Errors on polling two systems after upgrading

[kmayhue]

Hi, I have 2 devices out of 24 that will not poll after upgrading from 1.1.34 to 1.1.38. I've tried recreating the graphs and updating lib\snmp.php from development, but still get the below errors. I'm using CMD.PHP to do the polling and I can do an SNMP walk/Cacti Query with no issue on these failing devices. Everything else appears to be working. Any help would be greatly appreciated!

Operating System: Windows server 2016
Webserver: IIS10
Cacti:1.1.38 *upgraded from 1.1.34*
Spine:1.1.28
MySQL:5.7.20
PHP: 7.1.10
RRDTool (Cygwin or Win32 build):
Net-SNMP:
Cygwin (cygwin1.dll version):2.882

2018/07/20 11:54:32 - RECACHE STATS: Poller: RecacheTime:16.7699 DevicesRecached:2
2018/07/20 11:54:24 - PCOMMAND Device[29] Device[Bxxxx] WARNING: Recache Event Detected for Device
2018/07/20 11:54:16 - SYSTEM THOLD STATS: Time:0.5733 Tholds:27 TotalDevices:24 DownDevices:0 NewDownDevices:0
2018/07/20 11:54:16 - PCOMMAND Device[28] Device[x.x.x..197] WARNING: Recache Event Detected for Device
2018/07/20 11:54:14 - SYSTEM STATS: Time:13.0251 Method:cmd.php Processes:4 Threads:N/A Hosts:24 HostsPerProcess:6 DataSources:573 RRDsProcessed:318
2018/07/20 11:54:07 - POLLER: Poller[Main Poller] ASSERT: '87665511<10:3:35:28.77' failed. Recaching host 'Bxxxx', data query #5
2018/07/20 11:54:07 - POLLER: Poller[Main Poller] ASSERT: '87665210<10:3:35:28.77' failed. Recaching host 'Bxxxx', data query #4
2018/07/20 11:54:07 - POLLER: Poller[Main Poller] ASSERT: '87664702<10:3:35:28.75' failed. Recaching host 'Bxxxx', data query #1
2018/07/20 11:54:06 - POLLER: Poller[Main Poller] ASSERT: '9819849<1:3:21:18.92' failed. Recaching host 'x.x.x.197', data query #5
2018/07/20 11:54:06 - POLLER: Poller[Main Poller] ASSERT: '9819544<1:3:21:18.90' failed. Recaching host 'x.x.x.197', data query #4
2018/07/20 11:54:06 - POLLER: Poller[Main Poller] ASSERT: '9819012<1:3:21:18.83' failed. Recaching host 'x.x.x.197', data query #1
2018/07/20 11:49:33 - RECACHE STATS: Poller: RecacheTime:17.4015 DevicesRecached:2
2018/07/20 11:49:25 - PCOMMAND Device[29] Device[Bxxxx] WARNING: Recache Event Detected for Device
2018/07/20 11:49:17 - SYSTEM THOLD STATS: Time:0.2921 Tholds:27 TotalDevices:24 DownDevices:0 NewDownDevices:0
2018/07/20 11:49:16 - PCOMMAND Device[28] Device[x.x.x..197] WARNING: Recache Event Detected for Device
2018/07/20 11:49:15 - SYSTEM STATS: Time:14.1789 Method:cmd.php Processes:4 Threads:N/A Hosts:24 HostsPerProcess:6 DataSources:573 RRDsProcessed:318
2018/07/20 11:49:08 - POLLER: Poller[Main Poller] ASSERT: '87635356<10:3:30:29.49' failed. Recaching host 'Bxxxx', data query #5
2018/07/20 11:49:08 - POLLER: Poller[Main Poller] ASSERT: '87635037<10:3:30:29.47' failed. Recaching host 'Bxxxx', data query #4
2018/07/20 11:49:08 - POLLER: Poller[Main Poller] ASSERT: '87634476<10:3:30:29.46' failed. Recaching host 'Bxxxx', data query #1
2018/07/20 11:49:06 - POLLER: Poller[Main Poller] ASSERT: '9789620<1:3:16:19.58' failed. Recaching host 'x.x.x.197', data query #5
2018/07/20 11:49:06 - POLLER: Poller[Main Poller] ASSERT: '9789300<1:3:16:19.57' failed. Recaching host 'x.x.x.197', data query #4
2018/07/20 11:49:06 - POLLER: Poller[Main Poller] ASSERT: '9788722<1:3:16:19.43' failed. Recaching host 'x.x.x.197', data query #1

Debug:
2018/07/20 12:54:24 - RECACHE STATS: Poller: RecacheTime:9.4311 DevicesRecached:1
2018/07/20 12:54:15 - SYSTEM THOLD STATS: Time:0.3754 Tholds:27 TotalDevices:24 DownDevices:0 NewDownDevices:0
2018/07/20 12:54:15 - PCOMMAND Device[29] Device[Bxxxx] WARNING: Recache Event Detected for Device
2018/07/20 12:54:14 - SYSTEM STATS: Time:12.9408 Method:cmd.php Processes:4 Threads:N/A Hosts:24 HostsPerProcess:6 DataSources:573 RRDsProcessed:318
2018/07/20 12:54:07 - POLLER: Poller[Main Poller] Device[29] Device[Bxxxx] Graphs[Bxxxx - Traffic - Backup, Bxxxx - Traffic - Backup] DS[Bxxxx - Traffic - x.x.x.51 - ethernet_32770] SNMP: v2: Bxxxx, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.5, output: 787773737
2018/07/20 12:54:07 - POLLER: Poller[Main Poller] Device[29] Device[Bxxxx] Graphs[Bxxxx - Traffic - Backup, Bxxxx - Traffic - Backup] DS[Bxxxx - Traffic - x.x.x.51 - ethernet_32770] SNMP: v2: Bxxxx, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.5, output: 930464745
2018/07/20 12:54:07 - POLLER: Poller[Main Poller] Device[29] Device[Bxxxx] Graphs[Bxxxx - Traffic - Ethernet0, Bxxxx - Traffic - Ethernet0] DS[Bxxxx - Traffic - x.x.x.147 - ethernet_32769] SNMP: v2: Bxxxx, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.2, output: 683501720
2018/07/20 12:54:07 - POLLER: Poller[Main Poller] Device[29] Device[Bxxxx] Graphs[Bxxxx - Traffic - Ethernet0, Bxxxx - Traffic - Ethernet0] DS[Bxxxx - Traffic - x.x.x.147 - ethernet_32769] SNMP: v2: Bxxxx, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.2, output: 2232675790
2018/07/20 12:54:07 - POLLER: Poller[Main Poller] Device[29] Device[Bxxxx] Graphs[Bxxxx - CPU Utilization - CPUTotal, Bxxxx - CPU Utilization - CPUTotal] DS[Bxxxx - CPU Utilization - CPUTotal] SERVER: C:\inetpub\wwwroot\cacti\scripts\ss_host_cpu.php ss_host_cpu "Bxxxx" "29" "2:1611:10:800Numbers::::::" "get" "usage" "4000", output: 0
2018/07/20 12:54:07 - POLLER: Poller[Main Poller] Device[29] Device[Bxxxx] Graphs[Bxxxx - Used Space - Physical Memory, Bxxxx - Used Space - Physical Memory] DS[Bxxxx - Used Space - Physical Memory] SERVER: C:\inetpub\wwwroot\cacti\scripts\ss_host_disk.php ss_host_disk "Bxxxx" "29" "2:1611:10:800Numbers::::::" "get" "used" "5", output: 1254752256
2018/07/20 12:54:07 - POLLER: Poller[Main Poller] Device[29] Device[Bxxxx] Graphs[Bxxxx - Used Space - Physical Memory, Bxxxx - Used Space - Physical Memory] DS[Bxxxx - Used Space - Physical Memory] SERVER: C:\inetpub\wwwroot\cacti\scripts\ss_host_disk.php ss_host_disk "Bxxxx" "29" "2:1611:10:800Numbers::::::" "get" "total" "5", output: 4294443008
2018/07/20 12:54:07 - POLLER: Poller[Main Poller] Device[29] Device[Bxxxx] Graphs[Bxxxx - Used Space - C: Label: Serial Number 14fe7cf0, Bxxxx - Used Space - C: Label: Serial Number 14fe7cf0] DS[Bxxxx - Used Space - C: Label: Serial Number 14fe7cf0] SERVER: C:\inetpub\wwwroot\cacti\scripts\ss_host_disk.php ss_host_disk "Bxxxx" "29" "2:1611:10:800Numbers::::::" "get" "used" "2", output: 24069681152
2018/07/20 12:54:07 - POLLER: Poller[Main Poller] Device[29] Device[Bxxxx] Graphs[Bxxxx - Used Space - C: Label: Serial Number 14fe7cf0, Bxxxx - Used Space - C: Label: Serial Number 14fe7cf0] DS[Bxxxx - Used Space - C: Label: Serial Number 14fe7cf0] SERVER: C:\inetpub\wwwroot\cacti\scripts\ss_host_disk.php ss_host_disk "Bxxxx" "29" "2:1611:10:800Numbers::::::" "get" "total" "2", output: 53160701952
2018/07/20 12:54:06 - POLLER: Poller[Main Poller] Device[29] Device[Bxxxx] NOTICE: Spike Kill in Effect for 'Bxxxx'.
2018/07/20 12:54:06 - POLLER: Poller[Main Poller] ASSERT: '88039804<10:4:35:28.06' failed. Recaching host 'Bxxxx', data query #5
2018/07/20 12:54:06 - POLLER: Poller[Main Poller] Device[29] Device[Bxxxx] RECACHE DQ[SNMP - Get Processor Information] OID: .1.3.6.1.2.1.1.3.0, output: 10:4:35:28.06
2018/07/20 12:54:06 - POLLER: Poller[Main Poller] Device[29] Device[Bxxxx] NOTICE: Spike Kill in Effect for 'Bxxxx'.
2018/07/20 12:54:06 - POLLER: Poller[Main Poller] ASSERT: '88038896<10:4:35:28.05' failed. Recaching host 'Bxxxx', data query #4
2018/07/20 12:54:06 - POLLER: Poller[Main Poller] Device[29] Device[Bxxxx] RECACHE DQ[SNMP - Get Mounted Partitions] OID: .1.3.6.1.2.1.1.3.0, output: 10:4:35:28.05
2018/07/20 12:54:06 - POLLER: Poller[Main Poller] Device[29] Device[Bxxxx] NOTICE: Spike Kill in Effect for 'Bxxxx'.
2018/07/20 12:54:06 - POLLER: Poller[Main Poller] ASSERT: '88040082<10:4:35:28.01' failed. Recaching host 'Bxxxx', data query #1
2018/07/20 12:54:06 - POLLER: Poller[Main Poller] Device[29] Device[Bxxxx] RECACHE DQ[SNMP - Interface Statistics] OID: .1.3.6.1.2.1.1.3.0, output: 10:4:35:28.01
2018/07/20 12:54:06 - POLLER: Poller[Main Poller] Device[29] Device[Bxxxx] RECACHE: Processing 3 items in the auto reindex cache for 'Bxxxx'.
2018/07/20 12:54:06 - POLLER: Poller[Main Poller] Device[29] Device[Bxxxx] STATUS: Device 'Bxxxx' is UP.

[kmayhue]

After researching this more thoroughly in github, I saw a reference to the table poller_reindex having mixed formats (D:H:M:S and seconds) for the assert_value field, which I did see in my data. So I reverted the PHP modules cmd.php and snmp.php to version 1.1.35, which worked after two polling attempts. Please reference SjonHortensius posts at https://github.com/Cacti/cacti/issues/1634. Thank you SjonHortensius for the information!