Graph sudden choking

[areast]

Dear All

Currently user Windows 2008 R2 with cacti 0.8.8c
Found one of the graph sudden choking last few days ago

Found one abnormal log message
CMDPHP: Poller[0] ASSERT: '3430407287<' failed. Recaching host 'xxx.xxx.xx.xx', data query #43

turn on graph debug
d:/bin/rrdtool/rrdtool.exe graph - \
--imgformat=PNG \
--start="-86400" \
--end="-300" \
--title="Internet Router - Gi0/0" \
--rigid \
--base="1000" \
--height="120" \
--width="500" \
--alt-autoscale-max \
--lower-limit="0" \
--vertical-label="bits per second" \
--slope-mode \
--font TITLE:10: \
--font AXIS:7: \
--font LEGEND:8: \
--font UNIT:7: \
DEF:a="D\:/Inetpub/wwwroot/cacti/rra/internet_router_traffic_in_26014.rrd":"traffic_in":AVERAGE \
DEF:b="D\:/Inetpub/wwwroot/cacti/rra/internet_router_traffic_in_26014.rrd":"traffic_out":AVERAGE \
CDEF:cdefa="a,8,*" \
CDEF:cdefe="b,8,*" \
AREA:cdefa#00CF00FF:"Inbound" \
GPRINT:cdefa:LAST:" Current\:%8.2lf %s" \
GPRINT:cdefa:AVERAGE:"Average\:%8.2lf %s" \
GPRINT:cdefa:MAX:"Maximum\:%8.2lf %s\n" \
LINE1:cdefe#002A97FF:"Outbound" \
GPRINT:cdefe:LAST:"Current\:%8.2lf %s" \
GPRINT:cdefe:AVERAGE:"Average\:%8.2lf %s" \
GPRINT:cdefe:MAX:"Maximum\:%8.2lf %s\n" \
COMMENT:" \n" \
HRULE:211702007.71#FF0000FF:"95th Percentile" \
COMMENT:"(211.7 mbit in+out)"

[areast]

Also turn on DEBUG mode while the graph choking

sometime traffic_in output will become U
12/08/2015 02:26:18 PM - CMDPHP: Poller[0] Host[1440] DS[26015] SNMP: v3: <<IP ADDRESS>>, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.3, output: U

beside it also recache or reindex every polling ????
CMDPHP: Poller[0] Host[1440] RECACHE: Processing 3 items in the auto reindex cache

12/08/2015 02:41:26 PM - CMDPHP: Poller[0] Host[1440] DS[26015] SNMP: v3: <<IP ADDRESS>>, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.3, output: 193333730488754
12/08/2015 02:41:25 PM - CMDPHP: Poller[0] Host[1440] DS[26015] SNMP: v3: <<IP ADDRESS>>, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.3, output: 180565839811451
12/08/2015 02:41:25 PM - CMDPHP: Poller[0] Host[1440] DS[26014] SNMP: v3: <<IP ADDRESS>>, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.2, output: 180050119954309
12/08/2015 02:41:25 PM - CMDPHP: Poller[0] Host[1440] DS[26014] SNMP: v3: <<IP ADDRESS>>, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.2, output: 193451087271846
12/08/2015 02:41:25 PM - CMDPHP: Poller[0] Host[1440] DS[26013] SNMP: v3: <<IP ADDRESS>>, dsname: poolUsed, oid: .1.3.6.1.4.1.9.9.48.1.1.1.5.2, output: U
12/08/2015 02:41:25 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'<<IP ADDRESS>>', and OID:'.1.3.6.1.4.1.9.9.48.1.1.1.5.2'
12/08/2015 02:40:49 PM - CMDPHP: Poller[0] Host[1440] DS[26013] SNMP: v3: <<IP ADDRESS>>, dsname: poolFree, oid: .1.3.6.1.4.1.9.9.48.1.1.1.6.2, output: 43859264
12/08/2015 02:40:49 PM - CMDPHP: Poller[0] Host[1440] DS[26013] SNMP: v3: <<IP ADDRESS>>, dsname: poolLargest, oid: .1.3.6.1.4.1.9.9.48.1.1.1.7.2, output: 23110108
12/08/2015 02:40:49 PM - CMDPHP: Poller[0] Host[1440] DS[26012] SNMP: v3: <<IP ADDRESS>>, dsname: poolUsed, oid: .1.3.6.1.4.1.9.9.48.1.1.1.5.1, output: 44534104
12/08/2015 02:40:25 PM - CMDPHP: Poller[0] Host[1440] DS[26012] SNMP: v3: <<IP ADDRESS>>, dsname: poolFree, oid: .1.3.6.1.4.1.9.9.48.1.1.1.6.1, output: 710399888
12/08/2015 02:40:24 PM - CMDPHP: Poller[0] Host[1440] DS[26012] SNMP: v3: <<IP ADDRESS>>, dsname: poolLargest, oid: .1.3.6.1.4.1.9.9.48.1.1.1.7.1, output: 674906948
12/08/2015 02:40:24 PM - CMDPHP: Poller[0] Host[1440] DS[26011] SNMP: v3: <<IP ADDRESS>>, dsname: 5min_cpu, oid: .1.3.6.1.4.1.9.9.109.1.1.1.1.5.1, output: 24
12/08/2015 02:40:24 PM - CMDPHP: Poller[0] Host[1440] RECACHE: Processing 3 items in the auto reindex cache for '<<IP ADDRESS>>'.
12/08/2015 02:36:10 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'<<IP ADDRESS>>', and OID:'.1.3.6.1.2.1.1.3.0'
12/08/2015 02:31:36 PM - CMDPHP: Poller[0] Host[1440] DS[26015] SNMP: v3: <<IP ADDRESS>>, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.3, output: 193322862780418
12/08/2015 02:31:36 PM - CMDPHP: Poller[0] Host[1440] DS[26015] SNMP: v3: <<IP ADDRESS>>, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.3, output: 180556520088872
12/08/2015 02:31:36 PM - CMDPHP: Poller[0] Host[1440] DS[26014] SNMP: v3: <<IP ADDRESS>>, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.2, output: 180040842651316
12/08/2015 02:31:36 PM - CMDPHP: Poller[0] Host[1440] DS[26014] SNMP: v3: <<IP ADDRESS>>, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.2, output: 193440212478069
12/08/2015 02:31:36 PM - CMDPHP: Poller[0] Host[1440] DS[26013] SNMP: v3: <<IP ADDRESS>>, dsname: poolUsed, oid: .1.3.6.1.4.1.9.9.48.1.1.1.5.2, output: 19055296
12/08/2015 02:31:35 PM - CMDPHP: Poller[0] Host[1440] DS[26013] SNMP: v3: <<IP ADDRESS>>, dsname: poolFree, oid: .1.3.6.1.4.1.9.9.48.1.1.1.6.2, output: 43859264
12/08/2015 02:31:35 PM - CMDPHP: Poller[0] Host[1440] DS[26013] SNMP: v3: <<IP ADDRESS>>, dsname: poolLargest, oid: .1.3.6.1.4.1.9.9.48.1.1.1.7.2, output: 23110108
12/08/2015 02:31:35 PM - CMDPHP: Poller[0] Host[1440] DS[26012] SNMP: v3: <<IP ADDRESS>>, dsname: poolUsed, oid: .1.3.6.1.4.1.9.9.48.1.1.1.5.1, output: U
12/08/2015 02:31:35 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'<<IP ADDRESS>>', and OID:'.1.3.6.1.4.1.9.9.48.1.1.1.5.1'
12/08/2015 02:30:59 PM - CMDPHP: Poller[0] Host[1440] DS[26012] SNMP: v3: <<IP ADDRESS>>, dsname: poolFree, oid: .1.3.6.1.4.1.9.9.48.1.1.1.6.1, output: 710399888
12/08/2015 02:30:59 PM - CMDPHP: Poller[0] Host[1440] DS[26011] SNMP: v3: <<IP ADDRESS>>, dsname: 5min_cpu, oid: .1.3.6.1.4.1.9.9.109.1.1.1.1.5.1, output: U
12/08/2015 02:30:59 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'<<IP ADDRESS>>', and OID:'.1.3.6.1.4.1.9.9.109.1.1.1.1.5.1'
12/08/2015 02:30:22 PM - CMDPHP: Poller[0] Host[1440] RECACHE: Processing 3 items in the auto reindex cache for '<<IP ADDRESS>>'.
12/08/2015 02:26:42 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'<<IP ADDRESS>>', and OID:'.1.3.6.1.2.1.1.1.0'
12/08/2015 02:26:18 PM - CMDPHP: Poller[0] Host[1440] DS[26015] SNMP: v3: <<IP ADDRESS>>, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.3, output: 193317625085348
12/08/2015 02:26:18 PM - CMDPHP: Poller[0] Host[1440] DS[26015] SNMP: v3: <<IP ADDRESS>>, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.3, output: U
12/08/2015 02:26:18 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'<<IP ADDRESS>>', and OID:'.1.3.6.1.2.1.31.1.1.1.6.3'
12/08/2015 02:26:06 PM - CMDPHP: Poller[0] WARNING: SNMP Walk Timeout for Host:'<<IP ADDRESS>>', and OID:'.1.3.6.1.2.1.25.1'
12/08/2015 02:25:42 PM - CMDPHP: Poller[0] Host[1440] DS[26014] SNMP: v3: <<IP ADDRESS>>, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.2, output: 180035711742082
12/08/2015 02:25:42 PM - CMDPHP: Poller[0] Host[1440] DS[26014] SNMP: v3: <<IP ADDRESS>>, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.2, output: 193434415303680
12/08/2015 02:25:42 PM - CMDPHP: Poller[0] Host[1440] DS[26013] SNMP: v3: <<IP ADDRESS>>, dsname: poolUsed, oid: .1.3.6.1.4.1.9.9.48.1.1.1.5.2, output: 19055296
12/08/2015 02:25:42 PM - CMDPHP: Poller[0] Host[1440] DS[26013] SNMP: v3: <<IP ADDRESS>>, dsname: poolFree, oid: .1.3.6.1.4.1.9.9.48.1.1.1.6.2, output: 43859264
12/08/2015 02:25:30 PM - CMDPHP: Poller[0] Host[1440] DS[26013] SNMP: v3: <<IP ADDRESS>>, dsname: poolLargest, oid: .1.3.6.1.4.1.9.9.48.1.1.1.7.2, output: 23110108
12/08/2015 02:25:29 PM - CMDPHP: Poller[0] Host[1440] DS[26012] SNMP: v3: <<IP ADDRESS>>, dsname: poolUsed, oid: .1.3.6.1.4.1.9.9.48.1.1.1.5.1, output: 44534104
12/08/2015 02:25:29 PM - CMDPHP: Poller[0] Host[1440] DS[26012] SNMP: v3: <<IP ADDRESS>>, dsname: poolFree, oid: .1.3.6.1.4.1.9.9.48.1.1.1.6.1, output: 710399888
12/08/2015 02:25:29 PM - CMDPHP: Poller[0] Host[1440] DS[26012] SNMP: v3: <<IP ADDRESS>>, dsname: poolLargest, oid: .1.3.6.1.4.1.9.9.48.1.1.1.7.1, output: 674906948
12/08/2015 02:25:29 PM - CMDPHP: Poller[0] Host[1440] DS[26011] SNMP: v3: <<IP ADDRESS>>, dsname: 5min_cpu, oid: .1.3.6.1.4.1.9.9.109.1.1.1.1.5.1, output: 22
12/08/2015 02:25:29 PM - CMDPHP: Poller[0] Host[1440] RECACHE: Processing 3 items in the auto reindex cache for '<<IP ADDRESS>>'.

[BSOD2600]

recaching typically happens when cacti detects the device has been rebooted or number of indexes have changed. you can also force this per Data Query by changing the Reindex method to: Verify all

[areast]

Thanks BSOD2600

I have just delete the "Associated Data Queries" under the problem HOST
from SNMP - Interface Statistics with 'Uptime Goes Backwards"
Re add SNMP - Interface Statistics with "Verify All Fields" then Verbose Query

Not sure if this is the correct way to reindex or needed to modify with php command prompt?

[BSOD2600]

Not sure if this is the correct way to reindex or needed to modify with php command prompt?

Yes, it's the correct method to change the reindex method setting.

you should now see this occurring for this host+/data source(s) every polling cycle in the cacti.log.