SNMP Queries Return Data, But Data Isn't Graphed

[DudemanJenkins]

After some troubleshooting with cacti, I set the poller to run every 5 minutes and reflected this in cron. After reindexing hosts and rebuilding the cache for spine, my graphs not get data (as shown under the graph) but no actual graph is created with this data. However, this is only on certain devices, (all using the standard bits/s traffic interface). Since the logs are telling me everything's fine and the permissions are correct, I'm having issues determining exactly what the problem is.

[DudemanJenkins]

Sorry I was so frustrated I barely gave the details, my apologies:

- Verbose Query of spine checks out, no errors and no ignored hosts
- RRD Path is working and the cacti site folder is 777
- poller (spine) is set to run as root
- SNMPwalk checks out and returns appropriate data

All of this started to happen when I added a large switch (~200 interfaces) and a Cisco router ~40 interfaces. Both poll and get data (the switch I had to set a snmp timout of 1200 for it to get data, but since set it back to 500 in leu of issues).

Any information or lead would be appreciated.

Thanks!

[BSOD2600]

already looked through http://docs.cacti.net/manual:088:4_help.2_debugging ?

With debugging logging enabled for cacti.log, there are a few steps to validate a specific data source is working:
1) cacti checks if the host is up or down.
2) the poller queries the device (snmp, script, etc) and the data is returned.
3) the collected and cached data is then inserted into the *.rrd file (RRDTOOL UPDATE command issue).

Are all the above occurring ?

[DudemanJenkins]

1) Cacti sees the hosts as up:


2) Poller seems to get data from the devices:

01/15/2013 09:20:08 AM - SYSTEM STATS: Time:6.1067 Method:spine Processes:4 Threads:40 Hosts:212 HostsPerProcess:53 DataSources:14873 RRDsProcessed:5492
01/15/2013 09:20:08 AM - SPINE: Poller[0] Host[536] TH[1] Total Time: 4.2 Seconds
01/15/2013 09:20:08 AM - SPINE: Poller[0] Host[536] TH[1] DS[12445] SNMP: v2: sw1.thc.***************.net, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.219, value: 0
01/15/2013 09:20:08 AM - SPINE: Poller[0] Host[536] TH[1] DS[12448] SNMP: v2: sw1.thc.***************.net, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.222, value: 0
01/15/2013 09:20:08 AM - SPINE: Poller[0] Host[536] TH[1] DS[12453] SNMP: v2: sw1.thc.***************.net, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.227, value: 0
01/15/2013 09:20:08 AM - SPINE: Poller[0] Host[536] TH[1] DS[12322] SNMP: v2: sw1.thc.***************.net, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.60, value: 1061768647
01/15/2013 09:20:08 AM - SPINE: Poller[0] Host[536] TH[1] DS[12459] SNMP: v2: sw1.thc.***************.net, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.261, value: 11024
01/15/2013 09:20:08 AM - SPINE: Poller[0] Host[536] TH[1] DS[12464] SNMP: v2: sw1.thc.***************.net, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.266, value: 0
01/15/2013 09:20:08 AM - SPINE: Poller[0] Host[536] TH[1] DS[12464] SNMP: v2: sw1.thc.***************.net, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.266, value: 0
01/15/2013 09:20:08 AM - SPINE: Poller[0] Host[536] TH[1] DS[12465] SNMP: v2: sw1.thc.***************.net, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.267, value: 403264737
01/15/2013 09:20:08 AM - SPINE: Poller[0] Host[536] TH[1] DS[12465] SNMP: v2: sw1.thc.***************.net, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.267, value: 734645735
01/15/2013 09:20:08 AM - SPINE: Poller[0] Host[536] TH[1] DS[12466] SNMP: v2: sw1.thc.***************.net, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.268, value: 2400975992
01/15/2013 09:20:08 AM - SPINE: Poller[0] Host[536] TH[1] DS[12466] SNMP: v2: sw1.thc.***************.net, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.268, value: 278000460
01/15/2013 09:20:08 AM - SPINE: Poller[0] Host[536] TH[1] DS[12467] SNMP: v2: sw1.thc.***************.net, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.269, value: 0
01/15/2013 09:20:08 AM - SPINE: Poller[0] Host[536] TH[1] DS[12467] SNMP: v2: sw1.thc.***************.net, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.269, value: 0
01/15/2013 09:20:08 AM - SPINE: Poller[0] Host[536] TH[1] DS[12468] SNMP: v2: sw1.thc.***************.net, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.270, value: 0
01/15/2013 09:20:08 AM - SPINE: Poller[0] Host[536] TH[1] DS[12468] SNMP: v2: sw1.thc.***************.net, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.270, value: 0
01/15/2013 09:20:08 AM - SPINE: Poller[0] Host[536] TH[1] DS[12469] SNMP: v2: sw1.thc.***************.net, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.271, value: 0
01/15/2013 09:20:08 AM - SPINE: Poller[0] Host[536] TH[1] DS[12469] SNMP: v2: sw1.thc.***************.net, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.271, value: 0
01/15/2013 09:20:08 AM - SPINE: Poller[0] Host[536] TH[1] DS[12444] SNMP: v2: sw1.thc.***************.net, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.218, value: 0
01/15/2013 09:20:08 AM - SPINE: Poller[0] Host[536] TH[1] DS[12516] SNMP: v2: sw1.thc.***************.net, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.288, value: 0
01/15/2013 09:20:08 AM - SPINE: Poller[0] Host[536] TH[1] DS[12427] SNMP: v2: sw1.thc.***************.net, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.197, value: 0
01/15/2013 09:20:08 AM - SPINE: Poller[0] Host[536] TH[1] DS[12446] SNMP: v2: sw1.thc.***************.net, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.220, value: 0
01/15/2013 09:20:08 AM - SPINE: Poller[0] Host[536] TH[1] DS[12470] SNMP: v2: sw1.thc.***************.net, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.272, value: 0
01/15/2013 09:20:08 AM - SPINE: Poller[0] Host[536] TH[1] DS[12348] SNMP: v2: sw1.thc.***************.net, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.86, value: 0
01/15/2013 09:20:08 AM - SPINE: Poller[0] Host[536] TH[1] DS[12499] SNMP: v2: sw1.thc.***************.net, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.271, value: 0
01/15/2013 09:20:08 AM - SPINE: Poller[0] Host[536] TH[1] DS[12512] SNMP: v2: sw1.thc.***************.net, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.284, value: 0
01/15/2013 09:20:08 AM - SPINE: Poller[0] Host[536] TH[1] DS[12462] SNMP: v2: sw1.thc.***************.net, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.264, value: 1570454898
01/15/2013 09:20:08 AM - SPINE: Poller[0] Host[536] TH[1] DS[12438] SNMP: v2: sw1.thc.***************.net, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.212, value: 0
01/15/2013 09:20:08 AM - SPINE: Poller[0] Host[536] TH[1] DS[12463] SNMP: v2: sw1.thc.***************.net, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.265, value: 0
01/15/2013 09:20:08 AM - SPINE: Poller[0] Host[536] TH[1] DS[12457] SNMP: v2: sw1.thc.***************.net, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.231, value: 0
01/15/2013 09:20:08 AM - SPINE: Poller[0] Host[536] TH[1] DS[12429] SNMP: v2: sw1.thc.***************.net, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.199, value: 0
01/15/2013 09:20:08 AM - SPINE: Poller[0] Host[536] TH[1] DS[12500] SNMP: v2: sw1.thc.***************.net, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.272, value: 0
01/15/2013 09:20:08 AM - SPINE: Poller[0] Host[536] TH[1] DS[12516] SNMP: v2: sw1.thc.***************s.net, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.288, value: 0
01/15/2013 09:20:08 AM - SPINE: Poller[0] Host[536] TH[1] DS[12480] SNMP: v2: sw1.thc.***************.net, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.282, value: 0
01/15/2013 09:20:08 AM - SPINE: Poller[0] Host[536] TH[1] DS[12515] SNMP: v2: sw1.thc.***************.net, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.287, value: 0
01/15/2013 09:20:07 AM - SPINE: Poller[0] Host[536] TH[1] DS[12510] SNMP: v2: sw1.thc.***************.net, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.282, value: 0
01/15/2013 09:20:07 AM - SPINE: Poller[0] Host[536] TH[1] DS[12488] SNMP: v2: sw1.thc.***************.net, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.290, value: 0
01/15/2013 09:20:07 AM - SPINE: Poller[0] Host[536] TH[1] DS[12491] SNMP: v2: sw1.thc.***************.net, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.293, value: 0
01/15/2013 09:20:07 AM - SPINE: Poller[0] Host[536] TH[1] DS[12482] SNMP: v2: sw1.thc.***************.net, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.284, value: 0
01/15/2013 09:20:07 AM - SPINE: Poller[0] Host[536] TH[1] DS[12484] SNMP: v2: sw1.thc.***************.net, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.286, value: 0
01/15/2013 09:20:07 AM - SPINE: Poller[0] Host[536] TH[1] DS[12481] SNMP: v2: sw1.thc.***************.net, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.283, value: 0
01/15/2013 09:20:07 AM - SPINE: Poller[0] Host[536] TH[1] DS[12514] SNMP: v2: sw1.thc.***************.net, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.286, value: 0
01/15/2013 09:20:07 AM - SPINE: Poller[0] Host[536] TH[1] DS[12504] SNMP: v2: sw1.thc.***************.net, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.276, value: 0
01/15/2013 09:20:07 AM - SPINE: Poller[0] Host[536] TH[1] DS[12503] SNMP: v2: sw1.thc.***************.net, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.275, value: 0
01/15/2013 09:20:07 AM - SPINE: Poller[0] Host[536] TH[1] DS[12515] SNMP: v2: sw1.thc.***************.net, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.287, value: 0
01/15/2013 09:20:07 AM - SPINE: Poller[0] Host[536] TH[1] DS[12324] SNMP: v2: sw1.thc.***************.net, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.62, value: 0
01/15/2013 09:20:07 AM - SPINE: Poller[0] Host[536] TH[1] DS[12456] SNMP: v2: sw1.thc.***************.net, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.230, value: 0
01/15/2013 09:20:07 AM - SPINE: Poller[0] Host[536] TH[1] DS[12501] SNMP: v2: sw1.thc.***************.net, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.273, value: 1796788186
01/15/2013 09:20:07 AM - SPINE: Poller[0] Host[536] TH[1] DS[12511] SNMP: v2: sw1.thc.***************.net, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.283, value: 0
01/15/2013 09:20:07 AM - SPINE: Poller[0] Host[536] TH[1] DS[12507] SNMP: v2: sw1.thc.***************.net, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.279, value: 0
01/15/2013 09:20:07 AM - SPINE: Poller[0] Host[536] TH[1] DS[12513] SNMP: v2: sw1.thc.***************.net, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.285, value: 0
01/15/2013 09:20:07 AM - SPINE: Poller[0] Host[536] TH[1] DS[12450] SNMP: v2: sw1.thc.***************.net, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.224, value: 0
01/15/2013 09:20:07 AM - SPINE: Poller[0] Host[536] TH[1] DS[12474] SNMP: v2: sw1.thc.***************.net, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.276, value: 0
01/15/2013 09:20:07 AM - SPINE: Poller[0] Host[536] TH[1] DS[12470] SNMP: v2: sw1.thc.***************.net, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.272, value: 0
01/15/2013 09:20:07 AM - SPINE: Poller[0] Host[536] TH[1] DS[12493] SNMP: v2: sw1.thc.***************.net, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.295, value: 0
01/15/2013 09:20:07 AM - SPINE: Poller[0] Host[536] TH[1] DS[12472] SNMP: v2: sw1.thc.***************.net, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.274, value: 1406781218
01/15/2013 09:20:07 AM - SPINE: Poller[0] Host[536] TH[1] DS[12495] SNMP: v2: sw1.thc.***************.net, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.297, value: 0
01/15/2013 09:20:07 AM - SPINE: Poller[0] Host[536] TH[1] DS[12508] SNMP: v2: sw1.thc.***************.net, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.280, value: 0
01/15/2013 09:20:07 AM - SPINE: Poller[0] Host[536] TH[1] DS[12504] SNMP: v2: sw1.thc.***************.net, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.276, value: 0
01/15/2013 09:20:07 AM - SPINE: Poller[0] Host[536] TH[1] DS[12448] SNMP: v2: sw1.thc.***************.net, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.222, value: 0
01/15/2013 09:20:07 AM - SPINE: Poller[0] Host[536] TH[1] DS[12486] SNMP: v2: sw1.thc.***************.net, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.288, value: 0
01/15/2013 09:20:07 AM - SPINE: Poller[0] Host[536] TH[1] DS[12507] SNMP: v2: sw1.thc.***************.net, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.279, value: 0
01/15/2013 09:20:07 AM - SPINE: Poller[0] Host[536] TH[1] DS[12478] SNMP: v2: sw1.thc.***************.net, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.280, value: 0
01/15/2013 09:20:07 AM - SPINE: Poller[0] Host[536] TH[1] DS[12506] SNMP: v2: sw1.thc.***************.net, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.278, value: 0
01/15/2013 09:20:07 AM - SPINE: Poller[0] Host[536] TH[1] DS[12494] SNMP: v2: sw1.thc.***************.net, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.296, value: 0
01/15/2013 09:20:07 AM - SPINE: Poller[0] Host[584] TH[1] Total Time: 3.6 Seconds
01/15/2013 09:20:07 AM - SPINE: Poller[0] Host[584] TH[1] NOTE: There are '26' Polling Items for this Host

But even though it's returning data, I get this:



3) The RRD directory (/usr/share/cacti/site/rra/ ?) seems to have the right permissions, as the spine process is running as root:

drwxrwxrwx 2 root root 4096 Jan 14 14:28 rra

However nothing seems to be in that directory, even when I try to restructure the paths, nothing is created and I get a 0 output. From what I've read in other posts, these files are generated with the poller. The cache is also populated for it.

I have the poller cron'd for every 5 minutes:

*/5 * * * * php --define suhosin.memory_limit=512M /usr/share/cacti/site/poller.php 2>&1 >/dev/null | if [ -f /usr/bin/ts ] ; then ts ; else tee ; fi >> /var/log/cacti/poller-error.log

Thank you for your help with this.

[gandalf]

Are you running on ubuntu/debian? See paulgevers instructions for that platform, then. They restructured some paths and this might be related
R.

[DudemanJenkins]

I've checked over the ubuntu guide. This did help me locate the true RRD path (/var/lib/cacti/rra/) However the permissions seem to be correct for here : -rwxrwxrwx 1 www-data www-data

I'm running the logs in debug, and from what it's showing it seems the CACTI2RRD program is working:

01/16/2013 09:35:08 AM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /var/lib/cacti/rra/536/12428.rrd --template traffic_in:traffic_out 1358346903:827102689:2919986286
01/16/2013 09:35:08 AM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /var/lib/cacti/rra/536/12431.rrd --template traffic_in:traffic_out 1358346903:176771667:1199838929
01/16/2013 09:35:08 AM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /var/lib/cacti/rra/536/12434.rrd --template traffic_out:traffic_in 1358346903:0:0
01/16/2013 09:35:08 AM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /var/lib/cacti/rra/536/12435.rrd --template traffic_out:traffic_in 1358346903:0:0
01/16/2013 09:35:08 AM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /var/lib/cacti/rra/536/12436.rrd --template traffic_out:traffic_in 1358346903:956168297:3698493779
01/16/2013 09:35:08 AM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /var/lib/cacti/rra/536/12441.rrd --template traffic_out:traffic_in 1358346903:1480897394:1404713402
01/16/2013 09:35:08 AM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /var/lib/cacti/rra/536/12442.rrd --template traffic_out:traffic_in 1358346903:3180029008:1328226963
01/16/2013 09:35:08 AM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /var/lib/cacti/rra/536/12383.rrd --template traffic_out:traffic_in 1358346903:0:0
01/16/2013 09:35:08 AM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /var/lib/cacti/rra/536/12382.rrd --template traffic_out:traffic_in 1358346903:0:0
01/16/2013 09:35:08 AM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /var/lib/cacti/rra/536/12381.rrd --template traffic_out:traffic_in 1358346903:2231952552:268458896
01/16/2013 09:35:08 AM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /var/lib/cacti/rra/536/12380.rrd --template traffic_out:traffic_in 1358346903:0:0
01/16/2013 09:35:08 AM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /var/lib/cacti/rra/536/12377.rrd --template traffic_in:traffic_out 1358346903:0:0
01/16/2013 09:35:08 AM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /var/lib/cacti/rra/536/12376.rrd --template traffic_out:traffic_in 1358346903:863394996:3733448477
01/16/2013 09:35:08 AM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /var/lib/cacti/rra/536/12374.rrd --template traffic_out:traffic_in 1358346903:0:0
01/16/2013 09:35:08 AM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /var/lib/cacti/rra/536/12373.rrd --template traffic_in:traffic_out 1358346903:0:0
01/16/2013 09:35:08 AM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /var/lib/cacti/rra/536/12372.rrd --template traffic_out:traffic_in 1358346903:0:0

I've checked the database for errors using the cli script and it checked out.

Also it seems that many more of my interface graphs have stopped graphing as I've searched for and attempted fixes based on older forum posts, but the nature of the new problems are identical to the first.

[DudemanJenkins]

I've attempted re-creating a large amount of these graphs, however the ones that graph feel like they're chosen at random, it will recognize the interface and even the alias, but when it polls it, it gets a "U" return. Could this possibly be due to the fact I'm using Bits with Total Bandwith? Is that not supported on all devices/interfaces?

[gandalf]

To my knowledge, all interfaces in ifOperStatus "up" should report traffic data. To debug this, the debugging link listed above will help
R.

[DudemanJenkins]

SNMP Manual Query:

root@XXXXX:/var/log/cacti# snmpget -c XXXXXXXX -v 2c sw1.thc.XXXXXXXXX.net .1.3.6.1.2.1.2.2.1.10.60
IF-MIB::ifInOctets.60 = Counter32: 1081597487

The related data source:


The Graph that's generated:



RRD Fetch Average:

root@XXXXX:/usr/local/spine/bin# rrdtool fetch /var/lib/cacti/rra/536/12322.rrd AVERAGE
traffic_in traffic_out

1358348400: -nan -nan
1358348760: -nan -nan
1358349120: -nan -nan
1358349480: -nan -nan
1358349840: -nan -nan
1358350200: -nan -nan
1358350560: -nan -nan
1358350920: -nan -nan
1358351280: -nan -nan
1358351640: -nan -nan
1358352000: -nan -nan
1358352360: -nan -nan
1358352720: -nan -nan
1358353080: -nan -nan
1358353440: -nan -nan
1358353800: -nan -nan
1358354160: -nan -nan
1358354520: -nan -nan
1358354880: -nan -nan
1358355240: -nan -nan
1358355600: -nan -nan
1358355960: -nan -nan
1358356320: -nan -nan
1358356680: -nan -nan
1358357040: -nan -nan
1358357400: -nan -nan
1358357760: -nan -nan
1358358120: -nan -nan
1358358480: -nan -nan
1358358840: -nan -nan
1358359200: -nan -nan
1358359560: -nan -nan
1358359920: -nan -nan
1358360280: -nan -nan
1358360640: -nan -nan
1358361000: -nan -nan
1358361360: -nan -nan
1358361720: -nan -nan
1358362080: -nan -nan
1358362440: -nan -nan
1358362800: -nan -nan
1358363160: -nan -nan
1358363520: -nan -nan
1358363880: -nan -nan
1358364240: -nan -nan
1358364600: -nan -nan
1358364960: -nan -nan
1358365320: -nan -nan
1358365680: -nan -nan
1358366040: -nan -nan
1358366400: -nan -nan
1358366760: -nan -nan
1358367120: -nan -nan
1358367480: -nan -nan
1358367840: -nan -nan
1358368200: -nan -nan
1358368560: -nan -nan
1358368920: -nan -nan
1358369280: -nan -nan
1358369640: -nan -nan
1358370000: -nan -nan
1358370360: -nan -nan
1358370720: -nan -nan
1358371080: -nan -nan
1358371440: -nan -nan
1358371800: -nan -nan
1358372160: -nan -nan
1358372520: -nan -nan
1358372880: -nan -nan
1358373240: -nan -nan
1358373600: -nan -nan
1358373960: -nan -nan
1358374320: -nan -nan
1358374680: -nan -nan
1358375040: -nan -nan
1358375400: -nan -nan
1358375760: -nan -nan
1358376120: -nan -nan
1358376480: -nan -nan
1358376840: -nan -nan
1358377200: -nan -nan
1358377560: -nan -nan
1358377920: -nan -nan
1358378280: -nan -nan
1358378640: -nan -nan
1358379000: -nan -nan
1358379360: -nan -nan
1358379720: -nan -nan
1358380080: -nan -nan
1358380440: -nan -nan
1358380800: -nan -nan
1358381160: -nan -nan
1358381520: -nan -nan
1358381880: -nan -nan
1358382240: -nan -nan
1358382600: -nan -nan
1358382960: -nan -nan
1358383320: -nan -nan
1358383680: -nan -nan
1358384040: -nan -nan
1358384400: -nan -nan
1358384760: -nan -nan
1358385120: -nan -nan
1358385480: -nan -nan
1358385840: -nan -nan
1358386200: -nan -nan
1358386560: -nan -nan
1358386920: -nan -nan
1358387280: -nan -nan
1358387640: -nan -nan
1358388000: -nan -nan
1358388360: -nan -nan
1358388720: -nan -nan
1358389080: -nan -nan
1358389440: -nan -nan
1358389800: -nan -nan
1358390160: -nan -nan
1358390520: -nan -nan
1358390880: -nan -nan
1358391240: -nan -nan
1358391600: -nan -nan
1358391960: -nan -nan
1358392320: -nan -nan
1358392680: -nan -nan
1358393040: -nan -nan
1358393400: -nan -nan
1358393760: -nan -nan
1358394120: -nan -nan
1358394480: -nan -nan
1358394840: -nan -nan
1358395200: -nan -nan
1358395560: -nan -nan
1358395920: -nan -nan
1358396280: -nan -nan
1358396640: -nan -nan
1358397000: -nan -nan
1358397360: -nan -nan
1358397720: -nan -nan
1358398080: -nan -nan
1358398440: -nan -nan
1358398800: -nan -nan
1358399160: -nan -nan
1358399520: -nan -nan
1358399880: -nan -nan
1358400240: -nan -nan
1358400600: -nan -nan
1358400960: -nan -nan
1358401320: -nan -nan
1358401680: -nan -nan
1358402040: -nan -nan
1358402400: -nan -nan
1358402760: -nan -nan
1358403120: -nan -nan
1358403480: -nan -nan
1358403840: -nan -nan
1358404200: -nan -nan
1358404560: -nan -nan
1358404920: -nan -nan
1358405280: -nan -nan
1358405640: -nan -nan
1358406000: -nan -nan
1358406360: -nan -nan
1358406720: -nan -nan
1358407080: -nan -nan
1358407440: -nan -nan
1358407800: -nan -nan
1358408160: -nan -nan
1358408520: -nan -nan
1358408880: -nan -nan
1358409240: -nan -nan
1358409600: -nan -nan
1358409960: -nan -nan
1358410320: -nan -nan
1358410680: -nan -nan
1358411040: -nan -nan
1358411400: -nan -nan
1358411760: -nan -nan
1358412120: -nan -nan
1358412480: -nan -nan
1358412840: -nan -nan
1358413200: -nan -nan
1358413560: -nan -nan
1358413920: -nan -nan
1358414280: -nan -nan
1358414640: -nan -nan
1358415000: -nan -nan
1358415360: -nan -nan
1358415720: -nan -nan
1358416080: -nan -nan
1358416440: -nan -nan
1358416800: -nan -nan
1358417160: -nan -nan
1358417520: -nan -nan
1358417880: -nan -nan
1358418240: -nan -nan
1358418600: -nan -nan
1358418960: -nan -nan
1358419320: -nan -nan
1358419680: -nan -nan
1358420040: -nan -nan
1358420400: -nan -nan
1358420760: -nan -nan
1358421120: -nan -nan
1358421480: -nan -nan
1358421840: -nan -nan
1358422200: -nan -nan
1358422560: -nan -nan
1358422920: -nan -nan
1358423280: -nan -nan
1358423640: -nan -nan
1358424000: -nan -nan
1358424360: -nan -nan
1358424720: -nan -nan
1358425080: -nan -nan
1358425440: -nan -nan
1358425800: -nan -nan
1358426160: -nan -nan
1358426520: -nan -nan
1358426880: -nan -nan
1358427240: -nan -nan
1358427600: -nan -nan
1358427960: -nan -nan
1358428320: -nan -nan
1358428680: -nan -nan
1358429040: -nan -nan
1358429400: -nan -nan
1358429760: -nan -nan
1358430120: -nan -nan
1358430480: -nan -nan
1358430840: -nan -nan
1358431200: -nan -nan
1358431560: -nan -nan
1358431920: -nan -nan
1358432280: -nan -nan
1358432640: -nan -nan
1358433000: -nan -nan
1358433360: -nan -nan
1358433720: -nan -nan
1358434080: -nan -nan
1358434440: -nan -nan
1358434800: -nan -nan

Since there's -nan's I did the rrdtool info:

root@XXXXXX:/usr/local/spine/bin# rrdtool info /var/lib/cacti/rra/536/12322.rrd
filename = "/var/lib/cacti/rra/536/12322.rrd"
rrd_version = "0003"
step = 60
last_update = 1358434804
header_size = 3488
ds[traffic_in].index = 0
ds[traffic_in].type = "COUNTER"
ds[traffic_in].minimal_heartbeat = 120
ds[traffic_in].min = 0.0000000000e+00
ds[traffic_in].max = 1.0000000000e+07
ds[traffic_in].last_ds = "1081624224"
ds[traffic_in].value = NaN
ds[traffic_in].unknown_sec = 4
ds[traffic_out].index = 1
ds[traffic_out].type = "COUNTER"
ds[traffic_out].minimal_heartbeat = 120
ds[traffic_out].min = 0.0000000000e+00
ds[traffic_out].max = 1.0000000000e+07
ds[traffic_out].last_ds = "2242853201"
ds[traffic_out].value = NaN
ds[traffic_out].unknown_sec = 4
rra[0].cf = "AVERAGE"
rra[0].rows = 500
rra[0].cur_row = 488
rra[0].pdp_per_row = 1
rra[0].xff = 5.0000000000e-01
rra[0].cdp_prep[0].value = NaN
rra[0].cdp_prep[0].unknown_datapoints = 0
rra[0].cdp_prep[1].value = NaN
rra[0].cdp_prep[1].unknown_datapoints = 0
rra[1].cf = "AVERAGE"
rra[1].rows = 600
rra[1].cur_row = 375
rra[1].pdp_per_row = 1
rra[1].xff = 5.0000000000e-01
rra[1].cdp_prep[0].value = NaN
rra[1].cdp_prep[0].unknown_datapoints = 0
rra[1].cdp_prep[1].value = NaN
rra[1].cdp_prep[1].unknown_datapoints = 0
rra[2].cf = "AVERAGE"
rra[2].rows = 700
rra[2].cur_row = 507
rra[2].pdp_per_row = 6
rra[2].xff = 5.0000000000e-01
rra[2].cdp_prep[0].value = 0.0000000000e+00
rra[2].cdp_prep[0].unknown_datapoints = 0
rra[2].cdp_prep[1].value = 0.0000000000e+00
rra[2].cdp_prep[1].unknown_datapoints = 0
rra[3].cf = "AVERAGE"
rra[3].rows = 775
rra[3].cur_row = 490
rra[3].pdp_per_row = 24
rra[3].xff = 5.0000000000e-01
rra[3].cdp_prep[0].value = 0.0000000000e+00
rra[3].cdp_prep[0].unknown_datapoints = 12
rra[3].cdp_prep[1].value = 0.0000000000e+00
rra[3].cdp_prep[1].unknown_datapoints = 12
rra[4].cf = "AVERAGE"
rra[4].rows = 797
rra[4].cur_row = 659
rra[4].pdp_per_row = 288
rra[4].xff = 5.0000000000e-01
rra[4].cdp_prep[0].value = 0.0000000000e+00
rra[4].cdp_prep[0].unknown_datapoints = 36
rra[4].cdp_prep[1].value = 0.0000000000e+00
rra[4].cdp_prep[1].unknown_datapoints = 36
rra[5].cf = "MAX"
rra[5].rows = 500
rra[5].cur_row = 443
rra[5].pdp_per_row = 1
rra[5].xff = 5.0000000000e-01
rra[5].cdp_prep[0].value = NaN
rra[5].cdp_prep[0].unknown_datapoints = 0
rra[5].cdp_prep[1].value = NaN
rra[5].cdp_prep[1].unknown_datapoints = 0
rra[6].cf = "MAX"
rra[6].rows = 600
rra[6].cur_row = 85
rra[6].pdp_per_row = 1
rra[6].xff = 5.0000000000e-01
rra[6].cdp_prep[0].value = NaN
rra[6].cdp_prep[0].unknown_datapoints = 0
rra[6].cdp_prep[1].value = NaN
rra[6].cdp_prep[1].unknown_datapoints = 0
rra[7].cf = "MAX"
rra[7].rows = 700
rra[7].cur_row = 128
rra[7].pdp_per_row = 6
rra[7].xff = 5.0000000000e-01
rra[7].cdp_prep[0].value = -inf
rra[7].cdp_prep[0].unknown_datapoints = 0
rra[7].cdp_prep[1].value = -inf
rra[7].cdp_prep[1].unknown_datapoints = 0
rra[8].cf = "MAX"
rra[8].rows = 775
rra[8].cur_row = 515
rra[8].pdp_per_row = 24
rra[8].xff = 5.0000000000e-01
rra[8].cdp_prep[0].value = -inf
rra[8].cdp_prep[0].unknown_datapoints = 12
rra[8].cdp_prep[1].value = -inf
rra[8].cdp_prep[1].unknown_datapoints = 12
rra[9].cf = "MAX"
rra[9].rows = 797
rra[9].cur_row = 355
rra[9].pdp_per_row = 288
rra[9].xff = 5.0000000000e-01
rra[9].cdp_prep[0].value = -inf
rra[9].cdp_prep[0].unknown_datapoints = 36
rra[9].cdp_prep[1].value = -inf
rra[9].cdp_prep[1].unknown_datapoints = 36

This is one graph of several hundred that are having this issue. I hope this data proves more useful...

[BSOD2600]

Did you switch this device from 32 to 64bit counters per chance? It would appear the DS's are too small.

ds[traffic_in].max = 1.0000000000e+07
ds[traffic_in].last_ds = "1081624224"

that max only allows for:
10000000
while the data it last received was:
1081624224

Just a few decimal places too small .

What's reported for the ifSpeed on that specific port?

[DudemanJenkins]

I have not used the 64 bit counters before so I can safely say that's not the case.

Would there be a way to do some sort of mass fix for that? Or am I better off re-creating the graphs? Because new graphs are doing the same thing using the Bits with Total Bandwidth.

[BSOD2600]

rrdtool tune command could be used to fix... but since you dont have any data stored, no harm in deleting and starting fresh.

What does the ifSpeed for those interface(s) show up as? If its too low, this problem will just occur again

Code: Select all

snmpwalk -v 1 -c public localhost 1.3.6.1.2.1.2.2.1.5