Cacti 1.2.24 Two new switches not graphing any data, but SNMP is returning interface names and switch information

[kjstech]

We are running Cacti 1.2.24. We had 40 devices successfully polling and graphing. I added two more switches last week and the graphs are just blank. The switchport names come through and here is what I'm seeing in the logs:

2023/02/27 16:18:31 - SPINE: Poller[Main Poller] PID[43295] PT[139723855705856] Device[Tek-Cisco-Sw] HT[1] Total Time: 0.25 Seconds
2023/02/27 16:18:31 - SPINE: Poller[Main Poller] PID[43295] PT[139723855705856] Device[Tek-Cisco-Sw] HT[1] DS[Tek-Cisco-Sw - Traffic - Gi1/0/18] Graphs[Tek-Cisco-Sw - Gi1/0/18 - APC RIGHT PDU] TT[38.05] SNMP: v3: 10.100.0.254, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.10118, value: 58960779
2023/02/27 16:18:31 - SPINE: Poller[Main Poller] PID[43295] PT[139723855705856] Device[Tek-Cisco-Sw] HT[1] DS[Tek-Cisco-Sw - Traffic - Gi1/0/18] Graphs[Tek-Cisco-Sw - Gi1/0/18 - APC RIGHT PDU] TT[38.04] SNMP: v3: 10.100.0.254, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.10118, value: 1301789
2023/02/27 16:18:31 - SPINE: Poller[Main Poller] PID[43295] PT[139723855705856] Device[Tek-Cisco-Sw] HT[1] DS[Tek-Cisco-Sw - Traffic - Gi1/0/8] Graphs[Tek-Cisco-Sw - Gi1/0/8] TT[38.02] SNMP: v3: 10.100.0.254, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.10108, value: 0
2023/02/27 16:18:31 - SPINE: Poller[Main Poller] PID[43295] PT[139723855705856] Device[Tek-Cisco-Sw] HT[1] DS[Tek-Cisco-Sw - Traffic - Gi1/0/8] Graphs[Tek-Cisco-Sw - Gi1/0/8] TT[37.99] SNMP: v3: 10.100.0.254, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.10108, value: 0
2023/02/27 16:18:31 - SPINE: Poller[Main Poller] PID[43295] PT[139723855705856] Device[Tek-Cisco-Sw] HT[1] NOTE: There are '4' Polling Items for this Device
2023/02/27 16:18:31 - SPINE: Poller[Main Poller] PID[43295] PT[139723855705856] Device[Tek-Cisco-Sw] HT[1] DQ[SNMP - Interface Statistics] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 122019100 < output: 122022000)
2023/02/27 16:18:31 - SPINE: Poller[Main Poller] PID[43295] PT[139723855705856] DEBUG: Device[Tek-Cisco-Sw] HT[1] RECACHE: Processing 1 items in the auto reindex cache for '10.100.0.254'
2023/02/27 16:18:31 - SPINE: Poller[Main Poller] PID[43295] PT[139723855705856] Device[Tek-Cisco-Sw] Updating Full System Information Table
2023/02/27 16:18:31 - SPINE: Poller[Main Poller] PID[43295] PT[139723855705856] Device[Tek-Cisco-Sw] DEBUG: ICMP Device Alive, Try Count:1, Time:11.1780 ms
2023/02/27 16:18:31 - SPINE: Poller[Main Poller] PID[43295] PT[139723855705856] Device[Tek-Cisco-Sw] DEBUG: Received EINTR
2023/02/27 16:18:31 - SPINE: Poller[Main Poller] PID[43295] PT[139723855705856] Device[Tek-Cisco-Sw] DEBUG: Received EINTR
2023/02/27 16:18:31 - SPINE: Poller[Main Poller] PID[43295] PT[139723855705856] Device[Tek-Cisco-Sw] DEBUG: Attempting to ping 10.100.0.254, seq 0 (Retry 0 of 4)
2023/02/27 16:18:31 - SPINE: Poller[Main Poller] PID[43295] PT[139723855705856] Device[Tek-Cisco-Sw] DEBUG: Entering ICMP Ping
2023/02/27 16:18:31 - SPINE: Poller[Main Poller] PID[43295] PT[139723855705856] DEBUG: Device[Tek-Cisco-Sw] HT[1] In Poller, About to Start Polling
2023/02/27 16:18:09 - SYSTEM STATS: Time:8.8449 Method:spine Processes:8 Threads:2 Hosts:42 HostsPerProcess:6 DataSources:3780 RRDsProcessed:0
2023/02/27 16:18:02 - SPINE: Poller[Main Poller] PID[43202] PT[140548131743488] DEBUG: Device[Tek-Cisco-Sw] HT[1] DEBUG: HOST COMPLETE: About to Exit Device Polling Thread Function
2023/02/27 16:18:02 - SPINE: Poller[Main Poller] PID[43202] PT[140548131743488] Device[Tek-Cisco-Sw] HT[1] Total Time: 0.26 Seconds
2023/02/27 16:18:02 - SPINE: Poller[Main Poller] PID[43202] PT[140548131743488] Device[Tek-Cisco-Sw] HT[1] DS[Tek-Cisco-Sw - Traffic - Gi1/0/17] Graphs[Tek-Cisco-Sw - Gi1/0/17 - APC LEFT PDU] TT[33.23] SNMP: v3: 10.100.0.254, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.10117, value: 59120733
2023/02/27 16:18:02 - SPINE: Poller[Main Poller] PID[43202] PT[140548131743488] Device[Tek-Cisco-Sw] HT[1] DS[Tek-Cisco-Sw - Traffic - Gi1/0/17] Graphs[Tek-Cisco-Sw - Gi1/0/17 - APC LEFT PDU] TT[33.22] SNMP: v3: 10.100.0.254, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.10117, value: 893116
2023/02/27 16:18:02 - SPINE: Poller[Main Poller] PID[43202] PT[140548131743488] Device[Tek-Cisco-Sw] HT[1] DS[Tek-Cisco-Sw - Traffic - Gi1/0/7] Graphs[Tek-Cisco-Sw - Gi1/0/7] TT[33.20] SNMP: v3: 10.100.0.254, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.10107, value: 255435608
2023/02/27 16:18:02 - SPINE: Poller[Main Poller] PID[43202] PT[140548131743488] Device[Tek-Cisco-Sw] HT[1] DS[Tek-Cisco-Sw - Traffic - Gi1/0/7] Graphs[Tek-Cisco-Sw - Gi1/0/7] TT[33.17] SNMP: v3: 10.100.0.254, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.10107, value: 42705100
2023/02/27 16:18:01 - SPINE: Poller[Main Poller] PID[43202] PT[140548131743488] Device[Tek-Cisco-Sw] HT[1] NOTE: There are '4' Polling Items for this Device
2023/02/27 16:18:01 - SPINE: Poller[Main Poller] PID[43202] PT[140548131743488] Device[Tek-Cisco-Sw] HT[1] DQ[SNMP - Interface Statistics] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 122016000 < output: 122019100)


Any idea why 40 devices poll and graph fine and 2 other switches that are added the same way as other devices poll but do not graph any data?

[TheWitness]

The official release of 1.2.24 was today. You might want to download anything new since you pulled the 1.2.x branch and update your existing installs of spine and Cacti.

[kjstech]

Ok I just unpacked "the official" 1.2.24 release from github and moved my data and config.php over, ran through the initial setup and I'm back up and running.

My existing graphs are back up and running but I'm still seeing the two new switches not graph. I have blank graphs. Maybe I'll delete them and re-add them.

[kjstech]

Ok deleted and readded those last two switches but still not getting graphs.

Since deleting those two and re-adding now I lost tree view. Its just a white screen. I can do list view or preview view and find the switches that way.

Really weird. Tried Firefox and Chrome.

[kjstech]

Ok deleted and readded those last two switches but still not getting graphs.

Since deleting those two and re-adding now I lost tree view. Its just a white screen. I can do list view or preview view and find the switches that way.

Really weird. Tried Firefox and Chrome.

Edit: Tried publishing the tree again as soon as I turn it on and hit save it says
This tree has been locked for Editing on %1$s by %2$s.
And publish switches off.

Edit. Moved tree.php from my prior install to this 1.2.24 and its fixed.

[kjstech]

Ok I can say that the one switch, a Cisco if I go into realtime view and wait awhile, I start to see some traffic for a little then it starts going away and comes back. No graphs in regular view though, just -nan.

Image
https://imgur.com/a/a4rnS4x

[TheWitness]

Sounds like maybe a file permission issue. Run spine this way as a test;

Code: Select all

./spine -V 3 -S -R -f host_id -l host_id

The host_id is the numeric id of the host.

[kjstech]

Here's what I get for the first one:
spine -V 3 -S -R -f 99 -l 99
SPINE: Using spine config file [/etc/cacti/spine.conf]
SPINE: Version 1.2.6 starting
NOTE: Spine will support multithread device polling.
DEBUG: Initial Value of Active Threads is 0
SPINE: Active Threads is 1, Pending is 1
SPINE: Active Threads is 2, Pending is 2
Device[0] HT[1] Total Time: 0.00092 Seconds
POLLER: Active Threads is 1, Pending is 1
Device[99] INFO: SNMP Device '10.100.0.254' has timeout 500000 (500), retries 3
NOTE: Device[99] Checking for System Information Update
NOTE: Device[99] Updating Full System Information Table
Device[99] HT[1] DQ[1] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 128070000 < out put: 128082943)
Device[99] HT[1] NOTE: There are '6' Polling Items for this Device
Device[99] INFO: SNMP Device '10.100.0.254' has timeout 500000 (500), retries 3
Device[99] HT[1] DS[6185] SNMP: v3: 10.100.0.254, dsname: traffic_in, oid: .1.3. 6.1.2.1.31.1.1.1.6.10105, value: 3518860564
Device[99] HT[1] DS[6185] SNMP: v3: 10.100.0.254, dsname: traffic_out, oid: .1.3 .6.1.2.1.31.1.1.1.10.10105, value: 4601743271
Device[99] HT[1] DS[6195] SNMP: v3: 10.100.0.254, dsname: traffic_in, oid: .1.3. 6.1.2.1.31.1.1.1.6.10115, value: 0
Device[99] HT[1] DS[6195] SNMP: v3: 10.100.0.254, dsname: traffic_out, oid: .1.3 .6.1.2.1.31.1.1.1.10.10115, value: 0
Device[99] HT[1] DS[6205] SNMP: v3: 10.100.0.254, dsname: traffic_in, oid: .1.3. 6.1.2.1.31.1.1.1.6.10201, value: 0
Device[99] HT[1] DS[6205] SNMP: v3: 10.100.0.254, dsname: traffic_out, oid: .1.3 .6.1.2.1.31.1.1.1.10.10201, value: 0
Device[99] HT[1] Total Time: 0.32 Seconds
POLLER: Active Threads is 0, Pending is 0
SPINE: The Final Value of Threads is 0
Time: 0.4431 s, Threads: 2, Devices: 2

Here's what I get for the second one:
root@cacti:~# spine -V 3 -S -R -f 98 -l 98
SPINE: Using spine config file [/etc/cacti/spine.conf]
SPINE: Version 1.2.6 starting
NOTE: Spine will support multithread device polling.
DEBUG: Initial Value of Active Threads is 0
SPINE: Active Threads is 1, Pending is 1
Device[0] HT[1] Total Time: 0.00088 Seconds
POLLER: Active Threads is 1, Pending is 0
SPINE: Active Threads is 1, Pending is 1
Device[98] INFO: SNMP Device '10.100.0.101' has timeout 500000 (500), retries 3
NOTE: Device[98] Checking for System Information Update
NOTE: Device[98] Updating Full System Information Table
Device[98] HT[1] DQ[1] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 155047000 < output: 155051600)
Device[98] HT[1] NOTE: There are '6' Polling Items for this Device
Device[98] INFO: SNMP Device '10.100.0.101' has timeout 500000 (500), retries 3
Device[98] HT[1] DS[6157] SNMP: v1: 10.100.0.101, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.8, value: U
Device[98] HT[1] DS[6157] SNMP: v1: 10.100.0.101, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.8, value: U
Device[98] HT[1] DS[6167] SNMP: v1: 10.100.0.101, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.18, value: U
Device[98] HT[1] DS[6167] SNMP: v1: 10.100.0.101, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.18, value: U
Device[98] HT[1] DS[6177] SNMP: v1: 10.100.0.101, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.28, value: U
Device[98] HT[1] DS[6177] SNMP: v1: 10.100.0.101, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.28, value: U
Device[98] HT[1] Total Time: 0.15 Seconds
POLLER: Active Threads is 0, Pending is 0
SPINE: The Final Value of Threads is 0
Time: 0.2211 s, Threads: 2, Devices: 2

Still nothing graphed.

As far as permissions, all the rrd files, including the ones for these two are owned by www-data, 755.

[TheWitness]

The fact that you are getting U results may mean that you don't have full access to the device. If it's a Linux device, you may have to make adjustments in the snmpd.conf file to gain full access to the device. My snmpd.conf file is usually one line long...

rocommunity public

That's a beginner configuration, and of course you have to restart snmpd afterwards, but to be honest, in a corporate setting we don't use public, and we always use network ACL's.

[kjstech]

The bottom one was a brocade switch so I think I fixed access on that. The top one was cisco and that's returning values but just not graphing. Here's the spine for the borcade now that snmp is resolved.

spine -V 3 -S -R -f 98 -l 98
SPINE: Using spine config file [/etc/cacti/spine.conf]
SPINE: Version 1.2.6 starting
NOTE: Spine will support multithread device polling.
DEBUG: Initial Value of Active Threads is 0
SPINE: Active Threads is 1, Pending is 1
Device[0] HT[1] Total Time: 0.00076 Seconds
POLLER: Active Threads is 0, Pending is 0
SPINE: Active Threads is 1, Pending is 1
Device[98] INFO: SNMP Device '10.100.0.101' has timeout 500000 (500), retries 3
NOTE: Device[98] Checking for System Information Update
NOTE: Device[98] Updating Full System Information Table
Device[98] HT[1] DQ[1] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 155305000 < output: 155308200)
Device[98] HT[1] NOTE: There are '6' Polling Items for this Device
Device[98] INFO: SNMP Device '10.100.0.101' has timeout 500000 (500), retries 3
Device[98] HT[1] DS[6154] SNMP: v3: 10.100.0.101, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.5, value: 244320126438
Device[98] HT[1] DS[6154] SNMP: v3: 10.100.0.101, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.5, value: 1746989828897
Device[98] HT[1] DS[6164] SNMP: v3: 10.100.0.101, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.15, value: 0
Device[98] HT[1] DS[6164] SNMP: v3: 10.100.0.101, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.15, value: 0
Device[98] HT[1] DS[6174] SNMP: v3: 10.100.0.101, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.25, value: 0
Device[98] HT[1] DS[6174] SNMP: v3: 10.100.0.101, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.25, value: 0
Device[98] HT[1] Total Time: 0.17 Seconds
POLLER: Active Threads is 0, Pending is 0
SPINE: The Final Value of Threads is 0
Time: 0.2428 s, Threads: 2, Devices: 2


Graph details for one of the uplinks on this Brocade switch:
RRDtool Command:
/usr/bin/rrdtool graph - \
--imgformat=SVG \
--start='1677581520' \
--end='1677594486' \
--pango-markup \
--title='Tek-10g-Brocade - ethernet24 - Trunk to Cisco' \
--vertical-label='bits per second' \
--slope-mode \
--base=1000 \
--height=200 \
--width=700 \
--rigid \
--alt-autoscale-max \
--lower-limit='0' \
COMMENT:"From 2023/02/28 05\:52\:00 To 2023/02/28 09\:28\:06\c" \
COMMENT:" \n" \
--color BACK#F3F3F3 \
--color CANVAS#FDFDFD \
--color SHADEA#CBCBCB \
--color SHADEB#999999 \
--color GRID#C4C4C4 \
--color MGRID#1A1C1C \
--color FONT#000000 \
--color AXIS#2C4D43 \
--color ARROW#2C4D43 \
--color FRAME#2C4D43 \
--border 1 \
--font TITLE:11:'Arial' \
--font AXIS:8:'Arial' \
--font LEGEND:8:'Courier' \
--font UNIT:8:'Arial' \
--font WATERMARK:6:'Arial' \
--slope-mode \
--watermark 'Generated by Cacti®' \
DEF:a='/var/www/html/cacti/rra/tek-10g-brocade_traffic_in_6173.rrd':'traffic_in':MAX \
DEF:b='/var/www/html/cacti/rra/tek-10g-brocade_traffic_in_6173.rrd':'traffic_in':AVERAGE \
DEF:c='/var/www/html/cacti/rra/tek-10g-brocade_traffic_in_6173.rrd':'traffic_out':AVERAGE \
DEF:d='/var/www/html/cacti/rra/tek-10g-brocade_traffic_in_6173.rrd':'traffic_out':MAX \
CDEF:cdefa='a,8,*' \
CDEF:cdefb='b,8,*' \
CDEF:cdefg='c,8,*' \
CDEF:cdefh='d,8,*' \
CDEF:cdefbd='c,8,*,-1,*' \
LINE1:cdefa#00CF00FF: \
AREA:cdefb#00CF007F:'Inbound ' \
AREA:cdefb#00CF0033:'Inbound ' \
GPRINT:cdefb:LAST:'Current\:%8.2lf %s' \
GPRINT:cdefb:AVERAGE:'Average\:%8.2lf %s' \
GPRINT:cdefb:MAX:'Maximum\:%8.2lf %s\n' \
AREA:cdefg#002A9719: \
LINE1:cdefh#002A97FF: \
AREA:cdefg#002A977F:'Outbound' \
GPRINT:cdefg:LAST:'Current\:%8.2lf %s' \
GPRINT:cdefg:AVERAGE:'Average\:%8.2lf %s' \
LINE1:cdefb#000000FF: \
GPRINT:cdefg:MAX:'Maximum\:%8.2lf %s\n' \
LINE1:cdefbd#000000FF:
RRDtool Command lengths = 1713 characters.
RRDtool Says:
SVG/XML Output OK

[kjstech]

Actually the RRA files that were recreated when I deleted these two switches and recreated them were given www-data / www-data 644 permissions.

I just changed them to 755 lets see if they start growing more than the 1,596 KB they default as.

[kjstech]

I thought maybe this had to do with these two new switches I tried to add, which to this day still don't graph.

But then I added two new MLAGs on existing working Arista switches. Everything on those Arista switches graph except the two new MLAG interface graphs. Those graphs are larger and have nothing in them. They look out of place with the smaller, yet data filled graphs for other interfaces.

Do you think I hit the maximum number of graphs the system can handle? Everything seems to be working really well except for any new graphs. I have 1977 graphs.

[macan]

Try -> Console -> Troubleshooting -> datasources

[kjstech]

The only thing unique is the two new portchannel graphs I tried to create off existing working switches made the rrd files 644 www-data as the owner and group. Other ones are 755, www-data as the owner and group. Not sure why they are not ever created the same way.

For the two NEW switches that never worked, I thought maybe it was an rrd file permissions issue, but I fixed those all up too, to match other working graphs.

The other thing I noticed is the new graphs have a poller interval of 5 minutes whereas every other existing working graph has a poller interval of 30 seconds. I'm not sure why this is different or whats changed, but next I have to try to figure out where to change it.

IN the Troubleshooting > Data sources I get all green check marks

RRDfile Owner www-data:www-data -
Website runs as www-data -
Poller runs as www-data -
Is RRA Folder writeable by poller? /var/www/html/cacti/rra (Green Check)
Is RRDfile writeable by poller? /var/www/html/cacti/rra/aristasan-1_traffic_in_6207.rrd (Green Check)
Does the RRDfile Exist? Yes (Green Check)
Is the Data Source set as Active? Yes (Green Check)
Did the poller receive valid data? traffic_out = 11866909676634
traffic_in = 1499054976
Was the RRDfile updated? Yes
First Check TimeStamp 1679665710 -
Second Check TimeStamp 1679666010 -
Were we able to convert the title? AristaSAN-1 - Traffic - Port-Channel24 (Green Check)
Data Source matches the RRDfile? (Green Check)
Issues (Green Check)

Data Source Repair Recommendations
Waiting on Data Source Check to Complete

[kjstech]

I think I'm onto something,

Presets - DataProfiles - the second Upgrade Profile 1 (there are 3 under Near Realtime (every 30 seconds)).
Click on the number in the templates in the column. Find Interface - Traffic, check the box next to it, and at the bottom choose change profile and move it to Near Realtime (every 30 seconds).

Now go to the device's graphs and delete all of them (they dont work anyway so nothing lost here).
Now recreate graphs for the specific device.
Go to each graph (I find right clicking and opening in new tab works best) so that at the bottom in the graph debug mode text you get that "traffic_in_6265.rrd': No such file or directory" message.

Now on the server after you "touched" each graph once, it creates the files but as 644 permissions. if you ls -lah you'll see them and the new rrd files are 8.6M whereas the old ones are only 1.6M.
now chmod 755 devicename*

My graphs are now working. THis has been bothering me for a month.

Interestingly all new graphs, even the two new interfaces on my existing working switches have an up to date look to them. FIrst they are a little larger, which on the existing switches its weird becuase they are alongside other, smaller graphs. Second they are differnet colors. A blue and teal vs the old school green and semi-transparent blue.

No idea why this happened!