1.0.1 to 1.2.3 - good except for a few graphs wont draw

[kjstech]

Since we upgraded from cacti 1.0.1 to 1.2.3 (nice work by the way), we've noticed that a few of our devices just will not graph. The graph goes right up until we took the system down for the upgrade and then there are no plots on the graphs since then. I'm noticing this on our two core switches which are Extreme X690's which use SNMPv3 to connect. In the devices view the SNMPv3 query works and I can do a verbose query on the device and see all the data sources. On the graph trees I can do a realtime graph on any of these ports on these two core switches, wait and watch and I see traffic in real time.

So real time graphs work but plotting graphs over time since the upgrade to 1.2.3 do not work.

Here's what does work, without issue both before and after the upgrade:
Ecessa PL600 load balancers
Ubiquiti rocket m5 airos
Cisco 2911 and 2901 routers
Cisco 3750 g2, 3750x switches
Cisco 3560x switches
Meraki switches (very intermittent and broken up)
Brocade Ti24x 10gig switches
Extreme X450 G2 switches
localhost (ubuntu linux running cacti)
Eaton UPS
APC UPS (Very intermittent, some stopped graphing)
Cisco 5508 Wireless LAN Controller


Whats broken
Extreme X690 core licence switch
Dell PowerEdge FX2 IO io aggregator switch
Some APC UPS units
One Ubiquiti EdgeSwitch 24 lite (another one in our org works).

In the logs tab I've found this every so often which is interesting because it contains our two core switches, but the router it mentions is graphing normally.
2019/04/30 09:26:00 - CMDPHP WARNING: cmd.php poller over ran its polling intervale and therefore ending
2019/04/30 09:25:59 - POLLER: Poller[Main Poller] WARNING: You have 3 Devices with bad SNMP Indexes. Devices: Device[ExeRouter], Device[PurpleCore], Device[YellowCore] totalling 54 Data Sources. Please Either Re-Index, Delete or Disable these Data Sources.


First if I were to heed that warning, how would one Re-Index to try and eliminate that log from popping up, and do you think that is related to the inconsistent graphing? Thanks for the help!

[netniV]

The likelihood is that the graphs aren't being plotted as they aren't being polled due to the overruns. Are you having any devices iwth errors due to timeouts? What settings do you have against your poller for device/threads? Are you using spine or cmd.php? I'm assuming the latter based of the logs I saw above.

[kjstech]

The likelihood is that the graphs aren't being plotted as they aren't being polled due to the overruns. Are you having any devices iwth errors due to timeouts? What settings do you have against your poller for device/threads? Are you using spine or cmd.php? I'm assuming the latter based of the logs I saw above.

Ok I'm not 100% sure what particular settings you need. It worked fine in 1.0.1, and we upgraded to 1.2.3 and love the new look and overall its a great improvement. Its just a few devices stopped plotting since cacti was taken down for that upgrade.

If I go to Configuration > Settings > Poller tab here is what I have:
Data Collection and SNMP Agent Support Enabled
Poller type : cmd.php
Poller sync interval: Every 2 hours
Poller Interval: Every minute
Cron Interval: Every minute
Default Data Collector Processes: 4
Balance Process Load (enabled)
Debug Output Width and Disable increasing OID Check (disabled)
Remote Agent Timeout: 5 Seconds
SNMP Bulkwalk Fetch Size: 10


In Data Collection > Data Collectors

Code: Select all

                                         Proc/Threads | Polling Time | Avg/Max  | Devices | SNMP Gets | Scripts | Servers | Last Finished     | Last Update      | Last Sync
Collector Name: Main Poller (Running)             4/- |      59.91   | 59/63.73 |   44    |   3832    |    40   |    372  |  05-02 09:17:19   |  05-02 09:18:01  | N/A

So do you think its a poller issue, perhaps I need more threads or something because the realtime graph feature works, but that doesn't save any history back to the rrd files. Once I toggle realtime graph off then its lost and the graph is NaN for all values again.

The majority of our gear graphs well, but a few key components do not. I have one Ubuquti 24 port switch that isn't too important so I deleted the device completely (and all data sources) and then added the device back in and it still will not graph, despite SNMP query good and it detecting all of the interfaces for graphing 64-bit counters.

[kjstech]

Let me ask you this, one of the graphs is device 60, so I searched for Device[60] in cacti.log. There's thousands of lines like this:
POLLER: Poller[1] WARNING: You have 3 Devices with bad SNMP Indexes. Devices: Device[23], Device[60], Device[61] totalling 54 Data Sources. Please Either Re-Index, Delete or Disable these Data Sources.

Yes device 60 and 61 are not graphing. Device 23 is graphing fine so I'm not sure why that one is in the log entry.

My question is how do you do what its asking.... Where do I go to Re-Index these Data Sources? Would that be the fix?

[kjstech]

Ok well I changed to spine as the poller and now so far those missing graphs are working.

I think you were on to something and thanks for nudging me in the general direction. I believe you are right that the cmd poller was taking too long to complete.

[kjstech]

What's weird is chaning to spine fixed the missing graphs but it broke our second core switch graphs. On that Extreme X690 switch it stopped graphing right when I switched the poller.

Its device ID 61 so logged into the ubuntu host running cacti and spine I ran this to try to troubleshoot a spine issue. Nothing stands out.
# ./spine -V 3 -S -R --first=61 --last=61
SPINE: Using spine config file [/etc/cacti/spine.conf]
SPINE: Version 1.2.3 starting
NOTE: Spine will support multithread device polling.
DEBUG: Initial Value of Active Threads is 0
SPINE: Active Threads is 1, Pending is 1
DEBUG: In Poller, About to Start Polling of Device for Device ID 61
Device[61] DEBUG: Entering ICMP Ping
Device[61] DEBUG: ICMP Device Alive, Try Count:1, Time:0.2301 ms
Updating Full System Information Table
Device[61] HT[1] NOTE: There are '288' Polling Items for this Device
Device[61] HT[1] Total Time: 0.02 Seconds
Device[61] HT[1] DEBUG: HOST COMPLETE: About to Exit Device Polling Thread Function
DEBUG: The Value of Active Threads is 0 for Device ID 61
POLLER: Active Threads is 0, Pending is 0
SPINE: The Final Value of Threads is 0
Time: 1.0165 s, Threads: 2, Devices: 1

Realtime graphs on this device work fine. Editing the device returns SNMP interface names in a verbose SNMPv3 query, as well as the initial SNMP query at the top of the screen with the hostname and uptime.

The other logs regarding "bad SNMP Indexes" is resolved, I found they were actually bum graphs / data sources and deleted them.

[netniV]

What are you values for the poller when you edit it?

[kjstech]

What are you values for the poller when you edit it?

Ok I just have one called Main Poller.

8 processes and 2 threads.

[kjstech]

So our second Extreme X690 core switch never did update, despite the first one being fine.

So I deleted the graphs, data sources and the device completely. I added it back in, sucessfully queried SNMPv3 without issues. Generated all new graphs. I waited an hour and the graphs for this device still say "Website does not have write access to folder, may be unable to create/update RRDs: /rra/.

I call BS because I've seen this before after creating a device and you just had to wait a few minutes. Plus if I didn't have write access then NONE of my graphs would work. I have no idea why this one single device will not poll on SPINE, and only CMDPHP. The problem if I use CMDPHP, though this device would graph, there are more devices that do not graph at all (4 to 6 other devices). I'd rather have 1 non-working device than a few.

I have no idea how to correct this.

[camerabob]

Are both Spine and cmd.php running under the same user? Also, does the RRA folder where the files reside have write access?

[kjstech]

I'm not even sure where to check to see what user spine runs as, vs cmd.php.

I changed it from spine to cmd.php and at least the graphs on the core switch changed from that unable to access generic graphic, to actual blank graph grids, and finally some data.

Though why should it matter because the other devices graph correctly, so if I deleted this device and added it new, then the same rules should have applied as do other devices.

Is there any way to specify this one switch use cmdphp and the rest of the graphs use spine? I have to toggle back and forth to break / unbreak certain graphs.

[camerabob]

Find the path to Spine (Configurations > Paths > Path to Spine). Then from the CLI, find out who the owner is (I'm assuming Linux)

Code: Select all

ls -l /<path to spine>/spine

See if it is the same user that the poller is pushing

Code: Select all

[root@cacti ~]# cat /etc/cron.d/cacti
*/1 * * * * apache /usr/share/cacti/poller.php > /dev/null 2>&1

In my case, Apache is where cmd.php is running under. Spine would also need to be owned by apache or at least be able to be executed by other groups or users. I don't have spine running so I can't show the output, but it should be similar to:

Code: Select all

[root@cacti cacti]# ls -l cmd.php
-rwxr-xr-x 1 root root 30799 Mar 30 19:40 cmd.php

My permissions here are 755 so all can read and execute the file. Spine should have similar rights.

Hope that helps.

[kjstech]

Ok maybe you are on to something but here goes:

Code: Select all

#ls -l /var/www/cacti/cmd.php
-rw-r--r-- 1 root root 30799 Apr 17 16:04 /var/www/cacti/cmd.php

Nope, definitely not 755.

Code: Select all

#ls -l /usr/local/spine/bin/spine
-rwxr-xr-x 1 root root 389267 May  2 15:25 /usr/local/spine/bin/spine

This is 755.

Code: Select all

#cat /etc/cron.d/cacti
#MAILTO=root
#*/5 * * * * www-data php --define suhosin.memory_limit=512M /usr/share/cacti/site/poller.php 2>&1 >/dev/null | if [ -f /usr/bin/ts ]                                                         ; then ts ; else tee ; fi >> /var/log/cacti/poller-error.log

poller running as www-data

I just changed cmd.php to 755, let me see what that does.

[camerabob]

Any particular reason that your poller.php and cmd.php files are in different folders? Are they for the same version of Cacti with symlinks?

[kjstech]

Any particular reason that your poller and cmd.php files are in different folders? Are they for the same version of Cacti with symlinks?

I never used spine before. After upgrading to 1.2.3 I noticed a good handful of devices stopped graphing. While many still graphed normally, that's when I went to get spine (I think I used ubuntu's apt-get for it), and that's where it installed it.

I changed back to spine to break my one graph but keep all my other devices graphing. Sucks because the one graph that only works with cmdphp is a core switch, and while we have two core switches, its still a pretty important device to graph.

Both core switches are the same manufacturer, same firmware version, same SNMP v3 settings... so I'm confused why one only works with spine and the other only works with cmdphp.