Empty graphs in 1.2.7

[glennwelker]

This was a migration from another server running another 1.xx version.

We are seeing blank graphs. I suspected that maybe it was due to permissions on the rra folder but I recently created a new graph for one of our devices and it created the file. It just isn't detecting any data so it never updates them.

Ideas?

[netniV]

Yes, that sounds like a permission issue. Ensure that both the website and the poller are running under users that can create/update all the RRA files.

[glennwelker]

Thanks, maybe you can provide a little guidance.

I have the rra folder set to ownership by www-data. The poller is also running under www-data. The cacti user is cactiuser. How should I proceed?

[netniV]

What permissions do you have set against the folder/files? Do you have SELinux enabled?

[glennwelker]

rw rw r for www-data:www-data

[glennwelker]

Here is a snippet from my cacti log. I don't see anything for rrd files. Maybe they don't show up in this file?
2019/12/11 14:32:59 - SPINE: Poller[1] PID[21703] DEBUG: Initial Value of Active Threads is 0
2019/12/11 14:32:59 - SPINE: Poller[1] PID[21703] DEBUG: Valid Thread to be Created
2019/12/11 14:32:59 - SPINE: Poller[1] PID[21703] SPINE: Active Threads is 1, Pending is 1
2019/12/11 14:32:59 - SPINE: Poller[1] PID[21703] DEBUG: In Poller, About to Start Polling of Device for Device ID 0
2019/12/11 14:32:59 - SPINE: Poller[1] PID[21700] NOTE: Spine will support multithread device polling.
2019/12/11 14:32:59 - SPINE: Poller[1] PID[21700] DEBUG: Initial Value of Active Threads is 0
2019/12/11 14:32:59 - SPINE: Poller[1] PID[21700] DEBUG: Valid Thread to be Created
2019/12/11 14:32:59 - SPINE: Poller[1] PID[21700] SPINE: Active Threads is 1, Pending is 1
2019/12/11 14:32:59 - SPINE: Poller[1] PID[21700] DEBUG: In Poller, About to Start Polling of Device for Device ID 0
2019/12/11 14:32:59 - SPINE: Poller[1] PID[21703] Device[0] HT[1] Total Time: 0.0023 Seconds
2019/12/11 14:32:59 - SPINE: Poller[1] PID[21700] Device[0] HT[1] Total Time: 0.0013 Seconds
2019/12/11 14:32:59 - SPINE: Poller[1] PID[21703] Device[0] HT[1] DEBUG: HOST COMPLETE: About to Exit Device Polling Thread Function
2019/12/11 14:32:59 - SPINE: Poller[1] PID[21703] DEBUG: The Value of Active Threads is 0 for Device ID 0
2019/12/11 14:32:59 - SPINE: Poller[1] PID[21700] Device[0] HT[1] DEBUG: HOST COMPLETE: About to Exit Device Polling Thread Function
2019/12/11 14:32:59 - SPINE: Poller[1] PID[21703] POLLER: Active Threads is 0, Pending is 0
2019/12/11 14:32:59 - SPINE: Poller[1] PID[21703] DEBUG: Valid Thread to be Created
2019/12/11 14:32:59 - SPINE: Poller[1] PID[21700] DEBUG: The Value of Active Threads is 0 for Device ID 0
2019/12/11 14:32:59 - SPINE: Poller[1] PID[21703] SPINE: Active Threads is 1, Pending is 1
2019/12/11 14:32:59 - SPINE: Poller[1] PID[21700] POLLER: Active Threads is 0, Pending is 0
2019/12/11 14:32:59 - SPINE: Poller[1] PID[21703] DEBUG: In Poller, About to Start Polling of Device for Device ID 110
2019/12/11 14:32:59 - SPINE: Poller[1] PID[21700] DEBUG: Valid Thread to be Created
2019/12/11 14:32:59 - SPINE: Poller[1] PID[21700] SPINE: Active Threads is 1, Pending is 1
2019/12/11 14:32:59 - SPINE: Poller[1] PID[21700] DEBUG: In Poller, About to Start Polling of Device for Device ID 42
2019/12/11 14:32:59 - SPINE: Poller[1] PID[21703] Device[110] INFO: SNMP Device '10.24.2.249' has timeout 500000 (500), retries 3
2019/12/11 14:32:59 - SPINE: Poller[1] PID[21700] Device[42] INFO: SNMP Device '10.23.2.100' has timeout 500000 (500), retries 3
2019/12/11 14:32:59 - SPINE: Poller[1] PID[21703] Device[110] DEBUG Falling back to UDP Ping Due to SetUID Issues
2019/12/11 14:32:59 - SPINE: Poller[1] PID[21700] Device[42] DEBUG: Entering UDP Ping
2019/12/11 14:32:59 - SPINE: Poller[1] PID[21703] Device[110] DEBUG: Entering UDP Ping
2019/12/11 14:32:59 - SPINE: Poller[1] PID[21703] Device[110] DEBUG: UDP Device Alive, Try Count:1, Time:0.2711 ms
2019/12/11 14:32:59 - SPINE: Poller[1] PID[21703] Device[110] PING: Result UDP: Device is Alive
2019/12/11 14:32:59 - SPINE: Poller[1] PID[21703] Device[110] HT[1] RECACHE: Processing 3 items in the auto reindex cache for '10.24.2.249'
2019/12/11 14:32:59 - SPINE: Poller[1] PID[21700] Device[42] DEBUG: UDP Timeout, Try Count:1, Time:400.5680 ms
2019/12/11 14:32:59 - SPINE: Poller[1] PID[21700] Device[42] DEBUG: UDP Timeout, Try Count:2, Time:401.7501 ms
2019/12/11 14:32:59 - SPINE: Poller[1] PID[21700] Device[42] DEBUG: Entering SNMP Ping
2019/12/11 14:33:01 - SPINE: Poller[1] PID[21703] ERROR: Failed to get oid '.1.3.6.1.2.1.1.3.0' for Device[110]
2019/12/11 14:33:01 - SPINE: Poller[1] PID[21703] Device[110] HT[1] DQ[1] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: U < output: U)
2019/12/11 14:33:01 - SPINE: Poller[1] PID[21703] Device[110] HT[1] DQ[8] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: U < output: U)
2019/12/11 14:33:01 - SPINE: Poller[1] PID[21703] Device[110] HT[1] DQ[9] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: U < output: U)
2019/12/11 14:33:01 - SPINE: Poller[1] PID[21703] Device[110] HT[1] Total Time: 2 Seconds
2019/12/11 14:33:01 - SPINE: Poller[1] PID[21703] Device[110] HT[1] DEBUG: HOST COMPLETE: About to Exit Device Polling Thread Function
2019/12/11 14:33:01 - SPINE: Poller[1] PID[21703] DEBUG: The Value of Active Threads is 0 for Device ID 110
2019/12/11 14:33:01 - SPINE: Poller[1] PID[21703] POLLER: Active Threads is 0, Pending is 0
2019/12/11 14:33:01 - SPINE: Poller[1] PID[21703] SPINE: The Final Value of Threads is 0
2019/12/11 14:33:01 - SPINE: Poller[1] PID[21703] DEBUG: Thread Cleanup Complete
2019/12/11 14:33:01 - SPINE: Poller[1] PID[21703] DEBUG: PHP Script Server Pipes Closed
2019/12/11 14:33:01 - SPINE: Poller[1] PID[21703] DEBUG: Allocated Variable Memory Freed
2019/12/11 14:33:01 - SPINE: Poller[1] PID[21703] DEBUG: MYSQL Free & Close Completed
2019/12/11 14:33:01 - SPINE: Poller[1] PID[21703] DEBUG: Net-SNMP Close Completed
2019/12/11 14:33:01 - SPINE: Poller[1] PID[21703] Time: 2.1159 s, Threads: 1, Devices: 2
2019/12/11 14:33:01 - SPINE: Poller[1] PID[21700] ERROR: Failed to get oid '.1.3.6.1.2.1.1.3.0' for Device[42]

[glennwelker]

Hmm, just noticed that the one device that I added a graph for yesterday has updated its graphs. None of the other graphs are updating. I've noticed bad index errors. Seems like the migration to the new version is causing issues.

[netniV]

So it looks like you have a couple of issues:

• You need to set the SetUID attribute on the spine binary and ensure that the owner is root. You are trying to use ICMP Ping and without those steps being done, it will just fall back to UDP ping which may not be valid for the device/network/firewall configurations you have in place
• A device you are trying to query, is either blocking you, or not giving you any information. Make sure it does have SNMP enabled, allowed through it's firewall and that the Cacti server is allowed to view everything over SNMP

[glennwelker]

So frustrated. No progress. Still empty graphs. I have noticed issues but it still hasn't resolved my issue.

The last item I noticed is that the system had the incorrect path to the old rrd files. Since this was a migration, the full path is in the poller item table from the old system. But the new system didn't have the same path for the cacti install. I manually updated all paths to the correct path but it didn't fix our graph issue.

[glennwelker]

Some of the issues appear to be caused by Spine. I changed the poller to cmd.php and I started getting graphs. Not all graphs are working after the migration and I created another ticket for RRD cleaning which doesn't work.

[glennwelker]

Really trying to find a solution. Anyone?

Rebuilt Spine to make sure that there wasn't an issue with the newest version.

Runnning Cacti 1.2.7
Spine 1.2.7 from the cacti web site.

Firewall is not running.
No errors that I can see.

Spine seems to run but does not update graphs. It runs manually with no errors. If I switch over to cmd.php graphs are updated correctly.

I would really like to use Spine since we are monitoring servers which don't need a graph and don't get polled by the default poller.

[netniV]

Have you addressed the two points I raised above?

[glennwelker]

Yes, I thought that I said that in my previous replies. Maybe I am misunderstanding.

The first issue of SetOID was done.

The second issue I don't feel can apply to all of our graphs.
1. This is a migrated server that has been in place for years. All servers and network switches have SNMP properties set
2. The default poller works fine for the graphs. It is only when we use Spine that the graphs stop updating.

[glennwelker]

More logs. One device continues to graph with Spine. All other devices do not updates graphs unless we use cmd.php.

Seeing continuous warnings in the logs. Here is an example. Device 11 is one of the devices that it is not updating.
Line 13628: 2019/12/12 10:36:40 - SPINE: Poller[1] PID[8381] Device[11] WARNING: snmp_pdu_create(.1.3.6.1.2.1.1.1.0)
Line 13629: 2019/12/12 10:36:40 - SPINE: Poller[1] PID[8381] Device[11] WARNING: snmp_pdu_create(.1.3.6.1.2.1.1.1.0) [complete]
Line 13630: 2019/12/12 10:36:40 - SPINE: Poller[1] PID[8381] Device[11] WARNING: snmp_parse_oid(.1.3.6.1.2.1.1.1.0)
Line 13631: 2019/12/12 10:36:40 - SPINE: Poller[1] PID[8381] Device[11] WARNING: snmp_parse_oid(.1.3.6.1.2.1.1.1.0) [complete]
Line 13632: 2019/12/12 10:36:40 - SPINE: Poller[1] PID[8381] Device[11] WARNING: snmp_add_null_var(.1.3.6.1.2.1.1.1.0)
Line 13633: 2019/12/12 10:36:40 - SPINE: Poller[1] PID[8381] Device[11] WARNING: snmp_add_null_var(.1.3.6.1.2.1.1.1.0) [complete]
Line 13634: 2019/12/12 10:36:40 - SPINE: Poller[1] PID[8381] Device[11] WARNING: snmp_sess_sync_response(.1.3.6.1.2.1.1.1.0)
Line 13635: 2019/12/12 10:36:40 - SPINE: Poller[1] PID[8381] Device[11] WARNING: snmp_sess_sync_response(.1.3.6.1.2.1.1.1.0) [complete]
Line 13636: 2019/12/12 10:36:40 - SPINE: Poller[1] PID[8381] DEVDBG: Device[11] poll_result = snmp_get(host, '.1.3.6.1.2.1.1.1.0'); [complete]
Line 13637: 2019/12/12 10:36:40 - SPINE: Poller[1] PID[8381] DEVDBG: Device[11] poll_result = snmp_get(host, '.1.3.6.1.2.1.1.2.0');

[glennwelker]

OK, can someone help me go in a different direction? I am willing to delete all devices, data sources, and graphs but so far that hasn't made a difference.

I am fairly certain that the issue is in the database on not on the server.
Can someone point me to the right table?

I have two devices that create and update RRD files with Spine. The rest of the devices do not create or update the RRD files.
I have compared the data sources, devices and graphs so many times that I'm blue in the face. The working items are identical to the devices that do not update as far as settings go.

Hopeful that someone can help.