New RRDs (and graphs) not created

[stbecker]

I have a new graph that is not populating data in (nor creating a) RRD file for a script running on localhost. I've reviewed similar problems that were a result of file/directory permissions or selinux - those do not seem to apply. Please read below to help determine whether I missed a logical step, and I thank you in advance for your patience. I apologize if I've strayed from any standard rules for posting and will modify if necessary.

Steps taken thus far

• Create a script for troubleshooting that returns a single integer value. A standard linux command is used for troubleshooting.

  Code: Select all

  #!/usr/bin/bash
  /bin/date "+%H" | /bin/tr -d '\n'

  I've experimented on returning the newline, or even stripping that out - it makes no difference currently. Everything below will reference activePhoneContacts as a name, but clearly I'm just using an hour for test data right now.

• Create a data input method based on this script.
  

• Create a template for the data source including the output (input not necessary).
  

• Create a template for the graph including the associated item.
  

• Add a graph template to the localhost device. This doesn't create the graph, so I also use the link "create graphs for this device" where it shows that associated template.

Where I'm at now

The RRD should be created on the next polling cycle. The poller is working because other graphs are updating. Other graphs were created long ago. The RRD is not even being created.

Knowing that the data source includes a command to manually create RRDs, I tried that also and set the file ownership to match other pre-existing RRDs that work. I also setup the selinux file permissions to match the others. I tried disabling selinux temporarily, but that did not help either.

There are a couple dozen functional RRDs, but to be succinct, I list just one other by way of comparison.


I don't see references to my script in cacti.log in DEVEL or DEBUG mode, so I'm questioning whether the poller is using my script or not - the other graphs are all SNMP.

Troubleshooting data sources from console yields the following:


Hopefully I've provide enough information to identify a root cause here. Thank you for you patience in dissecting this lengthy description.

[TheWitness]

Run spine in Debug mode for that device, post the output:

Code: Select all

./spine -R -V 5 -S -H host_id

[stbecker]

[root@omawikiinf01 bin]# ./spine -R -V 5 -S -H host_id
SPINE: Using spine config file [../etc/spine.conf]
2023/11/06 09:09:23 - SPINE: Poller[1] DEBUG: The path_php_server variable is /usr/share/cacti/script_server.php
2023/11/06 09:09:23 - SPINE: Poller[1] DEBUG: The path_cactilog variable is /usr/share/cacti/log/cacti.log
DEBUG: The log_destination variable is 4 (STDOUT)
DEBUG: The path_php variable is /bin/php
DEBUG: The availability_method variable is 2
DEBUG: The ping_recovery_count variable is 3
DEBUG: The ping_failure_count variable is 2
DEBUG: The ping_method variable is 2
DEBUG: The ping_retries variable is 1
DEBUG: The ping_timeout variable is 400
DEBUG: The snmp_retries variable is 3
DEBUG: The log_perror variable is 1
DEBUG: The log_pwarn variable is 0
DEBUG: The boost_redirect variable is 0
DEBUG: The boost_rrd_update_enable variable is 0
DEBUG: The log_pstats variable is 0
DEBUG: The threads variable is 1
DEBUG: The polling interval is 300 seconds
DEBUG: The number of concurrent processes is 1
DEBUG: The script timeout is 25
DEBUG: The selective_device_debug variable is 11,1
DEBUG: The spine_log_level variable is 0
DEBUG: The number of php script servers to run is 1
DEBUG: Device List to be polled='host_id', TotalPHPScripts='1
DEBUG: The PHP Script Server is Required
DEBUG: The Maximum SNMP OID Get Size is 10
Selective Debug Devices 11,1
Version 1.1.38 starting
DEBUG: MySQL is Thread Safe!
DEBUG: Spine is running asroot.
SPINE: Initializing Net-SNMP API
DEBUG: Issues with SNMP Header Version information, assuming old version of Net-SNMP.
SPINE: Initializing PHP Script Server(s)
DEBUG: SS[0] PHP Script Server Routine Starting
DEBUG: SS[0] PHP Script Server About to FORK Child Process
DEBUG: SS[0] PHP Script Server Child FORK Success
DEBUG: SS[0] Confirmed PHP Script Server running using readfd[7], writefd[6]
NOTE: Spine will support multithread device polling.
FATAL: MySQL Error:'1054', Message:'Unknown column 'host_id' in 'where clause'' [0, Success] (Spine parent)
DEBUG: SS[0] Script Server Shutdown Started

Somehow my cacti table has an unknown column? Is this the result of me creating the RRD file manually?

[TheWitness]

Replace host_id with the actual id of the host/device.

[stbecker]

Yikes, thank you for being patient. Given the "U" response below, and a correlated post I saw from you, it's either timing out or may not have permissions. The script is returning just the hour from the date command for simplicity, so script duration shouldn't be an issue. And yet, it does take 40s to complete spine. ?

[root@omawikiinf01 bin]# ./spine -R -V 5 -S -H 1
SPINE: Using spine config file [../etc/spine.conf]
2023/11/07 22:15:53 - SPINE: Poller[1] DEBUG: The path_php_server variable is /usr/share/cacti/script_server.php
2023/11/07 22:15:53 - SPINE: Poller[1] DEBUG: The path_cactilog variable is /usr/share/cacti/log/cacti.log
DEBUG: The log_destination variable is 4 (STDOUT)
DEBUG: The path_php variable is /bin/php
DEBUG: The availability_method variable is 2
DEBUG: The ping_recovery_count variable is 3
DEBUG: The ping_failure_count variable is 2
DEBUG: The ping_method variable is 2
DEBUG: The ping_retries variable is 1
DEBUG: The ping_timeout variable is 400
DEBUG: The snmp_retries variable is 3
DEBUG: The log_perror variable is 1
DEBUG: The log_pwarn variable is 0
DEBUG: The boost_redirect variable is 0
DEBUG: The boost_rrd_update_enable variable is 0
DEBUG: The log_pstats variable is 0
DEBUG: The threads variable is 1
DEBUG: The polling interval is 300 seconds
DEBUG: The number of concurrent processes is 1
DEBUG: The script timeout is 25
DEBUG: The selective_device_debug variable is 11,1
DEBUG: The spine_log_level variable is 0
DEBUG: The number of php script servers to run is 1
DEBUG: Device List to be polled='1', TotalPHPScripts='0
DEBUG: The PHP Script Server is Not Required
DEBUG: The Maximum SNMP OID Get Size is 10
Selective Debug Devices 11,1
Version 1.1.38 starting
DEBUG: MySQL is Thread Safe!
DEBUG: Spine is running asroot.
SPINE: Initializing Net-SNMP API
DEBUG: Issues with SNMP Header Version information, assuming old version of Net-SNMP.
SPINE: Initializing PHP Script Server(s)
NOTE: Spine will support multithread device polling.
DEBUG: Initial Value of Active Threads is 0
DEBUG: Valid Thread to be Created
DEBUG: In Poller, About to Start Polling of Device for Device ID 1
SPINE: Active Threads is 1, Pending is 1
Device[1] PING: Result PING: Device does not require ping
Device[1] TH[1] RECACHE: Processing 2 items in the auto reindex cache for 'localhost
Device[1] TH[1] Recache DataQuery[1] OID: .1.3.6.1.2.1.1.3.0, output: U
Device[1] TH[1] ASSERT: 'U' .eq. 'U' failed. Recaching host 'localhost', data query #1
Device[1] TH[1] Recache DataQuery[2] OID: .1.3.6.1.2.1.1.3.0, output: U
Device[1] TH[1] ASSERT: 'U' .eq. 'U' failed. Recaching host 'localhost', data query #2
Device[1] TH[1] NOTE: There are '5' Polling Items for this Device
Device[1] TH[1] Total Time: 40 Seconds
Device[1] TH[1] DEBUG: HOST COMPLETE: About to Exit Device Polling Thread Function
DEBUG: The Value of Active Threads is 0 for Device ID 1
POLLR: Active Threads is 0, Pending is 0
SPINE: The Final Value of Threads is 0
DEBUG: Thread Cleanup Complete
DEBUG: PHP Script Server Pipes Closed
DEBUG: Allocated Variable Memory Freed
DEBUG: MYSQL Free & Close Completed
DEBUG: Net-SNMP Close Completed
Time: 41.0315 s, Threads: 1, Devices: 1
[root@omawikiinf01 bin]#

I should note that security changes have disabled SNMP to the host, though I'm not using SNMP for this input method. As another aside, I'll temporarily "setenforce 0" and try spine in debug again to see if that helps.

...UPDATE: disabling selinux did not make any difference from above.

[stbecker]

I really appreciate your direction and patience TheWitness!

Spine logs showed the timeout for an OID, even though I wasn't using snmp for this data source. The host device was set to use Ping for downed device detection. I can't speak to why an snmp failure would halt all other queries. But fixing snmpd on the device has allowed spine to complete in debug mode and show the correct data.

I'd be willing to bet that setting my device back to snmp for detection and letting a couple polling cycles go by will show the problem has been resolved.

UPDATE: confirmed resolved. Again, many thanks! #AlwaysLearning