TheWitness wrote:Not specific enough.
L
I ran spine in verbose mode, Gandalf suggested me to do it on a different thread. Quoting what I posted there:
Here is the output for the host with the problem:
Code: Select all
SPINE: Using spine config file [spine.conf]
06/22/2009 10:48:50 AM - SPINE: Poller[0] DEBUG: The path_php_server variable is /var/www/html/cacti/script_server.php
06/22/2009 10:48:50 AM - SPINE: Poller[0] DEBUG: The path_cactilog variable is /var/www/html/cacti/log/cacti.log
06/22/2009 10:48:50 AM - SPINE: Poller[0] DEBUG: The log_destination variable is 1 (FILE)
06/22/2009 10:48:50 AM - SPINE: Poller[0] DEBUG: The path_php variable is /usr/bin/php
06/22/2009 10:48:50 AM - SPINE: Poller[0] DEBUG: The availability_method variable is 0
06/22/2009 10:48:50 AM - SPINE: Poller[0] DEBUG: The ping_recovery_count variable is 3
06/22/2009 10:48:50 AM - SPINE: Poller[0] DEBUG: The ping_failure_count variable is 2
06/22/2009 10:48:50 AM - SPINE: Poller[0] DEBUG: The ping_method variable is 2
06/22/2009 10:48:50 AM - SPINE: Poller[0] DEBUG: The ping_retries variable is 1
06/22/2009 10:48:50 AM - SPINE: Poller[0] DEBUG: The ping_timeout variable is 400
06/22/2009 10:48:50 AM - SPINE: Poller[0] DEBUG: The snmp_retries variable is 3
06/22/2009 10:48:50 AM - SPINE: Poller[0] DEBUG: The log_perror variable is 1
06/22/2009 10:48:50 AM - SPINE: Poller[0] DEBUG: The log_pwarn variable is 0
06/22/2009 10:48:50 AM - SPINE: Poller[0] DEBUG: The boost_redirect variable is 0
06/22/2009 10:48:50 AM - SPINE: Poller[0] DEBUG: The log_pstats variable is 0
06/22/2009 10:48:50 AM - SPINE: Poller[0] DEBUG: The threads variable is 10
06/22/2009 10:48:50 AM - SPINE: Poller[0] DEBUG: The polling interval is 60 seconds
06/22/2009 10:48:50 AM - SPINE: Poller[0] DEBUG: The number of concurrent processes is 2
06/22/2009 10:48:50 AM - SPINE: Poller[0] DEBUG: The script timeout is 25
06/22/2009 10:48:50 AM - SPINE: Poller[0] DEBUG: The number of php script servers to run is 2
06/22/2009 10:48:50 AM - SPINE: Poller[0] DEBUG: Host List to be polled='22', TotalPHPScripts='0'
06/22/2009 10:48:50 AM - SPINE: Poller[0] DEBUG: The PHP Script Server is Not Required
06/22/2009 10:48:50 AM - SPINE: Poller[0] DEBUG: The Maximum SNMP OID Get Size is 10
06/22/2009 10:48:50 AM - SPINE: Poller[0] Version 0.8.7d starting
06/22/2009 10:48:50 AM - SPINE: Poller[0] DEBUG: MySQL is Thread Safe!
06/22/2009 10:48:50 AM - SPINE: Poller[0] SPINE: Initializing Net-SNMP API
06/22/2009 10:48:50 AM - SPINE: Poller[0] DEBUG: SNMP Header Version is 5.3.1
06/22/2009 10:48:50 AM - SPINE: Poller[0] DEBUG: SNMP Library Version is 5.3.1
06/22/2009 10:48:50 AM - SPINE: Poller[0] SPINE: Initializing PHP Script Server(s)
06/22/2009 10:48:50 AM - SPINE: Poller[0] DEBUG: Initial Value of Active Threads is 0
06/22/2009 10:48:50 AM - SPINE: Poller[0] DEBUG: Valid Thread to be Created
06/22/2009 10:48:50 AM - SPINE: Poller[0] DEBUG: The Value of Active Threads is 1
06/22/2009 10:48:50 AM - SPINE: Poller[0] DEBUG: In Poller, About to Start Polling of Host
06/22/2009 10:48:50 AM - SPINE: Poller[0] DEBUG: Valid Thread to be Created
06/22/2009 10:48:50 AM - SPINE: Poller[0] DEBUG: The Value of Active Threads is 2
06/22/2009 10:48:50 AM - SPINE: Poller[0] Host[0] DEBUG: HOST COMPLETE: About to Exit Host Polling Thread Function
06/22/2009 10:48:50 AM - SPINE: Poller[0] DEBUG: The Value of Active Threads is 1
06/22/2009 10:48:50 AM - SPINE: Poller[0] DEBUG: In Poller, About to Start Polling of Host
06/22/2009 10:48:50 AM - SPINE: Poller[0] Host[22] No Host Availability Method Selected
06/22/2009 10:48:50 AM - SPINE: Poller[0] Host[22] RECACHE: Processing 2 items in the auto reindex cache for 'myhostname'
06/22/2009 10:48:50 AM - SPINE: Poller[0] Host[22] ASSERT: '151176486' .lt. '0' failed. Recaching host 'myhostname', data query #1
06/22/2009 10:48:50 AM - SPINE: Poller[0] Host[22] NOTICE: Spike Kill in Effect for 'myhostname'
06/22/2009 10:48:50 AM - SPINE: Poller[0] Host[22] ASSERT: '151176486' .lt. '0' failed. Recaching host 'myhostname', data query #8
06/22/2009 10:48:50 AM - SPINE: Poller[0] Host[22] NOTICE: Spike Kill in Effect for 'myhostname'
06/22/2009 10:48:50 AM - SPINE: Poller[0] Host[22] NOTE: There are '12' Polling Items for this Host
06/22/2009 10:48:50 AM - SPINE: Poller[0] Host[22] DS[535] SNMP: v2: myhostname, dsname: ucd_load5min, oid: .1.3.6.1.4.1.2021.10.1.3.2, value: 0.23
06/22/2009 10:48:50 AM - SPINE: Poller[0] Host[22] DS[534] SNMP: v2: myhostname, dsname: ucd_load1min, oid: .1.3.6.1.4.1.2021.10.1.3.1, value: 0.19
06/22/2009 10:48:50 AM - SPINE: Poller[0] Host[22] DS[533] SNMP: v2: myhostname, dsname: ucd_load15min, oid: .1.3.6.1.4.1.2021.10.1.3.3, value: 0.23
06/22/2009 10:48:50 AM - SPINE: Poller[0] Host[22] DS[532] SNMP: v2: myhostname, dsname: ucd_ssCpuRawWait, oid: .1.3.6.1.4.1.2021.11.54.0, value: 0
06/22/2009 10:48:50 AM - SPINE: Poller[0] Host[22] DS[529] SNMP: v2: myhostname, dsname: ucd_ssCpuRawIdle, oid: .1.3.6.1.4.1.2021.11.53.0, value: 302935802
06/22/2009 10:48:50 AM - SPINE: Poller[0] Host[22] DS[530] SNMP: v2: myhostname, dsname: ucd_ssCpuRawKernel, oid: .1.3.6.1.4.1.2021.11.55.0, value: 235847456
06/22/2009 10:48:50 AM - SPINE: Poller[0] Host[22] DS[531] SNMP: v2: myhostname, dsname: ucd_ssCpuRawUser, oid: .1.3.6.1.4.1.2021.11.50.0, value: 263276848
06/22/2009 10:48:50 AM - SPINE: Poller[0] Host[22] DS[536] SNMP: v2: myhostname, dsname: ucd_memAvailReal, oid: .1.3.6.1.4.1.2021.4.6.0, value: 1803928
06/22/2009 10:48:50 AM - SPINE: Poller[0] Host[22] DS[537] SNMP: v2: myhostname, dsname: ucd_memTotalReal, oid: .1.3.6.1.4.1.2021.4.5.0, value: 16777216
06/22/2009 10:48:50 AM - SPINE: Poller[0] Host[22] DS[538] SNMP: v2: myhostname, dsname: ucd_memAvailSwap, oid: .1.3.6.1.4.1.2021.4.4.0, value: 29545648
06/22/2009 10:48:50 AM - SPINE: Poller[0] Host[22] DS[539] SNMP: v2: myhostname, dsname: ucd_memTotalSwap, oid: .1.3.6.1.4.1.2021.4.3.0, value: 32771800
06/22/2009 10:48:50 AM - SPINE: Poller[0] Host[22] DS[540] SNMP: v2: myhostname, dsname: ucd_hrSystemProcess, oid: .1.3.6.1.2.1.25.1.6.0, value: 126
06/22/2009 10:48:50 AM - SPINE: Poller[0] Host[22] DEBUG: HOST COMPLETE: About to Exit Host Polling Thread Function
06/22/2009 10:48:50 AM - SPINE: Poller[0] DEBUG: The Value of Active Threads is 0
06/22/2009 10:48:50 AM - SPINE: Poller[0] DEBUG: Thread Cleanup Complete
06/22/2009 10:48:50 AM - SPINE: Poller[0] DEBUG: PHP Script Server Pipes Closed
06/22/2009 10:48:50 AM - SPINE: Poller[0] DEBUG: Allocated Variable Memory Freed
06/22/2009 10:48:50 AM - SPINE: Poller[0] DEBUG: MYSQL Free & Close Completed
06/22/2009 10:48:50 AM - SPINE: Poller[0] Time: 0.1227 s, Threads: 10, Hosts: 2
the following lines are "suspect":
Code: Select all
06/22/2009 10:48:50 AM - SPINE: Poller[0] Host[22] RECACHE: Processing 2 items in the auto reindex cache for 'myhostname'
06/22/2009 10:48:50 AM - SPINE: Poller[0] Host[22] ASSERT: '151176486' .lt. '0' failed. Recaching host 'myhostname', data query #1
06/22/2009 10:48:50 AM - SPINE: Poller[0] Host[22] NOTICE: Spike Kill in Effect for 'myhostname'
06/22/2009 10:48:50 AM - SPINE: Poller[0] Host[22] ASSERT: '151176486' .lt. '0' failed. Recaching host 'myhostname', data query #8
06/22/2009 10:48:50 AM - SPINE: Poller[0] Host[22] NOTICE: Spike Kill in Effect for 'myhostname'
but I'm not sure what does "Spike Kill" mean. I ran the command several times and I get the recache event every single time, same for the "Spike Kill" notice.
This is a similar command launched for a working host with the OS, template and queries, I get the recache event, but not the "Spike Kill".
Code: Select all
SPINE: Using spine config file [spine.conf]
06/22/2009 10:52:44 AM - SPINE: Poller[0] DEBUG: The path_php_server variable is /var/www/html/cacti/script_server.php
06/22/2009 10:52:44 AM - SPINE: Poller[0] DEBUG: The path_cactilog variable is /var/www/html/cacti/log/cacti.log
06/22/2009 10:52:44 AM - SPINE: Poller[0] DEBUG: The log_destination variable is 1 (FILE)
06/22/2009 10:52:44 AM - SPINE: Poller[0] DEBUG: The path_php variable is /usr/bin/php
06/22/2009 10:52:44 AM - SPINE: Poller[0] DEBUG: The availability_method variable is 0
06/22/2009 10:52:44 AM - SPINE: Poller[0] DEBUG: The ping_recovery_count variable is 3
06/22/2009 10:52:44 AM - SPINE: Poller[0] DEBUG: The ping_failure_count variable is 2
06/22/2009 10:52:44 AM - SPINE: Poller[0] DEBUG: The ping_method variable is 2
06/22/2009 10:52:44 AM - SPINE: Poller[0] DEBUG: The ping_retries variable is 1
06/22/2009 10:52:44 AM - SPINE: Poller[0] DEBUG: The ping_timeout variable is 400
06/22/2009 10:52:44 AM - SPINE: Poller[0] DEBUG: The snmp_retries variable is 3
06/22/2009 10:52:44 AM - SPINE: Poller[0] DEBUG: The log_perror variable is 1
06/22/2009 10:52:44 AM - SPINE: Poller[0] DEBUG: The log_pwarn variable is 0
06/22/2009 10:52:44 AM - SPINE: Poller[0] DEBUG: The boost_redirect variable is 0
06/22/2009 10:52:44 AM - SPINE: Poller[0] DEBUG: The log_pstats variable is 0
06/22/2009 10:52:44 AM - SPINE: Poller[0] DEBUG: The threads variable is 10
06/22/2009 10:52:44 AM - SPINE: Poller[0] DEBUG: The polling interval is 60 seconds
06/22/2009 10:52:44 AM - SPINE: Poller[0] DEBUG: The number of concurrent processes is 2
06/22/2009 10:52:44 AM - SPINE: Poller[0] DEBUG: The script timeout is 25
06/22/2009 10:52:44 AM - SPINE: Poller[0] DEBUG: The number of php script servers to run is 2
06/22/2009 10:52:44 AM - SPINE: Poller[0] DEBUG: Host List to be polled='21', TotalPHPScripts='0'
06/22/2009 10:52:44 AM - SPINE: Poller[0] DEBUG: The PHP Script Server is Not Required
06/22/2009 10:52:44 AM - SPINE: Poller[0] DEBUG: The Maximum SNMP OID Get Size is 10
06/22/2009 10:52:44 AM - SPINE: Poller[0] Version 0.8.7d starting
06/22/2009 10:52:44 AM - SPINE: Poller[0] DEBUG: MySQL is Thread Safe!
06/22/2009 10:52:44 AM - SPINE: Poller[0] SPINE: Initializing Net-SNMP API
06/22/2009 10:52:44 AM - SPINE: Poller[0] DEBUG: SNMP Header Version is 5.3.1
06/22/2009 10:52:44 AM - SPINE: Poller[0] DEBUG: SNMP Library Version is 5.3.1
06/22/2009 10:52:44 AM - SPINE: Poller[0] SPINE: Initializing PHP Script Server(s)
06/22/2009 10:52:44 AM - SPINE: Poller[0] DEBUG: Initial Value of Active Threads is 0
06/22/2009 10:52:44 AM - SPINE: Poller[0] DEBUG: Valid Thread to be Created
06/22/2009 10:52:44 AM - SPINE: Poller[0] DEBUG: The Value of Active Threads is 1
06/22/2009 10:52:44 AM - SPINE: Poller[0] DEBUG: In Poller, About to Start Polling of Host
06/22/2009 10:52:44 AM - SPINE: Poller[0] DEBUG: Valid Thread to be Created
06/22/2009 10:52:44 AM - SPINE: Poller[0] DEBUG: The Value of Active Threads is 2
06/22/2009 10:52:44 AM - SPINE: Poller[0] DEBUG: In Poller, About to Start Polling of Host
06/22/2009 10:52:44 AM - SPINE: Poller[0] Host[0] DEBUG: HOST COMPLETE: About to Exit Host Polling Thread Function
06/22/2009 10:52:44 AM - SPINE: Poller[0] DEBUG: The Value of Active Threads is 1
06/22/2009 10:52:44 AM - SPINE: Poller[0] Host[21] No Host Availability Method Selected
06/22/2009 10:52:44 AM - SPINE: Poller[0] Host[21] RECACHE: Processing 2 items in the auto reindex cache for 'mygoodhost'
06/22/2009 10:52:44 AM - SPINE: Poller[0] Host[21] NOTE: There are '12' Polling Items for this Host
06/22/2009 10:52:44 AM - SPINE: Poller[0] Host[21] DS[528] SNMP: v2: mygoodhost, dsname: ucd_hrSystemProcess, oid: .1.3.6.1.2.1.25.1.6.0, value: 468
06/22/2009 10:52:44 AM - SPINE: Poller[0] Host[21] DS[527] SNMP: v2: mygoodhost, dsname: ucd_memTotalSwap, oid: .1.3.6.1.4.1.2021.4.3.0, value: 16780216
06/22/2009 10:52:44 AM - SPINE: Poller[0] Host[21] DS[526] SNMP: v2: mygoodhost, dsname: ucd_memAvailSwap, oid: .1.3.6.1.4.1.2021.4.4.0, value: 12092504
06/22/2009 10:52:44 AM - SPINE: Poller[0] Host[21] DS[525] SNMP: v2: mygoodhost, dsname: ucd_memTotalReal, oid: .1.3.6.1.4.1.2021.4.5.0, value: 12582912
06/22/2009 10:52:44 AM - SPINE: Poller[0] Host[21] DS[524] SNMP: v2: mygoodhost, dsname: ucd_memAvailReal, oid: .1.3.6.1.4.1.2021.4.6.0, value: 6732840
06/22/2009 10:52:44 AM - SPINE: Poller[0] Host[21] DS[523] SNMP: v2: mygoodhost, dsname: ucd_load5min, oid: .1.3.6.1.4.1.2021.10.1.3.2, value: 0.96
06/22/2009 10:52:44 AM - SPINE: Poller[0] Host[21] DS[522] SNMP: v2: mygoodhost, dsname: ucd_load1min, oid: .1.3.6.1.4.1.2021.10.1.3.1, value: 1.39
06/22/2009 10:52:44 AM - SPINE: Poller[0] Host[21] DS[521] SNMP: v2: mygoodhost, dsname: ucd_load15min, oid: .1.3.6.1.4.1.2021.10.1.3.3, value: 0.75
06/22/2009 10:52:44 AM - SPINE: Poller[0] Host[21] DS[517] SNMP: v2: mygoodhost, dsname: ucd_ssCpuRawIdle, oid: .1.3.6.1.4.1.2021.11.53.0, value: 632333902
06/22/2009 10:52:44 AM - SPINE: Poller[0] Host[21] DS[518] SNMP: v2: mygoodhost, dsname: ucd_ssCpuRawKernel, oid: .1.3.6.1.4.1.2021.11.55.0, value: 51424504
06/22/2009 10:52:44 AM - SPINE: Poller[0] Host[21] DS[519] SNMP: v2: mygoodhost, dsname: ucd_ssCpuRawUser, oid: .1.3.6.1.4.1.2021.11.50.0, value: 223157420
06/22/2009 10:52:44 AM - SPINE: Poller[0] Host[21] DS[520] SNMP: v2: mygoodhost, dsname: ucd_ssCpuRawWait, oid: .1.3.6.1.4.1.2021.11.54.0, value: 0
06/22/2009 10:52:44 AM - SPINE: Poller[0] Host[21] DEBUG: HOST COMPLETE: About to Exit Host Polling Thread Function
06/22/2009 10:52:44 AM - SPINE: Poller[0] DEBUG: The Value of Active Threads is 0
06/22/2009 10:52:44 AM - SPINE: Poller[0] DEBUG: Thread Cleanup Complete
06/22/2009 10:52:44 AM - SPINE: Poller[0] DEBUG: PHP Script Server Pipes Closed
06/22/2009 10:52:44 AM - SPINE: Poller[0] DEBUG: Allocated Variable Memory Freed
06/22/2009 10:52:44 AM - SPINE: Poller[0] DEBUG: MYSQL Free & Close Completed
06/22/2009 10:52:44 AM - SPINE: Poller[0] Time: 0.1263 s, Threads: 10, Hosts: 2
Thanks!
And