I have absolutely no clue what I am doing differently on our production box that causes this one host to constantly fail.
I have PHP 4.4.4 without SNMP extension compiled in, net-snmp 5.1.2 and I'm using cactid. This is what I get by polling manually:
Code: Select all
management:/www/cacti$ /usr/local/cactid/bin/cactid -f 191 -l 191 --verbosity=9 --snmponly -S
CACTID: Using cactid config file [/etc/cactid.conf]
WARNING: Unrecongized directive: Threads=10 in /etc/cactid.conf
CACTID: DEBUG: MySQL Query ID '0': 'SELECT value FROM settings WHERE name = 'path_webroot''
CACTID: DEBUG: MySQL Query ID '0': OK
CACTID: DEBUG: MySQL Query ID '1': 'SELECT value FROM settings WHERE name = 'path_cactilog''
CACTID: DEBUG: MySQL Query ID '1': OK
CACTID: DEBUG: The path_php_server variable is /www/cacti-0.8.6i/script_server.php
CACTID: DEBUG: The path_cactilog variable is /var/log/httpd/cacti.log
DEBUG: The log_destination variable is 4 (STDOUT)
DEBUG: MySQL Query ID '2': 'SELECT value FROM settings WHERE name = 'path_php_binary''
DEBUG: MySQL Query ID '2': OK
DEBUG: The path_php variable is /usr/local/bin/php
DEBUG: MySQL Query ID '3': 'SELECT value FROM settings WHERE name = 'availability_method''
DEBUG: MySQL Query ID '3': OK
DEBUG: The availability_method variable is 1
DEBUG: MySQL Query ID '4': 'SELECT value FROM settings WHERE name = 'ping_recovery_count''
DEBUG: MySQL Query ID '4': OK
DEBUG: The ping_recovery_count variable is 3
DEBUG: MySQL Query ID '5': 'SELECT value FROM settings WHERE name = 'ping_failure_count''
DEBUG: MySQL Query ID '5': OK
DEBUG: The ping_failure_count variable is 2
DEBUG: MySQL Query ID '6': 'SELECT value FROM settings WHERE name = 'ping_method''
DEBUG: MySQL Query ID '6': OK
DEBUG: The ping_method variable is 2
DEBUG: MySQL Query ID '7': 'SELECT value FROM settings WHERE name = 'ping_retries''
DEBUG: MySQL Query ID '7': OK
DEBUG: The ping_retries variable is 1
DEBUG: MySQL Query ID '8': 'SELECT value FROM settings WHERE name = 'ping_timeout''
DEBUG: MySQL Query ID '8': OK
DEBUG: The ping_timeout variable is 100
DEBUG: MySQL Query ID '9': 'SELECT value FROM settings WHERE name = 'log_perror''
DEBUG: MySQL Query ID '9': OK
DEBUG: The log_perror variable is 1
DEBUG: MySQL Query ID '10': 'SELECT value FROM settings WHERE name = 'log_pwarn''
DEBUG: MySQL Query ID '10': OK
DEBUG: The log_pwarn variable is 1
DEBUG: MySQL Query ID '11': 'SELECT value FROM settings WHERE name = 'log_pstats''
DEBUG: MySQL Query ID '11': OK
DEBUG: The log_pstats variable is 0
DEBUG: MySQL Query ID '12': 'SELECT value FROM settings WHERE name = 'max_threads''
DEBUG: MySQL Query ID '12': OK
DEBUG: The threads variable is 10
DEBUG: MySQL Query ID '13': 'SELECT value FROM settings WHERE name = 'poller_interval''
DEBUG: MySQL Query ID '13': OK
DEBUG: The polling interval is the system default
DEBUG: MySQL Query ID '14': 'SELECT value FROM settings WHERE name = 'concurrent_processes''
DEBUG: MySQL Query ID '14': OK
DEBUG: The number of concurrent processes is 1
DEBUG: MySQL Query ID '15': 'SELECT value FROM settings WHERE name = 'script_timeout''
DEBUG: MySQL Query ID '15': OK
DEBUG: The script timeout is 10
DEBUG: MySQL Query ID '16': 'SELECT value FROM settings WHERE name = 'php_servers''
DEBUG: MySQL Query ID '16': OK
DEBUG: The number of php script servers to run is 1
DEBUG: MySQL Query ID '17': 'SELECT action FROM poller_item WHERE action=2 AND host_id BETWEEN 191 AND 191 LIMIT 1'
DEBUG: MySQL Query ID '17': OK
DEBUG: StartHost='191', EndHost='191', TotalPHPScripts='0'
DEBUG: The PHP Script Server is Not Required
DEBUG: MySQL Query ID '18': 'SELECT value FROM settings WHERE name = 'max_get_size''
DEBUG: MySQL Query ID '18': OK
DEBUG: The Maximum SNMP OID Get Size is 10
CACTID: Version 0.8.6i starting
MYSQL: Connecting to MySQL database 'cacti' on 'alksdjf.de'...
MYSQL: Connected to MySQL database 'cacti' on 'alksdjf.de'...
CACTID: Initializing Net-SNMP API
DEBUG: SNMP Header Version is 5.1.2
DEBUG: SNMP Library Version is 5.1.2
CACTID: Initializing PHP Script Server(s)
DEBUG: MySQL Query ID '19': 'SELECT id FROM host WHERE disabled='' AND id BETWEEN 191 AND 191 ORDER BY id'
DEBUG: MySQL Query ID '19': OK
DEBUG: Initial Value of Active Threads is 0
DEBUG: Valid Thread to be Created
DEBUG: The Value of Active Threads is 1
DEBUG: In Poller, About to Start Polling of Host
MYSQL: Connected to MySQL database 'cacti' on 'alksdjf.de'...
MYSQL: Connected to MySQL database 'cacti' on 'alksdjf.de'...
DEBUG: MySQL Query ID '20': 'SELECT action, hostname, snmp_community, snmp_version, snmp_username, snmp_password, rrd_name, rrd_path, arg1, arg2, arg3, local_data_id, rrd_num, snmp_port, snmp_timeout FROM poller_item WHERE host_id=0 ORDER BY arg1'
DEBUG: MySQL Query ID '20': OK
Host[0] DEBUG: HOST COMPLETE: About to Exit Host Polling Thread Function
DEBUG: Valid Thread to be Created
DEBUG: The Value of Active Threads is 2
DEBUG: In Poller, About to Start Polling of Host
MYSQL: Connected to MySQL database 'cacti' on 'alksdjf.de'...
MYSQL: Connected to MySQL database 'cacti' on 'alksdjf.de'...
DEBUG: MySQL Query ID '21': 'SELECT id, hostname, snmp_community, snmp_username, snmp_password, snmp_version, snmp_port, snmp_timeout, status, status_event_count, status_fail_date, status_rec_date, status_last_error, min_time, max_time, cur_time, avg_time, total_polls, failed_polls, availability FROM host WHERE id=191'
DEBUG: MySQL Query ID '21': OK
DEBUG: The UDP Ping return_code was -1, errno was 111, total_time was 1909.0176
Host[191] PING Result: UDP: Host is Alive
Host[191] SNMP Result: SNMP not performed due to setting or ping result
DEBUG: MySQL Insert ID '0': 'update host set status='3', status_event_count='0', status_fail_date='0000-00-00 00:00:00', status_rec_date='0000-00-00 00:00:00', status_last_error='', min_time='0.941040', max_time='35.372970', cur_time='0.954510', avg_time='5.080469', total_polls='15', failed_polls='0', availability='100.0000' where id='191''
DEBUG: MySQL Insert ID '0': OK
DEBUG: MySQL Query ID '22': 'SELECT data_query_id, action, op, assert_value, arg1 FROM poller_reindex WHERE host_id=191'
DEBUG: MySQL Query ID '22': OK
Host[191] RECACHE: Processing 1 items in the auto reindex cache for '217.188.214.50'
DEBUG: MySQL Insert ID '1': 'update poller_reindex set assert_value='87659816' where host_id='191' and data_query_id='1' and arg1='.1.3.6.1.2.1.1.3.0''
DEBUG: MySQL Insert ID '1': OK
DEBUG: MySQL Query ID '23': 'SELECT action, hostname, snmp_community, snmp_version, snmp_username, snmp_password, rrd_name, rrd_path, arg1, arg2, arg3, local_data_id, rrd_num, snmp_port, snmp_timeout FROM poller_item WHERE host_id=191 ORDER BY arg1'
DEBUG: MySQL Query ID '23': OK
DEBUG: The Value of Active Threads is 1
Host[191] DS[3446] WARNING: Result from SNMP not valid. Partial Result: ...
Host[191] DS[3446] SNMP: v1: 217.188.214.50, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1, value: U
Host[191] DS[3455] WARNING: Result from SNMP not valid. Partial Result: ...
Host[191] DS[3455] SNMP: v1: 217.188.214.50, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.10, value: U
Host[191] DS[3478] SNMP: v1: 217.188.214.50, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.100000, value: 0
Host[191] DS[3479] SNMP: v1: 217.188.214.50, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.100010, value: 0
Host[191] DS[3480] SNMP: v1: 217.188.214.50, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.100049, value: 0
Host[191] DS[3456] WARNING: Result from SNMP not valid. Partial Result: ...
Host[191] DS[3456] SNMP: v1: 217.188.214.50, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.11, value: U
Host[191] DS[3457] WARNING: Result from SNMP not valid. Partial Result: ...
Host[191] DS[3457] SNMP: v1: 217.188.214.50, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.12, value: U
Host[191] DS[3458] WARNING: Result from SNMP not valid. Partial Result: ...
Host[191] DS[3458] SNMP: v1: 217.188.214.50, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.13, value: U
Host[191] DS[3459] WARNING: Result from SNMP not valid. Partial Result: ...
Host[191] DS[3459] SNMP: v1: 217.188.214.50, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.14, value: U
Host[191] DS[3460] WARNING: Result from SNMP not valid. Partial Result: ...
Host[191] DS[3460] SNMP: v1: 217.188.214.50, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.15, value: U
Host[191] DS[3461] WARNING: SNMP timeout detected [1000 ms], ignoring host '217.188.214.50'
Host[191] DS[3461] SNMP: v1: 217.188.214.50, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.16, value: U
Host[191] DS[3462] WARNING: SNMP timeout detected [1000 ms], ignoring host '217.188.214.50'
Host[191] DS[3462] SNMP: v1: 217.188.214.50, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.17, value: U
Host[191] DS[3463] WARNING: SNMP timeout detected [1000 ms], ignoring host '217.188.214.50'
Host[191] DS[3463] SNMP: v1: 217.188.214.50, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.18, value: U
Host[191] DS[3464] WARNING: SNMP timeout detected [1000 ms], ignoring host '217.188.214.50'
Host[191] DS[3464] SNMP: v1: 217.188.214.50, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.19, value: U
Host[191] DS[3447] WARNING: SNMP timeout detected [1000 ms], ignoring host '217.188.214.50'
Host[191] DS[3447] SNMP: v1: 217.188.214.50, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.2, value: U
Host[191] DS[3465] WARNING: SNMP timeout detected [1000 ms], ignoring host '217.188.214.50'
Host[191] DS[3465] SNMP: v1: 217.188.214.50, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.20, value: U
Host[191] DS[3466] WARNING: SNMP timeout detected [1000 ms], ignoring host '217.188.214.50'
Host[191] DS[3466] SNMP: v1: 217.188.214.50, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.21, value: U
Host[191] DS[3467] WARNING: SNMP timeout detected [1000 ms], ignoring host '217.188.214.50'
Host[191] DS[3467] SNMP: v1: 217.188.214.50, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.22, value: U
Host[191] DS[3468] WARNING: SNMP timeout detected [1000 ms], ignoring host '217.188.214.50'
Host[191] DS[3468] SNMP: v1: 217.188.214.50, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.23, value: U
Host[191] DS[3469] WARNING: SNMP timeout detected [1000 ms], ignoring host '217.188.214.50'
Host[191] DS[3469] SNMP: v1: 217.188.214.50, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.24, value: U
Host[191] DS[3470] WARNING: SNMP timeout detected [1000 ms], ignoring host '217.188.214.50'
Host[191] DS[3470] SNMP: v1: 217.188.214.50, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.25, value: U
DEBUG: MySQL Insert ID '2': 'INSERT INTO poller_output (local_data_id, rrd_name, time, output) VALUES (3446,'traffic_out','2006-11-19 20:07:42','U'),(3455,'traffic_out','2006-11-19 20:07:42','U'),(3478,'traffic_out','2006-11-19 20:07:42','0'),(3479,'traffic_out','2006-11-19 20:07:42','0'),(3480,'traffic_out','2006-11-19 20:07:42','0'),(3456,'traffic_out','2006-11-19 20:07:42','U'),(3457,'traffic_out','2006-11-19 20:07:42','U'),(3458,'traffic_out','2006-11-19 20:07:42','U'),(3459,'traffic_out','2006-11-19 20:07:42','U'),(3460,'traffic_out','2006-11-19 20:07:42','U'),(3461,'traffic_out','2006-11-19 20:07:42','U'),(3462,'traffic_out','2006-11-19 20:07:42','U'),(3463,'traffic_out','2006-11-19 20:07:42','U'),(3464,'traffic_out','2006-11-19 20:07:42','U'),(3447,'traffic_out','2006-11-19 20:07:42','U'),(3465,'traffic_out','2006-11-19 20:07:42','U'),(3466,'traffic_out','2006-11-19 20:07:42','U'),(3467,'traffic_out','2006-11-19 20:07:42','U'),(3468,'traffic_out','2006-11-19 20:07:42','U'),(3469,'traffic_out','2006-11-19 20:07:42','U'),(3470,'traffic_out','2006-11-19 20:07:42','U')'
DEBUG: MySQL Insert ID '2': OK
Host[191] DEBUG: HOST COMPLETE: About to Exit Host Polling Thread Function
DEBUG: The Value of Active Threads is 0
DEBUG: MySQL Insert ID '3': 'replace into settings (name,value) values ('date',NOW())'
DEBUG: MySQL Insert ID '3': OK
DEBUG: MySQL Insert ID '4': 'insert into poller_time (poller_id, start_time, end_time) values (0, NOW(), NOW())'
DEBUG: MySQL Insert ID '4': OK
DEBUG: Thread Cleanup Complete
DEBUG: PHP Script Server Pipes Closed
DEBUG: Allocated Variable Memory Freed
CACTID: Net-SNMP API Shutdown Completed
DEBUG: MYSQL Free & Close Completed
Time: 0.5535 s, Threads: 10, Hosts: 2
a) the "did not perform snmp due to settings" since the host is up and of course also enabled in cacti, and pinging it was obviously successful
b) the "timeout detected" message is obviously totally idiotic, since the whole poll does not take longer than 550 ms (see last line of output) and there can hardly be a multiple 1000ms timeout be hit.
Any ideas?