rrd's not getting updated

barneyg · Post by **barneyg** » Fri Dec 08, 2006 5:04 pm

I have been running cacti for about 2 months now with no issues. Overnight the rrd files stopped getting updates all at one time, and I can't figure out why. I see no errors in the logs other then some warnings about a few devices who aren't replying with snmp values. The permissions appear correct, and there are no errors in the log files. I've tried running cmd.php as well (cactid normally runs) with no luck. What else can I try?

Post by **gandalf** » Sat Dec 09, 2006 5:15 am

Please check cacti.log file size and overall disk partition usage. Make sure, crontab runs the poller every 5 mins. Run cactid manually using --verbosity=5 for a single host and post output
Reinhard

lungs71 · Post by **lungs71** » Sat Dec 09, 2006 8:30 am

Hi Reinhard - I work with barneyg. I have checked the items you suggested and do not see a problem there. Below is the output of the cactid manual run. This is a odd problem, we had not made any changes to the system in about 48hrs and for some reasion data just stopped being posted to the rra files. Any help would be great.

./cactid --verbosity=5 99 99
CACTID: Using cactid config file [cactid.conf]
CACTID: DEBUG: MySQL Query ID '0': 'SELECT value FROM settings WHERE name = 'path_webroot''
CACTID: DEBUG: MySQL Query ID '0': OK
CACTID: DEBUG: MySQL Query ID '1': 'SELECT value FROM settings WHERE name = 'path_cactilog''
CACTID: DEBUG: MySQL Query ID '1': OK
CACTID: DEBUG: The path_php_server variable is /var/www/html/script_server.php
CACTID: DEBUG: The path_cactilog variable is /var/www/html/log/cacti.log
CACTID: DEBUG: MySQL Query ID '2': 'SELECT value FROM settings WHERE name = 'log_destination''
CACTID: DEBUG: MySQL Query ID '2': OK
CACTID: DEBUG: The log_destination variable is 1 (FILE)
CACTID: DEBUG: MySQL Query ID '3': 'SELECT value FROM settings WHERE name = 'path_php_binary''
CACTID: DEBUG: MySQL Query ID '3': OK
CACTID: DEBUG: The path_php variable is /usr/bin/php
CACTID: DEBUG: MySQL Query ID '4': 'SELECT value FROM settings WHERE name = 'availability_method''
CACTID: DEBUG: MySQL Query ID '4': OK
CACTID: DEBUG: The availability_method variable is 2
CACTID: DEBUG: MySQL Query ID '5': 'SELECT value FROM settings WHERE name = 'ping_recovery_count''
CACTID: DEBUG: MySQL Query ID '5': OK
CACTID: DEBUG: The ping_recovery_count variable is 3
CACTID: DEBUG: MySQL Query ID '6': 'SELECT value FROM settings WHERE name = 'ping_failure_count''
CACTID: DEBUG: MySQL Query ID '6': OK
CACTID: DEBUG: The ping_failure_count variable is 2
CACTID: DEBUG: MySQL Query ID '7': 'SELECT value FROM settings WHERE name = 'ping_method''
CACTID: DEBUG: MySQL Query ID '7': OK
CACTID: DEBUG: The ping_method variable is 2
CACTID: DEBUG: MySQL Query ID '8': 'SELECT value FROM settings WHERE name = 'ping_retries''
CACTID: DEBUG: MySQL Query ID '8': OK
CACTID: DEBUG: The ping_retries variable is 1
CACTID: DEBUG: MySQL Query ID '9': 'SELECT value FROM settings WHERE name = 'ping_timeout''
CACTID: DEBUG: MySQL Query ID '9': OK
CACTID: DEBUG: The ping_timeout variable is 400
CACTID: DEBUG: MySQL Query ID '10': 'SELECT value FROM settings WHERE name = 'log_perror''
CACTID: DEBUG: MySQL Query ID '10': OK
CACTID: DEBUG: The log_perror variable is 1
CACTID: DEBUG: MySQL Query ID '11': 'SELECT value FROM settings WHERE name = 'log_pwarn''
CACTID: DEBUG: MySQL Query ID '11': OK
CACTID: DEBUG: The log_pwarn variable is 1
CACTID: DEBUG: MySQL Query ID '12': 'SELECT value FROM settings WHERE name = 'log_pstats''
CACTID: DEBUG: MySQL Query ID '12': OK
CACTID: DEBUG: The log_pstats variable is 1
CACTID: DEBUG: MySQL Query ID '13': 'SELECT value FROM settings WHERE name = 'max_threads''
CACTID: DEBUG: MySQL Query ID '13': OK
CACTID: DEBUG: The threads variable is 4
CACTID: DEBUG: MySQL Query ID '14': 'SELECT value FROM settings WHERE name = 'poller_interval''
CACTID: DEBUG: MySQL Query ID '14': OK
CACTID: DEBUG: The polling interval is the system default
CACTID: DEBUG: MySQL Query ID '15': 'SELECT value FROM settings WHERE name = 'concurrent_processes''
CACTID: DEBUG: MySQL Query ID '15': OK
CACTID: DEBUG: The number of concurrent processes is 1
CACTID: DEBUG: MySQL Query ID '16': 'SELECT value FROM settings WHERE name = 'script_timeout''
CACTID: DEBUG: MySQL Query ID '16': OK
CACTID: DEBUG: The script timeout is 25
CACTID: DEBUG: MySQL Query ID '17': 'SELECT value FROM settings WHERE name = 'php_servers''
CACTID: DEBUG: MySQL Query ID '17': OK
CACTID: DEBUG: The number of php script servers to run is 2
CACTID: DEBUG: MySQL Query ID '18': 'SELECT action FROM poller_item WHERE action=2 AND host_id BETWEEN 99 AND 99 LIMIT 1'
CACTID: DEBUG: MySQL Query ID '18': OK
CACTID: DEBUG: StartHost='99', EndHost='99', TotalPHPScripts='0'
CACTID: DEBUG: The PHP Script Server is Not Required
CACTID: DEBUG: MySQL Query ID '19': 'SELECT value FROM settings WHERE name = 'max_get_size''
CACTID: DEBUG: MySQL Query ID '19': OK
CACTID: DEBUG: The Maximum SNMP OID Get Size is 20
CACTID: CACTID: Version 0.8.6h starting
CACTID: MYSQL: Connecting to MySQL database 'cacti' on 'localhost'...
CACTID: MYSQL: Connected to MySQL database 'cacti' on 'localhost'...
CACTID: CACTID: Initializing Net-SNMP API
CACTID: DEBUG: SNMP Header Version is 5.1.2
CACTID: DEBUG: SNMP Library Version is 5.1.2
CACTID: CACTID: Initializing PHP Script Server(s)
CACTID: DEBUG: MySQL Query ID '20': 'SELECT id FROM host WHERE disabled='' AND id BETWEEN 99 AND 99 ORDER BY id'
CACTID: DEBUG: MySQL Query ID '20': OK
CACTID: DEBUG: Initial Value of Active Threads is 0
CACTID: DEBUG: Valid Thread to be Created
CACTID: DEBUG: The Value of Active Threads is 1
CACTID: DEBUG: In Poller, About to Start Polling of Host
CACTID: MYSQL: Connecting to MySQL database 'cacti' on 'localhost'...
CACTID: MYSQL: Connected to MySQL database 'cacti' on 'localhost'...
CACTID: DEBUG: MySQL Query ID '21': 'SELECT action,hostname,snmp_community,snmp_version,snmp_username,snmp_password,rrd_name,rrd_path,arg1,arg2,arg3,local_data_id,rrd_num,snmp_port,snmp_timeout FROM poller_item WHERE host_id=0 ORDER BY arg1'
CACTID: DEBUG: MySQL Query ID '21': OK
CACTID: Host[0] DEBUG: HOST COMPLETE: About to Exit Host Polling Thread Function
CACTID: DEBUG: Valid Thread to be Created
CACTID: DEBUG: The Value of Active Threads is 2
CACTID: DEBUG: In Poller, About to Start Polling of Host
CACTID: MYSQL: Connecting to MySQL database 'cacti' on 'localhost'...
CACTID: MYSQL: Connected to MySQL database 'cacti' on 'localhost'...
CACTID: DEBUG: MySQL Query ID '22': 'SELECT id, hostname,snmp_community,snmp_username,snmp_password,snmp_version,snmp_port,snmp_timeout,status,status_event_count,status_fail_date,status_rec_date,status_last_error,min_time,max_time,cur_time,avg_time,total_polls,failed_polls,availability FROM host WHERE id=99'
CACTID: DEBUG: MySQL Query ID '22': OK
CACTID: Host[99] SNMP Result: Host responded to SNMP
CACTID: DEBUG: MySQL Insert ID '0': 'update host set status='3',status_event_count='0', status_fail_date='0000-00-00 00:00:00',status_rec_date='0000-00-00 00:00:00',status_last_error='',min_time='9.999990',max_time='604.600000',cur_time='39.477110',avg_time='42.939307',total_polls='14226',failed_polls='0',availability='100.0000' where id='99''
CACTID: DEBUG: MySQL Insert ID '0': OK
CACTID: DEBUG: MySQL Query ID '23': 'SELECT data_query_id,action,op,assert_value,arg1 FROM poller_reindex WHERE host_id=99'
CACTID: DEBUG: MySQL Query ID '23': OK
CACTID: Host[99] RECACHE: Processing 1 items in the auto reindex cache for '192.168.117.8'
CACTID: DEBUG: The Value of Active Threads is 1
CACTID: DEBUG: MySQL Insert ID '1': 'update poller_reindex set assert_value='612526130' where host_id='99' and data_query_id='1' and arg1='.1.3.6.1.2.1.1.3.0''
CACTID: DEBUG: MySQL Insert ID '1': OK
CACTID: DEBUG: MySQL Query ID '24': 'SELECT action,hostname,snmp_community,snmp_version,snmp_username,snmp_password,rrd_name,rrd_path,arg1,arg2,arg3,local_data_id,rrd_num,snmp_port,snmp_timeout FROM poller_item WHERE host_id=99 ORDER BY arg1'
CACTID: DEBUG: MySQL Query ID '24': OK
CACTID: Host[99] DS[6305] SNMP: v1: 192.168.117.8, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.1, value: 1629739241
CACTID: Host[99] DS[6314] SNMP: v1: 192.168.117.8, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.10, value: 1657253067
CACTID: Host[99] DS[6315] SNMP: v1: 192.168.117.8, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.11, value: 0
CACTID: Host[99] DS[6316] SNMP: v1: 192.168.117.8, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.12, value: 1753378360
CACTID: Host[99] DS[6306] SNMP: v1: 192.168.117.8, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.2, value: 2100447460
CACTID: Host[99] DS[6307] SNMP: v1: 192.168.117.8, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.3, value: 0
CACTID: Host[99] DS[6311] SNMP: v1: 192.168.117.8, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.7, value: 0
CACTID: Host[99] DS[6312] SNMP: v1: 192.168.117.8, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.8, value: 1823717374
CACTID: Host[99] DS[6313] SNMP: v1: 192.168.117.8, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.9, value: 722534647
CACTID: Host[99] DS[6844] SNMP: v1: 192.168.117.8, dsname: discards_in, oid: .1.3.6.1.2.1.2.2.1.13.1, value: 0
CACTID: Host[99] DS[6853] SNMP: v1: 192.168.117.8, dsname: discards_in, oid: .1.3.6.1.2.1.2.2.1.13.10, value: 0
CACTID: Host[99] DS[6854] SNMP: v1: 192.168.117.8, dsname: discards_in, oid: .1.3.6.1.2.1.2.2.1.13.11, value: 0
CACTID: Host[99] DS[6855] WARNING: Result from SNMP not valid. Partial Result: ...
CACTID: Host[99] DS[6855] SNMP: v1: 192.168.117.8, dsname: discards_in, oid: .1.3.6.1.2.1.2.2.1.13.12, value: U
CACTID: Host[99] DS[6845] SNMP: v1: 192.168.117.8, dsname: discards_in, oid: .1.3.6.1.2.1.2.2.1.13.2, value: 0
CACTID: Host[99] DS[6846] SNMP: v1: 192.168.117.8, dsname: discards_in, oid: .1.3.6.1.2.1.2.2.1.13.3, value: 0
CACTID: Host[99] DS[6850] SNMP: v1: 192.168.117.8, dsname: discards_in, oid: .1.3.6.1.2.1.2.2.1.13.7, value: 0
CACTID: Host[99] DS[6851] SNMP: v1: 192.168.117.8, dsname: discards_in, oid: .1.3.6.1.2.1.2.2.1.13.8, value: 0
CACTID: Host[99] DS[6852] SNMP: v1: 192.168.117.8, dsname: discards_in, oid: .1.3.6.1.2.1.2.2.1.13.9, value: 0
CACTID: Host[99] DS[6844] SNMP: v1: 192.168.117.8, dsname: errors_in, oid: .1.3.6.1.2.1.2.2.1.14.1, value: 138044
CACTID: Host[99] DS[6853] SNMP: v1: 192.168.117.8, dsname: errors_in, oid: .1.3.6.1.2.1.2.2.1.14.10, value: 0
CACTID: Host[99] DS[6854] SNMP: v1: 192.168.117.8, dsname: errors_in, oid: .1.3.6.1.2.1.2.2.1.14.11, value: 0
CACTID: Host[99] DS[6855] WARNING: Result from SNMP not valid. Partial Result: ...
CACTID: Host[99] DS[6855] SNMP: v1: 192.168.117.8, dsname: errors_in, oid: .1.3.6.1.2.1.2.2.1.14.12, value: U
CACTID: Host[99] DS[6845] SNMP: v1: 192.168.117.8, dsname: errors_in, oid: .1.3.6.1.2.1.2.2.1.14.2, value: 769516
CACTID: Host[99] DS[6846] SNMP: v1: 192.168.117.8, dsname: errors_in, oid: .1.3.6.1.2.1.2.2.1.14.3, value: 0
CACTID: Host[99] DS[6850] SNMP: v1: 192.168.117.8, dsname: errors_in, oid: .1.3.6.1.2.1.2.2.1.14.7, value: 0
CACTID: Host[99] DS[6851] SNMP: v1: 192.168.117.8, dsname: errors_in, oid: .1.3.6.1.2.1.2.2.1.14.8, value: 1723
CACTID: Host[99] DS[6852] SNMP: v1: 192.168.117.8, dsname: errors_in, oid: .1.3.6.1.2.1.2.2.1.14.9, value: 0
CACTID: Host[99] DS[6305] SNMP: v1: 192.168.117.8, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.1, value: 1449680431
CACTID: Host[99] DS[6314] SNMP: v1: 192.168.117.8, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.10, value: 1413164613
CACTID: Host[99] DS[6315] SNMP: v1: 192.168.117.8, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.11, value: 0
CACTID: Host[99] DS[6316] SNMP: v1: 192.168.117.8, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.12, value: 1776628545
CACTID: Host[99] DS[6306] SNMP: v1: 192.168.117.8, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.2, value: 577274365
CACTID: Host[99] DS[6307] SNMP: v1: 192.168.117.8, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.3, value: 0
CACTID: Host[99] DS[6311] SNMP: v1: 192.168.117.8, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.7, value: 0
CACTID: Host[99] DS[6312] SNMP: v1: 192.168.117.8, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.8, value: 1840258850
CACTID: Host[99] DS[6313] SNMP: v1: 192.168.117.8, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.9, value: 3042701416
CACTID: Host[99] DS[6844] SNMP: v1: 192.168.117.8, dsname: discards_out, oid: .1.3.6.1.2.1.2.2.1.19.1, value: 52453
CACTID: Host[99] DS[6853] SNMP: v1: 192.168.117.8, dsname: discards_out, oid: .1.3.6.1.2.1.2.2.1.19.10, value: 0
CACTID: Host[99] DS[6854] SNMP: v1: 192.168.117.8, dsname: discards_out, oid: .1.3.6.1.2.1.2.2.1.19.11, value: 0
CACTID: Host[99] DS[6855] WARNING: Result from SNMP not valid. Partial Result: ...
CACTID: Host[99] DS[6855] SNMP: v1: 192.168.117.8, dsname: discards_out, oid: .1.3.6.1.2.1.2.2.1.19.12, value: U
CACTID: Host[99] DS[6845] SNMP: v1: 192.168.117.8, dsname: discards_out, oid: .1.3.6.1.2.1.2.2.1.19.2, value: 263709
CACTID: Host[99] DS[6846] SNMP: v1: 192.168.117.8, dsname: discards_out, oid: .1.3.6.1.2.1.2.2.1.19.3, value: 0
CACTID: Host[99] DS[6850] SNMP: v1: 192.168.117.8, dsname: discards_out, oid: .1.3.6.1.2.1.2.2.1.19.7, value: 0
CACTID: Host[99] DS[6851] SNMP: v1: 192.168.117.8, dsname: discards_out, oid: .1.3.6.1.2.1.2.2.1.19.8, value: 3601
CACTID: Host[99] DS[6852] SNMP: v1: 192.168.117.8, dsname: discards_out, oid: .1.3.6.1.2.1.2.2.1.19.9, value: 0
CACTID: Host[99] DS[6844] SNMP: v1: 192.168.117.8, dsname: errors_out, oid: .1.3.6.1.2.1.2.2.1.20.1, value: 0
CACTID: Host[99] DS[6853] SNMP: v1: 192.168.117.8, dsname: errors_out, oid: .1.3.6.1.2.1.2.2.1.20.10, value: 0
CACTID: Host[99] DS[6854] SNMP: v1: 192.168.117.8, dsname: errors_out, oid: .1.3.6.1.2.1.2.2.1.20.11, value: 0
CACTID: Host[99] DS[6855] WARNING: Result from SNMP not valid. Partial Result: ...
CACTID: Host[99] DS[6855] SNMP: v1: 192.168.117.8, dsname: errors_out, oid: .1.3.6.1.2.1.2.2.1.20.12, value: U
CACTID: Host[99] DS[6845] SNMP: v1: 192.168.117.8, dsname: errors_out, oid: .1.3.6.1.2.1.2.2.1.20.2, value: 1
CACTID: Host[99] DS[6846] SNMP: v1: 192.168.117.8, dsname: errors_out, oid: .1.3.6.1.2.1.2.2.1.20.3, value: 0
CACTID: Host[99] DS[6850] SNMP: v1: 192.168.117.8, dsname: errors_out, oid: .1.3.6.1.2.1.2.2.1.20.7, value: 0
CACTID: Host[99] DS[6851] SNMP: v1: 192.168.117.8, dsname: errors_out, oid: .1.3.6.1.2.1.2.2.1.20.8, value: 0
CACTID: Host[99] DS[6852] SNMP: v1: 192.168.117.8, dsname: errors_out, oid: .1.3.6.1.2.1.2.2.1.20.9, value: 0
CACTID: Host[99] DS[6202] SNMP: v1: 192.168.117.8, dsname: 5min_cpu, oid: .1.3.6.1.4.1.9.9.109.1.1.1.1.5.1, value: 2
CACTID: Host[99] DS[6200] SNMP: v1: 192.168.117.8, dsname: ProcMem, oid: .1.3.6.1.4.1.9.9.48.1.1.1.5.1, value: 5029916
CACTID: Host[99] DS[6207] SNMP: v1: 192.168.117.8, dsname: cisco_memused, oid: .1.3.6.1.4.1.9.9.48.1.1.1.5.1, value: 5030100
CACTID: Host[99] DS[6206] SNMP: v1: 192.168.117.8, dsname: cisco_memfree, oid: .1.3.6.1.4.1.9.9.48.1.1.1.6.1, value: 5448728
CACTID: Host[99] DS[6203] WARNING: Result from SNMP not valid. Partial Result: ...
CACTID: Host[99] DS[6203] SNMP: v1: 192.168.117.8, dsname: 5sec_cpu, oid: 1.3.6.1.4.1.9.9.109.1.1.1.1.3, value: U
CACTID: Host[99] DS[6201] WARNING: Result from SNMP not valid. Partial Result: ...
CACTID: Host[99] DS[6201] SNMP: v1: 192.168.117.8, dsname: 1min_cpu, oid: 1.3.6.1.4.1.9.9.109.1.1.1.1.4, value: U
CACTID: Host[99] DS[6205] SNMP: v1: 192.168.117.8, dsname: cisco_procmem, oid: 1.3.6.1.4.1.9.9.48.1.1.1.6.1, value: 5456932
CACTID: Host[99] DS[6204] SNMP: v1: 192.168.117.8, dsname: cisco_iomem, oid: 1.3.6.1.4.1.9.9.48.1.1.1.6.2, value: 2865712
CACTID: DEBUG: MySQL Insert ID '2': 'INSERT INTO poller_output (local_data_id,rrd_name,time,output) VALUE (6305,'traffic_in','2006-12-09 08:22:25','1629739241'),(6314,'traffic_in','2006-12-09 08:22:25','1657253067'),(6315,'traffic_in','2006-12-09 08:22:25','0'),(6316,'traffic_in','2006-12-09 08:22:25','1753378360'),(6306,'traffic_in','2006-12-09 08:22:25','2100447460'),(6307,'traffic_in','2006-12-09 08:22:25','0'),(6311,'traffic_in','2006-12-09 08:22:25','0'),(6312,'traffic_in','2006-12-09 08:22:25','1823717374'),(6313,'traffic_in','2006-12-09 08:22:25','722534647'),(6844,'discards_in','2006-12-09 08:22:25','0'),(6853,'discards_in','2006-12-09 08:22:25','0'),(6854,'discards_in','2006-12-09 08:22:25','0'),(6855,'discards_in','2006-12-09 08:22:25','U'),(6845,'discards_in','2006-12-09 08:22:25','0'),(6846,'discards_in','2006-12-09 08:22:25','0'),(6850,'discards_in','2006-12-09 08:22:25','0'),(6851,'discards_in','2006-12-09 08:22:25','0'),(6852,'discards_in','2006-12-09 08:22:25','0'),(6844,'errors_in','2006-12-09 08:22:25','138044'),(6853,'errors_in','2006-12-09 08:22:25','0'),(6854,'errors_in','2006-12-09 08:22:25','0'),(6855,'errors_in','2006-12-09 08:22:25','U'),(6845,'errors_in','2006-12-09 08:22:25','769516'),(6846,'errors_in','2006-12-09 08:22:25','0'),(6850,'errors_in','2006-12-09 08:22:25','0'),(6851,'errors_in','2006-12-09 08:22:25','1723'),(6852,'errors_in','2006-12-09 08:22:25','0')'
CACTID: DEBUG: MySQL Insert ID '2': OK
CACTID: DEBUG: MySQL Insert ID '3': 'INSERT INTO poller_output (local_data_id,rrd_name,time,output) VALUE (6305,'traffic_out','2006-12-09 08:22:25','1449680431'),(6314,'traffic_out','2006-12-09 08:22:25','1413164613'),(6315,'traffic_out','2006-12-09 08:22:25','0'),(6316,'traffic_out','2006-12-09 08:22:25','1776628545'),(6306,'traffic_out','2006-12-09 08:22:25','577274365'),(6307,'traffic_out','2006-12-09 08:22:25','0'),(6311,'traffic_out','2006-12-09 08:22:25','0'),(6312,'traffic_out','2006-12-09 08:22:25','1840258850'),(6313,'traffic_out','2006-12-09 08:22:25','3042701416'),(6844,'discards_out','2006-12-09 08:22:25','52453'),(6853,'discards_out','2006-12-09 08:22:25','0'),(6854,'discards_out','2006-12-09 08:22:25','0'),(6855,'discards_out','2006-12-09 08:22:25','U'),(6845,'discards_out','2006-12-09 08:22:25','263709'),(6846,'discards_out','2006-12-09 08:22:25','0'),(6850,'discards_out','2006-12-09 08:22:25','0'),(6851,'discards_out','2006-12-09 08:22:25','3601'),(6852,'discards_out','2006-12-09 08:22:25','0'),(6844,'errors_out','2006-12-09 08:22:25','0'),(6853,'errors_out','2006-12-09 08:22:25','0'),(6854,'errors_out','2006-12-09 08:22:25','0'),(6855,'errors_out','2006-12-09 08:22:25','U'),(6845,'errors_out','2006-12-09 08:22:25','1'),(6846,'errors_out','2006-12-09 08:22:25','0'),(6850,'errors_out','2006-12-09 08:22:25','0'),(6851,'errors_out','2006-12-09 08:22:25','0')'
CACTID: DEBUG: MySQL Insert ID '3': OK
CACTID: DEBUG: MySQL Insert ID '4': 'INSERT INTO poller_output (local_data_id,rrd_name,time,output) VALUE (6852,'errors_out','2006-12-09 08:22:25','0'),(6202,'5min_cpu','2006-12-09 08:22:25','2'),(6200,'ProcMem','2006-12-09 08:22:25','5029916'),(6207,'cisco_memused','2006-12-09 08:22:25','5030100'),(6206,'cisco_memfree','2006-12-09 08:22:25','5448728'),(6203,'5sec_cpu','2006-12-09 08:22:25','U'),(6201,'1min_cpu','2006-12-09 08:22:25','U'),(6205,'cisco_procmem','2006-12-09 08:22:25','5456932'),(6204,'cisco_iomem','2006-12-09 08:22:25','2865712')'
CACTID: DEBUG: MySQL Insert ID '4': OK
CACTID: Host[99] DEBUG: HOST COMPLETE: About to Exit Host Polling Thread Function
CACTID: DEBUG: The Value of Active Threads is 0
CACTID: DEBUG: MySQL Insert ID '5': 'replace into settings (name,value) values ('date',NOW())'
CACTID: DEBUG: MySQL Insert ID '5': OK
CACTID: DEBUG: MySQL Insert ID '6': 'insert into poller_time (poller_id, start_time, end_time) values (0, NOW(), NOW())'
CACTID: DEBUG: MySQL Insert ID '6': OK
CACTID: DEBUG: Thread Cleanup Complete
CACTID: DEBUG: PHP Script Server Pipes Closed
CACTID: DEBUG: Allocated Variable Memory Freed
CACTID: CACTID: Net-SNMP API Shutdown Completed
CACTID: DEBUG: MYSQL Free & Close Completed
CACTID: Time: 0.9250 s, Threads: 4, Hosts: 2

lungs71 · Post by **lungs71** » Sat Dec 09, 2006 9:09 am

Not sure if this is related but in an attempt to see if we can find the problem before we have to reload the server, Cacti monitoring has become fairly critical to us and the test/dev box can

lungs71 · Post by **lungs71** » Sat Dec 09, 2006 9:09 am

Not sure if this is related but in an attempt to see if we can find the problem before we have to reload the server, Cacti monitoring has become fairly critical to us and the test/dev box can

lungs71 · Post by **lungs71** » Sat Dec 09, 2006 9:12 am

Not sure if this is related but in an attempt to see if we can find the problem before we have to reload the server, Cacti monitoring has become fairly critical to us and the test/dev box can

lungs71 · Post by **lungs71** » Sat Dec 09, 2006 9:16 am

Not sure if this is realted but in an attempt to see if we can find the problem before we reload the server, Cacti has become fairly critical to us and the test/dev box can not support the load of all production. I installed MySQL Admin and we noticed the poller_output table has 9.4M rows in it, with a data free of 0, and it is highlighted like it is in error. I am not a DB guy so this could be in line, just thought I would put it out there.....

lungs71 · Post by **lungs71** » Sat Dec 09, 2006 9:18 am

Another thing we see is that MySQL is taking more than a normal about of processor.

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2375 mysql 16 0 132m 27m 2100 S 95.3 0.8 645:05.57 mysqld
15195 apache 15 0 785m 745m 1324 S 2.3 21.0 0:09.31 php
14833 apache 15 0 796m 669m 1324 S 1.7 18.9 0:09.60 php

Post by **gandalf** » Sat Dec 09, 2006 11:13 am

I suspected that from your output. The poller_output table is filled with each run. Then, the poller takes the data out of it to feed the rrd files. For this action it needs memory depending on the amount of entries in poller_output. Not enough memory = poller_output does not get emptied and fills up.
Change php.ini memory settings to some resonable size. Default is 8MB, 64 MB is good for normal operation. You may need more to get those data out. And, perhaps you need to increase php runtime (suppose 30 sec is default). You may need a HUGE value. Personally, I've never dealt with that much.
You may even "truncate table poller_output;" from mysql cli, but then, all data is gone
Even this is part of my "NaN Howto" ...

Reinhard

lungs71 · Post by **lungs71** » Sat Dec 09, 2006 6:57 pm

HI Reinhard - Well I tried but it did not help. I guess we will be rebuilding, I need to get back on line. I wish I knew what happened, oh well I guess it is a good opportunity to move from 0.8.6h to 0.8.6i. So you think we should try to pull the data from the rras or could this just migrate the problem? I'm not even sure if we can do it but it would be nice to have the history.

Thanks!!!

Post by **gandalf** » Sun Dec 10, 2006 11:57 am

You answer does not cope with all aspects I've suggested. At least, truncating the table and increasing php.ini memory should work. Then, a new cactid run would help me helping you
Reinhard

lungs71 · Post by **lungs71** » Sun Dec 10, 2006 2:36 pm

I'm sorry. I did attempt all solutions suggested and monitored, however they did not resolve the issue. I have re-built everything and so far everthing is ok.

rrd's not getting updated

rrd's not getting updated

Who is online