[SOLVED] WARNING: Poller Output Table not Empty

Atkatla · Post by **Atkatla** » Fri Mar 13, 2009 5:37 am

TheWitness wrote:Well, that was not what you had shown earlier.

nah, you confused me with cdnvballer.

I have a my.cnf, but it did not contain a line for max connections. I put

set-variable=max_connections=500

into [mysqld] and restarted mysql. Lets see if that helps. Reduced the number of threads from 40 to 20.

*later*
Didn't help:

Code: Select all

03/13/2009 12:13:43 PM - SYSTEM STATS: Time:11.9750 Method:spine Processes:2 Threads:20 Hosts:74 HostsPerProcess:37 DataSources:20074 RRDsProcessed:0 
03/13/2009 12:12:39 PM - SYSTEM STATS: Time:12.2468 Method:spine Processes:2 Threads:20 Hosts:74 HostsPerProcess:37 DataSources:20074 RRDsProcessed:0 
03/13/2009 12:11:37 PM - SYSTEM STATS: Time:12.4044 Method:spine Processes:2 Threads:20 Hosts:74 HostsPerProcess:37 DataSources:20074 RRDsProcessed:0 
03/13/2009 12:11:24 PM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 30265, Data Sources: mem_buffers(DS[2901]), (DS[2903]), proc(DS[2905]), (DS[2906]), (DS[2907]), hdd_free(DS[2910]), hdd_used(DS[2910]), hdd_free(DS[2911]), hdd_used(DS[2911]), 5sec(DS[2912]), 60sec(DS[2912]), 5sec(DS[2913]), 60sec(DS[2913]), 5sec(DS[2914]), 60sec(DS[2914]), 5sec(DS[2915]), 60sec(DS[2915]), 5sec(DS[2916]), 60sec(DS[2916]), 5sec(DS[2917]), 60sec(DS[2917]), Additional Issues Remain. Only showing first 20 
03/13/2009 12:11:01 PM - SYSTEM STATS: Time:0.3213 Method:spine Processes:2 Threads:20 Hosts:74 HostsPerProcess:37 DataSources:30265 RRDsProcessed:0 
03/13/2009 12:11:01 PM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting. 
03/13/2009 12:10:16 PM - SYSTEM STATS: Time:12.0629 Method:spine Processes:2 Threads:20 Hosts:74 HostsPerProcess:37 DataSources:20074 RRDsProcessed:0 
03/13/2009 12:09:06 PM - SYSTEM STATS: Time:17.7299 Method:spine Processes:2 Threads:20 Hosts:74 HostsPerProcess:37 DataSources:30265 RRDsProcessed:0 
03/13/2009 12:08:59 PM - SYSTEM BOOST STATS: Time:224.3655 RRDUpdates:126671 
03/13/2009 12:07:55 PM - SYSTEM STATS: Time:23.9394 Method:spine Processes:2 Threads:20 Hosts:74 HostsPerProcess:37 DataSources:30265 RRDsProcessed:0 
03/13/2009 12:06:30 PM - SYSTEM STATS: Time:26.0128 Method:spine Processes:2 Threads:20 Hosts:74 HostsPerProcess:37 DataSources:30265 RRDsProcessed:0 
03/13/2009 12:05:15 PM - SYSTEM STATS: Time:13.7799 Method:spine Processes:2 Threads:20 Hosts:74 HostsPerProcess:37 DataSources:30265 RRDsProcessed:0

Why does the poller runs into a timeout? And why does the DS number changes so drastcally? Cacti GUI says I have 24015.

Post by **TheWitness** » Fri Mar 13, 2009 7:34 pm

If your cron at 5 minutes and your poller interval at 1? If so, change both the cron and poller interval to 1 minute and see if the problem goes away. Otherwise I suspect a spine segfault. You need to make sure the poller use is authorized to write a core file and make sure that user has permissions to do so in the spine directory. Also, it would be nice if segfault's made it into your log, or root's mailbox.

TheWitness

Atkatla · Post by **Atkatla** » Mon Mar 16, 2009 7:58 am

I changed cron intervall to one minute in crontab and cacti settings. Result:

Code: Select all

03/16/2009 01:45:12 PM - SYSTEM STATS: Time:11.2962 Method:spine Processes:2 Threads:20 Hosts:74 HostsPerProcess:37 DataSources:25324 RRDsProcessed:0 
03/16/2009 01:45:11 PM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 8886, Data Sources: 5sec(DS[4001]), 60sec(DS[4001]), 5sec(DS[4002]), 60sec(DS[4002]), 5sec(DS[4003]), 60sec(DS[4003]), 5sec(DS[4004]), 60sec(DS[4004]), 5sec(DS[4005]), 60sec(DS[4005]), 5sec(DS[4006]), 60sec(DS[4006]), traffic_in(DS[4013]), traffic_out(DS[4013]), traffic_in(DS[4014]), traffic_out(DS[4014]), traffic_in(DS[4015]), traffic_out(DS[4015]), traffic_in(DS[4016]), traffic_out(DS[4016]), traffic_in(DS[4017]), Additional Issues Remain. Only showing first 20 
03/16/2009 01:45:00 PM - SYSTEM STATS: Time:58.8303 Method:spine Processes:2 Threads:20 Hosts:74 HostsPerProcess:37 DataSources:25336 RRDsProcessed:0 
03/16/2009 01:45:00 PM - POLLER: Poller[0] Maximum runtime of 58 seconds exceeded. Exiting. 
03/16/2009 01:44:08 PM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 13569, Data Sources: 5sec(DS[6660]), 60sec(DS[6660]), 5sec(DS[6661]), 60sec(DS[6661]), 5sec(DS[6662]), 60sec(DS[6662]), 5sec(DS[6663]), 60sec(DS[6663]), 5sec(DS[6664]), 60sec(DS[6664]), 5sec(DS[6665]), 60sec(DS[6665]), 5sec(DS[6666]), 60sec(DS[6666]), traffic_in(DS[6667]), traffic_out(DS[6667]), traffic_in(DS[6668]), traffic_out(DS[6668]), traffic_in(DS[6669]), traffic_out(DS[6669]), traffic_in(DS[6670]), Additional Issues Remain. Only showing first 20 
03/16/2009 01:43:10 PM - SYSTEM STATS: Time:9.1633 Method:spine Processes:2 Threads:20 Hosts:74 HostsPerProcess:37 DataSources:25343 RRDsProcessed:0 
03/16/2009 01:43:04 PM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 7218, Data Sources: 5sec(DS[6400]), 60sec(DS[6400]), 5sec(DS[6401]), 60sec(DS[6401]), traffic_in(DS[6402]), traffic_out(DS[6402]), traffic_in(DS[6403]), traffic_out(DS[6403]), traffic_in(DS[6404]), traffic_out(DS[6404]), traffic_in(DS[6405]), traffic_out(DS[6405]), traffic_in(DS[6406]), traffic_out(DS[6406]), traffic_in(DS[6407]), traffic_out(DS[6407]), traffic_in(DS[6408]), traffic_out(DS[6408]), traffic_in(DS[6409]), traffic_out(DS[6409]), traffic_in(DS[6410]), Additional Issues Remain. Only showing first 20

I have no errors concerning a segmentation fault in log file (log-level: low).
core files: I know that i can set the size with f.e. "ulimit -c 50000". But I dont know how to authorize a user to write core files. I only know it has something to do with the GNU debugger.

and make sure that user has permissions to do so in the spine directory

In /usr/local/spine or /usr/local/spine/bin/ ?

Post by **TheWitness** » Mon Mar 16, 2009 9:36 am

Grab a fresh copy of spine from svn.

Larry

Atkatla · Post by **Atkatla** » Tue Mar 17, 2009 4:08 am

Hmm, it seems that the errrors from my last post appeared only immediately after the poller restart after the change from 5min cron to 1minute. Before installing the SVN-spine I wanted to see if they stay but later they didnt occur. No POnE-Errors anymore. I made no other changes and didnt install the SVN yet. But sometimes a "Cron is out of sync with the Poller Interval! The Poller Interval is '60' seconds, with a maximum of a '300' second Cron, but 64 seconds have passed since the last poll!"
Does that indicate a too high CPU load? In late summer I will have to add 40% more data sources to reach the final expansion stage.

Do you want me to switch back to 5min cron and try the SVN spine to test the new version?

Post by **TheWitness** » Tue Mar 17, 2009 6:00 am

The cron and poller interval in the wb interface have to match reakity.

TheWitness

Atkatla · Post by **Atkatla** » Tue Mar 17, 2009 7:11 am

I think they already match. My crontabline:

*/1 * * * * cactiuser php /srv/www/htdocs/cacti/poller.php > /srv/www/htdocs/cacti/poller.log 2>&1

Cacti -> Sttings -> Poller:
Poller Interval: Every Minute
Cron Interval: every Minute

Post by **TheWitness** » Tue Mar 17, 2009 7:15 am

Did you repopulate your poller cache after making this change? You need to in order to distribute the sampling.

TheWitness

Atkatla · Post by **Atkatla** » Tue Mar 17, 2009 8:41 am

When I remember correctly I did it in the following order: disabled poller, changed cacti setting, changed crontab, made a reboot, rebuilded poller cache, enabled poller in GUI.

I did it again to be sure, (disable poller, rebuild cache, enable poller), but the "Cron is out of sync with the Poller Interval! The Poller Interval is '60' seconds, with a maximum of a '300' second Cron, but 64 seconds have passed since the last poll!" Message occured again periodically.

Post by **TheWitness** » Tue Mar 17, 2009 8:58 am

Maybe a vm or heavily loaded system? Anyway, using the -f option on poller.php will fix that.

Larry

Atkatla · Post by **Atkatla** » Wed Mar 18, 2009 3:15 am

Its not a VM, but 60-80% average load when the poller runs (about 15 seconds per minute) and 1-20% load during the other 45 seconds per minute. Hardware is a 2 GHz Dual-CoreXeon with 4GB RAM.
I did

Code: Select all

php -f [cactipath]/poller.php

but there are still sometimes these out-of-snyc-messages in the log from the last night.

Post by **TheWitness** » Wed Mar 18, 2009 7:18 am

Wrongo...

Code: Select all

*/1 * * * * php -q poller.php -f

What you did was the "-f" option of PHP and not poller.php.

TheWitness

Atkatla · Post by **Atkatla** » Thu Mar 19, 2009 9:39 am

vi /etc/crontab

Code: Select all

*/1 * * * * cactiuser php /srv/www/htdocs/cacti/poller.php --force > /srv/www/htdocs/cacti/poller.log 2>&1

But I got one of those out-of-syncs again. When I use -q, no results appear. I also have to use --force because -f is unknown:
# php -q poller.php -f

Code: Select all

ERROR: Invalid Argument: (-f) 
Cacti Poller Version 0.8.7d, Copyright 2007-2008 - The Cacti Group 
A simple command line utility to run the Cacti Poller. 
usage: poller.php [--force] [--debug|-d] 
Options: 
    --force        Override poller overrun detection and force a poller run 
    --debug|-d     Output debug information.  Similar to cacti's DEBUG logging level.

# php -q poller.php --force

Code: Select all

No log handling enabled - turning on stderr logging 
truncating unsigned value to 32 bits (2) 
truncating unsigned value to 32 bits (2) 
truncating unsigned value to 32 bits (2) 
[i]... [repeated about 70 times] [/i]
truncating unsigned value to 32 bits (2) 
PHP Warning:  key(): Passed variable is not an array or object in /srv/www/htdocs/cacti/plugins/weathermap/setup.php on line 457 
PHP Warning:  key(): Passed variable is not an array or object in /srv/www/htdocs/cacti/plugins/weathermap/setup.php on line 457 
[i][repeated about 50 times] [/i]
PHP Warning:  key(): Passed variable is not an array or object in /srv/www/htdocs/cacti/plugins/weathermap/setup.php on line 457

#457 in weathermap_setup.php:

Code: Select all

if( isset( $rrd_update_array{$file}['times'][key($rrd_update_array[$file]['times'])]{$dsname} ) )

aKeLa · Post by **aKeLa** » Fri Apr 24, 2009 2:08 pm

The final solution for this insue? Actualy i have this problem, but i don´t fix them. I prove all solutions... Help me please.

elgabito · Post by **elgabito** » Thu Apr 30, 2009 2:11 pm

This is marked as "SOLVED" - I have this issue and saw no resolution in this thread.

I am using 0.8.7d with patches installed - cmd.php

[SOLVED] WARNING: Poller Output Table not Empty

Who is online