Poller Stuck for Few Minutes on SQL Statements

[iczman]

I have a problem with poller waiting for a SQL statement to run and causing most of the delay in each polling cycle. The effect (I guess) of these delay is causing my script to be polled at irregular intervals. For example, the time between each script execution could vary anywhere from 1 to 10 minutes.

I'm not sure what other information I should provide to resolve this problem, but I'll attach a snippet of the debug lines from the log:

(using cacti 0.8.6j with all patch applied, using cmd.php)

Code: Select all

08/21/2007 06:02:20 AM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select poller_id,end_time from poller_time where poller_id = 0"
08/21/2007 06:02:20 AM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select  poller_output.output,  poller_output.time,  poller_output.local_data_id,  poller_item.rrd_path,  poller_item.\rrd_name,  poller_item.rrd_num  from (poller_output,poller_item)  where (poller_output.local_data_id=poller_item.local_data_id and poller_output.rrd_name=poller_item.rrd_name)  LI\MIT 10000"

... repeats for about a few (2+) minutes....

[iczman]

I found this website: http://bsod2600.home.comcast.net/install_windows.html
which suggests that if my log is filled with those lines from above, then my Cacti does not have debugging properly enabled (from the bottom of FAQ). I have to disagree because my log is also filled with other regular debugging information.

I'm no where near understanding why poller got stuck at those sql statements... TheWitness or gandalf, could you shed some light?

Thanks

[gandalf]

TheWitness or gandalf, could you shed some light?

I would have done so Iif I only could remember hwo to cure this. It WAS a well know issue some months ago. But I don't know anymore which spell to use.
Perhaps Larry knows?
Reinhard

[rony]

Do me a favor....

Run the following query when the poller is not running.

Code: Select all

select count(*) as total from poller_output;

The result should be 0 if the poller is not running. If it's is not, you have an issue.

1. Check the memory setting for php is set higher than 64 megs.
2. Run a check/repair on the poller_output table.
3. Make sure you are not running out of space on the volume where mysql tables are stored. Typically /var/lib/mysql.

[iczman]

select count(*) as total from poller_output;

I always get 0 every time I run the statement when poller isn't running.

1. Check the memory setting for php is set higher than 64 megs.

My php.in's memory_limit is already set at 128M. I also verified the setting with phpinfo().

2. Run a check/repair on the poller_output table.

Did check & repair on poller_output in phpmyadmin and both operations returned "OK".

3. Make sure you are not running out of space on the volume where mysql tables are stored. Typically /var/lib/mysql.

There are more than 130GB available on my disk.



I want to add some additional information. The poller usually takes a few seconds to execute those statement, but will occasionally (few time a day) take up to few minutes. I also reinstalled Cacti and nuked cacti database in mysql, but it still occurs once in a while.

[rony]

Um....

What is the system load like when this is occurring?

[iczman]

My system load is at minimal since it usually does nothing except running cacti poller.

I think I figured out my problem. One of my script was taking sweet time to run (300+ seconds) and poller was waiting for the result to come back. This also delayed my other scripts behind it and caused irregular execution intervals as I described earlier.

I will try to run the scripts asynchronously with crontab and see if the problem still exist.

Thanks for the help devs

[iczman]

Actually this brings up another question. Are there any plans to make Cacti poll each datasource in parallel, as opposed to running in serial as it is now? This could save some time to configure and modify custom scripts to run them asynchronously from cron.

I searched the forum and saw someone asked the same question, but it was from 2002 and I don't know if anything changed since then.