Boost: filling 'poller_output' and 'poller_output_boost'

[dawald]

Hi there,

first, as already mentioned, this plugin r0cks:

Code: Select all

05/09/2007 05:47:54 PM - SYSTEM STATS: Time:52.4289 Method:cactid Processes:1 Threads:15 Hosts:252 HostsPerProcess:252 DataSources:51569 RRDsProcessed:0
05/09/2007 05:57:10 PM - SYSTEM BOOST STATS: Time:556.2389 RRDUpdates:658063

But there is one question: when I'm looking at my mysql tables 'poller_output' and 'poller_output_boost' I see them growing all the time. Do we still need the table 'poller_output' or is all the polled data written directly to the 'poller_output_boost' table ?

There are a lot of old records that are not getting deleted. But I can't see entries for every polling cycle, if I'm sorting the records in the 'poller_output' by the column 'time'. Sometimes there are more than two hours without any entries. For me it looks like some SQL delete commands don't get executed properly.

Does anybody know how to deal with it ? After 48 hours of polling (576 polling cycles) 51569 datasources my 'poller_output' table contains 5756 old 'zombie-records'. So in average there are around ten records per polling cycle that don't get deleted, in the tables 'poller_output' and 'poller_output_boost'

Thanks for any advice.... I'm using cacti 0.8.6j with the latest patches, PIA 1.1 and BoostPlugin 1.2 on a UNIX maschine.

David

[EDIT]: maybe this post should get moved into the "Boost Plugin v1.2 Released" thread....

[TheWitness]

Edit <path_cacti>/poller.php and search for the word "tuncate". Add the following right after "db_execute("truncate table poller_time");"

db_execute("truncate table poller_output");

This is a required change and a bug introduced in Cacti 0.8.6j. However, it is making an issue with some of your data sources.

TheWitness

[dawald]

Thanks, TheWitness

I added the the line to my poller.php file.

David

[TheWitness]

David,

If you want to track when this occurs, look at the code I added to the Main Boost 1.2 post and then provide feedback here or there. I will get both. I would be interested to know if this is systemic and how large the problem is and if it's, for you anyway, damaged data source or just some slop that was left over in the table from some debugging exercise.

Thanks,

Larry

[dawald]

TheWitness,

thanks, I replaced the line

Code: Select all

db_execute("truncate table poller_output");

with the following code like documented in post http://forums.cacti.net/viewtopic.php?t=18430&start=75

Code: Select all

$rows = db_fetch_cell("SELECT count(*) FROM poller_output");

if ($rows > 0) {
   cacti_log("WRANING: Poller found partially populated poller_output table.", "POLLER");
   db_execute("truncate table poller_output");
}

Sometimes there are three days without any new 'zombie-lines' in my poller_output table. It is mostly the same data template that generates such problems. In my case it is the "Brocade Fibre Channel template (http://forums.cacti.net/viewtopic.php?t=6346)".

Now that I'm logging the 'errors' I can keep track when it occures. I will check what kind of datasources are causing problems. I let you know as soon as there are any results.

Thanks .. David

BTW: There is a small typo in your line, WRANING -> WARNING

[dawald]

The Witness,

out of 154 poller cycles, 30 contained the WARNING...
I disabled the line with the truncate on the table poller_output in the script to see what datasources are causing the problem. Maybe the logfile in debug mode will show me what exactly goes wrong.

David

[TheWitness]

I would definately like to get details around the problem data sources including taking a closer look at the scripts/snmp etc. generating them. More information on your environment will also be helpful. Like what poller you are running, etc.

TheWitness

[tekbot]

I had a similar problem with boost in a few of my environments. In the 10 second polling environment (see http://forums.cacti.net/about16482-90.html Page7) my graphs were all updating happily after the transition to boost for about 2 weeks. Then, early this morning, all my graphs just stopped updating. There was nothing informative being written to the debug log, and when I redirected the cacti_rrdsvc daemon to a logfile, *nothing* was being written to it (only the daemon Start/Stop Commands I issued). I found a few interesting things while troubleshooting it.

1) All the data from the polls was being written to the poller_output table (not the poller_output_boost table).
2) RRD On-Demand updating worked when I ran:

Code: Select all

 insert into poller_output_boost (select * from poller_output); 

however, the results from my polls were still not ever making it into the poller_output_boost table.
3) After truncating the poller_output table, boost was back in business and my graphs began updating again (with a short gap from the SQL insert statement to the truncate statement -- just a few minutes).

This makes me curious about how this plugin works. It would appear that with or without boost, the poller inserts into poller_output. So, a few questions on the theory of operation for future troubleshooting:

When does the poller_output_boost table get populated?
When poller_output_boost is populated from poller_output, is a delete or truncate statement on poller_output run?
Is there a safeguard on the query the inserts into poller_output_boost and deletes/truncates from poller_output to ensure data is never lost during the transition?
Is there a max table size to poller_output?

The fix for this issue, for me, was the truncate table command (and poller.php edit). This brings up the more important question: Is there any adverse effect of the truncate table addition to poller.php? I want to make sure that my other environments that use boost do not encounter this issue, but before that, I want to make sure that this won't cause some other issue I haven't thought of.

I hope that this fix helps some of you whose graphs have stopped updating. Before you do anything drastic, check the poller_output table -- If you're not seeing critical errors in the logs, your poller is probably still running and gathering data. You just need to move it somewhere where rrdtool / rrdupdate can get to it. I also hope that someone can shed some light on the questions I posted above.

[TheWitness]

My post above is the correct answer. It sounds like you may have had an issue as described above where you simply had too many "bad" records.

The log message above will indicate as such. You may have to review the contents of this table, in other words a better log message and research the odd data sources producing the errors.

TheWitness

[TheWitness]

Boost 2.0 is official. Please Check it Out Here:

http://forums.cacti.net/viewtopic.php?p=150337#150337

Regards,

TheWitness