Issue with weathermap and boost.

[Howie]

OK - but you didn't actually clear the weathermap_data table in the end, right?

[jmarshall28@gmail.com]

I did not. I hate to ask but how is that done?

[Howie]

You would do it from phpMyAdmin, or the mysql command-line tool.

In the command-line tool it would be "delete from weathermap_data". In phpMyAdmin there is a 'truncate' button, if I remember correctly.

I will need to go away and read the code I think. I would have expected the problem to continue after you disabled the use_poller_output, so now I'm not sure if I remember how it works inside

[jmarshall28@gmail.com]

I have deleted the weathermap data table contents. weathermap has since refilled the table.

Some specs. I disabled all maps but one. when weathermap runs without the set poller_output command and with boost disabled in cacti the wethermap run takes one second. When I renable boost within cacti and leave out the set poller_output command the weathermap run still only takes one second, but then the map is blank. When I add the set poller output command to the one active weathermap the run time increases from 1 second to 79 seconds.

When I vary the maps, say activate only one map that has one target per weathermap link, the run time increases to 15 second. But if I pick a weathermp that has lots of link target aggregation the run time for that map is 79 seconds. These times are with boost enabled and the set poll_output command in the weathermap config.

I have also noticed that if all maps are enabled and you add the set poller_output directive in only one map with boost enabled you get unintended results. Some weathermap runs you get the expected behavior of all maps but one being all zeros. The next run the maps are poplulated with non-zero utlization data. If you add the set poller_output directive to one other weathermap so a total of two out of 19 have the poller_output directive, then all maps are evaluated as if the directive is set for all of them? Is this a consequence of map groups?

[Howie]

No, data is read only from rrd files if rrd_use_poller_output is not set. It's perfectly possible to 'get lucky' and read valid data from the rrd files since Boost has just updated them though. You would also get valid data for targets where the graph is one that you look at, since looking at the graph will force boost to update that rrd file, too.

This definitely sounds like a index/sql related issue. I should have some time at the weekend to try it further.

[jmarshall28@gmail.com]

any luck with this?

[Howie]

I have two things to try.

1) Try running 2 cycles with DEBUG logging turned on. This will log a lot more information about what Weathermap is doing for each target. Specifically I'm interested in if it is 'missing' existing data, and trying to add new lines in weathermap_data each time through (which is why I said 2 cycles). It should also give some idea of where the big delay is.

2) Disable some of the extra queries made in the rrd plugin - this will stop things like cacti_graph_id being available in maps, but does reduce the number of queries per target. Look for the if statement that calls "UpdateCactiData" around line 180 in lib/datasources/WeatherMapDataSource_rrd.php and either remove it, or comment it out.

[Howie]

The next thing after that would probably be to try and get a copy of your Cacti database, and see if I can reproduce the problem here.

[jmarshall28@gmail.com]

Howie

I commented the line out in the php file and my runtimes are now

Last Completed Run: Mon, 12 Jul 2010 14:55:19 EDT: 19 maps were run in 7 seconds with 0 warnings.

Do you still need the log data?

[Howie]

Howie

I commented the line out in the php file and my runtimes are now

Last Completed Run: Mon, 12 Jul 2010 14:55:19 EDT: 19 maps were run in 7 seconds with 0 warnings.

Do you still need the log data?

I guess not, for now Let's see if it stays low.

I'll take a look at that part in the meantime, and try to figure out why it would kill your system so badly.

[jmarshall28@gmail.com]

still going strong!

Last Completed Run: Wed, 14 Jul 2010 09:38:18 EDT: 19 maps were run in 7 seconds with 0 warnings.

what do you want to try next?

[Howie]

OK, well in 0.98, there is a config-file option to disable that same code, so you wouldn't need to edit the code yourself. This particular feature (of adding in the extra info from the Cacti database) was already identified as a performance-impacting thing - I did a bunch of profiling and performance testing after 0.97a. So the next release is roughly twice as fast, and uses less memory.

As it happens, I'm just setting up a boost-based cacti 0.8.7g install for a customer at the moment, so I'll be able to do more testing with that before I hand it over. If you don't need those extra variables, then I think you are probably done. If you do need them, then I'll hopefully have some more answers in a week or so, once I have done some testing.

[jmarshall28@gmail.com]

Thanks for you help sir!

[Howie]

On my test VM, with a 200 target map, I have improved the performance a bit:

Code: Select all

07/15/2010 02:32:14 PM - WEATHERMAP: Poller[0] STATS: Weathermap 0.98dev run complete - Thu, 15 Jul 2010 14:32:14 BST: 2 maps were run in 60.164408 seconds with 1 warnings
07/15/2010 02:37:08 PM - WEATHERMAP: Poller[0] STATS: Weathermap 0.98dev run complete - Thu, 15 Jul 2010 14:37:08 BST: 2 maps were run in 55.540364 seconds with 1 warnings
07/15/2010 02:41:31 PM - WEATHERMAP: Poller[0] STATS: Weathermap 0.98dev run complete - Thu, 15 Jul 2010 14:41:31 BST: 2 maps were run in 15.023012 seconds with 1 warnings
07/15/2010 02:46:28 PM - WEATHERMAP: Poller[0] STATS: Weathermap 0.98dev run complete - Thu, 15 Jul 2010 14:46:28 BST: 2 maps were run in 14.074934 seconds with 1 warnings

So that's a 400% improvement without disabling any functionality The part we disabled before is still relatively slow though, especially with 1-minute polling limiting the maximum run time.

[jmarshall28@gmail.com]

when is that going gold?