Cacti first maps calculation reaaaaaalllllly slow !

[ncfrak]

There are a few changes to do with using indexes in queries that might also make a difference when you have so many targets, relative to datasources (almost 100%, or are they duplicated?).

I have to admit that I didn't quite understant what you meant
The TARGETs are datasources, or am I confusing something?

Code: Select all

        TARGET 8*dsstats:1025:traffic_in:traffic_out
        NODES rtr1151 rtr1122:SW50
        VIA 834 590

[Howie]

data_local = 2287

grep TARGET configs/*.conf | wc -l = 2184

You have almost as many targets as you have data sources in Cacti. So either you have a map showing every switch port, or some ports appear on every map...

Because weathermap would be doing (at least) one query per target, any small changes that make better use of indexes in the database could make a big difference. There is also a new switch in 0.98 that disables some additional queries - by default weathermap will fill in some extra variables with information about the interface. If you don't use 'cacti_ifSpeed' etc in your map, you can add ""SET rrd_no_cacti_extras 1" for another performance increase (in 0.98).

[ncfrak]

Oh OK I understand.
Yes in fact right now we are graphing only the ports that are in the weathermaps.

We will add other graphs later but now we are focused on weathermap

[Howie]

Here we go:
http://www.network-weathermap.com/files ... 98dev1.zip

For anyone else who wants to try it: go ahead, it shouldn't break anything, but it's not finished. In particular the manual isn't updated for some new features, and there is a new Map Properties page on the management screen that doesn't do much yet. Bug reports are welcome!

Most importantly: I won't be helping you install this as your first Weathermap install. Stick with 0.97a please.

CHANGELOG (incomplete) is:

Code: Select all

0.98    FIXED - static datasource plugin honours KILO
        FIXED - WriteConfig (i.e. editor) won't 'lose' absolute keyword from USESCALE
        FIXED - memory leak in poller code. Memory usage is MUCH lower now.
        FIXED - NINK colours were exchanged (thanks Deathwing00!)
        FIXED - check-gdbug.php shouldn't complain about empty ob_flush buffers anymore
        FIXED - SNMP DS should deal better with non-numeric (and blank) return values
        FIXED - absolute SCALE definitions didn't support K (thanks wwwdrich)
        CHANGE - lots of performance optimisations. Roughly twice as fast as 0.97a. (thanks ikorzha758 for starting this process, and DLNoah, jmarshall28 and llow for some input)
        CHANGE - no longer shows .-prefixed files in map config picker
        CHANGE - editor ignores attempts to rename nodes to have space in names
        ADDED - New Cacti integration for editor. Additional 'edit maps' permission. No need for editor-config.php anymore!
        ADDED - per-map runtime logging
        ADDED - per-map debugging (debug a single map in Cacti without running debug logging for the whole poller)
                ADDED - Cacti template for monitoring Weathermap performance/mapcount/warningcount/memory.
        ADDED - when running from Cacti, HTMLOUTPUTFILE, IMAGEOUTPUTFILE and IMAGEURI can be used to create a second copy of the output.
                ADDED - KEYBGCOLOR and KEYOUTLINECOLOR both accept 'none'
                ADDED - Ability to disable any warning with 'SET nowarn_WMxxx 1'
        ADDED - USEICONSCALE works to colour artificial icons
        ADDED - performance options to disable additional information checks in rrd and dsstats data plugins.
        ADDED - cactihost and cactimonitor targets both return 5 for 'unknown' (according to Cacti) (thanks djnono17)
        ADDED - new DATAOUTPUTFILE allows collected data to be written to a file for later use (automatically on in Cacti)
        ADDED - new wmdata: datasource plugin can read data from files produced by DATAOUTPUTFILE

[Djoul2706]

THANK YOU !

the effect is immediate !
we just installed it and it works perfectly.
the first rendering was 15s when it took more than 26 during the previous cycles. (no more "first cycle effect" with dsstats.)

we need now to check logs and stuffs to give you a more complete feedback !

Julien

[Howie]

Aha. Good news

Also, I'd forgotten that you'll be able to tell which map is taking longer if it does slow down - run time is recorded for each map now. You can see that time in the Properties page for each map.

There are also Cacti templates for a couple of graphs in random-bits/ to let you monitor Weathermap run time and memory usage using Cacti itself - and potentially have a Thold alert too.

[Howie]

THANK YOU !

the effect is immediate !
we just installed it and it works perfectly.
the first rendering was 15s when it took more than 26 during the previous cycles. (no more "first cycle effect" with dsstats.)

we need now to check logs and stuffs to give you a more complete feedback !

Julien

Hi Guys - any more feedback after a few weeks?

I know of two problems with that dev release:
* edit config in the editor is broken (something to do with the new permissions stuff I added)
* Weathermap doesn't deal with the case where there are no pre/post (or I guess datasource) plugins. The zip accidentally had neither the pre or post dirs.

Any others?

[thomas315]

Hello,

I've had the same issue with v0.97a (long maps calculation), i've tested your dev release. So here's some feedback :

- Map calculation is ok and twice faster than with 0.97a.

Logs says things like that :
- Couldn't open %s Plugin directory (%s). Things will probably go wrong. [WMWARN06]
- About to write image file. If this is the last message in your log, increase memory_limit in php.ini [WMPOLL01]

But the process seems to work fine.

The editor is working properly, but the cacti integration does not seems to be ok : $cacti_base is not set correctly. Cacti integration will be disabled in the editor. You might need to copy editor-config.php-dist to editor-config.php and edit it.
Even by adding the editor-config.php


Thomas

[ncfrak]

Hello,

Some feedback about this version. I did not have much time recently (and went on vacations )! Sorry for the long delay.

2 things:

1. We also experienced the $cacti_base warning when clicking on the editor link on the weathermap tab.
If i open the editor from the link on the page where there are the config files i don't get the warning (all the way down)

2. When picking a data source with the "Pick from Cacti" button, I can choose the link but when I click on it the window never closes with the message "This window should disappear in a moment."
So we have to enter manually the Info URL and 'Hover' Graph URL.

Except from that it's working great! Great job

Do you have plan for another release?

[Howie]

Yes - I have a list of bugs to fix, and some missing features (since I re-wrote the drawing code a little, comments don't appear in the right place, for example).

When I get time (the magic ingredient!) I'll get 0.98 officially released, with updated manual etc, and a new website.