Great Remote Poller Performance of Cacti 1.2

[phalek]

Just a remark on the Poller Performance of remote running pollers ... It's great !

The attached screenshot shows some of my pollers. All based on Raspberry Pis running the apt-get install of Cacti 1.1.38 upgraded to 1.2 from the Master.

As a side note: Make sure to have the master and remote pollers be as identical as possible. I'm running a CentOS 7 system as Master which has been isntalled from source and the Raspberry Pis are on Debian with the apt-get version.

The special setup of the apt-get install introduces a lot of issues for remote pollers. One is the symlink of css files which seem to force the poller to permanently trying to update these files causing up to 20 seconds delay on very far sites (ping > 250 ms).

All polling on the remote systems are from the mURLin plugin (php script server based) which has been changed to always use the local database for caching.

[netniV]

Very nice! Speaking of the mURLin plugin, have you committed that yet?

[cigamit]

Nice. An interesting tidbit is some of my early testing on the remote poller was with Raspberry PIs, as it was easier to detect what changes actually had a performance impact.

[phalek]

Hm ... one additional idea would be to actually start the polling only if the database sync and any file syncing has been completed.

Especially on very remote sites the file updates take a very long time and new pollers are being spawned all the way down the road making the system unresponsive.

Maybe a switch to manually enable/disable the actual polling (but leave the DB/File sync on) on the remote poller.
I think the current Enable/Disable functionality is a "Disable All" or "Enable All" thing, right ?

My total is 47 Raspberry Pi 3b+ system all over the world, so I can do a lot of testing here

[netniV]

I think that would be wise, especially during upgrades. For example, a lot changes between 1.1.38 and 1.2, so syncing those changes means that not all files would be in alignment when a poller is kicked off if it's only got a 64k link (for want of the worst kind of example). I'm not too sure how the poller currently handles that situation, still learning all it's nuances.

[phalek]

For slow links it's quite easy: The poller is being called multiple times resulting in a complete system instability and unresponsiveness.

For upgrades, it's complicated. When I upgraed to the 1.2 Beta, most pollers just stopped working as the cli_check php file was missing. Then afterwards it was complaining about missing features.

Although the database-based syncing is a convenient way of getting the pollers in sync with the master, it does have it's downsides.

A customer of mine managed to make the master fail by adding a 1GB test file for I/O measurement to the cacti main directory. Cacti tried to add that to the database and as a result the mysql connection constantly crashed (max_packet_size ... ).


Right now I have a LOCAL remote poller which is taking minutes to start as it tries to put the files from the DB to the local file system. Still trying to find the root cause of this, but if I just could disable this and enable the syncing one-by-one for each poller, that would make the whole setup a lot more stable. ( i.e. only sync if something has changed, and "something has changed" is defined by me and not a md5 hash .. )

INFO: I'm not filing a bug, as all these issues may just come from running a Centos (from source install) Master with Debian (apt-get installed) based remote pollers ...

UPDATE: I've now added a new global swithc "Enable Remote File Sync" to the poller tab and added some code to the resource_cache_out which check that settings and prevents the system to go through the resouce distribution. Based on that it looks like the "md5sum_path" function is actually causing the very long delay. need further checking on this ...

[netniV]

I think a size limit should certainly be in place, extremely large files hitting the database/syncing down to the pollers are not ideal. However, that might affect sending RRA's around.

[phalek]

I think a size limit should certainly be in place, extremely large files hitting the database/syncing down to the pollers are not ideal. However, that might affect sending RRA's around.

-> The size could be checked against the max_package setting of mysql, as anything bigger then that will cause the mysql connection to fail anyway ... ?

[netniV]

I would agree with that, with a warning about the issue in cacti.log.