Distributed Cacti - Ideas

[reeshimanu]

We currently use a HA clustered mysql cacti solution, using:

- mysql 5.0 real-time clustering/replication
- GFS on a 14-disk raid5 array (HP/compaq msa500) with two servers SCSI-attached (dl380g3)
- custom poller that divides polling amongst active nodes
- load-balanced web front-end with cisco CSS switches (sessions are synchronized with a GFS share)
- all datasources are polled on 1-minute intervals, with 1-minute granularity kept for 60 days

Please let me know if anyone is interested in more detail...

ADesimone

I'd love to see your work on the custom pollers; including your code changes.

I'd like to do the same thing using several poller machines all hooked into a single SAN environment.

Reeshi

I am also looking for similar requirement but unable to do can you please send me the document to do that.
Thanks
Mail id - reeshimanu@yahoo.co.in

[Marcel]

Hi,

In my company's environment, have setup a memcached to hold all polled data, in which a patch on poller.php and cmd.php would boost the polling to be as fast as it can.

We have the servers themselves inputing data to memcached. Then the poller polls the data on memcached. It's fairly easy to setup and it boosts the polling as for now:

08/01/2008 09:55:36 PM - SYSTEM STATS: Time:35.2860 Method:cmd.php Processes:4 Threads:N/A Hosts:1018 HostsPerProcess:255 DataSources:33705 RRDsProcessed:0

Without boost I would've been lost months ago. There is almost 30k graphs. And above are the number of servers are inputing memcached with data. Then, the sysadmins of the monitored servers themselves are able to setup cronjobs to input data to memcache.

This approach is very machine-resource saving, as Boost plugin. We've distributed the poller load accross every server that should be graphed. And the snmp data is done by some administrative hosts that have snmp access to the network infrastructure.

[jerrison]

Marcel,

would you care to elaborate how you patched poller and cmd.php (not spine?) and how exactly you set up memcached to work for you?

i´m in the process of setting up almost the same amount of hosts/datasources like you have so any help is really appreciated!

thanks in advance,

andre

[mikaelf]

Hi,

In my company's environment, have setup a memcached to hold all polled data, in which a patch on poller.php and cmd.php would boost the polling to be as fast as it can.

Thumbs up. cool soluation to keep temporary stuff out of mysql!

[i93sme]

One ideea would be independent polling machines, with independent storage + a daemon on each that will expose functions like: graph, get data.
Graphing a set of individual graphs from each machine would not be a problem

I already implemented that since the frontend (graph server) is not trusted by the backend (the polling machines), thus the frontend has access only to the graph image via a set of RPC exposed functions.

The only problem would be in graphing a graph that will need data from several RRDs from diffent machines. The only method I see is that the frontend will only gather data from the backend machines then graph that data. That would mean the frontend should be able to hold that amount of data in a cache of sort.

Another issue I don't see mentioned here is redundancy. Take this setup as a start:

- machine A - has local storage, polls hosts A1,A2,A3
- machine B - has local storage, polls hosts B1,B2,B3

if either one of them drops dead we lose the collected data.

A shared storage would mean I/O bottleneck

One ideea would be to use a mirroring mechanism:

A is mirrored by B and viceversa

The result of polling on A is first stored locally, then sent to B
The result of polling on B is first stored locally then sent to A

Results that are not sent successfull are stored locally on each machine, so that if a machine recovers we will update its mirrored RRDs

Recovery would mean:
- if disk is gone on dead machine, rsync its rrds from the other one. If not, skip this step
- replay 'mirroring' logs from the other machine

A rsync based solution is quite out of the question since it will get the RRDs fully sent due to the fact that the files are constanly changing.

[hol]

Hi all,

I have managed to set up a Cacti with, up to now, 5 servers...
1 server holding the GUI and RRDs, also running the extra plugins (thersold, weathermap etc)
1 server holding the DB
3 servers as pollers...

I am using boost and multipollerserver plugin and the hole system is working just perfect..

Time:98.9222 Method:spine Server:poller1 Processes:4 Threads:80 Hosts:551 HostsPerProcess:138 DataSources:27454 RRDsProcessed:0
Time:109.6663 Method:spine Server:poller2 Processes:4 Threads:80 Hosts:552 HostsPerProcess:138 DataSources:28460 RRDsProcessed:0
Time:111.8289 Method:spine Server:poller3 Processes:4 Threads:80 Hosts:550 HostsPerProcess:138 DataSources:32510 RRDsProcessed:0

Up to now, I don't have IO problems but I am thinking of implementing GlusterFS to share the I/O load over the servers.

I have found a nice article here: http://www.howtoforge.com/striping-acro ... untu-12.04

I have also tried to cluster the DB (active/active replication) and share load the pollers to the DBs. It didn't work as expected and the replication was broken frequently.

Vangelis

[der_doc]

Hi all,

after one year i have finished a new version of the multipollerserver plugin.

@hol
I read joyfull that the plugin is running of the hole system perfect.