cacti, collectd and externally updated remote rrds

[psyber]

I'm really behind the curve on collectd but after reading the docs I'm excited.
My very first thoughts were on how to display that data in Cacti. Then I found this post: http://forums.cacti.net/about28169.html
(Which BTW I would love to see something along these lines supported in cacti itself maybe a collectd plugin AND some way to better create ds and graphs for externally updated rrds)

I know there was discussion on one of the developer forums about designing a distributed cacti, and I make extensive use of externally updated rrds. I've played with rrd server some and now I'm wondering if I could make use of externally updated rrds that exist on remote servers. From a scalability standpoint where the number of data sources is your limiting factor, spreading the DSs across multiple servers makes a lot of sense. I think the rrd file location could be hacked in such a way if you passed it hostname:13900/path/to/rrd/ cacti could pass its graph commands through the socket to the remote host and have the img returned for display.

So the model I'm building in my head is collectd --->collectd server running an rrdserver <-----cacti connecting to the rrd server and passsing a graph command--->image from collectd server back to cacti for display...
This is all completely untested, I haven't even looked at the code yet, and quite possibly the network latency may kill the end user experience which would make this all a moot point. But I'm quite curious if anyone out there has tried something like this and what their experiences might be.

Thoughts? Comments? Lost my mind?

[gandalf]

We are thinking about a "mini Cacti", including spine, reading a writing to a central MySQL. That comes close to your structure. But there are currently no plans to support different pollers.
R.

[heybigben]

I also really like the idea of combining collectd and cacti. I like collectd because it includes a push model where we can have have a large number of compute hosts all reporting data over tcp channel to a central server where the data is logged to flat files via csv or directly to rrd. Also the config file is really easy to customize. I really like cacti for the tree views, and multiple methods of viewing data (list, preview, etc.) and the web access controls and realms.

So I had a slightly different approach to combine the two programs together. Use cacti to poll the collectd daemon for data via the collectd cli (or direct socket in future release) or have cacti use a custom data input method to grab the data from the cvs file. Then just have a script run in cron which queries collectd and determines the hostnames and then auto populates cacti with the device entries (if there missing). There probably a way to do this better as a cacti plugin but I haven't figured that out yet. By using the cacti poller to get data from collectd, it avoids the external update method - which doesn't seem to work with realtime or thold.

[TheWitness]

Quietly stating that "ganglia" is the right information propagation methodology, but very quietly.

TheWitness

[heybigben]

Both ganglia and collectd are great tools for cluster monitoring. I prefer collectd since it has more input and output options (i.e. cvs, rrd, etc.). But I would still really like to use cacti for the frontend and let those other tools gather the data. Much easier then snmp.

[TheWitness]

I would say plugin. The real issue that the former solves is the whole concept of aggregation/association. Stay well.