cactid/cmd.php polling based on hostname or description

[cscoman]

I currently have multiple data centers around the world and use cacti to collect statistics on all servers and network hardware. I am looking for a feature that would allow me to have a polling server in each data center than can run against only hosts in that data center. I tried using the start/end host id feature, but hosts are not added in an order that can support this very easily. It would be helpful if cactid/cmd.php would accept an option to only poll hosts that contain the subdomain for that data center. Example is to poll all hosts that are in the dfw sub domain. So all (hostname).dfw.(domain.com) hosts would be polled and results returned to the central DB server for graphing.


Thanks
Jason

[marnues]

This would be an amazing feature.
But it seems to me that this should be handled before cactid/cmd.php is called.
I recently made poller.php a very basic distributed poller, but all I do was start dividing up the host ids.
Of course, our core devices all have low ids and so the first poller does a lot more heavy lifting than the other 2.
Its not a huge difference in performance, but it would be nice to split things up based on template, hostname, or ip address.
I haven't delved much into the cactid internals (just enough to get cli options, really), but it seems to me that poller.php could then call cactid (or cmd.php, but you're probably not using it if you need this kind of feature) on each host id range that it finds

this would be some heavy lifting on poller.php's part (loads of SQL I imagine), but I think it would be pretty cool

[gandalf]

This feature was discussed extensively at the 1.CCC.eu conference. Personally, I would suggest to add a new field to each host, called "SITE". It will associate each host with a specific location, the SITE (perhaps SITEs should be held as a table on its own and presented as a dropdown for the DEVICE page).
Now, each of the "distributed" pollers should also be associated with exactly one SITE. It should get all relevant hosts by performing an SQL against host table using "WHERE SITE = <my site>".
I assume, that SITE would be a good filter even for people interested in data from their SITE only ...
just my two cents
Reinhard

BTB: Not yet solved: how to the pollers transfer the data to mysql? MySQL remote connections? Local MySQL db using one way replication with a central DB? Or a "local cache" (that could as well be a local MySQL db), replicating only on demand or on specific cycles to reduce network load?

[cscoman]

Reinhard,
I like your idea of the SITE field better than what I was thinking. As for your last thought. In my case I would be fine with using a remote mysql connection. I have the inter site connectivity to support it. I just want to have the polling done local to the site to help increase performance. I have a total of 250 devices and 3270 data sources so far and I keep adding more.

Jason

[gandalf]

Oh, that amount of ds should not be a problem. I know about sites using 40,000+ ds (we are currectly at about 15,000 with about 30-40 sec polling time). Sites above 100,000 ds may require the boost plugin. Even this size will not require distributed polling.
But situations may occur, where local polling is required (e.g. when using high latency sattelite links) and most of the time is consumed transferring SNMP requests
Reinhard

BTB: You may want toconsider using my Poller Templates and the latest Poller Performance plugin to trace your timings

[cscoman]

You are right, that is not a lot of data sources compared to a lot of sites. I really can not complain about current performance, I just want to move the checks to local systems. I am also interested in this as I have hardware behind firewalls that I can not allow my current cacti server in to monitor. I can however have a server behind the firewall do the polling and then pass the collected data out to the cacti server.

Jason

[gandalf]

I agree; that would be a reason for distributed polling.
But there's a problem as well. When polling centrally, you will have to open port 161 (or whatever you've configured for SNMP) for outbound requests only. Using "my" approach, the poller will try to open the mysql port for inbound requests.
Reinhard[/b]