Sizing Cacti to poll 12,000 nodes

[Ratbert]

My group would like to setup utilization polling on all switchports of about 12,000 switches. The goal is to offload almost all interface polling from Orion and free it up to primarily handle alerting and traps for the network infrastructure itself, while at the same time gaining per-port historical data when platform groups come to us wanting to know about the traffic patterns of a particular access port (which is not something we can currently support with Orion due to scaling issues.) We've used Cacti within the dept for pet-projects and one-off monitoring but weren't sure if anyone had scaled an implementation to this sort of size before and, if so, what sort of limitations or design considerations we need to consider. I would expect that disk I/O to the SAN would probably be the primary bottleneck for the collectors but I'm not sure what the common sizing recommendations would be. Can anyone with experience or knowledge with a large implementation speak to this?

[cigamit]

I haven't had to scale that high myself, but I know of a few large installations that are that big or bigger.

1. Use the boost plugin, it will help cut down on disk I/O a lot
2. Be extremely careful if you decide to install other plugins, a lot of them will slow you down greatly.
3. Be sure to change the poller_output table to a MEMORY table.
4. Build a separate box to house the MySQL server with tons of RAM (and properly configure it for large memory tables and caches)
5. If possible, use SSDs (or FusionIO Cards) for disk storage
6. If your network latency is normally pretty good, set the snmp timeouts low. If you have a device go down, you don't want it taking precious seconds while it waits on it
7. Set Down Host Detection to just SNMP Uptime. No need to send extra packets to Ping and SNMP test the switches.
8. Probably a ton of things I didn't mention but others will chime in