Huge Cacti installation - suggestions?

[ariesgeek]

Dell 2850 - 2 x 3.4GHz Xeon processor, 4GB RAM, RedHat Enterprise. Currently 0.8.6j with cactid, plan to use 0.8.7.

I have about a thousand devices, all either routers, switches, or similar network devices. We want to collect data for EVERYTHING. That is, there will be about 150,000 graphs. We need historical data for all ports on all switches, etc.

I've setup scripts to do the importing and graph configuring. It's slow but effective.

I've set PHP memory to 128M. An arbitrary number. Suggestions on this?

I have 2 processes with 15 cactid threads. Again, arbitrary numbers that sounded good. Suggestions?

So far 20,000 graphs have been setup (and counting as I type), and I'm up to around 70 seconds poll time. Not bad. I notice that as cactid is running, mysqld and php take turns tearing at 100% CPU utilization, or will drop to 50 and 50. At least according to top.

What I'm looking for is just suggestions. Performance suggestions. Threads? Processes? PHP settings? MySQL settings?

I saw the Boost plugin but want to wait until it's confirmed stable on 0.8.7, and of course that I've upgraded to 0.8.7, before installing.

I'd really be interested in hearing from someone with a similar installation.

Thanks!

[gandalf]

Dell 2850 - 2 x 3.4GHz Xeon processor, 4GB RAM, RedHat Enterprise. Currently 0.8.6j with cactid, plan to use 0.8.7.

Disk speed is of great importance. Use rrdtool 1.3 for optimal performance. Use SNMPV2 for your targets and vary the snmpbulkwalk factor.

I've set PHP memory to 128M. An arbitrary number. Suggestions on this?

The memory must be huge enough to hold a completely filled poller_output table. It is required, to change memory settings again when using BOOST

I have 2 processes with 15 cactid threads. Again, arbitrary numbers that sounded good. Suggestions?

The XEONs are dual cores, right? So use 4 processes at least. 15 threads is a good starting point. You may vary this value between 12 and 50 to find optimal settings

So far 20,000 graphs have been setup (and counting as I type), and I'm up to around 70 seconds poll time. Not bad.

20,000 graphs or 20,000 data sources? I recommend using my Poller Performace Graph Templates to follow cacti polling performance.

I notice that as cactid is running, mysqld and php take turns tearing at 100% CPU utilization, or will drop to 50 and 50. At least according to top.

For those setups, a seperate MySQL database machine is recommended with enough memory to hold virtually the whole cacti database in memory.

What I'm looking for is just suggestions. Performance suggestions. Threads? Processes? PHP settings? MySQL settings?

Unfortunatley, personally I don't have my.cnf for such a huge installation. THIS WILL SURELY BE OF GREAT IMPORTANCE.

I saw the Boost plugin but want to wait until it's confirmed stable on 0.8.7, and of course that I've upgraded to 0.8.7, before installing.

This will surely be required, sooner or later
Reinhard

[N3NCY]

You will no doubt need to use the "Boost Plugin" (that means not using the "Weathermap Plugin" at this time / current revision).
They key to a large Cacti configuration (as in my case) is using Boost to seperate two (2) key functions within Cacti:
1.) The poller (use Cactid aka Spine) must poll data completely seperate from updating the RRD files
2.) Boost updates the RRD files (every hour in my case) and is COMPLETELY seperated from the poller.

On my Dual CPU / Dual core Xeon (8GB RAM) the poller gets over 46,000 pieces of data in about 70 seconds and repeats every 5 minutes.
Boost updates about 8,000 graphs (disk susbsystem is RAID 5 - 4 15,000RPM SATA) in about 25 minutes - I am hopeful that a newer beta RRD Tool will reduce this time someday soon.
There is really no need to update the RRD files every 5 minutes since I don't look at every graph every polling cycle.
Boost has a function that updates only the RRD files I view if I want to look at data not yet commited to RRD in this cycle.
(Weathermap would need to use that function if you want to look at data before Boost updates the RRD files each hour)

[Howie]

You will no doubt need to use the "Boost Plugin" (that means not using the "Weathermap Plugin" at this time / current revision).

Now that 0.93 is out of the way, this is back on track again.

[ariesgeek]

Thank you all for your replies, but on my machine MySQL decides it's had enough somewhere around 35k - 40k graphs. Unfortunately having a seperate MySQL server isn't an option at this time.

So I've decided to cut back a bit and not graph switch interfaces unless needed. I'll keep the devices modeled, just no interface graphs. We can easily graph switch ports as needed this way.

Not as proactive as I would like, but still better than what we have now.

This will still be a significant Cacti installation, as I'll still probably have 10k - 20k graphs, and I'll be sure to post various stats in appropriate threads. So your advice is still apprecaited and will still be used.

So a quick question for the community regarding php memory limits. Based on a comment I saw from Gandalf, I'll probably have to set my php memory limit to somewhere from 1GB - 2GB. Does this sound reasonable? Will it cause problems elsewhere setting it so high?

[gandalf]

This will still be a significant Cacti installation, as I'll still probably have 10k - 20k graphs, and I'll be sure to post various stats in appropriate threads.

That's not an issue. We're running this size with a Dual XEON system an 4GB RAM with MySQL collocated. Polling time's around 40 sec. PHP Memory is set to 64MB for us. But this is largely related to PHP version. Start with 128/256 MB
Reinhard

[ariesgeek]

Thanks for the info.

At that level, I should be running 50 - 70 seconds, which isn't a problem. Any reason for me to run Boost still? It seems like although it's better for performance it may be a bit slower from the user perspective, no? I could move this question over to the appropriate forum if necessary.

[gandalf]

There's no need to worry about user issues. Use BOOST, if there's a real need, not earlier.
Reinhard

[rcaston]

There's no need to worry about user issues. Use BOOST, if there's a real need, not earlier.
Reinhard

Boost is greatness, but if they can't put enough ram into the box to make Boost resolve the issue, they could try throwing drive spindles at it.

10+ drives in a large raid 1+0 configuration just for the RRAs would be an ideal (possibly overkill?) way to eliminate the bottleneck they are having.

[lwhitb02]

Hi,

I'm also looking at doing a 150K data source installation. However, I currently only use Cacti in a 10 device, 250 DS install.

I don't want to purchsae a whole bunch of hardware and put a lot of time into the project unless there is a reasonable chance of success so I'm interested to know:

How many servers and what hardware spec would be recommended?

What other large installations are already in place using cacti?

What tools or processes are best to use to import and maintain large numbers of devices?

How big would the DB and RRD disk usage likely be for 150K DS?

What addons (just read about Boost) would be recommended?


Any interest would be appreciated. At this stage I'm also looking at commercial offerings but the idea of building something big with Cacti is appealing because it would probably provide a richer source of information than an off-shelf system if we invest some time into it.

[gandalf]

Please see http://forums.cacti.net/viewtopic.php?t=6206 for some numbers. PM TheWitness for more information. 150 k data sources should be ok, but depends on polling interval (1 min? 5 min? 15 min?)
Reinhard

[greginmadison]

Gandalf, When you say, "Enough memory to hold virtually the whole cacti database in memory".

What do you mean? Do you mean take a look at how much disk space is being consumed by all the .rdd files and this is how much memory MySQL should have to enable the memory only option for Boost?

I too am planning a large install monitoring 8000 switch ports (in/out traffic,cpu, discards), 50-60 routers and I estimate about 50k-60K rrds on a 5 min interval.

I plan on expanding the non-averaged DS to hold 7 days of data for all my in/out traffic so this may impact my ability to move to Boost if my poll times get too long. But if the files are too large it would require quite a bit of memory.

Also, what is the mysql command or other method to view the full size of the output poller (for php memory sizing)?

[gandalf]

Gandalf, When you say, "Enough memory to hold virtually the whole cacti database in memory".

What do you mean? Do you mean take a look at how much disk space is being consumed by all the .rdd files and this is how much memory MySQL should have to enable the memory only option for Boost?

No, rrd files hold the numbers, not the administrative data. I meant caching the mysql tables will improve performance.

I too am planning a large install monitoring 8000 switch ports (in/out traffic,cpu, discards), 50-60 routers and I estimate about 50k-60K rrds on a 5 min interval.

That's quite easy. We do about 23k data sources in about 45 sec. without boost and with a quite old rrdtool on a dual XEON with 4 GB of RAM. IMHO nothing special required.

I plan on expanding the non-averaged DS to hold 7 days of data for all my in/out traffic so this may impact my ability to move to Boost if my poll times get too long. But if the files are too large it would require quite a bit of memory.

No. See above. Latest rrdtool (>= 1.2.26) will be good on a quite recent Linux kernel.

Also, what is the mysql command or other method to view the full size of the output poller (for php memory sizing)?

Don't bother. Cacti 087b provides code for this.
Reinhard