Spine Tuning

[inzeos]

Trying to determine how to increase the performance of our Cacti installation.

03/14/2008 01:26:56 PM - SYSTEM STATS: Time:54.8899 Method:spine Processes:5 Threads:10 Hosts:35 HostsPerProcess:7 DataSources:94 RRDsProcessed:63
03/14/2008 01:25:56 PM - SYSTEM STATS: Time:54.9135 Method:spine Processes:5 Threads:10 Hosts:35 HostsPerProcess:7 DataSources:95 RRDsProcessed:60
03/14/2008 01:24:56 PM - SYSTEM STATS: Time:54.7340 Method:spine Processes:4 Threads:4 Hosts:35 HostsPerProcess:9 DataSources:99 RRDsProcessed:63
03/14/2008 01:23:56 PM - SYSTEM STATS: Time:54.7791 Method:spine Processes:4 Threads:4 Hosts:35 HostsPerProcess:9 DataSources:94 RRDsProcessed:58

Increased the processes and threads with minimal increases in the time it took to process. Currently using maximum of 20 OID's. 80% of the devices are being polled over WAN links ranging from 1.54Mbps to 6Mbps.

Machine running Cacti is a dual Xeon 2.8 Ghz w/ 2Gb of ram.

[gandalf]

What is the latency on the WAN links? This may turn out to be the key point!
Else, try to increase maxmimum OID per get request to about 50-60.
Add more RAM.
Use latest Linux Kernels and latest rrdtool (there was a quite big improvement due to avoiding cache mess up)
Reinhard

[inzeos]

The latency can be up to 150ms on the busier times of the day. I'll double check my kernel and rrdtool versions.

I didn't think it was a ram limitation as during the polling there are still free resources available.

[gandalf]

The latency can be up to 150ms on the busier times of the day. I'll double check my kernel and rrdtool versions.

I didn't think it was a ram limitation as during the polling there are still free resources available.

For that few hosts, you're correct. More RAM will not be the first issue. But when growing, you might hit the border, especially without newest rrdtool.
Reinhard

[inzeos]

I was able to do some performance tuning. I wanted to follow up here so that others might learn. We make extensive use of the Advanced Ping template / systems. It turns out that we had several devices that were setup with Advanced Ping; however, the packets were being filtered our prior to arriving at the devices. Once we disabled those advanced pings we saw a dramatic decrease in the time it took to run our polling interval. We were also able to decrease the threads and processes running.

[gandalf]

That's reasonable to to a significant timeout/retry setting that makes the poller wait most of the time. So an alternative (not a good one for sure) would be to reduce the timeout/retry
Reinhard