Scaling Cacti

[rmckenna]

Hi,

I have a question regarding scaling Cacti.
I have am running Cacti 0.8.7b on a the following hw:

CPU: 2x Xeon E5345, quad core (ie 8 cores)
RAM: 8GB

I want to add more nodes to my machine, but I am very iobound..

ie.

Code: Select all

Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00    74.13  0.00 417.93     0.00     1.92     9.42   113.60  271.39   2.24  93.48
dm-0              0.00     0.00  0.00  2.20     0.00     0.01     8.00     3.49 1587.87 226.12  49.75
dm-1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-2              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-3              0.00     0.00  0.00  1.54     0.00     0.01     8.00     0.92  600.35 157.64  24.28
dm-4              0.00     0.00  0.00 489.22     0.00     1.91     8.00   137.04  279.75   1.90  93.14
dm-5              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-6              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-7              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-8              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

My last run was this:

Last Run Statistics Time:247.7282 Method:cmd.php Processes:5 Threads:N/A Hosts:970 HostsPerProcess:194 DataSources:36023 RRDsProcessed:20185

running cmd.php (I have added and removed processes, without much reduction in time)

I am investigating upgrading to spine.
Would this reduce my reliance on the disk? I am far from CPU bound.

Else does anyone have any tips to help IO, other than modifying the hardware, and moving the SQL db onto separate disk. (Saving these options to last).

Also I notice that the last subset of device-id's takes the longest to process. Is there a recommended way to spread the host id's around. (the last lot of hosts added are WAN linux devices and take the longest to poll)

thanks in advance,
Rich

[Howie]

Boost should help more, by collecting multiple IOs together...

[rmckenna]

thanks Howie,

I will investigate the option of installing the boost plugin.
I removed some data points and change the number of processes to 16 and now I am able to poll in *just* under 300sec.

Does anyone else have any recommendations?
I am still interested to know if anyone has reduced their i/o using spine.
My RRD data file is ~7.7GB. I have read some recommendations for RRA size + 2GBs.

Code: Select all

@cacti rra]$ free -m
             total       used       free     shared    buffers     cached
Mem:          7983       7378        605          0        468       6049
-/+ buffers/cache:        860       7123
Swap:         5983          0       5983

thanks,
Rich

[Exo7]

which version of rrdtool are you using ?
1.2.26 disable read-ahead caching, which might help in your situation.
1.3 has a feature called "fadvise", which is smarter.

[Alice]

rrdool 1.2.26+ helps, spine helps much, boost helps a lot, separate mysql server helps, but not very much.

04/22/2009 07:13:05 PM - SYSTEM BOOST STATS: Time:44.0956 RRDUpdates:80257

04/22/2009 07:14:20 PM - SYSTEM STATS: Time:18.6741 Method:spine Processes:10 Threads:6 Hosts:271 HostsPerProcess:28 DataSources:26950 RRDsProcessed:0

Oh, and it's a dual dual-core xeon 3.2.

Code: Select all

root@stats2:~# free -m
             total       used       free     shared    buffers     cached
Mem:          2024       1584        439          0        223       1153
-/+ buffers/cache:        207       1816
Swap:          980          2        978