Scale cacti

[merlinthemagic]

I added processes (after reading the post closer) with strange results:

Server is 8 core with 4GB RAM:

10/09/2008 04:10:57 PM - SYSTEM STATS: Time:55.6282 Method:spine Processes:32 Threads:10 Hosts:4790 HostsPerProcess:150 DataSources:23196 RRDsProcessed:11234
10/09/2008 04:07:28 PM - SYSTEM STATS: Time:147.2908 Method:spine Processes:64 Threads:10 Hosts:4790 HostsPerProcess:75 DataSources:23196 RRDsProcessed:11234
10/09/2008 04:00:41 PM - SYSTEM STATS: Time:40.0560 Method:spine Processes:32 Threads:10 Hosts:4790 HostsPerProcess:150 DataSources:23196 RRDsProcessed:11234
10/09/2008 03:55:51 PM - SYSTEM STATS: Time:49.7520 Method:spine Processes:16 Threads:10 Hosts:4790 HostsPerProcess:300 DataSources:23196 RRDsProcessed:11234
10/09/2008 03:51:30 PM - SYSTEM STATS: Time:88.9863 Method:spine Processes:8 Threads:40 Hosts:4790 HostsPerProcess:599 DataSources:23196 RRDsProcessed:11234
10/09/2008 03:46:32 PM - SYSTEM STATS: Time:90.2485 Method:spine Processes:8 Threads:40 Hosts:4790 HostsPerProcess:599

[TheWitness]

Each process can have upto 10 script servers. If you are loaded with Ping, then by adding more processes, you grow the number of pings dramatically. If you have lot's of threads, they all have to wait on only 10 script servers.

So, think about the parallel pipeline. With 16 processes with each having 10 script servers, you can perform 160 concurrent pings. Thats a lot

How high do you foresee the script server count going?

TheWitness

[merlinthemagic]

Not sure what u mean by:

"How high do you foresee the script server count going?"

im currently running:

10/10/2008 09:15:52 PM - SYSTEM STATS: Time:50.9777 Method:spine Processes:32 Threads:10 Hosts:4888 HostsPerProcess:153 DataSources:32141 RRDsProcessed:14388

@

Maximum Threads per Process: 10
Number of PHP Script Servers: 10
Script and Script Server Timeout Value: 60
The Maximum SNMP OID's Per SNMP Get Request: 30

Cant seem to max out the resources, but then again maybe i dont quite understand how to messure them. I see this running "top" when the spine poll is just started:

top - 21:25:05 up 1 day, 6:19, 1 user, load average: 0.80, 1.33, 1.50
Tasks: 380 total, 1 running, 379 sleeping, 0 stopped, 0 zombie
Cpu(s): 28.4%us, 6.5%sy, 0.0%ni, 64.3%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0%st
Mem: 4147048k total, 3251576k used, 895472k free, 81520k buffers
Swap: 8193140k total, 44k used, 8193096k free, 1845940k cached

And like this when the poll is complete:

top - 21:24:11 up 1 day, 6:18, 1 user, load average: 1.00, 1.50, 1.57
Tasks: 153 total, 1 running, 152 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni, 99.9%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 4147048k total, 2288752k used, 1858296k free, 81432k buffers
Swap: 8193140k total, 44k used, 8193096k free, 1845936k cached

The Database server resides on a separate Dual Xeon 3.0 server on the same switch and subnet (if the database could slow the overall poll process).

We have full /16 that i would like to add (32768 additional ips @ 10 pings per poll (adv. ping)), and maybe another 900 switches (48 port cisco traffic and errors).

Is there any point in starting to play with CPU affinity? giving the NIC a dedicated CPU? dont think i can do process affinity on CentOS 5.2, maybe im wrong.

Maybe boost will help (separate boost server maybe?) just cant seem to get it working on 8.7b.

Ill jack up the ping count and let you know once we start getting close to the 300s poll time.

Thanks for your time.

MM