Scale cacti

merlinthemagic · Post by **merlinthemagic** » Fri May 18, 2007 4:43 pm

Hi All,

How do i scale cacti?

Currently running CentOS 4.4 cacti8.6j, cactid, boost 1.2 w/ dedicated (other server) MySQL server running memory tables for boost.

Poller runs off a Dual Xeon (non-HT) 2GB RAM, php max mem 256MB RAiD'ed SATA.

Load avg. <0.6

We add on avg. 250 Adv. Ping and 250 interface stats per month.

Problem is the 300s poller.php limit quickly becomes a problem.

It is like the poller (cactid) cannot utilize the resources like it is waiting for one host to reply with its stats before sending another request.. the processes / threads should fix that right?

05/18/2007 02:26:17 PM - SYSTEM STATS: Time:76.0036 Method:cactid Processes:20 Threads:10 Hosts:276 HostsPerProcess:14 DataSources:2156 RRDsProcessed:0

05/18/2007 02:41:51 PM - SYSTEM BOOST STATS: Time:33.6175 RRDUpdates:23738

This is from a new sever (same setup, thats how we scale it now... more servers) this way i can add 2 more C-class subnets and then i have to start another server.... and it is hard to figure out what subnets reside on what servers.

If money is not the problem what would you do? Quad core processor, more RAM? server load does not seem to be the problem.

How do i get a single cacti setup to hold 1000's of hosts and 10000' of data sources?

And i need the openness and ease of adding new scripts and data sources that Cacti gives... so no Orion Solarwinds for me thanks.

Help

Martin Madsen
Bel Air Internet

merlinthemagic · Post by **merlinthemagic** » Fri May 18, 2007 6:14 pm

Sorta fixed the issue, atleast im using the resources better now.

05/18/2007 04:09:09 PM - SYSTEM STATS: Time:23.8896 Method:cactid Processes:10 Threads:10 Hosts:276 HostsPerProcess:28 DataSources:2156 RRDsProcessed:0

I see the processor take a hit after the changes, never 100% used but sometimes down to 40% idle now (only when running the poller, rest of the time 99.8 idle).

This is the best result i could get main fix was to up the PHP Script servers from 1 to 10. Any reason it wont take hight values?

Im running php-5.1.6-3

Can someone tell me if there is any limit to how many hosts / datasources i can have on a single server IF hardware does not get maxed out? Is there a limited to how fast the poller can run?

Regards

Martin Madsen

Bel Air Internet

merlinthemagic · Post by **merlinthemagic** » Fri May 18, 2007 7:34 pm

... continuing my little monologue..

05/18/2007 05:30:25 PM - SYSTEM STATS: Time:24.4786 Method:cactid Processes:10 Threads:10 Hosts:353 HostsPerProcess:36 DataSources:3038 RRDsProcessed:0

Add 60 more hosts and almost 1000 data sources... time went up from 23.4........ to 24.4...

this is great

Martin Madsen
Bel Air Internet

Post by **TheWitness** » Fri May 18, 2007 8:03 pm

You can get the number down more if you simply reduce your processes. A dual system should have no more than 4. Then increase threads to 15. That equates to 60 concurrent devices queries at a time. Also, if you are only running SNMP, you can benefit further by reducing the number of concurrent script servers to 1. Keep in mind that Cactid is a connection hog and that your mysql server will have to handle many connections.

TheWitness

merlinthemagic · Post by **merlinthemagic** » Fri May 18, 2007 8:05 pm

the mysql server (seperate) has a 1GB key_buffer and 10000 max connections. That seems fine.

If i set the Php Script servers to "1" i exceed 300 sec.

Just wondering why the PHP script server max is set to 10.

Regards,

Martin Madsen
Bel Air Internet

Post by **TheWitness** » Fri May 18, 2007 8:08 pm

10000 connections is a bit high. Depending on the number of users, I would rather you do say 500.

TheWitness

merlinthemagic · Post by **merlinthemagic** » Fri May 18, 2007 8:17 pm

i got where i needed to be, but here is the compare 10x10 vs 4x15 (both running 10 script servers)

05/18/2007 06:15:25 PM - SYSTEM STATS: Time:23.4608 Method:cactid Processes:10 Threads:10 Hosts:353 HostsPerProcess:36 DataSources:3038 RRDsProcessed:0

05/18/2007 06:15:01 PM - CMDPHP: Poller[0] WARNING: Poller found partially populated poller_output table.

05/18/2007 06:10:31 PM - SYSTEM STATS: Time:29.5776 Method:cactid Processes:4 Threads:15 Hosts:353 HostsPerProcess:89 DataSources:3038 RRDsProcessed:0

Regards,

MM

Post by **TheWitness** » Fri May 18, 2007 8:25 pm

If you track your load average, you will be in better shape. Play more with the threads, but don't go too high.

What is your count of the following:

Script
Script Server
SNMP

You can get to this information by goting System Utilities and Veiw Poller Cache and counting the rows each way.

TheWitness

merlinthemagic · Post by **merlinthemagic** » Fri May 18, 2007 8:34 pm

Hey,

too many rows to count but im only using 2 kinds of data sources.

About 30 cisco switches 24 interfaces each, getting bandwidth stats:

SNMP Version: 1, Community: XXXXX, OID: .1.3.6.1.2.1.2.2.1.16.5 RRD: /var/www/html/cacti/rra/XXXX.rrd

About 300 ips being monitored via Adv. Ping:

Script Server: /var/www/html/cacti/scripts/ss_fping.php ss_fping xxx.xxx.xxx.xxx 10 ICMP
RRD: /var/www/html/cacti/rra/xxx_xxx_xxx_xxx_loss_623.rrd

Regards,

Martin Madsen

Post by **TheWitness** » Fri May 18, 2007 8:41 pm

So, you need to up your MAX OID GET requests per device to something around 50 or so, or until you start getting errors and then back off just a pinch from there. Also, you need to beef up your script servers as the advance ping is what they do.

Then the way the script servers work is that your pool of threads only have X script servers to handle their requests. So, if you have 15 processes and 5 script servers, you will have some waiting in line. If you think your system could handle it, you could have 15 threads and 15 script servers.

Having the MAX get size high, means that for your network switch, you are only doing 1 or two transactions for all you polling per device. The rest of the time is spent with the pings.

TheWitness

merlinthemagic · Post by **merlinthemagic** » Fri May 18, 2007 8:46 pm

I am running OiD get 50, i will play with it a bit more though if you think it could make a difference.

Setting the script servers to 15 might be a problem since the page states:

The number of concurrent script server processes to run per Cactid process. Settings between 1 and 10 are accepted. This parameter will help if you are running several threads and script server scripts.

The goal is to see the server run out of resources, if i ca do that ill get another more powerful server.

I will love it when i can start scaping some of the older servers and combine them on one.

Regards,

Martin

Post by **TheWitness** » Fri May 18, 2007 8:49 pm

It's been so long since I wrote it that I forgot that I placed a limit on it. So, goto the limit. If you think the server still have room, let me know. You can, of course, increase the number of script servers by simply increasing the number of processes. The bad thing about cactid, is the initial MIB loading.

That can be avoided if you compile net-snmp yourself or simply making sure net-snmp does not have any mibs to load...

TheWitness

merlinthemagic · Post by **merlinthemagic** » Fri May 18, 2007 8:58 pm

Cool Larry,

A few things to work on, ill recompile net-snmp, and see if i can figure out how you enforce that script server cap, so i can go beyond the 10.

Ill keep loading hosts on the box over the weekend, but its friday 7pm in LA, there are hotter things to do.

Thanks for all your quick and insightful help.

Regards,

Martin Madsen
Bel Air Internet

Post by **TheWitness** » Fri May 18, 2007 9:01 pm

No problem. When you configure net-snmp, the keyword is "--disable-mib-loading". Simply run "./configure --help". Mib loading slows down Cactid from polling and you can loose a few seconds of precious time waiting for X mibs to be loaded into memory.

Enjoy LA. I'll be there next week at Star Wars Celebration IV.

TheWitness

merlinthemagic · Post by **merlinthemagic** » Thu Oct 09, 2008 5:41 pm

hey,

How can i exceed the 10 php script server limit set under settings -> poller?

I have tons of host that are just being pinged, and it seems scriptservers are the way to fix that.

the server load is 0.30.

Cacti

Scale cacti

Scale cacti

PHP Script server

monolouge

hey

compare

cool

php script servers

Who is online