Submit your BOOST Statistics Here

[TheWitness]

Very nice demonstration of positive impact.

How many hosts? How many data sources? How many RRDs? Show some before and after "STATS".

TheWitness

[Jeb]

05/24/2007 02:19:22 PM - SYSTEM BOOST STATS: Time:221.6682 RRDUpdates:86368
05/24/2007 02:15:39 PM - SYSTEM STATS: Time:38.1070 Method:cactid Processes:8 Threads:16 Hosts:376 HostsPerProcess:47 DataSources:28971 RRDsProcessed:0
05/24/2007 02:10:40 PM - SYSTEM STATS: Time:39.4355 Method:cactid Processes:8 Threads:16 Hosts:376 HostsPerProcess:47 DataSources:28971 RRDsProcessed:0
05/24/2007 02:05:57 PM - SYSTEM STATS: Time:56.4675 Method:cactid Processes:8 Threads:16 Hosts:376 HostsPerProcess:47 DataSources:28970 RRDsProcessed:0
05/24/2007 02:05:42 PM - SYSTEM BOOST STATS: Time:299.0875 RRDUpdates:86055
05/24/2007 02:00:40 PM - SYSTEM STATS: Time:39.9136 Method:cactid Processes:8 Threads:16 Hosts:376 HostsPerProcess:47 DataSources:28970 RRDsProcessed:0
05/24/2007 01:55:38 PM - SYSTEM STATS: Time:37.3334 Method:cactid Processes:8 Threads:16 Hosts:376 HostsPerProcess:47 DataSources:28970 RRDsProcessed:0
05/24/2007 01:50:39 PM - SYSTEM STATS: Time:39.2261 Method:cactid Processes:8 Threads:16 Hosts:376 HostsPerProcess:47 DataSources:28970 RRDsProcessed:0
05/24/2007 01:49:16 PM - SYSTEM BOOST STATS: Time:213.1302 RRDUpdates:57637
05/24/2007 01:45:40 PM - SYSTEM STATS: Time:40.0143 Method:cactid Processes:8 Threads:16 Hosts:376 HostsPerProcess:47 DataSources:28970 RRDsProcessed:0
05/24/2007 01:40:39 PM - SYSTEM STATS: Time:38.7827 Method:cactid Processes:8 Threads:16 Hosts:376 HostsPerProcess:47 DataSources:28970 RRDsProcessed:0
05/24/2007 01:39:38 PM - SYSTEM BOOST STATS: Time:237.0712 RRDUpdates:86261
05/24/2007 01:35:39 PM - SYSTEM STATS: Time:38.9514 Method:cactid Processes:8 Threads:16 Hosts:376 HostsPerProcess:47 DataSources:28970 RRDsProcessed:0
05/24/2007 01:30:42 PM - SYSTEM STATS: Time:41.0048 Method:cactid Processes:8 Threads:16 Hosts:376 HostsPerProcess:47 DataSources:28970 RRDsProcessed:0
05/24/2007 01:25:40 PM - SYSTEM STATS: Time:39.4576 Method:cactid Processes:8 Threads:16 Hosts:376 HostsPerProcess:47 DataSources:28970 RRDsProcessed:0
05/24/2007 01:23:35 PM - SYSTEM BOOST STATS: Time:156.9922 RRDUpdates:57513
05/24/2007 01:20:54 PM - SYSTEM STATS: Time:54.0344 Method:cactid Processes:8 Threads:16 Hosts:376 HostsPerProcess:47 DataSources:28970 RRDsProcessed:0

Before... polling+update taken around 320seconds

This is really great ! I will be able to graph all things i want now
I was limited to around 30k datasource to be able to run the poller every 5 minutes...
I will post some graph later (i have just installed boost)
I will also try to do something to be able to update rrds used by weathermap after each poll (somethink like the on demand update for graphs)

[TheWitness]

Here is a pass of the new Cactid 0.8.6j/k and Boost 1.4

07/11/2007 05:05:40 AM - SYSTEM STATS: Time:38.6084 Method:cactid Processes:10 Threads:30 Hosts:3372 HostsPerProcess:338 DataSources:18583 RRDsProcessed:0

TheWitness

[nebj00la]

Here is a pass of the new Cactid 0.8.6j/k and Boost 1.4

07/11/2007 05:05:40 AM - SYSTEM STATS: Time:38.6084 Method:cactid Processes:10 Threads:30 Hosts:3372 HostsPerProcess:338 DataSources:18583 RRDsProcessed:0

TheWitness

Is it "safe" to try version 1.4 on a 0.8.6j install, I'd like to try and lower my load average. I'm averaging about 4-5 right now, and would like to be closer to 3.

[TheWitness]

I would go to 1.5. However, I have found that if you have to turn it off, that it's not working too well. If you do have to turn it off for some reason, force a run and then comment it out from config.php.

Hmm, Did I release 1.5. Have to check...

TheWitness

[disirk]

08/04/2007 07:53:18 PM - SYSTEM STATS: Time:197.6914 Method:cactid Processes:4 Threads:20 Hosts:566 HostsPerProcess:142 DataSources:99795 RRDsProcessed:0
08/04/2007 07:57:48 PM - SYSTEM STATS: Time:167.7309 Method:cactid Processes:4 Threads:20 Hosts:566 HostsPerProcess:142 DataSources:99826 RRDsProcessed:0
08/04/2007 08:03:56 PM - SYSTEM STATS: Time:235.2068 Method:cactid Processes:4 Threads:20 Hosts:566 HostsPerProcess:142 DataSources:99858 RRDsProcessed:0
08/04/2007 08:04:25 PM - SYSTEM BOOST STATS: Time:684.8934 RRDUpdates:290910
08/04/2007 08:07:53 PM - SYSTEM STATS: Time:172.7117 Method:cactid Processes:4 Threads:20 Hosts:566 HostsPerProcess:142 DataSources:99885 RRDsProcessed:0
08/04/2007 08:13:58 PM - SYSTEM STATS: Time:238.2603 Method:cactid Processes:4 Threads:20 Hosts:566 HostsPerProcess:142 DataSources:99912 RRDsProcessed:0
08/04/2007 08:17:32 PM - SYSTEM BOOST STATS: Time:751.7127 RRDUpdates:194710
08/04/2007 08:18:20 PM - SYSTEM STATS: Time:199.9822 Method:cactid Processes:4 Threads:20 Hosts:567 HostsPerProcess:142 DataSources:100407 RRDsProcessed:0
08/04/2007 08:23:06 PM - SYSTEM STATS: Time:185.9694 Method:cactid Processes:4 Threads:20 Hosts:567 HostsPerProcess:142 DataSources:100941 RRDsProcessed:0
08/04/2007 08:27:34 PM - SYSTEM STATS: Time:154.2108 Method:cactid Processes:4 Threads:20 Hosts:567 HostsPerProcess:142 DataSources:100941 RRDsProcessed:0
08/04/2007 08:28:25 PM - SYSTEM BOOST STATS: Time:624.8719 RRDUpdates:293256
08/04/2007 08:32:58 PM - SYSTEM STATS: Time:177.4861 Method:cactid Processes:4 Threads:20 Hosts:567 HostsPerProcess:142 DataSources:100941 RRDsProcessed:0
08/04/2007 08:37:59 PM - SYSTEM STATS: Time:178.8470 Method:cactid Processes:4 Threads:20 Hosts:567 HostsPerProcess:142 DataSources:100941 RRDsProcessed:0
08/04/2007 08:38:31 PM - SYSTEM BOOST STATS: Time:571.0101 RRDUpdates:196906
08/04/2007 08:43:09 PM - SYSTEM STATS: Time:188.7387 Method:cactid Processes:4 Threads:20 Hosts:567 HostsPerProcess:142 DataSources:100941 RRDsProcessed:0
08/04/2007 08:47:44 PM - SYSTEM STATS: Time:163.8680 Method:cactid Processes:4 Threads:20 Hosts:567 HostsPerProcess:142 DataSources:100943 RRDsProcessed:0

After messing with my hardware (note, 15k RPM drives are a must with many RRDs to update) I am now able to poll my entire network in less than 3 minutes, and also update all the RRD's. I've been running Cricket for years...and to poll the entire network using Cricket required 8+ minutes with each poll. This was even after a complete re-write of Cricket to make it use threads.

This plugin and whole system is great!

Thanks!

[TheWitness]

08/04/2007 07:53:18 PM - SYSTEM STATS: Time:197.6914 Method:cactid Processes:4 Threads:20 Hosts:566 HostsPerProcess:142 DataSources:99795 RRDsProcessed:0
08/04/2007 07:57:48 PM - SYSTEM STATS: Time:167.7309 Method:cactid Processes:4 Threads:20 Hosts:566 HostsPerProcess:142 DataSources:99826 RRDsProcessed:0
08/04/2007 08:03:56 PM - SYSTEM STATS: Time:235.2068 Method:cactid Processes:4 Threads:20 Hosts:566 HostsPerProcess:142 DataSources:99858 RRDsProcessed:0
08/04/2007 08:04:25 PM - SYSTEM BOOST STATS: Time:684.8934 RRDUpdates:290910
08/04/2007 08:07:53 PM - SYSTEM STATS: Time:172.7117 Method:cactid Processes:4 Threads:20 Hosts:566 HostsPerProcess:142 DataSources:99885 RRDsProcessed:0
08/04/2007 08:13:58 PM - SYSTEM STATS: Time:238.2603 Method:cactid Processes:4 Threads:20 Hosts:566 HostsPerProcess:142 DataSources:99912 RRDsProcessed:0
08/04/2007 08:17:32 PM - SYSTEM BOOST STATS: Time:751.7127 RRDUpdates:194710
08/04/2007 08:18:20 PM - SYSTEM STATS: Time:199.9822 Method:cactid Processes:4 Threads:20 Hosts:567 HostsPerProcess:142 DataSources:100407 RRDsProcessed:0
08/04/2007 08:23:06 PM - SYSTEM STATS: Time:185.9694 Method:cactid Processes:4 Threads:20 Hosts:567 HostsPerProcess:142 DataSources:100941 RRDsProcessed:0
08/04/2007 08:27:34 PM - SYSTEM STATS: Time:154.2108 Method:cactid Processes:4 Threads:20 Hosts:567 HostsPerProcess:142 DataSources:100941 RRDsProcessed:0
08/04/2007 08:28:25 PM - SYSTEM BOOST STATS: Time:624.8719 RRDUpdates:293256
08/04/2007 08:32:58 PM - SYSTEM STATS: Time:177.4861 Method:cactid Processes:4 Threads:20 Hosts:567 HostsPerProcess:142 DataSources:100941 RRDsProcessed:0
08/04/2007 08:37:59 PM - SYSTEM STATS: Time:178.8470 Method:cactid Processes:4 Threads:20 Hosts:567 HostsPerProcess:142 DataSources:100941 RRDsProcessed:0
08/04/2007 08:38:31 PM - SYSTEM BOOST STATS: Time:571.0101 RRDUpdates:196906
08/04/2007 08:43:09 PM - SYSTEM STATS: Time:188.7387 Method:cactid Processes:4 Threads:20 Hosts:567 HostsPerProcess:142 DataSources:100941 RRDsProcessed:0
08/04/2007 08:47:44 PM - SYSTEM STATS: Time:163.8680 Method:cactid Processes:4 Threads:20 Hosts:567 HostsPerProcess:142 DataSources:100943 RRDsProcessed:0

After messing with my hardware (note, 15k RPM drives are a must with many RRDs to update) I am now able to poll my entire network in less than 3 minutes, and also update all the RRD's. I've been running Cricket for years...and to poll the entire network using Cricket required 8+ minutes with each poll. This was even after a complete re-write of Cricket to make it use threads.

This plugin and whole system is great!

Thanks!

A note working with another customer late last week.

1. Bump your PHP memory (php.ini -> memory_limit) up to match your memory size table (assuming you are using memory tables for poller_output_boost).
2. Move your boost update interval to about an hour or so.
3. Move your record limit up to be effectively an hour at a time (2+m records)
4. If all you are using for in input method is snmp and your maximum poller_output field length is less than 50, reduce the column width on the poller_output_boot table to 50. This will allow some 8-10x the number of records to be held in your poller_output_boost table.
5. If using memory tables, make sure you have enough memory to handle all 2+m records. Maybe "max_heap_table=1G" or so.

My observation is that you are doing updates too often. These steps will reduce your load average.

TheWitness

[gandalf]

1. Bump your PHP memory (php.ini -> memory_limit) up to match your memory size table (assuming you are using memory tables for poller_output_boost).

Larry, is it possible to read out the MySQL memory size table and dynamically change the php.ini's memory setting? Just to avoid such issues?
Reinhard

[Frizz]

A note working with another customer late last week.

1. Bump your PHP memory (php.ini -> memory_limit) up to match your memory size table (assuming you are using memory tables for poller_output_boost).
2. Move your boost update interval to about an hour or so.
3. Move your record limit up to be effectively an hour at a time (2+m records)
4. If all you are using for in input method is snmp and your maximum poller_output field length is less than 50, reduce the column width on the poller_output_boot table to 50. This will allow some 8-10x the number of records to be held in your poller_output_boost table.
5. If using memory tables, make sure you have enough memory to handle all 2+m records. Maybe "max_heap_table=1G" or so.

My observation is that you are doing updates too often. These steps will reduce your load average.

TheWitness

Hi Larry,
thanks for very interisting letter of recommendation. I'm busy oneself with similar considerations. We're currently moving to new productive hardware, as we come up to the polling cyle limit .

• 08/01/2007 10:28:25 PM - SYSTEM STATS: Time:204.9392 Method:cactid Processes:6 Threads:30 Hosts:1940 HostsPerProcess:324 DataSources:73158 RRDsProcessed:36870
  08/01/2007 10:23:29 PM - SYSTEM STATS: Time:207.7885 Method:cactid Processes:6 Threads:30 Hosts:1940 HostsPerProcess:324 DataSources:73158 RRDsProcessed:36938

More HW details here http://forums.cacti.net/viewtopic.php?p ... ght=#51346

So we curently have a ratio of about 350 DS/sec. Compared with your stats you are at 500 DS/sec. I suppose it's mostly SNMP processing.

So my considerations are.

1. Where is the limit of DS/RRDs per box within the default polling cycle of 300 sec.
Is it theoretically limited at 150k DS/300sec ??
What are the key influence factors: I/O RAM, database RAM, PHP memory, CPUs, NFS mount performance, remote SQL performance ??

2. Where is the best balance between these 2 designs:
a.) Polling and database on a single box(8 core IBM445, 8GB RAM) -\- httpd and other plugins on the second box (equal HW)
b.) Only Polling on the single box (most memory for best I/O) -\- database and all other processes on the second box. With boost on the polling box and a high memory limit on the database box.

3. Will it be possible to build a determination matrix between all HW and SW influence factors and DS/RRDs per second?

Any further designs ideas and performance hints are desired.

Regards
Frizz

[TheWitness]

1. Bump your PHP memory (php.ini -> memory_limit) up to match your memory size table (assuming you are using memory tables for poller_output_boost).

Larry, is it possible to read out the MySQL memory size table and dynamically change the php.ini's memory setting? Just to avoid such issues?
Reinhard

Yes. Good idea

[TheWitness]

A note working with another customer late last week.

1. Bump your PHP memory (php.ini -> memory_limit) up to match your memory size table (assuming you are using memory tables for poller_output_boost).
2. Move your boost update interval to about an hour or so.
3. Move your record limit up to be effectively an hour at a time (2+m records)
4. If all you are using for in input method is snmp and your maximum poller_output field length is less than 50, reduce the column width on the poller_output_boot table to 50. This will allow some 8-10x the number of records to be held in your poller_output_boost table.
5. If using memory tables, make sure you have enough memory to handle all 2+m records. Maybe "max_heap_table=1G" or so.

My observation is that you are doing updates too often. These steps will reduce your load average.

TheWitness

Hi Larry,
thanks for very interisting letter of recommendation. I'm busy oneself with similar considerations. We're currently moving to new productive hardware, as we come up to the polling cyle limit .

• 08/01/2007 10:28:25 PM - SYSTEM STATS: Time:204.9392 Method:cactid Processes:6 Threads:30 Hosts:1940 HostsPerProcess:324 DataSources:73158 RRDsProcessed:36870
  08/01/2007 10:23:29 PM - SYSTEM STATS: Time:207.7885 Method:cactid Processes:6 Threads:30 Hosts:1940 HostsPerProcess:324 DataSources:73158 RRDsProcessed:36938

More HW details here http://forums.cacti.net/viewtopic.php?p ... ght=#51346

So we curently have a ratio of about 350 DS/sec. Compared with your stats you are at 500 DS/sec. I suppose it's mostly SNMP processing.

So my considerations are.

1. Where is the limit of DS/RRDs per box within the default polling cycle of 300 sec.
Is it theoretically limited at 150k DS/300sec ??
What are the key influence factors: I/O RAM, database RAM, PHP memory, CPUs, NFS mount performance, remote SQL performance ??

2. Where is the best balance between these 2 designs:
a.) Polling and database on a single box(8 core IBM445, 8GB RAM) -\- httpd and other plugins on the second box (equal HW)
b.) Only Polling on the single box (most memory for best I/O) -\- database and all other processes on the second box. With boost on the polling box and a high memory limit on the database box.

3. Will it be possible to build a determination matrix between all HW and SW influence factors and DS/RRDs per second?

Any further designs ideas and performance hints are desired.

Regards
Frizz

Frizz,

Tobi has been performing some tests with RRDtool 1.3 and what he has found is that with the new design, he is able to get a 3 to 5 x performance increase during updates. So, I would definately add that to the list.

However, the largest factor is MEMORY and of course the speed of your disks. So, my "best" recommendation is that on the box that actually has the RRDfiles, you need the fastest disks possible. I have been recommending RAID10 with 6 or more disks in a striped arrangement and from anywhere to 8-32GBytes of RAM (Cheap relatively speaking now).

Then, for the largest installs, the database always goes on a separate box with memory requirements requisite on the size of the installation and the use of the Database server. Is it dual use? If not, then it only needs enough memory for the key cache (MyISAM) and your memory tabkes (if any). If not using Memory tables, you need enough disk cache (MyISAM) to keep the various tables in memory. Of course they should be close to one another, latency wise, to increase performance.

Regards,

Larry

[TheWitness]

Frizz,

One more note on DS/Sec. The next release of Cactid, is much more aggressive at collection from multiple hosts. I have designed it, latency aside, to launch 10k collection threads per minute.

Larry

[gandalf]

I have designed it, latency aside, to launch 10k collection threads per minute.

Somebody would call this a DOS attack, won't you?
Reinhard

[TheWitness]

Assuming you have Cacti in the core and not out on the edge, it should only be considered efficient. In cases where you have concerns about crushing your network with snmp, I would not be too concerned as it is not a chatty protocol.

However, with 26 concurrent bulkwalk sessions on a Windows box, I was able to achieve 2.5mbps of network utilization. Not much these days.

TeWitness

[gandalf]

Assuming you have Cacti in the core and not out on the edge, it should only be considered efficient.

Hey, this was a joke! I'm REALLY impressed about the constant improvements on cactid. Each time I suppose, the current version is as fast as lightning and the ultimate performance is reached, you're setting the limits higher.
Reinhard