Submit Your CMD.PHP vs. SPINE Metrics Here

[gandalf]

Per theory, it should be set to 60 if all targets are working fine with that setting. It surely will reduce network load at minimum. If it also will reduce polling time will be clear if you add more devices.
Reinhard

[ben_c]

New install of cacti/cactid

10/15/2007 02:10:32 PM - SYSTEM STATS: Time:360.5152 Method:cactid Processes:2 Threads:5 Hosts:170 HostsPerProcess:85 DataSources:30912 RRDsProcessed:15748

Not getting the performance I had hoped, and I still haven't migrated all hosts over to it

IBM x346 Xeon 3.4Ghz
4Gb RAM
1 x raided 10k SCSI HDD's
Redhat AS 4

[gandalf]

IS this a dual CPU system? Which version of rrdtool used? We have about 7ß% of your numbers ds/rrd and require 40 sec polling time on a dual XEON. Please increase threads to 12-15. Most of the ds are SNMP?
Reinhard

[ben_c]

Hi Reinhard,

Yeah dual Xeon.

I've had some issues with the latest MySQL that was placing a lot of load on the system, i've done a kernel upgrade and all seems well.

[hylaride]

10/17/2007 01:26:35 PM - SYSTEM STATS: Time:34.7723 Method:cactid Processes:4 Threads:14 Hosts:229 HostsPerProcess:58 DataSources:4604 RRDsProcessed:2901

This is from 4-5 minutes from poller.php


Attached is the image as it went to cactid (the huge spike at the beginning). Since cacti is running of this machine, it's always running a load when checking itself I had to explain that to our nitwits in the NOC that the other half of the time the computer is idle...

[antifaction]

Didn't get much better. I'm lookign through my config now to check settings.

12/11/2007 01:10:16 PM - SYSTEM STATS: Time:14.1418 Method:spine Processes:16 Threads:4 Hosts:52 HostsPerProcess:4 DataSources:1950 RRDsProcessed:1279
12/11/2007 01:05:14 PM - SYSTEM STATS: Time:13.0296 Method:spine Processes:16 Threads:4 Hosts:52 HostsPerProcess:4 DataSources:1950 RRDsProcessed:1279
12/11/2007 01:00:14 PM - SYSTEM STATS: Time:13.2933 Method:spine Processes:16 Threads:4 Hosts:52 HostsPerProcess:4 DataSources:1950 RRDsProcessed:1279
12/11/2007 12:50:15 PM - SYSTEM STATS: Time:14.1304 Method:spine Processes:16 Threads:1 Hosts:52 HostsPerProcess:4 DataSources:1950 RRDsProcessed:1279


12/11/2007 12:40:16 PM - SYSTEM STATS: Time:15.3111 Method:cmd.php Processes:16 Threads:N/A Hosts:55 HostsPerProcess:4 DataSources:1950 RRDsProcessed:1270
12/11/2007 12:35:17 PM - SYSTEM STATS: Time:16.4388 Method:cmd.php Processes:16 Threads:N/A Hosts:55 HostsPerProcess:4 DataSources:1950 RRDsProcessed:1270
12/11/2007 12:30:17 PM - SYSTEM STATS: Time:15.7749 Method:cmd.php Processes:16 Threads:N/A Hosts:55 HostsPerProcess:4 DataSources:1950 RRDsProcessed:1270

[briang]

I'm currently seeing exceptional performance for a large windows based cacti install using cacti 0.8.7a/spine 0.8.7b/boost. Just wanted to post my configuration to hopefully save others the time and effort that it took me to get this right....

Current stats:

12/17/2007 09:44:23 AM - SYSTEM STATS: Time:23.6648 Method:spine Processes:1 Threads:8 Hosts:265 HostsPerProcess:265 DataSources:13930 RRDsProcessed:0

12/17/2007 09:44:42 AM - SYSTEM BOOST STATS: Time:282.2263 RRDUpdates:273150

My poller is running at a 1 minute interval with an RRA that keeps 30 days of granular data. This results in RRD files that are fairly large ~35 MB. Part of my initial performance problems were due to the amount of time needed to actually update the RRDs - about 80% of the polling cycle. All of my data is snmp (no scripts). The boost plugin addresssed the length of the update cycle.

My hardware specs:
CPU 2 x dual core - Xeon @ 2.33ghz; 4 GB RAM OS is w2k03

I'm using the latest version of boost from the svn as I was seeing a known issue with "Exceeded Run time limits" errors in my cacti.log using 1.7.

Here's the config that worked for me:
I have two scheduled tasks - standard one for the poller.php (5 min interval) and one for the boost (boost_poller.php -f -d) at a 20 min interval.

My poller settings:
Poller Type - spine
Poller Interval - Every Minute
Cron Interval - Every 5 minutes
Maximum Concurrent Poller Processes - 1
Maximum Threads Per Process - 8
Number of PHP Script Servers - 10
Script Server Timeout Value - 30
Maximum SNMP OIDs per SNMP get request - 60
Downed Host Detection - SNMP
(Important that Downed Host Detection is SNMP when definining an individual device as well.)

My.ini customizations (needed to allocate sufficient memory for boost.)
I used the mysql-huge.ini from the standard distribution with the following additional updates:
max_connections=250
max_user_connections=230
wait_timeout=600
max_heap_table_size=1500000000
tmp_table_size=1500000000
key_buffer_size=12000000

I also allocated more memory to PHP.
memory_limit = 384M

boost settings:
Enabled on Demand RRD Updating
How Often Should Boost Update All RRDs: 6 Hours
Maximum Records: 1000000
Maximum Records Per Pass: 100,000
Maximum MySQL insert string: 100000
Update String Length: 8100
Maximum RRD Update Script Run Time: 1 Hour

[koolaidrocks]

Here are the stats for my Cacti test box:

• Cacti 0.8.7a and Plugin Arch 1.4 w/ Monitor, MacTrack, Discovery, Weathermap and Thold

All of this runs on an old Compaq Deskpro EN sporting a 1GHz Pentium III w/ 512MB of RAM running Fedora Core 7. Needless to say, I was quite surprised with the results:

• 12/31/2007 10:37:18 AM - SYSTEM STATS: Time:16.3401 Method:spine Processes:1 Threads:5 Hosts:101 HostsPerProcess:101 DataSources:5352 RRDsProcessed:2823

Currently, only our switch infrastructure is added into Cacti. But the wireless and WAN devices will be added in soon enough.

[keefe007]

PHP 5.2.5
Cacti 0.8.7a

AMD Athlon 2800+
2 GB Ram


01/04/2008 12:22:14 AM - RECACHE STATS: RecacheTime:10.2970 HostsRecached:1
01/04/2008 12:22:03 AM - SYSTEM STATS: Time:2.3959 Method:spine Processes:2 Threads:15 Hosts:42 HostsPerProcess:21 DataSources:58 RRDsProcessed:31
01/04/2008 12:21:50 AM - SYSTEM STATS: Time:48.0235 Method:spine Processes:2 Threads:15 Hosts:42 HostsPerProcess:21 DataSources:638 RRDsProcessed:383
01/04/2008 12:20:05 AM - SYSTEM STATS: Time:4.3200 Method:spine Processes:2 Threads:15 Hosts:42 HostsPerProcess:21 DataSources:41 RRDsProcessed:20
01/04/2008 12:19:04 AM - SYSTEM STATS: Time:2.3350 Method:spine Processes:2 Threads:15 Hosts:42 HostsPerProcess:21 DataSources:41 RRDsProcessed:20
01/04/2008 12:18:07 AM - SYSTEM STATS: Time:6.3048 Method:spine Processes:2 Threads:15 Hosts:42 HostsPerProcess:21 DataSources:41 RRDsProcessed:20
01/04/2008 12:17:14 AM - RECACHE STATS: RecacheTime:10.0997 HostsRecached:1
01/04/2008 12:17:04 AM - SYSTEM STATS: Time:2.3558 Method:spine Processes:2 Threads:15 Hosts:42 HostsPerProcess:21 DataSources:58 RRDsProcessed:31
01/04/2008 12:16:48 AM - SYSTEM STATS: Time:47.2248 Method:spine Processes:2 Threads:15 Hosts:42 HostsPerProcess:21 DataSources:638 RRDsProcessed:383
01/04/2008 12:15:04 AM - SYSTEM STATS: Time:2.3276 Method:spine Processes:2 Threads:15 Hosts:42 HostsPerProcess:21 DataSources:41 RRDsProcessed:20
01/04/2008 12:11:04 AM - SYSTEM STATS: Time:2.4391 Method:spine Processes:2 Threads:15 Hosts:42 HostsPerProcess:21 DataSources:41 RRDsProcessed:20
01/04/2008 12:10:08 AM - SYSTEM STATS: Time:7.7261 Method:spine Processes:2 Threads:15 Hosts:42 HostsPerProcess:21 DataSources:41 RRDsProcessed:20
01/04/2008 12:09:03 AM - SYSTEM STATS: Time:2.2893 Method:spine Processes:2 Threads:15 Hosts:42 HostsPerProcess:21 DataSources:58 RRDsProcessed:31
01/04/2008 12:08:50 AM - SYSTEM STATS: Time:48.9131 Method:spine Processes:2 Threads:15 Hosts:42 HostsPerProcess:21 DataSources:638 RRDsProcessed:383

How are these stats?

[koolaidrocks]

PHP 5.2.5
Cacti 0.8.7a

AMD Athlon 2800+
2 GB Ram


01/04/2008 12:22:14 AM - RECACHE STATS: RecacheTime:10.2970 HostsRecached:1
01/04/2008 12:22:03 AM - SYSTEM STATS: Time:2.3959 Method:spine Processes:2 Threads:15 Hosts:42 HostsPerProcess:21 DataSources:58 RRDsProcessed:31
01/04/2008 12:21:50 AM - SYSTEM STATS: Time:48.0235 Method:spine Processes:2 Threads:15 Hosts:42 HostsPerProcess:21 DataSources:638 RRDsProcessed:383
01/04/2008 12:20:05 AM - SYSTEM STATS: Time:4.3200 Method:spine Processes:2 Threads:15 Hosts:42 HostsPerProcess:21 DataSources:41 RRDsProcessed:20
01/04/2008 12:19:04 AM - SYSTEM STATS: Time:2.3350 Method:spine Processes:2 Threads:15 Hosts:42 HostsPerProcess:21 DataSources:41 RRDsProcessed:20
01/04/2008 12:18:07 AM - SYSTEM STATS: Time:6.3048 Method:spine Processes:2 Threads:15 Hosts:42 HostsPerProcess:21 DataSources:41 RRDsProcessed:20
01/04/2008 12:17:14 AM - RECACHE STATS: RecacheTime:10.0997 HostsRecached:1
01/04/2008 12:17:04 AM - SYSTEM STATS: Time:2.3558 Method:spine Processes:2 Threads:15 Hosts:42 HostsPerProcess:21 DataSources:58 RRDsProcessed:31
01/04/2008 12:16:48 AM - SYSTEM STATS: Time:47.2248 Method:spine Processes:2 Threads:15 Hosts:42 HostsPerProcess:21 DataSources:638 RRDsProcessed:383
01/04/2008 12:15:04 AM - SYSTEM STATS: Time:2.3276 Method:spine Processes:2 Threads:15 Hosts:42 HostsPerProcess:21 DataSources:41 RRDsProcessed:20
01/04/2008 12:11:04 AM - SYSTEM STATS: Time:2.4391 Method:spine Processes:2 Threads:15 Hosts:42 HostsPerProcess:21 DataSources:41 RRDsProcessed:20
01/04/2008 12:10:08 AM - SYSTEM STATS: Time:7.7261 Method:spine Processes:2 Threads:15 Hosts:42 HostsPerProcess:21 DataSources:41 RRDsProcessed:20
01/04/2008 12:09:03 AM - SYSTEM STATS: Time:2.2893 Method:spine Processes:2 Threads:15 Hosts:42 HostsPerProcess:21 DataSources:58 RRDsProcessed:31
01/04/2008 12:08:50 AM - SYSTEM STATS: Time:48.9131 Method:spine Processes:2 Threads:15 Hosts:42 HostsPerProcess:21 DataSources:638 RRDsProcessed:383

How are these stats?

Well compared to my crappy test box...not very good. What's your memory_limit in php.ini set to?

[gandalf]

I won't bother with that few data sources. Most of the runtime will be dedicated to startup issues, I suppose
Reinhard

[antifaction]

02/11/2008 11:15:16 AM - SYSTEM STATS: Time:14.6410 Method:spine Processes:6 Threads:7 Hosts:154 HostsPerProcess:26 DataSources:5030 RRDsProcessed:3024
02/11/2008 11:10:14 AM - SYSTEM STATS: Time:13.6882 Method:spine Processes:6 Threads:7 Hosts:154 HostsPerProcess:26 DataSources:5030 RRDsProcessed:3024
02/11/2008 11:05:14 AM - SYSTEM STATS: Time:12.8746 Method:spine Processes:6 Threads:7 Hosts:154 HostsPerProcess:26 DataSources:5030 RRDsProcessed:3024
02/11/2008 11:00:16 AM - SYSTEM STATS: Time:14.6267 Method:spine Processes:6 Threads:7 Hosts:154 HostsPerProcess:26 DataSources:5030 RRDsProcessed:3024

Linux
0.8.7a
1GB memory

[bsdboy]

I started pushing the envelope a bit and my cacti box let me know today that it was time to look at something other then cmd.php. Seems to run much better, thanks!

Specs:
Cent/OS 4.4 on a Sun v60
dual-core hyper-threaded 2.8Ghz Xeon (4 cores total)
2G RAM

03/13/2008 01:20:02 PM - SYSTEM STATS: Time:299.8830 Method:cmd.php Processes:10 Threads:N/A Hosts:87 HostsPerProcess:9 DataSources:9033 RRDsProcessed:1794
03/13/2008 01:26:44 PM - SYSTEM STATS: Time:402.6165 Method:cmd.php Processes:10 Threads:N/A Hosts:84 HostsPerProcess:9 DataSources:8803 RRDsProcessed:7959
03/13/2008 01:29:47 PM - SYSTEM STATS: Time:286.7634 Method:cmd.php Processes:10 Threads:N/A Hosts:84 HostsPerProcess:9 DataSources:8803 RRDsProcessed:4115
03/13/2008 01:34:38 PM - SYSTEM STATS: Time:276.0497 Method:cmd.php Processes:10 Threads:N/A Hosts:84 HostsPerProcess:9 DataSources:8803 RRDsProcessed:4090
03/13/2008 01:39:43 PM - SYSTEM STATS: Time:282.4115 Method:cmd.php Processes:10 Threads:N/A Hosts:84 HostsPerProcess:9 DataSources:8803 RRDsProcessed:4175
03/13/2008 01:44:50 PM - SYSTEM STATS: Time:288.9465 Method:cmd.php Processes:10 Threads:N/A Hosts:84 HostsPerProcess:9 DataSources:8803 RRDsProcessed:4124
03/13/2008 01:49:49 PM - SYSTEM STATS: Time:287.3581 Method:cmd.php Processes:10 Threads:N/A Hosts:84 HostsPerProcess:9 DataSources:8803 RRDsProcessed:4133
03/13/2008 01:54:48 PM - SYSTEM STATS: Time:287.1003 Method:cmd.php Processes:10 Threads:N/A Hosts:84 HostsPerProcess:9 DataSources:8803 RRDsProcessed:4143
03/13/2008 01:59:38 PM - SYSTEM STATS: Time:276.7971 Method:cmd.php Processes:10 Threads:N/A Hosts:84 HostsPerProcess:9 DataSources:8803 RRDsProcessed:4036
03/13/2008 02:04:48 PM - SYSTEM STATS: Time:286.9911 Method:cmd.php Processes:10 Threads:N/A Hosts:84 HostsPerProcess:9 DataSources:8803 RRDsProcessed:4209
03/13/2008 02:09:43 PM - SYSTEM STATS: Time:281.8917 Method:cmd.php Processes:10 Threads:N/A Hosts:84 HostsPerProcess:9 DataSources:8803 RRDsProcessed:4005
03/13/2008 02:14:42 PM - SYSTEM STATS: Time:280.6889 Method:cmd.php Processes:10 Threads:N/A Hosts:84 HostsPerProcess:9 DataSources:8803 RRDsProcessed:4182
03/13/2008 02:19:49 PM - SYSTEM STATS: Time:287.8510 Method:cmd.php Processes:10 Threads:N/A Hosts:84 HostsPerProcess:9 DataSources:8803 RRDsProcessed:4196
03/13/2008 02:24:39 PM - SYSTEM STATS: Time:277.2017 Method:cmd.php Processes:10 Threads:N/A Hosts:84 HostsPerProcess:9 DataSources:8803 RRDsProcessed:4052
03/13/2008 02:29:54 PM - SYSTEM STATS: Time:292.4045 Method:cmd.php Processes:10 Threads:N/A Hosts:84 HostsPerProcess:9 DataSources:8803 RRDsProcessed:4203
03/13/2008 02:34:39 PM - SYSTEM STATS: Time:277.8951 Method:cmd.php Processes:10 Threads:N/A Hosts:84 HostsPerProcess:9 DataSources:8803 RRDsProcessed:4045
03/13/2008 02:39:50 PM - SYSTEM STATS: Time:288.5329 Method:cmd.php Processes:10 Threads:N/A Hosts:84 HostsPerProcess:9 DataSources:8803 RRDsProcessed:4202
03/13/2008 02:44:37 PM - SYSTEM STATS: Time:276.7351 Method:cmd.php Processes:10 Threads:N/A Hosts:84 HostsPerProcess:9 DataSources:8803 RRDsProcessed:4057
03/13/2008 02:49:40 PM - SYSTEM STATS: Time:279.7788 Method:cmd.php Processes:10 Threads:N/A Hosts:84 HostsPerProcess:9 DataSources:8803 RRDsProcessed:4143
03/13/2008 02:54:48 PM - SYSTEM STATS: Time:286.6913 Method:cmd.php Processes:10 Threads:N/A Hosts:84 HostsPerProcess:9 DataSources:8803 RRDsProcessed:4198
03/13/2008 02:59:40 PM - SYSTEM STATS: Time:278.5733 Method:cmd.php Processes:10 Threads:N/A Hosts:84 HostsPerProcess:9 DataSources:8803 RRDsProcessed:4069
03/13/2008 03:04:50 PM - SYSTEM STATS: Time:288.8808 Method:cmd.php Processes:10 Threads:N/A Hosts:84 HostsPerProcess:9 DataSources:8803 RRDsProcessed:4175
03/13/2008 03:06:13 PM - SYSTEM STATS: Time:72.1658 Method:cactid Processes:10 Threads:8 Hosts:84 HostsPerProcess:9 DataSources:8803 RRDsProcessed:4441
03/13/2008 03:11:03 PM - SYSTEM STATS: Time:61.4628 Method:cactid Processes:10 Threads:8 Hosts:84 HostsPerProcess:9 DataSources:8803 RRDsProcessed:4190
03/13/2008 03:16:00 PM - SYSTEM STATS: Time:58.9716 Method:cactid Processes:10 Threads:8 Hosts:84 HostsPerProcess:9 DataSources:8803 RRDsProcessed:4126

And here is current after I added the slower polls back in:

03/14/2008 11:06:02 AM - SYSTEM STATS: Time:60.6542 Method:cactid Processes:10 Threads:8 Hosts:101 HostsPerProcess:11 DataSources:13007 RRDsProcessed:5639
03/14/2008 11:11:05 AM - SYSTEM STATS: Time:63.2864 Method:cactid Processes:10 Threads:8 Hosts:101 HostsPerProcess:11 DataSources:13007 RRDsProcessed:5639
03/14/2008 11:16:02 AM - SYSTEM STATS: Time:61.6390 Method:cactid Processes:10 Threads:8 Hosts:101 HostsPerProcess:11 DataSources:13007 RRDsProcessed:5638
03/14/2008 11:21:07 AM - SYSTEM STATS: Time:65.9812 Method:cactid Processes:10 Threads:8 Hosts:101 HostsPerProcess:11 DataSources:13007 RRDsProcessed:5640

[gandalf]

I'd recommend to lower the proccesses a bit an double the threads instead. More memory would be fine (expect 30% reduction of the current runtime). Latest rrdtool (1.2.26 or later) along with a current 2.6.x kernel should reduce by another 30% or the like.
Reinhard

[bsdboy]

Went ahead and did some upgrades, still the same hardware but using 0.8.7b with spine 0.8.7a.

03/19/2008 10:25:36 PM - SYSTEM STATS: Time:34.4663 Method:spine Processes:8 Threads:10 Hosts:101 HostsPerProcess:13 DataSources:12691 RRDsProcessed:5475
03/19/2008 10:30:47 PM - SYSTEM STATS: Time:46.1244 Method:spine Processes:8 Threads:16 Hosts:101 HostsPerProcess:13 DataSources:12691 RRDsProcessed:5475
03/19/2008 10:35:35 PM - SYSTEM STATS: Time:34.1983 Method:spine Processes:8 Threads:16 Hosts:101 HostsPerProcess:13 DataSources:12691 RRDsProcessed:5475
03/19/2008 10:40:38 PM - SYSTEM STATS: Time:36.4121 Method:spine Processes:8 Threads:16 Hosts:101 HostsPerProcess:13 DataSources:12691 RRDsProcessed:5475