Submit Your CMD.PHP vs. SPINE Metrics Here

[melchandra]

Amen - thanks! :)

oops, think it should be: grep -rHn <pattern> *

HAHA, you're entirely right. Oops.

[knobdy]

I thought I posted this already (and even admitted to my idiotic misconfiguration of cactid the last time I tried to convert), but if I did it must have been the wrong forum area. Having run with cactid over night, here are the current results:

Code: Select all

04/25/2006 09:05:19 AM - SYSTEM STATS: Time:16.9793 Method:cactid Processes:2 Threads:8 Hosts:59 HostsPerProcess:30 DataSources:5309 RRDsProcessed:1453
04/25/2006 09:10:19 AM - SYSTEM STATS: Time:17.9754 Method:cactid Processes:2 Threads:8 Hosts:59 HostsPerProcess:30 DataSources:5309 RRDsProcessed:1510
04/25/2006 09:15:20 AM - SYSTEM STATS: Time:19.0551 Method:cactid Processes:2 Threads:8 Hosts:59 HostsPerProcess:30 DataSources:5309 RRDsProcessed:1569
04/25/2006 09:20:21 AM - SYSTEM STATS: Time:20.1951 Method:cactid Processes:2 Threads:8 Hosts:59 HostsPerProcess:30 DataSources:5309 RRDsProcessed:1569
04/25/2006 09:25:19 AM - SYSTEM STATS: Time:17.8080 Method:cactid Processes:2 Threads:8 Hosts:59 HostsPerProcess:30 DataSources:5309 RRDsProcessed:1525
04/25/2006 09:30:20 AM - SYSTEM STATS: Time:18.3497 Method:cactid Processes:2 Threads:8 Hosts:59 HostsPerProcess:30 DataSources:5309 RRDsProcessed:1525
04/25/2006 09:35:22 AM - SYSTEM STATS: Time:20.5984 Method:cactid Processes:2 Threads:8 Hosts:59 HostsPerProcess:30 DataSources:5309 RRDsProcessed:1525
04/25/2006 09:40:21 AM - SYSTEM STATS: Time:20.2469 Method:cactid Processes:2 Threads:8 Hosts:59 HostsPerProcess:30 DataSources:5309 RRDsProcessed:1554
04/25/2006 09:45:20 AM - SYSTEM STATS: Time:19.1965 Method:cactid Processes:2 Threads:8 Hosts:59 HostsPerProcess:30 DataSources:5309 RRDsProcessed:1512
04/25/2006 09:50:19 AM - SYSTEM STATS: Time:18.1279 Method:cactid Processes:2 Threads:8 Hosts:59 HostsPerProcess:30 DataSources:5309 RRDsProcessed:1558
04/25/2006 09:55:20 AM - SYSTEM STATS: Time:18.9790 Method:cactid Processes:2 Threads:8 Hosts:59 HostsPerProcess:30 DataSources:5309 RRDsProcessed:1554

How am I doing?

Here are some just prior to and after the change-over:

Code: Select all

04/24/2006 12:04:53 AM - SYSTEM MACTRACK STATS: Time:227.1363 ConcurrentProcesses:20 Devices:35
04/24/2006 04:01:00 PM - SYSTEM STATS: Time:57.8296 Method:cmd.php Processes:12 Threads:N/A Hosts:59 HostsPerProcess:5 DataSources:5309 RRDsProcessed:2638
04/24/2006 04:06:02 PM - SYSTEM STATS: Time:59.8644 Method:cmd.php Processes:12 Threads:N/A Hosts:59 HostsPerProcess:5 DataSources:5309 RRDsProcessed:2638
04/24/2006 04:11:03 PM - SYSTEM STATS: Time:61.5958 Method:cmd.php Processes:12 Threads:N/A Hosts:59 HostsPerProcess:5 DataSources:5309 RRDsProcessed:2638
04/24/2006 04:15:26 PM - SYSTEM STATS: Time:24.8374 Method:cactid Processes:12 Threads:8 Hosts:59 HostsPerProcess:5 DataSources:5309 RRDsProcessed:1569
04/24/2006 04:20:27 PM - SYSTEM STATS: Time:25.8656 Method:cactid Processes:12 Threads:8 Hosts:59 HostsPerProcess:5 DataSources:5309 RRDsProcessed:1569
04/19/2006 05:54:47 AM - SYSTEM MACTRACK STATS: Time:229.7355 ConcurrentProcesses:20 Devices:35

Oops, looking at this, it would seem I've got a problem still - over a thousand problems actually... http://forums.cacti.net/viewtopic.php?p=64338#64338

[KaiB]

My results:

04/26/2006 12:01:28 PM - SYSTEM STATS: Time:27.8974 Method:cactid Processes:2 Threads:2 Hosts:65 HostsPerProcess:33 DataSources:196 RRDsProcessed:148
04/26/2006 12:00:26 PM - SYSTEM STATS: Time:25.6139 Method:cactid Processes:2 Threads:2 Hosts:65 HostsPerProcess:33 DataSources:196 RRDsProcessed:148
04/26/2006 11:59:29 AM - SYSTEM STATS: Time:29.1076 Method:cactid Processes:2 Threads:2 Hosts:65 HostsPerProcess:33 DataSources:196 RRDsProcessed:148
04/26/2006 11:58:27 AM - SYSTEM STATS: Time:26.3604 Method:cactid Processes:2 Threads:2 Hosts:65 HostsPerProcess:33 DataSources:196 RRDsProcessed:148
04/26/2006 11:57:26 AM - SYSTEM STATS: Time:26.1172 Method:cactid Processes:2 Threads:2 Hosts:65 HostsPerProcess:33 DataSources:196 RRDsProcessed:148

04/26/2006 11:07:40 AM - SYSTEM STATS: Time:39.0367 Method:cmd.php Processes:2 Threads:N/A Hosts:65 HostsPerProcess:33 DataSources:196 RRDsProcessed:148
04/26/2006 11:06:43 AM - SYSTEM STATS: Time:42.6524 Method:cmd.php Processes:2 Threads:N/A Hosts:65 HostsPerProcess:33 DataSources:196 RRDsProcessed:148
04/26/2006 11:05:34 AM - SYSTEM STATS: Time:33.5517 Method:cmd.php Processes:2 Threads:N/A Hosts:65 HostsPerProcess:33 DataSources:196 RRDsProcessed:148
04/26/2006 11:04:39 AM - SYSTEM STATS: Time:38.4495 Method:cmd.php Processes:2 Threads:N/A Hosts:65 HostsPerProcess:33 DataSources:196 RRDsProcessed:148
04/26/2006 11:03:35 AM - SYSTEM STATS: Time:35.2310 Method:cmd.php Processes:2 Threads:N/A Hosts:65 HostsPerProcess:33 DataSources:196 RRDsProcessed:148

System:
2003 Server SP1
Dell PowerEdge SC 420
2.53GHz P4 Celeron
1GB RAM

[gandalf]

My results:

04/26/2006 12:01:28 PM - SYSTEM STATS: Time:27.8974 Method:cactid Processes:2 Threads:2 Hosts:65 HostsPerProcess:33 DataSources:196 RRDsProcessed:148
04/26/2006 12:00:26 PM - SYSTEM STATS: Time:25.6139 Method:cactid Processes:2 Threads:2 Hosts:65 HostsPerProcess:33 DataSources:196 RRDsProcessed:148
04/26/2006 11:59:29 AM - SYSTEM STATS: Time:29.1076 Method:cactid Processes:2 Threads:2 Hosts:65 HostsPerProcess:33 DataSources:196 RRDsProcessed:148
04/26/2006 11:58:27 AM - SYSTEM STATS: Time:26.3604 Method:cactid Processes:2 Threads:2 Hosts:65 HostsPerProcess:33 DataSources:196 RRDsProcessed:148
04/26/2006 11:57:26 AM - SYSTEM STATS: Time:26.1172 Method:cactid Processes:2 Threads:2 Hosts:65 HostsPerProcess:33 DataSources:196 RRDsProcessed:148

System:
2003 Server SP1
Dell PowerEdge SC 420
2.53GHz P4 Celeron
1GB RAM

For such a decent machine, this seems to be slow (for me). Are you running lots of scripts?
Reinhard

[TheWitness]

He needs to kick up the threads to somewhere from between 15-20.

Larry

[KaiB]

I did try to increase the threads to 3, it got a couple of seconds faster but i experienced result loss for some reason (gaps in graphs to stable hosts). Not sure if something else interupted the polling at the same time. Il give it another go today. Woud be nice if I coud get it faster as the polling cycles overlap from the stress of the nightly backup.

When it comes to scripts, I run the Advanced Ping to almost all hosts and a couple of others to special hosts.

[KaiB]

These where the highest stable results I achived.

04/28/2006 01:07:26 PM - SYSTEM STATS: Time:25.5384 Method:cactid Processes:4 Threads:3 Hosts:65 HostsPerProcess:17 DataSources:205 RRDsProcessed:154
04/28/2006 01:06:26 PM - SYSTEM STATS: Time:25.5731 Method:cactid Processes:4 Threads:3 Hosts:65 HostsPerProcess:17 DataSources:205 RRDsProcessed:154
04/28/2006 01:05:25 PM - SYSTEM STATS: Time:25.0394 Method:cactid Processes:4 Threads:3 Hosts:65 HostsPerProcess:17 DataSources:205 RRDsProcessed:154
04/28/2006 01:04:25 PM - SYSTEM STATS: Time:25.0031 Method:cactid Processes:4 Threads:3 Hosts:65 HostsPerProcess:17 DataSources:205 RRDsProcessed:154
04/28/2006 01:03:26 PM - SYSTEM STATS: Time:25.6107 Method:cactid Processes:4 Threads:3 Hosts:65 HostsPerProcess:17 DataSources:205 RRDsProcessed:154

More threads then this with any combination with processes resulted in cactid missbehaving. (POPEN timeouts and stable hosts marked as down.)

I can PM the log file if there is interest.

[TheWitness]

That is very interesting. What is your os and libc version? It appears that you are running many scripts. What type of scripts are they? If you have large quantities of scripts, you will have to watch load.

You can run 1 script per thread per process, so if you are running a perl environment that has a lot of overhead, it could cause timing issues as everything has to reload per script. Also, if you have concurrent processes to 3 and threads to 17, you could conceivably have 51 jobs in the run queue all attempting to reach a binary at the same time.

I am thinking that maybe I should limit the number of concurrent scripts to some level. Comments from long time listeners would be welcome.

This is the primary reason for the script server. In PHP5, I can load perl modules and run perl scripts much faster from PHP, without all the overhead, but it has never been a priority for me.

Thanks,

TheWitness

[gandalf]

I can PM the log file if there is interest.

More to check pollers performance to be found at
http://forums.cacti.net/viewtopic.php?t=9272 and
http://forums.cacti.net/viewtopic.php?t=12369
Reinhard

[KaiB]

Sorry for the late reply, been very busy at work lately.

What is your os and libc version? It appears that you are running many scripts. What type of scripts are they? If you have large quantities of scripts, you will have to watch load.

The OS is 2003 SP1
Got no idea how check the "libc" version number

Atm im running about 50 php scripts and 10 perl scripts and when it comes to load its 100% while the poller runs.

More to check pollers performance to be found at

Got the poller graph installed but not the performance debug log analyzer, didnt get it to work a while ago, dont remember why.

[TheWitness]

Interesting how it climbs slowly and then drops again. Could that be at about the same time as a reboot. Do you have any correlation to the increased numbers and something going on on the system.

Have you looked at open TCP ports or concurrent logins? There are scripts for both in the scripts section for those.

TheWitness

[gandalf]

Is it possible, that there are some scripts running on e.g. some log files that increase in size and are e.g. logrotated every day?
Reinhard

[cdukes]

Hey guys,
Just trying to make this as efficient as possible, here are a couple samples...do you have any tuning recommendations for me?

Code: Select all

05/10/2006 08:37:23 PM - SYSTEM STATS: Time:440.0827 Method:cactid Processes:1 Threads:15 Hosts:774 HostsPerProcess:774 DataSources:5211 RRDsProcessed:3708

Code: Select all

05/10/2006 08:39:59 PM - SYSTEM STATS: Time:296.9150 Method:cactid Processes:3 Threads:20 Hosts:774 HostsPerProcess:258 DataSources:5649 RRDsProcessed:4023

Code: Select all

5/10/2006 08:42:55 PM - SYSTEM STATS: Time:171.9149 Method:cactid Processes:3 Threads:20 Hosts:774 HostsPerProcess:258 DataSources:5650 RRDsProcessed:4024

Thanks!

[cdukes]

Seems to have gone down considerably now that I changed to 3 processes...

Code: Select all

Time:59.5912 Method:cactid Processes:3 Threads:20 Hosts:774 HostsPerProcess:258 DataSources:5650 RRDsProcessed:4024

[gandalf]

Recommendation is to choose about 2* (numbers of CPUs) as a starting point for #processes. Threads in the low 2-digit area should do.
Reinhard