Poor performance in cactid

hylaride · Post by **hylaride** » Mon Sep 17, 2007 10:59 am

I'm on a Solaris 10 x86 machine and am trying to move to cactid, due to the large number of hosts we have. When running the regular poller with about 1/10 of our total hosts it takes a little under a minute:

09/17/2007 11:48:35 AM - SYSTEM STATS: Time:46.4145 Method:cmd.php Processes:20 Threads:N/A Hosts:87 HostsPerProcess:5 DataSources:2087 RRDsProcessed:1329

However, upon switching to cactid, the performance becomes abysmal:

# /usr/local/php/bin/php /usr/local/httpd/htdocs/cacti/poller.php

09/17/2007 11:46:49 AM - POLLER: Poller[0] Maximum runtime of 292 seconds exceeded. Exiting.
09/17/2007 11:46:49 AM - SYSTEM STATS: Time:292.9064 Method:cactid Processes:20 Threads:20 Hosts:87 HostsPerProcess:5 DataSources:2087 RRDsProcessed:0

Warning: pclose(): 65 is not a valid stream resource in /usr/local/httpd/htdocs/cacti-0.8.6j/lib/rrd.php on line 47

Can somebody help me shed some light onto this? There are some weird errors in the cacti log:

09/17/2007 11:56:37 AM - CMDPHP: Poller[0] Host[73] DS[947] WARNING: Result from SNMP not valid. Partial Result:
09/17/2007 11:56:38 AM - CMDPHP: Poller[0] Host[84] DS[1137] WARNING: Result from SNMP not valid. Partial Result:
09/17/2007 11:56:39 AM - CMDPHP: Poller[0] WARNING: sockets support not enabled in PHP, falling back to SNMP ping
09/17/2007 11:56:40 AM - CMDPHP: Poller[0] WARNING: sockets support not enabled in PHP, falling back to SNMP ping
09/17/2007 11:56:40 AM - CMDPHP: Poller[0] Host[89] DS[1192] WARNING: Result from SNMP not valid. Partial Result:
09/17/2007 11:56:41 AM - CMDPHP: Poller[0] Host[64] DS[864] WARNING: Result from SNMP not valid. Partial Result:

Exo7 · Post by **Exo7** » Mon Sep 17, 2007 11:19 am

20 processes and 20 threads is quite a lot for cactid.

Here are my stats on a system with similar numbers of datasources, but less hosts.

Code: Select all

SYSTEM STATS: Time:9.1990 Method:cactid Processes:2 Threads:2 Hosts:37 HostsPerProcess:19 DataSources:3722 RRDsProcessed:1418

That's from a dell 1950, with 2 dual core xeon 3ghz and 2Gb of ram.

Others stats are available on this topic.

hylaride · Post by **hylaride** » Mon Sep 17, 2007 11:25 am

Yes, this is because it was having such poor performance so I was playing around.

Exo7 · Post by **Exo7** » Mon Sep 17, 2007 11:31 am

Do you have a lot of scripts or non snmp based polling ?

I didn't saw the "Warning: pclose(): 65 is not a valid stream resource in /usr/local/httpd/htdocs/cacti-0.8.6j/lib/rrd.php on line 47" part, maybe your performance problem is lurking here

hylaride · Post by **hylaride** » Mon Sep 17, 2007 12:47 pm

No, just standard snmp querying....

Post by **gandalf** » Mon Sep 17, 2007 2:25 pm

Please make sure to have all patches applied. Use latest cactid. Start with #of_processes =1-2 times the number of CPU cores, threads = 10 .. 15
Reinhard

Post by **TheWitness** » Mon Sep 17, 2007 3:37 pm

Your system does not meet the minimum software requirements for PHP. Therefore, you are having problems. If you are running solaris, you will definately encounter problems with Cactid. You might try the beta cactid 0.8.6j in the Announcement's forum.

However, to make things best the following is advised:

1) PHP 4.3.6 or greater
2) The following PHP modules should be enabled (php -m):
sockets
snmp
mysql

TheWitness

hylaride · Post by **hylaride** » Mon Sep 17, 2007 3:53 pm

I'm using cacti cacti-0.8.6j and cactid 0.8.6i the latest php, apache, etc (this is a brand new box which is replacing ancient freebsd 4.4 machines

I don't have sockets or snmp compiled directly into php, but I'm not using ICMP ping and I thought having cacti use net-snmp is more powerful instead of using php-snmp? am I wrong on this?

Anyways, I'll play around with adding them to php tomorrow as well as playing with a few more threads/processes settings and see where that gets me.

/usr/local/php/bin/php -v
PHP 5.2.4 (cli) (built: Sep 11 2007 10:37:12)
Copyright (c) 1997-2007 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2007 Zend Technologies

# /usr/local/php/bin/php -m
[PHP Modules]
ctype
date
dom
filter
hash
iconv
json
libxml
mysql
pcre
PDO
pdo_sqlite
posix
Reflection
session
SimpleXML
SPL
SQLite
standard
tokenizer
xml
xmlreader
xmlwriter
zlib

Post by **TheWitness** » Mon Sep 17, 2007 4:00 pm

Depends on what version of Cacti and what you plan on doing with snmp. As far as Cactid is concerned I would goto J. Much more stable.

TheWitness

Post by **TheWitness** » Mon Sep 17, 2007 4:49 pm

I suspect, with that polling time that cactid is either not starting or segfaulting. You need to run from the command line first.

Code: Select all

./cactid

Do it with your log level set to debug.

TheWitness

hylaride · Post by **hylaride** » Tue Sep 18, 2007 11:22 am

I've added sockets & snmp into php. Here are the results of funning cactid from the command line. It's not segfaulting...

Code: Select all

./cactid
CACTID:

There is nothing in the logfile at all (i've set it to debug).

Anything else to recommend before I try cactid 0.8.6j?

Exo7 · Post by **Exo7** » Tue Sep 18, 2007 11:26 am

you could always run a strace on cactid so see if it is hang somewhere

hylaride · Post by **hylaride** » Tue Sep 18, 2007 1:23 pm

Ever have one of those days where you know you're overworked?

I've found out what's going on...since the machine I'm running on doesn't have a compiler (i'm compiling everything on a dev machine...it's my organizations security requirement) i had to create /etc/cactid.conf myself, but typo'd it to /etc/cadtid.conf instead.

Maybe cactid should report that it can't find it? *shrugs*

Anyways, the time it takes for cactid to complete everyting is about the same:

Code: Select all

09/18/2007 01:26:35 PM - SYSTEM STATS: Time:95.2205 Method:cmd.php Processes:4 Threads:N/A Hosts:166 HostsPerProcess:42 DataSources:3804 RRDsProcessed:1865
09/18/2007 01:26:35 PM - SYSTEM STATS: Time:35.4808 Method:cmd.php Processes:4 Threads:N/A Hosts:166 HostsPerProcess:42 DataSources:3804 RRDsProcessed:1096
09/18/2007 01:27:45 PM - SYSTEM STATS: Time:44.5466 Method:cmd.php Processes:4 Threads:N/A Hosts:166 HostsPerProcess:42 DataSources:3804 RRDsProcessed:2648
09/18/2007 01:34:32 PM - SYSTEM STATS: Time:32.3592 Method:cactid Processes:4 Threads:20 Hosts:166 HostsPerProcess:42 DataSources:3804 RRDsProcessed:2977
09/18/2007 01:35:32 PM - SYSTEM STATS: Time:31.3704 Method:cactid Processes:4 Threads:10 Hosts:166 HostsPerProcess:42 DataSources:3804 RRDsProcessed:3335
09/18/2007 01:36:28 PM - SYSTEM STATS: Time:28.3728 Method:cactid Processes:4 Threads:10 Hosts:166 HostsPerProcess:42 DataSources:3804 RRDsProcessed:2775
09/18/2007 01:37:56 PM - SYSTEM STATS: Time:55.9658 Method:cactid Processes:4 Threads:10 Hosts:166 HostsPerProcess:42 DataSources:3804 RRDsProcessed:3074
09/18/2007 01:38:59 PM - SYSTEM STATS: Time:59.3730 Method:cactid Processes:4 Threads:10 Hosts:166 HostsPerProcess:42 DataSources:3804 RRDsProcessed:2376
09/18/2007 01:39:56 PM - SYSTEM STATS: Time:56.2181 Method:cactid Processes:4 Threads:10 Hosts:166 HostsPerProcess:42 DataSources:3804 RRDsProcessed:2376
09/18/2007 01:40:56 PM - SYSTEM STATS: Time:56.1538 Method:cactid Processes:4 Threads:10 Hosts:166 HostsPerProcess:42 DataSources:3804 RRDsProcessed:2376

Not tons better, but averages out a bit nicer. However, check out the load on my system after in the attached image...much better.

Post by **gandalf** » Tue Sep 18, 2007 2:00 pm

hylaride wrote:Maybe cactid should report that it can't find it? *shrugs*

There has been a bug report on this issue lately, well, it was mine

And it is solved, IMHO. But only with latest cactid (See Announcement Forum)
Reinhard

Poor performance in cactid

Poor performance in cactid

Who is online