High CPU utilization on localhost...

[aceves83]

Hi everyone:

The server hosting CACTI is having high CPU utilization since several servers were added (around 90). Before it had 115. For what I read it shouldn't be any problems handling more than 200+ devices.

This is the setup:

Server: Sun E4500
Procs: 2
Memory: 2GB
SO: Solaris 9

CACTI: cacti-0.8.6c
MySQL: mysql-4.0.17
PHP: php-4.3.4
rrdtool: rrdtool-1.0.46

I know they are not the latest versions, but it was working fine and an upgrade is not possible for the moment.

The recently added servers are Windows; the were added using the Windows 2000/XP template but most of them are 2003.

Currently there are around 246 'cacti' processes running.

The top command shows the following:

last pid: 2646; load averages: 77.15, 76.65, 76.91 16:40:30
326 processes: 251 sleeping, 59 running, 13 zombie, 3 on cpu
CPU states: 0.0% idle, 84.2% user, 15.8% kernel, 0.0% iowait, 0.0% swap
Memory: 2048M real, 180M free, 1384M swap in use, 2101M swap free

PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND
2461 cacti 1 42 2 9896K 8088K run 0:00 1.09% php
2503 cacti 1 42 2 9896K 8088K run 0:00 1.03% php
2507 cacti 1 42 2 9896K 8088K run 0:00 1.03% php
2515 cacti 1 42 2 9272K 7488K run 0:00 0.94% php
2505 cacti 1 52 2 9880K 8080K sleep 0:00 0.94% php
2479 cacti 1 53 2 9880K 8080K sleep 0:00 0.93% php
2495 cacti 1 43 2 9336K 7552K run 0:00 0.91% php
2523 cacti 1 52 2 9184K 7400K sleep 0:00 0.91% php
2498 cacti 1 43 2 9280K 7496K run 0:00 0.90% php
2468 cacti 1 53 2 9880K 8080K sleep 0:00 0.88% php
2443 cacti 1 43 2 9880K 8080K run 0:00 0.87% php
2527 cacti 1 42 2 8752K 6872K run 0:00 0.86% php
2422 cacti 1 53 2 9880K 8080K sleep 0:00 0.85% php
2519 cacti 1 52 2 9040K 7256K sleep 0:00 0.85% php
2353 cacti 1 53 2 11M 9256K sleep 0:00 0.84% php

I can open the CACTI home page but it is really slow and sometimes I get an error regarding the mysql database (After reloading the page one it works):

Cannot connect to MySQL server on 'localhost'. Please make sure you have specified a valid MySQL database name in 'include/config.php'.

When trying to connect to the mysql DB I got the following message:

ERROR 1040: Too many connections

Searching on the web I found that a parameter on the my.cnf must be changed to increase the number of connections but I did not find it on the my.conf file.

Any help / comments is really appreciated.

If you need further information let me know.

Thanks in advance for your help!

[cert-eh-fiable]

I'd first try to define what exactly is causing the server load.

The database issues and page load times are because the box is heavily loaded.

I'd shut down snmpd and see what happens to the server load. If it goes back to normal then you know it's probably a factor of the poller and rrdtool. It's possible to configure the poller to run more processes simultaneously which might help the machine not get so back logged.

Console > Settings > Poller

Check your cacti logs for more diagnostic help, enable more reporting in the logs for more details under Console > Settings > General.

Check your Console > Settings > Paths for any errors.

Check your Console > System Utilities > Technical Support for more information.

That's where I'd start. If the load on the machine doesn't resume to normal after killing snmpd then I'd look at other services and resource issues. I wouldn't change /etc/my.cnf just yet. I'd first query the db to see how many users are logged in.

Code: Select all

shell> mysql -u root -p

mysql> SHOW FULL PROCESSLIST;

[Gorbachov]

After adding the Windows servers probably the poll times were higher??

And one poller overlaps with the next because the first was not finished? This causes too many php process that handle too many mysql connections. This is leads to a dead loop. Probably you should check what is the poller responce for the new servers?


Cheers

[Linegod]

The default Windows device template uses quite a few scripts to collect data, versus a switch/router which uses SNMP to collect data. This, I am going to assume, is the root of your problem.

Cut and paste the top portion of 'Console/System Utilities/Technical Support' into the forum (no need for MySQL Table Information or anything below that).

As well, post the last couple of lines of your cacti.log (4 or 5 lines).

[aceves83]

I'd first try to define what exactly is causing the server load.

The database issues and page load times are because the box is heavily loaded.

I'd shut down snmpd and see what happens to the server load. If it goes back to normal then you know it's probably a factor of the poller and rrdtool. It's possible to configure the poller to run more processes simultaneously which might help the machine not get so back logged.

Console > Settings > Poller

I shutted down snmpd and still the server load was high.

Check your cacti logs for more diagnostic help, enable more reporting in the logs for more details under Console > Settings > General.

Check your Console > Settings > Paths for any errors.

Check your Console > System Utilities > Technical Support for more information.

On /usr/local/cacti/log

The error log shows lots of lines similar to:

PHP Warning: mysql_connect(): Too many connections in /usr/local/cacti-0.8.6c/lib/adodb/drivers/adodb-mysql.inc.php on line 339

The cacti.log shows lots of lines like:

CMDPHP: Poller[0] Host[262] WARNING: Result from SNMP not valid. Partial Result: No Such Object avail

That's where I'd start. If the load on the machine doesn't resume to normal after killing snmpd then I'd look at other services and resource issues. I wouldn't change /etc/my.cnf just yet. I'd first query the db to see how many users are logged in.

Code: Select all

shell> mysql -u root -p

mysql> SHOW FULL PROCESSLIST;

I need to reset the mysql root account, to see the how mnay users are looged in.

Thank you for your responses.

Hope you can give more feedback...

[aceves83]

Cut and paste the top portion of 'Console/System Utilities/Technical Support' into the forum (no need for MySQL Table Information or anything below that).

Hmm... There is not Technical Support option under System Utilities just the Poller Cache Administration and System Log Administration... I'm using CACTI 0.8.6c, I don't if this version does not have it.

As well, post the last couple of lines of your cacti.log (4 or 5 lines).

Here they are:

08/11/2009 01:56:29 PM - CMDPHP: Poller[0] Host[283] WARNING: Result from CMD not valid. Partial Result: Warning: mysql_conne
08/11/2009 01:56:30 PM - CMDPHP: Poller[0] Host[253] WARNING: Result from SNMP not valid. Partial Result: No Such Object avail
08/11/2009 01:56:30 PM - CMDPHP: Poller[0] Host[257] WARNING: Result from SNMP not valid. Partial Result: No Such Object avail
08/11/2009 01:56:27 PM - CMDPHP: Poller[0] ERROR: Cannot connect to MySQL server on 'localhost'. Please make sure you have specified a valid MySQL database name in 'include/config.php'.

Thanks for you help!

[Linegod]

I missed that you where running 0.8.6c - I would _highly_ recommend upgrading to 0.8.7e - there are dozens of bug/security fixes.

In the meantime, adjust your MySQL connections, so you can begin to determine where the other problems are:

/etc/my.cnf
<code>
max_connections=250
</code>

[aceves83]

I missed that you where running 0.8.6c - I would _highly_ recommend upgrading to 0.8.7e - there are dozens of bug/security fixes.

In the meantime, adjust your MySQL connections, so you can begin to determine where the other problems are:

/etc/my.cnf
<code>
max_connections=250
</code>

On the server the my.cnf file is actually under /usr/local/mysql/var, modified it and copied to /usr/local/mysql/etc, restarted mysql but the server load has not decreased, just a little bit...

There are several cacti processes running and some of them are defunct, I tried to kill them without luck... seems that I will have to reboot the server...

Thanks Linegod for all the help.

If anyone has more comments or ideas are more than welcome...

[cert-eh-fiable]

LG picked up on a good point. Upgrade to 0.8.7e. I missed that too. Best to start diagnosing with a known good installation.

[Mark.]

So, I'm having a similar problem...but running the latest cacti. I stopped snmpd and it brought my cpu down to 0%.

I guess that means it's a poller issue?
I'm using cmd.php (because spine gave me tons of problems). Is cmd.php supported?

0.8.7a
CentOS 5
SYSTEM STATS: Time:72.2249 Method:cmd.php Processes:5 Threads:N/A Hosts:105 HostsPerProcess:21 DataSources:2232 RRDsProcessed:1246


How would I continue to troubleshoot this? Any suggestions are welcome...plz. Thanks.

[brononi]

A good start is to check the poller timings...

My cron (/etc/crontab) was on 5 minutes.
And my poller (webmin) was on 1 minutes.

After putting this on 5 minutes, my cpu load drops from 120% towards 30%...