cacti 0.8.6d: polling process lock up

[rpingar]

after the upgrade from 0.8.6c to 0.8.6d the polling process using cmd.php is locking up. Without any particular message into the debug file.
It lock up every time during the execution of rrdtool.

I use rrdtool1.0.49 php4.3.10 mysql4.0.22

I poll snmp and scripts.

Thanks

Rosario

[williem]

Hey,

Have you tried running the poller.php from a cmd prompt to see if it gives you any messages? That is how I had to work on a problem with cactid.

Regards,

Willie

[rpingar]

I tried to run by command line but I got the same problem.

I'm using cmd.php not cactid.

thanks

[TheWitness]

Please state your environment:

Number of Hosts:
Number of Scripts:
Number of SNMP:
Number of RRDFiles:

Operating System Ver., MySQL Ver., PHP Ver., Net-SNMP Ver.

Then, set logging to DEBUG for one pass and run the following from the command line:

php <path_cacti>/cmd.php

Save your output and post. There are two patches that will be released related to poller.php. These only affect systems under very hight load with non-optimized mysql structure. I will include the files here.

There is likely going to be a lib/snmp.php patch that does not affect polling though.

In addition, please increase your maximum memory for PHP to something around 64MBytes. Settings can be found in the php.ini file.

TheWitness

[claudio]

After updating updating 0.8.5a to 0.8.6d the new poller script poller.php exceeds its timelimit.
I tried also cactid and patched the system withe the new cmd.php and poller.php of TheWitness, but no success...

Cacti infos:
Hosts: 52
Scripts: not a lot of
SNMP: ??
RRD Files: 309 (35MB)

System infos:
Win2K Server
MySQL 3.23.55-nt
Apache 2.0.54
PHP 4.3.11
NO NetSNMP

[claudio]

output of poller.php is:


05/04/2005 08:24:25 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: D:\Programs\webserver\cactid\cactid.exe, ARG
S: 0 71]
Waiting on 1/1 pollers.
Waiting on 1/1 pollers.
Waiting on 1/1 pollers.
Waiting on 1/1 pollers.
Waiting on 1/1 pollers.
Waiting on 1/1 pollers.
Waiting on 1/1 pollers.
Waiting on 1/1 pollers.
Waiting on 1/1 pollers.
Waiting on 1/1 pollers.
Waiting on 1/1 pollers.
Waiting on 1/1 pollers.
Waiting on 1/1 pollers.
Waiting on 1/1 pollers.
Waiting on 1/1 pollers.
Waiting on 1/1 pollers.
Waiting on 1/1 pollers.
Waiting on 1/1 pollers.
Waiting on 1/1 pollers.
Waiting on 1/1 pollers.
Waiting on 1/1 pollers.
....
...
..
.

Has anybody an idea, what I have made wrong?

[TheWitness]

You have orphaned Data Sources. Please note below

05/04/2005 06:42:53 PM - CMDPHP: Poller[0] PHP Script Server Started Properly
05/04/2005 06:42:55 PM - CMDPHP: Poller[0] Host[0] WARNING: Result from SNMP not valid. Partial Result:
05/04/2005 06:42:55 PM - CMDPHP: Poller[0] Host[0] SNMP: v0: None, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.2, output: U
05/04/2005 06:42:58 PM - CMDPHP: Poller[0] Host[0] WARNING: Result from SNMP not valid. Partial Result:
05/04/2005 06:42:58 PM - CMDPHP: Poller[0] Host[0] SNMP: v0: None, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.2, output: U
05/04/2005 06:43:00 PM - CMDPHP: Poller[0] Host[0] WARNING: Result from SNMP not valid. Partial Result:
05/04/2005 06:43:00 PM - CMDPHP: Poller[0] Host[0] SNMP: v0: None, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.2, output: U
05/04/2005 06:43:02 PM - CMDPHP: Poller[0] Host[0] WARNING: Result from SNMP not valid. Partial Result:
05/04/2005 06:43:02 PM - CMDPHP: Poller[0] Host[0] SNMP: v0: None, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.2, output: U
05/04/2005 06:43:04 PM - CMDPHP: Poller[0] Host[0] WARNING: Result from SNMP not valid. Partial Result:

Goto Data Sources Filter Host=None, then delete the abarent Data Sources and you should be back in business.

TheWitness

[TheWitness]

Also, increase your concurrent processes under Settings->Poller to something higher than 1 if you are using cmd.php.

TheWitness

[claudio]

OK, I fixed this two issues.

Now it works, when I execute cmd.php from the command line...

But when I execute poller.php:
D:\Programs\webserver\webpages\cacti>D:\Programs\webserver\php\php.exe d:\programs\webserve
05/05/2005 12:20:11 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: D:\
d:\programs\webserver\webpages\cacti/cmd.php 0 5]
05/05/2005 12:20:12 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: D:\
d:\programs\webserver\webpages\cacti/cmd.php 6 10]
05/05/2005 12:20:12 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: D:\
d:\programs\webserver\webpages\cacti/cmd.php 11 23]
05/05/2005 12:20:12 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: D:\
d:\programs\webserver\webpages\cacti/cmd.php 25 31]
05/05/2005 12:20:13 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: D:\
d:\programs\webserver\webpages\cacti/cmd.php 32 48]
05/05/2005 12:20:13 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: D:\
d:\programs\webserver\webpages\cacti/cmd.php 49 57]
05/05/2005 12:20:14 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: D:\
d:\programs\webserver\webpages\cacti/cmd.php 59 64]
05/05/2005 12:20:14 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: D:\
d:\programs\webserver\webpages\cacti/cmd.php 65 69]
05/05/2005 12:20:15 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: D:\
d:\programs\webserver\webpages\cacti/cmd.php 71 71]
Waiting on 9/9 pollers.
Waiting on 9/9 pollers.
Waiting on 9/9 pollers.
Waiting on 9/9 pollers.
Waiting on 9/9 pollers.
Waiting on 9/9 pollers.
Waiting on 9/9 pollers.
...
..
.

[claudio]

As a temporary hotfix i tried to run cmd.php as a scheduled task.
but now I have the following text in the log:

...
...
PHPSVR: Poller[0] DEBUG: PHP Script Server Shutdown request received, exiting
...
...

Gathering of snmp data seems to work...
PHP script memory is set to 32M. I also tried 128M with no success...

Any idea?

[TheWitness]

You need to run the following. Please keep in mind that shceduling cmd.php only makes this issue worse, so don't do it.

mysql -u root -p cacti
> truncate table poller_output;
> quit

Also, if you are running PHP 5.x, please insure that your Settings->Paths for PHP is set to <path_php>/php-win.exe". Then you should be back in business.

TheWitness

[claudio]

Hello

I changed the scheduled taskl back to poller.php and entered the mysql command as described.

Now the debug output is:

05/06/2005 02:09:21 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: D:\Programs\webserver\php\php.exe, ARGS: -q d:\programs\webserver\webpages\cacti/cmd.php 71 71]
05/06/2005 02:09:20 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: D:\Programs\webserver\php\php.exe, ARGS: -q d:\programs\webserver\webpages\cacti/cmd.php 65 69]
05/06/2005 02:09:20 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: D:\Programs\webserver\php\php.exe, ARGS: -q d:\programs\webserver\webpages\cacti/cmd.php 59 64]
05/06/2005 02:09:19 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: D:\Programs\webserver\php\php.exe, ARGS: -q d:\programs\webserver\webpages\cacti/cmd.php 49 57]
05/06/2005 02:09:19 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: D:\Programs\webserver\php\php.exe, ARGS: -q d:\programs\webserver\webpages\cacti/cmd.php 32 48]
05/06/2005 02:09:19 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: D:\Programs\webserver\php\php.exe, ARGS: -q d:\programs\webserver\webpages\cacti/cmd.php 25 31]
05/06/2005 02:09:18 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: D:\Programs\webserver\php\php.exe, ARGS: -q d:\programs\webserver\webpages\cacti/cmd.php 11 23]
05/06/2005 02:09:18 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: D:\Programs\webserver\php\php.exe, ARGS: -q d:\programs\webserver\webpages\cacti/cmd.php 6 10]
05/06/2005 02:09:18 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: D:\Programs\webserver\php\php.exe, ARGS: -q d:\programs\webserver\webpages\cacti/cmd.php 0 5]
05/06/2005 02:09:12 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.


...and the same with cactid:

05/06/2005 03:09:49 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: D:\Programs\webserver\cactid\cactid.exe, ARGS: 0 71]
05/06/2005 03:09:44 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.


Regards,
Claudio

[claudio]

I changed the IP adress in cactid.conf from the real address to the loopback ip (127.0.0.1).

Now everithing works. Cool !

(...except when I try to use cmd.php. But I don't care a pap of it, as long as cactid works...)

Thank you guys for help!

Greez,

Claudio

[hoganw]

I just upgraded to 'd' on Linux and suffer the same intermittant graphing, but I'm using cactid and not cmd.php.

Number of hosts: 167
Number of scripts: ~4 scripts, number of script data sources would probably be 600? I have over 1200 data sources total.
RRD files: 1730 (Removed some old/redundant data sources today)

OS: RHEL AS3 Update 4
MySQL: 3.23.58
PHP: 4.3.10
Net-SNMP: 5.0.9

The intermittant graphing was for SNMP and script data, I upped the SNMP timeout to 1 second and removed about 500 data sources, all SNMP, to try and aleviate the problem. So far it seems to be better but too soon to say it's fixed.

[EDIT] The post on a similar problem on Linux is here

[TheWitness]

http://forums.cacti.net/viewtopic.php?t=7486