cmd.php dying in 0.8.6j

[luxten]

Hi all

Here I have CentOS 4.4, php 4.3.9, net-snmp 5.1.2.
I just upgraded to 0.8.6j to fix the recent security issues.
Immediately after it, my drawings ceased being updated, and I found that poller.php is timing out.
Here's my log:

01/19/2007 06:38:28 PM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select poller_id,end_time from poller_time where poller_id = 0"
01/19/2007 06:38:28 PM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select poller_output.output, poller_output.time, poller_output.local_data_id, poller_item.rrd_path, poller_item.rrd_name, poller_item.rrd_num from (poller_output,poller_item) where (poller_output.local_data_id=poller_item.local_data_id and poller_output.rrd_name=poller_item.rrd_name) LIMIT 10000"
01/19/2007 06:38:29 PM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select poller_id,end_time from poller_time where poller_id = 0"
01/19/2007 06:38:29 PM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select poller_output.output, poller_output.time, poller_output.local_data_id, poller_item.rrd_path, poller_item.rrd_name, poller_item.rrd_num from (poller_output,poller_item) where (poller_output.local_data_id=poller_item.local_data_id and poller_output.rrd_name=poller_item.rrd_name) LIMIT 10000"
...
LOTS of these lines...
...
01/19/2007 06:39:51 PM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select poller_id,end_time from poller_time where poller_id = 0"
01/19/2007 06:39:51 PM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select poller_output.output, poller_output.time, poller_output.local_data_id, poller_item.rrd_path, poller_item.rrd_name, poller_item.rrd_num from (poller_output,poller_item) where (poller_output.local_data_id=poller_item.local_data_id and poller_output.rrd_name=poller_item.rrd_name) LIMIT 10000"
01/19/2007 06:39:51 PM - POLLER: Poller[0] Maximum runtime of 292 seconds exceeded. Exiting.
01/19/2007 06:39:51 PM - SYSTEM STATS: Time:293.0723 Method:cmd.php Processes:1 Threads:N/A Hosts:8 HostsPerProcess:8 DataSources:97 RRDsProcessed:5
01/19/2007 06:39:51 PM - CMDPHP: Poller[0] DEBUG: SQL Exec: "replace into settings (name,value) values ('stats_poller','Time:293.0723 Method:cmd.php Processes:1 Threads:N/A Hosts:8 HostsPerProcess:8 DataSources:97 RRDsProcessed:5')"
01/19/2007 06:39:51 PM - CMDPHP: Poller[0] DEBUG: SQL Cell: "select count(*) from poller_command"

I investigated the cause of this behaviur, and found that:
1) cmd.php dies without any warning soon after being run by poller.php.
2) poller.php tries forever to read poller_time, which cmd.php should update, but timeouts because cmd.php is just dead.
3) cmd.php dies when trying to ping a host (in this case localhost), more precisely it dies exactly when when it calls @snmpgetnext() in cacti_snmp_getnext() in snmp.php. In fact, the call to @snmpgetnext has been added in 0.8.6j.
4) @snmpgetnext is only available in PHP5, not in 4.
5) reverting snmpgetnext to snmpget as it used to be, solved the problem.

I think that in the code we should at least track the case of PHP4 and use snmpget instead of snmpgetnext;
and maybe poller.php should look for cmd.php dying abnormally, and report it in the log.
If cmd.php could die with some sort of debug message, I would appreciate it, but I really don't know if this can be done in PHP.

Comments are welcome.

Thank you for your attention
Luigi

[cbcinnovations]

Having the same issue... I am running CentOS 4.4, PHP 4.3.,NET-SNMP 5.3.1

-CBC-

[luxten]

I wrote that changing @snmpgetnext in @snmpget solved the problem. I had been optimistic. It only stops cmd.php to suddenly die.
I also had to change the line in ping.php saying:
$oid = ".1";
into:
$oid = ".1.3.6.1.2.1.1.3.0";
because if I do snmpget instead of snmpgetnext, I have to point to an available oid. I noticed I could have the same result using a retry count higher than 1, since the code switches to the ".1.3.6.1.2.1.1.3.0" oid when it retries.

So, a feature request is added to the ones in my previous post:
can't we have a new property for the host, saying what oid we should ask for pinging it, just as we can modify the SNMP port of the host?
In my opinion, it could be also helpful to have a similar property for the UDP port we use to ping the host, instead of the hard coded 33439. I have cacti in a firewalled environment, and could not use UDP ping because the firewall blocks this port.

[TheWitness]

1) Please verify if php_snmp is installed. Post your findings.
2) Please verify that you have a valid path the the binary snmpgetnext. Post your findings.
3) Please revert to ".1" and sniff utilizing Wireshark (Ethereal) and PM me the capture file.

This works perfectly fine on other systems.

TheWitness

[luxten]

1) Please verify if php_snmp is installed. Post your findings.

It's installed. Sorry for not metioning it explicitly. The fact that with 0.8.6i (and with 0.8.6j, with the little mod I was talking in my previous post) I had no problems should nearly show that the problem is in some change between cacti 0.8.6i and 0.8.6j.
The thing (I think) is that in php-snmp version 4.3, snmpgetnext is not implemented. I had a look at php source code and, althought I'm not a programmer, I found no mention about it in the snmp module. Instead, in php version 5, snmpgetnext is implemented.
In 0.8.6i we did not use snmpgetnext, so there was no problem even using php 4.3. Now in 0.8.6j, since bug#0000886, we use snmpgetnext. But it is not implemented in php-snmp v4.3, and my cmd.php crashes.

2) Please verify that you have a valid path the the binary snmpgetnext. Post your findings.

It's in /usr/bin, and it works. But I'm using snmp v1 so in lib/snmp.php I'm using SNMP_METHOD_PHP, and not SNMP_METHOD_BINARY.

3) Please revert to ".1" and sniff utilizing Wireshark (Ethereal) and PM me the capture file.

I sniffed using the filter 'port 33439 or port 161'. The first being the UDP port used for UDP pinging, the second the SNMP port. Absolutely no packet was captured.
Not surprinsing for me, because of the way cmd.php works: it first pings the host, then it retrieves the data it needs. My cmd.php dies as soon as it tries to ping the host (the first is localhost) with a call to snmpgetnext (which is not implemented in php 4.3, and cmd.php suddenly dies).
If I should sniff using a different filter please ask me. My system is in production, so sniffing without a filter produces a lot of unuseful, and potentially private, infos.
<opinion>
I love network analyzers very much, but unfortunately in this case this kind of tool is of little help, since cmd.php dies just before sending its first packet.
</opinion>

This works perfectly fine on other systems.

Hmm.. with cacti = 0.8.6j and PHP < 5? Because only in this situation this problem arises.
Another info: I tried to use both the options "Ping and SNMP" and "SNMP" for "Downed Host Detection" in the poller options. Since the SNMP ping is tried in both cases, cmd.php dies in both cases.

On my side, things return to work perfectly if I only apply the following patch that reverts to the old-style (pre-0.8.6j) SNMP ping style if we are on php < 5:

Code: Select all

--- lib/ping.php.lux    2007-01-20 08:49:25.000000000 -0500
+++ lib/ping.php        2007-01-20 12:21:05.000000000 -0500
@@ -289,6 +289,7 @@
                                return false;
                        }
 
+                       if (PHP_VERSION >= 5) {
                        $output = cacti_snmp_getnext($this->host["hostname"],
                                $this->host["snmp_community"],
                                $oid,
@@ -298,6 +299,18 @@
                                $this->host["snmp_port"],
                                $this->host["snmp_timeout"],
                                SNMP_CMDPHP);
+                       }else{
+                               /* no snmpgetnext here :( */
+                               $output = cacti_snmp_get($this->host["hostname"],
+                                       $this->host["snmp_community"],
+                                       ".1.3.6.1.2.1.1.3.0" ,
+                                       $this->host["snmp_version"],
+                                       $this->host["snmp_username"],
+                                       $this->host["snmp_password"],
+                                       $this->host["snmp_port"],
+                                       $this->host["snmp_timeout"],
+                                       SNMP_CMDPHP);
+                       }
 
                        /* determine total time +- ~10% */
                        $this->time = $this->get_time($this->precision);

Thank you.

[TheWitness]

Looking at the 4.4.4 source, it's there. I will go back to 4.3.9 and check as well.

TheWitness

Code: Select all

		} else if (st >= 2) {
			pdu = snmp_pdu_create(SNMP_MSG_GETNEXT);
			snmp_add_null_var(pdu, name, name_length);
		}

[TheWitness]

It's also in php 4.3.9. Going back even further...

[TheWitness]

It's in 4.3.0

Try the following. Create a and run a test script:

Code: Select all

<?php

$get_next = function_exists("snmpgetnext");

print "The getnext function '" . $get_next . "'\n";

?>

TheWitness

[TheWitness]

Hmm, even though it's in the source code, it does not appear to be in use..

[TheWitness]

Ok, try this...

TheWitness

[luxten]

It's in 4.3.0

Try the following. Create a and run a test script:

Code: Select all

<?php

$get_next = function_exists("snmpgetnext");

print "The getnext function '" . $get_next . "'\n";

?>

TheWitness

The output is:

Content-type: text/html
X-Powered-By: PHP/4.3.9

The getnext function ''




If I change snmpgetnext in snmpget I obtain:

Content-type: text/html
X-Powered-By: PHP/4.3.9

The getnext function '1'

[luxten]

Hmm, even though it's in the source code, it does not appear to be in use..

I noticed that in the php source package, is the file ext/snmp/snmp.c. There I found the array snmp_functions. In this array, looking in php 5, I found snmpgetnext; in php 4.3.9, I did not.
A look to the function php_snmp and to the documentation seems to confirm that snmpgetnext support began in v5 of php.

[luxten]

Ok, try this...

TheWitness

This does not work as is, but maybe with
(version_compare("5.0", phpversion(), ">="))
you wanted to say
(version_compare("5.0", phpversion(), "<="))

With this change, the patch works. Similar to my little patch, seems we had the same idea

[TheWitness]

How does this work?

Larry

[luxten]

How does this work?

Larry

Now it works.
Any hope to have this included in the mainstream package?

Thanks.