Hung snmpget processes

[bodazyphir]

Hello,

I have a cactic box that is humming away quiet nicely. there is one small glitch that unfortunately I cannot ignore , or fix. There are two particular boxes on my network that when Cacti polls , the snmpget process gets hung up and therefore does not die. As result after many polls , there are hundreds of these ophaned processes left running on the Cacti server till there are eventually so many the server dies.

As i said this ony happens with two machines on the whoe server. i have checked the mibs , they walk fine , i have deteled and recreated the boxes in cacti , still no luck. So I have resorted to running a script to manaully kill all the orphaned processes.

Any ideas ?

[gandalf]

Please find the hanging processes from "ps" command. You should see the timing information and number of retries. What do you see?
Reinhard

[bodazyphir]

I am getting the folloing L

cacti 1161 1 0 May18 ? 00:00:00 /usr/bin/snmpget -O fntev -c -v 2c -t 1 -r 3 ssprdnbu001.iggroup.local:161 .1.3.6.1.2.1.25.2.3.1.5.38
cacti 1249 1 0 May18 ? 00:00:00 /usr/bin/snmpget -O fntev -c -v 2c -t 1 -r 3 ssprdnbu001.iggroup.local:161 .1.3.6.1.2.1.25.2.3.1.5.3
cacti 1650 1 0 May18 ? 00:00:00 /usr/bin/snmpget -O fntev -c -v 2c -t 1 -r 3 ssprdnbu001.iggroup.local:161 .1.3.6.1.2.1.25.2.3.1.5.39
cacti 1998 1 0 May16 ? 00:00:00 /usr/bin/snmpget -O fntev -c -v 2c -t 1 -r 3 ssprdnbu001.iggroup.local:161 .1.3.6.1.2.1.25.2.3.1.6.39

Retries set to 3 ....

Also getting it on another server , but these are the only two example of it , out of a 200 + monitored devices.

/usr/bin/snmpget -O fntev -c -v 1 -t 1 -r 3 lsprdinf008.iggroup.local:161 .1.3.6.1.2.1.25.2.3.1.5.8
cacti 17765 1 0 May14 ? 00:00:00 /usr/bin/snmpget -O fntev -c -v 1 -t 1 -r 3 lsprdinf008.iggroup.local:161 .1.3.6.1.2.1.25.2.3.1.5.8
cacti 18554 1 0 May17 ? 00:00:00 /usr/bin/snmpget -O fntev -c -v 1 -t 1 -r 3 lsprdinf008.iggroup.local:161 .1.3.6.1.2.1.25.2.3.1.5.8
cacti 18934 1 0 May08 ? 00:00:00 /usr/bin/snmpget -O fntev -c -v 1 -t 1 -r 3 lsprdinf008.iggroup.local:161 .1.3.6.1.2.1.25.2.3.1.5.5
cacti 19337 1 0 May07 ? 00:00:00 /usr/bin/snmpget -O fntev -c -v 1 -t 1 -r 3 lsprdinf008.iggroup.local:161 .1.3.6.1.2.1.25.2.3.1.5.8
cacti 20509 1 0 May09 ? 00:00:00 /usr/bin/snmpget -O fntev -c -v 1 -t 1 -r 3 lsprdinf008.iggroup.local:161 .1.3.6.1.2.1.25.2.3.1.5.5

Your help is appreciated,

Stephen

[gandalf]

When running any of those commands, do they hang?
Reinhard

[bodazyphir]

yeah ...

You can walk the entire MiB but when you query a specific OiD the get hangs and leaves the process orphaned. Over a bit of time the hung process build up and eventually the server crashes. It only happens with two boxes out of over 200 being monitored. I have reinstalled the MiB's on both boxes , and they all function fine except for this weirdism.

appreciate you looking into this Gandalf

[gandalf]

Are those boxes running some sort of windows?
Reinhard

[bodazyphir]

Hi,


No they running Linux RHEL 4

Stephen

[bodazyphir]

Another clue I have picked up is in the poller.log I get this entry :

PHP Notice: Undefined variable: config in /var/www/html/cacti/lib/snmp.php on line 34
PHP Warning: Missing argument 9 for cacti_snmp_get() in /var/www/html/cacti/lib/snmp.php on line 40
PHP Warning: Missing argument 10 for cacti_snmp_get() in /var/www/html/cacti/lib/snmp.php on line 40
PHP Notice: Use of undefined constant SNMP_POLLER - assumed 'SNMP_POLLER' in /var/www/html/cacti/lib/snmp.php on line 40
PHP Fatal error: Call to undefined function: read_config_option() in /var/www/html/cacti/lib/snmp.php on line 45
PHP Notice: Undefined variable: config in /var/www/html/cacti/lib/snmp.php on line 34
PHP Warning: Missing argument 9 for cacti_snmp_get() in /var/www/html/cacti/lib/snmp.php on line 40
PHP Warning: Missing argument 10 for cacti_snmp_get() in /var/www/html/cacti/lib/snmp.php on line 40
PHP Notice: Use of undefined constant SNMP_POLLER - assumed 'SNMP_POLLER' in /var/www/html/cacti/lib/snmp.php on line 40
PHP Fatal error: Call to undefined function: read_config_option() in /var/www/html/cacti/lib/snmp.php on line 45


The line 40 and 45 problem seem to be a problem with the config files determination of the kind of platform it is on :

if ($config["cacti_server_os"] == "unix") {
define("SNMP_ESCAPE_CHARACTER", "'");
}else{
define("SNMP_ESCAPE_CHARACTER", "\"");
}

function cacti_snmp_get($hostname, $community, $oid, $version, $username, $password, $auth_proto, $priv_pass, $priv_proto, $context, $port = 161, $timeout = 500, $retries = 0, $environ = SNMP_POLLER) {
global $config;


Problem in the defintion of the escape character could perhaps cause a problem in the termination of an snmpget ?? What do you think ?

[gandalf]

I suppose you'r eusing an old version of scripts. Cacti as of 087x changed cacti_snmp_* functions to fully support SNMP V3. Please search for updated versions of your scripts or change them on your own.
Reinhard

[praveen_b744]

Me too working on the same cacti.

What scripts do I need to check?
Can you please tell me in detail?

We are using the following versions of snmp.
net-snmp-libs-5.1.2-11.el4_6.11.2
net-snmp-utils-5.1.2-11.el4_6.11.2
net-snmp-5.1.2-11.el4_6.11.2
net-snmp-devel-5.1.2-11.el4_6.11.2

And I hope we are using the correct and compatible version of the snmp for the cacti0.8.7b.

[gandalf]

What scripts do I need to check?

There's no general answer to this. All cacti provided scripts were changed by us. If you use scripts downloaded from the forum, this depends.

Reinhard