Host down, but manual polling on CLI is ok

[jmroth]

Hi there,

What do you make of the following?

Code: Select all

05/07/2016 05:35:11 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'x', and OID:'.1.3.6.1.2.1.1.3.0'
05/07/2016 05:35:11 PM - CMDPHP: Poller[0] Host[212] SNMP: Host did not respond to SNMP
05/07/2016 05:35:11 PM - CMDPHP: Poller[0] Host[212] ERROR: HOST EVENT: Host is DOWN Message: Host did not respond to SNMP

On the command line:

Code: Select all

# time snmpget -v 3 -x des -X xxx -a md5 -A xxx -u ro -l authPriv host .1.3.6.1.2.1.1.3.0
DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (11239138) 1 day, 7:13:11.38

real    0m0.072s
user    0m0.056s
sys     0m0.004s

Manually running "php -q ./cmd.php 212 212" as the debug guide suggests works fine.

Why does it time out when run via the poller/cron but works fine when run manually from the CLI ?

Cacti 0.8.8f (Linux), Poller Type: cmd.php, Downed Host Detection: SNMP Uptime

Thanks,
Marki

[micke2k]

Make sure you haven't filled anything out in the context column on snmp v3.

If you switch to v2 does it work?

[jmroth]

I don't know if v2 works. Unfortunately I can't just go there and change those systems.

Here's what I get with v3 when looking at Wireshark

Code: Select all

38995	2016-05-08 13:30:12.103382000	192.xxxxxxxxxx	172.xxxxxxxxxx	SNMP	188	get-request 1.3.6.1.2.1.1.3.0
38996	2016-05-08 13:30:12.104170000	172.xxxxxxxxxx	192.xxxxxxxxxx	SNMP	176	report 1.3.6.1.6.3.15.1.1.2.0

So instead of get-response I am getting "usmStatsNotInTimeWindows".

WTH?

This is SLES11 with php53-snmp-5.3.17-48.1

Time is synchronized. Anyway it still works on the CLI but not when run via poller/cmd.php.

[jmroth]

Oh and running the exact same snmp3_get() command as found in lib/snmp.php from the CLI is ok too.

Code: Select all

<?php
snmp_set_quick_print(0);

$r = snmp3_get("x:161", "ro", "authPriv", "MD5", "x", "DES", "x", ".1.3.6.1.2.1.1.3.0", 500000, 1);

print_r($r);

Code: Select all

# php test.php
Timeticks: (8248) 0:01:22.48

[jmroth]

This is sick...

The device is shown as down (with the SNMP errors shown in my previous post).

However on the actual device page, everything seems to work normally:

Code: Select all

SNMP Information
System:Linux hostname 3.12.51-52.39-default #1 SMP Fri Jan 15 20:03:12 UTC 2016 (16f5bac) x86_64
Uptime: 9467481 (1 days, 2 hours, 17 minutes)
Hostname: hostname 
Location: Server Room
Contact: Sysadmin (root@localhost)

The Data Queries also show results ("Success [33 Items, 11 Rows]" etc.)

What can this be?

Bye,

Marki

[jmroth]

This seems to have been an issue because of duplicated engine IDs on the remote hosts (cloned VMs).

How to make sure SNMPv3 Engine ID is not duplicated:

[*] Stop snmpd:

Code: Select all

 service snmpd stop

[*] Delete from snmpd config (location depends on distribution)

Code: Select all

 engineBoots 2
 oldEngineID .....

[*] Then:

Code: Select all

 service snmpd start
 service snmpd stop

[*] Then create the SNMPv3 users.

[*] Then restart and you're done.

Code: Select all

 service snmpd start