Poller claims host not responding to SNMP

[Feren]

Hello all,

I've done some digging and followed a few leads here on the forum and elsewhere on the net, but I keep getting stuck so I'm hoping that somebody here can help.

We are moving Cacti off of a Solaris 8 machine onto a Solaris 10 machine running Sun's "Glassfish" web stack (Apache, MySQL, PHP all precompiled). The migration of data, graphs, database, etc all went fine. However, on the new system all of my devices are showing as "Down." Cranking the logging level up shows the following in the log:

----------
12/10/2009 02:25:01 PM - CMDPHP: Poller[0] Host[2] SNMP: Host did not respond to SNMP
12/10/2009 02:25:01 PM - CMDPHP: Poller[0] Host[3] SNMP: Host did not respond to SNMP
12/10/2009 02:25:01 PM - CMDPHP: Poller[0] Host[4] SNMP: Host did not respond to SNMP
12/10/2009 02:25:01 PM - CMDPHP: Poller[0] Host[5] SNMP: Host did not respond to SNMP
12/10/2009 02:25:01 PM - CMDPHP: Poller[0] Host[6] SNMP: Host did not respond to SNMP
12/10/2009 02:25:01 PM - CMDPHP: Poller[0] Host[7] SNMP: Host did not respond to SNMP
12/10/2009 02:25:01 PM - CMDPHP: Poller[0] Host[10] SNMP: Host did not respond to SNMP
12/10/2009 02:25:01 PM - CMDPHP: Poller[0] Host[11] SNMP: Host did not respond to SNMP
12/10/2009 02:25:01 PM - CMDPHP: Poller[0] Host[12] SNMP: Host did not respond to SNMP
12/10/2009 02:25:01 PM - CMDPHP: Poller[0] Host[14] SNMP: Host did not respond to SNMP
12/10/2009 02:25:01 PM - CMDPHP: Poller[0] Host[16] SNMP: Host did not respond to SNMP
12/10/2009 02:25:01 PM - CMDPHP: Poller[0] Host[23] SNMP: Host did not respond to SNMP
12/10/2009 02:25:01 PM - CMDPHP: Poller[0] Host[18] SNMP: Host did not respond to SNMP
12/10/2009 02:25:01 PM - CMDPHP: Poller[0] Host[24] SNMP: Host did not respond to SNMP


and on and on for all the various hosts I have (approx 134 of them).
----------------------

While poller doesn't seem to be able to talk to the devices, I have no problem from the command line. I have ensured I'm using the exact same "snmpbulkwalk/snmpget/snmpwalk/" etc that is pointed to in the Settings, and I have no problem querying any of the devices. So I know it's not an ACL.

When I do a Verbose Query for "SNMP - Interface Statistics" on any of the devices I see "SNMP Error" in red on the top left and the following for Data Query Debug Information:

---------
+ Running data query [1].
+ Found type = '3' [snmp query].
+ Found data query XML file at '/var/opt/webstack/apache2/2.2/htdocs/cacti/resource/snmp_queries/interface.xml'
+ XML file parsed ok.
+ Executing SNMP walk for list of indexes @ '.1.3.6.1.2.1.2.2.1.1'
+ No SNMP data returned
+ Found data query XML file at '/var/opt/webstack/apache2/2.2/htdocs/cacti/resource/snmp_queries/interface.xml'
+ Found data query XML file at '/var/opt/webstack/apache2/2.2/htdocs/cacti/resource/snmp_queries/interface.xml'
+ Found data query XML file at '/var/opt/webstack/apache2/2.2/htdocs/cacti/resource/snmp_queries/interface.xml'
+ Found data query XML file at '/var/opt/webstack/apache2/2.2/htdocs/cacti/resource/snmp_queries/interface.xml'
+ Found data query XML file at '/var/opt/webstack/apache2/2.2/htdocs/cacti/resource/snmp_queries/interface.xml'
+ Found data query XML file at '/var/opt/webstack/apache2/2.2/htdocs/cacti/resource/snmp_queries/interface.xml'
+ Found data query XML file at '/var/opt/webstack/apache2/2.2/htdocs/cacti/resource/snmp_queries/interface.xml'
+ Found data query XML file at '/var/opt/webstack/apache2/2.2/htdocs/cacti/resource/snmp_queries/interface.xml'
+ Found data query XML file at '/var/opt/webstack/apache2/2.2/htdocs/cacti/resource/snmp_queries/interface.xml'
+ Found data query XML file at '/var/opt/webstack/apache2/2.2/htdocs/cacti/resource/snmp_queries/interface.xml'
+ Found data query XML file at '/var/opt/webstack/apache2/2.2/htdocs/cacti/resource/snmp_queries/interface.xml'
+ Found data query XML file at '/var/opt/webstack/apache2/2.2/htdocs/cacti/resource/snmp_queries/interface.xml'
+ Found data query XML file at '/var/opt/webstack/apache2/2.2/htdocs/cacti/resource/snmp_queries/interface.xml'
+ Found data query XML file at '/var/opt/webstack/apache2/2.2/htdocs/cacti/resource/snmp_queries/interface.xml'
------------------

I found in lib/snmp.php where the "Verbose Query" is built and duplicated the command on the CLI with success:

snmpbulkwalk -O Qn -c community -v 2c -t 1 -r 1 -Cr10 a.b.c.d:161 .1.3.6.1.2.1.2.2.1.1

It returned all the expected rows.....

I'm at a loss how I can debug this further to see where the error is. It's clearly inside this system, and it very well may be a misconfiguration on my part or a version mismatch (Since we went from Solaris 8 and a different version of Apache and PHP and NetSNMP to new versions on this machine), but I'm stumped.

Relevant version information:
* OS: Solaris 10
* Cacti: 0.8.7e
* Apache: Apache/2.2.11 (Unix) mod_ssl/2.2.11 OpenSSL/0.9.7d DAV/2 PHP/5.2.9
* PHP: 5.2.9
* Net-SNMP: 5.4.2.1
* RRDtool: 1.3.5

Any pointers or tips would be most appreciated. Thank you very much.

[gandalf]

Cacti does use two different methods to get snmp data: net-snmp or php-snmp. So please verify, if you are using php-snmp (php -m should tell you) or not
R.

[Feren]

Hello and thank you for the reply, Gandalf. I do not appear to be using php-snmp, as "php -m | grep snmp" returns no hit on my new server.

$ php -m
[PHP Modules]
apc
bcmath
bz2
calendar
ctype
curl
date
dba
dom
exif
filter
ftp
gd
gettext
hash
iconv
idn
imap
json
ldap
libxml
mbstring
mcrypt
memcache
mysql
mysqli
openssl
pcntl
pcre
PDO
pdo_mysql
pdo_sqlite
posix
Reflection
session
shmop
SimpleXML
soap
sockets
SPL
SQLite
standard
sysvmsg
sysvsem
sysvshm
tcpwrap
tidy
tokenizer
xml
xmlreader
xmlrpc
xmlwriter
xsl
zip
zlib

[Zend Modules]

I have executed the same command on my old server and do get back "snmp" as a module in the list. So that explains that difference, at least.

Is there a setting I can tweak inside Cacti to push it off php-snmp and over to net-snmp, or am I going to have to ditch this precompiled stack and roll my own?

Thanks for helping make sense of this for me. I thought I was going crazy....

----
Bit of follow-up after I posted above:

Confirmed using the "about" page that my Cacti is reporting "PHP SNMP Support: no"

Confirmed in Settings/General that I have "SNMP Utility Version" set to "NET-SNMP 5.x"

[gandalf]

Please install php-snmp, e.g. yum install php-snmp or the like and retry
R.

[TheWitness]

What about Console -> Settings -> Paths. Also, get with the Glassy dudes and tell them to include php-snmp. Yum does not work on Solaris I think (shyly).

TheWitness

[gandalf]

Yum does not work on Solaris I think (shyly).TheWitness

Oops. Missed that Solaris bit
R.

[Feren]

Sorry for the delay -- we're still relocating the datacenter and are in the final stages of relocating our home offices. Utterly crazy time....

What about Console -> Settings -> Paths.

Cacti Settings (Paths)

Required Tool Paths
snmpwalk Binary Path
The path to your snmpwalk binary.
[OK: FILE FOUND]
snmpget Binary Path
The path to your snmpget binary.
[OK: FILE FOUND]
snmpbulkwalk Binary Path
The path to your snmpbulkwalk binary.
[OK: FILE FOUND]
snmpgetnext Binary Path
The path to your snmpgetnext binary.
[OK: FILE FOUND]
RRDTool Binary Path
The path to the rrdtool binary.
[OK: FILE FOUND]
RRDTool Default Font Path
The path to the rrdtool default true type font for version 1.2 and above.
PHP Binary Path
The path to your PHP binary file (may require a php recompile to get this file).
[OK: FILE FOUND]
Logging
Cacti Log File Path
The path to your Cacti log file (if blank, defaults to /log/cacti.log)
[OK: FILE FOUND]

Also, get with the Glassy dudes and tell them to include php-snmp. Yum does not work on Solaris I think (shyly).

Yeah, no Yum on Solaris. Le sigh. And after dunning a quick check of a php file with "phpinfo();" to see how it was compiled... yeah, the Glassfish team didn't compile it with SNMP. Just everything else and the kitchen sink (I have no idea why not). So I'll be sending them a request to PLEASE add that to future revisions...

Any further thoughts, TheWitness?

[Feren]

Further to my point I'm so desperate at this point I'm willing to try compiling SNMP as a module for PHP to load since it currently doesn't have it compiled in from the Glassfish team.

Though I haven't a clue how to go about that.