SNMP Get Processor Info results in 0 rows for Win2k3 hosts

[Lord_Moebius]

Hi!

I am running Cacti 0.8.7d on Linux (SuSe SLES 10.1).

I want to monitor the CPU usage of our Windows 2003 servers using Cacti and added those hosts using the Windows 2000/XP Host template.

The "SNMP - Interface Statistics" query works fine, returning 14 items, 2 rows, and no problems with the SNMP queries "Host MIB - Logged in Users" and "Host MIB - Processes" either.

However, the "SNMP - Get Processor Information" query returns status "Success [0 Items, 0 Rows]", with the following debug information:

Code: Select all

+ Running data query [9].
+ Found type = '6 '[script query].
+ Found data query XML file at '/opt/cacti-0.8.7d/resource/script_server/host_cpu.xml'
+ XML file parsed ok.
+ Executing script for list of indexes '/usr/bin/php -q /opt/cacti-0.8.7d/scripts/ss_host_cpu.php MYHOST 25 2:161:500:1:20:XXXXX:::MD5::DES: index'
+ Executing script query '/usr/bin/php -q /opt/cacti-0.8.7d/scripts/ss_host_cpu.php MYHOST 25 2:161:500:1:20:XXXXX:::MD5::DES: query index'
+ Found data query XML file at '/opt/cacti-0.8.7d/resource/script_server/host_cpu.xml'
+ Found data query XML file at '/opt/cacti-0.8.7d/resource/script_server/host_cpu.xml'
+ Found data query XML file at '/opt/cacti-0.8.7d/resource/script_server/host_cpu.xml'

I don't seem to get any data.

snmpwalk runs fine:

Code: Select all

snmpwalk -v2c -c 'XXXXX' MYHOST processor
HOST-RESOURCES-MIB::hrProcessorFrwID.2 = OID: SNMPv2-SMI::zeroDotZero
HOST-RESOURCES-MIB::hrProcessorFrwID.3 = OID: SNMPv2-SMI::zeroDotZero
HOST-RESOURCES-MIB::hrProcessorFrwID.4 = OID: SNMPv2-SMI::zeroDotZero
HOST-RESOURCES-MIB::hrProcessorFrwID.5 = OID: SNMPv2-SMI::zeroDotZero
HOST-RESOURCES-MIB::hrProcessorFrwID.6 = OID: SNMPv2-SMI::zeroDotZero
HOST-RESOURCES-MIB::hrProcessorFrwID.7 = OID: SNMPv2-SMI::zeroDotZero
HOST-RESOURCES-MIB::hrProcessorFrwID.8 = OID: SNMPv2-SMI::zeroDotZero
HOST-RESOURCES-MIB::hrProcessorFrwID.9 = OID: SNMPv2-SMI::zeroDotZero
HOST-RESOURCES-MIB::hrProcessorLoad.2 = INTEGER: 0
HOST-RESOURCES-MIB::hrProcessorLoad.3 = INTEGER: 2
HOST-RESOURCES-MIB::hrProcessorLoad.4 = INTEGER: 0
HOST-RESOURCES-MIB::hrProcessorLoad.5 = INTEGER: 2
HOST-RESOURCES-MIB::hrProcessorLoad.6 = INTEGER: 4
HOST-RESOURCES-MIB::hrProcessorLoad.7 = INTEGER: 1
HOST-RESOURCES-MIB::hrProcessorLoad.8 = INTEGER: 1
HOST-RESOURCES-MIB::hrProcessorLoad.9 = INTEGER: 5

... and running the ss_host_cpu.php script manually does actually retrieves the correct data:

Code: Select all

php /opt/cacti/scripts/ss_host_cpu.php MYHOST 25 2:161:500:1:10:XXXXX:::MD5::DES: query usage
0!1
1!0
2!0
3!1
4!3
5!3
6!2
7!3

If I look at the query debug information, "query index" is there, but not "query usage". I assume it should be there, returning 8 rows.

When I turn on DEBUG logging, I get the following entries for this host:

Code: Select all

03/30/2009 11:55:04 AM - CMDPHP: Poller[0] Host[25] PING: UDP Ping Success (1.89 ms)
03/30/2009 11:55:04 AM - CMDPHP: Poller[0] Host[25] RECACHE: Processing 3 items in the auto reindex cache for 'MYHOST'.
03/30/2009 11:55:04 AM - CMDPHP: Poller[0] Host[25] DS[481] SNMP: v2: MYHOST, dsname: proc, oid: .1.3.6.1.2.1.25.1.6.0, output: 112
03/30/2009 11:55:04 AM - CMDPHP: Poller[0] Host[25] DS[480] SNMP: v2: MYHOST, dsname: users, oid: .1.3.6.1.2.1.25.1.5.0, output: 13
03/30/2009 11:55:04 AM - CMDPHP: Poller[0] Host[25] DS[479] SNMP: v2: MYHOST, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.65539, output: 494549702
03/30/2009 11:55:04 AM - CMDPHP: Poller[0] Host[25] DS[479] SNMP: v2: MYHOST, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.65539, output: 3911759620

Apparently, the poller does not even try to get the CPU data, so I guess the problem lies somewhere in the definition step for the Win2k3 hosts (i.e. in the "devices" screen).

I have searched the forum and Google and found quite a few entries with the same issue, but none specifies what exactly is going wrong and how to fix it.

I would appreciate any help to get this solved.
Thanks!

[BSOD2600]

It's refreshing to have a user do troubleshooting and try and help themself before posting. Bravo.

Try increasing the snmp timeout a lot and decreasing the number of OIDs per request for that host

Does running 'query index' manually return a list of indexes?

[Lord_Moebius]

Increasing the SNMP timeout from 500 to 5000 ms and clicking the green circle for "Reload Data Query" does not help. I have to wait for roughly 25 seconds after which I still get "Success [0 Items, 0 Rows]" (normal wait time for 500 ms is about 2 seconds).
Increasing to 10000 ms doubles the waiting the time with the same result.
The fact that the waiting time increases indicates to me that the timeout is actually reached.
Decreasing the maximum nr of OID's per request from 10 to 2 or 1 in combination with a timeout of 1000 or 5000 ms does not help.

Because of the apparent timeout, I used tcpdump:

Code: Select all

09:41:10.719914 IP CACTISERVER.7577 > MYHOST.161:  C=A23B GetBulk(29)  N=0 M=20 .1.3.6.1.2.1.25.3.3.1
09:41:15.723360 IP CACTISERVER.7577 > MYHOST.161:  C=A23B GetBulk(29)  N=0 M=20 .1.3.6.1.2.1.25.3.3.1
09:41:20.869780 IP CACTISERVER.7580 > MYHOST.161:  C=A23B GetBulk(29)  N=0 M=20 .1.3.6.1.2.1.25.3.3.1
09:41:25.872156 IP CACTISERVER.7580 > MYHOST.161:  C=A23B GetBulk(29)  N=0 M=20 .1.3.6.1.2.1.25.3.3.1
09:41:30.876386 IP CACTISERVER.7581 > MYHOST.161:  C=A23B GetBulk(29)  N=0 M=20 .1.3.6.1.2.1.25.3.3.1
09:41:35.880166 IP CACTISERVER.7581 > MYHOST.161:  C=A23B GetBulk(29)  N=0 M=20 .1.3.6.1.2.1.25.3.3.1
09:41:40.938384 IP CACTISERVER.7586 > MYHOST.161:  C=A$123B GetRequest(28)  .1.3.6.1.2.1.1.3.0
09:41:40.938584 IP MYHOST.161 > CACTISERVER.7586:  C=A$123B GetResponse(31)  .1.3.6.1.2.1.1.3.0=1670915

Apparently MYHOST does not respond to the GetBulk's from CACTISERVER, while the GetRequest works fine. But note the community string! It seems that for the bulkwalks, the $1 part is somehow "expanded" to an empty string, while the GetRequest has the correct community string!
The weird thing is that the community string is kept intact when I reload the data query for "SNMP - Interface Statistics". That query also uses bulkwalks and the community string is kept at A$123B as it should be.

I guess there is something funny in the code that is specific for the "SNMP - Get Processor Information" (and actually, the "SNMP - Get Mounted Partitions" query as well, which does not work for the same reason). I think the community string is not escaped correctly either in the scripts involved with these queries, or when passed as a parameter to these scripts.

[Lord_Moebius]

Solved!

Apparently, the XML's .../resource/script_server/host_cpu.xml and host_disk.xml require a pair of single quotes around the argument string:

Code: Select all

# diff host_cpu.xml host_cpu.xml.original
6c6
<       <arg_prepend>|host_hostname| |host_id| '|host_snmp_version|:|host_snmp_port|:|host_snmp_timeout|:|host_ping_retries|:|host_max_oids|:|host_snmp_community|:|host_snmp_username|:|host_snmp_password|:|host_snmp_auth_protocol|:|host_snmp_priv_passphrase|:|host_snmp_priv_protocol|:|host_snmp_context|'</arg_prepend>
---
>       <arg_prepend>|host_hostname| |host_id| |host_snmp_version|:|host_snmp_port|:|host_snmp_timeout|:|host_ping_retries|:|host_max_oids|:|host_snmp_community|:|host_snmp_username|:|host_snmp_password|:|host_snmp_auth_protocol|:|host_snmp_priv_passphrase|:|host_snmp_priv_protocol|:|host_snmp_context|</arg_prepend>


# diff host_disk.xml host_disk.xml.original
6c6
<       <arg_prepend>|host_hostname| |host_id| '|host_snmp_version|:|host_snmp_port|:|host_snmp_timeout|:|host_ping_retries|:|host_max_oids|:|host_snmp_community|:|host_snmp_username|:|host_snmp_password|:|host_snmp_auth_protocol|:|host_snmp_priv_passphrase|:|host_snmp_priv_protocol|:|host_snmp_context|'</arg_prepend>
---
>       <arg_prepend>|host_hostname| |host_id| |host_snmp_version|:|host_snmp_port|:|host_snmp_timeout|:|host_ping_retries|:|host_max_oids|:|host_snmp_community|:|host_snmp_username|:|host_snmp_password|:|host_snmp_auth_protocol|:|host_snmp_priv_passphrase|:|host_snmp_priv_protocol|:|host_snmp_context|</arg_prepend>

Now the arguments including the community string will appear in quotes in the query debug output, passing the arguments correctly.
It works!

Thanks for putting me on the right track!
(and please fix this in Cacti's next release)

[BSOD2600]

hmm good find. Although, looking through the Windows (ok *.zip download) of cacti 0.8.7 compared to 0.8.7d, I don't see those quotes present in either -- and they appear to work on at Windows Cacti box.

Needless to say though, if that fixed it for you, a bug should be created. http://www.cacti.net/bugs.php

[Lord_Moebius]

Note that Cacti is running on a Linux server, where the shell is involved in calling the scripts and is responsible for the unwanted expansion of $1. This probably does not occur when running on a Windows server, so the quotes are not needed there.

Anyway, it seems I celebrated too early. The poller is now querying CPU usage, but I don't get a valid result.

Code: Select all

From Cacti log:
03/31/2009 11:15:07 AM - PHPSVR: Poller[0] DEBUG: INC: '/opt/cacti-0.8.7d/scripts/ss_host_cpu.php' FUNC: 'ss_host_cpu' PARMS: 'MYHOST 25 '2:161:500:1:10:XXXXX:::MD5::DES:' get usage 0'
03/31/2009 11:15:07 AM - CMDPHP: Poller[0] Host[25] DS[505] WARNING: Result from SERVER not valid.  Partial Result: U

Manual poll using script server:
> php -q ./script_server.php
PHP Script Server has Started - Parent is cmd
/opt/cacti-0.8.7d/scripts/ss_host_cpu.php ss_host_cpu MYHOST 25 '2:161:500:1:10:XXXXX::MD5::DES:' get usage 0
U

Manual poll, direct call:
> php /opt/cacti-0.8.7d/scripts/ss_host_cpu.php  MYHOST 25 '2:161:500:1:10:XXXXX::MD5::DES:' get usage 0
2

Rebuilding the poller cache did not help, which makes sense, because it seems to use the correct query. However, I don't know yet why there is a difference in results between a direct call to ss_host_cpu.php and a call using the script server.

[Lord_Moebius]

After some debugging, I noticed that the quotes I added to the XML reappear in the SNMP version when the ss_host_cpu.php script is called from the script server.
When called directly from the shell, the quotes are automatically removed, but when called from the script server they are not.

To sum up:

PROBLEM
Using a Linux Cacti server to monitor Windows hosts using the "SNMP - Get Mounted Partitions" and "SNMP - Get Processor Information" templates will fail if the SNMP community string contains a $ character due to incorrect quoting/escaping. This will prevent the data queries to get any rows.

WORKAROUND (quite horrible)
Modify .../resource/script_server/host_cpu.xml and host_disk.xml so that the host_snmp_community parameter is properly quoted (see above for diffs).
This workaround will allow the creation of the appropiate data sources and graphs, but will break data gathering for these sources. The poller does not call the scripts through the shell, but uses the script server which passes the quotes along with the rest of the parameters.
Therefore, after creation of the data sources, the quotes from the XML files should be removed again, followed by a cache rebuild of the poller (see System Utilities).

If somebody knows a nicer workaround, please let me know.
In the meantime I will file a bug report.

[BSOD2600]

In the meantime I will file a bug report.

Thanks!

I'm sure others are tearing their hair out over this...

[cspenpen]

Solved!

Apparently, the XML's .../resource/script_server/host_cpu.xml and host_disk.xml require a pair of single quotes around the argument string:

Code: Select all

# diff host_cpu.xml host_cpu.xml.original
6c6
<       <arg_prepend>|host_hostname| |host_id| '|host_snmp_version|:|host_snmp_port|:|host_snmp_timeout|:|host_ping_retries|:|host_max_oids|:|host_snmp_community|:|host_snmp_username|:|host_snmp_password|:|host_snmp_auth_protocol|:|host_snmp_priv_passphrase|:|host_snmp_priv_protocol|:|host_snmp_context|'</arg_prepend>
---
>       <arg_prepend>|host_hostname| |host_id| |host_snmp_version|:|host_snmp_port|:|host_snmp_timeout|:|host_ping_retries|:|host_max_oids|:|host_snmp_community|:|host_snmp_username|:|host_snmp_password|:|host_snmp_auth_protocol|:|host_snmp_priv_passphrase|:|host_snmp_priv_protocol|:|host_snmp_context|</arg_prepend>


# diff host_disk.xml host_disk.xml.original
6c6
<       <arg_prepend>|host_hostname| |host_id| '|host_snmp_version|:|host_snmp_port|:|host_snmp_timeout|:|host_ping_retries|:|host_max_oids|:|host_snmp_community|:|host_snmp_username|:|host_snmp_password|:|host_snmp_auth_protocol|:|host_snmp_priv_passphrase|:|host_snmp_priv_protocol|:|host_snmp_context|'</arg_prepend>
---
>       <arg_prepend>|host_hostname| |host_id| |host_snmp_version|:|host_snmp_port|:|host_snmp_timeout|:|host_ping_retries|:|host_max_oids|:|host_snmp_community|:|host_snmp_username|:|host_snmp_password|:|host_snmp_auth_protocol|:|host_snmp_priv_passphrase|:|host_snmp_priv_protocol|:|host_snmp_context|</arg_prepend>

Now the arguments including the community string will appear in quotes in the query debug output, passing the arguments correctly.
It works!

Thanks for putting me on the right track!
(and please fix this in Cacti's next release)

I have this problem too.
Thank you for solving~!

[davidkillingsworth]

After some debugging, I noticed that the quotes I added to the XML reappear in the SNMP version when the ss_host_cpu.php script is called from the script server.
When called directly from the shell, the quotes are automatically removed, but when called from the script server they are not.

To sum up:

PROBLEM
Using a Linux Cacti server to monitor Windows hosts using the "SNMP - Get Mounted Partitions" and "SNMP - Get Processor Information" templates will fail if the SNMP community string contains a $ character due to incorrect quoting/escaping. This will prevent the data queries to get any rows.

WORKAROUND (quite horrible)
Modify .../resource/script_server/host_cpu.xml and host_disk.xml so that the host_snmp_community parameter is properly quoted (see above for diffs).
This workaround will allow the creation of the appropiate data sources and graphs, but will break data gathering for these sources. The poller does not call the scripts through the shell, but uses the script server which passes the quotes along with the rest of the parameters.
Therefore, after creation of the data sources, the quotes from the XML files should be removed again, followed by a cache rebuild of the poller (see System Utilities).

If somebody knows a nicer workaround, please let me know.
In the meantime I will file a bug report.

WOW! Thank You. I have a long string of random text as my community name, which does have a $ in it.

Cacti is running on Ubuntu 10.04 LTS
I'm polling a Windows 2008 server without an $ in the community name with no problem.
I was having trouble polling another Windows 2008 server with an $ in the community name and I could not get the following to produce any results, but was able to get "SNMP - Interface Statistics"

SNMP - Get Mounted Partitions
SNMP - Get Processor Information

After I removed the $ from the community name, the query were almost instantaneous and produced items and rows for both Get Mounted Partitions as well as Get Processor Information.

I would have never figured that it without this post.

Thanks!

[BSOD2600]

Is this still a problem in cacti 0.8.7i?