[solved] Physical Memory graph bugged for VM using dynamic

[joshvazquez]

I am using Windows Server 2008 R2 with Hyper-V to run VMs. I switched a particular VM to dynamic memory (http://technet.microsoft.com/en-us/libr ... 31766.aspx) today and immediately noticed something wrong with the Physical Memory graph. Physical memory was climbing to 6 GB, then 7 GB, then 8 GB, when the maximum for this VM is 5 GB!

The gap in the graph is the time at which dynamic memory was enabled. On the VM, I set the minimum memory to 512 MB and the maximum to 5120 MB. The blue graph should never be above 5.00 G here, and the red graph should be the actual RAM usage.

What I have determined from Hyper-V is that the memory demand for the VM is currently 8.85 GB. This is the amount that the VM is requesting to work with (this particular VM is RAM-hungry and we don't have enough for it). The VM only receives a maximum of 5.00 GB, so it seems that the graph is pulling the values for memory demand and something else instead of the actual capacity and usage values. Perhaps Windows uses nonstandard SNMP paths for these values or the SNMP poller is not dynamic-memory-aware.

I used the 0.8.8a all-in-one Cacti Windows installer.

[BSOD2600]

Not a Cacti problem.

This appears to be an issue with the way Microsoft decided to implement the counter in its SNMP agent. I'd guess there are more detailed memory counters for hyper-v/guest VMs. Possibly snmp-informant exposes this info or you'll have to write additional scripts to query it via WMI.

[joshvazquez]

Not a Cacti problem.

This appears to be an issue with the way Microsoft decided to implement the counter in its SNMP agent. I'd guess there are more detailed memory counters for hyper-v/guest VMs. Possibly snmp-informant exposes this info or you'll have to write additional scripts to query it via WMI.

Thanks for the info, I'll check that out.

[joshvazquez]

Not a Cacti problem.

This appears to be an issue with the way Microsoft decided to implement the counter in its SNMP agent. I'd guess there are more detailed memory counters for hyper-v/guest VMs. Possibly snmp-informant exposes this info or you'll have to write additional scripts to query it via WMI.

I've made a discovery.

Code: Select all

> snmpwalk -v 2c -c mykey myserver .1.3.6.1.2.1.25.2.3.1
...
HOST-RESOURCES-MIB::hrStorageDescr.7 = STRING: Physical Memory
HOST-RESOURCES-MIB::hrStorageAllocationUnits.7 = INTEGER: 65536 Bytes
HOST-RESOURCES-MIB::hrStorageSize.7 = INTEGER: 81898
HOST-RESOURCES-MIB::hrStorageUsed.7 = INTEGER: 77557
...

I'm assuming that I have to multiply the integer by 64k to get the actual value in bytes.

65536 * 81898 = 5367267328 B = 5118.6 MB ~= 5120 MB actual maximum RAM of the VM
65536 * 77557 = 5082775552 B = 4847.3 MB = approximately the current memory usage of the VM

To me, someone with very little knowledge of Cacti and SNMP, this seems to indicate that the values being reported by the Windows SNMP agent are correct. Please correct me if I'm wrong.

Thanks.

[BSOD2600]

To me, someone with very little knowledge of Cacti and SNMP, this seems to indicate that the values being reported by the Windows SNMP agent are correct. Please correct me if I'm wrong.

Sure looks correct.

[joshvazquez]

So is Cacti reporting it incorrectly, rrdtool, or something else? How should I troubleshoot this?

[BSOD2600]

Use a snmp tool like snmpwalk (net-snmp suite) and manually verify the memory sizes being reported. alternatively, you could increase the cacti logging level to medium which should show the raw values returned from the device. then when you're in doubt of the values being displayed in the cacti graph, you'd consult the log file and/or manual snmpwalk.

dynamic memory sounds to me that it would be expected the memory total/used would be constantly changing, which is what the graph reflects.

[joshvazquez]

Use a snmp tool like snmpwalk (net-snmp suite) and manually verify the memory sizes being reported.

Yes, I did this above and determined that the values being reported are correct (which you confirmed).

alternatively, you could increase the cacti logging level to medium which should show the raw values returned from the device. then when you're in doubt of the values being displayed in the cacti graph, you'd consult the log file and/or manual snmpwalk.

It looks like the wrong index is being polled. Data source 73 (virtual memory) is polling index 5, which is an optical drive. Data source 74 (physical memory) is polling index 6, which is virtual memory. They should be polling indexes 6 and 7 instead.

Code: Select all

...DS[73]...ss_host_disk...get used 5, output: 0
...DS[73]...get total 5, output: 0
...DS[74]...get used 6, output: 8818589696
...DS[74]...get total 6, output: 9637658624

So the virtual memory stats are being collected into the physical memory data source and displayed on the phyiscal memory graph, which explains why the virtual memory graph is blank. I tried deleting both the graph and data source for Virtual Memory, then recreating them. The new data source is still using the wrong index of 5 (although the index settings in the data source are Index Type = hrStorageDescr, Index Value = "Virtual Memory"). The client correctly reports the hrStorageDescr indexes as 6 and 7:

Code: Select all

HOST-RESOURCES-MIB::hrStorageDescr.5 = STRING: K:\
HOST-RESOURCES-MIB::hrStorageDescr.6 = STRING: Virtual Memory
HOST-RESOURCES-MIB::hrStorageDescr.7 = STRING: Physical Memory

Changing the index settings for the virtual memory data source to hrStorageIndex = 6, the virtual memory graph appears as it should. I can use this workaround for now, but any idea why it would be using the wrong index? I'd like to avoid this issue in the future and I might not notice if it happens to another client.

dynamic memory sounds to me that it would be expected the memory total/used would be constantly changing, which is what the graph reflects.

It should be constantly changing, but the maximum memory of the VM is 5 GB. The total memory of the VM can change from 512 MB to 5 GB, but not any higher.

[BSOD2600]

It looks like the wrong index is being polled. Data source 73 (virtual memory) is polling index 5, which is an optical drive. Data source 74 (physical memory) is polling index 6, which is virtual memory. They should be polling indexes 6 and 7 instead.

ah-ha, that would explain it.

What reindexing method do you have currently set? If the indexes change while the OS is up, the only option you have is to set it to Verify All Fields.

[joshvazquez]

It looks like the wrong index is being polled. Data source 73 (virtual memory) is polling index 5, which is an optical drive. Data source 74 (physical memory) is polling index 6, which is virtual memory. They should be polling indexes 6 and 7 instead.

ah-ha, that would explain it.

What reindexing method do you have currently set? If the indexes change while the OS is up, the only option you have is to set it to Verify All Fields.

Currently Uptime Goes Backwards.

Shouldn't it get the actual indexes when I delete the graph/data source and recreate them?

[BSOD2600]

Depends. Often you must manually refresh the indexes (click that green circle next to the data query), otherwise cacti will continue to use cached indexes in the poller cache (it only would refresh them in your case when the uptime changes).

[joshvazquez]

Depends. Often you must manually refresh the indexes (click that green circle next to the data query), otherwise cacti will continue to use cached indexes in the poller cache (it only would refresh them in your case when the uptime changes).

That seems to have worked. Thanks for your help.

[NCG]

If you change anything on the storage then the indexes change and this messes up your graphs.
I had this problem on Cluster nodes because the Quorum disk moves around the nodes.

It would be a good thing if Cacti can be configured to update the SNMP cache at regular moments.

[BSOD2600]

Change the reindex method to verify all fields for the Host MIB disk query on that device.