Disk Volumes changing OIDs

[BSOD2600]

What is the OID when its broken and after its fixed? The verbose query output of the data template will show the details without having to dig into the log file/poller cache.

[jblack]

The verbose query with the backup drive connected is:

Code: Select all

+ Running data query [8].
+ Found type = '6 '[script query].
+ Found data query XML file at 'E:/Cacti/www/resource/script_server/host_disk.xml'
+ XML file parsed ok.
+ Executing script for list of indexes 'c:\progra~1\php\php.exe -q E:\Cacti\www\scripts\ss_host_disk.php server1.mydomain.local 4 2:161:2000:1:10:public:::::: index'
+ Executing script query 'c:\progra~1\php\php.exe -q E:\Cacti\www\scripts\ss_host_disk.php server1.mydomain.local 4 2:161:2000:1:10:public:::::: query index'
+ Found item [hrStorageIndex='1'] index: 1
+ Found item [hrStorageIndex='2'] index: 2
+ Found item [hrStorageIndex='3'] index: 3
+ Found item [hrStorageIndex='4'] index: 4
+ Found item [hrStorageIndex='5'] index: 5
+ Found item [hrStorageIndex='6'] index: 6
+ Found item [hrStorageIndex='7'] index: 7
+ Executing script query 'c:\progra~1\php\php.exe -q E:\Cacti\www\scripts\ss_host_disk.php server1.mydomain.local 4 2:161:2000:1:10:public:::::: query description'
+ Found item [hrStorageDescr='A:'] index: 1
+ Found item [hrStorageDescr='C: Label: Serial Number d4b3954'] index: 2
+ Found item [hrStorageDescr='D:'] index: 3
+ Found item [hrStorageDescr='E: Label:Backup Serial Number 4c8808d'] index: 4
+ Found item [hrStorageDescr='F: Label:DATA Serial Number 98f1849'] index: 5
+ Found item [hrStorageDescr='Virtual Memory'] index: 6
+ Found item [hrStorageDescr='Physical Memory'] index: 7
+ Executing script query 'c:\progra~1\php\php.exe -q E:\Cacti\www\scripts\ss_host_disk.php server1.mydomain.local 4 2:161:2000:1:10:public:::::: query sau'
+ Found item [hrStorageAllocationUnits='0'] index: 1
+ Found item [hrStorageAllocationUnits='512'] index: 2
+ Found item [hrStorageAllocationUnits='0'] index: 3
+ Found item [hrStorageAllocationUnits='4096'] index: 4
+ Found item [hrStorageAllocationUnits='4096'] index: 5
+ Found item [hrStorageAllocationUnits='65536'] index: 6
+ Found item [hrStorageAllocationUnits='65536'] index: 7
+ Found data query XML file at 'E:/Cacti/www/resource/script_server/host_disk.xml'
+ Found data query XML file at 'E:/Cacti/www/resource/script_server/host_disk.xml'
+ Found data query XML file at 'E:/Cacti/www/resource/script_server/host_disk.xml'
+ Found data query XML file at 'E:/Cacti/www/resource/script_server/host_disk.xml'
+ Found data query XML file at 'E:/Cacti/www/resource/script_server/host_disk.xml'
+ Found data query XML file at 'E:/Cacti/www/resource/script_server/host_disk.xml'
+ Found data query XML file at 'E:/Cacti/www/resource/script_server/host_disk.xml'

This is a walk with the backup drive connected.

Code: Select all

C:\Windows\System32>snmpwalk -v 2c -c public server1.mydomain.local 1.3.6.1.2.1.25.2.3.1

HOST-RESOURCES-MIB::hrStorageIndex.1 = INTEGER: 1
HOST-RESOURCES-MIB::hrStorageIndex.2 = INTEGER: 2
HOST-RESOURCES-MIB::hrStorageIndex.3 = INTEGER: 3
HOST-RESOURCES-MIB::hrStorageIndex.4 = INTEGER: 4
HOST-RESOURCES-MIB::hrStorageIndex.5 = INTEGER: 5
HOST-RESOURCES-MIB::hrStorageIndex.6 = INTEGER: 6
HOST-RESOURCES-MIB::hrStorageIndex.7 = INTEGER: 7
HOST-RESOURCES-MIB::hrStorageType.1 = OID: HOST-RESOURCES-MIB::hrStorageTypes.5
HOST-RESOURCES-MIB::hrStorageType.2 = OID: HOST-RESOURCES-MIB::hrStorageTypes.4
HOST-RESOURCES-MIB::hrStorageType.3 = OID: HOST-RESOURCES-MIB::hrStorageTypes.7
HOST-RESOURCES-MIB::hrStorageType.4 = OID: HOST-RESOURCES-MIB::hrStorageTypes.4
HOST-RESOURCES-MIB::hrStorageType.5 = OID: HOST-RESOURCES-MIB::hrStorageTypes.4
HOST-RESOURCES-MIB::hrStorageType.6 = OID: HOST-RESOURCES-MIB::hrStorageTypes.3
HOST-RESOURCES-MIB::hrStorageType.7 = OID: HOST-RESOURCES-MIB::hrStorageTypes.2
HOST-RESOURCES-MIB::hrStorageDescr.1 = STRING: A:\
HOST-RESOURCES-MIB::hrStorageDescr.2 = STRING: C:\ Label:  Serial Number d4b39548
HOST-RESOURCES-MIB::hrStorageDescr.3 = STRING: D:\
HOST-RESOURCES-MIB::hrStorageDescr.4 = STRING: E:\ Label:Backup Serial Number 4c8808d1
HOST-RESOURCES-MIB::hrStorageDescr.5 = STRING: F:\ Label:DATA  Serial Number 98f18497
HOST-RESOURCES-MIB::hrStorageDescr.6 = STRING: Virtual Memory
HOST-RESOURCES-MIB::hrStorageDescr.7 = STRING: Physical Memory
HOST-RESOURCES-MIB::hrStorageAllocationUnits.1 = INTEGER: 0 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.2 = INTEGER: 512 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.3 = INTEGER: 0 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.4 = INTEGER: 4096 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.5 = INTEGER: 4096 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.6 = INTEGER: 65536 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.7 = INTEGER: 65536 Bytes
HOST-RESOURCES-MIB::hrStorageSize.1 = INTEGER: 0
HOST-RESOURCES-MIB::hrStorageSize.2 = INTEGER: 40965717
HOST-RESOURCES-MIB::hrStorageSize.3 = INTEGER: 0
HOST-RESOURCES-MIB::hrStorageSize.4 = INTEGER: 244192008
HOST-RESOURCES-MIB::hrStorageSize.5 = INTEGER: 174056242
HOST-RESOURCES-MIB::hrStorageSize.6 = INTEGER: 87583
HOST-RESOURCES-MIB::hrStorageSize.7 = INTEGER: 57309
HOST-RESOURCES-MIB::hrStorageUsed.1 = INTEGER: 0
HOST-RESOURCES-MIB::hrStorageUsed.2 = INTEGER: 37013547
HOST-RESOURCES-MIB::hrStorageUsed.3 = INTEGER: 0
HOST-RESOURCES-MIB::hrStorageUsed.4 = INTEGER: 232720253
HOST-RESOURCES-MIB::hrStorageUsed.5 = INTEGER: 129960341
HOST-RESOURCES-MIB::hrStorageUsed.6 = INTEGER: 54454
HOST-RESOURCES-MIB::hrStorageUsed.7 = INTEGER: 54107
HOST-RESOURCES-MIB::hrStorageAllocationFailures.1 = Counter32: 0
HOST-RESOURCES-MIB::hrStorageAllocationFailures.2 = Counter32: 0
HOST-RESOURCES-MIB::hrStorageAllocationFailures.3 = Counter32: 0
HOST-RESOURCES-MIB::hrStorageAllocationFailures.4 = Counter32: 0
HOST-RESOURCES-MIB::hrStorageAllocationFailures.5 = Counter32: 0
HOST-RESOURCES-MIB::hrStorageAllocationFailures.6 = Counter32: 0
HOST-RESOURCES-MIB::hrStorageAllocationFailures.7 = Counter32: 0

This verbose query with the backup drive disconnected.

Code: Select all

+ Running data query [8].
+ Found type = '6 '[script query].
+ Found data query XML file at 'E:/Cacti/www/resource/script_server/host_disk.xml'
+ XML file parsed ok.
+ Executing script for list of indexes 'c:\progra~1\php\php.exe -q E:\Cacti\www\scripts\ss_host_disk.php server1.mydomain.local 4 2:161:2000:1:10:public:::::: index'
+ Executing script query 'c:\progra~1\php\php.exe -q E:\Cacti\www\scripts\ss_host_disk.php server1.mydomain.local 4 2:161:2000:1:10:public:::::: query index'
+ Found item [hrStorageIndex='1'] index: 1
+ Found item [hrStorageIndex='2'] index: 2
+ Found item [hrStorageIndex='3'] index: 3
+ Found item [hrStorageIndex='4'] index: 4
+ Found item [hrStorageIndex='5'] index: 5
+ Found item [hrStorageIndex='6'] index: 6
+ Executing script query 'c:\progra~1\php\php.exe -q E:\Cacti\www\scripts\ss_host_disk.php server1.mydomain.local 4 2:161:2000:1:10:public:::::: query description'
+ Found item [hrStorageDescr='A:'] index: 1
+ Found item [hrStorageDescr='C: Label: Serial Number d4b3954'] index: 2
+ Found item [hrStorageDescr='D:'] index: 3
+ Found item [hrStorageDescr='F: Label:DATA Serial Number 98f1849'] index: 4
+ Found item [hrStorageDescr='Virtual Memory'] index: 5
+ Found item [hrStorageDescr='Physical Memory'] index: 6
+ Executing script query 'c:\progra~1\php\php.exe -q E:\Cacti\www\scripts\ss_host_disk.php server1.mydomain.local 4 2:161:2000:1:10:public:::::: query sau'
+ Found item [hrStorageAllocationUnits='0'] index: 1
+ Found item [hrStorageAllocationUnits='512'] index: 2
+ Found item [hrStorageAllocationUnits='0'] index: 3
+ Found item [hrStorageAllocationUnits='4096'] index: 4
+ Found item [hrStorageAllocationUnits='65536'] index: 5
+ Found item [hrStorageAllocationUnits='65536'] index: 6
+ Found data query XML file at 'E:/Cacti/www/resource/script_server/host_disk.xml'
+ Found data query XML file at 'E:/Cacti/www/resource/script_server/host_disk.xml'
+ Found data query XML file at 'E:/Cacti/www/resource/script_server/host_disk.xml'
+ Found data query XML file at 'E:/Cacti/www/resource/script_server/host_disk.xml'
+ Found data query XML file at 'E:/Cacti/www/resource/script_server/host_disk.xml'
+ Found data query XML file at 'E:/Cacti/www/resource/script_server/host_disk.xml'
+ Found data query XML file at 'E:/Cacti/www/resource/script_server/host_disk.xml'

This is the snmpwalk without the backup drive connected:

Code: Select all

C:\Windows\System32>snmpwalk -v 2c -c public server1.mydomain.local 1.3.6.1.2.1.25.2.3.1
HOST-RESOURCES-MIB::hrStorageIndex.1 = INTEGER: 1
HOST-RESOURCES-MIB::hrStorageIndex.2 = INTEGER: 2
HOST-RESOURCES-MIB::hrStorageIndex.3 = INTEGER: 3
HOST-RESOURCES-MIB::hrStorageIndex.4 = INTEGER: 4
HOST-RESOURCES-MIB::hrStorageIndex.5 = INTEGER: 5
HOST-RESOURCES-MIB::hrStorageIndex.6 = INTEGER: 6
HOST-RESOURCES-MIB::hrStorageType.1 = OID: HOST-RESOURCES-MIB::hrStorageTypes.5
HOST-RESOURCES-MIB::hrStorageType.2 = OID: HOST-RESOURCES-MIB::hrStorageTypes.4
HOST-RESOURCES-MIB::hrStorageType.3 = OID: HOST-RESOURCES-MIB::hrStorageTypes.7
HOST-RESOURCES-MIB::hrStorageType.4 = OID: HOST-RESOURCES-MIB::hrStorageTypes.4
HOST-RESOURCES-MIB::hrStorageType.5 = OID: HOST-RESOURCES-MIB::hrStorageTypes.3
HOST-RESOURCES-MIB::hrStorageType.6 = OID: HOST-RESOURCES-MIB::hrStorageTypes.2
HOST-RESOURCES-MIB::hrStorageDescr.1 = STRING: A:\
HOST-RESOURCES-MIB::hrStorageDescr.2 = STRING: C:\ Label:  Serial Number d4b39548
HOST-RESOURCES-MIB::hrStorageDescr.3 = STRING: D:\
HOST-RESOURCES-MIB::hrStorageDescr.4 = STRING: F:\ Label:DATA  Serial Number 98f18497
HOST-RESOURCES-MIB::hrStorageDescr.5 = STRING: Virtual Memory
HOST-RESOURCES-MIB::hrStorageDescr.6 = STRING: Physical Memory
HOST-RESOURCES-MIB::hrStorageAllocationUnits.1 = INTEGER: 0 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.2 = INTEGER: 512 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.3 = INTEGER: 0 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.4 = INTEGER: 4096 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.5 = INTEGER: 65536 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.6 = INTEGER: 65536 Bytes
HOST-RESOURCES-MIB::hrStorageSize.1 = INTEGER: 0
HOST-RESOURCES-MIB::hrStorageSize.2 = INTEGER: 40965717
HOST-RESOURCES-MIB::hrStorageSize.3 = INTEGER: 0
HOST-RESOURCES-MIB::hrStorageSize.4 = INTEGER: 174056242
HOST-RESOURCES-MIB::hrStorageSize.5 = INTEGER: 87583
HOST-RESOURCES-MIB::hrStorageSize.6 = INTEGER: 57309
HOST-RESOURCES-MIB::hrStorageUsed.1 = INTEGER: 0
HOST-RESOURCES-MIB::hrStorageUsed.2 = INTEGER: 37006196
HOST-RESOURCES-MIB::hrStorageUsed.3 = INTEGER: 0
HOST-RESOURCES-MIB::hrStorageUsed.4 = INTEGER: 129960341
HOST-RESOURCES-MIB::hrStorageUsed.5 = INTEGER: 54732
HOST-RESOURCES-MIB::hrStorageUsed.6 = INTEGER: 54431
HOST-RESOURCES-MIB::hrStorageAllocationFailures.1 = Counter32: 0
HOST-RESOURCES-MIB::hrStorageAllocationFailures.2 = Counter32: 0
HOST-RESOURCES-MIB::hrStorageAllocationFailures.3 = Counter32: 0
HOST-RESOURCES-MIB::hrStorageAllocationFailures.4 = Counter32: 0
HOST-RESOURCES-MIB::hrStorageAllocationFailures.5 = Counter32: 0
HOST-RESOURCES-MIB::hrStorageAllocationFailures.6 = Counter32: 0

It looks fairly normal. It takes the server a minute or so to update the SNMP statistics when the backup drive is disconnected. In this minute or so interium, the SNMPwalk is exactly the same as before the backup drive was disconnected.

What can I do next? Thanks so much for your help so far!

[BSOD2600]

All of that output looks normal.

In Data Sources -> select a device using the 'host mib hard drive space' template. Is the Index Type set to hrStorageDesc? If not, change that for a few devices since that should make Cacti track them via the description instead of index number.

I'm still surprised verify all fields doesn't work. You're running cacti 0.8.7g + patches, right? PIA 2.9?

[jblack]

In Data Sources -> select a device using the 'host mib hard drive space' template. Is the Index Type set to hrStorageDesc?

I picked a few of the data sources for the servers that the backup drives are on and all of them are set to "hrStorageDesc".

You're running cacti 0.8.7g + patches, right? PIA 2.9?

I'm running 0.8.7g . I've not installed any additional patches. I followed the guide to installing it for Windows. I downloaded the zip file for Windows and installed from that (About 3 weeks ago, it has been long enough, I don't know what guide I followed). It is definitely 0.8.7G as it says that version number. I don't think I did anything additional to patch.

Do I need to install any further patches?

I did some googling and found some patches here: http://www.cacti.net/downloads/patches/ ... e-patched/ . I haven't installed them but will if you think it will help.

I'm not using Plug-ins. Is it necessary that I do anything with PIA?

[BSOD2600]

Ah I had forgotten you used the windows installer, which contains the patches. Nope, no need for PIA 2.9 if you're not using plugins.

[jblack]

Ah I had forgotten you used the windows installer, which contains the patches. Nope, no need for PIA 2.9 if you're not using plugins.

So is there anything else I can do to help troubleshoot this problem? I appreciate your help so far. Thanks!

[jblack]

I started reading through the PHP code, and I think I know why the reindex method isn't working correctly.

In the cmd.php: Line 277

Code: Select all

				$reindex = db_fetch_assoc("select
					poller_reindex.data_query_id,
					poller_reindex.action,
					poller_reindex.op,
					poller_reindex.assert_value,
					poller_reindex.arg1
					from poller_reindex
					where poller_reindex.host_id=" . $item["host_id"]);

This happens right after we verify the host is up --- We then need to check if we need to do reindexing. So it runs this query against the DB to pull all the reindexing methods for each data query out. I did some digging and found that the host_id to one of the servers with the issue is 2.

So I ran select poller_reindex.data_query_id, poller_reindex.action, poller_reindex.op, poller_reindex.assert_value, poller_reindex.arg1 from poller_reindex where poller_reindex.host_id="2" and the result is:

Code: Select all

+---------------+--------+----+--------------+--------------------+
| data_query_id | action | op | assert_value | arg1               |
+---------------+--------+----+--------------+--------------------+
|             1 |      0 | <  | 149468090    | .1.3.6.1.2.1.1.3.0 |
|             9 |      0 | <  | 149468091    | .1.3.6.1.2.1.1.3.0 |
+---------------+--------+----+--------------+--------------------+

This only lists 2 data queries I need to reindex for ---- The server has 3 data queries associated with it. Processor, Network Interfaces, and Mounted Partitions.

I then ran the query: select * from host_snmp_query where host_id = 2;

This returned:

Code: Select all

+---------------+------------------+------------------------------+
| snmp_query_id | sort_field       | title_format                 |
+---------------+------------------+------------------------------+
|             1 | ifDescr          | |query_ifDescr|              |
|             8 | hrStorageDescr   | |query_hrStorageDescr|       |
|             9 | hrProcessorFrwID | CPU#|query_hrProcessorFrwID| |
+---------------+------------------+------------------------------+

So it looks like there isn't a reindex method in the database for query 8 which is associated with the hard drives. Any idea what I should have in the database for the reindex for the mounted partitions? Thanks!

[BSOD2600]

Nice detective work!

It appears your 'SNMP - Get Mounted Partitions' is using the PHP Script Server. As I look over the Cacti 0.8.7h beta1 changelog, I find the following:

-feature#0001924: Allow for reindex method "Index Count Changed" to apply to script (server) queries as well

which sounds exactly like the problem you're having. As I compare my ancient home Cacti installation, its SNMP - Get Mounted Partitions' still uses /script_queries/host_disk.xml. I think newer cacti installs must default to ss_host_disk.php while the legacy installations used host_disk.xml (http://docs.cacti.net/manual:087:3a_adv ... ta_queries validates this statement).

So, looks like you'll either need to sit and wait for cacti 0.8.7h to get released or revert to the older snmp implementation of host_disk.xml.

[jblack]

So, looks like you'll either need to sit and wait for cacti 0.8.7h to get released or revert to the older snmp implementation of host_disk.xml.

Or one could edit the PHP code to allow for reindexing of script server queries.

Normally you wouldn't have to edit the PHP, just insert data into the database for this to work, however the code for reindexing apparently deletes the entries in the poller_reindex table and doesn't reinsert them without a patch. I've patched the code so it works properly.

This is use at your OWN RISK! I've only tested it with cmd.php and with reindexing mounted partitions. Also be aware that I'm not using the script server to enumerate the mounted partitions (It didn't work for some reason), so I'm launching a separate instance of PHP. This added about 3s PER HOST to the runtime of the cmd.php. I'd suggest only adding this reindex method to hosts that will need it.

Anyway the file that needs patched is <CACTI INSTALL LOCATION>/lib/poller.php . The lines affected are at 211. We are going to change line 211 and then add 3 lines below it. The diff is as follows:

Code: Select all

211,214d210
<                       } else if ($data_query_type == DATA_INPUT_TYPE_QUERY_SCRIPT_SERVER) {
<                               if (isset($data_query_xml["arg_num_indexes"])) {
<                                       array_push($recache_stack, "insert into poller_reindex (host_id, data_query_id, action, op, assert_value, arg1) values ($host_id, $data_query_id, 1, '=', '$assert_value', '" . addslashes(read_config_option("path_php_binary")) . " -q " . addslashes(get_script_query_path((isset($data_query_xml["arg_prepend"]) ? $data_query_xml["arg_prepend"] . " ": "") . $data_query_xml["arg_num_indexes"], $data_query_xml["script_path"], $host_id)) . "')");
<                               }

If you don't want to add this in yourself, I've attached the PHP file below.

After you've patched the /lib/poller.php with the one below, you'll need to add database entries so reindexing occurs. From a MYSQL client run the following query:

Code: Select all

insert into poller_reindex (host_id, data_query_id, action, op, assert_value) VALUES (<HOST_ID>,8,1,'=',0);

Please note that the <HOST_ID> is the ID for host you want to query. You need to replace this with the HOST_ID. To find the host_id, in the GUI, click "Devices". The "ID" listed next to each device is the value that must be placed here. The next time the cmd.php runs, you should see something like: "05/05/2011 02:07:13 PM - PCOMMAND: Poller[0] Host[4] WARNING: Recache Event Detected for Host ". This is normal. After the host is recached the proper command should automatically be inserted into the database. If you'd like to check to make sure the value was rewritten to the database properly you can with this query:

Code: Select all

select * from poller_reindex where poller_reindex.host_id="<HOST_ID>";

Please DO NOT contact me for technical support for any of the previous mentioned patches or SQL queries.

[BSOD2600]

Nice.

Might as well file this as a bug and attach your patched code to it. Reference this post in the thread. Maybe the devs will want to include it.