Disk Volumes changing OIDs

[jblack]

Operating System: Windows 2003 R2 Server Standard 32-bit
Webserver: Apache 2.2.17
Cacti: .8.7G
Spine: .8.7G
MySQL: 5.5.11
PHP: 5.2.17
RRDTool (Cygwin or Win32 version): win32 1.2.X
Net-SNMP: 5.4.1.2
Cygwin (cygwin1.dll version): 1007.5.0.0
Plugin Architecture: Not using any plugins


We are currently testing out Cacti as a replacement for MRTG. We have some USB backup drives that we swap once a week for our backups. It appears that Cacti doesn't detect this and continues to graph the wrong information or nothing at all. We had the same problem with MRTG, but I fixed it with a Perl script that checks the description to ensure it is logging the proper volume.

I did a bit of searching and found someone who had a similar problem here: http://forums.cacti.net/viewtopic.php?f ... 76&start=0 . Quoting from that page:

To answer my own post. I seem to have stumbled upon a solution by accident.
Click on "Data Sources"
Pick the host in question
Click on the name of the partition in question
Under "Custom Data" change "Index Type" from "dskIndex" to "dskPath"

I checked on my server and the Index Type for the data source is set to "hrStorageDescr", which makes me think that Cacti doesn't check every time that the OIDs haven't changed. Does anyone have any suggestions on how I can fix this? Only a handful of the servers have the backup disks. Is there a way to have Cacti always check if the OIDs have changed?

I've attached a copy of a graph from one of the servers that has this issue.

Thanks!

[BSOD2600]

It depends on the index type and reindexing method used. more info on reindexing: http://docs.cacti.net/manual:087:6_refe ... ndex_hosts

Since your storage device keeps the same description, Cacti isnt detecting it should be reindexed. Either try changing the Index type to hrStorageIndex or you'll have to change the reindex method to verify all fields.

[jblack]

It depends on the index type and reindexing method used. more info on reindexing: http://docs.cacti.net/manual:087:6_refe ... ndex_hosts

Since your storage device keeps the same description, Cacti isnt detecting it should be reindexed. Either try changing the Index type to hrStorageIndex or you'll have to change the reindex method to verify all fields.

Now that option makes sense ----- I never knew or thought to think on what that meant. I switched it to "Index Count Changed", because when we unplug the USB drives, it should see one less drive. I'm sure it will work fine when we swap the drives again this week.

Thank you!

[jblack]

I guess I spoke too soon. The re-index method makes sense now, but it doesn't seem to work. I switched it to "Index Count Changed" and it doesn't detect when the OIDs change. I then tried "Verify All Fields" and it doesn't detect then either.


Does anyone know why these wouldn't work? Thank you!

[BSOD2600]

The cacti log with debug level set will reveal the details on why cacti isn't properly reindexing.

If you unplug and replugin a device between a polling cycle, the index count will remain the same and cacti won't (typically) do a reindex. verify all fields should've worked though....

[jblack]

If you unplug and replugin a device between a polling cycle, the index count will remain the same and cacti won't (typically) do a reindex. verify all fields should've worked though....

The hard drives are typically unplugged for about 2 hours, so Index Count Changed should be working.

I've attached the debug log for the poller. Server1 and Server2 are set to "Verify All Fields". The rest of the servers are set to "Index Count Changed". When I initially created all of these graphs I used "Uptime Goes Backwards" for the re-index method. Later, I deleted the data query from the devices and re-added them with the proper Re-Index Method. I never deleted the graphs or data files for the graphs. Should I have done that?

Thanks!

[BSOD2600]

I never deleted the graphs or data files for the graphs. Should I have done that?

No need. Deleting and re-adding the data query again will suffice.

I reviewed the cacti log, but it appears spine does something different with the output for determining if a reindex is needed. the cmd.php output (with debug logging) actually shows the details.

would it be possible to switch to cmd.php poller for a bit to see if it positively changes your issue?

[jblack]

would it be possible to switch to cmd.php poller for a bit to see if it positively changes your issue?

Sure. I've already switched to cmd.php . We switch backups on Friday, so I'll know then if this has fixed the issue (I'm thinking about unplugging them for an hour or so tomorrow so I know then). I don't think I'm polling enough devices to actually need spine, I just used it off the bat because it was more efficient. Thanks!

[jblack]

So I shut off one of the backup drives today. The physical memory dropped to 0 shortly thereafter telling me that it didn't detect that the drive OIDs had changed. I have the re-index method set to "Verify All Fields". Is there anything else we could try on this? Thanks!

[BSOD2600]

Have you migrated the script yet? read http://docs.cacti.net/manual:087:3a_adv ... ipt_server

Now since you're using cmd.php, can you post another log with debugging enabled?

[jblack]

Have you migrated the script yet? read http://docs.cacti.net/manual:087:3a_adv ... ipt_server

I'm only running 2 custom scripts to log pages on special printers. Both are Perl. Does this need to be done when I'm using the default scripts? I'm guessing I can't do this with perl scripts.

Now since you're using cmd.php, can you post another log with debugging enabled?

It should be attached. Server1 and Server2 are set to "Verify all fields" for the "SNMP - Get Mounted Partitions" data query. The rest of the servers are set to "Index Count Changed". Just so you know ---- Server2 is the server Cacti is running on (At least in this debug file --- I have to take server names/ IP addresses out due to paranoia).

Thanks!

[BSOD2600]

Naw, you can leave the perl scripts alone -- there is no perl script server in cacti.

On quick glance of the log file, it appears all is functioning properly now.

[jblack]

On quick glance of the log file, it appears all is functioning properly now.

In the log file maybe. The graph for physical memory dropped to 0 when I unplugged the backup drive. I'll wait and see what it does tomorrow, but I doubt it'll be different. Is there anything else we can do to troubleshoot this? --- If I go to the device page and refresh the query, it fixes itself then, but it would be nice if it were automatic. Thanks!

[BSOD2600]

We really need to catch the details in the log file to see why its breaking.

[jblack]

We really need to catch the details in the log file to see why its breaking.

What can I do? Let me know, I'm more than happy to help. I would give you access to the system, but it has some confidential data on it, so I'll do anything short of that. We swapped the backup drives today and it broke until I manually refreshed the query.

Thanks!