host Disk Space & very large drives

julian924s · Post by **julian924s** » Wed Dec 16, 2009 8:03 am

Hi - wonder if someone could with getting my cacti install to show Disk Space for very large partitions. Got a host with a "/" partition total size of 120G which is showing fine. But our "/data" partition which is 21T shows as a 1.88G partition when using "ucd/net - Get Monitored Partitions" or 4.0T when using "SNMP - Get Mounted Partitions"?

Platform details are:

Cacti Version - 0.8.7d
Plugin Architecture - 2.4
Poller Type - Cactid v
Server Info - Linux 2.6.18-92.1.22.el5
Web Server - Apache/2.2.3 (CentOS)
PHP - 5.1.6
PHP Extensions - libxml, xml, wddx, tokenizer, sysvshm, sysvsem, sysvmsg, standard, SimpleXML, sockets, SPL, shmop, session, Reflection, pspell, posix, mime_magic, iconv, hash, gmp, gettext, ftp, exif, date, curl, ctype, calendar, bz2, zlib, pcre, openssl, apache2handler, dbase, gd, memcache, mssql, mysql, mysqli, PDO, pdo_mysql, pdo_sqlite, snmp
MySQL - 5.0.77-log
RRDTool - 1.3.8
SNMP - 5.3.2.2
Plugins
Global Plugin Settings (settings - v0.5)
Host Info (hostinfo - v0.2)
Device Monitoring (monitor - v0.8.2)
Syslog Monitoring (syslog - v0.5.2)
Network Tools (tools - v0.3)
NTop Viewer (ntop - v0.1)
PHP Network Weathermap (weathermap - v0.96a)
Thresholds (thold - v0.3.9)
RRD File Cleaner (RRD Cleaner - v0.38)

Ideally I'd like to know if there is a command line way to verify the size being returned and is there a setting somewhere within the query that is limited the size of the partition being seen?

Thanks in advance - J.

Post by **gandalf** » Wed Dec 16, 2009 2:13 pm

Please see 4th link of my sig to find an alternate way of doing this. Purely SNMP, but not tested against such large devices as I do not own any of them

R.

julian924s · Post by **julian924s** » Fri Dec 18, 2009 2:50 am

Thanks - tried that and it gave the same results, the /data partition is reported with a Total size of 4.40T. The host itself is nothing unusual, a Centos server with a couple of disk shelves to make up the 21T partition. I'll take a look at the SNMP stuff and see if I can get anything from that.

Julian.

Post by **gandalf** » Fri Dec 18, 2009 8:41 am

Please post an SNMPwalk output against that part of the HOST MIB, that will help me a lot
R.

Post by **TheWitness** » Fri Dec 18, 2009 9:50 pm

Yea, this one's been around for some time and I think there was a post relative to the whole thing being a snmp related issue and a bit beyond our control. But you have my attention at the moment.

TheWitness

julian924s · Post by **julian924s** » Sun Dec 20, 2009 12:00 pm

Thanks - I looked at at the hrStorageTable.xml file and did an snmpwalk against .1.3.6.1.2.1.25.2.3.1 and got the following:

HOST-RESOURCES-MIB::hrStorageIndex.1 = INTEGER: 1
HOST-RESOURCES-MIB::hrStorageIndex.2 = INTEGER: 2
HOST-RESOURCES-MIB::hrStorageIndex.3 = INTEGER: 3
HOST-RESOURCES-MIB::hrStorageIndex.4 = INTEGER: 4
HOST-RESOURCES-MIB::hrStorageIndex.5 = INTEGER: 5
HOST-RESOURCES-MIB::hrStorageType.1 = OID: HOST-RESOURCES-TYPES::hrStorageOther
HOST-RESOURCES-MIB::hrStorageType.2 = OID: HOST-RESOURCES-TYPES::hrStorageRam
HOST-RESOURCES-MIB::hrStorageType.3 = OID: HOST-RESOURCES-TYPES::hrStorageVirtualMemory
HOST-RESOURCES-MIB::hrStorageType.4 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk
HOST-RESOURCES-MIB::hrStorageType.5 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk
HOST-RESOURCES-MIB::hrStorageDescr.1 = STRING: Memory Buffers
HOST-RESOURCES-MIB::hrStorageDescr.2 = STRING: Real Memory
HOST-RESOURCES-MIB::hrStorageDescr.3 = STRING: Swap Space
HOST-RESOURCES-MIB::hrStorageDescr.4 = STRING: /
HOST-RESOURCES-MIB::hrStorageDescr.5 = STRING: /data
HOST-RESOURCES-MIB::hrStorageAllocationUnits.1 = INTEGER: 1024 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.2 = INTEGER: 1024 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.3 = INTEGER: 1024 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.4 = INTEGER: 4096 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.5 = INTEGER: 4096 Bytes
HOST-RESOURCES-MIB::hrStorageSize.1 = INTEGER: 4046816
HOST-RESOURCES-MIB::hrStorageSize.2 = INTEGER: 4046816
HOST-RESOURCES-MIB::hrStorageSize.3 = INTEGER: 8385920
HOST-RESOURCES-MIB::hrStorageSize.4 = INTEGER: 35675630
HOST-RESOURCES-MIB::hrStorageSize.5 = INTEGER: 1074233344
HOST-RESOURCES-MIB::hrStorageUsed.1 = INTEGER: 167000
HOST-RESOURCES-MIB::hrStorageUsed.2 = INTEGER: 588612
HOST-RESOURCES-MIB::hrStorageUsed.3 = INTEGER: 0
HOST-RESOURCES-MIB::hrStorageUsed.4 = INTEGER: 370364
HOST-RESOURCES-MIB::hrStorageUsed.5 = INTEGER: 3177

Is that sufficient?

The host in question has the following disk setup:

Filesystem Size Used Avail Use% Mounted on
/dev/sda1 137G 1.5G 128G 2% /
tmpfs 2.0G 0 2.0G 0% /dev/shm
/dev/sdb 21T 13M 21T 1% /data

Or from fdisk:
Disk /dev/sda: 159.4 GB, 159450660864 bytes
Disk /dev/sdb: 21992.3 GB, 21992380039168 bytes

Thanks again - Julian.

Post by **TheWitness** » Sun Dec 20, 2009 12:19 pm

Do me a favor, walk the entire tree (your whole device) and let's see what we get. I see what is happening. Note the following:

Total Disk Space = (2e32 + 1074233344) * 4096 ~ 21TBytes

So, the counter rolled over one complete rotation and then added 1074233344 on to it. So, there is likely another non-standard OID that tracks the number of times the counter has rolled over, in this case "1".

Were you walking with snmpv2 or snmpv1? If snmpv1, try snmpv2 for both requests.

TheWitness

julian924s · Post by **julian924s** » Sun Dec 20, 2009 12:39 pm

Here you go, I've removed some IP & Interface stuff as not relevant. The snmpwalk used was using V2c.

Thanks - J.

Post by **TheWitness** » Sun Dec 20, 2009 12:59 pm

Did you get any errors on the stderr pipe when doing this, like truncating to 32bit's warning messages?

TheWitness

Post by **TheWitness** » Sun Dec 20, 2009 1:59 pm

Since you have not gotten back, do the following in the shell before you perform the walk, and you can also limit the walk to the host disk.

Code: Select all

export MIBS=if-mib
snmpwalk -c <whatever> -v 2c <hosty> <host-disk-mib>

If net-snmp is using the mibfile to truncate the 64bit values to 32bit, the setting of MIBS=... will prevent that tuncation.

TheWitness

julian924s · Post by **julian924s** » Mon Dec 21, 2009 1:16 pm

Hi - no errors at all, running snmp v2. Not entirely sure if this is what you wanted but is this about right?

iso.3.6.1.2.1.25.2.3.1.1.1 = INTEGER: 1
iso.3.6.1.2.1.25.2.3.1.1.2 = INTEGER: 2
iso.3.6.1.2.1.25.2.3.1.1.3 = INTEGER: 3
iso.3.6.1.2.1.25.2.3.1.1.4 = INTEGER: 4
iso.3.6.1.2.1.25.2.3.1.1.5 = INTEGER: 5
iso.3.6.1.2.1.25.2.3.1.2.1 = OID: iso.3.6.1.2.1.25.2.1.1
iso.3.6.1.2.1.25.2.3.1.2.2 = OID: iso.3.6.1.2.1.25.2.1.2
iso.3.6.1.2.1.25.2.3.1.2.3 = OID: iso.3.6.1.2.1.25.2.1.3
iso.3.6.1.2.1.25.2.3.1.2.4 = OID: iso.3.6.1.2.1.25.2.1.4
iso.3.6.1.2.1.25.2.3.1.2.5 = OID: iso.3.6.1.2.1.25.2.1.4
iso.3.6.1.2.1.25.2.3.1.3.1 = STRING: "Memory Buffers"
iso.3.6.1.2.1.25.2.3.1.3.2 = STRING: "Real Memory"
iso.3.6.1.2.1.25.2.3.1.3.3 = STRING: "Swap Space"
iso.3.6.1.2.1.25.2.3.1.3.4 = STRING: "/"
iso.3.6.1.2.1.25.2.3.1.3.5 = STRING: "/data"
iso.3.6.1.2.1.25.2.3.1.4.1 = INTEGER: 1024
iso.3.6.1.2.1.25.2.3.1.4.2 = INTEGER: 1024
iso.3.6.1.2.1.25.2.3.1.4.3 = INTEGER: 1024
iso.3.6.1.2.1.25.2.3.1.4.4 = INTEGER: 4096
iso.3.6.1.2.1.25.2.3.1.4.5 = INTEGER: 4096
iso.3.6.1.2.1.25.2.3.1.5.1 = INTEGER: 4046816
iso.3.6.1.2.1.25.2.3.1.5.2 = INTEGER: 4046816
iso.3.6.1.2.1.25.2.3.1.5.3 = INTEGER: 8385920
iso.3.6.1.2.1.25.2.3.1.5.4 = INTEGER: 35675630
iso.3.6.1.2.1.25.2.3.1.5.5 = INTEGER: 1074233344
iso.3.6.1.2.1.25.2.3.1.6.1 = INTEGER: 199756
iso.3.6.1.2.1.25.2.3.1.6.2 = INTEGER: 2423516
iso.3.6.1.2.1.25.2.3.1.6.3 = INTEGER: 0
iso.3.6.1.2.1.25.2.3.1.6.4 = INTEGER: 385143
iso.3.6.1.2.1.25.2.3.1.6.5 = INTEGER: 194842

Post by **TheWitness** » Mon Dec 21, 2009 3:23 pm

Is the machine your are reading from a 32 or 64bit machine. What about the remote host? I'll go check a few things out at net-snmp.org too.

TheWitness

Post by **TheWitness** » Mon Dec 21, 2009 3:30 pm

Here is the open bug report from the Sourceforge.net bug databases for this problem. It shows open. Note the "truncate" warnings on the page. If those are happening client side, then we can fool the system the way I indicated. However, it's likely a hack.

http://sourceforge.net/tracker/?func=de ... tid=112694

TheWitness

julian924s · Post by **julian924s** » Mon Dec 21, 2009 5:13 pm

Hi - both the server I'm reading from and the remote host are 64bit, Centos 5 installs.

J.

Post by **TheWitness** » Tue Dec 22, 2009 10:20 am

To fix this, net-snmp must be patched. The net-snmp guys believe that the agent can support 64bit volumes np (real big). However, the way that the agent is behaving does not reflect this.

Before I go and fix net-snmp's agent, can you please update to 5.5, and see if the hrStorageAllocationUnits increases for this volume? If it does, and the problem goes away, then I know it's fixed.

My guess is not. When I talk upbout upgrading, it will be on the monitored node and not the Cacti server, ok? To me, the fix in the snmpd agent would be trivial, but I don't want to re-invent the wheel if it's already been done.

TheWitness

host Disk Space & very large drives

host Disk Space & very large drives

Who is online