Linux Mount Point Problem?

[bcarpio]

I have an environment of about 70 servers AIX and RHEL 4.6 - 5.2 all hosts are displaying CPU, MEM, Load Average, Disk, Users etc.. perfectly except for ONE file system on one host (RHEL 4.6).

I have the following file system structure:

/opt - File System
/opt/app/oracle - File System

(well I have more then this but.. )

The /opt file system reports fine to Cacti through SNMP however /opt/app/oracle doesn't report through SNMP to cacti (and I'm not good enough to run an snmpwalk to see whats being reported to cacti). The file system isn't anything special it isn't OCFS2 (those actually report fine too) its just a standard LVM EXT3 file system but it reports as

Current: 0
Available: 0

When I log into the box I can see the file system is there, mounted its 50G and 30G available.

Is there some problem with SNMP on 4.6 and file systems mounted on other file systems?

Any advice?

Any idea what command string to use with snmpwalk to see whats being reported to cacti so I can try and troubleshoot that way?

Thanks.

[rony]

Check the snmpd.conf and make sure that you have it showing all volumes. I suspect that it is configured to just show that one volume.

[bcarpio]

I have the same exact snmpd.conf file pushed to all my hosts, I just double checked it and it included this:

includeAllDisks true

The server in question is also reporting the correct disk space for the following file systems:

/
/boot
/home
/opt
/var/log

However /opt/app/oracle continues to report
Current: 0
Available: 0

Unfortunately I didn't build this environment but this is the only server that has an /opt/app/oracle file system.

Any other ideas?

[rony]

How big is the volume in question.

This maybe a limitation of the agent.

[bcarpio]

/dev/mapper/VolGroup00-lv_optapporacle
50G 15G 33G 31% /opt/app/oracle

I don't think its a limit based on size, I have another system with a 1.6T file system and Caci and SNMP report on it just fine:

/dev/mapper/vg_oradata-lv_oradata
1.6T 8.0G 1.6T 1% /u01

[bcarpio]

Ok I have researched the problem a bit more.

When I do an snmpwalk server_name hrStorageSize I can see all my disks reporting EXCEPT FOR index 10 which is the /opt/app/oracle file system

HOST-RESOURCES-MIB::hrStorageSize.1 = INTEGER: 82437504
HOST-RESOURCES-MIB::hrStorageSize.2 = INTEGER: 82437504
HOST-RESOURCES-MIB::hrStorageSize.3 = INTEGER: 4194296
HOST-RESOURCES-MIB::hrStorageSize.4 = INTEGER: 1290144
HOST-RESOURCES-MIB::hrStorageSize.5 = INTEGER: 0
HOST-RESOURCES-MIB::hrStorageSize.6 = INTEGER: 0
HOST-RESOURCES-MIB::hrStorageSize.7 = INTEGER: 101086
HOST-RESOURCES-MIB::hrStorageSize.8 = INTEGER: 2580302
HOST-RESOURCES-MIB::hrStorageSize.9 = INTEGER: 3870460
HOST-RESOURCES-MIB::hrStorageSize.11 = INTEGER: 12901535
HOST-RESOURCES-MIB::hrStorageSize.12 = INTEGER: 2580302
HOST-RESOURCES-MIB::hrStorageSize.13 = INTEGER: 0
HOST-RESOURCES-MIB::hrStorageSize.14 = INTEGER: 0
HOST-RESOURCES-MIB::hrStorageSize.15 = INTEGER: 33042430
HOST-RESOURCES-MIB::hrStorageSize.16 = INTEGER: 30977278

See how 10 is missing?


I know hrStorageSize.10 should show up because of this

[root@ddcmgt005 rra]# snmpwalk -c COMMUNITYSTRING -v 1 server_name .1.3.6.1.4.1.2021.9.1.3
UCD-SNMP-MIB::dskDevice.1 = STRING: /dev/mapper/VolGroup00-lv_root
UCD-SNMP-MIB::dskDevice.2 = STRING: none
UCD-SNMP-MIB::dskDevice.3 = STRING: none
UCD-SNMP-MIB::dskDevice.4 = STRING: none
UCD-SNMP-MIB::dskDevice.5 = STRING: usbfs
UCD-SNMP-MIB::dskDevice.6 = STRING: /dev/sda1
UCD-SNMP-MIB::dskDevice.7 = STRING: none
UCD-SNMP-MIB::dskDevice.8 = STRING: /dev/mapper/VolGroup00-lv_home
UCD-SNMP-MIB::dskDevice.9 = STRING: /dev/mapper/VolGroup00-lv_opt
UCD-SNMP-MIB::dskDevice.10 = STRING: /dev/mapper/VolGroup00-lv_optapporacle
UCD-SNMP-MIB::dskDevice.11 = STRING: /dev/mapper/VolGroup00-lv_tmp
UCD-SNMP-MIB::dskDevice.12 = STRING: /dev/mapper/VolGroup00-lv_varlog


Then I checked the logs on the server that won't report disk space for /opt/app/oracle and I get this:

Jun 5 11:53:47 server_name snmpd[21640]: Couldn't open device /dev/mapper/VolGroup00-lv_optapporacle
Jun 5 11:53:47 server_name snmpd[21640]: statvfs dev/disk: Permission denied

Now I'm not sure why SNMP can't query the disk? Any suggestions?


EDIT: Ok, it helps when I read the log above the access denied log:

Jun 5 12:11:52 ddcerp004 kernel: audit(1212689512.811:20859): avc: denied { search } for pid=30908 comm="snmpd" name="app" dev=dm-2 ino=737281 scontext=root:system_r:snmpd_t tcontext=user_u:object_r:file_t tclass=dir

As you can see for some reason SELINUX is preventing access for SNMP to read this file system.

[phakesley]

I would take a look at your rrd file using rrdtool dump <rrd file> and look for the last update. Check the top of the output and you should see something like:

<ds>
<name> the name of the DS you are interested in.
......

Check that the max is set to NaN if not use

rrdtool tune <rrd file> --maximum <ds name>:U
this will set the maximum to NaN

Then wait for a couple of collection cycles and check the dump output for the DS. Check that last_ds and value have changed and that hopefull value now actually has some data in it rather than NaN which could be the problem.

Reason: Sometimes the DS is created with default vaules i.e. Min=0 and Max=100 so any value greater than 100 will not be stored in the RRD.

I had this problem a few weeks ago.

Regards,

Pete