Linux extended iostat template and scripts

[sebbs]

Same issue;
CentOS 5.5 x64 net-snmp-utils-5.3 / sysstat-7.0 / Cacti 8.7e

[/]# snmpwalk -v1 -c public MYBOX .1.3.6.1.3.1
End of MIB
[/]#

IOSTAT.cache
[/]# more /tmp/iostat.cache
Linux 2.6.18-164.9.1.el5 02/19/11

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.01 8.94 0.04 7.50 0.77 65.76 17.65 0.01 0.68 0.12 0.09
sda1 0.00 0.00 0.00 0.00 0.00 0.00 85.54 0.00 5.36 3.55 0.00
sda2 0.00 0.00 0.00 0.00 0.00 0.00 33.31 0.00 11.19 8.69 0.00
sda3 0.01 8.94 0.04 7.50 0.77 65.76 17.65 0.01 0.68 0.12 0.09
dm-0 0.00 0.00 0.05 5.38 0.68 21.52 8.18 0.01 1.22 0.06 0.03
dm-1 0.00 0.00 0.00 10.80 0.09 43.21 8.01 0.01 1.03 0.05 0.06
dm-2 0.00 0.00 0.00 0.25 0.00 0.98 8.02 0.00 0.68 0.31 0.01
dm-3 0.00 0.00 0.00 0.00 0.00 0.01 8.07 0.00 207.81 0.09 0.00
dm-4 0.00 0.00 0.00 0.01 0.00 0.04 8.00 0.00 0.55 0.12 0.00
hda 0.00 0.00 0.00 0.00 0.00 0.00 17.00 0.00 0.00 0.00 0.00

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 10.13 0.00 6.60 0.00 66.93 20.28 0.00 0.68 0.13 0.09
sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda3 0.00 10.13 0.00 6.60 0.00 66.93 20.28 0.00 0.68 0.13 0.09
dm-0 0.00 0.00 0.00 7.67 0.00 30.67 8.00 0.01 0.93 0.06 0.04
dm-1 0.00 0.00 0.00 7.37 0.00 29.47 8.00 0.01 1.09 0.05 0.03
dm-2 0.00 0.00 0.00 1.30 0.00 5.20 8.00 0.00 0.36 0.18 0.02
dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-4 0.00 0.00 0.00 0.40 0.00 1.60 8.00 0.00 0.83 0.17 0.01
hda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Seems to be polling....but the i get nothing from snmpwalk.

[taintedsquirrel]

For everyone with the "End of MIB" problems - please make sure you have modified the snmpd.conf file to allow the querying of the approperiate MIBs, for example and troubleshooting purposes only. This will give the community "public" full read access to the entire .1 MIB subtree.

Add this line to the snmpd.conf file -

view systemview included .1

[sunnyaj]

I am facing the same problem that many have faced.
However; the solution to that problem has not been suggested yet.

This is the output of the verbose query OR debug output

+ Running data query [15].
+ Found type = '3' [snmp query].
+ Found data query XML file at '/var/www/cacti/resource/snmp_queries/iostat.xml'
+ XML file parsed ok.
+ Executing SNMP walk for list of indexes @ '.1.3.6.1.3.1'
+ No SNMP data returned
+ Found data query XML file at '/var/www/cacti/resource/snmp_queries/iostat.xml'
+ Found data query XML file at '/var/www/cacti/resource/snmp_queries/iostat.xml'
+ Found data query XML file at '/var/www/cacti/resource/snmp_queries/iostat.xml'


snmpwalk from the CLI gives correct data.

SNMPv2-SMI::experimental.1.1.1 = INTEGER: 1
SNMPv2-SMI::experimental.1.1.2 = INTEGER: 2
SNMPv2-SMI::experimental.1.1.3 = INTEGER: 3
SNMPv2-SMI::experimental.1.1.4 = INTEGER: 4
SNMPv2-SMI::experimental.1.2.1 = STRING: "sda"
SNMPv2-SMI::experimental.1.2.2 = STRING: "sda1"
SNMPv2-SMI::experimental.1.2.3 = STRING: "sda2"
SNMPv2-SMI::experimental.1.2.4 = STRING: "sda3"
SNMPv2-SMI::experimental.1.3.1 = STRING: "0.00"
SNMPv2-SMI::experimental.1.3.2 = STRING: "0.00"
SNMPv2-SMI::experimental.1.3.3 = STRING: "0.00"
SNMPv2-SMI::experimental.1.3.4 = STRING: "0.00"
SNMPv2-SMI::experimental.1.4.1 = STRING: "1.80"
SNMPv2-SMI::experimental.1.4.2 = STRING: "0.00"
SNMPv2-SMI::experimental.1.4.3 = STRING: "0.00"

I can see logs suggesting snmp communication in /var/log/messages on my linux host


This is how my snmpd.conf looks

com2sec notConfigUser default public
group notConfigGroup v1 notConfigUser
group notConfigGroup v2c notConfigUser
view systemview included .1.3.6.1.2.1.1
view systemview included .1.3.6.1.2.1.25.1.1
view systemview included .1.3.6.1.3.1
access notConfigGroup "" any noauth exact systemview none none
syslocation Unknown (edit /etc/snmp/snmpd.conf)
syscontact Root <root@localhost> (configure /etc/snmp/snmp.local.conf)
pass .1.3.6.1.4.1.4413.4.1 /usr/bin/ucd5820stat
pass_persist .1.3.6.1.3.1 /usr/bin/perl /opt/BMIops/iostat-persist.pl

[mnnclub]

Yes, other script seems like fine but snmpwalk failed

But I could fix like this option for snmpd.conf you said, thanks a lot

view systemview included .1

[lshelton65gt]

I'm running into issues. On some machines the snmpwalk works fine. On others it does not. In the cases where it does not, I get nothing back. In all cases the cron job is creating the output file just fine. I've added the "view" option in the snmp.conf file and nothing. I've verified the the configurations across the machines are the same. The only thing I can think of is if there is some dependancy on the version of snmp.

I'm using varying versions of CentOS (4.8. 4.9, 5.5).

Any help would be GREATLY appreciated.

Thanks,
Lewis

[lshelton65gt]

OK, another issue.

Is there something special that I need to get this to work across a firewall. I have several machines that sit in a DMZ. I am currently gathering stats CPU, memory, etc on them. However the iostat is being stubborn. I can create the iostat graphs on the hosts, but they never show up in the Graph Management section. All machines outside the DMZ seem to work fine.

Any ideas?

Thanks,
Lewis

[Jericon]

I am facing the same problem that many have faced.
However; the solution to that problem has not been suggested yet.

This is the output of the verbose query OR debug output

+ Running data query [15].
+ Found type = '3' [snmp query].
+ Found data query XML file at '/var/www/cacti/resource/snmp_queries/iostat.xml'
+ XML file parsed ok.
+ Executing SNMP walk for list of indexes @ '.1.3.6.1.3.1'
+ No SNMP data returned
+ Found data query XML file at '/var/www/cacti/resource/snmp_queries/iostat.xml'
+ Found data query XML file at '/var/www/cacti/resource/snmp_queries/iostat.xml'
+ Found data query XML file at '/var/www/cacti/resource/snmp_queries/iostat.xml'

snmpwalk from the CLI gives correct data.

It looks like this is some kind of issue with cacti 0.8.7h. I tried it on a g instance and it worked just fine. Trying to see if it's something in the xml that can be tweaked, or if it is a bug in cacti itself.

[Jericon]

I am facing the same problem that many have faced.
However; the solution to that problem has not been suggested yet.

This is the output of the verbose query OR debug output

+ Running data query [15].
+ Found type = '3' [snmp query].
+ Found data query XML file at '/var/www/cacti/resource/snmp_queries/iostat.xml'
+ XML file parsed ok.
+ Executing SNMP walk for list of indexes @ '.1.3.6.1.3.1'
+ No SNMP data returned
+ Found data query XML file at '/var/www/cacti/resource/snmp_queries/iostat.xml'
+ Found data query XML file at '/var/www/cacti/resource/snmp_queries/iostat.xml'
+ Found data query XML file at '/var/www/cacti/resource/snmp_queries/iostat.xml'

snmpwalk from the CLI gives correct data.

It looks like this is some kind of issue with cacti 0.8.7h. I tried it on a g instance and it worked just fine. Trying to see if it's something in the xml that can be tweaked, or if it is a bug in cacti itself.

This is a bug in the iostats.xml file in cacti, that was probably just made this visible. In the iostats.xml file (the snmp_query file that is put in cacti) there is a field: index_order. ioName needs to be removed from it as that item does not exist.

OLD:

Code: Select all

<interface>
        <name>Get IOSTAT Devices</name>
        <description>Queries a host for a list of monitorable devices from iostat</description>
        <oid_index>.1.3.6.1.3.1</oid_index>
        <index_order>ioDescr:ioName:ioIndex</index_order>
        <index_order_type>numeric</index_order_type>

NEW:

Code: Select all

<interface>
        <name>Get IOSTAT Devices</name>
        <description>Queries a host for a list of monitorable devices from iostat</description>
        <oid_index>.1.3.6.1.3.1</oid_index>
        <index_order>ioDescr:ioIndex</index_order>
        <index_order_type>numeric</index_order_type>

[scobanx]

I cannot make pass_persist work on RHEL. I tried opening all debug information for snmpd but got nothing. It just does not work on RHEL. I tried to compile net-snmp bu results are same...

Anyone solved this issue?

[pwajda]

I followed installation instructions on Cacti 0.8.7i and I got this output when I do 'verbose query' on every data query derived from iostat template:

+ Running data query [10].
+ Found type = '3' [SNMP Query].
+ Found data query XML file at '/var/www/cacti/resource/snmp_queries/iostat.xml'
+ XML file parsed ok.
+ Invalid field <index_order>ioDescr:ioName:ioIndex</index_order>
+ Must contain <direction>input</direction> fields only

What's wrong?

[pwajda]

Solved.

Just in host to which those graphs belong to was no SNMP version defined...
My fault, but LOG should be more accurate

[chestr]

I followed installation instructions on Cacti 0.8.7i and I got this output when I do 'verbose query' on every data query derived from iostat template:

+ Running data query [10].
+ Found type = '3' [SNMP Query].
+ Found data query XML file at '/var/www/cacti/resource/snmp_queries/iostat.xml'
+ XML file parsed ok.
+ Invalid field <index_order>ioDescr:ioName:ioIndex</index_order>
+ Must contain <direction>input</direction> fields only

What's wrong?

i have this issue, but changing SNMP version not help(

[chestr]

I followed installation instructions on Cacti 0.8.7i and I got this output when I do 'verbose query' on every data query derived from iostat template:

+ Running data query [10].
+ Found type = '3' [SNMP Query].
+ Found data query XML file at '/var/www/cacti/resource/snmp_queries/iostat.xml'
+ XML file parsed ok.
+ Invalid field <index_order>ioDescr:ioName:ioIndex</index_order>
+ Must contain <direction>input</direction> fields only

What's wrong?

i have this issue, but changing SNMP version not help(

issue solved - in iostat.xml i changed ioDescr:ioName:ioIndex to ioDescr:ioIndex
founded on http://www.markround.com/archives/48-Li ... Cacti.html

[b0fh]

First, thanks for your scripts/templates. Works great on Solaris 11.

I am running a ZFS filesystem with a big number of disks in raid-z. Is there a way to monitor iostats for the whole zpool, too? At the moment I see iostats on a per-disk base.

Thanks!

[geoffmciver1]

For those of you with Centos problems.


this will be selinux.... and I've found that not all the audit messages are displayed in /var/log/audit/audit.log

I think there is a problem with the default selinux policy for snmpd with RHEL/centos...

heres the process I used to get it setup.

follow the instructions to setup...
test it via snmpwalks...

look in the audit log for problems.... they will look like below

Code: Select all

type=AVC msg=audit(1351798673.860:184072): avc:  denied  { read } for  pid=17694 comm="perl" name="iostat.pl" dev=sda3 ino=25593 scontext=unconfined_u:system_r:snmpd_t:s0 tcontext=unconfined_u:object_r:admin_home_t:s0 tclass=file
type=SYSCALL msg=audit(1351798673.860:184072): arch=c000003e syscall=2 success=no exit=-13 a0=ff6a90 a1=0 a2=1b6 a3=7fffe19f8ff0 items=0 ppid=17672 pid=17694 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=29351 comm="perl" exe="/usr/bin/perl" subj=unconfined_u:system_r:snmpd_t:s0 key=(null)
type=AVC msg=audit(1351798674.615:184073): avc:  denied  { read } for  pid=17695 comm="perl" name="iostat.pl" dev=sda3 ino=25593 scontext=unconfined_u:system_r:snmpd_t:s0 tcontext=unconfined_u:object_r:admin_home_t:s0 tclass=file
type=SYSCALL msg=audit(1351798674.615:184073): arch=c000003e syscall=2 success=no exit=-13 a0=10caa90 a1=0 a2=1b6 a3=7fff0d899c50 items=0 ppid=17672 pid=17695 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=29351 comm="perl" exe="/usr/bin/perl" subj=unconfined_u:system_r:snmpd_t:s0 key=(null)
type=AVC msg=audit(1351798677.730:184074): avc:  denied  { read } for  pid=17696 comm="perl" name="iostat.pl" dev=sda3 ino=25593 scontext=unconfined_u:system_r:snmpd_t:s0 tcontext=unconfined_u:object_r:admin_home_t:s0 tclass=file
type=SYSCALL msg=audit(1351798677.730:184074): arch=c000003e syscall=2 success=no exit=-13 a0=1d7ba90 a1=0 a2=1b6 a3=7fff8d8a9840 items=0 ppid=17672 pid=17696 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=29351 comm="perl" exe="/usr/bin/perl" subj=unconfined_u:system_r:snmpd_t:s0 key=(null)
type=AVC msg=audit(1351798678.464:184075): avc:  denied  { read } for  pid=17697 comm="perl" name="iostat.pl" dev=sda3 ino=25593 scontext=unconfined_u:system_r:snmpd_t:s0 tcontext=unconfined_u:object_r:admin_home_t:s0 tclass=file
type=SYSCALL msg=audit(1351798678.464:184075): arch=c000003e syscall=2 success=no exit=-13 a0=23e2a90 a1=0 a2=1b6 a3=7fff78eecbf0 items=0 ppid=17672 pid=17697 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=29351 comm="perl" exe="/usr/bin/perl" subj=unconfined_u:system_r:snmpd_t:s0 key=(null)
type=AVC msg=audit(1351798846.152:184132): avc:  denied  { open } for  pid=17765 comm="perl" name="iostat.pl" dev=sda3 ino=25593 scontext=unconfined_u:system_r:snmpd_t:s0 tcontext=unconfined_u:object_r:admin_home_t:s0 tclass=file
type=SYSCALL msg=audit(1351798846.152:184132): arch=c000003e syscall=2 success=no exit=-13 a0=1bd7a90 a1=0 a2=1b6 a3=7fff618f7b00 items=0 ppid=17672 pid=17765 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=29351 comm="perl" exe="/usr/bin/perl" subj=unconfined_u:system_r:snmpd_t:s0 key=(null)
type=AVC msg=audit(1351798846.157:184133): avc:  denied  { open } for  pid=17766 comm="perl" name="iostat.pl" dev=sda3 ino=25593 scontext=unconfined_u:system_r:snmpd_t:s0 tcontext=unconfined_u:object_r:admin_home_t:s0 tclass=file
type=SYSCALL msg=audit(1351798846.157:184133): arch=c000003e syscall=2 success=no exit=-13 a0=1235a90 a1=0 a2=1b6 a3=7fff3e6a6410 items=0 ppid=17672 pid=17766 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=29351 comm="perl" exe="/usr/bin/perl" subj=unconfined_u:system_r:snmpd_t:s0 key=(null)
type=AVC msg=audit(1351799006.534:184176): avc:  denied  { ioctl } for  pid=17824 comm="perl" path="/usr/local/bin/iostat.pl" dev=sda3 ino=25593 scontext=unconfined_u:system_r:snmpd_t:s0 tcontext=unconfined_u:object_r:admin_home_t:s0 tclass=file
type=SYSCALL msg=audit(1351799006.534:184176): arch=c000003e syscall=16 success=no exit=-13 a0=3 a1=5401 a2=7fff4f04a880 a3=48 items=0 ppid=17672 pid=17824 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=29351 comm="perl" exe="/usr/bin/perl" subj=unconfined_u:system_r:snmpd_t:s0 key=(null)
type=AVC msg=audit(1351799006.535:184177): avc:  denied  { getattr } for  pid=17824 comm="perl" path="/usr/local/bin/iostat.pl" dev=sda3 ino=25593 scontext=unconfined_u:system_r:snmpd_t:s0 tcontext=unconfined_u:object_r:admin_home_t:s0 tclass=file
type=SYSCALL msg=audit(1351799006.535:184177): arch=c000003e syscall=5 success=no exit=-13 a0=3 a1=f760a0 a2=f760a0 a3=7fff4f04a790 items=0 ppid=17672 pid=17824 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=29351 comm="perl" exe="/usr/bin/perl" subj=unconfined_u:system_r:snmpd_t:s0 key=(null)
type=AVC msg=audit(1351799006.538:184178): avc:  denied  { ioctl } for  pid=17825 comm="perl" path="/usr/local/bin/iostat.pl" dev=sda3 ino=25593 scontext=unconfined_u:system_r:snmpd_t:s0 tcontext=unconfined_u:object_r:admin_home_t:s0 tclass=file
type=SYSCALL msg=audit(1351799006.538:184178): arch=c000003e syscall=16 success=no exit=-13 a0=3 a1=5401 a2=7fff606601e0 a3=48 items=0 ppid=17672 pid=17825 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=29351 comm="perl" exe="/usr/bin/perl" subj=unconfined_u:system_r:snmpd_t:s0 key=(null)
type=AVC msg=audit(1351799006.538:184179): avc:  denied  { getattr } for  pid=17825 comm="perl" path="/usr/local/bin/iostat.pl" dev=sda3 ino=25593 scontext=unconfined_u:system_r:snmpd_t:s0 tcontext=unconfined_u:object_r:admin_home_t:s0 tclass=file
type=SYSCALL msg=audit(1351799006.538:184179): arch=c000003e syscall=5 success=no exit=-13 a0=3 a1=1a530a0 a2=1a530a0 a3=7fff606600f0 items=0 ppid=17672 pid=17825 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=29351 comm="perl" exe="/usr/bin/perl" subj=unconfined_u:system_r:snmpd_t:s0 key=(null)
type=AVC msg=audit(1351799157.759:184220): avc:  denied  { read } for  pid=17880 comm="perl" name="iostat.cache" dev=sda7 ino=77 scontext=unconfined_u:system_r:snmpd_t:s0 tcontext=system_u:object_r:system_cronjob_tmp_t:s0 tclass=file
type=SYSCALL msg=audit(1351799157.759:184220): arch=c000003e syscall=2 success=no exit=-13 a0=1d4afd0 a1=0 a2=1b6 a3=7fef6681cbe0 items=0 ppid=17672 pid=17880 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=29351 comm="perl" exe="/usr/bin/perl" subj=unconfined_u:system_r:snmpd_t:s0 key=(null)
type=AVC msg=audit(1351799157.769:184221): avc:  denied  { read } for  pid=17882 comm="perl" name="iostat.cache" dev=sda7 ino=77 scontext=unconfined_u:system_r:snmpd_t:s0 tcontext=system_u:object_r:system_cronjob_tmp_t:s0 tclass=file
type=SYSCALL msg=audit(1351799157.769:184221): arch=c000003e syscall=2 success=no exit=-13 a0=2087fd0 a1=0 a2=1b6 a3=7fcbfab2bbe0 items=0 ppid=17672 pid=17882 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=29351 comm="perl" exe="/usr/bin/perl" subj=unconfined_u:system_r:snmpd_t:s0 key=(null)

put the offending messages into a file....

use

Code: Select all

cat iostat-cacti | audit2allow -M iostat-cacti

to create a policy file...
import via

Code: Select all

semanage -i iostat-cacti.pp

test again... look for more messages and then .... rinse repeat.

if you get to a point where there are no more messages in the audit.log file and it still does not work then do this...

Code: Select all

semodule -DB

this will turn off the don't audit rules... "rules that don't get logged"

you will find that there are now more messages you need to use audit2allow on....

keep your file of messages you discover in the audit and just keep appending the new ones...

the reason why you have to keep doing this is that the process gets further each time you unblock a selinux road block.

selinux is great but it can be tricky to get a policy written...

when you are done you can turn the don't audit rules back on by

Code: Select all

semodule -B

good luck!