Graphing output from shell script using NET-SNMP-EXTEND-MIB

[sauce]

Hey everyone, long time Cacti user, first time poster. Would anyone have any advice on how to handle this?

I am trying to graph disk utilization of some large filesystems (20TB+). As I understand it, as of version 5.6, netsnmp is unable to handle this out of the box. So I have an idea to do it with snmpd's "extend" clause by invoking a shell script to do "df -k" or the like.

I imagine the output of the script would look something like this:

Code: Select all

# df -k | tail +2
/dev/md/dsk/d0       53957990 5739325 47679086    11%    /
/devices                   0       0       0     0%    /devices
ctfs                       0       0       0     0%    /system/contract
proc                       0       0       0     0%    /proc
mnttab                     0       0       0     0%    /etc/mnttab
swap                 16690560    1616 16688944     1%    /etc/svc/volatile
objfs                      0       0       0     0%    /system/object
sharefs                    0       0       0     0%    /etc/dfs/sharetab
/platform/sun4u-us3/lib/libc_psr/libc_psr_hwcap1.so.1
                     53957990 5739325 47679086    11%    /platform/sun4u-us3/lib/libc_psr.so.1
/platform/sun4u-us3/lib/sparcv9/libc_psr/libc_psr_hwcap1.so.1
                     53957990 5739325 47679086    11%    /platform/sun4u-us3/lib/sparcv9/libc_psr.so.1
fd                         0       0       0     0%    /dev/fd
swap                 16689160     216 16688944     1%    /tmp
swap                 16689040      96 16688944     1%    /var/run
macdata2             15061745664 11131043894 3930638063    74%    /macdata2
macdata1             20082327552 10527551399 9554715596    53%    /macdata1
macdata3             15061745664 3709858909 11351861901    25%    /macdata3
macdata4             20082327552 3961614389 16069684006    20%    /macdata4
macdata4/MCIS_Dump   20082327552 51019986 16069684006     1%    /macdata4/MCIS_Dump

And I would make a shell script and put the following in snmpd.conf:

Code: Select all

# cat /usr/local/share/snmp/snmpd.conf | grep extend
extend disk_utilization /usr/local/bin/disk_utilization.sh

And it would produce the following output when walking the NET-SNMP-EXTEND-MIB:

Code: Select all

$ snmpwalk -v2c -c secret fileserver NET-SNMP-AGENT-MIB::nsExtensions
NET-SNMP-EXTEND-MIB::nsExtendNumEntries.0 = INTEGER: 1
NET-SNMP-EXTEND-MIB::nsExtendCommand."zfs_utilization" = STRING: /usr/local/bin/disk_utilization.sh
NET-SNMP-EXTEND-MIB::nsExtendArgs."zfs_utilization" = STRING: 
NET-SNMP-EXTEND-MIB::nsExtendInput."zfs_utilization" = STRING: 
NET-SNMP-EXTEND-MIB::nsExtendCacheTime."zfs_utilization" = INTEGER: 5
NET-SNMP-EXTEND-MIB::nsExtendExecType."zfs_utilization" = INTEGER: exec(1)
NET-SNMP-EXTEND-MIB::nsExtendRunType."zfs_utilization" = INTEGER: run-on-read(1)
NET-SNMP-EXTEND-MIB::nsExtendStorage."zfs_utilization" = INTEGER: permanent(4)
NET-SNMP-EXTEND-MIB::nsExtendStatus."zfs_utilization" = INTEGER: active(1)
NET-SNMP-EXTEND-MIB::nsExtendOutput1Line."zfs_utilization" = STRING: Filesystem            kbytes    used   avail capacity  Mounted on
NET-SNMP-EXTEND-MIB::nsExtendOutputFull."zfs_utilization" = STRING: Filesystem            kbytes    used   avail capacity  Mounted on
/dev/md/dsk/d0       53957990 5739325 47679086    11%    /
/devices                   0       0       0     0%    /devices
ctfs                       0       0       0     0%    /system/contract
proc                       0       0       0     0%    /proc
mnttab                     0       0       0     0%    /etc/mnttab
swap                 16690200    1616 16688584     1%    /etc/svc/volatile
objfs                      0       0       0     0%    /system/object
sharefs                    0       0       0     0%    /etc/dfs/sharetab
/platform/sun4u-us3/lib/libc_psr/libc_psr_hwcap1.so.1 53957990 5739325 47679086    11%    /platform/sun4u-us3/lib/libc_psr.so.1
/platform/sun4u-us3/lib/sparcv9/libc_psr/libc_psr_hwcap1.so.1 53957990 5739325 47679086    11%    /platform/sun4u-us3/lib/sparcv9/libc_psr.so.1
fd                         0       0       0     0%    /dev/fd
swap                 16688800     216 16688584     1%    /tmp
swap                 16688680      96 16688584     1%    /var/run
zfsdata2             15061745664 11131043894 3930638063    74%    /zfsdata2
zfsdata1             20082327552 10527551399 9554715596    53%    /zfsdata1
zfsdata3             15061745664 3709858909 11351861901    25%    /zfsdata3
zfsdata4             20082327552 3961614389 16069684006    20%    /zfsdata4
zfsdata4/temp   20082327552 51019986 16069684006     1%    /zfsdata4/temp
NET-SNMP-EXTEND-MIB::nsExtendOutNumLines."zfs_utilization" = INTEGER: 19
NET-SNMP-EXTEND-MIB::nsExtendResult."zfs_utilization" = INTEGER: 0
NET-SNMP-EXTEND-MIB::nsExtendOutLine."zfs_utilization".1 = STRING: Filesystem            kbytes    used   avail capacity  Mounted on
NET-SNMP-EXTEND-MIB::nsExtendOutLine."zfs_utilization".2 = STRING: /dev/md/dsk/d0       53957990 5739325 47679086    11%    /
NET-SNMP-EXTEND-MIB::nsExtendOutLine."zfs_utilization".3 = STRING: /devices                   0       0       0     0%    /devices
NET-SNMP-EXTEND-MIB::nsExtendOutLine."zfs_utilization".4 = STRING: ctfs                       0       0       0     0%    /system/contract
NET-SNMP-EXTEND-MIB::nsExtendOutLine."zfs_utilization".5 = STRING: proc                       0       0       0     0%    /proc
NET-SNMP-EXTEND-MIB::nsExtendOutLine."zfs_utilization".6 = STRING: mnttab                     0       0       0     0%    /etc/mnttab
NET-SNMP-EXTEND-MIB::nsExtendOutLine."zfs_utilization".7 = STRING: swap                 16690200    1616 16688584     1%    /etc/svc/volatile
NET-SNMP-EXTEND-MIB::nsExtendOutLine."zfs_utilization".8 = STRING: objfs                      0       0       0     0%    /system/object
NET-SNMP-EXTEND-MIB::nsExtendOutLine."zfs_utilization".9 = STRING: sharefs                    0       0       0     0%    /etc/dfs/sharetab
NET-SNMP-EXTEND-MIB::nsExtendOutLine."zfs_utilization".10 = STRING: /platform/sun4u-us3/lib/libc_psr/libc_psr_hwcap1.so.1 53957990 5739325 47679086    11%    /platform/sun4u-us3/lib/libc_psr.so.1
NET-SNMP-EXTEND-MIB::nsExtendOutLine."zfs_utilization".11 = STRING: /platform/sun4u-us3/lib/sparcv9/libc_psr/libc_psr_hwcap1.so.1 53957990 5739325 47679086    11%    /platform/sun4u-us3/lib/sparcv9/libc_psr.so.1
NET-SNMP-EXTEND-MIB::nsExtendOutLine."zfs_utilization".12 = STRING: fd                         0       0       0     0%    /dev/fd
NET-SNMP-EXTEND-MIB::nsExtendOutLine."zfs_utilization".13 = STRING: swap                 16688800     216 16688584     1%    /tmp
NET-SNMP-EXTEND-MIB::nsExtendOutLine."zfs_utilization".14 = STRING: swap                 16688680      96 16688584     1%    /var/run
NET-SNMP-EXTEND-MIB::nsExtendOutLine."zfs_utilization".15 = STRING: zfsdata2             15061745664 11131043894 3930638063    74%    /zfsdata2
NET-SNMP-EXTEND-MIB::nsExtendOutLine."zfs_utilization".16 = STRING: zfsdata1             20082327552 10527551399 9554715596    53%    /zfsdata1
NET-SNMP-EXTEND-MIB::nsExtendOutLine."zfs_utilization".17 = STRING: zfsdata3             15061745664 3709858909 11351861901    25%    /zfsdata3
NET-SNMP-EXTEND-MIB::nsExtendOutLine."zfs_utilization".18 = STRING: zfsdata4             20082327552 3961614389 16069684006    20%    /zfsdata4
NET-SNMP-EXTEND-MIB::nsExtendOutLine."zfs_utilization".19 = STRING: zfsdata4/temp   20082327552 51019986 16069684006     1%    /zfsdata4/temp
$ 

[noname]

Was Net-SNMP v5.5 (and above) unable to handle 64-bit counters?
-> Net-SNMP : CHANGES
[PATCH 2449210]: add 64-bit disk usage statistics to UCD-SNMP-MIB::dskTable

I don't know whether appropriate templates which fit your purpose exist or not.
But I think, you had better to retrieve information partially (ex. per partition), than getting all at once.

For example, hmm.. well..

If you use 'sh' directive in snmpd.conf as follows,

Code: Select all

sh /            /bin/df -k /                 | /bin/awk 'NR==2{print $3}'
sh volatile     /bin/df -k /etc/svc/volatile | /bin/awk 'NR==2{print $3}'
sh macdata1     /bin/df -k /macdata1         | /bin/awk 'NR==2{print $3}'
sh macdata2     /bin/df -k /macdata2         | /bin/awk 'NR==2{print $3}'
sh macdata3     /bin/df -k /macdata3         | /bin/awk 'NR==2{print $3}'
sh macdata4     /bin/df -k /macdata4         | /bin/awk 'NR==2{print $3}'

Then (I guess) you can get disk usage by snmpwalk like as..

Code: Select all

% snmpwalk -v2c -c public localhost extTable
UCD-SNMP-MIB::extIndex.1 = INTEGER: 1
UCD-SNMP-MIB::extIndex.2 = INTEGER: 2
UCD-SNMP-MIB::extIndex.3 = INTEGER: 3
UCD-SNMP-MIB::extIndex.4 = INTEGER: 4
UCD-SNMP-MIB::extIndex.5 = INTEGER: 5
UCD-SNMP-MIB::extIndex.6 = INTEGER: 6
UCD-SNMP-MIB::extNames.1 = STRING: /
UCD-SNMP-MIB::extNames.2 = STRING: volatile
UCD-SNMP-MIB::extNames.3 = STRING: macdata1
UCD-SNMP-MIB::extNames.4 = STRING: macdata2
UCD-SNMP-MIB::extNames.5 = STRING: macdata3
UCD-SNMP-MIB::extNames.6 = STRING: macdata4
UCD-SNMP-MIB::extCommand.1 = STRING: /bin/df -k /                 | /bin/awk 'NR==2{print $3}'
UCD-SNMP-MIB::extCommand.2 = STRING: /bin/df -k /etc/svc/volatile | /bin/awk 'NR==2{print $3}'
UCD-SNMP-MIB::extCommand.3 = STRING: /bin/df -k /macdata1         | /bin/awk 'NR==2{print $3}'
UCD-SNMP-MIB::extCommand.4 = STRING: /bin/df -k /macdata2         | /bin/awk 'NR==2{print $3}'
UCD-SNMP-MIB::extCommand.5 = STRING: /bin/df -k /macdata3         | /bin/awk 'NR==2{print $3}'
UCD-SNMP-MIB::extCommand.6 = STRING: /bin/df -k /macdata4         | /bin/awk 'NR==2{print $3}'
UCD-SNMP-MIB::extResult.1 = INTEGER: 0
UCD-SNMP-MIB::extResult.2 = INTEGER: 0
UCD-SNMP-MIB::extResult.3 = INTEGER: 0
UCD-SNMP-MIB::extResult.4 = INTEGER: 0
UCD-SNMP-MIB::extResult.5 = INTEGER: 0
UCD-SNMP-MIB::extResult.6 = INTEGER: 0
UCD-SNMP-MIB::extOutput.1 = STRING: 5739325
UCD-SNMP-MIB::extOutput.2 = STRING: 1616
UCD-SNMP-MIB::extOutput.3 = STRING: 11131043894
UCD-SNMP-MIB::extOutput.4 = STRING: 10527551399
UCD-SNMP-MIB::extOutput.5 = STRING: 3709858909
UCD-SNMP-MIB::extOutput.6 = STRING: 3961614389

(To show OID by numeric, add '-On')

Now you can retrieve these information by SNMP from Cacti server.
Let's make data query by using 'Get SNMP Data (Indexed)' like as "ucd/net - Get Monitored Partitions".

But this is only my assumption, please not ask me detailed procedure.. ;)

[sauce]

I was wondering about that 64bit dskTable myself. Unfortunately I'm unable to get anything useful out of it, as I posted above. I'm probably doing it wrong.

Since this "extend script" has to be deployed on many machines, it would be much better to have just one reporting output from a single "df -k". That would be my ultimate goal (after getting it to work out of the box, that is).

[noname]

Output from plain 'df -k' command is too complex to handle via SNMP,
then I think you need to customize your script more ease to interpret..

At least, single OID should return single type of information.
Please see 'Scripts and Templates' forum for other samples.

[sauce]

Output from plain 'df -k' command is too complex to handle via SNMP,
then I think you need to customize your script more ease to interpret..

At least, single OID should return single type of information.
Please see 'Scripts and Templates' forum for other samples.

Thanks, I sort of figured that it wasn't good practice do do "df" output in one query, I just needed to hear someone confirm it. I will seek other ways to handle this. Thanks!