Linux TC QoS Queue Discipline Stats template

[Setarcos]

Hi Folks,

I have been playing with QoS rules on a Linux firewall on and off for the past few months, and gathered some tools for graphing throughput stats with Cacti. My setup is by no means complete, but I am posting it here in the hopes that someone else might find it useful.

Currently I only have queue discipline stats working, but have some (currently disabled) code to support classes as well. TC is used to retrieve stats from a Linux firewall and make them available via SNMP. The original developer's script didn't work with my versions of TC and Linux kernel so modifications were necessary: YMMV

This was developed on the following system:
CentOS 4 (RHEL4)
net-snmp 5.1.2
Cacti 0.8.7b
Kernel 2.6.9

Installation Instructions
-Copy snmp_persist.pl to /usr/local/sbin and ensure it is executable.
-Add the following line to snmpd.conf:

Code: Select all

pass_persist .1.3.6.1.4.1.2021.255 /usr/local/sbin/snmp_persist.pl

-Copy linux_qos.xml to th cacti/resource/snmp_queries directory
-From the Cacti web interface, load the cacti_host_template_linux_firewall_with_qos.xml host template.

Testing
-snmpwalk .1.3.6.1.4.1.2021.255 to ensure snmp_persist.pl is returning data. If it isn't, then please post the results of the following line:

Code: Select all

/sbin/tc -s -d qdisc show dev <DEVICE>

where <DEVICE> is the device with qdiscs defined (e.g. eth0)

[Setarcos]

Here is a custom aggregate graph that shows all of my queues using the data created using this template.

[DigitalXpert]

Hi There. I'd like to replicate what you have for monitoring the QoS on my server. However, I appear to be having trouble gathering the snmp data.

I have
pass_persist .1.3.6.1.4.1.2021.255 /usr/local/sbin/snmp_persist.pl
in my snmpd.conf file.

I've copied the template over and put the perl script in place making it +x.

When I run the verbose query from cacti I see:
+ Running data query [12].
+ Found type = '3' [snmp query].
+ Found data query XML file at '/var/www/html/admin/cacti/resource/snmp_queries/linux_qos.xml'
+ XML file parsed ok.
+ Executing SNMP walk for list of indexes @ '.1.3.6.1.4.1.2021.255.2.2.1'
+ No SNMP data returned
+ Found data query XML file at '/var/www/html/admin/cacti/resource/snmp_queries/linux_qos.xml'
+ Found data query XML file at '/var/www/html/admin/cacti/resource/snmp_queries/linux_qos.xml'
+ Found data query XML file at '/var/www/html/admin/cacti/resource/snmp_queries/linux_qos.xml'

I am shaping on eth2 and the output of
/sbin/tc -s -d qdisc show dev eth2
is:

qdisc htb 1: root r2q 10 default 20 direct_packets_stat 1 ver 3.17
Sent 24891912385 bytes 52195896 pkt (dropped 0, overlimits 12563597 requeues 1)
rate 0bit 0pps backlog 0b 3p requeues 1
qdisc sfq 10: parent 1:10 limit 127p quantum 1514b flows 127/1024 perturb 10sec
Sent 1741450922 bytes 26323347 pkt (dropped 0, overlimits 0 requeues 0)
rate 0bit 0pps backlog 0b 0p requeues 0
qdisc sfq 20: parent 1:20 limit 127p quantum 1514b flows 127/1024 perturb 10sec
Sent 14252231412 bytes 19286686 pkt (dropped 0, overlimits 0 requeues 1)
rate 0bit 0pps backlog 234b 3p requeues 1
qdisc sfq 30: parent 1:30 limit 127p quantum 1514b flows 127/1024 perturb 10sec
Sent 8898228737 bytes 1270849 pkt (dropped 0, overlimits 0 requeues 0)
rate 0bit 0pps backlog 0b 0p requeues 0
qdisc ingress ffff: parent ffff:fff1 ----------------
Sent 44508129430 bytes 56096512 pkt (dropped 2864, overlimits 0 requeues 0)
rate 0bit 0pps backlog 0b 0p requeues 0


Any thoughts where I may have gone wrong?

[Setarcos]

Hi DigitalXpert,

Can you restart snmpd, then snmpwalk .1.3.6.1.4.1.2021 on this box and let me know if you get anything back?

[mates007]

Hi DigitalXpert,
Can you restart snmpd, then snmpwalk .1.3.6.1.4.1.2021 on this box and let me know if you get anything back?

The same problem here.
Debian etch 2.6.24-etchnhalf.1-486

Code: Select all

/etc/snmp/snmpd.conf:
pass_persist .1.3.6.1.4.1.2021.255 /usr/local/sbin/snmp_persist.pl

# /sbin/tc -s -d qdisc show dev eth0
qdisc htb 1: root r2q 10 default 12 direct_packets_stat 18 ver 3.17
 Sent 1973833719 bytes 2758403 pkt (dropped 0, overlimits 1818462 requeues 0)
 rate 0bit 0pps backlog 0b 18p requeues 0
qdisc sfq 11: parent 1:11 limit 127p quantum 1514b flows 127/1024 perturb 10sec
 Sent 604422 bytes 10682 pkt (dropped 0, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0
qdisc sfq 12: parent 1:12 limit 127p quantum 1514b flows 127/1024 perturb 10sec
 Sent 1755893504 bytes 2139365 pkt (dropped 0, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 6706b 9p requeues 0
qdisc sfq 13: parent 1:13 limit 127p quantum 1514b flows 127/1024 perturb 10sec
 Sent 217313473 bytes 608338 pkt (dropped 0, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 3942b 9p requeues 0

# snmpwalk -v 1 -c pass 192.168.10.1 .1.3.6.1.4.1.2021.255
Returns nothing immediately

#/usr/local/sbin/snmp_persist.pl
Runs, but nothing happens, I canceled it after 60 seconds

# snmpwalk -v 1 -c pass 192.168.10.1 .1.3.6.1.4.1.2021
Returns a lot of stuff

[Setarcos]

Looks like something with the script may not be working properly.

Try running it manually from the command line. e.g:

Code: Select all

[brudy@set ~]$ /usr/local/sbin/snmp_persist.pl
getnext

Note: That last line is a <CR>

Should give you something like this:

Code: Select all

.1.3.6.1.4.1.2021.255.2.2.0.1
string
eth0

When things are working, it should also give you return values like these:

Code: Select all

getnext
.1.3.6.1.4.1.2021.255.2.2.1.1
.1.3.6.1.4.1.2021.255.2.2.1.2
string
sfq 10

Let me know what your system returns.

[DigitalXpert]

I ran

Code: Select all

snmpwalk localhost .1.3.6.1.4.1.2021

and all I got was

Code: Select all

snmpwalk: No securityName specified

I also tried running

Code: Select all

/usr/local/sbin/snmp_persist.pl
getnext

This resulted in no output at all. I'm pretty new at snmp. Perhaps I do not have my snmpd correct?

Here is my snmpd.conf

Code: Select all

com2sec local localhost private
com2sec mynetwork 10.0.0.0/24 public

group MyROGroup v1 mynetwork
group MyROGroup v2c mynetwork
group MyROGroup v1 local
group MyROGroup v2c local

view all included .1 80

access MyROGroup "" any noauth exact all none none
access MyRWGroup "" any noauth exact all all none

syslocation Unknown (edit /etc/snmp/snmpd.conf)
syscontact Root <root@localhost> (configure /etc/snmp/snmp.local.conf)

pass .1.3.6.1.4.1.4413.4.1 /usr/bin/ucd5820stat

pass_persist .1.3.6.1.4.1.2021.255   /usr/local/sbin/snmp_persist.pl

[Setarcos]

I ran

Code: Select all

snmpwalk localhost .1.3.6.1.4.1.2021

and all I got was

Code: Select all

snmpwalk: No securityName specified

You are missing the community name and SNMP version. For your config try:
snmpwalk -v2c -cprivate localhost .1.3.6.1.4.1.2021

I also tried running

Code: Select all

/usr/local/sbin/snmp_persist.pl
getnext

This resulted in no output at all. I'm pretty new at snmp. Perhaps I do not have my snmpd correct?

Did you hit enter one more time after the 'getnext' line?

Here is my snmpd.conf

Code: Select all

com2sec local localhost private
com2sec mynetwork 10.0.0.0/24 public

group MyROGroup v1 mynetwork
group MyROGroup v2c mynetwork
group MyROGroup v1 local
group MyROGroup v2c local

view all included .1 80

access MyROGroup "" any noauth exact all none none
access MyRWGroup "" any noauth exact all all none

syslocation Unknown (edit /etc/snmp/snmpd.conf)
syscontact Root <root@localhost> (configure /etc/snmp/snmp.local.conf)

pass .1.3.6.1.4.1.4413.4.1 /usr/bin/ucd5820stat

pass_persist .1.3.6.1.4.1.2021.255   /usr/local/sbin/snmp_persist.pl

This looks like a default config. You probably want to do some reading to set this up properly.


I attached a version of snmp_persist that will allow you to do some additional debugging. Try sticking this in a temporary directory and making it executable. On my firewall box, I get the following:

Code: Select all

[me@box tmp]$ ./snmp_persist-test.pl
Found NIC=lo
Found NIC=eth0
Found NIC=eth1
Found NIC=eth2
Found NIC=eth4
Found NIC=eth3
Found NIC=sit0
getnext

Matched 60048673 bytes 114234 pkts 0 dropped 49829 overlimits 0 requeues
Matched 673291 bytes 5941 pkts 0 dropped 0 overlimits 0 requeues
Matched 0 bytes 0 pkts 0 dropped 0 overlimits 0 requeues
Matched 4997551 bytes 25051 pkts 0 dropped 0 overlimits 0 requeues
Matched 0 bytes 0 pkts 0 dropped 0 overlimits 0 requeues
Matched 0 bytes 0 pkts 0 dropped 0 overlimits 0 requeues
Matched 54377831 bytes 83242 pkts 0 dropped 0 overlimits 0 requeues
.1.3.6.1.4.1.2021.255.2.2.0.1
string
eth0

[crazy72]

This would be great if an aggregate graph template should be createf for Cisco Qos.


Thx a lot for ur work.

Crazy

[z3rgl1ng]

Hi, i'm having some problems with my cacti setup now.
After i imported the new template i get this error on all interfaces:


Data Query [SNMP - Interface Statistics]
Error in data query.

When i try to edit Data Queries [edit: SNMP - Interface Statistics] i see this message:

Could not locate XML file.

Really strange

Any help would be apreciated.


------------------------------------------------------------------

Solved, after i imported the template my path to cacti changed, don't know why.

What i had in Data Queries [edit: SNMP - Interface Statistics]:
path_cacti/resource/snmp_queries/interface.xml

What should be in Data Queries [edit: SNMP - Interface Statistics]:
<path_cacti>/resource/snmp_queries/interface.xml

The same was for the new interfaces too...

This was tested on Gentoo 64bit

-------------------------------------------------------------------

The next problem is i don't get any output after NIC detection with the debugging script:

/root/monitor/snmp_persist-test.pl
Found NIC=eth1
Found NIC=eth0
Found NIC=lo
Found NIC=teql0
Found NIC=eth0
Found NIC=eth0

And the strange part is that it detects 3 eth0 cards...

--------------------------------------------------------------------

Ok solved this one too in a way:

Code: Select all

Use of uninitialized value in numeric gt (>) at /usr/local/sbin/snmp_persist.pl line 249, <STDIN> line 2.

This is what i'm getting after i run your script with the apropriate <CR>...
And i don't get it one time, it dosen't stops, i've let it run for 30sec and then ctrl +c...

And this is the message when i am running your debugging script:

Code: Select all

Matched  bytes  pkts  dropped  overlimits  requeues
Use of uninitialized value in numeric gt (>) at ./snmp_persist-test.pl line 251, <STDIN> line 2.
Use of uninitialized value in numeric gt (>) at ./snmp_persist-test.pl line 251, <STDIN> line 2.
Use of uninitialized value in numeric gt (>) at ./snmp_persist-test.pl line 251, <STDIN> line 2.
Use of uninitialized value in numeric gt (>) at ./snmp_persist-test.pl line 251, <STDIN> line 2.
Use of uninitialized value in numeric gt (>) at ./snmp_persist-test.pl line 251, <STDIN> line 2.
Use of uninitialized value in concatenation (.) or string at ./snmp_persist-test.pl line 142, <STDIN> line 2.
Use of uninitialized value in concatenation (.) or string at ./snmp_persist-test.pl line 142, <STDIN> line 2.
Use of uninitialized value in concatenation (.) or string at ./snmp_persist-test.pl line 142, <STDIN> line 2.
Use of uninitialized value in concatenation (.) or string at ./snmp_persist-test.pl line 142, <STDIN> line 2.
Use of uninitialized value in concatenation (.) or string at ./snmp_persist-test.pl line 142, <STDIN> line 2.

[z3rgl1ng]

New stuff, it shows 3 eth0 interfaces because i'm using vlans.
Now the other problem i don't get it why i'm getting lots of:

Code: Select all

Use of uninitialized value in numeric gt (>) at /usr/local/sbin/snmp_persist.pl line 249, <STDIN> line 2.

[Setarcos]

New stuff, it shows 3 eth0 interfaces because i'm using vlans.
Now the other problem i don't get it why i'm getting lots of:

Code: Select all

Use of uninitialized value in numeric gt (>) at /usr/local/sbin/snmp_persist.pl line 249, <STDIN> line 2.

It looks like the regex isn't matching your tc output for some reason.

Uncomment line 143 in snmp_persist-test.pl:

Code: Select all

print "Line->$line<-\n";

Also, if you could post the output of:
/sbin/tc -s -d qdisc show dev <DEVICE>

where <DEVICE> is the device with qdiscs defined (e.g. eth0)

This should help a bit.

[z3rgl1ng]

Well i have a lot of tc rules, since i have a lot of clients

Here is the start of the output you requested:

Code: Select all

qdisc htb 1: r2q 10 default 15 direct_packets_stat 0 ver 3.17
 Sent 44471262863 bytes 71982489 pkt (dropped 863530, overlimits 73233717 requeues 0)
 rate 0bit 0pps backlog 0b 353p requeues 0
qdisc pfifo 15: parent 1:15 limit 5p
 Sent 2285210 bytes 32444 pkt (dropped 135, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0
qdisc sfq 100: parent 1:100 limit 128p quantum 1514b flows 128/1024 perturb 10sec
 Sent 4694615 bytes 28858 pkt (dropped 0, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0
qdisc sfq 101: parent 1:101 limit 128p quantum 1514b flows 128/1024 perturb 10sec
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0
qdisc sfq 102: parent 1:102 limit 128p quantum 1514b flows 128/1024 perturb 10sec
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0

And here is the result after i uncomented that line:

Code: Select all

Matched  bytes  pkts  dropped  overlimits  requeues
Line-> Sent 31730 bytes 121 pkt (dropped 0, overlimits 0 requeues 0) <-
Use of uninitialized value in numeric gt (>) at ./monitor/snmp_persist-test.pl line 251, <STDIN> line 2.
Use of uninitialized value in numeric gt (>) at ./monitor/snmp_persist-test.pl line 251, <STDIN> line 2.
Use of uninitialized value in numeric gt (>) at ./monitor/snmp_persist-test.pl line 251, <STDIN> line 2.
Use of uninitialized value in numeric gt (>) at ./monitor/snmp_persist-test.pl line 251, <STDIN> line 2.
Use of uninitialized value in numeric gt (>) at ./monitor/snmp_persist-test.pl line 251, <STDIN> line 2.
Use of uninitialized value in concatenation (.) or string at ./monitor/snmp_persist-test.pl line 142, <STDIN> line 2.
Use of uninitialized value in concatenation (.) or string at ./monitor/snmp_persist-test.pl line 142, <STDIN> line 2.
Use of uninitialized value in concatenation (.) or string at ./monitor/snmp_persist-test.pl line 142, <STDIN> line 2.
Use of uninitialized value in concatenation (.) or string at ./monitor/snmp_persist-test.pl line 142, <STDIN> line 2.
Use of uninitialized value in concatenation (.) or string at ./monitor/snmp_persist-test.pl line 142, <STDIN> line 2.
Matched  bytes  pkts  dropped  overlimits  requeues
Line-> Sent 32440 bytes 128 pkt (dropped 0, overlimits 0 requeues 0) <-
Use of uninitialized value in numeric gt (>) at ./monitor/snmp_persist-test.pl line 251, <STDIN> line 2.
Use of uninitialized value in numeric gt (>) at ./monitor/snmp_persist-test.pl line 251, <STDIN> line 2.
Use of uninitialized value in numeric gt (>) at ./monitor/snmp_persist-test.pl line 251, <STDIN> line 2.
Use of uninitialized value in numeric gt (>) at ./monitor/snmp_persist-test.pl line 251, <STDIN> line 2.
Use of uninitialized value in numeric gt (>) at ./monitor/snmp_persist-test.pl line 251, <STDIN> line 2.
Use of uninitialized value in concatenation (.) or string at ./monitor/snmp_persist-test.pl line 142, <STDIN> line 2.
Use of uninitialized value in concatenation (.) or string at ./monitor/snmp_persist-test.pl line 142, <STDIN> line 2.
Use of uninitialized value in concatenation (.) or string at ./monitor/snmp_persist-test.pl line 142, <STDIN> line 2.
Use of uninitialized value in concatenation (.) or string at ./monitor/snmp_persist-test.pl line 142, <STDIN> line 2.
Use of uninitialized value in concatenation (.) or string at ./monitor/snmp_persist-test.pl line 142, <STDIN> line 2.

[Setarcos]

Matched bytes pkts dropped overlimits requeues
Line-> Sent 32440 bytes 128 pkt (dropped 0, overlimits 0 requeues 0) <-

Looks like your version of tc outputs 'pkt' instead of 'pkts' which is what the regex expected. Try the attached version of snmp_persist-test.pl that has an updated regex that should match both cases.

[z3rgl1ng]

Ok, that one works, but now i have a new problem, i replaced the /usr/local/sbin/snmp_... script with the one you posted now

The problem is cacti, i receive this error messages in error log when i add: Linux - QoS Qdisc

Code: Select all

11/26/2008 09:03:02 PM - PCOMMAND: Poller[0] Host[3] WARNING: Recache Event Detected for Host
11/26/2008 09:03:02 PM - SYSTEM STATS: Time:1.1201 Method:cmd.php Processes:1 Threads:N/A Hosts:7 HostsPerProcess:7 DataSources:32 RRDsProcessed:17
11/26/2008 09:03:01 PM - CMDPHP: Poller[0] Host[3] DS[35] WARNING: Result from SERVER not valid. Partial Result: U
11/26/2008 09:03:01 PM - CMDPHP: Poller[0] ASSERT: '6853263<319' failed. Recaching host '93.xxx.xx.xx', data query #9
11/26/2008 09:03:01 PM - CMDPHP: Poller[0] ASSERT: '6853263<319' failed. Recaching host '93.xxx.xx.xx', data query #8
11/26/2008 09:03:01 PM - CMDPHP: Poller[0] ASSERT: '6853262<317' failed. Recaching host '93.xxx.xx.xx', data query #1 

When i run it in debug mod it shows this:

Code: Select all

+ Running data query [11].
+ Found type = '3' [snmp query].
+ Found data query XML file at '/var/www/grafic/htdocs/cacti/resource/snmp_queries/linux_qos.xml'
+ XML file parsed ok.
+ Executing SNMP walk for list of indexes @ '.1.3.6.1.4.1.2021.255.2.2.1'
+ No SNMP data returned
+ Found data query XML file at '/var/www/grafic/htdocs/cacti/resource/snmp_queries/linux_qos.xml'
+ Found data query XML file at '/var/www/grafic/htdocs/cacti/resource/snmp_queries/linux_qos.xml'
+ Found data query XML file at '/var/www/grafic/htdocs/cacti/resource/snmp_queries/linux_qos.xml'