[HOWTO] Graphing Upstream Bandwidth by System User

[radicand]

Preface: Linux only
Limitations: needs iptables with 'owner' match support, needs a "clean" OUTPUT chain (though with small modifications this is not a limitation)

I have a few servers that basically lease out upstream bandwidth to a number of clients, and up until a few days ago, I could only see the interface traffic in Cacti. I had this crazy idea of breaking out graphs based on users, but couldn't find anything on how to do it. I hope this hacked-up-howto helps you out.

On my search I came across this excellent article by JM Roth on how to make graphs based on virtual interfaces, which hit close to what I wanted in practice but not quite. So in order to create system-user upsteam graphs, open JM's article and follow my modifications below:

He has a few bash scripts for different tasks. Rather than using his, you only need mine below:

iptables_user_eval.sh

Code: Select all

#!/bin/bash
[ $1 ] || exit 42;

case "$1" in

-g)
    iptables -vxnL OUTPUT | grep -v Chain | grep -v bytes | awk {'print $13'}
;;
-i)
    ipcount=`iptables -vxnL OUTPUT | grep -v Chain | grep -v bytes | awk {'print $2'} | wc -l`
    for i in `seq 1 $ipcount`; do
        echo $i;
    done
;;

-a)
    iptables -vxnL OUTPUT | grep -v Chain | grep -v bytes | awk {'print $2'}
;;

esac

Next, for each user on your system, create an iptables rule throwing them to the OUTPUT chain, where "1003" is the userid of a particular user (you can find uid's in /etc/passwd):

Code: Select all

iptables -A OUTPUT -m owner --uid-owner 1003 -j RETURN

After populating the chain, you'll want to move onto the SNMP configuration. I'm sure there's a much better non-SNMP method, but my servers I want data from are not the Cacti grapher machine. So SNMP it is.

At the bottom of your snmpd.conf, add the following (replacing the script path with the place you saved it):

Code: Select all

exec .1.3.6.1.4.1.2021.8.49 iplist /usr/share/snmp/iptables_user_eval.sh -g
exec .1.3.6.1.4.1.2021.8.49.42 ipindex /usr/share/snmp/iptables_user_eval.sh -i
exec .1.3.6.1.4.1.2021.8.50 traffic_in /usr/share/snmp/iptables_user_eval.sh -a
exec .1.3.6.1.4.1.2021.8.51 traffic_out /usr/share/snmp/iptables_user_eval.sh -a

Restart snmpd. You should be able to walk .1.3.6.1.4.1.2021.8 and see the userlist and bytestats. If not, now would be a good time to start troubleshooting. It's worth noting that JM uses a different OID in his howto. For whatever reason, I simply could not make it work with the ones he had listed. YMMV.

Now the hard part.

Add the following XML file to your snmp_queries subdirectory for your cacti installation (resource/snmp_queries/user_traffic.xml in my instance):

Code: Select all

<interface>
        <name>Get SNMP IPs</name>
        <description>Queries a host for a list of IPs with traffic monitored by iptables</description>
        <oid_index>.1.3.6.1.4.1.2021.8.49.42.101</oid_index>
        <oid_num_indexes>.1.3.6.1.4.1.2021.8.49.42.100.1</oid_num_indexes>
        <index_order>ipIP:ipIndex</index_order>
        <index_order_type>numeric</index_order_type>
        <index_title_format>|chosen_order_field|</index_title_format>

        <fields>
                <ipIndex>
                        <name>Index</name>
                        <method>walk</method>
                        <source>value</source>
                        <direction>input</direction>
                        <oid>.1.3.6.1.4.1.2021.8.49.42.101</oid>
                </ipIndex>
                <ipIP>
                        <name>IP Address</name>
                        <method>walk</method>
                        <source>value</source>
                        <direction>input</direction>
                        <oid>.1.3.6.1.4.1.2021.8.49.101</oid>
                </ipIP>
                <ipInBytes>
                        <name>Incoming Traffic</name>
                        <method>walk</method>
                        <source>value</source>
                        <direction>output</direction>
                        <oid>.1.3.6.1.4.1.2021.8.50.101</oid>
                </ipInBytes>
                <ipOutBytes>
                        <name>Outgoing Traffic</name>
                        <method>walk</method>
                        <source>value</source>
                        <direction>output</direction>
                        <oid>.1.3.6.1.4.1.2021.8.51.101</oid>
                </ipOutBytes>
        </fields>
</interface>

A few things to note now that you might have already been wondering about.
1) Why don't we have real downstream data?
A) We can not classify user-traffic on the downsteam side since we are unable to tell who it is going to when it enters the iptables chains.
2) Why are we then using the upsteam as the downstream here?
A) I wish there was a cleaner way to eliminate the extra OID and definition in the XML (and there likely is), but the only way I got it working was to do this for now. Don't fret, we'll modify the graph properly.
3) Were you really so lazy to only change the bare minimum from JM's XML?
A) Yes.

If you use JM's examples from here out, YMMV. I simply could not get anything to graph using his steps. So what I did is the below:

SKIP steps 3A and 3B in JM's howto. When you get to 3C, associate the dataquery xml (above instead of his) with the standard traffic graph. Pick your favorite. 3D is straightforward as is 3E.

When you finish making the graph, you'll want to immediately change the downsteam datasource to be that of your ethernet interface. Go into the Graph Management section, edit the graph, and make said change. Your graph should start looking normal now.

If you've actually managed to get things moving along, congratulations. I spent the better part of a day trying to figure it all out and make it work and I'm happy to say that I have succeeded.

From here out, you can start making any custom graph templates you want. See the below for one example. The top graph is the ethernet adapter traffic, followed by a combined graph showing two system users. Below that are the individual graphs for each user.