wrong network interface stats

hemry · Post by **hemry** » Thu Apr 12, 2007 12:07 pm

I've got running cacti on windows 2003 machine, it graphs cpu/disk usage and other statistic from linux hosts perfectly, however network statistics are somehow broken cacti graphs them, but they're just wrong, terribly wrong, instead of ~10mbits traffic it graphs 200, it even mixes incoming and outgoing figures. There are no errors in log file and cacti actually gets counters, I checked if i was polling right interface maybe 10 times, tried to specify in both ways by ifIndex or ifDescr still result is the same.

P.S. Just to assure you its not a byte/bit problem, i know the difference and i tried both, also its not the 32bit/64 bit issue counters are 32bit for sure (however tried both too).

BSOD2600 · Post by **BSOD2600** » Thu Apr 12, 2007 6:43 pm

Post a picture of the graph and the graph debug (click the little wrench icon). How are you sure the traffic values are wrong -- what other source are you validating against?

hemry · Post by **hemry** » Fri Apr 13, 2007 3:02 am

I'm sure graphs they are wrong, i'm validating against mrtg graphs. On eth1 there are almost no traffic this interface is only for management eg snmp and ssh it just cant be few hundred mbps.

Graph debug output:

RRDTool Command:

c:/cacti/rrdtool.exe graph - \
--imgformat=PNG \
--start=-86400 \
--end=-300 \
--title="news - Traffic - xxx.xxx.xx.xx (eth1)" \
--rigid \
--base=1000 \
--height=120 \
--width=500 \
--alt-autoscale-max \
--lower-limit=0 \
--vertical-label="bits per second" \
--slope-mode \
DEF:a="C\:/cacti/rra/news_traffic_in_75.rrd":traffic_in:AVERAGE \
DEF:b="C\:/cacti/rra/news_traffic_in_75.rrd":traffic_out:AVERAGE \
CDEF:cdefa=a,8,* \
CDEF:cdefe=b,8,* \
AREA:cdefa#00CF00:"Inbound" \
GPRINT:cdefa:LAST:" Current\:%8.2lf %s" \
GPRINT:cdefa:AVERAGE:"Average\:%8.2lf %s" \
GPRINT:cdefa:MAX:"Maximum\:%8.2lf %s\n" \
LINE1:cdefe#002A97:"Outbound" \
GPRINT:cdefe:LAST:"Current\:%8.2lf %s" \
GPRINT:cdefe:AVERAGE:"Average\:%8.2lf %s" \
GPRINT:cdefe:MAX:"Maximum\:%8.2lf %s"

RRDTool Says:

OK

RRDTool Command:

c:/cacti/rrdtool.exe graph - \
--imgformat=PNG \
--start=-86400 \
--end=-300 \
--title="news - Traffic - xxx.xx.x.xx (eth0)" \
--rigid \
--base=1000 \
--height=120 \
--width=500 \
--alt-autoscale-max \
--lower-limit=0 \
--vertical-label="bits per second" \
--slope-mode \
DEF:a="C\:/cacti/rra/news_traffic_in_77.rrd":traffic_in:AVERAGE \
DEF:b="C\:/cacti/rra/news_traffic_in_77.rrd":traffic_out:AVERAGE \
CDEF:cdefa=a,8,* \
CDEF:cdefe=b,8,* \
AREA:cdefa#00CF00:"Inbound" \
GPRINT:cdefa:LAST:" Current\:%8.2lf %s" \
GPRINT:cdefa:AVERAGE:"Average\:%8.2lf %s" \
GPRINT:cdefa:MAX:"Maximum\:%8.2lf %s\n" \
LINE1:cdefe#002A97:"Outbound" \
GPRINT:cdefe:LAST:"Current\:%8.2lf %s" \
GPRINT:cdefe:AVERAGE:"Average\:%8.2lf %s" \
GPRINT:cdefe:MAX:"Maximum\:%8.2lf %s"

RRDTool Says:

OK

BSOD2600 · Post by **BSOD2600** » Fri Apr 13, 2007 2:18 pm

Post the configuration mrtg is using to collect the data. Then look in cacti's snmp cache for the device in question on what OIDs its polling for the interface data.

hemry · Post by **hemry** » Fri Apr 13, 2007 3:11 pm

mrtg configuration :
Target[127.0.0.1_eth0]: \eth0:snmpfe@127.0.0.1:
SetEnv[127.0.0.1_eth0]: MRTG_INT_DESCR="eth0"
MaxBytes[127.0.0.1_eth0]: 12500000
Title[127.0.0.1_eth0]: Traffic Analysis for eth0
PageTop[127.0.0.1_eth0]: <h1>Traffic Analysis for eth0</h1>

mrtg is running on linux host localy, and i'm trying to poll with remote windows 2003 machine with cacti.

From poller cache:
fe106 - Traffic - xxx.xxx.xx.106/eth0 SNMP Version: 1, Community: public, OID: .1.3.6.1.2.1.2.2.1.10.2
RRD: c:\cacti\rra\fe106_traffic_in_56.rrd

fe106 - Traffic - xxx.xxx.xx.106/eth0 SNMP Version: 1, Community: public, OID: .1.3.6.1.2.1.2.2.1.16.2
RRD: c:\cacti\rra\fe106_traffic_in_56.rrd

In data source of device in question i use indexed snmp query
Index type ifDescr
Index valye eth0

BSOD2600 · Post by **BSOD2600** » Fri Apr 13, 2007 5:08 pm

Well mrtg looks to be pointing at the same OID that cacti is. Please post a mrtg graph and cacti graph. If the data truly is over 100mb, then you must use snmpv2 and 64bit templates.

hemry · Post by **hemry** » Fri Apr 13, 2007 5:16 pm

No data aint over 100mbit, the intarface is barely used at all.

Graphs are bellow, as you can see it just doesnt make sense...

BSOD2600 · Post by **BSOD2600** » Fri Apr 13, 2007 7:59 pm

Interesting.

Honestly, I don't believe this is a cacti problem, since its merely reading the snmp counters from linux. Are you running the latest version of net-snmp/ucd on linux?

Another possible thing is that you've got mrtg limited to 12500000 while cacti is limited to 100000000. Increase the mrtg limit to 100000000 (dont forget to restart the daemon) and see if that changes your graph.

One last test, change the logging level in cacti to medium for a while and watch the values cacti records for the Linux device in/out bandwidth and post it.

hemry · Post by **hemry** » Sat Apr 14, 2007 1:26 pm

At first as you suggested i changed mrtg's limit to 100000000, it didnt help the graphs still were different.

Then I started looking ant cacti.log snmp requests and output they get.
I found interesting thing, snmp requests from that windows machine ar somehow biassed.

From windows cacti host:
C:\net-snmp5.4\bin>snmpget -v1 -c public xxx.xxx.19.81 .1.3.6.1.2.1.2.2.1.16.2
IF-MIB::ifOutOctets.2 = Counter32: 1346982467

Localy on Linux:
HOST1:~ # snmpget -c public -v1 localhost .1.3.6.1.2.1.2.2.1.16.2
IF-MIB::ifOutOctets.2 = Counter32: 3139195935

tcpdump on linux:
HOST1:~ # tcpdump -n -i eth1 'host xxx.xx.65.106'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes
21:17:53.005678 IP xxx.xx.65.106.1853 > xxx.xxx.19.81.161: GetRequest(28) .1.3.6.1.2.1.2.2.1.16.2
21:17:53.007869 IP xxx.xxx.19.81.161 > xxx.xx.65.106.1853: GetResponse(34) .1.3.6.1.2.1.2.2.1.16.2=3148794333

From tcpdump we can see what cacti request did gets to the right machine, it gets one reply, but shows another.
I know its not cacti issue, but maybe someone know what to look for?

BSOD2600 · Post by **BSOD2600** » Sat Apr 14, 2007 9:29 pm

Heh, even more interesting.

So IP: xxx.xxx.19.81 = linux machine? On your linux machine, instead of using 'localhost' try putting in your public IP (same one Windows used). Same/different result?

hemry · Post by **hemry** » Sun Apr 15, 2007 6:55 am

Yes xxx.xxx.19.81 = linux box, If i try putting IP instead of localhost still the same result.
However i moved a bit forward, on windows i use net-snmp 5.4, my linux boxes were with net-snmp 5.2.1 or 5.4rc2, i tried to update them to 5.4 (final) now everything works.
So I suppose these versions are somehow incompatable?

New problem is what i just cant upgrade all snmpd's, its really a lot of them..

BSOD2600 you run cacti on windows, what are version of your net-snmp and snmpd's of your servers?

BSOD2600 · Post by **BSOD2600** » Sun Apr 15, 2007 12:15 pm

Weird, wonder what net-snmp changed between those versions.

I run the Microsoft SNMP Agent, not net-snmp.

hemry · Post by **hemry** » Sun Apr 15, 2007 2:07 pm

Your run microsoft snmp agent as snmpd servers, but what you run as a snmp client?

BSOD2600 · Post by **BSOD2600** » Sun Apr 15, 2007 8:01 pm

Primarily the PHP one since my devices are only snmpv1, but yes I do have net-snmp 5.3.1 installed too. I don't poll any linux device though.

Cacti

wrong network interface stats

wrong network interface stats

Who is online