Strange value on bandwidth

mrvu · Post by **mrvu** » Fri Mar 06, 2009 9:43 am

Hi all,

I have strange problem with Weathermap. I enable the rrd_use_poller_output option and then weathermap have this warning:

03/06/2009 09:21:14 PM - WEATHERMAP: Poller[0] [Map 9] HAN.ADSL.VB: ColourFromPercent: Scale DEFAULT doesn't cover -35494.654621103% for 4006-VB-R57 [WMWARN29]
03/06/2009 09:21:14 PM - WEATHERMAP: Poller[0] [Map 9] HAN.ADSL.VB: ColourFromPercent: Scale DEFAULT doesn't cover -6469.3136115641% for 4006-VB-R57 [WMWARN29]

The traffic of the interface that displayed on the map is defenitely wrong. But the traffic in the graph is seem to be correct.
Cacti Version 0.8.7c
Boost v2.1
Weathermap 0.95b

Please anyone help me on this problem

mrvu · Post by **mrvu** » Fri Mar 06, 2009 9:57 am

The network connection to router sometimes goes down. Does this affect the result?

Howie · Post by **Howie** » Fri Mar 06, 2009 10:07 am

It seems like you are getting large negative values from the rrd file...

Could you post the config for one of these links?

mrvu · Post by **mrvu** » Fri Mar 06, 2009 11:46 pm

This is the config for that link:

LINK DEFAULT
WIDTH 2
BWLABEL bits
BANDWIDTH 1G

LINK 4006-VB-R57
INFOURL /cacti/graph.php?rra_id=all&local_graph_id=1517 OVERLIBGRAPH /cacti/graph_image.php?local_graph_id=1517&rra_id=0&graph_nolegend=true&graph_height=100&graph_width=300
TARGET /var/www/html/cacti/rra/XXX-sw4006_vb_adsl_traffic_in_1515.rrd NODES 4006-VB-ADSL R57

There're also some strange positive bandwidth value. I guess the strange value appears when the router is unreachable or rebooting.

03/07/2009 10:45:05 AM - WEATHERMAP: Poller[0] [Map 9] HAN.ADSL.VB: ColourFromPercent: Clipped 3750-VB02_copy-R34 366815.32303227% to 100%
03/07/2009 10:45:05 AM - WEATHERMAP: Poller[0] [Map 9] HAN.ADSL.VB: ColourFromPercent: Clipped 3750-VB02_copy-R34 64057.620133973% to 100%
03/07/2009 10:45:05 AM - SYSTEM STATS: Time:2.8875 Method:spine Processes:2 Threads:1 Hosts:110 HostsPerProcess:55 DataSources:479 RRDsProcessed:0
03/07/2009 10:45:03 AM - SPINE: Poller[0] Host[110] Hostname[10.10.103.210] ERROR: HOST EVENT: Host is DOWN Message: Host did not respond to SNMP
03/07/2009 10:44:31 AM - WEATHERMAP: Poller[0] [Map 9] HAN.ADSL.VB: ColourFromPercent: Clipped 3750-VB02_copy-R34 366815.32303227% to 100%

Howie · Post by **Howie** » Sat Mar 07, 2009 2:31 am

mrvu wrote: There're also some strange positive bandwidth value. I guess the strange value appears when the router is unreachable or rebooting.

03/07/2009 10:45:05 AM - WEATHERMAP: Poller[0] [Map 9] HAN.ADSL.VB: ColourFromPercent: Clipped 3750-VB02_copy-R34 64057.620133973% to 100%

I think there must be more to it than that - I assume you aren't getting a 640Gbit spike on your Cacti graphs? Weathermap reads the same data.

The next thing to try would be to run with DEBUG logging for one poller cycle, then look for the WEATHERMAP ReadData lines in the log - they should show you what data is being given by rrdtool, which should provide some clues, hopefully.

Alternatively, you could also try adding
SET rrd_use_poller_output 1
to the top of your map file, and waiting for 3 poller cycles. That will start Weathermap pulling data directly from the poller, instead of looking in the rrd files. That should avoid any rrd-related problems, but it does make it a little harder to check what you are doing when you're editing maps, because changes to targets take a few cycles to show up.

mrvu · Post by **mrvu** » Sat Mar 07, 2009 4:26 am

Off course I have this setting in the map config (I use Boost)
SET rrd_use_poller_output 1

Cacti graphs display current Inbound/Outbound = 0 when the big value appears on the map (which is correct because I know there are some problems with the router at that moment)

I will try to turn debug on when I see the problem again. But the log must be huge

Howie · Post by **Howie** » Sat Mar 07, 2009 4:42 am

mrvu wrote:Off course I have this setting in the map config (I use Boost)
SET rrd_use_poller_output 1

Cacti graphs display current Inbound/Outbound = 0 when the big value appears on the map (which is correct because I know there are some problems with the router at that moment)

I will try to turn debug on when I see the problem again. But the log must be huge

Hmm, OK. This "problem"... is the router resetting it's counters?

mrvu · Post by **mrvu** » Sun Mar 08, 2009 10:15 am

I don't think the counter is reset because uptime of the router is unchaged (Uptime: 1662949672 (192 days, 11 hours, 18 minutes))

I had monitored the map for and notice the following behavior:
- First the bandwidth label go up very high in the map
- After few poller cycles, it display as zero
- Then it come back as normal

The graph have some NAN value at that time.

colejv · Post by **colejv** » Tue Mar 10, 2009 7:28 am

I've noticed the same behavior when using poller output. I am graphing over a hundred interfaces on one switch and occasionally 5-10% fail to read and display large positive or negative numbers, without poller output 0's are displayed.

Howie · Post by **Howie** » Tue Mar 10, 2009 7:33 am

colejv wrote:I've noticed the same behavior when using poller output. I am graphing over a hundred interfaces on one switch and occasionally 5-10% fail to read and display large positive or negative numbers, without poller output 0's are displayed.

Looking at the code, it doesn't actually do anything to check for NaNs

So it would take the 'N', convert that to 0, and use that as the current value. That would then look like a counter wrap-around, and give you huge values.

I'll try and organise a fix for this week - it's a busy week for Real Work though.

mcutting · Post by **mcutting** » Tue Mar 10, 2009 2:05 pm

Howie - just my 2cents worth.

I am also experiencing the same issue. Now running under Ubuntu (don't do Windows anymore for Cacti), and have noticed that this issue shows itself on both platforms.

I had an instance today where our MPLS router enjoyed 1078% outbound bandwidth, and the same inbound !

Howie · Post by **Howie** » Mon Mar 23, 2009 1:13 pm

If any of you are using 0.96test2 and get this problem regularly, could you please try dropping in the setup.php from subversion? ( this one )

I've added a check for 'U' values that sets the current value to 0, and otherwise ignores the update, so the first real update afterwards will be averaged over a longer time, but will be the right kind of value at least...

Still not sure what will happen when there's a reboot.

mrvu · Post by **mrvu** » Mon Mar 23, 2009 9:40 pm

Dear Howie,

Could you release a patch for 0.95b. I want to test but I don't have 0.96 installation here.

Howie · Post by **Howie** » Tue Mar 24, 2009 2:00 am

mrvu wrote:Dear Howie,

Could you release a patch for 0.95b. I want to test but I don't have 0.96 installation here.

No. I'm working to release 0.96 soon anyway - it's hard enough to find time to work on it as it is!

Strange value on bandwidth

Strange value on bandwidth

Who is online