Weathermap problems: over the top data

[xarses]

Hey guys, I've got an odd one. I've been working on install #1 for a while now and I've been noticing some oddness with one of the weathermaps I've created. every so often i will see what look like large negative 32 bit numbers as the value for the some of (but always the same set of some of) the link. Then the next polling cycle they will dispaly 8GB through ~24TB as the link speed. Suffice to say, the NOC has called a few times over it. I've also noted the same effect in Install #2. My most recent install (#3) has yet to produce these erroneous images and I think it may be involved with the interaction of the boost plug-in and weathermap. The interesting bit of information is that the values written to the rrd's appear to be valid as the graphs don't show these wild values.

I'll post some images shortly, as we've only just enabled the retention of each of the images.

Install 1
Cacti 8.7c
PA 2.3
Boost 2.1
Weathermap 0.95b

Install 2
Cacti 8.7e
Spine 0.8.7e
PA 2.4
Boost 2.5
Weathermap 0.95

Install 3
Cacti 0.8.7e
Spine 0.8.7e
Boost 3.0 (not on yet)
PA 2.4

[xarses]

Images attached

[Howie]

Interesting that they are huge-number of microbits... rather than less-huge-number of bits.

What do you have for INBWFORMAT/OUTBWFORMAT, because the standard settings don't do this...?

Second thing is to run with DEBUG logging for one cycle, if you are able to, and see what ReadData is logging as the raw data.

[xarses]

Howie,

Here's the global sections and config relevant to the EQ1-VC1 to EQ1-Core link. Other than controlling some of the link positions I'm almost entirely using code generated by the UI. I will try to turn on debug logging but it involves a lot of sifting on our 28K data sources.

Also, the images attached where from install #1.

Code: Select all

# Automatically generated by php-weathermap v0.95b

FONTDEFINE 100 docs/example/Vera.ttf 10

WIDTH 1536
HEIGHT 1024
HTMLSTYLE overlib
KEYFONT 2
TITLE Prod DC1
TIMEPOS 1340 15 Created: %b %d %Y %H:%M:%S

KEYPOS DEFAULT 10 11 Traffic Load
KEYTEXTCOLOR 0 0 0
KEYOUTLINECOLOR 0 0 0
KEYBGCOLOR 255 255 255
BGCOLOR 255 255 255
TITLECOLOR 0 0 0
TIMECOLOR 0 0 0
SCALE DEFAULT 0 0   192 192 192
SCALE DEFAULT 0 1   255 255 255
SCALE DEFAULT 1 10   140 0 255
SCALE DEFAULT 10 25   32 32 255
SCALE DEFAULT 25 40   0 192 255
SCALE DEFAULT 40 55   0 240 0
SCALE DEFAULT 55 70   240 240 0
SCALE DEFAULT 70 85   255 192 0
SCALE DEFAULT 85 100   255 0 0

SET nowarn_clipping 1
SET rrd_use_poller_output 1

# End of global section

# DEFAULT definitions:
NODE DEFAULT
        LABELFONT 2
        MAXVALUE 100

LINK DEFAULT
        WIDTH 2
        OVERLIBWIDTH 403
        OVERLIBHEIGHT 166
        BWFONT 1
        COMMENTFONT 100
        BWLABEL bits
        BANDWIDTH 1G


# End of DEFAULTS section
NODE node03289
        LABEL EQ1-Core1
        POSITION 1138 350

NODE node03590
        LABEL EQ1-VC1
        POSITION 915 255


LINK node03289-node03590
        INFOURL /cacti/graph.php?rra_id=all&local_graph_id=18664
        OVERLIBGRAPH /cacti/graph_image.php?local_graph_id=18664&rra_id=0&graph_
nolegend=true&graph_height=100&graph_width=300
        TARGET /var/www/cacti-0.8.7c/rra/eq1-core-1_traffic_in_22792.rrd
        NODES node03289 node03590
        BANDWIDTH 10G

[Howie]

Fair enough, I guessed it might be tricky.

In that case, lets do the next step and look in the database.

When you get this issue, please look in weathermap_data and find the relevant rows for that rrd file and paste them.

Are all the links that have this problem big busy ones? (I noticed the example there is 10G). I wonder if this is some counter-wrapping case that shows up more on larger links...

[xarses]

I'll see if i can capture this in the database.


regarding the link size, this also occurs on 45M, 100M, 1G, 2G, and 4G links

SV1-EDGE1 to SV1-FW1 = 100M
SV1-FW1 to SV1-Core = 1G
SV1-Core to p2p-qw-HQ = 45M
SV1-Core to SV1-LB2 = 4G

EQ1-Core2 to EQ1-LB[1-2] 4G

[xarses]

I was able to catch the data and grab it on the fly. It looks like the TB numbers may be caught from the difference between the two numbers. Interestingly enough, the boost table records the negative number timestamp as a U instead of a negative value.

Install 1 and install 3 are tracking some of the same interfaces but #3 isn't generating any of the weathermaps. I'll see if they track different values on the same cycle.

[xarses]

It seems that the core of this issue is that the 1 minute poll is over run on this hardware and data isn't retrieved. That said, we've tried to ignore the issue by modifying the weathermap_poller_output function in setup.php however results have been inconsistent.

I've added code before the switch on data_source_type_id to identify if a string "U" is present for $value, when found i try to "normalize" by bringing the previous's turns values forward. This works most of the time, but if the poller misses consecutive polls for the data source in question i will get back a massive delta when the ds is polled.

Code: Select all

                        $lastval = $required['last_value'];

                        // Xarses: Code to corect 'U' overflow
                        // Code will skip the issue and try to present same data as last time
                        $lastcalc = $required['last_calc'];

                        if ( $value == "U" ) {
                                $value = $lastval + ($lastcalc * $period);
                                cacti_log("poller_output: Caught STR _U_ NAN $file:$dsname -> $time:$value $lastval + ($lastcalc * $period)\n",true,"WEATHERMAP");
                        }

                        switch($required['data_source_type_id'])
                        {