[SOLVED] Links to single node show 0 traffic

[Hoffswell]

Ok, here's a weird one -

I have a single node, with two T1s running to it. It is defined fine within the weathermap config file, the data sources and graphs are populating fine as well.

The links read 0. Zero.

I actually monitor traffic on the neighboring router as well, and have tried to create a new node and links using the data from two more rrd files from the remote router.

overlib graph displays show the correct data, however!

No go!

Any ideas as to why this one node displays 0 all the time?

Thanks

[Howie]

Does the cacti.log show any warnings?

[Hoffswell]

Sorry for the delay

Yes, I see this:

Code: Select all

 02/26/2008 03:15:14 PM - WEATHERMAP: Poller[0] ReadData: LINK GR-MER, target: /srv/www/htdocs/cacti-0.8.7/rra/01-gr-wan_traffic_in_20.rrd on config line 141 had no valid data, according to WeatherMapDataSource_rrd

What do you suppose that means?

[Howie]

It means that for that one rrd file, there's no data in the last 800 seconds.

So two things: 1)could you paste the configuration of that link? 2) if you run Cacti with DEBUG logging turned on for a poller cycle (don't forget to turn it off), then take another look in the cacti.log. You should see a lot more logging from 'ReadData RRD'. Paste that?

[Hoffswell]

This rrdtool display shows that there's data there, so I'm thinking the RRD is ok?

There are two links to one campus that show up 0:

Code: Select all

LINK GR-MER
        INFOURL http://**/cacti/graph.php?rra_id=all&local_graph_id=17
        OVERLIBGRAPH http://**/cacti/graph_image.php?local_graph_id=17&rra_id=0&graph_nolegend=true&graph_height=100&graph_width=300
        TARGET /srv/www/htdocs/cacti-0.8.7/rra/01-gr-wan_traffic_in_20.rrd
        NODES GR MER
        VIA 70 520

LINK GR-MERa
        INFOURL http://**/cacti/graph.php?rra_id=all&local_graph_id=30
        OVERLIBGRAPH http://**/cacti/graph_image.php?local_graph_id=30&rra_id=0&graph_nolegend=true&graph_height=100&graph_width=300
        TARGET /srv/www/htdocs/cacti-0.8.7/rra/01-gr-wan_traffic_in_33.rrd
        NODES GR MER

And debug shows something like this for the in_20 rrd:

Code: Select all

 03/03/2008 08:20:13 AM - WEATHERMAP: Poller[0] ReadData: LINK GR-MER, target: /srv/www/htdocs/cacti-0.8.7/rra/01-gr-wan_traffic_in_20.rrd on config line 141 had no valid data, according to WeatherMapDataSource_rrd
03/03/2008 08:20:13 AM - WEATHERMAP: Poller[0] RRD ReadData: Running: /usr/bin/rrdtool fetch /srv/www/htdocs/cacti-0.8.7/rra/01-gr-wan_traffic_in_20.rrd AVERAGE --start now-800 --end now
03/03/2008 08:20:13 AM - WEATHERMAP: Poller[0] ReadData: New Target: /srv/www/htdocs/cacti-0.8.7/rra/01-gr-wan_traffic_in_20.rrd
03/03/2008 08:20:12 AM - WEATHERMAP: Poller[0] ReadData: LINK node03624-node02216, target: /srv/www/htdocs/cacti-0.8.7/rra/01-gr-wan_traffic_in_20.rrd on config line 149 had no valid data, according to WeatherMapDataSource_rrd
03/03/2008 08:20:12 AM - WEATHERMAP: Poller[0] RRD ReadData: Running: /usr/bin/rrdtool fetch /srv/www/htdocs/cacti-0.8.7/rra/01-gr-wan_traffic_in_20.rrd AVERAGE --start now-800 --end now
03/03/2008 08:20:12 AM - WEATHERMAP: Poller[0] ReadData: New Target: /srv/www/htdocs/cacti-0.8.7/rra/01-gr-wan_traffic_in_20.rrd
03/03/2008 08:20:02 AM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /srv/www/htdocs/cacti-0.8.7a/rra/01-gr-wan_traffic_in_20.rrd --template traffic_out:traffic_in 1204550401:1082443418:3376160658

If I look at the cacti charts, there is data there. It's just weathermap that displays 0.

[Howie]

And debug shows something like this for the in_20 rrd:

Code: Select all

 03/03/2008 08:20:13 AM - WEATHERMAP: Poller[0] ReadData: LINK GR-MER, target: /srv/www/htdocs/cacti-0.8.7/rra/01-gr-wan_traffic_in_20.rrd on config line 141 had no valid data, according to WeatherMapDataSource_rrd
03/03/2008 08:20:13 AM - WEATHERMAP: Poller[0] RRD ReadData: Running: /usr/bin/rrdtool fetch /srv/www/htdocs/cacti-0.8.7/rra/01-gr-wan_traffic_in_20.rrd AVERAGE --start now-800 --end now
03/03/2008 08:20:13 AM - WEATHERMAP: Poller[0] ReadData: New Target: /srv/www/htdocs/cacti-0.8.7/rra/01-gr-wan_traffic_in_20.rrd
03/03/2008 08:20:12 AM - WEATHERMAP: Poller[0] ReadData: LINK node03624-node02216, target: /srv/www/htdocs/cacti-0.8.7/rra/01-gr-wan_traffic_in_20.rrd on config line 149 had no valid data, according to WeatherMapDataSource_rrd
03/03/2008 08:20:12 AM - WEATHERMAP: Poller[0] RRD ReadData: Running: /usr/bin/rrdtool fetch /srv/www/htdocs/cacti-0.8.7/rra/01-gr-wan_traffic_in_20.rrd AVERAGE --start now-800 --end now
03/03/2008 08:20:12 AM - WEATHERMAP: Poller[0] ReadData: New Target: /srv/www/htdocs/cacti-0.8.7/rra/01-gr-wan_traffic_in_20.rrd
03/03/2008 08:20:02 AM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /srv/www/htdocs/cacti-0.8.7a/rra/01-gr-wan_traffic_in_20.rrd --template traffic_out:traffic_in 1204550401:1082443418:3376160658

That's not the DEBUG log. In DEBUG mode, you'll get lots more lines per TARGET, including the output from rrdtool (a seres of timestamps and values). That's what I was interested in, since it would hopefully offer some clue as to why Cacti can make a graph, but weathermap can't get any data.

[Hoffswell]

Sorry, Howie!

My search didn't include the data you were interested in.

How's this?

Code: Select all

3/03/2008 01:40:14 PM - WEATHERMAP: Poller[0] RRD ReadData: traditional style
03/03/2008 01:40:14 PM - WEATHERMAP: Poller[0] RRD ReadData: Target DS names are traffic_in and traffic_out
03/03/2008 01:40:14 PM - WEATHERMAP: Poller[0] ReadData: Matched for WeatherMapDataSource_rrd. Calling WeatherMapDataSource_rrd->ReadData()
03/03/2008 01:40:14 PM - WEATHERMAP: Poller[0] ReadData: New Target: /srv/www/htdocs/cacti-0.8.7/rra/01-gr-wan_traffic_in_33.rrd
03/03/2008 01:40:14 PM - WEATHERMAP: Poller[0]
03/03/2008 01:40:14 PM - WEATHERMAP: Poller[0] ReadData: Setting 0,0
03/03/2008 01:40:14 PM - WEATHERMAP: Poller[0] ReadData: LINK GR-MER, target: /srv/www/htdocs/cacti-0.8.7/rra/01-gr-wan_traffic_in_20.rrd on config line 141 had no valid data, according to WeatherMapDataSource_rrd
03/03/2008 01:40:14 PM - WEATHERMAP: Poller[0] RRD ReadData: Returning (-8,-8,0)
03/03/2008 01:40:14 PM - WEATHERMAP: Poller[0] --1204569000: nan nan
03/03/2008 01:40:14 PM - WEATHERMAP: Poller[0] --1204569300: nan nan
03/03/2008 01:40:14 PM - WEATHERMAP: Poller[0] --1204569600: nan nan
03/03/2008 01:40:14 PM - WEATHERMAP: Poller[0] --1204569900: nan nan
03/03/2008 01:40:14 PM - WEATHERMAP: Poller[0] --
03/03/2008 01:40:14 PM - WEATHERMAP: Poller[0] RRD ReadData: Headings are: traffic_in traffic_out
03/03/2008 01:40:14 PM - WEATHERMAP: Poller[0] RRD ReadData: Read 5 lines from rrdtool
03/03/2008 01:40:14 PM - WEATHERMAP: Poller[0] >
03/03/2008 01:40:14 PM - WEATHERMAP: Poller[0] > 1204569900: nan nan
03/03/2008 01:40:14 PM - WEATHERMAP: Poller[0] > 1204569600: nan nan
03/03/2008 01:40:14 PM - WEATHERMAP: Poller[0] > 1204569300: nan nan
03/03/2008 01:40:14 PM - WEATHERMAP: Poller[0] > 1204569000: nan nan
03/03/2008 01:40:14 PM - WEATHERMAP: Poller[0] RRD ReadData: Running: /usr/bin/rrdtool fetch /srv/www/htdocs/cacti-0.8.7/rra/01-gr-wan_traffic_in_20.rrd AVERAGE --start now-800 --end now
03/03/2008 01:40:14 PM - WEATHERMAP: Poller[0] RRD ReadData: traditional style
03/03/2008 01:40:14 PM - WEATHERMAP: Poller[0] RRD ReadData: Target DS names are traffic_in and traffic_out
03/03/2008 01:40:14 PM - WEATHERMAP: Poller[0] ReadData: Matched for WeatherMapDataSource_rrd. Calling WeatherMapDataSource_rrd->ReadData()
03/03/2008 01:40:14 PM - WEATHERMAP: Poller[0] ReadData: New Target: /srv/www/htdocs/cacti-0.8.7/rra/01-gr-wan_traffic_in_20.rrd
03/03/2008 01:40:14 PM - WEATHERMAP: Poller[0]
03/03/2008 01:40:14 PM - WEATHERMAP: Poller[0] ================== ReadData: Updating link data for all links and nodes
03/03/2008 01:40:14 PM - WEATHERMAP: Poller[0] Finished Initialising Plugins...

It seems like I have one nan entry:

Code: Select all

rrdtool fetch 01-gr-wan_traffic_in_20.rrd AVERAGE --start now-800

                     traffic_in         traffic_out

1204569600: 8.9405687333e+03 3.2355995644e+04
1204569900: 7.8475197000e+03 2.6494982756e+04
1204570200: 6.5129031767e+03 3.9430198234e+04
1204570500: nan nan

Before that, the 12047570200 had nan nan in it.

Code: Select all

# rrdtool first 01-gr-wan_traffic_in_20.rrd 

1204390500

Why wuold the first timestamp be nan nan?

Thanks for your help on this weird one, and my neophyte knowledge of rrd.

[Howie]

It's actually the last stamp - it's because weathermap asks for data from now-800 to now, and the last update was a little earlier. So the last line is almost always nan. And then Cacti shows you the log 'upside-down' with the most recent entry at the top, so you need to read that from the bottom-up.

That's a weird one then. You're not using Boost are you? That would possibly explain some delayed updates to rrd files. But you'd normally see it across all your files.

Your Cacti graph is definitely using that .rrd? (check the little spanner icon)

And where did the 'good' output there come from? because the logs show *all* NaNs...

[Hoffswell]

No, I'm not using plugins.

Weathermap looks at:

TARGET /srv/www/htdocs/cacti-0.8.7/rra/01-gr-wan_traffic_in_20.rrd

And when I look at cacti's setup, I see data source

/srv/www/htdocs/cacti-0.8.7a/rra/01-gr-wan_traffic_in_20.rrd

The displayed graphs look to point to this data source.

The spanner icon does report

Code: Select all

	

/usr/bin/rrdtool graph - \
--imgformat=PNG \
--start=-86400 \
--end=-300 \
--title="01-gr-wan - T1 GR - MER - Se2/1:0" \
--rigid \
--base=1000 \
--height=120 \
--width=500 \
--alt-autoscale-max \
--lower-limit=0 \
--vertical-label="bits per second" \
--slope-mode \
--font TITLE:12: \
--font AXIS:8: \
--font LEGEND:10: \
--font UNIT:8: \
DEF:a="/srv/www/htdocs/cacti-0.8.7a/rra/01-gr-wan_traffic_in_20.rrd":traffic_in:AVERAGE \
DEF:b="/srv/www/htdocs/cacti-0.8.7a/rra/01-gr-wan_traffic_in_20.rrd":traffic_in:MAX \
DEF:c="/srv/www/htdocs/cacti-0.8.7a/rra/01-gr-wan_traffic_in_20.rrd":traffic_out:AVERAGE \
DEF:d="/srv/www/htdocs/cacti-0.8.7a/rra/01-gr-wan_traffic_in_20.rrd":traffic_out:MAX \
CDEF:cdefa=a,8,* \
CDEF:cdefd=b,8,* \
CDEF:cdefe=c,8,* \
CDEF:cdefh=d,8,* \
AREA:cdefa#00CF00FF:"Inbound"  \
GPRINT:cdefa:LAST:" Current\:%8.2lf %s"  \
GPRINT:cdefa:AVERAGE:"Average\:%8.2lf %s"  \
GPRINT:cdefd:MAX:"Maximum\:%8.2lf %s\n"  \
LINE1:cdefe#002A97FF:"Outbound"  \
GPRINT:cdefe:LAST:"Current\:%8.2lf %s"  \
GPRINT:cdefe:AVERAGE:"Average\:%8.2lf %s"  \
GPRINT:cdefh:MAX:"Maximum\:%8.2lf %s\n"  \
COMMENT:"\n"  \
HRULE:515015.34#FF0000FF:"95th Percentile"  \
COMMENT:"(0.52 mbit in+out)" 

So the .rrds match


I might try to make a new config file, and start from scratch with this particular link for testing?

[Hoffswell]

I've actually just added 3 new circuits. They all are charting just fine in Cacti, but read 0 on my weathermap.

If I mouse over you can see the charts just fine.

[Howie]

I've actually just added 3 new circuits. They all are charting just fine in Cacti, but read 0 on my weathermap.

If I mouse over you can see the charts just fine.

Not sure what to suggest then. This stuff is the the output from rrdtool

Code: Select all

03/03/2008 01:40:14 PM - WEATHERMAP: Poller[0] > 
03/03/2008 01:40:14 PM - WEATHERMAP: Poller[0] > 1204569900: nan nan 
03/03/2008 01:40:14 PM - WEATHERMAP: Poller[0] > 1204569600: nan nan 
03/03/2008 01:40:14 PM - WEATHERMAP: Poller[0] > 1204569300: nan nan 
03/03/2008 01:40:14 PM - WEATHERMAP: Poller[0] > 1204569000: nan nan 

it's not how weathermap interprets or anything, it's literally the line-by-line output from rrdtool. So that implies that rrdtool is giving incorrect information, but only to weathermap and only for those files. It's not a permissions issue, because then there wouldn't be any output. It's not that the data isn't there because your graphs are updating (the timeline is moving forwards, right?).

Not a clue.

when you run

Code: Select all

/usr/bin/rrdtool fetch /srv/www/htdocs/cacti-0.8.7/rra/01-gr-wan_traffic_in_20.rrd AVERAGE --start now-800 --end now

from the command line, you get useful output?

[Hoffswell]

AH HA!

The weathermap editor is picking:

/srv/www/htdocs/cacti-0.8.7

But my current production cacti installation is

/srv/www/htdocs/cacti-0.8.7a

I actually use a symlink to /srv/www/htdocs/cacti

That fixed me right up.

You were right howie, weathermap wasn't reading the correct rrd file.


I fixed my problem by editing editor-config.php, and changing $cacti_base. It was pointing to the wrong directory.

Thanks!

[Howie]

D'oh!