'poller_output' Question

Support questions about the Network Weather Map plugin

Moderators: Developers, Moderators

CPF
Posts: 27
Joined: Sun Aug 28, 2005 8:25 am

'poller_output' Question

Post by CPF »

Hi,

I've recently started using the very excellent 1-minute poller support in Cacti v0.8.7b

This is outstanding and is working really well.

I'm now if the process of changing some of my Weathermaps over to using the 'poller_output' support.

The first one that I've changed is working great, but I'm seeing these errors in the log:

Code: Select all

04/24/2008 11:24:12 AM - SYSTEM STATS: Time:7.1532 Method:spine Processes:1 Threads:20 Hosts:40 HostsPerProcess:40 DataSources:217 RRDsProcessed:114
04/24/2008 11:24:13 AM - WEATHERMAP: Poller[0] [Map 12] Internetv3.conf: RRD ReadData: poller_output: traffic_in is not a valid RRD filename within this Cacti install.
04/24/2008 11:24:13 AM - WEATHERMAP: Poller[0] [Map 12] Internetv3.conf: RRD ReadData: poller_output: traffic_out is not a valid RRD filename within this Cacti install.
04/24/2008 11:24:13 AM - WEATHERMAP: Poller[0] [Map 12] Internetv3.conf: RRD ReadData: poller_output: traffic_in is not a valid RRD filename within this Cacti install.
04/24/2008 11:24:13 AM - WEATHERMAP: Poller[0] [Map 12] Internetv3.conf: RRD ReadData: poller_output: traffic_out is not a valid RRD filename within this Cacti install.
04/24/2008 11:24:13 AM - WEATHERMAP: Poller[0] [Map 12] Internetv3.conf: RRD ReadData: poller_output: traffic_in is not a valid RRD filename within this Cacti install.
04/24/2008 11:24:13 AM - WEATHERMAP: Poller[0] [Map 12] Internetv3.conf: RRD ReadData: poller_output: traffic_out is not a valid RRD filename within this Cacti install.
The map that it refers to, 'Internetv3.conf' is the one that I've changed to use 'poller_output' support.

The link information in this file is very basic:

Code: Select all

LINK ChessISPLine-Internet
	WIDTH 7
	USESCALE SMOOTH
	INFOURL /graph.php?rra_id=all&local_graph_id=752
	OVERLIBGRAPH /graph_image.php?local_graph_id=752&rra_id=0&graph_nolegend=true&graph_height=100&graph_width=300
	TARGET rrd:C:/Data/Apps/cacti/cactimaptest/rra/switch_-_128_traffic_in_771.rrd
	NODES ChessISPLine Internet
	BANDWIDTH 10M

LINK ADSLLine-Internet
	WIDTH 5
	USESCALE SMOOTH
	INFOURL /graph.php?rra_id=all&local_graph_id=754
	OVERLIBGRAPH /graph_image.php?local_graph_id=754&rra_id=0&graph_nolegend=true&graph_height=100&graph_width=300
	TARGET rrd:C:/Data/Apps/cacti/cactimaptest/rra/real-adsl-gw_traffic_in_773.rrd
	NODES ADSLLine Internet
	BANDWIDTH 8192K 448K

LINK AltyISPLine-Internet
	WIDTH 3
	USESCALE SMOOTH
	INFOURL /graph.php?rra_id=all&local_graph_id=753
	OVERLIBGRAPH /graph_image.php?local_graph_id=753&rra_id=0&graph_nolegend=true&graph_height=100&graph_width=300
	TARGET rrd:C:/Data/Apps/cacti/cactimaptest/rra/altsw12_traffic_in_772.rrd
	NODES AltyISPLine Internet
	BANDWIDTH 2M


Now, this error doesn't appear to be causing me any issues, but I just wanted to know if it's something that I should be worried about?

I've also already got my logging set to 'Quiet' in the settings section.

I've tried with and without the 'rrd:' in the TARGET line, but I still get the same errors.

Again, should I be worried, or should I just ignore this?

Thanks in advance,

Graham.
User avatar
Howie
Cacti Guru User
Posts: 5508
Joined: Thu Sep 16, 2004 5:53 am
Location: United Kingdom
Contact:

Post by Howie »

Does it carry on doing that after a few poller cycles? :-?

I can see that you might get it when you've just switched...
Weathermap 0.98a is out! & QuickTree 1.0. Superlinks is over there now (and built-in to Cacti 1.x).
Some Other Cacti tweaks, including strip-graphs, icons and snmp/netflow stuff.
(Let me know if you have UK DevOps or Network Ops opportunities, too!)
CPF
Posts: 27
Joined: Sun Aug 28, 2005 8:25 am

Post by CPF »

Hi Howie,
Does it carry on doing that after a few poller cycles?
I've just checked and I get the same errors for every poller cycle... So for a couple of hours so far... ;)

Cheers,

Graham.
User avatar
Howie
Cacti Guru User
Posts: 5508
Joined: Thu Sep 16, 2004 5:53 am
Location: United Kingdom
Contact:

Post by Howie »

OK - I'll have a play. I'm using 1 minute polling and weathermap on one of our installations too.
Weathermap 0.98a is out! & QuickTree 1.0. Superlinks is over there now (and built-in to Cacti 1.x).
Some Other Cacti tweaks, including strip-graphs, icons and snmp/netflow stuff.
(Let me know if you have UK DevOps or Network Ops opportunities, too!)
CPF
Posts: 27
Joined: Sun Aug 28, 2005 8:25 am

Post by CPF »

Cool,

Thanks for that. Let me know if there's any more info that you need, or anything that you'd like me to try.

Cheers,

Graham.
User avatar
Howie
Cacti Guru User
Posts: 5508
Joined: Thu Sep 16, 2004 5:53 am
Location: United Kingdom
Contact:

Post by Howie »

Figured it out. What is going on is that it tries to match the filename you give it against the ones in the cactidb. However, Cacti stores them as <rrd_path>/blah.rrd, not the full path, so I have to get rrd_path, and do a replace with the string <rrdpath> so that I can find it in the database.

If you use a format for the filename that doesn't exactly match rrdpath, then my replace fails, and so weathermap can't find which poller_output entry to use. The reason it still works is that it falls back to using regular rrdtool if it can't use poller_output.

So the solution is to change the path you use for the rrd file. My guess is that replacing the / with \ will fix it.
Weathermap 0.98a is out! & QuickTree 1.0. Superlinks is over there now (and built-in to Cacti 1.x).
Some Other Cacti tweaks, including strip-graphs, icons and snmp/netflow stuff.
(Let me know if you have UK DevOps or Network Ops opportunities, too!)
CPF
Posts: 27
Joined: Sun Aug 28, 2005 8:25 am

Post by CPF »

Hi Howie,

Thanks for the pointers, but no luck. I've tried the following permutations:

Code: Select all

TARGET rrd:C:/Data/Apps/cacti/cactimaptest/rra/<rrd file>
TARGET rrd:C:\Data\Apps\cacti\cactimaptest\rra\<rrd file>
TARGET rrd:c:/Data/Apps/cacti/cactimaptest/rra/<rrd file>
Where can I find the exact value of <rrdpath> that you mentioned?

In the 'settings' table in the database, the only similar row that I could see was 'path_webroot', which in my installation is:

Code: Select all

c:/Data/Apps/cacti/cactimaptest
Is this the right value?

I don't know if it helps / matters, but the weathermap config was initially built using the Editor. So the TARGET value in the LINKs was created by the editor tool, which must have pulled it from somewhere?

Any other thoughts?

Cheers,

Graham.
CPF
Posts: 27
Joined: Sun Aug 28, 2005 8:25 am

Post by CPF »

Hi Howie,

Just been doing some more thinking about this one. Where does weathermap look for the poller data?

I'll be the first to admit that I don't really know the inner working of the polling process. What I have noticed that the table 'poller_output' in my Cacti database is almost always empty. Is this right?

Code: Select all

mysql> select count(*) from poller_output;
+----------+
| count(*) |
+----------+
|        0 |
+----------+
1 row in set (0.00 sec)
If I constantly query it near to the seconds when the poller runs, it does briefly have some data, but this is removed almost instantly.

Is this right?
Is it me?

Is it possible that the data which weathermap is looking for is gone by the time that it queries this table (if that's how it works)?

Again, any pointers would be welcome.

Cheers,

Graham.
User avatar
Howie
Cacti Guru User
Posts: 5508
Joined: Thu Sep 16, 2004 5:53 am
Location: United Kingdom
Contact:

Post by Howie »

No, that's what should happen.

The way it works is (roughly, I'm no Cacti internals expert): Cacti poller collects data, inserts it into poller_output, and then a different part of the process reads that data out and updates your rrd files with it, deleting poller_output rows as it goes.

What the plugin architecture allows us to do is intercept various internal parts of Cacti and tinker with them before Cacti gets a chance to do it's normal thing. One of those 'hooks' is called poller_output, and sits in the path between the poller, and the poller_output table (or possibly poller_output and the rrd updater). Plugins like Thold and Weathermap can watch the data go past and pick it out to save somewhere else (weathermap_data, thold_data) before the poller removes it.

This saves weathermap from running 40 copies of rrdtool to fetch the data for your map, so it's quicker. If you use Boost, then the rrd files aren't even updated every poller cycle, so you *have* to use poller_output to get current data.
Last edited by Howie on Sun Apr 27, 2008 10:14 am, edited 1 time in total.
Weathermap 0.98a is out! & QuickTree 1.0. Superlinks is over there now (and built-in to Cacti 1.x).
Some Other Cacti tweaks, including strip-graphs, icons and snmp/netflow stuff.
(Let me know if you have UK DevOps or Network Ops opportunities, too!)
User avatar
Howie
Cacti Guru User
Posts: 5508
Joined: Thu Sep 16, 2004 5:53 am
Location: United Kingdom
Contact:

Post by Howie »

More constructively, try these changes to WeatherMapDataSource_rrd.php:

Around line 18 (just after the commented-out block):

Code: Select all

if($map->context=='cacti')
{
      debug("RRD DS: path_rra is ".$config["base_path"]."/rra - your rrd  pathname must be exactly this to use poller_output\n");
}
That will add an additional log line in DEBUG mode to tell you what path_rra is.

Around line 68 change this:

Code: Select all

$db_rrdname = realpath($rrdfile);
to this:

Code: Select all

$db_rrdname = $rrdfile;
to stop weathermap trying to be clever about 'fixing' paths.

Both those changes are in 0.96, but I think you are becoming a victim of the second problem, and the first change will help you with fixing it.

Make the changes and see if the errors go away. If not, run a poller-cycle with DEBUG on and look in the cacti.log afterwards for the path_rra line to see what your paths should look like. Don't forget to turn off DEBUG, because it'll impact your 1-minute polling.
Weathermap 0.98a is out! & QuickTree 1.0. Superlinks is over there now (and built-in to Cacti 1.x).
Some Other Cacti tweaks, including strip-graphs, icons and snmp/netflow stuff.
(Let me know if you have UK DevOps or Network Ops opportunities, too!)
CPF
Posts: 27
Joined: Sun Aug 28, 2005 8:25 am

Post by CPF »

Hi Howie,

Thanks again for all your help with this. I've made the changes that you suggested, but without success.

Here's some more information...

Tracking down one of the specific items that I'm interested in gives me this:

Code: Select all

mysql> select local_data_id,rrd_path from poller_item where local_data_id = 771;
+---------------+---------------------------------------------------------------------+
| local_data_id | rrd_path                                                            |
+---------------+---------------------------------------------------------------------+
|           771 | C:\Data\Apps\cacti\cactimaptest\rra\switch_-_128_traffic_in_771.rrd |
|           771 | C:\Data\Apps\cacti\cactimaptest\rra\switch_-_128_traffic_in_771.rrd |
+---------------+---------------------------------------------------------------------+
Right, now that I understand the role of the 'poller_output' table, I have managed to 'catch' it whilst it still had data in. The results are:

Code: Select all

mysql> select * from poller_output where local_data_id > 700;
+---------------+-------------+---------------------+------------+
| local_data_id | rrd_name    | time                | output     |
+---------------+-------------+---------------------+------------+
|           771 | traffic_in  | 2008-04-24 16:49:09 | 2146691502 |
|           771 | traffic_out | 2008-04-24 16:49:09 | 3605558618 |
+---------------+-------------+---------------------+------------+
To me, that suggests that the path is definitely:

Code: Select all

C:\Data\Apps\cacti\cactimaptest\rra\switch_-_128_traffic_in_771.rrd
So, just to be sure, my Weathermap config has:

Code: Select all

LINK ChessISPLine-Internet
	WIDTH 7
	USESCALE SMOOTH
	INFOURL /graph.php?rra_id=all&local_graph_id=752
	OVERLIBGRAPH /graph_image.php?local_graph_id=752&rra_id=0&graph_nolegend=true&graph_height=100&graph_width=300
	TARGET rrd:C:\Data\Apps\cacti\cactimaptest\rra\switch_-_128_traffic_in_771.rrd
	NODES ChessISPLine Internet
	BANDWIDTH 10M
As you suggested, I've also run with DEBUG logging on. I'll be honest and say that I don't fully understand the program flow, but here is the part which I believe is relevant:

Code: Select all

04/28/2008 10:18:12 AM - WEATHERMAP: Poller[0] DEBUG: [wmrrd_read_from_poller_output@WeatherMapDataSource_rrd.php:65] [Map 13] Internetv3.conf: RRD ReadData: poller_output style
04/28/2008 10:18:12 AM - WEATHERMAP: Poller[0] DEBUG: [wmrrd_read_from_poller_output@WeatherMapDataSource_rrd.php:73] [Map 13] Internetv3.conf: ******************************************************************
Checking weathermap_data
04/28/2008 10:18:12 AM - WEATHERMAP: Poller[0] DEBUG: [wmrrd_read_from_poller_output@WeatherMapDataSource_rrd.php:76] [Map 13] Internetv3.conf: RRD ReadData: poller_output - looking for 0 value
04/28/2008 10:18:12 AM - WEATHERMAP: Poller[0] DEBUG: [wmrrd_read_from_poller_output@WeatherMapDataSource_rrd.php:79] [Map 13] Internetv3.conf: RRD ReadData: poller_output - DS name is traffic_in
04/28/2008 10:18:12 AM - CMDPHP: Poller[0] DEBUG: SQL Row: "select * from weathermap_data where rrdfile='C:\\Data\\Apps\\cacti\\cactimaptest\\rra\\switch_-_128_traffic_in_771.rrd' and data_source_name='traffic_in'"
04/28/2008 10:18:12 AM - WEATHERMAP: Poller[0] DEBUG: [wmrrd_read_from_poller_output@WeatherMapDataSource_rrd.php:88] [Map 13] Internetv3.conf: RRD ReadData: poller_output - Adding new weathermap_data row for C:\Data\Apps\cacti\cactimaptest\rra\switch_-_128_traffic_in_771.rrd:traffic_in
04/28/2008 10:18:12 AM - CMDPHP: Poller[0] DEBUG: SQL Row: "select data_template_data.local_data_id from data_template_data,data_template_rrd where data_template_data.local_data_id=data_template_rrd.local_data_id and data_template_data.data_source_path='C:\\Data\\Apps\\cacti\\cactimaptest\\rra\\switch_-_128_traffic_in_771.rrd' and data_template_rrd.data_source_name='traffic_in'"
04/28/2008 10:18:12 AM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select data_template_rrd.data_source_name from data_template_data,data_template_rrd where data_template_data.local_data_id=data_template_rrd.local_data_id and data_template_data.data_source_path='C:\\Data\\Apps\\cacti\\cactimaptest\\rra\\switch_-_128_traffic_in_771.rrd'"
04/28/2008 10:18:12 AM - WEATHERMAP: Poller[0] [Map 13] Internetv3.conf: RRD ReadData: poller_output: traffic_in is not a valid RRD filename within this Cacti install.
So I'm not sure where or what's going wrong?!?

As before, thanks for your help with this, but for me, this is currently a test system, so there's no urgency. If it's just me, then don't burn too much time on it. I can always have another go when version 0.96 is ready.

Cheers,

Graham.
User avatar
Howie
Cacti Guru User
Posts: 5508
Joined: Thu Sep 16, 2004 5:53 am
Location: United Kingdom
Contact:

Post by Howie »

CPF wrote: To me, that suggests that the path is definitely:

Code: Select all

C:\Data\Apps\cacti\cactimaptest\rra\switch_-_128_traffic_in_771.rrd
I agree. But...

Code: Select all

04/28/2008 10:18:12 AM - CMDPHP: Poller[0] DEBUG: SQL Row: "select * from weathermap_data where rrdfile='C:\\Data\\Apps\\cacti\\cactimaptest\\rra\\switch_-_128_traffic_in_771.rrd' and data_source_name='traffic_in'"
that line should contain '<path_rra>' and not the full path, so evidently weathermap disagrees. Further back in the logs, you should have had a line telling you what Weathermap thinks path_rra is, if you made both the changes.
Weathermap 0.98a is out! & QuickTree 1.0. Superlinks is over there now (and built-in to Cacti 1.x).
Some Other Cacti tweaks, including strip-graphs, icons and snmp/netflow stuff.
(Let me know if you have UK DevOps or Network Ops opportunities, too!)
CPF
Posts: 27
Joined: Sun Aug 28, 2005 8:25 am

Post by CPF »

Hi Howie,

I did make both the changes, but there's no <path_rra> in the log file. I'm going to PM both the log and the modified 'WeatherMapDataSource_rrd.php' to make sure that it's not me!

Thanks again for the help.

Graham.
CPF
Posts: 27
Joined: Sun Aug 28, 2005 8:25 am

Post by CPF »

Hi Howie,

Thanks for the file that you PM'd me. This does now report the 'rra_path' line, as you suggested:

Code: Select all

05/06/2008 10:58:12 AM - WEATHERMAP: Poller[0] DEBUG: [Init@WeatherMapDataSource_rrd.php:15] [Map 13] Internetv3.conf: RRD DS: path_rra is C:/Data/Apps/cacti/cactimaptest/rra - your rrd pathname must be exactly this to use poller_output
Cool. So I've changed my config file to:

Code: Select all

TARGET rrd:C:/Data/Apps/cacti/cactimaptest/rra/switch_-_128_traffic_in_771.rrd
(I thought I'd tested this before, so maybe there were some other tweaks in there?)

I then let the poller run through again and all looked good:

Code: Select all

05/06/2008 10:58:12 AM - WEATHERMAP: Poller[0] DEBUG: [wmrrd_read_from_poller_output@WeatherMapDataSource_rrd.php:67] [Map 13] Internetv3.conf: RRD ReadData: poller_output style
05/06/2008 10:58:12 AM - WEATHERMAP: Poller[0] DEBUG: [wmrrd_read_from_poller_output@WeatherMapDataSource_rrd.php:74] [Map 13] Internetv3.conf: ******************************************************************
Checking weathermap_data
05/06/2008 10:58:12 AM - WEATHERMAP: Poller[0] DEBUG: [wmrrd_read_from_poller_output@WeatherMapDataSource_rrd.php:77] [Map 13] Internetv3.conf: RRD ReadData: poller_output - looking for 0 value
05/06/2008 10:58:12 AM - WEATHERMAP: Poller[0] DEBUG: [wmrrd_read_from_poller_output@WeatherMapDataSource_rrd.php:80] [Map 13] Internetv3.conf: RRD ReadData: poller_output - DS name is traffic_in
05/06/2008 10:58:12 AM - CMDPHP: Poller[0] DEBUG: SQL Row: "select * from weathermap_data where rrdfile='<path_rra>/switch_-_128_traffic_in_771.rrd' and data_source_name='traffic_in'"

05/06/2008 10:58:12 AM - WEATHERMAP: Poller[0] DEBUG: [wmrrd_read_from_poller_output@WeatherMapDataSource_rrd.php:115] [Map 13] Internetv3.conf: RRD ReadData: poller_output - found weathermap_data row
05/06/2008 10:58:12 AM - WEATHERMAP: Poller[0] DEBUG: [wmrrd_read_from_poller_output@WeatherMapDataSource_rrd.php:126] [Map 13] Internetv3.conf: RRD ReadData: poller_output - data is either too old, or too new
05/06/2008 10:58:12 AM - WEATHERMAP: Poller[0] DEBUG: [wmrrd_read_from_poller_output@WeatherMapDataSource_rrd.php:77] [Map 13] Internetv3.conf: RRD ReadData: poller_output - looking for 1 value
05/06/2008 10:58:12 AM - WEATHERMAP: Poller[0] DEBUG: [wmrrd_read_from_poller_output@WeatherMapDataSource_rrd.php:80] [Map 13] Internetv3.conf: RRD ReadData: poller_output - DS name is traffic_out
05/06/2008 10:58:12 AM - CMDPHP: Poller[0] DEBUG: SQL Row: "select * from weathermap_data where rrdfile='<path_rra>/switch_-_128_traffic_in_771.rrd' and data_source_name='traffic_out'"

05/06/2008 10:58:12 AM - WEATHERMAP: Poller[0] DEBUG: [wmrrd_read_from_poller_output@WeatherMapDataSource_rrd.php:115] [Map 13] Internetv3.conf: RRD ReadData: poller_output - found weathermap_data row
05/06/2008 10:58:12 AM - WEATHERMAP: Poller[0] DEBUG: [wmrrd_read_from_poller_output@WeatherMapDataSource_rrd.php:126] [Map 13] Internetv3.conf: RRD ReadData: poller_output - data is either too old, or too new
05/06/2008 10:58:12 AM - WEATHERMAP: Poller[0] DEBUG: [wmrrd_read_from_poller_output@WeatherMapDataSource_rrd.php:146] [Map 13] Internetv3.conf: RRD ReadData: poller_output - result is 0,0
05/06/2008 10:58:12 AM - WEATHERMAP: Poller[0] DEBUG: [wmrrd_read_from_poller_output@WeatherMapDataSource_rrd.php:147] [Map 13] Internetv3.conf: RRD ReadData: poller_output - ended
BUT, it looks like it's not returning any data:

Code: Select all

: poller_output - data is either too old, or too new
: poller_output - result is 0,0
I left it running for a good hour just to be sure, but no data is returned.

If I switch back to the RRDTool method (i.e. NOT Poller Output), the values on the map return correctly.

So, progress, which I'm very greatful for, but not quite there yet.

Is there anything else I can try?

Thanks again,

Graham
User avatar
Howie
Cacti Guru User
Posts: 5508
Joined: Thu Sep 16, 2004 5:53 am
Location: United Kingdom
Contact:

Post by Howie »

Could you have a look through your DEBUG cacti.log for lines that say 'poller_output: Got one! '?

This is output by the other side of the process, where weathermap looks at the poller data for 'interesting' things. It sounds like the TARGET part is correctly inserting lines in weathermap_data now.
Weathermap 0.98a is out! & QuickTree 1.0. Superlinks is over there now (and built-in to Cacti 1.x).
Some Other Cacti tweaks, including strip-graphs, icons and snmp/netflow stuff.
(Let me know if you have UK DevOps or Network Ops opportunities, too!)
Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests