'poller_output' Question

CPF · Post by **CPF** » Thu Apr 24, 2008 5:33 am

Hi,

I've recently started using the very excellent 1-minute poller support in Cacti v0.8.7b

This is outstanding and is working really well.

I'm now if the process of changing some of my Weathermaps over to using the 'poller_output' support.

The first one that I've changed is working great, but I'm seeing these errors in the log:

Code: Select all

04/24/2008 11:24:12 AM - SYSTEM STATS: Time:7.1532 Method:spine Processes:1 Threads:20 Hosts:40 HostsPerProcess:40 DataSources:217 RRDsProcessed:114
04/24/2008 11:24:13 AM - WEATHERMAP: Poller[0] [Map 12] Internetv3.conf: RRD ReadData: poller_output: traffic_in is not a valid RRD filename within this Cacti install.
04/24/2008 11:24:13 AM - WEATHERMAP: Poller[0] [Map 12] Internetv3.conf: RRD ReadData: poller_output: traffic_out is not a valid RRD filename within this Cacti install.
04/24/2008 11:24:13 AM - WEATHERMAP: Poller[0] [Map 12] Internetv3.conf: RRD ReadData: poller_output: traffic_in is not a valid RRD filename within this Cacti install.
04/24/2008 11:24:13 AM - WEATHERMAP: Poller[0] [Map 12] Internetv3.conf: RRD ReadData: poller_output: traffic_out is not a valid RRD filename within this Cacti install.
04/24/2008 11:24:13 AM - WEATHERMAP: Poller[0] [Map 12] Internetv3.conf: RRD ReadData: poller_output: traffic_in is not a valid RRD filename within this Cacti install.
04/24/2008 11:24:13 AM - WEATHERMAP: Poller[0] [Map 12] Internetv3.conf: RRD ReadData: poller_output: traffic_out is not a valid RRD filename within this Cacti install.

The map that it refers to, 'Internetv3.conf' is the one that I've changed to use 'poller_output' support.

The link information in this file is very basic:

Code: Select all

LINK ChessISPLine-Internet
	WIDTH 7
	USESCALE SMOOTH
	INFOURL /graph.php?rra_id=all&local_graph_id=752
	OVERLIBGRAPH /graph_image.php?local_graph_id=752&rra_id=0&graph_nolegend=true&graph_height=100&graph_width=300
	TARGET rrd:C:/Data/Apps/cacti/cactimaptest/rra/switch_-_128_traffic_in_771.rrd
	NODES ChessISPLine Internet
	BANDWIDTH 10M

LINK ADSLLine-Internet
	WIDTH 5
	USESCALE SMOOTH
	INFOURL /graph.php?rra_id=all&local_graph_id=754
	OVERLIBGRAPH /graph_image.php?local_graph_id=754&rra_id=0&graph_nolegend=true&graph_height=100&graph_width=300
	TARGET rrd:C:/Data/Apps/cacti/cactimaptest/rra/real-adsl-gw_traffic_in_773.rrd
	NODES ADSLLine Internet
	BANDWIDTH 8192K 448K

LINK AltyISPLine-Internet
	WIDTH 3
	USESCALE SMOOTH
	INFOURL /graph.php?rra_id=all&local_graph_id=753
	OVERLIBGRAPH /graph_image.php?local_graph_id=753&rra_id=0&graph_nolegend=true&graph_height=100&graph_width=300
	TARGET rrd:C:/Data/Apps/cacti/cactimaptest/rra/altsw12_traffic_in_772.rrd
	NODES AltyISPLine Internet
	BANDWIDTH 2M

Now, this error doesn't appear to be causing me any issues, but I just wanted to know if it's something that I should be worried about?

I've also already got my logging set to 'Quiet' in the settings section.

I've tried with and without the 'rrd:' in the TARGET line, but I still get the same errors.

Again, should I be worried, or should I just ignore this?

Thanks in advance,

Graham.

Howie · Post by **Howie** » Thu Apr 24, 2008 6:35 am

Does it carry on doing that after a few poller cycles?

I can see that you might get it when you've just switched...

CPF · Post by **CPF** » Thu Apr 24, 2008 6:49 am

Hi Howie,

Does it carry on doing that after a few poller cycles?

I've just checked and I get the same errors for every poller cycle... So for a couple of hours so far...

Cheers,

Graham.

Howie · Post by **Howie** » Thu Apr 24, 2008 6:59 am

OK - I'll have a play. I'm using 1 minute polling and weathermap on one of our installations too.

CPF · Post by **CPF** » Thu Apr 24, 2008 7:06 am

Cool,

Thanks for that. Let me know if there's any more info that you need, or anything that you'd like me to try.

Cheers,

Graham.

Howie · Post by **Howie** » Thu Apr 24, 2008 8:08 am

Figured it out. What is going on is that it tries to match the filename you give it against the ones in the cactidb. However, Cacti stores them as <rrd_path>/blah.rrd, not the full path, so I have to get rrd_path, and do a replace with the string <rrdpath> so that I can find it in the database.

If you use a format for the filename that doesn't exactly match rrdpath, then my replace fails, and so weathermap can't find which poller_output entry to use. The reason it still works is that it falls back to using regular rrdtool if it can't use poller_output.

So the solution is to change the path you use for the rrd file. My guess is that replacing the / with \ will fix it.

CPF · Post by **CPF** » Thu Apr 24, 2008 9:12 am

Hi Howie,

Thanks for the pointers, but no luck. I've tried the following permutations:

Code: Select all

TARGET rrd:C:/Data/Apps/cacti/cactimaptest/rra/<rrd file>
TARGET rrd:C:\Data\Apps\cacti\cactimaptest\rra\<rrd file>
TARGET rrd:c:/Data/Apps/cacti/cactimaptest/rra/<rrd file>

Where can I find the exact value of <rrdpath> that you mentioned?

In the 'settings' table in the database, the only similar row that I could see was 'path_webroot', which in my installation is:

Code: Select all

c:/Data/Apps/cacti/cactimaptest

Is this the right value?

I don't know if it helps / matters, but the weathermap config was initially built using the Editor. So the TARGET value in the LINKs was created by the editor tool, which must have pulled it from somewhere?

Any other thoughts?

Cheers,

Graham.

CPF · Post by **CPF** » Thu Apr 24, 2008 10:07 am

Hi Howie,

Just been doing some more thinking about this one. Where does weathermap look for the poller data?

I'll be the first to admit that I don't really know the inner working of the polling process. What I have noticed that the table 'poller_output' in my Cacti database is almost always empty. Is this right?

Code: Select all

mysql> select count(*) from poller_output;
+----------+
| count(*) |
+----------+
|        0 |
+----------+
1 row in set (0.00 sec)

If I constantly query it near to the seconds when the poller runs, it does briefly have some data, but this is removed almost instantly.

Is this right?
Is it me?

Is it possible that the data which weathermap is looking for is gone by the time that it queries this table (if that's how it works)?

Again, any pointers would be welcome.

Cheers,

Graham.

Howie · Post by **Howie** » Sun Apr 27, 2008 9:59 am

No, that's what should happen.

The way it works is (roughly, I'm no Cacti internals expert): Cacti poller collects data, inserts it into poller_output, and then a different part of the process reads that data out and updates your rrd files with it, deleting poller_output rows as it goes.

What the plugin architecture allows us to do is intercept various internal parts of Cacti and tinker with them before Cacti gets a chance to do it's normal thing. One of those 'hooks' is called poller_output, and sits in the path between the poller, and the poller_output table (or possibly poller_output and the rrd updater). Plugins like Thold and Weathermap can watch the data go past and pick it out to save somewhere else (weathermap_data, thold_data) before the poller removes it.

This saves weathermap from running 40 copies of rrdtool to fetch the data for your map, so it's quicker. If you use Boost, then the rrd files aren't even updated every poller cycle, so you *have* to use poller_output to get current data.

Howie · Post by **Howie** » Sun Apr 27, 2008 10:12 am

More constructively, try these changes to WeatherMapDataSource_rrd.php:

Around line 18 (just after the commented-out block):

Code: Select all

if($map->context=='cacti')
{
      debug("RRD DS: path_rra is ".$config["base_path"]."/rra - your rrd  pathname must be exactly this to use poller_output\n");
}

That will add an additional log line in DEBUG mode to tell you what path_rra is.

Around line 68 change this:

Code: Select all

$db_rrdname = realpath($rrdfile);

to this:

Code: Select all

$db_rrdname = $rrdfile;

to stop weathermap trying to be clever about 'fixing' paths.

Both those changes are in 0.96, but I think you are becoming a victim of the second problem, and the first change will help you with fixing it.

Make the changes and see if the errors go away. If not, run a poller-cycle with DEBUG on and look in the cacti.log afterwards for the path_rra line to see what your paths should look like. Don't forget to turn off DEBUG, because it'll impact your 1-minute polling.

CPF · Post by **CPF** » Mon Apr 28, 2008 6:00 am

Hi Howie,

Thanks again for all your help with this. I've made the changes that you suggested, but without success.

Here's some more information...

Tracking down one of the specific items that I'm interested in gives me this:

Code: Select all

mysql> select local_data_id,rrd_path from poller_item where local_data_id = 771;
+---------------+---------------------------------------------------------------------+
| local_data_id | rrd_path                                                            |
+---------------+---------------------------------------------------------------------+
|           771 | C:\Data\Apps\cacti\cactimaptest\rra\switch_-_128_traffic_in_771.rrd |
|           771 | C:\Data\Apps\cacti\cactimaptest\rra\switch_-_128_traffic_in_771.rrd |
+---------------+---------------------------------------------------------------------+

Right, now that I understand the role of the 'poller_output' table, I have managed to 'catch' it whilst it still had data in. The results are:

Code: Select all

mysql> select * from poller_output where local_data_id > 700;
+---------------+-------------+---------------------+------------+
| local_data_id | rrd_name    | time                | output     |
+---------------+-------------+---------------------+------------+
|           771 | traffic_in  | 2008-04-24 16:49:09 | 2146691502 |
|           771 | traffic_out | 2008-04-24 16:49:09 | 3605558618 |
+---------------+-------------+---------------------+------------+

To me, that suggests that the path is definitely:

Code: Select all

C:\Data\Apps\cacti\cactimaptest\rra\switch_-_128_traffic_in_771.rrd

So, just to be sure, my Weathermap config has:

Code: Select all

LINK ChessISPLine-Internet
	WIDTH 7
	USESCALE SMOOTH
	INFOURL /graph.php?rra_id=all&local_graph_id=752
	OVERLIBGRAPH /graph_image.php?local_graph_id=752&rra_id=0&graph_nolegend=true&graph_height=100&graph_width=300
	TARGET rrd:C:\Data\Apps\cacti\cactimaptest\rra\switch_-_128_traffic_in_771.rrd
	NODES ChessISPLine Internet
	BANDWIDTH 10M

As you suggested, I've also run with DEBUG logging on. I'll be honest and say that I don't fully understand the program flow, but here is the part which I believe is relevant:

Code: Select all

04/28/2008 10:18:12 AM - WEATHERMAP: Poller[0] DEBUG: [wmrrd_read_from_poller_output@WeatherMapDataSource_rrd.php:65] [Map 13] Internetv3.conf: RRD ReadData: poller_output style
04/28/2008 10:18:12 AM - WEATHERMAP: Poller[0] DEBUG: [wmrrd_read_from_poller_output@WeatherMapDataSource_rrd.php:73] [Map 13] Internetv3.conf: ******************************************************************
Checking weathermap_data
04/28/2008 10:18:12 AM - WEATHERMAP: Poller[0] DEBUG: [wmrrd_read_from_poller_output@WeatherMapDataSource_rrd.php:76] [Map 13] Internetv3.conf: RRD ReadData: poller_output - looking for 0 value
04/28/2008 10:18:12 AM - WEATHERMAP: Poller[0] DEBUG: [wmrrd_read_from_poller_output@WeatherMapDataSource_rrd.php:79] [Map 13] Internetv3.conf: RRD ReadData: poller_output - DS name is traffic_in
04/28/2008 10:18:12 AM - CMDPHP: Poller[0] DEBUG: SQL Row: "select * from weathermap_data where rrdfile='C:\\Data\\Apps\\cacti\\cactimaptest\\rra\\switch_-_128_traffic_in_771.rrd' and data_source_name='traffic_in'"
04/28/2008 10:18:12 AM - WEATHERMAP: Poller[0] DEBUG: [wmrrd_read_from_poller_output@WeatherMapDataSource_rrd.php:88] [Map 13] Internetv3.conf: RRD ReadData: poller_output - Adding new weathermap_data row for C:\Data\Apps\cacti\cactimaptest\rra\switch_-_128_traffic_in_771.rrd:traffic_in
04/28/2008 10:18:12 AM - CMDPHP: Poller[0] DEBUG: SQL Row: "select data_template_data.local_data_id from data_template_data,data_template_rrd where data_template_data.local_data_id=data_template_rrd.local_data_id and data_template_data.data_source_path='C:\\Data\\Apps\\cacti\\cactimaptest\\rra\\switch_-_128_traffic_in_771.rrd' and data_template_rrd.data_source_name='traffic_in'"
04/28/2008 10:18:12 AM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select data_template_rrd.data_source_name from data_template_data,data_template_rrd where data_template_data.local_data_id=data_template_rrd.local_data_id and data_template_data.data_source_path='C:\\Data\\Apps\\cacti\\cactimaptest\\rra\\switch_-_128_traffic_in_771.rrd'"
04/28/2008 10:18:12 AM - WEATHERMAP: Poller[0] [Map 13] Internetv3.conf: RRD ReadData: poller_output: traffic_in is not a valid RRD filename within this Cacti install.

So I'm not sure where or what's going wrong?!?

As before, thanks for your help with this, but for me, this is currently a test system, so there's no urgency. If it's just me, then don't burn too much time on it. I can always have another go when version 0.96 is ready.

Cheers,

Graham.

Howie · Post by **Howie** » Mon Apr 28, 2008 1:27 pm

CPF wrote: To me, that suggests that the path is definitely:
Code: Select all
C:\Data\Apps\cacti\cactimaptest\rra\switch_-_128_traffic_in_771.rrd

I agree. But...

Code: Select all

04/28/2008 10:18:12 AM - CMDPHP: Poller[0] DEBUG: SQL Row: "select * from weathermap_data where rrdfile='C:\\Data\\Apps\\cacti\\cactimaptest\\rra\\switch_-_128_traffic_in_771.rrd' and data_source_name='traffic_in'"

that line should contain '<path_rra>' and not the full path, so evidently weathermap disagrees. Further back in the logs, you should have had a line telling you what Weathermap thinks path_rra is, if you made both the changes.

CPF · Post by **CPF** » Tue Apr 29, 2008 3:02 am

Hi Howie,

I did make both the changes, but there's no <path_rra> in the log file. I'm going to PM both the log and the modified 'WeatherMapDataSource_rrd.php' to make sure that it's not me!

Thanks again for the help.

Graham.

CPF · Post by **CPF** » Tue May 06, 2008 8:55 am

Hi Howie,

Thanks for the file that you PM'd me. This does now report the 'rra_path' line, as you suggested:

Code: Select all

05/06/2008 10:58:12 AM - WEATHERMAP: Poller[0] DEBUG: [Init@WeatherMapDataSource_rrd.php:15] [Map 13] Internetv3.conf: RRD DS: path_rra is C:/Data/Apps/cacti/cactimaptest/rra - your rrd pathname must be exactly this to use poller_output

Cool. So I've changed my config file to:

Code: Select all

TARGET rrd:C:/Data/Apps/cacti/cactimaptest/rra/switch_-_128_traffic_in_771.rrd

(I thought I'd tested this before, so maybe there were some other tweaks in there?)

I then let the poller run through again and all looked good:

Code: Select all

05/06/2008 10:58:12 AM - WEATHERMAP: Poller[0] DEBUG: [wmrrd_read_from_poller_output@WeatherMapDataSource_rrd.php:67] [Map 13] Internetv3.conf: RRD ReadData: poller_output style
05/06/2008 10:58:12 AM - WEATHERMAP: Poller[0] DEBUG: [wmrrd_read_from_poller_output@WeatherMapDataSource_rrd.php:74] [Map 13] Internetv3.conf: ******************************************************************
Checking weathermap_data
05/06/2008 10:58:12 AM - WEATHERMAP: Poller[0] DEBUG: [wmrrd_read_from_poller_output@WeatherMapDataSource_rrd.php:77] [Map 13] Internetv3.conf: RRD ReadData: poller_output - looking for 0 value
05/06/2008 10:58:12 AM - WEATHERMAP: Poller[0] DEBUG: [wmrrd_read_from_poller_output@WeatherMapDataSource_rrd.php:80] [Map 13] Internetv3.conf: RRD ReadData: poller_output - DS name is traffic_in
05/06/2008 10:58:12 AM - CMDPHP: Poller[0] DEBUG: SQL Row: "select * from weathermap_data where rrdfile='<path_rra>/switch_-_128_traffic_in_771.rrd' and data_source_name='traffic_in'"

05/06/2008 10:58:12 AM - WEATHERMAP: Poller[0] DEBUG: [wmrrd_read_from_poller_output@WeatherMapDataSource_rrd.php:115] [Map 13] Internetv3.conf: RRD ReadData: poller_output - found weathermap_data row
05/06/2008 10:58:12 AM - WEATHERMAP: Poller[0] DEBUG: [wmrrd_read_from_poller_output@WeatherMapDataSource_rrd.php:126] [Map 13] Internetv3.conf: RRD ReadData: poller_output - data is either too old, or too new
05/06/2008 10:58:12 AM - WEATHERMAP: Poller[0] DEBUG: [wmrrd_read_from_poller_output@WeatherMapDataSource_rrd.php:77] [Map 13] Internetv3.conf: RRD ReadData: poller_output - looking for 1 value
05/06/2008 10:58:12 AM - WEATHERMAP: Poller[0] DEBUG: [wmrrd_read_from_poller_output@WeatherMapDataSource_rrd.php:80] [Map 13] Internetv3.conf: RRD ReadData: poller_output - DS name is traffic_out
05/06/2008 10:58:12 AM - CMDPHP: Poller[0] DEBUG: SQL Row: "select * from weathermap_data where rrdfile='<path_rra>/switch_-_128_traffic_in_771.rrd' and data_source_name='traffic_out'"

05/06/2008 10:58:12 AM - WEATHERMAP: Poller[0] DEBUG: [wmrrd_read_from_poller_output@WeatherMapDataSource_rrd.php:115] [Map 13] Internetv3.conf: RRD ReadData: poller_output - found weathermap_data row
05/06/2008 10:58:12 AM - WEATHERMAP: Poller[0] DEBUG: [wmrrd_read_from_poller_output@WeatherMapDataSource_rrd.php:126] [Map 13] Internetv3.conf: RRD ReadData: poller_output - data is either too old, or too new
05/06/2008 10:58:12 AM - WEATHERMAP: Poller[0] DEBUG: [wmrrd_read_from_poller_output@WeatherMapDataSource_rrd.php:146] [Map 13] Internetv3.conf: RRD ReadData: poller_output - result is 0,0
05/06/2008 10:58:12 AM - WEATHERMAP: Poller[0] DEBUG: [wmrrd_read_from_poller_output@WeatherMapDataSource_rrd.php:147] [Map 13] Internetv3.conf: RRD ReadData: poller_output - ended

BUT, it looks like it's not returning any data:

Code: Select all

: poller_output - data is either too old, or too new
: poller_output - result is 0,0

I left it running for a good hour just to be sure, but no data is returned.

If I switch back to the RRDTool method (i.e. NOT Poller Output), the values on the map return correctly.

So, progress, which I'm very greatful for, but not quite there yet.

Is there anything else I can try?

Thanks again,

Graham

Howie · Post by **Howie** » Tue May 06, 2008 9:02 am

Could you have a look through your DEBUG cacti.log for lines that say 'poller_output: Got one! '?

This is output by the other side of the process, where weathermap looks at the poller data for 'interesting' things. It sounds like the TARGET part is correctly inserting lines in weathermap_data now.

'poller_output' Question

'poller_output' Question

Who is online