Cacti no longer fetching, or writing traffic data

[frode]

Hello

My first post to the forum

I have used Cacti for a couple of year, and Weathermap even longer. Thank all the developers of this fantastic products

We have upgraded the core switch infrastructure, and I have seen the problem with gaps in our graphs because of the MAX value being reached, especially for the PO interfaces (port-channels).

Yesterday I was once again updating this, and was a bit annoyed that the default max value was so low, so I changed this in the data-template "Interface traffic". I tried to change both interface_in and interface_out default speed.

After a while I noticed that this obviously broke something, because all the links in weathermap became gray, and it was reported about 130 errors with the graph. Here is some of them:

08/03/2015 01:05:14 PM - WEATHERMAP: Poller[0] WARNING: [Map 5] DatasenterKjonerud: ReadData: LINK HIKT--DF10s4810-01-HIKT--DF10s4810-02, target: /usr/share/cacti/site/rra/hikt--df10s4810-01_traffic_in_570.rrd on config line 426 of /usr/share/cacti/site/plugins/weathermap/configs/DatasenterKjonerud had no valid data, according to WeatherMapDataSource_rrd
08/03/2015 01:05:15 PM - WEATHERMAP: Poller[0] WARNING: [Map 5] DatasenterKjonerud: ReadData: LINK HIKT--DM8024-01-node07817, target: /usr/share/cacti/site/rra/hikt-kjonerud-1etg-dm8024-01_traffic_in_495.rrd on config line 433 of /usr/share/cacti/site/plugins/weathermap/configs/DatasenterKjonerud had no valid data, according to WeatherMapDataSource_rrd
08/03/2015 01:05:15 PM - WEATHERMAP: Poller[0] WARNING: [Map 5] DatasenterKjonerud: ReadData: LINK node07817-HIKT--DF10s4810-01, target: /usr/share/cacti/site/rra/hikt--dm8024-01_traffic_in_541.rrd on config line 439 of /usr/share/cacti/site/plugins/weathermap/configs/DatasenterKjonerud had no valid data, according to WeatherMapDataSource_rrd
08/03/2015 01:05:15 PM - WEATHERMAP: Poller[0] WARNING: [Map 5] DatasenterKjonerud: ReadData: LINK node07817-HIKT--DF10s4810-02, target: /usr/share/cacti/site/rra/hikt--dm8024-01_traffic_in_542.rrd on config line 445 of /usr/share/cacti/site/plugins/weathermap/configs/DatasenterKjonerud had no valid data, according to WeatherMapDataSource_rrd
<SNIP>

It seems to me to be the same warning for all rrd's.

To complicate things I then tried to run a "rrdtool tune $i.rrd --maximum traffic_in:$new_value" on all the files.... (I had taken a backup.)

This trick didn't fix my problem, so I tried to revert both the template value and all the rrd files.... But to my surprise, I still have no data drawn in Weathermap.

I still have the same error messages.

I have found that it seems as if the traffic rrd files no longer is updated:

/usr/share/cacti/site/rra# date
Tue Aug 4 12:01:35 CEST 2015

/usr/share/cacti/site/rra# ls -l
totalt 31348
-rw-r--r-- 1 www-data www-data 47844 Aug 4 12:00 ha-c2960-raadhus-1_5min_cpu_109.rrd
-rw-r--r-- 1 www-data www-data 111212 Aug 4 09:23 ha-c2960-raadhus-1_traffic_in_305.rrd
-rw-r--r-- 1 www-data www-data 111212 Aug 4 09:23 ha-c2960-raadhus-1_traffic_in_306.rrd
-rw-r--r-- 1 www-data www-data 111212 Aug 4 09:23 ha-c2960-raadhus-1_traffic_in_307.rrd
-rw-r--r-- 1 www-data www-data 111212 Aug 4 09:23 ha-c2960-raadhus-1_traffic_in_308.rrd
-rw-r--r-- 1 www-data www-data 111208 Aug 4 09:23 ha-c2960-raadhus-1_traffic_in_610.rrd
-rw-r--r-- 1 www-data www-data 47844 Aug 4 12:00 ha-c2960-raadhus-2_5min_cpu_123.rrd
-rw-r--r-- 1 www-data www-data 111212 Aug 4 09:23 ha-c2960-raadhus-2_traffic_in_405.rrd
-rw-r--r-- 1 www-data www-data 111208 Aug 4 09:23 ha-c2960-raadhus-2_traffic_in_693.rrd
<SNIP>

(I have restored all the files at aprox 09:23 )


I have enabled DEBUG level, but can't find anything error or warning about writing to these files.


That's how far I have made it, with the help of google. Now I don't know what to do.

Can anyone point me in the right direction?


System info:
Debian 7.8
Cacti 0.8.8a+dfsg-5+deb7u4
Weathermap plugin 0.97c

[BSOD2600]

Sounds more like a weathermap issue...

Did you edit the min/max values in each Data Template for each interface? Thats where you want to edit. changing the Data Query defaults is a bad idea since the max is dynamically set based off the reported interface speed.

[frode]

I found some more information that might be relevant.

In the logfile i find this line:

08/05/2015 09:30:01 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 496, Data Sources: traffic_in(DS[295]), traffic_out(DS[295]), traffic_in(DS[296]), traffic_out(DS[296]), traffic_in(DS[297]), traffic_out(DS[297]), traffic_in(DS[305]), traffic_out(DS[305]), traffic_in(DS[306]), traffic_out(DS[306]), traffic_in(DS[307]), traffic_out(DS[307]), traffic_in(DS[308]), traffic_out(DS[308]), traffic_in(DS[309]), traffic_out(DS[309]), traffic_in(DS[310]), traffic_out(DS[310]), traffic_in(DS[311]), traffic_out(DS[311]), Additional Issues Remain. Only showing first 20
Waiting on 1 of 1 pollers.
Waiting on 1 of 1 pollers.
Waiting on 1 of 1 pollers
Waiting on 1 of 1 pollers.
Waiting on 1 of 1 pollers.
Waiting on 1 of 1 pollers.
Waiting on 1 of 1 pollers.
OK u:0.00 s:0.00 r:0.14
Waiting on 1 of 1 pollers.
Waiting on 1 of 1 pollers.
Waiting on 1 of 1 pollers.
Waiting on 1 of 1 pollers.
<SNIP about 750 almost equal lines, where about 95% is waiting for pollers>
08/05/2015 09:30:12 AM - SYSTEM STATS: Time:10.5053 Method:cmd.php Processes:1 Threads:N/A Hosts:47 HostsPerProcess:47 DataSources:539 RRDsProcessed:43

And if i count the number of traffic rrd files:

/usr/share/cacti/site/rra# ls *traffic_*|wc -l
248

So if as there are both DS for inbound and outbound, 248 files*2=496. The same number as issues found on the poller output table. It seems to me that Cacti poller actually is fetching the data, but can't put it into the rrd files.

/usr/share/cacti/site/rra# ls |grep -v traffic_in|wc -l
45

Close to the same number as RRDs processed.

I have checked the permissions on the files, and as you can see in the first post, the permissions on the files are the same as on files that do work.... (CPU ++)

BSOD2600: Thank you for your answer. It might be a weathermap issue... But is there any where I can look for more info/logs? There must be a "error writing to RRD" somewhere....

I didn't change the value on each interface for all the 248 interfaces. I went to Templates/Data Templates/Interface - Traffic, and changed the "Maximum Value". I think this was set to the default value 100000000 (But not sure. Didn't think of it as important to write down at that moment), and I changed it to 8589934592 (8Gbit). I'm not sure if this is what you reefer to as "Data Query defaults".

[frode]

Have tested the poller_output_empty.php script. Also enabled DEBUG loglevel, and it has puzzled me that I only see the following output:

<SNIP>
08/05/2015 10:12:02 AM - WEATHERMAP: Poller[0] WM poller_output: ENDING

08/05/2015 10:12:02 AM - WEATHERMAP: Poller[0] WM poller_output: STARTING

08/05/2015 10:12:02 AM - WEATHERMAP: Poller[0] WM poller_output: ENDING

08/05/2015 10:12:02 AM - WEATHERMAP: Poller[0] WM poller_output: STARTING

08/05/2015 10:12:02 AM - WEATHERMAP: Poller[0] WM poller_output: ENDING

08/05/2015 10:12:02 AM - WEATHERMAP: Poller[0] WM poller_output: STARTING

08/05/2015 10:12:02 AM - WEATHERMAP: Poller[0] WM poller_output: ENDING

08/05/2015 10:12:02 AM - WEATHERMAP: Poller[0] WM poller_output: STARTING

08/05/2015 10:12:02 AM - WEATHERMAP: Poller[0] WM poller_output: ENDING

08/05/2015 10:12:02 AM - WEATHERMAP: Poller[0] WM poller_output: STARTING

08/05/2015 10:12:02 AM - WEATHERMAP: Poller[0] WM poller_output: ENDING

08/05/2015 10:12:02 AM - WEATHERMAP: Poller[0] WM poller_output: STARTING

08/05/2015 10:12:02 AM - WEATHERMAP: Poller[0] WM poller_output: ENDING

08/05/2015 10:12:02 AM - WEATHERMAP: Poller[0] WM poller_output: STARTING

08/05/2015 10:12:02 AM - WEATHERMAP: Poller[0] WM poller_output: ENDING

08/05/2015 10:12:02 AM - WEATHERMAP: Poller[0] WM poller_output: STARTING
<SNIP>

Searched for "WM poller_output: STARTING", and found it to be located in /usr/share/cacti/site/plugins/weathermap/setup.php, at line 654. On line 662 there is a sql statement, that seems to return zero rows.

After poking around in phpmyadmin it seems to me that the table weathermap_data is empty. So it's not that strange that the SQL returns nothing:

SELECT DISTINCT weathermap_data.id, weathermap_data.last_value, weathermap_data.last_time, weathermap_data.data_source_name, data_template_data.data_source_path, data_template_data.local_data_id, data_template_rrd.data_source_type_id
FROM weathermap_data, data_template_data, data_template_rrd
WHERE weathermap_data.local_data_id = data_template_data.local_data_id
AND data_template_rrd.local_data_id = data_template_data.local_data_id
AND weathermap_data.local_data_id <>0

Then again, I don't understand why it still works for the CPU graphs.

Can anyone check what their weathermap_data table contains? Should it be empty? Is there a way to rebuild it? Or am I just barking up the wrong tree?

[frode]

Hello.

Sorry for all the postings, but keeping this post updated if others run into the same problem.

I splitted the cacti.log into run sections before and after the problem occurred, removed timestamp, and kompared the files.

I notice that it seems as if all the OIDs for my 64-bit values has been changed.

I did not have the problem at 12:50, but at 12:55

1.3.6.1.2.1.31.1.1.1.6 -> ifHCInOctets -> 64-bit version of ifInOctets

hurricane:/var/log/cacti# grep 1.3.6.1.2.1.31.1.1.1.6 cacti.log-error-1250 |wc -l
241
hurricane:/var/log/cacti# grep 1.3.6.1.2.1.31.1.1.1.6 cacti.log-error-1255 |wc -l
0

hurricane:/var/log/cacti# grep 1.3.6.1.2.1.31.1.1.1.6 cacti.log-error-1250 |tail -n 1
08/03/2015 12:50:15 PM - CMDPHP: Poller[0] Host[64] DS[742] SNMP: v2: 172.16.106.137, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.21, output: 107822057291

1.3.6.1.2.1.2.2.1.10 -> ifInOctets

/var/log/cacti# grep 1.3.6.1.2.1.2.2.1.10 cacti.log-error-1250 |wc -l
7
/var/log/cacti# grep 1.3.6.1.2.1.2.2.1.10 cacti.log-error-1255 |wc -l
248

/var/log/cacti# grep 1.3.6.1.2.1.2.2.1.10 cacti.log-error-1255 |tail -n 1
08/03/2015 12:55:14 PM - CMDPHP: Poller[0] Host[64] DS[742] SNMP: v2: 172.16.106.137, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.21, output: 451995041

Seems that by changing the template all the interfaces that did a 64-bit snmp request has been changed. Don't know if this is the actual problem, but pursuing this for now.

By the logs I probably can write up a sql script so I can update the poller_item table that holds the OID's. I'll post a update as soon I have tried it.

[frode]

Hmmmm.....

The first link I changed worked, but when I changed them all, it did not fix anything... In fact, the first link stopped working to.... I don't think the poller_item table work the way I thought.

Anyone know where to change this setting on each interface. I can't seem to find where this is located....

I have tested to enable two interfaces, as new 64-bit data sources, and the host now have these interfaces twice... The new one works. So at least I can get things to work again, but then I have to re-enable all 246 interfaces, and change the rrd number in the weathermap configs..... But then I also will loose the history..... Irritating.... Hmmm... Trying to see if there is any last options I have overlooked. Anyone have any suggestions? I'm on the verge of giving up here.

Is there some where I can see what rrd "commit" command that has been tryed? I'v got the feeling that Cacti has tried to write to the rrd file, but something has been wrong with format, or number size or something.

[frode]

I'm throwing in the towel.

We have been to long without the Weathermap, so I'm now starting to add all the interfaces, and removing the old once, and updating all the links in Weathermap.

If anyone have suggestions to what can be done, I will keep a couple of the broken interfaces, and can test it out, to help other in the same situation.

Seems to me that there are a couple of other cases, but still without solutions, like

http://forums.cacti.net/viewtopic.php?f=16&t=39474