rrdtool update leaves one data item nan, others updated

[oneaty]

I have a graph that's based on a script output and should be graphing 8 data items.

However, only the last seven are correctly showing values. The first one doesn't.

After much struggle, I found this tutorial on debugging nan's, which was excelent to freeze my random attempts to correct the issue and bring deeper understanding on how Cacti and rrd works.

I found one issue, which explains why my graph fails to show one data item: the rrdtool update statement does not reference the first data item in the data source:

Code: Select all

02/07/2015 05:09:02 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /var/lib/cacti/rra/humax_hg100r2_corr1_197.rrd --template corr2:corr3:corr4:corr5:corr6:corr7:corr8 1423336141:159:121:96:95:87:90:98

As seen above, corr1 data item is missing, despite the fact that the script is generating data for it:

Code: Select all

02/07/2015 04:37:02 PM - SPINE: Poller[0] Host[2] TH[1] DS[197] SCRIPT: /usr/share/cacti/site/scripts/humax_down_correctables, output: corr1:81 corr2:159 corr3:121 corr4:96 corr5:95 corr6:87 corr7:90 corr8:98

and mysql is also inserting data regarding that data item:

Code: Select all

02/07/2015 04:20:01 PM - SPINE: Poller[0] DEVDBG: SQL:'INSERT INTO poller_output (local_data_id, rrd_name, time, output) VALUES (197,'','2015-02-07 16:20:01','corr1:81 corr2:159 corr3:121 corr4:96 corr5:95 corr6:87 corr7:90 corr8:98') ON DUPLICATE KEY UPDATE output=VALUES(output)'

When I enter Cacti Data Template, the data item is there, as well as the others that are correctly updated by rrdtool. I believe this rrdtool info output confirms that:

Code: Select all

rrd_version = "0003"
step = 60
last_update = 1423337222
header_size = 9680
ds[corr1].index = 0
ds[corr1].type = "GAUGE"
ds[corr1].minimal_heartbeat = 120
ds[corr1].min = NaN
ds[corr1].max = NaN
ds[corr1].last_ds = "U"
ds[corr1].value = NaN
ds[corr1].unknown_sec = 2
ds[corr2].index = 1
ds[corr2].type = "GAUGE"
ds[corr2].minimal_heartbeat = 120
ds[corr2].min = NaN
ds[corr2].max = NaN
ds[corr2].last_ds = "159"
ds[corr2].value = 3,1800000000e+02
ds[corr2].unknown_sec = 0
ds[corr3].index = 2
ds[corr3].type = "GAUGE"
ds[corr3].minimal_heartbeat = 120
ds[corr3].min = NaN
ds[corr3].max = NaN
ds[corr3].last_ds = "121"
ds[corr3].value = 2,4200000000e+02
ds[corr3].unknown_sec = 0
ds[corr4].index = 3
ds[corr4].type = "GAUGE"
ds[corr4].minimal_heartbeat = 120
ds[corr4].min = NaN
ds[corr4].max = NaN
ds[corr4].last_ds = "96"
ds[corr4].value = 1,9200000000e+02
ds[corr4].unknown_sec = 0
ds[corr5].index = 4
ds[corr5].type = "GAUGE"
ds[corr5].minimal_heartbeat = 120
ds[corr5].min = NaN
ds[corr5].max = NaN
ds[corr5].last_ds = "95"
ds[corr5].value = 1,9000000000e+02
ds[corr5].unknown_sec = 0
ds[corr6].index = 5
ds[corr6].type = "GAUGE"
ds[corr6].minimal_heartbeat = 120
ds[corr6].min = NaN
ds[corr6].max = NaN
ds[corr6].last_ds = "87"
ds[corr6].value = 1,7400000000e+02
ds[corr6].unknown_sec = 0
ds[corr7].index = 6
ds[corr7].type = "GAUGE"
ds[corr7].minimal_heartbeat = 120
ds[corr7].min = NaN
ds[corr7].max = NaN
ds[corr7].last_ds = "90"
ds[corr7].value = 1,8000000000e+02
ds[corr7].unknown_sec = 0
ds[corr8].index = 7
ds[corr8].type = "GAUGE"
ds[corr8].minimal_heartbeat = 120
ds[corr8].min = NaN
ds[corr8].max = NaN
ds[corr8].last_ds = "98"
ds[corr8].value = 1,9600000000e+02
ds[corr8].unknown_sec = 0

What am I missing here?

[BSOD2600]

In the Data Input Method for your humax_down_correctables script, is the 'Update RRA' enabled for corr1?

[oneaty]

In the Data Input Method for your humax_down_correctables script, is the 'Update RRA' enabled for corr1?

In Output Fields view, I can see a field named Update RRD File, which is enabled.

When in Data Input Method view, the column Update RRA shows Selected for all output fields, including corr1.

[oneaty]

Not sure if this can help:

Code: Select all

mysql> select t_data_source_name, data_source_name, t_rrd_maximum, rrd_maximum, t_rrd_minimum, rrd_minimum, t_rrd_heartbeat, rrd_heartbeat from data_template_rrd where data_template_id = '88' order by data_source_name;
+--------------------+------------------+---------------+-------------+---------------+-------------+-----------------+---------------+
| t_data_source_name | data_source_name | t_rrd_maximum | rrd_maximum | t_rrd_minimum | rrd_minimum | t_rrd_heartbeat | rrd_heartbeat |
+--------------------+------------------+---------------+-------------+---------------+-------------+-----------------+---------------+
| NULL               | corr1            | NULL          | U           | NULL          | U           | NULL            |           120 |
|                    | corr1            |               | U           |               | U           |                 |           120 |
| NULL               | corr2            | NULL          | U           | NULL          | U           | NULL            |           120 |
|                    | corr2            |               | U           |               | U           |                 |           120 |
|                    | corr3            |               | U           |               | U           |                 |           120 |
| NULL               | corr3            | NULL          | U           | NULL          | U           | NULL            |           120 |
| NULL               | corr4            | NULL          | U           | NULL          | U           | NULL            |           120 |
|                    | corr4            |               | U           |               | U           |                 |           120 |
| NULL               | corr5            | NULL          | U           | NULL          | U           | NULL            |           120 |
|                    | corr5            |               | U           |               | U           |                 |           120 |
| NULL               | corr6            | NULL          | U           | NULL          | U           | NULL            |           120 |
|                    | corr6            |               | U           |               | U           |                 |           120 |
|                    | corr7            |               | U           |               | U           |                 |           120 |
| NULL               | corr7            | NULL          | U           | NULL          | U           | NULL            |           120 |
|                    | corr8            |               | U           |               | U           |                 |           120 |
| NULL               | corr8            | NULL          | U           | NULL          | U           | NULL            |           120 |
+--------------------+------------------+---------------+-------------+---------------+-------------+-----------------+---------------+
16 rows in set (0.00 sec)

[oneaty]

... and this:

Code: Select all

mysql> select data_name, input_output, update_rra, sequence from data_input_fields where data_input_id = '35' order by data_name;
+-----------+--------------+------------+----------+
| data_name | input_output | update_rra | sequence |
+-----------+--------------+------------+----------+
| corr1     | out          | on         |        0 |
| corr2     | out          | on         |        0 |
| corr3     | out          | on         |        0 |
| corr4     | out          | on         |        0 |
| corr5     | out          | on         |        0 |
| corr6     | out          | on         |        0 |
| corr7     | out          | on         |        0 |
| corr8     | out          | on         |        0 |
+-----------+--------------+------------+----------+
8 rows in set (0.00 sec)

and just in case there could be an inconsistency:

Code: Select all

mysql> select data_input_id, data_name, input_output, update_rra, sequence from data_input_fields where data_name like '%corr%' order by data_name;
+---------------+-----------+--------------+------------+----------+
| data_input_id | data_name | input_output | update_rra | sequence |
+---------------+-----------+--------------+------------+----------+
|            35 | corr1     | out          | on         |        0 |
|            35 | corr2     | out          | on         |        0 |
|            35 | corr3     | out          | on         |        0 |
|            35 | corr4     | out          | on         |        0 |
|            35 | corr5     | out          | on         |        0 |
|            35 | corr6     | out          | on         |        0 |
|            35 | corr7     | out          | on         |        0 |
|            35 | corr8     | out          | on         |        0 |
+---------------+-----------+--------------+------------+----------+
8 rows in set (0.00 sec)

[oneaty]

Sorry flooding the topic, but now I suspect the issue is when /lib/poller.php runs.

According to the log, corr1 field is not parsed:

Code: Select all

02/08/2015 02:05:02 PM - POLLER: Poller[0] Parsed MULTI output field 'corr8:98' [map corr8->corr8]
02/08/2015 02:05:02 PM - POLLER: Poller[0] Parsed MULTI output field 'corr7:90' [map corr7->corr7]
02/08/2015 02:05:02 PM - POLLER: Poller[0] Parsed MULTI output field 'corr6:87' [map corr6->corr6]
02/08/2015 02:05:02 PM - POLLER: Poller[0] Parsed MULTI output field 'corr5:95' [map corr5->corr5]
02/08/2015 02:05:02 PM - POLLER: Poller[0] Parsed MULTI output field 'corr4:101' [map corr4->corr4]
02/08/2015 02:05:02 PM - POLLER: Poller[0] Parsed MULTI output field 'corr3:130' [map corr3->corr3]
02/08/2015 02:05:02 PM - POLLER: Poller[0] Parsed MULTI output field 'corr2:299' [map corr2->corr2]

I looked into the code, but still couldn't find a clue...

[oneaty]

... getting closer:

Manually executing a simplified version of /lib/poller.php SELECT I get:

Code: Select all

mysql> select data_template_rrd.data_template_id, data_template_rrd.data_source_name, data_input_fields.data_name from (data_template_rrd,data_input_fields) where data_template_rrd.data_template_id = '88' and data_template_rrd.data_input_field_id=data_input_fields.id;
+------------------+------------------+-----------+
| data_template_id | data_source_name | data_name |
+------------------+------------------+-----------+
|               88 | corr8            | corr8     |
|               88 | corr6            | corr6     |
|               88 | corr7            | corr7     |
|               88 | corr4            | corr4     |
|               88 | corr5            | corr5     |
|               88 | corr3            | corr3     |
|               88 | corr2            | corr2     |
|               88 | corr3            | corr3     |
|               88 | corr4            | corr4     |
|               88 | corr5            | corr5     |
|               88 | corr6            | corr6     |
|               88 | corr7            | corr7     |
|               88 | corr8            | corr8     |
|               88 | corr2            | corr2     |
+------------------+------------------+-----------+
14 rows in set (0.00 sec)

corr1 is absent!

In table data_template_rrd there seems to be an issue regarding data_input_field_id value for corr1 entry:

Code: Select all

mysql> select data_source_name, rrd_maximum, rrd_minimum, rrd_heartbeat, data_input_field_id from data_template_rrd where data_template_id = '88' order by data_source_name;
+------------------+-------------+-------------+---------------+---------------------+
| data_source_name | rrd_maximum | rrd_minimum | rrd_heartbeat | data_input_field_id |
+------------------+-------------+-------------+---------------+---------------------+
| corr1            | U           | U           |           120 |                   0 |
| corr1            | U           | U           |           120 |                   0 |
| corr2            | U           | U           |           120 |                 383 |
| corr2            | U           | U           |           120 |                 383 |
| corr3            | U           | U           |           120 |                 384 |
| corr3            | U           | U           |           120 |                 384 |
| corr4            | U           | U           |           120 |                 385 |
| corr4            | U           | U           |           120 |                 385 |
| corr5            | U           | U           |           120 |                 386 |
| corr5            | U           | U           |           120 |                 386 |
| corr6            | U           | U           |           120 |                 387 |
| corr6            | U           | U           |           120 |                 387 |
| corr7            | U           | U           |           120 |                 388 |
| corr7            | U           | U           |           120 |                 388 |
| corr8            | U           | U           |           120 |                 389 |
| corr8            | U           | U           |           120 |                 389 |
+------------------+-------------+-------------+---------------+---------------------+
16 rows in set (0.00 sec)

I don't think I can go further on this: I have deleted and recreated everything for this graph (data input method, data template, graph template) and the corr1 issue remains.

[oneaty]

I created a new Data Input Method and Data Template.

This time,after each new Data Template Item, I monitored the database by running the command

Code: Select all

select data_source_name, rrd_maximum, rrd_minimum, rrd_heartbeat, data_input_field_id from data_template_rrd where data_template_id = '88' order by data_source_name;

Again, the field data_input_field_id was 0 for the first item, but not the following.

Then I update the min and max fields from U to 0 and data_input_field was correctly updated with an entry greater than 0.

Then I updated again min and max to U and this time data_input_field wasn't reset.

Important to note that the second field output field was also created for the first time with U for min and max fields.

So, to conclude, I think the error happens when you create the first data output field with U.

I consider this solved for now, considering that there is a workaround.

[BSOD2600]

Nice sleuthing.

[thenextdon13]

I've run into a similar problem, except that I _DO_ have data_input_field_id 's.

I'll hopefully update when i figure out more about what is going on.

[thenextdon13]

So, i added some print statements to poller.php:

Code: Select all

379                         }else{
380                                 /* below by camdenl*/
381                                 cacti_log("blahblah added value=" . $value, true, "POLLER");
382                                 $values = explode(" ", $value);
383 
384                                 $rrd_field_names = array_rekey(db_fetch_assoc("select
385                                         data_template_rrd.data_source_name,
386                                         data_input_fields.data_name
387                                         from (data_template_rrd,data_input_fields)
388                                         where data_template_rrd.data_input_field_id=data_input_fields.id
389                                         and data_template_rrd.local_data_id=" . $item["local_data_id"]), "data_name", "data_source_name");
390 
391                                 if (sizeof($values)) {
392                                 foreach($values as $value) {
393                                         $matches = explode(":", $value);
394                                         /* camdenl added below*/
395                                         cacti_log("blahblah added explodevalue=" . $value, true, "POLLER");
396 

And now i can see that for some reason poller is eating the first couple characters of my inputs:

Code: Select all

02/02/2016 03:37:35 PM - POLLER: Poller[0] blahblah added value=4_1:27 sg4_2:29 sg4_3:28 sg4_4:26 sg4_5:sg4_6:sg4_7:29 sg4_8:27
02/02/2016 03:38:05 PM - POLLER: Poller[0] blahblah added value=0_Temp:26.1 CPU1_Temp:24.2 CPU1_Temp:25.4 CPU1_Temp:24.8 Rear_Fan:1739 CPU_Fan:1088 HD_Fan0:1721 HD_Fan1:1721

The script output works as expected

Code: Select all

htdocs/cacti $ scripts/get_drivetemp_results 
sg4_1:27 sg4_2:29 sg4_3:28 sg4_4:26 sg4_5:sg4_6:sg4_7:29 sg4_8:27 htdocs/cacti $ scripts/get_sensors 
CPU0_Temp:26.1 CPU1_Temp:24.2 CPU1_Temp:25.4 CPU1_Temp:25.0 Rear_Fan:1739 CPU_Fan:1088 HD_Fan0:1721 HD_Fan1:1721