[HELP] Cacti not updating all datasources on rrdfile

[rsanto]

Hello,

I'm running Cacti with Spine and the Boost plugin. When running on Boost all the graphs with more than 4 elements started to get NAN values. Today I had to create new templates to measure temperature on 10 places on the same equipment and the same thing happened. Looking to the log it looks Spine is working fine but the values are not getting to boost:

11/13/2013 03:40:13 PM - SPINE: Poller[0] Host[441] TH[1] DS[214387] SCRIPT: perl /var/www/cacti/resource/scripts/cisco/10kCMTS_temperature.pl <IP> <SNMP_COMM>, output: index1:27 index2:35 index3:0 index4:0 index5:0 index6:40 index7:37 index8:37 index9:40 index10:38
11/13/2013 03:44:49 PM - BOOST: Poller[0] NOTE: Updating Local Data ID:'214387', Total of '1' Updates to Process
11/13/2013 03:44:49 PM - BOOST: Poller[0] CACTI2RRD: /usr/bin/rrdtool last /usr/share/cacti/rra/441/214387.rrd
11/13/2013 03:44:49 PM - BOOST: Poller[0] NOTE: Updating Local Data Id:'214387', Template:Outlet:SubSlot50:SubSlot51:SubSlot60, Output: 1384357211:35:0:0:0
11/13/2013 03:44:49 PM - BOOST: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /usr/share/cacti/rra/441/214387.rrd --template Outlet:SubSlot50:SubSlot51:SubSlot60 1384357211:35:0:0:0

And the graph is only showing values for the fields Outlet, SubSlot50, SubSlot51 and SubSlot60

Any idea how to solve this?

Many thanks

Cacti specs:

Cacti Version 0.8.8a
Cacti OS unix
SNMP Version NET-SNMP version: 5.5
RRDTool Version RRDTool 1.3.x
Hosts 777
Graphs 99865
Data Sources Script/Command: 483
SNMP: 12326
SNMP Query: 163171
Script Query: 12
Script - Script Server (PHP): 17
Script Query - Script Server: 850
Total: 176859
Poller Information
Interval 300
Type SPINE 0.8.8a Copyright 2002-2012 by The Cacti Group
Items Action[0]: 61842
Action[1]: 458
Action[2]: 1485
Total: 63785
Concurrent Processes 1
Max Threads 80
PHP Servers 10
Script Timeout 60
Max OID 15
Last Run Statistics Time:109.3010 Method:spine Processes:1 Threads:80 Hosts:722 HostsPerProcess:722 DataSources:63785 RRDsProcessed:0

[rsanto]

I changed log level to debug and got:

11/15/2013 12:09:19 PM - BOOST: Poller[0] CACTI2RRD: /usr/bin/rrdtool last /usr/share/cacti/rra/441/214387.rrd
11/15/2013 12:09:19 PM - CMDPHP: Poller[0] DEVEL: SQL Assoc: "SELECT data_template_rrd.data_source_name, data_input_fields.data_name FROM (data_template_rrd,data_input_fields) WHERE data_template_rrd.data_input_field_id=data_input_fields.id AND data_template_rrd.local_data_id=214387"

query response:

+------------------+-----------+
| data_source_name | data_name |
+------------------+-----------+
| Outlet | index2 |
| SubSlot50 | index3 |
| SubSlot51 | index4 |
| SubSlot60 | index5 |
| SubSlot61 | index6 |
| SubSlot70 | index7 |
| SubSlot71 | index8 |
| SubSlot80 | index9 |
| SubSlot81 | index10 |
+------------------+-----------+

Inlet is missing can't understand why, it seems to be there:

mysql> SELECT data_template_rrd.data_source_name, data_input_fields.data_name FROM (data_template_rrd,data_input_fields) WHERE data_source_name="Inlet" AND data_name="index1";
+------------------+-----------+
| data_source_name | data_name |
+------------------+-----------+
| Inlet | index1 |
| Inlet | index1 |
| Inlet | index1 |
+------------------+-----------+
3 rows in set (0.00 sec)

mysql> SELECT data_template_rrd.data_source_name, data_input_fields.data_name FROM (data_template_rrd,data_input_fields) WHERE data_source_name="Outlet" AND data_name="index2";
+------------------+-----------+
| data_source_name | data_name |
+------------------+-----------+
| Outlet | index2 |
| Outlet | index2 |
| Outlet | index2 |
+------------------+-----------+
3 rows in set (0.00 sec)

rest of log file...

11/15/2013 12:09:19 PM - BOOST: Poller[0] NOTE: Updating Local Data Id:'214387', Template:Outlet:SubSlot50:SubSlot51:SubSlot60, Output: 1384515613:35:0:0:0 1384515913:35:0:0:0 1384516213:35:0:0:0 1384516512:35:0:0:0 1384516812:35:0:0:0 1384517112:35:0:0:0
11/15/2013 12:09:19 PM - BOOST: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /usr/share/cacti/rra/441/214387.rrd --template Outlet:SubSlot50:SubSlot51:SubSlot60 1384515613:35:0:0:0 1384515913:35:0:0:0 1384516213:35:0:0:0 1384516512:35:0:0:0 1384516812:35:0:0:0 1384517112:35:0:0:0
11/15/2013 12:09:27 PM - BOOST: Poller[0] NOTE: Updating Local Data Id:'216293', Template:traffic_in:traffic_out, Output: 1384515608:0:21437767526696 1384515908:0:21438055267831 1384516209:0:21438367139703 1384516508:0:21438770226037 1384516809:0:21439129190732 1384517108:0:21439558668033
11/15/2013 12:09:27 PM - BOOST: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /usr/share/cacti/rra/314/216293.rrd --template traffic_in:traffic_out 1384515608:0:21437767526696 1384515908:0:21438055267831 1384516209:0:21438367139703 1384516508:0:21438770226037 1384516809:0:21439129190732 1384517108:0:21439558668033