hdd utilisation of various volumes not displayed

[marki4711]

Hello,

the error-message in the log-file that seems to be the root cause is often mentioned in the forum and I read multiple articles in the forum without finding a solution for the problem. This is the reason, why I open y new topic.

The error message is well known:
08/18/2011 04:07:01 PM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 12, Data Sources: hdd_total(DS[524]), hdd_used(DS[524]), hdd_percent(DS[538]), hdd_used(DS[538]), hdd_total(DS[539]), hdd_used(DS[556]), hdd_total(DS[723]), hdd_used(DS[723]), hdd_total(DS[724]), hdd_used(DS[724]), hdd_total(DS[725]), hdd_used(DS[725])

The problem happen only with monitoring the hdd utilisation (windows and unix) and when I use realtime-view on the volumes I get the correct values. I changed from spine to cmd.php and back. I recreated the graphs and got the same result that some volumes are not displayed. other volumes of the same host are created with the correct values but some do not work. I attached a screenshot of a server that has the described problem.

Any help is welcome!

BRs,
Matthias

General Information
Date Thu, 18 Aug 2011 16:15:11 +0200
Cacti Version 0.8.7g
Cacti OS unix
SNMP Version NET-SNMP version: 5.1.2
RRDTool Version RRDTool 1.2.x
Hosts 31
Graphs 256
Data Sources Script/Command: 10
SNMP: 83
SNMP Query: 78
Script - Script Server (PHP): 15
Script Query - Script Server: 114
Total: 300

[marki4711]

any idea?

BRs,
Matthias

[marki4711]

new information:
according to debug, all information are fetched correctly. The values are also available in the mysql-table poller_output but the graph shows only nan values. It seems that the transfer from poller_output to the rrd-files causes my troubles.

BRs,
Matthias

[marki4711]

Another strange behaviour:
I followed the instructions on http://docs.cacti.net/manual:088:6_refe ... r%20output to empty the poller_output table because I saw in mysql that this table is not empty but it doesn't work. The empty_poller_output-script says, that everything is alright but the table still contains values.

Clean Poller output:

Code: Select all

[root@lnzsx09 cli]# php -q poller_output_empty.php
There were 0, RRD updates made this pass

Result:

Code: Select all

mysql> select * from poller_output;
+---------------+-----------+---------------------+-------------+
| local_data_id | rrd_name  | time                | output      |
+---------------+-----------+---------------------+-------------+
|           743 | hdd_used  | 2011-08-25 08:16:02 | 30804463616 |
|           743 | hdd_total | 2011-08-25 08:16:02 | 42842714112 |
|           723 | hdd_used  | 2011-08-25 08:16:02 | 12720128    |
|           725 | hdd_used  | 2011-08-25 08:16:02 | 31530344448 |
|           725 | hdd_total | 2011-08-25 08:16:02 | 50770182144 |
|           723 | hdd_total | 2011-08-25 08:16:02 | 103531520   |
|           724 | hdd_total | 2011-08-25 08:16:02 | 10141114368 |
|           556 | hdd_used  | 2011-08-25 08:16:02 | 2042167296  |
|           724 | hdd_used  | 2011-08-25 08:16:02 | 2618830848  |
|           524 | hdd_used  | 2011-08-25 08:16:02 | 191569920   |
|           524 | hdd_total | 2011-08-25 08:16:02 | 2125418496  |
+---------------+-----------+---------------------+-------------+
11 rows in set (0.00 sec)

I expected, to see that table empty after issuing the poller_output_empty command.

[marki4711]

Next piece of the puzzle.

I followed the php-code and found the piece of source code that is responsible for leaving the rows in the poller_output-table.

poller.php:

Code: Select all

foreach ($results as $item) {
                        $unix_time = strtotime($item["time"]);
                        if (isset($rrd_update_array{$item["rrd_path"]}["times"][$unix_time])) {
                                if ($item["rrd_num"] <= sizeof($rrd_update_array{$item["rrd_path"]}["times"][$unix_time])) {
                                        db_execute("delete from poller_output where local_data_id='" . $item["local_data_id"] . "' and rrd_name='" . $item["rrd_name"] . "' and time='" . $item["time"] . "'");
                                }else{
                                        unset($rrd_update_array{$item["rrd_path"]}["times"][$unix_time]);
                                }
                        }
                }

Each value that is not transfered to the graph ends in the else-branch of this statement. I currently don't know why this happens but I keep searching. I would also be interested what the parameter rrd_num stands for. Would make it easier to understand the query.

BRs,
Matthias

[marki4711]

New knowledge:
As far as I understand the statement, it checks if enought data is available for a full update of the rrd. In my case I need three values for a harddrive (hdd_percent, hdd_total, hdd_used)

When I check the remaining rows in the poller_output-table, there are only two of these three values available. Most of the time it is hdd_total and hdd_used.

I consulted the manual and found an information that says "The poller cache holds all commands that cacti will issue during the polling process in an internal format". This made me check the poller cache and I found out that the number of entries in the poller-cache is identical to the number of entries in my poller_output table. This make me think that there are some entries missing in the poller-cache and this is the reason for my problems. Rebuilding the poller cache does not help, so I continue my investigation.

[marki4711]

Sigh, new knowledge but still no solution:
It seems that the Poller-Buffer is built with update_poller_cache in /lib/utility.php.

There is a part where outputs are used to generate the poller cache and it seems that the generation of the poller-cache only works, if the output hrStoragePercent is the third entry of the output-array. It doesn't work if hrStoragePercent is the first entry. The reason for the order of the output-entries is unknown.

BRs,
Matthias

[marki4711]

Yahoo!


I don't understand all influencing factors but it seems that my assumption regarting the order of the output-values is important. I changed a sql-query and now I don't have a single "WARNING: Poller Output Table not Empty." in my Logs and all graphs have values.

I changed this query in /lib/utility.php:

Code: Select all

     
     83                 $outputs = db_fetch_assoc("select
     84                         snmp_query_graph_rrd.snmp_field_name,
     85                         data_template_rrd.id as data_template_rrd_id
     86                         from (snmp_query_graph_rrd,data_template_rrd)
     87                         where snmp_query_graph_rrd.data_template_rrd_id=data_template_rrd.local_data_template_rrd_id
     88                         $output_type_sql
     89                         and snmp_query_graph_rrd.data_template_id=" . $data_input["data_template_id"] . "
     90                         and data_template_rrd.local_data_id=$local_data_id");

to

Code: Select all

     
     83                 $outputs = db_fetch_assoc("select
     84                         snmp_query_graph_rrd.snmp_field_name,
     85                         data_template_rrd.id as data_template_rrd_id
     86                         from (snmp_query_graph_rrd,data_template_rrd)
     87                         where snmp_query_graph_rrd.data_template_rrd_id=data_template_rrd.local_data_template_rrd_id
     88                         $output_type_sql
     89                         and snmp_query_graph_rrd.data_template_id=" . $data_input["data_template_id"] . "
     90                         and data_template_rrd.local_data_id=$local_data_id order by data_template_rrd.id");

I hope that a deveoper of Cacti takes a look at this change and if it makes sense to him and if it is possible to integrate the change to the official release. It took a lot of time to find this fix.

BRs,
Matthias

[erik.costa]

Great job!!!

Thanks for sharing with us.