Graphs not displaying on 'first' view -- refresh required

rilindo · Post by **rilindo** » Sat Apr 17, 2010 9:24 am

Besides weathermap, there was also thold and settings plugins. I moved them all out of the way, with the same unfortunate results.

Got Firebug installed and look at the responses. However, every time I looked at the response tab, I see a fully rendered image (even though it doesn't display on the main page). This continues even after I disabled preview tool-tips (which I thought was the problem).

Post by **TheWitness** » Sat Apr 17, 2010 9:25 am

Sounds like a browser issue then. Weird. What is the OS, OS Version, and Browser Version?

TheWitness

Post by **TheWitness** » Sat Apr 17, 2010 9:26 am

As you mentioned, it could also be boost taking too long. Also, do the following:

Code: Select all

mysql cacti
show create table poller_output_boost;

TheWitness

Post by **TheWitness** » Sat Apr 17, 2010 9:30 am

Also, you will note the following post. Has been a PITA for me lately.

http://forums.cacti.net/viewtopic.php?t=37070

Before you apply, you should write a simple script and look for long running queries on your database. Here is the script that I use.

Code: Select all

#!/bin/bash
while [ 1 ]; do
  clear
  mysql -e "show processlist" | grep -v Sleep | grep -v NULL
  sleep 2
done

If you run this script, all the slow running queries will be shown (most of them anyway, you may have to run "show full processlist" to get the whole query).

If you find a long running query, you can copy that query to the clipboard and then do the following from inside MySQL:

Code: Select all

explain <myquery>

If the explain comes back with "possible keys" being NULL for your queries, you have the aforementioned bug and best to goto InnoDB.

TheWitness

rilindo · Post by **rilindo** » Sat Apr 17, 2010 1:10 pm

Here you go - the structure.

mysql> show create table poller_output_boost;
+---------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+---------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| poller_output_boost | CREATE TABLE `poller_output_boost` (
`local_data_id` mediumint(8) unsigned NOT NULL DEFAULT '0',
`rrd_name` varchar(19) NOT NULL DEFAULT '',
`time` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
`output` varchar(50) NOT NULL DEFAULT '',
PRIMARY KEY (`local_data_id`,`rrd_name`,`time`) USING BTREE,
KEY `time_local_data_id` (`time`,`local_data_id`) USING BTREE,
KEY `local_data_id` (`local_data_id`) USING BTREE
) ENGINE=MEMORY DEFAULT CHARSET=latin1 ROW_FORMAT=FIXED |
+---------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

I have been running again against the slow queries I see in the slow log. What I see show up the most is the rra entries. Ran explain:

mysql> explain select rra.id, rra.steps, rra.rows, rra.name, rra.timespan, data_template_data.rrd_step from (graph_templates_item,data_template_data_rra,data_template_rrd,data_template_data,rra) where graph_templates_item.task_item_id=data_template_rrd.id and data_template_rrd.local_data_id=data_template_data.local_data_id and data_template_data.id=data_template_data_rra.data_template_data_id and data_template_data_rra.rra_id=rra.id and graph_templates_item.local_graph_id=7303 group by rra.id order by rra.timespan;
+----+-------------+------------------------+--------+------------------------------------------------------+----------------+---------+--------------------------------------------------+------+---------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+------------------------+--------+------------------------------------------------------+----------------+---------+--------------------------------------------------+------+---------------------------------+
| 1 | SIMPLE | graph_templates_item | ref | local_graph_id,task_item_id | local_graph_id | 3 | const | 10 | Using temporary; Using filesort |
| 1 | SIMPLE | data_template_rrd | eq_ref | PRIMARY,local_data_id,local_data_id_data_source_name | PRIMARY | 3 | cactidco2.graph_templates_item.task_item_id | 1 | |
| 1 | SIMPLE | data_template_data | ref | PRIMARY,local_data_id | local_data_id | 3 | cactidco2.data_template_rrd.local_data_id | 1 | |
| 1 | SIMPLE | rra | ALL | PRIMARY | NULL | NULL | NULL | 5 | Using join buffer |
| 1 | SIMPLE | data_template_data_rra | eq_ref | PRIMARY,data_template_data_id | PRIMARY | 6 | cactidco2.data_template_data.id,cactidco2.rra.id | 1 | Using index |
+----+-------------+------------------------+--------+------------------------------------------------------+----------------+---------+--------------------------------------------------+------+---------------------------------+
5 rows in set (0.00 sec)

I take it that the 4th entry is not supposed to return a "NULL" key, right?

Post by **TheWitness** » Sat Apr 17, 2010 1:25 pm

That looks all good. It's the "possible keys" column that we are worried about. How long do those queries run?

What is your key_buffer set to? What is your query_cache set to?

TheWitness

rilindo · Post by **rilindo** » Sat Apr 17, 2010 1:36 pm

Whoops, I should have waited.

I went and converted all the cacti tables into innodb. The problem still persists.

(that said, long term we probably needed to move to innodb fully anyway, so this is not a big deal).

Here are my current settings:

query_cache_size = 128M
key_buffer = 512M

At any event, does boost commit the files to disk before they get displayed din cacti? That would seem to be the case from the apacha logs, as I am not getting 404 messages. In that case, would the issue likely less to do with the database and more with something else (i.e. some sort of cache that I am missing)?

rilindo · Post by **rilindo** » Sat Apr 17, 2010 2:59 pm

As another experience, I copied the rra files over to a temporary ram disk to see if it is a disk latency problem. the problem still exists even with the rra files in ramdisk.

So at least it is not disk latency. :|

rilindo · Post by **rilindo** » Sun Apr 18, 2010 4:32 pm

Okay, coverted back to MyISAM and then I turned on DEBUG mode in cacti. Whenever I view a graph, I see this in the log:

04/18/2010 02:17:52 PM - CMDPHP: Poller[0] DEVEL: SQL Assoc: "SELECT name, file, function FROM plugin_hooks WHERE status = 1 AND hook = 'config_arrays'"
04/18/2010 02:17:52 PM - CMDPHP: Poller[0] DEVEL: SQL Assoc: "SELECT * FROM plugin_realms ORDER BY plugin, display"
04/18/2010 02:17:52 PM - CMDPHP: Poller[0] DEVEL: SQL Assoc: "SELECT name, file, function FROM plugin_hooks WHERE status = 1 AND hook = 'config_settings'"
04/18/2010 02:17:52 PM - CMDPHP: Poller[0] DEVEL: SQL Assoc: "SELECT name, file, function FROM plugin_hooks WHERE status = 1 AND hook = 'valid_host_fields'"
04/18/2010 02:17:52 PM - CMDPHP: Poller[0] DEVEL: SQL Assoc: "SELECT name, file, function FROM plugin_hooks WHERE status = 1 AND hook = 'config_form'"
04/18/2010 02:17:52 PM - CMDPHP: Poller[0] DEVEL: SQL Assoc: "SELECT name, file, function FROM plugin_hooks WHERE status = 1 AND hook = 'config_insert'"
04/18/2010 02:17:52 PM - CMDPHP: Poller[0] DEVEL: SQL Exec: "INSERT INTO poller_output_boost_processes (sock_int_value, status) VALUES ('829','OK')"

Running the queries manually, I get the following:

mysql> SELECT name, file, function FROM plugin_hooks WHERE status = 1 AND hook = 'config_arrays';
+----------+-----------+----------------------+
| name | file | function |
+----------+-----------+----------------------+
| internal | | plugin_config_arrays |
| boost | setup.php | boost_config_arrays |
+----------+-----------+----------------------+
2 rows in set (0.00 sec)

mysql> SELECT * FROM plugin_realms ORDER BY plugin, display;
+----+------------+----------------------------------+----------------------------------------+
| id | plugin | file | display |
+----+------------+----------------------------------+----------------------------------------+
| 1 | internal | plugins.php | Plugin Management |
| 3 | weathermap | weathermap-cacti-plugin-mgmt.php | Plugin -> Weathermap: Configure/Manage |
| 2 | weathermap | weathermap-cacti-plugin.php | Plugin -> Weathermap: View |
+----+------------+----------------------------------+----------------------------------------+
3 rows in set (0.00 sec)

mysql> SELECT name, file, function FROM plugin_hooks WHERE status = 1 AND hook = 'config_settings';
+-------+-----------+-----------------------+
| name | file | function |
+-------+-----------+-----------------------+
| boost | setup.php | boost_config_settings |
+-------+-----------+-----------------------+
1 row in set (0.00 sec)

mysql> SELECT name, file, function FROM plugin_hooks WHERE status = 1 AND hook = 'valid_host_fields';
Empty set (0.00 sec)

mysql> SELECT name, file, function FROM plugin_hooks WHERE status = 1 AND hook = 'config_form';
Empty set (0.00 sec)

mysql> SELECT name, file, function FROM plugin_hooks WHERE status = 1 AND hook = 'config_insert';
Empty set (0.00 sec)

A few questions

1) Should the poller boost except two records to return with the "config_array" query

2) Should the poller boost be in teh plugin_realms? It says it is installed cacti, so I would think it should exit in that able, though.

3) Why would the poller_boost look for fields that doesn't exists in plugin_hooks table:

mysql> select * from plugin_hooks;
+----+----------+------------------------------------+-----------+-----------------------------+--------+
| id | name | hook | file | function | status |
+----+----------+------------------------------------+-----------+-----------------------------+--------+
| 1 | internal | config_arrays | | plugin_config_arrays | 1 |
| 2 | internal | draw_navigation_text | | plugin_draw_navigation_text | 1 |
| 26 | boost | config_arrays | setup.php | boost_config_arrays | 1 |
| 27 | boost | config_settings | setup.php | boost_config_settings | 1 |
| 28 | boost | poller_on_demand | setup.php | boost_poller_on_demand | 1 |
| 29 | boost | poller_bottom | setup.php | boost_poller_bottom | 1 |
| 30 | boost | poller_command_args | setup.php | boost_poller_command_args | 1 |
| 31 | boost | draw_navigation_text | setup.php | boost_draw_navigation_text | 1 |
| 32 | boost | rrdtool_function_graph_cache_check | setup.php | boost_graph_cache_check | 1 |
| 33 | boost | rrdtool_function_fetch_cache_check | setup.php | boost_fetch_cache_check | 1 |
| 34 | boost | rrdtool_function_graph_set_file | setup.php | boost_graph_set_file | 1 |
| 35 | boost | prep_graph_array | setup.php | boost_prep_graph_array | 1 |
| 36 | boost | utilities_list | setup.php | boost_utilities_list | 1 |
| 37 | boost | utilities_action | setup.php | boost_utilities_action | 1 |
+----+----------+------------------------------------+-----------+-----------------------------+--------+

Post by **TheWitness** » Sun Apr 18, 2010 9:59 pm

That's not boost doing the calls, that the PIA doing that. All those queries are pretty much standard.

TheWitness

rilindo · Post by **rilindo** » Mon Apr 19, 2010 10:24 am

Fair enough. Sorry about that.

I should mention that I didn't originally setup this cacti installation, so it looking at the cacti tarball, it turns out that thold doesn't come with it. Looking back at the current install, it seems that person had put in an older version of thold. (the one that predates PIA). So it is likely that there is something with the install that is interfering with boost.

I'll try to reinstall cacti and see what happens (already cleaned up much of the old thold tables, which does not appear to have any affect).

rilindo · Post by **rilindo** » Mon Apr 19, 2010 1:06 pm

Cacti reinstalled. I think I cleaned out most of thold's entries in the database (in settings table). The problem still exists.

If it was a problem with an existing plugin, how would that come about? Would the issue arise from PIA? If so, any way to bypass PIA with boost?

rilindo · Post by **rilindo** » Mon Apr 19, 2010 1:36 pm

FWIW, aside from this problem, Boost is working wonderfully for us, as it allow us to scale with the HW we have. So short of re-engineering the infrastructure to load the RRAs directly to/from a ramdisk, this is really the way to go for us. Thanks for your work!

Post by **TheWitness** » Mon Apr 19, 2010 8:53 pm

Not too sure. Definitely some other plugin causing issues. Could be your PIA version too. You should be running 2.4++.

TheWitness

rilindo · Post by **rilindo** » Mon Apr 19, 2010 9:21 pm

Is 2.6 fine? I can try to downgrade to 2.4 and see what happens.

Also, in looking at the documentation, it actually doesn't make it clear as to how much memory I should allocate PHP. What type of setting would I change in php.ini (I had set them to 64/32 megabytes in various locations).

Graphs not displaying on 'first' view -- refresh required

Who is online