Graphs not displaying on 'first' view -- refresh required

[rilindo]

More updates:

Downgrading PIA did not address the issue.

However, I think there is some sort of misunderstanding on my part as the to the real issue and I apologize for that.

Essentially, it looks like Boost is working as intended, if on-demand is turned out.

04/23/2010 09:40:07 AM - SYSTEM STATS: Time:5.5949 Method:spine Processes:4 Threads:40 Hosts:892 HostsPerProcess:223 DataSources:14892 RRDsProcessed:0
04/23/2010 09:40:15 AM - SYSTEM BOOST STATS: Time:7.7700 RRDUpdates:14784

It appears that all the rrd info are being inserted into the database and then boost writes the out to disk. For the short term, this actually works for us to reduce the I/O.

What is not working is on-demand - once we turn it out, every request we made returns as a broken image.

My colleague and I have browse though the code and it looks like its not likely a boost problem. So whatever problem it is, it is outside of boost.

What is supposed to happen when you enable on-demand? Is cacti supposed to wait until boost writes to file or does it try to read the file immediately? If the former, what return code is it supposed t get from boost when it is done writing the rrd file to disk?

[TheWitness]

Remove all other plugins except boost, cause it works... Also, turn off image caching.

[rilindo]

Well, turning off image caching didn't work. However. . .

Just for the way of comparision, I setup a separate instance of cacti on my home server and right from the box, it works with no issues. So I started to compare the one at home and at work.

First thing I found is that poller_output was a memory table and the output field was a varchar (rather than text). Finally, it didn't have the right primary keys.

Got that all fixed. Still have the problem, though.

But it seems I am getting closer. It appears that the production cacti had boost before, but was removed for some unknown reason (the cacti version was 8.7). So one thing I wonder is whether there is any significant differences between the old version of boost and the current version (3.x) as how it needs to be configured in the database.

The other thing. . . is that it appears that poller_output table kept being updated and not emptied (like it should be, apparently), while the one at home appears to be properly truncated. I am not sure if that is notable or not, but as both the production and home cacti have the same cacti version now, I fairly mystified as to why the poller_output table behavior is different between the two.

[rilindo]

Okay, I think I fixed the poller_output problem. It appears that the collation is wrong:

mysql error: Illegal mix of collations (latin1_swedish_ci,IMPLICIT and (latin1_general_ci,IMPLICIT) for operation '='

When executing:

select poller_output.output, poller_output.time, poller_output.local_data_id, poller_item.rrd_path, poller_item.rrd_name, poller_item.rrd_num from (poller_output,poller_item) where (poller_output.local_data_id=poller_item.local_data_id and poller_output.rrd_name=poller_item.rrd_name) LIMIT 10000;

Fixed it by:

alter table poller_output modify rrd_name varchar(21) CHARACTER SET latin1 COLLATE latin1_swedish_ci NOT NULL;

Fixed the collation on the poller boost tables as well. It didn't resolve the main the problem, but it seems like I am getting close.

[gandalf]

Wow, what a find. Please open a bug report under http://bugs.cacti.net. We at least should check that condition
R.

[rilindo]

Wow, what a find. Please open a bug report under http://bugs.cacti.net. We at least should check that condition
R.

To be fair, I think the database I am working with is kinda odd to begin with, so I don't think it is a cacti problem. But I'll put up a bug report anyway for this particularly oddity.

[TheWitness]

I agree, but these things happen. The corrective action is why you buy support (aka Donate to the Cacti Group).

TheWitness

[rilindo]

OMG.

Unbelievable.

I think I found the problem.

I was playing around with my own home install of cacti, adding new hosts and adding features. Suddenly, I started to see broken images as well on my system.

I took out / disabled the extra hosts, but the problem still occurs. Then I realize that previously, I enabled most of the web events options in the Event Logging section as well as the Syslog options, as well as set the option to HIGH. So I turned them all off.

Problem went away.

I remembered that all the logging options were also enabled in our work cacti, so I turned those off.

The broken images went away.

I went and re-enabled the logging. The problem came back.

Turn off, the problem went away.

After leaving the logging options all disabled and left the logging level to low, I can verify that the broken images problem no longer exists (for now).

Poking around with my home system, it appears that if you set the poller logging level to MEDIUM or above, the images start coming up broken. Set it to LOW, and the problem goes away. This is successfully reproducible even if you have enabled all the logging options.

So in short:

NONE -> LOW - Graphs came up fine.
MEDIUM and ABOVE - Broken images.

Latency introduced by the logging, perhaps?

Of course, it would just be me being up late at night - even though I was able to reproduce the problem on both work and home servers, it could just be the sleep talking, so I'll rest up. That said, if somebody with a production boost were to duplicate this, that would be great. Just do the following:

- Go to Settings, click on "General" Tab.
- Set Poller Logging Level to "MEDIUM", "HIGH" or above.
- Save

That said, my apologies in advance if this is known issue.

[rilindo]

As for the donation, is it tax-deducible? If not, oh well, donation is sent.

[TheWitness]

The issue is fixed in the SVN. It will be released in boost 3.2.

TheWitness

[rilindo]

Nice. Thanks!

[chadd]

Just wanted to let you know that this problem seems to persist in the current version of Boost/Cacti/PIA (whatever was causing the problem).

I started seeing the problem where the graphs do not show up on first load, you have to do a refresh - saw this post, and changed my logging level from Debugging to High, and the problem went away.

Here are my versions:

Cacti: 0.8.7g
PIA: 2.8
Boost: 4.2

P.S. Thanks for figuring this out

[chadd]

Well, I have the latest cacti (g), spine (g), boost (4.3), and still have the problems found by rilindo. Before I followed his suggestion of turning logging to "low" and disabling the other logging features I was getting this message in my /var/log/apache2/error.log:

PHP Notice: Undefined offset: 8192 in /var/www/cacti/plugins/boost/setup.php on line 556

Now all the graphs that show up, show up on the first load - no "browser refresh" needed. However, there are many graphs that do not show up at all. This was the same as before, and hasn't changed. I am getting only these types of errors in the /var/log/apache2/error.log now:

ERROR: opening '/var/www/cacti/rra/840/99261.rrd': No such file or directory

I checked, and indeed, there is no file by that name, at that location.

Does anyone have a suggestion on fixing this? Thanks,

-chadd.

[chadd]

First. I confirmed that with boost 4.3 the problem of logging level still persists. Here are the details.

At logging level Debug, you get random graphs showing up - and most of the time a page refresh will make the graphs show up. Any logging level below that, the graphs show up fine - on first load.

My previous comments were when I had more issues - which I will take to another thread. They have to do with spine.

Thanks to all for your contributions.

-chadd.

[christiantf]

So in short:
NONE -> LOW - Graphs came up fine.
MEDIUM and ABOVE - Broken images.

Years later, I'd love to say thank you for finding the solution for this issue. I already digged into PHP and Apache config, but it was simply to turn off the DEBUG mode I used for another issue.

Regards, Christian