Boost problem - gaps when viewing graphs

[Jacket]

Hi,
I have a little problem with the boost plugin.
The problem is that when i open a graph, and stay on it's page, the graph stops being updated. Something like the attached graph. All the gaps there are from when someone had stayed on the page...
As you can see the graphs aren't being updated even after major boost updates...
I tried to flush the tmp directory, but no luck...
This is my last poller stats:

Code: Select all

02/27/2009 09:33:29 PM - SYSTEM STATS: Time:27.9893 Method:spine Processes:4 Threads:10 Hosts:1466 HostsPerProcess:367 DataSources:84348 RRDsProcessed:0

I have no boost overruns...
Image caching and Enable On Demand RRD Updating are on.

The specs of the machine are:
CPU: Core2Duo E8600
RAM: 8G
HDD: 2xWDC WD1500HLFS @10k RPM in RAID
mysql max heap size is 4g - maximum 1g is used for now.
Memory Limit for Boost and Poller is 2g - peak is 43mb.

[TheWitness]

"permissions". Apache MUST have RW to files.

1) Find the apache user:

ps -ef | grep httpd

2) Change the ownership on the rra directory:

chown -R cactiuser:<httpd_user> /var/www/html/cacti/rra

3) Set sticky bit on rra directory:

chmod +s /var/www/html/cacti/rra

TheWitness

[Jacket]

10x but unfortunately that didn't help I've allready set the owner of rra/ dir to www-data (apache user on debian) at cacti install (by the book). I haven't set the sticky bit, tried it now, but still the problem exists - if someone opens a graph and stays on the page, the graph stops being updated...
Here is my rra/ dir:

Code: Select all

drwsr-sr-x 1473 www-data www-data  32K 2009-02-27 18:18 rra

Is it possible that this have something to do with this:

you can change the index with following commands (based on boost 2.4 install script):

Code: Select all

ALTER TABLE poller_output_boost DROP PRIMARY KEY;
ALTER TABLE poller_output_boost ADD PRIMARY KEY USING BTREE (`local_data_id`,`rrd_name`,`time`);

I'm not exactly sure if it will make things better or worse. But you may give it a try.

Also i checked the apache logs, and there's nothing there about permissions...

[TheWitness]

Looks at your group permissions, I don't see write there...

TheWitness

[TheWitness]

The database optimization has nothing to do with your problems. I bet you are using SUSE right? What happens if you set the following:

Code: Select all

chmod -R 777 /var/www/html/cacti/rra
chmod +s /var/www/html/cacti/rra

TheWitness

[Jacket]

Set group permissions also - no change... after 2-3 minutes graphing stops...
Now tried chmodding to 777 and i'm wainting on one graph to see Allthough i don't like the idea of 777 but just for the test it's ok. I'll post the result ASAP.

Oh, and this is on Debian. Personally i'm a fan of Slackware most, but that machine was installed by one friend, and i'm in no mood of reinstalling it...

[TheWitness]

I would do a test:

1) Create a graph
2) Let poller_boost create the rrdfile
3) Check the permissions on that file
4) su to the apache user and attempt to touch the file.

TheWitness

[Jacket]

Did that:

Code: Select all

cactiregions:/usr/local/cacti/rra/4699# ls -lha
total 152K
drwxr-sr-x    2 www-data www-data 4.0K 2009-02-28 00:03 .
drwsrwsrwx 1474 www-data www-data  32K 2009-02-28 00:03 ..
-rw-r--r--    1 www-data www-data 109K 2009-02-28 00:03 158123.rrd
cactiregions:/usr/local/cacti/rra/4699# su www-data -c "touch 158123.rrd"
cactiregions:/usr/local/cacti/rra/4699#

Still one graph is open and is gaping...
Sometimes between refreshes some data appears, like on this graph. This is Hourly (1 Minute Average) graph. If i stay on that page for 30 minutes, it will be a huge gap on the Daily (5 Minute Average) graph. If i refresh in maximum 60 sec interval, no data appears at all, even on the Hourly graph... sorry if i'm not explaining this verry good.
It seems to me like on every request to view a graph, the data from the last poll is lost or something like that...

[TheWitness]

Ah, structured paths. I'm going to post a patch to boost in just a minute. This is a nasty little problem that's been introduced and neglected by me.

TheWitness

[TheWitness]

Ok, looked at structured paths in the boost code and the directory permissions should have been 775. So, your directory permissions are wrong and match the "OLD" setting for the rra using the sticky bit.

TheWitness

[Jacket]

It's 775 now, and still gaping, but strangely i just found one graph that is ok - it doesn't make gaps now, but have two older gaps from today. Verry strange...
So you found the problem to be with the structured paths? Sorry i forgot to mention that... Should i wait for a patch, or test something else?
Thanks for the help btw.

[TheWitness]

What I'd does the cacti poller run as? Make sure that user has rw permissions.

[Jacket]

poller runs as www-data, which as you can see has write permissions...
I found another graph. I think someone forgot to close it's browser, the host is 100% online, so i'm sure this isn't from the host itlesf.

[Jacket]

I tried the following (in the same order):
1. Stopped the poller cronjob - commented the line.
2. Waited for the last poller to complete.
3. Forced boost to update - php poller_boost.php -f.
4. Rebuild cacti's poller cache - php rebuild_poller_cache.php
5. Stopped mysql
6. Stopped apache
7. Started apache
8. Started mysql
9. Started the poller cronjob - uncommented the line.

At first all seemed fine for about 5 minutes, and then the problem started again
Can i do some kind of debugging of on-demand updating. I made a debug log for boost (last option in configuration), but it's not writing anything there - permissions are ok...

[Jacket]

Hmm i just found something strange..:

Code: Select all

02/28/2009 01:10:21 PM - SYSTEM STATS: Time:19.9386 Method:spine Processes:4 Threads:10 Hosts:1467 HostsPerProcess:367 DataSources:57326 RRDsProcessed:0

And this one si from the Data Sources page:

Code: Select all

Showing Rows 1 to 100 of 29939

Is this difference normal? If not, how can i fix this? I can recreate all my graphs, no problem, i use scripts to do that.
In fact, here's the script i use for 95% of my hosts:

Code: Select all

$conn = mysql_connect($dbhost, $dbuser, $dbpass) or die ('Error connecting to mysql');
mysql_select_db($dbname);
$result = mysql_query("SELECT * FROM host WHERE host_template_id='9' ORDER BY id") or die(mysql_error());
while ($row = mysql_fetch_assoc($result)) {
// Adds the data query "interface statistics"
system("php add_data_query.php --host-id=$row[id] --data-query-id=1 --reindex-method=1");
	for ($i = 1; $i <= 9; $i++) {
// Adds traffic graphs
system("php add_graphs.php --graph-type=ds --graph-template-id=2 --host-id=$row[id] --snmp-query-id=1 --snmp-query-type-id=13 --snmp-field=ifName --snmp-value=g$i");
// Adds unicast graphs
system("php add_graphs.php --graph-type=ds --graph-template-id=23 --host-id=$row[id] --snmp-query-id=1 --snmp-query-type-id=4 --snmp-field=ifName --snmp-value=g$i");
// Adds non-unicast graphs
system("php add_graphs.php --graph-type=ds --graph-template-id=24 --host-id=$row[id] --snmp-query-id=1 --snmp-query-type-id=3 --snmp-field=ifName --snmp-value=g$i");
	}
}