Boost Plugin v1.6 Released

cacti_owner · Post by **cacti_owner** » Mon Jun 04, 2007 7:37 am

Hello,
what might be a problem if "poller_boost.php -f -d" simply does nothing?
Only statistics page get's updated with information about "Last Start Time".

It displays nothing and it isn't proccessing any rrds, while everything else works fine (rrd's get updated when graphs are opened, info is being fed to poller_output_boost table and image cache directory is growing).

The problem is poller_output_boost has grown to much an and without mass rrd update sooner or later it will lead to crash and loss of data.

cacti 0.8.6j, Boost 1.2
mysq database type:MyISAM
Current Boost Status
Boost On Demand Updating: Running
Total Data Sources: 25078
Total Boost Records: 25831013
Table Size Estimates
Current Boost Table Size: 946 MBytes
Avg Bytes/Record: 36 Bytes
Max Allowed Boost Table Size: 262144 GBytes
Estimated Maximum Records: 7.81874935307E+12 Records
Previous Runtime
Last Start Time: 2007-06-04 15:15:03
Last Run Duration: Seconds
RRD's Updated:
Run Time Configuration
Update Frequency: 10 Minutes
Next Start Time:
Maximum Records: 1000000 Records
Maximum Allowed Runtime: 5 Minutes
Boost Server Details
Server Config Status: Disabled
Multiprocess Server: Multiple Process
Update Timeout: 2 Seconds
Server/Port: localhost@9050
Authorized Update Web Servers: 127.0.0.1
RRDtool Binary Used: /usr/bin/rrdupdate
Image Caching
Image Cacing Status: Enabled
Cache Directory: /opt/tinklas/PNG_cache/
Cached Files: 6234 Files
Cached Files Size: 216 MBytes

thank you for the response

Post by **TheWitness** » Mon Jun 04, 2007 7:45 am

I would venture to guess that you need to increase your php memory in php.ini. Go way high. This was working before right?

TheWitness

gninja · Post by **gninja** » Tue Jun 05, 2007 12:40 pm

Was working yesterday, (though I hadn't gotten the boost_server running) and today, my log hit 2gb (causing everything to fail).. After collecting some choice bits from the log, and then truncating it, I see that my graphs were only getting updated about once an hour, up until the 2gb point was hit. And there was a lot of:

06/05/2007 01:26:30 PM - CMDPHP: Poller[0] ERROR: SQL Exec Failed "INSERT INTO poller_output_boost (local_data_id, rrd_name, time, output) VALUES ('13794','','2007-06-05 04:50:20','file_reads: file_writes: file_fsyncs: insert_buffer_inserts: insert_buffer_merged: insert_buffer_merges: log_writes: buffer_pool_size: buffer_free_pages: buffer_database_pages: buffer_modified_pages: buffer_hit_rate: queries_inside: queries_queue: rows_inserted: rows_updated: rows_deleted: rows_read: spin_waits:0 spin_rounds:0 os_waits:0'), ('13792','','2007-06-05 04:50:20','file_reads: file_writes: file_fsyncs: insert_buffer_inserts: insert_buffer_merged: insert_buffer_merges: log_writes: buffer_pool_size: buffer_free_pages: buffer_database_pages: buffer_modified_pages: buffer_hit_rate: queries_inside: queries_queue: rows_inserted: rows_updated: rows_deleted: rows_read: spin_waits:0 spin_rounds:0 os_waits:0'), ('13791','','2007-06-05 04:50:20','file_reads: file_writes: file_fsyncs: insert_buffer_inserts: insert_buffer_merged: insert_buffer_merges: log_writes: buffer_pool_size: buffer_free_pages: buffer_database_pages: buffer_modified_pages: buffer_hit_rate: queries_inside: queries_queue: rows_inserted: rows_updated: rows_deleted: rows_read: spin_waits:0 spin_rounds:0 os_waits:0'), ('13790','','2007-06-05 04:50:20','file_reads: file_writes: file_fsyncs: insert_buffer_inserts: insert_buffer_merged: insert_buffer_merges: log_writes: buffer_pool_size: buffer_free_pages: buffer_database_pages: buffer_modified_pages: buffer_hit_rate: queries_inside: queries_queue: rows_inserted: rows_updated: rows_deleted: rows_read: spin_waits:0 spin_rounds:0 os_waits:0') ON DUPLICATE KEY UPDATE output=VALUES(output)"

Still digging into the problem, just thought I'd do a current brain dump in case anyone has a quick and easy answer.

gninja · Post by **gninja** » Tue Jun 05, 2007 1:00 pm

Hrm. My maximum allowable records was larger than what my table could support, and I generated enough records to fill my table, in less than my update interval. Didn't see a nice way to fix it, since I'd been getting occasional updates to the rrd files(?) so I truncated the poller_output_boost table, and set my settings to more sane values.

I was also seeing `poller_boost.php -f -d` output nothing, exiting with a status code of 255. So, if you're having similar problems, and a forced update does nothing, you might have the same problem.

gninja · Post by **gninja** » Tue Jun 05, 2007 2:38 pm

Nope. Still not working.

Last Start Time: 2007-06-05 19:33:04
Last Run Duration: 0.0357 Seconds
RRD's Updated: 0

Post by **TheWitness** » Tue Jun 05, 2007 9:42 pm

What about after the changes we made today?

Larry

cacti_owner · Post by **cacti_owner** » Wed Jun 06, 2007 2:12 am

I would venture to guess that you need to increase your php memory in php.ini. Go way high. This was working before right?

you was right...during the time when poller_boost.php was failing the poller_output_boost database became realy hudge.

to make it work i had to modify php.ini a little:

max_execution_time = 7200
max_input_time = 7200
memory_limi = 1638M

During the process i have noticed some coding lacks.

1. Depending on configuration, script was fetching 500.000 rows and processing approx. 2500 DS per hour and then failing as it's maximum 1-hour execution time was getting elapsed. So, you should consider increasing this setting up to "no limit".

2. Processed DS records were not removed from the database. Maybe it had to happen at the end of poller_boost in some form of truncate (and this step was not reached because of 1st problem), but I think it should be done in some more flexible way.

3. Processed DS count was dropping exponentially (after each hourly restart: 2500 2300 2000 .... 600...400 ). It might be realated to previous 2 problems, but it also may indicate that there are some places in the code where some variables (like arrays ) should be nulled.

4. During the process machine load never went over 25% (quad-proc Xeon 3.2GHz, 4Gigs RAM). The cacti's native poller.php is capable of doing 100%, so, boost code seems to be lacking of parllel processing.

Post by **TheWitness** » Wed Jun 06, 2007 9:01 pm

I think you need to check you IO wait during that time. RRDupdates are slowed by disk activity. Therefore, multiple processes don't often help.

TheWitness

leonardo_gyn · Post by **leonardo_gyn** » Thu Jun 21, 2007 5:03 pm

Got the plugin running in some minutes, specially for testing the Image Caching feature.

Well .... it works just fine !!

Anyway, i got a small problem .... images created on the specified directory are owned by httpd user. OK, no problem, they are correctly created and displayed, cause the directory has the right permissions.

BUT, as they are completly owned by httpd user, polling process is not capable of purging old files. So my directory just grow grow and nothing was deleted.

When I manually adjusted owner/group (chown) and permissions (chmod), purging happened.

Question is ..... is there any way of having this permissions adjustments made automatically when a new file is created ?

I'm running cacti on Unix enviroments. Is there some unix way of inheriting owner/group and permissions from the directory ?? I know Windows is capable of doing something like this, so files created inherit directory permissions. If this was possible on Unix, it would be easy to solve the problem.

Or ..... if some code changing is needed for manually adjusting files right after creating them, i see no problem at all !!

In the WORST case, i see no other way than setting a root cron job for doing that

Post by **TheWitness** » Thu Jun 21, 2007 8:43 pm

I have sent your request to a friend of mine. We'll see what he has to say. He knows these things very well.

TheWitness

sg · Post by sg » Fri Jun 22, 2007 7:47 am

Leonardo - TheWitness asked me to comment on this, as we talked about this very thing a few weeks ago!

Advanced UNIX permissions:

Let's assume we have a directory owned by the web user, with files within:

Code: Select all

# ls -al
total 16
drwxr-xr-x  3 apache apache 4096 Jun 22 08:30 .
drwxrwxrwt  7 root   root   4096 Jun 22 08:41 ..
drwxr-x---  2 apache apache 4096 Jun 22 08:42 mydir
-rw-r-----  1 apache apache    0 Jun 22 08:30 myfile

We want to allow the poller user ("cactiuser") to modify files within "mydir" and they must be accessible by the "apache" user.

Fast way: Add the poller user to the apache group:

Code: Select all

# grep apache /etc/group
apache:x:48:cactiuser

If you don't have permission/ability/etc. to modify this group, you can add "sticky" permission to the directory, and set the group ownership to cactiuser:

Code: Select all

# chmod g+rws mydir
# chown apache:cactiuser mydir
# ls -al
total 16
drwxr-xr-x  3 apache apache    4 Jun 22 08:30 .
drwxrwxrwt  7 root   root      4 Jun 22 08:41 ..
drwxrws---  2 apache cactiuser 4 Jun 22 08:42 mydir
-rw-r-----  1 apache apache    0 Jun 22 08:30 myfile

Now, when the apache user writes in "mydir", the files will be owned by apache:cactiuser:

Code: Select all

# su apache
# cd mydir
# touch newfile
total 8
drwxrws---  2 apache cactiuser 4096 Jun 22 08:45 .
drwxr-xr-x  3 apache apache    4096 Jun 22 08:30 ..
-rw-r--r--  1 apache cactiuser    0 Jun 22 08:45 newfile

All done!

Post by **TheWitness** » Fri Jun 22, 2007 7:59 am

Thanks Simon!

TheWitness

leonardo_gyn · Post by **leonardo_gyn** » Fri Jun 22, 2007 8:16 am

Thanks simon !! I'll try that and post results here, as i think this may be a common problem with cacti/boost users.

leonardo_gyn · Post by **leonardo_gyn** » Fri Jun 22, 2007 12:35 pm

Guys, i think i have an interesting situation here

My boost_image directory permissions are:

Code: Select all

[root@cacti cacti2]# ls -lsa | grep boost_images 
  4 drwxrwxr-x  2 cacti nobody   4096 Jun 22 12:18 boost_images
[root@cacti cacti2]#

cacti is my cacti user and nobody is my webserver user

Files inside boost_images directory are created by cacti web interface with the following permissions:

Code: Select all

-rw-r--r--  1 nobody nobody 49586 Jun 22 11:58 lgi_66_rrai_0_tsi_7.png
-rw-r--r--  1 nobody nobody 50766 Jun 22 11:58 lgi_67_rrai_0_tsi_7.png
-rw-r--r--  1 nobody nobody 36223 Jun 22 11:59 lgi_68_rrai_0_tsi_7.png

altough they are with owner/group set to webserver user, i can successfully erase files there when logged with cacti user:

[cacti@cacti boost_images]$ whoami
cacti
[cacti@cacti boost_images]$ ls -al lgi_39_rrai_0_tsi_7.png
-rw-r--r-- 1 nobody nobody 37688 Jun 22 11:51 lgi_39_rrai_0_tsi_7.png
[cacti@cacti boost_images]$ rm lgi_39_rrai_0_tsi_7.png
rm: remove write-protected regular file `lgi_39_rrai_0_tsi_7.png'? y
[cacti@cacti boost_images]$ ls -al lgi_39_rrai_0_tsi_7.png
ls: lgi_39_rrai_0_tsi_7.png: No such file or directory
[cacti@cacti boost_images]$

BUT ..... poller process, which should erase old files, is not erasing them. I have given enough time for them to became stale (1 hour according to poller_boost.php, i have even changed to 30 minutes). Even tough, stale files are not being erased.

I successfully erased nobody/nobody owned files logged with cacti user even before making changes proposed by Simon. If i can erase them in command line, cron jobs running with the same user have right permissions to erase them as well.

I made some tests and seems the the problem relies on 'is_writable' call. Seems that is_writable evaluate only file permissions without evaluating directory ones as well.

I ran this simple code with cacti user:

Code: Select all

if (is_writable('/home/httpd/html/cacti2/boost_images/lgi_93_rrai_0_tsi_7.png')) {
        echo "then";
} else {
        echo "else";
}

As i imagined, i got an 'else' return, meaning that is_writable returned false. Even tough, i modified the code and put a unlink to that tested file on else block and ...... the file was successfully erased, even with is_writable returning false !!

Based on that, seems that simply setting correct permissions on boost_images directory is enough for dealing with permissions mess because of cacti user and webserver user. BUT, that is_writable check is avoiding files to be correctly erased.

I'd like to propose some changes on that code. I dont know how, but i would like to propose some changes on it

What do you think on that ?

merlinthemagic · Post by **merlinthemagic** » Fri Jun 22, 2007 4:26 pm

Hey all,

Problem is that cacti_rrdsvc needs to be restarted about every 6 hours, if i dont, the on demand generation does not work and all i get is:

06/22/2007 09:56:43 AM - BOOST SERVER: Poller[0] WARNING: Broken connection detected '127.0.0.1'

06/22/2007 09:56:43 AM - BOOST: Poller[0] WARNING: RRD Update Warning 'FAILED' for Local Data ID '4508'
06/22/2007 09:56:43 AM - BOOST: Poller[0] ERROR: Timeout detected. Boost server is down. Contact support immediately!!
06/22/2007 09:56:43 AM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed "SELECT * FROM poller_output_boost_processes"
06/22/2007 09:56:43 AM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed "SELECT * FROM poller_output_boost_processes"
06/22/2007 09:56:43 AM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed "SELECT * FROM poller_output_boost_processes"
06/22/2007 09:56:43 AM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed "SELECT * FROM poller_output_boost_processes"

Tons of those, if i restart the process it shuts down fine and comes backup no problem, and after that the graphs are generated again. Now the poller and the boost updater seems unaffected by the this (still working when on demand update is down) it shows:

06/22/2007 09:43:16 AM - SYSTEM BOOST STATS: Time:114.8146 RRDUpdates:71604
06/22/2007 09:41:21 AM - SYSTEM STATS: Time:79.7330 Method:cactid Processes:10 Threads:10 Hosts:1607 HostsPerProcess:161 DataSources:12174 RRDsProcessed:0

06/22/2007 09:40:01 AM - CMDPHP: Poller[0] WARNING: Rows remaining in poller_output table, truncating

So it is just the on demand update that dies somehow....

Tried to fix it with a cronjob restarting the service every 3 hours (little more often that needed)... but that makes the kernel panick every night.

Boost works great besides from this.

CentOS 4.4, php-5.1.6-3, MySQL 5.0.27

Ideas?

Martin

Cacti

Boost Plugin v1.6 Released

problem

Boost sql failing

No problem!

problem with cacti_rrdsvc

Who is online