Spine take too long to pool and problems with Boost

General discussion about Plugins for Cacti

Moderators: Developers, Moderators

Post Reply
supportn2
Posts: 23
Joined: Thu Dec 17, 2009 8:24 am

Spine take too long to pool and problems with Boost

Post by supportn2 »

Hello,

First of all, thank you for being indulgent with my English.

I have a cacti server with an update interval of 1 minute. But actually, the cacti polling is between 40 and 60 seconds.

So I need to find a solution to add others devices.

Here my configuration :
Virtual Machine Cacti (4G RAM, 4vCPU)
cacti 0.8.7g
Poller Spine (last version)
Boost (Installed today) (last version)
weathermap (not yet configured) (last version)

Since I installed the boost plugin, I don't see any amelirations (perhaps 5 seconds maximum). Here my stats logs :

Code: Select all

08/23/2011 03:05:42 PM - SYSTEM STATS: Time:40.7419 Method:spine Processes:8 Threads:50 Hosts:108 HostsPerProcess:14 DataSources:9570 RRDsProcessed:0  
08/23/2011 03:04:40 PM - WEATHERMAP: Poller[0] STATS: Weathermap 0.97a run complete - Tue, 23 Aug 2011 15:04:40 CEST: 0 maps were run in 0 seconds with 0 warnings.  
08/23/2011 03:04:40 PM - SYSTEM STATS: Time:38.4890 Method:spine Processes:8 Threads:50 Hosts:108 HostsPerProcess:14 DataSources:9570 RRDsProcessed:0  
08/23/2011 03:03:39 PM - WEATHERMAP: Poller[0] STATS: Weathermap 0.97a run complete - Tue, 23 Aug 2011 15:03:39 CEST: 0 maps were run in 0 seconds with 0 warnings.  
08/23/2011 03:03:39 PM - SYSTEM STATS: Time:38.3446 Method:spine Processes:8 Threads:50 Hosts:108 HostsPerProcess:14 DataSources:9570 RRDsProcessed:0  
08/23/2011 03:02:39 PM - WEATHERMAP: Poller[0] STATS: Weathermap 0.97a run complete - Tue, 23 Aug 2011 15:02:39 CEST: 0 maps were run in 0 seconds with 0 warnings.  
08/23/2011 03:02:39 PM - SYSTEM STATS: Time:37.6755 Method:spine Processes:8 Threads:50 Hosts:108 HostsPerProcess:14 DataSources:9570 RRDsProcessed:0  
08/23/2011 03:01:38 PM - WEATHERMAP: Poller[0] STATS: Weathermap 0.97a run complete - Tue, 23 Aug 2011 15:01:38 CEST: 0 maps were run in 0 seconds with 0 warnings.  
08/23/2011 03:01:38 PM - SYSTEM STATS: Time:36.6374 Method:spine Processes:8 Threads:50 Hosts:108 HostsPerProcess:14 DataSources:9570 RRDsProcessed:0  
08/23/2011 03:00:52 PM - WEATHERMAP: Poller[0] STATS: Weathermap 0.97a run complete - Tue, 23 Aug 2011 15:00:52 CEST: 0 maps were run in 0 seconds with 0 warnings.  
08/23/2011 03:00:52 PM - SYSTEM STATS: Time:50.8189 Method:spine Processes:8 Threads:50 Hosts:108 HostsPerProcess:14 DataSources:9570 RRDsProcessed:0  
08/23/2011 03:00:16 PM - SYSTEM BOOST STATS: Time:99.3100 RRDUpdates:296573  
08/23/2011 02:59:42 PM - WEATHERMAP: Poller[0] STATS: Weathermap 0.97a run complete - Tue, 23 Aug 2011 14:59:42 CEST: 0 maps were run in 0 seconds with 0 warnings.  
08/23/2011 02:59:42 PM - SYSTEM STATS: Time:40.5223 Method:spine Processes:8 Threads:50 Hosts:108 HostsPerProcess:14 DataSources:9570 RRDsProcessed:0  
08/23/2011 02:58:37 PM - WEATHERMAP: Poller[0] STATS: Weathermap 0.97a run complete - Tue, 23 Aug 2011 14:58:37 CEST: 0 maps were run in 0 seconds with 0 warnings.
And it doesn't work correctly, because my graph are not correctly updated :
Image

Here my poller config :
Image

and my boost config
Image

and my boost statistics :
Image

My questions :
1 - Do you have any ideas to reduce the polling duration ? I want to reduce to 20-25 secondes (less than 30) the polling.
2 - Do you know why the boost plugin doesn't work correctly ?
3 - Is-it difficult to have a second cacti poller on an other VM, but with one GUI ?

Thank you !
Top
supportn2
Posts: 23
Joined: Thu Dec 17, 2009 8:24 am

Re: Spine take too long to pool and problems with Boost

Post by supportn2 »

supportn2 wrote: 2 - Do you know why the boost plugin doesn't work correctly ?
I think that is better now... There is less holes now...
Top
supportn2
Posts: 23
Joined: Thu Dec 17, 2009 8:24 am

Re: Spine take too long to pool and problems with Boost

Post by supportn2 »

In fact, I have many holes.

Here you can find a part of the logs :

Code: Select all

08/23/2011 05:33:31 PM - PHPSVR: Poller[0] WARNING: Script Server terminated with signal '15' in file:'ss_host_disk.php', function:'ss_host_disk', params:'FWLO93AUBENAPFAI.out.pri.ena..xxx.fr 45 2:161:3000:1:50:pccad:::::: get total 6'  
08/23/2011 05:33:31 PM - BOOST: Poller[0] WARNING: RRD Update Warning 'ERROR: /opt/cacti-0.8.7g/rra/opcdnre1_hdd_used_8855.rrd: expected timestamp not found in data source from 1314111612' for Local Data ID '8855'  
08/23/2011 05:33:31 PM - BOOST_SERVER: Poller[0] ERROR: /opt/cacti-0.8.7g/rra/opcdnre1_hdd_used_8855.rrd: expected timestamp not found in data source from 1314111612  
08/23/2011 05:33:31 PM - BOOST: Poller[0] WARNING: RRD Update Warning 'ERROR: /opt/cacti-0.8.7g/rra/opcdnre1_hdd_used_8852.rrd: expected timestamp not found in data source from 1314111612' for Local Data ID '8852'  
08/23/2011 05:33:31 PM - BOOST_SERVER: Poller[0] ERROR: /opt/cacti-0.8.7g/rra/opcdnre1_hdd_used_8852.rrd: expected timestamp not found in data source from 1314111612  
08/23/2011 05:33:30 PM - BOOST: Poller[0] WARNING: Stale Poller Data Found! Item Time:'1314111907', RRD Time:'1314111907' Ignoring Value!  
08/23/2011 05:33:30 PM - BOOST: Poller[0] WARNING: Stale Poller Data Found! Item Time:'1314111907', RRD Time:'1314111907' Ignoring Value!  
08/23/2011 05:33:30 PM - BOOST: Poller[0] WARNING: Stale Poller Data Found! Item Time:'1314111845', RRD Time:'1314111907' Ignoring Value!  
08/23/2011 05:33:30 PM - BOOST: Poller[0] WARNING: Stale Poller Data Found! Item Time:'1314111787', RRD Time:'1314111907' Ignoring Value!  
08/23/2011 05:33:30 PM - BOOST: Poller[0] WARNING: Stale Poller Data Found! Item Time:'1314111787', RRD Time:'1314111907' Ignoring Value!  
08/23/2011 05:33:30 PM - BOOST: Poller[0] WARNING: Stale Poller Data Found! Item Time:'1314111731', RRD Time:'1314111907' Ignoring Value!  
08/23/2011 05:33:30 PM - BOOST: Poller[0] WARNING: Stale Poller Data Found! Item Time:'1314111731', RRD Time:'1314111907' Ignoring Value!  
08/23/2011 05:33:30 PM - BOOST: Poller[0] WARNING: Stale Poller Data Found! Item Time:'1314111672', RRD Time:'1314111907' Ignoring Value!  
08/23/2011 05:33:30 PM - BOOST: Poller[0] WARNING: Stale Poller Data Found! Item Time:'1314111612', RRD Time:'1314111907' Ignoring Value!  
08/23/2011 05:33:30 PM - BOOST: Poller[0] WARNING: Stale Poller Data Found! Item Time:'1314111612', RRD Time:'1314111907' Ignoring Value!  
08/23/2011 05:33:30 PM - BOOST: Poller[0] WARNING: Stale Poller Data Found! Item Time:'1314111907', RRD Time:'1314111907' Ignoring Value!  
08/23/2011 05:33:30 PM - BOOST: Poller[0] WARNING: Stale Poller Data Found! Item Time:'1314111845', RRD Time:'1314111907' Ignoring Value!  
08/23/2011 05:33:30 PM - BOOST: Poller[0] WARNING: Stale Poller Data Found! Item Time:'1314111845', RRD Time:'1314111907' Ignoring Value!
Top
supportn2
Posts: 23
Joined: Thu Dec 17, 2009 8:24 am

Re: Spine take too long to pool and problems with Boost

Post by supportn2 »

No one ?
Top
noname
Cacti Guru User
Posts: 1566
Joined: Thu Aug 05, 2010 2:04 am
Location: Japan

Re: Spine take too long to pool and problems with Boost

Post by noname »

I'm not sure about your issue in detail, but..
Maximum Concurrent Poller Processes = 8
Maximum Threads per Process = 50
Number of PHP Script Servers = 5
total concurrent processes = 8 * (50 + 5) = 440

I think it's overworked.
Probably your MySQL can't accept many such connections. Try to reduce them.

See also: MySQL :: Server System Variables - max_connections
Top
supportn2
Posts: 23
Joined: Thu Dec 17, 2009 8:24 am

Re: Spine take too long to pool and problems with Boost

Post by supportn2 »

Thanks for the answer.

I reduce the Thread numbers to 10 per process... I will see if its better after some polling.

But have you an idea to reduce the poller duration ?

Thanks
Top
supportn2
Posts: 23
Joined: Thu Dec 17, 2009 8:24 am

Re: Spine take too long to pool and problems with Boost

Post by supportn2 »

Hi,

I think I have the holes only when I watch the graph. An dWhen I have Holes, I have this type of error (many) on logs :

Code: Select all

08/24/2011 01:07:38 PM - WEATHERMAP: Poller[0] Weathermap 0.97a starting - Normal logging mode. Turn on DEBUG in Cacti for more information  
08/24/2011 01:07:38 PM - SYSTEM STATS: Time:36.8803 Method:spine Processes:8 Threads:10 Hosts:108 HostsPerProcess:14 DataSources:9590 RRDsProcessed:0  
08/24/2011 01:07:09 PM - BOOST SERVER: Poller[0] WARNING: Broken connection detected '127.0.0.1'  
08/24/2011 01:07:09 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:"SHOW TABLES LIKE '%boost%'"  
08/24/2011 01:07:08 PM - BOOST: Poller[0] WARNING: RRD Update Warning 'FAILED' for Local Data ID '8911'  
08/24/2011 01:07:08 PM - BOOST: Poller[0] ERROR: Timeout detected. Boost server is down. Contact support immediately!!  
08/24/2011 01:07:08 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:"SELECT * FROM poller_output_boost_processes"  
08/24/2011 01:07:08 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:"SELECT * FROM poller_output_boost_processes"  
08/24/2011 01:07:08 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:"SELECT * FROM poller_output_boost_processes"  
08/24/2011 01:07:08 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:"SELECT * FROM poller_output_boost_processes"  
08/24/2011 01:07:08 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:"SELECT * FROM poller_output_boost_processes"  
08/24/2011 01:07:08 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:"SELECT * FROM poller_output_boost_processes"  
08/24/2011 01:07:08 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:"SELECT * FROM poller_output_boost_processes"  
08/24/2011 01:07:08 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:"SELECT * FROM poller_output_boost_processes"  
08/24/2011 01:07:08 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:"SELECT * FROM poller_output_boost_processes"  
08/24/2011 01:07:07 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:"SELECT * FROM poller_output_boost_processes"  
08/24/2011 01:07:07 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:"SELECT * FROM poller_output_boost_processes"  
08/24/2011 01:07:07 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:"SELECT * FROM poller_output_boost_processes"  
08/24/2011 01:07:07 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:"SELECT * FROM poller_output_boost_processes"  
08/24/2011 01:07:07 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:"SELECT * FROM poller_output_boost_processes"  
08/24/2011 01:07:07 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:"SELECT * FROM poller_output_boost_processes"  
08/24/2011 01:07:07 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:"SELECT * FROM poller_output_boost_processes"  
08/24/2011 01:07:07 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:"SELECT * FROM poller_output_boost_processes"  
08/24/2011 01:07:07 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:"SELECT * FROM poller_output_boost_processes"  
08/24/2011 01:07:07 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:"SHOW TABLES LIKE '%boost%'"  
08/24/2011 01:07:07 PM - BOOST SERVER: Poller[0] WARNING: Broken connection detected '127.0.0.1'  
08/24/2011 01:07:07 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:"SHOW TABLES LIKE '%boost%'"  
08/24/2011 01:07:06 PM - BOOST: Poller[0] WARNING: RRD Update Warning 'FAILED' for Local Data ID '8910'  
08/24/2011 01:07:06 PM - BOOST: Poller[0] ERROR: Timeout detected. Boost server is down. Contact support immediately!!  
08/24/2011 01:07:06 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:"SELECT * FROM poller_output_boost_processes"  
08/24/2011 01:07:06 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:"SELECT * FROM poller_output_boost_processes"  
08/24/2011 01:07:06 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:"SELECT * FROM poller_output_boost_processes"  
08/24/2011 01:07:06 PM - BOOST SERVER: Poller[0] WARNING: Broken connection detected '127.0.0.1'  
08/24/2011 01:07:06 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:"SHOW TABLES LIKE '%boost%'"  
08/24/2011 01:07:06 PM - BOOST: Poller[0] WARNING: RRD Update Warning 'FAILED' for Local Data ID '8909'  
08/24/2011 01:07:06 PM - BOOST: Poller[0] ERROR: Timeout detected. Boost server is down. Contact support immediately!!  
08/24/2011 01:07:06 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:"SELECT * FROM poller_output_boost_processes"  
08/24/2011 01:07:06 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:"SELECT * FROM poller_output_boost_processes"  
08/24/2011 01:07:06 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:"SELECT * FROM poller_output_boost_processes"  
08/24/2011 01:07:06 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:"SELECT * FROM poller_output_boost_processes"  
08/24/2011 01:07:06 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:"SELECT * FROM poller_output_boost_processes"  
08/24/2011 01:07:06 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:"SELECT * FROM poller_output_boost_processes"  
08/24/2011 01:07:05 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:"SELECT * FROM poller_output_boost_processes"  
08/24/2011 01:07:05 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'0', SQL:"SELECT * FROM poller_output_boost_processes"  ....
Top
noname
Cacti Guru User
Posts: 1566
Joined: Thu Aug 05, 2010 2:04 am
Location: Japan

Re: Spine take too long to pool and problems with Boost

Post by noname »

Just searching in the forum:
- http://forums.cacti.net/viewtopic.php?p=208600#p208600
Top
supportn2
Posts: 23
Joined: Thu Dec 17, 2009 8:24 am

Re: Spine take too long to pool and problems with Boost

Post by supportn2 »

I've already search on the forum, and when a put the timeout to 20s, I have the same errors and images loading is longer.

But, have you an idea to install cacti poller on multiple server ?

Thanks
Top
noname
Cacti Guru User
Posts: 1566
Joined: Thu Aug 05, 2010 2:04 am
Location: Japan

Re: Spine take too long to pool and problems with Boost

Post by noname »

Sorry, I'm not sure in detail.. and I haven't used boost plugin yet.
Top
supportn2
Posts: 23
Joined: Thu Dec 17, 2009 8:24 am

Re: Spine take too long to pool and problems with Boost

Post by supportn2 »

It isn't a problem...
Top
supportn2
Posts: 23
Joined: Thu Dec 17, 2009 8:24 am

Re: Spine take too long to pool and problems with Boost

Post by supportn2 »

I found why I has 40-50s poller duration.

When I deleted the standard "Host MIB - CPU Utilization" graph from 3 servers, I reduce my polling to 25s.

So now, I can disable my boost plugin.
Top
User avatar
TheWitness
Developer
Posts: 17007
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:
Contact TheWitness

Re: Spine take too long to pool and problems with Boost

Post by TheWitness »

Boost 5.0 corrects the connection broken with the boost server. HMIB corrects the stupid disk response issues. Nothing corrects running Cacti and storing RRDfiles on a VM other than lot's of alcohol.

TheWitness
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages

For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Top
Post Reply

Return to “Plugin General”

Who is online

Users browsing this forum: No registered users and 0 guests