[FIXED] Polling exceeds 1 minute

Tibo · Post by **Tibo** » Tue Sep 15, 2009 6:01 am

Hello everybody! As I am kind of new in Cacti and I request your help!

My problem is the following: my Cacti is configured to do polling every 1 minute but it takes now too long and for each polls, it's around 55-57 seconds so I am stuck because I cannot add new devices in my Cacti monitoring now although I don't have so many devices.

09/15/2009 11:06:54 AM - SYSTEM STATS: Time:54.9548 Method:spine Processes:1 Threads:6 Hosts:53 HostsPerProcess:53 DataSources:910 RRDsProcessed:553
09/15/2009 11:05:54 AM - SYSTEM STATS: Time:54.9374 Method:spine Processes:1 Threads:6 Hosts:53 HostsPerProcess:53 DataSources:910 RRDsProcessed:553
09/15/2009 11:04:56 AM - SYSTEM STATS: Time:56.6375 Method:spine Processes:1 Threads:6 Hosts:53 HostsPerProcess:53 DataSources:910 RRDsProcessed:553

Is it normal that it takes so long (I have only 57 hosts and 603 datasources). Could it be due to my RDD configuration as I set it to do a 1 minute consolidation for the 2 last months and 5 minutes consolidation for one year (without average)?

Name Steps Rows Timespan**
2 months - 1 min 1 89280 5356800
1 year - 5 min 5 115200 33053184

Any recommendations or advices to tune my settings? Let me know if you need more information to help me on this issue. FYI, it's a prod server so I cannot play too much with the settings in order to avoid as much as I can any data loss.

My poller settings in attachment.

And my server info:
Operating System: Windows 2003 server
Webserver: Apache 2.2.11 (win32)
Cacti: 0.8.7b
Spine: 0.8.7a
MySQL: 2.11.4
PHP: 5.2.5
RRDTool: 1.2.26
Net-SNMP: 5.4.2.1

mcutting · Post by **mcutting** » Tue Sep 15, 2009 6:20 am

Try increasing the number of processes, using amount of cpu cores x 2

engeishi · Post by **engeishi** » Tue Sep 15, 2009 6:33 am

And also try increasing Maximum Threads per Process to 10-15.
This document may help you.
http://docs.cacti.net/manual:087:3a_adv ... pine#spine

Tibo · Post by **Tibo** » Tue Sep 15, 2009 12:07 pm

I proceeded to some tests and I don't see any improvements at all. The time for one polling remains the same.

original setting: 1 process for 6 threads

tested with
2 for 6
1 for 10
2 for 15
4 for 15
4 for 10

Tibo · Post by **Tibo** » Wed Sep 16, 2009 4:59 am

Hello again. I did a lot of tests this morning changing the poller settings and it's not faster. In the best cases, I have the same time but in some cases I lost 2 or 3 seconds (roughly).

I don't really understand why Spine cannot be faster as my environment is not very huge: 53 hosts with 910 datasources and 553 RRDs processed.

For your information, it's a good server based on a Intel Xeon 3.4 Ghz (4 CPU) with 4 GB of RAM and the OS is a Windows Server 2003 (standard edition with SP2).

engeishi · Post by **engeishi** » Fri Sep 18, 2009 1:10 am

Isn't there possibility that several of your device has the problem to return data?
As you say, I also think your environment is not too large for Spine.

To pinpoints the cause, tern cacti.log to DEBUG level for one polling cycle,
and confirm how much time it takes to collect information on each device.

Tibo · Post by **Tibo** » Fri Sep 18, 2009 3:25 am

Is it normal that there is around 15 sec between the Spine poller time and the system stats time?

09/18/2009 09:13:57 AM - SYSTEM STATS: Time:57.6000 Method:spine Processes:1 Threads:6 Hosts:53 HostsPerProcess:53 DataSources:912 RRDsProcessed:554

09/18/2009 09:13:41 AM - SPINE: Poller[0] Time: 41.3430 s, Threads: 6, Hosts: 53

mcutting · Post by **mcutting** » Fri Sep 18, 2009 9:06 am

Tibo wrote:Hello again. I did a lot of tests this morning changing the poller settings and it's not faster. In the best cases, I have the same time but in some cases I lost 2 or 3 seconds (roughly).

I don't really understand why Spine cannot be faster as my environment is not very huge: 53 hosts with 910 datasources and 553 RRDs processed.

For your information, it's a good server based on a Intel Xeon 3.4 Ghz (4 CPU) with 4 GB of RAM and the OS is a Windows Server 2003 (standard edition with SP2).

You really should look at what you are polling. SPINE excels at SNMP queries, but scripts such as PERL and WMI will slow it down - particularly if you are querying a server in the Far East, when your Cacti box sits in the US (for example).

Have a look at your queries to see if you can get rid of what you don't really need. Another tool to look at is "pollperf" from Gandalf (search the forums) - this will tell you the runtime of each datasource and host during polling.

mcutting · Post by **mcutting** » Fri Sep 18, 2009 9:08 am

Tibo wrote:Is it normal that there is around 15 sec between the Spine poller time and the system stats time?

09/18/2009 09:13:57 AM - SYSTEM STATS: Time:57.6000 Method:spine Processes:1 Threads:6 Hosts:53 HostsPerProcess:53 DataSources:912 RRDsProcessed:554

09/18/2009 09:13:41 AM - SPINE: Poller[0] Time: 41.3430 s, Threads: 6, Hosts: 53

Depends on what plugins are running after the poller has finished collecting data. One way to check would be to disable all plugins, and let the poller have the limelight - the time you get with no plugins is the time it takes for your poller to run.

Tibo · Post by **Tibo** » Fri Sep 18, 2009 10:14 am

Concerning plugins, I have only Monitor, Thold and Weathermap but I will try to disable them and check how long it takes afterwards.

Tibo · Post by **Tibo** » Fri Sep 18, 2009 10:28 am

Oki so I disabled them and nothing changed at all... I am puzzled now...

Post by **TheWitness** » Fri Sep 18, 2009 1:14 pm

Set the log level to MEDIUM for one pass, and post your Cacti Log.

TheWitness

Tibo · Post by **Tibo** » Mon Sep 21, 2009 10:43 am

In attachment the log for one pass in medium level (I just remove the IP for security reason

).

Let me know if you find something interesting.

Surprisingly, there is a gap of 15 sec between the poller time and the system time for one pass. I tried to disable all my plugin as I reported before but it didn't change anything.

Post by **TheWitness** » Tue Sep 22, 2009 6:05 am

Your issue might be I/O wait. Your spine finishes at 37 seconds, but the poller does not finish until just under the bell. So, there is quite a bit of processing delay.

Here is your plan of attack:

Step 0

Increase Spine Threads to 20. However, prior to doing that, increase "max_connections" to 200 in my.cnf and restart mysql.

Step 1

Make this change

Code: Select all

mysql cacti
alter table poller_output modify column output varchar(60) not null default '', engine=memory;
quit;

Step 2

Send output to the following:

Code: Select all

mysql
show global variables
show global status

Step 3

Send output to the following:

Code: Select all

du -hs /var/www/html/cacti/rra

Step 4

Send output to the following:

Code: Select all

df -h /var/www/html/cacti/rra
df -h

Step 5

Send output to the following:
System Memory, System Cores, #Physical Volumnes, Is a VM?

TheWitness

Tibo · Post by **Tibo** » Thu Sep 24, 2009 5:00 am

Well, I installed from scratch with all lastest version of Cacti, Spine (and so on) on another server and with almost the same number of devices and datasources, my polling takes now 8.5 sec with the current polling settings...

So definitively, there is something weird on my first server. I will rebuilt it from scratch like my second one and it will fix the issue for sure

Thanks for your help!

[FIXED] Polling exceeds 1 minute

[FIXED] Polling exceeds 1 minute

Who is online