Cacti getting slower, ideas?

[LostyJai]

Hi guys,

Recently our Cacti has become rather slow loading graphs. When I click on the graphs tab, it takes probably over 1 minutes to load the tree and maybe another 1 minute to load the graphs. Overall anything takes a while to load.

Cacti is currently running as VM with Ubuntu LTS 10.04, 4GB RAM and 4 vCPUs. The load on the server fluctuates between 3-14 but the CPU usage may spike up to 80% for a few seconds, however it's usually 0%-20%. The MySQL database is also running on the same VM.

I also seem to have noticed gaps in my graphs. Here are my stats for the last 3 hours.

Code: Select all

11/03/2011 11:13:02 PM - SYSTEM STATS: Time:180.1412 Method:spine Processes:2 Threads:50 Hosts:2345 HostsPerProcess:1173 DataSources:28930 RRDsProcessed:18971
11/03/2011 11:08:05 PM - SYSTEM STATS: Time:183.6590 Method:spine Processes:2 Threads:50 Hosts:2345 HostsPerProcess:1173 DataSources:28930 RRDsProcessed:19014
11/03/2011 11:05:35 PM - SYSTEM STATS: Time:331.5603 Method:spine Processes:2 Threads:50 Hosts:2345 HostsPerProcess:1173 DataSources:28930 RRDsProcessed:14766
11/03/2011 10:58:26 PM - SYSTEM STATS: Time:200.9893 Method:spine Processes:2 Threads:50 Hosts:2345 HostsPerProcess:1173 DataSources:28930 RRDsProcessed:19064
11/03/2011 10:52:39 PM - SYSTEM STATS: Time:157.1752 Method:spine Processes:2 Threads:50 Hosts:2345 HostsPerProcess:1173 DataSources:28930 RRDsProcessed:19049
11/03/2011 10:48:23 PM - SYSTEM STATS: Time:143.6660 Method:spine Processes:2 Threads:50 Hosts:2345 HostsPerProcess:1173 DataSources:28930 RRDsProcessed:19062
11/03/2011 10:42:54 PM - SYSTEM STATS: Time:172.2603 Method:spine Processes:2 Threads:50 Hosts:2345 HostsPerProcess:1173 DataSources:28930 RRDsProcessed:19059
11/03/2011 10:37:33 PM - SYSTEM STATS: Time:151.7785 Method:spine Processes:2 Threads:50 Hosts:2345 HostsPerProcess:1173 DataSources:28930 RRDsProcessed:19035
11/03/2011 10:33:31 PM - SYSTEM STATS: Time:206.5621 Method:spine Processes:2 Threads:50 Hosts:2345 HostsPerProcess:1173 DataSources:28930 RRDsProcessed:19051
11/03/2011 10:28:07 PM - SYSTEM STATS: Time:182.9848 Method:spine Processes:2 Threads:50 Hosts:2345 HostsPerProcess:1173 DataSources:28930 RRDsProcessed:19050
11/03/2011 10:22:53 PM - SYSTEM STATS: Time:165.3769 Method:spine Processes:2 Threads:50 Hosts:2345 HostsPerProcess:1173 DataSources:28930 RRDsProcessed:19013
11/03/2011 10:17:41 PM - SYSTEM STATS: Time:158.8491 Method:spine Processes:2 Threads:50 Hosts:2345 HostsPerProcess:1173 DataSources:28930 RRDsProcessed:19071
11/03/2011 10:12:20 PM - SYSTEM STATS: Time:137.0180 Method:spine Processes:2 Threads:50 Hosts:2345 HostsPerProcess:1173 DataSources:28930 RRDsProcessed:19060
11/03/2011 10:07:38 PM - SYSTEM STATS: Time:155.9465 Method:spine Processes:2 Threads:50 Hosts:2345 HostsPerProcess:1173 DataSources:28930 RRDsProcessed:19029
11/03/2011 10:03:30 PM - SYSTEM STATS: Time:205.4486 Method:spine Processes:2 Threads:50 Hosts:2345 HostsPerProcess:1173 DataSources:28930 RRDsProcessed:19026
11/03/2011 09:57:59 PM - SYSTEM STATS: Time:175.9779 Method:spine Processes:2 Threads:50 Hosts:2345 HostsPerProcess:1173 DataSources:28930 RRDsProcessed:19070
11/03/2011 09:52:50 PM - SYSTEM STATS: Time:167.7335 Method:spine Processes:2 Threads:50 Hosts:2345 HostsPerProcess:1173 DataSources:28930 RRDsProcessed:19036
11/03/2011 09:48:29 PM - SYSTEM STATS: Time:148.5687 Method:spine Processes:2 Threads:50 Hosts:2345 HostsPerProcess:1173 DataSources:28930 RRDsProcessed:19026
11/03/2011 09:42:49 PM - SYSTEM STATS: Time:165.7150 Method:spine Processes:2 Threads:50 Hosts:2345 HostsPerProcess:1173 DataSources:28930 RRDsProcessed:19043
11/03/2011 09:37:59 PM - SYSTEM STATS: Time:177.7320 Method:spine Processes:2 Threads:50 Hosts:2345 HostsPerProcess:1173 DataSources:28930 RRDsProcessed:19071
11/03/2011 09:35:37 PM - SYSTEM STATS: Time:333.6340 Method:spine Processes:2 Threads:50 Hosts:2345 HostsPerProcess:1173 DataSources:28930 RRDsProcessed:18996
11/03/2011 09:27:43 PM - SYSTEM STATS: Time:161.5300 Method:spine Processes:2 Threads:50 Hosts:2345 HostsPerProcess:1173 DataSources:28930 RRDsProcessed:19027
11/03/2011 09:22:50 PM - SYSTEM STATS: Time:165.8856 Method:spine Processes:2 Threads:50 Hosts:2345 HostsPerProcess:1173 DataSources:28930 RRDsProcessed:19050
11/03/2011 09:19:39 PM - SYSTEM STATS: Time:277.0752 Method:spine Processes:2 Threads:50 Hosts:2345 HostsPerProcess:1173 DataSources:28930 RRDsProcessed:19040
11/03/2011 09:17:06 PM - SYSTEM STATS: Time:416.2720 Method:spine Processes:2 Threads:50 Hosts:2345 HostsPerProcess:1173 DataSources:28930 RRDsProcessed:17065
11/03/2011 09:14:43 PM - SYSTEM STATS: Time:555.7794 Method:spine Processes:2 Threads:50 Hosts:2345 HostsPerProcess:1173 DataSources:28930 RRDsProcessed:14018
11/03/2011 09:05:33 PM - SYSTEM STATS: Time:331.0218 Method:spine Processes:2 Threads:50 Hosts:2345 HostsPerProcess:1173 DataSources:28930 RRDsProcessed:5106
11/03/2011 08:57:36 PM - SYSTEM STATS: Time:154.5091 Method:spine Processes:2 Threads:50 Hosts:2345 HostsPerProcess:1173 DataSources:28930 RRDsProcessed:19072
11/03/2011 08:53:05 PM - SYSTEM STATS: Time:182.7114 Method:spine Processes:2 Threads:50 Hosts:2345 HostsPerProcess:1173 DataSources:28930 RRDsProcessed:19021
11/03/2011 08:49:08 PM - SYSTEM STATS: Time:147.6247 Method:spine Processes:2 Threads:50 Hosts:2345 HostsPerProcess:1173 DataSources:28930 RRDsProcessed:19060

My poller settings:

Code: Select all

Poller Processes: 2
Maximum Threads per Process: 50
Number of PHP Script Servers: 10

Just thinking out loud, would it be worthwhile to have two cacti servers. One server is purely for polling and the other is used to display graphs on the front end, the RRD files will be shared via NFS (stored on polling server and shared with frontend) and the database will be synced via replication.

Would that work?

[TheWitness]

This is all about disk I/O. Use the boost plugins, currently at version 5.1. But make sure your READ and more importantly COMPREHEND the README prior to turning it on.

[LostyJai]

This is all about disk I/O. Use the boost plugins, currently at version 5.1. But make sure your READ and more importantly COMPREHEND the README prior to turning it on.

Hey thanks for replying.

You say this is all about disk I/O, but when I click on those devices in my graph tree and it sits there "loading..." for a minute or two, I don't see much I/O happening on the server. Neither do I see any rrdtool process, which I assume is what it should be doing when creating those graphs.

I have since cloned the VM running it on our new infrastructure, reduced the vcpu to two, increased the RAM to 8GB and the load is comfortably sitting on 0-1.

What is the best way to tell where the bottleneck is?

[TheWitness]

It's disk I/O. I need not say any more. However, don't trust me, install and enable boost and prove me wrong. I've been doing this Cacti thing for 10 years. So, don't be surprised when I tell you that I told you so.

[LostyJai]

It's disk I/O. I need not say any more. However, don't trust me, install and enable boost and prove me wrong. I've been doing this Cacti thing for 10 years. So, don't be surprised when I tell you that I told you so.

I apologise if I sounded disrespectful. I just simply wanted to understand where the bottleneck was and I'll believe you it's the I/O.

If say my devices will double in the next year or so, would this VM still be able to handle the amount of I/O load? Or would I need to move it onto a physical server (possibly SSD drives)?

[TheWitness]

VM's are not good for Cacti unless on PCIe SSDs.