Looking for Cacti running on virtual machine guidelines

[cacti_is_awesome]

Hi,

I've few questions about running Cacti on virtual server for a large site.

We are using cacti setup v0.8.7.e, running vm RHEL 5.8, 4 CPUs, 16GB RAM, iSCSI storage. We are using Boost plugin as well.

Currently seeing performance issues with "spine" poller. When ran "spine -R", I see it's taking well over 300 seconds, thus timing out, not to mention lots of "SPINE: Poller[0] ERROR: Spine Timed Out While Processing Hosts Internal " messages seen in the cacti.log.

What can be done to improve Spine performance? Are there any best practices / capacity numbers available on forums that I can consider for using cacti running on VM?

Thanks in advance for your help.

Regards
Ray

Below are current stats about my env.
==============================================
Date Thu, 27 Sep 2012 10:30:32 -0700
Cacti Version 0.8.7e
Cacti OS unix
SNMP Version NET-SNMP version: 5.4.1
RRDTool Version RRDTool 1.2.x
Hosts 2566
Graphs 29983
Data Sources Script/Command: 51
SNMP: 34485
SNMP Query: 8358
Script - Script Server (PHP): 6569
Script Query - Script Server: 45
Total: 49508
Poller Information
Interval 300
Type spine
Items Action[0]: 52927
Action[1]: 51
Action[2]: 6639
Total: 59617
Concurrent Processes 7
Max Threads 50
PHP Servers 8
Script Timeout 20
Max OID 55
Last Run Statistics Time:299.0200 Method:spine Processes:7 Threads:50 Hosts:2566 HostsPerProcess:367 DataSources:59617 RRDsProcessed:0
PHP Information
PHP Version 5.2.5
PHP OS Linux
PHP uname Linux 2.6.18-308.11.1.el5 #1 SMP Fri Jun 15 15:41:53 EDT 2012 x86_64
PHP SNMP Not Installed
max_execution_time 30
memory_limit 512M
===================
Boost Stats
Runtime Statistics
Last Start Time: 2012-09-27 10:04:08
Last Run Duration: 2 minutes 26 seconds (8% of update frequency)
RRD Updates: 290533
Peak Poller Memory: 102.55 MBytes
Detailed Runtime Timers: RRDUpdates:290533 TotalTime:146 get_records:9.67 results_cycle:135.57 rrd_filename_and_template:32.51 rrd_lastupdate:18.13 rrdupdate:76.09 delete:0.35 timer_overhead:~2
Max Poller Memory Allowed: 2048 MBytes

[artagel]

What type of detection are using for dead hosts? ping? snmp uptime? I don't think spine uses the same implementation of detection as cmd.php. I've run into this problem myself.
Does it seem like it's getting stuck on down devices? Gandalf suggested using snmp uptime detection only in the past.
-Dan

[gandalf]

What type of detection are using for dead hosts? ping? snmp uptime? I don't think spine uses the same implementation of detection as cmd.php. I've run into this problem myself.
Does it seem like it's getting stuck on down devices? Gandalf suggested using snmp uptime detection only in the past.
-Dan

+1
As spine is coded in plain C, there are some tiny differences with those checks. Most important is the issue requiring root privilege for ICMP with spine.
R.

[cacti_is_awesome]

Thanks Gandalf & Dan,

I'm using SNMP only as the dead host detection.

Have you come across any large environment running cacti on VM?

Regards
Ray

[victorantunes]

gandalf,

does the issue of root privileges also extend to UDP/TCP ping, or is it strict to ICMP?

Is there a link providing information that I could use to analyze other possible downsides of using ping with spine?

thanks in advance.

[gandalf]

Thanks Gandalf & Dan,

I'm using SNMP only as the dead host detection.

Have you come across any large environment running cacti on VM?

Regards
Ray

We are running Cacti with ~100k data sources on a XEN environment. Is this "large"?
R.

[gandalf]

gandalf,

does the issue of root privileges also extend to UDP/TCP ping, or is it strict to ICMP?

Is there a link providing information that I could use to analyze other possible downsides of using ping with spine?

thanks in advance.

Please see ping.c code of spine. I'm not the bast C programmer. So my ability to understand that is limited. Have your own go ...
R.

[cacti_is_awesome]

Thanks Gandalf & Dan,

I'm using SNMP only as the dead host detection.

Have you come across any large environment running cacti on VM?

Regards
Ray

We are running Cacti with ~100k data sources on a XEN environment. Is this "large"?
R.

Gandalf,
~I'll consider ~100K as large env, given that I've ~45K data sources and seeing performance issues at my end. What are the system specifications? Could you please help answer below.
1. OS type/version Cacti is installed on your env.
2. Number of CPUs allocated to cacti vm
3. Memory allocated to vm
4. Cacti version
5. Type of storage iSCSI / NFS
6. Cacti version
7. # of Poller processes / Spine Threads
8. Boost plug-in installed?

Regards
Ray

[gandalf]

Gandalf,
~I'll consider ~100K as large env, given that I've ~45K data sources and seeing performance issues at my end. What are the system specifications? Could you please help answer below.
1. OS type/version Cacti is installed on your env.

RHEL6

2. Number of CPUs allocated to cacti vm

6

3. Memory allocated to vm

not sure; too few; about 4 GB???

4. Cacti version

Cacti 088a

5. Type of storage iSCSI / NFS

IBM SAN storage

6. Cacti version

again, 088a

7. # of Poller processes / Spine Threads

do not remember by heart. about 4/20

8. Boost plug-in installed?

sure, else we would be at a loss

R.

[cacti_is_awesome]

Thanks Gandalf, that helps!.

Regards
Ray