How to increase performance of Spine polling time?

[roxy]

I am running Version 0.8.8a and Linux platform. I am running into an issue with high polling time and periodic gaps in my graphs. I have tried adjusting the number of threads and also tried to use multiple processes instead of one. But still there is no improvement on the total polling time. Do you have any suggestion for improving spine's performance - I have lots of devices to monitor in Cacti.

05/04/2016 03:09:26 PM - SYSTEM STATS: Time:264.7665 Method:spine Processes:24 Threads:20 Hosts:2262 HostsPerProcess:95 DataSources:254876 RRDsProcessed:92164
05/04/2016 03:05:27 PM - SYSTEM STATS: Time:324.9506 Method:spine Processes:24 Threads:20 Hosts:2262 HostsPerProcess:95 DataSources:254876 RRDsProcessed:92176
05/04/2016 02:58:41 PM - SYSTEM STATS: Time:218.8851 Method:spine Processes:24 Threads:20 Hosts:2262 HostsPerProcess:95 DataSources:254876 RRDsProcessed:92194
05/04/2016 02:53:43 PM - SYSTEM STATS: Time:221.7222 Method:spine Processes:24 Threads:20 Hosts:2262 HostsPerProcess:95 DataSources:254876 RRDsProcessed:92183
05/04/2016 02:48:38 PM - SYSTEM STATS: Time:216.6530 Method:spine Processes:24 Threads:20 Hosts:2262 HostsPerProcess:95 DataSources:254876 RRDsProcessed:92133
05/04/2016 02:43:41 PM - SYSTEM STATS: Time:219.5430 Method:spine Processes:24 Threads:20 Hosts:2262 HostsPerProcess:95 DataSources:254876 RRDsProcessed:92177
05/04/2016 02:39:56 PM - SYSTEM STATS: Time:294.3475 Method:spine Processes:24 Threads:20 Hosts:2262 HostsPerProcess:95 DataSources:254876 RRDsProcessed:92147
05/04/2016 02:33:47 PM - SYSTEM STATS: Time:225.3324 Method:spine Processes:24 Threads:20 Hosts:2262 HostsPerProcess:95 DataSources:254876 RRDsProcessed:92177

Cacti poller is scheduled every 5 mins. Please suggest how I can improve performance of spine and can reduce poller time so that I can add more devices in the system.

I am using a good configuration box here. 24 CPU, 6 Cores each, 64GB memory.

[phalek]

You are probably hitting I/O limits.

Do you have SSD disks in place or "plain old harddisks" ?

You could look into boost to offload the rrd-updates.

You can also check if the script mentioned here helps you with the updating part: http://forums.cacti.net/viewtopic.php?f ... ol+wrapper

Check "top" for your io waits ...

[mikygee]

Hello,

I tend to have my rrd on a specific partition and mount that partition with the option noatime. This helps when files are read. In your case you might consider using a specific disk for your rrd files.
I don't use SSD drives because rrd tend to take space and ssd drives are more expensive.

Have you tried to increase the "The Maximum SNMP OID's Per SNMP Get Request" ?

You should also check if some devices are very slow to answer.
Maybe your cacti is not properly placed in your network ? If it's far from most devices then you'll have an extra overhead.

I monitor my I/O disk on my system.

Maybe it's not the disks but something else like the network card which receives a lot of little packets and interrupts the cpu all the time

[roxy]

I have already tried with higher "The Maximum SNMP OID's Per SNMP Get Request" but there is no improvement. RRD files are in local disk,

/dev/sda1 8063408 5442464 2211344 72% /
tmpfs 32982080 3205832 29776248 10% /dev/shm
/dev/sda6 2015824 1202856 710568 63% /home
/dev/sda5 4031680 2136088 1690792 56% /tmp
/dev/sda7 99150 5650 88380 7% /usr/app
/dev/sda2 8063408 5843136 1810672 77% /var

My Cacti is installed in /var/www/cacti and this is production system.

I have another system (dev/test) exactly similar setup with less number of devices (server with 2 CPU, 6 core each and 8 GB memory). It was taking around 190 - 210 Sec to complete poller and I installed boost to offload RRD update but I have not seen much improvement there as well.

05/05/2016 11:48:41 AM - SYSTEM STATS: Time:219.7454 Method:spine Processes:2 Threads:100 Hosts:1397 HostsPerProcess:699 DataSources:36567 RRDsProcessed:0
05/05/2016 11:43:26 AM - SYSTEM STATS: Time:204.4324 Method:spine Processes:2 Threads:100 Hosts:1397 HostsPerProcess:699 DataSources:36567 RRDsProcessed:0
05/05/2016 11:38:21 AM - SYSTEM STATS: Time:199.9224 Method:spine Processes:2 Threads:100 Hosts:1397 HostsPerProcess:699 DataSources:36567 RRDsProcessed:0
05/05/2016 11:33:28 AM - SYSTEM STATS: Time:206.3242 Method:spine Processes:2 Threads:100 Hosts:1397 HostsPerProcess:699 DataSources:36567 RRDsProcessed:0
05/05/2016 11:28:39 AM - SYSTEM STATS: Time:217.2716 Method:spine Processes:2 Threads:100 Hosts:1397 HostsPerProcess:699 DataSources:36567 RRDsProcessed:0
05/05/2016 11:23:17 AM - SYSTEM STATS: Time:194.6898 Method:spine Processes:2 Threads:100 Hosts:1397 HostsPerProcess:699 DataSources:36567 RRDsProcessed:0
05/05/2016 11:18:18 AM - SYSTEM STATS: Time:197.0485 Method:spine Processes:2 Threads:100 Hosts:1397 HostsPerProcess:699 DataSources:36567 RRDsProcessed:0

Boost plugin status:

Boost On Demand Updating: Idle
Total Data Sources: 36567
Pending Boost Records: 29302
Archived Boost Records: 0
Total Boost Records: 29302
Boost Storage Statistics
Database Engine: MEMORY
Current Boost Tables Size: 3 MBytes
Avg Bytes/Record: 91 Bytes
Max Record Length: 31 Bytes
Max Allowed Boost Table Size: 9 MBytes
Estimated Maximum Records: 99460 Records
Runtime Statistics
Last Start Time: 2016-05-05 11:48:42
Last Run Duration: 19 seconds (1% of update frequency)
RRD Updates: 32789
Peak Poller Memory: 14.04 MBytes
Detailed Runtime Timers: RRDUpdates:32789 TotalTime:20 get_records:0.92 results_cycle:18.63 rrd_filename_and_template:11.16 rrd_lastupdate:1.51 rrdupdate:1.56 delete:0.08 timer_overhead:~2
Max Poller Memory Allowed: 512 MBytes
Run Time Configuration
Update Frequency: 30 Minutes
Next Start Time: 2016-05-05 12:18:42
Maximum Records: 30000 Records
Maximum Allowed Runtime: 20 Minutes
Boost Server Details
Server Config Status: Disabled
Multiprocess Server: Multiple Process
Update Timeout: 2 Seconds
Server/Port: localhost@9050
Authorized Update Web Servers: 127.0.0.1
RRDtool Binary Used: /usr/bin/rrdtool

Cacti is in local disk here as well.

/dev/sda1 20641788 10712324 8880824 55% /
tmpfs 4029856 0 4029856 0% /dev/shm
/dev/sda6 2063504 905368 1053316 47% /home
/dev/sda7 2063504 42304 1916380 3% /tmp
/dev/sda8 106031216 12809828 87835304 13% /tools
/dev/sda5 2063504 438944 1519740 23% /opt
/dev/sda2 6190692 2228884 3647336 38% /var

Before doing Boost installation on prod, I want to check improvement on this test environment first.

I have few big Weathermaps configured on Cacti. Is there a possibility that WM put extra load on poller?
(Cacti Version: 0.8.8a, Plugin Architecture Version: 3.1, Weathermap 0.97b)

[phalek]

If boost doesn't give you much then it may really be network related. You can look into offloading the actual device polling to something like the multi pollen plugin

[roxy]

Thanks Phalek for your inputs.

I will try this plugin on test environment but Boost is already installed there so do I need to uninstall this first or Boost and Multi poller both can work together?

Any idea??

[phalek]

Try without boost first. I didn't test this combination yet.

[roxy]

Hi Phalek,

I want to use Cacti Multipoller server 0.2.2b as it is compatible with my Cacti version as mentioned some forum. But I cannot see any download link to download this version.
Can you please guide me where I can download this version?

Can it be possible to configure multiple poller in a single server so that I can assign each poller to a class of network devices? This will help to complete polling time within 300 sec?

Is it possible to install Cacti multi-poller on Cacti 0.8.8a or I need to upgrade Cacti to 0.8.8d first to install this plugin?

[phalek]

Go here for a list of all downloads: http://www.cacti-multipollerserver.de/i ... l?Itemid=0

[roxy]

I have recently built a test environment with 3 VM to implement Cacti Multi Poller installation so that I can see the performance. I faced lots of issue with poller time in my current system (Cacti single poller) as number of devices large and have heavy weathermaps.

I had below system with single poller (all in local system).
Cacti Version: 0.8.8a, Plugin Architecture Version: 3.1, MySQL 5.1.73

Now I have used clone of above in a new VM and then upgrade it to Cacti Version: 0.8.8d and installed Cacti Multi Poller v0.8.8d and configured all the VMs. Now I can see poller is working properly from all VMs and graphs are populated with data.

But I am facing an issue with graph tree. Graph tree view is broken after Cacti upgrade and installation of Multi-poller plugin. I am getting blank page when I am clicking on graph tree view.

Do you have any idea how can I fix this issue or how can I troubleshoot this issue?