[Solved] Graphs are not getting plotted for some hosts

[Eti]

Hi All,

I am quite new for CACTI and I have already added around 300 Alcatel,ZTE and CISCO routers and switches on our CACTI platform to monitor BW utilizations of router ports. Around 90% of the hosts are performing well on the platform without any issues. But for some hosts (both Alcatel and ZTE), graphs are not getting plotted continuously. Additionally, when I create a graph for these hosts, it take considerable time to generate RRD files in the rra directory. But for other hosts it is very fast. Please help me to solve this problem.
I herewith attached a sample graph and its debug information for your reference.

Capture 1.JPG (31.45 KiB) Viewed 7261 times

Capture 2.JPG (58.46 KiB) Viewed 7261 times

[camerabob]

Try changing the timeout on those hosts to see if maybe they just respond slower than the others. You could also increase the number of threads for those hosts but you should never use more threads than there are cores on the server. Most hosts never need more than two threads from what I have seen. Also check the logs to see if your poll completes in a decent time frame.

[Eti]

Hi camerabob,

First of all, thank you very much for your response and advises.

As per your instructions I did below changes and results are as below.

01. changing the timeout on those hosts : Changed this from 500 ms to 1000 ms but output is same as earlier
02. increase the number of threads for those hosts : Increased no. of thread from 1 to 2 but output is same as earlier
03. check the logs to see if your poll completes in a decent time frame : I have no idea about the log. I herewith attached latest log. if you can please help me to find the issue.

Thanks

[Eti]

Hi camerabob,

In addition to below result, I tried to update rrd file with rrdtool update command and file is getting updated. But specific rrd file is not getting updated with crone while all other files are getting updated with same crone. Thanks

[camerabob]

05/18/2018 12:04:02 PM - POLLER: Poller[0] WARNING: There are '1' detected as overrunning a polling process, please investigate
05/18/2018 12:04:05 PM - PHPSVR: Poller[0] Maximum runtime of 60 seconds exceeded for the Script Server. Exiting.
05/18/2018 12:04:05 PM - SPINE: Poller[0] ERROR: SS[0] PHP Script Server communications lost. Restarting PHP Script Server
05/18/2018 12:04:05 PM - SPINE: Poller[0] ERROR: Spine Timed Out While Processing Hosts Internal
05/18/2018 12:04:05 PM - SPINE: Poller[0] ERROR: Spine Timed Out While Processing Hosts Internal
05/18/2018 12:04:05 PM - SPINE: Poller[0] ERROR: SS[0] PHP Script Server communications lost. Restarting PHP Script Server

These lines require further analysis from someone that can decipher them. There should be no errors of this type. You need to figure out if the time overrun is the host in question that's failing your graph and figure out why it is taking so long to get a reply back to the Cacti server.

Additionally, I see you have 1 minute polling enabled. That one host may be taking more than a minute to reply and the response is getting clobbered by the next polling cycle.

05/18/2018 12:04:00 PM - SYSTEM STATS: Time:57.9935 Method:spine Processes:1 Threads:1 Hosts:351 HostsPerProcess:351 DataSources:5508 RRDsProcessed:2645

As you can see, your polls are being terminated at about 58-59 seconds to start the next run.

[cigamit]

You failed to mention what version of Cacti you are running.

Processes:1 Threads:1

You need to greatly increase this first. Changed Processes to equal the amount of CPU Cores in your Cacti server, and then change Threads to ~12. You will find these under Settings.

Only after getting your Poller to not overrun can you then figure out the other issues.

[Eti]

Hi Cigamit,

Thank You for your response. Please find my answers below.
01. version of Cacti : I am running CactiEz V0.7 (http://cactiez.cactiusers.org/) and version details are as below.
====================================================================================
Cacti Version 0.8.8a
Cacti OS unix
SNMP Version NET-SNMP version: 5.5
RRDTool Version RRDTool 1.3.x
Hosts 350
Graphs 2827
Data Sources SNMP: 42
SNMP Query: 2654
Script - Script Server (PHP): 146
Script Query - Script Server: 8
Total: 2850
Poller Information
Interval 60
Type SPINE 0.8.8 Copyright 2002-2012 by The Cacti Group
Items Action[0]: 5341
Action[2]: 167
Total: 5508
Concurrent Processes 1
Max Threads 1
PHP Servers 1
Script Timeout 25
Max OID 10
Last Run Statistics Time:58.2565 Method:spine Processes:1 Threads:1 Hosts:351 HostsPerProcess:351 DataSources:5508 RRDsProcessed:2645
PHP Information
PHP Version 5.3.3
PHP OS Linux
PHP uname Linux localhost.localdomain 2.6.32-279.9.1.el6.x86_64 #1 SMP Tue Sep 25 21:43:11 UTC 2012 x86_64
PHP SNMP Installed
max_execution_time 30
memory_limit 128M
====================================================================================

02. CPU Cores in your Cacti server : I am running Cacti on a virtual server and CPU information of my server as below.
====================================================================================
[root@localhost ~]# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 1
On-line CPU(s) list: 0
Thread(s) per core: 1
Core(s) per socket: 1
CPU socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 37
Stepping: 1
CPU MHz: 3000.000
BogoMIPS: 6000.00
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 25600K
NUMA node0 CPU(s): 0
================================

I hope this information will help you to isolate my issue.

Thanks

[Eti]

Hi camerabob,

Thanks for your response and I got your points.
Here is the ping test result from Cacti server to one of the problematic host.
What is the recommended value for polling interval for my case.
Thanks
======================================================================
[root@localhost ~]# ping 10.52.1.126
PING 10.52.1.126 (10.52.1.126) 56(84) bytes of data.
64 bytes from 10.52.1.126: icmp_seq=1 ttl=62 time=2.36 ms
64 bytes from 10.52.1.126: icmp_seq=2 ttl=62 time=6.16 ms
64 bytes from 10.52.1.126: icmp_seq=3 ttl=62 time=3.43 ms
64 bytes from 10.52.1.126: icmp_seq=4 ttl=62 time=2.37 ms
64 bytes from 10.52.1.126: icmp_seq=5 ttl=62 time=2.81 ms
64 bytes from 10.52.1.126: icmp_seq=6 ttl=62 time=2.28 ms
64 bytes from 10.52.1.126: icmp_seq=7 ttl=62 time=26.7 ms
64 bytes from 10.52.1.126: icmp_seq=8 ttl=62 time=2.40 ms
64 bytes from 10.52.1.126: icmp_seq=9 ttl=62 time=2.27 ms
64 bytes from 10.52.1.126: icmp_seq=10 ttl=62 time=2.51 ms
64 bytes from 10.52.1.126: icmp_seq=11 ttl=62 time=2.31 ms
64 bytes from 10.52.1.126: icmp_seq=12 ttl=62 time=2.38 ms
64 bytes from 10.52.1.126: icmp_seq=13 ttl=62 time=8.78 ms
64 bytes from 10.52.1.126: icmp_seq=14 ttl=62 time=14.2 ms
64 bytes from 10.52.1.126: icmp_seq=15 ttl=62 time=20.7 ms
64 bytes from 10.52.1.126: icmp_seq=16 ttl=62 time=43.9 ms
64 bytes from 10.52.1.126: icmp_seq=17 ttl=62 time=2.21 ms
64 bytes from 10.52.1.126: icmp_seq=18 ttl=62 time=2.26 ms
64 bytes from 10.52.1.126: icmp_seq=19 ttl=62 time=2.23 ms
64 bytes from 10.52.1.126: icmp_seq=20 ttl=62 time=2.18 ms
64 bytes from 10.52.1.126: icmp_seq=21 ttl=62 time=2.24 ms
64 bytes from 10.52.1.126: icmp_seq=22 ttl=62 time=2.25 ms
64 bytes from 10.52.1.126: icmp_seq=23 ttl=62 time=2.47 ms
64 bytes from 10.52.1.126: icmp_seq=24 ttl=62 time=2.93 ms
64 bytes from 10.52.1.126: icmp_seq=25 ttl=62 time=3.08 ms
64 bytes from 10.52.1.126: icmp_seq=26 ttl=62 time=6.44 ms
64 bytes from 10.52.1.126: icmp_seq=27 ttl=62 time=22.1 ms
64 bytes from 10.52.1.126: icmp_seq=28 ttl=62 time=2.36 ms
64 bytes from 10.52.1.126: icmp_seq=29 ttl=62 time=2.26 ms
64 bytes from 10.52.1.126: icmp_seq=30 ttl=62 time=2.37 ms
64 bytes from 10.52.1.126: icmp_seq=31 ttl=62 time=2.41 ms
64 bytes from 10.52.1.126: icmp_seq=32 ttl=62 time=2.29 ms
64 bytes from 10.52.1.126: icmp_seq=33 ttl=62 time=2.14 ms
64 bytes from 10.52.1.126: icmp_seq=34 ttl=62 time=2.19 ms
64 bytes from 10.52.1.126: icmp_seq=35 ttl=62 time=2.48 ms
64 bytes from 10.52.1.126: icmp_seq=36 ttl=62 time=2.42 ms
64 bytes from 10.52.1.126: icmp_seq=37 ttl=62 time=2.25 ms

--- 10.52.1.126 ping statistics ---
37 packets transmitted, 37 received, 0% packet loss, time 36480ms
rtt min/avg/max/mdev = 2.143/5.932/43.916/8.696 ms
=========================================================================

[Eti]

In addition to graph plotting issue, these problematic hosts taking significant time (in some cases more than 24hours) to generate rrd file when i create a graph for these hosts.

[cigamit]

Go into your settings and up the number of threads to 12 and PHP Script Servers to 3 and then after a bit report back what your poller time is.

[Eti]

Hi Cigamit,

Now it seems Ok. Up to now problematic graphs are getting plotted continuously without any issue.
And poller time has increased. please refer below.

05/19/2018 10:47:09 AM - SYSTEM STATS: Time:7.9241 Method:spine Processes:1 Threads:12 Hosts:351 HostsPerProcess:351 DataSources:5512 RRDsProcessed:2762

Anyway I will monitor it for next 24 hours and if i found any issue I will get back.

Thank you very much for your great supports.

[chanuka]

I have the same issue. The graph was not drawn frequently.