Cacti 0.8.8c using 2 Pollers ??? Spine and CMDPHP? :oops:

[Cactus84]

Hi All,

I currently use Cacti for 1 Minute Polling using spine as the poller.


I have been troubleshooting for hours trying to figure out why my poll times are are always at a static 58 seconds??


Then I found this in the logs.

Code: Select all

tail -f /var/www/cacti/log/cacti.log

01/13/2015 04:43:28 PM - CMDPHP: Poller[0] 23350616
01/13/2015 04:43:28 PM - CMDPHP: Poller[0] 11742172
01/13/2015 04:43:28 PM - CMDPHP: Poller[0] 18100592
01/13/2015 04:43:28 PM - CMDPHP: Poller[0] 6738812
01/13/2015 04:43:28 PM - CMDPHP: Poller[0] 76189
01/13/2015 04:43:28 PM - CMDPHP: Poller[0] 1795768
01/13/2015 04:43:28 PM - CMDPHP: Poller[0] 6115656
01/13/2015 04:43:28 PM - CMDPHP: Poller[0] 68735
01/13/2015 04:43:28 PM - CMDPHP: Poller[0] 8098429.1333
01/13/2015 04:43:28 PM - CMDPHP: Poller[0] 8552280
01/13/2015 04:43:28 PM - CMDPHP: Poller[0] 76189
01/13/2015 04:43:28 PM - CMDPHP: Poller[0] 4194295808
01/13/2015 04:43:28 PM - CMDPHP: Poller[0] 16609353728
01/13/2015 04:43:28 PM - CMDPHP: Poller[0] 103512064
01/13/2015 04:43:29 PM - CMDPHP: Poller[0] 103512064

For some reason Cacti is still using the old CMDPHP Poller and Not Spine??

Code: Select all

crontab -l -u apache

Code: Select all

#1 Minute Polling

* * * * *  /usr/bin/php /var/www/cacti/poller.php > /dev/null 2>&1

I have checked the root crontab and there is nothing.


I have attached my settings, is there anything obvious? Or perhaps a script is calling the old poller, thoughts and suggestions?

Thanks

[phalek]

Your polling is running over the 60 seconds and is then automatically stopped. You should see some "gaps" in your graphs then.

There should also be a warning message that the "poller" is "overrruning" the polling time.

And no, your Cacti is indeed using spine. The CMDPHP entries are some static text messages not actually reflecting the poller engine being used.

[Cactus84]

Thanks for your reply, Phalek.

I am getting these errors:

Code: Select all

SPINE: Poller[0] ERROR: Spine Timed Out While Processing Hosts Internal 

From memory this is an error when it cannot complete the full poll within the polling period.

Also there are no Gaps in the Graphs.

I did have some Hosts that were putting out ridiciulous times e.g: 30 seconds to complete scripts etc. But I trimmed the fat. I culled the Hosts from Cacti that were lagging badly so I guess I am still figuring out what's going on.

I have Installed Boost recently, but this made no difference to the Poll time, so then I removed it.

I am not sure exactly the cause hence why I posted my settings.

Thanks

[Cactus84]

Sigh getting these errors now:

01/15/2015 01:22:44 AM - SPINE: Poller[0] Host[271] ERROR: The NIFTY POPEN timed out
01/15/2015 01:22:34 AM - SPINE: Poller[0] Host[358] ERROR: The NIFTY POPEN timed out
01/15/2015 01:22:26 AM - SPINE: Poller[0] Host[312] ERROR: The NIFTY POPEN timed out
01/15/2015 01:22:26 AM - SPINE: Poller[0] Host[272] ERROR: The NIFTY POPEN timed out
01/15/2015 01:22:24 AM - SPINE: Poller[0] Host[271] ERROR: The NIFTY POPEN timed out
01/15/2015 01:22:24 AM - SPINE: Poller[0] Host[442] ERROR: The NIFTY POPEN timed out
01/15/2015 01:22:24 AM - SPINE: Poller[0] Host[358] ERROR: The NIFTY POPEN timed out

Might explain why my polling exceeds 1 minute, however I am shit out of luck trying to solve it.....

[Cactus84]

sigh....

Code: Select all

01/15/2015 01:27:01 AM - SYSTEM STATS: Time:58.8463 Method:spine Processes:3 Threads:15 Hosts:201 HostsPerProcess:67 DataSources:58413 RRDsP
rocessed:11802
01/15/2015 01:27:01 AM - POLLER: Poller[0] NOTE: Poller Int: '60', Cron Int: '60', Time Since Last: '59', Max Runtime '58', Poller Runs: '1'
01/15/2015 01:27:01 AM - POLLER: Poller[0] WARNING: There are '2' detected as overrunning a polling process, please investigate

[phalek]

Maybe this blog posting will help:

https://alexzeng.wordpress.com/2013/01/ ... bug-cacti/

[Cactus84]

Not sure what to make of the output....

Code: Select all

[root@cacti01 debug]# ./check_cacti.sh 30
Check poller command longer than 30 seconds at next 3 minutes...
Start time : 36:55
Sleep 29 seconds
Sleep 7 seconds
Time : 37:31
apache   44463 44461  0 01:37 ?        00:00:00 /bin/sh -c /usr/bin/php /var/www/cacti/poller.php > /dev/null 2>&1
apache   44464 44463 41 01:37 ?        00:00:12 /usr/bin/php /var/www/cacti/poller.php
apache   44473 44471  1 01:37 ?        00:00:00 /usr/bin/php -q /var/www/cacti/script_server.php spine 0
apache   44475 44471  1 01:37 ?        00:00:00 /usr/bin/php -q /var/www/cacti/script_server.php spine 0
apache   44480 44477  0 01:37 ?        00:00:00 /usr/bin/php -q /var/www/cacti/script_server.php spine 0
apache   44481 44471  1 01:37 ?        00:00:00 /usr/bin/php -q /var/www/cacti/script_server.php spine 0
apache   44487 44477  0 01:37 ?        00:00:00 /usr/bin/php -q /var/www/cacti/script_server.php spine 0
apache   44490 44471  1 01:37 ?        00:00:00 /usr/bin/php -q /var/www/cacti/script_server.php spine 0
apache   44510 44477  0 01:37 ?        00:00:00 /usr/bin/php -q /var/www/cacti/script_server.php spine 0
apache   44517 44471  1 01:37 ?        00:00:00 /usr/bin/php -q /var/www/cacti/script_server.php spine 0
apache   44520 44477  0 01:37 ?        00:00:00 /usr/bin/php -q /var/www/cacti/script_server.php spine 0
apache   44537 44477  0 01:37 ?        00:00:00 /usr/bin/php -q /var/www/cacti/script_server.php spine 0
Time : 37:34
apache   44463 44461  0 01:37 ?        00:00:00 /bin/sh -c /usr/bin/php /var/www/cacti/poller.php > /dev/null 2>&1
apache   44464 44463 43 01:37 ?        00:00:14 /usr/bin/php /var/www/cacti/poller.php
apache   44473 44471  1 01:37 ?        00:00:00 /usr/bin/php -q /var/www/cacti/script_server.php spine 0
apache   44475 44471  1 01:37 ?        00:00:00 /usr/bin/php -q /var/www/cacti/script_server.php spine 0
apache   44480 44477  0 01:37 ?        00:00:00 /usr/bin/php -q /var/www/cacti/script_server.php spine 0
apache   44481 44471  1 01:37 ?        00:00:00 /usr/bin/php -q /var/www/cacti/script_server.php spine 0
apache   44487 44477  0 01:37 ?        00:00:00 /usr/bin/php -q /var/www/cacti/script_server.php spine 0
apache   44490 44471  1 01:37 ?        00:00:00 /usr/bin/php -q /var/www/cacti/script_server.php spine 0
apache   44510 44477  0 01:37 ?        00:00:00 /usr/bin/php -q /var/www/cacti/script_server.php spine 0
apache   44517 44471  1 01:37 ?        00:00:00 /usr/bin/php -q /var/www/cacti/script_server.php spine 0
apache   44520 44477  0 01:37 ?        00:00:00 /usr/bin/php -q /var/www/cacti/script_server.php spine 0
apache   44537 44477  0 01:37 ?        00:00:00 /usr/bin/php -q /var/www/cacti/script_server.php spine 0

[Cactus84]

I am using 1 minute polling and this is in the code:

perhaps change this to 60 seconds?

Code: Select all

vim /var/www/cacti/script_server.php

  define("MAX_POLLER_RUNTIME", 300);

[Cactus84]

Nope made no difference....

[cigamit]

I would go back into Settings and change
"Script and Script Server Timeout" to 5 or 10.
If you have 1 script that isn't responding, and it is waiting on it for 180 seconds, you are definitely going to go over your polling time of 58 seconds.

[Cactus84]

Thanks for your replies.

I changed

Code: Select all

Script and Script Server Timeout Value to 5

This didn't make a difference, in the chance of bring polling down have people gone to the effort of finding Max OID's per device and going through every device in Cacti and changing this?

Here is some more information on the system:

Code: Select all

[root@cacti01 ~]# cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 45
model name      : Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
stepping        : 7
microcode       : 0x70d
cpu MHz         : 2700.000
cache size      : 20480 KB
physical id     : 0
siblings        : 4
core id         : 0
cpu cores       : 4
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt aes xsave avx hypervisor lahf_lm ida arat epb xsaveopt pln pts dtherm
bogomips        : 5400.00
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 45
model name      : Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
stepping        : 7
microcode       : 0x70d
cpu MHz         : 2700.000
cache size      : 20480 KB
physical id     : 0
siblings        : 4
core id         : 1
cpu cores       : 4
apicid          : 1
initial apicid  : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt aes xsave avx hypervisor lahf_lm ida arat epb xsaveopt pln pts dtherm
bogomips        : 5400.00
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 2
vendor_id       : GenuineIntel
cpu family      : 6
model           : 45
model name      : Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
stepping        : 7
microcode       : 0x70d
cpu MHz         : 2700.000
cache size      : 20480 KB
physical id     : 0
siblings        : 4
core id         : 2
cpu cores       : 4
apicid          : 2
initial apicid  : 2
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt aes xsave avx hypervisor lahf_lm ida arat epb xsaveopt pln pts dtherm
bogomips        : 5400.00
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 3
vendor_id       : GenuineIntel
cpu family      : 6
model           : 45
model name      : Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
stepping        : 7
microcode       : 0x70d
cpu MHz         : 2700.000
cache size      : 20480 KB
physical id     : 0
siblings        : 4
core id         : 3
cpu cores       : 4
apicid          : 3
initial apicid  : 3
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt aes xsave avx hypervisor lahf_lm ida arat epb xsaveopt pln pts dtherm
bogomips        : 5400.00
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

Code: Select all

[root@cacti01 ~]# free -m
             total       used       free     shared    buffers     cached
Mem:          7817       6875        942          5          0       4967
-/+ buffers/cache:       1907       5909
Swap:         1639          4       1635

disk I/O does not look to be a bottleneck, and I have XFS mount with: noatime,nodirtime,nobarriers.

I have been playing with new combinations.... with OIDS,Threads/Processes

seriously cannot get it down below 58 seconds

01/15/2015 08:07:00 AM - SYSTEM STATS: Time:58.4150 Method:spine Processes:3 Threads:15 Hosts:201 HostsPerProcess:67 DataSources:58413 RRDsProcessed:11802

any more ideas?

Thanks

[phalek]

Yes,

Enable debug logging and then check for the following entries:

Code: Select all

01/15/2015 05:50:24 AM - SPINE: Poller[0] Host[254] TH[1] Total Time: 15 Seconds
01/15/2015 05:50:19 AM - SPINE: Poller[0] Host[213] TH[1] Total Time: 13 Seconds
01/15/2015 05:50:17 AM - SPINE: Poller[0] Host[253] TH[1] Total Time: 8 Seconds
01/15/2015 05:50:14 AM - SPINE: Poller[0] Host[265] TH[1] Total Time: 2.3 Seconds
01/15/2015 05:50:14 AM - SPINE: Poller[0] Host[255] TH[1] Total Time: 4.4 Seconds
01/15/2015 05:50:13 AM - SPINE: Poller[0] Host[232] TH[1] Total Time: 6.5 Seconds

These tell you how long the polling for a specific host did actually take.

[Cactus84]

Phalek, thanks for your help I am slowly going through each host and trying to trim the fat to bring the polling time down.

Just a quick question, are we sure spine is polling every host?

Below it says 112 Hosts using the spine poller.

Code: Select all

SPINE: Poller[0] Time: 39.1741 s, Threads: 15, Hosts: 112

When the total hosts is: 196.

Code: Select all

01/17/2015 06:14:27 PM - SYSTEM STATS: Time:25.6755 Method:spine Processes:2 Threads:15 Hosts:196 HostsPerProcess:98 DataSources:58136 RRDs Processed:11776

[phalek]

You're using 2 processes, so there should be two entriess like these which add up to the total host numbers from the SYSTEM STATS entry.

[Cactus84]

Thanks for your help after fishing through the hosts I have made progress. Cheers.