[solved] Poller hits 300 seconds

[markruys]

Hi,

I can't get the poller run below 300 seconds, so I hope someone here has a suggestion for me.

Log:

Code: Select all

12/12/2014 03:49:59 PM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
12/12/2014 03:49:59 PM - SYSTEM STATS: Time:298.8840 Method:spine Processes:2 Threads:10 Hosts:44 HostsPerProcess:22 DataSources:1924 RRDsProcessed:0
12/12/2014 03:55:00 PM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
12/12/2014 03:55:00 PM - SYSTEM STATS: Time:298.8659 Method:spine Processes:3 Threads:10 Hosts:44 HostsPerProcess:15 DataSources:1924 RRDsProcessed:0
(END)

Top:

Code: Select all

top - 15:56:19 up 2 days,  3:42,  2 users,  load average: 0.82, 0.99, 0.79
Tasks: 115 total,   2 running, 113 sleeping,   0 stopped,   0 zombie
%Cpu0  : 11.2 us, 16.5 sy,  0.0 ni, 71.5 id,  0.0 wa,  0.0 hi,  0.0 si,  0.8 st
%Cpu1  :  2.9 us,  4.0 sy,  0.4 ni, 92.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.4 st
KiB Mem:    993048 total,   844020 used,   149028 free,    46264 buffers
KiB Swap:  2097148 total,    43864 used,  2053284 free.   487224 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                    
28676 www-data  20   0  325840  21620  10460 R  41.6  2.2   0:46.82 php                                                                                                        
12413 mysql     20   0 1342064  18636   3864 S   9.6  1.9 278:05.49 mysqld     

Code: Select all

$ ps waux | grep php
www-data   630  0.0  0.0   4444   652 ?        Ss   16:00   0:00 /bin/sh -c php --define suhosin.memory_limit=512M /usr/share/cacti/site/poller.php 2>&1 >/dev/null | if [ -f /usr/bin/ts ] ; then ts ; else tee ; fi >> /var/log/cacti/poller-error.log
www-data   631 43.3  2.1 325840 21624 ?        S    16:00   0:22 php --define suhosin.memory_limit=512M /usr/share/cacti/site/poller.php
www-data   632  0.0  0.0   4444   100 ?        S    16:00   0:00 /bin/sh -c php --define suhosin.memory_limit=512M /usr/share/cacti/site/poller.php 2>&1 >/dev/null | if [ -f /usr/bin/ts ] ; then ts ; else tee ; fi >> /var/log/cacti/poller-error.log

Poller Type: spine
Maximum Concurrent Poller Processes: 3
Maximum Threads per Process: 10
Number of PHP Script Servers: 3
The Maximum SNMP OID's Per SNMP Get Request: 25

Ubuntu 14.04, Cacti 0.8.8b+dfsg-5 =, Spine 0.8.8b-1

Load is low, IO is low, CPU idle, I see no spine in my process list, only a single PHP running.

[BSOD2600]

post the info from the first 3 sections in System Utilities > Technical Support.

some suggestions:
removing inefficient scripts which get invoked for each call/data source and replace with ones which are Data Queries (pure snmp) or PHP Script server. These scale much better.
optimize mysql and php configuration
configuring Boost plugin
separating web server/poller from MySQL.
Put *.rrd files on SSD or fast equivalent.

[cigamit]

According to the number of datasources, you aren't really monitoring a ton of stuff. It appears that the poller isn't exitting because something is getting stuck. Please post the above requested information. You should probably change your cacti log to MEDIUM and start digging through the results too, to see what is getting stuck.

[markruys]

Thanks for the advice.

I had the MySQL credentials wrong. This made PHP hang on 100% CPU. As soon as I fixed it, the runs took about 50 seconds.

I switched back from Spine to plain Cmd as the load of Spine was rather high (>10) and the speed in our case not significant better. Probably I could change this by playing with certain settings, but I'm happy with Cmd already. What surprised me a bit though was that under Spine, disk IO was 400 kB/s, with Cmd 600 kB/s (writes). Settings like logging level similar, I can't explain the difference.

Well, Cacti is collecting data again