Ugly performance problems.

jdr · Post by **jdr** » Fri Jan 12, 2007 4:22 pm

I have cacti with cactid on a Sun Fire V440 server with 8gb operating memory and 4 Ultra Sparc 1064MHz processors.

I have really bad performace, i was playing with optimizing mysql, mirror volumes etc, but there is all nothing. Recently i noticed, that when I run poller.php manually, it first gather data (no output), after that is long series of "OK's", than very long nothing happens, and everything ends. I saw, that in that moment betwen end of OK's and end of the process, I have one rrdtool process. The end of execution looks like:

Code: Select all

OK u:0.64 s:2.54 r:119.09
OK u:0.65 s:2.54 r:119.12
OK u:0.65 s:2.54 r:119.14

(the looooooooooooong break)

Code: Select all

01/12/2007 10:14:39 PM - POLLER: Poller[0] Maximum runtime of 292 seconds exceeded. Exiting.
01/12/2007 10:14:39 PM - SYSTEM STATS: Time:292.9227 Method:cactid Processes:12 Threads:12 Hosts:222 HostsPerProcess:19 DataSources:15519 RRDsProcessed:3975

And, when break occurs:

Code: Select all

# ps -efL|grep cacti
    root  3852  4806     1     1  0 22:20:30 pts/3    0:00 grep cacti
   cacti   219 18961     1     1  5 22:18:35 pts/2    0:11 /usr/local/bin/php /var/htdocs/cacti/poller.php
   cacti 18961  4660     1     1  0 21:45:51 pts/2    0:00 -bash
   cacti   718   717     1     1  1 22:18:48 pts/2    0:02 /usr/local/rrdtool-1.2.15/bin/rrdtool -
   cacti   717   219     1     1  0 22:18:48 pts/2    0:00 sh -c /usr/local/rrdtool-1.2.15/bin/rrdtool -

No, the pid of rrdtool process does not change.

On the other installation, on dual opteron with 2gb of ram (Sun Fire v20z), with about 20cisco switches monitored, ALL is completed within 6 seconds (!).
Do you have any suggestions, how to improve performance on this box?

Post by **TheWitness** » Fri Jan 12, 2007 10:03 pm

Are you upto date with patches? Please patch your system. Goto http://www.cacti.net/download_patches.php and update first.

TheWitness

Post by **TheWitness** » Fri Jan 12, 2007 10:07 pm

Also, goto Utilities -> View Poller Cache and provide a count of the number of 1) SNMP, 2) Script, 3) Script Server

With a server of your capacity and especially the amount of RAM you have to be having performance problems unless you simply have too much running on it outside of Cacti. Where is that 8GBytes going? It should be all disc cache too allowing poller.php to complete quickly.

TheWitness

Post by **TheWitness** » Fri Jan 12, 2007 10:12 pm

Also, how is CPU during polling? You may have too many processes running causing your system to overheat. I place "sleep" logic in the threads to reduce context switching. However, I have had to remove them due to issues with Solaris' POSIX thread model that causes threads to misbehave when executing sleep functions.

That might have something to do about it.

TheWitness

Post by **gandalf** » Sat Jan 13, 2007 12:51 pm

TheWitness wrote:Also, goto Utilities -> View Poller Cache and provide a count of the number of 1) SNMP, 2) Script, 3) Script Server

... or use my newest Templates for "Poller Statistics". They report these data on a 5 min base ...

(Sorry for advertising my own stuff. But I really like this script

)
Reinhard

Post by **TheWitness** » Mon Jan 15, 2007 10:51 pm

Me too

jdr · Post by **jdr** » Wed Jan 17, 2007 2:22 pm

Yeah, my system is patched with all security and recommended patches.
I was missing dec06-vulnerability-scripts-0.8.6i.patch and import_template_argument_space_removal.patch, but now i'm up-to-date.

SNMP: 15350
Script: 200
Script server: 0

The machine is doing only cacti and nagios, so it isn't very loaded.

Memory:

Code: Select all

# vmstat
 kthr      memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr m0 m1 m2 m1   in   sy   cs us sy id
 2 0 0 8202064 5026024 340 7024 454 752 750 0 0 0 0 0 0 743 15116 1267 39 12 49

There is a sample from cpu load during the cacti poller cycle (not sure if there are all processors, or one of them):

Code: Select all

# sar -u 10 30

SunOS sss 5.9 Generic_118558-38 sun4u    01/17/2007

20:34:53    %usr    %sys    %wio   %idle
20:35:03      25       9      27      39
20:35:13      77      23       0       0
20:35:23      79      21       0       0
20:35:33      68      21       5       6
20:35:43      36      14      22      29
20:35:53      39      11      23      27
20:36:03      44      14      23      18
20:36:13      27       8      28      37
20:36:23       3       2      17      78
20:36:33      40      12       8      41
20:36:43      46      14       7      32
20:36:53      20       9       6      65

Also, I've noticed some performance improvment when digging the logs:

Code: Select all

# grep 'SYSTEM STATS' log/cacti.log | tail -20
01/17/2007 06:42:22 PM - SYSTEM STATS: Time:141.9642 Method:cactid Processes:12 Threads:12 Hosts:223 HostsPerProcess:19 DataSources:15565 RRDsProcessed:7997
01/17/2007 06:47:42 PM - SYSTEM STATS: Time:161.1686 Method:cactid Processes:12 Threads:12 Hosts:223 HostsPerProcess:19 DataSources:15565 RRDsProcessed:7997
01/17/2007 06:52:31 PM - SYSTEM STATS: Time:149.7432 Method:cactid Processes:12 Threads:12 Hosts:223 HostsPerProcess:19 DataSources:15565 RRDsProcessed:7997
01/17/2007 06:57:35 PM - SYSTEM STATS: Time:154.8634 Method:cactid Processes:12 Threads:12 Hosts:223 HostsPerProcess:19 DataSources:15565 RRDsProcessed:7997
01/17/2007 07:08:06 PM - SYSTEM STATS: Time:485.1552 Method:cactid Processes:12 Threads:12 Hosts:223 HostsPerProcess:19 DataSources:15565 RRDsProcessed:7997
01/17/2007 07:08:15 PM - SYSTEM STATS: Time:194.2778 Method:cactid Processes:12 Threads:12 Hosts:223 HostsPerProcess:19 DataSources:15565 RRDsProcessed:7997
01/17/2007 07:12:28 PM - SYSTEM STATS: Time:147.0844 Method:cactid Processes:12 Threads:12 Hosts:223 HostsPerProcess:19 DataSources:15565 RRDsProcessed:7997
01/17/2007 07:17:38 PM - SYSTEM STATS: Time:156.8278 Method:cactid Processes:12 Threads:12 Hosts:223 HostsPerProcess:19 DataSources:15565 RRDsProcessed:7997
01/17/2007 07:22:35 PM - SYSTEM STATS: Time:154.0539 Method:cactid Processes:12 Threads:12 Hosts:223 HostsPerProcess:19 DataSources:15565 RRDsProcessed:7997
01/17/2007 07:27:24 PM - SYSTEM STATS: Time:142.8399 Method:cactid Processes:12 Threads:12 Hosts:223 HostsPerProcess:19 DataSources:15565 RRDsProcessed:7997
01/17/2007 07:35:59 PM - SYSTEM STATS: Time:358.4279 Method:cactid Processes:12 Threads:12 Hosts:223 HostsPerProcess:19 DataSources:15565 RRDsProcessed:7997
01/17/2007 07:36:08 PM - SYSTEM STATS: Time:67.8719 Method:cactid Processes:12 Threads:12 Hosts:223 HostsPerProcess:19 DataSources:15565 RRDsProcessed:1631
01/17/2007 07:42:18 PM - SYSTEM STATS: Time:137.3200 Method:cactid Processes:12 Threads:12 Hosts:223 HostsPerProcess:19 DataSources:15565 RRDsProcessed:7997
01/17/2007 07:47:31 PM - SYSTEM STATS: Time:150.5625 Method:cactid Processes:12 Threads:12 Hosts:223 HostsPerProcess:19 DataSources:15565 RRDsProcessed:7997
01/17/2007 07:52:40 PM - SYSTEM STATS: Time:159.6144 Method:cactid Processes:12 Threads:12 Hosts:223 HostsPerProcess:19 DataSources:15565 RRDsProcessed:7997
01/17/2007 07:57:36 PM - SYSTEM STATS: Time:155.8046 Method:cactid Processes:12 Threads:12 Hosts:223 HostsPerProcess:19 DataSources:15565 RRDsProcessed:7997
01/17/2007 08:06:06 PM - SYSTEM STATS: Time:365.3097 Method:cactid Processes:12 Threads:12 Hosts:223 HostsPerProcess:19 DataSources:15565 RRDsProcessed:7997
01/17/2007 08:06:19 PM - SYSTEM STATS: Time:78.7800 Method:cactid Processes:12 Threads:12 Hosts:223 HostsPerProcess:19 DataSources:15565 RRDsProcessed:1715
01/17/2007 08:12:18 PM - SYSTEM STATS: Time:137.1342 Method:cactid Processes:12 Threads:12 Hosts:223 HostsPerProcess:19 DataSources:15565 RRDsProcessed:7965
01/17/2007 08:17:30 PM - SYSTEM STATS: Time:149.5451 Method:cactid Processes:12 Threads:12 Hosts:223 HostsPerProcess:19 DataSources:15565 RRDsProcessed:7997

Gandalf, Even with search function, I couldn't find you Poller Statistic template. Could you please post a link?

jdr · Post by **jdr** » Wed Jan 17, 2007 2:38 pm

Actually, when running cacti, i can see on the beginning of polling process:

Code: Select all

# ps -efL|grep cacti | wc -l
     323
# ps -efL|grep cacti | wc -l
     265
# ps -efL|grep cacti | wc -l
     245

(I suspect, that cacti is polling data at this time), than

Code: Select all

# ps -efL|grep cacti | wc -l
      28
# ps -efL|grep cacti | wc -l
       9

And 9, 9, 9... then finish.

Post by **TheWitness** » Wed Jan 17, 2007 5:45 pm

Do me a favor and watch only the Cactid binaries using top and let me know how your polling goes it should be something like:

time with cactid binaries running (XXX seconds)
time without cactid binaries running (YYY seconds)

total polling time (XXX+YYY seconds)

Thanks,

TheWitness

jdr · Post by **jdr** » Thu Jan 18, 2007 9:24 am

Cactid polling time: almost 60seconds (i saw lots of php process at this time)
Then, rrdtool comes in action, which consumes lots of cpu power.
Total time: 240 seconds.

Code: Select all

# ps -efL|grep cacti|wc -l ; ps -efL|grep php|wc -l
     320
     136

Post by **gandalf** » Thu Jan 18, 2007 11:53 am

http://forums.cacti.net/viewtopic.php?t=18057
Reinhard

jdr · Post by **jdr** » Thu Jan 18, 2007 12:54 pm

Thanks, it's working. I'll put some graphs after reasonable (24h?) period of time.

Post by **TheWitness** » Thu Jan 18, 2007 1:34 pm

You may be a good cantidate for boost as your poller is finishing quite quickly. You should also increase your disk cache significantly. You should also verify that your write cache is NOT disabled. If you do the two following steps, you performance should get significantly better:

1) Increase Amount of Ram for Disk Cache
2) Verify/Enable Write Cache on your disks.

Make sure you have a UPS

TheWitness

jdr · Post by **jdr** » Fri Jan 19, 2007 8:34 am

While playing with tunning up solaris kernel and discs, i noticed:

Code: Select all

01/19/2007 01:41:34 PM - SYSTEM STATS: Time:93.8190 Method:cactid Processes:12 Threads:12 Hosts:223 HostsPerProcess:19 DataSources:15562 RRDsProcessed:1629
01/19/2007 01:49:27 PM - SYSTEM STATS: Time:266.1431 Method:cactid Processes:12 Threads:12 Hosts:223 HostsPerProcess:19 DataSources:15562 RRDsProcessed:7888
01/19/2007 01:55:01 PM - SYSTEM STATS: Time:300.4820 Method:cactid Processes:12 Threads:12 Hosts:223 HostsPerProcess:19 DataSources:15562 RRDsProcessed:7888
01/19/2007 02:00:04 PM - SYSTEM STATS: Time:303.9662 Method:cactid Processes:12 Threads:12 Hosts:223 HostsPerProcess:19 DataSources:15562 RRDsProcessed:7885
01/19/2007 02:07:34 PM - SYSTEM STATS: Time:453.0949 Method:cactid Processes:12 Threads:12 Hosts:223 HostsPerProcess:19 DataSources:15562 RRDsProcessed:7463
01/19/2007 02:20:12 PM - SYSTEM STATS: Time:311.5792 Method:cactid Processes:12 Threads:12 Hosts:223 HostsPerProcess:19 DataSources:15562 RRDsProcessed:8124
01/19/2007 02:28:23 PM - SYSTEM STATS: Time:503.2490 Method:cactid Processes:12 Threads:12 Hosts:223 HostsPerProcess:19 DataSources:15562 RRDsProcessed:13990
01/19/2007 02:31:28 PM - SYSTEM STATS: Time:387.2142 Method:cactid Processes:12 Threads:12 Hosts:223 HostsPerProcess:19 DataSources:15562 RRDsProcessed:7858

I have all the patches installed. Why number of RRDsProcessed differs between poller cycles?

Post by **TheWitness** » Fri Jan 19, 2007 8:51 am

Interesting, are you running the latest version 0.8.6j? If you are, then look into <path_cacti>/poller.php and change all lines that read:

process_poller_output($rrdtool_pipe)

to

process_poller_output($rrdtool_pipe, TRUE)

See if that makes a difference.

TheWitness

Ugly performance problems.

Ugly performance problems.

Who is online