Slow cactid performance

jgisler · Post by **jgisler** » Sun Oct 16, 2005 2:24 pm

Using cmd.php I was getting similar results with random failures to complete. I have switched to cactid but the results are still rather off.

Anyone have any ideas or directions in which to look to isolate the issue & get better results?

10/16/2005 03:04:58 AM - SYSTEM STATS: Time:296.7787 Method:cactid Processes:3 Threads:40 Hosts:411 HostsPerProcess:137 DataSources:1354 RRDsProcessed:681

Server:
Dual Xeon 2.8
2GIG's Memory
U160 SCSI
Fedora Core
Mysql 4.0.x
php-4.3.9
apache-1.3.3
net-snmp-5.2.1.2
rrd-tool-1.2.11

jgisler · Post by **jgisler** » Sun Oct 16, 2005 2:39 pm

cacti.log attached. Newer one for Ivan.

Post by **gandalf** » Sun Oct 16, 2005 2:44 pm

Just did some searching on the forum, but could not retrieve the post I'm remembering. But I suppose it was something like
Processes = 2* CPUs (nut don't know how to count HT CPUs)
Threads = about 8??? or was it 12???

I was wondering about the numbers of threads in your log. They constandly are keeping up at the upper limit of 40. But I'm just at home, so my laptop isn't an adequate comparison...

Reinhard
(btw) Please edit your post to include a link to your logfile for others that may read your post

Post by **TheWitness** » Sun Oct 16, 2005 3:13 pm

Here is one of your problems:

10/16/2005 12:31:47 PM - CACTID: Poller[0] Host[458] DEBUG: HOST COMPLETE: About to Exit Host Polling Thread Function
10/16/2005 12:31:47 PM - CACTID: Poller[0] DEBUG: The Value of Active Threads is 3
10/16/2005 12:32:09 PM - POLLER: Poller[0] CACTI2RRD: /usr/local/rrdtool-1.2.11/bin/rrdtool update /user1/home/sites/support/cacti86g/rra/server01097_server_cc_traffic_in_181.rrd --template traffic_in:traffic_out 1129491000:2867509627:1788586263
10/16/2005 12:32:09 PM - POLLER: Poller[0] CACTI2RRD: /usr/local/rrdtool-1.2.11/bin/rrdtool update /user1/home/sites/support/cacti86g/rra/server01108_server_cc_traffic_in_203.rrd --template traffic_in:traffic_out 1129491000:7250343:979303953
1

Note the large tie gap between your last cactid command and the first cacti2rrd command. Check the following:

1) You don't have enough connections in MySQL (I use 1000 in my production server)
2) PHP does not have enough memory to process the polling request.

Otherwise, 40 threads is a little high. I like to run with 2 processes and about 15-20 threads only.

TheWitness

jgisler · Post by **jgisler** » Sun Oct 16, 2005 3:31 pm

Will do, checking these items now. Thanks for the reply The Witness!

All help is always appreciated.

jgisler · Post by **jgisler** » Sun Oct 16, 2005 3:39 pm

The max connections was set for 300. I have increased it to 1200.

I have changed the processes to 2 & threads to 15.

jgisler · Post by **jgisler** » Sun Oct 16, 2005 4:24 pm

No change.

10/16/2005 02:19:57 PM - SYSTEM STATS: Time:296.6399 Method:cactid Processes:2 Threads:20 Hosts:411 HostsPerProcess:206 DataSources:1354 RRDsProcessed:680
10/16/2005 02:19:57 PM - CACTID: Poller[0] ERROR: Cactid Timed Out While Processing Hosts Internal

Post by **TheWitness** » Sun Oct 16, 2005 8:39 pm

What is your php memory limit set to? Also, describe your server load?

TheWitness

jgisler · Post by **jgisler** » Sun Oct 16, 2005 9:11 pm

Server load: 19:09:34 up 20:07, 1 user, load average: 2.72, 3.19, 3.1

84 processes: 83 sleeping, 1 running, 0 zombie, 0 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 75.2% 0.0% 6.8% 0.0% 0.0% 0.0% 317.2%
cpu00 18.3% 0.0% 1.5% 0.0% 0.0% 0.0% 80.0%
cpu01 21.5% 0.0% 0.9% 0.0% 0.0% 0.0% 77.4%
cpu02 22.1% 0.0% 2.1% 0.0% 0.0% 0.0% 75.6%
cpu03 13.3% 0.0% 2.1% 0.0% 0.0% 0.0% 84.4%
Mem: 2068432k av, 972232k used, 1096200k free, 0k shrd, 157304k buff
250340k active, 587068k inactive
Swap: 2048216k av, 0k used, 2048216k free 572252k cached

php memory setting: memory_limit = 96M

Post by **TheWitness** » Sun Oct 16, 2005 9:23 pm

Hmm, have your updated to Cactid 0.8.6f-1? If not please do so...

TheWitness

jgisler · Post by **jgisler** » Mon Oct 17, 2005 1:53 am

Yes, I am running the current Cactid-0.8.6f-1

jgisler · Post by **jgisler** » Mon Oct 17, 2005 1:21 pm

I changed the process to 4 & threads to 35. With it at 2 processes & 20 threads I was getting poller timed out errors. I'm no longer getting them with this configuration but poller times are still rather high.

10/17/2005 11:19:56 AM - SYSTEM STATS: Time:295.6674 Method:cactid Processes:4 Threads:35 Hosts:411 HostsPerProcess:103 DataSources:1354 RRDsProcessed:680
10/17/2005 11:19:55 AM - CACTID: Poller[0] ERROR: Cactid Timed Out While Processing Hosts Internal
10/17/2005 11:14:56 AM - SYSTEM STATS: Time:295.9162 Method:cactid Processes:4 Threads:35 Hosts:411 HostsPerProcess:103 DataSources:1354 RRDsProcessed:680
10/17/2005 11:14:55 AM - CACTID: Poller[0] ERROR: Cactid Timed Out While Processing Hosts Internal
10/17/2005 11:09:56 AM - SYSTEM STATS: Time:295.4880 Method:cactid Processes:4 Threads:35 Hosts:411 HostsPerProcess:103 DataSources:1354 RRDsProcessed:680
10/17/2005 11:09:56 AM - CACTID: Poller[0] ERROR: Cactid Timed Out While Processing Hosts Internal
10/17/2005 11:04:57 AM - SYSTEM STATS: Time:296.2099 Method:cactid Processes:4 Threads:35 Hosts:411 HostsPerProcess:103 DataSources:1354 RRDsProcessed:680
10/17/2005 11:04:56 AM - CACTID: Poller[0] ERROR: Cactid Timed Out While Processing Hosts Internal
10/17/2005 10:59:57 AM - SYSTEM STATS: Time:296.1440 Method:cactid Processes:4 Threads:35 Hosts:411 HostsPerProcess:103 DataSources:1354 RRDsProcessed:680
10/17/2005 10:59:56 AM - CACTID: Poller[0] ERROR: Cactid Timed Out While Processing Hosts Internal
10/17/2005 10:54:56 AM - SYSTEM STATS: Time:296.3692 Method:cactid Processes:4 Threads:35 Hosts:411 HostsPerProcess:103 DataSources:1354 RRDsProcessed:680
10/17/2005 10:54:56 AM - CACTID: Poller[0] ERROR: Cactid Timed Out While Processing Hosts Internal
10/17/2005 10:49:57 AM - SYSTEM STATS: Time:296.0255 Method:cactid Processes:4 Threads:35 Hosts:411 HostsPerProcess:103 DataSources:1354 RRDsProcessed:680
10/17/2005 10:49:56 AM - CACTID: Poller[0] ERROR: Cactid Timed Out While Processing Hosts Internal
10/17/2005 10:44:57 AM - SYSTEM STATS: Time:296.3683 Method:cactid Processes:4 Threads:35 Hosts:411 HostsPerProcess:103 DataSources:1354 RRDsProcessed:680
10/17/2005 10:44:56 AM - CACTID: Poller[0] ERROR: Cactid Timed Out While Processing Hosts Internal
10/17/2005 10:39:56 AM - SYSTEM STATS: Time:296.5498 Method:cactid Processes:4 Threads:35 Hosts:411 HostsPerProcess:103 DataSources:1354 RRDsProcessed:680
10/17/2005 10:39:55 AM - CACTID: Poller[0] ERROR: Cactid Timed Out While Processing Hosts Internal
10/17/2005 10:34:57 AM - SYSTEM STATS: Time:296.7765 Method:cactid Processes:4 Threads:35 Hosts:411 HostsPerProcess:103 DataSources:1354 RRDsProcessed:680
10/17/2005 10:34:56 AM - CACTID: Poller[0] ERROR: Cactid Timed Out While Processing Hosts Internal
10/17/2005 10:29:59 AM - SYSTEM STATS: Time:296.6197 Method:cactid Processes:4 Threads:35 Hosts:411 HostsPerProcess:103 DataSources:1354 RRDsProcessed:680
10/17/2005 10:29:58 AM - CACTID: Poller[0] ERROR: Cactid Timed Out While Processing Hosts Internal
10/17/2005 10:24:57 AM - SYSTEM STATS: Time:296.1638 Method:cactid Processes:4 Threads:35 Hosts:411 HostsPerProcess:103 DataSources:1354 RRDsProcessed:680
10/17/2005 10:24:57 AM - CACTID: Poller[0] ERROR: Cactid Timed Out While Processing Hosts Internal

jgisler · Post by **jgisler** » Mon Oct 17, 2005 2:39 pm

mysql info:

MySQL 4.0.18-log uptime 0 0:17:17 Mon Oct 17 12:36:45 2005

__ Key _________________________________________________________________
Buffer usage 632.00k of 384.00M %Used: 0.16
Write ratio 0.56
Read ratio 0.00

__ Questions ___________________________________________________________
Total 237.27k 228.80/s
QC Hits 194.29k 187.35/s %Total: 81.89
Com_ 20.45k 19.72/s 8.62
DMS 19.50k 18.80/s 8.22
COM_QUIT 2.69k 2.59/s 1.13
+Unknown 351 0.34/s 0.15
Slow 0 0.00/s 0.00 %DMS: 0.00
DMS 19.50k 18.80/s 8.22
SELECT 10.19k 9.82/s 4.29 52.26
DELETE 4.96k 4.79/s 2.09 25.46
INSERT 1.64k 1.58/s 0.69 8.39
UPDATE 1.63k 1.57/s 0.69 8.35
REPLACE 1.08k 1.04/s 0.46 5.55
Com_ 20.45k 19.72/s 8.62
change_db 20.44k 19.71/s 8.61
truncate 8 0.01/s 0.00
show_status 1 0.00/s 0.00

__ SELECT and Sort _____________________________________________________
Scan 1.35k 1.30/s %SELECT: 13.26
Range 8 0.01/s 0.08
Full join 642 0.62/s 6.30
Range check 0 0.00/s 0.00
Full rng join 0 0.00/s 0.00
Sort scan 475 0.46/s
Sort range 856 0.83/s
Sort mrg pass 0 0.00/s

__ Query Cache _________________________________________________________
Memory usage 5.12M of 64.00M %Used: 8.01
Block Fragmnt 0.86%
Hits 194.29k 187.3/s
Inserts 10.26k 9.90/s
Prunes 1 0.00/s
Insrt:Prune 10.26k:1 9.90/s
Hit:Insert 18.93:1

__ Table Locks _________________________________________________________
Waited 1.72k 1.66/s %Total: 6.08
Immediate 26.62k 25.67/s

__ Tables ______________________________________________________________
Open 91 of 512 %Cache: 17.77
Opened 127 0.12/s

__ Connections _________________________________________________________
Max used 29 of 1500 %Max: 1.93
Total 2.70k 2.60/s

__ Created Temp ________________________________________________________
Disk table 10 0.01/s
Table 445 0.43/s
File 0 0.00/s

jgisler · Post by **jgisler** » Mon Oct 17, 2005 4:09 pm

The servers storage consists of:

Operating system: RAID1
Data Storage: RAID5

jgisler · Post by **jgisler** » Mon Oct 17, 2005 4:13 pm

Post by The Witness:

Please make sure the MIBS environment variable is set to something basic like "MIBS=RFC_1215". What ever you do, don't set to ALL. Especially if you have a lot of MIB files in your MIBDIRS.

Where do I set this setting at? I removed all the mibs in the mibs directory.

Slow cactid performance

Slow cactid performance

Who is online