Ugly performance problems.

[jdr]

Okay, i set:
tune_t_fsflushr = 10
autoup = 60
bufhwm = 1572864
and also mounted /usr and /var with noatime option.
Nothing changed.
I start to consider, why does it take so long, since the drives are U320 SCSI.
Maybe RAID1 slow write on the filesystem so much?

[TheWitness]

I'm betting it's a Solaris thing. Call Sun and ask them why there hardware, you know %#$#'s.

TheWitness

[jdr]

Row 163:

Code: Select all

                        $rrds_processed = $rrds_processed + process_poller_output($rrdtool_pipe, TRUE);

Row 199:

Code: Select all

                        $rrds_processed = $rrds_processed + process_poller_output($rrdtool_pipe);

(changed to:

Code: Select all

                        $rrds_processed = $rrds_processed + process_poller_output($rrdtool_pipe,TRUE);

)

Call Sun and ask them why there hardware, you know %#$#'s.

Yeah, that's great idea =)

[TheWitness]

Did that change make a difference?

TheWitness

[jdr]

not-at-all

[jdr]

Code: Select all

01/23/2007 11:40:47 AM - SYSTEM STATS: Time:346.4342 Method:cactid Processes:12 Threads:12 Hosts:219 HostsPerProcess:19 DataSources:15455 RRDsProcessed:7944
[b]01/23/2007 11:41:21 AM - SYSTEM STATS: Time:80.1951 Method:cactid Processes:12 Threads:12 Hosts:219 HostsPerProcess:19 DataSources:15455 RRDsProcessed:1505[/b]
01/23/2007 11:49:08 AM - SYSTEM STATS: Time:248.1336 Method:cactid Processes:12 Threads:12 Hosts:219 HostsPerProcess:19 DataSources:15455 RRDsProcessed:7928
01/23/2007 11:54:23 AM - SYSTEM STATS: Time:262.6669 Method:cactid Processes:12 Threads:12 Hosts:219 HostsPerProcess:19 DataSources:15455 RRDsProcessed:7944
01/23/2007 11:59:27 AM - SYSTEM STATS: Time:266.5008 Method:cactid Processes:12 Threads:12 Hosts:219 HostsPerProcess:19 DataSources:15455 RRDsProcessed:7928
01/23/2007 12:09:11 PM - SYSTEM STATS: Time:551.4212 Method:cactid Processes:12 Threads:12 Hosts:219 HostsPerProcess:19 DataSources:15455 RRDsProcessed:7944
01/23/2007 12:10:51 PM - SYSTEM STATS: Time:350.2769 Method:cactid Processes:12 Threads:12 Hosts:219 HostsPerProcess:19 DataSources:15455 RRDsProcessed:7944
[b]01/23/2007 12:11:21 PM - SYSTEM STATS: Time:80.4930 Method:cactid Processes:12 Threads:12 Hosts:219 HostsPerProcess:19 DataSources:15455 RRDsProcessed:1589[/b]
01/23/2007 12:19:01 PM - SYSTEM STATS: Time:241.0694 Method:cactid Processes:12 Threads:12 Hosts:219 HostsPerProcess:19 DataSources:15455 RRDsProcessed:7939
01/23/2007 12:24:19 PM - SYSTEM STATS: Time:258.6829 Method:cactid Processes:12 Threads:12 Hosts:219 HostsPerProcess:19 DataSources:15455 RRDsProcessed:7886
01/23/2007 12:29:17 PM - SYSTEM STATS: Time:256.7209 Method:cactid Processes:12 Threads:12 Hosts:219 HostsPerProcess:19 DataSources:15455 RRDsProcessed:7872
01/23/2007 12:38:36 PM - SYSTEM STATS: Time:515.9245 Method:cactid Processes:12 Threads:12 Hosts:219 HostsPerProcess:19 DataSources:15455 RRDsProcessed:7944
01/23/2007 12:40:36 PM - SYSTEM STATS: Time:335.9743 Method:cactid Processes:12 Threads:12 Hosts:219 HostsPerProcess:19 DataSources:15455 RRDsProcessed:7944
01/23/2007 12:41:29 PM - SYSTEM STATS: Time:88.9258 Method:cactid Processes:12 Threads:12 Hosts:219 HostsPerProcess:19 DataSources:15455 RRDsProcessed:2062

Again, I have all patches installed, and ocasionally I can see, that not all the rrds was processed during the polling.

[TheWitness]

If you looks at your polling time, you will notice that you have multiple poller.php's stacked up on top of one another that will always do it.

TheWitness

[TheWitness]

I really think you need boost unless you can get Sun to answer why your disk write cache is not working. With 8GBytes of memory, you should have at least 6GBytes in your disk cache, which means that there is no excuse that your updates are taking longer than 70 seconds.

TheWitness

[jdr]

I also noticed lots of

Code: Select all

01/23/2007 12:55:33 PM - CACTID: Poller[0] Host[230] DS[10350] SS[0] WARNING: Result from SERVER not valid. Partial Result: ...

Even from localhost.

[TheWitness]

post the results to the following:

php -v

TheWitness

[jdr]

Code: Select all

# php -v
PHP 5.2.0 (cli) (built: Dec 21 2006 22:12:07)
Copyright (c) 1997-2006 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2006 Zend Technologies

[TheWitness]

Ok, you likely have invalid data sources. You should research it further.

TheWitness

[jdr]

Yeah, before contacting sun i decided to make some performance test.

Installed linux on HP ProLiant 360G3 (Dual Xeon 3.0), 4.0GB of RAM, Compaq Smart Array raid controller with 128Mbytes of memory and dual U320-scsi discs working on hardware raid-1.

Guess what:

Code: Select all

01/26/2007 06:57:40 PM - SYSTEM STATS: Time:158.0420 Method:cmd.php Processes:12 Threads:N/A Hosts:221 HostsPerProcess:19 DataSources:15510 RRDsProcessed:8031
01/26/2007 07:02:02 PM - SYSTEM STATS: Time:120.9942 Method:cmd.php Processes:12 Threads:N/A Hosts:221 HostsPerProcess:19 DataSources:15510 RRDsProcessed:7893
01/26/2007 07:06:59 PM - SYSTEM STATS: Time:118.0960 Method:cmd.php Processes:12 Threads:N/A Hosts:221 HostsPerProcess:19 DataSources:15510 RRDsProcessed:7891
01/26/2007 07:12:52 PM - SYSTEM STATS: Time:170.7156 Method:cmd.php Processes:12 Threads:N/A Hosts:221 HostsPerProcess:19 DataSources:15510 RRDsProcessed:7845
01/26/2007 07:17:22 PM - SYSTEM STATS: Time:140.7813 Method:cmd.php Processes:12 Threads:N/A Hosts:221 HostsPerProcess:19 DataSources:15510 RRDsProcessed:7873
01/26/2007 07:21:54 PM - SYSTEM STATS: Time:112.8219 Method:cmd.php Processes:12 Threads:N/A Hosts:221 HostsPerProcess:19 DataSources:15510 RRDsProcessed:7933
01/26/2007 07:26:57 PM - SYSTEM STATS: Time:115.3953 Method:cmd.php Processes:12 Threads:N/A Hosts:221 HostsPerProcess:19 DataSources:15510 RRDsProcessed:7933
01/26/2007 07:32:00 PM - SYSTEM STATS: Time:118.7431 Method:cmd.php Processes:12 Threads:N/A Hosts:221 HostsPerProcess:19 DataSources:15510 RRDsProcessed:7949

Please, notice, that method is cmd.php.
I just switched to cactid, waiting for results.

Nice, huh?

[jdr]

Code: Select all

01/26/2007 09:00:40 PM - SYSTEM STATS: Time:39.5860 Method:cactid Processes:16 Threads:12 Hosts:221 HostsPerProcess:14 DataSources:15510 RRDsProcessed:7949
01/26/2007 09:05:41 PM - SYSTEM STATS: Time:40.0492 Method:cactid Processes:16 Threads:12 Hosts:221 HostsPerProcess:14 DataSources:15510 RRDsProcessed:7949
01/26/2007 09:10:35 PM - SYSTEM STATS: Time:33.1920 Method:cactid Processes:6 Threads:8 Hosts:221 HostsPerProcess:37 DataSources:15510 RRDsProcessed:7921
01/26/2007 09:15:32 PM - SYSTEM STATS: Time:30.7133 Method:cactid Processes:4 Threads:6 Hosts:221 HostsPerProcess:56 DataSources:15510 RRDsProcessed:7949
01/26/2007 09:20:33 PM - SYSTEM STATS: Time:31.8100 Method:cactid Processes:4 Threads:6 Hosts:221 HostsPerProcess:56 DataSources:15510 RRDsProcessed:7949
01/26/2007 09:25:28 PM - SYSTEM STATS: Time:27.1171 Method:cactid Processes:2 Threads:12 Hosts:221 HostsPerProcess:111 DataSources:15510 RRDsProcessed:7921
01/26/2007 09:30:30 PM - SYSTEM STATS: Time:28.8296 Method:cactid Processes:1 Threads:16 Hosts:221 HostsPerProcess:221 DataSources:15510 RRDsProcessed:7949
01/26/2007 09:35:28 PM - SYSTEM STATS: Time:27.3642 Method:cactid Processes:2 Threads:16 Hosts:221 HostsPerProcess:111 DataSources:15510 RRDsProcessed:7949

This is over 10x speedup.
I need to figure out how to achieve it on sun, or move everything on this box and put this shitty sun on the shelf, you know, this one behind the window =)

Thanks for your help!

[jdr]

I have this feeling, that cause of problems on my sun is the fact, that despite we have multiple polling processess, the rrdtool process, that itself updates the rra's is only one.

When I switched from sun (4x1064MHz Ultra Sparc) to HP (2x3.06GHz Xeon) I noticed huge speedup.
Notice, that on the both systems there are hi-speed U-320 scsi drives (only the controller differs).

I assume, that it could be not neccessary disk writing problem, but one rrdtool process, which consumes all power from ONLY ONE cpu, while other remains doing "nothing".

Conclusion is, that if there were multiple rrdtool's updating processes, it would run much faster on systems, where we are more slower cpus.
Right?