Index  | Recent Threads  | Unanswered Threads  | Who's Active  | Guidelines  | Search
 

Quick Go »
No member browsing this thread
Thread Status: Active
Total posts in this thread: 156
Posts: 156   Pages: 16   [ Previous Page | 7 8 9 10 11 12 13 14 15 16 | Next Page ]
[ Jump to Last Post ]
Post new Thread
Author
Previous Thread This topic has been viewed 350403 times and has 155 replies Next Thread
Sekerob
Ace Cruncher
Joined: Jul 24, 2005
Post Count: 20043
Status: Offline
Reply to this Post  Reply with Quote 
Re: Poor crunching on this project

We're not talking the lost time, but the reason described and long discussed of the time that goes towards IO and why WCG changed the default work distribution for CEP2: https://secure.worldcommunitygrid.org/forums/...ad,30161_offset,50#300323
----------------------------------------
WCG Global & Research > Make Proposal Help: Start Here!
Please help to make the Forums an enjoyable experience for All!
[Oct 28, 2010 12:52:27 PM]   Link   Report threatening or abusive post: please login first  Go to top 
sk..
Master Cruncher
http://s17.rimg.info/ccb5d62bd3e856cc0d1df9b0ee2f7f6a.gif
Joined: Mar 22, 2007
Post Count: 2324
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Poor crunching on this project

Well that explains it; you are discussing something about USB IO and why the WCG changed the default work distribution for CEP2 in the Poor crunching on this project thread, instead of in your link to the Clean Energy Project Phase 2 Scheduling change or one of the many other similar or related threads that spawned from other CEP2 topics - not that I stick rigidly to topics, especially those with such poorly named titles.
[Oct 28, 2010 3:44:22 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Sekerob
Ace Cruncher
Joined: Jul 24, 2005
Post Count: 20043
Status: Offline
Reply to this Post  Reply with Quote 
Re: Poor crunching on this project

No, I'm talking about the 30-45 minutes most everyone experiences with CEP2 on Linux. The USB IO is another fish to fry... covered in the next post, nearly ready to post... waiting on completion of more C4CW tasks and an HFCC.

Sticking to the topic would be a major breath of fresh air... this one is about poor crunching (efficiency).
----------------------------------------
WCG Global & Research > Make Proposal Help: Start Here!
Please help to make the Forums an enjoyable experience for All!
[Oct 28, 2010 4:04:07 PM]   Link   Report threatening or abusive post: please login first  Go to top 
sk..
Master Cruncher
http://s17.rimg.info/ccb5d62bd3e856cc0d1df9b0ee2f7f6a.gif
Joined: Mar 22, 2007
Post Count: 2324
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Poor crunching on this project

OK, I will try to confine things to IO slowness, observed by most people for all tasks and especially CEP2 by some. I wish I still had my RAM drive to test this. Mind you the OCZ 80GB RevoDrive PCI-E SSD (OCZSSDPX-1RVD0080) would be much faster than the one I had - better chance of noticing any difference. If I had a 12 thread system and wanted just to crunch this project I might want one or two of these; could possibly save 12h runtime per day.
I guess triple channel RAM systems should fare better than dual channel, so it would be interesting to compare an i7 LGA 1366’s losses to an i7 LGA 1156.
Using a different hard drive for Virtual Memory would be better than using your primary drive and RAID systems should also be slightly faster.
Is there any way of creating a Virtual Drive from system memory?
Could you setup a VM only in RAM and run this solely from the VM?
Some Linux versions allow you to run them without installing them, would be interesting to see how these perform (if it is possible to run Boinc this way).
[Oct 28, 2010 5:00:53 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Sekerob
Ace Cruncher
Joined: Jul 24, 2005
Post Count: 20043
Status: Offline
Reply to this Post  Reply with Quote 
Re: Poor crunching on this project

Intermediate data is building... I'm hopeful to bring all sciences back to the 97-99%+ and CEP2 to 95% instead of 90-92%. The WTD extending helped already quite allot.

VM and RAM disk are from what I've read no big issue.... you may want to research that and add that to the pool of knowledge... substantially, something needs to back up what's in there on a periodic basis or booting becomes quite costly in progress loss... even for 24/7 crunchers.

Though all avenues on the swap file angle has not paid any dividend for crunching, it has made my quad system more user-responsive not having any, but that is a Linux thing. For Windows a VM is almost a must (W7 and maybe Vista and earlier) since MS uses strategies such as writing portions of RAM retained data to VM, so it can literally unmap it from RAM instantenously and point it to the VM mirror when RAM is actually needed for something else.
----------------------------------------
WCG Global & Research > Make Proposal Help: Start Here!
Please help to make the Forums an enjoyable experience for All!
[Oct 28, 2010 5:21:16 PM]   Link   Report threatening or abusive post: please login first  Go to top 
sk..
Master Cruncher
http://s17.rimg.info/ccb5d62bd3e856cc0d1df9b0ee2f7f6a.gif
Joined: Mar 22, 2007
Post Count: 2324
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Poor crunching on this project

Sorry, I meant Virtual Machine, not Virtual Memory.

In Windows, if you have a high amount of memory, if you match your Virtual Memory to physical memory, it should speed things up.
[Oct 28, 2010 5:40:28 PM]   Link   Report threatening or abusive post: please login first  Go to top 
X-Files 27
Senior Cruncher
Canada
Joined: May 21, 2007
Post Count: 391
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Poor crunching on this project

Intermediate data is building... I'm hopeful to bring all sciences back to the 97-99%+ and CEP2 to 95% instead of 90-92%. The WTD extending helped already quite allot.


My CEP2 usage:
I7 with 6GB = 95%-97%
Q6600 with 2GB = 90-92%

Sek, what's the amount ram on your rig? With your numbers and mine it seems more ram means higher efficiency.
----------------------------------------

[Oct 28, 2010 6:05:51 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Sekerob
Ace Cruncher
Joined: Jul 24, 2005
Post Count: 20043
Status: Offline
Reply to this Post  Reply with Quote 
Re: Poor crunching on this project

Usable by BOINC is 2.9GB on my Q6600. It's using 1.4 kind of constantly. The USB stick has all effectively 6.3GB allocated to BOINC of which 20% is used according BOINC stats tab... 1.27GB

More RAM for crunching has always been a speedupper. 90-92% pretty much is dictated by the disk IO and that's what I'm working to eliminate as much as possible... will doc a USB mem-stick trick that may help many.

Not heard back of that BFS kernel patch if it did anything for RaymondFO.
----------------------------------------
WCG Global & Research > Make Proposal Help: Start Here!
Please help to make the Forums an enjoyable experience for All!
----------------------------------------
[Edit 1 times, last edit by Sekerob at Oct 28, 2010 6:15:28 PM]
[Oct 28, 2010 6:14:48 PM]   Link   Report threatening or abusive post: please login first  Go to top 
codes
Advanced Cruncher
Joined: Oct 20, 2009
Post Count: 142
Status: Offline
Reply to this Post  Reply with Quote 
Re: Poor crunching on this project

Some more "sysstat" performance data. No significant disk I/O bottlenecks detected.


Last CEP2 job completed on this device:

CPU time: 12 Hours
Elapsed time: 12.4 Hours

<result>
<name>E200486_431_A.25.C21H13NOS2.127.3.set1d06_0</name>
<final_cpu_time>43200.006800</final_cpu_time>
<final_elapsed_time>44937.790886</final_elapsed_time>
<exit_status>0</exit_status>
<state>5</state>
<platform>i686-pc-linux-gnu</platform>
<version_num>619</version_num>
<stderr_out>

Job starts:

[07:26:42] Starting job 0,CPU time has been restored to 0.000000.
[07:29:07] Starting job 1,CPU time has been restored to 116.610272.
[07:36:38] Starting job 2,CPU time has been restored to 537.081350.
[10:47:09] Starting job 3,CPU time has been restored to 11198.435577.
[10:55:36] Starting job 4,CPU time has been restored to 11667.995193.
[11:03:05] Starting job 5,CPU time has been restored to 12091.264846.
[11:10:58] Starting job 6,CPU time has been restored to 12535.115370.
[11:18:36] Starting job 7,CPU time has been restored to 12967.189684.
[11:29:58] Starting job 8,CPU time has been restored to 13621.074278.
[11:37:24] Starting job 9,CPU time has been restored to 14046.812555.
[11:45:53] Starting job 10,CPU time has been restored to 14528.093389.
[12:06:55] Starting job 11,CPU time has been restored to 15741.876865.
[12:17:13] Starting job 12,CPU time has been restored to 16320.548893.
[13:06:41] Starting job 13,CPU time has been restored to 19164.929481.
[15:30:07] Starting job 14,CPU time has been restored to 27527.013251.
[17:39:39] Starting job 15,CPU time has been restored to 35087.966811.
Killing job because cpu time has been exceeded. Subjob start time = -266494131, Subjob current time = 1088496126
[19:55:25] Finished Job #15

Performance log entries (same time or very close to same) for job start times listed above:

# sar -u

07:17:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
07:17:01 AM all 0.03 98.22 1.76 0.00 0.00 0.00
07:27:01 AM all 0.06 96.53 1.58 1.30 0.00 0.53
07:37:01 AM all 0.03 91.82 7.06 0.22 0.00 0.87
10:47:01 AM all 0.03 93.81 6.15 0.01 0.00 0.00
10:57:01 AM all 0.02 92.82 6.23 0.13 0.00 0.79
11:07:01 AM all 0.02 94.64 4.78 0.20 0.00 0.37
11:17:01 AM all 0.03 93.87 5.42 0.17 0.00 0.51
11:27:01 AM all 0.03 95.86 3.66 0.02 0.00 0.43
11:37:01 AM all 0.03 95.79 3.69 0.05 0.00 0.44
11:47:01 AM all 0.02 94.22 4.61 0.21 0.00 0.93
12:07:01 PM all 0.02 95.95 3.50 0.16 0.00 0.36
12:17:01 PM all 0.02 94.12 5.70 0.15 0.00 0.00
12:27:01 PM all 0.03 98.20 1.17 0.34 0.00 0.26
12:37:01 PM all 0.02 95.36 4.50 0.12 0.00 0.00
01:07:01 PM all 0.02 97.90 1.62 0.01 0.00 0.45
03:27:01 PM all 0.03 98.41 1.55 0.01 0.00 0.00
03:37:01 PM all 0.02 95.99 3.50 0.00 0.00 0.49
05:37:01 PM all 0.02 98.44 1.54 0.00 0.00 0.00
05:47:01 PM all 0.02 93.57 5.60 0.36 0.00 0.45
07:47:02 PM all 0.03 94.01 5.96 0.00 0.00 0.00
07:57:02 PM all 0.07 91.93 5.71 1.73 0.00 0.57


The below averages are for all the performance recorded entries since midnight:

08:07:01 PM     CPU     %user     %nice   %system   %iowait    %steal     %idle
Average: all 0.03 96.05 3.77 0.07 0.00 0.08


Swap pages in/out:
# sar -W
08:07:02 PM  pswpin/s pswpout/s
Average: 0.00 0.00


Disk I/O performance:
# sar -d
08:07:02 PM       DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
Average: dev8-0 1.37 0.07 721.30 526.90 0.43 315.47 11.40 1.56


Memory performance:
# sar -r
08:07:02 PM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit
Average: 638499 1409089 68.82 259136 885776 351947 8.48

Interesting, %memused (68.82%) for all system/user processes is less than the 75% max memory setting in the BOINC Manager while in use. The CEP2 system requirements say "1,024 MB Memory Available". I wonder if I reduce system RAM from 2 GB to 1.5 GB what effect that would have on SWAP usage and any increased disk I/O activity bottleneck.
[Oct 29, 2010 1:33:23 AM]   Link   Report threatening or abusive post: please login first  Go to top 
mikaok
Senior Cruncher
Finland
Joined: Aug 8, 2006
Post Count: 489
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Poor crunching on this project

How come I only can have one CEP2 wu in progress at a time? Server refuses to send more, although when the one in progress finishes, the next one is downloaded confused
----------------------------------------
to infinity and beyond

[Oct 29, 2010 7:14:30 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Posts: 156   Pages: 16   [ Previous Page | 7 8 9 10 11 12 13 14 15 16 | Next Page ]
[ Jump to Last Post ]
Post new Thread