Index  | Recent Threads  | Unanswered Threads  | Who's Active  | Guidelines  | Search
 

Quick Go »
No member browsing this thread
Thread Status: Active
Total posts in this thread: 4
[ Jump to Last Post ]
Post new Thread
Author
Previous Thread This topic has been viewed 1141 times and has 3 replies Next Thread
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Improving efficiency

It was mentioned in the HPF2 section that

The techs want to add a feature that automatically detects computers that can't handle a particular type of task, and avoid sending them in that situation.


when talking about work units that error-ed out on certain computers, but why not take that one step further? Many people at WCG, myself included, are very competitive and try to maximize their contributions by monitoring their outputs by project and such: choosing the project that earns the highest point per day output. (note: as points are supposed to reflect beneficial contributions, this seems like a reasonable approach to me) Looking at that, it seems to me that it would not be to terribly difficult to make a way to follow averages for points per day and favor sending WUs to a computer that earns high marks for that project. (or at the very least have those statistics accessible to the user so that they could manually maximize their point count and thereby their contributions.)

It is important to note here that not all computers are equal in their strength at certain types of work. Some processors are more efficient at certain operation than others, not to mention the effect of the different amounts of cache, ram, bus speeds, etc. This may cause two computers sitting right next to each other to get different scores for different projects. For example, one computer may tend to do moderately well on CEP WUs, but the new HFCC struggles to get done, while another computer does HFCC just fine but struggles on CEP. As you might have guessed by now this is more than a hypothetical scenario. Both of these are at very close clock speeds with only slightly different amounts of ram (both over 1GB so not a factor for WCG). Many people will not go through the trouble of figuring out which is best for each computer, which will result in inefficiencies. The differences are hard to figure out just sifting through data, but a computer program could easily do such calculations is seconds.

Doing this manually could make the grid more efficient, while automatically would maximize such efficiency gains! cool


Hope that all makes sense!

Zach
[Mar 21, 2009 6:36:14 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Improving efficiency

Hi Otis,
I've been with WCG since 12/25/08. My (anal) accounting backround has hassled with this same situation. By Trial/Error and (probably way too much..) attention to completion times/points/etc. I've managed (i think) to maximize the output. Single processor machines up to 1.6Ghz seem to thrive on Rice and Muscular Dystrophy (with programmed cut off points), while it takes a dual processor w/1+Gig RAM to really 'get down' on the CEP, and other more complex studies. I have a 733Mhz and two old 566Mhz eMachines(both donations) that I increased RAM to the max (384meg) that crank out Rice units 21 of 24hrs regularly. Should hit Ruby-Rice by end of October. My 3 dual processor machines, meanwhile, work out on MuscDystrophy, Cancer, ChildCancer, CEP.
Just thought you'd like to know. I concur with your idea of having BOINC-WCG internally compute the best study for a certain machine. Would SAVE a ton of math/observation on our part.
Regards and Happy Crunching,
JR (ol43)
----------------------------------------
[Edit 1 times, last edit by Former Member at Sep 28, 2009 1:53:16 PM]
[Sep 28, 2009 1:49:26 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Sekerob
Ace Cruncher
Joined: Jul 24, 2005
Post Count: 20043
Status: Offline
Reply to this Post  Reply with Quote 
Re: Improving efficiency

The data is provided and updated frequently through the downloadable http://www.worldcommunitygrid.org/boinc/stats/host.gz file [very big in decompressed form]. An example of the information on 1 device of is:

<host>
<total_credit>16393.197724</total_credit>
<expavg_credit>0.076480</expavg_credit>
<expavg_time>1158057017.453430</expavg_time>
<p_vendor>AuthenticAMD</p_vendor>
<p_model>AMD Athlon(tm) XP 1700+</p_model>
<os_name>Linux</os_name>
<os_version>2.6.9-34.EL</os_version>
<create_time>1130851159</create_time>
<rpc_time>1152655444</rpc_time>
<timezone>-21600</timezone>
<ncpus>1</ncpus>
<p_fpops>759071089.284725</p_fpops>
<p_iops>1297318240.976990</p_iops>
<p_membw>1000000000.000000</p_membw>
<m_nbytes>528044032.000000</m_nbytes>
<m_cache>262144.000000</m_cache>
<m_swap>1073733632.000000</m_swap>
<d_total>39237181440.000000</d_total>
<d_free>31422160896.000000</d_free>
<n_bwup>38598.000000</n_bwup>
<n_bwdown>0.000000</n_bwdown>
<avg_turnaround>180410.906113</avg_turnaround>
<credit_per_cpu_sec>0.001190</credit_per_cpu_sec>


With this you can knock yourself out generating a matrix of CPU/OS/Credit second. Some sites such as BOINCstats already compile CPU/Hosts type RAC information from this.

Let us know what you come up with.

cheers
----------------------------------------
WCG Global & Research > Make Proposal Help: Start Here!
Please help to make the Forums an enjoyable experience for All!
[Sep 28, 2009 2:11:13 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Improving efficiency

Wow, I had given up on actually getting a response to this, then i run across it today and have two!

Ok, so I believe this is the log WCG makes on each client computer if I am not mistaken. (feel free to correct me at any time) But what would need to be done to implement what I am suggesting would simply to put in some extra fields and make it look like


<host>
<total_credit>16393.197724</total_credit>
<expavg_credit>0.076480</expavg_credit>
<expavg_credit_HCMD2>0.076480</expavg_credit_HCMD2>
<expavg_credit_IADS>0.085645</expavg_credit_IADS>
<expavg_credit_HFCC>0.086415</expavg_credit_HFCC>
<expavg_credit_HPF2>0.065891</expavg_credit_HPF2>
<expavg_credit_FAH>0.074365</expavg_credit_FAH>
<expavg_credit_CEP>0.015243</expavg_credit_CEP>
<expavg_credit_DDDT>0.046538</expavg_credit_DDDT>

<expavg_time>1158057017.453430</expavg_time>
<p_vendor>AuthenticAMD</p_vendor>
<p_model>AMD Athlon(tm) XP 1700+</p_model>
<os_name>Linux</os_name>
<os_version>2.6.9-34.EL</os_version>
<create_time>1130851159</create_time>
<rpc_time>1152655444</rpc_time>
<timezone>-21600</timezone>
<ncpus>1</ncpus>
<p_fpops>759071089.284725</p_fpops>
<p_iops>1297318240.976990</p_iops>
<p_membw>1000000000.000000</p_membw>
<m_nbytes>528044032.000000</m_nbytes>
<m_cache>262144.000000</m_cache>
<m_swap>1073733632.000000</m_swap>
<d_total>39237181440.000000</d_total>
<d_free>31422160896.000000</d_free>
<n_bwup>38598.000000</n_bwup>
<n_bwdown>0.000000</n_bwdown>
<avg_turnaround>180410.906113</avg_turnaround>
<credit_per_cpu_sec>0.001190</credit_per_cpu_sec>


I believe it is WCG program that accesses this information correct? (or is it Bionics?)
If it is the prior, the code simply need to add some conditions to run the most efficient one. Like this computer for example would request HFCC first, and if no work is available or it is on a lower priority, request IADS. It would just keep going down the line until it got some work. In this example, the computer above seems to have a large problem with CEP WUs as it only has a 0.015243 points per hour output, so would request that last.

This could be worked in with the existing structure of WCG's servers, and only ask for the most effective pairing in the queue. This situation would still clear the queue, and would be more effective at scientific thorough put than the current method. (all assuming points are an indication of the amount of the science done.)

If all of the statements above are correct, I do not see that this would be that difficult. If you would point me in the right direction (and I have access to the code - is it public?) I would gladly try to implement something like this myself and then send it in for WCG approval - but I have a feeling WCG would not do that...
----------------------------------------
[Edit 2 times, last edit by Former Member at Oct 14, 2009 8:16:24 PM]
[Oct 14, 2009 8:14:41 PM]   Link   Report threatening or abusive post: please login first  Go to top 
[ Jump to Last Post ]
Post new Thread