Index  | Recent Threads  | Unanswered Threads  | Who's Active  | Guidelines  | Search
 

Quick Go »
No member browsing this thread
Thread Status: Active
Total posts in this thread: 14
Posts: 14   Pages: 2   [ 1 2 | Next Page ]
[ Jump to Last Post ]
Post new Thread
Author
Previous Thread This topic has been viewed 3019 times and has 13 replies Next Thread
krakatuk
Advanced Cruncher
Germany
Joined: Oct 3, 2008
Post Count: 141
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Why don't we work on projects sequentially?

A long discussions about the equal share of resources between WCG projects in another thread (http://www.worldcommunitygrid.org/forums/wcg/viewthread_thread,31751_offset,0) inspired me to bring this idea, which I think would help us to make the world to a better place even much faster! Please let me try to explain:

Now we have about 10 projects which are running in parallel. Some of them are intermittent, some have technical issues. But biggest part of them are running sharing resources pretty equally. Doing so every single project would need a couple of years to complete.

Why don't we take one project first, crunch it down with the whole power of WCG and continue with the next one? We would need couple of months instead of long years to complete one particular project. Most of sciences we are crunching for are needed ASAP and would safe lives - saving one year on our side we would save a lot of lives all over the world!

You would say - there are some "open-end" or very long running projects like FAAH or HPF - they would block WCG completely. But in those cases it's still possible to split such projects in parts - like experiments in FAAH are. Every part should have a target, which would bring a benefit for scientists and give them work for quite a while. I assume for such a project it's not a big difference, to have the work done slowly with constant speed or to have it done with high speed, but in chunks while having breaks between chunks.

What would be needed if we would go this way - is a group of independent experts who would decide, which project goes first. Ok, it could possibly lead to an ethic problem. In this case it can be done on a "First Come - First Served" base. Long-running project, which have just finished it's nexts chunk of work, would come in the end of the queue.

Another problem, which can be faced doing this approach - crunchers dedicated to only one particular project. For whichever reason they may be not happy with this approach even if their prefered science will benefit from it. They should have a possibility to crunch the project they want if this project has available WUs in the time between full-power runs. We don't want to loose those people.
At the moment we have badges as additional motivation for crunchers. A lot of people are crunching for a project till they get the badge they want and then go for a next project where they still don't have the wanted badge. But the main idea in badges is, that it's much more enjoyable (don't know if it's the right word) to go for a short-term targets than to crunch without seeing the light at the end of the tunnel. If we would have short-running projects, it would be additional motivation for people and would make more fun for everybody here. E.g. something like "Let's crunch the Leishmaniasis away in the next two months!"

What we else have - are hardware-intensive projects like CEP2 and projects bringing only a small amount of work time to time. Those projects I would still leave running in parallel to whatever is currently running.

Regarding software developing efforts for WCG - it's nothing to do here. Just give a highest feeder priority to the project which has it's "showtime" and low priority to all the other projects. As soon the work for this project (or project chunk) is done - give the priority to the next project in the queue. And so on...


This idea is based on the "agile" methodology (I'm pretty sure our techs now what it is). I'm working in software development sector and we are using this software development methodology for a couple of years now. We are using it now everywhere, not only in software development. And that brought the company a much better "time-to-market" value then it has ever had before. I think it would be a great improvement for WCG as well.
If you want some more information abot this methodology, you can start here:
http://en.wikipedia.org/wiki/Agile_software_development

It would be great to know opinions on this from WCG techs, community advisors and all you community friends!
----------------------------------------

[Oct 2, 2011 7:22:01 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Why don't we work on projects sequentially?

Why don't we take one project first, crunch it down with the whole power of WCG and continue with the next one? We would need couple of months instead of long years to complete one particular project. Most of sciences we are crunching for are needed ASAP and would safe lives - saving one year on our side we would save a lot of lives all over the world!

There are multiple reasons.
1)we would't actually savingt years,we would just be transfering compyutation years to annother.We would do the same work in the same time but we would just be prolonging shortiening certain projects

2)WCG as a non profit project hosting service can't say to projects and there scientist "You will have close to zero cpu time because we are doing (project Y name) exclusively".No onw would submit anything if there if the chanche that for two years they dont crunch a single of your work units.

3)In the project submition form there is a phrase in the project sumbition form(the RFP at submition ) that states WCG want to treat every reshearch fairly.
"Estimate the total calendar time duration of the project, based on being able to use approximately 25% of the current grid power per the global statistics on www.worldcommunitygrid.org/stat/viewGlobal.do."
If we would do every project subthequencely it would be imposible to gage so we could get the extremely short or extremely long procjects.

4)when a project end(see HCC in a couple of months)There is alway a backlog of invalids /errors that need to be processed after the original work is done.With every project beung subsequential it wouldbe very trucky for the servers to cope with different combinations.

5)If it is not broken why fix it.But if there uis a solution that is worthwile we should fix it.

Other cruncher will also give you other reasons as well.

edit: spelling errors
edit2: i am not opposed to change but only is a solution that is worthwile to implement we should fix it.
----------------------------------------
[Edit 1 times, last edit by Former Member at Oct 2, 2011 8:28:11 PM]
[Oct 2, 2011 8:22:18 PM]   Link   Report threatening or abusive post: please login first  Go to top 
anhhai
Veteran Cruncher
Joined: Mar 22, 2005
Post Count: 839
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Why don't we work on projects sequentially?

this has been brought up before or at least something similiar. I myself was wondering if we should limit the number of active projects. However, the people at WCG give me several reason why it is not really needed. Below are a few that I remember.

1.) more projects usually attract more people
2.) as we crunch, the WUs are returned to the scientist and they are actively working on analyzing the results. So there is progress being made, and in some cases they modify future WUs base on the results. If we increase the results return to them by 6-8 fold, the scientist may not be able to analyze them any quicker.
3.) projects may not be able to generate 6-8 fold increase in WU
4.) it is more efficient for a system (your computer) to run different WUs

there are more, but that is what i remember
----------------------------------------

[Oct 2, 2011 8:28:55 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Why don't we work on projects sequentially?

Thanks anhhai,
1.) more projects usually attract more people
2.) as we crunch, the WUs are returned to the scientist and they are actively working on analyzing the results. So there is progress being made, and in some cases they modify future WUs base on the results. If we increase the results return to them by 6-8 fold, the scientist may not be able to analyze them any quicker.
3.) projects may not be able to generate 6-8 fold increase in WU

Very few of our projects have a known body of work at the start. So (2) is extremely important. Some of the projects add only a little work as they go along, but some, such as FAAH or HPF, are almost open-ended. (3) is almost as important. Many projects scramble to keep adding to their server capacity as results keep streaming in from WCG. Occasionally we have to slow them down until they can get an addition to their grant to buy more capacity.

Lawrence
[Oct 2, 2011 11:13:39 PM]   Link   Report threatening or abusive post: please login first  Go to top 
krakatuk
Advanced Cruncher
Germany
Joined: Oct 3, 2008
Post Count: 141
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Why don't we work on projects sequentially?

Thanks anhhai & Lawrence,

it's really surprising me, that we have already reached the state where not computer-power is the main bottleneck, but human- and other resources on scientist's side. Have to think about it...
----------------------------------------

[Oct 2, 2011 11:40:57 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Ingleside
Veteran Cruncher
Norway
Joined: Nov 19, 2005
Post Count: 974
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Why don't we work on projects sequentially?

it's really surprising me, that we have already reached the state where not computer-power is the main bottleneck, but human- and other resources on scientist's side. Have to think about it...

A few other reasons for not running only 1 project at a time:

User-side of things:
1: Not all computers meets the memory-requirements, disk-requirements or network-bandwidth-demands of some of the projects.
2: Anyone still with non-Intel Mac's, not all projects has any application for them.
3: Infrequent checkpointing coupled with usage-pattern can lead to little or nothing done with some of the projects (especially CEP2).
4: For some unknown reason, a project gives nothing but computer-errors or fails validation, while other projects is problem-free, and running memtest and cpu-test-programs doesn't show any errors.
5: Some users is only interested to run tasks taking around 1 hour, they won't run 10+ hour tasks.

Project-side of things:
6: A project with large downloads and/or uploads can exceed current disk-capasity or network-capasity.
7: The shorter the run-time of tasks, the higher the server-load. The server-load isn't linear, so shorter tasks increases both the chance of can't keep up, and the chance for errors and this gives outages.
8: If single project, any multi-batch errors can stop all progress. With multiple projects, only a single project will stop while majority of production can carry on unhindered.


It's a big advantage for WCG that not just the most powerful computers can contribute, but also the computers with little memory or little free disk-space. Also computers with mediocre network-bandwidth or monthly usage-caps can contribute. Also giving users a choise then it comes to run-times is an advantage, since not all users is interested in running the longest tasks.

If WCG only had a single "Project of the month" or something, chances are many of the users either couldn't, or wouldn't participate, and many of these users will either just quit or find another project. In either case, even if they're interested in next "Project of the month", getting these users back running WCG is very difficult.

Users having multiple projects to choose from is probably WCG's biggest advantage, so changing this would be a bad idea.
----------------------------------------


"I make so many mistakes. But then just think of all the mistakes I don't make, although I might."
----------------------------------------
[Edit 2 times, last edit by Ingleside at Oct 4, 2011 9:46:04 AM]
[Oct 4, 2011 9:43:13 AM]   Link   Report threatening or abusive post: please login first  Go to top 
krakatuk
Advanced Cruncher
Germany
Joined: Oct 3, 2008
Post Count: 141
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Why don't we work on projects sequentially?

Ingleside,

you are completely right. All your points are valid, but only for strictly sequentional approach.

What I'm suggesting is more a "semi-sequentional" way -
the idea is not to run ONLY ONE project, the idea is to run ONE project with high feeder priority and all the other with low feeder priority.
With other words, WCG server would send more work for one project to all the people who have chosen to run all projects.

Means, people who want to crunch on some other project would be able to do so. Only those crunchers who have selected many projects including the project which is currently having it's "high-prio" phase, would work almost only on this particular project.

We would leave everything as it is, but always run one project whith high feeder priority - turbo mode wink . And every project would get it's "high prio" phase after some time.


Just give a highest feeder priority to the project which has it's "showtime" and low priority to all the other projects. As soon the work for this project (or project chunk) is done - give the priority to the next project in the queue. And so on...

----------------------------------------

[Oct 4, 2011 12:05:26 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Why don't we work on projects sequentially?

...and there is also the case that Users probably prefer to choose, not let themselves be dictated to. I know for certain that there are some users who will only crunch Cancer, for personal reasons.

It aint bust so don't mess with it. Each User is able to set their own priorities which is the ultimate freedom of choice, rather than be forced to conform! wink
[Oct 4, 2011 12:42:06 PM]   Link   Report threatening or abusive post: please login first  Go to top 
krakatuk
Advanced Cruncher
Germany
Joined: Oct 3, 2008
Post Count: 141
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Why don't we work on projects sequentially?

...and there is also the case that Users probably prefer to choose, not let themselves be dictated to. I know for certain that there are some users who will only crunch Cancer, for personal reasons.

It aint bust so don't mess with it. Each User is able to set their own priorities which is the ultimate freedom of choice, rather than be forced to conform! wink

Fully agree with you, Alan! Users have to have choice - it's not a question at all!

And if you think about it, my approach would bring a big advantage for the people who are only crunching on one prefered project:

- Now all the WCG resources are equally shared between projects - if sombody is crunching a lot for one project, server will balance it by giving more other project's tasks to other users, who select to run many projects.

- If we would have one "high-prio"-project a time, we would have following situation:
If a project has it's "high-prio" phase, it has similar resources all the other project have when they have their "high-prio"-phase. When a project is currently not in "high-prio", it would have only recourses from people exclusively supporting this project.
Everybody who is crunching the project when it's currently not in "high-prio" phase, would bring a real additional advantage to this project.
----------------------------------------

[Oct 4, 2011 1:19:07 PM]   Link   Report threatening or abusive post: please login first  Go to top 
rembertw
Senior Cruncher
Belgium
Joined: Nov 21, 2005
Post Count: 275
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Why don't we work on projects sequentially?

My € 0,02 wink

As Lawrence said, your suggestion might create big problems on the scientist side.

On the other hand, there is nothing to stop you to make your own "project-of-the-month" by deselecting all projects (but one or 2 or ...), and selecting the "if no work..." selection. Which is, by the way, my current setting until I collect my sapphire in DDDT2. Although by now it is rather "project-of-the-year". biggrin

So, what you ask, you can select for yourself as individual, without forcing other crunchers to follow like lemmings and chasing them away towards other projects in the meantime. Also, if you create or join a team, it has been seen before that teams concentrate on a certain project for a time period. Even this may strain scientists servers if the participating power is large enough... I seem to remember that some time ago even 1 member caused havoc on WCG and project side (Kermit?) by adding and concentrating enormous amounts of computing power. I doubt WCG is interested in causing this chaos voluntarily to participating projects.

[Edit: Also don't forget, most members "set&forget", and are plenty happy to let WCG decide what distribution is best for the scientist side. After all, WCG knows (usually) best. I also seem to remember, that if a project needs a higher or lower share on occasion, that the techs can push the throttle the right direction.]
----------------------------------------
[Edit 1 times, last edit by rembertw at Oct 6, 2011 2:17:16 PM]
[Oct 6, 2011 2:10:42 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Posts: 14   Pages: 2   [ 1 2 | Next Page ]
[ Jump to Last Post ]
Post new Thread