I think I will give my GPUs a rest over Xmas and New Year.....

We send and receive data to/from the researchers in batches. While a batch is in progress but does not yet have all of its workunits completed, we have to store it on our system and the researchers don't get the benefit of the data in them until the whole batch is completed.

The Mapping Cancer Markers batches have 10,000 workunits per batch. Within in each batch, there are what we call "hard luck workunits". These are workunits where the first couple of assignments go to computers that don't disappear (the owner goes on vacation, they reformat their computer, they uninstall, etc). Therefore it takes 7 days before the next ones are sent out. If left to its own devices, then these next two might go to someone with a large cache and they happen to accidentally unplug their machine and the job ends in an error. Then the next one goes to someone else who never returns it. And so and so on. When we first started using the BOINC software there were workunits that took longer than 90 days to complete due to "bad luck".

As a result, the first copies that are sent out and any additional copies sent out in the first three days are assigned to any computer that asks for them. However, after that time they are only sent to computers that have proven themselves to quickly return jobs correctly. They are also assigned a shorter deadline (30% of the original deadline). This allows most batches to finish in about 160% of the original deadline (1 full deadline and then two shorter deadlines).

However, there are only certain number of these "reliable" hosts. If there are more jobs requiring reliable hosts than there are reliable hosts asking for jobs, then job distribution gets 'clogged' and we are not able to efficiently distribute the work. As a result, we want to minimize the amount of these reliable jobs.

Recently, people have been spinning up a number of EC2 Spot Instances to contribute to World Community Grid. These are virtual machines where you can say how much you are willing to spend and if there is sufficient excess capacity on Amazon's cloud and if no-one else is willing to pay more than you, then your machines will be started and will start running what ever workload you provide. The catch to this is that if excess capacity on EC2 shrinks or if someone offers to spend more money than you do, then your instances are stopped and removed.

What this means for us is that 32 core virtual machines were starting up, requesting 5*32 jobs from us, and then being killed off and the results are never returned. All 160+ of those jobs then become "no result" jobs 7 days later and generate 160 jobs that need reliable hosts. Enough of these were being done that it was exceeding the grid's ability to assign all of those 'reliable' jobs to reliable hosts.

So in order to reduce the number of reliable jobs being produced, we are now limiting new devices to only 1*(number of cores on device) jobs for the first 3 hours of its existence. This will reduce the number of jobs assigned and quickly abandoned, but also allow devices that continue to participate be able to get plenty of work soon when they start to finish their first jobs. Note that if a computer completes and reports a job as complete during this three hour window, they will still be able to get a new job to work on - they just won't be able to build up a buffer until the three hour window expires.

I want to note that there are also a lot of these ec2 spot instances that are doing some significant work for World Community Grid so we do not want to hinder their use and involvement. We are just taking steps to ensure that this works for everyone.