One thing I have noticed is that the task usually runs until 60-70% complete before declaring all 16 steps done. I have never seen this exceed 70% complete at all. Is this why all the estimated times are off?

I initially thought that the estimated time took into account that it might decide to go beyond 16 steps. Maybe this part could be investigated.

Dear gb009761gb009761 and others,

I guess we haven’t explained the CEP2 time-limit and workunit strategy properly, so we’ll make up for it today.

WCG projects commonly have a cutoff at 8h or – if the science is sufficiently flexible - are designed to just run that long (e.g., one can divide a job into 8h segments if they are pieced together easily). This takes into consideration that users like to finish a task within a foreseeable timeframe. The 8h is what IBM empirically determined as a sweet-spot between user preference and computational efficiency (you also don’t want to run more incrementally than necessary).

CEP2 wus can be continued and pieced together, but the overhead and additional traffic renders it more efficient to either finish them up in one go or for us to complete wus of particular interest on our cluster. Since we have a bit more leeway as an opt-in project we increased the time-limit from 8 to 12h, which we believe is a good compromise between our scientific goals and what’s necessary to keep the WCG participants (from your average Joe’s PC to a semiprofessional farmer) happy. There is actually a soft time-out after 10h (i.e., no new job gets started) and a hard time-out after 12h (i.e., the running job gets interrupted).

As discussed, our current ‘type 1’-wus (with the ‘set1d06’-extension) consist of 16 discrete jobs performed on a candidate molecule. We use a projection technique which makes this sequence of jobs very efficient. The time needed for each molecule primarily depends on its size and how complicated it is (there are a number of secondary factors). The molecules are becoming increasingly bigger in the course of the project.
The current ones are still small enough that most wus can actually be completed but as their size increases in the future only the first few jobs of a wu will finish.

From the 16 jobs the first 3 are most important because they provide the basic information; up to #8 the jobs have a good quality/cost-ratio so they are very attractive; after #12 jobs become much more demanding. These later ones are more sophisticated, give a more detailed and accurate description of the candidate molecule but we can do without them if necessary (they are the icing on the cake).

While incomplete results are perfectly fine for us, we certainly would not mind having a user option to increase the time-limit and get a larger number of completed wus. We already pitched the idea to IBM and will see whether or not it is doable.

A few more comments to points that were raised in this thread:

1) The cutoff has really nothing to do with bandwidth or storage issues. Later jobs actually don’t produce as much additional output as the essential first ones.

2) The validation is based on results which both wingmen have reached, but the more extensive results are kept (e.g., user1 made it to job15, user2 to job4; results up to job4 are compared to determine whether both runs were valid; if that is the case, the results from user1, i.e., up to job15, are kept). So nothing goes to waste here. Hence it is actually not that bad if someone only gets through job 2 or 3, since the wingmen probably makes it further (the slow one is then only used for validation).

3) There is some overhead lost if we have to finish up an interrupted job. It is not terribly bad and in the long run hard to avoid (otherwise we would have to cut the wus to always stay below 12h but then lose the benefit of the projection method).

4) Credit is awarded based on the time on the wu, regardless of whether it was finished or not (so this should be no reason for a deal breaker).

5) The completion-bar is unfortunately a rough tool since it is very hard to reliably predict how long a wu actually takes.

In summary, the time-limit is really all about being more attractive for users. It is a compromise we can live with and which does not overly impede on the science or efficiency. But we are happy to work with IBM to make it more flexible for people who like to run wus longer. For the default however we have to consider the whole range of WCG users and the 12h limit seems reasonable.

Thanks again for bringing this up and supporting our project.

Best wishes from

Your Harvard CEP team


P.S. We hope to publish some stats early in 2011.

Thanks cleanenergy for coming back on this query - it's good to hear that the earlier steps are the most critical (and hence, the ones most users should be able to process), with the latter helping to 'ice the cake'.

I, for one, certainly wouldn't mind these WU's running longer than 12 hours (or, has been suggested), cutting off at the next convenient checkpoint after that 12 hr limit, as I've seen numerous examples of WU's get 1, 2, and sometimes 2 1/2 hrs into the last step for it to simply be terminated - when, perhaps it only needed another 15-30 minutes to finish the job completely.

Cleanenergy --

Thank you very much for the details on CEP2 WUs. Very interesting and useful information!

I'll go ahead and sign even my slow machine up to do 1 CEP2 WU at a time now that I know that it is helpful even if only the first 3 jobs are completed.

Kate

I wonder what that does to the memory requirements. I am presently using a RAMDisk that is 5.8 GB in size for the Program Data/BOINC folder, which is fine for a quad-core running 4 work units at a time. (They actually never use above 4 GB, but I have to allow additional space to provide for downloads, which requires at least 2 GB free space.) But the system specifications for CEP2 state that 2,048 MB is required per work unit for storage, plus 1,024 MB more for running the program. So will the storage requirements get larger as the molecules get larger? How soon will that be? It is relevant to me for upgrading my system, and a month's notice would be helpful.