inderdaad.... grapjas!

Hi E. Frijters,
Killing it was a good idea. We have multiple threads on this topic, which can get confusing. olympic posted just once in this thread, back in February. Since then he has discovered that HPF2 can get 'stuck' but that starting it again from the last check point fixes the problem. He runs BOINC, so his fix consists of suspending the entire WCG project for several minutes in the PROJECTS tab, then unsuspending it. The HPF2 program picks up at the last check point and works to completion.

Lawrence

Some background and information on how HPF2 works on BOINC.

1) The goal of HPF2 is to generate about 33,000 structures predictions per gene. These predictions are then further analyzed by the researchers to identify the mostly likely structures for the protein. See the official description of the project for more info: http://homepages.nyu.edu/~rb133/Human_Proteome_Folding_Project.html

2) Each result returned by a computer contains between 10-35 predictions depending on a rough estimate of how tough the predictions will be to compute for the gene.

So on a given gene if we ask each computer to compute 25 structures, then it will take 1,320 results in order to generate the set of 33,000 structures. We could create one workunit for the gene and send it out enough times to make sure we get 1,320 results. However, with the way that BOINC works and this would create some inefficiencies in the system.

What we do is that we create a set of workunits that will generate the required number of results for the gene. Over the entire set of workunits for the gene, we need to generate an average of about 17.2 valid results per workunit. In this example where 25 results are being generated per result, this means that we will create 77 workunits for the gene - each of which needs to average 17.2 valid results with 25 structures per result. 77*17.2*25 = 33,110 structure predictions.

If we have a gene that is tougher and only 10 structures will be generated per result then we would create 192 workunits. 192*17.2*10 = 33,024 structure predictions.

In order to generate the average of 17.2 valid results per structure we only have three variables to modify:

1) Initial replicas sent out (initial_replicas)
2) The min quorum (min_quorum)
3) The min results in agreement within a quorum before accepting the result as valid (min_agreement)

The way that BOINC works is that for each workunit, BOINC will initially send out 'initial_replicas' initial copies of the workunit to be processed. As the results come back in, BOINC will wait until 'min_quorum' is reached before attempting to validate the workunit. During this phase, if a result is aborted or returned as an error, then an additional copy will be sent out. Once 'min_quorum' results are returned then validation is attempted. Validation is successful if at least 'min_agreement' results are determined to be valid. If there are not at least 'min_agreement' valid results, then all the results are marked as 'INCONCLUSIVE' and an additional result is sent out. Each time an additional result is returned, validation is attempted again. Once we have min_agreement valid results then the results will be marked as valid or invalid as appropriate and credit is awarded. Normally at this point BOINC would go ahead and 'assmilate' the result (this means to copy the results off to be returned to the researchers). However, we have modified the assmilator for HPF2 so that it will wait until the last result is returned or the result with the latest deadline has missed its deadline. Then the assimilator will collect all valid results returned to return to the scientists. Also - once the validation has been successful, no additional results will be sent out even if additional errors or invalid results are returned.

We currently have initial_replicas set to 19, min_quorum set to 15 and min_agreement at 13. This is yielding about 17.6 valid results per workunit. We want to keep it a somewhat above the 17.2 target to ensure that we have the required set of structure predictions.

Suspending and resuming WCG in the projects tab has worked for me so far to "unstick" HPF2's. You don't have to suspend it for long, I give it about 30 seconds.

It's a very strange anomaly indeed. It's completely random and other members of the quorum seem to complete the WU without issue. It's also rather rare. My machines process ~150 WU/day and sometimes they go many days without issue. 1 in 500-1000 WU would be my rough estimate of the "problem" WU's.