report a multi-day average of 15 good results consecutively is the other criterion, if still valid.

With so less and long running tasks, one can imagine that there are no reliable hosts at all.

The main problem is not the criteria. They may not have been sending enough of them out to find the reliable machines. A lot have gone to people who push the update button often enough instead.

Good Morning,

Ok. So a few things to note, this is a multi part process. We have the Indexer, createWork and the feeder.

The indexer checks to see what we have that is not in the database yet, it places them in an indexed state when found. We usually like to keep about 10 days indexed for most projects. This project is a bit different, but we have plenty indexed at the moment (even some from generation 001).

The createWork is what loads it into BOINC databases. Since BOINC requires xml to be stored in the database, we usually keep this buffer at about 48 hours to assist in keeping the database quick. But again, for this project we do something a bit different...we load a set number of results per half hour to artificially slow the project down. (Again, we hope to get this up to full speed in the future, which would be 30+k workunits in the wild at one time. 60+k results since redundancy)

The Feeder is the last part, it grabs what has been loaded by createWork. it pulls in work units to fill it based on weights set on our end. Say the feeder has 1000 spots and we give a weight of 50 to arp1, 25 to scc1, 15 to hstb, 10 to mcm1. The feeder would try to fill 500 slots with arp1 work. This happens every 5 seconds it tries to fill the empty slots. Members when they do a scheduler request pull from this feeder. The feeder attempts to pull higher priority results in first and then based on time stamp. This means that reliable results get pushed to the top first.

The problem wasn't the feeder, the problem was that we had a backlog of reliable results needing to be sent. This caused the createWork to believe it had over 21 work units loaded already on the grid. (If you haven't guessed we are loading 21 workuntis every 30 minutes). If we had only 10 reliable results waiting to send it would have only loaded 16 work units for arp1.

I am still trying to think what the best option is, because loading more work to keep the flow going is desirable, however, I need to have the resends be sent out and returned positively. I'm trying to think of a happy medium between the two extremes that would still keep the system quick and provide new work units to members.

I hope my long winded explanation helps, if it doesn't....please feel free to ask for clarification. :)

Thanks,
-Uplinger

Thanks for the background.

"(If you haven't guessed we are loading 21 workuntis every 30 minutes). If we had only 10 reliable results waiting to send it would have only loaded 16 work units for arp1."

Suppose that could be 28 minutes, 33 minutes, 38 minutes, i.e. on average, something that you cant tune a scheduler to which was the whole point of 'randomized distribution'

I'm math challenged BTW... 10 + 16 = 21. Guess you added some fuzzy logic.

Keith,

Many thanks for the detail. I guess writing it down gave you an opportunity to think while you did it. My only take is not very useful: it sounds like a process that works OK for a single project research effort, but for a multi-project effort like WCG I think I would have taken things in a different direction. But that sounds too much like "If you want to get there I wouldn't start from here".

I think enough people have made comments for someone of your expertise to weigh up the different requirements and to have a reasonable chance of sticking something together that works. But I do agree that an automatic, gently sloped fall-back to more relaxed 'reliable' constraints would seem to be necessary, even if not easily implemented in this environment.

Good luck!