Who's on first
CJSL
Gotta keep crunching...

First, to dispel a misconception: having an average runtime of "x" and a cache smaller than "x" DOES NOT mean that you get no units. I have a machine right now that has cache settings of store at least 0.01 days of work, and store an additional 0.1 days of work, that is crunching an ARP unit right now. That machine alone has crunched at least 4 units in the past several days. That machine has taken between 15.89 and 18.3 hours to complete and return units (which is obviously longer than 0.01 and .1 days). So, the idea that low cache numbers results in no units simply isn't correct.

Second, I feel like there's a difference in the value of "reliability" from the viewpoint of the project versus the viewpoint of some users and I don't think there's anything wrong with that. It seems like a lot of people aren't looking at it from the researchers' perspective. I understand the need (in some limited cases) or desire to cache units to crunch later. If you have intermittent internet, or a bandwidth cap, it allows people to still contribute to WCG when it's convenient. But there's always a tension between convenience ("we don't need the result right away, so we can wait a couple extra days") and what will make the project unworkable ("we've been waiting on this unit for x days now and it's holding us back from creating the next batch of units").

It sounds like they use "reliable" computers to primarily crunch in the second kind of case: re-sends on errored out, no response, aborted, or too late units. There, time would be of the essence. So what's wrong with WCG creating a definition of reliability based on "this computer makes the project run smoothest"? They are now "late" and at risk of slowing down the project, so it would be to the benefit of the project to send them to the hosts that have proven a consistently fast (or, in another word, "reliable") turnaround time. If it is redefined to allow hosts that do not crunch them immediately and quickly then that could slow down the project and these slowdowns compound quickly over time.

It's not a month into ARP, so there's no issue with testing things out to see what runs most smoothly, especially since they are not fully ramped up. But we should keep in mind that WCG is looking at the projects on a different level than the crunchers, and they may have different priorities and viewpoints. There needs to be some deference to those priorities. And to address an analogy by Mike.Gibson above, we are not the customer. The researchers are the customer. We, the crunchers, are a supplier of a good (computer power) that we donate to WCG, which is given to the customer. It's always good to know where you are in the supply chain

Are we seeing our first drop off on WU? HSTB Redux... down to 828 WU yesterday. I am running on fumes to get to gold.

Well said DrMason. I don't understand the first paragraph though:


I re-read the last page or two of posts to make sure, but I haven't seen anyone assert that low buffers means no ARP1 tasks, so it appears like you created and responded to a strawman to me. I know you didn't quote anybody, but can you please point me to somebody who made the argument that "low cache numbers results in no units" in case I missed someone taking that position?

At least my position is that small buffers require more frequent babysitting, meaning if I fill a 1 day buffer with MIP1 work then switch to ARP1, I would have to check every 1 day in order to make sure the device is 100% full of work instead of sitting idle. I prefer not to micromanage all this, hence why I set the period to 3 days, which means I fill up 3 days' worth of MIP1 work and whatever ARP1 work it receives is simply icing on the cake. But at least I don't have to babysit my devices except every 3 days.

DrMason said:

Gotcha dude! I didn't read back far enough. I guess that post confused me too, so sorry for dropping the strawman thing on you and getting all debatey.

Yeah, Mike's post I believe focused more on meeting the requirement for "reliable" fast turnaround time, which is hard to achieve on a really old system. My oldest system takes 36 hours to do a ARP1 work unit. My fastest knocks them out in about 12 hours.

DrMason said:

Yeah, I started out preferring a 0.1 day cache -- that way I get fresh work and turn it around immediately. My computers were super reliable and got a ton of repair work. But with such a tiny cache, sometimes the WCG maintenance window is 4 hours so that wouldn't cover it. And I've lost Internet for 1-2 days here so settled with 0.5 days then now 1 day cache. 1 day is kinda my sweet spot.

And yeah, even in a nice neighborhood here, every year we'll wake up and the neighborhood cable Internet node boxes are busted open and wiring or equipment is stolen. I mean, I hate Comcast with a thousand suns, but stealing Comcast gear knocks out Internet for a whole neighborhood.

DrMason said:

Oh wow, that's interesting. I'm aware of the L3 cache issues with Rosetta/MIP1 and maybe other projects, but I'm not optimizing MIP1 at this point.

My current system is mostly "ARP1 100% if possible," but since supply is scarce as the project ramps up, I just fill up with other stuff then untick the boxes so that only ARP1 is selected which gives me a 100% chance that the clients ask for ARP1 work instead of getting spread thin.

Sorry for misreading you man, and thanks for sharing your system on how you balance your workload.