I'm noticing fewer retries on the batches of MCM units I've been getting recently. Also, one recent ARP download went far faster than before (but there were still some retries).

Not sure if more resources are being brought to bear, or if people are turning down/off ARP. I've dropped down to one ARP per host (letting the ones still in the queue run, though).

The theme of this thread has recurred many times in other thread.

The problem is that the downloads for ARP are absolutely huge by comparison with MCM for instance. Whereas MCM files for download are sized in Bytes, ARP files are sized in MegaBytes and there are 10 instead of 1, so about 1 million times the size.

Also Krembil is the research arm of a University Hospital network so cannot afford the infrastructure to solve the problem, whereas IBM is a huge multinational corporatiion specialising in IT and had the resources to cope with surges in demand.

Once supply and demand settles there should be a vast improvement in downloading.

Everyone who wants to work on ARP seems to have set very high cache levels so it takes a long time to satisfy them on a restart. There would be little problem if everyone reduced their caches to say 2 units and gradually increased that as downloading became less congested. A few days would see improvement.

Mike

Can't WCG limit the number of ARP tasks that are available?

It must be pretty clear now how much data the system can handle. So limit the number of ARP tasks to something a bit less than that.

Or am I missing something?

Bubba, you are about two years too late to this discussion. We have been through this already a couple of years ago.

Yes, Krembil bit of more than they can chew, and have to work with what they have. Limited funding WAS mentioned as one issue. And yes, there had been folks volunteering to help administration, but that was rejected out of administrative reasons.

And one point that you have completely missed is that IBM wanted to get rid of WCG, as one of their cost saving measures by their bean counters. IBM is for years now in dire straits, They are firing people left and right (even have lawsuit for age discrimination going on) and have kind of lost their way more than a decade ago.

Ralf

I totally agree, but I'm not sure that they'd actually be able to get a sane solution even then -- unless whatever settings they arrived at managed to spread out the work evenly across the entire duration of a task deadline, there will still be problems when the (inevitable?) large numbers of retries for missed deadlines crop up all at once...

Also, the optimal level might be such that work gets swallowed up by those who cache larger numbers of tasks (whether because they have lots of CPU threads available, because they want enough work to tide them over system outages, or they run other projects that need a multi-day buffer). The network may cope well at the settings WCG choose, but there might be a lot of frustrated volunteers who only want small numbers per day but never seem to get any.

I reckon any real solution will require more (and more powerful) infrastructure (at which point the lack of a magic money tree might be a problem, as might the [possible] inability of the hosting service to provide a lot more physical bandwidth!)

Cheers - Al.

P.S. I posted the link to that news item (without commenting on the content here) because it appears a lot of folks hadn't read it [and I'd commented in detail there :-) ...] I've done the same in at least one other thread :-)