Index  | Recent Threads  | Unanswered Threads  | Who's Active  | Guidelines  | Search
 

Quick Go »
No member browsing this thread
Thread Status: Active
Total posts in this thread: 42
Posts: 42   Pages: 5   [ 1 2 3 4 5 | Next Page ]
[ Jump to Last Post ]
Post new Thread
Author
Previous Thread This topic has been viewed 5701 times and has 41 replies Next Thread
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Let's discuss this one more time

I know WCG has the statistics about which machines tend to return results within a 24 hr period on a continual basis.

It would likely be beneficial if there was a separate queue of WU's for each project that are specifically for machines that return the WU's within 24 hrs and dispatch those WU's only to the machines returning results very fast. Leave the default return date of 12 days / 14 day / 3 months as it is now don't change that.

It may not benefit the scientist, but it would reduce the backlog of validated wu's for those that are always waiting for others to return the WU's.

As it is right now; I've got a backlog of 45 WU's (and growing) that need validation. Getting the WU's validated as soon as possible using machines that have same day response, also has the opportunity to change the overall complexion of the WGC accomplishment graphs which are produced.

Anyway, just my thoughts.
[Jul 2, 2009 9:58:49 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Let's discuss this one more time

Here is knreed's most recent post in this area: http://www.worldcommunitygrid.org/forums/wcg/viewthread?thread=25998#236316
Lawrence
[Jul 2, 2009 11:25:45 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Let's discuss this one more time

Lawrence,

Thank you for the link in your reply. After reading the information in the link, it appears I failed to be as clear and precise in my prior post as I really needed to be.

Please permit me to try again with my position.

The observation was not for any particular project as I believe the post you referenced presented.

In my post, I had intended to supply sufficient information where the methodology, not necessarily a technique, could be designed and possibly implemented such that all WU's could fall into separate queues specifically to be dispatched to machines which return WU results within a 24 hour period.

This method could easily be used for high priority dispatching of WU's within a specific queue of WU's for a particular project. There appears to already be a mechanism in place for the express purpose of having the necessary WU's and the requisite validation WU's to be sent out for immediate processing forming quick turnaround. My suggestion, while similar is, I believe, just a tad different. As I see it, some set of WU's and their associated validation WU's would all be placed into this queue for nominal WU completion and return for validation to be completed within a 24 hr period.

There are some benefits for this type of implementation. Many folks, will stop carrying long queues of WU's that need to be completed. In other words, I suspect many will reduce their number of "Additional Days" of WU depth.

As I indicated before, I can only suspect this has an opportunity to change the complexion of how the entire WGC behaves in complete through put of WU's.

Here's my oldest WU in pending validation:



Here's my most recent WU in pending validation:



Now; if there were a queue for machines that returned WU's typically within 24 hrs were available and all the WU's listed in the WU status as shown in the above; I suspect the throughput of the WU's in general would;


  • show an overall improvement in WCG throughput (being defined as all WU's in a batch being verified quickly)
  • provide results back for analysis faster (presuming some things)
  • almost eliminate the "pending validation" for machines that respond within 24 hrs


Personally, I have my system set up so that it only starts to look for additional work just about the time a WU is about to complete, something like about 7 minutes of WU completion.

Does that present a better explanation about what I would like to see?

Thanks
---Barney
----------------------------------------
[Edit 1 times, last edit by Former Member at Jul 3, 2009 12:41:45 AM]
[Jul 3, 2009 12:06:04 AM]   Link   Report threatening or abusive post: please login first  Go to top 
gb009761
Master Cruncher
Scotland
Joined: Apr 6, 2005
Post Count: 3010
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Let's discuss this one more time

Personally, I think the crux of the matter is - being patient...

At times, we've all been eager to get x number of WU's validated (for whatever reason - although I'm guessing, mainly for "The Badge"), yet, once a WU has gone into PV status, I think it's very rare indeed for a WU to not be validated at some stage in the not too distant future.

Just from a technical point of view (and please, those who have far more knowledge/experience than I do on this matter, please feel free to jump in and correct me), the reissuing of WU's out to the 'reliable' computers, is done automatically, for a certain set criteria (e.g., a 'No Reply'). If the techs are going to have to figure a way of generating yet another series of 'quick return' queues, how many 'hoppers' are they going to have to ensure are filled? We're already having the odd occasional 'empty hopper' situation - just think what it'd be like when that number of hoppers were doubled over night... As we've seen from other threads, there are hoppers for each platform, project, CPU chip-set etc., and therefore, for 1 project (of which, as you know, we've got 9 currently up and running), there could be a dozen or so 'hoppers'...

So in summary, I think it's a nice idea, although rather impracticable - especially when all that's needed, is a bit of patience wink

Edit: Also, isn't there then a danger, that, due to the volume of WU's requiring a 'reliable computer', that they themselves get flooded out with WU's and become 'unreliable' - therefore, defeating the purpose?
----------------------------------------

----------------------------------------
[Edit 1 times, last edit by gb009761 at Jul 3, 2009 1:14:55 AM]
[Jul 3, 2009 1:11:00 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Sid2
Senior Cruncher
USA
Joined: Jun 12, 2007
Post Count: 259
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Let's discuss this one more time

Personally, I think the crux of the matter is - being patient...

So in summary, I think it's a nice idea, although rather impracticable - especially when all that's needed, is a bit of patience wink




If the system can reward the quick turnaround users with Betas. . . why couldn't the same system muster those same users together to get work turning faster?

I believe the crunching experience is enhanced with work turning in two days instead of two weeks.
----------------------------------------

[Jul 3, 2009 1:58:08 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Greg Lyke
Advanced Cruncher
Joined: May 30, 2008
Post Count: 50
Status: Offline
Reply to this Post  Reply with Quote 
Re: Let's discuss this one more time

I think that the idea is a good one, but like gb said, not really practical.

What I would be interested to see is how many people (as a percentage of users) have a full 10 days worth of projects in their queue. And then of those, how many of them connect to the internet every 1-3 days (or are constantly connected). If they are connecting fairly often then they really don't need a filled up 10 day queue do they? If there aren't that many people using the full 10 days who really need that amount of time, perhaps lowering the buffer down to 7 days is something to try.

Another alternative is only allowing a maximum of two days worth of work units per project at any one time (so if you wanted a 10 day queue buffer you would have to choose 5 different projects). While some people would be waiting/wingmen for those projects that are crunched later in the 10 day cycle, the total number of wingmen waiting should be reduced. This is because while days 1-2 of (for example) CEP subset 789 would be filled, he wouldn't be filling days 3-10 with 789's, those would be going out to different users (who may or may not have a fast turnaround). Some people would still have to wait for him to finish, but there wouldn't be as many waiting as in the current system.

Or you could just be patient & wait for everything to shake out in the end as it does now....
[Jul 3, 2009 2:57:28 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Let's discuss this one more time

Or take a trick from GPU grid. They give a 25% point bonus if the WU is returned within 24 hours... might not make everyone lower their buffer, but my buffer would definitely drop to less than 24 hours. cool

and I have a feeling it would make enough people change to make a noticeable impact...
----------------------------------------
[Edit 1 times, last edit by Former Member at Jul 3, 2009 5:12:01 AM]
[Jul 3, 2009 5:11:11 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Sid2
Senior Cruncher
USA
Joined: Jun 12, 2007
Post Count: 259
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Let's discuss this one more time

Or take a trick from GPU grid. They give a 25% point bonus if the WU is returned within 24 hours... might not make everyone lower their buffer, but my buffer would definitely drop to less than 24 hours. cool

and I have a feeling it would make enough people change to make a noticeable impact...



----------------------------------------

[Jul 3, 2009 8:04:32 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Let's discuss this one more time

Otis11
and I have a feeling it would make enough people change to make a noticeable impact...

And that in and of itself is the real crux of the matter.

As my images above easily demonstrated; its not about "being patient" it certainly revolves around the demonstrable benefit in WU throughput.

It's not about the badges, it is about getting WU's:

  • Processed quickly
  • Returned quickly
  • Validated quickly

So those with the big brains that can use the information for the purpose of mankind. (that as it is these days).

Seemingly it does not benefit anyone for WU's that are completed within hours of initial dispatching sitting and awaiting validation by others for 2+ weeks. Awaiting validation within a 24 hour period seems to be reasonable. WCG must know which machines return WU's within the shortest time periods. So creating a specialized queue containing the entire WU batch and putting that entire batch within the "fast Queue" seems a reasonable and responsible thing to request for numerous reasons.

Now, I'm certainly not suggesting that if machines suddenly start moving over, they be immediately placed into list of fast responders. This is something that requires a consistent track record, I don't know, say 10 consecutive days of returning results within 24 hrs?

Turn your machine off whilst you go on holiday for say a week; and you must again certify your machine as a fast responder? These are just thoughts and not truly intended to be implementation perspectives. The masters that run WCG have much better insight to this than I, a simple mortal.

When we had single core systems, that could hardly get a WU returned in 24 hours to 48 hours, the 7 / 12 / 14 / day return validation methodology appears to have been a sound and reasonable decision which ensured the most WU's were returned completed which eliminated copious re-work of WU's.

Today, many are returning WU results using unbelievably fast, multi-core CPU's tied to systems with almost incomprehensible quantities of memory; HDD space; and actual CPU processing capacity not available only a few years prior. These technological advances have been the single biggest contributor to being able to return many WU's within a 24 hour period of time. In my mind's eye, as it relates to not having a "queue for 24 hour turnaround" seems strikingly similar to one justifying using a howitzer as the appropriate tool to kill unwanted mice. biggrin

As an aside, while I can't say this with any real certainty, one could argue the point, should WCG implement a "Fast Queue" response for each and every project, additional projects regardless of size could benefit in reduced time to completion of the complete project.

BTW, I just looked at my PV queue.

When I started this thread, I was at a PV queue depth of 45. I'm quickly approaching a PV queue depth of 60. Should the results continue to pile up like this, some possibility exists my simple little machine could have a PV queue depth of over 100 before long. Surely WCG can do something about getting those of us in PV purgatory out of this mess.

Wouldn't it be simply amazing if by the adoption and implementation of this seemingly simple change, a project could get all it's WU's performed say 6 / 7 months earlier than initially anticipated and a cure for something could be found in time to save someone that would have otherwise been lost? Dreamy eyed, Pie in the Sky, perhaps... but what if it worked out to be the one it saved was you, one of yours or one of your friends, would it still be "impracticable?"

So wadaya say WCG guys can you do something to encourage a seemingly simple behavioral change within the community?

(Hey Camera guy, you gettin' all this? Opps sorry, I thought this was a Sham-Wow Commercial! biggrin )
----------------------------------------
[Edit 4 times, last edit by Former Member at Jul 3, 2009 8:42:49 AM]
[Jul 3, 2009 8:14:03 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Sekerob
Ace Cruncher
Joined: Jul 24, 2005
Post Count: 20043
Status: Offline
Reply to this Post  Reply with Quote 
Re: Let's discuss this one more time

Need more input XJ25?

Batches are generally not moved through the assimilator until 100% complete. Just a few 'reliable' devices to fall over and the gain is out the window, unless the deadlines are reduced accordingly.

What's a batch to a scientist if he needs a series of batches in an experiment? At > 90% already returned within 4 days (many moons ago it said 96%), the relative gain is minimal for WCG and as knreed wrote a few months ago, eventually squeezing the last 0.1% out becomes a disproportional effort. Disk storage is cheaper.

What's moved to the assimilators is removed from the public facing database, so better look quickly on your RS page. Other projects you frequently don't see completed work beyond 24 hours, here you see it for about 4 days after validation, at least and most of the time.

Separate,

How much work should be allocated to 'reliable' and where does the work go of the normal, no rush crunchers' repair no reply? Already there was a backlog developing for those tasks, so 'reliable' was increased to at least 2 days... that is, with my 2 day cache and rDCF of 1.5 am still getting them. Initial rush deadlines are now 40% of the original, speak 4 days for most projects.

Oh and Sid2, no quick returner is rewarded with Beta's. That's been a very long misunderstanding. Every client enrolled and contacting the feeders for more work at the right moment gets them! If there is one or the other FAQ around still saying so, let me know which and it will be amended.
----------------------------------------
WCG Global & Research > Make Proposal Help: Start Here!
Please help to make the Forums an enjoyable experience for All!
[Jul 3, 2009 8:39:06 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Posts: 42   Pages: 5   [ 1 2 3 4 5 | Next Page ]
[ Jump to Last Post ]
Post new Thread