Index  | Recent Threads  | Unanswered Threads  | Who's Active  | Guidelines  | Search
 

Quick Go »
No member browsing this thread
Thread Status: Active
Total posts in this thread: 59
Posts: 59   Pages: 6   [ Previous Page | 1 2 3 4 5 6 | Next Page ]
[ Jump to Last Post ]
Post new Thread
Author
Previous Thread This topic has been viewed 20600 times and has 58 replies Next Thread
Sgt.Joe
Ace Cruncher
USA
Joined: Jul 4, 2006
Post Count: 7846
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: 2023-08-17 Update (Weekend work unit shortage and OPNG issue)

Al, I think you are correct. My oldest valid work unit is from Aug. 15, but the wingman did not return the second unit until Aug. 21. All work units which have been validated with the wingman prior to Aug. 21 have now disappeared.

Those FL1-A units must be smaller because they are just flying through my Linux systems which are definitely not state of the art. On an I7-2600K they are taking anywhere from 30 minutes to 75 minutes. Even on my slowest system ( Xeon E5645) they are all under 2.5 hours, most less than 2 hours. Even on the Windows I7-3770 they are mostly around an hour plus or minus about 15 minutes.

Cheers
----------------------------------------
Sgt. Joe
*Minnesota Crunchers*
[Aug 22, 2023 11:42:16 AM]   Link   Report threatening or abusive post: please login first  Go to top 
TPCBF
Master Cruncher
USA
Joined: Jan 2, 2011
Post Count: 2173
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: 2023-08-17 Update (Weekend work unit shortage and OPNG issue)

P.S. If you have evidence of work units that are currently taking much more than 2 days to disappear after validation please publish some related work-unit numbers as I (for one) would be interested in seeing if there's a pattern amongst them... :-)

[1] Actually, my validated tasks have been disappearing consistently for quite a while -- it's not something that has just changed...

[2] I think it can take up to a day for all necessary result files to be saved after validation (at which point the WU is marked as ready for deletion); it then waits a day before actually clearing out the WU from the database. Hence the 48 hours or so...

Well Alan, you should know that I am crunching for WCG for more than just a couple of days, so I know how things should/used to work.... biggrin

As for for finding a pattern, well, that is a bit tough, at least on my side, for more than one reason. For one, any post that I try to make is delayed by well over 24h in almost all cases.
Second, WCG has hardly a track record of running consistently, which makes it hard to even establish a pattern.

I noticed the increased in over WUs even before the weekend, it became really noticeable starting Thursday.And yes, I noticed that some of the latest batch of SCC1 (not all of them!) are running maybe half the time "as usual" and that contributed to the increased number of WUs on the Results page. But I am convinced that this is not the ONLY reason.

As for those "waiting to be sent" WUs, this has been first mentioned shortly after the last "general outage" (streak counter now says 28 days, that's 4 weeks). As I mentioned before, when I checked, this seemed (for me, at least where I noticed it), macOS related WUs. Overall, those "Pending Verification" WUs has also increased to what I would call "disproportional". It used to be about a dozen WUs out of maybe 2500 WUs in Results at any given time, right now, it is 100 out of 4200.
"Pending Validation" is right now 256, which usually was maybe 150 to 200, so a not equally drastic increase.
But the most notable change for me is that with 1300 WUs "in progress", there are now 2400 WUs marked valid, which used to be in the past a 50:50 split.

The valid WUs go back now to 8/11, that's 11 days, and the oldest one is showing
Result name  OS type OS version Status Sent time Time due/ Return time Cpu time/ Elapsed time Claimed credit/ Granted credit
SCC1_0004202_MyoD1-C_37002_0 Microsoft Windows 10 Core x64 Edition, (10.00.19045.00) Valid 2023-08-10 20:40:24 UTC 2023-08-11 21:36:10 UTC 3 / 3 106.8 / 76.6
SCC1_0004202_MyoD1-C_37002_1 Microsoft Windows 10 Enterprise x64 Edition, (10.00.22621.00) Valid 2023-08-11 21:36:24 UTC 2023-08-21 13:00:57 UTC 4.11 / 4.11 46.4 / 76.6
SCC1_0004202_MyoD1-C_37002_2 Other


So, while I received and returned this WU within 25h, the wingman's WU has been sitting likely in some hoarder's cache for 10 days, and now we are stuck due to the system answering with adding an oddball "other" WU.
And then people got mad at me when I was calling out those hoarders as selfish and detrimental to the overall situation,.

Something is amiss, and it doesn't help the project in any form, if anyone (including WCG Towers and fellow crunchers) are just turning a blind eye and look the other way, just because "alles so schön bunt hier"... sad


Ralf

PS: it's 8:10am PST, let's see when this post is finally showing up,, and more importantly, there is an acknowledgment of an existing problem on WCG Tower's side
[Aug 22, 2023 3:13:59 PM]   Link   Report threatening or abusive post: please login first  Go to top 
TPCBF
Master Cruncher
USA
Joined: Jan 2, 2011
Post Count: 2173
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: 2023-08-17 Update (Weekend work unit shortage and OPNG issue)

Here's one of those "waiting to be sent" WUs, having my WU sitting in PV jail for +7 days (after the wingman failed to perform) now
Project name: Smash Childhood Cancer
Created: Aug. 8, 2023 - 18:55 UTC
Name: SCC1_0004191_MyoD1-C_59026
Minimum Quorum: 2
Replication: 2
Result name OS type OS version Status Sent time Time due/ Return time Cpu time/elapsed time
Claimed credit/ Granted credit
SCC1_0004191_MyoD1-C_59026_0 Microsoft Windows 11 Enterprise x64 Edition, (10.00.22621.00) No Reply 2023-08-08 23:49:40 UTC 2023-08-14 23:49:40 UTC
SCC1_0004191_MyoD1-C_59026_1 Microsoft Windows 10 Professional x64 Edition, (10.00.19045.00) Pending Validation 2023-08-08 23:50:06 UTC 2023-08-09 13:20:09 UTC 3.9 / 4.05 76.8 / 0
SCC1_0004191_MyoD1-C_59026_2 Waiting to be sent

[Aug 22, 2023 3:27:50 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Sgt.Joe
Ace Cruncher
USA
Joined: Jul 4, 2006
Post Count: 7846
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: 2023-08-17 Update (Weekend work unit shortage and OPNG issue)

And then people got mad at me when I was calling out those hoarders as selfish and detrimental to the overall situation,.

I would agree with you on hoarding being detrimental to good throughput. I am pretty sure a good portion of the hoarding comes from those who are afraid of outages and are trying to hedge their bets. If the supply of work units remains steady, that should lessen the incentive to hoard a disproportionate amount of work units. At the moment that is a pretty big "if."I think a cache of 2-3 days should be more than sufficient. If you run out of work units that is the purpose of a backup project. If your chosen projects run dry, take a vacation, unplug your machines, and just relax. I am certainly not going to obsess about something over which I have no control.
Cheers
----------------------------------------
Sgt. Joe
*Minnesota Crunchers*
[Aug 22, 2023 5:59:36 PM]   Link   Report threatening or abusive post: please login first  Go to top 
TigerLily
Senior Cruncher
Joined: May 26, 2023
Post Count: 280
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: 2023-08-17 Update (Weekend work unit shortage and OPNG issue)

Hi all,

Thanks for sharing this information and your insights. As mentioned, this has been forwarded to the tech team and I will provide updates as they become available.
[Aug 22, 2023 8:13:45 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Blount
Veteran Cruncher
Joined: Aug 19, 2005
Post Count: 587
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: 2023-08-17 Update (Weekend work unit shortage and OPNG issue)

Looks like the SCC queue has dried up. No new WUs and a few resend.
The MCM queue is dribbling out small numbers of WU.

Update: MCMs. WU supply is better. SCC steady new WUs. Last night a flood of resents, now back to new WUs.
----------------------------------------
[Edit 1 times, last edit by Blount at Aug 26, 2023 2:52:30 PM]
[Aug 24, 2023 7:56:02 PM]   Link   Report threatening or abusive post: please login first  Go to top 
TPCBF
Master Cruncher
USA
Joined: Jan 2, 2011
Post Count: 2173
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: 2023-08-17 Update (Weekend work unit shortage and OPNG issue)

Well, something's still not quite right.

While there was a new supply of SCC1 WUs over night, I noticed that a lot of those already returned ended up with a status of "Pending verification" (176 vs an already high +70 a few days ago), also a relatively large number of both SCC1 and MCM1 stuck in validation.
Not sure if this is partially still from the backlog of last week but that should not effect newly send WUs. Unless a lot of hoarders are still biting more than they can chew....
Not much time this weekend to check more closely, let's see how this all develops by Monday.. confused


Ralf
[Aug 26, 2023 3:53:10 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Grumpy Swede
Master Cruncher
Svíþjóð
Joined: Apr 10, 2020
Post Count: 2494
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: 2023-08-17 Update (Weekend work unit shortage and OPNG issue)

I haven't really paid attention to WCG for a couple of weeks now. is there any news about when we can expect OPN1 and OPNG tasks to start flowing again?
[Sep 1, 2023 11:41:13 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Mike.Gibson
Ace Cruncher
England
Joined: Aug 23, 2007
Post Count: 12594
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: 2023-08-17 Update (Weekend work unit shortage and OPNG issue)

Short answer - No!
[Sep 2, 2023 2:42:33 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Speedy51
Veteran Cruncher
New Zealand
Joined: Nov 4, 2005
Post Count: 1326
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: 2023-08-17 Update (Weekend work unit shortage and OPNG issue)

I haven't really paid attention to WCG for a couple of weeks now. is there any news about when we can expect OPN1 and OPNG tasks to start flowing again?

We will know when they start flowing again when they appear in our tasks list :-)
----------------------------------------

[Sep 2, 2023 9:09:49 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Posts: 59   Pages: 6   [ Previous Page | 1 2 3 4 5 6 | Next Page ]
[ Jump to Last Post ]
Post new Thread