Adri,

I wondered about doing that the first time round, but opted against it because I couldn't be sure there wasn't also something else wrong with the file that didn't show as a syntax error... So I just hope that is the only error in that data file :-)

One side-effect of users fixing the data file might be to disguise the problem, and if [as appears to be the case] they have suspended they suspend SCC1 supply with a view to identifying and removing the remains of the bad batch (as appeared to happen last time), any "repaired" jobs still out in the field probably won't count for anything (depending on how they "remove" the bad WUs...)

[Edit:] I thought they had suspended SCC1 to do some clean-up as there didn't seem to be any new SCC1 of any type for quite a long time... However, new SCC1 tasks started turning up late this afternoon, so perhaps it was just an overnight precaution (their time, not UTC...)

I'm more concerned about how a second bad batch got turned into active WUs after they'd had to deal with the first one -- if it had already been delivered by the scientists, could it not have been checked[1] (and either repaired before WU generation or suppressed. as appropriate!); if it was a new delivery, why hadn't the scientists checked the flex file and repaired it before shipping?

And if there are still more bad batches already in the pipeline, I hope they get culled or cured in advance :-)

Cheers - Al.

[1] I don't know how automated the process of accepting SCC1 work and making WUs is, so that might not be as easy as it sounds :-(

[Edited in light of the [apparent] resumption of SCC1 supply, including bad batch cases...]

Clever workaround adriverhoef, but yeah, it won't address the root cause of what caused it in the first place, and the odds are very low that the whole batch won't be invalidated and re-issued.

To save you being there all night and you thought about using Boinc Tasks this will allow you to cancel all tasks ready to start. I do recommend setting "no new tasks" before cancelling tasks waiting to start :-)

Adri,

Thanks for posting about those, as I'd been aborting any I spotted but hadn't followed up to see what happened to them!

It looks as if something was done about these bad WUs some time between about 17:00 and 19:00 UTC on 2nd June (WCG afternoon shift?) as any tasks of mine that failed (or that I aborted if I spotted them) before that period ended up with retries (up to about the same time interval), whereas after then any tasks that were sent back (or aborted) didn't get retries!

As for those two examples, I think "Too Late" may also appear for returned tasks that are for "Don't need" cases, and as retries don't seem to be going out for MyoD1-C tasks any longer and tasks already out there are being Server Aborted it looks as if they may have [finally] marked the bad work units as unwanted!

An unwelcome current side-effect of whatever they've done is that the only available SCC1 work now seems to be retries for MyoD1-A/B work-units :-( -- It now being the weekend (or almost so in WCG's time-zone!), it'll be interesting to see if any new work shows up before Monday and in the two+ hours since I first posted this. the tap has been turned on again, and there are still occasional MyoD1-C tasks amongst them (but not many...)

I hope they post something about what is happening regarding the ongoing problems with MyoD1-C batches[1]...

Cheers - Al.

[1] And if that includes the information that the only thing wrong with the flex file was that missing space, it might legitimize your work-around :-) -- not that tampering with data files should ever be acceptable, even in what seems to be a good cause... :-) :-)

Al,

That may be the result of (types A and B) tasks needing wingmen to resolve the "unreliable" status that you get after processing any, each and every task from the faulty batch. All faulty tasks together are creating a hausse (upturn) in tasks (of types A and B) needing verification. Also, type C is still being sent out at a slow pace, because of the resends for types A and B needing verification. Important: the system is holding up and still hasn't collapsed.

Also, new workunits for types A and B are being distributed, albeit still scarcely.

Agreed. It seemed like a good idea at first, but in the end it only led to a lot of wasted cycles (and one Valid(*1)). It should probably never be acceptable in any way but to point out and document the error.

Adri
[*1] (Output generated by 'wcgstats -frrre* SCC1_0004176_MyoD1-C_0299')
workunit 311931323

SCC1_0004176_MyoD1-C_0299_0  Fedora Linux  Valid  2023-06-01T21:21:18  2023-06-02T09:46:32  0.77/0.78  69.0/69.0
	Logfile:
	<core_client_version>7.20.2</core_client_version>
	<stderr_txt>
	INFO: result number = 0
	INFO: No state to restore.  Start from the beginning.
	[10:58:41] Number of tasks = 1
	[10:58:41] Running task 0,CPU time at start of task 0 was 0.000000
	[10:58:41] ./cmpd-1100299.pdbqt size = 19 3 ../../projects/www.worldcommunitygrid.org/scc1.MyoD1-C.pdbqt size = 1268 0
	[11:45:29] Finished task #0 cpu time used 2784.904472
	11:45:29 (1000920): called boinc_finish(0)
	
	</stderr_txt>

PS Crystal Pellet, nice try!