My company uses virtual servers. I think in this case my admins would just re-allocate some processors to the virtual validator from somewhere else until everything was re-balanced.

@supdood,

Neat observation -- I was wondering if/when we'd get any evidence that those systems weren't effectively a black hole :-)

I haven't seen a single one return a task for SCC1 yet, but after seeing your message I went through my MCM1 records and found three from the [apparent] early days of those systems firing up - two of them managed to have elapsed time less than double the CPU time, but this wonderful example also showed up (output is from one of my wingman monitoring scripts, slightly edited...)

Task MCM1_0203538_9998_1 was returned by [My Ryzen 5600H] at 2023-09-12T15:59:37+0000:
  Work-unit 374236575 created 2023-09-12T11:53:58+0000
  Sent date 2023-09-12T11:54:00+0000,  deadline 2023-09-18T11:54:00+0000.
  CPU time 1.54008 hours,  elapsed time 1.54102 hours,
  status is Valid
  The workunit has 2 potential results:  wingman data follows.
  MCM1_0203538_9998_0 assigned to [redacted] on Linux
    O/S version is 5.15.107+
    time sent was 2023-09-12T11:54:00+0000
    due time was 2023-09-18T11:54:00+0000
    returned time was 2023-09-14T11:56:16+0000
    status is Valid
    CPU time 5.52996 hours,  elapsed time 28.67212 hours 

That is one seriously stalling CPU!

I'm in total agreement about "getting through the work" -- I currently have over 250 MCM1 tasks and 550 SCC1 tasks Pending Validations, and 720 of them are waiting on replies from the cluster :-( The first 100 or so of those tasks aren't far off initial deadline now...

I have this horrible vision of the BOINC transitioner setting tens or hundreds of thousands of No Reply markers over the next few days as more of the cluster nodes start missing deadlines, with the same sort of consequences we saw a few weeks ago (in particular, retries taking ages to be sent out because there were just so many of them...)

Here's hoping the WCG folks are aware of this and can reach out to whoever is running those systems[*1]. There might even be expertise amongst forum members that could help resolve any issues regarding configuration of those systems (but as we can't identify the owners we can't offer to do it ourselves!)

Cheers - Al.

[*1] Whilst finding a way to simply block them would reduce the problem, that wouldn't be polite, would it? :-)

Or not. Some of my tasks as going through as reliable quorum 1, but most are still unreliable quorum 2 and getting this cluster as wingman. PV tasks still climbing...

And, sadly, the majority of the tasks released as “no reply” are now “waiting to be sent”.

Just a thought, but in the latter IBM days, Kevin or Keith said that there was a big mainframe that crunched intermittently when it didn't have any of its official work to do.

Could this be the problem?

Mike