There does appear to be a problem with the validator, most of my pending jobs have two completed WUs and should have validated.

The system is completely stopped from the looks of things. The only wild guess I have is that they are waiting on a new drive. The last message we got was that they expected a quick fix on Monday, so it sure would be nice if they just came on and said that they hit a bump and will be back once they have the thingy and have it installed.

It's not quite completely stalled, but it is very sluggish! I'm seeing small numbers of PVal jail inmates escaping, but almost all newly completed work gets stuck -- even OPNG tasks without a wingman! (I'm only waiting for 2 SCC1 now -- that was 20+ a couple of days ago)

Possible reasons?

• If the validators need to access non-BOINC storage to work, that may be a source of delay;
  
• Similarly, if final results are written to non-BOINC storage, that might help explain why some validated tasks seem to get stuck afterwards (I've checked several, and their flags suggest files have not yet been deleted, so no assimilation possible);
  
• the data centre's update to their scheduler on or around 13th October apparently caused it to kill off quite a lot of running systems, and whilst any relevant WCG systems should've been restarted by now, I wonder if something is still stopping them periodically;
  
• WCG may [still?] be turning various parts of the system on and off in efforts to clear out the tail end of the mess caused by that storm of non-returned jobs a while back (I've still got WUs not signed that have jobs from that cluster in their history...
  

The couple of references to non-BOINC storage in that list may well relate to that virtual disc loss and restoration...

As you said, some information would be nice, but don't forget that the WCG tech team is also the MCM tech team, so some of them may be busy elsewhere :-)

Cheers - Al

P.S. Here's one extreme MCM1 task to show some of the issues we've been having...

MCM1_0204016_2007  (WU 389244523 created 2023-09-24 05:41:32)
  0:  sent 2023-09-24 05:41:37,  due 2023-09-30 05:41:37  (** mine **)
      Status Pending Validation  Returned 2023-09-24 12:50:23 
  1:  sent 2023-09-24 05:41:35,  due 2023-09-30 05:41:35:       
      Status No Reply  (** from that cluster **)
  2:  sent 2023-10-14 08:04:51,  due 2023-10-17 08:04:51:       
      Status Pending Validation  Returned 2023-10-14 19:52:27
      This is a retry for task 1 -- delay of 14 days, 2 hours before sending

And that's its state at around 14:00 UTC on 2023-10-20 -- the last 6 days in PVal jail!

We have 216 pages of pending results out of 698 pages of results. So, about ~31% are pending.