Seems like we ran out of tasks again. Last ones I got before midnight UTC were all resends.

Edit: well, my crying helped, got 3 more resends, but nothing fresh.

We're back in business!

We have an update on the Lab page (https://www.cs.toronto.edu/~juris/jlab/wcg.html):

MCM1 Workunit Availability: The recurring problem where older servers/VMs in our private cloud lose their DHCP lease on all interfaces and effectively go down, has been causing the quorum of coordinator nodes that accept jobs and assign them to workers ("the scheduler"), to stop accepting job submissions when this crash coincides with a second issue that renders the coordinators unable to recover their quorum. Building on the work we did to generate ARP1 and MAM1 workunits locally on WCG servers, we are migrating the MCM1 workunit delivery to Kubernetes, which should permanently resolve the issue, and increase workunit supply overall. Initially we had planned to complete this work after the release of MAM1 7.05, which has been a significant refactor, but given the frequency of failures we are moving it up and will complete the migration this week.

MAM1 7.05 - Why is it Taking so Long? MAM1 is being refactored to use LibTorch and run also on NVIDIA GPUs. LibTorch vastly simplifies the checkpointing logic, and should resolve the unsafe memory access and resume from/checkpointing crashes in previous beta releases as well as significantly improve performance.

You are not alone. Have a look at Unixchick’s unofficial status page thread, here.

This is a server issue. No fresh WUs are flowing at the moment. you might get a resend if you are lucky