We are aware of the issue with sporadic generation of workunits. The cause is our job scheduling infrastructure, we are attempting to resolve this today and will update volunteers tomorrow on the result.
Several jobs that we schedule on the backend write lockfiles that will prevent the job from running again, if the previous run is still ongoing. However, there are several points in the scripts that launch these jobs where a non-critical failure will result in the lockfile preventing future work from being indexed and created. We began experiencing these failures after we had multiple server "crashes" in the past two weeks, a lingering issue with the DHCP agent becoming "stuck", eventually resulting in non-responsive VMs as their DHCP leases expire and are not renewed. This has happened to multiple of the six workunit management servers where backend jobs to generate workunits are run, and one of the upload/download servers, in the past week or so. This has resulted in several problems, that we are trying to resolve to bring the system back into a stable state.
The "modulo 4" issue identified with the MCM1 validators for example, was caused by one of these crashes. As surmised in the forums, the number of validators is scaled up by re-launching them as services on the Mesos cluster with a specific modulus. We have investigated the logs for the validators and despite the other issues we are facing with the Mesos cluster at the moment, we are seeing validations go through. The rate of validation should improve as we restore stability to the Mesos cluster and the services the jobs that run on Mesos interact with.
Unfortunately, these problems are caused by part of the system running in the oldest servers (which we started to use in 2021), which we will move out of once the new system becomes available.