I've seen this over the last several weeks, not just in the last few days. I just had a batch of results validate three hours after they were returned. From 10-30 minutes to 180?

Yes, my results were either quorum of one WUs or, for quorums or more than one, I completed the quorum and the WUs are ready to validate and have credit given. I considered the possibility of a backlog delaying things but we would be having recurring backlogs, enough to really get noticed. I believe I've seen cases where any such backlog would have to build up extremely rapidly as I've seen WUs get near instantaneous validation only to have WUs returned an hour or so later see this 2-3 hour delay. It's been off and on since early October I believe.

There are a set of scripts that run 4 times a day. These scripts load new work into BOINC to send, archive results into their proper directories, update status for workunits that errored, notify us of these errors, etc.

There is one step that is very db intensive that takes about 20-30 minutes to run. There is another step that is moderately db intensive that takes about 40 minutes to run.

When the 'very intensive' step runs, we stop the backend daemons (transitioner, validators, asssimilators) to minimize load on the db. If we let these keep running, we saw transaction timeouts and clients being able to request work. However, about a month ago (due to the increasing load) we needed to also stop the daemons while the moderately db intensive step ran. This means that 4 times a day, we stop those backend processes for about 1-1.5 hours. It then takes another 1-1.5 hours for the transitioner to catch up.

This is being done to keep feeding work to the clients while these back-end processes run. Once we move to our new hardware (with four times the memory and processors) we will experience much shorter outages. I expect that we will not need to stop the daemons during the moderately intensive script and the time to run the very intensive should be a lot shorter.