Just an update. We are currently running well (good supply of workunits and the transitioner is caught up). The validators are also caught up.

We are looking to track down an issue that is causing a deadlock that causes the transitioners to exit when they try to find new workunits to transition. This is an issue because once the transitioners stop running, we eventually run out of work and validation falls way behind.

As we look into what is causing this deadlock, we expect the transitioners to get knocked offline now and and again. As a result, we may experience temporary delays in validation and shortages for work. We are working to get this resolved and we appreciate your patience while we work through this.

We are going to take a short outage of the feeder and BOINC daemons. During this time boinc-client will not be able to report work completed or get new work.

We are attempting to improve the performance of one index that is at the core of many of our current challenges.

We appreciate your patience while we continue to work through this.

This outage is done and we are seeing the performance we previously expected. We are going to watch things closely over the next few days and make sure the issue does not occur again. Thanks to everyone for their patience during this past week and a half.

Things have continued to run well over the past 12 hours or so. Now that we think we have gotten to the root cause I can give some more detail about what was going on. This is highly technical for those who are interested.

When we did the update to apply the Spectre and Meltdown patches, we started to see various performance related issues on the systems. Since the issues we saw were so highly correlated in time to the application of those patches, we spent a lot of time investigating that. However, during our investigation some information started to appear that didn't make sense with that being the explanation. In particular we discovered that these queries were not performing like we expected:

The table workunit has an index called wu_timeout that consists of transition_time. That means that the performance of a query that has transition_time in the where clause should perform significantly faster using the index then not using the index. However, we were seeing the exact opposite which was very odd.

After reading and looking at various performance metrics we reached the conclusion that something during the Dec 6th updates had someone damaged these indexes and we eventually decided that they needed to be dropped and recreated. That is what we did during the short outage yesterday. Immediately after that we saw the following performance:

The query time using the index went from 26.5 seconds down to a tiny fraction of a second. The overall load and contention on the database dropped dramatically and the system started to behave like it had been before.

We looked a little further and saw a few other indexes that were exhibiting similar symptoms so we rebuilt those as well. As a result, the database is backing to performing beautifully and the team's stress level has gone way down.

Based on this, we still feel that Spectre and Meltdown have caused a performance hit on our database servers of somewhere between 20-30%. However, the database servers were sized to handle at least a doubling of load so we are still very comfortably within our performance capabilities.

Thankyou Kevin for both those very interesting posts - and, of course, for all your, and your colleagues effort in resolving the situation.

I'm curious to know, do you, or would you ever consider, rebuilding the database indexes on a regular/set schedule (say, once a week/month)?, as when I was working for IBM several years ago, we had regular housekeeping jobs that did just that.