First of all - I apologize for our low level of communication. The problem was more serious than I expected and I've been doing some serious digging in to understand the issues.

There are two issues we are experiencing.

1) The BOINC 'working directory' where we store the files uploaded from you as well as store the files ready for you to download went up to 97% full. We are adding more storage to address this issue. I'm still looking at why the storage jumped up in space relatively quickly that caught us unaware.

2) The SAN storage for the MySQL database that backs up the grid is hitting a very high load. The short term change is that while we are adding additional storage, we are getting some storage from a different SAN array and allocating a new filesystem from that allocation. We will put the binary logs on that filesystem which will alleviate some the storage contention. Longer term we are going to look at the use of the MySQL 5.5 compression and look at the block sizes for the SAN/RAID/FileSystem/MySQL to make sure that they are properly in alignment.

Right now we are running the file_deleters, validators and assimilators only (not the transitioner). This is allowing the system to catch up on the back log for these queues. Once they are caught up, I will turn the transitioner back on and allow things to catch up. We should catch up faster once we get the new filesystem for the MySQL binary logs.

Even longer term we are working to attempt to forecast the load we can handle on the current setup (we are always doing this) and planning for when we will need to expand.

We will still proceed with the filesystem update tomorrow. We have had to manually intervene on the filesystem every few days due to what appears to be a slow memory leak in the software. We are anxious to get that resolved as it is disruptive to keep having to stop all access to the filesystem and unmount and remount it.

The addition of storage for MySQL should be happening in the next 30-40 minutes. Once that is done we will stop the db server, make some changes, start it back up and after the initial start up period we should be able to catch up fairly quickly.