Wow, what a morning!! Milkyway also went down this morning and GPUGrid (which has huge upload files) finally nailed the lid on the coffin for my farm. All died from communications errors. I booted everything and let BOINC sort it out ... which it finally did.

Let's not do this again anytime soon.

Post-mortem:
It's now the day after the outage, and I just found this thread.
For some reason, I was unable to access the WCG forums or any more than about 2% of the website during the outage. I just got the "The application is unavailable at this time, please try again later." page, or a blank page, or an SQL error message. Yet it seems others were able to make posts in this thread. ??? [Edit]: I'm bookmarked into http://www.worldcommunitygrid.org/index.jsp

Main reason for this post:
Did anyone else check what was happening in the Transfers tab of their BOINC client(s)? With the WCG server off-line, you would expect that the uploads of each file would try to start, fail, and time out. Not this time. There must have been some front-end server still active. The files would upload - you could see the kb/s and their progress, reach 100% transferred, and then fail. For every file, BOINC would back off for a time that increased up to about 2 (?) hours, and then repeat the process. Over and over, with more and more files as they arrived in the queue. The Net traffic going into the WCG data centre must have been incredible, but it didn't seem to slow down.

Having multiple machines crunching, I suspended network activity in all but the slowest. I let that continue so that I could see when the system was back up.

For any of you whose ISP counts uploads and charges $$ for over quota, you could be in for a nasty surprise when your next bill arrives.

I know that the techs were probably tearing their hair out over other issues, but I suggest that if such an event happens again, the front-end server or whatever zombie was handling our uploads, gets disabled.

Suggestion #1 was going to be that WCG have an emergency substitute website server available to display a single page on which the techs could post simple status reports. However, it seems that some people could access the forum, so maybe there's no need.

Rick,
Nobody has been able to access the forum for several hours. knreeds' post in the Known Issues has been posted at 14:17 UTC and the whole thing was already down sometime around 7 UTC for me.
Then after the first steps of the repair operations the website has been accessible again and the forum as a consequence.
Then uploads have been possible again.
And finally reporting and new tasks fetching.

Regarding unsuccessful uploads and their possible cost:
Since there was nothing at the other end of the pipe what you could see from the Transfer tab every time the client retried uploads was a few blocks pushed in the pipe (enough for a small file to reach 100 %).
Then upon receiving no acknowledgement at all from the other end the client entered back-off mode for a random time.
Therefore very few kilobytes of data were sent to your ISP during these attempts.

If Rick's jobs were CEP2 it may have been different since the Harvard server could receive files, and if it needs WCG server's availability to complete the process it's a different story, unfortunately.
But Rick did not mention CEP2 in his post so far.

On the other hand he mentions something which was not true, at least for a set of files without any CEP2 tasks:Every time I have watched a retry BOINC was starting to transfer a few files as I described, and then it backed off the whole list without trying for every file in the list.

Sekerob and JmBoullier has commented on the server-side of things, so I'll keep it on how client handles errors.

In v6.10.xx and older clients each file-transfer (upload or download) will have it's own random backoff in case of errors, this is between 1 minute and 4 hours, and always starts with 1 minute and increases in case of errors.

v6.10.xx also has a new Project-wide backoff, so with 3 transfer-errors in a row to same project (**), the whole project will get a random backoff, and this again will be between 1 minute and 4 hours. This is so fast, multi-core computers that can have many file-transfers in case of problems, won't be continuously trying a different transfer as the individual file-backoffs times-out. While project has a project-wide backoff, any new uploads won't be tried immediately, but will wait for the project-wide backoff to count-down.

The next serie of BOINC-clients that is currently in development, v6.12.xx, will have an additional change, since instead of the 1 minute to 4 hours random backoff, it's increased to 10 minute to 12 hours random backoff, and the same for project-wide backoff.


(**) In case a project has multiple upload-servers and/or download-servers, all the listed servers will be tried before project-wide backoff kicks-in, so there will be more than 3 errors before you'll get a project-wide backoff in this instance. AFAIK WCG doesn't have multiple servers, except for the linux-only CEP2.