A quick update:

We have been using GPFS - a clustered file system to mount SAN storage across several servers. The file systems that store workunit data for downloads and that store the results when they are uploaded are on GPFS. For reasons that we are still trying to diagnose, the GPFS software caused a kernel panic on the webservers which locked them up (and they had to be physically rebooted) and caused the outage.

We are still using GPFS on some of the backend servers, but the web servers are now accessing the data via NFS. We are continuing to investigate what happened with GPFS but full connectivity to members has been restored.

The BOINC clients should be able to upload and download work easily at this time. They will automatically attempt to reconnect once the back-off timers expire so no action is required by the members.

We appreciate your patience.

We were able to locate a spot on the GPFS filesystem that had been corrupted. We have corrected the problem (a few files were lost in the process but they were workunit input files so some folks will get errors downloading workunits, but no results were lost). The BOINC grid and website have been up and stable for the past hour. We continue to watch and monitor the behavior of the system.