A second real-time failover file system would prohibitively expensive and difficult to maintain given the volume of work that moves on and off of the file system.

However, we are now fully using GPFS as the file system for the BOINC file system. These means the following:

• Each of the servers that has access to the file system has the storage mounted directly rather than through an intermediary servers (previously we had a single point of failure for the server that mounted the SAN storage which shared the filesystem via NFS to the other servers)
  
• Each server that mounts the storage has a redundant HBA cards with redundant pathing to the SAN storage array
  
• The SAN storage array is a highly available solution that supports multiple customers (not just World Community Grid)
  

It is unlikely that we will lose access to the file system. However, human error could always occur and that could cause issues.

There is always more money that could be spent to making the system more available and more reliable. But the costs to make it ever more reliable become increasingly expensive with diminishing returns.

As far as I can recall (without digging through notes and history) there have been 3 major outages of the BOINC system since we started using it in 2005. One was right before Christmas 2006 and was significant because BOINC primarily ran off of one server at that time without redundancy. The second was when we migrated to our new hosting environment. The third was last week. There have been other outages ranging from a few minutes to a few hours, but nothing beyond that.

Our scheduled up time is above 99% over our 4-5 year history (I could argue it is above 99.5% but that would require me getting more accurate). I personally think that is a reasonable availability for a system that does not involve finances, medical, or other life threatening or critical systems.

This thread asks the wrong question. Is a failover system possible? Of course it is. Is it necessary and cost justifiable? Absolutely not. Kevin says it perfectly correctly. This is not a critical system as defined by the business world. Whether a project is down two hours, two days, two weeks or two months is inconsequential (except for the whining of crunchers).
I find that even a one nine of availably (90.00%) achieved by WCG is simply amazing for a data center of this type. Even commercial banks and hospitals can only achieve 2 nines and only the most critical data centers acheive 3 nines. Removing every single point of failure is all but impossible except for the most mission critical data centers. In my (non-humble) opinion, WCG has greatly exceeded expectations for availability and I am certain they have met or exceed their service level agreements with their clients. Moving to 99.90% would roughly double the expenditure and I hope that if their budget doubles that they spend it on capacity and not redundancy.

I know it is hard to remember sometimes but the DC technology was not created to keep crunchers entertained and its primary work product is not points and badges.
Having said that … Where’s my UD credits, Kevin?
Cheers and crunch on.

Saw allot of this in the past 48 hours... clouds... forecast for today ... cloudless. Good harvest day for the olives.