I presume you have a quad core system because you stated you are now running only 3 CEP2 to to one FAAH. Has this allowed CEP2 to behave over time ? If so, it seems to point to a potential bottleneck someplace in your system. You have stated you use only high end components and have otherwise maximally stressed your system with other applications and experienced no errors. I am at a loss for where that might be. It also does not explain why you were able to run for an extensive period of time with CEP2 without a problem. The only other item which comes to mind is a transient problem with your power supply, but given your care with components, that is probably unlikely. Other than that I am at a deadend for any further suggestions but would be curious to know if you ever do figure out what the cause is. Good luck.
Cheers

Part of why I choose enterprise class Western Digital 10k RPM Velociraptors as my primary workstation hard drives is their superior random access speeds. In testing, these drives outperform all of my 7200 RPM drives. You have to move from SATA to SAS to get better performance. The more recent models I am using also feature TLER, so the possible bottleneck of error recovery is mitigated. (These drives also feature an "ice pack," a 3.5" heat sink surrounding the 2.5" drive, which means they run at the same or less temps as my 7200 RPM drives given the same ambient temp, and mount in a 3.5" bay.)

I already have write caching enabled these drives, because each of my PC's has a battery backup.

The failures of CEP2 are not occurring during periods of high drive activity, which is why I don't think drive throughput is the problem. There is high drive activity as each CEP2 workunit starts (the first 3 or 4 seconds), but the activity quickly drops to nominal. It's between 10 and 15 seconds into the run that CEP2 fails.

In any event, if Win7's Resource Monitor is to be trusted, the rate at which data is being written comes no where near the known top write speed of this drive, even on random writes, and that high drive activity has moderated well before CEP2 fails. If all four CEP2 processes were starting or doing heavy writes at the same time, then it would seem more plausible that it was reaching the the limits of the drive.

I have been running this PC since I joined in May of this year with "Run always" selected on the Activity menu and the preferences set to use 100% of processors and 100% of CPU time while processor usage is less than 50%. If I know I'm going to push the limit often on the 50% threshold (e.g. video editing), I simply suspend BOINC.

In all other respects, I am very proud of the efficiency/low overhead on this PC. For example, the last validated CEP2 workunit from this afternoon (while the PC did not have an active user) shows 5.16 hours CPU time out of 5.19 Elapsed hours time. At the time, it was running alongside another CEP2 and two FAAH workunits.

I have changed "Run always" to "Run based on preferences," to see if that might stabilize the situation. I have also kept FAAH in the rotation, and set CEP2 to max at 3 workunits at the time. It makes it impossible for all four cores to run CEP2 at the same time.

I must confess some bewilderment at the lack of a useful error code/log. As a Windows desktop app developer, I always write error events to the log (and optionally other events), so that there is at least some information that can support troubleshooting. This often gives me valuable information to convince a user/client that they have a bad drive or a flaky network connection.

I'm going to let it run this way for a while and see how it behaves.

Agreed, Sgt Joe. Doesn't make sense to me either. It would run 24 to 36 hours without a problem, and then suddenly start throwing errors.

We get very high quality/stable power from the electricity provider in our area, and I have good battery backup on that PC, which conditions the power. All my PC's also have high efficiency power supplies that are rated for at least twice the wattage that they require, so total power requirements shouldn't be an issue. With heavy primary drive activity, all four cores at 100% and the LED/LCD monitor "awake," this rig maxes around 150W draw on a 900W battery backup.

As to running 3 CEP2 to 1 FAAH, I can only manage that if I micromanage the queue. I have BOINC on that PC set for half a day's cache. Since CEP2 is currently limited to 3 workunits at the time, it keeps a much larger cache of FAAH than CEP2, so there are times when only FAAH workunits are running at the time. Wish I could tell it to do 3 to 1 . . .

...or you could do the recommended setting for this project and choose to run it on just 50% of your CPU's cores (which, by coincidence or not, was the same as your old setting of running only 2 units at a time).

My two cents on the matter are these: even if the error you see is because of the huge amount of I/O on whatever type of storage you have and assuming you will somehow get that fixed, I would bet it won't be long before you will start seeing another nice error: no heartbeat from client for more than x seconds...exiting. And that, my friend, is the CPU getting clogged up. So my advice: go back to running it on 2 cores. If you have other devices that run other sciences, maybe you can add a unit or two there, to compensate for the other 2 cores that you won't be running CEP2 on.

Oh, and by the way, the fact that BOINC refused to give you anymore units is just a fail safe put in place for exactly this type of situations where a machine spits out only bad results. You would put a leash on it too, right? But no worries: as soon as you get it fixed, your daily quota will increase little by little and since the maximum is not that far, you shouldn't even notice when you will be back to normal.

Hope this helps. Cheers!