Suggestion of a failure control mechanism based on the introduction of Result Status tallies on a per project basis.

I quite often see posts about what I describe as run-away-failures; where tasks from one project continuously fail on systems. These are often the result of non-WCG processes, but none the less effect some projects more than others. My suggestion is that the servers keep an Error Tally list by system and when the error number is too great, the server sends an email to the cruncher to advise of the problem and only sends other task types.

The email could notify the cruncher that their project selection has been changed (if they only selected one project) to run different tasks because that project’s fail rate on that system was too high, and that they should restart their system (often resolves this situation). If the cruncher has more than one project, then they would just stop crunching one of their chosen projects until the cruncher selects it again (following a system restart, hopefully).