working again...

There was a set of 20 batches for MCM1 that almost every workunit in the batch reached the maximum file size before the job terminated early and sent back the data it had computed so far. This resulted in a rapid filling of the filesystem. We were able to clear some space to get work flowing again and things are now stable.

The newer batches that are now flowing do not have the same situation so it looks like we are past this issue but we will be watching closely.

The workunits were not bad. What happens is that the some of the types of analysis that is run by the different algorithms output quantities of data. This sometimes results in large result files. Because this project had the potential to create some very large data sets, we worked with the researchers to add controls to detect when a data set was particular large and stop the analysis at a known good point and return the result early. The researchers will then evaluate that work that was done and it will either be good enough for what they need or they will run the remain bit of the job on their local grid or resubmit the remaining bit of work in future workunits.