ARP1_0033636_089_5

Currently at 79% but I don't expect anything from it. All previous copies errored with Unhandled Exception.

Again this morning, no new work available - only the occasional resend.

Mike

Once again no work after 0000 UTC stats run and machine is idle at the moment. I hope someone looks at the WUs that are getting errors soon so we don't end up in the extreme laggard situation again like the last time

I got 6 new tasks at 13:14 GMT (UTC). All non-priority 099 x 5 and 101.

There does seem to be a link to the statistics runs.

I have reported the issue.

Mike

I should have posted before this - I apologize for the delay.

There was an issue with the object storage system starting around 1:00 UTC Saturday morning. Due to this issue we were unable to create the next generation of workunits for the project except occasionally when the issue wasn't present. The issue was intermittent but each time it happened it caused the assimilators to stop running. We had a period mid-day Saturday and again on Sunday when we were able to run for a few hours before the issue re-occurred. That allowed us to catch up on the backlog and get a large set of work out. It has been running again since about 8 hours ago so I'm hopeful that things are stable again.

The object storage is used to archive the outputs used for each generation so that we can re-run a line starting from a previous generation if needed. We only save the past 6 generations due to the amount of storage required (we used to only save 3 but in the case of the 6 workunits we had to restart, 3 generations proved to not be enough).

There is a small number of workunits that are having an issue (ARP1_0033558_091 and ARP1_0033636_089 are among them). I need to collect and send the data back to Delft but I had some critical transition work I needed to do first. I'm hoping to be able to send that information to them soon.