Yes we should have monitoring for this. We do have monitoring setup for many of the scenarios that happen. In this case the feeder went completely dry, the monitors we have in place test to make sure that the feeder isn't clogged with all resends (an indication that there is a work unit issue or science application issue). So, my main task today will be to update that monitoring to include projects that are completely empty.

Thanks,
-Uplinger

I have just forced the script that creates those reports to run. They should get updated files soon.

Thanks,
-Uplinger

+1

Greetings,

We are extremely sorry about the work unit outage that occurred this weekend. My initial investigation shows that a the scripts that push the workunits into the database was stuck with an illegal lock file. It appears that the lock file was illegal because both servers that attempt to load work happened to create it within the same time. This has been the same mechanism we have used for many years without issue.

I will be adding additional code the the create work scripts to check for an extended period of time and send alerts to our team if the lock file is illegal. I will also see if I can add additional monitoring from an external server to check to make sure there is work available on the servers. At the moment, I need to think what the best possible way of doing this is.

On a personal side, usually I log in and check the grid health every day. Starting thursday of last week, I was spending some time with my twin brother for a long weekend. Since I did not get any alerts due to no checks for the feeder being completely empty as mentioned before, I assumed all was well.

Again, we will be adding some more monitoring so this issue can be caught earlier and work can be consistently flowing. Thank you for your patience with us on this.

Thanks,
-Uplinger

Unfortunately there were/are no news, statements or reportings on https://secure.worldcommunitygrid.org/about_us/displayNews.do and Twitter.