I just had this too, both on the main site and on the forums, at the same time. But it's definitely intermittent.
Load balancer? Hardware being recycled mid-session?

So a long explanation. As part of moving to the IBM Cloud we have started using the email delivery offering (http://www.softlayer.com/email-delivery) that is available. Although we had a pretty good setup at our old environment, setting up SPF, DKIM, encryption for sending, processing bounces, etc is really not what we want to spend our time working on so we have stopped doing it ourselves and we are using an external service now.

The old environment worked pretty well and when we sent mass emails like the one we sent today in the old environment we would identify around 0.5% or so of emails as "bounced". Once an email bounces we disable sending emails to those accounts in the future. We also processed unsubscribe requests (this is all part of maintaining good email lists and being respectful of your users).

However, we had custom built the method to identify bounced messages and so we missed some cases. Our new email delivery system is not missing those. As a result, the new system is identifying a lot more "bounced" emails than our old system did.

The new email delivery system notifies us of a bounced email by calling an API on our server for each bounced email address.

Due to the much better infrastructure that we have now, we sent out the first 20,000 emails in about 3 minutes. This resulted in a surge in "bounced" email API call back to our servers at a rate that exceeded our expectations and resulted in the site going offline temporarily. We spent the next couple of hours scaling up the API and letting the backlog of bounces get processed. Once those two tasks were done we were able to resume sending the emails at a more nuanced pace and the site has been behaving normally since then.

Since we will have processed the backlog of bounces after this sending, we will be able to scale up our sending next time because our API is better prepared to handle the load and there will be fewer bounces generated by the sending of the email.

me too

@SekeRob

What page are you trying to access? If you are using Chrome can you do the following:

• click on the three dots on the top right to open more options
  
• click on 'more tools'
  
• click on develop tools
  
• go to the network tab
  
• make sure "preserve log" is checked
  

Then access the website and recreate the issue. Once it has captured it, can you expand it to full screen, take a screen shot and send it to support@worldcommunitygrid.org? Also - if you could send a screen shot of the pop up as well that would be great.

I think what you describe is different then the 503 issue.

Mid-night visit to the fridge, to get a cold glass of water, and peek into cruncher and sure enough, the message box was waiting on a reply to continue or break off, at page 1 of 1234 (x250 results). This was about 1:35 UTC and went shuteye again. All had ran through this morning. Then in the hunt for the missed, an hour runtime is long to cause lots of changes ad interim, let alone the 1.5 hours of added stall, and again, page 1 hung, then ran through to the end.

As statistically it is happening most often at page 1, as if someone needs waking up to load a reel in the tape room, now programmed an added sleep of 30 seconds to try same page-offset again and ping when it's daytime, one time only, be silent when at night. In a loop, do until, max 5 times, then the message box.

Far as I know, the Results Status page data remains accessible 24/24, no lock out periods. Previously this occurred very occasional, which is why a msg box was set, to give me a chance to check the intertubes, but the ISP has been very stable. Anyway, now the backoff loop is 2.5 minutes.