Hi everyone,

Some good news to share before the weekend. Our data centre was able to fix some of the networking issues that should help us to stabilize the flow of workunits from the data centre to your devices. We ultimately aim to prevent the transient issues that have been the most commonly reported bug from volunteers during the testing phase and prevent further interruptions to the supply of workunits available to our volunteers.

Working with personnel at the data centre, our engineers were able to find a temporary workaround for the networking issue whereby migrating VMs to dedicated physical hosts with a previously configured virtualized interface to the VLAN; thus, the switch that provides access to the storage layer of the WCG was successful in connecting additional VMs to the storage layer. Oversubscribing some of these new VMs that will join the production Aurora/Mesos cluster to the physical resources of the bare metal hosts allowed us to fit the VMs, and will increase the overall capacity of the workunit processing backend by roughly ~45%. This will provide an additional upload/download server to help mediate the issues reported by volunteers in awaiting download/upload of workunits (and reduce the manual workarounds we had to do to make workunits available again).

Currently, we are running diagnostics on these servers, and will put them into service as part of the production Aurora/Mesos cluster as soon as we are finished. The oversubscription to the physical resources of the host should not affect the performance of the VMs, as the newly migrated VMs we are putting into service were co-located specifically with VMs that use those resources intermittently or sparingly (Build Servers, DevOps, internally hosted apps, etc.)

It may seem like a small advance - but besides directly helping with the workunit management - this will also free up some time for our tech team to finish other pressing challenges, and brings us closer to a more stable WCG.

Thank you for your support, patience and understanding.
- WCG team