Greetings,
There have been some questions asked recently about our environment, which we recently migrated to IBM Cloud.

We have three different environments at IBM Cloud. Each is isolated from each other except for a few specific connections (like allowing monitoring).

The first is our “Admin/DevOps” environment. This is available only inside the IBM firewall to house tools that our team use to help us run and manage the program. It is where we run programs like Sonar (code quality analysis tool), Jenkins, Jira (task management tool), Confluence (collaborative team wiki), Gitlab (code repository) and Nagios (monitoring tool). These are installed on 2 Linux (Red Hat 7) virtual machines.

The second is our QA environment where we carry out our testing. We are running 5 Linux virtual machines in this environment. This is enough servers to match each “flavor” of server that we have running in production (see below)

The third is our production environment. We are running 11 Linux physical machines in this environment.

The “flavors” of servers we have in production are as follows:

• 2 servers (24 cores, 64GB RAM, 20 Gbps connections) host our load balancers (HAProxy)
  
• 2 servers (24 cores, 64GB RAM, 20 Gbps connections) host the BOINC scheduler and the website (Apache HTTPD (web server) and WebSphere Application Server)
  
• 2 servers (28 cores, 512GB RAM, 20 Gbps connections) host the databases (DB2 and MariaDB)
  
• 5 servers (28 cores, 64GB RAM, 20 Gbps connections) host the science data, run the BOINC daemons and handle file transfers (Apache HTTPD, Apache Mesos (distributed task scheduling tool), Apache Aurora (Mesos framework for long running jobs), IBM Spectrum Scale (clustered storage tool))
  

We are using IBM Spectrum Scale, Apache Aurora, and Apache Mesos so that when we run any of our backend tasks such as the BOINC validators, workunit building tools or result aggregation scripts, the tasks is distributed to whichever science server has the least work and the task is run there. Each server has the same access to files and the database as the others. This setup makes it easy to manage this infrastructure. It also makes it very easy to additional capacity by just adding additional servers and configuring them like the others.

One question that some people will ask is if we are in the cloud why are we using physical machines. There are two answers to this question. The first is that IBM Cloud has made it as easy to provision a physical server as it is to provision a virtual server. We do lose a few things like being able to easily take a snapshot of the server and save it, but since we used automation heavily during setup, it is relatively easy for us to recreate a server from scratch. The second reason is that we use a lot of bandwidth and file I/O. These are things that have proven to be a challenge for the industry to get to perform well at scale when virtualized. A lot of research is being done on this and it will probably change in the next few years. However, we wanted to make sure that we had the I/O capabilities we needed and going physical was the best way to ensure that.

We are using two automation tools. We have used Ansible for a lot of the low level setup and configuration of the servers. We are using IBM UrbanCode Deploy for the ongoing deployment of changes for our different applications and processes.

Now, to the question of what caused the two major outages that we have had recently. Both of them have occurred when the IBM Spectrum Scale determined that it couldn't trust the data on a set of disks on one of the servers following a reboot. In order to restore operations, the software needed to perform a long recovery process to scan and check the data. IBM Spectrum Scale is very defensive in order to make sure that the data it provides access to is reliable (this is a good thing). The reason this happened is more complicated. The long and the short of it is that we are working with a internal operations team that we haven't worked with before. There were some questions about who was responsible for certain activities, which resulted in some things being set up wrong that triggered these issues. While this is embarrassing to our team, we are confident we can resolve these issues quickly and look forward to leveraging the new capabilities we have built into our new environment.