The Importance of Designing for Failures

Given the recent AWS Availability Zone (AZ) failure in the Sydney region I thought now was as good of a time as any to reiterate the importance of designing your AWS infrastructure to survive failures.

First, it’s important to understand that just because you’ve decided to move all (or some of) your IT infrastructure to AWS doesn’t mean that you no longer have to design for failures.

Cloud Computing regardless of your chosen platform is not a silver bullet.

Based on the public post-mortem released by AWS it seems that although a large number of services were affected primarily, Elastic Compute Cloud (EC2) and Elastic Block Storage (EBS) devices were the most affected. This article will of discuss some of the strategies you can employ to help avoid failures with both EC2 and EBS.

Elastic Block Storage (EBS) Strategies

AWS automatically replicates your EBS volume data across multiple servers within a single AZ, which is fine, but if you want to design for failure you need to get your EBS data into another AZ or, if required, another region.

Replicating EBS volumes into another Availability Zone (AZ)

By default, EBS volumes are tied to the AZ where they were created. To move EBS volumes, you need to do a couple of things. First, you need to create an EBS snapshot, and then you can create a new EBS volume from that snapshot in another AZ. Simple stuff and you can automate this process by using the AWS CLI or Boto and scheduling the job(s) via Lambda

Copying EBS snapshots to another AWS region

If you require even more protection, once you have created an EBS snapshot you can use the copy command in the EBS console (or the AWS CLI) to copy that snapshot to another region.

Elastic Compute Cloud (EC2) Strategies

First, you should create your infrastructure in AWS in a highly available, fault-tolerant architecture. You should be utilizing Elastic Load Balancers (ELB) to distribute traffic across AZ’s, and you should be using Auto-scaling groups to make sure that your infrastructure can scale up (and down).

If for some reason, you’re not able to do this, then you should be making regular snapshots of your EC2 instances (and their EBS volumes) and copying them to another region in the event of an AWS infrastructure failure.

Creating EC2 Amazon Machines Images (AMI) and Copying them to another Region

Using the EC2 dashboard (or AWS CLI, or Boto), you need to create an AMI of any EC2 instances you need to restore in the event of an issue. To make sure that you get a good snapshot, you should stop the EC2 instance first - although it’s not required. Once the AMI has been created make sure that you copy it - you can use the console, CLI or Boto to do this. Just like the EBS section you can (and should) automate this process.