Monitoring Burstable T2 EC2 Instances

Recently, I was working with a client who was leveraging Burstable T2 EC2 instances for their web infrastructure - something which I regularly recommend to my clients. In this situation though we were receiving reports from the application team about odd behaviour. On what seemed like a random schedule the websites would stop responding to requests entirely or complete the requests at a reduced rate.

We immediately started to comb through our logs in an attempt to determine the cause and implement a fix. Based on CloudWatch metrics which we were already capturing it appeared as though we had a memory leak in one of the applications running on the Apache Server.

After a conversation with the application team, we decided to vertically scale the EC2 instances to get more memory and setup an automatic reboot for the web server EC2 instances, not a great solution, but it solved the issue for the time being.

We also decided to supplement our current monitoring solution with DataDog. After configuring the agent and collecting a few days of data, it became apparent we were not dealing with a memory leak. Something else was going on.

As we were brainstorming other possible causes, somebody casually mentioned that we were using T2 instances for the web servers. I immediately opened up CloudWatch and pulled up the CPUCredit Balance metric and voila - there it was - zero credits available.

The web servers were consistently over their baseline! Mystery solved. We were able to quickly modify the EC2 instances to General Purpose (M4) instances - reduce their size and the problem was solved.

The moral of this story is, first not to avoid T2 instances, they are fantastic for spiky workloads and are very cost-effective, but make sure you have CloudWatch (or something else) monitoring in place to ensure you have a positive CPU Credit Balance, and your solution is not experiencing throttling.

Now that we’ve identified the issue and implemented a permanent fix, we’re investigating opportunities to leverage serverless solutions to deliver these websites in the future. By using a Serverless architecture not only will we be able to significantly reduce our monthly costs, but also avoid issues like running out of EC2 CPU Credits in the future.