Bug-Byte 1: AWS EC2 Auto Scaling Group Instances Fails on Launch

Have you ever encountered an obscure error like the one below in the AWS Autoscaling Console?

    
        
Client.InternalError: Client error on launch.

    

Often vague errors like these can be traced back to encryption-related permissions issues. AWS intentionally obfuscates Key Management Service (KMS) errors to prevent malicious actors from gaining information from error messages, which can sometimes make debugging a frustrating experience.

If the Amazon Machine Image (AMI) you are using for your Auto Scaling Group is encrypted with a KMS key, and you are seeing this issue in your Autoscaling Group, check the policy on your KMS key. It should allow access to the KMS key from the AutoScaling Service Role, like so:

    
        
{
   "Sid": "Allow service-linked role use of the customer managed key",
   "Effect": "Allow",
   "Principal": {
       "AWS": [ 
            "arn:aws:iam::123456789012:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
       ]
   },
   "Action": [
       "kms:Encrypt",
       "kms:Decrypt",
       "kms:ReEncrypt*",
       "kms:GenerateDataKey*",
       "kms:DescribeKey"
   ],
   "Resource": "*"
}

    

This also applies to encrypted Elastic Block Store (EBS) volumes.

I hope that helped, good luck out there in the cloud.