CloudWatch Logs Retention Periods

Brett Gillett

If you’re not familiar with CloudWatch Logs its a feature of the CloudWatch service which allows us to persist logs from applications and operating systems on the AWS platform.

Once you’ve installed - and configured - the Unified CloudWatch Log agent, you’re able to gather logs from both EC2 and Server instances (on-premise).

By default, CloudWatch Logs stores log data indefinitely. This is fantastic, but we need to remember that we pay for log storage. While the costs are not high, this is one of those services that can quietly sneak up on you and end up costing a fair amount every month.

Before looking at how to configure log retention, let’s talk about some CloudWatch Logs terms.

  • Log Event: This is a single log record sent to the CloudWatch Logs service from the agent.
  • Log Stream: A collection of Log Events from a single source. For example, logs coming from a single EC2 instance - let’s say an Apache webserver.
  • Log Group: A collection of related Log Streams. Using my previous example, let’s say you have a fleet of Apache web servers for a specific application. You could organize all those logs stream into a single Log Group. This is where you set your retention period.
CloudWatch Logs - Event, Stream and Group

Setting the Retention Period

Retention periods are set on each Log Group, and as always, can be done via the AWS Management Console, the CLI, or by using one of the AWS-provided SDKs.

Here’s an example of how you could set the retention period of a newly created CloudWatch Log Group using AWS CloudFormation.

        Type: AWS::Logs::LogGroup
            KmsKeyId: !Ref keyId
            LogGroupName: !Sub '${AWS::StackName}-lg'
            RetentionInDays: !Ref retainInDays

NOTE: I’ve included KMS here - which is new(ish) to CloudFormation.

If you have existing CloudWatch Log Groups and need to update them my recommendation would be to script this. One idea may be to create a Lambda function which loops through all the Log Groups in a region and sets a standard retention period.

If you have just a few Log Groups, you can do this easily through the Management Console.

In the end, it’s up to you to decide on how to set retention periods on your Log Groups. My suggestion would be to establish a standard log retention policy and apply it universally across all your Log Groups.

Brett Gillett


Like what you read? Why not subscribe to the weekly Orbit newsletter and get content before everyone else?