How to Save Big on Big Data in AWS
Is your S3 data accumulating and costing more than you would like? Cloud storage has a way of creeping upward, especially as applications become hungrier for data. Thankfully, if you’re just noticing this issue there’s a good chance you can leverage lifecycle policies to manage your data in a more sustainable way. One of our customers saw a 70% reduction in daily S3 costs after implementing appropriate lifecycle policies for their S3 data.
A lifecycle policy is a rule you can specify in S3 that defines how the service should treat your data as it ages. Possible actions include expiring old data, transferring it to a cheaper quick-access storage tier, or archiving it for posterity. The greatest cost savings come from knowing the optimal lifecycle for your data.
For example, if you frequently access data within the first 30 days of adding it to S3, access it somewhat less frequently in the subsequent 90 days, and rarely access afterwards but must retain archives for 2 years, then you would save 87% using this lifecycle policy compared to keeping the data in S3 Standard (assuming each bucket is stable at 5TB):
- After 30 days, transition data to S3 Standard Infrequent Access.
- After 90 days, transition data to Glacier Deep Archive.
- After 730 days, delete the data.
This kind of policy applies an expiration date to the data in your S3 bucket, ensuring that you don’t pay for data you aren’t using. If your S3 bucket is not currently configured to expire old data, you could achieve the savings seen in the example above (or more) by applying a similar lifecycle policy.
Another way to implement lifecycle policies from the get go without having to configure resources manually is to use the Intelligent-Tiering storage class. This class of S3 storage will automatically transition between Standard, Infrequent Access, and Archive tiers based on how recently the data was accessed. Depending on your use case, this can result in large savings with minimal effort, especially if your access patterns are unknown or unpredictable.
The bottom line is that data hoarding is one of the main contributors to cloud storage cost creep, but it can be managed with the consistent application of data lifecycle policies. If you take the time to consider your data lifecycle, you could greatly reduce your storage costs.
Check out this page for more information on S3 Storage Classes: https://aws.amazon.com/s3/storage-classes/