Enabling logs and configuring them is an important step in securing your AWS environment, however you have to also be using them effectively! Just monitoring your environment isn’t enough, you should also be proactive with the data you’re collecting. Today I’ll focus on VPC Flow Logs, where and how they’re stored, and how to use that data to troubleshoot common network issues.
First off, determine if VPC Flow Logs are enabled. There are two possible ways to deliver flow logs data; to CloudWatch Logs or to an S3 bucket. If Flow Logs already exist you’ll be able to see the destination.
Otherwise you’ll need to create a new one. For this example I’ve selected to filter ALL traffic but that can be narrowed down to Accept only and Reject only as well. Regardless of which destination you choose you’ll need to have it set up ahead of time, so you’ll need to make a CloudWatch Log Group, or an S3 Bucket before you can point Flow Logs to it. You’ll also need an IAM role which you can set up beforehand, or select “Set Up Permissions” to generate a role.
Once the Flow Logs have been created and populated with data we can start using it for things like queries.
Here’s what the raw flow log data looks like in a CloudWatch Log group. While using the console is a good start, we can do better.
By using the CloudWatch Logs Insights we can get AWS to do all the heavy lifting for us. Select Insights under Logs and then choose your log group. There are also example queries accessible from the folder button on the right.
When using an S3 bucket as the destination, a table has to be set up that queries can be run from. We’ll be using AWS Athena to run those queries and set up the initial table. Those familiar with SQL will feel right at home. I will provide the code here but it’s also available in the AWS Documentation.
|
|
The important part is editing the LOCATION set here to your S3 bucket that stores the VPC flow logs you want to query.
You may also need to create a partition to be able to read the data as mentioned in step 4 of the documentation. The code you’ll need looks like this:
|
|
AWS provides some query examples in the documentation but I will also be listing some other queries here as well.
While these two queries are basic, they will test if your Log Group or Athena tables are set up correctly. The ‘limit’ is there to keep the query from taking a long time to parse your data and can be optionally inserted into any other query in both CloudWatch Logs and Athena.
|
|
|
|
This code checks for all traffic coming through a specific AWS ENI. This is great for when you’re troubleshooting specific instances or other network connections to narrow down traffic and errors.
|
|
|
|
|
|
|
|
This query checks to see how many times an inbound IP address has tried to hit the instance in question using the same source port.
|
|
|
|
Today you’ve learned about VPC Flow Logs, where they can go, and how you can use them to query data in meaningful ways. There are plenty of other queries you could try, and AWS also allows you to monitor pretty much every service they provide in some way. Remember that monitoring is just the first step, it’s how you use that data that counts.
Like what you read? Why not subscribe to the weekly Orbit newsletter and get content before everyone else?