Using CloudWatch to Monitor VPN Connections


Travers Annan

VPNs are essential for modern internet security, and most organizations use them as part of their IT infrastructure. Cloud IT systems are no different, however in a cloud environment there are some interesting opportunities for automating the monitoring and maintenance of IT resources. A great example in AWS is that you can set up CloudWatch alarms to track the status of your resources and take automatic action in case of a failure event. Once you have a functional alarm, you can do things like send notification emails, repair resources, and run serverless code using AWS Lambda. By leveraging the power of serverless functions, the alarm responses you can design are only limited by your creativity.

As an example, we will set up some alarms to monitor our VPN connections and send us email and slack notifications if they go down. To accomplish this, we will create CloudFormation templates for our VPN alarms and SNS topic, a slack notifying python script, and a serverless config file for our Lambda deployment.

The first resource we need to deploy is the SNS topic, as the other CloudFormation templates reference the topic ARN. The template for this resource is pretty straightforward, deploying a named SNS topic with a subscription. Note: you should change the subscription endpoint to the email address where you want to receive notifications.

SNS topic template:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
Description: Add topic for Cloudwatch alarm notifications

Resources:
  alarmTopic:
    Type: "AWS::SNS::Topic"
    Properties:
      TopicName: VpnAlarmNotificationTopic
      Subscription:
        - Endpoint: <your.address@somewhere.com>
          Protocol: email

Outputs:
  VpnAlarmNotificationTopicArn:
    Description: 'Output the arn of the alarm topic.'
    Value: !Ref alarmTopic
    Export:
      Name: VpnAlarmNotificationTopicArn
After the SNS topic is deployed, the next step is to deploy the actual alarms. Plug in the ID and outside tunnel IPs of the VPN connection you want to monitor. If you don’t have a VPN connection deployed in your AWS account and want to set one up, take a look at the documentation here. If not, then this concept can just as easily be adapted to another type of resource, you’ll just have to do some tweaking. This template deploys an alarm for each of the two tunnels in a site-to-site VPN connection. These alarms trigger if their tunnel is in a down state for 3 consecutive 60 second periods, sending a message to the SNS topic.

VPN alarm template:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
Description: Add Cloudwatch alarms for a vpn connection

Parameters:
  vpnConnectionId:
    Type: String
    Default: <yourVpnConnectionId>
  tunnelIp0:
    Type: String
    Default: <yourFirstOutsideTunnelIp>
  tunnelIp1:
    Type: String
    Default: <yourSecondOutsideTunnelId>

Resources:
  vpnDownAlarm0:
    Type: "AWS::CloudWatch::Alarm"
    Properties:
      AlarmDescription: !Sub "Alarm for the vpn tunnel ${tunnelIp0} to ${vpnConnectionId}"
      Dimensions:
        - Name: VpnId
          Value: !Sub "${vpnConnectionId}"
        - Name: TunnelIpAddress
          Value: !Sub "${tunnelIp0}"
      AlarmActions:
        - !ImportValue VpnAlarmNotificationTopicArn
      Threshold: 0
      ComparisonOperator: LessThanOrEqualToThreshold
      EvaluationPeriods: 3
      Period: 60
      Statistic: Average
      MetricName: TunnelState
      Namespace: "AWS/VPN"

  vpnDownAlarm1:
    Type: "AWS::CloudWatch::Alarm"
    Properties:
      AlarmDescription: !Sub "Alarm for the vpn tunnel ${tunnelIp1} to ${vpnConnectionId}"
      Dimensions:
        - Name: VpnId
          Value: !Sub "${vpnConnectionId}"
        - Name: TunnelIpAddress
          Value: !Sub "${tunnelIp1}"
      AlarmActions:
        - !ImportValue VpnAlarmNotificationTopicArn
      Threshold: 0
      ComparisonOperator: LessThanOrEqualToThreshold
      EvaluationPeriods: 3
      Period: 60
      Statistic: Average
      MetricName: TunnelState
      Namespace: "AWS/VPN"

After the alarms are deployed, we will use the serverless framework to deploy the Lambda function that will be sending our Slack notifications. For this to work, we must first create a slack bot, then add the OAuth token and channel id to the SSM Parameter Store. Once we have those parameters, we can deploy our serverless project. The Lambda function is configured to trigger whenever the SNS topic sends a message, so that we receive a notification every time one of our VPN connection tunnels goes into an alarm state.

Serverless YML file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
service: SlackNotifier

plugins:
  - serverless-pseudo-parameters

frameworkVersion: ">=1.34.0"

layers:
  Slack:
    path: layers/slack-layer

package:
  individually: true
  exclude:
    - README.md
    - package_script.txt
    - src/venv/**
    - node_modules/**
    - package.json
    - package-lock.json

provider:
  name: aws
  runtime: python3.7
  stage: ${opt:stage, 'development'}
  region: ca-central-1
  app: ${opt:app,'insights'}
  memorySize: 128
  iamRoleStatements:
    - Effect: Allow
      Action:
        - "ssm:DescribeParameters"
      Resource:
        - "*"
    - Effect: Allow
      Action:
        - "ssm:GetParameters"
        - "ssm:GetParameter"
      Resource:
        - "arn:aws:ssm:#{AWS::Region}:#{AWS::AccountId}:parameter/slack_channel"
        - "arn:aws:ssm:#{AWS::Region}:#{AWS::AccountId}:parameter/slack_oauth"

functions:
  PipelineNotifier:
    description: If a monitored VPN tunnel goes down in this account, send a message in slack.
    handler: src/SlackNotifier.lambda_handler
    layers:
      - {Ref: SlackLambdaLayer}
    events:
      - sns:
          arn: ${cf:alarm-sns-topic.VpnAlarmNotificationTopicArn}

Here is the python code that makes up the body of the Lambda function:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
import logging
import os
from dotenv import load_dotenv
from slack import WebClient
from slack.errors import SlackApiError
import json
import boto3

TOKEN = 'slack_oauth'
CHANNEL = 'slack_channel'


def lambda_handler(event, context):
    print(json.dumps(event, indent=4))
    logging.basicConfig(level=logging.DEBUG)
    load_dotenv()
    client = WebClient(token=get_parameter(TOKEN))
    print(event)
    msg = 'VPN connection down, alarm triggered: \n\t{}'.format(event['Records'][0]['Sns']['Subject'])'
    response = client.chat_postMessage(
        channel=get_parameter(CHANNEL),
        text=msg
    )
    print(response)


def get_parameter(param_name):
    ssm_client = boto3.client('ssm')
    response = ssm_client.get_parameter(
        Name=param_name,
        WithDecryption=True
    )
    print(response)
    return response['Parameter']['Value']

The code is pretty simple and doesn’t include any error handling, but it should send a notification to your Slack channel of choice, provided the slack bot is set up correctly. We used SSM to store our parameters to avoid the bad practice of hard-coding OAuth credentials into our function, which can be a big security risk.

Once the Lambda function is deployed, you should start seeing both email and Slack messages if one of your tunnels goes down for more than 3 minutes at a time. This kind of alarm solution is very versatile, thanks to the variety of resource metrics you can track and the response possibilities. That potential makes CloudWatch alarms another great tool to have in your monitoring toolkit that shouldn’t be overlooked.


Travers Annan


Orbit

Like what you read? Why not subscribe to the weekly Orbit newsletter and get content before everyone else?