Monitoring AWS Health Events


Travers Annan

Monitoring AWS Health Events

Every so often, infrastructure in an AWS availability zone (or region) breaks down or a bug is discovered with a service. When this happens, AWS sends a notification to affected accounts and an email to the account owner. That’s pretty useful if you are the sole owner of an account, but what if you need those emails to reach multiple users? You could always forward them, but that requires action on your part. Instead, let’s take you out of the equation by setting up a system that will automatically forward those health notifications to a group of users.

In order to set up this notification system, we will be using the Serverless framework to deploy a Lambda function, an EventBridge rule, and an SNS topic. The rule will listen for events generated by the AWS Health service and pass them to the Lambda function, which will push the relevant event information to the SNS topic, sending a notification email to the users.

1
2
3
4
5
{
  "source": [
    "Aws.health"
  ]
}

One thing to note about the AWS Health service is that it only generates events on EventBridge if your account contains resources affected by AWS Health issues, so you may see Health Dashboard notifications that this solution does not capture. This just means that the notification was not relevant to your current AWS environment.

Once we have the event structure, we can set up a Lambda function to deliver our notifications. The Serverless framework is my preferred Lambda deployment tool because of how quickly you can get a project up and running. Here is the code that configures and deploys the notification function:

1
2
3
4
5
6
7
8
9
functions:
  HealthNotifier:
    handler: src/sns-health-notifier.lambda_handler
    events:
    - eventBridge:
      # Mind your indentation
        pattern:
          source:
            - 'aws.health'

And here is the python code for the actual notification function:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
import boto3
import json


def lambda_handler(event, context):
    sns_client = boto3.client('sns')
    topicName = 'snsScheduledChangesHealthDashboard'

    topic = createTopic(sns_client, topicName)
    eventJson = json.dumps(event, sort_keys=True, indent=4)
    publish(sns, topic, eventJson)


def createTopic(client, name):
    # create_topic is idempotent, i.e. if the requester already owns a   topic with
    # the specified name, that topic's ARN is returned without creating a new topic.
    topic = client.create_topic(Name=name, Tags=[{
        'Key': 'purpose',
        'Value': 'snsHealthDashNotifier'
    }])
    return topic['TopicArn']


def publish(client, topic, msg):
    response = client.publish(
        TopicArn=topic,
        Message=msg,
        Subject='New Health Event Detected',
        MessageAttributes={
            'string': {
                'DataType': 'String',
                'StringValue': 'String'
            }
        }
    )

The notifier is a fairly simple program, it takes the event JSON sent to the function by EventBridge and dumps it into a string. From there it calls the create_topic() SNS API(), which returns the ARN of an SNS topic named snsHealthNotifier or creates one if that topic does not already exist. With that topic ARN, we are able to call the publish() SNS API, which sends the event string as an email to all subscribers of the snsHealthNotifier topic. The email will contain the full event JSON, which is most likely far more information than your users actually need, but once you see the structure of an AWS Health event you can refine this function to parse out the most relevant information.

This solution is not exclusive to AWS Health, you could use it to monitor any number of important events in an AWS account as long as you know the patterns of those events. For instance, what if you wanted to know when a new EC2 instance is stood up/down in your account, or if someone was messing with your S3 buckets?

You could set up some rules in your serveless.yml file that look something like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
- eventBridge:
        pattern:
          source:
            - 'aws.ec2'
          detail-type:
            - 'EC2 Instance State-change Notification'
          detail:
            state:
              - 'running'
              - 'terminated'
- eventBridge:
        pattern:
          source:
            - 'aws.s3'
          detail-type:
            - 'AWS API Call via CloudTrail'
          detail:
            eventSource:
              - 's3.amazonaws.com'
            eventName:
              - 'DeleteBucket'
              - 'DeleteBucketPolicy'
              - 'PutBucketPolicy'

The only issue with SNS is that the message formatting and style leaves something to be desired. For this use case, however, a pretty email is not really the priority, and SNS fits the bill just fine. If you really need to adjust the formatting of these emails, you might want to look into using something like SES (Simple Email Service) instead. Another thing you could do is use the Slack API to send notifications directly to your work channels, skipping the email part completely.

Example simple Slack messaging Lambda function:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
import logging
import os
from dotenv import load_dotenv
from slack import WebClient
import json


# The SSM parameters with your oauth key and slack channel name
TOKEN = 'slack_oauth'
CHANNEL = 'slack_channel'


def lambda_handler(event, context):
    logging.basicConfig(level=logging.DEBUG)
    load_dotenv()    
    
    event_str = json.dumps(event, indent=4))

    client = WebClient(token=get_parameter(TOKEN))

    response = client.chat_postMessage(
        channel=get_parameter(CHANNEL),
        text=event_str
    )
    print(response)


def get_parameter(param_name):
    ssm_client = boto3.client('ssm')
    response = ssm_client.get_parameter(
        Name=param_name,
        WithDecryption=True
    )
    print(response)
    return response['Parameter']['Value']

This simple script will send your event JSON to a Slack channel of your choice, provided you have a Slack bot with Oauth and channel-write permissions set up. Furthermore, because these solutions are Lambda functions, you could tie in any number of services with the right API calls, like stopping an EC2 instance or logging some data to CloudWatch. That’s the real power of EventBridge, the ability to tie together multiple services through event triggers. When it comes to event notifications in AWS, the sky really is the limit, just be careful not to get lost in the cloud(s).


Travers Annan


Orbit

Like what you read? Why not subscribe to the weekly Orbit newsletter and get content before everyone else?