We send (and receive) a lot of email both for the @dwyl App and our newsletter.
We need a simple, scalable & maintainable way of sending email, and most importantly we needed to know with certainty:

• Are our emails being delivered?
• How many emails are bouncing?
• Are we attempting to re-send email to addresses that have "bounced"? (i.e. wasting money?!)
• Are people opening / reading the email?
• Do people engage with the content of the email? (click through)
• If someone no longer wants to receive emails (too many or not relevant), do we have a reliable way for them to unsubscribe?

This project is our quest to answer these questions.

The aws-ses-lambda function does three related things¹:

1. Send emails.
2. Parse AWS SNS notifications related to the emails that were sent.
3. Save the parsed SNS notification data for aggregation and visualisation.

The How? section below explains how each of these functions works.

This diagram explains the context where aws-ses-lambda is used:

Edit this diagram

The Email App receives requests from the Auth App and and triggers the aws-ses-lambda function.

The aws-ses-lambda function Sends email and handles SNS notifications for bounce events.

As the name of this project suggests, we are using AWS Lambda, to handle all email-related tasks via AWS SES.

If you (or anyone else on your team) are new to AWS Lambda, see: github.com/dwyl/learn-aws-lambda

In this section we will break down how the lambda works.

Thanks to the work we did earlier on sendemail, sending emails using AWS Simple Email Service (SES) from our Lambda function is very simple.

We just need to follow the setup instructions in github.com/dwyl/sendemail#how including creating a /templates directory, then create a handler function:

const sendemail = require('sendemail').email;

module.exports = function send (event, callback) {
  return sendemail(event.template, event, callback);
};

Don't you just love it when things are that simple?!
All the data required for sending an email is received in the Lambda event object.

The required keys in the event object are:

• email - the email address we want to send an email to.
• name - the name of the person we are sending the email to. (if your email messages aren't personal, don't send them!)
• subject - the subject of the email you are sending.
• template - the template you want to send.

It works flawlessly.

The full code is: lib/send.js

After an email is sent using AWS SES, AWS keeps track of the status of the emails e.g delivered, bounce or complaint.
By subscribing to AWS Simple Notification System (SNS) notifications, we can keep track of the status.

There are a few steps for setting up SNS notifications for SES events, so we created detailed setup instructions: SETUP.md

Once you have configured the SNS Topic, used the topic for SES notifications and set the topic as the trigger for the lambda function, it's time to parse the notifications.

Thankfully this is also really simple code!

let json = {};
if(event && event.Records && event.Records.length > 0) {
  const msg = JSON.parse(event.Records[0].Sns.Message);
  json.messageId = msg.mail.messageId;
  json.notificationType = msg.notificationType + ' ' + msg.bounce.bounceType;
}

We are only interested in the messageId and notificationType. This code is included in lib/parse.js

During MVP we are only interested in the emails that bounce. So we are only parsing the bounce event. Gmail does not send delivery notifications, so we will need to implement a workaround. See: dwyl/email#1

More detail on the various SES SNS notifications: https://docs.aws.amazon.com/ses/latest/DeveloperGuide/event-publishing-retrieving-sns-examples.html

Once we have parsed the SNS notifications for SES events, we need to save the data back to our PostgreSQL database so that we can build our analytics dashboard!

This again is pretty simple code; we just invoke http_request with the json data we want to send to the Phoenix App:

const json = parse(event); // parse SNS event see: step 2.
http_request(json, callback); // json data & lambda callback argument

View the complete code in index.js and the supporting http_request function in lib/http_request.js

The http_request function wraps the Node.js core http.request method with a few basic options and allows us to pass in a json Object to send to the Phoenix App.

In order for all parts of the Lambda function to work, we need to ensure that all environment variables are defined.

For the complete list of required environment variables, please see the .env_sample file.

Copy the .env_sample file and create a .env file:

cp .env_sample .env

Then update all the values in the file so that they are the real values.

Once you have a .env file with all the correct environment variables, it's time to deploy the Lambda function to AWS!

Run the following command in your terminal:

npm run deploy

You should see output similar to the following:

- - - - - - - - > Lambda Function Deployed:
{
  FunctionName: 'aws-ses-lambda-v1',
  FunctionArn: 'arn:aws:lambda:eu-west-1:123456789247:function:aws-ses-lambda-v1',
  Runtime: 'nodejs12.x',
  Role: 'arn:aws:iam::123456789247:role/service-role/LambdaExecRole',
  Handler: 'index.handler',
  CodeSize: 8091768,
  Description: 'A complete solution for sending email via AWS SES using Lambda',
  Timeout: 42,
  MemorySize: 128,
  LastModified: '2020-03-05T23:42:56.809+0000',
  CodeSha256: 'jvOg/+8y9UwBcLeTprMRIEvT0ryun1bdjzrAJXAk5m8=',
  Version: '$LATEST',
  Environment: { Variables: { EMAIL_APP_URL: 'phemail.herokuapp.com' } },
  TracingConfig: { Mode: 'PassThrough' },
  RevisionId: '42442cee-d506-4aa5-aec5-d7fb73145a58',
  State: 'Active',
  LastUpdateStatus: 'Successful'
}
- - - - - - - - > took 8.767 seconds

Ensure you follow all the instructions in SETUP.md to get the SNS Topic to trigger the Lambda function for SES notifications.

Enable debugging by setting the NODE_ENV=test environment variable.

Now the latest event will be saved to: https://ademoapp.s3.eu-west-1.amazonaws.com/event.json

And SNS messages are saved to: https://ademoapp.s3.eu-west-1.amazonaws.com/sns.json

There are way more reasons why we are handcrafting this app than the ones stated above.
We see email as our primary feedback mechanism and thus "operationally strategic", not merely "transactional". i.e. not something to be "outsourced" to a "black box" provider that "takes care of everything" for us. We want to have full control and deep insights into our email system.
By using a decoupled lambda function to send email and subscribe to SNS events we keep all the AWS specific functionality in a single place. This is easy to reason about, maintain and extend when required. In the future, if we decide to switch email sending provider, (or run our own email service), we can simply re-write the sendemail and parse_notification functions and not need to touch our email analytics dashboard at all!

For now SES is by far the cheapest and superbly reliable way to send email. We are very happy to let AWS take care of this part of our stack.

¹ The aws-ses-lambda function does 3 things because they relate to the unifying theme of sending email via SES and tracking the status of the sent emails via SNS. We could have split these 3 bits of functionality into separate repositories and deploy them separately as distinct lambda functions, however in our experience having too many lambda functions can quickly become a maintenance headache. We chose to group them together because they are small, easy to reason about and work well as a team! If you feel strongly about the UNIX Philosophy definitely split out the functions in your own fork/implementation.
The code for this Lambda function is less than 100 lines and can be read in 10 minutes. The sendemail module which the Lambda uses to send emails via AWS SES is 38 lines of code. See: lib/index.js it's mostly comments which make it very beginner friendly.