AWS Lambda function written in Ruby for recording API Gateway post request data to DynamoDB.
This will take anything that you post to it and record it in a DynamoDB table.
This project serves as an example of how to make a serverless microservice with Ruby and AWS Lambda, and how to manage the deployment and the creation of the cloud DynamoDB table using SAM and AWS CloudFormation.
This is also an exmaple of how to do local development using SAM Local and dynamodb-local.
The original motivator for this data logger is that we needed a web hook for capturing data from a third-party data source for our data warehouse. We don't want to directly connect our data warehouse database to the world through HTTP. So we use DynamoDB as a dead drop. This web hook microservice collects the data postings, and ETL code in the data warehouse comes along later and processes the data.
These instructions are specifically for setting up this project for development using AWS Cloud9.
Create a new Cloud9 environment and clone this Git repository to it. It's simplest if you set it up so that the root of the repository is in ~/environment in the Cloud9 environment.
Once you have the project in ~/environment, cd to that directory.
rvm use 2.5.0 --install
Using Linuxbrew to install SAM works great on the AWS AMI:
SAM Local is just the application server for running Lambda functions locally. It doesn't provide other AWS services. If you want to develop Lambda functions that use DynamoDB, then you will need DynamoDB Local.
First, create a network for SAM Local to reach your local DynamoDB:
docker network create sam-local
Then install and run DynamoDB Local, on that Docker network:
docker run -d -v "$PWD":/dynamodb_local_db -p 8000:8000 --network sam-local --name dynamodb amazon/dynamodb-local
Once you have DynamoDB running on port 8000, create some tables:
rake dynamodb:create
If you need to drop those tables and re-create them, then do this:
rake dynamodb:delete
To scan the current contents of your dynamodb-data-logger-web-hook-items table:
rake dynamodb:scan
Build, and use SAM Local for development:
sam build --base-dir lambda && sam local start-api -p 8080 --docker-network sam-local --env-vars env.json
Port 8080 is important if you're using AWS Cloud9.
The --env-vars parameter loads environment variables from the env.json file.
The -docker-network parameter enables it to connect to the DynamoDB container. SAM Local runs in a container, so without this you can't connect to the database.
Once you have an HTTP server running, you can send an HTTP request to it:
curl -A "DataSourceName" -d "param1=value1¶m2=value2" -X POST "http://localhost:8080/items"
If all goes well, it will return 200 response with a JSON representation of the record that it just posted to DynamoDB.
2019-04-04 21:19:17 127.0.0.1 - - [04/Apr/2019 21:19:17] "POST /items HTTP/1.1" 200 -
{"id":"1611588b-24a5-49ff-9d27-be1bc4397d6a","created_at":"2019-04-04T21:19:17+00:00","body":"param1=value1¶m2=value2","source":"DataSourceName"}
Note that it stores the User-Agent HTTP header as the data source for the record.
You can also bypass the HTTP server and invoke it directly:
sam build --base-dir lambda && echo "{\"body\":\"TEST\", \"source\": \"DataSourceName\"}" | sam local invoke PostSignup --docker-network sam-local --env-vars=env.json
You won't get an HTTP response. Instead, you will see the hash that the function returns:
{"statusCode":200,"body":"{\"id\":\"c094d3e3-e765-4f04-88bc-4c82352f4bd6\",\"created_at\":\"2019-04-04T21:19:46+00:00\",\"body\":\"TEST\",\"source\":\"DataSourceName\"}"}
(The extra whitespace is for formatting, from the Awesome Print gem.)
You can provide a source parameter and the DynamoDB record will be tagged with it. You can either provide that parameter as a ?source= URL parameter, or as one of the event parameters if you're bypassing HTTP. As demonstrated above.
This project uses SAM to deploy to the cloud using AWS CloudFormation.
First, create a bucket for CloudFormation to use during the deployment.
aws s3api create-bucket --bucket dynamodb-data-logger-web-hook
Then deploy with:
rake sam:build && rake sam:package && rake sam:deploy
To deploy to production instead of development:
rake sam:deploy[production]
Each subsequent time that you want to spin up a development environment, do this:
rake develop:start
To stop those things:
rake develop:stop
To stop and restart everything:
rake develop:restart