An AWS lambda containing utility functions for AWS batch
Contributions are welcome.
Install the following applications:
Run pipenv install --dev
to install both production and development
requirements, and pipenv shell
to activate the virtual environment. For more
information see the pipenv docs.
After activating the virtual environment, run pre-commit install
to install
the pre-commit git hook.
First, make any needed updates to the base requirements in Pipfile
, then use
pipenv
to regenerate both Pipfile.lock
and requirements.txt
.
$ pipenv update --dev
We use pipenv
to control versions in testing, but sam
relies on
requirements.txt
directly for building the lambda artifact, so we dynamically
generate requirements.txt
from Pipfile.lock
before building the artifact.
The file must be created in the CodeUri
directory specified in
template.yaml
.
$ pipenv requirements > requirements.txt
Additionally, pre-commit
manages its own requirements.
$ pre-commit autoupdate
Use a Lambda-like docker container to build the Lambda artifact
$ sam build --use-container
Tests are defined in the tests
folder in this project, and dependencies are
managed with pipenv
. Install the development dependencies and run the tests
using coverage
.
$ pipenv run coverage run -m pytest tests/ -svv
Automated testing will upload coverage results to Coveralls.
Running integration tests requires docker
$ sam local invoke HelloWorldFunction --event events/event.json
Deployments are sent to the
Sage cloudformation repository
which requires permissions to upload to Sage
bootstrap-awss3cloudformationbucket-19qromfd235z9
and
essentials-awss3lambdaartifactsbucket-x29ftznj6pqw
buckets.
sam package --template-file .aws-sam/build/template.yaml \
--s3-bucket essentials-awss3lambdaartifactsbucket-x29ftznj6pqw \
--output-template-file .aws-sam/build/lambda-batch-trigger.yaml
aws s3 cp .aws-sam/build/lambda-batch-trigger.yaml s3://bootstrap-awss3cloudformationbucket-19qromfd235z9/lambda-batch-trigger/master/
Publishing the lambda makes it available in your AWS account. It will be accessible in the serverless application repository.
sam publish --template .aws-sam/build/lambda-batch-trigger.yaml
Making the lambda publicly accessible makes it available in the global AWS serverless application repository
aws serverlessrepo put-application-policy \
--application-id <lambda ARN> \
--statements Principals=*,Actions=Deploy
The list of inputs for this lambda can be found in the Parameters
section of the
template.yaml file.
We recommend installing this lambda as a nested cloudformation stack.
Add this snippet to the parent cloudformation stack:
LambdaBatchTrigger:
Type: 'AWS::CloudFormation::Stack'
Properties:
TemplateURL: 'https://bootstrap-awss3cloudformationbucket-19qromfd235z9.s3.amazonaws.com/lambda-batch-trigger/master/lambda-batch-trigger.yaml'
Parameters:
JobName: !Sub '${AWS::StackName}-my-job'
JobQueue: !Ref JobQueue
JobDefinition: !Ref JobDefition
Then deploy the parent stack.
Alternatively it can be deployed with sceptre by creating the following file config/prod/lambda-batch-trigger.yaml
template:
type: http
url: https://bootstrap-awss3cloudformationbucket-19qromfd235z9.s3.amazonaws.com/lambda-batch-trigger/master/lambda-batch-trigger.yaml
stack_name: "lambda-batch-trigger"
stack_tags:
Department: "Platform"
Project: "Infrastructure"
OwnerEmail: "[email protected]"
parameters:
JobName: my-batch-job
JobQueue: !stack_output_external script-runner::JobQueueArn
JobDefinition: !stack_output_external script-runner::JobDefinitionArn
Install the lambda using sceptre:
sceptre --var "profile=my-profile" --var "region=us-east-1" launch prod/lambda-batch-trigger.yaml
Steps to deploy from AWS console.
- Login to AWS
- Access the serverless application repository -> Available Applications
- Select application to install
- Enter Application settings
- Click Deploy
Once this lambda is deployed as a cloudformation stack, locate the SubmitJobApi
output of the
stack and make a request to it with the curl command. It will look something like this
curl https://jmu6d85zp3.execute-api.us-east-1.amazonaws.com/Prod/submitjob
That should trigger the batch job.
This lambda writes debug messages out to a cloudwatch log group, it should be under
/aws/lambda/..
. Locate it and view the messages.
We have setup our CI to automate a releases. To kick off the process just create a tag (i.e 0.0.1) and push to the repo. The tag must be the same number as the current version in template.yaml. Our CI will do the work of deploying and publishing the lambda.