AWS Fargate and Fluentd based Log Aggregator

A reference architecture for running a Fluentd Log Aggregator on AWS Fargate, which forwards logs to Kinesis Firehose. See Building a scalable log solution aggregator with AWS Fargate, Fluentd, and Amazon Kinesis Data Firehose.

Building the Fluentd Docker image

First, customize the Fluentd configuration file in src. Then, build it into the docker image:

docker build -t fluentd-aggregator src

Launching the CloudFormation template

The Fluentd aggregator is designed to be deployed to a VPC with private subnets behind a NAT Gateway. The aws-samples/ecs-refarch-cloudformation repository has a CloudFormation template which will create a VPC with two public and two private subnets: vpc.yaml.

Then create an ECS Cluster (if you don't already have one):

aws ecs create-cluster --cluster-name fargate-fluentd-tutorial

Next, create an ECS Task Execution IAM role.

Finally, launch template.yml:

aws cloudformation deploy --template-file template.yml --stack-name aggregator-service \
--parameter-overrides EnvironmentName=fluentd-aggregator-service \
DockerImage=<Fluentd image> \
VPC=<your VPC ID> \
Subnets=<private subnet 1, private subnet 2> \
Cluster=fargate-fluentd-tutorial \
ExecutionRoleArn=<your task execution IAM role ARN> \
MinTasks=2 \
MaxTasks=4 \
--capabilities CAPABILITY_NAMED_IAM

License

This library is licensed under the Apache 2.0 License.