Unit Challenge

Challenge for OutSystems DevOps Experts.

Solution Outline

This is a description of the main steps taken to solve the challenge.

AWS

1. Created an S3 bucket (iislogs-upload) to upload the IIS log files to.

2. Created a Lambda function (iislogs-importer) triggered by "All object create events" in the S3 bucket.
   Used the provided Python code with some minor changes.
   Gave it 128 MB memory and a 30 sec timeout.
   Source code is available here.

3. Created a CloudWatch log group (/aws/lambda/iislogs-importer) for the Lambda function to write into.

4. Created a Role (iislogs-role) to grant the Lambda function access to the S3 bucket and CloudWatch Logs.
   The role actually uses a policy for that (iislogs-policy), which restricts the access to the aforementioned S3 bucket and CloudWatch log group.

5. Created a CloudFormation template to automate the creation (and deletion) of all of the above.
   Source code is available here.

Functionbeat

1. Configured Functionbeat to:

• Use the index lifecycle management (ILM) feature in Elasticsearch
• Not touch the index template
• Use an ingest pipeline in Elasticsearch

Full configuration file is available here.

Elastic Cloud

1. Created a template for all iislogs* indexes in Elasticsearch.
   Source code is available here.

2. Created an ingest pipeline to set the IIS date and time (present in the log file that is uploaded) as the @timestamp field for the Elasticsearch documents.
   Source code is available here.

3. Created the iislogs* index pattern in Kibana and a dashboard that puts the 2 requested visualisations together.
   Created user dashboard_user to access Kibana and see this dashboard. Password provided via a separate channel.

To view data in the Kibana dashboard, make sure to set the time range to Jul 20, 2019 @ 00:00:00.000 → Jul 20, 2019 @ 23:30:00.00, as this is the time range for the data in the example log file that was made available here.

Room for Improvements

For a better production grade solution, some improvements can be made.

1. Take the code of the Labmbda function out of the CloudFormation template file.
   The Labmbda function code was copied and pasted as inline in the CloudFormation template file.