terraform-ghost-log-forwarder

Terraform module which deploys a Ghost log forwarder to AWS for sending ALB logs to the Ghost platform.

Refer to the Log Based Discovery documentation for more on how this is used in the Ghost platform.

Considerations

• Ensure resources created in AWS are unique to avoid naming conflict errors.
• S3 bucket versioning must be enabled on the source bucket to allow for S3 bucket replication to be configured.
• Only replicate new log files to the log forwarder bucket. Do not replicate existing objects.
• The ghost provider requires an API key with read:log_forwarders and write:log_forwarders permissions.
  • Use the API Keys page to generate a new key.
  • Using an invalid or expired API key will result in an unexpected status 401 error.

Example

The following example deploys a log forwarder and configures S3 ojbect replication to copy log files from an existing source S3 bucket.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.81.0"
    }
    ghost = {
      source  = "ghostsecurity/ghost",
      version = "~> 0.1.0"
    }
  }
}

provider "aws" {
  region = "us-east-2"
}

# Configure the provider. A valid API key must exist in the GHOST_API_KEY
# environment variable with read:log_forwarders and write:log_forwarders permissions.
provider "ghost" {
}

# This must be updated to reference an existing S3 bucket that is receiving logs
# from your application load balancer.
data "aws_s3_bucket" "source" {
  bucket = "source-bucket-name"
}

# Deploy the Ghost log forwarder.
# Change the name to something meaningful in your organization.
module "dev-alb-forwarder" {
  source = "ghostsecurity/log-forwarder/ghost"
  name   = "example-forwarder"
}

data "aws_s3_bucket" "dest" {
  bucket = module.dev-alb-forwarder.s3_input_bucket
}

# The following resources configure an example S3 replication policy to
# copy logs from the source bucket to the log forwarder bucket so that they
# will be processed and sent to the Ghost platform.
data "aws_iam_policy_document" "assume_role" {
  statement {
    effect = "Allow"

    principals {
      type        = "Service"
      identifiers = ["s3.amazonaws.com"]
    }

    actions = ["sts:AssumeRole"]
  }
}

resource "aws_iam_role" "replication" {
  name               = "example-replication-role"
  assume_role_policy = data.aws_iam_policy_document.assume_role.json
}

data "aws_iam_policy_document" "replication" {
  statement {
    effect = "Allow"

    actions = [
      "s3:GetReplicationConfiguration",
      "s3:ListBucket",
    ]

    resources = [data.aws_s3_bucket.source.arn]
  }

  statement {
    effect = "Allow"

    actions = [
      "s3:GetObjectVersionForReplication",
      "s3:GetObjectVersionAcl",
      "s3:GetObjectVersionTagging",
    ]

    resources = ["${data.aws_s3_bucket.source.arn}/*"]
  }

  statement {
    effect = "Allow"

    actions = [
      "s3:ReplicateObject",
      "s3:ReplicateDelete",
      "s3:ReplicateTags",
    ]

    resources = ["${data.aws_s3_bucket.dest.arn}/*"]
  }
}

resource "aws_iam_policy" "replication" {
  name   = "example-replication-policy"
  policy = data.aws_iam_policy_document.replication.json
}

resource "aws_iam_role_policy_attachment" "replication" {
  role       = aws_iam_role.replication.name
  policy_arn = aws_iam_policy.replication.arn
}

resource "aws_s3_bucket_replication_configuration" "replication" {
  role   = aws_iam_role.replication.arn
  bucket = data.aws_s3_bucket.source.id

  rule {
    filter {
      prefix = "AWSLogs"
    }

    status = "Enabled"

    delete_marker_replication {
      status = "Enabled"
    }

    destination {
      bucket        = data.aws_s3_bucket.dest.arn
      storage_class = "STANDARD"
    }
  }
}

Providers

Name	Version
aws	>= 5.81.0
ghost	>= 0.1.0

Outputs

Name	Description
s3_input_bucket	The name of the input bucket

Inputs

Name	Description	Type	Default	Required
name	The name for this log forwarder as it will be displayed in the Ghost platform UI.	string	n/a	yes
tags	Map of tags to assign to all resources. By default resources are tagged with ghost:forwarder_id and ghost:forwarder_name.	map(string)	{}	no

Resources

Name	Type
aws_iam_policy.lambda_bucket_access	resource
aws_iam_policy.log_shipper_policy	resource
aws_iam_role.log_converter_role	resource
aws_iam_role.log_shipper_role	resource
aws_iam_role_policy_attachment.log_converter_bucket_access	resource
aws_iam_role_policy_attachment.log_converter_logging	resource
aws_iam_role_policy_attachment.log_shipper_role_bucket_access	resource
aws_lambda_function.log_converter	resource
aws_lambda_permission.log_converter	resource
aws_s3_bucket.ingest_bucket	resource
aws_s3_bucket.input_bucket	resource
aws_s3_bucket_lifecycle_configuration.input_bucket	resource
aws_s3_bucket_notification.ingest	resource
aws_s3_bucket_notification.input	resource
aws_s3_bucket_versioning.input	resource
aws_sqs_queue.ingest_bucket_notifications	resource
aws_sqs_queue_policy.ingest_bucket_notifications	resource
ghost_aws_log_source.source	resource
ghost_log_forwarder.forwarder	resource
aws_caller_identity.current	data source
aws_iam_policy.basic_execution_role	data source
aws_iam_policy_document.lambda_assume_role	data source
aws_iam_policy_document.lambda_bucket_access	data source
aws_iam_policy_document.log_shipper_assume_role	data source
aws_iam_policy_document.log_shipper_policy	data source
aws_iam_policy_document.s3_object_notification	data source
aws_region.current	data source