S3 Plugin: Implement Server-Side Encryption option #8646

msolters · 2024-03-28T03:13:59Z

Introduces a server_side_encryption parameter for the S3 [OUTPUT] plugin. Possible values are AES256 and aws:kms, as per AWS API documentation: https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html#API_PutObject_ResponseSyntax
If either value is provided, the x-amz-server-side-encryption header will be included with S3 requests with the corresponding value set.

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

Example configuration file for the change
Debug log output from testing the change

Example config:

---
service:
  http_server: off
  http_listen: 0.0.0.0
  http_port: 2029
  flush: 1
  daemon: off
  log_level: info

pipeline:
  inputs:
    - name: tail
      read_from_head: true
      path: /etc/fluent/logs/*
      refresh_interval: 1
      ignore_older: 1m

  outputs:
    - name: stdout
      match: "*"
    - name: s3
      match: "*"
      bucket: "foobar"
      server_side_encryption: AES256

This shows the S3 output config loading correctly. Obviously the bucket isn't real in this config but the config load throws no error:

root@1969e5ffe6ff:/# fluent-bit/bin/fluent-bit -c /cfg/config.yaml
Fluent Bit v3.0.1
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

___________.__                        __    __________.__  __          ________
\_   _____/|  |  __ __   ____   _____/  |_  \______   \__|/  |_  ___  _\_____  \
 |    __)  |  | |  |  \_/ __ \ /    \   __\  |    |  _/  \   __\ \  \/ / _(__  <
 |     \   |  |_|  |  /\  ___/|   |  \  |    |    |   \  ||  |    \   / /       \
 \___  /   |____/____/  \___  >___|  /__|    |______  /__||__|     \_/ /______  /
     \/                     \/     \/               \/                        \/

[2024/03/28 02:40:34] [ info] [fluent bit] version=3.0.1, commit=62b337ef78, pid=12
[2024/03/28 02:40:34] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/03/28 02:40:34] [ info] [cmetrics] version=0.7.0
[2024/03/28 02:40:34] [ info] [ctraces ] version=0.4.0
[2024/03/28 02:40:34] [ info] [input:tail:tail.0] initializing
[2024/03/28 02:40:34] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2024/03/28 02:40:34] [error] [input:tail:tail.0] read error, check permissions: /etc/fluent/logs/*
[2024/03/28 02:40:34] [ warn] [input:tail:tail.0] error scanning path: /etc/fluent/logs/*
[2024/03/28 02:40:34] [ info] [output:stdout:stdout.0] worker #0 started
[2024/03/28 02:40:34] [ info] [output:s3:s3.1] Using upload size 100000000 bytes
[2024/03/28 02:40:34] [error] [/src/fluent-bit/src/flb_http_client.c:1239 errno=111] Connection refused
[2024/03/28 02:40:34] [ warn] [imds] unable to evaluate IMDS version
...

When the bucket is real (and IAM permissions exist) this does correctly upload to SSE-enabled buckets.

A negative example, where server_side_encryption: wumbo, fails to load as expected:

root@1969e5ffe6ff:/# fluent-bit/bin/fluent-bit -c /cfg/config.yaml
Fluent Bit v3.0.1
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

___________.__                        __    __________.__  __          ________
\_   _____/|  |  __ __   ____   _____/  |_  \______   \__|/  |_  ___  _\_____  \
 |    __)  |  | |  |  \_/ __ \ /    \   __\  |    |  _/  \   __\ \  \/ / _(__  <
 |     \   |  |_|  |  /\  ___/|   |  \  |    |    |   \  ||  |    \   / /       \
 \___  /   |____/____/  \___  >___|  /__|    |______  /__||__|     \_/ /______  /
     \/                     \/     \/               \/                        \/

[2024/03/28 02:42:31] [ info] [fluent bit] version=3.0.1, commit=62b337ef78, pid=27
[2024/03/28 02:42:31] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/03/28 02:42:31] [ info] [cmetrics] version=0.7.0
[2024/03/28 02:42:31] [ info] [ctraces ] version=0.4.0
[2024/03/28 02:42:31] [ info] [input:tail:tail.0] initializing
[2024/03/28 02:42:31] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2024/03/28 02:42:31] [error] [input:tail:tail.0] read error, check permissions: /etc/fluent/logs/*
[2024/03/28 02:42:31] [ warn] [input:tail:tail.0] error scanning path: /etc/fluent/logs/*
[2024/03/28 02:42:31] [ info] [output:stdout:stdout.0] worker #0 started
[2024/03/28 02:42:31] [ info] [output:s3:s3.1] Using upload size 100000000 bytes
[2024/03/28 02:42:31] [error] [aws_compress] unknown sse type: wumbo
[2024/03/28 02:42:31] [error] [output:s3:s3.1] unknown server-side encryption type: wumbo
[2024/03/28 02:42:31] [error] [output] failed to initialize 's3' plugin
[2024/03/28 02:42:31] [error] [engine] output initialization failed
[2024/03/28 02:42:32] [ info] [input] pausing tail.0
[2024/03/28 02:42:32] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2024/03/28 02:42:32] [ info] [output:stdout:stdout.0] thread worker #0 stopped

Starting without server_side_encryption being defined at all also doesn't crash:

root@1969e5ffe6ff:/# fluent-bit/bin/fluent-bit -c /cfg/config.yaml
Fluent Bit v3.0.1
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

___________.__                        __    __________.__  __          ________
\_   _____/|  |  __ __   ____   _____/  |_  \______   \__|/  |_  ___  _\_____  \
 |    __)  |  | |  |  \_/ __ \ /    \   __\  |    |  _/  \   __\ \  \/ / _(__  <
 |     \   |  |_|  |  /\  ___/|   |  \  |    |    |   \  ||  |    \   / /       \
 \___  /   |____/____/  \___  >___|  /__|    |______  /__||__|     \_/ /______  /
     \/                     \/     \/               \/                        \/

[2024/03/28 03:11:07] [ info] [fluent bit] version=3.0.1, commit=62b337ef78, pid=31
[2024/03/28 03:11:07] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/03/28 03:11:07] [ info] [cmetrics] version=0.7.0
[2024/03/28 03:11:07] [ info] [ctraces ] version=0.4.0
[2024/03/28 03:11:07] [ info] [input:tail:tail.0] initializing
[2024/03/28 03:11:07] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2024/03/28 03:11:07] [error] [input:tail:tail.0] read error, check permissions: /etc/fluent/logs/*
[2024/03/28 03:11:07] [ warn] [input:tail:tail.0] error scanning path: /etc/fluent/logs/*
[2024/03/28 03:11:07] [ info] [output:stdout:stdout.0] worker #0 started
[2024/03/28 03:11:07] [ info] [output:s3:s3.1] Using upload size 100000000 bytes
[2024/03/28 03:11:07] [error] [/src/fluent-bit/src/flb_http_client.c:1239 errno=111] Connection refused
[2024/03/28 03:11:07] [ warn] [imds] unable to evaluate IMDS version

Attached Valgrind output that shows no leaks or memory corruption was found

Valgrind heap summary with server_side_encryption: AES256:

[2024/03/28 03:34:43] [ info] [output:s3:s3.1] thread worker #0 stopping...
[2024/03/28 03:34:43] [ info] [output:s3:s3.1] thread worker #0 stopped
==12==
==12== HEAP SUMMARY:
==12==     in use at exit: 0 bytes in 0 blocks
==12==   total heap usage: 8,566 allocs, 8,566 frees, 949,054 bytes allocated
==12==
==12== All heap blocks were freed -- no leaks are possible
==12==
==12== For lists of detected and suppressed errors, rerun with: -s
==12== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

Run local packaging test showing all targets (including any new ones) build.
Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

Documentation required for this feature

Documentation PR here

Backporting

Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

- Introduces a server_side_encryption parameter for the S3 [OUTPUT] plugin. Possible values are AES256 and aws:kms, as per AWS API documentation: https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html#API_PutObject_ResponseSyntax - If either value is provided, the x-amz-server-side-encryption header will be included with S3 requests with the corresponding value set. Signed-off-by: Mark Solters <[email protected]>

PettitWesley · 2024-04-17T22:11:16Z

src/aws/flb_aws_sse.c

+ * AWS plugins that support sse will have these options.
+ * Referenced function should return -1 on error and 0 on success.
+ */
+static const struct sse_option sse_options[] = {


@edsiper is this a recommended style/technique for FLB C code? We have similar bits of code elsewhere in the project that have a set of values but that don't validate them in this way with a object list...

src/aws/flb_aws_sse.c

As noted per feedback this code is unrelated to compression Signed-off-by: Mark Solters <[email protected]>

PettitWesley

Thanks!

github-actions · 2024-08-05T01:56:42Z

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

github-actions · 2024-12-14T02:08:07Z

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

msolters requested review from PettitWesley, edsiper, leonardo-albertovich, fujimotos and koleini as code owners March 28, 2024 03:14

github-actions bot added the docs-required label Mar 28, 2024

msolters force-pushed the s3-server-side-encryption branch from 62b337e to 34a63d5 Compare March 28, 2024 03:38

msolters mentioned this pull request Mar 28, 2024

out_s3: document server_side_encryption fluent/fluent-bit-docs#1347

Open

patrick-stephens added the ok-package-test Run PR packaging tests label Mar 29, 2024

msolters temporarily deployed to pr March 29, 2024 15:19 — with GitHub Actions Inactive

msolters temporarily deployed to pr March 29, 2024 15:39 — with GitHub Actions Inactive

msolters temporarily deployed to pr March 29, 2024 15:40 — with GitHub Actions Inactive

PettitWesley reviewed Apr 17, 2024

View reviewed changes

src/aws/flb_aws_sse.c Outdated Show resolved Hide resolved

PettitWesley previously approved these changes Apr 17, 2024

View reviewed changes

msolters commented May 2, 2024

View reviewed changes

src/aws/flb_aws_sse.c Outdated Show resolved Hide resolved

AWS SSE: Correct error language

dd168e0

As noted per feedback this code is unrelated to compression Signed-off-by: Mark Solters <[email protected]>

msolters dismissed PettitWesley’s stale review via dd168e0 May 2, 2024 03:43

msolters temporarily deployed to pr May 3, 2024 22:15 — with GitHub Actions Inactive

msolters temporarily deployed to pr May 3, 2024 22:16 — with GitHub Actions Inactive

PettitWesley approved these changes May 3, 2024

View reviewed changes

msolters temporarily deployed to pr May 3, 2024 22:36 — with GitHub Actions Inactive

PettitWesley approved these changes May 6, 2024

View reviewed changes

github-actions bot added the Stale label Aug 5, 2024

github-actions bot removed the Stale label Aug 17, 2024

github-actions bot added the Stale label Dec 14, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

S3 Plugin: Implement Server-Side Encryption option #8646

S3 Plugin: Implement Server-Side Encryption option #8646

msolters commented Mar 28, 2024 •

edited

Loading

PettitWesley Apr 17, 2024

PettitWesley left a comment

github-actions bot commented Aug 5, 2024

github-actions bot commented Dec 14, 2024

S3 Plugin: Implement Server-Side Encryption option #8646

Are you sure you want to change the base?

S3 Plugin: Implement Server-Side Encryption option #8646

Conversation

msolters commented Mar 28, 2024 • edited Loading

PettitWesley Apr 17, 2024

Choose a reason for hiding this comment

PettitWesley left a comment

Choose a reason for hiding this comment

github-actions bot commented Aug 5, 2024

github-actions bot commented Dec 14, 2024

msolters commented Mar 28, 2024 •

edited

Loading