Plug S3 plugin changes into multi-export S3 origins

[jhiemstrawisc]

This PR plugs upstream changes in the S3 plugin for XRootD into Pelican. Under the new setup, here are a few example ways to configure an S3 origin:

# Set up an origin using the full exports blocks
Origin:
  StorageType: "s3"
  # Some S3 config options remain top-level
  S3UrlStyle: <path or virtual>
  S3Region: us-east-1
  S3ServiceUrl: https://my-s3-url.com (eg https://s3dev.chtc.wisc.edu)
  Exports:
    # Other S3 configurations move into the exports block
    - S3Bucket: first-bucket
      S3AccessKeyfile: /path/to/access.key (this bucket requires auth)
      S3SecretKeyfile: /path/to/secret.key (this bucket requires auth)
      FederationPrefix: /second/prefix
      Capabilities: [ some capabilities here ]
   - S3Bucket: second-bucket
      FederationPrefix: /second/prefix
      Capabilities: [ some capabilities here ]

# Set up an origin using the old top-level config options
Origin:
  StorageType: "s3"
  S3UrlStyle: <path or virtual>
  S3Region: us-east-1
  S3ServiceUrl: https://my-s3-url.com (eg https://s3dev.chtc.wisc.edu)
  S3Bucket: my-bucket
  S3AccessKeyfile: /path/to/access.key (this bucket requires auth)
  S3SecretKeyfile: /path/to/secret.key (this bucket requires auth)
  FederationPrefix: /my/prefix
  # Various capabilities omitted for brevity

# Please don't do it this way, but in theory it's possible. ExportVolumes is only really meant for CLI/env var compat.
Origin:
  StorageType: "s3"
  S3UrlStyle: <path or virtual>
  S3Region: us-east-1
  S3ServiceUrl: https://my-s3-url.com (eg https://s3dev.chtc.wisc.edu)
  S3AccessKeyfile: /path/to/access.key (this bucket requires auth)
  S3SecretKeyfile: /path/to/secret.key (this bucket requires auth)
  ExportVolumes:
    - "my-bucket:/first/prefix"
    - "different-bucket:/second/prefix
  # Various capabilities omitted for brevity

And finally, from the command line:

pelican origin serve -m s3 -v my-bucket:/my/prefix --service-url https://s3.us-east-1.amazonaws.com --region us-east-1 --bucket-access-keyfile /path/to/access.key --bucket-secret-keyfile /path/to/secret.key

Here's something funky I did (open to negative feedback on it):
Right now we have an origin that exports all of AWS public data, and to make something similar possible under this new setup, we need a way to tell Pelican how to export an entire S3 endpoint, potentially without knowing all the buckets (many thousands in the case of AWS open data). I achieved this by deciding that to export an entire S3 endpoint, no bucket should be provided, and we assume ALL buckets at the endpoint are public. For example, this config will export all of AWS open data:

Origin:
  StorageType: "s3"
  S3UrlStyle: "path"
  S3Region: "us-east-1"
  S3ServiceUrl:  https://s3.us-east-1.amazonaws.com
  Exports:
    - FederationPrefix: /aws-open-data
      Capabilities: ["PublicReads", "Writes", "Listings", "DirectReads"]

The peculiarity in this setup is that unlike other S3 exports where the bucket name is abstracted away from the user by FederationPrefix, in this setup objects are accessed by /aws-open-data/<bucket>/<object>

For example, my usual test file is to get the file MD5SUMS from the noaa-wod-pds bucket. In this setup, it comes from /aws-open-data/noaa-wod-pds/MD5SUMS

[jhiemstrawisc]

@brianhlin This is the PR I indicated I'd like you to glance over with an eye toward "how do people actually configure S3 origins". The PR description should have the various setup bits, and I think you know how all of that gets converted to env vars. Let me know if you see any issues!

[brianhlin]

Nothing jumps out at me immediately but I think I need a little bit to digest the configs.

I achieved this by deciding that to export an entire S3 endpoint, no bucket should be provided, and we assume ALL buckets at the endpoint are public

Seems potentially dangerous. Maybe we should have folks opt into this with a special config option?

[turetske]

I achieved this by deciding that to export an entire S3 endpoint, no bucket should be provided, and we assume ALL buckets at the endpoint are public.

What happens if this is done and a bucket at the endpoint isn't public?

[jhiemstrawisc]

I achieved this by deciding that to export an entire S3 endpoint, no bucket should be provided, and we assume ALL buckets at the endpoint are public.

What happens if this is done and a bucket at the endpoint isn't public?

There are no associated S3 credentials, so everything is prevented by lack of authentication.

[brianhlin]

This config makes it look like S3 and POSIX origins are going to be mutually exclusive. Is that intentional?

There are no associated S3 credentials, so everything is prevented by lack of authentication.

Even still, it feels like we're entering foot gun territory here

[jhiemstrawisc]

This config makes it look like S3 and POSIX origins are going to be mutually exclusive. Is that intentional?

Yep, that's always been the case. We can either operate in S3 mode or posix mode.

[turetske]

Is there any way to get an end-to-end s3 test going with both a configuration file setup and a command line setup? Or is that really not available because we don't have a good way of setting up a test S3 origin?

[jhiemstrawisc]

Is there any way to get an end-to-end s3 test going with both a configuration file setup and a command line setup? Or is that really not available because we don't have a good way of setting up a test S3 origin?

I'm happy to write one, but I don't know the best way to set up an S3 endpoint in the process. I can take a peek at programmatically creating a bucket and S3 credentials in Minio, but wasn't able to get that working when I tried previously. One alternative is to spin up the origin and point it at an AWS open data bucket, but that sounds like a test that's asking for trouble. If you think it's still worth the risk, I'll set it up.

[turetske]

Is there any way to get an end-to-end s3 test going with both a configuration file setup and a command line setup? Or is that really not available because we don't have a good way of setting up a test S3 origin?

I'm happy to write one, but I don't know the best way to set up an S3 endpoint in the process. I can take a peek at programmatically creating a bucket and S3 credentials in Minio, but wasn't able to get that working when I tried previously. One alternative is to spin up the origin and point it at an AWS open data bucket, but that sounds like a test that's asking for trouble. If you think it's still worth the risk, I'll set it up.

That's fair. I guess I would like confirmation from you that you've tested the following locally:

1. Tested both the command line and configuration file setups with S3 with both the bucket name provided and not provided
2. Pull the data via a cache that has access to an S3 origin (one with the bucket name provided, one with the bucket name not provided)

[jhiemstrawisc]

Okay, I got rid of the Minio dependency and pointed the origin tests at an AWS endpoint serving historical data (which, being historical, should never change). That allowed me to test various configuration setups and make sure the file pulled had the correct contents.

[turetske]

Mostly questions I want clarified.

[turetske]

Very tentatively approving assuming all the local tests path.

If everything fails after merging even with the dev-container changes, revert the PR.

[jhiemstrawisc]

Just adding breadcrumbs here in case we end up needing them -- we're merging even though tests aren't passing because
these tests don't actually run with the container changes I've also set up in this PR to install the new upstream S3 plugin version, and there's no way to get them to do that UNLESS we do that in a separate PR and merge that first.

The problem is that the upstream changes are breaking (even though we keep the configuration in Pelican the same), so merging the updated container will break lots of other stuff. All the tests for this PR are currently passing when I run locally. Famous last words!