Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement access to public S3 bucket without credentials #286

Closed
wants to merge 16 commits into from
Closed

Implement access to public S3 bucket without credentials #286

Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
bf198dd
Upgrade dependencies
kaklakariada Nov 6, 2023
2f1535a
Adapt extension unit tests
kaklakariada Nov 6, 2023
4b960af
Fix vulnerability in npm dependency
kaklakariada Nov 6, 2023
ecf492d
Enable integration tests after EM upgrade
kaklakariada Nov 6, 2023
f14c0ad
Upgrade jersey-common to fix failing extension tests
kaklakariada Nov 6, 2023
356d113
Adapt tests to new error message
kaklakariada Nov 6, 2023
a80915b
Use base class for integration tests
kaklakariada Nov 6, 2023
3d5a776
Add changelog entry
kaklakariada Nov 7, 2023
608442b
Add release date
kaklakariada Nov 7, 2023
a74752d
Upgrade Exasol DB version
kaklakariada Nov 8, 2023
533f1fd
Upgrade plugin
kaklakariada Nov 8, 2023
1685141
Remove unnecessary exclusion
kaklakariada Nov 8, 2023
f6b3de3
Upgrade testcontainer version
kaklakariada Nov 8, 2023
3e69b1e
Apply suggestions from code review
kaklakariada Nov 8, 2023
0ddcc55
Implement aws anonymous mode
Shmuma Nov 8, 2023
b8c3d83
Merge branch 'main' into feature/283_public_s3
Shmuma Nov 8, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 16 additions & 6 deletions src/main/scala/com/exasol/cloudetl/bucket/S3Bucket.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,9 @@ final case class S3Bucket(path: String, params: StorageProperties) extends Bucke
)
}

private[this] def isAnonymousAWSParams(properties: StorageProperties): Boolean =
properties.getString(S3_ACCESS_KEY).isEmpty && properties.getString(S3_SECRET_KEY).isEmpty

/**
* @inheritdoc
*
Expand Down Expand Up @@ -83,15 +86,22 @@ final case class S3Bucket(path: String, params: StorageProperties) extends Bucke
properties
}

conf.set("fs.s3a.access.key", mergedProperties.getString(S3_ACCESS_KEY))
conf.set("fs.s3a.secret.key", mergedProperties.getString(S3_SECRET_KEY))

if (mergedProperties.containsKey(S3_SESSION_TOKEN)) {
if (isAnonymousAWSParams(mergedProperties)) {
conf.set(
"fs.s3a.aws.credentials.provider",
classOf[org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider].getName()
classOf[org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider].getName()
)
conf.set("fs.s3a.session.token", mergedProperties.getString(S3_SESSION_TOKEN))
} else {
conf.set("fs.s3a.access.key", mergedProperties.getString(S3_ACCESS_KEY))
conf.set("fs.s3a.secret.key", mergedProperties.getString(S3_SECRET_KEY))

if (mergedProperties.containsKey(S3_SESSION_TOKEN)) {
conf.set(
"fs.s3a.aws.credentials.provider",
classOf[org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider].getName()
)
conf.set("fs.s3a.session.token", mergedProperties.getString(S3_SESSION_TOKEN))
}
}

properties.getProxyHost().foreach { proxyHost =>
Expand Down