[Question] What does fluss need to be configured to support Minio's S3?

[biao-lvwan]

Search before asking

• I searched in the issues and found nothing similar.

Fluss version

0.5.0

Minimal reproduce step

1，fluss config remote.data.dir: s3://fluss/storage, appear Region is not set
2，user lakehouse.sh Dynamic session credentials for Fluss: No AWS Credentials

What doesn't meet your expectations?

1,The data can be written to remote.data.dir, but an error message will appear and the Region is not set
2,use lakehouse.sh ,Can't read data from Flink's S3 configuration? Where do I need to configure it specifically,Can't I read the S3 configuration configured in Flink

Anything else?


Are you willing to submit a PR?

• I'm willing to submit a PR!

[luoyuxia]

@biao-lvwan Thanks for reporting! You'll need to set s3.region in Fluss's server.yaml to solve the excpetion as the doc said . For the region, you can just use us-east-1 as the region of Minio's s3...
But I have to remind you that when reading from s3, Fluss client requires to obtain STS token from S3 by the method:

 AWSSecurityTokenService stsClient =
                AWSSecurityTokenServiceClientBuilder.standard()
                        .withRegion(region)
                        .withCredentials(
                                new AWSStaticCredentialsProvider(
                                        new BasicAWSCredentials(accessKey, secretKey)))
                        .build();

But unfortunately, it's not supported by Minio's s3.......
So you may still see a exception like Failed to update tokens ... in client side, but I believe the exception won't cause the client read fail..

[biao-lvwan]

@biao-lvwan Thanks for reporting! You'll need to set in Fluss's to solve the excpetion as the doc said . But I have to remind you that when reading from s3, Fluss client requires to obtain STS token from S3 by the method:s3.region``server.yaml

 AWSSecurityTokenService stsClient =
                AWSSecurityTokenServiceClientBuilder.standard()
                        .withRegion(region)
                        .withCredentials(
                                new AWSStaticCredentialsProvider(
                                        new BasicAWSCredentials(accessKey, secretKey)))
                        .build();

But unfortunately, it's not supported by Minio's s3....... So you may see a exception like in client side, but I believe the exception won't cause the client read fail..Failed to update tokens ...

Yes, this error does not affect the remote data transfer to Minio, but when opening lakehouse.sh, Start to download kv snapshot files to local directory for bucket， This task has been submitted to the Flink cluster, and after startup, insert，tm, It will report an error that the configuration has not been obtained，Do I need to configure server. yaml in the Flink cluster?

[luoyuxia]

@biao-lvwan Thanks for reporting! You'll need to set in Fluss's to solve the excpetion as the doc said . But I have to remind you that when reading from s3, Fluss client requires to obtain STS token from S3 by the method: s3.regionserver.yaml ``

 AWSSecurityTokenService stsClient =
                AWSSecurityTokenServiceClientBuilder.standard()
                        .withRegion(region)
                        .withCredentials(
                                new AWSStaticCredentialsProvider(
                                        new BasicAWSCredentials(accessKey, secretKey)))
                        .build();

But unfortunately, it's not supported by Minio's s3....... So you may see a exception like in client side, but I believe the exception won't cause the client read fail..Failed to update tokens ...

Yes, this error does not affect the remote data transfer to Minio, but when opening lakehouse.sh, Start to download kv snapshot files to local directory for bucket， This task has been submitted to the Flink cluster, and after startup, insert，tm, It will report an error that the configuration has not been obtained，Do I need to configure server. yaml in the Flink cluster?

Thanks for explantion. Now, I got you.. Unfortunately you can't read with Minio's S3 since Fluss client can't get the a credential token from your Minio's S3 for the reason I said before. Fluss will use S3's api to obtain the credential token, but Minio don't support such API..

[biao-lvwan]

@biao-lvwan Thanks for reporting! You'll need to set in Fluss's to solve the excpetion as the doc said . But I have to remind you that when reading from s3, Fluss client requires to obtain STS token from S3 by the method:server.yaml ``` s3.region`

 AWSSecurityTokenService stsClient =
                AWSSecurityTokenServiceClientBuilder.standard()
                        .withRegion(region)
                        .withCredentials(
                                new AWSStaticCredentialsProvider(
                                        new BasicAWSCredentials(accessKey, secretKey)))
                        .build();

But unfortunately, it's not supported by Minio's s3....... So you may see a exception like in client side, but I believe the exception won't cause the client read fail..Failed to update tokens ...

Yes, this error does not affect the remote data transfer to Minio, but when opening lakehouse.sh, Start to download kv snapshot files to local directory for bucket， This task has been submitted to the Flink cluster, and after startup, insert，tm, It will report an error that the configuration has not been obtained，Do I need to configure server. yaml in the Flink cluster?

Thanks for explantion. Now, I got you.. Unfortunately you can't read with Minio's S3 since Fluss client can't get the a credential token from your Minio's S3 for the reason I said before. Fluss will use S3's api to obtain the credential token, but Minio don't support such API..

thx

[luoyuxia]

But I think maybe Fluss can use the API that Minio also supports to obtain the token like Trino does. Seems trino use leverage sts endpoint to do that..

[biao-lvwan]

But I think maybe Fluss can use the API that Minio also supports to obtain the token like Trino does. Seems trino use leverage sts endpoint to do that..

Thank you, I'll give it a try