Adds S3 GUCs #81
Draft
Adds S3 GUCs #81
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We add an option for `COPY FROM` called `match_by_name` which matches Parquet file fields to PostgreSQL table columns `by their names` rather than `by their order` in the schema. By default, the option is `false`. The option is useful when field order differs between the Parquet file and the table, but their names match. **!!IMPORTANT!!**: This is a breaking change. Before the PR, we match always by name. This is a bit strict and not common way to match schemas. (e.g. COPY FROM csv at postgres or COPY FROM of duckdb match by field position by default) This is why we match by position by default and have a COPY FROM option `match_by_name` that can be set to true for the old behaviour. Closes #39.
To make it more Postgres way, we define 3 s3 related GUCs. - `pg_parquet.aws_config_file`: an absolute path to the configuration file used by s3 client. Note that when set, the GUC overrides `AWS_CONFIG_FILE` environment variable. By default the GUC is unset, - `pg_parquet.aws_shared_credentials_file`: an absolute path to the shared credentials file used by s3 client. Note that when set, the GUC overrides `AWS_SHARED_CREDENTIALS_FILE` environment variable. By default the GUC is unset, - `pg_parquet.aws_profile`: the profile name used by s3 client. Note that when set, the GUC overrides `AWS_PROFILE` environment variable. By default the GUC is unset. These GUCs can only be set by a superuser. You can easily set these GUCs to use different configurations per session. You do not need to restart your session to change the config file, shared credentials file or profile name. Closes #70.
aykut-bozkurt
commented
Nov 28, 2024
| // when the environment variable is not set, the default value | ||
| // (~/.aws/credentials) is used by object_store. | ||
| if let Some(aws_shared_credentials_file) = AWS_SHARED_CREDENTIALS_FILE.get() { | ||
| std::env::set_var( |
There was a problem hiding this comment.
do not set environment variables as it could be unexpected for other extensions... there should be a way to configure object_store.
|
Not sure if we want to support GUCs though. We need to sync GUCs with s3 env vars. This logic better be in a middleware. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
To make it more Postgres way, we define 3 s3 related GUCs.
pg_parquet.aws_config_file: an absolute path to the configuration file used by s3 client. Note that when set, the GUC overridesAWS_CONFIG_FILEenvironment variable. By default the GUC is unset,pg_parquet.aws_shared_credentials_file: an absolute path to the shared credentials file used by s3 client. Note that when set, the GUC overridesAWS_SHARED_CREDENTIALS_FILEenvironment variable. By default the GUC is unset,pg_parquet.aws_profile: the profile name used by s3 client. Note that when set, the GUC overridesAWS_PROFILEenvironment variable. By default the GUC is unset.These GUCs can only be set by a superuser. You can easily set these GUCs to use different configurations per session. You do not need to restart your session to change the config file, shared credentials file or profile name.
Closes #70.