Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add new ENV config S3_SERVICE_REMOVE_X_AMZ_HEADERS #241

Merged
merged 10 commits into from
May 2, 2024
Merged

Add new ENV config S3_SERVICE_REMOVE_X_AMZ_HEADERS #241

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
8dad170
Add S3_SERVICE_REMOVE_X_AMZ_HEADERS header flag for use case where yo…
gawsoftpl Apr 27, 2024
9ffe12e
Update docs
gawsoftpl Apr 27, 2024
0adbe91
Update docs
gawsoftpl Apr 27, 2024
35bcbba
Update docs
gawsoftpl Apr 27, 2024
91de535
set S3_SERVICE_REMOVE_X_AMZ_HEADERS as not required
gawsoftpl Apr 29, 2024
b791c7c
Add header HEADER_PREFIXES_ALLOWED for force allow some headers from …
gawsoftpl Apr 30, 2024
713da5f
Type correction
gawsoftpl Apr 30, 2024
8170509
- Fix Readme docs about HEADER_PREFIXES_ALLOWED
gawsoftpl May 1, 2024
8690f15
Add unit test testEditHeadersWithAllowedPrefixes
gawsoftpl May 1, 2024
8691f03
Add Header_prefixes_alloweed to docker-entrypoint
gawsoftpl May 1, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 33 additions & 2 deletions common/etc/nginx/include/s3gateway.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,14 @@ const S3_STYLE = process.env['S3_STYLE'];
* @type {Array<String>}
* */
const ADDITIONAL_HEADER_PREFIXES_TO_STRIP = utils.parseArray(process.env['HEADER_PREFIXES_TO_STRIP']);

/**
* Additional header prefixes to allow from the response before sending to the
* client. This is opposite to HEADER_PREFIXES_TO_STRIP.
* @type {Array<String>}
* */
const ADDITIONAL_HEADER_PREFIXES_ALLOWED = utils.parseArray(process.env['HEADER_PREFIXES_ALLOWED']);

/**
* Default filename for index pages to be read off of the backing object store.
* @type {string}
Expand All @@ -100,15 +108,19 @@ function editHeaders(r) {
r.method === 'HEAD' &&
_isDirectory(decodeURIComponent(r.variables.uri_path));

/* Strips all x-amz- headers from the output HTTP headers so that the
/* Strips all x-amz- (if x-amz- is not in ADDITIONAL_HEADER_PREFIXES_ALLOWED) headers from the output HTTP headers so that the
* requesters to the gateway will not know you are proxying S3. */
if ('headersOut' in r) {
for (const key in r.headersOut) {
const headerName = key.toLowerCase()
/* We delete all headers when it is a directory head request because
* none of the information is relevant for passing on via a gateway. */
if (isDirectoryHeadRequest) {
delete r.headersOut[key];
} else if (_isHeaderToBeStripped(key.toLowerCase(), ADDITIONAL_HEADER_PREFIXES_TO_STRIP)) {
} else if (
!_isHeaderToBeAllowed(headerName, ADDITIONAL_HEADER_PREFIXES_ALLOWED)
&& _isHeaderToBeStripped(headerName, ADDITIONAL_HEADER_PREFIXES_TO_STRIP)
gawsoftpl marked this conversation as resolved.
Show resolved Hide resolved
) {
delete r.headersOut[key];
}
}
Expand All @@ -128,6 +140,7 @@ function editHeaders(r) {
* sent on to the requesting client.
* @param headerName {string} Lowercase HTTP header name
* @param additionalHeadersToStrip {Array<string>} array of additional headers to remove
* @param additionalHeadersToAllow {Array<string>} array of additional headers to allow
gawsoftpl marked this conversation as resolved.
Show resolved Hide resolved
* @returns {boolean} true if header should be removed
*/
function _isHeaderToBeStripped(headerName, additionalHeadersToStrip) {
Expand All @@ -145,6 +158,24 @@ function _isHeaderToBeStripped(headerName, additionalHeadersToStrip) {
return false;
}

/**
* Determines if a given HTTP header should be force allowed from requesting client.
* @param headerName {string} Lowercase HTTP header name
* @param additionalHeadersToAllow {Array<string>} array of additional headers to allow
* @returns {boolean} true if header should be removed
*/
function _isHeaderToBeAllowed(headerName, additionalHeadersToAllow) {

for (let i = 0; i < additionalHeadersToAllow.length; i++) {
const headerToAllow = additionalHeadersToAllow[i];
if (headerName.indexOf(headerToAllow, 0) >= 0) {
return true;
}
}

return false;
}

/**
* Outputs the timestamp used to sign the request, so that it can be added to
* the 'Date' header and sent by NGINX.
Expand Down
5 changes: 4 additions & 1 deletion docs/getting_started.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,11 +39,14 @@ running as a Container or as a Systemd service.
| `PROXY_CACHE_VALID_FORBIDDEN` | No | | `30s` | Sets caching time for response code 403 |
| `PROVIDE_INDEX_PAGE` | No | `true`, `false` | `false` | Flag which returns the index page if there is one when requesting a directory. |
| `JS_TRUSTED_CERT_PATH` | No | | | Enables the `js_fetch_trusted_certificate` directive when retrieving AWS credentials and sets the path (on the container) to the specified path |
| `HEADER_PREFIXES_TO_STRIP` | No | | | A list of HTTP header prefixes that exclude headers client responses. List should be specified in lower-case and a semicolon (;) should be used to as a deliminator between values. For example: `x-goog-;x-something-` |
| `HEADER_PREFIXES_TO_STRIP` | No | | | A list of HTTP header prefixes that exclude headers client responses. List should be specified in lower-case and a semicolon (;) should be used to as a deliminator between values. For example: `x-goog-;x-something-` |
| `HEADER_PREFIXES_ALLOWED` | No | | | A list of HTTP header allowed prefixes from headers client responses. List should be specified in lower-case and a semicolon (;) should be used to as a deliminator between values. For example: `x-amz-;x-something-` |
gawsoftpl marked this conversation as resolved.
Show resolved Hide resolved
| `CORS_ENABLED` | No | `true`, `false` | `false` | Flag that enables CORS headers on GET requests and enables pre-flight OPTIONS requests. If enabled, this will add CORS headers for "fully open" cross domain requests by default, meaning all domains are allowed, similar to the settings show in [this example](https://enable-cors.org/server_nginx.html). CORS settings can be fine-tuned by overwriting the [`cors.conf.template`](/common/etc/nginx/templates/gateway/cors.conf.template) file. |
| `CORS_ALLOWED_ORIGIN` | No | | | value to set to be returned from the CORS `Access-Control-Allow-Origin` header. This value is only used if CORS is enabled. (default: \*) |
| `STRIP_LEADING_DIRECTORY_PATH` | No | | | Removes a portion of the path in the requested URL (if configured). Useful when deploying to an ALB under a folder (eg. www.mysite.com/somepath). |
| `PREFIX_LEADING_DIRECTORY_PATH` | No | | | Prefix to prepend to all S3 object paths. Useful to serve only a subset of an S3 bucket. When used in combination with `STRIP_LEADING_DIRECTORY_PATH`, this allows the leading path to be replaced, rather than just removed. |
|


If you are using [AWS instance profile credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html),
you will need to omit the `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and `AWS_SESSION_TOKEN` variables from
Expand Down
Loading