Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate removed networking variables #900

Merged
merged 4 commits into from
Nov 9, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 2 additions & 4 deletions content/en/contributing/integration-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -147,16 +147,14 @@ region=eu-central-1
- Use the client `fixtures` and other fixtures for resource creation instead of methods from `aws_stack.py`
- While using the environment variable `TEST_TARGET=AWS_CLOUD`, the boto client will be automatically configured to target AWS instead of LocalStack.
- Configure your AWS profile/credentials:
- When running the test, set the environment variable `AWS_PROFILE` to the profile name you chose in the previous step. Eexample: `AWS_PROFILE=ls-sandbox`
- When running the test, set the environment variable `AWS_PROFILE` to the profile name you chose in the previous step. Example: `AWS_PROFILE=ls-sandbox`
- Ensure that all resources are cleaned up even when the test fails and even when other fixture cleanup operations fail!
- Testing against AWS might require additional roles and policies.

Here is how a useful environment configuration for testing against AWS could look like:

```bash
EDGE_BIND_HOST=0.0.0.0;
DEBUG=1;
DNS_ADDRESS=0;
DEBUG=1; # enables debug logging
TEST_DISABLE_RETRIES_AND_TIMEOUTS=1;
TEST_TARGET=AWS_CLOUD;
AWS_DEFAULT_REGION=us-east-1;
Expand Down
19 changes: 3 additions & 16 deletions content/en/getting-started/faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,23 +117,10 @@ sudo sysctl -w net.ipv4.ip_forward=1

### Why can't my other Docker containers reach LocalStack?

Using LocalStack inside a Docker network with multiple other containers can lead to connectivity issues from/to those containers. For example, a container which attempts to deploy a stack and interact with the services directly, from within the same Docker network.
Using LocalStack inside a Docker network with multiple other containers can lead to connectivity issues from/to those containers.
For example, a container which attempts to deploy a stack and interact with the services directly, from within the same Docker network.

To resolve this, set `HOSTNAME_EXTERNAL` for correct response values for endpoints. Use network aliases to ensure resolution of `localhost.localstack.cloud` to the correct container:

```yaml
...
networks:
default:
aliases:
- localhost.localstack.cloud
sdkstack:
image: ubuntu
command: ["bash", "-c", "apt update && apt install -y curl && sleep 5 && curl -v http://localhost.localstack.cloud:4566/_localstack/health"]
...
```

Note the missing `network: bridge` here! The default bridge network does not support DNS name resolution.
Refer to our [network troubleshooting guide]({{< ref "references/network-troubleshooting" >}}) covering several scenarios.

### How to resolve the pull rate limit issue for LocalStack's Docker image?

Expand Down
10 changes: 2 additions & 8 deletions content/en/references/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,9 @@ Options that affect the core LocalStack system.
| - | - | - |
| `DEBUG` | `0` (default) \|`1`| Flag to increase log level and print more verbose logs (useful for troubleshooting issues)|
| `IMAGE_NAME`| `localstack/localstack` (default), `localstack/localstack:0.11.0` | Specific name and tag of LocalStack Docker image to use.|
| `GATEWAY_LISTEN`| `127.0.0.1:4566` (default in host mode), `0.0.0.0:4566` (default in Docker mode) | Configures the bind addresses of LocalStack. It has the form `<ip address>:<port>(,<ip address>:<port>)*`. |
| `GATEWAY_LISTEN`| `0.0.0.0:4566` (default in Docker mode) `127.0.0.1:4566` (default in host mode) | Configures the bind addresses of LocalStack. It has the form `<ip address>:<port>(,<ip address>:<port>)*`. LocalStack Pro adds port `443`. |
| `LOCALSTACK_HOST`| `localhost.localstack.cloud:4566` (default) | This is interpolated into URLs and addresses that are returned by LocalStack. It has the form `<hostname>:<port>`. |
| `USE_SSL` | `0` (default) | Whether to return URLs using HTTP (`0`) or HTTPS (`1`). |
| `LEGACY_DIRECTORIES` | `0` (default) | Use legacy method of managing internal filesystem layout. See [Filesystem Layout]({{< ref "filesystem" >}}). |
| `PERSISTENCE` | `0` (default) | Enable persistence. See [Persistence Mechanism]({{< ref "persistence-mechanism" >}}) and [Filesystem Layout]({{< ref "filesystem" >}}). |
| `MAIN_CONTAINER_NAME` | `localstack_main` (default) | Specify the main docker container name |
Expand Down Expand Up @@ -308,17 +309,10 @@ These configurations are deprecated and will be removed in the upcoming major ve

| Variable | Example Values | Description |
| - | - | - |
| `USE_SSL` | `false` (default) | **Deprecated.** Whether to use https://... URLs with SSL encryption. Deprecated as of version 0.11.3. Each service endpoint now supports multiplexing HTTP/HTTPS traffic over the same port. |
| `DATA_DIR`| blank (disabled/default), `/tmp/localstack/data` | **Deprecated.** Local directory for saving persistent data. This option is deprecated since LocalStack v1 and will be ignored. Please use `PERSISTENCE`. Using this option will set `PERSISTENCE=1` as a deprecation path. The state will be stored in your LocalStack volume in the `state/` directory |
| `HOST_TMP_FOLDER` | `/some/path` | **Deprecated.** Temporary folder on the host that gets mounted as `$TMPDIR/localstack` into the LocalStack container. Required only for Lambda volume mounts when using `LAMBDA_REMOTE_DOCKER=false.` |
| `TMPDIR`| `/tmp` (default)| **Deprecated.** Temporary folder on the host running the CLI and inside the LocalStack container .|
| `<SERVICE>_BACKEND` | | **Deprecated.** Custom endpoint URL to use for a specific service, where `<SERVICE>` is the uppercase service name.
| `<SERVICE>_PORT_EXTERNAL` | `4567` | **Deprecated.** Port number to expose a specific service externally . `SQS_PORT_EXTERNAL`, e.g. , is used when returning queue URLs from the SQS service to the client. |
| `LOCALSTACK_HOSTNAME` | `http://${LOCALSTACK_HOSTNAME}:4566` | **Deprecated.** Name of the host where LocalStack services are available. Use this hostname as endpoint in order to access the services from within your Lambda functions (e.g., to store an item to DynamoDB or S3 from a Lambda). This option is read-only. Use `LOCALSTACK_HOST` instead. |
| `HOSTNAME_EXTERNAL` | `localhost` (default) | **Deprecated.** Name of the host to expose the services externally. This host is used, e.g., when returning queue URLs from the SQS service to the client. Use `LOCALSTACK_HOST` instead. |
| `EDGE_BIND_HOST` | `127.0.0.1` (default), `0.0.0.0` (docker)| **Deprecated.** Address the edge service binds to. Use `GATEWAY_LISTEN` instead. |
| `EDGE_PORT` | `4566` (default)| **Deprecated.** Port number for the edge service, the main entry point for all API invocations. |
| `EDGE_FORWARD_URL` | | **Deprecated.** Optional target URL to forward all edge requests to (e.g., for distributed deployments)
| `INIT_SCRIPTS_PATH` | `/some/path` | **Deprecated.** Before 1.0, this was used to configure the path to the initializing files with extensions `.sh` that were found in `/docker-entrypoint-initaws.d`. This has been replaced by the [init-hook system](https://docs.localstack.cloud/references/init-hooks/). |
| `ES_CUSTOM_BACKEND` | `http://elasticsearch:9200` | **Deprecated.** Use [`OPENSEARCH_CUSTOM_BACKEND`](#opensearch) instead. URL to a custom elasticsearch backend cluster. If this is set to a valid URL, then localstack will not create elasticsearch cluster instances, but instead forward all domains to the given backend (see [Custom Elasticsearch Backends]({{< ref "elasticsearch#custom-elasticsearch-backends" >}})). |
| `ES_MULTI_CLUSTER` | `0`\|`1` | **Deprecated.** Use [`OPENSEARCH_MULTI_CLUSTER`](#opensearch) instead. When activated, LocalStack will spawn one Elasticsearch cluster per domain. Otherwise all domains will share a single cluster instance. This is ignored if `ES_CUSTOM_BACKEND` is set. |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,17 +6,19 @@ tags:
- networking
---

If you have created a resource using LocalStack, such as an OpenSearch cluster or RDS database, you may need to access it from your application. Typically, these resources are accessible through a URL or a hostname provided by LocalStack. While the environment variable `HOSTNAME_EXTERNAL` can often control the returned hostname, this approach may only work for some cases. This guide will explore different scenarios and provide detailed instructions on accessing resources created by LocalStack under different scenarios.

{{<alert title="Note">}}
An additional environment variable is available: `LOCALSTACK_HOSTNAME`. It is primarily used for communication within LocalStack and should not be used. We are currently working on streamlining the configuration under different scenarios.
{{</alert>}}
If you have created a resource using LocalStack, such as an OpenSearch cluster or RDS database, you may need to access it from your application.
Typically, these resources are accessible through a URL or a hostname provided by LocalStack.
By default, LocalStack returns the hostname `localhost.localstack.cloud`, which resolves to LocalStack using DNS.
For special environments (e.g., proxies), the [configuration]({{< ref "configuration" >}}) `LOCALSTACK_HOST` customizes the URLs returned by LocalStack.
This guide will explore different scenarios and provide detailed instructions on accessing resources created by LocalStack under different scenarios.

## From your host

{{< figure src="../images/3.svg" width="400" >}}

For example, suppose you have created an OpenSearch cluster using LocalStack and want to access it from the same computer. In such a case, you can set the `HOSTNAME_EXTERNAL` environment variable to specify the desired hostname that will be returned. Check out the [service-specific documentation]({{<ref "user-guide/aws/feature-coverage">}}) for more details.
For example, suppose you have created an OpenSearch cluster using LocalStack and want to access it from the same computer.
In such a case, you can set the `LOCALSTACK_HOST` environment variable to specify the desired hostname and port that will be returned.
Check out the [service-specific documentation]({{<ref "user-guide/aws/feature-coverage">}}) for more details.

## From a container LocalStack created

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Suppose you have LocalStack installed on your machine and want to access it usin
Additionally, if you bind a domain name to `localhost`, ensure that you are not subject to [DNS rebind protection]({{< ref "user-guide/tools/transparent-endpoint-injection/dns-server#dns-rebind-protection" >}}).
{{</alert>}}

You can also use the `EDGE_PORT` [configuration variable]({{< ref "references/configuration" >}}) to change the exposed port if necessary.
You can also use the `GATEWAY_LISTEN` [configuration variable]({{< ref "references/configuration" >}}) to change the exposed port if necessary.

## From a container LocalStack created

Expand Down Expand Up @@ -217,7 +217,7 @@ LocalStack must listen to the address of the host, or `0.0.0.0`.

{{<tabpane>}}
{{<tab header="CLI" lang="bash">}}
EDGE_BIND_HOST="0.0.0.0" localstack start
GATEWAY_LISTEN="0.0.0.0" localstack start
{{</tab>}}
{{<tab header="Docker" lang="bash">}}
# this command exposes ports on all interfaces by default
Expand Down
5 changes: 3 additions & 2 deletions content/en/user-guide/aws/elastic-kubernetes-service/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,12 +100,13 @@ You can now use ECR (Elastic Container Registry) images within your EKS environm

#### Initial configuration

To modify the return value of resource URIs for most services, including ECR, you can utilize the `HOSTNAME_EXTERNAL` variable in the [configuration]({{< ref "configuration" >}}). By default, ECR returns a `repositoryUri` starting with `localhost`, such as: `localhost:<port>/<repository-name>`. However, if you set the `HOSTNAME_EXTERNAL` to `localhost.localstack.cloud`, the ECR will provide a `repositoryUri` like `localhost.localstack.cloud:<port>/<repository_name>`.
To modify the return value of resource URIs for most services, including ECR, you can utilize the `LOCALSTACK_HOST` variable in the [configuration]({{< ref "configuration" >}}).
By default, ECR returns a `repositoryUri` starting with `localhost.localstack.cloud`, such as: `localhost.localstack.cloud:<port>/<repository-name>`.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@simonrw That's the new default, right?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes this is correct


{{< alert title="Notes" >}}
In this section, we assume that `localhost.localstack.cloud` resolves in your environment, and LocalStack is connected to a non-default bridge network. For more information, refer to the article about [DNS rebind protection]({{< ref "dns-server#dns-rebind-protection" >}}).

If the domain `localhost.localstack.cloud` does not resolve on your host, you can still proceed without setting `HOSTNAME_EXTERNAL`. However it is suggested to use `localhost.localstack.cloud` as the registry in your pod configuration.
If the domain `localhost.localstack.cloud` does not resolve on your host, you can still proceed by setting `LOCALSTACK_HOST=localhost` (not recommended).

LocalStack will take care of the DNS resolution of `localhost.localstack.cloud` within ECR itself, allowing you to use the `localhost:<port>/<repository_name>` URI for tagging and pushing the image on your host.
{{< / alert >}}
Expand Down
13 changes: 9 additions & 4 deletions content/en/user-guide/ci/gitlab-ci/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,8 @@ We recommend taking the following steps:
- Add GitLab CI variables using the keyword variables to include `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_DEFAULT_REGION`.
- Create a service using the keyword `services` and reference `docker:20.10.16-dind` to pull the latest Docker image and assign an alias for the container (`docker` in our case). Use the `command` option to disable TLS with `--tls=false`.
- Install LocalStack and/or AWS-related dependencies to define the commands that should be run before all builds.
- In the `before_script` section, add commands to set up the `LOCALSTACK_HOSTNAME` and `HOSTNAME_EXTERNAL` environment variables and append `localhost.localstack.cloud` to `/etc/hosts` using the IP address of the `docker` service.
- In the script section, pull the `localstack/localstack` Docker image, start LocalStack in detached mode, and run your LocalStack-related tests.
- In the `script` section, append `localhost.localstack.cloud` to `/etc/hosts` using the IP address of the `docker` service.
- In the `script` section, pull the `localstack/localstack` Docker image, start LocalStack in detached mode, and run your LocalStack-related tests.

The following example Gitlab CI job config (`.gitlab-ci.yml`) executes these steps, creates a new S3 bucket, copies some content to the bucket, and checks the content of the bucket:

Expand All @@ -39,8 +39,6 @@ test:
AWS_DEFAULT_REGION: us-east-1
DOCKER_HOST: tcp://docker:2375
DOCKER_TLS_CERTDIR: ""
LOCALSTACK_HOSTNAME: localhost.localstack.cloud
HOSTNAME_EXTERNAL: localhost.localstack.cloud

services:
- name: docker:20.10.16-dind
Expand All @@ -62,6 +60,13 @@ test:
- aws --endpoint http://localhost.localstack.cloud:4566 s3 ls s3://test/
```

<details>
<summary>For LocalStack versions before 3.0.0</summary>
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks a bit awkward but probably needed

Under test>variables, add:<br>
LOCALSTACK_HOSTNAME: localhost.localstack.cloud<br>
HOSTNAME_EXTERNAL: localhost.localstack.cloud.
</details>

{{< alert title="Note">}}
While working with a Docker-in-Docker (`dind`) setup, the Docker runner requires `privileged` mode. You must always use `privileged = true` in your GitLab CI's `config.toml` file while setting up LocalStack in GitLab CI runners. For more information, see [GitLab CI Docker-in-Docker](https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#use-docker-in-docker-executor) documentation.
{{< /alert >}}
Expand Down
3 changes: 1 addition & 2 deletions content/en/user-guide/integrations/aws-cdk/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,8 +47,7 @@ $ cdklocal --help

The following environment variables can be configured:

* `EDGE_PORT`: Port under which LocalStack edge service is accessible (default: `4566`)
* `LOCALSTACK_HOSTNAME`: Target host under which LocalStack edge service is accessible (default: `localhost`)
* `AWS_ENDPOINT_URL`: The endpoint URL (i.e., protocol, host, and port) to connect to LocalStack (default: `http://localhost:4566`)
* `LAMBDA_MOUNT_CODE`: Whether to use local Lambda code mounting (via setting `hot-reload` S3 bucket name)


Expand Down
3 changes: 1 addition & 2 deletions content/en/user-guide/integrations/aws-sam/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,5 +40,4 @@ Please make sure to replace all `sam` calls with `samlocal` when following the A

### Configuration

* `EDGE_PORT`: Port number under which the LocalStack edge service is available (default: `4566`)
* `LOCALSTACK_HOSTNAME`: Host under which the LocalStack edge service is available (default: `localhost`)
* `AWS_ENDPOINT_URL`: The endpoint URL (i.e., protocol, host, and port) to connect to LocalStack (default: `http://localhost:4566`)
5 changes: 2 additions & 3 deletions content/en/user-guide/integrations/sdks/python/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,11 +37,10 @@ client = boto3.client("lambda")
...
```

Alternatively, if you prefer to (or need to) set the endpoints directly, you can use the `$LOCALSTACK_HOSTNAME` environment variable which is available when executing user code (e.g., Lambda functions, ECS containers) in LocalStack:
Alternatively, if you prefer to (or need to) set the endpoints directly, you can use the environment variable `AWS_ENDPOINT_URL`, which is available when executing user code (e.g., Lambda functions) in LocalStack:
```python3
import os
endpoint_url = f"http://{os.getenv("LOCALSTACK_HOSTNAME")}:{os.getenv("EDGE_PORT")}"
client = boto3.client("lambda", endpoint_url=endpoint_url)
client = boto3.client("lambda", endpoint_url=os.getenv("AWS_ENDPOINT_URL"))
...
```

Expand Down
13 changes: 6 additions & 7 deletions content/en/user-guide/integrations/serverless-framework/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,23 +106,22 @@ Usually, all of LocalStack's services are available via a specific port on local

Since the Lambda functions execute within the LocalStack Docker container, Lambda functions cannot access other services via the usual localhost endpoint.

Instead, LocalStack provides a special environment variable `LOCALSTACK_HOSTNAME` which contains the internal endpoint of the LocalStack services from within its runtime environment.
Instead, LocalStack provides a special environment variable `AWS_ENDPOINT_URL` which contains the internal endpoint of the LocalStack services from within its runtime environment.

Hence, you need to configure the Lambda functions to use the `LOCALSTACK_HOSTNAME` endpoint when accessing other AWS services in LocalStack.
Hence, you need to configure the Lambda functions to use the `AWS_ENDPOINT_URL` endpoint when accessing other AWS services in LocalStack.

In Python, this may look something like. The code detects if it is running in LocalStack by checking if the `LOCALSTACK_HOSTNAME` variable exists and then configures the endpoint URL accordingly.
In Python, this may look something like. The code detects if it is running in LocalStack by checking if the `AWS_ENDPOINT_URL` variable exists and then configures the endpoint URL accordingly.

```python
...
if 'LOCALSTACK_HOSTNAME' in os.environ:
dynamodb_endpoint = 'http://%s:4566' % os.environ['LOCALSTACK_HOSTNAME']
dynamodb = boto3.resource('dynamodb', endpoint_url=dynamodb_endpoint)
if 'AWS_ENDPOINT_URL' in os.environ:
dynamodb = boto3.resource('dynamodb', endpoint_url=os.environ['AWS_ENDPOINT_URL'])
else:
dynamodb = boto3.resource('dynamodb')
...
```

> Ideally, we want to make LocalStack's Lambda execution environment "LocalStack-agnostic", so that you are not required to adjust endpoints in your function code anymore. You want to help us with that? [Drop us a line in Slack](https://localstack-community.slack.com)!.
In LocalStack Pro, no code changes are required using our [Transparent Endpoint Injection]({{< ref "user-guide/tools/transparent-endpoint-injection" >}}).

## Deploying to LocalStack
You can now deploy your Serverless service to LocalStack.
Expand Down