Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: Add docs for connecting airflow to aws secrets backend #866

Merged
merged 1 commit into from
Oct 6, 2023
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 61 additions & 0 deletions airflow/plural/docs/aws-secrets-backend.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
## Connecting to AWS Secrets Backend

Airflow allows you the opportunity to connect to various services as a Secrets Backend as an alternative to using the
Airflow UI to manage connections. One of these services is [AWS Secrets Manager](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/secrets-backends/aws-secrets-manager.html).
Once you add below configurations, Airflow will be able to retrieve Secrets from AWS Secrets Manager (provided that they
have the same prefixes specified in the `KWARGS` config).

In this scenario, the prefixes are `airflow/connections` & `airflow/variables`, so any values stored under the
`airflow/connections` prefix would be treated the same as an object stored in the `Admin >> Connections` menu of the
Airflow UI. Any values stored under the `airflow/variables` prefix would be treated the same as an object stored in the
`Admin >> Variables` menu of the Airflow UI.

### edit values.yaml

You'll then want to edit `airflow/helm/airflow/values.yaml` in your installation repo with something like:

```yaml
airflow:
airflow:
airflow:
config:
AIRFLOW__SECRETS__BACKEND: airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend
AIRFLOW__SECRETS__BACKEND_KWARGS: '{"connections_prefix": "airflow/connections","variables_prefix":
"airflow/variables"}'
```

Alternatively, you should be able to do this in the configuration section for airflow in your plural console as well.

### add policy to AWS role

When installing the Airflow Application, Plural added a default role for Airflow. The role will be called
`<your-cluster-name>-airflow`. You will need to add a policy to that role to allow it to access AWS Secrets Manager. You
can use this policy as a starting point:

```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"secretsmanager:GetRandomPassword",
"secretsmanager:ListSecrets"
],
"Resource": "*"
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": "secretsmanager:*",
"Resource": "arn:aws:secretsmanager:<insert-aws-region>:<insert-aws-account-number>:secret:airflow/*"
}
]
}
```

### redeploy

From there, you should be able to run `plural build --only airflow && plural deploy --commit "use aws secrets manager
backend"` to use the secrets backend
Loading