Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Granular env-based solution for "connection string"-like config #112

Open
slint opened this issue Mar 24, 2024 · 2 comments · May be fixed by inveniosoftware/invenio-app-rdm#2918 or inveniosoftware/invenio-config#57
Open

Granular env-based solution for "connection string"-like config #112

slint opened this issue Mar 24, 2024 · 2 comments · May be fixed by inveniosoftware/invenio-app-rdm#2918 or inveniosoftware/invenio-config#57

Comments

@slint
Copy link
Member

slint commented Mar 24, 2024

It's common practice to be able to configure services that rely on connection strings/URIs (e.g. DB, OpenSearch, RabbitMQ, Redis) by individually setting parts of the string via env variables. This allows:

  • reusing config in sub-charts (e.g. passing RABBITMQ_USER to both the web/worker config, but also the RabbitMQ sub-chart)
  • placing only sensitive information of the config in a secret and exposing as an env var

On the application-side building the SQLALCHEMY_DATABASE_URI config would look something like:

import os

# First check if the full value is set
if os.environ.get("SQLALCHEMY_DATABASE_URI"):
    SQLALCHEMY_DATABASE_URI = os.environ.get("SQLALCHEMY_DATABASE_URI")
else:
    # Check parts of the config
    db_host = os.environ.get("DB_HOST")
    db_password = os.environ.get("DB_PASSWORD")
    db_user = os.environ.get("DB_USER")
    db_name = os.environ.get("DB_NAME")
    db_port = os.environ.get("DB_PORT")

    if all([db_host, db_password, db_user, db_name, db_port]):
        SQLALCHEMY_DATABASE_URI = f"postgresql://{db_user}:{db_password}@{db_host}:{db_port}/{db_name}"
    else:
        SQLALCHEMY_DATABASE_URI = f"postgresql://invenio:password@localhost:5432/invenio"

Important

This requires development in the application modules that configure services like the DB, OpenSearch, RabbitMQ, etc. so that they support loading this type of config.
@lindhe
Copy link
Contributor

lindhe commented Mar 25, 2024 •
edited
Loading

Ah, yes! This is an important improvement, in my opinion!

An adjacent topic to this is if we can provide credentials using secrets! It would be a great improvement on security if we could set all credentials using secrets (which is not possible today, if I recall correctly). We may also want to consider using credentials from the secrets generated by our chart dependencies.

I think these topics should be part of this issue, but if you think otherwise I can create a separate issue for them.

@lindhe lindhe mentioned this issue Mar 25, 2024
Open
@slint
Copy link
Member Author

slint commented Mar 25, 2024

@lindhe exactly, I tried to (badly 😅) summarize this in:

placing only sensitive information of the config in a secret and exposing as an env var

We touched a bit on this on Discord, and I brought it up at the InvenioRDM workshop last week, where people agreed based on their experience with other Helm Charts approach to secrets.

Samk13 added a commit to Samk13/invenio-app-rdm that referenced this issue Nov 21, 2024
@Samk13
config: granular env-based solution for connection strings
85be1aa 
* build db uri
* build redis url
* build mq url

partially closes: inveniosoftware/helm-invenio#112
@Samk13 Samk13 linked a pull request Nov 21, 2024 that will close this issue
Open
10 tasks
Samk13 added a commit to Samk13/invenio-app-rdm that referenced this issue Nov 21, 2024
@Samk13
config: granular env-based solution for connection strings
fb44dbe 
* build db uri
* build redis url
* build mq url

partially closes: inveniosoftware/helm-invenio#112
Samk13 added a commit to Samk13/invenio-config that referenced this issue Nov 26, 2024
@Samk13
config: granular env-based solution for connection strings
bf02100 
Add logic to build connection urls from env vars if available
needed for helm charts security best practices. includes:
* Build db uri
* Build redis url
* Build mq url

Partially closes: inveniosoftware/helm-invenio#112
Samk13 added a commit to Samk13/invenio-app-rdm that referenced this issue Nov 26, 2024
@Samk13
config: granular env-based solution for connection strings
c4e3c53 
use invenio-config util to build connection string

partially closes: inveniosoftware/helm-invenio#112
@Samk13 Samk13 linked a pull request Nov 26, 2024 that will close this issue
Open
10 tasks
Samk13 added a commit to Samk13/invenio-config that referenced this issue Nov 26, 2024
@Samk13
config: granular env-based solution for connection strings
6ceb654 
Add logic to build connection urls from env vars if available
needed for helm charts security best practices. includes:
* Build db uri
* Build redis url
* Build mq url

Partially closes: inveniosoftware/helm-invenio#112
Samk13 added a commit to Samk13/invenio-config that referenced this issue Nov 27, 2024
@Samk13
config: granular env-based solution for connection strings
8f4b0e6 
Add logic to build connection urls from env vars if available
needed for helm charts security best practices. includes:
* Build db uri
* Build redis url
* Build mq url

Partially closes: inveniosoftware/helm-invenio#112
Samk13 added a commit to Samk13/invenio-config that referenced this issue Nov 27, 2024
@Samk13
config: granular env-based solution for connection strings
2e1e160 
Add logic to build connection urls from env vars if available
needed for helm charts security best practices. includes:
* Build db uri
* Build redis url
* Build mq url

Partially closes: inveniosoftware/helm-invenio#112
Samk13 added a commit to Samk13/invenio-config that referenced this issue Nov 28, 2024
@Samk13
config: granular env-based solution for connection strings
f6d0167 
Add logic to build connection urls from env vars if available
needed for helm charts security best practices. includes:
* Build db uri
* Build redis url
* Build mq url

Partially closes: inveniosoftware/helm-invenio#112
Samk13 added a commit to Samk13/invenio-config that referenced this issue Nov 28, 2024
@Samk13
config: granular env-based solution for connection strings
a196bc1 
Add logic to build connection urls from env vars if available
needed for helm charts security best practices. includes:
* Build db uri
* Build redis url
* Build mq url

Partially closes: inveniosoftware/helm-invenio#112
Samk13 added a commit to Samk13/invenio-config that referenced this issue Nov 28, 2024
@Samk13
config: granular env-based solution for connection strings
c131f74 
Add logic to build connection urls from env vars if available
needed for helm charts security best practices. includes:
* Build db uri
* Build redis url
* Build mq url

Partially closes: inveniosoftware/helm-invenio#112
Samk13 added a commit to Samk13/invenio-config that referenced this issue Nov 28, 2024
@Samk13
config: granular env-based solution for connection strings
8cc9cfe 
Add logic to build connection urls from env vars if available
needed for helm charts security best practices. includes:
* Build db uri
* Build redis url
* Build mq url

Partially closes: inveniosoftware/helm-invenio#112
Samk13 added a commit to Samk13/invenio-config that referenced this issue Nov 28, 2024
@Samk13
config: granular env-based solution for connection strings
5eeef92 
Add logic to build connection urls from env vars if available
needed for helm charts security best practices. includes:
* Build db uri
* Build redis url
* Build mq url

Partially closes: inveniosoftware/helm-invenio#112
Samk13 added a commit to Samk13/invenio-config that referenced this issue Nov 28, 2024
@Samk13
config: granular env-based solution for connection strings
e52709c 
Add logic to build connection urls from env vars if available
needed for helm charts security best practices. includes:
* Build db uri
* Build redis url
* Build mq url

Partially closes: inveniosoftware/helm-invenio#112
Samk13 added a commit to Samk13/invenio-app-rdm that referenced this issue Nov 28, 2024
@Samk13
config: granular env-based solution for connection strings
d12d53a 
use invenio-config util to build connection string

partially closes: inveniosoftware/helm-invenio#112
Samk13 added a commit to Samk13/invenio-app-rdm that referenced this issue Dec 2, 2024
@Samk13
config: granular env-based solution for connection strings
e0a468e 
use invenio-config util to build connection string

partially closes: inveniosoftware/helm-invenio#112
Samk13 added a commit to Samk13/invenio-config that referenced this issue Dec 2, 2024
@Samk13
config: granular env-based solution for connection strings
c8d1038 
Add logic to build connection urls from env vars if available
needed for helm charts security best practices. includes:
* Build db uri
* Build redis url
* Build mq url

Partially closes: inveniosoftware/helm-invenio#112
Samk13 added a commit to Samk13/invenio-config that referenced this issue Dec 2, 2024
@Samk13
config: granular env-based solution for connection strings
1f4d2aa 
Add logic to build connection urls from env vars if available
needed for helm charts security best practices. includes:
* Build db uri
* Build redis url
* Build mq url

Partially closes: inveniosoftware/helm-invenio#112
Samk13 added a commit to Samk13/invenio-config that referenced this issue Dec 3, 2024
@Samk13
config: granular env-based solution for connection strings
4afd6e5 
Add logic to build connection urls from env vars if available
needed for helm charts security best practices. includes:
* Build db uri
* Build redis url
* Build mq url

Partially closes: inveniosoftware/helm-invenio#112
Samk13 added a commit to Samk13/invenio-config that referenced this issue Dec 3, 2024
@Samk13
config: granular env-based solution for connection strings
77ec675 
Add logic to build connection urls from env vars if available
needed for helm charts security best practices. includes:
* Build db uri
* Build redis url
* Build mq url

Partially closes: inveniosoftware/helm-invenio#112
Samk13 added a commit to Samk13/invenio-config that referenced this issue Dec 3, 2024
@Samk13
config: granular env-based solution for connection strings
b9c6913 
Add logic to build connection urls from env vars if available
needed for helm charts security best practices. includes:
* Build db uri
* Build redis url
* Build mq url

Partially closes: inveniosoftware/helm-invenio#112
Samk13 added a commit to Samk13/invenio-config that referenced this issue Dec 3, 2024
@Samk13
config: granular env-based solution for connection strings
8859a27 
Add logic to build connection urls from env vars if available
needed for helm charts security best practices. includes:
* Build db uri
* Build redis url
* Build mq url

Partially closes: inveniosoftware/helm-invenio#112
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@slint @lindhe