diff --git a/.gitignore b/.gitignore new file mode 100644 index 00000000..17866c09 --- /dev/null +++ b/.gitignore @@ -0,0 +1,4 @@ +*.iml +.idea/ +.DS_Store +.vscode/ \ No newline at end of file diff --git a/configuration-utils/_oidc.config.json b/configuration-utils/_oidc.config.json index d77a72b2..d6bfae7c 100644 --- a/configuration-utils/_oidc.config.json +++ b/configuration-utils/_oidc.config.json @@ -3,6 +3,10 @@ "default": false, "description": "Enables/disables OIDC authentication. When enabled, few other configuration options must also be set." }, + "acl.oidc.ropc.flow.enabled": { + "default": false, + "description": "Enables/disables Resource Owner Password Credentials flow." + }, "acl.oidc.host": { "default": "", "description": "OIDC provider hostname, required when OIDC is enabled." @@ -35,6 +39,14 @@ "default": "", "description": "Client name assigned to QuestDB in the OIDC server, required when OIDC is enabled." }, + "acl.oidc.redirect.uri": { + "default": "", + "description": "The redirect URI tells the OIDC server where to redirect the user after successful authentication. If not set, the Web Console defaults it to the location where it was loaded from (`window.location.href`)." + }, + "acl.oidc.scope": { + "default": "openid", + "description": "The OIDC server should ask consent for the list of scopes provided in this property. The scope `openid` is mandatory, and always should be included." + }, "acl.oidc.authorization.endpoint": { "default": "/as/authorization.oauth2", "description": "OIDC Authorization Endpoint, the default value should work for the Ping Identity Platform." diff --git a/operations/openid-connect-oidc-integration.mdx b/operations/openid-connect-oidc-integration.mdx index 609bb5f4..22a6cac6 100644 --- a/operations/openid-connect-oidc-integration.mdx +++ b/operations/openid-connect-oidc-integration.mdx @@ -3,7 +3,7 @@ title: OpenID Connect (OIDC) Integration description: "" --- -import Screenshot from "@theme/Screenshot"; +import Screenshot from "@theme/Screenshot" OpenID Connect (OIDC) integrates with Identity Providers (IdP) external to QuestDB. @@ -27,7 +27,7 @@ OpenID Connect (OIDC) is only available in QuestDB Enterprise. See our example using [PingFederate with Active Directory](/docs/guides/active-directory-pingfederate/). -## Architecture +## Architecture overview Altogether, the architecture appears as such: @@ -45,13 +45,14 @@ We can break it down into core components. QuestDB's interactive UI. Users must authenticate before accessing the database via the interface. -The [Web Console](/docs/web-console/) uses PKCE (Proof Key for Code Exchange) to secure the -authentication and authorization flow. +The [Web Console](/docs/web-console/) uses PKCE (Proof Key for Code Exchange) to +secure the authentication and authorization flow. -In OAuth2/OIDC terms, the [Web Console](/docs/web-console/) is referred to as the _client_, and it is -assigned an identifier, the _Client Id._ +In OAuth2/OIDC terms, the [Web Console](/docs/web-console/) is referred to as +the _client_, and it is assigned an identifier: the **Client Id**. -Each application integrates via OIDC should be given a different Client Id. +Each application which integrates via OIDC should be given a different **Client +Id**. ### OIDC Provider @@ -99,21 +100,22 @@ It consists of ten steps... ### 1. Secret generation -First the [Web Console](/docs/web-console/) generates a cryptographically strong random secret, called -the _code verifier_. +First the [Web Console](/docs/web-console/) generates a cryptographically strong +random secret called the _code verifier_. -The secret is hashed using the _SHA256 algorithm_, the result is the _code +The secret is hashed using the _SHA256 algorithm_. The result is the _code challenge_. -After PKCE initialization the [Web Console](/docs/web-console/) requests an _authorization code_ from -the OIDC Provider. +After PKCE initialization the [Web Console](/docs/web-console/) requests an +_authorization code_ from the OIDC Provider. It calls the Authorization endpoint with a few parameters, including the: -- Client Id -- `openid` and `profile` scopes +- **Client Id** +- requested scopes (the list of scopes are configurable, default is `openid` + only) - code challenge -- method which was used to generate it from the code verifier (SHA256) +- algorithm used to generate the code challenge from the code verifier (SHA256) When the Authorization Server receives the request, it checks if the user has been authenticated already: @@ -125,7 +127,7 @@ been authenticated already: Identity Provider for authentication. ```bash title="Authorization code request example" -https://oidc.provider:443/as/authorization.oauth2?client_id=questdb&response_type=code&scope=openid+profile&redirect_uri=https%3A%2F%2Fquestdb.host%3A9000&code_challenge=IwZ-WuypAY3fMtvismbj1MQUe5CzMgrBa87nYcgFoLQ&code_challenge_method=S256 +https://oidc.provider:443/as/authorization.oauth2?client_id=questdb&response_type=code&scope=openid&redirect_uri=https%3A%2F%2Fquestdb.host%3A9000&code_challenge=IwZ-WuypAY3fMtvismbj1MQUe5CzMgrBa87nYcgFoLQ&code_challenge_method=S256 ``` ### 2. Prove identity @@ -137,7 +139,7 @@ This could be a username with: - a password, - an OTP - facial recognition via a mobile app -- anything supported by the Identity Provider. +- or anything else supported by the Identity Provider. **Worried about exposing the token?** It is rather opaque and does not contain +> user details. To carry out permission checks, the database has to know more about the user. + For this, QuestDB has a User Info Cache. If it finds a valid entry with the access token in the cache, steps 8 and 9 are -skipped. +skipped: ```bash title="Query request example" https://questdb.host:9999/exec?query=select%20current_user() @@ -249,8 +255,10 @@ No user information in the cache, or stale information? QuestDB uses the access token to request user information from the OIDC Provider's User Info endpoint. -This call also serves as token validation, because if the token is not real or -has been expired, the User Info endpoint replies with an error. +This call also serves as token validation. + +If the token is not real or has been expired, the User Info endpoint replies +with an error: ```bash title="User info request example" https://oidc.provider:443/idp/userinfo.openid @@ -260,27 +268,31 @@ Authorization: Bearer gslpJtzmmi6RwaPSx0dYGD4tEkom ### 9. Receive user information If the access token is valid, QuestDB receives the required user information -from the endpoint, and it will update its cache. +from the endpoint, then updates its cache. -The cache improves performance, as the database does not have to turn to the -OIDC Provider on every single request. +The cache improves performance, as QuestDB does not have to turn to the OIDC +Provider on every single request. -Cache expiry is configurable in QuestDB. +Do note that cache expiry is configurable: -```json title="User info response example" +```json title="User info response example with Active Directory groups" { "sub": "externalUser", "name": "External User", - "groups": ["externalGroup1", "externalGroup2"] + "groups": [ + "CN=TestGroup1,OU=DC Users,DC=ad,DC=quest,DC=dev", + "CN=TestGroup2,OU=DC Users,DC=ad,DC=quest,DC=dev" + ] } ``` ### 10. Permission check -With the help of the user information QuestDB can carry out permission checks +With the help of the user information, QuestDB can carry out +[permission checks](#user-permissions). -If the permission check is successful, the database will processes the request, -and then sends the results back. +If the permission check is successful, the database will process the request, +and then sends the results back: ```json title="Query response example" { @@ -297,10 +309,280 @@ and then sends the results back. } ``` +## Interactive clients + +Any interactive client - a UI, Jupyter notebook, CLI - can integrate with an +OIDC provider. However, the level of support will vary between these tools. + +Interactive clients usually fall into one of the following categories: + +- Browser-based clients with support for HTTP redirects; this includes the Web + Console or any javascript UI +- Applications running in a browser without support for redirects, such as + Jupyter notebooks +- Non-browser based clients, usually some kind of command line interface (CLI) + or a standalone application, such as Microsoft Access + +### Browser-based clients + +If the tool is browser based and can handle HTTP redirects, it can implement two +possible flows to request an access token.: + +1. [Authorization Code](https://openid.net/specs/openid-connect-core-1_0.html#CodeFlowAuth) + flow **(Recommended, more secure)** + +2. [Implicit](https://openid.net/specs/openid-connect-core-1_0.html#ImplicitFlowAuth) + flow + +The Web Console implements the +[Authorization Code Flow with PKCE](https://oauth.net/2/pkce), which is a +special version of the Authorization Code flow designed for mobile apps and +single page applications. + +Regardless of which flow is used by the web or mobile application, the requested +access token can be used for authentication and authorization when communicating +with QuestDB as explained in the [above 7th step](#7-database-access). + +### Jupyter notebook + +JupyterHub can integrate with OAuth2 providers using OAuthenticator, as +described in its +[documentation](https://jupyterhub.readthedocs.io/en/stable/explanation/oauth.html). +The OAuthenticator documentation also contains +[examples](https://oauthenticator.readthedocs.io/en/latest/tutorials/provider-specific-setup/index.html) +using different identity providers. + +If Jupyter notebooks are used without JupyterHub, one option for OAuth2 +integration is to use the +[Resource Owner Password Credentials](https://oauth.net/2/grant-types/password) +flow. It is likely that to enable this flow in your OAuth2 provider will require +additional setup. The Resource Owner Password Credentials flow is legacy, and +should be used as a last resort. + +We can use the code below to acquire an access token in our notebook: + +```python +from urllib import request, parse +import json + +url = "https://oidc.provider:443/as/token.oauth2" +data = parse.urlencode( { + "grant_type": "password", + "username": "testuser", + "password": "testpwd", + "scope": "openid", + "client_id": "testclient" +} ).encode() +req = request.Request(url=url, data=data) +req.add_header("Content-Type", "application/x-www-form-urlencoded") +with request.urlopen(req) as f: + body = f.read().decode(f.headers.get_content_charset()) + resp = json.loads(body) + access_token = resp["access_token"] +``` + +This token can be used to authenticate with QuestDB: + +```python +query = parse.urlencode({ + "query": "select current_user()" +}) +req = request.Request(f"http://localhost:9000/exec?{query}") +req.add_header("Authorization", f"Bearer {access_token}") +with request.urlopen(req) as f: + body = f.read().decode(f.headers.get_content_charset()) + resp = json.loads(body) + print(resp) +``` + +#### Externalizing credentials + +The above example saves the user's credentials into the notebook, potentially +exposing them to others. One way to improve this is to use environment variables +or files to externalize the username and password. + +Here is an example using the `dotenv` library. + +First we need to create a file named `.env` with the settings: + +```python +username=testuser +password=testpwd +``` + +Then load it in our notebook, and use it to request tokens: + +```python +from dotenv import load_dotenv +import os +from urllib import request, parse +import json + +load_dotenv() +user = os.environ.get("username") +pwd = os.environ.get("password") + +url = "https://oidc.provider:443/as/token.oauth2" +data = parse.urlencode( { + "grant_type": "password", + "username": user, + "password": pwd, + "scope": "openid", + "client_id": "testclient" +} ).encode() +req = request.Request(url=url, data=data) +req.add_header("Content-Type", "application/x-www-form-urlencoded") +with request.urlopen(req) as f: + body = f.read().decode(f.headers.get_content_charset()) + resp = json.loads(body) + access_token = resp["access_token"] +``` + +#### Enable ROPC in QuestDB + +The Resource Owner Password Credentials flow can be enabled in QuestDB within +`server.conf`: + +``` +acl.oidc.ropc.flow.enabled = true +``` + +Now we can use Basic Authentication to simplify our code. We send the +credentials to QuestDB, and the database will validate the credentials against +the OAuth2 provider. + +```python +from dotenv import load_dotenv +import os +from urllib import request +import base64 + +load_dotenv() +user = os.environ.get("username") +pwd = os.environ.get("password") + +query = parse.urlencode({ + "query": "select current_user()" +}) +req = request.Request(f"http://localhost:9000/exec?{query}") +b64credentials = base64.standard_b64encode(f"{user}:{pwd}".encode()).decode() +req.add_header("Authorization", f"Basic {b64credentials}") +with request.urlopen(req) as f: + body = f.read().decode(f.headers.get_content_charset()) + resp = json.loads(body) + print(resp) +``` + +We can also use a postgres client to connect to the database: + +:::note + +QuestDB never persists the user's credentials. + +::: + +```python +import psycopg as pg +from dotenv import load_dotenv +import os + +load_dotenv() +user = os.environ.get("username") +pwd = os.environ.get("password") + +conn_str = f"user={user} password={pwd} host=localhost port=8812 dbname=qdb" +with pg.connect(conn_str, autocommit=True) as connection: + with connection.cursor() as cur: + cur.execute("select current_user()") + records = cur.fetchall() + for row in records: + print(row) +``` + +### CLI, standalone applications + +When using CLI tools, such as `psql`, or standalone applications like Microsoft +Access, the best option may be the Resource Owner Password Credentials flow. + +The user logs in with their SSO credentials, and the server validates the +details with the OAuth2 provider: + +```shell +% psql -h localhost -p 8812 -U testuser +Password for user testuser: +psql (14.2, server 11.3) +Type "help" for help. + +testldap=> +testldap=> +``` + +## Non-interactive clients + +Non-interactive clients are usually jobs or standalone applications, such as a +client for ingesting data. It is practical to manage their credentials via an +OAuth2 provider too. + +As seen in the Jupyter notebook examples, the clients can request a token +themselves and then use it to authorise data ingestion: + +```python +import json +import os +import requests +import pandas as pd +from dotenv import load_dotenv +from questdb.ingress import Sender + +load_dotenv() +user = os.environ.get("username") +pwd = os.environ.get("password") + +token_endpoint = "https://oidc.provider:443/as/token.oauth2" +response = requests.post(token_endpoint, + data={"grant_type": "password", + "client_id": "testclient", + "username": user, + "password": pwd, + "scope": "openid"}, + headers={"Content-Type": "application/x-www-form-urlencoded"}) + +response_body = response.content.decode("utf-8") +tokens = json.loads(response_body) +access_token = tokens["access_token"] + +conf = f"http::addr=localhost:9000;token={access_token};" +with Sender.from_conf(conf) as sender: + df = pd.read_csv("data.csv") + df["ts"] = pd.to_datetime(df["ts"]) + sender.dataframe(df, table_name="foo", at="ts") +``` + +Alternatively, a user may rely on QuestDB to authenticate them via the OAuth2 +provider when the Resource Owner Password Credentials flow is enabled on the +server side: + +```python +import os +import pandas as pd +from dotenv import load_dotenv +from questdb.ingress import Sender + +load_dotenv() +user = os.environ.get("username") +pwd = os.environ.get("password") + +conf = f"http::addr=localhost:9000;username={user};password={pwd};" +with Sender.from_conf(conf) as sender: + df = pd.read_csv("data.csv") + df["ts"] = pd.to_datetime(df["ts"]) + sender.dataframe(df, table_name="foo", at="ts") +``` + ## User permissions -As mentioned earlier, QuestDB requires user information to be able to construct -the user's access list. +QuestDB requires additional user information to be able to construct the user's +access list. As a reminder, the access list is the list of permissions that determines what the user can and cannot do. @@ -314,16 +596,16 @@ Provider. Since external users are not managed by QuestDB, permissions cannot be granted to them directly. -Instead the database expects a list of groups, called the _groups claim_ to be +Instead, the database expects a list of groups, called the _groups claim_ to be present in the user information. -These external group names are then mapped to QuestDB's own groups. +These external group names are mapped to QuestDB's own groups. The access list of the external user consists of the permissions granted to those groups: