Merge pull request #211 from lsst-sqre/tickets/DM-28120

[DM-28120] Disable prestart database initialization, add email and name

.pre-commit-config.yaml

@@ -45,16 +45,16 @@ repos:
         additional_dependencies:
           # Manually mirror dev dependencies related to eslint here
           - [email protected]
-          - eslint@7.21.0
+          - eslint@7.24.0
           - [email protected]
           - [email protected]
           - [email protected]
-          - [email protected].1
+          - [email protected].2
           - [email protected]
           - [email protected]
-          - eslint-plugin-prettier@3.3.1
-          - eslint-plugin-react@7.22.0
+          - eslint-plugin-prettier@3.4.0
+          - eslint-plugin-react@7.23.2
           - [email protected]
           - [email protected]
-          - react@16.14.0
-          - react-dom@16.14.0
+          - react@17.0.2
+          - react-dom@17.0.2

CHANGELOG.rst

@@ -2,28 +2,40 @@
 Change log
 ##########
 
-2.0.0 (unreleased)
+2.0.0 (2021-04-23)
 ==================
 
 As of this release, Gafaelfawr now uses opaque tokens for all internal authentication and only issues JWTs as part of its OpenID Connect server support.
 All existing sessions and tokens will be invalidated by this upgrade and all users will have to reauthenticate.
 
 Gafaelfawr now requires a SQL database.
-Its URL must be set as the ``database_url`` configuration parameter.
+Its URL must be set as the ``config.databaseUrl`` Helm chart parameter.
 
 As of this release, Gafaelfawr now uses FastAPI instead of aiohttp.
-OpenAPI documentation is available via the ``/docs`` route but not yet fleshed out or reviewed for accuracy.
+OpenAPI documentation is available via the ``/auth/docs`` and ``/auth/redoc`` routes.
 
 - Eliminate internal JWTs, including the old session and session handle system, in favor of opaque tokens.
 - Add a new token API under ``/auth/api/v1`` for creating, modifying, viewing, and deleting tokens.
-  This will be the basis of a new token management UI.
+  This is the basis of the new token management UI.
+  API documentation is published under ``/auth/docs`` and ``/auth/redoc``.
+- Add support for several classes of tokens for different purposes.
+  Add additional token metadata to record the purpose of a token.
+- Add caching of internal and notebook tokens.
+  Issue new internal and notebook tokens when the previous token is half-expired.
+- Add support for a bootstrap token that can be used to dynamically create other tokens or configure administrators.
+- Add support for maintaining Kubernetes secrets containing Gafaelfawr service tokens for applications that need to make authenticated calls on their own behalf.
 - Replace the ``/auth/tokens`` UI with a new UI using React and Gatsby.
   Currently, it supports viewing all the tokens for a user, creating and editing user tokens, revoking tokens, viewing token information with the token change history, and searching the token change history.
 - Protected applications no longer receive a copy of the user's authentication token.
+  They must request a delegated token if they want one.
 - The ``/auth`` route now supports requesting a notebook or internal delegated token for the application.
 - Use FastAPI instead of aiohttp, and use httpx to make internal requests.
 - Add ``/.well-known/openid-configuration`` route to provide metadata about the internal OpenID Connect server.
   This follows the OpenID Connect Discovery 1.0 specification.
+- Enforce constraints on valid usernames matching GitHub's constraints, except without allowing capital letters.
+- Be more careful in interpreting ``isMemberOf`` claims from the upstream OpenID Connect provider and discard more invalid data.
+- Only document and support installing Gafaelfawr via the Helm chart.
+- Update all dependencies.
 
 1.5.0 (2020-09-16)
 ==================

Dockerfile

@@ -64,11 +64,11 @@ ENV GAFAELFAWR_UI_PATH=/app/ui/public
 # Make sure we use the virtualenv
 ENV PATH="/opt/venv/bin:$PATH"
 
-# Copy over the prestart script that handles database setup.
-COPY scripts/prestart.sh /app/prestart.sh
-
 # We use a module name other than app, so tell the base image that.  This
 # does not copy the app into /app as is recommended by the base Docker
 # image documentation and instead relies on the module search path as
 # modified by the virtualenv.
 ENV MODULE_NAME=gafaelfawr.main
+
+# Run on port 8080 instead of the FastAPI default for backward compatibility.
+ENV PORT=8080

docs/applications.rst

@@ -159,11 +159,17 @@ The value of that annotation is a comma-separated list of desired headers.
 ``X-Auth-Request-User``
     The username of the authenticated user.
 
+``X-Auth-Request-Name``
+    The name of the authenticated user, if available.
+
+``X-Auth-Request-Email``
+    The email address of the authenticated user, if available.
+
 ``X-Auth-Request-Uid``
     The numeric UID of the authenticated user if the user has one.
 
 ``X-Auth-Request-Groups``
-    If the token lists groups in an ``isMemberOf`` claim, the names of the groups will be returned, comma-separated, in this header.
+    The names of groups of the authenticated user, comma-separated, if any.
 
 ``X-Auth-Request-Token``
     If a notebook or internal token was requested, it will be provided as the value of this header.

docs/arch/providers.rst

@@ -25,7 +25,8 @@ When configured to use an OpenID Connect provider, Gafaelfawr obtains the ID tok
 
 - Username is taken from the claim identified by the ``username_claim`` setting.
 - UID is taken from the claim identified by the ``uid_claim`` setting and is converted to a number.
-- Name is taken from the ``name`` claim.
+- Name is taken from the ``name`` claim if it exists.
+- Email address is taken from the ``email`` claim if it exists.
 - Groups are taken from the ``isMemberOf`` claim if it exists.
 - The scope of the token will be based on the group membership from ``isMemberOf`` and the ``config.groupMapping`` Helm chart value.
   See :ref:`scopes` for more details.
@@ -40,6 +41,7 @@ GitHub does not issue JWTs, so the token created after GitHub authentication is
 The username will be taken from the ``login`` value returned by the ``/user`` API route, forced to lowercase.
 The UID will be taken from the ``id`` value returned by the ``/user`` API route.
 The name will be taken from the ``name`` value returned by the ``/user`` API route.
+The email address will be taken from the address tagged primary in the addresses returned by the ``/user/emails`` API route.
 The group membership will be taken from the user's team membership.
 See :ref:`github-groups` for more details.
 The scope of the token will be based on the group membership and the ``group_mapping`` configuration setting.

requirements/dev.txt

@@ -338,9 +338,9 @@ hyperframe==6.0.1 \
     # via
     #   h2
     #   selenium-wire
-identify==2.2.3 \
-    --hash=sha256:398cb92a7599da0b433c65301a1b62b9b1f4bb8248719b84736af6c0b22289d6 \
-    --hash=sha256:4537474817e0bbb8cea3e5b7504b7de6d44e3f169a90846cbc6adb0fc8294502
+identify==2.2.4 \
+    --hash=sha256:9bcc312d4e2fa96c7abebcdfb1119563b511b5e3985ac52f60d9116277865b2e \
+    --hash=sha256:ad9f3fa0c2316618dc4d840f627d474ab6de106392a4f00221820200f490f5a8
     # via pre-commit
 idna==2.10 \
     --hash=sha256:b307872f855b18632ce0c21c5e45be78c0ea7ae4c15c828c20788b26921eb3f6 \
@@ -630,9 +630,9 @@ pysocks==1.7.1 \
     --hash=sha256:2725bd0a9925919b9b51739eea5f9e2bae91e83288108a9ad338b2e3a4435ee5 \
     --hash=sha256:3f8804571ebe159c380ac6de37643bb4685970655d3bba243530d6558b799aa0
     # via selenium-wire
-pytest-asyncio==0.14.0 \
-    --hash=sha256:2eae1e34f6c68fc0a9dc12d4bea190483843ff4708d24277c41568d6b6044f1d \
-    --hash=sha256:9882c0c6b24429449f5f969a5158b528f39bde47dc32e85b9f0403965017e700
+pytest-asyncio==0.15.1 \
+    --hash=sha256:2564ceb9612bbd560d19ca4b41347b54e7835c2f792c504f698e05395ed63f6f \
+    --hash=sha256:3042bcdf1c5d978f6b74d96a151c4cfb9dcece65006198389ccd7e6c60eb1eea
     # via -r requirements/dev.in
 pytest-cov==2.11.1 \
     --hash=sha256:359952d9d39b9f822d9d29324483e7ba04a3a17dd7d05aa6beb7ea01e359e5f7 \
@@ -825,9 +825,9 @@ urllib3==1.26.4 \
     # via
     #   requests
     #   selenium
-virtualenv==20.4.3 \
-    --hash=sha256:49ec4eb4c224c6f7dd81bb6d0a28a09ecae5894f4e593c89b0db0885f565a107 \
-    --hash=sha256:83f95875d382c7abafe06bd2a4cdd1b363e1bb77e02f155ebe8ac082a916b37c
+virtualenv==20.4.4 \
+    --hash=sha256:09c61377ef072f43568207dc8e46ddeac6bcdcaf288d49011bda0e7f4d38c4a2 \
+    --hash=sha256:a935126db63128861987a7d5d30e23e8ec045a73840eeccb467c148514e29535
     # via pre-commit
 webcolors==1.11.1 \
     --hash=sha256:76f360636957d1c976db7466bc71dcb713bb95ac8911944dffc55c01cb516de6 \

requirements/main.txt

@@ -539,41 +539,41 @@ sniffio==1.2.0 \
     # via
     #   httpcore
     #   httpx
-sqlalchemy==1.4.9 \
-    --hash=sha256:065ac7331b87494a86bf3dc4430c1ee7779d6dc532213c528394ddd00804e518 \
-    --hash=sha256:099e63ffad329989080c533896267c40f9cb38ed5704168f7dae3afdda121e10 \
-    --hash=sha256:0d8aab144cf8d31c1ac834802c7df4430248f74bd8b3ed3149f9c9eec0eafe50 \
-    --hash=sha256:230b210fc6d1af5d555d1d04ff9bd4259d6ab82b020369724ab4a1c805a32dd3 \
-    --hash=sha256:25aaf0bec9eadde9789e3c0178c718ae6923b57485fdeae85999bc3089d9b871 \
-    --hash=sha256:29816a338982c30dd7ee76c4e79f17d5991abb1b6561e9f1d72703d030a79c86 \
-    --hash=sha256:2e1b8d31c97a2b91aea8ed8299ad360a32d60728a89f2aac9c98eef07a633a0e \
-    --hash=sha256:343c679899afdc4952ac659dc46f2075a2bd4fba87ca0df264be838eecd02096 \
-    --hash=sha256:386f215248c3fb2fab9bb77f631bc3c6cd38354ca2363d241784f8297d16b80a \
-    --hash=sha256:457a1652bc1c5f832165ff341380b3742bfb98b9ceca24576350992713ad700f \
-    --hash=sha256:4e554872766d2783abf0a11704536596e8794229fb0fa63d311a74caae58c6c5 \
-    --hash=sha256:4edff2b4101a1c442fb1b17d594a5fdf99145f27c5eaffae12c26aef2bb2bf65 \
-    --hash=sha256:690fbca2a208314504a2ab46d3e7dae320247fcb1967863b9782a70bf49fc600 \
-    --hash=sha256:6c6090d73820dcf04549f0b6e80f67b46c8191f0e40bf09c6d6f8ece2464e8b6 \
-    --hash=sha256:7bdb0f972bc35054c05088e91cec8fa810c3aa565b690bae75c005ee430e12e8 \
-    --hash=sha256:815a8cdf9c0fa504d0bfbe83fb3e596b7663fc828b73259a20299c01330467aa \
-    --hash=sha256:a28c7b96bc5beef585172ca9d79068ae7fa2527feaa26bd63371851d7894c66f \
-    --hash=sha256:a8763fe4de02f746666161b130cc3e5d1494a6f5475f5622f05251739fc22e55 \
-    --hash=sha256:b0266e133d819d33b555798822606e876187a96798e2d8c9b7f85e419d73ef94 \
-    --hash=sha256:bb97aeaa699c43da62e35856ab56e5154d062c09a3593a2c12c67d6a21059920 \
-    --hash=sha256:bce6eaf7b9a3a445911e225570b8fd26b7e98654ac9f308a8a52addb64a2a488 \
-    --hash=sha256:c4485040d86d4b3d9aa509fd3c492de3687d9bf52fb85d66b33912ad068a088c \
-    --hash=sha256:c6f228b79fd757d9ca539c9958190b3a44308f743dc7d83575aa0891033f6c86 \
-    --hash=sha256:cde2cf3ee76e8c538f2f43f5cf9252ad53404fc350801191128bab68f335a8b2 \
-    --hash=sha256:cfa4a336de7d32ae30b54f7b8ec888fb5c6313a1b7419a9d7b3f49cdd83012a3 \
-    --hash=sha256:cfbf2cf8e8ef0a1d23bfd0fa387057e6e522d55e43821f1d115941d913ee7762 \
-    --hash=sha256:e26791ac43806dec1f18d328596db87f1b37f9d8271997dd1233054b4c377f51 \
-    --hash=sha256:e7d262415e4adf148441bd9f10ae4e5498d6649962fabc62a64ec7b4891d56c5 \
-    --hash=sha256:e9e95568eafae18ac40d00694b82dc3febe653f81eee83204ef248563f39696d \
-    --hash=sha256:ec7c33e22beac16b4c5348c41cd94cfee056152e55a0efc62843deebfc53fcb4 \
-    --hash=sha256:f239778cf03cd46da4962636501f6dea55af9b4684cd7ceee104ad4f0290e878 \
-    --hash=sha256:f31757972677fbe9132932a69a4f23db59187a072cc26427f56a3082b46b6dac \
-    --hash=sha256:fbdcf9019e92253fc6aa0bcd5937302664c3a4d53884c425c0caa994e56c4421 \
-    --hash=sha256:fc82688695eacf77befc3d839df2bc7ff314cd1d547f120835acdcbac1a480b8
+sqlalchemy==1.4.11 \
+    --hash=sha256:0140f6dac2659fa6783e7029085ab0447d8eb23cf4d831fb907588d27ba158f7 \
+    --hash=sha256:034b42a6a59bf4ddc57e5a38a9dbac83ccd94c0b565ba91dba4ff58149706028 \
+    --hash=sha256:03a503ecff0cc2be3ad4dafd220eaff13721edb11c191670b7662932fb0a5c3a \
+    --hash=sha256:069de3a701d33709236efe0d06f38846b738b19c63d45cc47f54590982ba7802 \
+    --hash=sha256:1735e06a3d5b0793d5ee2d952df8a5c63edaff6383c2210c9b5c93dc2ea4c315 \
+    --hash=sha256:19633df6be629200ff3c026f2837e1dd17908fb1bcea860290a5a45e6fa5148e \
+    --hash=sha256:1e14fa32969badef9c309f55352e5c46f321bd29f7c600556caacdaa3eddfcf6 \
+    --hash=sha256:31e941d6db8b026bc63e46ef71e877913f128bd44260b90c645432626b7f9a47 \
+    --hash=sha256:452c4e002be727cb6f929dbd32bbc666a0921b86555b8af09709060ed3954bd3 \
+    --hash=sha256:45a720029756800628359192630fffdc9660ab6f27f0409bd24d9e09d75d6c18 \
+    --hash=sha256:4a2e7f037d3ca818d6d0490e3323fd451545f580df30d62b698da2f247015a34 \
+    --hash=sha256:4a7d4da2acf6d5d068fb41c48950827c49c3c68bfb46a1da45ea8fbf7ed4b471 \
+    --hash=sha256:4ad4044eb86fbcbdff2106e44f479fbdac703d77860b3e19988c8a8786e73061 \
+    --hash=sha256:4f631edf45a943738fa77612e85fc5c5d3fb637c4f5a530f7eedd1a7cd7a70a7 \
+    --hash=sha256:6389b10e23329dc8b5600c1a84e3da2628d0f437d8a5cd05aefd1470ec571dd1 \
+    --hash=sha256:6ebd58e73b7bd902688c0bb8dbabb0c36b756f02cc7b27ad5efa2f380c611f95 \
+    --hash=sha256:7180830ea1082b96b94884bc352b274e29b45151b6ee911bf1fd79cba2de659b \
+    --hash=sha256:789be639501445d85fd4ca41d04f0f5c6cbb6deb0c6826aaa6f22774fe84ef94 \
+    --hash=sha256:7d89add44938ea4f52c7641d5805c9e154fed4381e874ef3221483eeb191a96d \
+    --hash=sha256:842b0d4698381aac047f8ae57409c90b7e63ebabf5bc02814ddc8eaefd13499e \
+    --hash=sha256:8f96d4b6a49d3f0f109365bb6303ae5d266d3f90280ca68cf8b2c46032491038 \
+    --hash=sha256:961b089e64c2ad29ad367487dd3ba1aa3eeba56bc82037ce91732baaa0f6ca90 \
+    --hash=sha256:96de1d4a2e05d4a017087cb29cd6a8ebfeecfd0e9f872880b1a589f011c1c02e \
+    --hash=sha256:98214f04802a3fc740038744d8981a8f2fdca710f791ca125fc4792737d9f3a7 \
+    --hash=sha256:9cf94161cb55507cee147bf8abcfd3c076b353ad18743296764dd81108ea74f8 \
+    --hash=sha256:9fdf0713166f33e5e6ea98cf59deb305cb323131277f6880de6c509f468076f8 \
+    --hash=sha256:a41ab83ecfadf38a47bdfaf4e488f71579df47a711e1ab1dce30d34c7c25bd00 \
+    --hash=sha256:ac14fee167653ec6dee32d6aa4d501d90ae1bfbbc3eb5816940bccf227f0d617 \
+    --hash=sha256:b8b7d66ee8b8ac272adce0af1342a60854f0d89686e6d3318127a6a82a2f765c \
+    --hash=sha256:bb1072fdf48ba870c0fe81bee8babe4ba2f096fb56bb4f3e0c2386a7626e405c \
+    --hash=sha256:cd823071b97c1a6ac3af9e43b5d861126a1304033dcd18dfe354a02ec45642fe \
+    --hash=sha256:d08173144aebdf30c21a331b532db16535cfa83deed12e8703fa6c67c0894ffc \
+    --hash=sha256:e7d76312e904aa4ea221a92c0bc2e299ad46e4580e2d72ca1f7e6d31dce5bfab \
+    --hash=sha256:f772e4428d413c0affe2a34836278fbe9df9a9c0940705860c2d3a4b50af1a66
     # via
     #   -r requirements/main.in
     #   alembic

src/gafaelfawr/config.py

@@ -193,7 +193,7 @@ class Settings(BaseSettings):
     redis_password_file: Optional[str] = None
     """File containing the password to use when connecting to Redis."""
 
-    bootstrap_token: Optional[Token] = None
+    bootstrap_token: Optional[str] = None
     """Bootstrap authentication token.
 
     This token can be used with specific routes in the admin API to change the
@@ -295,11 +295,12 @@ def _valid_loglevel(cls, v: str) -> str:
         return v
 
     @validator("bootstrap_token", pre=True)
-    def _valid_bootstrap_token(cls, v: Optional[str]) -> Optional[Token]:
+    def _valid_bootstrap_token(cls, v: Optional[str]) -> Optional[str]:
         if not v:
             return None
         try:
-            return Token.from_str(v)
+            Token.from_str(v)
+            return v
         except Exception as e:
             raise ValueError(f"bootstrap_token not a valid token: {str(e)}")
 
@@ -676,6 +677,9 @@ def from_file(cls, path: str) -> Config:
         }
 
         # Build the Config object.
+        bootstrap_token = None
+        if settings.bootstrap_token:
+            bootstrap_token = Token.from_str(settings.bootstrap_token)
         issuer_config = IssuerConfig(
             iss=settings.issuer.iss,
             kid=settings.issuer.key_id,
@@ -731,7 +735,7 @@ def from_file(cls, path: str) -> Config:
             session_secret=session_secret.decode(),
             redis_url=settings.redis_url,
             redis_password=redis_password,
-            bootstrap_token=settings.bootstrap_token,
+            bootstrap_token=bootstrap_token,
             proxies=tuple(settings.proxies if settings.proxies else []),
             after_logout_url=str(settings.after_logout_url),
             issuer=issuer_config,

src/gafaelfawr/constants.py

@@ -6,6 +6,9 @@
 COOKIE_NAME = "gafaelfawr"
 """Name of the state cookie."""
 
+HTTP_TIMEOUT = 10.0
+"""Timeout (in seconds) for outbound HTTP requests to auth providers."""
+
 KUBERNETES_TOKEN_TYPE_LABEL = "gafaelfawr.lsst.io/token-type"
 """Label storing the token type of Gafaelfawr-managed secrets."""
 

src/gafaelfawr/database.py

@@ -11,19 +11,58 @@
 from typing import TYPE_CHECKING
 
 import structlog
-from sqlalchemy import create_engine
+from sqlalchemy import create_engine, select
 from sqlalchemy.exc import OperationalError
 from sqlalchemy.orm import Session
 
 from gafaelfawr.models.admin import Admin
+from gafaelfawr.schema import Admin as SQLAdmin
 from gafaelfawr.schema import drop_schema, initialize_schema
 from gafaelfawr.storage.admin import AdminStore
 from gafaelfawr.storage.transaction import TransactionManager
 
 if TYPE_CHECKING:
+    from structlog.stdlib import BoundLogger
+
     from gafaelfawr.config import Config
 
-__all__ = ["initialize_database"]
+__all__ = ["create_session", "initialize_database"]
+
+
+def create_session(config: Config, logger: BoundLogger) -> Session:
+    """Create a new database session.
+
+    Checks that the database is available and retries in a loop for 10s if it
+    is not.
+
+    Parameters
+    ----------
+    config : `gafaelfawr.config.Config`
+        The Gafaelfawr configuration.
+
+    Returns
+    -------
+    session : `sqlalchemy.orm.Session`
+        The database session.
+    """
+    for _ in range(5):
+        try:
+            engine = create_engine(config.database_url)
+            session = Session(bind=engine)
+            session.execute(select(SQLAdmin))
+            return session
+        except OperationalError:
+            logger.info("database not ready, waiting two seconds")
+            time.sleep(2)
+            continue
+
+    # If we got here, we failed five times.  Try one last time without
+    # catching exceptions so that we raise the appropriate exception to our
+    # caller.
+    engine = create_engine(config.database_url)
+    session = Session(bind=engine)
+    session.execute(select(Admin))
+    return session
 
 
 def initialize_database(config: Config, reset: bool = False) -> None:
@@ -55,7 +94,8 @@ def initialize_database(config: Config, reset: bool = False) -> None:
             logger.info("database not ready, waiting two seconds")
             time.sleep(2)
             continue
-        logger.info("initialized database schema")
+        if success:
+            logger.info("initialized database schema")
         break
     if not success:
         msg = "database schema initialization failed (database not reachable?)"

src/gafaelfawr/dependencies/http_client.py

@@ -6,6 +6,8 @@
 
 from httpx import AsyncClient
 
+from gafaelfawr.constants import HTTP_TIMEOUT
+
 __all__ = ["http_client_dependency"]
 
 
@@ -20,5 +22,5 @@ async def http_client_dependency() -> AsyncIterator[AsyncClient]:
 
     This dependency should eventually move into the Safir framework.
     """
-    async with AsyncClient() as client:
+    async with AsyncClient(timeout=HTTP_TIMEOUT) as client:
         yield client