Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🐛 [BUG]: Unable to disable JupyterHub behind Twitcher #443

Closed
tlvu opened this issue Apr 4, 2024 · 8 comments · Fixed by #446
Closed

🐛 [BUG]: Unable to disable JupyterHub behind Twitcher #443

tlvu opened this issue Apr 4, 2024 · 8 comments · Fixed by #446
Assignees
@mishaschwartz
Labels
bug Something isn't working

Comments

@tlvu
Copy link
Collaborator

tlvu commented Apr 4, 2024

Summary

Setting JUPYTERHUB_AUTHENTICATOR_AUTHORIZATION_URL="" in env.local is unable to disable JupyterHub behind Twitcher. It breaks the spawn of the personal JupyterLab server completely.

Details

Error in docker logs jupyterhub when JUPYTERHUB_AUTHENTICATOR_AUTHORIZATION_URL="" is set in env.local: Error starting server lvu: Invalid URL '': No scheme supplied. Perhaps you meant https://?

Full error log:

[I 2024-04-04 16:19:18.682 JupyterHub log:191] 302 GET /jupyter/hub/ -> /jupyter/hub/spawn ([email protected]) 89.01ms
[D 2024-04-04 16:19:18.843 JupyterHub scopes:877] Checking access to /jupyter/hub/spawn via scope servers
[D 2024-04-04 16:19:18.843 JupyterHub scopes:690] Argument-based access to /jupyter/hub/spawn via servers
[D 2024-04-04 16:19:18.845 JupyterHub pages:209] Serving options form for lvu
[I 2024-04-04 16:19:18.875 JupyterHub log:191] 200 GET /jupyter/hub/spawn ([email protected]) 41.23ms
[D 2024-04-04 16:19:21.646 JupyterHub scopes:877] Checking access to /jupyter/hub/spawn via scope servers
[D 2024-04-04 16:19:21.647 JupyterHub scopes:690] Argument-based access to /jupyter/hub/spawn via servers
[D 2024-04-04 16:19:21.647 JupyterHub pages:257] Triggering spawn with supplied form options for lvu
[D 2024-04-04 16:19:21.647 JupyterHub base:344] Refreshing auth for lvu
[E 2024-04-04 16:19:21.650 JupyterHub pages:313] Error starting server lvu: Invalid URL '': No scheme supplied. Perhaps you meant https://?
    Traceback (most recent call last):
    NoneType: None
    
[E 2024-04-04 16:19:21.650 JupyterHub pages:266] Failed to spawn single-user server with form
    Traceback (most recent call last):
      File "/usr/local/lib/python3.10/dist-packages/jupyterhub/handlers/pages.py", line 262, in _post
        return await self._wrap_spawn_single_user(
      File "/usr/local/lib/python3.10/dist-packages/jupyterhub/handlers/pages.py", line 317, in _wrap_spawn_single_user
        raise web.HTTPError(
    tornado.web.HTTPError: HTTP 500: Internal Server Error (Unhandled error starting server lvu)
    
[I 2024-04-04 16:19:21.654 JupyterHub log:191] 200 POST /jupyter/hub/spawn?_xsrf=[secret] ([email protected]) 10.75ms
[I 2024-04-04 16:20:05.381 JupyterHub log:191] 200 GET /jupyter/hub/login (@172.18.0.1) 7.34ms

Full log when it works (that switch is not set), starting from the exact same line GET /jupyter/hub/ -> /jupyter/hub/spawn:

[I 2024-04-04 16:24:33.887 JupyterHub log:191] 302 GET /jupyter/hub/ -> /jupyter/hub/spawn ([email protected]) 92.05ms
[D 2024-04-04 16:24:33.959 JupyterHub scopes:877] Checking access to /jupyter/hub/spawn via scope servers
[D 2024-04-04 16:24:33.959 JupyterHub scopes:690] Argument-based access to /jupyter/hub/spawn via servers
[D 2024-04-04 16:24:33.960 JupyterHub pages:209] Serving options form for lvu
[I 2024-04-04 16:24:33.974 JupyterHub log:191] 200 GET /jupyter/hub/spawn ([email protected]) 17.31ms
[D 2024-04-04 16:24:36.638 JupyterHub scopes:877] Checking access to /jupyter/hub/spawn via scope servers
[D 2024-04-04 16:24:36.638 JupyterHub scopes:690] Argument-based access to /jupyter/hub/spawn via servers
[D 2024-04-04 16:24:36.638 JupyterHub pages:257] Triggering spawn with supplied form options for lvu
[D 2024-04-04 16:24:36.639 JupyterHub base:344] Refreshing auth for lvu
[D 2024-04-04 16:24:36.661 JupyterHub base:961] Initiating spawn for lvu
[D 2024-04-04 16:24:36.661 JupyterHub base:965] 0/100 concurrent spawns
[D 2024-04-04 16:24:36.661 JupyterHub base:970] 0 active servers
[I 2024-04-04 16:24:37.012 JupyterHub provider:659] Creating oauth client jupyterhub-user-lvu
[D 2024-04-04 16:24:37.450 JupyterHub user:794] Calling Spawner.start for lvu
[I 2024-04-04 16:24:37.451 JupyterHub dockerspawner:1262] pulling pavics/workflow-tests:py39-230601-1-update240116
[I 2024-04-04 16:24:37.641 JupyterHub log:191] 302 POST /jupyter/hub/spawn?_xsrf=[secret] -> /jupyter/hub/spawn-pending/lvu?_xsrf=[secret] ([email protected]) 1004.86ms
[D 2024-04-04 16:24:37.786 JupyterHub scopes:877] Checking access to /jupyter/hub/spawn-pending/lvu via scope servers
[D 2024-04-04 16:24:37.787 JupyterHub scopes:690] Argument-based access to /jupyter/hub/spawn-pending/lvu via servers
[I 2024-04-04 16:24:37.788 JupyterHub pages:398] lvu is pending spawn
[I 2024-04-04 16:24:37.805 JupyterHub log:191] 200 GET /jupyter/hub/spawn-pending/lvu?_xsrf=[secret] ([email protected]) 23.01ms
[D 2024-04-04 16:24:37.949 JupyterHub dockerspawner:1027] Getting container 'jupyter-lvu'
[I 2024-04-04 16:24:37.952 JupyterHub dockerspawner:1033] Container 'jupyter-lvu' is gone
[D 2024-04-04 16:24:37.959 JupyterHub dockerspawner:1205] Starting host with config: {REMOVED}
[D 2024-04-04 16:24:38.888 JupyterHub scopes:877] Checking access to /jupyter/hub/api/users/lvu/server/progress via scope read:servers
[D 2024-04-04 16:24:38.888 JupyterHub scopes:690] Argument-based access to /jupyter/hub/api/users/lvu/server/progress via read:servers
[I 2024-04-04 16:24:40.576 JupyterHub dockerspawner:1311] Created container jupyter-lvu (id: f68bcca) from image pavics/workflow-tests:py39-230601-1-update240116
[I 2024-04-04 16:24:40.576 JupyterHub dockerspawner:1335] Starting container jupyter-lvu (id: f68bcca)
[D 2024-04-04 16:24:42.706 JupyterHub spawner:1384] Polling subprocess every 30s
[D 2024-04-04 16:24:42.707 JupyterHub dockerspawner:972] Persisting state for lvu: container name=jupyter-lvu, id=f68bcca87532c4d696a28d476337deee7bcde561062012900fe375b1a65c6807

To Reproduce

Steps to reproduce the behavior:

  1. birdhouse-deploy version 2.1.2 (but should be broken since 1.36.0 when JupyterHub was put behind Twitcher)
  2. set export JUPYTERHUB_AUTHENTICATOR_AUTHORIZATION_URL="" in env.local
  3. login to JupyterHub will work
  4. but spawning the personal server will fail with the log above server-side

Environment

Information Value
Server/Platform URL
Version Tag/Commit 2.1.2
Related issues/PR #358 (tag 1.36.0)
Related components jupyterhub, twitcher, magpie
Custom configuration

Concerned Organizations

@mishaschwartz @fmigneault
@tlvu tlvu added the bug Something isn't working label Apr 4, 2024
@tlvu
Copy link
Collaborator Author

tlvu commented Apr 4, 2024

@mishaschwartz I merged the pending Ouranosinc/jupyterhub#29 so you are clear to do any fixes in that repo.

@mishaschwartz
Copy link
Collaborator

@tlvu

Is JUPYTERHUB_CRYPT_KEY set?
If so, then the MagpieAuthenticator.refresh_pre_spawn is set to True and the authorization_url is required.

@mishaschwartz
Copy link
Collaborator

@tlvu Ouranosinc/jupyterhub#30 should cover this edge case

@tlvu
Copy link
Collaborator Author

tlvu commented Apr 4, 2024

Is JUPYTERHUB_CRYPT_KEY set?
If so, then the MagpieAuthenticator.refresh_pre_spawn is set to True and the authorization_url is required.

@mishaschwartz Oh indeed I also had JUPYTERHUB_CRYPT_KEY set. I was trying out all the new options, didn't know they were mutually exclusive.

Confirmed that removing JUPYTERHUB_CRYPT_KEY while having JUPYTERHUB_AUTHENTICATOR_AUTHORIZATION_URL="", I was able to login and spawn the server fine.

@tlvu
Copy link
Collaborator Author

tlvu commented Apr 4, 2024

Ouranosinc/jupyterhub#30 should cover this edge case

So I've read the PR, so the problem is not due to both JUPYTERHUB_CRYPT_KEY and JUPYTERHUB_AUTHENTICATOR_AUTHORIZATION_URL="" are mutually exclusive but due to a previous auth_state left behind?

So both of these options can be set at the same time and it should work?

@tlvu tlvu mentioned this issue Apr 4, 2024
Merged
@mishaschwartz
Copy link
Collaborator

So both of these options can be set at the same time and it should work?

Yeah exactly

@tlvu
Copy link
Collaborator Author

tlvu commented Apr 5, 2024

So both of these options can be set at the same time and it should work?

Yeah exactly

Just to be clear. Your fix is to make these 2 options not mutually exclusive. Otherwise, in the current state, they are mutually exclusive? Just so I fully understand the problem.

@mishaschwartz
Copy link
Collaborator

@tlvu

As of Ouranosinc/jupyterhub#30 the behaviour is as follows:

  • JUPYTERHUB_AUTHENTICATOR_AUTHORIZATION_URL is set:
    • only users that have permission to access jupyterhub (as defined by magpie service permissions) can log in to jupyterhub
    • if JUPYTERHUB_CRYPT_KEY is set, jupyterhub will also:
      • check if the user still has permission when they spawn a server
      • check if the user still has permission at most every JUPYTERHUB_AUTHENTICATOR_REFRESH_AGE seconds
      • store the login cookies for the user in the jupyterhub database. A user can access these in their jupyterlab session to make accessing other resources that are protected behind twitcher easier.
  • JUPYTERHUB_AUTHENTICATOR_AUTHORIZATION_URL is not set:
    • any user that exists in magpie is allowed to log in to jupyterhub
    • JUPYTERHUB_CRYPT_KEY can be set or not and will have no effect

@mishaschwartz mishaschwartz mentioned this issue Apr 8, 2024
Merged
mishaschwartz added a commit that referenced this issue Apr 9, 2024
@mishaschwartz
Jupyterhub: fix authentication bug when changing settings (#446)
9aa96cd 
## Overview

If the `JUPYTERHUB_AUTHENTICATOR_AUTHORIZATION_URL` variable or
`JUPYTERHUB_CRYPT_KEY` is unset without clearing the jupyterhub
database, users could no longer spawn jupyterlab servers.


## Changes

**Non-breaking changes**
- New component version jupyterhub:4.1.4-20240408

## Related Issue / Discussion

- Resolves #443 

## Additional Information

Links to other issues or sources.

Related to: Ouranosinc/jupyterhub#30

## CI Operations

<!--
The test suite can be run using a different DACCS config with
``birdhouse_daccs_configs_branch: branch_name`` in the PR description.
To globally skip the test suite regardless of the commit message use
``birdhouse_skip_ci`` set to ``true`` in the PR description.
Note that using ``[skip ci]``, ``[ci skip]`` or ``[no ci]`` in the
commit message will override ``birdhouse_skip_ci`` from the PR
description.
-->

birdhouse_daccs_configs_branch: master
birdhouse_skip_ci: false
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mishaschwartz mishaschwartz

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Jupyterhub: fix authentication bug when changing settings bird-house/birdhouse-deploy
2 participants
@mishaschwartz @tlvu