Argo-Jupyter Scheduler parameters behave differently than Jupyter-Scheduler

[ericdatakelly]

Describe the bug

I tried to demo the Jupyter Scheduler with a simple notebook that prints the value of parameter a, but it fails with the message "Workflow failed".

Expected behavior

The notebook should run the cell (print the value of a) and be available to download from the Notebook Jobs table. Note that Jupyter Scheduler will create a new cell at the top of the notebook and add the parameter.

OS and architecture in which you are running Nebari

ASW cloud

How to Reproduce the problem?

1. Create a notebook with a single code cell containing print(a)
2. Make sure your environment has papermill installed
3. Click on the Jupyter Scheduler icon, choose your environment, add parameter a with value of 2 (or whatever number), choose Run now, click Create
4. Reload the Notebook jobs table to refresh until the job is complete
5. Check the status for "Failed" or "Completed". If completed, click the download arrow under "Output files" and check that the notebook printed the value of the parameter.

Command output

No traceback, just the failure message in the job output.

Versions and dependencies used.

Nebari v2023.7.2.dev60+gd58ebb1

Compute environment

AWS

Integrations

No response

Anything else?

No response

[iameskild]

Thanks for reporting this @ericdatakelly!

The Jupyter-Scheduler we're using is a modified version of Jupyter-Scheduler (we call it Argo-Jupyter-Scheduler), the parameter field works slightly differently. They get converted to environment variables that can be used as follows a = os.environ.get('a'). This is briefly mentioned in the docs here.

We can investigate what it would take to match the base Jupyter-Scheduler.