Skip to content
This repository has been archived by the owner on Apr 10, 2024. It is now read-only.

Commit

Permalink
Stable URLs, s3 link improvements (#79)
Browse files Browse the repository at this point in the history 
* add uat base urls and buckets

* config-based urls: maapsec

* dps_magic readme typo

* config based urls: submit_jobs

* config based urls: dps_info

* config based urls: pull_projects

* revert timeout setting change

* config based urls: ssh_info

* add default_host field for explicitness

* remove hard coded url from dps_info client

* clean up hard coded url references

* remove unnecessary state param

* Update dps_magic to use maap_environments.json

* send presigned s3 url req to maap api

* fix ci build error

* Update entrypoint.sh

* Replace statedb environment with server lookup

* replace env statedb with server lookup

* set ade_server in client once during startup

* fix keycloak callback error

* submit jobs convert to tabs to spaces

* fix submit jobs getMetrics result key

* fix dps_info ref to state

* Use maap-py for MAAP API (#73)

* use maap-py for list jobs
* use maap-py for desribe process
* use maap-py for get job result
* use maap-py for get job metrics
* use maap-py for get job status
* use maap-py for get execute inputs
* use maap-py for delete job
* use maap-py for dismiss job
* use maap-py for get capabilities
* use maap-py for publish algorithm
* use maap-py for delete algorithm
* use maap-py for execute job
* fix mismatching keys in submit_jobs to maap-py
* use maap-py for register algorithm

* move edsc url to configuration file

* Registration updates (#74)

* fix mismatching keys in submit_jobs to maap-py
* use maap-py for register algorithm
* pass username to defaultvalues helper
* prepend "demo-${username}-" to tutorial hello world algorithm name
* use regex to match hello-world repo under any group or user
* add dynamic queue selection for algo registration
* properly re-populate algo config with memory selection and add instructions for disk size

* make maap-users org folder read-only

* Fix IStateDB bug

* update hello-world url regex to allow git token in url

* fix org bucket mount line

* update ops search port to 30052

* Parse new api xml format (#76)

* updated parsing for list and describe algorithms
* updated parsing for grabbing execute params and automatically passing job queue

* comment out yaml overwrite for now

* changing default env to ops for now

* Fix terminal webapp setting

* Update entrypoint.sh

* s3link checks if path is folder or if file in valid directory (#77)

* user s3 folder is from under maap-users, debugging presigned s3 url

* s3 url needed profile token, not proxy-ticket

* revert change having s3 folders point to same prefix

* remove tabs side panel wip

* s3 url handler using wrong token name to get orgs

* pass token to get s3url

* getOrgs token different from API proxy-ticket

* debugging

* check s3 paths against mounted paths instead of orgs call

* decode output for string manipulation

* typo

* parse df command for mounted folders only

* remove unused variable

* add selector for s3 link expiration time

* typo

* Add text reminding s3link expiration

* add button to copy s3 link to clipboard

* fix copy s3link to clipboard

* rm unused code and only show copy s3 link button if there is a valid link

* remove duplicated code from merge

* add a few lines about 3d data (#81)

* Update maap code snippet to include default url

* Update entrypoint.sh

* Add comment to che machine name env selector

* Add che7 test config

* Don't use the default maap-py api host

* Fix devche7 letter case

* Update README.md

* remove password file, replace with iam roles

* change edsc sever port

* remove references to che installers

* Update workspace bucket

* Remove s3fs password file reference

* repo.uat.maap-project.org

* Update README.md

* Update handlers.py

* Missed one

* Update supported MAS domains

* remove references to che installers (#84)

* replace org mount with shared workspace mount

* start ssh in entrypoint, add username namespace

* Revert shared ws mount and remove read-only

* Replace memory with queue

* switch from ip address of worker to host address of ade

* update pull projects for che7

* Dev update

* Readme

* Updated entrypoint.sh to make the fix to run Jupyter under Che in 'single-host' mode more generic. Should accommodate Jupyter 2.x & 3.x now.

* Update entrypoint.sh

Added echo in empty else.

* fix key error in pull projects when no proj exists

* Update maap_environments.json

* Added the inline python code to generate the url prefix (#86)

* Updated preview url generation to be more robust so that it'll work with multiple workspaces and won't need the MACHINE_NAME variable in the devfile.

* Also fix directories for publickey authentication.

* Fixed entrypoint.sh to work in the current UAT environment too

* remove s3fs and mounting refernces (#87)

* Reattempt PREVIEW_URL fetch on failure

* update ssh service to find nodeport (#88)

* remove sucess message that actually fails

* Add workspace argument to presigned s3 url paths

* Fix environment pre-ops 'workspace_bucket' name

* Remove timestamp from algo reg

* update readme

Co-authored-by: bsatoriu <[email protected]>
Co-authored-by: Debellis <[email protected]>
Co-authored-by: George Chang <[email protected]>
Co-authored-by: George Chang <[email protected]>
  • Loading branch information
@echyam @bsatoriu
@mayadebellis @gchang
5 people authored Jun 3, 2021
1 parent 803ef8e commit 8b5bc7f
Show file tree
Hide file tree
Showing 26 changed files with 829 additions and 602 deletions.
174 changes: 33 additions & 141 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# MAAP Jupyter Interface

This repo includes JupyterLab extensions that have been built for the MAAP project (https://che-k8s.maap.xyz/)
This repo includes JupyterLab extensions that have been built for the MAAP project (https://www.maap-project.org/)

In order to use each of these extensions they must be installed and enabled in your environment. Instructions for each extension can be
found in the respective folder. Make sure each extension's dependencies are installed first.
Expand All @@ -16,8 +16,8 @@ Create a new Conda environment and install common dependencies and versions:
conda create --name maap-ade
conda activate maap-ade
conda install conda=4.7.12 jupyterlab=2.1.4 nodejs=10.13.0 gitpython=3.0.2
jupyter labextension install jupyterlab_toastify@3.3.0 --no-build
npm i jupyterlab_toastify@3.3.0
jupyter labextension install jupyterlab_toastify@3.0.0
npm i jupyterlab_toastify@3.0.0
pip install plotly==4.0.0
jupyter labextension install @jupyter-widgets/[email protected]
```
Expand Down Expand Up @@ -68,143 +68,35 @@ Some Jupyter Extensions/Resources we have found helpful:
In JupyterLab's update to the stable 1.0 version, they have also updated and added lots of documentation on extension
development. I recommend taking a look at [this](https://jupyterlab.readthedocs.io/en/stable/developer/extension_dev.html).

## Deploying Extensions as Part of Eclipse Che
### Dockerizing
Our development process involves building and running an extension locally in jupyterlab using a conda env before
installing it on the che server. To enable an extension in Che, it must be included in the base docker image/stack that a
Che workspace is launched with. The dockerfile that extensions are included in is the `Dockerfile` and the highest level
in this repo. At the point of adding your extension into the Docker image, some minor changes may have to be made
(mainly path issues). This will be explained in the bullets below.

An instance of this repository lives on the Che server under `~/che/dockerfiles/maap-jupyter-ide`. Once an extension has been tested locally, rebuild the docker
image with your new extensions.


- Add your install to the Dockerfile. For example:
```bash
# jlab pull projects into /projects directory
COPY pull_projects /pull_projects
RUN cd /pull_projects && npm run build
RUN cd /pull_projects && jupyter labextension link .
RUN cd /pull_projects && pip install -e .
RUN cd /pull_projects && jupyter serverextension enable --py pull_projects --sys-prefix

```
- If your extension includes a server extension you also need to modify `entrypoint.sh`. This is because jupyter
server extensions function off of having a standard base url, but in the context of che the url is not what jupyter
thinks it is.
- Here is some magic that fixes it (add this line and replace with the path to where
your `load_jupyter_server_extension` function is)
## Steps for Contributing
### Each Contribution
- Clone repository locally to test and develop extensions. Create your own branch and once you have tested it and are ready to contribute, put a PR into the develop branch.
- After you have validated your code is working locally, the next step is to test it in the MAAP ADE in the devlopment environment.
- When there is an update to the develop branch, such as merging in a PR, the CI will be kicked off automatically and will build the image necessary for use in the MAAP ADE. This image will end in the `:develop` tag. This is intended to be used to test in the development environment.
- When updates in develop have been tested in the MAAP ADE and deemed stable, merge them into master.
### Ready for Release
- Once there have been changes that warrant a new release to the operations environment, create a new Github release from the master branch. Name the release the next corresponding number following the `v3.0.1` schema. In the notes of the release include a changelog of the updates included in that release
- Then, update the existing `stable` tag to point to the same place as the release that you just created. This can easily be done on the command line by checking out the latest code on master (or whatever commit the release was created from) and running the following:
```
git tag -f stable
git push -f origin stable
```
- The CI will also be kicked off for `stable` and `v` tags. The `stable` tag is used in the devfiles in the operations environment and automatically pulled in each new workspace, so it is very important pushing changes to this tag have been thuroughly tested. The `v` tag is not used in any environment but is used to track versions and the changes we role out. If there is an issue found in the `stable` tag we can roll it back to the previous `v` release by checking out that release and running the lines above.
- The CI trigger information can be found .github/workflows/gitlab-trigger.yml and in the actions tab

## Docker Images and CI Process
- The CI and images built off of this code live on MAAP's hosted GitLab (https://mas.maap-project.org/root).
- Triggered from maap-jupyter-ide
- Will build corresponding maap-jupyter-ide branch on all of the base images listed in the base_images.txt file
- Triggered from Gitlab
- If triggered from a [base image](https://mas.maap-project.org/root/ade-base-images) being updated, it will only rebuild that base image with map-jupyter-ide’s stable tag on top of it.
- If triggered from an update in the [jupyter-image](https://mas.maap-project.org/root/jupyter-image) repo, it will rebuild all of the base images listed in base_iamges.txt with maap-jupyter-ide’s stable tag.
- Troubleshooting
- If the image failed to build for an unexpected reason, it is likely because the gitlab runner ran out of space. Ssh onto the runner, clean up the docker images (you can get rid of anything - everything important is on s3 backed registries) and restart the failed job.


## Troubleshooting
- If your extension includes a server extension you also need to modify `entrypoint.sh`. This is because jupyter server extensions function off of having a standard base url, but in the context of che the url is not what jupyter thinks it is.
- Here is some magic that fixes it (add this line and replace with the path to where your `load_jupyter_server_extension` function is)
```bash
perl -pi -e "s|web_app.settings\['base_url'\]|'/'|g" /show_ssh_info/show_ssh_info/__init__.py
```
- Then rebuild the docker image. `microk8s.docker build -t localhost:32000/che-jupyter-lab-ide .`
- Push! `microk8s.docker push localhost:32000/che-jupyter-lab-ide `
- Now when you build a new workspace with the `localhost:32000/che-jupyter-lab-ide` image it will automatically
fetch the new image. (found in the stack's `Recipe` or `Raw Configuration`)
- you can also specify the image tag to use in your build on the stack if you want to use a previous build
- Any change pushed to `microk8s.docker push localhost:32000/che-jupyter-lab-ide ` will affect the default stacks
on all user accounts. If you are testing something, you can create your own image and your own stack to play around with.
### Che Stacks
To make your custom docker image available to users in Che, you need to make a new stack that creates workspaces using your image and make it available to users.
Below is an example stack configuration using our locally built dockerized juptyer image with MAAP extensions installed.
Make sure to replace the image name in `workspaceConfig.environments.default.recipe.image` with the location of your image.
In order for SSH-ing into the workspace to be possible, the `org.eclipse.che.exec` and `org.eclipse.che.ssh` installers must be enabled under `workspaceConfig.environments.default.machines.ws/jupyter.installers`.
#### Che Stack Raw Configuration
```
{
"scope": "general",
"description": "Use this one. Stable jupyter. No extra packages",
"creator": "b07e3a58-ed50-4a6e-be17-fcf49ff8b242",
"tags": [
"MAAP",
"JUPYTER",
"STABLE"
],
"workspaceConfig": {
"defaultEnv": "default",
"environments": {
"default": {
"recipe": {
"contentType": "text/x-yaml",
"type": "kubernetes",
"content": "kind: List\nitems:\n - \n apiVersion: v1\n kind: Pod\n metadata:\n name: ws\n labels:\n name: ws\n spec:\n containers:\n - \n name: jupyter\n image: 'localhost:32000/stable-ide:latest'\n resources:\n limits:\n memory: 1024Mi\n securityContext:\n privileged: true\n - \n apiVersion: v1\n kind: Service\n metadata:\n name: ws\n spec:\n type: NodePort\n ports:\n - \n port: 22\n selector:\n name: ws\n \n "
},
"machines": {
"ws/jupyter": {
"env": {
"MACHINE_NAME": "WS_JUPYTER"
},
"servers": {
"jupyter": {
"path": "/",
"attributes": {
"cookiesAuthEnabled": "true",
"type": "ide",
"secure": "true"
},
"protocol": "http",
"port": "3100"
}
},
"installers": [
"org.eclipse.che.exec",
"org.eclipse.che.ssh"
],
"volumes": {
"projects": {
"path": "/projects"
}
},
"attributes": {}
}
}
}
},
"projects": [],
"commands": [],
"name": "default",
"attributes": {},
"links": []
},
"components": [],
"name": "maap-jupyter-ide",
"id": "stacktdo2q0ixhv7cge00"
}
```
#### Enabling Privileged Docker Containers
1. Cluster Privileges
- in `/var/snap/microk8s/current/args/kubelet` and `/var/snap/microk8s/current/args/kube-apiserver`, append `--allow-privileged`
- restart both services:
```
sudo systemctl restart snap.microk8s.daemon-apiserver
sudo systemctl restart snap.microk8s.daemon-kubelet
```
2. Che Permissions
- in `che/dockerfiles/init/manifest/che.env`, set `CHE_DOCKER_PRIVILEGED=true` under the Privileged Mode section
- restart Che
#### Creating and Sharing Stacks
To create a stack, you write a raw configuration with all the che and docker settings your workspace will require, including installers, volumes, docker run tags, docker images, etc. See the example above.
To share a stack, you will need to be the owner (creator) of the stack.
Go to the homepage of where your Che instance is hosted and add `/swagger` to the end of the url for an interface with Che's API. Under the `permissions`section, make a POST request with the users you want to share with and the id of your stack (shows up at the bottom of the configuration after creation).
POST body:
```
{
"userId": "*",
"domainId": "stack",
"instanceId": "${STACK_ID}",
"actions": [
"read",
"search"
]
}
```
reference: https://www.eclipse.org/che/docs/che-6/stacks.html#sharing-stacks-and-system-stacks
.
9 changes: 4 additions & 5 deletions dps_info/src/jobinfo.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -141,14 +141,12 @@ export class JobTable extends Widget {
}

async _getJobList(me:JobTable) {
// console.log(this._username);
const res:RequestResult = await getJobs(me._state, me._username);
if(res.ok){
let json_response:any = res.json();
INotification.success("Get user jobs success.");
// console.log(json_response);
if (json_response['status_code'] === 200){
// let resp = json_response['result'];
INotification.success("Get user jobs success.");

me._table = json_response['result'];
JOBS = json_response['jobs'];
DISPLAYS = json_response['displays'];
Expand Down Expand Up @@ -1130,8 +1128,9 @@ export class JobWidget extends Widget {
const res:RequestResult = await getJobs(this._state, this._username);
if(res.ok){
let json_response:any = res.json();
INotification.success("Get user jobs success.");
if (json_response['status_code'] === 200){
INotification.success("Get user jobs success.");

me._table = json_response['table'];
JOBS = json_response['jobs'];
DISPLAYS = json_response['displays'];
Expand Down
20 changes: 18 additions & 2 deletions edsc_extension/edsc_extension/handlers.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,25 @@
import nbformat
from notebook.base.handlers import IPythonHandler
import subprocess
import functools
import json
import maap
from maap.maap import MAAP

@functools.lru_cache(maxsize=128)
def get_maap_config(host):
path_to_json = os.path.join(os.path.dirname(os.path.abspath(__file__)), '../..', 'maap_environments.json')

with open(path_to_json) as f:
data = json.load(f)

match = next((x for x in data if host in x['ade_server']), None)
maap_config = next((x for x in data if x['default_host'] == True), None) if match is None else match

return maap_config

def maap_api(host):
return get_maap_config(host)['api_server']

class GetGranulesHandler(IPythonHandler):
def printUrls(self, granules):
Expand All @@ -18,7 +34,7 @@ def printUrls(self, granules):

def get(self):

maap = MAAP()
maap = MAAP(maap_api(self.request.host))
cmr_query = self.get_argument('cmr_query', '')
limit = str(self.get_argument('limit', ''))
print("cmr_query", cmr_query)
Expand All @@ -35,7 +51,7 @@ def get(self):

class GetQueryHandler(IPythonHandler):
def get(self):
maap = MAAP()
maap = MAAP(maap_api(self.request.host))
cmr_query = self.get_argument('cmr_query', '')
limit = str(self.get_argument('limit', ''))
query_type = self.get_argument('query_type', 'granule')
Expand Down
Loading

0 comments on commit 8b5bc7f

Please sign in to comment.