Refactoring/#258 used a non root user to run jupyter in the docker im…

…age ai lab (#231)

* Use separate user for running Jupyter
Additionally Same as AMI and VM Editions also the Docker Edition uses a
non-root user for executing the ansible installation.
* added dependency to gpg-agent
* Added become for disabling core dumps
* Improved Ansible output for pip install
* fixed Ansible warning about providing a list of dictionaries to vars:
See https://docs.ansible.com/ansible/latest/porting_guides/porting_guide_8.html#deprecated
* Updated tests for docker Edition
* Increased time out for commiting the Docker container
As integration tests in test_create_dss_docker_image.py failed with timeout
error in test_install_notebook_connector during docker.commit().
* Updated dependencies to fix vulnerabilities
* Added free disk space to workflows/check_ci.yaml
* Use user "jupyter" for EC2, too
* Updated documentation
* Updated dependency network-manager
[CodeBuild]
---------

Co-authored-by: Torsten Kilias <[email protected]>

.github/workflows/check_ci.yaml

@@ -14,6 +14,25 @@ jobs:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0
+
+     - name: Free disk space
+       uses: jlumbroso/free-disk-space@main
+       with:
+         tool-cache: true
+         large-packages: false
+
+     - name: Free disk space by removing large directories
+       run: |
+         sudo rm -rf /usr/local/graalvm/
+         sudo rm -rf /usr/local/.ghcup/
+         sudo rm -rf /usr/local/share/powershell
+         sudo rm -rf /usr/local/share/chromium
+         sudo rm -rf /usr/local/lib/node_modules
+         sudo rm -rf /opt/ghc
+
+     - name: Show available disk space
+       run: df -h
+
      - name: Setup Python & Poetry Environment
        uses: ./.github/actions/prepare_poetry_env
 
@@ -39,6 +58,7 @@ jobs:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0
+
      - name: Free disk space
        uses: jlumbroso/free-disk-space@main
        with:

doc/changes/changes_1.1.0.md

@@ -34,4 +34,4 @@ n/a
 * #217: Changed notebook-connector dependency, now installing it from PyPi.
 * #220: Changed default ports in the external database configuration.
 * #221: Changed wording in the main configuration notebook, as suggested by PM.
-
+* #66: Used a non-root user to run Jupyter in the Docker Image ai-lab

doc/developer_guide/aws.md

@@ -32,7 +32,7 @@ After the export has finished, the cloudformation stack and the keypair is remov
 
 Installs all dependencies via Ansible:
 * installs and configures Jupyter
-* installs Docker and adds the user `ubuntu` to the docker group
+* installs Docker and adds the user `jupyter` to the docker group
 * changes the netplan configuration. This is necessary to have proper network configuration when running the VM image
 
 Finally, the default password will be set, and also the password will be marked as expired, such that the user will be forced to enter a new password during initial login.

doc/user_guide/ami-usage.md

@@ -38,6 +38,8 @@ Check the [AWS documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuid
      - Click button "Launch instance"
 6. As soon as the machine becomes available you can connect per ssh with user `ubuntu`: `ssh -i your_key.pem ubuntu@the_new_ec_instance`
 
+**Please note**: When you want to modify the settings of jupyterlab or to install additional pip packages into the virtual env `jupyterenv`, you need to prefix your commands with `sudo --login --user jupyter`.
+
 ## Login
 
 See [Login to AMI and VM Editions](login-vm-and-ami.md) for logging in to the system.

doc/user_guide/docker/docker-usage.md

@@ -39,14 +39,14 @@ In this scenario the AI-Lab Docker container does not need access to the Docker
 The following command will
 * Download the Docker image for the specified version `$VERSION` of the AI-Lab if the image of the specified version is not yet available in your Docker service
 * Run a Docker container using this image
-* Mount the volume `$VOLUME` to the directory `/root/notebooks` inside the container
+* Mount the volume `$VOLUME` to the directory `/home/jupyter/notebooks` inside the container
   * If the volume does not exist yet, then it will be created automatically.
 * Forward port `49494` on the [daemon machine](prerequisites.md) to allow connections from all IP addresses matched by `$LISTEN_IP`
 
 ```shell
 docker run \
   --name ${CONTAINER_NAME} \
-  --volume ${VOLUME}:/root/notebooks \
+  --volume ${VOLUME}:/home/jupyter/notebooks \
   --publish ${LISTEN_IP}:49494:49494 \
   exasol/ai-lab:${VERSION}
 ```
@@ -68,7 +68,7 @@ In this scenario you must enable the AI-Lab Docker container to access the Docke
 ```shell
 docker run \
   --name ${CONTAINER_NAME} \
-  --volume ${VOLUME}:/root/notebooks \
+  --volume ${VOLUME}:/home/jupyter/notebooks \
   --volume /var/run/docker.sock:/var/run/docker.sock \
   --publish ${LISTEN_IP}:49494:49494 \
   exasol/ai-lab:${VERSION}
@@ -122,8 +122,8 @@ port to the same port then you can connect with http://localhost:49494.
 └─┘┴  ─┴┘┴ ┴ ┴ └─┘   ┴ └─┘└─┘┴└─  └┘└─┘┴   ┴  ┴ └─┘┴└─  ┴  ┴ ┴└─┘└─┘└┴┘└─┘┴└──┴┘ o
 
 The default password is "ailab".
-To update the password, log in to the Docker container as the user root and run
-    /root/jupyterenv/bin/jupyter-lab server password
+To update the password, log in to the Docker container as the user jupyter and run
+    /home/jupyter/jupyterenv/bin/jupyter-lab server password
 ```
 
 Using an internet browser you then can connect to the Jupyter server running in the Docker container in order to follow the tutorials presented by a set of Jupyter notebooks, see [Connecting to Jupyter Service](../jupyter.md#open-jupyter-in-your-browser).

doc/user_guide/docker/managing-user-data.md

@@ -16,8 +16,8 @@ The Docker Edition of Exasol AI-Lab allows you to
 ## Basics
 
 In order to save your changes persistently, to reuse, backup, and restore them, the AI-Lab
-* Keeps the notebook files as well as the SCS in the directory `/root/notebooks` inside the Docker container
-* Uses a so-called [_Docker Volume_](https://docs.docker.com/storage/volumes) mounted to `/root/notebooks` to store the content independent of the Docker container
+* Keeps the notebook files as well as the SCS in the directory `/home/jupyter/notebooks` inside the Docker container
+* Uses a so-called [_Docker Volume_](https://docs.docker.com/storage/volumes) mounted to `/home/jupyter/notebooks` to store the content independent of the Docker container
 
 ## Additional Environment Variables
 

doc/user_guide/jupyter.md

@@ -1,13 +1,9 @@
 ## Open Jupyter In Your Browser
 
-Root location
-* For Exasol AI-Lab's VM and AMI editions the root location is `$ROOT=/home/ubuntu`.
-* For the Docker edition the root location is `$ROOT=/root`.
-
 | Item                | Location or value                         |
 |---------------------|-------------------------------------------|
-| Virtual environment | location `/$ROOT/jupyterenv`              |
-| Location notebooks  | location `/$ROOT/notebooks`               |
+| Virtual environment | location `/home/jupyter/jupyterenv`       |
+| Location notebooks  | location `/home/jupyter/notebooks`        |
 | Password            | `ailab`                                   |
 | HTTP Port           | `49494` (or the port you forwarded it to) |
 

doc/user_guide/login-vm-and-ami.md

@@ -1,5 +1,6 @@
 ## Login to AMI and VM Editions
 
+<!-- I assume the login user for AMI and VM Editions is still ubuntu. -->
 Username: **ubuntu**
 
 At the first login to the AI-Lab (image or AMI) you will be prompted to change your password.

exasol/ds/sandbox/lib/dss_docker/create_image.py

@@ -112,7 +112,12 @@ def _docker_file(self) -> importlib_resources.abc.Traversable:
 
     def _start_container(self) -> DockerContainer:
         self._start = datetime.now()
-        docker_client = docker.from_env()
+        # default timeout is 60 seconds.
+        docker_client = docker.from_env(timeout=600)
+        try:
+            return docker_client.containers.get(self.container_name)
+        except:
+            pass
         docker_file = self._docker_file()
         _logger.info(f"Creating docker image {self.image_name} from {docker_file}")
         if self.registry is not None:
@@ -157,6 +162,7 @@ def _commit_container(
         notebook_folder_initial = get_fact(facts, "notebook_folder", "initial")
         conf = {
             "Entrypoint": entrypoint(facts),
+            "User": get_fact(facts, "jupyter", "user"),
             "Cmd": [],
             "Volumes": {notebook_folder_final: {}, },
             "ExposedPorts": {f"{port}/tcp": {}},
@@ -174,7 +180,7 @@ def _cleanup(self, container: DockerContainer):
         if container is None:
             return
         if self.keep_container:
-            _logger.info("Keeping container running")
+            _logger.info(f"Keeping container {self.container_name} running")
             return
         _logger.info("Stopping container")
         container.stop()

exasol/ds/sandbox/runtime/ansible/ai_lab_docker_playbook.yml

@@ -4,22 +4,39 @@
   vars:
       ansible_python_interpreter: python3
   tasks:
-    - name: Add docker container to inventory
+    - name: Add Docker Container to Ansible Inventory
       add_host:
         name: "{{docker_container}}"
         groups: docker_container_group
         ansible_connection: docker
 
+- name: Prepare Ansible Access
+  hosts: docker_container_group
+  gather_facts: true
+  vars:
+    my_ansible_user: "ubuntu"
+    jupyter_user: "jupyter"
+    jupyter_group: "jupyter"
+    jupyter_user_home: "/home/jupyter"
+    need_sudo: no
+  tasks:
+    - import_tasks: apt_update.yml
+    - name: Ansible Access
+      include_role:
+        name: ansible_access
+
 - name: Setup AI-Lab Docker Container
   hosts: docker_container_group
+  remote_user: "ubuntu"
   gather_facts: true
   vars:
-      ansible_python_interpreter: python3
-      user_name: "{{ ansible_user_id }}"
-      user_home: "{{ ansible_env.HOME }}"
-      initial_notebook_folder: "{{ user_home }}/notebook-defaults"
-      need_sudo: false
-      docker_integration_test: true
+    ansible_python_interpreter: python3
+    user_name: "jupyter"
+    user_group: "jupyter"
+    user_home: "/home/jupyter"
+    initial_notebook_folder: "{{ user_home }}/notebook-defaults"
+    need_sudo: yes
+    docker_integration_test: true
   tasks:
     - import_tasks: general_setup_tasks.yml
     - import_tasks: cleanup_tasks.yml

exasol/ds/sandbox/runtime/ansible/apt_update.yml

@@ -0,0 +1,5 @@
+- name: Update and Upgrade apt Packages
+  apt:
+    upgrade: yes
+    update_cache: yes
+  become: "{{need_sudo}}"

exasol/ds/sandbox/runtime/ansible/create_jupyter_user.yml

@@ -0,0 +1,4 @@
+- name: Add user 'jupyter' for running jupyter
+  ansible.builtin.user:
+    name: "{{ user_name }}"
+    home: "{{ user_home }}"

exasol/ds/sandbox/runtime/ansible/ec2_playbook.yml

@@ -2,12 +2,14 @@
   hosts: ec2
   vars:
       ansible_python_interpreter: /usr/bin/python3
-      user_name: ubuntu
-      user_home: /home/ubuntu
+      user_name: jupyter
+      user_home: /home/jupyter
+      user_group: jupyter
       initial_notebook_folder: "{{ user_home }}/notebooks"
       need_sudo: yes
   remote_user: ubuntu
   tasks:
+    - import_tasks: apt_update.yml
     - import_tasks: general_setup_tasks.yml
     - import_tasks: ec2_setup_tasks.yml
     - import_tasks: cleanup_tasks.yml

exasol/ds/sandbox/runtime/ansible/general_setup_tasks.yml

@@ -1,4 +1,4 @@
-- name: Set facts for entry point
+- name: Set facts for Entry Point
   vars:
     jupyter_virtualenv: "{{user_home}}/jupyterenv"
   ansible.builtin.set_fact:
@@ -15,14 +15,12 @@
         initial: "{{ initial_notebook_folder }}"
         final: "{{user_home}}/notebooks"
     cacheable: yes
-- name: Update and upgrade apt packages
-  apt:
-    upgrade: yes
-    update_cache: yes
-  become: "{{need_sudo}}"
 - name: Install rsync
   include_role:
     name: rsync
+- name: Create Group and User for Running Jupyter Server
+  include_role:
+    name: jupyter_user
 - name: Copy Entry Point Script
   include_role:
     name: entrypoint
@@ -36,7 +34,14 @@
     jupyterlab_password: "{{ dss_facts.jupyter.password }}"
     jupyterlab_notebook_folder_initial: "{{ dss_facts.notebook_folder.initial }}"
     jupyterlab_notebook_folder: "{{ dss_facts.notebook_folder.final }}"
-- name: Clear pip cache
+- name: Change Owner of All Files and Dirs in Home Directory
+  ansible.builtin.file:
+    path: "{{ user_home }}"
+    owner: "{{ user_name }}"
+    group: "{{ user_group }}"
+    recurse: true
+  become: "{{ need_sudo }}"
+- name: Clear pip Cache
   ansible.builtin.file:
     path: /root/.cache/pip
     state: absent

exasol/ds/sandbox/runtime/ansible/roles/ansible_access/defaults/main.yml

@@ -0,0 +1,4 @@
+---
+
+apt_dependencies:
+  - sudo=1.8.31-1ubuntu1.5

exasol/ds/sandbox/runtime/ansible/roles/ansible_access/tasks/main.yml

@@ -0,0 +1,16 @@
+- name: Install sudo
+  apt:
+    name: "{{apt_dependencies}}"
+    state: latest
+    install_recommends: false
+
+- name: Add dedicated user for executing ansible tasks
+  ansible.builtin.user:
+    name: "{{ my_ansible_user }}"
+- name: Add to sudoers
+  community.general.sudoers:
+    name: Add user ansible
+    user: "{{ my_ansible_user }}"
+    state: present
+    nopassword: true
+    commands: ALL

exasol/ds/sandbox/runtime/ansible/roles/coredumps/tasks/main.yml

@@ -4,3 +4,4 @@
     regexp: ^[^#].*core
     line: '*        hard       core      0'
     create: true
+  become: "{{ need_sudo }}"

exasol/ds/sandbox/runtime/ansible/roles/docker/defaults/main.yml

@@ -2,5 +2,6 @@
 
 apt_dependencies:
   - apt-transport-https=2.0.10
+  - gpg-agent=2.2.19-3ubuntu2.2
   - ca-certificates=20230311ubuntu0.20.04.1
   - software-properties-common=0.99.9.12

exasol/ds/sandbox/runtime/ansible/roles/docker/tasks/main.yml

@@ -3,6 +3,7 @@
     name: "{{apt_dependencies}}"
     state: latest
     update_cache: true
+    install_recommends: false
   become: "{{need_sudo}}"
 
 - name: Add Docker GPG apt Key
@@ -22,11 +23,12 @@
     name: docker-ce
     state: latest
     update_cache: true
+    install_recommends: false
   become: "{{need_sudo}}"
 
 - name: Adding docker users (for use without sudo)
   user:
     name: "{{user_name}}"
     append: yes
     groups: docker
-  become: "{{need_sudo}}"
+  become: "{{need_sudo}}"

exasol/ds/sandbox/runtime/ansible/roles/entrypoint/tasks/main.yml

@@ -3,3 +3,6 @@
     src: "entrypoint.py"
     dest: "{{ dss_facts.entrypoint }}"
     mode: 0644
+    group: "{{ user_group }}"
+    owner: "{{ user_name }}"
+  become: "{{need_sudo}}"

exasol/ds/sandbox/runtime/ansible/roles/jupyter/defaults/main.yml

@@ -2,6 +2,7 @@
 
 jupyterlab_ip: '*'
 jupyterlab_config: "{{user_home}}/.jupyter/jupyter_lab_config.py"
+jupyterlab_config_json: "{{user_home}}/.jupyter/jupyter_server_config.json"
 jupyterlab_default_url: "/lab/tree/start.ipynb"
 
 # overridden in include_role

exasol/ds/sandbox/runtime/ansible/roles/jupyter/files/requirements_dependencies.txt → exasol/ds/sandbox/runtime/ansible/roles/jupyter/files/notebook_requirements.txt

@@ -3,7 +3,7 @@ numpy==1.23.1
 pandas==1.4.3
 scikit-learn==1.0.2
 matplotlib==3.7.4
-jupysql==0.10.7
+jupysql==0.10.10
 sqlalchemy_exasol==4.6.3
 stopwatch.py==2.0.1
 boto3==1.26.163