Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactoring/#258 used a non root user to run jupyter in the docker image ai lab #231

Merged
Show file tree
Hide file tree
Changes from 34 commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
7bacff9
Use separate user for running Jupyter
ckunki Feb 23, 2024
6663562
Used a non-root user to run Jupyter in the Docker Image ai-lab
ckunki Feb 26, 2024
194e9d1
Additional changes based on experiments
ckunki Feb 27, 2024
5befe3e
Fixed ansible tasks
ckunki Feb 27, 2024
1e510fc
remove commented line
ckunki Feb 27, 2024
69de9d8
Added become for disabling core dumps
ckunki Feb 27, 2024
d9eed17
poetry lock
ckunki Feb 27, 2024
af1a929
Fixed test
ckunki Feb 27, 2024
c776849
Improved Ansible output for pip install
ckunki Feb 27, 2024
36117ea
Improved Ansible output for pip install
ckunki Feb 27, 2024
5229b26
Merge branch 'main' into refactoring/#258-Used_a_non-root_user_to_run…
ckunki Feb 27, 2024
2304741
Removed test scaffolds
ckunki Feb 27, 2024
b030baf
Capitalized name of Ansible task
ckunki Feb 27, 2024
a66235d
Updated comments in Ansible tasks
ckunki Feb 27, 2024
71ecf25
Updated user guide for Docker Edition
ckunki Feb 27, 2024
514cdd0
Updated tests for docker Edition
ckunki Feb 27, 2024
50b4432
Increased time out for commiting the Docker container
ckunki Feb 27, 2024
b595ae6
Added comment for timeout
ckunki Feb 27, 2024
97b5635
Show name of docker container when keeping it alive
ckunki Feb 28, 2024
b8229da
Moved integration/test_create_dss_docker_image to codebuild
ckunki Feb 28, 2024
f347df9
[CodeBuild]
ckunki Feb 28, 2024
e0a0cae
Fixed import for ci tests
ckunki Feb 28, 2024
dc426ad
[CodeBuild]
ckunki Feb 28, 2024
c254cee
Updated dependencies to fix vulnerabilities
ckunki Feb 28, 2024
0235d4d
[CodeBuild]
ckunki Feb 28, 2024
92781ad
Added free disk space to workflows/check_ci.yaml
ckunki Feb 28, 2024
62004a3
Use user "jupyter" for EC2, too
ckunki Feb 28, 2024
3c19388
Updated documentation
ckunki Feb 28, 2024
d9d6205
[CodeBuild]
ckunki Feb 28, 2024
7ee7abf
Updated dependency network-manager
ckunki Feb 28, 2024
887d558
[CodeBuild]
ckunki Feb 28, 2024
8ace780
Fixed test test_jupyter_password_message_shown
ckunki Feb 28, 2024
427696b
Removed comment
ckunki Feb 28, 2024
3467495
Merge branch 'main' into refactoring/#258-Used_a_non-root_user_to_run…
ckunki Feb 28, 2024
524656a
Fixed review findings
ckunki Feb 28, 2024
dfb6b5e
Fixed review findings
ckunki Feb 28, 2024
0574c95
Update exasol/ds/sandbox/runtime/ansible/roles/jupyter/templates/etc/…
ckunki Feb 29, 2024
1778e81
Fixed review finding
ckunki Feb 29, 2024
b10fa2e
Fixed failing test
ckunki Feb 29, 2024
ab3b254
Merge branch 'main' into refactoring/#258-Used_a_non-root_user_to_run…
ckunki Feb 29, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions .github/workflows/check_ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,25 @@ jobs:
- uses: actions/checkout@v3
with:
fetch-depth: 0

- name: Free disk space
uses: jlumbroso/free-disk-space@main
with:
tool-cache: true
large-packages: false

- name: Free disk space by removing large directories
run: |
sudo rm -rf /usr/local/graalvm/
sudo rm -rf /usr/local/.ghcup/
sudo rm -rf /usr/local/share/powershell
sudo rm -rf /usr/local/share/chromium
sudo rm -rf /usr/local/lib/node_modules
sudo rm -rf /opt/ghc

- name: Show available disk space
run: df -h

- name: Setup Python & Poetry Environment
uses: ./.github/actions/prepare_poetry_env

Expand All @@ -39,6 +58,7 @@ jobs:
- uses: actions/checkout@v3
with:
fetch-depth: 0

- name: Free disk space
uses: jlumbroso/free-disk-space@main
with:
Expand Down
2 changes: 1 addition & 1 deletion doc/changes/changes_1.1.0.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,4 +33,4 @@ n/a
* #217: Changed notebook-connector dependency, now installing it from PyPi.
* #220: Changed default ports in the external database configuration.
* #221: Changed wording in the main configuration notebook, as suggested by PM.

* #66: Used a non-root user to run Jupyter in the Docker Image ai-lab
2 changes: 1 addition & 1 deletion doc/developer_guide/aws.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ After the export has finished, the cloudformation stack and the keypair is remov

Installs all dependencies via Ansible:
* installs and configures Jupyter
* installs Docker and adds the user `ubuntu` to the docker group
* installs Docker and adds the user `jupyter` to the docker group
* changes the netplan configuration. This is necessary to have proper network configuration when running the VM image

Finally, the default password will be set, and also the password will be marked as expired, such that the user will be forced to enter a new password during initial login.
Expand Down
1 change: 1 addition & 0 deletions doc/user_guide/ami-usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ Check the [AWS documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuid
- For the storage we recommend to keep the pre-selected 100GB volume
- Click button "Launch instance"
6. As soon as the machine becomes available you can connect per ssh with user `ubuntu`: `ssh -i your_key.pem ubuntu@the_new_ec_instance`
<!-- I assume the ssh user for AMI and VM Editions is still ubuntu. -->
ckunki marked this conversation as resolved.
Show resolved Hide resolved

## Login

Expand Down
8 changes: 4 additions & 4 deletions doc/user_guide/docker/docker-usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,14 +39,14 @@ In this scenario the AI-Lab Docker container does not need access to the Docker
The following command will
* Download the Docker image for the specified version `$VERSION` of the AI-Lab if the image of the specified version is not yet available in your Docker service
* Run a Docker container using this image
* Mount the volume `$VOLUME` to the directory `/root/notebooks` inside the container
* Mount the volume `$VOLUME` to the directory `/home/jupyter/notebooks` inside the container
* If the volume does not exist yet, then it will be created automatically.
* Forward port `49494` on the [daemon machine](prerequisites.md) to allow connections from all IP addresses matched by `$LISTEN_IP`

```shell
docker run \
--name ${CONTAINER_NAME} \
--volume ${VOLUME}:/root/notebooks \
--volume ${VOLUME}:/home/jupyter/notebooks \
--publish ${LISTEN_IP}:49494:49494 \
exasol/ai-lab:${VERSION}
```
Expand All @@ -68,7 +68,7 @@ In this scenario you must enable the AI-Lab Docker container to access the Docke
```shell
docker run \
--name ${CONTAINER_NAME} \
--volume ${VOLUME}:/root/notebooks \
--volume ${VOLUME}:/home/jupyter/notebooks \
--volume /var/run/docker.sock:/var/run/docker.sock \
--publish ${LISTEN_IP}:49494:49494 \
exasol/ai-lab:${VERSION}
Expand Down Expand Up @@ -123,7 +123,7 @@ port to the same port then you can connect with http://localhost:49494.

The default password is "ailab".
To update the password, log in to the Docker container as the user root and run
/root/jupyterenv/bin/jupyter-lab server password
/home/jupyter/jupyterenv/bin/jupyter-lab server password
ckunki marked this conversation as resolved.
Show resolved Hide resolved
```

Using an internet browser you then can connect to the Jupyter server running in the Docker container in order to follow the tutorials presented by a set of Jupyter notebooks, see [Connecting to Jupyter Service](../jupyter.md#open-jupyter-in-your-browser).
Expand Down
4 changes: 2 additions & 2 deletions doc/user_guide/docker/managing-user-data.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,8 @@ The Docker Edition of Exasol AI-Lab allows you to
## Basics

In order to save your changes persistently, to reuse, backup, and restore them, the AI-Lab
* Keeps the notebook files as well as the SCS in the directory `/root/notebooks` inside the Docker container
* Uses a so-called [_Docker Volume_](https://docs.docker.com/storage/volumes) mounted to `/root/notebooks` to store the content independent of the Docker container
* Keeps the notebook files as well as the SCS in the directory `/home/jupyter/notebooks` inside the Docker container
* Uses a so-called [_Docker Volume_](https://docs.docker.com/storage/volumes) mounted to `/home/jupyter/notebooks` to store the content independent of the Docker container

## Additional Environment Variables

Expand Down
8 changes: 2 additions & 6 deletions doc/user_guide/jupyter.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,9 @@
## Open Jupyter In Your Browser

Root location
* For Exasol AI-Lab's VM and AMI editions the root location is `$ROOT=/home/ubuntu`.
* For the Docker edition the root location is `$ROOT=/root`.

| Item | Location or value |
|---------------------|-------------------------------------------|
| Virtual environment | location `/$ROOT/jupyterenv` |
| Location notebooks | location `/$ROOT/notebooks` |
| Virtual environment | location `/home/jupyter/jupyterenv` |
| Location notebooks | location `/home/jupyter/notebooks` |
| Password | `ailab` |
| HTTP Port | `49494` (or the port you forwarded it to) |

Expand Down
1 change: 1 addition & 0 deletions doc/user_guide/login-vm-and-ami.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
## Login to AMI and VM Editions

<!-- I assume the login user for AMI and VM Editions is still ubuntu. -->
Username: **ubuntu**

At the first login to the AI-Lab (image or AMI) you will be prompted to change your password.
Expand Down
7 changes: 6 additions & 1 deletion exasol/ds/sandbox/lib/dss_docker/create_image.py
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,10 @@ def _docker_file(self) -> importlib_resources.abc.Traversable:
def _start_container(self) -> DockerContainer:
self._start = datetime.now()
docker_client = docker.from_env()
try:
return docker_client.containers.get(self.container_name)
except:
pass
Comment on lines +117 to +120
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added these lines to speed up testing Ansible tasks

docker_file = self._docker_file()
_logger.info(f"Creating docker image {self.image_name} from {docker_file}")
if self.registry is not None:
Expand Down Expand Up @@ -157,6 +161,7 @@ def _commit_container(
notebook_folder_initial = get_fact(facts, "notebook_folder", "initial")
conf = {
"Entrypoint": entrypoint(facts),
"User": get_fact(facts, "jupyter", "user"),
"Cmd": [],
"Volumes": {notebook_folder_final: {}, },
"ExposedPorts": {f"{port}/tcp": {}},
Expand All @@ -174,7 +179,7 @@ def _cleanup(self, container: DockerContainer):
if container is None:
return
if self.keep_container:
_logger.info("Keeping container running")
_logger.info(f"Keeping container {self.container_name} running")
return
_logger.info("Stopping container")
container.stop()
Expand Down
31 changes: 24 additions & 7 deletions exasol/ds/sandbox/runtime/ansible/ai_lab_docker_playbook.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,22 +4,39 @@
vars:
ansible_python_interpreter: python3
tasks:
- name: Add docker container to inventory
- name: Add Docker Container to Ansible Inventory
add_host:
name: "{{docker_container}}"
groups: docker_container_group
ansible_connection: docker

- name: Prepare Ansible Access
hosts: docker_container_group
gather_facts: true
vars:
my_ansible_user: "ubuntu"
jupyter_user: "jupyter"
jupyter_group: "jupyter"
jupyter_user_home: "/home/jupyter"
need_sudo: no
tasks:
- import_tasks: apt_update.yml
- name: Ansible Access
include_role:
name: ansible_access

- name: Setup AI-Lab Docker Container
hosts: docker_container_group
remote_user: "ubuntu"
gather_facts: true
vars:
ansible_python_interpreter: python3
user_name: "{{ ansible_user_id }}"
user_home: "{{ ansible_env.HOME }}"
initial_notebook_folder: "{{ user_home }}/notebook-defaults"
need_sudo: false
docker_integration_test: true
ansible_python_interpreter: python3
user_name: "jupyter"
user_group: "jupyter"
user_home: "/home/jupyter"
initial_notebook_folder: "{{ user_home }}/notebook-defaults"
need_sudo: yes
docker_integration_test: true
tasks:
- import_tasks: general_setup_tasks.yml
- import_tasks: cleanup_tasks.yml
5 changes: 5 additions & 0 deletions exasol/ds/sandbox/runtime/ansible/apt_update.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
- name: Update and Upgrade apt Packages
apt:
upgrade: yes
update_cache: yes
become: "{{need_sudo}}"
4 changes: 4 additions & 0 deletions exasol/ds/sandbox/runtime/ansible/create_jupyter_user.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
- name: Add user 'jupyter' for running jupyter
ansible.builtin.user:
name: "{{ user_name }}"
home: "{{ user_home }}"
6 changes: 4 additions & 2 deletions exasol/ds/sandbox/runtime/ansible/ec2_playbook.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,14 @@
hosts: ec2
vars:
ansible_python_interpreter: /usr/bin/python3
user_name: ubuntu
user_home: /home/ubuntu
user_name: jupyter
user_home: /home/jupyter
user_group: jupyter
initial_notebook_folder: "{{ user_home }}/notebooks"
need_sudo: yes
remote_user: ubuntu
tasks:
- import_tasks: apt_update.yml
- import_tasks: general_setup_tasks.yml
- import_tasks: ec2_setup_tasks.yml
- import_tasks: cleanup_tasks.yml
17 changes: 12 additions & 5 deletions exasol/ds/sandbox/runtime/ansible/general_setup_tasks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,14 +15,12 @@
initial: "{{ initial_notebook_folder }}"
final: "{{user_home}}/notebooks"
cacheable: yes
- name: Update and upgrade apt packages
apt:
upgrade: yes
update_cache: yes
become: "{{need_sudo}}"
- name: Install rsync
include_role:
name: rsync
- name: Create Group and User for Running Jupyter Server
include_role:
name: jupyter_user
- name: Copy Entry Point Script
include_role:
name: entrypoint
Expand All @@ -36,7 +34,16 @@
jupyterlab_password: "{{ dss_facts.jupyter.password }}"
jupyterlab_notebook_folder_initial: "{{ dss_facts.notebook_folder.initial }}"
jupyterlab_notebook_folder: "{{ dss_facts.notebook_folder.final }}"
- name: Change Owner of All Files and Dirs in Home Directory
ansible.builtin.file:
path: "{{ user_home }}"
owner: "{{ user_name }}"
group: "{{ user_group }}"
recurse: true
become: "{{ need_sudo }}"
- name: Clear pip cache
# TBD: do we need become: "{{ need_sudo }}" here?
# should folder /root be replaced by {{ user_home }} ?
ckunki marked this conversation as resolved.
Show resolved Hide resolved
ansible.builtin.file:
path: /root/.cache/pip
state: absent
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
---

apt_dependencies:
- sudo=1.8.31-1ubuntu1.5
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
- name: Install sudo
apt:
name: "{{apt_dependencies}}"
state: latest
install_recommends: false

- name: Add dedicated user for executing ansible tasks
ansible.builtin.user:
name: "{{ my_ansible_user }}"
- name: Add to sudoers
community.general.sudoers:
name: Add user ansible
user: "{{ my_ansible_user }}"
state: present
nopassword: true
commands: ALL
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,4 @@
regexp: ^[^#].*core
line: '* hard core 0'
create: true
become: "{{ need_sudo }}"
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,6 @@

apt_dependencies:
- apt-transport-https=2.0.10
- gpg-agent=2.2.19-3ubuntu2.2
- ca-certificates=20230311ubuntu0.20.04.1
- software-properties-common=0.99.9.12
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
name: "{{apt_dependencies}}"
state: latest
update_cache: true
install_recommends: false
become: "{{need_sudo}}"

- name: Add Docker GPG apt Key
Expand All @@ -22,11 +23,12 @@
name: docker-ce
state: latest
update_cache: true
install_recommends: false
become: "{{need_sudo}}"

- name: Adding docker users (for use without sudo)
user:
name: "{{user_name}}"
append: yes
groups: docker
become: "{{need_sudo}}"
become: "{{need_sudo}}"
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,6 @@
src: "entrypoint.py"
dest: "{{ dss_facts.entrypoint }}"
mode: 0644
group: "{{ user_group }}"
owner: "{{ user_name }}"
become: "{{need_sudo}}"
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

jupyterlab_ip: '*'
jupyterlab_config: "{{user_home}}/.jupyter/jupyter_lab_config.py"
jupyterlab_config_json: "{{user_home}}/.jupyter/jupyter_server_config.json"
jupyterlab_default_url: "/lab/tree/start.ipynb"

# overridden in include_role
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ numpy==1.23.1
pandas==1.4.3
scikit-learn==1.0.2
matplotlib==3.7.4
jupysql==0.10.7
jupysql==0.10.10
sqlalchemy_exasol==4.6.3
stopwatch.py==2.0.1
boto3==1.26.163
Expand Down

This file was deleted.

Loading
Loading