Doc/#203 updated documentation (#208)

* Updated user guide
* Updated changes file
* Based on feedback from PM
* Added screenshot
* Use docker_client.login instead parameter auth_config which doesn't exist for docker_client.build

Co-authored-by: Mikhail Beck <[email protected]>
Co-authored-by: Torsten Kilias <[email protected]>

README.md

@@ -1,16 +1,20 @@
 # Exasol AI-Lab
 
-## Overview
+The Exasol AI-Lab is a pre-configured container designed to empower data scientists. It streamlines common data science and AI tasks, including data loading, preparation, exploration, model training, and deployment. Whether you’re a seasoned practitioner or just getting started, the AI-Lab provides a hassle-free experience.
 
-This project provides an automated mechanism to build and export virtual machines images (AWS AMI, VMDK,...)
-enabling users to try out data science algorithms in Jupyter notebooks connected to the Exasol database.
+![Transformers Extension](doc/user_guide/ai-lab-screenshot.png)
 
-## Where to find the VM images
+Key Features:
+* Jupyter Notebook Environment: The heart of the AI-Lab is a robust Jupyter Notebook environment. It’s where you’ll work on your data science projects.
+* Exasol Integration: Leverage Exasol’s power for your AI and machine learning use cases. The AI-Lab includes essential Exasol packages, extensions, and configuration tasks.
+* Example Notebooks: Jumpstart your work with ready-to-use example notebooks. Explore classic machine learning scenarios (think scikit-learn), seamlessly integrate Exasol with AWS Sagemaker, and tap into Huggingface models directly within Exasol.
 
-The release process will automatically store the links to the images in the [release notes](https://github.com/exasol/ai-lab/releases/latest), as there will be a specific AMI per release.
-Please check the user guide about details of the image.
+Feel free to explore the Exasol AI-Lab and unleash your data science potential!
 
-## Links
+The AI-Lab is available in multiple [Editions](doc/user_guide/editions.md) involving different technology stacks, see also common [System Requirements](doc/user_guide/system-requirements.md).
+
+After downloading the required files and having started the AI-Lab you can connect to AI-Lab's [Jupyter Service](doc/user_guide/jupyter.md).
+
+## Additional Links
 
-* [User Guide](doc/user_guide/user_guide.md)
 * [Developer Guide](doc/developer_guide/developer_guide.md)

doc/changes/changelog.md

@@ -1,5 +1,5 @@
 # Changes
 
-* [0.2.1](changes_0.2.1.md)
+* [1.0.0](changes_1.0.0.md)
 * [0.2.0](changes_0.2.0.md)
 * [0.1.0](changes_0.1.0.md)

doc/changes/changes_0.2.1.md → doc/changes/changes_1.0.0.md

@@ -1,6 +1,6 @@
-# AI-Lab 0.2.1, released 2024-??-??
+# AI-Lab 1.0.0 released 2024-02-20
 
-Code name: Fix Cloud Storage Notebook
+Code name: First Official Release
 
 ## Summary
 
@@ -11,6 +11,8 @@ Impact and delimitation
 * Also AWS codebuild image need to be upgraded from `aws/codebuild/standard:5.0` to `6.0`.
 * The Jupyterlab notebooks and their libraries remain on Python `3.8` for now.
 
+Additionally the release updates the User Guide.
+
 ## AI-Lab-Release
 
 Version: 0.2.1
@@ -41,7 +43,7 @@ n/a
 
 ## Documentation
 
-n/a
+* #203: Updated User Guide
 
 ## Refactoring
 

doc/developer_guide/testing.md

@@ -11,7 +11,7 @@ docker image in the tests in `integration/test_create_dss_docker_image.py`
 simply add CLI option `--dss-docker-image` when calling `pytest`:
 
 ```shell  
-poetry run pytest --dss-docker-image exasol/ai-lab:0.2.0 
+poetry run pytest --dss-docker-image exasol/ai-lab:1.0.0 
 ```
 
 #### Executing tests involving AWS resources

doc/user_guide/ami_usage.md → doc/user_guide/ami-usage.md

@@ -1,6 +1,12 @@
-# Using the Amazon Machine Image (AMI Edition)
+# AI-Lab AMI Edition (Amazon Machine Image)
 
-__Important__: The AMI is currently only available in the AWS region `eu-central-1`. If you want to use the image in another region, you need to copy it before. Check the [AWS documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/CopyingAMIs.html) for details about how to copy the image.
+The ID of the AMI (Amazon Machine Image) is mentioned in the [release notes](https://github.com/exasol/ai-lab/releases/latest) and can be used to start an EC2-instance in your AWS account.
+
+The naming scheme is: "_Exasol-AI-Lab-${VERSION}_", e.g. "_Exasol-AI-Lab-5.0.0_"
+
+The AMI is currently only available in the AWS region `eu-central-1`.
+
+Check the [AWS documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/CopyingAMIs.html) for details about how to copy the image.
 
 ## Overview
 
@@ -19,15 +25,19 @@ __Important__: The AMI is currently only available in the AWS region `eu-central
     - Create  inbound rules:
       - One of type `ssh`
       - If you plan to connect to the Jupyter lab, add another rule of type "Custom TCP" with port 49494.
-        - **Important**: With this rule you expose the Jupyter lab to the internet; anybody who has access to the password will be able to execute commands. For a minimum of security you should change the default Jupyter password. Details about how to do that will be shown in the login screen when you login via ssh. However, you should consider to use only `ssh` with port forwarding.
+        - **Important**: With this rule you expose the Jupyter lab to the internet; anybody who has access to the password will be able to execute commands. For a minimum of security you should change the default Jupyter password. Details about how to do that will be shown in the login screen when you log in via ssh. However, you should consider to use only `ssh` with port forwarding.
  4. Go back to the EC2 console
  5. Launch the EC2 instance:
      - In the navigation bar on the left select "Instances"
      - Click button "Launch instances"
-     - At field "Application and OS Images" select the AMI id of the sandbox (found in the [release notes](https://github.com/exasol/ai-lab/releases/latest))
+     - At field "Application and OS Images" select the AMI ID of the sandbox (found in the [release notes](https://github.com/exasol/ai-lab/releases/latest))
      - Select an appropriate instance type (at least "t2.small" or similar)
      - Choose your key pair
-     - Choose the security group which your created in step 3.
+     - Choose the security group which you created in step 3.
      - For the storage we recommend to keep the pre-selected 100GB volume
      - Click button "Launch instance"
 6. As soon as the machine becomes available you can connect per ssh with user `ubuntu`: `ssh -i your_key.pem ubuntu@the_new_ec_instance`
+
+## Login
+
+See [Login to AMI and VM Editions](login-vm-and-ami.md) for logging in to the system.

doc/user_guide/docker/docker-usage.md

@@ -0,0 +1,162 @@
+# AI-Lab Docker Edition
+
+Using Exasol AI-Lab Docker Edition requires some specific [prerequisites](prerequisites.md) but also offers additional benefits.
+
+The [Operating System and Setup Guide](prerequisites.md#operating-systems-and-setups) helps you with the initial system setup.
+
+AI-Lab also offers a [short introduction](intro.md) to Docker Images and Containers if you are new to this technology.
+
+## Defining Environment Variables
+
+The Unix shell commands in the following sections will use some environment variables. By this you can adapt the commands to your specific preferences while still being able to execute them literally:
+* Variable `VERSION` refers to the version of Exasol AI-Lab Docker Edition you want to use, alternatively you can use `latest`.
+* Variable `VOLUME` is expected to contain the name of your Docker volume, see [Managing User Data](managing-user-data.md).
+* Variable `LISTEN_IP` defines the range of IP-addresses allowed to connect to the forwarded Jupyter port.
+  * `0.0.0.0` means all IP-addresses are allowed.
+  * For local setups, we recommend `127.0.0.1`.
+  * Please contact your IT department if there are security restrictions.
+* Variable `CONTAINER_NAME` defines the name for the AI-Lab Docker container. Giving the running container a name makes it easier to refer to it when [stopping](#stopping-the-ai-lab-docker-container) or [restarting](#restarting-a-stopped-container) it.
+
+Here is an example:
+
+```shell
+VERSION=1.0.0
+LISTEN_IP=0.0.0.0
+VOLUME=my-vol
+CONTAINER_NAME=ai-lab
+```
+
+## Creating a Docker Container for the AI- Lab from the AI-Lab Docker Image
+
+You can use an Exasol database with AI-Lab in two ways
+* [External Exasol database](#ai-lab-with-external-exasol-database-only)
+* Integrated [Exasol Docker-DB](#ai-lab-with-integrated-exasol-docker-db) managed by AI-Lab
+
+### AI-Lab with External Exasol Database Only
+
+In this scenario the AI-Lab Docker container does not need access to the Docker daemon.
+
+The following command will
+* Download the Docker image for the specified version `$VERSION` of the AI-Lab if the image of the specified version is not yet available in your Docker service
+* Run a Docker container using this image
+* Mount the volume `$VOLUME` to the directory `/root/notebooks` inside the container
+  * If the volume does not exist yet, then it will be created automatically.
+* Forward port `49494` on the [daemon machine](prerequisites.md) to allow connections from all IP addresses matched by `$LISTEN_IP`
+
+```shell
+docker run \
+  --name ${CONTAINER_NAME} \
+  --volume ${VOLUME}:/root/notebooks \
+  --publish ${LISTEN_IP}:49494:49494 \
+  exasol/ai-lab:${VERSION}
+```
+
+If you want to use a newer version of the AI-Lab then please [delete the Docker volumes](managing-user-data.md#replacing-the-docker-volume) created with older versions.
+
+Additional options
+* Add option `--detach` to run the container in the background but please note that the initial welcome message with instructions will be hidden then, see also Command [`docker logs`](https://docs.docker.com/engine/reference/commandline/container_logs/) and section [Stopping the AI-Lab Docker Container](#stopping-the-ai-lab-docker-container).
+* If port `49494` is not available on your daemon machine you can forward port `49494` of the Jupyter server in the Docker container to another port, e.g. `55555`, on the daemon machine with `--publish ${LISTEN_IP}:55555:49494`
+
+### AI-Lab with Integrated Exasol Docker-DB
+
+In this scenario you must enable the AI-Lab Docker container to access the Docker daemon.
+
+**Please note**
+* Additional [Limitations and security risks](prerequisites.md#enabling-exasol-ai-lab-to-use-docker-features) apply.
+* Only file system objects on the daemon machine can be mounted. This applies to ordinary directories as well as the `docker.sock`.
+* On Windows mounting `docker.sock` only works with Docker Desktop with WSL 2.
+```shell
+docker run \
+  --name ${CONTAINER_NAME} \
+  --volume ${VOLUME}:/root/notebooks \
+  --volume /var/run/docker.sock:/var/run/docker.sock \
+  --publish ${LISTEN_IP}:49494:49494 \
+  exasol/ai-lab:${VERSION}
+```
+
+## Stopping the AI-Lab Docker Container
+
+If you used one of the commands given in preceding sections without option `--detach` then you can stop the AI-Lab Docker container by simply pressing Ctrl-C.
+
+If you used option `--detach` then you need to use the following command:
+
+```shell
+docker stop ${CONTAINER_NAME}
+```
+
+See also https://docs.docker.com/engine/reference/commandline/container_stop/.
+
+## Restarting a Stopped Container
+
+In general we recommend to restart a stopped container instead of creating a new one. This has the benefit of keeping [additional dependencies](#installing-additional-dependencies) that you did install:
+
+```shell
+docker start ${CONTAINER_NAME}
+```
+
+**Please note**
+* If you accidently created a new AI-Lab Docker container and the stopped container still exists please remove the new container and restart the existing one.
+* If there is no stopped AI-Lab container and you are using Exasol Docker-DB you need to link the newly created container to the Exasol Docker-DB network `db_network_DemoDb`.
+
+```shell
+docker network connect db_network_DemoDb <CONTAINER>
+```
+
+See also https://docs.docker.com/engine/reference/commandline/network_connect/.
+
+## Connecting to the Jupyter Service
+
+When starting AI-Lab as a Docker container the command line will display a welcome message showing connection instructions and a reminder to change the default password:
+
+```
+$ docker run --publish 0.0.0.0:$PORT:49494 exasol/ai-lab:$VERSION
+Server for Jupyter has been started successfully.
+
+You can connect with http://<host>:<port>
+
+If using a Docker daemon on your local machine and did forward the
+port to the same port then you can connect with http://localhost:49494.
+
+┬ ┬┌─┐┌┬┐┌─┐┌┬┐┌─┐  ┬ ┬┌─┐┬ ┬┬─┐   ┬┬ ┬┌─┐┬ ┬┌┬┐┌─┐┬─┐  ┌─┐┌─┐┌─┐┌─┐┬ ┬┌─┐┬─┐┌┬┐ ┬
+│ │├─┘ ││├─┤ │ ├┤   └┬┘│ ││ │├┬┘   ││ │├─┘└┬┘ │ ├┤ ├┬┘  ├─┘├─┤└─┐└─┐││││ │├┬┘ ││ │
+└─┘┴  ─┴┘┴ ┴ ┴ └─┘   ┴ └─┘└─┘┴└─  └┘└─┘┴   ┴  ┴ └─┘┴└─  ┴  ┴ ┴└─┘└─┘└┴┘└─┘┴└──┴┘ o
+
+The default password is "ai-lab".
+To update the password, log in to the Docker container as the user root and run
+    /root/jupyterenv/bin/jupyter-lab server password
+```
+
+Using an internet browser you then can connect to the Jupyter server running in the Docker container in order to follow the tutorials presented by a set of Jupyter notebooks, see [Connecting to Jupyter Service](../jupyter.md#open-jupyter-in-your-browser).
+
+For the parameter `<host>`: If your daemon machine is identical to the machine your browser is running on then you can replace `<host>` by `localhost` otherwise please use the IP address of the daemon machine.
+
+The following section explains how to log in to the Docker container to change settings, such as the default password.
+
+## Logging in to the Docker container
+
+To update the password you must log in to the Docker container.
+
+First, you need to find out the container's ID. The following command shows the list of currently running Docker containers.
+
+```shell
+docker ps
+```
+
+Here is a sample output
+
+```
+CONTAINER ID   IMAGE     COMMAND        NAMES
+1199447716d4   image:2   "entrypoint"   funny_rabbit
+```
+
+The following command enables you to log in as the user `root` to the specified container:
+
+```shell
+docker exec --user root -it ${CONTAINER_NAME} bash
+```
+
+## Installing Additional Dependencies
+
+See [User Guide](../jupyter.md#installing-additional-dependencies).
+
+Please note: Removing the docker container `docker rm <container>` will discard all dependencies that have been installed additionally.