#308, #298, #318. #322: Updated and cleanup dependencies, re-enable …

…tests, move ibis notebook to new download URL, (#319)

* #308: Updated Dependencies of Jupyter Environment and pyproject.toml
* #298: Fixed vulnerabilities 
* #322: Made URL in IBIS notebook point to a new example data bucket
* #318: Re-enabled tests for cloud storage notebook

* Updated version constraint for stopwatch
* Replaced GitHub dependency in file test_dependencies.txt for notebook tests by PyPI dependency.
* Fixed CloudFormation lint tests
* Fixed name of docker registry in integration test to satisfy legal names for docker registries.
* Updated changes file
* Changed version to 3.0.0
* Fixed cloud storage notebook test
* Fixed cloud_store_config.ipynb temporarily by correctly reimplement bfs_utils.put_file


---------

Co-authored-by: Torsten Kilias <[email protected]>

.github/workflows/notebook_tests.yaml

@@ -11,6 +11,7 @@ jobs:
       fail-fast: false
       matrix:
         nb_test:
+          - { name: "CSE notebook", test_file: "nbtest_cloud.py" }
           - { name: "sklearn notebook", test_file: "nbtest_sklearn.py" }
           - { name: "ibis notebook", test_file: "nbtest_ibis.py" }
           - { name: "SLC notebook", test_file: "nbtest_script_languages_container.py" }

doc/changes/changelog.md

@@ -1,6 +1,6 @@
 # Changes
 
-* [2.1.0](changes_2.1.0.md)
+* [3.0.0](changes_3.0.0.md)
 * [2.0.0](changes_2.0.0.md)
 * [1.0.0](changes_1.0.0.md)
 * [0.2.0](changes_0.2.0.md)

doc/changes/changes_3.0.0.md

@@ -0,0 +1,87 @@
+# AI-Lab 3.0.0 released 2024-08-26
+
+Code name: More Notebooks, TE-Update, Exasol SaaS, Python 3.10
+
+## Summary
+
+This release adds Jupyter tutorials for using the [IBIS dataframe library](https://ibis-project.org/) and for building [Exasol Script Language Containers (SLC)](https://docs.exasol.com/db/latest/database_concepts/udf_scripts/adding_new_packages_script_languages.htm#ScriptLanguageContainer).
+
+Transitive dependency `transformers-extension` (TE) via `notebook-connector` has been updated to [TE version 2.0.0](https://github.com/exasol/transformers-extension/releases/2.0.0). This fixes an error in saving and loading of the model but required a breaking change regarding the storage format of the TE model.
+
+**Users of the AI-Lab must redeploy TE into their database and upload the models once again.**
+
+The release adds support for parameters for SaaS instances of Exasol database to the configuration page and fixes vulnerability `CVE-2024-23342` by updating dependencies and also updates the operating system from ubuntu 20.04 to 22.04 and Python version to 3.10 in the published images for Docker, AMI, and virtual machines.
+
+Additionally, this release fixes the following vulnerabilities by updating dependencies:
+* Vulnerability CVE-2024-23342 in transitive dependency via `localstack` to `ecdsa` vulnerably to Minerva timing attack on P-256 in `python-ecdsa`.
+* Vulnerability CVE-2024-5206 in dependency `scikit-learn` versions below `1.5.0` caused by sensitive data leakage.
+* Vulnerability CVE-2024-35195 in dependency `requests` in versions below `2.32.0` caused by requests `Session` object not verifying requests after making first request with `verify=False`.
+* Vulnerability CVE-2024-37891 in transitive dependency via `boto3` to `urllib3` in versions below `2.2.2` caused by proxy-authorization request header not to be stripped during cross-origin redirects.
+
+The release ignores vulnerability CVE-2024-33663 in transitive dependency via `localstack` to `python-jose` `3.3.0` caused by algorithm confusion with OpenSSH ECDSA keys as there is no newer version of `python-jose` available and the dependency only affects tests.
+
+## Features
+
+* #277: Added the SaaS database parameters to the configuration page.
+* #279: Made the notebooks tests running in SaaS as well as in the Docker-DB.
+* #19: Added SLC notebook
+* #301: Added CloudFront distribution for example data S3 bucket
+* #273: Added `jupyterenv/bin` to environment variable `PATH` for running Jupyter Server
+
+## Security
+
+* #207: Fixed vulnerability CVE-2024-23342 by updating dependency ecdsa
+* #298: Fixed vulnerabilities by updating dependencies
+
+## Bug Fixes
+
+* #303: Fixed AWS Codebuild
+* #313: Fixed differing versions of dependency `scikit-learn`
+
+## Documentation
+
+* #249: Added a troubleshooting section to the user guide documenting died kernel for transformers/te_init.ipynb.
+* #284: Improved User Guide regarding Docker volumes
+* #322: Made URL in IBIS notebook point to new example data bucket https://dut5tonqye28.cloudfront.net/ai_lab/flight-info
+
+## Refactoring
+
+* #267: Switched CodeBuildWaiter to use tenacity
+* #276: Started using the new ITDE Manager interface in the notebook-connector 0.2.9
+* #282: Updated python version to Python 3.10
+* #295: Made notebook-tests mandatory for merge
+* #193: Ignored warnings in notebook tests
+* #297: Reduced log level for transitive libraries in notebook tests
+* #307: Made the notebook tests running in parallel; moved common steps from test jobs to a composite action
+* #308: Removed redundant dependencies from file `notebook_requirements.txt`.
+* #318: Re-enabled disabled notebook tests
+
+## Dependency Updates
+
+### Jupyter Environment Dependencies
+
+In File [notebook_requirements.txt](https://github.com/exasol/ai-lab/blob/main/exasol/ds/sandbox/runtime/ansible/roles/jupyter/files/notebook_requirements.txt):
+* Removed dependency to `uncertainties`
+* Updated `scikit-learn:1.0.2` to `1.5.1`
+* Updated `matplotlib:3.7.4` to `3.9.2`
+* Updated `jupysql:0.10.10` to `0.10.12`
+* Relaxed declaration of dependency `stopwatch.py:2.0.1` to `2.*` to avoid inconsistencies with ITDE
+* Updated `exasol-notebook-connector:0.2.8` to `0.2.9`
+* Updated `ipywidgets:8.1.1` to `8.1.3`
+
+### Dependencies in `ai-lab/pyproject.toml`
+
+* Updated `boto3:1.34.144` to `1.35.2`
+* Updated `ansible:9.8.0` to `10.3.0`
+* Updated `rich:12.6.0` to `13.7.1`
+* Updated `pandas:1.5.3` to `2.2.2`
+* Updated `tenacity:8.5.0` to `9.0.0`
+* Updated `importlib-metadata:7.2.1` to `8.4.0`
+* Updated `pytest-check-links:0.9.3` to `0.10.1`
+* Updated `pytest:7.4.4` to `8.3.2`
+* Updated `cfn-lint:0.65.1` to `1.10.3`
+* Updated `localstack:3.5.0` to `3.6.0`
+* Updated `docker:6.1.3` to `7.1.0`
+* Updated `fabric:2.7.1` to `3.2.2`
+* Updated `requests:2.31.0` to `2.32.3`
+* Removed explicit declaration of dependency `numpy`

doc/developer_guide/testing.md

@@ -19,7 +19,7 @@ poetry run exasol/ds/sandbox/main.py \
 To use an existing docker image in the tests in `integration/test_create_dss_docker_image.py`, simply add the CLI option `--dss-docker-image` when calling `pytest`:
 
 ```shell
-poetry run pytest --dss-docker-image exasol/ai-lab:2.1.0
+poetry run pytest --dss-docker-image exasol/ai-lab:3.0.0
 ```
 
 ## Tests for Jupyter Notebooks

doc/user_guide/docker/docker-usage.md

@@ -21,7 +21,7 @@ The Unix shell commands in the following sections will use some environment vari
 Here is an example:
 
 ```shell
-VERSION=2.1.0
+VERSION=3.0.0
 LISTEN_IP=0.0.0.0
 VOLUME=my-vol
 CONTAINER_NAME=ai-lab

doc/user_guide/vm-edition/win-vbox.md

@@ -9,7 +9,7 @@
 ## Select Virtual machine Name and Operating System
 
 * Create a new virtual machine
-* Enter a name for your virtual machine, e.g. "Exasol-AI-Lab-2.1.0"
+* Enter a name for your virtual machine, e.g. "Exasol-AI-Lab-3.0.0"
 * Select a folder to store the VM image to
 * Select operating system "Linux", e.g. version "Ubuntu 22.04"
 * Click button "Next"

exasol/ds/sandbox/runtime/ansible/roles/jupyter/files/jupyter_requirements.txt

@@ -1,5 +1,2 @@
 jupyterlab==4.1.1
-# enable interactive Javascript widgets in the notebooks
-ipywidgets==8.1.1
 pexpect==4.8.0
-pyexasol==0.16.1