From 9f466305e64f2de3a0518f1737f102ead4036e41 Mon Sep 17 00:00:00 2001 From: mibe Date: Tue, 15 Oct 2024 10:03:16 +0100 Subject: [PATCH 1/2] #144 updated the user guide, prepared the release --- doc/changes/changes_0.11.0.md | 5 +- doc/user_guide/user_guide.md | 110 +++++------------- exasol_sagemaker_extension/deploy.py | 4 - poetry.lock | 105 ++++++++--------- pyproject.toml | 2 +- .../deployment/test_deploy_cli.py | 11 +- 6 files changed, 92 insertions(+), 145 deletions(-) diff --git a/doc/changes/changes_0.11.0.md b/doc/changes/changes_0.11.0.md index 160d0dd..b1507fe 100644 --- a/doc/changes/changes_0.11.0.md +++ b/doc/changes/changes_0.11.0.md @@ -1,12 +1,13 @@ # SageMaker Extension 0.11.0, released T.B.D. -Code name: T.B.D. +Simplified installation. ## Summary -T.B.D. +Using a single installation command. Using the pytest plugins for testing. ### Refactoring * #140: Used the pytest plugins for testing. * #142: Made a unified deployment CLI command. +* #144: Updated the installation section of the user guide. diff --git a/doc/user_guide/user_guide.md b/doc/user_guide/user_guide.md index 99d6fd6..d4d4d08 100644 --- a/doc/user_guide/user_guide.md +++ b/doc/user_guide/user_guide.md @@ -4,10 +4,9 @@ Exasol Sagemaker Extension provides a Python library together with Exasol Script and UDFs that train Machine Learning Models on data stored in Exasol using AWS SageMaker Autopilot service. -The extension basically exports a given Exasol table into AWS S3, and then triggers -Machine Learning training using the AWS Autopilot service with the specified parameters. -In addition, the training status can be polled using the auxiliary scripts provided -within the scope of the project. +The extension exports a given Exasol table into AWS S3, and then triggers Machine Learning training +using the AWS Autopilot service with the specified parameters. The training status can be polled using +the auxiliary scripts provided within the scope of the project. ## Table of Contents @@ -25,82 +24,39 @@ within the scope of the project. ## Installation -### Install The Built Archive -- Install the packaged sagemaker-extension project as follows (Please check [the latest release](https://github.com/exasol/sagemaker-extension/releases/latest)): -```buildoutcfg -pip install exasol_sagemaker_extension.whl -``` -### The Pre-built Language Container - -This extension requires the installation of a Language Container in the Exasol Database. -The Script Language Container is a way to install the required programming language and -necessary dependencies in the Exasol Database so the UDF scripts can be executed. - -The Language Container is downloaded and installed by executing the -deployment script below. Please make sure that the version of the Language Container matches the -installed version of the Sagemaker Extension Package. See [the latest release](https://github.com/exasol/sagemaker-extension/releases) on Github. - - ```buildoutcfg - python -m exasol_sagemaker_extension.deploy language-container - ``` -Please refer to the [Language Container Deployment Guide](https://github.com/exasol/python-extension-common/blob/main/doc/user_guide/user-guide.md#language-container-deployer) for details about this command. - -### Scripts Deployment - -Deploy all necessary scripts to the specified ```SCHEMA``` in Exasol using the following python cli command: +### Install the Python Package +The Sagemaker Extension package can be installed using pip: +```shell +pip install exasol-sagemaker-extension +``` -```buildoutcfg -python -m exasol_sagemaker_extension.deployment.deploy_cli +### Deploy the Extension to the Database +The Sagemaker Extension must be deployed to the database using the following command: +```shell +python -m exasol_sagemaker_extension.deploy ``` -The choice of options is primarily determined by the storage backend being used - On-Prem or SaaS. - -### List of options - -The table below lists all available options. It shows which ones are applicable for On-Prem and for SaaS backends. -Unless stated otherwise in the comments column, the option is required for either or both backends. - -Some of the values, like passwords, are considered confidential. For security reasons, it is recommended to store -those values in environment variables instead of providing them in the command line. The names of the environment -variables are given in the comments column, where applicable. Alternatively, it is possible to put just the name of -an option in the command line, without providing its value. In this case, the command will prompt to enter the value -interactively. For long values, such as the SaaS account id, it is more practical to copy/paste the value from -another source. - -| Option name | On-Prem | SaaS | Comment | -|:-----------------------------|:-------:|:----:|:-------------------------------------------------------| -| dsn | [x] | | i.e. | -| db-user | [x] | | | -| db-pass | [x] | | Env. [DB_PASSWORD] | -| saas-url | | [x] | Optional, Env. [SAAS_HOST] | -| saas-account-id | | [x] | Env. [SAAS_ACCOUNT_ID] | -| saas-database-id | | [x] | Optional, Env. [SAAS_DATABASE_ID] | -| saas-database-name | | [x] | Optional, provide if the database_id is unknown | -| saas-token | | [x] | Env. [SAAS_TOKEN] | -| schema | [x] | [x] | DB schema to deploy the scripts in | -| ssl-cert-path | [x] | [x] | Optional | -| [no_]use-ssl-cert-validation | [x] | [x] | Optional boolean, defaults to True | -| ssl-client-cert-path | [x] | | Optional | -| ssl-client-private-key | [x] | | Optional | -| develop | [x] | [x] | Optional, if True, causes re-generation of the scripts | -| verbose | [x] | [x] | Optional, if True produces verbose output | - -### TLS/SSL options - -The `--ssl-cert-path` is needed if the TLS/SSL certificate is not in the OS truststore. -Generally speaking, this certificate is a list of trusted CA. It is needed for the server's certificate -validation by the client. -The option `--use-ssl-cert-validation`is the default, it can be disabled with `--no-use-ssl-cert-validation`. -One needs to exercise caution when turning the certificate validation off as it potentially lowers the security of the -Database connection. -The "server" certificate described above shall not be confused with the client's own certificate. -In some cases, this certificate may be requested by a server. The client certificate may or may not include -the private key. In the latter case, the key may be provided as a separate file. - -### AWS Connection Object - - Create an Exasol connection object with AWS credentials that has -AWS Sagemaker Execution permission. The connection will encapsulate the address of the AWS S3 bucket where the exported data will be stored. +The deployment includes the installation of the Script Language Container (SLC) and several +scripts. The SLC is a way to install the required programming language and necessary dependencies +in the Exasol Database so that UDF scripts can be executed. The version of the installed SLC must +match the version of the Sagemaker Extension Package. See [the latest release](https://github.com/exasol/sagemaker-extension/releases) on Github. + +For information about the available options common to all Exasol extensions please refer to the +[documentation](https://github.com/exasol/python-extension-common/blob/0.8.0/doc/user_guide/user-guide.md) +in the Exasol Python Extension Common package. +In addition, this extension provides the following installation options: + +| Option name | Default | Comment | +|:--------------------|:-------:|:-----------------------------------------------| +| [no-]deploy-slc | True | Install SLC as part of the deployment | +| [no-]deploy-scripts | True | Install scripts as part of the deployment | +| [no-]to-print | False | Print SQL statements instead of executing them | +| [no-]develop | False | Re-generate the scripts | + +### Create AWS Connection Object +Create an Exasol connection object with AWS credentials that has AWS Sagemaker Execution permission. +The connection will encapsulate the address of the AWS S3 bucket where the exported data will be stored. For more information please check the [Create Connection in Exasol](https://docs.exasol.com/sql/create_connection.htm?Highlight=connection) document. Below is a template of the query that will create the required connection object. ```buildoutcfg @@ -110,8 +66,6 @@ Below is a template of the query that will create the required connection object IDENTIFIED BY '' ``` - - ## Execution of Training ### Execute Autopilot Training - Example usage of the AWS Sagemaker Autopilot service in Exasol is as follows. diff --git a/exasol_sagemaker_extension/deploy.py b/exasol_sagemaker_extension/deploy.py index 1556783..886732f 100644 --- a/exasol_sagemaker_extension/deploy.py +++ b/exasol_sagemaker_extension/deploy.py @@ -29,10 +29,6 @@ def deploy(deploy_slc: bool, deploy_scripts: bool, **kwargs): if deploy_slc: - # Workaround for the issue#78 in PEC - if StdParams.path_in_bucket.name in kwargs and kwargs[StdParams.path_in_bucket.name] is None: - kwargs[StdParams.path_in_bucket.name] = '' - slc_deployer = LanguageContainerDeployerCli( container_url_arg=CONTAINER_URL_ARG, container_name_arg=CONTAINER_NAME_ARG) diff --git a/poetry.lock b/poetry.lock index aa2a55e..c0902ed 100644 --- a/poetry.lock +++ b/poetry.lock @@ -13,13 +13,13 @@ files = [ [[package]] name = "anyio" -version = "4.6.0" +version = "4.6.2.post1" description = "High level compatibility layer for multiple asynchronous event loop implementations" optional = false python-versions = ">=3.9" files = [ - {file = "anyio-4.6.0-py3-none-any.whl", hash = "sha256:c7d2e9d63e31599eeb636c8c5c03a7e108d73b345f064f1c19fdc87b79036a9a"}, - {file = "anyio-4.6.0.tar.gz", hash = "sha256:137b4559cbb034c477165047febb6ff83f390fc3b20bf181c1fc0a728cb8beeb"}, + {file = "anyio-4.6.2.post1-py3-none-any.whl", hash = "sha256:6d170c36fba3bdd840c73d3868c1e777e33676a69c3a72cf0a0d5d6d8009b61d"}, + {file = "anyio-4.6.2.post1.tar.gz", hash = "sha256:4c8bc31ccdb51c7f7bd251f51c609e038d63e34219b44aa86e47576389880b4c"}, ] [package.dependencies] @@ -30,7 +30,7 @@ typing-extensions = {version = ">=4.1", markers = "python_version < \"3.11\""} [package.extras] doc = ["Sphinx (>=7.4,<8.0)", "packaging", "sphinx-autodoc-typehints (>=1.2.0)", "sphinx-rtd-theme"] -test = ["anyio[trio]", "coverage[toml] (>=7)", "exceptiongroup (>=1.2.0)", "hypothesis (>=4.0)", "psutil (>=5.9)", "pytest (>=7.0)", "pytest-mock (>=3.6.1)", "trustme", "uvloop (>=0.21.0b1)"] +test = ["anyio[trio]", "coverage[toml] (>=7)", "exceptiongroup (>=1.2.0)", "hypothesis (>=4.0)", "psutil (>=5.9)", "pytest (>=7.0)", "pytest-mock (>=3.6.1)", "trustme", "truststore (>=0.9.1)", "uvloop (>=0.21.0b1)"] trio = ["trio (>=0.26.1)"] [[package]] @@ -721,13 +721,13 @@ simplejson = ">=3.16.0" [[package]] name = "exasol-python-extension-common" -version = "0.7.0" +version = "0.8.0" description = "A collection of common utilities for Exasol extensions." optional = false python-versions = "<4.0.0,>=3.10.0" files = [ - {file = "exasol_python_extension_common-0.7.0-py3-none-any.whl", hash = "sha256:ee4b77263648ca97716ccb372c2257b61141d5ce35d75b30bda63e788c214801"}, - {file = "exasol_python_extension_common-0.7.0.tar.gz", hash = "sha256:7e6b10dc5593d63a1333e57d8d5b8ceb0f5eec632b8a31740f1a03fbc0c3eabc"}, + {file = "exasol_python_extension_common-0.8.0-py3-none-any.whl", hash = "sha256:109edc9d878b2cdb82b79255264206f0661e1f55850178f31d663bea6d8026f5"}, + {file = "exasol_python_extension_common-0.8.0.tar.gz", hash = "sha256:765a6601a880924c68a0d9306764752129b7144390ff1662a425a31df1c9083a"}, ] [package.dependencies] @@ -1410,20 +1410,21 @@ nicer-shell = ["ipython"] [[package]] name = "networkx" -version = "3.3" +version = "3.4.1" description = "Python package for creating and manipulating graphs and networks" optional = false python-versions = ">=3.10" files = [ - {file = "networkx-3.3-py3-none-any.whl", hash = "sha256:28575580c6ebdaf4505b22c6256a2b9de86b316dc63ba9e93abde3d78dfdbcf2"}, - {file = "networkx-3.3.tar.gz", hash = "sha256:0c127d8b2f4865f59ae9cb8aafcd60b5c70f3241ebd66f7defad7c4ab90126c9"}, + {file = "networkx-3.4.1-py3-none-any.whl", hash = "sha256:e30a87b48c9a6a7cc220e732bffefaee585bdb166d13377734446ce1a0620eed"}, + {file = "networkx-3.4.1.tar.gz", hash = "sha256:f9df45e85b78f5bd010993e897b4f1fdb242c11e015b101bd951e5c0e29982d8"}, ] [package.extras] -default = ["matplotlib (>=3.6)", "numpy (>=1.23)", "pandas (>=1.4)", "scipy (>=1.9,!=1.11.0,!=1.11.1)"] +default = ["matplotlib (>=3.7)", "numpy (>=1.24)", "pandas (>=2.0)", "scipy (>=1.10,!=1.11.0,!=1.11.1)"] developer = ["changelist (==0.5)", "mypy (>=1.1)", "pre-commit (>=3.2)", "rtoml"] -doc = ["myst-nb (>=1.0)", "numpydoc (>=1.7)", "pillow (>=9.4)", "pydata-sphinx-theme (>=0.14)", "sphinx (>=7)", "sphinx-gallery (>=0.14)", "texext (>=0.6.7)"] -extra = ["lxml (>=4.6)", "pydot (>=2.0)", "pygraphviz (>=1.12)", "sympy (>=1.10)"] +doc = ["intersphinx-registry", "myst-nb (>=1.1)", "numpydoc (>=1.8.0)", "pillow (>=9.4)", "pydata-sphinx-theme (>=0.15)", "sphinx (>=7.3)", "sphinx-gallery (>=0.16)", "texext (>=0.6.7)"] +example = ["cairocffi (>=1.7)", "contextily (>=1.6)", "igraph (>=0.11)", "momepy (>=0.7.2)", "osmnx (>=1.9)", "scikit-learn (>=1.5)", "seaborn (>=0.13)"] +extra = ["lxml (>=4.6)", "pydot (>=3.0.1)", "pygraphviz (>=1.14)", "sympy (>=1.10)"] test = ["pytest (>=7.2)", "pytest-cov (>=4.0)"] [[package]] @@ -1897,13 +1898,13 @@ test = ["coverage", "mypy", "ruff", "wheel"] [[package]] name = "pyparsing" -version = "3.1.4" +version = "3.2.0" description = "pyparsing module - Classes and methods to define and execute parsing grammars" optional = false -python-versions = ">=3.6.8" +python-versions = ">=3.9" files = [ - {file = "pyparsing-3.1.4-py3-none-any.whl", hash = "sha256:a6a7ee4235a3f944aa1fa2249307708f893fe5717dc603503c6c7969c070fb7c"}, - {file = "pyparsing-3.1.4.tar.gz", hash = "sha256:f86ec8d1a83f11977c9a6ea7598e8c27fc5cddfa5b07ea2241edbbde1d7bc032"}, + {file = "pyparsing-3.2.0-py3-none-any.whl", hash = "sha256:93d9577b88da0bbea8cc8334ee8b918ed014968fd2ec383e868fb8afb1ccef84"}, + {file = "pyparsing-3.2.0.tar.gz", hash = "sha256:cbf74e27246d595d9a74b186b810f6fbb86726dbf3b9532efb343f6d7294fe9c"}, ] [package.extras] @@ -2107,29 +2108,29 @@ files = [ [[package]] name = "pywin32" -version = "307" +version = "308" description = "Python for Window Extensions" optional = false python-versions = "*" files = [ - {file = "pywin32-307-cp310-cp310-win32.whl", hash = "sha256:f8f25d893c1e1ce2d685ef6d0a481e87c6f510d0f3f117932781f412e0eba31b"}, - {file = "pywin32-307-cp310-cp310-win_amd64.whl", hash = "sha256:36e650c5e5e6b29b5d317385b02d20803ddbac5d1031e1f88d20d76676dd103d"}, - {file = "pywin32-307-cp310-cp310-win_arm64.whl", hash = "sha256:0c12d61e0274e0c62acee79e3e503c312426ddd0e8d4899c626cddc1cafe0ff4"}, - {file = "pywin32-307-cp311-cp311-win32.whl", hash = "sha256:fec5d27cc893178fab299de911b8e4d12c5954e1baf83e8a664311e56a272b75"}, - {file = "pywin32-307-cp311-cp311-win_amd64.whl", hash = "sha256:987a86971753ed7fdd52a7fb5747aba955b2c7fbbc3d8b76ec850358c1cc28c3"}, - {file = "pywin32-307-cp311-cp311-win_arm64.whl", hash = "sha256:fd436897c186a2e693cd0437386ed79f989f4d13d6f353f8787ecbb0ae719398"}, - {file = "pywin32-307-cp312-cp312-win32.whl", hash = "sha256:07649ec6b01712f36debf39fc94f3d696a46579e852f60157a729ac039df0815"}, - {file = "pywin32-307-cp312-cp312-win_amd64.whl", hash = "sha256:00d047992bb5dcf79f8b9b7c81f72e0130f9fe4b22df613f755ab1cc021d8347"}, - {file = "pywin32-307-cp312-cp312-win_arm64.whl", hash = "sha256:b53658acbfc6a8241d72cc09e9d1d666be4e6c99376bc59e26cdb6223c4554d2"}, - {file = "pywin32-307-cp313-cp313-win32.whl", hash = "sha256:ea4d56e48dc1ab2aa0a5e3c0741ad6e926529510516db7a3b6981a1ae74405e5"}, - {file = "pywin32-307-cp313-cp313-win_amd64.whl", hash = "sha256:576d09813eaf4c8168d0bfd66fb7cb3b15a61041cf41598c2db4a4583bf832d2"}, - {file = "pywin32-307-cp313-cp313-win_arm64.whl", hash = "sha256:b30c9bdbffda6a260beb2919f918daced23d32c79109412c2085cbc513338a0a"}, - {file = "pywin32-307-cp37-cp37m-win32.whl", hash = "sha256:5101472f5180c647d4525a0ed289ec723a26231550dbfd369ec19d5faf60e511"}, - {file = "pywin32-307-cp37-cp37m-win_amd64.whl", hash = "sha256:05de55a7c110478dc4b202230e98af5e0720855360d2b31a44bb4e296d795fba"}, - {file = "pywin32-307-cp38-cp38-win32.whl", hash = "sha256:13d059fb7f10792542082f5731d5d3d9645320fc38814759313e5ee97c3fac01"}, - {file = "pywin32-307-cp38-cp38-win_amd64.whl", hash = "sha256:7e0b2f93769d450a98ac7a31a087e07b126b6d571e8b4386a5762eb85325270b"}, - {file = "pywin32-307-cp39-cp39-win32.whl", hash = "sha256:55ee87f2f8c294e72ad9d4261ca423022310a6e79fb314a8ca76ab3f493854c6"}, - {file = "pywin32-307-cp39-cp39-win_amd64.whl", hash = "sha256:e9d5202922e74985b037c9ef46778335c102b74b95cec70f629453dbe7235d87"}, + {file = "pywin32-308-cp310-cp310-win32.whl", hash = "sha256:796ff4426437896550d2981b9c2ac0ffd75238ad9ea2d3bfa67a1abd546d262e"}, + {file = "pywin32-308-cp310-cp310-win_amd64.whl", hash = "sha256:4fc888c59b3c0bef905ce7eb7e2106a07712015ea1c8234b703a088d46110e8e"}, + {file = "pywin32-308-cp310-cp310-win_arm64.whl", hash = "sha256:a5ab5381813b40f264fa3495b98af850098f814a25a63589a8e9eb12560f450c"}, + {file = "pywin32-308-cp311-cp311-win32.whl", hash = "sha256:5d8c8015b24a7d6855b1550d8e660d8daa09983c80e5daf89a273e5c6fb5095a"}, + {file = "pywin32-308-cp311-cp311-win_amd64.whl", hash = "sha256:575621b90f0dc2695fec346b2d6302faebd4f0f45c05ea29404cefe35d89442b"}, + {file = "pywin32-308-cp311-cp311-win_arm64.whl", hash = "sha256:100a5442b7332070983c4cd03f2e906a5648a5104b8a7f50175f7906efd16bb6"}, + {file = "pywin32-308-cp312-cp312-win32.whl", hash = "sha256:587f3e19696f4bf96fde9d8a57cec74a57021ad5f204c9e627e15c33ff568897"}, + {file = "pywin32-308-cp312-cp312-win_amd64.whl", hash = "sha256:00b3e11ef09ede56c6a43c71f2d31857cf7c54b0ab6e78ac659497abd2834f47"}, + {file = "pywin32-308-cp312-cp312-win_arm64.whl", hash = "sha256:9b4de86c8d909aed15b7011182c8cab38c8850de36e6afb1f0db22b8959e3091"}, + {file = "pywin32-308-cp313-cp313-win32.whl", hash = "sha256:1c44539a37a5b7b21d02ab34e6a4d314e0788f1690d65b48e9b0b89f31abbbed"}, + {file = "pywin32-308-cp313-cp313-win_amd64.whl", hash = "sha256:fd380990e792eaf6827fcb7e187b2b4b1cede0585e3d0c9e84201ec27b9905e4"}, + {file = "pywin32-308-cp313-cp313-win_arm64.whl", hash = "sha256:ef313c46d4c18dfb82a2431e3051ac8f112ccee1a34f29c263c583c568db63cd"}, + {file = "pywin32-308-cp37-cp37m-win32.whl", hash = "sha256:1f696ab352a2ddd63bd07430080dd598e6369152ea13a25ebcdd2f503a38f1ff"}, + {file = "pywin32-308-cp37-cp37m-win_amd64.whl", hash = "sha256:13dcb914ed4347019fbec6697a01a0aec61019c1046c2b905410d197856326a6"}, + {file = "pywin32-308-cp38-cp38-win32.whl", hash = "sha256:5794e764ebcabf4ff08c555b31bd348c9025929371763b2183172ff4708152f0"}, + {file = "pywin32-308-cp38-cp38-win_amd64.whl", hash = "sha256:3b92622e29d651c6b783e368ba7d6722b1634b8e70bd376fd7610fe1992e19de"}, + {file = "pywin32-308-cp39-cp39-win32.whl", hash = "sha256:7873ca4dc60ab3287919881a7d4f88baee4a6e639aa6962de25a98ba6b193341"}, + {file = "pywin32-308-cp39-cp39-win_amd64.whl", hash = "sha256:71b3322d949b4cc20776436a9c9ba0eeedcbc9c650daa536df63f0ff111bb920"}, ] [[package]] @@ -2963,25 +2964,25 @@ test = ["websockets"] [[package]] name = "windows-curses" -version = "2.3.3" +version = "2.4.0" description = "Support for the standard curses module on Windows" optional = false python-versions = "*" files = [ - {file = "windows_curses-2.3.3-cp310-cp310-win32.whl", hash = "sha256:859011e77d7d9d9eaabb4fe081760cec966454bdea18a9f3a98e6d10e802fb61"}, - {file = "windows_curses-2.3.3-cp310-cp310-win_amd64.whl", hash = "sha256:cb0eaf41b90c0b7e33269a6cf34900d8b0d7f3fcf2c1f9dd8191bac11955b711"}, - {file = "windows_curses-2.3.3-cp311-cp311-win32.whl", hash = "sha256:3b3d01720724532794e81f0496d18c1284791b6e5b1fa7a59ec878d1c5134dde"}, - {file = "windows_curses-2.3.3-cp311-cp311-win_amd64.whl", hash = "sha256:a5d90950656fc0d85aa3d395ec7f454cc7136add512d40a9c5e8ff29a0ce4905"}, - {file = "windows_curses-2.3.3-cp312-cp312-win32.whl", hash = "sha256:e24b68a1c029d3ff521cf9325ba0f533b303e22ed148952a5a5e6396f29f536c"}, - {file = "windows_curses-2.3.3-cp312-cp312-win_amd64.whl", hash = "sha256:eaea51f169d9940a9bad1edda1759f63795daeb2174b2a45c895148058970689"}, - {file = "windows_curses-2.3.3-cp36-cp36m-win32.whl", hash = "sha256:0d7fb1d799ce3458c9e7215ffa2e8c5e347dabeef1227350b62db6030ebd4c24"}, - {file = "windows_curses-2.3.3-cp36-cp36m-win_amd64.whl", hash = "sha256:1db4246cc019d31614d29e0a26bb8792b0a1a9ba4fbb82ceaf85e5607c7a2857"}, - {file = "windows_curses-2.3.3-cp37-cp37m-win32.whl", hash = "sha256:cb654716c400b24c6410e09415ea682307cf874ee76b109d9c6587a42ac7ed62"}, - {file = "windows_curses-2.3.3-cp37-cp37m-win_amd64.whl", hash = "sha256:ba4a6a0d2d56ad834a1241fb5ce83186322cca87295ef4a768ede870ad33d597"}, - {file = "windows_curses-2.3.3-cp38-cp38-win32.whl", hash = "sha256:a3604136cc0284e2284c7172bb81ba6ff914f820c2eb156e5dde71dafd98fff9"}, - {file = "windows_curses-2.3.3-cp38-cp38-win_amd64.whl", hash = "sha256:15d148658e8967fd3826c907c4ad094d35a7a3605eb0e2d1bbad0b5c81b9df9f"}, - {file = "windows_curses-2.3.3-cp39-cp39-win32.whl", hash = "sha256:b61d1d949b508de4e73e8b4848e243c5d9de691c242b2c8d458a991e5e80ad18"}, - {file = "windows_curses-2.3.3-cp39-cp39-win_amd64.whl", hash = "sha256:23b71f3a44cc5a5b92e12d3a519e840e54f8252ff3ef867b630963d5fe7124dd"}, + {file = "windows_curses-2.4.0-cp310-cp310-win32.whl", hash = "sha256:525fa12689aa54cd8e4cfa6d03f3d4d336f2e9ef0b6db4fdb4d3c1b22a71180b"}, + {file = "windows_curses-2.4.0-cp310-cp310-win_amd64.whl", hash = "sha256:4621042a28f48e2240314a624eb36f18121483d47f12c488cad9670f402ba6d7"}, + {file = "windows_curses-2.4.0-cp311-cp311-win32.whl", hash = "sha256:90fde3341b2361f45716cfbe44bd253ac6a07dfde7262f3afdcf7cc0285c35ac"}, + {file = "windows_curses-2.4.0-cp311-cp311-win_amd64.whl", hash = "sha256:0c32be4aa97887febff175a8a6b3deeecc06c1a3f47ecbb5f62de6bf4165d6db"}, + {file = "windows_curses-2.4.0-cp312-cp312-win32.whl", hash = "sha256:e165db84de30039f6da6bded9aa2f88e3c1ae7be11dc7c84416cbe6ba0ac4f29"}, + {file = "windows_curses-2.4.0-cp312-cp312-win_amd64.whl", hash = "sha256:f50d02833a7e7866b9f42d62febbdf83c6703575240e9bbdd1446d09d70505d3"}, + {file = "windows_curses-2.4.0-cp36-cp36m-win32.whl", hash = "sha256:fcb0591fc6e647556f685327a6f3e6923afc8a16c35436665df692556b78e708"}, + {file = "windows_curses-2.4.0-cp36-cp36m-win_amd64.whl", hash = "sha256:e1f137d347e300e6163fa0aa31f7a02f6da42e43a90ce423bc929fb06752385e"}, + {file = "windows_curses-2.4.0-cp37-cp37m-win32.whl", hash = "sha256:6771fb73f5a66d281a6f5e474dcfe07a4dd64460ff17c72ed9dd35fdf65fbfc8"}, + {file = "windows_curses-2.4.0-cp37-cp37m-win_amd64.whl", hash = "sha256:68d633cdb7e82ec207184ab3be74c058e184cd34f76407801cf451be4ee85861"}, + {file = "windows_curses-2.4.0-cp38-cp38-win32.whl", hash = "sha256:342b786c82fda5f9592593b2f8f4078de40b31baaf331be57af4c3c9a2e40061"}, + {file = "windows_curses-2.4.0-cp38-cp38-win_amd64.whl", hash = "sha256:c77973c5af419c3d9bcb99908dc9791102813026d6abdda9236e44d9342c9579"}, + {file = "windows_curses-2.4.0-cp39-cp39-win32.whl", hash = "sha256:6f481322360dbcdfa37b30654c89aa2920f4528eaefec3a50fa6a6d07be49069"}, + {file = "windows_curses-2.4.0-cp39-cp39-win_amd64.whl", hash = "sha256:441189fb71c455ed0575ffb1b840d5857c01c4dda441cbbbd34a6216e7207bfa"}, ] [[package]] @@ -3085,4 +3086,4 @@ type = ["pytest-mypy"] [metadata] lock-version = "2.0" python-versions = ">=3.10,<4.0" -content-hash = "d83dd23af51005bcf6ee050ba211bec8793fec21b498b054d7de57ca307a70be" +content-hash = "79acfff4c7f912ecd94f4b4d11e0924dc999ed291f0638ca94c4f30ef305f84e" diff --git a/pyproject.toml b/pyproject.toml index 86e2ccd..188a757 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -29,7 +29,7 @@ sagemaker = "^2.59.1" pyexasol = ">=0.26.0,<1" importlib-resources = "^6.4.0" click = "^8.0.3" -exasol-python-extension-common = ">=0.7.0,<1" +exasol-python-extension-common = ">=0.8.0,<1" [tool.poetry.dev-dependencies] pytest = "^7.1" diff --git a/tests/integration_tests/deployment/test_deploy_cli.py b/tests/integration_tests/deployment/test_deploy_cli.py index 04b4f4b..60bed32 100644 --- a/tests/integration_tests/deployment/test_deploy_cli.py +++ b/tests/integration_tests/deployment/test_deploy_cli.py @@ -1,5 +1,5 @@ from click.testing import CliRunner -from exasol.python_extension_common.cli.std_options import StdParams +from exasol.python_extension_common.cli.std_options import StdParams, get_cli_arg from exasol_sagemaker_extension.deploy import deploy_command from exasol_sagemaker_extension.deployment.language_container import export_slc @@ -40,14 +40,9 @@ def test_deploy_cli(pyexasol_connection, cli_args): pyexasol_connection.execute(f'CREATE SCHEMA IF NOT EXISTS "{DB_SCHEMA}"') - def std_param_to_opt(std_param: StdParams) -> str: - # This function should have been implemented in the StdParams - return f'--{std_param.name.replace("_", "-")}' - with export_slc() as container_file: - args_string = (f'{cli_args} ' - f'{std_param_to_opt(StdParams.schema)} "{DB_SCHEMA}" ' - f'{std_param_to_opt(StdParams.container_file)} "{container_file}"') + args_string = ' '.join([cli_args, get_cli_arg(StdParams.schema, DB_SCHEMA), + get_cli_arg(StdParams.container_file, container_file)]) runner = CliRunner() result = runner.invoke(deploy_command, args=args_string, catch_exceptions=False) assert result.exit_code == 0 From 41b0c525d7cfcb2c83f44d20ffafced4f5f5c4d1 Mon Sep 17 00:00:00 2001 From: mibe Date: Tue, 15 Oct 2024 11:16:55 +0100 Subject: [PATCH 2/2] #144 addressed review comments --- doc/user_guide/user_guide.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/doc/user_guide/user_guide.md b/doc/user_guide/user_guide.md index d4d4d08..46f8f9e 100644 --- a/doc/user_guide/user_guide.md +++ b/doc/user_guide/user_guide.md @@ -4,7 +4,7 @@ Exasol Sagemaker Extension provides a Python library together with Exasol Script and UDFs that train Machine Learning Models on data stored in Exasol using AWS SageMaker Autopilot service. -The extension exports a given Exasol table into AWS S3, and then triggers Machine Learning training +The extension exports a given Exasol table into AWS S3, and then triggers Machine Learning training using the AWS Autopilot service with the specified parameters. The training status can be polled using the auxiliary scripts provided within the scope of the project. @@ -52,10 +52,9 @@ In addition, this extension provides the following installation options: | [no-]deploy-slc | True | Install SLC as part of the deployment | | [no-]deploy-scripts | True | Install scripts as part of the deployment | | [no-]to-print | False | Print SQL statements instead of executing them | -| [no-]develop | False | Re-generate the scripts | ### Create AWS Connection Object -Create an Exasol connection object with AWS credentials that has AWS Sagemaker Execution permission. +Create an Exasol connection object with AWS credentials that have AWS Sagemaker Execution permission. The connection will encapsulate the address of the AWS S3 bucket where the exported data will be stored. For more information please check the [Create Connection in Exasol](https://docs.exasol.com/sql/create_connection.htm?Highlight=connection) document. Below is a template of the query that will create the required connection object.