Merge branch 'main' into feature/#255-Rename_data_science_sandbox_to_…

…exasol-ai-lab

doc/changes/changes_0.1.0.md

@@ -48,3 +48,4 @@ Version: 0.1.0
 ## Documentation
 
 * #58: Added location of Juypter notebooks to developer guide
+* #145: Split Development Guide

doc/developer_guide/aws.md

@@ -0,0 +1,68 @@
+## AWS Build and Release Workflow
+
+The following diagram shows the high-level steps to generate the images:
+
+![image info](./img/create-vm-overview.drawio.png)
+
+### Setup EC2
+
+Creates an EC2 instance based on an Ubuntu AMI via AWS Cloudformation. It also creates a new keypair dynamically.
+After the export has finished, the cloudformation stack and the keypair is removed.
+
+### Install
+
+Installs all dependencies via Ansible:
+* installs Poetry
+* installs and configures Jupyter
+* installs Docker and adds the user `ubuntu` to the docker group
+* clones the script-languages-release repository
+* changes the netplan configuration. This is necessary to have proper network configuration when running the VM image
+
+Finally, the default password will be set, and also the password will be marked as expired, such that the user will be forced to enter a new password during initial login.
+Also, the ssh password authentication will be enabled, and for security reasons the folder "~/.ssh" will be removed.
+
+### Export
+
+The export creates an AMI based on the running EC2 instance and exports the AMI as VM image in the default formats to a S3 bucket.
+
+## Release
+
+The release is executed in a AWS Codebuild job, the following diagram shows the flow.
+
+![image info](./img/create-vm-release.drawio.png)
+
+## AWS S3 Bucket
+
+The bucket has private access. In order to control access, the Bucket cloudformation stack also contains a Cloudfront distribution. Public Https access is only possibly through Cloudfront. Another stack contains a Web application firewall (WAF), which will be used by the Cloudfront distribution. Due to restrictions in AWS, the WAF stack needs to be deployed in region "us-east-1". The WAF stack provides two rules which aim to minimize a possible bot attack:
+
+| Name                 | Explanation                                                                               | Priority |
+|----------------------|-------------------------------------------------------------------------------------------|----------|
+| VMBucketRateLimit    | Declares the minimum possible rate limit for access: 100 requests in a 5 min interval.    | 0        |
+| CAPTCHA              | Forces a captcha action for any IP which does not match a predefined set of IP-addresses. | 1        |
+
+## Involved Cloudformation stacks
+
+The following diagram shows the involved cloudformation stacks:
+![image info](./img/cloudformation-stacks.drawio.png)
+
+The following resources are permanent and need to be deployed using the "deploy" [commands](#deployment-commands):
+* `DATA-SCIENCE-SANDBOX-VM-Bucket`
+* `DATA-SCIENCE-SANDBOX-CI-TEST-CODEBUILD`
+* `DATA-SCIENCE-SANDBOX-RELEASE-CODEBUILD`
+
+The EC2-stack lives only during the creation of a new sandbox image.
+
+## Tagging
+
+Each of the involved resources might cause costs: cloudformation stacks, AMI, EC2 key-pairs.
+
+To enable keeping track of all these resources, the implementation tags them after creation with a specific keyword (called __asset-id__).
+
+The S3 objects are identified by the prefix in the S3 bucket.
+
+The command tags only the dynamically created entities with the *asset-id* but not the permanent cloudformation stacks.
+
+The command `show-aws-assets` lists all assets which were created during the execution.
+* This is very useful if an error occured.
+* If the creation of a sandbox finished normally the list should contain only the AMI, images (S3 objects) and the export tasks (one for each image).
+

doc/developer_guide/ci.md

@@ -0,0 +1,32 @@
+## Running tests in the CI
+
+The project has two types of CI tests:
+* unit tests and integration tests which run in a Github workflow
+* A system test which runs on a AWS Codebuild
+
+Both ci tests need to pass before the approval of a Github PR.
+The Github workflow will run on each push to a branch in the Github repository. However, the AWS Codebuild will only run after you push a commit containing the string "[CodeBuild]" in the commit message, see [Executing AWS CodeBuild](#executing-aws-codebuild).
+
+## Executing AWS CodeBuild
+
+Use the following git commands to execute the AWS CodeBuild script:
+
+```shell
+git commit -m "[CodeBuild]" --allow-empty && git push
+```
+
+This will trigger a webhook that was installed by an AWS template into the git-Repository.
+* The webhook is defined in file `exasol/ds/sandbox/templates/ci_code_build.jinja.yaml`
+* and calls `aws-code-build/ci/buildspec.yaml`
+* which then executes `test/codebuild/test_ci*.py`
+
+The CodeBuild will take about 20 minutes to complete.
+
+## Running AWS CodeBuild locally
+
+To run these tests locally please use
+
+ ```shell 
+ export DSS_RUN_CI_TEST=true; poetry run test/codebuild/test_ci.py 
+```
+

doc/developer_guide/commands.md

@@ -0,0 +1,73 @@
+# Commands
+
+The commands offered by the DSS CLI can be organized into three groups:
+
+| Group                | Usage                                   |
+|----------------------|-----------------------------------------|
+| Release Commands     | during the release                      |
+| Deployment Commands  | to deploy infrastructure onto AWS cloud |
+| Development Commands | to identify problems or for testing     |
+
+## Release commands
+
+The following commands are used during the release AWS Codebuild job:
+* `create-vm`: Create a new AMI and VM images.
+* `update-release`: Update release notes of an existing Github release.
+* `start-release-build`: Start the release on AWS codebuild.
+* `create-docker-image`: Create a Docker image for data-science-sandbox and deploy it to hub.docker.com/exasol/data-science-sandbox.
+
+Script `start-release-build`:
+* Is usually called from github workflow `release_droid_upload_github_release_assets.yml`.
+* Requires environment variable `GH_TOKEN` to contain a valid token for access to Github.
+* Requires to specify CLI option `--upload-url`.
+
+This operation usually takes around than 1:40 hours.
+
+## Developer commands
+
+All other commands provide a subset of the features of the release commands, and can be used to identify problems or simulate the release:
+* `export-vm`: Create a new VM image from a running EC2-Instance.
+* `install-dependencies`: Start an ansible-installation onto an existing EC-2 instance.
+* `reset-password`: Reset password on a remote EC-2-instance via ansible.
+* `setup-ec2`: Start a new EC2 instance (based on an Ubuntu AMI).
+* `setup-ec2-and-install-dependencies`: Start a new EC2 instance and install dependencies via Ansible.
+  * The script will print the required SSH login for manual inspection or interaction with the EC2 instance.
+  * The instance is kept running until the user presses Ctrl-C.
+* `show-aws-assets`: Show AWS entities associated with a specific keyword (called __asset-id__).
+* `start-test-release-build`: (For testing) Creates a release on Github and forwards it to the AWS Codebuild which creates VM images in various formats and attaches them to the Github release.
+* `make-ami-public`: Change permissions of an existing AMI such that it becomes public.
+
+Script `start-test-release-build` requires environment variable `GH_TOKEN` to contain a valid token for access to Github.
+
+## Deployment commands
+
+The following commands can be used to deploy the infrastructure onto a given AWS account:
+* `setup-ci-codebuild`: Deploy the AWS Codebuild cloudformation stack which will run the ci-test.
+* `setup-vm-bucket`: Deploy the AWS Bucket cloudformation stack which will be used to deploy the VM images.
+* `setup-release-codebuild`: Deploy the AWS Codebuild cloudformation stack which will be used for the release-build.
+* `setup-vm-bucket-waf`: Deploy the AWS Codebuild cloudformation stack which contains the WAF Acl configuration for the Cloudfront distribution of the VM Bucket.
+
+For all deployment commands:
+* Don't forget to specify CLI option `--aws-profile`.
+* Ensure the related AWS stack does not exist. If there was a rollback then please delete the stack manually, otherwise the script will fail.
+
+If `setup-release-codebuild` or `setup-ci-codebuild` fails with error message "_Failed to create webhook. Repository not found or permission denied._" then
+* Ensure to grant sufficient access permissions to the Github user used by the script.
+* You can use a Github "_Repository role_" for that.
+* The repository role must include the following permissions
+  * Inherit the permissions from default role "Write"
+  * Additional repository permission "Manage webhooks"
+* In AWS you can configure the Github token by a resource with logical ID `CodeBuildCredentials`
+  * Please note: There must be only one stack containing such a resource.
+  * The definition of the AWS resource `CodeBuildCredentials` can use credentials from tha AWS secret manager.
+
+```yaml
+Resources:
+  CodeBuildCredentials:
+    Type: AWS::CodeBuild::SourceCredential
+    Properties:
+      ServerType: GITHUB
+      AuthType: PERSONAL_ACCESS_TOKEN
+      Username: "{{resolve:secretsmanager:github_personal_token:SecretString:github_user_name}}"
+      Token: "{{resolve:secretsmanager:github_personal_token:SecretString:github_personal_token}}"
+```