Skip to content

Commit

Permalink
Update master
Browse files Browse the repository at this point in the history
Changelist:

- #944: Removed exaudfclient_bin_py3 (#975) 
- #967: Added ctpg script options parser (#968) 
- #972 and #973: Refactorings for Script Options parser (#976) 
- #977: Fixed Trivy update cache workflow (#978) 
- #969: Use new CTPG parser in java vm (#970) 
- #980: Moved flavor template-Exasol-all-java-17 to script-languages repo (#984)
- #971: Removed dead code in Java VM (#985) 
- Refactorings in ScriptOptions parser (#987) 
- #988: Adapted Java integration tests for new ScriptOptions parser (#995) 
- #997: Added documentation for Rollback of Script-Languages-Container (#999)
- #878: Fixed jvmoption and parameters containing a space (#1001) 
- #1002: Use ECR fallback repository for Trivy caching (#1004) 
- #1006: Updated Dependencies on top of 8.4.0 (#1007) 
- #958: Fix conda version conflict (#1010) 
- #1009: Removed NVIDIA GPU Driver Libraries from template-CUDA SLC (#1012)
- Bumped version to 9.0.0 (#1013) 
- #1014: Fixed Push Docker Test Container workflow (#1015)
  • Loading branch information
tomuben authored Nov 14, 2024
2 parents ae613bf + 4bedaba commit f2edb41
Show file tree
Hide file tree
Showing 221 changed files with 1,084 additions and 2,437 deletions.
2 changes: 1 addition & 1 deletion .current_gitmodules

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

29 changes: 29 additions & 0 deletions .github/workflows/check_bazel_build.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
name: Check Bazel Build

on:
push:
branches:
- develop
pull_request:


jobs:
build:
runs-on: ubuntu-22.04
env:
UDF_CLIENT_ENV_FILE: /tmp/.udf_client_env
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
submodules: 'recursive'

- name: Install UDF Client Dependencies
run: |
sudo bash scripts/installUdfClientDeps.sh "$UDF_CLIENT_ENV_FILE"
- uses: r-lib/actions/setup-r@v2
- name: Build
run: |
source "$UDF_CLIENT_ENV_FILE"
bazel build --lockfile_mode=off --config no-tty -c dbg --config python --config java --config fast-binary --config r --verbose_failures
working-directory: ./exaudfclient/
42 changes: 0 additions & 42 deletions .github/workflows/check_bazel_tests.yml

This file was deleted.

92 changes: 92 additions & 0 deletions .github/workflows/check_rollback_7.1.30_script.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
name: Check Rollback Script 7.1.30

on:
push:
branches:
- develop
pull_request:

jobs:
Check:
runs-on: ubuntu-20.04
strategy:
fail-fast: false
matrix:
include:
- db-version: "7.1.29"
expected-original-alter-system-parameter: "ALTER SYSTEM SET SCRIPT_LANGUAGES='R=builtin_r JAVA=builtin_java PYTHON3=builtin_python3'"
expected-new-alter-system-command: "ALTER SYSTEM SET SCRIPT_LANGUAGES='JAVA=builtin_java PYTHON3=builtin_python3 R=localzmq+protobuf:///bfsdefault/default/EXAClusterOS/ScriptLanguages-standard-EXASOL-7.1.0-slc-v6.0.0-VYP23K36/?lang=r#/buckets/bfsdefault/default/EXAClusterOS/ScriptLanguages-standard-EXASOL-7.1.0-slc-v6.0.0-VYP23K36/exaudf/exaudfclient_py3';"
support-python2: "no"
- db-version: "7.1.19"
expected-original-alter-system-parameter: "ALTER SYSTEM SET SCRIPT_LANGUAGES='PYTHON=builtin_python R=builtin_r JAVA=builtin_java PYTHON3=builtin_python3';"
expected-new-alter-system-command: "ALTER SYSTEM SET SCRIPT_LANGUAGES='JAVA=builtin_java PYTHON3=builtin_python3 PYTHON=localzmq+protobuf:///bfsdefault/default/EXAClusterOS/ScriptLanguages-standard-EXASOL-7.1.0-slc-v4.0.0-CM4RWW6R/?lang=python#/buckets/bfsdefault/default/EXAClusterOS/ScriptLanguages-standard-EXASOL-7.1.0-slc-v4.0.0-CM4RWW6R/exaudf/exaudfclient R=localzmq+protobuf:///bfsdefault/default/EXAClusterOS/ScriptLanguages-standard-EXASOL-7.1.0-slc-v4.0.0-CM4RWW6R/?lang=r#/buckets/bfsdefault/default/EXAClusterOS/ScriptLanguages-standard-EXASOL-7.1.0-slc-v4.0.0-CM4RWW6R/exaudf/exaudfclient_py3';"
support-python2: "yes"
- db-version: "7.1.6"
expected-original-alter-system-parameter: "ALTER SYSTEM SET SCRIPT_LANGUAGES='PYTHON=builtin_python R=builtin_r JAVA=builtin_java PYTHON3=builtin_python3';"
expected-new-alter-system-command: "ALTER SYSTEM SET SCRIPT_LANGUAGES='JAVA=builtin_java PYTHON3=builtin_python3 PYTHON=localzmq+protobuf:///bfsdefault/default/EXAClusterOS/ScriptLanguages-release-standard-exasol-7.1.0-1.1.0/?lang=python#/buckets/bfsdefault/default/EXAClusterOS/ScriptLanguages-release-standard-exasol-7.1.0-1.1.0/exaudf/exaudfclient R=localzmq+protobuf:///bfsdefault/default/EXAClusterOS/ScriptLanguages-release-standard-exasol-7.1.0-1.1.0/?lang=r#/buckets/bfsdefault/default/EXAClusterOS/ScriptLanguages-release-standard-exasol-7.1.0-1.1.0/exaudf/exaudfclient_py3';"
support-python2: "yes"
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
submodules: 'recursive'

- name: Setup Python & Poetry Environment
uses: exasol/python-toolbox/.github/actions/[email protected]
with:
python-version: "3.10"
poetry-version: '1.8.2'
- name: Start ITDE
run: poetry run itde spawn-test-environment --environment-name exasol_test --database-port-forward 8888 --bucketfs-port-forward 6583 --docker-db-image-version "${{ matrix.db-version }}"
- name: Install dependencies
run: sudo apt update && sudo apt install -y perl default-jre
- name: Install EXAPlus
run: |
curl -o exaplus.tar.gz https://exasol-script-languages-dependencies.s3.eu-central-1.amazonaws.com/EXAplus-7.0.11.tar.gz && tar xf exaplus.tar.gz
echo "/tmp/EXAplus-7.0.11" >> $GITHUB_PATH
working-directory: /tmp
- name: Check if get_original_script_languages_parameters.sql works
run: |
RESULT=`exaplus -x -q -c localhost:8888 -u sys -p exasol -f ./get_original_script_languages_parameter.sql`
echo "$RESULT" | grep "$EXPECTED_ORIGINAL_ALTER_SYSTEM_PARAMETER" || (echo SCRIPT_LANGUAGES parameter not as expected && exit 1)
env:
EXPECTED_ORIGINAL_ALTER_SYSTEM_PARAMETER: "${{ matrix.expected-original-alter-system-parameter }}"

working-directory: doc/user_guide/resources
- name: Install a simple R UDF
run: |
sleep 120
exaplus -x -q -c localhost:8888 -u sys -p exasol -f install_r_udf.sql
exaplus -x -q -c localhost:8888 -u sys -p exasol -sql "SELECT TEST.R_DEMO();"
working-directory: .github/workflows/scripts
- name: Install a simple Python2 UDF
if: ${{ matrix.support-python2=='yes' }}
run: |
exaplus -x -q -c localhost:8888 -u sys -p exasol -f install_python2_udf.sql
exaplus -x -q -c localhost:8888 -u sys -p exasol -sql "SELECT TEST.TEST_PYTHON_VERSION();"
working-directory: .github/workflows/scripts
- name: Check if pre_update_check.sql works for R only
if: ${{ matrix.support-python2=='no' }}
run: |
RESULT=`exaplus -x -q -c localhost:8888 -u sys -p exasol -f ./pre_update_check.sql`
echo "$RESULT" | grep "You have 1 UDFs using the pre-shipped R/Python2"
working-directory: doc/user_guide/resources
- name: Check if pre_update_check.sql works for R and Python2
if: ${{ matrix.support-python2=='yes' }}
run: |
RESULT=`exaplus -x -q -c localhost:8888 -u sys -p exasol -f ./pre_update_check.sql`
echo "$RESULT" | grep "You have 2 UDFs using the pre-shipped R/Python2"
working-directory: doc/user_guide/resources
- name: Run construct_alter_system_command_before_update.sql
run: |
RESULT=`exaplus -c localhost:8888 -u sys -p exasol -f ./construct_alter_system_command_before_update.sql`
echo "$RESULT" | tr -d '\n' | grep "$EXPECTED_NEW_ALTER_SYSTEM_PARAMETER" || (echo ALTER SYSTEM command not as expected && exit 1)
exaplus -x -q -c localhost:8888 -u sys -p exasol -sql "$EXPECTED_NEW_ALTER_SYSTEM_PARAMETER"
working-directory: doc/user_guide/resources
env:
EXPECTED_NEW_ALTER_SYSTEM_PARAMETER: "${{ matrix.expected-new-alter-system-command }}"
- name: Test if R UDF still works
run: exaplus -x -q -c localhost:8888 -u sys -p exasol -sql "SELECT TEST.R_DEMO();"
- name: Test if Python2 UDF still works
if: ${{ matrix.support-python2=='yes' }}
run: exaplus -x -q -c localhost:8888 -u sys -p exasol -sql "SELECT TEST.TEST_PYTHON_VERSION();"
12 changes: 9 additions & 3 deletions .github/workflows/push_docker_test_container.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -16,20 +16,26 @@ jobs:
with:
fetch-depth: 0
submodules: 'recursive'
- name: Setup Python & Poetry Environment
uses: exasol/python-toolbox/.github/actions/[email protected]
with:
python-version: "3.10"
poetry-version: '1.8.2'

- name: Build TestContainer
run: ./exaslct build-test-container --force-rebuild
run: poetry run exaslct build-test-container --force-rebuild
- name: Push TestContainer image
if: ${{ (github.ref == 'refs/heads/master') || (github.ref == 'refs/heads/main') }}
run: |
./exaslct push-test-container --source-docker-username "$SECRET_DOCKER_USER_NAME" --target-docker-username "$SECRET_DOCKER_USER_NAME" --force-push true
poetry run exaslct push-test-container --source-docker-username "$SECRET_DOCKER_USER_NAME" --target-docker-username "$SECRET_DOCKER_USER_NAME" --force-push true
env: # Set the secret as an input
SECRET_DOCKER_USER_NAME: ${{ secrets.DOCKER_USER_NAME }}
TARGET_DOCKER_PASSWORD: ${{ secrets.DOCKER_TOKEN }}
SOURCE_DOCKER_PASSWORD: ${{ secrets.DOCKER_TOKEN }}
- name: Push ci-release-test TestContainer image
if: ${{ (github.ref != 'refs/heads/master') && (github.ref != 'refs/heads/main') }}
run: |
./exaslct push-test-container --source-docker-username "$SECRET_DOCKER_USER_NAME" --target-docker-username "$SECRET_DOCKER_USER_NAME" --force-push true --source-docker-tag-prefix ci_release_test --target-docker-tag-prefix ci_release_test
poetry run exaslct push-test-container --source-docker-username "$SECRET_DOCKER_USER_NAME" --target-docker-username "$SECRET_DOCKER_USER_NAME" --force-push true --source-docker-tag-prefix ci_release_test --target-docker-tag-prefix ci_release_test
env: # Set the secret as an input
SECRET_DOCKER_USER_NAME: ${{ secrets.DOCKER_USER_NAME }}
TARGET_DOCKER_PASSWORD: ${{ secrets.DOCKER_TOKEN }}
Expand Down
10 changes: 10 additions & 0 deletions .github/workflows/scripts/install_python2_udf.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
CREATE SCHEMA IF NOT EXISTS TEST;
OPEN SCHEMA TEST;
--/

CREATE OR REPLACE PYTHON SCALAR SCRIPT "TEST_PYTHON_VERSION" () RETURNS VARCHAR(2000) AS
import sys
def run(ctx):
return sys.version
/

10 changes: 10 additions & 0 deletions .github/workflows/scripts/install_r_udf.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
CREATE SCHEMA IF NOT EXISTS TEST;
OPEN SCHEMA TEST;

--/
CREATE OR REPLACE R SCALAR SCRIPT TEST.R_DEMO() RETURNS VARCHAR(2000) AS

run <- function(ctx) {
"Minimal R language UDF"
}
/
6 changes: 3 additions & 3 deletions .github/workflows/update_trivy_cache.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ jobs:
steps:
- name: Run install dependencies
run: |
sudo apt update && sudo apt install -y awscli curl wget apt-transport-https gnupg lsb-release
sudo apt update && sudo apt install -y curl wget apt-transport-https gnupg lsb-release
wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | sudo apt-key add -
echo deb https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -sc) main | sudo tee -a /etc/apt/sources.list.d/trivy.list
sudo apt-get update
Expand All @@ -22,8 +22,8 @@ jobs:

- name: Fetch trivy databases
run: |
trivy image --download-java-db-only
trivy image --download-db-only
trivy image --download-java-db-only || (echo "Falling back to ECR repository...." && trivy image --download-java-db-only --java-db-repository public.ecr.aws/aquasecurity/trivy-java-db)
trivy image --download-db-only || (echo "Falling back to ECR repository....." && trivy image --download-db-only --db-repository public.ecr.aws/aquasecurity/trivy-db)
- name: Create tar gz databases
run: |
Expand Down
1 change: 0 additions & 1 deletion aws-code-build/build_config.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@
"ignore": {
"paths": [
".github",
"config",
"doc",
"emulator",
"githooks",
Expand Down
14 changes: 7 additions & 7 deletions aws-code-build/ci/buildspec.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

14 changes: 7 additions & 7 deletions aws-code-build/release/buildspec.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

68 changes: 0 additions & 68 deletions config/.bcc.env

This file was deleted.

13 changes: 0 additions & 13 deletions config/README.md

This file was deleted.

3 changes: 0 additions & 3 deletions config/build-env

This file was deleted.

Loading

0 comments on commit f2edb41

Please sign in to comment.