From b6d144f41109717a19aad2e109e20b5c6c61c90e Mon Sep 17 00:00:00 2001 From: Bernardo Meireles <67381633+bcmeireles@users.noreply.github.com> Date: Tue, 3 Dec 2024 17:03:43 +0000 Subject: [PATCH] Create GH Actions CI Pipeline w/ Expected Failed Tests (#248) ### Summary Created a workflow to run the tests for `dbt-dremio` when a PR is created ### Description Since we have 22 failing tests that we plan to work on fixing, the current pipeline will just ignore pytest's exit code on error and will compare the failing tests with the tests that are expected to fail. The pipeline from https://github.com/dremio/dbt-dremio/pull/249 is to be deployed once all the tests are fixed and it does not use this comparison approach The test suites are also ran separately (folder by folder) as some tests will fail when trying to run all tests consecutively from the root test folder A list of expected failures can be found (and edited) in `.github/expected_failures.txt` ### Test Results Failing (expected) tests: ``` tests/component/test_profile_template.py::TestProfileTemplate::test_cloud_options tests/functional/adapter/dbt_clone/test_dbt_clone.py::TestCloneNotPossibleDremio::test_can_clone_false tests/functional/adapter/dremio_specific/test_drop_temp_table.py::TestDropTempTableDremio::test_drop_temp_table tests/functional/adapter/dremio_specific/test_schema_parsing.py::TestSchemaParsingDremio::test_schema_with_dots tests/functional/adapter/dremio_specific/test_verify_ssl.py::TestVerifyCertificateDremio::test_insecure_request_warning_not_exist tests/functional/adapter/grants/test_incremental_grants.py::TestIncrementalGrantsDremio::test_incremental_grants tests/functional/adapter/grants/test_invalid_grants.py::TestInvalidGrantsDremio::test_invalid_grants tests/functional/adapter/grants/test_model_grants.py::TestTableGrantsDremio::test_view_table_grants tests/functional/adapter/grants/test_model_grants.py::TestViewGrantsDremio::test_view_table_grants tests/functional/adapter/grants/test_seed_grants.py::TestSeedGrantsDremio::test_seed_grants tests/functional/adapter/grants/test_snapshot_grants.py::TestSnapshotGrantsDremio::test_snapshot_grants tests/functional/adapter/relation/test_get_relation_last_modified.py::TestGetLastRelationModified::test_get_last_relation_modified tests/functional/adapter/unit_testing/test_unit_testing.py::TestDremioUnitTestCaseInsensitivity::test_case_insensitivity tests/functional/adapter/unit_testing/test_unit_testing.py::TestDremioUnitTestInvalidInput::test_invalid_input tests/functional/adapter/unit_testing/test_unit_testing.py::TestDremioUnitTestingTypes::test_unit_test_data_type tests/hooks/test_model_hooks.py::TestPrePostModelHooksInConfigDremio::test_pre_and_post_model_hooks_model tests/hooks/test_model_hooks.py::TestPrePostModelHooksInConfigKwargsDremio::test_pre_and_post_model_hooks_model tests/hooks/test_model_hooks.py::TestPrePostModelHooksInConfigWithCountDremio::test_pre_and_post_model_hooks_model_and_project tests/hooks/test_model_hooks.py::TestPrePostModelHooksOnSnapshotsDremio::test_hooks_on_snapshots tests/hooks/test_model_hooks.py::TestPrePostSnapshotHooksInConfigKwargsDremio::test_hooks_on_snapshots tests/hooks/test_run_hooks.py::TestPrePostRunHooksDremio::test_pre_and_post_run_hooks tests/hooks/test_run_hooks.py::TestPrePostRunHooksDremio::test_pre_and_post_seed_hooks ``` ### Changelog - created `.github/workflows/ci.yml` ### Related Issue https://github.com/dremio/dbt-dremio/issues/39 --- .github/expected_failures.txt | 22 +++ .github/workflows/ci.yml | 346 ++++++++++++++++++++++++++++++++++ 2 files changed, 368 insertions(+) create mode 100644 .github/expected_failures.txt create mode 100644 .github/workflows/ci.yml diff --git a/.github/expected_failures.txt b/.github/expected_failures.txt new file mode 100644 index 0000000..c92e6da --- /dev/null +++ b/.github/expected_failures.txt @@ -0,0 +1,22 @@ +tests/component/test_profile_template.py::TestProfileTemplate::test_cloud_options +tests/functional/adapter/dbt_clone/test_dbt_clone.py::TestCloneNotPossibleDremio::test_can_clone_false +tests/functional/adapter/dremio_specific/test_drop_temp_table.py::TestDropTempTableDremio::test_drop_temp_table +tests/functional/adapter/dremio_specific/test_schema_parsing.py::TestSchemaParsingDremio::test_schema_with_dots +tests/functional/adapter/dremio_specific/test_verify_ssl.py::TestVerifyCertificateDremio::test_insecure_request_warning_not_exist +tests/functional/adapter/grants/test_incremental_grants.py::TestIncrementalGrantsDremio::test_incremental_grants +tests/functional/adapter/grants/test_invalid_grants.py::TestInvalidGrantsDremio::test_invalid_grants +tests/functional/adapter/grants/test_model_grants.py::TestViewGrantsDremio::test_view_table_grants +tests/functional/adapter/grants/test_model_grants.py::TestTableGrantsDremio::test_view_table_grants +tests/functional/adapter/grants/test_seed_grants.py::TestSeedGrantsDremio::test_seed_grants +tests/functional/adapter/grants/test_snapshot_grants.py::TestSnapshotGrantsDremio::test_snapshot_grants +tests/functional/adapter/relation/test_get_relation_last_modified.py::TestGetLastRelationModified::test_get_last_relation_modified +tests/functional/adapter/unit_testing/test_unit_testing.py::TestDremioUnitTestingTypes::test_unit_test_data_type +tests/functional/adapter/unit_testing/test_unit_testing.py::TestDremioUnitTestCaseInsensitivity::test_case_insensitivity +tests/functional/adapter/unit_testing/test_unit_testing.py::TestDremioUnitTestInvalidInput::test_invalid_input +tests/hooks/test_model_hooks.py::TestPrePostModelHooksOnSnapshotsDremio::test_hooks_on_snapshots +tests/hooks/test_model_hooks.py::TestPrePostModelHooksInConfigDremio::test_pre_and_post_model_hooks_model +tests/hooks/test_model_hooks.py::TestPrePostModelHooksInConfigWithCountDremio::test_pre_and_post_model_hooks_model_and_project +tests/hooks/test_model_hooks.py::TestPrePostModelHooksInConfigKwargsDremio::test_pre_and_post_model_hooks_model +tests/hooks/test_model_hooks.py::TestPrePostSnapshotHooksInConfigKwargsDremio::test_hooks_on_snapshots +tests/hooks/test_run_hooks.py::TestPrePostRunHooksDremio::test_pre_and_post_run_hooks +tests/hooks/test_run_hooks.py::TestPrePostRunHooksDremio::test_pre_and_post_seed_hooks \ No newline at end of file diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000..c0beee2 --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,346 @@ +name: CI + +on: + push: + branches: + - main + pull_request: + branches: + - main + +jobs: + run-tests: + name: Setup and Run Tests + runs-on: ubuntu-latest + env: + RETRY_COUNT: 12 # number of retries for health checks + SLEEP_INTERVAL: 5 # Sleep duration in seconds between retries + MINIO_HEALTH_URL: http://localhost:9000/minio/health/live + DREMIO_HEALTH_URL: http://localhost:9047 + + steps: + - name: Check out repository + uses: actions/checkout@v2 + + - name: Set up Environment Variables + run: | + echo "DREMIO_SOFTWARE_HOST=localhost" >> $GITHUB_ENV + echo "DREMIO_SOFTWARE_USERNAME=dremio" >> $GITHUB_ENV + echo "DREMIO_SOFTWARE_PASSWORD=dremio123" >> $GITHUB_ENV + echo "DREMIO_DATALAKE=dbt_test_source" >> $GITHUB_ENV + echo "DREMIO_DATABASE=dbt_test" >> $GITHUB_ENV + echo "DBT_TEST_USER_1=dbt_test_user_1" >> $GITHUB_ENV + echo "DBT_TEST_USER_2=dbt_test_user_2" >> $GITHUB_ENV + echo "DBT_TEST_USER_3=dbt_test_user_3" >> $GITHUB_ENV + + - name: Create Docker Network + run: | + docker network create ci-network + + - name: Start MinIO Service + run: | + docker run -d \ + --network ci-network \ + --name minio \ + -p 9000:9000 \ + -p 9001:9001 \ + -e "MINIO_ROOT_USER=admin" \ + -e "MINIO_ROOT_PASSWORD=password" \ + minio/minio server /data --console-address ":9001" + + - name: Start Dremio Service + run: | + docker run -d \ + --network ci-network \ + --name dremio \ + -p 31010:31010 \ + -p 9047:9047 \ + -e "DREMIO_JAVA_SERVER_EXTRA_OPTS=-Ddebug.addDefaultUser=true" \ + dremio/dremio-oss + + - name: Install MinIO Client (mc) + run: | + curl -O https://dl.min.io/client/mc/release/linux-amd64/mc + chmod +x mc + sudo mv mc /usr/local/bin/ + + - name: Create MinIO bucket + run: | + for i in $(seq 1 $RETRY_COUNT); do + if docker exec minio curl -s $MINIO_HEALTH_URL; then + echo "MinIO is up." + break + fi + echo "Waiting for MinIO to become ready..." + sleep 5 + done + if ! docker exec minio curl -s $MINIO_HEALTH_URL; then + echo "MinIO did not become ready in time." + exit 1 + fi + + # Set alias to MinIO + mc alias set myminio http://localhost:9000 admin password + + echo "Creating bucket dbtdremios3" + mc mb myminio/dbtdremios3 + + echo "Setting bucket policy to public" + mc policy set public myminio/dbtdremios3 + + echo "Listing all buckets to verify" + mc ls myminio + + - name: "Create Dremio S3 Source" + run: | + sudo apt-get update + sudo apt-get install -y curl jq + + for i in $(seq 1 $RETRY_COUNT); do + if docker exec dremio curl -s $DREMIO_HEALTH_URL; then + echo "Dremio is up." + break + fi + echo "Waiting for Dremio to become ready..." + sleep 5 + done + if ! docker exec dremio curl -s $DREMIO_HEALTH_URL; then + echo "Dremio did not become ready in time." + exit 1 + fi + + echo "Logging into Dremio to obtain auth token..." + AUTH_TOKEN=$(curl -s -X POST "http://localhost:9047/apiv2/login" \ + -H "Content-Type: application/json" \ + --data "{\"userName\":\"dremio\", \"password\":\"dremio123\"}" | jq -r .token) + + # Check if AUTH_TOKEN is not empty + if [ -z "$AUTH_TOKEN" ]; then + echo "Failed to obtain Dremio auth token" + exit 1 + fi + + echo "Obtained Dremio auth token: $AUTH_TOKEN" + + echo "Creating the S3 source in Dremio..." + curl -s -X PUT "http://localhost:9047/apiv2/source/dbt_test_source" \ + -H "Content-Type: application/json" \ + -H "Authorization: _dremio$AUTH_TOKEN" \ + --data "{\"name\":\"dbt_test_source\",\"config\":{\"credentialType\":\"ACCESS_KEY\",\"accessKey\":\"admin\",\"accessSecret\":\"password\",\"secure\":false,\"externalBucketList\":[],\"enableAsync\":true,\"enableFileStatusCheck\":true,\"rootPath\":\"/\",\"defaultCtasFormat\":\"ICEBERG\",\"propertyList\":[{\"name\":\"fs.s3a.path.style.access\",\"value\":\"true\"},{\"name\":\"fs.s3a.endpoint\",\"value\":\"minio:9000\"},{\"name\":\"dremio.s3.compat\",\"value\":\"true\"}],\"whitelistedBuckets\":[],\"isCachingEnabled\":false,\"maxCacheSpacePct\":100},\"type\":\"S3\",\"metadataPolicy\":{\"deleteUnavailableDatasets\":true,\"autoPromoteDatasets\":false,\"namesRefreshMillis\":3600000,\"datasetDefinitionRefreshAfterMillis\":3600000,\"datasetDefinitionExpireAfterMillis\":10800000,\"authTTLMillis\":86400000,\"updateMode\":\"PREFETCH_QUERIED\"}}" + + echo "S3 Source created in Dremio" + + - name: Install Dependencies + run: | + pip install -r dev_requirements.txt + pip install . + + - name: Create dbt test users + run: | + curl 'http://localhost:9047/api/v3/user' \ + -H "Authorization: _dremio$AUTH_TOKEN" \ + -H 'Content-Type: application/json' \ + --data-raw '{"firstName":"dbt","lastName":"user1","name":"dbt_test_user_1","email":"dbt_test_user_1@dremio.com","password":"dremio123"}' + + curl 'http://localhost:9047/api/v3/user' \ + -H "Authorization: _dremio$AUTH_TOKEN" \ + -H 'Content-Type: application/json' \ + --data-raw '{"firstName":"dbt","lastName":"user2","name":"dbt_test_user_2","email":"dbt_test_user_2@dremio.com","password":"dremio123"}' + + curl 'http://localhost:9047/api/v3/user' \ + -H "Authorization: _dremio$AUTH_TOKEN" \ + -H 'Content-Type: application/json' \ + --data-raw '{"firstName":"dbt","lastName":"user3","name":"dbt_test_user_3","email":"dbt_test_user_3@dremio.com","password":"dremio123"}' + + echo "users created" + + - name: Create dbt projects + run: | + dbt init test_cloud_options < tests/.env + DREMIO_SOFTWARE_HOST=localhost + DREMIO_SOFTWARE_USERNAME=dremio + DREMIO_SOFTWARE_PASSWORD=dremio123 + DREMIO_DATALAKE=dbt_test_source + DREMIO_DATABASE=dbt_test + DBT_TEST_USER_1=dbt_test_user_1 + DBT_TEST_USER_2=dbt_test_user_2 + DBT_TEST_USER_3=dbt_test_user_3 + EOF + + - name: Create Reports Directory + run: | + mkdir -p reports + + - name: Run tests + run: | + #!/bin/bash + set -e + + echo "Starting tests" + + test_dirs=$(find tests/ -type f \( -name 'test_*.py' -o -name '*_test.py' \) -exec dirname {} \; | sort -u) + + echo "$test_dirs" + + for dir in $test_dirs; do + echo "Running tests in directory: $dir" + # Generate a safe report filename + report_file="reports/$(echo "$dir" | tr '/' '_').txt" + echo "Saving report to: $report_file" + pytest "$dir" | tee "$report_file" + done + + echo "All tests executed." + + - name: Upload tests report as artifact + uses: actions/upload-artifact@v3 + with: + name: all-tests-reports + path: reports/ + + upload-individual-test-reports: + name: Upload Tests Artifacts + runs-on: ubuntu-latest + needs: run-tests + + steps: + - name: Download test reports + uses: actions/download-artifact@v3 + with: + name: all-tests-reports + path: reports/ + + - name: Upload individual test reports + uses: actions/upload-artifact@v3 + with: + name: individual-test-reports + path: reports/*.txt + + verify-failures: + name: Verify Expected Test Failures + runs-on: ubuntu-latest + needs: [run-tests, upload-individual-test-reports] + + steps: + - name: Check out repository + uses: actions/checkout@v2 + + - name: Download All Test Reports + uses: actions/download-artifact@v3 + with: + name: all-tests-reports + path: reports/ + + - name: Set Up Python Environment + uses: actions/setup-python@v4 + with: + python-version: '3.9' + + - name: Extract Actual Failed Tests + run: | + shopt -s globstar + grep "FAILED" reports/**/*.txt | awk '{print $2}' | sort > actual_failures_sorted.txt + + - name: Sort Expected Failures + run: sort .github/expected_failures.txt > expected_failures_sorted.txt + + - name: Compare Actual Failures with Expected Failures + run: | + echo "Expected Failures:" + cat expected_failures_sorted.txt + echo "" + echo "Actual Failures:" + cat actual_failures_sorted.txt + echo "" + + # Identify unexpected failures + unexpected_failures=$(comm -13 expected_failures_sorted.txt actual_failures_sorted.txt) + + # Identify missing expected failures + missing_failures=$(comm -23 expected_failures_sorted.txt actual_failures_sorted.txt) + + # Initialize exit code + exit_code=0 + + if [ -n "$unexpected_failures" ]; then + echo "Unexpected test failures detected:" + echo "$unexpected_failures" + exit_code=1 + fi + + if [ -n "$missing_failures" ]; then + echo "Expected test failures that did not occur (they passed):" + echo "$missing_failures" + exit_code=1 + fi + + if [ $exit_code -eq 0 ]; then + echo "All failed tests are expected, and all expected failures have occurred." + else + echo "Verification failed: There are unexpected or missing test failures." + fi + + exit $exit_code \ No newline at end of file