-
Notifications
You must be signed in to change notification settings - Fork 124
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
dbt-athena-community support #203
Conversation
Local run for info:
|
Re: #133 (comment) (need for
|
Note - dbt-labs/dbt-athena#161 effectively added a large subset of external tables functionality in dbt-athena itself. Might be worth trying to refactor that and utilise it to cut down on the duplicated logic in here |
@brabster what's needed to get this PR approved? I'm happy to contribute. |
@aidan-o-boyle-kroo hi there! I've just pulled this, it is still working on dbt-athena-community 1.4.6 and works against latest 1.6.1 too. $ ATHENA_TEST_DBNAME=AwsDataCatalog AWS_REGION=eu-west-2 ATHENA_TEST_BUCKET=my-redacted_bucket ATHENA_TEST_WORKGROUP=primary ./run_test.sh athena
Setting up virtual environment for dbt-athena
Changing working directory: integration_tests
Starting integration tests
19:25:28 Running with dbt=1.6.3
19:25:29 Installing ../
19:25:29 Installed from <local @ ../>
19:25:29 Installing dbt-labs/dbt_utils
19:25:29 Installed from version 0.8.0
19:25:29 Updated version available: 1.1.1
19:25:29
19:25:29 Updates available for packages: ['dbt-labs/dbt_utils']
Update your versions in packages.yml, then run dbt deps
19:25:32 Running with dbt=1.6.3
19:25:32 Registered adapter: athena=1.6.1
19:25:32 Unable to do partial parsing because config vars, config profile, or config target have changed
19:25:34 Found 1 seed, 2 tests, 5 sources, 0 exposures, 0 metrics, 619 macros, 0 groups, 0 semantic models
19:25:34
19:25:37 Concurrency: 1 threads (target='athena')
19:25:37
19:25:37 1 of 1 START seed file dbt_external_tables_integration_tests_athena.people ..... [RUN]
19:25:48 1 of 1 OK loaded seed file dbt_external_tables_integration_tests_athena.people . [CREATE 200 in 11.03s]
19:25:48
19:25:48 Finished running 1 seed in 0 hours 0 minutes and 14.18 seconds (14.18s).
19:25:48
19:25:48 Completed successfully
19:25:48
19:25:48 Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1
19:25:51 Running with dbt=1.6.3
19:25:51 Registered adapter: athena=1.6.1
19:25:51 Found 1 seed, 2 tests, 5 sources, 0 exposures, 0 metrics, 619 macros, 0 groups, 0 semantic models
19:25:51 No prep necessary, skipping
19:25:54 Running with dbt=1.6.3
19:25:54 Registered adapter: athena=1.6.1
19:25:54 Unable to do partial parsing because config vars, config profile, or config target have changed
19:25:57 Found 1 seed, 2 tests, 5 sources, 0 exposures, 0 metrics, 619 macros, 0 groups, 0 semantic models
19:25:57 1 of 4 START external source dbt_external_tables_integration_tests_athena.people_csv_unpartitioned
19:25:58 1 of 4 (1) drop table if exists `AwsDataCatalog`.`dbt_external_tables_integration_tests_ath...
19:25:59 1 of 4 (1) OK -1
19:25:59 1 of 4 (2) create external table `AwsDataCatalog`.`dbt_external_tables_integration_tests_at...
19:26:01 1 of 4 (2) OK -1
19:26:01 2 of 4 START external source dbt_external_tables_integration_tests_athena.people_csv_partitioned
19:26:02 2 of 4 (1) drop table if exists `AwsDataCatalog`.`dbt_external_tables_integration_tests_ath...
19:26:03 2 of 4 (1) OK -1
19:26:03 2 of 4 (2) create external table `AwsDataCatalog`.`dbt_external_tables_integration_tests_at...
19:26:05 2 of 4 (2) OK -1
19:26:05 2 of 4 (3) alter table `AwsDataCatalog`.`dbt_external_tables_integration_tests_athena`.`peo...
19:26:08 2 of 4 (3) OK -1
19:26:08 3 of 4 START external source dbt_external_tables_integration_tests_athena.people_csv_multipartitioned
19:26:10 3 of 4 (1) drop table if exists `AwsDataCatalog`.`dbt_external_tables_integration_tests_ath...
19:26:11 3 of 4 (1) OK -1
19:26:11 3 of 4 (2) create external table `AwsDataCatalog`.`dbt_external_tables_integration_tests_at...
19:26:12 3 of 4 (2) OK -1
19:26:12 3 of 4 (3) alter table `AwsDataCatalog`.`dbt_external_tables_integration_tests_athena`.`peo...
19:26:49 3 of 4 (3) OK -1
19:26:49 3 of 4 (4) alter table `AwsDataCatalog`.`dbt_external_tables_integration_tests_athena`.`peo...
19:27:25 3 of 4 (4) OK -1
19:27:25 3 of 4 (5) alter table `AwsDataCatalog`.`dbt_external_tables_integration_tests_athena`.`peo...
19:27:59 3 of 4 (5) OK -1
19:27:59 4 of 4 START external source dbt_external_tables_integration_tests_athena.people_csv_multipartitioned_hive_compatible
19:27:59 4 of 4 (1) drop table if exists `AwsDataCatalog`.`dbt_external_tables_integration_tests_ath...
19:28:00 4 of 4 (1) OK -1
19:28:00 4 of 4 (2) create external table `AwsDataCatalog`.`dbt_external_tables_integration_tests_at...
19:28:01 4 of 4 (2) OK -1
19:28:01 4 of 4 (3) msck repair table `AwsDataCatalog`.`dbt_external_tables_integration_tests_athena...
19:28:04 4 of 4 (3) OK -1
19:28:07 Running with dbt=1.6.3
19:28:07 Registered adapter: athena=1.6.1
19:28:07 Unable to do partial parsing because config vars, config profile, or config target have changed
19:28:09 Found 1 seed, 2 tests, 5 sources, 0 exposures, 0 metrics, 619 macros, 0 groups, 0 semantic models
19:28:09 1 of 4 START external source dbt_external_tables_integration_tests_athena.people_csv_unpartitioned
19:28:10 1 of 4 SKIP
19:28:10 2 of 4 START external source dbt_external_tables_integration_tests_athena.people_csv_partitioned
19:28:12 2 of 4 (1) alter table `AwsDataCatalog`.`dbt_external_tables_integration_tests_athena`.`peo...
19:28:16 2 of 4 (1) OK -1
19:28:16 3 of 4 START external source dbt_external_tables_integration_tests_athena.people_csv_multipartitioned
19:28:17 3 of 4 (1) alter table `AwsDataCatalog`.`dbt_external_tables_integration_tests_athena`.`peo...
19:28:53 3 of 4 (1) OK -1
19:28:53 3 of 4 (2) alter table `AwsDataCatalog`.`dbt_external_tables_integration_tests_athena`.`peo...
19:29:29 3 of 4 (2) OK -1
19:29:29 3 of 4 (3) alter table `AwsDataCatalog`.`dbt_external_tables_integration_tests_athena`.`peo...
19:30:03 3 of 4 (3) OK -1
19:30:03 4 of 4 START external source dbt_external_tables_integration_tests_athena.people_csv_multipartitioned_hive_compatible
19:30:04 4 of 4 (1) msck repair table `AwsDataCatalog`.`dbt_external_tables_integration_tests_athena...
19:30:06 4 of 4 (1) OK -1
19:30:09 Running with dbt=1.6.3
19:30:09 Registered adapter: athena=1.6.1
19:30:09 Found 1 seed, 2 tests, 5 sources, 0 exposures, 0 metrics, 619 macros, 0 groups, 0 semantic models
19:30:09
19:30:10 Concurrency: 1 threads (target='athena')
19:30:10
19:30:11 1 of 2 START test dbt_utils_source_equality_athena_external_people_csv_partitioned_id__first_name__last_name__email__ref_people_ [RUN]
19:30:14 1 of 2 PASS dbt_utils_source_equality_athena_external_people_csv_partitioned_id__first_name__last_name__email__ref_people_ [PASS in 3.83s]
19:30:14 2 of 2 START test dbt_utils_source_equality_athena_external_people_csv_unpartitioned_id__first_name__last_name__email__ref_people_ [RUN]
19:30:17 2 of 2 PASS dbt_utils_source_equality_athena_external_people_csv_unpartitioned_id__first_name__last_name__email__ref_people_ [PASS in 2.77s]
19:30:17
19:30:17 Finished running 2 tests in 0 hours 0 minutes and 7.89 seconds (7.89s).
19:30:17
19:30:17 Completed successfully
19:30:17
19:30:17 Done. PASS=2 WARN=0 ERROR=0 SKIP=0 TOTAL=2 I've love to get it merged, will remove draft label. Main concerns would be:
I am depending on my fork for multiple projects now - you can kick tyres and check it's working for you that way I guess. |
could we mock athena ? https://github.com/getmoto/moto/blob/master/IMPLEMENTATION_COVERAGE.md#athena |
We could - I'm not sure how effective a test that would be, and I'm not sure what the maintainers need in order to merge the PR. @jeremyyeo can you advise on what we'd need to do to get this PR merged in? 🙇♂️ |
This PR has been marked as Stale because it has been open with no activity as of late. If you would like the PR to remain open, please comment on the PR or else it will be closed in 7 days. |
I'd also like to contribute whatever it takes to get this merged. This would be really helpful for our team. |
@dataders who should we add as reviewer to merge this one? 🙏🏻 |
…ome additional docs
I've just set it up again with latest dbt-athena-community against my personal AWS account. All appears to still be working fine, integration tests run and pass. I've added an example of minimal IAM permissions and defaulted a config value to assist with any future test automation setup. Also checked that the implementation does its own drop-if logic and so doesn't appear to inherit any inappropriate housekeeping behaviour from the adapter.
|
This PR has been marked as Stale because it has been open with no activity as of late. If you would like the PR to remain open, please comment on the PR or else it will be closed in 7 days. |
Although we are closing this PR as stale, it can still be reopened to continue development. Just add a comment to notify the maintainers. |
Description & motivation
resolves #274
PR based on #133
Checklist