Add Github workflow to populate the persistent source schema #715

plypaul · 2023-08-08T18:13:20Z

Description

When the definitions of the tables used in tests change, the persistent source schema needs to be repopulated manually. This PR adds a Github workflow to do this that can be trigged with the Reload Test Data in SQL Engines label.

WilliamDee · 2023-08-08T20:57:43Z

Makefile

 .PHONY: test-bigquery
 test-bigquery:
 	hatch -v run bigquery-env:pytest -vv -n $(PARALLELISM) $(ADDITIONAL_PYTEST_OPTIONS) metricflow/test/

+.PHONY: populate-persistent-source-schema-bigquery
+populate-persistent-source-schema:


add -bigquery suffix

Nice catch. Apparently, if the name it wrong, make still succeeds with nothing to do

plypaul · 2023-09-19T05:58:44Z

Updated logic to run the workflow when the label is set, and remove the label when workflow finishes. Hopefully Github's concurrency controls work as expected, and this seems good enough for a try.

tlento

Sure, let's give it a go. Please address inline comments before merge.

tlento · 2023-09-19T17:22:55Z

.github/workflows/cd-sql-engine-populate-persistent-source-schema.yaml

+on:
+  pull_request:
+    types: [labeled]


Please add workflow_dispatch so we don't have to label random PRs if something gets borked off main.

Added, but iterating on this PR has been a huge pain as it can't be tested locally.

tlento · 2023-09-19T17:25:34Z

.github/workflows/cd-sql-engine-populate-persistent-source-schema.yaml

+jobs:
+  snowflake-populate:
+    environment: DW_INTEGRATION_TESTS
+    if: github.event.action == 'labeled' && github.event.label.name == 'Reload Test Data in SQL Engines'


This may need an update for workflow_dispatch to fire, but it should be simple enough.

tlento · 2023-09-19T17:30:50Z

Makefile

 .PHONY: test-bigquery
 test-bigquery:
 	hatch -v run bigquery-env:pytest -vv -n $(PARALLELISM) $(ADDITIONAL_PYTEST_OPTIONS) metricflow/test/

+.PHONY: populate-persistent-source-schema-bigquery
+populate-persistent-source-schema-bigquery:
+	hatch -v run bigquery-env:pytest -vv $(ADDITIONAL_PYTEST_OPTIONS) $(POPULATE_PERSISTENT_SOURCE_SCHEMA)


I think it's better to define a make command that just works, instead of one that relies on someone knowing that they have to set up the ADDITIONAL_PYTEST_OPTIONS=--use-persistent-source-schema environment flag.

I just know I'm going to perpetually be in a loop of:

make populate-persistent....databricks <fail> make -e "ADDITIONAL_PYTEST_OPTIONS=--use-persistent-source-schema" populate-persistent...databricks

That's fair. Updated.

tlento · 2023-09-19T17:31:49Z

.github/workflows/cd-sql-engine-populate-persistent-source-schema.yaml

+  group: POPULATE_PERSISTENT_SOURCE_SCHEMA
+  cancel-in-progress: true


I am very curious to see how this works.

tlento · 2023-09-19T17:32:18Z

.github/workflows/cd-sql-engine-populate-persistent-source-schema.yaml

+          additional-pytest-options: ${{ env.ADDITIONAL_PYTEST_OPTIONS }}
+          make-target: "populate-persistent-source-schema-databricks"
+
+  remove-label:


Oh if this works we should TOTALLY add it to the sql engine tests...... I've got some updates I want to make over there so I can do that once this is in.

tlento · 2023-09-19T17:35:15Z

.github/workflows/cd-sql-engine-populate-persistent-source-schema.yaml

+
+name: Reload Test Data in SQL Engines
+
+# We don't want multiple workflows trying to create the same table.


Yeah I guess if this ever happens in different schemas the last one in will be wrong anyway.

plypaul added the Skip Changelog label Aug 8, 2023

cla-bot bot added the cla:yes label Aug 8, 2023

plypaul added the Reload Test Data in SQL Engines Should be run when test data changes label Aug 8, 2023

plypaul temporarily deployed to DW_INTEGRATION_TESTS August 8, 2023 18:45 — with GitHub Actions Inactive

plypaul force-pushed the plypaul--53--populate-persistent-source-schema-gha branch from e283ae8 to b7950bf Compare August 8, 2023 20:05

plypaul added Reload Test Data in SQL Engines Should be run when test data changes and removed Reload Test Data in SQL Engines Should be run when test data changes labels Aug 8, 2023

plypaul force-pushed the plypaul--53--populate-persistent-source-schema-gha branch from b7950bf to a739d97 Compare August 8, 2023 20:10

plypaul added Reload Test Data in SQL Engines Should be run when test data changes and removed Reload Test Data in SQL Engines Should be run when test data changes labels Aug 8, 2023

plypaul temporarily deployed to DW_INTEGRATION_TESTS August 8, 2023 20:12 — with GitHub Actions Inactive

plypaul force-pushed the plypaul--53--populate-persistent-source-schema-gha branch from a739d97 to 7ba0977 Compare August 8, 2023 20:38

plypaul added Reload Test Data in SQL Engines Should be run when test data changes and removed Reload Test Data in SQL Engines Should be run when test data changes labels Aug 8, 2023

plypaul had a problem deploying to DW_INTEGRATION_TESTS August 8, 2023 20:41 — with GitHub Actions Error

plypaul force-pushed the plypaul--53--populate-persistent-source-schema-gha branch from 7ba0977 to c9d00fe Compare August 8, 2023 20:44

plypaul added Reload Test Data in SQL Engines Should be run when test data changes and removed Reload Test Data in SQL Engines Should be run when test data changes labels Aug 8, 2023

plypaul temporarily deployed to DW_INTEGRATION_TESTS August 8, 2023 20:44 — with GitHub Actions Inactive

plypaul marked this pull request as ready for review August 8, 2023 20:55

plypaul requested a review from WilliamDee August 8, 2023 20:55

plypaul force-pushed the plypaul--53--populate-persistent-source-schema-gha branch from c9d00fe to 9278608 Compare August 8, 2023 20:57

WilliamDee approved these changes Aug 8, 2023

View reviewed changes

plypaul force-pushed the plypaul--53--populate-persistent-source-schema-gha branch from 9278608 to 0f6a14c Compare August 8, 2023 23:02

plypaul added Reload Test Data in SQL Engines Should be run when test data changes and removed Reload Test Data in SQL Engines Should be run when test data changes labels Aug 8, 2023

plypaul temporarily deployed to DW_INTEGRATION_TESTS August 8, 2023 23:03 — with GitHub Actions Inactive

plypaul temporarily deployed to DW_INTEGRATION_TESTS September 19, 2023 02:20 — with GitHub Actions Inactive

github-actions bot removed the Reload Test Data in SQL Engines Should be run when test data changes label Sep 19, 2023

plypaul force-pushed the plypaul--53--populate-persistent-source-schema-gha branch 2 times, most recently from 7d3b842 to 0977192 Compare September 19, 2023 05:56

plypaul requested a review from tlento September 19, 2023 05:58

tlento approved these changes Sep 19, 2023

View reviewed changes

plypaul force-pushed the plypaul--53--populate-persistent-source-schema-gha branch from 0977192 to dc1ad0d Compare September 19, 2023 19:21

plypaul added the Reload Test Data in SQL Engines Should be run when test data changes label Sep 19, 2023

plypaul force-pushed the plypaul--53--populate-persistent-source-schema-gha branch from dc1ad0d to 3f88ff4 Compare September 19, 2023 19:26

plypaul added Reload Test Data in SQL Engines Should be run when test data changes and removed Reload Test Data in SQL Engines Should be run when test data changes labels Sep 19, 2023

plypaul had a problem deploying to DW_INTEGRATION_TESTS September 19, 2023 19:26 — with GitHub Actions Failure

plypaul had a problem deploying to DW_INTEGRATION_TESTS September 19, 2023 19:27 — with GitHub Actions Failure

Add Github workflow to populate the persistent source schema.

609c080

plypaul force-pushed the plypaul--53--populate-persistent-source-schema-gha branch from 3f88ff4 to 609c080 Compare September 19, 2023 19:33

plypaul added Reload Test Data in SQL Engines Should be run when test data changes and removed Reload Test Data in SQL Engines Should be run when test data changes labels Sep 19, 2023

plypaul temporarily deployed to DW_INTEGRATION_TESTS September 19, 2023 19:33 — with GitHub Actions Inactive

github-actions bot removed the Reload Test Data in SQL Engines Should be run when test data changes label Sep 19, 2023

plypaul merged commit e285189 into main Sep 19, 2023
22 checks passed

plypaul deleted the plypaul--53--populate-persistent-source-schema-gha branch September 19, 2023 19:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Github workflow to populate the persistent source schema #715

Add Github workflow to populate the persistent source schema #715

plypaul commented Aug 8, 2023 •

edited

Loading

WilliamDee Aug 8, 2023

plypaul Aug 8, 2023

plypaul commented Sep 19, 2023

tlento left a comment

tlento Sep 19, 2023

plypaul Sep 19, 2023

tlento Sep 19, 2023

tlento Sep 19, 2023 •

edited

Loading

plypaul Sep 19, 2023

tlento Sep 19, 2023

tlento Sep 19, 2023

tlento Sep 19, 2023

		group: POPULATE_PERSISTENT_SOURCE_SCHEMA
		cancel-in-progress: true


		name: Reload Test Data in SQL Engines

		# We don't want multiple workflows trying to create the same table.

Add Github workflow to populate the persistent source schema #715

Add Github workflow to populate the persistent source schema #715

Conversation

plypaul commented Aug 8, 2023 • edited Loading

Description

Choose a reason for hiding this comment

Choose a reason for hiding this comment

plypaul commented Sep 19, 2023

tlento left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tlento Sep 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

plypaul commented Aug 8, 2023 •

edited

Loading

tlento Sep 19, 2023 •

edited

Loading