Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(actions): add docs propagation action for columns #130

Merged
merged 10 commits into from
Sep 11, 2024

Conversation

shirshanka
Copy link
Collaborator

@shirshanka shirshanka commented Sep 7, 2024

This PR introduces a new DocPropagationAction for column-level documentation propagation.

Key features include:

  1. Automated propagation of documentation changes to downstream and sibling schema fields
  2. Configuration options to enable/disable propagation at global and column levels
  3. Support for ADD, MODIFY, and REMOVE operations on documentation
  4. Integration with DataHub's event system, processing EntityChangeEvents and MetadataChangeLogs
  5. Ability to fetch and apply global settings from DataHub
  6. Detailed logging and statistics tracking for monitoring and debugging

As seen in DataHub townhall - https://youtu.be/hdqgjxiW-zI?t=2051

Copy link

github-actions bot commented Sep 7, 2024

Unit Test Results (build & test)

63 tests  ±0   63 ✅ ±0   4s ⏱️ ±0s
 1 suites ±0    0 💤 ±0 
 1 files   ±0    0 ❌ ±0 

Results for commit 6fc4fb6. ± Comparison against base commit c5f6dba.

♻️ This comment has been updated with latest results.

# No need to propagate to self
return None

if not dataset_urn.startswith("urn:li:dataset"):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we have utils e.g. from datahub.metadata.urns import Dataset or guess_entity_type - this startswith check feels a bit too hardcoded

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i have seen this in a code quite a bit recently. i agree with avoiding raw strings for urn checks. seems like python sdk should have internal utilities for all of this

smoke-test/tests/consistency_utils.py Outdated Show resolved Hide resolved
@shirshanka shirshanka merged commit 36aa93e into acryldata:main Sep 11, 2024
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants