Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add course licensing model. #2

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Conversation

anfbermudezme
Copy link
Contributor

@anfbermudezme anfbermudezme commented Sep 2, 2024

Ticket related

Description

This PR is dedicated to exploring and understanding the capabilities of dbt (Data Build Tool). The main objective of this PR is to familiarize ourselves with dbt's core features, including data modeling, testing, documentation, and deployment workflows. Key activities in this exploration include:

  • Setting Up dbt Environment: Configured the dbt environment to connect with our data warehouse.
  • Creating Sample Models: Developed initial data models to understand how dbt transforms raw data into clean datasets.
  • Implementing Tests: Added basic tests to validate data quality and integrity.
  • Generating Documentation: Explored dbt's documentation generation feature to see how it can enhance data lineage and transparency.
  • Experimenting with Macros: Created simple macros to test dbt’s ability to handle reusable SQL code and logic.
  • Deploying Models: Tested dbt's deployment process to understand how models are materialized in our data warehouse.

Purpose:

This PR is purely experimental and serves as a foundational step to evaluate how dbt can fit into our data pipeline architecture. Feedback and insights gained from this exploration will guide future decisions on adopting dbt as part of our data engineering toolkit.

Changes made

  • Add institution_administrators_count.sql filtering the data by the maximum time_last_dumped
  • Add schema.yml documenting the table created and the type of the data
  • Add source.yml adding the principal tables located in event_sink
  • Set the https://github.com/openedx/aspects-dbt.git version

Ho to test

  • Run you tutor local environment
  • Go to you superset local URL http://0.0.0.0:8088
  • Enter in SQLab
  • Verufy that the table is created, in this case institution_administrator_count.

Notes:

  • This PR is not intended for production use and will not impact any existing data pipelines.
  • Future PRs will build on these initial findings to optimize and scale dbt's use within our workflows.

@anfbermudezme anfbermudezme changed the title Vue/padv 1563 feat: add course licensing model. Sep 2, 2024
@anfbermudezme anfbermudezme force-pushed the vue/PADV-1563 branch 2 times, most recently from 3e57692 to 9747649 Compare September 2, 2024 20:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant