[Feature] Standardized Marketing Models

CHANGELOG.md

@@ -1,3 +1,33 @@
+# dbt_hubspot v0.19.0
+[PR #146](https://github.com/fivetran/dbt_hubspot/pull/146) includes the following updates:
+
+## 🚨 Breaking Changes 🚨
+> New fields are being brought in and a field is changed, so we recommend a `--full-refresh` after upgrading.
+
+- Now brings in unsubscribed metrics into the following end models to ensure they compile in the new standardized marketing models (if they are not available, they are cast as nulls):
+  - `hubspot__contacts`: `total_unsubscribes`
+  - `hubspot__email_campaigns`: `total_unsubscribes`
+  - `hubspot__email_sends`: `unsubscribes`, `was_unsubscribed`
+- Corrected misspelling of `was_unsubscribed` in `hubspot__email_sends`. 
+- Removed `unsubscribes` from `default_metrics` in the `adjust_email_metrics` macro, as unsubscribes is not a default email event. 
+
+## Feature Updates
+- Addition of the following standardized marketing models.
+
+| **model**                | **description**                                                                                                      |
+| ------------------------ | -------------------------------------------------------------------------------------------------------------------- |
+| [hubspot__marketing_campaigns_standardized](https://fivetran.github.io/dbt_hubspot/#!/model/model.hubspot.hubspot__marketing_campaigns_standardized)   |   This model constructs a comprehensive, denormalized analytical table that enables reporting on campaign metrics from your marketing platform. It’s designed to align with the schema of the `*_marketing_campaigns_standardized` model found in Hubspot, Marketo, Iterable, Klaviyo, Mailchimp, Qualtrics, Salesforce Marketing Cloud, and Twilio.  To see the kinds of insights this model can generate, explore example visualizations in the [Fivetran Marketing Model Streamlit App](https://fivetran-marketing-model.streamlit.app/). Visit the app for more details.    |
+| [hubspot__marketing_contacts_standardized](https://fivetran.github.io/dbt_hubspot/#!/model/model.hubspot.hubspot__marketing_contacts_standardized)   |   This model constructs a comprehensive, denormalized analytical table that enables reporting on contacts from your marketing platform. It’s designed to align with the schema of the `*_marketing_contacts_standardized` model found in Hubspot, Marketo, Iterable, Klaviyo, Mailchimp, Qualtrics, Salesforce Marketing Cloud, and Twilio. To see the kinds of insights this model can generate, explore example visualizations in the [Fivetran Marketing Model Streamlit App](https://fivetran-marketing-model.streamlit.app/). Visit the app for more details.     |
+| [hubspot__marketing_events_standardized](https://fivetran.github.io/dbt_hubspot/#!/model/model.hubspot.hubspot__marketing_events_standardized)        |   This model constructs a comprehensive, denormalized analytical table that enables reporting on events from your marketing platform. It’s designed to align with the schema of the `*_marketing_events_standardized` model found in Hubspot, Marketo, Iterable, Klaviyo, Mailchimp, Qualtrics, Salesforce Marketing Cloud, and Twilio. To see the kinds of insights this model can generate, explore example visualizations in the [Fivetran Marketing Model Streamlit App](https://fivetran-marketing-model.streamlit.app/). Visit the app for more details.    |
+
+
+- These models construct comprehensive, denormalized analytical tables that enables reporting on campaigns, contacts, and events from your marketing platform. It’s designed to align with the schema of the `*__line_item_enhanced` model found in Hubspot, Marketo, Iterable, Klaviyo, Mailchimp, Qualtrics, Salesforce Marketing Cloud, and Twilio, offering standardized reporting across various marketing platforms. To see the kinds of insights this model can generate, explore example visualizations in the [Fivetran Marketing Model Streamlit App](https://fivetran-marketing-model.streamlit.app/). Visit the app for more details.
+  -  This model is currently disabled by default. You may enable it by setting the `hubspot__standardized_marketing_model_enabled` as `true` in your `dbt_project.yml`.
+
+
+## Under the Hood:
+- Added consistency tests within `integration_tests` for the `hubspot__contacts`, `hubspot__email_campaigns` and `hubspot__email_sends` models.
+
 # dbt_hubspot v0.18.0
 
 [PR #144](https://github.com/fivetran/dbt_hubspot/pull/144) includes the following updates:

README.md

@@ -41,6 +41,14 @@ The following table provides a detailed list of all models materialized within t
 | [hubspot__email_event_*](https://fivetran.github.io/dbt_hubspot/#!/model/model.hubspot.hubspot__email_event_bounce)   | Each record represents an email event in Hubspot, joined with relevant tables to make them analysis-ready.           |
 | [hubspot__email_sends](https://fivetran.github.io/dbt_hubspot/#!/model/model.hubspot.hubspot__email_sends)     | Each record represents a sent email in Hubspot, enriched with metrics about opens, clicks, and other email activity. |
 | [hubspot__engagement_*](https://fivetran.github.io/dbt_hubspot/#!/model/model.hubspot.hubspot__engagement_calls)    | Each record represents an engagement event in Hubspot, joined with relevant tables to make them analysis-ready.      |
+| [hubspot__marketing_campaigns_standardized](https://fivetran.github.io/dbt_hubspot/#!/model/model.hubspot.hubspot__marketing_campaigns_standardized)   |   This model constructs a comprehensive, denormalized analytical table that enables reporting on campaign metrics from your marketing platform. It’s designed to align with the schema of the `*_marketing_campaigns_standardized` model found in Hubspot, Marketo, Iterable, Klaviyo, Mailchimp, Qualtrics, Salesforce Marketing Cloud, and Twilio.  To see the kinds of insights this model can generate, explore example visualizations in the [Fivetran Marketing Model Streamlit App](https://fivetran-marketing-model.streamlit.app/). Visit the app for more details.    |
+| [hubspot__marketing_contacts_standardized](https://fivetran.github.io/dbt_hubspot/#!/model/model.hubspot.hubspot__marketing_contacts_standardized)   |   This model constructs a comprehensive, denormalized analytical table that enables reporting on contacts from your marketing platform. It’s designed to align with the schema of the `*_marketing_contacts_standardized` model found in Hubspot, Marketo, Iterable, Klaviyo, Mailchimp, Qualtrics, Salesforce Marketing Cloud, and Twilio. To see the kinds of insights this model can generate, explore example visualizations in the [Fivetran Marketing Model Streamlit App](https://fivetran-marketing-model.streamlit.app/). Visit the app for more details.     |
+| [hubspot__marketing_events_standardized](https://fivetran.github.io/dbt_hubspot/#!/model/model.hubspot.hubspot__marketing_events_standardized)        |   This model constructs a comprehensive, denormalized analytical table that enables reporting on events from your marketing platform. It’s designed to align with the schema of the `*_marketing_events_standardized` model found in Hubspot, Marketo, Iterable, Klaviyo, Mailchimp, Qualtrics, Salesforce Marketing Cloud, and Twilio. To see the kinds of insights this model can generate, explore example visualizations in the [Fivetran Marketing Model Streamlit App](https://fivetran-marketing-model.streamlit.app/). Visit the app for more details.    |
+
+
+## Example Visualizations
+Curious what these models can do? Check out example visualizations from the [hubspot__marketing_campaigns_standardized](https://fivetran.github.io/dbt_hubspot/#!/model/model.hubspot.hubspot__marketing_campaigns_standardized) model in the [Fivetran Marketing Model Streamlit App](https://fivetran-marketing-model.streamlit.app/), and see how you can use these models in your own reporting. Below is a screenshot of an example report—explore the app for more.
+
 <!--section-end-->
 
 # 🎯 How do I use the dbt package?
@@ -75,7 +83,7 @@ Include the following hubspot package version in your `packages.yml` file:
 ```yaml
 packages:
   - package: fivetran/hubspot
-    version: [">=0.18.0", "<0.19.0"] # we recommend using ranges to capture non-breaking changes automatically
+    version: [">=0.19.0", "<0.20.0"] # we recommend using ranges to capture non-breaking changes automatically
 
 ```
 Do **NOT** include the `hubspot_source` package in this file. The transformation package itself has a dependency on it and will install the source package as well.
@@ -155,6 +163,20 @@ vars:
 ```
 ## (Optional) Step 5: Additional configurations
 
+### Enabling Standardized Marketing Models
+This package contains the following standardized models:
+
+These models are designed to align with the schema of other models found in similar marketing packages like Marketo, Iterable, Klaviyo, Mailchimp, Qualtrics, Salesforce Marketing Cloud, and Twilio to offering standardized reporting across various billing platforms.
+
+These models constructs comprehensive, denormalized analytical table that enables reporting on key campaign, contact, and event metrics from your marketing platform.
+
+ To see the kinds of insights this model can generate, explore example visualizations in the [Fivetran Marketing Model Streamlit App](https://fivetran-marketing-model.streamlit.app/). For the time being, this model is disabled by default. If you would like to enable this model you will need to adjust the `hubspot__standardized_marketing_model_enabled` variable to be `true` within your `dbt_project.yml`:
+
+```yml
+vars:
+  hubspot__standardized_marketing_model_enabled: true # false by default.
+```
+
 ### Configure email metrics
 This package allows you to specify which email metrics (total count and total unique count) you would like to be calculated for specified fields within the `hubspot__email_campaigns` model. By default, the `email_metrics` variable below includes all the shown fields. If you would like to remove any field metrics from the final model, you may copy and paste the below snippet within your root `dbt_project.yml` and remove any fields you want to be ignored in the final model.
 

dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'hubspot'
-version: '0.18.0'
+version: '0.19.0'
 
 config-version: 2
 require-dbt-version: [">=1.3.0", "<2.0.0"]

integration_tests/ci/sample.profiles.yml

@@ -16,13 +16,13 @@ integration_tests:
       pass: "{{ env_var('CI_REDSHIFT_DBT_PASS') }}"
       dbname: "{{ env_var('CI_REDSHIFT_DBT_DBNAME') }}"
       port: 5439
-      schema: hubspot_integration_tests_57
+      schema: hubspot_integration_tests_60
       threads: 8
     bigquery:
       type: bigquery
       method: service-account-json
       project: 'dbt-package-testing'
-      schema: hubspot_integration_tests_57
+      schema: hubspot_integration_tests_60
       threads: 8
       keyfile_json: "{{ env_var('GCLOUD_SERVICE_KEY') | as_native }}"
     snowflake:
@@ -33,7 +33,7 @@ integration_tests:
       role: "{{ env_var('CI_SNOWFLAKE_DBT_ROLE') }}"
       database: "{{ env_var('CI_SNOWFLAKE_DBT_DATABASE') }}"
       warehouse: "{{ env_var('CI_SNOWFLAKE_DBT_WAREHOUSE') }}"
-      schema: hubspot_integration_tests_57
+      schema: hubspot_integration_tests_60
       threads: 8
     postgres:
       type: postgres
@@ -42,13 +42,13 @@ integration_tests:
       pass: "{{ env_var('CI_POSTGRES_DBT_PASS') }}"
       dbname: "{{ env_var('CI_POSTGRES_DBT_DBNAME') }}"
       port: 5432
-      schema: hubspot_integration_tests_57
+      schema: hubspot_integration_tests_60
       threads: 8
     databricks:
       catalog: "{{ env_var('CI_DATABRICKS_DBT_CATALOG') }}"
       host: "{{ env_var('CI_DATABRICKS_DBT_HOST') }}"
       http_path: "{{ env_var('CI_DATABRICKS_DBT_HTTP_PATH') }}"
-      schema: hubspot_integration_tests_57
+      schema: hubspot_integration_tests_60
       threads: 8
       token: "{{ env_var('CI_DATABRICKS_DBT_TOKEN') }}"
       type: databricks

integration_tests/dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'hubspot_integration_tests'
-version: '0.18.0'
+version: '0.19.0'
 
 profile: 'integration_tests'
 config-version: 2
@@ -9,13 +9,17 @@ models:
   # +schema: "hubspot_{{ var('directed_schema','dev') }}" ## To be used for validation testing
 
 vars:
-  hubspot_schema: hubspot_integration_tests_57
+  hubspot_schema: hubspot_integration_tests_60
   hubspot_service_enabled: true
   # hubspot_sales_enabled: true # enable when generating docs
   # hubspot_marketing_enabled: true # enable when generating docs
   # hubspot_contact_merge_audit_enabled: true # enable when generating docs
   # hubspot_using_all_email_events: true # enable when generating docs
   # hubspot_merged_deal_enabled: true # enable when generating docs 
+
+  # hubspot__standardized_marketing_model_enabled: true # enable when utilizing standardized marketing models + generating docs
+  # hubspot_email_event_status_change_enabled: true #enable when testing email unsubscribes
+
   hubspot_company_property_history_identifier: "company_property_history_data"
   hubspot_company_identifier: "company_data"
   hubspot_contact_identifier: "contact_data"

integration_tests/packages.yml

@@ -1,2 +1,2 @@
 packages:
-  - local: ../
+  - local: ../

integration_tests/tests/consistency/row_comparisons/consistency_campaigns.sql

@@ -0,0 +1,46 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+-- this test ensures the daily_activity end model matches the prior version
+with prod as (
+    select *
+    from {{ target.schema }}_hubspot_prod.hubspot__email_campaigns
+),
+
+dev as (
+    select *
+    from {{ target.schema }}_hubspot_dev.hubspot__email_campaigns
+),
+
+prod_not_in_dev as (
+    -- rows from prod not found in dev
+    select * from prod
+    except distinct
+    select * from dev
+),
+
+dev_not_in_prod as (
+    -- rows from dev not found in prod
+    select * from dev
+    except distinct
+    select * from prod
+),
+
+final as (
+    select
+        *,
+        'from prod' as source
+    from prod_not_in_dev
+
+    union all -- union since we only care if rows are produced
+
+    select
+        *,
+        'from dev' as source
+    from dev_not_in_prod
+)
+
+select *
+from final

integration_tests/tests/consistency/row_comparisons/consistency_contacts.sql

@@ -0,0 +1,46 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+-- this test ensures the daily_activity end model matches the prior version
+with prod as (
+    select *
+    from {{ target.schema }}_hubspot_prod.hubspot__contacts
+),
+
+dev as (
+    select *
+    from {{ target.schema }}_hubspot_dev.hubspot__contacts
+),
+
+prod_not_in_dev as (
+    -- rows from prod not found in dev
+    select * from prod
+    except distinct
+    select * from dev
+),
+
+dev_not_in_prod as (
+    -- rows from dev not found in prod
+    select * from dev
+    except distinct
+    select * from prod
+),
+
+final as (
+    select
+        *,
+        'from prod' as source
+    from prod_not_in_dev
+
+    union all -- union since we only care if rows are produced
+
+    select
+        *,
+        'from dev' as source
+    from dev_not_in_prod
+)
+
+select *
+from final

integration_tests/tests/consistency/row_comparisons/consistency_email_sends.sql

@@ -0,0 +1,46 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+-- this test ensures the daily_activity end model matches the prior version
+with prod as (
+    select *
+    from {{ target.schema }}_hubspot_prod.hubspot__email_sends
+),
+
+dev as (
+    select *
+    from {{ target.schema }}_hubspot_dev.hubspot__email_sends
+),
+
+prod_not_in_dev as (
+    -- rows from prod not found in dev
+    select * from prod
+    except distinct
+    select * from dev
+),
+
+dev_not_in_prod as (
+    -- rows from dev not found in prod
+    select * from dev
+    except distinct
+    select * from prod
+),
+
+final as (
+    select
+        *,
+        'from prod' as source
+    from prod_not_in_dev
+
+    union all -- union since we only care if rows are produced
+
+    select
+        *,
+        'from dev' as source
+    from dev_not_in_prod
+)
+
+select *
+from final

integration_tests/tests/consistency/row_counts/consistency_campaign_count.sql

@@ -0,0 +1,21 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+-- this test is to make sure the rows counts are the same between versions
+with prod as (
+    select count(*) as prod_rows
+    from {{ target.schema }}_hubspot_prod.hubspot__email_campaigns
+),
+
+dev as (
+    select count(*) as dev_rows
+    from {{ target.schema }}_hubspot_dev.hubspot__email_campaigns
+)
+
+-- test will return values and fail if the row counts don't match
+select *
+from prod
+join dev
+    on prod.prod_rows != dev.dev_rows

integration_tests/tests/consistency/row_counts/consistency_contact_count.sql

@@ -0,0 +1,21 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+-- this test is to make sure the rows counts are the same between versions
+with prod as (
+    select count(*) as prod_rows
+    from {{ target.schema }}_hubspot_prod.hubspot__contacts
+),
+
+dev as (
+    select count(*) as dev_rows
+    from {{ target.schema }}_hubspot_dev.hubspot__contacts
+)
+
+-- test will return values and fail if the row counts don't match
+select *
+from prod
+join dev
+    on prod.prod_rows != dev.dev_rows

integration_tests/tests/consistency/row_counts/consistency_email_send_count.sql

@@ -0,0 +1,21 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+-- this test is to make sure the rows counts are the same between versions
+with prod as (
+    select count(*) as prod_rows
+    from {{ target.schema }}_hubspot_prod.hubspot__email_sends
+),
+
+dev as (
+    select count(*) as dev_rows
+    from {{ target.schema }}_hubspot_dev.hubspot__email_sends
+)
+
+-- test will return values and fail if the row counts don't match
+select *
+from prod
+join dev
+    on prod.prod_rows != dev.dev_rows

macros/adjust_email_metrics.sql

@@ -2,7 +2,7 @@
 
 {% set base_cols = dbt_utils.get_filtered_columns_in_relation(ref(base_ref))|map('lower') %}
 
-{% set default_metrics = ['bounces', 'clicks', 'deferrals', 'deliveries', 'drops', 'forwards', 'opens', 'prints', 'spam_reports', 'unsubscribes'] %}
+{% set default_metrics = ['bounces', 'clicks', 'deferrals', 'deliveries', 'drops', 'forwards', 'opens', 'prints', 'spam_reports'] %}
 {% set email_metrics = var(var_name, default_metrics)|map('lower') %}
 
 {# Only keep metrics found in the base ref #}