PR #150 includes the following updates:
-
Introduced a new
category
column to the following upstream models (see dbt_hubspot_source CHANGELOG notes). This association field differentiates records by either HUBSPOT_DEFINED (default label) or USER_DEFINED (custom label) and was introduced to the Hubspot connector in October 2024. See the connector release notes for more.stg_hubspot__deal_company
stg_hubspot__deal_contact
stg_hubspot__engagement_company
stg_hubspot__engagement_contact
stg_hubspot__engagement_deal
stg_hubspot__ticket_company
stg_hubspot__ticket_contact
stg_hubspot__ticket_deal
stg_hubspot__ticket_engagement
-
Since new columns were added upstream, a
--full-refresh
is needed.
- Updated the respective seed files in the integration_tests folder to property test for the new
category
field. - Updated seed files to make consistent with the seed files in
dbt_hubspot_source
.
PR #148 includes the following updates:
- Removed
hubspot__daily_ticket_history
,hubspot__tickets
from the quickstart.yml since these models may not always be present.
PR #147 includes the following updates:
⚠️ Since the following changes result in the table format changing, we recommend running a--full-refresh
after upgrading to this version to avoid possible incremental failures.
- We have made this a breaking change due to upstream changes that may alter your schema. While changes are made 'behind the scenes' to now allow models to successfully run with both
hubspot__pass_through_all_columns
andhubspot__<>_pass_through_columns
, this may be a breaking change due to leveraging theremove_duplicate_and_prefix_from_columns
macro. This is a breaking change because this macro can remove duplicate fields, resulting in an impact to your schema. For more information refer to the upstream dbt_hubspot_source v0.16.0 release notes.
- Updated seed data to include fields with special syntax in order to test the above changes.
PR #144 includes the following updates:
⚠️ Since the following changes result in the table format changing, we recommend running a--full-refresh
after upgrading to this version to avoid possible incremental failures.
-
For Databricks All-Purpose clusters, incremental models will now be materialized using the delta table format (previously parquet).
- Delta tables are generally more performant than parquet and are also more widely available for Databricks users. This will also prevent compilation issues on customers' managed tables.
-
For Databricks SQL Warehouses, incremental materialization will not be used due to the incompatibility of the
insert_overwrite
strategy.
- The
is_incremental_compatible
macro has been added and will returntrue
if the target warehouse supports our chosen incremental strategy.- This update was applied as there have been other Databricks runtimes discovered (ie. an endpoint and external runtime) which do not support the
insert_overwrite
incremental strategy used.
- This update was applied as there have been other Databricks runtimes discovered (ie. an endpoint and external runtime) which do not support the
- Added integration testing for Databricks SQL Warehouse.
- Added consistency tests for
hubspot__daily_ticket_history
.
PR #142 includes the following updates:
- Fixed the
fivetran_utils.enabled_vars
conditional by adding thehubspot_contact_list_member_enabled
variable in thehubspot__contact_lists
model. This solves for compilation errors when thecontact_list
source table is not synced in the destination. Ifhubspot_contact_list_member_enabled
istrue
,int_hubspot__email_metrics__by_contact_list
is brought in as a dependency, and ignored otherwise.
- Updated the
integration_tests/dbt_project.yml
variables to be global to ensure more effective testing of our seed data. - Updated
property_closed_date
andproperty_createdate
datatypes inticket_data
to cast as timestamp to fix datetime data type issues in BigQuery tests. - Updated the maintainer PR template to resemble the most up to date format.
- Removed the check docs GitHub Action as it is no longer necessary.
PR #140 includes the following updates:
- Included explicit datatype casts to
{{ dbt.type_string() }}
within the join ofcontact_merge_audit.vid_to_merge
tocontacts.contact_id
in theint_hubspot__contact_merge_adjust
model.- This update was required to address a bug where the IDs in the join would overflow to bigint or be interpreted as strings. This change ensures the join fields have matching datatypes.
PR #137 includes the following updates:
- The following are now dependent on the
hubspot_email_event_sent_enabled
variable being defined astrue
: -hubspot__contact_lists
email metric fields -hubspot__contact
email metric fields -hubspot__email_campaigns
model -int_hubspot__email_metrics__by_contact_list
model
Note: This is a breaking change for users who have the
hubspot_email_event_sent_enabled
variable set tofalse
as this update will change the above behaviors of the package. For all other users, these changes will not be breaking.
PR #135 includes the following updates:
- Added unique tests on
event_id
for event-based models,contact_id
forhubspot__contacts
,contact_list_id
forhubspot__contact_lists
,deal_id
forhubspot__deals
,deal_stage_id
forhubspot__deal_stages
, andcompany_id
forhubspot__companies
, on the condition the titular fields have not been deleted. We would advise running through these to ensure they work successfully.
- Updates the
hubspot__deal_stages
andhubspot__deals
models to remove stale deals that have been merged since into other deals. In addition we've introduced a new fieldmerged_deal_ids
that lists all historic merged deals for each deal.- Please note you must have the underlying
merged_deals
table to take advantage of the above mentioned merged deal update. This may be enabled by settinghubspot_merged_deal_enabled
to True (by default this will be False). Also,hubspot_sales_enabled
andhubspot_deal_enabled
must not be set to False (by default these are True). See Step 4 of the README for more details.
- Please note you must have the underlying
- The following adjustments have been made concerning the
hubspot_contact_enabled
variable:- The
email_events_joined()
macro now includes conditional logic to include contact information only if thehubspot_contact_enabled
variable is defined astrue
. - The
int_hubspot__email_metrics__by_contact_list
model is now dependent on thehubspot_contact_enabled
variable being defined astrue
. - The
hubspot__contact_lists
model now takes into consideration thehubspot_contact_enabled
variable being defined astrue
before joining in email metric information to the contact lists data.
- The
- Modified the config enabled logic within the
hubspot__email_sends
model to be dependent on thehubspot_email_event_sent_enabled
variable beingtrue
in addition to thehubspot_marketing_enabled
andhubspot_email_event_enabled
variables.
- Added quickstart.yml for Quickstart customers.
- Added
not_null
tests that were previously missing todeal_id
forhubspot__deals
anddeal_stage_id
forhubspot__deal_stages
.
PR #129 includes the following updates:
- Updated model
int_hubspot__deals_enhanced
so that fields from theowner
source are not included whenhubspot_owner_enabled
is set to false.
- Included auto-releaser GitHub Actions workflow to automate future releases.
PR #127 includes the following updates:
- Updated variables used to determine if engagements are enabled in
hubspot__contacts
to also check variablehubspot_engagement_contact_enabled
.
-
The following changes stem from changes made in the source package. See the source package v0.14.0 CHANGELOG entry for more details.
-
Added the following staging models, along with documentation and tests:
stg_hubspot__property
stg_hubspot__property_option
- These tables can be disabled by setting
hubspot_property_enabled: False
in your dbt_project.yml vars. See Step 4 of the README for more details.
-
When including a passthrough
property_hs_*
column, you now have the option to include the corresponding, human-readable labels.- The above-mentioned
property
tables are required for this feature. If you do not have them and have to disable them, unfortunately you will not be able to use this feature. - See the Adding property label section of the README for more details on how to enable this feature!
- We recommend being selective with the label columns you add. As you add more label columns, your run time will increase due to the underlying logic requirements.
- This update applies to models:
hubspot__companies
hubspot__contacts
hubspot__deals
hubspot__tickets
- The above-mentioned
- Within the source package the
created_at
andclosed_at
fields in the below mentioned staging models have been renamed tocreated_date
andclosed_date
respectively to be consistent with the source data. Additionally, this will ensure there are no duplicate column errors when passing through allproperty_*
columns, which could potentially conflict withproperty_created_at
orproperty_closed_at
. (PR #119)stg_hubspot__company
- Impacts
hubspot__companies
- Impacts
stg_hubspot__contact
- Impacts
hubspot__contacts
- Impacts
stg_hubspot__deal
- Impacts
hubspot__deals
- Impacts
stg_hubspot__ticket
- Impacts
hubspot__tickets
- Impacts
Introducing Service end models! These are disabled by default but can be enabled by setting hubspot_service_enabled
to true
(PR #123):
hubspot__tickets
- Docshubspot__daily_ticket_history
- Docs- See additional configurations for the history model in README
- Addition of the following variables to allow the disabling of the
*_property_history
models if they are not being leveraged. All variables aretrue
by default. (PR #122)hubspot_company_property_history_enabled
hubspot_contact_property_history_enabled
hubspot_deal_property_history_enabled
- Updates to the seed files and seed file configurations for the package integration tests to ensure updates are properly tested. (PR #122)
- This release will be a breaking change due to the removal of below dependencies.
- Removes the dependencies on dbt-expectations and dbt-date. (PR #118)
- Specifically we removed the
dbt_expectations.expect_column_values_to_be_unique
test that was used to validate uniqueness under given conditions for the primary keys among the end models. We will be working to replace this with a similar test. (PR #118)
- This release includes breaking changes as a result of upstream changes within the
v0.12.0
release of the dbt_hubspot_source package. Please see below for the relevant breaking change release notes from the source package. (PR #120) - The following models in the dbt_hubspot_source package now use a custom macro to remove the property_hs_ prefix in staging columns, while also preventing duplicates. If de-prefixed columns match existing ones (e.g.,
property_hs_meeting_outcome
vs.meeting_outcome
), the macro favors theproperty_hs_
field, aligning with the latest HubSpot API update. (PR #115)stg_hubspot__engagement_call
stg_hubspot__engagement_company
stg_hubspot__engagement_contact
stg_hubspot__engagement_deal
stg_hubspot__engagement_email
stg_hubspot__engagement_meeting
stg_hubspot__engagement_note
stg_hubspot__engagement_task
stg_hubspot__ticket
stg_hubspot__ticket_company
stg_hubspot__ticket_contact
stg_hubspot__ticket_deal
stg_hubspot__ticket_engagement
stg_hubspot__ticket_property_history
PR #114 includes the following updates:
This change is made breaking in part to updates applied to the upstream dbt_hubspot_source package following upgrades to ensure compatibility with the HubSpot v3 API updates. Please see below for the relevant upstream changes:
- Following the May 2023 connector update the HubSpot connector now syncs the below parent and child tables from the new v3 API. As a result the dependent fields and field names from the downstream staging models have changed depending on the fields available in your HubSpot data. Now the respective staging models will sync the required fields for the dbt_hubspot downstream transformations and all of your
property_hs_*
fields. Please be aware that theproperty_hs_*
will be truncated from the field name in the staging and downstream models. The impacted sources (and relevant staging models) are below:
- `ENGAGEMENT`
- `ENGAGEMENT_CALL`
- `ENGAGEMENT_COMPANY`
- `ENGAGEMENT_CONTACT`
- `ENGAGEMENT_DEAL`
- `ENGAGEMENT_EMAIL`
- `ENGAGEMENT_MEETING`
- `ENGAGEMENT_NOTE`
- `ENGAGEMENT_TASK`
- `TICKET`
- `TICKET_COMPANY`
- `TICKET_CONTACT`
- `TICKET_DEAL`
- `TICKET_ENGAGEMENT`
- `TICKET_PROPERTY_HISTORY`
-
Please note that while these changes are breaking, the package has been updated to ensure backwards compatibility with the pre HubSpot v3 API updates. As a result, you may see some
null
fields which are artifacts of the pre v3 API HubSpot version. Be sure to inspect the relevant field descriptions for an understanding of which fields remain for backwards compatibility purposes. These fields will be removed once all HubSpot connectors are upgraded to the v3 API. -
The
engagements_joined
macro has been adjusted to account for the HubSpot v3 API changes. In particular, the fields that used to only be available in theengagements
source are now available in the individualengagement_[type]
sources. As such, coalesce statements were added to ensure the correct populated fields are used following the join. Further, to avoid ambiguous columns adbt-utils.star
was added to remove the explicitly declared columns within the coalesce statements.- The coalesce is only included for backwards compatibility. This will be removed in a future release when all HubSpot connectors are on the latest API version.
- The
base_model
argument used for theengagements_joined
macro has been updated to be an explicitref
as opposed to the previously usedvar
within the below models:hubspot__engagement_calls
hubspot__engagement_emails
hubspot__engagement_meetings
hubspot__engagement_notes
hubspot__engagement_tasks
- This update was applied as the
var
result was producing inconsistent results during compile and runtime when leveraging thedbt-utils.star
macro. However, the explicitref
always provided consistent results.
- This update was applied as the
- As new fields were added in the v3 API updates, and old fields were removed, the documentation was updated to reflect the v3 API consistent fields. Please take note if you are still using the pre v3 API, you will find the following end models no longer have complete field documentation coverage:
hubspot__engagement_calls
hubspot__engagement_emails
hubspot__engagement_meetings
hubspot__engagement_notes
hubspot__engagement_tasks
Explicitly casts join fields (engagement_id
and deal_id
) in hubspot__deals
as the appropriate data types to avoid potential errors in joining. PR #113
These changes are made breaking due to changes in the source.
- Columns
updated_at
andcreated_at
were added to the following sources and their corresponding staging models in the source package:DEAL_PIPELINE
DEAL_PIPELINE_STAGE
TICKET_PIPELINE
TICKET_PIPELINE_STAGE
- As a result, the following columns have been added (#111):
- Model
hubspot__deals
:deal_pipeline_created_at
deal_pipeline_updated_at
- Model
hubspot__deal_stages
:deal_pipeline_stage_created_at
deal_pipeline_stage_updated_at
- Model
- Documentation has also been updated with these new columns. (#111)
- Updated README to include the variables
hubspot_ticket_deal_enabled
andhubspot_owner_enabled
. (#111)
- Modified the
unnest
logic in themerge_contacts
macro for Redshift users to reduce runtime of theint_hubspot__contact_merge_adjust
model. (#110) - Updated seed data for testing newly added columns. (#111)
- @kcraig-ats (#110)
See upstream hubspot_source
release notes here.
- A new variable was added
hubspot_using_all_email_events
to allow package users to remove filtered email events from thestg_hubspot__email_event
staging model as well as the relevant downstream reporting models. This is crucial for HubSpot users who greatly take advantage of marking events as filtered in order to provide accurate reporting. (#104)- The
hubspot_using_all_email_events
variable istrue
by default. Set the variable tofalse
to filter out specified email events in your staging and downstream models. (#104)
- The
- Introduced new macro
adjust_email_metrics
to prevent failures that may occur when certain tables are disabled or enabled. This macro removes any metrics that are unavailable from the defaultemail_metrics
list, ensuring that the models runs smoothly. It's worth noting that you can still manually set the email_metrics list as described in the README's (Optional) Step 5: Additional configurations. (#105)
- The
email_event_data.csv
seed file was updated to include events that are listed astrue
for filtered_events. This is to effectively test the above mentioned feature update. (#104) - Included
hubspot_using_all_email_events: false
as a variable declared in the finalrun_models.sh
step to ensure our integration tests gain coverage over this new feature and variable. (#104)- See the source package CHANGELOG for updates made to the staging layer in
dbt_hubspot_source v0.9.1
.
- See the source package CHANGELOG for updates made to the staging layer in
- Updated the following models to utilize the
adjust_email_metrics
macro (#105):- hubspot__contacts
- hubspot__contact_lists
- hubspot__email_campaigns
- int_hubspot__email_metrics__by_contact_list
- Incorporated the new
fivetran_utils.drop_schemas_automation
macro into the end of each Buildkite integration test job. (#103) - Updated the pull request templates. (#103)
In November 2022, the Fivetran Hubspot connector switched to v3 of the Hubspot CRM API, which deprecated the CONTACT_MERGE_AUDIT
table and stored merged contacts in a field in the CONTACT
table. This has not been rolled out to BigQuery warehouses yet. BigQuery connectors with the CONTACT_MERGE_AUDIT
table enabled will continue to sync this table until the new CONTACT.property_hs_calculated_merged_vids
field and API version becomes available to them.
This release introduces breaking changes around how contacts are merged in order to align with the above connector changes. It is, however, backwards-compatible.
PR #100 applies the following changes:
- Updates logic around the recently deprecated
CONTACT_MERGE_AUDIT
table.- The package now leverages the new
property_hs_calculated_merged_vids
field to filter out merged contacts. This is the default behavior for all destinations, including BigQuery (the package will run successfully but not actually merge any contacts). This is achieved by the package's newmerge_contacts()
macro. - Backwards-compatibility: the package will only reference the old
CONTACT_MERGE_AUDIT
table to merge contacts ifhubspot_contact_merge_audit_enabled
is explicitly set totrue
in your rootdbt_project.yml
file. - The
int_hubspot__contact_merge_adjust
model (where the package performs contact merging) is now materialized as a table by default. This used to be ephemeral, but the reworked merge logic requires this model to be either a table or view.
- The package now leverages the new
PR #100 applies the following changes:
- Updates seed data to test new merging paradigm.
See the source package CHANGELOG for updates made to the staging layer in dbt_hubspot_source v0.9.0
.
- Following the release of
v0.8.0
, the end model uniqueness tests were not updated to account for the added flexibility of the inclusion of deleted records. As such the respective end model tests have been adjusted to test uniqueness only on non-deleted records. (#94)
- Addition of the dbt-expectations package to be used for more robust testing of uniqueness for end models. (#94)
PR #92 incorporates the following changes:
-
The below models/macros have new soft deleted record flags explicitly added to them:
is_contact_deleted
added inmacros/email_events_joined
which can affect the following downstream models found inmodels/marketing/email_events/
:hubspot__email_event_bounce
hubspot__email_event_clicks
hubspot__email_event_deferred
hubspot__email_event_delivered
hubspot__email_event_dropped
hubspot__email_event_forward
hubspot__email_event_open
hubspot__email_event_print
hubspot__email_event_sent
hubspot__email_event_spam_report
hubspot__email_event_status_change
is_deal_pipeline_deleted
andis_deal_pipeline_stage_deleted
added inmodels/sales/intermediate/int_hubspot__deals_enhanced
which can affect the following downstream models found inmodels/sales/
:hubspot__deal_stages
hubspot__deals
is_deal_pipeline_deleted
,is_deal_pipeline_stage_deleted
andis_deal_deleted
added inmodels/sales/hubspot__deal_stages
-
Soft deleted records are also now implicitly inherited (e.g. via a
select *
) from thedbt_hubspot_source
package's staging models and can affect the below final models (for more information, see dbt_hubspot_source PR #96):- Soft deleted company records (
companies.is_company_deleted
) included in the below models:sales/hubspot__companies
- Soft deleted deal records (
deals.is_deal_deleted
) included in the below models:sales/hubspot__deal_stages
sales/hubspot__deals
- Soft deleted contact list records (
contact_lists.is_contact_list_deleted
) included in the below models:marketing/hubspot__contact_lists
- Soft deleted contact records (
contacts.is_contact_deleted
) included in the below models:marketing/hubspot__contacts
- Soft deleted email sends records (
sends.is_contact_deleted
) included in the below models:marketing/hubspot__email_sends
- Soft deleted company records (
-
For completeness, soft deleted records are also now included for
ticket*
tables, however does not currently affect this package directly and includes the following staging models:stg_hubspot__ticket
(is_ticket_deleted
)stg_hubspot__ticket_pipeline_stage
(is_ticket_pipeline_stage_deleted
)stg_hubspot__ticket_pipeline
(is_ticket_pipeline_deleted
)
PR #86 includes the following breaking changes:
- Dispatch update for dbt-utils to dbt-core cross-db macros migration. Specifically
{{ dbt_utils.<macro> }}
have been updated to{{ dbt.<macro> }}
for the below macros:any_value
bool_or
cast_bool_to_text
concat
date_trunc
dateadd
datediff
escape_single_quotes
except
hash
intersect
last_day
length
listagg
position
replace
right
safe_cast
split_part
string_literal
type_bigint
type_float
type_int
type_numeric
type_string
type_timestamp
array_append
array_concat
array_construct
- For
current_timestamp
andcurrent_timestamp_in_utc
macros, the dispatch AND the macro names have been updated to the below, respectively:dbt.current_timestamp_backcompat
dbt.current_timestamp_in_utc_backcompat
dbt_utils.surrogate_key
has also been updated todbt_utils.generate_surrogate_key
. Since the method for creating surrogate keys differ, we suggest all users do afull-refresh
for the most accurate data. For more information, please refer to dbt-utils release notes for this update.- Dependencies on
fivetran/fivetran_utils
have been upgraded, previously[">=0.3.0", "<0.4.0"]
now[">=0.4.0", "<0.5.0"]
.
- 🧱 Databricks compatibility! (PR #89)
PR #84 incorporates the following updates:
- Added column descriptions that were missing in our documentation.
- Updated the docs as there were issues with the previous deployment. (#82)
- Introduced a GitHub workflow to ensure docs are re-built prior to merges to
main
. (#82)
- The README had holdover merge artifacts included within it. Those artifacts have since been removed in this release. #80
- Updated README documentation updates for easier navigation and setup of the dbt package (#67).
- Included
hubspot_[source_table_name]_identifier
variable for additional flexibility within the package when source tables are named differently (#67). - Adds
hubspot_ticket_deal_enabled
variable (default value=False
) to disable modelling and testing of theticket_deal
source table. If there are no associations between tickets and deals in your Hubspot environment, this table will not exist (#79).
- The
hubspot__deal_stages
model has undergone two major changes (#78):- The stage and pipeline label columns now reflect where the deal was at a certain point of time (from
date_stage_entered
todate_stage_exited
). Previously, this information reflected the deal's current stage and pipeline in every record associated with the deal. - This model previously passed through all fields from the parent deals model. We removed these fields, as they are all present in
hubspot__deals
final model and do not change across deal stages. If you would like to joinDEAL
fields into thehubspot__deal_stages
model, joinhubspot__deal_stages
withhubspot__deals
ondeal_id
.
- The stage and pipeline label columns now reflect where the deal was at a certain point of time (from
- Consistently renames
property_dealname
,property_closedate
, andproperty_createdate
todeal_name
,closed_at
, andcreated_at
, respectively, in thedeals
model. Previously, ifhubspot__pass_through_all_columns = true
, only the prefixproperty_
was removed from the names of these fields, while they were completely renamed todeal_name
,closed_at
, andcreated_at
ifhubspot__pass_through_all_columns = false
(#79).
- Typo fix and spelling correction within the README. (#70)
- Spelling correction of the variable names within the README. (#74)
- Added integration testing to support the new
stg_hubspot__deal_contact
model to thedbt_hubspot_source
package. - Updated the below models such that the respective
lead()
functions also account forfield_name
in partitionshubspot__contact_history
hubspot__company_history
hubspot__deal_history
- Added test cases for the above models that can be found, respectively, in:
models/marketing/history/history.yml
models/sales/history/history.yml
- Updated the
engagements_joined
macro to conditionally include relevant fields if their respective variables are enabled. Previously if a variable was disabled then thehubspot__engagement_calls
model, which depends on theengagements_joined
macro, would error out on the missing fields that were from the disabled variables. (#65)
- Modified the join conditions within the
deal_fields_joined
cte of int_hubspot__deals_enhanced model to leverage the more appropriateleft join
rather than theusing
condition. This is to allow for correct and accurate joins across warehouses. (#60)
🎉 dbt v1.0.0 Compatibility 🎉
- Adjusts the
require-dbt-version
to now be within the range [">=1.0.0", "<2.0.0"]. Additionally, the package has been updated for dbt v1.0.0 compatibility. If you are using a dbt version <1.0.0, you will need to upgrade in order to leverage the latest version of the package.- For help upgrading your package, I recommend reviewing this GitHub repo's Release Notes on what changes have been implemented since your last upgrade.
- For help upgrading your dbt project to dbt v1.0.0, I recommend reviewing dbt-labs upgrading to 1.0.0 docs for more details on what changes must be made.
- Upgrades the package dependency to refer to the latest
dbt_hubspot_source
. Additionally, the latestdbt_hubspot_source
package has a dependency on the latestdbt_fivetran_utils
. Further, the latestdbt_fivetran_utils
package also has a dependency ondbt_utils
[">=0.8.0", "<0.9.0"].- Please note, if you are installing a version of
dbt_utils
in yourpackages.yml
that is not in the range above then you will encounter a package dependency error.
- Please note, if you are installing a version of
Refer to the relevant release notes on the Github repository for specific details for the previous releases. Thank you!