Merge pull request #30 from fivetran/bugfix/catch-all-recent-unsubscr…

…ibes

[Bug Fix] Catch all recent unsubscribes by updating partitions

.github/ISSUE_TEMPLATE/bug-report.yml

@@ -83,6 +83,6 @@ body:
       description: Our team will assess this issue and let you know if we will add it to a future sprint. However, if you would like to expedite the solution, we encourage you to contribute to the package via a PR. Our team will then work with you to approve and merge your contributions as soon as possible.
       options:
         - label: Yes.
-        - label: Yes, but I will need assistance and will schedule time during our [office hours](https://calendly.com/fivetran-solutions-team/fivetran-solutions-team-office-hours) for guidance
+        - label: Yes, but I will probably need assistance.
         - label: No.
           required: false

.github/ISSUE_TEMPLATE/config.yml

@@ -1,7 +1,7 @@
 contact_links:
-  - name: Ask a question during our office hours
-    url: https://calendly.com/fivetran-solutions-team/fivetran-solutions-team-office-hours
-    about: Schedule time during the external office hours block with the Fivetran Analytics Engineering team for support
+  - name: Provide feedback to our dbt package team
+    url: https://www.surveymonkey.com/r/DQ7K7WW
+    about: Fill out our survey form to provide valuable feedback to the Fivetran team developing and maintaining the dbt packages.
   - name: Fivetran connector question
     url: https://support.fivetran.com/hc
     about: Have a question about your connector? Check out the Fivetran support portal for more details.

.github/ISSUE_TEMPLATE/feature-request.yml

@@ -34,7 +34,7 @@ body:
       description: Our team will assess this feature and let you know if we will add it to a future sprint. However, if you would like to expedite the feature, we encourage you to contribute to the package via a PR. Our team will then work with you to approve and merge your contributions as soon as possible.
       options:
         - label: Yes.
-        - label: Yes, but I will need assistance and will schedule time during your [office hours](https://calendly.com/fivetran-solutions-team/fivetran-solutions-team-office-hours) for guidance.
+        - label: Yes, but I will probably need assistance.
         - label: No.
           required: false
   - type: textarea

.github/PULL_REQUEST_TEMPLATE/maintainer_pull_request_template.md

@@ -4,48 +4,26 @@
 **This PR will result in the following new package version:**
 <!--- Please add details around your decision for breaking vs non-breaking version upgrade. If this is a breaking change, were backwards-compatible options explored? -->
 
-**Please detail what change(s) this PR introduces and any additional information that should be known during the review of this PR:**
+**Please provide the finalized CHANGELOG entry which details the relevant changes included in this PR:**
+<!--- Copy/paste the CHANGELOG for this version below. -->
 
 ## PR Checklist
 ### Basic Validation
 Please acknowledge that you have successfully performed the following commands locally:
-- [ ] dbt compile
-- [ ] dbt run –full-refresh
-- [ ] dbt run
-- [ ] dbt test
-- [ ] dbt run –vars (if applicable)
+- [ ] dbt run –full-refresh && dbt test
+- [ ] dbt run (if incremental models are present) && dbt test
 
 Before marking this PR as "ready for review" the following have been applied:
-- [ ] The appropriate issue has been linked and tagged
-- [ ] You are assigned to the corresponding issue and this PR
+- [ ] The appropriate issue has been linked, tagged, and properly assigned
+- [ ] All necessary documentation and version upgrades have been applied
+- [ ] docs were regenerated (unless this PR does not include any code or yml updates)
 - [ ] BuildKite integration tests are passing
+- [ ] Detailed validation steps have been provided below
 
 ### Detailed Validation
-Please acknowledge that the following validation checks have been performed prior to marking this PR as "ready for review":
-- [ ] You have validated these changes and assure this PR will address the respective Issue/Feature.
-- [ ] You are reasonably confident these changes will not impact any other components of this package or any dependent packages.
-- [ ] You have provided details below around the validation steps performed to gain confidence in these changes.
+Please share any and all of your validation steps:
 <!--- Provide the steps you took to validate your changes below. -->
 
-### Standard Updates
-Please acknowledge that your PR contains the following standard updates:
-- Package versioning has been appropriately indexed in the following locations:
-    - [ ] indexed within dbt_project.yml
-    - [ ] indexed within integration_tests/dbt_project.yml
-- [ ] CHANGELOG has individual entries for each respective change in this PR
-    <!--- If there is a parallel upstream change, remember to reference the corresponding CHANGELOG as an individual entry.  -->
-- [ ] README updates have been applied (if applicable)
-    <!--- Remember to check the following README locations for common updates. →
-        <!--- Suggested install range (needed for breaking changes) →
-        <!--- Dependency matrix is appropriately updated (if applicable) →
-        <!--- New variable documentation (if applicable) -->
-- [ ] DECISIONLOG updates have been updated (if applicable)
-- [ ] Appropriate yml documentation has been added (if applicable)
-
-### dbt Docs
-Please acknowledge that after the above were all completed the below were applied to your branch:
-- [ ] docs were regenerated (unless this PR does not include any code or yml updates)
-
 ### If you had to summarize this PR in an emoji, which would it be?
 <!--- For a complete list of markdown compatible emojis check our this git repo (https://gist.github.com/rxaviers/7360908)  --> 
-:dancer:
+:dancer:

.github/workflows/auto-release.yml

@@ -0,0 +1,13 @@
+name: 'auto release'
+on:
+  pull_request:
+    types:
+      - closed
+    branches:
+      - main
+
+jobs:
+  call-workflow-passing-data:
+    if: github.event.pull_request.merged
+    uses: fivetran/dbt_package_automations/.github/workflows/auto-release.yml@main
+    secrets: inherit

CHANGELOG.md

@@ -1,3 +1,17 @@
+# dbt_iterable_source v0.8.1
+[PR #30](https://github.com/fivetran/dbt_iterable_source/pull/30) introduces the following updates: 
+
+## Bug Fixes
+- IMPORTANT: This only impacts customers who are using Fivetran connectors that were setup before August 2023, when `user_unsubscribed_message_type_history` and `user_unsubscribed_channel_history` were the tables being used for tracking unsubscribe events.
+- Updated the `stg_iterable__user_unsub_message_type` model to partition the `latest_batch_index` value on `message_type_id` along with `email`. This brings in all unsubscribes for the `unique_user_key` across all message types for an email.
+- Updated `stg_iterable__user_unsubscribed_channel` and `stg_iterable__user_unsubscribed_channel` to partition the `latest_batch_index` value on `channel_id` along with `email`. This brings in all unsubscribes for the `unique_user_key` across all channels for an email.
+- These updates ensure that `iterable__user_unsubscriptions` in the `dbt_iterable` package brings in all the latest unsubscribing behavior for a user by channel and by message type, and not just the last unsubscription for a user based on the `updated_at` value.
+
+## Under the Hood
+- Updated seed files to effectively run and test that the new partitions worked as expected.
+- Updated [pull request and issue templates](https://github.com/fivetran/dbt_iterable_source/tree/v0.8.1/.github).
+- Included auto-releaser GitHub Actions workflow to automate future releases.
+
 # dbt_iterable_source v0.8.0
 
 ## API Updates

README.md

@@ -156,5 +156,4 @@ We highly encourage and welcome contributions to this package. Check out [this d
 
 # 🏪 Are there any resources available?
 - If you have questions or want to reach out for help, please refer to the [GitHub Issue](https://github.com/fivetran/dbt_iterable_source/issues/new/choose) section to find the right avenue of support for you.
-- If you would like to provide feedback to the dbt package team at Fivetran or would like to request a new dbt package, fill out our [Feedback Form](https://www.surveymonkey.com/r/DQ7K7WW).
-- Have questions or want to just say hi? Book a time during our office hours [on Calendly](https://calendly.com/fivetran-solutions-team/fivetran-solutions-team-office-hours) or email us at [email protected].
+- If you would like to provide feedback to the dbt package team at Fivetran or would like to request a new dbt package, fill out our [Feedback Form](https://www.surveymonkey.com/r/DQ7K7WW).

dbt_project.yml

@@ -1,7 +1,7 @@
 config-version: 2
 name: 'iterable_source'
 
-version: '0.8.0'
+version: '0.8.1'
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 
 models:

integration_tests/ci/sample.profiles.yml

@@ -16,13 +16,13 @@ integration_tests:
       pass: "{{ env_var('CI_REDSHIFT_DBT_PASS') }}"
       dbname: "{{ env_var('CI_REDSHIFT_DBT_DBNAME') }}"
       port: 5439
-      schema: iterable_source_integration_tests_3
+      schema: iterable_source_integration_tests_4
       threads: 8
     bigquery:
       type: bigquery
       method: service-account-json
       project: 'dbt-package-testing'
-      schema: iterable_source_integration_tests_3
+      schema: iterable_source_integration_tests_4
       threads: 8
       keyfile_json: "{{ env_var('GCLOUD_SERVICE_KEY') | as_native }}"
     snowflake:
@@ -33,7 +33,7 @@ integration_tests:
       role: "{{ env_var('CI_SNOWFLAKE_DBT_ROLE') }}"
       database: "{{ env_var('CI_SNOWFLAKE_DBT_DATABASE') }}"
       warehouse: "{{ env_var('CI_SNOWFLAKE_DBT_WAREHOUSE') }}"
-      schema: iterable_source_integration_tests_3
+      schema: iterable_source_integration_tests_4
       threads: 8
     postgres:
       type: postgres
@@ -42,13 +42,13 @@ integration_tests:
       pass: "{{ env_var('CI_POSTGRES_DBT_PASS') }}"
       dbname: "{{ env_var('CI_POSTGRES_DBT_DBNAME') }}"
       port: 5432
-      schema: iterable_source_integration_tests_3
+      schema: iterable_source_integration_tests_4
       threads: 8
     databricks:
       catalog: "{{ env_var('CI_DATABRICKS_DBT_CATALOG') }}"
       host: "{{ env_var('CI_DATABRICKS_DBT_HOST') }}"
       http_path: "{{ env_var('CI_DATABRICKS_DBT_HTTP_PATH') }}"
-      schema: iterable_source_integration_tests_3
+      schema: iterable_source_integration_tests_4
       threads: 8
       token: "{{ env_var('CI_DATABRICKS_DBT_TOKEN') }}"
       type: databricks

integration_tests/dbt_project.yml

@@ -1,12 +1,12 @@
 config-version: 2
 name: 'iterable_source_integration_tests'
 
-version: '0.8.0'
+version: '0.8.1'
 profile: 'integration_tests'
 
 vars:
   iterable_source:
-    iterable_schema: iterable_source_integration_tests_3
+    iterable_schema: iterable_source_integration_tests_4
     iterable_campaign_history_identifier: "campaign_history_data"
     iterable_campaign_label_history_identifier: "campaign_label_history_data"
     iterable_campaign_list_history_identifier: "campaign_list_history_data"

integration_tests/requirements.txt

@@ -1,7 +1,7 @@
-dbt-snowflake>=1.3.0,<2.0.0
-dbt-bigquery>=1.3.0,<2.0.0
-dbt-redshift>=1.3.0,<2.0.0
-dbt-postgres>=1.3.0,<2.0.0
-dbt-spark>=1.3.0,<2.0.0
-dbt-spark[PyHive]>=1.3.0,<2.0.0
-dbt-databricks>=1.6.0,<2.0.0
+dbt-snowflake>=1.3.0,<1.8.0
+dbt-bigquery>=1.3.0,<1.8.0
+dbt-redshift>=1.3.0,<1.8.0
+dbt-postgres>=1.3.0,<1.8.0
+dbt-spark>=1.3.0,<1.8.0
+dbt-spark[PyHive]>=1.3.0,<1.8.0
+dbt-databricks>=1.6.0,<1.8.0

integration_tests/seeds/message_type_data.csv

@@ -1,3 +1,6 @@
 id,name,channel_id,_fivetran_deleted,_fivetran_synced,created_at,frequency_cap,rate_limit_per_minute,subscription_policy,updated_at
 50806,Event Invites,13508,TRUE,2021-03-10 14:17:29.467,2023-10-25 15:30:00,5,100,"Gold",2023-10-25 16:45:00
-50830,Event Invites,14206,FALSE,2021-06-14 14:17:32.568,2023-10-25 15:30:00,5,1000,"Basic",2023-10-25 15:45:00
+50830,Event Invites,14206,FALSE,2021-06-14 14:17:32.568,2023-10-25 15:30:00,5,1000,"Basic",2023-10-25 15:45:00
+20246,Event Invites,11249,FALSE,2021-06-14 14:17:32.568,2023-10-25 15:30:00,5,1000,"Basic",2023-10-25 15:45:00
+20248,Event Invites,11249,FALSE,2021-06-14 14:17:32.568,2023-10-25 15:30:00,5,1000,"Basic",2023-10-25 15:45:00
+20247,Purchases,11250,FALSE,2021-06-14 14:17:32.568,2023-10-25 15:30:00,5,1000,"Basic",2023-10-25 15:45:00

integration_tests/seeds/user_unsubscribed_channel_history_data.csv

@@ -1,3 +1,6 @@
 email,channel_id,updated_at,_fivetran_synced
 [email protected],11249,2020-09-28 06:16:07.000,2020-09-28 06:17:46.188
-[email protected],11249,2020-09-28 04:21:18.000,2020-09-28 06:17:45.341
+[email protected],11249,2020-09-28 04:21:18.000,2020-09-28 06:17:45.341
+[email protected],11249,2020-09-29 04:21:18.000,2020-09-28 06:17:45.341
+[email protected],11250,2020-09-28 04:21:18.001,2020-09-28 06:17:45.341
+[email protected],11250,2020-09-29 04:21:18.001,2020-09-28 06:17:45.341

integration_tests/seeds/user_unsubscribed_message_type_history_data.csv

@@ -1,3 +1,8 @@
 message_type_id,email,updated_at,_fivetran_synced
 20246,[email protected],2020-08-08 04:44:39.000,2020-08-08 04:48:17.099
-20246,[email protected],2020-08-14 16:16:32.000,2020-08-14 16:22:42.638
+20246,[email protected],2020-08-14 16:16:32.000,2020-08-14 16:22:42.638
+20246,[email protected],2020-08-14 16:16:32.000,2020-08-14 16:22:42.638
+20246,[email protected],2020-08-15 16:16:32.001,2020-08-14 16:22:42.638
+20247,[email protected],2020-08-14 16:16:32.001,2020-08-14 16:22:42.638
+20247,[email protected],2020-08-15 16:16:32.000,2020-08-14 16:22:42.638
+20248,[email protected],2020-08-14 16:16:32.000,2020-08-14 16:22:42.638

models/stg_iterable__user_unsub_message_type.sql

@@ -36,7 +36,7 @@ final as (
         {{ dbt_utils.generate_surrogate_key(['_fivetran_id', 'email', 'message_type_id','updated_at']) }} as unsub_message_type_unique_key,
 
         {% if does_table_exist('user_unsubscribed_message_type') == false %}
-        rank() over(partition by email order by updated_at desc) as latest_batch_index,
+        rank() over(partition by email, message_type_id order by updated_at desc) as latest_batch_index,
         {% else %}
         1 as latest_batch_index,
         {% endif %}

models/stg_iterable__user_unsubscribed_channel.sql

@@ -33,7 +33,7 @@ final as (
         {{ dbt_utils.generate_surrogate_key(['_fivetran_id', 'channel_id', 'email', 'updated_at']) }} as unsub_channel_unique_key,
 
         {% if does_table_exist('user_unsubscribed_channel') == false %}
-        rank() over(partition by email order by updated_at desc) as latest_batch_index,
+        rank() over(partition by email, channel_id order by updated_at desc) as latest_batch_index,
         {% else %}
         1 as latest_batch_index,
         {% endif %}