BigQuery Web Model v1.0.4

Patch release for: BigQuery Web: Fix upsert limit in commit_table procedure (Close #75)

This is to make sure the upsert_lookback_days variable works as it was intended to.

upsert_lookback_days:  default 30. Period of time (in days) to look back over the production table in order to find rows to delete when upserting data. Where performance is not a concern, should be set to as long a value as possible.

No need to run a migration playbook to upgrade.

Snowflake Web Model v1.0.2

This release contains the following changes:

Snowflake Web: Optimise partition pruning (#140)

No need to run a migration playbook to upgrade.

Redshift Web Model v1.3.1

This is a patch release containing the following fix:

• Redshift Web: load_tstamp missing from table definition #135

It is to unblock users using the Redshift web model who are using RDB Loader 4.0.0 and above which causes the data model to fail due to the added load_tstamp field in the events table.

Redshift Web Model v1.3.0

This is a patch release containing the following fixes:

• Change SORTKEY encoding to RAW (#129) (thanks @mark-walle!)
• Fix column lengths in manifest tables (#131)

It also introduces a pattern to make manual steps for upgrading models more transparent and accessible.

Since column definitions change in this release, manual steps are required to bring existing tables into line with the new table definitions. Where in previous releases we provided the steps to do so in the release notes, these steps can now be found in the 99-migrations folder, and playbooks are provided.

This release contains the migration for Redshift web 1.2.0 -> 1.3.0. A subsequent maintenance release will include migrations for previous releases of all models.

Snowflake Web Model v1.0.1

This patch release contains the following fixes:

• Update column check stored procedure ( #125)
• Fix varchar length for pseudonymized fields (#122)
• Remove start_date variable from users module (#123)
• Update copyright notices ( #124)
• Fix logic in users_sessions_this_run to account for sparse data (#120)
• Fix varchar length for yauaa columns (#97)
• Fix se_label column length in events_staged (#109)

Although there are no breaking changes, pre-existing users of the package will have to relax the following columns manually in order for the changes to take effect:

alter table if exists {{.output_schema}}.base_session_id_manifest{{.entropy}}
 alter column session_id set data type varchar(128);
 
alter table if exists {{.output_schema}}.users_manifest{{.entropy}}
 alter column domain_userid set data type varchar(128);
 
alter table if exists {{.scratch_schema}}.events_staged{{.entropy}}
 alter column se_label set data type varchar(4096);
 
alter table if exists {{.output_schema}}.page_views{{.entropy}}
 alter column agent_name set data type varchar,
  agent_name_version set data type varchar,
  agent_name_version_major set data type varchar,
  agent_version set data type varchar,
  agent_version_major set data type varchar,
  device_brand set data type varchar,
  device_name set data type varchar,
  device_version set data type varchar,
  layout_engine_name set data type varchar,
  layout_engine_name_version set data type varchar,
  layout_engine_name_version_major set data type varchar,
  layout_engine_version set data type varchar,
  layout_engine_version_major set data type varchar,
  operating_system_name set data type varchar,
  operating_system_name_version set data type varchar,
  operating_system_version set data type varchar;

alter table if exists {{.scratch_schema}}.page_views_staged{{.entropy}}
 alter column agent_name set data type varchar,
  agent_name_version set data type varchar,
  agent_name_version_major set data type varchar,
  agent_version set data type varchar,
  agent_version_major set data type varchar,
  device_brand set data type varchar,
  device_name set data type varchar,
  device_version set data type varchar,
  layout_engine_name set data type varchar,
  layout_engine_name_version set data type varchar,
  layout_engine_name_version_major set data type varchar,
  layout_engine_version set data type varchar,
  layout_engine_version_major set data type varchar,
  operating_system_name set data type varchar,
  operating_system_name_version set data type varchar,
  operating_system_version set data type varchar;
 
alter table if exists {{.output_schema}}.sessions{{.entropy}}
 alter column agent_name set data type varchar,
  agent_name_version set data type varchar,
  agent_name_version_major set data type varchar,
  agent_version set data type varchar,
  agent_version_major set data type varchar,
  device_brand set data type varchar,
  device_name set data type varchar,
  device_version set data type varchar,
  layout_engine_name set data type varchar,
  layout_engine_name_version set data type varchar,
  layout_engine_name_version_major set data type varchar,
  layout_engine_version set data type varchar,
  layout_engine_version_major set data type varchar,
  operating_system_name set data type varchar,
  operating_system_name_version set data type varchar,
  operating_system_version set data type varchar;

Snowflake Mobile Model v1.1.0

This release contains the following changes:

• Fixes column order in events_this_run to allow for events_staged table migration (#102)
• Updates commit_table procedure to throw error messages (#103)
• Updates update_manifest procedure to throw error message (#104)

This release contains breaking changes to the model, however all data in the derived tables will be preserved. To upgrade to this version of the model:

• Ensure that the model has successfully completed it's previous run and all data in mobile_events_staged has been consumed by all modules i.e. mobile_events_staged is empty.
• Drop mobile_events_staged using: DROP TABLE {{scratch_schema}}.mobile_events_staged{{entropy}};
• Run the model as per usual.

BigQuery Mobile Model v1.1.0

This release contains the following changes:

• Fix column order in mobile_events_this_run to allow for successful table migration of mobile_events_staged in the event of new columns (#99)
• Add mobile_staging_reconcilition to temp tables validation config (#100)

This release contains breaking changes to the model, however all data in the derived tables will be preserved. To upgrade to this version of the model:

• Ensure that the model has successfully completed it's previous run and all data in mobile_events_staged has been consumed by all modules i.e. mobile_events_staged is empty.
• Drop mobile_events_staged using: DROP TABLE {{scratch_schema}}.mobile_events_staged{{entropy}};
• Run the model as per usual.

Redshift Mobile Model v1.1.0

This release contains the following changes:

• Increases session_id character limit in the sessions manifest (#93)
• Adds simplified configs (#89)
• Sets app errors module to disabled by default (#88)
• Adds model_tstamp to derived tables. This allows for the easy identification of new rows within a table since the last run. This is useful for building incremental models off the Snowplow derived tables in tools like dbt (#82)
• Moves app errors columns to end of table (#83)
• Remove redundant CTE from user-aggs step (#80)

This release contains breaking changes to the model. To upgrade to this version of the model:

• Teardown all existing tables from the mobile model using the teardown_all config.
• Replace the standard components of the model with the new release.
• Re-run the model

Snowflake Mobile Model v1.0.0

Merge pull request #86 from snowplow/release/mobile/snowflake/1.0.0

Add Snowflake mobile model v1 (Close #85)

BigQuery Mobile Model v1.0.0

Merge pull request #74 from snowplow/release/mobile/bigquery/1.0.0

Release/mobile/bigquery/1.0.0