3.0.0-beta1

What's Changed

• Remove all analytics from the dataform CLI by @GJMcGowan in #1683
• Remove a bunch of random packages we don't use by @GJMcGowan in #1684
• Add notebook options to configs proto by @Ekrekr in #1685

Full Changelog: 3.0.0-beta.0...3.0.0-beta.1

3.0.0-beta.0

Description of changes since 2.9.0 have been moved to https://github.com/dataform-co/dataform/releases/tag/3.0.0.

What's Changed

• Remove non bigquery warehouses by @Ekrekr in #1550
• Remove dynamic warehouse inference, and more clearly differentiate core and api's SQL adapters by @Ekrekr in #1554
• Consolidate cli package by @Ekrekr in #1557
• Merge main to v3 by @Ekrekr in #1563
• Deprecate support for old type .sql files by @Ekrekr in #1564
• Add initial action config proto definitions by @Ekrekr in #1576
• Merge main to main_v3 by @Ekrekr in #1579
• Remove gen index in favor of main by @Ekrekr in #1571
• Read workflow_settings.yaml by @Ekrekr in #1580
• Validate workflow settings fields by @Ekrekr in #1581
• Remove deprecated run cache and gcloud project ID fields by @Ekrekr in #1584
• Move concurrency and action retries from project config proto to the CLI by @Ekrekr in #1585
• Merge main to main_v3 3 by @Ekrekr in #1590
• Populate warehouse as BigQuery by default by @Ekrekr in #1591
• Update v3 to use the apache 2.0 license by @Ekrekr in #1594
• Stop using ProtobufJS's verify method (it doesn't do much) by @Ekrekr in #1596
• Clean up action proto constructors in prep for proto config definitions by @Ekrekr in #1597
• Add basic actions.yaml and notebook support by @Ekrekr in #1595
• Move custom variables test to the new testing structure by @Ekrekr in #1601
• Add auto assertion database override to core test by @Ekrekr in #1602
• Move core/tasks.ts to CLI, and minor cleanup by @Ekrekr in #1598
• Add more test coverage of variables and project config overrides for Dataform Core by @Ekrekr in #1603
• Add licenses as prefixes to output bundles by @Ekrekr in #1605
• Throw a more interpretable error when a =v3 Core version by @Ekrekr in #1607
• Action builders and path utils fixes by @Ekrekr in #1606
• Prevent non bq warehouse from being set in workflow settings by @Ekrekr in #1609
• Strip notebook cell outputs by @Ekrekr in #1613
• Add version to workflow settings by @Ekrekr in #1610
• Move common target constructor methods to the action builder by @Ekrekr in #1626
• Add basic support for actions.yaml reading SQL files as operations by @Ekrekr in #1628
• Add support for declarations in action config files by @Ekrekr in #1630
• Add support for Tables to action config files by @Ekrekr in #1631
• Add action config support for the remaining SQL based action types by @Ekrekr in #1633
• Improve proto validation errors where possible by @Ekrekr in #1635
• Make filenames defined in action config files be treated as relative to the action config file by @Ekrekr in #1636
• Better errors for invalid declarations, add more test coverage by @Ekrekr in #1637
• Add an example of a workflow containing notebooks/SQL scripts, with tests by @Ekrekr in #1644
• Make the CLI use and default to workflow_settings.yaml by @Ekrekr in #1648
• Update Dataform CLI npm installs to best-effort by @Ekrekr in #1649
• Merge to v3 4 by @Ekrekr in #1654
• Remove dead resolve code by @Ekrekr in #1652
• Make main_test more concise by making filenames optional by @Ekrekr in #1651
• Remove more dead code, including navigator column descriptors and tools by @Ekrekr in #1655
• Remove warehouse option from the CLI, more tidying by @Ekrekr in #1653
• Make v3 JS API act the same as v2 by @Ekrekr in #1657
• Remove inline tables functionality by @Ekrekr in #1658
• Update proto to use a separate compiled graph, with UX review changes by @Ekrekr in #1656
• Core path tidying by @Ekrekr in #1659
• Export configs proto, bump version by @Ekrekr in #1661
• Migrate core sqlx syntax and assertions context functions tests to _test style by @Ekrekr in #1665
• Better cli errors during compilation by @Ekrekr in #1669
• Replace cli spec tests with _test format by @Ekrekr in #1670
• Improve generated workflow_settings.yaml field order by @Ekrekr in #1671
• Fix config targets by @Ekrekr in #1673
• Remove JS context from SQL files by @Ekrekr in #1672
• Lazy / stateless installation by @Ekrekr in #1675
• Don't check we are on main branch for pre-releases by @lewish in #1664
• Merge main to main_v3 5 by @Ekrekr in #1678
• Bump ip from 1.1.5 to 1.1.9 by @dependabot in #1679
• Merge v3 into main by @Ekrekr in #1680
• Bump v3 to beta by @Ekrekr in #1682

Full Changelog: 2.9.0...3.0.0-beta.0

2.9.0

What's Changed

• Bump follow-redirects from 1.15.3 to 1.15.4 by @dependabot in #1645
• Include type declarations with @dataform/core by @bmagyarkuti in #1647
• Allow 'partitionBy' and 'clusterBy' params for bigquery materialized views by @vlad-ogol in #1660

New Contributors

• @vlad-ogol made their first contribution in #1660

Full Changelog: 2.8.4...2.9.0

2.8.4

What's Changed

• Fix ActionIndex.find() when callers pass an unknown database/schema value. by @BenBirt in #1643

Full Changelog: 2.8.3...2.8.4

2.8.3

What's Changed

• Make Vscode Buildable Again by @GJMcGowan in #1632
• VSCode: avoid errors upon completion requests by @barna-emarsys in #1627
• VSCode: Add ability to configure compile command, add option to skip compilation on save by @bmagyarkuti in #1639
• Escape backslashes and backticks in multiline string literals by @diasdauletov in #1641

New Contributors

• @barna-emarsys made their first contribution in #1627
• @bmagyarkuti made their first contribution in #1639

Full Changelog: 2.8.2...2.8.3

2.8.2

What's Changed

• Fix the @dataform/cli --table_prefix flag. by @BenBirt in #1623
• Append a compile error when resolve() is unable to resolve the given reference to an action. by @BenBirt in #1624

Full Changelog: 2.8.1...2.8.2

2.8.1

What's Changed

• Make a packages readme by @Ekrekr in #1612
• Fix doc link, and tidying by @Ekrekr in #1614
• Updates mssql dependencies by @bit-shifter in #1618
• Remove unused babel-preset-env dependency, which transitively removes a dependency on babel-traverse (CVE-2023-45133). by @BenBirt in #1619
• Bump version number. by @BenBirt in #1620

New Contributors

• @bit-shifter made their first contribution in #1618

Full Changelog: 2.8.0...2.8.1

2.8.0: Packaging and compilation performance improvements

The highlights of this release are two significant performance improvements for GCP Dataform projects:

• The @dataform/core NPM package now no longer has any dependencies, with all dependency packages bundled in to existing minified bundle. This reduces the size of installation significantly and should improve package installation performance and reliability on Dataform on GCP. See #1552 for more info.
• We've rolled out a change that reduces the amount of work required to decode/encode compiled graphs that we expect to improve compilation performance on Dataform on GCP by ~2x. See #1570 for more info.

What's Changed

• Remove formatter test files from examples by @Ekrekr in #1558
• Improve example projects by @Ekrekr in #1559
• Bump postcss from 8.3.9 to 8.4.31 by @dependabot in #1548
• Allow ArrayBuffer args and return from the main function, more than doubling compilation speed for large projects by @Ekrekr in #1570
• Remove snowflake test dependency on S3 by @lewish in #1583
• Minify @dataform/core and bundle all dependencies by @lewish in #1552
• Bump browserify-sign from 4.2.1 to 4.2.2 by @dependabot in #1586
• Bump axios from 0.21.2 to 1.6.0 by @dependabot in #1574
• Add support for publishing pre-release/next packages (and stage 2.8.0-beta.1) by @lewish in #1587
• Update analytics-node to unabandoned version by @claydiffrient in #1588
• Update version of snowflake-sdk to 1.9.1 by @claydiffrient in #1592
• Add project config override cli options by @Ekrekr in #1454

New Contributors

• @claydiffrient made their first contribution in #1588

Full Changelog: 2.7.0...2.8.0

2.7.0: Updates for Dataform GCP incremental SQL

From version 2.7.0 onwards, Dataform projects running on Google Cloud Platform will use updated SQL generation logic for incremental insert tables (tables of type incremental without a uniqueKey specified).

Explicit column names

Column names will be explicitly listed in the insert call, which is inline with OSS Dataform behaviour and prevents schema mismatch during insert, for example:

• if source_table has columns in a different order that the target - it can lead to data corruption (column values can be swapped during insert)
• if source_table has different number of columns - the insert into fails since target table columns count does not match with the source

For the new script the incremental query has to list all target table columns (can list other extra columns, but at least must contain target table columns) in any order.

Example of new generated SQL:

INSERT INTO $target_table 
    ($target_columns_list) -- listing target columns
    SELECT target_columns_list -- reordering columns so that subquery column order matches the target column order
    FROM (
        $incrementalQuery
    );

Execution within a procedure

In order to facilitate explicit columns, the new code is executed within a procedure, which will be created on the fly. For example:

EXECUTE IMMEDIATE
"""
CREATE OR REPLACE PROCEDURE $procedure_name()
BEGIN
    $preOperations
    $incrementalInsertStatement
    $postOperations
END;
"""
CALL $procedure_name();
DROP PROCEDURE IF EXISTS $procedure_name;

2.6.8

Fixing bug for escaping target name and canonical target name. PR: #1551