Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How does one test compilation code changes locally? #1629

Closed
mattkocak opened this issue Dec 30, 2023 · 1 comment · May be fixed by #1786
Closed

How does one test compilation code changes locally? #1629

mattkocak opened this issue Dec 30, 2023 · 1 comment · May be fixed by #1786

Comments

@mattkocak
Copy link

mattkocak commented Dec 30, 2023

Hello, this is a question rather than an issue. I'm becoming more familiar with the code base and have been testing changes that I've been making locally. I've been having trouble figuring out how I can test compile step code changes locally, though.

I've completed the setup as described in the contributing documentation and generally things are working well. However, I haven't managed to figure out how to test changes regarding the compile step.

Here's what I've done:

  1. Create a new project with ./scripts/run init bigquery my-project --default-database my-project --default-location EU
  2. Create credentials file with ./scripts/run init-creds bigquery my-project
  3. Add the following test.sqlx file to my-project/definitions:
config {
  type: "table",
  testProperty: "test"
}

SELECT
  CURRENT_TIMESTAMP()
  1. Make some changes to table.ts that I would expect to be reflected when compiling this file (console.logs, updating strings, etc.)
  2. Run ./scripts/run compile my-project
  3. Instead of seeing the changes that I would expect, it appears that the compile command is using code from /my-project/node_modules/@dataform/core/bundle.js (so the published version of Dataform), rather than using the code that I modified. Below is the output that I see after running the command:
Compiling...

Compiled 1 action(s).
1 dataset(s):
  dataform.test [table]

  Compilation errors:
  definitions/test.sqlx: Error: Unexpected property "testProperty" in table config. Supported properties are: ["type","disabled","protected","name","redshift","bigquery","snowflake","sqldatawarehouse","presto","tags","uniqueKey","dependencies","hermetic","schema","assertions","database","columns","description","materialized"]
    at Object.t.checkExcessProperties (/Users/matthewkocak/Documents/GitHub/dataform/my-project/node_modules/@dataform/core/bundle.js:1:209544)
    at a.config (/Users/matthewkocak/Documents/GitHub/dataform/my-project/node_modules/@dataform/core/bundle.js:1:252775)
    at t.Session.sqlxAction (/Users/matthewkocak/Documents/GitHub/dataform/my-project/node_modules/@dataform/core/bundle.js:1:383183)
    at /Users/matthewkocak/Documents/GitHub/dataform/my-project/definitions/test.sqlx:1:24
    at VM2 Wrapper.apply (/Users/matthewkocak/Documents/GitHub/dataform/bazel-bin/packages/@dataform/cli/bin.sh.runfiles/npm/node_modules/vm2/lib/bridge.js:485:11)
    at NodeVM.run (/Users/matthewkocak/Documents/GitHub/dataform/bazel-bin/packages/@dataform/cli/bin.sh.runfiles/npm/node_modules/vm2/lib/nodevm.js:497:23)
    at CustomResolver.loadJS (/Users/matthewkocak/Documents/GitHub/dataform/bazel-bin/packages/@dataform/cli/bin.sh.runfiles/npm/node_modules/vm2/lib/resolver.js:210:6)
    at CustomResolver.loadJS (/Users/matthewkocak/Documents/GitHub/dataform/bazel-bin/packages/@dataform/cli/bin.sh.runfiles/npm/node_modules/vm2/lib/resolver-compat.js:63:65)
    at /Users/matthewkocak/Documents/GitHub/dataform/bazel-bin/packages/@dataform/cli/bin.sh.runfiles/npm/node_modules/vm2/lib/resolver.js:84:14
    at VM2 Wrapper.apply (/Users/matthewkocak/Documents/GitHub/dataform/bazel-bin/packages/@dataform/cli/bin.sh.runfiles/npm/node_modules/vm2/lib/bridge.js:485:11)

I expected the error, since I added the testProperty to test.sqlx, but I expected the output text to reflect string changes that I had made in the table.ts. In the call stack, you can see that /my-project/node_modules/@dataform/core/bundle.js is being used to complete the compile step, rather than my local files.

When I run the run command using ./scripts/run run my-project, it appears that the compile step in this command still uses the published version of Dataform, but then the run/execution step of this command uses my local files. So I'm able to see the expected changes that I made locally for this step in the output.

Long post where I'm sure I'm just testing things incorrectly in my local development environment. How can I run the compile command in my local environment and have it reflect changes that I had made to the code?

@BenBirt
Copy link
Collaborator

BenBirt commented Jan 4, 2024

Thanks for the question! For the future - it might make more sense to post these kinds of things in "Discussions".

Firstly - it's important to note the difference between @dataform/core vs. @dataform/cli. If you make changes to (just) the CLI, ./scripts/run compile ... will use those changes. However, as per your question, if the changes are in @dataform/core, ./scripts/run compile ... will not automatically pick up your changes. This is because whatever directory you're pointing compilation at will have its own version of @dataform/core installed.

The simplest way around this is to simply overwrite that directory's core.bundle.js (for @dataform/core) with one built freshly: bazel build packages/@dataform/core:bundle.js && cp bazel-bin/packages/@dataform/core/bundle.js <target_directory>/node_modules/@dataform/core/ && ./scripts/run compile ....

It's probably reasonable to make scripts/run do this automatically - perhaps not as default behaviour, but in a flag to the script - I'd be happy to accept a PR for that, if you want to do it!

That said, obviously if you want to test things more thoroughly/systematically, please do add unit tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants