You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hello, this is a question rather than an issue. I'm becoming more familiar with the code base and have been testing changes that I've been making locally. I've been having trouble figuring out how I can test compile step code changes locally, though.
I've completed the setup as described in the contributing documentation and generally things are working well. However, I haven't managed to figure out how to test changes regarding the compile step.
Here's what I've done:
Create a new project with ./scripts/run init bigquery my-project --default-database my-project --default-location EU
Create credentials file with ./scripts/run init-creds bigquery my-project
Add the following test.sqlx file to my-project/definitions:
Make some changes to table.ts that I would expect to be reflected when compiling this file (console.logs, updating strings, etc.)
Run ./scripts/run compile my-project
Instead of seeing the changes that I would expect, it appears that the compile command is using code from /my-project/node_modules/@dataform/core/bundle.js (so the published version of Dataform), rather than using the code that I modified. Below is the output that I see after running the command:
Compiling...
Compiled 1 action(s).
1 dataset(s):
dataform.test [table]
Compilation errors:
definitions/test.sqlx: Error: Unexpected property "testProperty" in table config. Supported properties are: ["type","disabled","protected","name","redshift","bigquery","snowflake","sqldatawarehouse","presto","tags","uniqueKey","dependencies","hermetic","schema","assertions","database","columns","description","materialized"]
at Object.t.checkExcessProperties (/Users/matthewkocak/Documents/GitHub/dataform/my-project/node_modules/@dataform/core/bundle.js:1:209544)
at a.config (/Users/matthewkocak/Documents/GitHub/dataform/my-project/node_modules/@dataform/core/bundle.js:1:252775)
at t.Session.sqlxAction (/Users/matthewkocak/Documents/GitHub/dataform/my-project/node_modules/@dataform/core/bundle.js:1:383183)
at /Users/matthewkocak/Documents/GitHub/dataform/my-project/definitions/test.sqlx:1:24
at VM2 Wrapper.apply (/Users/matthewkocak/Documents/GitHub/dataform/bazel-bin/packages/@dataform/cli/bin.sh.runfiles/npm/node_modules/vm2/lib/bridge.js:485:11)
at NodeVM.run (/Users/matthewkocak/Documents/GitHub/dataform/bazel-bin/packages/@dataform/cli/bin.sh.runfiles/npm/node_modules/vm2/lib/nodevm.js:497:23)
at CustomResolver.loadJS (/Users/matthewkocak/Documents/GitHub/dataform/bazel-bin/packages/@dataform/cli/bin.sh.runfiles/npm/node_modules/vm2/lib/resolver.js:210:6)
at CustomResolver.loadJS (/Users/matthewkocak/Documents/GitHub/dataform/bazel-bin/packages/@dataform/cli/bin.sh.runfiles/npm/node_modules/vm2/lib/resolver-compat.js:63:65)
at /Users/matthewkocak/Documents/GitHub/dataform/bazel-bin/packages/@dataform/cli/bin.sh.runfiles/npm/node_modules/vm2/lib/resolver.js:84:14
at VM2 Wrapper.apply (/Users/matthewkocak/Documents/GitHub/dataform/bazel-bin/packages/@dataform/cli/bin.sh.runfiles/npm/node_modules/vm2/lib/bridge.js:485:11)
I expected the error, since I added the testProperty to test.sqlx, but I expected the output text to reflect string changes that I had made in the table.ts. In the call stack, you can see that /my-project/node_modules/@dataform/core/bundle.js is being used to complete the compile step, rather than my local files.
When I run the run command using ./scripts/run run my-project, it appears that the compile step in this command still uses the published version of Dataform, but then the run/execution step of this command uses my local files. So I'm able to see the expected changes that I made locally for this step in the output.
Long post where I'm sure I'm just testing things incorrectly in my local development environment. How can I run the compile command in my local environment and have it reflect changes that I had made to the code?
The text was updated successfully, but these errors were encountered:
Thanks for the question! For the future - it might make more sense to post these kinds of things in "Discussions".
Firstly - it's important to note the difference between @dataform/core vs. @dataform/cli. If you make changes to (just) the CLI, ./scripts/run compile ... will use those changes. However, as per your question, if the changes are in @dataform/core, ./scripts/run compile ... will not automatically pick up your changes. This is because whatever directory you're pointing compilation at will have its own version of @dataform/core installed.
The simplest way around this is to simply overwrite that directory's core.bundle.js (for @dataform/core) with one built freshly: bazel build packages/@dataform/core:bundle.js && cp bazel-bin/packages/@dataform/core/bundle.js <target_directory>/node_modules/@dataform/core/ && ./scripts/run compile ....
It's probably reasonable to make scripts/run do this automatically - perhaps not as default behaviour, but in a flag to the script - I'd be happy to accept a PR for that, if you want to do it!
That said, obviously if you want to test things more thoroughly/systematically, please do add unit tests.
Hello, this is a question rather than an issue. I'm becoming more familiar with the code base and have been testing changes that I've been making locally. I've been having trouble figuring out how I can test compile step code changes locally, though.
I've completed the setup as described in the contributing documentation and generally things are working well. However, I haven't managed to figure out how to test changes regarding the compile step.
Here's what I've done:
./scripts/run init bigquery my-project --default-database my-project --default-location EU
./scripts/run init-creds bigquery my-project
test.sqlx
file tomy-project/definitions
:table.ts
that I would expect to be reflected when compiling this file (console.logs, updating strings, etc.)./scripts/run compile my-project
/my-project/node_modules/@dataform/core/bundle.js
(so the published version of Dataform), rather than using the code that I modified. Below is the output that I see after running the command:I expected the error, since I added the
testProperty
totest.sqlx
, but I expected the output text to reflect string changes that I had made in thetable.ts
. In the call stack, you can see that/my-project/node_modules/@dataform/core/bundle.js
is being used to complete the compile step, rather than my local files.When I run the run command using
./scripts/run run my-project
, it appears that the compile step in this command still uses the published version of Dataform, but then the run/execution step of this command uses my local files. So I'm able to see the expected changes that I made locally for this step in the output.Long post where I'm sure I'm just testing things incorrectly in my local development environment. How can I run the compile command in my local environment and have it reflect changes that I had made to the code?
The text was updated successfully, but these errors were encountered: