Make scripts more powerful

[nmattia]

I've been wanting to extract more value from the dfx toolchain in builds. I think dfx could be useful to build and deploy more than simple canisters. In particular I think the interface with user-defined scripts in custom builds could be improved.

In general I think scripts could leverage environment variables to drive the build. Here's something that just isn't doable right now:

Assume two custom canisters; frontend and backend. The frontend canister requires knowing the backend canister ID (for agent-js to make calls), and assume extra heavy assets need to be uploaded to the frontend canister via HTTP after the canister was installed. With a few tweaks to the dfx toolchain, the build, preinstall and postinstall scripts could look like this:

Simplified & robust build

# dfx.json build for frontend
npm run build
cp dist/* "$DFX_BUILD_OUTPUT"

Note: Here the build script (specified in dfx.json) leverages DFX_BUILD_OUTPUT (set by dfx) to avoid having to specify a wasm: value in dfx.json which has to be kept in sync with the script. Exact semantics may depend based on canister type (e.g. custom may simply copy a single wasm file)

Dynamically setting canister IDs in preinstall

# dfx.json preinstall for frontend
echo "dfx_canister_args::$(didc encode '(record { backend_id = "'"$DFX_CANISTER_ID_backend"'" })')"

Note: Here the preinstall script would leverage two new features:

1. If frontend specifies a dependency on backend, then frontend could read some information about the already deployed backend canister, like its canister ID (through the DFX_CANISTER_ID_<canister ID> environment variable)
2. Inspired by GitHub Actions and build.rs output instructions, the script could set candid arguments for the frontend canister dynamically. This is highly desirable to not have to bake canister IDs in canisters (see also: Don't incite developers to bake the canister ID in the frontend #2628, Dynamic Candid arguments #2626) (note that this would be made nicer even with something like Advanced Candid argument operations #2623 for didc)

Connecting to the canister postinstall

# dfx.json postinstall for frontend
curl --data-binary "@./assets/terribly-big-banner.png" "$DFX_CANISTER_URL/upload"

Note: Here the post install script would read information learned during the canister installation (or rather, creation) to perform some (ideally idempotent) operations like uploading assets. Proper local URLs when developing (without &canisterId hacks) may need to be implemented to make this work seamlessly both when deploying to mainnet and when deploying to a locally running replica: #2651

---

Goes without saying that I'm happy to contribute code, just want to agree with the team first on the direction.

[bitdivine]

Note that canister IDs are already available as env vars, so there is already a solution IF the canister IDs need to be compiled in.

That said, I like the idea of being able to specify other canister IDs as install args. I don't actually know whether arguments can be passed during upgrade, in case the list of canister dependencies changes.

[nmattia]

Note that canister IDs are already available as env vars, so there is already a solution IF the canister IDs need to be compiled in.

To clarify, I would need canister IDs in a as-of-yet-non-existent pre-install script, which could also set canister arguments. I definitely do not want canister IDs to be compiled in.

[bitdivine]

Please note that the sdk team is somewhat resource constrained. Maybe a good way of getting things done is to agree a strategy with the community & SDK team, then submit PRs. The CI is also thorough but slow, some love to make it faster would make the sdk team faster.

[adamspofford-dfinity]

The variable CANISTER_ID_backend is already available, as is CANISTER_CANDID_PATH_backend, CANISTER_ID/CANISTER_CANDID_PATH (for self), DFX_NETWORK, and DFX_VERSION.

[sesi200]

But isn't that only during build and not for arguments specified in dfx.json?

[nmattia]

The variable CANISTER_ID_backend is already available

Same reply as to @bitdivine above:

I would need canister IDs in a as-of-yet-non-existent pre-install script, which could also set canister arguments. I definitely do not want canister IDs to be compiled in.

Moreover, I think it is not a good idea to have the canister IDs available in the build scripts: the canister may not have been created yet, and it incites developers to bake the canister ID at build time

[sesi200]

Right now canisters won't even attempt build if they don't have a canister ID defined yet

[nmattia]

Why is that? Why would a node project need to know the final URL where it'll be hosted just so that you can run npm build? I believe we really need to stop thinking of "the canister ID" as a build input, and instead allow canisters to be built and tested without making any assumptions about where they'll be running; doing anything else is throwing away decades of software best practices

[bitdivine]

Is it worth splitting this up and moving forward on the uncontentious parts while we discuss the big vision behind the harder parts? For example, I see nothing at all wrong with $DFX_BUILD_OUTPUT. The only question is whether we should assume that jq is part of the developer's toolchain, in which case one could argue that jq .canisters[env.DFX_CANISTER].wasm dfx.json` is enough. I would say that this is a worthwhile addition. Can we have a thumbs up/downvote on this?

Note: There is an existing env var CANISTER_CANDID_PATH_{canister.name} so presumably to match we would name this var CANISTER_WASM_PATH_{canister.name}.

[nmattia]

Yes, most definitely. Although to be fair, there's already tickets about most of the points mentioned above (see links in first message) although I wanted to give an idea of the bigger vision/integration to incite people to act on said tickets :) Will nonetheless start by creating tickets for the following:

• DFX_BUILD_OUTPUT Add environment variable for expected build output path #2684
• canister URLs
• "pre_install" script + dynamic candid args

[bitdivine]

Likewise, can we have a vote on DFX_CANISTER_URL? I think we need to unpack this a bit. What is the URL of the nns-dapp?

• https://nns.ic0.app ? If yes, where should that be specified? Note that the URL will differ by network.
• https://qoctq-giaaa-aaaaa-aaaea-cai.ic0.app ? Again, if this, where is the network URL (https://ic0.app) specified?

Note: I personally think that the URL should be the normal login URL used by end users, so https://nns.ic0.app in the above example. For canisters without a custom domain there is no distinction but where there is a custom domain, using the common URL is better IMO.

Note: I would love to have a cli command: dfx canister url CANISTER_NAME

Question: Should there be a URL env var per canister, as for canister IDs? I suspect that the answer is yes.

Waiting with a vote until this has been thrashed out a bit.

[bitdivine]

Note: Env vars are published here: https://internetcomputer.org/docs/current/references/cli-reference/dfx-envars/