feat: esm.sh support, add Fresh example

[meatcar]

esm.sh uses the User-Agent to determine the required target. Let's emulate deno's User-Agent to ensure that we get the same sources that deno receives, and the hashes align.

I'm trying to package an app that uses fresh and relies quite heavily on esm.sh.

This PR fixes the hash mismatch, but another error pops up that I can't intuit how to resolve. Any ideas?

$ nix flake check
error: builder for '/nix/store/hcfgbjicrk308hx1c1lpskhzfpg8qjwg-esm-simple.drv' failed with exit code 1;
       last 10 log lines:
       > Executing denoRestoreCacheHook
       > Finished denoRestoreCacheHook
       > Running phase: patchPhase
       > Running phase: updateAutotoolsGnuConfigScriptsPhase
       > Running phase: configurePhase
       > no configure script, doing nothing
       > Running phase: buildPhase
       > error: Expected a JavaScript or TypeScript module, but identified a Unknown module. Importing these types of modules is currently not supported.
       >   Specifier: https://esm.sh/v135/[email protected]/
       >     at file:///build/esm-simple/main.ts:1:19
       For full logs, run 'nix log /nix/store/hcfgbjicrk308hx1c1lpskhzfpg8qjwg-esm-simple.drv'.

As a side-note, the timing of this repo couldn't have been better, I was banging my head agains the wall a few days ago trying to get deno2nix to work. I really appreciate it, glad to help out in any way possible. 😁

[meatcar]

Ah, looks like Deno preserves the headers in the .metadata.json file, and uses content-type to determine the module type. Gotta figure out how to do that with nix machinery, the x-typescript-types header could be useful as well.

[nekowinston]

Hey, thanks for contributing so early!
I'm also still learning more about Deno internals as I'm playing around with this repo.

I quite naively assumed that esm.sh dependencies would work the same as deno.land 😅
Bit surprised that it will probably need a separate fetcher, looks like we'll have to write one from scratch, as I dont see a way in https://github.com/NixOS/nixpkgs/blob/4c501306af1ab6c19491fdafebb30fd097eb42c5/pkgs/build-support/fetchurl/builder.sh to pipe the header response to $out, passthru, or similar.
I'll try to have a look at this tomorrow, it's getting late here. 🙂

[meatcar]

I can get fetchurl to output just the headers with a -I curl flag. Including the headers with the request with -D - doesn't work, as fetchurl saves the request body with --output, and not just the stdout.

We can call fetchurl twice, once for the header and once for the body. A better option would do just one HTTP request. Maybe that would be a custom fetcher perhaps using deno internals for robustness? Maybe getting a patch into nixpkgs' fetchurl with some backwards compatibility considerations?

I'll think on it and make another attempt when I'm back at the keyboard.

[nekowinston]

I think we're possibly pretty stuck here, if we want to solve this cleanly. 😓

We can call fetchurl twice, once for the header and once for the body.

Storing the header data would change the output, so the sha256sum in deno.lock would differ from the output; the shasum only validates the actual dependency download, not any headers that are sent along with it.

We could make a special exception for esm.sh since it's so widely used in the Deno ecosystem, and try to assume response types in some cases. I've pushed some changes onto this branch to play around with this, the basic esm-simple check passes. I had to push the fresh check onto GitHub because my local nix on darwin runs into the old

sandbox-exec: pattern serialization length 69665 exceeds maximum (65535)

😅

Edit: I think I'm making progress with a similar strategy of assuming content-type from the file extensions, Fresh still isn't building because of other dependencies (esbuild), but I think I'm getting there.

[nekowinston]

We have a successful build! 🎉

I hope that if Fresh manages to build with its relative complexity, that it's good enough for 99% of cases. 🤞

[meatcar]

This is awesome! You're so fast! I've got some ideas to improve packaging Fresh, but I'll open a separate issue.

EDIT: Never mind, looks like #9 addresses them. Cheers!

[nekowinston]

Glad it seems to work for you. 😄

I've got some ideas to improve packaging Fresh, but I'll open a separate issue.

Please do open that issue, I'd love to get input to make these builders as useful as possible! I'm looking at the nixpkgs NPM/Rust/Flutter builders for inspiration, in addition to Deno specific stuff like the bin wrappers:

nix-deno/nix/hooks/deno-install-hook.sh

Lines 13 to 21 in bb2771a

	mkdir -p "$out/bin"
	makeWrapper \
	@hostDeno@ \
	"$out/bin/@binaryName@" \
	--set DENO_DIR "$depsOut" \
	--set DENO_NO_UPDATE_CHECK 1 \
	--set DENO_REPL_HISTORY "" \
	--set NPM_CONFIG_REGISTRY "$npmRegistryUrl" \
	--add-flags "@denoFlags@"