Selective binary by OS and Node version 🤔

[rtritto]

After uWebSockets.js is installed, all OS and Node versions are built.

Reproduction

• yarn init -y
• yarn set version berry
• yarn add uWebSockets.js@github:uNetworking/uWebSockets.js#v20.49.0
• open directory /.yarn/unplugged/uWebSockets.js-https-<HASH>/node_modules/uWebSockets.js
• .node files are 24 (uws_darwin_*, uws_linux_*, uws_win32_*)

Expected

In directory /.yarn/unplugged/uWebSockets.js-https-<HASH>/node_modules/uWebSockets.js there is only 1 file filtered by OS and Node version

[uNetworkingAB]

It downloads all binaries because npm client has no idea of the platform, no support to download only a subset of binaries like literally all other package managers on earth do. So it's a tradeoff between being easily downloaded with npm client and saving disk space.

[rtritto]

So the issue is both npm and yarn.
I opened an issue to Yarn Berry yarnpkg/berry#6565

[rtritto]

@uNetworkingAB

yarnpkg/berry#6565 (comment)

It's not an issue in Yarn Berry or any other package manager. The issue is how uNetworking/uWebSockets.js is distributed. They don't use os and cpu fields of package.json and instead distribute all platform specific files in a single package:
https://github.com/uNetworking/uWebSockets.js/tree/v20.49.0

[larixer]

It downloads all binaries because npm client has no idea of the platform, no support to download only a subset of binaries like literally all other package managers on earth do.

The platform can be communicated to npm client via os and cpu fields in package.json. The JavaScript package managers should also support installing packages from GitHub subfolders. Combining these two features might work in your case to decrease network bandwidth during package installation at a price of some complication of releasing process on GitHub. E.g. you can keep binaries in some branch, but use multiple subfolders, like native-linux-x64, native-darwin-x64, etc, each subfolder having package.json with os and cpu fields for the target platform. Platform-agnostic and user-facing package can be kept in some other branch and should be tagged on each release as you do it now. User-facing package can then have all binary packages corresponding to its version as dependencies or optionalDependencies. Will this work well with all JavaScript package managers or not is hard to tell, in theory - it should, I am not aware of the package that do it already, so, YMMV and it's an open question if additional complexity for distribution is worth it.

[uNetworkingAB]

Very interesting! Do you think it will work with the same user experience?

npm install uNetworking/uWebSockets.js#v20.48.0

and

require("uWebSockets.js")

?

[larixer]

@uNetworkingAB Yes, it should work with the same user user experience.

[porsager]

It will still download the various abi versions, but that's still an improvement! I can make a quick sample setup for you to see

[porsager]

In my projects I'm just using this loader and a postinstall to only download a single binary, so that's an option too:

import fs from 'node:fs'
import path from 'node:path'
import https from 'node:https'
import { pipeline } from 'node:stream/promises'
import { createRequire } from 'node:module'

const binaryName = `uws_${ process.platform }_${ process.arch }_${ process.versions.modules }.node`
const binaryPath = path.join(import.meta.dirname, binaryName)

fs.existsSync(binaryPath) || await download()

let uws
try {
  uws = createRequire(import.meta.url)(binaryPath)
} catch(e) {
  await download()
  uws = createRequire(import.meta.url)(binaryPath)
}

export default uws

async function download(url = 'https://raw.githubusercontent.com/uNetworking/uWebSockets.js/v20.47.0/' + binaryName, retries = 0) {
  return new Promise((resolve, reject) => {
    https.get(url, async res => {
      if (retries > 10)
        return reject(new Error('Could not download uWebSockets binary - too many redirects - latest: ' + res.headers.location))

      if (res.statusCode === 302)
        return (res.destroy(), resolve(download(res.headers.location, retries + 1)))

      if (res.statusCode !== 200)
        return reject(new Error('Could not download uWebSockets binary - error code: ' + res.statusCode + ' - ' + url))

      pipeline(res, fs.createWriteStream(binaryPath)).then(resolve, reject)
    })
    .on('error', reject)
    .end()
  })
}

If the postinstall doesn't run, it'll download it on first run after instead.

[larixer]

@porsager @uNetworkingAB I have put together a demo repo:
https://github.com/larixer/multi-platform

You can check how it works via:
npm init
npm add larixer/multi-platform#1.0.0

You can then go into node_modules and see that user-facing multi-platform and one for your current platform package is installed, provided, if you are on Linux, Win32 or Mac.

Note, that I have used 4 branches, 1 for user-facing package main, and 3 other for linux-x64, darwin-x64 and win32-x64, because in fact support for Git subfolders is not that great in JS package managers, but they support tags and branches and this should be enough.

[rtritto]

There is a way to select also by Node version?

[larixer]

There is a way to select also by Node version?

There is engines field where you can put required Node version, but as far as I know npm will not skip installing the dependency if Node version is not compatible, it just warns about Node version mismatch, so it's not possible to use engines like os and cpu to selectively skip dependency installation.

[rtritto]

There is a way to select also by Node version?

Can we use the release tag?

Eg:

• v20.49.0 (default) OR v20.49.0-22 for latest Node version
• v20.49.0-21 for Node v21
• v20.49.0-20 for Node v20
• v20.49.0-18 for Node v18

[uNetworkingAB]

You can have multiple nodejs versions on the same platform so installing all 3 is preferrable IMO.

Thanks for making the demo, it works for me on Linux but I had issues getting it to work on arm64 macOS. I made a fork but could not get it to work for some reason. Anyways -

If someone wants to make a full demo, using the actual binaries in latest release, and making it so that require("uWebSockets.js") works on macOS arm64, x64, Windows x64, Linux x64, arm64 - please do so and provide a branch I can test against. If that branch works, I will definitely move to this approach, as it would be a seamless optimization.

My point re. full demo is that I do not have time to play with this and this is a very low prio issue. So I can take a look at the finished full demo if someone makes it.

[webcarrot]

Off topic: Binary without SSL and H3 (pure http 1.1) would also be nice.

[uNetworkingAB]

That's way too specific. If you value disk space, use ws.

[porsager]

Ok - I've made 2 PRs ready master branch changes and binary branch changes, and a sample release on my fork you can try out.

npm i porsager/uWebSockets.js#v20.50.0

I've tried to keep it as close to the current setup as possible.

• node_modules folder size goes from 139mb to ~6mb
• on my sloppy 5g internet connection install time goes from ~25s to ~6s
• running a release after binaries have been built takes ~1min

I've left the current build.yml script completely as it is, and added a new release script that allows you to release by running it from the actions tab and inputting the version you want to release. It takes care of making git tags and individual package.jsons for every single binary, and the main package.json with optionalDependencies that makes package managers only install the needed binary.

uws-release.mp4

If anything fails you can simply delete the tags, and run again.

Only downside is all the friggin tags, but I think that's definitely worth the tradeoff.

The only maintenance should be in adding new ABI versions to the version map in release.yml when a new node version should be supported.