[RRFC] Support installing from only a lock file

[gingermusketeer]

Motivation ("The Why")

When developing in a docker environment, rebuilding images and installing dependencies can be quite slow. Using best practices it is recommended to have a docker file with the following steps:

1. Copy package.json and package-lock.json files into image
2. Install dependencies
3. Copy rest of application

Doing it this way ensure that we can skip step 2 if nothing has changed in step 1. Ideally this means that step 1 should only contain the minimum to install the dependencies in step 2.

For npm v9 the minimum needed to be able to install dependencies in a production like setting such as a docker file is both the package.json and package-lock.json.

Relying on package.json has some downsides because it is used for a lot of other configuration:

• npm scripts
• eslint configuration
• jest configuration
• repository and licence information.

If any of these change then step 2 of the docker build cannot be skipped.

Example

• Solving this would speed up docker builds for people where package.json has changed but package-lock.json has not changed.

How

Current Behaviour

• Currently both package.json and package-lock.json is needed to be able to install packages.

Trying to install with only a package-lock.json file present results in the following error:

npm ERR! code ENOENT
npm ERR! syscall open
npm ERR! path /path/package.json
npm ERR! errno -2
npm ERR! enoent ENOENT: no such file or directory, open '/path/package.json'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent 

Desired Behaviour

• npm install should support installing from only a package-lock.json
• npm ci should support installing from only a package-lock.json.

References

• Potentially related RRFC [RRFC] npm install --from-lockfile #415
• Recommended docker file for node

[ljharb]

A lockfile doesn't have sufficient information. Why would it be valuable to duplicate the information in package.json into the lockfile?

It seems like a smarter diffing algorithm would help you here - for example, don't just diff the files, but delete the keys you don't care about, sort the remaining top-level keys, and then diff them.

[gingermusketeer]

A smarter diffing algorithm would definitely be an alternative. It does make it more complicated however as it would necessitate a tool being installed or copy pasted between projects.

When you say that the information in a lock file doesn't have sufficient information, what is missing? My understanding was that lock files had everything necessary.

As of npm v7, lockfiles include enough information to gain a complete picture of the package tree, reducing the need to read package.json files, and allowing for significant performance improvements.

source

This suggests the need for a package.json are reduced so reducing further might be possible.

Thanks for taking the time to read the proposal!

[ljharb]

reducing, not eliminating, is what i read from that.

[gingermusketeer]

After a big more learning and experimenting related to this I have a work around which involves:

1. Generating a package-docker.json file on postinstall of the package.json
2. Copying package-docker.json in place of package.json prior to the dependency install step.
3. Installing dependencies
4. Replacing minimal package.json with actual file so npm scripts etc are available.

Code

The postinstall step is:

 "postinstall": "node -e \"require('node:fs').writeFileSync( 'package-docker.json', JSON.stringify({...require('./package-lock.json').packages[''], overrides: require('./package.json').overrides}, null, 2) );\""

In a more readable format:

require("node:fs").writeFileSync(
  "package-docker.json",
  JSON.stringify(
    {
      ...require("./package-lock.json").packages[""],
      overrides: require("./package.json").overrides,
    },
    null,
    2
  )
);

For step 1 I was able to generate this completely from the package-lock.json file except for overrides so it looks like most of the information needed is in the package-lock.json.

The problem with this workaround is that it will most likely break tools like dependabot, renovate or any other tool which only expects package-lock.json to change when package.json changes.