Deterministic Zip (Reproducible Checksum)

[beanaroo]

❤️ First of all, thank you for such a well documented library!
I do not consider myself proficient in typescript/javascript but I was able to make quick and easy use of fflate.

Use Case

I would like to produce deterministic zip archives of source code to prevent unnecessary redeploys of AWS Lambda Functions and Layers. This can be done in Terraform by specifying source_code_hash of the upload.

Alternatives

I have achieved this in the past using shell scripts or Python

Looking for a native Node.js method, I came across deterministic-zip and had success after applying @troyready's patch (and converting to Typescript).

Trying fflate

I was able to achieve the desired effect with the following:

import Fs from '@supercharge/fs';
import { type Zippable, zipSync } from 'fflate';
import path from 'path';

/**
 * Produce a deterministic zip archive from directory contents
 * to prevent unnecessary redeploys. (Consecutive checksums should match)
 *
 * - Sorts files before zip
 * - Overrides file modification time
 *
 * @param outFile Filepath of zip file to produce
 * @param srcDir Path of directory to zip
 * @param destDir Destination directory inside the zip
*
 * @example Package dependencies for Lambda Layer
 * ```ts
 * await zipDir('layer.zip', 'node_modules', 'nodejs/node_modules');
 * ```
 */
export const zipDir = async (outFile: string, srcDir: string, destDir = '') => {
  const files = await Fs.allFiles(srcDir);
  const zipContent: Zippable = {};
  files.sort().forEach((file) => {
    const zippedPath = path.join(destDir, path.relative(srcDir, file));
    const fileBuf = Fs.readFileSync(file);
    zipContent[zippedPath] = fileBuf.valueOf();
  });
  const zipped = zipSync(zipContent, {
    level: 4 as const,
    mtime: new Date('1/1/1980'),
  });
  Fs.writeFileSync(outFile, zipped);
};

Verified using zipcmp and sha256

Performance

I have not yet attempted to optimize the above by trying things like async zip and/or streaming. The above performs well enough for my needs (archiving node_modules), compressing 18k files to 85MB in ~25 seconds.

Observation

Passing in fileBuf: Buffer resulted in different CRCs on each invocation. Switching to fileBuf.valueOf(): UInt8Array did the trick.

---

💡 Sharing in case somebody finds this useful or would like to critique.