Z_SYNC_FLUSH support

[BenoitZugmeyer]

What can't you do right now?

I have this use case. I'd like to concatenate deflate streams, so I need to use Z_SYNC_FLUSH before finishing the stream to have deterministic trailing bytes. Without this, I would have to inflate the stream to know where it finishes.

An optimal solution

When invoking .push(...), add a parameter to ask for a sync flush, ex:

stream.push(data, SYNC_FLUSH)

How is this done by other libraries?

Pako allows to pass an arbitrary flush mode to push: https://github.com/nodeca/pako/blob/62cb729e7813176ce2d2694b89c8724680fca383/lib/deflate.js#L178-L200

[101arrowz]

You can just call stream.flush() to flush the stream.

const stream = new fflate.Deflate((chunk, final) => {
  /* handle stream chunks here */
});

stream.push(data);
stream.flush();

[BenoitZugmeyer]

Thank you! I tried your suggestion but I didn't manage to get something working. Here is my test case, replicating the Stack Overflow answer but with "raw" deflate to avoid dealing with headers and adler32 trailers:

import * as fflate from "fflate";
import * as pako from "pako";
import * as zlib from "zlib";

async function test(name, compress) {
  // Compress data with the provided implementation.
  const compressedBuffer = await compress(
    new TextEncoder().encode("Hello, World!"),
  );

  // Concatenate the compressed data twice, and add an empty block at the end.
  const full = concatBuffers([
    compressedBuffer,
    compressedBuffer,
    new Uint8Array([0x03, 0x00]),
  ]);

  try {
    console.log(name.padEnd(6, " "), zlib.inflateRawSync(full).toString());
  } catch (e) {
    console.log(name.padEnd(6, " "), "error:", e.message);
  }
}

test("fflate", (inputBuffer) => {
  let chunks = [];
  const deflate = new fflate.Deflate((data) => {
    chunks.push(data);
  });
  deflate.push(inputBuffer);
  deflate.flush();
  return concatBuffers(chunks);
});

test("pako", (inputBuffer) => {
  const deflate = new pako.Deflate({ raw: true });
  deflate.push(inputBuffer, pako.constants.Z_SYNC_FLUSH);
  return concatBuffers(deflate.chunks);
});

test("zlib", (inputBuffer) => {
  return new Promise((resolve) => {
    let chunks = [];
    const deflate = zlib.createDeflateRaw();
    deflate.on("data", (chunk) => {
      chunks.push(chunk);
    });
    deflate.write(inputBuffer);
    deflate.flush(zlib.constants.Z_SYNC_FLUSH, () => {
      resolve(concatBuffers(chunks));
    });
  });
});

function concatBuffers(buffers) {
  const totalLength = buffers.reduce((acc, bytes) => acc + bytes.length, 0);
  const result = new Uint8Array(totalLength);
  let offset = 0;
  for (const bytes of buffers) {
    result.set(bytes, offset);
    offset += bytes.length;
  }
  return result;
}

When run in Node.js, it prints:

fflate error: invalid distance too far back
pako   Hello, World!Hello, World!
zlib   Hello, World!Hello, World!

I'm probably missing something... do you think this is achievable with fflate now?

[101arrowz]

You are correct, this is not currently possible with fflate. The fix should be relatively simple, I'll look into it for the next version.

[BenoitZugmeyer]

I know this is a niche use-case, thank you very much for considering this!