gunzip reads only the first chunk of files

[han-ysys]

hello,
I've been testing the AsyncGunzip with:

this.files.map(async (file) => {
  const gzip = new AsyncGunzip();
  const fileReader = file.stream().getReader();
  gzip.ondata = (_, data, final) => {
    if (data) {
      console.log(data);
    }
    if (final) {
      gzip.terminate();
      console.log('done');
    }
  };
  while (true) {
    const { done, value } = await fileReader.read();
    if (done) {
      gzip.push(new Uint8Array(0), true);
      break;
    }
    gzip.push(value);
  }
});

The files are about 6MB and AsyncGunzip outputs only the first chunk of files with a Uint8Array(65058)

After printing several empty Uint8Array, it says 'done'.

Another try without streaming stuff:

Promise.all(
  this.files.map((file) => {
    const reader = new FileReader();
    reader.readAsArrayBuffer(file);
    reader.onerror = () => {
      Notify.create({
        message: 'Failed to read file' + reader.error,
        color: 'negative',
      });
    };
    return new Promise<ArrayBuffer>((resolve) => {
      reader.onload = () => {
        resolve(reader.result as ArrayBuffer);
      };
    });
  })
).then((buffers) => {
  buffers.map((buffer) => {
    const bytes = new Uint8Array(buffer);
    console.log(bytes);
    const gunzip = new AsyncGunzip();
    gunzip.ondata = (err, data, final) => {
      if (err) {
        console.error(err);
      } else {
        console.log(data);
        if (final) {
          gunzip.terminate();
          console.log('done');
        }
      }
    };
    gunzip.push(bytes);
  });
});

This snippet produced logs like:

It is wired no 'done' is printed.
I've also tried the synchronous gunzip function and it acts the same.
What did I do wrong please?

[101arrowz]

You don't need to call terminate() manually unless you need to abort the process; workers are cleaned up automatically once the stream is done. What you've done won't break but it isn't needed.

Can you confirm that the following fails:

const results = await Promise.all(
  this.files.map(async file =>
    gunzipSync(new Uint8Array(await file.arrayBuffer()))
  )
);
console.log(results);

If you get the results you expect then there may be an issue with your code I'm not seeing. If that also doesn't work, I'll have to see the file to debug further.

[han-ysys]

Thanks for the quick reply.
The snippet you provide won't work either. It returns an even shorter part. :(

Tell me if you need to know more about my environment.

[han-ysys]

There are several gzipped files being decompressed.

❯ file YQ1_R1_001.fastq.gz
YQ1_R1_001.fastq.gz: gzip compressed data, original size modulo 2^32 37204

❯ file LS2042_FDDP202663764-1a_1.clean.fq.gz
LS2042_FDDP202663764-1a_1.clean.fq.gz: gzip compressed data, from Unix, original size modulo 2^32 0

❯ file alignment\ issue/FT2307_1.fq.gz
alignment issue/FT2307_1.fq.gz: Blocked GNU Zip Format (BGZF; gzip compatible), block length 10142

Out of the three files, only the second one (LS2042_FDDP202663764-1a_1.clean.fq.gz) can be decompressed successfully. The other two files behave the same way as I mentioned in the main thread.

All of them are valid fastq files when decompressed using the Archive Utility tool on Mac. I wonder how this issue may be derived from.

[101arrowz]

This might be the same issue as #102. I will try to fix this again, though I had trouble doing so last time I tried.

[han-ysys]

Appreciate the recent 0.8.0 update! I've encountered some challenges while utilizing the onmember property for processing multimember files. Specifically, I've noticed that AsyncGunzip.ondata receives a final=true value when it reaches the end of a member. Could you please clarify whether this behaviour is intended? Does this imply that I need to manually segment the file into blobs in order to read it? If not, I'm curious about the correct approach to accessing the data within streams.

[101arrowz]

Apologies for the delayed response - I think this was intended behavior but you bring up a good point that it's hard to know when the stream has actually ended. I'll rethink the API design here.

[101arrowz]

I actually haven't been able to reproduce this issue - it seems to get a true only on the very last chunk. Can you send a reproduction?

[han-ysys]

My bad. It works great now for me. Thanks for the help!