Decompress gz JSON without JSON.parse

[jimmykane]

Hi there guys,

We are really impressed by this lib. We were used to PAKO and wow , the JS web needed something like this.

We would like to parse some compressed and huge JSONs but typically (ie with Pako) you would do something like this

string = uncompress(data)

Then you would need to JSON.parse(string)

That results into high memory usage.

Without trying to use the lib and in regards to using streams when parsing the data , is it possible to get a plain JS object (JSON) , without having to cast to the string first ?

[101arrowz]

First I want to make sure you've verified that high memory usage is actually a problem. JSON strings are almost always less memory-intensive than the JS object they represent, i.e. '{"a": "1234"}' actually costs way less memory than the JS object {a: "1234"} because object keys and strings are UTF-16 in JavaScript, while most JSON data is UTF-8. After parsing the JSON string, it should be garbage collected (if not, you might be able to use an inner scope to force it).

If this is actually a problem, you will need to use a parser built in JavaScript. As for streaming, I'm not sure how useful that would be, given that the entire point is to reduce memory usage but you will need the full object at the end anyway. You should accumulate into a single Uint8Array and pass it through this. But honestly a string shouldn't have so much memory usage over just the Uint8Array (at most 2x, for JSON scale this isn't usually so bad).

[jimmykane]

Thanks a lot for the reply @101arrowz really appreciated. Indeed we need first to very the problem.

I ll come back with more reports if needed but this answer is more than sufficient