Recommended approach to generate URL valid base64 strings.

[axyz]

Hi, I am trying to use fflate to compress some text content in a way that can be embedded in a URL query param.
current working implementation: https://codesandbox.io/s/optimistic-vaughan-ub8gk?file=/src/index.ts

This implementation only works passing the true argument to compressSync and decompressSync functions in order to use latin characters instead of the full utf-8 range resulting on larger strings.

Ideally I would like to use the full utf-8 and use base64 to encode that as binary data, but it seems to fail due to "invalid characters"

My guess is that btoa and atob are not utf-8 aware and may fail with non ascii characters. What would be the suggested approach here? Using some utf-8 aware base64 conversion? Or maybe use the streaming API parsing 2 bytes at once? I'm sure it is a quite common use cases and there may be some working examples out there, so wanted to ask here before investigating further.

Thanks.

[101arrowz]

You're actually already using the most effective strategy possible for what you're trying to do. If you log the compressed and base64 encoded lengths, you'll see that btoa(strFromU8(compressedData, true)) is always 33% larger than compressedData, which is the optimal result for Base64 encoding. You're correct that btoa and atob don't support Unicode: they only support Latin-1, a 256 character subset. Regardless, if you try to do strToU8(strFromU8(compressedData)) it will actually not work because compressedData is not valid UTF-8 in the first place.

The one change you can make is how you encode the original string. Right now you're assuming your input is only Latin-1, i.e. adding emojis will break it. If you change your call for the string to Uint8Array conversion from strToU8(str, true) to strToU8(str), you get full unicode support. See this sandbox.

It's not optimal but for Base64 this is as good as it gets. Let me know if you have any questions.

[axyz]

Thanks a lot