Use transparent zstd over HTTP for fetching and exporting packages

[stapelberg]

• figure out how zstd over HTTP is typically done for maximum compatibility
  • the client sends e.g. an Accept-Encoding: gzip, deflate header
  • the server replies with a Content-Encoding: gzip header
  • zstd is the value that can be used for zstandard as per RFC 8478, section 6.2
    • Upstream discussion at Content-Encoding and other HTTP Headers facebook/zstd#315
    • not yet documented e.g. on MDN
• distri export should support this
  • blocked on Add zstandard (zstd, .zst) support lpar/gzipped#14 being merged
• distri HTTP client should support asking for zstd versions
• zstd-compress all packages of the current release
• ensure nginx behind repo.distr1.org serves zstd files when asked with the appropriate header, i.e. build https://github.com/tokers/zstd-nginx-module
• CloudFlare seems to strip Accept-Encoding: zstd right now: https://twitter.com/zekjur/status/1266855979041292290
• recommend mirror operators support zstd
• In addition to accept-encoding, we could consider fetching the corresponding .zst file explicitly for squashfs files, perhaps in a happy eyeballs-like scheme. This increases support across the whole landscape, e.g. with mirrors that don’t support accept-encoding yet, or CDNs or proxies etc.
• we should add a test to verify that zstd is faster :)

This exploration shows how zstd beats both sequential gzip and parallel gzip significantly for decompressing larger packages such as qemu:

We took a few measurements with the respective uncompress tools on stream: https://youtu.be/dLr_6jJ4N7Y?t=4172

• gunzip: 2.95s (100%)
• unpigz: 1.50 (50%)
• unzstd: 0.51s (17%)

…and then measured distri package installation, which confirm zstd’s effectiveness in our use-case:

• qemu with gzip: 2020/05/30 17:24:30 done, 359.07 MB/s (1149903681 bytes in 3.05s)
• qemu with zstd: 2020/05/30 17:24:39 done, 603.67 MB/s (1149903681 bytes in 1.81s)