Thoughts on "Content-defined Merkle Trees for Efficient Container Delivery" (2021)

[fenollp]

Today there is layer reuse while pushing or pulling images. However, layers may differ by just a few percent of their contents.
The article at https://arxiv.org/abs/2104.02158 describes a Merkle tree on the contents of each layer.

What's the team's thoughts on this potential technique?

I was thinking it might help with

• Incremental export transfer #1224
• users of cache-from

[AkihiroSuda]

Chunk-level dedupe is already supported with Nydus: https://github.com/moby/buildkit/blob/master/docs/nydus.md

However, Nydus isn't popular as its layer blobs are not compatible with OCI tar.gz layers.

There was a brainstorming to bring chunk-level dedupe to OCI "v2", but the discussion has got stuck: http://web.archive.org/web/20200919032824/https://hackmd.io/@cyphar/ociv2-brainstorm#Reduced-Duplication