The S5 Network: An IPFS alternative that breaks the spec

[redsolver]

Hey everyone, I've been loosely following the development of Iroh and was curious to see if this project could actually get IPFS to scale efficiently. Some months ago after some research and testing I arrived at the conclusion that the core spec of IPFS is not really efficient and can't scale well enough, so I decided to build my own "flavor" of IPFS. The core differences are:

• everything happens on the file level, not block-level. so a giant 1 TiB file only needs a single lookup on the p2p network
• verified streaming is still possible, because S5 uses the blake3 hash function, which is a lot faster than sha256 and a merkle tree on the inside, which makes the streaming possible
• S5 uses HTTP/s (by default) to send file data to clients instead of over a p2p connection. This makes it work in the browser by default and makes it easy to utilize existing CDN infrastructure
• to repeat, the p2p network is only there to answer the question "which HTTP/S urls can be used to download this specific hash"
• because the p2p messages are really lightweight, they can be relayed with minimal resource usage and less total connections are needed
  the file concept is explained here: https://docs.s5.ninja/concepts/content-addressed-data.html

Here's a demo web app that streams large video files directly from the S5 network (so not proxied using a gateway) and fully verifies the integrity of every single byte using BLAKE3/bao in Rust compiled to WASM: https://tube5.app/

After reading https://n0.computer/blog/a-new-direction-for-iroh/, I noticed that you seem to have encountered a lot of similar issues that I tried to solve with my project. I would be really interested to know how you are approaching a new IPFS 2.0 from the technical side and maybe there might even be some possibility to work together :)

[b5]

Woooooooah cool! There's a lot of overlap here. For example:

• We're also big bao fans (to the point of forking to make it work (async)[github.com/n0-computer/abao/])
• Fully agreed on one identifier per file (byte blob in our case)

• We're looking harder at QUIC than HTTP/s, but I will admit I'm very interested to hear how your experience with HTTP/s has gone 😄
• The draft design we're working on takes a very similar cut between content routing & data transfer: p2p system is a giant pointer machine that delegates to some other mechanism for actually moving bytes. We're still thinking DHT

I'm going to make a point of reading through the S5 docs & will report back here, but if you want to email my github handle at n0.computer, I'd love to set up some time to chat!

[redsolver]

• awesome, I'm using a fork of bao too that increases the hash chunk size to 256 KiB to reduce the metadata that needs to be stored in addition to the byte data. with default bao settings it would be ~6% overhead and allow streaming/verification on the 1024-byte level. Fortunately the chunk size for bao doesn't matter too much, because the BLAKE3 hash and CID stay the same if it changes, so that's convenient - different nodes could use different settings for it and all files/byte-data would still have the same CID if the content is the same. S5 attaches the bao outboard bytes of a file to the end of it when storing it and S5 CIDs include the file size to make range requests easy that only fetch the bao outboard part of a stored file
• S5 does byte blobs too, things like filenames and mime types are not stored in the file itself, to make deduplication on the network a lot more efficient. Instead, file extension and name can be (optionally) attached to the CID or just be included in a more complex metadata format, like the one used on tube5.app. it supports multiple resolutions, audio/video/test formats and stuff like that. I made a little CID explorer tool for it: https://cid.one/#z5TTp1umafoRLvi8VHEU6fXtUxkpjas2N5KPKhZyaLff9ZZ5
• For p2p connections, TCP is used atm but I planned to add support for QUIC and WebSocket (for browser-based nodes). HTTP/HTTPS is used to download and stream files/byte-data from storage providers, like S3 buckets, CDNs, local storage, Arweave, Estuary (Filecoin) or the Sia Network. From my understanding HTTP/3 uses QUIC under the hood, so it should provide most advantages of QUIC while still being fully backwards-compatible with HTTP/2 and HTTP/1.1
• The S5 network does not use a DHT yet (all queries are just flooded to all nodes atm), but it will likely be required for scaling the network to a ton of peers efficiently. S5 is still pretty early and I'm trying to focus on a solid base spec first (like blake3/bao and CID formats) and then most of the specific protocol details can still be improved later - it just needs to be designed with upgradeability in mind. speaking of upgradeability, all S5 CID formats are designed to not collide with any IPFS or IPLD formats.

[pcfreak30]

Hello, I thought I would drop in as well and say hi! I am interested in this overall as I am collaborating with @redsolver between his S5 project, and my project Lume Web (Git) which is still in heavy BUIDL mode.

Background

To put it simply, Lume aims to be the web3 hub for the community, to decentralize browsing the new web we are building. It is creating 2 backend infrastructure components. a P2P network that can bridge any service (think socks proxy with DHT topics) and provide an RPC service. This is powered by hypercore.

The second is a data storage portal for the Sia network. The project is grant funded by the Sia Foundation.

Why this matters/why im here in this discussion regarding S5?

Lume will be using the concepts, formats, and data structures from S5 as we are in the same ecosystem/tribe. Simply put, anything S5 does, Lume will aim to support within reason. Put another way, both S5 and Lume are two different networks born out of the Sia ecosystem (and all the failures we have hit already collectively) which aim to interop for the end user. We are both working on different "problem spaces" as well, but have similar interests and goals. You can also view S5 and Lume as Geth vs Nethermind from the ETH ecosystem. Different clients/nodes trying to do similar purposes.

I am also actively researching how to trustlessly download/support the current IPFS in browser since I have a network that can proxy over WS raw data.

As such I have an interest in following along and being involved in any work around this for my projects interest.

I hope I did not ramble too much.

Kudos!