Parallelism across multiple nodes over a slow link?

[remi6397]

So I was wondering, would it be possible to have the model split across multiple nodes communicating over a slow link?

My use case would be to split the model into <4 GB chunks ([32-bit] WebAssembly memory limit) distributed across peers connected via WebRTC.

BLOOM already caters to a use case similar to mine (i.e. Petals), except for that its requirements exceed the resources available in this case…

[remi6397]

I just thought that maybe I could split the [7B] model file in half, where the first half would contain layers 0-15 and the second half would contain layers 16-31. I could then distribute those partial models across the nodes and forward the contents of the inpL tensor on each slice boundary. Or am I missing something?