Marshal to multiple buffers

[howardjohn]

In my use case, I am sending messages that have many repeated fields (tens or hundreds of thousands). This is the proto if it is relevant.

Ideally we would split this up at the application level, but for various reasons we can't do this.

Instead, it would be nice if we could marshal to multiple slices, rather than one giant one, using mem.BufferSlice.

This could look something like this:

mem.BufferSlice{
  everythingSerializedBesidesResources,
  resource1,
  resource2,
  resource3,
   ...
}

A specialization of this is that in our particular use case, the repeated field is an anypb.Any -- this is preserialized into []byte. We are already caching this serialization, so it would be ideal to not copy that at all and instead just reference the slice in the mem.BufferSlice. That would then look like...

mem.BufferSlice{
  everythingSerializedBesidesResources,
  resource1BesidesBytesValue, resource1BytesValue,
  resource2BesidesBytesValue, resource2BytesValue,
   ...
}

I put together a POC doing this and saw pretty promising results:

benchmark            ns/op 
golang-protobuf      11116     
vtprotobuf           2858      
hand-optimized       125       

(code is here, its not correct but should be correct-enough to get a broad idea of the performance. It could be faster, simpler, and more correct).

TBH I mostly think this is likely out of scope for this library and pretty usage-specific, but wanted to raise it in case others were interested. I am not really sure how this would even be exposed in the API. At minimum we would need a new marshal interface that returns mem.BufferSlice and some way to indicate when/how to split messages.