Consider changing serialization of optional payload

[PhilippGackstatter]

We could consider improving our serialization of optional types in serix, specifically of the optional payload in a Transaction Essence.

The Tagged Data Payload is optional, and the way we implement this optionality is by having a Payload Length prefix of type uint32 that, if 0, indicates that the type is absent. If it's there, the payload length holds the entire length of the optional payload, which is unnecessary. Additionally, we handle this payload specially in serix, which also seems unnecessary, and if removed, would reduce the serializer code size and its complexity.

It would suffice to have a byte indicating whether the type is absent (e.g. 0) or present (e.g. 1), and then continue reading the schema of whatever type is specified since the next byte would be that type's prefix byte. This way, we wouldn't need special code for optional payloads, but would even enable optional types more generically.
See also https://github.com/iotaledger/tips/blob/main/tips/TIP-0020/tip-0020.md#serialized-layout for the layout of the current Transaction and how the optional payload is serialized, which still matches what we have in code today.

[alexsporn]

You still need a Payload Length in the Transaction, so that you know how many bytes should be read. For an optional field with a fixed size this extra byte you mentioned works, but not for variable size fields, for those we always require the length first.

[PhilippGackstatter]

Why would it not be possible to read the bytes on-demand from the stream? You know the layout of a Tagged Data Payload, so if there was an indication that it's present, you can try to

• read the type byte, and if it's tagged data
• read the uint8 tag length, then the tag
• read the uint32 data length, then the data.

I haven't looked into how serix does this exactly, but something similar must be done when reading a Metadata Feature. You only know the Features Count, and based on that you'll try to read each feature. You don't have a "Features Length" or something like that that tells you how long the entire features part is, or the individual one. There is no upfront info on how long the metadata is.

So, without having dove deeper into serix, I don't get why we need the Payload Length specifiying the entire length of the following payload when the tagged data itself has a well-defined layout, just like a Metadata Feature.

[alexsporn]

That comparison does not work. Because of the Features Count you explicitely know that you will have a certain amount of typed features (all having a type of a specified size (uint8 or uint32)) and after you read all of them you are done reading the whole Features field.
For optional fields how can you know if the byte you are reading is from the object or from the object after that one? Relying on a specific type byte being present is hacky and is not guaranteed to be always safe. Because what happens if the field after the optional one also has a type that can have the same value?

[thibault-martinez]

I am also in favour of removing the Payload Length. Payload is a type like any other and prefixing the length is not necessary. In the SDK we parse it because the protocol wants it but we don't use it. Payload is self-contained and can be read on demand.

[muXxer]

Maybe come to a final conclusion when we do #653 .
The good thing about having a specific Payload Length is, that you can skip the whole payload in the deserialization if you don't want to check the payload itself.

[muXxer]

We decided to keep the payload length, so we can skip parsing the payload if not needed.