fix: Provide smallcaps encoding in addition to capData

[erights]

Based on
https://github.com/endojs/endo/blob/38078aec70975d87a0a76918d0ba788e87420581/packages/marshal/src/marshal.js

This key thing about this PR is that the unserializer understands both new smallcaps and old capData protocols, switching between them based on a leading #. The serialized defaults to the old protocol, based on the useSmallcaps option to makeMarshal. With this defaulting to off, everything is trivially green, but that puts us in a position to switch different subsystems on separately as we put in the work to do so. In this PR itself, only test-marshal-smallcaps.js does so.

See #1283 for an experiment of changing the default useSmallcaps to true, to see what breaks. However, be aware that some tests and even some modules in this PR explicitly set useSmallcaps to false, evading this experiment, because they're known to not yet work with smallcaps.

SECURITY CONCERN

Even though our serializer, as of this PR, does not send smallcaps by default, others can, including malformed smallcaps. For capData, we've reviewed the code for robustness in the face of malformed capData. REVIEWERS, please review the unserializer for robustness against malformed smallcaps. If we cannot be confident that we're at least as robust against malformed smallcaps as we were against malformed capData, that would be a reason to delay this PR until we are.

[erights]

Ready for review!

@arirubinstein for some reason, I cannot add you as an official reviewer. Please consider yourself a reviewer anyway. Thanks.

[mhofman]

I only did a quick pass to understand the smallcaps encoding. If I understand it correctly (it would be nice to have a description to the design and/or comparisons between capdata and smallcaps), the main difference with capdata is the way BigInt, remotables/promises and certain specific values are encoded as strings instead of being encoded as qclass "objects".

While I see how this may result in less verbose (and potentially more readable) output, I'm somewhat surprised it'd lead to such an improvement in performance.

[erights]

I'm somewhat surprised it'd lead to such an improvement in performance.

We have not measured this, and so do not know. I can explain why I expect a big reduction in allocations, but my pre-measurement intuitions are often wrong. @dtribble , we need to measure this!

The measured increase in performance was Agoric/agoric-sdk#6174 + #1280 . @dtribble measured this, but only for pattern matching. I suspect that at least the endo-level changes generalize, but again we should be skeptical until we measure. We need to more realistically measure this!

[erights]

If I understand it correctly (it would be nice to have a description to the design and/or comparisons between capdata and smallcaps),

Yes. (FWIW, I did not expect us to decide today to put this into review immediately, or I would have been more ready. But yes, this needs to be written.)

the main difference with capdata is the way BigInt, remotables/promises and certain specific values are encoded as strings instead of being encoded as qclass "objects".

Yes. This means that strings themselves do not always encode as the same string, which also requires that we hilbert the string dimension.

[mhofman]

Copy over from #1284 (review)

[erights]

All done. PTAL

[gibson042]

Looks good except for verification that each decoded object property name is a string.

We should also create a ticket to update agoric-sdk get-flattened-publication.sh for decoding smallcaps after this lands.

[erights]

Looks good except for verification that each decoded object property name is a string.

Note that passStyleOf(x) === 'tagged' and makeTagged(tag, payload) both validate that the tag is a string, so marshal doesn't need to.

[erights]

Great suggestions! Done. PTAL.

[gibson042]

A few final nits, but I trust your judgment on all of them. Looks like this is it!