Server-side JSON-Patch OT

[tomalec]

Starcounter version: 2.4.999.191

Issue type

• Bug
• Feature request
• Suggestion
• Question
• Cannot reproduce
• Urgent

Issue description

During investigation of HeadsOne issue (fast clicking on create button) https://github.com/Starcounter/HeadsOmni/issues/665#issuecomment-440784899
from reading the Starcounter.XSON code I figured out (I'm not a pro in C#, so please, tell me that I'm wrong) that we do not transform JSON Patches against JSON Patches on server-side - meaning we do not have JSON Patch Operational Transformation (OT).

• we do some kind of transformations on Handle - when we apply patch to the JSON tree,
• but we do not transform JSON Patches against JSON Patches.

OT - Operational Transformation - is about transforming operations not the object. That's what makes it fast and simple.

Reproduction code snippet

Send two, separate but consecutive Web Socket messages from client(browser), that changes a property in JSON view-model.

[{"op":"replace","path":"/_ver#c$","value":1},{"op":"test","path":"/_ver#s","value":0},{"op":"replace","path":"/BlendingProvider_0/Content/Products_0/CurrentPage/Products_0/CreateProductTrigger$","value":"1"}] //C1
[{"op":"replace","path":"/_ver#c$","value":2},{"op":"test","path":"/_ver#s","value":0},{"op":"replace","path":"/BlendingProvider_0/Content/Products_0/CurrentPage/Products_0/CreateProductTrigger$","value":"2"}] //C2

Server-side of an app should remove the modified object in response to the first message:

[{"op":"replace","path":"/_ver#s","value":1},{"op":"test","path":"/_ver#c$","value":1},{"op":"remove","path":"/BlendingProvider_0/Content/Products_0/CurrentPage"}] //S1

Expected behaviour

Server-side JSON Patch OT, should transform
C2' = C2 ∘ S1 regardless of JSON tree.

[{"op":"replace","path":"/_ver#c$","value":2},{"op":"test","path":"/_ver#s","value":1}] //C2'

Actual behaviour - Exception stack trace

Unknown property in path. Patch: '{"op":"replace","path":"/BlendingProvider_0/Content/Products_0/CurrentPage/Products_0/CreateProductTrigger$","value":"2"}'. Property: 'CreateProductTrigger$'. Viewmodel versions: {client (_ver#c$): 3, clients serverversion: 2, server (_ver#s): 3}.

Related issues:

• Implement server side Palindrom to connect to a Starcounter server from another server #341
• https://github.com/Starcounter/AdminTrack/issues/459
• https://github.com/Starcounter/Palindrom.Net/issues/8
• https://github.com/Starcounter/TechComm/issues/360
• Make reconnecting/resetting more symmetric to use Palindrom.js as server in Node Palindrom/Palindrom#91
• Define the protocol properly Palindrom/Palindrom#61
• ... I bet there are more issues about making Palindrom.Net, NodeJS Palindrom, Writing a white paper on JSON Patch OT and so on.

//cc @warpech @eriohl @erikvk @alshakero

[eriohl]

Unfortunately only Christian is/was fully familiar with how this stuff works and is supposed to work. So it is not entirely trivial answering questions about it. Not sure if Joachim was involved in this as well but if so it was a while ago.

He also started up a rewrite of it is but he didn't get very far. (if you lack access to it is a little disabled code riddled with TODOs, so it is probably not all that useful).

So it is probably quite a large project cleaning this up. And for me it is not entirely clear on how to approach this. Up to @erikvk though. Let me know if and how I can help.

[erikvk]

cc @dan31 @eriohl @tomalec

I should let you know that I'm currently busy with the high-prio migrator tool for the root data model (AKA Doctor M) used in mapping, and will be unable to look into this before I'm done with that, i.e. sometime mid-December.

[miyconst]

As far as I remember from discussions with Christian, this feature was never implemented, and it's lack was hitting us every now and then.

[miyconst]

Christian also told me that there is a way to setup server side JSON Patch applicator to ignore patches to non-existing objects instead of crashing, but now I was not able to find how to use that feature, maybe it's only available internally in Level1/XSON.

[warpech]

The roadmap https://github.com/Starcounter/CompanyTrack/blob/master/Roadmap/Roadmap.md considers long and short-term solution to the lack of this feature. @dan31, @eriohl do you remember what is the short-term solution that is meant there?

I think the only way to fix this is to implement the actual transformations. It should not be super hard, given that we aleady have a reference implementation in JS: https://github.com/Palindrom/JSON-Patch-OT-agent

[eriohl]

I don't remember discussing that part nor do I understand how to short term fix it given that there isn't any OT there to fix. But maybe it is possible just to use the existing implementation given that is is possible to embed Node.js in .NET? It requires no input other the patches right?

[tomalec]

It requires no input other the patches right?

Just patches (with/and version numbers), JSON tree is not needed.

[tomalec]

On Thursday 11th April @eriohl @erikvk @dan31 and me had a meeting to gather common understanding of current implementations and plan the architecture and the way to go with the new server-side implementations.

I'd like to write down some highlights from the meeting.

• I explained the relations between our client-side packages that covers entire JSON Patch flow:

  • https://github.com/Palindrom/Palindrom
  • https://github.com/Palindrom/JSON-Patch-Queue
  • https://github.com/Palindrom/JSON-Patch-OT-agent
  • https://github.com/Palindrom/JSON-Patch-OT
  • https://github.com/Palindrom/JSONPatcherProxy
  • https://github.com/Starcounter-Jack/JSON-Patch
  • visualization of OT at https://github.com/tomalec/PuppetJs-operational-transformation
  • docs at
    • https://palindrom.github.io/docs/,
    • https://github.com/Palindrom/Palindrom/wiki/Server-communication,
    • https://docs.starcounter.io/topic-guides/blendable-web-apps/palindrom,
    • https://docs.starcounter.io/topic-guides/blendable-web-apps/sessions#client-server-synchronization-with-json-patch

• We agreed that it makes sense to introduce similar architecture on server-side as it gives maintainable and comprehensible modules and allows to keep OT processing in separate process. It would also make it easier in future to publish Palindrom server/Starcounter-lite to be engagement step for the community.

• We also agreed to start with implementing serve-side OT first as this is a missing feature, that may lead to data inconsistency for clients running on production. Then we could focus on improving dirty checking and code refactoring.

• Server-side implementation could be started as translation from JS to C# of our current modules, as they are stable and abstract enough.

• Server-side implementation could make use of https://github.com/json-patch/json-patch-tests and tests written for https://github.com/Starcounter-Jack/JSON-Patch and Palindrom packages

Please correct me, if I wrote something wrong or missed something.

[erikvk]

@tomalec Thanks for the writeup!

[miyconst]

Archiving.