Version Management & Schema Validation

[hackerwins]

Description:

1. Overview

Yorkie is a document store for real-time collaboration, allowing users to express models in a JSON-like format. This issue defines a schema validation and version management system for consistent document management.

2. Necessity

• Consistently manage JSON-like model representations with high flexibility
• Restrict document editing within specific schema boundaries
• Maintain stable document structures through schema version management

3. Schema Definition and Management

3.1 Schema Creation

• Create schemas in the Dashboard
• Use TypeScript's Type Alias syntax
• Support Yorkie built-in types (yorkie.Tree, yorkie.Text, etc.)

3.2 Version Management

• Manage schemas immutably
• Store updates as new versions (v1, v2, v3, ...)

3.3 Schema Usage

• Ability to specify a particular schema when creating documents in the SDK

const doc = new yorkie.Document('doc1');
await client.attach(doc, {
  schema: `doc-schema:v1` // optional
});

4. Implementation Devices

4.1 Schema Validation Device (A)

• Location: Dashboard or server
• Function: Validate user-defined schemas
• Process: Tokenization → Syntax Analysis → Semantic Analysis
• Current Status: Syntax analysis implemented (yorkie-schema repository)

4.2 Document Change Validation Device (B)

• Location: Hybrid approach (SDK + Server)
• Function: Validate Document changes
• Implementation: Use Ruleset based on AST generated by Device A
• Current Status: We need to implement a prototype

5. System Architecture

Overall Structure

graph TB
    User[User] --> Dashboard
    Dashboard --> |Schema Definition| SchemaValidationA[Device A. Schema Validator]
    SchemaValidationA --> |Validated Schema| Server
    Server --> |Validated Schema| SDK
    User --> |Document Editing| SDK
    SDK --> |Basic Validation| ChangeValidationB[Device B. Change Validator]
    Server --> |Store Schema/Document| Database[(Database)]

Loading

Device B. Change Validator

sequenceDiagram
    participant Editor
    participant SDK
    participant ChangeValidator    
    participant Server

    Editor->>SDK: Document.Attach()
    SDK->>Server: Request AttachDocument
    Server-->>SDK: Return Document with Schema

    SDK->>ChangeValidator: Build Ruleset
    ChangeValidator-->>SDK: Ruleset Ready
    SDK-->>Editor: Attach Successful

    Editor->>SDK: Document.Update()
    SDK->>ChangeValidator: Generate Change
    ChangeValidator->>ChangeValidator: Validate Change Against Rules
    
    alt Validation Successful
        ChangeValidator-->>SDK: Validation Passed
        SDK-->>Editor: Update Successful
    else Validation Failed
        ChangeValidator-->>SDK: Validation Failed
        SDK-->>Editor: Validation Failed Error
    end

Loading

6. Key Considerations and Improvements

6.1 Performance Optimization

• Performance testing for large documents and complex schemas
• Optimize validation speed in real-time collaboration environments

6.2 Offline Support

• Include basic validation logic in the SDK
• Develop temporary validation and synchronization strategies for offline states

6.3 Migration Strategy

• Define migration process for existing documents when schema versions change
• Establish backward compatibility maintenance plans

6.4 Custom Conflict Resolution

• Implement handling logic for validation failures in the SDK
• Develop conflict resolution strategies for synchronization processes

Why:

This issue aims to enhance Yorkie's document management capabilities and provide a more stable real-time collaboration environment. Continuous improvement will be necessary during the implementation process, considering performance, user experience, and the characteristics of real-time collaboration.

[blurfx]

I'm interested in this topic, can I get involved?