Apply Command

[ankitrgadiya]

Context

The CLI supports creating all the basic objects except Deployments currently. Deployment also happens to be the most complicated object with a lot of runtime GUIDs as part of it. Initializing it through the CLI flags is not possible. Instead, the idea is to take the kubectl approach and implement a generic apply command that accepts YAML/JSON based manifests.

The user experience will be to write the Deployments (and other objects) in a declarative YAML manifest (similar to Kubernetes manifests) and then call the apply command.

rio apply -f resources.yaml

Define Manifests

First part of the problem is, the core resources are currently not defined as Kubernetes-style manifests. We want to solve this problem at the CLI first. That means, in the beginning only CLI will understand the manifests and it will be able to translate it into the Objects available in SDK that API Servers understand.

Rapyuta.io CRD Document is a work-in-progress document that defines core resources in Kubernetes-style Manifests.

JSON Schema Diagram [Open with Diagrams.net]

Resource	Sample Manifest	JSON Schema
Project	Project Manifest	Project Schema
Secret	Secret Manifest	Secret Schema
Build	Build Manifest	Build Schema
Network	Network Manifest	Network Schema
Device	Device Manifest	Device Schema
Deployment	WIP	WIP

Parse and Validate

Instead of writing hand-written Parsers and Validators for every resource, we are taking the approach of using JSON Schema. JSON Schema is an evolving specification that lets you define resources with validations in a JSON format. There are libraries built for common languages that can parse the Schema and make them available as langauge native objects.

Note: For compatibility purposes, we have decided to use the Draft 07. We've so far found it to be available in most popular libraries out there.

The parser will use the Munch library to convert the YAML/JSON-based dictionary into an attribute-access style Object. For validation of the objects, fastjsonschema's generated validation code will be used.

The following Abstract Class is defined in the riocli.model module that every object must implement. This gives a standard way to process the Objects in the Apply command.

class Model(ABC, Munch):

    def apply(self, client: Client) -> typing.Any:
        """
        Implementation of the Apply using the hooks defined below.
        """
        obj = self.find_object(client)
        if not obj:
            return self.create_object(client)

        return self.update_object(client, obj)

    @abstractmethod
    def find_object(self, client: Client) -> typing.Any:
        """
        Hook for finding the existing resource for the Object.
        If falsy-value is returned, Model will conclude that the Object is 
        not present and will move on to the create hook.

        If truthy-value is returned, Model will pass the value in the 
        update hook, if defined.
        """
        pass

    @abstractmethod
    def create_object(self, client: Client) -> typing.Any:
        """
        Hook for creating the resource. The returned value will be returned 
        from `apply` function.
        """
        pass

    @abstractmethod
    def update_object(self, client: Client, obj: typing.Any) -> typing.Any:
        """
        Hook for updating the resource. This will be called if the Object is found.
        """
        pass

    @classmethod
    @abstractmethod
    def pre_process(cls, client: Client, d: typing.Dict) -> None:
        """
        Hook for pre_processing the dictionary before it gets validated or 
        parsed into the Object. For e.g.: Subsitute GUIDs instead of names.
        """
        pass

    @staticmethod
    @abstractmethod
    def validate(d):
        """
        Hook for validating the dictionary. Use the generated function 
        from the `fastjsonschema` library.
        """
        pass

    @classmethod
    def from_dict(cls, client: Client, d: typing.Dict):
        """
        Validate the data using the `validate` hook and then initialize the Object 
        using Munch.
        """
        cls.pre_process(client, d)
        cls.validate(d)
        return cls(munchify(d))

Other Libraries for JSONSchema:

• Python
  • warlock
  • fastjsonschema
  • jsonschema
  • json-schema-to-class
  • python-jsonschema-objects
• Go
  • gojsonschema
  • jsonschema

Translate and Serialize

Since the API does not yet understand the new Manifest format, CLI needs to translate the manifests into v1 API Objects. In the first iteration, the translation is hand-implemented using Python code. SDK already has v1 API Objects defined and implements methods to operate on them. SDK also takes care of the serialization.

The Model implementation for each object will also implement the to_v1 method generating the SDK v1 Object.

class Project(Model)

    ...
    
    def to_v1(self) -> v1Project:
        return v1Project(self.metadata.name)

[Experimental] Declarative JSON Translation

We can use a JSON Translation language to define the translations instead of hand-coding it. One such library is JSONata. But the Python binding is not installable in its current form and it does not have Windows support (jsonata-js/jsonata#415).

Additionally, even after converting to the desired JSON, we cannot use the v1 Objects. We can directly send the payload to the APIs but then we'll have to either use internal functions from SDK or implement it again here.

[ankitrgadiya]

Warlock

Repo: https://github.com/bcwaldon/warlock/

Warlock has a single-method API to generate Classes from the JSON Schema. The class validates and generates Objects using kwargs. Warlock is built on top of the popular jsonschema library and the code footprint is very small.

Warlock looks like an abondonded project in its current form. Last commit is from Feb, 2020 and last release from May, 2019. The current version supports jsonschema ~=3.x.x that gives us JSON Schema Draft 07 support. Because of the small code footprint, it is possible to fork and update it easily.

Sample Code:

from requests import get
from warlock import model_factory
from yaml import safe_load

GIST_SCHEMA_URL = "https://gist.githubusercontent.com/ankitrgadiya/29b7b4be28a26b8071437f2b1f36ce3e/raw/project.json"
GIST_MANIFEST_URL = "https://gist.githubusercontent.com/ankitrgadiya/29b7b4be28a26b8071437f2b1f36ce3e/raw/manifest.yaml"

schema = get(GIST_SCHEMA_URL).json()
manifest = safe_load(get(GIST_MANIFEST_URL).text)

Project = model_factory(schema, name="project")
p = Project(**manifest)
print(p)

[ankitrgadiya]

fastjsonschema

Website: https://horejsek.github.io/python-fastjsonschema/

The fastjsonschema library implements validations for JSON Schema that claims to be the fastest library for the purpose. The library provides three modes:

• Validate by directly passing in the Schema and Data (Compiles on call and validates).
• Compile the Schema and use it in the life-cycle of the Application (Compiles and caches).
• Compile and dump in Python File (Compiles and generates the code that can be used in multiple Application runs).

It looks like an active project with last commit on March, 2022 and last release in January, 2022. The current version implements Draft 04, 06 and 07.

Sample Code:

from requests import get
from yaml import safe_load
from fastjsonschema import validate, compile, compile_to_code

GIST_SCHEMA_URL = "https://gist.githubusercontent.com/ankitrgadiya/29b7b4be28a26b8071437f2b1f36ce3e/raw/project.json"
GIST_MANIFEST_URL = "https://gist.githubusercontent.com/ankitrgadiya/29b7b4be28a26b8071437f2b1f36ce3e/raw/manifest.yaml"

schema = get(GIST_SCHEMA_URL).json()
manifest = safe_load(get(GIST_MANIFEST_URL).text)

validate(schema, manifest)

validation = compile(schema)
validation(manifest)

code = compile_to_code(schema)
print(code)

Generated Code