neuIP-5: Subset schema & identification (name tbc) discussion

[djfnd]

Starting a fresh thread for discussion regarding the currently-in-development proposal for neuIP-5 led by @neatonk.

Current PRs:

• neuIP-5: Subset Schema Definition and Identification using JSON Schema and JCS neuIPs#15

Pre-discussion can be found here

---

Abstract

This neuIP proposes a standard for defining and identifying subset schemas in Neume, utilizing JSON Schema and the JSON Canonicalization Scheme (JCS). This proposal introduces schemas as defining sets—each with a unique deterministic identifier—wherein set membership is determined by schema validation and subsets may be explicitly partitioned using derived subset schemas. This proposed standard enhances data partitioning, subset reconciliation, and execution orchestration, in line with the vision of Neume as a credibly neutral network of decentralized nodes collaborating to produce a shared dataset.

Motivation

The motivation for this proposal is multifaceted, with the primary objective of catalyizing Neume's evolution into a network capable of serving a variety of applications and handling the scale of the data required. To realize this vision, we need a robust mechanism to accurately identify and validate data subsets in a decentralized environment.

Currently, Neume lacks a deterministic method for identifying subsets and their schemas, which complicates subset partitioning and reconciliation. Additionally, the project lacks an automatic method for optimizing and scheduling the execution of multiple strategies based on their respective input and output datasets. This proposal provides a promising path to address these challenges by introducing deterministic schema Identifiers, helping to manage the escalating scale of data consumed by Neume.

This neuIP confronts these limitations by proposing JSON Schema-based definitions, validations, and derivations, paired with deterministic identifiers using JCS. By leveraging these identifiers, Neume nodes will be able to coordinate set reconciliation, validate candidate data against their schemas, orchestrate execution graphs based on their input and output datasets, and assemble ad-hoc strategy combinations for producing required output sets. Essentially, deterministic schema Identifiers are a crucial facilitator for Neume to scale efficiently, supporting flexible dataset partitioning and segmentation of the Neume schema into numerous, potentially overlapping, sub-schemas.

The benefits of this proposal align with the principles of social scalability and decentralization, fundamental to the Neume project. By introducing JSON Schema-based definitions, validations, derivations, and deterministic identifiers using JCS, we establish a foundation for a network of independent nodes each operating on subsets of interest, promoting Neume’s growth as a network. As such, this neuIP marks a significant step towards realizing our vision for Neume.