Merge pull request #28 from oasisprotocol/kostko/off-chain-rt-logic

ADR 24: Runtime Off-chain Logic (ROFL)

0024-off-chain-runtime-logic.md

@@ -0,0 +1,348 @@
+# ADR 0024: Runtime Off-chain Logic (ROFL)
+
+## Component
+
+Oasis Core, Oasis SDK
+
+## Changelog
+
+- 2024-02-26: Notifications
+- 2023-11-27: Initial draft
+
+## Status
+
+Proposed
+
+## Context
+
+Sometimes we may want the runtime compute nodes to run additional off-chain
+logic that communicates with the on-chain state securely (e.g. ensuring that the
+off-chain logic is being run by the same node operator, is properly attested
+when running in a TEE, etc.).
+
+The off-chain logic may then perform non-deterministic and potentially expensive
+things (like remote HTTPS requests or complex local computation) and securely
+interact with the on-chain logic via transactions.
+
+The main use case driving this proposal is support for running attested light
+client committees that read and verify information from other chains, then make
+this information available to Oasis runtimes with no additional trust
+assumptions.
+
+## Decision
+
+While similar functionality can be implemented entirely independently on the
+application layer (and such solutions already exist), this proposal attempts to
+reuse the same security and attestation infrastructure that is already available
+for on-chain parts of the runtimes, specifically:
+
+- Compute nodes and runtime binary distribution and execution can stay the same
+  as it has been for existing node operators that run the runtimes. Handling of
+  the off-chain logic part should be done transparently if the runtime provides
+  it.
+
+- Existing attestation, consensus and freshness proof flows can be leveraged for
+  ensuring that the off-chain logic is running in a secure environment.
+
+One important consideration is also whether to have the off-chain logic part of
+the same runtime binary or have it as a completely separate binary running in
+its own process. This proposal decides on the latter to ensure that the
+off-chain TCB is completely separate from the on-chain TCB. Given that the logic
+running off-chain can be much more complex and can interact with untrusted
+external services, ensuring this separation is important as a defense-in-depth
+measure.
+
+The proposed architecture extends the composition of the runtime so that it now
+contains the following components:
+
+- **Runtime On-chain Logic (RONL)** is what has existed as the sole runtime
+  component before this proposal. It contains the logic (and TCB) that is
+  responsible for executing the deterministic on-chain logic of the runtime.
+
+- **Runtime Off-chain Logic (ROFL)** is an optional runtime component that may
+  run in parallel with RONL and is part of its own TCB. It also uses the same
+  general runtime framework and RHP, but instead of implementing the on-chain
+  batch scheduling, execution and query methods, it only implements specific
+  notification hooks that can trigger arbitrary actions.
+
+Both RONL and ROFL are managed as independent runtimes by the Oasis Node as
+host, using the existing runtime host architecture. Failure of ROFL does not
+affect RONL which can proceed to run as usual.
+
+### Attestation
+
+An assumption made in this proposal is that both RONL and ROFL components are
+developed and built together, by the same entity, and are part of the same
+release. This means that we can simplify attestation by making RONL being able
+to attest ROFL by being aware of its exact identity.
+
+The idea is that during the release build process, ROFL is built first, its
+signer-independent identity (e.g. MRENCLAVE) is measured and included during
+compilation of RONL. The signer-dependent part of identity (e.g. MRSIGNER) is
+assumed to be the same for both and can be read from trusted CPU state (since it
+may not be available during the build process due to offline signing).
+
+Alternatively, one can imagine a proposal where the ROFL identity is backed by
+some sort of on-chain governance process defined in the RONL component. Defining
+such a mechanism is outside the scope of this proposal.
+
+The process for ROFL attestation proceeds as follows:
+
+1. **Remote Attestation.** The normal runtime attestation flow is initiated by
+   the host. As a result of this flow, the `node.CapabilityTEE` structure is
+   generated which includes the remote attestation quote and additional data.
+
+2. **Node Endorsement.** The host verifies the `node.CapabilityTEE` structure
+   and if deemed correct, it signs it using the node's identity key and the
+   following domain separation context:
+
+   ```
+   oasis-core/node: endorse TEE capability
+   ```
+
+   The signature is stored in a new structure `EndorsedCapabilityTEE` which is
+   defined as follows:
+
+   ```go
+   type EndorsedCapabilityTEE struct {
+      // CapabilityTEE is the TEE capability structure to be endorsed.
+      CapabilityTEE CapabilityTEE `json:"capability_tee"`
+
+      // NodeEndorsement is the node endorsement signature.
+      NodeEndorsement signature.Signature `json:"node_endorsement"`
+   }
+   ```
+
+3. **Updating Node-Endorsed CapabilityTEE in ROFL.** The `EndorsedCapabilityTEE`
+   is sent to ROFL to be stored and available for establishing secure EnclaveRPC
+   sessions.
+
+4. **RONL Verification.** When establishing a new session with RONL, the
+   endorsed TEE capability is presented during session establishment. RONL
+   verifies the quote, ensures the enclave identity is one of the known
+   identities set at compile-time and verifies the node endorsement against the
+   locally known node identity (both RONL and ROFL must be from the same node).
+
+   If all the checks pass, a secure EnclaveRPC session is established.
+
+This flow needs to be repeated whenever RAK changes for any reason and also
+periodically to ensure freshness (consistent with the quote policy configured
+for the runtime in the consensus layer).
+
+### Updates to the ORC Manifest
+
+The ORC manifest is extended with a field that can specify extra components
+which currently include ROFL binaries in a similar way as we already support
+regular runtime binaries (e.g. specifying the executable and SGX metadata).
+
+The manifest is updated as follows:
+
+```go
+// Manifest is a deserialized runtime bundle manifest.
+type Manifest struct {
+    // ... existing fields omitted ...
+
+    // Components are the additional runtime components.
+    Components []*Component `json:"components,omitempty"`
+}
+
+// ComponentKind is the kind of a component.
+type ComponentKind string
+
+const (
+  // ComponentInvalid is an invalid component.
+  ComponentInvalid ComponentKind = ""
+  // ComponentRONL is the on-chain logic component.
+  ComponentRONL ComponentKind = "ronl"
+  // ComponentROFL is the off-chain logic component.
+  ComponentROFL ComponentKind = "rofl"
+)
+
+// Component is a runtime component.
+type Component struct {
+  // Kind is the component kind.
+  Kind ComponentKind `json:"kind"`
+
+  // Name is the name of the component that can be used to filter components
+  // when multiple are provided by a runtime.
+  Name string `json:"name,omitempty"`
+
+  // Executable is the name of the runtime ELF executable file.
+  Executable string `json:"executable"`
+
+  // SGX is the SGX specific manifest metadata if any.
+  SGX *SGXMetadata `json:"sgx,omitempty"`
+}
+```
+
+The top-level `executable` and `sgx` fields are supported for backwards
+compatibility and implicitly define a new `Component` of kind `ComponentRONL`.
+
+### Updates to the Runtime Host Protocol
+
+This proposal includes some non-breaking updates to the Runtime Host Protocol in
+order to support the ROFL component, as follows:
+
+- **Consensus Block Notification.** No updates are required to facilitate
+  notifications about consensus layer blocks as this is already handled as part
+  of the existing RHP flow. The only change is that for ROFL, these
+  notifications invoke a hook that can be implemented by the runtime.
+
+- **Runtime Transaction Submission.** A new method `HostSubmitTx` is introduced
+  which allows ROFL to submit transactions to the runtime. It works by queueing
+  the transaction in the transaction pool (local queue) for later scheduling.
+
+  ```go
+  type HostSubmitTxRequest struct {
+      // RuntimeID is the identifier of the target runtime.
+      RuntimeID common.Namespace `json:"runtime_id"`
+      // Data is the raw transaction data.
+      Data []byte `json:"data"`
+      // Wait specifies whether the call should wait until the transaction is
+      // included in a block.
+      Wait bool `json:"wait,omitempty"`
+      // Prove specifies whether the response should include a proof of
+      // transaction being included in a block.
+      Prove bool `json:"prove,omitempty"`
+  }
+  ```
+
+- **Notify Registration.** A new method `HostRegisterNotify` is introduced which
+  allows ROFL to register to be notified by the host when specific events occur.
+  Note that delivery of these notifications is best effort as a dishonest host
+  may withold notification delivery or generate spurious notifications.
+
+  Registering for notifications overwrites any previous configuration.
+
+  ```go
+  type HostRegisterNotifyRequest struct {
+      // RuntimeBlock subscribes to runtime block notifications.
+      RuntimeBlock bool `json:"runtime_block,omitempty"`
+      // RuntimeEvent subscribes to runtime event emission notifications.
+      RuntimeEvent *struct {
+          // Tags specifies which event tags to subscribe to.
+          Tags [][]byte `json:"tags,omitempty"`
+      } `json:"runtime_event,omitempty"`
+  }
+  ```
+
+- **Notification Delivery.** A new method `RuntimeNotify` is introduced which
+  allows the host to deliver event notifications based on previously registered
+  notifiers.
+
+  ```go
+  type RuntimeNotifyRequest struct {
+      // RuntimeBlock notifies about a new runtime block.
+      RuntimeBlock *roothash.AnnotatedBlock `json:"runtime_block,omitempty"`
+      // RuntimeEvent notifies about a specific runtime event being emitted.
+      RuntimeEvent *struct {
+          // Block is the block header of the block that emitted the event.
+          Block *roothash.AnnotatedBlock `json:"block"`
+          // Tags are the matching tags that were emitted.
+          Tags [][]byte `json:"tags"`
+      } `json:"runtime_event,omitempty"`
+  }
+  ```
+
+- **RONL-ROFL Communication.** The existing EnclaveRPC is reused to facilitate
+  the communication between the two components if/when needed. For this purpose
+  the endpoint identifier `ronl` is made available in the ROFL host method
+  handler to address the RONL component.
+
+- **Updating Node-Endorsed CapabilityTEE in ROFL.** A new method
+  `RuntimeCapabilityTEEUpdateEndorsementRequest` is introduced which allows the
+  node to refresh the `EndorsedCapabilityTEE` for ROFL.
+
+  ```go
+  type RuntimeCapabilityTEEUpdateEndorsementRequest struct {
+      // EndorsedCapabilityTEE is an endorsed TEE capability.
+      EndorsedCapabilityTEE node.EndorsedCapabilityTEE `json:"ect"`
+  }
+  ```
+
+### Updates to EnclaveRPC RAK Binding
+
+Version 2 of the `RAKBinding` structure is introduced for establishment of
+EnclaveRPC sessions, as follows:
+
+```rust
+pub enum RAKBinding {
+    // ... previous versions omitted ...
+
+    /// V2 format which supports endorsed CapabilityTEE structures.
+    #[cbor(rename = 2)]
+    V2 {
+        ect: EndorsedCapabilityTEE,
+        binding: Signature,
+    },
+}
+```
+
+Additionally, the relevant EnclaveRPC session implementation is updated to
+facilitate thew new authentication mechanism via endorsed TEE capabilities and
+the session demultiplexer is updated to support authentication policies on
+incoming connections.
+
+### Updates to the Runtime Host Sandbox
+
+This proposal updates the runtime host sandbox to support optionally allowing
+external network requests. These are then allowed only for the ROFL component
+(if any is available for a runtime).
+
+The following modifications are required:
+
+- When setting up the Bubblewrap sandbox, `--share-net` is passed to share the
+  network namespace with the sandboxed runtime. All other namespaces are still
+  unshared.
+
+- The runtime loader is modified to accept an additional argument
+  `--allow-network` which then changes the usercall extension to pass through
+  any address passed in the `connect_stream` handler.
+
+### Configuration
+
+ROFL may require additional configuration which it may do through one of several
+ways:
+
+- **On-chain Configuration.** Configuration for the ROFL component may be stored
+  in on-chain state. ROFL would then query the current configuration and apply
+  it locally.
+
+- **Local Per-Node Configuration.** In case some per-node configuration is
+  required (e.g. to allow the node operator to override a default), the existing
+  runtime local configuration mechanism can be used where configuration is
+  provided as part of the RHP handshake. All configuration for ROFL should be
+  contained under the `rofl` configuration key.
+
+### Untrusted Local Storage
+
+ROFL may utilize the existing untrusted node-local storage to store things like
+sealed data local to the node. This store is shared between RONL and ROFL, but
+all ROFL keys are transparently prefixed by `rofl.` on the host such that only
+RONL can see (but not necessarily read) ROFL's keys but not vice versa.
+
+### Updates to the Oasis SDK
+
+A convenient way to develop ROFL modules alongside the on-chain support
+functionality should be implemented in the Oasis SDK, including a convenient way
+for ROFL to submit runtime transactions in a way that can be verified on-chain
+as coming from a specific node/runtime instance.
+
+## Consequences
+
+### Positive
+
+- Oasis runtimes can easily be extended with arbitrary off-chain logic that can
+  securely interact with on-chain functionality.
+
+- Node operators do not need to perform much additional configuration in order
+  to support the new off-chain logic.
+
+### Negative
+
+- Additional complexity is introduced to the Runtime Host Protocol and to the
+  node binary.
+
+### Neutral
+
+## References

README.md

@@ -115,4 +115,5 @@ The following records currently exist:
 * [ADR 0020](0020-governance-delegator-votes.md) - Governance Support for Delegator Votes
 * [ADR 0021](0021-keymanager-ephemeral-secrets.md) - Forward-Secret Ephemeral Secrets
 * [ADR 0022](0022-keymanager-master-secrets.md) - Forward-Secret Master Secrets
+* [ADR 0024](0024-off-chain-runtime-logic.md) - Runtime Off-chain Logic (ROFL)
 <!-- markdownlint-enable line-length -->