golemfactory · prekucki · Sep 12, 2024 · Feb 7, 2024 · Feb 7, 2024 · Feb 8, 2024
diff --git a/README.md b/README.md
@@ -1,3 +1,34 @@
-This repository contains documentation and artifacts defining the architecture for the Golem Project.
+# Golem Architecture
 
+This repository contains documentation and artifacts defining the protocol and architecture for the Golem Project.
+
+
+This repository contains specification on protocol level and should be baseline for
+future implementation. That means that implementation details of [yagna](https://github.com/golemfactory/yagna)
+or SDKs should be documented in corresponding repositories, not here.
+
+## Content
+
+### [Daemon REST API](https://golemfactory.github.io/ya-client/)
+
+REST API specifications (OpenAPI, yaml) are maintained in [repository](https://github.com/golemfactory/ya-client/tree/master/specs).
+Documentation is generated [here](https://golemfactory.github.io/ya-client/).
+Read this documentation if you need to interact with daemon directly without using SDKs. 
+
+### [Golem market protocol properties standards](./standards/README.md)
+
+Describes market protocol properties hierarchy and protocol design conventions.
+Read if you are looking for meaning of specific properties.
+
+### [Features specification](./specs/README.md)
+
+Specification of Golem features. This directory contains unified description of protocol.
+Read here if you want to know how certain features work, or you need to implement interactions
+between Nodes on SDK on daemon level.
+
+### [Golem Amendment Proposals](./gaps/Readme.md) (GAPs)
+
+Process of proposing and discussing changes to Golem protocol.
+Read this chapter if you want to observe progressive development of the protocol and motivations
+behind design decisions.
 
diff --git a/gaps/Readme.md b/gaps/Readme.md
@@ -0,0 +1,19 @@
+# Golem Amendment Proposals (GAPs)
+
+## List of approved GAPs
+
+
+| Link                                                                         | Description             | Category          |
+|------------------------------------------------------------------------------|-------------------------|-------------------|
+| [GAP-1](./gap-1_gap_process/gap-1_gap_process.md)                            | GAP process             | Internal process  |
+| [GAP-3](./gap-3_mid_agreement_payments/gap-3_mid_agreement_payments.md)      | Mid-Agreement payments  | Payments          |
+| [GAP-4](./gap-4_comp_manifest/gap-4_comp_manifest.md)                        | Computation Manifest    | Provider security |
+| [GAP-5](./gap-5_payload_manifest/gap-5_payload_manifest.md)                  | Payload Manifest        | Provider security |
+| [GAP-6](./gap-6_tech_council_process/gap-6_tech_council_process.md)          | Tech Council process    | Internal process  |
+| [GAP-16](./gap-16_golem_deploy/gap-16_golem_deploy.md)                       | Golem deploy            |                   |
+| [GAP-17](./gap-17_offline_requestor_model/gap-17_offline_requestor_model.md) | Offline Requestor model |                   |
+| [GAP-23](./gap-23_inbound_network_proxy/gap-23_inbound_network_proxy.md)     | Inbound network proxy   |                   |
+| [GAP-25](./gap-25_golem_certificates/gap-25_golem_certificates.md)           | Golem certificates      | Provider security |
+| [GAP-31](./gap-31_node_descriptor/gap-31_node_descriptor.md)                 | Node descriptor         | Provider security |
+| [GAP-32](./gap-32_experimental_features/gap-32_experimental_features.md)     | Experimental features   | Internal process  |
+| [GAP-35](./gap-35_gpu_pci_capability/gap-35_gpu_pci_capability.md)           | GPU pci capability      |                   |
diff --git a/specs/README.md b/specs/README.md
@@ -1 +1,15 @@
-REST API specifications (OpenAPI, yaml) are maintained in https://github.com/golemfactory/ya-client/tree/master/specs .
+# Specifications
+
+## [Capabilities](./capabilities.md)
+
+List of Golem capabilities.
+
+The Golem specification outlines a range of features that Golem Nodes have the option to implement. However, each implementation
+retains the autonomy to select which features to incorporate. Capabilities provide the most detailed level at which decisions
+can be made regarding which functionalities are included.
+
+## Features
+
+### Execution
+
+- [ExeUnit command progress reporting](./command-progress.md)
diff --git a/specs/capabilities.md b/specs/capabilities.md
@@ -1,7 +1,9 @@
 # Golem Capability List
 
-| Capability                    | Yagna package version | Backwards-compatible? | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
-|-------------------------------|-----------------------|-----------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| 
-| Multi-Activity Agreement      | 0.5.0                 | Yes                   | Negotiate ability to create multiple activities under single Agreement. Use (which?) property in Demand/Offer to indicate node's support for Multi-Activity. If counterparty does not support Multi-Activity, the node falls back to single Activity per Agreement behaviour.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
-| Restart Proposal Negotiations | 0.7.0                 | Yes                   | Agent is allowed to restart negotiations, by sending `Counter Proposal`, after he rejected Proposal at some point. Counter-party will receive regular `ProposalEvent` in this case. Only Agent rejecting Proposal has initiative in restarting negotiations, rejected Agent can only wait for this to happen. To indicate, that Proposal rejection isn't final and negotiations can be restarted later, Agent can set `golem.proposal.rejection.is-final` (bool) field in `Reason` structure. If this value is set to false, Agent can free any state related to this negotiation. The same field can be set in `Reason` sent in `Reject Agreement` operation. Requestor can send new counter Proposal after some period of time or propose the same Agreement for the second time. (No change to specification) |
-| Payment version protocol      | 0.15.0                | No                    | [Payment protocol version](./payment_version.md) (`golem.com.payment.protocol.version`) shows what payment capabilities each node implements. New payment features require higher version.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
+
+| Capability                    | Backwards-compatible? | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | Property                                                                                    |
+|-------------------------------|-----------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------|
+| Multi-Activity Agreement      | Yes                   | Negotiate ability to create multiple activities under single Agreement. Use `golem.srv.caps.multi-activity` property in Demand/Offer to indicate node's support for Multi-Activity. If counterparty does not support Multi-Activity, the node falls back to single Activity per Agreement behaviour.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | `golem.srv.caps.multi-activity`                                                             |
+| Restart Proposal Negotiations | Yes                   | Agent is allowed to restart negotiations, by sending `Counter Proposal`, after he rejected Proposal at some point. Counter-party will receive regular `ProposalEvent` in this case. Only Agent rejecting Proposal has initiative in restarting negotiations, rejected Agent can only wait for this to happen. To indicate, that Proposal rejection isn't final and negotiations can be restarted later, Agent can set `golem.proposal.rejection.is-final` (bool) field in `Reason` structure. If this value is set to false, Agent can free any state related to this negotiation. The same field can be set in `Reason` sent in `Reject Agreement` operation. Requestor can send new counter Proposal after some period of time or propose the same Agreement for the second time. (No change to specification) |                                                                                             |
+| Command progress reporting    | Yes                   | ExeUnit can report progress of executed commands. [Specification](./command-progress.md)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | `golem.activity.caps.transfer.report-progress` `golem.activity.caps.deploy.report-progress` |
+| Payment version protocol      | No                    | [Payment protocol version](./payment_version.md) (`golem.com.payment.protocol.version`) shows what payment capabilities each node implements. New payment features require higher version.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | `golem.com.payment.protocol.version`                                                        |
diff --git a/specs/command-progress.md b/specs/command-progress.md
@@ -0,0 +1,82 @@
+# Command progress reporting
+
+Downloading large images and files poses problem for Requestor controlling ExeUnit.
+Since he doesn't have any information about Provider's internet bandwidth, there is no
+way to estimate download time. Slow internet connection is indistinguishable from Provider
+that just doesn't work and is unable to finish task. This results in wasted resources and
+non-optimal Requestor behavior.
+
+
+This specification describes generic mechanisms allowing to report progress of different
+commands from ExeUnit to Reqeustors. Moreover, it describes current implementation of ExeUnit.
+
+## Specification
+
+### Capabilities
+
+ExeUnit supporting progress reporting should put following properties in the Offer:
+- `golem.activity.caps.transfer.report-progress` - if he can report `transfer` command progress
+- `golem.activity.caps.deploy.report-progress` - if he can report `deploy` command progress
+
+To support progress for any other command, specification should be extended with new properties.
+Current ExeUnit implementation supports only these 2 commands.
+
+### Progress parameters
+
+To enable progress events Requestor agent has to attach progress parameters to [ExeScriptCommand](https://golemfactory.github.io/ya-client/?urls.primaryName=Activity%20API#model-RunCommand).
+Example `deploy` and `transfer` command:
+```json
+[
+  {
+    "deploy": {
+      "progress" : { "updateInterval" : "300ms", "updateStep" : null }
+    }
+  },
+  {
+    "transfer": {
+      "from": "http://34.244.4.185:8000/LICENSE",
+      "to": "container:/input/file_in",
+      "progress" : { "updateInterval" : null, "updateStep" : 1048576 }
+    }
+  }
+]
+```
+
+`ProgressArgs` structure is described [here](https://golemfactory.github.io/ya-client/index.html?urls.primaryName=Activity%20API#model-ProgressArgs).
+
+
+### Listening to events 
+
+Events will be available on [getStreamingBatchResults](https://golemfactory.github.io/ya-client/index.html?urls.primaryName=Activity%20API#/requestor_control/getExecBatchResults) endpoint (with `text/event-stream` accept header).
+Event structure is defined [here](https://golemfactory.github.io/ya-client/index.html?urls.primaryName=Activity%20API#model-RuntimeEventKindProgress).
+
+Example:
+```json
+{
+  "batch_id": "5c9a8f0e13dd49edb7fa570a10b1b14b",
+  "index": 0,
+  "timestamp": "2024-02-09T15:12:07.540318580",
+  "kind": {
+    "progress": {
+      "step": [
+        0,
+        1
+      ],
+      "message": "Transfer retried",
+      "progress": [
+        60,
+        1024
+      ],
+      "unit": "Bytes"
+    }
+  }
+}
+```
+
+You must check specific ExeUnit documentation to know implementation details not covered
+by this specification.
+
+## List of supporting ExeUnits
+
+- [ya-exe-unit](https://github.com/golemfactory/yagna/blob/master/docs/provider/exe-unit/command-progress.md) (`deploy` and `transfer` commands)
+- [ya-runtime-ai](https://github.com/golemfactory/ya-runtime-ai) (only `deploy` command)
diff --git a/standards/0-commons/golem/activity/caps.md b/standards/0-commons/golem/activity/caps.md
@@ -26,3 +26,17 @@ Indicates the data transmission protocols available for TRANSFER operation on th
 ### **Examples**
 * `golem.activity.caps.transfer.protocol:List=["https"]` - Declares availability of HTTPS protocol for data transfer
 * `golem.activity.caps.transfer.protocol=["http","https","gftp"]` - Declares availability of HTTP, HTTPS and "GFTP" protocols for file transfer
+
+## `golem.activity.caps.transfer.report-progress : bool [Fact]`
+
+### Describes: Offer
+
+Indicates that ExeUnit is capable of sending `transfer` command progress events according to protocol
+described [here](https://github.com/golemfactory/yagna/blob/master/docs/provider/exe-unit/command-progress.md).
+
+## `golem.activity.caps.deploy.report-progress : bool [Fact]`
+
+### Describes: Offer
+
+Indicates that ExeUnit is capable of sending `deploy` command progress events according to protocol
+described [here](https://github.com/golemfactory/yagna/blob/master/docs/provider/exe-unit/command-progress.md).
diff --git a/standards/1-node/node/geo.md b/standards/1-node/node/geo.md
@@ -1,4 +1,8 @@
 # Golem Identity Properties - Geography
+
+> [!NOTE]
+> Unused
+
 Namespace defining location/geography aspects of a Golem node.
 
 ## Common Properties

diff --git a/standards/2-service/srv/comp.md b/standards/2-service/srv/comp.md
@@ -1,6 +1,9 @@
 # Computation Platform properties 
 Generic properties describing the Computation Platform aspects.
 
+> [!NOTE]
+> Unused
+
 ## Common Properties
 
 (Not applicable)

diff --git a/standards/README.md b/standards/README.md
@@ -29,6 +29,11 @@ The Golem Standards repository consists of following sections/categories:
 * [2-service](2-service) - A collection of namespaces containing properties which describe a Golem service or resource. A hierarchy of different **categories of services/resources** is defined.
 * [3-commercial](3-commercial) - A collection of namespaces containing properties which describe commercial aspects of Golem ecosystem. All properties referring to **pricing**, **payments** or **licenses** are defined here.
 
+### Standard properties - Cheat sheet
+
+List fo Golem Factory supported properties:
+[Cheat sheet](cheat_sheet.md)
+
 ## Generic conventions
 
 ### ["Fact" vs "Negotiable" properties](#negotiable-properties)
@@ -59,5 +64,3 @@ There are a couple of reasons for the `Negotiable` property convention:
 
 Properties (and usage conters) marked as `Deprecated` are still supported in the current version of the Standard, but will be removed in one of subsequent versions of the Standard. The deprecation rules as indicated in the [Golem Compatibility Policy](https://handbook.golem.network/see-also/compatibility-policy#deprecation-policy) shall be followed.
 
-## Standard properties - Cheat sheet
-[Cheat sheet](cheat_sheet.md)