Define ping/pong payload extensions (#348)

* Define ping/pong payload extensions

* fix: resolve PR concerns

* fix: update links

* fix: add capabilities message and outline how to upgrade a standard message

* Update ping-custom-payload-extensions.md

* fix: remove  networks that don't exist yet

* fix: some wording in capabilities payload

* fix: remove word version

* fix: push changes before going to bed

* fix: spelling issues

* fix: resolve PR concerns

* fix: update PR to resolve some of Kim's concerns

* fix: try to address some of deme's feedback

* fix: spelling error

beacon-chain/beacon-network.md

@@ -76,12 +76,10 @@ The beacon chain network supports the following protocol messages:
 
 #### `Ping.custom_data` & `Pong.custom_data`
 
-In the beacon chain network the `custom_payload` field of the `Ping` and `Pong` messages is the serialization of an SSZ Container specified as `custom_data`:
+In the beacon chain network the `custom_payload` field of the `Ping` and `Pong` messages. The first packet between another client MUST be [Type 0: Client Info, Radius, and Capabilities Payload](../ping-payload-extensions/extensions/type-0.md). Then upgraded to the latest payload supported by both of the clients.
 
-```python
-custom_data = Container(data_radius: uint256)
-custom_payload = serialize(custom_data)
-```
+List of currently supported payloads, by latest to oldest.
+-  [Type 1 Basic Radius Payload](../ping-payload-extensions/extensions/type-1.md)
 
 ### Routing Table
 

canonical-transaction-index/canonical-transaction-index-network.md

@@ -56,12 +56,10 @@ The canonical transaction index network supports the following protocol messages
 
 #### `Ping.custom_data` & `Pong.custom_data`
 
-In the canonical transaction index network the `custom_payload` field of the `Ping` and `Pong` messages is the serialization of an SSZ Container specified as `custom_data`:
+In the canonical transaction index network the `custom_payload` field of the `Ping` and `Pong` messages. The first packet between another client MUST be [Type 0: Client Info, Radius, and Capabilities Payload](../ping-payload-extensions/extensions/type-0.md). Then upgraded to the latest payload supported by both of the clients.
 
-```
-custom_data = Container(data_radius: uint256)
-custom_payload = SSZ.serialize(custom_data)
-```
+List of currently supported payloads, by latest to oldest.
+-  [Type 1 Basic Radius Payload](../ping-payload-extensions/extensions/type-1.md)
 
 
 ### Routing Table 

history/history-network.md

@@ -61,12 +61,15 @@ The history network supports the following protocol messages:
 
 #### `Ping.custom_data` & `Pong.custom_data`
 
-In the history network the `custom_payload` field of the `Ping` and `Pong` messages is the serialization of an SSZ Container specified as `custom_data`:
+In the history network the `custom_payload` field of the `Ping` and `Pong` messages. The first packet between another client MUST be [Type 0: Client Info, Radius, and Capabilities Payload](../ping-payload-extensions/extensions/type-0.md). Then upgraded to the latest payload supported by both of the clients.
+
+List of currently supported payloads, by latest to oldest.
+-  [Type 2 History Radius Payload](../ping-payload-extensions/extensions/type-2.md)
+
+
+
+
 
-```python
-custom_data = Container(data_radius: uint256)
-custom_payload = SSZ.serialize(custom_data)
-```
 
 ### Routing Table
 

ping-payload-extensions/extensions.md

@@ -0,0 +1,12 @@
+# This document lists defined extensions
+This is a list and short description of all the extensions
+
+- Extensions can be supported by all sub-networks or a subsection
+
+
+| Type number  |  Name | Supported sub-networks  | Short Description  | Is standard extension  |
+|---|---|---|---|---|
+| [0](extensions/type-0.md)  | Client Info, Radius, and Capabilities | All  | Returns client info e.x. `trin/0.1.1-2b00d730/linux-x86_64/rustc1.81.0`, the nodes radius and a list of enabled extensions  |  Yes  |
+| [1](extensions/type-1.md)  |  Basic Radius Payload | State, Beacon | Provides the nodes Radius  |  No  |
+| [2](extensions/type-2.md)  | History Radius Payload  |  History | Provides the nodes radius and ephemeral header count  |  No  |
+| [65535](extensions/type-65535.md)  | Error Response  |  All | Returns an error for respective ping message  |  Yes  |

ping-payload-extensions/extensions/template.md

@@ -0,0 +1,26 @@
+# [Title]
+
+[Description]
+
+Ping payload
+```python
+
+[Payload] = SSZ.serialize(Container([Key Value Pairs]))
+
+
+[Container Name] = Container(
+  type: [Type Number],
+  payload: [Payload]
+)
+```
+
+Pong payload
+```python
+
+[Payload] = SSZ.serialize(Container([Key Value Pairs]))
+
+[Container Name] = Container(
+  type: [Type Number],
+  payload: [Payload]
+)
+```

ping-payload-extensions/extensions/type-0.md

@@ -0,0 +1,43 @@
+# Client Info and Capabilities Payload
+
+## Client Info 
+Client info are ASCII hex encoded strings.
+
+Client info strings consist of 4 parts
+- client name (e.x. `trin`,`fluffy`)
+- client version + short commit (e.x. `0.1.1-2b00d730`)
+- operating system + cpu archtecture (e.x. `linux-x86_64`)
+- programming language + language version (e.x. `rustc1.81.0`)
+
+Example 
+- String: `trin/0.1.1-2b00d730/linux-x86_64/rustc1.81.0`
+- Hex encoding: `0x7472696E2F302E312E312D32623030643733302F6C696E75782D7838365F36342F7275737463312E38312E30`
+
+#### Privacy Concerns
+Clients can optionally return an empty string for privacy reasons, this is not recommended as client info helps researchers understand the network.
+
+## Capabilities
+Portal clients can only have max 400 extensions enabled per sub-network.
+
+This payload provides a list of u16's each u16 provide in the list corresponds to an enabled extension type.
+
+## Payload Outline
+
+Ping and Pong Payload
+```python
+
+MAX_CLIENT_INFO_BYTE_LENGTH = 200
+MAX_CAPABILITIES_LENGTH = 400
+
+client_info_and_capabilities = SSZ.serialize(Container(
+    client_info: ByteList[MAX_CLIENT_INFO_BYTE_LENGTH]
+    data_radius: U256
+    capabilities: List[u16, MAX_CAPABILITIES_LENGTH]
+))
+
+CapabilitiesPayload = Container(
+  type: 0,
+  payload: client_info_and_capabilities
+)
+```
+

ping-payload-extensions/extensions/type-1.md

@@ -0,0 +1,15 @@
+# Basic Radius Payload
+
+A basic Ping/Pong payload which only contains the nodes radius
+
+Ping and Pong Payload
+```python
+
+basic_radius = SSZ.serialize(Container(data_radius: U256))
+
+BasicRadiusPayload = Container(
+  type: 1,
+  payload: basic_radius
+)
+```
+

ping-payload-extensions/extensions/type-2.md

@@ -0,0 +1,14 @@
+# History Radius Payload
+
+A specialized radius payload for the history network which contains field for how many ephemeral headers the node holds.
+
+Ping and Pong Payload
+```python
+
+history_radius = SSZ.serialize(Container(data_radius: U256, ephemeral_header_count=U16))
+
+HistoryRadiusPayload = Container(
+  type: 2,
+  payload: history_radius
+)
+```

ping-payload-extensions/extensions/type-65535.md

@@ -0,0 +1,33 @@
+# Error Payload
+
+If the ping receiver can't handle the ping for any reason the pong should return an error payload
+
+Pong payload
+```python
+
+# Max ASCII hex encoded strings length
+MAX_ERROR_BYTE_LENGTH = 300
+
+error_payload = SSZ.serialize(Container(error_code: u16, message: ByteList[MAX_ERROR_BYTE_LENGTH]))
+
+ErrorPayload = Container(
+  type: 65535,
+  payload: error_payload
+)
+```
+
+### Error Code's
+
+#### 0: Extension not supported
+This code should be returned if the extension isn't supported. This error should only be returned if 
+- The extension isn't supported
+- The extension isn't a required extension for specified Portal Network.
+
+#### 1: Requested data not found
+This error code should be used when a client is unable to provide the necessary data for the response.
+
+#### 2: Failed to decode payload
+Wasn't able to decode the payload
+
+#### 3: System error
+A critical error happened and the ping can't be processed

ping-payload-extensions/ping-custom-payload-extensions.md

@@ -0,0 +1,63 @@
+# Ping Custom Payload Extensions
+
+## Motivation
+
+Ping Payload Extensions. Messages on Portal are primarily made of ping/pong responses. This framework allows Portal clients to implement `non standard extensions` which don't require a breaking change to deploy to the network. A more flexible way to extend the Protocol without bloating the core specification [Portal-Wire-Protocol](../portal-wire-protocol.md) or requiring every client to agree to add a feature a subset of clients want.
+
+# Type's
+
+There are 65536 unique type ids.
+
+Types 0-10_000 and 65436-65535 are reserved for for future upgrades.
+
+The rest are first come first serve, but they should still be defined in this repo to avoid overlaps.
+
+
+## Requirements
+
+All payloads used in the Ping `custom_payload` MUST follow the `Ping Custom Payload Extensions` format.
+
+## Custom Payload Extensions Format
+
+- **type**: numeric identifier which tells clients how the `payload` field should be decoded.
+- **payload**: the SSZ encoded extension payload
+
+
+```python
+CustomPayloadExtensionsFormat = Container(
+  type: u16,
+  payload: ByteList[max_length=1100]
+)
+```
+
+## Ping vs Pong
+The relationship between Ping and Pong message will be determined by the requirements of the type.
+
+Currently type 0, 1, and 2 are symmetrical, having the same payload for both request and response. This symmetry is not required. Extensions may define separate payloads for Ping and Pong within the same type.
+
+
+### Error responses
+If the ping receiver can't handle the ping for any reason the pong should return an error payload
+
+[Type 65535: The definition of error responses can be found here](extensions/type-65535.md)
+
+## Standard extensions
+
+A standard extension is an extension which all nodes on the network MUST support. Nodes can send these without requiring a `Type 0: Client Info, Radius, and Capabilities Payload` request to discover what extensions the client supports. 
+
+Changing standard extensions is considered a breaking change.
+
+List of some standard extensions
+- [Type 0: Client Info, Radius, and Capabilities Payload](extensions/type-0.md): useful for finding Client Info, Radius, and ping extensions a client supports
+- [Type 65535: Error Payload](extensions/type-65535.md): this payload can only be used as a response to a ping
+
+# Non standard extensions
+Non standard extensions are extensions in which you can't assume all other clients support, to use a non standard extension it is required that Portal clients first send a [Type 0: Client Info, Radius, and Capabilities Payload](extensions/type-0.md) packet, then upgrade to use their desired non standard extensions.
+
+## What is the [Type 0: Client Info, Radius, and Capabilities Payload](extensions/type-0.md) for
+It is for Portal implementations which want to see what extensions a peer supports. Not all extensions need to be implemented by all parties. So in order for a peer to find if an extension is implemented a [Type 0: Client Info, Radius, and Capabilities Payload](extensions/type-0.md) should be exchanged.
+
+Non-required extension's offer a way for Portal implementations to offer extended functionality to its users without requiring every Portal implementing party to agree to a new feature. This allows for a diverse set of use cases to be fulfilled without requiring every implementer implement every extension, or requiring the need to bloat the minimal [Portal-Wire-Protocol](../portal-wire-protocol.md) with new `Message Types`.
+
+## How do sub-network standard extension's work
+sub-network standard extension's are fundamental extensions that are required for a Portal sub-network to function. They must be supported by the sub-networks implementations. Changing a standard extension requires a breaking change. 

state/state-network.md

@@ -56,12 +56,10 @@ The execution state network supports the following protocol messages:
 
 #### `Ping.custom_data` & `Pong.custom_data`
 
-In the execution state network the `custom_payload` field of the `Ping` and `Pong` messages is the serialization of an SSZ Container specified as `custom_data`:
+In the execution state network network the `custom_payload` field of the `Ping` and `Pong` messages. The first packet between another client MUST be [Type 0: Client Info, Radius, and Capabilities Payload](../ping-payload-extensions/extensions/type-0.md). Then upgraded to the latest payload supported by both of the clients.
 
-```
-custom_data = Container(data_radius: uint256)
-custom_payload = SSZ.serialize(custom_data)
-```
+List of currently supported payloads, by latest to oldest.
+-  [Type 1 Basic Radius Payload](../ping-payload-extensions/extensions/type-1.md)
 
 #### POKE Mechanism
 

transaction-gossip/transaction-gossip.md

@@ -26,19 +26,7 @@ The network uses the PING/PONG/FINDNODES/FOUNDNODES/OFFER/ACCEPT messages from t
 
 The network uses the standard XOR distance function.
 
-### PING payload
-
-```
-Ping.custom_payload := ssz_serialize(custom_data)
-custom_data         := Container(transaction_radius: uint256)
-```
-
-### PONG payload
-
-```
-Pong.custom_payload := ssz_serialize(custom_data)
-custom_data         := Container(transaction_radius: uint256)
-```
+In the execution state network the `custom_payload` field of the `Ping` and `Pong` messages is the serialization of an SSZ Container specified as [Type 1 Basic Radius Payload](../ping-payload-extensions/extensions/type-1.md)
 
 ## Content Keys