From de9bfd4d4f7e939bf62d74dd6f7735cb108d70a2 Mon Sep 17 00:00:00 2001 From: mocax <35949852+mocax@users.noreply.github.com> Date: Mon, 29 Jan 2018 22:52:35 -0800 Subject: [PATCH 1/6] Create PROTOCOL.md Draft the wire protocol spec for Twirp based on latest discussions. It reflects the existing Twirp wire spec, plus best practice from proto3, gRPC, HTTP, and JSON. The spec should be usable for developers to build clients and servers using Twirp wire protocol without much dependency. --- PROTOCOL.md | 193 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 193 insertions(+) create mode 100644 PROTOCOL.md diff --git a/PROTOCOL.md b/PROTOCOL.md new file mode 100644 index 00000000..0ab12501 --- /dev/null +++ b/PROTOCOL.md @@ -0,0 +1,193 @@ +# Twirp Wire Protocol + +This document defines the Twirp wire protocol over HTTP. + +## Conventions + +The requirement level keywords "MUST", "MUST NOT", "REQUIRED", +"SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", +and "OPTIONAL" used in this document are to be interpreted as +described in [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt). + +The grammar rules used in this document are using [ABNF +syntax](https://tools.ietf.org/html/rfc5234). + +## Overview + +The Twirp wire protocol is a simple RPC protocol based on HTTP and +Protocol Buffers (proto). The protocol uses HTTP URLs to specify the +RPC endpoints, and sends/receives proto messages as HTTP +request/response bodies. + +To use Twirp, developers first define their APIs using proto files, +then use Twirp tools to generate the client and the server libraries. +The generated libraries implement the Twirp wire protocol, using the +standard HTTP library provided by the programming language runtime or +the operating system. Once the client and the server are implemented, +the client can communicate with the server by making RPC calls. + +The Twirp wire protocol supports both binary and JSON encodings of +proto messages, and works with any HTTP client and any HTTP version. +However, certain capabilities may be limited by the actual HTTP +library being used. + +### URLs + +**URL ::= Base-URL "/" [ Package "." ] Interface "/" Method** + +The Twirp wire protocol uses HTTP URLs to directly specify the RPC +endpoints on the server for sending the requests. such direct mapping +makes the request routing simple and efficient. The Twirp URLs have +the following components. + +* **Base-URL** is the virtual location of a Twirp API server, which is + typically published via API documentation or service discovery. For + example, "https://example.com/apis". + +* **Package** is the proto `package` name for an API, which is often + considered as an API version. For example, + `example.calendar.v1`. This component is omitted if the API + definition doesn't have a package name. + +* **Interface** is the proto `service` name for an API. For example, + `CalendarService`. + +* **Method** is the proto `rpc` name for an API method. For example, + `CreateEvent`. + +### Requests + +**Request ::= Request-Headers Request-Body** + +Twirp always uses HTTP POST method to send requests, because it +closely matches the semantics of RPC methods. + +The **Request-Headers** are normal HTTP headers. The Twirp wire +protocol uses the following headers. + +* **Authorization** header is often used to pass user credentials + from the client to the server, such as OAuth access token or + JWT token. + +* **Content-Type** header indicates the proto message encoding, which + should be one of "application/x-protobuf", "application/json". The + server uses this value to decide how to parse the request body, + and encode the response body. + +* **User-Agent** header indicates the client application and its + runtime environment. While the server should not use this + information for request processing, this header is heavily used + for analytics and troubleshooting purposes. + +* **RPC-Timeout** header indicates the client-specified request + timeout in seconds, such as "10". If **RPC-Timeout** is omitted, the + server should use a pre-configured timeout value, by default it + should be 10 seconds. + +The **Request-Body** is the encoded request message, contained in the +HTTP request body. The encoding is specified by the `Content-Type` +header. + +### Responses + +**Response ::= Response-Headers Response-Body** + +The **Response-Headers** are just normal HTTP response headers. The +Twirp wire protocol uses the following headers. + +* **Content-Type** The value should be either "application/x-protobuf" + or "application/json" to indicate the encoding of the response + message. It must match the "Content-Type" header in the request. + +The **Request-Body** is the encoded response message contained in the +HTTP response body. The encoding is specified by the `Content-Type` +header. + +## Example + +The following example shows a simple Echo API definition and its +corresponding wire payloads. + +The example assumes the server base URL is "https://example.com". + +```proto +package twirp; + +service Echo { + rpc Hello(HelloRequest) returns (HelloResponse); +} + +message HelloRequest { + string message; +} + +message HelloResponse { + string message; +} +``` + +**Proto Request** + +``` +POST /twirp/Echo/Hello HTTP/1.1 +Host: example.com +Content-Type: application/x-protobuf +Content-Length: 15 + + +``` + +**JSON Request** + +``` +POST /twirp/Echo/Hello HTTP/1.1 +Host: example.com +Content-Type: application/json +Content-Length: 27 + +{"message":"Hello, World!"} +``` + +**Proto Response** + +``` +HTTP/1.1 200 OK +Content-Type: application/x-protobuf +Content-Length: 15 + + +``` + +**JSON Response** + +``` +HTTP/1.1 200 OK +Content-Type: application/json +Content-Length: 27 + +{"message":"Hello, World!"} +``` + +## Errors + +If an error occurs when the server processes a request, the server +must return an error payload as the response message, and correctly +set the HTTP status code. Please see +[`google.rpc.Code`](https://github.com/googleapis/googleapis/blob/master/google/rpc/code.proto) +on how to map typical server errors to HTTP status codes. + +### Timeout errors + +For a single request, there is a client-specified timeout and a +server-configured timeout. If a request misses the server-configured +timeout, the server must return a `503` error. If a request misses +the client-specified timeout that it is shorter then the server- +configured timeout, the server must return a `504` error. +This allows more accurate measurement of the server availability. + +### Network errors + +If a client fails to reach the server due to network errors, the +client library must report HTTP status code `502` to the client +application. This helps users distinguishing network errors from +server errors. From 77e721bdb36551fa277cbe7ba43bfc438a6133f3 Mon Sep 17 00:00:00 2001 From: mocax <35949852+mocax@users.noreply.github.com> Date: Mon, 29 Jan 2018 23:38:33 -0800 Subject: [PATCH 2/6] Use . instead of / as separator --- PROTOCOL.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/PROTOCOL.md b/PROTOCOL.md index 0ab12501..8b872ceb 100644 --- a/PROTOCOL.md +++ b/PROTOCOL.md @@ -129,7 +129,7 @@ message HelloResponse { **Proto Request** ``` -POST /twirp/Echo/Hello HTTP/1.1 +POST /twirp.Echo/Hello HTTP/1.1 Host: example.com Content-Type: application/x-protobuf Content-Length: 15 @@ -140,7 +140,7 @@ Content-Length: 15 **JSON Request** ``` -POST /twirp/Echo/Hello HTTP/1.1 +POST /twirp.Echo/Hello HTTP/1.1 Host: example.com Content-Type: application/json Content-Length: 27 From c9c3d98621556ee402eecb6263b48aebbb9828a0 Mon Sep 17 00:00:00 2001 From: mocax <35949852+mocax@users.noreply.github.com> Date: Tue, 30 Jan 2018 13:41:14 -0800 Subject: [PATCH 3/6] Update PROTOCOL.md Addressing review comments. --- PROTOCOL.md | 95 +++++++++++++++++++++-------------------------------- 1 file changed, 38 insertions(+), 57 deletions(-) diff --git a/PROTOCOL.md b/PROTOCOL.md index 8b872ceb..d0c1b490 100644 --- a/PROTOCOL.md +++ b/PROTOCOL.md @@ -1,16 +1,7 @@ # Twirp Wire Protocol -This document defines the Twirp wire protocol over HTTP. - -## Conventions - -The requirement level keywords "MUST", "MUST NOT", "REQUIRED", -"SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", -and "OPTIONAL" used in this document are to be interpreted as -described in [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt). - -The grammar rules used in this document are using [ABNF -syntax](https://tools.ietf.org/html/rfc5234). +This document defines the Twirp wire protocol over HTTP. The +current protocol version is v5. ## Overview @@ -28,28 +19,30 @@ the client can communicate with the server by making RPC calls. The Twirp wire protocol supports both binary and JSON encodings of proto messages, and works with any HTTP client and any HTTP version. -However, certain capabilities may be limited by the actual HTTP -library being used. ### URLs -**URL ::= Base-URL "/" [ Package "." ] Interface "/" Method** +In [ABNF syntax](https://tools.ietf.org/html/rfc5234), Twirp's URLs +have the following format: + +**URL ::= Base-URL "/twirp/" [ Package "." ] Service "/" Method** -The Twirp wire protocol uses HTTP URLs to directly specify the RPC -endpoints on the server for sending the requests. such direct mapping +The Twirp wire protocol uses HTTP URLs to specify the RPC +endpoints on the server for sending the requests. Such direct mapping makes the request routing simple and efficient. The Twirp URLs have the following components. * **Base-URL** is the virtual location of a Twirp API server, which is - typically published via API documentation or service discovery. For - example, "https://example.com/apis". + typically published via API documentation or service discovery. + Currently, it should only contain URL `scheme` and `authority`. For + example, "https://example.com". * **Package** is the proto `package` name for an API, which is often considered as an API version. For example, `example.calendar.v1`. This component is omitted if the API definition doesn't have a package name. -* **Interface** is the proto `service` name for an API. For example, +* **Service** is the proto `service` name for an API. For example, `CalendarService`. * **Method** is the proto `rpc` name for an API method. For example, @@ -57,41 +50,23 @@ the following components. ### Requests -**Request ::= Request-Headers Request-Body** - Twirp always uses HTTP POST method to send requests, because it closely matches the semantics of RPC methods. The **Request-Headers** are normal HTTP headers. The Twirp wire protocol uses the following headers. -* **Authorization** header is often used to pass user credentials - from the client to the server, such as OAuth access token or - JWT token. - * **Content-Type** header indicates the proto message encoding, which - should be one of "application/x-protobuf", "application/json". The + should be one of "application/protobuf", "application/json". The server uses this value to decide how to parse the request body, and encode the response body. -* **User-Agent** header indicates the client application and its - runtime environment. While the server should not use this - information for request processing, this header is heavily used - for analytics and troubleshooting purposes. - -* **RPC-Timeout** header indicates the client-specified request - timeout in seconds, such as "10". If **RPC-Timeout** is omitted, the - server should use a pre-configured timeout value, by default it - should be 10 seconds. - The **Request-Body** is the encoded request message, contained in the HTTP request body. The encoding is specified by the `Content-Type` header. ### Responses -**Response ::= Response-Headers Response-Body** - The **Response-Headers** are just normal HTTP response headers. The Twirp wire protocol uses the following headers. @@ -111,6 +86,8 @@ corresponding wire payloads. The example assumes the server base URL is "https://example.com". ```proto +syntax = "proto3"; + package twirp; service Echo { @@ -131,7 +108,7 @@ message HelloResponse { ``` POST /twirp.Echo/Hello HTTP/1.1 Host: example.com -Content-Type: application/x-protobuf +Content-Type: application/protobuf Content-Length: 15 @@ -152,7 +129,7 @@ Content-Length: 27 ``` HTTP/1.1 200 OK -Content-Type: application/x-protobuf +Content-Type: application/protobuf Content-Length: 15 @@ -170,24 +147,28 @@ Content-Length: 27 ## Errors -If an error occurs when the server processes a request, the server -must return an error payload as the response message, and correctly -set the HTTP status code. Please see -[`google.rpc.Code`](https://github.com/googleapis/googleapis/blob/master/google/rpc/code.proto) -on how to map typical server errors to HTTP status codes. +Twirp error responses are always JSON-encoded, regardless of +the request's Content-Type, with a corresponding +`Content-Type: application/json` header. This ensures that +the errors are human-readable in any setting. -### Timeout errors +Twirp errors are a JSON object with three keys: -For a single request, there is a client-specified timeout and a -server-configured timeout. If a request misses the server-configured -timeout, the server must return a `503` error. If a request misses -the client-specified timeout that it is shorter then the server- -configured timeout, the server must return a `504` error. -This allows more accurate measurement of the server availability. +* **code**: One of the Twirp error codes as a string. +* **msg**: A human-readable message describing the error + as a string. +* **meta**: An object with string keys and values holding + arbitrary additional metadata describing the error. -### Network errors +Example: +``` +{ + "code": "permission_denied", + "msg": "thou shall not pass", + "meta": { + "target": "Balrog" + } +} +``` -If a client fails to reach the server due to network errors, the -client library must report HTTP status code `502` to the client -application. This helps users distinguishing network errors from -server errors. +For more information, see https://github.com/twitchtv/twirp/wiki/Errors. From 9ba45910a1044a9b96a7437d0c2de5e1ddec329e Mon Sep 17 00:00:00 2001 From: mocax <35949852+mocax@users.noreply.github.com> Date: Tue, 30 Jan 2018 17:09:34 -0800 Subject: [PATCH 4/6] Use application/protobuf --- PROTOCOL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/PROTOCOL.md b/PROTOCOL.md index d0c1b490..7da8e95d 100644 --- a/PROTOCOL.md +++ b/PROTOCOL.md @@ -70,7 +70,7 @@ header. The **Response-Headers** are just normal HTTP response headers. The Twirp wire protocol uses the following headers. -* **Content-Type** The value should be either "application/x-protobuf" +* **Content-Type** The value should be either "application/protobuf" or "application/json" to indicate the encoding of the response message. It must match the "Content-Type" header in the request. From da5f789ff882fb5c81ebf42a56451100b266cfe1 Mon Sep 17 00:00:00 2001 From: mocax <35949852+mocax@users.noreply.github.com> Date: Thu, 1 Feb 2018 20:43:47 -0800 Subject: [PATCH 5/6] Update PROTOCOL.md Change proto package name to example.echoer. --- PROTOCOL.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/PROTOCOL.md b/PROTOCOL.md index 7da8e95d..1d0e0d89 100644 --- a/PROTOCOL.md +++ b/PROTOCOL.md @@ -78,7 +78,7 @@ The **Request-Body** is the encoded response message contained in the HTTP response body. The encoding is specified by the `Content-Type` header. -## Example +### Example The following example shows a simple Echo API definition and its corresponding wire payloads. @@ -88,7 +88,7 @@ The example assumes the server base URL is "https://example.com". ```proto syntax = "proto3"; -package twirp; +package example.echoer; service Echo { rpc Hello(HelloRequest) returns (HelloResponse); @@ -106,9 +106,9 @@ message HelloResponse { **Proto Request** ``` -POST /twirp.Echo/Hello HTTP/1.1 +POST /twirp/example.echoer.Echo/Hello HTTP/1.1 Host: example.com -Content-Type: application/protobuf +Content-Type: application/x-protobuf Content-Length: 15 @@ -117,7 +117,7 @@ Content-Length: 15 **JSON Request** ``` -POST /twirp.Echo/Hello HTTP/1.1 +POST /twirp/example.echoer.Echo/Hello HTTP/1.1 Host: example.com Content-Type: application/json Content-Length: 27 @@ -129,7 +129,7 @@ Content-Length: 27 ``` HTTP/1.1 200 OK -Content-Type: application/protobuf +Content-Type: application/x-protobuf Content-Length: 15 From e87649b5c7bb2b0ab9d5aa0812e1acba79b4a759 Mon Sep 17 00:00:00 2001 From: mocax <35949852+mocax@users.noreply.github.com> Date: Thu, 1 Feb 2018 20:45:49 -0800 Subject: [PATCH 6/6] Update PROTOCOL.md Use google/protobuf. --- PROTOCOL.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/PROTOCOL.md b/PROTOCOL.md index 1d0e0d89..0c8908a2 100644 --- a/PROTOCOL.md +++ b/PROTOCOL.md @@ -108,7 +108,7 @@ message HelloResponse { ``` POST /twirp/example.echoer.Echo/Hello HTTP/1.1 Host: example.com -Content-Type: application/x-protobuf +Content-Type: application/protobuf Content-Length: 15 @@ -129,7 +129,7 @@ Content-Length: 27 ``` HTTP/1.1 200 OK -Content-Type: application/x-protobuf +Content-Type: application/protobuf Content-Length: 15