chore(docs) Add proxy-tunnel docs and small api/typescript tidy up (#748

) Co-authored-by: Oleksii Sholik <[email protected]>
electric-sql · Dec 13, 2023 · 017bca9 · 017bca9
1 parent be5fbbb
commit 017bca9
Show file tree

Hide file tree

Showing 11 changed files with 193 additions and 72 deletions.
diff --git a/components/electric/README.md b/components/electric/README.md
@@ -104,6 +104,7 @@ The Electric application is configured using environment variables. Everything t
 | `ELECTRIC_INSTANCE_ID`      | `electric`            | Unique identifier of this Electric instance when running in a cluster (not yet supported). When running locally, you can use any string                                                                                                                                                                                                                                                              |
 
 [1]: https://electric-sql.com/docs/usage/data-modelling/migrations#migrations-proxy
+[2]: https://electric-sql.com/docs/api/cli#proxy-tunnel
 
 **Authentication**
 

diff --git a/docs/api/cli.md b/docs/api/cli.md
@@ -0,0 +1,91 @@
+---
+title: CLI & Generator
+description: >-
+  Command-line interface for the `npx electric-sql generate` script and other commands.
+sidebar_position: 30
+---
+
+# CLI Commands
+
+The Electric client library comes with a CLI providing a couple of helpful commands for building Electric apps:
+
+- [`generate`](#generate) - The type-safe client generator; this command builds the client library for your applications to use.
+- [`proxy-tunnel`](#proxy-tunnel) - A tool to enable you to connect to the Electric migrations proxy when it's deployed behind an restrictive firewall, or with a hosting provider, that only allows incoming HTTP connections.
+
+These commands are all executed in the form `npx electric-sql [command-name]` from within a project where you have installed the [client library](../usage/installation/client).
+
+## `generate`
+
+To interface with your database from within a JavaScript or TypeScript application you need an Electric client (see <DocPageLink path="usage/data-access/client" /> for context). To generate an Electric client, make sure that your database and the Electric sync service are up and running.
+Then, from within the root folder of your application run:
+
+```shell
+npx electric-sql generate
+```
+
+This will download all migrations from the database, bundle them with your application, and generate the Electric client.
+Note that you can use an optional `--watch` flag to automatically re-generate the client on every database migration.
+
+### Options
+
+The generate command accepts a number of arguments:
+
+```shell
+npx electric-sql generate [--service <url>] [--proxy <url>] [--out <path>] [--watch [<pollingInterval>]]
+```
+
+All arguments are optional and are described below:
+
+| Argument | Value | Description |
+|----------|-------|-------------|
+| <span className="no-wrap">`--service`</span> | `<url>` | Provides the url to connect to the Electric sync service. If not provided, it uses the url set in the `ELECTRIC_URL` environment variable. If that variable is not set, it resorts to the default url which is `http://localhost:5133`. |
+| <span className="no-wrap">`--proxy`</span> | `<url>` | Provides the url to connect to Electric's database proxy. If not provided, it uses the url set in the `ELECTRIC_PROXY_URL` environment variable. If that variable is not set, it resorts to the default url which is <span class="break-all">`postgresql://prisma:proxy_password@localhost:65432/electric`</span>. |
+| <span className="no-wrap">`--out`</span> | `<path>` | Specifies where to output the generated client. Defaults to `./src/generated/client` |
+| <span className="no-wrap">`--watch`</span> | `<pollingInterval>` | Run the generator in watch mode. Accepts an optional polling interval (in milliseconds) which defines how often to poll Electric for database migrations. The default polling interval is 1000ms. |
+
+Note that the `--watch` flag can be used with or without a polling interval:
+
+```shell
+npx electric-sql generate --watch
+# or with an explicit polling interval
+npx electric-sql generate --watch 5000
+```
+
+## `proxy-tunnel`
+
+Some hosting providers only allow HTTP connections, which poses a challenge for deploying Electric to their platforms since it uses a separate port for connections to the [migrations proxy](../usage/data-modelling/migrations#migrations-proxy). In order to enable connecting to run migrations and use the generate command in these setups, you can enable a special "Proxy Tunnel" that tunnels the Postgres Proxy TCP connection over a Websocket to the Electric sync service. This is enabled on the sync service by setting the environment variable `PG_PROXY_PORT=http`.
+
+The `npx electric-sql proxy-tunnel` command is provided to forward TCP traffic from your local machine to the Electric Postgres Proxy when it has tunneling enabled. It binds to a local port, allowing you to use the generator command, perform migrations, and connect with psql.
+
+To connect to the service, and create a local proxy tunnel:
+
+```shell
+npx electric-sql proxy-tunnel --service http://my.electric.host:5133 --local-port 65431
+```
+
+Then to run migrations, if you are using [@databases/pg-migrations](https://www.atdatabases.org/docs/pg-migrations) as we do in our [starter template](../quickstart/?setup=generator#setup), you can run this in another shell:
+
+```shell
+npx pg-migrations apply --database postgres://postgres:proxy_password@localhost:65431 --directory ./db/migrations
+```
+
+To then use the `generate` command to create your client:
+
+```shell
+npx electric-sql generate --service http://my.electric.host:5133 --proxy postgresql://prisma:proxy_password@localhost:65431/electric
+```
+
+### Options
+
+The proxy-tunnel command accepts a number of arguments:
+
+```shell
+npx electric-sql proxy-tunnel [--service <url>] [--local-port <port>]
+```
+
+All arguments are optional and are described below:
+
+| Argument | Value | Description |
+|----------|-------|-------------|
+| <span className="no-wrap">`--service`</span> | `<url>` | Provides the url to connect to the Electric sync service. If not provided, it uses the url set in the `ELECTRIC_URL` environment variable. If that variable is not set, it resorts to the default url which is `http://localhost:5133`. |
+| <span className="no-wrap">`--local-port`</span> | `<port>` | The local port to bind to; this will be forwarded to the Electric sync service, and defaults to `65432`. |
diff --git a/docs/api/clients/typescript.md b/docs/api/clients/typescript.md
@@ -7,12 +7,12 @@ sidebar_position: 10
 
 # Typescript client
 
-The <DocPageLink path="api/generator" /> page explained how to generate an Electric client for your application.
-In this section, we demonstrate the usage of a generated Electric client
-for an issue tracking application where users participate in projects,
-projects have issues, and users can comment on issues
-(in which case they are said to be the author of that comment).
-The data model for this application can be found on the <DocPageLink path="api/clients/typescript" /> page.
+The Typescript client provides a number of functions for developing front-end applications with Electric: 
+
+- [Authenticating](../../usage/auth/) with the sync service
+- [Synchronising database](#shapes) to a local SQLite database
+- [Type-safe data access](#queries) to read and update the database
+- [Reactive live queries](#live-queries) that update in realtime as the database changes
 
 ## Configuration
 
@@ -93,9 +93,13 @@ ELECTRIC_URL=http://localhost:5133 DEBUG_MODE=true npm run start
 
 ## Instantiation
 
-To instantiate an Electric client
-we need to electrify our database,
-passing along the generated database schema.
+A Typescript client comprises of:
+
+1. SQLite database connection from a [supported driver](../../integrations/drivers/)
+2. A client schema [generated using the generator command](../cli.md#generate)
+3. A [configuration object](#configuration)
+
+To instantiate the client, these are passed to an `electrify` function that is specific to your SQLite database driver and platform.
 
 ```ts
 import { schema } from './generated/client'
@@ -111,7 +115,7 @@ const conn = await ElectricDatabase.init('electric.db', '')
 const electric = await electrify(conn, schema, config)
 ```
 
-The electrify call returns a promise that will resolve to an `ElectricClient` for our database.
+The `electrify` call returns a promise that will resolve to an `ElectricClient` for our database.
 The client exposes the following interface:
 
 ```ts
@@ -145,9 +149,82 @@ In addition, one can execute raw SQL queries using the `electric.db.raw` and `el
 Raw queries should be used with caution as they are unchecked and may cause the sync service to stop if they are ill-formed.
 Therefore, only use raw queries for features that are not supported by our regular API.
 
-## Authentication
+## Configuration
+
+The Electric client has a few configuration options that are defined on the `ElectricConfig` type available in
+`electric-sql/config`. At a minimum, you have to include in the config object the URL to your instance of the
+[sync service](../../usage/installation/service) and an [auth token](../../usage/auth), for example:
+
+```ts
+const config: ElectricConfig = {
+  url: 'http://my-app-domain',
+  auth: {
+    token: 'eyJ0eXAiOiJKV1QiLCJhbGciOiJub25lIn0...'
+  }
+}
+```
+
+### Available options
+
+- `auth: AuthConfig`
+
+   Authentication object that includes an auth `token` and an optional `clientId`.
+
+   `token` must be a JWT that the Electric sync service will be able to validate.
+
+   `clientId` is a unique identifier for this particular client or device. If omitted, a random UUID will be generated
+   the first time this client connects to the sync service.
+
+- `url?: string` (default: `"http://localhost:5133"`)
+
+   URL of the Electric sync service to connect to.
+
+   Should have the following format:
+
+   ```
+   protocol://<host>:<port>[?ssl=true]
+   ```
+
+   If the protocol is `https` or `wss` then `ssl` defaults to true. Otherwise it defaults to false.
+
+   If port is not provided, defaults to 443 when ssl is enabled or 80 when it isn't.
+
+- `debug?: boolean` (default: `false`)
+
+  Activate debug mode which logs the replication messages that are exchanged between the client and the sync service.
+
+- `timeout?: number` (default: `3000`)
+
+  Timeout (in milliseconds) for RPC requests.
+
+  Needs to be large enough for the server to have time to deliver the full initial subscription data
+  when the client subscribes to a shape for the first time.
+
+
+- `connectionBackOffOptions?: ConnectionBackOffOptions`
+
+   Configuration of the backoff strategy used when trying to reconnect to the Electric sync service after a failed
+   connection attempt.
 
-## Connectivity
+### Configuring example apps
+
+In our example apps and in apps created with `npx create-electric-app`, the `url` and `debug` options are looked up as
+`ELECTRIC_URL` and `DEBUG_MODE` environment variables, respectively.
+
+So, for example, to include the URL of a hosted instance of Electric into the production build of your app, put it in
+the `ELECTRIC_URL` environment variable when running your build command:
+
+```shell
+ELECTRIC_URL=https://my-app-domain.com npm run build
+# or
+ELECTRIC_URL=wss://my-app-domain.com npm run build
+```
+
+To run your app in development with debug mode enabled:
+
+```shell
+ELECTRIC_URL=http://localhost:5133 DEBUG_MODE=true npm run start
+```
 
 ## Shapes
 
@@ -198,13 +275,6 @@ When a table is not yet synced, it exists on the device's local database but is
 If you try to read from an unsynced table you will get empty results and a warning will be logged:
 > Reading from unsynced table memberships
 
-### `discover`
-
-- Subsetting
-- Retention
-- Derived shapes
-- Sync boundaries
-
 ## Queries
 
 As explained before, every Electric client contains a `db` property representing the electrified database.

diff --git a/docs/api/generator.md b/docs/api/generator.md
diff --git a/docs/api/service.md b/docs/api/service.md
@@ -103,4 +103,4 @@ It's extremely helpful to leave telemetry enabled if you can.
 
 | Variable                                           | Description                                                                                      |
 | -------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
-| `ELECTRIC_TELEMETRY`<p>&nbsp;&nbsp;(`enabled`)</p> | <p>Telemetry mode. Telemetry is enabled by default. Set to `disabled` to disable collection.</p> |
+| `ELECTRIC_TELEMETRY`<p>&nbsp;&nbsp;(`enabled`)</p> | <p>Telemetry mode. Telemetry is enabled by default. Set to `disabled` to disable collection.</p> |
diff --git a/docs/integrations/backend/prisma.md b/docs/integrations/backend/prisma.md
@@ -62,7 +62,7 @@ dotenv -e .env.proxy -- npx prisma migrate dev
 
 ## Generator
 
-Electric uses a customised version of the [zod-prisma-types](https://github.com/chrishoermann/zod-prisma-types) Prisma client generator to [generate](../../api/generator.md) the [Electric Client](../../usage/data-access/client.md).
+Electric uses a customised version of the [zod-prisma-types](https://github.com/chrishoermann/zod-prisma-types) Prisma client generator to [generate](../../api/cli.md#generate) the [Electric Client](../../usage/data-access/client.md).
 
 ## Event sourcing
 

diff --git a/docs/quickstart/index.md b/docs/quickstart/index.md
@@ -23,7 +23,7 @@ If you'd prefer to understand a bit more about the system before jumping into co
 
 ## Setup
 
-Get setup quickly using the [`create-electric-app`](../api/generator.md) starter app. Or install, run and integrate the components yourself.
+Get setup quickly using the `create-electric-app` starter app. Or install, run and integrate the components yourself.
 
 <Tabs groupId="setup" queryString>
   <TabItem value="generator" label="Use the starter">

diff --git a/docs/reference/architecture.md b/docs/reference/architecture.md
@@ -20,7 +20,7 @@ ElectricSQL has three primary components. [Postgres](../usage/installation/postg
 
 [![High level component diagramme](./_images/high-level-components.png)](./_images/high-level-components.jpg)
 
-Inside the Client there is a [generated](../api/generator.md) type-safe [data access library](../usage/data-access/client.md). A [satellite replication process](https://github.com/electric-sql/electric/blob/main/clients/typescript/src/satellite/process.ts) (per named local database), SQLite [driver adapters](../integrations/drivers/index.md) and a [reactivity system](#reactivity) with [front-end framework integrations](../integrations/frontend/index.md).
+Inside the Client there is a [generated](../api/cli.md#generate) type-safe [data access library](../usage/data-access/client.md). A [satellite replication process](https://github.com/electric-sql/electric/blob/main/clients/typescript/src/satellite/process.ts) (per named local database), SQLite [driver adapters](../integrations/drivers/index.md) and a [reactivity system](#reactivity) with [front-end framework integrations](../integrations/frontend/index.md).
 
 ### Topology
 
@@ -41,7 +41,7 @@ The Postgres schema is managed using any [standard migration tooling](../usage/d
 
 ### Bundling
 
-Electric provides an HTTP "status API" to access the electrified DDL schema. In development, the [generator script](../api/generator.md) calls this API to pull down the schema and uses it to:
+Electric provides an HTTP "status API" to access the electrified DDL schema. In development, the [generator command](../api/cli.md#generate) calls this API to pull down the schema and uses it to:
 
 1. generate a type-safe data access library
 2. write an array of SQLite-compatible migrations into an importable Javascript file

diff --git a/docs/usage/data-access/client.md b/docs/usage/data-access/client.md
@@ -11,7 +11,7 @@ Generate this Typescript client as part of your build process, instantiate it wi
 
 ## Generating the client
 
-Use the [generator script](../../api/generator.md) to generate the [Typescript client](../installation/client.md):
+Use the [generator command](../../api/cli.md#generate) to generate the [Typescript client](../installation/client.md):
 
 ```shell
 npx electric-sql generate

diff --git a/docs/usage/data-modelling/migrations.md b/docs/usage/data-modelling/migrations.md
@@ -214,6 +214,8 @@ There are two environment variables that configure the proxy in Electric:
 
 - `PG_PROXY_PORT` (default `65432`). This is the TCP port that [**Electric sync service**](../../api/service.md) will listen on. You should connect to it in order to pass through the migration proxy. Since the proxy speaks fluent Postgres, you can connect to it via any Postgres-compatible tool, e.g. `psql -U electric -p 65432 electric`
 
+   Some deployment targets restrict you to a single port for your service. On these platforms, `PG_PROXY_PORT` can be set to a special `http` value. This enables the use of the [Proxy Tunnel](../../api/cli.md#proxy-tunnel). Additionally, to set both the TCP port and enable the Proxy Tunnel, use a value such as `http:65432`.
+
 - `PG_PROXY_PASSWORD` (no default). Access to the proxy is controlled by password (see below for information on the username). You must set this password here and pass it to any application hoping to connect to the proxy.
 
 You should be able to connect to the proxy directly using `psql` as outlined above and run any DDLX/migration commands you like. These will be validated, captured, and streamed to any connected clients automatically: