From 017bca9f6377743d6c68c7b78a7a14eb8e0478e9 Mon Sep 17 00:00:00 2001 From: Sam Willis Date: Wed, 13 Dec 2023 12:50:24 +0000 Subject: [PATCH] chore(docs) Add proxy-tunnel docs and small api/typescript tidy up (#748) Co-authored-by: Oleksii Sholik --- components/electric/README.md | 1 + docs/api/cli.md | 91 ++++++++++++++++++++ docs/api/clients/typescript.md | 108 +++++++++++++++++++----- docs/api/generator.md | 43 ---------- docs/api/service.md | 2 +- docs/integrations/backend/prisma.md | 2 +- docs/quickstart/index.md | 2 +- docs/reference/architecture.md | 4 +- docs/usage/data-access/client.md | 2 +- docs/usage/data-modelling/migrations.md | 2 + docs/usage/installation/client.md | 8 +- 11 files changed, 193 insertions(+), 72 deletions(-) create mode 100644 docs/api/cli.md delete mode 100644 docs/api/generator.md diff --git a/components/electric/README.md b/components/electric/README.md index 32165dc0fc..a3b7f5f82a 100644 --- a/components/electric/README.md +++ b/components/electric/README.md @@ -104,6 +104,7 @@ The Electric application is configured using environment variables. Everything t | `ELECTRIC_INSTANCE_ID` | `electric` | Unique identifier of this Electric instance when running in a cluster (not yet supported). When running locally, you can use any string | [1]: https://electric-sql.com/docs/usage/data-modelling/migrations#migrations-proxy +[2]: https://electric-sql.com/docs/api/cli#proxy-tunnel **Authentication** diff --git a/docs/api/cli.md b/docs/api/cli.md new file mode 100644 index 0000000000..53a87af0da --- /dev/null +++ b/docs/api/cli.md @@ -0,0 +1,91 @@ +--- +title: CLI & Generator +description: >- + Command-line interface for the `npx electric-sql generate` script and other commands. +sidebar_position: 30 +--- + +# CLI Commands + +The Electric client library comes with a CLI providing a couple of helpful commands for building Electric apps: + +- [`generate`](#generate) - The type-safe client generator; this command builds the client library for your applications to use. +- [`proxy-tunnel`](#proxy-tunnel) - A tool to enable you to connect to the Electric migrations proxy when it's deployed behind an restrictive firewall, or with a hosting provider, that only allows incoming HTTP connections. + +These commands are all executed in the form `npx electric-sql [command-name]` from within a project where you have installed the [client library](../usage/installation/client). + +## `generate` + +To interface with your database from within a JavaScript or TypeScript application you need an Electric client (see for context). To generate an Electric client, make sure that your database and the Electric sync service are up and running. +Then, from within the root folder of your application run: + +```shell +npx electric-sql generate +``` + +This will download all migrations from the database, bundle them with your application, and generate the Electric client. +Note that you can use an optional `--watch` flag to automatically re-generate the client on every database migration. + +### Options + +The generate command accepts a number of arguments: + +```shell +npx electric-sql generate [--service ] [--proxy ] [--out ] [--watch []] +``` + +All arguments are optional and are described below: + +| Argument | Value | Description | +|----------|-------|-------------| +| `--service` | `` | Provides the url to connect to the Electric sync service. If not provided, it uses the url set in the `ELECTRIC_URL` environment variable. If that variable is not set, it resorts to the default url which is `http://localhost:5133`. | +| `--proxy` | `` | Provides the url to connect to Electric's database proxy. If not provided, it uses the url set in the `ELECTRIC_PROXY_URL` environment variable. If that variable is not set, it resorts to the default url which is `postgresql://prisma:proxy_password@localhost:65432/electric`. | +| `--out` | `` | Specifies where to output the generated client. Defaults to `./src/generated/client` | +| `--watch` | `` | Run the generator in watch mode. Accepts an optional polling interval (in milliseconds) which defines how often to poll Electric for database migrations. The default polling interval is 1000ms. | + +Note that the `--watch` flag can be used with or without a polling interval: + +```shell +npx electric-sql generate --watch +# or with an explicit polling interval +npx electric-sql generate --watch 5000 +``` + +## `proxy-tunnel` + +Some hosting providers only allow HTTP connections, which poses a challenge for deploying Electric to their platforms since it uses a separate port for connections to the [migrations proxy](../usage/data-modelling/migrations#migrations-proxy). In order to enable connecting to run migrations and use the generate command in these setups, you can enable a special "Proxy Tunnel" that tunnels the Postgres Proxy TCP connection over a Websocket to the Electric sync service. This is enabled on the sync service by setting the environment variable `PG_PROXY_PORT=http`. + +The `npx electric-sql proxy-tunnel` command is provided to forward TCP traffic from your local machine to the Electric Postgres Proxy when it has tunneling enabled. It binds to a local port, allowing you to use the generator command, perform migrations, and connect with psql. + +To connect to the service, and create a local proxy tunnel: + +```shell +npx electric-sql proxy-tunnel --service http://my.electric.host:5133 --local-port 65431 +``` + +Then to run migrations, if you are using [@databases/pg-migrations](https://www.atdatabases.org/docs/pg-migrations) as we do in our [starter template](../quickstart/?setup=generator#setup), you can run this in another shell: + +```shell +npx pg-migrations apply --database postgres://postgres:proxy_password@localhost:65431 --directory ./db/migrations +``` + +To then use the `generate` command to create your client: + +```shell +npx electric-sql generate --service http://my.electric.host:5133 --proxy postgresql://prisma:proxy_password@localhost:65431/electric +``` + +### Options + +The proxy-tunnel command accepts a number of arguments: + +```shell +npx electric-sql proxy-tunnel [--service ] [--local-port ] +``` + +All arguments are optional and are described below: + +| Argument | Value | Description | +|----------|-------|-------------| +| `--service` | `` | Provides the url to connect to the Electric sync service. If not provided, it uses the url set in the `ELECTRIC_URL` environment variable. If that variable is not set, it resorts to the default url which is `http://localhost:5133`. | +| `--local-port` | `` | The local port to bind to; this will be forwarded to the Electric sync service, and defaults to `65432`. | \ No newline at end of file diff --git a/docs/api/clients/typescript.md b/docs/api/clients/typescript.md index b780b66723..6b51fbad00 100644 --- a/docs/api/clients/typescript.md +++ b/docs/api/clients/typescript.md @@ -7,12 +7,12 @@ sidebar_position: 10 # Typescript client -The page explained how to generate an Electric client for your application. -In this section, we demonstrate the usage of a generated Electric client -for an issue tracking application where users participate in projects, -projects have issues, and users can comment on issues -(in which case they are said to be the author of that comment). -The data model for this application can be found on the page. +The Typescript client provides a number of functions for developing front-end applications with Electric: + +- [Authenticating](../../usage/auth/) with the sync service +- [Synchronising database](#shapes) to a local SQLite database +- [Type-safe data access](#queries) to read and update the database +- [Reactive live queries](#live-queries) that update in realtime as the database changes ## Configuration @@ -93,9 +93,13 @@ ELECTRIC_URL=http://localhost:5133 DEBUG_MODE=true npm run start ## Instantiation -To instantiate an Electric client -we need to electrify our database, -passing along the generated database schema. +A Typescript client comprises of: + +1. SQLite database connection from a [supported driver](../../integrations/drivers/) +2. A client schema [generated using the generator command](../cli.md#generate) +3. A [configuration object](#configuration) + +To instantiate the client, these are passed to an `electrify` function that is specific to your SQLite database driver and platform. ```ts import { schema } from './generated/client' @@ -111,7 +115,7 @@ const conn = await ElectricDatabase.init('electric.db', '') const electric = await electrify(conn, schema, config) ``` -The electrify call returns a promise that will resolve to an `ElectricClient` for our database. +The `electrify` call returns a promise that will resolve to an `ElectricClient` for our database. The client exposes the following interface: ```ts @@ -145,9 +149,82 @@ In addition, one can execute raw SQL queries using the `electric.db.raw` and `el Raw queries should be used with caution as they are unchecked and may cause the sync service to stop if they are ill-formed. Therefore, only use raw queries for features that are not supported by our regular API. -## Authentication +## Configuration + +The Electric client has a few configuration options that are defined on the `ElectricConfig` type available in +`electric-sql/config`. At a minimum, you have to include in the config object the URL to your instance of the +[sync service](../../usage/installation/service) and an [auth token](../../usage/auth), for example: + +```ts +const config: ElectricConfig = { + url: 'http://my-app-domain', + auth: { + token: 'eyJ0eXAiOiJKV1QiLCJhbGciOiJub25lIn0...' + } +} +``` + +### Available options + +- `auth: AuthConfig` + + Authentication object that includes an auth `token` and an optional `clientId`. + + `token` must be a JWT that the Electric sync service will be able to validate. + + `clientId` is a unique identifier for this particular client or device. If omitted, a random UUID will be generated + the first time this client connects to the sync service. + +- `url?: string` (default: `"http://localhost:5133"`) + + URL of the Electric sync service to connect to. + + Should have the following format: + + ``` + protocol://:[?ssl=true] + ``` + + If the protocol is `https` or `wss` then `ssl` defaults to true. Otherwise it defaults to false. + + If port is not provided, defaults to 443 when ssl is enabled or 80 when it isn't. + +- `debug?: boolean` (default: `false`) + + Activate debug mode which logs the replication messages that are exchanged between the client and the sync service. + +- `timeout?: number` (default: `3000`) + + Timeout (in milliseconds) for RPC requests. + + Needs to be large enough for the server to have time to deliver the full initial subscription data + when the client subscribes to a shape for the first time. + + +- `connectionBackOffOptions?: ConnectionBackOffOptions` + + Configuration of the backoff strategy used when trying to reconnect to the Electric sync service after a failed + connection attempt. -## Connectivity +### Configuring example apps + +In our example apps and in apps created with `npx create-electric-app`, the `url` and `debug` options are looked up as +`ELECTRIC_URL` and `DEBUG_MODE` environment variables, respectively. + +So, for example, to include the URL of a hosted instance of Electric into the production build of your app, put it in +the `ELECTRIC_URL` environment variable when running your build command: + +```shell +ELECTRIC_URL=https://my-app-domain.com npm run build +# or +ELECTRIC_URL=wss://my-app-domain.com npm run build +``` + +To run your app in development with debug mode enabled: + +```shell +ELECTRIC_URL=http://localhost:5133 DEBUG_MODE=true npm run start +``` ## Shapes @@ -198,13 +275,6 @@ When a table is not yet synced, it exists on the device's local database but is If you try to read from an unsynced table you will get empty results and a warning will be logged: > Reading from unsynced table memberships -### `discover` - -- Subsetting -- Retention -- Derived shapes -- Sync boundaries - ## Queries As explained before, every Electric client contains a `db` property representing the electrified database. diff --git a/docs/api/generator.md b/docs/api/generator.md deleted file mode 100644 index c2e75df5c1..0000000000 --- a/docs/api/generator.md +++ /dev/null @@ -1,43 +0,0 @@ ---- -title: Generator script -description: >- - Command-line interface for the `npx electric-sql generate` script. -sidebar_position: 30 ---- - -# Generator script - -To interface with your database from within a JavaScript or TypeScript application you need an Electric client (see for context). To generate an Electric client, make sure that your database and the Electric sync service are up and running. -Then, from within the root folder of your application run: - -```shell -npx electric-sql generate -``` - -This will download all migrations from the database, bundle them with your application, and generate the Electric client. -Note that you can use an optional `--watch` flag to automatically re-generate the client on every database migration. - -## Options - -The generate script accepts a number of arguments: - -```shell -npx electric-sql generate [--service ] [--proxy ] [--out ] [--watch []] -``` - -All arguments are optional and are described below: - -| Argument | value | description | -|----------|-------|-------------| -| `--service` | `` | Provides the url to connect to the Electric sync service. If not provided, it uses the url set in the `ELECTRIC_URL` environment variable. If that variable is not set, it resorts to the default url which is `http://localhost:5133`. | -| `--proxy` | `` | Provides the url to connect to Electric's database proxy. If not provided, it uses the url set in the `ELECTRIC_PROXY_URL` environment variable. If that variable is not set, it resorts to the default url which is `postgresql://prisma:proxy_password@localhost:65432/electric`. | -| `--out` | `` | Specifies where to output the generated client. Defaults to `./src/generated/client` | -| `--watch` | `` | Run the generator in watch mode. Accepts an optional polling interval (in milliseconds) which defines how often to poll Electric for database migrations. The default polling interval is 1000ms. | - -Note that the `--watch` flag can be used with or without a polling interval: - -```shell -npx electric-sql --watch -# or with an explicit polling interval -npx electric-sql --watch 5000 -``` diff --git a/docs/api/service.md b/docs/api/service.md index 977cb27a64..6047b1edf4 100644 --- a/docs/api/service.md +++ b/docs/api/service.md @@ -103,4 +103,4 @@ It's extremely helpful to leave telemetry enabled if you can. | Variable | Description | | -------------------------------------------------- | ------------------------------------------------------------------------------------------------ | -| `ELECTRIC_TELEMETRY`

  (`enabled`)

|

Telemetry mode. Telemetry is enabled by default. Set to `disabled` to disable collection.

| +| `ELECTRIC_TELEMETRY`

  (`enabled`)

|

Telemetry mode. Telemetry is enabled by default. Set to `disabled` to disable collection.

| \ No newline at end of file diff --git a/docs/integrations/backend/prisma.md b/docs/integrations/backend/prisma.md index 20dd013aec..8c0462e3a2 100644 --- a/docs/integrations/backend/prisma.md +++ b/docs/integrations/backend/prisma.md @@ -62,7 +62,7 @@ dotenv -e .env.proxy -- npx prisma migrate dev ## Generator -Electric uses a customised version of the [zod-prisma-types](https://github.com/chrishoermann/zod-prisma-types) Prisma client generator to [generate](../../api/generator.md) the [Electric Client](../../usage/data-access/client.md). +Electric uses a customised version of the [zod-prisma-types](https://github.com/chrishoermann/zod-prisma-types) Prisma client generator to [generate](../../api/cli.md#generate) the [Electric Client](../../usage/data-access/client.md). ## Event sourcing diff --git a/docs/quickstart/index.md b/docs/quickstart/index.md index ea30b2fe53..dcd01d21a6 100644 --- a/docs/quickstart/index.md +++ b/docs/quickstart/index.md @@ -23,7 +23,7 @@ If you'd prefer to understand a bit more about the system before jumping into co ## Setup -Get setup quickly using the [`create-electric-app`](../api/generator.md) starter app. Or install, run and integrate the components yourself. +Get setup quickly using the `create-electric-app` starter app. Or install, run and integrate the components yourself. diff --git a/docs/reference/architecture.md b/docs/reference/architecture.md index a4e5363ca6..408354f885 100644 --- a/docs/reference/architecture.md +++ b/docs/reference/architecture.md @@ -20,7 +20,7 @@ ElectricSQL has three primary components. [Postgres](../usage/installation/postg [![High level component diagramme](./_images/high-level-components.png)](./_images/high-level-components.jpg) -Inside the Client there is a [generated](../api/generator.md) type-safe [data access library](../usage/data-access/client.md). A [satellite replication process](https://github.com/electric-sql/electric/blob/main/clients/typescript/src/satellite/process.ts) (per named local database), SQLite [driver adapters](../integrations/drivers/index.md) and a [reactivity system](#reactivity) with [front-end framework integrations](../integrations/frontend/index.md). +Inside the Client there is a [generated](../api/cli.md#generate) type-safe [data access library](../usage/data-access/client.md). A [satellite replication process](https://github.com/electric-sql/electric/blob/main/clients/typescript/src/satellite/process.ts) (per named local database), SQLite [driver adapters](../integrations/drivers/index.md) and a [reactivity system](#reactivity) with [front-end framework integrations](../integrations/frontend/index.md). ### Topology @@ -41,7 +41,7 @@ The Postgres schema is managed using any [standard migration tooling](../usage/d ### Bundling -Electric provides an HTTP "status API" to access the electrified DDL schema. In development, the [generator script](../api/generator.md) calls this API to pull down the schema and uses it to: +Electric provides an HTTP "status API" to access the electrified DDL schema. In development, the [generator command](../api/cli.md#generate) calls this API to pull down the schema and uses it to: 1. generate a type-safe data access library 2. write an array of SQLite-compatible migrations into an importable Javascript file diff --git a/docs/usage/data-access/client.md b/docs/usage/data-access/client.md index b17f96fd61..c017281d73 100644 --- a/docs/usage/data-access/client.md +++ b/docs/usage/data-access/client.md @@ -11,7 +11,7 @@ Generate this Typescript client as part of your build process, instantiate it wi ## Generating the client -Use the [generator script](../../api/generator.md) to generate the [Typescript client](../installation/client.md): +Use the [generator command](../../api/cli.md#generate) to generate the [Typescript client](../installation/client.md): ```shell npx electric-sql generate diff --git a/docs/usage/data-modelling/migrations.md b/docs/usage/data-modelling/migrations.md index da34b22351..f4deb64896 100644 --- a/docs/usage/data-modelling/migrations.md +++ b/docs/usage/data-modelling/migrations.md @@ -214,6 +214,8 @@ There are two environment variables that configure the proxy in Electric: - `PG_PROXY_PORT` (default `65432`). This is the TCP port that [**Electric sync service**](../../api/service.md) will listen on. You should connect to it in order to pass through the migration proxy. Since the proxy speaks fluent Postgres, you can connect to it via any Postgres-compatible tool, e.g. `psql -U electric -p 65432 electric` + Some deployment targets restrict you to a single port for your service. On these platforms, `PG_PROXY_PORT` can be set to a special `http` value. This enables the use of the [Proxy Tunnel](../../api/cli.md#proxy-tunnel). Additionally, to set both the TCP port and enable the Proxy Tunnel, use a value such as `http:65432`. + - `PG_PROXY_PASSWORD` (no default). Access to the proxy is controlled by password (see below for information on the username). You must set this password here and pass it to any application hoping to connect to the proxy. You should be able to connect to the proxy directly using `psql` as outlined above and run any DDLX/migration commands you like. These will be validated, captured, and streamed to any connected clients automatically: diff --git a/docs/usage/installation/client.md b/docs/usage/installation/client.md index f212dd8f4e..500f4b9491 100644 --- a/docs/usage/installation/client.md +++ b/docs/usage/installation/client.md @@ -6,7 +6,7 @@ description: >- sidebar_position: 50 --- -Install the [`electric-sql`](https://www.npmjs.com/package/electric-sql) library and [`npx electric-sql generate`](../../api/generator.md) your database [Client](../data-access/client.md). +Install the [`electric-sql`](https://www.npmjs.com/package/electric-sql) library and [`npx electric-sql generate`](../../api/cli.md#generate) your database [Client](../data-access/client.md). ## Install @@ -71,7 +71,7 @@ Your local app will pick up the changes. ## Generate -Use the [Generator script](../../api/generator.md) to generate a type-safe database client. First, make sure your [sync service is running](./service.md). +Use the [Generator command](../../api/cli.md#generate) to generate a type-safe database client. First, make sure your [sync service is running](./service.md). Then run: @@ -81,11 +81,11 @@ npx electric-sql generate By default this will connect to the [sync service](./service.md) on `http://localhost:5133` and output generated files to a `./src/generated/client` folder. -See for the command-line options. +See for the command-line options. ### Outputs -The generator script outputs a database schema, type-safe database client and bundled SQL migrations. See for details on how to import and use these when instantiating a database client. +The generator command outputs a database schema, type-safe database client and bundled SQL migrations. See for details on how to import and use these when instantiating a database client. ### Pre-build script