custom api docs (#4346)

* custom api docs * metrics sql docs * metrics sql docs * broken doc link * review comments * Aligning custom api docs with refactored docs structure We recently overhauled our docs completely, with refactored content and a new structure. I'm pushing this commit to try to align the new custom API docs with these changes. For context, see: #4355 * Readjusting order in project files reference --------- Co-authored-by: Anshul Khandelwal <[email protected]> Co-authored-by: Andrew Tsao <[email protected]>
rilldata · Mar 17, 2024 · ff8388d · ff8388d
1 parent 3391bc1
commit ff8388d
Show file tree

Hide file tree

Showing 10 changed files with 245 additions and 7 deletions.
diff --git a/docs/docs/develop/import-data.md b/docs/docs/develop/import-data.md
diff --git a/docs/docs/integrate/custom-api.md b/docs/docs/integrate/custom-api.md
@@ -0,0 +1,43 @@
+---
+title: "Custom API Integration"
+description: How to integrate custom APIs with your application
+sidebar_label: "Custom API Integration"
+sidebar_position: 11
+---
+
+Rill exposes [custom APIs](/integrate/custom-apis/index.md) you have created under `apis` folder as HTTP endpoints 
+at `https://admin.rilldata.com/v1/organizations/<org-name>/projects/<project-name>/runtime/api/<api-name>`.
+
+## Accessing custom APIs
+You need to do a POST request to the API endpoint with A bearer token in the `Authorization` header.
+
+```bash
+curl -X POST https://admin.rilldata.com/v1/organizations/<org-name>/projects/<project-name>/runtime/api/<api-name>[?query-args] \
+-H "Authorization: Bearer <token>"
+```
+
+There are two types of bearer tokens that you can use to access the custom APIs:
+1. **Service Account Token**: You can use a service account token to access the custom APIs.
+    Read more about [Service Account Tokens](../reference/cli/service). 
+
+    :::note
+    1. Service accounts have full access to all APIs so if there are security policies defined for metrics view being used in metrics_sql API, they will not be enforced.
+    2. Also since there is no user context available, this means if the api being called uses `{{ .user.<attr> }}` in SQL templating, it will fail.
+    :::
+
+2. **User Token**: You can use a user token to access the custom API when you want user context and enforce security policies defined for the [metrics view](/build/dashboards/dashboards.md) being used in the `metrics_sql` API.
+    To get user token you need to perform a handshake with Rill's [credentials API](https://admin.rilldata.com/v1/organizations/<org-name>/projects/<project-name>/credentials) using a service account token. Example:
+
+    ```bash
+    curl -X POST https://admin.rilldata.com/v1/organizations/<org-name>/projects/<project-name>/credentials \
+    -H "Authorization: Bearer <service-account-token>"
+   --data-raw '{
+      "user_email":"<user-email>"
+    }'
+    ``` 
+   The API accepts the following parameters:
+    | Parameter | Description                                                                                                                                                                                    | Required                         |
+    | --- |------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------|
+    | user_email | The email of the user for which token is being asked for                                                                                                                                                | No (either this or `attributes`) |
+    | attributes | Json payload containing user attributes used for enforcing policies. When using this make sure to pass all the attributes used in your security policy like `email`, `domain` and `admin`| No (either this or `user_email`) |
+    | ttl_seconds | The time to live for the iframe URL                                                                                                                                                            | No (Default: 86400)              |
diff --git a/docs/docs/integrate/custom-apis/_category_.yml b/docs/docs/integrate/custom-apis/_category_.yml
@@ -0,0 +1,4 @@
+position: 14
+label: Custom APIs
+collapsible: true
+collapsed: true
diff --git a/docs/docs/integrate/custom-apis/index.md b/docs/docs/integrate/custom-apis/index.md
@@ -0,0 +1,35 @@
+---
+title: "Custom API"
+description: Create your own API to pull data out in flexible manner 
+sidebar_label: "Custom API"
+sidebar_position: 30
+---
+
+Rill allows you to create custom APIs to pull data out in a flexible manner. You can write custom SQL queries and expose them as an API endpoint.
+
+## Create a custom API
+
+To create a custom API, create a new yaml file under `apis` directory in your Rill project. Currently, 
+we support two types of custom APIs:
+
+1. **SQL API**: You can write a SQL query and expose it as an API endpoint. This is useful when you want to directly 
+    write queries against a [model](/build/models/models.md) that you have created. It should have the following structure:
+
+    ```yaml
+    kind: api
+    sql: SELECT abc FROM my_table
+    ```
+    where `my_table` is your model name. Read more details about [SQL apis](./sql-api.md).
+
+2. **Metrics SQL API**: You can write a SQL query referring to metrics definition and dimensions defined in the [metrics view](/build/dashboards/dashboards.md). 
+It should have the following structure:
+
+    ```yaml
+    kind: api
+    metrics_sql: SELECT dimension, AGGREGATE(measure) FROM my_metrics GROUP BY dimension
+    ```
+    where `my_metrics` is your metrics view name, `measure` is a custom metrics that you have defined. 
+    Read more details about [Metrics SQL API](./metrics-sql-api.md).
+
+## How to use custom APIs
+Refer to the integration docs [here](/integrate/custom-api.md) to learn how to use custom APIs in your application.
diff --git a/docs/docs/integrate/custom-apis/metrics-sql-api.md b/docs/docs/integrate/custom-apis/metrics-sql-api.md
@@ -0,0 +1,23 @@
+---
+title: Metrics SQL API
+sidebar_label: Metrics SQL API
+sidebar_position: 50
+---
+
+Metrics SQL API allows you to query data from your predefined [metrics view](/build/dashboards/dashboards.md) using [Metrics SQL](./metrics-sql.md) dialect. 
+
+Example:
+
+```yaml
+kind: api
+metrics_sql: SELECT dimension, AGGREGATE(measure) FROM my_metrics GROUP BY dimension
+```
+
+where `my_metrics` is your metrics view name, `measure` is a metrics that you have defined in the metrics view and `dimension` is a dimension defined in the metrics view.
+
+Read more the dialect here: [Metrics SQL](./metrics-sql.md).
+
+## Templating
+
+It supports templating the same way as [SQL API Templating](./sql-api.md#sql-templating).
+
diff --git a/docs/docs/integrate/custom-apis/metrics-sql.md b/docs/docs/integrate/custom-apis/metrics-sql.md
@@ -0,0 +1,68 @@
+---
+title: "Metrics SQL Syntax and Capabilities"
+description: Learn how to utilize SQL for extracting data from metrics views effectively.
+sidebar_label: "Metrics SQL"
+sidebar_position: 60
+---
+
+
+<!-- WARNING: There are links to this page in source code. If you move it, find and replace the links and consider adding a redirect in docusaurus.config.js. -->
+
+Metrics SQL is a specialized SQL dialect designed exclusively for querying data from metrics views.
+
+## Querying Fundamentals
+Metrics SQL transforms queries that reference `dimensions` and `measures` within a `metrics view` into their corresponding database columns or expressions. This transformation is based on the mappings defined in a metrics view YAML configuration, enabling reuse of dimension or measure defintions. Additionally any security policies defined in metrics_view is also inherited.
+
+## Example: Crafting a Metrics SQL Query
+
+Consider a metrics view configured as follows:
+```
+title: Ad Bids
+model: ad_bids
+timeseries: timestamp
+dimensions:
+  - label: Publisher
+    name: publisher
+    column: toUpper(publisher)
+    description: ""
+  - label: Domain
+    name: domain
+    column: domain
+    description: ""
+measures:
+  - name: total_records
+    label: Total records
+    expression: COUNT(*)
+    description: ""
+    format_preset: humanize
+    valid_percent_of_total: true
+```
+
+To query this view, a user might write a Metrics SQL query like:
+```
+SELECT publisher, domain, AGGREGATE(total_records) FROM metrics_view GROUP BY publisher, domain
+```
+This Metrics SQL is internally translated to a standard SQL query as follows:
+```
+SELECT toUpper(publisher) AS publisher, domain AS domain, COUNT(*) AS total_records FROM ad_bids GROUP BY publisher, domain
+```
+
+## Supported SQL Features
+
+- **SELECT** statements with plain `dimension` and `measure` references. The measures need to be accessed with a custom function `AGGREGATE(measure_name)`
+- A single **FROM** clause referencing a `metrics view`.
+- **WHERE** clause that can reference selected `dimensions` only.
+- Operators in **WHERE** and **HAVING** clauses include `=`, `!=`, `>`, `>=`, `<`, `<=`, IN, LIKE, AND, OR, and parentheses for structuring the expression.
+- **HAVING** clause for filtering on aggregated results, referencing selected dimension and measure names. Supports the same expression capabilities as the WHERE clause.
+- **ORDER BY** clause for sorting the results.
+- **LIMIT** and **OFFSET** clauses for controlling the result set size and pagination.
+
+
+**Caution** : The Metrics SQL feature is currently evolving. We are dedicated to enhancing the synatx by introducing additional SQL features, while striving to maintain support for existing syntax. However, please be advised that backward compatibility cannot be guaranteed at all times. Additionally, users should be aware that there may be untested edge cases in the current implementation. We appreciate your understanding as we work to refine and improve this feature.
+
+## Security and Compliance
+Queries executed via Metrics SQL are subject to the security policies and access controls defined in the metrics view YAML configuration, ensuring data security and compliance.
+
+
+## Limitations
+Metrics SQL is specifically designed for querying metrics views and may not support all features found in standard SQL. Its primary focus is on providing an efficient and easy way to extract data within the constraints of metrics view configurations.
diff --git a/docs/docs/integrate/custom-apis/sql-api.md b/docs/docs/integrate/custom-apis/sql-api.md
@@ -0,0 +1,47 @@
+---
+title: SQL API
+sidebar_label: SQL API
+sidebar_position: 40
+---
+
+You can write a SQL query and expose it as an API endpoint. This is useful when you want to directly write queries 
+against a [model](/build/models/models.md) or [source](../../reference/project-files/sources.md) that you have created. 
+It should have the following structure:
+
+```yaml
+kind: api
+sql: SELECT abc FROM my_table
+```
+
+where `my_table` is your model or source name.
+
+## SQL Templating
+
+You can use templating to make your SQL query dynamic. We support:
+ - Dynamic arguments that can be passed in as query params during api call using `{{ .args.<param-name> }}`
+ - User attributes like email, domain and admin if available using `{{ .user.<attr> }}` (see integration docs [here](/integrate/custom-api.md) for when user attributes are available)
+ - Conditional statements using go ang sprig templating functions (see resources at the end for more details) 
+
+For example:
+
+`my-api.yaml`:
+```yaml
+kind: api
+sql: |
+  SELECT count("measure")
+    {{ if ( .user.admin ) }} ,dim  {{ end }} 
+    FROM my_table WHERE date = '{{ .args.date }}' 
+    {{ if ( .user.admin ) }} group by 2 {{ end }}
+```
+
+will expose an API endpoint like `https://admin.rilldata.com/v1/organizations/<org-name>/projects/<project-name>/runtime/api/my-api?date=2021-01-01`.
+If the user is an admin, the API will return the count of `measure` by `dim` for the given date. If the user is not an admin, the API will return the count of `measure` for the given date.
+
+See integration docs [here](/integrate/custom-api.md) to learn how to these are passed in while calling the API.
+
+## Useful resources
+
+- [Official docs](https://pkg.go.dev/text/template) (Go)
+- [Learn Go Template Syntax](https://developer.hashicorp.com/nomad/tutorials/templates/go-template-syntax) (HashiCorp)
+- [Sprig Function Documentation](http://masterminds.github.io/sprig/)
+
diff --git a/docs/docs/integrate/embedding.md b/docs/docs/integrate/embedding.md
@@ -264,12 +264,12 @@ public class DashboardController {
 
 The API accepts the following parameters:
 
-| Parameter | Description                                                                                                                                                                                    | Required                         |
-| --- |------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------|
-| resource | The name of the dashboard to embed                                                                                                                                                             | Yes                              |
-| user_email | The email of the user to embed the dashboard for                                                                                                                                               | No (either this or `attributes`) |
-| attributes | Json payload to be put in the access token, used to pass attributes to the dashboard for enforcing policies. When using this make sure to pass all the attributes used in your security policy | No (either this or `user_email`) |
-| ttl_seconds | The time to live for the iframe URL                                                                                                                                                            | No (Default: 86400)              |
+| Parameter | Description                                                                                                                                                                                     | Required                         |
+| --- |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------|
+| resource | The name of the dashboard to embed                                                                                                                                                              | Yes                              |
+| user_email | The email of the user to embed the dashboard for                                                                                                                                                | No (either this or `attributes`) |
+| attributes | Json payload to be put in the access token, used to pass attributes to the dashboard for enforcing policies. When using this make sure to pass all the attributes used in your security policy like `email`, `domain` and `admin` | No (either this or `user_email`) |
+| ttl_seconds | The time to live for the iframe URL                                                                                                                                                             | No (Default: 86400)              |
 
 The response will contain an `iframeSrc` value that can be used to embed the dashboard in your application. It will also contain a `ttlSeconds` value, which indicates how long the iframe URL will be valid for. After the TTL has elapsed, the iframe URL needs be refreshed. Here's an example response:
 

diff --git a/docs/docs/reference/project-files/apis.md b/docs/docs/reference/project-files/apis.md
@@ -0,0 +1,18 @@
+---
+title: API YAML
+sidebar_label: API YAML
+sidebar_position: 50
+---
+
+In your Rill project directory, create a new file name `<api-name>.yaml` in the `apis` directory containing a custom API definition.
+See comprehensive documentation on how to define and use [custom APIs](/integrate/custom-apis/index.md)
+
+## Properties
+
+_**`kind`**_ — should always be `api` _(required)_
+
+Either one of the following:
+
+- _**`sql`**_ — General SQL query referring a [model](/build/models/models.md) _(required)_
+
+- _**`metrics_sql`**_ — SQL query referring metrics definition and dimensions defined in the [metrics view](/build/dashboards/dashboards.md) _(required)_
diff --git a/docs/docs/reference/project-files/themes.md b/docs/docs/reference/project-files/themes.md
@@ -1,7 +1,7 @@
 ---
 title: Theme YAML
 sidebar_label: Theme YAML
-sidebar_position: 41
+sidebar_position: 60
 hide_table_of_contents: true
 ---