diff --git a/docs/blog/0.38.md b/docs/blog/0.38.md index 04202d07951..6f7dabdbfa8 100644 --- a/docs/blog/0.38.md +++ b/docs/blog/0.38.md @@ -37,7 +37,7 @@ Read more about how to configure Themes by visiting our docs page [here](../../r ## Available Time Ranges Rill has always treated time as an essential part of the product and with this release you will get even more capabilities around customizing selectable time periods and comparison periods. -The `dashboard.yaml` introduces a new property called [available_time_ranges](/reference/project-files/dashboards) that allows you define time ranges in [this format](../../reference/rill-iso-extensions). +The `dashboard.yaml` introduces a new property called [available_time_ranges](/reference/project-files/explore-dashboards) that allows you define time ranges in [this format](../../reference/rill-iso-extensions). You can either define this on a dashboard level or roll it out to all our dashboards via project defaults. ## Bug Fixes and Misc diff --git a/docs/blog/0.41.md b/docs/blog/0.41.md index d9d9d3833d6..b9d92e372b5 100644 --- a/docs/blog/0.41.md +++ b/docs/blog/0.41.md @@ -26,7 +26,7 @@ We have added a new driver for Clickhouse that allows you to have a live connect Refer docs [here](/reference/olap-engines/clickhouse) for more details. ## Default comparisons -Previously in Rill, time comparison were turned on by default. We have now switched to comparisons being off by default and also introduced new syntax to the [`dashboard.yaml`](/reference/project-files/dashboards) that allows you to specify either your own time or dimensional comparison modes. +Previously in Rill, time comparison were turned on by default. We have now switched to comparisons being off by default and also introduced new syntax to the [`dashboard.yaml`](/reference/project-files/explore-dashboards) that allows you to specify either your own time or dimensional comparison modes. ## Dimension expressions In addition to being able to define a specific column in your dataset as a dimension you can now supply an expression instead. That allows you to either visually transform the data without touching the underlying data model or for example to un-nest a nested column directly from your dimensions. @@ -35,7 +35,7 @@ In addition to being able to define a specific column in your dataset as a dimen If Pepperidge farm remembers so should Rill. We will now remember which dimensions and measures you were looking at and will automatically apply those for you when you come back to a dashboard. ## Default Dimensions and Measures -Dashboards can get crowded, the number of dimensions and measures grows over time. You can now specify which dimensions and measures that should be visible by default on the dashboard through new syntax in the [`dashboard.yaml`](/reference/project-files/dashboards). This allows you to control which measures and dimensions that should be visible for your users, making the initial cognitive load smaller while still allowing you to add more to the dashboard as you need them. +Dashboards can get crowded, the number of dimensions and measures grows over time. You can now specify which dimensions and measures that should be visible by default on the dashboard through new syntax in the [`dashboard.yaml`](/reference/project-files/explore-dashboards). This allows you to control which measures and dimensions that should be visible for your users, making the initial cognitive load smaller while still allowing you to add more to the dashboard as you need them. ## Support for environment-specific YAML configurations We love our local developer tool but when working with very large datasets locally even the best of laptop can slow to a crawl. diff --git a/docs/docs/build/dashboards/customize.md b/docs/docs/build/dashboards/customize.md index e975bdb4d37..c43af1b66f1 100644 --- a/docs/docs/build/dashboards/customize.md +++ b/docs/docs/build/dashboards/customize.md @@ -11,25 +11,14 @@ You will find below some common customizations and dashboard configurations that :::info Dashboard properties -For a full list of available dashboard properties and configurations, please see our [Dashboard YAML](/reference/project-files/dashboards.md) reference page. +For a full list of available dashboard properties and configurations, please see our [Dashboard YAML](/reference/project-files/explore-dashboards.md) reference page. ::: -### Changing Available Filters -**`default_time_range`** +**`time_ranges`** -Default time range controls the data analyzed on initial page load. Setting the default time range improves user experience by setting to most frequently used period - in particular, avoiding `all time` if you have a large datasource but only analyze more recent data. - -The value must be either a valid [ISO 8601 duration](https://en.wikipedia.org/wiki/ISO_8601#Durations) (for example, `PT12H` for 12 hours, `P1M` for 1 month, or `P26W` for 26 weeks) or one of the [Rill ISO 8601 extensions](../../reference/rill-iso-extensions.md#extensions). - -**`smallest_time_grain`** - -Smallest time grain available for your users. Rill will try to infer the smallest time grain. One of the most common reasons to change this setting is your data has timestamps but is actually in hourly or daily increments. The valid values are: `millisecond`, `second`, `minute`, `hour`, `day`, `week`, `month`, `quarter`, `year`. - -**`available_time_ranges`** - -One of the more important configurations, available time ranges allow you to change the defaults in the time dropdown for periods to select. Updating this list allows users to quickly change between the most common analyses like day over day, recent weeks, or period to date. The range must be a valid [ISO 8601 duration](https://en.wikipedia.org/wiki/ISO_8601#Durations) or one of the [Rill ISO 8601 extensions](../../reference/rill-iso-extensions.md#extensions) - example below. +One of the more important configurations, available time ranges allow you to change the defaults in the time dropdown for periods to select. Updating this list allows users to quickly change between the most common analyses like day over day, recent weeks, or period to date. The range must be a valid [ISO 8601 duration](https://en.wikipedia.org/wiki/ISO_8601#Durations) or one of the [Rill ISO 8601 extensions](../../reference/rill-iso-extensions.md#extensions). ```yaml available_time_ranges: @@ -41,30 +30,66 @@ One of the more important configurations, available time ranges allow you to cha - rill-WTD ## Week-To-date ``` -**`available_time_zones`** +**`time_zones`** Rill will automatically select several time zones that should be pinned to the top of the time zone selector. It should be a list of [IANA time zone identifiers](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones). You can add or remove from this list for the relevant time zones for your team. +```yaml + - America/Los_Angeles + - America/Chicago + - America/New_York + - Europe/London + - Europe/Paris + - Asia/Jerusalem + - Europe/Moscow + - Asia/Kolkata + - Asia/Shanghai + - Asia/Tokyo + - Australia/Sydney +``` + ### Setting Default Views for Dashboards +:::tip +Starting from version 0.50, the default views have been consolidated into a single YAML struct, `defaults:`. +::: -**`default_dimensions`** +**`time_range`** -For dashboards with wide tables, setting default dimensions is a good way to make sure that users can focus on the primary analyses and ensure a positive first experience. Each dimension listed under the `default_dimensions` setting would appear on the screen, while the remainder of the dimensions would be hidden (and still available for selection under filters). Common use cases for setting default fields would be to simplify dashboards on initial load, to narrow the dashboard to the most used fields, and to avoid high cardinality fields (that may take longer to load, but are used less often so improve performance). An example addition to the `dashboard.yaml` file is below. +Default time range controls the data analyzed on initial page load. Setting the default time range improves user experience by setting to most frequently used period - in particular, avoiding `all time` if you have a large datasource but only analyze more recent data. + +The value must be either a valid [ISO 8601 duration](https://en.wikipedia.org/wiki/ISO_8601#Durations) (for example, `PT12H` for 12 hours, `P1M` for 1 month, or `P26W` for 26 weeks) or one of the [Rill ISO 8601 extensions](../../reference/rill-iso-extensions.md#extensions). + + +**`dimensions`** + +For dashboards with wide tables, setting default dimensions is a good way to make sure that users can focus on the primary analyses and ensure a positive first experience. Each dimension listed under the `dimensions` setting would appear on the screen, while the remainder of the dimensions would be hidden (and still available for selection under filters). Common use cases for setting default fields would be to simplify dashboards on initial load, to narrow the dashboard to the most used fields, and to avoid high cardinality fields (that may take longer to load, but are used less often so improve performance). An example addition to the `dashboard.yaml` file is below. ```yaml -default_dimensions: - - column1 - - column2 +defaults: + dimensions: + - column1 + - column2 ``` :::warning Column vs. Name Usage The `column` property is used by default from the column name in your underlying source. If you decide to use the `name` property, you'd replace the column above with the field name. ::: -**`default_measures`** +**`measures`** A list of measures that should be visible by default. Operates the same as the `default_dimensions` configuration. When selecting measures, by default, consider hiding more computationally intensive measures like `count distinct` or other complicated expressions to improve performance. +```yaml +defaults: + measures: + - measure_1 + - measure_1 +``` + +**`security`** + +Defining security policies for your data is crucial for security. For more information on this, please refer to our [Dashboard Access Policies](/manage/security.md) + ## Changing Themes & Colors In your Rill project directory, create a `.yaml` file in any directory. Rill will automatically ingest the theme next time you run `rill start` or deploy to Rill Cloud and change the color scheme of your dashboard. @@ -80,7 +105,7 @@ colors: Once you have created that file, update the `dashboard.yaml` with the following configuration (we typically add this at the top along with time zones, time series and other configurations): -`default_theme: ` +`theme: ` :::info Theme properties @@ -88,28 +113,30 @@ For more details about configuring themes, you can refer to our [Theme YAML](/re ::: -## Dashboard Measure Formatting +## Example -Formatting for individual measures on a dashboard can be controlled through the `format_preset` and `format_d3` properties. +```yaml +type: explore -### Euro currency +title: Title of your explore dashboard +description: a description +metrics_view: -Instead of displaying a numeric measure in USD currency, the euro currency can be used instead through `format_preset`. In the following example `dashboard.yaml`, the `Total Revenue` measure will be displayed using euros in the resulting dashboard. +dimensions: '*' #can use regex +measures: '*' #can use regex -```yaml -type: metrics_view -title: example_dashboard -model: example_model -measures: - - label: "Total Revenue" - expression: SUM(revenue) - name: total_revenue - description: "This measure calculates the total sum of revenue" - format_preset: currency_eur -``` +theme: #your default theme -:::info Measure formatting options +time_ranges: #was available_time_ranges +time_zones: #was available_time_zones -For a full list of available dashboard measure formatting options, please see our [Dashboard YAML](/reference/project-files/dashboards.md) reference page. +defaults: #define all the defaults within here + dimensions: + measures: + time_range: + comparison_mode: + comparison_dimension: -::: \ No newline at end of file +security: + access: #only access can be set on dashboard level, see metric view for detailed access policies +``` \ No newline at end of file diff --git a/docs/docs/build/dashboards/dashboards.md b/docs/docs/build/dashboards/dashboards.md index 68b3fc6e186..298697ad0e3 100644 --- a/docs/docs/build/dashboards/dashboards.md +++ b/docs/docs/build/dashboards/dashboards.md @@ -5,78 +5,41 @@ sidebar_label: Create Dashboards sidebar_position: 00 --- -In Rill, your dashboards are defined by _metric definitions_. Metric definitions are composed of: -* _**A model**_ - A data model creating a One Big Table that will power the dashboard. -* _**A timeseries**_ - A column from your model that will underlie x-axis data in the line charts. Time will be truncated into different time periods. -* _**Measures**_ - Numerical aggregates of columns from your data model shown on the y-axis of the line charts and the "big number" summaries. -* _**Dimensions**_ - Categorical columns from your data model whose values are shown in _leaderboards_ and allows you to look at segments or attributes of your data (and filter / slice accordingly). +In Rill, dashboards are one of many components that access the metrics layer. Currently, it is possible to create an explore dashboard but more features are on the way! -:::note Dashboard Properties -For more details about available configurations and properties, check our [Dashboard YAML](/reference/project-files/dashboards) reference page. -::: - -## Creating valid metrics - -### Timeseries - -Your timeseries must be a column from your data model of [type](https://duckdb.org/docs/sql/data_types/timestamp) `TIMESTAMP`, `TIME`, or `DATE`. If your source has a date in a different format, you can apply [time functions](https://duckdb.org/docs/sql/functions/timestamp) to transform these fields into valid timeseries types. - -### Measures - -Measures are numeric aggregates of columns from your data model. A measure must be defined with [DuckDB SQL](https://duckdb.org/docs/sql/introduction.html) aggregation functions and expressions on columns from your data model. The following operators and functions are allowed in measure expressions: - -* Any DuckDB SQL [numeric](https://duckdb.org/docs/sql/functions/numeric) operators and functions -* This set of DuckDB SQL [aggregates](https://duckdb.org/docs/sql/aggregates): `AVG`, `COUNT`, `FAVG`,`FIRST`, `FSUM`, `LAST`, `MAX`, `MIN`, `PRODUCT`, `SUM`, `APPROX_COUNT_DISTINCT`, `APPROX_QUANTILE`, `STDDEV_POP`, `STDDEV_SAMP`, `VAR_POP`, `VAR_SAMP`. -* [Filtered aggregates](https://duckdb.org/docs/sql/query_syntax/filter.html) can be used to filter set of rows fed to the aggregate functions - -As an example, if you have a table of sales events with the sales price and customer id, you could calculate the following metrics with these aggregates and expressions: -* Number of sales: `COUNT(*)` -* Total revenue: `SUM(sales_price)` -* Revenue per customer: `CAST(SUM(sales_price) AS FLOAT)/CAST(COUNT(DISTINCT customer_id) AS FLOAT)` -* Number of orders with order value more than $100 : `count(*) FILTER (WHERE order_val > 100)` - -You can also add labels, descriptions, and your choice of number formatting to customize how they are shown in the dashboard. +![img](/img/build/dashboard/explore-dashboard.png) +* _**metrics_view**_ - A metrics view that powers the dashboard +* _**measures**_ - `*` Which measures to include or exclude from the metrics view, using a wildcard will include all. +* _**dimensions**_ - `*` Which dimensions to include or exclude from the metrics view, using a wildcard will include all. -### Dimensions +When including dimensions and measures only the named resources will be included. +Rill also supports the ability to exclude a set of named dimensions and measures. -Dimensions are used for exploring segments and filtering the dashboard. Valid dimensions can be any type and are selected using the drop down menu. You can also add labels and descriptions to your dimensions to customize how they are shown in the dashboard. +```yaml +metrics_view: my_metrics_view +dimensions: [country, region, product_category] # Only these three dimensions will be included +measures: + exclude: [profit] # All measures except profit will be included +``` -## Updating dashboards +:::tip +Starting in version 0.50, metrics view has been separated from dashboard. This allows for a cleaner, more accessible metrics layer and the ability to build various dashboards and components on top of a single metrics layer. For more information on why we decided to do this, please refer to the following: [Why separate the dashboard and metrics layer](/concepts/metrics-layer) -### Using the UI / code -When you add a metrics definition using the UI, a code definition will automatically be created as a YAML file in your Rill project within the dashboards directory by default. However, you can also create metrics definitions more directly by creating the artifact. +For migration steps, see [Migrations](/manage/migration#v049---v050). +::: -In your Rill project directory, after the `dashboard_name.yaml` file is created in the `dashboards` directory, its configuration or definition can be updated as needed by updating the YAML file directly, using the following template as an example: -```yaml -type: metrics_view -model: model_name -title: Dashboard name -default_time_range: "" -smallest_time_grain: "" -timeseries: timestamp_column_name - -dimensions: - - column: model_column_1 - label: Column label 1 - description: "" - # Add more dimensions here +:::note Dashboard Properties +For more details about available configurations and properties, check our [Dashboard YAML](/reference/project-files/explore-dashboards) reference page. +::: -measures: - - label: "Count" - name: "count" - expression: count(*) - description: "" - format_preset: "humanize" - # Add more measures here -``` -:::info Check our reference documentation +### Preview a Dashboard in Rill Developer +Once a dashboard is ready to preview, before [deploying to Rill Cloud](/deploy/deploy-dashboard/), you can preview the dashboard in Rill Developer. Especially if you are setting up [dashboard policies](/manage/security), it is recommended to preview and test the dashboard before deploying. -For more information about available dashboard properties, feel free to check our [reference YAML documentation](/reference/project-files/dashboards). +![preview](/img/build/dashboard/preview-dashboard.png) -::: ### Multi-editor and external IDE support diff --git a/docs/docs/build/metrics-view/_category_.yml b/docs/docs/build/metrics-view/_category_.yml new file mode 100644 index 00000000000..c9fae99b73f --- /dev/null +++ b/docs/docs/build/metrics-view/_category_.yml @@ -0,0 +1,4 @@ +position: 38 +label: Create Metrics Views +collapsible: true +collapsed: true diff --git a/docs/docs/build/metrics-view/customize.md b/docs/docs/build/metrics-view/customize.md new file mode 100644 index 00000000000..5a749f6077c --- /dev/null +++ b/docs/docs/build/metrics-view/customize.md @@ -0,0 +1,77 @@ +--- +title: "Customization" +description: Alter metrics view features +sidebar_label: "Customization" +sidebar_position: 30 +--- + +## Common Customizations + +You will find below some common customizations and metrics-view configurations that are available for end users. + +:::info Metric View properties + +For a full list of available dashboard properties and configurations, please see our [Metrics View YAML](/reference/project-files/metrics-view.md) reference page. +::: + + +**`smallest_time_grain`** + +Smallest time grain available for your users. Rill will try to infer the smallest time grain. One of the most common reasons to change this setting is your data has timestamps but is actually in hourly or daily increments. The valid values are: `millisecond`, `second`, `minute`, `hour`, `day`, `week`, `month`, `quarter`, `year`. + +**`first_day_of_week`** + +The start of the week defined as an integer. The valid values are 1 through 7 where Monday=`1` and Sunday=`7` _(optional) + +**`first_month_of_year`** + + +The first month of the year for time grain aggregation. The valid values are 1 through 12 where January=`1` and December=`12` _(optional)_. + + +**`security`** + +Defining security policies for your data is crucial for security. For more information on this, please refer to our [Dashboard Access Policies](/manage/security.md) + +## Example + +```yaml +version: 1 #defines version +type: metrics_view # metrics_view + +title: The title of your metrics_view +display_name: The display_name +description: A description +model: refernce the model or table, +database / database_schema: #not sure what this is used for. + +timeseries: your timeseries column + +smallest_time_grain: #defines the smallest time grain + +first_day_of_week: #defines first day of week +first_month_of_year: #defines first month of year + +dimensions: #your dimensions, can be copied from dashboard.yaml + - name: + label: + column/expression: + property: + description + unnest: + uri: + +measures: #your measures, can be copied from dashboard.yaml + - name: + label: + type: + expressions: + window: + per: + requires: + description: + format_preset / format_d3: + valid_percent_of_total: + +security: #your security policies can be copied from dashboard.yaml +``` \ No newline at end of file diff --git a/docs/docs/build/dashboards/expressions.md b/docs/docs/build/metrics-view/expressions.md similarity index 83% rename from docs/docs/build/dashboards/expressions.md rename to docs/docs/build/metrics-view/expressions.md index 88884433523..8c0f72027d2 100644 --- a/docs/docs/build/dashboards/expressions.md +++ b/docs/docs/build/metrics-view/expressions.md @@ -7,7 +7,7 @@ sidebar_position: 20 ## Overview -Within the dashboard yaml, you can apply aggregate sql expressions to create derived metrics or non-aggregate expressions to adjust dimension settings. SQL expressions are specific to the underlying OLAP engine so keep that in mind when editing directly in the yaml. +Within the metrcs view yaml, you can apply aggregate sql expressions to create derived metrics or non-aggregate expressions to adjust dimension settings. SQL expressions are specific to the underlying OLAP engine so keep that in mind when editing directly in the yaml. We continually get questions about common metric definitions and other tricks so will update this page frequently. [Please let us know](../../contact.md) if you have questions or are stuck on an expression so we can add more examples. @@ -19,11 +19,11 @@ Rill's modeling layer provides open-ended SQL compatibility for complex SQL quer ## Measure Expressions -Measure expressions can take any sql numeric function, a set of aggregates and apply filters to create derived metrics. Reminder on basic expressions are available in the [create dashboard definition](dashboards.md). +Measure expressions can take any SQL numeric function, a set of aggregates and apply filters to create derived metrics. Reminder on basic expressions are available in the [create metrics-view definition](metrics-view.md#measures). ### Metric Formatting -In addition to standard presents, you can also use `format_d3` to control the formatting of a measure in the dashboard using a [d3-format string](https://d3js.org/d3-format). If an invalid format string is supplied, measures will be formatted with `format_preset: humanize`. Measures cannot have both `format_preset` and `format_d3` entries. _(optional; if neither `format_preset` nor `format_d3` is supplied, measures will be formatted with the `humanize` preset)_ +In addition to standard presents, you can also use `format_d3` to control the formatting of a measure in the metrics view using a [d3-format string](https://d3js.org/d3-format). If an invalid format string is supplied, measures will be formatted with `format_preset: humanize`. Measures cannot have both `format_preset` and `format_d3` entries. _(optional; if neither `format_preset` nor `format_d3` is supplied, measures will be formatted with the `humanize` preset)_ - **Example**: to show a measure using fixed point formatting with 2 digits after the decimal point, your measure specification would include: `format_d3: ".2f"`. - **Example**: to show a measure using grouped thousands with two significant digits, your measure specification would include: `format_d3: ",.2r"`. diff --git a/docs/docs/build/metrics-view/metrics-view.md b/docs/docs/build/metrics-view/metrics-view.md new file mode 100644 index 00000000000..8ce2c7a19a9 --- /dev/null +++ b/docs/docs/build/metrics-view/metrics-view.md @@ -0,0 +1,99 @@ +--- +title: Create Metrics Views +description: Create metrics-view using source data and models with time, dimensions, and measures +sidebar_label: Create Metrics Views +sidebar_position: 00 +--- + + +
+ +In Rill, your metrics view is defined by _metric definitions_. Metric definitions are composed of: +* _**model**_ - A data model creating a One Big Table that will power the metrics view. +* _**timeseries**_ - A column from your model that will underlie x-axis data in the line charts. Time will be truncated into different time periods. +* _**measures**_ - Numerical aggregates of columns from your data model shown on the y-axis of the line charts and the "big number" summaries. +* _**dimensions**_ - Categorical columns from your data model whose values are shown in _leaderboards_ and allows you to look at segments or attributes of your data (and filter / slice accordingly). + + +:::tip +Starting in version 0.50, metrics view has been separated from dashboard. This allows for a cleaner, more accessible metrics layer and the ability to build various dashboards and components on top of a single metrics layer. For more information on why we decided to do this, please refer to the following: [Why separate the dashboard and metrics layer](/concepts/metrics-layer) + +For migration steps, see [Migrations](/manage/migration#v049---v050). +::: + +## Creating valid metrics + +### Timeseries + +Your timeseries must be a column from your data model of [type](https://duckdb.org/docs/sql/data_types/timestamp) `TIMESTAMP`, `TIME`, or `DATE`. If your source has a date in a different format, you can apply [time functions](https://duckdb.org/docs/sql/functions/timestamp) to transform these fields into valid timeseries types. + +### Measures + +Measures are numeric aggregates of columns from your data model. A measure must be defined with [DuckDB SQL](https://duckdb.org/docs/sql/introduction.html) aggregation functions and expressions on columns from your data model. The following operators and functions are allowed in measure expressions: + +* Any DuckDB SQL [numeric](https://duckdb.org/docs/sql/functions/numeric) operators and functions +* This set of DuckDB SQL [aggregates](https://duckdb.org/docs/sql/aggregates): `AVG`, `COUNT`, `FAVG`,`FIRST`, `FSUM`, `LAST`, `MAX`, `MIN`, `PRODUCT`, `SUM`, `APPROX_COUNT_DISTINCT`, `APPROX_QUANTILE`, `STDDEV_POP`, `STDDEV_SAMP`, `VAR_POP`, `VAR_SAMP`. +* [Filtered aggregates](https://duckdb.org/docs/sql/query_syntax/filter.html) can be used to filter set of rows fed to the aggregate functions + +As an example, if you have a table of sales events with the sales price and customer id, you could calculate the following metrics with these aggregates and expressions: +* Number of sales: `COUNT(*)` +* Total revenue: `SUM(sales_price)` +* Revenue per customer: `CAST(SUM(sales_price) AS FLOAT)/CAST(COUNT(DISTINCT customer_id) AS FLOAT)` +* Number of orders with order value more than $100 : `count(*) FILTER (WHERE order_val > 100)` + +You can also add labels, descriptions, and your choice of number formatting to customize how they are shown. + + +### Dimensions + +Dimensions are used for exploring segments and filtering. Valid dimensions can be any type and are selected using the drop down menu. You can also add labels and descriptions to your dimensions to customize how they are shown. + + +## Updating the Metrics View + +Whether you prefer the UI or YAML artifacts, Rill supports both methods for updating your metrics view. + +### Using the Visual Metrics Editor + +![visual-metric-editor](/img/build/metrics-view/visual-metrics-editor.png) + +When you add a metrics definition using the UI, a code definition will automatically be created as a YAML file in your Rill project within the metrics directory by default. + +### Using YAML +You can also create metrics definitions more directly by creating the artifact. + +In your Rill project directory, after the `metrics-view.yaml` file is created in the `metrics` directory, its configuration or definition can be updated as needed by updating the YAML file directly, using the following template as an example: + +```yaml +# Metrics View YAML +# Reference documentation: https://docs.rilldata.com/reference/project-files/metrics_views + +version: 1 +type: metrics_view + +table: example_table # Choose a table to underpin your metrics +timeseries: timestamp_column # Choose a timestamp column (if any) from your table + +dimensions: + - column: category + label: "Category" + description: "Description of the dimension" + +measures: + - expression: "SUM(revenue)" + label: "Total Revenue" + description: "Total revenue generated" + +``` +:::info Check our reference documentation + +For more information about available metrics view properties, feel free to check our [reference YAML documentation](/reference/project-files/metrics-view. + +::: + + +### Multi-editor and external IDE support + +Rill Developer is meant to be developer friendly and has been built around the idea of keystroke-by-keystroke feedback when modeling your data, allowing live interactivity and a real-time feedback loop to iterate quickly (or make adjustments as necessary) with your models and dashboards. Additionally, Rill Developer has support for the concept of "hot reloading", which means that you can keep two windows of Rill open at the same time and/or use a preferred editor of choice, such as VSCode, side-by-side with the dashboard that you're actively developing! + +![hot-reload-0-36](https://cdn.rilldata.com/docs/release-notes/36_hot_reload.gif) \ No newline at end of file diff --git a/docs/docs/build/models/environments.md b/docs/docs/build/models/environments.md index 2a8cb32471a..5c1588db81d 100644 --- a/docs/docs/build/models/environments.md +++ b/docs/docs/build/models/environments.md @@ -83,7 +83,7 @@ env: :::info Hierarchy of inheritance and property overrides As a general rule of thumb, properties that have been specified at a more _granular_ level will supercede or override higher level properties that have been inherited. Therefore, in order of inheritance, Rill will prioritize properties in the following order: -1. Individual [source](/reference/project-files/sources.md)/[model](/reference/project-files/models.md)/[dashboard](/reference/project-files/dashboards.md) object level properties (e.g. `source.yaml` or `dashboard.yaml`) +1. Individual [source](/reference/project-files/sources.md)/[model](/reference/project-files/models.md)/[dashboard](/reference/project-files/explore-dashboards.md) object level properties (e.g. `source.yaml` or `dashboard.yaml`) 2. [Environment](/docs/build/models/environments.md) level properties (e.g. a specific property that have been set for `dev`) 3. [Project-wide defaults](/reference/project-files/rill-yaml.md#project-wide-defaults) for a specific property and resource type diff --git a/docs/docs/build/models/models.md b/docs/docs/build/models/models.md index d069767254c..2999467bbff 100644 --- a/docs/docs/build/models/models.md +++ b/docs/docs/build/models/models.md @@ -11,7 +11,7 @@ sidebar_position: 00 Data transformations in Rill Developer are powered by DuckDB and their dialect of SQL (DuckDB SQL). Please visit [DuckDB SQL documentation](https://duckdb.org/docs/sql/introduction) to learn about how to write your queries. -For additional tips on commonly used expressions (either in models or dashboard definitions) visit our [common expressions page](../dashboards/expressions.md). +For additional tips on commonly used expressions (either in models or dashboard definitions) visit our [common expressions page](../metrics-view/expressions.md). ## Adding a data model diff --git a/docs/docs/build/olap/olap.md b/docs/docs/build/olap/olap.md index 099b4a51bf5..16f68e75ea6 100644 --- a/docs/docs/build/olap/olap.md +++ b/docs/docs/build/olap/olap.md @@ -10,7 +10,7 @@ sidebar_position: 00 There are two ways to define an OLAP engine within Rill. 1. Set the [default OLAP engine](../../reference/project-files/rill-yaml#configuring-the-default-olap-engine) via the rill.yaml file. -2. Set the [OLAP engine](../../reference/project-files/dashboards.md) for a specific dashboard. +2. Set the [OLAP engine](../../reference/project-files/explore-dashboards.md) for a specific dashboard. The OLAP engine set on the dashboard will take precedence over the project-level defined OLAP engine. diff --git a/docs/docs/build/structure/structure.md b/docs/docs/build/structure/structure.md index b67d7bd6100..3706fbd56a4 100644 --- a/docs/docs/build/structure/structure.md +++ b/docs/docs/build/structure/structure.md @@ -7,9 +7,10 @@ sidebar_position: 00 After creating your initial set of sources, models, and dashboards, you may have noticed the following _native_ folders that exist in your Rill project directory: - [Sources](/reference/project-files/sources) - [Models](/reference/project-files/models) -- [Dashboards](/reference/project-files/dashboards) +- [Metrics View](/reference/project-files/metrics-view) +- [Dashboards](/reference/project-files/explore-dashboards) -By default, any new sources, models, and dashboards will be created in their respective native folder. However, this does not necessarily have to be the case and Rill Developer allows a flexible project directory structure, including nested folders or even storing objects in non-native folders. This is a powerful feature that allows you as a developer to organize your project to meet your specific team needs. +By default, any new sources, models, metrics view and dashboards will be created in their respective native folder. However, this does not necessarily have to be the case and Rill Developer allows a flexible project directory structure, including nested folders or even storing objects in non-native folders. This is a powerful feature that allows you as a developer to organize your project to meet your specific team needs. ## Adding new resources or parent folders diff --git a/docs/docs/concepts/metrics-layer.md b/docs/docs/concepts/metrics-layer.md new file mode 100644 index 00000000000..97094db42b5 --- /dev/null +++ b/docs/docs/concepts/metrics-layer.md @@ -0,0 +1,46 @@ +--- +title: "What is a Metrics View?" +sidebar_label: "What is a Metrics View?" +sidebar_position: 11 +hide_table_of_contents: true +--- + +What is a metrics layer, and why did we decide to split the dashboard into two individual components? + + +## What is a Metrics Layer? + +A metrics layer is a `centralized framework` used to define and organize **key metrics** for your organization. Having a centralized layer allows an organization to easily manage and re-use calculations across various reports, dashboard, and data tools. As Rill continues to grow, we decided to separate metrics layer from the dashboard configuration. + + +### Historically, in Rill... + +
+ +Historically in Rill, the metrics layer and dashboard configuration were a single file. As seen above, the metrics would be defined **inside** a dashboard YAML file along with the dashboard components and dashboard customizations. However, as we continue to create more features, we found that this was not the best approach. In order to create a metrics layer in Rill as a first class resource and not a consequence of dashboards, we found it necessary to split the two resources into their own files. Thus, the metrics-view was born. + +:::tip +Starting from version 0.50, the operation of creating a dashboard via AI will create a metrics-view and dashboard separately in their own respective folders and navigate you to a preview of your dashboard. If you find that some of the metrics need to be modified, you will need to navigate to your [metrics/model_name_metrics.yaml](/build/metrics-view/) file. + + +Assuming that you have the ' * ', select all, in your dashboard configurations, any changes will automatically be changed on your [explore dashboard](/build/dashboards/). +::: + +## Splitting the Dashboard into two components, Metrics-view and Dashboard Configuration +Splitting the metrics view into its own component allows us more freedom to continue building Rill and adding new additional features. Instead of querying a dashboard for data, we would be querying the metrics-layer. The dashboard will directly query the metrics-view along with many new components that are currently being developed. + +### New Metrics View as an independent object in Rill + +![img](/img/concepts/metrics-view/metrics-view-components.png) + +### (Explore) Dashboard + +With the split of metrics view, dashboard configurations experienced an overhaul. Instead of defining measure and dimensions, you will now reference the object into your dashboard. What this allows is creating customized dashboards for specific viewers and reusability of a single metrics-view in multiple dashboards! + +![img](/img/concepts/metrics-view/explore-dashboard.png) + +For more information on what is changed, please review the [migrations](/manage/migration) and the [reference page](/reference/project-files/explore-dashboards). + +## More to come! + +New features are being developed as we speak! If you're curious, feel free to [contact us](/contact)! \ No newline at end of file diff --git a/docs/docs/concepts/operational.md b/docs/docs/concepts/operational.md index f0fc24eb39d..9f1319e37b3 100644 --- a/docs/docs/concepts/operational.md +++ b/docs/docs/concepts/operational.md @@ -1,7 +1,7 @@ --- title: Operational BI vs. Traditional BI sidebar_label: What is Operational BI -sidebar_position: 11 +sidebar_position: 10 hide_table_of_contents: true --- diff --git a/docs/docs/contact.md b/docs/docs/contact.md index bde936e7ddb..c1dc9045934 100644 --- a/docs/docs/contact.md +++ b/docs/docs/contact.md @@ -11,7 +11,7 @@ Rill is part of the open source community and you can find the source code of ou ## Discord -Curious to see if someone else has encountered an issue before, looking for help getting started, interested in tapping into the collective knowledge of others, or simply want to engage other members of the Rill community? Rill's community can be found on Discord and we'd love for you to be a part of it! Please click [here](https://discord.gg/DJ5qcsxE2m) to get access to our dedicated Discord channel. +Curious to see if someone else has encountered an issue before, looking for help getting started, interested in tapping into the collective knowledge of others, or simply want to engage other members of the Rill community? Rill's community can be found on Discord and we'd love for you to be a part of it! Please click [here](https://bit.ly/3bbcSl9) to get access to our dedicated Discord channel. ## Contacting Support If you'd like to get in contact with the Rill team directly and/or contact Rill Support, there are a few different options available. diff --git a/docs/docs/deploy/deploy-dashboard/deploy-dashboard.md b/docs/docs/deploy/deploy-dashboard/deploy-dashboard.md index fb83abafb8f..af099f99f7b 100644 --- a/docs/docs/deploy/deploy-dashboard/deploy-dashboard.md +++ b/docs/docs/deploy/deploy-dashboard/deploy-dashboard.md @@ -17,7 +17,7 @@ Cloud datastores will typically require service keys to access data. Make sure t The flow diagram below shows two options for deploying an existing project. -**Deploy via the UI or CLI using `rill project upload`**: +**Deploy via the UI or CLI using `rill project deploy`**: ```mermaid graph LR; A(Local code files); @@ -40,7 +40,7 @@ graph LR; ``` ## Deploying a project via the UI -Starting from **v0.48**, we have introduced the possibility to push dashboards _directly from Rill Developer's UI to Rill Cloud_. On the dashboard page, you can select the `deploy to share` button and follow the steps to deploy to Rill Cloud. +Starting from **v0.48**, we have introduced the possibility to push dashboards _directly from Rill Developer's UI to Rill Cloud_. On the dashboard page, you can select the `Deploy` button and follow the steps to deploy to Rill Cloud.
@@ -60,7 +60,7 @@ You can easily check the version of Rill that you are using in Rill Developer by rill --version ``` -If you are on an older version of Rill, it is **strongly recommended** to [upgrade](home/install.md#upgrade) to the latest version. +If you are on an older version of Rill, it is **strongly recommended** to [upgrade](home/install.md#upgrade-to-the-newest-version) to the latest version. ::: diff --git a/docs/docs/explore/dashboard-101.md b/docs/docs/explore/dashboard-101.md index 81dd2923c64..bbce26d7f15 100644 --- a/docs/docs/explore/dashboard-101.md +++ b/docs/docs/explore/dashboard-101.md @@ -12,7 +12,7 @@ sidebar_position: 00 Depending on whether you are opening Rill Developer or logging into Rill Cloud, you will either the default "Getting started" landing page or a list of all projects available to your user. For the purposes of this article, we will assume that your project has already been [deployed to Rill Cloud](../deploy/deploy-dashboard/) and that you are looking to consume your dashboards in a production capacity. -After logging into [Rill Cloud](https://ui.rilldata.com), you should see all projects within your [organization](../manage/project-management#organization) that is available and/or has been granted permissions to your user profile. Within each project, you'll then be able to access the corresponding individual dashboards that belong to a particular Rill project. +After logging into [Rill Cloud](https://ui.rilldata.com), you should see all projects within your [organization](/manage/project-management#organization) that is available and/or has been granted permissions to your user profile. Within each project, you'll then be able to access the corresponding individual dashboards that belong to a particular Rill project. ![Rill Cloud landing page](../../static/img/explore/dashboard101/rill-cloud-landing-page.png) @@ -36,12 +36,12 @@ The main screen of any Rill dashboard is called the _Explore_ page. As seen abov - _**Explore or Pivot:**_ You can switch the view from _explore_ to [_pivot_](https://docs.rilldata.com/explore/filters/pivot) by selecting either from the UI (see `pink` box) -- _**Alerts, Bookmarks and Sharing:**_ You can create an [alert](./alerts/alerts.md) by selecting the bell, customizing the default view of the dashboard (see `purple` box) to a predefined set of metrics, dimensions, and filters by selecting the [bookmark](bookmarks.md), or share the dashboard ([internally](../manage/user-management#step-1---admin-invites-user) or [externally](./public-url.md)) by clicking the `Share` button. +- _**Alerts, Bookmarks and Sharing:**_ You can create an [alert](./alerts/alerts.md) by selecting the bell, customizing the default view of the dashboard (see `purple` box) to a predefined set of metrics, dimensions, and filters by selecting the [bookmark](bookmarks.md), or share the dashboard ([internally](/manage/user-management#option-1---admin-invites-user) or [externally](./public-url.md)) by clicking the `Share` button. ### Metrics Panel -- _**Measures:**_ All _**metrics**_ that are available in the underlying model \ are viewable on the left-hand side, broken out with summary numbers (e.g. eCPM) and timeseries visualizations (based on your configured `timeseries` column in your [dashboard YAML](/reference/project-files/dashboards.md)). You can add or remove any metric from the page by simply selecting them from the dropdown above the charts (see `yellow` box). If you select any specific measure, you will be navigating to the [Time Dimension Detail](https://docs.rilldata.com/explore/filters/tdd). +- _**Measures:**_ All _**metrics**_ that are available in the underlying model \ are viewable on the left-hand side, broken out with summary numbers (e.g. eCPM) and timeseries visualizations (based on your configured `timeseries` column in your [dashboard YAML](/reference/project-files/explore-dashboards.md)). You can add or remove any metric from the page by simply selecting them from the dropdown above the charts (see `yellow` box). If you select any specific measure, you will be navigating to the [Time Dimension Detail](https://docs.rilldata.com/explore/filters/tdd). ### Dimensions Leaderboard Panel @@ -58,7 +58,7 @@ After drilling into a leaderboard (or what we sometimes refer to as a _toplist_ :::tip Don't have a Rill project or dashboard deployed yet? -If you want to get hands on and see what interacting with a Rill dashboard feels like, we have a set of [demo projects](https://ui.rilldata.com/demo) already deployed on Rill Cloud and publicly available for everyone to try out. These [same projects](../home/get-started#example-projects) are also available on Github and can be deployed locally using Rill Developer. +If you want to get hands on and see what interacting with a Rill dashboard feels like, we have a set of [demo projects](https://ui.rilldata.com/demo) already deployed on Rill Cloud and publicly available for everyone to try out. These [same projects](/home/get-started#example-projects-repository) are also available on Github and can be deployed locally using Rill Developer. ::: diff --git a/docs/docs/explore/filters/filters.md b/docs/docs/explore/filters/filters.md index 424f2a12606..5ac4c8455db 100644 --- a/docs/docs/explore/filters/filters.md +++ b/docs/docs/explore/filters/filters.md @@ -20,7 +20,7 @@ You can also change settings in the dashboard configuration to hide certain fiel Or, an administrator can set the default view of a dashboard by [bookmarking the view](../bookmarks.md) as Home. ::: -## Dimension & Measure Filters +## Dimension & Measure & Time Filters ### Filter by Dimensions @@ -55,6 +55,16 @@ Metric filters are a good way to "sort" by two different metrics. First, apply y As an example - to see most active enterprise customers - filter all customers with revenue greater than $1000 then sorted by number of users increased descending. ::: +### Filter by Scrubbing + +For a specific view into your time series graph, you can interactively scrub directly on the time series graph. + + + +
+ +This allows the ability for a more detailed view into your time series without having to change the overall time series filter for quick access to measures. + ## Time & Dimension Comparisons ![img](/img/explore/filters/time-series.png) @@ -66,7 +76,7 @@ Time comparison is a key feature of Rill and recommended to find insights quicke
:::note -Default and comparison periods can be configured in your dashboard file. Visit [Dashboard Reference](/reference/project-files/dashboards) for more details +Default and comparison periods can be configured in your dashboard file. Visit [Dashboard Reference](/reference/project-files/explore-dashboards) for more details ::: @@ -87,4 +97,5 @@ De-select the comparison option or clear the filter bar to remove your compariso
:::note For more advanced time and dimension comparisons, visit the [Time Dimension Detail](tdd.md) page. -::: \ No newline at end of file +::: + diff --git a/docs/docs/explore/public-url.md b/docs/docs/explore/public-url.md index 8764b284d29..70f5af9703b 100644 --- a/docs/docs/explore/public-url.md +++ b/docs/docs/explore/public-url.md @@ -33,12 +33,16 @@ In fact, users who click on a public shareable URL cannot see the parent filters ::: +## How to manage public URLs -### How to manage public URLs via the CLI -:::tip +### via the UI +You can now manage public URLs via the UI. You will find a new "settings" tab in the Rill Cloud UI as an administrator. -Starting from v.0.48, `public-url` has been rebranded to `public-url`. +![img](/img/explore/publicurl/public-url-settings.png) +### via the CLI +:::tip +Starting from v.0.48, `public-url` has been rebranded to `public-url`. ::: ``` rill public-url @@ -83,14 +87,4 @@ rill public-url delete If you are interested in creating a public URL directly from the CLI, you can do so by passing the required parameters. (You can use the --help flag to see what additional flags are required.) -## Manage public URLs in Rill Cloud - -import ComingSoon from '@site/src/components/ComingSoon'; - - - - -
-Historically (pre 0.48), user management was only possible via the CLI. Now, it is also possible to do so via the UI! -
diff --git a/docs/docs/manage/migration.md b/docs/docs/manage/migration.md new file mode 100644 index 00000000000..b1efeeb0850 --- /dev/null +++ b/docs/docs/manage/migration.md @@ -0,0 +1,121 @@ +--- +title: "Migrations" +description: For documenting required migrations +sidebar_label: "Migrations" +sidebar_position: 60 +--- + +Please refer to the changes below and recommended migration steps. + +## Migration + +### v0.49 -> v0.50 + +Due to the [separation of dashboards to metrics layer and dashboards](/concepts/metrics-layer), you will need to review your current dashboards and make the following changes (note: Legacy dashboards will continue to function.): + +**[Sample Legacy Dashboard Contents](https://docs.rilldata.com/reference/project-files/explore-dashboards):** + +```yaml +title: #needs to be defined on metrics-view and dashboard +model: #defined on metrics-view +type: #defined on both, explore or metrics-view +timeseries: #defined on metrics-view + +smallest_time_grain: #defined in metrics-view, + +default_...: #defined in dashboard + dimensions: + measures: + comparison: + ... + + +measures: #defined in metrics-view, + ... + +dimensions: #defined in metrics-view, + ... + +security: #defined on both metrics-view and dashboard but different capabilities + ... + +theme: #defined in dashboard + +available_time_zones: #defined in dashboard as time_zones: +available_time_ranges: #defined in dashboard as time_ranges: + +first_day_of_week: #defined in metrics-view, +first_month_of_year: #defined in metrics-view, + +``` + +**[Metrics_View YAML](/reference/project-files/metrics-view):** +```yaml +version: 1 #defines version +type: metrics_view # metrics_view + +title: The title of your metrics_view +display_name: The display_name +description: A description +model / table: refernce the model or table, +database / database_schema: #if using a different OLAP engine, refers to database and schema (usually not required) + +timeseries: your timeseries column + +smallest_time_grain: #defines the smallest time grain + +first_day_of_week: #defines first day of week +first_month_of_year: #defines first month of year + +dimensions: #your dimensions, can be copied from dashboard.yaml + - name: + label: + column/expression: + property: + description: + unnest: + uri: + +measures: #your measures, can be copied from dashboard.yaml + - name: + label: + type: + expressions: + window: + per: + requires: + description: + format_preset / format_d3: + valid_percent_of_total: + +security: #your security policies can be copied from dashboard.yaml +``` + +**[Explore dashboard YAML](/reference/project-files/explore-dashboards):** +```yaml +type: explore + +title: Title of your explore dashboard +description: a description +metrics_view: + +dimensions: '*' #can use regex +measures: '*' #can use regex + +theme: #your default theme + +time_ranges: #was available_time_ranges +time_zones: #was available_time_zones + +defaults: #define all the defaults within here + dimensions: + measures: + time_range: + comparison_mode: + comparison_dimension: + +security: + access: #only access can be set on dashboard level, see metric view for detailed access policy +``` + +For any questions, please [contact the team](https://docs.rilldata.com/contact) via Slack, email, Discord or the in-app chat! \ No newline at end of file diff --git a/docs/docs/manage/project-management.md b/docs/docs/manage/project-management.md index 6e72e091832..00aef7e1767 100644 --- a/docs/docs/manage/project-management.md +++ b/docs/docs/manage/project-management.md @@ -11,23 +11,30 @@ Once a project is ready to be deployed onto Rill Cloud, as an admin, you will ne An Organization in Rill is the parent management object and encompasses how your team or organization interfaces with Rill Cloud. Organizations are designed to hold the differnet components of your Rill project. Projects exist within an organization, which itself contains sources, models, dashboards, and other resources that belong to your standalone Rill projects. -If you'd like to create, edit, modify, or delete an organization, you can do this either via the CLI or UI. +If you'd like to create, edit, modify, or delete an organization, run the following command. -### CLI ``` -rill org +rill org +Manage organisations + +Usage: + rill org [command] + +Available Commands: + create Create organization + edit Edit organization details + switch Switch to other organization + list List all organizations + delete Delete organization + rename Rename organization + +Global Flags: + --api-token string Token for authenticating with the cloud API + --format string Output format (options: "human", "json", "csv") (default "human") + -h, --help Print usage + --interactive Prompt for missing required parameters (default true) ``` - -### UI -import ComingSoon from '@site/src/components/ComingSoon'; - - - -
-a -
- :::tip [Access to Rill can be granted on the organization level](user-management.md#adding-a-member-to-the-organization). diff --git a/docs/docs/manage/security.md b/docs/docs/manage/security.md index 25da0e6960a..f85dca49432 100644 --- a/docs/docs/manage/security.md +++ b/docs/docs/manage/security.md @@ -28,7 +28,7 @@ You define access policies for dashboards under the `security` key in a dashboar ![access](../../static/img/manage/security/access.png) -See the [Dashboard YAML](/reference/project-files/dashboards) reference docs for all the available fields. +See the [Dashboard YAML](/reference/project-files/explore-dashboards) reference docs for all the available fields. See the [Examples](#examples) below for how to set up each type of configuration. @@ -125,7 +125,7 @@ security: - id ``` -Alternatively, you can explicitly define the dimensions and measures to include using the `include` key. It uses the same syntax as `exclude` and automatically excludes all names not explicitly defined in the list. See the [Dashboard YAML](/reference/project-files/dashboards) reference for details. +Alternatively, you can explicitly define the dimensions and measures to include using the `include` key. It uses the same syntax as `exclude` and automatically excludes all names not explicitly defined in the list. See the [Dashboard YAML](/reference/project-files/explore-dashboards) reference for details. ### Use wildcards to select all dimensions and measures diff --git a/docs/docs/manage/settings.md b/docs/docs/manage/settings.md new file mode 100644 index 00000000000..f30722cefba --- /dev/null +++ b/docs/docs/manage/settings.md @@ -0,0 +1,29 @@ +--- +title: "Settings" +description: For other Settings related to Rill Cloud +sidebar_label: "Settings" +sidebar_position: 55 +--- + +Settings in Rill Cloud give an easy access to manage objects and ensure that everything is working as expected. + + + +:::tip +Starting from version 0.50, we are beginning to port the ability to manage objects in Rill Cloud within the Rill Cloud UI. This includes, public URLs (currently released), user and group management, and more! +::: + +## Public URL + +Within the settings page, you can copy the URL of an existing public URL, delete existing public URLs, and check owner and last access date. + +![img](/img/manage/settings/rill-cloud-settings.png) + +import ComingSoon from '@site/src/components/ComingSoon'; + +## User and User Groups + + + +
+
\ No newline at end of file diff --git a/docs/docs/manage/user-management.md b/docs/docs/manage/user-management.md index 10dc3e5cb9e..8f855c3e297 100644 --- a/docs/docs/manage/user-management.md +++ b/docs/docs/manage/user-management.md @@ -1,7 +1,7 @@ --- title: User Management sidebar_label: Users -sidebar_position: 20 +sidebar_position: 22 --- In Rill Cloud, access can be granted at the organization, project, or group level using the Rill CLI. diff --git a/docs/docs/manage/usergroup-management.md b/docs/docs/manage/usergroup-management.md index 331898c1ff7..f86d563de1a 100644 --- a/docs/docs/manage/usergroup-management.md +++ b/docs/docs/manage/usergroup-management.md @@ -1,7 +1,7 @@ --- title: User Group Management sidebar_label: User Groups -sidebar_position: 22 +sidebar_position: 20 --- import ComingSoon from '@site/src/components/ComingSoon'; diff --git a/docs/docs/reference/olap-engines/clickhouse.md b/docs/docs/reference/olap-engines/clickhouse.md index f52ae64f350..c2b696310cf 100644 --- a/docs/docs/reference/olap-engines/clickhouse.md +++ b/docs/docs/reference/olap-engines/clickhouse.md @@ -128,5 +128,5 @@ Note that you must `cd` into the Git repository that your project was deployed f ## Additional Notes - At the moment, we do not officially support modeling with ClickHouse. If this is something you're interested in, please [contact us](../../contact.md). -- For dashboards powered by ClickHouse, [measure definitions](/build/dashboards/dashboards.md#measures) are required to follow standard [ClickHouse SQL](https://clickhouse.com/docs/en/sql-reference) syntax. +- For dashboards powered by ClickHouse, [measure definitions](/build/metrics-view/metrics-view.md#measures) are required to follow standard [ClickHouse SQL](https://clickhouse.com/docs/en/sql-reference) syntax. - Because string columns in ClickHouse can theoretically contain [arbitrary binary data](https://github.com/ClickHouse/ClickHouse/issues/2976#issuecomment-416694860), if your column contains invalid UTF-8 characters, you may want to first cast the column by applying the `toValidUTF8` function ([see ClickHouse documentation](https://clickhouse.com/docs/en/sql-reference/functions/string-functions#tovalidutf8)) before reading the table into Rill to avoid any downstream issues. \ No newline at end of file diff --git a/docs/docs/reference/olap-engines/druid.md b/docs/docs/reference/olap-engines/druid.md index 504c98e9aad..19ef9234375 100644 --- a/docs/docs/reference/olap-engines/druid.md +++ b/docs/docs/reference/olap-engines/druid.md @@ -82,4 +82,4 @@ Note that you must `cd` into the Git repository that your project was deployed f ## Additional Notes - At the moment, we do not support modeling with Druid. If this is something you're interested in, please [contact us](../../contact.md). -- For dashboards powered by Druid, [measure definitions](/build/dashboards/dashboards.md#measures) are required to follow standard [Druid SQL](https://druid.apache.org/docs/latest/querying/sql/) syntax. \ No newline at end of file +- For dashboards powered by Druid, [measure definitions](/build/metrics-view/metrics-view.md#measures) are required to follow standard [Druid SQL](https://druid.apache.org/docs/latest/querying/sql/) syntax. \ No newline at end of file diff --git a/docs/docs/reference/olap-engines/duckdb.md b/docs/docs/reference/olap-engines/duckdb.md index 7d3db7e9c59..7e9f4a420ee 100644 --- a/docs/docs/reference/olap-engines/duckdb.md +++ b/docs/docs/reference/olap-engines/duckdb.md @@ -25,5 +25,5 @@ Well now you can! For more details, see our page on [Using Multiple OLAP Engines ## Additional Notes -- For dashboards powered by DuckDB, [measure definitions](/build/dashboards/dashboards.md#measures) are required to follow standard [DuckDB SQL](https://duckdb.org/docs/sql/introduction) syntax. +- For dashboards powered by DuckDB, [measure definitions](/build/metrics-view/metrics-view.md#measures) are required to follow standard [DuckDB SQL](https://duckdb.org/docs/sql/introduction) syntax. - Known issue around creating a DuckDB source via the UI, you will need to create the YAML file manually. \ No newline at end of file diff --git a/docs/docs/reference/olap-engines/multiple-olap.md b/docs/docs/reference/olap-engines/multiple-olap.md index a13d9f7275a..5a37c6ff6ed 100644 --- a/docs/docs/reference/olap-engines/multiple-olap.md +++ b/docs/docs/reference/olap-engines/multiple-olap.md @@ -69,7 +69,7 @@ For more information about available configurations for `rill.yaml`, please see ### Setting the OLAP Engine in the dashboard YAML -For each dashboard that is using a separate OLAP engine (than the default), you'll want to set the `connector` and `table` properties in the underlying [dashboard YAML](../project-files/dashboards.md) configuration to the OLAP engine and corresponding [external table](../../concepts/OLAP#external-olap-tables) that exists in your OLAP store respectively. +For each dashboard that is using a separate OLAP engine (than the default), you'll want to set the `connector` and `table` properties in the underlying [dashboard YAML](../project-files/explore-dashboards.md) configuration to the OLAP engine and corresponding [external table](../../concepts/OLAP#external-olap-tables) that exists in your OLAP store respectively. ```yaml type: metrics_view diff --git a/docs/docs/reference/olap-engines/pinot.md b/docs/docs/reference/olap-engines/pinot.md index df49e93fc09..ed6ba048478 100644 --- a/docs/docs/reference/olap-engines/pinot.md +++ b/docs/docs/reference/olap-engines/pinot.md @@ -84,7 +84,7 @@ Multi-Valued dimensions needed to be defined in the dashboard yaml as expression name: RandomAirports description: "Random Airports" ``` -Refer to the [Dashboard YAML](../project-files/dashboards) reference page for all dimension properties detail. +Refer to the [Dashboard YAML](../project-files/explore-dashboards) reference page for all dimension properties detail. :::note @@ -95,4 +95,4 @@ Pinot does not support unnest function so don't set `unnest` property to true in ## Additional Notes - At the moment, we do not support modeling with Pinot. If this is something you're interested in, please [contact us](../../contact.md). -- For dashboards powered by Pinot, [measure definitions](../../build/dashboards/dashboards.md#measures) are required to follow [Pinot SQL](https://docs.pinot.apache.org/users/user-guide-query/querying-pinot) syntax. \ No newline at end of file +- For dashboards powered by Pinot, [measure definitions](../../build/metrics-view/metrics-view.md#measures) are required to follow [Pinot SQL](https://docs.pinot.apache.org/users/user-guide-query/querying-pinot) syntax. \ No newline at end of file diff --git a/docs/docs/reference/project-files/connectors.md b/docs/docs/reference/project-files/connectors.md index 3bca792afad..ca22fe9b1b8 100644 --- a/docs/docs/reference/project-files/connectors.md +++ b/docs/docs/reference/project-files/connectors.md @@ -27,7 +27,7 @@ Select + Add -> Data -> Connect an OLAP engine :::tip A note on OLAP engines -By defining the `connector` parameter in a [dashboard's YAML](dashboards.md) file, you can have multiple OLAP engines in a single project. +By defining the `connector` parameter in a [dashboard's YAML](explore-dashboards.md) file, you can have multiple OLAP engines in a single project. ::: diff --git a/docs/docs/reference/project-files/dashboards.md b/docs/docs/reference/project-files/dashboards.md deleted file mode 100644 index 02f0745c9d7..00000000000 --- a/docs/docs/reference/project-files/dashboards.md +++ /dev/null @@ -1,107 +0,0 @@ ---- -title: Dashboard YAML -sidebar_label: Dashboard YAML -sidebar_position: 30 -hide_table_of_contents: true ---- - -In your Rill project directory, create a `.yaml` file in the `dashboards` directory. Rill will ingest the dashboard definition next time you run `rill start`. - -:::tip Did you know? - -Files that are *nested at any level* under your native `dashboards` directory will be assumed to be metric definitions, i.e. dashboards (unless **otherwise** specified by the `type` property). - -::: - -## Properties - -**`type`** — Refers to the resource type and must be `metrics_view` _(required)_. - -**`database_schema`** — Refers to the database or schema to use in the OLAP engine (to be used in conjunction with `table`). Otherwise, will use the default database or schema if not specified _(optional)_. - -**`model`** — Refers to the **model** powering the dashboard with no path specified; should only be used for [Rill models](/build/models/models.md) _(either **model** or **table** is required)_. - -**`table`** — Refers to the **table** powering the dashboard with no path specified; should be used instead of `model` for dashboards created directly from [sources](/build/connect/connect.md) and/or from [external OLAP tables](../../concepts/OLAP.md#external-olap-tables) _(either **table** or **model** is required)_. - -**`title`** — Refers to the display name for the dashboard _(required)_. - -**`timeseries`** — Refers to the timestamp column from your model that will underlie x-axis data in the line charts. If not specified, the line charts will not appear _(optional)_. - -**`connector`** — Refers to the OLAP engine, if using [multiple OLAP engines](../olap-engines/multiple-olap.md). - -**`default_time_range`** — Refers to the default time range shown when a user initially loads the dashboard. The value must be either a valid [ISO 8601 duration](https://en.wikipedia.org/wiki/ISO_8601#Durations) (for example, `PT12H` for 12 hours, `P1M` for 1 month, or `P26W` for 26 weeks) or one of the [Rill ISO 8601 extensions](../rill-iso-extensions.md#extensions) (default). If not specified, defaults to the full time range of the `timeseries` column _(optional)_. - -**`smallest_time_grain`** — Refers to the smallest time granularity the user is allowed to view in the dashboard. The valid values are: `millisecond`, `second`, `minute`, `hour`, `day`, `week`, `month`, `quarter`, `year` _(optional)_. - -**`first_day_of_week`** — Refers to the first day of the week for time grain aggregation (for example, Sunday instead of Monday). The valid values are 1 through 7 where Monday=`1` and Sunday=`7` _(optional)_. - -**`first_month_of_year`** — Refers to the first month of the year for time grain aggregation. The valid values are 1 through 12 where January=`1` and December=`12` _(optional)_. - -**`available_time_zones`** — Refers to the time zones that should be pinned to the top of the time zone selector. It should be a list of [IANA time zone identifiers](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones). By adding one or more time zones will make the dashboard time zone aware and allow users to change current time zone within the dashboard _(optional)_. - -**`default_theme`** — Refers to the default theme to apply to the dashboard. A valid theme must be defined in the project. Read this [page](./themes.md) for more detailed information about themes _(optional)_. - -**`default_comparison`** - Defines which should be the default comparison mode. Default to `none` _(optional)_. - - **`mode`** - comparison mode - - `none` - no comparison - - `time` - time, will pick the comparison period depending on `default_time_range` - - `dimension` - dimension comparison mode - - **`dimension`** - for dimension mode, specify the comparison dimension by name - -**`dimensions`** — Relates to exploring segments or [dimensions](/build/dashboards/dashboards.md#dimensions) of your data and filtering the dashboard _(required)_. - - **`column`** — a categorical column _(required)_ - - **`expression`** a non-aggregate expression such as `string_split(domain, '.')`. One of `column` and `expression` is required but cannot have both at the same time _(required)_ - - **`name`** — a stable identifier for the dimension _(optional)_ - - **`label`** — a label for your dashboard dimension _(optional)_ - - **`description`** — a freeform text description of the dimension for your dashboard _(optional)_ - - **`unnest`** - if true, allows multi-valued dimension to be unnested (such as lists) and filters will automatically switch to "contains" instead of exact match _(optional)_ - -**`measures`** — Used to define the numeric [aggregates](/build/dashboards/dashboards.md#measures) of columns from your data model _(required)_. - - **`expression`** — a combination of operators and functions for aggregations _(required)_ - - **`name`** — a stable identifier for the measure _(required)_ - - **`label`** — a label for your dashboard measure _(optional)_ - - **`description`** — a freeform text description of the dimension for your dashboard _(optional)_ - - **`valid_percent_of_total`** — a boolean indicating whether percent-of-total values should be rendered for this measure _(optional)_ - - **`format_d3`** — controls the formatting of this measure in the dashboard using a [d3-format string](https://d3js.org/d3-format). If an invalid format string is supplied, measures will be formatted with `format_preset: humanize` (described below). Measures cannot have both `format_preset` and `format_d3` entries. _(optional; if neither `format_preset` nor `format_d3` is supplied, measures will be formatted with the `humanize` preset)_ - - **Example**: to show a measure using fixed point formatting with 2 digits after the decimal point, your measure specification would include: `format_d3: ".2f"`. - - **Example**: to show a measure using grouped thousands with two significant digits, your measure specification would include: `format_d3: ",.2r"`. - - **`format_preset`** — controls the formatting of this measure in the dashboard according to option specified below. Measures cannot have both `format_preset` and `format_d3` entries. _(optional; if neither `format_preset` nor `format_d3` is supplied, measures will be formatted with the `humanize` preset)_ - - `humanize` — round off numbers in an opinionated way to thousands (K), millions (M), billions (B), etc. - - `none` — raw output - - `currency_usd` — output rounded to 2 decimal points prepended with a dollar sign: `$` - - `currency_eur` — output rounded to 2 decimal points prepended with a euro symbol: `€` - - `percentage` — output transformed from a rate to a percentage appended with a percentage sign - - `interval_ms` — time intervals given in milliseconds are transformed into human readable time units like hours (h), days (d), years (y), etc. - -**`available_time_ranges`** — Overrides the list of default time range selections available in the dropdown. Note that `All Time` and `Custom` selections are always available _(optional)_. - - **`range`** — a valid [ISO 8601 duration](https://en.wikipedia.org/wiki/ISO_8601#Durations) or one of the [Rill ISO 8601 extensions](../rill-iso-extensions.md#extensions) for the selection _(required)_ - - **`comparison_offsets`** — list of time comparison options for this time range selection _(optional)_. Must be one of the [Rill ISO 8601 extensions](../rill-iso-extensions.md#extensions). - - **Example**: - ```yaml - available_time_ranges: - - PT15M // Simplified syntax to specify only the range - - PT1H - - PT6H - - P7D - - range: P5D // Advanced syntax to specify comparison_offsets as well - comparison_offsets: - - rill-PP - - rill-PW - - P4W - - rill-TD // Today - - rill-WTD // Week-To-date - ``` - -**`default_dimensions`** - Provides the list of dimensions that should be visible on the dashboard upon initial render by default _(optional)_. - -**`default_measures`** - Provides the list of measures that should be visible on the dashboard upon initial render by default _(optional)_. - -**`security`** - Defines a [security policy](/manage/security) for the dashboard _(optional)_. - - **`access`** - Expression indicating if the user should be granted access to the dashboard. If not defined, it will resolve to `false` and the dashboard won't be accessible to anyone. Needs to be a valid SQL expression that evaluates to a boolean _(optional)_. - - **`row_filter`** - SQL expression to filter the underlying model by. Can leverage templated user attributes to customize the filter for the requesting user. Needs to be a valid SQL expression that can be injected into a `WHERE` clause _(optional)_. - - **`exclude`** - List of dimension or measure names to exclude from the dashboard. If `exclude` is defined all other dimensions and measures are included _(optional)_. - - **`if`** - Expression to decide if the column should be excluded or not. It can leverage templated user attributes. Needs to be a valid SQL expression that evaluates to a boolean _(required)_. - - **`names`** - List of fields to exclude. Should match the `name` of one of the dashboard's dimensions or measures _(required)_. - - **`include`** - List of dimension or measure names to include in the dashboard. If `include` is defined all other dimensions and measures are excluded _(optional)_. - - **`if`** - Expression to decide if the column should be included or not. It can leverage templated user attributes. Needs to be a valid SQL expression that evaluates to a boolean _(required)_. - - **`names`** - List of fields to include. Should match the `name` of one of the dashboard's dimensions or measures _(required)_. diff --git a/docs/docs/reference/project-files/explore-dashboards.md b/docs/docs/reference/project-files/explore-dashboards.md new file mode 100644 index 00000000000..d3c5447eed0 --- /dev/null +++ b/docs/docs/reference/project-files/explore-dashboards.md @@ -0,0 +1,84 @@ +--- +title: Explore Dashboard YAML +sidebar_label: Explore Dashboard YAML +sidebar_position: 30 +hide_table_of_contents: true +--- + +In your Rill project directory, create a explore dashboard, `.yaml`, file in the `dashboards` directory. Rill will ingest the dashboard definition next time you run `rill start`. + + +## Properties + +**`type`** — Refers to the resource type and must be `explore` _(required)_. + +**`metrics_view`** — Refers to the metrics_view resource _(required)_. + +**`title`** — Refers to the display name for the dashboard _(required)_. + +**`description`** - A description for the project. _(optional)_. + +**`dimensions`** - List of dimensions to include, defaults to `*`. _(optional)_. + + - **`exclude`** - Inversely a list of dimensions to exclude. Will ignore include if exclude is specified. _(optional)_. + +**`measures`** - List of measures to include, defaults to `*`. _(optional)_. + + - **`exclude`** - Inversely a list of measures to exclude. Will ignore include if exclude is specified. _(optional)_. + +**`defaults`** - defines the defaults YAML struct + + - **`time_range`** — Refers to the default time range shown when a user initially loads the dashboard. The value must be either a valid [ISO 8601 duration](https://en.wikipedia.org/wiki/ISO_8601#Durations) (for example, `PT12H` for 12 hours, `P1M` for 1 month, or `P26W` for 26 weeks) or one of the [Rill ISO 8601 extensions](../rill-iso-extensions.md#extensions) (default). If not specified, defaults to the full time range of the `timeseries` column _(optional)_. + + + - **`comparison_mode`** - comparison mode _(optional)_. + - `none` - no comparison + - `time` - time, will pick the comparison period depending on `default_time_range` + - `dimension` - dimension comparison mode + + - **`comparison_dimension`** - for dimension mode, specify the comparison dimension by name _(optional)_. + + - **`dimensions`** - Provides the default dimensions to load on viewing the dashboard. _(optional)_. + + - **`measures`** - Provides the default measures to load on viewing the dashboard. _(optional)_. + + **Default Example:** + ```yaml + defaults: #define all the defaults within here + dimensions: + - dim_1 + - dim_2 + measures: + - measure_1 + - measure_2 + time_range: P1M + comparison_mode: dimension #time, none + comparison_dimension: filename +``` + +**`time_ranges`** — Overrides the list of default time range selections available in the dropdown. Note that `All Time` and `Custom` selections are always available _(optional)_. + - **`range`** — a valid [ISO 8601 duration](https://en.wikipedia.org/wiki/ISO_8601#Durations) or one of the [Rill ISO 8601 extensions](../rill-iso-extensions.md#extensions) for the selection _(required)_ + - **`comparison_offsets`** — list of time comparison options for this time range selection _(optional)_. Must be one of the [Rill ISO 8601 extensions](../rill-iso-extensions.md#extensions). + + **Example**: + ```yaml + time_ranges: + - PT15M // Simplified syntax to specify only the range + - PT1H + - PT6H + - P7D + - range: P5D // Advanced syntax to specify comparison_offsets as well + comparison_offsets: + - rill-PP + - rill-PW + - P4W + - rill-TD // Today + - rill-WTD // Week-To-date + ``` + +**`time_zones`** — Refers to the time zones that should be pinned to the top of the time zone selector. It should be a list of [IANA time zone identifiers](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones). By adding one or more time zones will make the dashboard time zone aware and allow users to change current time zone within the dashboard _(optional)_. + +**`theme`** — Refers to the default theme to apply to the dashboard. A valid theme must be defined in the project. Read this [page](./themes.md) for more detailed information about themes _(optional)_. + +**`security`** - Defines a [security policy](/manage/security) for the dashboard _(optional)_. + - **`access`** - Expression indicating if the user should be granted access to the dashboard. If not defined, it will resolve to `false` and the dashboard won't be accessible to anyone. Needs to be a valid SQL expression that evaluates to a boolean _(optional)_. \ No newline at end of file diff --git a/docs/docs/reference/project-files/index.md b/docs/docs/reference/project-files/index.md index 93d36cdb734..aa72cfd554a 100644 --- a/docs/docs/reference/project-files/index.md +++ b/docs/docs/reference/project-files/index.md @@ -11,7 +11,7 @@ When you create sources, models, and dashboards, these objects are represented a :::info Working with resources outside their native folders -It is possible to define resources (such as [sources](sources.md), [models](models.md), [dashboards](dashboards.md), [custom APIs](apis.md), or [themes](themes.md)) within any nested folder within your Rill project directory. However, for any YAML configuration file, it is then imperative that the `type` property is then appropriately defined within the underlying resource configuration or Rill will not able to resolve the resource type correctly! +It is possible to define resources (such as [sources](sources.md), [models](models.md), [metrics-views](metrics-view.md), [dashboards](explore-dashboards.md), [custom APIs](apis.md), or [themes](themes.md)) within any nested folder within your Rill project directory. However, for any YAML configuration file, it is then imperative that the `type` property is then appropriately defined within the underlying resource configuration or Rill will not able to resolve the resource type correctly! ::: diff --git a/docs/docs/reference/project-files/metrics-view.md b/docs/docs/reference/project-files/metrics-view.md new file mode 100644 index 00000000000..f7a79d53ab0 --- /dev/null +++ b/docs/docs/reference/project-files/metrics-view.md @@ -0,0 +1,76 @@ +--- +title: Metrics View YAML +sidebar_label: Metrics View YAML +sidebar_position: 28 +hide_table_of_contents: true +--- + +In your Rill project directory, create a metrics view, `.yaml`, file in the `metrics` directory. Rill will ingest the metric view definition next time you run `rill start`. + + +## Properties + +**`version`** - Refers to the version of the metrics view _(required)_. + +**`type`** — Refers to the resource type and must be `metrics_view` _(required)_. + +**`title`** — Refers to the display name for the metrics view _(required)_. + +**`description`** - A description for the project. _(optional)_. + +**`database`** - Refers to the database to use in the OLAP engine (to be used in conjunction with `table`). Otherwise, will use the default database or schema if not specified _(optional)_. + +**`database_schema`** — Refers to the schema to use in the OLAP engine (to be used in conjunction with `table`). Otherwise, will use the default database or schema if not specified _(optional)_. + +**`watermark`** - A SQL expression that tells us the max timestamp that the metrics are considered valid for. Usually does not need to be overwritten, _(optional)_. + +**`timeseries`** — Refers to the timestamp column from your model that will underlie x-axis data in the line charts. If not specified, the line charts will not appear _(optional)_. + +**`connector`** — Refers to the OLAP engine, if you are not using DuckDB, IE: [ClickHouse OLAP engine](../olap-engines/multiple-olap.md). _(optional)_. + +**`model`** — Refers to the **model** powering the dashboard with no path specified; should only be used for [Rill models](/build/models/models.md) _(either **model** or **table** is required)_. + +**`table`** — Refers to the **table** powering the dashboard with no path specified; should be used instead of `model` for dashboards create from [external OLAP tables](../../concepts/OLAP.md#external-olap-tables) _(either **table** or **model** is required)_. + + +**`dimensions`** — Relates to exploring segments or [dimensions](/build/metrics-view/metrics-view.md#dimensions) of your data and filtering the dashboard _(required)_. + - **`column`** — a categorical column _(required)_ + - **`expression`** a non-aggregate expression such as `string_split(domain, '.')`. One of `column` and `expression` is required but cannot have both at the same time _(required)_ + - **`name`** — a stable identifier for the dimension _(optional)_ + - **`label`** — a label for your dimension _(optional)_ + - **`description`** — a freeform text description of the dimension _(optional)_ + - **`unnest`** - if true, allows multi-valued dimension to be unnested (such as lists) and filters will automatically switch to "contains" instead of exact match _(optional)_ + +**`measures`** — Used to define the numeric [aggregates](/build/metrics-view/metrics-view.md#measures) of columns from your data model _(required)_. + - **`expression`** — a combination of operators and functions for aggregations _(required)_ + - **`name`** — a stable identifier for the measure _(required)_ + - **`label`** — a label for your measure _(optional)_ + - **`description`** — a freeform text description of the dimension _(optional)_ + - **`valid_percent_of_total`** — a boolean indicating whether percent-of-total values should be rendered for this measure _(optional)_ + - **`format_d3`** — controls the formatting of this measure using a [d3-format string](https://d3js.org/d3-format). If an invalid format string is supplied, measures will be formatted with `format_preset: humanize` (described below). Measures cannot have both `format_preset` and `format_d3` entries. _(optional; if neither `format_preset` nor `format_d3` is supplied, measures will be formatted with the `humanize` preset)_ + - **Example**: to show a measure using fixed point formatting with 2 digits after the decimal point, your measure specification would include: `format_d3: ".2f"`. + - **Example**: to show a measure using grouped thousands with two significant digits, your measure specification would include: `format_d3: ",.2r"`. + - **`format_preset`** — controls the formatting of this measure according to option specified below. Measures cannot have both `format_preset` and `format_d3` entries. _(optional; if neither `format_preset` nor `format_d3` is supplied, measures will be formatted with the `humanize` preset)_ + - `humanize` — round off numbers in an opinionated way to thousands (K), millions (M), billions (B), etc. + - `none` — raw output + - `currency_usd` — output rounded to 2 decimal points prepended with a dollar sign: `$` + - `currency_eur` — output rounded to 2 decimal points prepended with a euro symbol: `€` + - `percentage` — output transformed from a rate to a percentage appended with a percentage sign + - `interval_ms` — time intervals given in milliseconds are transformed into human readable time units like hours (h), days (d), years (y), etc. + + +**`smallest_time_grain`** — Refers to the smallest time granularity the user is allowed to view. The valid values are: `millisecond`, `second`, `minute`, `hour`, `day`, `week`, `month`, `quarter`, `year` _(optional)_. + +**`first_day_of_week`** — Refers to the first day of the week for time grain aggregation (for example, Sunday instead of Monday). The valid values are 1 through 7 where Monday=`1` and Sunday=`7` _(optional)_. + +**`first_month_of_year`** — Refers to the first month of the year for time grain aggregation. The valid values are 1 through 12 where January=`1` and December=`12` _(optional)_. + +**`security`** - Defines a [security policy](/manage/security) for the dashboard _(optional)_. + - **`access`** - Expression indicating if the user should be granted access to the dashboard. If not defined, it will resolve to `false` and the dashboard won't be accessible to anyone. Needs to be a valid SQL expression that evaluates to a boolean _(optional)_. + - **`row_filter`** - SQL expression to filter the underlying model by. Can leverage templated user attributes to customize the filter for the requesting user. Needs to be a valid SQL expression that can be injected into a `WHERE` clause _(optional)_. + - **`exclude`** - List of dimension or measure names to exclude from the dashboard. If `exclude` is defined all other dimensions and measures are included _(optional)_. + - **`if`** - Expression to decide if the column should be excluded or not. It can leverage templated user attributes. Needs to be a valid SQL expression that evaluates to a boolean _(required)_. + - **`names`** - List of fields to exclude. Should match the `name` of one of the dashboard's dimensions or measures _(required)_. + - **`include`** - List of dimension or measure names to include in the dashboard. If `include` is defined all other dimensions and measures are excluded _(optional)_. + - **`if`** - Expression to decide if the column should be included or not. It can leverage templated user attributes. Needs to be a valid SQL expression that evaluates to a boolean _(required)_. + - **`names`** - List of fields to include. Should match the `name` of one of the dashboard's dimensions or measures _(required)_. diff --git a/docs/docs/reference/project-files/rill-yaml.md b/docs/docs/reference/project-files/rill-yaml.md index 04e47a22953..50eebfae2ee 100644 --- a/docs/docs/reference/project-files/rill-yaml.md +++ b/docs/docs/reference/project-files/rill-yaml.md @@ -33,7 +33,7 @@ Please see our reference documentation on [OLAP Engines](../olap-engines/olap-en ## Project-wide defaults -In `rill.yaml`, project-wide defaults can be specified for a resource type within a project. Unless otherwise specified, _individual resources will inherit any defaults_ that have been specified in `rill.yaml`. For available properties that can be configured, please refer to the YAML specification for each individual resource type - [sources](sources.md), [models](models.md), and [dashboards](dashboards.md) +In `rill.yaml`, project-wide defaults can be specified for a resource type within a project. Unless otherwise specified, _individual resources will inherit any defaults_ that have been specified in `rill.yaml`. For available properties that can be configured, please refer to the YAML specification for each individual resource type - [sources](sources.md), [models](models.md), and [dashboards](explore-dashboards.md) :::note Use plurals when specifying project-wide defaults @@ -44,7 +44,7 @@ In your `rill.yaml`, the top level property for the resource type needs to be ** For example, the following YAML configuration below will set a project-wide default for: - **Sources** - Configure a [source refresh](/build/connect/source-refresh.md). - **Models** - Automatically materialize the models as tables instead of views (the default behavior if unspecified). -- **Dashboards** - Set the [first day of the week](dashboards.md) for timeseries aggregations to be Sunday along with defining available timezones on a dashboard. +- **Dashboards** - Set the [first day of the week](explore-dashboards.md) for timeseries aggregations to be Sunday along with defining available timezones on a dashboard. ```yaml title: My Rill Project @@ -67,7 +67,7 @@ dashboards: :::info Hierarchy of inheritance and property overrides As a general rule of thumb, properties that have been specified at a more _granular_ level will supercede or override higher level properties that have been inherited. Therefore, in order of inheritance, Rill will prioritize properties in the following order: -1. Individual [source](/reference/project-files/sources.md)/[model](/reference/project-files/models.md)/[dashboard](/reference/project-files/dashboards.md) object level properties (e.g. `source.yaml` or `dashboard.yaml`) +1. Individual [source](/reference/project-files/sources.md)/[model](/reference/project-files/models.md)/[dashboard](/reference/project-files/explore-dashboards.md) object level properties (e.g. `source.yaml` or `dashboard.yaml`) 2. [Environment](/docs/build/models/environments.md) level properties (e.g. a specific property that have been set for `dev`) 3. [Project-wide defaults](/reference/project-files/rill-yaml.md#project-wide-defaults) for a specific property and resource type diff --git a/docs/docs/tutorials/administration/alerts.md b/docs/docs/tutorials/administration/project/alerts.md similarity index 100% rename from docs/docs/tutorials/administration/alerts.md rename to docs/docs/tutorials/administration/project/alerts.md diff --git a/docs/docs/tutorials/administration/credential-envvariable-mangement.md b/docs/docs/tutorials/administration/project/credential-envvariable-mangement.md similarity index 100% rename from docs/docs/tutorials/administration/credential-envvariable-mangement.md rename to docs/docs/tutorials/administration/project/credential-envvariable-mangement.md diff --git a/docs/docs/tutorials/administration/github.md b/docs/docs/tutorials/administration/project/github.md similarity index 100% rename from docs/docs/tutorials/administration/github.md rename to docs/docs/tutorials/administration/project/github.md diff --git a/docs/docs/tutorials/administration/project-maintanence.md b/docs/docs/tutorials/administration/project/project-maintanence.md similarity index 91% rename from docs/docs/tutorials/administration/project-maintanence.md rename to docs/docs/tutorials/administration/project/project-maintanence.md index a3b7e60321a..f53b9fe5d58 100644 --- a/docs/docs/tutorials/administration/project-maintanence.md +++ b/docs/docs/tutorials/administration/project/project-maintanence.md @@ -126,26 +126,10 @@ rill project refresh --source [--model ] ## Settings - - -
-Historically (pre 0.48), user management was only possible via the CLI. Now, it is also possible to do so via the UI! - - +An additional page for administrators to manage objects in Rill Cloud. ### Public URL Management - - -
-Historically (pre 0.48), user management was only possible via the CLI. Now, it is also possible to do so via the UI! - -
- -### Environmental Variable Management - -
-Historically (pre 0.48), user management was only possible via the CLI. Now, it is also possible to do so via the UI! +Along with the CLI, you can also view and manage the public URLs from the Settings page. As an administrator, you can re-copy the URL or delete the public URL. +![img](/img/tutorials/admin/settings-public-url.png) -
-
\ No newline at end of file diff --git a/docs/docs/tutorials/administration/user-group-management.md b/docs/docs/tutorials/administration/user/user-group-management.md similarity index 100% rename from docs/docs/tutorials/administration/user-group-management.md rename to docs/docs/tutorials/administration/user/user-group-management.md diff --git a/docs/docs/tutorials/administration/user-management.md b/docs/docs/tutorials/administration/user/user-management.md similarity index 100% rename from docs/docs/tutorials/administration/user-management.md rename to docs/docs/tutorials/administration/user/user-management.md diff --git a/docs/docs/tutorials/other/Rill Cloud/share-dashboard-publicly.md b/docs/docs/tutorials/other/Rill Cloud/share-dashboard-publicly.md new file mode 100644 index 00000000000..9aa7e186295 --- /dev/null +++ b/docs/docs/tutorials/other/Rill Cloud/share-dashboard-publicly.md @@ -0,0 +1,61 @@ +--- +title: "Share your dashboards via Public URL" +sidebar_label: "Sharing Dashboards Publicly" +hide_table_of_contents: false +tags: + - Rill Cloud +--- + +There might be occasions that sharing a dashboard to a 'non-user' is required in your workflow. In order to accomdate such situations, you can send a public URL of your dashboard with set parameters and with a expiration date if required. + + +### Accessing your Dashboard in Rill Cloud +Once you have deployed your project to Rill Cloud, you should be able to access the project via the following URL: + +[Go to Rill Cloud](https://ui.rilldata.com/) + +You can select your project from the list and select your dashboard. + +![img](/img/tutorials/201/rill-cloud-projects.png) + +### Adding Filters and Creating the Public URL + +When sharing to your end-user, it is likely that you will want to specific a specific amount of filters. Once set, the end-user cannot view or modify the set filters so they will only be allowed to view a set portion of the dasboard that you define. + + +Once your dashboard is ready, you can create the public URL via the `Share` button. + +![img](/img/tutorials/other/public-url/share-public-url.png) + + +### Managing Public URLS in Rill Cloud + +**via UI** + +Public URL can also be managed via the Settings page in Rill Cloud. Please refer to the [administrators guide](https://docs.rilldata.com/tutorials/administration/project-maintanence#public-url-management)! + + + +**via CLI** + +Checking the public URL can be done by running the following: + +```bash +rill public-url list --project + ID DASHBOARD FILTER CREATED BY CREATED ON LAST USED ON EXPIRES ON + -------------------------------------- ------------- ------------------------------------- ----------------------- --------------------- --------------------- ------------ + 3564c499-c8bd-4c1c-bab8-c33a218a5009 advanced_metrics_view_explore (author_name IN ["Alexey Milovidov""]) roy.endo@rilldata.com 2024-09-30 09:34:41 2024-09-30 09:34:41 + cab99113-d5a8-468d-981e-213e41c7d1bf advanced_metrics_view_explore roy.endo@rilldata.com 2024-09-30 09:29:26 2024-09-30 09:34:32 + +NOTE: For security reasons, the actual URLs can't be displayed after creation. +``` +As you can see, you can receive information on who created, what filters, when it expires, etc. + +### Example Public URL + +Feel free to take a look at the created public URLs and note the difference between a dashboard with a filter and without. + +[Public URL without filter](https://ui.rilldata.com/rill_learn/my-rill-tutorial/-/share/rill_mgc_boCupdujFIo0I0DFL7yoO3bAGdbaSqXWUn5OXIlXL8VeDDTENARBPv) + + +[Public URL with Filter](https://ui.rilldata.com/rill_learn/my-rill-tutorial/-/share/rill_mgc_1iB5JJ7CPz4g59vTihTnlYOkvqn8mYivtNxSpnvgnhc1IY56Pi86hs) \ No newline at end of file diff --git a/docs/docs/tutorials/other/Rill Cloud/views.md b/docs/docs/tutorials/other/Rill Cloud/views.md new file mode 100644 index 00000000000..bd278bbe5f9 --- /dev/null +++ b/docs/docs/tutorials/other/Rill Cloud/views.md @@ -0,0 +1,11 @@ +--- +title: "Save your Most Common Views" +sidebar_label: "Save your Most Common Views" +hide_table_of_contents: false +tags: + - Rill Cloud +--- + +Whether you're an administrator in Rill Cloud or just a user, you can set up views for dashboards used frequently. + +## \ No newline at end of file diff --git a/docs/docs/tutorials/other/Rill Cloud/visual-metric-editor-rc.md b/docs/docs/tutorials/other/Rill Cloud/visual-metric-editor-rc.md new file mode 100644 index 00000000000..ec99c9c0736 --- /dev/null +++ b/docs/docs/tutorials/other/Rill Cloud/visual-metric-editor-rc.md @@ -0,0 +1,16 @@ +--- +title: "Edit Measures and Dimensions on the Fly" +sidebar_label: "Edit Measures and Dimensions on the Fly" +hide_table_of_contents: false +tags: + - Rill Cloud +--- + +import ComingSoon from '@site/src/components/ComingSoon'; + + + + +
+Waiting for Visual Metric Editor in RC +
diff --git a/docs/docs/tutorials/other/_category_.yml b/docs/docs/tutorials/other/_category_.yml new file mode 100644 index 00000000000..a29146f3a5c --- /dev/null +++ b/docs/docs/tutorials/other/_category_.yml @@ -0,0 +1,5 @@ +position: 200 +label: " " +collapsible: true +collapsed: true +className: hidden \ No newline at end of file diff --git a/docs/docs/tutorials/other/add-column-dimension.md b/docs/docs/tutorials/other/add-column-dimension.md index e9a3be83969..6d7b564572c 100644 --- a/docs/docs/tutorials/other/add-column-dimension.md +++ b/docs/docs/tutorials/other/add-column-dimension.md @@ -1,44 +1,48 @@ --- title: "How do I add a new dimension to my dashboard?" -sidebar_label: "How to: Add Dimension to Dashboard" +sidebar_label: "Add/Modify Dimension to Dashboard via YAML" sidebar_position: 10 hide_table_of_contents: false tags: - OLAP:DuckDB - - OLAP:ClickHouse --- - -Changes to the underlying data may require modification to your SQL / Dashboard in order to start visualizing your new data. - You will need to add the dimension/s back in Rill Developer and [push the changes to Rill Cloud](/tutorials/rill_advanced_features/advanced_developer/update-rill-cloud) when you're ready. ## Rill Developer -Depending on whether your dashboard is created directly from your source or via a model (check if your dashboard as a `table:` or `model:` it it's YAML), you will need to check the corresponding file to make the changes. +Depending on whether your dashboard was created directly from your source or via a model, you will need to ensure that all the components are updated. ### Checking the Sources You need to check that the new column is being ingested into Rill. If you have a select * statement, go ahead and select the `refresh` button and confirm that the new column is listed. If not, add the new column into your select statement and select `refresh`. -If your dashboard is created directly from the source, navigate to [adding the new dimension to the Dashboard](#adding-the-new-dimension-to-the-dashboard). If not, continue to the model to make changes. +If your dashboard is created directly from the source, navigate to [adding the new dimension to the metrics view](/tutorials/other/add-column-dimension#adding-the-new-dimension-to-the-metrics-view). If not, continue to the model to make changes. -![source](/img/tutorials/other/source-new-dimension.png) +![source](/img/tutorials/other/new-dimension/source-new-dimension.png) --- -### Dashboard created from a Model +### Metrics View created from a Model -After confirming that the sources have ingested the new data, you can [modify the model to include these new dimensions / measures](https://docs.rilldata.com/build/models/). If you need to make any transformations, you can do so here or [in the dashboard layer](https://docs.rilldata.com/build/dashboards/expressions). +After confirming that the sources have ingested the new data, you can [modify the model to include these new dimensions / measures](https://docs.rilldata.com/build/models/). If you need to make any transformations, you can do so here or [in the metrics view layer](https://docs.rilldata.com/build/dashboards/expressions). -![model](/img/tutorials/other/model-new-dimension.png) +![model](/img/tutorials/other/new-dimension/model-new-dimension.png) --- -### Adding the new Dimension to the Dashboard +### Adding the new Dimension to the Metrics view + +You can add the dimension / measure to the metrics layer. Notice in the right panel, your newly created dimension or metric can be seen in the right panel. You can add either from the Visual metric editor or the YAML. + +![metrics](/img/tutorials/other/new-dimension/metrics-new-dimension.png) + + +### Adding the dimension to the Dashboard + +Finally, you can preview the dashboard to see if the new dimension/measure is viewable. If not, please double check the dashboard YAML. -Finally, you can add the dimension / measure to the dashboard. Notice in the right panel, your newly created dimension or metric can be seen in the right panel. -![dashboard](/img/tutorials/other/dashboard-new-dimension.png) +![dashboard](/img/tutorials/other/new-dimension/dashboard-new-dimension.png) ## Rill Cloud diff --git a/docs/docs/tutorials/other/component-variable-freedom.md b/docs/docs/tutorials/other/component-variable-freedom.md new file mode 100644 index 00000000000..4899a1c77e9 --- /dev/null +++ b/docs/docs/tutorials/other/component-variable-freedom.md @@ -0,0 +1,29 @@ +--- +title: "Using Variables in Components, to what extent?" +sidebar_label: "Filter your Components for your own personal View" +sidebar_position: 10 +hide_table_of_contents: false +tags: + - Canvas Dashboard + - Canvas Component +--- + +## An idea + +What if you create a component that is COMPLETELY based off of variables? +The source is selectable and depending on which source your selecting, this filters the filters on which columns are measures are available. + + +selector_source.yaml -> outputs 'source_name' to all components + +source_name -> selector_dimensions.yaml -> outputs dimensions -> all components +source_name -> selector_measure.yaml -> outputs measures -> all components + +source_name and dimension -> selector_group_by -> outputs a groupby for components defaults to "" + +source_name, measure +x, measure_0 +x, measure_1 +y, measure_0 +y, measure_1 + diff --git a/docs/docs/tutorials/other/create-map-component.md b/docs/docs/tutorials/other/create-map-component.md new file mode 100644 index 00000000000..f46ca1ee491 --- /dev/null +++ b/docs/docs/tutorials/other/create-map-component.md @@ -0,0 +1,82 @@ +--- +title: "How to create a map Component in Rill for Canvas Dashboards" +sidebar_label: "Visualize Maps in Rill" +sidebar_position: 10 +hide_table_of_contents: false +tags: + - Canvas Dashboard + - Canvas Component +--- + +We will assume that you've already gotten started on build canvas dashboards and are interested in creating a map component. If not, please refer to the [canvas dashboard](/tutorials/rill_advanced_features/canvas_dashboards/getting-started) section. + +### Import the source into Rill +If you have your data source already available, go ahead and add it now. If not, you can use the sample dataset hosted here: +``` +https://cdn.rilldata.com/examples/superstore.csv +``` +We'll use the above dataset for this guide. + + +### Create the Model / Dashboard + +In our case, our data is already prepared, but if not, you can create a model from the superstore table. Next, you can create a dashboard from the model/source. Take note of the final name of the dashboard as you will need this when you are selecting the data via `metrics_sql`. + +Creating the dashboard with AI will not add the `name:` key to the dimensions, so you will have to add this to the state dimension. + +```yaml + - label: State + column: State + name: state +``` + +### Create the Map Component + +From the UI, select +Add -> Component. +This will open the default component sample graph. Delete all the information under `type:component`. + +First, let's create the `data:` key. We'll use the metrics_sql and select one measure and one dimension for the generated dashboard. + +```yaml +data: + metrics_sql: > + select state, total_sales from superstore_model_dashboard #this is the name of your dashboard.yaml + + +vega_lite: > +``` + +You should see a table appear with the data from the SQL query. + +![img](/img/tutorials/other/maps/data.png) + + +### Using Generate using AI feature + +Select `Generate using AI` and fill in the prompt as shown below. IN order to ensure that our data is what we expect, let's create a simple bar chart first. + +``` +Make me a bar chart using the states and total_sales column data. +``` +As we can see from the image below, the results automatically overwrites the vega_lite section and creates a spec that displays a bar_chart. + +![img](/img/tutorials/other/maps/bar.png) + + +Once confirmed, we want to use the AI feature again to generate a map based on the total_sales per state and visualize this is as a USA map. In order to do so, you will need to download the public geoJSON file for USA ([Available in the public folder of my-rill-project in our rill example repository](https://github.com/rilldata/rill-examples/blob/main/my-rill-tutorial/public/us-states.json)) + +Once this is added to public/us-states.json of your Rill project directory, we can try to ask OpenAI to create a map visualization. Note this might take a few tiems and it's not perfect. + + +In my example, I input the following prompt: +``` +Using table as the data, I want to look up the state column from public/us-states.json. The GeoJSON data has its features inside the features array, and I want to match the state field with properties.NAME. Do not add fields: ["geometry"]. Instead, assign the entire GeoJSON feature as geo, without isolating any specific fields. I will use geo for shape encoding, and color will be determined by total_sales. The projection should be albersUsa. +``` +:::note +As this is using AI to generate the the vega_lite spec, there is no guarantee that you will get the results you are looking for everytime. In the above example, we had to provide quite a detailed prompt in order for the generated result to satisfy the requirements and a correct chart to be displayed. +::: + +![img](/img/tutorials/other/maps/map_fixed.png) + + +Just like that, you have a vega-lite powered map that you can add to your canvas dashboard. Like any other component, you can add input variables to filter this map and make an interactive experience for your users. \ No newline at end of file diff --git a/docs/docs/tutorials/other/custom-charts.md b/docs/docs/tutorials/other/custom-charts.md new file mode 100644 index 00000000000..33bbade87df --- /dev/null +++ b/docs/docs/tutorials/other/custom-charts.md @@ -0,0 +1,89 @@ +--- +title: "Using Natural lanugage to Create Components for Canvas Dashboards" +sidebar_label: "Using Natural lanugage to create Components" +sidebar_position: 10 +hide_table_of_contents: false +tags: + - Canvas Dashboard + - Canvas Component +--- + +While we specialize in time-series charts on our Explore and Pivot view, we understand that there is always a need for a more customized view for specific individuals or groups. However, writing out specs to create a picture-perfect dashboard can be time-consuming. +In this guide, we will go over the steps to create your own custom components using natural language to create a unique canvas dashboard. You can follow along with the `rill-openrtb-prog-ads` project in our [Rill Examples repository](https://github.com/rilldata/rill-examples/tree/main/rill-openrtb-prog-ads). + + +## Rill Components + +Rill Components are developed in Rill Developer and are used as single units within a Canvas Dashboard. Please review [our documentation on Canvas Dashboards](https://docs.rilldata.com/build/canvasdashboards/) or the [Advanced Features walkthrough](https://docs.rilldata.com/tutorials/rill_advanced_features/canvas_dashboards/getting-started) for more information on this concept. + + +### Creating a Component Overview + +In order to create a component in Rill you need to define the following : + +**`type`** - `component` *(required)* + +**`data`** - This section defines which data should be exposed to the component. Various different data resolvers are available such as `metrics_sql`, `api` and `sql` *(required)* + +**`renderer`** - This defines the visualization spec, can be vega_lite, or any [Rill Authored Component](https://docs.rilldata.com/build/canvasdashboards/components#rill-authored-components) + +In this example, as we will be using the natural language to create our component we will use the `vega_lite:` renderer. + +## Create a Component + +Navigate to Rill Developer, and select `Component` under the `Add` dropdown. + +![img](/img/tutorials/other/natural-language/adding-component.png) + +### Preparing the Data +Before we can start asking Rill to create a visualization, we will need some data to be available on the Component. You can choose between running a raw `SQL` query against the available models, a `metris_sql` against the metrics layer or using an available `API`. + +In this example, we will be using a `metrics_sql` and select the following measure and dimension: +```yaml +data: + metrics_sql: > + select impressions, placement_type from bids +``` +![img](/img/tutorials/other/natural-language/component-data.png) +:::note +In order to call the measure and dimension by name, the `name` key must be defined on the dashboard level. Please refer to the dashboard, `bids`, for more information. +::: + +Once we see that data is available in the table below the chart, we can select the `Generate using AI` button and type in a prompt. + +### Using Natural Language + +``` +Create a bar chart +``` +![img](/img/tutorials/other/natural-language/bar-chart.png) + +When finished, the UI will automatically update with the corresponding vega_lite spec and a simple bar chart should be displayed above the table. If you want to modify the look, or after you've changed the data, re-create the chart, you can do so. The Generate using AI UI will save your previous runs of the component and can be refreshed. + +### Making some modifications +For example, let's modify the `metrics_sql` and pull in the time-series column, `__time`. +```yaml + metrics_sql: | + select impressions, placement_type, __time from bids +``` + +From here, let's provide a bit more detailed request. + +``` +create a stacked bar chart with __time over impression and color the stack by placement_type. +span the chart the whole width of the display +``` + +![img](/img/tutorials/other/natural-language/stacked-bar-chart.png) + +As you can see, the request sends out a more detailed vega-lite spec that fits the request and the chart looks better. In addition to this, you can have a conversation-like chat with the UI. For example, instead of modifying the whole prompt, you can say something like, + +``` +Actually I want this to be a line chart. +``` + +and the chart will update with all the same components but as a line chart. + +![img](/img/tutorials/other/natural-language/line-chart.png) + +As the graphs are creating a vega-lite spec, you can create many different types of graphs for your needs. For a full example, please refer to [Vega-lite's example page](https://vega.github.io/vega-lite/examples/), and explore our [GitHub repositories](https://github.com/rilldata/rill-examples/) for more examples. \ No newline at end of file diff --git a/docs/docs/tutorials/other/dashboard-row-policies.md b/docs/docs/tutorials/other/dashboard-row-policies.md new file mode 100644 index 00000000000..a5a2c66f433 --- /dev/null +++ b/docs/docs/tutorials/other/dashboard-row-policies.md @@ -0,0 +1,124 @@ +--- +title: "Creating Dashboard Policies" +sidebar_label: "Access Policies for Partner Facing Dashboards" +hide_table_of_contents: false +--- + +For this example, we will use the OpenRTB Programmatic Advertising project available in our [examples repostory.](https://github.com/rilldata/rill-examples/tree/main/rill-openrtb-prog-ads). + + +## Dashboard Access Policies + +While there are times that you can completely limit a user from viewing a dashboard by setting up [dashboard-level access](https://docs.rilldata.com/manage/security#restrict-dashboard-access-to-users-matching-specific-criteria), there are times that you might want to re-use the same dashboard but limit the view based on the user. + +Depending on how you will manage the row policies, depends highly on the type of data and the domain. IE, if the domain of the user matches with a column's value, then you will not need to create any mapping. However, in this example, we will cover creating a mapping file, and mapping domains to different values in the column, "Pub Name". + +Let's look at the possible values for Pub Name: [Click Here](https://ui.rilldata.com/demo/rill-openrtb-prog-ads/auction) + +Some values include: Disney, Pluto TV, LG USA, ... + +## Dashboard Access via Mapping Table +### Creating the Mapping Table +There are many ways to set up the mapping file. Whether it's directly in a model SQL, or reading from a S3 bucket. You have the freedom to decide, in this example, we will make it directly in the models/model.sql file. + +```SQL +-- Model SQL +-- Reference documentation: https://docs.rilldata.com/reference/project-files/models + + SELECT * FROM (VALUES + ('Disney', 'domain.com'), + ('Disney', 'rilldata.com'), + ... -- add your email and pub name here! + ) AS t(PubName, domain) +``` + +From this SQL file, we create a table that will map the `rilldata.com` domain to `Disney`. You will need to modify the file to add an email domain that fits your email and add a `Pub Name` from one that exists in the demo dashboard. + +### Creating the Row level Dashboard Policy +Now that this is created, you have a few options on which level you want to create the security level policy. Some questions to ask yourself is: +1. Am I using the metrics view on other components other than the dashboard? IE: APIs, canvas dashboards +2. If I am using the metrics view in other locations, how strict do I want the metrics layer to be? + +For most situations, you would define the dashboard policies at the metrics-view level. So let's do that. Let's create a new metrics view, `auction_data_model_metrics_row_policies.yaml`, and copy the contents of `auction_data_model_metrics.yaml` into it. + +In our new file, we want to define the following security rule: +```yaml +security: + access: true + row_filter: "Pub_Name IN (SELECT PubName FROM mapping WHERE domain = '{{ .user.domain }}')" +``` + +From our created model `mapping`, we are running the following SQL statement. + +```SQL +SELECT PubName FROM mapping WHERE domain = '{{ .user.domain }}' +``` + +Using the login information from our current user account, .user.domain will extract the domain from your email. In my case, rilldata.com is being extracted. Since the row `rilldata.com` matches, it returns the value in column PubName, 'Disney'. + +This translated back into the query, runs: + +```SQL +security: + access: true + row_filter: "Pub_Name IN `Disney`" +``` + +Which results in the current view: + +![img](/img/tutorials/other/row-policy/row-policy-view.png) + + +### Additional Set Up Possibilities + +This is a relatively straight forward example of row policies in Rill. By setting up a mapping file, you can allow specific data to be visible for specific individuals. + +Let's say you hired a contractor to assist with several customers. Mapping their domain or even email to the accounts will grant them visibility to only that specific data without having to create a new dashboard, metrics view, etc. + +Grant Rill Data to three values in 'Pub Name', but for Roy only LG USA. You would need to modify the SQL statement in the metrics-view to also accommodate email. +```SQL +-- Model SQL +-- Reference documentation: https://docs.rilldata.com/reference/project-files/models + + SELECT * FROM (VALUES + ('Disney', 'domain.com', ''), + ('Disney', 'rilldata.com', ''), + ('LG USA', 'rilldata.com', 'roy.endo@rilldata.com'), + ('Pluto', 'rilldata.com', '') + ) AS t(PubName, domain, email) +``` +```yaml + row_filter: "Pub_Name IN (SELECT PubName FROM mapping WHERE domain = '{{ .user.domain }}' {{ if .user.email}} AND email = '{{.user.email}}' {{ end }})" +``` + +For more possibilities on attributes please see [our documentation](https://docs.rilldata.com/manage/security#user-attributes). + + +## Dashboard Access (Column-level access) + +Another use-case is removing columns for user or groups that do not need them. Going back to our example openrtb project, let's say you are creating this dashboard for bids but do not want to provide specific information to non-company viewers. While you could create a new dashboard and remove some dimensions and measures, you can also use the same dashboard with a specific security policy. + +```yaml +security: + access: true #access is provided but + exclude: #if the use is not part of rilldata, then exclude: + - if: "'{{ .user.domain }}' != 'rilldata.com'" + names: + - bid_floor_bucket + - auction_type + - win_rate +``` + + + +In this case, if the user is not part of the Rill Data, they will not be able to view the listed dimension and measures. This is especially the case if your dashboard has customer sensitive information that should not be viewable. + + +## Dashboard Access + +This can be set both on the metric view level and at the dashboard level. Simply define an access statement like below: + +```yaml +security: + access: "{{ .user.admin }} OR '{{ .user.domain }}' == 'example.com'" +``` diff --git a/docs/docs/tutorials/other/deep-dive-incremental-modeling.md b/docs/docs/tutorials/other/deep-dive-incremental-modeling.md new file mode 100644 index 00000000000..6f085650eca --- /dev/null +++ b/docs/docs/tutorials/other/deep-dive-incremental-modeling.md @@ -0,0 +1,122 @@ +--- +title: "Deep Dive into Incremental Modeling" +sidebar_label: "Understanding Incremental Modeling" +sidebar_position: 10 +hide_table_of_contents: false +--- + +In this article, we will, along with some code, go through a detailed step by step example of how to use incremental modeling. + +:::note Requirements +In order to follow all the contets this guide, you will need access to a cloud based provider (s3, gcs) and datawarehouse (bq, snowflake) as well as be comfortable with running some basic python code (this will be provided). +::: + +## Getting Started + +Before starting the guide, it is a good idea to review the documentation on [incremental modeling]( and [splits](. I will assume that you've already have gone through the trial and have some basic understanding of the concepts. + +Ensure that you have python installed on your system so that we can use some basic scripts to write files / run SQL queries to update the data manually. + + +## Cloud storage Incremental Modeling + +In order to understand and control the new data ingestion, we will be using a python code that manually writes into GCS / S3 based on the following format `/YYYY/MM/DD/HH/MM/SS/rilldata-incremental-model.csv`. + +A few things to note: +1. On first run, the program will try to read your credentials. If it does not exist, it will request for you locate the JSON. +2. The code will request a number of rows to write to your bucket. +3. The number of rows will increase the column, `number` so that we can see the increasing values. +4. Please modify line 142 and add you bucket name: ` bucket_name = '' # replace with your bucket name` + +Let's try to run the code and see what the first run looks like. + +```bash +python3 deepdive.py + +Enter the path to your GCS credentials JSON file: /Users/royendo/Downloads/rilldata.json +Max 'number' value found across all files: 0 +Enter the number of new rows to add: 10 +File uploaded to GCS at path: 2024/09/30/17/38/44/rilldata-incremental-model.csv +Successfully updated rilldata-incremental-model.csv with 10 new rows. +``` + +Now, lets open Rill and create an Incremental Model. + +### Creating the Incremental Model. + +```yaml +type: model +incremental: true + +splits: + glob: glob: gs://rendo-test/**/rilldata-incremental-model.csv + +sql: SELECT * FROM read_csv('{{ .split.uri }}', auto_detect=true, ignore_errors=1, header=true) +``` + +Let's take a minute to check out the split that was just created when reading in the initial data. +```bash +rill project splits deepdive --local + KEY DATA EXECUTED ON ELAPSED ERROR + ---------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------ ---------------------- --------- ------- + 591d2ae89c74e7c11516097fbf3c1723 {"path":"2024/09/30/17/38/44/rilldata-incremental-model.csv","uri":"gs://rendo-test/2024/09/30/17/38/44/rilldata-incremental-model.csv"} 2024-09-30T23:49:39Z 319ms + ``` + +Run the code again to insert 5 more rows. You won't see the new data yet populated in Rill even when refreshing the page. You'll have to run a refresh of the model. Let's do that. + +### Refreshing the Incremental Model + +```bash +rill project refresh --model deepdive --local +Refresh initiated. Check the project logs for status updates. +``` + +```bash +rill project splits deepdive --local + KEY DATA EXECUTED ON ELAPSED ERROR + ---------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------ ---------------------- --------- ------- + 591d2ae89c74e7c11516097fbf3c1723 {"path":"2024/09/30/17/38/44/rilldata-incremental-model.csv","uri":"gs://rendo-test/2024/09/30/17/38/44/rilldata-incremental-model.csv"} 2024-09-30T23:49:39Z 319ms + 442b7a42276b914d64eace9aab917a34 {"path":"2024/09/30/17/50/04/rilldata-incremental-model.csv","uri":"gs://rendo-test/2024/09/30/17/50/04/rilldata-incremental-model.csv"} 2024-09-30T23:53:05Z 165ms + ``` + +What about if we create several files and refresh them all together? + +```bash + rill project splits deepdive --local + KEY (5) DATA EXECUTED ON ELAPSED ERROR + ---------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------ ---------------------- --------- ------- + 591d2ae89c74e7c11516097fbf3c1723 {"path":"2024/09/30/17/38/44/rilldata-incremental-model.csv","uri":"gs://rendo-test/2024/09/30/17/38/44/rilldata-incremental-model.csv"} 2024-09-30T23:49:39Z 319ms + 442b7a42276b914d64eace9aab917a34 {"path":"2024/09/30/17/50/04/rilldata-incremental-model.csv","uri":"gs://rendo-test/2024/09/30/17/50/04/rilldata-incremental-model.csv"} 2024-09-30T23:53:05Z 165ms + 1fcda4bee740beb41f2237aa7711de95 {"path":"2024/09/30/17/54/02/rilldata-incremental-model.csv","uri":"gs://rendo-test/2024/09/30/17/54/02/rilldata-incremental-model.csv"} 2024-09-30T23:55:38Z 175ms + 2596a58289ffa507ff17af7459d5be16 {"path":"2024/09/30/17/55/32/rilldata-incremental-model.csv","uri":"gs://rendo-test/2024/09/30/17/55/32/rilldata-incremental-model.csv"} 2024-09-30T23:55:38Z 169ms + 2d33015a8e0b091ce44ef4c0961eda91 {"path":"2024/09/30/17/55/35/rilldata-incremental-model.csv","uri":"gs://rendo-test/2024/09/30/17/55/35/rilldata-incremental-model.csv"} 2024-09-30T23:55:39Z 160ms + ``` + + Note that while they are in different folder paths in GCS, these are all executed on, or refreshed at the same time stamp, as you would expect. Rill is reading for the new files that did not exist before **or** files that have changed since last ingestion and is incrementally refreshing these. + +### Automatic Refreshes +After pushing to Rill Cloud, you will want to have an automatic refresh of the model, so you aren't running a refresh command manually as we just did. In this case, you can setup a refresh like below. + +```yaml +refresh: + cron: 0 0 * * * + ``` + + +### Per file or per directory? + +Depending on your use case and how your data is set up, there are a few things to consider for how you set up your file hierarchy and incremental modeling. + +The current setup, creates a split per file. This doesn't have any adverse affects as we only have 1 file in each directory. But what if you had mulitple files in a directory that when modified, you want to ensure that **all** the files in the directory are refreshed and synced? + + ```yaml +glob: + path: glob: gs://rendo-test/**/*.csv + partition: directory + ``` + + + + +## Data Warehouse Incremental Modeling + diff --git a/docs/docs/tutorials/other/embed-dashboard.md b/docs/docs/tutorials/other/embed-dashboard.md new file mode 100644 index 00000000000..e69b988c93f --- /dev/null +++ b/docs/docs/tutorials/other/embed-dashboard.md @@ -0,0 +1,117 @@ +--- +title: "How to Embed a Dashboard" +sidebar_label: "How to Embed your Rill Dashboards" +sidebar_position: 10 +hide_table_of_contents: false +--- + +The following guide is based on the example repository: [Rill Embedding Example](https://github.com/rilldata/rill-embedding-example). We will create the same three embed dashboards in this guide. (We will not go over how to create the web application) To view our publicly available web application: [Click here!](https://rill-embedding-example.netlify.app/) + +## Embedding a dashboard + +To begin, please review [the documentation](https://docs.rilldata.com/integrate/embedding) on embedding Rill Dashsboards via iframes and the various requirements after you have deployed your project to Rill Cloud. + + +We will continue assuming that you've: + +1. Reviewed and understand how the authenticated iframe URL are generated and passed to the frontend application, +2. Created your service token via: +```bash +rill service create +``` +3. Reviewed the documentation for available parameters + +### Using the demo dashboard + +We will be using the [demo dashboard](https://ui.rilldata.com/demo/rill-openrtb-prog-ad) that contains three dashboards. However, you will need to use your own dashboard as the token you created is specifically for your organization. Note the dashboard ID will be used, not the title. + + +- Spend vs Request Canvas Dashboard (spend-request-canvas-dashboard) +- Progammatic Ads Auction (auction) +- Programmtic Ads Bids (bids) + +To test whether we are able to generate a iframe URL, please run the following from the CLI. Please replace the `org-name`, `project-name`, `rill-svc-token`, `dashboard-name` and your `user-email`. +```bash +curl -X POST --location 'https://admin.rilldata.com/v1/organizations//projects//iframe' \ +--header 'Content-Type: application/json' \ +--header 'Authorization: Bearer ' \ +--data-raw '{ +"resource": "", +"user_email":"" +}' +``` +Your results should provide you a iframeSrc URL. Open this in your browser and see if you are able to view and interact with your dashboard. This should work exactly like the dashboard page within Rill Cloud. + +``` +"iframeSrc":"https://ui.rilldata.com/-/embed?access_token=... +``` + +Note, if you are trying to embed a dashboard that has navigation, or is a canvas dashboard, you will need to add a few components into `--data-raw`. + +**Navigation Enabled:** +```bash +--data-raw '{ +"resource": "bids", + "navigation": true, +"user_email":"email@domain.com" +``` + + +**Canvas Dashboard:** +```bash +--data-raw '{ +"resource": "canvas", +"kind": "rill.runtime.v1.Dashboard", +"user_email":"email@domain.com" +}' +``` + +Once you can confirm that all the dashboards are working as designed, you can embed these into your site. However, note that there is a ttl_seconds paramter that is default to 86400 seconds, 24 hours. This will keep the iframe URL alive for only 1 day. Therefore, creating a static site would require you to manually create this URL and update the site accordingly. + +Please refer to the examples in the documents for different ways to generate the iframe URL. +https://docs.rilldata.com/integrate/embedding#backend-build-an-iframe-url + + +### Sample JavaScript Code: [Click me for source!](https://github.com/rilldata/rill-embedding-example/blob/main/pages/api/iframe.js) +Taken from our example repository, you can create a js file that retrieves the iframe URL. Note that the service token will be retrieved from an environmental variable, but all the other components are defined before fetching the URL. + +```js +// Get the secret Rill service token from an environment variable. +const rillServiceToken = process.env.RILL_SERVICE_TOKEN; + +// Configure the dashboard to request an iframe URL for. +// Note that the organization must be the same as the one the service token is associated with. +const rillOrg = "demo"; +const rillProject = "rill-openrtb-prog-ads"; +const rillDashboard = "bids"; + +// This is a serverless function that makes an authenticated request to the Rill API to get an iframe URL for a dashboard. +// The iframe URL is then returned to the client. +// Iframe URLs must be requested from the backend to prevent exposing the Rill service token to the browser. +export default async function handler(req, res) { + try { + const url = `https://admin.rilldata.com/v1/organizations/${rillOrg}/projects/${rillProject}/iframe`; + const response = await fetch(url, { + method: 'POST', + headers: { + 'Content-Type': 'application/json', + Authorization: `Bearer ${rillServiceToken}`, + }, + body: JSON.stringify({ + resource: rillDashboard, + // You can pass additional parameters for row-level security policies here. + // For details, see: https://docs.rilldata.com/integrate/embedding + }), + }); + const data = await response.json(); + if (response.ok) { + res.json(data); + } else { + throw new Error(data.message); + } + } catch (error) { + res.status(500).json({ error: error.message }); + } +} +``` + diff --git a/docs/docs/tutorials/other/trial-check.md b/docs/docs/tutorials/other/trial-check.md index 0d0e16fb480..eb84d27fb8c 100644 --- a/docs/docs/tutorials/other/trial-check.md +++ b/docs/docs/tutorials/other/trial-check.md @@ -1,7 +1,16 @@ --- -title: "How do I check my trial information" -sidebar_label: "How to: Check Trial Information" +title: "How do I check my trial information?" +sidebar_label: "How to Check my Trial Information" hide_table_of_contents: false tags: - CLI ---- \ No newline at end of file +--- + +import ComingSoon from '@site/src/components/ComingSoon'; + + + + +
+Waiting for Self serve billing +
diff --git a/docs/docs/tutorials/other/yaml-vs-ui.md b/docs/docs/tutorials/other/yaml-vs-ui.md new file mode 100644 index 00000000000..773fef2c985 --- /dev/null +++ b/docs/docs/tutorials/other/yaml-vs-ui.md @@ -0,0 +1,6 @@ +--- +title: "BI-as-Code vs. UI" +sidebar_label: "BI-as-Code (YAML) vs. UI" +hide_table_of_contents: false +--- + diff --git a/docs/docs/tutorials/rill_advanced_features/advanced_developer/advanced-modeling.md b/docs/docs/tutorials/rill_advanced_features/advanced_developer/1-advanced-modeling.md similarity index 100% rename from docs/docs/tutorials/rill_advanced_features/advanced_developer/advanced-modeling.md rename to docs/docs/tutorials/rill_advanced_features/advanced_developer/1-advanced-modeling.md diff --git a/docs/docs/tutorials/rill_advanced_features/advanced_developer/advanced-dashboard.md b/docs/docs/tutorials/rill_advanced_features/advanced_developer/2-advanced-dashboard.md similarity index 61% rename from docs/docs/tutorials/rill_advanced_features/advanced_developer/advanced-dashboard.md rename to docs/docs/tutorials/rill_advanced_features/advanced_developer/2-advanced-dashboard.md index ae2c1e795d8..9f931808371 100644 --- a/docs/docs/tutorials/rill_advanced_features/advanced_developer/advanced-dashboard.md +++ b/docs/docs/tutorials/rill_advanced_features/advanced_developer/2-advanced-dashboard.md @@ -7,10 +7,10 @@ sidebar_position: 16 ## Let's make a new dashboard -As we have learned in the previous course, we will need to setup the dashboard based on the new column names. -Let's create a new dashboard via the UI. It should be named `dashboard_1.yaml`. Let's copy the contents from our old dashboard and make some changes. +As we have learned in the previous course, we will need to set up the metrics-view based on the new column names. +Let's create a new metrics-view via the UI. It should be named `advanced_metrics-view.yaml`. Let's copy the contents from our old dashboard and make some changes. -First, we will want to change the `model` value to the new model name `advaned_commits___model` +First, we will want to change the `table` value to the new model name `advaned_commits___model` Add two new dimensions: `directory path` and `commit_msg`. @@ -31,66 +31,98 @@ It depends. Depending on the size of data, type of measure, and what you are caluclating, you can choose either. Sometimes it would be better if you are dealing with a lot of data to front load the calculation on the SQL level so your dashboards load faster. However, the way OLAP engines work (linke avg of avg article), you might get incorrect data by doing certain calculations in the SQL level. You'll have to test and see which works for you! ::: -After making the above changes, you should be able to view your new dashboard! - -![img](/img/tutorials/204/advanced-dashboard.png)
- Example Working Dashboard - - - + Example Working metrics-view ```yaml -# Dashboard YAML -# Reference documentation: https://docs.rilldata.com/reference/project-files/dashboards +# Metrics View YAML +# Reference documentation: https://docs.rilldata.com/reference/project-files/metrics_views +version: 1 type: metrics_view -title: "My Advanced Tutorial Project" -#table: example_table # Choose a table to underpin your dashboard -model: advanced_commits___model - -timeseries: author_date # Select an actual timestamp column (if any) from your table +table: advanced_commits___model # Choose a table to underpin your metrics +timeseries: author_date # Choose a timestamp column (if any) from your table dimensions: - column: directory_path label: "The directory" description: "The directory path" + name: directory_path - column: filename label: "The filename" description: "The name of the modified filename" + name: filename - column: author_name label: "The Author's Name" description: "The name of the author of the commit" + name: author_name - column: commit_msg label: "The commit message" description: "The commit description attached." + name: commit_msg measures: - expression: "SUM(total_line_changes)" label: "Total number of Lines changed" description: "the total number of lines changes, addition and deletion" + name: total_line_changes -- name: net_line_changes - expression: "SUM(net_line_changes)" +- expression: "SUM(net_line_changes)" label: "Net number of Lines changed" description: "the total net number of lines changes" + name: net_line_changes + - expression: "SUM(num_commits)" label: "Number of Commits" description: "The total number of commits" + name: num_commits - expression: "(SUM(deleted_lines)/(SUM(deleted_lines)+SUM(added_lines)))" label: "Code Deletion Percent %" description: "The percent of code deletion" format_preset: percentage ``` -
+### Create the dashboard + +Similarly to the Basics course, we can create an explore-dashboard on top of this metrics view by selecting `Create explore`. You're preview should look something like the below! + +![img](/img/tutorials/204/advanced-dashboard.png) + +Along with the dimensions and measures, you can define `theme:`, time zones, time ranges, and [security policies](https://docs.rilldata.com/manage/security). Feel free to test by uncommenting the parameters and seeing how it changes the explore dashboard. + +```yaml +# Explore YAML +# Reference documentation: https://docs.rilldata.com/reference/project-files/explores + +type: explore + +title: "advanced_metrics_view dashboard" +metrics_view: advanced_metrics_view + +dimensions: '*' +measures: '*' + +# theme: theme.yaml + +#time_ranges: +# - PT15M // Simplified syntax to specify only the range +# - PT1H +# - PT6H +# - P7D + +#time_zones: +# - America/New_York + +#security: +# access: "{{ .user.admin }} AND '{{ .user.domain }}' == 'rilldata.com'" +``` import DocsRating from '@site/src/components/DocsRating'; diff --git a/docs/docs/tutorials/rill_advanced_features/advanced_developer/update-rill-cloud.md b/docs/docs/tutorials/rill_advanced_features/advanced_developer/3-update-rill-cloud.md similarity index 84% rename from docs/docs/tutorials/rill_advanced_features/advanced_developer/update-rill-cloud.md rename to docs/docs/tutorials/rill_advanced_features/advanced_developer/3-update-rill-cloud.md index e7c41c08d2f..62410c89a1b 100644 --- a/docs/docs/tutorials/rill_advanced_features/advanced_developer/update-rill-cloud.md +++ b/docs/docs/tutorials/rill_advanced_features/advanced_developer/3-update-rill-cloud.md @@ -39,7 +39,8 @@ Your branch is up to date with 'origin/main'. Untracked files: (use "git add ..." to include in what will be committed) .DS_Store - dashboards/dashboard_1.yaml + explore-dashboards/advanced_metrics_view_explore.yaml + metrics/advanced_metrics_view.yaml models/advanced_commits___model.sql tmp/ @@ -49,20 +50,12 @@ nothing added to commit but untracked files present (use "git add" to track) Let's try again after adding the mentioned files. ``` -git add dashboards/dashboard_1.yaml +git add explore-dashboards/advanced_metrics_view_explore.yaml +git add metrics/advanced_metrics_view.yaml git add models/advanced_commits___model.sql git commit -m "Adding new files" [main 68b293e] Adding new files - 2 files changed, 91 insertions(+) - create mode 100644 dashboards/dashboard_1.yaml - create mode 100644 models/advanced_commits___model.sql - - Enumerating objects: 9, done. - Counting objects: 100% (9/9), done. - Delta compression using up to 12 threads - Compressing objects: 100% (6/6), done. - Writing objects: 100% (6/6), 1.57 KiB | 1.57 MiB/s, done. - Total 6 (delta 0), reused 2 (delta 0), pack-reused 0 + ... To https://github.com/royendo/my-rill-tutorial.git 59beefe..68b293e main -> main branch 'main' set up to track 'origin/main'. diff --git a/docs/docs/tutorials/rill_advanced_features/canvas_dashboards/getting-started.md b/docs/docs/tutorials/rill_advanced_features/canvas_dashboards/1-getting-started.md similarity index 82% rename from docs/docs/tutorials/rill_advanced_features/canvas_dashboards/getting-started.md rename to docs/docs/tutorials/rill_advanced_features/canvas_dashboards/1-getting-started.md index 84de634fc7f..a80a8a2b450 100644 --- a/docs/docs/tutorials/rill_advanced_features/canvas_dashboards/getting-started.md +++ b/docs/docs/tutorials/rill_advanced_features/canvas_dashboards/1-getting-started.md @@ -4,6 +4,9 @@ description: Creating Canvas Dashboards in Rill sidebar_label: "Getting Started" sidebar_position: 6 --- +:::note +Canvas Dashboards are still not released yet and is hidden behind a feature flag. Breaking changes may be pushed at any time and may cause your dashboards to no longer function. Please proceed with that in mind! +::: ## Creating Canvas Dashboards in Rill @@ -40,7 +43,8 @@ Under the + Add dropdown, select More to find the chart and custom dashboard com ![img](/img/tutorials/301/add-custom-dashboard.png) -Once you select either of these, a dedicated folder `components` and `canvasdashboards` will be created. +Once you select either of these, a dedicated folder `components` and `canvas-dashboards` will be created. + import DocsRating from '@site/src/components/DocsRating'; diff --git a/docs/docs/tutorials/rill_advanced_features/canvas_dashboards/2-template-charts.md b/docs/docs/tutorials/rill_advanced_features/canvas_dashboards/2-template-charts.md new file mode 100644 index 00000000000..41925defc82 --- /dev/null +++ b/docs/docs/tutorials/rill_advanced_features/canvas_dashboards/2-template-charts.md @@ -0,0 +1,66 @@ +--- +title: "Canvas Dashboards" +description: Using Template charts +sidebar_label: "Template Components" +sidebar_position: 10 +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +## Using Rill Custom Components Templates + +Rill provides a few templates for Rill-Authored charts. We will use the metrics_view, `advanced_metrics_view` to create a few custom components. For a more extensive list of examples, please see [our reference page](https://docs.rilldata.com/reference/project-files/components#Examples)! + +### Components: + + + + + +```yaml +# Component YAML +# Reference documentation: https://docs.rilldata.com/reference/project-files/components + +type: component + +kpi: + metrics_view: advanced_metrics_view + time_range: P1W + measure: net_line_changes #if name is defined + comparison_range: P1W + + ``` + + + + +```yaml +# Component YAML +# Reference documentation: https://docs.rilldata.com/reference/project-files/components + +type: component + +data: + metrics_sql: | + select + measure_2, + date_trunc('day', author_date) as date, + from advanced_metrics_view + where author_date > '2024-07-14' + +bar_chart: + x: date + y: measure_2 + ``` + + + + + + + +import DocsRating from '@site/src/components/DocsRating'; + +--- + \ No newline at end of file diff --git a/docs/docs/tutorials/rill_advanced_features/canvas_dashboards/vega-lite.md b/docs/docs/tutorials/rill_advanced_features/canvas_dashboards/3-vega-lite.md similarity index 89% rename from docs/docs/tutorials/rill_advanced_features/canvas_dashboards/vega-lite.md rename to docs/docs/tutorials/rill_advanced_features/canvas_dashboards/3-vega-lite.md index 10de93edef1..93f35a97760 100644 --- a/docs/docs/tutorials/rill_advanced_features/canvas_dashboards/vega-lite.md +++ b/docs/docs/tutorials/rill_advanced_features/canvas_dashboards/3-vega-lite.md @@ -18,7 +18,7 @@ Let's start by creating a chart for our Canvas dashboard. ![project-view](/img/tutorials/301/add-custom-dashboard.png) -### Creating a bar graph that calculates the top contributors to the Repository +### Creating a bar graph that calculates the top 5 contributors to the Repository Let's use the `advanced_commits__model` table to obtain our data. @@ -48,6 +48,7 @@ WITH commit_file_stats AS ( ) SELECT author_date, + cast(author_date as date) as date, author_name, directory_path, filename, @@ -69,7 +70,7 @@ GROUP BY --directory_path, filename, author_name, author_date ALL ORDER BY - directory_path DESC + author_date DESC ``` ::: @@ -88,10 +89,11 @@ data: author_name, sum(net_line_changes) as net_lines from advanced_commits___model - where author_date > '2024-07-21 00:00:00 Z' + where author_date > '2024-07-21 00:00:00' + {{ if .args.author }} AND author_name = '{{ .args.author }}' {{ end }} + group by author_name order by net_lines desc - limit 5 net_lines desc limit 5 ``` @@ -133,14 +135,16 @@ vega_lite: | ``` +If you are unfamiliar with creating vega-lite specs, you can use natural language to create components. Please see the following how-to for more information! [Using Natural Language to create Components](https://docs.rilldata.com/tutorials/other/custom-charts) + **Custom Component Complete** With everything completed, you should have a YAML file with the following contents and a custom component that looks something like the following. ![img](/img/tutorials/301/top-contributors.png) ```yaml -# Chart YAML -# Reference documentation: https://docs.rilldata.com/reference/project-files/charts +# Component YAML +# Reference documentation: https://docs.rilldata.com/reference/project-files/components type: component diff --git a/docs/docs/tutorials/rill_advanced_features/canvas_dashboards/vega-lite2.md b/docs/docs/tutorials/rill_advanced_features/canvas_dashboards/4-vega-lite2.md similarity index 99% rename from docs/docs/tutorials/rill_advanced_features/canvas_dashboards/vega-lite2.md rename to docs/docs/tutorials/rill_advanced_features/canvas_dashboards/4-vega-lite2.md index 9f724989e30..c9d3f066a97 100644 --- a/docs/docs/tutorials/rill_advanced_features/canvas_dashboards/vega-lite2.md +++ b/docs/docs/tutorials/rill_advanced_features/canvas_dashboards/4-vega-lite2.md @@ -144,8 +144,8 @@ With both components ready, your dashboard should look something like this: ![img](/img/tutorials/301/normalized.png) ```yaml -# Chart YAML -# Reference documentation: https://docs.rilldata.com/reference/project-files/charts +# Component YAML +# Reference documentation: https://docs.rilldata.com/reference/project-files/components type: component diff --git a/docs/docs/tutorials/rill_advanced_features/canvas_dashboards/canvas-dashboards.md b/docs/docs/tutorials/rill_advanced_features/canvas_dashboards/5-canvas-dashboards.md similarity index 95% rename from docs/docs/tutorials/rill_advanced_features/canvas_dashboards/canvas-dashboards.md rename to docs/docs/tutorials/rill_advanced_features/canvas_dashboards/5-canvas-dashboards.md index 0f4fb89c335..9c5f20d36c0 100644 --- a/docs/docs/tutorials/rill_advanced_features/canvas_dashboards/canvas-dashboards.md +++ b/docs/docs/tutorials/rill_advanced_features/canvas_dashboards/5-canvas-dashboards.md @@ -42,7 +42,7 @@ Finally, using some of the Rill template KPI charts, we can finish off the Canva
If you had any issues building the Canvas dashboard, please refer to the YAML below. ```yaml -type: dashboard +type: canvas columns: 13 gap: 2 @@ -101,6 +101,8 @@ items: These are just two simple custom graphs that can be built using Vega Lite. Please refer to Vega Lite [documentation](https://vega.github.io/vega-lite/docs/) and [examples](https://vega.github.io/vega-lite/examples/) for further inspiration on how you can build your very own Canvas dashboard. +The current canvas dashboard is nice but it lacks the ability to filter based on different dimenions. Let's add that. + import DocsRating from '@site/src/components/DocsRating'; --- diff --git a/docs/docs/tutorials/rill_advanced_features/canvas_dashboards/6-filters.md b/docs/docs/tutorials/rill_advanced_features/canvas_dashboards/6-filters.md new file mode 100644 index 00000000000..062218b1602 --- /dev/null +++ b/docs/docs/tutorials/rill_advanced_features/canvas_dashboards/6-filters.md @@ -0,0 +1,104 @@ +--- +title: "Filtering the Canvas Dashboards" +description: Canvas Dashboard Filtering +sidebar_label: "Filtering the Canvas Dashboard" +sidebar_position: 9 +--- + +Filtering the dashboard can be done via components. For information on `input`, `output` and `variables`, please see [our documentation] (). + + +### Creating a Selector Component +For our example, we will create a selector component that filters on the distinct values in the `author_name` column in our `commits___model` model. + +A few things to note, since this component will be used as an `input` into other components, we define the `output` parameter. There's a hard limit of 10,000 rows. + +```yaml +# Component YAML +# Reference documentation: https://docs.rilldata.com/reference/project-files/components + +type: component + +output: + name: author_name + type: string + +data: + sql: SELECT DISTINCT author_name FROM commits___model limit 10000 + +select: + valueField: "author_name" + placeholder: "Author's Name" + ``` + +We can now add the component to our dashboard. Selecting you'll see the distinct authors listed. However, selecting an author doesnt change the dashboards, why? This is because we haven't set the input in the component's YAML. + +![img](/img/tutorials/301/selector.png) + + +### Defining the input from the Selector +Using the stacked bar chart, let's define the input from the selector's output parameter. + + +```yaml +... +input: + - name: author_name + type: string + value: "" + + +data: + sql: | + select + author_name, + sum(added_lines) as added_lines, + sum(deleted_lines) as deleted_lines, + from advanced_commits___model + where author_date > '2024-07-21 00:00:00 Z' + {{ if .args.author_name }} AND author_name = '{{ .args.author_name }}' {{ end }} + group by author_name +... +``` + +Let's take a second to understand the SQL. We are checking that if the author_name argument exists, we wil append `AND author_name...` to the SQL query. As we have defined value as "", this author_name is not being used. + +![img](/img/tutorials/301/component-filter-on.png) + +However, you can see that if we add an actual author to this key-pair the chart changes. + +![img](/img/tutorials/301/component-filter-off.png) + +Let's change it back to the original empty value as we do not want to default on the single author view. Now let's navigate back to the canvas dashboard and add the variables to be used. + +```yaml +type: dashboard +columns: 13 +gap: 2 + +variables: + - name: author + type: string + value: "" + + - name: timegrain + type: string + value: "P1W" #you can also define a default value for the variable + +items: + - component: + markdown: + content: "ClickHouse Repo Overview" +... +``` + +Now upon selection of the author dropdown, we can see the stacked bar chart change. You can make the same changes to the Highest Contributor chart, as well. See below for a sample of a completed canvas dashboard. + + +![img](/img/tutorials/301/canvas-dashboard-filters.png) + + +import DocsRating from '@site/src/components/DocsRating'; + +--- + \ No newline at end of file diff --git a/docs/docs/tutorials/rill_advanced_features/canvas_dashboards/template-charts.md b/docs/docs/tutorials/rill_advanced_features/canvas_dashboards/template-charts.md deleted file mode 100644 index 79ff5c8e2bc..00000000000 --- a/docs/docs/tutorials/rill_advanced_features/canvas_dashboards/template-charts.md +++ /dev/null @@ -1,185 +0,0 @@ ---- -title: "Canvas Dashboards" -description: Using Template charts -sidebar_label: "Template Components" -sidebar_position: 10 ---- - -import Tabs from '@theme/Tabs'; -import TabItem from '@theme/TabItem'; - -## Using Rill Custom Components Templates - -Rill provides a few templates for Rill-Authored charts. We will use the dashboard, `dashboard_1` to create a few custom dashboards. For a more extensive list of examples, please see [our reference page](https://docs.rilldata.com/reference/project-files/components#Examples)! - -### Charts: - - - - - -```yaml -# Chart YAML -# Reference documentation: https://docs.rilldata.com/reference/project-files/charts - -type: component - -data: - metrics_sql: | - select - measure_2, - date_trunc('day', author_date) as date - from dashboard_1 - where author_date > '2024-07-14 00:00:00 Z' - -bar_chart: - x: date - y: measure_2 - - - ``` - - - - - - ```yaml -# Chart YAML -# Reference documentation: https://docs.rilldata.com/reference/project-files/charts - -type: component - -data: - metrics_sql: | - select - measure_0, - date_trunc('day', author_date) as date - from dashboard_1 - where author_date > '2024-07-14 00:00:00 Z' - -line_chart: - x: date - y: measure_0 - - ``` - - - - ```yaml -# Chart YAML -# Reference documentation: https://docs.rilldata.com/reference/project-files/charts - -type: component - - - ``` - - - - - ```yaml -# Chart YAML -# Reference documentation: https://docs.rilldata.com/reference/project-files/charts - -type: component - - - ``` - - - - - - ```yaml -# Chart YAML -# Reference documentation: https://docs.rilldata.com/reference/project-files/charts - -type: component - - - ``` - - - - - -### Others: - - - - -```yaml -# Chart YAML -# Reference documentation: https://docs.rilldata.com/reference/project-files/charts - -type: component - -kpi: - metric_view: dashboard_1 - time_range: P1W - measure: net_line_changes #if name parameter is defined on measure - comparison_range: P1W - ``` - - - - ```yaml -# Chart YAML -# Reference documentation: https://docs.rilldata.com/reference/project-files/charts - -type: component - - - ``` - - - - ```yaml -# Chart YAML -# Reference documentation: https://docs.rilldata.com/reference/project-files/charts - -type: component - - - ``` - - - - ```yaml -# Chart YAML -# Reference documentation: https://docs.rilldata.com/reference/project-files/charts - -type: component - -table: - measures: - - net_line_changes - metric_view: "dashboard_1" - time_range: "P3M" - #comparison_range: "P3M" - - row_dimensions: - - author_name - #col_dimensions: - # - filename - ``` - - - - ```yaml -# Chart YAML -# Reference documentation: https://docs.rilldata.com/reference/project-files/charts - -type: component - - - ``` - - - - - -import DocsRating from '@site/src/components/DocsRating'; - ---- - \ No newline at end of file diff --git a/docs/docs/tutorials/rill_advanced_features/custom_api/create-api.md b/docs/docs/tutorials/rill_advanced_features/custom_api/create-api.md index 84eb87b1661..3e81eca7430 100644 --- a/docs/docs/tutorials/rill_advanced_features/custom_api/create-api.md +++ b/docs/docs/tutorials/rill_advanced_features/custom_api/create-api.md @@ -18,11 +18,12 @@ Now that the access has been created, we can make the actual custom APIs. For th For the `SQL_api`, we will retreive the author's with the most net line changes from the model. ```sql +sql: | SELECT author_name, sum(net_line_changes) as net_line_changes, FROM advanced_commits___model - where author_date > '2024-07-01 00:00:00 Z' + where author_date > '2024-07-01 00:00:00' group by author_name order by net_line_changes DESC limit 10 @@ -30,13 +31,14 @@ For the `SQL_api`, we will retreive the author's with the most net line changes -For `metrics_view_api`, we will use `dashboard_1` and run the following SQL query: +For `metrics_view_api`, we will use `advanced_metrics_view` and run the following SQL query: ```sql +metrics_sql: | SELECT author_name, net_line_changes - FROM dashboard_1 - where author_date > '2024-07-01 00:00:00 Z' + FROM advanced_metrics_view + where author_date > '2024-07-01 00:00:00' order by net_line_changes DESC limit 10 ``` @@ -51,7 +53,7 @@ Both of these SQL queries will return the same data, why? As discussed when creating the measure, we defined the name of the measure so we can use the name in the SQL query. You can test the API's output with the following syntax, `http://localhost:9009/v1/instances/default/api/` -Once you have confirmed that the local running APIs work as expected, we can select update [or push changes to GitHub repository](../../rill_advanced_features/advanced_developer/update-rill-cloud.md) to push the changes to your project. +Once you have confirmed that the local running APIs work as expected, we can select update [or push changes to GitHub repository](https://docs.rilldata.com/tutorials/rill_advanced_features/advanced_developer/update-rill-cloud) to push the changes to your project. :::note If the `update` button is not available on your current UI, you can find this on the dashboard page! diff --git a/docs/docs/tutorials/rill_advanced_features/custom_api/getting-started.md b/docs/docs/tutorials/rill_advanced_features/custom_api/getting-started.md index 6711bc8ff6e..61c78b00512 100644 --- a/docs/docs/tutorials/rill_advanced_features/custom_api/getting-started.md +++ b/docs/docs/tutorials/rill_advanced_features/custom_api/getting-started.md @@ -37,19 +37,18 @@ Created service "my-api-service" in org "Rill_Learn". Access token: rill_svc_ ``` -Once this is created, you can use the service access token to create a user token. Using [dashboard access policies](https://docs.rilldata.com/manage/security), we can add the following to our [dashboard_1.yaml](../advanced_developer/advanced-dashboard.md) file. +Once this is created, you can use the service access token to create a user token. Using [dashboard access policies](https://docs.rilldata.com/manage/security), we can add the following to our [advanced_metrics_view_explore.yaml](https://docs.rilldata.com/tutorials/advanced_developer/advanced-dashboard.md) file. ```yaml security: access: "{{ .user.admin }} AND '{{ .user.domain }}' == 'rilldata.com'" ``` -This access policy gives access to the dashboard for admins who's email domain is rilldata.com. For the user token below, please select an email for a user that is [a viewer](../../administration/user-management.md) to the project, `my-rill-tutorial`. +This access policy gives access to the dashboard for admins who's email domain is rilldata.com. For the user token below, please select an email for a user that is [a viewer](https://docs.rilldata.com/tutorials/administration/user-management) to the project, `my-rill-tutorial`. ```bash -curl -X POST https://admin.rilldata.com/v1/organizations/Rill_Learn/projects/my-rill-tutorial/credentials \ --H "Authorization: Bearer rill_svc_" ---data-raw '{ - "user_email":"" +curl -X POST https://admin.rilldata.com/v1/organizations//projects//credentials \ +-H "Authorization: Bearer rill_svc_" --data-raw '{ + "user_email": "email@domaim.com" }' ``` diff --git a/docs/docs/tutorials/rill_advanced_features/custom_api/test-api.md b/docs/docs/tutorials/rill_advanced_features/custom_api/test-api.md index 3dc3b6e5a4e..eff3e0f50fd 100644 --- a/docs/docs/tutorials/rill_advanced_features/custom_api/test-api.md +++ b/docs/docs/tutorials/rill_advanced_features/custom_api/test-api.md @@ -32,14 +32,18 @@ Based on the access policy, we expect that the API does not work as the user doe Dashboard API: (metrics_view_api) ```bash +curl https://admin.rilldata.com/v1/orgs/Rill_Learn/projects/my-rill-tutorial/runtime/api/metrics_view_api \ ... + {"error":"action not allowed"} ``` Underlying SQL table: (SQL_api) ```bash +curl https://admin.rilldata.com/v1/orgs/Rill_Learn/projects/my-rill-tutorial/runtime/api/SQL_api \ ... + [{"author_name":"avogar","net_line_changes":16331},{"author_name":"Sema Checherinda","net_line_changes":8118},{"author_name":"Blargian","net_line_changes":5629},{"author_name":"Max K","net_line_changes":1904},{"author_name":"robot-clickhouse","net_line_changes":1899},{"author_name":"Raúl Marín","net_line_changes":1434},{"author_name":"János Benjamin Antal","net_line_changes":1168},{"author_name":"yariks5s","net_line_changes":1078},{"author_name":"Nikita Taranov","net_line_changes":1035},{"author_name":"Antonio Andelic","net_line_changes":1032}]% ``` ### Developing APIs -For further information about our custom APIs, I recommened referring to the following [documentation](https://docs.rilldata.com/integrate/custom-api) and [references](https://docs.rilldata.com/reference/project-files/apis). \ No newline at end of file +For further information about our custom APIs, please refer to the following [documentation](https://docs.rilldata.com/integrate/custom-api) and [references](https://docs.rilldata.com/reference/project-files/apis). \ No newline at end of file diff --git a/docs/docs/tutorials/rill_advanced_features/incremental_models/1-introduction.md b/docs/docs/tutorials/rill_advanced_features/incremental_models/1-introduction.md new file mode 100644 index 00000000000..2765509bad9 --- /dev/null +++ b/docs/docs/tutorials/rill_advanced_features/incremental_models/1-introduction.md @@ -0,0 +1,46 @@ +--- +title: "Splits and Increments" +description: "Whats the differences" +sidebar_label: "Splits and Incremental Models" +sidebar_position: 1 +--- + +In order to help with data ingestion into Rill, we will introduce the concepts of [splits](https://docs.rilldata.com/build/advancedmodels/splits) and [incremental models](https://docs.rilldata.com/build/advancedmodels/incremental) Before diving into our ClickHouse project, let's understand what each of these are used for and do. + +## Incremental Model + +An incremental model is defined using the following key pair. + +```yaml +incremental: true +``` +Once this is enabled, this allows Rill to configure the model YAML as an incrementing model. +In some following examples, we will use both a time based incremental and glob based increments. + +## Splits + +Splits in models are enabled by defining the split parameter as seen below: + +```yaml +splits: + sql/glob: some splits definition +``` + +Depending on your data, this can be defined as a `SQL:` statement or a `glob:` pattern. Once configured, Rill will try to split your existing data into smaller subcategories which allows you to refresh specific sections only instead of reingesting the whole dataset. (only when incremental is enabled) + +By running the following command, you can see all the available splits and run a refresh command on a specific key or keys. +```bash +rill project split +``` + +```bash +rill project refresh --model --split +``` + + +Let's look at a few simple examples before diving into our ClickHouse project. + +import DocsRating from '@site/src/components/DocsRating'; + +--- + \ No newline at end of file diff --git a/docs/docs/tutorials/rill_advanced_features/incremental_models/2-cloud-storage-splits.md b/docs/docs/tutorials/rill_advanced_features/incremental_models/2-cloud-storage-splits.md new file mode 100644 index 00000000000..b8269f645fb --- /dev/null +++ b/docs/docs/tutorials/rill_advanced_features/incremental_models/2-cloud-storage-splits.md @@ -0,0 +1,112 @@ +--- +title: "Splits with Cloud Storage" +description: "Getting Started with Splits" +sidebar_label: "Cloud Storage: Splits and Incremental Models" +sidebar_position: 12 +--- + +Now that we understand what [Incremental Models](https://docs.rilldata.com/build/advancedmodels/incremental) and [splits](https://docs.rilldata.com/build/advancedmodels/splits) are, let's try to apply them to our project. + +Since our ClickHouse data is hosted in GCS/S3, we will be using glob based splits, instead of the example sql select statement. + +### Let's create a basic split model. +In the previous courses, we used a GCS connection to import ClickHouse's repository commit history. Let's go ahead and assume we are using the same folder structure. + +``` +#gs://rilldata-public/github-analytics/Clickhouse/YYYY/MM/filename_YYYY_MM.parquet + +gs://rilldata-public/github-analytics/Clickhouse/*/*/commits_.parquet +gs://rilldata-public/github-analytics/Clickhouse/*/*/modified_files_*.parquet +``` +1. Create a YAML file: `splits-tutorial.yaml` + +2. Use `glob:` resolver to load files from GCS +```yaml +type: model + +splits: + glob: + connector: gcs + path: gs://rilldata-public/github-analytics/Clickhouse/*/*/commits_*.parquet + +``` +3. Set the SQL statement to user the URI. +```yaml +sql: SELECT * FROM read_parquet('{{ .split.uri }}') +``` + +Once you save the file, Rill will start to ingest all the splits from GCS. This may take a few minutes. You can see the progress of the ingestion from the CLI. + +### Viewing errors in splits + +If you see any errors in the UI regarding your splits, you may need to check the status by selecting "view splits" + +![img](/img/tutorials/302/splits-refresh-ui.png) + + +Or, you can check this via the CLI running: +```bash +rill project splits --local +``` + +Once completed you should see the following: + +![img](/img/tutorials/302/splits.png) + +### Refreshing Splits + +Let's say a specific split in your model had some formatting issues. After fixing the data, you can either select `Refresh Split` in the UI or find the split ID by running `rill project splits -- --local`. Once found, you can run the following command that will only refresh the specific split, instead of the whole model. + +```bash +rill project refresh --model --split +``` + + +## What is Incremental Modeling? +Once splits are set up, you can use incremental modeling to load only new data when refreshing a dataset. This becomes important when your data is large and it does not make sense to reload all the data when trying to ingest new data. + +### Let's create an Incremental model for our commits and modified files sources. + +0. Create a file CH_incremental_commits.yaml and CH_incremental_modified_files.yal + +1. After copying the previous YAML contents, set `incremental` to true (For modified_files, make sure you change the file name!) + +2. You can manually setup a `splits_watermark` but since our data is using the `glob` key, it is automatically set to the `updated_on` field. + +3. Let's set up a `refresh` based on `cron` that runs daily at 8AM UTC. +``` +refresh: + cron: "0 8 * * *" +``` + +Once Rill ingests the data, your UI should look something like this: +![img](/img/tutorials/302/incremental.png) + + +Your YAML should look like the following: + +```yaml +type: model + +incremental: true +refresh: + cron: "0 8 * * *" + +splits: + glob: + connector: gcs + path: gs://rilldata-public/github-analytics/Clickhouse/*/*/commits_*.parquet #modified_files_*.parquet + +sql: SELECT * FROM read_parquet('{{ .split.uri }}') +``` + +You now have a working incremental model that refreshed new data based on the `updated_on` key at 8AM UTC everyday. Along with writing to the default OLAP engine, DuckDB, we have also added some features to use staging tables for connectors that do not have direct read/write capabilities. + + +Once this is created + + +import DocsRating from '@site/src/components/DocsRating'; + +--- + \ No newline at end of file diff --git a/docs/docs/tutorials/rill_advanced_features/incremental_models/cloud-storage-splits.md b/docs/docs/tutorials/rill_advanced_features/incremental_models/cloud-storage-splits.md deleted file mode 100644 index 7d1afa9ddf0..00000000000 --- a/docs/docs/tutorials/rill_advanced_features/incremental_models/cloud-storage-splits.md +++ /dev/null @@ -1,102 +0,0 @@ ---- -title: "Splits with Cloud Storage" -description: "Getting Started with Splits" -sidebar_label: "Cloud Storage: Splits and Incremental Models" -sidebar_position: 12 ---- - -Another advanced concept within Rill is using [Incremental Models](https://docs.rilldata.com/build/advancedmodels/incremental). To understand incremental models, we will also need to discuss [splits](https://docs.rilldata.com/build/advancedmodels/splits). - - -## Understanding Splits in Models - -Here’s how it works at a high level: - -- **Split Definition**: Each row from the result set becomes one "split". The model processes each split separately. -- **Execution Strategy**: - - **First Split**: Runs without incremental processing. - - **Subsequent Splits**: Run incrementally, following the output connector's `incremental_strategy` (either append or merge for SQL connectors). - -### Let's create a basic split model. -In the previous courses, we used a GCS connection to import ClickHouse's repository commit history. In this guide, we will use S3. The format of the files are the same, you just need to change `gs` to `s3` - - -``` -s3://rilldata-public/github-analytics/Clickhouse/*/*/commits_*.parquet -``` -1. Create a YAML file: `S3-splits-tutorial.yaml` - -2. Use `glob:` resolver to load files from S3 -```yaml -splits: - glob: - connector: s3 - path: s3://rilldata-public/github-analytics/Clickhouse/*/*/commits_*.parquet - -``` -3. Set the SQL statement to user the URI. -```yaml -sql: SELECT * FROM read_parquet('{{ .split.uri }}') -``` - -### Handling errors in splits -If you see any errors in the UI regarding split, you may need to check the status. You can do this via the CLI running: -```bash -rill project splits -- --local -``` - -Once completed you should see the following: - -![img](/img/tutorials/302/splits.png) - -### Refreshing Splits - -Let's say a specific split in your model had some formatting issues. After fixing the data, you would need to find the key for the split and run `rill project splits -- --local`. Once found, you can run the following command that will only refresh the specific split, instead of the whole model. - -```bash -rill project refresh --model --split -``` - - -## What is Incremental Modeling? -Once splits are setup, you can use incremental modeling to load only new data when refreshing a dataset. This becomes important when your data is large and it does not make sense to reload all the data when trying to ingest new data. - -### Let's modify the split model to add incremental modeling. - -1. Set `incremental` to true - -2. You can manually setup a `splits_watermark` but since our data is using the `glob` key, it is automatically set to the `updated_on` field. - -3. Let's set up a `refresh` based on `cron` that runs daily at 8AM UTC. -``` -refresh: - cron: "0 8 * * *" -``` - -Once Rill ingests the data, your UI should looks something like this: -![img](/img/tutorials/302/incremental.png) - -Your YAML should look like the following: - -```yaml -type: model - -incremental: true -refresh: - cron: 0 8 * * * - -splits: - glob: - connector: s3 - path: s3://rilldata-public/github-analytics/Clickhouse/*/*/commits_*.parquet - -sql: SELECT * FROM read_parquet('{{ .split.uri }}') - -``` - -You now have a working incremental model that refreshed new data based on the `updated_on` key at 8AM UTC everyday. Along with writing to the default OLAP engine, DuckDB, we have also added some features to use staging tables for connectors that do not have direct read/write capabilities. - -import DocsRating from '@site/src/components/DocsRating'; - ---- - \ No newline at end of file diff --git a/docs/docs/tutorials/rill_advanced_features/incremental_models/simple-examples/incremental_now.md b/docs/docs/tutorials/rill_advanced_features/incremental_models/simple-examples/incremental_now.md new file mode 100644 index 00000000000..56a54a18dc8 --- /dev/null +++ b/docs/docs/tutorials/rill_advanced_features/incremental_models/simple-examples/incremental_now.md @@ -0,0 +1,85 @@ +--- +title: "Incremental Models" +description: "Start with basics" +sidebar_label: "Basic Incremental Models" +sidebar_position: 1 +--- +Before enabling incremental on the model, let's take a look at the following model YAML file, now.yaml. +### Sample YAML +```yaml +# This model outputs the current time every time it is refreshed. +type: model +refresh: + cron: 0 0 * * * +sql: SELECT now() AS inserted_on +``` + + + +To understand what this is doing, let's go ahead and select the Refresh button. This button performs the same command as the below in the CLI. + +```bash +rill project refresh --model now --local +``` +:::note +Since we're using Rill Developer, we will need to add the `--local` flag to the refresh commands or else this will refresh the project on Rill Cloud! +::: + +After the model refreshes, you should see the inserted_on value change. +![img](/img/tutorials/302/now.png) +>update this next week + + +### Enable Increments on our Model + +As mentioned previously, the `incremental: true` tells Rill that this model is incremental. You will see that the UI changes slightly when this is enabled. Not only will you be able to full refresh, but also incrementally refresh. + +```yaml +type: model +refresh: + cron: 0 0 * * * +sql: SELECT now() AS inserted_on +incremental: true +``` + +After adding the following, let's run the same command. What's different? + +![img](/img/tutorials/302/now-incremental.png) +>>update this + +Instead of overwriting the same row, we are now appending the new values of now() into the table. With the refresh enabled to run at midnight on every night, you should see the amount of rows increase each day at midnight UTC. Next, let's take a moment to review `states:`. + + +### States in Models + +Lastly, we can add a `state:` key that allows us to manually define some sort of state to use as our incrementing key. + + +```yaml +#filename: incremental_state.yaml +type: model +refresh: + cron: 0 0 * * * +sql: SELECT {{ if incremental }} {{ .state.max_val }} + 1 {{ else }} 0 {{ end}} AS val, now() AS inserted_on +state: + sql: SELECT MAX(val) as max_val FROM incremental_state +incremental: true +``` +:::note +In more realistic cases, we could select the MAX(time_stamp) which will grab the latest time_stamp that your current model includes. Then, based on this it would incrementally refresh your model to only insert the new data. This would continue to occur. However, keep in mind that any 'old' data that gets added outside of Rill would not be detected. +::: +Along with the inserted_on column, we are also creating a val column that defaults to 0. Then on each run, if incremental, increases this value. The state retrieves the max value of 'val' as max_val. + +```SQL +SELECT +{{ if incremental }} + {{ .state.max_val }} + 1 {{ else }} 0 +{{ end}} +``` + +![img](/img/tutorials/302/now-state.png) + +import DocsRating from '@site/src/components/DocsRating'; + +--- + \ No newline at end of file diff --git a/docs/docs/tutorials/rill_advanced_features/incremental_models/simple-examples/split_now.md b/docs/docs/tutorials/rill_advanced_features/incremental_models/simple-examples/split_now.md new file mode 100644 index 00000000000..63b9e3086ac --- /dev/null +++ b/docs/docs/tutorials/rill_advanced_features/incremental_models/simple-examples/split_now.md @@ -0,0 +1,107 @@ +--- +title: "Split Models" +description: "Start with basics" +sidebar_label: "Basic Split Models" +sidebar_position: 2 +--- + +As mentioned, we can define a split by adding the `splits:` key with some defining parameters. Splits are a special case of incremental model states. Let's look at the following example. + +```yaml +type: model + +splits: + sql: SELECT range AS num FROM range(0,10) +sql: SELECT {{ .split.num }} AS num, now() AS inserted_on +``` + +In this simple example, we set up 10 splits [range(0,10)] that have a single row with the same now() function as defined earlier. To confirm this we can run the following: + +```bash +rill project splits splits_range --local + KEY (10) DATA EXECUTED ON ELAPSED ERROR + ---------------------------------- ----------- ---------------------- --------- ------- + ff7416f774dfb086006d0b4696c214e1 {"num":0} 2024-09-18T02:32:01Z 145ms + 69401118e166742864f35f1a77ffe07d {"num":1} 2024-09-18T02:32:01Z 0s + 555ef019f87b5a57ec7b057476fe9d38 {"num":2} 2024-09-18T02:32:01Z 0s + 09a5530e62c87e41848a02680f4422d4 {"num":3} 2024-09-18T02:32:01Z 1ms + ecc2e2f9deb13509547ebcc2e5c55116 {"num":4} 2024-09-18T02:32:01Z 1ms + 1e3ddd76525fefd5ac9989d9b6c4727e {"num":5} 2024-09-18T02:32:01Z 1ms + 25684ad5ef8f1965b597edeeb8004afa {"num":6} 2024-09-18T02:32:01Z 0s + 8142d250d75d0a20883333c00c3962d5 {"num":7} 2024-09-18T02:32:01Z 0s + 0d6962a0746cb896ce87250808a50051 {"num":8} 2024-09-18T02:32:01Z 1ms + 727d91a916260837579d5e42ad696dd9 {"num":9} 2024-09-18T02:32:01Z 0s + ``` + + If you try to refresh a single split, you'll receive the following error: + + ```bash +rill project refresh --model splits_range --split ff7416f774dfb086006d0b4696c214e1 --local +Error: can't refresh splits on model "splits_range" because it is not incremental +``` + +### Incremental Split Model +```yaml +type: model + +splits: + sql: SELECT range AS num FROM range(0,10) +sql: SELECT {{ .split.num }} AS num, now() AS inserted_on +incremental: true + +output: + incremental_strategy: merge + unique_key: [num] +``` + +Similarily to the above, let's run `rill project splits ` to get the key_ids. + +```bash +rill project splits splits_range_incremental --local + KEY (10) DATA EXECUTED ON ELAPSED ERROR + ---------------------------------- ----------- ---------------------- --------- ------- + ff7416f774dfb086006d0b4696c214e1 {"num":0} 2024-09-18T03:17:16Z 103ms + 69401118e166742864f35f1a77ffe07d {"num":1} 2024-09-18T03:17:16Z 1ms + 555ef019f87b5a57ec7b057476fe9d38 {"num":2} 2024-09-18T03:17:16Z 0s + 09a5530e62c87e41848a02680f4422d4 {"num":3} 2024-09-18T03:17:16Z 0s + ecc2e2f9deb13509547ebcc2e5c55116 {"num":4} 2024-09-18T03:17:16Z 0s + 1e3ddd76525fefd5ac9989d9b6c4727e {"num":5} 2024-09-18T03:17:16Z 0s + 25684ad5ef8f1965b597edeeb8004afa {"num":6} 2024-09-18T03:17:16Z 0s + 8142d250d75d0a20883333c00c3962d5 {"num":7} 2024-09-18T03:17:16Z 0s + 0d6962a0746cb896ce87250808a50051 {"num":8} 2024-09-18T03:17:16Z 0s + 727d91a916260837579d5e42ad696dd9 {"num":9} 2024-09-18T03:17:16Z 0s +``` + +Using the above information, we'll refresh the top split. + +```bash +rill project refresh --model splits_range_incremental --split ff7416f774dfb086006d0b4696c214e1 --local +Refresh initiated. Check the project logs for status updates. +``` + +Then, rerun the splits command to see that the EXECUTED ON columns has been updated. +```bash +royendo@Roys-MacBook-Pro-2 modeling % rill project splits splits_range_incremental --local + KEY (10) DATA EXECUTED ON ELAPSED ERROR + ---------------------------------- ----------- ---------------------- --------- ------- + ff7416f774dfb086006d0b4696c214e1 {"num":0} 2024-09-18T03:17:58Z 1ms + 69401118e166742864f35f1a77ffe07d {"num":1} 2024-09-18T03:17:16Z 1ms + 555ef019f87b5a57ec7b057476fe9d38 {"num":2} 2024-09-18T03:17:16Z 0s + 09a5530e62c87e41848a02680f4422d4 {"num":3} 2024-09-18T03:17:16Z 0s + ecc2e2f9deb13509547ebcc2e5c55116 {"num":4} 2024-09-18T03:17:16Z 0s + 1e3ddd76525fefd5ac9989d9b6c4727e {"num":5} 2024-09-18T03:17:16Z 0s + 25684ad5ef8f1965b597edeeb8004afa {"num":6} 2024-09-18T03:17:16Z 0s + 8142d250d75d0a20883333c00c3962d5 {"num":7} 2024-09-18T03:17:16Z 0s + 0d6962a0746cb896ce87250808a50051 {"num":8} 2024-09-18T03:17:16Z 0s + 727d91a916260837579d5e42ad696dd9 {"num":9} 2024-09-18T03:17:16Z 0s +``` + + +The above is a static model. The split is defined by a set range(0,10) so there is no reason for us to put a refresh key-pair on this. However, real data is likely not static and will require some sort of refresh when you push to Rill Cloud. + +Let's take a look at a more realistic example, our ClickHouse project. + +import DocsRating from '@site/src/components/DocsRating'; + +--- + \ No newline at end of file diff --git a/docs/docs/tutorials/rill_advanced_features/incremental_models/staging-connectors.md b/docs/docs/tutorials/rill_advanced_features/incremental_models/staging-connectors.md index a5ce4c49d1d..087920375f2 100644 --- a/docs/docs/tutorials/rill_advanced_features/incremental_models/staging-connectors.md +++ b/docs/docs/tutorials/rill_advanced_features/incremental_models/staging-connectors.md @@ -5,7 +5,7 @@ sidebar_label: "Staging Connectors" sidebar_position: 13 --- -As mentioned, there are some connections that are not natively supported such as Snowflake to Clickhouse. In order to successfully import data, there are times where a staging table is required. +There are some connections that are not natively supported such as Snowflake to Clickhouse. In order to successfully ingest data from these types of sources, there are times where a staging table is required. :::tip requirements diff --git a/docs/docs/tutorials/rill_basics/launch.md b/docs/docs/tutorials/rill_basics/1-launch.md similarity index 100% rename from docs/docs/tutorials/rill_basics/launch.md rename to docs/docs/tutorials/rill_basics/1-launch.md diff --git a/docs/docs/tutorials/rill_basics/import.md b/docs/docs/tutorials/rill_basics/2-import.md similarity index 100% rename from docs/docs/tutorials/rill_basics/import.md rename to docs/docs/tutorials/rill_basics/2-import.md diff --git a/docs/docs/tutorials/rill_basics/model.md b/docs/docs/tutorials/rill_basics/3-model.md similarity index 100% rename from docs/docs/tutorials/rill_basics/model.md rename to docs/docs/tutorials/rill_basics/3-model.md diff --git a/docs/docs/tutorials/rill_basics/dashboard.md b/docs/docs/tutorials/rill_basics/4-dashboard.md similarity index 51% rename from docs/docs/tutorials/rill_basics/dashboard.md rename to docs/docs/tutorials/rill_basics/4-dashboard.md index 117f58ee8b4..aeffc87822e 100644 --- a/docs/docs/tutorials/rill_basics/dashboard.md +++ b/docs/docs/tutorials/rill_basics/4-dashboard.md @@ -1,6 +1,6 @@ --- -title: "4. Metrics View (Dashboard)" -sidebar_label: "4. Create the Dashboard" +title: "4. Metrics View and Dashboard" +sidebar_label: "4. Create the Metrics View and Dashboard" sidebar_position: 3 hide_table_of_contents: false tags: @@ -8,38 +8,41 @@ tags: --- ### What is a Metrics View? +:::note +Starting from version 0.50, we have officially split the metrics-view and dashboard and rolled out [visual metric editor](#via-the-ui) What this means is that we have a separate file for the metrics layer and a dashboard built on top of this. For more reasoning behind this change, please refer to our documentation. +::: +A metrics view is a layer in which you can create and define your measures and dimensions. Once you have defined your measures and dimensions, you can build the dashboard. -A metrics view is a dashboard where you can visualize and drill down into your data. ### Let's create a metrics view! Now that the model is created, we can create a metrics-view. There are two ways to do so: -1. Generate dashboard with AI -2. Start Simple using the +Add, Dashboard +1. Generate metrics with AI +2. Start Simple using the +Add, Metrics
- How does Generate dashboard with AI work? + How does Generate metrics with AI work? - We send a set of YAML and project files to GPT-4o to suggest the dimensions, measures, and various other key pairs for your dashboard. + We send a set of YAML and project files to OpenAI to suggest the dimensions, measures, and various other key pairs for your dashboard.
-Let's go ahead and create a simple dashboard via the UI and build onto it. +Let's go ahead and create a simple metrics layer via the UI and build onto it. - -
+ +![img](/img/tutorials/102/create-metrics-view-ui.png) As you can see, the default dashboard YAML is as follows: ```yaml -# Dashboard YAML -# Reference documentation: https://docs.rilldata.com/reference/project-files/dashboards +# Metrics View YAML +# Reference documentation: https://docs.rilldata.com/reference/project-files/metrics_views -type: metrics_view #the type is required for all YAML files in Rill to define the type +version: 1 +type: metrics_view -title: "Dashboard Title" -table: example_table # Choose a table [or model] to underpin your dashboard / -timeseries: timestamp_column # Select an actual timestamp column (if any) from your table +table: example_table # Choose a table to underpin your metrics +timeseries: timestamp_column # Choose a timestamp column (if any) from your table dimensions: - column: category @@ -50,36 +53,52 @@ measures: - expression: "SUM(revenue)" label: "Total Revenue" description: "Total revenue generated" - ``` For now, you'll see a red box around the UI and the preview button grayed out. This indicates something is wrong with the YAML. -## Fixing the Dashboard +## Fixing the Metrics View +From here, we have two options! +Starting from version 0.50, we have introduced the [visual-metric-editor](/build/metrics-view/#using-the-visual-metrics-editor), in the top right corner, you can select whether you want to modify the YAML directly or use a UI tool. + +## Via the UI +When navigating to the visual metric editor, you will see the following: + +![img](/img/tutorials/102/new-viz-editor.png) + +We can go ahead and change the following components as directed in the UI: + +- model: commits___model +- time columns: author_date +- measures: sum(added_lines) +- dimensions: author_name + + +![img](/img/tutorials/102/basic-viz-editor.png) + +Once finished, the red border will disappear and your explore dashboard is ready to be created. If you need further information on each component see the next section, via the YAML. + + +## Via the YAML Let's go over each component and what they are in order to better understand the metrics-view and how to fix the dashboard. -### Type ### +### Type ```yaml +version: 1 type: metrics_view ``` The type is a Rill required key pair as it indicates to Rill what type of file this is. Whether a `source`, `metrics_vew`, `connector`, etc. We can keep this as is. --- -### Title and underlying table ### +### Underlying Table ### ```yaml -title: "My Tutorial dashboard" - -#table: example_table # OR -model: commits___model #_ _ _, there are three underbars there! +table: commits___model # Note that this has 3 "_"! ``` -The title will be displayed in the metrics-view UI and should be defined as required.
-Depending on your use-case, `table` or `model` will be used to underpin your dashboard.
- -Let's go ahead and change the title to `My Tutorial dashboard` and comment out the table and add `model: commits___model`. +The underlying table can be defined here, let's change it to `commits___model`. --- @@ -87,7 +106,7 @@ Let's go ahead and change the title to `My Tutorial dashboard` and comment out t ```yaml timeseries: author_date # Select an actual timestamp column (if any) from your table ``` -The timeseries column is a Date type column within your table. While technically this is not required, many of the features require a timeseries column and your resulting dashboard will lack key functionality. Let's set this to `author_date`. +The time-series column is a date type column within your table. Let's set this to `author_date`. --- ### Dimensions ### @@ -119,8 +138,11 @@ Our first measure will be: `SUM(added_lines)`. --- -At this point our dashboard should be previewable! -Let's go ahead a select preview so we can take a look. You should see something similar to the below. +## Create the Explore dashboard +At this point our metrics view is ready! Let's rename the metrics to `commits___model_metrics.yaml` and select `Create Explore`. + + +This will create an explore-dashboards folder with a very simple YAML file. Let's go ahead a select preview to see what it looks like. You should see something similar to the below. ![simple](/img/tutorials/103/simple-dashboard.png) @@ -129,48 +151,19 @@ Let's go ahead a select preview so we can take a look. You should see something We can definitely do better than that! --- -For a quick summary on the different components that we modified and it's respective parts in the dashboard UI. - - +For a quick summary on the different components that we modified, and its respective parts in the dashboard UI.
-
- Dashboard not working? - - Go ahead and copy the YAML below into your dashboard. - ```yaml -# Dashboard YAML -# Reference documentation: https://docs.rilldata.com/reference/project-files/dashboards - -type: metrics_view - -title: "My Tutorial Project" -#table: example_table # Choose a table to underpin your dashboard -model: commits___model - -timeseries: author_date # Select an actual timestamp column (if any) from your table - -dimensions: - - column: author_name - label: "The Author's Name" - description: "The name of the author of the commit" - -measures: - - expression: "SUM(added_lines)" - label: "Sum of Added lines" - description: "The aggregate sum of added_lines column." -``` -
-## Adding more functionalty +## Adding more Functionality -Let's add further dimensions and measures. For +Let's add further dimensions and measure to the metrics layer and see the changes to the explore dashboard. ### Dimensions - From our dataset, we can add more dimensions to allow more filtering and exploration of the measures we will create. +From our dataset, we can add more dimensions to allow more filtering and exploration of the measures we will create. Add the following dimensions, with title and description. - author_name @@ -184,29 +177,29 @@ Let's add further dimensions and measures. For - sum(deleted_lines) -You may need to reference the dashboard YAML reference guide to figure out the above. Your final output should look something like this! +You may need to reference the metrics view YAML reference guide to figure out the above. Your final output should look something like this! ![finished](/img/tutorials/103/Completed-100-dashboard.png)
- Working Dashboard YAML + Working Metrics View YAML ```yaml -# Dashboard YAML -# Reference documentation: https://docs.rilldata.com/reference/project-files/dashboards +# Metrics View YAML +# Reference documentation: https://docs.rilldata.com/reference/project-files/metrics_views +version: 1 type: metrics_view -title: "My Tutorial Project" -#table: example_table # Choose a table to underpin your dashboard -model: commits___model +table: commits___model timeseries: author_date # Select an actual timestamp column (if any) from your table dimensions: - column: author_name - label: "The Author's Name" - description: "The name of the author of the commit" + name: author_name + label: The Author's Name + description: The name of the author of the commit - column: author_timezone label: "The Author's TZ" @@ -215,35 +208,31 @@ dimensions: - column: filename label: "The filename" description: "The name of the modified filename" - + measures: - - expression: "SUM(added_lines)" - label: "Sum of Added lines" - description: "The aggregate sum of added_lines column." + - expression: SUM(added_lines) + name: added_lines + label: Sum of Added lines + format_preset: humanize + description: The aggregate sum of added_lines column. + valid_percent_of_total: true - expression: "SUM(deleted_lines)" label: "Sum of deleted lines" description: "The aggregate sum of deleted_lines column." + ```
-import ComingSoon from '@site/src/components/ComingSoon'; - - -## Modifying the metrics view via the UI - - - -
-**Main components of the metrics view Editor** +### Completed visual metrics editor -Now that we've discussed the YAML and have a general understanding of how it works, I want to show you how to use the metric view editor! This is our UI based editor that allows for a point and click method to modify the metric-view. +If you decide to build out the metrics view via the UI, it should look something like below! +![img](/img/tutorials/103/visual-metric-editor.png) -
diff --git a/docs/docs/tutorials/rill_clickhouse/r_ch_launch.md b/docs/docs/tutorials/rill_clickhouse/1-r_ch_launch.md similarity index 100% rename from docs/docs/tutorials/rill_clickhouse/r_ch_launch.md rename to docs/docs/tutorials/rill_clickhouse/1-r_ch_launch.md diff --git a/docs/docs/tutorials/rill_clickhouse/r_ch_connect.md b/docs/docs/tutorials/rill_clickhouse/2-r_ch_connect.md similarity index 90% rename from docs/docs/tutorials/rill_clickhouse/r_ch_connect.md rename to docs/docs/tutorials/rill_clickhouse/2-r_ch_connect.md index 441572ecb19..c998a0b4de8 100644 --- a/docs/docs/tutorials/rill_clickhouse/r_ch_connect.md +++ b/docs/docs/tutorials/rill_clickhouse/2-r_ch_connect.md @@ -13,7 +13,7 @@ import TabItem from '@theme/TabItem'; ## Default OLAP connection and Connect to ClickHouse -Within Rill you can set the default OLAP connection on the [project level](https://docs.rilldata.com/reference/project-files/rill-yaml) or the [dashboard level](https://docs.rilldata.com/reference/project-files/dashboards). +Within Rill you can set the default OLAP connection on the [project level](https://docs.rilldata.com/reference/project-files/rill-yaml) or the [dashboard level](https://docs.rilldata.com/reference/project-files/explore-dashboards). For this course, we will set it up on the project level so all of our dashboards will be based on our ClickHouse table. :::tip @@ -39,12 +39,15 @@ Once updated, we can create the clickhouse connection by selection `+Add Data` > ![clickhouse](/img/tutorials/ch/clickhouse-connector.png) :::tip -You can obtain the credentials from your ClickHouse Cloud account by following: +You can obtain the credentials from your ClickHouse Cloud account by clicking the `Connect` button in the left panel.: -UPDATE THIS +![clickhouse](/img/tutorials/ch/clickhouse-cloud-credential.png) +``` +"https://:?username=&password=&secure=true&skip_verify=true" +``` ::: -Once this is created, a `clickhouse.yaml` file in the `connectors` folder. +Once this is created, a `clickhouse.yaml` file will appear in the `connectors` folder. Example for a locally running ClickHouse server: ```yaml diff --git a/docs/docs/tutorials/rill_clickhouse/r_ch_dashboard.md b/docs/docs/tutorials/rill_clickhouse/3-r_ch_dashboard.md similarity index 67% rename from docs/docs/tutorials/rill_clickhouse/r_ch_dashboard.md rename to docs/docs/tutorials/rill_clickhouse/3-r_ch_dashboard.md index 1b380b40189..968d4fc8616 100644 --- a/docs/docs/tutorials/rill_clickhouse/r_ch_dashboard.md +++ b/docs/docs/tutorials/rill_clickhouse/3-r_ch_dashboard.md @@ -1,6 +1,6 @@ --- -title: "3. Create Dashboard in Rill" -sidebar_label: "3. Create Dashboard in Rill" +title: "3. Create Metrics-view and Dashboard in Rill" +sidebar_label: "3. Create Metrics-view and Dashboard in Rill" sidebar_position: 4 hide_table_of_contents: false tags: @@ -15,22 +15,32 @@ If you noticed in the previous screenshot, we had a table called `uk_price_paid` In the case that you have not already added this table to your local or Cloud database, please follow the step on [ClickHouse's site](https://clickhouse.com/docs/en/getting-started/example-datasets/uk-price-paid) for the steps to do so! ::: -### Create dashboard -Let's create a dashboard based on the table via the `generate dashboard via AI`. +### Create metrics-view + +Let's create a metrics-view based on the table via the `Generate metrics via AI`.
### What are we looking at? -Returning back to the Metric view, select `Edit Metrics` in the UI, let's do a quick run through of the components within the dashboard. +This is our metrics-view, where we can define measures and dimensions to be used on dashboards. ```yaml +# Metrics view YAML +# Reference documentation: https://docs.rilldata.com/reference/project-files/explore-dashboards +# This file was generated using AI. + +version: 1 type: metrics_view + title: UK Price Paid Metrics +connector: clickhouse table: uk_price_paid timeseries: date + dimensions: + ... measures: @@ -41,11 +51,17 @@ While we go into more details in our [Rill Basics course](/tutorials/rill_basics --- -`Dimensions` - These are our categorial columns that we can use on the dashboard to filter and slice; +`timeseries` - This is our time column that is used on as our x-axis for graphs. + +`connector` - this is our manually defined ClickHouse connector + +`dimensions` - These are our categorial columns that we can use on the dashboard to filter and slice; + +`measures` - These are our numerical aggregates defined in the metrics layer. We can see functions such as MAX(), COUNT(), and AVG() used on the underlying table. -`Measures` - These are our numerical aggregates defined in the metrics layer. We can see functions such as MAX(), COUNT(), and AVG() used on the underlying table. +### Create the explore dashboard -`Timeseries` - This is our time column that is used on as our x-axis for graphs. +When you're ready, you can create the visualization on top of the metric layer. Let's select `create explore`. This will create a simple explore-dashboards/uk_price_paid_metrics_explore.yaml file that reads in all the dimensions and measures. For more information on the available key-pairs, please refer to the [reference documentation.](https://docs.rilldata.com/reference/project-files/explores) --- diff --git a/docs/sidebars.js b/docs/sidebars.js index 10d0b01e62d..0c414397029 100644 --- a/docs/sidebars.js +++ b/docs/sidebars.js @@ -45,7 +45,7 @@ const sidebars = { }, -/** + { type: 'category', label: "Rill's Advanced Features", @@ -69,8 +69,7 @@ const sidebars = { ] }, - - { + /* {　waiting for canvas dashboard release type: 'category', label: 'Rill Canvas Dashboards', items: [ @@ -79,20 +78,33 @@ const sidebars = { 'tutorials/rill_advanced_features/canvas_dashboards/vega-lite', 'tutorials/rill_advanced_features/canvas_dashboards/vega-lite2', 'tutorials/rill_advanced_features/canvas_dashboards/canvas-dashboards', - ] - }, + 'tutorials/rill_advanced_features/canvas_dashboards/filters' - { + ] + }, +*/ +/* + { some changes to syntax and adding 'tasks' type: 'category', label: 'Incremental Models', + items: [ + 'tutorials/rill_advanced_features/incremental_models/introduction', + { + type: 'category', + label: 'Basic Incremental and Split Model Examples', + items: [ + 'tutorials/rill_advanced_features/incremental_models/simple-examples/incremental_now', + 'tutorials/rill_advanced_features/incremental_models/simple-examples/split_now', + ] + }, 'tutorials/rill_advanced_features/incremental_models/cloud-storage-splits', 'tutorials/rill_advanced_features/incremental_models/data-warehouse-splits', 'tutorials/rill_advanced_features/incremental_models/staging-connectors' ] }, - + */ { type: 'category', label: 'Custom APIs', @@ -104,7 +116,6 @@ const sidebars = { }, ] }, - */ { type: 'category', label: 'Administration Topics', @@ -119,20 +130,18 @@ const sidebars = { type: 'category', label: 'User Management', items: [ - 'tutorials/administration/user-management', - 'tutorials/administration/user-group-management', + 'tutorials/administration/user/user-management', + 'tutorials/administration/user/user-group-management', ] }, { type: 'category', label: 'Project Management', items: [ - 'tutorials/administration/project-maintanence', - 'tutorials/administration/alerts', - 'tutorials/administration/credential-envvariable-mangement', - 'tutorials/administration/github', - - + 'tutorials/administration/project/project-maintanence', + 'tutorials/administration/project/alerts', + 'tutorials/administration/project/credential-envvariable-mangement', + 'tutorials/administration/project/github', ] }, @@ -166,6 +175,17 @@ const sidebars = { description: 'For guides that are not quite Rill related but needs consideration', items: [ 'tutorials/other/add-column-dimension', + 'tutorials/other/dashboard-row-policies', + // 'tutorials/other/custom-charts', + // 'tutorials/other/create-map-component', + // 'tutorials/other/component-variable-freedom', + 'tutorials/other/embed-dashboard', + // 'tutorials/other/deep-dive-incremental-modeling', + 'tutorials/other/Rill Cloud/share-dashboard-publicly', + // 'tutorials/other/Rill Cloud/views', + // 'tutorials/other/Rill Cloud/visual-metric-editor-rc', + // 'tutorials/other/yaml-vs-ui', + // 'tutorials/other/trial-check', 'tutorials/other/avg_avg', ] diff --git a/docs/src/components/CustomDocCard.js b/docs/src/components/CustomDocCard.js index c349019ec7b..7bd73c82561 100644 --- a/docs/src/components/CustomDocCard.js +++ b/docs/src/components/CustomDocCard.js @@ -2,24 +2,13 @@ import React from 'react'; import Link from '@docusaurus/Link'; // Define images or colors for specific documents -const cardStyles = { - 'tutorials/rill_basics': { - backgroundImage: 'url(/img/guide-image.jpg)', - backgroundColor: '#f9f9f9', - }, - 'tutorials/rill_advanced_features/overview': { - backgroundImage: 'url(/img/overview-image.jpg)', - backgroundColor: '#eef5f9', - }, - // Add more styles for different docs here -}; + const CustomDocCard = ({ item }) => { // Get custom style (image or background color) based on docId - const style = cardStyles[item.docId] || {}; return ( - +

{item.label}

diff --git a/docs/src/css/custom.scss b/docs/src/css/custom.scss index a3ccc41c509..1936da66914 100644 --- a/docs/src/css/custom.scss +++ b/docs/src/css/custom.scss @@ -278,7 +278,7 @@ li.theme-doc-sidebar-item-category-level-2 .menu__list-item-collapsible { display: block; margin-left: auto; margin-right: auto; - + width: 100%; } // CSS changes to CustomCard Component diff --git a/docs/static/img/build/dashboard/explore-dashboard.png b/docs/static/img/build/dashboard/explore-dashboard.png new file mode 100644 index 00000000000..958b0ac97bd Binary files /dev/null and b/docs/static/img/build/dashboard/explore-dashboard.png differ diff --git a/docs/static/img/build/dashboard/preview-dashboard.png b/docs/static/img/build/dashboard/preview-dashboard.png new file mode 100644 index 00000000000..051846303a3 Binary files /dev/null and b/docs/static/img/build/dashboard/preview-dashboard.png differ diff --git a/docs/static/img/build/metrics-view/visual-metrics-editor.gif b/docs/static/img/build/metrics-view/visual-metrics-editor.gif new file mode 100644 index 00000000000..c02a85c9a2a Binary files /dev/null and b/docs/static/img/build/metrics-view/visual-metrics-editor.gif differ diff --git a/docs/static/img/build/metrics-view/visual-metrics-editor.png b/docs/static/img/build/metrics-view/visual-metrics-editor.png new file mode 100644 index 00000000000..9c277c54a29 Binary files /dev/null and b/docs/static/img/build/metrics-view/visual-metrics-editor.png differ diff --git a/docs/static/img/build/structure/adding-nested-folder.png b/docs/static/img/build/structure/adding-nested-folder.png index c2e17ae8676..a6d3fde9143 100644 Binary files a/docs/static/img/build/structure/adding-nested-folder.png and b/docs/static/img/build/structure/adding-nested-folder.png differ diff --git a/docs/static/img/build/structure/adding-objects.png b/docs/static/img/build/structure/adding-objects.png index 5a207d59979..3d28d5caecc 100644 Binary files a/docs/static/img/build/structure/adding-objects.png and b/docs/static/img/build/structure/adding-objects.png differ diff --git a/docs/static/img/concepts/metrics-view/explore-dashboard.png b/docs/static/img/concepts/metrics-view/explore-dashboard.png new file mode 100644 index 00000000000..78dec5be1ee Binary files /dev/null and b/docs/static/img/concepts/metrics-view/explore-dashboard.png differ diff --git a/docs/static/img/concepts/metrics-view/metrics-view-components.png b/docs/static/img/concepts/metrics-view/metrics-view-components.png new file mode 100644 index 00000000000..1da94472387 Binary files /dev/null and b/docs/static/img/concepts/metrics-view/metrics-view-components.png differ diff --git a/docs/static/img/concepts/metrics-view/old-dashboard.png b/docs/static/img/concepts/metrics-view/old-dashboard.png new file mode 100644 index 00000000000..5be10cab8bd Binary files /dev/null and b/docs/static/img/concepts/metrics-view/old-dashboard.png differ diff --git a/docs/static/img/explore/filters/scrub-graph.gif b/docs/static/img/explore/filters/scrub-graph.gif new file mode 100644 index 00000000000..7fc24a2bab2 Binary files /dev/null and b/docs/static/img/explore/filters/scrub-graph.gif differ diff --git a/docs/static/img/explore/publicurl/public-url-settings.png b/docs/static/img/explore/publicurl/public-url-settings.png new file mode 100644 index 00000000000..18064fd9374 Binary files /dev/null and b/docs/static/img/explore/publicurl/public-url-settings.png differ diff --git a/docs/static/img/manage/settings/rill-cloud-settings.png b/docs/static/img/manage/settings/rill-cloud-settings.png new file mode 100644 index 00000000000..1b8679d4ab1 Binary files /dev/null and b/docs/static/img/manage/settings/rill-cloud-settings.png differ diff --git a/docs/static/img/tutorials/102/Add-Dashboard.gif b/docs/static/img/tutorials/102/Add-Dashboard.gif deleted file mode 100644 index 2aa42bdb14d..00000000000 Binary files a/docs/static/img/tutorials/102/Add-Dashboard.gif and /dev/null differ diff --git a/docs/static/img/tutorials/102/basic-viz-editor.png b/docs/static/img/tutorials/102/basic-viz-editor.png new file mode 100644 index 00000000000..c9ea6d1f6ed Binary files /dev/null and b/docs/static/img/tutorials/102/basic-viz-editor.png differ diff --git a/docs/static/img/tutorials/102/create-metrics-view-ui.png b/docs/static/img/tutorials/102/create-metrics-view-ui.png new file mode 100644 index 00000000000..d522d3cd5bf Binary files /dev/null and b/docs/static/img/tutorials/102/create-metrics-view-ui.png differ diff --git a/docs/static/img/tutorials/102/new-viz-editor.png b/docs/static/img/tutorials/102/new-viz-editor.png new file mode 100644 index 00000000000..a397dffbf30 Binary files /dev/null and b/docs/static/img/tutorials/102/new-viz-editor.png differ diff --git a/docs/static/img/tutorials/103/visual-metric-editor.png b/docs/static/img/tutorials/103/visual-metric-editor.png new file mode 100644 index 00000000000..2c1c79626c8 Binary files /dev/null and b/docs/static/img/tutorials/103/visual-metric-editor.png differ diff --git a/docs/static/img/tutorials/301/add-custom-dashboard.png b/docs/static/img/tutorials/301/add-custom-dashboard.png index b11b8838d85..e1a1de6a09e 100644 Binary files a/docs/static/img/tutorials/301/add-custom-dashboard.png and b/docs/static/img/tutorials/301/add-custom-dashboard.png differ diff --git a/docs/static/img/tutorials/301/add_custom_chart_dashboard.png b/docs/static/img/tutorials/301/add_custom_chart_dashboard.png deleted file mode 100644 index d5a6fef822b..00000000000 Binary files a/docs/static/img/tutorials/301/add_custom_chart_dashboard.png and /dev/null differ diff --git a/docs/static/img/tutorials/301/canvas-dashboard-filters.png b/docs/static/img/tutorials/301/canvas-dashboard-filters.png new file mode 100644 index 00000000000..8c192e06439 Binary files /dev/null and b/docs/static/img/tutorials/301/canvas-dashboard-filters.png differ diff --git a/docs/static/img/tutorials/301/component-filter-off.png b/docs/static/img/tutorials/301/component-filter-off.png new file mode 100644 index 00000000000..a909a8e8982 Binary files /dev/null and b/docs/static/img/tutorials/301/component-filter-off.png differ diff --git a/docs/static/img/tutorials/301/component-filter-on.png b/docs/static/img/tutorials/301/component-filter-on.png new file mode 100644 index 00000000000..c0d05e6c2df Binary files /dev/null and b/docs/static/img/tutorials/301/component-filter-on.png differ diff --git a/docs/static/img/tutorials/301/selector.png b/docs/static/img/tutorials/301/selector.png new file mode 100644 index 00000000000..d6c24fb333e Binary files /dev/null and b/docs/static/img/tutorials/301/selector.png differ diff --git a/docs/static/img/tutorials/302/now-incremental.png b/docs/static/img/tutorials/302/now-incremental.png new file mode 100644 index 00000000000..1ec45d2f64d Binary files /dev/null and b/docs/static/img/tutorials/302/now-incremental.png differ diff --git a/docs/static/img/tutorials/302/now-state.png b/docs/static/img/tutorials/302/now-state.png new file mode 100644 index 00000000000..762e3bef40a Binary files /dev/null and b/docs/static/img/tutorials/302/now-state.png differ diff --git a/docs/static/img/tutorials/302/now.png b/docs/static/img/tutorials/302/now.png new file mode 100644 index 00000000000..a336d85bf14 Binary files /dev/null and b/docs/static/img/tutorials/302/now.png differ diff --git a/docs/static/img/tutorials/302/splits-refresh-ui.png b/docs/static/img/tutorials/302/splits-refresh-ui.png new file mode 100644 index 00000000000..ed92c278d47 Binary files /dev/null and b/docs/static/img/tutorials/302/splits-refresh-ui.png differ diff --git a/docs/static/img/tutorials/admin/settings-public-url.png b/docs/static/img/tutorials/admin/settings-public-url.png new file mode 100644 index 00000000000..5470c9ea16d Binary files /dev/null and b/docs/static/img/tutorials/admin/settings-public-url.png differ diff --git a/docs/static/img/tutorials/ch/clickhouse-cloud-credential.png b/docs/static/img/tutorials/ch/clickhouse-cloud-credential.png new file mode 100644 index 00000000000..bc945f76d8d Binary files /dev/null and b/docs/static/img/tutorials/ch/clickhouse-cloud-credential.png differ diff --git a/docs/static/img/tutorials/other/dashboard-new-dimension.png b/docs/static/img/tutorials/other/dashboard-new-dimension.png deleted file mode 100644 index 0a1e06ba70b..00000000000 Binary files a/docs/static/img/tutorials/other/dashboard-new-dimension.png and /dev/null differ diff --git a/docs/static/img/tutorials/other/maps/bar.png b/docs/static/img/tutorials/other/maps/bar.png new file mode 100644 index 00000000000..c152971b8b5 Binary files /dev/null and b/docs/static/img/tutorials/other/maps/bar.png differ diff --git a/docs/static/img/tutorials/other/maps/data.png b/docs/static/img/tutorials/other/maps/data.png new file mode 100644 index 00000000000..b3550681a16 Binary files /dev/null and b/docs/static/img/tutorials/other/maps/data.png differ diff --git a/docs/static/img/tutorials/other/maps/map_broken.png b/docs/static/img/tutorials/other/maps/map_broken.png new file mode 100644 index 00000000000..ccd12f94d45 Binary files /dev/null and b/docs/static/img/tutorials/other/maps/map_broken.png differ diff --git a/docs/static/img/tutorials/other/maps/map_fixed.png b/docs/static/img/tutorials/other/maps/map_fixed.png new file mode 100644 index 00000000000..28a22bf07e3 Binary files /dev/null and b/docs/static/img/tutorials/other/maps/map_fixed.png differ diff --git a/docs/static/img/tutorials/other/natural-language/adding-component.png b/docs/static/img/tutorials/other/natural-language/adding-component.png new file mode 100644 index 00000000000..97dde05e714 Binary files /dev/null and b/docs/static/img/tutorials/other/natural-language/adding-component.png differ diff --git a/docs/static/img/tutorials/other/natural-language/bar-chart.png b/docs/static/img/tutorials/other/natural-language/bar-chart.png new file mode 100644 index 00000000000..e76f0cc9669 Binary files /dev/null and b/docs/static/img/tutorials/other/natural-language/bar-chart.png differ diff --git a/docs/static/img/tutorials/other/natural-language/component-data.png b/docs/static/img/tutorials/other/natural-language/component-data.png new file mode 100644 index 00000000000..34e62743113 Binary files /dev/null and b/docs/static/img/tutorials/other/natural-language/component-data.png differ diff --git a/docs/static/img/tutorials/other/natural-language/line-chart.png b/docs/static/img/tutorials/other/natural-language/line-chart.png new file mode 100644 index 00000000000..bc16987230a Binary files /dev/null and b/docs/static/img/tutorials/other/natural-language/line-chart.png differ diff --git a/docs/static/img/tutorials/other/natural-language/stacked-bar-chart.png b/docs/static/img/tutorials/other/natural-language/stacked-bar-chart.png new file mode 100644 index 00000000000..f722276f830 Binary files /dev/null and b/docs/static/img/tutorials/other/natural-language/stacked-bar-chart.png differ diff --git a/docs/static/img/tutorials/other/new-dimension/dashboard-new-dimension.png b/docs/static/img/tutorials/other/new-dimension/dashboard-new-dimension.png new file mode 100644 index 00000000000..2d07a1ea46f Binary files /dev/null and b/docs/static/img/tutorials/other/new-dimension/dashboard-new-dimension.png differ diff --git a/docs/static/img/tutorials/other/new-dimension/metrics-new-dimension.png b/docs/static/img/tutorials/other/new-dimension/metrics-new-dimension.png new file mode 100644 index 00000000000..ff71fb5aacf Binary files /dev/null and b/docs/static/img/tutorials/other/new-dimension/metrics-new-dimension.png differ diff --git a/docs/static/img/tutorials/other/model-new-dimension.png b/docs/static/img/tutorials/other/new-dimension/model-new-dimension.png similarity index 100% rename from docs/static/img/tutorials/other/model-new-dimension.png rename to docs/static/img/tutorials/other/new-dimension/model-new-dimension.png diff --git a/docs/static/img/tutorials/other/source-new-dimension.png b/docs/static/img/tutorials/other/new-dimension/source-new-dimension.png similarity index 100% rename from docs/static/img/tutorials/other/source-new-dimension.png rename to docs/static/img/tutorials/other/new-dimension/source-new-dimension.png diff --git a/docs/static/img/tutorials/other/public-url/share-public-url.png b/docs/static/img/tutorials/other/public-url/share-public-url.png new file mode 100644 index 00000000000..8878d2b38ce Binary files /dev/null and b/docs/static/img/tutorials/other/public-url/share-public-url.png differ diff --git a/docs/static/img/tutorials/other/row-policy/row-policy-view.png b/docs/static/img/tutorials/other/row-policy/row-policy-view.png new file mode 100644 index 00000000000..ce4f532bfa6 Binary files /dev/null and b/docs/static/img/tutorials/other/row-policy/row-policy-view.png differ diff --git a/runtime/testruntime/testdata/ad_bids/dashboards/ad_bids_metrics_view.yaml b/runtime/testruntime/testdata/ad_bids/dashboards/ad_bids_metrics_view.yaml index 37fba421c2e..5267a682f83 100644 --- a/runtime/testruntime/testdata/ad_bids/dashboards/ad_bids_metrics_view.yaml +++ b/runtime/testruntime/testdata/ad_bids/dashboards/ad_bids_metrics_view.yaml @@ -1,5 +1,5 @@ # Dashboard YAML -# Reference documentation: https://docs.rilldata.com/reference/project-files/dashboards +# Reference documentation: https://docs.rilldata.com/reference/project-files/explore-dashboards type: metrics_view title: Ad Bids diff --git a/runtime/testruntime/testdata/ad_bids_clickhouse/dashboards/ad_bids_metrics_view.yaml b/runtime/testruntime/testdata/ad_bids_clickhouse/dashboards/ad_bids_metrics_view.yaml index 052f26e8f2b..6f77c247651 100644 --- a/runtime/testruntime/testdata/ad_bids_clickhouse/dashboards/ad_bids_metrics_view.yaml +++ b/runtime/testruntime/testdata/ad_bids_clickhouse/dashboards/ad_bids_metrics_view.yaml @@ -1,5 +1,5 @@ # Dashboard YAML -# Reference documentation: https://docs.rilldata.com/reference/project-files/dashboards +# Reference documentation: https://docs.rilldata.com/reference/project-files/explore-dashboards type: metrics_view title: Ad Bids