Merge main

docs/blog/0.52.md

@@ -0,0 +1,62 @@
+---
+
+date: 2024-12-12
+image: https://github.com/rilldata/rill/assets/5587788/b30486f6-002a-445d-8a1b-955b6ec0066d
+
+---
+
+# Rill 0.52 - Visual Explore Editing, human readable URLs, env variables in the cloud and breadcrumbs!
+
+:::note
+⚡ Rill Developer is a tool that makes it effortless to transform your datasets with SQL and create fast, exploratory dashboards. Rill Cloud can then help to enable shared collaboration at scale.
+
+To [try out Rill Developer, check out these instructions](/home/install) and [join us over on Discord](https://bit.ly/3bbcSl9) to meet the team behind the product as well as other users. In addition, once you have a project set up in Rill Developer, you can then [deploy the project](/deploy/deploy-dashboard) and collaborate with others / invite your team members by [logging into Rill Cloud](https://ui.rilldata.com)!
+:::
+
+![release-0 52](<https://cdn.rilldata.com/docs/release-notes/release052.gif>)
+
+## Visual Explore Editing
+With our 0.51 release we introduced the ability to visually edit metrics and dimensions in the metrics view. Now we are extending the same capabilities also for Explore dashboards. So if you prefer a UX experience over editing yaml files directly then this is for you!
+
+## Human Readable URLs
+As you navigate around inside of a Rill dashboard what you see on the screen is always present in the URL so that you can easily share that URL with other users and they will see exactly what you are seeing on the screen. Previously we encoded this URL so that it was very hard to read, change or make updates to but with our 0.52 release the URL will now be fully human readable. This allows you to more easily embed a Rill dashboard into intranets and dynamically change what the user sees on the screen and when you get sent a link from someone it's easier to understand what the actual URL will show on the screen.
+
+## Environment Variables - Manage in Rill Cloud
+In Rill Cloud you can now visually manage your environment variables for a project. Add, update or delete them directly from Rill Cloud without having to fire up the Rill CLI. Change a password env variable and all your developers have to do is a rill env pull and they are up-to-date!
+
+## Project Breadcrumbs in Rill Developer
+As you are navigating around in Rill Developer you will now find breadcrumbs that shows you the associated resources so that you can quickly navigate between them as you are making changes. An early quality of life holiday present!
+
+## Bug Fixes and Misc
+- Added support for STS AssumeRole in Athena connector.
+- Support locate formatting using `d3_locale` in metrics view.
+- Added UI for Environment Variables in Rill Cloud.
+- Removed "Back to Home" from Embed Dashboards.
+- Improved Big number KPI formatting rules.
+- Default to UTC when no timezones are selected.
+- Environmental Variables are now available in security policies.
+- Port not required when connecting to Cloud OLAP URL.
+- Improved UI: Improved project user list UI.
+- Improved UI: Improved buttons to return to Explore main page.
+- Improved UX: Disable slack notification by default.
+- Improved UX: Refresh button should initiate incremental refresh where possible.
+- Improved UX: Support infinite query when listing org users and invites.
+- Improved UX: Offer to copy to clipboard CTA when creating public URL
+- Improved UX: Users able to drag and select multiple values to copy in pivot.
+- Improved UX: Add clear button on pivot header for rows and columns.
+- Improved UX: Added error handling for row access policies.
+- fixed Leaderboard column sorting behavior.
+- fixed `Select all` button in the dimension leaderboard.
+- fixed Scroll position in pivot table when loading new rows. 
+- fixed environmental overrides in rill.yaml.
+- fixed custom compare time ranges in reports or alerts.
+- fixed leaderboard header no label issue, defaults to name.
+- fixed case issues with bookmarks.
+- fixed autogenerated metric view indentation and default parameters.
+- fixed issue when switching dashboards based on the same metrics view in Rill Cloud.
+- fixed formatting in TDD when formatting is not defined, use humanize.
+- fixed issue in embed dashboard when pivot disabled, `Start pivot` still visible.
+- fixed issue where could not search by name in Org search.
+- fixed issue where limit was not being applied to exports.
+- fixed issue when disabling compare dimension in TDD not being respected.
+- fixed issue where ClickHouse would auto unnest in metrics view.

runtime/compilers/rillv1/parse_node.go

@@ -73,6 +73,9 @@ type commonYAML struct {
 	Kind *string `yaml:"kind"`
 	// Name is usually inferred from the filename, but can be specified manually.
 	Name string `yaml:"name"`
+	// Namespace is an optional value to group resources by.
+	// It currently just gets pre-pended to the resource name in the format `<namespace>/<name>`.
+	Namespace string `yaml:"namespace"`
 	// Refs are a list of other resources that this resource depends on. They are usually inferred from other fields, but can also be specified manually.
 	Refs []yaml.Node `yaml:"refs"`
 	// ParserConfig enables setting file-level parser config.
@@ -283,6 +286,11 @@ func (p *Parser) parseStem(paths []string, ymlPath, yml, sqlPath, sql string) (*
 		}
 	}
 
+	// If a namespace was provided in YAML, prepend it to the name.
+	if cfg != nil && cfg.Namespace != "" {
+		res.Name = cfg.Namespace + ":" + res.Name
+	}
+
 	// If resource kind is not set in YAML or SQL, try to infer it from the context
 	if res.Kind == ResourceKindUnspecified {
 		if strings.HasPrefix(paths[0], "/sources") {

runtime/compilers/rillv1/parser_test.go

@@ -2041,6 +2041,50 @@ refresh:
 	requireResourcesAndErrors(t, p, []*Resource{m1, m2}, nil)
 }
 
+func TestNamespace(t *testing.T) {
+	ctx := context.Background()
+	repo := makeRepo(t, map[string]string{
+		`rill.yaml`: ``,
+		`models/m1.yaml`: `
+type: model
+sql: SELECT 1
+`,
+		`explores/e1.yaml`: `
+type: explore
+namespace: foo
+metrics_view: missing
+`,
+	})
+
+	resources := []*Resource{
+		{
+			Name:  ResourceName{Kind: ResourceKindModel, Name: "m1"},
+			Paths: []string{"/models/m1.yaml"},
+			ModelSpec: &runtimev1.ModelSpec{
+				RefreshSchedule: &runtimev1.Schedule{RefUpdate: true},
+				InputConnector:  "duckdb",
+				InputProperties: must(structpb.NewStruct(map[string]any{"sql": `SELECT 1`})),
+				OutputConnector: "duckdb",
+			},
+		},
+		{
+			Name:  ResourceName{Kind: ResourceKindExplore, Name: "foo:e1"},
+			Paths: []string{"/explores/e1.yaml"},
+			Refs:  []ResourceName{{Kind: ResourceKindMetricsView, Name: "missing"}},
+			ExploreSpec: &runtimev1.ExploreSpec{
+				DisplayName:        "Foo: E1",
+				MetricsView:        "missing",
+				DimensionsSelector: &runtimev1.FieldSelector{Selector: &runtimev1.FieldSelector_All{All: true}},
+				MeasuresSelector:   &runtimev1.FieldSelector{Selector: &runtimev1.FieldSelector_All{All: true}},
+			},
+		},
+	}
+
+	p, err := Parse(ctx, repo, "", "", "duckdb")
+	require.NoError(t, err)
+	requireResourcesAndErrors(t, p, resources, nil)
+}
+
 func requireResourcesAndErrors(t testing.TB, p *Parser, wantResources []*Resource, wantErrors []*runtimev1.ParseError) {
 	// Check errors
 	// NOTE: Assumes there's at most one parse error per file path

runtime/compilers/rillv1/util.go

@@ -18,6 +18,9 @@ func ToDisplayName(name string) string {
 	name = strings.ReplaceAll(name, "_", " ")
 	name = strings.ReplaceAll(name, "-", " ")
 
+	// Replace colons with colon-space.
+	name = strings.ReplaceAll(name, ":", ": ")
+
 	// Capitalize the first letter.
 	name = cases.Title(language.English).String(name)
 

web-admin/src/features/bookmarks/BookmarksDropdownMenuContent.svelte

@@ -80,6 +80,7 @@
     $schemaResp.data?.schema,
     $exploreState,
     defaultExplorePreset,
+    $metricsViewTimeRange.data?.timeRangeSummary,
   );
   $: filteredBookmarks = searchBookmarks(categorizedBookmarks, searchText);
 

web-admin/src/features/bookmarks/selectors.ts

@@ -8,7 +8,10 @@ import { getDashboardStateFromUrl } from "@rilldata/web-common/features/dashboar
 import { useMetricsViewTimeRange } from "@rilldata/web-common/features/dashboards/selectors";
 import { useExploreState } from "@rilldata/web-common/features/dashboards/stores/dashboard-stores";
 import type { MetricsExplorerEntity } from "@rilldata/web-common/features/dashboards/stores/metrics-explorer-entity";
-import { timeControlStateSelector } from "@rilldata/web-common/features/dashboards/time-controls/time-control-store";
+import {
+  getTimeControlState,
+  timeControlStateSelector,
+} from "@rilldata/web-common/features/dashboards/time-controls/time-control-store";
 import { convertExploreStateToURLSearchParams } from "@rilldata/web-common/features/dashboards/url-state/convertExploreStateToURLSearchParams";
 import { ResourceKind } from "@rilldata/web-common/features/entity-management/resource-selectors";
 import { useExploreValidSpec } from "@rilldata/web-common/features/explores/selectors";
@@ -20,6 +23,7 @@ import {
   type V1ExploreSpec,
   type V1MetricsViewSpec,
   type V1StructType,
+  type V1TimeRangeSummary,
 } from "@rilldata/web-common/runtime-client";
 import type { QueryClient } from "@tanstack/query-core";
 import { derived, get, type Readable } from "svelte/store";
@@ -61,6 +65,7 @@ export function categorizeBookmarks(
   schema: V1StructType | undefined,
   exploreState: MetricsExplorerEntity,
   defaultExplorePreset: V1ExplorePreset,
+  timeRangeSummary: V1TimeRangeSummary | undefined,
 ) {
   const bookmarks: Bookmarks = {
     home: undefined,
@@ -76,6 +81,7 @@ export function categorizeBookmarks(
       schema ?? {},
       exploreState,
       defaultExplorePreset,
+      timeRangeSummary,
     );
     if (isHomeBookmark(bookmarkResource)) {
       bookmarks.home = bookmark;
@@ -143,19 +149,27 @@ export function convertBookmarkToUrlSearchParams(
   schema: V1StructType,
   exploreState: MetricsExplorerEntity | undefined,
   defaultExplorePreset: V1ExplorePreset,
+  timeRangeSummary: V1TimeRangeSummary | undefined,
 ) {
   const exploreStateFromBookmark = getDashboardStateFromUrl(
     bookmarkResource.data ?? "",
     metricsViewSpec,
     exploreSpec,
     schema,
   );
+  const finalExploreState = {
+    ...(exploreState ?? {}),
+    ...exploreStateFromBookmark,
+  } as MetricsExplorerEntity;
   return convertExploreStateToURLSearchParams(
-    {
-      ...(exploreState ?? {}),
-      ...exploreStateFromBookmark,
-    } as MetricsExplorerEntity,
+    finalExploreState,
     exploreSpec,
+    getTimeControlState(
+      metricsViewSpec,
+      exploreSpec,
+      timeRangeSummary,
+      finalExploreState,
+    ),
     defaultExplorePreset,
   );
 }
@@ -167,6 +181,7 @@ function parseBookmark(
   schema: V1StructType,
   exploreState: MetricsExplorerEntity,
   defaultExplorePreset: V1ExplorePreset,
+  timeRangeSummary: V1TimeRangeSummary | undefined,
 ): BookmarkEntry {
   const url = new URL(get(page).url);
   url.search = convertBookmarkToUrlSearchParams(
@@ -176,6 +191,7 @@ function parseBookmark(
     schema,
     exploreState,
     defaultExplorePreset,
+    timeRangeSummary,
   );
   return {
     resource: bookmarkResource,

web-admin/src/features/dashboards/query-mappers/utils.ts

@@ -1,5 +1,6 @@
 import { createInExpression } from "@rilldata/web-common/features/dashboards/stores/filter-utils";
 import type { MetricsExplorerEntity } from "@rilldata/web-common/features/dashboards/stores/metrics-explorer-entity";
+import { getTimeControlState } from "@rilldata/web-common/features/dashboards/time-controls/time-control-store";
 import { PreviousCompleteRangeMap } from "@rilldata/web-common/features/dashboards/time-controls/time-range-mappers";
 import { convertExploreStateToURLSearchParams } from "@rilldata/web-common/features/dashboards/url-state/convertExploreStateToURLSearchParams";
 import { getDefaultExplorePreset } from "@rilldata/web-common/features/dashboards/url-state/getDefaultExplorePreset";
@@ -182,6 +183,7 @@ export async function getExplorePageUrl(
   const url = new URL(`${curPageUrl.protocol}//${curPageUrl.host}`);
   url.pathname = `/${organization}/${project}/explore/${exploreName}`;
 
+  const metricsViewSpec = metricsView?.metricsView?.state?.validSpec ?? {};
   const exploreSpec = explore?.explore?.state?.validSpec ?? {};
   const metricsViewName = exploreSpec.metricsView;
 
@@ -206,6 +208,12 @@ export async function getExplorePageUrl(
   url.search = convertExploreStateToURLSearchParams(
     exploreState,
     exploreSpec,
+    getTimeControlState(
+      metricsViewSpec,
+      exploreSpec,
+      fullTimeRange?.timeRangeSummary,
+      exploreState,
+    ),
     getDefaultExplorePreset(exploreSpec, fullTimeRange),
   );
   return url.toString();

web-admin/src/features/projects/status/ProjectGlobalStatusIndicator.svelte

@@ -30,12 +30,17 @@
               .length > 0
           );
         },
+        refetchOnMount: true,
+        refetchOnWindowFocus: true,
       },
     },
   );
   $: hasResourceErrors = $hasResourceErrorsQuery.data;
 
-  $: projectParserQuery = useProjectParser(queryClient, instanceId);
+  $: projectParserQuery = useProjectParser(queryClient, instanceId, {
+    refetchOnMount: true,
+    refetchOnWindowFocus: true,
+  });
   $: hasParseErrors =
     $projectParserQuery?.data?.projectParser.state.parseErrors.length > 0;
 </script>

web-admin/src/features/projects/status/ProjectParseErrors.svelte

@@ -8,10 +8,21 @@
   import { createRuntimeServiceGetResource } from "@rilldata/web-common/runtime-client";
   import { runtime } from "@rilldata/web-common/runtime-client/runtime-store";
 
-  $: projectParserQuery = createRuntimeServiceGetResource($runtime.instanceId, {
-    "name.kind": ResourceKind.ProjectParser,
-    "name.name": SingletonProjectParserName,
-  });
+  $: ({ instanceId } = $runtime);
+
+  $: projectParserQuery = createRuntimeServiceGetResource(
+    instanceId,
+    {
+      "name.kind": ResourceKind.ProjectParser,
+      "name.name": SingletonProjectParserName,
+    },
+    {
+      query: {
+        refetchOnMount: true,
+        refetchOnWindowFocus: true,
+      },
+    },
+  );
   $: ({ isLoading, isSuccess, data, error } = $projectParserQuery);
 
   $: parseErrors = data?.resource?.projectParser.state.parseErrors;
@@ -30,7 +41,7 @@
   {:else if isSuccess}
     {#if parseErrors && parseErrors.length > 0}
       <ul class="border rounded-sm">
-        {#each parseErrors as error}
+        {#each parseErrors as error (error.message)}
           <li
             class="flex gap-x-5 justify-between py-1 px-9 border-b border-gray-200 bg-red-50 font-mono last:border-b-0"
           >

web-admin/src/features/projects/status/ProjectResources.svelte

@@ -16,8 +16,10 @@
 
   let isReconciling = false;
 
+  $: ({ instanceId } = $runtime);
+
   $: resources = createRuntimeServiceListResources(
-    $runtime.instanceId,
+    instanceId,
     // All resource "kinds"
     undefined,
     {
@@ -30,6 +32,7 @@
           );
         },
         refetchOnMount: true,
+        refetchOnWindowFocus: true,
         refetchInterval: isReconciling ? 500 : false,
       },
     },

web-common/package.json

@@ -25,6 +25,7 @@
     "@codemirror/lang-yaml": "^6.1.1",
     "@codemirror/language": "^6.10.3",
     "@codemirror/lint": "^6.8.2",
+    "@codemirror/merge": "^6.7.4",
     "@codemirror/search": "^6.5.6",
     "@codemirror/state": "^6.4.1",
     "@codemirror/view": "^6.34.1",