Enable Multiple Data Sources #109

SeanLeRoy · 2024-05-21T18:04:04Z

Description
This changeset enables users to select more than one data source in a single query. This is largely a proof-of-concept implementation of this feature as I think we could spend more time to make this much more performant like by perhaps creating views that are representations of a query's data sources.

How to review
Most of the changes to look for are in the SQLBuilder & logics.tsx

Testing
Tested on desktop and in the web adding various types of data source combination and performing queries.

aswallace

Having some trouble testing locally. Main thing is that when I use two different source files that should have the same columns, it shows up like this with null columns:

The data from these two sources came from the same file, I just downloaded them as two separate files and then re-selected them each as datasources in the same query

Also, if I add a third source I get

Error: Parser Error: syntax error at or near "JOIN"
LINE 5: ...,355,118,721"."File Path") AS "File Path"
            FROM "file-selection-Tue...

aswallace · 2024-05-22T03:53:24Z

packages/core/state/selection/logics.ts

@@ -578,7 +608,7 @@ const replaceDataSourceLogic = createLogic({
        deps.ctx.replaceDataSourceAction = deps.action;
        const queries = selectionSelectors.getQueries(deps.getState());
        const updatedQueries = queries.map((query) => {
-            if (query.parts.source?.name !== replacementDataSource.name) {
+            if (query.parts.sources[0]?.name !== replacementDataSource.name) {


Wouldn't load for me locally until I put query.parts.source?.[0]?.name. It's possible that this was just a first time load error?

packages/core/services/FileService/DatabaseFileService/index.ts

packages/web/src/services/DatabaseServiceWeb.ts

packages/core/components/QueryPart/QueryDataSource.tsx

BrianWhitneyAI · 2024-05-22T20:44:26Z

Seeing this when trying to load in various datasets locally.

fro variance set

:9001/?source=%7B%22…ri%22%3A%7B%7D%7D:1 Uncaught (in promise) Catalog Error: Table with name Variance Paper Dataset does not exist!
Did you mean "pg_database"?
LINE 4:             FROM "Variance Paper Dataset"

And this for the second dataset

localhost/:1 Uncaught (in promise) Catalog Error: Table with name AIND Image Atlas - AIND Image Atlas does not exist!
Did you mean "pg_settings"?
LINE 4:             FROM "AIND Image Atlas - AIND Image Atlas"

aswallace · 2024-06-10T22:43:43Z

I'm now able to select, filter, group on multiple data sources, both web and desktop. Still having issues with downloading manifests:
Web:

I can download successfully for non AICS sources, but when I try to download from AICS source it claims to download successfully, but then the browser gives me a "File wasn't available on site" error and doesn't actually complete the download

Desktop:

Still having the header issue with downloading CSVs
I'm still seeing the error Brian mentioned when it tries to reload an external data source, and the modal + error won't go away even if I select a new file

SeanLeRoy · 2024-06-13T21:33:38Z

I'm now able to select, filter, group on multiple data sources, both web and desktop. Still having issues with downloading manifests: Web:

I can download successfully for non AICS sources, but when I try to download from AICS source it claims to download successfully, but then the browser gives me a "File wasn't available on site" error and doesn't actually complete the download
This was an issue with the type I was sending to the databaseservice, should work now.

Desktop:

Still having the header issue with downloading CSVs
Added option "HEADER" to CSV exports, should work

I'm still seeing the error Brian mentioned when it tries to reload an external data source, and the modal + error won't go away even if I select a new file
Fixed, the cache of annotations was saving [] as a valid list of annotations, seems to be working now

This should all be fixed now.

aswallace · 2024-06-13T23:51:31Z

This should all be fixed now.

Confirmed that these fixes all worked for me!
Only running into a couple small issues now:

Desktop: Sometimes when I try to add a new query/data source it adds it to the old/current query rather than creating a new one? I'm not exactly sure how to reproduce
If the columns aren't all exactly the same for both data sources (e.g., one is missing one column), I get errors (first screenshot is desktop, second is web):

Saw that when a query with multiple data sources has a source changed, it always throws an error even if it's successful: Uncaught (in promise) Catalog Error: Table with name AICS-13, AICS-5 does not exist! Did you mean "AICS-13"? FROM "AICS-13, AICS-5"

Also two future feature requests:

would be really nice if we could delete queries without opening them, since clicking on them to delete means the app automatically attempts to run the query, which can be annoying if it's an old data source I don't intend to replace
Some errors don't render in the file list UI, so it can look like it's just loading forever if you don't know to check console

SeanLeRoy added 2 commits May 20, 2024 20:33

Add multi data source support

85f628f

UI prompts and logics for multi data source working

fe48abb

SeanLeRoy self-assigned this May 21, 2024

SeanLeRoy added 3 commits May 21, 2024 11:05

Ensure lack of dupe data

2888478

Improve UX of adding/removing sources

d8c1ad1

Adjust unit tests

709a179

SeanLeRoy changed the title ~~Feature/multi data source support~~ Enable Multiple Data Sources May 21, 2024

Update from main

3cf6cd0

SeanLeRoy marked this pull request as ready for review May 21, 2024 19:44

SeanLeRoy requested review from aswallace and BrianWhitneyAI May 21, 2024 19:44

aswallace reviewed May 22, 2024

View reviewed changes

SeanLeRoy added 6 commits May 31, 2024 10:11

Create view before multi-data source

2a921e4

Refactor DatabaseService into abstract class

c943ae1

Update tests

d2adee6

Address PR Feedback

f925ce8

Update from main

f3b9922

Merge branch 'main' into feature/multi-data-source-support

6ce48ef

SeanLeRoy added 4 commits June 13, 2024 13:47

Fix download over web

2a95603

Fix data source replacement

a6cbd15

Fix CSV export on desktop

a6decd4

Adjust test

dcd5e5a

aswallace self-requested a review June 14, 2024 17:17

aswallace approved these changes Jun 14, 2024

View reviewed changes

SeanLeRoy added 2 commits June 17, 2024 09:36

Merge branch 'main' into feature/multi-data-source-support

02cfef3

Update tests after update from main

9ecd29c

SeanLeRoy merged commit 0a53f42 into main Jun 17, 2024
3 checks passed

SeanLeRoy deleted the feature/multi-data-source-support branch June 17, 2024 16:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable Multiple Data Sources #109

Enable Multiple Data Sources #109

SeanLeRoy commented May 21, 2024 •

edited

Loading

aswallace left a comment

aswallace May 22, 2024

BrianWhitneyAI commented May 22, 2024 •

edited

Loading

aswallace commented Jun 10, 2024 •

edited

Loading

SeanLeRoy commented Jun 13, 2024

aswallace commented Jun 13, 2024

Enable Multiple Data Sources #109

Enable Multiple Data Sources #109

Conversation

SeanLeRoy commented May 21, 2024 • edited Loading

aswallace left a comment

Choose a reason for hiding this comment

aswallace May 22, 2024

Choose a reason for hiding this comment

BrianWhitneyAI commented May 22, 2024 • edited Loading

aswallace commented Jun 10, 2024 • edited Loading

SeanLeRoy commented Jun 13, 2024

aswallace commented Jun 13, 2024

SeanLeRoy commented May 21, 2024 •

edited

Loading

BrianWhitneyAI commented May 22, 2024 •

edited

Loading

aswallace commented Jun 10, 2024 •

edited

Loading