Feature/use csv as data source

[SeanLeRoy]

Description of Changes

The FMS File Explorer App is usually powered by its connection to the File Explorer Service which is fed by the FMS MongoDB, this changeset makes it possible to select a separate data source that the app will run queries against itself using DuckDB. For the moment (until #64) I added this concept into the existing Collections and added a prompt for users to select a data source when one isn't detected (like if unable to connect to the File Explorer Service).

How to Review

Much of what is different about this is within the DatabaseFileService, DatabaseAnnotationService, & the various logics files. I'd begin by taking a look there. There is quite a bit to still do to this feature to make it a fully-fledged concept (see next steps below) - but I need to tie this work of in a bow for now to prioritize other work.

Testing

Tested running various CSV data sources & with unit tests.

Next Steps

• Simplify dataset/collection concept
• Support web deploys
• Support all data types from CSV/JSON/Parquet sources

[SeanLeRoy]

Not changed, just moved/renamed from index.ts

[SeanLeRoy]

Not changed, just moved/renamed from index.ts

[kmitcham]

Individual comments as I left them

[aswallace]

I pulled the changes to try this out locally, and ran into an issue with the filters. Here's what I did:

• Used the regular Allen Institute FMS data source to generate a csv out of the first 10 files. Selected the first ~20 or so annotations as columns
• Switched to the "Create from file" collection and selected the csv I just downloaded

I can download files and use the Annotation Hierarchy as expected. But while I can filter without issue on the regular annotation properties, I had issues filtering on file properties (file_id, file_name etc)-- it lets me select values to filter on but I can't deselect or reset.
Tried testing further by adding my own test column to the csv, and while the column does show up as an option, it has a similar issue in the filters where I can select values but not de-select (but unlike file props, does show up as a medallion that I can remove via the FilterDisplayBar)

[SeanLeRoy]

I pulled the changes to try this out locally, and ran into an issue with the filters. Here's what I did:

• Used the regular Allen Institute FMS data source to generate a csv out of the first 10 files. Selected the first ~20 or so annotations as columns
• Switched to the "Create from file" collection and selected the csv I just downloaded

I can download files and use the Annotation Hierarchy as expected. But while I can filter without issue on the regular annotation properties, I had issues filtering on file properties (file_id, file_name etc)-- it lets me select values to filter on but I can't deselect or reset. Tried testing further by adding my own test column to the csv, and while the column does show up as an option, it has a similar issue in the filters where I can select values but not de-select (but unlike file props, does show up as a medallion that I can remove via the FilterDisplayBar)

Thank you for testing this out! I should've commented on this, but I was saving fixing the file properties for when I have combined with your work (hoping to keep this open until you merge your PR). I think a lot of what that will entail involves managing how the filters are send to the "backend" so I'd like to not have a weird merge with your work involving putting the filters into the list, let me know what you think of the plan

[aswallace]

Thank you for testing this out! I should've commented on this, but I was saving fixing the file properties for when I have combined with your work (hoping to keep this open until you merge your PR). I think a lot of what that will entail involves managing how the filters are send to the "backend" so I'd like to not have a weird merge with your work involving putting the filters into the list, let me know what you think of the plan

That sounds good, makes sense!

[SeanLeRoy]

Closing, ready to merge once more changes are applied to this branch through new PRs