How should data providers host datasets they're adding to scivision catalog?

[edwardchalstrey1]

I was just having a chat with @acocac about how data providers wishing to submit a dataset to the scivision dataset catalog should host their data, i.e. where should it be uploaded to on the web.

Perhaps the option that makes the most sense since we're already asking them to create an intake catalog file, is for the data provider to create an intake data package, which uploads the data to something called "Anaconda cloud".

Another option is pangeo forge which looks like you have the option to choose a cloud provider of choice.

Thinking about @evangeline-corcoran 's project data in particular, we thought that it's important to make sure that it remains possible to query the model catalog with a private dataset, even if you don't add this dataset to the scivision dataset catalog - see #143 - so perhaps this kind of consideration may impact which of the above options we recommend?

Also @mooniean @evangeline-corcoran @acocac - looking at ds4s 38 and ds4s 37, perhaps the next tasks after you have created the intake drivers will be to figure out how to host your data e.g. create an intake data package?

[mooniean]

the packages on cryoEM (synthetic or the ones that we can open with the reader) would be more dynamic: i.e., the synthetic makes the data as we need it, and the reader loads the dataset (hosted at the emdb) requested.

[edwardchalstrey1]

This has also come up as something to think about - how load_dataset() works in such cases

[mooniean]

I made an issue on the ds4s management git: https://github.com/alan-turing-institute/DS4S-project-management/issues/64

[ots22]

How should data providers host datasets they're adding to scivision catalog?

My feeling is that data hosted at any publicly accessible URI should be acceptable in the catalogue. There are many places where data is hosted and we don't want to insist on anywhere in particular, at the exclusion of other places.

Is this question really about having a few suggested options, for someone who has a local dataset and wants to host it publicly somewhere?

Or perhaps, are there additional things we should require (we might insist that a dataset has a DOI for example)?

[edwardchalstrey1]

I think having suggested options is good, but yes also agree that any public URI makes sense, I guess we were thinking as much about the example datasets that @acocac and the other post-docs are working on, as other data providers