On upload of Folder Keep Folder Structure

[shlake]

Now that Dataverse supports File Path (and preserves the directory structure of a zipped file), it would be nice to have Dataverse keep the structure of a folder on a NON-zipped folder.

[mheppler]

@shlake could you please just clarify your use case for me a little more detail.

Do you mean that you would to drag a folder from your desktop, into the upload widget on the Upload Files pg? Are there any other uploads methods you have tried this with?

We are working to collect use cases for our design efforts in Capsulization and Packaging of Replication Objects in Dataverse #6085, and I want to make sure we have your use case included in our inventory.

[shlake]

@mheppler yes, dragging a folder from my desktop to the upload widget on the upload files page. The folder here: "Carpentry Slides" has a subdirectory, but after I drag the folder to the upload page, the structure is not kept on dataverse (no File Path).

Also note that on the File Upload page the button "Select Files to Add" ONLY allows files, you cannot chose a directory - if you select a directory, it "opens". You must select files only.

I have not tried the command line API.

[scolapasta]

@shlake this is a current limitation of the upload widget we are using. as @mheppler mentioned we've discussed this and are going to investigate adding this ability to this widget - I thought we had created the issue, but cannot find it. We can link here once we do.

[pdurbin]

a current limitation of the upload widget we are using

Here's the upstream issue, at least: primefaces/primefaces#2705

@adam3smith seems to think folder upload via browser is tricky: https://twitter.com/adam42smith/status/1232417189988311046

Any maybe it is. I'm not sure.

I did find an answer with 7 upvotes here, which gives me hope: https://stackoverflow.com/questions/42239663/upload-folder-and-all-its-content-in-javascript

[jggautier]

Until solutions for having the upload component keep the folder structure are investigated and implemented, can we do something to let the depositor know, before they start uploading folders, that any structure will be flattened, and suggest putting the files and folders into a compressed file so that Dataverse remembers how the files are organized and can re-create the folder structure when people download the files?

Recently a depositor who emailed us was confused because of this limitation, which led me to this issue, but I'm not sure how often people try to upload folders, how aware they are or how soon they become aware that the folder structure isn't preserved, and if or how they figure out that uploading certain types of archive files work better (other than contacting a repository's support team).

[pdurbin]

@shlake have you tried the new fancy https://github.com/gdcc/dvwebloader by the one and only @qqmyers ?

It might be good for what ails ya. 😄

[shlake]

@qqmyers is dvwebloader just for S3 storage, or any?

I thought I saw (or heard) somewhere that it will not work for local storage (which UVa still has).

[qqmyers]

dvwebloader at present is only S3 and only direct upload. There's no reason, other than time and $, that it couldn't be extended to use the normal file upload API as well. (Although our non-direct file upload API at present only allows upload of one file at a time - the dataset gets saved/reindexed after each one, so making dvwebloader handle non-direct/file store upload would still be much less efficient than S3/direct (unless/until that is updated in some way)).
FWIW: DVUploader implements both methods, so it will work with file stores already - with the performance being lower when its not direct. (Also - basically anything DVUploader can do, i.e. in terms of various flags/options, dvwebloader could also do if we think it makes sense.)

[pdurbin]

@shlake my impression is that you've been trying out dvwebloader and it's working for you. Closing. If not, please let us know!