Feature Request: Provide different ways to upload and store attachments, and drag attachment to card to upload it

[fnkr]

Large databases are not easy to handle and to backup.
It would be useful to store attachments as file in the local filesystem (or maybe on a Cloud Storage provider like Google Cloud Storage or Amazon S3?).

[shekar73]

i agree 👍

[ocdtrekkie]

Local filesystem would definitely be nice. Not sure someone who wants a self-hosted board is going to really want to store data on Google or Amazon clouds though, as that largely defeats the point.

[dcworldwide]

Users may or may not find value in self hosted attachments. I'd prefer if i could like to my google drive account. Any ideas of how we might achieve non-gridfs storage?

[xet7]

I will add S3, Google Drive support etc. I'll add this to my todo list.

[ahashibon]

supporting something like owncloud or next cloud would be great, also supporting access to davros on sandstorm would also be pretty cool! I doubt that using google, or AS would help much, this defeats the point point of own hosting !

[xet7]

Planning to use this package https://atmospherejs.com/malibun23/files

[GavinLilly]

Storing attachments in a different manner would certainly help organisations using Wekan self-hosted. The last thing they want is to have data stored in many places. In my mind AWS and GDrive would be the most applicable places to store and they're both supported by the "files" package

[razum2um]

@ocdtrekkie @ahashibon please, note, s3 does not "defeat the point" because it's a protocol and there's https://github.com/minio/minio self-hosted alternative with the same api (@rossmansfield mentioned it as well)

[razum2um]

@xet7 @blaggacao did you have time to get back on this btw? could you maybe add hacktoberfest topic to the repo?

p.s oh, it seems like tech discussion continues here #3272

[razum2um]

@rossmansfield feel free to plegde anything https://www.bountysource.com/ or in crypto https://gitcoin.co/ do you still need this?

[blaggacao]

The current implementation on #3273 is just a special implementation of a storage backend towards GridFS MongoDB Buckets.

With the same strategy, those can be stored anywhere given a connector is provided. See for example here. I'd assume, storing to file system is as simple as not deleting the uploaded files and not storing them into a specific backend.

[razum2um]

@blaggacao thanks for such a quick reply, btw, what do you think about another idea: given there's already a self-hosted minio and some images inside - can we just render direct img src= to the direct location, i.e. not uploading twice, but specifying the link?

[blaggacao]

Yes the download location of ostrio:files is configurable. Upload goes through meteor, though. With upload comes the proper initialization of the image reference in the mongo database. So

given there's already a self-hosted minio and some images inside

would be rather awkward to support. Though nothing speaks against using a self hosted minion and populate it through wekan uploads.

[razum2um]

already a self-hosted minio and some images inside

would be rather awkward to support

oh, sorry, didn't realised input can contain just html #2635 (comment)

[blaggacao]

Here is where a hook intercepts the upload and stores it into the back-end (and deletes the uploaded files):

https://github.com/wekan/wekan/pull/3273/files#diff-4a858d7364de26a72d82ca64882d740c4ecf2f5efe5eab5bd481a3a909d49518R26

[xet7]

From @razum2um

feel free to plegde anything https://www.bountysource.com/ or in crypto https://gitcoin.co/ do you still need this?

For Wekan, current payment options are at https://wekan.team/commercial-support/ . Those BountySource and Gitcoin or anything else are not in use at Wekan, it was already a lot of work to move away from BountySource #3177 .

[xet7]

@mfilser @NotTheEvilOne @blaggacao

Any ideas how to fix my first try of AWS S3 code 21e2eab so that after upload and optionally checking file with ClamAV etc, it would upload file to S3 ? And show attachments at cards, be able to move files to S3, etc? Beginnings of those changes are at feature-s3 branch https://github.com/wekan/wekan/tree/feature-s3

Related https://github.com/veliovgroup/Meteor-Files/blob/master/docs/aws-s3-integration.md

[NotTheEvilOne]

Hi,

Fortunately from the validation part no changes are required at all. Validation and storage movement are separated steps by now and can easily work with your changes. I'll try to find some time to look more detailed into the S3 branch over the holidays.

[xet7]

@NotTheEvilOne

I fixed some typos and added some more code so that WeKan starts properly with those S3 related changes, but I have not yet tested does it work. Have you made any changes?

028633b

[xet7]

@mfilser @NotTheEvilOne

I listed related S3 commits at https://github.com/wekan/wekan/blob/master/CHANGELOG.md#upcoming-wekan--release

I merged feature-s3 branch to master, and deleted feature-s3 branch, because it seems WeKan starts anyway with these changes.

Currently I'm getting this error, when trying to click button to move attachment to S3. Trying to figure it out.

I20221226-06:37:48.337(2)? Exception while invoking method 'moveAttachmentToStorage' TypeError: this.classFileStoreStrategyS3 is not a constructor
I20221226-06:37:48.338(2)?     at FileStoreStrategyFactory.getFileStrategy (models/lib/fileStoreStrategy.js:56:15)
I20221226-06:37:48.338(2)?     at models/lib/fileStoreStrategy.js:458:52
I20221226-06:37:48.338(2)?     at Array.forEach (<anonymous>)
I20221226-06:37:48.338(2)?     at moveToStorage (models/lib/fileStoreStrategy.js:456:33)
I20221226-06:37:48.338(2)?     at MethodInvocation.moveAttachmentToStorage (models/attachments.js:147:7)
I20221226-06:37:48.339(2)?     at packages/check/match.js:118:15
I20221226-06:37:48.339(2)?     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1329:12)
I20221226-06:37:48.339(2)?     at Object._failIfArgumentsAreNotAllChecked (packages/check/match.js:116:43)
I20221226-06:37:48.339(2)?     at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1899:18)
I20221226-06:37:48.339(2)?     at getCurrentMethodInvocationResult (packages/ddp-server/livedata_server.js:772:38)
I20221226-06:37:48.339(2)?     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1329:12)
I20221226-06:37:48.339(2)?     at packages/ddp-server/livedata_server.js:791:46
I20221226-06:37:48.339(2)?     at new Promise (<anonymous>)
I20221226-06:37:48.340(2)?     at Session.method (packages/ddp-server/livedata_server.js:739:23)
I20221226-06:37:48.340(2)?     at packages/ddp-server/livedata_server.js:603:43

[xet7]

I got some S3 example working at https://github.com/wekan/meteor-files-website where I made these changes wekan/meteor-files-website@c56185e so that it would start properly with:

1. settings.json copied to parent directory
2. S3 settings added
3. Started with start.sh

So with that example, I'm able to upload to S3. I'll look what is different with that example, and current WeKan code, what would be required to get WeKan code working.

[xet7]

I asked from Meteor-Files is there possibility for Custom Endpoint for MinIO veliovgroup/Meteor-Files#862

[xet7]

It's possible to use MinIO and other cloud storage as filesystem mounted with Rclone, docs here:
https://github.com/wekan/wekan/wiki/Rclone

It works with move to filesystem button (not move to S3 button, that does not work yet).

[verdel]

@xet7, do I understand correctly that the current WeKan code using Minio.client still doesn't work?

I pass an object in the S3 environment variable that contains:

{"s3":{"endpoint": "https://storage.googleapis.com", "port": "443", "sslEnabled": "true", "bucket": "bucket-name", "key": "client-key", "secret": "client-secret"}}

but I still get the error Exception while invoking method 'moveAttachmentToStorage' TypeError: this.classFileStoreStrategyS3 is not a constructor.

[xet7]

@verdel

There is some file store strategy pattern using code here, but some is still missing from being able to move files to/from S3 compatible storage:

https://github.com/wekan/wekan/tree/main/models/lib

[verdel]

@xet7

Yes, I took a closer look at the code and found the reason.

In this part, the FileStoreStrategyFactory instance is initialized, which is then passed to the moveToStorage() function when calling the moveAttachmentToStorage() method.

When initializing the FileStoreStrategyFactory class, its constructor should include the classFileStoreStrategyS3 class and the name of the S3 bucket (at least that seems to have been the intention).

Since this is not happening, later in the code, when the classFileStoreStrategyS3 instance is initialized, the mentioned error occurs because at that moment this.classFileStoreStrategyS3 is undefined.

Further down the code, there are additional issues that will prevent it from working correctly:

• During the initialization of the FileStoreStrategyFactory class instance, an instance of the MongoInternals.NpmModule.GridFSBucket class needs to be passed to the constructor as the s3Bucket parameter (not a bucket name), which, for example, is used here

• This issue is related to the fact that the FileStoreStrategyS3 class was copied from the FileStoreStrategyGridFs class and its implementation of the methods getReadStream(), getWriteStream(), writeStreamFinished(), and unlink() was not fully reworked

• In the fileStoreStrategy.js file, the initialized FilesCollection class is not imported

It seems to me that the implementation of working with S3 in its current form is still too early to add to release versions.

Perhaps it would be wise to temporarily disable this functionality and wait for its full implementation?

[xet7]

@verdel

Hmm, disable how?

[mfilser]

I made this strategy classes (design pattern), but only implemented GridFs and Filesystem, not S3. I wonder who made the S3 implementation as it's probably missing a lot of code to get it working. Also, I doubt the S3 code was successfully tested to have it in an official wekan release?

[verdel]

@xet7, ideally, all the code related to the S3 implementation should be moved to a separate dev branch and removed from the release branch. It will also be necessary to remove the Move all attachments to S3 button from the Settings section and the Move attachment to S3 button from the attachment properties in the card.

After these changes, a new release will need to be issued.

@mfilser, the author of the commits that added the implementation is @xet7