Should motion POST API take bytes or a download URL?

[hannahhoward]

We’ve described the post API as taking bytes, but I wonder if the post API makes better sense as something that gets a URL to fetch data from.

Essentially this would mirror the PULL approach that is eventually used at the SP level.

Long term, this seems way more flexible - especially for very large blobs - but also doesn’t preclude an S3 integration — it would just move storage of the blob above the motion API.

[hannahhoward]

One risk with external download URL: the download could get removed before it's downloaded.

[willscott]

right - it's a more complex interaction: do we promise "exactly once" pulls? when can they invalidate the url on their side.

i don't think we should force this mode, and would say the simpler POST is probably where we should start for the MVP and then add a pull mode for larger data in a subsequent iteration?

[xinaxu]

Certainly valuable but not for MVP. IMO, this feature should belong to a different application that can be built on top of Project motion APIs - and then can add more sources such as FTP support, IPFS support, etc.

[hannahhoward]

My point is why take bytes directly in the POST at all? Why not go right to a download URL?

Taking bytes means storing bytes-- it adds complexity. Pulling bytes is actually easier.

[hannahhoward]

and it's just an HTTP interface -- doesn't even need SSL cause it's all in the same system.

[xinaxu]

If the MVP only offers this low level API we will need to validate with isv if this pull mode works for them.

What will the design be like with this approach for s3 integration? Is it going to be something like accept a file from S3 API and then tell motion to work on this file using this internal URL?

Also I think that pull approach is going to be come a duplicate of the work Singularity is already doing.

[masih]

If the rationale is to cope with mid upload disruption then taking in a URL is certainly an option but not the only one. For example, we can push bytes and allow resumption.

I also think this is out of the scope for MVP.