How Singularity can help project motion

[xinaxu]

Starting a discussion to capture initial thoughts for how project motion can depends on Singularity to achieve core functionality

Initialization

• Project motion should setup environment variables correctly for Singularity
  • Database connection string
  • Dataset worker concurrency - default to nproc / 4
  • Lotus API - default to GLIF
  • HTTP Hostname - for distributing CAR file in boost deal proposal
  • Retry Policy parameters
• Project motion should also setup port forwarding to expose the Singularity HTTP service for CAR download
• Project motion should call Singularity API to
  • create a default dataset and default datasource
  • —inline-preparation - whether to export CAR files
  • —output-dir - for storing CAR files
  • —staging-dir - for storing uploaded files
  • —delete-after-export - whether to delete uploaded files after exporting to CAR files
  • —target-piece-size 32G|64G
  • —pad-piece - whether to pad the CAR file if it’s not full
  • —scan-interval - how often should the staging folder to be scanned
  • —max-wait — how long to wait before we pack a CAR file that is not full

POST /v0/data

There are two approaches for how POST works with Singularity

1. Approach One: Motion is responsible of managing the staging folder and tells Singularity when a new file is ready to grab. - that's the one I'm in favor of.
   1. Less change in Singularity. The implementation of API spec resides in Project motion
   2. Singularity current implementation delays CID calculation. If Motion needs the CID in /GET, it needs to make sure it’s calculated the same way (using same chunker/hasher)
   3. How it works
      1. Motion accepts the request, save it to staging folder and calls /dataset/push API with the file path. Motion manages lifecycle of the staging folder
      2. Motion will handle the case where the blob already exist
      3. Singularity returns itemID and and delays the preparation
      4. Motion saves mapping of dataID ↔ Singularity ItemID
      5. Motion can query the status of this item using the itemID
2. Motion forward the stream to Singularity and let Singularity manage the staging folder
   1. Singularity can calculate CID while accepting the stream
   2. How it works
      1. Motion accepts the request and return necessary response header
      2. Motion send the binary stream in the body(octet-stream) to Singularity /dataset/upload API (multipart-formdata)
      3. Singularity saves the stream to a file in staging folder
      4. Singularity calculates the CID while streaming
      5. Singularity returns both CID and itemID
      6. Singularity will also return whether the blob already exists
      7. Motion saves mapping of dataID ↔ Singularity ItemID
      8. Motion can query the status of this item using the itemID

GET /v0/data/{id}

With /POST approach 1, Motion owns the staging folder and the implementation of this API

With /POST approach 2, Motion will make a call to Singularity content provider service and return the file stream in the body.

GET /v0/data/{id}/status

Singularity already has everything in the database. How it works

1. Motion calls Singularity API GET /item/<item_id>/deals
2. Singularity returns all deals for this item
3. Motion converts it to the its own format and return
4. “LastVerified” is not there - need research on how to get it from lotus chain state