This guide is mostly useful for people (including Storj Labs employees) who want to develop and/or gain more in-depth knowledge about Storj's edge services.
Edge services bridge the gap between the native integration and existing applications written to talk with S3 object storage (Gateway-ST and Gateway-MT). Link Sharing Service allows to quickly share a file or an entire directory via URL.
- MinIO fork upon which Gateway-ST and Gateway-MT are based
- Gateway-ST
- Gateway-MT and Link Sharing Service (this repository)
Auth Service maps existing access grants to S3-specific Access Key ID/Secret Access Key pairs on demand. The primary reason for using Auth Service is that it's hard to pass long access grants in Access Key ID/Secret Access Key. See this for a more in-depth explanation of how Gateway-MT works with Auth Service. Link Sharing Service can work with and without Auth Service (it's easier and more secure with Auth Service). Gateway-ST does not need Auth Service.
See Public Roadmap filtered for Edge Team.
See also the GitHub project including current items the team is working on.
- Gateway-ST (also Gateway-MT's object layer) documentation
- Auth Service, Gateway-MT and Link Sharing Service
- Blueprints (see this for an explanation of what blueprints are)
TODO(artur): write about this. Present different approaches, including storj/up.
A suite of integration tests for Gateway-MT can be run on the checked out code.
make integration-run will start a new integration environment using Docker and run all the integration tests.
At the end of the run, you can run make integration-env-purge to remove the integration environment.
This requires docker, git, and openssl to be installed on your local machine for this to work.
We run mint tests based on MinIO's mint (gateway-mint) on every commit to Gateway-MT against Gateway-MT, as well as in the fork's repository itself.
To run the tests:
make integration-env-start integration-mint-tests
You can also run a specific test using the TEST environment variable:
TEST=aws-sdk-php make integration-mint-tests
We run S3 tests based on Splunk's fork (which is better suited for us) of Ceph's S3 tests (splunk-s3-tests) on every commit to Gateway-MT against Gateway-MT, as well as in the fork's repository itself.
To run the tests:
make integration-env-start integration-splunk-tests
We currently don't run performance tests. We are planning to performance-test Gateway-MT/Gateway-ST using COSBench and warp.
We collect metrics and telemetry metrics for Gateway-ST (if consent has been granted), similarly to what this blueprint proposes.