Skip to content

Commit

Permalink
docs: chris' suggestions; dockerfile rename
Browse files Browse the repository at this point in the history
  • Loading branch information
hopeyen committed Jan 16, 2024
1 parent c39ffdd commit dc67ebd
Show file tree
Hide file tree
Showing 7 changed files with 22 additions and 36 deletions.
File renamed without changes.
23 changes: 9 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,32 +1,28 @@
# File Exchange
# File Hosting Service

## Introduction
Enable file sharing as a exchange, aim for a decentralized, efficient, and verifiable market, with scalable, performant, and secure software.

File Exchange is a decentralized, peer-to-peer data sharing platform designed for efficient and verifiable file sharing. It leverages a combination of technologies including Hash commitments on IPFS for file discovery and verification, chunk data transfer and micropayments reducing trust requirements between clients and servers, and HTTPS over HTTP2 for secure and efficient data transfer. The system is built with scalability, performance, integrity, and security in mind, aiming to create a robust market for file sharing.


File Exchange leverages IPFS for file discovery and verification, ensuring that each piece of data shared is authentic and unaltered. The use of SHA2-256 for hashing provides a balance of speed and security, making the system both fast and impenetrable to known cryptographic attacks. Furthermore, the adoption of HTTPS over HTTP2 with range requests ensures that all data transfers are not only swift but also secure, safeguarding against common internet vulnerabilities and minimizing risks per transaction.
File Hosting Service (FHS) is a marketplace for sharing file data and is part of The Graph Network's [World of Data Services](https://forum.thegraph.com/t/gip-0042-a-world-of-data-services/3761).

FHS is a decentralized, peer-to-peer data sharing platform designed for efficient and trust-minimised file sharing that is payments-enabled. It leverages a combination of technologies including hash commitments on IPFS for file discovery and verification, chunked data transfer and micropayments reducing trust requirements between clients and servers, and secure and efficient data transfers via HTTP2. The system is built with scalability, performance, integrity, and security in mind, aiming to create a robust market for file sharing.

## Target Audience

This documentation is tailored for individuals who have a basic understanding of decentralized technologies, peer-to-peer networks, and cryptographic principles. Whether you are an indexer running various blockchain nodes looking for sharing and verifying your data, an indexer looking to launch service for a new chain, or simply a user interested in the world of decentralized file sharing, this guide aims to provide you with a clear and comprehensive understanding of how File Service operates.

## Features

- Decentralized File Sharing: Utilize peer-to-peer networks for direct file transfers, eliminating central points of failure.
- Decentralized File Sharing: FHS uses direct connections for file transfers, eliminating central points of failure.
- IPFS Integration: Employ IPFS for efficient and reliable file discovery and content verification.
- SHA2-256 Hashing: Ensure data integrity through robust cryptographic hashing.
- HTTPS over HTTP2: Leverage the latest web protocols for secure and efficient data transfer.
- SHA2-256 Hashing: Ensure data integrity through robust and incremental cryptographic hashing.
- HTTP2 and TLS: Leverage the latest web protocols for secure and efficient data transfer.

**To be supported:**
- Micropayments Support: Implement a system of micropayments to facilitate fair compensation and reduce trust requirements.
- Scalability and Performance: Designed with a focus on handling large volumes of data and high user traffic.
- User-Friendly Interface: Intuitive design for easy navigation and operation.

More details can be found in [Feature checklist](docs/feature_checklist.md)

More details can be found in [Feature Checklist](docs/feature_checklist.md)

## Upgrading

Expand All @@ -44,12 +40,11 @@ You may learn background information on various components of the exchange

4. **Blockchain**: [World of data services](https://forum.thegraph.com/t/gip-0042-a-world-of-data-services/3761), [flatfiles for Ethereum](https://github.com/streamingfast/firehose-ethereum), [use case](https://eips.ethereum.org/EIPS/eip-4444).


## Documentation

#### [Design Principle](docs/architecture.md)
#### [Architecture](docs/architecture.md)

#### [Entity Definition](docs/manifest.md)
#### [Entity Definitions](docs/manifest.md)

#### [Contracts](docs/contracts.md)

Expand Down
10 changes: 1 addition & 9 deletions docs/architecture.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,8 @@
## Architecture and Components

### Decentralized Architecture

Unlike traditional centralized systems where data is stored and managed through a single entity, File Service distributes data across a network of nodes. This decentralization ensures resilience against failures and attacks as there is no single point of failure; it also enhances privacy and control as users are not reliant on a central authority for data management.

The architecture is underpinned by a peer-to-peer (P2P) network framework, where each node in the network can act as a client and/or a server. This setup facilitates direct file sharing among users without the need for intermediaries, and as more users join and contribute to the network, the system becomes more robust and capable of handling larger volumes of data.


### Generic Diagram

![Diagram](./file-exchange.png)

![Diagram](./fhs.png)

### Key Components

Expand Down
9 changes: 4 additions & 5 deletions docs/client_guide.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
# File Sharing Client

Tired of data drudgery? Imagine a world where you skip the tedious task of indexing data and start serving queries in a flash! That's the magic of our P2P file sharing network. Instead of spending hours to months catching up to the head of a chain, you can tap into a vast pool of indexed information, ready and waiting for your queries.
Tired of data drudgery? Imagine a world where you skip the tedious task of indexing data and start serving queries in a flash! That's the magic of our file sharing network. Instead of spending hours to months catching up to the head of a chain, you can tap into a vast pool of indexed information, ready and waiting for your queries.

This document provides an overview of the file sharing client.

## Functionality

The client facilitates data retrieval from the P2P file sharing network. Users can specify desired files by their content addressable IDs (CIDs) and utilize various features:
The client facilitates data retrieval from FHS. The client implements:

- File retrieval: Download entire files or specific chunks at a time.
- Payment: Pay for file access using tokens deposited in an Escrow account on-chain.
Expand All @@ -25,7 +25,7 @@ After determining the Bundle CID, client should supply a local path for writing

### CLI example
```
➜ file-exchange git:(main) ✗ cargo run -p file-exchange downloader \
$ file-exchange downloader \
--ipfs-hash QmHash \
--indexer-endpoints http://localhost:5678,http://localhost:5677 \
--free-query-auth-token 'Bearer auth_token' \
Expand All @@ -35,12 +35,11 @@ After determining the Bundle CID, client should supply a local path for writing
--provider "arbitrum-sepolia-rpc-endpoint"
```


### Requirements

To use the client effectively, you will need:

- Content Address: The CID of the desired file.
- Bundle Manifest CID: The CID of the Bundle Manifest you want to download.
- Local Path: A directory where the downloaded file will be stored. (Later will be a generic storage path, enabling cloud storage access)
- Wallet: A blockchain wallet containing tokens for escrow payments.
- Indexer Endpoints: A list of available server addresses.
Expand Down
File renamed without changes
8 changes: 4 additions & 4 deletions docs/publisher_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,16 +8,16 @@ To start, you would need to provide several configurations

### Requirements

Publisher must have read access to all files contained in the package. The publisher publish 1 Bundle at a time and is not responsible for hosting the file after publishing. The publisher should chunk all the files in the package and generate a hash for all the chunks. Then the publisher will build a hierarchy with the hashes. Currently, publisher simply put chunk hashes in a list for each individual files, publish individual file manifests, then they build a Bundle that contains a list of the file manifest addresses.
Publisher must have read access to all files contained in the Bundle. The publisher publish 1 Bundle at a time and is not responsible for hosting the file after publishing. The publisher should chunk all the files in the package and generate a hash for all the chunks. Then the publisher will build a hierarchy with the hashes. Currently, the Publisher simply put chunk hashes in a list for each individual file, publish individual file manifests, then they build a Bundle that contains a list of the file manifest addresses.

> More exploration for hashing/packaging architecture

### CLI example
```
➜ file-exchange git:(main) ✗ cargo run -p file-exchange publisher \
$ file-exchange publisher \
--read-dir ./example-file/ \
--Bundle-name "blah" \
--bundle-name "blah" \
--file-names example0017686312.dbin,example-create-17686085.dbin \
--file-type flatfiles \
--file-version 0.0.0 \
Expand All @@ -27,5 +27,5 @@ Publisher must have read access to all files contained in the package. The publi

For more information
```
➜ file-exchange git:(main) ✗ cargo run -p file-exchange --help
$ file-exchange --help
```
8 changes: 4 additions & 4 deletions docs/server_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,13 @@

You hold the key to unlocking the network's true potential. By sharing your meticulously indexed data, you become the architect of information accessibility. Imagine, your contribution fueling a vibrant ecosystem of knowledge, where fellow indexers can build upon your work, unleashing a torrent of information to the world. In return, your generosity is rewarded with precious tokens, a testament to the invaluable role you play in this decentralized revolution. Become the data hero we need, and together, let us build a brighter future fueled by open access and boundless knowledge!

This document provides an overview of the P2P file sharing server, intended for those familiar with blockchain nodes and large datasets.
This document provides an overview of the file sharing server, intended for those familiar with blockchain nodes and large datasets.

Jump to [Quick Start](###getting-started)

## File Transfer Protocol

The server utilizes HTTP2 over HTTPS for secure and efficient file transfer. This ensures data integrity and confidentiality while leveraging the performance benefits of HTTP2.
The server utilizes HTTP2 with TLS for secure and efficient file transfer. This ensures data integrity and confidentiality while leveraging the performance benefits of HTTP2.


## Access Control
Expand Down Expand Up @@ -53,15 +53,15 @@ The server utilizes a combination of open-source technologies for optimal perfor

CLI example
```
✗ cargo run -p file-exchange server \
$ file-exchange server \
--host 0.0.0.0 \
--port 5678 \
--mnemonic "seed phrase" \
--admin-auth-token "imadmin" \
--free-query-auth-token "imafriend" \
--bundles "QmHash00:./example-file/,QmHash01:BUNDLE_PATH"
```
Run `cargo run -p file-exchange --help` for more configurations and the corresponding ENV variable names.
Run `file-exchange --help` for more configurations and the corresponding ENV variable names.

3. Access the server via the **admin** endpoint.

Expand Down

0 comments on commit dc67ebd

Please sign in to comment.