From dc67ebdf4b3650f1b897a9f3a8f4f54de3ac95a6 Mon Sep 17 00:00:00 2001 From: hopeyen Date: Tue, 16 Jan 2024 10:28:52 -0600 Subject: [PATCH] docs: chris' suggestions; dockerfile rename --- ...bfile-exchange => Dockerfile.file-exchange | 0 README.md | 23 +++++++----------- docs/architecture.md | 10 +------- docs/client_guide.md | 9 +++---- docs/{subfile-exchange.png => fhs.png} | Bin docs/publisher_guide.md | 8 +++--- docs/server_guide.md | 8 +++--- 7 files changed, 22 insertions(+), 36 deletions(-) rename Dockerfile.subfile-exchange => Dockerfile.file-exchange (100%) rename docs/{subfile-exchange.png => fhs.png} (100%) diff --git a/Dockerfile.subfile-exchange b/Dockerfile.file-exchange similarity index 100% rename from Dockerfile.subfile-exchange rename to Dockerfile.file-exchange diff --git a/README.md b/README.md index 63f91f0..0b42a07 100644 --- a/README.md +++ b/README.md @@ -1,13 +1,10 @@ -# File Exchange +# File Hosting Service ## Introduction -Enable file sharing as a exchange, aim for a decentralized, efficient, and verifiable market, with scalable, performant, and secure software. -File Exchange is a decentralized, peer-to-peer data sharing platform designed for efficient and verifiable file sharing. It leverages a combination of technologies including Hash commitments on IPFS for file discovery and verification, chunk data transfer and micropayments reducing trust requirements between clients and servers, and HTTPS over HTTP2 for secure and efficient data transfer. The system is built with scalability, performance, integrity, and security in mind, aiming to create a robust market for file sharing. - - -File Exchange leverages IPFS for file discovery and verification, ensuring that each piece of data shared is authentic and unaltered. The use of SHA2-256 for hashing provides a balance of speed and security, making the system both fast and impenetrable to known cryptographic attacks. Furthermore, the adoption of HTTPS over HTTP2 with range requests ensures that all data transfers are not only swift but also secure, safeguarding against common internet vulnerabilities and minimizing risks per transaction. +File Hosting Service (FHS) is a marketplace for sharing file data and is part of The Graph Network's [World of Data Services](https://forum.thegraph.com/t/gip-0042-a-world-of-data-services/3761). +FHS is a decentralized, peer-to-peer data sharing platform designed for efficient and trust-minimised file sharing that is payments-enabled. It leverages a combination of technologies including hash commitments on IPFS for file discovery and verification, chunked data transfer and micropayments reducing trust requirements between clients and servers, and secure and efficient data transfers via HTTP2. The system is built with scalability, performance, integrity, and security in mind, aiming to create a robust market for file sharing. ## Target Audience @@ -15,18 +12,17 @@ This documentation is tailored for individuals who have a basic understanding of ## Features -- Decentralized File Sharing: Utilize peer-to-peer networks for direct file transfers, eliminating central points of failure. +- Decentralized File Sharing: FHS uses direct connections for file transfers, eliminating central points of failure. - IPFS Integration: Employ IPFS for efficient and reliable file discovery and content verification. -- SHA2-256 Hashing: Ensure data integrity through robust cryptographic hashing. -- HTTPS over HTTP2: Leverage the latest web protocols for secure and efficient data transfer. +- SHA2-256 Hashing: Ensure data integrity through robust and incremental cryptographic hashing. +- HTTP2 and TLS: Leverage the latest web protocols for secure and efficient data transfer. **To be supported:** - Micropayments Support: Implement a system of micropayments to facilitate fair compensation and reduce trust requirements. - Scalability and Performance: Designed with a focus on handling large volumes of data and high user traffic. - User-Friendly Interface: Intuitive design for easy navigation and operation. -More details can be found in [Feature checklist](docs/feature_checklist.md) - +More details can be found in [Feature Checklist](docs/feature_checklist.md) ## Upgrading @@ -44,12 +40,11 @@ You may learn background information on various components of the exchange 4. **Blockchain**: [World of data services](https://forum.thegraph.com/t/gip-0042-a-world-of-data-services/3761), [flatfiles for Ethereum](https://github.com/streamingfast/firehose-ethereum), [use case](https://eips.ethereum.org/EIPS/eip-4444). - ## Documentation -#### [Design Principle](docs/architecture.md) +#### [Architecture](docs/architecture.md) -#### [Entity Definition](docs/manifest.md) +#### [Entity Definitions](docs/manifest.md) #### [Contracts](docs/contracts.md) diff --git a/docs/architecture.md b/docs/architecture.md index e444b9b..165532b 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,16 +1,8 @@ ## Architecture and Components -### Decentralized Architecture - -Unlike traditional centralized systems where data is stored and managed through a single entity, File Service distributes data across a network of nodes. This decentralization ensures resilience against failures and attacks as there is no single point of failure; it also enhances privacy and control as users are not reliant on a central authority for data management. - -The architecture is underpinned by a peer-to-peer (P2P) network framework, where each node in the network can act as a client and/or a server. This setup facilitates direct file sharing among users without the need for intermediaries, and as more users join and contribute to the network, the system becomes more robust and capable of handling larger volumes of data. - - ### Generic Diagram -![Diagram](./file-exchange.png) - +![Diagram](./fhs.png) ### Key Components diff --git a/docs/client_guide.md b/docs/client_guide.md index dd75e19..7feb953 100644 --- a/docs/client_guide.md +++ b/docs/client_guide.md @@ -1,12 +1,12 @@ # File Sharing Client -Tired of data drudgery? Imagine a world where you skip the tedious task of indexing data and start serving queries in a flash! That's the magic of our P2P file sharing network. Instead of spending hours to months catching up to the head of a chain, you can tap into a vast pool of indexed information, ready and waiting for your queries. +Tired of data drudgery? Imagine a world where you skip the tedious task of indexing data and start serving queries in a flash! That's the magic of our file sharing network. Instead of spending hours to months catching up to the head of a chain, you can tap into a vast pool of indexed information, ready and waiting for your queries. This document provides an overview of the file sharing client. ## Functionality -The client facilitates data retrieval from the P2P file sharing network. Users can specify desired files by their content addressable IDs (CIDs) and utilize various features: +The client facilitates data retrieval from FHS. The client implements: - File retrieval: Download entire files or specific chunks at a time. - Payment: Pay for file access using tokens deposited in an Escrow account on-chain. @@ -25,7 +25,7 @@ After determining the Bundle CID, client should supply a local path for writing ### CLI example ``` -➜ file-exchange git:(main) ✗ cargo run -p file-exchange downloader \ +$ file-exchange downloader \ --ipfs-hash QmHash \ --indexer-endpoints http://localhost:5678,http://localhost:5677 \ --free-query-auth-token 'Bearer auth_token' \ @@ -35,12 +35,11 @@ After determining the Bundle CID, client should supply a local path for writing --provider "arbitrum-sepolia-rpc-endpoint" ``` - ### Requirements To use the client effectively, you will need: -- Content Address: The CID of the desired file. +- Bundle Manifest CID: The CID of the Bundle Manifest you want to download. - Local Path: A directory where the downloaded file will be stored. (Later will be a generic storage path, enabling cloud storage access) - Wallet: A blockchain wallet containing tokens for escrow payments. - Indexer Endpoints: A list of available server addresses. diff --git a/docs/subfile-exchange.png b/docs/fhs.png similarity index 100% rename from docs/subfile-exchange.png rename to docs/fhs.png diff --git a/docs/publisher_guide.md b/docs/publisher_guide.md index 995e0fc..a09559c 100644 --- a/docs/publisher_guide.md +++ b/docs/publisher_guide.md @@ -8,16 +8,16 @@ To start, you would need to provide several configurations ### Requirements -Publisher must have read access to all files contained in the package. The publisher publish 1 Bundle at a time and is not responsible for hosting the file after publishing. The publisher should chunk all the files in the package and generate a hash for all the chunks. Then the publisher will build a hierarchy with the hashes. Currently, publisher simply put chunk hashes in a list for each individual files, publish individual file manifests, then they build a Bundle that contains a list of the file manifest addresses. +Publisher must have read access to all files contained in the Bundle. The publisher publish 1 Bundle at a time and is not responsible for hosting the file after publishing. The publisher should chunk all the files in the package and generate a hash for all the chunks. Then the publisher will build a hierarchy with the hashes. Currently, the Publisher simply put chunk hashes in a list for each individual file, publish individual file manifests, then they build a Bundle that contains a list of the file manifest addresses. > More exploration for hashing/packaging architecture ### CLI example ``` -➜ file-exchange git:(main) ✗ cargo run -p file-exchange publisher \ +$ file-exchange publisher \ --read-dir ./example-file/ \ - --Bundle-name "blah" \ + --bundle-name "blah" \ --file-names example0017686312.dbin,example-create-17686085.dbin \ --file-type flatfiles \ --file-version 0.0.0 \ @@ -27,5 +27,5 @@ Publisher must have read access to all files contained in the package. The publi For more information ``` -➜ file-exchange git:(main) ✗ cargo run -p file-exchange --help +$ file-exchange --help ``` diff --git a/docs/server_guide.md b/docs/server_guide.md index 88da97f..c2c5dad 100644 --- a/docs/server_guide.md +++ b/docs/server_guide.md @@ -2,13 +2,13 @@ You hold the key to unlocking the network's true potential. By sharing your meticulously indexed data, you become the architect of information accessibility. Imagine, your contribution fueling a vibrant ecosystem of knowledge, where fellow indexers can build upon your work, unleashing a torrent of information to the world. In return, your generosity is rewarded with precious tokens, a testament to the invaluable role you play in this decentralized revolution. Become the data hero we need, and together, let us build a brighter future fueled by open access and boundless knowledge! -This document provides an overview of the P2P file sharing server, intended for those familiar with blockchain nodes and large datasets. +This document provides an overview of the file sharing server, intended for those familiar with blockchain nodes and large datasets. Jump to [Quick Start](###getting-started) ## File Transfer Protocol -The server utilizes HTTP2 over HTTPS for secure and efficient file transfer. This ensures data integrity and confidentiality while leveraging the performance benefits of HTTP2. +The server utilizes HTTP2 with TLS for secure and efficient file transfer. This ensures data integrity and confidentiality while leveraging the performance benefits of HTTP2. ## Access Control @@ -53,7 +53,7 @@ The server utilizes a combination of open-source technologies for optimal perfor CLI example ``` -✗ cargo run -p file-exchange server \ +$ file-exchange server \ --host 0.0.0.0 \ --port 5678 \ --mnemonic "seed phrase" \ @@ -61,7 +61,7 @@ CLI example --free-query-auth-token "imafriend" \ --bundles "QmHash00:./example-file/,QmHash01:BUNDLE_PATH" ``` -Run `cargo run -p file-exchange --help` for more configurations and the corresponding ENV variable names. +Run `file-exchange --help` for more configurations and the corresponding ENV variable names. 3. Access the server via the **admin** endpoint.