Merge branch 'main' into july2024/improve-gsfa-perf

.devcontainer/devcontainer.json

@@ -0,0 +1,6 @@
+{
+  "image": "mcr.microsoft.com/devcontainers/universal:2",
+  "features": {
+    "ghcr.io/devcontainers/features/go:1": {}
+  }
+}

.github/workflows/build-release.yml

@@ -10,12 +10,12 @@ jobs:
     runs-on: ubuntu-20.04
 
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
         with:
           fetch-depth: 0
 
       - name: Setup go env
-        uses: actions/setup-go@v3
+        uses: actions/setup-go@v5
         with:
           go-version: '1.21'
           check-latest: true
@@ -24,7 +24,7 @@ jobs:
         run: make compile-all
 
       - name: Publish to release
-        uses: softprops/action-gh-release@v1
+        uses: softprops/action-gh-release@v2
         if: startsWith(github.ref, 'refs/tags/')
         with:
           files: |

.github/workflows/jekyll-gh-pages.yml

@@ -30,21 +30,21 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
       - name: Setup Ruby
         uses: ruby/setup-ruby@v1
         with:
           bundler-cache: true # runs 'bundle install' and caches installed gems automatically
           cache-version: 0 # Increment this number if you need to re-download cached gems
       - name: Setup Pages
-        uses: actions/configure-pages@v3
+        uses: actions/configure-pages@v5
       - name: Build with Jekyll
         # Outputs to the './_site' directory by default
         run: bundle exec jekyll build --source ./site --baseurl "${{ steps.pages.outputs.base_path }}"
         env:
           JEKYLL_ENV: production
       - name: Upload artifact
-        uses: actions/upload-pages-artifact@v2
+        uses: actions/upload-pages-artifact@v3
 
   # Deployment job
   deploy:
@@ -56,5 +56,5 @@ jobs:
     steps:
       - name: Deploy to GitHub Pages
         id: deployment
-        uses: actions/deploy-pages@v2
+        uses: actions/deploy-pages@v4
 

.github/workflows/tests.yml

@@ -19,11 +19,11 @@ jobs:
     runs-on: ${{ matrix.os }}
     steps:
       - name: Install Go
-        uses: actions/setup-go@v3
+        uses: actions/setup-go@v5
         with:
           go-version: ${{ matrix.go-version }}
       - name: Checkout code
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
         with:
           fetch-depth: 0
       - name: Test

README.md

@@ -4,7 +4,7 @@
 
 This is currently in RFC stage, which means that it is not intended for production use and that there may be breaking changes to the format, the CLI utilities or any other details related to the project.
 
-For more documentation, please visit [https://old-faithful.net](https://old-faithful.net).
+For more documentation, please visit [https://docs.old-faithful.net](https://docs.old-faithful.net).
 
 > ❗ **Request for comment**: We are currently looking for feedback and comments on the new archival format and the RPC server setup. We invite all interested parties to test the archival access and open issues on this repo with questions/comments/requests for improvements.
 
@@ -110,113 +110,6 @@ Flags:
 - `--verify`: Verify the indexes after generation. (optional)
 - `--network=<network>`: Which network to use for the gsfa index. Defaults to `mainnet` (other options: `testnet`, `devnet`). (optional)
 
-## RPC server proxying
-
-The RPC server provides a proxy mode which allows it to forward traffic it can't serve to a downstream RPC server. To configure this, simply provide the command line argument `--proxy=/path/to/faithful-proxy-config.json` pointing it to a config file. The config file should look like this:
-
-```json
-{
-	"target": "https://api.mainnet-beta.solana.com",
-	"headers": {
-		"My-Header": "My-Value"
-	},
-	"proxyFailedRequests": true
-}
-```
-
-The `proxyFailedRequests` flag will make the RPC server proxy not only RPC methods that it doesn't support, but also retry requests that failed to be served from the archives (e.g. a `getBlock` request that failed to be served from the archives because that epoch is not available).
-
-### Log Levels
-
-You can set the desired log verbosity level by using the `-v` flag. The levels are from 0 to 5, where 0 is the least verbose and 5 is the most verbose. The default level is 2.
-
-Example:
-
-```bash
-faithful-cli rpc -v=5 455.yml
-```
-
-### RPC server from old-faithful.net
-
-We are hosting data on old-faithful.net for testing and cloning purposes. This allows you to run a sample test server without downloading any data. You can run a fully remote server like this:
-
-```
-$ ./tools/run-rpc-server-remote.sh 0
-```
-
-This will create a server that hosts epoch 0.
-
-### RPC server with local indexes
-
-For ongoing testing, we strongly recommend that you download at least the indexes for best performance. If you have local indexes downloaded you can use the following helper script:
-
-```
-$ ./tools/run-rpc-server-local-indexes.sh 0 ./epoch0
-```
-
-There is a utility script in the `tools` folder that will download the indexes hosted on old-faithful.net. The indexes will also be available on Filecoin.
-
-```
-$ mkdir epoch0
-$ cd epoch0
-$ ../tools/download-indexes.sh 0 ./epoch0
-$ ../tools/download-gsfa.sh 0 ./epoch0
-```
-
-### RPC server running fully locally
-
-If you have a local copy of a CAR archive and the indexes and run a RPC server servicing data from them. For example:
-
-```bash
-/usr/local/bin/faithful-cli rpc \
-    --listen $PORT \
-    /path/to/epoch-455.yml
-```
-
-You can download the CAR files either via Filecoin or via the bucket provided by Triton. There are helper scripts in the `tools` folder. To download the full epoch data:
-
-```bash
-$ mkdir epoch0
-$ cd epoch0
-$ ../tools/download-epoch.sh 0
-$ ../tools/download-indexes.sh 0
-$ ../tools/download-gsfa.sh 0
-```
-
-Once files are downloaded there are also utility scripts to run the server:
-```bash
-$ ./tools/run-rpc-server-local.sh 0 ./epoch0
-```
-
-This will host epoch 0 from the data available in the folder epoch0.
-
-### Filecoin RPC server
-
-The filecoin RPC server allows provide getBlock, getTransaction and getSignaturesForAddress powered by Filecoin. This requires access to indexes. The indexes allow you to lookup transaction signatures, block numbers and addresses and map them to Filecoin CIDs.
-
-You can run it in the following way:
-
-```bash
-faithful-cli rpc 455.yml
-```
-
-The config file points faithful to the location of the required indexes (`455.yaml`):
-```bash
-indexes:
-  slot_to_cid: './epoch-455.car.bafyreibkequ55hyrhyk6f24ctsofzri6bjykh76jxl3zju4oazu3u3ru7y.slot-to-cid.index'
-  sig_to_cid: './epoch-455.car.bafyreibkequ55hyrhyk6f24ctsofzri6bjykh76jxl3zju4oazu3u3ru7y.sig-to-cid.index'
-  sig_exists: './epoch-455.car.bafyreibkequ55hyrhyk6f24ctsofzri6bjykh76jxl3zju4oazu3u3ru7y.sig-exists.index'
-  gsfa: './epoch-455.car.gsfa.index'
-```
-
-Due to latency in fetching signatures, typically the getSignaturesForAddress index needs to be stored in a local directory, but the other indexes can be fetched via HTTP or via local file system access. If you provide a URL, you need to make sure that the url supports HTTP Range requests. S3 or similar APIs will support this.
-
-There is a mode in which you can use a remote gSFA index, which limits it to only return signatures and not additional transaction meta data. In this mode, you can use a remote gSFA index. To enable this mode run faithful-cli in the following way:
-
-```bash
-faithful-cli rpc -gsfa-only-signatures=true 455.yml 
-```
-
 ### Filecoin fetch via CID
 
 If you already know the CID of the data you are looking for you can fetch it via `faithful-cli fetch <cid>`. This requires no further indexes and can also be used to recursively fetch data for example for an epoch. To avoid fetching the full dataset for an epoch (100s of GB) you probably want to pass the parameter `--dag-scope=block` to fetch only the particular CID entity that you are interested in.

accum/block.go

@@ -190,6 +190,17 @@ func (obj ObjectWithMetadata) RawSection() ([]byte, error) {
 	return buf, nil
 }
 
+func (obj ObjectWithMetadata) RawSectionSize() int {
+	sectionLen := len(obj.Cid.Bytes()) + len(obj.ObjectData)
+	lenBytes := leb128.FromUInt64(uint64(sectionLen))
+
+	// Size is:
+	// length of LEB128-encoded section length +
+	// length of CID bytes +
+	// length of object data
+	return len(lenBytes) + sectionLen
+}
+
 // {
 // 	raw, err := objm.RawSection()
 // 	if err != nil {