Updating documentation (#61)

site/indexes.markdown

@@ -13,6 +13,7 @@ Indexes will be needed to map Solana's block numbers, transaction signatures and
  - slot-to-cid: Lookup a CID based on a slot number
  - tx-to-cid: Lookup a CID based on a transaction signature
  - cid-to-offset: Index for a specific CAR file, used by the local rpc server (see above) to find CIDs in a car file
+ - sig-exists: Index for quick checking whether a specific signature exists in an epoch or not.
 
 In addition to these Old Faithful supports an index called `gsfa` that maps Solana addresses to a list of transaction signatures.
 
@@ -36,16 +37,23 @@ COMMANDS:
    sig-to-cid
    all
    gsfa
+   sig-exists     
    help, h        Shows a list of commands or help for one command
 
 OPTIONS:
    --help, -h  show help
 ```
 
-For example, to generate the three indexes cid-to-offset, slot-to-cid, sig-to-cid you would run:
+For example, to generate the three indexes cid-to-offset, slot-to-cid, sig-to-cid, sig-exists you would run:
 
 ```
 faithful-cli index all epoch-107.car .
 ```
 
-This would generate the indexes in the current dir for epoch-107.
+This would generate the indexes in the current dir for epoch-107. 
+
+The optional GSFA index would need to be run separately as follows:
+
+```
+faithful-cli index gsfa epoch-107.car .
+```

site/roadmap.markdown

@@ -7,7 +7,8 @@ nav_order: 6
 
 # Roadmap
 
-- Multi-epoch indexing and retrievals
-- Full archives on filecoin and old-faithful.net
-- Continuous uploads of new epochs to Filecoin and old-faithful.net
+For a full list of open issues, see our github - https://github.com/rpcpool/yellowstone-faithful/issues.
+
 - Root index/IPNS for all solana history
+- Support for jsonParsed
+

site/rpc-server.markdown

@@ -23,6 +23,8 @@ It can query data from multiple locations:
 
 ## Quickstart: RPC server from old-faithful.net
 
+The Quickstart RPC server is useful for testing and development. It allows you to spin up an RPC server that hosts a single epoch, either from Filecoin or from CAR files/indexes hosted locally on filesystem or on a separate HTTP endpoint. The quickstart server is available with the commands `faithful-cli rpc-server-car` and `faithful-cli rpc-server-filecoin`.  
+
 We are hosting data on old-faithful.net for testing and cloning purposes. This allows you to run a sample test server without downloading any data. There is a simple tool that you can run for this available from [https://github.com/rpcpool/yellowstone-faithful/tree/main/tools](https://github.com/rpcpool/yellowstone-faithful/tree/main/tools). You can run a fully remote server like this:
 
 ```
@@ -38,4 +40,80 @@ $ ./tools/run-rpc-server-remote.sh 475
 
 While this is a possible runtime scenario, for blocks this is very, very slow. We recommend downloading indexes locally but keeping the data remote for good performance. See this more as a demonstration that a fully remote use scenario is theoretically possible. More details about how to run with local indexes can be found [here](/rpc-server/old-faithful-net/).
 
-New epochs are being added continuously, but the upload cadence is currently a little irregular due to pipeline development work.
+## Production server
+
+The production server comes with multiple features that the Quickstart server doesn't. It allows you to serve multiple (or all) epochs, with support for dynamically adding new epochs during runtime. You can also make multiple different filecoin settings, such as using a whitelisted provider or excluding certain Filecoin storage providers.
+
+The production server is available via the `faithful-cli rpc`  command.
+
+### Configuration files 
+
+To run a Faithful RPC server you need to specify configuration files for the epoch(s) you want to host. For multi-epoch support you need to generate epoch config files for the epochs that you want to host. An epoch config file looks like this:
+
+```
+data:
+  car:
+    uri: /faithful/493/epoch-493.car
+  filecoin:
+    enable: false
+epoch: 493
+indexes:
+  cid_to_offset:
+    uri: /faithful/493/epoch-493.car.bafyreidlbcsg46dn5mqppioijyqb5cn6j23rkcoazl7skif74kpa3lihxa.cid-to-offset.index
+  sig_to_cid:
+    uri: /faithful/493/epoch-493.car.bafyreidlbcsg46dn5mqppioijyqb5cn6j23rkcoazl7skif74kpa3lihxa.sig-to-cid.index
+  slot_to_cid:
+    uri: /faithful/493/epoch-493.car.bafyreidlbcsg46dn5mqppioijyqb5cn6j23rkcoazl7skif74kpa3lihxa.slot-to-cid.index
+  sig_exists:
+    uri: /faithful/493/epoch-493.car.bafyreidlbcsg46dn5mqppioijyqb5cn6j23rkcoazl7skif74kpa3lihxa.sig-exists.index
+```
+
+The `uri` parameter supports both HTTP URIs as well as file based ones.
+
+If you want you can also run the RPC server using some (or all) epochs via Filecoin:
+
+```
+data:
+  filecoin:
+    enable: true
+    root_cid: bafyreigq7w4bwspbsf7j4ykov34fcf6skrn663n4ywfalgxlhp7o5nes5a
+epoch: 494
+indexes:
+  cid_to_offset:
+    uri: /faithful/494/epoch-494.car.bafyreigq7w4bwspbsf7j4ykov34fcf6skrn663n4ywfalgxlhp7o5nes5a.cid-to-offset.index
+  sig_to_cid:
+    uri: /faithful/494/epoch-494.car.bafyreigq7w4bwspbsf7j4ykov34fcf6skrn663n4ywfalgxlhp7o5nes5a.sig-to-cid.index
+  slot_to_cid:
+    uri: /faithful/494/epoch-494.car.bafyreigq7w4bwspbsf7j4ykov34fcf6skrn663n4ywfalgxlhp7o5nes5a.slot-to-cid.index
+  sig_exists:
+    uri: /faithful/494/epoch-494.car.bafyreigq7w4bwspbsf7j4ykov34fcf6skrn663n4ywfalgxlhp7o5nes5a.sig-exists.index
+```
+
+This requires specifying the root CID for the epoch. You can retrieve the root CID for any epoch from `https://files.old-faithful.net/<epoch>/epoch-<epoch>.cid` e.g. `https://files.old-faithful.net/489/epoch-489.cid`. You can mix and match epochs hosted in different ways, but each epoch can only be served from a single source. 
+
+In this config file you can optionally add `gsfa` to point to the directory of the gsfa index for the epoch.
+
+We host sample config files for each epoch under the path `https://files.old-faithful.net/<epoch>/<epoch>.yml` which will point at indexes and CAR files hosted remotely at https://files.old-faithful.net. You will typically want to at least clone the indexes locally if you want to use this for any production purposes. 
+
+
+### Systemd unit
+
+A sample systemd unit that looks for epoch config files in `/etc/faithful/epochs` is listed below. You can start this up and then add the epoch yaml configs into this folder to support new epochs.
+
+```
+[Unit]
+Description=Faithful RPC server
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=simple
+ExecStart=/usr/local/bin/faithful-cli rpc --listen ":8899"  --watch -p -vv /etc/faithful/epochs                                                                                  
+Restart=always
+LimitNOFILE=1500000
+LimitNPROC=2000000
+
+
+[Install]
+WantedBy=multi-user.target
+```