From f25ffd561a1150755025b2d9054702a54f4ba0b2 Mon Sep 17 00:00:00 2001 From: Linus Kendall Date: Mon, 6 Nov 2023 07:46:56 +0530 Subject: [PATCH] Add more docs about production server (#68) Adds some more docs about production server user. --- site/production-server.markdown | 97 +++++++++++++++++++++++++++++++++ site/rpc-server.markdown | 77 -------------------------- 2 files changed, 97 insertions(+), 77 deletions(-) create mode 100644 site/production-server.markdown diff --git a/site/production-server.markdown b/site/production-server.markdown new file mode 100644 index 00000000..480683ad --- /dev/null +++ b/site/production-server.markdown @@ -0,0 +1,97 @@ +--- +layout: page +title: Production server +permalink: /rpc-server/production/ +parent: RPC server +nav_order: 2 +--- + +# Production server + +The production server comes with multiple features that the Quickstart server doesn't. It allows you to serve multiple (or all) epochs, with support for dynamically adding new epochs during runtime. You can also make multiple different filecoin settings, such as using a whitelisted provider or excluding certain Filecoin storage providers. + +The production server is available via the `faithful-cli rpc` command. + +## Configuration files + +To run a Faithful RPC server you need to specify configuration files for the epoch(s) you want to host. For multi-epoch support you need to generate epoch config files for the epochs that you want to host. An epoch config file looks like this: + +``` +data: + car: + uri: /faithful/493/epoch-493.car + filecoin: + enable: false +epoch: 493 +indexes: + cid_to_offset: + uri: /faithful/493/epoch-493.car.bafyreidlbcsg46dn5mqppioijyqb5cn6j23rkcoazl7skif74kpa3lihxa.cid-to-offset.index + sig_to_cid: + uri: /faithful/493/epoch-493.car.bafyreidlbcsg46dn5mqppioijyqb5cn6j23rkcoazl7skif74kpa3lihxa.sig-to-cid.index + slot_to_cid: + uri: /faithful/493/epoch-493.car.bafyreidlbcsg46dn5mqppioijyqb5cn6j23rkcoazl7skif74kpa3lihxa.slot-to-cid.index + sig_exists: + uri: /faithful/493/epoch-493.car.bafyreidlbcsg46dn5mqppioijyqb5cn6j23rkcoazl7skif74kpa3lihxa.sig-exists.index +``` + +The `uri` parameter supports both HTTP URIs as well as file based ones. + +If you want you can also run the RPC server using some (or all) epochs via Filecoin: + +``` +data: + filecoin: + enable: true + root_cid: bafyreigq7w4bwspbsf7j4ykov34fcf6skrn663n4ywfalgxlhp7o5nes5a +epoch: 494 +indexes: + cid_to_offset: + uri: /faithful/494/epoch-494.car.bafyreigq7w4bwspbsf7j4ykov34fcf6skrn663n4ywfalgxlhp7o5nes5a.cid-to-offset.index + sig_to_cid: + uri: /faithful/494/epoch-494.car.bafyreigq7w4bwspbsf7j4ykov34fcf6skrn663n4ywfalgxlhp7o5nes5a.sig-to-cid.index + slot_to_cid: + uri: /faithful/494/epoch-494.car.bafyreigq7w4bwspbsf7j4ykov34fcf6skrn663n4ywfalgxlhp7o5nes5a.slot-to-cid.index + sig_exists: + uri: /faithful/494/epoch-494.car.bafyreigq7w4bwspbsf7j4ykov34fcf6skrn663n4ywfalgxlhp7o5nes5a.sig-exists.index +``` + +This requires specifying the root CID for the epoch. You can retrieve the root CID for any epoch from `https://files.old-faithful.net//epoch-.cid` e.g. `https://files.old-faithful.net/489/epoch-489.cid`. You can mix and match epochs hosted in different ways, but each epoch can only be served from a single source. + +In this config file you can optionally add `gsfa` to point to the directory of the gsfa index for the epoch. + +We host sample config files for each epoch under the path `https://files.old-faithful.net//.yml` which will point at indexes and CAR files hosted remotely at https://files.old-faithful.net. You will typically want to at least clone the indexes locally if you want to use this for any production purposes. + +## Proxying support + +The production RPC server provides a proxy mode which allows it to forward traffic it can't serve to a downstream RPC server. To configure this, simply provide the command line argument `--proxy` pointing it to a regular Solana RPC node. + +## Split routing + +The production RPC server can be deployed in split-routing mode via a Layer 7 proxy such as Nginx, Envoy or HAproxy. In this mode, you would forward requests for recent blocks or transactions to a regular Solana RPC server, and then forward requests for historical blocks or transactions to Old Faithful. + +Our recommendation is that you configure the Solana validator to keep at least 2 days of local history (approximately 1 bn shreds) but **without** access to a Bigtable history store. You can then first let requests be routed to the Solana validator and only if those fail to be found, forward the request to the Faithful RPC. + +Another option can be to configure faithful to sit as the first line proxy and then use the built in proxy mode to forward RPC traffic it cannot service downstream to a vanilla Solana RPC node. + + +## Systemd unit + +A sample systemd unit that looks for epoch config files in `/etc/faithful/epochs` is listed below. You can start this up and then add the epoch yaml configs into this folder to support new epochs. + +``` +[Unit] +Description=Faithful RPC server +After=network-online.target +Wants=network-online.target + +[Service] +Type=simple +ExecStart=/usr/local/bin/faithful-cli rpc --listen ":8899" --watch -p -vv /etc/faithful/epochs +Restart=always +LimitNOFILE=1500000 +LimitNPROC=2000000 + + +[Install] +WantedBy=multi-user.target +``` diff --git a/site/rpc-server.markdown b/site/rpc-server.markdown index 2648484c..a3ea1853 100644 --- a/site/rpc-server.markdown +++ b/site/rpc-server.markdown @@ -40,80 +40,3 @@ $ ./tools/run-rpc-server-remote.sh 475 While this is a possible runtime scenario, for blocks this is very, very slow. We recommend downloading indexes locally but keeping the data remote for good performance. See this more as a demonstration that a fully remote use scenario is theoretically possible. More details about how to run with local indexes can be found [here](/rpc-server/old-faithful-net/). -## Production server - -The production server comes with multiple features that the Quickstart server doesn't. It allows you to serve multiple (or all) epochs, with support for dynamically adding new epochs during runtime. You can also make multiple different filecoin settings, such as using a whitelisted provider or excluding certain Filecoin storage providers. - -The production server is available via the `faithful-cli rpc` command. - -### Configuration files - -To run a Faithful RPC server you need to specify configuration files for the epoch(s) you want to host. For multi-epoch support you need to generate epoch config files for the epochs that you want to host. An epoch config file looks like this: - -``` -data: - car: - uri: /faithful/493/epoch-493.car - filecoin: - enable: false -epoch: 493 -indexes: - cid_to_offset: - uri: /faithful/493/epoch-493.car.bafyreidlbcsg46dn5mqppioijyqb5cn6j23rkcoazl7skif74kpa3lihxa.cid-to-offset.index - sig_to_cid: - uri: /faithful/493/epoch-493.car.bafyreidlbcsg46dn5mqppioijyqb5cn6j23rkcoazl7skif74kpa3lihxa.sig-to-cid.index - slot_to_cid: - uri: /faithful/493/epoch-493.car.bafyreidlbcsg46dn5mqppioijyqb5cn6j23rkcoazl7skif74kpa3lihxa.slot-to-cid.index - sig_exists: - uri: /faithful/493/epoch-493.car.bafyreidlbcsg46dn5mqppioijyqb5cn6j23rkcoazl7skif74kpa3lihxa.sig-exists.index -``` - -The `uri` parameter supports both HTTP URIs as well as file based ones. - -If you want you can also run the RPC server using some (or all) epochs via Filecoin: - -``` -data: - filecoin: - enable: true - root_cid: bafyreigq7w4bwspbsf7j4ykov34fcf6skrn663n4ywfalgxlhp7o5nes5a -epoch: 494 -indexes: - cid_to_offset: - uri: /faithful/494/epoch-494.car.bafyreigq7w4bwspbsf7j4ykov34fcf6skrn663n4ywfalgxlhp7o5nes5a.cid-to-offset.index - sig_to_cid: - uri: /faithful/494/epoch-494.car.bafyreigq7w4bwspbsf7j4ykov34fcf6skrn663n4ywfalgxlhp7o5nes5a.sig-to-cid.index - slot_to_cid: - uri: /faithful/494/epoch-494.car.bafyreigq7w4bwspbsf7j4ykov34fcf6skrn663n4ywfalgxlhp7o5nes5a.slot-to-cid.index - sig_exists: - uri: /faithful/494/epoch-494.car.bafyreigq7w4bwspbsf7j4ykov34fcf6skrn663n4ywfalgxlhp7o5nes5a.sig-exists.index -``` - -This requires specifying the root CID for the epoch. You can retrieve the root CID for any epoch from `https://files.old-faithful.net//epoch-.cid` e.g. `https://files.old-faithful.net/489/epoch-489.cid`. You can mix and match epochs hosted in different ways, but each epoch can only be served from a single source. - -In this config file you can optionally add `gsfa` to point to the directory of the gsfa index for the epoch. - -We host sample config files for each epoch under the path `https://files.old-faithful.net//.yml` which will point at indexes and CAR files hosted remotely at https://files.old-faithful.net. You will typically want to at least clone the indexes locally if you want to use this for any production purposes. - - -### Systemd unit - -A sample systemd unit that looks for epoch config files in `/etc/faithful/epochs` is listed below. You can start this up and then add the epoch yaml configs into this folder to support new epochs. - -``` -[Unit] -Description=Faithful RPC server -After=network-online.target -Wants=network-online.target - -[Service] -Type=simple -ExecStart=/usr/local/bin/faithful-cli rpc --listen ":8899" --watch -p -vv /etc/faithful/epochs -Restart=always -LimitNOFILE=1500000 -LimitNPROC=2000000 - - -[Install] -WantedBy=multi-user.target -```