Fix process-compose demo (#1654)

Fixes loading of the seeded .env before running hydra-node processes. Also fixes issues with hydra-node persistence and aligns the configuration closer to the demo/docker-compose.yaml. This will now copy relevant files from the nix store in "prepare-devnet.sh" so this can run from outside a hydra working copy using: ``` nix run github:cardano-scaling/hydra#demo ```
cardano-scaling · Sep 24, 2024 · 2a36ef7 · 2a36ef7
1 parent b1c3914
commit 2a36ef7
Show file tree

Hide file tree

Showing 9 changed files with 63 additions and 41 deletions.
diff --git a/.gitignore b/.gitignore
@@ -30,10 +30,6 @@ hspec-results.md
 !index.html
 
 # Getting started / demo instructions
+/demo/.env
 /demo/devnet/
-/demo/persistence/
 /docs/static/haddock/
-
-# demo
-/devnet
-.env
diff --git a/demo/.envrc b/demo/.envrc
@@ -1 +1,4 @@
 use flake .#demo
+
+export CARDANO_NODE_SOCKET_PATH=devnet/node.socket
+export CARDANO_NODE_NETWORK_ID=42
diff --git a/demo/prepare-devnet.sh b/demo/prepare-devnet.sh
@@ -7,15 +7,18 @@ set -eo pipefail
 BASEDIR=${BASEDIR:-$(realpath $(dirname $(realpath $0))/..)}
 TARGETDIR=${TARGETDIR:-devnet}
 
-[ -d "$TARGETDIR" ] && { echo "Cleaning up directory $TARGETDIR" ; rm -r $TARGETDIR ; }
+[ -d "$TARGETDIR" ] && { echo "Cleaning up directory $TARGETDIR" ; rm -rf $TARGETDIR ; }
 
-cp -af "$BASEDIR/hydra-cluster/config/devnet/" "$TARGETDIR"
+cp -af "$BASEDIR/hydra-cluster/config/devnet" "$TARGETDIR"
+chmod u+w -R "$TARGETDIR"
 cp -af "$BASEDIR/hydra-cluster/config/credentials" "$TARGETDIR"
+chmod u+w -R "$TARGETDIR"
+
 echo '{"Producers": []}' > "$TARGETDIR/topology.json"
 sed -i.bak "s/\"startTime\": [0-9]*/\"startTime\": $(date +%s)/" "$TARGETDIR/genesis-byron.json" && \
 sed -i.bak "s/\"systemStart\": \".*\"/\"systemStart\": \"$(date -u +%FT%TZ)\"/" "$TARGETDIR/genesis-shelley.json"
 
 find $TARGETDIR -type f -name '*.skey' -exec chmod 0400 {} \;
+
 mkdir "$TARGETDIR/ipc"
 echo "Prepared devnet, you can start the cluster now"
-
diff --git a/demo/seed-devnet.sh b/demo/seed-devnet.sh
@@ -12,7 +12,7 @@ if [[ -n ${1} ]]; then
     echo >&2 "Using provided cardano-cli command: ${1}"
     $(${1} version > /dev/null)
     CCLI_CMD=${1}
-    DEVNET_DIR=${SCRIPT_DIR}/devnet
+    DEVNET_DIR=devnet
 fi
 
 HYDRA_NODE_CMD=

diff --git a/docs/docs/getting-started-without-docker.md b/docs/docs/getting-started-without-docker.md
@@ -9,7 +9,11 @@ import Tabs from '@theme/Tabs';
 import TabItem from '@theme/TabItem';
 ```
 
-> Running the demo without Docker containers, but with plain executables and scripts.
+This tutorial guides you through the same setup as the [docker-based one](./getting-started), but without using Docker containers and only using executables and scripts.
+
+:::info Shortcut using nix
+All steps of this tutorial are also available in a combined [process-compose](https://github.com/F1bonacc1/process-compose) setup via `nix run .#demo`
+:::
 
 ## Preparation
 
@@ -18,16 +22,21 @@ Make sure that you have a `cardano-node`, `hydra-node` and `hydra-tui` executabl
 - use `nix develop .#demo` or
 - `cabal build` and `cabal exec` them (do not forget the `--` before passing further arguments).
 
-:::info Tip for tmux users
-In the `demo` nix shell, there is a `run-tmux` script which starts a new `tmux` session with multiple windows and panes executing all the commands below!
-:::
+All further commands are written as if executed from the `demo` folder in the project repository:
 
-All further commands are written as if executed from the `demo` folder in the project repository, so make sure to `cd demo` before continuing.
+```shell
+cd demo
+```
 
 :::info Tip for nix-direnv users
 Allowing the `demo/.envrc` will ensure you have the nix shell environment available whenever you are in the `demo/` directory. To use this, opt-in via `direnv allow` after `cd demo`.
 :::
 
+:::info Tmux shortcut
+In the `demo` nix shell, there is a `run-tmux` script which starts a new `tmux` session with multiple windows and panes executing all the commands below!
+:::
+
+
 ## Set up the devnet
 
 First, let's prepare and start an ad-hoc, single `cardano-node` devnet using our configuration. Note that this will create a `devnet` directory in your current working directory:

diff --git a/docs/docs/getting-started.md b/docs/docs/getting-started.md
@@ -5,7 +5,7 @@ To get started quickly, we'll walk you through the standard demo setup, which in
 * A single `cardano-node` producing blocks used as a (very fast) local `devnet`
 * The `hydra-tui` example for clients to interact with the individual `hydra-node`.
 
-This tutorial uses [Docker](https://www.docker.com/get-started) to install the nodes, so ensure Docker is installed. If you want to explore alternative ways of running the tools, see the [variant tutorial](./getting-started-without-docker.md) or the [testnet tutorial](./tutorial/index.md), which uses pre-built binaries. The documentation pages on [installation](./installation) and [configuration](./configuration) provide more details.
+This tutorial uses [Docker](https://www.docker.com/get-started) to install the nodes, so ensure Docker is installed. If you want to explore alternative ways of running the tools, see a [variant of this tutorial](./getting-started-without-docker.md) or the [testnet tutorial](./tutorial/index.md), which uses pre-built binaries. The documentation pages on [installation](./installation) and [configuration](./configuration) provide more details.
 
 Additionally, the `hydra-tui` uses the HTTP/WebSocket API provided by the `hydra-node` behind the scenes. The [testnet tutorial](./tutorial/index.md) will show how to use this API using low-level commands, or you can see the [API reference](https://hydra.family/head-protocol/unstable/api-reference) for more details.
 

diff --git a/docs/sidebars.js b/docs/sidebars.js
@@ -10,6 +10,11 @@ module.exports = {
       className: "sidebar-header",
     },
     "getting-started",
+    {
+      type: "doc",
+      label: "... without Docker",
+      id: "getting-started-without-docker",
+    },
     "tutorial/index",
     {
       type: "html",

diff --git a/flake.nix b/flake.nix
@@ -135,7 +135,6 @@
             };
           process-compose."demo" = import ./nix/hydra/demo.nix {
             inherit system pkgs inputs self;
-            demoDir = ./demo;
             inherit (pkgsLatest) process-compose;
             inherit (pkgs) cardano-node cardano-cli;
             inherit (hydraPackages) hydra-node hydra-tui;

diff --git a/nix/hydra/demo.nix b/nix/hydra/demo.nix
@@ -8,24 +8,20 @@
 , hydra-node
 , hydra-tui
 , self
-, demoDir
 , process-compose
 }: {
-  # httpServer.enable = true;
   package = process-compose;
   settings = {
-    log_location = "./devnet/logs/all.log";
+    log_location = "devnet/logs/process-compose.log";
     log_level = "debug";
     environment = {
-      BASEDIR = "./";
-      SCRIPT_DIR = "./";
       CARDANO_NODE_SOCKET_PATH = "devnet/node.socket";
       CARDANO_NODE_NETWORK_ID = "42";
     };
 
     processes = {
       prepare-devnet = {
-        command = "${demoDir}/prepare-devnet.sh";
+        command = "${self}/demo/prepare-devnet.sh";
       };
       cardano-node = {
         command = ''
@@ -42,85 +38,94 @@
         depends_on."prepare-devnet".condition = "process_completed";
       };
       seed-devnet = {
-        command = "${demoDir}/seed-devnet.sh ${cardano-cli}/bin/cardano-cli ${hydra-node}/bin/hydra-node";
+        command = ''
+          ${self}/demo/seed-devnet.sh ${cardano-cli}/bin/cardano-cli ${hydra-node}/bin/hydra-node
+        '';
         depends_on."cardano-node".condition = "process_log_ready";
       };
       hydra-node-alice = {
         command = ''
-          source .env &&
+          # (Re-)Export all variables from .env
+          set -a; source .env; set +a
           ${hydra-node}/bin/hydra-node \
           --node-id 1 \
           --port 5001 \
           --api-port 4001 \
           --monitoring-port 6001 \
           --peer 127.0.0.1:5002 \
           --peer 127.0.0.1:5003 \
-          --hydra-signing-key ${demoDir}/alice.sk \
-          --hydra-verification-key ${demoDir}/bob.vk \
-          --hydra-verification-key ${demoDir}/carol.vk \
-          --hydra-scripts-tx-id  $HYDRA_SCRIPTS_TX_ID \
+          --hydra-signing-key ${self}/demo/alice.sk \
+          --hydra-verification-key ${self}/demo/bob.vk \
+          --hydra-verification-key ${self}/demo/carol.vk \
+          --hydra-scripts-tx-id $$HYDRA_SCRIPTS_TX_ID \
           --cardano-signing-key devnet/credentials/alice.sk \
           --cardano-verification-key devnet/credentials/bob.vk \
           --cardano-verification-key devnet/credentials/carol.vk \
           --ledger-protocol-parameters devnet/protocol-parameters.json \
           --testnet-magic 42 \
           --node-socket devnet/node.socket \
-          --persistence-dir devnet
+          --persistence-dir devnet/persistence/alice \
+          --contestation-period 3
         '';
         ready_log_line = "NodeIsLeader";
         depends_on."seed-devnet".condition = "process_completed";
       };
       hydra-node-bob = {
         command = ''
-          source .env &&
+          # (Re-)Export all variables from .env
+          set -a; source .env; set +a
           ${hydra-node}/bin/hydra-node \
           --node-id 2 \
           --port 5002 \
           --api-port 4002 \
           --monitoring-port 6002 \
           --peer 127.0.0.1:5001 \
           --peer 127.0.0.1:5003 \
-          --hydra-signing-key ${demoDir}/bob.sk \
-          --hydra-verification-key ${demoDir}/alice.vk \
-          --hydra-verification-key ${demoDir}/carol.vk \
-          --hydra-scripts-tx-id  $HYDRA_SCRIPTS_TX_ID \
+          --hydra-signing-key ${self}/demo/bob.sk \
+          --hydra-verification-key ${self}/demo/alice.vk \
+          --hydra-verification-key ${self}/demo/carol.vk \
+          --hydra-scripts-tx-id $$HYDRA_SCRIPTS_TX_ID \
           --cardano-signing-key devnet/credentials/bob.sk \
           --cardano-verification-key devnet/credentials/alice.vk \
           --cardano-verification-key devnet/credentials/carol.vk \
           --ledger-protocol-parameters devnet/protocol-parameters.json \
           --testnet-magic 42 \
           --node-socket devnet/node.socket \
-          --persistence-dir devnet
+          --persistence-dir devnet/persistence/bob \
+          --contestation-period 3
         '';
         ready_log_line = "NodeIsLeader";
         depends_on."seed-devnet".condition = "process_completed";
       };
       hydra-node-carol = {
         command = ''
-          source .env &&
+          # (Re-)Export all variables from .env
+          set -a; source .env; set +a
           ${hydra-node}/bin/hydra-node \
           --node-id 3 \
           --port 5003 \
           --api-port 4003 \
           --monitoring-port 6003 \
           --peer 127.0.0.1:5001 \
           --peer 127.0.0.1:5002 \
-          --hydra-signing-key ${demoDir}/carol.sk \
-          --hydra-verification-key ${demoDir}/alice.vk \
-          --hydra-verification-key ${demoDir}/bob.vk \
-          --hydra-scripts-tx-id  $HYDRA_SCRIPTS_TX_ID \
+          --hydra-signing-key ${self}/demo/carol.sk \
+          --hydra-verification-key ${self}/demo/alice.vk \
+          --hydra-verification-key ${self}/demo/bob.vk \
+          --hydra-scripts-tx-id $$HYDRA_SCRIPTS_TX_ID \
           --cardano-signing-key devnet/credentials/carol.sk \
           --cardano-verification-key devnet/credentials/alice.vk \
           --cardano-verification-key devnet/credentials/bob.vk \
           --ledger-protocol-parameters devnet/protocol-parameters.json \
           --testnet-magic 42 \
           --node-socket devnet/node.socket \
-          --persistence-dir devnet
+          --persistence-dir devnet/persistence/carol \
+          --contestation-period 3
         '';
         ready_log_line = "NodeIsLeader";
         depends_on."seed-devnet".condition = "process_completed";
       };
       hydra-tui-alice = {
+        working_dir = "./demo";
         command = ''
           ${hydra-tui}/bin/hydra-tui \
             --connect 0.0.0.0:4001 \
@@ -132,6 +137,7 @@
         depends_on."hydra-node-alice".condition = "process_started";
       };
       hydra-tui-bob = {
+        working_dir = "./demo";
         command = ''
           ${hydra-tui}/bin/hydra-tui \
             --connect 0.0.0.0:4002 \
@@ -143,6 +149,7 @@
         depends_on."hydra-node-bob".condition = "process_started";
       };
       hydra-tui-carol = {
+        working_dir = "./demo";
         command = ''
           ${hydra-tui}/bin/hydra-tui \
             --connect 0.0.0.0:4003 \