feat(docs): reorganize top-level menus in online docs (#929)

docs/consuming_tests/index.md

@@ -17,7 +17,7 @@ Here's a top-level comparison of the different methods of consuming tests:
 
 !!! note "Running `blocktest`, `statetest`, directly within the execution-spec-tests framework"
 
-    It's possible to execute `evm blocktest` directly within the execution-spec-tests framework. This is intended to verify fixture generation, see [Debugging `t8n` Tools](../getting_started/debugging_t8n_tools.md).
+    It's possible to execute `evm blocktest` directly within the execution-spec-tests framework. This is intended to verify fixture generation, see [Debugging `t8n` Tools](../filling_tests/debugging_t8n_tools.md).
 
 !!! note "Generating test fixtures using a `t8n` tool via `fill` is not considered to be the actual test"
 

docs/getting_started/executing_tests_command_line.md → docs/filling_tests/filling_tests_command_line.md

@@ -1,4 +1,4 @@
-# Executing Tests at a Prompt
+# Filling Tests at a Prompt
 
 The execution-spec-tests test framework uses the [pytest framework](https://docs.pytest.org/en/latest/) for test case collection and execution. The `fill` command is essentially an alias for `pytest`, which uses several [custom pytest plugins](../library/pytest_plugins/index.md) to run transition tools against test cases and generate JSON fixtures.
 
@@ -33,7 +33,7 @@ fill --collect-only -k warm_coinbase -vv
 
 ## Execution
 
-By default, test cases are executed for all forks already deployed to mainnet, but not for forks still under active development, i.e., as of time of writing, Q2 2023:
+By default, test cases are filled for all forks already deployed to mainnet, but not for forks still under active development, i.e., as of time of writing, Q2 2023:
 
 ```console
 fill
@@ -77,16 +77,16 @@ fill tests/shanghai/eip3651_warm_coinbase/test_warm_coinbase.py::test_warm_coinb
 ## Execution for Development Forks
 
 !!! note ""
-    By default, test cases are not executed with upcoming Ethereum forks so that they can be readily executed against the `evm` tool from the latest `geth` release.
+    By default, test cases are not filled for upcoming Ethereum forks so that they can be readily filled using the `evm` tool from the latest `geth` release.
 
-    In order to execute test cases for an upcoming fork, ensure that the `evm` tool used supports that fork and features under test and use the `--until` or `--fork` flag.
+    In order to fill test cases for an upcoming fork, ensure that the `evm` tool used supports that fork and features under test and use the `--until` or `--fork` flag.
 
     For example, as of Q2 2023, the current fork under active development is `Cancun`:
     ```console
     fill --until Cancun
     ```
 
-    See: [Executing Tests for Features under Development](./executing_tests_dev_fork.md).
+    See: [Filling Tests for Features under Development](./filling_tests_dev_fork.md).
 
 ## Debugging the `t8n` Command
 

docs/getting_started/executing_tests_dev_fork.md → docs/filling_tests/filling_tests_dev_fork.md

@@ -1,4 +1,4 @@
-# Executing Tests for Features under Development
+# Filling Tests for Features under Development
 
 ## Requirements
 
@@ -45,4 +45,4 @@ By default, execution-spec-tests only generates fixtures for forks that have bee
 
 ## VS Code Setup
 
-By default, VS Code's Testing View will only show tests for stable forks. To show tests for development forks, uncomment the relevant line in the `python.testing.pytestArgs` configuration section of included settings file (`.vscode/settings.json`) to enable the `--until=FORK` flag. See [VS Code Setup](./setup_vs_code.md) for help finding the settings files.
+By default, VS Code's Testing View will only show tests for stable forks. To show tests for development forks, uncomment the relevant line in the `python.testing.pytestArgs` configuration section of included settings file (`.vscode/settings.json`) to enable the `--until=FORK` flag. See [VS Code Setup](../getting_started/setup_vs_code.md) for help finding the settings files.

docs/getting_started/executing_tests_vs_code.md → docs/filling_tests/filling_tests_vs_code.md

@@ -1,6 +1,6 @@
-# Executing Tests in VS Code
+# Filling Tests in VS Code
 
-Prerequisite: [VS Code Setup](./setup_vs_code.md).
+Prerequisite: [VS Code Setup](../getting_started/setup_vs_code.md).
 
 ## Exploring Test Cases
 
@@ -11,9 +11,9 @@ Implemented test cases can be explored in VS Code's "Testing" View; click on the
 </figure>
 
 !!! note "Testing EVM Features Under Active Development"
-     See [the VS Code section](./executing_tests_dev_fork.md#executing-tests-for-features-under-development) in [Executing Tests for Features under Development](./executing_tests_dev_fork.md) to explore tests targeting EVM features under development.
+     See [the VS Code section](./filling_tests_dev_fork.md#executing-tests-for-features-under-development) in [Filling Tests for Features under Development](./filling_tests_dev_fork.md) to explore tests targeting EVM features under development.
 
-## Executing and Debugging Test Cases
+## Filling and Debugging Test Cases
 
 <figure markdown>
  ![VS Code Testing Tab](./img/vs_code_executing_tests.png){ width=auto align=center}

docs/getting_started/quick_start.md → docs/filling_tests/getting_started.md

@@ -1,41 +1,6 @@
-# Quick Start
+# Getting Started: Filling Tests
 
-## Prerequisites
-
-The tools provided by [execution-spec-tests](https://github.com/ethereum/execution-spec-tests) use `uv` ([docs.astral.sh/uv](https://docs.astral.sh/uv/)) to manage their dependencies and virtual environment.
-
-`uv` can be installed via curl (recommended; can self-update) or pip (requires Python, can't self-update):
-
-=== "curl"
-
-    ```console
-    curl -LsSf https://astral.sh/uv/install.sh | sh
-    ```
-
-=== "pip"
-
-    ```console
-    pip install uv
-    ```
-
-If installed via `curl`, `uv` will download Python for your target platform if one of the required versions (Python 3.10, 3.11 or 3.12) is not available natively.
-
-## Installation
-
-Clone [execution-spec-tests](https://github.com/ethereum/execution-spec-tests) and install its dependencies:
-
-```console
-git clone https://github.com/ethereum/execution-spec-tests
-cd execution-spec-tests
-uv sync --all-extras
-uv run solc-select use 0.8.24 --always-install
-```
-
-See [Installation Troubleshooting](./installation_troubleshooting.md) if you encounter issues.
-
-## Exploring and Filling Test Cases
-
-By default, JSON test fixtures are generated from this repository's Python test cases using the [Ethereum Execution Layer Specification](https://github.com/ethereum/execution-specs) (EELS) reference implementation. The resulting JSON fixtures can be executed against execution clients to verify consensus. The process of generating fixtures is often referred to as "filling".
+This guide describes how to get started with `fill` and commonly-used command-line options.
 
 1. Explore test cases via `--collect-only` and search for test cases that combine `PUSH0` and `DELEGATECALL` in the EVM functionality introduced in the Shanghai hard fork:
 
@@ -84,13 +49,9 @@ By default, JSON test fixtures are generated from this repository's Python test
     ./fixtures/.meta/report_fill.html
     ```
 
-## Installation Troubleshooting
-
-If you encounter issues during installation, see the [Installation Troubleshooting](./installation_troubleshooting.md) guide.
-
 ## Next Steps
 
-1. Learn [useful command-line flags](./executing_tests_command_line.md).
-2. [Execute tests for features under development](./executing_tests_dev_fork.md) via the `--fork` flag.
-3. _Optional:_ [Configure VS Code](./setup_vs_code.md) to auto-format Python code and [execute tests within VS Code](./executing_tests_vs_code.md#executing-and-debugging-test-cases).
+1. Learn [useful command-line flags](./filling_tests_command_line.md).
+2. [Execute tests for features under development](./filling_tests_dev_fork.md) via the `--fork` flag.
+3. _Optional:_ [Configure VS Code](../getting_started/setup_vs_code.md) to auto-format Python code and [execute tests within VS Code](./filling_tests_vs_code.md#executing-and-debugging-test-cases).
 4. Implement a new test case, see [Writing Tests](../writing_tests/index.md).

docs/filling_tests/index.md

@@ -0,0 +1,30 @@
+# Filling Tests
+
+Execution of test cases against clients is a two-step process:
+
+1. JSON test fixtures are generated from the Python test cases found in `./tests` using `fill` and an EVM transition tool (`t8n`) implementation.
+2. Clients "consume" the JSON fixtures via either a dedicated, client-specific interface or a testing environment such as Hive.
+
+The process of generating fixtures is often referred to as "filling" the tests.
+
+!!! note "The `execute` command"
+
+    The `execute` command directly executes Python test cases against a client via its RPC without using generated JSON fixtures. For all other methods of testing clients, the JSON fixtures are required. For more information, see [Executing Tests](../executing_tests/index.md).
+
+## Transition Tools (`t8n`)
+
+The `fill` command requires an EVM `t8n` tool provided by most clients in order to generate the JSON fixtures. The `t8n` tool is mainly responsible for calculating the post-state of the EVM after executing a transaction, most relevantly, it calculates the updated state root.
+
+## Ethereum Execution Layer Specification (EELS)
+
+By default, the [Ethereum Execution Layer Specification](https://github.com/ethereum/execution-specs) (EELS) reference implementation of the `t8n` tool is used to generate test fixtures for all forks that have been deployed to Ethereum mainnet. We strong encourage EIP authors to provide a reference implementation of their EIP in EELS, so that it can be used to generate test fixtures for features under active development.
+
+## Limitations of Filling
+
+The "fill-consume" method follows a differential testing approach: A reference implementation is used to generate JSON test fixtures, which can then be executed against other EVM clients. However:
+
+!!! warning "Successfully filling does not guarantee correctness"
+
+    Some tests cases, particularly those without straightforward post-checks, such as certain gas calculations, may allow subtle inconsistencies to slip through during filling.
+
+    **Consequently, filling the tests does not ensure the client’s correctness. Clients must consume the tests to be considered correctly tested, even if that client was used to fill the tests.**

docs/filling_tests/transition_tool_support.md

@@ -0,0 +1,12 @@
+# Transition Tool Support
+
+The following transition tools are supported by the framework:
+
+| Client | `t8n` Tool | Tracing Support |
+| -------| ---------- | --------------- |
+| [ethereum/evmone](https://github.com/ethereum/evmone) | `evmone-t8n` | Yes |
+| [ethereum/execution-specs](https://github.com/ethereum/execution-specs) | [`ethereum-spec-evm-resolver`](https://github.com/petertdavies/ethereum-spec-evm-resolver) | Yes |
+| [ethereumjs](https://github.com/ethereumjs/ethereumjs-monorepo) | [`ethereumjs-t8ntool.sh`](https://github.com/ethereumjs/ethereumjs-monorepo/tree/master/packages/vm/test/t8n) | No |
+| [ethereum/go-ethereum](https://github.com/ethereum/go-ethereum) | [`evm t8n`](https://github.com/ethereum/go-ethereum/tree/master/cmd/evm) | Yes |
+| [hyperledger/besu](https://github.com/hyperledger/besu/tree/main/ethereum/evmtool) | [`evmtool t8n-server`](https://github.com/hyperledger/besu/tree/main/ethereum/evmtool) | Yes             |
+| [status-im/nimbus-eth1](https://github.com/status-im/nimbus-eth1) | [`t8n`](https://github.com/status-im/nimbus-eth1/blob/master/tools/t8n/readme.md) | Yes |

docs/getting_started/installation.md

@@ -0,0 +1,36 @@
+# Installation
+
+## Prerequisites
+
+The tools provided by [execution-spec-tests](https://github.com/ethereum/execution-spec-tests) use `uv` ([docs.astral.sh/uv](https://docs.astral.sh/uv/)) to manage their dependencies and virtual environment.
+
+`uv` can be installed via curl (recommended; can self-update) or pip (requires Python, can't self-update):
+
+=== "curl"
+
+    ```console
+    curl -LsSf https://astral.sh/uv/install.sh | sh
+    ```
+
+=== "pip"
+
+    ```console
+    pip install uv
+    ```
+
+If installed via `curl`, `uv` will download Python for your target platform if one of the required versions (Python 3.10, 3.11 or 3.12) is not available natively.
+
+## Installation
+
+Clone [execution-spec-tests](https://github.com/ethereum/execution-spec-tests) and install its dependencies:
+
+```console
+git clone https://github.com/ethereum/execution-spec-tests
+cd execution-spec-tests
+uv sync --all-extras
+uv run solc-select use 0.8.24 --always-install
+```
+
+## Installation Troubleshooting
+
+If you encounter issues during installation, see the [Installation Troubleshooting](./installation_troubleshooting.md) guide.

docs/getting_started/overview.md

@@ -0,0 +1,82 @@
+# Execution Spec Tests
+
+<figure markdown>  <!-- markdownlint-disable MD033 (MD033=no-inline-html) -->
+  ![Execution Spec Tests Logo](img/execution_spec_tests.png){width="400"}
+</figure>
+
+@ethereum/execution-spec-tests is both a collection of test cases and a framework in Python to generate tests for Ethereum execution clients implemented.
+
+The framework collects and executes the test cases in order to generate _test fixtures_ (JSON) which can be consumed by any execution client to verify their implementation of [ethereum/execution-specs](https://github.com/ethereum/execution-specs). The fixtures, which define state transition and block tests, are generated by the framework using one of the `t8n` command-line tools that are provided by most execution clients, see below for an overview of the supported `t8n` tools.
+
+```mermaid
+---
+title: Test Fixture Generation with execution-spec-tests
+---
+flowchart LR
+  style C stroke:#333,stroke-width:2px
+  style D stroke:#333,stroke-width:2px
+  style G stroke:#F9A825,stroke-width:2px
+  style H stroke:#F9A825,stroke-width:2px
+  
+  subgraph ethereum/go-ethereum
+    C[<code>evm t8n</code>\nexternal executable]
+  end
+
+  subgraph ethereum/solidity
+    D[<code>solc</code>\nexternal executable]
+  end
+
+  subgraph ethereum/EIPs
+    E(<code>EIPS/EIP-*.md</code>\nSHA digest via Github API)
+  end
+
+  subgraph "ethereum/execution-spec-tests"
+    A(<code>./tests/**/*.py</code>\nPython Test Cases)
+    B([<code>$ fill ./tests/</code>\nPython Framework])
+  end
+
+  subgraph Test Fixture Consumers
+    subgraph ethereum/hive
+      G([<code>$ hive ...</code>\nGo Test Framework])
+    end
+    H([Client executables])
+  end
+
+  C <-.-> B  
+  D <-.-> B
+  A --> B
+  E <-.-> |retrieve latest spec version\ncheck tested spec version| B
+  B -->|output| F(<code>./fixtures/**/*.json</code>\nJSON Test Fixtures)
+  F -->|input| G
+  F -->|input| H
+```
+
+The generated test fixtures can be used:
+
+1. Directly by client teams' test frameworks, and,
+2. In the integration tests executed in the @ethereum/hive framework.
+
+More information on how to use and download the [released test fixtures](https://github.com/ethereum/execution-spec-tests/releases) can be found [here](../consuming_tests/index.md).
+
+## Relationship to ethereum/tests
+
+This collection of tests is relatively new (test case development started Q4, 2022) and mainly targets recent and upcoming Ethereum specification changes. It does not replace, but rather complements the existing tests in [ethereum/tests](https://github.com/ethereum/tests).
+
+## Motivation
+
+The motivation to implement test cases in [ethereum/execution-spec-tests](https://github.com/ethereum/execution-spec-tests) is:
+
+1. To implement test cases as code and ensure that changes, due to spec changes, for example, can be easily made. Moreover, changes are easily understandable and available in version control.
+2. To avoid the 2-step approach often used in [ethereum/tests](https://github.com/ethereum/tests):
+    1. Code (often unavailable) -> Test case (YAML).
+    2. Test case (YAML) -> Fixtures (JSON).
+
+!!! success "Contributing"
+    Contributions via [PR](https://github.com/ethereum/execution-spec-tests/pulls) are welcome!
+
+!!! bug "Reporting a Vulnerability"
+
+    Care is required when adding PRs or issues for functionality that is live on Ethereum mainnet. Please report vulnerabilities and verify bounty eligibility via the [bug bounty program](https://bounty.ethereum.org).
+
+    - **Please do not create a PR with a vulnerability visible.**
+    - **Please do not file a public ticket mentioning the vulnerability.**

docs/getting_started/setup_vs_code.md

@@ -52,4 +52,4 @@ Trust the `execution-specs-test` repository when opening in VS Code to be prompt
 
 ## Configuration for Testing EVM Features Under Active Development
 
-An additional step is required to enable fixture generations for features from forks that are under active development and have not been deployed to mainnet, see [Executing Tests for Features under Development](./executing_tests_dev_fork.md#vs-code-setup).
+An additional step is required to enable fixture generations for features from forks that are under active development and have not been deployed to mainnet, see [Filling Tests for Features under Development](../filling_tests/filling_tests_dev_fork.md#vs-code-setup).