-
Notifications
You must be signed in to change notification settings - Fork 108
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
test: simulation import and export (#3033)
* debug sim test * start modifuing for v50 * add sim test v1 * add sim export * move simulation tests to a separate directory * remove unused functions * add comments * enable gov module * enable gov module * fix lint error * add todos for BasicManager * add todos for BasicManager * add cleanup for TestFullAppSimulation * register legacy router * added a doc for simulation testing * undo db close for multi threaded test * add description for tests * remove go module from simulation tests * make basicsmanager private * Update Makefile Co-authored-by: Francisco de Borja Aranda Castillejo <[email protected]> * Update changelog.md Co-authored-by: Francisco de Borja Aranda Castillejo <[email protected]> * format simulation.md * add test for import export * test sim after import * add bench test * remove bench * add changelog * remove cosmos utils * remove print lines * add comments * ci: Add simulation tests workflow * merge ci workflow * Update .github/workflows/simulation-test.yaml Co-authored-by: semgrep-code-zeta-chain[bot] <181804379+semgrep-code-zeta-chain[bot]@users.noreply.github.com> * Update simulation-test.yaml Update simulation-test workflow * add branch sim-import-export to trigger ci run * format files * uncomment simulation tests from CI * fix defer functions * Rename sim tests file, add result aggregation and merge jobs * ignore simulation tests from codecov * Add runs on to matrix conditional job * add default make-targets * bump go * Update .github/workflows/sim.yaml workflow name Co-authored-by: Alex Gartner <[email protected]> * Delete main branch from sim test workflow * improve formatting for simulation.md * move simulation tests to root directory * remove panic for Import Export Test * Trigger Sim Tests on Labeled PRs or when any file from x/**/* is changed * update sim.yaml * Update sim.yaml jobs needs * update sim.yaml * update codecov.yml * fixed some comments * remove directory sim * remove directory sim * format directory * remove utils file * add to codecov --------- Co-authored-by: Francisco de Borja Aranda Castillejo <[email protected]> Co-authored-by: Julian Rubino <[email protected]> Co-authored-by: semgrep-code-zeta-chain[bot] <181804379+semgrep-code-zeta-chain[bot]@users.noreply.github.com> Co-authored-by: Julian Rubino <[email protected]> Co-authored-by: Alex Gartner <[email protected]>
- Loading branch information
6 people
authored
Nov 1, 2024
1 parent
b58046f
commit 2e5d794
Showing
12 changed files
with
775 additions
and
231 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,111 @@ | ||
| name: sim | ||
|
|
||
| on: | ||
| push: | ||
| branches: | ||
| - develop | ||
| pull_request: | ||
| types: [opened, synchronize, labeled] | ||
| schedule: | ||
| - cron: "0 6 * * *" | ||
| workflow_dispatch: | ||
| inputs: | ||
| make-targets: | ||
| description: 'Comma separated list of make targets to run (e.g., test-sim-nondeterminism, test-sim-fullappsimulation)' | ||
| required: true | ||
| default: 'test-sim-nondeterminism' | ||
|
|
||
| concurrency: | ||
| group: simulation-${{ github.head_ref || github.sha }} | ||
| cancel-in-progress: true | ||
|
|
||
| jobs: | ||
| changed-files: | ||
| runs-on: ubuntu-latest | ||
| outputs: | ||
| modified_files: ${{ steps.changes.outputs.modified_files }} | ||
| steps: | ||
| - name: Checkout repository | ||
| uses: actions/checkout@v3 | ||
|
|
||
| - name: Get changed files in x directory | ||
| id: changes | ||
| run: | | ||
| echo "::set-output name=modified_files::$(git diff --name-only --diff-filter=ACMRT ${{ github.event.pull_request.base.sha }} ${{ github.sha }} | grep '^x/' | xargs)" | ||
| matrix-conditionals: | ||
| needs: changed-files | ||
| if: | | ||
| contains(github.event.pull_request.labels.*.name, 'SIM_TESTS') || needs.changed-files.outputs.modified_files | ||
| runs-on: ubuntu-22.04 | ||
| outputs: | ||
| SIM_TEST_NOND: ${{ steps.matrix-conditionals.outputs.SIM_TEST_NOND }} | ||
| SIM_TEST_FULL: ${{ steps.matrix-conditionals.outputs.SIM_TEST_FULL }} | ||
| SIM_TEST_IMPORT_EXPORT: ${{ steps.matrix-conditionals.outputs.SIM_TEST_IMPORT_EXPORT }} | ||
| SIM_TEST_AFTER_IMPORT: ${{ steps.matrix-conditionals.outputs.SIM_TEST_AFTER_IMPORT }} | ||
| steps: | ||
| - id: matrix-conditionals | ||
| uses: actions/github-script@v7 | ||
| with: | ||
| script: | | ||
| const makeTargetsInput = context.payload.inputs ? context.payload.inputs['make-targets'] : null; | ||
| const defaultTargets = ['test-sim-nondeterminism', 'test-sim-fullappsimulation', 'test-sim-import-export', 'test-sim-after-import']; | ||
| const makeTargets = makeTargetsInput ? makeTargetsInput.split(',') : defaultTargets; | ||
| core.setOutput('SIM_TEST_NOND', makeTargets.includes('test-sim-nondeterminism')); | ||
| core.setOutput('SIM_TEST_FULL', makeTargets.includes('test-sim-fullappsimulation')); | ||
| core.setOutput('SIM_TEST_IMPORT_EXPORT', makeTargets.includes('test-sim-import-export')); | ||
| core.setOutput('SIM_TEST_AFTER_IMPORT', makeTargets.includes('test-sim-after-import')); | ||
| simulation-tests: | ||
| needs: | ||
| - matrix-conditionals | ||
| if: | | ||
| contains(github.event.pull_request.labels.*.name, 'SIM_TESTS') || needs.changed-files.outputs.modified_files | ||
| runs-on: ubuntu-22.04 | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| include: | ||
| - make-target: "test-sim-nondeterminism" | ||
| condition: ${{ needs.matrix-conditionals.outputs.SIM_TEST_NOND == 'true' }} | ||
| - make-target: "test-sim-fullappsimulation" | ||
| condition: ${{ needs.matrix-conditionals.outputs.SIM_TEST_FULL == 'true' }} | ||
| - make-target: "test-sim-import-export" | ||
| condition: ${{ needs.matrix-conditionals.outputs.SIM_TEST_IMPORT_EXPORT == 'true' }} | ||
| - make-target: "test-sim-after-import" | ||
| condition: ${{ needs.matrix-conditionals.outputs.SIM_TEST_AFTER_IMPORT == 'true' }} | ||
| name: ${{ matrix.make-target }} | ||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@v3 | ||
|
|
||
| - name: Set up Go | ||
| uses: actions/setup-go@v4 | ||
| with: | ||
| go-version: '1.22' | ||
|
|
||
| - name: Install dependencies | ||
| run: make runsim | ||
|
|
||
| - name: Run ${{ matrix.make-target }} | ||
| if: ${{ matrix.condition }} | ||
| run: | | ||
| make ${{ matrix.make-target }} | ||
| sim-ok: | ||
| needs: | ||
| - simulation-tests | ||
| if: | | ||
| contains(github.event.pull_request.labels.*.name, 'SIM_TESTS') || needs.changed-files.outputs.modified_files | ||
| runs-on: ubuntu-22.04 | ||
| steps: | ||
| - name: Aggregate Results | ||
| run: | | ||
| result="${{ needs.simulation-tests.result }}" | ||
| if [[ $result == "success" || $result == "skipped" ]]; then | ||
| exit 0 | ||
| else | ||
| exit 1 | ||
| fi |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -81,3 +81,4 @@ ignore: | |
| - "precompiles/**/*.json" | ||
| - "precompiles/**/*.sol" | ||
| - "precompiles/**/*.gen.go" | ||
| - "simulation/*.go" | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,34 +1,78 @@ | ||
| # Zetachain simulation testing | ||
| ## Overview | ||
| The blockchain simulation tests how the blockchain application would behave under real life circumstances by generating | ||
| and sending randomized messages.The goal of this is to detect and debug failures that could halt a live chain,by providing | ||
| logs and statistics about the operations run by the simulator as well as exporting the latest application state. | ||
|
|
||
| and sending randomized messages.The goal of this is to detect and debug failures that could halt a live chain by | ||
| providing logs and statistics about the operations run by the simulator as well as exporting the latest application | ||
| state. | ||
|
|
||
| ## Simulation tests | ||
|
|
||
| ### Nondeterminism test | ||
| Nondeterminism test runs a full application simulation , and produces multiple blocks as per the config | ||
| It checks the determinism of the application by comparing the apphash at the end of each run to other runs | ||
| The test certifies that , for the same set of operations ( irrespective of what the operations are ), we would reach the same final state if the initial state is the same | ||
| The test certifies that, for the same set of operations (regardless of what the operations are), we | ||
| would reach the same final state if the initial state is the same | ||
| Approximate run time is 2 minutes. | ||
| ```bash | ||
| make test-sim-nondeterminism | ||
| ``` | ||
|
|
||
| ### Full application simulation test | ||
| Full application runs a full app simulation test with the provided configuration. | ||
| At the end of the run it tries to export the genesis state to make sure the export works. | ||
| At the end of the run, it tries to export the genesis state to make sure the export works. | ||
| Approximate run time is 2 minutes. | ||
| ```bash | ||
| make test-sim-full-app | ||
| ``` | ||
|
|
||
| ### Import Export simulation test | ||
| The import export simulation test runs a full application simulation | ||
| and exports the application state at the end of the run. | ||
| This state is then imported into a new simulation. | ||
| At the end of the run, we compare the keys for the application state for both the simulations | ||
| to make sure they are the same. | ||
| Approximate run time is 2 minutes. | ||
| ```bash | ||
| make test-sim-import-export | ||
| ``` | ||
|
|
||
| ### Import and run simulation test | ||
| This simulation test exports the application state at the end of the run and imports it into a new simulation. | ||
| Approximate run time is 2 minutes. | ||
| ```bash | ||
| make test-sim-after-import | ||
| ``` | ||
|
|
||
| ### Multi seed long test | ||
| Multi seed long test runs a full application simulation with multiple seeds and multiple blocks.This runs the test for a longer duration compared to the multi seed short test | ||
| Multi seed long test runs a full application simulation with multiple seeds and multiple blocks. | ||
| It uses the `runsim` tool to run the same test in parallel threads. | ||
| Approximate run time is 30 minutes. | ||
| ```bash | ||
| make test-sim-multi-seed-long | ||
| ``` | ||
|
|
||
| ### Multi seed short test | ||
| Multi seed short test runs a full application simulation with multiple seeds and multiple blocks. This runs the test for a longer duration compared to the multi seed long test | ||
| Multi seed short test runs a full application simulation with multiple seeds and multiple blocks. | ||
| It uses the `runsim` tool to run the same test in parallel threads. | ||
| This test is a shorter version of the Multi seed long test. | ||
| Approximate run time is 10 minutes. | ||
| ```bash | ||
| make test-sim-multi-seed-short | ||
| ``` | ||
| ``` | ||
|
|
||
| ### Import Export long test | ||
| This test runs the import export simulation test for a longer duration. | ||
| It uses the `runsim` tool to run the same test in parallel threads. | ||
| Approximate run time is 30 minutes. | ||
| ```bash | ||
| make test-sim-import-export-long | ||
| ``` | ||
|
|
||
| ### Import and run simulation test long | ||
| This test runs the import and run simulation test for a longer duration. | ||
| It uses the `runsim` tool to run the same test in parallel threads. | ||
| Approximate run time is 30 minutes. | ||
| ```bash | ||
| make test-sim-after-import-long | ||
| ``` | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,4 @@ | ||
| package sim | ||
| package simulation | ||
|
|
||
| import ( | ||
| "flag" | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.