Skip to content

Development How To Add Frameworks or Tests

Jump to bottom
ajohnston edited this page Jan 13, 2021 · 7 revisions

Adding a New Test

To create a new framework you can start the new test initialization wizard with the command:

tfb --new

This will walk you through the scaffolding process. You will still need to follow the steps in Additional Information below.

Adding a New Test Without the Installation Wizard

When adding a new framework or new test to an existing framework, please follow these steps:

  • Add/Update a test docker file.
  • Add/Update a benchmark_config.json.
  • Implement at least one test (we'd be even happier with six completed tests). When creating a database test, use the table/collection hello_world. Our database setup scripts are stored inside the config/ folder if you need to see the database schema.
  • Add/Update a README for your framework.
  • Add/Update a README for your language.

Additional Information

  • Make sure that any packages/dependencies you are using or any git source repositories you are pulling in are locked down to specific version numbers or releases.
  • Test your framework appropriately.
    • Ensure the framework tests implemented pass in your local environment.
    • GitHub Actions will test the framework when a pull request is opened. Tip: Use your own GitHub Actions queue and know the outcome of the tests before you submit a merge request. See GitHub Actions Tips and Tricks for more details.
  • When all tests are passing, submit a pull request following the pull request procedure.

Tuning By Environment

Our official round results come from two different environments. If you'd like to tune your tests based on those environments, you can make use of the BENCHMARK_ENV environment variable. To do that, add the following line to your test's dockerfile:

ARG BENCHMARK_ENV

Then you'll have access to the $BENCHMARK_ENV variable throughout your dockerfile and scripts. See the environments section for the values of that variable on each environment.

To change this value during local testing, use the --results-environment flag.

Example:

tfb --test gemini --results-environment Local
Clone this wiki locally