Writing tests

Testing philosophy

• Whenever you encounter a bug, add a (failing) test for it. Then fix the bug.
• Whenever you modify or add a feature, write a test for it! Writing tests before writing the actual implementation helps keeping your API clean.
• Make unit tests as atomic as possible. A unit test should run in the blink of an eye.
• Document why you wrote your test - developers will thank you for it once it fails.

Types of tests

1. Local tests (tests/ folder): An extensive suite of (hundreds of) tests of the AiiDA python API and command line interface that can be run locally on your development machine. Also run on GitHub actions at every commit on pull requests to aiida-core.
2. Benchmark tests (tests/benchmark/ folder): A handful of performance benchmarks of common operations (see results). Run on GitHub actions at every commit to the develop branch of aiida-core.
3. System tests (.github/system_tests folder): A handful of tests that require infrastructure that is difficult to set up and manage locally as part of a test run (e.g. an AiiDA daemon, a remote computer) or would otherwise interfere with the standard test infrastructure. Run on GitHub actions at every commit on pull requests to aiida-core.
4. Stress tests (.molecule folder): Medium- to long-running tests that stress the AiiDA engine. They are not part of the automated CI pipeline.

Setting up your local test environment

1. Install extra dependencies: pip install -e .[tests]
2. (optional) Set up a test profile for faster testing :
   • with quicksetup: verdi quicksetup --profile test_profile --test-profile
   • with setup: verdi setup --profile test_profile_1 --test-profile
   • Note: The database name, and the repository folder need to start with test_ . This is to avoid accidental data loss (tests modify the database).

You can also use tox, which automates the creation of local virtual environments within which to run the tests. The configuration is provided in pyproject.toml.

Running the local test suite

• Run all tests: pytest or tox
  • By default this will use pgtest to set up a temporary postgres cluster that is used to run the tests.
  • If you have a test profile set up (see above), you can run the tests directly there by running:

    AIIDA_TEST_PROFILE=test_profile_1 pytest

  • Alternatively, you can export the variable export AIIDA_TEST_PROFILE=test_profile_1. Then you don't have to specify it each time you run the tests.
• Run subset of tests: pytest tests/path/to/test

Transport tests

For transport tests, you need to be able to ssh into your local machine (localhost) without password, i.e.

• Make sure to have a ssh server running.
• Configure a ssh key for your user on your machine, and add your public key to the authorized keys of localhost.

Linux (Ubuntu)

• sudo apt install openssh-server
• Create an ssh key (if you don't have one already)
• Add it to ~/.ssh/authorized_keys, e.g. using ssh-copy-id localhost
• For security reasons, you might want to disallow ssh connections from outside your local machine: Edit /etc/ssh/sshd_config, changing #ListenAddress 0.0.0.0 to ListenAddress 127.0.0.1 (note the missing #).
• ssh localhost should now work

RabbitMQ Tests

For tests that require the RabbitMQ message broker, you can set up RabbitMQ locally or there is also .docker/docker-rabbitmq.yml which can spin up a docker container running it and exposing the requisite ports (see instructions for use within the file).

Writing new tests

AiiDA uses pytest as the testing framework. It provides a number of useful fixtures that take care of setting up a database, AiiDA profile etc.

• Create a test_MODULENAME.py in the appropriate subfolder of the tests directory or directly add to an existing python file.
• Write a python function starting with test_
• Reuse existing fixtures where possible or add your own

Note: Some tests still use the AiidaTestCase unittest class. Make an effort to migrate such cases to pytest when you are touching them.