Writing tests

Testing philosophy

• Whenever you encounter a bug, add a (failing) test for it. Then fix the bug.
• Whenever you modify or add a feature, write a test for it! Writing tests before writing the actual implementation helps keeping your API clean.
• Make unit tests as atomic as possible. A unit test should run in the blink of an eye.
• Document why you wrote your test - developers will thank you for it once it fails.

Types of tests

1. Local tests (tests/ folder): An extensive suite of (hundreds of) tests of the AiiDA python API and command line interface that can be run locally on your development machine. Also run on GitHub actions at every commit on pull requests to aiida-core.
2. Benchmark tests (tests/benchmark/ folder): A handful of performance benchmarks of common operations ([results](https://aiidateam.github.io/aiida-core/dev/bench/ubuntu-18.04/django/)). Run on GitHub actions at every commit to the develop branch of aiida-core.
3. System tests (.github/system_tests folder): A handful of tests that require infrastructure that is difficult to set up manage locally as part of a test run (e.g. an AiiDA daemon, a remove computer) or would otherwise interfere with the standard test infrastructure. Run on GitHub actions at every commit on pull requests to aiida-core.
4. Stress tests (.molecule folder): Medium- to long-running tests that stress the AiiDA engine. Run on our local Jenkins runner at every commit to pull requests to aiida-core.

Setting up your local test environment

1. Install extra dependencies: pip install -e .[testing]
2. (optional) Set up a test profile (starting with test_) for faster testing:
   • with quicksetup: verdi quicksetup test_profile_1
   • with setup: verdi setup --profile test_profile_1
   • Note: The profile name, the database name, and the repository folder need to start with test_ . This is to avoid accidental data loss (tests modify the database).

Running the local test suite

• Run all tests: pytest
  • By default this will use pgtest to set up a temporary postgres cluster that is used to run the tests.
  • If you have a test profile set up (see above), you can run the tests directly there by setting:

    export AIIDA_TEST_PROFILE=test_profile_1
    export AIIDA_TEST_BACKEND=django

• Run subset of tests: pytest tests/path/to/test

Transport tests

For transport tests, you need to be able to ssh into your local machine (localhost) without password, i.e.

• Make sure to have a ssh server running.
• Configure a ssh key for your user on your machine, and add your public key to the authorized keys of localhost.

Linux (Ubuntu)

• sudo apt install openssh-server
• Create an ssh key (if you don't have one already)
• Add it to ~/.ssh/authorized_keys, e.g. using ssh-copy-id localhost
• For security reasons, you might want to disallow ssh connections from outside your local machine: Edit /etc/ssh/sshd_config, changing #ListenAddress 0.0.0.0 to ListenAddress 127.0.0.1 (note the missing #).
• ssh localhost should now work

Writing new tests

AiiDA uses pytest as the testing framework. It provides a number of useful fixtures that take care of setting up a database, AiiDA profile etc.

• Create a test_MODULENAME.py in the appropriate subfolder of the tests directory or directly add to an existing python file.
• Write a python function starting with test_
• Reuse existing fixtures where possible or add your own

Note: Some tests still use the AiidaTestCase unittest class. Make an effort to migrate such cases to pytest when you are touching them.