Skip to content

Writing tests

Jump to bottom
Leopold Talirz edited this page Dec 9, 2018 · 15 revisions

Setting up your test environment

  1. Install extra dependencies: pip install -e .[testing]
  2. Set up a test profile (starting with test_):
    • with quicksetup: verdi quicksetup test_profile_1
    • with setup: verdi -p test_profile_1 setup
    • Note: The profile name, the database name, and the repository folder need to start with test_ . This is to avoid accidental data loss (tests modify the database).

Running existing tests

  • Run all tests: verdi -p test_profile devel tests
  • Run subset of tests: verdi -p test_profile devel tests aiida.transport db.generic

Use TAB completion to get the full list of tests.

Transport tests

For transport tests, you need to be able to ssh into your local machine (localhost) without password, i.e.

  • Make sure to have a ssh server running.
  • Configure a ssh key for your user on your machine, and add your public key to the authorized keys of localhost.

Linux (Ubuntu)

  • sudo apt install openssh-server
  • Create an ssh key (if you don't have one already)
  • Add it to ~/.ssh/authorized_keys, e.g. using ssh-copy-id localhost
  • For security reasons, you might want to disallow ssh connections from outside your local machine: Edit /etc/ssh/sshd_config, changing #ListenAddress 0.0.0.0 to ListenAddress 127.0.0.1 (note the missing #).
  • ssh localhost should now work

Testing philosophy

  • Whenever you encounter a bug, add a (failing) test for it. Then fix the bug.
  • Whenever you modify or add a feature, write a test for it! Writing tests before writing the actual implementation helps keeping your API clean.
  • Make unit tests as atomic as possible. A unit test should run in the blink of an eye.
  • Document why you wrote your test - developers will thank you for it once it fails.

Writing a new test

There are three types of tests in AiiDA:

  1. Tests that do not require the database (for example, testing the creation of paths in k-space, the functionality of a transport plugin, ...)
  2. Tests that require the database, but do not require submission of jobs (for example, verifying that node attributes can be correctly queried, that the transitive closure table is correctly generated, ...)
  3. Tests that require the submission of jobs

We now describe how to write each of these type of tests.

1. Tests that do not need the database

  • Create a test_MODULENAME.py inside the package you want to test.
  • For each group of tests create a new subclass of unittest.TestCase and then create the tests as methods using the unittest module.
  • Verify that the parent module of the test is listed in the base_allowed_test_folders property of the Devel class, inside aiida.cmdline.commands.devel (for autodiscovery).

For an example, see aiida.common.test_utils.

2. Tests that need the database

These tests use the testing functionality of Django, adapted to run smoothly with AiiDA.

  • Create a new python file under aiida.backends.djsite.db.subtests

  • For each group of tests create a new subclass of aiida.backends.djsite.db.testbase.AiidaTestCase.

  • add a new entry to the db_test_list inside aiida.backends.djsite.db.testbase. E.g.:

    db_test_list = {
  ...
  'newtests': 'aiida.backends.djsite.db.subtests.mynewtestsmodule',
  ...
}

    makes the tests available under:

    verdi devel tests db.newtests

Note:

  • The AiidaTestCase takes care of
    • creating a temporary database
    • populating the database with default data for each test class (in particular, a computer and a user; see the AiidaTestCase.setUpClass() method).
    • cleaning the database when all tests of the class have finished
  • Run these tests only using verdi devel tests - or you risk corrupting your database.

3. Tests that require the submission of jobs

These tests require an external engine to submit the calculations and then check the results at job completion. We use for this a continuous integration server, and the best approach is to write suitable workflows to run simulations and then verify the results at the end.

Reusing test code

Some test modules contain dedicated routines to simplify the creation of new tests.

For example, any method defined in aiida.transport.plugins.test_all_plugins that is decorated by @run_for_all_plugins will automatically run for all available transport plugins (testing e.g., to copy files, remove folders, etc.) - in addition to the plugin specific tests inside aiida.transport.plugins.test_ssh and aiida.transport.plugins.test_local.

Clone this wiki locally