Improvements to Testing

[srivarra]

Motivation

As the codebase is gets larger, testing becomes more complex. Lets use this issue to gather everyone's grievances, problems they've dealt with, and potential solutions. Feel free to add anything you guys come across with!

After we feel that everything has been discussed and potential solutions have been addressed, we should make a design document and a repository project.

[srivarra]

Jupyter Notebook Tests

• The Jupyter Notebook tests are very complex, and are a bit inflexible.
• Currently in notebooks_test.py we have separate tests for each notebook containing calls to utility functions which run cells in the notebook. Each function runs several cells in sequence. The granularity could be increased here.
• We can use a class based system, where we have 1 class per notebook. And 1 test per tagged notebook cell.
  • This would allow us to add a _setup() function for each class which pytest recognizes. Think of it like an __init__() method for regular Python classes.
    • There we can create dummy data, set paths up, etc...
• We can run then run tests in the same order that a user would run the notebook by using pytest-order.

[srivarra]

Organization

We should remove the tests out of the ark folder and place them to a separate folder called tests. So that when we pip install it will not install the tests themselves.

[alex-l-kong]

The main thing to keep in mind about the tests is that each one should be simple. Several of the test functions could use improvement on this front:

• They contain duplicated code for testing different parameters
• They contain complex (and oftentimes duplicate) data generation processes
• They run the same test repeatedly (particularly prevalent for validation)
• They test multiple things (ex. file path checks + column name checks + output checks)

Some way we can address this:

• Consistent parametrization: if you're just testing different parameter values that don't affect the resulting output, use @parametrize.
• Fixtures: if your tests require massive DataFrames or arrays, consider using a fixture. This will avoid having to generate this within the test itself, and will also allow you to share the fixture across multiple tests
• Test classes: these are helpful for organizing tests that may have a large amount of shared parameters and a variety of different checks/outputs. Using these can also provide a better way to break up test functions so they only test one thing each.