Requirements for a better test suite

[rafmudaf]

The current test suite was adapted from the legacy FAST 8 Cert Tests. These test cases provided a readily available path for establishing automated tests in a modern framework. Now that the testing framework has been established, its worth reevaluating the existing test suite and possibly redesigning with specific requirements.

Some initial thoughts on requirements are:

• Execution on the order of 10 minutes
• Solutions should be testable with single and double precision
  • Single precision should pass with a larger tolerance
• Well scoped
  • High-level tests to fully execute the OpenFAST system
  • Mid-level tests to execute a given algorithm (BEM or Morison)

[ebranlard]

I agree with these requirements. Here are some of my thoughts/experience running the tests.

I would even aim for 1-10s (of CPU time) per case, max 1min. The benefits are multiple: less CPU time, smaller file size, less accumulation of errors (which can be inevitable when numerics/precision are changed).

I believe it's a good idea and possible to get single and double to match (I've lately seen a potential single/double issue with BEM, but hopefully we'll fix it).

It'd be great to aim for debug and release to match as well. Since I typically compile for single/debug when I develop). The tolerance can be increased for debug.

A careful selection of channels will help (some small loads and displacements might be just noise and we should remove them). Finding and removing the noisy/sensitive channels and focusing on the physical ones is a bit tedious, subjective, and could hide some bugs, but I think it's worth it. Maybe the testing framework can be tolerant if some channels are removed.

A focus on more driver tests is a good idea (typically faster tests).

A framework for low (subroutine level) and mid-level tests would be great. I would prefer having the source code of these tests close to the code (in the src folder, or a test folder nearby), and with an easy compilation/running of these tests.

I'd be great to have the r-tests "less far" from the openfast root and from the cmake build. I think there are a bit too much sub-directory to navigate at the moment, and it's a tiny bit frustrating to switch between the build/reg-tests/... and reg-tests/r-tests/... (when debugging testcases, or setting up testcases). I'm not sure what the solution could be. (NOTE: in the ad-driver branch, I've updated the python code so that the content of the build/reg-tests folder is always rewritten. Currently, it's only done at the first pass, which is not desirable when developing tests).

Keeping the r-test folder small would be great (Right now, merging/updating of r-test is tricky if you don't have a good chunk of the history). I don't know what the solution could be (duplicating/rebasing the repo every year)..

EDIT:
Automatic rebaselin-ing would be great.

I'd be happy to focus on storing baselines for linux gfortran. Maybe have separate baselines for other platform only when release are done?

Platform independent tests... Maybe adding some kind of list of exceptions for each test for cases when strong difference exists between platforms/release/precision.

Being able to use the test suite within our development workflow would help a lot (this goes back to my point about the directory being far and split into two). Currently it is difficult to create a small test on the go, while developing. For instance, someone sends you a model that used to work but doesn't anymore. You'd want to put the folder next to the tests, generate a reference, and then easily run that case while changing the code, and once in a while rerun all the other tests. In the process you'll likely simplify the input files several times. It's a typical workflow for me at least (which right now I achieve using a Makefile). If there is a way to integrate that workflow in the test suite without having to edit too many files in too many different folders, it would be awesome! I think for most practical cases, it would work to have the ctest being executed in the folder of the r-test directly. Some kind of optional suffix could be added to the output files if ones needs to compare e.g. debug/release. This would simplify the testing process for me and help generating new tests without having to switch between two folders.