Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move test suite description to RTD #101

Merged
merged 3 commits into from
Oct 15, 2024
Merged

Move test suite description to RTD #101

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a1680a3
Move test suite description to RTD
dweindl Oct 15, 2024
f010c5a
Merge branch 'develop' into test_rtd
dweindl Oct 15, 2024
75ab989
Merge branch 'develop' into test_rtd
dweindl Oct 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 0 additions & 35 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,38 +46,3 @@ given the forward method and a predecessor model with 2 estimated parameters,
if there are no models with 3 estimated parameters, but some models with 4
estimated parameters, then the search may return candidate models with 4
estimated parameters.

## Test cases

Several test cases are provided, to test the compatibility of a
PEtab-compatible calibration tool with different PEtab Select features.

The test cases are available in the `test_cases` directory, and are provided in
the model format.

| Test ID | Criterion | Method | Model space files | Compressed format | Constraints files | Predecessor (initial) models files |
|----------------------------------------------|-----------|------------------|-------------------|-------------------|-------------------|------------------------------------|
| 0001 | (all) | (only one model) | 1 | | | |
| 0002<sup>[1](#test_case_0002)</sup> | AIC | forward | 1 | | | |
| 0003 | BIC | all | 1 | Yes | | |
| 0004 | AICc | backward | 1 | | 1 | |
| 0005 | AIC | forward | 1 | | | 1 |
| 0006 | AIC | forward | 1 | | | |
| 0007<sup>[2](#test_case_0007_and_0008)</sup> | AIC | forward | 1 | | | |
| 0008<sup>[2](#test_case_0007_and_0008)</sup> | AICc | backward | 1 | | | |
| 0009<sup>[3](#test_case_0009)</sup> | AICc | FAMoS | 1 | Yes | | Yes |

<a name="test_case_0002">1</a>. Model `M1_0` differs from `M1_1` in three
parameters, but only 1 additional estimated parameter. The effect of this on
model selection criteria needs to be clarified. Test case 0006 is a duplicate
of 0002 that doesn't have this issue.

<a name="test_case_0007_and_0008">2</a>. Noise parameter is removed, noise is
fixed to `1`.

<a name="test_case_0009">3</a>. This is a computationally expensive problem to
solve. Developers can try a model selection initialized with the provided
predecessor model, which is a model start that reproducibly finds the expected
model. To solve the problem reproducibly <i>ab initio</i>, on the order of 100
random model starts are required. This test case reproduces the model selection
problem presented in https://doi.org/10.1016/j.cels.2016.01.002 .
1 change: 1 addition & 0 deletions doc/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ Welcome to petab-select's documentation!
:caption: Contents:

examples
Test Suite <test_suite>
api
problem_definition

Expand Down
88 changes: 88 additions & 0 deletions doc/test_suite.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
Model selection test suite
==========================

Several test cases are provided, to test the compatibility of a
PEtab-compatible calibration tool with different PEtab Select features.

The test cases are available in the ``test_cases`` directory, and are provided in
the model format.

.. list-table::
:header-rows: 1

* - Test ID
- Criterion
- Method
- Model space files
- Compressed format
- Constraints files
- Predecessor (initial) models files
* - 0001
- (all)
- (only one model)
- 1
-
-
-
* - 0002 [#f1]_
- AIC
- forward
- 1
-
-
-
* - 0003
- BIC
- all
- 1
- Yes
-
-
* - 0004
- AICc
- backward
- 1
-
- 1
-
* - 0005
- AIC
- forward
- 1
-
-
- 1
* - 0006
- AIC
- forward
- 1
-
-
-
* - 0007 [#f2]_
- AIC
- forward
- 1
-
-
-
* - 0008 [#f2]_
- AICc
- backward
- 1
-
-
-
* - 0009 [#f3]_
- AICc
- FAMoS
- 1
- Yes
-
- Yes

.. [#f1] Model ``M1_0`` differs from ``M1_1`` in three parameters, but only 1 additional estimated parameter. The effect of this on model selection criteria needs to be clarified. Test case 0006 is a duplicate of 0002 that doesn't have this issue.

.. [#f2] Noise parameter is removed, noise is fixed to ``1``.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The (1,2) that appears in the rendered docs is useless/confusing but fine to leave in

Copy link
Member Author

@dweindl dweindl Oct 15, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If your browser window is small enough, they will do something meaningful 😅

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then maybe I've finally met the person that benefits from max-linewidth=80...


.. [#f3] This is a computationally expensive problem to solve. Developers can try a model selection initialized with the provided predecessor model, which is a model start that reproducibly finds the expected model. To solve the problem reproducibly *ab initio*, on the order of 100 random model starts are required. This test case reproduces the model selection problem presented in https://doi.org/10.1016/j.cels.2016.01.002.
Loading