diff --git a/doc/examples/workflow_cli.ipynb b/doc/examples/workflow_cli.ipynb index 21f42da5..46c5a516 100644 --- a/doc/examples/workflow_cli.ipynb +++ b/doc/examples/workflow_cli.ipynb @@ -45,6 +45,7 @@ "Iterations of model selection start and end with `start_iteration` and `end_iteration`.\n", "\n", "In each call to `petab_select start_iteration`, the following options are required:\n", + "\n", "- `--problem`: The PEtab Select problem YAML file, which normally defines the method too\n", "- `--state`: A file that is used to store the state of the problem (e.g., such that models are not revisited)\n", "- `--output`: A file to store the models proposed by PEtab Select\n", @@ -156,9 +157,10 @@ "id": "bee59532-f6bd-4d9e-8a94-a9b0302efab1", "metadata": {}, "source": [ - "At this point, the calibration tool should calibrate the models, and save the calibration results to disk in the PEtab Select model YAML format. For this example, we have stored the results in \"model_selection/calibrated_models_1.yaml\".\n", + "At this point, the calibration tool should calibrate the models, and save the calibration results to disk in the PEtab Select model YAML format. For this example, we have stored the results in `model_selection/calibrated_models_1.yaml`.\n", "\n", "Next, we finalize the iteration by calling `petab_select end_iteration`, which requires:\n", + "\n", "- `--state`: see `start_iteration`\n", "- `--output-models`: A file used to store the models from this iteration of calibration. Note that, if the user has supplied calibration results from previous model selection jobs, then this `end_iteration` output might differ from the output of the calibration tool. This `end_iteration` output should be considered the real set of models calibrated in this iteration.\n", "- `--output-metadata`: A file where metadata from the iteration is stored. This includes the signal of whether model selection has terminated." diff --git a/doc/examples/workflow_python.ipynb b/doc/examples/workflow_python.ipynb index eaf0acc2..2a203987 100644 --- a/doc/examples/workflow_python.ipynb +++ b/doc/examples/workflow_python.ipynb @@ -98,7 +98,7 @@ " )\n", "\n", "\n", - "print(\"Information about the model selection problem.\")\n", + "print(\"Information about the model selection problem:\")\n", "print(select_problem)" ] }, @@ -120,6 +120,7 @@ "The model space is then used to find neighbors to the initial model. A candidate space is used to calculate distances between models, and whether a candidate model represents a valid move in model space.\n", "\n", "The built-in `ForwardCandidateSpace` uses the following properties to identify candidate models:\n", + "\n", "- previously estimated parameters must remain estimated;\n", "- the number of estimated parameters must increase; and\n", "- this increase must be minimal.\n", @@ -305,7 +306,7 @@ "Model ID: M1_2-000\n", "Criterion.AIC: 140\n", "\n", - "\u001b[1mBEST MODEL OF CURRENT ITERATION\u001b[0m\n", + "\u001B[1mBEST MODEL OF CURRENT ITERATION\u001B[0m\n", "Model subspace ID: M1_3\n", "PEtab YAML location: model_selection/petab_problem.yaml\n", "Custom model parameters: {'k1': 'estimate', 'k2': 0.1, 'k3': 0}\n", @@ -355,7 +356,7 @@ "Model ID: M1_5-000\n", "Criterion.AIC: -70\n", "\n", - "\u001b[1mBEST MODEL OF CURRENT ITERATION\u001b[0m\n", + "\u001B[1mBEST MODEL OF CURRENT ITERATION\u001B[0m\n", "Model subspace ID: M1_6\n", "PEtab YAML location: model_selection/petab_problem.yaml\n", "Custom model parameters: {'k1': 'estimate', 'k2': 'estimate', 'k3': 0}\n", @@ -398,7 +399,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "\u001b[1mBEST MODEL OF CURRENT ITERATION\u001b[0m\n", + "\u001B[1mBEST MODEL OF CURRENT ITERATION\u001B[0m\n", "Model subspace ID: M1_7\n", "PEtab YAML location: model_selection/petab_problem.yaml\n", "Custom model parameters: {'k1': 'estimate', 'k2': 'estimate', 'k3': 'estimate'}\n", diff --git a/doc/problem_definition.rst b/doc/problem_definition.rst index 4734cb9b..eb05699d 100644 --- a/doc/problem_definition.rst +++ b/doc/problem_definition.rst @@ -1,21 +1,16 @@ Problem definition and file formats =================================== -Model selection problems for PEtab Select are defined by the following -components: +Model selection problems for PEtab Select are defined by the following files: -* A general description of the model selection problem -* Model space files that specify the initial set of candidate models -* Constraints files that further restrict the model space -* Predecessor model files that are used to initialize the model selection - process +#. a general description of the model selection problem, +#. a specification of the model space, and +#. (optionally) a specification of the initial candidate model. The different file formats are described below. -Column or key names that are surrounded by square brackets -(e.g. \[constraint_files\]) are optional. -Selection problem ------------------ +1. Selection problem +-------------------- A YAML file with a description of the model selection problem. @@ -25,81 +20,105 @@ A YAML file with a description of the model selection problem. criterion: # string. method: # string. model_space_files: # list[string]. Filenames. - constraint_files: # list[string] (optional). Filenames. - predecessor_model_files: # list[string] (optional). Filenames. - -- ``format_version``: The version of the model selection extension format (e.g. ``'beta_1'``) -- ``criterion``: The criterion by which models should be compared (e.g. ``'AIC'``) -- ``method``: The method by which model candidates should be generated (e.g. ``'forward'``) + candidate_space_arguments: # list[string] (optional). Filenames. + +- ``format_version``: The version of the model selection extension format + (e.g. ``1``) +- ``criterion``: The criterion by which models should be compared + (e.g. ``AIC``) +- ``method``: The method by which model candidates should be generated + (e.g. ``forward``) - ``model_space_files``: The filenames of model space files. -- ``constraint_files``: The filenames of constraint files. -- ``predecessor_model_files``: The filenames of predecessor (initial) model files. +- ``candidate_space_arguments``: Additional arguments used to generate + candidate models during model selection. For example, an initial candidate + model can be specified with the following code, where + ``predecessor_model.yaml`` is a valid model file. Additional arguments are + provided in the documentation of the ``CandidateSpace`` class. + +.. code-block:: yaml -Model space ------------ + candidate_space_arguments: + predecessor_model: predecessor_model.yaml -A TSV with candidate models, in compressed or uncompressed format. +2. Model space +-------------- + +A TSV file with candidate models, in compressed or uncompressed format. +Each row defines a model subspace, by specifying value(s) that each parameter +can take. The models in a model subspace are all combinations of values across +all parameters. .. list-table:: :header-rows: 1 * - ``model_subspace_id`` - ``petab_yaml`` - - ``sbml`` - ``parameter_id_1`` - ... - ``parameter_id_n`` - * - (Unique) [string] - - [string] + * - (unique) [string] - [string] - - [string/float] OR [; delimited list of string/float] + - (``;``-delimited list) [string/float] + - ... - ... - - [string/float] OR [; delimited list of string/float] - ``model_subspace_id``: An ID for the model subspace. -- ``petab_yaml``: The PEtab YAML filename that serves as the base for a model. -- ``sbml``: An SBML filename. If the PEtab YAML file specifies multiple SBML models, this can select a specific model by model filename. -- ``parameter_id_1``...``parameter_id_n``: Parameter IDs that are specified to take specific values or be estimated. Example valid values are: +- ``petab_yaml``: The PEtab YAML filename that serves as the basis of all + models in this subspace. +- ``parameter_id_1`` ... ``parameter_id_n``: Specify the values that a + parameter can take in the model subspace. For example, this could be: - - uncompressed format: + - a single value - ``0.0`` - ``1.0`` - ``estimate`` - - compressed format: + - one of several possible values, as a ``;``-delimited list - - ``0.0;1.1;estimate`` (the parameter can take the values ``0.0`` or ``1.1``, or be estimated according to the PEtab problem) + - ``0.0;1.1;estimate`` (the parameter can take the values ``0.0`` or + ``1.1``, or be estimated) -Constraints ------------ +Using the ``;``-delimited list format, a model subspace that has two parameters +(``p1, p2``) and six models: -A TSV file with constraints. +- ``p1:=0, p2:=10`` +- ``p1:=0, p2:=20`` +- ``p1:=0, p2:=estimate`` +- ``p1:=estimate, p2:=10`` +- ``p1:=estimate, p2:=20`` +- ``p1:=estimate, p2:=estimate`` + +can be specified like .. list-table:: :header-rows: 1 - * - ``petab_yaml`` - - ``if`` - - ``constraint`` - * - [string] - - [SBML L3 Formula expression] - - [SBML L3 Formula expression] - -- ``petab_yaml``: The filename of the PEtab YAML file that this constraint applies to. -- ``if``: As a single YAML can relate to multiple models in the model space file, this ensures the constraint is only applied to the models that match this ``if`` statement -- ``constraint``: If a model violates this constraint, it is skipped during the model selection process and not optimized. - -Model(s) (Predecessor models / model interchange / report) ----------------------------------------------------------- - -- *Predecessor models* are used to initialize an appropriate model selection method. Model IDs should be unique here and compared to model IDs in any model space files. -- *Model interchange* refers to the format used to transfer model information between PEtab Select and a PEtab-compatible calibration tool, during the model selection process. -- *Report* refers to the final results of the model selection process, which may include calibration results from any calibrated models, or just the select model. - -Here, the format for a single model is shown. Multiple models can be specified as a YAML list of the same format. - -The only required key is the PEtab YAML, as a model requires a PEtab problem. All other keys are maybe required, for the different uses of the format (e.g., the report format should include ``estimated_parameters``), or at different stages of the model selection process (the PEtab-compatible calibration tool should provide ``criteria`` for model comparison). + * - model_subspace_id + - petab_yaml + - p1 + - p2 + * - subspace1 + - petab_problem.yaml + - 0;estimate + - 10;20;estimate + +3. Model(s) (Predecessor models / model interchange / report) +------------------------------------------------------------- + +- *Predecessor models* are used to initialize a compatible model selection + method. +- *Model interchange* refers to the format used to transfer model information + between PEtab Select and a PEtab-compatible calibration tool, during the + model selection process. +- *Report* refers to the final results of the model selection process, which + may include calibration results from any calibrated models, or just the + selected model. + +Here, the format for a single model is shown. Multiple models can be specified +as a YAML list of the same format. The only required key is the ``petab_yaml``, +as a model requires a PEtab problem. Other keys are required in different +contexts (for example, model comparison will require ``criteria``). .. code-block:: yaml @@ -107,10 +126,11 @@ The only required key is the PEtab YAML, as a model requires a PEtab problem. Al estimated_parameters: # dict[string, float] (optional). Parameter ID => parameter value. model_hash: # string (optional). model_id: # string (optional). + model_subspace_id: # string (optional). + model_subspace_indices: # string (optional). parameters: # dict[string, float] (optional). Parameter ID => parameter value or "estimate". petab_yaml: # string. predecessor_model_hash: # string (optional). - sbml: # string (optional). - ``criteria``: The value of the criterion by which model selection was performed, at least. Optionally, other criterion values too. - ``estimated_parameters``: Parameter estimates, not only of parameters specified to be estimated in a model space file, but also parameters specified to be estimated in the original PEtab problem of the model. @@ -121,4 +141,3 @@ The only required key is the PEtab YAML, as a model requires a PEtab problem. Al - ``parameters``: The parameters from the problem (either values or ``'estimate'``) (a specific combination from a model space file, but uncalibrated). - ``petab_yaml``: Same as in model space files. - ``predecessor_model_hash``: The hash of the model that preceded this model during the model selection process. -- ``sbml``: Same as in model space files. diff --git a/petab_select/constants.py b/petab_select/constants.py index 692e0198..bba90ac6 100644 --- a/petab_select/constants.py +++ b/petab_select/constants.py @@ -49,7 +49,6 @@ PREDECESSOR_MODEL_HASH = "predecessor_model_hash" PETAB_PROBLEM = "petab_problem" PETAB_YAML = "petab_yaml" -SBML = "sbml" HASH = "hash" # MODEL_SPACE_FILE_NON_PARAMETER_COLUMNS = [MODEL_ID, PETAB_YAML] diff --git a/test/pypesto/test_pypesto.py b/test/pypesto/test_pypesto.py index d64de876..de6f2576 100644 --- a/test/pypesto/test_pypesto.py +++ b/test/pypesto/test_pypesto.py @@ -19,9 +19,10 @@ # Set to `[]` to test all test_cases = [ - #'0006', - #'0002', - #'0008', + # '0006', + # '0002', + # '0008', + # "0009", ] # Do not use computationally-expensive test cases in CI diff --git a/test_cases/0009/predecessor_model.yaml b/test_cases/0009/predecessor_model.yaml index c2cba72d..4471224b 100644 --- a/test_cases/0009/predecessor_model.yaml +++ b/test_cases/0009/predecessor_model.yaml @@ -1,8 +1,8 @@ criteria: NLLH: inf estimated_parameters: {} -model_hash: cf9e9633bb9566b26275cc273997fe8c873e4b017e84da13889951f93b6a2950b334af9cc97b582bd43626834d09cd028803056c97dd4692f1e4b209258752a7 -model_id: cf9e9633bb9566b26275cc273997fe8c873e4b017e84da13889951f93b6a2950b334af9cc97b582bd43626834d09cd028803056c97dd4692f1e4b209258752a7 +model_hash: M-01000001111110010011010011010001 +model_id: M-01000001111110010011010011010001 model_subspace_id: M model_subspace_indices: - 0 @@ -71,4 +71,4 @@ parameters: a_k16_k08k16: 1 a_k16_k12k16: estimate petab_yaml: petab/petab_problem.yaml -predecessor_model_hash: None +predecessor_model_hash: null