From 59c8d975733a12a36cd45c74d676a9201b815458 Mon Sep 17 00:00:00 2001 From: Jon Clucas Date: Wed, 4 Sep 2024 14:15:17 -0400 Subject: [PATCH 1/3] :memo: Update `FROM` base from `default` to `blank` --- bin/build | 19 +++++++++++----- docs/_sources/developer/utils.rst | 12 ++++++++++ .../developer/workflows/cpac_pipeline.rst | 2 +- .../user/pipelines/pipeline_config.rst | 7 ++++-- docs/_sources/user/pipelines/preconfig.rst | 22 ++++++++++--------- 5 files changed, 43 insertions(+), 19 deletions(-) diff --git a/bin/build b/bin/build index ff9cbdef6..b5ec69d14 100755 --- a/bin/build +++ b/bin/build @@ -1,11 +1,18 @@ #!/bin/bash # link references -CPAC_DIRECTORY=$(python -c "import os, CPAC; print(os.path.abspath(os.path.dirname(CPAC.__file__)))") -ln ${CPAC_DIRECTORY}/resources/configs/1.7-1.8-nesting-mappings.yml docs/_sources/references/1.7-1.8-nesting-mappings.yml || true -ln ${CPAC_DIRECTORY}/resources/configs/1.7-1.8-deprecations.yml docs/_sources/references/1.7-1.8-deprecations.yml || true -ln ${CPAC_DIRECTORY}/resources/configs/pipeline_config_default.yml docs/_sources/references/default_pipeline.yml || true -ln ${CPAC_DIRECTORY}/resources/configs/group_config_template.yml docs/_sources/references/group_config_template.yml || true +CPAC_DIRECTORY=$(python -c "from importlib import resources; print(str(resources.files('CPAC')))") +ln "${CPAC_DIRECTORY}/resources/configs/1.7-1.8-nesting-mappings.yml" docs/_sources/references/1.7-1.8-nesting-mappings.yml || true +ln "${CPAC_DIRECTORY}/resources/configs/1.7-1.8-deprecations.yml" docs/_sources/references/1.7-1.8-deprecations.yml || true +shopt -s globstar nullglob +for PRECONFIG in "${CPAC_DIRECTORY}"/resources/configs/pipeline_config_*.yml +do + FILENAME=$(basename "$PRECONFIG") + CONFIG_NAME="${FILENAME#pipeline_config_}" + CONFIG_NAME="${CONFIG_NAME%.yml}" + ln "${PRECONFIG}" "docs/_sources/references/${CONFIG_NAME}_pipeline.yml" || true +done +ln "${CPAC_DIRECTORY}/resources/configs/group_config_template.yml" docs/_sources/references/group_config_template.yml || true # build docs -sphinx-build -b html docs/_sources docs/$1 +sphinx-build -b html docs/_sources "docs/$1" diff --git a/docs/_sources/developer/utils.rst b/docs/_sources/developer/utils.rst index ee6bf514f..8c36d01d7 100644 --- a/docs/_sources/developer/utils.rst +++ b/docs/_sources/developer/utils.rst @@ -22,6 +22,18 @@ Configuration :members: :undoc-members: +.. automodule:: CPAC.utils.configuration.configuration + :members: + :undoc-members: + +.. automodule:: CPAC.utils.configuration.diff + :members: + :undoc-members: + +.. automodule:: CPAC.utils.configuration.yaml_template + :members: + :undoc-members: + Create FMRIB's Local Analysis of Mixed Effects (FLAME) Model Files ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/docs/_sources/developer/workflows/cpac_pipeline.rst b/docs/_sources/developer/workflows/cpac_pipeline.rst index a1e351790..2d86075a5 100644 --- a/docs/_sources/developer/workflows/cpac_pipeline.rst +++ b/docs/_sources/developer/workflows/cpac_pipeline.rst @@ -7,7 +7,7 @@ C-PAC Pipeline Construction Configuration Object ==================== -The 'c' variable in cpac_pipeline.py is a `Configuration object `_, which is an abstract representation used to store the options for the pipeline configuration YAML after it is read in by C-PAC. Each attribute of the Configuration object is a key from the YAML file, with values mirroring the YAML. +The 'c' variable in cpac_pipeline.py is a :py:class:`~CPAC.utils.configuration.configuration.Configuration` object, which is an abstract representation used to store the options for the pipeline configuration YAML after it is read in by C-PAC. Each attribute of the Configuration object is a key from the YAML file, with values mirroring the YAML. Strategy Object =============== diff --git a/docs/_sources/user/pipelines/pipeline_config.rst b/docs/_sources/user/pipelines/pipeline_config.rst index e3f671451..205f818ec 100644 --- a/docs/_sources/user/pipelines/pipeline_config.rst +++ b/docs/_sources/user/pipelines/pipeline_config.rst @@ -51,7 +51,10 @@ If you want to base a pipeline on another pipeline configuration YAML file, you FROM: /path/to/pipeline.yml -in your pipeline configuration file. You can use the name of a :doc:`preconfigured pipeline ` instead of a filepath if you want to base a configuration file on a preconfigured pipeline. If ``FROM`` is not specified, the pipeline will be based on :doc:`the default pipeline `. +in your pipeline configuration file. You can use the name of a :doc:`preconfigured pipeline ` instead of a filepath if you want to base a configuration file on a preconfigured pipeline. If ``FROM`` is not specified, the pipeline will be based on :doc:`the blank pipeline `. + +.. versionchanged:: 1.8.5 + From version 1.8.0 to version 1.8.5, unspecified keys were based on the default configuration rather than the blank preconfiguration. C-PAC will include all expected keys from the pipeline file specified in ``FROM`` (or the default pipeline if none is specified). Any keys specified in a pipeline configuration file will take precedence over the same key in the ``FROM`` base configuration, but all omitted keys will retain their values from the ``FROM`` base configuration. @@ -59,7 +62,7 @@ From terminal, you can quickly generate a default pipeline configuration YAML fi cpac utils pipe-config new-template -You can then edit the file as needed. For values that you want to leave at the default, you can either leave the key as-is, or you can remove the key, and C-PAC will automatically use value from the default pipeline configuration (or from the pipeline specified in ``FROM``). +You can then edit the file as needed. For values that you want to leave at the default, you can either leave the key as-is, or you can add the line ``FROM: default``, remove the key, and C-PAC will automatically use value from the default pipeline configuration (or from the pipeline specified in ``FROM``). If you want to run the analysis from terminal:: diff --git a/docs/_sources/user/pipelines/preconfig.rst b/docs/_sources/user/pipelines/preconfig.rst index 1d3a6d50e..47ca1e4eb 100644 --- a/docs/_sources/user/pipelines/preconfig.rst +++ b/docs/_sources/user/pipelines/preconfig.rst @@ -7,7 +7,7 @@ human default: The Default Pipeline ----------------------------- -Pipeline Configuration YAML: `https://github.com/FCP-INDI/C-PAC/blob/main/CPAC/resources/configs/pipeline_config_default.yml `_ +:doc:`Pipeline Configuration YAML ` .. note:: @@ -54,6 +54,8 @@ References abcd-options ------------ +:doc:`Pipeline Configuration YAML ` + .. warning:: :doc:`/user/known-issues/FCP-INDI/C-PAC/2104` @@ -61,7 +63,7 @@ abcd-options anat-only: Default with Anatomical Preprocessing Only ----------------------------------------------------- -Pipeline Configuration YAML: `https://github.com/FCP-INDI/C-PAC/blob/main/CPAC/resources/configs/pipeline_config_anat-only.yml `_ +:doc:`Pipeline Configuration YAML ` Based on the preprocessing decisions of the default pipeline, this preconfiguration allows you to immediately kick off a run with only anatomical preprocessing selected. This includes: @@ -72,7 +74,7 @@ Based on the preprocessing decisions of the default pipeline, this preconfigurat preproc: Default without Derivatives ------------------------------------ -Pipeline Configuration YAML: `https://github.com/FCP-INDI/C-PAC/blob/main/CPAC/resources/configs/pipeline_config_preproc.yml `_ +:doc:`Pipeline Configuration YAML ` Based on the preprocessing decisions of the default pipeline, this preconfiguration allows you to preprocess all of your data, without launching into calculation of outputs and data derivatives. This includes: @@ -90,10 +92,10 @@ Functional: * Nuisance correction & filtering * Registration to template (via ANTs/ITK) -fmriprep-options: fmriprep-Options Pipeline +fmriprep-options: fMRIPrep options Pipeline ------------------------------------------- -Pipeline Configuration YAML: `https://github.com/FCP-INDI/C-PAC/blob/main/CPAC/resources/configs/pipeline_config_fmriprep-options.yml `_ +:doc:`Pipeline Configuration YAML ` This pipeline is designed to increase reproducibility with the preprocessing results of the fmriprep pipeline package\ :footcite:`fMRI16` produced by the `Poldrack Lab at Stanford University `_. @@ -111,7 +113,7 @@ References ndmg: Neurodata's 'ndmg-f' Pipeline ----------------------------------- -Pipeline Configuration YAML: `https://github.com/FCP-INDI/C-PAC/blob/main/CPAC/resources/configs/pipeline_config_ndmg.yml `_ +:doc:`Pipeline Configuration YAML ` This pipeline is the result of `Neurodata's `_ study to converge upon the intersection of pipeline configuration decisions that maximizes discriminability between participants' data, drawing from the connectome graphs produced (labeled 'ndmg_graph' in the C-PAC output directory). This pipeline invokes a minimal set of preprocessing. @@ -131,7 +133,7 @@ References rbc-options: ReproBrainChart Options Pipeline --------------------------------------------- -Pipeline Configuration YAML: `https://github.com/FCP-INDI/C-PAC/blob/main/CPAC/resources/configs/pipeline_config_rbc-options.yml `_ +:doc:`Pipeline Configuration YAML ` RBC-options pipeline was built and integrated in C-PAC based on the Reproducible Brain Charts initiative, which aims to aggregate and harmonize phenotypic and neuroimage data to delineate node mechanisms regarding developmental basis of psychopathology in youth and yield reproducible growth charts of brain development\ :footcite:`Hoff21`. @@ -146,7 +148,7 @@ non-human primate monkey: Default with Monkey Preprocessing ----------------------------------------- -Pipeline Configuration YAML: `https://github.com/FCP-INDI/C-PAC/blob/main/CPAC/resources/configs/pipeline_config_monkey.yml `_ +:doc:`Pipeline Configuration YAML ` This pipeline is based on the work of Xu et al.\ :footcite:`Xu19` and nhp-ABCD-BIDS-pipeline.\ :footcite:`Stur20` @@ -188,14 +190,14 @@ testing benchmark-ANTS: C-PAC Benchmark with ANTs Registration ------------------------------------------------------ -Pipeline Configuration YAML: `https://github.com/FCP-INDI/C-PAC/blob/main/CPAC/resources/configs/pipeline_config_benchmark-ANTS.yml `_ +:doc:`Pipeline Configuration YAML ` The benchmark pipeline has remained mostly unchanged since the project's inception, and is used at the end of each release cycle to ensure the results of C-PAC's key outputs have not changed. It is designed to test a wide range of pipeline options. This pipeline is based on registration-to-template using the ANTs/ITK toolset, as this decision impacts many other aspects of the pipeline further downstream. benchmark-FNIRT: C-PAC Benchmark with FSL FNIRT Registration ------------------------------------------------------------ -Pipeline Configuration YAML: `https://github.com/FCP-INDI/C-PAC/blob/main/CPAC/resources/configs/pipeline_config_benchmark-FNIRT.yml `_ +:doc:`Pipeline Configuration YAML ` The benchmark pipeline has remained mostly unchanged since the project's inception, and is used at the end of each release cycle to ensure the results of C-PAC's key outputs have not changed. It is designed to test a wide range of pipeline options. This pipeline is based on registration-to-template using the FSL FLIRT & FNIRT, as this decision impacts many other aspects of the pipeline further downstream. From 09d86a7f5ae1f67572999eb35a2aac8a3120f344 Mon Sep 17 00:00:00 2001 From: Jon Clucas Date: Thu, 5 Sep 2024 09:58:11 -0400 Subject: [PATCH 2/3] :loud_sound: Log linking --- bin/build | 1 + docs/_sources/developer/utils.rst | 4 ---- 2 files changed, 1 insertion(+), 4 deletions(-) diff --git a/bin/build b/bin/build index b5ec69d14..8a60d429c 100755 --- a/bin/build +++ b/bin/build @@ -10,6 +10,7 @@ do FILENAME=$(basename "$PRECONFIG") CONFIG_NAME="${FILENAME#pipeline_config_}" CONFIG_NAME="${CONFIG_NAME%.yml}" + echo "ln \"${PRECONFIG}\" \"docs/_sources/references/${CONFIG_NAME}_pipeline.yml\" || true" ln "${PRECONFIG}" "docs/_sources/references/${CONFIG_NAME}_pipeline.yml" || true done ln "${CPAC_DIRECTORY}/resources/configs/group_config_template.yml" docs/_sources/references/group_config_template.yml || true diff --git a/docs/_sources/developer/utils.rst b/docs/_sources/developer/utils.rst index 8c36d01d7..1a8094e23 100644 --- a/docs/_sources/developer/utils.rst +++ b/docs/_sources/developer/utils.rst @@ -18,10 +18,6 @@ BIDS Data Configuration Configuration ^^^^^^^^^^^^^ -.. automodule:: CPAC.utils.configuration - :members: - :undoc-members: - .. automodule:: CPAC.utils.configuration.configuration :members: :undoc-members: From 1b384d23fa51f91b63d3212e89da1baaac3cd06b Mon Sep 17 00:00:00 2001 From: Jon Clucas Date: Thu, 5 Sep 2024 11:45:47 -0400 Subject: [PATCH 3/3] :recycle: Literal-include YAML configs --- docs/_sources/conf.py | 24 ++++++++++++++++++++++ docs/_sources/user/pipelines/preconfig.rst | 20 +++++++++--------- 2 files changed, 34 insertions(+), 10 deletions(-) diff --git a/docs/_sources/conf.py b/docs/_sources/conf.py index ba3e2d7b4..33df2135a 100644 --- a/docs/_sources/conf.py +++ b/docs/_sources/conf.py @@ -13,6 +13,7 @@ # serve to show the default. import os +from pathlib import Path import re import sys from typing import Any @@ -28,6 +29,7 @@ from nipype import __version__ as _nipype_version import semver from pybtex.plugin import register_plugin +import yaml sys.path.append(os.path.dirname(__file__)) @@ -139,7 +141,29 @@ def yaml_to_rst(path): open(path, 'w').write(''.join(lines)) +def literal_include_config_page(filepath: Path) -> None: + """Create a page with a literalinclude of a config file.""" + if not filepath.exists(): + return + with filepath.open("r", encoding="utf8") as _f: + config: dict[str, Any] = yaml.safe_load(_f) + title: str = config["pipeline_setup"]["pipeline_name"] + with (Path("user/pipelines") / + filepath.name.replace("_pipeline.yml", ".rst") + ).open("w", encoding="utf8") as _f: + _f.write(f""".. title:: {title} pipeline configuration YAML file + +.. literalinclude:: /{filepath} + :language: YAML +""") + return + + yaml_to_rst('references/1.7-1.8-deprecations.yml') +for ref in Path("references").iterdir(): + if str(ref).endswith("_pipeline.yml"): + # create literalinclude page + literal_include_config_page(ref) # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the diff --git a/docs/_sources/user/pipelines/preconfig.rst b/docs/_sources/user/pipelines/preconfig.rst index 47ca1e4eb..05d3dab25 100644 --- a/docs/_sources/user/pipelines/preconfig.rst +++ b/docs/_sources/user/pipelines/preconfig.rst @@ -7,7 +7,7 @@ human default: The Default Pipeline ----------------------------- -:doc:`Pipeline Configuration YAML ` +:doc:`Pipeline Configuration YAML ` .. note:: @@ -54,7 +54,7 @@ References abcd-options ------------ -:doc:`Pipeline Configuration YAML ` +:doc:`Pipeline Configuration YAML ` .. warning:: @@ -63,7 +63,7 @@ abcd-options anat-only: Default with Anatomical Preprocessing Only ----------------------------------------------------- -:doc:`Pipeline Configuration YAML ` +:doc:`Pipeline Configuration YAML ` Based on the preprocessing decisions of the default pipeline, this preconfiguration allows you to immediately kick off a run with only anatomical preprocessing selected. This includes: @@ -74,7 +74,7 @@ Based on the preprocessing decisions of the default pipeline, this preconfigurat preproc: Default without Derivatives ------------------------------------ -:doc:`Pipeline Configuration YAML ` +:doc:`Pipeline Configuration YAML ` Based on the preprocessing decisions of the default pipeline, this preconfiguration allows you to preprocess all of your data, without launching into calculation of outputs and data derivatives. This includes: @@ -95,7 +95,7 @@ Functional: fmriprep-options: fMRIPrep options Pipeline ------------------------------------------- -:doc:`Pipeline Configuration YAML ` +:doc:`Pipeline Configuration YAML ` This pipeline is designed to increase reproducibility with the preprocessing results of the fmriprep pipeline package\ :footcite:`fMRI16` produced by the `Poldrack Lab at Stanford University `_. @@ -113,7 +113,7 @@ References ndmg: Neurodata's 'ndmg-f' Pipeline ----------------------------------- -:doc:`Pipeline Configuration YAML ` +:doc:`Pipeline Configuration YAML ` This pipeline is the result of `Neurodata's `_ study to converge upon the intersection of pipeline configuration decisions that maximizes discriminability between participants' data, drawing from the connectome graphs produced (labeled 'ndmg_graph' in the C-PAC output directory). This pipeline invokes a minimal set of preprocessing. @@ -133,7 +133,7 @@ References rbc-options: ReproBrainChart Options Pipeline --------------------------------------------- -:doc:`Pipeline Configuration YAML ` +:doc:`Pipeline Configuration YAML ` RBC-options pipeline was built and integrated in C-PAC based on the Reproducible Brain Charts initiative, which aims to aggregate and harmonize phenotypic and neuroimage data to delineate node mechanisms regarding developmental basis of psychopathology in youth and yield reproducible growth charts of brain development\ :footcite:`Hoff21`. @@ -148,7 +148,7 @@ non-human primate monkey: Default with Monkey Preprocessing ----------------------------------------- -:doc:`Pipeline Configuration YAML ` +:doc:`Pipeline Configuration YAML ` This pipeline is based on the work of Xu et al.\ :footcite:`Xu19` and nhp-ABCD-BIDS-pipeline.\ :footcite:`Stur20` @@ -190,14 +190,14 @@ testing benchmark-ANTS: C-PAC Benchmark with ANTs Registration ------------------------------------------------------ -:doc:`Pipeline Configuration YAML ` +:doc:`Pipeline Configuration YAML ` The benchmark pipeline has remained mostly unchanged since the project's inception, and is used at the end of each release cycle to ensure the results of C-PAC's key outputs have not changed. It is designed to test a wide range of pipeline options. This pipeline is based on registration-to-template using the ANTs/ITK toolset, as this decision impacts many other aspects of the pipeline further downstream. benchmark-FNIRT: C-PAC Benchmark with FSL FNIRT Registration ------------------------------------------------------------ -:doc:`Pipeline Configuration YAML ` +:doc:`Pipeline Configuration YAML ` The benchmark pipeline has remained mostly unchanged since the project's inception, and is used at the end of each release cycle to ensure the results of C-PAC's key outputs have not changed. It is designed to test a wide range of pipeline options. This pipeline is based on registration-to-template using the FSL FLIRT & FNIRT, as this decision impacts many other aspects of the pipeline further downstream.