BIDS data I/O specifications

[ghisvail]

This thread aggregates ideas for a design proposal for BIDS data reading and writing in Clinica with the new Pydra-based engine.

After careful reflection, I am thinking of the following design:

Reading

def bids_data_reader(
    dataset_path: PathLike[str],
    output_files: Mapping[str, dict],
    subject_ids: Sequence[str] | None = None,
    session_ids: Sequence[str] | None = None,
    **kwargs,
) -> FunctionalTask:
    ...

where:

argument	type	example	description
dataset_path	path-like	Path("/path/to/bids/dataset")	Root path to the BIDS dataset to read from
output_files	mapping of str to dict)	{"t1w": {"datatype": "anat", "suffix": "T1w"}}	Which BIDS files will be read and exposed as named outputs of the task
subject_ids	sequence of str (optional)	["sub-P01", "sub-P20"]	Restrict reading to the following subjects
session_ids	sequence of str (optional)	["ses-M00", "ses-M00"]	Restrict reading to the following sessions

In this example, the reader is configured to fetch T1w images for session ses-M00 of participant sub-P01 and session ses-M00 of participant sub-P20. subject_id and session_id would be typically used to implement our filtering feature with a user-provided TSV file containing the combinations of participant - session to keep.

Writing

def bids_data_writer(
    dataset_path: PathLike[str], input_files: Mapping[str, dict], **kwargs
) -> FunctionalTask:
    ...

where:

argument	type	example	description
dataset_path	path-like	Path("/path/to/bids/dataset")	Root path to the BIDS dataset to write to
input_files	mapping of str to dict)	{"t1w_corrected": {"datatype": "anat", "suffix": "T1w", "entities": {"desc": "corrected"}}}	Which BIDS files will be written from the named inputs of the task

In this example, the writer is configured to write a derivative T1w image with an additional desc entity.

Usage

wf = pydra.Workflow(...)

wf.add(
    read_task = bids_data_reader(
        dataset_path=Path("/path/to/input/dataset"),
        output_files={"t1w": {"datatype": "anat", "suffix": "T1w"}},
        name="read_dataset",
    )
)

wf.add(
    correct_image(
        input_image=wf.read_dataset.lzout.t1w,
        name="correct_t1w",
    )
)

wf.add(
    write_task = bids_data_writer(
        dataset_path=Path("/path/to/output/dataset"),
        output_files={"t1w_corrected": {
            "datatype": "anat",
            "suffix": "T1w",
            "entities": {"desc": "corrected"},
        }},
        t1w_corrected=wf.correct_t1w.lzout.output_image,
        name="write_dataset",
    )
)

Support for CAPS outputs

One important challenge is to support our own CAPS outputs, which are not strictly BIDS compliant. I believe the combination of datatype and entities should make the design flexible enough to support most CAPS files (the ones where the filename is BIDS-compliant at least) with calls like:

writer = bids_data_writer(
    input_files={
        "xfm": {
            "datatype": "anat",
            "suffix": "affine",
            "entities": {
                "space": "MNI152NLin2009cSym",
                "res": "1x1x1",
            },
        }
    },
)

writer.inputs.dataset_path = "/path/to/derivative"
writer.inputs.subject_id = "sub-ADNI029S1384"
writer.inputs.session_id = "ses-M000"
writer.inputs.xfm = some_task.lzout.xfm_file
# Writer should copy the transform file produced by the upstream task to
# /path/to/derivative/sub-ADNI029S1384/ses-M000/anat/sub-ADNI029S1384_ses-M000_space-MNI152NLin2009cSym_res-1x1x1_affine.mat 

For less compliant outputs such as FreeSurfer or longitudinal computations, we could still use datatype to tag them and let them provide their respective parameters. For instance:

writer = bids_data_writer(
    input_files={
        "freesurfer_dir": {
            datatype="caps_freesurfer",  # custom type for FreeSurfer
            subdir="t1/freesurfer_cross_sectional",  # subdir within sub / ses
    },
}

writer.inputs.dataset_path = "/path/to/caps"
writer.inputs.subject_id = "sub-01"
writer.inputs.session_id = "ses-2011"
writer.inputs.freesurfer_dir = some_task.lzout.subjects_dir
# Writer should copy the upstream task's subjects_dir to
# /path/to/caps/subjects/sub-01/ses-2011/t1/freesurfer_cross_sectional/sub-01_ses-2011