Enhance code with BIDS abstractions

[omar-rifai]

Code that handles BIDS datasets is scattered across the code base. It would be useful to have abstract classes that centralize the manipulation of BIDS (and derivative?) directories.

Following a discussion with @ghisvail and @MatthieuJoulot, the following abstractions could be introduced:

• class BIDSReader(): Centralizes methods for reading BIDS datasets and extracting information from the data structure

• class BIDSWriter(): Centralizes methods for writing BIDS datasets end enforcing coherence with specifications

• class BIDSModality(datatype: str, suffix: str, entities: list[str, str], participant_id: str, session_id: str): abstract imaging data for easier querying and manipulation in above classes

• class BIDS[json metadata](): enforces schema on required json files (cf PR430 for an example with the required dataset_description.json)

• class BIDS[phenotype](): enforces schema on phenotype files

[omar-rifai]

Look into https://github.com/bids-standard/pybids

[ghisvail]

Comparison of active BIDS implementations

Source: https://bids.neuroimaging.io/benefits.html

Software	Last release	Documented	Tested	Abstractions	Limitations
pybids	0.14 (2021-11-09)	Yes	Yes	BIDSLayout	Query operations only, no support for BIDS derivative.
mne-bids	0.9 (2021-11-23)	Yes	Yes	BIDSPath	Restricted to MEG, EEG and iEEG specifications.
ancp-bids	0.0.13 (2021-09-09)	No	Yes	Dataset, Query, Entity, Suffix, File	Experimental
bidskit	2019.02.20 (2019-02-20)	Yes	No	BIDSTree	Write operations only, focused on anatomical MRI
dcm2bids	2.1.6 (2021-02-27)	Yes	Yes	Participant, Acquisition, Sidecar, SidecarPairing	No API documentation, not designed for re-use
heudiconv	0.10.0 (2021-10-08)	Yes	Yes	None	Not designed for re-use
bidscoin	3.7.0 (2021-12-24)	No	Yes	File-based schema definition	Not designed for re-use
bidsme	1.3.5r8 (2021-11-22)	No	No	BIDSTable, BIDSField, BIDSSession	Incompatible license
phys2bids	2.6.0 (2021-06-18)	Yes	Yes	None	Restricted to physio, not designed for re-use
BIDSHandler	0.2.0 (2019-02-05)	Yes	No	BIDSTree, Project, Subject, Session, Scan	Focused on MEG, experimental, unmaintained

Possible validation against bids-examples and bids-langloc public datasets.

[ghisvail]

There is also a limited effort to actually model the specification in pybids. In my opinion, both mne-bids and ancp-bids actually attempt to properly define a model for manipulating BIDS metadata in addition to raw files. BIDSHandler pushes the level of abstraction even further, but it has been inactive since 2019 and is focused on the MEG BIDS extension according to the README.

[omar-rifai]

Could we add a column to the table where we have the main limitation for each library? I feel like this is something that will be useful down the road since going forward with our own implementation is a costly investment.

[ghisvail]

Great suggestion, I'll add it.

[ghisvail]

Updated

[ghisvail]

I have added the Documented attribute and remove the Type checked one, since none of them are anyway.

[ghisvail]

Here are the main usage of BIDS within Clinica which I have been able to identify from my recent audit:

Converters

• Write modality-agnostic files following the BIDS specification
• Compute paths to modality-specific files for raw datasets
• Write dataset description file for raw datasets

Pipelines

• Compute paths to modality-specific files for raw datasets

• Detect availability of a given modality in a raw dataset

  ex: T1, PET, DWI, ...

• Read modality-agnostic files for participant / session

• Compute path to files with custom BIDS entities in derivatives

  ex: FWHM, DWI PREPROCESSING, T1 FREESURFER, ...

• Understand the group concept from a CAPS output

[ghisvail]

For now, most of these are handled by raw path manipulation and is lacking proper data modeling / encapsulation for high-order concepts such as BIDS entities or CAPS groups.

[ghisvail]

Here is my personnal summary of the BIDS spec with Clinica's usage in mind. I have highlighted a few concepts and domain knowledge, which may be worth abstracting over.

BIDS_specs_summary.md

[ghisvail]

The next step is to find a design that model these concepts well enough to capture all these business rules, whilst making invalid values impossible to represent. Hence some focus on type safety in my analysis of the existing ecosystem earlier.

[ghisvail]

We are also focusing on the structural aspect of the spec. Whether the content of the metadata, sidecar and scan files are compliant is not the topic here and would likely involve other abstractions to model the necessary constraints, which we don't have a use for (yet).

[ghisvail]

Apart for dataset_description.json which we have addressed for the converters, and should also provide for our derivatives in the near future.

[ghisvail]

Typesafe construction of BIDS path

BIDS-like paths

This is an idea for a BIDS abstraction for safer handling of BIDS-like paths. I am making the clear distinction between BIDS-compliant and BIDS-like paths. The latter corresponds to the general formalism of what a BIDS filename should look like. The former would be validated against a specific modality which mandates additional constraints such as what entities and extensions are valid and their ordering in the naming scheme.

For instance:

• sub-P01_ses-M00_T1w.nii.gz is both a valid BIDS-like and BIDS-compliant name as per the anat/T1w convention.
• ses-M00_sub-P01_T1w.nii.gz satisfies a BIDS-like formalism but is not BIDS-compliant because the ordering of the subject and session entities is wrong.
• sub-P01_T1w_acq-lowres.nii.gz is not BIDS-like because the suffix does not appear before the extension. This is an issue we typically have in our CAPS specification at the moment.
• sub-P01_acq-lowres_acq-PIB_pet.nii.gz is not BIDS-like because the acquisition entity is repeated multiple times.

Proposal: define a BIDS path abstraction which would capture these constraints.

path = Subject("P01") / Session("M00") / Suffix("T1w") / Extension(".nii.gz")
print(path.directory)
>>> 'sub-P01/ses-M00/anat'
print(path.filename)
>>> 'sub-P01_ses-M00_T1w.nii.gz'

# Raise en exception, or return an invalid BIDS-like type.
path = Subject("P01") / Suffix("T1w") / Acquisition("lowres") / Extension(".nii.gz")

# Raise en exception, or return an invalid BIDS-like type.
path = Subject("P01") / Acquisition("lowres") / Acquisition("PIB") / Suffix("pet") / Extension(".nii.gz")

Here I am re-using the / operator introduced by pathlib to combine path components.

An alternative notation could be BIDSPath.join([Subject(...), Session(...), Suffix(...), Extension(...)]) with the same effect on validation underneath.

BIDS-compliant paths

BIDS-compliance basically links a BIDS-like path to a specific version of a BIDS specification (raw or derivative). As the BIDS specification progresses, new modalities are introduced whilst ambiguous definitions are deprecated.

For instance,

• sub-P01_ses-M00_pet.nii.gz is compliant with BIDS version 1.6 and above, where the PET modality was introduced, but not with earlier versions of the specification.

• sub-P01_ses-M00_PD.nii.gz used to be compliant with earlier versions of BIDS but is now deprecated in favour of more explicit suffixes such as PDw and PDmap.

Proposal: make this notion of compliance explicit to transform a BIDS-like path to a compliant one.

bids_like_path = Subject("P01") / Modality("pet") / Suffix(".nii.gz")

# Returns a BIDS-compliant path. 
maybe_bids_path = BIDSSpec(version="1.6.0").validate(bids_like_path)

# Raise an exception, or return as BIDS like path.
maybe_bids_path = BIDSSpec(version="1.4.0").validate(bids_like_path)

[omar-rifai]

This looks great ! I have a couple of questions though about the design:

• Are all of the "basic" objects you manipulate going to inherit from a common type? (e.g BIDSPath type ?)
• In the first example (prior to introducting the BIDSSpec class), you mention raising an exception on execution. Is the check for a valid path going to be done by default with respect to a globally-set BIDSSpec version?

[ghisvail]

Are all of the "basic" objects you manipulate going to inherit from a common type? (e.g BIDSPath type ?)

Entities such as Subject, Session or Acquisition will likely inherit from a common Entity ancestor. I am not sure yet for Entity, Suffix and Extension. The goal remains to define an algebra which only accept valid BIDS composition rules, such as Entity / [Entity | Suffix], Suffix / Extension and Extension cannot be composed further.

Is the check for a valid path going to be done by default with respect to a globally-set BIDSSpec version?

BIDS-likeness, i.e. whether a filename is structurally compliant with BIDS, can be checked without a specification.

BIDS-compliance requires BIDS-likeness plus a validation against a BIDS specification.

[ghisvail]

Operations on BIDS filenames

Construction

path = Subject("P01") / Suffix("T1w") / Extension(".nii.gz")
t1w = BIDSFilename(path)

t1w.directory
>>> 'sub-P01/anat'

t1w.name
>>> 'sub-P01_T1w.nii.gz'

t1w.path
>>> 'sub-P01/anat/sub-P01_T1w.nii.gz'

sidecar = t1w.with_extension(Extension(".json"))
sidecar.name
>>> 'sub-P01_T1w.json'

pet = t1w.with_suffix(Suffix("pet")).path
>>> 'sub-P01/pet/sub-P01_pet.nii.gz'

other_subject = t1w.with_entity(Subject('P02')).path
>>> 'sub-P02/anat/sub-P02_T1w.nii.gz'

Querying

path = Subject("P01") / Run(1) / Suffix("T1w") / Extension(".nii.gz")
filename = BIDSFilename(path)

filename.entity("sub")
>>> "P01"
filename.entity("run")
>>> 1
filename.entity("ses")
>>> None
filename.suffix
>>> "T1w"
filename.extension
>>> ".nii.gz"

[ghisvail]

BIDS model and schema

There is a partial definition of the BIDS specification to a machine-readable format. The latter is used to generate the template sections in the web version.

Other terms of the BIDS taxonomy are defined such as modalities, metadata, associated data, templates or top-level files.

[ghisvail]

Domain analysis

BIDS dataset

BIDS dataset is the highest level concept of the BIDS specification. It is the first term introduced in the common principles and is used consistently within the specification to refer to the whole set of neuroimaging data and metadata.

There are two types of BIDS dataset: raw and derivative. Raw datasets must strictly comply with a certain version of a BIDS specification, whilst derivative datasets should follow their own subset with more relaxed constraints.

The nominal set of attributes defining a dataset are its name, version of the BIDS specification and dataset type, all of which must be persisted to a dataset_description.json file located at the root path.

BIDS data and files

A BIDS dataset aggregates imaging data and metadata as a tree-like hierarchy of modality-agnostic (MAF) and modality-specific files (MSF). MAF may stored in tabular or key-valued formats and MSF are stored in native recording format with additional metadata stored as a separate key-value mapping. The prescribed file formats are TSV for tabular data, JSON for key-valued data and compressed NIfTI for imaging data.

Modality-specific data are always located at the bottom of the tree, whilst modality-agnostic data may appear anywhere above in the hierarchy, based on which level of metadata their corresponding data are associated with. Modality-specific metadata may appear anywhere in the tree and be composed and redefined following the BIDS inheritance principle.

BIDS specification

Given a BIDS dataset, the list of valid data (both MAF ans MSF), filenaming scheme, optional and required metadata are defined by a specific version of the BIDS specification. The latter uses concepts such as entity, modality, datatype, suffix, extension to model the list of possible file collections available within a BIDS hierarchy.

The BIDS specification has been improved with BIDS extension proposals introducing new modalities, refinement of the required and optional modality-specific metadata, deprecation of ambiguous suffixes, etc...

As far as BIDS derivatives are concerned, there were only recommandations up until version 1.6.0 of the BIDS specification. Version 1.7.0 introduced additional concepts and datatypes which may be used to define a custom specification for pipeline outputs. Official rules for custom specifications include:

• Set the DatasetTypeattribute to derivative in the dataset description.
• Conform to the BIDS-like file hierarchy and naming convention (no file duplicates, single suffix, no duplicate entities, inheritance principle, etc..).
• Do not reuse entities defined in the BIDS rawdata specification.
• Prefix the derived data filename with the entities from the source rawdata.

Storage convention

A BIDS file (MAF or MSF) is composed of a suffix and an extension, and may prefixed by an ordered set of entities. The suffix identifies the nature of the recording and the extension its filetype.

Entities are key-value pairs in the form <prefix>-<value>, which are used to disambiguate recordings within the BIDS hierarchy. The presence of an entity with value X usually implies the existence of the same entity with a value other than X, apart in rare use cases documented in certain modalities. Entity values may be alphanumerical (called label) or strictly positive integers (called index). Some entities further restrict their valid labels to a set of values (like an enum) or on / off (like a flag).

For example:

• participants.tsv -> suffix: participants, extension: tsv -> tabular data for the subject metadata level
• sub-01_sessions.json -> suffix: sessions, extension: json, entities: sub-01 -> json sidecar for the session metadata level. Implies existence of other subjects within the original dataset.
• sub-01_run-01_T1w.nii.gz -> suffix: T1w, extension: nii.gz, entities: [sub-01, run-01] -> imaging data for a T1-weighted acquisition. Implies the existence of other subjects and multiple repeats of the same acquisition.

All MSF are associated with a datatype (for instance anat or pet) and a modality which may group several datatypes together (for instance mri and pet). Some MAFs may be associated with a datatype, such as phenotype. Those files are stored within a subfolder named after their datatype:

• phenotype/acds.tsv -> datatype: phenotype, suffix: acds, extension: tsv -> phenotypic tabular data for the ACDS assessment.
• dwi/[<source_entities>]_dwi.nii.gz -> datatype: dwi, suffix: dwi, extension: nii.gz -> imaging data for the DWI modality.

Inheritance principle

MSF metadata can be factored out and recomposed following the inheritance principle. Common metadata belonging to a specific acquisition may be defined anywhere in the metadata hierarchy (subject, session or scan) and redefined multiple times, provided redefinition occurs at most once per metadata level. Composition is done by aggregating attributes from top to bottom, overriding already defined values. Multiple (re)definitions of MSF metadata is not recommended but made possible by this principle.

In the most extreme case, the following files:

T1w.json                                      # Common to the whole dataset
sub-<label>/sub-<label>_T1w.json              # Common within this participant
sub-<label>/anat/sub-<label>_run-01_T1w.json  # Specific to this scan instance

can all co-exist.

[ghisvail]

Domain modeling

BIDS dataset

The main abstraction for querying a BIDS dataset. Successful instantiation of a Dataset class should represent a valid object in the BIDS sense. When opened from a path, it should deserialize dataset_description.json, extract its mandatory attributes (Name, BIDSVersion and DatasetType) and immediatly return an object. The path could be the root BIDS folder where dataset_description.json is stored or the direct path to the file.

# bids/dataset.py

class DatasetDescription:
    name: str
    bids_version: str
    dataset_type: "raw" | "derivative"


class Dataset:
    ...


def open(path: PathLike) -> Dataset | None:
    ...


dataset = open(path)

BIDS specification

This abstraction is required to handle both BIDS raw and derivative datasets. In the case of raw datasets, the BIDS version is enough to identify specifications supported by the dataset. For derivatives however, it needs to be specified manually.

# bids/specification.py

class Specification:
    ...


# bids/dataset.py

class Dataset:
    specification: Specification

    def with_specification(self, spec: Specification) -> Dataset:
        ...

spec = Specification(...)   # Define the BIDS specification for the derivative
dataset = open(path).with_specification(spec)

We could also use the proposed convention that the root BIDS path be named with a prefix identifying the derivative, like <pipeline>-<hash>. For instance, a path like /path/to/bids/derivative/t1linear-aabbccdd would be an instance (aabbccdd) of the derivative produced by the t1linear pipeline. Opening the latter would mean finding a specification named t1linear.

Regardless, since we would have to manage several variations of the BIDS specification, i.e. one per version and one per derivative at first, there is a need for a registry abstraction to which we would declare and query instances of BIDS specifications by version or derivative name.

# bids/specification_registry.py

def register(spec: Specification, version: str, derivative: str | None):
    ...

def get(version: str, derivative: str | None) -> Specification | None:
    ...

It will likely be done with a decorator applied to the specification definition. They are functionally equivalent.

BIDS template

A BIDS specification models the space of valid BIDS hierarchies and associated metadata. A BIDS template materialize one such hierarchy in this space, onto which search queries may be applied to compute valid BIDS paths for MAF or MSF collections.

# bids/template.py

class Template:
    def apply(query) -> Iterable[Path]:
        ...

# bids/specification.py

class Specification:
    def get_template(self) -> Template:
        ...

BIDS file collections

File collections model entity-linked data corresponding to a specific suffix. This would aggregate files sharing a common suffix, plus metadata gathered in the tree as per the inheritance rules. This is where the distinction of modality-agnostic and modality-specific models would happen.

# bids/file_collection.py


class ModalityAgnosticFileCollection:
    datatype: Optional[str] 


class ModalitySpecificFileCollection:
    modality: str
    datatype: str

BIDS paths

Models a path to a valid BIDS file within a file collection.

# bids/path.py

class Path:
    entities: Optional[List[Entity]]
    suffix: Suffix
    extension: Extension

   def __str__(self) -> str:
       # As [<entities>]_<suffix>.<extension>
        ...

BIDS components

Those are the low-level components composing a BIDS path, i.e. entities, suffixes and extensions.

An entity is modeled with a name, description, prefix and value type (label or index at first, maybe extended later to enum and flag if needed). Prefix must be alphanumerical.

A suffix is an alphanumerical value.

An extension is composed of alphanumerical values joined with dots.

bids/path.py

class Entity:
    name: str
    description: str
    prefix: str
    value_type: "label" | "index"
    value: str

class Suffix:
    name: str
    description: str
    value: str

class Extension
    name: str
    description: str
    value: str

There is likely a pattern to be factored out here, but I would have to progress further in the implementation to find the best way. So far, this is close to what's being modeled in the schema of the BIDS specification.

[NicolasGensollen]

Playing a little bit with the implementation of low level concepts above, I was wondering how we should handle data types in BIDS Paths.

More precisely, let's say I have a Path class defined as above and implementing a from_path constructor such that you can instantiate a Path object from its PathLike representation:

path = "sub-01/ses-M00/anat/sub-01_ses-M00_T1w.nii.gz"
# `from_path()` constructor takes a pathLike and returns a proper Path class.
p = Path.from_path(path)

Then, extension and suffix are easy:

assert p.extension.value == ".nii.gz"
assert p.suffix.value == "T1w"

For entities, if I understand correctly your proposal @ghisvail , we should get the following:

assert len(p.entities) == 2
assert p.entities[0].name == "subject"
assert p.entities[0].prefix == "sub"
assert p.entities[0].value_type == "label"
assert p.entities[0].value == "01"
assert p.entities[1].name == "session"
assert p.entities[1].prefix == "ses"
assert p.entities[1].value_type == "label"
assert p.entities[1].value == "M00"

Now, should we have

assert str(p) == path

Or

assert str(p) == "sub-01_ses-M00_T1w.nii.gz"

I'm pretty sure we want the former, but then we have to get the data type somehow (i.e. anat in this case).

ATM, I see two main opposite ways to do that:

• Have a general hardcoded mapping from the suffix to the data type (e.g T1w -> anat, bold -> func...)
• Add a data_type attribute to the Path class

but I was wondering if you thought of something different @ghisvail ?

[ghisvail]

We'll probably need a datatype attribute somewhere. It could be directly at the path level, or via some sort of association (datatype, suffix). I am still not sure which abstraction should hold this responsability yet.

I'd expect to get something like:

class Path:
    ...

    @property
    def name(self) -> str:
        # sub-01_ses-M00_T1w.nii.gz
        ...

    @property
    def relpath(self) -> str
        # Maybe if required?
        # sub-01/ses-M00/anat/sub-01_ses-M00_T1w.nii.gz
        ...
    
    @property
    def path(self) -> Path:
        # Path(<root>/sub-01/ses-M00/anat/sub-01_ses-M00_T1w.nii.gz)
        ...

[ghisvail]

But I am open to other suggestions. Other specifications may arise as implementation progresses :-)

[ghisvail]

Answering my own question, a suffix may not be sufficient to guess the datatype in all cases. For instance a mask suffix may be present under anat, func or dwi. Consequently, if one were to parse a BIDS compliant filename, multiple candidate results may be returned.

[NicolasGensollen]

Thanks for your fast reply! I agree that storing the data type at the Path level is probably the cleanest solution. I'll see how it goes when implementing.