Introducing chaining rule for derivatives

[CPernet]

Your idea

As many are writing derivatives extensions, and I was working on that with @melanieganz and @mnoergaard, the idea came that we may need a chaining rule. For raw data, the location of entities is based on a priori rules. For derivatives, it would be based on how entities apply.

Example for this proposal

<pipeline_name>/
    tpl-<label>/
        [cohort-<label>/]
           [<datatype>/]
               tpl-<label>_<source_entities>[_cohort-<label>][_atlas-<label>][seg-<label>][_scale-<label>][_res-<label>][_den-<label>][_desc-<label>]_<suffix>.<extension>

In this work, [_res-][_den-] applies to tpl- and [_scale-] applies to atlas-. If we introduce a chain rule, then we get this representation:

<pipeline_name>/
    tpl-<label>/
        [cohort-<label>/]
           [<datatype>/]
               tpl-<label>_][_res-<label>][_den-<label>]<source_entities>[_cohort-<label>][_atlas-<label>][_scale-<label>[seg-<label>][_desc-<label>]_<suffix>.<extension>

Again, this is not specific to this example; it works for any derivative one builds with operators seen as entities. We have, of course, the 255-long character limits, so we do not want mega-long names that are not human-friendly, but it makes sense to have a few entities and we think the order helps to understand what the file represents.

[yarikoptic]

Unless order is explicitly specified (like now in the schema; or explicitly e.g. as for #1809) it would make it impossible to construct/match a filename given a set (unordered) of entities. IMHO that would be a considerable deviation from current BIDS behavior and thus would need a very careful consideration.

[CPernet]

Yeah I was more thinking of a recommendation rather than a validator enforced one. A reason is also that some entities might 'apply' to different othera depending on use sase and tracking there is just a nightmare. A recommendation like when we say use as little entities as needed would suffice, just to encourage human readability.

[yarikoptic]

sorry, I am not following how "recommendation" would work by breaking agreed upon order (since BIDS relies on having an order) .

[CPernet]

Right now for derivatives, it does not follow any order besides sub-X_suffix.ext, so I'm proposing we start by introducing the chaining rule, ie order entities as they apply to the data. I cannot break something that does not exist (no derivatives validation)

[effigies]

The schema validator does validate derivatives, and that is now the default web validator.

[yarikoptic]

Even if validator did not validate the order, AFAIK the order is mandated by the standard e.g. per https://bids-specification.readthedocs.io/en/stable/schema/index.html#ordering-rules for any BIDS dataset (raw, derivative, or any other), so it would be just a matter of extending validator with support for validating the order or changing BIDS specification to allow arbitrary order in some DatasetTypes.

[CPernet]

hey both for pointing to these -- despite the link I could not locate a rule file, I can see the script that loads entities, etc .. but the yml about chain? -- my point was, however, that when there is not an explicit rule, the chaining should be based on usage (for how one built the derivative file)

[CPernet]

also relates to #1895 with desc- as, or not, enforced last entity -- same topic really @arokem