Allow filtering of tag-lists

[danielhuppmann]

Working with ever-longer lists of variables in the common-definitions repo, I'm wondering how to more efficiently structure the creation of combinations.

One idea is to add filters, e.g.

- Secondary Energy|{Secondary Fuel Level 3|ccs true}:
     description: Production of {Secondary Fuel Level 3}
     ...

which would only take the elements from the tag-list that have an attribute ccs: True. Alternative simple filters would be tier- or level-attributes.

An alternative solution would be to add a dedicated filter-attribute, like

- Secondary Energy|{Secondary Fuel Level 3}:
     tag-filter:
       - Secondary Fuel Level 3:
            ccs: True
     description: Production of {Secondary Fuel Level 3}
     ...

Any thoughts @phackstock @dc-almeida?

[phackstock]

Puh, that's a tricky one, option 1 is certainly more efficient but option 2 is more explicit and covers more complex filters.
What kind of filter operations would you imagine @danielhuppmann? "Simple" attribute equality matching or more complex logic like "use the tag only if the value for given tag is in ['something', 'something else*', ...]".
In any case, implementation wise option 2 should be a lot easier since option 1 would require string parsing and variable type conversion. In your above case we'd need to cast "true" to a boolean which is something that we could only do by trying. In the case of integer values such as 1, we might then run into the issue that the result of the parsing would depend on the order of type conversions that we try. If we try casting to bool first, 1 would become True if we tried casting to int, 1 would come 1 and if we tried casting to float 1 would become 1.0.
Having written the above, I'd prefer option 2.

[danielhuppmann]

Yeah, probably option 2 makes more sense.

To give some context, look at this example: https://github.com/IAMconsortium/common-definitions/blob/main/definitions/variable/energy/tag_secondary_energy_carriers_level-3.yaml

There are at least three dimensions where the filter could be useful:

• A level-attribute allowing us to remove the three separate level-yaml-files
• A CCS flag whether this fuel/technology can be combined with CCS, so that "with/without CCS" sub-category variables can easily be created
• An emission-mapping to specify which species can be a result of a particular technology (e.g., burning hydrogen does not produce CO2 but other air pollutants)