Custom modifiers: feedback and suggestions

[alexander-held]

I gave the custom modifier implementation #1991 another try and have a few questions / suggestions I would like to provide here. I figured it might be useful to do that in a discussion where threads are possible and I did not want to clutter discussion in the PR.

How do I define a parameter that enters multiple different expressions?

Take this example:

import pyhf
from pyhf.experimental.modifiers import add_custom_modifier

new_params = {
    "x": {"inits": (1.0,), "bounds": ((-5.0,5.0),)},
}
expanded_pyhf = add_custom_modifier("custom", ["x"], new_params)

model = pyhf.Model(
{
    "channels": [
        {
            "name": "SR",
            "samples": [
                {
                    "name": "signal_quadratic",
                    "data": [10, 8],
                    "modifiers": [{"name": "f1", "type": "custom", "data": {"expr": "2*x**2"}}],
                },
#                 {
#                     "name": "signal_linear",
#                     "data": [0.1, 0.1],
#                     "modifiers": [{"name": "f2", "type": "custom", "data": {"expr": "x"}}]
#                 }
            ],
        }
    ]
},
    modifier_set=expanded_pyhf,
    poi_name='',
    validate=False,
    batch_size = None
)

print(model.expected_data([-1.0]))

This prints expected yields of [20. 16.], which makes sense given the scaling by $2\cdot (-1)^2$.

If you now uncomment the signal_linear part, the result is [-9.8 -7.8]. What I want here to happen is a result of [20, 16] + (-1)*[0.1, 0.1] = [19.9, 15.9]. I am not sure if this is a bug in the implementation or me not using the feature correctly. Such setups should be fairly common in EFT scenarios.

Understanding the implementation conceptually / potential streamlining

To me, add_custom_modifier() passed into modifier_set declares that there is a parameter with some name, and that parameter should be a free parameter in the fit. I can then go on and use this parameter in expressions. It’s not obvious to me why the type of such expression actually need to match what I initially defined, as opposed to just being a generic custom type that will always just work. Is there a need to couple them tightly? What I want is any expression to just be able to use any parameter ideally, but I understand I somewhere need to declare “free” parameters.

Here is the way I think about this (the following is a proposed interface, not something that works right now):

extra_parameters = {"x": {"inits": (1.0), "bounds": ...}, "y": {...}}
pyhf.Model(
    {
        "channels": [
            {
                "samples": [
                    {
                        "name": "signal_1",
                        "data": [1, 2],
                        "modifiers": [{"name": "f1", "type": "custom_mod", "data": {"expr": "x**2 + y"}}]
                    },
                    {
                        "name": "signal_2",
                        "data": [5, 3],
                        "modifiers": [{"name": "f2", "type": "custom_mod", "data": {"expr": "y"}}]
                    }
                ]
            }
        ]
    },
    extra_parameters = extra_parameters
)

You would declare which extra parameters you need, and then freely use them in any expressions for modifiers of a type matching some reserved keyword like "custom_mod".

This would strip out a lot of things that from a user perspective aren’t immediately useful and a bit harder to remember to write down. It’s also not clear to me how the current interface essentially has you define a custom modifier with a name you pick, but then what that modifier does still depends on the expression. That expression is still defined per entry of the modifier acting on a sample, so I’m not sure what that custom modifier actually means or why I’d need one with a second unique name.

Put differently, the current implementation feels like it mixes concepts of parameters and modifiers a bit: modifiers operate on samples and are controlled by parameters. Parameters are just degrees of freedom in the model (or expressions thereof, to me an expression would still feel more parameter-like than a modifier).

[matthewfeickert]

Thanks very much @alexander-held. This is great feedback to have. I'll be offline for next week, but I hope that we can iterate on all of this in the very near future.

[amartone21]

Hi,

I'm writing here because I wanted to try the custom modifier solution but I cannot import the add_custom_modifier function because: 'No module named 'pyhf.experimental''. I am using pyhf version 0.7.4.

[alexander-held]

Hi, this API has not made it to any release yet, it is sitting on the branch feat/addCustomModifierPython via the PR #1991. You need to install pyhf from this branch in order to use this functionality at the moment.

[amartone21]

After playing around with it, I also have the same questions as Alexander.
I am working on the HWW off-shell analysis and for this reason I need to implement something like N= (mu - sqrt(mu)S + (1-sqrt(mu))V + sqrt(mu)SVI + B.
I tried the following but I don't know if that may work:

new_params = {
    'm1': {'inits': (1.0,), 'bounds': ((0,10.0),)},
    'm2': {'inits': (1.0,), 'bounds': ((1.0,1.0),)},  
                                                        
}

expanded_pyhf = add_custom_modifier('customfunc', ['m1', 'm2'], new_params)
SBI_model = pyhf.Model(
    {
        'channels': [
            {
                'name': 'singlechannel',
                'samples': [
                    {
                        'name': 'signal', 
                        'data': hists['sig'],
                        'modifiers': [
                            {
                                'name': 'mu',
                                'type': 'customfunc',
                                'data': {'expr': 'm1 - sqrt(m1)'}}, 
                        ],
                    },
                    {
                        'name': 'Interfering background', 
                        'data': hists["IntB"], 
                        'modifiers': [
                        {
                            'name': 'f_V',
                            'type': 'customfunc',
                            'data': {'expr': 'm2 -sqrt(m1)'}},  
                            
                    ]},
                    {
                        'name': 'Non Interfering background', 
                        'data': hists["NonIntB"], 
                        'modifiers': [
                        {
                            'name': 'f_b',
                            'type': 'customfunc',
                            'data': {'expr': 'm2'}},  
                            
                    ]},
                    {
                        'name': 'SBI',
                        'data': hists['sbi'],
                        'modifiers':[
                            {
                                'name': 'f_int',
                                'type': 'customfunc',
                                'data': {'expr': 'sqrt(m1)'}}, 
                    ]},
                ],
            }
        ]
    },
    modifier_set=expanded_pyhf,
    poi_name='m1',
    validate=False,
    batch_size = 1
)

I am basically using the m2 parameters as 1 since - when writing 1 instead of a parameter - I am not able to build the model.
By doing so, however, I am introducing some kind of correlation between B and V and that would not be ideal.
I think this is due to the fact that I still confuse parameters and modifiers so I also would like to understand them better.