Speed loss when adding categories

[jagrundy]

Hi!

In our ATLAS analysis, we have noticed a significance speed difference between our pyhf limit setting framework and another independent framework, and wanted your thoughts/advice. I'm a member of the HH → 4b analysis and one of results is a limit on HH production cross-section at various κλ (the Higgs self-coupling modifier) points. We have a number of categories in the analysis, the addition of which has slowed our pyhf-based limit setting framework down significantly. Put simply, we use a root finder and pyhf.infer.hypotest to find the μ corresponding to CLs = 0.05. For example, for κλ = 1, the time taken to get an expected limit on μ with error bands is ~25 hours 39 minutes. An alternative framework used by the HH combination team, quickstats, which uses ROOT, extracted the same limit in ~79 seconds. We found this difference very surprising! Please let us know your thoughts and if you want any more information.

Cheers!
James

[alexander-held]

Hi James, there are a number of things that may have an impact here. As a first step, I would suggest comparing the runtime of a single maximum-likelihood estimate (MLE) fit. A few more thoughts:

• pyhf does not prune gammas (modifiers for MC statistical uncertainty): Pruning of gammas #662. If you have many bins, and apply pruning, that may significantly reduce the number of parameters in the ROOT setup.
• I don't know the quickstats limit-finding implementation, but there are some scripts used in ATLAS that are extremely efficient at ROOT finding by using features of the problem. How may fits do you need for convergence in the various setups? What threshold are you using to determine convergence? Usually the POI resolution does not need to be very fine, allowing to potentially stop early.
• Have you tried different pyhf backends? Which minimizer are you using? That may significantly change the runtime.

When you say "categories", do you mean channels in pyhf (= phase space regions)? That may point to the gamma point above.

If you are interested in another datapoint, you may also want to check out the implementation in cabinetry: code / docs. This uses a few things to speed up the procedure, but is typically still surpassed by optimized implementations (some more tricks need to be ported to this still).

[alexander-held]

No, this is currently not technically possible. See #662.

[jagrundy]

Okay, fair enough. I'm just dropping all stat errors from the pyhf fit to see if this causes a big speed up. If so, then this difference is likely from the pruning, as you say.

[jagrundy]

Actually, I've dropped all stat errors from the workspace, but the speed doesn't seem to have improved significantly.

[lukasheinrich]

interesting, can you pass us the workspace privately .. we can do some more checks

[jagrundy]

Cool, thank you. I can email it over. Who should I address it to?

[jagrundy]

Just for some more information about the workspaces, here's the summary of pyhf inspect:

Summary
Channels	38
Samples	114
Parameters	305
Modifiers	305

[jagrundy]

Thank you for the replies. I'm now looking at setting up a comparison of the single MLE fit. To answer the questions in the meantime - we're using jax right now but have tested others in the past and haven't seen a significant difference. Regarding the number of samples, in each channel we have two signal models and background so 38 channels x (2 sig + 1 bkg) = 114 samples.

[lukasheinrich]

Do you allocate a different sample per channel? This might be a clue. Pyhf wouldn't optimise that

[jagrundy]

Yes, we did. I can look at changing that now. Will let you know the result.

[jagrundy]

I've reduced the number of samples to three but there doesn't seem to be any appreciable speed up from an initial test.

[AlkaidCheng]

Hi, the default settings for quickstats is roughly (in ROOT language):
-minimizer strategy = 1
-minimizer type = Minuit2
-minimizer algorithm = Migrad
-minimizer tolerance = 1.
-constant term optimization + Cache-and-track optimization = True

From my test, the entire limit setting takes 27 seconds to finish for the 4b workspace mentioned.

[kratsg]

Hi, there's been no update. @AlkaidCheng is this still an open discussion? Re-open/comment if so.