pyhf 2019 into 2020 Roadmap

[matthewfeickert]

pyhf 2019 into 2020 Roadmap

Overview and Goals

We will follow loosely Seibert’s Heirarchy of Needs

(Stan Seibert, SciPy 2019)

As a general overview that will include:

• Improvements to docs
  • Add lots of examples
  • Add at least 5 well documented case studies
• Issue cleanup
• Adding core feature support
• "pyhf evolution": integration with columnar data analysis systems
• GPU support and testing
• Publications
  • Submit pyhf to JOSS
  • Submit pyhf to pyOpenSci
  • Start pyhf paper in 2020
• Align with IRIS-HEP Analysis Systems NSF milestones

Time scale

The roadmap will be executed over mostly Quarter 3 of 2019 through Quarter 1 of 2020, with some projects continuing into Quarter 2 of 2020

• 2019-Q3
• 2019-Q4
• 2020-Q1
• (2020-Q2)

Roadmap

1. Documentation and Deployment
   • Add docstrings to all functions and classes (Issues Add docstrings #38, Add docstrings to all of the public pyhf.utils API #349) [2019-Q3]
   • Greatly revise and expand examples (Issues Test all the Jupyter notebooks in the CI #168, Add notebook demoing pyhf xml2json #202, Add documentation on cli commands (including piping) #212, Example Notebooks Have Bugs And Dont Run #325, Add learn notebook describing the interpolators (code2 and code4) #342, Add docstrings to all of the public pyhf.utils API #349, plan sprint for adding function docs #367) [2019-Q3 → 2019-Q4]
     • Add small case studies with published sbottom likelihood from HEPData
   • Move to scikit-hep GitHub organization [2019-Q3]
   • Develop a release schedule/criteria [2019-Q4]
   • Automate deployment with Azure pipeline (talk with Henry Schreiner) (Issue Investigate Using Azure Pipelines for CI #517) [2019-Q3]
   • Finalize logo and add it to website (Issue logo #453) [2019-Q3 → 2019-Q4]
   • Write submission to JOSS (Issue Write JOSS Submission #502) and write submission to pyOpenSci [2019-Q4 → 2020-Q2]
   • Contribute to IRIS-HEP Analysis Systems Milestones "Initial roadmap for ecosystem coherency" and "Initial roadmap for high-level cyberinfrastructure components of analysis system" [2019-Q4 → 2020-Q2]
2. Revision and Maintenance
   • Add tests using HEPData published sbottom likelihoods (Issue add public sbottom likelihood as test #518) [2019-Q3]
   • Add CI with GitHub Actions and Azure Pipelines (PR Add CI through GitHub Actions #527, Issue Investigate Using Azure Pipelines for CI #517) [2019-Q3]
   • Investigate rewrite of pytest fixtures to use modern pytest (Issue Fix Fixture use in pytest #370) [2019-Q3 → 2019-Q4]
   • Factorize out the statistical fitting portion into pyhf.infer (PR refactor: Create infer subpackage to hold our statistics code #531) [2019-Q3 → 2019-Q4]
   • Bug squashing at large [2019-Q3 → 2020-Q2]
     • Unexpected use cases (Issues Confusing doctest behavior from modifiers.modifier #324, Example Notebooks Have Bugs And Dont Run #325, single bin fit with s = 0.0, b = 1.9, deltab = 0.4 gives NaN #529)
     • Computational edge cases (Issues tensorflow hessians calculation produces nans #332, fix handling of non-physical data when calculating staterrors #445)
   • Make sure that all backends reproduce sbottom results [2019-Q4 → 2020-Q2]
3. Development
   • Batch support (PR [WIP] batched evaluation (for parameters) #503) [2019-Q3]
   • Add ParamViewer support (PR Add ParamViewer class  #519) [2019-Q3]
   • Add setting of NPs constant/fixed [2019-Q3] (PR feat: expand API to enable pull/impact plots #653)
   • Implement pdf as subclass of distributions (PR Add probability module #551) [2019-Q3]
   • Add sampling with toys (PR feat: Add toy sampling #558) [2019-Q3]
   • Make general modeling choices (e.g., Issue Should Poisson[0,0] be handled especially? #293) [2019-Q4 → 2020-Q1]
   • Add "discovery" test stats (p0) (PR feat: Add discovery p0 test statistic #520) [2019-Q4 → 2020-Q1]
   • Add better Model creation [2019-Q4 → 2020-Q1]
   • Add background model support (Issue background-only workspaces cannot be built as models #514) [2019-Q4 → 2020-Q1]
   • Develop interface for the optimizers similar to tensor/backend [2019-Q4 → 2020-Q1]
   • Migrate to TensorFlow v2.0 (PR feat: Migrate to TensorFlow v2.0 #541) [2019-Q4]
   • Drop Python 2.7 support at end of 2019 (Issue Drop Python 2.7 support #469) [2019-Q4 (last week of December 2019)]
   • Finalize public API [2020-Q1]
   • Integrate Statisfactory API [2020-Q1]
4. Research
   • Add Statisfactory integrations (Issue Possible to integrate CMS's Combine workflow? #344, zfit Issue 120) [2019-Q4]
   • Hardware acceleration scaling studies (Issues Acquire access to GPU cluster for testing #93, Investigate TPU performance #301) [2019-Q4 → 2020-Q1]
   • Speedup through Numba (Issue Investigate possible speedups through Numba #364) [2019-Q3 → 2019-Q4]
   • Dask backend (Issue Research: Dask integration #259) [2019-Q3 → 2020-Q1]
   • Attempt to use pyhf as fitting tool for full Analysis Systems pipeline test in early 2020 [2019-Q4 → 2020-Q1]
   • pyhf should satisfy IRIS-HEP Analysis Systems Milestone "GPU/accelerator-based implementation of statistical and other appropriate components" [2020-Q1 → 2020-Q2] and contributes to "Benchmarking and assessment of prototype analysis system components" [2020-Q3 → 2020-Q4]

Roadmap as Gantt Chart

Presentations During Roadmap Timeline

• Talk at IRIS-HEP Institute Retreat (September 12-13th, 2019)
• Talk at PyHEP 2019 (October 16-18th, 2019)
• Talk at CHEP 2019 (November 4-8th, 2019)
• Poster at CHEP 2019 (November 4-8th, 2019)

[kratsg]

We need to update roadmap in pyhf

[matthewfeickert]

We need to update roadmap in pyhf

The ROADMAP.md should get updated in PRs that affect it and then those results translated here. What needs to get updated? Or are you saying that we need a review and revision of goals and timelines?

[kratsg]

https://github.com/scikit-hep/pyhf/blob/master/docs/governance/ROADMAP.md

• " Add setting of NPs constant/fixed [2019-Q3]" was already done. I updated it in this issue...