New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug/challenger comparisons #45

Merged

thorrester merged 3 commits into main from bug/challenger-comparisons

Aug 28, 2024

Member

thorrester commented Aug 27, 2024

Pull Request

Short Summary

The ModelChallenger does not currently handle failures well if a model, metric or run is missing. This PR adds better handling logic.

New Challenger Flow:

Scenario: No champion provided

In the case when no champion is provided, ModelChallenger will attempt to load the latest model
- If no model is found --> challenger wins
- If models are found, ModelChallenger will return the first version that is not the challenger version and has a runcard
- if a model is found but there is no associated runcard and metrics --> challenger wins
- If a model and runcard exists, but the metric does not --> challenger wins
- If model, runcard and metric exists --> run battle report

Scenario: List of champions provided

If a champion model does not exist --> challenger wins
If a champion does not have a runcard --> challenger wins
if metric is missing --> challenger wins
If model, runcard and metric exist --> run battle report

thorrester added 3 commits

August 27, 2024 06:59


          Update _hw_metrics.py

4c89990


          challenger error handing

64aaeff


          error handling logic

2a3ab7e

github-actions bot commented Aug 27, 2024

Dependency Review

✅ No vulnerabilities or license issues found.

Scanned Manifest Files

thorrester marked this pull request as ready for review

August 27, 2024 21:49

thorrester merged commit f537388 into main

14 checks passed

thorrester deleted the bug/challenger-comparisons branch

August 28, 2024 10:42

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet