Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mean reciprocal rank (MRR) of leaderboard scores #3451

Merged
merged 1 commit into from
Dec 8, 2023

Conversation

cmanzo
Copy link
Contributor

@cmanzo cmanzo commented Nov 30, 2023

@ mention of reviewers

@Didayolo

Purpose of the changes contained in this PR.

For a new competition that we will launch soon on CodaLab, we would like to use the mean reciprocal rank (MRR) as a summary metrics for the overall evaluation of a group of scores in the leaderboard. However, at the moment, Codalab only allows using the (weighted) average operation over the rank of scores.
This is mentioned in some comments on #2736 and #3449.

Issues this PR resolves

This PR includes the operation to calculate the MRR over leaderboard scores.

A checklist for hand testing

  • create a competition bundle with a column of the leaderboard performing the MRR of some metrics. You can use the modified IRIS competition bundle.
  • upload a few submissions to the competition.
  • check that the MRR column provides the correct calculation and ranking, similar as in the screenshot below.
    Screenshot 2023-11-30 at 10 25 25

Any relevant files for testing

I have created a modified IRIS competition bundle that has, as leaderboard columns, both the MRR and the Avg of the original metrics (Prediction Score and Duration).

Checklist

  • Code review by me and @JesusPinedaC
  • Hand tested by me
  • I'm proud of my work (code changes are very minor though!)
  • Code review by reviewer
  • Hand tested by reviewer
  • CircleCi tests are passing
  • Ready to merge

@Didayolo
Copy link
Member

Didayolo commented Dec 7, 2023

I first merge this into codalab:mrr in order to trigger the automatic testing.

@codecov-commenter
Copy link

Codecov Report

Attention: 13 lines in your changes are missing coverage. Please review.

Comparison is base (927279f) 60.45% compared to head (f06e051) 60.38%.

Files Patch % Lines
codalab/apps/web/models.py 23.52% 13 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##              mrr    #3451      +/-   ##
==========================================
- Coverage   60.45%   60.38%   -0.07%     
==========================================
  Files         173      173              
  Lines       11396    11411      +15     
==========================================
+ Hits         6889     6891       +2     
- Misses       4507     4520      +13     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@Didayolo Didayolo merged commit 3c28c2f into codalab:mrr Dec 8, 2023
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants