Is it possible to give an empty score to a value in the leaderboard?

[gorkamunoz]

We are creating a competition in which we will have various subtasks. Participants do not have to participate in all subtasks. Is there a way of giving a non-numeric value to a given value in the leaderboard? For example, if the participant only participates in Task1, but not Task2, the column of Task1 would have a score but the column of Task2 something like ' - '. Then the rank with respect to Task2 would put this submission in last position.

Thanks for the help!

[Didayolo]

Hi,

I am not sure but I think by default if the scoring program returns nothing, it puts a - into the leaderboard instead of a numeric value. You can also choose a value for no-score (like -1).

Then, if you don't know, you can combine the results from various task with something like this in competition.yaml file:

leaderboard:
  columns:
    auc_binary:
      label: Classification (AUC ROC)
      leaderboard: &id001
        label: RESULTS
        rank: 1
      rank: 2
      sort: desc
    auc_binary_2:
      label: Feature Selection (AUC ROC)
      leaderboard: *id001
      rank: 3
      sort: desc
    ave_score:
      label: < Rank >
      leaderboard: *id001
      rank: 1
      sort: asc
      computed:
        operation: Avg
        fields: auc_binary, auc_binary_2
  leaderboards:
    RESULTS: *id001

In this case we rank participants with the average rank among the two tasks.

[gorkamunoz]

Hi Adrien, thanks for the response. After some tests, I see that if you have various scores, and you give nothing for one of them, the scoring program is not able to compute all the rest of the scores and puts (-)- in the first rank and nothing in the others. We would like to have a - in the tasks where the submission has not participated but numerical values in the rest.

For example, my competition.yaml will have something like:

columns:
        correct:
            leaderboard: *RESULTS
            label: correct
            rank: 1
        other:
            leaderboard: *RESULTS
            label: other
            rank: 2

If in scores.txt there is only, e.g. correct = 1 or even other = 1, the leaderboard will look like:

If I put anything else, e.g. correct: 1 other: False, correct: 1 other: - or correct: 1 other: the submissions fail.

On another topic, now that you comment on the avg, is it possible to define a different average over the ranks? We are interested in using the mean reciprocal rank, which is the mean of the inverse of the ranks. Is it possible to define such average? Where is the operation Avg defined?

Thanks again for your help!

[Didayolo]

Hi Gorka,

Sorry for the late reply.

We would like to have a - in the tasks where the submission has not participated but numerical values in the rest.

I'm not sure how to do that. I remember having coded a multi-task scoring program which doesn't fail when a task is missing. But it returns a specific value for the missing task. Here is the code if it ever helps.

Is it possible to define a different average over the ranks?

This is a very good question. I never used something else.
But quickly searching into the code, it seems that there is only 'Max' and 'Average':

class SubmissionComputedScore(models.Model):
    scoredef = models.OneToOneField(SubmissionScoreDef, related_name='computed_score')
    operation = models.CharField(max_length=10, choices=(('Max', 'Max'),
                                                        ('Avg', 'Average')))

You can do a pull request to add new operations if you wish.

[Franck-Dernoncourt]

+1 for having some way to indicate - or something that isn't a value in the column of some task (e.g., N/A). Using -1 confuses participants, e.g. see https://groups.google.com/d/msg/semeval-2020-task-6-all/ATqDbf8yoYQ/Sg0xaAF4CAAJ

[Didayolo]

For future: scoring programs should be able to return None for failures.

[Didayolo]

Merged.

[cmanzo]

On another topic, now that you comment on the avg, is it possible to define a different average over the ranks? We are interested in using the mean reciprocal rank, which is the mean of the inverse of the ranks. Is it possible to define such average? Where is the operation Avg defined?

Please check this PR #3451

[Didayolo]

By the way, on Codabench I think leaderboard can have N/A scores.