Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speeding up mean_iou metric computation #569

Merged
merged 2 commits into from
Apr 18, 2024
Merged

Speeding up mean_iou metric computation #569

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c0bacad
Change Features from Sequence to Image
qubvel Apr 8, 2024
6da1cd8
Fix example
qubvel Apr 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions metrics/mean_iou/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,13 +27,13 @@ For binary (two classes) or multi-class segmentation, the *mean IoU* of the imag

## How to Use

The Mean IoU metric takes two numeric arrays as input corresponding to the predicted and ground truth segmentations:
The Mean IoU metric takes two lists of numeric 2D arrays as input corresponding to the predicted and ground truth segmentations:
```python
>>> import numpy as np
>>> mean_iou = evaluate.load("mean_iou")
>>> predicted = np.array([[2, 2, 3], [8, 2, 4], [3, 255, 2]])
>>> ground_truth = np.array([[1, 2, 2], [8, 2, 1], [3, 255, 1]])
>>> results = mean_iou.compute(predictions=predicted, references=ground_truth, num_labels=10, ignore_index=255)
>>> results = mean_iou.compute(predictions=[predicted], references=[ground_truth], num_labels=10, ignore_index=255)
Copy link
Contributor

@NielsRogge NielsRogge Apr 8, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this be a breaking change? Cause currently the metric works when passing 2D arrays for predicted and ground_truth, but this PR would require it to be a list of 2D arrays?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like it doesn't work with 2D arrays originally, only with a list of them.

I tested it locally and got the following error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/piakubovskii/Projects/evaluate/.venv/lib/python3.10/site-packages/evaluate/module.py", line 450, in compute
    self.add_batch(**inputs)
  File "/Users/piakubovskii/Projects/evaluate/.venv/lib/python3.10/site-packages/evaluate/module.py", line 541, in add_batch
    raise ValueError(error_msg) from None
ValueError: Predictions and/or references don't match the expected format.
Expected format: {'predictions': Sequence(feature=Sequence(feature=Value(dtype='uint16', id=None), length=-1, id=None), length=-1, id=None), 'references': Sequence(feature=Sequence(feature=Value(dtype='uint16', id=None), length=-1, id=None), length=-1, id=None)},
Input predictions: [[  2   2   3]
 [  8   2   4]
 [  3 255   2]],
Input references: [[  1   2   2]
 [  8   2   1]
 [  3 255   1]]

There is also an issue opened 3 weeks ago with the same error:
#563

Maybe backward compatibility have been broken earlier

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, cc'ing @lhoestq here, I assume we can safely merge it in that case

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

indeed metrics .compute() take lists of references and predictions per image, not just a pair of reference and prediction

```

### Inputs
Expand Down
5 changes: 2 additions & 3 deletions metrics/mean_iou/mean_iou.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -281,10 +281,9 @@ def _info(self):
citation=_CITATION,
inputs_description=_KWARGS_DESCRIPTION,
features=datasets.Features(
# 1st Seq - height dim, 2nd - width dim
{
"predictions": datasets.Sequence(datasets.Sequence(datasets.Value("uint16"))),
"references": datasets.Sequence(datasets.Sequence(datasets.Value("uint16"))),
"predictions": datasets.Image(),
"references": datasets.Image(),
}
),
reference_urls=[
Expand Down
Loading