Precision and recall metrics doesn't work with scikit-learn==1.6.0

[mgrddsj]

Description

The metric "precision" and "recall" doesn't work with the latest scikit-learn==1.6.0, which may have API changes.

At the time of writing this issue, the API changelog of scikit-learn hasn't updated, but it seems like precision_score() from sklearn.metrics has changed to returning float instead of its original type.

Other metrics may have similar issues, these two are what I've encountered.

Temporary Fix

Downgrade scikit-learn to 1.5.2.

pip install scikit-learn==1.5.2

Steps to Reproduce

1. Install evaluate with pip (which will automatically install the latest scikit-learn==1.6.0
2. Run the following code

import evaluate
precision = evaluate.load("precision")
precision.compute(predictions=[0, 1], references=[0, 1])

Error Message

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[24], line 1
----> 1 precision.compute(predictions=[0, 1], references=[0, 1])

File /opt/conda/lib/python3.10/site-packages/evaluate/module.py:467, in EvaluationModule.compute(self, predictions, references, **kwargs)
    465 inputs = {input_name: self.data[input_name] for input_name in self._feature_names()}
    466 with temp_seed(self.seed):
--> 467     output = self._compute(**inputs, **compute_kwargs)
    469 if self.buf_writer is not None:
    470     self.buf_writer = None

File ~/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--precision/4e7f439a346715f68500ce6f2be82bf3272abd3f20bdafd203a2c4f85b61dd5f/precision.py:145, in Precision._compute(self, predictions, references, labels, pos_label, average, sample_weight, zero_division)
    126 def _compute(
    127     self,
    128     predictions,
   (...)
    134     zero_division="warn",
    135 ):
    136     score = precision_score(
    137         references,
    138         predictions,
   (...)
    143         zero_division=zero_division,
    144     )
--> 145     return {"precision": float(score) if score.size == 1 else score}

AttributeError: 'float' object has no attribute 'size'

[Cozmo25]

The same issue exists with f1_score and same workaround fixes issue

[rbelew]

@Cozmo25 could you tell me how one can patch this dependency locally, until a new version of the evaluate is available? i have v0.4.3 and have this issue. I see the change that has made here: https://github.com/ScandEval/ScandEval/pyproject.toml but don't know how/if this percolates into the evaluate module?

[Cozmo25]

@rbelew you should be able to pip install -U scikit-learn==1.5.2 in your virtual environment to downgrade the version after all other dependencies are installed

[Maxwell-Jia]

I've submitted a PR (#656) to fix this issue.

The problem occurs because scikit-learn 1.6.0 sometimes returns float values instead of numpy arrays for single-value results. The PR modifies the return statement to handle both cases safely.

Temporary solutions until the PR is merged:

1. Downgrade scikit-learn to 1.5.2 as mentioned above
2. Or use the fixed version from my fork: evaluate

The PR should fix this issue for precision, recall, and f1 metrics while maintaining compatibility with both scikit-learn 1.6.0 and earlier versions.