May I ask if MTEB supports evaluating downloaded data？

[BtlWolf]

During the MTEB evaluation process, it usually downloads relevant task datasets, but I already have these datasets. Is there any way to specify the path

[Samoed]

MTEB downloads datasets using datasets, but there’s no way to specify a custom path for the datasets.

[isaac-chung]

Currently I don't believe there is a clear way. But we certainly welcome contributions!

The MTEB.run() method accepts kwargs, but I have not tried it this way.

mteb/mteb/evaluation/MTEB.py

Line 409 in 0753aba

kwargs: Additional arguments to be passed to `_run_eval` method and task.load_data.

For non-retrieval tasks, the AbsTask.load_data() uses all kwargs in the TaskMetadata's dataset dict. Right now, most tasks specifies "path" and "revision".

mteb/mteb/abstasks/AbsTask.py

Line 192 in 0753aba

self.dataset = datasets.load_dataset(**self.metadata_dict["dataset"]) # type: ignore

So perhaps, you could try either:

1. Install MTEB in edit mode and overwrite the dataset's path to point to the desired path, or
2. Install MTEB normally, create a new dataset, and inherit from the dataset class, and overwrite the dataset dict, e.g.

class NewDataset(AmazonPolarityClassification):
    metadata.dataset = {"path": YOUR_PATH}

Let us know if any of these work :)

[KennethEnevoldsen]

We could implement the following:

# one task
task = mteb.get_task("AmazonPolarityClassification", dataset_kwargs={...})

# more tasks at once
tasks = mteb.get_tasks(tasks = ["AmazonPolarityClassification"], dataset_kwargs={"AmazonPolarityClassification": {...}})

However, I am unsure if we want users to be able to overwrite kwargs.

[KennethEnevoldsen]

related to #1714