Add usage instructions and new audio-to-text hard task configuration

EvolvingLMMs-Lab · Dec 5, 2024 · f580d13 · f580d13
1 parent fb56c5c
commit f580d13
Show file tree

Hide file tree

Showing 2 changed files with 28 additions and 0 deletions.
diff --git a/lmms_eval/tasks/mix_evals/README.md b/lmms_eval/tasks/mix_evals/README.md
@@ -2,6 +2,31 @@
 
 [Homepage](https://mixeval-x.github.io/) / [arXiv](https://arxiv.org/abs/2410.13754)
 
+## Usage
+
+Tasks:
+
+- `mix_evals_image2text`
+    - `mix_evals_image2text_freeform`
+    - `mix_evals_image2text_mc`
+- `mix_evals_image2text_hard`
+    - `mix_evals_image2text_freeform_hard`
+    - `mix_evals_image2text_mc_hard`
+- `mix_evals_video2text`
+    - `mix_evals_video2text_freeform`
+    - `mix_evals_video2text_mc`
+- `mix_evals_video2text_hard`
+    - `mix_evals_video2text_freeform_hard`
+    - `mix_evals_video2text_mc_hard`
+- `mix_evals_audio2text`
+    - `mix_evals_audio2text_freeform`
+- `mix_evals_audio2text_hard`
+    - `mix_evals_audio2text_freeform_hard`
+
+```bash
+lmms-eval --model=llava_vid --model_args=pretrained=lmms-lab/LLaVA-NeXT-Video-7B --tasks=mix_evals_video2text --batch_size=1 --log_samples --output_path=./logs/
+```
+
 ## Citation
 
 ```bib

diff --git a/lmms_eval/tasks/mix_evals/audio2text/mix_evals_audio2text_hard.yaml b/lmms_eval/tasks/mix_evals/audio2text/mix_evals_audio2text_hard.yaml
@@ -0,0 +1,3 @@
+group: mix_evals_audio2text_hard
+task:
+- mix_evals_audio2_text_freeform_hard