New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add compositionality benchmarks #97

Open

mehdidc opened this issue Jun 30, 2023 · 8 comments

Labels

Collaborator

mehdidc commented Jun 30, 2023 •

edited

Loading

HarmanDotpy commented Jul 6, 2023

listing some others:
SugarCREPE: https://github.com/RAIVNLab/sugar-crepe/tree/main
VL-Checklist: https://github.com/om-ai-lab/VL-CheckList
Winoground: https://huggingface.co/datasets/facebook/winoground
maybe SVO (verb understanding): https://github.com/deepmind/svo_probes
maybe VALSE: https://github.com/Heidelberg-NLP/VALSE

Collaborator Author

mehdidc commented Jul 7, 2023

Thank you very much @HarmanDotpy !

mehdidc added the new datasets label

mehdidc mentioned this issue

Support sugar crepe for compositionality evaluation #102

Merged

Collaborator Author

mehdidc commented Jul 11, 2023

Ok SugarCREPE is there, will look at the rest

Collaborator Author

mehdidc commented Dec 1, 2023

Another one to consider: https://github.com/arijitray1993/COLA, https://cs-people.bu.edu/array/research/cola/

Collaborator Author

mehdidc commented Jan 7, 2024 •

edited

Loading

Also:

https://arxiv.org/abs/2310.19785 (What's "up" with vision-language models? Investigating their struggle with spatial reasoning)
https://arxiv.org/abs/2310.08577 (Visual Data-Type Understanding does not emerge from Scaling Vision-Language Models)
https://github.com/yonatanbitton/wysiwyr (What You See is What You Read? Improving Text-Image Alignment Evaluation)

mehdidc mentioned this issue

Support winoground #116

Merged

vishaal27 commented Jan 15, 2024 •

edited

Loading

Add MMVP-VLM (winoground-style dataset): https://github.com/tsb0601/MMVP?tab=readme-ov-file
https://huggingface.co/datasets/MMVP/MMVP_VLM/

vishaal27 commented Feb 8, 2024

Add https://github.com/Top34051/colorswap

escorciav commented Oct 30, 2024 •

edited

Loading

@mehdidc sugarcrepe is there & runs, but there is an issue with the run all functionality. Refer to #128

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment