Skip to content

Issues: open-compass/VLMEvalKit

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

42 Open 190 Closed
42 Open 190 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Discrepancy in MM Math Evaluation Results: Possible Issue with Answer Extraction in VLMEvalkit
#638 opened Dec 1, 2024 by mantle2048
2
Invalid url of English and Chinese split of OlympiadBench
#634 opened Nov 30, 2024 by aiiph4
执行MVBench评测时，发现两个问题，修改已放在评论中的附件里
#633 opened Nov 29, 2024 by Xiaoqi-Zhou916
@FangXinyu-0913
1
The evaluation device_map setting is not optimized for quantized models using the ExLlamaV2 backend.
#629 opened Nov 26, 2024 by WeiweiZhang1
Where is the Mathverse result？
#611 opened Nov 19, 2024 by StarUniversus
1
llava-onevision multi-image
#607 opened Nov 15, 2024 by jun0wanan
1
Error: module 'torch.library' has no attribute 'register_fake'
#605 opened Nov 15, 2024 by tjasmin111
1
Ovis1.5-Llama3-8B在Hallusion Bench上的指标和榜单上的指标差距过大
#595 opened Nov 13, 2024 by LIRENDA621
1
How soon is the 'News' section in the README updated after a new Benchmark or Model is added?
#590 opened Nov 10, 2024 by Baiqi-Li
1
Evaluations get stuck on the last 4 questions awaiting confirm
#588 opened Nov 9, 2024 by XuGW-Kevin
4
ChartQA eval bug
#584 opened Nov 7, 2024 by lemonliu1992
Corrupted MathVista testmini data WIP
#578 opened Nov 5, 2024 by LeoDu0314
1
AI2d gpt和claude3.5官方分数非常高 awaiting confirm
#577 opened Nov 5, 2024 by Violettttee
1
Qwen-VL-Max-0809 MME中celebrity测出来和榜单结果差距有10分左右 awaiting confirm
#571 opened Nov 2, 2024 by lihua8848
7
[Bug] RuntimeError: world_size (8) is not equal to tensor_model_parallel_size (1) x pipeline_model_parallel_size (1) awaiting confirm
#561 opened Oct 31, 2024 by Wiselnn570
2
LLaVA v1.5 7b very low performance on TextVQA
#554 opened Oct 28, 2024 by wufeim
The result of gpt4o on MathVista is 61.6, which is lower than the 62.7 on the list. awaiting confirm
#553 opened Oct 28, 2024 by kkk123698745
3
评测MVBench时报错
#551 opened Oct 25, 2024 by helloworld01001
2
How to eval with multigpu when using device_map='auto'
#528 opened Oct 18, 2024 by qianwangn
@FangXinyu-0913
1
Different evaluate result between different VLMEvalKit version
#523 opened Oct 16, 2024 by terry-for-github
@kennymckormick
2
Are images compressed in tsv files?
#517 opened Oct 12, 2024 by ywwynm
3
textonly benchmarks
#497 opened Sep 30, 2024 by JcWang20
@PhoenixZ810
1
[Help Wanted] the alignment with official accuracy in llama3.2-vision Feature Request Extra attention is needed
#493 opened Sep 29, 2024 by droidXrobot
8
Reproducing QWen2VL Results on Video Benchmarks with VLMEvalKit
#484 opened Sep 23, 2024 by aniki-ly
@FangXinyu-0913
4
【提问】关于语音模型的评测 Feature Request Extra attention is needed
#473 opened Sep 19, 2024 by xx0412
1
ProTip! Find all open issues with in progress development work with linked:pr.