Qwen-VL-Max-0809 MME中celebrity测出来和榜单结果差距有10分左右

[lihua8848]

测试代码都是VLMEvalKit，我只改了api为qwen-vl-max-0809，以及测MME的celebrity，prompt这些都没动，计算scores的方法也没动，为什么和榜单上的差异这么大

[lihua8848]

第三张图是前几天那个榜单上的，请问1101更新为什么消失了？

[kennymckormick]

Hi, @lihua8848 ,
Recently we have added Qwen2-VL-72B to the leaderboard. As the Qwen Team introduced, the model behind Qwen-VL-Max-0809 is Qwen2-VL-72B, so we remove the Qwen-VL-Max-0809 entry. But maybe we can add it back to the OpenVLM Leaderboard later.

For the performance issue, I can help re-conduct the evaluation to see the numbers. Also, note that since Qwen-VL-Max is a proprietary API model and the sample number is small for each single capability in MME, the evaluation results can be variable.

[lihua8848]

It would be great if you could help me reproduce the MME celebrity results. How can I assist you? I'm using the following API for testing.

[kennymckormick]

@lihua8848

Basically, my current evaluation results align with yours.

[kennymckormick]

The results of Qwen-VL-Max-0809 are added back to the leaderboard.

[lihua8848]

The celebrity metric is still incorrect

[kennymckormick]

@lihua8848
To clarify, for API models, there is no so called correct performance (since the model behind may change frequently). The entry on the leaderboard only represents the performance at the evaluation time. I will add a new field Evaluation Time to the leaderboard to make it clearer.