Regarding the evaluation of OpenVLA on the SIMPLER simulation platform

[leonwu117]

Dear author, first of all, I would like to sincerely thank you for summarizing and publishing your work on the current popular visual-language action models. As I am planning to conduct research based on OpenVLA, I followed your previous attempt to evaluate OpenVLA on SIMPLER. However, I found that the evaluation of OpenVLA on the Widox dataset resulted in a success rate of only 0. Therefore, I would like to ask if you have any better methods to reproduce the results presented in the OpenVLA paper, or if you have any suggestions. Relly appreciate that!

[DelinQu]

Hi! Thanks for your attention 🤗
As far as I know, openvla does not perform well on the WidowX, likely due to the camera's orientation being perpendicular to the table, which lacks geometric information such as vanishing line. Additionally, the control method in the simulation itself might differ from other environments. xuanlinli17 suspects that the issue is related to data augmentation in #issuecomment, but I think this might not be the primary cause. Recently, CogACT has achieved impressively remarkable results on simpler environments based on OpenVLA. Similarly, I'm extremely curious and would strongly suggest you submit a new issue to the repository.