Forward pass of the model - how to pass videos?

[esh04]

I see this is how you run inference on the model:

    output = model.generate(
              input_ids, 
              return_dict_in_generate=True,
              output_scores=True,
              images_or_videos=videos,
              modal_list=['video'],
              max_new_tokens=128,
              do_sample=True,
              temperature=0.2,
              use_cache=False,
          )

I wanted to know how to run training code. The forward pass of the model does not support modal_list and images_or_videos - then how do I pass the videos? I want to be able to pass labels as I need to calculate probabilities. (so I need logits)

[LiangMeng89]

Hello,I'm a phD student from ZJU, I also use videollama2 to do my own research,we create a WeChat group to discuss some issues of videollama2 and help each other,could you join us? Please contact me: WeChat number == LiangMeng19357260600, phone number == +86 19357260600,e-mail == [email protected].