Repos for Training and Finetuning (1 already available!)

[kolabearafk]

Is there an existing issue for this?

• I have searched the existing issues and checked the recent builds/commits

What would your feature do ?

Is there any released training code or published paper mentioning the training methods used for this model?

Proposed workflow

N/A

Additional information

No response

[ExponentialML]

I can take a shot to see if this works with current available implementation floating around.

If we're just to training the CrossAttention layers (finetuning the Psuedo Conv3D layers are tricky) and limiting the size to 256x256, It may (this is a big if) be able to fit in 24GB of VRAM.

Also, I don't know if they used a DDPM scheduler for training or the Gaussian Diffusion scheduler for training as I don't know the correlating paper for this implementation. It seems to be a mix of video diffusion and Make-A-Video.

Either way, the process should be very simple if we reference the training methods we have floating around.

1. Add noise to video latents based on timestep.
2. Forward through 3D conditional unet with the noisy latents.
3. Calculate the loss with the model prediction and the noisy latents.

I'm also curious since the model already has a sufficient amount of data, you may be able to fine tune it in an unconditional way (no prompts, just video data).

[ExponentialML]

I created a repository for Text2Video finetuning here using the recent Diffusers addition. Let me know how it goes if you give it a shot!

https://github.com/ExponentialML/Text-To-Video-Finetuning

[kabachuha]

Incredible! @ExponentialML, I'll post it on Reddit if you don't mind?

Upd: posted here https://www.reddit.com/r/StableDiffusion/comments/11zhy1b/wake_up_samurai_modelscope_text2video_finetuning/

[kolabearafk]

@ExponentialML Wow, truly amazing. Can't wait to try it. Thank you!

[ExponentialML]

@kabachuha Didn't realize you posted it. All good, thanks for doing it!

[23Rj20]

@ExponentialML Hey can you please look at this error, for finetuning it is not able to locate the files even though they are present in that folder. Pease look at this issue I need an urgent fix for this.

I have uploaded te necessary screenshot to understand the error.
@kabachuha Can you also take a look at this please.