Controlnet training method for playgroundV2.5 model

[baizh0u]

When I try to train a controlnet model for playgroundV2.5, but there were some issues happened.
The general training code in diffusers for controlnet sdxl is not fit for EDM model, training is ok, loss looks ok, but the result is broke, after 400 steps the image looks like snow pieces. although I tied to set the scheduler to EDM type....
I think is mostly due to the EDM mean and EDM std inside the VAE config and the vae scale factor is also different from normal sdxl model.
I think something should change for encode the images.
How can I fix it?

[sayakpaul]

This should be discussion as it doesn't relate to any bug in the library.

You can refer to #7126 that added support for doing LoRA DreamBooth with the Playground model and apply similar changes to the ControlNet training script.