Dim Mismatch Using Stable Diffusion Turbo Weights

[laetokang]

Hello,

I am a newbie who recently developed an interest in video generation. Thanks to this interest, I was able to train and infer a model that generates portraits using the Stable Diffusion v1.5 backbone.

Next, I plan to train using the Stable Diffusion Turbo weights for real-time generation.

However, I have encountered a challenge due to a dimension mismatch. Could you please advise on how to typically resolve such issues?

Thank you!

Traceback (most recent call last):
File "/workspace/workspace/EMOPortrait/train_stage_1.py", line 771, in
main(config)
File "/workspace/workspace/EMOPortrait/train_stage_1.py", line 593, in main
model_pred = net(
File "/root/anaconda3/envs/feature_b/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/root/anaconda3/envs/feature_b/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/root/anaconda3/envs/feature_b/lib/python3.10/site-packages/accelerate/utils/operations.py", line 581, in forward
return model_forward(*args, **kwargs)
File "/root/anaconda3/envs/feature_b/lib/python3.10/site-packages/accelerate/utils/operations.py", line 569, in call
return convert_to_fp32(self.model_forward(*args, **kwargs))
File "/root/anaconda3/envs/feature_b/lib/python3.10/site-packages/torch/amp/autocast_mode.py", line 16, in decorate_autocast
return func(*args, **kwargs)
File "/workspace/workspace/EMOPortrait/train_stage_1.py", line 83, in forward
self.reference_unet(
File "/root/anaconda3/envs/feature_b/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/root/anaconda3/envs/feature_b/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/root/anaconda3/envs/feature_b/lib/python3.10/site-packages/diffusers/models/unet_2d_condition.py", line 1075, in forward
sample, res_samples = downsample_block(
File "/root/anaconda3/envs/feature_b/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/root/anaconda3/envs/feature_b/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/root/anaconda3/envs/feature_b/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py", line 1160, in forward
hidden_states = attn(
File "/root/anaconda3/envs/feature_b/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/root/anaconda3/envs/feature_b/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/root/anaconda3/envs/feature_b/lib/python3.10/site-packages/diffusers/models/transformer_2d.py", line 392, in forward
hidden_states = block(
File "/root/anaconda3/envs/feature_b/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/root/anaconda3/envs/feature_b/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/root/anaconda3/envs/feature_b/lib/python3.10/site-packages/diffusers/models/attention.py", line 323, in forward
attn_output = self.attn2(
File "/root/anaconda3/envs/feature_b/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/root/anaconda3/envs/feature_b/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/root/anaconda3/envs/feature_b/lib/python3.10/site-packages/diffusers/models/attention_processor.py", line 522, in forward
return self.processor(
File "/root/anaconda3/envs/feature_b/lib/python3.10/site-packages/diffusers/models/attention_processor.py", line 1137, in call
key = attn.to_k(encoder_hidden_states, *args)
File "/root/anaconda3/envs/feature_b/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/root/anaconda3/envs/feature_b/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/root/anaconda3/envs/feature_b/lib/python3.10/site-packages/diffusers/models/lora.py", line 430, in forward
out = super().forward(hidden_states)
File "/root/anaconda3/envs/feature_b/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward
return F.linear(input, self.weight, self.bias)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x768 and 1024x320)

• Model Architecture Based on Stable Diffusion Turbo
  

• Model Architecture Based on Stable Diffusion v1.5
  

[tolgacangoz]

[laetokang]

Thank you for your reply. To be more specific, I am currently working on a portrait generation task using a model called AniPortrait. I plan to train it using the video data I have collected. However, the inference speed of the currently trained model is too slow, which is why I want to train and infer using a Turbo-based model to see if it improves performance.

[tolgacangoz]

What happens if you use all the speeding-up methods for SD v1.5? I see that AniPortrait uses DDIMScheduler for inferencing. LCMScheduler and TCDScheduler utilize very few numbers of inference steps. But I am unsure if these would be proper for this kind of video generation task.

[tolgacangoz]

For SD Turbo training, could you be sure that you use its proper components? Also, I don't know if one should adapt Adversarial Diffusion Distillation to this training somehow.

[laetokang]

I found that the performance of the SD Turbo model was inferior to that of the SD v1.5 in generating images, contrary to my expectations.
While the v1.5 model generated images that closely reflected the information from the trained image data, the Turbo model produced images in a different style. This issue persisted even though the parameters were set to default during training.