[bug]: SD3.5 & Flux T5 models should be interchangeable. (also other tokenizers and textencoder might be the same)

[skunkworxdark]

Is there an existing issue for this problem?

• I have searched the existing issues

Operating system

Windows

GPU vendor

Nvidia (CUDA)

GPU model

No response

GPU VRAM

24GB

Version number

v5.4.2rc1

Browser

chrome

Python dependencies

No response

What happened

Select an SD3.5 main model and a Flux T5 sub-model, the generations work fine. However, when a Flux main model is selected with an SD3.5 T5 sub-model it fails with an assertion error when you try to generate.

What you expected to happen

The T5 models should work both ways. I think the issue is which SD3.5 sub-model is returned to the generation routine. With SD3.5 all sub-models are called the same name so you can't tell from the UI only.

How to reproduce the problem

No response

Additional context

I've not looked into it in depth but it might be related to the textencoder# and tokenizer_# folder definition differences between flux and sd3.5. t5 on flux is #2 but on sd3.5 it is #3. seems like the ui might be displaying the wrong sd3.5 sub model when a flux model is selected which is harder for the user to notice as all the sd3.5 model parts are all called "SD3.5 Size" no matter if it is a vae, t5 or clip model.

If they can all use the same sub models then there is a huge saving in storage space if they are shared. ~10Gb alone for just sd3.5 medium and large is common, so at least that could be saved. Then you also have textencoders and tokenizers for flux that seem to be shareable as well.

Discord username

Skunkworxdark