Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flux-controlnet #438

Open
apepkuss opened this issue Oct 14, 2024 · 2 comments
Open

Flux-controlnet #438

apepkuss opened this issue Oct 14, 2024 · 2 comments

Comments

@apepkuss
Copy link

Hi guys,

I'm trying XLabs-AI/flux-controlnet-canny-v3 with stable-diffusion.cpp and run into the following error message:

[ERROR] model.cpp:1694 - tensor 'input_hint_block.10.bias' has wrong shape in model file: got [16, 1, 1, 1], expected [96, 1, 1, 1]
[WARN ] model.cpp:1592 - process tensor failed: 'input_hint_block.10.bias'
[ERROR] model.cpp:1710 - load tensors from file failed
[ERROR] control.hpp:449  - load control net tensors from model loader failed
new_sd_ctx_t failed

I am not sure if it was caused by the model itself. Anyone here ran into this issue? The entire log can be found below. Thanks a lot!

Log 
sd --diffusion-model flux1-dev-Q4_0.gguf --vae ae.safetensors --clip_l clip_l.safetensors --t5xxl t5xxl-Q8_0.gguf --control-net flux-canny-controlnet-v3.safetensors --control-image input_canny.png -p "cyberpank dining room, full hd, cinematic" -v
Option: 
    n_threads:         4
    mode:              txt2img
    model_path:        
    wtype:             unspecified
    clip_l_path:       clip_l.safetensors
    t5xxl_path:        t5xxl-Q8_0.gguf
    diffusion_model_path:   flux1-dev-Q4_0.gguf
    vae_path:          ae.safetensors
    taesd_path:        
    esrgan_path:       
    controlnet_path:   flux-canny-controlnet-v3.safetensors
    embeddings_path:   
    stacked_id_embeddings_path:   
    input_id_images_path:   
    style ratio:       20.00
    normalize input image :  false
    output_path:       output.png
    init_img:          
    control_image:     input_canny.png
    clip on cpu:       false
    controlnet cpu:    false
    vae decoder on cpu:false
    strength(control): 0.90
    prompt:            cyberpank dining room, full hd, cinematic
    negative_prompt:   
    min_cfg:           1.00
    cfg_scale:         7.00
    guidance:          3.50
    clip_skip:         -1
    width:             512
    height:            512
    sample_method:     euler_a
    schedule:          default
    sample_steps:      20
    strength(img2img): 0.75
    rng:               cuda
    seed:              42
    batch_count:       1
    vae_tiling:        false
    upscale_repeats:   1
System Info: 
    BLAS = 0
    SSE3 = 1
    AVX = 1
    AVX2 = 1
    AVX512 = 0
    AVX512_VBMI = 0
    AVX512_VNNI = 0
    FMA = 1
    NEON = 0
    ARM_FMA = 0
    F16C = 1
    FP16_VA = 0
    WASM_SIMD = 0
    VSX = 0
[DEBUG] stable-diffusion.cpp:180  - Using CPU backend
[INFO ] stable-diffusion.cpp:202  - loading clip_l from 'clip_l.safetensors'
[INFO ] model.cpp:793  - load clip_l.safetensors using safetensors format
[DEBUG] model.cpp:861  - init from 'clip_l.safetensors'
[INFO ] stable-diffusion.cpp:209  - loading t5xxl from 't5xxl-Q8_0.gguf'
[INFO ] model.cpp:790  - load t5xxl-Q8_0.gguf using gguf format
[DEBUG] model.cpp:807  - init from 't5xxl-Q8_0.gguf'
WARNING: Behavior may be unexpected when allocating 0 bytes for ggml_calloc!
[INFO ] stable-diffusion.cpp:216  - loading diffusion model from 'flux1-dev-Q4_0.gguf'
[INFO ] model.cpp:790  - load flux1-dev-Q4_0.gguf using gguf format
[DEBUG] model.cpp:807  - init from 'flux1-dev-Q4_0.gguf'
WARNING: Behavior may be unexpected when allocating 0 bytes for ggml_calloc!
[INFO ] stable-diffusion.cpp:223  - loading vae from 'ae.safetensors'
[INFO ] model.cpp:793  - load ae.safetensors using safetensors format
[DEBUG] model.cpp:861  - init from 'ae.safetensors'
[INFO ] stable-diffusion.cpp:235  - Version: Flux Dev 
[INFO ] stable-diffusion.cpp:266  - Weight type:                 f16
[INFO ] stable-diffusion.cpp:267  - Conditioner weight type:     f16
[INFO ] stable-diffusion.cpp:268  - Diffusion model weight type: q4_0
[INFO ] stable-diffusion.cpp:269  - VAE weight type:             f32
[DEBUG] stable-diffusion.cpp:271  - ggml tensor size = 400 bytes
[DEBUG] clip.hpp:171  - vocab size: 49408
[DEBUG] clip.hpp:182  -  trigger word img already in vocab
[DEBUG] ggml_extend.hpp:1046 - clip params backend buffer size =  235.06 MB(RAM) (196 tensors)
[DEBUG] ggml_extend.hpp:1046 - t5 params backend buffer size =  9083.77 MB(RAM) (219 tensors)
[DEBUG] ggml_extend.hpp:1046 - flux params backend buffer size =  6394.53 MB(RAM) (780 tensors)
[DEBUG] ggml_extend.hpp:1046 - vae params backend buffer size =  94.57 MB(RAM) (138 tensors)
[DEBUG] stable-diffusion.cpp:398  - loading weights
[DEBUG] model.cpp:1530 - loading tensors from clip_l.safetensors
[DEBUG] model.cpp:1530 - loading tensors from t5xxl-Q8_0.gguf
[INFO ] model.cpp:1685 - unknown tensor 'text_encoders.t5xxl.encoder.embed_tokens.weight | q8_0 | 2 [4096, 32128, 1, 1, 1]' in model file
[DEBUG] model.cpp:1530 - loading tensors from flux1-dev-Q4_0.gguf
[DEBUG] model.cpp:1530 - loading tensors from ae.safetensors
[INFO ] control.hpp:434  - loading control net from 'flux-canny-controlnet-v3.safetensors'
[DEBUG] ggml_extend.hpp:1046 - control_net params backend buffer size =  511.14 MB(RAM) (340 tensors)
[INFO ] model.cpp:793  - load flux-canny-controlnet-v3.safetensors using safetensors format
[DEBUG] model.cpp:861  - init from 'flux-canny-controlnet-v3.safetensors'
[DEBUG] model.cpp:1530 - loading tensors from flux-canny-controlnet-v3.safetensors
[INFO ] model.cpp:1685 - unknown tensor 'controlnet_blocks.0.bias | bf16 | 1 [3072, 1, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'controlnet_blocks.0.weight | bf16 | 2 [3072, 3072, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'controlnet_blocks.1.bias | bf16 | 1 [3072, 1, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'controlnet_blocks.1.weight | bf16 | 2 [3072, 3072, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.img_attn.norm.key_norm.scale | bf16 | 1 [128, 1, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.img_attn.norm.query_norm.scale | bf16 | 1 [128, 1, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.img_attn.proj.bias | bf16 | 1 [3072, 1, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.img_attn.proj.weight | bf16 | 2 [3072, 3072, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.img_attn.qkv.bias | bf16 | 1 [9216, 1, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.img_attn.qkv.weight | bf16 | 2 [3072, 9216, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.img_mlp.0.bias | bf16 | 1 [12288, 1, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.img_mlp.0.weight | bf16 | 2 [3072, 12288, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.img_mlp.2.bias | bf16 | 1 [3072, 1, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.img_mlp.2.weight | bf16 | 2 [12288, 3072, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.img_mod.lin.bias | bf16 | 1 [18432, 1, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.img_mod.lin.weight | bf16 | 2 [3072, 18432, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.txt_attn.norm.key_norm.scale | bf16 | 1 [128, 1, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.txt_attn.norm.query_norm.scale | bf16 | 1 [128, 1, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.txt_attn.proj.bias | bf16 | 1 [3072, 1, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.txt_attn.proj.weight | bf16 | 2 [3072, 3072, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.txt_attn.qkv.bias | bf16 | 1 [9216, 1, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.txt_attn.qkv.weight | bf16 | 2 [3072, 9216, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.txt_mlp.0.bias | bf16 | 1 [12288, 1, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.txt_mlp.0.weight | bf16 | 2 [3072, 12288, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.txt_mlp.2.bias | bf16 | 1 [3072, 1, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.txt_mlp.2.weight | bf16 | 2 [12288, 3072, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.txt_mod.lin.bias | bf16 | 1 [18432, 1, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.txt_mod.lin.weight | bf16 | 2 [3072, 18432, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.img_attn.norm.key_norm.scale | bf16 | 1 [128, 1, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.img_attn.norm.query_norm.scale | bf16 | 1 [128, 1, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.img_attn.proj.bias | bf16 | 1 [3072, 1, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.img_attn.proj.weight | bf16 | 2 [3072, 3072, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.img_attn.qkv.bias | bf16 | 1 [9216, 1, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.img_attn.qkv.weight | bf16 | 2 [3072, 9216, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.img_mlp.0.bias | bf16 | 1 [12288, 1, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.img_mlp.0.weight | bf16 | 2 [3072, 12288, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.img_mlp.2.bias | bf16 | 1 [3072, 1, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.img_mlp.2.weight | bf16 | 2 [12288, 3072, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.img_mod.lin.bias | bf16 | 1 [18432, 1, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.img_mod.lin.weight | bf16 | 2 [3072, 18432, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.txt_attn.norm.key_norm.scale | bf16 | 1 [128, 1, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.txt_attn.norm.query_norm.scale | bf16 | 1 [128, 1, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.txt_attn.proj.bias | bf16 | 1 [3072, 1, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.txt_attn.proj.weight | bf16 | 2 [3072, 3072, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.txt_attn.qkv.bias | bf16 | 1 [9216, 1, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.txt_attn.qkv.weight | bf16 | 2 [3072, 9216, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.txt_mlp.0.bias | bf16 | 1 [12288, 1, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.txt_mlp.0.weight | bf16 | 2 [3072, 12288, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.txt_mlp.2.bias | bf16 | 1 [3072, 1, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.txt_mlp.2.weight | bf16 | 2 [12288, 3072, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.txt_mod.lin.bias | bf16 | 1 [18432, 1, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.txt_mod.lin.weight | bf16 | 2 [3072, 18432, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'guidance_in.in_layer.bias | bf16 | 1 [3072, 1, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'guidance_in.in_layer.weight | bf16 | 2 [256, 3072, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'guidance_in.out_layer.bias | bf16 | 1 [3072, 1, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'guidance_in.out_layer.weight | bf16 | 2 [3072, 3072, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'img_in.bias | bf16 | 1 [3072, 1, 1, 1, 1]' in model file
[INFO ] model.cpp:1685 - unknown tensor 'img_in.weight | bf16 | 2 [64, 3072, 1, 1, 1]' in model file
[ERROR] model.cpp:1694 - tensor 'input_hint_block.10.bias' has wrong shape in model file: got [16, 1, 1, 1], expected [96, 1, 1, 1]
[WARN ] model.cpp:1592 - process tensor failed: 'input_hint_block.10.bias'
[ERROR] model.cpp:1710 - load tensors from file failed
[ERROR] control.hpp:449  - load control net tensors from model loader failed
new_sd_ctx_t failed
@grauho
Copy link
Contributor

grauho commented Oct 15, 2024

I don't think Flux has control net support yet.

@apepkuss
Copy link
Author

You're right. I got the answer from XLabs AI.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@apepkuss @grauho