[Bug]: Stable Lora args missing in API calls

[dotsimulate]

Is there an existing issue for this?

• I have searched the existing issues and checked the recent builds/commits of both this extension and the webui

Are you using the latest version of the extension?

• I have the modelscope text2video extension updated to the lastest version and I still have the issue.

What happened?

It seems like the new Stable Lora args causes errors with the API calls when the render.py is calling process_modelscope. I just tested on a fully fresh install of both a1111 and extension and I get the same error using the API as I normally do, and with the example call in the local docs.

At first, I thought it could be fixed similar to #192 in scripts/t2vhelpers/render.py by separating out the lora args. But after a few attempts of some fixes, I am unsure how to fix or separate the correct args or where to set defaults for them in order to allow the API to work as it did before. There is a very good chance I am missing something obvious and will be looking at this again in the morning.

It is working as expected in the webui.

error message on the API call -
Traceback (most recent call last):
File "F:\SD-Master\7.5\stable-diffusion-webui/extensions/sd-webui-text2video/scripts\t2v_helpers\render.py", line 30, in run
vids_pack = process_modelscope(args_dict, args)
File "F:\SD-Master\7.5\stable-diffusion-webui/extensions/sd-webui-text2video/scripts\modelscope\process_modelscope.py", line 70, in process_modelscope
stable_lora_processor.process(pipe, *stable_lora_args)
File "F:\SD-Master\7.5\stable-diffusion-webui\venv\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
TypeError: StableLoraScript.process() missing 8 required positional arguments: 'lora_files_selection', 'lora_alpha', 'use_bias', 'use_linear', 'use_conv', 'use_emb', 'use_multiplier', and 'use_time'
Exception occurred: StableLoraScript.process() missing 8 required positional arguments: 'lora_files_selection', 'lora_alpha', 'use_bias', 'use_linear', 'use_conv', 'use_emb', 'use_multiplier', and 'use_time'

Steps to reproduce the problem

1. Launch Webui with --api and the text2video extension installed
2. Go to http://127.0.0.1:7860/docs and scroll down to the sample API call
3. Add example args and hit execute. This is the same exact error I've also gotten from Touchdesigner when running API calls

What should have happened?

API working as expected.

WebUI and Deforum extension Commit IDs

webui commit id - f865d3e11647dfd6c7b2cdf90dde24680e58acd8
txt2vid commit id - 8f0af8c

Torch version

torch: 2.0.1+cu118
xformers: 0.0.20  
python: 3.10.9

What GPU were you using for launching?

4090

On which platform are you launching the webui backend with the extension?

Local PC setup (Windows)

Settings

{'prompt': 'A vibrant cityscape drawn with pastels, flowing paint transforming the scene', 'n_prompt': '(cutscene), (low quality, VHS, artifacts), iphone, tex', 'sampler': 'UniPC', 'model': 'zs2_448w', 'steps': 24, 'frames': 60, 'seed': 2196745325, 'cfg_scale': 14, 'width': 448, 'height': 256, 'eta': 0.0, 'fps': 24, 'batch_count': 1, 'do_vid2vid': False, 'strength': None, 'vid2vid_startFrame': None, 'inpainting_frames': None, 'inpainting_weights': None, 'add_soundtrack': None}

also same error with
{'prompt': 'A vibrant cityscape drawn with pastels, flowing paint transforming the scene', 'n_prompt': '(cutscene), (low quality, VHS, artifacts), iphone, tex', 'sampler': 'UniPC', 'model': 'zs2_448w', 'steps': 24, 'frames': 60, 'seed': 3579263276, 'cfg_scale': 14, 'width': 448, 'height': 256, 'eta': 0.0, 'fps': 24, 'batch_count': 1, 'do_vid2vid': False, 'strength': None, 'vid2vid_startFrame': None, 'inpainting_frames': None, 'inpainting_weights': None, 'add_soundtrack': None, 'lora_files_selection': None, 'lora_alpha': 0.0, 'use_bias': False, 'use_linear': False, 'use_conv': False, 'use_emb': False, 'use_multiplier': False, 'use_time': False}

Console logs

venv "F:\SD-Master\7.5\stable-diffusion-webui\venv\Scripts\Python.exe"
Python 3.10.9 (tags/v3.10.9:1dd9be6, Dec  6 2022, 20:01:21) [MSC v.1934 64 bit (AMD64)]
Version: v1.4.1
Commit hash: f865d3e11647dfd6c7b2cdf90dde24680e58acd8
Installing requirements

Launching Web UI with arguments: --api --xformers
Loading weights [6ce0161689] from F:\SD-Master\7.5\stable-diffusion-webui\models\Stable-diffusion\v1-5-pruned-emaonly.safetensors
preload_extensions_git_metadata for 10 extensions took 0.16s
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 6.6s (import torch: 1.9s, import gradio: 1.0s, import ldm: 0.5s, other imports: 1.2s, load scripts: 1.1s, create ui: 0.6s, gradio launch: 0.2s).
Creating model from config: F:\SD-Master\7.5\stable-diffusion-webui\configs\v1-inference.yaml
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Applying attention optimization: xformers... done.
Textual inversion embeddings loaded(0):
Model loaded in 4.7s (load weights from disk: 1.0s, create model: 0.4s, apply weights to model: 1.1s, apply half(): 1.0s, move model to device: 1.1s).
text2video — The model selected is: zs2_448w (ModelScope-like)
 text2video extension for auto1111 webui
Git commit: 8f0af8c9
Starting text2video
Pipeline setup
config namespace(framework='pytorch', task='text-to-video-synthesis', model={'type': 'latent-text-to-video-synthesis', 'model_args': {'ckpt_clip': 'open_clip_pytorch_model.bin', 'ckpt_unet': 'text2video_pytorch_model.pth', 'ckpt_autoencoder': '../VQGAN_autoencoder.pth', 'max_frames': 16, 'tiny_gpu': 1}, 'model_cfg': {'unet_in_dim': 4, 'unet_dim': 320, 'unet_y_dim': 768, 'unet_context_dim': 1024, 'unet_out_dim': 4, 'unet_dim_mult': [1, 2, 4, 4], 'unet_num_heads': 8, 'unet_head_dim': 64, 'unet_res_blocks': 2, 'unet_attn_scales': [1, 0.5, 0.25], 'unet_dropout': 0.1, 'temporal_attention': 'True', 'num_timesteps': 1000, 'mean_type': 'eps', 'var_type': 'fixed_small', 'loss_type': 'mse'}}, pipeline={'type': 'latent-text-to-video-synthesis'})
Traceback (most recent call last):
  File "F:\SD-Master\7.5\stable-diffusion-webui/extensions/sd-webui-text2video/scripts\t2v_helpers\render.py", line 30, in run
    vids_pack = process_modelscope(args_dict, args)
  File "F:\SD-Master\7.5\stable-diffusion-webui/extensions/sd-webui-text2video/scripts\modelscope\process_modelscope.py", line 70, in process_modelscope
    stable_lora_processor.process(pipe, *stable_lora_args)
  File "F:\SD-Master\7.5\stable-diffusion-webui\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
TypeError: StableLoraScript.process() missing 8 required positional arguments: 'lora_files_selection', 'lora_alpha', 'use_bias', 'use_linear', 'use_conv', 'use_emb', 'use_multiplier', and 'use_time'
Exception occurred: StableLoraScript.process() missing 8 required positional arguments: 'lora_files_selection', 'lora_alpha', 'use_bias', 'use_linear', 'use_conv', 'use_emb', 'use_multiplier', and 'use_time'

Additional information

I did attempt to fill in these args with a few values on my side to see if I could get the error to go away, but nothing that I tried seemed to work or change the error code. I was alerted of this by someone else also using the API, so I know its not something with my setup.

[ExponentialML]

Hi. I'm unable to resolve this at the moment, but I think I could lead you in the right direction (I haven't looked at it in full). I don't know if it hooks into the current UI process through Gradio, or strictly does a POST directly to FastAPI.

I had a look in the API code, and it seems that the instance and/or args isn't passed to the API endpoint.

You would have to do an import something along the lines of:

from stable_lora.scripts.lora_webui import StableLoraScriptInstance
StableLoraScript = StableLoraScriptInstance

Here is how the script process works:

# You would use the default args (as a list) + stable_lora_ui.
stable_lora_ui = StableLoraScript.ui()

stable_lora_processor = StableLoraScriptInstance

# extra_args = args + stable_lora_ui  
stable_lora_args = stable_lora_processor.process_extension_args(all_args=extra_args) 

# pipe = ModelScope model (pipe.unet, pipe.text_encoder, etc.)
stable_lora_processor.process(pipe, *stable_lora_args)

The script relies on Gradio to process the args, which is why I can't give an assured fix at the moment for the API without looking into it.

[dotsimulate]

I will try looking into that this week. Thank you. I want to start testing the Stable Lora stuff, but will need to find a fix for the API first.

[farrukhhameed50]

I have the same issue while sending a request via API. I'm also looking into it.

I'm running the webui on an A40.

Here are my logs:

text2video — The model selected is: base (ModelScope-like)
text2video extension for auto1111 webui
Git commit: 8f0af8c
Starting text2video
Pipeline setup
config namespace(framework='pytorch', task='text-to-video-synthesis', model={'type': 'latent-text-to-video-synthesis', 'model_args': {'ckpt_clip': 'open_clip_pytorch_model.bin', 'ckpt_unet': 'text2video_pytorch_model.pth', 'ckpt_autoencoder': 'VQGAN_autoencoder.pth', 'max_frames': 16, 'tiny_gpu': 1}, 'model_cfg': {'unet_in_dim': 4, 'unet_dim': 320, 'unet_y_dim': 768, 'unet_context_dim': 1024, 'unet_out_dim': 4, 'unet_dim_mult': [1, 2, 4, 4], 'unet_num_heads': 8, 'unet_head_dim': 64, 'unet_res_blocks': 2, 'unet_attn_scales': [1, 0.5, 0.25], 'unet_dropout': 0.1, 'temporal_attention': 'True', 'num_timesteps': 1000, 'mean_type': 'eps', 'var_type': 'fixed_small', 'loss_type': 'mse'}}, pipeline={'type': 'latent-text-to-video-synthesis'})
Traceback (most recent call last):
File "/home/far/sd/stable-diffusion-webui/extensions/sd-webui-text2video/scripts/t2v_helpers/render.py", line 30, in run
vids_pack = process_modelscope(args_dict, args)
File "/home/far/sd/stable-diffusion-webui/extensions/sd-webui-text2video/scripts/modelscope/process_modelscope.py", line 70, in process_modelscope
stable_lora_processor.process(pipe, *stable_lora_args)
File "/home/far/sd/stable-diffusion-webui/venv/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
TypeError: process() missing 8 required positional arguments: 'lora_files_selection', 'lora_alpha', 'use_bias', 'use_linear', 'use_conv', 'use_emb', 'use_multiplier', and 'use_time'
Exception occurred: process() missing 8 required positional arguments: 'lora_files_selection', 'lora_alpha', 'use_bias', 'use_linear', 'use_conv', 'use_emb', 'use_multiplier', and 'use_time'

[farrukhhameed50]

Any update on this. I'm unable to resolve the issue.

[selvaprakash]

We are also looking for a fix for this issue. Earlier it was working for us. So, I tried using a older version of the extension. That didnt work even on the ui side.
Any date on when we can expect a fix?

[occurrentarts]

Hey guys, I encountered this bug which prevented me from being able to run the extension and I determined a temporary hack to get around it. I commented out lines 68-70 in process_modelscope.py so it looks like this:

#TODO Wrap this in a list so that we can process this for future extensions.
#stable_lora_processor = StableLoraScriptInstance
#stable_lora_args = stable_lora_processor.process_extension_args(all_args=extra_args)
#stable_lora_processor.process(pipe, *stable_lora_args)

As far as I can tell this disables the processing of the LORA model which isn't ideal but seems to allow the extension to continue working. Obviously would be great to be able to keep using the LORAs module but this seems to work for the time being. Hopefully this helps someone.