Not work in google colab.

[Rondalter]

I am running it in colab, and since the modification that added the photomaker it does not work.
You click on generate and it stays blocked.

[machineminded]

I will need to make a separate configuration script for unix-like environments. Shouldn't be too much work. I am not familiar with Google Colab but can make an effort.

[machineminded]

@Rondalter I did go in and update the notebook. I am trying to use it but I don't think it will work because it is saying I have zero compute resources available. I'm not familiar with how Colab works but if you could test it out that would be great.

[Rondalter]

equirement already satisfied: pygit2==1.12.2 in /usr/local/lib/python3.10/dist-packages (1.12.2)
Requirement already satisfied: cffi>=1.9.1 in /usr/local/lib/python3.10/dist-packages (from pygit2==1.12.2) (1.16.0)
Requirement already satisfied: pycparser in /usr/local/lib/python3.10/dist-packages (from cffi>=1.9.1->pygit2==1.12.2) (2.21)
/content/Fooocus-inswapper
[System ARGV] ['launch.py', '--share']
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]
Fooocus version: 2.1.864
Version mismatch for tb-nightly: Installed version 2.16.0a20240202 does not meet requirement tb-nightly
Installing requirements
Total VRAM 15102 MB, total RAM 12979 MB
Set vram state to: NORMAL_VRAM
Always offload VRAM
Device: cuda:0 Tesla T4 : native
VAE dtype: torch.float32
Using pytorch cross attention
Exception in thread Thread-2 (worker):
Traceback (most recent call last):
File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/usr/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/content/Fooocus-inswapper/modules/async_worker.py", line 25, in worker
import modules.default_pipeline as pipeline
File "/content/Fooocus-inswapper/modules/default_pipeline.py", line 1, in
import modules.core as core
File "/content/Fooocus-inswapper/modules/core.py", line 1, in
from modules.patch import patch_all
File "/content/Fooocus-inswapper/modules/patch.py", line 11, in
import ldm_patched.modules.sd1_clip
File "/content/Fooocus-inswapper/ldm_patched/modules/sd1_clip.py", line 3, in
from transformers import CLIPTokenizer
ImportError: cannot import name 'CLIPTokenizer' from 'transformers' (/usr/local/lib/python3.10/dist-packages/transformers/init.py)
2024-02-02 18:25:31.015317: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-02-02 18:25:31.015376: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-02-02 18:25:31.021640: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-02-02 18:25:32.883777: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Refiner unloaded.
model_type EPS
UNet ADM Dimension 2816
Using pytorch attention in VAE
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
Using pytorch attention in VAE
extra {'cond_stage_model.clip_l.logit_scale', 'cond_stage_model.clip_l.text_projection'}
left over keys: dict_keys(['cond_stage_model.clip_l.transformer.text_model.embeddings.position_ids'])
Base model loaded: /content/Fooocus-inswapper/models/checkpoints/juggernautXL_v8Rundiffusion.safetensors
Request to load LoRAs [['sd_xl_offset_example-lora_1.0.safetensors', 0.1], ['None', 1.0], ['None', 1.0], ['None', 1.0], ['None', 1.0]] for model [/content/Fooocus-inswapper/models/checkpoints/juggernautXL_v8Rundiffusion.safetensors].
Loaded LoRA [/content/Fooocus-inswapper/models/loras/sd_xl_offset_example-lora_1.0.safetensors] for UNet [/content/Fooocus-inswapper/models/checkpoints/juggernautXL_v8Rundiffusion.safetensors] with 788 keys at weight 0.1.
Fooocus V2 Expansion: Vocab with 642 words.
/usr/local/lib/python3.10/dist-packages/torch/_utils.py:776: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
return self.fget.get(instance, owner)()
Fooocus Expansion engine loaded for cuda:0, use_fp16 = True.
Requested to load SDXLClipModel
Requested to load GPT2LMHeadModel
Loading 2 new models
[Fooocus Model Management] Moving model(s) has taken 0.60 seconds
Controls: [textbox, textbox, checkboxgroup, radio, radio, slider, textbox, slider, slider, dropdown, dropdown, slider, dropdown, slider, dropdown, slider, dropdown, slider, dropdown, slider, dropdown, slider, checkbox, textbox, radio, image, checkboxgroup, image, textbox, image, image, slider, slider, radio, image, slider, slider, radio, image, slider, slider, radio, image, slider, slider, radio, checkbox, image, number, checkbox, image, image, image, image]
Running on local URL: http://127.0.0.1:7865/
Running on public URL: https://9e35fd3f3edf6da406.gradio.live/

This share link expires in 72 hours. For free permanent hosting and GPU upgrades, run gradio deploy from Terminal to deploy to Spaces (https://huggingface.co/spaces)

[Rondalter]

it stays at that point.
I have once managed to generate a basic image.
But if you try any option, an error appears.

[machineminded]

On the free tier of Google Colab, how much VRAM is allocated? I think it would need at least 16GB, but even that is probably not enough for use with Fooocus. Maybe 18GB or so. On my local with a 3090, Fooocus is using just over 16GB during generation.

[machineminded]

When I run it, I am only allocated 15GB of VRAM on the free tier. If you are using the free T4 then this could be the issue.

[machineminded]

I should have asked if this was for inswapper or PhotoMaker.

[zachysaur]

Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]
Fooocus version: 2.1.864
Version mismatch for tb-nightly: Installed version 2.16.0a20240202 does not meet requirement tb-nightly
Installing requirements
Total VRAM 15102 MB, total RAM 12979 MB
Set vram state to: NORMAL_VRAM
Always offload VRAM
Device: cuda:0 Tesla T4 : native
VAE dtype: torch.float32
Using pytorch cross attention
Exception in thread Thread-2 (worker):
Traceback (most recent call last):
File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/usr/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/content/Fooocus-inswapper/modules/async_worker.py", line 25, in worker
import modules.default_pipeline as pipeline
File "/content/Fooocus-inswapper/modules/default_pipeline.py", line 1, in
import modules.core as core
File "/content/Fooocus-inswapper/modules/core.py", line 1, in
from modules.patch import patch_all
File "/content/Fooocus-inswapper/modules/patch.py", line 11, in
import ldm_patched.modules.sd1_clip
File "/content/Fooocus-inswapper/ldm_patched/modules/sd1_clip.py", line 3, in
from transformers import CLIPTokenizer
ImportError: cannot import name 'CLIPTokenizer' from 'transformers' (/usr/local/lib/python3.10/dist-packages/transformers/init.py)
2024-02-02 18:25:31.015317: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-02-02 18:25:31.015376: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-02-02 18:25:31.021640: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-02-02 18:25:32.883777: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Refiner unloaded.
model_type EPS
UNet ADM Dimension 2816
Using pytorch attention in VAE
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
Using pytorch attention in VAE
extra {'cond_stage_model.clip_l.logit_scale', 'cond_stage_model.clip_l.text_projection'}
left over keys: dict_keys(['cond_stage_model.clip_l.transformer.text_model.embeddings.position_ids'])
Base model loaded: /content/Fooocus-inswapper/models/checkpoints/juggernautXL_v8Rundiffusion.safetensors
Request to load LoRAs [['sd_xl_offset_example-lora_1.0.safetensors', 0.1], ['None', 1.0], ['None', 1.0], ['None', 1.0], ['None', 1.0]] for model [/content/Fooocus-inswapper/models/checkpoints/juggernautXL_v8Rundiffusion.safetensors].
Loaded LoRA [/content/Fooocus-inswapper/models/loras/sd_xl_offset_example-lora_1.0.safetensors] for UNet [/content/Fooocus-inswapper/models/checkpoints/juggernautXL_v8Rundiffusion.safetensors] with 788 keys at weight 0.1.
Fooocus V2 Expansion: Vocab with 642 words.
/usr/local/lib/python3.10/dist-packages/torch/_utils.py:776: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
return self.fget.get(instance, owner)()
Fooocus Expansion engine loaded for cuda:0, use_fp16 = True.
Requested to load SDXLClipModel
Requested to load GPT2LMHeadModel
Loading 2 new models
[Fooocus Model Management] Moving model(s) has taken 0.60 seconds
Controls: [textbox, textbox, checkboxgroup, radio, radio, slider, textbox, slider, slider, dropdown, dropdown, slider, dropdown, slider, dropdown, slider, dropdown, slider, dropdown, slider, dropdown, slider, checkbox, textbox, radio, image, checkboxgroup, image, textbox, image, image, slider, slider, radio, image, slider, slider, radio, image, slider, slider, radio, image, slider, slider, radio, checkbox, image, number, checkbox, image, image, image, image]
Running on local URL: http://127.0.0.1:7865/
Running on public URL: https://9e35fd3f3edf6da406.gradio.live/

This share link expires in 72 hours. For free permanent hosting and GPU upgrades, run gradio deploy from Terminal to deploy to Spaces (https://huggingface.co/spaces)

[EntonyHop]

the same issue, infinite generation of even a normal image. The style button doesn't load either. The fooocus itself doesn't work, I think it's the problem.

[machineminded]

I am still working thru this. It will take some time but hopefully soon.

[EntonyHop]

Let us know when you fix the problem, all work has stopped. Thx

[machineminded]

@EntonyHop Understood - I will try to get this working.

[machineminded]

I am am able to see normal Fooocus features working now in another branch. I found the issue with the inswapper integration on linux and will resolve that next.

[machineminded]

This has been difficult for me to test. But I am making progress. We need a better way to manage the queue:

lllyasviel#2271

Additionally, I believe they have some code in place that detects if you are running Stable Diffusion on the free tier, so the VM/container gets shut down. I don't have a paid subscription otherwise this would be easier for me to test. Will consider paying in order to get this issue closed up.

[zachysaur]

try to kaggle

[machineminded]

try to kaggle

Thanks for the suggestion. I'll check it out!

[mashb1t]

This has been difficult for me to test. But I am making progress. We need a better way to manage the queue:

lllyasviel#2271

Additionally, I believe they have some code in place that detects if you are running Stable Diffusion on the free tier, so the VM/container gets shut down. I don't have a paid subscription otherwise this would be easier for me to test. Will consider paying in order to get this issue closed up.

no additional code, but args for better resource allocation: lllyasviel#1710

[dmitryalexander]

does in-swapper work for colab now?

[mashb1t]

@dmitryalexander Fooocus (this repository) does not have an inswapper implementation. Please check out https://github.com/machineminded/Fooocus-inswapper maintained by @machineminded.

omg, nvm, wrong repository. I'm so sorry 😬

[dmitryalexander]

@dmitryalexander Fooocus (this repository) does not have an inswapper implementation. Please check out https://github.com/machineminded/Fooocus-inswapper maintained by @machineminded.

omg, nvm, wrong repository. I'm so sorry 😬

yeah i got the inswapper working, but i couldn't get the instantID or the photo maker to work

[machineminded]

@dmitryalexander Is it just not working in colab? What is the error? I don't have a good way to support colab because I'd have to pay for it to use SD.

[dmitryalexander]

it's only 10$ a month for colab. but yeah, the instant id, it just doesn't activate... it's like it's invisible and the photomaker, it says something like trigger word activated more than once... and it doesn't work from there, i guess that problem happens on some CUDA cards. but apparently the main branch fixed that issue, but i wasn't able to find out how.

[machineminded]

Can you show me the actual log output or a screenshot if you get a chance?

[dmitryalexander]

Traceback (most recent call last): File "/content/Fooocus-inswapper/modules/async_worker.py", line 1047, in worker handler(task) File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "/content/Fooocus-inswapper/modules/async_worker.py", line 915, in handler imgs = generate_photomaker(photomaker_prompt, photomaker_source_images, photomaker_negative_prompt, steps, task['task_seed'], width, height, guidance_scale, loras, sampler_name, scheduler_name, async_task) File "/content/Fooocus-inswapper/modules/pm.py", line 119, in generate_photomaker images = pipe( File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "/content/Fooocus-inswapper/photomaker/pipeline.py", line 344, in call ) = self.encode_prompt_with_trigger_word( File "/content/Fooocus-inswapper/photomaker/pipeline.py", line 176, in encode_prompt_with_trigger_word raise ValueError( ValueError: PhotoMaker currently does not support multiple trigger words in a single prompt. Trigger word: img, Prompt: completely normal prompt nothing to see here. Total time: 0.47 seconds

[dmitryalexander]

apparently the problem was that there's some kind of space before the trigger word... or something like that?

[machineminded]

Oh, yeah. This is how PhotoMaker works: you need to put "img" in the prompt after "man" or "woman". So your prompt with PhotoMaker enabled might be something like:

"woman img, smile, long hair, standing in a library, incredible lighting"

Check out some of the documentation here:

https://github.com/TencentARC/PhotoMaker?tab=readme-ov-file#use-like-diffusers

"img" is the default trigger word.

[dmitryalexander]

interesting....i'll try it out :thanks: