Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Calling to() is not supported for 4-bit quantized models with the installed version of bitsandbytes. The current device is cuda:0. If you intended to move the model, please install bitsandbytes >= 0.43.2. #35442

Open
1 of 3 tasks
Kulbuntu opened this issue Dec 28, 2024 · 2 comments

Comments

@Kulbuntu
Copy link

Kulbuntu commented Dec 28, 2024

System Info

Transformers : version 4.48.0.dev0
OS : Arch Linux 6.12.6 Kernel
Python version : 3.11.0
Rocm : 6.3.1 version
GPU : AMD 6800 XT

Who can help?

No response

Information

  • The official example scripts
  • My own modified scripts

Tasks

  • [X ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
  • My own task or dataset (give details below)

Reproduction

Hi guys i got a big problem with this error. Few hours ago i was able to use CXH Joy Caption for make a detailled prompt from an image. Sadly it don't work anymore and i don't know at all how i made it work for the first time
When i launch CXH Joy caption it give me this error even with the good bitsandbytes version.
i saw @younesbelkada made a script `from transformers import AutoModelForCausalLM, AutoConfig, AutoTokenizer
import torch

model_path="tiiuae/falcon-40b-instruct"

config = AutoConfig.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True, load_in_4bit=True, device_map="auto")

tokenizer = AutoTokenizer.from_pretrained("tiiuae/falcon-40b-instruct")

input_text = "Describe the solar system."
input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to("cuda")

outputs = model.generate(input_ids, max_length=10)
print(tokenizer.decode(outputs[0]))`

But i don't know to use it and where to use it for solve this issue .
Here you'll see my terminal log from my last session with ComfyUI.
If someone could help me it could be cool.
I'm pretty in all this world.

Here is the log.
`(bit) [bryan@archlinux ComfyUI]$ python main.py
[START] Security scan
DEPRECATION: Loading egg at /home/bryan/.pyenv/versions/3.11.1/envs/bit/lib/python3.11/site-packages/flash_attn-2.7.2.post1-py3.11.egg is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to use pip for package installation. Discussion can be found at pypa/pip#12330
[DONE] Security scan

ComfyUI-Manager: installing dependencies done.

** ComfyUI startup time: 2024-12-28 21:52:02.458268
** Platform: Linux
** Python version: 3.11.1 (main, Dec 26 2024, 18:00:13) [GCC 14.2.1 20240910]
** Python executable: /home/bryan/.pyenv/versions/bit/bin/python
** ComfyUI Path: /home/bryan/Documents/ComfyUI
** Log path: /home/bryan/Documents/ComfyUI/comfyui.log
DEPRECATION: Loading egg at /home/bryan/.pyenv/versions/3.11.1/envs/bit/lib/python3.11/site-packages/flash_attn-2.7.2.post1-py3.11.egg is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to use pip for package installation. Discussion can be found at pypa/pip#12330
DEPRECATION: Loading egg at /home/bryan/.pyenv/versions/3.11.1/envs/bit/lib/python3.11/site-packages/flash_attn-2.7.2.post1-py3.11.egg is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to use pip for package installation. Discussion can be found at pypa/pip#12330

Prestartup times for custom nodes:
0.9 seconds: /home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI-Manager

Total VRAM 16368 MB, total RAM 31903 MB
pytorch version: 2.5.1+rocm6.2
WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for:
PyTorch 2.5.1+cu121 with CUDA 1201 (you have 2.5.1+rocm6.2)
Python 3.11.11 (you have 3.11.1)
Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers)
Memory-efficient attention, SwiGLU, sparse and more won't be available.
Set XFORMERS_MORE_DETAILS=1 for more details
xformers version: 0.0.29
Set vram state to: NORMAL_VRAM
Device: cuda:0 AMD Radeon RX 6800 XT : native
Using sub quadratic optimization for attention, if you have memory or speed issues try using: --use-split-cross-attention
[Prompt Server] web root: /home/bryan/Documents/ComfyUI/web
Traceback (most recent call last):
File "/home/bryan/Documents/ComfyUI/nodes.py", line 2089, in load_custom_node
module_spec.loader.exec_module(module)
File "", line 940, in exec_module
File "", line 241, in _call_with_frames_removed
File "/home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI_LayerStyle/init.py", line 35, in
imported_module = importlib.import_module(".py.{}".format(name), name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/bryan/.pyenv/versions/3.11.1/lib/python3.11/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1206, in _gcd_import
File "", line 1178, in _find_and_load
File "", line 1149, in _find_and_load_unlocked
File "", line 690, in _load_unlocked
File "", line 940, in exec_module
File "", line 241, in _call_with_frames_removed
File "/home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI_LayerStyle/py/inner_shadow_v2.py", line 3, in
from .imagefunc import log, tensor2pil, pil2tensor, image2mask, shift_image, expand_mask, chop_image_v2, chop_mode_v2
File "/home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI_LayerStyle/py/imagefunc.py", line 30, in
from skimage import img_as_float, img_as_ubyte
ModuleNotFoundError: No module named 'skimage'

Cannot import /home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI_LayerStyle module for custom nodes: No module named 'skimage'
Traceback (most recent call last):
File "/home/bryan/Documents/ComfyUI/nodes.py", line 2089, in load_custom_node
module_spec.loader.exec_module(module)
File "", line 936, in exec_module
File "", line 1073, in get_code
File "", line 1130, in get_data
FileNotFoundError: [Errno 2] No such file or directory: '/home/bryan/Documents/ComfyUI/custom_nodes/joy-caption-batch/init.py'

Cannot import /home/bryan/Documents/ComfyUI/custom_nodes/joy-caption-batch module for custom nodes: [Errno 2] No such file or directory: '/home/bryan/Documents/ComfyUI/custom_nodes/joy-caption-batch/init.py'
[VideoHelperSuite] - WARNING - Failed to import imageio_ffmpeg
Traceback (most recent call last):
File "/home/bryan/Documents/ComfyUI/nodes.py", line 2089, in load_custom_node
module_spec.loader.exec_module(module)
File "", line 940, in exec_module
File "", line 241, in _call_with_frames_removed
File "/home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI_LayerStyle_Advance/init.py", line 35, in
imported_module = importlib.import_module(".py.{}".format(name), name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/bryan/.pyenv/versions/3.11.1/lib/python3.11/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1206, in _gcd_import
File "", line 1178, in _find_and_load
File "", line 1149, in _find_and_load_unlocked
File "", line 690, in _load_unlocked
File "", line 940, in exec_module
File "", line 241, in _call_with_frames_removed
File "/home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI_LayerStyle_Advance/py/get_color_tone.py", line 3, in
from .imagefunc import log, tensor2pil, gaussian_blur, get_image_color_tone, get_image_color_average, RGB_to_HSV, Hex_to_RGB
File "/home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI_LayerStyle_Advance/py/imagefunc.py", line 30, in
from skimage import img_as_float, img_as_ubyte
ModuleNotFoundError: No module named 'skimage'

Cannot import /home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI_LayerStyle_Advance module for custom nodes: No module named 'skimage'
Total VRAM 16368 MB, total RAM 31903 MB
pytorch version: 2.5.1+rocm6.2
xformers version: 0.0.29
Set vram state to: NORMAL_VRAM
Device: cuda:0 AMD Radeon RX 6800 XT : native

Loading: ComfyUI-Manager (V2.55.5)

ComfyUI Version: v0.3.10-4-g4b5bcd8 | Released on '2024-12-27'

Import times for custom nodes:
0.0 seconds: /home/bryan/Documents/ComfyUI/custom_nodes/websocket_image_save.py
0.0 seconds: /home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI-Unload-Model
0.0 seconds (IMPORT FAILED): /home/bryan/Documents/ComfyUI/custom_nodes/joy-caption-batch
0.0 seconds: /home/bryan/Documents/ComfyUI/custom_nodes/Comfyui_JC2
0.0 seconds: /home/bryan/Documents/ComfyUI/custom_nodes/Comfyui_joytag
0.0 seconds: /home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI-Miaoshouai-Tagger
0.0 seconds: /home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI_SLK_joy_caption_two
0.0 seconds: /home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI-GGUF
0.0 seconds: /home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI-Custom-Scripts
0.0 seconds (IMPORT FAILED): /home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI_LayerStyle_Advance
0.0 seconds: /home/bryan/Documents/ComfyUI/custom_nodes/mikey_nodes
0.0 seconds: /home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI-LTXTricks
0.0 seconds: /home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI-VideoHelperSuite
0.0 seconds: /home/bryan/Documents/ComfyUI/custom_nodes/Comfyui_joy-caption-alpha-two
0.0 seconds (IMPORT FAILED): /home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI_LayerStyle
0.0 seconds: /home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI-Manager
0.3 seconds: /home/bryan/Documents/ComfyUI/custom_nodes/Comfyui_CXH_joy_caption
0.3 seconds: /home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI_NYJY
0.3 seconds: /home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI-LTXVideo

Starting server

To see the GUI go to: http://127.0.0.1:8188
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/model-list.json
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/github-stats.json
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/custom-node-list.json
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/alter-list.json
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/extension-node-map.json
^Agot prompt
/home/bryan/Documents/ComfyUI/models/clip/siglip-so400m-patch14-384
Loading VLM's custom vision model
/home/bryan/Documents/ComfyUI/custom_nodes/Comfyui_CXH_joy_caption/Joy_caption_alpha.py:237: FutureWarning: You are using torch.load with weights_only=False (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for weights_only will be flipped to True. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via torch.serialization.add_safe_globals. We recommend you start setting weights_only=True for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
checkpoint = torch.load(clip_model_path, map_location='cpu')
/home/bryan/Documents/ComfyUI/models/LLM/Meta-Llama-3.1-8B-bnb-4bit
Successfully modified 'base_model_name_or_path' value in '/home/bryan/Documents/ComfyUI/models/Joy_caption_alpha/text_model/adapter_config.json'.
Loading tokenizer
Loading LLM
Loading VLM's custom text model
Unused kwargs: ['_load_in_4bit', '_load_in_8bit', 'quant_method']. These kwargs are not used in <class 'transformers.utils.quantization_config.BitsAndBytesConfig'>.
We will use 90% of the memory on device 0 for storing the model, and 10% for the buffer to avoid OOM. You can set max_memory in to a higher value to use more memory (at your own risk).
!!! Exception during processing !!! Calling to() is not supported for 4-bit quantized models with the installed version of bitsandbytes. The current device is cuda:0. If you intended to move the model, please install bitsandbytes >= 0.43.2.
Traceback (most recent call last):
File "/home/bryan/Documents/ComfyUI/execution.py", line 328, in execute
output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/bryan/Documents/ComfyUI/execution.py", line 203, in get_output_data
return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/bryan/Documents/ComfyUI/execution.py", line 174, in _map_node_over_list
process_inputs(input_dict, i)
File "/home/bryan/Documents/ComfyUI/execution.py", line 163, in process_inputs
results.append(getattr(obj, func)(**inputs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/bryan/Documents/ComfyUI/custom_nodes/Comfyui_CXH_joy_caption/Joy_caption_alpha.py", line 283, in gen
self.loadCheckPoint()
File "/home/bryan/Documents/ComfyUI/custom_nodes/Comfyui_CXH_joy_caption/Joy_caption_alpha.py", line 261, in loadCheckPoint
text_model = AutoModelForCausalLM.from_pretrained(CHECKPOINT_PATH, device_map="auto",
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/bryan/.pyenv/versions/bit/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 564, in from_pretrained
return model_class.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/bryan/.pyenv/versions/bit/lib/python3.11/site-packages/transformers/modeling_utils.py", line 4288, in from_pretrained
dispatch_model(model, **device_map_kwargs)
File "/home/bryan/.pyenv/versions/bit/lib/python3.11/site-packages/accelerate/big_modeling.py", line 498, in dispatch_model
model.to(device)
File "/home/bryan/.pyenv/versions/bit/lib/python3.11/site-packages/transformers/modeling_utils.py", line 3089, in to
raise ValueError(
ValueError: Calling to() is not supported for 4-bit quantized models with the installed version of bitsandbytes. The current device is cuda:0. If you intended to move the model, please install bitsandbytes >= 0.43.2.

Prompt executed in 3.92 seconds
^C
Stopped server
(bit) [bryan@archlinux ComfyUI]$ python -m bitsandbytes
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++ BUG REPORT INFORMATION ++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++ OTHER +++++++++++++++++++++++++++
CUDA specs: CUDASpecs(highest_compute_capability=(10, 3), cuda_version_string='62', cuda_version_tuple=(6, 2))
PyTorch settings found: CUDA_VERSION=62, Highest Compute Capability: (10, 3).
WARNING: CUDA versions lower than 11 are currently not supported for LLM.int8().
You will be only to use 8-bit optimizers and quantization routines!
To manually override the PyTorch CUDA version please see: https://github.com/TimDettmers/bitsandbytes/blob/main/docs/source/nonpytorchcuda.mdx
The directory listed in your path is found to be non-existent: local/archlinux
The directory listed in your path is found to be non-existent: @/tmp/.ICE-unix/935,unix/archlinux
The directory listed in your path is found to be non-existent: /usr/etc/pyenv.d
The directory listed in your path is found to be non-existent: /usr/local/etc/pyenv.d
The directory listed in your path is found to be non-existent: /etc/pyenv.d
The directory listed in your path is found to be non-existent: /usr/lib/pyenv/hooks
The directory listed in your path is found to be non-existent: /org/freedesktop/DisplayManager/Session1
The directory listed in your path is found to be non-existent: /etc/gtk/gtkrc
The directory listed in your path is found to be non-existent: /home/bryan/.gtkrc
The directory listed in your path is found to be non-existent: /etc/gtk-2.0/gtkrc
The directory listed in your path is found to be non-existent: /Sessions/1
The directory listed in your path is found to be non-existent: /org/freedesktop/DisplayManager/Seat0
The directory listed in your path is found to be non-existent: //debuginfod.archlinux.org
The directory listed in your path is found to be non-existent: /Windows/1
CUDA SETUP: WARNING! CUDA runtime files not found in any environmental path.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++ DEBUG INFO END ++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Checking that the library is importable and CUDA is callable...
SUCCESS!
Installation was successful!
`

Expected behavior

It should return a prompt for an image but it don't

@Kulbuntu Kulbuntu added the bug label Dec 28, 2024
@LysandreJik
Copy link
Member

WDYT @SunMarc

@SunMarc
Copy link
Member

SunMarc commented Dec 30, 2024

Can you check your version of bitsandbytes @Kulbuntu , the error happens because your version is too low.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants