LLaVA extension

[oobabooga]

In case anyone missed the PR by @Wojtab: #1487

README: https://github.com/oobabooga/text-generation-webui/tree/main/extensions/llava

This is a multimodal model that understands the contents of images much better than previous models. It uses about 11GB VRAM.

Usage:

python download-model.py wojtab/llava-13b-v0-4bit-128g
python3 server.py --model wojtab_llava-13b-v0-4bit-128g --chat  --extensions llava

[BarfingLemurs]

I was trying this picture test for making a website. It doesn't work the way I thought for now, the output is only visible with --verbose since chat mode can't render these things properly.

It is very neat and I hope some templates can be made for making websites from pictures.

[viperwasp]

Does anyone know of a better/updated model? Or is there just this 13B and nothing else? I would love to get like 15B, 30B, version of this. Or a GGML version. Any information would be welcomed. Thanks.

[tig3rmast3r]

there is no extensions/llava anymore
is it still possible to load LLaVA within text-generation-webui ?
thanks

EDIT: nevermind i found instruction on the main page thanks

[NaggingDaivy]

Hello,

Where did you find instruction for installing LLAVA on text-generation-webui ? I can't find any information on that on LLAVA website, neither on text-generation-webui's github.

[tig3rmast3r]

https://github.com/oobabooga/text-generation-webui/tree/main/extensions/multimodal
btw i wasn't able to let it work with Llava, only Vicuna

[TomLucidor]

I am also thinking of something similar but for the more SOTA models

• https://github.com/magic-research/bubogpt
• https://github.com/QwenLM/Qwen-VL
• https://github.com/THUDM/CogVLM
• https://github.com/UX-Decoder/LLaVA-Grounding

[R3xpook]

CogVLM like absolutely i would even say it reach GPT level of accuracy completly obliterate mini gpt4, llava, blip instruct

[darkacorn]

id love to have a way to get cogvlm integrated .. ! sadly not a easy feast - talked to quite a few people in the quant real (casper from autoawq/turbo from exllama) .. its really not a easy push

info from turbo was about 50h initial integration work .. and then the upkeep