Using previously cached huggingface models and locally installed Lora Finetunes

[filmo]

I've been using Python to fine tune various models with Lora on an Ubuntu 22.04 LTS system

Using the transformers library, it caches the models in:

~/.cache/huggingface

which I believe is the standard location for caching. I have 100s of gigs of models.

Is it possible to set up oobabooga to use the existing huggingface cache instead of downloading duplicate copies to text-generation-webui/models ??

Likewise for local Lora fine-tunes ?? We don't want to push our Loras to a public repository. I tried entering the path to my local Lora in "Download custom model or LoRA" field in the off chance that it would work, but it does not (appears to only want to query the huggingface public repositories)

[nosajthenitram]

I implemented a version of HF cache support this weekend and it seems to work. There's more to be done, but the models can be queried in the cache and loaded. I'll issue a pull request if others are interested.

[FilipeD]

Did this make it? I've found myself in this situation too many times so I finally (this past week june'24) decided to do something. I also placed a question about it and ended up hacking my way to (locally) put an end to my problems. Still, when checking if anyone had spoted the questions, I found this post of yours of 1 year ago. It's kind of annoying to find that it hasn't (to my best knowledge) been dealt with.

[sinogermany]

To whom looking for a quick solution - I've got a little hack here, until the codebase properly loads the cached models.

If you are using Linux or WSL2

create a make_models_dir.py file (Thank you CodeLLama !) :

import os
import sys
from pathlib import Path
import shutil

def main(base_dir):
    base_path = Path(base_dir)

    # Make sure base_dir exists
    if not base_path.exists():
        os.mkdir(base_dir)

    # Clear up everything in base_dir
    for item in base_path.iterdir():
        if item.is_file():
            os.remove(item)
        else:
            shutil.rmtree(item)

    # Iterate over directories starting with 'models--'
    for org_repo_path in Path(os.path.join(Path.home(), '.cache/huggingface/hub')).glob('models--*'):
        org_repo_name = org_repo_path.stem[len("models--"):]
        hash_path = next((org_repo_path / 'snapshots').iterdir())
        hash_value = hash_path.stem
        symlink_path = base_path / f"{org_repo_name}--{hash_value}"
        os.symlink(hash_path, symlink_path)

if __name__ == "__main__":
    main(sys.argv[1])

Then I called it like this:

python3 make_models_dir.py ~/dev/huggingface-models-sym-link

And after that start your webui like this:

./start_linux.sh --model-dir ~/dev/huggingface-models-sym-link

If you are using windows:

Go run this in WSL2. If you just HAVE TO run it in Windows:

1. Copy paste the models to the models directory, takes you few minutes to do.
2. If you're not switching models at runtime, do something like

./start_windows.sh --model-dir <cache-dir>\models--ORG--REPO\snapshots

That gives you a slightly awkward models list but at least you know what it is.

Worked for me. Enjoy 🤗

[Ch4mpa9ne]

I have a simpler and more convenient way here, which follows the same principle as sinogermany. Both methods use symbolic links. Below is the process. You can replicate this process to create symbolic links for all models, or you can use a script to achieve it.

ln -s ~/.cache/huggingface/hub/models--THUDM--chatglm2-6b/snapshots/a8c811e5922c4775741639645746bd37c55d57d6/ ~/model_link/models--THUDM--chatglm2-6b

ln -s ~/.cache/huggingface/hub/models--THUDM--chatglm-6b/snapshots/8b7d33596d18c5e83e2da052d05ca4db02e60620/ ~/model_link/models--THUDM--chatglm-6b

python server.py --chat --trust-remote-code --listen --model-dir ~/model_link

This will provide you with a list based on your custom model names.