Enable multiple LoRa adapters

[drbh]

This PR is a work in progress to add support for mutliple loras to be loaded at startup and then use 0 or 1 adapters in a request by specifying the adapter id.

Example usage

download adapter without auto merging

text-generation-server download-weights predibase/dbpedia
text-generation-server download-weights predibase/customer_support

start server with multiple LoRa adapters

LORA_ADAPTERS=predibase/customer_support,predibase/dbpedia \
text-generation-launcher --model-id mistralai/Mistral-7B-v0.1

sending request without adapter id

curl 127.0.0.1:3000/generate \
    -X POST \
    -H 'Content-Type: application/json' \
    -d '{
    "inputs": "What are 3 unique words that describe you?",
    "parameters": {
    "max_new_tokens": 40
  }
}'

{
  "generated_text": "\n\nI’m a very passionate person. I’m very driven. I’m very determined.\n\nWhat is your favorite thing about being a teacher?\n\nI love the fact"
}

with first LoRa adapter specified

curl 127.0.0.1:3000/generate \
    -X POST \
    -H 'Content-Type: application/json' \
    -d '{
    "inputs": "What are 3 unique words that describe you?",
    "parameters": {
    "max_new_tokens": 40,
    "adapter_id": "predibase/customer_support"
  }
}'

{
  "generated_text": "\n\nI’m not sure if I can come up with 3 unique words that describe me, but I’ll try.\n\n1. Creative\n2. Funny\n3."
}

with second LoRa adapter specified

curl 127.0.0.1:3000/generate \
    -X POST \
    -H 'Content-Type: application/json' \
    -d '{
    "inputs": "You are given the title and the body of an article below. Please determine the type of the article.### Title: Great White Whale\n\n### Body: Great White Whale is the debut album by the Canadian rock band Secret and Whisper. The album was in the works for about a year and was released on February 12 2008.",                                                                                             
    "parameters": {
    "max_new_tokens": 40,
    "adapter_id": "predibase/dbpedia"
  }
}'

{
  "generated_text": "8"
}

[flozi00]

Hey guys, as I am switched to lorax and started contributing there a lot after the first license change I am happy to see the PR got opened

I would be happy if you are open for some questions and discussion about this.
What do you think about following the lorax style for the api so it could be a drop in replacement, especially for the openai endpoint?
Furthermore are you open for some kernel optimizations like punica, used in lorax and vllm, to minimize the overhead and enable efficient batching of different adapters ?

I'd would be happy to contribute here too.

[drbh]

hi @flozi00 thanks for the feedback! can you share more about the lorax style api? I see that in lorax you can specify the adapter via the model field in the chat endpoint, is that the feature you're referring to? regarding kernel optimizations, yes we are very interested and plan on using optimized kernels/punica (planning on diving into this specific thing tomorrow/this week). Any suggestions/contributions/patches related to this are always appreciated!

[flozi00]

Yes, i mean the "adapter_id" inside "parameters" for the tgi api (as you did it, i see now), and the "model" in the openai api :)

[drbh]

update:

This PR's implementation has been updated to align with the great work done by the lorax team. This implementation tries to use the same layers when possible and only diverges to work with TGI's recent updates/improvements and limits lora to loading at startup. Current changes allow weights to be loaded similar to Lorax, however there are still issues with generation to be resolved, and other refactors

[flozi00]

Looks like you are successfully adopting the lorax code
Please tell me if you need any help with this feature

[drbh]

@flozi00 generation with loras is mostly stable, just focusing on the rebase then refactors now. And thank you 🙂 a review once the PR is ready would be super helpful!

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[tgaddair]

Thanks for the shoutout in the docs! It's quite interesting to see things come full circle, maybe we should chat about merging our projects.

[drbh]

of course @tgaddair thank you for the awesome work! thats an interesting idea and we are always aiming to improve TGI. We appreciate any contributions/discussions about features that may be helpful to our users

[flozi00]

Thanks for the shoutout in the docs! It's quite interesting to see things come full circle, maybe we should chat about merging our projects.

I'd love to migrate to tgi again 👍 And of course trying to contribute here too @tgaddair

[drbh]

hi @xiadingZ in this PR lora adapters are loaded from the HUGGINGFACE_HUB_CACHE directory. Once you've downloaded the lora locally, you can specify the id like LORA_ADAPTERS=predibase/customer_support which will use the local lora model.

once this initial lora work is merged we'll follow up with other improvement such as easier ways to specify lora path, and etc

[xiadingZ]

hi @xiadingZ in this PR lora adapters are loaded from the HUGGINGFACE_HUB_CACHE directory. Once you've downloaded the lora locally, you can specify the id like LORA_ADAPTERS=predibase/customer_support which will use the local lora model.

once this initial lora work is merged we'll follow up with other improvement such as easier ways to specify lora path, and etc

Hi, @drbh I can try your methods with downloaded lora. But I have a lora adapter trained locally. It doesn't have a directory structure such as blobs, refs, snapshots from huggingface. how can I place it in HUGGINGFACE_HUB_CACHE and load it?

I set HUGGINGFACE_HUB_CACHE as /root/lora_adapters and my directory structure is:

[danieldk]

Forgot to add: we probably want an integration test as well.

[danieldk]

Thanks for all the changes! Looks ready to merge to me after the small nit that breaks CI is fixed.

[drbh]

@danieldk thanks for the review! I've fixed the nits and CI passes. Going to go ahead and merge based on your last approval

[mhou7712]

Hi @flozi00, is it possible to look at my issue #2143 and let me know any suggestion? Thanks.