[Bug] tgi can not download model meta-llama/Meta-Llama-3.1-70B-Instruct

[lianhao]

Priority

P3-Medium

OS type

Ubuntu

Hardware type

Xeon-GNR

Installation method

• Pull docker images from hub.docker.com
• Build docker images from source

Deploy method

• Docker compose
• Docker
• Kubernetes
• Helm

Running nodes

Single Node

What's the version?

github commit 74fe45d

Description

With the recent merged PR #613, we use huggingface/downloader:0.17.3 image as the InitContainer to download the image. However, it seems that image doesn't acknowledge the HF_TOKEN to download models requires special permission.

Reproduce steps

helm install tgi . -f gaudi-values.yaml --set global.HUGGINGFACEHUB_API_TOKEN=${HFTOKEN} --set LLM_MODEL_ID=meta-llama/Meta-Llama-3.1-70B-Instruct

Raw log

downloading https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct/resolve/945c8663693130f8be2ee66210e062158b2a9693/original/params.json to /data/tmpk1qo4jmt
downloading https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct/resolve/945c8663693130f8be2ee66210e062158b2a9693/.gitattributes to /data/tmpi1exvwm9
downloading https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct/resolve/945c8663693130f8be2ee66210e062158b2a9693/USE_POLICY.md to /data/tmp885sss8a
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/huggingface_hub/utils/_errors.py", line 261, in hf_raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct/resolve/945c8663693130f8be2ee66210e062158b2a9693/.gi
tattributes

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/huggingface-cli", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/huggingface_hub/commands/huggingface_cli.py", line 49, in main
    service.run()
  File "/usr/local/lib/python3.11/site-packages/huggingface_hub/commands/download.py", line 156, in run
    print(self._download())  # Print path to downloaded files
          ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/huggingface_hub/commands/download.py", line 191, in _download
    return snapshot_download(
           ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/huggingface_hub/_snapshot_download.py", line 239, in snapshot_download
    thread_map(
  File "/usr/local/lib/python3.11/site-packages/tqdm/contrib/concurrent.py", line 69, in thread_map
    return _executor_map(ThreadPoolExecutor, fn, *iterables, **tqdm_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/tqdm/contrib/concurrent.py", line 51, in _executor_map
    return list(tqdm_class(ex.map(fn, *iterables, chunksize=chunksize), **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/tqdm/std.py", line 1170, in __iter__
    for obj in iterable:
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 619, in result_iterator
    yield _result_or_cancel(fs.pop())
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 317, in _result_or_cancel
    return fut.result(timeout)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/huggingface_hub/_snapshot_download.py", line 214, in _inner_hf_hub_download
    return hf_hub_download(
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/huggingface_hub/file_download.py", line 1431, in hf_hub_download
    http_get(
  File "/usr/local/lib/python3.11/site-packages/huggingface_hub/file_download.py", line 524, in http_get
    hf_raise_for_status(r)
  File "/usr/local/lib/python3.11/site-packages/huggingface_hub/utils/_errors.py", line 277, in hf_raise_for_status
    raise GatedRepoError(message, response) from e
huggingface_hub.utils._errors.GatedRepoError: 401 Client Error. (Request ID: Root=1-675a96a6-33ce89f905d89f186658f45e;1af87390-148b-4e0d-b94e-1c10a5b83e95)

Cannot access gated repo for url https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct/resolve/945c8663693130f8be2ee66210e062158b2a9693/.gitattributes.
Access to model meta-llama/Llama-3.1-70B-Instruct is restricted. You must have access to it and be authenticated to access it. Please log in.