Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SafetensorError: Error while deserializing header: HeaderTooLarge #34

Open
smruthi49 opened this issue Oct 27, 2024 · 6 comments
Open

Comments

@smruthi49
Copy link

smruthi49 commented Oct 27, 2024

, line 313, in load_file
    with safe_open(filename, framework="pt", device=device) as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
safetensors_rust.SafetensorError: Error while deserializing header: HeaderTooLarge

Received the above error while running :

python weights/convert_safetensor_to_pt.py

Note:
Ran the above only after fixing the error in requirements.txt.
Downloaded the ckpt files from HF
I am running this in PopOS

@tajchert
Copy link

This seems related to my issue at #18 as it has same error message.

@smruthi49
Copy link
Author

smruthi49 commented Oct 27, 2024

I cleared the error. I did not download the weights properly from HF.
I had to do this before cloning from HF.

git lfs install

Now the weights are downloaded properly and the "python weights/convert_safetensor_to_pt.py" works as well.

@tajchert
Copy link

Interesting, I will re-do my steps as I already had lfs installed.

@callbacked
Copy link

callbacked commented Oct 27, 2024

doing git lfs install downloads the tools required to clone large file size objects (which is what we need since the weights are big).

what you need to do afterwards is run git lfs clone [repo] to fully clone the repo and its weights

alternatively, if you already have the repo cloned through git clone you can do git lfs pull to download the remaining large files (i.e. the weights that we need).

@yadong-lu
Copy link
Collaborator

Feel free to raise any questions or close the issue if you think the problem is resolved :) Thanks.

@abrichr
Copy link

abrichr commented Oct 30, 2024

See #52 for installing and running git lfs commands in the Dockerfile.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants