Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add example notebook and argument for 8-bit-inference #82

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

orangetin
Copy link
Member

@orangetin orangetin commented Mar 31, 2023

This PR:

  • Adds the argument --load-in-8bit for inference
  • Adds an example jupyter/Colab notebook that can run bot.py inference (quantized) on a free Colab account (would have crashed after 5 prompts without quantizing. The user has the option to remove the 8-bit argument if running on a non-free account).
  • Update transformers==4.21.1 to transformers==4.27.4 because:
    • It adds support for 8-bit quantization to the model class
    • It shows a progress bar when loading the model which can be helpful for consumer hardware
  • Update documentation to reflect recent changes (new model and example notebook)
  • Fix typos

Note: the links to 'Open in Colab' have been modified to how it should look after the merge. For testing purposes, use this branch with the original links instead. Open In Colab

Solves #42

@orangetin orangetin marked this pull request as draft April 12, 2023 15:44
@orangetin
Copy link
Member Author

orangetin commented Apr 12, 2023

Turned to draft because it can be improved a bit as suggested by @exander77

@orangetin orangetin marked this pull request as ready for review April 18, 2023 21:36
@orangetin
Copy link
Member Author

This is ready for review.

The recent commit simplified the ChatModel class with the load-in-8bit arg and fixed an issue which was preventing it from being passed alongside CPU-offload.

@orangetin orangetin requested a review from justusc May 23, 2023 23:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant