Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add ipex readme #595

Merged
merged 15 commits into from
Mar 22, 2024
28 changes: 28 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,34 @@ where `extras` can be one or more of `ipex`, `neural-compressor`, `openvino`, `n

# Quick tour

## IPEX
Below are examples of how to use IPEX model to generate texts.
### generate
```diff
import torch
from transformers import AutoTokenizer, AutoConfig
- from transformers import AutoModelForCausalLM
+ from optimum.intel.ipex import IPEXModelForCausalLM

config = AutoConfig.from_pretrained("gpt2")
model = IPEXModelForCausalLM.from_pretrained(
jiqing-feng marked this conversation as resolved.
Show resolved Hide resolved
"gpt2",
jiqing-feng marked this conversation as resolved.
Show resolved Hide resolved
config=config,
torch_dtype=torch.bfloat16,
export=True,
)
tokenizer = AutoTokenizer.from_pretrained("gpt2")
input_sentence = ["Answer the following yes/no question by reasoning step-by-step please. Can you write a whole Haiku in a single tweet?"]
model_inputs = tokenizer(input_sentence, return_tensors="pt")
generation_kwargs = dict(max_new_tokens=32, do_sample=False, num_beams=4, num_beam_groups=1, no_repeat_ngram_size=2, use_cache=True)

generated_ids = model.generate(**model_inputs, **generation_kwargs)
output = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(output)
```

For more details, please refer to the [documentation](https://intel.github.io/intel-extension-for-pytorch/#introduction).

echarlaix marked this conversation as resolved.
Show resolved Hide resolved
## Neural Compressor

Dynamic quantization can be used through the Optimum command-line interface:
Expand Down
Loading