Update bnb.md with example for OpenAI (#11718)

vllm-project · Jan 4, 2025 · d1d4939 · d1d4939
1 parent 9c93636
commit d1d4939
Showing 1 changed file with 7 additions and 0 deletions.
diff --git a/docs/source/quantization/bnb.md b/docs/source/quantization/bnb.md
@@ -37,3 +37,10 @@ model_id = "huggyllama/llama-7b"
 llm = LLM(model=model_id, dtype=torch.bfloat16, trust_remote_code=True, \
 quantization="bitsandbytes", load_format="bitsandbytes")
 ```
+## OpenAI Compatible Server
+
+Append the following to your 4bit model arguments:
+
+```
+--quantization bitsandbytes --load-format bitsandbytes
+```