[Frontend] add build_server and make run_server async#107

Closed

dtrifiro wants to merge 13 commits intoopendatahub-io:mainfrom dtrifiro:entrypoint-split-server

+667-371

Commits on Jul 23, 2024

[Misc] Add ignored layers for fp8 quantization (vllm-project#6657 )
mgoin
authored
[Frontend] Add Usage data in each chunk for chat_serving. vllm-project#6540 (vllm-project#6652 )
yecohn
authored
[Model] Pipeline Parallel Support for DeepSeek v2 (vllm-project#6519 )
tjohnson31415
authored
Bump transformers version for Llama 3.1 hotfix and patch Chameleon (vllm-project#6690 )
ywang96
authored
[build] relax wheel size limit (vllm-project#6704 )
youkaichao
authored
[CI] Add smoke test for non-uniform AutoFP8 quantization (vllm-project#6702 )
mgoin
authored
[Bugfix] StatLoggers: cache spec decode metrics when they get collected. (vllm-project#6645 )
tdoublep
authored
[bitsandbytes]: support read bnb pre-quantized model (vllm-project#5753 )

thesues
and
mgoin
authored