Code

Sync main with upstream#18

Code

Merged

z103cb merged 19 commits intoopendatahub-io:mainopendatahub-io/vllm:mainfrom z103cb:main_updatez103cb/opendatahub_vllm:main_updateCopy head branch name to clipboardMay 9, 2024

+1,226-604

Commits

Commits on May 8, 2024

• [Build/CI] Fixing 'docker run' to re-enable AMD CI tests. (vllm-project#4642)

  

  Alexei-V-Ivanov-AMD

  authored and

  z103cb

  committed
  52f24fc

• [Bugfix] Fixed error in slice_lora_b for MergedQKVParallelLinearWithLora (vllm-project#4609)

  

  FurtherAI

  authored and

  z103cb

  committed
  3574e04

• [Core][Optimization] change copy-on-write from dict[int, list] to list (vllm-project#4648)

  

  youkaichao

  authored and

  z103cb

  committed
  8be3803

• [Bug fix][Core] fixup ngram not setup correctly (vllm-project#4551)

  Show description for 9aa0af7
  

  4 peopleauthored and

  z103cb

  committed
  9aa0af7

• [Core][Distributed] support cpu&device in broadcast tensor dict (vllm-project#4660)

  Show description for aedb607
  

  youkaichao

  authored and

  z103cb

  committed
  aedb607

Commits on May 9, 2024

• [Core] Optimize sampler get_logprobs (vllm-project#4594)

  

  rkooo567

  authored and

  z103cb

  committed
  4ae5247

• [CI] Make mistral tests pass (vllm-project#4596)

  

  rkooo567

  authored and

  z103cb

  committed
  6c53a43

• [Bugfix][Kernel] allow non-power-of-2 for prefix prefill with alibi (vllm-project#4573)

  

  DefTruth

  authored and

  z103cb

  committed
  21d8850

• [Misc] Add get_name method to attention backends (vllm-project#4685)

  

  WoosukKwon

  authored and

  z103cb

  committed
  bfef5df

• [Core] Faster startup for LoRA enabled models (vllm-project#4634)

  

  Yard1

  authored and

  z103cb

  committed
  b74cdc3

• [Core][Optimization] change python dict to pytorch tensor for blocks to swap (vllm-project#4659)

  

  youkaichao

  authored and

  z103cb

  committed
  8f229ec

• [CI/Test] fix swap test for multi gpu (vllm-project#4689)

  

  youkaichao

  authored and

  z103cb

  committed
  cc3a658

• [Misc] Use vllm-flash-attn instead of flash-attn (vllm-project#4686)

  

  WoosukKwon

  authored and

  z103cb

  committed
  3ab3a02

• [Dynamic Spec Decoding] Auto-disable by the running queue size (vllm-project#4592)

  Show description for c08e3f1
  

  2 peopleauthored and

  z103cb

  committed
  c08e3f1

• [Speculative decoding] [Bugfix] Fix overallocation in ngram + spec logprobs (vllm-project#4672)

  

  cadedaniel

  authored and

  z103cb

  committed
  3f6229f

• [Bugfix] Fine-tune gptq_marlin configs to be more similar to marlin (vllm-project#4626)

  

  alexm-neuralmagic

  authored and

  z103cb

  committed
  626f3f4

• [Frontend] add tok/s speed metric to llm class when using tqdm (vllm-project#4400)

  Show description for 16fd703
  

  2 peopleauthored and

  z103cb

  committed
  16fd703

• [Frontend] Move async logic outside of constructor (vllm-project#4674)

  

  DarkLight1337

  authored and

  z103cb

  committed
  361c86f

• [Misc] Remove unnecessary ModelRunner imports (vllm-project#4703)

  

  WoosukKwon

  authored and

  z103cb

  committed
  f8de4ff