Code

[pull] main from vllm-project:main#15

Code

Closed

pull[bot] wants to merge 84 commits intoopendatahub-io:mainopendatahub-io/vllm:mainfrom vllm-project:mainvllm-project/vllm:mainCopy head branch name to clipboard

+8,986-2,936

Commits

Commits on Apr 30, 2024

• [BugFix] fix num_lookahead_slots missing in async executor (#4165)

  Show description for 4bb53e2
  

  leiwen83

  and

  wenlei03

  authored
  4bb53e2

• [Doc] add visualization for multi-stage dockerfile (#4456)

  Show description for b31a1fb
  

  prashantgupta24

  and

  ywang96

  authored
  b31a1fb

• [Kernel] Support Fp8 Checkpoints (Dynamic + Static) (#4332)

  Show description for 111815d
  

  6 peopleauthored
  111815d

• [Frontend] Support complex message content for chat completions endpoint (#3467)

  Show description for a494140
  

  3 peopleauthored
  a494140

• [Frontend] [Core] Tensorizer: support dynamic num_readers, update version (#4467)

  alpayariyak

  authored
  715c2d8

• [Bugfix][Minor] Make ignore_eos effective (#4468)

  bigPYJ1151

  authored
  dd1a50a

• fix_tokenizer_snapshot_download_bug (#4493)

  kingljl

  authored
  6ad58f4

Commits on May 1, 2024

• Unable to find Punica extension issue during source code installation (#4494)

  Show description for ee37328
  

  kingljl

  and

  simon-mo

  authored
  ee37328

• [Core] Centralize GPU Worker construction (#4419)

  njhill

  authored
  2e240c6

• [Misc][Typo] type annotation fix (#4495)

  HarryWu99

  authored
  f458112

• [Misc] fix typo in block manager (#4453)

  Juelianqvq

  authored
  a822eb3

• Allow user to define whitespace pattern for outlines (#4305)

  robcaulk

  authored
  c3845d8

• [Misc]Add customized information for models (#4132)

  jeejeelee

  authored
  d6f4bd7

• [Test] Add ignore_eos test (#4519)

  rkooo567

  authored
  6f1df80

• [Bugfix] Fix the fp8 kv_cache check error that occurs when failing to obtain the CUDA version. (#4173)

  Show description for a88bb9b
  AnyISalIn

  authored
  a88bb9b

• [Bugfix] Fix 307 Redirect for /metrics (#4523)

  robertgshaw2-neuralmagic

  authored
  4dc8026

• [Doc] update(example model): for OpenAI compatible serving (#4503)

  fpaupier

  authored
  e491c7e

• [Bugfix] Use random seed if seed is -1 (#4531)

  sasha0552

  authored
  6990912

• [CI/Build][Bugfix] VLLM_USE_PRECOMPILED should skip compilation (#4534)

  Show description for 8b798ee
  tjohnson31415

  authored
  8b798ee

• [Speculative decoding] Add ngram prompt lookup decoding (#4237)

  Show description for b38e42f
  

  leiwen83

  and

  wenlei03

  authored
  b38e42f

• [Core] Enable prefix caching with block manager v2 enabled (#4142)

  Show description for 24750f4
  

  3 peopleauthored
  24750f4

• [Core] Add multiproc_worker_utils for multiprocessing-based workers (#4357)

  njhill

  authored
  a657bfc

• [Kernel] Update fused_moe tuning script for FP8 (#4457)

  Show description for 24bb4fe
  pcmoritz

  authored
  24bb4fe

• [Bugfix] Add validation for seed (#4529)

  sasha0552

  authored
  c47ba4a

• [Bugfix][Core] Fix and refactor logging stats (#4336)

  esmeetu

  authored
  3a922c1

• [Core][Distributed] fix pynccl del error (#4508)

  youkaichao

  authored
  6ef09b0

• [Misc] Remove Mixtral device="cuda" declarations (#4543)

  Show description for c9d852d
  pcmoritz

  authored
  c9d852d

• [Misc] Fix expert_ids shape in MoE (#4517)

  WoosukKwon

  authored
  826b82a

Commits on May 2, 2024

• [MISC] Rework logger to enable pythonic custom logging configuration to be provided (#4273)

  

  Danny Guinther

  authored
  b8afa8b

• [Bug fix][Core] assert num_new_tokens == 1 fails when SamplingParams.n is not 1 and max_tokens is large & Add tests for preemption (#4451)

  rkooo567

  authored
  0d62fe5

• [CI]Add regression tests to ensure the async engine generates metrics (#4524)

  ronensc

  authored
  5e401bc

• [mypy][6/N] Fix all the core subdirectory typing (#4450)

  Show description for cf8cac8
  

  rkooo567

  and

  cadedaniel

  authored
  cf8cac8

• [Core][Distributed] enable multiple tp group (#4512)

  Show description for 2a85f93
  

  youkaichao

  and

  zhuohan123

  authored
  2a85f93

• [Kernel] Support running GPTQ 8-bit models in Marlin (#4533)

  alexm-neuralmagic

  authored
  7038e8b

• [mypy][7/N] Cover all directories (#4555)

  rkooo567

  authored
  fb087af

• [Misc] Exclude the tests directory from being packaged (#4552)

  itechbear

  authored
  5ad60b0

• [BugFix] Include target-device specific requirements.txt in sdist (#4559)

  markmc

  authored
  1ff0c73

• [Misc] centralize all usage of environment variables (#4548)

  youkaichao

  authored
  5b8a7c1

• [kernel] fix sliding window in prefix prefill Triton kernel (#4405)

  Show description for 32881f3
  

  mmoskal

  and

  rkooo567

  authored
  32881f3

• [CI/Build] AMD CI pipeline with extended set of tests. (#4267)

  Show description for 9b5c9f9
  

  Alexei-V-Ivanov-AMD

  and

  simon-mo

  authored
  9b5c9f9

• [Core] Ignore infeasible swap requests. (#4557)

  rkooo567

  authored
  0f8a914

Commits on May 3, 2024

• [Core][Distributed] enable allreduce for multiple tp groups (#4566)

  youkaichao

  authored
  344a5d0

• [BugFix] Prevent the task of _force_log from being garbage collected (#4567)

  Atry

  authored
  808632d

• [Misc] remove chunk detected debug logs (#4571)

  DefTruth

  authored
  ce3f1ee

• [Doc] add env vars to the doc (#4572)

  youkaichao

  authored
  2d7bce9

• [Core][Model runner refactoring 1/N] Refactor attn metadata term (#4518)

  rkooo567

  authored
  3521ba4

• [Bugfix] Allow "None" or "" to be passed to CLI for string args that default to None (#4586)

  mgoin

  authored
  7e65477

• Fix/async chat serving (#2727)

  schoennenbeck

  authored
  f8e7add

• [Kernel] Use flashinfer for decoding (#4353)

  Show description for 43c413e
  

  LiuXiaoxuanPKU

  and

  LiuXiaoxuanPKU

  authored
  43c413e

• [Speculative decoding] Support target-model logprobs (#4378)

  cadedaniel

  authored
  ab50275

• [Misc] add installation time env vars (#4574)

  youkaichao

  authored
  344bf7c

Commits on May 4, 2024

• [Misc][Refactor] Introduce ExecuteModelData (#4540)

  comaniac

  authored
  bc8ad68

• [Doc] Chunked Prefill Documentation (#4580)

  rkooo567

  authored
  36fb68f

• [Kernel] Support MoE Fp8 Checkpoints for Mixtral (Static Weights with Dynamic/Static Activations) (#4527)

  Show description for 2a05201
  mgoin

  authored
  2a05201

• [CI] check size of the wheels (#4319)

  simon-mo

  authored
  021b1a2

• [Bugfix] Fix inappropriate content of model_name tag in Prometheus metrics (#3937)

  DearPlanet

  authored
  4302987

Commits on May 5, 2024

• bump version to v0.4.2 (#4600)

  simon-mo

  authored
  8d8357c

• [CI] Reduce wheel size by not shipping debug symbols (#4602)

  simon-mo

  authored
  c7f2cf2

• Disable cuda version check in vllm-openai image (#4530)

  zhaoyang-star

  authored
  0650e59

Commits on May 6, 2024

• [Bugfix] Fix asyncio.Task not being subscriptable (#4623)

  DarkLight1337

  authored
  323f27b

• [CI] use ccache actions properly in release workflow (#4629)

  simon-mo

  authored
  e186d37

• [CI] Add retry for agent lost (#4633)

  cadedaniel

  authored
  19cb471

• Update lm-format-enforcer to 0.10.1 (#4631)

  noamgat

  authored
  bd99d22

Commits on May 7, 2024

• [Kernel] Make static FP8 scaling more robust (#4570)

  Show description for a98187c
  pcmoritz

  authored
  a98187c

• [Core][Optimization] change python dict to pytorch tensor (#4607)

  youkaichao

  authored
  63575bc

• [Build/CI] Fixing 'docker run' to re-enable AMD CI tests. (#4642)

  Alexei-V-Ivanov-AMD

  authored
  478aed5

• [Bugfix] Fixed error in slice_lora_b for MergedQKVParallelLinearWithLora (#4609)

  FurtherAI

  authored
  10760da

• [Core][Optimization] change copy-on-write from dict[int, list] to list (#4648)

  youkaichao

  authored
  469f85c

• [Bug fix][Core] fixup ngram not setup correctly (#4551)

  Show description for 8344f77
  

  4 peopleauthored
  8344f77

Commits on May 8, 2024

• [Core][Distributed] support cpu&device in broadcast tensor dict (#4660)

  Show description for cc466a3
  youkaichao

  authored
  cc466a3

• [Core] Optimize sampler get_logprobs (#4594)

  rkooo567

  authored
  d7740ea

• [CI] Make mistral tests pass (#4596)

  rkooo567

  authored
  f6a5930

• [Bugfix][Kernel] allow non-power-of-2 for prefix prefill with alibi (#4573)

  DefTruth

  authored
  0f9a6e3

• [Misc] Add get_name method to attention backends (#4685)

  WoosukKwon

  authored
  5510cf0

• [Core] Faster startup for LoRA enabled models (#4634)

  Yard1

  authored
  ad932a2

• [Core][Optimization] change python dict to pytorch tensor for blocks to swap (#4659)

  youkaichao

  authored
  20cfcde

• [CI/Test] fix swap test for multi gpu (#4689)

  youkaichao

  authored
  230c4b3

• [Misc] Use vllm-flash-attn instead of flash-attn (#4686)

  WoosukKwon

  authored
  89579a2

• [Dynamic Spec Decoding] Auto-disable by the running queue size (#4592)

  Show description for f942efb
  

  comaniac

  and

  cadedaniel

  authored
  f942efb

• [Speculative decoding] [Bugfix] Fix overallocation in ngram + spec logprobs (#4672)

  cadedaniel

  authored
  8b9241b

Commits on May 9, 2024

• [Bugfix] Fine-tune gptq_marlin configs to be more similar to marlin (#4626)

  alexm-neuralmagic

  authored
  e288df0

• [Frontend] add tok/s speed metric to llm class when using tqdm (#4400)

  Show description for 16bc0a0
  

  MahmoudAshraf97

  and

  mgoin

  authored
  16bc0a0

• [Frontend] Move async logic outside of constructor (#4674)

  DarkLight1337

  authored
  f12b20d

• [Misc] Remove unnecessary ModelRunner imports (#4703)

  WoosukKwon

  authored
  190bc83