Code

Sync with [email protected]#103

Code

Merged

dtrifiro merged 99 commits intoopendatahub-io:mainopendatahub-io/vllm:mainfrom dtrifiro:sync-with-upstreamdtrifiro/vllm:sync-with-upstreamCopy head branch name to clipboardJul 23, 2024

+11,844-3,885

Commits

Commits on Jul 15, 2024

• [Docs] Add Google Cloud to sponsor list (vllm-project#6450)

  WoosukKwon

  authored
  3dee97b

• [Misc] Add CustomOp Interface to UnquantizedFusedMoEMethod (vllm-project#6289)

  WoosukKwon

  authored
  ec9933f

• [CI/Build][TPU] Add TPU CI test (vllm-project#6277)

  Show description for 4552e37
  

  WoosukKwon

  and

  khluu

  authored
  4552e37

Commits on Jul 16, 2024

• Pin sphinx-argparse version (vllm-project#6453)

  Show description for d6f3b3d
  khluu

  authored
  d6f3b3d

• [BugFix][Model] Jamba - Handle aborted requests, Add tests and fix cleanup bug (vllm-project#6425)

  Show description for 9ad32da
  

  mzusman

  and

  Mor Zusman

  authored
  9ad32da

• [Bugfix][CI/Build] Test prompt adapters in openai entrypoint tests (vllm-project#6419)

  g-eoj

  authored
  d92b3c5

• [Docs] Announce 5th meetup (vllm-project#6458)

  WoosukKwon

  authored
  37d7766

• [CI/Build] vLLM cache directory for images (vllm-project#6444)

  DarkLight1337

  authored
  d970115

• [Frontend] Support for chat completions input in the tokenize endpoint (vllm-project#5923)

  sasha0552

  authored
  7a3d2a5

• [Misc] Fix typos in spec. decode metrics logging. (vllm-project#6470)

  Show description for 7508a3d
  tdoublep

  authored
  7508a3d

• [Core] Use numpy to speed up padded token processing (vllm-project#6442)

  peng1999

  authored
  2bb0489

• [CI/Build] Remove "boardwalk" image asset (vllm-project#6460)

  DarkLight1337

  authored
  38ef948

• [doc][misc] remind to cancel debugging environment variables (vllm-project#6481)

  Show description for 9f4ccec
  youkaichao

  authored
  9f4ccec

• [Hardware][TPU] Support MoE with Pallas GMM kernel (vllm-project#6457)

  WoosukKwon

  authored
  c467dff

• [Doc] Fix the lora adapter path in server startup script (vllm-project#6230)

  Jeffwan

  authored
  94162be

• [Misc] Log spec decode metrics (vllm-project#6454)

  comaniac

  authored
  160e1d8

• [Kernel][Attention] Separate Attention.kv_scale into k_scale and v_scale (vllm-project#6081)

  mgoin

  authored
  978aed5

• [ci][distributed] add pipeline parallel correctness test (vllm-project#6410)

  youkaichao

  authored
  09c2eb8

Commits on Jul 17, 2024

• [misc][distributed] improve tests (vllm-project#6488)

  youkaichao

  authored
  7f62077

• [misc][distributed] add seed to dummy weights (vllm-project#6491)

  youkaichao

  authored
  ce37be7

• [Distributed][PP] only create embedding & lm head when necessary (vllm-project#6455)

  Show description for 1d094fd
  wushidonguc

  authored
  1d094fd

• [ROCm] Cleanup Dockerfile and remove outdated patch (vllm-project#6482)

  hongxiayang

  authored
  1038388

• [Misc][Speculative decoding] Typos and typing fixes (vllm-project#6467)

  Show description for a19e8d3
  ShangmingCai

  authored
  a19e8d3

• [Doc][CI/Build] Update docs and tests to use vllm serve (vllm-project#6431)

  DarkLight1337

  authored
  5bf35a9

• [Bugfix] Fix for multinode crash on 4 PP (vllm-project#6495)

  Show description for 5fa6e98
  andoorve

  authored
  5fa6e98

• [TPU] Remove multi-modal args in TPU backend (vllm-project#6504)

  WoosukKwon

  authored
  e09ce75

• [Misc] Use torch.Tensor for type annotation (vllm-project#6505)

  WoosukKwon

  authored
  a9a2e74

• [Core] Refactor _prepare_model_input_tensors - take 2 (vllm-project#6164)

  comaniac

  authored
  2fa4623

• [DOC] - Add docker image to Cerebrium Integration (vllm-project#6510)

  milo157

  authored
  a38524f

• [Bugfix] Fix Ray Metrics API usage (vllm-project#6354)

  Yard1

  authored
  5f0b993

• [Core] draft_model_runner: Implement prepare_inputs on GPU for advance_step (vllm-project#6338)

  alexm-neuralmagic

  authored
  e76466d

Commits on Jul 18, 2024

• [ Kernel ] FP8 Dynamic-Per-Token Quant Kernel (vllm-project#6511)

  Show description for b5241e4
  

  varun-sundar-rabindranath

  and

  Varun Sundar Rabindranath

  authored
  b5241e4

• [Model] Pipeline parallel support for Mixtral (vllm-project#6516)

  comaniac

  authored
  b5af8c2

• [ Kernel ] Fp8 Channelwise Weight Support (vllm-project#6487)

  robertgshaw2-neuralmagic

  authored
  18fecc3

• [core][model] yet another cpu offload implementation (vllm-project#6496)

  Show description for 1c27d25
  

  youkaichao

  and

  mgoin

  authored
  1c27d25

• [BugFix] Avoid secondary error in ShmRingBuffer destructor (vllm-project#6530)

  njhill

  authored
  d25877d

• [Core] Introduce SPMD worker execution using Ray accelerated DAG (vllm-project#6032)

  Show description for 61e5927
  

  ruisearch42

  and

  stephanie-wang

  authored
  61e5927

• [Misc] Minor patch for draft model runner (vllm-project#6523)

  comaniac

  authored
  8a74c68

• [BugFix][Frontend] Use LoRA tokenizer in OpenAI APIs (vllm-project#6227)

  Show description for e2fbaee
  

  njhill

  and

  DarkLight1337

  authored
  e2fbaee

• [Bugfix] Update flashinfer.py with PagedAttention forwards - Fixes Gemma2 OpenAI Server Crash (vllm-project#6501)

  noamgat

  authored
  c8a7d51

• [TPU] Refactor TPU worker & model runner (vllm-project#6506)

  WoosukKwon

  authored
  4634c87

• [ Misc ] Improve Min Capability Checking in compressed-tensors (vllm-project#6522)

  robertgshaw2-neuralmagic

  authored
  58ca663

• [ci] Reword Github bot comment (vllm-project#6534)

  khluu

  authored
  ecdb462

• [Model] Support Mistral-Nemo (vllm-project#6548)

  mgoin

  authored
  15c6a07

• Fix PR comment bot (vllm-project#6554)

  Show description for 2d4733b
  khluu

  authored
  2d4733b

• [ci][test] add correctness test for cpu offloading (vllm-project#6549)

  youkaichao

  authored
  f53b8f0

• [Kernel] Implement fallback for FP8 channelwise using torch._scaled_mm (vllm-project#6552)

  tlrmchlsmth

  authored
  4ffffcc

Commits on Jul 19, 2024

• [CI/Build] Build on Ubuntu 20.04 instead of 22.04 (vllm-project#6517)

  tlrmchlsmth

  authored
  1689219

• Add support for a rope extension method (vllm-project#6553)

  simon-mo

  authored
  c5df56f

• [Core] Multiprocessing Pipeline Parallel support (vllm-project#6130)

  Show description for b5672a1
  

  njhill

  and

  andoorve

  authored
  b5672a1

• [Bugfix] Make spec. decode respect per-request seed. (vllm-project#6034)

  Show description for d4201e0
  

  tdoublep

  and

  njhill

  authored
  d4201e0

• [ Misc ] non-uniform quantization via compressed-tensors for Llama (vllm-project#6515)

  robertgshaw2-neuralmagic

  authored
  dbe5588

• [Bugfix][Frontend] Fix missing /metrics endpoint (vllm-project#6463)

  DarkLight1337

  authored
  6366efc

• [BUGFIX] Raise an error for no draft token case when draft_tp>1 (vllm-project#6369)

  wooyeonlee0

  authored
  a921e86

• [Model] RowParallelLinear: pass bias to quant_method.apply (vllm-project#6327)

  Show description for a5314e8
  tdoublep

  authored
  a5314e8

• [Bugfix][Frontend] remove duplicate init logger (vllm-project#6581)

  dtrifiro

  authored
  51f8aa9

• [Misc] Small perf improvements (vllm-project#6520)

  Yard1

  authored
  9ed82e7

• [Docs] Update docs for wheel location (vllm-project#6580)

  simon-mo

  authored
  30efe41

• [Bugfix] [SpecDecode] AsyncMetricsCollector: update time since last collection (vllm-project#6578)

  tdoublep

  authored
  f0bbfaf

• [bugfix][distributed] fix multi-node bug for shared memory (vllm-project#6597)

  youkaichao

  authored
  07eb6f1

• [ Kernel ] Enable Dynamic Per Token fp8 (vllm-project#6547)

  robertgshaw2-neuralmagic

  authored
  4cc24f0

• [Docs] Update PP docs (vllm-project#6598)

  andoorve

  authored
  45ceb85

Commits on Jul 20, 2024

• [build] add ib in image for out-of-the-box infiniband support (vllm-project#6599)

  Show description for e81522e
  youkaichao

  authored
  e81522e

• [ Kernel ] FP8 Dynamic Per Token Quant - Add scale_ub (vllm-project#6593)

  Show description for 2e26564
  

  varun-sundar-rabindranath

  and

  Varun Sundar Rabindranth

  authored
  2e26564

• [Core] Allow specifying custom Executor (vllm-project#6557)

  Yard1

  authored
  7bd8200

• Pipeline Parallel: Guard for KeyErrors at request abort (vllm-project#6587)

  Show description for 3f8d42c
  tjohnson31415

  authored
  3f8d42c

• [Misc] Consolidate and optimize logic for building padded tensors (vllm-project#6541)

  DarkLight1337

  authored
  9042d68

• [ Misc ] fbgemm checkpoints (vllm-project#6559)

  robertgshaw2-neuralmagic

  authored
  683e3cb

• [Bugfix][CI/Build][Hardware][AMD] Fix AMD tests, add HF cache, update CK FA, add partially supported model notes (vllm-project#6543)

  mawong-amd

  authored
  06d6c5f

• [ Kernel ] Enable fp8-marlin for fbgemm-fp8 models (vllm-project#6606)

  robertgshaw2-neuralmagic

  authored
  9364f74

• [Misc] Fix input_scale typing in w8a8_utils.py (vllm-project#6579)

  mgoin

  authored
  f952bbc

• [ Bugfix ] Fix AutoFP8 fp8 marlin (vllm-project#6609)

  robertgshaw2-neuralmagic

  authored
  082ecd8

Commits on Jul 21, 2024

• [Frontend] Move chat utils (vllm-project#6602)

  Show description for d7f4178
  

  DarkLight1337

  and

  ywang96

  authored
  d7f4178

• [Spec Decode] Disable Log Prob serialization to CPU for spec decoding for both draft and target models. (vllm-project#6485)

  sroy745

  authored
  14f91fe

• [Misc] Remove abused noqa (vllm-project#6619)

  WoosukKwon

  authored
  b6df37f

• [Model] Refactor and decouple phi3v image embedding (vllm-project#6621)

  Isotr0py

  authored
  25e778a

• [Kernel][Core] Add AWQ support to the Marlin kernel (vllm-project#6612)

  alexm-neuralmagic

  authored
  396d92d

Commits on Jul 22, 2024

• [Model] Initial Support for Chameleon (vllm-project#5770)

  ywang96

  authored
  c9eef37

• [Misc] Add a wrapper for torch.inference_mode (vllm-project#6618)

  WoosukKwon

  authored
  42de2ce

• [Bugfix] Fix vocab_size field access in llava_next.py (vllm-project#6624)

  jaywonchung

  authored
  89c1c6a

• [Frontend] Refactor prompt processing (vllm-project#4028)

  Show description for 739b61a
  

  DarkLight1337

  and

  ywang96

  authored
  739b61a

• [Bugfix][Kernel] Use int64_t for indices in fp8 quant kernels (vllm-project#6649)

  tlrmchlsmth

  authored
  fea59c7

• [ci] Use different sccache bucket for CUDA 11.8 wheel build (vllm-project#6656)

  Show description for 69d5ae3
  khluu

  authored
  69d5ae3

• [Core] Support dynamically loading Lora adapter from HuggingFace (vllm-project#6234)

  Show description for 42c7f66
  

  Jeffwan

  and

  Yard1

  authored
  42c7f66

• [ci][build] add back vim in docker (vllm-project#6661)

  youkaichao

  authored
  5a96ee5

Commits on Jul 23, 2024

• [Misc] Remove deprecation warning for beam search (vllm-project#6659)

  WoosukKwon

  authored
  bdf5fd1

• [Core] Modulize prepare input and attention metadata builder (vllm-project#6596)

  comaniac

  authored
  e0c1575

• [Bugfix] Fix null modules_to_not_convert in FBGEMM Fp8 quantization (vllm-project#6665)

  cli99

  authored
  c5e8330

• [Misc] Enable chunked prefill by default for long context models (vllm-project#6666)

  WoosukKwon

  authored
  729171a

• [misc] add start loading models for users information (vllm-project#6670)

  youkaichao

  authored
  7c2749a

• add tqdm when loading checkpoint shards (vllm-project#6569)

  Show description for e519ae0
  

  3 peopleauthored
  e519ae0

• [Misc] Support FP8 kv cache scales from compressed-tensors (vllm-project#6528)

  mgoin

  authored
  9e0b558

• [doc][distributed] doc for setting up multi-node environment (vllm-project#6529)

  Show description for c051bfe
  youkaichao

  authored
  c051bfe

• [Misc] Manage HTTP connections in one place (vllm-project#6600)

  DarkLight1337

  authored
  97234be

• [misc] only tqdm for first rank (vllm-project#6672)

  youkaichao

  authored
  c520124

• [VLM][Model] Support image input for Chameleon (vllm-project#6633)

  ywang96

  authored
  22fa2e3

• support ignore patterns in model loader (vllm-project#6673)

  simon-mo

  authored
  3eda4ec

• Bump version to v0.5.3 (vllm-project#6674)

  simon-mo

  authored
  bb2fc08

• Sync with [email protected]

  Show description for c2ef5a8
  dtrifiro

  committed
  c2ef5a8