Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pull #1

Open
wants to merge 148 commits into
base: main
Choose a base branch
from
Open

pull #1

wants to merge 148 commits into from

Conversation

codybum
Copy link

@codybum codybum commented Jul 26, 2024

What does this PR do?

Fixes # (issue)

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Was this discussed/approved via a Github issue or the discord / slack channel? Please add a link
    to it if that's the case.
  • Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

arnavgarg1 and others added 30 commits July 3, 2024 15:48
@arnavgarg1
Infer dtype from model config when not explicitly specified (#534)
24cb494
@magdyksaleh
Lorax NER (#531)
a3ad209
@Infernaught
Fix gemma2 (#539)
35f666a
@magdyksaleh
no warm up (#540)
2dd5277
@tgaddair @noyoshi
Preload adapters during init (#543)
5c25e26 
Co-authored-by: Noah Yoshida <[email protected]>
@tgaddair
Move kv cache allocation to router to ensure correct block allocation (…
5a7a1be 
…#545)
@ajtejankar
Fix : compile bug causing models to error with 'lora' key not found (#…
1adc076 
…547)
@tgaddair
Tokenize inputs in router (#548)
452ac73
@tgaddair
Add support for Llama 3 rotary embeddings (#551)
240079b
@tgaddair
Apply chat template in router to properly validate input length (#538)
59631a0
@tgaddair
Fix: short circuit download, load, offload for preloaded adapters (#552)
07addea
@ajtejankar
Update PyTorch, CUDA, vLLM, and Bitsandbytes (#553)
5cefe6e
@ajtejankar
Fix the attention bug caused by upgrading vLLM (#555)
2e81331
@bdalal
Allow eager_prefill to be set in Helm chart (#557)
15a38d5
@tgaddair
Added missing nvidia-ml-py package (#558)
d1a4d09
@ajtejankar @tgaddair
Support FP8 for Mistral (#559)
91ef7a8 
Co-authored-by: Travis Addair <[email protected]>
@magdyksaleh
Move NER output formatting to server (#561)
efa4bff
@ajtejankar
(WIP) Support FP8 for LLaMa (#562)
ea5d74b
@noyoshi
parse headers for errored requests (#564)
3b2cc05
@noyoshi
handle folders for predibase (#565)
e594ce0
@thincal
feat: support lazy loading the lora module for reducing the loading p… (
2e47e77 
#434)
@tgaddair
Fix LM head interaction with Medusa (#567)
8454a82
@noyoshi
bump version (#569)
387e17e
@tgaddair
Fix adapter mask when using speculative decoding + LM head LoRA (#570)
6a3d65b
@magdyksaleh
enable mistral nemo (#568)
da37b5e
@arnavgarg1
Doc updates for Medusa training (#544)
25fd9ca
@noyoshi
include prompt and generated tokens as part of logs (#574)
2e5d97c
@magdyksaleh
add support for classification in bert (#573)
a3f91a6
@magdyksaleh
Support classify batch (#577)
12dc740
@tgaddair
Fix outlines compatibility with speculative decoding (#578)
ca2e643
ajtejankar and others added 30 commits November 27, 2024 12:44
@ajtejankar
Add support for targeting cross_attn layers in mllama (#693)
c96ff88
@magdyksaleh
Make MEMORY_WIGGLE_ROOM a custom arg (#701)
5e0b003
@magdyksaleh
Openai embedding endpoint (#702)
07b77db
@magdyksaleh
fix alerts (#704)
79e4283
@magdyksaleh
fix classification int test (#706)
61b40a8
@andrejTodoroski21
Added Http Status code documentation (#703)
482aa2b
@tgaddair
Remove restriction on using compile + sharding (#707)
63c5eb3
@ajtejankar
Fix decoder_input_details bug (#705)
2af302d
@wvaneaton
Update lorax images (#709)
2aac736
@magdyksaleh
Fix integration tests (#708)
3b25a39 
Co-authored-by: Ubuntu <[email protected]>
@wvaneaton
Update lorax logo (#710)
9a63db1
@magdyksaleh
Fix integration tests2 (#712)
37123f4 
Co-authored-by: Ubuntu <[email protected]>
@wvaneaton
Update index.md (#711)
64c4891
@wvaneaton
Update index.md (#713)
ea6a62a
@ajtejankar
Fix FlashInfer + Medusa bug (#715)
4d7f468
@magdyksaleh
some metrics and logging for batch classify (#714)
e314845
@magdyksaleh
Fix stella model and client <> embedding compatibility (#717)
abf9f39
@magdyksaleh
make tokeziner name required for dev ex (#718)
170aae1
@arnavgarg1
Adds support for the Solar architecture (#719)
ddd1c2d
@magdyksaleh
Memory estimate fixes (#720)
6dfb215
@magdyksaleh
fix and add test (#721)
69bb989
@arnavgarg1
Fix residual summation for the Solar architecture (#723)
52710ea
@magdyksaleh
Better health check (#724)
7dbabda
@arnavgarg1
Support JSON response for /metrics endpoint, add client method for me…
7cd7d06 
…trics (#725)
@arnavgarg1
Add tokenizer-name arg to router launch in developer docs (#727)
0c486c9
@arnavgarg1
Fix lm_head with tied word embeddings for Qwen2 and Qwen2.5 (#728)
eef95f1
@jppgks
IBM Granite support (#730)
269004f
@ajtejankar
Adds support for BGMV kernels needed to run Phi-4 (#731)
667e248
@dependabot @arnavgarg1
Bump jinja2 from 3.1.4 to 3.1.5 in /server (#729)
e36d9c8 
Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Arnav Garg <[email protected]>
@noyoshi
offline mode support (#735)
0f91a7e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.