Code

Speculative#1308

Code

Merged

OlivierDehaene merged 38 commits intomainhuggingface/text-generation-inference:mainfrom medusa2huggingface/text-generation-inference:medusa2Copy head branch name to clipboardDec 11, 2023

+1,349-282

Commits

Commits on Dec 5, 2023

• Tmp work for medusa.

  Narsil

  committed
  d7d07d4

• Tmp.

  Narsil

  committed
  3fae84d

• Speculative medusa (illegal address Paged).

  Narsil

  committed
  243e9c3

• Speedup 2x.

  Show description for aa442dc
  Narsil

  committed
  aa442dc

• Modifying the protobuf.

  Narsil

  committed
  cda627e

• Non breaking router.

  Narsil

  committed
  b4d97d5

• Medusa + ngram

  Narsil

  committed
  657ccd8

• Working state except all params ??

  Narsil

  committed
  e7e0734

• Speculative decoding + mistral

  Narsil

  committed
  7ed07bc

• Propagate speculate

  Narsil

  committed
  bdd9596

• Updating launcher + docs.

  Narsil

  committed
  1f46bc4

• Need to update params since tensor changed.

  Narsil

  committed
  970e57b

• Needed to regenerate params tests + fix simple tests

  Narsil

  committed
  79f9afb

• Remove pdb comments.

  Narsil

  committed
  d99f281

• Cargo fmt

  Narsil

  committed
  2697920

• Revert falcon load modification.

  Narsil

  committed
  f576598

• C'mon falcon.

  Show description for 8efff84
  Narsil

  committed
  8efff84

• cargo update.

  Narsil

  committed
  4d6efe3

• Working around falcon tests.

  Narsil

  committed
  e808222

• Address comments.

  Narsil

  committed
  be481a4

• Fix.

  Narsil

  committed
  cb8a168

• Fmt.

  Narsil

  committed
  5aa3a01

• Fixing some simple stuff, adding speculate to budget.

  Narsil

  committed
  09839b0

• Fixing infer iterator.

  Narsil

  committed
  9bf31fe

• Fix no speculation.

  Narsil

  committed
  fdef00c

• Cargo fmt.

  Narsil

  committed
  a3cc5a9

Commits on Dec 6, 2023

• Improve create_n_gram degradation.

  Show description for 7b34445
  Narsil

  committed
  7b34445

• Include a few fixes

  Show description for f6958ea
  Narsil

  committed
  f6958ea

• Discard all params modifications, we're not running ngram speculation

  Show description for 6350c11
  Narsil

  committed
  6350c11

• Revert integration tests modifications.

  Narsil

  committed
  b3c1492

• Remove ngram debug code

  Narsil

  committed
  3a8b192

• Updating medusa test + Speeding ngram immensely by just making a smple

  Show description for d2b42f6
  Narsil

  committed
  d2b42f6

• Updated.

  Narsil

  committed
  3a79fbc

• Old llama test.

  Narsil

  committed
  abc8d48

Commits on Dec 8, 2023

• Fixing medusa off by ones.

  Narsil

  committed
  ba16994

• Removing dead code.

  Narsil

  committed
  e95a5a8

Commits on Dec 9, 2023

• Update medusa sampling.

  Narsil

  committed
  b6519b5

Commits on Dec 11, 2023

• Apply suggestions from code review

  Show description for 0006fab
  

  Narsil

  and

  OlivierDehaene

  authored
  0006fab