Code

Add llama.cpp backend #2723

Code

Open

mfuntowicz wants to merge 92 commits intomainhuggingface/text-generation-inference:mainfrom feat-backend-llamacpphuggingface/text-generation-inference:feat-backend-llamacppCopy head branch name to clipboard

+1,934-2

Commits

Commits on Nov 14, 2024

• feat(llamacpp): initial commit

  Show description for aa1fcba
  mfuntowicz

  committed
  aa1fcba

• feat(llamacpp): correctly handle CMAKE_BUILD_TYPE for spdlog macros

  mfuntowicz

  committed
  7d1f8a2

• feat(llamacpp): initial end2end build

  mfuntowicz

  committed
  52d57dc

• misc(cmake): add parameter to build specific cuda arch

  mfuntowicz

  committed
  e4432d3

• misc(cmake): wut

  mfuntowicz

  committed
  fa89d1e

• feat(llamacpp): enable cuda

  mfuntowicz

  committed
  05ad684

• feat(backend): correctly load llama.cpp model from llama api and not gpt2

  mfuntowicz

  committed
  0911076

• feat(backend): tell cmake to build llama-common and link to it

  mfuntowicz

  committed
  098c669

• feat(backend): add some initial decoding steps

  mfuntowicz

  committed
  45d5a6a

• feat(backend): use llama_token as TokenId type

  mfuntowicz

  committed
  92bb113

• feat(backend): minor refactor

  mfuntowicz

  committed
  d4b5be1

• feat(backend): expose frequency and repetition penalties

  mfuntowicz

  committed
  37faeb3

• chore(backend): minor formatting

  mfuntowicz

  committed
  f9c2486

• feat(backend): wip Rust binding

  mfuntowicz

  committed
  355d8a5

• feat(backend): build and link through build.rs

  mfuntowicz

  committed
  e4d803c

• misc(build): handle different lib destination folder lib/lib64

  mfuntowicz

  committed
  f0859c2

• misc(build): refactor build type detection in cmake

  mfuntowicz

  committed
  179309b

• feat(llamacpp): expose number of threads for the backend when constructing the model

  mfuntowicz

  committed
  a316c53

• feat(llamacpp): wip explosion

  mfuntowicz

  committed
  0c1dd0e

• misc(offline): link correctly

  mfuntowicz

  committed
  dbc5b7a

• misc(offline): expose more parameters for generate

  mfuntowicz

  committed
  6115904

• feat(backend): entirely rewrite backend

  mfuntowicz

  committed
  b98c635

• misc(offline): update offline tester

  mfuntowicz

  committed
  6a5f6b0

• feat(backend): full rework of the backend internal to safer c++

  mfuntowicz

  committed
  d52b4c4

• misc(offline): match rework

  mfuntowicz

  committed
  3af2c68

• feat(backend): add mapping for ignore_eos_token stopping criteria

  mfuntowicz

  committed
  f39edc7

• feat(backend): add logit parameter in the callback fn

  mfuntowicz

  committed
  d4aee42

• feat(backend): bind incoming request to the server

  mfuntowicz

  committed
  612f2f9

• feat(backend): avoid dropping the boxed stream at the end of the callback

  mfuntowicz

  committed
  b50dcdd

• feat(backend): somewhat generates the final infer response

  mfuntowicz

  committed
  3e82f14

• feat(backend): fix invalid reference to ctx instead of context in release build

  mfuntowicz

  committed
  bd8f0f1

• feat(backend): correctly link to shared fmt and spdlog instead of static

  mfuntowicz

  committed
  2cdfed9

• chore: unsued variables

  mfuntowicz

  committed
  86a2ae6

• feat(backend): fix memory leaking on llama_sampler when the decode ends

  mfuntowicz

  committed
  7b0a56f

• feat(backend): remove static from inner_fw visitor as it leads to invalid memory locations

  mfuntowicz

  committed
  31d9254

• misc(lint): make clippy happier

  mfuntowicz

  committed
  188442f

• feat(backend): add number of generated tokens in the callback

  mfuntowicz

  committed
  05ff551

• feat(backend): correctly handle the max_new_tokens case for is_eos

  mfuntowicz

  committed
  06424aa

• feat(backend): make eog clearer on c++ side

  mfuntowicz

  committed
  11c593d

• feat(backend): refactor the callback to handle intermediate and end inference message

  mfuntowicz

  committed
  5b7a951

• misc(ffi): remove unused ffi mapping

  mfuntowicz

  committed
  958c72a

• feat(backend): add early stopping criteria from TGI stream callback

  mfuntowicz

  committed
  1473259

• feat(backend): expose tokenizer to the GenerationContext to decode token

  mfuntowicz

  committed
  1149186

• misc(backend): decrease log verbosity in callback

  mfuntowicz

  committed
  52208f5

• misc(cmake): use url deps and not git repo

  mfuntowicz

  committed
  62dba1a

• misc(backend): missing header <variant>

  mfuntowicz

  committed
  5884218

• feat(backend): avoid copy constructor

  mfuntowicz

  committed
  a1154b1

• chore(backend): minor fixes mostly format

  mfuntowicz

  committed
  7eec0f7

• feat(backend): dockerfile

  mfuntowicz

  committed
  a7afde4

• feat(dockerfile): build process

  mfuntowicz

  committed
  2065282

• feat(backend): handle all the tokenization failure and send back to the client

  mfuntowicz

  committed
  26d0266

• misc(cmake): remove dependency on fmt

  mfuntowicz

  committed
  cf17928

• misc(cmake): use URL base llama.cpp repo

  mfuntowicz

  committed
  4f5397c

• feat(backend): simplify overall cpp structure

  mfuntowicz

  committed
  86d30ae

• feat(backend): remove reinterpret_cast converting from uint32_t to llama_token(int32_t)

  mfuntowicz

  committed
  6915fa3

• feat(backend): remove unused function

  mfuntowicz

  committed
  7e2890f

• feat(backend): fix invalid reference to context in release mode

  mfuntowicz

  committed
  488ba93

• feat(backend): use std::ranges to map uint32_t to llama_token

  mfuntowicz

  committed
  363d5e4

• chore(backend): minor improvements

  mfuntowicz

  committed
  02cd6fe

• dockerfile(backend): initial working version of llama.cpp container

  mfuntowicz

  committed
  daf1631

• feat(backend): simplify Rust callback

  mfuntowicz

  committed
  57b2154

• feat(backend): wrap Arc tokenizer to avoid duplicating

  mfuntowicz

  committed
  6f059c4

• feat(backend): update llamacpp to 4077

  mfuntowicz

  committed
  70c90ad

• misc(build): improve build process

  mfuntowicz

  committed
  23d2bcf

Commits on Nov 20, 2024

• feat(backend): multistream inference on CPU

  mfuntowicz

  committed
  5335bf9

Commits on Nov 21, 2024

• feat(backend): bind thread and memory affinity for thread

  mfuntowicz

  committed
  50c3766

• feat(backend): correctly setup llama_context providing n_threads and n_ubatch

  mfuntowicz

  committed
  84eead2

Commits on Nov 22, 2024

• feat(backend): rely on multi consumer queue to scheduler workers

  mfuntowicz

  committed
  5a85661

• misc(docker): add numa lib as dependency

  mfuntowicz

  committed
  30ae996

• misc(backend): allow rebinding numa core affinity

  mfuntowicz

  committed
  2d9465d

• misc(license): update LICENSE

  mfuntowicz

  committed
  4ee2ee5

• misc(doc): c++ documentation

  mfuntowicz

  committed
  b9c04b9

• misc(doc): rust documentation

  mfuntowicz

  committed
  862a519

• chore: remove unrelated change to trtllm

  mfuntowicz

  committed
  9025a26

Commits on Nov 28, 2024

• Update Dockerfile.llamacpp as per review

  Show description for bbe95ca
  

  mfuntowicz

  and

  Hugoch

  authored
  bbe95ca

• Update Dockerfile.llamacpp as per review

  Show description for d918e6a
  

  mfuntowicz

  and

  Hugoch

  authored
  d918e6a

• feat(backend): remove core overriding in the Rust backend

  mfuntowicz

  committed
  274cfce

• feat(backend): use the new batch api from llama

  mfuntowicz

  committed
  8e89793

• feat(backend): fix when num_cores_per_instance is equals to zero with the size of the generated core allocation

  mfuntowicz

  committed
  298367c

• feat(backend): add some test to the backend for core allocation

  mfuntowicz

  committed
  929a2fc

• feat(backend): add guard in case top_k = 0

  mfuntowicz

  committed
  df72c56

• feat(backend): add missing temperature parameter

  mfuntowicz

  committed
  9d659f1

• misc(offline): update model creation as std::shared_ptr

  mfuntowicz

  committed
  6c5a75b

• feat(backend): update llama.cpp to 4215

  mfuntowicz

  committed
  b1ebc8f

• feat(backend): create llama_context_params with default factory

  mfuntowicz

  committed
  dc6435e

• feat(backend): use new batch API to generate tokens

  mfuntowicz

  committed
  b10eaab

• feat: Fix Cmakelist to allow building on Darwin platform (#2785)

  Show description for 59b0ef3
  Hugoch

  authored
  59b0ef3

Commits on Nov 29, 2024

• feat(backend): correctly link to all libraries

  mfuntowicz

  committed
  f5c4cee

• feat(backend): add mimalloc memory allocator to the container

  mfuntowicz

  committed
  db41776

• feat(backend): better map exception throw on C++ side

  mfuntowicz

  committed
  c9f6c3a

• feat(backend): use c++ defined types for llama.cpp

  mfuntowicz

  committed
  e0dda9b

Commits on Dec 12, 2024

• misc: use return Ok(())

  Show description for 182ffaf
  

  mfuntowicz

  and

  co42

  authored
  182ffaf