Code

merge from upstream#19

Code

Merged

l3utterfly merged 121 commits intolayla-buildl3utterfly/llama.cpp:layla-buildfrom mergel3utterfly/llama.cpp:mergeCopy head branch name to clipboardMay 16, 2024

+63,524-44,866

Commits

Commits on May 3, 2024

• llama : rename ctx to user_data in progress_callback (ggerganov#7045)

  Show description for 433def2
  danbev

  authored
  433def2

• convert.py : add python logging instead of print() (ggerganov#6511)

  Show description for a2ac89d
  mofosyne

  authored
  a2ac89d

Commits on May 4, 2024

• tests : add test-tokenizer-0.sh + fix some tokenizers (ggerganov#7036)

  Show description for 92139b9
  ggerganov

  authored
  92139b9

• If first token generated from the server is the stop word the server will crash (ggerganov#7038)

  Show description for 03fb8a0
  maor-ps

  authored
  03fb8a0

• Fix Linux /sys cpu path to guess number of cores (ggerganov#7064)

  viric

  authored
  fcd84a0

• Tidy Android Instructions README.md (ggerganov#7016)

  Show description for cf768b7
  

  Jeximo

  and

  slaren

  authored
  cf768b7

• gguf-split: add --no-tensor-first-split (ggerganov#7072)

  ngxson

  authored
  8425001

Commits on May 5, 2024

• py : logging and flake8 suppression refactoring (ggerganov#7081)

  Show description for 6fbd432
  mofosyne

  authored
  6fbd432

• command-r : add BPE pre-tokenization (ggerganov#7063)

  Show description for 889bdd7
  

  dranger003

  and

  ggerganov

  authored
  889bdd7

• readme : add note that LLaMA 3 is not supported with convert.py (ggerganov#7065)

  lyledean1

  authored
  ca36326

• Disable benchmark on forked repo (ggerganov#7034)

  Show description for 8f8acc8
  CISC

  authored
  8f8acc8

• Adding support for the --numa argument for llama-bench. (ggerganov#7080)

  kunnis

  authored
  628b299

Commits on May 6, 2024

• minor : fix trailing whitespace

  ggerganov

  committed
  bcdee0d

• flake.lock: Update (ggerganov#7079)

  Show description for b3a995b
  

  ggerganov

  and

  github-actions[bot]

  authored
  b3a995b

• Add an option to build without CUDA VMM (ggerganov#7067)

  Show description for 858f6b7
  WilliamTambellini

  authored
  858f6b7

Commits on May 7, 2024

• ci : add GG_BUILD_EXTRA_TESTS_0 env (ggerganov#7098)

  Show description for 947d3ad
  ggerganov

  authored
  947d3ad

• docs: fix typos (ggerganov#7124)

  Show description for 04976db
  omahs

  authored
  04976db

• main : update log text (EOS to EOG) (ggerganov#7104)

  Show description for 3af34c1
  RhinoDevel

  authored
  3af34c1

• readme : update hot topics

  ggerganov

  authored
  53d6c52

• server : update readme with undocumented options (ggerganov#7013)

  K-Mistele

  authored
  260b7c6

• Fix OLMo HF to GGUF conversion (ggerganov#6910)

  nopperl

  authored
  b6aa670

• server: fix incorrectly reported token probabilities (ggerganov#7125)

  Show description for af0a5b6
  JohannesGaessler

  authored
  af0a5b6

Commits on May 8, 2024

• Fixed save_imatrix to match old behaviour for MoE (ggerganov#7099)

  Show description for 48b2f9c
  

  jukofyork

  and

  slaren

  authored
  48b2f9c

• Further tidy on Android instructions README.md (ggerganov#7077)

  Show description for c780e75
  Jeximo

  authored
  c780e75

• metal : fix unused warning

  ggerganov

  committed
  c0e6fbf

• ggml : introduce bfloat16 support (ggerganov#6412)

  Show description for 3855416
  jart

  authored
  3855416

• compare-llama-bench.py: add missing basicConfig (ggerganov#7138)

  Show description for acdce3c
  mofosyne

  authored
  acdce3c

• py : also print the normalizers

  ggerganov

  committed
  7e0b6a7

• convert : add BPE pre-tokenization for DBRX (ggerganov#7132)

  Show description for 4cd621c
  dranger003

  authored
  4cd621c

• clean up json_value & server_log (ggerganov#7142)

  ngxson

  authored
  1fd9c17

• llama : add BPE pre-tokenization for Qwen2 (ggerganov#7114)

  Show description for 229ffff
  

  jklj077

  and

  ggerganov

  authored
  229ffff

• convert.py : --vocab-only generates false but valid params (ggerganov#7027)

  Show description for ad211ed
  20kdc

  authored
  ad211ed

• server : add_special option for tokenize endpoint (ggerganov#7059)

  JohanAR

  authored
  911b390

• sgemm : AVX Q4_0 and Q8_0 (ggerganov#6891)

  Show description for 465263d
  netrunnereve

  authored
  465263d

• main : add --conversation / -cnv flag (ggerganov#7108)

  

  Dawid Potocki

  authored
  83330d8

• metal : use vm_allocate instead of posix_memalign on macOS (ggerganov#7078)

  Show description for 26458af
  giladgd

  authored
  26458af

• server : add themes + favicon (ggerganov#6848)

  Show description for bd1871f
  jboero

  authored
  bd1871f

• Revert "llava : add support for moondream vision language model (ggerganov#6899)"

  Show description for 9da243b
  ggerganov

  committed
  9da243b

• JSON: [key] -> .at(key), assert() -> GGML_ASSERT (ggerganov#7143)

  JohannesGaessler

  authored
  c12452c

• Introduction of CUDA Graphs to LLama.cpp (ggerganov#6766)

  Show description for bc4bba3
  

  agray3

  and

  slaren

  authored
  bc4bba3

• convert-hf : save memory with lazy evaluation (ggerganov#7075)

  Show description for f98eb31
  compilade

  authored
  f98eb31

• cmake : fix typo (ggerganov#7151)

  cebtenzzre

  authored
  4426e29

Commits on May 9, 2024

• TypoFix (ggerganov#7162)

  AhmedZeer

  authored
  07cd41d

• opencl : alignment size converted from bits to bytes (ggerganov#7090)

  Show description for 4734524
  

  albertjin

  and

  cebtenzzre

  authored
  4734524

• gguf-py : add special token modification capability (ggerganov#7166)

  Show description for 2284216
  CISC

  authored
  2284216

• llama : update llama_timings.n_p_eval setting (ggerganov#7160)

  Show description for fd9f92b
  danbev

  authored
  fd9f92b

• Add warning if token is invalid (ggerganov#7173)

  Galunid

  authored
  f31ec12

• CUDA: generalize FP16 fattn vec kernel (ggerganov#7061)

  Show description for a743d76
  JohannesGaessler

  authored
  a743d76

• llama3 custom regex split (ggerganov#6965)

  Show description for 43248e5
  

  4 peopleauthored
  43248e5

• readme : add app (ggerganov#6371)

  Show description for 0961d86
  l3utterfly

  authored
  0961d86

• readme : add scheduled server workflow status badge

  ggerganov

  authored
  d46dbc7

• Vulkan Bugfixes and Improvements (ggerganov#7084)

  Show description for befddd0
  0cc4m

  authored
  befddd0

• eval-callback : fix conversion to float (ggerganov#7184)

  slaren

  authored
  eaf4bd8

Commits on May 10, 2024

• Minor arithmetic improvement to mmvq wrapper kernel (ggerganov#7172)

  OuadiElfarouki

  authored
  8c570c9

• llava : fix moondream support (ggerganov#7163)

  Show description for d11afd6
  abetlen

  authored
  d11afd6

• Main+: optionally allow special tokens from user in interactive mode (ggerganov#7097)

  Show description for f89fe27
  hanishkvc

  authored
  f89fe27

• Fix memory bug in grammar parser (ggerganov#7194)

  Show description for 4e38809
  jart

  authored
  4e38809

• llama : use n_vocab to differentiate between mistral 7B and llama3 8B (ggerganov#7200)

  slaren

  authored
  25c6e82

• convert : print "ignore_merges" field

  ggerganov

  committed
  8c66024

• metal : fix flash attention kernel requirements (ggerganov#7169)

  Show description for 18e4376
  ggerganov

  authored
  18e4376

• llama-bench : add pp+tg test type (ggerganov#7199)

  slaren

  authored
  e849648

Commits on May 11, 2024

• ggml : full ALiBi support (ggerganov#7192)

  Show description for 9cb317f
  ggerganov

  authored
  9cb317f

• llama : add Jina Embeddings architecture (ggerganov#6826)

  Show description for b83cc3f
  

  JoanFM

  and

  ggerganov

  authored
  b83cc3f

• server: fix reported top tokens for temperature 0 (ggerganov#7203)

  JohannesGaessler

  authored
  5ae3426

• llama : lookup word in vocab before doing BPE merges (ggerganov#7193)

  Show description for f99e1e4
  

  3 peopleauthored
  f99e1e4

• server : free llama_batch on exit (ggerganov#7212)

  Show description for 9886313
  stevegrubb

  authored
  9886313

• convert : skip unaccessible HF repos (ggerganov#7210)

  CrispStrobe

  authored
  3292733

• build: fix and ignore msvc warnings (ggml/805)

  

  iboB

  authored and

  ggerganov

  committed
  ef0d5e3

• feat: implemented sigmoid function (ggml/806)

  Show description for f5ef34e
  

  justcho5

  authored and

  ggerganov

  committed
  f5ef34e

• sync : ggml

  Show description for fae9d23
  ggerganov

  committed
  fae9d23

• convert-hf : support bfloat16 conversion (ggerganov#7158)

  Show description for 5a41992
  compilade

  authored
  5a41992

• fix system prompt handling (ggerganov#7153)

  ngxson

  authored
  72c177c

• Scripting & documenting debugging one test without anything else in the loop. (ggerganov#7096)

  Show description for fed0108
  josh-ramer

  authored
  fed0108

• ggml : resolve merge (ggml/0)

  Show description for 325756d
  ggerganov

  committed
  325756d

• metal : fix indent (ggml/0)

  ggerganov

  committed
  6aeff24

• sync : ggml

  ggerganov

  committed
  1622ac0

• metal : fix warnings (skipme) (#0)

  ggerganov

  committed
  7bd4ffb

Commits on May 12, 2024

• remove convert-lora-to-ggml.py (ggerganov#7204)

  slaren

  authored
  b228aba

• cmake : fix version cmp (ggerganov#7227)

  ggerganov

  committed
  6f1b636

• CUDA: add FP32 FlashAttention vector kernel (ggerganov#7188)

  Show description for dc685be
  JohannesGaessler

  authored
  dc685be

Commits on May 13, 2024

• [SYCL] update CI with oneapi 2024.1 (ggerganov#7235)

  Show description for 0d5cef7
  

  arthw

  and

  NeoZhangJianyu

  authored
  0d5cef7

• [SYCL] Add oneapi runtime dll files to win release package (ggerganov#7241)

  Show description for cbf7589
  

  arthw

  and

  NeoZhangJianyu

  authored
  cbf7589

• change default temperature of OAI compat API from 0 to 1 (ggerganov#7226)

  Show description for e586ee4
  Kartoffelsaft

  authored
  e586ee4

• convert.py: Outfile default name change and additional metadata support (ggerganov#4858)

  Show description for b1f8af1
  mofosyne

  authored
  b1f8af1

• llama : rename jina tokenizers to v2 (ggerganov#7249)

  Show description for 9aa6724
  JoanFM

  authored
  9aa6724

• [SYCL] rm wait() (ggerganov#7233)

  arthw

  authored
  948f4ec

• perplexity: add BF16 vs. FP16 results (ggerganov#7150)

  JohannesGaessler

  authored
  1c570d8

• llava-cli: fix base64 prompt (ggerganov#7248)

  Adriankhl

  authored
  30e7033

• llama : less KV padding when FA is off (ggerganov#7257)

  Show description for 614d3b9
  ggerganov

  authored
  614d3b9

• convert-hf : support direct Q8_0 conversion (ggerganov#7234)

  Show description for ee52225
  compilade

  authored
  ee52225

Commits on May 14, 2024

• docs: Fix typo and update description for --embeddings flag (ggerganov#7026)

  Show description for 27f65d6
  louixs

  authored
  27f65d6

• Add left recursion check: quit early instead of going into an infinite loop (ggerganov#7083)

  Show description for e0f5561
  nuchi

  authored
  e0f5561

• move ndk code to a new library (ggerganov#6951)

  eltonkola

  authored
  efc8f76

• llama : disable pipeline parallelism with nkvo (ggerganov#7265)

  slaren

  authored
  5416002

• ggml : add RPC backend (ggerganov#6829)

  Show description for 5e31828
  rgerganov

  authored
  5e31828

• Revert "move ndk code to a new library (ggerganov#6951)" (ggerganov#7282)

  Show description for 1265c67
  mofosyne

  authored
  1265c67

• server: free sampling contexts on exit (ggerganov#7264)

  Show description for 4f02636
  stevegrubb

  authored
  4f02636

• ggml : optimize for ppc64le using VSX intrinsics (ggml/784)

  Show description for 0d26d8c
  

  penghongbo

  and

  ggerganov

  committed
  0d26d8c

• ggml : expose SSE3 and SSSE3 for MSVC when AVX is available (whisper/2128)

  

  przemoc

  authored and

  ggerganov

  committed
  182adef

• ggml : try fix ppc64 (whisper/0)

  ggerganov

  committed
  c3c88f2

• metal : tune soft_max number of threads (whisper/0)

  ggerganov

  committed
  f308ea7

• sync : ggml

  Show description for a5e3fde
  ggerganov

  committed
  a5e3fde

• metal : support FA without mask + add asserts (ggerganov#7278)

  Show description for e8a7fd4
  ggerganov

  authored
  e8a7fd4

• script : sync ggml-rpc

  ggerganov

  committed
  9f77348

Commits on May 15, 2024

• server bench: fix bench not waiting for model load (ggerganov#7284)

  JohannesGaessler

  authored
  583fd6b

• ggml : add ggml_upscale_ext (ggml/814)

  Show description for 48aa8fd
  

  balisujohn

  and

  ggerganov

  committed
  48aa8fd

• sync : ggml

  ggerganov

  committed
  29499bb

• embedding : free the batch after execution (ggerganov#7297)

  dm4

  authored
  ea3b059

• Add missing " (ggerganov#7303)

  AidanBeltonS

  authored
  9a17ab9

• ggml : tag ggml_tensor::backend as deprecated (ggerganov#7290)

  slaren

  authored
  344f912

• Avoid unnecessarily disabling CUDA graphs (ggerganov#7302)

  Show description for dc02098
  agray3

  authored
  dc02098

• ggml : use dynamic thread scheduling for matrix multiplication (ggerganov#6915)

  Show description for e1b40ac
  kunnis

  authored
  e1b40ac

• readme : remove stray double quote (ggerganov#7310)

  Show description for 8f7080b
  danbev

  authored
  8f7080b

Commits on May 16, 2024

• Add support for properly optimized Windows ARM64 builds with LLVM and MSVC (ggerganov#7191)

  Show description for 13ad16a
  

  max-krasnyansky

  and

  ggerganov

  authored
  13ad16a

• ci: fix bin/Release path for windows-arm64 builds (ggerganov#7317)

  Show description for 172b782
  max-krasnyansky

  authored
  172b782

• doc: add references to hugging face GGUF-my-repo quantisation web tool. (ggerganov#7288)

  Show description for ad52d5c
  

  3 peopleauthored
  ad52d5c

• grammar, json, llama: replace push on emplace if it possible (ggerganov#7273)

  GermanAizek

  authored
  0350f58

• convert : get general.name from model dir, not its parent (ggerganov#5615)

  Show description for dda64fc
  

  cebtenzzre

  and

  mofosyne

  authored
  dda64fc

• rpc : add command line arg for specifying backend memory

  Show description for 3b3963c
  rgerganov

  committed
  3b3963c

• rpc : get available mem for the CPU backend

  Show description for 9afdffe
  rgerganov

  committed
  9afdffe

• Merge branch 'layla-build' into merge

  l3utterfly

  authored
  37df164