Code

Gg/flash attn#9

Code

Merged

l3utterfly merged 116 commits intol3utterfly:test-flash-attnl3utterfly/llama.cpp:test-flash-attnfrom ggerganov:gg/flash-attnggerganov/llama.cpp:gg/flash-attnCopy head branch name to clipboardApr 19, 2024

+4,711-362

Commits

Commits on Jan 18, 2024

• ggml : add ggml_flash_attn_ext API

  ggerganov

  committed
  a1c004e

  Copy full SHA for a1c004e

Commits on Jan 19, 2024

• ggml : fix GQA support in ggml_flash_attn_ext

  ggerganov

  committed
  fa7ebcc

  Copy full SHA for fa7ebcc

Commits on Jan 20, 2024

• Merge branch 'master' into gg/flash-attn

  ggerganov

  committed
  c3cdfff

  Copy full SHA for c3cdfff

• ggml : online attention (CPU)

  ggerganov

  committed
  a9681fe

  Copy full SHA for a9681fe

Commits on Jan 21, 2024

• metal : initial implementation

  ggerganov

  committed
  1173f49

  Copy full SHA for 1173f49

• metal : f16 precision

  ggerganov

  committed
  528da75

  Copy full SHA for 528da75

• metal : reduce branches

  ggerganov

  committed
  52ae085

  Copy full SHA for 52ae085

• metal : specialize for head size

  ggerganov

  committed
  b973258

  Copy full SHA for b973258

• wip : 8 rows per simd group

  ggerganov

  committed
  8cde449

  Copy full SHA for 8cde449

• wip : 4 rows per simd group

  ggerganov

  committed
  f31955f

  Copy full SHA for f31955f

• wip : template for rows per warp

  ggerganov

  committed
  a4b6341

  Copy full SHA for a4b6341

• metal : parallelize across KV size

  ggerganov

  committed
  77d08f3

  Copy full SHA for 77d08f3

• metal : parallel reduce across heads

  ggerganov

  committed
  17720fa

  Copy full SHA for 17720fa

Commits on Jan 25, 2024

• metal : efficient flash_attn_f16 implementation

  ggerganov

  committed
  1446a12

  Copy full SHA for 1446a12

• metal : avoid redundant loads of the attention

  ggerganov

  committed
  d917746

  Copy full SHA for d917746

• metal : scale and mask in matrix form

  ggerganov

  committed
  432ad04

  Copy full SHA for 432ad04

• metal : fix comment

  ggerganov

  committed
  40ea8cd

  Copy full SHA for 40ea8cd

• llama : avoid ggml_cast, use F32 query

  ggerganov

  committed
  f9ca5dc

  Copy full SHA for f9ca5dc

• metal : add parallel reduce version (disabled)

  ggerganov

  committed
  6fea843

  Copy full SHA for 6fea843

Commits on Jan 28, 2024

• Merge branch 'master' into gg/flash-attn

  ggerganov

  committed
  b3dd7d9

  Copy full SHA for b3dd7d9

• metal : move output into local memory + optimize

  Show description for 77f6976
  ggerganov

  committed
  77f6976

  Copy full SHA for 77f6976

• metal : add tests, fix scaling, support C > 32

  ggerganov

  committed
  ecc466a

  Copy full SHA for ecc466a

• metal : improve precision

  ggerganov

  committed
  3a428a1

  Copy full SHA for 3a428a1

• ggml : fix f16 mad

  ggerganov

  committed
  8612864

  Copy full SHA for 8612864

• Merge branch 'master' into gg/flash-attn

  ggerganov

  committed
  0ad44ba

  Copy full SHA for 0ad44ba

• metal : minor

  ggerganov

  committed
  134c81c

  Copy full SHA for 134c81c

• metal : support Q > 8

  ggerganov

  committed
  1db22d7

  Copy full SHA for 1db22d7

Commits on Jan 29, 2024

• tests : add ATTN tests

  ggerganov

  committed
  4794821

  Copy full SHA for 4794821

• metal : disable buffer allocation logs

  ggerganov

  committed
  abeaf0d

  Copy full SHA for abeaf0d

• tests : more

  ggerganov

  committed
  c6c1132

  Copy full SHA for c6c1132

• metal : faster inner loop for C == 32

  ggerganov

  committed
  5fcb9c1

  Copy full SHA for 5fcb9c1

Commits on Jan 30, 2024

• metal : fix array initialization

  ggerganov

  committed
  d073e4f

  Copy full SHA for d073e4f

• tests : ifdef

  ggerganov

  committed
  78df552

  Copy full SHA for 78df552

• Merge branch 'master' into gg/flash-attn

  ggerganov

  committed
  3d03bcb

  Copy full SHA for 3d03bcb

Commits on Jan 31, 2024

• Merge branch 'master' into gg/flash-attn

  ggerganov

  committed
  2ddc9bb

  Copy full SHA for 2ddc9bb

• ggml : switch to padded F16 mask for ggml_soft_max, ggml_flash_attn_ext

  ggerganov

  committed
  8ad92dc

  Copy full SHA for 8ad92dc

Commits on Feb 1, 2024

• ggml : fix ggml_soft_max mask requirement

  ggerganov

  committed
  910b15b

  Copy full SHA for 910b15b

• cuda : fix soft_max to use correct mask size

  ggerganov

  committed
  2e46013

  Copy full SHA for 2e46013

• cuda : add flash_attn kernel (wip)

  ggerganov

  committed
  5a19a9f

  Copy full SHA for 5a19a9f

• Merge branch 'master' into gg/flash-attn

  ggerganov

  committed
  41d136b

  Copy full SHA for 41d136b

• metal : optimize softmax for C > 32

  ggerganov

  committed
  56e45a2

  Copy full SHA for 56e45a2

• metal : optimize softmax

  ggerganov

  committed
  cda5a60

  Copy full SHA for cda5a60

• tests : minor fix

  ggerganov

  committed
  c6769b9

  Copy full SHA for c6769b9

• cuda : avoid zeroing fragments

  ggerganov

  committed
  db1f3c4

  Copy full SHA for db1f3c4

Commits on Feb 2, 2024

• tests : update dims

  ggerganov

  committed
  12eaa22

  Copy full SHA for 12eaa22

• cuda : fix __hisinf() result check

  ggerganov

  committed
  b68a112

  Copy full SHA for b68a112

Commits on Feb 3, 2024

• cuda : avoid warp_reduce for smax

  ggerganov

  committed
  b150abe

  Copy full SHA for b150abe

• cuda : use int instead of int64_t

  Show description for 7c34655
  ggerganov

  committed
  7c34655

  Copy full SHA for 7c34655

• cuda : make loops use the same loop values

  Show description for 1f8a592
  ggerganov

  committed
  1f8a592

  Copy full SHA for 1f8a592

• cuda : unroll some of the loops

  ggerganov

  committed
  92472ea

  Copy full SHA for 92472ea

• cuda : avoid __hisinf branches

  ggerganov

  committed
  c51f27c

  Copy full SHA for c51f27c

• cuda : use half2 in softmax

  ggerganov

  committed
  b958151

  Copy full SHA for b958151

• cuda : switch to 1 warp for bs > 16

  ggerganov

  committed
  a7b4715

  Copy full SHA for a7b4715

• cuda : speed-up reduce part of the kernel

  ggerganov

  committed
  3b1c4e7

  Copy full SHA for 3b1c4e7

• cuda : unroll Q*K^T loop

  ggerganov

  committed
  5b263dd

  Copy full SHA for 5b263dd

• cuda : fix -INF block check

  ggerganov

  committed
  e04ff39

  Copy full SHA for e04ff39

• cuda : simplify softmax

  ggerganov

  committed
  cfd9732

  Copy full SHA for cfd9732

• cuda : fix matrix names

  ggerganov

  committed
  ef68fac

  Copy full SHA for ef68fac

Commits on Feb 4, 2024

• cuda : minor

  ggerganov

  committed
  1846e92

  Copy full SHA for 1846e92

Commits on Feb 12, 2024

• Merge branch 'master' into gg/flash-attn

  ggerganov

  committed
  6875997

  Copy full SHA for 6875997

Commits on Feb 19, 2024

• Merge branch 'master' into gg/flash-attn

  ggerganov

  committed
  31109ca

  Copy full SHA for 31109ca

• llama : adapt to F16 KQ_pos

  ggerganov

  committed
  f249c99

  Copy full SHA for f249c99

Commits on Mar 3, 2024

• Merge branch 'master' into gg/flash-attn

  ggerganov

  committed
  02a645e

  Copy full SHA for 02a645e

• llama : adapt new models to F16 KQ_mask

  ggerganov

  committed
  6aefd11

  Copy full SHA for 6aefd11

Commits on Mar 4, 2024

• Merge branch 'master' into gg/flash-attn

  ggerganov

  committed
  e307882

  Copy full SHA for e307882

• ggml : fix F16 store (ARM NEON)

  ggerganov

  committed
  58c7f61

  Copy full SHA for 58c7f61

Commits on Mar 22, 2024

• Merge branch 'master' into gg/flash-attn

  ggerganov

  committed
  9495d39

  Copy full SHA for 9495d39

• llama : fix type of KQ_mask and KQ_pos

  ggerganov

  committed
  3a468e6

  Copy full SHA for 3a468e6

• ggml : fix CPU soft_max

  ggerganov

  committed
  0953212

  Copy full SHA for 0953212

Commits on Mar 24, 2024

• tests : add hs=256

  ggerganov

  committed
  e425810

  Copy full SHA for e425810

Commits on Mar 27, 2024

• Merge branch 'master' into gg/flash-attn

  ggerganov

  committed
  013721d

  Copy full SHA for 013721d

• cuda : fix build

  ggerganov

  committed
  6be02b5

  Copy full SHA for 6be02b5

Commits on Mar 28, 2024

• metal : improve perf via smaller int registers

  ggerganov

  committed
  57c03b7

  Copy full SHA for 57c03b7

• Merge branch 'master' into gg/flash-attn

  ggerganov

  committed
  3e318e7

  Copy full SHA for 3e318e7

• cuda : adapt soft_max to F16 mask and pos

  ggerganov

  committed
  08e69c5

  Copy full SHA for 08e69c5

Commits on Apr 2, 2024

• CUDA: faster FlashAttention, kernel for bs == 1

  

  JohannesGaessler

  authored and

  ggerganov

  committed
  75aa7b4

  Copy full SHA for 75aa7b4

• 16 cols for Phi-2

  

  JohannesGaessler

  authored and

  ggerganov

  committed
  d59ac67

  Copy full SHA for d59ac67

• no vec for hs, no hs==256 ncols==32 for Volta

  

  JohannesGaessler

  authored and

  ggerganov

  committed
  81da919

  Copy full SHA for 81da919

• adjust kernel selection logic

  

  JohannesGaessler

  authored and

  ggerganov

  committed
  269374e

  Copy full SHA for 269374e

• 4 warps, 256 stride for all D

  

  JohannesGaessler

  authored and

  ggerganov

  committed
  cca6d02

  Copy full SHA for cca6d02

• no ncols == 64

  

  JohannesGaessler

  authored and

  ggerganov

  committed
  68d793b

  Copy full SHA for 68d793b

• Multiple parallel blocks for batch size 1

  

  JohannesGaessler

  authored and

  ggerganov

  committed
  3f777ac

  Copy full SHA for 3f777ac

• fix compile warnings

  

  JohannesGaessler

  authored and

  ggerganov

  committed
  e1ecd3b

  Copy full SHA for e1ecd3b

• fix excessive KQ_b loads

  

  JohannesGaessler

  authored and

  ggerganov

  committed
  bb0d51a

  Copy full SHA for bb0d51a

• fix cmake build

  

  JohannesGaessler

  authored and

  ggerganov

  committed
  c63dfdf

  Copy full SHA for c63dfdf

• fix KV cache padding, NaN from INFINITY (#6438)

  JohannesGaessler

  authored
  ee19a4a

  Copy full SHA for ee19a4a

Commits on Apr 5, 2024

• Merge branch 'master' into gg/flash-attn

  ggerganov

  committed
  89961de

  Copy full SHA for 89961de

Commits on Apr 16, 2024

• gguf : add special tokens metadata for FIM/Infill (#6689)

  Show description for 4fbd809
  danbev

  authored
  4fbd809

  Copy full SHA for 4fbd809

• perplexity : require positive --ctx-size arg (#6695)

  ggerganov

  authored
  58227ff

  Copy full SHA for 58227ff

• gritlm : add --outdir option to hf.sh script (#6699)

  Show description for 8a56075
  danbev

  authored
  8a56075

  Copy full SHA for 8a56075

• llama : add qwen2moe (#6074)

  Show description for f4dea7d
  

  simonJJJ

  and

  ggerganov

  authored
  f4dea7d

  Copy full SHA for f4dea7d

• llama : add StableLM2 12B (#6635)

  Show description for dbceec8
  

  ashishdatta

  and

  compilade

  authored
  dbceec8

  Copy full SHA for dbceec8

• ggml : add llamafile sgemm (#6414)

  Show description for 8cc91dc
  jart

  authored
  8cc91dc

  Copy full SHA for 8cc91dc

• ggml : fix llamafile sgemm wdata offsets (#6710)

  Show description for 666867b
  ggerganov

  authored
  666867b

  Copy full SHA for 666867b

• llama : make general.name optional (#6709)

  ggerganov

  authored
  532c173

  Copy full SHA for 532c173

• convert : fix autoawq gemma (#6704)

  Show description for facb8b5
  dengzheng-cloud

  authored
  facb8b5

  Copy full SHA for facb8b5

Commits on Apr 17, 2024

• Merge branch 'master' into gg/flash-attn

  ggerganov

  committed
  2c41180

  Copy full SHA for 2c41180

• llama : flash_attn cparam + fix defrag

  ggerganov

  committed
  599ce84

  Copy full SHA for 599ce84

• server: support flash_attn param

  phymbert

  committed
  4053857

  Copy full SHA for 4053857

• readme : add UI (#6724)

  Show description for 8dd1ec8
  

  yaroslavyaroslav

  and

  ggerganov

  authored
  8dd1ec8

  Copy full SHA for 8dd1ec8

• llamafile : tmp disable + build sgemm.o when needed (#6716)

  Show description for 3b8f1ec
  ggerganov

  authored
  3b8f1ec

  Copy full SHA for 3b8f1ec

• server: bench: enable flash_attn param

  phymbert

  committed
  5668c79

  Copy full SHA for 5668c79

Commits on Apr 18, 2024

• llama : fix compatibility with old 2 expert models (#6735)

  slaren

  authored
  c71bfd7

  Copy full SHA for c71bfd7

• CUDA: refactor host code, dyn. par. blocks

  JohannesGaessler

  committed
  34f93bb

  Copy full SHA for 34f93bb

• fix flash_attn_vec_f16 race condition

  JohannesGaessler

  committed
  6a3b842

  Copy full SHA for 6a3b842

• flush softmax exp below threshold to 0

  JohannesGaessler

  committed
  ef9e159

  Copy full SHA for ef9e159

• store temp KQ in registers

  JohannesGaessler

  committed
  a5b0e2d

  Copy full SHA for a5b0e2d

• Calculate KQ as FP32 if KQV has GGML_PREC_F32

  JohannesGaessler

  committed
  0bc67dd

  Copy full SHA for 0bc67dd

• Add __hgt2_mask implementation for CUDA 11

  JohannesGaessler

  committed
  2f538b9

  Copy full SHA for 2f538b9

• fix KQ FP32 precision fpr parallel_blocks > 1

  JohannesGaessler

  committed
  87968de

  Copy full SHA for 87968de

• llama-bench : add -fa,--flash-attn arg

  ggerganov

  committed
  260cdb2

  Copy full SHA for 260cdb2

• metal : add BS=1 kernel for flash attention (#6508)

  Show description for 105332c
  ggerganov

  authored
  105332c

  Copy full SHA for 105332c

• Qwen2 : assume tied weights if lm_head/output weights is missing (#6738)

  jklj077

  authored
  e11b2e6

  Copy full SHA for e11b2e6

• Merge branch 'master' into gg/flash-attn

  ggerganov

  committed
  fa9e8c6

  Copy full SHA for fa9e8c6

• metal : use F32 attention accumulators

  ggerganov

  committed
  c16a7c2

  Copy full SHA for c16a7c2

• batched-bench : add fattn arg

  ggerganov

  committed
  9ca8698

  Copy full SHA for 9ca8698