Code

🚨All attention refactor🚨#35235

Code

Merged

Cyrilvallez merged 99 commits intomainhuggingface/transformers:mainfrom all-attention-refactorhuggingface/transformers:all-attention-refactorCopy head branch name to clipboardDec 18, 2024

+5,635-9,778

Commits

Commits on Dec 16, 2024

• refactor LlamaAttention

  

  ArthurZucker

  authored and

  Cyrilvallez

  committed
  79cb53c

• minimal changes

  

  ArthurZucker

  authored and

  Cyrilvallez

  committed
  4bb485b

• fix llama

  

  ArthurZucker

  authored and

  Cyrilvallez

  committed
  f370907

• update

  

  ArthurZucker

  authored and

  Cyrilvallez

  committed
  d3ef539

• modular gemmas

  

  ArthurZucker

  authored and

  Cyrilvallez

  committed
  45eac58

• modular nits

  

  ArthurZucker

  authored and

  Cyrilvallez

  committed
  e52af49

• modular updates

  

  ArthurZucker

  authored and

  Cyrilvallez

  committed
  5ed37ae

• nits

  

  ArthurZucker

  authored and

  Cyrilvallez

  committed
  38cafc1

• simplify

  

  ArthurZucker

  authored and

  Cyrilvallez

  committed
  a862eac

• gpt2

  

  ArthurZucker

  authored and

  Cyrilvallez

  committed
  5639b81

• more modualr and fixes

  

  ArthurZucker

  authored and

  Cyrilvallez

  committed
  452d8ed

• granite

  

  ArthurZucker

  authored and

  Cyrilvallez

  committed
  81a0b66

• modular modular modular

  

  ArthurZucker

  authored and

  Cyrilvallez

  committed
  bc72c3f

• nits

  

  ArthurZucker

  authored and

  Cyrilvallez

  committed
  48caa89

• update

  

  ArthurZucker

  authored and

  Cyrilvallez

  committed
  df68dd0

• qwen2 + starcoder2

  Cyrilvallez

  committed
  0325dc4

• mostly gemma2

  Cyrilvallez

  committed
  ecd814b

• Update image_processing_auto.py

  Cyrilvallez

  committed
  f5fc638

• fix

  Cyrilvallez

  committed
  5e56d9c

• Update modular_starcoder2.py

  Cyrilvallez

  committed
  598b7bb

• fix

  Cyrilvallez

  committed
  0f565fb

• remove all copied from attentions

  Cyrilvallez

  committed
  c9ac84d

• remove gcv

  ArthurZucker

  committed
  d189fe7

• make fix-copies

  ArthurZucker

  committed
  9c83d96

• oups

  ArthurZucker

  committed
  138368e

• oups2.0

  ArthurZucker

  committed
  7225a4f

• fix some modulars + all copied from

  Cyrilvallez

  committed
  a3b9195

• should be good now

  ArthurZucker

  committed
  8d93708

• Merge branch 'all-attention-refactor' of github.com:huggingface/transformers into all-attention-refactor

  Cyrilvallez

  committed
  3cc2b4d

• Merge branch 'all-attention-refactor' of github.com:huggingface/transformers into all-attention-refactor

  Cyrilvallez

  committed
  074e469

• revert unwanted changes

  Cyrilvallez

  committed
  54d9b95

• Update modeling_decision_transformer.py

  Cyrilvallez

  committed
  944e26e

• finish cleanup

  Cyrilvallez

  committed
  911833f

• Update modeling_olmo.py

  Cyrilvallez

  committed
  ea26910

• consistency

  Cyrilvallez

  committed
  bc421af

• re-add gradient checkpointing attribute

  Cyrilvallez

  committed
  8664ddc

• fix

  Cyrilvallez

  committed
  607e928

• style

  Cyrilvallez

  committed
  4612595

• make config necessary

  Cyrilvallez

  committed
  20c376c

• bis

  Cyrilvallez

  committed
  0ac9db2

• bis

  Cyrilvallez

  committed
  349b7ab

• Update modeling_my_new_model2.py

  Cyrilvallez

  committed
  defa88f

• is_causal attr

  Cyrilvallez

  committed
  fbf4b55

• fix

  Cyrilvallez

  committed
  9104d0a

• remove past kv return from decoder layer

  Cyrilvallez

  committed
  0b09340

• fix

  Cyrilvallez

  committed
  46a0df7

• default rope config

  Cyrilvallez

  committed
  aedd88a

• correctly fix rope config

  Cyrilvallez

  committed
  57e9b49

• fix bias

  Cyrilvallez

  committed
  fe90ec0

• fix gpt2 attention output

  Cyrilvallez

  committed
  a3f50d0

• fix test

  Cyrilvallez

  committed
  6a92c70

• fix inits

  Cyrilvallez

  committed
  a28ad19

• fix default sdpa

  Cyrilvallez

  committed
  9bd6c94

• fix default sdpa implementation

  Cyrilvallez

  committed
  fae05e1

• harmonize classes

  Cyrilvallez

  committed
  838d211

• fix mistral

  Cyrilvallez

  committed
  e0d10f6

• fix sliding window models

  Cyrilvallez

  committed
  b275fdc

• mixtral

  Cyrilvallez

  committed
  71eb6a2

• be more explicit

  Cyrilvallez

  committed
  4e25753

• style

  Cyrilvallez

  committed
  1e8712b

• fix

  Cyrilvallez

  committed
  854537b

Commits on Dec 17, 2024

• several fixes

  Cyrilvallez

  committed
  99bddf0

• Update modeling_dbrx.py

  Cyrilvallez

  committed
  2f666b3

• fix test

  Cyrilvallez

  committed
  bafa020

• olmo + phi

  Cyrilvallez

  committed
  00a98e7

• rotary

  Cyrilvallez

  committed
  8c25411

• syle

  Cyrilvallez

  committed
  4bb2f25

• phi

  Cyrilvallez

  committed
  44ff5e3

• phi again

  Cyrilvallez

  committed
  95f7b96

• again

  Cyrilvallez

  committed
  7d55036

• kwargs

  Cyrilvallez

  committed
  24ac9ab

• Update test_modeling_common.py

  Cyrilvallez

  committed
  bd8ede8

• skip fx tracing tests

  Cyrilvallez

  committed
  0d3d3e3

• Update modeling_utils.py

  Cyrilvallez

  committed
  49135d0

• gemma 2

  Cyrilvallez

  committed
  f80a2c3

• again

  Cyrilvallez

  committed
  3e461bd

• Update modeling_recurrent_gemma.py

  Cyrilvallez

  committed
  7a882d5

• gemma2

  Cyrilvallez

  committed
  7870073

• granite

  Cyrilvallez

  committed
  5b4ebaa

• style

  Cyrilvallez

  committed
  7bdf61c

• starcoder

  Cyrilvallez

  committed
  7d5b0b5

• Update sdpa_attention.py

  Cyrilvallez

  committed
  70ef2fd

• switch args

  Cyrilvallez

  committed
  b8429c5

• Update modeling_mllama.py

  Cyrilvallez

  committed
  533657c

• fix

  Cyrilvallez

  committed
  fe20d63

• cache type tests

  Cyrilvallez

  committed
  248a607

• gpt2

  Cyrilvallez

  committed
  4646014

• Update test_modeling_common.py

  Cyrilvallez

  committed
  ad16b1b

• fix

  Cyrilvallez

  committed
  1df6e29

• consistency

  Cyrilvallez

  committed
  6c01005

• fix shape with encoder

  Cyrilvallez

  committed
  f651cd0

• should be the last one

  Cyrilvallez

  committed
  98b7f97

• tests non model

  Cyrilvallez

  committed
  88e2fe5

Commits on Dec 18, 2024

• most comments

  Cyrilvallez

  committed
  5a3bdc4

• small oupsi

  Cyrilvallez

  committed
  f3923b6

• be more explicit in modulars

  Cyrilvallez

  committed
  a6a2ff9

• more explicit modulars

  Cyrilvallez

  committed
  aeea33b

• CIs! it works locally

  Cyrilvallez

  committed
  ec3bef3

• add kwargs to _flash_attention_forward

  Cyrilvallez

  committed
  fc74e39