Code

Add Zamba2#34517

Code

Draft

pglorio wants to merge 56 commits intohuggingface:mainhuggingface/transformers:mainfrom Zyphra:zamba2Zyphra/transformers_zamba:zamba2Copy head branch name to clipboard

+5,207-39

Commits

Commits on Oct 24, 2024

• First commit

  pglorio

  committed
  acd25b7

Commits on Oct 28, 2024

• Finish model implementation

  pglorio

  committed
  70639b8

Commits on Oct 29, 2024

• First commit

  pglorio

  committed
  d111b98

• Finish model implementation

  pglorio

  committed
  8f36dba

• Merge branch 'zamba2' of https://github.com/Zyphra/transformers_zamba into zamba2

  pglorio

  committed
  f0c547c

Commits on Oct 30, 2024

• Register zamba2

  pglorio

  committed
  700fbf0

Commits on Nov 4, 2024

• generated modeling and configuration

  pglorio

  committed
  70a6021

Commits on Nov 5, 2024

• Merge pull request #2 from Zyphra/main

  Show description for 88c4b26
  pglorio

  authored
  88c4b26

• generated modeling and configuration

  pglorio

  committed
  685906a

• added hybrid cache

  pglorio

  committed
  4da8d5f

• fix attention_mask in mamba

  pglorio

  committed
  6b5a9be

• dropped unused loras

  pglorio

  committed
  248350d

• fix flash2

  pglorio

  committed
  d1d2c66

• Merge pull request #3 from Zyphra/main

  Show description for eb6063e
  pglorio

  authored
  eb6063e

Commits on Nov 6, 2024

• config docstrings

  

  root

  committed
  5f5d01e

Commits on Nov 7, 2024

• fix config and fwd pass

  pglorio

  committed
  c1b7647

• make fixup fixes

  pglorio

  committed
  979b99b

Commits on Nov 9, 2024

• text_modeling_zamba2

  pglorio

  committed
  9d9b2eb

• Merge pull request #4 from Zyphra/main

  Show description for 3a457f5
  pglorio

  authored
  3a457f5

• small fixes

  pglorio

  committed
  549d4cb

Commits on Nov 11, 2024

• make fixup fixes

  pglorio

  committed
  987bba9

• Merge pull request #5 from Zyphra/main

  Show description for ffc2a58
  pglorio

  authored
  ffc2a58

• Fix modular model converter

  pglorio

  committed
  9adf85e

Commits on Nov 19, 2024

• added inheritances in modular, renamed zamba cache

  pglorio

  committed
  904da4e

• Merge pull request #6 from Zyphra/main

  Show description for 4725983
  pglorio

  authored
  4725983

• modular rebase

  pglorio

  committed
  0be27d7

• Rebase

  pglorio

  committed
  cc0c549

Commits on Nov 20, 2024

• new modular conversion

  pglorio

  committed
  ac77a09

• fix generated modeling file

  pglorio

  committed
  e59980e

• fixed import for Zamba2RMSNormGated

  pglorio

  committed
  73a647a

Commits on Nov 21, 2024

• modular file cleanup

  pglorio

  committed
  c2b72a5

• rebase

  pglorio

  committed
  0eb39a5

• make fixup and model tests

  pglorio

  committed
  10a0b1e

Commits on Nov 23, 2024

• dropped inheritance for Zamba2PreTrainedModel

  pglorio

  committed
  0270667

• make fixup and unit tests

  pglorio

  committed
  189c8c5

Commits on Dec 5, 2024

• Add inheritance of rope from GemmaRotaryEmbedding

  pglorio

  committed
  fa5f79e

• moved rope to model init

  pglorio

  committed
  8079ae0

• drop del self.self_attn and del self.feed_forward

  pglorio

  committed
  d6206eb

• Rebase onto upstream

  pglorio

  committed
  f832699

• fix tests

  pglorio

  committed
  cf613b7

Commits on Dec 7, 2024

• renamed lora -> adapter

  pglorio

  committed
  337faed

• rewrote adapter implementation

  pglorio

  committed
  f1b31a1

• rebase

  pglorio

  committed
  8925c15

• fixed tests

  pglorio

  committed
  11fdd47

Commits on Dec 18, 2024

• Merge branch 'main' into zamba2

  pglorio

  committed
  02dd042

Commits on Dec 19, 2024

• Fix torch_forward in mamba2 layer

  pglorio

  committed
  5d0a5d4

• Fix torch_forward in mamba2 layer

  pglorio

  committed
  ef055c9

• Fix torch_forward in mamba2 layer

  pglorio

  committed
  b993a78

• Dropped adapter in-place sum

  pglorio

  committed
  bf93251

• removed rope from attention init

  pglorio

  committed
  99708af

• updated rope

  pglorio

  committed
  d9b4a50

• created get_layers method

  pglorio

  committed
  095d853

Commits on Dec 20, 2024

• rebase

  pglorio

  committed
  10ebad5

• make fixup fix

  pglorio

  committed
  99e343e

• make fixup fixes

  pglorio

  committed
  4e40975

• make fixup fixes

  pglorio

  committed
  61bb32f