Code

Fix grad accum when using loss_mask#842

Code

Open

Aphoh wants to merge 5 commits intostanford-crfm:mainstanford-crfm/levanter:mainfrom Aphoh:aphoh/fix-grad-accumAphoh/levanter:aphoh/fix-grad-accumCopy head branch name to clipboard

+198-166

Commits

Commits on Dec 12, 2024

• Fix grad accum

  Aphoh

  committed
  dc6a871

• Also sum reduce in microbatched fn

  Aphoh

  committed
  d8fa2d7

Commits on Dec 17, 2024

• Try moving all the logic into the loss fn

  Aphoh

  committed
  fcf15ab

Commits on Dec 18, 2024

• Switch to sum -> divide

  Aphoh

  committed
  3ae3014

• Fix CI

  Aphoh

  committed
  3e60320