Code

Modernize MosaicBERT#440

Code

Open

Skylion007 wants to merge 19 commits intomosaicml:mainmosaicml/examples:mainfrom Skylion007:skylion007/add-fa2-to-bertSkylion007/mosaicml-examples:skylion007/add-fa2-to-bertCopy head branch name to clipboard

+648-474

Commits

Commits on Jan 2, 2024

• Modernize MosaicBERT

  Skylion007

  committed
  b809a7b

• Reviewer suggestions & import refactor

  Skylion007

  committed
  dbe8d64

Commits on Jan 4, 2024

• Fix broken tests

  Skylion007

  committed
  6a8d85b

• Cast to bfloat16 if possible

  Skylion007

  committed
  4cbdec1

• Update streaming kwarg

  Skylion007

  committed
  6ecb798

• Improve HALF_DTYPE selection

  Skylion007

  committed
  eb93f3b

• revert back to old HALF selection

  Skylion007

  committed
  58d9ffe

Commits on Jan 5, 2024

• update documentation

  jacobfulano

  committed
  c6d58e4

• clean up documentation

  jacobfulano

  committed
  d5ec672

• update yamls to mosaicml/pytorch:2.1.1_cu121-python3.10-ubuntu20.04

  jacobfulano

  committed
  bcd10d2

• clean up comments and documentation

  jacobfulano

  committed
  e9e1213

• linting

  jacobfulano

  committed
  ea82b1b

• small formatting for pre-commit hook

  jacobfulano

  committed
  26313a5

• linting and type: ignore

  jacobfulano

  committed
  329ed51

• linting

  jacobfulano

  committed
  95a4753

• type ignore

  jacobfulano

  committed
  28ccb4f

• yapf precommit hook

  jacobfulano

  committed
  2cdb6af

• remove bug for self.slopes

  jacobfulano

  committed
  cb78d90

Commits on Jan 8, 2024

• remove fused_layernorm with apex dependency and include low_precision_layernorm

  jacobfulano

  committed
  c44effa