Code

Autotp training#6922

Code

Open

inkcherry wants to merge 51 commits intomicrosoft:mastermicrosoft/DeepSpeed:masterfrom inkcherry:autotp_traininginkcherry/DeepSpeed:autotp_trainingCopy head branch name to clipboard

+1,461-143

Commits

Commits on Apr 22, 2024

• auto tp training

  inkcherry

  committed
  674a873

Commits on Apr 23, 2024

• update parallel_states

  inkcherry

  committed
  a2e4c47

Commits on Nov 19, 2024

• Merge branch 'master' into HEAD

  inkcherry

  committed
  f4eb142

• WA skips assertions, the loss remains exactly consistent with the low-precision version before the rebase, but the grad norm differs (display issue)

  inkcherry

  committed
  dd081ed

Commits on Nov 22, 2024

• save/load ckpt & save/load hf model basic POC

  inkcherry

  committed
  cdaed2f

Commits on Nov 27, 2024

• finish all the basic functionalities

  inkcherry

  committed
  9aad0e7

Commits on Nov 28, 2024

• update

  inkcherry

  committed
  2bb11fd

Commits on Dec 2, 2024

• use groups for parallel_states

  inkcherry

  committed
  e75c1c2

• enable bwd allreduce, enable scale loss by gas

  inkcherry

  committed
  840a5f2

Commits on Dec 4, 2024

• add dataloader check

  inkcherry

  committed
  60bd6ab

• refactor autoTP step1

  inkcherry

  committed
  9266383

Commits on Dec 5, 2024

• rm parallel_states

  inkcherry

  committed
  07174a9

• refactor autoTP step2

  inkcherry

  committed
  ee6323e

Commits on Dec 10, 2024

• update ut step1

  inkcherry

  committed
  6461b84

Commits on Dec 11, 2024

• update

  inkcherry

  committed
  4d73011

• add uts

  inkcherry

  committed
  c79c3bb

Commits on Dec 12, 2024

• finished all ut code base

  inkcherry

  committed
  97e659c

• addllr scheduler test

  inkcherry

  committed
  a15905b

• refine ut

  inkcherry

  committed
  e9802b0

Commits on Dec 15, 2024

• fix bcast_objlist

  inkcherry

  committed
  88b8acf

• refine layers.py

  inkcherry

  committed
  868be0b

• refine gather

  inkcherry

  committed
  3788e07

Commits on Dec 16, 2024

• pass codegen350M +TP2 ut

  inkcherry

  committed
  27b24f6

• add mode choice

  inkcherry

  committed
  3d7b89f

• fix chatglm

  inkcherry

  committed
  47a6b0b

• fix chatglm2 with transformers=4.40 version

  inkcherry

  committed
  3a23997

• uneven

  inkcherry

  committed
  e3ec46e

• fix uneven

  inkcherry

  committed
  9685879

• fix training

  inkcherry

  committed
  7b99b03

Commits on Dec 17, 2024

• refine code

  inkcherry

  committed
  570645f

• remove skip bcase&reduce

  inkcherry

  committed
  3729b64

• fix typo

  inkcherry

  committed
  62d8858

• format

  inkcherry

  committed
  dd17313

Commits on Dec 18, 2024

• refine code

  inkcherry

  committed
  93cf6f5

• refine code

  inkcherry

  committed
  87c4bc2

• refine

  inkcherry

  committed
  1714bb5

Commits on Dec 19, 2024

• update yuan

  inkcherry

  committed
  dadf915

• optimize usage of move function

  inkcherry

  committed
  86c9399

• refine args usage

  inkcherry

  committed
  2526dc6

• format

  inkcherry

  committed
  c9fd699

• zero1 compatible

  inkcherry

  committed
  797e71f

Commits on Dec 22, 2024

• remove wa

  inkcherry

  committed
  86ae65e

• fix cpu device name

  inkcherry

  committed
  3e40024

Commits on Dec 23, 2024

• fix lm-head

  inkcherry

  committed
  7d94b77

• add detach

  inkcherry

  committed
  b297950

• fix ipex intergration

  inkcherry

  committed
  67ce220

Commits on Dec 25, 2024

• fix tied_embedding

  inkcherry

  committed
  f818be9

Commits on Jan 2, 2025

• Merge remote-tracking branch 'origin/master' into autotp_training

  inkcherry

  committed
  11c98f6

• format

  inkcherry

  committed
  e22b625

Commits on Jan 6, 2025

• Merge branch 'master' into autotp_training

  tjruwase

  authored
  8531b64

• Merge branch 'master' into autotp_training

  loadams

  authored
  8d19e01