Skip to content

Commit

Permalink
Migrate GPT-2 to new tracer (#875)
Browse files Browse the repository at this point in the history
## Description
Migrated GPT-2 example to work with new tracer based pippy.

examples/hf/hf_utils.py contains utility to generate inputs for
HuggingFace models.

Model architecture:
```
GPT2ForSequenceClassification(
  (transformer): GPT2Model(
    (wte): Embedding(50257, 768)
    (wpe): Embedding(1024, 768)
    (drop): Dropout(p=0.1, inplace=False)
    (h): ModuleList(
      (0-11): 12 x GPT2Block(
        (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
        (attn): GPT2Attention(
          (c_attn): Conv1D()
          (c_proj): Conv1D()
          (attn_dropout): Dropout(p=0.1, inplace=False)
          (resid_dropout): Dropout(p=0.1, inplace=False)
        )
        (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
        (mlp): GPT2MLP(
          (c_fc): Conv1D()
          (c_proj): Conv1D()
          (act): NewGELUActivation()
          (dropout): Dropout(p=0.1, inplace=False)
        )
      )
    )
    (ln_f): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
  )
  (score): Linear(in_features=768, out_features=2, bias=False)
)
```

## Run
```
$ torchrun --nproc-per-node 4 pippy_gpt2.py
```

## Output
https://gist.github.com/kwen2501/b9ed6158d8d0dc90b16824aa6abd8d72
  • Loading branch information
kwen2501 authored Nov 30, 2023
1 parent 2146783 commit 92038fb
Show file tree
Hide file tree
Showing 6 changed files with 373 additions and 291 deletions.
49 changes: 0 additions & 49 deletions examples/hf/gpt2/README.md

This file was deleted.

211 changes: 0 additions & 211 deletions examples/hf/gpt2/pippy_gpt2.py

This file was deleted.

20 changes: 0 additions & 20 deletions examples/hf/gpt2/pippy_sbatch.sh

This file was deleted.

11 changes: 0 additions & 11 deletions examples/hf/gpt2/pippy_wrapper.sh

This file was deleted.

Loading

0 comments on commit 92038fb

Please sign in to comment.