Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
## Description Migrated GPT-2 example to work with new tracer based pippy. examples/hf/hf_utils.py contains utility to generate inputs for HuggingFace models. Model architecture: ``` GPT2ForSequenceClassification( (transformer): GPT2Model( (wte): Embedding(50257, 768) (wpe): Embedding(1024, 768) (drop): Dropout(p=0.1, inplace=False) (h): ModuleList( (0-11): 12 x GPT2Block( (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.1, inplace=False) ) (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (mlp): GPT2MLP( (c_fc): Conv1D() (c_proj): Conv1D() (act): NewGELUActivation() (dropout): Dropout(p=0.1, inplace=False) ) ) ) (ln_f): LayerNorm((768,), eps=1e-05, elementwise_affine=True) ) (score): Linear(in_features=768, out_features=2, bias=False) ) ``` ## Run ``` $ torchrun --nproc-per-node 4 pippy_gpt2.py ``` ## Output https://gist.github.com/kwen2501/b9ed6158d8d0dc90b16824aa6abd8d72
- Loading branch information