-
Notifications
You must be signed in to change notification settings - Fork 82
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Adding packedBERT deep dive walkthrough to Tutorials
- Loading branch information
Showing
7 changed files
with
1,790 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
# A deeper dive into faster fine-tuning with Packed BERT | ||
|
||
This notebook is a walkthrough diving deeper into fine-tuning with a more online-capable version of PackedBERT, building on the original Packed BERT used for BERT pre-training. It explores all of the modifications made for preprocessing, dataset creation, the model itself and postprocessing required to implement packing for BERT (and similar models) using an example of sequence classification. The code is broken down and explained step by step, explaining the logic behind the choices made. | ||
|
||
You can try the [notebooks](https://console.paperspace.com/github/gradient-ai/Graphcore-HuggingFace?file=%2Fpacked-bert) that use these functions in Optimum Graphcore in Paperspace. | ||
|
||
Before starting the walkthrough, ensure you have all requirements installed with | ||
|
||
``` | ||
pip install -r requirements.txt | ||
``` |
Binary file added
BIN
+40.7 KB
tutorials/blogs_code/finetuning-packedBERT/images/attention-mask-draw.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+196 KB
tutorials/blogs_code/finetuning-packedBERT/images/packing-creator-example.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+79.4 KB
tutorials/blogs_code/finetuning-packedBERT/images/pooling-draw-edit.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
transformers==4.25.1 | ||
datasets | ||
numpy | ||
matplotlib | ||
torch==1.13.0 |
1,773 changes: 1,773 additions & 0 deletions
1,773
tutorials/blogs_code/finetuning-packedBERT/walkthrough.ipynb
Large diffs are not rendered by default.
Oops, something went wrong.