This repository contains the materials for the course "Hands-on Transformers: Fine-Tune your own BERT and GPT" on 22. August 2023 by Moritz Laurer for the Data Science Summer School 2023
The summer school session was recorded and the full video is available here: https://ds3.ai/2023/transformer.html
1_open_source_toolkit.ipynb: An overview of accessible hardware (Google Colab) and easy-to-use software (Hugging Face Transformers) to use Transformers without a PhD in deep learning.
2_inside_transformers.ipynb: A high-level breakdown of the main components for using Transformers (models & tokenizer).
3_tune_bert.ipynb: An end-to-end training pipeline for training and evaluating a BERT-base model.
4_tune_bert_nli.ipynb: An end-to-end training pipeline for training and evaluating a BERT-NLI model.
5_data_quality_and_cleaning.ipynb: Illustrations of data quality issues in established datasets and a tool (CleanLab) for automatic data cleaning.
6_annotation_interface_argilla.ipynb: A demo of the free data annotation interface Argilla for creating your own training and test data.
7_tune_generative_llm.ipynb: An end-to-end training pipeline for training and evaluating a generative LLM (FLAN-T5).
While Transformer models like BERT and GPTs are becoming more popular, there is a persistent misconception that they are very complicated to use. This workshop will demonstrate that this is not the case anymore. There are amazing open-source packages like Hugging Face Transformers that enable anyone with some programming knowledge to use, train and evaluate Transformers.
We will start with an intuitive introduction to transfer learning and discuss its added value for social science use-cases as well as limitations. We will then look at the open-source ecosystem and free hardware options to train Transformers. Building upon a high-level explanation of the main components of Transformers in Hugging Face’s implementation, we will then fine-tune different BERT and GPT models and discuss important aspects of fine-tuning and evaluation.
The code demonstrations will be in Python, but participants without prior knowledge of Python or Transformers are explicitly invited to participate. You will leave the workshop with Jupyter notebooks that enable you to train your own Transformer with your own data for your future research projects.
Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0). You are free to copy, change and re-use the code. If you are using larger portions of the codebase, please provide proper attribution. See full license text here.