main_notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# System Project Notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tabel of Contents\n",
    "1. **Overview**\n",
    "\n",
    "2. **Decomposition and Domain Expertise**\n",
    "   \n",
    "3. **Data**\n",
    "* Data Source\n",
    "* Model Vocabulary\n",
    "\n",
    "4. **Design** \n",
    "* Hyperparameters\n",
    "* Pipeline\n",
    "\n",
    "5. **Training**\n",
    "* PyTorch and PyTorch Lightning\n",
    "\n",
    "6. **Visualizations**\n",
    "* Tensorboard\n",
    "* Weights and Biases\n",
    "\n",
    "7. **Diagnosis/Evaluation**\n",
    "* Default Model\n",
    "* Comparing Models\n",
    "\n",
    "8. **Conclusion and Takeaways**\n",
    "\n",
    "9. **References**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import pickle\n",
    "\n",
    "from model import Collater, SimpleDataset, Transformer, evaluate, load_model, pairs_to_tensors, train\n",
    "from data import PolynomialLanguage, train_test_split, load_file\n",
    "\n",
    "import torch\n",
    "import pytorch_lightning as pl\n",
    "from torch.utils.data import DataLoader\n",
    "from utils import get_device"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pyTorch VERSION: 1.13.0\n",
      "pyTorch Lightning VERSION: 1.7.7\n",
      "--------------------------------------------------\n",
      "CUDA VERSION 11.6\n",
      "Available devices  1\n",
      "Active CUDA Device: GPU 0\n"
     ]
    }
   ],
   "source": [
    "print('pyTorch VERSION:', torch.__version__)\n",
    "print('pyTorch Lightning VERSION:', pl.__version__)\n",
    "print('-' * 50)\n",
    "\n",
    "print('CUDA VERSION', torch.version.cuda)\n",
    "print ('Available devices ', torch.cuda.device_count())\n",
    "print('Active CUDA Device: GPU', torch.cuda.current_device())\n",
    "\n",
    "device = get_device()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Overview\n",
    "\n",
    "My system project will tackle sequence-to-sequence prediction using a Transformer AI model. This falls under the category of neural machine translation where a deep neural network will be used. This is a common task among natural language processing applications.\n",
    "\n",
    "To be clear, this problem does not *need* neural machine translation to solve. It is likely that there are hardcoded solutions for distributed mutiplication of this type. However, applying a machine learning perspective to this problem is a fun and interesting challenge.\n",
    "\n",
    "My project with not be using human language, but rather algebraic polynomial expansion. An example can be shown here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "         SOURCE (src)                    TARGET (trg)             \n",
      "------------------------------------------------------------------\n",
      "         -5*h*(5-2*h)          |          10*h**2-25*h         \n",
      "------------------------------------------------------------------\n",
      "          s*(8*s-21)           |          8*s**2-21*s          \n",
      "------------------------------------------------------------------\n",
      "       (21-t)*(-6*t-4)         |        6*t**2-122*t-84        \n",
      "------------------------------------------------------------------\n",
      "       (21-5*c)*(3*c-7)        |       -15*c**2+98*c-147       \n",
      "------------------------------------------------------------------\n",
      "          4*n*(n+22)           |          4*n**2+88*n          \n",
      "------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "from data import load_file\n",
    "\n",
    "inputs, outputs = load_file(\"inputs/test_set.txt\") # loading test set\n",
    "\n",
    "print(f\"{'SOURCE (src)':^30}{'TARGET (trg) ':^36}\")\n",
    "print(\"-\" * 66)\n",
    "for i in range(5):\n",
    "    print(f\"{inputs[i]:^30} | {outputs[i]:^30}\")\n",
    "    print(\"-\" * 66)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The input to my model is a factored form of a polynomial sequence. The output is the expanded form of the polynomial sequence."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Decomposition/Domain Expertise\n",
    "\n",
    "Predicting a word based on previous words is a fundamental task for NLP. As human beings, we extrapolate and form sequences of thought almost intuititively. If we want to be able to communicate with intelligence machines, we are going to need to teach them language that we already understand. This is the first step in creating a system that can learn to communicate with us.\n",
    "\n",
    "If we can build a model with natural language ability, this would reduce the workload for many communication-based tasks and greatly increase the speed of communication. In a TED talk by Sam Harris about the safety of AI, he mentions, \"Electronic circuits function about a million times faster than biochemical ones. (...) The AI should think about a million times faster than the people who built it\" https://youtu.be/8nt3edWLgIg. The speed of computational processing would allow us to achieve new insight into the world around us. This would allow us to explore new ideas and concepts that we have never been able to before.\n",
    "\n",
    "Natural language processing is a continously developing field of study, and I only have a grasp on the basics. Using polynomial expansion allows exploration of the deep learning process with a greatly reduced \"vocabulary\". "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Data\n",
    "\n",
    "The data for this project was provided by a machine learning company for an interview application assessment. It is a set of 1,000,000 factored and expanded polynomials, line-by-line, located in a .txt file. I have split the original data file into a training set and a test set. The training set is 80% of the original data and the test set is 20% of the original data.\n",
    "\n",
    "Link to source data: https://scale-static-assets.s3-us-west-2.amazonaws.com/ml-interview/expand/train.txt\n",
    "\n",
    "The data exploration and analysis process is located in the Python Notebook file: exploratory_data_analysis.ipynb.\n",
    "\n",
    "In order for the model to process this textual data, a number of process must take place which are outlined below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setting path variables\n",
    "train_set_path = \"inputs/train_set.txt\"\n",
    "test_set_path = \"inputs/test_set.txt\"\n",
    "model_path = \"models/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First 3 pairs: \n",
      "['6*i*(-4*i-16)', '-24*i**2-96*i']\n",
      "['-4*s*(-4*s-27)', '16*s**2+108*s']\n",
      "['-6*o*(20-4*o)', '24*o**2-120*o']\n"
     ]
    }
   ],
   "source": [
    "train_set_pairs = PolynomialLanguage.load_pairs(train_set_path) # loading train set\n",
    "\n",
    "# print the first 3 pairs in the training set\n",
    "print('First 3 pairs: ')\n",
    "for pair in train_set_pairs[:3]:\n",
    "    print(pair)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "creating vocabs: 100%|██████████| 800000/800000 [00:06<00:00, 126573.69it/s]\n",
      "creating tensors: 100%|██████████| 760000/760000 [00:12<00:00, 60114.24it/s]\n",
      "creating tensors: 100%|██████████| 40000/40000 [00:00<00:00, 69714.98it/s]\n"
     ]
    }
   ],
   "source": [
    "# Hyperparameter for train vs validation split\n",
    "ratio = 0.95\n",
    "\n",
    "src_lang, trg_lang = PolynomialLanguage.create_vocabs(train_set_pairs) # creating source and target language objects\n",
    "train_pairs, val_pairs = train_test_split(train_set_pairs, ratio=ratio) # split for validation\n",
    "train_tensors = pairs_to_tensors(train_pairs, src_lang, trg_lang) # converting train pairs to tensors\n",
    "val_tensors = pairs_to_tensors(val_pairs, src_lang, trg_lang) # converting val pairs to tensors\n",
    "\n",
    "\n",
    "save_to_pickle = {\"src_lang.pickle\": src_lang, \"trg_lang.pickle\": trg_lang,}\n",
    "\n",
    "for k, v in save_to_pickle.items(): # saving source and target language objects\n",
    "    with open(os.path.join(model_path, k), \"wb\") as file_out:\n",
    "        pickle.dump(v, file_out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Source (src) Vocabulary: \n",
      "{'<pad>': 0, '<sos>': 1, '<eos>': 2, '<unk>': 3, '6': 4, '*': 5, 'i': 6, '(': 7, '-': 8, '4': 9, '1': 10, ')': 11, 's': 12, '2': 13, '7': 14, 'o': 15, '0': 16, 'n': 17, '3': 18, 'z': 19, '5': 20, '+': 21, 'y': 22, '8': 23, 'cos': 24, 'c': 25, 'j': 26, '9': 27, 't': 28, 'k': 29, 'a': 30, 'x': 31, 'h': 32, 'sin': 33, '**': 34, 'tan': 35}\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Target (trg) Vocabulary: \n",
      "{'<pad>': 0, '<sos>': 1, '<eos>': 2, '<unk>': 3, '-': 4, '2': 5, '4': 6, '*': 7, 'i': 8, '**': 9, '9': 10, '6': 11, '1': 12, 's': 13, '+': 14, '0': 15, '8': 16, 'o': 17, 'n': 18, '7': 19, '5': 20, '3': 21, 'z': 22, 'y': 23, 'cos': 24, '(': 25, 'c': 26, ')': 27, 'j': 28, 't': 29, 'k': 30, 'a': 31, 'x': 32, 'h': 33, 'sin': 34, 'tan': 35}\n",
      "----------------------------------------------------------------------------------------------------\n",
      "\n",
      "First 2 tensors: (src, trg)\n",
      "(tensor([ 1,  7, 12,  8, 10, 27, 11,  5,  7, 12, 21, 18, 16, 11,  2]), tensor([ 1, 13,  9,  5, 14, 12, 12,  7, 13,  4, 20, 19, 15,  2]))\n",
      "----------------------------------------------------------------------------------------------------\n",
      "(tensor([ 1,  8, 18,  5, 15,  5,  7, 15, 21, 10, 20, 11,  2]), tensor([ 1,  4, 21,  7, 17,  9,  5,  4,  6, 20,  7, 17,  2]))\n",
      "----------------------------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# printing the vocabulary\n",
    "print('Source (src) Vocabulary: ')\n",
    "print(src_lang.word2index)\n",
    "print('-'*100)\n",
    "print('Target (trg) Vocabulary: ')\n",
    "print(trg_lang.word2index)\n",
    "print('-'*100)\n",
    "print()\n",
    "# print the first 2 tensors in the training set\n",
    "print('First 2 tensors: (src, trg)')\n",
    "for tensor in train_tensors[:2]:\n",
    "    print(tensor)\n",
    "    print('-'*100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking above at each source and target tensor, we can see all tensors start with a 1 and end with a 2. This is used to indicate the start and end of a sequence. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Design\n",
    "\n",
    "### Hyperparameters\n",
    "\n",
    "Rules for Transformer Models:\n",
    "* The hidden_dim must be divisible by the number of heads.\n",
    "* The batch_size should ideally be as large as possible for a given GPU. This greatly increases the speed of training due to parallelization. I have 12GB of VRAM on my GPU, so I can use a batch size larger than the standard values of 32 or 64."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Specifying hyperparameters for model\n",
    "hid_dim=256 # default 256 \n",
    "enc_layers=3 # number of encoder layers, default 3\n",
    "dec_layers=3 # number of decoder layers, default 3\n",
    "enc_heads=8 # number of self-attention heads in encoder, default 8\n",
    "dec_heads=8 # number of self-attention heads in decoder, default 8\n",
    "enc_pf_dim=512 # position-wise feedforward dimension in encoder, default 512\n",
    "dec_pf_dim=512 # position-wise feedforward dimension in decoder, default 512\n",
    "enc_dropout=0.1 # dropout in encoder, default 0.1\n",
    "dec_dropout=0.1 # dropout in decoder, default 0.1\n",
    "\n",
    "# For batch processing\n",
    "num_workers=8 # basically the number of parallel processes for *loading* data, default 8, can be up to the number of CPU cores\n",
    "batch_size=512 # number of examples in a batch\n",
    "\n",
    "# Specifying hyperparameters for training\n",
    "val_check_interval=0.2 # how often to check validation loss (as a proportion of training steps)\n",
    "max_epochs=10 # maximum number of epochs to train for\n",
    "clip=1 # gradient clipping\n",
    "fast_dev_run=True # Show the output of the training loop without having to actually train the model. It is set to False for training."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pipeline\n",
    "\n",
    "Collating and Batching:\n",
    "\n",
    "PyTorch has a method to batch data that helps to take advantage of parallelization. This allows for the input data to be passed to the model in smaller pieces. \n",
    "\n",
    "The Transformer model class does allow for variable length input and target sequences, but not within individual batches. To fix the input and target lengths and minimize the amount of padding, a custom collator class and corresponding functions are used to pad the input and target sequences to the maximum length of the batch. Using a padding value of 0 allows us to extend the length of the vectors without affecting the dot product calculations.\n",
    "\n",
    "The PyTorch DataLoader class is used to create the batches. This function returns an iterator data structure that can be used to iterate over the batches. The DataLoader requires a custom dataset class with three methods: `__init__`, `__len__`, and `__getitem__`. The `__init__` method initializes the dataset and the `__len__` method returns the length of the dataset. The `__getitem__` method returns the input and target sequences for a given index. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "collate_fn = Collater(src_lang, trg_lang) # initializing collate function\n",
    "\n",
    "# the data is loaded in batches and then iterated over\n",
    "train_dataloader = DataLoader(SimpleDataset(train_tensors), # SimpleDataset is a custom dataset class\n",
    "                              batch_size=batch_size, # batch size\n",
    "                              collate_fn=collate_fn, # collate function\n",
    "                              num_workers=num_workers) # number of parallel processes\n",
    "\n",
    "val_dataloader = DataLoader(SimpleDataset(val_tensors),\n",
    "                            batch_size=batch_size,\n",
    "                            collate_fn=collate_fn,\n",
    "                            num_workers=num_workers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of batches in training set:  1485\n",
      "Batch:  0\n",
      "Source Length:  25 Target Length:  24\n",
      "--------------------------------------------------\n",
      "Batch:  1\n",
      "Source Length:  25 Target Length:  25\n",
      "--------------------------------------------------\n",
      "Batch:  2\n",
      "Source Length:  27 Target Length:  25\n",
      "--------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "print('Number of batches in training set: ', len(train_dataloader))\n",
    "\n",
    "i = 0\n",
    "\n",
    "for src, trg in train_dataloader:\n",
    "    print('Batch: ', i)\n",
    "    print('Source Length: ', src.shape[1], 'Target Length: ', trg.shape[1]) # the length of the tensor is the second value in the shape\n",
    "    i += 1\n",
    "    print('-'*50)\n",
    "    if i == 3:\n",
    "        break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Source (src) tensor with padding \"0\": \n",
      "tensor([[ 1,  7,  8, 23,  5, 17,  8, 13, 16, 11,  5,  7, 17,  8,  9, 11,  2,  0,\n",
      "          0,  0,  0,  0,  0,  0,  0,  0,  0]])\n"
     ]
    }
   ],
   "source": [
    "# showing the padding within the tensors in src\n",
    "print('Source (src) tensor with padding \"0\": ')\n",
    "print(src[:1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The source tensor is padded to the maximum length sequence of the batch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initializing the model using the hyperparameters\n",
    "model = Transformer(src_lang=src_lang, trg_lang=trg_lang, hid_dim=hid_dim, \n",
    "                    enc_layers=enc_layers, dec_layers=dec_layers, enc_heads=enc_heads, \n",
    "                    dec_heads=dec_heads, enc_pf_dim=enc_pf_dim, dec_pf_dim=dec_pf_dim, \n",
    "                    enc_dropout=enc_dropout, dec_dropout=dec_dropout)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Training\n",
    "\n",
    "#### PyTorch and PyTorch Lightning\n",
    "For the purposed of this overview, the model has already been fully trained. \n",
    "\n",
    "During training, the below cell: model = train(...) was run. PyTorch Lightning produces a number of files to help keep track of the model training process. The model checkpoint is located in the models folder. [Basic Checkpoint Documentation](https://pytorch-lightning.readthedocs.io/en/stable/common/checkpointing_basic.html)\n",
    "\n",
    "This .ckpt file is updated after every epoch of training, in case the training process is interrupted. The model checkpoint is also used to load the model for inference.\n",
    "\n",
    "To show what the output of PyTorch Lighting looks like, I have passed in a parameter called \"fast_dev_run\". This parameter is generally used for debugging purposes, but it is useful for showing the output PyTorch Lightning produces during training.\n",
    "\n",
    "During training, there are a number of progess bars that update with each batch and validation pass. Because the model is in fast_dev_run mode, only one step is shown.\n",
    "\n",
    "---\n",
    "\n",
    "**DO NOT RUN THE CODE IN THE NEXT CELL WITH FAST_DEV_RUN = FALSE UNLESS YOU WANT TO TRAIN THE MODEL**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using 16bit native Automatic Mixed Precision (AMP)\n",
      "GPU available: True (cuda), used: True\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "HPU available: False, using: 0 HPUs\n",
      "Running in `fast_dev_run` mode: will run the requested loop using 1 batch(es). Logging and checkpointing is suppressed.\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "\n",
      "  | Name      | Type             | Params\n",
      "-----------------------------------------------\n",
      "0 | encoder   | Encoder          | 1.6 M \n",
      "1 | decoder   | Decoder          | 2.4 M \n",
      "2 | criterion | CrossEntropyLoss | 0     \n",
      "-----------------------------------------------\n",
      "4.0 M     Trainable params\n",
      "0         Non-trainable params\n",
      "4.0 M     Total params\n",
      "8.065     Total estimated model params size (MB)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6c0698e66933493fa730f812e9f0f85e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Training: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e401bf23c1a248b4b99425c4c315d2f6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`Trainer.fit` stopped: `max_steps=1` reached.\n"
     ]
    }
   ],
   "source": [
    "# ignoring the warning from pytorch lightning about the model file already existing\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "# Initializing the trainer with the hyperparameters and training the model\n",
    "model = train(model, train_dataloader, val_dataloader, val_check_interval, max_epochs, clip, model_path, fast_dev_run)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visualizations\n",
    "\n",
    "### TensorBoard / Weights and Biases (Wandb)\n",
    "\n",
    "**Tensorboard** is nice to track individual training runs and also visualizing the model graph. (https://www.tensorflow.org/tensorboard)\n",
    "\n",
    "**Wandb** is a nice tool to track multiple training runs and compare the results. (https://wandb.ai/site)\n",
    "\n",
    "I found that being able to tangibly see the differences between the model training configurations in Wandb is more valuable, and I am able to share the results with others more easily.\n",
    "\n",
    "To view the interactive Wandb dashboard, click here: https://wandb.ai/chrisvaisnor/system_project?workspace=user-chrisvaisnor\n",
    "\n",
    "Below are some screenshots from each dashboard. \n",
    "\n",
    "#### TensorBoard\n",
    "\n",
    "<img src=\"tensorBoard_outputs/loss.png\" alt=\"drawing\" width=\"900\"/>\n",
    "<img src=\"tensorBoard_outputs/parallel_coords_view.png\" alt=\"drawing\" width=\"900\"/> \n",
    "<img src=\"tensorBoard_outputs/model.png\" alt=\"drawing\" width=\"900\"/>\n",
    "\n",
    "#### Weights and Biases (Wandb)\n",
    "\n",
    "<img src=\"wandb_outputs/loss.png\" alt=\"drawing\" width=\"900\"/>\n",
    "<img src=\"wandb_outputs/runs_4.png\" alt=\"drawing\" width=\"900\"/>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_set_pairs = PolynomialLanguage.load_pairs(test_set_path) # loading test set\n",
    "\n",
    "# Loading the saved model checkpoint located in /models.\n",
    "model_red = load_model(model_path, model_ckpt=\"model_red_default.ckpt\") # default model\n",
    "model_orange = load_model(model_path, model_ckpt=\"model_orange.ckpt\") # for comparison later"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Diagnosis/Evaluation\n",
    "\n",
    "#### Default Model\n",
    "\n",
    "For simplicity, the evaluate function calls the model.predict() function which transforms and batches the data, implicitly. This is in contrast to earlier in the notebook when the data processessing was shown explicitly. The saved model is passed into the function as a parameter, in addition to the test set pairs and batch size.\n",
    "\n",
    "Depending on the hardware used, this cell can take between 5 minutes to over an hour. The test set is 200,000 lines long.\n",
    "\n",
    "Scoring is based on if the model's prediction is exactly equal to the target. \n",
    "\n",
    "A (1) is returned if the prediction is correct and a (0) is returned if the prediction is incorrect. The average of these values is the accuracy score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "creating prediction tensors: 100%|██████████| 200000/200000 [00:01<00:00, 134501.39it/s]\n",
      "predict batch num: 100%|██████████| 391/391 [04:16<00:00,  1.52it/s]\n",
      "scoring: 100%|██████████| 200000/200000 [00:00<00:00, 3127661.84it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "---- Test Case 0 ----\n",
      "Input = -5*h*(5-2*h)\n",
      "Target = 10*h**2-25*h\n",
      "Predicted = 10*h**2-25*h\n",
      "score = 1\n",
      "\n",
      "---- Test Case 1 ----\n",
      "Input = s*(8*s-21)\n",
      "Target = 8*s**2-21*s\n",
      "Predicted = 8*s**2-21*s\n",
      "score = 1\n",
      "\n",
      "---- Test Case 2 ----\n",
      "Input = (21-t)*(-6*t-4)\n",
      "Target = 6*t**2-122*t-84\n",
      "Predicted = 6*t**2-122*t-84\n",
      "score = 1\n",
      "\n",
      "---- Test Case 3 ----\n",
      "Input = (21-5*c)*(3*c-7)\n",
      "Target = -15*c**2+98*c-147\n",
      "Predicted = -15*c**2+98*c-147\n",
      "score = 1\n",
      "\n",
      "---- Test Case 4 ----\n",
      "Input = 4*n*(n+22)\n",
      "Target = 4*n**2+88*n\n",
      "Predicted = 4*n**2+88*n\n",
      "score = 1\n",
      "\n",
      "---- Test Case 5 ----\n",
      "Input = (k+2)*(5*k+29)\n",
      "Target = 5*k**2+39*k+58\n",
      "Predicted = 5*k**2+39*k+58\n",
      "score = 1\n",
      "\n",
      "---- Test Case 6 ----\n",
      "Input = (k-15)*(2*k+29)\n",
      "Target = 2*k**2-k-435\n",
      "Predicted = 2*k**2+k-435\n",
      "score = 0\n",
      "\n",
      "---- Test Case 7 ----\n",
      "Input = (i-20)*(i+24)\n",
      "Target = i**2+4*i-480\n",
      "Predicted = i**2+4*i-480\n",
      "score = 1\n",
      "\n",
      "---- Test Case 8 ----\n",
      "Input = -4*c*(c+2)\n",
      "Target = -4*c**2-8*c\n",
      "Predicted = -4*c**2-8*c\n",
      "score = 1\n",
      "\n",
      "---- Test Case 9 ----\n",
      "Input = -6*j*(-7*j-11)\n",
      "Target = 42*j**2+66*j\n",
      "Predicted = 42*j**2+66*j\n",
      "score = 1\n",
      "--------------------------------------------------\n",
      "Number Correct:  177508\n",
      "Number of Examples:  200000\n",
      "Final Score = 0.8875\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.88754"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate(model_red, test_set_pairs, batch_size=batch_size, print_examples=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on the current training configuration, the model is able to achieve a score of 88.7% on the test set! In addition, test case 6 was very close to correct. \n",
    "\n",
    "In the future, a more specific evaluation metric could be used to determine how close the model's prediction is to the target. This would more accurately reflect the model's performance and especially help with natural language processing tasks.\n",
    "\n",
    "### Comparing performance with two different configurations\n",
    "\n",
    "* Red = (hidden_dim=256, layers=3, batch_size=512) **DEFAULT**\n",
    "* Orange = (hidden_dim=256, layers=3, batch_size=256)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "creating prediction tensors: 100%|██████████| 200000/200000 [00:01<00:00, 128078.86it/s]\n",
      "predict batch num: 100%|██████████| 391/391 [04:18<00:00,  1.51it/s]\n",
      "scoring: 100%|██████████| 200000/200000 [00:00<00:00, 3233601.11it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------------\n",
      "Number Correct:  177489\n",
      "Number of Examples:  200000\n",
      "Final Score = 0.8874\n",
      "----------------------------------------------------------------------------------------------------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "creating prediction tensors: 100%|██████████| 200000/200000 [00:01<00:00, 131581.80it/s]\n",
      "predict batch num: 100%|██████████| 782/782 [04:34<00:00,  2.85it/s]\n",
      "scoring: 100%|██████████| 200000/200000 [00:00<00:00, 3350149.96it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------------\n",
      "Number Correct:  174827\n",
      "Number of Examples:  200000\n",
      "Final Score = 0.8741\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.874135"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate(model_red, test_set_pairs, batch_size=512, print_examples=False)\n",
    "print('-'*100)\n",
    "evaluate(model_orange, test_set_pairs, batch_size=256, print_examples=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the case of the two evaluations above, the models have a very similar accuracy score. The model that used a batch size of 512 was able to train and evaluate 5-10% faster.\n",
    "\n",
    "The batch size is the number of training examples that are passed through the model at once. The larger the batch size, the more memory is required. There is a concensus that larger batch sizes can reduce model accuracy. The thought is that there is less opportunity for the model to learn from each example. In my case, increasing batch size did NOT reduce model accuracy. This is likely due to a number of factors.\n",
    "\n",
    "* The dataset is large enough that the model has plenty of examples to learn from. If the dataset was under 100k lines, then accuracy may have been reduced.\n",
    "* There is enough steps per epoch, and total epochs, to allow the model to converge to an accurate solution."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion and Takeaways\n",
    "\n",
    "The Transformer is a powerful model for sequence-to-sequence tasks and this project was a great introduction to the model. Going in-depth into the model's architecture and the training process was a great learning experience. My skills in PyTorch have grown exponentially and I was introduced to PyTorch Lightning, TensorBoard, and Weights and Biases, all of which I will continue to use and develop with in the future.\n",
    "\n",
    "Taking a look at the system section of the Weights and Biases dashboard, I also learned how GPU wattage, utilization, and memory usage can be tracked. \n",
    "\n",
    "To compare two of the training configurations, lets use the following two runs evaluated earlier:\n",
    "* hid_dim=256, layers=3, batch_size=256 (orange)\n",
    "* hid_dim=256, layers=3, batch_size=512 (red) (default)\n",
    "\n",
    "Looking at the GPU power usage in watts and the training time in minutes, there are a few things to learn.\n",
    "\n",
    "* The larger batch size (red) trained in 22 minutes and averaged 123 watts.\n",
    "    * 22m x 123w = 162,360 joules\n",
    "* The smaller batch size (orange) trained in 27 minutes and averaged 108 watts.\n",
    "    * 27m x 108w = 174,960 joules\n",
    "\n",
    "<img src=\"wandb_outputs/gpu_watts.png\" alt=\"drawing\" width=\"600\"/>\n",
    "\n",
    "* This leads to a [relative difference](https://docs.oracle.com/en/cloud/saas/planning-budgeting-cloud/pfusu/insights_metrics_RPD.html) of 12.9% in power usage and 20.4% in training time.\n",
    "\n",
    "In this case, having used a larger batch size, even if it included more VRAM and GPU watts, was cost effective. Because the data set was large enough, there was no penalty to accuracy. This is useful to know when training models in the future.\n",
    "\n",
    "There is new software and hardware being developed every day and I am excited to see what the future holds for machine learning and artificial intelligence. This project has given me a backbone to build on and continously reference from. The next generation of transformer models, such as [Generative Pre-Training Transformers (GPT)](https://en.wikipedia.org/wiki/GPT-3) and [Bidirecitonal Encoder Representations from Transformer (BERT)](https://en.wikipedia.org/wiki/BERT_(language_model)), are now the cutting edge of natural language processing. With the foundational skills I have gained from this project, I am excited to dig deeper into these models and the future of NLP."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# References\n",
    "\n",
    "* Attention Is All You Need (Original Paper) https://arxiv.org/abs/1706.03762\n",
    "* The Illustrated Transformer (Jay Lammar) http://jalammar.github.io/illustrated-transformer/\n",
    "* The Annotated Transformer (Harvard Team) http://nlp.seas.harvard.edu/annotated-transformer/\n",
    "* PyTorch https://pytorch.org/\n",
    "* PyTorch Lightning https://www.pytorchlightning.ai/\n",
    "* TensorBoard https://www.tensorflow.org/tensorboard\n",
    "* Transformer Models (PyTorch) https://pytorch.org/tutorials/beginner/transformer_tutorial.html\n",
    "* Mastering Transformers (O'Reilly Book) https://learning.oreilly.com/library/view/mastering-transformers/9781801077651/\n",
    "* Mastering PyTorch (O'Reilly Book) https://learning.oreilly.com/library/view/mastering-pytorch/9781789614381/\n",
    "* Transformers for Natural Language Processing (O'Reilly Book) https://learning.oreilly.com/library/view/transformers-for-natural/9781803247335/\n",
    "* Seq-to-Seq Models https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html\n",
    "* PyTorch-seq2seq https://github.com/bentrevett/pytorch-seq2seq\n",
    "* PyTorch Datasets and DataLoaders https://pytorch.org/tutorials/beginner/basics/data_tutorial.html\n",
    "* Seq2Seq (https://github.com/jaymody/seq2seq-polynomial)\n",
    "* TensorFlow Transformer Tutorial (Lilian Weng) https://github.com/lilianweng/transformer-tensorflow\n",
    "* PyTorch Tutorials https://pytorch.org/tutorials/\n",
    "* Cuda Toolkit https://developer.nvidia.com/cuda-toolkit\n",
    "* Anaconda https://www.anaconda.org/"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8.13 ('system_project_env')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13 (default, Oct 21 2022, 23:50:54) \n[GCC 11.2.0]"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "1440ca659962f92585b915ea9c35fda159f7dda4777faeac167d697a04f9a992"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}