Sentiment_Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "853c2426",
   "metadata": {},
   "source": [
    "# Sentiment Analysis\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "a14a95a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv\n",
    "import time\n",
    "from collections import Counter\n",
    "\n",
    "import numpy as np\n",
    "import torch\n",
    "from torch.utils.data import DataLoader\n",
    "from torchtext.vocab import Vocab\n",
    "from torchtext.data.utils import get_tokenizer\n",
    "from tqdm.notebook import tqdm"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5cf091df",
   "metadata": {},
   "source": [
    "# Parsing the Raw Data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "14e9cfd0",
   "metadata": {},
   "outputs": [],
   "source": [
    "TRAIN_PATH = \"./data/reviews_train.tsv\"\n",
    "VALIDATION_PATH = \"./data/reviews_val.tsv\"\n",
    "TEST_PATH = \"./data/reviews_test.tsv\"\n",
    "\n",
    "def load_data(data_path):\n",
    "    with open(data_path, \"rb\") as f:\n",
    "        lines = f.readlines()\n",
    "    lines = [l.decode(\"cp1252\").strip().split(\"\\t\") for l in lines]\n",
    "    examples = []\n",
    "    for line in lines[1:]:\n",
    "        examples.append(dict(zip(lines[0], line)))\n",
    "        \n",
    "    return examples\n",
    "\n",
    "train_examples = load_data(TRAIN_PATH)\n",
    "validation_examples = load_data(VALIDATION_PATH)\n",
    "test_examples = load_data(TEST_PATH)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "833c8a11",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ratio of Training to validation examples: 8.0\n",
      "Ratio of validation to test examples: 1.0\n"
     ]
    }
   ],
   "source": [
    "print(f'Ratio of Training to validation examples: {len(train_examples)/len(validation_examples)}')\n",
    "print(f'Ratio of validation to test examples: {len(validation_examples)/len(test_examples)}')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7802947",
   "metadata": {},
   "source": [
    "- Same number of training and validation examples; ratio ===1.0.\n",
    "- Ratio of training to validation examples is 8.0\n",
    "- 80% of the data is used as training data, 10% used for testing and 10% used for validation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "8f667ed8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "========Average Training Typical review length=======\n",
      "Typical length of 445.5155\n",
      "========Average Testing Typical review length=======\n",
      "Typical length of 415.37\n",
      "========Average validation Typical review length=======\n",
      "Typical length of 452.862\n"
     ]
    }
   ],
   "source": [
    "print(\"========Average Training Typical review length=======\")\n",
    "length_a = []\n",
    "for i in range(len(train_examples)):\n",
    "    length_a.append(len(train_examples[i]['text']))\n",
    "print(f\"Typical length of {sum(length_a)/len(train_examples)}\")\n",
    "\n",
    "print(\"========Average Testing Typical review length=======\")\n",
    "length_b = []\n",
    "for i in range(len(test_examples)):\n",
    "    length_b.append(len(test_examples[i]['text']))\n",
    "print(f\"Typical length of {sum(length_b)/len(test_examples)}\")\n",
    "      \n",
    "print(\"========Average validation Typical review length=======\")\n",
    "length_c = []\n",
    "for i in range(len(validation_examples)):\n",
    "    length_c.append(len(validation_examples[i]['text']))\n",
    "print(f\"Typical length of {sum(length_c)/len(validation_examples)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "455b4d49",
   "metadata": {},
   "source": [
    "- Typical (Average) review length for samples in training set is 445.5155\n",
    "- Typical (Average) review length for samples in test set is 415.37\n",
    "- Typical (Average) review length for samples in validation set is 452.862"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "2390976a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 1962 positive and 2038 negative reviews in the training set data.\n",
      "ratio of positive to negative reviews in training set: 0.9627085377821394.\n",
      "\n",
      "There are 261 positive and 239 negative reviews in the testing set data.\n",
      "ratio of positive to negative reviews in testing set: 1.0920502092050208.\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "1.0920502092050208"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def get_pos_to_neg_ratio(examples,name):\n",
    "    positive,negative = [],[]\n",
    "    for i in range(len(examples)):\n",
    "        if examples[i]['sentiment'] == '1':\n",
    "            positive.append(len(examples[i]['sentiment']))\n",
    "        elif examples[i]['sentiment'] == '-1':\n",
    "            negative.append(len(examples[i]['sentiment']))\n",
    "    print(f'There are {len(positive)} positive and {len(negative)} negative reviews in the {name} data.')\n",
    "\n",
    "    print(f\"ratio of positive to negative reviews in {name}: {len(positive)/len(negative)}.\\n\")\n",
    "    return len(positive)/len(negative)\n",
    "\n",
    "get_pos_to_neg_ratio(train_examples,'training set')\n",
    "get_pos_to_neg_ratio(test_examples, 'testing set')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e53f3fa",
   "metadata": {},
   "source": [
    "- The ratio positive to negative reviews in the train set is 0.967.\n",
    "    - This means that the amount of positive and negative reviews are almost the same.\n",
    "    - There are more negative reviews in the training set than positive.\n",
    "    - There are more positive reviews in the test set than negative\n",
    "- The dataset is **balanced**. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ddf2a6e6",
   "metadata": {},
   "source": [
    "- Based on this ratio, Accuracy will be an appropraite evaluation metric."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e42a059",
   "metadata": {},
   "source": [
    "# Data Preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d88efa24",
   "metadata": {},
   "source": [
    "# GloVe Embeddings\n",
    "\n",
    "\n",
    "GloVe is an unsupervised learning algorithm for obtaining vector representations for words (word embeddings), similar to Word2Vec. Training is performed by looking at words that co-occur in a very large dataset of text, with the goal of mapping each word to a dense vector represenation. Words that have similar meanings (**semantics**) generally have GloVe vectors that are nearby.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "8fecd377",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the 0.04656 0.21318 -0.0074364 -0.45854 -0.035639 0.23643 -0.28836 0.21521 -0.13486 -1.6413 -0.26091 0.032434 0.056621 -0.043296 -0.021672 0.22476 -0.075129 -0.067018 -0.14247 0.038825 -0.18951 0.29977 0.39305 0.17887 -0.17343 -0.21178 0.23617 -0.063681 -0.42318 -0.11661 0.093754 0.17296 -0.33073 0.49112 -0.68995 -0.092462 0.24742 -0.17991 0.097908 0.083118 0.15299 -0.27276 -0.038934 0.54453 0.53737 0.29105 -0.0073514 0.04788 -0.4076 -0.026759 0.17919 0.010977 -0.10963 -0.26395 0.07399 0.26236 -0.1508 0.34623 0.25758 0.11971 -0.037135 -0.071593 0.43898 -0.040764 0.016425 -0.4464 0.17197 0.046246 0.058639 0.041499 0.53948 0.52495 0.11361 -0.048315 -0.36385 0.18704 0.092761 -0.11129 -0.42085 0.13992 -0.39338 -0.067945 0.12188 0.16707 0.075169 -0.015529 -0.19499 0.19638 0.053194 0.2517 -0.34845 -0.10638 -0.34692 -0.19024 -0.2004 0.12154 -0.29208 0.023353 -0.11618 -0.35768 0.062304 0.35884 0.02906 0.0073005 0.0049482 -0.15048 -0.12313 0.19337 0.12173 0.44503 0.25147 0.10781 -0.17716 0.038691 0.08153 0.14667 0.063666 0.061332 -0.075569 -0.37724 0.01585 -0.30342 0.28374 -0.042013 -0.040715 -0.15269 0.07498 0.15577 0.10433 0.31393 0.19309 0.19429 0.15185 -0.10192 -0.018785 0.20791 0.13366 0.19038 -0.25558 0.304 -0.01896 0.20147 -0.4211 -0.0075156 -0.27977 -0.19314 0.046204 0.19971 -0.30207 0.25735 0.68107 -0.19409 0.23984 0.22493 0.65224 -0.13561 -0.17383 -0.048209 -0.1186 0.0021588 -0.019525 0.11948 0.19346 -0.4082 -0.082966 0.16626 -0.10601 0.35861 0.16922 0.07259 -0.24803 -0.10024 -0.52491 -0.17745 -0.36647 0.2618 -0.012077 0.08319 -0.21528 0.41045 0.29136 0.30869 0.078864 0.32207 -0.041023 -0.1097 -0.092041 -0.12339 -0.16416 0.35382 -0.082774 0.33171 -0.24738 -0.048928 0.15746 0.18988 -0.026642 0.063315 -0.010673 0.34089 1.4106 0.13417 0.28191 -0.2594 0.055267 -0.052425 -0.25789 0.019127 -0.022084 0.32113 0.068818 0.51207 0.16478 -0.20194 0.29232 0.098575 0.013145 -0.10652 0.1351 -0.045332 0.20697 -0.48425 -0.44706 0.0033305 0.0029264 -0.10975 -0.23325 0.22442 -0.10503 0.12339 0.10978 0.048994 -0.25157 0.40319 0.35318 0.18651 -0.023622 -0.12734 0.11475 0.27359 -0.21866 0.015794 0.81754 -0.023792 -0.85469 -0.16203 0.18076 0.028014 -0.1434 0.0013139 -0.091735 -0.089704 0.11105 -0.16703 0.068377 -0.087388 -0.039789 0.014184 0.21187 0.28579 -0.28797 -0.058996 -0.032436 -0.0047009 -0.17052 -0.034741 -0.11489 0.075093 0.099526 0.048183 -0.073775 -0.41817 0.0041268 0.44414 -0.16062 0.14294 -2.2628 -0.027347 0.81311 0.77417 -0.25639 -0.11576 -0.11982 -0.21363 0.028429 0.27261 0.031026 0.096782 0.0067769 0.14082 -0.013064 -0.29686 -0.079913 0.195 0.031549 0.28506 -0.087461 0.0090611 -0.20989 0.053913\n",
      "\n",
      ", -0.25539 -0.25723 0.13169 -0.042688 0.21817 -0.022702 -0.17854 0.10756 0.058936 -1.3854 0.58509 0.036501 -0.19846 0.19613 0.40929 0.15702 -0.15305 0.050447 0.30045 -0.11295 -0.017043 0.18593 0.19982 0.20053 -0.63141 -0.12622 0.2951 -0.26282 -0.15831 0.0012383 0.011784 0.58758 -0.15914 0.27731 -0.82343 -0.21134 0.013414 0.19637 -0.4147 0.0010276 0.13422 -0.14205 0.051545 0.34993 -0.29868 -0.3209 0.19566 0.47886 0.10744 0.010004 0.18503 0.080694 0.20739 -0.097365 -0.039448 0.020151 -0.17378 0.25679 0.24198 -0.351 0.18759 0.0063857 0.18395 -0.13929 0.0081855 -0.63109 0.29832 0.31731 0.13022 -0.32284 -0.050343 -0.114 0.12097 0.14687 -0.33244 -0.055789 -0.05849 0.27551 -0.043855 0.039664 0.15162 -0.086627 0.067729 0.23146 0.015351 -0.15142 -0.031975 0.45181 -0.068806 -0.077058 0.055193 0.054596 -0.24708 0.031113 -0.12826 0.12782 -0.46708 -0.026264 0.010387 -0.33174 0.17277 -0.26894 0.20467 -0.16181 -0.041519 -0.014878 0.10279 0.18868 -0.23396 -0.018436 -0.14747 -0.32685 -0.022055 -0.054 0.16264 0.27095 -0.22792 -0.0077006 0.11206 -0.039787 -0.11906 0.021773 0.05528 -0.13318 -0.056867 0.008304 -0.027021 0.23447 0.086864 0.12009 -0.30726 0.0024735 0.29041 -0.044887 0.12297 0.13077 0.090807 -0.39141 0.080546 0.18724 -0.097481 0.10397 0.11492 0.17775 -0.18167 0.24652 0.20136 -0.23395 -0.35018 -0.14061 0.17091 -0.095465 -0.10962 -0.09836 0.15344 0.08868 -0.22048 -0.13803 -0.11288 -0.08534 0.072735 -0.12732 -0.1964 -0.10586 0.0020616 0.13496 0.058912 -0.043979 -0.091375 0.24408 0.16872 0.24297 -0.43983 0.47089 -0.018595 0.16146 0.19828 -0.17237 -0.0026998 0.52097 -0.080197 0.43324 -0.066261 0.04324 0.084954 -0.14836 -0.41936 0.15988 -0.18411 0.1321 0.27476 0.27279 -0.13465 -0.091238 -0.32523 0.27936 0.023296 -0.33472 0.016878 -0.055544 0.92915 -0.33914 -0.14791 0.017301 0.18272 0.35108 -0.11438 0.13228 -0.021064 -0.27453 -0.10081 -0.046296 0.21689 -0.056319 0.14651 -0.023536 0.068026 -0.045453 -0.23851 -0.33868 0.31396 -0.031914 -0.019217 0.0018715 -0.13328 0.070148 -0.039761 0.070801 0.0018422 -0.12646 0.028675 -0.095728 0.26673 -0.35536 0.15286 0.064565 0.12647 0.23397 -0.046058 0.13519 -0.14549 0.23031 0.42066 0.16267 -0.16541 -0.0020155 0.080653 -0.30025 -0.076014 0.070612 0.3157 0.05352 -0.10721 -0.1366 0.32214 0.2004 0.11609 -0.22501 0.12155 -0.10851 -0.063187 -0.24553 -0.059751 0.068787 -0.11627 -0.0083402 0.0052044 -0.20159 -0.023663 0.17562 -0.31475 -0.11162 -0.12492 0.10949 -0.26913 0.34893 -1.6997 -0.2447 0.30292 0.05672 -0.31737 0.083612 0.095949 -0.1759 0.10235 0.36808 -0.3438 0.20607 0.19135 0.10992 0.075968 -0.014359 -0.073794 0.22176 0.14652 0.56686 0.053307 -0.2329 -0.12226 0.35499\n",
      "\n",
      ". -0.12559 0.01363 0.10306 -0.10123 0.098128 0.13627 -0.10721 0.23697 0.3287 -1.6785 0.22393 0.12409 -0.086708 0.3301 0.34375 -0.00087582 -0.29658 0.24417 -0.11592 -0.035742 -0.01083 0.20776 0.29285 -0.073491 -0.18598 -0.2009 -0.095366 0.0063732 -0.1362 0.092028 -0.039957 0.19027 -0.10456 0.002767 -0.71742 -0.12915 -0.0013451 0.27002 -0.053023 0.22148 0.13881 -0.15051 -0.1915 0.16402 0.097484 0.056841 0.39789 0.40725 0.14802 0.21569 -0.10671 -0.10232 0.02481 -0.221 -0.01072 0.14234 -0.28242 0.19254 0.08672 -0.3897 0.11321 0.0013779 0.0064009 -0.16206 -0.082153 -0.55397 0.36789 -0.0040159 0.2071 -0.37157 0.25135 -0.19544 -0.047059 0.17155 -0.24036 -0.046086 0.19429 -0.18939 -0.0071974 0.069481 0.059175 -0.17585 0.10653 0.16933 -0.036122 0.029911 -0.1183 0.13916 -0.037951 0.1069 -0.26069 -0.10307 -0.12272 -0.15032 -0.042409 0.013354 -0.2851 0.011248 0.16073 -0.16384 0.21233 -0.18476 -0.00090874 0.066687 0.16918 -0.35004 0.099016 0.46393 -0.19462 0.10346 -0.25668 -0.36516 -0.18963 -0.21933 0.024634 0.065627 -0.1112 -0.164 0.010874 -0.084688 -0.14923 -0.070223 0.028887 0.083497 -0.016193 -0.0024926 0.17186 0.0098749 0.080237 0.14774 0.043206 0.27716 0.57697 -0.041297 0.12765 -0.091517 0.14132 0.087579 0.093224 0.015346 -0.19856 0.017277 -0.10708 -0.013059 -0.37227 0.078568 0.16677 -0.15359 -0.33294 0.036986 0.11697 0.039781 0.038464 -0.16247 0.4128 -0.077491 0.04549 0.1133 0.0082177 -0.25052 0.070966 -0.11388 -0.11503 -0.11014 0.10499 0.15878 -0.27023 -0.011006 0.00076057 0.33902 0.25564 0.16342 -0.56019 0.13055 0.076311 -0.028334 0.28721 -0.027844 -0.11561 0.34925 -0.1242 0.21405 0.24116 -0.031343 0.10913 -0.24755 -0.045429 -0.082178 -0.18831 0.18446 -0.097074 0.32395 0.10658 -0.26676 -0.27311 0.017181 0.25796 -0.28048 0.3079 -0.218 0.87415 -0.12297 0.10991 -0.29797 0.13394 0.10615 -0.10789 -0.35976 -0.18311 -0.45133 0.034967 -0.19847 0.21965 0.08152 0.2581 0.040173 0.031394 0.19069 0.0758 -0.060638 0.20739 0.009839 -0.2693 0.066515 -0.10711 0.0059916 0.23284 -0.058663 0.098993 -0.081464 0.067004 -0.14305 0.25506 -0.31971 -0.03107 -0.092451 0.2944 0.28947 -0.059804 0.24286 -0.16755 0.042031 0.51261 0.24525 -0.65983 0.062456 0.052204 -0.025717 -0.080613 0.080869 0.22821 -0.10217 -0.20719 -0.012123 0.34916 0.086527 0.066288 -0.099828 0.25843 0.11943 -0.13667 -0.43962 0.23704 0.031296 0.074701 -0.22387 0.0078162 -0.19016 0.044444 0.20191 -0.20814 -0.28382 0.10427 -0.21098 0.18865 0.31659 -2.0753 -0.071045 0.52419 0.056023 -0.25295 -0.062168 -0.10989 -0.35755 -0.079244 0.37472 -0.28353 0.16337 0.11165 -0.098002 0.060148 -0.15619 -0.11949 0.23445 0.081367 0.24618 -0.15242 -0.34224 -0.022394 0.13684\n",
      "\n",
      "of -0.076947 -0.021211 0.21271 -0.72232 -0.13988 -0.12234 -0.17521 0.12137 -0.070866 -1.5721 -0.22464 0.04269 -0.4018 0.21006 0.014288 0.41628 0.017165 0.071732 0.0069246 0.18107 -0.15412 0.14933 -0.030493 0.29918 0.029479 -0.036147 -0.061125 0.083918 -0.12398 -0.10077 -0.0054142 0.3371 -0.25612 0.44388 -0.68922 0.1802 0.34898 -0.052284 -0.26226 -0.47109 0.21647 -0.4002 -0.049986 0.011376 0.54994 -0.22791 0.095873 0.47693 -0.056727 -0.17895 0.11756 0.14662 0.048948 0.13587 -0.093821 0.45968 -0.32062 0.29911 0.20656 -0.18503 -0.2769 -0.022545 0.70698 -0.23815 0.16437 -0.55044 -0.0010615 0.12266 0.11898 0.23985 0.29815 0.013207 0.16316 -0.61334 -0.37051 0.19444 -0.13621 -0.30426 -0.37715 0.065299 -0.15995 -0.56516 0.074696 0.40184 0.19328 0.041802 0.20572 0.28971 0.34783 0.33873 -0.10052 -0.16397 -0.15236 -0.086815 0.36522 0.14969 -0.40859 0.23106 0.17162 -0.60545 0.086019 0.37043 0.17937 -0.40282 -0.62471 -0.055919 0.15092 0.12554 -0.45344 0.34417 0.40042 -0.049512 -0.29969 -0.31761 0.30023 0.090029 0.3106 -0.033077 -0.21995 -0.40396 -0.34443 -0.21248 -0.37636 0.21835 -0.1785 -0.17261 0.16391 0.22753 0.2686 0.57541 -0.14912 0.20413 0.22187 -0.27014 0.068253 0.29115 -0.067943 0.10623 -0.16281 0.19939 -0.48613 0.035688 -0.12373 0.13707 0.33359 -0.12713 -0.31711 -0.13962 -0.04288 -0.0014614 0.76883 -0.41705 -0.092911 0.16315 0.29202 0.12119 -0.076683 0.14131 -0.093406 -0.042796 0.13738 0.014278 0.11918 -0.34215 -0.19076 -0.12499 0.24648 0.42259 0.091966 0.45351 0.14437 0.1878 -0.85876 0.059621 -0.32242 0.28627 0.12427 0.0090984 -0.1891 0.16638 0.099881 -0.048553 -0.026257 0.099904 0.12406 -0.015416 -0.29707 -0.4044 -0.17258 0.36468 -0.014118 -0.11889 -0.11686 -0.14124 0.28012 0.067644 0.1485 -0.35702 0.29626 0.36004 1.019 -0.067307 -0.11588 -0.2178 0.070191 0.23154 -0.13849 0.26441 0.28742 0.1941 -0.0060504 0.44105 0.12416 -0.27745 -0.25729 0.10992 0.18362 -0.34522 -0.21861 -0.18825 -0.037454 -0.20862 -0.25216 0.060842 0.068595 0.10275 0.10745 -0.061288 0.19725 -0.27739 -0.022559 0.052794 -0.24083 0.09199 0.30959 0.054999 0.063676 -0.087357 -0.34495 0.22793 -0.42405 0.24536 0.55708 0.19126 -0.797 -0.2048 0.32545 0.09235 0.084791 -0.16433 -0.066568 -0.099249 0.31526 -0.44465 0.087281 0.3288 -0.017809 -0.23855 -0.12848 0.041509 0.46728 0.48214 0.10548 0.065805 0.067221 0.13321 -0.27856 0.015532 0.30026 0.38748 -0.14401 -0.16131 0.17678 0.16448 -0.3244 0.007937 -2.2836 0.096945 0.66131 0.16857 -0.028877 -0.10791 -0.027445 -0.25695 0.046686 0.23087 -0.076458 0.27127 0.25185 0.054947 -0.36673 -0.38603 0.3029 0.015747 0.34036 0.47841 0.068617 0.18351 -0.29183 -0.046533\n",
      "\n",
      "to -0.25756 -0.057132 -0.6719 -0.38082 -0.36421 -0.082155 -0.010955 -0.082047 0.46056 -1.8477 -0.11258 -0.12955 0.27254 0.0072891 0.26038 0.12096 -0.23193 0.03226 -0.29472 -0.67594 -0.33844 -0.23297 0.1102 0.18816 -0.45184 -0.33833 0.11274 0.4949 -0.042132 0.079961 -0.013146 0.062284 0.20223 0.038279 -1.1154 -0.1214 0.089846 0.29702 -0.055794 -0.46021 -0.13194 0.087357 -0.27865 0.14981 0.25536 0.16698 -0.04452 0.067588 -0.11772 -0.13452 0.28694 -0.39844 -0.12806 -0.47818 0.067802 0.20353 -0.30677 0.60789 -0.18588 0.11997 -0.040508 -0.06586 0.30621 -0.055824 0.039448 -0.4557 0.21081 0.25889 0.14666 0.3095 0.14343 0.10524 0.15788 0.103 0.32211 -0.27939 -0.17139 0.32202 0.10784 -0.28209 0.12611 -0.23913 -0.089638 -0.39179 -0.26402 0.36796 -0.23691 0.62503 -0.027226 -0.038851 -0.37359 0.045442 -0.17169 -0.54477 -0.091772 -0.32952 -0.25522 0.087106 0.048685 -0.31684 -0.064198 0.044885 -0.49587 0.429 0.36198 -0.14682 0.20122 -0.030038 0.18736 0.31626 0.059023 -0.13473 -0.40664 -0.52983 0.072459 0.082531 -0.32495 -0.006691 0.43564 -0.12976 -0.26293 0.15975 0.25062 -0.064732 -0.30689 0.16084 0.4583 0.38565 0.12612 -0.00080485 0.18055 0.24757 0.5568 0.21701 0.33105 0.11991 0.0027257 -0.10679 -0.16922 0.25277 0.36024 -0.4762 -0.20035 0.38473 -0.71653 -0.13788 -0.12201 -0.16979 -0.29624 -0.010344 0.19216 0.063375 0.30977 -0.19759 0.57419 0.34018 -0.34795 -0.085304 -0.028243 -0.39182 0.23797 -0.092997 0.11981 -0.2368 0.098179 0.083047 0.12652 0.16026 -0.14166 0.14296 -0.12868 0.18181 -0.42337 0.18061 -0.051635 -0.17987 -0.097071 0.0013755 -0.42539 0.56817 0.057432 0.31577 0.18918 0.27639 -0.39906 0.18594 -0.54231 0.61376 0.37921 0.1366 -0.39842 0.58557 0.054558 0.061801 0.32546 -0.19615 -0.034329 -0.013225 0.67854 -0.16672 0.92439 0.11952 0.040351 0.35368 -0.25573 0.26648 -0.020954 0.18535 0.062376 -0.22976 -0.074563 -0.3289 0.28535 0.46959 -0.54324 0.14124 0.19964 0.1541 0.024155 0.144 0.30989 -0.15989 -0.11611 0.09102 -0.50317 -0.33662 0.059168 0.43838 -0.1428 -0.0053718 0.046505 0.21546 0.065006 -0.35175 -0.17137 0.20467 -0.07173 0.40879 0.20295 -0.014211 -0.21898 -0.12831 0.3224 0.17608 -0.60267 0.036737 0.54848 -0.47682 -0.56556 -0.083633 0.032302 -0.25262 0.39481 -0.019623 0.62547 -0.11369 -0.25727 0.073363 0.18437 0.14587 0.32708 -0.52049 0.037555 0.023667 -0.068237 -0.22916 0.017755 -0.18394 0.55107 -0.23965 0.39187 -0.017785 0.43113 0.27181 -0.16043 -0.347 -2.4194 -0.028952 0.95085 0.05804 -0.23623 0.18914 0.31192 0.23064 -0.30309 -0.18603 0.07618 0.37337 -0.14444 -0.028793 -0.012806 -0.59707 0.31734 -0.25267 0.54384 0.063007 -0.049795 -0.16043 0.046744 -0.070621\n",
      "\n",
      "and 0.038466 -0.039792 0.082747 -0.38923 -0.21431 0.1702 -0.025657 0.09578 0.2386 -1.6342 0.14332 -0.037958 -0.019583 0.38494 0.097319 0.29697 -0.34523 0.11742 -0.024189 0.16013 0.09824 0.12811 -0.17482 0.20976 -0.22362 -0.20656 0.24428 0.066875 -0.12594 -0.015706 0.064986 0.4754 -0.055405 0.54286 -0.75188 -0.083218 0.17896 0.073084 -0.3033 -0.17416 -0.17147 -0.1192 0.038308 -0.2066 0.088679 -0.055993 0.361 0.38658 -0.055434 0.097699 0.3686 -0.326 0.13023 -0.29897 -0.24709 0.051869 0.030422 0.18586 -0.046117 -0.14765 0.35895 0.10094 -0.087822 -0.17514 -0.25403 -0.35855 0.15801 -0.027074 0.12565 -0.17509 -0.13126 -0.13916 0.053628 -0.049429 0.051938 -0.048684 0.071719 0.080952 -0.20018 -0.10871 -0.26707 -0.35727 0.3712 0.016709 -0.034959 -0.047711 0.0024827 0.10847 0.0089053 -0.14874 0.046014 0.42702 -0.24684 0.12193 -0.27579 0.25844 -0.20991 -0.086667 0.14767 -0.17441 0.17054 -0.30868 -0.08797 -0.17195 -0.11743 0.12146 0.069268 0.13311 -0.13565 -0.24855 -0.0026393 -0.71169 -0.32594 -0.36397 0.053331 0.35714 -0.30035 0.041583 -0.11996 -0.02368 0.016728 0.15869 -0.16491 -0.2782 -0.13788 0.10178 -0.24177 0.096347 0.2367 0.20885 -0.28867 0.10772 0.15562 -0.012284 -0.22161 -0.1017 0.1257 -0.29579 -0.0089604 0.35075 0.020088 0.020389 0.0038884 0.31869 -0.19848 0.060458 0.28148 -0.23499 -0.17035 -0.22323 0.57975 0.26464 -0.10124 0.081058 0.14029 0.066431 -0.12212 0.040034 0.027548 -0.1476 0.31438 -0.23046 0.2645 -0.10945 -0.39033 0.1266 -0.030951 0.067389 0.16807 0.22373 0.13502 0.38235 -0.52807 0.54013 -0.043062 0.093211 0.045211 -0.24291 -0.49781 0.26425 0.0264 0.14347 0.11763 -0.011614 0.097932 -0.26632 -0.22143 0.25156 0.08128 0.10937 -0.12199 0.019255 -0.33463 -0.18181 0.064724 0.22921 -0.032425 -0.27295 0.31983 0.16134 0.93692 -0.1214 -0.012617 0.25274 0.24615 0.13214 0.16092 0.20576 -0.051567 -0.3784 0.19761 0.16993 -0.087151 0.026922 0.38035 0.083349 0.24715 -0.1094 0.15459 -0.051741 0.16604 -0.21335 -0.030744 -0.14574 -0.50462 0.34825 -0.12343 0.17733 0.2857 -0.30467 -0.15095 0.30346 -0.15678 0.064804 -0.073008 0.26499 0.16312 0.11889 -0.63938 0.15598 -0.23643 0.59644 0.38748 0.3358 -0.58647 0.12584 0.36144 -0.33622 0.38128 -0.10348 0.18825 -0.33686 0.0058178 -0.1345 0.55511 -0.056443 0.15094 -0.28438 0.025488 0.20392 0.052712 -0.45719 0.089267 0.095469 -0.19022 -0.17101 -0.37599 -0.182 -0.065605 -0.061388 -0.19467 -0.070368 -0.23977 0.39253 -0.21283 0.17221 -1.867 -0.22609 0.53976 -0.3358 -0.48587 -0.050246 0.31694 -0.15536 0.12244 0.38356 -0.1389 0.41886 0.23664 -0.31113 0.045194 -0.20405 -0.21097 -0.11025 0.021766 0.44129 0.32797 -0.33427 0.011807 0.059703\n",
      "\n",
      "in -0.44399 0.12817 -0.25247 -0.18582 -0.16614 0.25909 -0.22678 -0.069229 -0.077204 -1.5814 0.10753 0.15123 -0.039371 0.2006 0.15792 0.43324 -0.21626 0.025275 -0.16592 -0.29108 0.0021523 0.084663 0.36784 -0.032838 0.092125 -0.40108 0.1446 -0.25876 -0.11703 0.098804 -0.04319 0.022361 -0.37503 0.44156 -0.67719 -0.21881 -0.11522 0.18124 -0.58965 0.18788 0.2861 -0.28068 -0.0632 0.54372 -0.025678 0.11458 0.031563 0.52909 -0.088527 0.28774 -0.10665 0.09264 0.1355 -0.18419 0.4314 -0.020869 -0.25236 -0.22679 0.45321 0.043271 -0.13449 0.0049912 -0.14535 -0.29764 0.16016 -0.6684 0.27327 0.15247 0.073077 -0.19881 -0.087157 -0.044633 0.1115 -0.051241 -0.70671 0.023404 0.36067 -0.055768 -0.22948 -0.11894 0.08605 -0.15969 -0.34213 0.2975 0.23921 0.093323 -0.10786 -0.054065 -0.21083 0.13899 0.11314 0.23076 -0.0022573 0.24149 -0.080955 0.04065 -0.378 0.0082898 -0.4385 -0.28115 0.25843 0.15749 -0.043514 0.10557 -0.14622 0.1586 0.19999 0.42149 0.15598 -0.0075512 -0.098827 -0.29451 -0.25065 -0.10596 0.31253 0.029076 0.058986 0.23656 -0.20147 -0.3718 -0.064728 -0.31944 0.28854 -0.25287 -0.16296 -0.081592 0.32626 -0.17376 0.29944 -0.16743 -0.21161 0.262 0.40972 -0.04841 -0.19614 -0.055705 -0.48783 -0.063714 -0.12616 0.55167 -0.12978 -0.20938 -0.14516 0.28025 -0.33003 -0.10742 0.35705 0.019434 -0.19469 0.13617 0.29435 -0.1825 0.092357 0.14214 0.20934 -0.099252 -0.25698 0.11528 0.1976 -0.24459 -0.085121 -0.26033 -0.09665 -0.099706 0.1462 0.12409 0.44139 -0.082185 0.2398 0.33155 0.29472 -0.02574 -0.6724 0.17099 -0.44336 0.40155 0.035535 -0.33381 -0.30127 0.59352 0.06213 0.22637 0.12551 0.059679 0.55479 0.10582 -0.48949 0.33181 -0.07972 -0.35039 0.032213 0.20024 0.36333 -0.12796 0.003225 -0.045665 0.09832 -0.070857 -0.34799 0.27443 1.1264 0.053398 -0.079054 -0.13053 0.074163 -0.22822 -0.040045 0.25848 -0.41994 0.0017782 0.1322 0.22764 -0.20604 -0.013716 0.48998 0.63756 -0.11187 0.11135 -0.19104 -0.13215 0.0023801 -0.24815 -0.48574 0.19432 0.065064 -0.15393 0.14434 0.10069 -0.058673 -0.18539 0.050425 0.27535 -0.24751 -0.050139 0.29191 0.018815 0.36727 0.12812 -0.10293 0.16095 -0.30712 -0.17172 0.39917 0.3457 -0.53083 -0.3528 0.50422 0.14786 -0.071239 -0.019673 0.12915 0.15139 0.02374 0.059252 0.36249 -0.20221 0.21818 0.031545 0.36705 -0.046894 0.13735 -0.057606 0.027779 -0.033237 0.050452 -0.42195 -0.29115 -0.28406 0.25065 -0.23917 -0.16317 -0.1791 -0.12767 0.50467 0.21975 -0.32816 -2.1376 0.087901 0.47143 0.48375 -0.55058 -0.081249 -0.22943 -0.10266 -0.1082 0.37007 -0.4184 0.4504 -0.09231 -0.0079179 -0.2745 -0.037237 0.10104 0.10798 0.37727 0.87977 0.33583 -0.20043 -0.082191 -0.06255\n",
      "\n",
      "a -0.29712 0.094049 -0.096662 -0.344 -0.18483 -0.12329 -0.11656 -0.099692 0.17265 -1.6386 0.1022 0.034253 -0.24213 0.048212 0.2706 0.35996 -0.40779 0.23785 0.24245 -0.31107 -0.48287 0.26101 0.11878 0.51011 -0.31115 -0.37005 0.11625 -0.42061 -0.034391 0.11809 -0.48955 0.33693 -0.13387 0.34924 -0.837 0.56691 -0.5367 0.38559 -0.28975 0.6247 0.48489 0.36357 0.35868 0.089597 -0.05536 0.11218 0.25683 -0.076713 -0.19422 -0.18243 -0.32841 0.064014 0.051971 0.20494 0.099948 -0.2716 -0.46416 0.10966 0.38606 -0.38436 -0.38772 0.28296 0.62533 -0.21405 -0.3754 -0.57913 -0.1251 -0.17104 0.43539 -0.18758 0.017773 0.35395 0.30074 -0.19863 -0.37743 0.11643 -0.18239 0.021961 -0.37256 0.023748 -0.096721 0.1289 -0.064794 -0.23133 0.45646 0.14918 -0.039145 0.39048 -0.113 -0.031962 0.036758 0.24962 -0.11699 -0.57764 0.57354 -0.14273 0.2503 -0.059743 -0.18562 -0.068998 -0.03661 0.15586 0.19729 -0.33594 -0.16771 0.17228 0.12988 -0.064598 -0.60267 -0.43089 -0.288 0.5353 -0.085713 0.089749 0.93015 0.16516 -0.14327 -0.060793 0.065698 -0.011137 0.24506 0.3287 0.012378 0.34808 0.047224 -0.20778 0.62321 0.14988 0.30187 -0.32872 0.3404 0.22678 -0.53792 -0.11482 0.41729 0.15157 0.15916 -0.12746 0.23922 0.202 -0.20951 0.36117 0.11522 -0.028047 -0.72466 -0.011893 0.031583 0.1739 -0.1262 0.20053 0.33085 -0.26827 0.28153 -0.044422 0.46704 0.049405 -0.46265 -0.27114 0.097589 0.30885 -0.10546 -0.018832 0.18444 -0.015528 0.047652 -0.1205 -0.2874 0.084141 0.53238 0.2009 0.035318 -0.44411 -0.86037 0.2761 -0.16407 -0.23766 -0.53769 -0.046391 0.55086 0.24044 0.31387 0.24542 0.19868 0.27393 -0.24355 -0.4129 -0.46545 0.60012 -0.26015 0.1929 0.36024 0.18589 0.037866 -0.077942 -0.099914 0.034607 -0.12247 -0.27205 -0.13819 0.23319 0.96767 0.34317 0.044022 -0.31471 0.014663 -0.071928 -0.45375 0.029313 -0.22576 0.20704 0.029607 -0.41761 0.2603 0.23784 0.083533 0.12032 -0.19662 -0.2885 -0.25891 0.71359 0.28543 -0.0017777 -0.10196 0.47297 -0.006194 -0.024529 0.11389 0.059219 0.18821 -0.15918 0.19517 0.23143 -0.016216 -0.51149 0.36894 0.55447 0.17006 -0.081063 0.14756 0.43106 0.62111 -0.020083 -0.051027 -0.42335 -0.95094 -0.098285 0.49542 -0.1441 0.18742 -0.1606 -0.073652 0.067304 -0.097287 -0.27716 0.80758 0.20659 -0.34734 -0.17897 0.48319 0.59006 -0.22616 0.28727 0.49545 0.41908 -0.37884 0.023117 -0.11783 0.079173 0.5605 -0.17425 -0.33199 -0.72821 0.10046 0.041775 0.22655 -0.12865 -2.3832 0.2206 0.51407 0.08616 -0.21772 -0.055862 -0.040358 -0.22208 -0.042596 0.037001 -0.50139 0.10573 -0.16145 0.3616 0.075972 -0.42426 -0.3967 0.32683 0.62049 0.34719 0.26952 0.059717 -0.22853 0.29602\n",
      "\n",
      "\" 0.6947 0.22184 0.10526 0.012382 -0.2558 -0.32645 -0.48287 0.51755 -0.0872 -2.0289 0.35021 0.045363 -0.58554 -0.16147 0.35592 -0.12698 -0.389 0.027275 0.10101 0.010271 0.30711 0.32771 0.11769 0.24432 0.23224 0.88493 0.084191 0.019132 0.41142 0.17872 0.28685 0.32401 -0.1452 0.13602 -1.0021 0.15376 0.27661 -0.24772 -0.62897 0.47619 -0.35871 -0.82761 -0.41504 0.43143 -0.0085019 0.019796 0.30738 -0.06202 -0.41845 -0.0065542 -0.5164 0.14782 -0.0714 0.14323 -0.012224 0.22951 -0.086194 0.29748 0.46222 -0.56759 0.2868 0.068718 0.25969 0.1065 -0.38421 -0.24288 0.3652 -0.28178 0.02779 0.21189 -0.21838 -0.26429 -0.2053 -0.050045 -0.14675 0.43928 0.12568 0.2448 0.23945 -0.13081 0.061735 0.44339 0.66748 -0.30867 0.31239 0.050992 -0.20818 0.53632 0.017549 -0.62452 -0.017933 -0.15915 0.04842 -0.18007 -0.2852 0.24385 0.04839 0.0049169 0.070413 -0.25744 0.24731 0.076947 0.053461 0.36677 -0.17137 0.089144 -0.17755 -0.44276 0.58558 0.15683 0.04322 0.016737 0.026973 -0.44068 0.19796 -0.65142 0.0039067 0.064482 -0.18919 -0.42223 0.37854 -0.46292 0.01278 -0.080519 0.26873 -0.56258 -0.15363 0.53182 -0.091645 -0.49802 -0.1925 0.18915 -0.17388 -0.14453 -0.033144 -0.13589 -0.52306 -0.60742 0.066981 0.049356 0.039682 0.70251 0.19068 -0.0053267 0.026991 -0.45241 -0.11205 0.38563 -0.26384 0.66098 0.25433 -0.048764 -0.19554 0.21586 0.086974 -0.3031 0.23631 -0.12765 -0.013964 -0.5626 0.23265 -0.67844 0.083024 -0.33026 -0.83967 0.24366 -0.079742 -0.12785 -0.29069 0.075679 0.52596 0.26064 -0.17329 -0.27323 0.3122 0.32645 0.22285 0.86193 0.28423 -0.23161 0.50005 -0.081095 -0.14249 -0.33155 -0.39187 -0.33697 -0.28684 -0.077314 0.56339 0.14105 0.20169 0.53144 0.94135 -0.22078 -0.32236 -0.19445 -0.63195 -0.55729 -0.3093 -0.36269 1.3423 0.45712 0.095516 1.0522 -0.025737 -0.16478 -0.10284 0.49699 0.2979 0.084515 -0.37817 -0.32549 -0.25269 -0.19468 0.66512 -0.30682 0.27146 0.66712 0.28062 -0.63802 -0.20536 -0.058433 -0.3065 -0.10404 -0.35549 0.1365 0.071077 -0.24535 0.070016 -0.19833 -0.10835 -0.29371 -0.25792 -0.20339 -0.062024 -0.043849 -0.31949 -0.36047 -0.070991 -0.21235 -0.46852 -0.484 -0.16648 0.23164 -0.32893 0.087816 -0.081276 0.21032 0.40517 -0.051471 -0.091353 0.058312 -0.76205 0.39375 0.29967 -0.23293 0.19127 -0.21633 0.56827 0.014206 -0.16902 -0.53718 -0.20777 0.17868 -0.27838 -0.075129 -0.055254 -0.27411 0.019909 0.57986 0.077334 -0.046309 -0.286 -0.16093 0.30357 -0.11045 -1.8737 0.04662 0.88636 -0.074213 -0.21623 0.14911 -0.74491 0.15774 0.22384 -0.0085548 0.065872 0.056001 0.37591 0.080234 -0.32401 -0.39791 -0.35475 -0.064987 -0.35959 0.27407 0.28326 0.34871 -0.63079 0.38089\n",
      "\n",
      "'s -0.001272 0.36514 -0.077363 -0.26559 0.17987 0.15347 -0.15338 0.43267 -0.13364 -1.716 0.069153 -0.0079778 0.17582 0.28543 0.38667 0.44506 0.37369 0.17313 -0.19106 0.29733 0.83032 0.15301 0.25848 -0.046363 -0.32512 0.17635 0.16375 -0.021848 -0.33696 -0.3572 -0.14684 0.73045 -0.096603 0.35377 -1.2056 0.51558 0.24781 -0.1109 -0.17139 -0.47703 0.15555 0.24988 -0.12376 0.62701 -0.31196 0.19281 -0.10957 0.2915 -0.66135 -0.21631 -0.07754 -0.10975 0.070786 0.43514 0.33533 -0.0010721 -0.1285 0.54657 0.11623 0.063463 -0.16129 0.27799 0.023294 -0.16092 -0.020701 -0.042522 -0.013445 0.21933 0.13208 -0.08651 -0.15157 0.045004 0.22081 -0.3302 0.56354 0.13779 0.18951 -0.034288 -0.39261 0.479 0.025599 0.20667 -0.14304 -0.10036 0.26442 0.1191 -0.22048 -0.089946 -0.20849 -0.33173 -0.0465 -0.039867 -0.23478 -0.27917 0.009119 -0.36536 -0.25293 0.027104 0.11066 -0.39852 0.10786 0.074682 0.030254 0.2018 0.11648 -0.24248 0.38339 0.63444 -0.16731 0.24386 0.0578 -0.10832 0.016448 -0.077361 0.11561 0.11861 0.1586 0.33599 0.15333 -0.47698 -0.11412 -0.15998 0.18273 -0.46594 -0.022473 -0.22373 -0.28015 0.16083 0.063174 0.16524 0.17235 0.78742 -0.43266 -0.24705 0.34706 0.13913 0.16906 -0.19882 -0.10414 0.26749 0.17295 0.078849 0.58617 -0.24936 -0.30504 -0.40331 0.034397 -0.21471 -0.30321 -0.28646 0.5263 0.11751 0.24854 -0.14418 0.70013 0.065482 -0.20346 0.64396 0.63971 0.03147 0.39379 0.027267 0.80518 0.22757 -0.18351 -0.39773 -0.066513 0.15471 -0.13131 0.091 0.28648 0.48654 -1.2863 -0.352 -0.143 -0.10963 0.34816 0.1008 -0.45295 0.35414 0.2452 0.50821 0.13088 -0.0088304 0.11829 0.48728 0.037797 -0.31463 -0.84104 0.26889 -0.7261 0.037926 -0.61905 -0.090255 -0.060257 0.50222 0.2757 -0.034749 -0.35541 0.071836 1.7241 -0.53972 0.32166 -0.25462 -0.22476 0.21367 -0.25695 0.10421 0.25238 -0.038572 -0.31734 0.037501 0.28885 -0.4438 0.33324 0.24069 -0.23782 -0.35156 0.47037 -0.23308 0.096621 -0.083021 -0.32734 0.31101 -0.1255 0.18077 0.1032 -0.27682 -0.86378 -0.33937 -0.32415 0.070228 -0.0019138 -0.0039554 -0.071345 0.1667 0.31996 0.24855 0.6906 0.27341 0.16493 -0.35739 0.24218 -0.064732 -0.45739 0.16461 -0.12408 -0.30335 -0.24301 0.22162 0.32573 -0.041798 0.26151 -0.12885 0.55704 -0.34012 -0.12046 0.17767 0.23785 -0.50045 -0.19556 -0.11327 0.081786 -0.43105 -0.28843 0.39246 -0.017525 -0.089589 -0.021078 0.57783 -0.12208 -0.01884 0.39227 0.10745 0.18686 -0.60034 -1.7264 -0.35302 0.53372 0.40274 0.2427 0.26233 -0.21278 0.26437 -0.20814 0.13909 0.066891 0.23215 -0.20509 0.099417 -0.32295 0.01604 0.2739 0.046497 0.22002 0.50164 0.63324 -0.18361 -0.76518 0.39206\n",
      "\n"
     ]
    }
   ],
   "source": [
    "with open('../../glove.6B/glove.6B.300d.txt') as f:\n",
    "    for _ in range(10): #first 10 lines\n",
    "        print(f.readline())"
   ]
  },
  {
   "cell_type": "raw",
   "id": "862383fc",
   "metadata": {},
   "source": [
    "!head ../../glove.6B/glove.6B.300d.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "5b105b32",
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_glove_vectors(file_path):\n",
    "    print(\"Loading Glove Model.\")\n",
    "    glove_model = {}\n",
    "    with open(file_path,'r',encoding=\"utf8\") as f:\n",
    "        for line in f:\n",
    "            split_line = line.split()\n",
    "            word = split_line[0]\n",
    "            embedding = np.array(split_line[1:],dtype=np.float64)\n",
    "            glove_model[word] = embedding\n",
    "        print(f'{len(glove_model)} words loaded.')\n",
    "    return glove_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "ce06b0f9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading Glove Model.\n",
      "400000 words loaded.\n"
     ]
    }
   ],
   "source": [
    "# Load the embeddings!\n",
    "\n",
    "GLOVE_PATH = \"../../glove.6B/glove.6B.300d.txt\"\n",
    "\n",
    "glove = load_glove_vectors(GLOVE_PATH)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "bbb47f04",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 400000 tokens/words having GloVe Embeddings.\n",
      "The Shape of each embedding =====> (300,)\n"
     ]
    }
   ],
   "source": [
    "#we take a word and obtain the shape of its embedding.\n",
    "#On the assumption that every word in the glove model has the same number of embeddings.\n",
    "print(f'There are {len(glove)} tokens/words having GloVe Embeddings.') \n",
    "print(f'The Shape of each embedding =====> {np.array(list(glove.values())[0]).shape}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f3930ce",
   "metadata": {},
   "source": [
    "- There are 400000 words/token that have GloVe embeddings.\n",
    "- Each embedding has a shape of (300,) i.e (300 by 1)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "b8a80043",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_iter = [(int(x['sentiment']), x['summary']+ ' ' +x['text'])  for x in train_examples]\n",
    "validation_iter = [(int(x['sentiment']), x['summary']+ ' ' +x['text'])  for x in validation_examples]\n",
    "test_iter = [(int(x['sentiment']), x['summary']+ ' ' +x['text'])  for x in test_examples]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "fc99d5bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = get_tokenizer('basic_english')\n",
    "\n",
    "counter = Counter()\n",
    "for (label, text) in train_iter:\n",
    "    counter.update(tokenizer(text))\n",
    "vocab = Vocab(counter, min_freq=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43181c92",
   "metadata": {},
   "source": [
    "##  Sequence Padding\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "d1521cf2",
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_sequence(text,pad_len,pad_token='<pad>'):\n",
    "    sequence_words,sequence_id = [],[]\n",
    "    tokenized_words = tokenizer(text)\n",
    "    if len(tokenized_words) <=100:\n",
    "        remainder_length = pad_len - len(tokenized_words)\n",
    "        sequence_words = tokenized_words+[pad_token]*remainder_length\n",
    "    else:\n",
    "        sequence_words = tokenized_words[:100]\n",
    "    for word in sequence_words:\n",
    "        sequence_id.append(vocab.stoi[word])\n",
    "    return sequence_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "42bf03ed",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([  8,  10,  26, 113,   2,   2,   1,   1,   1,   1,   1,   1,   1,\n",
       "         1,   1,   1,   1,   1,   1,   1,   1,   1,   1,   1,   1,   1,\n",
       "         1,   1,   1,   1,   1,   1,   1,   1,   1,   1,   1,   1,   1,\n",
       "         1,   1,   1,   1,   1,   1,   1,   1,   1,   1,   1,   1,   1,\n",
       "         1,   1,   1,   1,   1,   1,   1,   1,   1,   1,   1,   1,   1,\n",
       "         1,   1,   1,   1,   1,   1,   1,   1,   1,   1,   1,   1,   1,\n",
       "         1,   1,   1,   1,   1,   1,   1,   1,   1,   1,   1,   1,   1,\n",
       "         1,   1,   1,   1,   1,   1,   1,   1,   1])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pad_len = 100\n",
    "pad_token = '<pad>'\n",
    "\n",
    "text_pipeline = lambda x: [j for j in generate_sequence(x,pad_len)]\n",
    "## check text_pipeline functionality\n",
    "np.array(text_pipeline(train_examples[90]['summary']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "fd921b9e",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Prepare dataloaders\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "def collate_batch(batch):\n",
    "    label_list, text_list = [], []\n",
    "    for (_label, _text) in batch:\n",
    "         label_list.append(_label)\n",
    "         processed_text = torch.unsqueeze(torch.tensor(text_pipeline(_text), dtype=torch.int64), 0)\n",
    "         text_list.append(processed_text)\n",
    "\n",
    "    label_list = torch.tensor(label_list, dtype=torch.int64)\n",
    "    \n",
    "    text_list = torch.cat(text_list, dim=0)\n",
    "    return label_list.to(device), text_list.to(device)\n",
    "\n",
    "train_dataloader = DataLoader(list(train_iter), batch_size=8, shuffle=True, collate_fn=collate_batch)\n",
    "validation_dataloader = DataLoader(list(validation_iter), batch_size=8, shuffle=False, collate_fn=collate_batch)\n",
    "test_dataloader = DataLoader(list(test_iter), batch_size=8, shuffle=False, collate_fn=collate_batch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "7a3a9aec",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of each input tensors(text) ====> torch.Size([8, 100])\n",
      "Shape of each output tensors(labels) ======> torch.Size([8])\n"
     ]
    }
   ],
   "source": [
    "label, text = next(iter(train_dataloader))\n",
    "print(f'Shape of each input tensors(text) ====> {text.shape}')\n",
    "print(f'Shape of each output tensors(labels) ======> {label.shape}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5666001b",
   "metadata": {},
   "source": [
    "- Shape of input tensors(text) is (8, 100)\n",
    "- Shape of output tensors(labels) is (8)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92459333",
   "metadata": {},
   "source": [
    "# GloVe Embedding Layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "bd3cbfd0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(15030, 300)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "matrix_len = len(glove.keys())\n",
    "glove_dim = len(list(glove.values())[0])\n",
    "weights_matrix = [glove[token] if token in glove.keys() else np.zeros(glove_dim) for token in list(vocab.stoi.keys())] \n",
    "weights_matrix = np.array(weights_matrix)\n",
    "## Confirm weight_matrix shape.\n",
    "weights_matrix.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "8dbbe2b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "def glove_emb_layer(weights_matrix, trainable=False):\n",
    "    num_embeddings, embedding_dim = weights_matrix.shape\n",
    "    emb_layer = torch.nn.Embedding.from_pretrained(torch.from_numpy(weights_matrix.astype(np.float32)))\n",
    "\n",
    "    if not trainable:\n",
    "        emb_layer.weight.requires_grad = False\n",
    "\n",
    "    return emb_layer, num_embeddings, embedding_dim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "e7b9f122",
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(model, dataloader, criterion, optimizer, epoch):\n",
    "    model.train()\n",
    "    total_acc, total_count = 0, 0\n",
    "    log_interval = 100\n",
    "    start_time = time.time()\n",
    "\n",
    "    for idx, (label, text) in tqdm(enumerate(dataloader), total=len(dataloader)):\n",
    "        optimizer.zero_grad()\n",
    "        predicted_label = model(text)\n",
    "        loss = criterion(predicted_label, label.float())\n",
    "        loss.backward()\n",
    "        torch.nn.utils.clip_grad_norm_(model.parameters(), 0.1)\n",
    "        optimizer.step()\n",
    "        total_acc += (predicted_label * label > 0).sum().item()\n",
    "        total_count += label.size(0)\n",
    "        if idx % log_interval == 0 and idx > 0:\n",
    "            elapsed = time.time() - start_time\n",
    "            print('| epoch {:3d} | {:5d}/{:5d} batches '\n",
    "                  '| train accuracy {:8.3f}'.format(epoch, idx, len(dataloader),\n",
    "                                              total_acc/total_count))\n",
    "            total_acc, total_count = 0, 0\n",
    "            start_time = time.time()\n",
    "\n",
    "def evaluate(model, dataloader, criterion):\n",
    "    model.eval()\n",
    "    total_acc, total_count = 0, 0\n",
    "\n",
    "    with torch.no_grad():\n",
    "        for idx, (label, text) in enumerate(dataloader):\n",
    "            predicted_label = model(text)\n",
    "            loss = criterion(predicted_label, label)\n",
    "            total_acc += (predicted_label * label > 0).sum().item()\n",
    "            total_count += label.size(0)\n",
    "    return total_acc/total_count\n",
    "\n",
    "def train_loop(model, criterion):\n",
    "    # Hyperparameters\n",
    "    EPOCHS = 15 # epoch\n",
    "    LR = 5  # learning rate\n",
    "    BATCH_SIZE = 64\n",
    "\n",
    "    optimizer = torch.optim.SGD(model.parameters(), lr=LR)\n",
    "    scheduler = torch.optim.lr_scheduler.StepLR(optimizer, 1.0, gamma=0.1)\n",
    "    total_accu = None\n",
    "\n",
    "    train_dataset = list(train_iter)\n",
    "    test_dataset = list(test_iter)\n",
    "    num_train = int(len(train_dataset) * 0.95)\n",
    "\n",
    "    for epoch in range(1, EPOCHS + 1):\n",
    "        epoch_start_time = time.time()\n",
    "\n",
    "        train(model, train_dataloader, criterion, optimizer, epoch)\n",
    "        accu_val = evaluate(model, validation_dataloader, criterion)\n",
    "        if total_accu is not None and total_accu > accu_val:\n",
    "          scheduler.step()\n",
    "        else:\n",
    "           total_accu = accu_val\n",
    "        print('-' * 59)\n",
    "        print('| end of epoch {:3d} | time: {:5.2f}s | '\n",
    "              'valid accuracy {:8.3f} '.format(epoch,\n",
    "                                               time.time() - epoch_start_time,\n",
    "                                               accu_val))\n",
    "        print('-' * 59)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9487544",
   "metadata": {},
   "source": [
    "# Embedding Bag Model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "317d85e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "class EmbeddingBagModel(torch.nn.Module):\n",
    "\n",
    "    def __init__(self, vocab_size, embed_dim):\n",
    "        super(EmbeddingBagModel, self).__init__()\n",
    "        self.hidden = torch.nn.EmbeddingBag(vocab_size, embed_dim)\n",
    "        self.output = torch.nn.Linear(embed_dim,1)\n",
    "\n",
    "    def forward(self, text):\n",
    "        hidden = self.hidden(text)\n",
    "        hidden = self.output(hidden)\n",
    "        final_output = hidden.squeeze()\n",
    "        return final_output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "e2326907",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6563419224004940b6faa09c6315a682",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch   1 |   100/  500 batches | train accuracy    0.524\n",
      "| epoch   1 |   200/  500 batches | train accuracy    0.557\n",
      "| epoch   1 |   300/  500 batches | train accuracy    0.596\n",
      "| epoch   1 |   400/  500 batches | train accuracy    0.632\n",
      "-----------------------------------------------------------\n",
      "| end of epoch   1 | time: 23.67s | valid accuracy    0.560 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b330b63f4c1a4a9bb4335df8d343ce65",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch   2 |   100/  500 batches | train accuracy    0.599\n",
      "| epoch   2 |   200/  500 batches | train accuracy    0.656\n",
      "| epoch   2 |   300/  500 batches | train accuracy    0.657\n",
      "| epoch   2 |   400/  500 batches | train accuracy    0.635\n",
      "-----------------------------------------------------------\n",
      "| end of epoch   2 | time: 22.06s | valid accuracy    0.664 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "52d34e8e1ba74c20931d9574fc270bf0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch   3 |   100/  500 batches | train accuracy    0.655\n",
      "| epoch   3 |   200/  500 batches | train accuracy    0.677\n",
      "| epoch   3 |   300/  500 batches | train accuracy    0.632\n",
      "| epoch   3 |   400/  500 batches | train accuracy    0.677\n",
      "-----------------------------------------------------------\n",
      "| end of epoch   3 | time: 21.93s | valid accuracy    0.748 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "aa6e50d8199548f291affc6842f6cd39",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch   4 |   100/  500 batches | train accuracy    0.673\n",
      "| epoch   4 |   200/  500 batches | train accuracy    0.682\n",
      "| epoch   4 |   300/  500 batches | train accuracy    0.681\n",
      "| epoch   4 |   400/  500 batches | train accuracy    0.696\n",
      "-----------------------------------------------------------\n",
      "| end of epoch   4 | time: 22.83s | valid accuracy    0.588 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "beeed32e0c8c44be97c26d51665ddb02",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch   5 |   100/  500 batches | train accuracy    0.750\n",
      "| epoch   5 |   200/  500 batches | train accuracy    0.777\n",
      "| epoch   5 |   300/  500 batches | train accuracy    0.769\n",
      "| epoch   5 |   400/  500 batches | train accuracy    0.771\n",
      "-----------------------------------------------------------\n",
      "| end of epoch   5 | time: 22.93s | valid accuracy    0.754 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "49141b66cefa43cab2aee60e5c36a282",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch   6 |   100/  500 batches | train accuracy    0.785\n",
      "| epoch   6 |   200/  500 batches | train accuracy    0.782\n",
      "| epoch   6 |   300/  500 batches | train accuracy    0.790\n",
      "| epoch   6 |   400/  500 batches | train accuracy    0.790\n",
      "-----------------------------------------------------------\n",
      "| end of epoch   6 | time: 20.72s | valid accuracy    0.758 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cabff13a9cd9427494e4ceb0ca864169",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch   7 |   100/  500 batches | train accuracy    0.795\n",
      "| epoch   7 |   200/  500 batches | train accuracy    0.791\n",
      "| epoch   7 |   300/  500 batches | train accuracy    0.792\n",
      "| epoch   7 |   400/  500 batches | train accuracy    0.802\n",
      "-----------------------------------------------------------\n",
      "| end of epoch   7 | time: 21.23s | valid accuracy    0.768 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6a82a1404138496a91a042e472a84c59",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch   8 |   100/  500 batches | train accuracy    0.769\n",
      "| epoch   8 |   200/  500 batches | train accuracy    0.794\n",
      "| epoch   8 |   300/  500 batches | train accuracy    0.780\n",
      "| epoch   8 |   400/  500 batches | train accuracy    0.794\n",
      "-----------------------------------------------------------\n",
      "| end of epoch   8 | time: 21.14s | valid accuracy    0.762 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "935d620b1a084f00a7675064d998c5da",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch   9 |   100/  500 batches | train accuracy    0.822\n",
      "| epoch   9 |   200/  500 batches | train accuracy    0.791\n",
      "| epoch   9 |   300/  500 batches | train accuracy    0.779\n",
      "| epoch   9 |   400/  500 batches | train accuracy    0.789\n",
      "-----------------------------------------------------------\n",
      "| end of epoch   9 | time: 21.11s | valid accuracy    0.770 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "24bf8bfb70e9482aad21f9f375642e83",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch  10 |   100/  500 batches | train accuracy    0.801\n",
      "| epoch  10 |   200/  500 batches | train accuracy    0.806\n",
      "| epoch  10 |   300/  500 batches | train accuracy    0.816\n",
      "| epoch  10 |   400/  500 batches | train accuracy    0.779\n",
      "-----------------------------------------------------------\n",
      "| end of epoch  10 | time: 20.73s | valid accuracy    0.764 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "221dd79baa1a4fe18645b91cfe044ab3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch  11 |   100/  500 batches | train accuracy    0.811\n",
      "| epoch  11 |   200/  500 batches | train accuracy    0.812\n",
      "| epoch  11 |   300/  500 batches | train accuracy    0.810\n",
      "| epoch  11 |   400/  500 batches | train accuracy    0.785\n",
      "-----------------------------------------------------------\n",
      "| end of epoch  11 | time: 21.39s | valid accuracy    0.764 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c75be1b3ea874659b423cb14bb4bb9a9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch  12 |   100/  500 batches | train accuracy    0.795\n",
      "| epoch  12 |   200/  500 batches | train accuracy    0.806\n",
      "| epoch  12 |   300/  500 batches | train accuracy    0.795\n",
      "| epoch  12 |   400/  500 batches | train accuracy    0.824\n",
      "-----------------------------------------------------------\n",
      "| end of epoch  12 | time: 23.29s | valid accuracy    0.766 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b9a99d71060f451b9c090312816d137e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch  13 |   100/  500 batches | train accuracy    0.804\n",
      "| epoch  13 |   200/  500 batches | train accuracy    0.792\n",
      "| epoch  13 |   300/  500 batches | train accuracy    0.820\n",
      "| epoch  13 |   400/  500 batches | train accuracy    0.799\n",
      "-----------------------------------------------------------\n",
      "| end of epoch  13 | time: 22.08s | valid accuracy    0.766 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "02ef15d9e22a4058bbe72ad6b39e72c6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch  14 |   100/  500 batches | train accuracy    0.819\n",
      "| epoch  14 |   200/  500 batches | train accuracy    0.820\n",
      "| epoch  14 |   300/  500 batches | train accuracy    0.795\n",
      "| epoch  14 |   400/  500 batches | train accuracy    0.787\n",
      "-----------------------------------------------------------\n",
      "| end of epoch  14 | time: 21.34s | valid accuracy    0.766 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7c9cf98731de4facb4e724a8bd9a78f2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch  15 |   100/  500 batches | train accuracy    0.788\n",
      "| epoch  15 |   200/  500 batches | train accuracy    0.816\n",
      "| epoch  15 |   300/  500 batches | train accuracy    0.809\n",
      "| epoch  15 |   400/  500 batches | train accuracy    0.811\n",
      "-----------------------------------------------------------\n",
      "| end of epoch  15 | time: 20.84s | valid accuracy    0.766 \n",
      "-----------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "criterion = torch.nn.MSELoss()\n",
    "vocab_size = len(vocab)\n",
    "emsize = 128\n",
    "embedding_bag_model = EmbeddingBagModel(vocab_size, emsize).to(device)\n",
    "\n",
    "train_loop(embedding_bag_model, criterion)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "9b5864e0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.766"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate(embedding_bag_model, validation_dataloader, criterion)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db608571",
   "metadata": {},
   "source": [
    "- The validation accuracy for our Embedding Bag model is 0.766 (76.6%)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec182810",
   "metadata": {},
   "source": [
    "# Average Embedding  Model\n",
    "\n",
    "GloVe embedding layer to convert our input sequence into a matrix of embeddings. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "40751d48",
   "metadata": {},
   "outputs": [],
   "source": [
    "class AverageEmbeddingModel(torch.nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "        super(AverageEmbeddingModel, self).__init__()\n",
    "        emb_layer, num_embeddings, embedding_dim = glove_emb_layer(weights_matrix, trainable=True)\n",
    "        self.embedding = emb_layer\n",
    "        self.output = torch.nn.Linear(embedding_dim,1)\n",
    "        \n",
    "    \n",
    "\n",
    "    def forward(self, text):\n",
    "        embedded = self.embedding(text)\n",
    "        avg_embedding = torch.mean(embedded,1)\n",
    "        output = self.output(avg_embedding)\n",
    "        final_output = output.squeeze()\n",
    "        return final_output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "2d1db50d",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "873b783fb2824b0ea05647383999aa89",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch   1 |   100/  500 batches | train accuracy    0.502\n",
      "| epoch   1 |   200/  500 batches | train accuracy    0.578\n",
      "| epoch   1 |   300/  500 batches | train accuracy    0.549\n",
      "| epoch   1 |   400/  500 batches | train accuracy    0.601\n",
      "-----------------------------------------------------------\n",
      "| end of epoch   1 | time:  4.00s | valid accuracy    0.622 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "eb69dcec85ca4313b7319f63ab7a6aeb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch   2 |   100/  500 batches | train accuracy    0.613\n",
      "| epoch   2 |   200/  500 batches | train accuracy    0.642\n",
      "| epoch   2 |   300/  500 batches | train accuracy    0.618\n",
      "| epoch   2 |   400/  500 batches | train accuracy    0.669\n",
      "-----------------------------------------------------------\n",
      "| end of epoch   2 | time:  2.64s | valid accuracy    0.646 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7598a61f50b6411a9963fb90c82316fb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch   3 |   100/  500 batches | train accuracy    0.656\n",
      "| epoch   3 |   200/  500 batches | train accuracy    0.656\n",
      "| epoch   3 |   300/  500 batches | train accuracy    0.681\n",
      "| epoch   3 |   400/  500 batches | train accuracy    0.654\n",
      "-----------------------------------------------------------\n",
      "| end of epoch   3 | time:  2.23s | valid accuracy    0.672 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2e0a839b7c184a8f8e74eb4a9c96d239",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch   4 |   100/  500 batches | train accuracy    0.651\n",
      "| epoch   4 |   200/  500 batches | train accuracy    0.657\n",
      "| epoch   4 |   300/  500 batches | train accuracy    0.690\n",
      "| epoch   4 |   400/  500 batches | train accuracy    0.662\n",
      "-----------------------------------------------------------\n",
      "| end of epoch   4 | time:  1.95s | valid accuracy    0.682 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "303d928f845f4c14b367db47fdc0249a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch   5 |   100/  500 batches | train accuracy    0.681\n",
      "| epoch   5 |   200/  500 batches | train accuracy    0.682\n",
      "| epoch   5 |   300/  500 batches | train accuracy    0.652\n",
      "| epoch   5 |   400/  500 batches | train accuracy    0.699\n",
      "-----------------------------------------------------------\n",
      "| end of epoch   5 | time:  1.96s | valid accuracy    0.670 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e5130f107dc04ef2ac36a5332cc33c02",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch   6 |   100/  500 batches | train accuracy    0.829\n",
      "| epoch   6 |   200/  500 batches | train accuracy    0.807\n",
      "| epoch   6 |   300/  500 batches | train accuracy    0.787\n",
      "| epoch   6 |   400/  500 batches | train accuracy    0.814\n",
      "-----------------------------------------------------------\n",
      "| end of epoch   6 | time:  2.44s | valid accuracy    0.764 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ba8d0b3c4fde4da0966dfcc6e4437ee3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch   7 |   100/  500 batches | train accuracy    0.803\n",
      "| epoch   7 |   200/  500 batches | train accuracy    0.804\n",
      "| epoch   7 |   300/  500 batches | train accuracy    0.790\n",
      "| epoch   7 |   400/  500 batches | train accuracy    0.826\n",
      "-----------------------------------------------------------\n",
      "| end of epoch   7 | time:  1.97s | valid accuracy    0.766 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e80578b946454465b8628b96a91bb150",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch   8 |   100/  500 batches | train accuracy    0.813\n",
      "| epoch   8 |   200/  500 batches | train accuracy    0.812\n",
      "| epoch   8 |   300/  500 batches | train accuracy    0.804\n",
      "| epoch   8 |   400/  500 batches | train accuracy    0.789\n",
      "-----------------------------------------------------------\n",
      "| end of epoch   8 | time:  2.01s | valid accuracy    0.782 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6d7d6ec95e8e498e887606153d529ed1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch   9 |   100/  500 batches | train accuracy    0.824\n",
      "| epoch   9 |   200/  500 batches | train accuracy    0.824\n",
      "| epoch   9 |   300/  500 batches | train accuracy    0.791\n",
      "| epoch   9 |   400/  500 batches | train accuracy    0.792\n",
      "-----------------------------------------------------------\n",
      "| end of epoch   9 | time:  1.94s | valid accuracy    0.786 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c546c501f7aa47b78f599cd577c6fc84",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch  10 |   100/  500 batches | train accuracy    0.802\n",
      "| epoch  10 |   200/  500 batches | train accuracy    0.790\n",
      "| epoch  10 |   300/  500 batches | train accuracy    0.823\n",
      "| epoch  10 |   400/  500 batches | train accuracy    0.802\n",
      "-----------------------------------------------------------\n",
      "| end of epoch  10 | time:  2.00s | valid accuracy    0.770 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ef27f07bd52e4195b74adcfedabe9df6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch  11 |   100/  500 batches | train accuracy    0.834\n",
      "| epoch  11 |   200/  500 batches | train accuracy    0.835\n",
      "| epoch  11 |   300/  500 batches | train accuracy    0.784\n",
      "| epoch  11 |   400/  500 batches | train accuracy    0.830\n",
      "-----------------------------------------------------------\n",
      "| end of epoch  11 | time:  1.98s | valid accuracy    0.796 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7734487d0b014416a43196ca85b45bd5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch  12 |   100/  500 batches | train accuracy    0.811\n",
      "| epoch  12 |   200/  500 batches | train accuracy    0.840\n",
      "| epoch  12 |   300/  500 batches | train accuracy    0.797\n",
      "| epoch  12 |   400/  500 batches | train accuracy    0.824\n",
      "-----------------------------------------------------------\n",
      "| end of epoch  12 | time:  1.95s | valid accuracy    0.790 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c45cee5969674a47a809a7f0c326f764",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch  13 |   100/  500 batches | train accuracy    0.847\n",
      "| epoch  13 |   200/  500 batches | train accuracy    0.818\n",
      "| epoch  13 |   300/  500 batches | train accuracy    0.821\n",
      "| epoch  13 |   400/  500 batches | train accuracy    0.819\n",
      "-----------------------------------------------------------\n",
      "| end of epoch  13 | time:  1.95s | valid accuracy    0.786 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b305e2a7316f40c296070067aefb8fbd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch  14 |   100/  500 batches | train accuracy    0.839\n",
      "| epoch  14 |   200/  500 batches | train accuracy    0.818\n",
      "| epoch  14 |   300/  500 batches | train accuracy    0.829\n",
      "| epoch  14 |   400/  500 batches | train accuracy    0.828\n",
      "-----------------------------------------------------------\n",
      "| end of epoch  14 | time:  2.40s | valid accuracy    0.786 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "704c1f0eeece431a881ed242ed6a6bb1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch  15 |   100/  500 batches | train accuracy    0.817\n",
      "| epoch  15 |   200/  500 batches | train accuracy    0.841\n",
      "| epoch  15 |   300/  500 batches | train accuracy    0.826\n",
      "| epoch  15 |   400/  500 batches | train accuracy    0.818\n",
      "-----------------------------------------------------------\n",
      "| end of epoch  15 | time:  1.96s | valid accuracy    0.786 \n",
      "-----------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "criterion = torch.nn.MSELoss()\n",
    "vocab_size = len(vocab)\n",
    "emsize = 128\n",
    "embedding_bag_model = AverageEmbeddingModel().to(device)\n",
    "\n",
    "train_loop(embedding_bag_model, criterion)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "4b2f8bb6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.786"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate(embedding_bag_model, validation_dataloader, criterion)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df7ea118",
   "metadata": {},
   "source": [
    "**What was the validation accuracy for our Average Embedding model?**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5ce1043",
   "metadata": {},
   "source": [
    "- The validation accuracy for our Average Embedding model is 0.786 (78.6%)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6eb8aa57",
   "metadata": {},
   "source": [
    "# BiLSTM Model\n",
    "\n",
    "A **Bidirectional Long Short-Term Memory Network (BiLSTM)**, a type of **recurrent neural network (RNN)**.  When an LSTM is bidirectional, that means that we are using two LSTMs, one that process the sequence in the forwards direction (from the first word to last), and another to process backwords (from the last word to the first). Using a bidirectional approach allows our model to learn more complex relationships in our input sequences.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "5ac2ede2",
   "metadata": {},
   "outputs": [],
   "source": [
    "class BiLSTMModel(torch.nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "        super(BiLSTMModel, self).__init__()\n",
    "        self.emb_layer, self.num_embeddings, self.embedding_dim = glove_emb_layer(weights_matrix, trainable=True)\n",
    "        self.embedding = self.emb_layer\n",
    "        self.lstm = torch.nn.LSTM(glove_dim,glove_dim,num_layers=1,batch_first=True,bidirectional=True)\n",
    "        self.output = torch.nn.Linear(self.embedding_dim*2,1)\n",
    "\n",
    "    def forward(self, text):\n",
    "        embedded = self.embedding(text)\n",
    "        output, (h_n, c_n) = self.lstm(embedded)\n",
    "        average_vector = torch.mean(output,1)\n",
    "        output = self.output(average_vector)\n",
    "        final_output = output.squeeze()\n",
    "        return final_output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "9ca8c8a0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "379623fb28fb414fa8b47a54184c4342",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch   1 |   100/  500 batches | train accuracy    0.545\n",
      "| epoch   1 |   200/  500 batches | train accuracy    0.586\n",
      "| epoch   1 |   300/  500 batches | train accuracy    0.601\n",
      "| epoch   1 |   400/  500 batches | train accuracy    0.596\n",
      "-----------------------------------------------------------\n",
      "| end of epoch   1 | time: 547.56s | valid accuracy    0.664 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8f331a74d56840a5b996b161f98b9960",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch   2 |   100/  500 batches | train accuracy    0.702\n",
      "| epoch   2 |   200/  500 batches | train accuracy    0.662\n",
      "| epoch   2 |   300/  500 batches | train accuracy    0.710\n",
      "| epoch   2 |   400/  500 batches | train accuracy    0.734\n",
      "-----------------------------------------------------------\n",
      "| end of epoch   2 | time: 547.65s | valid accuracy    0.760 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d0cfbaf0060b476a985bf2cab88f1a04",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch   3 |   100/  500 batches | train accuracy    0.748\n",
      "| epoch   3 |   200/  500 batches | train accuracy    0.792\n",
      "| epoch   3 |   300/  500 batches | train accuracy    0.760\n",
      "| epoch   3 |   400/  500 batches | train accuracy    0.824\n",
      "-----------------------------------------------------------\n",
      "| end of epoch   3 | time: 546.26s | valid accuracy    0.834 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7517228d75ea46d190ff31322256bcab",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch   4 |   100/  500 batches | train accuracy    0.824\n",
      "| epoch   4 |   200/  500 batches | train accuracy    0.835\n",
      "| epoch   4 |   300/  500 batches | train accuracy    0.814\n",
      "| epoch   4 |   400/  500 batches | train accuracy    0.834\n",
      "-----------------------------------------------------------\n",
      "| end of epoch   4 | time: 554.36s | valid accuracy    0.830 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "598df7486ea84af7b8d429b42f2605d0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch   5 |   100/  500 batches | train accuracy    0.859\n",
      "| epoch   5 |   200/  500 batches | train accuracy    0.866\n",
      "| epoch   5 |   300/  500 batches | train accuracy    0.873\n",
      "| epoch   5 |   400/  500 batches | train accuracy    0.870\n",
      "-----------------------------------------------------------\n",
      "| end of epoch   5 | time: 573.50s | valid accuracy    0.842 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1c7c9577780d4940ab2263ca68b08073",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch   6 |   100/  500 batches | train accuracy    0.877\n",
      "| epoch   6 |   200/  500 batches | train accuracy    0.863\n",
      "| epoch   6 |   300/  500 batches | train accuracy    0.875\n",
      "| epoch   6 |   400/  500 batches | train accuracy    0.865\n",
      "-----------------------------------------------------------\n",
      "| end of epoch   6 | time: 545.58s | valid accuracy    0.836 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fb9a78f3f35b4c4580c8bb3d001a8990",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch   7 |   100/  500 batches | train accuracy    0.869\n",
      "| epoch   7 |   200/  500 batches | train accuracy    0.874\n",
      "| epoch   7 |   300/  500 batches | train accuracy    0.894\n",
      "| epoch   7 |   400/  500 batches | train accuracy    0.887\n",
      "-----------------------------------------------------------\n",
      "| end of epoch   7 | time: 552.23s | valid accuracy    0.846 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "32f6978cec93457d891be84cb6c45a45",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch   8 |   100/  500 batches | train accuracy    0.907\n",
      "| epoch   8 |   200/  500 batches | train accuracy    0.869\n",
      "| epoch   8 |   300/  500 batches | train accuracy    0.875\n",
      "| epoch   8 |   400/  500 batches | train accuracy    0.881\n",
      "-----------------------------------------------------------\n",
      "| end of epoch   8 | time: 566.40s | valid accuracy    0.846 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ef692ee88e95494bb90c34cd6b04dee3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch   9 |   100/  500 batches | train accuracy    0.886\n",
      "| epoch   9 |   200/  500 batches | train accuracy    0.879\n",
      "| epoch   9 |   300/  500 batches | train accuracy    0.859\n",
      "| epoch   9 |   400/  500 batches | train accuracy    0.887\n",
      "-----------------------------------------------------------\n",
      "| end of epoch   9 | time: 541.85s | valid accuracy    0.846 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bca8dd21bacd40efb1c6aff534240889",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch  10 |   100/  500 batches | train accuracy    0.895\n",
      "| epoch  10 |   200/  500 batches | train accuracy    0.884\n",
      "| epoch  10 |   300/  500 batches | train accuracy    0.882\n",
      "| epoch  10 |   400/  500 batches | train accuracy    0.876\n",
      "-----------------------------------------------------------\n",
      "| end of epoch  10 | time: 564.12s | valid accuracy    0.846 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "799a3a1940f74563bcfc8bf64b56fa36",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch  11 |   100/  500 batches | train accuracy    0.884\n",
      "| epoch  11 |   200/  500 batches | train accuracy    0.886\n",
      "| epoch  11 |   300/  500 batches | train accuracy    0.885\n",
      "| epoch  11 |   400/  500 batches | train accuracy    0.901\n",
      "-----------------------------------------------------------\n",
      "| end of epoch  11 | time: 550.45s | valid accuracy    0.844 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8a8baa8391e243928772f1c317fa9522",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch  12 |   100/  500 batches | train accuracy    0.900\n",
      "| epoch  12 |   200/  500 batches | train accuracy    0.870\n",
      "| epoch  12 |   300/  500 batches | train accuracy    0.897\n",
      "| epoch  12 |   400/  500 batches | train accuracy    0.892\n",
      "-----------------------------------------------------------\n",
      "| end of epoch  12 | time: 546.59s | valid accuracy    0.846 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3fe5015a41404e49b8fe4b3da575f027",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch  13 |   100/  500 batches | train accuracy    0.912\n",
      "| epoch  13 |   200/  500 batches | train accuracy    0.880\n",
      "| epoch  13 |   300/  500 batches | train accuracy    0.886\n",
      "| epoch  13 |   400/  500 batches | train accuracy    0.884\n",
      "-----------------------------------------------------------\n",
      "| end of epoch  13 | time: 546.90s | valid accuracy    0.846 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "798653f94077442696ea2414f120a378",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch  14 |   100/  500 batches | train accuracy    0.894\n",
      "| epoch  14 |   200/  500 batches | train accuracy    0.902\n",
      "| epoch  14 |   300/  500 batches | train accuracy    0.891\n",
      "| epoch  14 |   400/  500 batches | train accuracy    0.876\n",
      "-----------------------------------------------------------\n",
      "| end of epoch  14 | time: 545.44s | valid accuracy    0.846 \n",
      "-----------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6c5677dd50a7481dbeb00c80a4849a93",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| epoch  15 |   100/  500 batches | train accuracy    0.895\n",
      "| epoch  15 |   200/  500 batches | train accuracy    0.887\n",
      "| epoch  15 |   300/  500 batches | train accuracy    0.876\n",
      "| epoch  15 |   400/  500 batches | train accuracy    0.894\n",
      "-----------------------------------------------------------\n",
      "| end of epoch  15 | time: 550.11s | valid accuracy    0.848 \n",
      "-----------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "criterion = torch.nn.MSELoss()\n",
    "vocab_size = len(vocab)\n",
    "emsize = 128\n",
    "bilstm_model = BiLSTMModel().to(device)\n",
    "\n",
    "train_loop(bilstm_model, criterion)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "08c3fc49",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.848"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate(bilstm_model, validation_dataloader, criterion)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fac3dbe6",
   "metadata": {},
   "source": [
    "- The validation accuracy for the BiLSTM model is 0.848 (84.8%)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "305fbce7",
   "metadata": {},
   "source": [
    "# Model Selection"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd49dabf",
   "metadata": {},
   "source": [
    "- The best performing model is BiLSTM Model with validation accuracy of 0.848.\n",
    "- The least performing model is the Embedding Bag Model with accuracy of 0.766."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3636cde7",
   "metadata": {},
   "source": [
    "- The best performing models were the most complex. The model ranked the way they did due to **increasing Complexity.**\n",
    "     - As model complexity increases, performance on the data used to build the model (training data) improves.\n",
    "     - However, performance on the validation set improves up to a point,then start getting worse (overfitting)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "85af7575",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.858"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate(bilstm_model, test_dataloader, criterion) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c528c3c",
   "metadata": {},
   "source": [
    "- the test accuracy for the best model(BiLSTM Model) is 0.858. \n",
    "    - It's a good performance."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conv-ai",
   "language": "python",
   "name": "conv-ai"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}