6. Transformers for Sentiment Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6 - Transformers for Sentiment Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Trong notebook này, ta sẽ sử dụng transformer model, được giới thiệu lần đầu tiên trong paper [này](https://arxiv.org/pdf/1706.03762.pdf). Đặc biệt, ta sẽ sử dụng mô hình BERT (Bidirectional Encoder Representations from Transformers) từ paper [này](https://arxiv.org/pdf/1810.04805.pdf)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Transformer models có kích thước lớn hơn rất nhiều các models ở notebooks trước. Ta sẽ sử dụng [thư viện transformer](https://github.com/huggingface/transformers) để lấy pre-trained transformers models và sử dụng chúng như embedding layers của mình. Ta sẽ freeze (không train) transformer và chỉ train phần còn lại của model, no sẽ học từ các biểu diễn sinh ra bởi transformer. Trong trường hợp này, ta sẽ sử dụng *multi-layer bi-directional GRU*, tuy nhiên bất cứ mô hình nào cũng có thể học được từ biểu diễn của transformers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preparing Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "import random\n",
    "import numpy as np\n",
    "\n",
    "SEED = 1234 \n",
    "\n",
    "random.seed(SEED)\n",
    "np.random.seed(SEED)    \n",
    "torch.manual_seed(SEED)\n",
    "torch.backends.cudnn.deterministic = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Transformer được train với một tập vocab cụ thể, điều đó có nghĩa là chúng ta cần train chính xác với tập vocab và cần tokenize data giống với cách mà transformer được tokenize."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> May mắn thay, thư viện transformer có sẵn **tokenizer** cho mỗi transformer model mà nó cung cấp. Trong trường hợp này, chúng ta sử dụng model BERT lower tất cả các từ. Ta load tokenizer bằng cách sử dụng: `bert-base-uncased`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import BertTokenizer, BertModel\n",
    "\n",
    "tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> `tokenizer` có attribute `vocab` chứa vobulary thực mà ta sẽ sử dụng. Ta cùng tìm hiểu về vocab này."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "30522"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.vocab_size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Ta sử dụng `tokenizer` như sau:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['this', 'is', 'a', 'test', 'sentence']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_sentence = \"This is a test sentence\"\n",
    "\n",
    "tokenized = tokenizer.tokenize(test_sentence)\n",
    "tokenized"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Ta còn có thể số hóa các tokens trong vocab bằng cách sử dụng `tokenizer.convert_tokens_to_ids`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[2023, 2003, 1037, 3231, 6251]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.convert_tokens_to_ids(tokenized)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Transformer còn được train với các tokens đặc biệt dùng để đánh dấu bắt đầu và kết thúc câu, chi tiết tại [đây](https://huggingface.co/docs/transformers/model_doc/bert#transformers.BertModel). Bên cạnh đó, ta còn có thể lấy được các padding và unknow token."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **NOTE**: tokenizer có thuộc tính bắt đầu sequence và kết thúc sequence (`bos_token` và `eos_token`) tuy nhiên không nên sử dụng chúng trong transformer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CLS] [SEP] [PAD] [UNK]\n"
     ]
    }
   ],
   "source": [
    "init_token = tokenizer.cls_token\n",
    "eos_token = tokenizer.sep_token\n",
    "pad_token = tokenizer.pad_token\n",
    "unk_token = tokenizer.unk_token\n",
    "\n",
    "print(init_token, eos_token, pad_token, unk_token)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Ta có thể lấy chỉ số của các tokens đặc biệt này bằng cách sử dụng vocab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "101 102 0 100\n"
     ]
    }
   ],
   "source": [
    "init_token_idx = tokenizer.convert_tokens_to_ids(init_token)\n",
    "eos_token_idx = tokenizer.convert_tokens_to_ids(eos_token)\n",
    "pad_token_idx = tokenizer.convert_tokens_to_ids(pad_token)\n",
    "unk_token_idx = tokenizer.convert_tokens_to_ids(unk_token)\n",
    "\n",
    "print(init_token_idx, eos_token_idx, pad_token_idx, unk_token_idx)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> chúng ta cần phải xử lý thêm 1 vấn đề nữa là model được train trên các sequences với chiều dài tối đa đã được định sẵn nên model sẽ không biết cách xử lý những sequence có chiều dài lớn hơn maximum length đã được chỉ định ở trên. Để kiểm tra maximum length, ta sử dụng `max_model_input_sizes` cho phiên bản transformer mà ta sử dụng:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "512\n"
     ]
    }
   ],
   "source": [
    "max_input_length = tokenizer.max_model_input_sizes['bert-base-uncased']\n",
    "print(max_input_length)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Trước đây, ta sử dụng `SpaCy` tokenizer để tokenize các sentences. Tuy nhiên giờ đây ta cần một hàm được truyền vào trường `TEXT` để thực hiện tokenize. hàm đó cần thực hiện thêm cả việc cắt ngắn câu nếu nó vượt quá maximum length cho trước. Lưu ý là maximum length phải trừ đi 2 vì ta cần thêm 2 tokens vào đầu và cuối mỗi sentences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokenize_and_cut(sentence):\n",
    "    tokens = tokenizer.tokenize(sentence)\n",
    "    tokens = tokens[:max_input_length-2]\n",
    "    return tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Tiếp theo, ta thực hiện xây dựng các trường. Transformer yêu cầu batch dimension trước, nên ta set `batch_first = True`. Do ta đã có sẵn vocabulary cho các text, được cung cấp bởi transformer nên ta set `use_vocab = False` để thông báo với torchtext rằng ta tự xử lý vấn đề vocab. Tiếp theo, ta cần truyền hàm `tokenize_and_cut` để làm tokenizer. Đối số `preprocessing` là một hàm nhận đầu vào là các sentences đã được tokenized, đó sẽ là nơi chúng ta chuyển tokens thành các chỉ số tương ứng của chúng. Cuối cùng là các tokens đặc biệt, lưu ý là ta truyền vào chỉ số của chúng chứ không phải bản thân tokens (dạng string)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchtext.legacy import data\n",
    "\n",
    "TEXT = data.Field(batch_first = True,\n",
    "                    use_vocab = False,\n",
    "                    tokenize = tokenize_and_cut,\n",
    "                    preprocessing = tokenizer.convert_tokens_to_ids,\n",
    "                    init_token = init_token_idx,\n",
    "                    eos_token = eos_token_idx,\n",
    "                    pad_token = pad_token_idx,\n",
    "                    unk_token = unk_token_idx)\n",
    "\n",
    "LABEL = data.LabelField(dtype = torch.float)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Load data và tạo train, test, valid set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchtext.legacy import datasets\n",
    "\n",
    "train_data, test_data = datasets.IMDB.splits(TEXT, LABEL)\n",
    "\n",
    "train_data, valid_data = train_data.split(random_state = random.seed(SEED))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of training examples: 17500\n",
      "Number of validation examples: 7500\n",
      "Number of testing examples: 25000\n"
     ]
    }
   ],
   "source": [
    "print(f'Number of training examples: {len(train_data)}')\n",
    "print(f'Number of validation examples: {len(valid_data)}')\n",
    "print(f'Number of testing examples: {len(test_data)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Kiểm tra 1 instance để đảm bảo nó đã được số hóa."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'text': [1996,\n",
       "  18458,\n",
       "  1997,\n",
       "  6644,\n",
       "  9016,\n",
       "  4627,\n",
       "  2066,\n",
       "  2009,\n",
       "  2453,\n",
       "  2031,\n",
       "  2242,\n",
       "  2000,\n",
       "  3749,\n",
       "  1012,\n",
       "  1037,\n",
       "  2177,\n",
       "  1997,\n",
       "  2267,\n",
       "  13496,\n",
       "  2044,\n",
       "  4399,\n",
       "  1006,\n",
       "  1999,\n",
       "  1996,\n",
       "  2991,\n",
       "  1029,\n",
       "  1007,\n",
       "  3632,\n",
       "  2000,\n",
       "  1037,\n",
       "  7001,\n",
       "  6644,\n",
       "  1999,\n",
       "  1996,\n",
       "  5249,\n",
       "  2073,\n",
       "  2028,\n",
       "  2011,\n",
       "  2028,\n",
       "  2027,\n",
       "  2024,\n",
       "  4457,\n",
       "  2011,\n",
       "  2019,\n",
       "  16100,\n",
       "  5771,\n",
       "  5983,\n",
       "  7865,\n",
       "  1012,\n",
       "  1026,\n",
       "  7987,\n",
       "  1013,\n",
       "  1028,\n",
       "  1026,\n",
       "  7987,\n",
       "  1013,\n",
       "  1028,\n",
       "  6854,\n",
       "  1010,\n",
       "  1996,\n",
       "  2034,\n",
       "  20423,\n",
       "  2003,\n",
       "  2073,\n",
       "  2151,\n",
       "  6556,\n",
       "  3787,\n",
       "  1997,\n",
       "  2143,\n",
       "  3737,\n",
       "  2644,\n",
       "  1012,\n",
       "  6644,\n",
       "  9016,\n",
       "  2003,\n",
       "  2210,\n",
       "  2062,\n",
       "  2084,\n",
       "  2267,\n",
       "  4268,\n",
       "  2559,\n",
       "  2005,\n",
       "  3348,\n",
       "  1010,\n",
       "  22017,\n",
       "  4371,\n",
       "  1010,\n",
       "  3331,\n",
       "  2512,\n",
       "  1011,\n",
       "  2644,\n",
       "  2055,\n",
       "  2498,\n",
       "  1010,\n",
       "  1998,\n",
       "  3773,\n",
       "  2129,\n",
       "  2116,\n",
       "  1042,\n",
       "  1011,\n",
       "  9767,\n",
       "  2027,\n",
       "  2064,\n",
       "  2131,\n",
       "  2046,\n",
       "  1015,\n",
       "  1024,\n",
       "  2871,\n",
       "  2781,\n",
       "  2030,\n",
       "  2174,\n",
       "  2146,\n",
       "  2023,\n",
       "  6752,\n",
       "  2003,\n",
       "  1012,\n",
       "  1026,\n",
       "  7987,\n",
       "  1013,\n",
       "  1028,\n",
       "  1026,\n",
       "  7987,\n",
       "  1013,\n",
       "  1028,\n",
       "  1996,\n",
       "  4268,\n",
       "  2552,\n",
       "  1998,\n",
       "  10509,\n",
       "  5236,\n",
       "  2135,\n",
       "  2000,\n",
       "  2673,\n",
       "  2105,\n",
       "  2068,\n",
       "  1012,\n",
       "  2028,\n",
       "  1997,\n",
       "  2068,\n",
       "  2005,\n",
       "  6013,\n",
       "  9418,\n",
       "  2008,\n",
       "  1996,\n",
       "  3096,\n",
       "  7865,\n",
       "  2038,\n",
       "  10372,\n",
       "  2014,\n",
       "  3456,\n",
       "  1010,\n",
       "  2061,\n",
       "  2054,\n",
       "  2515,\n",
       "  2016,\n",
       "  2079,\n",
       "  1029,\n",
       "  2016,\n",
       "  7906,\n",
       "  21146,\n",
       "  6455,\n",
       "  2014,\n",
       "  3456,\n",
       "  7989,\n",
       "  2000,\n",
       "  2202,\n",
       "  5372,\n",
       "  2966,\n",
       "  3086,\n",
       "  2005,\n",
       "  2014,\n",
       "  8710,\n",
       "  1012,\n",
       "  1996,\n",
       "  3496,\n",
       "  2003,\n",
       "  2210,\n",
       "  2062,\n",
       "  2084,\n",
       "  1037,\n",
       "  7977,\n",
       "  2041,\n",
       "  1012,\n",
       "  1999,\n",
       "  2178,\n",
       "  3496,\n",
       "  1010,\n",
       "  7945,\n",
       "  2844,\n",
       "  2013,\n",
       "  1000,\n",
       "  2879,\n",
       "  6010,\n",
       "  2088,\n",
       "  1000,\n",
       "  4152,\n",
       "  19026,\n",
       "  2006,\n",
       "  1996,\n",
       "  2192,\n",
       "  2011,\n",
       "  2070,\n",
       "  4845,\n",
       "  2040,\n",
       "  2069,\n",
       "  2758,\n",
       "  1000,\n",
       "  28470,\n",
       "  1000,\n",
       "  1998,\n",
       "  7777,\n",
       "  2000,\n",
       "  2079,\n",
       "  16894,\n",
       "  14590,\n",
       "  2006,\n",
       "  2216,\n",
       "  2040,\n",
       "  4133,\n",
       "  2279,\n",
       "  2000,\n",
       "  2032,\n",
       "  1012,\n",
       "  2065,\n",
       "  2017,\n",
       "  2064,\n",
       "  3275,\n",
       "  2041,\n",
       "  1996,\n",
       "  3114,\n",
       "  2005,\n",
       "  2339,\n",
       "  1996,\n",
       "  1000,\n",
       "  28470,\n",
       "  1000,\n",
       "  4845,\n",
       "  2001,\n",
       "  2443,\n",
       "  1010,\n",
       "  1045,\n",
       "  1005,\n",
       "  1040,\n",
       "  2293,\n",
       "  2000,\n",
       "  2113,\n",
       "  1012,\n",
       "  4312,\n",
       "  1010,\n",
       "  7945,\n",
       "  18551,\n",
       "  1037,\n",
       "  3748,\n",
       "  3899,\n",
       "  1998,\n",
       "  3632,\n",
       "  2125,\n",
       "  2000,\n",
       "  9378,\n",
       "  2010,\n",
       "  19026,\n",
       "  2192,\n",
       "  1999,\n",
       "  1037,\n",
       "  2087,\n",
       "  3497,\n",
       "  19450,\n",
       "  3636,\n",
       "  1012,\n",
       "  2178,\n",
       "  4845,\n",
       "  7777,\n",
       "  2000,\n",
       "  4530,\n",
       "  1042,\n",
       "  1011,\n",
       "  9767,\n",
       "  1999,\n",
       "  24868,\n",
       "  2000,\n",
       "  2673,\n",
       "  2105,\n",
       "  2032,\n",
       "  1998,\n",
       "  5607,\n",
       "  29384,\n",
       "  1012,\n",
       "  2339,\n",
       "  1029,\n",
       "  2115,\n",
       "  3984,\n",
       "  2003,\n",
       "  2004,\n",
       "  2204,\n",
       "  2004,\n",
       "  3067,\n",
       "  999,\n",
       "  1026,\n",
       "  7987,\n",
       "  1013,\n",
       "  1028,\n",
       "  1026,\n",
       "  7987,\n",
       "  1013,\n",
       "  1028,\n",
       "  7945,\n",
       "  2844,\n",
       "  2003,\n",
       "  1996,\n",
       "  2069,\n",
       "  4845,\n",
       "  2007,\n",
       "  2151,\n",
       "  5038,\n",
       "  1999,\n",
       "  2023,\n",
       "  3185,\n",
       "  1012,\n",
       "  2002,\n",
       "  5363,\n",
       "  2000,\n",
       "  5475,\n",
       "  2111,\n",
       "  2091,\n",
       "  1999,\n",
       "  1011,\n",
       "  2090,\n",
       "  1996,\n",
       "  13175,\n",
       "  1998,\n",
       "  7491,\n",
       "  1998,\n",
       "  1042,\n",
       "  1008,\n",
       "  1008,\n",
       "  1008,\n",
       "  1061,\n",
       "  1008,\n",
       "  1008,\n",
       "  999,\n",
       "  9767,\n",
       "  2008,\n",
       "  2111,\n",
       "  2024,\n",
       "  6886,\n",
       "  2105,\n",
       "  1012,\n",
       "  2043,\n",
       "  1996,\n",
       "  4268,\n",
       "  4995,\n",
       "  1005,\n",
       "  1056,\n",
       "  13175,\n",
       "  1010,\n",
       "  2027,\n",
       "  2024,\n",
       "  2383,\n",
       "  2030,\n",
       "  3331,\n",
       "  2055,\n",
       "  3348,\n",
       "  2030,\n",
       "  3331,\n",
       "  14652,\n",
       "  2000,\n",
       "  1996,\n",
       "  2060,\n",
       "  4639,\n",
       "  3494,\n",
       "  2040,\n",
       "  2024,\n",
       "  2130,\n",
       "  2062,\n",
       "  1006,\n",
       "  2065,\n",
       "  2008,\n",
       "  2003,\n",
       "  2825,\n",
       "  1007,\n",
       "  10041,\n",
       "  2594,\n",
       "  2084,\n",
       "  1996,\n",
       "  4268,\n",
       "  999,\n",
       "  1996,\n",
       "  10041,\n",
       "  8872,\n",
       "  2007,\n",
       "  2019,\n",
       "  26264,\n",
       "  1997,\n",
       "  3438,\n",
       "  2012,\n",
       "  2190,\n",
       "  2089,\n",
       "  2022,\n",
       "  2028,\n",
       "  1997,\n",
       "  1996,\n",
       "  5409,\n",
       "  3772,\n",
       "  5841,\n",
       "  1045,\n",
       "  2031,\n",
       "  2412,\n",
       "  2464,\n",
       "  1999,\n",
       "  1037,\n",
       "  3185,\n",
       "  1012,\n",
       "  2017,\n",
       "  2831,\n",
       "  2055,\n",
       "  2111,\n",
       "  2025,\n",
       "  2652,\n",
       "  2007,\n",
       "  1037,\n",
       "  2440,\n",
       "  5877,\n",
       "  1010,\n",
       "  2023,\n",
       "  2079,\n",
       "  8024,\n",
       "  2987,\n",
       "  1005,\n",
       "  1056,\n",
       "  2130,\n",
       "  2113,\n",
       "  2129,\n",
       "  2000,\n",
       "  2424,\n",
       "  1996,\n",
       "  5329,\n",
       "  999,\n",
       "  8840,\n",
       "  2140,\n",
       "  999,\n",
       "  1045,\n",
       "  2001,\n",
       "  2066,\n",
       "  1010,\n",
       "  1000,\n",
       "  2097,\n",
       "  2017,\n",
       "  3531,\n",
       "  3844,\n",
       "  2039,\n",
       "  2525,\n",
       "  1029,\n",
       "  999,\n",
       "  1000,\n",
       "  2002,\n",
       "  3084,\n",
       "  1996,\n",
       "  4845,\n",
       "  5889,\n",
       "  2298,\n",
       "  2066,\n",
       "  11067,\n",
       "  2229,\n",
       "  999,\n",
       "  1996,\n",
       "  2069,\n",
       "  2112,\n",
       "  2008,\n",
       "  1045,\n",
       "  4066,\n",
       "  1997,\n",
       "  4669,\n",
       "  2001,\n",
       "  7945,\n",
       "  1005,\n",
       "  1055,\n",
       "  12459,\n",
       "  2466,\n",
       "  1006,\n",
       "  2348,\n",
       "  2175,\n",
       "  2854,\n",
       "  1007,\n",
       "  2055,\n",
       "  1996,\n",
       "  4315,\n",
       "  22043,\n",
       "  2094,\n",
       "  9116,\n",
       "  8975,\n",
       "  3124,\n",
       "  1012,\n",
       "  1999,\n",
       "  7636,\n",
       "  1010,\n",
       "  7945,\n",
       "  2056,\n",
       "  2008,\n",
       "  2002,\n",
       "  2018,\n",
       "  1037,\n",
       "  2307,\n",
       "  3066,\n",
       "  1997,\n",
       "  4847,\n",
       "  2005,\n",
       "  2472,\n",
       "  12005,\n",
       "  12211,\n",
       "  1012],\n",
       " 'label': 'neg'}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vars(train_data.examples[6])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Ta còn có thể sử dụng `convert_ids_to_tokens` để chuyển từ chỉ số về dạng string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['the',\n",
       " 'premise',\n",
       " 'of',\n",
       " 'cabin',\n",
       " 'fever',\n",
       " 'starts',\n",
       " 'like',\n",
       " 'it',\n",
       " 'might',\n",
       " 'have',\n",
       " 'something',\n",
       " 'to',\n",
       " 'offer',\n",
       " '.',\n",
       " 'a',\n",
       " 'group',\n",
       " 'of',\n",
       " 'college',\n",
       " 'teens',\n",
       " 'after',\n",
       " 'finals',\n",
       " '(',\n",
       " 'in',\n",
       " 'the',\n",
       " 'fall',\n",
       " '?',\n",
       " ')',\n",
       " 'goes',\n",
       " 'to',\n",
       " 'a',\n",
       " 'resort',\n",
       " 'cabin',\n",
       " 'in',\n",
       " 'the',\n",
       " 'woods',\n",
       " 'where',\n",
       " 'one',\n",
       " 'by',\n",
       " 'one',\n",
       " 'they',\n",
       " 'are',\n",
       " 'attacked',\n",
       " 'by',\n",
       " 'an',\n",
       " 'unseen',\n",
       " 'flesh',\n",
       " 'eating',\n",
       " 'virus',\n",
       " '.',\n",
       " '<',\n",
       " 'br',\n",
       " '/',\n",
       " '>',\n",
       " '<',\n",
       " 'br',\n",
       " '/',\n",
       " '>',\n",
       " 'unfortunately',\n",
       " ',',\n",
       " 'the',\n",
       " 'first',\n",
       " 'paragraph',\n",
       " 'is',\n",
       " 'where',\n",
       " 'any',\n",
       " 'remote',\n",
       " 'elements',\n",
       " 'of',\n",
       " 'film',\n",
       " 'quality',\n",
       " 'stop',\n",
       " '.',\n",
       " 'cabin',\n",
       " 'fever',\n",
       " 'is',\n",
       " 'little',\n",
       " 'more',\n",
       " 'than',\n",
       " 'college',\n",
       " 'kids',\n",
       " 'looking',\n",
       " 'for',\n",
       " 'sex',\n",
       " ',',\n",
       " 'boo',\n",
       " '##ze',\n",
       " ',',\n",
       " 'talking',\n",
       " 'non',\n",
       " '-',\n",
       " 'stop',\n",
       " 'about',\n",
       " 'nothing',\n",
       " ',',\n",
       " 'and',\n",
       " 'seeing',\n",
       " 'how',\n",
       " 'many',\n",
       " 'f',\n",
       " '-',\n",
       " 'bombs',\n",
       " 'they',\n",
       " 'can',\n",
       " 'get',\n",
       " 'into',\n",
       " '1',\n",
       " ':',\n",
       " '40',\n",
       " 'minutes',\n",
       " 'or',\n",
       " 'however',\n",
       " 'long',\n",
       " 'this',\n",
       " 'mess',\n",
       " 'is',\n",
       " '.',\n",
       " '<',\n",
       " 'br',\n",
       " '/',\n",
       " '>',\n",
       " '<',\n",
       " 'br',\n",
       " '/',\n",
       " '>',\n",
       " 'the',\n",
       " 'kids',\n",
       " 'act',\n",
       " 'and',\n",
       " 'react',\n",
       " 'stupid',\n",
       " '##ly',\n",
       " 'to',\n",
       " 'everything',\n",
       " 'around',\n",
       " 'them',\n",
       " '.',\n",
       " 'one',\n",
       " 'of',\n",
       " 'them',\n",
       " 'for',\n",
       " 'instance',\n",
       " 'discovers',\n",
       " 'that',\n",
       " 'the',\n",
       " 'skin',\n",
       " 'virus',\n",
       " 'has',\n",
       " 'infected',\n",
       " 'her',\n",
       " 'legs',\n",
       " ',',\n",
       " 'so',\n",
       " 'what',\n",
       " 'does',\n",
       " 'she',\n",
       " 'do',\n",
       " '?',\n",
       " 'she',\n",
       " 'keeps',\n",
       " 'sha',\n",
       " '##ving',\n",
       " 'her',\n",
       " 'legs',\n",
       " 'failing',\n",
       " 'to',\n",
       " 'take',\n",
       " 'proper',\n",
       " 'medical',\n",
       " 'attention',\n",
       " 'for',\n",
       " 'her',\n",
       " 'wounds',\n",
       " '.',\n",
       " 'the',\n",
       " 'scene',\n",
       " 'is',\n",
       " 'little',\n",
       " 'more',\n",
       " 'than',\n",
       " 'a',\n",
       " 'gross',\n",
       " 'out',\n",
       " '.',\n",
       " 'in',\n",
       " 'another',\n",
       " 'scene',\n",
       " ',',\n",
       " 'rider',\n",
       " 'strong',\n",
       " 'from',\n",
       " '\"',\n",
       " 'boy',\n",
       " 'meets',\n",
       " 'world',\n",
       " '\"',\n",
       " 'gets',\n",
       " 'bitten',\n",
       " 'on',\n",
       " 'the',\n",
       " 'hand',\n",
       " 'by',\n",
       " 'some',\n",
       " 'kid',\n",
       " 'who',\n",
       " 'only',\n",
       " 'says',\n",
       " '\"',\n",
       " 'pancakes',\n",
       " '\"',\n",
       " 'and',\n",
       " 'likes',\n",
       " 'to',\n",
       " 'do',\n",
       " 'karate',\n",
       " 'kicks',\n",
       " 'on',\n",
       " 'those',\n",
       " 'who',\n",
       " 'sit',\n",
       " 'next',\n",
       " 'to',\n",
       " 'him',\n",
       " '.',\n",
       " 'if',\n",
       " 'you',\n",
       " 'can',\n",
       " 'figure',\n",
       " 'out',\n",
       " 'the',\n",
       " 'reason',\n",
       " 'for',\n",
       " 'why',\n",
       " 'the',\n",
       " '\"',\n",
       " 'pancakes',\n",
       " '\"',\n",
       " 'kid',\n",
       " 'was',\n",
       " 'included',\n",
       " ',',\n",
       " 'i',\n",
       " \"'\",\n",
       " 'd',\n",
       " 'love',\n",
       " 'to',\n",
       " 'know',\n",
       " '.',\n",
       " 'anyway',\n",
       " ',',\n",
       " 'rider',\n",
       " 'pets',\n",
       " 'a',\n",
       " 'wild',\n",
       " 'dog',\n",
       " 'and',\n",
       " 'goes',\n",
       " 'off',\n",
       " 'to',\n",
       " 'wash',\n",
       " 'his',\n",
       " 'bitten',\n",
       " 'hand',\n",
       " 'in',\n",
       " 'a',\n",
       " 'most',\n",
       " 'likely',\n",
       " 'contaminated',\n",
       " 'creek',\n",
       " '.',\n",
       " 'another',\n",
       " 'kid',\n",
       " 'likes',\n",
       " 'to',\n",
       " 'drop',\n",
       " 'f',\n",
       " '-',\n",
       " 'bombs',\n",
       " 'in',\n",
       " 'reacting',\n",
       " 'to',\n",
       " 'everything',\n",
       " 'around',\n",
       " 'him',\n",
       " 'and',\n",
       " 'shoot',\n",
       " 'squirrels',\n",
       " '.',\n",
       " 'why',\n",
       " '?',\n",
       " 'your',\n",
       " 'guess',\n",
       " 'is',\n",
       " 'as',\n",
       " 'good',\n",
       " 'as',\n",
       " 'mine',\n",
       " '!',\n",
       " '<',\n",
       " 'br',\n",
       " '/',\n",
       " '>',\n",
       " '<',\n",
       " 'br',\n",
       " '/',\n",
       " '>',\n",
       " 'rider',\n",
       " 'strong',\n",
       " 'is',\n",
       " 'the',\n",
       " 'only',\n",
       " 'kid',\n",
       " 'with',\n",
       " 'any',\n",
       " 'recognition',\n",
       " 'in',\n",
       " 'this',\n",
       " 'movie',\n",
       " '.',\n",
       " 'he',\n",
       " 'tries',\n",
       " 'to',\n",
       " 'calm',\n",
       " 'people',\n",
       " 'down',\n",
       " 'in',\n",
       " '-',\n",
       " 'between',\n",
       " 'the',\n",
       " 'yelling',\n",
       " 'and',\n",
       " 'screaming',\n",
       " 'and',\n",
       " 'f',\n",
       " '*',\n",
       " '*',\n",
       " '*',\n",
       " 'y',\n",
       " '*',\n",
       " '*',\n",
       " '!',\n",
       " 'bombs',\n",
       " 'that',\n",
       " 'people',\n",
       " 'are',\n",
       " 'throwing',\n",
       " 'around',\n",
       " '.',\n",
       " 'when',\n",
       " 'the',\n",
       " 'kids',\n",
       " 'aren',\n",
       " \"'\",\n",
       " 't',\n",
       " 'yelling',\n",
       " ',',\n",
       " 'they',\n",
       " 'are',\n",
       " 'having',\n",
       " 'or',\n",
       " 'talking',\n",
       " 'about',\n",
       " 'sex',\n",
       " 'or',\n",
       " 'talking',\n",
       " 'nonsense',\n",
       " 'to',\n",
       " 'the',\n",
       " 'other',\n",
       " 'adult',\n",
       " 'characters',\n",
       " 'who',\n",
       " 'are',\n",
       " 'even',\n",
       " 'more',\n",
       " '(',\n",
       " 'if',\n",
       " 'that',\n",
       " 'is',\n",
       " 'possible',\n",
       " ')',\n",
       " 'idiot',\n",
       " '##ic',\n",
       " 'than',\n",
       " 'the',\n",
       " 'kids',\n",
       " '!',\n",
       " 'the',\n",
       " 'idiot',\n",
       " 'cop',\n",
       " 'with',\n",
       " 'an',\n",
       " 'iq',\n",
       " 'of',\n",
       " '60',\n",
       " 'at',\n",
       " 'best',\n",
       " 'may',\n",
       " 'be',\n",
       " 'one',\n",
       " 'of',\n",
       " 'the',\n",
       " 'worst',\n",
       " 'acting',\n",
       " 'jobs',\n",
       " 'i',\n",
       " 'have',\n",
       " 'ever',\n",
       " 'seen',\n",
       " 'in',\n",
       " 'a',\n",
       " 'movie',\n",
       " '.',\n",
       " 'you',\n",
       " 'talk',\n",
       " 'about',\n",
       " 'people',\n",
       " 'not',\n",
       " 'playing',\n",
       " 'with',\n",
       " 'a',\n",
       " 'full',\n",
       " 'deck',\n",
       " ',',\n",
       " 'this',\n",
       " 'do',\n",
       " '##rk',\n",
       " 'doesn',\n",
       " \"'\",\n",
       " 't',\n",
       " 'even',\n",
       " 'know',\n",
       " 'how',\n",
       " 'to',\n",
       " 'find',\n",
       " 'the',\n",
       " 'cards',\n",
       " '!',\n",
       " 'lo',\n",
       " '##l',\n",
       " '!',\n",
       " 'i',\n",
       " 'was',\n",
       " 'like',\n",
       " ',',\n",
       " '\"',\n",
       " 'will',\n",
       " 'you',\n",
       " 'please',\n",
       " 'shut',\n",
       " 'up',\n",
       " 'already',\n",
       " '?',\n",
       " '!',\n",
       " '\"',\n",
       " 'he',\n",
       " 'makes',\n",
       " 'the',\n",
       " 'kid',\n",
       " 'actors',\n",
       " 'look',\n",
       " 'like',\n",
       " 'genius',\n",
       " '##es',\n",
       " '!',\n",
       " 'the',\n",
       " 'only',\n",
       " 'part',\n",
       " 'that',\n",
       " 'i',\n",
       " 'sort',\n",
       " 'of',\n",
       " 'liked',\n",
       " 'was',\n",
       " 'rider',\n",
       " \"'\",\n",
       " 's',\n",
       " 'scary',\n",
       " 'story',\n",
       " '(',\n",
       " 'although',\n",
       " 'go',\n",
       " '##ry',\n",
       " ')',\n",
       " 'about',\n",
       " 'the',\n",
       " 'der',\n",
       " '##ange',\n",
       " '##d',\n",
       " 'bowling',\n",
       " 'alley',\n",
       " 'guy',\n",
       " '.',\n",
       " 'in',\n",
       " 'interviews',\n",
       " ',',\n",
       " 'rider',\n",
       " 'said',\n",
       " 'that',\n",
       " 'he',\n",
       " 'had',\n",
       " 'a',\n",
       " 'great',\n",
       " 'deal',\n",
       " 'of',\n",
       " 'respect',\n",
       " 'for',\n",
       " 'director',\n",
       " 'eli',\n",
       " 'roth',\n",
       " '.']"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokens = tokenizer.convert_ids_to_tokens(vars(train_data.examples[6])['text'])\n",
    "tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Bên trên, ta mới chỉ xử lý vocabulary cho text, còn label ta vẫn phải tự xây dựng vocabulary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "LABEL.build_vocab(train_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Tạo iterator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "BATCH_SIZE = 64\n",
    "\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "\n",
    "train_iterator, valid_iterator, test_iterator = data.BucketIterator.splits(\n",
    "    (train_data, valid_data, test_data),\n",
    "    batch_size = BATCH_SIZE,\n",
    "    device = device)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build the Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> hãy chắc chắn rằng ta load model cùng với model ta áp dụng tokenizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Downloading: 100%|██████████| 570/570 [00:00<00:00, 225kB/s]\n",
      "Downloading: 100%|██████████| 440M/440M [01:09<00:00, 6.32MB/s] \n"
     ]
    }
   ],
   "source": [
    "from transformers import BertModel\n",
    "\n",
    "bert = BertModel.from_pretrained('bert-base-uncased')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Tiếp theo, ta thực hiện xây dựng model. <br>\n",
    "> Thay vì sử dụng embedding layer để có được word embeddings, ta sẽ sử dụng pre-trained transformer model. Các embedding vectors này sẽ được truyền vào GRU để thực hiện phân tích cảm xúc của các sentences. Chúng ta sẽ lấy embedding dimension size (gọi là `hidden_size`) từ transformer thông qua các attribute của nó."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Bên trong hàm foward, ta đặt transformer bên trong `no_grad` để đảm bảo phần transformer không được tính đạo hàm. Transformer trả về embeddings cho toàn bộ sequence và một *pooled output*. Tài liệu [này](https://huggingface.co/docs/transformers/model_doc/bert#transformers.BertModel) chỉ ra rằng *pooled output* sẽ kém hơn trong việc tổng hợp ý nghĩa câu đầu vào, tốt hơn hết ta nên lấy trung bình hoặc pooling hidden-states của toàn bộ input sentence, do đó ta không sử dụng *pooled output*. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "\n",
    "class BERTGRUSentiment(nn.Module):\n",
    "    def __init__(self,\n",
    "                 bert,\n",
    "                 hidden_dim,\n",
    "                 output_dim,\n",
    "                 n_layers,\n",
    "                 bidirectional,\n",
    "                 dropout):\n",
    "        \n",
    "        super().__init__()\n",
    "        \n",
    "        self.bert = bert\n",
    "        \n",
    "        embedding_dim = bert.config.to_dict()['hidden_size']\n",
    "        \n",
    "        self.rnn = nn.GRU(embedding_dim,\n",
    "                          hidden_dim,\n",
    "                          num_layers = n_layers,\n",
    "                          bidirectional = bidirectional,\n",
    "                          batch_first = True,\n",
    "                          dropout = 0 if n_layers < 2 else dropout)\n",
    "        \n",
    "        self.out = nn.Linear(hidden_dim * 2 if bidirectional else hidden_dim, output_dim)\n",
    "        \n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "\n",
    "\n",
    "\n",
    "    def forward(self, text):\n",
    "            \n",
    "            #text = [batch size, sent len]\n",
    "                    \n",
    "            with torch.no_grad():\n",
    "                embedded = self.bert(text)[0]\n",
    "                    \n",
    "            #embedded = [batch size, sent len, emb dim]\n",
    "            \n",
    "            _, hidden = self.rnn(embedded)\n",
    "            \n",
    "            #hidden = [n layers * n directions, batch size, emb dim]\n",
    "            \n",
    "            if self.rnn.bidirectional:\n",
    "                hidden = self.dropout(torch.cat((hidden[-2,:,:], hidden[-1,:,:]), dim = 1))\n",
    "            else:\n",
    "                hidden = self.dropout(hidden[-1,:,:])\n",
    "                    \n",
    "            #hidden = [batch size, hid dim]\n",
    "            \n",
    "            output = self.out(hidden)\n",
    "            \n",
    "            #output = [batch size, out dim]\n",
    "            \n",
    "            return output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "HIDDEN_DIM = 256\n",
    "OUTPUT_DIM = 1\n",
    "N_LAYERS = 2\n",
    "BIDIRECTIONAL = True\n",
    "DROPOUT = 0.25\n",
    "\n",
    "model = BERTGRUSentiment(bert,\n",
    "                         HIDDEN_DIM,\n",
    "                         OUTPUT_DIM,\n",
    "                         N_LAYERS,\n",
    "                         BIDIRECTIONAL,\n",
    "                         DROPOUT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The model has 112,241,409 trainable parameters\n"
     ]
    }
   ],
   "source": [
    "def count_parameters(model):\n",
    "    return sum(p.numel() for p in model.parameters() if p.requires_grad)\n",
    "\n",
    "print(f'The model has {count_parameters(model):,} trainable parameters')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Tổng có 112M tham số, tuy nhiên 110M tham số thuộc transformer (và ta không cần train chúng, chỉ cần train 2M tham số còn lại)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Để đóng băng các weights (không train chúng), ta cần set `requires_grad = False`. Để thực hiện việc này, ta chỉ cần lặp xuyên suốt `named_parameters`, nếu nó thuộc `bert`, ta đóng băng."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "for name, param in model.named_parameters():\n",
    "    if name.startswith('bert'):\n",
    "        param.requires_grad = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The model has 2,759,169 trainable parameters\n"
     ]
    }
   ],
   "source": [
    "print(f'The model has {count_parameters(model):,} trainable parameters')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Oh, bây giờ model chỉ còn 2M7 tham số cần train."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Chúng ta sẽ kiểm tra tên của các parameters có thể train để chắc chắn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rnn.weight_ih_l0\n",
      "rnn.weight_hh_l0\n",
      "rnn.bias_ih_l0\n",
      "rnn.bias_hh_l0\n",
      "rnn.weight_ih_l0_reverse\n",
      "rnn.weight_hh_l0_reverse\n",
      "rnn.bias_ih_l0_reverse\n",
      "rnn.bias_hh_l0_reverse\n",
      "rnn.weight_ih_l1\n",
      "rnn.weight_hh_l1\n",
      "rnn.bias_ih_l1\n",
      "rnn.bias_hh_l1\n",
      "rnn.weight_ih_l1_reverse\n",
      "rnn.weight_hh_l1_reverse\n",
      "rnn.bias_ih_l1_reverse\n",
      "rnn.bias_hh_l1_reverse\n",
      "out.weight\n",
      "out.bias\n"
     ]
    }
   ],
   "source": [
    "for name, param in model.named_parameters():\n",
    "    if param.requires_grad:\n",
    "        print(name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train the Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.optim as optim\n",
    "\n",
    "optimizer = optim.Adam(model.parameters())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "criterion = nn.BCEWithLogitsLoss()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = model.to(device)\n",
    "criterion = criterion.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "def binary_accuracy(preds, y):\n",
    "    \"\"\"\n",
    "    Returns accuracy per batch, i.e. if you get 8/10 right, this returns 0.8, NOT 8\n",
    "    \"\"\"\n",
    "\n",
    "    #round predictions to the closest integer\n",
    "    rounded_preds = torch.round(torch.sigmoid(preds))\n",
    "    correct = (rounded_preds == y).float() #convert into float for division \n",
    "    acc = correct.sum() / len(correct)\n",
    "    return acc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(model, iterator, optimizer, criterion):\n",
    "    \n",
    "    epoch_loss = 0\n",
    "    epoch_acc = 0\n",
    "    \n",
    "    model.train()\n",
    "    \n",
    "    for batch in iterator:\n",
    "        \n",
    "        optimizer.zero_grad()\n",
    "        \n",
    "        predictions = model(batch.text).squeeze(1)\n",
    "        \n",
    "        loss = criterion(predictions, batch.label)\n",
    "        \n",
    "        acc = binary_accuracy(predictions, batch.label)\n",
    "        \n",
    "        loss.backward()\n",
    "        \n",
    "        optimizer.step()\n",
    "        \n",
    "        epoch_loss += loss.item()\n",
    "        epoch_acc += acc.item()\n",
    "        \n",
    "    return epoch_loss / len(iterator), epoch_acc / len(iterator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate(model, iterator, criterion):\n",
    "    \n",
    "    epoch_loss = 0\n",
    "    epoch_acc = 0\n",
    "    \n",
    "    model.eval()\n",
    "    \n",
    "    with torch.no_grad():\n",
    "    \n",
    "        for batch in iterator:\n",
    "\n",
    "            predictions = model(batch.text).squeeze(1)\n",
    "            \n",
    "            loss = criterion(predictions, batch.label)\n",
    "            \n",
    "            acc = binary_accuracy(predictions, batch.label)\n",
    "\n",
    "            epoch_loss += loss.item()\n",
    "            epoch_acc += acc.item()\n",
    "        \n",
    "    return epoch_loss / len(iterator), epoch_acc / len(iterator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "def epoch_time(start_time, end_time):\n",
    "    elapsed_time = end_time - start_time\n",
    "    elapsed_mins = int(elapsed_time / 60)\n",
    "    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n",
    "    return elapsed_mins, elapsed_secs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Do model cần truyền data vào transformer và xử lý nên mô hình train khá lâu."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[1;32m~\\AppData\\Local\\Temp\\ipykernel_6624\\2881294158.py\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      7\u001b[0m     \u001b[0mstart_time\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mtime\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtime\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      8\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 9\u001b[1;33m     \u001b[0mtrain_loss\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mtrain_acc\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mtrain\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mtrain_iterator\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0moptimizer\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcriterion\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     10\u001b[0m     \u001b[0mvalid_loss\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mvalid_acc\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mevaluate\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mvalid_iterator\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcriterion\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     11\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Temp\\ipykernel_6624\\4261676746.py\u001b[0m in \u001b[0;36mtrain\u001b[1;34m(model, iterator, optimizer, criterion)\u001b[0m\n\u001b[0;32m     10\u001b[0m         \u001b[0moptimizer\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mzero_grad\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     11\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 12\u001b[1;33m         \u001b[0mpredictions\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mmodel\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mbatch\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtext\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msqueeze\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     13\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     14\u001b[0m         \u001b[0mloss\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mcriterion\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mpredictions\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mbatch\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mlabel\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mc:\\Users\\longln3\\Anaconda3\\envs\\python3.7.6\\lib\\site-packages\\torch\\nn\\modules\\module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[1;34m(self, *input, **kwargs)\u001b[0m\n\u001b[0;32m   1049\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[0;32m   1050\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[1;32m-> 1051\u001b[1;33m             \u001b[1;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m*\u001b[0m\u001b[0minput\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1052\u001b[0m         \u001b[1;31m# Do not call functions when jit is used\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1053\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m[\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Temp\\ipykernel_6624\\1392172365.py\u001b[0m in \u001b[0;36mforward\u001b[1;34m(self, text)\u001b[0m\n\u001b[0;32m     34\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     35\u001b[0m             \u001b[1;32mwith\u001b[0m \u001b[0mtorch\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mno_grad\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 36\u001b[1;33m                 \u001b[0membedded\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mbert\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtext\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     37\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     38\u001b[0m             \u001b[1;31m#embedded = [batch size, sent len, emb dim]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mc:\\Users\\longln3\\Anaconda3\\envs\\python3.7.6\\lib\\site-packages\\torch\\nn\\modules\\module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[1;34m(self, *input, **kwargs)\u001b[0m\n\u001b[0;32m   1049\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[0;32m   1050\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[1;32m-> 1051\u001b[1;33m             \u001b[1;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m*\u001b[0m\u001b[0minput\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1052\u001b[0m         \u001b[1;31m# Do not call functions when jit is used\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1053\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m[\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mc:\\Users\\longln3\\Anaconda3\\envs\\python3.7.6\\lib\\site-packages\\transformers\\models\\bert\\modeling_bert.py\u001b[0m in \u001b[0;36mforward\u001b[1;34m(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask, past_key_values, use_cache, output_attentions, output_hidden_states, return_dict)\u001b[0m\n\u001b[0;32m    979\u001b[0m             \u001b[0moutput_attentions\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0moutput_attentions\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    980\u001b[0m             \u001b[0moutput_hidden_states\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0moutput_hidden_states\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 981\u001b[1;33m             \u001b[0mreturn_dict\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mreturn_dict\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    982\u001b[0m         )\n\u001b[0;32m    983\u001b[0m         \u001b[0msequence_output\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mencoder_outputs\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mc:\\Users\\longln3\\Anaconda3\\envs\\python3.7.6\\lib\\site-packages\\torch\\nn\\modules\\module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[1;34m(self, *input, **kwargs)\u001b[0m\n\u001b[0;32m   1049\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[0;32m   1050\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[1;32m-> 1051\u001b[1;33m             \u001b[1;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m*\u001b[0m\u001b[0minput\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1052\u001b[0m         \u001b[1;31m# Do not call functions when jit is used\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1053\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m[\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mc:\\Users\\longln3\\Anaconda3\\envs\\python3.7.6\\lib\\site-packages\\transformers\\models\\bert\\modeling_bert.py\u001b[0m in \u001b[0;36mforward\u001b[1;34m(self, hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, past_key_values, use_cache, output_attentions, output_hidden_states, return_dict)\u001b[0m\n\u001b[0;32m    573\u001b[0m                     \u001b[0mencoder_attention_mask\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    574\u001b[0m                     \u001b[0mpast_key_value\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 575\u001b[1;33m                     \u001b[0moutput_attentions\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    576\u001b[0m                 )\n\u001b[0;32m    577\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mc:\\Users\\longln3\\Anaconda3\\envs\\python3.7.6\\lib\\site-packages\\torch\\nn\\modules\\module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[1;34m(self, *input, **kwargs)\u001b[0m\n\u001b[0;32m   1049\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[0;32m   1050\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[1;32m-> 1051\u001b[1;33m             \u001b[1;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m*\u001b[0m\u001b[0minput\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1052\u001b[0m         \u001b[1;31m# Do not call functions when jit is used\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1053\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m[\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mc:\\Users\\longln3\\Anaconda3\\envs\\python3.7.6\\lib\\site-packages\\transformers\\models\\bert\\modeling_bert.py\u001b[0m in \u001b[0;36mforward\u001b[1;34m(self, hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, past_key_value, output_attentions)\u001b[0m\n\u001b[0;32m    495\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    496\u001b[0m         layer_output = apply_chunking_to_forward(\n\u001b[1;32m--> 497\u001b[1;33m             \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfeed_forward_chunk\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mchunk_size_feed_forward\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mseq_len_dim\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mattention_output\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    498\u001b[0m         )\n\u001b[0;32m    499\u001b[0m         \u001b[0moutputs\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m(\u001b[0m\u001b[0mlayer_output\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;33m+\u001b[0m \u001b[0moutputs\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mc:\\Users\\longln3\\Anaconda3\\envs\\python3.7.6\\lib\\site-packages\\transformers\\modeling_utils.py\u001b[0m in \u001b[0;36mapply_chunking_to_forward\u001b[1;34m(forward_fn, chunk_size, chunk_dim, *input_tensors)\u001b[0m\n\u001b[0;32m   1815\u001b[0m         \u001b[1;32mreturn\u001b[0m \u001b[0mtorch\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcat\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0moutput_chunks\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdim\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mchunk_dim\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1816\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1817\u001b[1;33m     \u001b[1;32mreturn\u001b[0m \u001b[0mforward_fn\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m*\u001b[0m\u001b[0minput_tensors\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32mc:\\Users\\longln3\\Anaconda3\\envs\\python3.7.6\\lib\\site-packages\\transformers\\models\\bert\\modeling_bert.py\u001b[0m in \u001b[0;36mfeed_forward_chunk\u001b[1;34m(self, attention_output)\u001b[0m\n\u001b[0;32m    506\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    507\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0mfeed_forward_chunk\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mattention_output\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 508\u001b[1;33m         \u001b[0mintermediate_output\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mintermediate\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mattention_output\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    509\u001b[0m         \u001b[0mlayer_output\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0moutput\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mintermediate_output\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mattention_output\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    510\u001b[0m         \u001b[1;32mreturn\u001b[0m \u001b[0mlayer_output\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mc:\\Users\\longln3\\Anaconda3\\envs\\python3.7.6\\lib\\site-packages\\torch\\nn\\modules\\module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[1;34m(self, *input, **kwargs)\u001b[0m\n\u001b[0;32m   1049\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[0;32m   1050\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[1;32m-> 1051\u001b[1;33m             \u001b[1;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m*\u001b[0m\u001b[0minput\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1052\u001b[0m         \u001b[1;31m# Do not call functions when jit is used\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1053\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m[\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mc:\\Users\\longln3\\Anaconda3\\envs\\python3.7.6\\lib\\site-packages\\transformers\\models\\bert\\modeling_bert.py\u001b[0m in \u001b[0;36mforward\u001b[1;34m(self, hidden_states)\u001b[0m\n\u001b[0;32m    409\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    410\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0mforward\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mhidden_states\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 411\u001b[1;33m         \u001b[0mhidden_states\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdense\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mhidden_states\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    412\u001b[0m         \u001b[0mhidden_states\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mintermediate_act_fn\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mhidden_states\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    413\u001b[0m         \u001b[1;32mreturn\u001b[0m \u001b[0mhidden_states\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mc:\\Users\\longln3\\Anaconda3\\envs\\python3.7.6\\lib\\site-packages\\torch\\nn\\modules\\module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[1;34m(self, *input, **kwargs)\u001b[0m\n\u001b[0;32m   1049\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[0;32m   1050\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[1;32m-> 1051\u001b[1;33m             \u001b[1;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m*\u001b[0m\u001b[0minput\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1052\u001b[0m         \u001b[1;31m# Do not call functions when jit is used\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1053\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m[\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mc:\\Users\\longln3\\Anaconda3\\envs\\python3.7.6\\lib\\site-packages\\torch\\nn\\modules\\linear.py\u001b[0m in \u001b[0;36mforward\u001b[1;34m(self, input)\u001b[0m\n\u001b[0;32m     94\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     95\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0mforward\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0minput\u001b[0m\u001b[1;33m:\u001b[0m \u001b[0mTensor\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;33m->\u001b[0m \u001b[0mTensor\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 96\u001b[1;33m         \u001b[1;32mreturn\u001b[0m \u001b[0mF\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mlinear\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0minput\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mweight\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mbias\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     97\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     98\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0mextra_repr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;33m->\u001b[0m \u001b[0mstr\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mc:\\Users\\longln3\\Anaconda3\\envs\\python3.7.6\\lib\\site-packages\\torch\\nn\\functional.py\u001b[0m in \u001b[0;36mlinear\u001b[1;34m(input, weight, bias)\u001b[0m\n\u001b[0;32m   1845\u001b[0m     \u001b[1;32mif\u001b[0m \u001b[0mhas_torch_function_variadic\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0minput\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mweight\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1846\u001b[0m         \u001b[1;32mreturn\u001b[0m \u001b[0mhandle_torch_function\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mlinear\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m(\u001b[0m\u001b[0minput\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mweight\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0minput\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mweight\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mbias\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mbias\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1847\u001b[1;33m     \u001b[1;32mreturn\u001b[0m \u001b[0mtorch\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_C\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_nn\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mlinear\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0minput\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mweight\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mbias\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1848\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1849\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "N_EPOCHS = 5\n",
    "\n",
    "best_valid_loss = float('inf')\n",
    "\n",
    "for epoch in range(N_EPOCHS):\n",
    "    \n",
    "    start_time = time.time()\n",
    "    \n",
    "    train_loss, train_acc = train(model, train_iterator, optimizer, criterion)\n",
    "    valid_loss, valid_acc = evaluate(model, valid_iterator, criterion)\n",
    "        \n",
    "    end_time = time.time()\n",
    "        \n",
    "    epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
    "        \n",
    "    if valid_loss < best_valid_loss:\n",
    "        best_valid_loss = valid_loss\n",
    "        torch.save(model.state_dict(), './models/tut6-model.pt')\n",
    "    \n",
    "    print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')\n",
    "    print(f'\\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')\n",
    "    print(f'\\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.load_state_dict(torch.load('./models/tut6-model.pt'))\n",
    "\n",
    "test_loss, test_acc = evaluate(model, test_iterator, criterion)\n",
    "\n",
    "print(f'Test Loss: {test_loss:.3f} | Test Acc: {test_acc*100:.2f}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Ta thực hiện các bước sau: <br>\n",
    ">> * Tokenize input sequence\n",
    ">> * Cắt chúng sau cho có chiều dài nhỏ hơn hoặc bằng maximum length.\n",
    ">> * Thêm ký tự đặc biệt vào sequence, chuyển sang dạng tensor, tạo batch dimension giả và truyền vào model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict_sentiment(model, tokenizer, sentence):\n",
    "    model.eval()\n",
    "    tokens = tokenizer.tokenize(sentence)\n",
    "    tokens = tokens[:max_input_length-2]\n",
    "    indexed = [init_token_idx] + tokenizer.convert_tokens_to_ids(tokens) + [eos_token_idx]\n",
    "    tensor = torch.LongTensor(indexed).to(device)\n",
    "    tensor = tensor.unsqueeze(0)\n",
    "    prediction = torch.sigmoid(model(tensor))\n",
    "    return prediction.item()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predict_sentiment(model, tokenizer, \"This film is terrible\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predict_sentiment(model, tokenizer, \"This film is great\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## END"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.7.6 ('python3.7.6')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "94b83b56f38ffef65d6a4ee563210b313d606c3429660e46922f3bd794e4159a"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}