sentiment-app-deployment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating a Sentiment Analysis Web App\n",
    "### XGBoost AWS SageMaker\n",
    "_SageMaker, Lambda, API, CloudWatch_\n",
    "\n",
    "---\n",
    "Put an overview of the notebook here\n",
    "\n",
    "## Outline\n",
    "1. [Download the data](#download)\n",
    "2. [Process and prepare the data](#process)\n",
    "3. [Upload data to S3](#upload)\n",
    "4. [Build and train the Pytorch model](#train)\n",
    "5. [Test the trained model](#test)\n",
    "6. [Deploy the trained model](#deploy)\n",
    "7. [Use the deployed model for inference](#use)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='download'></a>\n",
    "## Download the Data\n",
    "\n",
    "The notebook and model use the [IMDb dataset](http://ai.stanford.edu/~amaas/data/sentiment/).\n",
    "\n",
    "> Maas, Andrew L., et al. [Learning Word Vectors for Sentiment Analysis](http://ai.stanford.edu/~amaas/data/sentiment/). In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, 2011."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mkdir: cannot create directory ‘../data’: File exists\n",
      "--2021-12-26 15:48:58--  http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz\n",
      "Resolving ai.stanford.edu (ai.stanford.edu)... 171.64.68.10\n",
      "Connecting to ai.stanford.edu (ai.stanford.edu)|171.64.68.10|:80... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 84125825 (80M) [application/x-gzip]\n",
      "Saving to: ‘../data/aclImdb_v1.tar.gz’\n",
      "\n",
      "../data/aclImdb_v1. 100%[===================>]  80.23M  60.7MB/s    in 1.3s    \n",
      "\n",
      "2021-12-26 15:48:59 (60.7 MB/s) - ‘../data/aclImdb_v1.tar.gz’ saved [84125825/84125825]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%mkdir ../data\n",
    "!wget -O ../data/aclImdb_v1.tar.gz http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz\n",
    "!tar -zxf ../data/aclImdb_v1.tar.gz -C ../data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='process'></a>\n",
    "## Process and Prepare the Data\n",
    "\n",
    "---\n",
    "### Read in Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# necessary imports \n",
    "import os\n",
    "import glob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def read_imbd_data(data_dir='../data/aclImdb'):\n",
    "    \"\"\" Read in IMDb data from aclImdb folder. Creates data and label dictionaries.\n",
    "    \n",
    "        Arguments:\n",
    "        - data_dir: (str) Directory of the data\n",
    "        \n",
    "        Returns:\n",
    "        - data: (dict) Movie review\n",
    "        - labels: (dict) Movie review labels\n",
    "    \"\"\"\n",
    "    data = {}\n",
    "    labels = {}\n",
    "    \n",
    "    # create paths to read in review data\n",
    "    for data_type in ['train', 'test']:\n",
    "        data[data_type] = {}\n",
    "        labels[data_type] = {}\n",
    "        \n",
    "        for sentiment in ['pos', 'neg']:\n",
    "            data[data_type][sentiment] = []\n",
    "            labels[data_type][sentiment] = []\n",
    "            \n",
    "            # join path names\n",
    "            path = os.path.join(data_dir, data_type, sentiment, '*.txt')\n",
    "            files = glob.glob(path)\n",
    "            \n",
    "            # open each review and label. Append to dictionaries and label with binary vars\n",
    "            for f in files:\n",
    "                with open(f) as review:\n",
    "                    data[data_type][sentiment].append(review.read())\n",
    "                    labels[data_type][sentiment].append(1 if sentiment == 'pos' else 0)\n",
    "                    \n",
    "            assert len(data[data_type][sentiment]) == len(labels[data_type][sentiment]), \\\n",
    "                       \"{}: data size does not equal {}: label size.\".format(data_type, sentiment)\n",
    "                    \n",
    "    return data, labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "IMDb Reviews: Train = 12500 pos / 12500 neg <--> Test = 12500 pos / 12500 neg\n"
     ]
    }
   ],
   "source": [
    "# read in data and display length of train and test data\n",
    "data, labels = read_imbd_data()\n",
    "print(\"IMDb Reviews: Train = {} pos / {} neg <--> Test = {} pos / {} neg\".format(len(data['train']['pos']),\n",
    "                                                                                    len(data['train']['neg']),\n",
    "                                                                                    len(labels['test']['pos']),\n",
    "                                                                                    len(labels['test']['pos'])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'My family goes back to New Orleans late 1600\\'s early 1700\\'s and in watching the movie I knew it was a history my grand-parents never talked about, but we knew it existed. I have cousins obviously black aka African Americans and others who can \"pass\" as white and chose not to. It\\'s a hard history to watch when you realize that it\\'s your family they\\'re talking about and that Cane River is all a part of that history. It makes me want to cry and it makes me want to kick the \\'arse\\' of my great grandfathers who owned those plantations and wonder in awe of how my great grandmothers of African heritage lived under that oppressive and yet aristocratic existence...And at the same time had I not come out of that history, I probably wouldn\\'t be the successful business woman I am today living successfully in a fairly integrated world. The acting was both excellent and fair depending upon the actor, but it is a movie that NEEDED to be made. Anne Rice is incredible and I ask myself, why is she \\'symbolically\\' writing about my family and I\\'m not. I recommend this movie to everyone. Leza'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['train']['pos'][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labels['train']['pos'][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### Create Feature and Target Sets\n",
    "Combine the training and test data/labels and shuffle to creat feature and target sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.utils import shuffle\n",
    "\n",
    "import nltk\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.stem.porter import *\n",
    "import re\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "import pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def combine_imdb_data(data, labels):\n",
    "    \"\"\" Combine pos and neg reviews from the training and test data.\n",
    "        \n",
    "        Arguments:\n",
    "        - data: (dict) Unprocessed reviews\n",
    "        - labels: (dict) Sentiment label, 1 pos --> 0 neg\n",
    "        \n",
    "        Returns:\n",
    "        - train_X, test_X: features\n",
    "        - train_y, test_y: targets\n",
    "    \"\"\"\n",
    "    # combine positive and negative reviews and labels\n",
    "    train_data = data['train']['pos'] + data['train']['neg']\n",
    "    test_data = data['test']['pos'] + data['test']['neg']\n",
    "    train_labels = labels['train']['pos'] + labels['train']['neg']\n",
    "    test_labels = labels['test']['pos'] + labels['test']['neg']\n",
    "    \n",
    "    # shuffle reviews and labels within training and test sets\n",
    "    train_data, train_labels = shuffle(train_data, train_labels)\n",
    "    test_data, test_labels = shuffle(test_data, test_labels)\n",
    "    \n",
    "    return train_data, test_data, train_labels, test_labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "IMDb Data Length: Train data = 25000, Test data = 25000\n"
     ]
    }
   ],
   "source": [
    "train_X, test_X, train_y, test_y = combine_imdb_data(data, labels)\n",
    "print(\"IMDb Data Length: Train data = {}, Test data = {}\".format(len(train_X), len(test_X)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This film is very interesting. I have seen it twice and it seems Glover hit the nail on the head with what he claims to he wants to accomplish. I for one can relate to the outrage that the filmmaker clearly expresses against the current thoughtless corporate drivel that is an onslaught in our every media center, and the things that we as a culture are supposed to not \"think\" about due to corporate media control. The outrage that Glover expresses through the \"outrageous\" elements in the films is both clear in its visceral aggressiveness and beautiful in its poetic potency. I am glad I saw this film and it is even clearer that Glover is up to something interesting with part two of what will be a trilogy. It is fine! EVERYTHING IS FINE. See that also. People that dismiss this film as \"thoughtless\" or \"pretentious\" are really missing the boat. This is an intelligent films. If you can see it with his live show he performs before with his books, that is also very wroth while. The way you get in to his mindset is really something. You will have an experience! \n",
      "\n",
      "1\n"
     ]
    }
   ],
   "source": [
    "# take a look at a review and it's corresponding label\n",
    "print(train_X[20], '\\n')\n",
    "print(train_y[20])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### Process the Data\n",
    "Remove the html formatting and convert the review into a list of words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def review_to_words(review):\n",
    "    \"\"\" Converts a review string to a list of words. Removes html\n",
    "        formatting, stopwords and morphological endings of common\n",
    "        words.\n",
    "        \n",
    "        Arguments:\n",
    "        - review: (str) String of words that make up review\n",
    "        \n",
    "        Returns:\n",
    "        - words: (list) List of processed words in a review\n",
    "    \n",
    "    \"\"\" \n",
    "    nltk.download('stopwords', quiet=True)\n",
    "    stemmer = PorterStemmer()\n",
    "    \n",
    "    text = BeautifulSoup(review, 'html.parser').get_text() # remove html tags\n",
    "    text = re.sub(r\"[^a-zA-z0-9]\", \" \", text.lower()) \n",
    "    words = text.split() # split the string into a list of words\n",
    "    words = [word for word in words if word not in stopwords.words('english')] # remove stopwords\n",
    "    words = [stemmer.stem(word) for word in words] # stem words\n",
    "    \n",
    "    return words\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['film', 'interest', 'seen', 'twice', 'seem', 'glover', 'hit', 'nail', 'head', 'claim', 'want', 'accomplish', 'one', 'relat', 'outrag', 'filmmak', 'clearli', 'express', 'current', 'thoughtless', 'corpor', 'drivel', 'onslaught', 'everi', 'media', 'center', 'thing', 'cultur', 'suppos', 'think', 'due', 'corpor', 'media', 'control', 'outrag', 'glover', 'express', 'outrag', 'element', 'film', 'clear', 'viscer', 'aggress', 'beauti', 'poetic', 'potenc', 'glad', 'saw', 'film', 'even', 'clearer', 'glover', 'someth', 'interest', 'part', 'two', 'trilog', 'fine', 'everyth', 'fine', 'see', 'also', 'peopl', 'dismiss', 'film', 'thoughtless', 'pretenti', 'realli', 'miss', 'boat', 'intellig', 'film', 'see', 'live', 'show', 'perform', 'book', 'also', 'wroth', 'way', 'get', 'mindset', 'realli', 'someth', 'experi']\n"
     ]
    }
   ],
   "source": [
    "words = review_to_words(train_X[20])\n",
    "print(words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "cache_dir = os.path.join(\"../cache\", \"sentiment_analysis\")\n",
    "os.makedirs(cache_dir, exist_ok=True) \n",
    "\n",
    "def preprocess_data(train_data, test_data, train_labels, test_labels,\n",
    "                    cache_dir=cache_dir, cache_file='preprocesssed_data.pkl'):\n",
    "    \"\"\" Convert each review to words and read from the cache file if \n",
    "        available. \n",
    "    \n",
    "    \"\"\"\n",
    "    # if cache file exists try to read from it\n",
    "    cache_data = None\n",
    "    if cache_file is not None:\n",
    "        try:\n",
    "            with open(os.path.join(cache_dir, cache_file), \"rb\") as f: # open and read binary file\n",
    "                cache_data = pickle.load(f)\n",
    "                print(\"Reading preprocessed data from cache file: {}\".format(cache_file))\n",
    "        except:\n",
    "            pass\n",
    "        \n",
    "    # if cache data does not exist create it\n",
    "    if cache_data is None:\n",
    "        # process data to create list of words for each review\n",
    "        train_words = [review_to_words(review) for review in train_data]\n",
    "        test_words = [review_to_words(review) for review in test_data]\n",
    "        \n",
    "        # write to cache file if it doesn't exist\n",
    "        if cache_file is not None:\n",
    "            cache_data = dict(train_words=train_words, test_words=test_words,\n",
    "                              train_labels=train_labels, test_labels=test_labels)\n",
    "            with open(os.path.join(cache_dir, cache_file), \"wb\") as f:\n",
    "                pickle.dump(cache_data, f)\n",
    "            print(\"Wrote preprocessed data to cache file: {}\".format(cache_file))\n",
    "            \n",
    "    else:\n",
    "        # unpack data from cache file\n",
    "        train_words = cache_data['train_words']\n",
    "        test_words = cache_data['test_words']\n",
    "        train_labels = cache_data['train_labels']\n",
    "        test_labels = cache_data['test_labels']\n",
    "        \n",
    "    return train_words, test_words, train_labels, test_labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading preprocessed data from cache file: preprocesssed_data.pkl\n"
     ]
    }
   ],
   "source": [
    "train_X, test_X, train_y, test_y = preprocess_data(train_X, test_X, train_y, test_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "227"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(train_X[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### Transform the Data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "import joblib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_dict(data, vocab_size = 5000):\n",
    "    \"\"\" Construct and return a dictionary mapping each of the most \n",
    "        frequently appearing words to a unique integer.\n",
    "        \n",
    "    \"\"\"\n",
    "    \n",
    "    word_count = {} # A dict storing the words that appear in the reviews along with how often they occur\n",
    "    for sentance in data:\n",
    "        for word in sentance:\n",
    "            if word not in word_count:\n",
    "                word_count[word] = 1\n",
    "            else:\n",
    "                word_count[word] += 1\n",
    "\n",
    "    sorted_words = sorted(word_count, key=word_count.get, reverse=True)\n",
    "    \n",
    "    word_dict = {} # This is what we are building, a dictionary that translates words into integers\n",
    "    for idx, word in enumerate(sorted_words[:vocab_size - 2]): # The -2 is so that we save room for the 'no word'\n",
    "        word_dict[word] = idx + 2                              # 'infrequent' labels\n",
    "        \n",
    "    return word_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take a look at the dictionary to make sure everything looks good. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_dict = build_dict(train_X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['movi', 'film', 'one', 'like', 'time']\n"
     ]
    }
   ],
   "source": [
    "most_freq = [key for idx, (key, val) in enumerate(word_dict.items()) if idx < 5]\n",
    "print(most_freq)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save `word_dict`\n",
    "\n",
    "Later on when we construct an endpoint which processes a submitted review we will need to make use of the `word_dict` which we have created. As such, we will save it to a file now for future use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_dir = '../data/pytorch' # folder that will store the data\n",
    "if not os.path.exists(data_dir): # check if the folder exists\n",
    "    os.makedirs(data_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(os.path.join(data_dir, 'word_dict.pkl'), \"wb\") as f:\n",
    "    pickle.dump(word_dict, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transform the Reviews\n",
    "Convert the reviews into their integer sequence representation. Shorter reviews will be padded with `0` or `1` for no word and infrequent word representation. Longer reviews will be truncated to 500 characters. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "def convert_and_pad(word_dict, review, pad=500):\n",
    "    NOWORD = 0\n",
    "    INFREQ = 1\n",
    "    current_review = [NOWORD] * pad\n",
    "        \n",
    "    for idx, word in enumerate(review[:pad]):\n",
    "        if word in word_dict:\n",
    "            current_review[idx] = word_dict[word]\n",
    "        else:\n",
    "            current_review[idx] = INFREQ\n",
    "    \n",
    "    return current_review, min(len(review), pad)\n",
    "\n",
    "def convert_reviews(word_dict, data, pad=500):\n",
    "    \"\"\" Convert each review to an integer sequence representation. Truncate\n",
    "        to 500 chars and pad with 0s and 1s accordingly.\n",
    "        \n",
    "        Arguments:\n",
    "        - word_dict: (dict) word mapping dictionary\n",
    "        - data: reviews\n",
    "        - pad: (int) length to truncate to\n",
    "        \n",
    "        Returns:\n",
    "        - train_X: feature\n",
    "        - train_X_len: length of feature set\n",
    "    \"\"\"    \n",
    "    result = []\n",
    "    lengths = []\n",
    "    \n",
    "    for review in data:\n",
    "        current_review, leng = convert_and_pad(word_dict, review, pad)\n",
    "        result.append(current_review)\n",
    "        lengths.append(leng)\n",
    "        \n",
    "    return np.array(result), np.array(lengths)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_X, train_X_len = convert_reviews(word_dict, train_X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_X, test_X_len = convert_reviews(word_dict, test_X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1974 3354    1    1   42 3932  618   91    1  219  279  816   37  207\n",
      "   89  401   11    1    1    1  122 1803 1386  345   89  175   52  237\n",
      " 2286 1349 1938  170  471 1352 2566   48  271  345  126 2162 2336   50\n",
      " 1425    1   50    1   89    1   10   31   62 2829 3845   45 3439    5\n",
      "  451  816  707   29    1  237 2502  875    1  345  116    6   85  296\n",
      "   14   37 2502   62  603 2393 4211    1 4632   26   85  105 1707 1205\n",
      "  273 2143  593  400 1733 1087 1899 1160 2829   78 3439 2829   28   35\n",
      "   45  202   30 1087 1899   33   63   17 4674 3905  658 2336    1  945\n",
      "  359 1877  352    1 2991 1425  136 1238  321    1  711   44   72 2272\n",
      "    1  356  193   52 1279   50  128 3932    1    1   26  193   60  794\n",
      " 3048 3728 3330   33 1948  489 1345    1    1  384 1342 2503   70   26\n",
      "  465    2  658  593  400  397   19 1205 1899   60  427  219 4815  633\n",
      "    1    5   37    1   96   83  208  169 1087  593    1    1   52 2272\n",
      "    1   29  891 3905  593    1    1  785  344    4   48  192  777   92\n",
      " 1707  593  400 1342 2503   27  713  852   47 3933   90  381 3400 2503\n",
      " 1451    2 1239  596  552    1    1  111    4   90   85  457 4211    1\n",
      "  238  548  164    0    0    0    0    0    0    0    0    0    0    0\n",
      "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "    0    0    0    0    0    0    0    0    0    0] \n",
      "\n",
      "159\n"
     ]
    }
   ],
   "source": [
    "print(train_X[0], '\\n')\n",
    "print(train_X_len[20])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='upload'></a>\n",
    "## Upload the Data to S3\n",
    "Save the data locally and upload to S3 later. Note that the format has to be in the form `label`, `length`, `review`.\n",
    "\n",
    "### Save and process locally\n",
    "It is important to note the format of the data that we are saving as we will need to know it when we write the training code. In our case, each row of the dataset has the form `label`, `length`, `review[500]` where `review[500]` is a sequence of 500 integers representing the words in the review."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "# necessary imports\n",
    "import pandas as pd\n",
    "import sagemaker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.concat([pd.DataFrame(train_y), pd.DataFrame(train_X_len), pd.DataFrame(train_X)], axis=1) \\\n",
    "        .to_csv(os.path.join(data_dir, 'train.csv'), header=False, index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Upload the data\n",
    "Create a SageMaker session, role, and bucket. Upload the data to the default S3 bucket.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is an object that represents the SageMaker session that we are currently operating in. This\n",
    "# object contains some useful information that we will need to access later such as our region.\n",
    "session = sagemaker.Session()\n",
    "\n",
    "bucket = session.default_bucket()\n",
    "prefix = 'sagemaker/sentiment_rnn'\n",
    "\n",
    "# This is an object that represents the IAM role that we are currently assigned. When we construct\n",
    "# and launch the training job later we will need to tell it what IAM role it should have. Since our\n",
    "# use case is relatively simple we will simply assign the training job the role we currently have.\n",
    "role = sagemaker.get_execution_role()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_data = session.upload_data(path=data_dir, bucket=bucket, key_prefix=prefix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sagemaker-us-west-2-904606187431\n"
     ]
    }
   ],
   "source": [
    "print(session.default_bucket())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='train'></a>\n",
    "## Build and Train the Pytorch Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[37m# PROGRAMMER: Justin Bellucci \u001b[39;49;00m\n",
      "\u001b[37m# DATE CREATED: 07_31_2020                                  \u001b[39;49;00m\n",
      "\u001b[37m# REVISED DATE: 12_23_2021\u001b[39;49;00m\n",
      "\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mtorch\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mnn\u001b[39;49;00m \u001b[34mas\u001b[39;49;00m \u001b[04m\u001b[36mnn\u001b[39;49;00m\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mtorch\u001b[39;49;00m\n",
      "\n",
      "\u001b[34mclass\u001b[39;49;00m \u001b[04m\u001b[32mLSTMClassifier\u001b[39;49;00m(nn.Module):\n",
      "    \u001b[33m\"\"\" LSTM based RNN to perform sentiment analysis\u001b[39;49;00m\n",
      "\u001b[33m    \u001b[39;49;00m\n",
      "\u001b[33m    \"\"\"\u001b[39;49;00m\n",
      "    \u001b[34mdef\u001b[39;49;00m \u001b[32m__init__\u001b[39;49;00m(\u001b[36mself\u001b[39;49;00m, vocab_size, embedding_dim, hidden_dim, n_layers=\u001b[34m2\u001b[39;49;00m):\n",
      "        \u001b[33m\"\"\" Initialize the model by setting up the various \u001b[39;49;00m\n",
      "\u001b[33m            layers.\u001b[39;49;00m\n",
      "\u001b[33m        \"\"\"\u001b[39;49;00m\n",
      "        \u001b[36msuper\u001b[39;49;00m(LSTMClassifier, \u001b[36mself\u001b[39;49;00m).\u001b[32m__init__\u001b[39;49;00m()\n",
      "        \n",
      "        \u001b[36mself\u001b[39;49;00m.vocab_size = vocab_size\n",
      "        \u001b[36mself\u001b[39;49;00m.n_layers = n_layers\n",
      "        \u001b[36mself\u001b[39;49;00m.hidden_dim = hidden_dim\n",
      "        \n",
      "        \u001b[36mself\u001b[39;49;00m.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx=\u001b[34m0\u001b[39;49;00m)\n",
      "        \u001b[36mself\u001b[39;49;00m.lstm = nn.LSTM(embedding_dim, hidden_dim, n_layers)\n",
      "\n",
      "        \u001b[36mself\u001b[39;49;00m.fc = nn.Linear(in_features=hidden_dim, out_features=\u001b[34m1\u001b[39;49;00m)\n",
      "        \u001b[36mself\u001b[39;49;00m.sigmoid = nn.Sigmoid()\n",
      "\n",
      "        \n",
      "    \u001b[34mdef\u001b[39;49;00m \u001b[32mforward\u001b[39;49;00m(\u001b[36mself\u001b[39;49;00m, x):\n",
      "        \n",
      "        batch_size = x.size(\u001b[34m0\u001b[39;49;00m)\n",
      "        x = x.t()\n",
      "        reviews_lengths = x[\u001b[34m0\u001b[39;49;00m,:] \u001b[37m# first row is the length of original review\u001b[39;49;00m\n",
      "        reviews = x[\u001b[34m1\u001b[39;49;00m:,:] \u001b[37m# second row to end is each review\u001b[39;49;00m\n",
      "        \n",
      "        embeds = \u001b[36mself\u001b[39;49;00m.embedding(reviews)\n",
      "        lstm_out, _ = \u001b[36mself\u001b[39;49;00m.lstm(embeds)\n",
      "        \u001b[37m# stack lstm layers before they go into fully connected layer\u001b[39;49;00m\n",
      "        lstm_out = lstm_out.contiguous().view(-\u001b[34m1\u001b[39;49;00m, \u001b[36mself\u001b[39;49;00m.hidden_dim)\n",
      "        fc_out = \u001b[36mself\u001b[39;49;00m.fc(lstm_out)\n",
      "\n",
      "        sig_out = \u001b[36mself\u001b[39;49;00m.sigmoid(fc_out)\n",
      "        \u001b[37m# reshape sig_out (seq_length, batch_size, output_size)\u001b[39;49;00m\n",
      "        sig_out = sig_out.view(-\u001b[34m1\u001b[39;49;00m, batch_size, \u001b[34m1\u001b[39;49;00m) \n",
      "        \n",
      "        \u001b[37m# here we want to grab the value of the output which\u001b[39;49;00m\n",
      "        \u001b[37m# corresponds to the last word in the original sequence.\u001b[39;49;00m\n",
      "        \u001b[37m# then we grab all in the batch. ex: out = sig_out[165, (0,50)]\u001b[39;49;00m\n",
      "        out = sig_out[reviews_lengths - \u001b[34m1\u001b[39;49;00m, \u001b[36mrange\u001b[39;49;00m(batch_size)]\n",
      "        \n",
      "        \u001b[34mreturn\u001b[39;49;00m out.squeeze() \u001b[37m# squeeze dimensions\u001b[39;49;00m\n"
     ]
    }
   ],
   "source": [
    "!pygmentize train/model.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loading in a bit of data to test the model before we use GPU to train on Sagemaker. This is important to identify any mistakes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "# necessary imports\n",
    "import torch\n",
    "import torch.utils.data\n",
    "import torch.nn as nn\n",
    "\n",
    "import torch.optim as optim\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 50\n",
    "\n",
    "# read in the fist 250 rows\n",
    "train_sample = pd.read_csv(os.path.join(data_dir, 'train.csv'), header=None, names=None, nrows=250)\n",
    "\n",
    "# turn the Pandas DF into Tensors. Labels are first.\n",
    "train_sample_y = torch.from_numpy(train_sample[[0]].values).float().squeeze()\n",
    "train_sample_X = torch.from_numpy(train_sample.drop([0], axis=1).values).long()\n",
    "\n",
    "# build dataset using TensorDataset() from Pytorch\n",
    "train_sample_dataset = torch.utils.data.TensorDataset(train_sample_X, train_sample_y)\n",
    "\n",
    "# build the dataloader using DataLoader() from Pytorch\n",
    "train_sample_loader = torch.utils.data.DataLoader(train_sample_dataset, batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>240</th>\n",
       "      <th>241</th>\n",
       "      <th>242</th>\n",
       "      <th>243</th>\n",
       "      <th>244</th>\n",
       "      <th>245</th>\n",
       "      <th>246</th>\n",
       "      <th>247</th>\n",
       "      <th>248</th>\n",
       "      <th>249</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>227</td>\n",
       "      <td>84</td>\n",
       "      <td>89</td>\n",
       "      <td>146</td>\n",
       "      <td>60</td>\n",
       "      <td>332</td>\n",
       "      <td>93</td>\n",
       "      <td>330</td>\n",
       "      <td>108</td>\n",
       "      <td>330</td>\n",
       "      <td>...</td>\n",
       "      <td>19</td>\n",
       "      <td>221</td>\n",
       "      <td>164</td>\n",
       "      <td>141</td>\n",
       "      <td>126</td>\n",
       "      <td>43</td>\n",
       "      <td>123</td>\n",
       "      <td>180</td>\n",
       "      <td>69</td>\n",
       "      <td>41</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1974</td>\n",
       "      <td>6</td>\n",
       "      <td>69</td>\n",
       "      <td>2863</td>\n",
       "      <td>300</td>\n",
       "      <td>634</td>\n",
       "      <td>7</td>\n",
       "      <td>709</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>16</td>\n",
       "      <td>537</td>\n",
       "      <td>907</td>\n",
       "      <td>311</td>\n",
       "      <td>979</td>\n",
       "      <td>2</td>\n",
       "      <td>55</td>\n",
       "      <td>2513</td>\n",
       "      <td>180</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3354</td>\n",
       "      <td>2</td>\n",
       "      <td>120</td>\n",
       "      <td>506</td>\n",
       "      <td>174</td>\n",
       "      <td>1050</td>\n",
       "      <td>3</td>\n",
       "      <td>217</td>\n",
       "      <td>1133</td>\n",
       "      <td>1505</td>\n",
       "      <td>...</td>\n",
       "      <td>439</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>657</td>\n",
       "      <td>2</td>\n",
       "      <td>620</td>\n",
       "      <td>1</td>\n",
       "      <td>34</td>\n",
       "      <td>1</td>\n",
       "      <td>77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>48</td>\n",
       "      <td>356</td>\n",
       "      <td>2</td>\n",
       "      <td>2026</td>\n",
       "      <td>6</td>\n",
       "      <td>177</td>\n",
       "      <td>16</td>\n",
       "      <td>68</td>\n",
       "      <td>...</td>\n",
       "      <td>7</td>\n",
       "      <td>915</td>\n",
       "      <td>211</td>\n",
       "      <td>12</td>\n",
       "      <td>340</td>\n",
       "      <td>72</td>\n",
       "      <td>95</td>\n",
       "      <td>6</td>\n",
       "      <td>261</td>\n",
       "      <td>2668</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>497</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>498</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>499</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>500</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>501</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>502 rows × 250 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      0    1    2     3    4     5    6    7     8     9    ...  240  241  \\\n",
       "0       1    0    1     1    0     1    1    1     1     1  ...    1    0   \n",
       "1     227   84   89   146   60   332   93  330   108   330  ...   19  221   \n",
       "2    1974    6   69  2863  300   634    7  709     1     1  ...   16  537   \n",
       "3    3354    2  120   506  174  1050    3  217  1133  1505  ...  439    1   \n",
       "4       1    1   48   356    2  2026    6  177    16    68  ...    7  915   \n",
       "..    ...  ...  ...   ...  ...   ...  ...  ...   ...   ...  ...  ...  ...   \n",
       "497     0    0    0     0    0     0    0    0     0     0  ...    0    0   \n",
       "498     0    0    0     0    0     0    0    0     0     0  ...    0    0   \n",
       "499     0    0    0     0    0     0    0    0     0     0  ...    0    0   \n",
       "500     0    0    0     0    0     0    0    0     0     0  ...    0    0   \n",
       "501     0    0    0     0    0     0    0    0     0     0  ...    0    0   \n",
       "\n",
       "     242  243  244  245  246   247  248   249  \n",
       "0      1    0    0    1    0     1    1     1  \n",
       "1    164  141  126   43  123   180   69    41  \n",
       "2    907  311  979    2   55  2513  180    16  \n",
       "3      3  657    2  620    1    34    1    77  \n",
       "4    211   12  340   72   95     6  261  2668  \n",
       "..   ...  ...  ...  ...  ...   ...  ...   ...  \n",
       "497    0    0    0    0    0     0    0     0  \n",
       "498    0    0    0    0    0     0    0     0  \n",
       "499    0    0    0    0    0     0    0     0  \n",
       "500    0    0    0    0    0     0    0     0  \n",
       "501    0    0    0    0    0     0    0     0  \n",
       "\n",
       "[502 rows x 250 columns]"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_sample.transpose()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1974,    6,   69,  ...,  356, 1192,   50],\n",
      "        [3354,    2,  120,  ...,   46, 1260,   10],\n",
      "        [   1,    1,   48,  ...,   23,  158, 1715],\n",
      "        ...,\n",
      "        [   0,    0,    0,  ...,    0,    0,    0],\n",
      "        [   0,    0,    0,  ...,    0,    0,    0],\n",
      "        [   0,    0,    0,  ...,    0,    0,    0]])\n",
      "tensor([[ 303,  135,  138,  ...,    1,    1,  161],\n",
      "        [  70,    2,  810,  ..., 1464,  172,   53],\n",
      "        [ 102,  514,  408,  ..., 3358,  597,  269],\n",
      "        ...,\n",
      "        [   0,    0,    0,  ...,    0,    0,    0],\n",
      "        [   0,    0,    0,  ...,    0,    0,    0],\n",
      "        [   0,    0,    0,  ...,    0,    0,    0]])\n",
      "tensor([[ 487,  135,  581,  ...,  796,   19,  967],\n",
      "        [   1,    1,    2,  ...,    3,  443, 1950],\n",
      "        [  12,    3,  497,  ...,  161,  198, 3269],\n",
      "        ...,\n",
      "        [   0,    0,    0,  ...,    0,    0,    0],\n",
      "        [   0,    0,    0,  ...,    0,    0,    0],\n",
      "        [   0,    0,    0,  ...,    0,    0,    0]])\n",
      "tensor([[2494,  815,  207,  ..., 2045,    2,   21],\n",
      "        [   5, 2684,  473,  ...,  214,    5,   38],\n",
      "        [ 205,  140, 1027,  ...,   75,    4,    4],\n",
      "        ...,\n",
      "        [   0,    0,    0,  ...,    0,    0,    0],\n",
      "        [   0,    0,    0,  ...,    0,    0,    0],\n",
      "        [   0,    0,    0,  ...,    0,    0,    0]])\n",
      "tensor([[ 381,   12,   19,  ..., 2513,  180,   16],\n",
      "        [ 111,    2,    5,  ...,   34,    1,   77],\n",
      "        [ 648, 4637,  459,  ...,    6,  261, 2668],\n",
      "        ...,\n",
      "        [   0,    0,    0,  ...,    0,    0,    0],\n",
      "        [   0,    0,    0,  ...,    0,    0,    0],\n",
      "        [   0,    0,    0,  ...,    0,    0,    0]])\n"
     ]
    }
   ],
   "source": [
    "for idx, (x, y) in enumerate(train_sample_loader):\n",
    "#     print(x.t()[0,:])\n",
    "    print(x.t()[1:,:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training with the small sample dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %load_ext autoreload\n",
    "# %autoreload 2\n",
    "\n",
    "from train.model import LSTMClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_sample(model, train_loader, epochs, optimizer, criterion, device):\n",
    "    \"\"\" Train a sample dataset in Jupyter notebook.\n",
    "    \"\"\"\n",
    "    for e in range(epochs):\n",
    "        model.train() # put model in training mode\n",
    "        total_loss = 0\n",
    "        \n",
    "        for batch in train_loader:\n",
    "            batch_X, batch_y = batch\n",
    "            \n",
    "            # move to GPU if available\n",
    "            batch_X = batch_X.to(device)\n",
    "            batch_y = batch_y.to(device)\n",
    "            \n",
    "#             h = tuple([each.data for each in h])\n",
    "            \n",
    "            # train the model\n",
    "            optimizer.zero_grad() # zero gradients\n",
    "            out= model.forward(batch_X)\n",
    "            \n",
    "            loss = criterion(out, batch_y)\n",
    "            loss.backward()\n",
    "            nn.utils.clip_grad_norm_(model.parameters(), 5)\n",
    "            optimizer.step()\n",
    "            \n",
    "            total_loss += loss.data.item()\n",
    "        print(\"Epoch: {}, BCELoss: {}\".format(e+1, total_loss/len(train_loader)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using cpu to train...\n",
      "Epoch: 1, BCELoss: 0.6973053932189941\n",
      "Epoch: 2, BCELoss: 0.6876913785934449\n",
      "Epoch: 3, BCELoss: 0.6794354557991028\n",
      "Epoch: 4, BCELoss: 0.6701034545898438\n",
      "Epoch: 5, BCELoss: 0.6552833437919616\n"
     ]
    }
   ],
   "source": [
    "embedding_dim = 32\n",
    "hidden_dim = 100\n",
    "vocab_size = 5000\n",
    "epochs = 5\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "print(\"Using {} to train...\".format(device))\n",
    "\n",
    "model = LSTMClassifier(vocab_size, embedding_dim, hidden_dim).to(device)\n",
    "optimizer = optim.Adam(model.parameters(), lr=0.001)\n",
    "criterion = torch.nn.BCELoss()\n",
    "\n",
    "train_sample(model, train_sample_loader, epochs, optimizer, criterion, device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='train'></a>\n",
    "## Train the Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "# necessary imports\n",
    "from sagemaker.pytorch import PyTorch\n",
    "import boto3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "pytorch_estimator = PyTorch(entry_point=\"train.py\",\n",
    "                            source_dir=\"train\",\n",
    "                            role=role,\n",
    "                            framework_version='1.0.0',\n",
    "                            py_version='py3',\n",
    "                            instance_count=1,\n",
    "                            instance_type='ml.p2.xlarge',\n",
    "                            hyperparameters={'epochs': 15,\n",
    "                                             'hidden_dim': 300})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-12-26 16:24:09 Starting - Starting the training job...\n",
      "2021-12-26 16:24:11 Starting - Launching requested ML instancesProfilerReport-1640535848: InProgress\n",
      "......\n",
      "2021-12-26 16:25:39 Starting - Preparing the instances for training............\n",
      "2021-12-26 16:27:39 Downloading - Downloading input data\n",
      "2021-12-26 16:27:39 Training - Downloading the training image......\n",
      "2021-12-26 16:28:40 Training - Training image download completed. Training in progress.\u001b[34mbash: cannot set terminal process group (-1): Inappropriate ioctl for device\u001b[0m\n",
      "\u001b[34mbash: no job control in this shell\u001b[0m\n",
      "\u001b[34m2021-12-26 16:28:25,618 sagemaker-containers INFO     Imported framework sagemaker_pytorch_container.training\u001b[0m\n",
      "\u001b[34m2021-12-26 16:28:25,648 sagemaker_pytorch_container.training INFO     Block until all host DNS lookups succeed.\u001b[0m\n",
      "\u001b[34m2021-12-26 16:28:28,673 sagemaker_pytorch_container.training INFO     Invoking user training script.\u001b[0m\n",
      "\u001b[34m2021-12-26 16:28:29,091 sagemaker-containers INFO     Module train does not provide a setup.py. \u001b[0m\n",
      "\u001b[34mGenerating setup.py\u001b[0m\n",
      "\u001b[34m2021-12-26 16:28:29,091 sagemaker-containers INFO     Generating setup.cfg\u001b[0m\n",
      "\u001b[34m2021-12-26 16:28:29,091 sagemaker-containers INFO     Generating MANIFEST.in\u001b[0m\n",
      "\u001b[34m2021-12-26 16:28:29,092 sagemaker-containers INFO     Installing module with the following command:\u001b[0m\n",
      "\u001b[34m/usr/bin/python -m pip install -U . -r requirements.txt\u001b[0m\n",
      "\u001b[34mProcessing /opt/ml/code\u001b[0m\n",
      "\u001b[34mCollecting pandas (from -r requirements.txt (line 1))\u001b[0m\n",
      "\u001b[34m  Downloading https://files.pythonhosted.org/packages/c3/e2/00cacecafbab071c787019f00ad84ca3185952f6bb9bca9550ed83870d4d/pandas-1.1.5-cp36-cp36m-manylinux1_x86_64.whl (9.5MB)\u001b[0m\n",
      "\u001b[34mCollecting numpy (from -r requirements.txt (line 2))\n",
      "  Downloading https://files.pythonhosted.org/packages/45/b2/6c7545bb7a38754d63048c7696804a0d947328125d81bf12beaa692c3ae3/numpy-1.19.5-cp36-cp36m-manylinux1_x86_64.whl (13.4MB)\u001b[0m\n",
      "\u001b[34mCollecting nltk (from -r requirements.txt (line 3))\n",
      "  Downloading https://files.pythonhosted.org/packages/90/35/8848a41a983a923aa2496fbe4ee4df796c6b32b2a59f04c2b2047aafd3c3/nltk-3.6.6-py3-none-any.whl (1.5MB)\u001b[0m\n",
      "\u001b[34mCollecting beautifulsoup4 (from -r requirements.txt (line 4))\n",
      "  Downloading https://files.pythonhosted.org/packages/69/bf/f0f194d3379d3f3347478bd267f754fc68c11cbf2fe302a6ab69447b1417/beautifulsoup4-4.10.0-py3-none-any.whl (97kB)\u001b[0m\n",
      "\u001b[34mCollecting html5lib (from -r requirements.txt (line 5))\n",
      "  Downloading https://files.pythonhosted.org/packages/6c/dd/a834df6482147d48e225a49515aabc28974ad5a4ca3215c18a882565b028/html5lib-1.1-py2.py3-none-any.whl (112kB)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied, skipping upgrade: python-dateutil>=2.7.3 in /usr/local/lib/python3.6/dist-packages (from pandas->-r requirements.txt (line 1)) (2.8.0)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied, skipping upgrade: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas->-r requirements.txt (line 1)) (2019.1)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied, skipping upgrade: click in /usr/local/lib/python3.6/dist-packages (from nltk->-r requirements.txt (line 3)) (7.0)\u001b[0m\n",
      "\u001b[34mCollecting regex>=2021.8.3 (from nltk->-r requirements.txt (line 3))\u001b[0m\n",
      "\u001b[34m  Downloading https://files.pythonhosted.org/packages/93/57/277ac1dc8b7e49f48c6fb74929dd32d27ea5e3c504a766f4420d28a673ef/regex-2021.11.10-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (670kB)\u001b[0m\n",
      "\u001b[34mCollecting joblib (from nltk->-r requirements.txt (line 3))\n",
      "  Downloading https://files.pythonhosted.org/packages/3e/d5/0163eb0cfa0b673aa4fe1cd3ea9d8a81ea0f32e50807b0c295871e4aab2e/joblib-1.1.0-py2.py3-none-any.whl (306kB)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied, skipping upgrade: tqdm in /usr/local/lib/python3.6/dist-packages (from nltk->-r requirements.txt (line 3)) (4.32.1)\u001b[0m\n",
      "\u001b[34mCollecting soupsieve>1.2 (from beautifulsoup4->-r requirements.txt (line 4))\n",
      "  Downloading https://files.pythonhosted.org/packages/72/a6/fd01694427f1c3fcadfdc5f1de901b813b9ac756f0806ef470cfed1de281/soupsieve-2.3.1-py3-none-any.whl\u001b[0m\n",
      "\u001b[34mRequirement already satisfied, skipping upgrade: six>=1.9 in /usr/local/lib/python3.6/dist-packages (from html5lib->-r requirements.txt (line 5)) (1.12.0)\u001b[0m\n",
      "\u001b[34mCollecting webencodings (from html5lib->-r requirements.txt (line 5))\n",
      "  Downloading https://files.pythonhosted.org/packages/f4/24/2a3e3df732393fed8b3ebf2ec078f05546de641fe1b667ee316ec1dcf3b7/webencodings-0.5.1-py2.py3-none-any.whl\u001b[0m\n",
      "\u001b[34mBuilding wheels for collected packages: train\n",
      "  Running setup.py bdist_wheel for train: started\u001b[0m\n",
      "\u001b[34m  Running setup.py bdist_wheel for train: finished with status 'done'\n",
      "  Stored in directory: /tmp/pip-ephem-wheel-cache-e5xphzl9/wheels/35/24/16/37574d11bf9bde50616c67372a334f94fa8356bc7164af8ca3\u001b[0m\n",
      "\u001b[34mSuccessfully built train\u001b[0m\n",
      "\u001b[34mInstalling collected packages: numpy, pandas, regex, joblib, nltk, soupsieve, beautifulsoup4, webencodings, html5lib, train\n",
      "  Found existing installation: numpy 1.16.4\n",
      "    Uninstalling numpy-1.16.4:\u001b[0m\n",
      "\u001b[34m      Successfully uninstalled numpy-1.16.4\u001b[0m\n",
      "\u001b[34m  Found existing installation: pandas 0.24.2\u001b[0m\n",
      "\u001b[34m    Uninstalling pandas-0.24.2:\u001b[0m\n",
      "\u001b[34m      Successfully uninstalled pandas-0.24.2\u001b[0m\n",
      "\u001b[34mSuccessfully installed beautifulsoup4-4.10.0 html5lib-1.1 joblib-1.1.0 nltk-3.6.6 numpy-1.19.5 pandas-1.1.5 regex-2021.11.10 soupsieve-2.3.1 train-1.0.0 webencodings-0.5.1\u001b[0m\n",
      "\u001b[34mYou are using pip version 18.1, however version 21.3.1 is available.\u001b[0m\n",
      "\u001b[34mYou should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n",
      "\u001b[34m2021-12-26 16:28:46,451 sagemaker-containers INFO     Invoking user script\u001b[0m\n",
      "\u001b[34mTraining Env:\u001b[0m\n",
      "\u001b[34m{\n",
      "    \"additional_framework_parameters\": {},\n",
      "    \"channel_input_dirs\": {\n",
      "        \"training\": \"/opt/ml/input/data/training\"\n",
      "    },\n",
      "    \"current_host\": \"algo-1\",\n",
      "    \"framework_module\": \"sagemaker_pytorch_container.training:main\",\n",
      "    \"hosts\": [\n",
      "        \"algo-1\"\n",
      "    ],\n",
      "    \"hyperparameters\": {\n",
      "        \"hidden_dim\": 300,\n",
      "        \"epochs\": 15\n",
      "    },\n",
      "    \"input_config_dir\": \"/opt/ml/input/config\",\n",
      "    \"input_data_config\": {\n",
      "        \"training\": {\n",
      "            \"TrainingInputMode\": \"File\",\n",
      "            \"S3DistributionType\": \"FullyReplicated\",\n",
      "            \"RecordWrapperType\": \"None\"\n",
      "        }\n",
      "    },\n",
      "    \"input_dir\": \"/opt/ml/input\",\n",
      "    \"is_master\": true,\n",
      "    \"job_name\": \"sagemaker-pytorch-2021-12-26-16-24-08-429\",\n",
      "    \"log_level\": 20,\n",
      "    \"master_hostname\": \"algo-1\",\n",
      "    \"model_dir\": \"/opt/ml/model\",\n",
      "    \"module_dir\": \"s3://sagemaker-us-west-2-904606187431/sagemaker-pytorch-2021-12-26-16-24-08-429/source/sourcedir.tar.gz\",\n",
      "    \"module_name\": \"train\",\n",
      "    \"network_interface_name\": \"eth0\",\n",
      "    \"num_cpus\": 4,\n",
      "    \"num_gpus\": 1,\n",
      "    \"output_data_dir\": \"/opt/ml/output/data\",\n",
      "    \"output_dir\": \"/opt/ml/output\",\n",
      "    \"output_intermediate_dir\": \"/opt/ml/output/intermediate\",\n",
      "    \"resource_config\": {\n",
      "        \"current_host\": \"algo-1\",\n",
      "        \"hosts\": [\n",
      "            \"algo-1\"\n",
      "        ],\n",
      "        \"network_interface_name\": \"eth0\"\n",
      "    },\n",
      "    \"user_entry_point\": \"train.py\"\u001b[0m\n",
      "\u001b[34m}\u001b[0m\n",
      "\u001b[34mEnvironment variables:\u001b[0m\n",
      "\u001b[34mSM_HOSTS=[\"algo-1\"]\u001b[0m\n",
      "\u001b[34mSM_NETWORK_INTERFACE_NAME=eth0\u001b[0m\n",
      "\u001b[34mSM_HPS={\"epochs\":15,\"hidden_dim\":300}\u001b[0m\n",
      "\u001b[34mSM_USER_ENTRY_POINT=train.py\u001b[0m\n",
      "\u001b[34mSM_FRAMEWORK_PARAMS={}\u001b[0m\n",
      "\u001b[34mSM_RESOURCE_CONFIG={\"current_host\":\"algo-1\",\"hosts\":[\"algo-1\"],\"network_interface_name\":\"eth0\"}\u001b[0m\n",
      "\u001b[34mSM_INPUT_DATA_CONFIG={\"training\":{\"RecordWrapperType\":\"None\",\"S3DistributionType\":\"FullyReplicated\",\"TrainingInputMode\":\"File\"}}\u001b[0m\n",
      "\u001b[34mSM_OUTPUT_DATA_DIR=/opt/ml/output/data\u001b[0m\n",
      "\u001b[34mSM_CHANNELS=[\"training\"]\u001b[0m\n",
      "\u001b[34mSM_CURRENT_HOST=algo-1\u001b[0m\n",
      "\u001b[34mSM_MODULE_NAME=train\u001b[0m\n",
      "\u001b[34mSM_LOG_LEVEL=20\u001b[0m\n",
      "\u001b[34mSM_FRAMEWORK_MODULE=sagemaker_pytorch_container.training:main\u001b[0m\n",
      "\u001b[34mSM_INPUT_DIR=/opt/ml/input\u001b[0m\n",
      "\u001b[34mSM_INPUT_CONFIG_DIR=/opt/ml/input/config\u001b[0m\n",
      "\u001b[34mSM_OUTPUT_DIR=/opt/ml/output\u001b[0m\n",
      "\u001b[34mSM_NUM_CPUS=4\u001b[0m\n",
      "\u001b[34mSM_NUM_GPUS=1\u001b[0m\n",
      "\u001b[34mSM_MODEL_DIR=/opt/ml/model\u001b[0m\n",
      "\u001b[34mSM_MODULE_DIR=s3://sagemaker-us-west-2-904606187431/sagemaker-pytorch-2021-12-26-16-24-08-429/source/sourcedir.tar.gz\u001b[0m\n",
      "\u001b[34mSM_TRAINING_ENV={\"additional_framework_parameters\":{},\"channel_input_dirs\":{\"training\":\"/opt/ml/input/data/training\"},\"current_host\":\"algo-1\",\"framework_module\":\"sagemaker_pytorch_container.training:main\",\"hosts\":[\"algo-1\"],\"hyperparameters\":{\"epochs\":15,\"hidden_dim\":300},\"input_config_dir\":\"/opt/ml/input/config\",\"input_data_config\":{\"training\":{\"RecordWrapperType\":\"None\",\"S3DistributionType\":\"FullyReplicated\",\"TrainingInputMode\":\"File\"}},\"input_dir\":\"/opt/ml/input\",\"is_master\":true,\"job_name\":\"sagemaker-pytorch-2021-12-26-16-24-08-429\",\"log_level\":20,\"master_hostname\":\"algo-1\",\"model_dir\":\"/opt/ml/model\",\"module_dir\":\"s3://sagemaker-us-west-2-904606187431/sagemaker-pytorch-2021-12-26-16-24-08-429/source/sourcedir.tar.gz\",\"module_name\":\"train\",\"network_interface_name\":\"eth0\",\"num_cpus\":4,\"num_gpus\":1,\"output_data_dir\":\"/opt/ml/output/data\",\"output_dir\":\"/opt/ml/output\",\"output_intermediate_dir\":\"/opt/ml/output/intermediate\",\"resource_config\":{\"current_host\":\"algo-1\",\"hosts\":[\"algo-1\"],\"network_interface_name\":\"eth0\"},\"user_entry_point\":\"train.py\"}\u001b[0m\n",
      "\u001b[34mSM_USER_ARGS=[\"--epochs\",\"15\",\"--hidden_dim\",\"300\"]\u001b[0m\n",
      "\u001b[34mSM_OUTPUT_INTERMEDIATE_DIR=/opt/ml/output/intermediate\u001b[0m\n",
      "\u001b[34mSM_CHANNEL_TRAINING=/opt/ml/input/data/training\u001b[0m\n",
      "\u001b[34mSM_HP_HIDDEN_DIM=300\u001b[0m\n",
      "\u001b[34mSM_HP_EPOCHS=15\u001b[0m\n",
      "\u001b[34mPYTHONPATH=/usr/local/bin:/usr/lib/python36.zip:/usr/lib/python3.6:/usr/lib/python3.6/lib-dynload:/usr/local/lib/python3.6/dist-packages:/usr/lib/python3/dist-packages\u001b[0m\n",
      "\u001b[34mInvoking script with the following command:\u001b[0m\n",
      "\u001b[34m/usr/bin/python -m train --epochs 15 --hidden_dim 300\u001b[0m\n",
      "\u001b[34mUsing cuda to train...\u001b[0m\n",
      "\u001b[34mCreating dataloader...\u001b[0m\n",
      "\u001b[34mModel loaded with embedding_dim: 42, hidden_dim: 300, vocab_size: 5000\u001b[0m\n",
      "\u001b[34mEpoch: 1, BCELoss: 0.6413684146744865\u001b[0m\n",
      "\u001b[34mEpoch: 2, BCELoss: 0.5438943347152398\u001b[0m\n",
      "\u001b[34mEpoch: 3, BCELoss: 0.4684590490496888\u001b[0m\n",
      "\u001b[34mEpoch: 4, BCELoss: 0.4117989473196925\u001b[0m\n",
      "\u001b[34mEpoch: 5, BCELoss: 0.35855578281441514\u001b[0m\n",
      "\u001b[34mEpoch: 6, BCELoss: 0.3317445662556862\u001b[0m\n",
      "\u001b[34mEpoch: 7, BCELoss: 0.3083164901757727\u001b[0m\n",
      "\u001b[34mEpoch: 8, BCELoss: 0.30728521061186886\u001b[0m\n",
      "\u001b[34mEpoch: 9, BCELoss: 0.2757405483601045\u001b[0m\n",
      "\u001b[34mEpoch: 10, BCELoss: 0.2683771094497369\u001b[0m\n",
      "\u001b[34mEpoch: 11, BCELoss: 0.2551763662878348\u001b[0m\n",
      "\u001b[34mEpoch: 12, BCELoss: 0.2526549651306503\u001b[0m\n",
      "\u001b[34mEpoch: 13, BCELoss: 0.21736777284923864\u001b[0m\n",
      "\u001b[34mEpoch: 14, BCELoss: 0.2269030100837046\u001b[0m\n",
      "\u001b[34mEpoch: 15, BCELoss: 0.19942333260361028\u001b[0m\n",
      "\u001b[34m2021-12-26 16:42:57,402 sagemaker-containers INFO     Reporting training SUCCESS\u001b[0m\n",
      "\n",
      "2021-12-26 16:43:34 Uploading - Uploading generated training model\n",
      "2021-12-26 16:43:34 Completed - Training job completed\n",
      "Training seconds: 974\n",
      "Billable seconds: 974\n"
     ]
    }
   ],
   "source": [
    "pytorch_estimator.fit({'training': training_data})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='deploy'></a>\n",
    "## Deployment\n",
    "\n",
    "Attach previous training job if necessary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Model Artifacts:  s3://sagemaker-us-west-2-904606187431/sagemaker-pytorch-2021-12-26-16-24-08-429/output/model.tar.gz\n",
      "Region:  us-west-2\n",
      "Role:  arn:aws:iam::904606187431:role/service-role/AmazonSageMaker-ExecutionRole-20211223T092713\n",
      "s3 Bucket:  sagemaker-us-west-2-904606187431\n",
      "Session:  <sagemaker.session.Session object at 0x7f9624eaa1d0>\n",
      "S3 Train Data Prefix:  sagemaker/sentiment_rnn\n"
     ]
    }
   ],
   "source": [
    "# training_job_name = \"pytorch-training-2021-12-23-18-17-23-313\"\n",
    "# # attach estimator to existing training job\n",
    "# pytorch_estimator = PyTorch.attach(training_job_name)\n",
    "\n",
    "client = boto3.client(service_name=\"sagemaker\")\n",
    "runtime = boto3.client(service_name=\"sagemaker-runtime\")\n",
    "\n",
    "model_artifacts = pytorch_estimator.model_data\n",
    "print(\"\\nModel Artifacts: \", model_artifacts)\n",
    "\n",
    "region = boto_session.region_name\n",
    "print(\"Region: \",region)\n",
    "print(\"Role: \",role)\n",
    "print(\"s3 Bucket: \",bucket)\n",
    "print(\"Session: \",session)\n",
    "print(\"S3 Train Data Prefix: \",prefix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install -U sagemaker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "image_uri = sagemaker.image_uris.retrieve(framework=\"pytorch\",\n",
    "                                          region=region,\n",
    "                                          version=\"1.8.0\",\n",
    "                                          py_version=\"py3\",\n",
    "                                          image_scope='inference', \n",
    "                                          instance_type='ml.m4.4xlarge')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model name: pytorch-serverless2021-12-26-17-36-41\n",
      "Model Arn: arn:aws:sagemaker:us-west-2:904606187431:model/pytorch-serverless2021-12-26-17-36-41\n"
     ]
    }
   ],
   "source": [
    "from time import gmtime, strftime\n",
    "\n",
    "model_name = \"pytorch-serverless\" + strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())\n",
    "print(\"Model name: \" + model_name)\n",
    "\n",
    "# # dummy environment variables\n",
    "byo_container_env_vars = {\"SAGEMAKER_CONTAINER_LOG_LEVEL\": \"20\", \"SOME_ENV_VAR\": \"myEnvVar\"}\n",
    "\n",
    "create_model_response = client.create_model(\n",
    "    ModelName=model_name,\n",
    "    Containers=[\n",
    "        {\n",
    "            \"Image\": image_uri,\n",
    "            \"Mode\": \"SingleModel\",\n",
    "            \"ModelDataUrl\": model_artifacts,\n",
    "            \"Environment\": byo_container_env_vars,\n",
    "        }\n",
    "    ],\n",
    "    ExecutionRoleArn=role,\n",
    ")\n",
    "\n",
    "print(\"Model Arn: \" + create_model_response[\"ModelArn\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Endpoint Configuration Creation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Endpoint Configuration Arn: arn:aws:sagemaker:us-west-2:904606187431:endpoint-config/pytorch-serverless-epc2021-12-26-17-36-58\n"
     ]
    }
   ],
   "source": [
    "pytorch_epc_name = \"pytorch-serverless-epc\" + strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())\n",
    "\n",
    "endpoint_config_response = client.create_endpoint_config(\n",
    "    EndpointConfigName=pytorch_epc_name,\n",
    "    ProductionVariants=[\n",
    "        {\n",
    "            \"VariantName\": \"byoVariant\",\n",
    "            \"ModelName\": model_name,\n",
    "            \"ServerlessConfig\": {\n",
    "                \"MemorySizeInMB\": 4096,\n",
    "                \"MaxConcurrency\": 1,\n",
    "            },\n",
    "        },\n",
    "    ],\n",
    ")\n",
    "\n",
    "print(\"Endpoint Configuration Arn: \" + endpoint_config_response[\"EndpointConfigArn\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Serverless Endpoint Creation\n",
    "Now that we have an endpoint configuration, we can create a serverless endpoint and deploy our model to it. When creating the endpoint, provide the name of your endpoint configuration and a name for the new endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Endpoint Arn: arn:aws:sagemaker:us-west-2:904606187431:endpoint/pytorch-serverless-ep2021-12-26-17-37-58\n"
     ]
    }
   ],
   "source": [
    "endpoint_name = \"pytorch-serverless-ep\" + strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())\n",
    "\n",
    "create_endpoint_response = client.create_endpoint(\n",
    "    EndpointName=endpoint_name,\n",
    "    EndpointConfigName=pytorch_epc_name,\n",
    ")\n",
    "\n",
    "print(\"Endpoint Arn: \" + create_endpoint_response[\"EndpointArn\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Failed\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'EndpointName': 'pytorch-serverless-ep2021-12-26-17-37-58',\n",
       " 'EndpointArn': 'arn:aws:sagemaker:us-west-2:904606187431:endpoint/pytorch-serverless-ep2021-12-26-17-37-58',\n",
       " 'EndpointConfigName': 'pytorch-serverless-epc2021-12-26-17-36-58',\n",
       " 'EndpointStatus': 'Failed',\n",
       " 'FailureReason': 'Unable to successfully stand up your model within the allotted 180 second timeout. Please ensure that downloading your model artifacts, starting your model container and passing the ping health checks can be completed within 180 seconds.',\n",
       " 'CreationTime': datetime.datetime(2021, 12, 26, 17, 37, 58, 444000, tzinfo=tzlocal()),\n",
       " 'LastModifiedTime': datetime.datetime(2021, 12, 26, 17, 47, 25, 759000, tzinfo=tzlocal()),\n",
       " 'ResponseMetadata': {'RequestId': 'd43a872c-0610-4c15-b7a2-7b4c1afdc0a0',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'x-amzn-requestid': 'd43a872c-0610-4c15-b7a2-7b4c1afdc0a0',\n",
       "   'content-type': 'application/x-amz-json-1.1',\n",
       "   'content-length': '581',\n",
       "   'date': 'Sun, 26 Dec 2021 17:47:35 GMT'},\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# wait for endpoint to reach a terminal state (InService) using describe endpoint\n",
    "import time\n",
    "\n",
    "describe_endpoint_response = client.describe_endpoint(EndpointName=endpoint_name)\n",
    "\n",
    "while describe_endpoint_response[\"EndpointStatus\"] == \"Creating\":\n",
    "    describe_endpoint_response = client.describe_endpoint(EndpointName=endpoint_name)\n",
    "    print(describe_endpoint_response[\"EndpointStatus\"])\n",
    "    time.sleep(15)\n",
    "\n",
    "describe_endpoint_response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "accuracy_score(test_y, predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "763104351884.dkr.ecr.us-west-2.amazonaws.com/pytorch-training:1.8.0-gpu-py3\n"
     ]
    }
   ],
   "source": [
    "image_uri = pytorch_estimator.training_image_uri()\n",
    "print(image_uri)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.pytorch import PyTorchModel\n",
    "\n",
    "# pytorch_model = PyTorchModel(model_data=model_artifacts,\n",
    "#                              role=role,\n",
    "#                              framework_version='1.8.0',\n",
    "#                              source_dir='serve',\n",
    "#                              entry_point='predict.py',\n",
    "#                              py_version='py3')\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Creation\n",
    "Create a model by providing your model artifacts, the container image URI, environment variables for the container (if applicable), a model name, and the SageMaker IAM role."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model name: pytorch-serverless-2021-12-a24-02-32-05\n"
     ]
    }
   ],
   "source": [
    "from time import gmtime, strftime\n",
    "\n",
    "model_name = \"pytorch-serverless-\" + strftime(\"%Y-%m-a%d-%H-%M-%S\", gmtime())\n",
    "print(\"Model name: \" + model_name)\n",
    "\n",
    "\n",
    "# create_model_response = client.create_model(\n",
    "#     ModelName=model_name,\n",
    "#     ExecutionRoleArn=role,\n",
    "#     Containers=[\n",
    "#         {\n",
    "#             \"Image\": pytorch_uri,\n",
    "#             \"Mode\": \"SingleModel\",\n",
    "#             \"ModelDataUrl\": model_artifacts,\n",
    "#         }\n",
    "#     ],\n",
    "#     ExecutionRoleArn=role,\n",
    "# )\n",
    "\n",
    "# print(\"Model Arn: \" + create_model_response[\"ModelArn\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Endpoint Configuration Creation\n",
    "\n",
    "This is where you can adjust the <b>Serverless Configuration</b> for your endpoint. The current max concurrent invocations for a single endpoint, known as <b>MaxConcurrency</b>, can be any value from <b>1 to 50</b>, and <b>MemorySize</b> can be any of the following: <b>1024 MB, 2048 MB, 3072 MB, 4096 MB, 5120 MB, or 6144 MB</b>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Endpoint Configuration Arn: arn:aws:sagemaker:us-west-2:904606187431:endpoint-config/pytorch-serverless-epc2021-12-24-02-35-11\n"
     ]
    }
   ],
   "source": [
    "pytorch_epc_name = \"pytorch-serverless-epc\" + strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())\n",
    "\n",
    "endpoint_config_response = client.create_endpoint_config(\n",
    "    EndpointConfigName=pytorch_epc_name,\n",
    "    ProductionVariants=[\n",
    "        {\n",
    "            \"VariantName\": \"byoVariant\",\n",
    "            \"ModelName\": \"pytorch-serverless-2021-12-24-model-01\",\n",
    "            \"ServerlessConfig\": {\n",
    "                \"MemorySizeInMB\": 4096,\n",
    "                \"MaxConcurrency\": 1,\n",
    "            },\n",
    "        },\n",
    "    ],\n",
    ")\n",
    "\n",
    "print(\"Endpoint Configuration Arn: \" + endpoint_config_response[\"EndpointConfigArn\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Serverless Endpoint Creation\n",
    "Now that we have an endpoint configuration, we can create a serverless endpoint and deploy our model to it. When creating the endpoint, provide the name of your endpoint configuration and a name for the new endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Endpoint Arn: arn:aws:sagemaker:us-west-2:904606187431:endpoint/pytorch-serverless-ep2021-12-24-02-35-43\n"
     ]
    }
   ],
   "source": [
    "endpoint_name = \"pytorch-serverless-ep\" + strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())\n",
    "\n",
    "create_endpoint_response = client.create_endpoint(\n",
    "    EndpointName=endpoint_name,\n",
    "    EndpointConfigName=pytorch_epc_name,\n",
    ")\n",
    "\n",
    "print(\"Endpoint Arn: \" + create_endpoint_response[\"EndpointArn\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Creating\n",
      "Failed\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'EndpointName': 'pytorch-serverless-ep2021-12-24-02-35-43',\n",
       " 'EndpointArn': 'arn:aws:sagemaker:us-west-2:904606187431:endpoint/pytorch-serverless-ep2021-12-24-02-35-43',\n",
       " 'EndpointConfigName': 'pytorch-serverless-epc2021-12-24-02-35-11',\n",
       " 'EndpointStatus': 'Failed',\n",
       " 'FailureReason': 'Unable to successfully stand up your model within the allotted 180 second timeout. Please ensure that downloading your model artifacts, starting your model container and passing the ping health checks can be completed within 180 seconds.',\n",
       " 'CreationTime': datetime.datetime(2021, 12, 24, 2, 35, 43, 964000, tzinfo=tzlocal()),\n",
       " 'LastModifiedTime': datetime.datetime(2021, 12, 24, 2, 45, 57, 215000, tzinfo=tzlocal()),\n",
       " 'ResponseMetadata': {'RequestId': 'c8e1c04e-7c12-4cd3-bd1e-bc35509e3669',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'x-amzn-requestid': 'c8e1c04e-7c12-4cd3-bd1e-bc35509e3669',\n",
       "   'content-type': 'application/x-amz-json-1.1',\n",
       "   'content-length': '581',\n",
       "   'date': 'Fri, 24 Dec 2021 02:45:57 GMT'},\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# wait for endpoint to reach a terminal state (InService) using describe endpoint\n",
    "import time\n",
    "\n",
    "describe_endpoint_response = client.describe_endpoint(EndpointName=endpoint_name)\n",
    "\n",
    "while describe_endpoint_response[\"EndpointStatus\"] == \"Creating\":\n",
    "    describe_endpoint_response = client.describe_endpoint(EndpointName=endpoint_name)\n",
    "    print(describe_endpoint_response[\"EndpointStatus\"])\n",
    "    time.sleep(15)\n",
    "\n",
    "describe_endpoint_response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='use'></a>\n",
    "## Use the Deployed Model for Inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_review = 'The simplest pleasures in life are the best, and this film is one of them. Combining a rather basic storyline of love and adventure this movie transcends the usual weekend fair with wit and unmitigated charm.'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: Convert test_review into a form usable by the model and save the results in test_data\n",
    "def convert_test_review(test_review):\n",
    "    words = review_to_words(test_review)\n",
    "    test_data = [np.array(convert_and_pad(word_dict, words)[0])]\n",
    "    \n",
    "    return test_data\n",
    "\n",
    "test_data = convert_test_review(test_review)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = runtime.invoke_endpoint(\n",
    "    EndpointName=endpoint_name,\n",
    "    Body=test_data,\n",
    "    ContentType=\"text/csv\",\n",
    ")\n",
    "\n",
    "print(response[\"Body\"].read())"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_pytorch_p36",
   "language": "python",
   "name": "conda_pytorch_p36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}