Applying_Basic_Text_Classification_experiments.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook Outline\n",
    "* **Data transform and storage**  I have taken data directly from the original source. All the data is present as a txt file. We will run a pre-process procedure to clean a data little bit and store it in more usable format later i.e in a `csv` file.\n",
    "* **Data cleaning** Since we have text data in hand, we are going to apply standard cleaning pipelines using various nlp tools available. \n",
    "\n",
    "\n",
    "\n",
    "DataSource <a href = \"http://ai.stanford.edu/~amaas/data/sentiment/\">here</a>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-24T08:56:07.846964Z",
     "start_time": "2020-05-24T08:56:07.843965Z"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from tqdm import tqdm\n",
    "import os\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-24T08:55:42.047898Z",
     "start_time": "2020-05-24T08:55:42.043899Z"
    }
   },
   "outputs": [],
   "source": [
    "data_source_path = \"./IMDB_raw_movie_review_data/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-24T08:56:24.347734Z",
     "start_time": "2020-05-24T08:56:24.339733Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['cleaned_imdb_reviews.csv',\n",
       " 'imdb.vocab',\n",
       " 'imdbEr.txt',\n",
       " 'README',\n",
       " 'test',\n",
       " 'train',\n",
       " 'transformed_data_reviews.csv']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.listdir(data_source_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['labeledBow.feat',\n",
       " 'neg',\n",
       " 'pos',\n",
       " 'unsup',\n",
       " 'unsupBow.feat',\n",
       " 'urls_neg.txt',\n",
       " 'urls_pos.txt',\n",
       " 'urls_unsup.txt']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.listdir(data_source_path+\"/train\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['labeledBow.feat', 'neg', 'pos', 'urls_neg.txt', 'urls_pos.txt']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    " os.listdir(data_source_path+\"/test\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['0_3.txt',\n",
       " '10000_4.txt',\n",
       " '10001_4.txt',\n",
       " '10002_1.txt',\n",
       " '10003_1.txt',\n",
       " '10004_3.txt',\n",
       " '10005_3.txt',\n",
       " '10006_4.txt',\n",
       " '10007_1.txt',\n",
       " '10008_2.txt']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.listdir(data_source_path+\"/train/neg\")[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['0_2.txt',\n",
       " '10000_4.txt',\n",
       " '10001_1.txt',\n",
       " '10002_3.txt',\n",
       " '10003_3.txt',\n",
       " '10004_2.txt',\n",
       " '10005_2.txt',\n",
       " '10006_2.txt',\n",
       " '10007_4.txt',\n",
       " '10008_4.txt']"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.listdir(data_source_path+\"/test/neg\")[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* As we can see data is divided into `test` and `train` folders. And the `labels` are represented as folders i.e `pos` and `neg` here.\n",
    "* each review is represented as `/train/neg/0_3.txt` i.e id for this review is `0`, star rating for that review on a 1-10 scale is `3` and the label for this review is `neg` and the data belongs to `train-set`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Data transformation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "sets = ['train','test']\n",
    "labels = ['pos','neg']\n",
    "\n",
    "labels_dict = dict(pos = 0,neg = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def transform_data(base_data_source_path):\n",
    "    \"\"\"\n",
    "    This function will read each review and it's corresponding label and other useful info one by one.\n",
    "    Will perform a small clean-up step i.e strip operation to get rid of extra spaced and store the review\n",
    "    and it's information in a dataFrame row.\n",
    "    At the end it will return the populated dataFrame\n",
    "    \n",
    "    parameters :\n",
    "    base_data_source_path - base path where the dataset is present in raw form\n",
    "    \n",
    "    returns :\n",
    "    a dataFrame populated with reviews and some meta-data\n",
    "    \"\"\"\n",
    "    \n",
    "    df_columns = ['id','text','rating','label']\n",
    "    \n",
    "    rows = []\n",
    "    \n",
    "    for set_ in tqdm(sets,total = len(sets),unit = \"set \"):\n",
    "        \n",
    "        for label in labels:\n",
    "            current_path = base_data_source_path+f\"{set_}/{label}\"\n",
    "            files = os.listdir(current_path)\n",
    "            for f in tqdm(files,total = len(files),unit = \"review \",leave = False):\n",
    "                file_name_tokens = f.split(\".\")[0].split(\"_\")\n",
    "                \n",
    "                review_id = file_name_tokens[0]\n",
    "                \n",
    "                review_rating = file_name_tokens[1]\n",
    "                \n",
    "                review_data = None\n",
    "                \n",
    "                with open(current_path+f\"/{f}\",\"r\", encoding=\"utf8\") as Fp:\n",
    "                    review_data = Fp.read()\n",
    "                \n",
    "                review_data = review_data.strip()\n",
    "                \n",
    "                rows.append((review_id,review_data,review_rating,label))\n",
    "        \n",
    "    return pd.DataFrame(columns = df_columns,data = rows)    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|                                                                                                          | 0/2 [00:00<?, ?set /s]\n",
      "  0%|                                                                                                   | 0/12500 [00:00<?, ?review /s]\u001b[A\n",
      "  3%|██▋                                                                                    | 394/12500 [00:00<00:03, 3901.03review /s]\u001b[A\n",
      "  6%|█████▌                                                                                 | 805/12500 [00:00<00:02, 3961.46review /s]\u001b[A\n",
      " 10%|████████▍                                                                             | 1232/12500 [00:00<00:02, 4049.23review /s]\u001b[A\n",
      " 13%|███████████▎                                                                          | 1638/12500 [00:00<00:02, 4052.44review /s]\u001b[A\n",
      " 16%|██████████████                                                                        | 2044/12500 [00:00<00:02, 4042.61review /s]\u001b[A\n",
      " 19%|████████████████▊                                                                     | 2435/12500 [00:00<00:02, 4001.87review /s]\u001b[A\n",
      " 23%|███████████████████▋                                                                  | 2860/12500 [00:00<00:02, 4061.56review /s]\u001b[A\n",
      " 26%|██████████████████████▍                                                               | 3268/12500 [00:00<00:02, 4067.05review /s]\u001b[A\n",
      " 29%|█████████████████████████▏                                                            | 3653/12500 [00:00<00:02, 3938.06review /s]\u001b[A\n",
      " 33%|████████████████████████████                                                          | 4086/12500 [00:01<00:02, 4036.65review /s]\u001b[A\n",
      " 36%|██████████████████████████████▉                                                       | 4496/12500 [00:01<00:01, 4043.46review /s]\u001b[A\n",
      " 39%|█████████████████████████████████▉                                                    | 4926/12500 [00:01<00:01, 4105.36review /s]\u001b[A\n",
      " 43%|████████████████████████████████████▊                                                 | 5349/12500 [00:01<00:01, 4129.84review /s]\u001b[A\n",
      " 46%|███████████████████████████████████████▊                                              | 5780/12500 [00:01<00:01, 4182.26review /s]\u001b[A\n",
      " 50%|██████████████████████████████████████████▋                                           | 6201/12500 [00:01<00:01, 4178.10review /s]\u001b[A\n",
      " 53%|█████████████████████████████████████████████▌                                        | 6618/12500 [00:01<00:01, 4150.73review /s]\u001b[A\n",
      " 56%|████████████████████████████████████████████████▍                                     | 7040/12500 [00:01<00:01, 4171.26review /s]\u001b[A\n",
      " 60%|███████████████████████████████████████████████████▎                                  | 7457/12500 [00:01<00:01, 4158.41review /s]\u001b[A\n",
      " 63%|██████████████████████████████████████████████████████▏                               | 7873/12500 [00:01<00:01, 4073.36review /s]\u001b[A\n",
      " 66%|████████████████████████████████████████████████████████▉                             | 8281/12500 [00:02<00:01, 3979.93review /s]\u001b[A\n",
      " 69%|███████████████████████████████████████████████████████████▋                          | 8680/12500 [00:02<00:00, 3901.15review /s]\u001b[A\n",
      " 73%|██████████████████████████████████████████████████████████████▍                       | 9073/12500 [00:02<00:00, 3898.16review /s]\u001b[A\n",
      " 76%|█████████████████████████████████████████████████████████████████▎                    | 9490/12500 [00:02<00:00, 3975.91review /s]\u001b[A\n",
      " 79%|████████████████████████████████████████████████████████████████████▎                 | 9928/12500 [00:02<00:00, 4077.68review /s]\u001b[A\n",
      " 83%|██████████████████████████████████████████████████████████████████████▎              | 10337/12500 [00:02<00:00, 4069.17review /s]\u001b[A\n",
      " 86%|█████████████████████████████████████████████████████████████████████████            | 10745/12500 [00:02<00:00, 3977.13review /s]\u001b[A\n",
      " 89%|███████████████████████████████████████████████████████████████████████████▊         | 11144/12500 [00:02<00:00, 3876.52review /s]\u001b[A\n",
      " 92%|██████████████████████████████████████████████████████████████████████████████▌      | 11562/12500 [00:02<00:00, 3951.63review /s]\u001b[A\n",
      " 96%|█████████████████████████████████████████████████████████████████████████████████▎   | 11959/12500 [00:02<00:00, 3887.39review /s]\u001b[A\n",
      " 99%|███████████████████████████████████████████████████████████████████████████████████▉ | 12349/12500 [00:03<00:00, 3867.98review /s]\u001b[A\n",
      "                                                                                                                                       \u001b[A\n",
      "  0%|                                                                                                   | 0/12500 [00:00<?, ?review /s]\u001b[A\n",
      "  3%|██▋                                                                                    | 381/12500 [00:00<00:03, 3773.42review /s]\u001b[A\n",
      "  6%|█████▍                                                                                 | 775/12500 [00:00<00:03, 3810.82review /s]\u001b[A\n",
      "  9%|████████                                                                              | 1169/12500 [00:00<00:02, 3837.44review /s]\u001b[A\n",
      " 13%|██████████▊                                                                           | 1578/12500 [00:00<00:02, 3909.87review /s]\u001b[A\n",
      " 16%|█████████████▋                                                                        | 1995/12500 [00:00<00:02, 3973.05review /s]\u001b[A\n",
      " 19%|████████████████▎                                                                     | 2374/12500 [00:00<00:02, 3916.30review /s]\u001b[A\n",
      " 22%|███████████████████▎                                                                  | 2800/12500 [00:00<00:02, 4013.44review /s]\u001b[A\n",
      " 26%|██████████████████████▎                                                               | 3237/12500 [00:00<00:02, 4102.56review /s]\u001b[A\n",
      " 29%|█████████████████████████▏                                                            | 3666/12500 [00:00<00:02, 4157.04review /s]\u001b[A\n",
      " 33%|████████████████████████████▏                                                         | 4098/12500 [00:01<00:02, 4192.39review /s]\u001b[A\n",
      " 36%|███████████████████████████████▏                                                      | 4539/12500 [00:01<00:01, 4243.10review /s]\u001b[A\n",
      " 40%|██████████████████████████████████                                                    | 4957/12500 [00:01<00:01, 4123.93review /s]\u001b[A\n",
      " 43%|████████████████████████████████████▉                                                 | 5366/12500 [00:01<00:01, 4028.62review /s]\u001b[A\n",
      " 46%|███████████████████████████████████████▊                                              | 5780/12500 [00:01<00:01, 4049.49review /s]\u001b[A\n",
      " 49%|██████████████████████████████████████████▌                                           | 6185/12500 [00:01<00:01, 4049.61review /s]\u001b[A\n",
      " 53%|█████████████████████████████████████████████▎                                        | 6589/12500 [00:01<00:01, 3986.84review /s]\u001b[A\n",
      " 56%|████████████████████████████████████████████████▏                                     | 6995/12500 [00:01<00:01, 4008.49review /s]\u001b[A\n",
      " 59%|██████████████████████████████████████████████████▉                                   | 7396/12500 [00:01<00:01, 3961.44review /s]\u001b[A\n",
      " 62%|█████████████████████████████████████████████████████▌                                | 7793/12500 [00:01<00:01, 3804.46review /s]\u001b[A\n",
      " 66%|████████████████████████████████████████████████████████▍                             | 8212/12500 [00:02<00:01, 3912.44review /s]\u001b[A\n",
      " 69%|███████████████████████████████████████████████████████████▏                          | 8605/12500 [00:02<00:01, 3848.64review /s]\u001b[A\n",
      " 72%|██████████████████████████████████████████████████████████████                        | 9025/12500 [00:02<00:00, 3936.63review /s]\u001b[A\n",
      " 75%|████████████████████████████████████████████████████████████████▊                     | 9421/12500 [00:02<00:00, 3943.61review /s]\u001b[A\n",
      " 79%|███████████████████████████████████████████████████████████████████▌                  | 9817/12500 [00:02<00:00, 3800.72review /s]\u001b[A\n",
      " 82%|█████████████████████████████████████████████████████████████████████▍               | 10207/12500 [00:02<00:00, 3829.94review /s]\u001b[A\n",
      " 85%|████████████████████████████████████████████████████████████████████████             | 10599/12500 [00:02<00:00, 3845.19review /s]\u001b[A\n",
      " 88%|██████████████████████████████████████████████████████████████████████████▊          | 11006/12500 [00:02<00:00, 3909.98review /s]\u001b[A\n",
      " 91%|█████████████████████████████████████████████████████████████████████████████▌       | 11398/12500 [00:02<00:00, 3878.11review /s]\u001b[A\n",
      " 95%|████████████████████████████████████████████████████████████████████████████████▍    | 11822/12500 [00:02<00:00, 3968.87review /s]\u001b[A\n",
      " 98%|███████████████████████████████████████████████████████████████████████████████████▏ | 12241/12500 [00:03<00:00, 4021.07review /s]\u001b[A\n",
      " 50%|█████████████████████████████████████████████████                                                 | 1/2 [00:06<00:06,  6.33s/set ]\u001b[A\n",
      "  0%|                                                                                                   | 0/12500 [00:00<?, ?review /s]\u001b[A\n",
      "  3%|██▊                                                                                    | 405/12500 [00:00<00:03, 4010.10review /s]\u001b[A\n",
      "  6%|█████▍                                                                                 | 788/12500 [00:00<00:02, 3954.26review /s]\u001b[A\n",
      "  9%|████████                                                                              | 1172/12500 [00:00<00:02, 3907.33review /s]\u001b[A\n",
      " 13%|██████████▊                                                                           | 1564/12500 [00:00<00:02, 3911.10review /s]\u001b[A\n",
      " 16%|█████████████▍                                                                        | 1955/12500 [00:00<00:02, 3899.12review /s]\u001b[A\n",
      " 19%|████████████████▎                                                                     | 2373/12500 [00:00<00:02, 3968.01review /s]\u001b[A\n",
      " 22%|███████████████████▏                                                                  | 2791/12500 [00:00<00:02, 4029.31review /s]\u001b[A\n",
      " 26%|██████████████████████                                                                | 3215/12500 [00:00<00:02, 4078.48review /s]\u001b[A\n",
      " 29%|████████████████████████▊                                                             | 3607/12500 [00:00<00:02, 4017.23review /s]\u001b[A\n",
      " 32%|███████████████████████████▍                                                          | 3996/12500 [00:01<00:02, 3966.03review /s]\u001b[A\n",
      " 35%|██████████████████████████████▎                                                       | 4412/12500 [00:01<00:02, 4010.66review /s]\u001b[A\n",
      " 39%|█████████████████████████████████▍                                                    | 4853/12500 [00:01<00:01, 4122.65review /s]\u001b[A\n",
      " 42%|████████████████████████████████████▏                                                 | 5261/12500 [00:01<00:01, 4060.66review /s]\u001b[A\n",
      " 45%|██████████████████████████████████████▉                                               | 5665/12500 [00:01<00:01, 4029.80review /s]\u001b[A\n",
      " 49%|█████████████████████████████████████████▊                                            | 6075/12500 [00:01<00:01, 4039.03review /s]\u001b[A\n",
      " 52%|████████████████████████████████████████████▌                                         | 6478/12500 [00:01<00:01, 4036.31review /s]\u001b[A\n",
      " 55%|███████████████████████████████████████████████▍                                      | 6892/12500 [00:01<00:01, 4066.87review /s]\u001b[A\n",
      " 59%|██████████████████████████████████████████████████▍                                   | 7337/12500 [00:01<00:01, 4162.98review /s]\u001b[A\n",
      " 62%|█████████████████████████████████████████████████████▌                                | 7783/12500 [00:01<00:01, 4247.85review /s]\u001b[A\n",
      " 66%|████████████████████████████████████████████████████████▍                             | 8209/12500 [00:02<00:01, 4251.47review /s]\u001b[A\n",
      " 69%|███████████████████████████████████████████████████████████▍                          | 8635/12500 [00:02<00:00, 4178.92review /s]\u001b[A\n",
      " 73%|██████████████████████████████████████████████████████████████▍                       | 9067/12500 [00:02<00:00, 4207.94review /s]\u001b[A\n",
      " 76%|█████████████████████████████████████████████████████████████████▎                    | 9489/12500 [00:02<00:00, 4186.49review /s]\u001b[A\n",
      " 79%|████████████████████████████████████████████████████████████████████▏                 | 9908/12500 [00:02<00:00, 4101.46review /s]\u001b[A\n",
      " 83%|██████████████████████████████████████████████████████████████████████▎              | 10340/12500 [00:02<00:00, 4164.64review /s]\u001b[A\n",
      " 86%|█████████████████████████████████████████████████████████████████████████▏           | 10758/12500 [00:02<00:00, 4156.81review /s]\u001b[A\n",
      " 90%|████████████████████████████████████████████████████████████████████████████         | 11192/12500 [00:02<00:00, 4197.91review /s]\u001b[A\n",
      " 93%|██████████████████████████████████████████████████████████████████████████████▉      | 11613/12500 [00:02<00:00, 4201.53review /s]\u001b[A\n",
      " 96%|█████████████████████████████████████████████████████████████████████████████████▉   | 12044/12500 [00:02<00:00, 4233.49review /s]\u001b[A\n",
      "100%|████████████████████████████████████████████████████████████████████████████████████▊| 12468/12500 [00:03<00:00, 4053.24review /s]\u001b[A\n",
      "                                                                                                                                       \u001b[A\n",
      "  0%|                                                                                                   | 0/12500 [00:00<?, ?review /s]\u001b[A\n",
      "  3%|██▊                                                                                    | 396/12500 [00:00<00:03, 3959.92review /s]\u001b[A\n",
      "  6%|█████▌                                                                                 | 807/12500 [00:00<00:02, 3992.13review /s]\u001b[A\n",
      " 10%|████████▍                                                                             | 1232/12500 [00:00<00:02, 4054.53review /s]\u001b[A\n",
      " 13%|███████████▎                                                                          | 1638/12500 [00:00<00:02, 4056.17review /s]\u001b[A\n",
      " 16%|█████████████▉                                                                        | 2022/12500 [00:00<00:02, 3988.79review /s]\u001b[A\n",
      " 19%|████████████████▋                                                                     | 2420/12500 [00:00<00:02, 3986.12review /s]\u001b[A\n",
      " 23%|███████████████████▋                                                                  | 2858/12500 [00:00<00:02, 4085.22review /s]\u001b[A\n",
      " 26%|██████████████████████▋                                                               | 3292/12500 [00:00<00:02, 4158.45review /s]\u001b[A\n",
      " 30%|█████████████████████████▍                                                            | 3694/12500 [00:00<00:02, 4103.32review /s]\u001b[A\n",
      " 33%|████████████████████████████▍                                                         | 4126/12500 [00:01<00:02, 4153.99review /s]\u001b[A\n",
      " 36%|███████████████████████████████▏                                                      | 4531/12500 [00:01<00:01, 4072.50review /s]\u001b[A\n",
      " 39%|█████████████████████████████████▉                                                    | 4932/12500 [00:01<00:02, 3767.88review /s]\u001b[A\n",
      " 43%|████████████████████████████████████▋                                                 | 5341/12500 [00:01<00:01, 3848.18review /s]\u001b[A\n",
      " 46%|███████████████████████████████████████▌                                              | 5752/12500 [00:01<00:01, 3923.12review /s]\u001b[A\n",
      " 49%|██████████████████████████████████████████▎                                           | 6145/12500 [00:01<00:01, 3878.67review /s]\u001b[A\n",
      " 52%|████████████████████████████████████████████▉                                         | 6540/12500 [00:01<00:01, 3888.32review /s]\u001b[A\n",
      " 55%|███████████████████████████████████████████████▋                                      | 6930/12500 [00:01<00:01, 3880.21review /s]\u001b[A\n",
      " 59%|██████████████████████████████████████████████████▍                                   | 7335/12500 [00:01<00:01, 3929.59review /s]\u001b[A\n",
      " 62%|█████████████████████████████████████████████████████▏                                | 7730/12500 [00:01<00:01, 3924.01review /s]\u001b[A\n",
      " 65%|███████████████████████████████████████████████████████▉                              | 8123/12500 [00:02<00:01, 3914.08review /s]\u001b[A\n",
      " 68%|██████████████████████████████████████████████████████████▌                           | 8515/12500 [00:02<00:01, 3904.14review /s]\u001b[A\n",
      " 72%|█████████████████████████████████████████████████████████████▌                        | 8948/12500 [00:02<00:00, 4011.65review /s]\u001b[A\n",
      " 75%|████████████████████████████████████████████████████████████████▌                     | 9382/12500 [00:02<00:00, 4093.22review /s]\u001b[A\n",
      " 78%|███████████████████████████████████████████████████████████████████▍                  | 9793/12500 [00:02<00:00, 4086.01review /s]\u001b[A\n",
      " 82%|█████████████████████████████████████████████████████████████████████▍               | 10213/12500 [00:02<00:00, 4119.54review /s]\u001b[A\n",
      " 85%|████████████████████████████████████████████████████████████████████████▎            | 10626/12500 [00:02<00:00, 4073.86review /s]\u001b[A\n",
      " 88%|███████████████████████████████████████████████████████████████████████████          | 11034/12500 [00:02<00:00, 4051.41review /s]\u001b[A\n",
      " 92%|█████████████████████████████████████████████████████████████████████████████▊       | 11449/12500 [00:02<00:00, 4068.50review /s]\u001b[A\n",
      " 95%|████████████████████████████████████████████████████████████████████████████████▊    | 11877/12500 [00:02<00:00, 4117.82review /s]\u001b[A\n",
      " 98%|███████████████████████████████████████████████████████████████████████████████████▋ | 12306/12500 [00:03<00:00, 4155.89review /s]\u001b[A\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:12<00:00,  6.29s/set ]\u001b[A\n"
     ]
    }
   ],
   "source": [
    "df_reviews = transform_data(data_source_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>text</th>\n",
       "      <th>rating</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7388</th>\n",
       "      <td>53</td>\n",
       "      <td>Home Room really surprised me. In comparison t...</td>\n",
       "      <td>10</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19803</th>\n",
       "      <td>5323</td>\n",
       "      <td>I can't believe that in the 34 prior comments,...</td>\n",
       "      <td>3</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24898</th>\n",
       "      <td>990</td>\n",
       "      <td>Do not waste your time or your money on this m...</td>\n",
       "      <td>1</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46823</th>\n",
       "      <td>7141</td>\n",
       "      <td>Okay, first the good thing : If you saw the tr...</td>\n",
       "      <td>1</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15874</th>\n",
       "      <td>1788</td>\n",
       "      <td>Giving credit where it's due, only the technic...</td>\n",
       "      <td>3</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22354</th>\n",
       "      <td>761</td>\n",
       "      <td>Absolute garbage. The reason that this is so t...</td>\n",
       "      <td>1</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>307</th>\n",
       "      <td>10277</td>\n",
       "      <td>This is very nearly a perfect film. The ideas ...</td>\n",
       "      <td>9</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41824</th>\n",
       "      <td>2642</td>\n",
       "      <td>\"Milo\" is yet another answer to a question nob...</td>\n",
       "      <td>4</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23201</th>\n",
       "      <td>8382</td>\n",
       "      <td>There was a Bugs Bunny cartoon titled \"Baby Bu...</td>\n",
       "      <td>1</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31104</th>\n",
       "      <td>4244</td>\n",
       "      <td>\"Bride of Chucky\" is one of the better horror ...</td>\n",
       "      <td>10</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23445</th>\n",
       "      <td>8601</td>\n",
       "      <td>Ok, where do we start with this little gem? Mu...</td>\n",
       "      <td>1</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48800</th>\n",
       "      <td>8921</td>\n",
       "      <td>this is only the second time i have been moved...</td>\n",
       "      <td>1</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18963</th>\n",
       "      <td>4568</td>\n",
       "      <td>Not an altogether bad start for the program --...</td>\n",
       "      <td>2</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41882</th>\n",
       "      <td>2695</td>\n",
       "      <td>Watching this movie and then listening to the ...</td>\n",
       "      <td>3</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>10022</td>\n",
       "      <td>Aro Tolbukhin burnt alive seven people in a Mi...</td>\n",
       "      <td>7</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11022</th>\n",
       "      <td>8671</td>\n",
       "      <td>Armageddon PPV&lt;br /&gt;&lt;br /&gt;The last PPV of 2006...</td>\n",
       "      <td>7</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45857</th>\n",
       "      <td>6272</td>\n",
       "      <td>It is hard to imagine anyone making a Tom Crui...</td>\n",
       "      <td>1</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25573</th>\n",
       "      <td>10516</td>\n",
       "      <td>Bridget Fonda has disappointed me several time...</td>\n",
       "      <td>10</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36701</th>\n",
       "      <td>9282</td>\n",
       "      <td>This is the first movie I've seen from Singapo...</td>\n",
       "      <td>9</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23002</th>\n",
       "      <td>8202</td>\n",
       "      <td>Is it a remake og the Thing (1982/1951), i thi...</td>\n",
       "      <td>4</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          id                                               text rating label\n",
       "7388      53  Home Room really surprised me. In comparison t...     10   pos\n",
       "19803   5323  I can't believe that in the 34 prior comments,...      3   neg\n",
       "24898    990  Do not waste your time or your money on this m...      1   neg\n",
       "46823   7141  Okay, first the good thing : If you saw the tr...      1   neg\n",
       "15874   1788  Giving credit where it's due, only the technic...      3   neg\n",
       "22354    761  Absolute garbage. The reason that this is so t...      1   neg\n",
       "307    10277  This is very nearly a perfect film. The ideas ...      9   pos\n",
       "41824   2642  \"Milo\" is yet another answer to a question nob...      4   neg\n",
       "23201   8382  There was a Bugs Bunny cartoon titled \"Baby Bu...      1   neg\n",
       "31104   4244  \"Bride of Chucky\" is one of the better horror ...     10   pos\n",
       "23445   8601  Ok, where do we start with this little gem? Mu...      1   neg\n",
       "48800   8921  this is only the second time i have been moved...      1   neg\n",
       "18963   4568  Not an altogether bad start for the program --...      2   neg\n",
       "41882   2695  Watching this movie and then listening to the ...      3   neg\n",
       "25     10022  Aro Tolbukhin burnt alive seven people in a Mi...      7   pos\n",
       "11022   8671  Armageddon PPV<br /><br />The last PPV of 2006...      7   pos\n",
       "45857   6272  It is hard to imagine anyone making a Tom Crui...      1   neg\n",
       "25573  10516  Bridget Fonda has disappointed me several time...     10   pos\n",
       "36701   9282  This is the first movie I've seen from Singapo...      9   pos\n",
       "23002   8202  Is it a remake og the Thing (1982/1951), i thi...      4   neg"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_reviews.sample(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(50000, 4)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_reviews.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# save the transformed_data\n",
    "\n",
    "df_reviews.to_csv(\"./IMDB_raw_movie_review_data/transformed_data_reviews.csv\",index = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data-cleaning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read the transformed-data\n",
    "\n",
    "df_reviews = pd.read_csv(data_source_path+\"transformed_data_reviews.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>text</th>\n",
       "      <th>rating</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>Bromwell High is a cartoon comedy. It ran at t...</td>\n",
       "      <td>9</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10000</td>\n",
       "      <td>Homelessness (or Houselessness as George Carli...</td>\n",
       "      <td>8</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>10001</td>\n",
       "      <td>Brilliant over-acting by Lesley Ann Warren. Be...</td>\n",
       "      <td>10</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>10002</td>\n",
       "      <td>This is easily the most underrated film inn th...</td>\n",
       "      <td>7</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>10003</td>\n",
       "      <td>This is not the typical Mel Brooks film. It wa...</td>\n",
       "      <td>8</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      id                                               text  rating label\n",
       "0      0  Bromwell High is a cartoon comedy. It ran at t...       9   pos\n",
       "1  10000  Homelessness (or Houselessness as George Carli...       8   pos\n",
       "2  10001  Brilliant over-acting by Lesley Ann Warren. Be...      10   pos\n",
       "3  10002  This is easily the most underrated film inn th...       7   pos\n",
       "4  10003  This is not the typical Mel Brooks film. It wa...       8   pos"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_reviews.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# nlp\n",
    "\n",
    "import spacy\n",
    "import nltk\n",
    "from bs4 import BeautifulSoup\n",
    "from sklearn.feature_extraction import stop_words\n",
    "from nltk.stem import SnowballStemmer\n",
    "from nltk.tokenize import word_tokenize\n",
    "import re\n",
    "import string\n",
    "\n",
    "lemmatizer = SnowballStemmer(language = \"english\")\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "\n",
    "stop_words_sklearn = list(stop_words.ENGLISH_STOP_WORDS)\n",
    "stop_words_spacy = list(nlp.Defaults.stop_words)\n",
    "\n",
    "stop_words = list(set(stop_words_sklearn + stop_words_spacy)) # all the stop words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pre-process pipeline\n",
    "\n",
    "def cleanpunc(review):\n",
    "    nopunc = [char for char in review if char not in string.punctuation]\n",
    "    nopunc = ''.join(nopunc)\n",
    "    return nopunc\n",
    "\n",
    "def remove_html_content(review):\n",
    "    \"\"\"\n",
    "    remove html content if any using bs4\n",
    "    \"\"\"\n",
    "    return BeautifulSoup(review).get_text()\n",
    "\n",
    "def remove_stop_words(review):\n",
    "    review = word_tokenize(review)\n",
    "    return \"\".join([word for word in review if not word in stop_words])\n",
    "    \n",
    "    \n",
    "def lemmatization(review):\n",
    "    review = word_tokenize(review)\n",
    "    return \"\".join([lemmatizer.stem(word) for word in review]) \n",
    "\n",
    "\n",
    "def clean_data(reviews):\n",
    "    \"\"\"\n",
    "    the transform fn that will apply all the transforms\n",
    "    \"\"\"\n",
    "    transformed_reviews = []\n",
    "    \n",
    "    for review in tqdm(reviews,total = reviews.shape[0],unit = \"review \"):\n",
    "        if len(review) != 0:\n",
    "            review  = review.lower()\n",
    "            review = remove_html_content(review)\n",
    "            review = cleanpunc(review)\n",
    "            review = remove_stop_words(review)\n",
    "            review_lem = lemmatization(review)\n",
    "            \n",
    "            transformed_reviews.append(\"\".join(review_lem))\n",
    "            \n",
    "    return np.array(transformed_reviews)       "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 133.32review /s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array(['bromwelhghcrtooncomedrntmeprogrmschoollfetecher35yertechprofessledbelevbromwelhghstrcloserrelttecherscrmblsurvvfnncnsghtstudentrghtpthettecherpomppettstutremndschoolknewstudentswepsodstudentrepettrburnschoolmmedreclhghclssclnenspectormscktecherstudentwelcombromwelhghexpectdultgethnkbromwelhghfrfetchptsnt',\n",
       "       'homelesshouselessgeorgcrlnsttessuyerplnhelpstreetconsdhumngoschoolworkvotemtterpeoplthnkhomelesslostcusworrthngrcsmwrrqpressurkdsucceedtechnologelectnfltworrtheyllendstreetsbutgvenbetlvestreetmonthluxurhomeentertnsetbthroompcturwllcomputtresurlkehomelessgoddrdboltlessonmelbrookdrectstrboltplyrchmnworlddecdbetsssrvljeffertmborlvestreetthrtdyluxurboltsucceedwntfuturprojectmkebuldbetboltthrownstreetbrceletlegmontorstepsdewlkhesgvenncknmpeptovgrntwrttenforehedboltmeetchrctncludwomnmolllesleynnwrrenexdncgotdvorclosehomeplslorhowrdmorrfumeteddwlsonstreettheyrsurvvorboltsnthesrechmutulgreementlkerchfghtflghtkllklledwhlloveconnectmollboltwsntnecessrplotlfestnkmelbrookobservflmprorcomedshowtendercomprslpstckworkblzesddlyoungfrnkenstenspceblmtterlkehvevlubllosedyhndmkestupdbetlkerchpeopldontknowmoneymybhomelessnstedlkemonopolmoneyormybflmnsprhelp'],\n",
       "      dtype='<U848')"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's apply the above pipeline\n",
    "\n",
    "# df_reviews.text = clean_data(df_reviews.text)\n",
    "clean_data(df_reviews.text.values[:2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['b r o m w e l   h i g h   c a r t o o n   c o m e d i   r a n   t i m e   p r o g r a m   s c h o o l   l i f e   t e a c h e r   3 5   y e a r   t e a c h   p r o f e s s   l e a d   b e l i e v   b r o m w e l   h i g h   s a t i r   c l o s e r   r e a l i t i   t e a c h e r   s c r a m b l   s u r v i v   f i n a n c i   i n s i g h t   s t u d e n t   r i g h t   p a t h e t   t e a c h e r   p o m p   p e t t i   s i t u a t   r e m i n d   s c h o o l   k n e w   s t u d e n t   s a w   e p i s o d   s t u d e n t   r e p e a t   t r i   b u r n   s c h o o l   i m m e d i   r e c a l   h i g h   c l a s s i c   l i n e   i n s p e c t o r   i m   s a c k   t e a c h e r   s t u d e n t   w e l c o m   b r o m w e l   h i g h   e x p e c t   a d u l t   a g e   t h i n k   b r o m w e l   h i g h   f a r   f e t c h   p i t i   i s n t',\n",
       "       'h o m e l e s s   h o u s e l e s s   g e o r g   c a r l i n   s t a t e   i s s u   y e a r   p l a n   h e l p   s t r e e t   c o n s i d   h u m a n   g o   s c h o o l   w o r k   v o t e   m a t t e r   p e o p l   t h i n k   h o m e l e s s   l o s t   c a u s   w o r r i   t h i n g   r a c i s m   w a r   i r a q   p r e s s u r   k i d   s u c c e e d   t e c h n o l o g   e l e c t   i n f l a t   w o r r i   t h e y l l   e n d   s t r e e t s b u t   g i v e n   b e t   l i v e   s t r e e t   m o n t h   l u x u r i   h o m e   e n t e r t a i n   s e t   b a t h r o o m   p i c t u r   w a l l   c o m p u t   t r e a s u r   l i k e   h o m e l e s s   g o d d a r d   b o l t   l e s s o n m e l   b r o o k   d i r e c t   s t a r   b o l t   p l a y   r i c h   m a n   w o r l d   d e c i d   b e t   s i s s i   r i v a l   j e f f e r i   t a m b o r   l i v e   s t r e e t   t h i r t i   d a y   l u x u r i   b o l t   s u c c e e d   w a n t   f u t u r   p r o j e c t   m a k e   b u i l d   b e t   b o l t   t h r o w n   s t r e e t   b r a c e l e t   l e g   m o n i t o r   s t e p   s i d e w a l k   h e s   g i v e n   n i c k n a m   p e p t o   v a g r a n t   w r i t t e n   f o r e h e a d   b o l t   m e e t   c h a r a c t   i n c l u d   w o m a n   m o l l i   l e s l e y   a n n   w a r r e n   e x d a n c   g o t   d i v o r c   l o s e   h o m e   p a l   s a i l o r   h o w a r d   m o r r i   f u m e   t e d d i   w i l s o n   s t r e e t   t h e y r   s u r v i v o r   b o l t   i s n t   h e s   r e a c h   m u t u a l   a g r e e m e n t   l i k e   r i c h   f i g h t   f l i g h t   k i l l   k i l l e d w h i l   l o v e   c o n n e c t   m o l l i   b o l t   w a s n t   n e c e s s a r i   p l o t   l i f e   s t i n k   m e l   b r o o k   o b s e r v   f i l m   p r i o r   c o m e d i   s h o w   t e n d e r   c o m p a r   s l a p s t i c k   w o r k   b l a z e   s a d d l   y o u n g   f r a n k e n s t e i n   s p a c e b a l   m a t t e r   l i k e   h a v e   v a l u a b l   l o s e   d a y   h a n d   m a k e   s t u p i d   b e t   l i k e   r i c h   p e o p l   d o n t   k n o w   m o n e y   m a y b   h o m e l e s s   i n s t e a d   l i k e   m o n o p o l i   m o n e y o r   m a y b   f i l m   i n s p i r   h e l p'],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_reviews.text.values[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 178,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save the cleaned reviews\n",
    "\n",
    "df_reviews.to_csv(data_source_path+\"cleaned_imdb_reviews.csv\",index = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Some visuals and  feature eng."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 179,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.style.use(\"ggplot\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 180,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read the cleaned reviews\n",
    "\n",
    "df_reviews = pd.read_csv(data_source_path+\"cleaned_imdb_reviews.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 181,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>text</th>\n",
       "      <th>rating</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>bromwell high cartoon comedy ran time programs...</td>\n",
       "      <td>9</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10000</td>\n",
       "      <td>homelessness houselessness george carlin state...</td>\n",
       "      <td>8</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>10001</td>\n",
       "      <td>brilliant overacting lesley ann warren best dr...</td>\n",
       "      <td>10</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>10002</td>\n",
       "      <td>easily underrated film inn brooks cannon sure ...</td>\n",
       "      <td>7</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>10003</td>\n",
       "      <td>typical mel brooks film slapstick movies actua...</td>\n",
       "      <td>8</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      id                                               text  rating label\n",
       "0      0  bromwell high cartoon comedy ran time programs...       9   pos\n",
       "1  10000  homelessness houselessness george carlin state...       8   pos\n",
       "2  10001  brilliant overacting lesley ann warren best dr...      10   pos\n",
       "3  10002  easily underrated film inn brooks cannon sure ...       7   pos\n",
       "4  10003  typical mel brooks film slapstick movies actua...       8   pos"
      ]
     },
     "execution_count": 181,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_reviews.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "rating\n",
       "1     10122\n",
       "2      4586\n",
       "3      4961\n",
       "4      5331\n",
       "7      4803\n",
       "8      5859\n",
       "9      4607\n",
       "10     9731\n",
       "dtype: int64"
      ]
     },
     "execution_count": 182,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# group by on ratings\n",
    "\n",
    "df_reviews.groupby('rating').size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAEJCAYAAAB/pOvWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3X9U1HW+x/HnCIHiCA0z/gjDH4RuNxWxIMluwiq7a9va0s+9/thN007KXk2oVip3293aou4aLKm5p7p6b3bqdCrZ1sq7lyXwHLm2o4AhJeJGpkcNYfDH+AMEPvcPj5PjF38N6mC8Huf4x3z4fL/f9+fDyGu+3893ZmzGGIOIiMgpegS7ABER6XoUDiIiYqFwEBERC4WDiIhYKBxERMRC4SAiIhYKB+nShgwZwrPPPhvsMq5ov/3tb4mPj78sx6qqquLmm2+mZ8+eDBky5LIcUy4NhcMVKi0tjdmzZ1vad+3ahc1mo6Sk5PIX1QmzZ88mLS3N0u52u8nKyrr8BUlAfvWrXxEZGcnWrVtxu90d9tmzZw/Tpk1jxIgRhIaGkp6eft77/8Mf/sBtt91GZGQkNpuNXbt2ddjvxRdfZPDgwfTs2ZMxY8bwt7/9LaDxdGcKB7lgLS0tl6RvR/r27Uvv3r07tQ+5fGpra0lNTWXIkCH07du3wz7Nzc1ER0eTnZ19QcFwcts777yTp5566ox98vPzefrpp3nmmWeoqKjgBz/4AZMnT+azzz67oGN1e0auSKmpqWbWrFmW9p07dxrAfPLJJ8YYY1paWkxWVpYZOHCgCQsLMwMGDDA/+9nP/LZ56623zOjRo014eLgZPHiwycrKMl6v1+9YDz74oFm0aJEZMGCAcblcHdb0ySefGMCsWbPG3HrrrSY8PNy8/PLLxuPxmGnTppnY2FjTs2dPM3z4cPPHP/7RtLe3G2OMefrppw3g92/FihXGGGMGDx5snnnmGd8xBg8ebH7961+b+fPnG4fDYfr162ceffRR09ra6utz5MgR89BDD5nIyEhz9dVXm7lz55qcnBxz3XXX+fps2bLF/PCHPzRRUVEmIiLCXH/99ea///u/zzrnH374obnxxhtNWFiY6du3r5k7d67fPD3wwANm4sSJ5s9//rMZNGiQ6dOnj7nzzjtNfX39WfdbWFhoEhMTTa9evUxUVJRJTk425eXlvp9v377d3HvvvcbhcJhevXqZUaNGmb/+9a/GGHPOuT05v6eO3Rhj/va3v5lx48aZnj17mpiYGDNjxgzT0NBw1jp3795tfvazn5moqCjTs2dPk5qaatxutzHGmLq6Osvv8Omnnz7r/k6dswt18rm2c+dOv/b29nYTExNjnnjiCb/2pKQk88ADD1zwcbqz0CBlklwmL7/8Mu+88w6rVq0iLi6Ob775hvXr1/t+vnLlSrKysigoKODWW29l165d/Pu//zv79u3jjTfe8PV75513mDZtGn//+99pa2s76zEfffRRXnzxRUaNGsVVV11Fc3Mzo0aNIjs7G4fDwfr165kzZw7R0dHMnDmTxx57jNraWurq6nj//fcBiIqKOuuYFi5cyKeffkp5ebnvEsXMmTMBWLhwIX/5y1944403+N73vsfKlStZtmyZ3yvZKVOmMHLkSMrKyujZsyc1NTVnHddnn33GnXfeybx581i1ahVfffUVDz/8MIcOHfKbJ7fbTd++ffnwww85ePAgU6ZM4bHHHuO//uu/Otzv3r17ue+++3j22We57777OHbsGBUVFYSGhvp+Pm7cOEaNGsUHH3zANddcw5YtW+jR48RJ/7nmtiPFxcX89Kc/5YUXXmDlypXs37+fX/3qV9x1112UlpZis9ks2xhjyMjIoLm5mTVr1hAVFcWzzz7LD37wA2pra4mNjWXPnj0kJyczbdo0FixYgN1uP+N8XipfffUVu3fvZtKkSX7tkyZN4q233rrs9VzRgp1OEpjzPXOYP3+++f73v+/3SvJUgwcPNq+88opfW2lpqQGMx+PxHWvYsGGmra3trDWdfDV3rlfgJ+tKT0/3PZ41a5ZJTU3tsL7TzxwmT57s1+dHP/qR+bd/+zdjjDFer9eEhYWZ1157za/P2LFj/V49R0ZG+s5Ozsf06dNNcnKyX1thYaGx2Wzmq6++MsaceBXscrnMsWPHfH2ef/55M2DAgDPut7y83ACmrq6uw58vWrTI9O/f3+8M5VxOn9vTzxxSU1PNwoUL/bbZsWOHAUxFRUWH+ywqKjKAqa6u9rUdO3bMDBgwwPzud7/ztZ3++zqXi33msH79egOYmpoav/YlS5aYiIiICz5Od6Y1h++4mTNnUlVVRXx8PHPmzOG9997zrQPs27ePHTt2kJ2djd1u9/27/fbbAdi+fbtvPzfddJPv1eq53HzzzX6P29vbyc3NJTExEZfLhd1uZ/ny5ezYsSOgMSUmJvo9HjhwIN98842v5paWFlJSUvz63HLLLX6PH3vsMd8i+G9/+1vKy8vPeszq6mrGjx/v15aamooxhs8//9zX9i//8i+Eh4d3WFtHEhIS+NGPfsTIkSO56667+NOf/sTOnTt9P9+0aRPjxo0747pLIHPrdrvJz8/3+53fcMMNwIk1gzON3+l0+voBhIeHM3bsWKqrq894rECNGDHCV9uIESMuyj47OiOSM1M4XKHCw8M5cOCApX3//v0A9OzZEzjxh7Suro4//vGPhIWF8cgjj5CYmMjBgwdpb28H4E9/+hOVlZW+f5s3b6a2tpZRo0b59nshi8Kn9128eDHPP/888+bN43//93+prKxk9uzZAS9Wh4WF+T222Wy+sZzadja//vWv2bZtG/fffz9btmwhJSWFRYsWnXWbM+3z1PaOajNn+eDjkJAQPv74Y4qLi0lOTua9995j+PDhrFmz5rzGEsjctre3s3DhQr/feWVlJbW1tb4XBuca50nGmEvyR/ejjz7y1fXRRx+d93bXXHMNcOJy3Km++eYbBgwYcFFr/K5TOFyhrr/+ejZt2mS5Tv6Pf/yDHj16MGzYMF+b3W7nrrvuoqCggI0bN/LFF19QWlpK//79iY2Npaamhvj4eMu/kwHTWevWrWPSpEnMmjWLMWPGEB8fb3mFGhYWds61jPMRHx9PWFgY//d//+fXvmHDBkvfuLg4MjMzeffdd/n973/PK6+8csb9jhgxgtLSUr+2k9fnT301HQibzcbNN9/Mk08+ybp160hNTWXFihXAiTO29evXc/jw4Q63PZ+5PV1SUhLV1dUd/s7PtE4wYsQIGhoa/M6Smpub+cc//nHRXtmfavDgwb6aBg8efN7bDRkyhJiYGP7nf/7Hr33t2rX867/+68Uu8ztN4XCFmjNnDnv37mXmzJls2rSJf/7zn7z99ts8+eST/OIXv8DpdALwH//xH7z55ptUV1dTV1fHf/7nfxISEsLw4cOBE/eNFxQU8Oyzz7JlyxZqamooLCzk4Ycfvmi1fu9736OkpIRPPvmEbdu2sWjRIj799FO/PkOHDmXr1q1UV1fT0NBAc3NzQMfq3bs3Dz/8MIsWLWLNmjVs27aNp556ii+++ML3Ctfr9fLLX/6S4uJi6urqqKioYO3atWf9I//4449TXl5OdnY2W7duZe3atcybN49p06YxaNCggGoFKCsr45lnnuHTTz/l66+/5u9//zufffaZr5bMzEza29v56U9/yvr166mrq2PNmjV8/PHHwPnN7el+//vf85e//IWsrCwqKyv55z//ydq1a5k1axZHjx7tcJsJEyZw8803M3XqVNavX8+WLVv4xS9+wbFjx5g7d+4Fj/vkWYHH48Hr9foen8vXX39NZWWl75Ln559/7tsPnAjaxx9/nLy8PFatWsXWrVvJyclh8+bNer/MhQrymod0wubNm83kyZPNNddcYyIiIszIkSPNCy+8YJqbm319li9fbm688UbTp08f07t3b5OUlGQKCwv99rN69WqTkpJievXqZfr06WNGjx7tt8h4psXv051pkXD//v3mvvvuM3369DHR0dEmMzPTLFq0yAwePNjXp7Gx0dx+++0mMjLynLeynr7gefpi9slbWfv06WOioqLM3LlzzSOPPGJGjhxpjDHm6NGjZsqUKWbIkCEmPDzc9O3b19x///3m66+/Puv4Tr2V1eVymTlz5nR4K+up3njjDXO2/2Zbtmwxt99+u+nfv78JCwszgwYNMo899pjf77CmpsZkZGSYyMhI06tXL5OQkGA+/PDD857bjm5lXbdunZk4caKx2+2+W3kfeeQRc/z48TPWevqtrOPHj/fdynrS+S5Ic9ptryf/ncsDDzzQ4Xan31zwwgsvmNjYWBMWFmZGjx5t1q5de859iz+bMfomOPnumzBhAg6Hg/feey/YpYhcEfQ+B/nOqaqqory8nFtuuYWWlhbeeOMNPvnkkwta2BTp7hQO8p1js9l45ZVXmD9/Pu3t7Vx//fWsXr36rHfiiIg/XVYSEREL3a0kIiIWCgcREbG4otccdu/eHewSOsXlctHQ0BDsMroEzYU/zYc/zce3OjMXMTEx591XZw4iImKhcBAREQuFg4iIWCgcRETEQuEgIiIWCgcREbFQOIiIiIXCQURELM75Jrhly5ZRXl5OVFQUixcvBk58WUpeXh779u2jb9++ZGVlYbfbMcawYsUKKioqCA8PJzMzk7i4OABKSkp4//33Abj77rtJS0sD4Msvv2Tp0qW0tLQwZswYZs6cqe96FREJsnOGQ1paGpMmTWLp0qW+tsLCQkaNGkVGRgaFhYUUFhYyffp0Kioq2Lt3LwUFBdTW1vLaa6/x3HPP4fV6effdd8nNzQUgJyeHpKQk7HY7r776Kg8//DDDhg3j+eefp7KykjFjxly6EQNtD915Sfd/JiGvfhCU44qIXKhzXla64YYbLN8r63a7SU1NBSA1NRW32w3Axo0bGT9+PDabjeHDh3P48GGampqorKwkISEBu92O3W4nISGByspKmpqaOHr0KMOHD8dmszF+/HjfvkREJHgC+mylAwcO4HA4AHA4HBw8eBAAj8eDy+Xy9XM6nXg8Hjwej+87jQGio6M7bD/Z/0yKioooKioCIDc31+9YF+KbgLbqvNPrDQ0NDXgM3zWaC3+aD3+aj29drrm4qB+819FXQ5xp/cBms3XY/2zS09NJT0/3Pb7SPojr9Hr1YWLf0lz403z403x8q0t/8F5UVBRNTU0ANDU1ERkZCZx45X9q0Y2NjTgcDqKjo2lsbPS1ezweHA4HTqfTr72xsZHo6OhAShIRkYsooHBISkqitLQUgNLSUpKTk33t69atwxjDtm3biIiIwOFwkJiYyObNm/F6vXi9XjZv3kxiYiIOh4NevXqxbds2jDGsW7eOpKSkizc6EREJyDkvK+Xn5/P5559z6NAh5syZw/33309GRgZ5eXkUFxfjcrnIzs4GYMyYMZSXlzN//nzCwsLIzMwEwG63c8899/DEE08AcO+99/oWuWfPns2yZctoaWkhMTHxkt+pJCIi53ZFf4d0oF/201VuZdV11G9pLvxpPvxpPr51udYcruhvghMRCZZgvchkddllOYw+PkNERCwUDiIiYqFwEBERC4WDiIhYKBxERMRC4SAiIhYKBxERsVA4iIiIhcJBREQsFA4iImKhcBAREQuFg4iIWCgcRETEQuEgIiIWCgcREbFQOIiIiIXCQURELBQOIiJioXAQERELhYOIiFgoHERExELhICIiFgoHERGxUDiIiIiFwkFERCwUDiIiYqFwEBERC4WDiIhYKBxERMRC4SAiIhYKBxERsQjtzMZr1qyhuLgYm81GbGwsmZmZ7N+/n/z8fLxeL0OHDmXevHmEhoZy/PhxlixZwpdffkmfPn1YsGAB/fr1A2D16tUUFxfTo0cPZs6cSWJi4kUZnIiIBCbgMwePx8PHH39Mbm4uixcvpr29nbKyMlatWsUdd9xBQUEBvXv3pri4GIDi4mJ69+7Nyy+/zB133MGbb74JwK5duygrK+Oll17iqaee4vXXX6e9vf3ijE5ERALSqctK7e3ttLS00NbWRktLC1dffTXV1dWkpKQAkJaWhtvtBmDjxo2kpaUBkJKSwpYtWzDG4Ha7GTduHFdddRX9+vVjwIABbN++vXOjEhGRTgn4slJ0dDSTJ09m7ty5hIWFMXr0aOLi4oiIiCAkJMTXx+PxACfONJxOJwAhISFERERw6NAhPB4Pw4YN89vvyW1OV1RURFFREQC5ubm4XK6Aav8moK067/R6Q0NDAx7Dd43mwp/mw19XnI9g/R25XHMRcDh4vV7cbjdLly4lIiKCl156icrKyjP2N8ZY2mw2W4ftZ5Kenk56errvcUNDw4UVHWSn1+tyua64MVwqmgt/mg9/mo9vtba2BjwXMTEx59034MtKVVVV9OvXj8jISEJDQxk7diw1NTUcOXKEtrY24MTZQnR0NABOp5PGxkYA2traOHLkCHa73a/99G1ERCQ4Ag4Hl8tFbW0tzc3NGGOoqqri2muvZcSIEWzYsAGAkpISkpKSALjpppsoKSkBYMOGDYwYMQKbzUZSUhJlZWUcP36c+vp69uzZQ3x8fOdHJiIiAQv4stKwYcNISUlh4cKFhISEMGTIENLT07nxxhvJz8/n7bffZujQoUyYMAGACRMmsGTJEubNm4fdbmfBggUAxMbGcsstt5CdnU2PHj2YNWsWPXro7RciIsFkMxdy0b+L2b17d0DbtT1050Wu5PyEvPqB32NdR/2W5sKf5sNfV5yPYP0d6b+6rGuvOYiIyHeXwkFERCwUDiIiYqFwEBERC4WDiIhYKBxERMRC4SAiIhYKBxERsVA4iIiIhcJBREQsFA4iImKhcBAREQuFg4iIWCgcRETEQuEgIiIWCgcREbFQOIiIiIXCQURELBQOIiJioXAQERELhYOIiFgoHERExELhICIiFgoHERGxUDiIiIiFwkFERCwUDiIiYqFwEBERC4WDiIhYKBxERMRC4SAiIhYKBxERsQjtzMaHDx9m+fLl7Ny5E5vNxty5c4mJiSEvL499+/bRt29fsrKysNvtGGNYsWIFFRUVhIeHk5mZSVxcHAAlJSW8//77ANx9992kpaV1emAiIhK4ToXDihUrSExM5NFHH6W1tZXm5mZWr17NqFGjyMjIoLCwkMLCQqZPn05FRQV79+6loKCA2tpaXnvtNZ577jm8Xi/vvvsuubm5AOTk5JCUlITdbr8oAxQRkQsX8GWlI0eO8MUXXzBhwgQAQkND6d27N263m9TUVABSU1Nxu90AbNy4kfHjx2Oz2Rg+fDiHDx+mqamJyspKEhISsNvt2O12EhISqKysvAhDExGRQAV85lBfX09kZCTLli1jx44dxMXFMWPGDA4cOIDD4QDA4XBw8OBBADweDy6Xy7e90+nE4/Hg8XhwOp2+9ujoaDweT4fHLCoqoqioCIDc3Fy//V2IbwLaqvNOrzc0NDTgMXzXaC78aT78dcX5CNbfkcs1FwGHQ1tbG3V1dTz44IMMGzaMFStWUFhYeMb+xhhLm81m67DvmdrT09NJT0/3PW5oaLjAqoPr9HpdLtcVN4ZLRXPhT/PhT/PxrdbW1oDnIiYm5rz7BnxZyel04nQ6GTZsGAApKSnU1dURFRVFU1MTAE1NTURGRvr6nzqgxsZGHA4H0dHRNDY2+to9Ho/vzENERIIj4DOHq6++GqfTye7du4mJiaGqqoprr72Wa6+9ltLSUjIyMigtLSU5ORmApKQk1q5dy6233kptbS0RERE4HA4SExN566238Hq9AGzevJmpU6denNGJyGXR9tCdl3T/Z7qEE/LqB5f0uN1Zp+5WevDBBykoKKC1tZV+/fqRmZmJMYa8vDyKi4txuVxkZ2cDMGbMGMrLy5k/fz5hYWFkZmYCYLfbueeee3jiiScAuPfee3WnkohIkHUqHIYMGeK7BfVUv/nNbyxtNpuN2bNnd7ifCRMm+O56EhGR4NM7pEVExELhICIiFgoHERGxUDiIiIiFwkFERCwUDiIiYqFwEBERC4WDiIhYdOpNcCLncr4fq3ApPuFSH60gEjidOYiIiIXCQURELBQOIiJioXAQERELhYOIiFgoHERExEK3snYTl/qbukTku0VnDiIiYqFwEBERC4WDiIhYaM1B5DtC60pyMenMQURELBQOIiJioXAQERELrTlcRqdfE74UH1Mt0p1oneXS0ZmDiIhYKBxERMRC4SAiIhYKBxERsVA4iIiIhcJBREQsFA4iImKhcBAREYtOvwmuvb2dnJwcoqOjycnJob6+nvz8fLxeL0OHDmXevHmEhoZy/PhxlixZwpdffkmfPn1YsGAB/fr1A2D16tUUFxfTo0cPZs6cSWJiYqcHJiIigev0mcNHH33EwIEDfY9XrVrFHXfcQUFBAb1796a4uBiA4uJievfuzcsvv8wdd9zBm2++CcCuXbsoKyvjpZde4qmnnuL111+nvb29s2WJiEgndCocGhsbKS8vZ+LEiQAYY6iuriYlJQWAtLQ03G43ABs3biQtLQ2AlJQUtmzZgjEGt9vNuHHjuOqqq+jXrx8DBgxg+/btnSlLREQ6qVOXlVauXMn06dM5evQoAIcOHSIiIoKQkBAAoqOj8Xg8AHg8HpxOJwAhISFERERw6NAhPB4Pw4YN8+3z1G1OV1RURFFREQC5ubm4XK6A6tZnGnUPgT4/uoLQ0NALrl/P6+4hkOdGQMcJdMNNmzYRFRVFXFwc1dXV5+xvjLG02Wy2DtvPJD09nfT0dN/jhoaG895Wup8r+fnhcrmu6Prl0mltbQ34uRETE3PefQMOh5qaGjZu3EhFRQUtLS0cPXqUlStXcuTIEdra2ggJCcHj8RAdHQ2A0+mksbERp9NJW1sbR44cwW63+9pPOnUbEREJjoDXHKZOncry5ctZunQpCxYsYOTIkcyfP58RI0awYcMGAEpKSkhKSgLgpptuoqSkBIANGzYwYsQIbDYbSUlJlJWVcfz4cerr69mzZw/x8fGdH5mIiATson+fw7Rp08jPz+ftt99m6NChTJgwAYAJEyawZMkS5s2bh91uZ8GCBQDExsZyyy23kJ2dTY8ePZg1axY9eujtFyIiwWQzF3LRv4vZvXt3QNvpC0K6h5BXPwh2CQELZM1Bz+vuof/qssuy5qCX6CIiYqFwEBERC4WDiIhYKBxERMRC4SAiIhYKBxERsVA4iIiIhcJBREQsFA4iImKhcBAREQuFg4iIWCgcRETEQuEgIiIWCgcREbFQOIiIiIXCQURELBQOIiJioXAQERELhYOIiFgoHERExELhICIiFgoHERGxUDiIiIiFwkFERCwUDiIiYqFwEBERC4WDiIhYKBxERMRC4SAiIhYKBxERsVA4iIiIhcJBREQsQgPdsKGhgaVLl7J//35sNhvp6en8+Mc/xuv1kpeXx759++jbty9ZWVnY7XaMMaxYsYKKigrCw8PJzMwkLi4OgJKSEt5//30A7r77btLS0i7K4EREJDABh0NISAg///nPiYuL4+jRo+Tk5JCQkEBJSQmjRo0iIyODwsJCCgsLmT59OhUVFezdu5eCggJqa2t57bXXeO655/B6vbz77rvk5uYCkJOTQ1JSEna7/aINUkRELkzAl5UcDofvlX+vXr0YOHAgHo8Ht9tNamoqAKmpqbjdbgA2btzI+PHjsdlsDB8+nMOHD9PU1ERlZSUJCQnY7XbsdjsJCQlUVlZehKGJiEigAj5zOFV9fT11dXXEx8dz4MABHA4HcCJADh48CIDH48Hlcvm2cTqdeDwePB4PTqfT1x4dHY3H4+nwOEVFRRQVFQGQm5vrt78L8U1AW8mVJtDnR1cQGhp6wfXred09BPLcCOg4nd3BsWPHWLx4MTNmzCAiIuKM/YwxljabzdZh3zO1p6enk56e7nvc0NBwgdVKd3IlPz9cLtcVXb9cOq2trQE/N2JiYs67b6fuVmptbWXx4sXcdtttjB07FoCoqCiampoAaGpqIjIyEjhxpnDqgBobG3E4HERHR9PY2Ohr93g8vjMPEREJjoDDwRjD8uXLGThwID/5yU987UlJSZSWlgJQWlpKcnKyr33dunUYY9i2bRsRERE4HA4SExPZvHkzXq8Xr9fL5s2bSUxM7OSwRESkMwK+rFRTU8O6desYNGgQjz/+OABTpkwhIyODvLw8iouLcblcZGdnAzBmzBjKy8uZP38+YWFhZGZmAmC327nnnnt44oknALj33nt1p5KISJDZTEeLAVeI3bt3B7Rd20N3XuRKpCsKefWDYJcQsEDWHPS87h76ry7r+msOIiLy3aRwEBERC4WDiIhYKBxERMRC4SAiIhYKBxERsVA4iIiIhcJBREQsFA4iImKhcBAREQuFg4iIWCgcRETEQuEgIiIWCgcREbFQOIiIiIXCQURELBQOIiJioXAQERELhYOIiFgoHERExELhICIiFgoHERGxUDiIiIiFwkFERCwUDiIiYqFwEBERC4WDiIhYKBxERMRC4SAiIhYKBxERsVA4iIiIhcJBREQsQoNdwEmVlZWsWLGC9vZ2Jk6cSEZGRrBLEhHptrrEmUN7ezuvv/46Tz75JHl5eaxfv55du3YFuywRkW6rS4TD9u3bGTBgAP379yc0NJRx48bhdruDXZaISLfVJS4reTwenE6n77HT6aS2ttbSr6ioiKKiIgByc3OJiYkJ7IAfbgxsO5HL6IKf33pedxsB/+27AF3izMEYY2mz2WyWtvT0dHJzc8nNzb0cZV1yOTk5wS6hy9Bc+NN8+NN8fOtyzUWXCAen00ljY6PvcWNjIw6HI4gViYh0b10iHK677jr27NlDfX09ra2tlJWVkZSUFOyyRES6rS6x5hASEsKDDz7IH/7wB9rb2/n+979PbGxssMu65NLT04NdQpehufCn+fCn+fjW5ZoLm+nogr+IiHRrXeKykoiIdC0KBxERsegSaw7dSUNDA0uXLmX//v3YbDbS09P58Y9/HOyygq69vZ2cnByio6O79W2Lhw8fZvny5ezcuRObzcbcuXMZPnx4sMsKmjVr1lBcXIzNZiM2NpbMzEzCwsKCXdZls2zZMsrLy4mKimLx4sXBQLPBAAADOklEQVQAeL1e8vLy2LdvH3379iUrKwu73X7Rj61wuMxCQkL4+c9/TlxcHEePHiUnJ4eEhASuvfbaYJcWVB999BEDBw7k6NGjwS4lqFasWEFiYiKPPvoora2tNDc3B7ukoPF4PHz88cfk5eURFhbGSy+9RFlZGWlpacEu7bJJS0tj0qRJLF261NdWWFjIqFGjyMjIoLCwkMLCQqZPn37Rj63LSpeZw+EgLi4OgF69ejFw4EA8Hk+QqwquxsZGysvLmThxYrBLCaojR47wxRdfMGHCBABCQ0Pp3bt3kKsKrvb2dlpaWmhra6OlpaXbvf/phhtusJwVuN1uUlNTAUhNTb1kHzWkM4cgqq+vp66ujvj4+GCXElQrV65k+vTp3f6sob6+nsjISJYtW8aOHTuIi4tjxowZ9OzZM9ilBUV0dDSTJ09m7ty5hIWFMXr0aEaPHh3ssoLuwIEDvpB0OBwcPHjwkhxHZw5BcuzYMRYvXsyMGTOIiIgIdjlBs2nTJqKionxnU91ZW1sbdXV1/PCHP+TFF18kPDycwsLCYJcVNF6vF7fbzdKlS/nzn//MsWPHWLduXbDL6jYUDkHQ2trK4sWLue222xg7dmywywmqmpoaNm7cyC9/+Uvy8/PZsmULBQUFwS4rKJxOJ06nk2HDhgGQkpJCXV1dkKsKnqqqKvr160dkZCShoaGMHTuWbdu2BbusoIuKiqKpqQmApqYmIiMjL8lxdFnpMjPGsHz5cgYOHMhPfvKTYJcTdFOnTmXq1KkAVFdX89e//pX58+cHuarguPrqq3E6nezevZuYmBiqqqq69Y0KLpeL2tpampubCQsLo6qqiuuuuy7YZQVdUlISpaWlZGRkUFpaSnJy8iU5jt4hfZlt3bqV3/zmNwwaNMj3ybNTpkzhxhtvDHJlwXcyHLrzraxfffUVy5cvp7W1lX79+pGZmXlJblO8UrzzzjuUlZUREhLCkCFDmDNnDldddVWwy7ps8vPz+fzzzzl06BBRUVHcf//9JCcnk5eXR0NDAy6Xi+zs7EvyHFE4iIiIhdYcRETEQuEgIiIWCgcREbFQOIiIiIXCQURELBQOIiJioXAQERGL/wfsmgee3l+4kwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist(df_reviews.rating)\n",
    "plt.title(\"User ratings on scale of 1-10\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observation** - Most reviews are rated either 1 or 10. No review has ratings = `5` or `6`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 184,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"4\" halign=\"left\">text</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>unique</th>\n",
       "      <th>top</th>\n",
       "      <th>freq</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>label</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>neg</th>\n",
       "      <td>25000</td>\n",
       "      <td>24695</td>\n",
       "      <td>got movie free job similar movies watched low ...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pos</th>\n",
       "      <td>25000</td>\n",
       "      <td>24883</td>\n",
       "      <td>loved todays variety solely cooking great stim...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        text                                                               \n",
       "       count unique                                                top freq\n",
       "label                                                                      \n",
       "neg    25000  24695  got movie free job similar movies watched low ...    3\n",
       "pos    25000  24883  loved todays variety solely cooking great stim...    5"
      ]
     },
     "execution_count": 184,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_reviews[['text','label']].groupby(\"label\").describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pos    25000\n",
       "neg    25000\n",
       "Name: label, dtype: int64"
      ]
     },
     "execution_count": 185,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# class distribution\n",
    "\n",
    "df_reviews['label'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAO4AAAD8CAYAAABw8JiyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAG0RJREFUeJzt3XlYlOXeB/DvMwPDsA0om4ApmyCQqAmmotg57h6zzquW9eaSYq7F63XM5TXLUltMU092XvWo5cEtt1OalSWJkJkihiWCAoq7xCL7Mtvz/kHOEREcFLjve+b3uS6vmmGWL+Pz9b6fZ55FkmVZBiFEKArWAQghTUfFJURAVFxCBETFJURAVFxCBETFJURAVlfciRMnYuDAgU16jp+fH5YuXfrI7/2wryNJErZu3drg7eb01FNPITY2tsHbza25PltrY8M6AGm6mzdvwtXV1azHbt26FePGjYO5X9fv27cPNjbNv1jExsYiOzsbiYmJde5PSUmBg4NDs7+fpaPiCqhdu3bN/pparRYqlQpt27Zt9tdujIeHR6u+n6WwuqnyvU6fPo1hw4bB09MTTk5OiIqKwrffflvvcVVVVYiNjYVGo4G7uzvmzZsHo9Fo+rler8fixYvh7+8PtVqN8PBwrF+/vsl5jhw5goiICKjVakRERODIkSP1HnPvVHnjxo0IDQ2FWq2Gm5sbYmJicO3aNSQmJmLcuHGm50iShIkTJwKonQJPnjwZixYtgre3N3x9fU333zs1NhqNmD9/Ptzd3aHRaBAbG4uqqirTz+/3nKVLl8LPzw8AsHjxYmzatAlHjx415fjss88A1J8ql5WVYerUqfDw8IBarUZkZCS+++47089zc3MhSRJ27dqFp59+Gg4ODggICEB8fHwTP2mxWX1xS0tLMXbsWCQmJuL06dMYMmQIRo4ciQsXLtR53McffwwfHx+kpKRg1apVWLt2LVavXm36eWxsLPbt24f169cjIyMDb775JubNm4dNmzaZneXGjRsYMWIEevTogdOnT2PlypWIi4tr9DmpqamYNm0aFixYgPPnzyMxMRHjx48HAPTp0wdr164FUDu9vnnzJtasWWN67q5du5Cfn4+EhAT88MMPDb7Hnj17UFhYiOTkZGzbtg379+/HvHnzzP695syZgxdffBG9e/c25Xj++efv+9hJkybh0KFD2Lp1K3755RdER0djxIgRyMzMrPO4+fPnY9y4cfj111/x3HPP4eWXX0ZWVpbZmYQnW5kJEybIAwYMaPQxERER8tKlS023O3bsKPft27fOYxYsWCD7+vrKsizLFy9elCVJkjMyMuo85u2335a7du1a53WWLFnS4PsuXLhQ7tChg6zT6Uz3HThwQAYgx8fHm+67+/a+fftkjUYjl5SU3Pc14+Pj5fv9Nffv31/u1KmTbDAY6t0/efLkOrc7duwo6/V6033r16+XVSqVXF5eft/nyLIsL1myRO7YsaPp9uTJk+X+/fvXy3H3Z5KVlSUDkA8ePFjnMd27d5dffvllWZZl+dKlSzIAeeXKlaaf63Q62dHRUV63bt19PwNLZPUjbn5+PmbMmIHOnTvD1dUVTk5OSE9Px+XLl+s8rnfv3nVuR0dH4/r16ygtLcWpU6cgyzIiIyPh5ORk+vPuu+82aRQ4d+4cevbsWWfjUN++fRt9zqBBgxAQEAB/f3+MHTsWGzZsQEFBgVnv16NHDygUD14EevbsCaVSabodHR0NrVaLnJwcs97HXOfOnQMAxMTE1Lk/JiYG6enpde7r1q2b6f9tbGzg5eWFvLy8Zs3DM6vfODVx4kRcuXIFy5cvh7+/P+zt7TF27FhotdpGnyfftZX2zrruTz/9VG8LqSRJZmeRZbne4x/0fCcnJ5w6dQrHjh3D4cOHsW7dOsydOxcJCQno0aNHo891dHQ0O9u9Oe+mUCjq3afT6R7qtRt6v3s/B5VKVee2JEl1tjlYOqsfcZOSkjBjxgyMHDkSXbp0gbe3Ny5evFjvcT///HOd28ePH4ePjw80Go2pIFeuXEFQUFCdP4GBgWZnCQ8Px4kTJ2AwGEz3/fjjjw98nlKpRExMDN555x2kpqbC29sb27dvB/CfBfzu12yqlJSUOs8/fvw4VCqV6Xfz9PTEjRs36jzn9OnTdW6rVKoHZggPDwdQ+3dyt+TkZNPPSC2rL25ISAi2bduG3377DWlpaXjhhRfuu4ClpaVh8eLFuHDhArZv3441a9Zg9uzZAICgoCBMmjQJU6ZMQXx8PLKzs3HmzBls3rwZH3zwgdlZpk+fjvz8fLzyyivIyMhAQkICFi5c2OhzvvzyS6xatQqpqam4cuUKvvjiC1y9ehVhYWEAAH9/fwDA/v37kZ+fj/LycrPz3FFYWIiZM2ciIyMDBw8exKJFizBlyhTTiD1w4EAcPnwYu3btQnZ2Nt5//30kJyfXeQ1/f39kZmYiPT0dBQUFqKmpqfc+gYGBGDNmDGbMmIFDhw4hMzMTcXFxOHv2LF5//fUm57ZkVl/cTz/9FEajET179sSzzz6LoUOHIioqqt7jXn31VVy+fBmRkZGYNWsWpk+fbiouAGzYsAGzZ8/GsmXLEBYWhgEDBmDLli0ICAgwO4uvry8OHDiAkydPolu3boiLi8NHH33U6HPatGmDAwcOYOjQoQgODsbcuXPxxhtvYNKkSQCAqKgoxMXFYdq0afDy8sKsWbPMznPH6NGj4ezsjL59+2Ls2LEYPnw4li9fbvr5hAkTMHPmTMyaNQuRkZG4evUqXnvttTqvMXnyZERFRaFPnz7w8PDAjh077vteGzduxJAhQ/DSSy+ha9euOHbsGL766it07ty5ybktmSTfu3JCCOGe1Y+4hIiIikuIgKi4hAiIikuIgKi4hAiIikuIgKi4hAiIikuIgKi4hAiIikuIgKi4hAiIikuIgKi4hAiIikuIgKi4hAiIimvhcnNz0blzZ0yYMAEREREYPXo0KisrkZCQgO7du6NLly6YNGmS6YwU8+fPR1hYGCIiIjBnzhzG6UlD6EB6C5ebmwt/f3/8+OOPiI6OxqRJkxAQEID169cjISEBwcHBGD9+PJ544gmMHz8evXv3RmZmJiRJQnFxsdmXOiGti0ZcK/DYY48hOjoaAPDSSy8hISEB/v7+CA4OBlB76pmkpCRoNBqo1WrTyd3pmj78ouJaAXNPEWtjY4OTJ09i1KhR+OKLLzB06NAWTkYeFhXXCly5cgXHjx8HAOzYsQMDBw5Ebm4usrOzAQDx8fHo378/ysvLUVJSguHDh2P16tVIS0tjGZs0wupPiG4NQkNDsWXLFkydOhWdOnXCmjVr0KtXL4wZMwZ6vR5RUVGYNm0aioqK8Mwzz6C6uhqyLGPVqlWso5MG0MYpC5ebm4sRI0bg7NmzrKOQZkRTZUIERCMuIQKiEZcQAVFxCREQbVUWnLGsFIbbBTAU5sNYUQZZrweMhtr/yjKgUEBSKgGFEpKtLRTOLlC6eUDZ1gMKe9rBQlRUXM4Zy0qhzcmENuc8DPm3YCgqgKEoH4aiQhhvF0DW1r/qnbkkewco27hD2bb2j6KtO2y8fKAK6gzbgBAo1PbN+JuQ5kQbpzhiKml2BrRZGdBmZ8Bw6zqbMAoFbNr7QdUpFKqgUCozZ6i4DMk6LarPpKDqRBJqfjkB/c1rrCM1TqGA7WP+UPfoA/snY6AKjaidhpNWR8VtZYaSYlSfTEbViSRUp52AXFXJOtJDU2hcoY6Khv2TMVA/0ZvWmVsRFbcV6At+R+XRb1H181FoM38DjEbWkZqfrQrqiEjY94qBQ8wQKJycWSeyaFTcFiLLMmrSTqL84B5UnUwCDAbWkVqNZKeGQ8xgOI14DqogupJ8S6DiNjNjeRkqDh9A+Td7ob92mXUc5lTB4XD6yxg4xAyCpLJjHcdiUHGbiTYnE+Vf7Ubl0UOQa6pZx+GOQuMCx4Ej4TR8FGy827OOIzwq7iPSXc5B8ZZPUH0iiXUUMSiUcBz0NDQvvgIbd0/WaYRFxX1I+t9voWTrOlQe+doyNza1MMnODk4jnodmzEQonDWs4wiHittEhpJilH6+CeVf7wV0WtZxhCc5OkMzegKcnhkLhZ2adRxhUHHNZKyuQtm/t6FsXzzkygrWcSyO0s0Dmhdi4Tj4GUhK2hP3Qai4Zqg+k4KiNUtgyLvBOorFsw0KRdvZb0HlF8Q6CteouI0wVlWiePMaVHyzr/ZIG9I6bGzh8kIsnMdMoNG3AVTcBtAoyx6Nvg2j4t6DRlnO0Oh7X1Tcu9Sc/QWFH71FoyyHbINC4fb6Eti292MdhQtU3D+UfbUbxRtWWNU+xaKRHBzhNncZ7KP6so7CnNUXV9brcXvd8tqpMeGfQgGXCTOhGT2BdRKmrLq4hpJiFL47FzVnT7OOQprI4alhaBv3htUeuGC1xdVeykLBkr/R+qzAVMFhcH9jJZRuHqyjtDqrLG7lsR9Q9NFbkKurWEchj0jR1h3ub6yAXcjjrKO0KqsrbtmBz1G8fgV91WNBJJUd3BYuh31kNOsorcaqilu6Nx4lm9ewjkFago0t3Oa/B4feT7FO0iqsprglOzaidOs61jFIS1Iq4TZnCRxiBrNO0uKs4hIkVForYTCg8MNFqEz6jnWSFmfxxS3d+y8qrTUxGlC4YhEqjyeyTtKiLLq4Zft3omTz31nHIK3NYEDh+wtQdeoY6yQtxmKLW5H4LYo3rGQdg7Ci16Fw2VzUZJ5lnaRFWGRxtVkZuP33JfSVj5WTtTUoWDYHhsJ81lGancUV11BUgIKlcyDXPPxV7IjlMN5ZHh7hqoY8sqjiyjotCpa9DkNBHusohCPaC+ko+vtS1jGalUUVt2jte7XX5iHkHpVHvkHp3n+xjtFsLKa4Zf/ehsrDB1jHIBwr+WytxWxptojiVv/yM4rpax/yIEYjCpcvhO5aLuskj0z44hrKSlC48i3ASGeuIA8mV5Sj8MNFkA161lEeifDFLV63AsbbhaxjEIHosjNQtkfs9V2hi1t1PBGVid+wjkEEVLL9n9DmZrOO8dCELa6hrARFn7zHOgYRlV6HolVvCztlFra4NEUmj0rkKbOQxaUpMmkuok6ZhSsuTZFJsxJ0yixccUt3bKQpMmlWuuwMVHy3n3WMJhGquPq8Gyg/uId1DGKBSnf8E8aaatYxzCZUcUu2rgP0OtYxiAUyFOaj/MudrGOYTZjianOzUZn4LesYxIKV7tkCY1kp6xhmEaa4JVs+AYxG1jGIBZMrylC6+zPWMcwiRHFr0tNQfTKZdQxiBcq/+hz6gt9Zx3ggIYpb/NnHrCMQKyHX1KB0+z9Zx3gg7otblXoc2nNnWMcgVqTi+/3Q37rOOkajuC9u+YHPWUcg1sZoQPnXfH/tyHVx9Xk3UJ36E+sYxApVfL+f6xPMcV3c8q/30pZkwoSxtASVyd+zjtEgbosr67So+F6s3dCIZeF5Lz1ui1uZfBjGktusYxArpj1/FtrsTNYx7ovb4vL8rx2xHuUHd7OOcF9cFlebcx7azF9ZxyAElUcPwVhexjpGPVwWt4LOj0w4IddUozLpEOsY9XBZ3Kqfj7KOQIhJ1Ykk1hHq4a642ktZMPx+k3UMQkyqz5yCsbqKdYw6uCtu1QkabQlndFpUpx5nnaIO7opbzeG0hBDepstcFddQVABtVgbrGITUU53yI2SO9uLjqrhVJ5LoKvKES8bSYmgz+PmKkr/iEsIpnra/cFNcWa9Hza8prGMQ0iCeNlBxU1zd5WzINfweRkWI7solGKv5OIUrN8XldWduQkyMBugunmedAgBXxaWtyYR/vCyn3BRXx8kHQkhjqLh3kfV6aC+Jd8U0Yn2ouHfRXc4GdFrWMQh5IP3Vy1xsoOKiuLRhigiDkw1UVFxCmkibw3555aK4hrwbrCMQYjY9B8srH8UtKmAdgRCzGTlYXjkpbj7rCISYjYfllXlxZb0extJi1jEIMRsPM0TmxTUUF9KhfEQohqJC1hE4KG4h+2kHIU0hV1XAWFXJNAPz4vKwok9IU7GeLjMvLusPgJCHwXoDFfviltGGKSIe1htUmRcXOj3rBIQ0nZ7tcsu8uLLRwDoCIU0mG9gut8yLCwONuERAjItrw/TdAZz36YKMwTNZxyCkSXq5+KIzw/dnXtyf1R2wlQ7FJYJpa9+OaXGZT5VtFBLrCIQ0mY2CbXWYF1dJxSUCYj3gMC8u6w+AkIfBesBhXlxnNfPVbEKajPVyy7y4Ho52rCMQ0mQeTmyXW+bFdWP8ARDyMNwdVUzfn3lxWX8AhDSVnY0CzmpbphmYF5f1lIOQpuJhsGFeXJWNAhraQEUE4s7BYMO8uAAfHwQh5uJhlshHcTmYehBiLjcOllcuittOo2YdgRCz8bC8clHcEE9n1hEIMVuIpxPrCHwUt7MXFZeIg4fllYviBns6QSnRPsuEf+1d7Zl/hwtwUly1rRJ+bg6sYxDyQDyMtgAnxQWA0HZ8fCCENCaUiltXqJeGdQRCHoiXAYab4vIyBSGkMbwsp9wUN9jTifnByYQ05jFONkwBHBVXbatEV18X1jEIaVBPv7asI5hwU1wAiAl0Zx2BkAbFBPGzfPJVXI4+GELu5qhSIqpDG9YxTLgq7mNtHOBP3+cSDj3p1xa2Sn7qwk+SP/Sj6TLhEG/LJXfFpfVcwhulJKFvgBvrGHVwV9wuvi5o48DHJndCAKCLjwauDuyPwb0bd8VVSBL6BtCoS/jB40ZT7ooLAMPD27GOQAiA2isWDAnlb3nksriRHdrQ1mXChZhAd3g6sz/H1L24LC4AjOrmyzoCIRjdnc/lkNvi/iXcG/a2StYxiBXza+uAnh352c3xbtwW18nOBkNDvVjHIFbsvzie9XFbXIDfaQqxfGpbBZ5+3Jt1jAZxXdxgT2dE+NARQ6T1DQn1gpMdv1fY4Lq4APDcEzTqktY3plt71hEaxX1xB4Z40VdDpFX1D3JHCCdnumgI98VVKiRM7xvIOgaxEkpJwox+/C9v3BcXAP4U7IHHvelkcqTlDQv3QoC7I+sYDyREcQFgVgz//woSsamUCkyNDmAdwyzCFLdHhzbo7c/nl+HEMozu7svFBb3MIUxxAWBmv0DQeSBJS3BUKfFyLz/WMcwmVHFDvJwxmPamIi1gXM8OcLUX5zhwoYoLADP6BcCB9mEmzcjHRY0Xe3RgHaNJhCuuj4s9ZvWnDVWkeUgAFg0Nhb1KrMFAuOICwOhuvojs4Mo6BrEA/9XNF5EcnXbVXEIWV5IkvDEklA77I4/Ex0WN1wSdvQlZXADwdbXHq4J+6IS9O1NkBxW/BxI0RtjiAjRlJg9P1CnyHUIXl6bM5GGIPEW+Q+jiArVT5jkDOrGOQQRho5CweHiYsFPkO4QvLgCM7OKD57rzffwk4cPf/twJ3duLv3plEcUFgNl/DuLqamqEP6O6+mK0hfwDbzHFtVEo8N7Ix9He1Z51FMKhJx5ztahVKospLgC42NtixV+7wFGwvWBIy/JxUeODkY/DhqPLZD4qy/lN/hDo7oR3/hIOBR1GRAA42Cqx8q8R3F2061FZXHGB2os0TesrxgHRpOVIABYPD0OQhxPrKM3OIosLAC/38sMYOi+zVZszIBh/CvZgHaNFWGxxAeD1AcF4JoLfk1qTlvM/TwXhuScsYwvy/Vh0cSVJwv8O7oxhYXTwvTWZ3jcA/x0l1vG1TWXRxQVqL5T91rAwKq+VmBrtj0m9/VjHaHGSLMsy6xCtwSjLePe7THz5603WUUgLea1/EMb1tOyR9g6rKS4AyLKMDxMuYPcv11lHIc1IQu2GKEtep72XVRX3jk9/zsX/JV+E1f3iFsjORoE3h4Za3UkErbK4AJCUXYA3D6ajQmtgHYU8JE9nO6x4tgtC21nfVS6strgAcLGgAn/796+4VlzFOgppoi4+Gnz4bATcHC1rjyhzWXVxAaCkSocF+88i5cpt1lGImZ5+3BsLBofA1oL2PW4qqy8uAOiNRqw+ko3PT19jHYU0QilJeO2pILwY+RjrKMxRce+y/7cbWJmQhUodrffyxs1RhbeHh+FJP7p+FEDFred6cRWWHsrAqSvFrKOQPwwL88KcAcHQqMW5REhLo+LehyzL2JN2HR8fzUEVjb7MuDmqsGBwCPoHWeaBAo+CitsIGn3ZoVG2cVTcB6DRt3XRKGseKq6ZbpRU4R/JF/FdRh7tcdUCVEoFRnXzxeTefnAR6HKXrFBxm+h8Xhk+Sc7B8UtFrKNYBIUEDAtrh6nR/vB2oRP9mYuK+5BSr97GJ0dz8NvNUtZRhBUT5I4Z/QIQ6G55p5ZpaVTcR5SYlY9/JOfgUmEl6yjC6ObrglkxgehqAScmZ4WK2wwMRhmHz+dh9y/XceZ6Ces4XFJIQJ8ANzzfvT16+buxjiM8Km4zy/q9HHvSruHbc3m0BxaANg62GNnFB6O6+tA6bDOi4raQ8ho9vk6/hT1p13GpsIJ1nFYX4eOC0d19MTDE06oPBmgpVNxWkHrlNg6m38KxiwUoqtSxjtNifFzUiAlyx9OPeyPY05l1HItGxW1FRlnGbzdKkZSdj+ScAuE3aEkAwrw1iAl0R0yQu0WeeJxXVFyGrt6uRHJOAZKyC5B2vQQGI/9/FXY2CkR1bIOYQHf0C3SHu5Md60hWiYrLiWqdAVn55ci4VYaMvFJk3irDpcJKGBj+9aiUCgR5OCK0nQahXs7o7OWMQHdHi7p4lqiouBy7u8wXfi/DrdJqFFRoUVBeg5JqfbO8h4TaLb/uTnZwd7SDj4sanamk3KPiCkqrN6KgogYF5VrTf0urdTAYZRhkGXqDDKMsQ6mQav9IEmyUElztVXB3UsHd0Q4eTnZo62gLGwWVUzRUXEIERP/UEiIgKi4hAqLiEiIgKi4hAqLiEiIgKi6HcnNzERoaiilTpiA8PByDBw9GVVUVcnJyMHToUPTo0QP9+vVDZmYmACAnJwe9evVCVFQU3nzzTTg50a6Hlo6Ky6msrCzMnDkT6enpcHV1xd69e/HKK6/g448/RmpqKlasWIEZM2YAAOLi4hAXF4eUlBT4+PgwTk5aA32Py6Hc3FwMGjQIWVlZAIAPPvgAOp0Oy5YtQ0hIiOlxNTU1yMjIgJubG/Ly8mBjY4PS0lL4+PigvLycVXzSCmxYByD3Z2f3n533lUol8vLy4OrqirS0NIapCC9oqiwIjUYDf39/7N69G0Dt+Z7PnDkDAOjVqxf27t0LANi5cyezjKT1UHEFsm3bNmzatAldu3ZFeHg4vvzySwDA6tWr8dFHH6Fnz564efMmXFxcGCclLY3WcS1AZWUl7O3tIUkSdu7ciR07dphKTSwTreNagNTUVMyaNQuyLMPV1RWbN29mHYm0MBpxCREQreMSIiAqLiECouISIiAqLiECouISIiAqLiECouISIiAqLiECouISIiAqLiECouISIiAqLiECouISIiAqLiECouISIiAqLiECouISIqD/B6xkPIglp11SAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.pie(df_reviews['label'].value_counts(),labels = ['pos','neg'])\n",
    "plt.title(\"label distribution\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observation** - Perfectly balanced class labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 187,
   "metadata": {},
   "outputs": [],
   "source": [
    "reviews = df_reviews[['text','label']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 188,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>bromwell high cartoon comedy ran time programs...</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>homelessness houselessness george carlin state...</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>brilliant overacting lesley ann warren best dr...</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>easily underrated film inn brooks cannon sure ...</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>typical mel brooks film slapstick movies actua...</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text label\n",
       "0  bromwell high cartoon comedy ran time programs...   pos\n",
       "1  homelessness houselessness george carlin state...   pos\n",
       "2  brilliant overacting lesley ann warren best dr...   pos\n",
       "3  easily underrated film inn brooks cannon sure ...   pos\n",
       "4  typical mel brooks film slapstick movies actua...   pos"
      ]
     },
     "execution_count": 188,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 189,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add a review length feature \n",
    "\n",
    "reviews['length'] = reviews['text'].apply(len)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 190,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>label</th>\n",
       "      <th>length</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>bromwell high cartoon comedy ran time programs...</td>\n",
       "      <td>pos</td>\n",
       "      <td>487</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>homelessness houselessness george carlin state...</td>\n",
       "      <td>pos</td>\n",
       "      <td>1290</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>brilliant overacting lesley ann warren best dr...</td>\n",
       "      <td>pos</td>\n",
       "      <td>517</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>easily underrated film inn brooks cannon sure ...</td>\n",
       "      <td>pos</td>\n",
       "      <td>361</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>typical mel brooks film slapstick movies actua...</td>\n",
       "      <td>pos</td>\n",
       "      <td>307</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text label  length\n",
       "0  bromwell high cartoon comedy ran time programs...   pos     487\n",
       "1  homelessness houselessness george carlin state...   pos    1290\n",
       "2  brilliant overacting lesley ann warren best dr...   pos     517\n",
       "3  easily underrated film inn brooks cannon sure ...   pos     361\n",
       "4  typical mel brooks film slapstick movies actua...   pos     307"
      ]
     },
     "execution_count": 190,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(min,max review lenghts) -  (17, 8625)\n"
     ]
    }
   ],
   "source": [
    "print(\"(min,max review lenghts) - \",(reviews.length.min(),reviews.length.max()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAEJCAYAAAB/pOvWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XtcVPW+//HXyHjDUZwZRINIxcveZSGcsNB2gsjjdNrW3m5zd7VSu1MaWj202951KuPsHckhNSsNt9XZ7h4esatdCMHKPBvlcryUeMv0iCIMKiMVCt/fH/6ch7RAYUABeT8fjx6P5jvf9V3f9WE571lrzayxGWMMIiIip+jU2hMQEZG2R+EgIiIWCgcREbFQOIiIiIXCQURELBQOIiJioXA4Tz3zzDMMHjz4rIydk5ODzWZj79699T5uaUuWLMFut5+Vsf2xZ88exo4dS48ePbDZbK0yB5vNxttvv31O1zl58mQSExPP6TrPZMCAATz//POtPY3zksKhHZk8eTI2mw2bzYbdbsflcjFy5EieffZZPB5Pnb6PPvoo69ata/TYgwcP5plnnmlU31GjRlFSUkJoaGhTpn9Ge/fuxWazkZOTU6f9pptu4v/+7/9adF3NMWfOHEpLSyksLKSkpKRV5lBSUsLEiRNbZd2t4fnnn2fAgAGtPY0OReHQzlx99dWUlJTwww8/8OWXX3LPPffw97//nWHDhlFcXOzr53A4CA4ObvH1V1dX06VLF/r160enTudm9+nevTt9+/Y9J+tqjG3btnHFFVcwZMgQ+vXr16hljDEcO3asxebQr18/unXr1mLjifySwqGdOfnCHBoayrBhw5g6dSr//Oc/CQwM5P777/f1++Vppb1793LDDTcQHBxM9+7diYiI4K9//SsA8fHx7Nixg2effdZ3ZPL999/7Thd99NFH/OY3v6Fbt268/vrrDZ5GKigo4IorrqBbt24MGzaMzz//3PdcQ8vY7XaWLFkCQHh4OABjxozBZrP53inWd1rp448/5vLLL6dr166EhISQlJTE0aNHfc+fPAXy+uuv079/f3r16sXvf/97Dh48eNr6VlZWct9999GnTx+6detGTEwMn332me95m83GF198wZtvvonNZmPy5Mn1jnNyzqtXryY6OpquXbvy6aefAvD5559z1VVX0b17d8LCwpgyZQrl5eW+5wICAtizZ0+d8f7xj3/QrVs3Dh065JvHqaeVvF4vDz/8MGFhYQQGBhIdHc2KFSt8z0+aNIlJkyb5HmdkZGCz2Vi0aJGv7c477+TGG288bX1+admyZURFRdGtWzcGDBjAzJkz6/wd4uPjufvuu3nuuefo168fLpeLyZMn1+lTW1vLE088QZ8+fXA4HNx8882kpaX5/uZLlizh6aefZvfu3b7989Sj3Orqah5++GFcLhd9+/bl0Ucfpaamxvf8V199xVVXXUXPnj3p2bMnw4cP9/0t5DSMtBt33nmnGTt2bL3P/fWvfzU2m82UlpYaY4z585//bAYNGuR7/vrrrzdjx441BQUFZteuXSY7O9v813/9lzHGmPLycjNgwADzyCOPmJKSElNSUmKOHz9uVq9ebQDzq1/9yrz33ntm586dZs+ePb72PXv2GGOM7/HgwYPNBx98YLZs2WKmTp1qunXrZvbu3Vunz8llTgoICDAZGRnGGGPy8/MNYP77v//blJSU+LYlIyPDBAQE+JYpKioyAQEBJjk52WzZssV8/PHHJjw83EyaNKlOrXr16mVuvvlms3HjRvP111+biy66yNxxxx2nrfHEiRNN//79zSeffGK2bNlipk+fbjp37my+/fZbY4wxJSUlZuTIkebWW281JSUl5tChQ/WOk5GRYWw2m4mJiTFffPGF2bFjhyktLTVffPGF6d69u0lPTzfFxcXmn//8p4mPjzdXX321qa2tNTU1NSYsLMzMmTOnznjjxo0zN954o+8xYN566y1jjDG1tbUmPj7exMXFmS+//NLs2LHDvPbaa6Zz584mKyvLGGPM4sWLzQUXXOBbftKkSaZPnz7m5ptv9rWFh4ebV199tcHa/HL/y8jIML179zZLly41O3bsMLm5ueayyy6r83eIi4szQUFBJjk52Xz77bdm1apVJigoyPzpT3/y9UlNTTU9evQwS5cuNcXFxSY1NdU4nU7f37yqqsrMmjXLXHjhhb79s7Ky0hhjTP/+/U3v3r3Niy++aIqLi82yZctMQECAefPNN40xxhw/ftw4nU4zY8YMU1xcbIqLi82KFSvMmjVrGtxOOUHh0I6cLhxWrVplAPM///M/xhhrOERGRpo///nPDY49aNAgy/MnX9CXLl1ab/svw2HRokW+PseOHTMXXXSRefLJJ+td5qRTw2HPnj0GMKtXr67T55fhMGnSJDNixIg6fVauXGlsNpv5/vvvjTEnahUcHGx++uknX58XX3zR9OvXr8EabNu2zQDmo48+qtMeHR1tpkyZ4nscFxdn7rrrrgbHOTlnwPIiFBcXZ2bNmlWnbffu3QYwBQUFxhhjZs2aZS6++GLf8wcOHDB2u918+OGHvrZTw2H16tWma9eulqCaMmWK+f3vf2+MMeb77783gNm8ebMxxpiwsDDz0ksvmZCQEGOMMcXFxQYwW7dubXCbfrn/9e/f3xImubm5BjAej8e3vZdddlmdPvfdd5+JjY31PQ4NDTVPPfVUnT433XRTnb/5c889Z/r372+ZU//+/c31119fp+2aa67xhZ7H46l3n5Iz02ml84T5//dPbOjTM8nJycyZM4crr7ySWbNmsWbNmkaPfcUVVzSq38iRI33/b7fbueKKK9iyZUuj19NYmzdvZvTo0XXa4uLiMMbUWd/FF19M165dfY/DwsI4cOBAg+OeXPaXY48ePZrNmzf7NdcRI0bUeZyXl0daWhoOh8P33yWXXAKcuJYBJ07vfPvtt+Tl5QHw97//HbfbzTXXXFPvOvLy8qiuriYsLKzOuG+//bZvzP79+zNw4ECys7PZunUrhw4dIikpiZ9++olNmzaRnZ1NWFgYQ4cObdR2HTx4kN27dzNz5sw667z22msB2L59u69vVFRUnWVP/TscOXKEffv2ERsbW6fPqfvSmZxufKfTyd13380111zDtddeS0pKClu3bm302B1Z2/l8oDTLpk2bsNlsRERE1Pv8lClT+Ld/+zc++eQTVq9ezbXXXssf/vCHRn0cskePHn7NyZxyw9+TF69PbaupqaG2ttavsRsKwVPbu3TpYnnO+HETYmOMXx9ZDQgIsFw0rq2tZdasWdx+++2W/icvbl988cXExMSwdOlSRowYwdKlS7n11lsb/DhvbW0tQUFBvjA51ak1SEhI4IsvviAgIIDf/OY3dO/endGjR/PFF1+wdu1aEhISGr1tJ/9u//mf/8mYMWMsz1944YX1zgFO/B1OLn+mNzWNcbrxAd544w0efvhhPvvsMz7//HOefvpp5s2bx3333ef3OjsCHTmcB44cOcKrr77K2LFjcbvdDfa74IILmDJlCkuXLmXx4sW88847HDlyBDjxD+zUi3j+OPWjs8ePHycvL4+LL74YgJCQEAD27dvn61NYWFjnxfrkP/IzzWPYsGHk5ubWacvNzcVms/nehftj2LBhAJajqi+//NL3XHPFxMSwefNmBg8ebPnP4XD4+t1xxx0sW7aMoqIi8vPzufPOO0875qFDh/jpp58sY1500UW+fgkJCeTm5pKVlcXYsWN9bV988QU5OTlNCoe+ffsSHh7O1q1b692Wxn6SKigoiNDQUL755ps67b/8GHZz989LL72UmTNnsmrVKu666y5ef/11v8fqKBQO7Ux1dTX79++npKSELVu28Oabb3LFFVfw888/8+qrrza43EMPPcTHH3/Mjh072Lx5MytWrCA8PJyePXsCMHDgQL7++mt++OEHysrK/HpHn5KSwscff8y3337LAw88wIEDB3jggQeAE9+j6N+/P8888wzfffcdX331FTNmzKjzjjE4OBiHw8Fnn33G/v37qaioqHc9jz32GPn5+cycOZPvvvuOTz75hGnTpnHbbbfVeTFsqkGDBvHHP/6RpKQkPv30U7777jsefvhhNm3axGOPPeb3uKf693//d9577z1mzJhBYWEhO3bs4JNPPuGuu+7ixx9/9PW75ZZbOHz4MJMnTyYyMpLhw4c3OGZCQgKJiYlMmDCBzMxMdu7cyYYNG3jllVd444036vSrqKjg/fff9wVBQkICq1atorS0tEnhAPDCCy+Qnp7O888/z6ZNm9i6dSsrV65s8jvyRx55hLS0NN555x22bdtGWloan332WZ19Y+DAgezfv59vvvmGsrIyqqqqGjX29u3bmTVrFl999RW7d+/mm2++4csvv2zWm4iOQuHQznz55ZdccMEFhIeHc9VVV/H6669z6623smnTptN+I9oYQ3JyMpdeeimjR4/m6NGjrFq1yvcP8Nlnn+Xw4cP86le/ok+fPvzwww9NnttLL73E008/TVRUFF9//TXvvfee7/SC3W7nH//4B6WlpURHR/Pggw/ywgsv1PmuRKdOnZg/fz7vvvsu4eHhREdH17ueyMhI3n//fXJzcxk+fDi3334748aNY+HChU2e8y8tWrSIa665hkmTJjF8+HC+/vprPvzwQ3796183e2w48THd7OxsNm7cyNVXX01kZCQzZsygZ8+edO7c2dcvODiYcePGUVhYyB133HHaMW02G++//z4TJkxg5syZ/PrXv2bcuHF89NFHDBo0yNevX79+XHLJJfTs2dNX28jISHr37s2gQYOaHKy333477777Lh999BFXXHEFI0aM4JlnniEsLKxJ4yQnJ/PQQw/x8MMPEx0dzbp163jkkUfqHH2MHz+eP/7xj4wbN44+ffrwl7/8pVFj9+jRg23btnHzzTczdOhQbrjhBkaNGsW8efOaNMeOyGb8OQkrInIWTZ06laKiIjZs2NDaU+mwdEFaRFrVvn37yMzMZMyYMQQEBPDBBx+wdOlSvbtvZTpyEJFWdeDAAW666Sb+93//13dRfdq0adxzzz2tPbUOTeEgIiIWuiAtIiIWCgcREbFo1xekT/1CVWMFBwdTVlZ2FmbT/qk2DVNt6qe6NKyt1qaxv8OiIwcREbFQOIiIiIXCQURELBQOIiJioXAQERELhYOIiFic8aOsCxYsID8/n6CgIFJTU+s89/777/P222+zaNEievXqhTGGjIwMCgoK6Nq1K0lJSb4fn8nJyfH94PmECROIj48HYOfOncyfP5/q6mqio6OZMmVKs374Q0REmu+MRw7x8fE88cQTlvaysjI2btxIcHCwr62goID9+/eTnp7Ovffey6JFiwDwer0sX76cOXPmMGfOHJYvX47X6wVO/ErTfffdR3p6Ovv376ewsLCltk1ERPx0xnC45JJL6vxC1Ul/+9vfuO222+q8y1+/fj2jR4/GZrMxdOhQjh49SkVFBYWFhURGRvp+ZzYyMpLCwkIqKir48ccfGTp0KDabjdGjR9f7U4ciInJu+fUN6fXr1+NyuRgwYECddo/HU+dIwu124/F48Hg8dX6+0uVy1dt+sn9DsrKyyMrKAk786tip62osu93e5OUO/GFUve19M9c2ef1tmT+16ShUm/qpLg1r77Vpcjj8/PPPrFixgqeeesryXH03eD3dD8E39YawiYmJJCYm+h7789X0lvxKe1v8anxztNWv+7cFqk39VJeGtdXanLXbZxw4cIDS0lIee+wxHnzwQcrLy5k1axaHDh3C7XbXKUZ5eTlOpxOXy0V5ebmv3ePx4HQ6cbvdddrLy8txuVxNnZKIiLSwJofDRRddxKJFi5g/fz7z58/H7XbzH//xH/Tu3ZuYmBjWrFmDMYbi4mICAwNxOp1ERUVRVFSE1+vF6/VSVFREVFQUTqeT7t27U1xcjDGGNWvWEBMTcza2U0REmuCMp5XS0tLYsmULlZWV3H///dx4440kJCTU2zc6Opr8/HymT59Oly5dSEpKAsDhcHDDDTfw+OOPAzBx4kTfRe67776bBQsWUF1dTVRUVIM/Ki8iIudOu/4luHN1y+6ae35Xb3vAG+83ef1tWVs9R9oWqDb1U10a1lZro1t2i4iI3xQOIiJioXAQERELhYOIiFgoHERExELhICIiFgoHERGxUDiIiIiFwkFERCwUDiIiYqFwEBERC4WDiIhYKBxERMRC4SAiIhYKBxERsVA4iIiIhcJBREQsFA4iImKhcBAREQuFg4iIWNjP1GHBggXk5+cTFBREamoqAG+99RYbNmzAbrfTt29fkpKS6NGjBwCZmZlkZ2fTqVMnpkyZQlRUFACFhYVkZGRQW1vL2LFjGT9+PAClpaWkpaXh9XoZOHAg06ZNw24/47REROQsOuORQ3x8PE888USdtsjISFJTU3nppZe44IILyMzMBGDv3r2sXbuWl19+mSeffJLFixdTW1tLbW0tixcv5oknnmDu3Ll8/fXX7N27F4C3336bcePGkZ6eTo8ePcjOzj4LmykiIk1xxnC45JJLcDgcddqGDx9OQEAAAEOHDsXj8QCQl5fHqFGj6Ny5MyEhIfTr14/t27ezfft2+vXrR9++fbHb7YwaNYq8vDyMMWzevJnY2FjgRBDl5eW19DaKiEgTNfv8TXZ2NqNGjQLA4/EwZMgQ33Mul8sXHG6329fudrvZtm0blZWVBAYG+oLm1P71ycrKIisrC4CUlBSCg4ObPF+73d7k5Q400O7P+tsyf2rTUag29VNdGtbea9OscFixYgUBAQFcffXVABhj6u1XX7vNZmvy+hITE0lMTPQ9Lisra/IYwcHBfi1Xn5Yap61oydqcb1Sb+qkuDWurtQkNDW1UP78/rZSTk8OGDRuYPn2674Xe7XZTXl7u6+PxeHC5XJb28vJynE4nPXv2pKqqipqamjr9RUSkdfkVDoWFhbz33nvMmjWLrl27+tpjYmJYu3Ytx44do7S0lJKSEgYPHsygQYMoKSmhtLSU48ePs3btWmJiYrDZbAwbNox169YBJwInJiamZbZMRET8dsbTSmlpaWzZsoXKykruv/9+brzxRjIzMzl+/DjPPfccAEOGDOHee+8lPDyckSNHMnPmTDp16sRdd91Fp04n8mfq1Km88MIL1NbWMmbMGMLDwwG47bbbSEtLY9myZQwcOJCEhISzuLkiItIYNtPQhYJ2YN++fU1exp/zgDX3/K7e9oA33m/y+tuytnqOtC1QbeqnujSsrdbmrF9zEBGR85fCQURELBQOIiJioXAQERELhYOIiFgoHERExELhICIiFgoHERGxUDiIiIiFwkFERCwUDiIiYqFwEBERC4WDiIhYKBxERMRC4SAiIhYKBxERsVA4iIiIhcJBREQsFA4iImKhcBAREQuFg4iIWNjP1GHBggXk5+cTFBREamoqAF6vl7lz53Lw4EH69OnDjBkzcDgcGGPIyMigoKCArl27kpSUREREBAA5OTmsWLECgAkTJhAfHw/Azp07mT9/PtXV1URHRzNlyhRsNttZ2lwREWmMMx45xMfH88QTT9RpW7lyJZdddhnp6elcdtllrFy5EoCCggL2799Peno69957L4sWLQJOhMny5cuZM2cOc+bMYfny5Xi9XgDeeOMN7rvvPtLT09m/fz+FhYUtvY0iItJEZwyHSy65BIfDUactLy+PuLg4AOLi4sjLywNg/fr1jB49GpvNxtChQzl69CgVFRUUFhYSGRmJw+HA4XAQGRlJYWEhFRUV/PjjjwwdOhSbzcbo0aN9Y4mISOs542ml+hw+fBin0wmA0+nkyJEjAHg8HoKDg3393G43Ho8Hj8eD2+32tbtcrnrbT/ZvSFZWFllZWQCkpKTUWVdj2e32Ji93oIF2f9bflvlTm45Ctamf6tKw9l4bv8KhIcYYS1tD1w9sNlu9/U8nMTGRxMRE3+OysrKmTZATL+j+LFeflhqnrWjJ2pxvVJv6qS4Na6u1CQ0NbVQ/vz6tFBQUREVFBQAVFRX06tULOPHO/9RilJeX43Q6cblclJeX+9o9Hg9OpxO3212nvby8HJfL5c+URESkBfkVDjExMeTm5gKQm5vLiBEjfO1r1qzBGENxcTGBgYE4nU6ioqIoKirC6/Xi9XopKioiKioKp9NJ9+7dKS4uxhjDmjVriImJabmtExERv5zxtFJaWhpbtmyhsrKS+++/nxtvvJHx48czd+5csrOzCQ4OZubMmQBER0eTn5/P9OnT6dKlC0lJSQA4HA5uuOEGHn/8cQAmTpzou8h99913s2DBAqqrq4mKiiI6OvpsbauIiDSSzTT1xH8bsm/fviYv4895wJp7fldve8Ab7zd5/W1ZWz1H2haoNvVTXRrWVmtzVq85iIjI+U3hICIiFgoHERGxUDiIiIiFwkFERCwUDiIiYqFwEBERC4WDiIhYKBxERMRC4SAiIhYKBxERsVA4iIiIhcJBREQsFA4iImKhcBAREQuFg4iIWCgcRETEQuEgIiIWCgcREbFQOIiIiIW9OQt/+OGHZGdnY7PZCA8PJykpiUOHDpGWlobX62XgwIFMmzYNu93OsWPHmDdvHjt37qRnz54kJycTEhICQGZmJtnZ2XTq1IkpU6YQFRXVIhsnIiL+8fvIwePxsGrVKlJSUkhNTaW2tpa1a9fy9ttvM27cONLT0+nRowfZ2dkAZGdn06NHD1555RXGjRvHO++8A8DevXtZu3YtL7/8Mk8++SSLFy+mtra2ZbZORET80qzTSrW1tVRXV1NTU0N1dTW9e/dm8+bNxMbGAhAfH09eXh4A69evJz4+HoDY2Fg2bdqEMYa8vDxGjRpF586dCQkJoV+/fmzfvr15WyUiIs3i92kll8vF9ddfzwMPPECXLl0YPnw4ERERBAYGEhAQ4Ovj8XiAE0cabrcbgICAAAIDA6msrMTj8TBkyJA6455c5peysrLIysoCICUlheDg4CbP2263N3m5Aw20+7P+tsyf2nQUqk39VJeGtffa+B0OXq+XvLw85s+fT2BgIC+//DKFhYUN9jfGWNpsNlu97Q1JTEwkMTHR97isrKxpk+bEC7o/y9WnpcZpK1qyNucb1aZ+qkvD2mptQkNDG9XP79NKGzduJCQkhF69emG327nyyivZunUrVVVV1NTUACeOFlwuFwBut5vy8nIAampqqKqqwuFw1Gn/5TIiItI6/D5yCA4OZtu2bfz888906dKFjRs3MmjQIIYNG8a6deu46qqryMnJISYmBoDLL7+cnJwchg4dyrp16xg2bBg2m42YmBjS09O57rrrqKiooKSkhMGDB7fYBjZFzT2/a5X1ioi0NX6Hw5AhQ4iNjWXWrFkEBAQwYMAAEhMT+Zd/+RfS0tJYtmwZAwcOJCEhAYCEhATmzZvHtGnTcDgcJCcnAxAeHs7IkSOZOXMmnTp14q677qJTp/bx9YuGwiTgjffP8UxERFqWzTTlpH8bs2/fviYvc7rzgC115NBew6GtniNtC1Sb+qkuDWurtTnr1xxEROT8pXAQERELhYOIiFgoHERExELhICIiFgoHERGxUDiIiIiFwkFERCwUDiIiYqFwEBERC4WDiIhYKBxERMRC4SAiIhYKBxERsVA4iIiIhcJBREQsFA4iImKhcBAREQuFg4iIWCgcRETEQuEgIiIW9uYsfPToURYuXMiePXuw2Ww88MADhIaGMnfuXA4ePEifPn2YMWMGDocDYwwZGRkUFBTQtWtXkpKSiIiIACAnJ4cVK1YAMGHCBOLj45u9YSIi4r9mhUNGRgZRUVE88sgjHD9+nJ9//pnMzEwuu+wyxo8fz8qVK1m5ciWTJk2ioKCA/fv3k56ezrZt21i0aBFz5szB6/WyfPlyUlJSAJg9ezYxMTE4HI4W2UAREWk6v08rVVVV8e2335KQkACA3W6nR48e5OXlERcXB0BcXBx5eXkArF+/ntGjR2Oz2Rg6dChHjx6loqKCwsJCIiMjcTgcOBwOIiMjKSwsbIFNExERf/l95FBaWkqvXr1YsGABu3fvJiIigsmTJ3P48GGcTicATqeTI0eOAODxeAgODvYt73a78Xg8eDwe3G63r93lcuHxeOpdZ1ZWFllZWQCkpKTUGa+x7HZ7g8sdaPJo9fNnXm3B6WrT0ak29VNdGtbea+N3ONTU1LBr1y6mTp3KkCFDyMjIYOXKlQ32N8ZY2mw2W719G2pPTEwkMTHR97isrKyJsz7xwu3Pck1xtsc/W85Fbdor1aZ+qkvD2mptQkNDG9XP79NKbrcbt9vNkCFDAIiNjWXXrl0EBQVRUVEBQEVFBb169fL1P7VQ5eXlOJ1OXC4X5eXlvnaPx+M78hARkdbhdzj07t0bt9vNvn37ANi4cSMXXnghMTEx5ObmApCbm8uIESMAiImJYc2aNRhjKC4uJjAwEKfTSVRUFEVFRXi9XrxeL0VFRURFRbXApomIiL+a9WmlqVOnkp6ezvHjxwkJCSEpKQljDHPnziU7O5vg4GBmzpwJQHR0NPn5+UyfPp0uXbqQlJQEgMPh4IYbbuDxxx8HYOLEifqkkohIK7OZ+i4GtBMnj1qa4nTnAWvu+V1zpwRAwBvvt8g451pbPUfaFqg29VNdGtZWa3PWrzmIiMj5S+EgIiIWCgcREbFQOIiIiIXCQURELBQOIiJioXAQERELhYOIiFgoHERExELhICIiFgoHERGxUDiIiIiFwkFERCyadctuqV9Dd3dtr3drFZGOR0cOIiJioXAQERELhYOIiFgoHERExELhICIiFgoHERGxaPZHWWtra5k9ezYul4vZs2dTWlpKWloaXq+XgQMHMm3aNOx2O8eOHWPevHns3LmTnj17kpycTEhICACZmZlkZ2fTqVMnpkyZQlRUVLM3TERE/NfsI4ePP/6YsLAw3+O3336bcePGkZ6eTo8ePcjOzgYgOzubHj168MorrzBu3DjeeecdAPbu3cvatWt5+eWXefLJJ1m8eDG1tbXNnZaIiDRDs8KhvLyc/Px8xo4dC4Axhs2bNxMbGwtAfHw8eXl5AKxfv574+HgAYmNj2bRpE8YY8vLyGDVqFJ07dyYkJIR+/fqxffuQKtWmAAAM2ElEQVT25kxLRESaqVnhsGTJEiZNmoTNZgOgsrKSwMBAAgICAHC5XHg8HgA8Hg9utxuAgIAAAgMDqaysrNP+y2VERKR1+H3NYcOGDQQFBREREcHmzZvP2N8YY2mz2Wz1tjckKyuLrKwsAFJSUggODm78hP8/u93e4HIHmjxa0/gz33PpdLXp6FSb+qkuDWvvtfE7HLZu3cr69espKCigurqaH3/8kSVLllBVVUVNTQ0BAQF4PB5cLhcAbreb8vJy3G43NTU1VFVV4XA4fO0nnbrMLyUmJpKYmOh7XFZW1uR5BwcH+7VcS2it9TZWa9amrVNt6qe6NKyt1iY0NLRR/fw+rXTrrbeycOFC5s+fT3JyMpdeeinTp09n2LBhrFu3DoCcnBxiYmIAuPzyy8nJyQFg3bp1DBs2DJvNRkxMDGvXruXYsWOUlpZSUlLC4MGD/Z2WiIi0gBa/K+ttt91GWloay5YtY+DAgSQkJACQkJDAvHnzmDZtGg6Hg+TkZADCw8MZOXIkM2fOpFOnTtx111106qSvX4iItCabacpJ/zZm3759TV7mdId6Dd1qu6W09Vt2t9XD4LZAtamf6tKwtlqbs35aSUREzl8KBxERsVA4iIiIhcJBREQs9BvS55B+W1pE2gsdOYiIiIXCQURELBQOIiJioXAQERELhYOIiFgoHERExELhICIiFgoHERGxUDiIiIiFwkFERCwUDiIiYqFwEBERC4WDiIhYKBxERMSiQ96y+2z/VnRTnW4+up23iLQGHTmIiIiF30cOZWVlzJ8/n0OHDmGz2UhMTOS3v/0tXq+XuXPncvDgQfr06cOMGTNwOBwYY8jIyKCgoICuXbuSlJREREQEADk5OaxYsQKACRMmEB8f3yIbJyIi/vE7HAICArj99tuJiIjgxx9/ZPbs2URGRpKTk8Nll13G+PHjWblyJStXrmTSpEkUFBSwf/9+0tPT2bZtG4sWLWLOnDl4vV6WL19OSkoKALNnzyYmJgaHw9FiGykiIk3j92klp9Ppe+ffvXt3wsLC8Hg85OXlERcXB0BcXBx5eXkArF+/ntGjR2Oz2Rg6dChHjx6loqKCwsJCIiMjcTgcOBwOIiMjKSwsbIFNExERf7XIBenS0lJ27drF4MGDOXz4ME6nEzgRIEeOHAHA4/EQHBzsW8btduPxePB4PLjdbl+7y+XC4/HUu56srCyysrIASElJqTNeY9nt7esavD/b6C+73X5O19eeqDb1U10a1t5r0+xXyp9++onU1FQmT55MYGBgg/2MMZY2m81Wb9+G2hMTE0lMTPQ9Lisra+Jsz+2LbUvwZxv9FRwcfE7X156oNvVTXRrWVmsTGhraqH7N+rTS8ePHSU1N5eqrr+bKK68EICgoiIqKCgAqKiro1asXcOJI4dRClZeX43Q6cblclJeX+9o9Ho/vyENERFqH3+FgjGHhwoWEhYVx3XXX+dpjYmLIzc0FIDc3lxEjRvja16xZgzGG4uJiAgMDcTqdREVFUVRUhNfrxev1UlRURFRUVDM3S0REmsPv00pbt25lzZo1XHTRRTz22GMA3HLLLYwfP565c+eSnZ1NcHAwM2fOBCA6Opr8/HymT59Oly5dSEpKAsDhcHDDDTfw+OOPAzBx4kR9UukUDX1BTl+OE5GzyWbquxjQTuzbt6/JywQHB3PgD6POwmzOrbMRDm31HGlboNrUT3VpWFutzTm55iAiIucnhYOIiFgoHERExELhICIiFgoHERGxaF/3khAffcRVRM4mHTmIiIiFwkFERCwUDiIiYqFrDucZXYsQkZagIwcREbFQOIiIiIXCQURELHTNoYPQtQgRaQodOYiIiIWOHDq4U48oDpzSriMKkY5NRw4iImKhIwepl65RiHRsCgdpEoWGSMegcJAWodAQOb8oHOSsUmiItE9tJhwKCwvJyMigtraWsWPHMn78+NaekpxFDYVGQxQmIudWmwiH2tpaFi9ezFNPPYXb7ebxxx8nJiaGCy+8sLWnJm1EU8MEFCgizdEmwmH79u3069ePvn37AjBq1Cjy8vIUDtIs/gRKfRQy0hG1iXDweDy43W7fY7fbzbZt2yz9srKyyMrKAiAlJYXQ0FC/1hf+0Xr/Jiodmr/72/lOdWlYe65Nm/gSnDHG0maz2SxtiYmJpKSkkJKS4ve6Zs+e7fey5zvVpmGqTf1Ul4a199q0iXBwu92Ul5f7HpeXl+N0OltxRiIiHVubCIdBgwZRUlJCaWkpx48fZ+3atcTExLT2tEREOqw2cc0hICCAqVOn8sILL1BbW8uYMWMIDw8/K+tKTEw8K+OeD1Sbhqk29VNdGtbea2Mz9Z3wFxGRDq1NnFYSEZG2ReEgIiIWbeKaw7nS0W7RUVZWxvz58zl06BA2m43ExER++9vf4vV6mTt3LgcPHqRPnz7MmDEDh8OBMYaMjAwKCgro2rUrSUlJREREAJCTk8OKFSsAmDBhAvHx8a24ZS2ntraW2bNn43K5mD17NqWlpaSlpeH1ehk4cCDTpk3Dbrdz7Ngx5s2bx86dO+nZsyfJycmEhIQAkJmZSXZ2Np06dWLKlClERUW18lY139GjR1m4cCF79uzBZrPxwAMPEBoa2uH3mw8//JDs7GxsNhvh4eEkJSVx6NCh83OfMR1ETU2Neeihh8z+/fvNsWPHzKOPPmr27NnT2tM6qzwej9mxY4cxxpiqqiozffp0s2fPHvPWW2+ZzMxMY4wxmZmZ5q233jLGGLNhwwbzwgsvmNraWrN161bz+OOPG2OMqaysNA8++KCprKys8//ngw8++MCkpaWZF1980RhjTGpqqvnqq6+MMca89tpr5tNPPzXGGPPJJ5+Y1157zRhjzFdffWVefvllY4wxe/bsMY8++qiprq42Bw4cMA899JCpqalphS1pWa+88orJysoyxhhz7Ngx4/V6O/x+U15ebpKSkszPP/9sjDmxr6xevfq83Wc6zGmlU2/RYbfbfbfoOJ85nU7fO7ju3bsTFhaGx+MhLy+PuLg4AOLi4nx1WL9+PaNHj8ZmszF06FCOHj1KRUUFhYWFREZG4nA4cDgcREZGUlhY2Grb1VLKy8vJz89n7NixwIkvY27evJnY2FgA4uPj69Tm5Lve2NhYNm3ahDGGvLw8Ro0aRefOnQkJCaFfv35s3769VbanpVRVVfHtt9+SkJAAgN1up0ePHtpvOHGkWV1dTU1NDdXV1fTu3fu83Wc6zGmlxt6i43xVWlrKrl27GDx4MIcPH/Z9ydDpdHLkyBHgRI2Cg4N9y7jdbjwej6V2LpcLj8dzbjfgLFiyZAmTJk3ixx9/BKCyspLAwEACAgKAutt5ag0CAgIIDAyksrISj8fDkCFDfGOeD7UpLS2lV69eLFiwgN27dxMREcHkyZM7/H7jcrm4/vrreeCBB+jSpQvDhw8nIiLivN1nOsyRg2nkLTrORz/99BOpqalMnjyZwMDABvs1pUbtvXYbNmwgKCjId2R1Jg3Vpr729q6mpoZdu3bxr//6r/zlL3+ha9eurFy5ssH+HWW/8Xq95OXlMX/+fF577TV++umn0x4Jtfd9psOEQ0e9Rcfx48dJTU3l6quv5sorrwQgKCiIiooKACoqKujVqxdwokZlZWW+ZU/WyOVy1amdx+Np97XbunUr69ev58EHHyQtLY1NmzaxZMkSqqqqqKmpAU5sp8vlAuruPzU1NVRVVeFwOCz71anLtFdutxu32+17dxsbG8uuXbs6/H6zceNGQkJC6NWrF3a7nSuvvJKtW7eet/tMhwmHjniLDmMMCxcuJCwsjOuuu87XHhMTQ25uLgC5ubmMGDHC175mzRqMMRQXFxMYGIjT6SQqKoqioiK8Xi9er5eioqK2+emKJrj11ltZuHAh8+fPJzk5mUsvvZTp06czbNgw1q1bB5z4pM3JfeTyyy8nJycHgHXr1jFs2DBsNhsxMTGsXbuWY8eOUVpaSklJCYMHD26tzWoRvXv3xu12s2/fPuDEi+KFF17Y4feb4OBgtm3bxs8//4wxxleX83Wf6VDfkM7Pz+dvf/ub7xYdEyZMaO0pnVXfffcdf/rTn7jooot8h/O33HILQ4YMYe7cuZSVlREcHMzMmTN9H0lcvHgxRUVFdOnShaSkJAYNGgRAdnY2mZmZwImPJI4ZM6bVtqulbd68mQ8++IDZs2dz4MABy8cSO3fuTHV1NfPmzWPXrl04HA6Sk5N9vz+yYsUKVq9eTadOnZg8eTLR0dGtvEXN9/3337Nw4UKOHz9OSEgISUlJGGM6/H7z7rvvsnbtWgICAhgwYAD3338/Ho/nvNxnOlQ4iIhI43SY00oiItJ4CgcREbFQOIiIiIXCQURELBQOIiJioXAQERELhYOIiFj8P5BeT6+xNSCwAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot the distributions of len\n",
    "\n",
    "plt.hist(reviews.length,bins = 50)\n",
    "plt.title(\"Distribution of review lengths\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    50000.000000\n",
       "mean       748.574380\n",
       "std        586.307924\n",
       "min         17.000000\n",
       "25%        385.000000\n",
       "50%        548.000000\n",
       "75%        914.250000\n",
       "max       8625.000000\n",
       "Name: length, dtype: float64"
      ]
     },
     "execution_count": 193,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# some information about the lengths\n",
    "\n",
    "reviews.length.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observation** - looks like we have a very long review i.e having length = 8677. Let's see that review"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 195,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'match 1 tag team table match bubba ray spike dudley vs eddie guerrero chris benoit bubba ray spike dudley started things tag team table match eddie guerrero chris benoit according rules match opponents tables order win benoit guerrero heated early taking turns hammering spike bubba ray german suplex benoit bubba took wind dudley brother spike tried help brother referee restrained benoit guerrero ganged corner benoit stomping away bubba guerrero set table outside spike dashed ring somersaulted rope guerrero outside recovering taking care spike guerrero slipped table ring helped wolverine set tandem set double superplex middle rope bubba table spike knocked table right brother came crashing guerrero benoit propped table corner tried irish whip spike bubba dashed blocked brother bubba caught lifted opponents body drops bubba slammed guerrero spike stomped wolverine rope bubba held benoit bay spike soar wassup headbutt shortly benoit latched spike crossface match continued spike tapped bubba came brothers rescue managed sprawl benoit table bubba leapt middle rope benoit moved sent bubba crashing wood opponents didnt force table bubba allowed stay match man eliminated shortly spike eddie table dudley dawg ring apron outside benoit spike table moments later score seconds bubba nailed bubba bomb benoit table gave dudleys win winner bubba ray spike dudleymatch 2 cruiserweight championship jamie noble vs billy kidman billy kidman challenged jamie noble brought nidia ring cruiserweight championship noble kidman locked tumbled ring raced inside grappled kidman thwarted nobles moves noble fled outside ring nidia gave encouragement fight spread outside ring noble threw girlfriend challenger kidman tossed nidia aside taken modified arm bar noble continued attack kidmans injured arm ring kidmans injured harm hampered offense continued battle hard noble tried kidman away powerbomb challenger countered facebuster kidman went finish things shooting star press noble broke attempt kidman went shooting star press time noble rolled harms way noble flipped kidman power bomb soon got pin retain wwe cruiserweight championship winner jamie noblematch 3 european championship william regal vs jeff hardy william regal took jeff hardy attempt win european championship jeff catapulted regal rope took hurracanrana ring apron ring jeff hit whisper wind knock regal loop jeff went swanton bomb regal got knees hit jeff devastating shot jeff managed surprise regal quick rollup got pin european championship regal started bawling seeing hardy celebrate way ramp winner jeff hardymatch 4 chris jericho vs john cena chris jericho promised end john cenas career match vengeance came jericho tried teach cena lesson match began suplexing mat jericho continued knock cena ring cockiness got better rope jericho began showboat allowed cena grab superplex cena followed tiltawhirl slam taken nasty dropkick gut rookie recovered hit belly belly suplex y2j away jericho launched lionsault cena dodged jericho nailed bulldog connected lionsault cover goaded cena feet walls jericho cena ideas reversing pin attempt getting 123 jericho went berserk match winner john cenamatch 5 intercontinental championship rvd vs brock lesnar disqualification big thing mr payperview tangled intercontinental championship line brock grabbed title ref draped shoulder momentarily glaring rvd van dam s quickness gave brock fits early big man rolled ring kicked steel steps frustration brock pulled began charge paul heyman beaming ringside brock slammed rvd hard floor outside ring brock began overpower rvd throwing ease rope rvd landed painfully suffer having spine cracked steel ring steps fight returned ring brock squeezing rvd ribs rvd broke away soon leveled brock kick temple rvd followed rolling thunder brock managed kick twocount fight looked like soon rvd went fivestar frog splash brock hoisted van dam shoulder went f5 rvd whirled brock ddt followed frog splash went pin heyman pulled ref ring ref immediately called disqualification soon traded blows heyman rvd leapt brock rope threatened hit van terminator heyman grabbed rvds leg brock picked champ time connected f5 steel chair winner rvdmatch 6 booker t vs big booker t faced big oneonone withstood booker ts kicks punches slapped booker corner thrown ring booker picked chair ringside big punched bookers face booker tried game choking camera cable ringside booker smashed tv monitor spanish announcers position shows skull delivered scissors kick men table booker crawled ring big staggered moments later grabbed bookers throat met low blow kick face booker climbed rope nailed somersaulting leg drop pin winner booker tannouncement triple h entered ring thunderous ovation fans hoped learn game end competing speak eric bishoff stopped game apologize getting involved personal business triple h signed raw bischoff promised personal life come play bischoff said hes spent past years networking hollywood said looking breakout wwe superstar talking triple h bischoff guaranteed triple h signed raw hed getting opportunities coming way stephanie mcmahon stepped issue pitch said personal history triple h know said unstoppable bischoff cut begged stop stephanie cited triple h told bischoff said triple h talent charisma bischoff said young time didnt know lot experience stephanie continued bicker forth triple h stepped microphone game said easy screw triple h went shake bischoffs hand pulled away said devil knows doesnt know shawn michaels came shake things hbk said thing wanted cause trouble didnt want involved remembered pledging bring triple h nwo hbk said theres world triple h better friends hbk told friend imagine making bischoffs life living hell triple h said tempting offer turned hugged hbk making official switch raw triple h hbk left bischoff gloated victory bischoff said difference hes got testicles doesnt stephanie whacked bischoff head leftmatch 7 tag team championship match christian lance storm vs hollywood hogan edge match started loud usa chants hogan shoving christian ropes ring canadians took edge scored kick christians head planted facebuster storm tag hogan hogan began hulk soon caught christian big boot leg drop storm broke count christian tossed hogan ring storm superkicked icon edge tagged soon dropped opponents speared corner turnbuckles missed spear strom hit ref hard instead edge nailed ddt ref count test raced took hogan leveled edge boot storm tried pin edge kicked riksihi sprinted fend test allowing edge recover spear storm christian distracted ref y2j dashed clocked edge tag team championship storm rolled got pinfall win title winners new tag team champions christian lance stormmatch 8 wwe undisputed championship triple threat match rock vs kurt angle undertaker wwes successful superstars lined triple threat match undisputed championship hanging balance taker rock got face face kurt angle begging attention got attention form beat form men soon taker spilled ring rock brawled angle angle gave series suplexes took rock great countered ddt managed twocount fight continued outside ring taker coming life clotheslining angle repeatedly smacking rock taker rock got ring taker dropped rock sidewalk slam twocount rock rebounded grabbed taker throat chokeslammed angle broke pin attempt likely given rock title rock retaliated latching ankle lock kurt angle angle reversed rock bottomed peoples champion soon rock disposed angle hit peoples elbow undertaker angle tried advantage disabling great outside ring covering taker kicked count outside ring rock took big swig nearby water bottle spewed liquid takers face blind champion taker didnt stay disabled long managed overpower rock turn attention angle taker landed guillotine leg drop angle laying ring apron rock picked time break pin attempt kurt angle taker nailed rock ddt set chokeslam angle tried sneaking steel chair taker caught tomfoolery smacked hands referee got caught ensuing didnt angle knock taker silly steel chair angle went cover taker rock lay prone dead man got shoulder angle tried pin rock kicked rock got landed angle sharpshooter angle looked like tap taker kicked rock submission hold taker picked rock crashed ride dead man covered win angle raced picked taker ankle lock taker went delirious pain managed counter picked angle ride angle triangle choke looked like taker pass rock broke angles hold caught ankle lock rock got hold watched taker chokeslam angle rocky hit rock taker refused kicked angle whirled taker angle slam rock bottomed great pinned winner new wwe champion rockfinally decent ppv lately ppv werent good winner ppv'"
      ]
     },
     "execution_count": 195,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews[reviews['length'] == 8625]['text'].iloc[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Modeling\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 196,
   "metadata": {},
   "outputs": [],
   "source": [
    "# now let's convert the text into numerical representation.\n",
    "\n",
    "from sklearn.feature_extraction.text import CountVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 197,
   "metadata": {},
   "outputs": [],
   "source": [
    "bow_transformer = CountVectorizer().fit(reviews.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 199,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "221238"
      ]
     },
     "execution_count": 199,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(bow_transformer.vocabulary_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 200,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'typical mel brooks film slapstick movies actually plot followable leslie ann warren movie fantastic underrated actress moments fleshed bit scenes probably cut room worth price rent acting good overall brooks good job characteristic speaking directly audience warren best actor movie fume sailor played parts'"
      ]
     },
     "execution_count": 200,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_message = reviews.text.iloc[4]\n",
    "sample_message"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 202,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_transformed = bow_transformer.transform([sample_message])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 205,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  (0, 5822)\t1\n",
      "  (0, 6217)\t1\n",
      "  (0, 6425)\t1\n",
      "  (0, 6531)\t1\n",
      "  (0, 12048)\t1\n",
      "  (0, 16465)\t1\n",
      "  (0, 22506)\t1\n",
      "  (0, 23844)\t1\n",
      "  (0, 28692)\t2\n",
      "  (0, 35264)\t1\n",
      "  (0, 47685)\t1\n",
      "  (0, 54211)\t1\n",
      "  (0, 69666)\t1\n",
      "  (0, 72027)\t1\n",
      "  (0, 74301)\t1\n",
      "  (0, 75097)\t1\n",
      "  (0, 78030)\t1\n",
      "  (0, 82868)\t2\n",
      "  (0, 104470)\t1\n",
      "  (0, 112468)\t1\n",
      "  (0, 122865)\t1\n",
      "  (0, 127029)\t1\n",
      "  (0, 129017)\t2\n",
      "  (0, 129575)\t1\n",
      "  (0, 142232)\t1\n",
      "  (0, 144714)\t1\n",
      "  (0, 149395)\t1\n",
      "  (0, 149753)\t1\n",
      "  (0, 153469)\t1\n",
      "  (0, 153836)\t1\n",
      "  (0, 161589)\t1\n",
      "  (0, 165519)\t1\n",
      "  (0, 167374)\t1\n",
      "  (0, 169332)\t1\n",
      "  (0, 178569)\t1\n",
      "  (0, 182238)\t1\n",
      "  (0, 203368)\t1\n",
      "  (0, 204653)\t1\n",
      "  (0, 211461)\t2\n",
      "  (0, 217748)\t1\n",
      "shape :  (1, 221238)\n"
     ]
    }
   ],
   "source": [
    "# a sparse matrix\n",
    "\n",
    "print(sample_transformed)\n",
    "\n",
    "print(\"shape : \",sample_transformed.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 207,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of entire transformed data :  (50000, 221238)\n",
      "Non-Zero elements :  4365356\n"
     ]
    }
   ],
   "source": [
    "# now let's transform the entire reviews\n",
    "\n",
    "reviews_transformed_bow = bow_transformer.transform(reviews.text)\n",
    "\n",
    "print(\"Shape of entire transformed data : \",reviews_transformed_bow.shape)\n",
    "\n",
    "# since it will be a sparse matrix we can count no of non-zero elements\n",
    "\n",
    "print(\"Non-Zero elements : \",reviews_transformed_bow.nnz)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 208,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now we have transformed the text into numerical form.\n",
    "# But we can also assign a weights to each word by using Tf-Idf \n",
    "\n",
    "from sklearn.feature_extraction.text import TfidfTransformer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 210,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf_idf_transformer = TfidfTransformer().fit(reviews_transformed_bow)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 218,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  (0, 217748)\t0.09540094558207422\n",
      "  (0, 211461)\t0.37656471727729884\n",
      "  (0, 204653)\t0.15556757911139552\n",
      "  (0, 203368)\t0.12590501523469033\n",
      "  (0, 182238)\t0.14676835873213936\n",
      "  (0, 178569)\t0.16099404381002486\n",
      "  (0, 169332)\t0.07874105947387403\n",
      "  (0, 167374)\t0.19930806413934105\n",
      "  (0, 165519)\t0.12468049502771246\n",
      "  (0, 161589)\t0.12792560668677466\n",
      "  (0, 153836)\t0.0917520041958287\n",
      "  (0, 153469)\t0.1553923035722736\n",
      "  (0, 149753)\t0.07274147234514643\n",
      "  (0, 149395)\t0.09571654478972869\n",
      "  (0, 144714)\t0.11453226369558171\n",
      "  (0, 142232)\t0.11472993830944923\n",
      "  (0, 129575)\t0.06918975589696753\n",
      "  (0, 129017)\t0.0829982523776984\n",
      "  (0, 127029)\t0.10674294639955594\n",
      "  (0, 122865)\t0.17741901690859563\n",
      "  (0, 112468)\t0.1748164292574346\n",
      "  (0, 104470)\t0.09919484093384666\n",
      "  (0, 82868)\t0.10945238499662319\n",
      "  (0, 78030)\t0.27084597030567475\n",
      "  (0, 75097)\t0.2750744049283481\n",
      "  (0, 74301)\t0.20402200751680974\n",
      "  (0, 72027)\t0.04429582437975536\n",
      "  (0, 69666)\t0.12758537871527953\n",
      "  (0, 54211)\t0.16208703814168568\n",
      "  (0, 47685)\t0.12212317390926004\n",
      "  (0, 35264)\t0.20507043558596\n",
      "  (0, 28692)\t0.3508281357964893\n",
      "  (0, 23844)\t0.09089006432773569\n",
      "  (0, 22506)\t0.07307257169777742\n",
      "  (0, 16465)\t0.10154478124798691\n",
      "  (0, 12048)\t0.17055423397998706\n",
      "  (0, 6531)\t0.08221426825943193\n",
      "  (0, 6425)\t0.11684631121743641\n",
      "  (0, 6217)\t0.09964136788154215\n",
      "  (0, 5822)\t0.07052143391947997\n"
     ]
    }
   ],
   "source": [
    "# let's see the tf-idf transformed version of the sample we have used earlier\n",
    "sample_transformed_tfidf = tf_idf_transformer.transform(sample_transformed)\n",
    "print(sample_transformed_tfidf) # each word has now a probablity assigned to it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 214,
   "metadata": {},
   "outputs": [],
   "source": [
    "# transform all the reviews\n",
    "\n",
    "reviews_transformed_tfidf = tf_idf_transformer.transform(reviews_transformed_bow)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 215,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(50000, 221238)"
      ]
     },
     "execution_count": 215,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews_transformed_tfidf.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's apply `Naive bayes classifier`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 216,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**In case of Naive bayes**\n",
    "\n",
    "There is only one `hyper parameter` to tune i.e $\\alpha $ Which represents `Additive (Laplace/Lidstone) smoothing parameter.`\n",
    "\n",
    "* $\\alpha=0$ i.e no smoothing will lead to high variance i.e under-fitting\n",
    "* $\\alpha=large-value$ will lead to high bias i.e under fitting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 217,
   "metadata": {},
   "outputs": [],
   "source": [
    "review_classifier = MultinomialNB().fit(reviews_transformed_tfidf,reviews.label)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 219,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>label</th>\n",
       "      <th>length</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>typical mel brooks film slapstick movies actua...</td>\n",
       "      <td>pos</td>\n",
       "      <td>307</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text label  length\n",
       "4  typical mel brooks film slapstick movies actua...   pos     307"
      ]
     },
     "execution_count": 219,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's check the classifier\n",
    "\n",
    "# data \n",
    "\n",
    "reviews[reviews.text == sample_message]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 220,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prediction from model\n",
    "\n",
    "predicted_sample = review_classifier.predict(sample_transformed_tfidf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 221,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['pos'], dtype='<U3')"
      ]
     },
     "execution_count": 221,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predicted_sample # same as true label"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have a working model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 222,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's predict all data points\n",
    "\n",
    "predicted_all = review_classifier.predict(reviews_transformed_tfidf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 223,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's look at some classification metrics\n",
    "\n",
    "from sklearn.metrics import classification_report,confusion_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 233,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_classification_report(true_labels,predicted_labels):\n",
    "    \"\"\"\n",
    "    to print classification report and confusion matrix\n",
    "    \"\"\"\n",
    "    # print classification report\n",
    "    print(\"Classification report \")\n",
    "    print(classification_report(true_labels,predicted_labels))\n",
    "\n",
    "    print(\"*\"*100)\n",
    "\n",
    "    print(\"Confusion metrics \")\n",
    "    print(pd.DataFrame(data = confusion_matrix(true_labels,predicted_labels)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 234,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification report \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         neg       0.91      0.92      0.92     25000\n",
      "         pos       0.92      0.91      0.92     25000\n",
      "\n",
      "    accuracy                           0.92     50000\n",
      "   macro avg       0.92      0.92      0.92     50000\n",
      "weighted avg       0.92      0.92      0.92     50000\n",
      "\n",
      "****************************************************************************************************\n",
      "Confusion metrics \n",
      "       0      1\n",
      "0  23114   1886\n",
      "1   2223  22777\n"
     ]
    }
   ],
   "source": [
    "print_classification_report(reviews.label,predicted_all)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Looks like we have a nice classifier but wait we did a `blunder`. We applied classification report on training data as we did not have test data**\n",
    "\n",
    "**Hence we have no idea about the true predictive power of our classifier**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Let's apply the model in right way** \n",
    "\n",
    "We will also use pipelines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 235,
   "metadata": {},
   "outputs": [],
   "source": [
    "# perform test and train split\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 236,
   "metadata": {},
   "outputs": [],
   "source": [
    "review_train,review_test,label_train,label_test = train_test_split(reviews.text.values,reviews.label.values,\n",
    "                                                                    test_size = 0.2, stratify = reviews.label.values,\n",
    "                                                                  random_state = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 239,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(40000,) (10000,)\n"
     ]
    }
   ],
   "source": [
    "print(review_train.shape,review_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 274,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pipeline to perform the steps in one go\n",
    "review_pipeline = Pipeline(steps = [\n",
    "    ('bow_transform',CountVectorizer()),\n",
    "    ('tfidf_transform',TfidfTransformer()),\n",
    "    ('classifier',MultinomialNB())\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 275,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "         steps=[('bow_transform',\n",
       "                 CountVectorizer(analyzer='word', binary=False,\n",
       "                                 decode_error='strict',\n",
       "                                 dtype=<class 'numpy.int64'>, encoding='utf-8',\n",
       "                                 input='content', lowercase=True, max_df=1.0,\n",
       "                                 max_features=None, min_df=1,\n",
       "                                 ngram_range=(1, 1), preprocessor=None,\n",
       "                                 stop_words=None, strip_accents=None,\n",
       "                                 token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "                                 tokenizer=None, vocabulary=None)),\n",
       "                ('tfidf_transform',\n",
       "                 TfidfTransformer(norm='l2', smooth_idf=True,\n",
       "                                  sublinear_tf=False, use_idf=True)),\n",
       "                ('classifier',\n",
       "                 MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True))],\n",
       "         verbose=False)"
      ]
     },
     "execution_count": 275,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "review_pipeline.fit(review_train,label_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 276,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions_test = review_pipeline.predict(review_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 277,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification report \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         neg       0.86      0.87      0.86      5000\n",
      "         pos       0.87      0.85      0.86      5000\n",
      "\n",
      "    accuracy                           0.86     10000\n",
      "   macro avg       0.86      0.86      0.86     10000\n",
      "weighted avg       0.86      0.86      0.86     10000\n",
      "\n",
      "****************************************************************************************************\n",
      "Confusion metrics \n",
      "      0     1\n",
      "0  4358   642\n",
      "1   730  4270\n"
     ]
    }
   ],
   "source": [
    "# print the classification report\n",
    "\n",
    "print_classification_report(label_test,predictions_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 263,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's also see what is the accuracy \n",
    "\n",
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 264,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8628"
      ]
     },
     "execution_count": 264,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy_score(label_test,predictions_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Looks like we have a pretty good model**\n",
    "\n",
    "**But What about cross validation and hyper-parameter search ?**\n",
    "\n",
    "\n",
    "- we can use `sklearn.model_selection.GridSearchCV` or `sklearn.model_selection.RandomizedSearchCV` to find best hyper-parameter. But how does `sklearn` do that -- By using `Cross validation`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 265,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV,RandomizedSearchCV"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In case of `GridSearchCV` we have to provide a list of possible hyper-parameters as a dictionary in case we have only one hyper-parameter to tune or a list of dictionaries. On the other hand in `RandomizedSearchCV` we have to provide a distribution from which the value of hyper-parameters will be sampled randomly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 292,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import f1_score,make_scorer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 302,
   "metadata": {},
   "outputs": [],
   "source": [
    "# modeling using GridSearchCV\n",
    "\n",
    "f1_scorer = make_scorer(f1_score, pos_label=\"pos\")\n",
    "\n",
    "parameter_to_tune = {\"classifier__alpha\":[0.5,0.6,0.7,0.8]} # pipeline name of the our algo or estimator\n",
    "\n",
    "model_grid_search = GridSearchCV(review_pipeline,\n",
    "                                 param_grid = parameter_to_tune,\n",
    "                                 scoring = f1_scorer,cv = 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 303,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=3, error_score='raise-deprecating',\n",
       "             estimator=Pipeline(memory=None,\n",
       "                                steps=[('bow_transform',\n",
       "                                        CountVectorizer(analyzer='word',\n",
       "                                                        binary=False,\n",
       "                                                        decode_error='strict',\n",
       "                                                        dtype=<class 'numpy.int64'>,\n",
       "                                                        encoding='utf-8',\n",
       "                                                        input='content',\n",
       "                                                        lowercase=True,\n",
       "                                                        max_df=1.0,\n",
       "                                                        max_features=None,\n",
       "                                                        min_df=1,\n",
       "                                                        ngram_range=(1, 1),\n",
       "                                                        preprocessor=None,\n",
       "                                                        stop_words=None,\n",
       "                                                        strip_accents...\n",
       "                                        TfidfTransformer(norm='l2',\n",
       "                                                         smooth_idf=True,\n",
       "                                                         sublinear_tf=False,\n",
       "                                                         use_idf=True)),\n",
       "                                       ('classifier',\n",
       "                                        MultinomialNB(alpha=1.0,\n",
       "                                                      class_prior=None,\n",
       "                                                      fit_prior=True))],\n",
       "                                verbose=False),\n",
       "             iid='warn', n_jobs=None,\n",
       "             param_grid={'classifier__alpha': [0.5, 0.6, 0.7, 0.8]},\n",
       "             pre_dispatch='2*n_jobs', refit=True, return_train_score=False,\n",
       "             scoring=make_scorer(f1_score, pos_label=pos), verbose=0)"
      ]
     },
     "execution_count": 303,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_grid_search.fit(review_train,label_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 305,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "         steps=[('bow_transform',\n",
       "                 CountVectorizer(analyzer='word', binary=False,\n",
       "                                 decode_error='strict',\n",
       "                                 dtype=<class 'numpy.int64'>, encoding='utf-8',\n",
       "                                 input='content', lowercase=True, max_df=1.0,\n",
       "                                 max_features=None, min_df=1,\n",
       "                                 ngram_range=(1, 1), preprocessor=None,\n",
       "                                 stop_words=None, strip_accents=None,\n",
       "                                 token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "                                 tokenizer=None, vocabulary=None)),\n",
       "                ('tfidf_transform',\n",
       "                 TfidfTransformer(norm='l2', smooth_idf=True,\n",
       "                                  sublinear_tf=False, use_idf=True)),\n",
       "                ('classifier',\n",
       "                 MultinomialNB(alpha=0.7, class_prior=None, fit_prior=True))],\n",
       "         verbose=False)"
      ]
     },
     "execution_count": 305,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_grid_search.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 306,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8614950065570464"
      ]
     },
     "execution_count": 306,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's check model score on test data\n",
    "\n",
    "model_grid_search.score(review_test,label_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 307,
   "metadata": {},
   "outputs": [],
   "source": [
    "predicted_test_gridsearch = model_grid_search.predict(review_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 308,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification report \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         neg       0.86      0.87      0.86      5000\n",
      "         pos       0.87      0.85      0.86      5000\n",
      "\n",
      "    accuracy                           0.86     10000\n",
      "   macro avg       0.86      0.86      0.86     10000\n",
      "weighted avg       0.86      0.86      0.86     10000\n",
      "\n",
      "****************************************************************************************************\n",
      "Confusion metrics \n",
      "      0     1\n",
      "0  4357   643\n",
      "1   730  4270\n"
     ]
    }
   ],
   "source": [
    "print_classification_report(label_test,predicted_test_gridsearch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 323,
   "metadata": {},
   "outputs": [],
   "source": [
    "# modeling using randomizedSearchCV\n",
    "\n",
    "from scipy.stats import uniform\n",
    "\n",
    "\n",
    "distributions = dict(classifier__alpha=uniform(loc=0, scale=4))\n",
    "\n",
    "\n",
    "model_randSearch = RandomizedSearchCV(review_pipeline,distributions,\n",
    "                                 scoring = f1_scorer,cv = 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 324,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomizedSearchCV(cv=3, error_score='raise-deprecating',\n",
       "                   estimator=Pipeline(memory=None,\n",
       "                                      steps=[('bow_transform',\n",
       "                                              CountVectorizer(analyzer='word',\n",
       "                                                              binary=False,\n",
       "                                                              decode_error='strict',\n",
       "                                                              dtype=<class 'numpy.int64'>,\n",
       "                                                              encoding='utf-8',\n",
       "                                                              input='content',\n",
       "                                                              lowercase=True,\n",
       "                                                              max_df=1.0,\n",
       "                                                              max_features=None,\n",
       "                                                              min_df=1,\n",
       "                                                              ngram_range=(1,\n",
       "                                                                           1),\n",
       "                                                              preprocessor=None,\n",
       "                                                              stop_words=None,\n",
       "                                                              strip_a...\n",
       "                                              MultinomialNB(alpha=1.0,\n",
       "                                                            class_prior=None,\n",
       "                                                            fit_prior=True))],\n",
       "                                      verbose=False),\n",
       "                   iid='warn', n_iter=10, n_jobs=None,\n",
       "                   param_distributions={'classifier__alpha': <scipy.stats._distn_infrastructure.rv_frozen object at 0x000001DA349740C8>},\n",
       "                   pre_dispatch='2*n_jobs', random_state=None, refit=True,\n",
       "                   return_train_score=False,\n",
       "                   scoring=make_scorer(f1_score, pos_label=pos), verbose=0)"
      ]
     },
     "execution_count": 324,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_randSearch.fit(review_train,label_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 325,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "         steps=[('bow_transform',\n",
       "                 CountVectorizer(analyzer='word', binary=False,\n",
       "                                 decode_error='strict',\n",
       "                                 dtype=<class 'numpy.int64'>, encoding='utf-8',\n",
       "                                 input='content', lowercase=True, max_df=1.0,\n",
       "                                 max_features=None, min_df=1,\n",
       "                                 ngram_range=(1, 1), preprocessor=None,\n",
       "                                 stop_words=None, strip_accents=None,\n",
       "                                 token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "                                 tokenizer=None, vocabulary=None)),\n",
       "                ('tfidf_transform',\n",
       "                 TfidfTransformer(norm='l2', smooth_idf=True,\n",
       "                                  sublinear_tf=False, use_idf=True)),\n",
       "                ('classifier',\n",
       "                 MultinomialNB(alpha=0.5448322875490428, class_prior=None,\n",
       "                               fit_prior=True))],\n",
       "         verbose=False)"
      ]
     },
     "execution_count": 325,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_randSearch.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 326,
   "metadata": {},
   "outputs": [],
   "source": [
    "predicted_test_Randsearch = model_randSearch.predict(review_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 328,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification report \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         neg       0.86      0.87      0.86      5000\n",
      "         pos       0.87      0.86      0.86      5000\n",
      "\n",
      "    accuracy                           0.86     10000\n",
      "   macro avg       0.86      0.86      0.86     10000\n",
      "weighted avg       0.86      0.86      0.86     10000\n",
      "\n",
      "****************************************************************************************************\n",
      "Confusion metrics \n",
      "      0     1\n",
      "0  4354   646\n",
      "1   724  4276\n"
     ]
    }
   ],
   "source": [
    "print_classification_report(label_test,predicted_test_Randsearch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**As we can see we are not observing any improvement in our model at all**\n",
    "\n",
    "Yup no use of hyper-parameter search in this case. Or we may have failed to find best hyper-param space.\n",
    "\n",
    "**Let's try one more simple model i.e `logistic regression`**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 329,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 330,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create pipeline for logistic regression\n",
    "\n",
    "\n",
    "logistic_review_pipeline = Pipeline(steps = [\n",
    "    ('bow_transform',CountVectorizer()),\n",
    "    ('tfidf_transform',TfidfTransformer()),\n",
    "    ('classifier',LogisticRegression())\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 331,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "         steps=[('bow_transform',\n",
       "                 CountVectorizer(analyzer='word', binary=False,\n",
       "                                 decode_error='strict',\n",
       "                                 dtype=<class 'numpy.int64'>, encoding='utf-8',\n",
       "                                 input='content', lowercase=True, max_df=1.0,\n",
       "                                 max_features=None, min_df=1,\n",
       "                                 ngram_range=(1, 1), preprocessor=None,\n",
       "                                 stop_words=None, strip_accents=None,\n",
       "                                 token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "                                 tokenizer=None, vocab...\n",
       "                ('tfidf_transform',\n",
       "                 TfidfTransformer(norm='l2', smooth_idf=True,\n",
       "                                  sublinear_tf=False, use_idf=True)),\n",
       "                ('classifier',\n",
       "                 LogisticRegression(C=1.0, class_weight=None, dual=False,\n",
       "                                    fit_intercept=True, intercept_scaling=1,\n",
       "                                    l1_ratio=None, max_iter=100,\n",
       "                                    multi_class='warn', n_jobs=None,\n",
       "                                    penalty='l2', random_state=None,\n",
       "                                    solver='warn', tol=0.0001, verbose=0,\n",
       "                                    warm_start=False))],\n",
       "         verbose=False)"
      ]
     },
     "execution_count": 331,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logistic_review_pipeline.fit(review_train,label_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 332,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification report \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         neg       0.91      0.88      0.89      5000\n",
      "         pos       0.89      0.91      0.90      5000\n",
      "\n",
      "    accuracy                           0.90     10000\n",
      "   macro avg       0.90      0.90      0.90     10000\n",
      "weighted avg       0.90      0.90      0.90     10000\n",
      "\n",
      "****************************************************************************************************\n",
      "Confusion metrics \n",
      "      0     1\n",
      "0  4413   587\n",
      "1   462  4538\n"
     ]
    }
   ],
   "source": [
    "predicted_test_log = logistic_review_pipeline.predict(review_test)\n",
    "\n",
    "print_classification_report(label_test,predicted_test_log)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Wow looks like logistic regression is performing well**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 339,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's do some hyper-param search for logistic reg\n",
    "\n",
    "f1_scorer = make_scorer(f1_score, pos_label=\"pos\")\n",
    "\n",
    "parameter_to_tune = {\"classifier__C\":[1,2,3,4,5,6,7,8,9,10,10,11,12]} # pipeline name of the our algo or estimator\n",
    "\n",
    "model_grid_search = GridSearchCV(logistic_review_pipeline,\n",
    "                                 param_grid = parameter_to_tune,\n",
    "                                 scoring = f1_scorer,cv = 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 340,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=5, error_score='raise-deprecating',\n",
       "             estimator=Pipeline(memory=None,\n",
       "                                steps=[('bow_transform',\n",
       "                                        CountVectorizer(analyzer='word',\n",
       "                                                        binary=False,\n",
       "                                                        decode_error='strict',\n",
       "                                                        dtype=<class 'numpy.int64'>,\n",
       "                                                        encoding='utf-8',\n",
       "                                                        input='content',\n",
       "                                                        lowercase=True,\n",
       "                                                        max_df=1.0,\n",
       "                                                        max_features=None,\n",
       "                                                        min_df=1,\n",
       "                                                        ngram_range=(1, 1),\n",
       "                                                        preprocessor=None,\n",
       "                                                        stop_words=None,\n",
       "                                                        strip_accents...\n",
       "                                                           l1_ratio=None,\n",
       "                                                           max_iter=100,\n",
       "                                                           multi_class='warn',\n",
       "                                                           n_jobs=None,\n",
       "                                                           penalty='l2',\n",
       "                                                           random_state=None,\n",
       "                                                           solver='warn',\n",
       "                                                           tol=0.0001,\n",
       "                                                           verbose=0,\n",
       "                                                           warm_start=False))],\n",
       "                                verbose=False),\n",
       "             iid='warn', n_jobs=None,\n",
       "             param_grid={'classifier__C': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 10,\n",
       "                                           11, 12]},\n",
       "             pre_dispatch='2*n_jobs', refit=True, return_train_score=False,\n",
       "             scoring=make_scorer(f1_score, pos_label=pos), verbose=0)"
      ]
     },
     "execution_count": 340,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_grid_search.fit(review_train,label_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 341,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "         steps=[('bow_transform',\n",
       "                 CountVectorizer(analyzer='word', binary=False,\n",
       "                                 decode_error='strict',\n",
       "                                 dtype=<class 'numpy.int64'>, encoding='utf-8',\n",
       "                                 input='content', lowercase=True, max_df=1.0,\n",
       "                                 max_features=None, min_df=1,\n",
       "                                 ngram_range=(1, 1), preprocessor=None,\n",
       "                                 stop_words=None, strip_accents=None,\n",
       "                                 token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "                                 tokenizer=None, vocab...\n",
       "                ('tfidf_transform',\n",
       "                 TfidfTransformer(norm='l2', smooth_idf=True,\n",
       "                                  sublinear_tf=False, use_idf=True)),\n",
       "                ('classifier',\n",
       "                 LogisticRegression(C=5, class_weight=None, dual=False,\n",
       "                                    fit_intercept=True, intercept_scaling=1,\n",
       "                                    l1_ratio=None, max_iter=100,\n",
       "                                    multi_class='warn', n_jobs=None,\n",
       "                                    penalty='l2', random_state=None,\n",
       "                                    solver='warn', tol=0.0001, verbose=0,\n",
       "                                    warm_start=False))],\n",
       "         verbose=False)"
      ]
     },
     "execution_count": 341,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_grid_search.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 342,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification report \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         neg       0.91      0.89      0.90      5000\n",
      "         pos       0.89      0.91      0.90      5000\n",
      "\n",
      "    accuracy                           0.90     10000\n",
      "   macro avg       0.90      0.90      0.90     10000\n",
      "weighted avg       0.90      0.90      0.90     10000\n",
      "\n",
      "****************************************************************************************************\n",
      "Confusion metrics \n",
      "      0     1\n",
      "0  4430   570\n",
      "1   457  4543\n"
     ]
    }
   ],
   "source": [
    "predicted_test_log = model_grid_search.predict(review_test)\n",
    "\n",
    "print_classification_report(label_test,predicted_test_log)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['a', 'b', 'v', 'c', 'd']"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[\"a\",\"b\",\"v\"] + [\"c\",\"d\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1, 2, 3, 4, 5, 6]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[1,2,3] + [4,5,6]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['abc', 'abc', 'abc', 'abc', 'abc', 'abc', 'abc', 'abc', 'abc', 'abc']"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[\"abc\"]*10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.7.5 64-bit ('pytorch_env': conda)",
   "language": "python",
   "name": "python37564bitpytorchenvconda1a599ba2df0f415ab16aa82c434e76bb"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}