4.4.5_NLP_caesar_v_hamlet.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction \n",
    "This notebook attempts to build 4 models that predict the classification of a random sentence in the Shakespeare plays, Julius Caesar and Hamlet.  Two methods are used define the features of the models.  The first method vectorizes each sentence to determine the most relevant words or features for each corpus.  The second method tokenizes the words and finds the most common 1000.  These words (features) are then combined to create a unique set that will comprise the final columns of the dataset.  \n",
    "\n",
    "Some additional methods are then used to further distinguish a sentence from each of the corpora.\n",
    "\n",
    "The first step is to import the necessary modules and import the text data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     C:\\Users\\Fred\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import scipy\n",
    "import sklearn\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import re\n",
    "from nltk.corpus import gutenberg, stopwords\n",
    "from collections import Counter\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "\n",
    "import nltk\n",
    "nltk.download('punkt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import the nlp library, spacy\n",
    "import spacy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package gutenberg to\n",
      "[nltk_data]     C:\\Users\\Fred\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package gutenberg is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.download('gutenberg')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: en_core_web_sm==2.0.0 from https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.0.0/en_core_web_sm-2.0.0.tar.gz#egg=en_core_web_sm==2.0.0 in c:\\users\\fred\\anaconda3\\lib\\site-packages (2.0.0)\n",
      "\n",
      "    Linking successful\n",
      "    C:\\Users\\Fred\\Anaconda3\\lib\\site-packages\\en_core_web_sm -->\n",
      "    C:\\Users\\Fred\\Anaconda3\\lib\\site-packages\\spacy\\data\\en\n",
      "\n",
      "    You can now load the model via spacy.load('en')\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# this takes a long time\n",
    "\n",
    "### need to run as administrator from Anaconda3 promplt\n",
    "\n",
    "!python -m spacy download en"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['austen-emma.txt', 'austen-persuasion.txt', 'austen-sense.txt', 'bible-kjv.txt', 'blake-poems.txt', 'bryant-stories.txt', 'burgess-busterbrown.txt', 'carroll-alice.txt', 'chesterton-ball.txt', 'chesterton-brown.txt', 'chesterton-thursday.txt', 'edgeworth-parents.txt', 'melville-moby_dick.txt', 'milton-paradise.txt', 'shakespeare-caesar.txt', 'shakespeare-hamlet.txt', 'shakespeare-macbeth.txt', 'whitman-leaves.txt']\n"
     ]
    }
   ],
   "source": [
    "print(gutenberg.fileids())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Clean and explore the data  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Utility function for standard text cleaning.\n",
    "def text_cleaner(text):\n",
    "    # Visual inspection identifies a form of punctuation spaCy does not\n",
    "    # recognize: the double dash '--'.  Better get rid of it now!\n",
    "    text = re.sub(r'--',' ',text)\n",
    "    text = re.sub(\"[\\[].*?[\\]]\", \"\", text)\n",
    "    text = ' '.join(text.split())\n",
    "    return text\n",
    "    \n",
    "# Load and clean the data.\n",
    "caesar = gutenberg.raw('shakespeare-caesar.txt')\n",
    "hamlet = gutenberg.raw('shakespeare-hamlet.txt')\n",
    "\n",
    "# The Chapter indicator is idiosyncratic\n",
    "caesar = re.sub(r'Chapter \\d+', '', caesar)\n",
    "hamlet = re.sub(r'CHAPTER .*', '', hamlet)\n",
    "    \n",
    "caesar = text_cleaner(caesar[:int(len(caesar)/7)])\n",
    "hamlet = text_cleaner(hamlet[:int(len(hamlet)/10)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "15656\n",
      "15867\n"
     ]
    }
   ],
   "source": [
    "print(len(caesar))\n",
    "print(len(hamlet))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Parse the cleaned novels. This can take a bit.\n",
    "# had to do this in an admin terminal.......\n",
    "nlp = spacy.load('en')\n",
    "caesar_doc = nlp(caesar)\n",
    "hamlet_doc = nlp(hamlet)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>(Actus, Primus, .)</td>\n",
       "      <td>Shake_C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>(Scoena, Prima, .)</td>\n",
       "      <td>Shake_C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>(Enter, Flauius, ,, Murellus, ,, and, certaine, Commoner...</td>\n",
       "      <td>Shake_C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>(Flauius, .)</td>\n",
       "      <td>Shake_C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>(Hence, :, home, you, idle, Creatures, ,, get, you, home...</td>\n",
       "      <td>Shake_C</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                             0        1\n",
       "0                                           (Actus, Primus, .)  Shake_C\n",
       "1                                           (Scoena, Prima, .)  Shake_C\n",
       "2  (Enter, Flauius, ,, Murellus, ,, and, certaine, Commoner...  Shake_C\n",
       "3                                                 (Flauius, .)  Shake_C\n",
       "4  (Hence, :, home, you, idle, Creatures, ,, get, you, home...  Shake_C"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Group into sentences.\n",
    "caesar_sents = [[sent, \"Shake_C\"] for sent in caesar_doc.sents]\n",
    "hamlet_sents = [[sent, \"Shake_H\"] for sent in hamlet_doc.sents]\n",
    "\n",
    "# Combine the sentences from the two novels into one data frame.\n",
    "sentences = pd.DataFrame(caesar_sents + hamlet_sents)\n",
    "\n",
    "pd.set_option('max_colwidth', 60)\n",
    "\n",
    "sentences.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Using tf-idf to establish a list of \"common\" words to use as features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "#reading in the data, this time in the form of paragraphs\n",
    "caesar = gutenberg.paras('shakespeare-caesar.txt')\n",
    "\n",
    "#processing\n",
    "caesar_paras = []\n",
    "for paragraph in caesar: \n",
    "    for i in range(len(paragraph)):        \n",
    "        para = paragraph[i]\n",
    "        \n",
    "        # removing the double-dash from all words\n",
    "        # para = [re.sub(r'--', '', word) for word in para]\n",
    "        # Forming each paragraph into a string and adding it to the list of strings.\n",
    "        \n",
    "        caesar_paras.append(' '.join(para))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Build a model to generate a list of the most common words or features by vectorizing the sentences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of features: 535\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "X_train, X_test = train_test_split(caesar_paras, test_size=0.25, random_state=0)\n",
    "\n",
    "vectorizer = TfidfVectorizer(max_df=0.5, # drop words that occur in more than half the paragraphs\n",
    "                             min_df=4, # only use words that appear at least twice (or 4 times, etc...)\n",
    "                             stop_words='english', \n",
    "                             lowercase=True, #convert everything to lower case (since Alice in Wonderland has the HABIT of CAPITALIZING WORDS for EMPHASIS)\n",
    "                             use_idf=True,# we definitely want to use inverse document frequencies in our weighting\n",
    "                             norm=u'l2', # Applies a correction factor so that longer paragraphs and shorter paragraphs get treated equally\n",
    "                             smooth_idf=True # Adds 1 to all document frequencies, as if an extra document existed that used every word once.  Prevents divide-by-zero errors\n",
    "                            )\n",
    "\n",
    "\n",
    "#Applying the vectorizer\n",
    "caesar_paras_tfidf = vectorizer.fit_transform(caesar_paras)\n",
    "print(\"Number of features: %d\" % caesar_paras_tfidf.get_shape()[1])\n",
    "\n",
    "# #splitting into training and test sets\n",
    "# X_train_tfidf, X_test_tfidf= train_test_split(caesar_paras_tfidf, test_size=0.4, random_state=0)\n",
    "\n",
    "# #Reshapes the vectorizer output into something people can read\n",
    "# X_train_tfidf_csr = X_train_tfidf.tocsr()\n",
    "\n",
    "# #number of paragraphs\n",
    "# n = X_train_tfidf_csr.shape[0]\n",
    "\n",
    "# #A list of dictionaries, one per paragraph\n",
    "# tfidf_bypara = [{} for _ in range(0,n)]\n",
    "\n",
    "# List of features\n",
    "terms_c = vectorizer.get_feature_names()\n",
    "# print(terms_c)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "#reading in the data, this time in the form of paragraphs\n",
    "hamlet = gutenberg.paras('shakespeare-hamlet.txt')\n",
    "\n",
    "#processing\n",
    "hamlet_paras = []\n",
    "for paragraph in hamlet: \n",
    "    for i in range(len(paragraph)):        \n",
    "        para = paragraph[i]\n",
    "        \n",
    "        # removing the double-dash from all words\n",
    "        # para = [re.sub(r'--', '', word) for word in para]\n",
    "        # Forming each paragraph into a string and adding it to the list of strings.\n",
    "        \n",
    "        hamlet_paras.append(' '.join(para))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Build a model to generate a list of the most common words or features by vectorizing the sentences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of features: 696\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "X_train, X_test = train_test_split(caesar_paras, test_size=0.25, random_state=0)\n",
    "\n",
    "vectorizer = TfidfVectorizer(max_df=0.5, # drop words that occur in more than half the paragraphs\n",
    "                             min_df=4, # only use words that appear at least twice (or 4 times, etc...)\n",
    "                             stop_words='english', \n",
    "                             lowercase=True, #convert everything to lower case (since Alice in Wonderland has the HABIT of CAPITALIZING WORDS for EMPHASIS)\n",
    "                             use_idf=True,# we definitely want to use inverse document frequencies in our weighting\n",
    "                             norm=u'l2', # Applies a correction factor so that longer paragraphs and shorter paragraphs get treated equally\n",
    "                             smooth_idf=True # Adds 1 to all document frequencies, as if an extra document existed that used every word once.  Prevents divide-by-zero errors\n",
    "                            )\n",
    "\n",
    "\n",
    "#Applying the vectorizer\n",
    "hamlet_paras_tfidf = vectorizer.fit_transform(hamlet_paras)\n",
    "print(\"Number of features: %d\" % hamlet_paras_tfidf.get_shape()[1])\n",
    "\n",
    "# #splitting into training and test sets\n",
    "# X_train_tfidf, X_test_tfidf= train_test_split(hamlet_paras_tfidf, test_size=0.4, random_state=0)\n",
    "\n",
    "# #Reshapes the vectorizer output into something people can read\n",
    "# X_train_tfidf_csr = X_train_tfidf.tocsr()\n",
    "\n",
    "# #number of paragraphs\n",
    "# n = X_train_tfidf_csr.shape[0]\n",
    "\n",
    "# #A list of dictionaries, one per paragraph\n",
    "# tfidf_bypara = [{} for _ in range(0,n)]\n",
    "\n",
    "# List of features\n",
    "terms_h = vectorizer.get_feature_names()\n",
    "# print(terms_h)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a list of the most common words in each corpus."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Bag of words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "854\n"
     ]
    }
   ],
   "source": [
    "# Utility function to create a list of the 2000 most common words.\n",
    "def bag_of_words(text):\n",
    "    \n",
    "    # Filter out punctuation and stop words.\n",
    "    allwords = [token.lemma_\n",
    "                for token in text\n",
    "                if not token.is_punct\n",
    "                and not token.is_stop]\n",
    "    \n",
    "    # Return the most common words.\n",
    "    return [item[0] for item in Counter(allwords).most_common(500)]\n",
    "    \n",
    "\n",
    "# Creates a data frame with features for each word in our common word set.\n",
    "# Each value is the count of the times the word appears in each sentence.\n",
    "def bow_features(sentences, common_words):\n",
    "    \n",
    "    # Scaffold the data frame and initialize counts to zero.\n",
    "    df = pd.DataFrame(columns=common_words)\n",
    "    df['text_sentence'] = sentences[0]\n",
    "    df['text_source'] = sentences[1]\n",
    "    df.loc[:, common_words] = 0\n",
    "    \n",
    "    # Process each row, counting the occurrence of words in each sentence.\n",
    "    for i, sentence in enumerate(df['text_sentence']):\n",
    "        \n",
    "        # Convert the sentence to lemmas, then filter out punctuation,\n",
    "        # stop words, and uncommon words.\n",
    "        words = [token.lemma_\n",
    "                 for token in sentence\n",
    "                 if (\n",
    "                     not token.is_punct\n",
    "                     and not token.is_stop\n",
    "                     and token.lemma_ in common_words\n",
    "                 )]\n",
    "        \n",
    "        # Populate the row with word counts.\n",
    "        for word in words:\n",
    "            df.loc[i, word] += 1\n",
    "        \n",
    "        # This counter is just to make sure the kernel didn't hang.\n",
    "        if i % 50 == 0:\n",
    "            print(\"Processing row {}\".format(i))\n",
    "            \n",
    "    return df\n",
    "\n",
    "# Set up the bags.\n",
    "caesarwords = bag_of_words(caesar_doc)\n",
    "hamletwords = bag_of_words(hamlet_doc)\n",
    "\n",
    "# print(caesarwords)\n",
    "\n",
    "# Combine bags to create a set of unique words. (just add 'terms' here to get the final list)\n",
    "common_words = set(caesarwords + hamletwords)\n",
    "\n",
    "# print the number of common words found in caesar and hamlet\n",
    "print(len(common_words))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add the words / features found from the 2 methods 'bag_of_words' and 'tf-idf'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1404\n"
     ]
    }
   ],
   "source": [
    "common_words_with_tf = set(caesarwords + hamletwords + terms_c + terms_h)\n",
    "print(len(common_words_with_tf))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Processing row 0\n",
      "Processing row 50\n",
      "Processing row 100\n",
      "Processing row 150\n",
      "Processing row 200\n",
      "Processing row 250\n",
      "Processing row 300\n",
      "Processing row 350\n",
      "Processing row 400\n",
      "Processing row 450\n",
      "Processing row 500\n",
      "Processing row 550\n",
      "(551, 1406)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>to</th>\n",
       "      <th>blame</th>\n",
       "      <th>wonder</th>\n",
       "      <th>return'd</th>\n",
       "      <th>bear</th>\n",
       "      <th>apron</th>\n",
       "      <th>thunder</th>\n",
       "      <th>old</th>\n",
       "      <th>iust</th>\n",
       "      <th>great</th>\n",
       "      <th>...</th>\n",
       "      <th>fauour</th>\n",
       "      <th>true</th>\n",
       "      <th>windowes</th>\n",
       "      <th>burn</th>\n",
       "      <th>choose</th>\n",
       "      <th>be</th>\n",
       "      <th>morne</th>\n",
       "      <th>compact</th>\n",
       "      <th>text_sentence</th>\n",
       "      <th>text_source</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>(Actus, Primus, .)</td>\n",
       "      <td>Shake_C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>(Scoena, Prima, .)</td>\n",
       "      <td>Shake_C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>(Enter, Flauius, ,, Murellus, ,, and, certaine, Commoner...</td>\n",
       "      <td>Shake_C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>(Flauius, .)</td>\n",
       "      <td>Shake_C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>(Hence, :, home, you, idle, Creatures, ,, get, you, home...</td>\n",
       "      <td>Shake_C</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 1406 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  to blame wonder return'd bear apron thunder old iust great  ... fauour true  \\\n",
       "0  0     0      0        0    0     0       0   0    0     0  ...      0    0   \n",
       "1  0     0      0        0    0     0       0   0    0     0  ...      0    0   \n",
       "2  0     0      0        0    0     0       0   0    0     0  ...      0    0   \n",
       "3  0     0      0        0    0     0       0   0    0     0  ...      0    0   \n",
       "4  0     0      0        0    0     0       0   0    0     0  ...      0    0   \n",
       "\n",
       "  windowes burn choose be morne compact  \\\n",
       "0        0    0      0  0     0       0   \n",
       "1        0    0      0  0     0       0   \n",
       "2        0    0      0  0     0       0   \n",
       "3        0    0      0  0     0       0   \n",
       "4        0    0      0  0     0       0   \n",
       "\n",
       "                                                 text_sentence text_source  \n",
       "0                                           (Actus, Primus, .)     Shake_C  \n",
       "1                                           (Scoena, Prima, .)     Shake_C  \n",
       "2  (Enter, Flauius, ,, Murellus, ,, and, certaine, Commoner...     Shake_C  \n",
       "3                                                 (Flauius, .)     Shake_C  \n",
       "4  (Hence, :, home, you, idle, Creatures, ,, get, you, home...     Shake_C  \n",
       "\n",
       "[5 rows x 1406 columns]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create our data frame with features. This can take a while to run.\n",
    "word_counts = bow_features(sentences, common_words_with_tf)\n",
    "print(word_counts.shape)\n",
    "word_counts.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>to</th>\n",
       "      <th>blame</th>\n",
       "      <th>wonder</th>\n",
       "      <th>return'd</th>\n",
       "      <th>bear</th>\n",
       "      <th>apron</th>\n",
       "      <th>thunder</th>\n",
       "      <th>old</th>\n",
       "      <th>iust</th>\n",
       "      <th>great</th>\n",
       "      <th>...</th>\n",
       "      <th>true</th>\n",
       "      <th>windowes</th>\n",
       "      <th>burn</th>\n",
       "      <th>choose</th>\n",
       "      <th>be</th>\n",
       "      <th>morne</th>\n",
       "      <th>compact</th>\n",
       "      <th>text_sentence</th>\n",
       "      <th>text_source</th>\n",
       "      <th>num_words</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>(Actus, Primus, .)</td>\n",
       "      <td>Shake_C</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>(Scoena, Prima, .)</td>\n",
       "      <td>Shake_C</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>(Enter, Flauius, ,, Murellus, ,, and, certaine, Commoner...</td>\n",
       "      <td>Shake_C</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>(Flauius, .)</td>\n",
       "      <td>Shake_C</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>(Hence, :, home, you, idle, Creatures, ,, get, you, home...</td>\n",
       "      <td>Shake_C</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 1407 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  to blame wonder return'd bear apron thunder old iust great  ... true  \\\n",
       "0  0     0      0        0    0     0       0   0    0     0  ...    0   \n",
       "1  0     0      0        0    0     0       0   0    0     0  ...    0   \n",
       "2  0     0      0        0    0     0       0   0    0     0  ...    0   \n",
       "3  0     0      0        0    0     0       0   0    0     0  ...    0   \n",
       "4  0     0      0        0    0     0       0   0    0     0  ...    0   \n",
       "\n",
       "  windowes burn choose be morne compact  \\\n",
       "0        0    0      0  0     0       0   \n",
       "1        0    0      0  0     0       0   \n",
       "2        0    0      0  0     0       0   \n",
       "3        0    0      0  0     0       0   \n",
       "4        0    0      0  0     0       0   \n",
       "\n",
       "                                                 text_sentence text_source  \\\n",
       "0                                           (Actus, Primus, .)     Shake_C   \n",
       "1                                           (Scoena, Prima, .)     Shake_C   \n",
       "2  (Enter, Flauius, ,, Murellus, ,, and, certaine, Commoner...     Shake_C   \n",
       "3                                                 (Flauius, .)     Shake_C   \n",
       "4  (Hence, :, home, you, idle, Creatures, ,, get, you, home...     Shake_C   \n",
       "\n",
       "  num_words  \n",
       "0         3  \n",
       "1         3  \n",
       "2        12  \n",
       "3         2  \n",
       "4        11  \n",
       "\n",
       "[5 rows x 1407 columns]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_counts['num_words'] = word_counts['text_sentence'].apply(lambda x: len(x))\n",
    "word_counts.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following functions count the number of adverbs, interjections and pronouns and adds the columns to the word_counts dataframe in hopes that these additional features will improve the accuracy of the 4 models:  Random Forest, Logistic Regression, Gradient Boosting, and Support Vector Machines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Count the adverbs and add the column to the word_counts dataframe\n",
    "\n",
    "def count_verbs(txt):\n",
    "    sentences = nltk.sent_tokenize(str(txt))\n",
    "    count = 0\n",
    "    for sentence in sentences:    \n",
    "        text = nltk.word_tokenize(sentence)\n",
    "        tag = nltk.pos_tag(text)\n",
    "        a = pd.Series(tag)\n",
    "        a = a.map(lambda x: 1 if x[1] == \"RB\" else 0).sum()\n",
    "#       count = count + a\n",
    "    return a\n",
    "\n",
    "word_counts['count_adverbs'] = word_counts['text_sentence'].apply(lambda x: count_verbs(x))\n",
    "\n",
    "# -------------------------------------------------------------------------------------------------------\n",
    "# Count the interjections and add the column to the dataframe\n",
    "\n",
    "def count_verbs(txt):\n",
    "    sentences = nltk.sent_tokenize(str(txt))\n",
    "    count = 0\n",
    "    for sentence in sentences:    \n",
    "        text = nltk.word_tokenize(sentence)\n",
    "        tag = nltk.pos_tag(text)\n",
    "        a = pd.Series(tag)\n",
    "        a = a.map(lambda x: 1 if x[1] == \"UH\" else 0).sum()\n",
    "#       count = count + a\n",
    "    return a\n",
    "\n",
    "word_counts['count_inter'] = word_counts['text_sentence'].apply(lambda x: count_verbs(x))\n",
    "\n",
    "# ------------------------------------------------------------------------------------------------------\n",
    "# Count the pronouns and add the column to the dataframe\n",
    "\n",
    "def count_verbs(txt):\n",
    "    sentences = nltk.sent_tokenize(str(txt))\n",
    "    count = 0\n",
    "    for sentence in sentences:    \n",
    "        text = nltk.word_tokenize(sentence)\n",
    "        tag = nltk.pos_tag(text)\n",
    "        a = pd.Series(tag)\n",
    "        a = a.map(lambda x: 1 if x[1] == \"PRP\" else 0).sum()\n",
    "#       count = count + a\n",
    "    return a\n",
    "\n",
    "word_counts['count_pronoun'] = word_counts['text_sentence'].apply(lambda x: count_verbs(x))\n",
    "\n",
    "# ------------------------------------------------------------------------------------------------------\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the first 5 lines of the dataframe to make sure the knew features are captured."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>to</th>\n",
       "      <th>blame</th>\n",
       "      <th>wonder</th>\n",
       "      <th>return'd</th>\n",
       "      <th>bear</th>\n",
       "      <th>apron</th>\n",
       "      <th>thunder</th>\n",
       "      <th>old</th>\n",
       "      <th>iust</th>\n",
       "      <th>great</th>\n",
       "      <th>...</th>\n",
       "      <th>choose</th>\n",
       "      <th>be</th>\n",
       "      <th>morne</th>\n",
       "      <th>compact</th>\n",
       "      <th>text_sentence</th>\n",
       "      <th>text_source</th>\n",
       "      <th>num_words</th>\n",
       "      <th>count_adverbs</th>\n",
       "      <th>count_inter</th>\n",
       "      <th>count_pronoun</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>(Actus, Primus, .)</td>\n",
       "      <td>Shake_C</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>(Scoena, Prima, .)</td>\n",
       "      <td>Shake_C</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>(Enter, Flauius, ,, Murellus, ,, and, certaine, Commoner...</td>\n",
       "      <td>Shake_C</td>\n",
       "      <td>12</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>(Flauius, .)</td>\n",
       "      <td>Shake_C</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>(Hence, :, home, you, idle, Creatures, ,, get, you, home...</td>\n",
       "      <td>Shake_C</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 1410 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  to blame wonder return'd bear apron thunder old iust great  ... choose be  \\\n",
       "0  0     0      0        0    0     0       0   0    0     0  ...      0  0   \n",
       "1  0     0      0        0    0     0       0   0    0     0  ...      0  0   \n",
       "2  0     0      0        0    0     0       0   0    0     0  ...      0  0   \n",
       "3  0     0      0        0    0     0       0   0    0     0  ...      0  0   \n",
       "4  0     0      0        0    0     0       0   0    0     0  ...      0  0   \n",
       "\n",
       "  morne compact                                                text_sentence  \\\n",
       "0     0       0                                           (Actus, Primus, .)   \n",
       "1     0       0                                           (Scoena, Prima, .)   \n",
       "2     0       0  (Enter, Flauius, ,, Murellus, ,, and, certaine, Commoner...   \n",
       "3     0       0                                                 (Flauius, .)   \n",
       "4     0       0  (Hence, :, home, you, idle, Creatures, ,, get, you, home...   \n",
       "\n",
       "  text_source num_words count_adverbs count_inter count_pronoun  \n",
       "0     Shake_C         3             0           0             0  \n",
       "1     Shake_C         3             0           0             0  \n",
       "2     Shake_C        12             0           0             0  \n",
       "3     Shake_C         2             0           0             0  \n",
       "4     Shake_C        11             0           0             2  \n",
       "\n",
       "[5 rows x 1410 columns]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_counts.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(551, 1410)\n"
     ]
    }
   ],
   "source": [
    "print(word_counts.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build the models  \n",
    "The \"text_source\" is what the models are trying to predict."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Random Forest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Fred\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training set score: 0.9848484848484849\n",
      "\n",
      "Test set score: 0.7058823529411765\n",
      "\n",
      "Confusion Matrix:\n",
      "[[77 32]\n",
      " [33 79]]\n",
      "\n",
      "Raw Accuracy Score:\n",
      "0.7058823529411765\n"
     ]
    }
   ],
   "source": [
    "from sklearn import ensemble\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "rfc = ensemble.RandomForestClassifier()\n",
    "Y = word_counts['text_source']\n",
    "X = np.array(word_counts.drop(['text_sentence','text_source'], 1))\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, \n",
    "                                                    Y,\n",
    "                                                    test_size=0.4,\n",
    "                                                    random_state=0)\n",
    "# normalize the training data\n",
    "X_train = sklearn.preprocessing.normalize(X_train)\n",
    "\n",
    "# normalize the test data\n",
    "X_test = sklearn.preprocessing.normalize(X_test)\n",
    "\n",
    "train = rfc.fit(X_train, y_train)\n",
    "\n",
    "print('Training set score:', rfc.score(X_train, y_train))\n",
    "print('\\nTest set score:', rfc.score(X_test, y_test))\n",
    "\n",
    "# Confusion Matrix\n",
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "y_test_pred = rfc.predict(X_test)\n",
    "\n",
    "print('\\nConfusion Matrix:')\n",
    "print(sklearn.metrics.confusion_matrix(y_test, y_test_pred, labels=None, sample_weight=None))\n",
    "print('\\nRaw Accuracy Score:')\n",
    "print(sklearn.metrics.accuracy_score(y_test, y_test_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Fred\\Anaconda3\\lib\\site-packages\\sklearn\\linear_model\\logistic.py:433: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(330, 1408) (330,)\n",
      "Training set scorae: 0.6515151515151515\n",
      "\n",
      "Test set score: 0.5520361990950227\n",
      "\n",
      "Confusion Matrix:\n",
      "[[ 15  94]\n",
      " [  5 107]]\n",
      "\n",
      "Raw Accuracy Score:\n",
      "0.5520361990950227\n"
     ]
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "lr = LogisticRegression(penalty='l2') # No need to specify l2 as it's the default. But we put it for demonstration.\n",
    "train = lr.fit(X_train, y_train)\n",
    "print(X_train.shape, y_train.shape)\n",
    "print('Training set scorae:', lr.score(X_train, y_train))\n",
    "print('\\nTest set score:', lr.score(X_test, y_test))\n",
    "\n",
    "# Confusion Matrix\n",
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "y_test_pred = lr.predict(X_test)\n",
    "\n",
    "print('\\nConfusion Matrix:')\n",
    "print(sklearn.metrics.confusion_matrix(y_test, y_test_pred, labels=None, sample_weight=None))\n",
    "print('\\nRaw Accuracy Score:')\n",
    "print(sklearn.metrics.accuracy_score(y_test, y_test_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Gradient Boosting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training set score: 0.9363636363636364\n",
      "\n",
      "Test set score: 0.7692307692307693\n",
      "\n",
      "Confusion Matrix:\n",
      "[[ 67  42]\n",
      " [  9 103]]\n",
      "\n",
      "Raw Accuracy Score:\n",
      "0.7692307692307693\n"
     ]
    }
   ],
   "source": [
    "clf = ensemble.GradientBoostingClassifier()\n",
    "train = clf.fit(X_train, y_train)\n",
    "\n",
    "print('Training set score:', clf.score(X_train, y_train))\n",
    "print('\\nTest set score:', clf.score(X_test, y_test))\n",
    "\n",
    "# Confusion Matrix\n",
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "y_test_pred = clf.predict(X_test)\n",
    "\n",
    "print('\\nConfusion Matrix:')\n",
    "print(sklearn.metrics.confusion_matrix(y_test, y_test_pred, labels=None, sample_weight=None))\n",
    "print('\\nRaw Accuracy Score:')\n",
    "print(sklearn.metrics.accuracy_score(y_test, y_test_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Support Vector Machines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training set score: 0.5393939393939394\n",
      "\n",
      "Test set score: 0.5067873303167421\n",
      "\n",
      "Confusion Matrix:\n",
      "[[  0 109]\n",
      " [  0 112]]\n",
      "\n",
      "Raw Accuracy Score:\n",
      "0.5067873303167421\n"
     ]
    }
   ],
   "source": [
    "from sklearn.svm import SVC\n",
    "\n",
    "sv_c = SVC(gamma='auto')\n",
    "train = sv_c.fit(X_train, y_train)\n",
    "\n",
    "print('Training set score:', sv_c.score(X_train, y_train))\n",
    "print('\\nTest set score:', sv_c.score(X_test, y_test))\n",
    "\n",
    "# Confusion Matrix\n",
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "y_test_pred = sv_c.predict(X_test)\n",
    "\n",
    "print('\\nConfusion Matrix:')\n",
    "print(sklearn.metrics.confusion_matrix(y_test, y_test_pred, labels=None, sample_weight=None))\n",
    "print('\\nRaw Accuracy Score:')\n",
    "print(sklearn.metrics.accuracy_score(y_test, y_test_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluation and Conclusion  \n",
    "\n",
    "All of the models were evaluated using only about 10% of the corpora.  The initial accuracy scores (for test data set) for the 4 models are as follows:  \n",
    "\n",
    "- Random Forest:  0.76    \n",
    "- Logistic Regression:  0.79  \n",
    "- Gradient Boosting:  0.78  \n",
    "- Support Vector Machines:  0.51  \n",
    "\n",
    "Adjusting the number of 'most_common' words in bag of words from 2000 to 500 and requiring words to appear at least 4 times instead of twice in term frequency:\n",
    "\n",
    "- Random Forest:  **0.76**    \n",
    "- Logistic Regression:  **0.80**    \n",
    "- Gradient Boosting:  **0.78**  \n",
    "- Support Vector Machines:  **0.51**  \n",
    "\n",
    "With the additional feature of **'number of words in each sentence' and 'parts of speach'**:\n",
    "\n",
    "- Random Forest:  0.71  \n",
    "- Logistic Regression:  0.55  \n",
    "- Gradient Boosting:  0.77  \n",
    "- Support Vector Machines:  0.51  \n",
    "\n",
    "Adding a 'number of words per sentence' feature looks like it just added a bunch of noise to the data.  The accuracy scores got worse.  This is probably expected because the author of each corpus is the same (Shakespeare).  His writing is probably similar enough so that the number of words per sentence for each corpus would be similar.  \n",
    "\n",
    "Adding the parts of speach as a feature did not seem to help either.  Again, since the author is the same in this case, there would not be a significant variance in the use of **pronouns, adverbs or injunctions**.  If the works considered here were written with many years in between, the accuracy might improve with these added features.  However, Caesar and Hamlet were both written in 1599.\n",
    "\n",
    "The data looks balanced from the Confusion Matrix values; however, the Confusion Matrix for the Support Vector Machines model yielded 0 True Negatives and 0 False Negatives.  So, if classified everything as either a True Positive (112 occurences) or a False Positive (109 instances).  \n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}