Classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text classification\n",
    "\n",
    "- Assign input text into categories (classes):\n",
    "  - Filtering (e.g. spam / non-spam)\n",
    "  - Topic identification (e.g. politics, sports, business)\n",
    "  - Sentiment analysis (positive / negative / neutral)\n",
    "  - Language identification (Finnish, English, ...)\n",
    "  - Authorship attribution (who wrote this text?)\n",
    "  - (and many more)\n",
    "  \n",
    "---  \n",
    "\n",
    "**Example**: classify each of the following email subject lines as `spam` or `ham` (non-spam)\n",
    "\n",
    "```\n",
    "  Subject: want to make more money ?\n",
    "  Subject: during / after hours contact information\n",
    "  Subject: learn to save on medications at discount pharmacy\n",
    "  Subject: invitation to dinner\n",
    "  Subject: your 60 second auto loan will be accepted\n",
    "  Subject: diet medications online\n",
    "  Subject: december preliminary production estimate\n",
    "  Subject: first deliveries - comstock oil & gas and united oil & mineral\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Learning to classify text\n",
    "  \n",
    "- In this lesson, focus on **supervised** classification\n",
    "  - We assume a *training set* of example inputs (texts) with correct outputs (classes)\n",
    "  - (Unsupervised classification (clustering) covered separately)\n",
    "- Machine learning approach:\n",
    "  - Given example texts and their classes $\\textbf{y} = \\{ y_1, y_2, \\ldots y_n \\}$\n",
    "  - Represent each document as a feature vector $\\textbf{x} = \\{ x_1, x_2, \\ldots x_n \\}$  \n",
    "  - Learn a function $f(X) \\rightarrow Y$ to predict the class given text features\n",
    "- Note: data is unordered, and there are no dependencies between texts (cf. sequence tagging)\n",
    "\n",
    "### Overview\n",
    "\n",
    "- Tokenization: split text into words (and non-word elements such as punctuation)\n",
    "- Feature extraction: create discrete features representing the text\n",
    "- Vectorization: map feature representation (human-readable) to vector form (machine-readable)\n",
    "- Training: given vectorized training data and classes, train machine learning model\n",
    "- Prediction: given vectorized test (or dev) data, predict classes\n",
    "- Evaluation: compare predicted to true classes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Spam classification\n",
    "\n",
    "We'll use a version of the [Enron-Spam](http://www2.aueb.gr/users/ion/data/enron-spam/) corpus created by [Metsis et al. (2006)](http://nlp.cs.aueb.gr/pubs/ceas2006_paper.pdf) based on the [Enron email dataset](https://www.cs.cmu.edu/~./enron/) as an example text classification task.\n",
    "\n",
    "First, let's have a look at the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "spam    16798\n",
      "ham     16545\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "\n",
    "def class_counts(df, label='class'):\n",
    "    return df[label].value_counts().to_string(header=None)\n",
    "\n",
    "\n",
    "text_data = pd.read_csv('data/enron-spam/enron-spam-subjects.tsv', sep='\\t', names=('class', 'id', 'text'))\n",
    "text_data = text_data[['class', 'text']]    # drop IDs\n",
    "\n",
    "print(class_counts(text_data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>class</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>spam</td>\n",
       "      <td>what up , , your cam babe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>spam</td>\n",
       "      <td>want to make more money ?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>spam</td>\n",
       "      <td>food for thoughts</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>spam</td>\n",
       "      <td>miningnews . net newsletter - tuesday , januar...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>spam</td>\n",
       "      <td>your pharmacy ta</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  class                                               text\n",
       "0  spam                          what up , , your cam babe\n",
       "1  spam                          want to make more money ?\n",
       "2  spam                                  food for thoughts\n",
       "3  spam  miningnews . net newsletter - tuesday , januar...\n",
       "4  spam                                   your pharmacy ta"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_data[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>class</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>33338</th>\n",
       "      <td>ham</td>\n",
       "      <td>bullets 9 / 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33339</th>\n",
       "      <td>ham</td>\n",
       "      <td>eog material</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33340</th>\n",
       "      <td>ham</td>\n",
       "      <td>associate / analyst fall recruiting</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33341</th>\n",
       "      <td>ham</td>\n",
       "      <td>tw bullets 1 / 26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33342</th>\n",
       "      <td>ham</td>\n",
       "      <td>dashboard enhancements</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      class                                 text\n",
       "33338   ham                        bullets 9 / 1\n",
       "33339   ham                         eog material\n",
       "33340   ham  associate / analyst fall recruiting\n",
       "33341   ham                    tw bullets 1 / 26\n",
       "33342   ham               dashboard enhancements"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_data[-5:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(Note that we're working with raw text: there are no lemmas, part-of-speech tags or similar.)\n",
    "\n",
    "Split randomly into training, development and test sets. We'll again work on the training and development sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "- Training -\n",
      "spam    10071\n",
      "ham      9934\n",
      "\n",
      "- Development -\n",
      "spam    3360\n",
      "ham     3309\n",
      "\n",
      "- Test -\n",
      "spam    3367\n",
      "ham     3302\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "\n",
    "train_texts, devel_and_test_texts = train_test_split(text_data, test_size=0.4, random_state=1234)\n",
    "devel_texts, test_texts = train_test_split(devel_and_test_texts, test_size=0.5, random_state=5678)\n",
    "\n",
    "for label, dataset in (('Training', train_texts),\n",
    "                       ('Development', devel_texts),\n",
    "                       ('Test', test_texts)):\n",
    "    print('- {} -'.format(label))\n",
    "    print(class_counts(dataset))\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature representation\n",
    "\n",
    "Machine learning methods do not understand text, so we need to represent it as features, using e.g.:\n",
    "\n",
    "* Word forms, lemmas, parts-of-speech, and their combinations\n",
    "* Word or character N-grams (pairs, triples, etc. of consecutive words/characters)\n",
    "* Document metadata (author, time of creation, ...)\n",
    "* ...\n",
    "\n",
    "Simple document representation:\n",
    "\n",
    "* Split text into words, create a feature for each (unique) word\n",
    "* Note that this discards word order → no distinction between e.g. \"man bites dog\" and \"dog bites man\"\n",
    "* *Bag-of-words* representations such as this often also encode the number of occurrences of each word\n",
    "* This can be surprisingly effective: many things about texts can be learned even if you only know which words occur"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['spam', 'fw : upto 50 % off on prescrlpt 1 on drogs zgpxruoeyyimjhq e']\n",
      "\n",
      "('spam',\n",
      " {'%': True,\n",
      "  '1': True,\n",
      "  '50': True,\n",
      "  ':': True,\n",
      "  'drogs': True,\n",
      "  'e': True,\n",
      "  'fw': True,\n",
      "  'off': True,\n",
      "  'on': True,\n",
      "  'prescrlpt': True,\n",
      "  'upto': True,\n",
      "  'zgpxruoeyyimjhq': True})\n"
     ]
    }
   ],
   "source": [
    "from collections import Counter\n",
    "\n",
    "from pprint import pprint as pretty_print\n",
    "\n",
    "\n",
    "def word_features(text):\n",
    "    features = {}\n",
    "    for word in text.split():\n",
    "        features[word] = True\n",
    "    return features\n",
    "\n",
    "\n",
    "def classes_and_features(dataset, featurizer=word_features):\n",
    "    classes, feats = [], []\n",
    "    for class_, text in dataset.values:\n",
    "        classes.append(class_)\n",
    "        feats.append(featurizer(text))\n",
    "    return classes, feats\n",
    "\n",
    "\n",
    "train_classes, train_data = classes_and_features(train_texts)\n",
    "devel_classes, devel_data = classes_and_features(devel_texts)\n",
    "test_classes, test_data = classes_and_features(test_texts)\n",
    "\n",
    "pretty_print(list(train_texts.values[0]))\n",
    "print()\n",
    "pretty_print((train_classes[0], train_data[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reminder: Naive Bayes\n",
    "\n",
    "* Simple probabilistic classifier based on class priors (`P(spam)`) and conditional feature probabilities (`P(viagra|spam)`)\n",
    "* Estimate probabilities based on counts of classes and (feature, class) pairs in training data\n",
    "    * Prior: `P(spam) = count(spam) / count(*)`\n",
    "    * Conditional: `P(viagra|spam) = count(viagra, spam) / count(*, spam)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Most Informative Features\n",
      "                    2001 = True              ham : spam   =    143.8 : 1.0\n",
      "                   meter = True              ham : spam   =    135.5 : 1.0\n",
      "               interview = True              ham : spam   =     75.4 : 1.0\n",
      "                 quality = True             spam : ham    =     70.0 : 1.0\n",
      "                   group = True              ham : spam   =     63.2 : 1.0\n",
      "                 revised = True              ham : spam   =     61.8 : 1.0\n",
      "                       > = True             spam : ham    =     59.4 : 1.0\n",
      "                       % = True             spam : ham    =     57.8 : 1.0\n",
      "            presentation = True              ham : spam   =     57.1 : 1.0\n",
      "                    2000 = True              ham : spam   =     56.4 : 1.0\n",
      "              california = True              ham : spam   =     54.9 : 1.0\n",
      "              nomination = True              ham : spam   =     54.9 : 1.0\n",
      "                      em = True             spam : ham    =     54.9 : 1.0\n",
      "                    hour = True              ham : spam   =     47.9 : 1.0\n",
      "              conference = True              ham : spam   =     45.2 : 1.0\n",
      "                       = = True             spam : ham    =     44.1 : 1.0\n",
      "                      80 = True             spam : ham    =     41.1 : 1.0\n",
      "                     gas = True              ham : spam   =     40.1 : 1.0\n",
      "               agreement = True              ham : spam   =     39.5 : 1.0\n",
      "               submitted = True              ham : spam   =     39.5 : 1.0\n",
      "                     low = True             spam : ham    =     39.0 : 1.0\n",
      "                    save = True             spam : ham    =     36.7 : 1.0\n",
      "                      iv = True              ham : spam   =     36.2 : 1.0\n",
      "                      01 = True              ham : spam   =     35.4 : 1.0\n",
      "                     buy = True             spam : ham    =     34.7 : 1.0\n"
     ]
    }
   ],
   "source": [
    "from nltk import NaiveBayesClassifier\n",
    "\n",
    "\n",
    "nb_classifier = NaiveBayesClassifier.train(zip(train_data, train_classes))\n",
    "\n",
    "nb_classifier.show_most_informative_features(25)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "93.04%\n"
     ]
    }
   ],
   "source": [
    "from nltk import classify\n",
    "from nltk.metrics import scores\n",
    "\n",
    "pred_classes = nb_classifier.classify_many(devel_data)\n",
    "\n",
    "print('{:.2%}'.format(scores.accuracy(devel_classes, pred_classes)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks OK, but accuracy is not very informative. Here we care very much about high recall, i.e. losing as few good (ham) emails as possible."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reminder: metrics\n",
    "\n",
    "In the context of binary classification where one class is positive (e.g. `ham`) and the other negative (e.g. `spam`):\n",
    "\n",
    "* Accuracy: fraction of predicted that are correct, i.e. identical to the true class\n",
    "* Precision: fraction of predicted positives that are correct\n",
    "* Recall: fraction of positives that were correctly predicted\n",
    "* F-score: harmonic mean of precision and recall\n",
    "\n",
    "More precisely\n",
    "\n",
    "* True positives (**tp**): prediction **true** (i.e. correct), predicted class **positive**\n",
    "* False negative (**fn**): prediction **false** (i.e. incorrect), predicted class **negative**\n",
    "* True negative (**tn**): prediction **true**, predicted class **negative**\n",
    "* False positive (**fp**): prediction **false**, predicted class **positive**\n",
    "\n",
    "and\n",
    "\n",
    "* Accuracy: (tp + tn)/(tp + fp + fn + tn)\n",
    "* Precision (p): tp / (tp + fp)\n",
    "* Recall (r): tp / (tp + fn)\n",
    "* F-score: 2 * p * r / (p + r)\n",
    "\n",
    "(Specifically, that's $F_1$-score, the balanced harmonic mean.)\n",
    "\n",
    "Or, in code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy :\t93.04%\n",
      "Precision:\t94.49%\n",
      "Recall   :\t91.30%\n",
      "F-score  :\t92.87%\n"
     ]
    }
   ],
   "source": [
    "def counts(true_classes, pred_classes, positive_class='ham'):\n",
    "    tp, fp, fn, tn = 0, 0, 0, 0\n",
    "    for true, pred in zip(true_classes, pred_classes):\n",
    "        if pred == true:\n",
    "            if pred == positive_class:\n",
    "                tp += 1\n",
    "            else:\n",
    "                tn += 1\n",
    "        else:    # pred != true\n",
    "            if pred == positive_class:\n",
    "                fp += 1\n",
    "            else:\n",
    "                fn += 1\n",
    "    return tp, fp, fn, tn\n",
    "\n",
    "\n",
    "def accuracy(true_classes, pred_classes):\n",
    "    tp, fp, fn, tn = counts(true_classes, pred_classes)\n",
    "    return (tp + tn) / (tp + fp + fn + tn)\n",
    "\n",
    "\n",
    "def precision(true_classes, pred_classes, positive_class='ham'):\n",
    "    tp, fp, fn, _ = counts(true_classes, pred_classes, positive_class)\n",
    "    return tp / (tp + fp)\n",
    "\n",
    "\n",
    "def recall(true_classes, pred_classes, positive_class='ham'):\n",
    "    tp, fp, fn, _ = counts(true_classes, pred_classes, positive_class)\n",
    "    return tp / (tp + fn)\n",
    "\n",
    "\n",
    "def f1_score(true_classes, pred_classes, positive_class='ham'):\n",
    "    p = precision(true_classes, pred_classes, positive_class)\n",
    "    r = recall(true_classes, pred_classes, positive_class)\n",
    "    return 2 * p * r / (p + r)\n",
    "\n",
    "\n",
    "print('Accuracy :\\t{:.2%}'.format(accuracy(devel_classes, pred_classes)))\n",
    "print('Precision:\\t{:.2%}'.format(precision(devel_classes, pred_classes)))\n",
    "print('Recall   :\\t{:.2%}'.format(recall(devel_classes, pred_classes)))\n",
    "print('F-score  :\\t{:.2%}'.format(f1_score(devel_classes, pred_classes)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why not just accuracy?\n",
    "\n",
    "Precision, recall and F-score are used in particular when positives are only a small fraction of the data. Consider e.g. an information retrieval setting where\n",
    "\n",
    "* The total number of documents is 1000\n",
    "* There are 10 documents about apples (the fruit)\n",
    "* There are 20 documents about Apple (the company)\n",
    "* Looking for information about fruit, a user queries two systems for \"apple\":\n",
    "  - **System 1** returns nothing (it never does)\n",
    "  - **System 2** returns the 10 fruit documents (positive) and the 20 Apple, Inc. documents (negative)\n",
    "  \n",
    "Evaluation:\n",
    "\n",
    "* **System 1** accuracy: 990 / 1000 = 0.99\n",
    "* **System 2** accuracy: 980 / 1000 = 0.98\n",
    "\n",
    "But\n",
    "\n",
    "* **System 1** precision, recall and F-score are zero\n",
    "* **System 2** precision: 10 / 30 = 0.33\n",
    "* **System 2** recall: 10 / 10 = 1.0\n",
    "* **System 2** F-score: 0.5\n",
    "\n",
    "The system that returns nothing has higher accuracy, but is performs lower in terms of precision, recall, and F-score.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Support Vector Machines\n",
    "\n",
    "* Machine learning method that finds (hyper)plane that optimally separates data\n",
    "* Original formulation as linear classifier in 1963, non-linear extension in 1992\n",
    "* Reasonably fast and highly competitive in many tasks, including text classification\n",
    "* Simple case: separate points in 2D (can't visualize >10,000D) by finding plane with *maximum margin* to nearest points"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://upload.wikimedia.org/wikipedia/commons/2/20/Svm_separating_hyperplanes.png\" width=\"35%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align:center; color:gray; font-size:80%\">(Figure from <a href=\"https://commons.wikimedia.org/wiki/File:Svm_separating_hyperplanes.png\">Wikimedia Commons</a>)</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* The method learns a weight vector **w** defining the orientation of the plane and a bias term $b$ defining its position"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://upload.wikimedia.org/wikipedia/commons/7/72/SVM_margin.png\" width=\"35%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align:center; color:gray; font-size:80%\">(Figure from <a href=\"https://commons.wikimedia.org/wiki/File:Svm_separating_hyperplanes.png\">Wikimedia Commons</a>)</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Convex optimization problem (unique minimum) with efficient optimization methods (esp. linear time for linear SVM)\n",
    "* The solution is determined by the points \"supporting\" the margin (hence the name)\n",
    "\n",
    "\n",
    "In general, data may not be linearly separable, and a \"hard\" margin may not be optimal when if it is\n",
    "\n",
    "* To address this, the soft-margin SVM formulation allows for some, i.e. points that are inside the margin\n",
    "* Tradeoff between maximizing the margin and minimizing errors controlled by cost parameter C\n",
    "    * (as $C \\rightarrow \\infty$, the solution apporaches a hard margin)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SVM in practice"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature vectorization\n",
    "\n",
    "* SVM implementations use integer indices (0, 1, 2, ...) to identify features\n",
    "    * (The same goes for other methods we have used, but the ML libraries provided a transparent string mapping)\n",
    "* Before training, we explicitly map our feature sets to vectors (\"vectorize\" the features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Examples from feature index ↔ name mapping:\n",
      "\n",
      "     0 ↔ !\n",
      "   544 ↔ 53813\n",
      "  1088 ↔ admin\n",
      "  1632 ↔ arterial\n",
      "  2176 ↔ blonde\n",
      "  2720 ↔ ce\n",
      "  3264 ↔ condescend\n",
      "  3808 ↔ decertify\n",
      "  4352 ↔ dummies\n",
      "  4896 ↔ eveis\n",
      "  5440 ↔ friedman\n",
      "  5984 ↔ harvested\n",
      "  6528 ↔ india\n",
      "  7072 ↔ klno\n",
      "  7616 ↔ luther\n",
      "  8160 ↔ model\n",
      "  8704 ↔ obligations\n",
      "  9248 ↔ peer\n",
      "  9792 ↔ print\n",
      " 10336 ↔ regressions\n",
      " 10880 ↔ scarify\n",
      " 11424 ↔ solutions\n",
      " 11968 ↔ tailgate\n",
      " 12512 ↔ turned\n",
      " 13056 ↔ ward\n",
      " 13600 ↔ zpalx\n"
     ]
    }
   ],
   "source": [
    "import sklearn.svm\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "\n",
    "def basic_tokenizer(txt):\n",
    "    \"\"\"Simple whitespace tokenizer\"\"\"\n",
    "    return txt.split()\n",
    "\n",
    "\n",
    "# Map texts to sparse matrix of token counts (bag-of-words)\n",
    "# Here, vectorizer.fit() learns the vocabulary, and\n",
    "# vectorizer.transform() performs the mapping.\n",
    "# (There is also a combined fit_transform() function.)\n",
    "vectorizer = CountVectorizer(tokenizer=basic_tokenizer)\n",
    "vectorizer.fit(train_texts['text'])\n",
    "train_X = vectorizer.transform(train_texts['text'])\n",
    "\n",
    "feature_names = vectorizer.get_feature_names()\n",
    "\n",
    "print('Examples from feature index ↔ name mapping:\\n')\n",
    "feature_num = train_X.shape[1]\n",
    "for index in range(0, feature_num, int(feature_num/25)):\n",
    "    print('{:6d} ↔ {}'.format(index, feature_names[index]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "texts x features: (20005, 13616)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<20005x13616 sparse matrix of type '<class 'numpy.int64'>'\n",
       "\twith 132462 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"texts x features: {}\".format(train_X.shape))\n",
    "\n",
    "train_X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Each text has now been mapped to a long vector (rows of the matrix)\n",
    "    * the dimensions correspond to words and the values to their counts in the text\n",
    "* Note that almost all of the values will be zero: no text contains more than a small fraction of possible words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"figs/bow_vector.png\" width=\"60%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align:center; font-size:80%\">A vector representation of the sentence \"a vector representation of a sentence\"</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature indices and values for an examples sentence:\n",
      "\n",
      "     4 %\n",
      "    83 1\n",
      "   509 50\n",
      "   920 :\n",
      "  4313 drogs\n",
      "  4391 e\n",
      "  5517 fw\n",
      "  8742 off\n",
      "  8813 on\n",
      "  9747 prescrlpt\n",
      " 12693 upto\n",
      " 13575 zgpxruoeyyimjhq\n"
     ]
    }
   ],
   "source": [
    "print('Feature indices and values for an examples sentence:\\n')\n",
    "\n",
    "for index in train_X[0].nonzero()[1]:\n",
    "    print('{:6d}'.format(index), feature_names[index])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearSVC(C=1.0, class_weight=None, dual=True, fit_intercept=True,\n",
       "     intercept_scaling=1, loss='squared_hinge', max_iter=10000,\n",
       "     multi_class='ovr', penalty='l2', random_state=None, tol=0.0001,\n",
       "     verbose=0)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.svm import LinearSVC\n",
    "\n",
    "\n",
    "svm_classifier = LinearSVC(max_iter=10000)\n",
    "svm_classifier.fit(train_X, train_classes)    # note that classes are strings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy : 92.89%\n",
      "Precision: 92.97%\n",
      "Recall   : 92.69%\n",
      "F-score  : 92.83%\n"
     ]
    }
   ],
   "source": [
    "devel_X = vectorizer.transform(devel_texts['text'])\n",
    "\n",
    "pred_classes = svm_classifier.predict(devel_X)\n",
    "\n",
    "# svm_classifier.score(devel_X, devel_classes)\n",
    "\n",
    "print('Accuracy : {:.2%}'.format(accuracy(devel_classes, pred_classes)))\n",
    "print('Precision: {:.2%}'.format(precision(devel_classes, pred_classes)))\n",
    "print('Recall   : {:.2%}'.format(recall(devel_classes, pred_classes)))\n",
    "print('F-score  : {:.2%}'.format(f1_score(devel_classes, pred_classes)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Huh, no better than Naive Bayes. Let's peek inside"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <style>\n",
       "    table.eli5-weights tr:hover {\n",
       "        filter: brightness(85%);\n",
       "    }\n",
       "</style>\n",
       "\n",
       "\n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "        \n",
       "\n",
       "    \n",
       "\n",
       "        \n",
       "            \n",
       "                \n",
       "                \n",
       "    \n",
       "        <p style=\"margin-bottom: 0.5em; margin-top: 0em\">\n",
       "            <b>\n",
       "    \n",
       "        y=spam\n",
       "    \n",
       "</b>\n",
       "\n",
       "top features\n",
       "        </p>\n",
       "    \n",
       "    <table class=\"eli5-weights\"\n",
       "           style=\"border-collapse: collapse; border: none; margin-top: 0em; table-layout: auto; margin-bottom: 2em;\">\n",
       "        <thead>\n",
       "        <tr style=\"border: none;\">\n",
       "            \n",
       "                <th style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\" title=\"Feature weights. Note that weights do not account for feature value scales, so if feature values have different scales, features with highest weights might not be the most important.\">\n",
       "                    Weight<sup>?</sup>\n",
       "                </th>\n",
       "            \n",
       "            <th style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">Feature</th>\n",
       "            \n",
       "        </tr>\n",
       "        </thead>\n",
       "        <tbody>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 83.12%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.659\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        learn\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 83.14%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.655\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        rolex\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 83.87%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.553\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        money\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 84.70%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.441\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        girls\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 84.71%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.439\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        invite\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 84.73%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.437\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        rotor\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 84.76%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.433\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        404\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 84.79%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.429\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        doctor\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 84.81%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.426\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        1776\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 84.87%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.417\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        impaired\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 85.08%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.390\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        2004\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 85.21%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.373\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        passenger\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 85.22%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.372\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        applying\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 85.28%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.364\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        appointment\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 85.30%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.361\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        565\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 85.40%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.347\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        kin\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 85.60%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.321\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        adv\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 85.60%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.321\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        spam\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 85.73%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.305\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        her\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 85.78%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.297\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        savings\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 85.87%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.286\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        logo\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 85.95%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.275\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        2005\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 85.95%); border: none;\">\n",
       "                <td colspan=\"2\" style=\"padding: 0 0.5em 0 0.5em; text-align: center; border: none; white-space: nowrap;\">\n",
       "                    <i>&hellip; 4965 more positive &hellip;</i>\n",
       "                </td>\n",
       "            </tr>\n",
       "        \n",
       "\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 85.85%); border: none;\">\n",
       "                <td colspan=\"2\" style=\"padding: 0 0.5em 0 0.5em; text-align: center; border: none; white-space: nowrap;\">\n",
       "                    <i>&hellip; 3427 more negative &hellip;</i>\n",
       "                </td>\n",
       "            </tr>\n",
       "        \n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 85.85%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.288\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        arrived\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 85.80%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.295\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        transmission\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 85.72%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.305\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        ferc\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 85.72%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.305\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        gtc\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 85.70%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.308\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        license\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 85.66%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.314\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        standards\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 85.51%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.334\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        bout\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 85.45%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.341\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        expertfinder\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 85.35%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.354\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        peoplefinder\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 85.28%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.363\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        presentation\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 85.27%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.365\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        activated\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 85.25%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.367\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        reactions\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 85.03%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.396\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        texas\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 84.89%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.416\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        united\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 84.87%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.418\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        draft\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 84.87%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.418\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        responding\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 84.65%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.448\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        devon\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 84.38%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.484\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        clickathome\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 84.22%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.506\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        tragedy\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 84.15%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.515\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        ordered\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 83.83%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.559\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        dave\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 83.44%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.613\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        vince\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 83.36%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.624\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        sap\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 83.17%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.651\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        morning\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 81.36%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.910\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        enron\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 81.28%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.922\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        guess\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 81.13%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.943\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        clerk\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 80.00%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -2.112\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        connection\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "\n",
       "        </tbody>\n",
       "    </table>\n",
       "\n",
       "            \n",
       "        \n",
       "\n",
       "        \n",
       "\n",
       "\n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import eli5\n",
    "\n",
    "\n",
    "eli5.show_weights(svm_classifier, vec=vectorizer, top=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(Note that only one class is shown as the task is binary: positive features for one class are negative for the other.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Recap: vector representation of text\n",
    "\n",
    "(Expanding on yesterday's explanation)\n",
    "\n",
    "Assume the following is our entire (training) dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "texts = [\n",
    "    \"money money money\",\n",
    "    \"invitation to dinner\",\n",
    "    \"buy medications online\",\n",
    "    \"production estimate online\",\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at mapping these texts into bag-of-words vectors.\n",
    "\n",
    "Before we do, recall that if we mapped each texts into bag-of-words features, it would look something like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'money': 3},\n",
       " {'invitation': 1, 'to': 1, 'dinner': 1},\n",
       " {'buy': 1, 'medications': 1, 'online': 1},\n",
       " {'production': 1, 'estimate': 1, 'online': 1}]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from collections import Counter\n",
    "\n",
    "\n",
    "featurized = []\n",
    "for text in texts:\n",
    "    features = Counter()\n",
    "    for word in text.split():\n",
    "        features[word] += 1\n",
    "    featurized.append(dict(features))\n",
    "\n",
    "featurized"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(In Python 3.6+ the keys in `dict` appear in insertion order. This implementation detail should not be misundersood as suggesting that the representation preserves order!) \n",
    "\n",
    "Now with vectors. First, learn the vocabulary and create the index ↔ string mapping:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>buy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>dinner</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>estimate</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>invitation</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>medications</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>money</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>online</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>production</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>to</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             0\n",
       "0          buy\n",
       "1       dinner\n",
       "2     estimate\n",
       "3   invitation\n",
       "4  medications\n",
       "5        money\n",
       "6       online\n",
       "7   production\n",
       "8           to"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from pandas import DataFrame    # DataFrame is just here to make output look nice\n",
    "\n",
    "\n",
    "def basic_tokenizer(txt):\n",
    "    return txt.split()\n",
    "\n",
    "\n",
    "vectorizer = CountVectorizer(tokenizer=basic_tokenizer)\n",
    "vectorizer.fit(texts)    # learn vocabulary, create index <-> string mapping\n",
    "\n",
    "feature_names = vectorizer.get_feature_names()\n",
    "\n",
    "DataFrame(feature_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This simply establishes the mapping between vector indices (0, 1, 2, ...) and feature names (\"buy\", \"dinner\", \"estimate\", ...).\n",
    "\n",
    "Next, extract bag-of-words features and map each text to a vector. (Both done by `CountVectorizer.transform()`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = vectorizer.transform(texts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This gives a sparse matrix, but that's just for computational efficiency. Let's look at the equivalent dense version instead:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   0  1  2  3  4  5  6  7  8\n",
       "0  0  0  0  0  0  3  0  0  0\n",
       "1  0  1  0  1  0  0  0  0  1\n",
       "2  1  0  0  0  1  0  1  0  0\n",
       "3  0  0  1  0  0  0  1  1  0"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dense_X = X.todense()\n",
    "\n",
    "DataFrame(dense_X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each row corresponds to one of our texts (in their original order) and each column to one of the features (here, words).\n",
    "\n",
    "Looking at just the first row:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   0  1  2  3  4  5  6  7  8\n",
       "0  0  0  0  0  0  3  0  0  0"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "DataFrame(dense_X[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The only non-zero value is 3 in index 5, i.e."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'money'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feature_names[5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'money money money'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "texts[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   0  1  2  3  4  5  6  7  8\n",
       "0  0  1  0  1  0  0  0  0  1"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "DataFrame(dense_X[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('dinner', 'invitation', 'to')"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feature_names[1], feature_names[3], feature_names[8]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'invitation to dinner'"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "texts[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note:\n",
    "\n",
    "* The vector representation is equivalent to (contains the same information as) the bag-of-words feature representation\n",
    "    * (One can be mapped into the other using `feature_names`)\n",
    "* The vector representation does *not* allow us to reconstruct the original texts -- word order is lost\n",
    "* In practical applications, the vectors will be large (length of vocabulary) and mostly zeros, so efficient implementation requires sparse representations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Analysis and improvements\n",
    "\n",
    "We have now carefully processed our data and trained an SVM, but it is performing no better than Naive Bayes. We'll next look at some of the many things we can do to analyze and improve the machine learning-based text classifiers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hyperparameters\n",
    "\n",
    "* Machine learning methods learn their primary parameters (e.g. feature weights) in training\n",
    "* Most methods additionally have various *hyperparameters* that affect the model and training process\n",
    "\n",
    "Let's first look at a simple hyperparameter, class weights. This allows us to say e.g. that we care about the \"ham\" class more than the \"spam\" class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Defaults: precision 92.97%, recall 92.69%, f-score 92.83%\n",
      "Ham-2   : precision 88.64%, recall 95.95%, f-score 92.15%\n",
      "Ham-10  : precision 79.13%, recall 98.40%, f-score 87.72%\n"
     ]
    }
   ],
   "source": [
    "def prec_rec_f(gold, pred):\n",
    "    return precision(gold, pred), recall(gold, pred), f1_score(gold, pred)\n",
    "\n",
    "\n",
    "def eval_and_report(label, classifier, train_X, train_Y, test_X, test_Y):\n",
    "    classifier.fit(train_X, train_Y)\n",
    "    prec, rec, f = prec_rec_f(test_Y, classifier.predict(test_X))\n",
    "    print('{:8s}: precision {:.2%}, recall {:.2%}, f-score {:.2%}'.format(label, prec, rec, f))\n",
    "\n",
    "\n",
    "svm_classifier = LinearSVC(max_iter=10000)    # the default weight is 1 for all classes\n",
    "eval_and_report('Defaults', svm_classifier, train_X, train_classes, devel_X, devel_classes)\n",
    "\n",
    "svm_classifier = LinearSVC(max_iter=10000, class_weight={ 'ham': 2, 'spam': 0.5 })\n",
    "eval_and_report('Ham-2', svm_classifier, train_X, train_classes, devel_X, devel_classes)\n",
    "\n",
    "svm_classifier = LinearSVC(max_iter=10000, class_weight={ 'ham': 10, 'spam': 0.1 })\n",
    "eval_and_report('Ham-10', svm_classifier, train_X, train_classes, devel_X, devel_classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* The class weight acts as a multiplier for the cost of errors (points in the margin)\n",
    "    * Increasing one class weight without decreasing others would effecively increase C\n",
    "* The drop in $F_1$-score is expected: we're moving away from balanced precision and recall"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Precision-recall tradeoff\n",
    "\n",
    "* The class weighting above is one possible way to control the tradeoff beween precision and recall\n",
    "* Because SVM predicts class *scores*, we can pick our own threshold for predicting a class (default threshold is 0)\n",
    "    * It is trivial to reach 100% recall: just predict all positive (recall = tp/(tp+fn))\n",
    "    * It's *not* generally possible to assure 100% precision: all-negative doesn't work (precision = tp/(tp+fp) → 0/0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"figs/pr_curve.png\" width=\"25%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align:center; font-size:80%\">Idealized precision-recall curve</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot a precision-recall curve (note the use of `decision_function` instead of `predict`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEKCAYAAAARnO4WAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAHDNJREFUeJzt3Xt0HOWd5vHvry+SbMt3yRh8k43NxUDARHggCfcAxtmxd5LdBDY3ZjMwSYbMnAzJhIQ5mcSZHJKTGeacTJhwWZgMnBMuISdZJ8A6WTDhMjixvDYOtiMizEUyBst3+Sapu3/7R5WktixbjbqkbnU9n3N03FX99luvy/JTb79V9Za5OyIiUvkSpW6AiIiMDAW+iEhMKPBFRGJCgS8iEhMKfBGRmFDgi4jEhAJfRCQmFPgiIjGhwBcRiYlUqTZcV1fnDQ0Npdq8iMiotG7dup3uXj+Uz5Ys8BsaGmhqairV5kVERiUze2Oon9WQjohITCjwRURiQoEvIhITCnwRkZhQ4IuIxMSggW9m95vZDjN7+Tjvm5l938xazGyjmZ0ffTNFRKRYhfTwfwQsOcH71wILwp+bgB8W3ywREYnaoNfhu/uzZtZwgiLLgQc8eFbiGjObZGYnu/v2E9X7zv4j3PGr5nfV2Cilkwk+fuEcpoyrKlkbRERGUhQ3Xs0AWvOW28J1xwS+md1E8C2Aqunz+dfVLRFs/t3reYzv2/uPcEHDFDozWTozOTq7c3TncmSyTiabozsX/pl1Mj3rc0e/l8l6v9c5Tpk0hq//l4V0Z3s+E9aRdbqyuaBszo96vyvbV8eEMSmuOWs6Znbcv0O25/M5JxtuN5MN1mVzfdvM5oKfnPf8BJ/N5cLX3vM6vxxHfSabo6+M5322f5nwvWxYR9/rcH1YJp00Ll5QT84dd/K2Dd7zWc9rc46jtvOn557M+Jr0EP/tj217z3IyYdRWl+xeRJFhZ4U8xDzs4f/S3c8e4L1fAt9x9+fD5aeAr7j7CW+jbWxs9FLdadtxpJtFK35NJnfiv3s6aaQSCVJJI5UwUskE6fDPVNJI97zXu95Ys3V3JG2sSiWoG1dFdxjY+QeHTM4p4J+t5BIGyYSRsOAnmTAOdGYiqXv2lLFkc4770QeY/INHz2vvObD54PvtrFMmcObJE9h1oJObr1jAe+dMjqS9IlExs3Xu3jiUz0bRndkGzMpbnhmuK1vja9Ks/tJldBzJUJNOUJ1OUp1KUJ1KkE4mSCWCcDpRD/t4duw/wqpNb5NIDHRASJBOWu82epZTiQRVqeDP3Ye6uO+514L1ee8nE9a3LmEkw7qPPij1rUsmespZX+gmjKQZCSN4nQhfmx0TzL1lrOezx4Z3T5mEheUS5L0eeN8d7sry8lv7MPo+29MGO+oAAWZ92zcDM/jhM6+y73A36WQiKG99f7djDzAc/Z4F/6YDtT3rzg+feZWdBzp5bF0bAKub26lJJ8IDbnCkmD6hhqm1VeG3qOCA0pXJ0d7RyXmzJ3HJgrre9b1/Zp1seKA+1JWlfnw10yfUkE4lmDt1XO+3sbl1Y6lKJunO5Zg+oYZx+rYhEYuih/8h4GZgKfAnwPfdffFgdZayhy8ymGdfaWd18w7SyeBg23Gkmw2te6mrre7tEPQciHcd7OSFll1HfT5h9L6fShjJZHAA3Huo+1235eIFdYytSjK+Js0HzzyJsVVJzp01iQk1qSF1SmR0G9Yevpk9BFwG1JlZG/APQBrA3e8CniAI+xbgEPDnQ2mISDm55LR6Ljmt8AkJc7ng/EwyceJvOIe6MhzuyrL/SIbtew/3Dg82v93Re5DYuvMAmazTuucQz72yk63tB9m29zBA77ePHsmEMb++lpmTx9CVzdGVybFl+34+fuEcLpw3le5Mju7wnFPv62yOjs4Mp0wcw9iqJN3huZ/OTJbxNWkmjU0H6zI5MrkcqUSCC0+dSk0qQSqpW3dGs4J6+MNBPXyRwu052EXrnkPsOtDFxrZ97DzQyYbWvfxxRwezp4ylKhyOfKl1L4OcmiravLpxnDNzIt3ZHK27D3NBwxRe33WQ5eedwpknT6A7PPBMm1BD0ozubA4zqK1OkUxY78UEdbVV+oYyBMX08BX4IhVm36FuNm/fT1UqOF/U81OVTJAO1+3Y30lnJhusDw8W7R1969Lh+SPD+PWWd3hn3xE2tO6l+Z0Opk+ooSad4PVdhyJp7/mzJ3HlmScxa8pYcjln6TknU5XSN4njUeCLSEltemsfG1r3Mr4mTVUyweu7DjImnew9mORyzoHODNmcU5VK8Pjvt7O1/SA7D3QOWN/HGmcxe+pYlp5zMrMmj9FQUh4FvoiMWu0dnezoOMKR7hzfWLmJ32/bN2C5y0+v56QJNVSlEiw5ezrnzpwUyyuZFPgiUjHcnfaOTp5v2clvt+7m0XWtuMOYdJLD3dnecrXVKb5y7Rl8fPHs454kr0QKfBGJhYOdGTa07uXN3Yd4fON2nm/ZyQUNk7n9w+cwf9r4UjdvRCjwRSR23J3H1rXxtZ/9nu6sU1udoiuTY9HsSZwzYyINdeP42AWzSFfY+L8CX0Riq2XHAW796UZebT/AngFubHvwM4u5eEHh91SUOwW+iEie7fsOc/Udz9KRN3fTpy6aw8cumMVZp0wsYcuKp8AXERnAz9a38R//+QYbWvf2rrto3lSuPHMaV5wxjXn1tSVs3dAo8EVETuBId5af/r82bvvZsQ/uu+sT53PVwukkR8mVPgp8EZECdGdztOw4wAstO/nHx7cc8/4vv/ABzp5R3kM+CnwRkXfJ3Xm+ZSePNrXxi5fe6l1fnUrw3Y+8h+XnnVKWc/0o8EVEinTXb17lO0/+4ah18+rH8fQtl5WmQcdRTOBX1gWqIiJD9NlLT6Xl29fy08+9j9NPCm7i2tp+kIZbH+cnTa2DfHp0UA9fRGQAuw508tG7X+TV9oO9676y5Aw+d9mpJWyVevgiIpGbWlvNU7dcxsM3Xdi77rv/5w98+/HNJWxVcRT4IiIncOG8qbz+nQ/x/FcuB+De516jZUdHiVs1NAp8EZECzJw8lm8uOwuAD97xLP+0qplMNlfiVr07CnwRkQJ9+n0N/NmiGQD8YHUL8297kkUrfsWLr+4a5JPlQYEvIvIu/MvHzuPZL1/O31y5gJp0gj2Hurn+3jXc/ZtXS920QSnwRUTepdlTx/LFq07jD9+6lhsvngvA7U/+oex7+gp8EZEi3Pahhdz9yfcCcP29a2h+u3xP6CrwRUSKdM1Z07n9w+cAcM+zW0vcmuNT4IuIROD6xbP59EVz+N8btrFt7+FSN2dACnwRkYjcdGlwF+69ZdrLV+CLiERkxqQx/NdFM3h47ZvsOtBZ6uYcQ4EvIhKhz156Kp2ZHP/+wuulbsoxFPgiIhGaP62WaxZO50f/+TodR459qHopKfBFRCL2+ctP5UBnhu+tai51U46iwBcRidh7Zk4C4IEX3yCbK80U9AMpKPDNbImZNZtZi5ndOsD7c8zsKTPbaGbPmNnM6JsqIjJ6zJ9WC8CpX3uChlsf55nmHSVuUQGBb2ZJ4E7gWmAhcL2ZLexX7J+AB9z9PcAK4PaoGyoiMpo89tmLuOask1g0O+jt3/Dva1nxi80lnWGzkB7+YqDF3be6exfwMLC8X5mFwNPh69UDvC8iEiuTxlZx9ycb+dnn389fXjIPgPtfeI3L//mZkrWpkMCfAeQ/0LEtXJfvJeDD4es/A8ab2dTimyciMvp9demZvPzNa0gnjdbdh/nlxrdK0o6oTtp+CbjUzNYDlwLbgGz/QmZ2k5k1mVlTe3t7RJsWESl/tdUpnvu7KwC4+cfr+d1ru0e8DYUE/jZgVt7yzHBdL3d/y90/7O6LgNvCdXv7V+Tu97h7o7s31tfXF9FsEZHRZ/rEGm5beiYAf/Pw+hHffiGBvxZYYGZzzawKuA5YmV/AzOrMrKeurwL3R9tMEZHKcOMl86itTrF935ER3/agge/uGeBmYBWwBXjU3TeZ2QozWxYWuwxoNrNXgJOAbw9Te0VERr3z50wG4OVt+0Z0u+ZempsCGhsbvampqSTbFhEppef+2M4n7/sdAL/72pVMm1BT8GfNbJ27Nw5lu7rTVkRkhH1gfh3Tw5D/wHdXj9h2FfgiIiPMzFjztSsB6MrmeGf/yIznK/BFRErkjo+eC8CTv98+IttT4IuIlMjlp08D4KHftQ5SMhoKfBGREpk8rorFDVNofqeDF1p2Dvv2FPgiIiX01aVnAPCPj28Z9m0p8EVESmjR7OCa/Lf3HR72bSnwRURKbHHDFPYc6mbfoeF9JKICX0SkxP7qivkAfOvxzcO6HQW+iEiJvf/UYDb5x9a18cjaN4dtOwp8EZESSyUT3H9DMFvCz9cP31z5CnwRkTJwxRkn8d45k3lx665h24YCX0SkTKQSBsADL74+LPUr8EVEysQP/sf5ANz73NZhqV+BLyJSJurHVzNxTJqp46qHpX4FvohIGVk8dwqv7Tw4LHUr8EVEykgqYew73E3HkehvwlLgi4iUkUtPqwegvaMz8roV+CIiZWTyuCoAnnz57cjrVuCLiJSRSxYEPfzvrWqOvO5U5DWKiMiQjalKMn9aLfsOawxfRKTiXdAwBffo61Xgi4iUmepUgp0HdNJWRKTi7T3UBcChrkyk9SrwRUTKzLmzJgFwuCsbab0KfBGRMlOTTgLQnY12IF+BLyJSZtLJIJq7s7lI61Xgi4iUmZ5pkrfvOxJpvQp8EZEyM3FsGoADndFei6/AFxEpM7MmjwHgrmeinRe/oMA3syVm1mxmLWZ26wDvzzaz1Wa23sw2mtnSSFspIhIjc6aOA6A6HW2ffNDazCwJ3AlcCywErjezhf2K/T3wqLsvAq4D/i3SVoqIxEg6meDiBXV0HBn56/AXAy3uvtXdu4CHgeX9yjgwIXw9ERi+x66LiMTA+JoUBzqjDfxCJk+bAbTmLbcBf9KvzDeAX5nZF4BxwAcjaZ2ISEyNr06zP+IJ1KIaILoe+JG7zwSWAg+a2TF1m9lNZtZkZk3t7e0RbVpEpPKMrU6yo6OTzkx0d9sWEvjbgFl5yzPDdfk+AzwK4O4vAjVAXf+K3P0ed29098b6+vqhtVhEJAbGVQUDMHsORtfLLyTw1wILzGyumVURnJRd2a/Mm8CVAGZ2JkHgqwsvIjJEC06qBeDN3Yciq3PQwHf3DHAzsArYQnA1ziYzW2Fmy8JitwA3mtlLwEPADe7DMZuziEg8TAkfdfjGroOR1VnQE6/c/QngiX7rvp73ejPw/shaJSISc2edMhGAP+44EFmdutNWRKQM9fTwE2aR1anAFxEpY8+3RHc6VIEvIlKmUglj+oQxkdWnwBcRKVOnTx9PlNe/KPBFRMpUMmFkFfgiIpUvYUY2p8AXEal4yYQR5R1NCnwRkTKVMNTDFxGJg4RpDF9EJBaSCSOnHr6ISOUzg6Y39kRWnwJfRKRMHeiMbi58UOCLiJSthScHT46NasZMBb6ISJm6cN4UAA5G1NNX4IuIlKnqVDLS+hT4IiIxocAXEYkJBb6ISEwo8EVEYkKBLyJS5pxo7rZV4IuIlKlcOI/O+jf3RlKfAl9EpEydfcpEIHjUYRQU+CIiZao6HUR0VPOnKfBFRMqUhR37XERTJCvwRUTKVCJM/KgeZK7AFxEpUz2BryEdEZEKl9CQjohIPJh6+CIi8dDTw9cYvohIhesbwx/BwDezJWbWbGYtZnbrAO//i5ltCH9eMbNobgsTEYmxqE/apgYrYGZJ4E7gKqANWGtmK919c08Zd/9iXvkvAIuiaZ6ISHz1XIefjSjxC+nhLwZa3H2ru3cBDwPLT1D+euChKBonIhJnyXAQ/3DXyD3icAbQmrfcFq47hpnNAeYCTxffNBGReEuGXfwt2/dHUl/UJ22vAx5z9wEPR2Z2k5k1mVlTe3t7xJsWEaksibCHP3lcVTT1FVBmGzArb3lmuG4g13GC4Rx3v8fdG929sb6+vvBWiojE1Ny6cXRlcpHUVUjgrwUWmNlcM6siCPWV/QuZ2RnAZODFSFomIiIY8NSWdyKpa9DAd/cMcDOwCtgCPOrum8xshZktyyt6HfCwR3WHgIiIkMk5B7uykdx8NehlmQDu/gTwRL91X++3/I2iWyMiIkdZePIE3tx9iPaOTqZNqCmqLt1pKyJSxq44YxoAXdnix/EV+CIiZaznWvwobr5S4IuIlLFUMgj87qwCX0SkoqWTQUxnchrSERGpaD0TqG1s21d8XUXXICIiw6axYTIAL29T4IuIVLS62moAxlQli65LgS8iUuaqUgk6jmSKrkeBLyJS5gz46bq2outR4IuIlLmEGZ0RTKCmwBcRKXNXn3USVani41qBLyJS5iaPraJagS8iUvlSCdPUCiIicZBMGhkFvohI5VMPX0QkJpKJBNmckysy9BX4IiJl7lBncNPV9v1HiqpHgS8iUubOmjEBgEyRD0FR4IuIlLmeKZK7irz5SoEvIlLmegJ/98GuoupR4IuIlLkJNWkANrTuLaoeBb6ISJk7OxzD73m+7VAp8EVEylx1KpgL/2Bntqh6FPgiImUuHT7IfM3WXUXVo8AXESlzFj7X9pRJY4qqR4EvIjIKzJw8BnfdaSsiUvGSCSOrwBcRqXxJK34CNQW+iMgokEgYOfXwRUQqn3r4IiIxkUgYRc6dVljgm9kSM2s2sxYzu/U4ZT5qZpvNbJOZ/bi4ZomISL5kgqKv0kkNVsDMksCdwFVAG7DWzFa6++a8MguArwLvd/c9ZjatqFaJiMhRujI5VjfvKKqOQnr4i4EWd9/q7l3Aw8DyfmVuBO509z0A7l5cq0RE5CiZrFM/vrqoOgoJ/BlAa95yW7gu32nAaWb2gpmtMbMlA1VkZjeZWZOZNbW3tw+txSIiMXTOzIm8s7+zqDqiOmmbAhYAlwHXA/ea2aT+hdz9HndvdPfG+vr6iDYtIlL5xlUPOgI/qEICfxswK295ZrguXxuw0t273f014BWCA4CIiERgRpHz6EBhgb8WWGBmc82sCrgOWNmvzM8JeveYWR3BEM/WolsnIiIAWHFT4QMFBL67Z4CbgVXAFuBRd99kZivMbFlYbBWwy8w2A6uBL7t7cfN4iohIL6P4xC9oUMjdnwCe6Lfu63mvHfjb8EdERCI2Ij18EREpvSKfbhjUUXwVIiIy3KIY0lHgi4iMAhrSERGJCYsg8RX4IiKjQAQdfAW+iMhooCEdEZGYUA9fRCQmEhFcl6nAFxEZBdTDFxGJC12lIyISD7rTVkQkJnSnrYhITOiyTBGRmNBJWxGRmOjOedF1KPBFREaBhqlji65DgS8iMgokdVmmiEg86E5bEZGYSKiHLyISD8kI0lqBLyIyCqiHLyISE0mN4YuIxIN6+CIiMVGTThZdhwJfRGQUqKutKroOBb6IyChgGtIREYkHzYcvIhITukpHRCQmdJWOiEhMjNgDUMxsiZk1m1mLmd06wPs3mFm7mW0If/6i+KaJiEiPKHr4qcEKmFkSuBO4CmgD1prZSnff3K/oI+5+c9EtEhGRY4zU9MiLgRZ33+ruXcDDwPKitywiIgUbqSGdGUBr3nJbuK6/j5jZRjN7zMxmFd80ERHpYWZFh35UJ21/ATS4+3uAXwP/MVAhM7vJzJrMrKm9vT2iTYuIxIMX+VjbQgJ/G5DfY58ZrstrhO9y985w8X8B7x2oIne/x90b3b2xvr5+KO0VEYmtW689o6jPFxL4a4EFZjbXzKqA64CV+QXM7OS8xWXAlqJaJSIix/jspacW9flBr9Jx94yZ3QysApLA/e6+ycxWAE3uvhL4azNbBmSA3cANRbVKREQiZ17soNAQNTY2elNTU0m2LSIyWpnZOndvHMpndaetiEhMKPBFRGJCgS8iEhMKfBGRmFDgi4jEhAJfRCQmSnZZppl1AM0l2Xj5qQN2lroRZUL7oo/2RR/tiz6nu/v4oXxw0BuvhlHzUK8lrTRm1qR9EdC+6KN90Uf7oo+ZDfkGJg3piIjEhAJfRCQmShn495Rw2+VG+6KP9kUf7Ys+2hd9hrwvSnbSVkRERpaGdEREYmLYA9/MlphZs5m1mNmtA7xfbWaPhO//1swahrtNpVLAvvhbM9scPiryKTObU4p2joTB9kVeuY+YmZtZxV6hUci+MLOPhr8bm8zsxyPdxpFSwP+R2Wa22szWh/9PlpaincPNzO43sx1m9vJx3jcz+364nzaa2fkFVezuw/ZDMH/+q8A8oAp4CVjYr8zngbvC19cBjwxnm0r1U+C+uBwYG77+XJz3RVhuPPAssAZoLHW7S/h7sQBYD0wOl6eVut0l3Bf3AJ8LXy8EXi91u4dpX1wCnA+8fJz3lwJPAgZcCPy2kHqHu4e/GGhx963u3gU8DCzvV2Y5fc/AfQy40iyK57OXnUH3hbuvdvdD4eIagsdJVqJCfi8AvgV8Fzgyko0bYYXsixuBO919D4C77xjhNo6UQvaFAxPC1xOBt0awfSPG3Z8leJjU8SwHHvDAGmBSvycPDmi4A38G0Jq33BauG7CMu2eAfcDUYW5XKRSyL/J9huAIXokG3RfhV9RZ7v74SDasBAr5vTgNOM3MXjCzNWa2ZMRaN7IK2RffAD5hZm3AE8AXRqZpZefd5glQ2jtt5TjM7BNAI3BpqdtSCmaWAO5Aj8rskSIY1rmM4Fvfs2Z2jrvvLWmrSuN64Efu/s9mdhHwoJmd7e65UjdsNBjuHv42YFbe8sxw3YBlzCxF8DVt1zC3qxQK2ReY2QeB24Bl7t45Qm0baYPti/HA2cAzZvY6wRjlygo9cVvI70UbsNLdu939NeAVggNApSlkX3wGeBTA3V8Eagjm2YmbgvKkv+EO/LXAAjOba2ZVBCdlV/YrsxL4dPj6vwFPe3hWosIMui/MbBFwN0HYV+o4LQyyL9x9n7vXuXuDuzcQnM9Y5u6V+BDkQv6P/Jygd4+Z1REM8WwdyUaOkEL2xZvAlQBmdiZB4LePaCvLw0rgU+HVOhcC+9x9+2AfGtYhHXfPmNnNwCqCM/D3u/smM1sBNLn7SuA+gq9lLQQnKa4bzjaVSoH74ntALfCT8Lz1m+6+rGSNHiYF7otYKHBfrAKuNrPNQBb4srtX3LfgAvfFLcC9ZvZFghO4N1RiB9HMHiI4yNeF5yv+AUgDuPtdBOcvlgItwCHgzwuqtwL3lYiIDEB32oqIxIQCX0QkJhT4IiIxocAXEYkJBb6ISEwo8KVimVnWzDaY2ctm9gszmxRx/TeY2Q/C198wsy9FWb9I1BT4UskOu/t57n42wT0ef1XqBomUkgJf4uJF8iaXMrMvm9nacC7xb+at/1S47iUzezBc96fhsxrWm9n/NbOTStB+kaJp8jSpeGaWJLgd/75w+WqCuWgWE8wnvtLMLiGYw+nvgfe5+04zmxJW8Txwobu7mf0F8HcEd3yKjCoKfKlkY8xsA0HPfgvw63D91eHP+nC5luAAcC7wE3ffCeDuPfORzwQeCecbrwJeG5nmi0RLQzpSyQ67+3nAHIKefM8YvgG3h+P757n7fHe/7wT1/CvwA3c/B/hLggm7REYdBb5UvPApYn8N3BJOwb0K+J9mVgtgZjPMbBrwNPDfzWxquL5nSGcifVPPfhqRUUpDOhIL7r7ezDYC17v7g+HUui+Gs5IeAD4Rzsz4beA3ZpYlGPK5geApSz8xsz0EB4W5pfg7iBRLs2WKiMSEhnRERGJCgS8iEhMKfBGRmFDgi4jEhAJfRCQmFPgiIjGhwBcRiQkFvohITPx/48vjxeVPxtkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "from sklearn.metrics import precision_recall_curve\n",
    "\n",
    "\n",
    "svm_classifier = LinearSVC(max_iter=10000)\n",
    "svm_classifier.fit(train_X, train_classes)\n",
    "pred_scores = svm_classifier.decision_function(devel_X)\n",
    "\n",
    "if svm_classifier.classes_[1] == 'spam':    \n",
    "    pred_scores = -pred_scores    # classifier got the positive wrong, so negate scores\n",
    "    \n",
    "precisions, recalls, thresholds = precision_recall_curve(devel_texts[\"class\"], pred_scores, pos_label=\"ham\")\n",
    "\n",
    "df = DataFrame(zip(precisions, recalls), columns=['Precision', 'Recall'])\n",
    "df.plot(x='Recall', y='Precision', legend=None) ;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sidenote: output probabilities\n",
    "\n",
    "SVM is not inherently probabilistic (i.e. its predictions are not probabilities), but there is a probabilistic formulation ([Platt 1999](https://www.researchgate.net/profile/John_Platt/publication/2594015_Probabilistic_Outputs_for_Support_Vector_Machines_and_Comparisons_to_Regularized_Likelihood_Methods/links/004635154cff5262d6000000.pdf)). The implementations we could easily use here are pretty slow, so won't be running them, but it would look something like this:\n",
    "\n",
    "<pre><code>\n",
    "from sklearn.svm import SVC\n",
    "\n",
    "[...]\n",
    "\n",
    "train_X = vectorizer.fit_transform(train_texts['text'])\n",
    "bigram_classifier = <b>SVC</b>(kernel='linear', <b>probability=True</b>)\n",
    "bigram_classifier.fit(train_X, train_texts['class'])\n",
    "\n",
    "devel_X = bigram_vectorizer.transform(devel_texts['text'])\n",
    "pred_probs = bigram_classifier.<b>predict_proba</b>(devel_X)\n",
    "</code></pre>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Regularization, overfitting and underfitting\n",
    "\n",
    "Among the most important hyperparameters are those controlling *regularization* (broadly, techiques that constrain a model to avoid overfitting)\n",
    "\n",
    "Bias-variance tradeoff:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"figs/bias-variance.png\" width=\"60%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align:center; font-size:80%\">Illustration of bias and variance.</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* A simple model may generalize better even if it doesn't fit the training data as well as a complex one\n",
    "* ... but *too* simple a model won't even be able to fit training data\n",
    "\n",
    "Recall that the SVM parameter C controls the cost of errors (data points in the margin).\n",
    "\n",
    "Let's vary the regularization parameter C and measure performance on the training and development."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "C=  0.001, train F-score: 87.63%, dev F-score: 86.77%\n",
      "C=  0.010, train F-score: 93.54%, dev F-score: 91.64%\n",
      "C=  0.100, train F-score: 97.13%, dev F-score: 93.12%\n",
      "C=  1.000, train F-score: 98.79%, dev F-score: 92.83%\n",
      "C= 10.000, train F-score: 99.06%, dev F-score: 91.77%\n",
      "C=100.000, train F-score: 99.17%, dev F-score: 90.00%\n"
     ]
    }
   ],
   "source": [
    "results = []\n",
    "for C in (0.001, 0.01, 0.1, 1, 10, 100):\n",
    "    svm_classifier = LinearSVC(C=C, max_iter=100000)\n",
    "    svm_classifier.fit(train_X, train_classes)\n",
    "    train_f = f1_score(train_classes, svm_classifier.predict(train_X))\n",
    "    devel_f = f1_score(devel_classes, svm_classifier.predict(devel_X))\n",
    "    print('C={:7.3f}, train F-score: {:.2%}, dev F-score: {:.2%}'.format(C, train_f, devel_f))\n",
    "    results.append({ 'C': C, 'Train F': train_f, 'Devel F': devel_f })"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(Note log scale for C and how training gets slower with higher values of C)\n",
    "\n",
    "Plot the results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAEOCAYAAABrSnsUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xd8VFX+//HXJz2QAqTQQu+hQ0AREbCgKIoiQgBRFERXsSH+Fnb97lrWVXfXVVxdFRUFC6EogkoRURasJEDovSWhhhZa+pzfH3cgIQKZJJPcmeTzfDzm4cydc+985hrue+65954rxhiUUkpVbT52F6CUUsp+GgZKKaU0DJRSSmkYKKWUQsNAKaUUGgZKKaXQMFBKKYWGgVJKKTQMlFJKoWGglFIK8LO7gKIiIyNN48aN7S5DKaW8yqpVq44YY6JKO7/HhUHjxo1JSkqyuwyllPIqIrK3LPNrN5FSSikNA6WUUhoGSiml8MBjBheTm5tLWloaWVlZdpfisYKCgoiJicHf39/uUpRSXsgrwiAtLY3Q0FAaN26MiNhdjscxxnD06FHS0tJo0qSJ3eUopbyQV3QTZWVlERERoUFwCSJCRESE7jkppUrNK/YMAA2CYuj6Uaryyct3kJPvICfPemTnXfi68POy8powsNPRo0e57rrrADh48CC+vr5ERVnXdqxcuZKAgIBil3HfffcxceJEWrVq5dJnvv/++0yaNIn69esD0LlzZz788MNSfgOlVHGMMZfc0P5uI1z0/d+9l39Bu+xzy7jE/OeeW23yz792VOAt6jUMXBAREUFycjIAzz77LCEhIUyYMOGCNsYYjDH4+Fy85600G/IRI0bw+uuvl7xgpSqRrNx8TmbmcjIrl4zMXE5m5ln/zcrlVFbehRvZIhvhnMIb4Uts6AtvzN3F10cI8PUhwM/58PUh0K/gdaCfD0H+PoQF+Tmn+Z5vH1honoCLPA+8xHtdXilbzRoGZbBjxw5uu+02OnfuzJo1a1iyZAnPPfccq1evJjMzk6FDh/KXv/wFgKuvvpo333yTdu3aERkZyUMPPcTChQupVq0a8+bNIzo62uZvo1T5cDgMp7LzOJl5bmN+8Q37ufes19b0jMxclzbS/r5SaGPqe9GNaEigHwHVLr6BDfDzIfB3G1jfCzfARdsU3Uj7FrT39fG+bluvC4PnvtrIpv0n3brM2Hph/PXWtqWad8uWLUyfPp24uDgAXn75ZWrVqkVeXh59+/Zl8ODBxMbGXjBPRkYGvXv35uWXX2b8+PFMnTqViRMn/m7Zn376KcuWLQNg/Pjx3HPPPaWqUamyys7L/93G+2ShjfdFN+xZuWSczeVUdh7mMt0dPgKhQf6EB1uPsGA/6oQHERZ07rXzEeR3/vW5tiGBfgT4+uDjhRtfT+N1YeBpmjVrdj4IAGbMmMEHH3xAXl4e+/fvZ9OmTb8Lg+DgYPr37w9A165dWbFixUWXrd1Eyl0cDsPpnDwyzl74q/xkMb/Kz03LLubXudXlUbDxrh0WRIvokEIb+HMb9IIN/rnpIQF+ujH3AF4XBqX9BV9eqlevfv759u3bmTx5MitXrqRGjRrcfffdFz3ds/ABZ19fX/Ly8iqkVuX9HA7D3mNnOX42p0i3S16RDfqFv9ZPZeVe9mCkCIQG+hFezbmxDvKnRXSItfGuduGv8sIb9XMb9kA/34pbCapceF0YeLKTJ08SGhpKWFgYBw4cYPHixdx00012l6UqgcMns5i9Ko2ZiamkHDt70TaBfj7nu1DCgvyICgmkWVTI+Y174Y33hb/S/QkN1F/nVZ2GgRt16dKF2NhYWrduTaNGjejZs6fdJSkvlu8wLNt6mBkrU/lh62HyHYYrm9biod7NqBsedEFXS1iQP0H++utclZ6Yyx3ZsUFcXJwpej+DzZs306ZNG5sq8h66niqHtONnmZWYyqykNA6ezCIyJIA7u8YQ360hTSKrF78AVSWJyCpjTFzxLS9O9wyU8gC5+Q6+23SIGYmprNieDkCvFlH89dZYrmtTmwA/rxg5RnkxDQOlbLT7yBkSElP4fFUaR07nUDc8iEevbcFdXWNoUKua3eWpKkTDQKkKlpWbz6INB5mxMoXfdh/D10e4tnU0w7o3oHfLaK+8YEl5Pw0DpSrI1oOnmLEyhblr9pGRmUuDWsE8fWMrBneNoXZYkN3lqSpOw0CpcnQ2J4+v1x5gRmIKa1JO4O8r3Ni2DvHdGnJVswg9nVN5DA0DpcrB+rQMZiSmMD95P6ez82gWVZ1nbmnDoC4x1Kpe/Ci3SlU0l8JARG4CJgO+wPvGmJeLvN8ImApEAceAu40xac73/gHcgnUjnSXA48bTzmd1ga+vL+3btyc3Nxc/Pz/uuecennzyyUuOUloae/bsYcCAAWzYsOF309u0aXPB8NeuDp2tKs7JrFzmJe8nYWUKG/efJNDPh1s61GVY94bENaqp95xQHq3YMBARX+At4AYgDUgUkfnGmE2Fmv0LmG6MmSYi1wIvASNF5CqgJ9DB2e5HoDewzH1foWIEBwefH8b68OHDDB8+nJMnT/Lcc89VyOc3a9bs/Ocrz2GMYdXe48xYmco36/eTleugTd0wnh/YloGd6hMerPekVt7BlZ+13YEdxphdxpgcIAEYWKRNLPC98/kPhd43QBAQAAQC/sChshZtt+joaKZMmcKbb76JMYb8/HyefvppunXrRocOHXj33XcBiI+P55tvvjk/36hRo5gzZ84l2yvvcfxMDu+v2EW/15Yz+J1fWLThAHd0rs+8R3qy4LGruadHYw0C5VVc6SaqD6QWep0GXFGkzVpgEFZX0h1AqIhEGGN+EZEfgAOAAG8aYzaXqeKFE+Hg+jIt4nfqtIf+LxffrpCmTZuSn5/P4cOHmTdvHuHh4SQmJpKdnU3Pnj3p168fQ4cOZdasWdxyyy3k5OSwdOlS3n77bT744IOLtr9cN8LOnTvp1KkTAD179uStt94q01dWJedwGH7ddZQZiaks3nCQnHwHnRrU4JU72zOgQz2qB+ohOOW93PXXOwF4U0RGAcuBfUC+iDQH2gAxznZLRKSXMeaCMZtFZCwwFqBhw4ZuKqnifPvtt6xbt445c+YA1v0Ktm/fTv/+/Xn88cfJzs5m0aJFXHPNNQQHB1+yfcuWLS/5GdpNZJ/Dp7KY4xwkbu/Rs4QF+TH8ioYM7daANnXD7C5PKbdwJQz2AQ0KvY5xTjvPGLMfa88AEQkB7jTGnBCRB4BfjTGnne8tBHoAK4rMPwWYAtbYRJetpoS/4MvLrl278PX1JTo6GmMM//nPf7jxxht/165Pnz4sXryYmTNnEh8fD3DJ9nv27KmI0pUL8h2G5dvSmbEyhaVbrEHiujepxRPXt6B/u7o6KJyqdFw5ZpAItBCRJiISAMQD8ws3EJFIETm3rElYZxYBpAC9RcRPRPyxDh6XrZvIA6Snp/PQQw8xbtw4RIQbb7yRt99+m9zcXAC2bdvGmTNnABg6dCgffvghK1asOD+c9eXaK3vtO5HJa0u20euV77nvo0RW7T3O6KubsPSp3sx6sAd3dI7RIFCVUrF7BsaYPBEZByzGOrV0qjFmo4g8DyQZY+YDfYCXRMRgdRM94px9DnAtsB7rYPIiY8xX7v8a5S8zM5NOnTqdP7V05MiRjB8/HoAxY8awZ88eunTpgjGGqKgovvzySwD69evHyJEjGThw4PlTQS/XXlW83HwHSzcfJiExhf9tswaJu7p5JM8MiOV6HSROVRE6hHUlouupZPYePUNCYipzVqWRfiqb2mGBDIlrwJC4BjpInPI6OoS1UiWQnZfP4o2HSFiZws87j+IjcG3raOK7NaRPqyj8fHUvQFVNGgaqSth+6BQzVqbyxZo0TpzNJaZmMBP6tWRw1wbUCddB4pTSMFCVVmZOPl+v209CYiqr9h7H31foF1uH+O4N6NksUgeJU6oQrwkDY4yO7XIZnnbsx04b9mWQkJjCvDX7OZWdR9PI6vzp5tYM6hJDZEig3eUp5ZG8IgyCgoI4evQoERERGggXYYzh6NGjBAVV3e6OU1m5zF+7n4SVqazfl0Ggnw83t69LfLcGdG9SS/9ulCqGV4RBTEwMaWlppKen212KxwoKCiImJqb4hpWIMYbVKSdIWJnC1+sOkJmbT+s6oTx3W1tu71Sf8Go6NpBSrvKKMPD396dJkyZ2l6E8xImzOXyxeh8zE1PZeugU1QJ8GdipHvHdG9IxJlz3ApQqBa8IA6WMMfy66xgJiSks3HCQnDwHHWPCeWlQe27tWI8QHSROqTLRf0HK4/284wh//nIDu4+cITTIj/huDYjv1pDYejpInFLuomGgPNrMxBT+PHcDjSKq8e8hHbm5vQ4Sp1R50DBQHsnhMPxj8Vbe+d9OereM4s3hnQkN0gPCSpUXDQPlcTJz8hk/K5mFGw5y95UNefbWtjpMhFLlTMNAeZTDp7J4YFoS6/Zl8H8DYrm/Z2M9O0ipCqBhoDzGloMnGf1REsfO5DBlZBw3xNa2uySlqgwNA+URlm09zLjP1lA90JfZD/WgXf1wu0tSqkrRMFC2+/iXPfx1/kZa1wnjg1Fx1A0PtrskpaocDQNlm3yH4cVvNjP1p91c3yaayfGdqa4XjyllC/2Xp2xxJjuPxxPW8N3mw9zfswl/vqUNvjqktFK20TBQFe5ARiajP0piy8GTvDCwLSN7NLa7JKWqPA0DVaE27Mtg9LREzmTnM3VUN/q0ira7JKUUGgaqAi3ZdIjHZqyhVvUA5vyhO63r6NhCSnkKDQNV7owxfPDjbl5csJkO9cN57944okOr7o14lPJEGgaqXOXlO3j2q4188msK/dvV4d9DOhEcoAPNKeVpXBrwRURuEpGtIrJDRCZe5P1GIrJURNaJyDIRiSn0XkMR+VZENovIJhFp7L7ylSc7lZXL/dOS+OTXFB7q3Yy3hnfRIFDKQxW7ZyAivsBbwA1AGpAoIvONMZsKNfsXMN0YM01ErgVeAkY635sOvGiMWSIiIYDDrd9AeaS042cZ/VESO9NP8/Kg9sR3b2h3SUqpy3Clm6g7sMMYswtARBKAgUDhMIgFxjuf/wB86WwbC/gZY5YAGGNOu6lu5cGSU08wZloS2Xn5TLu/Oz2bR9pdklKqGK50E9UHUgu9TnNOK2wtMMj5/A4gVEQigJbACRH5QkTWiMg/nXsaqpJasP4AQ9/9heAAH+Y+fJUGgVJewl2DxE8AeovIGqA3sA/Ix9rz6OV8vxvQFBhVdGYRGSsiSSKSlJ6e7qaSVEUyxvD2sp08/Olq2tUP58uHe9I8OtTuspRSLnIlDPYBDQq9jnFOO88Ys98YM8gY0xn4s3PaCay9iGRjzC5jTB5W91GXoh9gjJlijIkzxsRFRUWV8qsou+TkOZj4+XpeWbSFWzvW49MxVxAREmh3WUqpEnDlmEEi0EJEmmCFQDwwvHADEYkEjhljHMAkYGqheWuISJQxJh24FkhyV/HKfhlnc/nDp6v4eedRHru2OU9c3xIfHWNIKa9TbBgYY/JEZBywGPAFphpjNorI80CSMWY+0Ad4SUQMsBx4xDlvvohMAJaKdbuqVcB75fNVVEXbe/QM932USOqxs7x6V0fu7BpT/ExKKY8kxhi7a7hAXFycSUrSnQdPl7TnGGM/XoXDGN69uytXNI2wuySlqjQRWWWMiSvt/HoFsiqxecn7eHr2OurXDGbqqG40iaxud0lKqTLSMFAuM8bwxtIdvPbdNro3qcW7d3elZvUAu8tSSrmBhoFySXZePhM/X8/cNfsY1KU+Lw1qT6CfXjKiVGWhYaCKdexMDg9+nETinuNM6NeSR/o2xzofQClVWWgYqMvamX6a+z9K5EBGFm8M68xtHevZXZJSqhxoGKhL+mXnUR76ZBV+PsKMB66ka6OadpeklConGgbqomYnpfKnuetpFFGdqfd2o2FENbtLUkqVIw0DdQGHw/Dqkq289cNOejaP4L8juhIe7G93WUqpcqZhoM7Lys1nwuy1fL3uAPHdGvDC7e3w93XXWIZKKU+mYaAAOHI6mwemJ5GceoJJ/Vsz9pqmesaQUlWIhoFi+6FT3PdRIkdOZ/P2iC7c1K6u3SUppSqYhkEVt2J7Og9/spqgAF9mju1BxwY17C5JKWUDDYMqbMbKFJ75cgMtokP4YFQ36tcItrskpZRNNAyqIIfD8PKiLUxZvoveLaN4c3hnQoP0jCGlqjINgyrmbE4eTyQk8+2mQ4y8shF/vTUWPz1jSKkqT8OgCjl0Mosx05LYsD+DvwyI5b6ejfWMIaUUoGFQZWzaf5LR0xLJyMzlvZFxXB9b2+6SlFIeRMOgCvhhy2HGfbaa0CB/Zj3Yg3b1w+0uSSnlYTQMKrlpP+/hua820qZuGB/c24064UF2l6SU8kAaBpVUvsPwwteb+OjnPVzfpjaT4ztRPVD/dyulLk63DpXQ6ew8Hpuxhu+3HGbM1U2YdHMbfH30QLFS6tI0DCqZ/ScyGT0tiW2HTvHC7e0YeWUju0tSSnkBDYNKZH1aBqOnJXI2J5+po7rRu2WU3SUppbyES1cbichNIrJVRHaIyMSLvN9IRJaKyDoRWSYiMUXeDxORNBF5012Fqwst3niQIe/+gr+vD5//4SoNAqVUiRQbBiLiC7wF9AdigWEiEluk2b+A6caYDsDzwEtF3n8BWF72clVRxhjeW76Lhz5ZRcs6ocx95Cpa1Qm1uyyllJdxZc+gO7DDGLPLGJMDJAADi7SJBb53Pv+h8Psi0hWoDXxb9nJVYbn5Dv785QZeXLCZ/u3qkPDAlUSH6qmjSqmScyUM6gOphV6nOacVthYY5Hx+BxAqIhEi4gO8Ckwoa6HqQiezcrn/o0Q++y2FP/RpxpvDuhAc4Gt3WUopL+WuA8gTgDdFZBRWd9A+IB94GFhgjEm73Bg4IjIWGAvQsGFDN5VUeaUeO8v9HyWy+8gZ/nFnB4Z0a2B3SUopL+dKGOwDCm9tYpzTzjPG7Me5ZyAiIcCdxpgTItID6CUiDwMhQICInDbGTCwy/xRgCkBcXJwp7ZepClanHGfs9CRy8hxMH92dq5pF2l2SUqoScCUMEoEWItIEKwTigeGFG4hIJHDMGOMAJgFTAYwxIwq1GQXEFQ0C5bqv1+3nqVlrqR0WRMLYbjSPDrG7JKVUJVHsMQNjTB4wDlgMbAZmGWM2isjzInKbs1kfYKuIbMM6WPxiOdVbJRljeOuHHYz7bA3t6ocz9+GrNAiUUm4lxnhWr0xcXJxJSkqyuwyPkZPn4E9z1zNnVRq3dazHPwZ3IMhfDxQrpS4kIquMMXGlnV+vQPZgJ87m8ODHq/ht9zEeu64FT17fQm9Go5QqFxoGHupARiYj3vuNtOOZvDa0I3d0jil+JqWUKiUNAw+U7zA8npDMoZNZfDLmCro3qWV3SUqpSk7DwAO987+drNx9jH/d1VGDQClVIVwaqE5VnLWpJ3htyTZu6VCXO7sUvdBbKaXKh4aBBzmTnccTM5OJCg3k77e314PFSqkKo91EHuRv32xiz9EzfDbmSsKr+dtdjlKqCtE9Aw+xaMNBZqxM5cFrmtGjWYTd5SilqhjdM/AAh05mMfGLdbSrH8b4G1raXY7nMAbSkmD9bEj5BQJDIagGBNeE4HP/rXnx14FhoN1sSrlMw8BmDofhqVlrycrNZ3J8ZwL8dGeNw1usAFg/G07sBd9AaNQD8vPg+B7YvwayTkDu2UsvQ3whKLxIYFwiOIJrXhgyvtpFp6oeDQObTf1pNz/uOMLf72hPs6gqPN7QiRTY8Dms/xwOrQfxgaZ9oM9EaD0AgsJ+P09ulhUKmcedj8LPCz2yTsDZI3B0u/N1xuVrCQgtFBqXC44iAeNfTfdGlNfSMLDRxv0Z/GPRVm6Irc2w7lXwngRnjsKmubB+jtUNBBDTHfr/E9reDiHRl5/fPwj860BonZJ9riPfCoSLBUjWRQLl8JaC547cSy/XN6CY4LhEF1dgOPjoHqGyl4aBTTJz8nk8IZnwav68cmeHqnMaafYp2LIANsyBnd+DIw+iWsO1/wft7oRaTcq/Bh9fqFbLepSEMZBz5tKhUTRcTqRC5jqrbc7pyyxYrIC41B5H0Wkh0VCzie6FKLfSMLDJSws3s+Pwaabf351a1QPsLqd85eXAju+sYwBbF0JeJoQ3gB7joP1dULutd2zYRCAwxHpQwj25vJyLBMhFAuVcm+O7C9pwkZGFQ+pA46utR5NroFZT71iHymNpGNhg6eZDTP9lL6OvbsI1LaPsLqd8OByw9ycrADbNszZy1SKg8whoNxgaXFG1ukb8Aqxf9MV1fRXlcEB2RpE9jr2w5yfYs8LawwIIresMh17WfzUcVAlpGFSw9FPZ/L8562hdJ5Snb2xldznuZQwcWGsFwIYv4NR+8K8ObQZYewBN++iZOiXl41PQPUShLrS4+631fXSHFQq7V8Cu/1nrHiCs/oXhULOxhoO6LA2DCmSM4ek5azmdnceMsVdWnpvUHNlh/UJdP9vaOPn4Q4sboP3foGV/CKhmd4WVkwhEtrAe58LhyLaCcNixFNbNtNqGNygSDo3srV15HA2DCjT9l70s25rOc7e1pWXtULvLKZuTB2DjF1YA7F8DiLWRuepRaHNbyQ/OqrITgahW1qPbGCsc0rfAnh9h93LYthjWzrDa1mjoDAZnONSogmezqQtoGFSQbYdO8eKCzfRtFcU9Pbz0V1nmcdg039oL2L0CMFC3E/R7EdoNgrB6dleoChOB6DbWo/sD1vGH9M1WOOxZAVsXQPKnVtuajQvtOfSCcB0xt6rReyBXgKzcfG5/6yfST2Wz6IlriAoNtLsk1+WchW2LrGsBtn9rnWdfq5l1DKD9YKuLQnknhwMOb7KCYc+P1iPrhPVezSbQpFdBOITVtbdWVSy9B7IX+OfirWw5eIqpo+K8IwjycwsORm752jpHPqQOXPGgdS1Avc56MLIy8PGBOu2sx5V/sC7GO7SxIBw2zoPV0622tZoVCoerS36hn/J4GgblbPm2dD74cTf39GjEta1r213OpTkckLbSCoCNX1rDNwSFW90/7e+CRj2ti7VU5eXjC3U7WI8ej1jhcHB9QThs+AJWfWS1jWjhDAdn11JJT5lVHke7icrRsTM53Pj6csKD/fn60as98+yhQxudg8J9Dhkp4BcErfpbAdD8evDzgj0ZVTHy8+DguoJw2PsL5Jyy3otsVRAOja6GkEp6/YwHq5BuIhG5CZgM+ALvG2NeLvJ+I2AqEAUcA+42xqSJSCfgbSAMyAdeNMbMLG2x3sQYwx8/X0fG2Vw+uq+bZwXB8T3OQeHmWH3G4gvN+sK1f4bWt1hDRStVlK8f1O9iPXo+boXDgbXOcFgByTMg8X2rbVQb59XRvaxwqK736PB0xe4ZiIgvsA24AUgDEoFhxphNhdrMBr42xkwTkWuB+4wxI0WkJWCMMdtFpB6wCmhjjDlxqc+rLHsGM1amMOmL9TxzSxvG9GpqdzlwOh02zrX2AtJWWtMaXGkdBG57B1SPtLc+5f3yc2F/ckE4pPxaMMx4dNtC4dBTTz0uB2XdM3AlDHoAzxpjbnS+ngRgjHmpUJuNwE3GmFSxRlzLMMb8bsxhEVkLDDbGbL/U51WGMNiZfpoBb/xI10Y1mX5/d3x8bDrYmnUStnxjBcCuZWDyrX+U7QdbB4L1wiNVnvJzYd/qgm6llF+tcakQqN2uUDhc5bzCWpVFRXQT1QdSC71OA64o0mYtMAirK+kOIFREIowxRwsV2h0IAHaWtlhvkJPn4ImEZAL9fXh1SMeKD4LcLNixxAqAbYshLwvCG1q79e0HW4PCKVURfP2h4RXW45oJ1mB9+1db16jsWQGrPoTf3gYE6rS3DkQ36QUNe1ijtaoK5a6ziSYAb4rIKGA5sA/rGAEAIlIX+Bi41xjjKDqziIwFxgI0bNjQTSXZ47XvtrF+Xwbv3N2V2mFBFfOhjnzrH9f62bDpK2tgs2qR0OUe60BwTDc9FVTZzy8AGl5pPXo/DXnZsG9VQTgkvg+/vgWIdUbTuWscGvWwzmxT5cot3URF2ocAW4wxMc7XYcAy4O/GmDnFFeTN3US/7DzK8Pd/Jb5bA14a1KF8P8wY61fW+jnWweDThyAgBNrcau0BNOljHfBTylvkZsG+JGc4/Ggd28rPse56V7fjheGgJzn8TkUcM/DDOoB8HdYv/kRguDFmY6E2kcAxY4xDRF4E8o0xfxGRAGAh8JUx5nVXCvLWMMg4m8tNk5cT7O/L149dTbWActoQp28rGBTu2C7r7lot+lkB0PIm8A8un89VqqLlZkJaYqFwSLSugBdfqNfJCoZmfaHxNVVrOPRLKPdjBsaYPBEZByzGOrV0qjFmo4g8DyQZY+YDfYCXRMRgdRM94px9CHANEOHsQgIYZYxJLm3BnsgYw5/mrif9VDZfPHyV+4MgY5/zVNDZ1nneiHVDk6uftPYE9OCbqoz8g62/8ybXWK9zzlp7C3t+tALil7fgp9etoTO6jYZOI/QspTLQi87cYM6qNCbMXsvTN7bikb7N3bPQ/FxrELF1s62bxGCgXhfrGEDbO3SsGKVyzlh3zlv5HqT+Cn7B0OEu6PaAdcyhiin3bqKK5m1hsPfoGW6evIK29cOZ8cCV+Lrj7CFjYO6D1lj0ES0KBoWLaFb2ZStVGR1YB4nvWT+e8jKta2i6P2ANp+5XyW8r66QD1dkoN9/B4wnJ+PgIrw3t5J4gAFj2khUEfZ+xTsnTM4GUury6HeC2/8ANz8OaT60zkz4fDdWjoesoiLtPh1gvhh51KYP/fL+D5NQT/P2O9tSv4aYDt8kz4H+vQKe7NQiUKqngmnDVOHh0NYyYY42wu/yf8Fo7mHWPdazBw3pDPIXuGZRS0p5jvPn9dgZ1qc+tHd30i2P3Cpj/qHXAbMBrGgRKlZaPj3Xr1RY3wLHdkPQBrP4YNs2zxk3qPgY6xENgiN2Vegw9ZlAKJ7NyuXnyCkRgwWO9CA1yw03e07fCBzdAaF24f7FegamUu+Wctc7KWznFOisvMAw6DrOOLVSCmzSV9ZiBdhOVwl/nbeRARhY8O0RoAAATY0lEQVSvD+3sniA4nQ6f3mVdMzB8lgaBUuUhoBp0GQkPLofRS6zrcpKmwptxMH2gNY6XI7/45VRS2k1UQvOS9zF3zT6euL4FXRu54fz+3ExIGAanD8Oob3TwOKXKmwg06G49bnwRVk2zQiFhOIQ3gLj7raFcqthIvrpnUAKpx87yzNwNdGlYg3HuuJ7A4YAvxkJaEtz5HsR0LfsylVKuC4m2xkl6Yj0M+RhqNoalz8G/Y2HuQ5C2yu4KK4zuGbgo32EYPysZA7w+tDN+vm7I0aXPwub50O9F60pipZQ9fP0g9jbrcXizdWrq2gRYO8O62LP7A9B2EPhX0OCTNtA9Axe9vWwHiXuO8/zAtjSMqFb2BSZ9CD9NhrjR1v1mlVKeIboN3PIqjN8M/f8JOafhyz/Av9vAkr/C8b12V1guNAxckJx6gte/286tHetxR+f6ZV/g9u/gm6esAeb6/0NPIVXKEwWFwRVj4ZGVcM886yY8P78Bb3SCGcNg5/dWV28lod1ExTiTnccTCWuoHRbE325vh5R1w31wA8weBdGxMHiqDjOtlKcTgaZ9rMeJVOumPKumwdYFENHcGgup0zCvv+eC7hkU4/mvNrH32Fn+PaQj4cFlPI305AH4bIh1ocvwmTomu1LepkYDuO4vMH4T3DEFgmrAoj/Cq23g6yfh0Kbil+Gh9GfpZSxcf4CZSak80rcZVzSNKNvCsk9bQZCVAfcthHA3dDcppezhFwgdh1qPfautA85rPrVOUW10tXWFc+sB1q0/vYTuGVzCgYxMJn6xng4x4TxxfcuyLcyRbw2adWgDDP6wSg6vq1SlVb8L3P5f64Dz9c9BRorVFfx6e1j2Cpw6ZHeFLtEwuAiHw/DUrLXk5DmYHN8Z/7KeRrpoEmxbZB0sbtnPPUUqpTxL9Qi4+gl4LBmGJVjHBZf9HV5rC3Puh72/ePQgedpNdBHv/7iLn3ce5ZU729MksnrZFvbr27DyXegxzjpXWSlVufn4Qqv+1uPIDmuQvDWfWuMi1W5vdSG1vwsCyrhtcTPdMyhiw74M/rl4Kze1rcOQuAZlW9iWb6y9gtYD4IYX3FOgUsp7RDaHm16CpzbDgNfBOOCrx61rFhb9CY7utLvC83TU0kIyc/IZ8J8VnM7OY9Hj11CzehnukLR/DXx4M0S1tsYcCnDDhWpKKe9mDKT8Yt2qc/N8cORB8xusXoPm11t7FaWkdzpzoxcXbGJn+hk+HXNF2YLgRAp8NhSqRVqnkGoQKKXAumah0VXW4+QBWO0cJO+zIVCjEXQbA53vhmq1Krw07SZy+m7TIT75NYWx1zSlZ/MyjFaYlWEFQW4WjJhtDYSllFJFhdWFPhPhiQ3WBahh9WHJ/1ldSPMegf3JFVqO7hkAh09l8f8+X0ds3TCe6leG00jzc2HWvXBkG9z9OUS3dl+RSqnKyS8A2t1pPQ5ugMT3YN0sWPMJxHS3upBiB1rXNpSjKr9n4HAYJsxex5nsPN4Y1olAv1L22RkD34yHXT/ArZOtS9eVUqok6rSzth/jN8ONL8HZo/DFA9bpqUtfgIy0cvtol8JARG4Ska0iskNEJl7k/UYislRE1onIMhGJKfTevSKy3fm4153Fu8O0X/awfFs6zwyIpXl0GYaH+Ol1WD0dek2w+vyUUqq0gmtAj4dhXJLVy1A/Dla8Cq93gJl3w+7lbr9modiziUTEF9gG3ACkAYnAMGPMpkJtZgNfG2Omici1wH3GmJEiUgtIAuIAA6wCuhpjjl/q8yrybKItB09y25s/0at5JO/fG1f6Qeg2fAFz7rN28wa9b92MWyml3On4Hutg8+rpkHncOlOx2xjoGA+BoRVyD+TuwA5jzC5jTA6QAAws0iYW+N75/IdC798ILDHGHHMGwBLgptIW605Zufk8PiOZsCA/XhncofRBkPKbdUekBlfCwP9qECilykfNxnDD81YX0sD/gl8QLJhgDZK34OkyL96VLVd9ILXQ6zTntMLWAoOcz+8AQkUkwsV5bfHKoi1sPXSKf97VkciQUh6YObbLun9xeH2I/6xS3wVJKeUh/IOh8wgYuwzGLIXWN8Oqj8q8WHf9jJ0A9BaRNUBvYB+Q7+rMIjJWRJJEJCk9Pd1NJV3a/7al8+FPexh1VWP6tirlqZ9nj8GnQ6wrCkfMscYlUUqpiiICMXEwaAo8Wfahs10Jg31A4XEZYpzTzjPG7DfGDDLGdAb+7Jx2wpV5nW2nGGPijDFxUVFRJfwKJXP0dDYTZq+lZe0QJvYv5amfednWQZwTeyF+BkQ0c2+RSilVEiFl3266EgaJQAsRaSIiAUA8ML9wAxGJFJFzy5oETHU+Xwz0E5GaIlIT6OecZgtjDH/8fD0ZZ3OZHN+ZIP9SnEZqDMx/FPb+BLe/DY16uL9QpZSqYMWGgTEmDxiHtRHfDMwyxmwUkedF5DZnsz7AVhHZBtQGXnTOewx4AStQEoHnndNs8dnKFL7bfIg/9m9Nm7phpVvIspdh3Uzo+wy0H+zeApVSyiZVZqC6HYdPM+A/K+jWuBbT7uuOj08pzh5KngFfPgSdRsDAt/RG9kopj1ERp5Z6vZw8B48nrCHY35dX7+pYuiDYvcLqHmpyjTUUrQaBUqoSqRJjE726ZCsb959kysiuRIeV4vTP9G0wcwTUagpDPrbGElFKqUqk0u8Z/LzjCFOW72L4FQ3p17ZOyRdwOh0+HQy+AdYopME13F+kUkrZrFLvGZw4m8P4WWtpElmdZ25pU/IF5GZaF5WdPgSjFkDNRu4vUimlPECl3TMwxjDpi/UcPZPNG/GdqRZQwtxzOGDug5CWBIPeg5iu5VOoUkp5gEobBrNXpbFww0Ge6teKdvXDS76Apc/CpnnQ7wWIva3Y5kop5c0qZRjsOXKGZ+dvpEfTCMb2alryBSR9CD9NhrjR0GOc+wtUSikPU+nCIDffweMzk/H39eHVIaU4jXTHd/DNU9ZNqvv/Q08hVUpVCZXuAPIbS7ezNvUEbw3vQr0awSWb+eAGmDUKomPhrg/Bt9KtHqWUuqhKtWewcvcx3vphB3d1jeGWDnVLNvPJA/DZEAgMgeEzIbAMdz1TSikvU2l++mZk5vLkzGQa1KrGX29rW7KZs0/DjKGQeQLuX2jdn0AppaqQShMGf5m3gYMns5jzUA9CAkvwtRz58PkYOLgehs2Euh3Lr0illPJQlaKb6Ms1+5iXvJ8nrmtB54Y1Szbz4j/BtoXWweKW/cqnQKWU8nBeHwapx87yf19uoFvjmjzct3nJZv71HfjtHbjyEej+QPkUqJRSXsCrwyAv38GTM5MB+PeQTviW5DTSLQtg0URoPcC6sEwppaowrz5m8PaynSTtPc7k+E40qFXN9Rn3r4HPR0O9ztZQEz6luOOZUkpVIl67Z7Am5TivL93O7Z3qMbBTCc7+OZEKnw2FapEwLAECShAiSilVSXnlnsHp7DyemJlMnbAgnr+9neszZmVY1xLkZsE98yC0dvkVqZRSXsQrw+C5+RtJPXaWhLE9CAvyd22m/FyYdS8c2QYj5kB0KYa0VkqpSsrruom+WXeA2avSeKRvc7o3qeXaTMZY4w3t+sG6ZWWzvuVbpFJKeRmvCoP9JzKZ9MU6OjaowWPXtXB9xp9eh9XToNdT0GVk+RWolFJeymvCIN9hGD8rmTyHYfLQTvj7ulj6xrnw3bPQ7k7o+0y51qiUUt7Ka44ZvLdiF7/uOsY/BnegcWR112ZKXQlfPAgNroSB/wUfr8k+pZSqUC5tHUXkJhHZKiI7RGTiRd5vKCI/iMgaEVknIjc7p/uLyDQRWS8im0VkUmmKXJ+WwavfbuXm9nW4q2uMazMd2wUz4iGsHsR/Bv5BpflopZSqEooNAxHxBd4C+gOxwDARiS3S7BlgljGmMxAP/Nc5/S4g0BjTHugKPCgijUtS4NmcPB6fuYaI6oH8/Y72iCs3mzl7DD4dAsZhnTlUPaIkH6mUUlWOK3sG3YEdxphdxpgcIAEYWKSNAcKcz8OB/YWmVxcRPyAYyAFOlqTAv32zmd1HzvDvoR2pUS2g+BnysmHmSDix19ojiCzheEVKKVUFuRIG9YHUQq/TnNMKexa4W0TSgAXAo87pc4AzwAEgBfiXMeZY0Q8QkbEikiQiSenp6eenf7vxIJ/9lsLYa5pyVbPI4is1BuY/Bnt/tI4RNLrKha+nlFLKXUdUhwEfGWNigJuBj0XEB2uvIh+oBzQBnhKR392h3hgzxRgTZ4yJi4qKAuDwySz++Pk62tUP46kbWrlWxf9egXUJ0PfP0OEut3wxpZSqClwJg31Ag0KvY5zTChsNzAIwxvwCBAGRwHBgkTEm1xhzGPgJiCvuAx0Ow1Oz15KZm8/rQzsT4OdCmWsTYNlL0GkEXPO0C19LKaXUOa6EQSLQQkSaiEgA1gHi+UXapADXAYhIG6wwSHdOv9Y5vTpwJbCluA/88Oc9rNh+hP8bEEvz6JDiK9y9AuaNg8a9rCuMXTnIrJRS6rxiw8AYkweMAxYDm7HOGtooIs+LyG3OZk8BD4jIWmAGMMoYY7DOQgoRkY1YofKhMWbd5T4vKzefVxZu4fo2tRnevWHx3yB9G8wcAbWawNCPwc+Fg8xKKaUuINY223OENWhlWj30Fose70VESODlG585Au9fBzlnYMx3ULNxhdSolFKeRkRWGWOK7Ya/FI+7Ajk7z8G/7upYfBDkZloXlZ06CKO+0SBQSqky8LgwaBEdQu+WUZdv5HDA3IcgLQmGTIOYUoehUkopPDAMgvxduAXl0udg05dwwwsQW/T6N6WUUiXlfSO3rfrIGpI67n646tFimyullCqed4XBjqXw9Xhofj30/6eeQqqUUm7iPWFwaKN128roNjD4Q/D1uB4upZTyWt4RBicPWKOQBobA8FkQFFb8PEoppVzm+T+vc87AjKGQeRzuXwjhRcfIU0opVVaeHQaOfJgzGg6uh2EJULej3RUppVSl5NlhsPhPsG0h3PwvaHmj3dUopVSl5bnHDH59B357B658GLo/YHc1SilVqXlmGGxdCIsnQatboN/f7K5GKaUqPc8Lg9yzMOd+6/jAne+BjwtXJCullCoTzwuDY7ugWgQMmwkB1e2uRimlqgTPCwOAEbMhtLbdVSilVJXheWEQ3da6ylgppVSF8bww0PGGlFKqwnleGCillKpwGgZKKaU0DJRSSmkYKKWUQsNAKaUUGgZKKaXQMFBKKQWIMcbuGi4gIqeArW5ebDiQ4ca2l2rj6vTLvS78PBI4UkwtJaXrovgaS9u2JOvClWkVtS5Ksh5cba/rovg27l4XrYwxocUVfEnGGI96AEnlsMwp7mx7qTauTr/c6yLPdV1U0nXhyrSKWhclWQ+6Lirvuqgq3URfubntpdq4Ov1yr0tSa2nouijd8t29LlyZVlHroqTL1nVRsvZesS48sZsoyRgTZ3cdnkDXRQFdFwV0XRTQdVGgrOvCE/cMpthdgAfRdVFA10UBXRcFdF0UKNO68Lg9A6WUUhXPE/cMlFJKVTANA6WUUhoGSimlvCwMRKSNiLwjInNE5A9212MnEbldRN4TkZki0s/ueuwkIk1F5AMRmWN3LXYQkeoiMs359zDC7nrsVNX/Fgor8TbCnRdsFHPhxVTgMLChyPSbsK443gFMdHFZPsAnFVW7h6+LmsAHdn8nD1kXc+z+PnasF2AkcKvz+Uy7a/eEv5HK9LfghnXh0jaiIr/ENUCXwl8C8AV2Ak2BAGAtEAu0B74u8oh2znMbsBAYbvf/GLvXhXO+V4Eudn8nD1kXlWYDUML1Mgno5Gzzmd2127kuKuPfghvWhUvbCD8qiDFmuYg0LjK5O7DDGLMLQEQSgIHGmJeAAZdYznxgvoh8A3xWfhWXH3esCxER4GVgoTFmdflWXH7c9XdR2ZRkvQBpQAyQjJd1/bqihOtiU8VWV7FKsi5EZDMl2EbY/YdTH0gt9DrNOe2iRKSPiLwhIu8CC8q7uApWonUBPApcDwwWkYfKszAblPTvIkJE3gE6i8ik8i7ORpdaL18Ad4rI25T/EB6e4qLrogr9LRR2qb+LEm0jKmzPwB2MMcuAZTaX4RGMMW8Ab9hdhycwxhwFKlsguswYcwa4z+46PEFV/1sorKTbCLv3DPYBDQq9jnFOq4p0XRTQdXFxul4K6Loo4JZ1YXcYJAItRKSJiAQA8cB8m2uyi66LArouLk7XSwFdFwXcsi4qLAxEZAbwC9BKRNJEZLQxJg8YBywGNgOzjDEbK6omu+i6KKDr4uJ0vRTQdVGgPNeFDlSnlFLK9m4ipZRSHkDDQCmllIaBUkopDQOllFJoGCillELDQCmlFBoGSrlEROqISIKI7BSRVSKyQERa2l2XUu7iVWMTKWUH5wixc4Fpxph457SOQG1gm521KeUuGgZKFa8vkGuMeefcBGPMWhvrUcrttJtIqeK1A1bZXYRS5UnDQCmllIaBUi7YCHS1uwilypOGgVLF+x4IFJGx5yaISAcR6WVjTUq5lYaBUsUw1tC+dwDXO08t3Qi8BBy0tzKl3EeHsFZKKaV7BkoppTQMlFJKoWGglFIKDQOllFJoGCillELDQCmlFBoGSiml0DBQSikF/H/RQZSSmiDzhAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "DataFrame(results).plot(x='C', y=['Train F', 'Devel F'], logx=True) ;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a typical pattern:\n",
    "\n",
    "* For small values of C, the SVM *underfits*: performance is low even on the training data\n",
    "* For large values of C, it *overfits*: performance on training data gets close to 100%, but performance on unseen data drops\n",
    "\n",
    "(The following illustrations are regression instead of classification, but the idea is the same.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"figs/overfitting.png\" width=50%>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align:center; font-size:80%\">Overfitting: A better fit for training data (filled circles) does not guarantee generalization to unseen data (empty circles)</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"figs/underfitting.png\" width=50%>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align:center; font-size:80%\">Underfitting: a simpler model isn't necessarily better. (Filled circles represent training data, empty circles unseen data.)</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature weighting\n",
    "\n",
    "* We have so far used present/absent (1/0) or simple occurrence counts as feature values\n",
    "* Features can be assigned any values, and weighting important features highly (and vice versa) can help ML methods learn\n",
    "    * (Ignoring common *stopwords* such as \"a\" and \"the\" can be seen as an extreme example of this)\n",
    "\n",
    "### TF-IDF weighting\n",
    "\n",
    "* Popular and effective statistic measuring the importance of word in a set of documents\n",
    "* Term frequency (TF($w$)): count of word $w$ in current document (cf. counts in bags-of-words)\n",
    "* Inverse Document Frequency (IDF): 1 / fraction of documents where word occurs\n",
    "\n",
    "One possible formulation (of many):\n",
    "\n",
    "$$\\textrm{TF-IDF}(w) = \\textrm{TF}(w)\\cdot\\log\\left(\\frac{\\textrm{N}}{\\textrm{DF}(w)}\\right)$$\n",
    "\n",
    "* N is the total number of documents in the data\n",
    "* DF($w$) is the number of documents where word $w$ occurs\n",
    "\n",
    "Let's try that out:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Count  F-score  : 92.83%\n",
      "TF-IDF F-score  : 93.56%\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "\n",
    "def vectorize_and_train(train, vectorizer):\n",
    "    vectorizer.fit(train['text'])\n",
    "    train_X = vectorizer.transform(train['text'])\n",
    "    classifier = LinearSVC(C=1, max_iter=10000)\n",
    "    classifier.fit(train_X, train['class'])\n",
    "    return classifier\n",
    "\n",
    "\n",
    "def predict_and_evaluate(classifier, test, vectorizer):\n",
    "    test_X = vectorizer.transform(test['text'])\n",
    "    pred_classes = classifier.predict(test_X)\n",
    "    return f1_score(test['class'], pred_classes)\n",
    "\n",
    "\n",
    "count_vectorizer = CountVectorizer(tokenizer=basic_tokenizer)\n",
    "tfidf_vectorizer = TfidfVectorizer(tokenizer=basic_tokenizer)\n",
    "\n",
    "count_classifier = vectorize_and_train(train_texts, count_vectorizer)\n",
    "tfidf_classifier = vectorize_and_train(train_texts, tfidf_vectorizer)\n",
    "\n",
    "print('Count  F-score  : {:.2%}'.format(predict_and_evaluate(count_classifier, devel_texts, count_vectorizer)))\n",
    "print('TF-IDF F-score  : {:.2%}'.format(predict_and_evaluate(tfidf_classifier, devel_texts, tfidf_vectorizer)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nice! How did the learned weights change?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <style>\n",
       "    table.eli5-weights tr:hover {\n",
       "        filter: brightness(85%);\n",
       "    }\n",
       "</style>\n",
       "\n",
       "\n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "        \n",
       "\n",
       "    \n",
       "\n",
       "        \n",
       "            \n",
       "                \n",
       "                \n",
       "    \n",
       "        <p style=\"margin-bottom: 0.5em; margin-top: 0em\">\n",
       "            <b>\n",
       "    \n",
       "        y=spam\n",
       "    \n",
       "</b>\n",
       "\n",
       "top features\n",
       "        </p>\n",
       "    \n",
       "    <table class=\"eli5-weights\"\n",
       "           style=\"border-collapse: collapse; border: none; margin-top: 0em; table-layout: auto; margin-bottom: 2em;\">\n",
       "        <thead>\n",
       "        <tr style=\"border: none;\">\n",
       "            \n",
       "                <th style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\" title=\"Feature weights. Note that weights do not account for feature value scales, so if feature values have different scales, features with highest weights might not be the most important.\">\n",
       "                    Weight<sup>?</sup>\n",
       "                </th>\n",
       "            \n",
       "            <th style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">Feature</th>\n",
       "            \n",
       "        </tr>\n",
       "        </thead>\n",
       "        <tbody>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 83.12%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.659\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        learn\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 83.14%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.655\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        rolex\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 83.87%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.553\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        money\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 84.70%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.441\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        girls\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 84.71%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.439\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        invite\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 84.73%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.437\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        rotor\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 84.76%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.433\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        404\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 84.79%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.429\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        doctor\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 84.81%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.426\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        1776\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 84.87%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.417\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        impaired\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 84.87%); border: none;\">\n",
       "                <td colspan=\"2\" style=\"padding: 0 0.5em 0 0.5em; text-align: center; border: none; white-space: nowrap;\">\n",
       "                    <i>&hellip; 4980 more positive &hellip;</i>\n",
       "                </td>\n",
       "            </tr>\n",
       "        \n",
       "\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 84.89%); border: none;\">\n",
       "                <td colspan=\"2\" style=\"padding: 0 0.5em 0 0.5em; text-align: center; border: none; white-space: nowrap;\">\n",
       "                    <i>&hellip; 3452 more negative &hellip;</i>\n",
       "                </td>\n",
       "            </tr>\n",
       "        \n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 84.89%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.416\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        united\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 84.87%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.418\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        draft\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 84.87%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.418\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        responding\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 84.65%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.448\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        devon\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 84.38%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.484\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        clickathome\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 84.22%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.506\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        tragedy\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 84.15%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.515\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        ordered\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 83.83%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.559\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        dave\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 83.44%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.613\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        vince\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 83.36%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.624\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        sap\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 83.17%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.651\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        morning\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 81.36%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.910\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        enron\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 81.28%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.922\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        guess\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 81.13%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.943\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        clerk\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 80.00%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -2.112\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        connection\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "\n",
       "        </tbody>\n",
       "    </table>\n",
       "\n",
       "            \n",
       "        \n",
       "\n",
       "        \n",
       "\n",
       "\n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eli5.show_weights(count_classifier, vec=count_vectorizer, top=25)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <style>\n",
       "    table.eli5-weights tr:hover {\n",
       "        filter: brightness(85%);\n",
       "    }\n",
       "</style>\n",
       "\n",
       "\n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "        \n",
       "\n",
       "    \n",
       "\n",
       "        \n",
       "            \n",
       "                \n",
       "                \n",
       "    \n",
       "        <p style=\"margin-bottom: 0.5em; margin-top: 0em\">\n",
       "            <b>\n",
       "    \n",
       "        y=spam\n",
       "    \n",
       "</b>\n",
       "\n",
       "top features\n",
       "        </p>\n",
       "    \n",
       "    <table class=\"eli5-weights\"\n",
       "           style=\"border-collapse: collapse; border: none; margin-top: 0em; table-layout: auto; margin-bottom: 2em;\">\n",
       "        <thead>\n",
       "        <tr style=\"border: none;\">\n",
       "            \n",
       "                <th style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\" title=\"Feature weights. Note that weights do not account for feature value scales, so if feature values have different scales, features with highest weights might not be the most important.\">\n",
       "                    Weight<sup>?</sup>\n",
       "                </th>\n",
       "            \n",
       "            <th style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">Feature</th>\n",
       "            \n",
       "        </tr>\n",
       "        </thead>\n",
       "        <tbody>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 86.42%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +2.282\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        2004\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 87.15%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +2.108\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        money\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 87.42%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +2.046\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        2005\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 87.91%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.933\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        %\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 88.10%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.890\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        adv\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 88.12%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.884\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        http\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 88.28%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.849\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        learn\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 88.35%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.832\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        appointment\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 88.37%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.827\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        account\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 88.37%); border: none;\">\n",
       "                <td colspan=\"2\" style=\"padding: 0 0.5em 0 0.5em; text-align: center; border: none; white-space: nowrap;\">\n",
       "                    <i>&hellip; 6888 more positive &hellip;</i>\n",
       "                </td>\n",
       "            </tr>\n",
       "        \n",
       "\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 88.33%); border: none;\">\n",
       "                <td colspan=\"2\" style=\"padding: 0 0.5em 0 0.5em; text-align: center; border: none; white-space: nowrap;\">\n",
       "                    <i>&hellip; 4686 more negative &hellip;</i>\n",
       "                </td>\n",
       "            </tr>\n",
       "        \n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 88.33%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.838\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        texas\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 88.15%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.878\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        ordered\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 88.08%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.895\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        vince\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 87.96%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.920\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        conference\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 87.94%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.924\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        interview\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 87.80%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.958\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        cera\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 87.75%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.969\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        presentation\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 87.43%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -2.044\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        list\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 87.41%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -2.046\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        sap\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 87.10%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -2.119\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        morning\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 87.08%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -2.125\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        gas\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 87.07%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -2.128\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        energy\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 86.84%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -2.182\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        2000\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 86.40%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -2.286\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        meter\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 86.12%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -2.354\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        connection\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 80.00%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -3.966\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        enron\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "\n",
       "        </tbody>\n",
       "    </table>\n",
       "\n",
       "            \n",
       "        \n",
       "\n",
       "        \n",
       "\n",
       "\n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eli5.show_weights(tfidf_classifier, vec=tfidf_vectorizer, top=25)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## N-gram features\n",
    "\n",
    "* Create features from sequences of *N* consecutive words or characters\n",
    "* Word N-grams, the more common choice, provide (limited) sequence information\n",
    "* Character N-grams capture (some) word-internal information and can help with e.g. misspelled words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Char n-gram F-score  : 93.58%\n",
      "Bigram F-score  : 94.38%\n"
     ]
    }
   ],
   "source": [
    "char_vectorizer = TfidfVectorizer(analyzer='char', ngram_range=(3,5))\n",
    "bigram_vectorizer =  TfidfVectorizer(tokenizer=basic_tokenizer, ngram_range=(1,2))\n",
    "\n",
    "char_classifier = vectorize_and_train(train_texts, char_vectorizer)\n",
    "bigram_classifier = vectorize_and_train(train_texts, bigram_vectorizer)\n",
    "\n",
    "print('Char n-gram F-score  : {:.2%}'.format(predict_and_evaluate(char_classifier, devel_texts, char_vectorizer)))\n",
    "print('Bigram F-score  : {:.2%}'.format(predict_and_evaluate(bigram_classifier, devel_texts, bigram_vectorizer)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <style>\n",
       "    table.eli5-weights tr:hover {\n",
       "        filter: brightness(85%);\n",
       "    }\n",
       "</style>\n",
       "\n",
       "\n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "        \n",
       "\n",
       "    \n",
       "\n",
       "        \n",
       "            \n",
       "                \n",
       "                \n",
       "    \n",
       "        <p style=\"margin-bottom: 0.5em; margin-top: 0em\">\n",
       "            <b>\n",
       "    \n",
       "        y=spam\n",
       "    \n",
       "</b>\n",
       "\n",
       "top features\n",
       "        </p>\n",
       "    \n",
       "    <table class=\"eli5-weights\"\n",
       "           style=\"border-collapse: collapse; border: none; margin-top: 0em; table-layout: auto; margin-bottom: 2em;\">\n",
       "        <thead>\n",
       "        <tr style=\"border: none;\">\n",
       "            \n",
       "                <th style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\" title=\"Feature weights. Note that weights do not account for feature value scales, so if feature values have different scales, features with highest weights might not be the most important.\">\n",
       "                    Weight<sup>?</sup>\n",
       "                </th>\n",
       "            \n",
       "            <th style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">Feature</th>\n",
       "            \n",
       "        </tr>\n",
       "        </thead>\n",
       "        <tbody>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 82.45%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.786\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        you\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 82.69%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.751\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        <span style=\"background-color: hsl(120, 80%, 70%); margin: 0 0.1em 0 0\" title=\"A space symbol\">&emsp;</span>a<span style=\"background-color: hsl(120, 80%, 70%); margin: 0 0 0 0.1em\" title=\"A space symbol\">&emsp;</span>\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 83.08%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.696\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        <span style=\"background-color: hsl(120, 80%, 70%); margin: 0 0.1em 0 0\" title=\"A space symbol\">&emsp;</span>!<span style=\"background-color: hsl(120, 80%, 70%); margin: 0 0 0 0.1em\" title=\"A space symbol\">&emsp;</span>\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 83.10%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.693\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        gent<span style=\"background-color: hsl(120, 80%, 70%); margin: 0 0 0 0.1em\" title=\"A space symbol\">&emsp;</span>\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 83.59%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.622\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        <span style=\"background-color: hsl(120, 80%, 70%); margin: 0 0.1em 0 0\" title=\"A space symbol\">&emsp;</span>,<span style=\"background-color: hsl(120, 80%, 70%); margin: 0 0 0 0.1em\" title=\"A space symbol\">&emsp;</span>\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 83.64%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.616\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        <span style=\"background-color: hsl(120, 80%, 70%); margin: 0 0.1em 0 0\" title=\"A space symbol\">&emsp;</span>$<span style=\"background-color: hsl(120, 80%, 70%); margin: 0 0 0 0.1em\" title=\"A space symbol\">&emsp;</span>\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 84.28%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.526\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        med\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 84.89%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.443\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        ns<span style=\"background-color: hsl(120, 80%, 70%); margin: 0 0.1em 0 0.1em\" title=\"A space symbol\">&emsp;</span>!<span style=\"background-color: hsl(120, 80%, 70%); margin: 0 0 0 0.1em\" title=\"A space symbol\">&emsp;</span>\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 85.93%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.303\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        <span style=\"background-color: hsl(120, 80%, 70%); margin: 0 0.1em 0 0\" title=\"A space symbol\">&emsp;</span>[<span style=\"background-color: hsl(120, 80%, 70%); margin: 0 0 0 0.1em\" title=\"A space symbol\">&emsp;</span>\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 86.21%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.265\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        :<span style=\"background-color: hsl(120, 80%, 70%); margin: 0 0.1em 0 0.1em\" title=\"A space symbol\">&emsp;</span>ord\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 86.43%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.237\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        <span style=\"background-color: hsl(120, 80%, 70%); margin: 0 0.1em 0 0\" title=\"A space symbol\">&emsp;</span>%<span style=\"background-color: hsl(120, 80%, 70%); margin: 0 0 0 0.1em\" title=\"A space symbol\">&emsp;</span>\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 86.65%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.208\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        <span style=\"background-color: hsl(120, 80%, 70%); margin: 0 0.1em 0 0\" title=\"A space symbol\">&emsp;</span>?<span style=\"background-color: hsl(120, 80%, 70%); margin: 0 0 0 0.1em\" title=\"A space symbol\">&emsp;</span>\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 87.07%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.155\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        dic\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 87.15%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.145\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        <span style=\"background-color: hsl(120, 80%, 70%); margin: 0 0.1em 0 0\" title=\"A space symbol\">&emsp;</span>e<span style=\"background-color: hsl(120, 80%, 70%); margin: 0 0 0 0.1em\" title=\"A space symbol\">&emsp;</span>\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 87.29%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.127\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        <span style=\"background-color: hsl(120, 80%, 70%); margin: 0 0.1em 0 0\" title=\"A space symbol\">&emsp;</span>]<span style=\"background-color: hsl(120, 80%, 70%); margin: 0 0 0 0.1em\" title=\"A space symbol\">&emsp;</span>\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 87.59%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.088\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        :<span style=\"background-color: hsl(120, 80%, 70%); margin: 0 0.1em 0 0.1em\" title=\"A space symbol\">&emsp;</span>mon\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 87.61%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.086\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        no<span style=\"background-color: hsl(120, 80%, 70%); margin: 0 0 0 0.1em\" title=\"A space symbol\">&emsp;</span>\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 87.76%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.067\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        free\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 87.79%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.063\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        w<span style=\"background-color: hsl(120, 80%, 70%); margin: 0 0.1em 0 0.1em\" title=\"A space symbol\">&emsp;</span>sc\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 87.79%); border: none;\">\n",
       "                <td colspan=\"2\" style=\"padding: 0 0.5em 0 0.5em; text-align: center; border: none; white-space: nowrap;\">\n",
       "                    <i>&hellip; 79823 more positive &hellip;</i>\n",
       "                </td>\n",
       "            </tr>\n",
       "        \n",
       "\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 87.63%); border: none;\">\n",
       "                <td colspan=\"2\" style=\"padding: 0 0.5em 0 0.5em; text-align: center; border: none; white-space: nowrap;\">\n",
       "                    <i>&hellip; 55667 more negative &hellip;</i>\n",
       "                </td>\n",
       "            </tr>\n",
       "        \n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 87.63%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.083\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        met\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 87.62%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.085\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        ron<span style=\"background-color: hsl(0, 80%, 70%); margin: 0 0 0 0.1em\" title=\"A space symbol\">&emsp;</span>\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 87.53%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.096\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        ron\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 87.51%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.099\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        000\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 87.45%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.107\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        <span style=\"background-color: hsl(0, 80%, 70%); margin: 0 0.1em 0 0\" title=\"A space symbol\">&emsp;</span>note\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 87.33%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.122\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        ject<span style=\"background-color: hsl(0, 80%, 70%); margin: 0 0 0 0.1em\" title=\"A space symbol\">&emsp;</span>\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 87.32%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.123\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        var\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 87.28%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.127\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        <span style=\"background-color: hsl(0, 80%, 70%); margin: 0 0.1em 0 0\" title=\"A space symbol\">&emsp;</span>proj\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 87.16%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.143\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        mete\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 87.11%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.149\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        meter\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 86.98%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.166\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        ees\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 86.97%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.167\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        ath\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 86.85%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.182\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        nron<span style=\"background-color: hsl(0, 80%, 70%); margin: 0 0 0 0.1em\" title=\"A space symbol\">&emsp;</span>\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 86.67%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.206\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        <span style=\"background-color: hsl(0, 80%, 70%); margin: 0 0.1em 0 0\" title=\"A space symbol\">&emsp;</span>2000\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 86.66%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.207\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        cha\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 86.49%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.229\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        <span style=\"background-color: hsl(0, 80%, 70%); margin: 0 0.1em 0 0\" title=\"A space symbol\">&emsp;</span>01\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 86.29%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.255\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        gas\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 86.27%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.258\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        <span style=\"background-color: hsl(0, 80%, 70%); margin: 0 0.1em 0 0\" title=\"A space symbol\">&emsp;</span>&amp;<span style=\"background-color: hsl(0, 80%, 70%); margin: 0 0 0 0.1em\" title=\"A space symbol\">&emsp;</span>\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 85.94%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.301\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        2000\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 85.91%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.305\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        eol\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 85.30%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.387\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        tw<span style=\"background-color: hsl(0, 80%, 70%); margin: 0 0 0 0.1em\" title=\"A space symbol\">&emsp;</span>\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 84.58%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.485\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        nro\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 84.55%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.488\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        enron\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 84.55%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.488\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        nron\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 84.47%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.499\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        enro\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 83.94%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.574\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        enr\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 83.51%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.634\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        <span style=\"background-color: hsl(0, 80%, 70%); margin: 0 0.1em 0 0\" title=\"A space symbol\">&emsp;</span>/<span style=\"background-color: hsl(0, 80%, 70%); margin: 0 0 0 0.1em\" title=\"A space symbol\">&emsp;</span>\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 81.76%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.888\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        e<span style=\"background-color: hsl(0, 80%, 70%); margin: 0 0.1em 0 0.1em\" title=\"A space symbol\">&emsp;</span>:<span style=\"background-color: hsl(0, 80%, 70%); margin: 0 0 0 0.1em\" title=\"A space symbol\">&emsp;</span>\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 81.44%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.934\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        w<span style=\"background-color: hsl(0, 80%, 70%); margin: 0 0.1em 0 0.1em\" title=\"A space symbol\">&emsp;</span>:<span style=\"background-color: hsl(0, 80%, 70%); margin: 0 0 0 0.1em\" title=\"A space symbol\">&emsp;</span>\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 80.39%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -2.093\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        fw<span style=\"background-color: hsl(0, 80%, 70%); margin: 0 0.1em 0 0.1em\" title=\"A space symbol\">&emsp;</span>:<span style=\"background-color: hsl(0, 80%, 70%); margin: 0 0 0 0.1em\" title=\"A space symbol\">&emsp;</span>\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 80.00%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -2.153\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        re<span style=\"background-color: hsl(0, 80%, 70%); margin: 0 0.1em 0 0.1em\" title=\"A space symbol\">&emsp;</span>:<span style=\"background-color: hsl(0, 80%, 70%); margin: 0 0 0 0.1em\" title=\"A space symbol\">&emsp;</span>\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "\n",
       "        </tbody>\n",
       "    </table>\n",
       "\n",
       "            \n",
       "        \n",
       "\n",
       "        \n",
       "\n",
       "\n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eli5.show_weights(char_classifier, vec=char_vectorizer, top=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <style>\n",
       "    table.eli5-weights tr:hover {\n",
       "        filter: brightness(85%);\n",
       "    }\n",
       "</style>\n",
       "\n",
       "\n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "        \n",
       "\n",
       "    \n",
       "\n",
       "        \n",
       "            \n",
       "                \n",
       "                \n",
       "    \n",
       "        <p style=\"margin-bottom: 0.5em; margin-top: 0em\">\n",
       "            <b>\n",
       "    \n",
       "        y=spam\n",
       "    \n",
       "</b>\n",
       "\n",
       "top features\n",
       "        </p>\n",
       "    \n",
       "    <table class=\"eli5-weights\"\n",
       "           style=\"border-collapse: collapse; border: none; margin-top: 0em; table-layout: auto; margin-bottom: 2em;\">\n",
       "        <thead>\n",
       "        <tr style=\"border: none;\">\n",
       "            \n",
       "                <th style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\" title=\"Feature weights. Note that weights do not account for feature value scales, so if feature values have different scales, features with highest weights might not be the most important.\">\n",
       "                    Weight<sup>?</sup>\n",
       "                </th>\n",
       "            \n",
       "            <th style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">Feature</th>\n",
       "            \n",
       "        </tr>\n",
       "        </thead>\n",
       "        <tbody>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 82.90%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +3.042\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        your\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 86.31%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +2.214\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        $\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 86.42%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +2.188\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        you\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 86.69%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +2.127\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        %\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 86.76%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +2.112\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        free\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 87.06%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +2.044\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        stock\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 88.00%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.835\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        online\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 88.01%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.833\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        new schedule\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 88.16%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.799\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        2005\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 88.39%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.750\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        2004\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 88.68%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.688\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        best\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 88.75%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.673\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        account\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 88.77%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.669\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        software\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 88.96%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.628\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        pharmacy\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 88.99%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.622\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        great\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 89.01%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.618\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        : fyi\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 89.07%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.605\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        money\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 89.21%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.575\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        life\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 89.49%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        +1.518\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        rolex\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "        \n",
       "            <tr style=\"background-color: hsl(120, 100.00%, 89.49%); border: none;\">\n",
       "                <td colspan=\"2\" style=\"padding: 0 0.5em 0 0.5em; text-align: center; border: none; white-space: nowrap;\">\n",
       "                    <i>&hellip; 30904 more positive &hellip;</i>\n",
       "                </td>\n",
       "            </tr>\n",
       "        \n",
       "\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 89.50%); border: none;\">\n",
       "                <td colspan=\"2\" style=\"padding: 0 0.5em 0 0.5em; text-align: center; border: none; white-space: nowrap;\">\n",
       "                    <i>&hellip; 23199 more negative &hellip;</i>\n",
       "                </td>\n",
       "            </tr>\n",
       "        \n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 89.50%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.516\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        vince\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 89.45%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.525\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        hpl\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 89.42%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.532\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        [ fwd\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 89.36%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.546\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        update\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 89.31%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.556\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        research\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 89.29%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.560\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        prc\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 89.29%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.560\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        sap\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 89.17%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.584\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        risk\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 89.04%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.613\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        draft\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 89.03%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.613\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        01\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 88.84%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.655\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        visit\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 88.76%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.671\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        deal\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 88.75%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.674\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        morning\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 88.63%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.697\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        texas\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 88.31%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.768\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        power\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 88.28%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.774\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        eol\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 88.11%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.811\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        tw\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 88.04%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.827\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        list\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 88.02%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.831\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        conference\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 87.91%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.853\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        presentation\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 87.72%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.896\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        interview\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 87.68%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.905\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        resume\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 87.44%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -1.958\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        confirmation of\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 87.05%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -2.045\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        2001\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 86.69%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -2.128\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        gas\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 86.46%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -2.181\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        2000\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 86.44%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -2.184\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        meeting\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 86.36%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -2.202\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        fw :\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 85.75%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -2.345\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        energy\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 85.55%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -2.392\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        meter\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "            <tr style=\"background-color: hsl(0, 100.00%, 80.00%); border: none;\">\n",
       "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
       "        -3.806\n",
       "    </td>\n",
       "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
       "        enron\n",
       "    </td>\n",
       "    \n",
       "</tr>\n",
       "        \n",
       "\n",
       "        </tbody>\n",
       "    </table>\n",
       "\n",
       "            \n",
       "        \n",
       "\n",
       "        \n",
       "\n",
       "\n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "    \n",
       "\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eli5.show_weights(bigram_classifier, vec=bigram_vectorizer, top=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Comparing results (aka. making your numbers look bigger)\n",
    "\n",
    "(Don't take this too seriously)\n",
    "\n",
    "Recall the performance of the simple initial Naive Bayes classifier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Naive Bayes accuracy: 93.04%\n"
     ]
    }
   ],
   "source": [
    "pred_classes = nb_classifier.classify_many(devel_data)\n",
    "nb_accuracy = scores.accuracy(devel_classes, pred_classes)\n",
    "print('Naive Bayes accuracy: {:.2%}'.format(nb_accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And our latest, finest SVM:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SVM accuracy: 94.47%\n"
     ]
    }
   ],
   "source": [
    "devel_X = bigram_vectorizer.transform(devel_texts['text'])\n",
    "pred_classes = bigram_classifier.predict(devel_X)\n",
    "svm_accuracy = scores.accuracy(devel_classes, pred_classes)\n",
    "print('SVM accuracy: {:.2%}'.format(svm_accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Was it worth it?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Difference: 1.42% points\n"
     ]
    }
   ],
   "source": [
    "print('Difference: {:.2%} points'.format(svm_accuracy-nb_accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This doesn't look terribly impressive. Let's change perspective:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<h1>Relative reduction in error: 20.47%</h1>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.core.display import HTML\n",
    "\n",
    "\n",
    "nb_error = 1 - nb_accuracy\n",
    "svm_error = 1 - svm_accuracy\n",
    "relative_reduction_in_error = (nb_error - svm_error) / nb_error\n",
    "\n",
    "HTML('<h1>Relative reduction in error: {:.2%}</h1>'.format(relative_reduction_in_error))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Much better."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Improve the model further\n",
    "\n",
    "There are many ways that we could still try to further improve the model, for example\n",
    "\n",
    "* Additional hyperparameter optimization\n",
    "* Feature weighting variants (see [TfidfTransformer](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfTransformer.html))\n",
    "* Different values of N for N-gram representations\n",
    "* Combination of character and word features\n",
    "\n",
    "(Exercise)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Finnish topic classification\n",
    "\n",
    "On the server in `/course_data/textmine/suomi24/suomi24.tsv` you can find a Finnish topic classification dataset in the format used here. Train at least one SVM classifier on this data and evaluate its performance.\n",
    "\n",
    "(Exercise. Note that this does not require an understanding of Finnish.)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}