bow_classifier.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.5"
    },
    "colab": {
      "name": "bow_classifier.ipynb",
      "provenance": [],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/TurkuNLP/intro-to-nlp/blob/master/bow_classifier.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TX0_SdtpjayU"
      },
      "source": [
        "You can wget this notebook from https://raw.githubusercontent.com/TurkuNLP/intro-to-nlp/master/bow_classifier.ipynb\n",
        "\n",
        "# Text classification\n",
        "\n",
        "## Objectives of this lecture\n",
        "\n",
        "* Introduce text classification as a task\n",
        "* Introduce basic approaches to text classification, especially the bag-of-words model\n",
        "* Show practical implementation of the concepts in the previous two lectures (intro and machine learning primer)\n",
        "* Train a simple classifier on the IMDB dataset\n",
        "\n",
        "## Purporse of these materials\n",
        "\n",
        "* These materials support, but do not replace the lectures\n",
        "* Presence on lectures highly recommended\n",
        "* Make notes!\n",
        "\n",
        "## Text classification\n",
        "\n",
        "* Text in, class label out\n",
        "* Often thought especially in terms of \"Document classification\" --- document in, class label(s) out\n",
        "  * Movie review -> star rating\n",
        "  * Email -> ham or spam\n",
        "  * News article -> list of topics from a pre-defined set of topic categories\n",
        "  * Comment in online discussion -> abusive or not\n",
        "  * Customer email / call to customer service -> topic and relevant business process\n",
        "  * Customer feedback -> needs reaction or not\n",
        "  * ...\n",
        "  \n",
        " * Document classification can be seen as a supervised classification problem with pre-defined classes\n",
        " * A direct application of the standard approach:\n",
        "   1. prepare training/dev/test data\n",
        "   2. extract features\n",
        "   3. train your favorite classifier\n",
        "   4. test on held-out data\n",
        " "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UjdK-azOjayh"
      },
      "source": [
        "# Bag-of-words model\n",
        "\n",
        "* BoW is the simplest way to model documents for classification\n",
        "* The document is reduced to a **set** of features, in the simplest case the words\n",
        "\n",
        "\n",
        "* Feature vector: a vector with as many dimensions as we have unique features, and a non-zero value set for every feature present in our example\n",
        "* Values: 1/0 (present/absent) or oftentimes TF/IDF weights (why?)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZNAeRmEUjayi"
      },
      "source": [
        "# IMDB data\n",
        "\n",
        "* Movie review sentiment positive/negative\n",
        "* Some 25,000 examples, 50:50 split of classes (why is this number relevant?)\n",
        "* Current state-of-the-art is about 95% accuracy (what is accuracy?)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!wget -nc https://github.com/TurkuNLP/intro-to-nlp/raw/master/Data/imdb_train.json\n"
      ],
      "metadata": {
        "id": "Difs_bAtnwEP",
        "outputId": "cad6189b-376b-4c8c-9e16-3e4ac52aa5db",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2022-01-26 13:42:07--  https://github.com/TurkuNLP/intro-to-nlp/raw/master/Data/imdb_train.json\n",
            "Resolving github.com (github.com)... 192.30.255.113\n",
            "Connecting to github.com (github.com)|192.30.255.113|:443... connected.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://raw.githubusercontent.com/TurkuNLP/intro-to-nlp/master/Data/imdb_train.json [following]\n",
            "--2022-01-26 13:42:07--  https://raw.githubusercontent.com/TurkuNLP/intro-to-nlp/master/Data/imdb_train.json\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.109.133, 185.199.110.133, 185.199.108.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.109.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 33944099 (32M) [text/plain]\n",
            "Saving to: ‘imdb_train.json’\n",
            "\n",
            "imdb_train.json     100%[===================>]  32.37M  --.-KB/s    in 0.1s    \n",
            "\n",
            "2022-01-26 13:42:08 (274 MB/s) - ‘imdb_train.json’ saved [33944099/33944099]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4HmC3Ragjayi",
        "outputId": "48a3a736-13cf-47f9-cb2f-69a0fb368280",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "import json\n",
        "import random\n",
        "with open(\"imdb_train.json\") as f:\n",
        "    data=json.load(f)\n",
        "random.shuffle(data) #play it safe! (why?)\n",
        "print(\"class label:\", data[0][\"class\"])\n",
        "print(\"text:\",data[0][\"text\"])"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "class label: neg\n",
            "text: Everything in this film is bad , the story , the acting , the effects but its funny , funny , funny !!!Scott Valentine with the army uniform thats ten sizes too big is so bad with the permanent attempt at a scowl on his face as the leader of a special ops group its hilarious ! The ''terrorists'' are as scary and realistic as the ''raptors'' , this is so phoney and bad at everything it tries you have to laugh .The part where the giant T-REX who somehow snuck on board a ship and then somehow got below is blown up and you see the metal pole sticking up where its head was is the perfect ending .If your into bad films , this is the pot of gold , the mona lisa of b-b-bad !!!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iN2zVSoVjayo"
      },
      "source": [
        "## Bag of words in practice\n",
        "\n",
        "* We will need to build a feature vector for every example\n",
        "* We will need to know the class for every example\n",
        "\n",
        "* Build a data matrix with dimensionality (number of examples, number of possible features), and a value for each feature, 0/1 for binary features, TF-IDF weights are also a typical choice\n",
        "\n",
        "It is quite useless to do all this ourselves, so we will use ready-made classes and functions mostly from scikit"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pUaLFcNjjayp",
        "outputId": "23d901d4-347a-4129-b694-29d87c075cb4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# We need to gather the texts and labels into separate lists\n",
        "texts=[one_example[\"text\"] for one_example in data]\n",
        "labels=[one_example[\"class\"] for one_example in data]\n",
        "print(\"This many texts\",len(texts))\n",
        "print(\"This many labels\",len(labels))\n",
        "print()\n",
        "for label,text in list(zip(labels,texts))[:20]:\n",
        "    print(label,text[:50]+\"...\")\n",
        "\n"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "This many texts 25000\n",
            "This many labels 25000\n",
            "\n",
            "neg Everything in this film is bad , the story , the a...\n",
            "pos Jafar Panahi's comedy-drama \\Offside\\\" portrays so...\n",
            "pos This is a great movie that everyone should see. It...\n",
            "pos I saw this movie with a bunch of friends and altho...\n",
            "pos Having previously seen the abridged print presente...\n",
            "neg In my opinion, this movie's title should be change...\n",
            "neg Well. Where to begin. Let's just say this; avoid t...\n",
            "neg Flat, ordinary thriller about a conniving woman wh...\n",
            "pos This is one of the most guilty pleasure movies eve...\n",
            "neg If Alien, Jurassic Park and countless other sci fi...\n",
            "pos SPOILERS 9/11 is a very good and VERY realistic do...\n",
            "neg This is one of the funniest movies i've ever seen....\n",
            "neg I never like to comment on a good film but when it...\n",
            "neg Words can't describe how utterly stupid this story...\n",
            "neg I'd give it a 2/10.  I was really, really disappoi...\n",
            "pos This movie is one of the funniest, saddest and mos...\n",
            "neg What can possibly said about this movie other than...\n",
            "pos Flynn, known mostly for his swashbuckling roles (a...\n",
            "neg Over the past year, Uwe Boll has shown marginal im...\n",
            "pos To all the miserable people who have done everythi...\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Que9798Ujayp"
      },
      "source": [
        "## Sklearn text vectorizers\n",
        "\n",
        "* Vectorizers take care of turning inputs into feature vectors\n",
        "* Also build the feature name to feature index mapping:\n",
        "    * `.fit()` learn the mapping\n",
        "    * `.fit_transform()` learn and apply the mapping\n",
        "    * `.transform()` apply the learned mapping\n",
        "    * (What is the difference?)\n",
        "\n",
        "Let's first try on a tiny example, then work our way to the real data:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "X2nYLTb7jayq",
        "outputId": "9ebfdd4e-a877-4036-9f0b-9ee4f523cadc",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "import sklearn\n",
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "\n",
        "vectorizer=CountVectorizer()\n",
        "\n",
        "#just two short document\n",
        "toy_data=[\"More precious than gold: Why the metal palladium is soaring\",\"The price of the precious metal palladium has soared on the global commodities markets.\"]\n",
        "\n",
        "vectorizer.fit(toy_data)\n",
        "print(\"Unique features:\")\n",
        "print(vectorizer.get_feature_names())\n",
        "print()\n",
        "print(\"Feature vectors (sparse format):\")\n",
        "print(vectorizer.transform(toy_data))"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Unique features:\n",
            "['commodities', 'global', 'gold', 'has', 'is', 'markets', 'metal', 'more', 'of', 'on', 'palladium', 'precious', 'price', 'soared', 'soaring', 'than', 'the', 'why']\n",
            "\n",
            "Feature vectors (sparse format):\n",
            "  (0, 2)\t1\n",
            "  (0, 4)\t1\n",
            "  (0, 6)\t1\n",
            "  (0, 7)\t1\n",
            "  (0, 10)\t1\n",
            "  (0, 11)\t1\n",
            "  (0, 14)\t1\n",
            "  (0, 15)\t1\n",
            "  (0, 16)\t1\n",
            "  (0, 17)\t1\n",
            "  (1, 0)\t1\n",
            "  (1, 1)\t1\n",
            "  (1, 3)\t1\n",
            "  (1, 5)\t1\n",
            "  (1, 6)\t1\n",
            "  (1, 8)\t1\n",
            "  (1, 9)\t1\n",
            "  (1, 10)\t1\n",
            "  (1, 11)\t1\n",
            "  (1, 12)\t1\n",
            "  (1, 13)\t1\n",
            "  (1, 16)\t3\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function get_feature_names is deprecated; get_feature_names is deprecated in 1.0 and will be removed in 1.2. Please use get_feature_names_out instead.\n",
            "  warnings.warn(msg, category=FutureWarning)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ufN4edxojayq",
        "outputId": "d71f76a0-844b-48c7-e562-dfc096523e28",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "#...and in a more understandable format\n",
        "#...these are the feature vectors of our two toy documents\n",
        "print(vectorizer.transform(toy_data).todense())"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[0 0 1 0 1 0 1 1 0 0 1 1 0 0 1 1 1 1]\n",
            " [1 1 0 1 0 1 1 0 1 1 1 1 1 1 0 0 3 0]]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pugfeTZYjayr",
        "outputId": "99125612-3c59-4921-9947-ff90ca805e27",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# only features seen in the training data can be taken into account!\n",
        "print(vectorizer.transform([\"unseen words only\"]).todense())"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bdnun-Kejays",
        "outputId": "45ac7d95-4361-49ce-cebd-9068787bf71e",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "\n",
        "vectorizer=CountVectorizer(max_features=100000,binary=True,ngram_range=(1,1))\n",
        "feature_matrix=vectorizer.fit_transform(texts)\n",
        "print(\"shape=\",feature_matrix.shape)\n",
        "print(\"what did we get? ->\", feature_matrix.__class__)"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "shape= (25000, 74849)\n",
            "what did we get? -> <class 'scipy.sparse.csr.csr_matrix'>\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LUIu865Mjays",
        "outputId": "c6771ff4-47b7-42eb-de4a-19fe781b50c7"
      },
      "source": [
        "print(feature_matrix)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "  (0, 18292)\t1\n",
            "  (0, 46050)\t1\n",
            "  (0, 38755)\t1\n",
            "  (0, 66339)\t1\n",
            "  (0, 32435)\t1\n",
            "  (0, 46680)\t1\n",
            "  (0, 24202)\t1\n",
            "  (0, 68640)\t1\n",
            "  (0, 4753)\t1\n",
            "  (0, 2662)\t1\n",
            "  (0, 60351)\t1\n",
            "  (0, 402)\t1\n",
            "  (0, 72365)\t1\n",
            "  (0, 36865)\t1\n",
            "  (0, 67117)\t1\n",
            "  (0, 67125)\t1\n",
            "  (0, 6334)\t1\n",
            "  (0, 25734)\t1\n",
            "  (0, 9304)\t1\n",
            "  (0, 73342)\t1\n",
            "  (0, 66367)\t1\n",
            "  (0, 65747)\t1\n",
            "  (0, 62338)\t1\n",
            "  (0, 3258)\t1\n",
            "  (0, 21827)\t1\n",
            "  :\t:\n",
            "  (24999, 3538)\t1\n",
            "  (24999, 57668)\t1\n",
            "  (24999, 49147)\t1\n",
            "  (24999, 66526)\t1\n",
            "  (24999, 66621)\t1\n",
            "  (24999, 31868)\t1\n",
            "  (24999, 13574)\t1\n",
            "  (24999, 31669)\t1\n",
            "  (24999, 68412)\t1\n",
            "  (24999, 45388)\t1\n",
            "  (24999, 66329)\t1\n",
            "  (24999, 45656)\t1\n",
            "  (24999, 50496)\t1\n",
            "  (24999, 72910)\t1\n",
            "  (24999, 33779)\t1\n",
            "  (24999, 18839)\t1\n",
            "  (24999, 72239)\t1\n",
            "  (24999, 17239)\t1\n",
            "  (24999, 29971)\t1\n",
            "  (24999, 70493)\t1\n",
            "  (24999, 12484)\t1\n",
            "  (24999, 51610)\t1\n",
            "  (24999, 15621)\t1\n",
            "  (24999, 67385)\t1\n",
            "  (24999, 25109)\t1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_MUyzxhbjayt",
        "outputId": "159bb452-f63e-4da2-9873-6c70c8c76583",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "print(vectorizer.get_feature_names_out()[:1000])"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['00' '000' '0000000000001' '00001' '00015' '000s' '001' '003830' '006'\n",
            " '007' '0079' '0080' '0083' '0093638' '00am' '00pm' '00s' '01' '01pm' '02'\n",
            " '020410' '029' '03' '04' '041' '05' '050' '06' '06th' '07' '08' '087'\n",
            " '089' '08th' '09' '0f' '0ne' '0r' '0s' '10' '100' '1000' '1000000'\n",
            " '10000000000000' '1000lb' '1000s' '1001' '100b' '100k' '100m' '100min'\n",
            " '100mph' '100s' '100th' '100x' '100yards' '101' '101st' '102' '102nd'\n",
            " '103' '104' '1040' '1040a' '1040s' '105' '1050' '105lbs' '106' '106min'\n",
            " '107' '108' '109' '10am' '10lines' '10mil' '10min' '10minutes' '10p'\n",
            " '10pm' '10s' '10star' '10th' '10x' '10yr' '11' '110' '1100' '11001001'\n",
            " '1100ad' '111' '112' '1138' '114' '1146' '115' '116' '117' '11f' '11m'\n",
            " '11th' '12' '120' '1200' '1200f' '1201' '1202' '123' '12383499143743701'\n",
            " '125' '125m' '127' '128' '12a' '12hr' '12m' '12mm' '12s' '12th' '13'\n",
            " '130' '1300' '1300s' '131' '1318' '132' '134' '135' '135m' '136' '137'\n",
            " '138' '139' '13k' '13s' '13th' '14' '140' '1408' '140hp' '1415' '142'\n",
            " '145' '1454' '146' '147' '1473' '149' '1492' '14a' '14ieme' '14s' '14th'\n",
            " '14yr' '14ème' '15' '150' '1500' '1500s' '150_worst_cases_of_nepotism'\n",
            " '150k' '150m' '151' '152' '153' '1547' '155' '156' '1561' '157' '158'\n",
            " '1594' '15mins' '15minutes' '15s' '15th' '16' '160' '1600' '1600s'\n",
            " '160lbs' '161' '1610' '163' '164' '165' '166' '1660s' '168' '169' '1692'\n",
            " '16ieme' '16k' '16mm' '16s' '16th' '16x9' '16ème' '16éme' '17' '170'\n",
            " '1700' '1700s' '1701' '171' '175' '177' '1775' '1780s' '1790s' '1794'\n",
            " '1798' '17million' '17th' '18' '180' '1800' '1800mph' '1800s' '1801'\n",
            " '1805' '1809' '180d' '1812' '1813' '18137' '1814' '1816' '1820' '1824'\n",
            " '183' '1830' '1832' '1836' '1837' '1838' '1839' '1840' '1840s' '1844'\n",
            " '1846' '1847' '185' '1850' '1850ies' '1850s' '1852' '1853' '1854' '1855'\n",
            " '1859' '1860' '1860s' '1861' '1862' '1863' '1864' '1865' '1870' '1870s'\n",
            " '1871' '1873' '1874' '1875' '1876' '188' '1880' '1880s' '1881' '1886'\n",
            " '1887' '1888' '1889' '188o' '1890' '1890s' '1892' '1893' '1894' '1895'\n",
            " '1896' '1897' '1898' '1899' '18a' '18s' '18th' '18year' '19' '190' '1900'\n",
            " '1900s' '1901' '1902' '1903' '1904' '1905' '1906' '1907' '1908' '1909'\n",
            " '1910' '1910s' '1911' '1912' '1913' '1914' '1915' '1916' '1917' '1918'\n",
            " '1919' '192' '1920' '1920ies' '1920s' '1921' '1922' '1923' '1924' '1925'\n",
            " '1926' '1927' '1928' '1929' '1930' '1930ies' '1930s' '1931' '1932' '1933'\n",
            " '1934' '1935' '1936' '1937' '1938' '1939' '193o' '194' '1940' '1940s'\n",
            " '1941' '1942' '1943' '1944' '1945' '1946' '1947' '1948' '1949' '1949er'\n",
            " '195' '1950' '1950s' '1951' '1952' '1953' '1954' '1955' '1956' '1957'\n",
            " '1958' '1959' '1960' '1960s' '1961' '1961s' '1962' '1963' '1964' '1965'\n",
            " '1966' '1967' '1968' '1969' '197' '1970' '1970ies' '1970s' '1971' '1972'\n",
            " '1973' '1974' '1975' '1976' '1977' '1978' '1979' '19796' '197o' '1980'\n",
            " '1980ies' '1980s' '1981' '1982' '1982s' '1983' '1983s' '1984' '1984ish'\n",
            " '1985' '1986' '1987' '1988' '1989' '1990' '1990s' '1991' '1992' '1993'\n",
            " '1994' '1995' '1996' '1997' '1998' '1999' '19k' '19th' '19thc' '1am'\n",
            " '1and' '1d' '1h' '1h30' '1h40' '1h40m' '1h53' '1hour' '1hr' '1million'\n",
            " '1min' '1mln' '1o' '1s' '1st' '1ton' '1tv' '1½' '1ç' '20' '200' '2000'\n",
            " '20000' '20001' '2000ad' '2000s' '2001' '2002' '2003' '2004' '2004s'\n",
            " '2005' '2006' '2007' '2008' '2009' '200ft' '200th' '201' '2010' '2012'\n",
            " '2013' '2015' '2017' '2019' '2020' '2022' '2023' '2030' '2031' '2033'\n",
            " '2035' '2036' '2038' '204' '2040' '2044' '2046' '2047' '2050' '2053'\n",
            " '2054' '206' '2060' '2070' '2080' '209' '2090' '20c' '20ft' '20k' '20m'\n",
            " '20mins' '20minutes' '20mn' '20p' '20perr' '20s' '20th' '20ties'\n",
            " '20widow' '20x' '20year' '20yrs' '21' '210' '2100' '214' '215' '2151'\n",
            " '216' '21699' '21849889' '21849890' '21849907' '21st' '22' '220' '2200'\n",
            " '221' '2210' '22101' '222' '223' '225' '2257' '225mins' '227' '22d'\n",
            " '22h45' '22nd' '23' '230lbs' '230mph' '231' '232' '233' '236' '237' '23d'\n",
            " '23rd' '24' '240' '2400' '241' '242' '248' '2480' '249' '24m30s' '24th'\n",
            " '24years' '25' '250' '2500' '250000' '25million' '25mins' '25s' '25th'\n",
            " '25yo' '25yrs' '26' '260' '2600' '261k' '262' '2642' '269' '26th' '27'\n",
            " '270' '272' '273' '274' '275' '2772' '278' '27th' '27x41' '28' '280'\n",
            " '285' '28th' '29' '29th' '2am' '2d' '2fast' '2furious' '2h' '2h30'\n",
            " '2hour' '2hours' '2hr' '2hrs' '2in' '2inch' '2k' '2more' '2nd' '2oo4'\n",
            " '2oo5' '2pac' '2point4' '2s' '2x4' '30' '300' '3000' '300ad' '300c'\n",
            " '300lbs' '300mln' '3012' '303' '305' '30am' '30ish' '30k' '30lbs' '30min'\n",
            " '30mins' '30pm' '30s' '30something' '30th' '30ties' '31' '3199' '31st'\n",
            " '32' '3200' '320x180' '32lb' '32nd' '33' '330am' '330mins' '332960073452'\n",
            " '336th' '33m' '34' '345' '3462' '34th' '35' '350' '3500' '3516' '356'\n",
            " '357' '35c' '35mins' '35mm' '35pm' '35th' '35yr' '36' '360' '365' '36th'\n",
            " '37' '370' '372' '378' '38' '38k' '38th' '39' '395' '39th' '3am' '3bs'\n",
            " '3d' '3dvd' '3k' '3lbs' '3m' '3mins' '3p' '3p0' '3pm' '3po' '3rd' '3rds'\n",
            " '3th' '3who' '3x5' '3yrs' '40' '400' '4000' '401k' '405' '409' '40am'\n",
            " '40min' '40mins' '40mph' '40s' '40th' '41' '42' '420' '425' '428' '42nd'\n",
            " '43' '430' '44' '440' '442nd' '44c' '44yrs' '45' '450' '4500' '451' '454'\n",
            " '45am' '45min' '45mins' '45s' '46' '465' '469' '47' '475' '477' '47s'\n",
            " '48' '480m' '480p' '48hrs' '49' '498' '49th' '4am' '4cylinder' '4d'\n",
            " '4eva' '4ever' '4f' '4h' '4hrs' '4k' '4kids' '4m' '4o' '4pm' '4th' '4w'\n",
            " '4ward' '4x' '4x4' '50' '500' '5000' '500000' '500ad' '500db' '500lbs'\n",
            " '502' '50c' '50ft' '50ies' '50ish' '50k' '50min' '50mins' '50s' '50th'\n",
            " '50usd' '51' '51b' '51st' '52' '5200' '5250' '529' '52s' '53' '53m' '54'\n",
            " '5400' '540i' '54th' '55' '5539' '555' '55th' '56' '57' '571' '576' '578'\n",
            " '57d' '58' '58th' '59' '598947' '59th' '5hrs' '5ive' '5kph' '5million'\n",
            " '5min' '5mins' '5s' '5seconds' '5th' '5x' '5x5' '5years' '5yo' '5yrs'\n",
            " '60' '600' '6000' '607' '608' '60ies' '60ish' '60mph' '60s' '60th'\n",
            " '60ties' '61' '618' '62' '6200' '62229249' '63' '637' '63rd' '64' '65'\n",
            " '65m' '66' '660' '666' '66er' '67' '6723' '67th' '68' '68th' '69' '69th'\n",
            " '6am' '6b' '6ft' '6hours' '6k' '6million' '6pm' '6th' '6wks' '6yo' '70'\n",
            " '700' '701' '707' '70ies' '70m' '70mm' '70s' '70th' '71' '713' '72'\n",
            " '72nd' '73' '7300' '735' '737' '74' '740' '740il' '747' '747s' '74th'\n",
            " '75' '750' '75054' '75c' '75m' '76' '7600' '77' '78' '788' '78rpm' '79'\n",
            " '79th' '7days' '7even' '7eventy' '7ft' '7ish' '7mm' '7th' '7½th' '80'\n",
            " '800' '8000' '80ies' '80ish' '80min' '80s' '80yr' '81' '817' '819' '82'\n",
            " '820' '8217' '8230' '83' '84' '849' '84f' '84s' '85' '850' '850pm' '86'\n",
            " '86s' '87' '8700' '8763' '878' '87minutes' '88' '88min' '89' '89or' '89s'\n",
            " '8bit' '8ftdf' '8k' '8mm' '8o' '8p' '8pm' '8star' '8th' '8u' '8½' '90'\n",
            " '900' '9000' '90210' '905' '90c' '90ish' '90min' '90mins' '90s' '91'\n",
            " '911' '914' '917' '92' '921' '92fs' '92nd' '93' '937' '94' '9484' '94s'\n",
            " '94th' '95' '950' '95th' '96' '97' '970' '974th' '978' '98' '987'\n",
            " '98minutes' '99' '998' '999' '9999' '99cents' '99p' '99½' '9_' '9am'\n",
            " '9as' '9do' '9ers' '9is' '9lbs' '9mm']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x2nSO-PQjayu"
      },
      "source": [
        "Now we have the feature matrix done!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XKfzJ-6xjayu"
      },
      "source": [
        "# Data split\n",
        "\n",
        "* Train data - all training based on it (this includes the vectorizer!)\n",
        "* Development data - set the parameters\n",
        "* Test data - used for nothing during training, produce final results"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "K55AK1Qljayu"
      },
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "train_texts, dev_texts, train_labels, dev_labels=train_test_split(texts,labels,test_size=0.2)\n",
        "vectorizer=CountVectorizer(max_features=100000,binary=True,ngram_range=(1,1))\n",
        "feature_matrix_train=vectorizer.fit_transform(train_texts)\n",
        "feature_matrix_dev=vectorizer.transform(dev_texts)\n",
        "\n",
        "\n"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B9uhKjCDjayv",
        "outputId": "2107573c-acf3-4169-d7e6-e31a2f83cfbd",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "print(feature_matrix_train.shape)\n",
        "print(feature_matrix_dev.shape)"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(20000, 68538)\n",
            "(5000, 68538)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f79LMVbLjayv"
      },
      "source": [
        "# Classifier train\n",
        "\n",
        "* Let us try the venerable, if a bit outdated SVM\n",
        "* Linear SVM for simplicity"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "duOkP-Z3jayw",
        "outputId": "cfdcfb26-05ac-493a-b80b-1ee4c3dfaf56",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "import sklearn.svm\n",
        "classifier=sklearn.svm.LinearSVC(C=0.0005,verbose=1)\n",
        "classifier.fit(feature_matrix_train, train_labels)"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[LibLinear]"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "LinearSVC(C=0.0005, verbose=1)"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OXvPTx4tjayx"
      },
      "source": [
        "# Test\n",
        "\n",
        "* For a quick test we can use the .score() method"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qMD4pdcfjayx",
        "outputId": "f6de0c3d-f0a5-4483-d03d-31cab9ce0aa4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "print(\"DEV\",classifier.score(feature_matrix_dev, dev_labels))\n",
        "print(\"TRAIN\",classifier.score(feature_matrix_train, train_labels))\n"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "DEV 0.8746\n",
            "TRAIN 0.8949\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A0gojaA_jayx"
      },
      "source": [
        "* Try varying the C value and observe the results"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Pdgnte5njayz",
        "outputId": "573687b4-a0bd-48e4-d033-819793faa8f9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "import sklearn.metrics\n",
        "predictions_dev=classifier.predict(feature_matrix_dev)\n",
        "print(predictions_dev)\n",
        "print(sklearn.metrics.confusion_matrix(dev_labels,predictions_dev))\n",
        "print(sklearn.metrics.accuracy_score(dev_labels,predictions_dev))"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['neg' 'pos' 'pos' ... 'pos' 'neg' 'neg']\n",
            "[[2126  359]\n",
            " [ 268 2247]]\n",
            "0.8746\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5HxnFQF6jayz"
      },
      "source": [
        "# Trained classifier save and load\n",
        "\n",
        "* We fitted two things: the vectorizer and the classifier\n",
        "* If we want to reuse them later, we need to save them\n",
        "* You can use `pickle` in most cases"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dBjeFPqojay0"
      },
      "source": [
        "import pickle\n",
        "\n",
        "with open(\"saved_model.pickle\",\"wb\") as f:\n",
        "    pickle.dump((classifier,vectorizer),f)"
      ],
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "naCxQK5njay0"
      },
      "source": [
        "* let's try to load and test"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "j9DiwbOajay0",
        "outputId": "b31e950d-4a31-494a-881d-3cf12bcb38ad",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "with open(\"saved_model.pickle\",\"rb\") as f:\n",
        "    classifier_loaded,vectorizer_loaded=pickle.load(f)\n",
        "\n",
        "feature_matrix_dev_loaded=vectorizer_loaded.transform(dev_texts)\n",
        "print(\"DEV - loaded (should match the score above)\",classifier_loaded.score(feature_matrix_dev, dev_labels))\n",
        "\n"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "DEV - loaded (should match the score above) 0.8746\n"
          ]
        }
      ]
    }
  ]
}