basic_nlp.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/TurkuNLP/intro-to-nlp/blob/master/basic_nlp.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8ejPP-C8WFL4"
   },
   "source": [
    "# Basic NLP\n",
    "\n",
    "Themes:\n",
    "1. What is Natural Language Processing (NLP) / Language Technology?\n",
    "2. Getting textual data\n",
    "3. Text Segmentation\n",
    "4. Word Frequencies\n",
    "5. Text Normalization\n",
    "\n",
    "## 1. Natural Language Processing / Language Technology\n",
    "\n",
    "* Natural language = human language\n",
    "    * Compared to formal and artificial languages, for example programming languages\n",
    "    * **variety and ambiguity:** the same meaning can be expressed many different ways (variety), and the same surface realization can express many different meanings based on context (ambiguity)\n",
    "    \n",
    "![human_language.png](https://github.com/TurkuNLP/intro-to-nlp/blob/master/figs/human_language.png?raw=1)\n",
    "    \n",
    "* NLP means different computer-based methods to analyze, understant or generate human language\n",
    "* Wikipedia: \"Natural language processing (NLP) is a subfield of linguistics, computer science, information engineering, and artificial intelligence concerned with the interactions between computers and human (natural) languages, in particular how to program computers to process and analyze large amounts of natural language data.\"\n",
    "\n",
    "## NLP Applications\n",
    "\n",
    "### Simple counting\n",
    "* How many words/characters an essay have?\n",
    "* Which are the most frequently used words in Finnish?\n",
    "\n",
    "### Spell Checker\n",
    "* Highlight spelling errors in text editors\n",
    "* Auto correction in mobile phones\n",
    "\n",
    "### Search Engines\n",
    "* Google, Bing, Baidu\n",
    "\n",
    "### Speech Recognition / Assistant\n",
    "* Speech-to-text\n",
    "* Apple Siri, Google Now\n",
    "\n",
    "### Machine Translation\n",
    "| ![machine_translation.png](https://github.com/TurkuNLP/intro-to-nlp/blob/master/figs/machine_translation.png?raw=1)|\n",
    "|:--:|\n",
    "| *Source: Google Translate* |\n",
    "\n",
    "### Text Generation\n",
    "| ![talktotransformer.png](https://github.com/TurkuNLP/intro-to-nlp/blob/master/figs/talktotransformer.png?raw=1) |\n",
    "|:--:|\n",
    "| *Source: https://app.inferkit.com/demo* |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "exS9UrNnWFMF"
   },
   "source": [
    "## 2. Getting Textual Data\n",
    "\n",
    "* Data is everywhere you can see written or spoken language\n",
    "* **Corpus** is a structured and documented collection of data\n",
    "    * Text Corpus = A collection of written text\n",
    "    * Speech corpus = A collection of spoken language (audio)\n",
    "* Preferably in an easily machine readable format\n",
    "* Raw text: A collection of unprocessed text\n",
    "\n",
    "![raw_text.png](https://github.com/TurkuNLP/intro-to-nlp/blob/master/figs/raw_text.png?raw=1)\n",
    "\n",
    "* Annotated text corpora: A collection of text with markings\n",
    "    * For each document/sentence, mark its language or topic\n",
    "    * For each word, mark its part-of-speech category\n",
    "    * etc.\n",
    "    * human or machine generated\n",
    "\n",
    "![annotated_text.png](https://github.com/TurkuNLP/intro-to-nlp/blob/master/figs/annotated_text.png?raw=1)\n",
    "![labeled_text.png](https://github.com/TurkuNLP/intro-to-nlp/blob/master/figs/labeled_text.png?raw=1)\n",
    "\n",
    "* Corpus properties\n",
    "    * Language\n",
    "    * Size\n",
    "    * Structure (E.g. Do we store full documents or individual sentences)\n",
    "    * Domain (Do we store certain type of documents, e.g. only news articles, or any text documents)\n",
    "        * Closed-domain, e.g. news corpus\n",
    "        * Open-domain, anything and everything, no boundaries\n",
    "    \n",
    "### Textual Data / Corpora sources: \n",
    "* **Web pages**\n",
    "    * crawler = internet bot that systematically browses the www by following hyperlinks\n",
    "        * Start with a list of seed URLs --> While visiting an URL, identify all hyperlinks and add them to the list of URLs to visit --> Continue until the list of URLs to visit is empty\n",
    "    * Web crawling / web scraping = save all data while browsing the www\n",
    "    * Remember: You need to be polite!\n",
    "        * Be aware of the local rules\n",
    "        * Tell who you are and why you are crawling\n",
    "        * Crawl delay!!!\n",
    "        * robots.txt\n",
    "* **News papers** (online or scanned paper versions)\n",
    "* **Discussion forums**\n",
    "* **Twitter**\n",
    "    * Crawling Twitter is super easy, has API and ready-made libraries to access it\n",
    "        * You need Twitter account, create an app and identification tokens for it\n",
    "        * Sign in developer.twitter.com → click apps → create an app → go to Keys and Tokens → click create\n",
    "        * To create new apps, you need to apply for a developer account or be invited to an existing educational team\n",
    "    * How to download tweets: https://github.com/jmnybl/notebook-examples/blob/master/twitter_dl.ipynb\n",
    "    \n",
    "|![twitter_interface.png](https://github.com/TurkuNLP/intro-to-nlp/blob/master/figs/twitter_interface.png?raw=1)|\n",
    "|:--:|\n",
    "| *Source: Twitter* |\n",
    "\n",
    "![twitter_json.png](https://github.com/TurkuNLP/intro-to-nlp/blob/master/figs/twitter_json.png?raw=1)\n",
    "    \n",
    "\n",
    "* **A lot of ready made (raw text and annotated) corpora exists!**\n",
    "    * Wikipedia: All Wikipedia articles for a language as one downloadable file (https://dumps.wikimedia.org/)\n",
    "    * Ready-made corpora for Finnish language: Kielipankki (https://www.kielipankki.fi/aineistot/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MTzVjBSuWFMH"
   },
   "source": [
    "### Example corpus: IMDB Dataset\n",
    "\n",
    "* IMDB Dataset (English) (http://ai.stanford.edu/~amaas/data/sentiment/index.html)\n",
    "    * A collection of IMDB movie reviews with a labeling to positive and negative reviews\n",
    "    * Negative review: Low score --> bad movie\n",
    "    * Positive review: High score --> good movie\n",
    "    \n",
    "| ![imdb-website.png](https://github.com/TurkuNLP/intro-to-nlp/blob/master/figs/imdb-website.png?raw=1) |\n",
    "|:--:|\n",
    "| *Source: imdb.com* |\n",
    "    \n",
    "* Let's inspect how to read the dataset!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "uMYk8jNxWFMJ",
    "outputId": "d46ca928-ca18-4fb8-8994-9a71093f4e65"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "File ‘imdb_train.json’ already there; not retrieving.\n",
      "\n",
      "Data type: <class 'list'>\n",
      "First item type: <class 'dict'>\n",
      "First item: {'class': 'pos', 'text': \"With all this stuff going down at the moment with MJ i've started listening to his music, watching the odd documentary here and there, watched The Wiz and watched Moonwalker again. Maybe i just want to get a certain insight into this guy who i thought was really cool in the eighties just to maybe make up my mind whether he is guilty or innocent. Moonwalker is part biography, part feature film which i remember going to see at the cinema when it was originally released. Some of it has subtle messages about MJ's feeling towards the press and also the obvious message of drugs are bad m'kay.  Visually impressive but of course this is all about Michael Jackson so unless you remotely like MJ in anyway then you are going to hate this and find it boring. Some may call MJ an egotist for consenting to the making of this movie BUT MJ and most of his fans would say that he made it for the fans which if true is really nice of him.  The actual feature film bit when it finally starts is only on for 20 minutes or so excluding the Smooth Criminal sequence and Joe Pesci is convincing as a psychopathic all powerful drug lord. Why he wants MJ dead so bad is beyond me. Because MJ overheard his plans? Nah, Joe Pesci's character ranted that he wanted people to know it is he who is supplying drugs etc so i dunno, maybe he just hates MJ's music.  Lots of cool things in this like MJ turning into a car and a robot and the whole Speed Demon sequence. Also, the director must have had the patience of a saint when it came to filming the kiddy Bad sequence as usually directors hate working with one kid let alone a whole bunch of them performing a complex dance scene.  Bottom line, this movie is for people who like MJ on one level or another (which i think is most people). If not, then stay away. It does try and give off a wholesome message and ironically MJ's bestest buddy in this movie is a girl! Michael Jackson is truly one of the most talented people ever to grace this planet but is he guilty? Well, with all the attention i've gave this subject....hmmm well i don't know because people can be different behind closed doors, i know this for a fact. He is either an extremely nice but stupid guy or one of the most sickest liars. I hope he is not the latter.\"}\n"
     ]
    }
   ],
   "source": [
    "!wget -nc https://github.com/TurkuNLP/intro-to-nlp/raw/master/Data/imdb_train.json\n",
    "\n",
    "import json # JSON encoder and decoder: store python data structures (e.g. lists and dictionaries) as text files\n",
    "\n",
    "with open(\"imdb_train.json\", \"rt\", encoding=\"utf-8\") as f:\n",
    "    data = json.load(f)\n",
    "    \n",
    "print(\"Data type:\", type(data))\n",
    "print(\"First item type:\", type(data[0]))\n",
    "print(\"First item:\", data[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "nqtcq1YmWFMN",
    "outputId": "359a7ad5-e6db-4d2c-f377-aaf13f14f1f4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of documents: 25000\n",
      "25000\n",
      "With all this stuff going down at the moment with MJ i've started listening to his music, watching the odd documentary here and there, watched The Wiz and watched Moonwalker again. Maybe i just want to get a certain insight into this guy who i thought was really cool in the eighties just to maybe make up my mind whether he is guilty or innocent. Moonwalker is part biography, part feature film which i remember going to see at the cinema when it was originally released. Some of it has subtle messages about MJ's feeling towards the press and also the obvious message of drugs are bad m'kay.  Visually impressive but of course this is all about Michael Jackson so unless you remotely like MJ in anyway then you are going to hate this and find it boring. Some may call MJ an egotist for consenting to the making of this movie BUT MJ and most of his fans would say that he made it for the fans which if true is really nice of him.  The actual feature film bit when it finally starts is only on for 20 minutes or so excluding the Smooth Criminal sequence and Joe Pesci is convincing as a psychopathic all powerful drug lord. Why he wants MJ dead so bad is beyond me. Because MJ overheard his plans? Nah, Joe Pesci's character ranted that he wanted people to know it is he who is supplying drugs etc so i dunno, maybe he just hates MJ's music.  Lots of cool things in this like MJ turning into a car and a robot and the whole Speed Demon sequence. Also, the director must have had the patience of a saint when it came to filming the kiddy Bad sequence as usually directors hate working with one kid let alone a whole bunch of them performing a complex dance scene.  Bottom line, this movie is for people who like MJ on one level or another (which i think is most people). If not, then stay away. It does try and give off a wholesome message and ironically MJ's bestest buddy in this movie is a girl! Michael Jackson is truly one of the most talented people ever to grace this planet but is he guilty? Well, with all the attention i've gave this subject....hmmm well i don't know because people can be different behind closed doors, i know this for a fact. He is either an extremely nice but stupid guy or one of the most sickest liars. I hope he is not the latter.\n"
     ]
    }
   ],
   "source": [
    "# How many documents the dataset have?\n",
    "print(\"Number of documents:\", len(data))\n",
    "\n",
    "documents = [document[\"text\"] for document in data] # right now we only need the text field for each document\n",
    "print(len(documents))\n",
    "print(documents[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ziutcOnEWFMO"
   },
   "source": [
    "## 3. Text Segmentation\n",
    "\n",
    "* **Segmentation:** Divide bigger units into smaller ones\n",
    "* In many cases text needs to be segmented into sentences and/or words\n",
    "* Why?\n",
    "\n",
    "\n",
    "* **Tokenization / word segmentation:** Segment text into individual tokens\n",
    "* **Sentence splitting / sentence segmentation:** Segment text into individual sentences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9TXhAZEGWFMQ"
   },
   "source": [
    "```\n",
    "Extremely bad customer service\n",
    "\n",
    "Do not go to this salon, especially if you have to get your hair straightened. They did a very bad job with my hair \n",
    "and were extremely rude when I went back to ask them why it didn't work for my hair. Rude, insensitive, discourteous people!!!!!\n",
    "```\n",
    "(*Text source: https://github.com/UniversalDependencies/UD_English-EWT*)\n",
    "\n",
    "\n",
    "**Tokenized:**\n",
    "```\n",
    "Extremely bad customer service\n",
    "\n",
    "Do not go to this salon , especially if you have to get your hair straightened . They did a very bad job with my hair \n",
    "and were extremely rude when I went back to ask them why it did n't work for my hair . Rude , insensitive , discourteous people !!!!!\n",
    "```\n",
    "\n",
    "**Sentence splitted:**\n",
    "```\n",
    "Extremely bad customer service\n",
    "\n",
    "Do not go to this salon, especially if you have to get your hair straightened.\n",
    "\n",
    "They did a very bad job with my hair and were extremely rude when I went back to ask them why it didn't work for my hair.\n",
    "\n",
    "Rude, insensitive, discourteous people!!!!!\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "E4dd60DpWFMR"
   },
   "source": [
    "### Tokenization: HOW?\n",
    "\n",
    "* **Naive method 1:** Split from whitespace characters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "tdfax4N4WFMR",
    "outputId": "ee9b3ca5-371b-4d14-c910-0db497a3d5ef"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extremely\n",
      "bad\n",
      "customer\n",
      "service\n",
      "Do\n",
      "not\n",
      "go\n",
      "to\n",
      "this\n",
      "salon,\n",
      "especially\n",
      "if\n",
      "you\n",
      "have\n",
      "to\n",
      "get\n",
      "your\n",
      "hair\n",
      "straightened.\n",
      "They\n",
      "did\n",
      "a\n",
      "very\n",
      "bad\n",
      "job\n",
      "with\n",
      "my\n",
      "hair\n",
      "and\n",
      "were\n",
      "extremely\n",
      "rude\n",
      "when\n",
      "I\n",
      "went\n",
      "back\n",
      "to\n",
      "ask\n",
      "them\n",
      "why\n",
      "it\n",
      "didn't\n",
      "work\n",
      "for\n",
      "my\n",
      "hair.\n",
      "Rude,\n",
      "insensitive,\n",
      "discourteous\n",
      "people!!!!!\n"
     ]
    }
   ],
   "source": [
    "text=\"\"\"Extremely bad customer service\n",
    "\n",
    "Do not go to this salon, especially if you have to get your hair straightened. \\\n",
    "They did a very bad job with my hair and were extremely rude when I went back to \\\n",
    "ask them why it didn't work for my hair. Rude, insensitive, discourteous people!!!!!\"\"\"\n",
    "\n",
    "tokenized_text = text.split() # split(): Return a list of the words in the string, using whitespace as the delimiter string.\n",
    "\n",
    "for w in tokenized_text:\n",
    "    print(w)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cXT_iPR2WFMS"
   },
   "source": [
    "* **Naive method 2:** Split from whitespace characters, take into account punctuation\n",
    "* Regular expressions!\n",
    "\n",
    "| ![email_validation_small.png](https://github.com/TurkuNLP/intro-to-nlp/blob/master/figs/email_validation_small.png?raw=1) | \n",
    "|:--:| \n",
    "| *Source: facebook.com* |\n",
    "\n",
    "\n",
    "    * Define search patters\n",
    "    * Find this kind of patterns from raw text, or find-and-replace if needed\n",
    "* Find all punctuation characters, and replace with whitespace+punctiation character\n",
    "    * *book.* --> *book .*\n",
    "    * *people!!!!!* --> *people !!!!!*\n",
    "    * How about clitics in English? [don't, can't, cannot?]\n",
    "    * 2-(14-hydroxypentadecyl)-4-methyl-5-oxo-2,5-dihydrofuran-3-carboxylic acid ???\n",
    "    * Usually it's not that important how exactly you do it, just be consistent!\n",
    "        * consistent = always do it the same way\n",
    "        * If you download two datasets which are already tokenized, the tokenization may differ and you need to be aware of it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "b0omWr9QWFMT",
    "outputId": "1ef41972-18f7-4c93-c213-9ae26c967076"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extremely bad customer service\n",
      "\n",
      "Do not go to this salon , especially if you have to get your hair straightened . They did a very bad job with my hair and were extremely rude when I went back to ask them why it did n't work for my hair . Rude , insensitive , discourteous people !!!!!\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "tokenized = re.sub(r'([.,!?]+)', r' \\1', text) # replace . , ! ? with whitespace+character(s), '+' means one or more\n",
    "tokenized = re.sub(r\"(n't)\", r\" \\1\", tokenized) # clitics\n",
    "\n",
    "print(tokenized) # Note: this is still string, apply simple whitespace splitting to get a list of tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WvF1NWsNWFMX"
   },
   "source": [
    "* **Naive method 2** works quite well for English, Finnish, Swedish...\n",
    "    * Approx. 97-99% correct on clean text\n",
    "    * Many existing tokenizers are just (a bunch of) regular expressions\n",
    "    * Can be hundreds of different rules...\n",
    "\n",
    "\n",
    "* How about other languages, does it work for all?\n",
    "\n",
    ".\n",
    "\n",
    ".\n",
    "\n",
    ".\n",
    "\n",
    ".\n",
    "\n",
    ".\n",
    "\n",
    ".\n",
    "\n",
    ".\n",
    "\n",
    ".\n",
    "\n",
    "**Nope! Why not?**\n",
    "\n",
    ".\n",
    "\n",
    ".\n",
    "\n",
    ".\n",
    "\n",
    ".\n",
    "\n",
    ".\n",
    "\n",
    ".\n",
    "\n",
    ".\n",
    "\n",
    "* All languages do not use whitespace or punctuation, or the meaning of those may be different.\n",
    "* Chinese, Thai, Vietnamese\n",
    "\n",
    "![tokenization.png](https://github.com/TurkuNLP/intro-to-nlp/blob/master/figs/tokenization.png?raw=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XfQmIvngWFMY"
   },
   "source": [
    "* **Naive algorithm:**\n",
    "    1) Build a vocabulary for the language\n",
    "    2) Start from the beginning of the text, and find the longest matching word\n",
    "    3) Split the matching word and continue from the next remaining character\n",
    "* *the table down there* --> *thetabledownthere* --> *theta bled own there*\n",
    "    * Does not work well for English, but in Chinese words are usually 2-4 characters long, so the simple algorithm works better\n",
    "    * Where to get the dictionary?\n",
    "    \n",
    "**Tokenization: State-of-the-art**\n",
    "* State-of-the-art = The best existing method currently known\n",
    "* Machine learning\n",
    "    * Collect raw (untokenized) text for the language you are interested in, and manually tokenize it.\n",
    "    * Train a classifier\n",
    "    * The trained classifier can be used to tokenize new text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5xMDiYq9WFMY"
   },
   "source": [
    "#### Sentence splitting: HOW?\n",
    "\n",
    "* **Naive method 1:** What kind of punctuation characters end the sentence?\n",
    "    * yes: . ! ?\n",
    "    * no: ,\n",
    "* Define a list of sentence-final punctuation, and always split on those.\n",
    "* Problems?\n",
    "\n",
    "![sentence_splitting.png](https://github.com/TurkuNLP/intro-to-nlp/blob/master/figs/sentence_splitting.png?raw=1)\n",
    "\n",
    "\n",
    "* **Solution 1:** Define a list of rules to identify when punctuation does not end a sentence\n",
    "    * List of known abbreviations, list of regular expression to regocnize numbers etc. (*The cost was approx. 1.5 million euros.*)\n",
    "    * How about missing punctuation? Other languages?\n",
    "    \n",
    "**Sentence splitting: State-of-the-art**\n",
    "* Machine learning\n",
    "    * Collect raw text for the language you are interested in, and manually sentence segment it.\n",
    "    * Train a classifier\n",
    "    * The trained classifier can be used to sentence segment new text\n",
    "    \n",
    "## Try UDPipe machine learned tokenizer and sentence splitter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ce6O2NqgWFMY",
    "outputId": "dfd88eca-ca71-4231-bd0e-4f64514dbd01"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "File ‘en.segmenter.udpipe’ already there; not retrieving.\n",
      "\n",
      "Requirement already satisfied: ufal.udpipe in /usr/local/lib/python3.6/dist-packages (1.2.0.3)\n",
      "With all this stuff going down at the moment with MJ i 've started listening to his music , watching the odd documentary here and there , watched The Wiz and watched Moonwalker again .\n",
      "Maybe i just want to get a certain insight into this guy who i thought was really cool in the eighties just to maybe make up my mind whether he is guilty or innocent .\n",
      "Moonwalker is part biography , part feature film which i remember going to see at the cinema when it was originally released .\n",
      "Some of it has subtle messages about MJ's feeling towards the press and also the obvious message of drugs are bad m'kay .\n",
      "Visually impressive but of course this is all about Michael Jackson so unless you remotely like MJ in anyway then you are going to hate this and find it boring .\n",
      "Some may call MJ an egotist for consenting to the making of this movie BUT MJ and most of his fans would say that he made it for the fans which if true is really nice of him .\n",
      "The actual feature film bit when it finally starts is only on for 20 minutes or so excluding the Smooth Criminal sequence and Joe Pesci is convincing as a psychopathic all powerful drug lord .\n",
      "Why he wants MJ dead so bad is beyond me .\n",
      "Because MJ overheard his plans ?\n",
      "Nah , Joe\n",
      "Pesci 's character ranted that he wanted people to know it is he who is supplying drugs etc so i dunno , maybe he just hates MJ's music .\n",
      "Lots of cool things in this like MJ turning into a car and a robot and the whole Speed Demon sequence .\n",
      "Also , the director must have had the patience of a saint when it came to filming the kiddy Bad sequence as usually directors hate working with one kid let alone a whole bunch of them performing a complex dance scene .\n",
      "Bottom line , this movie is for people who like MJ on one level or another ( which i think is most people ) .\n",
      "If not , then stay away .\n",
      "It does try and give off a wholesome message and ironically MJ's bestest buddy in this movie is a girl !\n",
      "Michael Jackson is truly one of the most talented people ever to grace this planet but is he guilty ?\n",
      "Well , with all the attention i 've gave this subject .... hmmm well i do n't know because people can be different behind closed doors , i know this for a fact .\n",
      "He is either an extremely nice but stupid guy or one of the most sickest liars .\n",
      "I hope he is not the latter .\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Let's try to tokenize and sentence split the IMDB data with UDPipe machine learned segmenter!\n",
    "# Documentation: https://ufal.mff.cuni.cz/udpipe/users-manual\n",
    "# Training data: \n",
    "# Finnish (intro-to-nlp/Data/fi.segmenter.udpipe): https://github.com/UniversalDependencies/UD_Finnish-TDT v.2.2\n",
    "# English (intro-to-nlp/Data/en.segmenter.udpipe): https://github.com/UniversalDependencies/UD_English-EWT v.2.2\n",
    "# Swedish (intro-to-nlp/Data/sv.segmenter.udpipe): https://github.com/UniversalDependencies/UD_Swedish-Talbanken v.2.2\n",
    "\n",
    "!wget -nc https://github.com/TurkuNLP/intro-to-nlp/raw/master/Data/en.segmenter.udpipe\n",
    "\n",
    "!pip3 install ufal.udpipe\n",
    "\n",
    "import ufal.udpipe as udpipe\n",
    "\n",
    "model = udpipe.Model.load(\"en.segmenter.udpipe\")\n",
    "pipeline = udpipe.Pipeline(model,\"tokenize\",\"none\",\"none\",\"horizontal\") # horizontal: returns one sentence per line, with words separated by a single space\n",
    "\n",
    "segmented_document = pipeline.process(documents[0])\n",
    "\n",
    "print(segmented_document)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "M9BtnHDeWFMZ"
   },
   "source": [
    "## 4. Word frequencies\n",
    "\n",
    "* How many times each word appears in the corpus?\n",
    "* How many unique words the corpus has?\n",
    "    * vocabulary size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "KaPTwF3NWFMa",
    "outputId": "00f7ac03-8415-472d-df25-7f85f2f0efc5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Most common tokens: [('the', 11464), (',', 10974), ('.', 10515), ('a', 6291), ('and', 6269), ('of', 5723), ('to', 5221), ('is', 4310), ('in', 3421), ('I', 3115), ('it', 3101), ('that', 2723), ('\"', 2633), (\"'s\", 2432), ('this', 2287), ('\\\\', 2250), ('was', 2013), ('-', 1980), ('with', 1812), ('as', 1711)]\n",
      "Vocabulary size: 21939\n"
     ]
    }
   ],
   "source": [
    "from collections import Counter\n",
    "\n",
    "token_counter = Counter()\n",
    "for doc in documents[:1000]: # IMDB documents\n",
    "    tokenized = pipeline.process(doc)\n",
    "    tokens = tokenized.split() # after segmenter, we can do whitespace splitting\n",
    "    token_counter.update(tokens)\n",
    "\n",
    "print(\"Most common tokens:\", token_counter.most_common(20))\n",
    "print(\"Vocabulary size:\", len(token_counter))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UI4Tme-MWFMb"
   },
   "source": [
    "### Stop words\n",
    "\n",
    "* Commonly used functional words with little semantic meaning\n",
    "* Typically the most frequent words in the corpus\n",
    "* The idea is to densify the data by removing these \"meaningless\" words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "cXQF3pzdWFMb",
    "outputId": "a92a53a4-9053-4e2d-e7d7-b653814a1f92"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of tokens: 47\n",
      "Tokens: [(\"'s\", 2432), ('film', 1630), ('movie', 1596), (\"n't\", 1237), ('one', 1004), ('like', 729), (\"'\", 685), ('good', 634), ('would', 527), ('time', 488), ('really', 445), ('even', 430), ('story', 425), ('see', 397), ('could', 383), ('get', 364), ('people', 361), ('much', 345), ('bad', 340), ('well', 334), ('great', 326), ('made', 311), ('first', 310), ('way', 307), ('make', 305), ('also', 299), ('think', 279), ('movies', 278), ('films', 275), ('characters', 275), ('many', 268), ('character', 267), ('show', 266), ('acting', 250), ('ever', 246), ('watch', 241), ('seen', 240), ('plot', 240), ('love', 229), ('never', 225), ('little', 220), ('best', 218), ('say', 217), ('two', 216), ('know', 214), ('life', 213), ('end', 206)]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to /home/jmnybl/nltk_data...\n",
      "[nltk_data]   Unzipping corpora/stopwords.zip.\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "nltk.download('stopwords') # download the stopwords dataset\n",
    "\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "# take 150 most common words from the IMDB corpus and filter out stop words and punctuation\n",
    "filtered_tokens = []\n",
    "punctuation_chars = '. , : ( ) ! ? \" = & - ; ... \\\\ '.split() # list of punctuation symbols to ignore\n",
    "for word, count in token_counter.most_common(150):\n",
    "    if word.lower() in stopwords.words(\"english\") or word in punctuation_chars:\n",
    "        continue\n",
    "    filtered_tokens.append((word, count))\n",
    "print(\"Number of tokens:\", len(filtered_tokens))\n",
    "print(\"Tokens:\", filtered_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dPqSB58sWFMc"
   },
   "source": [
    "Quotes from the internet search:\n",
    "\n",
    "* *A stop word is a commonly used word (such as “the”, “a”, “an”, “in”) that a search engine has been programmed to ignore.* (geeksforgeeks.org)\n",
    "* *Stop words are words which are filtered out before processing of natural language data (text).* (Wikipedia)\n",
    "\n",
    "**Not necessarily true with modern machine learning techniques!**\n",
    "\n",
    "* Another approach: Do not remove anything but give a higher importance to more meaningful words\n",
    "\n",
    "\n",
    "### tf-idf weighting\n",
    "\n",
    "* TF = term frequency *tf(t, d)*, how many times the term *t* appears in the **document** *d*\n",
    "* DF = document frequency *df(t)*, in how many documents (out of all documents) the term *t* appears\n",
    "* IDF = inverse document frequency, *m/df(t)*, where *m* is the total number of documents in your collection\n",
    "* TF-IDF = **tf(t, d) * idf(t)**\n",
    "    * Usually calculated using logaritmic scale --> tf(t, d) * log(idf(t)) or log(1 + tf(t,d)) * log(idf(t))\n",
    "    \n",
    "| ![log.png](https://github.com/TurkuNLP/intro-to-nlp/blob/master/figs/log.png?raw=1) |\n",
    "|:--:|\n",
    "| *Source: Wikipedia* |\n",
    "    \n",
    "* common in information retrieval, also used in document classification\n",
    "* scale down the impact of tokens that occur very frequently in many documents and are hence empirically less informative than words that occur in a small fraction of the documents\n",
    "\n",
    "### Examples of idf-weights calculated using natural logarithm (ln) and a Finnish corpus\n",
    "\n",
    "![idf.png](https://github.com/TurkuNLP/intro-to-nlp/blob/master/figs/idf.png?raw=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8Rv5xWXSWFMc"
   },
   "source": [
    "## 5. Text Normalization\n",
    "\n",
    "* Remove certain \"randomness\" from the data\n",
    "* Try to reduce uncommon cases\n",
    "* Normalization techniques involve:\n",
    "  * Tokenization\n",
    "  * Punctuation removal\n",
    "  * Capitalization / Lowercasing\n",
    "  * Accent removal\n",
    "  * Stemming / Lemmatization\n",
    "  * ...\n",
    "\n",
    "### Stemming and lemmatization\n",
    "\n",
    "* Densify data by removing inflectional variation\n",
    "\n",
    "* **Stemming:** Determine the word root by removing inflectional affixes \n",
    "    * play, plays, playing, played --> play\n",
    "    * activate, active, activated, activation --> activ\n",
    "    * koira, koiran, koiralla, koirilla --> koir\n",
    "    * koirasta --> koir\n",
    "* Risk of overstemming or understemming: two separate inflected words are stemmed to the same root, or inflections of the same word are stemmed to different roots\n",
    "* Does not take into account the context (lives --> live / life, koirasta --> koira / koiras)\n",
    "\n",
    "\n",
    "* **Lemmatization:** Determine the base (dictionary) form of the word\n",
    "    * play, plays, playing, played --> play\n",
    "    * activate, active, activated, activation --> activate, active, activate, activation\n",
    "    * koira, koiran, koiralla, koirilla --> koira\n",
    "    * koirasta --> koira / koiras\n",
    "* Generally better, but also computationally heavier and more complex method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "x2yfKSNkWFMd",
    "outputId": "72fd28bd-d391-4e49-a268-6bb7b5170783"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "with all this stuff go down at the moment with mj i'v start listen to his music, watch the odd documentari here and there, watch the wiz and watch moonwalk again. mayb i just want to get a certain insight into this guy who i thought was realli cool in the eighti just to mayb make up my mind whether he is guilti or innocent. moonwalk is part biography, part featur film which i rememb go to see at the cinema when it was origin released. some of it has subtl messag about mj feel toward the press and also the obvious messag of drug are bad m'kay. visual impress but of cours this is all about michael jackson so unless you remot like mj in anyway then you are go to hate this and find it boring. some may call mj an egotist for consent to the make of this movi but mj and most of his fan would say that he made it for the fan which if true is realli nice of him. the actual featur film bit when it final start is onli on for 20 minut or so exclud the smooth crimin sequenc and joe pesci is convinc as a psychopath all power drug lord. whi he want mj dead so bad is beyond me. becaus mj overheard his plans? nah, joe pesci charact rant that he want peopl to know it is he who is suppli drug etc so i dunno, mayb he just hate mj music. lot of cool thing in this like mj turn into a car and a robot and the whole speed demon sequence. also, the director must have had the patienc of a saint when it came to film the kiddi bad sequenc as usual director hate work with one kid let alon a whole bunch of them perform a complex danc scene. bottom line, this movi is for peopl who like mj on one level or anoth (which i think is most people). if not, then stay away. it doe tri and give off a wholesom messag and iron mj bestest buddi in this movi is a girl! michael jackson is truli one of the most talent peopl ever to grace this planet but is he guilty? well, with all the attent i'v gave this subject....hmmm well i don't know becaus peopl can be differ behind close doors, i know this for a fact. he is either an extrem nice but stupid guy or one of the most sickest liars. i hope he is not the latter.\n"
     ]
    }
   ],
   "source": [
    "from nltk.stem.snowball import SnowballStemmer\n",
    "\n",
    "stemmer = SnowballStemmer(\"english\")\n",
    "\n",
    "print(\" \".join(stemmer.stem(w) for w in documents[0].split()))\n"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "include_colab_link": true,
   "name": "basic_nlp.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}