ex07.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Exercise 07: TF-IDF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The following exercise uses results from our parsing to calculate a *term frequency - inverse document frequency* (TF-IDF) metric to construct *feature vectors* per document. First we'll load a *stopword* list, for common words to ignore from the analysis:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import pynlp\n",
    "\n",
    "stopwords = pynlp.load_stopwords(\"stop.txt\")\n",
    "print(stopwords)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Next, we'll use a function from our `pynlp` library to iterate through the keywords for one of the parsed HTML documents:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%sx ls *.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "json_file = \"a1.json\"\n",
    "\n",
    "for lex in pynlp.lex_iter(json_file):\n",
    "    print(lex)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We need to initialize some data structures for counting keywords. BTW, if you've heard about how Big Data projects use [word count](http://spark.apache.org/examples.html) programs to demonstrate their capabilities, here's a major use case for that. Even so, our examples are conceptually simple, built for relatively small files, and are not intended to scale:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "\n",
    "files = [\"a4.json\", \"a3.json\", \"a2.json\", \"a1.json\"]\n",
    "files_tf = {}\n",
    "\n",
    "d = len(files)\n",
    "df = defaultdict(int)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Iterate through each parsed file, tallying counts for `tf` for each document while also tallying counts for `df` across all documents:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "for json_file in files:\n",
    "    tf = defaultdict(int)\n",
    "\n",
    "    for lex in pynlp.lex_iter(json_file):\n",
    "        if (lex.pos != \".\") and (lex.root not in stopwords):\n",
    "            tf[lex.root] += 1\n",
    "\n",
    "    files_tf[json_file] = tf\n",
    "\n",
    "    for word in tf.keys():\n",
    "        df[word] += 1\n",
    "\n",
    "## print results for just the last file in the sequence\n",
    "print(json_file, files_tf[json_file])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Let's take a look at the `df` results overall. If there are low-information common words in the list that you'd like to filter out, move them to your *stopword* list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "for word, count in sorted(df.items(), key=lambda kv: kv[1], reverse=True):\n",
    "  print(word, count)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Finally, we make a second pass through the data, using the `df` counts to normalize `tf` counts, calculating the `tfidf` metrics for each keyword:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import math\n",
    "\n",
    "for json_file in files:\n",
    "    tf = files_tf[json_file]\n",
    "    keywords = []\n",
    "\n",
    "    for word, count in tf.items():\n",
    "        tfidf = float(count) * math.log((d + 1.0) / (df[word] + 1.0))\n",
    "        keywords.append((json_file, tfidf, word,))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Let's take a look at the results for one of the files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "for json_file, tfidf, word in sorted(keywords, key=lambda x: x[1], reverse=True):\n",
    "    print(\"%s\\t%7.4f\\t%s\" % (json_file, tfidf, word))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Question: how does that vector of ranked keywords compare with your reading of the text from the HTML file?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}