t81_558_class_11_02_word2vec.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_11_02_word2vec.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 11: Natural Language Processing and Speech Recognition**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 11 Material\n",
    "\n",
    "* Part 11.1: Getting Started with Spacy in Python [[Video]](https://www.youtube.com/watch?v=A5BtU9vXzu8&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_11_01_spacy.ipynb)\n",
    "* **Part 11.2: Word2Vec and Text Classification** [[Video]](https://www.youtube.com/watch?v=nWxtRlpObIs&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_11_02_word2vec.ipynb)\n",
    "* Part 11.3: What are Embedding Layers in Keras [[Video]](https://www.youtube.com/watch?v=OuNH5kT-aD0&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_11_03_embedding.ipynb)\n",
    "* Part 11.4: Natural Language Processing with Spacy and Keras [[Video]](https://www.youtube.com/watch?v=BKgwjhao5DU&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_11_04_text_nlp.ipynb)\n",
    "* Part 11.5: Learning English from Scratch with Keras and TensorFlow [[Video]](https://www.youtube.com/watch?v=Y1khuuSjZzc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN&index=58) [[Notebook]](t81_558_class_11_05_english_scratch.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: not using Google CoLab\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    %tensorflow_version 2.x\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 11.2: Word2Vec and Text Classification\n",
    "\n",
    "Word2vec is a group of related models that data scientists use to produce word embeddings, which are numeric representations for words.  For example, a word embedding lexicon may provide a 100-number vector for each word in the English dictionary. Word2vec is one such embedding.\n",
    "\n",
    "Word2vec is implemented by shallow, two-layer neural networks that trained to reconstruct linguistic contexts of words. Word2vec takes as its input a large corpus of text and produces a vector space, typically of several hundred dimensions, with each unique word in the corpus assigned a corresponding vector in high dimension space. Similar words will have similar vectors in this high dimension space.[[Cite:mikolov2013efficient]](https://arxiv.org/abs/1301.3781)\n",
    "\n",
    "### Suggested Software for Word2Vec\n",
    "\n",
    "The following URLs provide useful software and data for working with Word2vec.\n",
    "\n",
    "* [GoogleNews Vectors](https://code.google.com/archive/p/word2vec/), [GitHub Mirror](https://github.com/mmihaltz/word2vec-GoogleNews-vectors)\n",
    "* [Python Gensim](https://radimrehurek.com/gensim/)\n",
    "\n",
    "The Python package Gensim is used in this chapter to work with word2vec vectors.  It is also necessary to load the embedding lookup table.  The following code can download this table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "C:\\Users\\jheaton\\.keras\\datasets\\GoogleNews-vectors-negative300.bin.gz\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.utils import get_file\n",
    "\n",
    "try:\n",
    "    path = get_file('GoogleNews-vectors-negative300.bin.gz', origin='https://s3.amazonaws.com/dl4j-distribution/GoogleNews-vectors-negative300.bin.gz')\n",
    "except:\n",
    "    print('Error downloading')\n",
    "    raise\n",
    "    \n",
    "print(path)    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code loads the vector lookup tables and prepares Gensim for use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gensim\n",
    "\n",
    "# Not that the path below refers to a location on my hard drive.\n",
    "# You should download GoogleNews Vectors (see suggested software above)\n",
    "model = gensim.models.KeyedVectors.load_word2vec_format(path, binary=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Word2vec makes each word a vector.  We are using the 300-number vector, which can be seen for the word \"hello\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "w = model['hello']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "300\n"
     ]
    }
   ],
   "source": [
    "print(len(w))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-0.05419922  0.01708984 -0.00527954  0.33203125 -0.25       -0.01397705\n",
      " -0.15039062 -0.265625    0.01647949  0.3828125  -0.03295898 -0.09716797\n",
      " -0.16308594 -0.04443359  0.00946045  0.18457031  0.03637695  0.16601562\n",
      "  0.36328125 -0.25585938  0.375       0.171875    0.21386719 -0.19921875\n",
      "  0.13085938 -0.07275391 -0.02819824  0.11621094  0.15332031  0.09082031\n",
      "  0.06787109 -0.0300293  -0.16894531 -0.20800781 -0.03710938 -0.22753906\n",
      "  0.26367188  0.012146    0.18359375  0.31054688 -0.10791016 -0.19140625\n",
      "  0.21582031  0.13183594 -0.03515625  0.18554688 -0.30859375  0.04785156\n",
      " -0.10986328  0.14355469 -0.43554688 -0.0378418   0.10839844  0.140625\n",
      " -0.10595703  0.26171875 -0.17089844  0.39453125  0.12597656 -0.27734375\n",
      " -0.28125     0.14746094 -0.20996094  0.02355957  0.18457031  0.00445557\n",
      " -0.27929688 -0.03637695 -0.29296875  0.19628906  0.20703125  0.2890625\n",
      " -0.20507812  0.06787109 -0.43164062 -0.10986328 -0.2578125  -0.02331543\n",
      "  0.11328125  0.23144531 -0.04418945  0.10839844 -0.2890625  -0.09521484\n",
      " -0.10351562 -0.0324707   0.07763672 -0.13378906  0.22949219  0.06298828\n",
      "  0.08349609  0.02929688 -0.11474609  0.00534058 -0.12988281  0.02514648\n",
      "  0.08789062  0.24511719 -0.11474609 -0.296875   -0.59375    -0.29492188\n",
      " -0.13378906  0.27734375 -0.04174805  0.11621094  0.28320312  0.00241089\n",
      "  0.13867188 -0.00683594 -0.30078125  0.16210938  0.01171875 -0.13867188\n",
      "  0.48828125  0.02880859  0.02416992  0.04736328  0.05859375 -0.23828125\n",
      "  0.02758789  0.05981445 -0.03857422  0.06933594  0.14941406 -0.10888672\n",
      " -0.07324219  0.08789062  0.27148438  0.06591797 -0.37890625 -0.26171875\n",
      " -0.13183594  0.09570312 -0.3125      0.10205078  0.03063965  0.23632812\n",
      "  0.00582886  0.27734375  0.20507812 -0.17871094 -0.31445312 -0.01586914\n",
      "  0.13964844  0.13574219  0.0390625  -0.29296875  0.234375   -0.33984375\n",
      " -0.11816406  0.10644531 -0.18457031 -0.02099609  0.02563477  0.25390625\n",
      "  0.07275391  0.13574219 -0.00138092 -0.2578125  -0.2890625   0.10107422\n",
      "  0.19238281 -0.04882812  0.27929688 -0.3359375  -0.07373047  0.01879883\n",
      " -0.10986328 -0.04614258  0.15722656  0.06689453 -0.03417969  0.16308594\n",
      "  0.08642578  0.44726562  0.02026367 -0.01977539  0.07958984  0.17773438\n",
      " -0.04370117 -0.00952148  0.16503906  0.17285156  0.23144531 -0.04272461\n",
      "  0.02355957  0.18359375 -0.41601562 -0.01745605  0.16796875  0.04736328\n",
      "  0.14257812  0.08496094  0.33984375  0.1484375  -0.34375    -0.14160156\n",
      " -0.06835938 -0.14648438 -0.02844238  0.07421875 -0.07666016  0.12695312\n",
      "  0.05859375 -0.07568359 -0.03344727  0.23632812 -0.16308594  0.16503906\n",
      "  0.1484375  -0.2421875  -0.3515625  -0.30664062  0.00491333  0.17675781\n",
      "  0.46289062  0.14257812 -0.25       -0.25976562  0.04370117  0.34960938\n",
      "  0.05957031  0.07617188 -0.02868652 -0.09667969 -0.01281738  0.05859375\n",
      " -0.22949219 -0.1953125  -0.12207031  0.20117188 -0.42382812  0.06005859\n",
      "  0.50390625  0.20898438  0.11230469 -0.06054688  0.33203125  0.07421875\n",
      " -0.05786133  0.11083984 -0.06494141  0.05639648  0.01757812  0.08398438\n",
      "  0.13769531  0.2578125   0.16796875 -0.16894531  0.01794434  0.16015625\n",
      "  0.26171875  0.31640625 -0.24804688  0.05371094 -0.0859375   0.17089844\n",
      " -0.39453125 -0.00156403 -0.07324219 -0.04614258 -0.16210938 -0.15722656\n",
      "  0.21289062 -0.15820312  0.04394531  0.28515625  0.01196289 -0.26953125\n",
      " -0.04370117  0.37109375  0.04663086 -0.19726562  0.3046875  -0.36523438\n",
      " -0.23632812  0.08056641 -0.04248047 -0.14648438 -0.06225586 -0.0534668\n",
      " -0.05664062  0.18945312  0.37109375 -0.22070312  0.04638672  0.02612305\n",
      " -0.11474609  0.265625   -0.02453613  0.11083984 -0.02514648 -0.12060547\n",
      "  0.05297852  0.07128906  0.00063705 -0.36523438 -0.13769531 -0.12890625]\n"
     ]
    }
   ],
   "source": [
    "print(w)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code below shows the distance between two words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.4796925\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "w1 = model['king']\n",
    "w2 = model['queen']\n",
    "\n",
    "dist = np.linalg.norm(w1-w2)\n",
    "\n",
    "print(dist)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This shows the classic word2vec equation of **queen = (king - man) + female**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('queen', 0.7118192911148071),\n",
       " ('monarch', 0.6189674139022827),\n",
       " ('princess', 0.5902431607246399),\n",
       " ('crown_prince', 0.5499460697174072),\n",
       " ('prince', 0.5377321243286133),\n",
       " ('kings', 0.5236844420433044),\n",
       " ('Queen_Consort', 0.5235945582389832),\n",
       " ('queens', 0.5181134343147278),\n",
       " ('sultan', 0.5098593235015869),\n",
       " ('monarchy', 0.5087411999702454)]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.most_similar(positive=['woman', 'king'], negative=['man'])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code shows which item does not belong with the others."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\jheaton\\Miniconda3\\envs\\tensorflow\\lib\\site-packages\\gensim\\models\\keyedvectors.py:877: FutureWarning: arrays to stack must be passed as a \"sequence\" type such as list or tuple. Support for non-sequence iterables such as generators is deprecated as of NumPy 1.16 and will raise an error in the future.\n",
      "  vectors = vstack(self.word_vec(word, use_norm=True) for word in used_words).astype(REAL)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'dog'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.doesnt_match(\"house garage store dog\".split())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code shows the similarity between two words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5633577"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.similarity('iphone', 'android')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code shows which words are most similar to the given one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('dogs', 0.8680490255355835),\n",
       " ('puppy', 0.8106428384780884),\n",
       " ('pit_bull', 0.780396044254303),\n",
       " ('pooch', 0.7627376317977905),\n",
       " ('cat', 0.7609457969665527),\n",
       " ('golden_retriever', 0.7500901818275452),\n",
       " ('German_shepherd', 0.7465174198150635),\n",
       " ('Rottweiler', 0.7437615394592285),\n",
       " ('beagle', 0.7418621778488159),\n",
       " ('pup', 0.7406911253929138)]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.most_similar('dog')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.7 (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}