LAK 2022 - spaCy demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "02039c3f-6165-4ed3-ae35-94f825a440a4",
   "metadata": {},
   "source": [
    "*This notebook is released under a Creative Commons Attribution 4.0 International license ([https://creativecommons.org/licenses/by/4.0/](https://creativecommons.org/licenses/by/4.0/)).*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b42db6b5",
   "metadata": {},
   "source": [
    "# LAK 2022 spaCy demo\n",
    "\n",
    "[spaCy](https://spacy.io) is a general-purpose, opinionated, high-performance, and very modular NLP toolkit for Python.  It has various bindings in other languages, including R (via [spacyr](https://spacyr.quanteda.io/articles/using_spacyr.html)) and Julia (via the still-experimental [spaCy.jl](https://spacy.io/universe/project/spaCy.jl)).  This notebook contains a general demo of some of its most useful parts.\n",
    "\n",
    "**This notebook should be run inside a dedicated Anaconda environment.**  It will use the `conda` command-line tool to install all needed dependencies."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "969a4c50",
   "metadata": {},
   "source": [
    "## Notebook/environment setup\n",
    "\n",
    "Let's get the important stuff installed and set up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "00da7b99",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6a1366b",
   "metadata": {},
   "source": [
    "spaCy has support for GPU acceleration.  Run this cell if you have an NVidia GPU in your system that's CUDA-compatible.  If you're not sure, skip this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ace7da6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!conda install --yes --prefix {sys.prefix} \\\n",
    "    pytorch torchvision torchaudio cudatoolkit=11.3 conda-forge::cupy \\\n",
    "    -c pytorch"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5772aa3b",
   "metadata": {},
   "source": [
    "Install spaCy and other required libaries for running this notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9b1241da",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!conda install --yes --prefix {sys.prefix} -c conda-forge scikit-learn tqdm gensim"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b1188d7-20d7-477f-829c-f16924c65c11",
   "metadata": {},
   "source": [
    "Some setup for things we'll be using throughout the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c0d4f237-2cc0-43ed-8a03-803814817ce1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split, GridSearchCV\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from tqdm.notebook import tqdm as _tqdm\n",
    "\n",
    "def tqdm(*args, **kwargs):\n",
    "    return _tqdm(*args, ncols=1000, **kwargs)\n",
    "\n",
    "def fit_random_forest(x, y):\n",
    "    # Generate a train-test split, fit a Random Forest to the train split\n",
    "    # with default parameters, and evaluate its score on the test set.\n",
    "    train_x, test_x, train_y, test_y = train_test_split(\n",
    "        x, y,\n",
    "        train_size=0.8,\n",
    "        stratify=y,\n",
    "        random_state=0,\n",
    "    )\n",
    "    clf = RandomForestClassifier(n_jobs=2).fit(train_x, train_y)\n",
    "    print(f\"{type(clf).__name__} accuracy on test set: {clf.score(test_x, test_y):.2%}\")\n",
    "\n",
    "np.set_printoptions(threshold=100, linewidth=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a270e80-02ab-4c94-aa7e-713f0e2d65aa",
   "metadata": {},
   "source": [
    "# spaCy Quickstart\n",
    "\n",
    "Installing spaCy is easy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f423719c-419d-495a-8916-7b0367e51b92",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# Through conda--recommended\n",
    "!conda install --yes --prefix {sys.prefix} -c conda-forge spacy\n",
    "\n",
    "# Through pip--running on CPU\n",
    "# !{sys.executable} -m pip install -U spacy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37ee035c",
   "metadata": {},
   "source": [
    "The core of spaCy is the *model:* a processing and annotation pipeline for text.\n",
    "\n",
    "spaCy has [a lot of pre-trained models](https://spacy.io/models) available in several languages that you can easily download and start using.  All the models are drop-in replacements for one another: just download a new one, change what model is being loaded, and that's it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "754473ce-9b34-4c48-82e5-2d000cd0a7be",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# Download a spaCy model.\n",
    "# Replace en_core_web_lg with whatever model you want.\n",
    "!{sys.executable} -m spacy download en_core_web_lg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2a0ea3a1",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\andersonh\\Miniconda3\\envs\\LAK2022\\lib\\site-packages\\cupy\\_environment.py:213: UserWarning: CUDA path could not be detected. Set CUDA_PATH environment variable if CuPy fails to load.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "# Load the model for use.\n",
    "import spacy\n",
    "\n",
    "# Prefer running the model on GPU, if available.\n",
    "# spacy.prefer_gpu()\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_lg\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28ba4fc9",
   "metadata": {},
   "source": [
    "## Running the Pipeline\n",
    "\n",
    "To run a piece of text through the model, just call the model (like a function) on the string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "aacaefc1",
   "metadata": {},
   "outputs": [],
   "source": [
    "text = (\n",
    "    \"Grace Brewster Murray Hopper \"\n",
    "    \"was an American computer scientist and United States Navy rear admiral. One \"\n",
    "    \"of the first programmers of the Harvard Mark I computer, she was a pioneer of \"\n",
    "    \"computer programming who invented one of the first linkers. Hopper was the first \"\n",
    "    \"to devise the theory of machine-independent programming languages, and the \"\n",
    "    \"FLOW-MATIC programming language she created using this theory was later extended \"\n",
    "    \"to create COBOL, an early high-level programming language still in use today.\"\n",
    ")\n",
    "doc = nlp(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5450fd9",
   "metadata": {},
   "source": [
    "The `doc` object now contains a spaCy [`Doc` object](https://spacy.io/api/doc), which is a transformed and annotated version of the text.  We can interact with this object just like a Python list to access the token-level annotations.  The model is a pipeline of text transformation and annotation steps:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "305cdc87-5c63-4135-94b5-72e1ae55685c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tok2vec\n",
      "tagger\n",
      "parser\n",
      "attribute_ruler\n",
      "lemmatizer\n",
      "ner\n"
     ]
    }
   ],
   "source": [
    "for step_name, step_fn in nlp.pipeline:\n",
    "    print(step_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18219c59-2bc2-47c1-81a0-38c1dea2fc56",
   "metadata": {},
   "source": [
    "### Some of the available token-level annotations\n",
    "\n",
    "For a full list, see the [documentation on `Token` object.](https://spacy.io/api/token)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "8f7c23bd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TOKEN          LEMMA          PART OF SPEECH  STOPWORD? SYNTACTIC ROLE  SYNTACTIC HEAD  MORPHOLOGY\n",
      "Grace          Grace          PROPN           False     compound        Hopper          Number=Sing\n",
      "Brewster       Brewster       PROPN           False     compound        Hopper          Number=Sing\n",
      "Murray         Murray         PROPN           False     compound        Hopper          Number=Sing\n",
      "Hopper         Hopper         PROPN           False     nsubj           was             Number=Sing\n",
      "was            be             AUX             True      ROOT            was             Mood=Ind|Number=Sing|Person=3|Tense=Past|VerbForm=Fin\n",
      "an             an             DET             True      det             scientist       Definite=Ind|PronType=Art\n",
      "American       american       ADJ             False     amod            scientist       Degree=Pos\n",
      "computer       computer       NOUN            False     compound        scientist       Number=Sing\n",
      "scientist      scientist      NOUN            False     attr            was             Number=Sing\n",
      "and            and            CCONJ           True      cc              scientist       ConjType=Cmp\n"
     ]
    }
   ],
   "source": [
    "print(f\"{'TOKEN':<15}{'LEMMA':<15}{'PART OF SPEECH':<16}{'STOPWORD?':<10}\"\n",
    "      f\"{'SYNTACTIC ROLE':<16}{'SYNTACTIC HEAD':<16}{'MORPHOLOGY'}\")\n",
    "for tok in doc[:10]:\n",
    "    print(f\"{tok.text:<15}{tok.lemma_:<15}{tok.pos_:<16}{tok.is_stop!s:<10}\"\n",
    "          f\"{tok.dep_:<16}{tok.head.text:<16}{tok.morph}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffb2ae90",
   "metadata": {},
   "source": [
    "### Named entity recognition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "c8c71030",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ENTITY                        ENTITY TYPE\n",
      "Grace Brewster                ORG (Companies, agencies, institutions, etc.)\n",
      "Murray Hopper                 PERSON (People, including fictional)\n",
      "American                      NORP (Nationalities or religious or political groups)\n",
      "United States Navy            ORG (Companies, agencies, institutions, etc.)\n",
      "One                           CARDINAL (Numerals that do not fall under another type)\n",
      "one                           CARDINAL (Numerals that do not fall under another type)\n",
      "first                         ORDINAL (\"first\", \"second\", etc.)\n",
      "Hopper                        ORG (Companies, agencies, institutions, etc.)\n",
      "first                         ORDINAL (\"first\", \"second\", etc.)\n",
      "FLOW-MATIC                    ORG (Companies, agencies, institutions, etc.)\n",
      "COBOL                         ORG (Companies, agencies, institutions, etc.)\n",
      "today                         DATE (Absolute or relative dates or periods)\n"
     ]
    }
   ],
   "source": [
    "print(f\"{'ENTITY':<30}ENTITY TYPE\")\n",
    "for ent in doc.ents:\n",
    "    print(f\"{ent.text:<30}{ent.label_} ({spacy.explain(ent.label_)})\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "158c89fd-714f-4374-86cf-f40a7c2321a6",
   "metadata": {},
   "source": [
    "### Sentence tokenization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "f94e39db",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Grace Brewster Murray Hopper was an American computer scientist and United States Navy rear admiral.\n",
      "\n",
      "One of the first programmers of the Harvard Mark I computer, she was a pioneer of computer programming who invented one of the first linkers.\n",
      "\n",
      "Hopper was the first to devise the theory of machine-independent programming languages, and the FLOW-MATIC programming language she created using this theory was later extended to create COBOL, an early high-level programming language still in use today.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for sent in doc.sents:\n",
    "    print(sent)\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1975d120-3f46-4ecf-80c6-97bff8565cc8",
   "metadata": {},
   "source": [
    "### Similarity queries\n",
    "\n",
    "Similarities between `Doc` objects are between 0 and 1, with higher values indicating greater similarity.  The similarity is based on "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "aa6e8570-6ed3-41f6-bac4-328b69801886",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8016854705531046\n",
      "0.6165512506011797\n",
      "0.600524898806014\n"
     ]
    }
   ],
   "source": [
    "dog = nlp(\"dog\")\n",
    "cat = nlp(\"cat\")\n",
    "story = nlp(\"My pet won't stop clawing the couch!\")\n",
    "\n",
    "print(dog.similarity(cat))\n",
    "print(story.similarity(dog))\n",
    "print(story.similarity(cat))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bc9c34d-5b96-4391-af96-9ec545e4c6dd",
   "metadata": {},
   "source": [
    "### Text vectorization\n",
    "\n",
    "spaCy models apply vectors to individual token, spans of tokens, and whole documents, and stores them in the `.vector` attribute.  Different models have different approaches to vectorization--`en_core_web_lg` static GloVe vectors trained on Common Crawl; `*_trf` models use contextual vectors generated by Transformer neural networks; `*_sm` and `*_md` use token\n",
    "\n",
    "(the similarity queries we just saw are just the cosine similarities between the two pieces of text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "21f552bc-3fb9-478d-a455-df59775bc09b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Grace: [-0.25481  0.4372   0.21204 ...  0.18271 -0.45479 -0.18673]\n",
      "Brewster: [-0.56768  -0.26426   0.089199 ...  0.086851 -0.77606   0.24845 ]\n",
      "...\n",
      "Whole document:  [-0.01057081  0.12500678  0.02788484 ...  0.04178689 -0.10531644  0.08284231]\n"
     ]
    }
   ],
   "source": [
    "print(f\"{doc[0]}: {doc[0].vector}\")\n",
    "print(f\"{doc[1]}: {doc[1].vector}\")\n",
    "print(\"...\")\n",
    "print(f\"Whole document: \", doc.vector)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "815ab50f-c5a1-4158-8bd4-4623e0b36f09",
   "metadata": {},
   "source": [
    "## Processing lots of text at once\n",
    "\n",
    "`nlp.pipe()` is the easiest way to process lots of texts at once.  It supports parallelism and lazy-loading of the texts, and it returns a generator--so it's perfect for processing more texts than you can fit in RAM at once.  We can also disable processing steps that we don't want to apply to our texts, which can speed things up a lot for large document collections.  We can also specify a batch size to control how many documents are processed at once (this is more important when the model is running on a GPU, not so much when running on CPU).\n",
    "\n",
    "Note: when using multiple processes, there'll be a pretty big spike in memory use, and a small delay before things start running.  The model needs to be copied into each worker process that gets spawned, which can take some time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "40dbefe2-bc21-402a-b29f-18e522d77681",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<generator object Language.pipe at 0x000001F959D88C80>\n",
      "[I'm a document!, I, too, am a document!, Look at that, another document., Who keeps putting all these documents here?]\n"
     ]
    }
   ],
   "source": [
    "my_texts = [\n",
    "    \"I'm a document!\",\n",
    "    \"I, too, am a document!\",\n",
    "    \"Look at that, another document.\",\n",
    "    \"Who keeps putting all these documents here?\"\n",
    "]\n",
    "\n",
    "my_processed_texts = nlp.pipe(\n",
    "    my_texts,\n",
    "    n_process=1,           # single worker process\n",
    "    batch_size=256,        # buffer 256 documents per worker\n",
    "    disable=[              # Processing steps to disable/skip--for speed\n",
    "        \"tok2vec\",         # token vectorization\n",
    "        \"parser\",          # syntax parser\n",
    "        \"tagger\",          # part-of-speech tagger\n",
    "        \"ner\",             # named entity recognition\n",
    "        \"attribute_ruler\", # various rules-based transformarions\n",
    "        \"lemmatizer\",      # lemmatization\n",
    "    ]\n",
    ")\n",
    "\n",
    "# .pipe() returns a generator...\n",
    "print(my_processed_texts)\n",
    "\n",
    "# ...so we have to call list() or explicitly iterate through it\n",
    "# to get processed documents back.\n",
    "print(list(my_processed_texts))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7bee737d-c1d0-468a-83cb-3474e5151186",
   "metadata": {},
   "source": [
    "(to keep the code simple, the rest of the examples won't disable pipeline components or use multiple processes.  But feel free to experiment with changing these settings!)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb7ce714",
   "metadata": {},
   "source": [
    "# Text Classification\n",
    "spaCy is very useful for a wide range of text classification workflows.  We'll try three different approaches to the same binary classification task: predict whether an Amazon review was >3 or <=3 stars, based just on the review text.\n",
    "## Getting the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "fdede1d7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d6c055c383ee4011a0ee7b9a79296eba",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|                                                                                                         …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Downlod the data.  You'll need to do this manually if you don't have wget installed.\n",
    "# !wget --no-clobber http://snap.stanford.edu/data/amazon/productGraph/categoryFiles/reviews_Video_Games_5.json.gz\n",
    "\n",
    "# Now, process the reviews.\n",
    "import gzip\n",
    "import json\n",
    "import random\n",
    "\n",
    "# Extract text and create our classification target: True for >3 stars, False for <=3\n",
    "reviews = [\n",
    "    (i[\"reviewText\"], i[\"overall\"] > 3)\n",
    "    for i in map(\n",
    "        json.loads,\n",
    "        tqdm(gzip.open(\"reviews_Video_Games_5.json.gz\", \"rt\"), total=231780)\n",
    "    )\n",
    "]\n",
    "\n",
    "# Resample to a smaller subset for the sake of this demo.\n",
    "positive = [t for t in reviews if t[1] == True]\n",
    "negative = [t for t in reviews if t[1] == False]\n",
    "resampled_reviews = positive[:2_000] + negative[:2_000]\n",
    "random.shuffle(resampled_reviews)\n",
    "review_texts = [i[0] for i in resampled_reviews]\n",
    "review_scores = [i[1] for i in resampled_reviews]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59ec5374-c590-4cdd-921e-04bdb4ad0486",
   "metadata": {},
   "source": [
    "## Using spaCy's Text Vectorization\n",
    "\n",
    "One of the easiest approaches is to use spaCy's text vectors as features for a predictive model.  Since they're just numpy arrays, we can use them with `scikit-learn`, `keras`, `pytorch`, or any other library we want.\n",
    "\n",
    "`nlp.make_doc(text)` is probably the fastest way to get these vectors.  It will just run tokenization + vectorization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "7eca51a6-93cb-4f4e-938e-59484f170cd8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ef2cd594fc8943a88f647b77087fdb1d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|                                                                                                         …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.04479684  0.19694032 -0.13956733 ... -0.06383649 -0.00293173  0.03228827]\n",
      " [ 0.00834184  0.13011977 -0.07146262 ...  0.03144804  0.0060417   0.013466  ]\n",
      " [ 0.01021749  0.12195702 -0.07611904 ... -0.05591595 -0.00627276  0.0879147 ]\n",
      " ...\n",
      " [-0.04679825  0.18046853 -0.10175125 ... -0.08377992  0.00120563  0.08296575]\n",
      " [ 0.01141285  0.15055837 -0.14503597 ... -0.06690759  0.0610111   0.09047323]\n",
      " [-0.02274511  0.20885104 -0.1764832  ... -0.02571562  0.07243696  0.1297923 ]]\n",
      "(4000, 300)\n",
      "RandomForestClassifier accuracy on test set: 77.62%\n"
     ]
    }
   ],
   "source": [
    "# nlp.make_doc is the fastest way to get just the vector representation\n",
    "# of a document.  This will run the tokenizer and vectorizer components.\n",
    "text_vectors = np.array([\n",
    "    i.vector\n",
    "    for i in map(nlp.make_doc, tqdm(review_texts))\n",
    "])\n",
    "print(text_vectors)\n",
    "print(text_vectors.shape)\n",
    "fit_random_forest(text_vectors, review_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39d284f4-ece2-405b-a326-e1f8d5b26a34",
   "metadata": {},
   "source": [
    "## Using spaCy for Text Preprocessing\n",
    "\n",
    "Sometimes you need a bag-of-words model (e.g. for interpretability).  We can use spaCy to do some pretty fine-grained text preprocessing.  Let's lemmatize all our reviews and remove stopwords and punctuation tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "df85ce09-590a-4da7-85b4-e50618a6dc2a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "49d180a7419b40f29d8e80b7425b4bfd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|                                                                                                         …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before cleaning: I enjoyed Riven more than Myst. Riven was more difficult than Myst, for me, but I still only needed ...\n",
      "After cleaning:  enjoyed riven myst riven difficult myst needed hints complete riven highly recommend riven adventure...\n",
      "RandomForestClassifier accuracy on test set: 76.25%\n"
     ]
    }
   ],
   "source": [
    "# Lemmatize and remove stopwords+punctuation.  THis requires\n",
    "# running most of the spaCy model's pipelines to get these annoations.\n",
    "cleaned_texts = []\n",
    "to_disable=[\"tok2vec\", \"ner\"]\n",
    "processed_documents = nlp.pipe(tqdm(review_texts), disable=to_disable)\n",
    "for doc in processed_documents:\n",
    "    cleaned = [\n",
    "        tok.lemma_.lower()\n",
    "        for tok in doc\n",
    "        if not (tok.is_stop or tok.is_punct)\n",
    "    ]\n",
    "    cleaned = \" \".join(cleaned)\n",
    "    cleaned_texts.append(cleaned)\n",
    "    \n",
    "print(f\"Before cleaning: {review_texts[0][:100]}...\")\n",
    "print(f\"After cleaning:  {cleaned_texts[0][:100]}...\")\n",
    "\n",
    "# Now we feed this through some scikit-learn text preprocessing tools.\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "bow_texts = CountVectorizer().fit_transform(cleaned_texts)\n",
    "fit_random_forest(bow_texts, review_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ddb01c8-aa88-4185-b6c6-a3e2704f3d9a",
   "metadata": {},
   "source": [
    "We could also use the cleaned texts as inputs for topic models like LDA or LSA."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "bb5f1597-3acc-4f51-9251-0aab2caadc06",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Topic 0: game, time, like, games, mario, play, super, good, version, characters\n",
      "Topic 1: game, like, games, play, best, graphics, great, level, time, good\n",
      "Topic 2: game, games, like, play, mario, time, good, 2, graphics, best\n",
      "Topic 3: game, games, great, play, like, fun, graphics, time, good, best\n",
      "Topic 4: game, like, games, good, graphics, time, play, fun, way, better\n"
     ]
    }
   ],
   "source": [
    "from gensim.corpora import Dictionary\n",
    "from gensim.models.ldamodel import LdaModel\n",
    "corpus = [i.split() for i in cleaned_texts]\n",
    "id2word = Dictionary(corpus)\n",
    "corpus = [id2word.doc2bow(i) for i in corpus]\n",
    "\n",
    "model = LdaModel(corpus, num_topics=5, id2word=id2word)\n",
    "for i, words in model.show_topics(formatted=False):\n",
    "    print(f\"Topic {i}: {', '.join(i[0] for i in words)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0116aa89-e67a-4cb6-b9bd-d0ef42564691",
   "metadata": {},
   "source": [
    "(Gensim is another excellent library for a wide range of NLP tasks, more focused on unsupervised learning tasks like topic modeling and building word embedding models.  But we don't have time to go into it today)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "642c5dd9",
   "metadata": {},
   "source": [
    "## Training a spaCy Model for Text Classification\n",
    "\n",
    "We can also tweak the spaCy models to do the classification themselves, rather than just using them for feature extraction.  The steps required to do this:\n",
    "1. Generate a config file containing all of our training settings.\n",
    "2. Convert our data into one of spaCy's file formats.\n",
    "3. Train + evaluate the model.\n",
    "\n",
    "The config file makes it very easy to re-train our model in exactly the same way, and it has a *lot* of settings we could tweak.  Once the training is done, we get a spaCy model that we can `spacy.load()` and use like any other model to annotate texts with the categories we trained it on.\n",
    "\n",
    "### Generate the config files\n",
    "We'll generate a pretty basic config file, and let spaCy fill in sensible defaults for all the settings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "60831a59",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[i] Generated config template specific for your use case\n",
      "- Language: en\n",
      "- Pipeline: textcat\n",
      "- Optimize for: efficiency\n",
      "- Hardware: CPU\n",
      "- Transformer: None\n",
      "[+] Auto-filled config with all values\n",
      "[+] Saved config\n",
      "default_config.cfg\n",
      "You can now add your data and train your pipeline:\n",
      "python -m spacy train default_config.cfg --paths.train ./train.spacy --paths.dev ./dev.spacy\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\andersonh\\Miniconda3\\envs\\LAK2022\\lib\\site-packages\\cupy\\_environment.py:213: UserWarning: CUDA path could not be detected. Set CUDA_PATH environment variable if CuPy fails to load.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[!] Nothing to auto-fill: base config is already complete\n",
      "[+] Saved config\n",
      "config.cfg\n",
      "You can now add your data and train your pipeline:\n",
      "python -m spacy train config.cfg --paths.train ./train.spacy --paths.dev ./dev.spacy\n",
      "\n",
      "Config file contents:\n",
      "[paths]\n",
      "train = null\n",
      "dev = null\n",
      "vectors = null\n",
      "init_tok2vec = null\n",
      "\n",
      "[system]\n",
      "gpu_allocator = null\n",
      "seed = 0\n",
      "\n",
      "[nlp]\n",
      "lang = \"en\"\n",
      "pipeline = [\"textcat\"]\n",
      "batch_size = 1000\n",
      "disabled = []\n",
      "before_creation = null\n",
      "after_creation = null\n",
      "after_pipeline_creation = null\n",
      "tokenizer = {\"@tokenizers\":\"spacy.Tokenizer.v1\"}\n",
      "\n",
      "[components]\n",
      "\n",
      "[components.textcat]\n",
      "factory = \"textcat\"\n",
      "scorer = {\"@scorers\":\"spacy.textcat_scorer.v1\"}\n",
      "threshold = 0.5\n",
      "\n",
      "[components.textcat.model]\n",
      "@architectures = \"spacy.TextCatBOW.v2\"\n",
      "exclusive_classes = true\n",
      "ngram_size = 1\n",
      "no_output_layer = false\n",
      "nO = null\n",
      "\n",
      "[corpora]\n",
      "\n",
      "[corpora.dev]\n",
      "@readers = \"spacy.Corpus.v1\"\n",
      "path = ${paths.dev}\n",
      "max_length = 0\n",
      "gold_preproc = false\n",
      "limit = 0\n",
      "augmenter = null\n",
      "\n",
      "[corpora.train]\n",
      "@readers = \"spacy.Corpus.v1\"\n",
      "path = ${paths.train}\n",
      "max_length = 0\n",
      "gold_preproc = false\n",
      "limit = 0\n",
      "augmenter = null\n",
      "\n",
      "[training]\n",
      "dev_corpus = \"corpora.dev\"\n",
      "train_corpus = \"corpora.train\"\n",
      "seed = ${system.seed}\n",
      "gpu_allocator = ${system.gpu_allocator}\n",
      "dropout = 0.1\n",
      "accumulate_gradient = 1\n",
      "patience = 1600\n",
      "max_epochs = 0\n",
      "max_steps = 20000\n",
      "eval_frequency = 200\n",
      "frozen_components = []\n",
      "annotating_components = []\n",
      "before_to_disk = null\n",
      "\n",
      "[training.batcher]\n",
      "@batchers = \"spacy.batch_by_words.v1\"\n",
      "discard_oversize = false\n",
      "tolerance = 0.2\n",
      "get_length = null\n",
      "\n",
      "[training.batcher.size]\n",
      "@schedules = \"compounding.v1\"\n",
      "start = 100\n",
      "stop = 1000\n",
      "compound = 1.001\n",
      "t = 0.0\n",
      "\n",
      "[training.logger]\n",
      "@loggers = \"spacy.ConsoleLogger.v1\"\n",
      "progress_bar = false\n",
      "\n",
      "[training.optimizer]\n",
      "@optimizers = \"Adam.v1\"\n",
      "beta1 = 0.9\n",
      "beta2 = 0.999\n",
      "L2_is_weight_decay = true\n",
      "L2 = 0.01\n",
      "grad_clip = 1.0\n",
      "use_averages = false\n",
      "eps = 0.00000001\n",
      "learn_rate = 0.001\n",
      "\n",
      "[training.score_weights]\n",
      "cats_score = 1.0\n",
      "cats_score_desc = null\n",
      "cats_micro_p = null\n",
      "cats_micro_r = null\n",
      "cats_micro_f = null\n",
      "cats_macro_p = null\n",
      "cats_macro_r = null\n",
      "cats_macro_f = null\n",
      "cats_macro_auc = null\n",
      "cats_f_per_type = null\n",
      "cats_macro_auc_per_type = null\n",
      "\n",
      "[pretraining]\n",
      "\n",
      "[initialize]\n",
      "vectors = ${paths.vectors}\n",
      "init_tok2vec = ${paths.init_tok2vec}\n",
      "vocab_data = null\n",
      "lookups = null\n",
      "before_init = null\n",
      "after_init = null\n",
      "\n",
      "[initialize.components]\n",
      "\n",
      "[initialize.tokenizer]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\andersonh\\Miniconda3\\envs\\LAK2022\\lib\\site-packages\\cupy\\_environment.py:213: UserWarning: CUDA path could not be detected. Set CUDA_PATH environment variable if CuPy fails to load.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "# If you're running this at home: try changing `-o efficiency` to `-o accuracy`.\n",
    "# The model will be much larger and slower to train, but should be more accurate\n",
    "# (though it may not make much difference for how little data we're using).\n",
    "!{sys.executable} -m spacy init config \\\n",
    "    -F \\\n",
    "    -p textcat \\\n",
    "    -l en \\\n",
    "    -o efficiency \\\n",
    "    default_config.cfg\n",
    "\n",
    "# This does not always change the config, but usually it'll fill in sensible defaults.\n",
    "!{sys.executable} -m spacy init fill-config default_config.cfg config.cfg\n",
    "\n",
    "print()\n",
    "print(\"Config file contents:\")\n",
    "print(open(\"config.cfg\").read())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4274322c",
   "metadata": {},
   "source": [
    "### Convert documents to the right format\n",
    "We need to:\n",
    "1. Convert our texts to spaCy `Doc`s, but we only need to run tokenization (not any other steps).  `nlp.make_doc()` is a very fast way to do this.\n",
    "2. Add a `.cats` attribute, which should be a dictionar in the form: {category1: True, category2: False, ...}, where the category this document in is lbeled `True` and all others are `False`.\n",
    "3. Put all our documents into a `DocBin` so we can save it to disk.  (this is the format spaCy expects for training its models).\n",
    "\n",
    "The spaCy models are much more data hungry, so we'll use a bigger set of the reviews for this than we have for previous model examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "a5685a67",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2fb0d433b0814574a5748e5fb80ecfeb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|                                                                                                         …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import spacy\n",
    "\n",
    "# 10k positive and 10k negative examples\n",
    "# resampled_reviews = positive[:10_000] + negative[:10_000]\n",
    "# print(resampled_reviews)\n",
    "# random.shuffle(resampled_reviews)\n",
    "\n",
    "# Create the Document objects and set their .cats attributes\n",
    "docs = []\n",
    "for (text, category) in tqdm(resampled_reviews):\n",
    "    doc = nlp.make_doc(text)\n",
    "    doc.cats = {\"Positive\": category, \"Negative\": not category}\n",
    "    docs.append(doc)\n",
    "\n",
    "# train-dev-test split: 16k train, 2k each for dev/test\n",
    "# Create them as DocBin objects so we can easily save them to file.\n",
    "random.shuffle(docs)\n",
    "train = spacy.tokens.DocBin(docs=docs[:3200])\n",
    "dev   = spacy.tokens.DocBin(docs=docs[3200:3600])\n",
    "test  = spacy.tokens.DocBin(docs=docs[3600:])\n",
    "\n",
    "# Save the splits to file\n",
    "train.to_disk(\"data/train.spacy\")\n",
    "dev.to_disk(\"data/dev.spacy\")\n",
    "test.to_disk(\"data/test.spacy\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f39b99c",
   "metadata": {},
   "source": [
    "### Train + evaluate the model\n",
    "Now we can train the model.  We'll pass some arguments to override what might be in the config file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "9cf6653f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[i] Saving to output directory: text_categorization\n",
      "[i] Using GPU: 0\n",
      "\u001b[1m\n",
      "=========================== Initializing pipeline ===========================\u001b[0m\n",
      "[+] Initialized pipeline\n",
      "\u001b[1m\n",
      "============================= Training pipeline =============================\u001b[0m\n",
      "[i] Pipeline: ['textcat']\n",
      "[i] Initial learn rate: 0.001\n",
      "E    #       LOSS TEXTCAT  CATS_SCORE  SCORE \n",
      "---  ------  ------------  ----------  ------\n",
      "  0       0          0.12       31.15    0.31\n",
      "  0     200        102.42       68.80    0.69\n",
      "  0     400         81.19       76.38    0.76\n",
      "  0     600         68.26       80.04    0.80\n",
      "  0     800         76.99       76.66    0.77\n",
      "  0    1000         50.70       81.20    0.81\n",
      "  0    1200         62.01       80.71    0.81\n",
      "  0    1400         58.68       78.42    0.78\n",
      "  0    1600         43.77       79.73    0.80\n",
      "  0    1800         45.28       82.20    0.82\n",
      "  0    2000         40.09       80.99    0.81\n",
      "  0    2200         29.04       84.73    0.85\n",
      "  1    2400         37.40       85.12    0.85\n",
      "  1    2600          3.83       83.85    0.84\n",
      "  1    2800         15.59       83.49    0.83\n",
      "  1    3000          6.96       84.18    0.84\n",
      "  1    3200         11.52       83.19    0.83\n",
      "  1    3400          6.37       83.70    0.84\n",
      "  2    3600          0.70       82.21    0.82\n",
      "  2    3800          1.18       84.40    0.84\n",
      "  2    4000          0.92       83.15    0.83\n",
      "[+] Saved pipeline to output directory\n",
      "text_categorization\\model-last\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\andersonh\\Miniconda3\\envs\\LAK2022\\lib\\site-packages\\cupy\\_environment.py:213: UserWarning: CUDA path could not be detected. Set CUDA_PATH environment variable if CuPy fails to load.\n",
      "  warnings.warn(\n",
      "[2022-03-21 15:40:36,750] [INFO] Set up nlp object from config\n",
      "[2022-03-21 15:40:36,759] [INFO] Pipeline: ['textcat']\n",
      "[2022-03-21 15:40:36,763] [INFO] Created vocabulary\n",
      "[2022-03-21 15:40:36,763] [INFO] Finished initializing nlp object\n",
      "[2022-03-21 15:40:48,604] [INFO] Initialized pipeline components: ['textcat']\n"
     ]
    }
   ],
   "source": [
    "!{sys.executable} -m spacy train \\\n",
    "    config.cfg \\\n",
    "    --paths.train ./data/train.spacy \\\n",
    "    --paths.dev ./data/dev.spacy \\\n",
    "    --output ./text_categorization \\\n",
    "    --gpu-id 0 \\\n",
    "    --nlp.batch_size 1024"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f59980fc",
   "metadata": {},
   "source": [
    "Evaluate the model's performance on the testing set, and save the results to a JSON file for later reference:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "5b104c21",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Contents of the evaluation data file:\n",
      "[i] Using GPU: 0\n",
      "\u001b[1m\n",
      "================================== Results ==================================\u001b[0m\n",
      "\n",
      "TOK                 100.00\n",
      "TEXTCAT (macro F)   82.24 \n",
      "SPEED               232736\n",
      "\n",
      "\u001b[1m\n",
      "=========================== Textcat F (per label) ===========================\u001b[0m\n",
      "\n",
      "               P       R       F\n",
      "Positive   85.28   80.00   82.56\n",
      "Negative   79.31   84.74   81.93\n",
      "\n",
      "\u001b[1m\n",
      "======================== Textcat ROC AUC (per label) ========================\u001b[0m\n",
      "\n",
      "           ROC AUC\n",
      "Positive      0.89\n",
      "Negative      0.89\n",
      "\n",
      "[+] Saved results to TestSetEvaluation.json\n",
      "{'cats_auc_per_type': {'Negative': 0.8933333333, 'Positive': 0.8932581454},\n",
      " 'cats_f_per_type': {'Negative': {'f': 0.8193384224,\n",
      "                                  'p': 0.7931034483,\n",
      "                                  'r': 0.8473684211},\n",
      "                     'Positive': {'f': 0.8255528256,\n",
      "                                  'p': 0.8527918782,\n",
      "                                  'r': 0.8}},\n",
      " 'cats_macro_auc': 0.8932957393,\n",
      " 'cats_macro_f': 0.822445624,\n",
      " 'cats_macro_p': 0.8229476632,\n",
      " 'cats_macro_r': 0.8236842105,\n",
      " 'cats_micro_f': 0.8225,\n",
      " 'cats_micro_p': 0.8225,\n",
      " 'cats_micro_r': 0.8225,\n",
      " 'cats_score': 0.822445624,\n",
      " 'cats_score_desc': 'macro F',\n",
      " 'speed': 232735.6422415499,\n",
      " 'token_acc': 1.0,\n",
      " 'token_f': 1.0,\n",
      " 'token_p': 1.0,\n",
      " 'token_r': 1.0}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\andersonh\\Miniconda3\\envs\\LAK2022\\lib\\site-packages\\cupy\\_environment.py:213: UserWarning: CUDA path could not be detected. Set CUDA_PATH environment variable if CuPy fails to load.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "!{sys.executable} -m spacy evaluate \\\n",
    "    ./text_categorization/model-best \\\n",
    "    ./data/test.spacy \\\n",
    "    --gpu-id 0 \\\n",
    "    -o TestSetEvaluation.json\n",
    "\n",
    "from pprint import pprint\n",
    "print()\n",
    "print(\"Contents of the evaluation data file:\")\n",
    "pprint(json.load(open(\"TestSetEvaluation.json\")))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d092d1da",
   "metadata": {},
   "source": [
    "Now, we can load the `model-best` file from the output folder, just like any other spaCy model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "a18f6368",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'Positive': 0.07472464442253113, 'Negative': 0.9252753257751465}\n",
      "{'Positive': 0.5814142823219299, 'Negative': 0.41858571767807007}\n"
     ]
    }
   ],
   "source": [
    "my_model = spacy.load(\"./text_categorization/model-best\")\n",
    "\n",
    "negative_example = (\n",
    "    \"I hate this terrible, awful, garbage game.  \"\n",
    "    \"This is the worst thing I've ever played.  \"\n",
    "    \"The developers should be absolutely ashamed of themselves.\"\n",
    ")\n",
    "positive_example = (\n",
    "    \"Wow, this might be my favorite game ever!  \"\n",
    "    \"I can't believe how much fun I'm having.  This game rules.  \"\n",
    "    \"Definitely a contender for Game of the Year for me.\"\n",
    ")\n",
    "\n",
    "print(my_model(negative_example).cats)\n",
    "print(my_model(positive_example).cats)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae7db4d6",
   "metadata": {},
   "source": [
    "## The spaCy universe\n",
    "\n",
    "It's fairly easy to write custom pipeline components that do all sorts of things to documents, like adding customized annotations and functionality.  There is a long, but non-exhaustive list, on [the spaCy site.](https://spacy.io/universe)\n",
    "\n",
    "One that I use a lot is PyTextRank, which is an implementation of the TextRank algorithm for automatic text summarization.  It's very easy to use (but it's not available through `conda`, so we have to `pip install` it):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e1c22472",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "!{sys.executable} -m pip install --user pytextrank"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "74a11de1",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<pytextrank.base.BaseTextRankFactory at 0x1f959d7fbe0>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import spacy\n",
    "import pytextrank\n",
    "nlp_textrank = spacy.load(\"en_core_web_lg\")\n",
    "nlp_textrank.add_pipe(\"textrank\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3455d61f",
   "metadata": {},
   "source": [
    "And that's it!  Now we just run a document through the `nlp` model and access the special methods and attributes that get added.  By convention, spaCy extensions all add attributes and methods to `doc._.something`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "4b24f6b3",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SENTENCE 1:\n",
      "I started playing games on my laptop and bought a few new games to build my collection.\n",
      "\n",
      "SENTENCE 2:\n",
      "Game plays well and looks gorgeous when image spanned across three monitors.\n",
      "\n",
      "SENTENCE 3:\n",
      "My lady dug out her kids old gamecube and games and started playing her old favorite Harvest Moon titles.\n",
      "\n",
      "SENTENCE 4:\n",
      "My living room is fairly large, so these chords make it nice to play games which require the old controllers.\n",
      "\n",
      "SENTENCE 5:\n",
      "Innovation is necessary, and appreciated, but games like BOF3 need never disappear.\n",
      "\n",
      "SENTENCE 6:\n",
      "it good play game I guess well\n",
      "\n",
      "SENTENCE 7:\n",
      "With all the complaints aside, Breath of Fire 3 is fun game to play through.\n",
      "\n",
      "SENTENCE 8:\n",
      "Now  I still play SNES, playing such good games as Super Godzilla, Breath of  Fire I, and the Final Fantasy series.\n",
      "\n",
      "SENTENCE 9:\n",
      "If you like racing games you should check this out.\n",
      "\n",
      "SENTENCE 10:\n",
      "If you enjoy racing games, this is not one you should ignore.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "to_summarize = \"\\n\".join([i[0] for i in positive[:100]])\n",
    "\n",
    "textrank = nlp_textrank(to_summarize)._.textrank\n",
    "summary = textrank.summary(limit_phrases=10, limit_sentences=10)\n",
    "for (i, sent) in enumerate(summary):\n",
    "    print(f\"SENTENCE {i+1}:\")\n",
    "    print(sent.text.strip())\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6975ad9",
   "metadata": {},
   "source": [
    "# Some things not covered\n",
    "\n",
    "spaCy is a surprisingly deep library, and there's a lot I didn't cover.  A few big things:\n",
    "- Not all models have exactly the same capabilities, and some prioritise speed over accuracy.  The website has pretty thorough documentation on model specs, so you can choose the best pre-trained model for your work.\n",
    "- You can use different word vectors if you want.  You can provide static word vectors or pre-train your model to get better starting values for the embedding layers.\n",
    "- spaCy Projects give nice ways to package up an entire workflow and make it easy to redistribute.\n",
    "- You can train your own part-of-speech tagger, syntactic analyzer, named entity recognition, etc. components, as long as you have enough annotated data.\n",
    "- You can write your own pipeline components\n",
    "- You can easily interface with various tranformer models from Huggingface's `transformers` library by installing `spacy-transformers`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1fa0323-c34f-4ea3-a79f-e0842eacd59d",
   "metadata": {},
   "source": [
    "# Contact\n",
    "\n",
    "Henry Anderson ([henry.anderson@uta.edu](mailto:henry.anderson@uta.edu)]"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}