task_8_transitions_solved.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyMxlVACuP3aA/RpLJWsqD5Q",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/TurkuNLP/intro-to-nlp/blob/master/task_8_transitions_solved.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Task 8: POS transition probabilities\n",
        "\n",
        "In the lecture, we briefly saw the concept of hidden markov models and transition probabilies, as applied to POS tags. In the simplest case, these probabilities model the sequences of POS tag pairs such that e.g. probability of DET -> NOUN will be the probability of seeing a NOUN, having seen a DET (determiner), i.e. more formally P(NOUN|DET). We also had the intuition that for example DET -> NOUN should be much larger than, say DET -> VERB. And of course, since these are probabilities, sum of P(x|y) over all x should sum up\n",
        "to 1 for any given y. These probabilities can be easily estimated by counting from the data, i.e. the probability of DET -> NOUN transition, i.e. P(NOUN|DET) is simply the count of how many times you saw NOUN following a DET, divided by how many times you saw DET.\n",
        "\n",
        "Your task is to pick a Universal Dependencies dataset of your choice, e.g. UD_English-EWT training data, calculate these transition probabilities, pretty-print them if you can, and check that our intuitions hold, i.e. that for example DET -> NOUN is substantially more likely than, say, DET -> VERB.\n",
        "\n"
      ],
      "metadata": {
        "id": "-OJBvK7u7PT6"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MyPHNDWjtwB8"
      },
      "outputs": [],
      "source": [
        "# Grab the data\n",
        "\n",
        "!wget https://github.com/UniversalDependencies/UD_English-EWT/archive/refs/heads/master.zip\n",
        "!unzip master.zip"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# I guess this conllu library is quite useful after all\n",
        "\n",
        "!pip install conllu"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rgoEzQoJulk-",
        "outputId": "7850ff88-f720-4102-de84-c04e13339b95"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting conllu\n",
            "  Downloading conllu-4.5.3-py2.py3-none-any.whl (16 kB)\n",
            "Installing collected packages: conllu\n",
            "Successfully installed conllu-4.5.3\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import conllu\n",
        "inp_data=\"UD_English-EWT-master/en_ewt-ud-train.conllu\"\n",
        "conllu_data=conllu.parse(open(inp_data).read())"
      ],
      "metadata": {
        "id": "HSomZ9g8uDCo"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Let's get the counts into a numpy 2D array, but\n",
        "# of course there are very many ways to achieve\n",
        "# this same job\n",
        "\n",
        "import numpy as np\n",
        "import itertools\n",
        "\n",
        "# I grabbed this list from the documentation on the UD pages\n",
        "# and I add \"START\" tag to mark the start of the sentence\n",
        "all_tags= \"START ADJ ADP PUNCT ADV AUX SYM INTJ CCONJ X NOUN DET PROPN NUM VERB PART PRON SCONJ _\".split()\n",
        "\n",
        "def idx(tag):\n",
        "    \"\"\"Utility function which turns a tag into a 0-based integer index\"\"\"\n",
        "    return all_tags.index(tag)\n",
        "\n",
        "counts=np.zeros((len(all_tags),len(all_tags)),dtype=float) #0-filled array of the correct size\n",
        "\n",
        "for sentence in conllu_data:\n",
        "    #get the sequence of POS tags in the sentence, pre-pended with START\n",
        "    pos_sequence=[\"START\"]+[token['upos'] for token in sentence]\n",
        "    #now loop over all tag pairs - there are so many ways to do that, this one is quite elegant I think\n",
        "    for pos_from, pos_to in itertools.pairwise(pos_sequence):\n",
        "        pos_from_i=idx(pos_from)\n",
        "        pos_to_i=idx(pos_to)\n",
        "        counts[pos_from_i,pos_to_i]+=1\n",
        "\n",
        "probs=counts/np.sum(counts,axis=1)[:,None] #normalize into probabilities by dividing with row sums\n",
        "\n",
        "# Print a few to see if they make any sense, and they do\n",
        "print(\"DET -> NOUN\", probs[idx(\"DET\"),idx(\"NOUN\")])\n",
        "print(\"DET -> VERB\", probs[idx(\"DET\"),idx(\"VERB\")])\n",
        "print(\"DET -> DET\", probs[idx(\"DET\"),idx(\"DET\")])\n",
        "print(\"AUX -> VERB\", probs[idx(\"AUX\"),idx(\"VERB\")])\n",
        "print(\"VERB -> AUX\", probs[idx(\"VERB\"),idx(\"AUX\")])\n",
        "\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JU-fa73wt7NO",
        "outputId": "70086060-f34a-4131-fe66-5e8a334bc68b"
      },
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "DET -> NOUN 0.5875207067918278\n",
            "DET -> VERB 0.018958218295600956\n",
            "DET -> DET 0.009877906620038039\n",
            "AUX -> VERB 0.33824561403508774\n",
            "VERB -> AUX 0.006738783472246853\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Let's make sure we don't repeat the mistake on division\n",
        "array=np.array([[1,2],[3,4]])\n",
        "print(\"sum1\",np.sum(array))\n",
        "print(\"sum2\",np.sum(array,0))\n",
        "print(\"sum3\",np.sum(array,1))\n",
        "sums=np.sum(array,1)\n",
        "print(\"Sums as row vector\",sums)\n",
        "print(\"Sums as column vector\", sums[:,None])\n",
        "print(\"division with sums as row vector\", array/sums) ## Sums interpreted as row vector\n",
        "print(\"division with sums as column vector\", array/sums[:,None]) #Make the sums into a column vector\n",
        "\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2kfzpIB8e79g",
        "outputId": "7fe6841a-7da6-4d34-d421-c3c1ac5f7b78"
      },
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "sum1 10\n",
            "sum2 [4 6]\n",
            "sum3 [3 7]\n",
            "Sums as row vector [3 7]\n",
            "Sums as column vector [[3]\n",
            " [7]]\n",
            "division with sums as row vector [[0.33333333 0.28571429]\n",
            " [1.         0.57142857]]\n",
            "division with sums as column vector [[0.33333333 0.66666667]\n",
            " [0.42857143 0.57142857]]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Now we can do some pretty-printing\n",
        "# this is what google thought to be a good way, can't argue with that :)\n",
        "import pandas\n",
        "df = pandas.DataFrame(probs, columns=all_tags, index=all_tags)\n",
        "df"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 665
        },
        "id": "Q13C7TzB4dyt",
        "outputId": "fe5e81cb-ebd5-409d-8f19-b6b43286cca0"
      },
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "       START       ADJ       ADP     PUNCT       ADV       AUX       SYM  \\\n",
              "START    0.0  0.040816  0.043686  0.035236  0.075016  0.025351  0.007733   \n",
              "ADJ      0.0  0.054495  0.076782  0.129904  0.013815  0.003358  0.001526   \n",
              "ADP      0.0  0.071204  0.030347  0.024165  0.015960  0.000843  0.003316   \n",
              "PUNCT    0.0  0.053647  0.050289  0.084960  0.061534  0.027565  0.005388   \n",
              "ADV      0.0  0.141085  0.091577  0.171644  0.089295  0.040976  0.002976   \n",
              "AUX      0.0  0.106589  0.032827  0.016608  0.140741  0.079064  0.000780   \n",
              "SYM      0.0  0.041074  0.011058  0.044234  0.004739  0.004739  0.017378   \n",
              "INTJ     0.0  0.001477  0.008863  0.385524  0.033973  0.019202  0.002954   \n",
              "CCONJ    0.0  0.087659  0.027973  0.013014  0.081975  0.049065  0.004039   \n",
              "X        0.0  0.016548  0.044917  0.264775  0.002364  0.004728  0.000000   \n",
              "NOUN     0.0  0.011413  0.209989  0.293733  0.035728  0.073236  0.003532   \n",
              "DET      0.0  0.234002  0.011473  0.012823  0.015768  0.001595  0.002147   \n",
              "PROPN    0.0  0.014374  0.068617  0.267368  0.018138  0.067334  0.003251   \n",
              "NUM      0.0  0.034500  0.081410  0.281708  0.007694  0.011417  0.033755   \n",
              "VERB     0.0  0.044467  0.184918  0.078959  0.069206  0.006739  0.001951   \n",
              "PART     0.0  0.040028  0.007832  0.019318  0.046467  0.070310  0.000522   \n",
              "PRON     0.0  0.038272  0.038379  0.055985  0.051100  0.298658  0.001020   \n",
              "SCONJ    0.0  0.025527  0.013024  0.009638  0.030216  0.015369  0.000000   \n",
              "_        0.0  0.002679  0.000765  0.000000  0.007271  0.297742  0.000000   \n",
              "\n",
              "           INTJ     CCONJ         X      NOUN       DET     PROPN       NUM  \\\n",
              "START  0.032286  0.023198  0.012277  0.062022  0.100446  0.111687  0.039142   \n",
              "ADJ    0.000305  0.042894  0.001221  0.517631  0.005037  0.065639  0.008319   \n",
              "ADP    0.000281  0.005845  0.001293  0.162190  0.358379  0.132910  0.040238   \n",
              "PUNCT  0.011479  0.109402  0.008746  0.127050  0.064032  0.098001  0.032563   \n",
              "ADV    0.000992  0.026094  0.001290  0.014188  0.045838  0.009227  0.018256   \n",
              "AUX    0.000312  0.001949  0.000234  0.011852  0.080312  0.006394  0.009045   \n",
              "SYM    0.001580  0.039494  0.000000  0.137441  0.006319  0.041074  0.609795   \n",
              "INTJ   0.023634  0.011817  0.000000  0.019202  0.014771  0.025111  0.008863   \n",
              "CCONJ  0.002992  0.000150  0.000598  0.145400  0.097233  0.075991  0.014361   \n",
              "X      0.000000  0.011820  0.513002  0.068558  0.007092  0.018913  0.007092   \n",
              "NOUN   0.000584  0.072098  0.001547  0.125427  0.007414  0.008115  0.005546   \n",
              "DET    0.000000  0.000614  0.000675  0.587521  0.009878  0.075710  0.015154   \n",
              "PROPN  0.000171  0.066051  0.000342  0.085387  0.005048  0.228953  0.035763   \n",
              "NUM    0.000248  0.029536  0.003227  0.356664  0.004468  0.040457  0.086126   \n",
              "VERB   0.000975  0.017512  0.000355  0.078826  0.185006  0.023231  0.012680   \n",
              "PART   0.000000  0.002436  0.000000  0.076923  0.014619  0.010964  0.003133   \n",
              "PRON   0.001610  0.011326  0.000483  0.130864  0.018894  0.006280  0.003435   \n",
              "SCONJ  0.001563  0.003126  0.000260  0.036207  0.133889  0.054181  0.007815   \n",
              "_      0.000000  0.000000  0.000000  0.091466  0.000765  0.171068  0.000765   \n",
              "\n",
              "           VERB      PART      PRON     SCONJ         _  \n",
              "START  0.059630  0.004703  0.251993  0.035555  0.039222  \n",
              "ADJ    0.008930  0.031598  0.011830  0.022897  0.003816  \n",
              "ADP    0.006463  0.000843  0.133191  0.003091  0.009441  \n",
              "PUNCT  0.081446  0.010542  0.122677  0.030923  0.019756  \n",
              "ADV    0.195357  0.017660  0.083639  0.035619  0.014287  \n",
              "AUX    0.338246  0.118441  0.046628  0.008265  0.001715  \n",
              "SYM    0.022117  0.001580  0.012638  0.003160  0.001580  \n",
              "INTJ   0.354505  0.000000  0.073855  0.004431  0.011817  \n",
              "CCONJ  0.173373  0.015707  0.167091  0.017053  0.026328  \n",
              "X      0.009456  0.002364  0.021277  0.004728  0.002364  \n",
              "NOUN   0.054117  0.028810  0.039464  0.024548  0.004699  \n",
              "DET    0.018958  0.000368  0.005031  0.000123  0.008160  \n",
              "PROPN  0.066393  0.046287  0.010096  0.006417  0.010010  \n",
              "NUM    0.011169  0.002234  0.008687  0.004468  0.002234  \n",
              "VERB   0.014453  0.065526  0.169578  0.039679  0.005941  \n",
              "PART   0.690741  0.005395  0.008702  0.001740  0.000870  \n",
              "PRON   0.264412  0.012292  0.027483  0.006817  0.032689  \n",
              "SCONJ  0.157072  0.010419  0.444908  0.022923  0.033863  \n",
              "_      0.014925  0.000000  0.412553  0.000000  0.000000  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-dd808481-5729-4fa9-a08d-c4acf2fafaec\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>START</th>\n",
              "      <th>ADJ</th>\n",
              "      <th>ADP</th>\n",
              "      <th>PUNCT</th>\n",
              "      <th>ADV</th>\n",
              "      <th>AUX</th>\n",
              "      <th>SYM</th>\n",
              "      <th>INTJ</th>\n",
              "      <th>CCONJ</th>\n",
              "      <th>X</th>\n",
              "      <th>NOUN</th>\n",
              "      <th>DET</th>\n",
              "      <th>PROPN</th>\n",
              "      <th>NUM</th>\n",
              "      <th>VERB</th>\n",
              "      <th>PART</th>\n",
              "      <th>PRON</th>\n",
              "      <th>SCONJ</th>\n",
              "      <th>_</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>START</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.040816</td>\n",
              "      <td>0.043686</td>\n",
              "      <td>0.035236</td>\n",
              "      <td>0.075016</td>\n",
              "      <td>0.025351</td>\n",
              "      <td>0.007733</td>\n",
              "      <td>0.032286</td>\n",
              "      <td>0.023198</td>\n",
              "      <td>0.012277</td>\n",
              "      <td>0.062022</td>\n",
              "      <td>0.100446</td>\n",
              "      <td>0.111687</td>\n",
              "      <td>0.039142</td>\n",
              "      <td>0.059630</td>\n",
              "      <td>0.004703</td>\n",
              "      <td>0.251993</td>\n",
              "      <td>0.035555</td>\n",
              "      <td>0.039222</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>ADJ</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.054495</td>\n",
              "      <td>0.076782</td>\n",
              "      <td>0.129904</td>\n",
              "      <td>0.013815</td>\n",
              "      <td>0.003358</td>\n",
              "      <td>0.001526</td>\n",
              "      <td>0.000305</td>\n",
              "      <td>0.042894</td>\n",
              "      <td>0.001221</td>\n",
              "      <td>0.517631</td>\n",
              "      <td>0.005037</td>\n",
              "      <td>0.065639</td>\n",
              "      <td>0.008319</td>\n",
              "      <td>0.008930</td>\n",
              "      <td>0.031598</td>\n",
              "      <td>0.011830</td>\n",
              "      <td>0.022897</td>\n",
              "      <td>0.003816</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>ADP</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.071204</td>\n",
              "      <td>0.030347</td>\n",
              "      <td>0.024165</td>\n",
              "      <td>0.015960</td>\n",
              "      <td>0.000843</td>\n",
              "      <td>0.003316</td>\n",
              "      <td>0.000281</td>\n",
              "      <td>0.005845</td>\n",
              "      <td>0.001293</td>\n",
              "      <td>0.162190</td>\n",
              "      <td>0.358379</td>\n",
              "      <td>0.132910</td>\n",
              "      <td>0.040238</td>\n",
              "      <td>0.006463</td>\n",
              "      <td>0.000843</td>\n",
              "      <td>0.133191</td>\n",
              "      <td>0.003091</td>\n",
              "      <td>0.009441</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>PUNCT</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.053647</td>\n",
              "      <td>0.050289</td>\n",
              "      <td>0.084960</td>\n",
              "      <td>0.061534</td>\n",
              "      <td>0.027565</td>\n",
              "      <td>0.005388</td>\n",
              "      <td>0.011479</td>\n",
              "      <td>0.109402</td>\n",
              "      <td>0.008746</td>\n",
              "      <td>0.127050</td>\n",
              "      <td>0.064032</td>\n",
              "      <td>0.098001</td>\n",
              "      <td>0.032563</td>\n",
              "      <td>0.081446</td>\n",
              "      <td>0.010542</td>\n",
              "      <td>0.122677</td>\n",
              "      <td>0.030923</td>\n",
              "      <td>0.019756</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>ADV</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.141085</td>\n",
              "      <td>0.091577</td>\n",
              "      <td>0.171644</td>\n",
              "      <td>0.089295</td>\n",
              "      <td>0.040976</td>\n",
              "      <td>0.002976</td>\n",
              "      <td>0.000992</td>\n",
              "      <td>0.026094</td>\n",
              "      <td>0.001290</td>\n",
              "      <td>0.014188</td>\n",
              "      <td>0.045838</td>\n",
              "      <td>0.009227</td>\n",
              "      <td>0.018256</td>\n",
              "      <td>0.195357</td>\n",
              "      <td>0.017660</td>\n",
              "      <td>0.083639</td>\n",
              "      <td>0.035619</td>\n",
              "      <td>0.014287</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>AUX</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.106589</td>\n",
              "      <td>0.032827</td>\n",
              "      <td>0.016608</td>\n",
              "      <td>0.140741</td>\n",
              "      <td>0.079064</td>\n",
              "      <td>0.000780</td>\n",
              "      <td>0.000312</td>\n",
              "      <td>0.001949</td>\n",
              "      <td>0.000234</td>\n",
              "      <td>0.011852</td>\n",
              "      <td>0.080312</td>\n",
              "      <td>0.006394</td>\n",
              "      <td>0.009045</td>\n",
              "      <td>0.338246</td>\n",
              "      <td>0.118441</td>\n",
              "      <td>0.046628</td>\n",
              "      <td>0.008265</td>\n",
              "      <td>0.001715</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>SYM</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.041074</td>\n",
              "      <td>0.011058</td>\n",
              "      <td>0.044234</td>\n",
              "      <td>0.004739</td>\n",
              "      <td>0.004739</td>\n",
              "      <td>0.017378</td>\n",
              "      <td>0.001580</td>\n",
              "      <td>0.039494</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.137441</td>\n",
              "      <td>0.006319</td>\n",
              "      <td>0.041074</td>\n",
              "      <td>0.609795</td>\n",
              "      <td>0.022117</td>\n",
              "      <td>0.001580</td>\n",
              "      <td>0.012638</td>\n",
              "      <td>0.003160</td>\n",
              "      <td>0.001580</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>INTJ</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.001477</td>\n",
              "      <td>0.008863</td>\n",
              "      <td>0.385524</td>\n",
              "      <td>0.033973</td>\n",
              "      <td>0.019202</td>\n",
              "      <td>0.002954</td>\n",
              "      <td>0.023634</td>\n",
              "      <td>0.011817</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.019202</td>\n",
              "      <td>0.014771</td>\n",
              "      <td>0.025111</td>\n",
              "      <td>0.008863</td>\n",
              "      <td>0.354505</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.073855</td>\n",
              "      <td>0.004431</td>\n",
              "      <td>0.011817</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>CCONJ</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.087659</td>\n",
              "      <td>0.027973</td>\n",
              "      <td>0.013014</td>\n",
              "      <td>0.081975</td>\n",
              "      <td>0.049065</td>\n",
              "      <td>0.004039</td>\n",
              "      <td>0.002992</td>\n",
              "      <td>0.000150</td>\n",
              "      <td>0.000598</td>\n",
              "      <td>0.145400</td>\n",
              "      <td>0.097233</td>\n",
              "      <td>0.075991</td>\n",
              "      <td>0.014361</td>\n",
              "      <td>0.173373</td>\n",
              "      <td>0.015707</td>\n",
              "      <td>0.167091</td>\n",
              "      <td>0.017053</td>\n",
              "      <td>0.026328</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>X</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.016548</td>\n",
              "      <td>0.044917</td>\n",
              "      <td>0.264775</td>\n",
              "      <td>0.002364</td>\n",
              "      <td>0.004728</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.011820</td>\n",
              "      <td>0.513002</td>\n",
              "      <td>0.068558</td>\n",
              "      <td>0.007092</td>\n",
              "      <td>0.018913</td>\n",
              "      <td>0.007092</td>\n",
              "      <td>0.009456</td>\n",
              "      <td>0.002364</td>\n",
              "      <td>0.021277</td>\n",
              "      <td>0.004728</td>\n",
              "      <td>0.002364</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NOUN</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.011413</td>\n",
              "      <td>0.209989</td>\n",
              "      <td>0.293733</td>\n",
              "      <td>0.035728</td>\n",
              "      <td>0.073236</td>\n",
              "      <td>0.003532</td>\n",
              "      <td>0.000584</td>\n",
              "      <td>0.072098</td>\n",
              "      <td>0.001547</td>\n",
              "      <td>0.125427</td>\n",
              "      <td>0.007414</td>\n",
              "      <td>0.008115</td>\n",
              "      <td>0.005546</td>\n",
              "      <td>0.054117</td>\n",
              "      <td>0.028810</td>\n",
              "      <td>0.039464</td>\n",
              "      <td>0.024548</td>\n",
              "      <td>0.004699</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>DET</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.234002</td>\n",
              "      <td>0.011473</td>\n",
              "      <td>0.012823</td>\n",
              "      <td>0.015768</td>\n",
              "      <td>0.001595</td>\n",
              "      <td>0.002147</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000614</td>\n",
              "      <td>0.000675</td>\n",
              "      <td>0.587521</td>\n",
              "      <td>0.009878</td>\n",
              "      <td>0.075710</td>\n",
              "      <td>0.015154</td>\n",
              "      <td>0.018958</td>\n",
              "      <td>0.000368</td>\n",
              "      <td>0.005031</td>\n",
              "      <td>0.000123</td>\n",
              "      <td>0.008160</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>PROPN</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.014374</td>\n",
              "      <td>0.068617</td>\n",
              "      <td>0.267368</td>\n",
              "      <td>0.018138</td>\n",
              "      <td>0.067334</td>\n",
              "      <td>0.003251</td>\n",
              "      <td>0.000171</td>\n",
              "      <td>0.066051</td>\n",
              "      <td>0.000342</td>\n",
              "      <td>0.085387</td>\n",
              "      <td>0.005048</td>\n",
              "      <td>0.228953</td>\n",
              "      <td>0.035763</td>\n",
              "      <td>0.066393</td>\n",
              "      <td>0.046287</td>\n",
              "      <td>0.010096</td>\n",
              "      <td>0.006417</td>\n",
              "      <td>0.010010</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NUM</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.034500</td>\n",
              "      <td>0.081410</td>\n",
              "      <td>0.281708</td>\n",
              "      <td>0.007694</td>\n",
              "      <td>0.011417</td>\n",
              "      <td>0.033755</td>\n",
              "      <td>0.000248</td>\n",
              "      <td>0.029536</td>\n",
              "      <td>0.003227</td>\n",
              "      <td>0.356664</td>\n",
              "      <td>0.004468</td>\n",
              "      <td>0.040457</td>\n",
              "      <td>0.086126</td>\n",
              "      <td>0.011169</td>\n",
              "      <td>0.002234</td>\n",
              "      <td>0.008687</td>\n",
              "      <td>0.004468</td>\n",
              "      <td>0.002234</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>VERB</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.044467</td>\n",
              "      <td>0.184918</td>\n",
              "      <td>0.078959</td>\n",
              "      <td>0.069206</td>\n",
              "      <td>0.006739</td>\n",
              "      <td>0.001951</td>\n",
              "      <td>0.000975</td>\n",
              "      <td>0.017512</td>\n",
              "      <td>0.000355</td>\n",
              "      <td>0.078826</td>\n",
              "      <td>0.185006</td>\n",
              "      <td>0.023231</td>\n",
              "      <td>0.012680</td>\n",
              "      <td>0.014453</td>\n",
              "      <td>0.065526</td>\n",
              "      <td>0.169578</td>\n",
              "      <td>0.039679</td>\n",
              "      <td>0.005941</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>PART</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.040028</td>\n",
              "      <td>0.007832</td>\n",
              "      <td>0.019318</td>\n",
              "      <td>0.046467</td>\n",
              "      <td>0.070310</td>\n",
              "      <td>0.000522</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.002436</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.076923</td>\n",
              "      <td>0.014619</td>\n",
              "      <td>0.010964</td>\n",
              "      <td>0.003133</td>\n",
              "      <td>0.690741</td>\n",
              "      <td>0.005395</td>\n",
              "      <td>0.008702</td>\n",
              "      <td>0.001740</td>\n",
              "      <td>0.000870</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>PRON</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.038272</td>\n",
              "      <td>0.038379</td>\n",
              "      <td>0.055985</td>\n",
              "      <td>0.051100</td>\n",
              "      <td>0.298658</td>\n",
              "      <td>0.001020</td>\n",
              "      <td>0.001610</td>\n",
              "      <td>0.011326</td>\n",
              "      <td>0.000483</td>\n",
              "      <td>0.130864</td>\n",
              "      <td>0.018894</td>\n",
              "      <td>0.006280</td>\n",
              "      <td>0.003435</td>\n",
              "      <td>0.264412</td>\n",
              "      <td>0.012292</td>\n",
              "      <td>0.027483</td>\n",
              "      <td>0.006817</td>\n",
              "      <td>0.032689</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>SCONJ</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.025527</td>\n",
              "      <td>0.013024</td>\n",
              "      <td>0.009638</td>\n",
              "      <td>0.030216</td>\n",
              "      <td>0.015369</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.001563</td>\n",
              "      <td>0.003126</td>\n",
              "      <td>0.000260</td>\n",
              "      <td>0.036207</td>\n",
              "      <td>0.133889</td>\n",
              "      <td>0.054181</td>\n",
              "      <td>0.007815</td>\n",
              "      <td>0.157072</td>\n",
              "      <td>0.010419</td>\n",
              "      <td>0.444908</td>\n",
              "      <td>0.022923</td>\n",
              "      <td>0.033863</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>_</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.002679</td>\n",
              "      <td>0.000765</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.007271</td>\n",
              "      <td>0.297742</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.091466</td>\n",
              "      <td>0.000765</td>\n",
              "      <td>0.171068</td>\n",
              "      <td>0.000765</td>\n",
              "      <td>0.014925</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.412553</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-dd808481-5729-4fa9-a08d-c4acf2fafaec')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-dd808481-5729-4fa9-a08d-c4acf2fafaec button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-dd808481-5729-4fa9-a08d-c4acf2fafaec');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-40019440-4f75-4aeb-aba5-088c41c09a23\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-40019440-4f75-4aeb-aba5-088c41c09a23')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-40019440-4f75-4aeb-aba5-088c41c09a23 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "df",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 19,\n  \"fields\": [\n    {\n      \"column\": \"START\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.0,\n        \"min\": 0.0,\n        \"max\": 0.0,\n        \"num_unique_values\": 1,\n        \"samples\": [\n          0.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"ADJ\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.05595804246571349,\n        \"min\": 0.0014771048744460858,\n        \"max\": 0.2340020860175471,\n        \"num_unique_values\": 19,\n        \"samples\": [\n          0.04081632653061224\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"ADP\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.05716934757988592,\n        \"min\": 0.0007654037504783774,\n        \"max\": 0.209988616130068,\n        \"num_unique_values\": 19,\n        \"samples\": [\n          0.04368622448979592\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"PUNCT\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.12277063742180962,\n        \"min\": 0.0,\n        \"max\": 0.38552437223042835,\n        \"num_unique_values\": 19,\n        \"samples\": [\n          0.035235969387755105\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"ADV\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.036676867628216245,\n        \"min\": 0.002364066193853428,\n        \"max\": 0.14074074074074075,\n        \"num_unique_values\": 19,\n        \"samples\": [\n          0.07501594387755102\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"AUX\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.08878716423113968,\n        \"min\": 0.0008429807800382151,\n        \"max\": 0.2986580783682233,\n        \"num_unique_values\": 19,\n        \"samples\": [\n          0.025350765306122448\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"SYM\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.008037256028254969,\n        \"min\": 0.0,\n        \"max\": 0.03375527426160337,\n        \"num_unique_values\": 17,\n        \"samples\": [\n          0.007732780612244898\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"INTJ\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.008892153862459343,\n        \"min\": 0.0,\n        \"max\": 0.03228635204081633,\n        \"num_unique_values\": 16,\n        \"samples\": [\n          0.03228635204081633\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"CCONJ\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.029717171662728786,\n        \"min\": 0.0,\n        \"max\": 0.10940184288614711,\n        \"num_unique_values\": 19,\n        \"samples\": [\n          0.023198341836734693\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"X\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.11732036392274275,\n        \"min\": 0.0,\n        \"max\": 0.5130023640661938,\n        \"num_unique_values\": 16,\n        \"samples\": [\n          0.012276785714285714\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"NOUN\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.1617724270694691,\n        \"min\": 0.011851851851851851,\n        \"max\": 0.5875207067918278,\n        \"num_unique_values\": 19,\n        \"samples\": [\n          0.062021683673469385\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"DET\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.08917347507296303,\n        \"min\": 0.0007654037504783774,\n        \"max\": 0.3583792289535799,\n        \"num_unique_values\": 19,\n        \"samples\": [\n          0.10044642857142858\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"PROPN\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.061957534541672264,\n        \"min\": 0.006280193236714976,\n        \"max\": 0.22895277207392198,\n        \"num_unique_values\": 19,\n        \"samples\": [\n          0.11168686224489796\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"NUM\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.13698521330714164,\n        \"min\": 0.0007654037504783774,\n        \"max\": 0.6097946287519748,\n        \"num_unique_values\": 19,\n        \"samples\": [\n          0.039142219387755105\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"VERB\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.17624599536575197,\n        \"min\": 0.00646285264695965,\n        \"max\": 0.6907413853115211,\n        \"num_unique_values\": 19,\n        \"samples\": [\n          0.05963010204081633\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"PART\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.029716297360767474,\n        \"min\": 0.0,\n        \"max\": 0.11844054580896686,\n        \"num_unique_values\": 18,\n        \"samples\": [\n          0.00470344387755102\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"PRON\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.1328338699699971,\n        \"min\": 0.005030983495919995,\n        \"max\": 0.4449075280020839,\n        \"num_unique_values\": 19,\n        \"samples\": [\n          0.25199298469387754\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"SCONJ\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.013623393156699104,\n        \"min\": 0.0,\n        \"max\": 0.03967902110303245,\n        \"num_unique_values\": 19,\n        \"samples\": [\n          0.03555484693877551\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"_\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.012406168292652448,\n        \"min\": 0.0,\n        \"max\": 0.0392219387755102,\n        \"num_unique_values\": 19,\n        \"samples\": [\n          0.0392219387755102\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 28
        }
      ]
    }
  ]
}