intro_2023_exercise_10_solution.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyP43kdS1PS5i8wmHqJ3P1eQ",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "gpuClass": "standard"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/TurkuNLP/intro-to-nlp/blob/master/intro_2023_exercise_10_solution.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Creating word embeddings\n",
        "\n",
        "This notebook demonstrates how to create new `word2vec` word embeddings using `gensim`."
      ],
      "metadata": {
        "id": "vZLv6qie_I4C"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "---\n",
        "\n",
        "## Setup\n",
        "\n",
        "We'll use [`gensim`](https://pypi.org/project/gensim/) to induce the word embeddings and (following the [sentence splitting and tokenization notebook](https://github.com/TurkuNLP/intro-to-nlp/blob/master/sentence_splitting_and_tokenization.ipynb)) the [`sentence-splitter`](https://pypi.org/project/sentence-splitter/) package to split sentences."
      ],
      "metadata": {
        "id": "-ZrlWj4K_xoq"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "f77-4b34uZtp"
      },
      "outputs": [],
      "source": [
        "!pip install --quiet gensim sentence-splitter"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!wget -nc http://dl.turkunlp.org/TKO_7095_2023/fiwiki-20221120-sample.txt"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "c2VZifm9zbg8",
        "outputId": "cb87af1f-8685-40b7-d400-2ef532d59692"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "File ‘fiwiki-20221120-sample.txt’ already there; not retrieving.\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import regex\n",
        "\n",
        "from sentence_splitter import SentenceSplitter"
      ],
      "metadata": {
        "id": "fobDYf0LC8Jq"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Download and read data\n",
        "\n",
        "We'll use the English data here, but you can switch out to the Finnish by commenting out the lines loading the Finnish and uncommenting the lines loading the English data. "
      ],
      "metadata": {
        "id": "KSy9UUbIAdrK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# !wget -nc http://dl.turkunlp.org/TKO_7095_2023/fiwiki-20221120-sample.txt\n",
        "!wget -nc http://dl.turkunlp.org/TKO_7095_2023/enwiki-20220301-sample.txt"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "luC-7TGKAyYF",
        "outputId": "db8c1fec-b479-4278-9b07-68be57d2ddb0"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "File ‘enwiki-20220301-sample.txt’ already there; not retrieving.\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#paragraphs = open('fiwiki-20221120-sample.txt').readlines()\n",
        "paragraphs = open('enwiki-20220301-sample.txt').readlines()"
      ],
      "metadata": {
        "id": "v9Fr7EgSzuzZ"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Check what we have"
      ],
      "metadata": {
        "id": "JvZKJHvRBmEG"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print('Total paragraphs:', len(paragraphs))\n",
        "print('Total characters:', sum(len(p) for p in paragraphs))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PXBr35fh2fH3",
        "outputId": "2f2672f6-17b7-4980-f360-137cf567d625"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Total paragraphs: 1000000\n",
            "Total characters: 526082203\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "So, that's a million paragraphs totaling over 500 million characters. This is a reasonably large corpus, though still notably smaller than corpora on which word2vec models are generally trained."
      ],
      "metadata": {
        "id": "h6zAQxlwLON3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "for i, p in enumerate(paragraphs[:10]):\n",
        "    print(f'paragraph {i}: {p}', end='')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pvz88gaZBqc-",
        "outputId": "d47a9177-d853-4e15-a303-ce6b7017b4db"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "paragraph 0: Incumbent CM Mayawati began her campaign on 27 January at a rally in Bijnor. On 15 January, she released the BSP's list of candidates for all the 403 constituencies. The list included 88 candidates belonging to SCs, 113 from OBCs, 85 religious minorities and 117 upper castes, out of which 74 are Brahmins.\n",
            "paragraph 1: As a member of the OSS Research and Analysis Division, Wheeler had government security clearance to received secret and confidential \"ditto\" copies of monthly and semi-monthly reports of political developments throughout the world. Wheeler is alleged to have passed these reports as well as handwritten and typewritten material of cable reports from the State Department and the OSS to Soviet intelligence.  Wheeler is alleged to have provided information on the organization and policies of British intelligence services and furnished memoranda prepared by the Foreign Nationalities Branch of OSS on material relating to the particular racial groups and activities within the United States. These allegations, launched during McCarthyism resulted in no charges ever being brought against Wheeler, who contended that the charges of espionage were a response to his outspoken criticism of the failings of the American economic and political system.\n",
            "paragraph 2: Lexicographic preferences have also been found useful in describing choices under uncertainty. Let wi = wi(x, p(x)) be the ith utility from a vector of prospects x = (x1,…, xk,…) which has associated probabilities p(x) = (p1(x1),…,pk(xn)).  Note that the criterion function wi depends explicitly not only on x but also on the probabilities associated with x. One may then define w¡* as the satisficing level of w¡ and proceed to define lexicographic preference in the customary manner, with v(x) = [v1(x),v2(x),…] and vi(x) = min (wi (x, p(x), wi*) for all i.  It is  straightforward to reproduce expected utility as a particular case by defining some primitive utility function u(.) that depends only on the individual xk.  In this case, one may write w1(x, p(x)•u(x)), where u(x) = [u(x1), u(x2),…] and set w1* sufficiently high to obtain the usual subjective expected utility.  If other criteria are allowed to come into play, on the other hand, different possibilities arise.  For example, for some i, wi may depend only on the probability of ruinous levels of x, or on the maximal values of x.\n",
            "paragraph 3: Wesley Worrall, Steven Bedford, and Matthew Tate were dishonourably discharged from the camp, Kirk Woodend chose to leave voluntarily and Adam Oakley was medically discharged, Dale Tate (Matthew Tate's brother) deserted the section and never returned to the Platoon; his brother deserted with him, but later returned. Simon Pinkney and Adrian Turton left the camp, however their exits were never aired.\n",
            "paragraph 4: The club also signed several new players: Toutziarakis, Saloustros, Dimakos, Releford, Evans, Lypovyy, Milošević, Hall, Ellis, Murry, and Prather. They finished the season in 4th place overall in the league standings, with a record of 19–15 (regular season & playoffs). That same season, the team also participated in the 2017–18 Greek Cup competition, in which they again finished in 10th place, after beating Koroivos and Lavrio, but then losing to Kolossos Rodou.\n",
            "paragraph 5: Before the Roman conquest, the ancient Abellinum was a centre of the Samnite Hirpini, located on the Civita hill some  outside the current town, in what is now Atripalda. The city could correspond to the ancient Velecha, documented by coins found in the area. Abellinum was conquered by the Romans in 293 BC, changing name several times in the following centuries (Veneria, Livia, Augusta, Alexandriana, and Abellinatium). However, the construction of a true Roman town occurred only after the conquest by Lucius Cornelius Sulla in 89 BC.\n",
            "paragraph 6: Lumbini provincial assembly is the unicameral legislative assembly consisting of 87 members. Candidates for each constituency are chosen by the political parties or stand as independents. Each constituency elects one member under the first past the post (FPTP) system of election. The current constitution specifies that sixty percent of the members should be elected from the first past the post system and forty percent through the party-list proportional representation (PR) system. Women should account for one-third of total members elected from each party. If one-third percentage are not elected, the party that fails to ensure so shall have to elect one-third of the total number as women through the party-list proportional representation.\n",
            "paragraph 7: Ćirić was chair of the Permanent Conference of the Cities and Municipalities in Yugoslavia during his time as mayor. In early 2002, he signed an accord with a representative of the National Association of Municipalities in Bulgaria for greater cooperation between the local governments of both countries. In early 2003, Ćirić reached an agreement with Radio Television of Serbia for Radio Niš to be privatized after one year of transitional funding. He was defeated by Smiljko Kostić of New Serbia by a significant margin in the October 2004 municipal election.\n",
            "paragraph 8: The Wellington Cathedral of St Paul (also called St Paul's Cathedral or Wellington Cathedral) is an Anglican church in the city of Wellington, New Zealand. It is the mother church of the Diocese of Wellington and the cathedral of the Bishop of Wellington. Situated in Thorndon, the main entrance to the cathedral is on Hill Street, at its junction with Molesworth Street; it is located close to the parliament precinct.\n",
            "paragraph 9: The Cardinals won the NL Central and faced the Dodgers in the NLDS. In Game 1, DeRosa had three hits, including an RBI double, but the Cardinals lost 5–3. He had two hits in Game 2 and scored a run that put the Cardinals ahead in seventh, but they ultimately lost 3–2 following an error by Matt Holliday. For the second year in a row, DeRosa's team was swept by the Dodgers in the NLDS. DeRosa, who became a free agent at the end of the year, used the offseason to undergo surgery that sought to repair a torn tendon sheath in his left wrist. He also became a free agent.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Split sentences and tokenize\n",
        "\n",
        "Following the [sentence splitting and tokenization notebook](https://github.com/TurkuNLP/intro-to-nlp/blob/master/sentence_splitting_and_tokenization.ipynb)).\n",
        "\n",
        "(The line `%%time` is Jupyter [\"cell magic\"](https://ipython.readthedocs.io/en/stable/interactive/magics.html) to print execution time.) "
      ],
      "metadata": {
        "id": "LyehaGtuCOqI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "\n",
        "splitter = SentenceSplitter(language='fi')\n",
        "\n",
        "sentences = [s for p in paragraphs for s in splitter.split(p)]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "49jRa74z0dMe",
        "outputId": "9ea64e2e-7a79-4633-9084-d8b338904158"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 15min 26s, sys: 3.96 s, total: 15min 30s\n",
            "Wall time: 15min 50s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "\n",
        "TOKENIZE_RE = regex.compile(r'([[:alnum:]]+|\\S)')\n",
        "\n",
        "tokenized = [TOKENIZE_RE.findall(s) for s in sentences]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zBQzf6ItIVxw",
        "outputId": "e00da441-87f2-4082-9c90-844d96634e9b"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 1min 23s, sys: 10.9 s, total: 1min 34s\n",
            "Wall time: 1min 34s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "So, with this approach, total time for sentence-splitting was ~15 min, and tokenization was just over one minute for the million paragraphs.\n",
        "\n",
        "See a few examples"
      ],
      "metadata": {
        "id": "c7shIiclKEUJ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "for i, s in enumerate(tokenized[:10]):\n",
        "    print(f'Sentence {i}:', s)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FrlQm9j8KFx8",
        "outputId": "3cf7f956-5aa2-4dc9-b192-57868ef35e36"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Sentence 0: ['Incumbent', 'CM', 'Mayawati', 'began', 'her', 'campaign', 'on', '27', 'January', 'at', 'a', 'rally', 'in', 'Bijnor', '.']\n",
            "Sentence 1: ['On', '15', 'January', ',', 'she', 'released', 'the', 'BSP', \"'\", 's', 'list', 'of', 'candidates', 'for', 'all', 'the', '403', 'constituencies', '.']\n",
            "Sentence 2: ['The', 'list', 'included', '88', 'candidates', 'belonging', 'to', 'SCs', ',', '113', 'from', 'OBCs', ',', '85', 'religious', 'minorities', 'and', '117', 'upper', 'castes', ',', 'out', 'of', 'which', '74', 'are', 'Brahmins', '.']\n",
            "Sentence 3: ['As', 'a', 'member', 'of', 'the', 'OSS', 'Research', 'and', 'Analysis', 'Division', ',', 'Wheeler', 'had', 'government', 'security', 'clearance', 'to', 'received', 'secret', 'and', 'confidential', '\"', 'ditto', '\"', 'copies', 'of', 'monthly', 'and', 'semi', '-', 'monthly', 'reports', 'of', 'political', 'developments', 'throughout', 'the', 'world', '.']\n",
            "Sentence 4: ['Wheeler', 'is', 'alleged', 'to', 'have', 'passed', 'these', 'reports', 'as', 'well', 'as', 'handwritten', 'and', 'typewritten', 'material', 'of', 'cable', 'reports', 'from', 'the', 'State', 'Department', 'and', 'the', 'OSS', 'to', 'Soviet', 'intelligence', '.']\n",
            "Sentence 5: ['Wheeler', 'is', 'alleged', 'to', 'have', 'provided', 'information', 'on', 'the', 'organization', 'and', 'policies', 'of', 'British', 'intelligence', 'services', 'and', 'furnished', 'memoranda', 'prepared', 'by', 'the', 'Foreign', 'Nationalities', 'Branch', 'of', 'OSS', 'on', 'material', 'relating', 'to', 'the', 'particular', 'racial', 'groups', 'and', 'activities', 'within', 'the', 'United', 'States', '.']\n",
            "Sentence 6: ['These', 'allegations', ',', 'launched', 'during', 'McCarthyism', 'resulted', 'in', 'no', 'charges', 'ever', 'being', 'brought', 'against', 'Wheeler', ',', 'who', 'contended', 'that', 'the', 'charges', 'of', 'espionage', 'were', 'a', 'response', 'to', 'his', 'outspoken', 'criticism', 'of', 'the', 'failings', 'of', 'the', 'American', 'economic', 'and', 'political', 'system', '.']\n",
            "Sentence 7: ['Lexicographic', 'preferences', 'have', 'also', 'been', 'found', 'useful', 'in', 'describing', 'choices', 'under', 'uncertainty', '.']\n",
            "Sentence 8: ['Let', 'wi', '=', 'wi', '(', 'x', ',', 'p', '(', 'x', ')', ')', 'be', 'the', 'ith', 'utility', 'from', 'a', 'vector', 'of', 'prospects', 'x', '=', '(', 'x1', ',', '…', ',', 'xk', ',', '…', ')', 'which', 'has', 'associated', 'probabilities', 'p', '(', 'x', ')', '=', '(', 'p1', '(', 'x1', ')', ',', '…', ',', 'pk', '(', 'xn', ')', ')', '.']\n",
            "Sentence 9: ['Note', 'that', 'the', 'criterion', 'function', 'wi', 'depends', 'explicitly', 'not', 'only', 'on', 'x', 'but', 'also', 'on', 'the', 'probabilities', 'associated', 'with', 'x', '.']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Alternative using UDpipe\n",
        "\n",
        "You can alternatively perform sentence splitting and tokenization using [UDPipe](https://lindat.mff.cuni.cz/services/udpipe/) as follows, but note that this is considerably slower."
      ],
      "metadata": {
        "id": "NKIdPdQuDM6T"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip3 install --quiet ufal.udpipe"
      ],
      "metadata": {
        "id": "XgvITBKl9Qa1"
      },
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!wget -nc https://github.com/TurkuNLP/intro-to-nlp/raw/master/Data/fi.segmenter.udpipe\n",
        "!wget -nc https://github.com/TurkuNLP/intro-to-nlp/raw/master/Data/en.segmenter.udpipe"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qZ69rUob9NZ-",
        "outputId": "5efa9543-1266-4e53-db00-336bf07eeb3f"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "File ‘fi.segmenter.udpipe’ already there; not retrieving.\n",
            "\n",
            "File ‘en.segmenter.udpipe’ already there; not retrieving.\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import ufal.udpipe as udpipe\n",
        "\n",
        "#model = udpipe.Model.load('fi.segmenter.udpipe')\n",
        "model = udpipe.Model.load('en.segmenter.udpipe')\n",
        "\n",
        "pipeline = udpipe.Pipeline(model, 'tokenize', 'none', 'none', 'horizontal')"
      ],
      "metadata": {
        "id": "4PVECfMC9L71"
      },
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "(We're only processing the first 10,000 paragraphs here as this is quite slow.)"
      ],
      "metadata": {
        "id": "Av30KKl2EN0N"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "\n",
        "udpipe_segmented = [pipeline.process(p) for p in paragraphs[:10000]]"
      ],
      "metadata": {
        "id": "VUlv-46g-SJQ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "88dc1421-65eb-4c22-df1e-39edfd13a489"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 3min 22s, sys: 544 ms, total: 3min 23s\n",
            "Wall time: 3min 27s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "For UDPipe, the sentence splitting and tokenization took over three minutes for just 10,000 paragraphs."
      ],
      "metadata": {
        "id": "W3WoPtqyRPSG"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "udpipe_sentences = [s for t in udpipe_segmented for s in t.split('\\n')]\n",
        "udpipe_tokenized = [s.split() for s in udpipe_sentences]"
      ],
      "metadata": {
        "id": "Gh9NXA4w_rvP"
      },
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "for i, s in enumerate(udpipe_tokenized[:10]):\n",
        "    print(f'UDpipe sentence {i}:', s)"
      ],
      "metadata": {
        "id": "CbZKFPcc_5h0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "46f3516c-48d3-4295-910a-bff2fb487761"
      },
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "UDpipe sentence 0: ['Incumbent', 'CM', 'Mayawati', 'began', 'her', 'campaign', 'on', '27', 'January', 'at', 'a', 'rally', 'in', 'Bijnor', '.']\n",
            "UDpipe sentence 1: ['On', '15', 'January', ',', 'she', 'released', 'the', 'BSP', \"'s\", 'list', 'of', 'candidates', 'for', 'all', 'the', '403', 'constituencies', '.']\n",
            "UDpipe sentence 2: ['The', 'list', 'included', '88', 'candidates', 'belonging', 'to', 'SCs', ',', '113', 'from', 'OBCs', ',', '85', 'religious', 'minorities', 'and', '117', 'upper', 'castes', ',', 'out', 'of', 'which', '74', 'are', 'Brahmins', '.']\n",
            "UDpipe sentence 3: []\n",
            "UDpipe sentence 4: ['As', 'a', 'member', 'of', 'the', 'OSS', 'Research', 'and', 'Analysis', 'Division', ',', 'Wheeler', 'had', 'government', 'security', 'clearance', 'to', 'received', 'secret', 'and', 'confidential', '\"', 'ditto', '\"', 'copies', 'of', 'monthly', 'and', 'semi-monthly', 'reports', 'of', 'political', 'developments', 'throughout', 'the', 'world', '.']\n",
            "UDpipe sentence 5: ['Wheeler', 'is', 'alleged', 'to', 'have', 'passed', 'these', 'reports', 'as', 'well', 'as', 'handwritten', 'and', 'typewritten', 'material', 'of', 'cable', 'reports', 'from', 'the', 'State', 'Department', 'and', 'the', 'OSS', 'to', 'Soviet', 'intelligence', '.']\n",
            "UDpipe sentence 6: ['Wheeler', 'is', 'alleged', 'to', 'have', 'provided', 'information', 'on', 'the', 'organization', 'and', 'policies', 'of', 'British', 'intelligence', 'services', 'and', 'furnished', 'memoranda', 'prepared', 'by', 'the', 'Foreign', 'Nationalities', 'Branch', 'of', 'OSS', 'on', 'material', 'relating', 'to', 'the', 'particular', 'racial', 'groups', 'and', 'activities', 'within', 'the', 'United', 'States', '.']\n",
            "UDpipe sentence 7: ['These', 'allegations', ',', 'launched', 'during', 'McCarthyism', 'resulted', 'in', 'no', 'charges', 'ever', 'being', 'brought', 'against', 'Wheeler', ',', 'who', 'contended', 'that', 'the', 'charges', 'of', 'espionage', 'were', 'a', 'response', 'to', 'his', 'outspoken', 'criticism', 'of', 'the', 'failings', 'of', 'the', 'American', 'economic', 'and', 'political', 'system', '.']\n",
            "UDpipe sentence 8: []\n",
            "UDpipe sentence 9: ['Lexicographic', 'preferences', 'have', 'also', 'been', 'found', 'useful', 'in', 'describing', 'choices', 'under', 'uncertainty', '.']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Create word2vec model\n",
        "\n",
        "We'll create a word2vec model using `gensim` mostly with default parameters. For details on the parameters of this class, see <https://radimrehurek.com/gensim/models/word2vec.html#gensim.models.word2vec.Word2Vec>"
      ],
      "metadata": {
        "id": "9mEcFyScI68h"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "\n",
        "from gensim.models import Word2Vec\n",
        "\n",
        "model = Word2Vec(\n",
        "    sentences=tokenized,\n",
        "    vector_size=100,\n",
        ")"
      ],
      "metadata": {
        "id": "Lq8bLlyKubhd",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "20c26281-9827-41b0-dfcd-e20330869f37"
      },
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 25min 21s, sys: 8.91 s, total: 25min 30s\n",
            "Wall time: 15min 41s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Test with Finnish words (if you've trained a Finnish model)"
      ],
      "metadata": {
        "id": "SiSaErvlJs7J"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# for word in ('hyvä', 'huono', 'koira', 'kissa', 'kuningas', 'kuningatar'):\n",
        "#     print(f'{word}:\\t', [w for w, s in model.wv.most_similar(word)])"
      ],
      "metadata": {
        "id": "IxRTSxQOz0Wi"
      },
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Test with English words (if you've trained an English model)"
      ],
      "metadata": {
        "id": "WHlxSwzgJydK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "for word in ('good', 'bad', 'dog', 'cat', 'king', 'queen'):\n",
        "    print(f'{word}:\\t', [w for w, s in model.wv.most_similar(word)])"
      ],
      "metadata": {
        "id": "AXyKK8XeAUOn",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "dfe49b5c-8c03-44b7-bc95-3c1f17508e16"
      },
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "good:\t ['bad', 'decent', 'perfect', 'tough', 'nice', 'sensible', 'genuine', 'wonderful', 'great', 'little']\n",
            "bad:\t ['good', 'terrible', 'horrible', 'tough', 'careless', 'sad', 'poor', 'funny', 'crazy', 'stupid']\n",
            "dog:\t ['cat', 'rabbit', 'donkey', 'pet', 'monkey', 'puppy', 'cow', 'fox', 'goat', 'horse']\n",
            "cat:\t ['rabbit', 'dog', 'monkey', 'snake', 'spider', 'creature', 'wolf', 'beast', 'monster', 'donkey']\n",
            "king:\t ['prince', 'emperor', 'monarch', 'sultan', 'ruler', 'duke', 'queen', 'Emperor', 'shah', 'kings']\n",
            "queen:\t ['princess', 'prince', 'king', 'empress', 'bride', 'consort', 'monarch', 'dowager', 'regent', 'duchess']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Not too bad a result for a half-hour run! Some observations:\n",
        "\n",
        "* Perhaps surprisingly, antonyms (words with opposite meaning) show up in the lists of most similar words. This is a known issue with word embeddings (see e.g. [Scheible _et al._ 2013](https://aclanthology.org/I13-1056.pdf)) and makes sense if you think about it: for example, _bad_ can appear in many of the same contexts as _good_.\n",
        "* These word embeddings group animal words with both _dog_ and _cat_, but don't show the level of granularity we got with the embeddings trained on more data, where different types of dogs and cats were more similar to the words _dog_ and _cat_ (respectively) than other animals.\n",
        "* We see a similar phenomenon with _king_ and _queen_ as with animals, with many words related to royalty and ruling classes showing up in both, with more limited distinction by e.g. gender than found in embeddings trained on more data."
      ],
      "metadata": {
        "id": "NHrP2KgqUe5A"
      }
    }
  ]
}