intro_2023_exercise_3_solution.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyNMFSVrfW+yupGGXM3TMyYd",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/TurkuNLP/intro-to-nlp/blob/master/intro_2023_exercise_3_solution.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Exercise task 3: text classification corpora (solution)\n",
        "\n",
        "This notebook shows an example solution for exercise 3.\n"
      ],
      "metadata": {
        "id": "Q68rWd9Gs93X"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "---\n",
        "\n",
        "## Setup\n",
        "\n",
        "Install and import packages"
      ],
      "metadata": {
        "id": "t1Rt-3CrtJOX"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "OaBL4Jfa9AxW",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9c4fbac8-abf9-4d23-98e1-ae8e1ff90f58"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m469.0/469.0 KB\u001b[0m \u001b[31m8.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m132.9/132.9 KB\u001b[0m \u001b[31m4.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m110.5/110.5 KB\u001b[0m \u001b[31m5.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m212.2/212.2 KB\u001b[0m \u001b[31m9.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.0/1.0 MB\u001b[0m \u001b[31m6.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m199.8/199.8 KB\u001b[0m \u001b[31m10.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m158.8/158.8 KB\u001b[0m \u001b[31m7.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.2/114.2 KB\u001b[0m \u001b[31m2.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m264.6/264.6 KB\u001b[0m \u001b[31m8.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h"
          ]
        }
      ],
      "source": [
        "!pip install --quiet datasets"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import datasets\n",
        "\n",
        "from collections import Counter"
      ],
      "metadata": {
        "id": "6IVHn9Sb9CSB"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Adjust verbosity for the datasets library. (This only affects what shows on screen.) "
      ],
      "metadata": {
        "id": "r-x7r-JZtPeC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "datasets.disable_progress_bar()\n",
        "datasets.logging.set_verbosity_error()"
      ],
      "metadata": {
        "id": "DiDIonD4Ke2z"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "---\n",
        "\n",
        "## Print dataset info\n",
        "\n",
        "Python function that takes a dataset name as an argument, loads the dataset, and prints dataset description, relative sizes of subsets of the dataset, and label distribution in `'train'`."
      ],
      "metadata": {
        "id": "n15PTuO-tW8x"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def print_dataset_info(name):\n",
        "  dataset = datasets.load_dataset(name)\n",
        "  builder = datasets.load_dataset_builder(name)\n",
        "  print(f'--- Description for {name} ---')\n",
        "  print(builder.info.description)\n",
        "  print(f'--- Statistics for {name} ---')\n",
        "\n",
        "  print('Relative sizes of subsets:')\n",
        "  subsets = dataset.keys()\n",
        "  total = sum(dataset[s].num_rows for s in subsets)\n",
        "  for s in sorted(subsets):\n",
        "    ratio = dataset[s].num_rows/total\n",
        "    print(f'    {s}: {ratio:.1%} ({dataset[s].num_rows}/{total})')\n",
        "\n",
        "  print('Label distribution in \"train\" subset:')\n",
        "  label_counts = Counter(dataset['train']['label'])\n",
        "  label_names = dataset['train'].features['label'].names\n",
        "  total = sum(label_counts.values())\n",
        "  for k, v in label_counts.items():\n",
        "    print(f'    {label_names[k]}: {v/total:.1%}')\n",
        "\n",
        "  print()"
      ],
      "metadata": {
        "id": "NeNAHrim9GXi"
      },
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Run the function on the datasets `'emotion'`, `'rotten_tomatoes'`, `'snli'`, `'sst2'`, `'emo'`."
      ],
      "metadata": {
        "id": "_uT8AeTIt2IU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "for name in ('emotion', 'rotten_tomatoes', 'snli', 'sst2', 'emo',):\n",
        "  print_dataset_info(name)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bE9xZzYI92EH",
        "outputId": "50b9fdec-6c42-4021-e1ff-4900dcee06de"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading and preparing dataset emotion/split to /root/.cache/huggingface/datasets/emotion/split/1.0.0/cca5efe2dfeb58c1d098e0f9eeb200e9927d889b5a03c67097275dfb5fe463bd...\n",
            "Dataset emotion downloaded and prepared to /root/.cache/huggingface/datasets/emotion/split/1.0.0/cca5efe2dfeb58c1d098e0f9eeb200e9927d889b5a03c67097275dfb5fe463bd. Subsequent calls will reuse this data.\n",
            "--- Description for emotion ---\n",
            "Emotion is a dataset of English Twitter messages with six basic emotions: anger, fear, joy, love, sadness, and surprise. For more detailed information please refer to the paper.\n",
            "\n",
            "--- Statistics for emotion ---\n",
            "Relative sizes of subsets:\n",
            "    test: 10.0% (2000/20000)\n",
            "    train: 80.0% (16000/20000)\n",
            "    validation: 10.0% (2000/20000)\n",
            "Label distribution in \"train\" subset:\n",
            "    sadness: 29.2%\n",
            "    anger: 13.5%\n",
            "    love: 8.2%\n",
            "    surprise: 3.6%\n",
            "    fear: 12.1%\n",
            "    joy: 33.5%\n",
            "\n",
            "Downloading and preparing dataset rotten_tomatoes/default to /root/.cache/huggingface/datasets/rotten_tomatoes/default/1.0.0/40d411e45a6ce3484deed7cc15b82a53dad9a72aafd9f86f8f227134bec5ca46...\n",
            "Dataset rotten_tomatoes downloaded and prepared to /root/.cache/huggingface/datasets/rotten_tomatoes/default/1.0.0/40d411e45a6ce3484deed7cc15b82a53dad9a72aafd9f86f8f227134bec5ca46. Subsequent calls will reuse this data.\n",
            "--- Description for rotten_tomatoes ---\n",
            "Movie Review Dataset.\n",
            "This is a dataset of containing 5,331 positive and 5,331 negative processed\n",
            "sentences from Rotten Tomatoes movie reviews. This data was first used in Bo\n",
            "Pang and Lillian Lee, ``Seeing stars: Exploiting class relationships for\n",
            "sentiment categorization with respect to rating scales.'', Proceedings of the\n",
            "ACL, 2005.\n",
            "\n",
            "--- Statistics for rotten_tomatoes ---\n",
            "Relative sizes of subsets:\n",
            "    test: 10.0% (1066/10662)\n",
            "    train: 80.0% (8530/10662)\n",
            "    validation: 10.0% (1066/10662)\n",
            "Label distribution in \"train\" subset:\n",
            "    pos: 50.0%\n",
            "    neg: 50.0%\n",
            "\n",
            "Downloading and preparing dataset snli/plain_text to /root/.cache/huggingface/datasets/snli/plain_text/1.0.0/1f60b67533b65ae0275561ff7828aad5ee4282d0e6f844fd148d05d3c6ea251b...\n",
            "Dataset snli downloaded and prepared to /root/.cache/huggingface/datasets/snli/plain_text/1.0.0/1f60b67533b65ae0275561ff7828aad5ee4282d0e6f844fd148d05d3c6ea251b. Subsequent calls will reuse this data.\n",
            "--- Description for snli ---\n",
            "The SNLI corpus (version 1.0) is a collection of 570k human-written English\n",
            "sentence pairs manually labeled for balanced classification with the labels\n",
            "entailment, contradiction, and neutral, supporting the task of natural language\n",
            "inference (NLI), also known as recognizing textual entailment (RTE).\n",
            "\n",
            "--- Statistics for snli ---\n",
            "Relative sizes of subsets:\n",
            "    test: 1.8% (10000/570152)\n",
            "    train: 96.5% (550152/570152)\n",
            "    validation: 1.8% (10000/570152)\n",
            "Label distribution in \"train\" subset:\n",
            "    neutral: 33.2%\n",
            "    contradiction: 33.3%\n",
            "    entailment: 33.3%\n",
            "    contradiction: 0.1%\n",
            "\n",
            "Downloading and preparing dataset sst2/default to /root/.cache/huggingface/datasets/sst2/default/2.0.0/9896208a8d85db057ac50c72282bcb8fe755accc671a57dd8059d4e130961ed5...\n",
            "Dataset sst2 downloaded and prepared to /root/.cache/huggingface/datasets/sst2/default/2.0.0/9896208a8d85db057ac50c72282bcb8fe755accc671a57dd8059d4e130961ed5. Subsequent calls will reuse this data.\n",
            "--- Description for sst2 ---\n",
            "The Stanford Sentiment Treebank consists of sentences from movie reviews and\n",
            "human annotations of their sentiment. The task is to predict the sentiment of a\n",
            "given sentence. We use the two-way (positive/negative) class split, and use only\n",
            "sentence-level labels.\n",
            "\n",
            "--- Statistics for sst2 ---\n",
            "Relative sizes of subsets:\n",
            "    test: 2.6% (1821/70042)\n",
            "    train: 96.2% (67349/70042)\n",
            "    validation: 1.2% (872/70042)\n",
            "Label distribution in \"train\" subset:\n",
            "    negative: 44.2%\n",
            "    positive: 55.8%\n",
            "\n",
            "Downloading and preparing dataset emo/emo2019 to /root/.cache/huggingface/datasets/emo/emo2019/1.0.0/3bb182a8ea21ffa4656a89f870d16a7b75abb79f07cf990436beb9320d1d6ddd...\n",
            "Dataset emo downloaded and prepared to /root/.cache/huggingface/datasets/emo/emo2019/1.0.0/3bb182a8ea21ffa4656a89f870d16a7b75abb79f07cf990436beb9320d1d6ddd. Subsequent calls will reuse this data.\n",
            "--- Description for emo ---\n",
            "In this dataset, given a textual dialogue i.e. an utterance along with two previous turns of context, the goal was to infer the underlying emotion of the utterance by choosing from four emotion classes - Happy, Sad, Angry and Others.\n",
            "\n",
            "--- Statistics for emo ---\n",
            "Relative sizes of subsets:\n",
            "    test: 15.4% (5509/35669)\n",
            "    train: 84.6% (30160/35669)\n",
            "Label distribution in \"train\" subset:\n",
            "    others: 49.6%\n",
            "    angry: 18.3%\n",
            "    sad: 18.1%\n",
            "    happy: 14.1%\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "> What patterns can you notice in the relative sizes of the subsets? Can you tell why this might be? \n",
        "\n",
        "The training set is always clearly the largest of the subsets, constituting 80% - 96.5% of the data for all datasets.\n",
        "\n",
        "This is because once you have enough data in the **validation** or **test** subsets to reasonably accurately estimate performance (e.g. a few thousand examples), there's little benefit to increasing their size, while increasing the size of the size of the **training** set generally allows for better classifiers to be trained on the dataset."
      ],
      "metadata": {
        "id": "Ra2Az8gJuhdR"
      }
    }
  ]
}