DevRev_Experimentation_GPU.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/ris27hav/devrevs_domain_specific_qa/blob/main/DevRev_Experimentation_GPU.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Instructions To Experiment\n",
        "\n",
        "1. For experimentation purposes, connect to a GPU runtime.\n",
        "2. Run the implementation section of the notebook.\n",
        "3. Select the parameters from the Execution section and then run the cell.\n",
        "4. Run the remaining cells to get the metrics on the selected parameters."
      ],
      "metadata": {
        "id": "d20R266Ugl_8"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U38_GxU5ZWFh"
      },
      "source": [
        "## Implementation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JyKtVXm661x3"
      },
      "source": [
        "### Load Data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xdk3n4f899fc"
      },
      "source": [
        "Load validation data for testing, based on missing data in the training data from squad 2.0 dataset. Round 1 data contains themes that are not present in training data. While, round 2 data contains themes that are present in training data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "f5cqzE_kBBj_",
        "outputId": "96873998-bf0f-4161-ef70-b0327dc8cda5"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: gdown in /usr/local/lib/python3.8/dist-packages (4.4.0)\n",
            "Collecting gdown\n",
            "  Downloading gdown-4.6.0-py3-none-any.whl (14 kB)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.8/dist-packages (from gdown) (3.9.0)\n",
            "Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.8/dist-packages (from gdown) (4.6.3)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.8/dist-packages (from gdown) (1.15.0)\n",
            "Requirement already satisfied: requests[socks] in /usr/local/lib/python3.8/dist-packages (from gdown) (2.25.1)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.8/dist-packages (from gdown) (4.64.1)\n",
            "Requirement already satisfied: chardet<5,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from requests[socks]->gdown) (4.0.0)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests[socks]->gdown) (1.24.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests[socks]->gdown) (2022.12.7)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests[socks]->gdown) (2.10)\n",
            "Requirement already satisfied: PySocks!=1.5.7,>=1.5.6 in /usr/local/lib/python3.8/dist-packages (from requests[socks]->gdown) (1.7.1)\n",
            "Installing collected packages: gdown\n",
            "  Attempting uninstall: gdown\n",
            "    Found existing installation: gdown 4.4.0\n",
            "    Uninstalling gdown-4.4.0:\n",
            "      Successfully uninstalled gdown-4.4.0\n",
            "Successfully installed gdown-4.6.0\n"
          ]
        }
      ],
      "source": [
        "!pip install --upgrade --no-cache-dir gdown"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PDsZ3veewUjM"
      },
      "outputs": [],
      "source": [
        "import gdown\n",
        "\n",
        "def download_validation_data(round = 1):\n",
        "    \"\"\"Download the validation data (4 csv files)\"\"\"\n",
        "    assert round in [1,2], \"round can be 1 or 2\"\n",
        "    ids = [\n",
        "        [\n",
        "            \"15WPYOD3ZLShFq_NRtiBHbpz3RTvc8ZWR\",\n",
        "            \"15yxIF27NvEa3l12yNy6F5h8lGCJ2n7rf\",\n",
        "            \"1Ilpxyj_0T-1KzQMdVSEbSmc1ybxOv69G\",\n",
        "            \"1nkEDQZJY6_cAEVw3JlaKCgz0C6mDSYiv\"\n",
        "        ],\n",
        "        [\n",
        "            \"1-3fMldkBVsTAX3W5JewdAdlUG_agexG0\",\n",
        "            \"1-59pQe8TH7UaORF1RSqzFWybMJShdf1U\",\n",
        "            \"1-AbnJRRHQiTU5zyUdDC2gUwbIGkEF5l6\",\n",
        "            \"1-Px6FFj043L7lbAEBOAMSy2bdoPiVNhy\"\n",
        "        ]\n",
        "    ]\n",
        "    for id in ids[round-1]:\n",
        "        url = f\"https://drive.google.com/u/1/uc?id={id}&export=download\"\n",
        "        gdown.download(url, quiet=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UzHw72nd6_wA"
      },
      "source": [
        "### Generate Embeddings"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_FarPEbyBsIh"
      },
      "source": [
        "For a given theme, break its paragraphs into sentences and store their paragraph id. Load sentence encoder and calculate embeddings for the sentences from paragraphs and the queries."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "g5jb4sYXKUvk",
        "outputId": "db28da62-4353-42be-fa48-cf84df305d8f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting sentence-transformers\n",
            "  Downloading sentence-transformers-2.2.2.tar.gz (85 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m86.0/86.0 KB\u001b[0m \u001b[31m6.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting transformers<5.0.0,>=4.6.0\n",
            "  Downloading transformers-4.26.0-py3-none-any.whl (6.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.3/6.3 MB\u001b[0m \u001b[31m79.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: tqdm in /usr/local/lib/python3.8/dist-packages (from sentence-transformers) (4.64.1)\n",
            "Requirement already satisfied: torch>=1.6.0 in /usr/local/lib/python3.8/dist-packages (from sentence-transformers) (1.13.1+cu116)\n",
            "Requirement already satisfied: torchvision in /usr/local/lib/python3.8/dist-packages (from sentence-transformers) (0.14.1+cu116)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.8/dist-packages (from sentence-transformers) (1.21.6)\n",
            "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.8/dist-packages (from sentence-transformers) (1.0.2)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from sentence-transformers) (1.7.3)\n",
            "Requirement already satisfied: nltk in /usr/local/lib/python3.8/dist-packages (from sentence-transformers) (3.7)\n",
            "Collecting sentencepiece\n",
            "  Downloading sentencepiece-0.1.97-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m78.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting huggingface-hub>=0.4.0\n",
            "  Downloading huggingface_hub-0.12.0-py3-none-any.whl (190 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m190.3/190.3 KB\u001b[0m \u001b[31m23.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: packaging>=20.9 in /usr/local/lib/python3.8/dist-packages (from huggingface-hub>=0.4.0->sentence-transformers) (23.0)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.8/dist-packages (from huggingface-hub>=0.4.0->sentence-transformers) (2.25.1)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.8/dist-packages (from huggingface-hub>=0.4.0->sentence-transformers) (3.9.0)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.8/dist-packages (from huggingface-hub>=0.4.0->sentence-transformers) (4.4.0)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.8/dist-packages (from huggingface-hub>=0.4.0->sentence-transformers) (6.0)\n",
            "Collecting tokenizers!=0.11.3,<0.14,>=0.11.1\n",
            "  Downloading tokenizers-0.13.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.6/7.6 MB\u001b[0m \u001b[31m89.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.8/dist-packages (from transformers<5.0.0,>=4.6.0->sentence-transformers) (2022.6.2)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.8/dist-packages (from nltk->sentence-transformers) (7.1.2)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.8/dist-packages (from nltk->sentence-transformers) (1.2.0)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.8/dist-packages (from scikit-learn->sentence-transformers) (3.1.0)\n",
            "Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /usr/local/lib/python3.8/dist-packages (from torchvision->sentence-transformers) (7.1.2)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests->huggingface-hub>=0.4.0->sentence-transformers) (1.24.3)\n",
            "Requirement already satisfied: chardet<5,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from requests->huggingface-hub>=0.4.0->sentence-transformers) (4.0.0)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests->huggingface-hub>=0.4.0->sentence-transformers) (2.10)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests->huggingface-hub>=0.4.0->sentence-transformers) (2022.12.7)\n",
            "Building wheels for collected packages: sentence-transformers\n",
            "  Building wheel for sentence-transformers (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for sentence-transformers: filename=sentence_transformers-2.2.2-py3-none-any.whl size=125938 sha256=8d547d0e822d889bf0c9cf3bd14c0356d7443c0cd0859d5eb57f96f13859c3e1\n",
            "  Stored in directory: /root/.cache/pip/wheels/5e/6f/8c/d88aec621f3f542d26fac0342bef5e693335d125f4e54aeffe\n",
            "Successfully built sentence-transformers\n",
            "Installing collected packages: tokenizers, sentencepiece, huggingface-hub, transformers, sentence-transformers\n",
            "Successfully installed huggingface-hub-0.12.0 sentence-transformers-2.2.2 sentencepiece-0.1.97 tokenizers-0.13.2 transformers-4.26.0\n"
          ]
        }
      ],
      "source": [
        "!pip install -U sentence-transformers"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5-iuMv8R7BPv",
        "outputId": "c8e2ce28-9b69-40f6-ca47-1ef2a57b5e8b"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
          ]
        }
      ],
      "source": [
        "import nltk\n",
        "nltk.download('punkt')\n",
        "\n",
        "def para_to_sentences(para):\n",
        "    \"\"\"Splits a paragraph into sentences.\"\"\"\n",
        "    para = para.replace('\\n', ' ').replace('\\t', ' ').replace('\\x00', ' ')\n",
        "    return nltk.sent_tokenize(para)\n",
        "\n",
        "def load_sents_from_para(paras):\n",
        "    \"\"\"Spilits a list of paragraphs into sentences and returns the sentences\n",
        "    and their corresponding paragraph id\"\"\"\n",
        "    sents = []\n",
        "    para_id = []\n",
        "    for i,p in enumerate(paras):\n",
        "        new_sents = para_to_sentences(p['paragraph'])\n",
        "        sents += new_sents\n",
        "        para_id += [i]*len(new_sents)\n",
        "    return sents, para_id"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!gdown https://drive.google.com/file/d/137tZvp-iTMR2xIogasglSH4jTTLW4_Sf/view --fuzzy --no-cookies"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cb0gs9K_sfTe",
        "outputId": "4ab35139-ae07-4cdb-83da-08e00144d247"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading...\n",
            "From: https://drive.google.com/uc?id=137tZvp-iTMR2xIogasglSH4jTTLW4_Sf\n",
            "To: /content/finetuned_mpnet_triplet.zip\n",
            "100% 405M/405M [00:03<00:00, 112MB/s]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!unzip /content/finetuned_mpnet_triplet.zip"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "c8gNsfv-smc3",
        "outputId": "c380b294-150b-4fc9-ff56-0e676496eb68"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Archive:  /content/finetuned_mpnet_triplet.zip\n",
            "   creating: kaggle/working/finetuned_mpnet_triplet/\n",
            "  inflating: kaggle/working/finetuned_mpnet_triplet/config.json  \n",
            "   creating: kaggle/working/finetuned_mpnet_triplet/eval/\n",
            "  inflating: kaggle/working/finetuned_mpnet_triplet/vocab.txt  \n",
            "  inflating: kaggle/working/finetuned_mpnet_triplet/special_tokens_map.json  \n",
            "  inflating: kaggle/working/finetuned_mpnet_triplet/pytorch_model.bin  \n",
            "  inflating: kaggle/working/finetuned_mpnet_triplet/modules.json  \n",
            "  inflating: kaggle/working/finetuned_mpnet_triplet/tokenizer_config.json  \n",
            "   creating: kaggle/working/finetuned_mpnet_triplet/1_Pooling/\n",
            "  inflating: kaggle/working/finetuned_mpnet_triplet/1_Pooling/config.json  \n",
            "  inflating: kaggle/working/finetuned_mpnet_triplet/tokenizer.json  \n",
            "  inflating: kaggle/working/finetuned_mpnet_triplet/config_sentence_transformers.json  \n",
            "  inflating: kaggle/working/finetuned_mpnet_triplet/README.md  \n",
            "   creating: kaggle/working/finetuned_mpnet_triplet/2_Normalize/\n",
            "  inflating: kaggle/working/finetuned_mpnet_triplet/sentence_bert_config.json  \n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fQKHrVVbDvwN"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "import tensorflow_hub as hub\n",
        "import torch\n",
        "from sentence_transformers import SentenceTransformer\n",
        "from transformers import AutoTokenizer, AutoModel\n",
        "\n",
        "def load_encoder(encoder=\"universal-sentence-encoder-qa-v3\"):\n",
        "    \"\"\"Load Google's Universal Sentence Encoder for QA\"\"\"\n",
        "    if encoder == \"universal-sentence-encoder-qa-v3\":\n",
        "        module_url = \"https://tfhub.dev/google/universal-sentence-encoder-qa/3\"\n",
        "        model = hub.load(module_url)\n",
        "    elif encoder == \"mpnet-base-v2\":\n",
        "        model = SentenceTransformer('sentence-transformers/all-mpnet-base-v2')\n",
        "        # model = SentenceTransformer('/content/kaggle/working/finetuned_mpnet_triplet')\n",
        "    elif encoder == \"mpnet-base-v2-fine-tuned\":\n",
        "        model = SentenceTransformer('/content/kaggle/working/finetuned_mpnet_triplet')\n",
        "    elif encoder == \"distilroberta-v1\":\n",
        "        model = SentenceTransformer('sentence-transformers/all-distilroberta-v1')\n",
        "    elif encoder == \"minilm-l12-v2\":\n",
        "        model = SentenceTransformer('sentence-transformers/all-MiniLM-L12-v2')\n",
        "    elif encoder == \"SimCSE\":\n",
        "        tokenizer = AutoTokenizer.from_pretrained(\"princeton-nlp/sup-simcse-bert-base-uncased\")\n",
        "        model = AutoModel.from_pretrained(\"princeton-nlp/sup-simcse-bert-base-uncased\")\n",
        "        device = torch.device('cuda:0')\n",
        "        model.to(device)\n",
        "        model = (tokenizer, model)\n",
        "    else:\n",
        "        raise \"Unknown sentence encoder\"\n",
        "    return model\n",
        "\n",
        "def get_embeddings_guse(sents, paras, para_id, model, sents_type=\"Context\"):\n",
        "    \"\"\"Calculate embeddings for given list of sentences based on its type\n",
        "    i.e. either its a Question or a Context\"\"\"\n",
        "    if sents_type == \"Question\":\n",
        "        return model.signatures['question_encoder'](\n",
        "            tf.constant(sents)\n",
        "        )['outputs']\n",
        "    else:\n",
        "        contexts = [\n",
        "            paras[para_id[i]]['paragraph'] for i in range(len(sents))\n",
        "        ]\n",
        "        return model.signatures['response_encoder'](\n",
        "            input = tf.constant(sents),\n",
        "            context = tf.constant(contexts)             # can play with this\n",
        "        )['outputs']\n",
        "\n",
        "def get_embeddings_st(sents, model):\n",
        "    return model.encode(sents)\n",
        "\n",
        "def get_embeddings_simcse(sents, bundle):\n",
        "    tokenizer, model = bundle\n",
        "    tokens = tokenizer(sents, padding=True, truncation=True, return_tensors=\"pt\")\n",
        "    device = torch.device('cuda:0')\n",
        "    tokens = tokens.to(device)\n",
        "    with torch.no_grad():\n",
        "        embeds = model(**tokens, output_hidden_states=True, return_dict=True).pooler_output\n",
        "    return embeds.cpu()\n",
        "\n",
        "def get_embeddings(encoder_name, sents, paras, para_id, model, sents_type=\"Context\"):\n",
        "    if encoder_name == \"universal-sentence-encoder-qa-v3\":\n",
        "        return get_embeddings_guse(sents, paras, para_id, model, sents_type)\n",
        "    elif encoder_name in [\"mpnet-base-v2\", \"distilroberta-v1\", \"minilm-l12-v2\", \"mpnet-base-v2-fine-tuned\"]:\n",
        "        return get_embeddings_st(sents, model)\n",
        "    elif encoder_name == \"SimCSE\":\n",
        "        return get_embeddings_simcse(sents, model)\n",
        "    else:\n",
        "        raise \"Unknown Sentence Encoder\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BTEq0Khe7DM0"
      },
      "source": [
        "### Nearest Neighbour Search"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xo6xW9HsHoMl"
      },
      "source": [
        "Based on the embeddings calculated, indexes them based on L2 distance and then applies nearest neighbour search to get top k closest sentences for each query"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZWSJfQaeHZlT",
        "outputId": "066d04a5-58c3-410c-9e4b-677f89badf17"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: transformers in /usr/local/lib/python3.8/dist-packages (4.26.0)\n",
            "Collecting faiss-gpu\n",
            "  Downloading faiss_gpu-1.7.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (85.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m85.5/85.5 MB\u001b[0m \u001b[31m8.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /usr/local/lib/python3.8/dist-packages (from transformers) (0.13.2)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.8/dist-packages (from transformers) (2.25.1)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.8/dist-packages (from transformers) (2022.6.2)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.8/dist-packages (from transformers) (1.21.6)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.8/dist-packages (from transformers) (6.0)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.8/dist-packages (from transformers) (3.9.0)\n",
            "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.8/dist-packages (from transformers) (4.64.1)\n",
            "Requirement already satisfied: huggingface-hub<1.0,>=0.11.0 in /usr/local/lib/python3.8/dist-packages (from transformers) (0.12.0)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.8/dist-packages (from transformers) (23.0)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.8/dist-packages (from huggingface-hub<1.0,>=0.11.0->transformers) (4.4.0)\n",
            "Requirement already satisfied: chardet<5,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from requests->transformers) (4.0.0)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests->transformers) (1.24.3)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests->transformers) (2.10)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests->transformers) (2022.12.7)\n",
            "Installing collected packages: faiss-gpu\n",
            "Successfully installed faiss-gpu-1.7.2\n"
          ]
        }
      ],
      "source": [
        "!pip install -U transformers faiss-gpu"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6SwA5Hhc7JMO"
      },
      "outputs": [],
      "source": [
        "import faiss\n",
        "import numpy as np\n",
        "\n",
        "def get_k_nearest_neighbours(source_embeds, target_embeds, k = 10):\n",
        "    \"\"\"Returns k nearest neighbours of target_embeds in source_embeds\"\"\"\n",
        "    index = faiss.IndexFlatL2(source_embeds.shape[1])\n",
        "    index.add(np.array(source_embeds))\n",
        "    return index.search(np.array(target_embeds), k)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Uxnd1xuWqVeA"
      },
      "source": [
        "### Check previously answered queries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OSpVs8P8qoMh"
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xccnbbAf7J1o"
      },
      "source": [
        "### Context Generation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aJgYZNGdR5ni"
      },
      "source": [
        "Generates a context for a given query and its nearest neighbours. Also provides a method to get the paragraph id given the start idx of the answer."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cdyftt2i7SMV"
      },
      "outputs": [],
      "source": [
        "def get_context(query_id, query, sents, paras, para_ids, nearest_neighbours, distances, option, m, ctx_threshold, distance_threshold):\n",
        "    \"\"\"Generate the context for a given query and store the para_id for\n",
        "    each sentence\"\"\"\n",
        "    if option in [1, 2]:\n",
        "        if option == 1:\n",
        "            m = 0\n",
        "        context = \"\"\n",
        "        context_para_ids, sent_length = [], []\n",
        "        for sent_id, dist in zip(nearest_neighbours, distances):\n",
        "            # if dist > distance_threshold*distances[0]:\n",
        "            #     break\n",
        "            for j in range(-m, m+1):\n",
        "                cur_id = sent_id + j\n",
        "                if cur_id >= 0 and cur_id < len(para_ids) and para_ids[sent_id] == para_ids[cur_id]:\n",
        "                    context += sents[cur_id] + ' '\n",
        "                    context_para_ids.append(paras[para_ids[cur_id]]['id'])\n",
        "                    sent_length.append(len(sents[cur_id]))\n",
        "            if len(context.split()) >= ctx_threshold:\n",
        "                break\n",
        "    # else:\n",
        "\n",
        "    sum = -1\n",
        "    for i in range(len(sent_length)):\n",
        "        sum += sent_length[i] + 1\n",
        "        sent_length[i] = sum\n",
        "    return context.strip(), context_para_ids, sent_length\n",
        "\n",
        "def para_id_retriever(start_idx, sent_length, context_para_ids):\n",
        "    \"\"\"Given start index of the answer, return the id of the paragraph\n",
        "    in which the answer belongs\"\"\"\n",
        "    if start_idx == -1:\n",
        "        return -1\n",
        "    for j in range(len(sent_length)):\n",
        "        if start_idx <= sent_length[j]:\n",
        "            return context_para_ids[j]\n",
        "    return context_para_ids[-1]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "raCs2XgbZpTG"
      },
      "source": [
        "### Load QA model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "actaNQ_CafqR"
      },
      "source": [
        "Given a theme, download the corresponding fine-tuned QA model and load the QA pipeline"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AEP0g4-3a8g_",
        "outputId": "7249e999-b4db-4426-a993-2877433db767"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: transformers in /usr/local/lib/python3.8/dist-packages (4.26.0)\n",
            "Requirement already satisfied: sentencepiece in /usr/local/lib/python3.8/dist-packages (0.1.97)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.8/dist-packages (from transformers) (1.21.6)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.8/dist-packages (from transformers) (2022.6.2)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.8/dist-packages (from transformers) (23.0)\n",
            "Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /usr/local/lib/python3.8/dist-packages (from transformers) (0.13.2)\n",
            "Requirement already satisfied: huggingface-hub<1.0,>=0.11.0 in /usr/local/lib/python3.8/dist-packages (from transformers) (0.12.0)\n",
            "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.8/dist-packages (from transformers) (4.64.1)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.8/dist-packages (from transformers) (2.25.1)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.8/dist-packages (from transformers) (3.9.0)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.8/dist-packages (from transformers) (6.0)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.8/dist-packages (from huggingface-hub<1.0,>=0.11.0->transformers) (4.4.0)\n",
            "Requirement already satisfied: chardet<5,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from requests->transformers) (4.0.0)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests->transformers) (2022.12.7)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests->transformers) (2.10)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests->transformers) (1.24.3)\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting optimum[onnxruntime-gpu]\n",
            "  Downloading optimum-1.6.3-py3-none-any.whl (227 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m227.4/227.4 KB\u001b[0m \u001b[31m17.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: huggingface-hub>=0.8.0 in /usr/local/lib/python3.8/dist-packages (from optimum[onnxruntime-gpu]) (0.12.0)\n",
            "Requirement already satisfied: torch>=1.9 in /usr/local/lib/python3.8/dist-packages (from optimum[onnxruntime-gpu]) (1.13.1+cu116)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from optimum[onnxruntime-gpu]) (23.0)\n",
            "Requirement already satisfied: transformers[sentencepiece]>=4.26.0 in /usr/local/lib/python3.8/dist-packages (from optimum[onnxruntime-gpu]) (4.26.0)\n",
            "Requirement already satisfied: numpy<1.24.0 in /usr/local/lib/python3.8/dist-packages (from optimum[onnxruntime-gpu]) (1.21.6)\n",
            "Collecting coloredlogs\n",
            "  Downloading coloredlogs-15.0.1-py2.py3-none-any.whl (46 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m46.0/46.0 KB\u001b[0m \u001b[31m4.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: sympy in /usr/local/lib/python3.8/dist-packages (from optimum[onnxruntime-gpu]) (1.7.1)\n",
            "Collecting evaluate\n",
            "  Downloading evaluate-0.4.0-py3-none-any.whl (81 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m81.4/81.4 KB\u001b[0m \u001b[31m11.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting datasets>=1.2.1\n",
            "  Downloading datasets-2.9.0-py3-none-any.whl (462 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m462.8/462.8 KB\u001b[0m \u001b[31m40.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting onnx\n",
            "  Downloading onnx-1.13.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m13.5/13.5 MB\u001b[0m \u001b[31m98.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting protobuf==3.20.1\n",
            "  Downloading protobuf-3.20.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.0/1.0 MB\u001b[0m \u001b[31m68.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting onnxruntime-gpu>=1.9.0\n",
            "  Downloading onnxruntime_gpu-1.13.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (115.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m115.3/115.3 MB\u001b[0m \u001b[31m8.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: requests>=2.19.0 in /usr/local/lib/python3.8/dist-packages (from datasets>=1.2.1->optimum[onnxruntime-gpu]) (2.25.1)\n",
            "Requirement already satisfied: pyarrow>=6.0.0 in /usr/local/lib/python3.8/dist-packages (from datasets>=1.2.1->optimum[onnxruntime-gpu]) (9.0.0)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.8/dist-packages (from datasets>=1.2.1->optimum[onnxruntime-gpu]) (6.0)\n",
            "Requirement already satisfied: fsspec[http]>=2021.11.1 in /usr/local/lib/python3.8/dist-packages (from datasets>=1.2.1->optimum[onnxruntime-gpu]) (2023.1.0)\n",
            "Collecting responses<0.19\n",
            "  Downloading responses-0.18.0-py3-none-any.whl (38 kB)\n",
            "Requirement already satisfied: tqdm>=4.62.1 in /usr/local/lib/python3.8/dist-packages (from datasets>=1.2.1->optimum[onnxruntime-gpu]) (4.64.1)\n",
            "Collecting multiprocess\n",
            "  Downloading multiprocess-0.70.14-py38-none-any.whl (132 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m132.0/132.0 KB\u001b[0m \u001b[31m16.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pandas in /usr/local/lib/python3.8/dist-packages (from datasets>=1.2.1->optimum[onnxruntime-gpu]) (1.3.5)\n",
            "Requirement already satisfied: aiohttp in /usr/local/lib/python3.8/dist-packages (from datasets>=1.2.1->optimum[onnxruntime-gpu]) (3.8.3)\n",
            "Collecting xxhash\n",
            "  Downloading xxhash-3.2.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (213 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m213.0/213.0 KB\u001b[0m \u001b[31m23.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: dill<0.3.7 in /usr/local/lib/python3.8/dist-packages (from datasets>=1.2.1->optimum[onnxruntime-gpu]) (0.3.6)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.8/dist-packages (from huggingface-hub>=0.8.0->optimum[onnxruntime-gpu]) (4.4.0)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.8/dist-packages (from huggingface-hub>=0.8.0->optimum[onnxruntime-gpu]) (3.9.0)\n",
            "Requirement already satisfied: flatbuffers in /usr/local/lib/python3.8/dist-packages (from onnxruntime-gpu>=1.9.0->optimum[onnxruntime-gpu]) (1.12)\n",
            "Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /usr/local/lib/python3.8/dist-packages (from transformers[sentencepiece]>=4.26.0->optimum[onnxruntime-gpu]) (0.13.2)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.8/dist-packages (from transformers[sentencepiece]>=4.26.0->optimum[onnxruntime-gpu]) (2022.6.2)\n",
            "Requirement already satisfied: sentencepiece!=0.1.92,>=0.1.91 in /usr/local/lib/python3.8/dist-packages (from transformers[sentencepiece]>=4.26.0->optimum[onnxruntime-gpu]) (0.1.97)\n",
            "Collecting humanfriendly>=9.1\n",
            "  Downloading humanfriendly-10.0-py2.py3-none-any.whl (86 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m86.8/86.8 KB\u001b[0m \u001b[31m11.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting onnx\n",
            "  Downloading onnx-1.12.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m13.1/13.1 MB\u001b[0m \u001b[31m53.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.8/dist-packages (from sympy->optimum[onnxruntime-gpu]) (1.2.1)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets>=1.2.1->optimum[onnxruntime-gpu]) (6.0.4)\n",
            "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets>=1.2.1->optimum[onnxruntime-gpu]) (4.0.2)\n",
            "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets>=1.2.1->optimum[onnxruntime-gpu]) (1.8.2)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets>=1.2.1->optimum[onnxruntime-gpu]) (22.2.0)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets>=1.2.1->optimum[onnxruntime-gpu]) (1.3.3)\n",
            "Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets>=1.2.1->optimum[onnxruntime-gpu]) (2.1.1)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets>=1.2.1->optimum[onnxruntime-gpu]) (1.3.1)\n",
            "Requirement already satisfied: chardet<5,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from requests>=2.19.0->datasets>=1.2.1->optimum[onnxruntime-gpu]) (4.0.0)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests>=2.19.0->datasets>=1.2.1->optimum[onnxruntime-gpu]) (2.10)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests>=2.19.0->datasets>=1.2.1->optimum[onnxruntime-gpu]) (2022.12.7)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests>=2.19.0->datasets>=1.2.1->optimum[onnxruntime-gpu]) (1.24.3)\n",
            "Collecting urllib3<1.27,>=1.21.1\n",
            "  Downloading urllib3-1.26.14-py2.py3-none-any.whl (140 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m140.6/140.6 KB\u001b[0m \u001b[31m13.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas->datasets>=1.2.1->optimum[onnxruntime-gpu]) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas->datasets>=1.2.1->optimum[onnxruntime-gpu]) (2022.7.1)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.8/dist-packages (from python-dateutil>=2.7.3->pandas->datasets>=1.2.1->optimum[onnxruntime-gpu]) (1.15.0)\n",
            "Installing collected packages: xxhash, urllib3, protobuf, multiprocess, humanfriendly, onnx, coloredlogs, responses, onnxruntime-gpu, datasets, evaluate, optimum\n",
            "  Attempting uninstall: urllib3\n",
            "    Found existing installation: urllib3 1.24.3\n",
            "    Uninstalling urllib3-1.24.3:\n",
            "      Successfully uninstalled urllib3-1.24.3\n",
            "  Attempting uninstall: protobuf\n",
            "    Found existing installation: protobuf 3.19.6\n",
            "    Uninstalling protobuf-3.19.6:\n",
            "      Successfully uninstalled protobuf-3.19.6\n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "tensorflow 2.9.2 requires protobuf<3.20,>=3.9.2, but you have protobuf 3.20.1 which is incompatible.\n",
            "tensorboard 2.9.1 requires protobuf<3.20,>=3.9.2, but you have protobuf 3.20.1 which is incompatible.\n",
            "googleapis-common-protos 1.58.0 requires protobuf!=3.20.0,!=3.20.1,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.20.1 which is incompatible.\n",
            "google-cloud-translate 3.8.4 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.20.1 which is incompatible.\n",
            "google-cloud-language 2.6.1 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.20.1 which is incompatible.\n",
            "google-cloud-firestore 2.7.3 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.20.1 which is incompatible.\n",
            "google-cloud-datastore 2.11.1 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.20.1 which is incompatible.\n",
            "google-cloud-bigquery 3.4.2 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.20.1 which is incompatible.\n",
            "google-cloud-bigquery-storage 2.18.1 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.20.1 which is incompatible.\n",
            "google-api-core 2.11.0 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.20.1 which is incompatible.\u001b[0m\u001b[31m\n",
            "\u001b[0mSuccessfully installed coloredlogs-15.0.1 datasets-2.9.0 evaluate-0.4.0 humanfriendly-10.0 multiprocess-0.70.14 onnx-1.12.0 onnxruntime-gpu-1.13.1 optimum-1.6.3 protobuf-3.20.1 responses-0.18.0 urllib3-1.26.14 xxhash-3.2.0\n"
          ]
        }
      ],
      "source": [
        "!pip install transformers sentencepiece\n",
        "# !pip install optimum[onnxruntime]\n",
        "!pip install optimum[onnxruntime-gpu]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8YXEXyUNv1XO"
      },
      "outputs": [],
      "source": [
        "from transformers import AutoTokenizer, AutoModelForQuestionAnswering\n",
        "from optimum.onnxruntime import ORTModelForQuestionAnswering, ORTOptimizer, ORTQuantizer\n",
        "from optimum.onnxruntime.configuration import OptimizationConfig, AutoQuantizationConfig\n",
        "from optimum.pipelines import pipeline\n",
        "\n",
        "\n",
        "# load vanilla transformers and convert to onnx\n",
        "def load_optimized_model_pipeline(model_id, save_path, use_onnx, optimize, quantize):\n",
        "    task = 'question-answering'\n",
        "    tokenizer = AutoTokenizer.from_pretrained(model_id)\n",
        "    if use_onnx:\n",
        "        # model = ORTModelForQuestionAnswering.from_pretrained(model_id, from_transformers=True)\n",
        "        model = ORTModelForQuestionAnswering.from_pretrained(model_id, from_transformers=True, provider=\"CUDAExecutionProvider\")\n",
        "    else:\n",
        "        model = AutoModelForQuestionAnswering.from_pretrained(model_id)\n",
        "\n",
        "    if optimize:\n",
        "        optimizer = ORTOptimizer.from_pretrained(model)\n",
        "        optimization_config = OptimizationConfig(optimization_level=99) # enable all optimizations\n",
        "        optimizer.optimize(save_dir=save_path, optimization_config=optimization_config)\n",
        "        # model = ORTModelForQuestionAnswering.from_pretrained(save_path, file_name='model_optimized.onnx')\n",
        "        model = ORTModelForQuestionAnswering.from_pretrained(save_path, file_name='model_optimized.onnx', provider=\"CUDAExecutionProvider\")\n",
        "\n",
        "    if quantize:\n",
        "        quantizer = ORTQuantizer.from_pretrained(model)\n",
        "        qconfig = AutoQuantizationConfig.avx512_vnni(is_static=False, per_channel=True)\n",
        "        quantizer.quantize(save_dir=save_path, quantization_config=qconfig)\n",
        "        # model = ORTModelForQuestionAnswering.from_pretrained(save_path, file_name=\"model_optimized_quantized.onnx\")\n",
        "        model = ORTModelForQuestionAnswering.from_pretrained(save_path, file_name=\"model_optimized_quantized.onnx\", provider=\"CUDAExecutionProvider\")\n",
        "\n",
        "    # save onnx checkpoint and tokenizer\n",
        "    optimum_qa = pipeline(\n",
        "        task, model=model, tokenizer=tokenizer, handle_impossible_answer=True\n",
        "    )\n",
        "\n",
        "    return optimum_qa"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NNzi6Bz-mGED"
      },
      "outputs": [],
      "source": [
        "import gdown\n",
        "import json\n",
        "\n",
        "\n",
        "def download_qa_models():\n",
        "    file_url = \"https://drive.google.com/file/d/1912D_F3GAkUmFOGQqf5boZaaz5g5-1RC/view?usp=sharing\"\n",
        "    gdown.download(url=file_url, output='model_urls.json', quiet=False, fuzzy=True)\n",
        "    with open('model_urls.json') as fo:\n",
        "        model_urls = json.load(fo)\n",
        "    model_urls = model_urls[\"urls\"]\n",
        "    for url in model_urls:\n",
        "        gdown.download_folder(url, quiet=True, use_cookies=False)\n",
        "    print(\"All models downloaded successfully.\")\n",
        "\n",
        "def create_mapping():\n",
        "    file_url = \"https://drive.google.com/file/d/1P6dp7f2m67-iPaUbaNZiDYTmTH7Mw9ec/view?usp=share_link\"\n",
        "    gdown.download(url=file_url, output='clusters.json', quiet=False, fuzzy=True)\n",
        "    mapping = {}\n",
        "    with open('clusters.json') as fo:\n",
        "        map = json.load(fo)\n",
        "    for _, cluster_dict in map.items():\n",
        "        for cluster, themes in cluster_dict.items():\n",
        "            cluster = int(cluster)\n",
        "            for theme in themes:\n",
        "                mapping[theme] = cluster\n",
        "    return mapping\n",
        "\n",
        "def load_fine_tuned_model(theme, mapping):\n",
        "    cluster = mapping[theme]\n",
        "    task = 'question-answering'\n",
        "    model_id = f'/content/qamodels/electra-base-squad2-finetuned-squad-{cluster}'\n",
        "\n",
        "    tokenizer = AutoTokenizer.from_pretrained(model_id)\n",
        "    model = ORTModelForQuestionAnswering.from_pretrained(model_id, from_transformers=True, provider=\"CUDAExecutionProvider\")\n",
        "\n",
        "    optimum_qa = pipeline(\n",
        "        task, model=model, tokenizer=tokenizer, handle_impossible_answer=True\n",
        "    )\n",
        "    return optimum_qa\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0zfjkSO7apHi"
      },
      "outputs": [],
      "source": [
        "import gdown\n",
        "import json\n",
        "from zipfile import ZipFile\n",
        "\n",
        "def load_model_links(json_link):\n",
        "    \"\"\"Downloads the JSON that contains the links to models and tokenizer\"\"\"\n",
        "    gdown.download(json_link, quiet=True)\n",
        "    with open('theme_wise_models.json') as f:\n",
        "        model_links = json.load(f)\n",
        "    return model_links\n",
        "\n",
        "def load_theme_model_pipeline(theme, tokenizer_link, model_links, model_id=\"\"):\n",
        "    \"\"\"Given a theme, loads the corresponding QA model\"\"\"\n",
        "    task = \"question-answering\"\n",
        "    if not model_id:\n",
        "        gdown.download(tokenizer_link, \"tokenizer.zip\", quiet=True)\n",
        "        gdown.download(model_links[theme]['link'], \"model.zip\", quiet=True)\n",
        "        with ZipFile(\"tokenizer.zip\", 'r') as zObject:\n",
        "            zObject.extractall()\n",
        "        with ZipFile(\"model.zip\", 'r') as zObject:\n",
        "            zObject.extractall()\n",
        "        tokenizer_path = \"tokenizer\"\n",
        "        model_path = model_links[theme]['path']\n",
        "        model = ORTModelForQuestionAnswering.from_pretrained(model_path)  # Vanilla model\n",
        "        model = ORTModelForQuestionAnswering.from_pretrained(\n",
        "            model_path, file_name=\"model_optimized_quantized.onnx\", provider=\"CUDAExecutionProvider\"\n",
        "        )\n",
        "        # model = ORTModelForQuestionAnswering.from_pretrained(\n",
        "        #     model_path, file_name=\"model_optimized_quantized.onnx\"\n",
        "        # )                                        # Optimized and Quantized Model\n",
        "        tokenizer = AutoTokenizer.from_pretrained(tokenizer_path)\n",
        "    else:\n",
        "        model = ORTModelForQuestionAnswering.from_pretrained(\n",
        "            model_id, from_transformers=True, provider=\"CUDAExecutionProvider\"\n",
        "        )\n",
        "        # model = ORTModelForQuestionAnswering.from_pretrained(\n",
        "        #     model_id, from_transformers=True\n",
        "        # )\n",
        "        tokenizer = AutoTokenizer.from_pretrained(model_id)\n",
        "    optimum_qa = pipeline(\n",
        "        task, model=model, tokenizer=tokenizer, handle_impossible_answer=True\n",
        "    )\n",
        "    return optimum_qa"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W7Jcd0sw7TOf"
      },
      "source": [
        "### Run QA pipeline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GWsSMRk3kv5Z"
      },
      "source": [
        "Predicts the answer given query and context in the required format"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8cf_Mlpe7Z3L"
      },
      "outputs": [],
      "source": [
        "def predict(query_id, query, context, qa_model, pred_paras, sent_length, context_para_ids):\n",
        "    \"\"\"Predict the answer given a query and a context\"\"\"\n",
        "    prediction = qa_model(question=query, context=context)\n",
        "    ans = {\n",
        "        \"question_id\": query_id,\n",
        "        \"answers\": [prediction['answer']],\n",
        "        \"paragraph_id\": -1,\n",
        "        \"context\": context                # Extra info\n",
        "    }\n",
        "    if prediction['answer'] != \"\":\n",
        "        ans[\"paragraph_id\"] = para_id_retriever(\n",
        "            prediction['start'], sent_length, context_para_ids\n",
        "        )\n",
        "    return ans"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2y-4btD4GC0v"
      },
      "outputs": [],
      "source": [
        "def divide_context(passes, context, sent_length):\n",
        "    context_list = []\n",
        "    rem_sents = len(sent_length)\n",
        "    passes = min(passes, rem_sents)\n",
        "    passes_left = passes\n",
        "    i, end = -1, -1\n",
        "    for j in range(passes):\n",
        "        i += int(rem_sents / passes_left)\n",
        "        context_list.append(context[end+1:sent_length[i]])\n",
        "        end = sent_length[i]\n",
        "        rem_sents -= rem_sents / passes_left\n",
        "        passes_left -= 1\n",
        "    return context_list\n",
        "\n",
        "def get_best_prediction(query, context_list, qa_model):\n",
        "    best_prediction = {'answer': \"\"}\n",
        "    text_start = 0\n",
        "    for context in context_list:\n",
        "        prediction = qa_model(question=query, context=context)\n",
        "        if prediction['answer'] != \"\" and (best_prediction['answer'] == \"\" or prediction['score'] > best_prediction['score']):\n",
        "            prediction['start'] += text_start\n",
        "            best_prediction = prediction\n",
        "        text_start += len(context)\n",
        "    return best_prediction\n",
        "\n",
        "def multiple_pass_prediction(passes, query_id, query, context, qa_model, pred_paras, sent_length, context_para_ids):\n",
        "    context_list = divide_context(passes, context, sent_length)\n",
        "    prediction = get_best_prediction(query, context_list, qa_model)\n",
        "    ans = {\n",
        "        \"question_id\": query_id,\n",
        "        \"answers\": [prediction['answer']],\n",
        "        \"paragraph_id\": -1,\n",
        "        \"context\": context                # Extra info\n",
        "    }\n",
        "    if prediction['answer'] != \"\":\n",
        "        ans[\"paragraph_id\"] = para_id_retriever(\n",
        "            prediction['start'], sent_length, context_para_ids\n",
        "        )\n",
        "    return ans"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DlfTtpesQFcK"
      },
      "outputs": [],
      "source": [
        "import time\n",
        "from tqdm import tqdm\n",
        "\n",
        "def predict_theme_wise(paras, ques, pred_out, encoder_name, sents_encoder, qa_pipeline, ctx_option, k, m, ctx_threshold, distance_threshold, qa_passes):\n",
        "    \"\"\"Predicts the answers for all queries of a particular theme\"\"\"\n",
        "    ann_inference_time, qna_inference_time = 0., 0.\n",
        "    theme = ques[0][\"theme\"]\n",
        "    print(f'Theme: {theme}')\n",
        "\n",
        "    # Preprocessing of contexts\n",
        "    sents, para_id = load_sents_from_para(paras)\n",
        "    sents_embed = get_embeddings(\n",
        "        encoder_name, sents, paras, para_id, sents_encoder, sents_type=\"Context\"\n",
        "    )\n",
        "\n",
        "    # Nearest Neighbour Search\n",
        "    start_time = time.time()\n",
        "    ques_list = [q['question'] for q in ques]\n",
        "    ques_embed = get_embeddings(\n",
        "        encoder_name, ques_list, None, None, sents_encoder, sents_type=\"Question\"\n",
        "    )\n",
        "    D, I = get_k_nearest_neighbours(sents_embed, ques_embed, k)\n",
        "    ann_inference_time = (time.time() - start_time)*1000.\n",
        "\n",
        "    pred_paras = [\n",
        "        [paras[para_id[sent_idx]]['id'] for sent_idx in I[i]]\n",
        "        for i in range(len(I))\n",
        "    ]\n",
        "\n",
        "    start_time = time.time()\n",
        "    for i in tqdm(range(len(ques))):\n",
        "        q = ques[i]\n",
        "\n",
        "        # Context Generation\n",
        "        context, context_para_ids, sent_length = get_context(\n",
        "            q[\"id\"], q['question'], sents, paras, para_id, I[i], D[i],\n",
        "            ctx_option, m, ctx_threshold, distance_threshold\n",
        "        )\n",
        "\n",
        "        # Answer Prediction and Paragraph Retrieval\n",
        "        ans = multiple_pass_prediction(\n",
        "            qa_passes, q[\"id\"], q['question'], context, qa_pipeline,\n",
        "            pred_paras[i], sent_length, context_para_ids\n",
        "        )\n",
        "        pred_out.append(ans)\n",
        "\n",
        "    # Print Inference Time\n",
        "    qna_inference_time = (time.time() - start_time)*1000.\n",
        "    print(\n",
        "        f'Avg. ANN IT = {round(ann_inference_time/len(ques), 2)} ms, ' +\n",
        "        f'Avg. QnA IT = {round(qna_inference_time/len(ques),2)} ms\\n'\n",
        "    )\n",
        "    return (ann_inference_time, qna_inference_time)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QHsjvLAgTn3g"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "\n",
        "def predict_multiple_themes(params):\n",
        "    \"\"\"Predicts the answers for queries from multiple (num_themes) themes\"\"\"\n",
        "    # Load paras and queries\n",
        "    paragraphs = json.loads(pd.read_csv(\"input_paragraph.csv\").to_json(orient=\"records\"))\n",
        "    questions = json.loads(pd.read_csv(\"input_question.csv\").to_json(orient=\"records\"))\n",
        "    theme_intervals = json.loads(pd.read_csv(\"theme_interval.csv\").to_json(orient=\"records\"))\n",
        "    pred_out = []\n",
        "    theme_inf_time = {}\n",
        "\n",
        "    # Number of themes for prediction\n",
        "    if params['num_themes'] == -1 or params['num_themes'] > len(theme_intervals):\n",
        "        params['num_themes'] = len(theme_intervals)\n",
        "\n",
        "    # if using pretrained model\n",
        "    if params['use_pretrained']:\n",
        "        qa_pipeline = load_optimized_model_pipeline(\n",
        "            params['model_id'], '/content/model.onnx', params['use_onnx'],\n",
        "            params['optimize'], params['quantize']\n",
        "        )\n",
        "\n",
        "    # Predict for each theme\n",
        "    themes = [\n",
        "        'IPod', 'Wayback_Machine', 'Web_browser', 'DevRev',\n",
        "        '2008_Sichuan_earthquake', 'Nanjing', 'Canadian_Armed_Forces',\n",
        "        'Cardinal_(Catholicism)', 'Heresy', 'Mary_(mother_of_Jesus)',\n",
        "        'Human_Development_Index', 'Warsaw_Pact', 'Materialism',\n",
        "        'Pub', 'Southampton', 'Catalan_language', 'Dialect', 'Paper',\n",
        "        'Adult_contemporary_music', 'Hard_rock', 'The_Times',\n",
        "        'United_States_dollar', 'Immunology', 'Imamah_(Shia_doctrine)',\n",
        "        'Grape', 'Everton_F.C.', 'Great_Plains', 'Biodiversity',\n",
        "        'Federal_Bureau_of_Investigation', 'Unknown'\n",
        "    ]\n",
        "    for theme_interval in theme_intervals[:params['num_themes']]:\n",
        "        theme = theme_interval[\"theme\"]\n",
        "        if theme not in themes:\n",
        "            continue\n",
        "        if not params['use_pretrained']:\n",
        "            qa_pipeline = load_theme_model_pipeline(\n",
        "                theme, params['tok_link'], params['qam_links'],\n",
        "            )\n",
        "        theme_ques = questions[int(theme_interval[\"start\"]) - 1: int(theme_interval[\"end\"])]\n",
        "        theme_paras = [p for p in paragraphs if p[\"theme\"] == theme]\n",
        "        execution_time = predict_theme_wise(\n",
        "            theme_paras, theme_ques, pred_out, params['encoder_name'],\n",
        "            params['encoder'], qa_pipeline, params['ctx_option'], params['k'],\n",
        "            params['m'], params['ctx_threshold'], params['distance_threshold'],\n",
        "            params['qa_passes']\n",
        "        )\n",
        "        theme_inf_time[theme] = execution_time\n",
        "\n",
        "    # Export predictions\n",
        "    pred_df = pd.DataFrame.from_records(pred_out)\n",
        "    pred_df.to_csv('output_prediction.csv', index=False)\n",
        "\n",
        "    return theme_inf_time"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XNJggLTqUSrg"
      },
      "source": [
        "### Evaluation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aCkWmI4ZUU8i"
      },
      "source": [
        "Evaluates and prints statistics of the predictions by the given pipeline. Metrics include the F1 Score, Paragraph Accuracy, Mean Rank of the gold paragraph, performance on true positives and negatives, inference times, etc."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GI9fmizthXlR"
      },
      "outputs": [],
      "source": [
        "import string, re\n",
        "from collections import Counter\n",
        "\n",
        "def normalize_answer(s):\n",
        "    \"\"\"Lower text and remove punctuation, articles and extra whitespace.\"\"\"\n",
        "    def remove_articles(text):\n",
        "        regex = re.compile(r'\\b(a|an|the)\\b', re.UNICODE)\n",
        "        return re.sub(regex, ' ', text)\n",
        "\n",
        "    def white_space_fix(text):\n",
        "        return ' '.join(text.split())\n",
        "\n",
        "    def remove_punc(text):\n",
        "        exclude = set(string.punctuation)\n",
        "        return ''.join(ch for ch in text if ch not in exclude)\n",
        "\n",
        "    def lower(text):\n",
        "        return text.lower()\n",
        "\n",
        "    return white_space_fix(remove_articles(remove_punc(lower(s))))\n",
        "\n",
        "\n",
        "def get_tokens(s):\n",
        "    if not s:\n",
        "        return []\n",
        "    return normalize_answer(s).split()\n",
        "\n",
        "\n",
        "def calc_f1(a_gold, a_pred):\n",
        "    \"\"\"Calulates F1 score, given prediction and a gold answer\"\"\"\n",
        "    gold_toks = get_tokens(a_gold)\n",
        "    pred_toks = get_tokens(a_pred)\n",
        "    common = Counter(gold_toks) & Counter(pred_toks)\n",
        "    num_same = sum(common.values())\n",
        "\n",
        "    if len(gold_toks) == 0 or len(pred_toks) == 0:\n",
        "        # If either is no-answer, then F1 is 1 if they agree, 0 otherwise\n",
        "        return int(gold_toks == pred_toks)\n",
        "\n",
        "    if num_same == 0:\n",
        "        return 0\n",
        "\n",
        "    precision = 1.0 * num_same / len(pred_toks)\n",
        "    recall = 1.0 * num_same / len(gold_toks)\n",
        "    f1 = (2 * precision * recall) / (precision + recall)\n",
        "    return f1\n",
        "\n",
        "\n",
        "def calc_max_f1(predicted, ground_truths):\n",
        "    \"\"\"Calulates the max F1 score, given prediction and the gold answers\"\"\"\n",
        "    max_f1 = 0\n",
        "    for ground_truth in ground_truths:\n",
        "        f1 = calc_f1(str(predicted), str(ground_truth))\n",
        "        max_f1 = max(max_f1, f1)\n",
        "    return max_f1"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MafV6F1JnSgo"
      },
      "outputs": [],
      "source": [
        "from ast import literal_eval\n",
        "import csv\n",
        "import pandas as pd\n",
        "\n",
        "def evaluate_metrics():\n",
        "    \"\"\"Calculate metrics using the predictions and the ground truths\"\"\"\n",
        "    metrics = {}\n",
        "    # Load questions, prediction and ground_truth csv\n",
        "    questions = pd.read_csv(\"input_question.csv\")\n",
        "    pred = pd.read_csv(\"output_prediction.csv\")\n",
        "    truth = pd.read_csv(\"ground_truth.csv\")\n",
        "\n",
        "    # String to list and numbers conversion\n",
        "    truth.paragraph_id = truth.paragraph_id.apply(literal_eval)\n",
        "    truth.answers = truth.answers.apply(literal_eval)\n",
        "    pred.answers = pred.answers.apply(literal_eval)\n",
        "\n",
        "    preds_eval = [\n",
        "        ['theme', 'question', 'gold_para', 'gold_ans', 'context_used',\n",
        "         'pred_para', 'pred_ans', 'ans_in_context', 'para_acc', 'f1']\n",
        "    ]\n",
        "\n",
        "    # Go thorugh each prediction and update the metrics\n",
        "    for idx in pred.index:\n",
        "        q_id = pred[\"question_id\"][idx]\n",
        "        q_rows = questions.loc[questions['id'] == q_id].iloc[-1]\n",
        "        theme = q_rows[\"theme\"]\n",
        "        truth_row = truth.loc[truth['question_id'] == q_id].iloc[-1]\n",
        "        truth_paragraph_id = [ int(i) for i in truth_row[\"paragraph_id\"] ]\n",
        "        predicted_paragraph = pred[\"paragraph_id\"][idx]\n",
        "        predicted_ans = pred[\"answers\"][idx][0]\n",
        "\n",
        "        cur_pred = [\n",
        "            theme, q_rows['question'], truth_paragraph_id, truth_row[\"answers\"],\n",
        "            pred['context'][idx], predicted_paragraph, predicted_ans, 0, 0, 0.\n",
        "        ]\n",
        "\n",
        "        if theme not in metrics.keys():\n",
        "            metrics[theme] = {\n",
        "                \"total_positive\": 0,\n",
        "                \"total_negative\": 0,\n",
        "                \"true_positive\": 0,\n",
        "                \"true_negative\": 0,\n",
        "                'ansInCtx': 0,\n",
        "                \"total_predictions\": 0,\n",
        "                \"total_ctx_len\": 0,\n",
        "                \"f1_sum\": 0\n",
        "            }\n",
        "\n",
        "        if truth_paragraph_id == []:\n",
        "            metrics[theme][\"total_negative\"] += 1\n",
        "        else:\n",
        "            metrics[theme][\"total_positive\"] += 1\n",
        "            for ans in truth_row[\"answers\"]:\n",
        "                if ans in pred['context'][idx]:\n",
        "                    metrics[theme][\"ansInCtx\"] += 1\n",
        "                    cur_pred[7] = 1\n",
        "                    break\n",
        "\n",
        "        metrics[theme][\"total_ctx_len\"] += len(pred['context'][idx].split())\n",
        "\n",
        "        if predicted_paragraph in truth_paragraph_id:\n",
        "            # Increase TP for that theme.\n",
        "            metrics[theme][\"true_positive\"] = metrics[theme][\"true_positive\"] + 1\n",
        "            cur_pred[8] = 1\n",
        "\n",
        "        # -1 prediction in case there is no paragraph which can answer the query.\n",
        "        if predicted_paragraph == -1 and truth_paragraph_id == []:\n",
        "            # Increase TN.\n",
        "            metrics[theme][\"true_negative\"] = metrics[theme][\"true_negative\"] + 1\n",
        "            cur_pred[8] = 1\n",
        "\n",
        "        # Increase total predictions for that theme.\n",
        "        metrics[theme][\"total_predictions\"] = metrics[theme][\"total_predictions\"] + 1\n",
        "        if truth_row[\"answers\"] == []:\n",
        "            truth_row[\"answers\"] = [\"\"]\n",
        "        f1 = calc_max_f1(predicted_ans, truth_row[\"answers\"])\n",
        "        metrics[theme][\"f1_sum\"] = metrics[theme][\"f1_sum\"] + f1\n",
        "\n",
        "        cur_pred[9] = f1\n",
        "        preds_eval.append(cur_pred)\n",
        "\n",
        "    with open('predictions_evaluation.csv', 'w') as f:\n",
        "        write = csv.writer(f)\n",
        "        write.writerows(preds_eval)\n",
        "\n",
        "    return metrics"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "J-WRArKplZ4o"
      },
      "outputs": [],
      "source": [
        "def show(val, sz, dec = 0):\n",
        "    \"\"\"Prints the value and adds whitespaces so that characters printed = sz\"\"\"\n",
        "    val_str = str(round(val, dec))\n",
        "    return ' '*(max(0, sz - len(val_str))) + val_str\n",
        "\n",
        "\n",
        "def calculate_score(theme_inf_time, inf_time_threshold = 1000.0):\n",
        "    \"\"\"Calculates and prints theme-wise as well as aggregated metrics score\"\"\"\n",
        "    metrics = evaluate_metrics()\n",
        "    final_para_score = 0.0\n",
        "    final_qa_score = 0.0\n",
        "    q, aic, tait, tqit, totp, totn, tp, tn, tf1 = 0, 0, 0., 0., 0, 0, 0, 0, 0.\n",
        "    total_ctx_len = 0.0\n",
        "\n",
        "    print('Theme             | Queries |  AIT: (ANN) + (QnA) = Total | ansInCtx'\n",
        "        ' % | TP % (TotP) | TN % (TotN) | Para Acc | F1 Score | Avg Ctx Size'\n",
        "    )\n",
        "    print('------------------|---------|-----------------------------|-------'\n",
        "        '-----|-------------|-------------|----------|----------|-------------'\n",
        "    )\n",
        "\n",
        "    # Print theme wise metrics score\n",
        "    for theme in metrics:\n",
        "        inf_time_score = 1.0\n",
        "        metric = metrics[theme]\n",
        "        para_score = (metric[\"true_positive\"] + metric[\"true_negative\"]) / metric[\"total_predictions\"]\n",
        "        qa_score = metric[\"f1_sum\"] / metric[\"total_predictions\"]\n",
        "        avg_ann_inf_time = theme_inf_time[theme][0] / metric[\"total_predictions\"]\n",
        "        avg_qna_inf_time = theme_inf_time[theme][1] / metric[\"total_predictions\"]\n",
        "        ctx_length = metric[\"total_ctx_len\"] * 1. / metric[\"total_predictions\"]\n",
        "\n",
        "        avg_inf_time = avg_ann_inf_time + avg_qna_inf_time\n",
        "        if avg_inf_time > inf_time_threshold:\n",
        "            inf_time_score = inf_time_threshold / avg_inf_time\n",
        "\n",
        "        q += metric[\"total_predictions\"]\n",
        "        aic += metric[\"ansInCtx\"]\n",
        "        tait += theme_inf_time[theme][0]\n",
        "        tqit += theme_inf_time[theme][1]\n",
        "        totp += metric[\"total_positive\"]\n",
        "        totn += metric[\"total_negative\"]\n",
        "        tp += metric[\"true_positive\"]\n",
        "        tn += metric[\"true_negative\"]\n",
        "        tf1 += metric[\"f1_sum\"]\n",
        "        total_ctx_len += metric[\"total_ctx_len\"]\n",
        "        final_qa_score += inf_time_score * qa_score\n",
        "        final_para_score += inf_time_score * para_score\n",
        "\n",
        "        print(f'{(theme + \" \"*17)[:17]} | '\n",
        "            f'{show(metric[\"total_predictions\"],7)} | '\n",
        "            f'{show(avg_ann_inf_time,6,2)} + {show(avg_qna_inf_time,6,2)} = '\n",
        "            f'{show(avg_inf_time,6,2)} ms | '\n",
        "            f'{show(metric[\"ansInCtx\"]*100./max(1,metric[\"total_positive\"]),8,2)} '\n",
        "            f'% | {show(int(metric[\"true_positive\"]*100./max(1,metric[\"total_positive\"])),3)}% '\n",
        "            f'({show(metric[\"total_positive\"],4)}) | '\n",
        "            f'{show(int(metric[\"true_negative\"]*100./max(1,metric[\"total_negative\"])),3)}% '\n",
        "            f'({show(metric[\"total_negative\"],4)}) | '\n",
        "            f'{show(para_score,8,4)} | {show(qa_score,8,4)} | {show(ctx_length,7,1)}')\n",
        "\n",
        "    final_qa_score /= len(metrics)\n",
        "    final_para_score /= len(metrics)\n",
        "    # Print Aggregated Metrics Score\n",
        "    print(f'------------------|---------|-----------------------------|'\n",
        "        f'------------|-------------|-------------|----------|----------'\n",
        "        f'|--------------')\n",
        "    print(f'Grand Total       | {show(q,7)} | {show(tait/max(1,q),6,2)} + '\n",
        "        f'{show(tqit/max(1,q),6,2)} = {show((tait+tqit)/max(1,q),6,2)} ms |'\n",
        "        f'{show(aic*100./max(1,totp),9,2)} % | {show(int(tp*100./max(1,totp)),3)}% '\n",
        "        f'({show(totp,4)}) | {show(int(tn*100./max(1,totn)),3)}% '\n",
        "        f'({show(totn,4)}) | {show((tp+tn)/max(1,q),8,4)} | '\n",
        "        f'{show(tf1/max(1,q),8,4)} | {show(total_ctx_len/q,7,1)}')\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_Q_1CZAKZZ1w"
      },
      "source": [
        "## Execution"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IVHQXSfMSPmA",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 177,
          "referenced_widgets": [
            "5e46ca6cebaa497bb3542c85689fa117",
            "9c07e959cb5d4e258c6068ac8f115937",
            "665352ce795348028e9873845109bcc3",
            "50636167e4484b8b9dbcc313c3fb22bb",
            "ea1a9a9253d3427b9fccbc21082476e2",
            "d1333a25055d4c179e352bbd18329d56",
            "7cb95e77b0554fe8aa182823dda95199",
            "ebfa6f7b16e643eaba0070455a42c407",
            "ac4e90976a65403e9d0e15eb5ab772b0",
            "cb4fe34398864c8193747a1ca2c40d4e",
            "75e802335a78483091ac3f15b428070a",
            "8fdb9e2851374dbd867a6ab0924a77ee",
            "cc2e0a13ae0d47f8a47db163b3a4d3cb",
            "4c3213c235db43fb81c09882882cd450",
            "847580e6e30c489b92141d08e2de6d37",
            "9fdf189253744c72bd42b5907b1949f3",
            "9a9c74ebbd334ce998a33d59643f1240",
            "efb04f5e8cb741d0afebc0f48f9cfa2c",
            "d30fc6bf02e9407db50fba22ae2fee4d",
            "3ca0a533b4b94904b141dce121b85566",
            "cc0beb05f7244dd2a1c5023444837846",
            "674fa0d39deb4879b6dce947051689aa",
            "ea971bb214894a919bd86597c746cbe3",
            "6e915df9688044aba21a1907b8a9f96f",
            "3eb9929610e94362a31abf6f2ec38a12",
            "13ed22d90c1b4107acc2f8dc5bf9d363",
            "fd29c20e9249400089610ad7ed4add24",
            "aa58fed4497a4aff8e9bb7316e7a9785",
            "e214f60e233d4d04bed4569542cfb072",
            "9630eac4580d40c78b05bd473126b9db",
            "c51453b6361e43e897a78b1b5c0cedc2",
            "eb6db8b1f5fc41fba752a7400cb2cb59",
            "e25723644cab48c0bffe5dff7a5a6459",
            "93fc6edb6c9a4922a15f7dd6ff0172da",
            "979cf28032aa404cb11ce79fcd662954",
            "189bb73c5b024388a10d2ff17de27134",
            "9f11c6b8f2144e57864618aa73050669",
            "2effa6df7cd44b7386d43b9a1153b67b",
            "32d7320c36e047f194a63bc7348dd486",
            "e6217fa25df94b11bcad04ace5943027",
            "e85e8399e02b4a5783bbe622a9231b9c",
            "eb0f9afaa66c4bb8bd3d1a5992e2b5d5",
            "c7ac9ad92d5e4d62a5857c8ee3656093",
            "ac95d86d816641afaf273b31f2040a13",
            "f75c2adcf7f54d5c98a4febc130fd0ad",
            "b1bae9234b664cf4814dcfb8bc83ee13",
            "53ba79f7ba804105b387b0860ef05cca",
            "dc7ed138503542d4a6ee6583f2c15123",
            "5ccc7d6ec71842b1bf8e772dc46434a6",
            "da9963741e684cd4a680903328053ae3",
            "440e641feb82497dbf87415b49e6ed0c",
            "bd6974c44f1149b1bec7c1934970c1af",
            "3b2337a8da274229bdc482832294a298",
            "8de015da5a874636a88a040226991336",
            "f6e0cfeade5d4d69aeb4567708814ee7"
          ]
        },
        "outputId": "e5f139ed-963b-49b9-bc09-56c106b28b93"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading (…)okenizer_config.json:   0%|          | 0.00/252 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "5e46ca6cebaa497bb3542c85689fa117"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading (…)lve/main/config.json:   0%|          | 0.00/689 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "8fdb9e2851374dbd867a6ab0924a77ee"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading (…)solve/main/vocab.txt:   0%|          | 0.00/232k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "ea971bb214894a919bd86597c746cbe3"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading (…)cial_tokens_map.json:   0%|          | 0.00/112 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "93fc6edb6c9a4922a15f7dd6ff0172da"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading (…)\"pytorch_model.bin\";:   0%|          | 0.00/438M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "f75c2adcf7f54d5c98a4febc130fd0ad"
            }
          },
          "metadata": {}
        }
      ],
      "source": [
        "# Download validation data\n",
        "# data 1 contains queries for new themes, while data 2 contains queries for old themes\n",
        "validation_data = 1 #@param [\"1\", \"2\"] {type:\"raw\"}\n",
        "# Choose -1 to test on all themes\n",
        "num_themes_to_test = 10 #@param {type:\"integer\"}\n",
        "\n",
        "download_validation_data(round = validation_data)\n",
        "\n",
        "# Dowload sentence encoder and qa models\n",
        "sentence_encoder = \"SimCSE\" #@param [\"universal-sentence-encoder-qa-v3\", \"mpnet-base-v2\", \"distilroberta-v1\", \"minilm-l12-v2\", \"SimCSE\", \"mpnet-base-v2-fine-tuned\"]\n",
        "sents_encoder = load_encoder(sentence_encoder)\n",
        "# %cd '/content/qamodels'\n",
        "# download_qa_models()\n",
        "# %cd '..'\n",
        "# models_mapping = create_mapping()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "i2sheww6oVzq"
      },
      "outputs": [],
      "source": [
        "indexing_library = \"faiss\" #@param [\"faiss\"]\n",
        "search_previously_answered_queries = False #@param {type:\"boolean\"}\n",
        "context_generation = \"top-k nearest sentences\" #@param [\"top-k nearest sentences\", \"top-k nearest sentences with window of m sentences\", \"paragraphs of top k sentences\"]\n",
        "context_option = 1\n",
        "if context_generation == \"top-k nearest sentences with window of m sentences\":\n",
        "    context_option = 2\n",
        "elif context_generation == \"paragraphs of top k sentences\":\n",
        "    context_option = 3\n",
        "k = 10 #@param {type:\"slider\", min:1, max:15, step:1}\n",
        "m = 1 #@param {type:\"slider\", min:1, max:3, step:1}\n",
        "context_length_threshold = 205 #@param {type:\"slider\", min:100, max:250, step:5}\n",
        "distance_threshold = 1.8 #@param {type:\"slider\", min:1.0, max:1.8, step:0.05}\n",
        "\n",
        "use_pretrained_model_for_QA = True #@param {type:\"boolean\"}\n",
        "model_id = \"PremalMatalia/electra-base-best-squad2\" #@param [\"twmkn9/albert-base-v2-squad2\", \"deepset/minilm-uncased-squad2\", \"deepset/electra-base-squad2\", \"deepset/roberta-base-squad2\", \"deepset/deberta-v3-base-squad2\", \"deepset/roberta-base-squad2-distilled\", \"PremalMatalia/electra-base-best-squad2\", \"PremalMatalia/roberta-base-best-squad2\"]\n",
        "use_onnx = True #@param {type:\"boolean\"}\n",
        "optimize_model = True #@param {type:\"boolean\"}\n",
        "quantize_model = False #@param {type:\"boolean\"}\n",
        "tokenizer_link = \"https://drive.google.com/u/1/uc?id=1Rq9kXnOpbY1FsDBjHtlx4i7scrnk_0A9&export=download\" #@param {type:\"string\"}\n",
        "model_links_json = \"https://drive.google.com/u/1/uc?id=1usU8GcPTzIakelkJd7ChvQGxqwlEJxoz&export=download\" #@param {type:\"string\"}\n",
        "num_qa_passes = 1 #@param {type:\"slider\", min:1, max:5, step:1}\n",
        "\n",
        "params = {\n",
        "    'encoder_name': sentence_encoder,\n",
        "    'encoder': sents_encoder,\n",
        "    # 'models_mapping': models_mapping,\n",
        "    'tok_link': tokenizer_link,\n",
        "    'ctx_option': context_option,\n",
        "    'k': k,\n",
        "    'm': m,\n",
        "    'ctx_threshold': context_length_threshold,\n",
        "    'distance_threshold': distance_threshold,\n",
        "    'use_pretrained': use_pretrained_model_for_QA,\n",
        "    'model_id': model_id,\n",
        "    'use_onnx': use_onnx,\n",
        "    'optimize': optimize_model,\n",
        "    'quantize': quantize_model,\n",
        "    'num_themes': num_themes_to_test,\n",
        "    'qa_passes': num_qa_passes\n",
        "}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FL1j7Ll5ce7d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 976,
          "referenced_widgets": [
            "a45951df22984efdbe00b4d65daa4dc6",
            "855090a3f45d4eebb987cbbd2bf271bc",
            "06e1376dab834540a042a897dc61683a",
            "2c4fecbc235d4b929479e5ce587e72ab",
            "9eba878120744c22b7c8374a01335689",
            "4a12b088d0bb4d2d84a6417fe3c00b03",
            "1f56d944546b4c16aac4b511a05286d4",
            "07e61381ff0a4e6ebb6dd1b068c38c2e",
            "95827f07fecc473f8b0743bd52bfa49a",
            "0c4f2cd3070248949da77b13ea04f43e",
            "a0fc0bed83524a63a284ad6277a53a46",
            "4e18146581d342dc9aea70c73b9059fd",
            "26bf94bc382f4b8aa47734030fc6d789",
            "6ef8ac763cc74a3ab76758db98196195",
            "2badb4ab562b4378acf6f1d03b2983e3",
            "dfaf25d204c84b91bcc7629db9f17f80",
            "15f0a57013de4dc5854b81c36365beb0",
            "2cae8ec3495f43d5be727a5e82f192c5",
            "b721da0ecb1047c0a89d165b83b9ea05",
            "c89d7f6e1eaa435f85c0a21988af0895",
            "c70ead75d16241a1a802507ec26e6f45",
            "17c0422b6280417081c2b4ee0d5eb7bc",
            "56008c0784d948d790939ab3ad2416fc",
            "6a73776ef54243f5bc34ef43673440e3",
            "7480a8f7d4e34f909f068d209df08266",
            "821d39b2154d4619b96570233b05efb3",
            "7a7b5180f61144e3bd6cca311d9f1261",
            "7b12523306a2406881d7a49f175a90d9",
            "b30a4fcd572c4e6eb7fb8b4cc49df921",
            "2f7e5fc0c0cf497faa278a005c0d9fe1",
            "57afc18618274219a57e66aa040a0aa1",
            "d02486d7710840dcb756709f0b426ecb",
            "841b3278610448559ca0547ce9299b62",
            "c88ff834bce04877a8263fc9aa8c9658",
            "8ef4eba9a1614c0c930ee261532ef6ee",
            "b2f75b902e114860871d56e94da64e13",
            "f71958b283ea4183bb5bc3b6b7388fb2",
            "d5128c7a02e74386920bc377ce190da4",
            "a39af2f41f744091abbb834921b91bd4",
            "d9e1ed5f431b4b6ebeb58801ad180b9f",
            "682e6725dfa0430d82182b8443042bde",
            "227bce91d5604a7c938b5bdfaaab89f5",
            "a026aab1f317479fa790edbb51603fda",
            "6bb7e8237d1a4b22a43721e50663e00a",
            "65e5a635331f483ea00b389342aef4a7",
            "ef9409853ab0400981b156aaf773f2a2",
            "b3b5662e1c5544ac8343ae68707a6914",
            "ad31c2c720d24a4ca5c926fd60f32875",
            "7d2b9b0d591d4cdcb9365adcfeb31d51",
            "f68adc2e9af742a89fbc42e6997ce123",
            "833e0e3e5de24f8daca1da0c8b79ae68",
            "f41cf80f0b674031acdbbea6dc12fe6e",
            "1fe5f85c0f9d445599f0a42850c1916f",
            "7bf3729a79174620be770f06a4f44e03",
            "ec529a8662a14d01bba58ebc4913b7c9",
            "44665a4b5de54ad4b76d2195d5f931bf",
            "541b159c4252417fb820f104c8c448a6",
            "6316c1d751184feba6b4cbf206125e77",
            "ed6aab02518b491abc4a1c8d1d26ec72",
            "65de4b3151fa4ee1a148c98f0dd26b0d",
            "715d912540ee4e8b837a6933fdf74033",
            "899d25fd97474279be41d8ca127e5ed1",
            "91d136f49faa4fbf98e89b7501c1fcf1",
            "d4b42a978220400cafb0c87fe4cdbd9d",
            "d32888b6ba354a5082e46456ce3645f6",
            "7779a29f6b474020854c6e5a3e038a14"
          ]
        },
        "outputId": "ed88d49f-67ec-4f7e-acbd-9cfd01a06219"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading (…)okenizer_config.json:   0%|          | 0.00/549 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "a45951df22984efdbe00b4d65daa4dc6"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading (…)solve/main/vocab.txt:   0%|          | 0.00/232k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "4e18146581d342dc9aea70c73b9059fd"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading (…)/main/tokenizer.json:   0%|          | 0.00/466k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "56008c0784d948d790939ab3ad2416fc"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading (…)cial_tokens_map.json:   0%|          | 0.00/112 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "c88ff834bce04877a8263fc9aa8c9658"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading (…)lve/main/config.json:   0%|          | 0.00/783 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "65e5a635331f483ea00b389342aef4a7"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading (…)\"pytorch_model.bin\";:   0%|          | 0.00/436M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "44665a4b5de54ad4b76d2195d5f931bf"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.8/dist-packages/optimum/onnxruntime/configuration.py:706: FutureWarning: disable_embed_layer_norm will be deprecated soon, use disable_embed_layer_norm_fusion instead, disable_embed_layer_norm_fusion is set to True.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Theme: IPod\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|██████████| 326/326 [00:07<00:00, 42.86it/s]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Avg. ANN IT = 1.57 ms, Avg. QnA IT = 23.36 ms\n",
            "\n",
            "Theme: 2008_Sichuan_earthquake\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|██████████| 521/521 [00:11<00:00, 45.08it/s]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Avg. ANN IT = 1.16 ms, Avg. QnA IT = 22.2 ms\n",
            "\n",
            "Theme: Wayback_Machine\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|██████████| 208/208 [00:04<00:00, 49.62it/s]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Avg. ANN IT = 1.82 ms, Avg. QnA IT = 20.18 ms\n",
            "\n",
            "Theme: Canadian_Armed_Forces\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|██████████| 396/396 [00:08<00:00, 47.00it/s]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Avg. ANN IT = 1.15 ms, Avg. QnA IT = 21.29 ms\n",
            "\n",
            "Theme: Cardinal_(Catholicism)\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|██████████| 322/322 [00:06<00:00, 46.58it/s]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Avg. ANN IT = 1.87 ms, Avg. QnA IT = 21.48 ms\n",
            "\n",
            "Theme: Human_Development_Index\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|██████████| 168/168 [00:03<00:00, 42.45it/s]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Avg. ANN IT = 1.49 ms, Avg. QnA IT = 23.6 ms\n",
            "\n",
            "Theme: Heresy\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|██████████| 204/204 [00:04<00:00, 46.15it/s]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Avg. ANN IT = 1.57 ms, Avg. QnA IT = 21.69 ms\n",
            "\n",
            "Theme: Warsaw_Pact\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|██████████| 146/146 [00:03<00:00, 41.85it/s]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Avg. ANN IT = 1.41 ms, Avg. QnA IT = 23.94 ms\n",
            "\n",
            "Theme: Materialism\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|██████████| 203/203 [00:04<00:00, 45.69it/s]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Avg. ANN IT = 1.6 ms, Avg. QnA IT = 21.91 ms\n",
            "\n",
            "Theme: Pub\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|██████████| 356/356 [00:07<00:00, 48.58it/s]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Avg. ANN IT = 1.86 ms, Avg. QnA IT = 20.61 ms\n",
            "\n"
          ]
        }
      ],
      "source": [
        "theme_inf_time = predict_multiple_themes(params)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZC_D_IucatMq"
      },
      "outputs": [],
      "source": [
        "# Key Metrics -\n",
        "#\n",
        "# AIT - Average Inference Time\n",
        "#   ANN: AIT for Approximate Nearest Neighbour Search (including time for encoding queries)\n",
        "#   QnA: AIT for Question Answering on generated context\n",
        "# ansInCtx - For how many \"answerable\" queries, the answer exists in the generated context? (in %)\n",
        "# TP - True Postives\n",
        "# TN - True Negatives\n",
        "# TotP - Total Positives\n",
        "# TotN - Total Negatives\n",
        "# Para Acc - Paragraph Accuracy for retrieved paragraph id\n",
        "# F1 SCore - F1 Score of the answer predicted\n",
        "# Avg Ctx Size - Average length of the generated context (number of words)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bhiRGUe9VCYp",
        "outputId": "30c77b0f-dc8d-4bb8-c9bc-bc5925e59c8e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Theme             | Queries |  AIT: (ANN) + (QnA) = Total | ansInCtx % | TP % (TotP) | TN % (TotN) | Para Acc | F1 Score | Avg Ctx Size\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|-------------\n",
            "IPod              |     326 |   1.46 +  20.07 =  21.53 ms |    79.45 % |  76% ( 326) |   0% (   0) |   0.7638 |   0.7091 |   202.3\n",
            "2008_Sichuan_eart |     521 |   1.21 +  22.36 =  23.57 ms |    81.19 % |  77% ( 521) |   0% (   0) |   0.7735 |   0.7462 |   212.4\n",
            "Wayback_Machine   |     208 |   1.91 +  22.31 =  24.21 ms |    81.01 % |  64% (  79) |  89% ( 129) |   0.7981 |   0.8011 |   205.5\n",
            "Canadian_Armed_Fo |     396 |   1.19 +  21.55 =  22.73 ms |    74.86 % |  71% ( 179) |  91% ( 217) |   0.8258 |   0.8018 |   222.0\n",
            "Cardinal_(Catholi |     322 |    2.0 +  23.33 =  25.33 ms |     73.5 % |  66% ( 117) |  92% ( 205) |   0.8323 |    0.843 |   216.4\n",
            "Human_Development |     168 |   1.53 +  22.56 =   24.1 ms |    74.67 % |  60% (  75) |  89% (  93) |   0.7619 |   0.7839 |   211.0\n",
            "Heresy            |     204 |   1.64 +  23.25 =  24.89 ms |    90.32 % |  84% (  93) |  89% ( 111) |   0.8725 |   0.8674 |   210.0\n",
            "Warsaw_Pact       |     146 |   1.43 +  22.67 =   24.1 ms |    76.92 % |  67% (  52) |  92% (  94) |   0.8356 |   0.8354 |   231.9\n",
            "Materialism       |     203 |   1.62 +  21.24 =  22.86 ms |    81.48 % |  74% (  54) |  98% ( 149) |   0.9212 |   0.9159 |   179.5\n",
            "Pub               |     356 |   1.96 +  21.08 =  23.04 ms |    85.96 % |  82% ( 356) |   0% (   0) |   0.8258 |   0.8191 |   209.9\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|--------------\n",
            "Grand Total       |    2850 |   1.56 +  21.94 =   23.5 ms |    80.78 % |  75% (1852) |  92% ( 998) |   0.8147 |   0.8013 |   210.6\n"
          ]
        }
      ],
      "source": [
        "calculate_score(theme_inf_time, inf_time_threshold = 1000.0) # with simcse and electra-best (optimized)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VbjbmRzOxSAJ"
      },
      "source": [
        "## Results"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DDxE7DH90sss"
      },
      "source": [
        "### Experimentation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_4eigUBJO3eO"
      },
      "source": [
        "#### Pretrained Question Answering Models\n",
        "electra vs **electra-best** vs roberta-large vs roberta-distilled vs deberta vs albert vs minilm vs roberta"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "StRciZid0feI",
        "outputId": "cd9ccc6d-7aef-408f-bfd1-06c8b83808af"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Theme             | Queries |  AIT: (ANN) + (QnA) = Total | RPinTopK % | TP % (TotP) | TN % (TotN) | Para Acc | Final PA | F1 Score | Final F1\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|----------|---------\n",
            "IPod              |     326 |   1.07 +  14.46 =  15.53 ms |    98.16 % |  82% ( 326) |   0% (   0) |   0.8221 |  0.82209 |   0.8183 |  0.81834\n",
            "2008_Sichuan_eart |     521 |   0.91 +  17.42 =  18.33 ms |    94.43 % |  80% ( 521) |   0% (   0) |   0.8061 |  0.80614 |   0.7967 |  0.79671\n",
            "Wayback_Machine   |     208 |    1.1 +  17.33 =  18.44 ms |    98.08 % |  69% (  79) |  93% ( 129) |   0.8462 |  0.84615 |   0.8526 |  0.85256\n",
            "Canadian_Armed_Fo |     396 |   0.89 +  21.18 =  22.07 ms |    98.23 % |  82% ( 179) |  94% ( 217) |   0.8889 |  0.88889 |   0.8779 |  0.87788\n",
            "Cardinal_(Catholi |     322 |   1.12 +  18.03 =  19.15 ms |     97.2 % |  71% ( 117) |  95% ( 205) |   0.8696 |  0.86957 |   0.8704 |  0.87045\n",
            "Human_Development |     168 |    1.2 +   17.5 =   18.7 ms |    97.62 % |  73% (  75) |  87% (  93) |   0.8095 |  0.80952 |   0.8491 |  0.84911\n",
            "Heresy            |     204 |   1.11 +   16.9 =  18.01 ms |    98.04 % |  78% (  93) |  93% ( 111) |   0.8676 |  0.86765 |   0.8562 |  0.85618\n",
            "Warsaw_Pact       |     146 |   1.01 +  24.65 =  25.67 ms |    97.95 % |  76% (  52) |  97% (  94) |   0.9041 |  0.90411 |   0.9266 |  0.92661\n",
            "Materialism       |     203 |   1.05 +  16.37 =  17.42 ms |    98.52 % |  87% (  54) |  97% ( 149) |   0.9458 |  0.94581 |   0.9317 |  0.93174\n",
            "Pub               |     356 |   1.22 +  15.89 =  17.11 ms |    94.94 % |  82% ( 356) |   0% (   0) |   0.8287 |  0.82865 |   0.8206 |  0.82064\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|----------|---------\n",
            "Grand Total       |    2850 |   1.05 +  17.74 =  18.79 ms |    96.95 % |  80% (1852) |  94% ( 998) |   0.8519 |  0.85886 |   0.8495 |  0.86002\n"
          ]
        }
      ],
      "source": [
        "# with mpnet and electra-base (optimized)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "n8L16VoD5S8Z",
        "outputId": "cbfc41f3-4329-41ed-93b7-7da3d6a3d6f0"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Theme             | Queries |  AIT: (ANN) + (QnA) = Total | ansInCtx % | TP % (TotP) | TN % (TotN) | Para Acc | Final PA | F1 Score | Final F1\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|----------|---------\n",
            "IPod              |     326 |   1.14 +  15.67 =   16.8 ms |    93.87 % |  87% ( 326) |   0% (   0) |   0.8742 |  0.87423 |   0.8597 |  0.85967\n",
            "2008_Sichuan_eart |     521 |   0.97 +  18.65 =  19.62 ms |    93.09 % |  86% ( 521) |   0% (   0) |   0.8656 |  0.86564 |   0.8561 |  0.85607\n",
            "Wayback_Machine   |     208 |   1.18 +  18.68 =  19.86 ms |    88.61 % |  73% (  79) |  93% ( 129) |   0.8558 |  0.85577 |   0.8619 |  0.86194\n",
            "Canadian_Armed_Fo |     396 |   0.94 +  21.54 =  22.47 ms |    93.85 % |  88% ( 179) |  92% ( 217) |   0.9066 |  0.90657 |   0.8851 |  0.88513\n",
            "Cardinal_(Catholi |     322 |   1.22 +  22.73 =  23.95 ms |    88.89 % |  79% ( 117) |  93% ( 205) |    0.882 |  0.88199 |   0.8752 |  0.87523\n",
            "Human_Development |     168 |   1.25 +  18.89 =  20.14 ms |    89.33 % |  74% (  75) |  84% (  93) |   0.8036 |  0.80357 |   0.8453 |  0.84526\n",
            "Heresy            |     204 |   1.18 +  18.01 =  19.19 ms |     91.4 % |  87% (  93) |  90% ( 111) |   0.8922 |  0.89216 |   0.8856 |  0.88559\n",
            "Warsaw_Pact       |     146 |   1.07 +  23.98 =  25.05 ms |    96.15 % |  78% (  52) |  94% (  94) |   0.8904 |  0.89041 |   0.9155 |  0.91553\n",
            "Materialism       |     203 |   1.12 +  17.32 =  18.44 ms |    94.44 % |  87% (  54) |  95% ( 149) |    0.936 |  0.93596 |   0.9265 |  0.92653\n",
            "Pub               |     356 |   1.26 +   16.6 =  17.86 ms |    91.01 % |  86% ( 356) |   0% (   0) |   0.8652 |  0.86517 |    0.858 |    0.858\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|----------|---------\n",
            "Grand Total       |    2850 |   1.11 +  19.06 =  20.17 ms |    92.33 % |  85% (1852) |  92% ( 998) |   0.8779 |  0.87715 |   0.8729 |  0.87689\n"
          ]
        }
      ],
      "source": [
        "# with mpnet and electra-best (optimized)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jnT3T5LIMk8a",
        "outputId": "57476560-03a8-48dd-afc5-28e32622463a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Theme             | Queries |  AIT: (ANN) + (QnA) = Total | ansInCtx % | TP % (TotP) | TN % (TotN) | Para Acc | Final PA | F1 Score | Final F1\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|----------|---------\n",
            "IPod              |     326 |   1.28 +  45.97 =  47.25 ms |    93.87 % |  88% ( 326) |   0% (   0) |   0.8834 |  0.88344 |   0.8505 |   0.8505\n",
            "2008_Sichuan_eart |     521 |   0.97 +  57.54 =  58.51 ms |    93.09 % |  85% ( 521) |   0% (   0) |   0.8599 |  0.85988 |   0.8329 |  0.83294\n",
            "Wayback_Machine   |     208 |    1.2 +  55.56 =  56.76 ms |    88.61 % |  77% (  79) |  95% ( 129) |   0.8846 |  0.88462 |   0.8802 |   0.8802\n",
            "Canadian_Armed_Fo |     396 |   0.99 +  70.48 =  71.47 ms |    93.85 % |  82% ( 179) |  95% ( 217) |   0.8939 |  0.89394 |   0.8676 |  0.86757\n",
            "Cardinal_(Catholi |     322 |   1.29 +  61.88 =  63.17 ms |    88.89 % |  82% ( 117) |  98% ( 205) |   0.9255 |  0.92547 |   0.9199 |  0.91987\n",
            "Human_Development |     168 |   1.36 +  56.69 =  58.05 ms |    89.33 % |  66% (  75) |  89% (  93) |   0.7917 |  0.79167 |   0.8226 |  0.82259\n",
            "Heresy            |     204 |   1.24 +  55.22 =  56.46 ms |     91.4 % |  89% (  93) |  96% ( 111) |   0.9314 |  0.93137 |   0.9221 |  0.92209\n",
            "Warsaw_Pact       |     146 |   1.11 +  74.05 =  75.16 ms |    96.15 % |  76% (  52) |  97% (  94) |   0.9041 |  0.90411 |   0.9163 |  0.91634\n",
            "Materialism       |     203 |   1.19 +  53.76 =  54.95 ms |    94.44 % |  77% (  54) |  97% ( 149) |   0.9261 |  0.92611 |   0.9102 |   0.9102\n",
            "Pub               |     356 |   1.32 +  53.41 =  54.73 ms |    91.01 % |  85% ( 356) |   0% (   0) |   0.8567 |  0.85674 |   0.8402 |  0.84023\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|----------|---------\n",
            "Grand Total       |    2850 |   1.17 +   58.2 =  59.37 ms |    92.33 % |  84% (1852) |  96% ( 998) |   0.8842 |  0.88573 |   0.8695 |  0.87625\n"
          ]
        }
      ],
      "source": [
        "# with mpnet and roberta-large (optimized)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8Ug0qfYE7OdQ",
        "outputId": "704be2b1-9d84-4f5f-8f52-6cd302c7d391"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Theme             | Queries |  AIT: (ANN) + (QnA) = Total | RPinTopK % | TP % (TotP) | TN % (TotN) | Para Acc | Final PA | F1 Score | Final F1\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|----------|---------\n",
            "IPod              |     326 |    1.2 +  15.33 =  16.53 ms |    98.16 % |  87% ( 326) |   0% (   0) |   0.8773 |   0.8773 |   0.8228 |  0.82285\n",
            "2008_Sichuan_eart |     521 |   0.88 +   21.6 =  22.47 ms |    94.43 % |  85% ( 521) |   0% (   0) |   0.8599 |  0.85988 |   0.8014 |  0.80144\n",
            "Wayback_Machine   |     208 |   1.17 +  18.54 =  19.71 ms |    98.08 % |  67% (  79) |  90% ( 129) |   0.8173 |  0.81731 |   0.8096 |  0.80964\n",
            "Canadian_Armed_Fo |     396 |   0.86 +  21.46 =  22.32 ms |    98.23 % |  83% ( 179) |  93% ( 217) |   0.8889 |  0.88889 |   0.8337 |  0.83366\n",
            "Cardinal_(Catholi |     322 |   1.14 +   18.9 =  20.04 ms |     97.2 % |  79% ( 117) |  94% ( 205) |   0.8882 |   0.8882 |   0.8718 |  0.87177\n",
            "Human_Development |     168 |   1.16 +  16.89 =  18.04 ms |    97.62 % |  70% (  75) |  87% (  93) |   0.7976 |  0.79762 |   0.8201 |  0.82011\n",
            "Heresy            |     204 |   1.09 +  16.52 =   17.6 ms |    98.04 % |  86% (  93) |  90% ( 111) |   0.8873 |  0.88725 |   0.8667 |  0.86675\n",
            "Warsaw_Pact       |     146 |   0.99 +  21.86 =  22.85 ms |    97.95 % |  67% (  52) |  95% (  94) |   0.8562 |  0.85616 |   0.8467 |  0.84671\n",
            "Materialism       |     203 |   1.03 +  16.05 =  17.08 ms |    98.52 % |  75% (  54) |  94% ( 149) |   0.8966 |  0.89655 |   0.8722 |  0.87223\n",
            "Pub               |     356 |   1.17 +  15.95 =  17.12 ms |    94.94 % |  85% ( 356) |   0% (   0) |   0.8567 |  0.85674 |   0.8201 |  0.82014\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|----------|---------\n",
            "Grand Total       |    2850 |   1.05 +   18.6 =  19.65 ms |    96.95 % |  83% (1852) |  92% ( 998) |   0.8663 |  0.86259 |   0.8324 |  0.83653\n"
          ]
        }
      ],
      "source": [
        "# with mpnet and roberta-distilled (optimized)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4JL_ZBWv0feJ",
        "outputId": "38b9457c-7fbf-497b-f570-6ad1ac458821"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Theme             | Queries |  AIT: (ANN) + (QnA) = Total | RPinTopK % | TP % (TotP) | TN % (TotN) | Para Acc | Final PA | F1 Score | Final F1\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|----------|---------\n",
            "IPod              |     326 |   1.06 + 888.17 = 889.23 ms |    98.16 % |  86% ( 326) |   0% (   0) |    0.865 |  0.86503 |   0.7159 |   0.7159\n",
            "2008_Sichuan_eart |     521 |   0.88 + 1038.32 = 1039.2 ms |    94.43 % |  86% ( 521) |   0% (   0) |   0.8618 |  0.82929 |   0.7841 |  0.75456\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|----------|---------\n",
            "Grand Total       |     847 |   0.95 + 980.53 = 981.48 ms |    95.87 % |  86% ( 847) |   0% (   0) |    0.863 |  0.84716 |   0.7579 |  0.73523\n"
          ]
        }
      ],
      "source": [
        "# with mpnet and deberta base (no onnx support)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eZOvOJnE0feJ",
        "outputId": "c38f147d-28c3-41ea-be8e-0e4590b34c82"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Theme             | Queries |  AIT: (ANN) + (QnA) = Total | RPinTopK % | TP % (TotP) | TN % (TotN) | Para Acc | Final PA | F1 Score | Final F1\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|----------|---------\n",
            "IPod              |     326 |   1.06 +  15.24 =   16.3 ms |    98.16 % |  84% ( 326) |   0% (   0) |   0.8466 |  0.84663 |   0.7069 |  0.70693\n",
            "2008_Sichuan_eart |     521 |   0.92 +  17.27 =  18.19 ms |    94.43 % |  86% ( 521) |   0% (   0) |   0.8618 |   0.8618 |   0.7469 |   0.7469\n",
            "Wayback_Machine   |     208 |    1.1 +  17.62 =  18.72 ms |    98.08 % |  63% (  79) |  93% ( 129) |   0.8173 |  0.81731 |   0.8022 |   0.8022\n",
            "Canadian_Armed_Fo |     396 |   0.87 +  22.36 =  23.22 ms |    98.23 % |  74% ( 179) |  92% ( 217) |   0.8434 |  0.84343 |   0.7722 |  0.77218\n",
            "Cardinal_(Catholi |     322 |    1.5 +  20.78 =  22.28 ms |     97.2 % |  77% ( 117) |  94% ( 205) |    0.882 |  0.88199 |   0.8345 |  0.83449\n",
            "Human_Development |     168 |   1.33 +  20.19 =  21.52 ms |    97.62 % |  61% (  75) |  87% (  93) |    0.756 |  0.75595 |   0.7372 |  0.73716\n",
            "Heresy            |     204 |   1.14 +  19.92 =  21.05 ms |    98.04 % |  83% (  93) |  88% ( 111) |   0.8627 |  0.86275 |   0.8105 |  0.81052\n",
            "Warsaw_Pact       |     146 |   1.01 +  25.74 =  26.74 ms |    97.95 % |  71% (  52) |  94% (  94) |    0.863 |  0.86301 |   0.8528 |  0.85276\n",
            "Materialism       |     203 |   1.06 +  19.12 =  20.18 ms |    98.52 % |  75% (  54) |  93% ( 149) |   0.8867 |   0.8867 |   0.8508 |  0.85078\n",
            "Pub               |     356 |   1.61 +  18.46 =  20.07 ms |    94.94 % |  85% ( 356) |   0% (   0) |   0.8511 |  0.85112 |   0.7173 |   0.7173\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|----------|---------\n",
            "Grand Total       |    2850 |   1.15 +  19.24 =  20.39 ms |    96.95 % |  81% (1852) |  92% ( 998) |   0.8509 |  0.84707 |   0.7729 |  0.78312\n"
          ]
        }
      ],
      "source": [
        "# with mpnet and albert_base (optimized)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xN5XMTnZ0feJ",
        "outputId": "b4860db1-0a6b-492b-b528-306eb81864b2"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Theme             | Queries |  AIT: (ANN) + (QnA) = Total | RPinTopK % | TP % (TotP) | TN % (TotN) | Para Acc | Final PA | F1 Score | Final F1\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|----------|---------\n",
            "IPod              |     326 |   1.06 +   6.77 =   7.83 ms |    98.16 % |  84% ( 326) |   0% (   0) |   0.8466 |  0.84663 |   0.7855 |  0.78547\n",
            "2008_Sichuan_eart |     521 |   0.89 +   8.09 =   8.99 ms |    94.43 % |  85% ( 521) |   0% (   0) |   0.8503 |  0.85029 |    0.775 |  0.77496\n",
            "Wayback_Machine   |     208 |   1.12 +   8.01 =   9.13 ms |    98.08 % |  69% (  79) |  80% ( 129) |   0.7644 |  0.76442 |   0.7456 |  0.74559\n",
            "Canadian_Armed_Fo |     396 |   0.87 +   9.33 =   10.2 ms |    98.23 % |  78% ( 179) |  85% ( 217) |   0.8207 |  0.82071 |   0.7772 |  0.77722\n",
            "Cardinal_(Catholi |     322 |   1.07 +   8.19 =   9.26 ms |     97.2 % |  78% ( 117) |  89% ( 205) |    0.854 |  0.85404 |   0.8324 |   0.8324\n",
            "Human_Development |     168 |   1.17 +   7.87 =   9.04 ms |    97.62 % |  69% (  75) |  81% (  93) |   0.7619 |   0.7619 |   0.7705 |  0.77048\n",
            "Heresy            |     204 |   1.09 +   7.74 =   8.83 ms |    98.04 % |  86% (  93) |  87% ( 111) |   0.8676 |  0.86765 |   0.8457 |  0.84569\n",
            "Warsaw_Pact       |     146 |   0.99 +   10.4 =  11.38 ms |    97.95 % |  80% (  52) |  86% (  94) |   0.8425 |  0.84247 |   0.8329 |  0.83289\n",
            "Materialism       |     203 |   1.04 +   7.43 =   8.47 ms |    98.52 % |  79% (  54) |  91% ( 149) |   0.8818 |  0.88177 |   0.8658 |  0.86582\n",
            "Pub               |     356 |   1.18 +   7.14 =   8.32 ms |    94.94 % |  86% ( 356) |   0% (   0) |    0.868 |  0.86798 |   0.8104 |  0.81043\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|----------|---------\n",
            "Grand Total       |    2850 |   1.03 +   8.03 =   9.06 ms |    96.95 % |  82% (1852) |  86% ( 998) |     0.84 |  0.83578 |   0.7995 |   0.8041\n"
          ]
        }
      ],
      "source": [
        "# with mpnet and minilm (optimized)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "M1ZsXMte0feJ",
        "outputId": "1a861dc8-5e3e-4e1e-bf88-590f18f50ea4"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Theme             | Queries |  AIT: (ANN) + (QnA) = Total | RPinTopK % | TP % (TotP) | TN % (TotN) | Para Acc | Final PA | F1 Score | Final F1\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|----------|---------\n",
            "IPod              |     326 |   1.25 +  19.26 =  20.51 ms |    98.16 % |  85% ( 326) |   0% (   0) |   0.8589 |   0.8589 |   0.7811 |  0.78113\n",
            "2008_Sichuan_eart |     521 |   1.02 +  20.31 =  21.34 ms |    94.43 % |  84% ( 521) |   0% (   0) |   0.8484 |  0.84837 |   0.7828 |  0.78281\n",
            "Wayback_Machine   |     208 |   1.37 +  18.11 =  19.48 ms |    98.08 % |  62% (  79) |  85% ( 129) |   0.7644 |  0.76442 |   0.7542 |  0.75419\n",
            "Canadian_Armed_Fo |     396 |   0.99 +  21.72 =  22.72 ms |    98.23 % |  71% ( 179) |  89% ( 217) |   0.8131 |  0.81313 |    0.774 |  0.77395\n",
            "Cardinal_(Catholi |     322 |   1.29 +  20.16 =  21.45 ms |     97.2 % |  77% ( 117) |  91% ( 205) |   0.8665 |  0.86646 |   0.8467 |  0.84675\n",
            "Human_Development |     168 |   1.44 +  18.87 =  20.31 ms |    97.62 % |  69% (  75) |  81% (  93) |   0.7619 |   0.7619 |   0.7715 |  0.77147\n",
            "Heresy            |     204 |   1.36 +  18.43 =  19.78 ms |    98.04 % |  81% (  93) |  81% ( 111) |   0.8186 |  0.81863 |   0.7891 |  0.78908\n",
            "Warsaw_Pact       |     146 |   1.18 +  24.72 =   25.9 ms |    97.95 % |  67% (  52) |  92% (  94) |   0.8356 |  0.83562 |   0.8255 |  0.82547\n",
            "Materialism       |     203 |   1.27 +  18.04 =  19.31 ms |    98.52 % |  75% (  54) |  94% ( 149) |   0.8966 |  0.89655 |   0.8736 |   0.8736\n",
            "Pub               |     356 |   1.44 +  18.15 =  19.59 ms |    94.94 % |  81% ( 356) |   0% (   0) |   0.8174 |  0.81742 |   0.7638 |  0.76385\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|----------|---------\n",
            "Grand Total       |    2850 |   1.23 +  19.78 =  21.01 ms |    96.95 % |  80% (1852) |  88% ( 998) |   0.8323 |  0.82814 |   0.7926 |  0.79623\n"
          ]
        }
      ],
      "source": [
        "# with mpnet and roberta (optimized)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uFTXHe2dOqOR"
      },
      "source": [
        "#### Sentence Encoders\n",
        "**mpnet** vs distilroberta vs minilm vs universal sentence encoder"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tUFrmi_oXoWe",
        "outputId": "cd9ccc6d-7aef-408f-bfd1-06c8b83808af"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Theme             | Queries |  AIT: (ANN) + (QnA) = Total | RPinTopK % | TP % (TotP) | TN % (TotN) | Para Acc | Final PA | F1 Score | Final F1\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|----------|---------\n",
            "IPod              |     326 |   1.07 +  14.46 =  15.53 ms |    98.16 % |  82% ( 326) |   0% (   0) |   0.8221 |  0.82209 |   0.8183 |  0.81834\n",
            "2008_Sichuan_eart |     521 |   0.91 +  17.42 =  18.33 ms |    94.43 % |  80% ( 521) |   0% (   0) |   0.8061 |  0.80614 |   0.7967 |  0.79671\n",
            "Wayback_Machine   |     208 |    1.1 +  17.33 =  18.44 ms |    98.08 % |  69% (  79) |  93% ( 129) |   0.8462 |  0.84615 |   0.8526 |  0.85256\n",
            "Canadian_Armed_Fo |     396 |   0.89 +  21.18 =  22.07 ms |    98.23 % |  82% ( 179) |  94% ( 217) |   0.8889 |  0.88889 |   0.8779 |  0.87788\n",
            "Cardinal_(Catholi |     322 |   1.12 +  18.03 =  19.15 ms |     97.2 % |  71% ( 117) |  95% ( 205) |   0.8696 |  0.86957 |   0.8704 |  0.87045\n",
            "Human_Development |     168 |    1.2 +   17.5 =   18.7 ms |    97.62 % |  73% (  75) |  87% (  93) |   0.8095 |  0.80952 |   0.8491 |  0.84911\n",
            "Heresy            |     204 |   1.11 +   16.9 =  18.01 ms |    98.04 % |  78% (  93) |  93% ( 111) |   0.8676 |  0.86765 |   0.8562 |  0.85618\n",
            "Warsaw_Pact       |     146 |   1.01 +  24.65 =  25.67 ms |    97.95 % |  76% (  52) |  97% (  94) |   0.9041 |  0.90411 |   0.9266 |  0.92661\n",
            "Materialism       |     203 |   1.05 +  16.37 =  17.42 ms |    98.52 % |  87% (  54) |  97% ( 149) |   0.9458 |  0.94581 |   0.9317 |  0.93174\n",
            "Pub               |     356 |   1.22 +  15.89 =  17.11 ms |    94.94 % |  82% ( 356) |   0% (   0) |   0.8287 |  0.82865 |   0.8206 |  0.82064\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|----------|---------\n",
            "Grand Total       |    2850 |   1.05 +  17.74 =  18.79 ms |    96.95 % |  80% (1852) |  94% ( 998) |   0.8519 |  0.85886 |   0.8495 |  0.86002\n"
          ]
        }
      ],
      "source": [
        "# with mpnet and electra"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qIGrUN_eXoWd",
        "outputId": "1e387a8e-1212-4396-c85c-6ab9ab351bcc"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Theme             | Queries |  AIT: (ANN) + (QnA) = Total | RPinTopK % | TP % (TotP) | TN % (TotN) | Para Acc | Final PA | F1 Score | Final F1\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|----------|---------\n",
            "IPod              |     326 |   0.56 +  13.61 =  14.16 ms |    95.71 % |  76% ( 326) |   0% (   0) |   0.7669 |  0.76687 |   0.7529 |  0.75287\n",
            "2008_Sichuan_eart |     521 |   0.46 +   17.3 =  17.76 ms |    94.43 % |  80% ( 521) |   0% (   0) |     0.81 |  0.80998 |      0.8 |  0.80002\n",
            "Wayback_Machine   |     208 |   0.59 +  16.71 =  17.31 ms |    98.08 % |  79% (  79) |  91% ( 129) |   0.8702 |  0.87019 |    0.867 |  0.86699\n",
            "Canadian_Armed_Fo |     396 |   0.48 +  18.67 =  19.15 ms |    97.47 % |  79% ( 179) |  95% ( 217) |   0.8838 |  0.88384 |   0.8678 |  0.86782\n",
            "Cardinal_(Catholi |     322 |   0.56 +  17.29 =  17.84 ms |    96.89 % |  70% ( 117) |  93% ( 205) |    0.854 |  0.85404 |   0.8566 |   0.8566\n",
            "Human_Development |     168 |   0.58 +  15.97 =  16.55 ms |    96.43 % |  64% (  75) |  88% (  93) |   0.7738 |  0.77381 |     0.81 |  0.81005\n",
            "Heresy            |     204 |   0.54 +  16.99 =  17.54 ms |    98.04 % |  82% (  93) |  93% ( 111) |   0.8873 |  0.88725 |   0.8669 |  0.86685\n",
            "Warsaw_Pact       |     146 |   0.52 +  21.33 =  21.85 ms |    97.26 % |  71% (  52) |  95% (  94) |   0.8699 |  0.86986 |   0.8855 |  0.88552\n",
            "Materialism       |     203 |   0.55 +  16.29 =  16.84 ms |    100.0 % |  88% (  54) |  98% ( 149) |   0.9606 |  0.96059 |   0.9564 |  0.95637\n",
            "Pub               |     356 |   0.62 +  16.19 =  16.81 ms |    95.79 % |  85% ( 356) |   0% (   0) |   0.8511 |  0.85112 |   0.8448 |  0.84485\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|----------|---------\n",
            "Grand Total       |    2850 |   0.54 +  16.92 =  17.46 ms |    96.63 % |  79% (1852) |  94% ( 998) |    0.847 |  0.85276 |   0.8418 |  0.85079\n"
          ]
        }
      ],
      "source": [
        "# with distilroberta and electra"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zldksakbXoWd",
        "outputId": "93864887-8976-45eb-f8ab-4a3f34431b37"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Theme             | Queries |  AIT: (ANN) + (QnA) = Total | RPinTopK % | TP % (TotP) | TN % (TotN) | Para Acc | Final PA | F1 Score | Final F1\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|----------|---------\n",
            "IPod              |     326 |   0.48 +  14.62 =   15.1 ms |    96.93 % |  80% ( 326) |   0% (   0) |   0.8067 |  0.80675 |   0.7684 |  0.76841\n",
            "2008_Sichuan_eart |     521 |   0.37 +  18.15 =  18.52 ms |    93.47 % |  79% ( 521) |   0% (   0) |   0.7965 |  0.79655 |    0.776 |  0.77597\n",
            "Wayback_Machine   |     208 |   0.44 +  16.59 =  17.03 ms |    96.15 % |  65% (  79) |  93% ( 129) |   0.8317 |  0.83173 |   0.8253 |  0.82532\n",
            "Canadian_Armed_Fo |     396 |   0.39 +  19.68 =  20.07 ms |    97.73 % |  79% ( 179) |  92% ( 217) |   0.8687 |  0.86869 |   0.8533 |  0.85333\n",
            "Cardinal_(Catholi |     322 |   0.44 +  17.76 =   18.2 ms |    96.27 % |  66% ( 117) |  96% ( 205) |   0.8571 |  0.85714 |   0.8547 |   0.8547\n",
            "Human_Development |     168 |   0.45 +  16.03 =  16.48 ms |    94.64 % |  58% (  75) |  89% (  93) |    0.756 |  0.75595 |   0.8012 |  0.80123\n",
            "Heresy            |     204 |   0.42 +  17.19 =  17.61 ms |    97.55 % |  80% (  93) |  93% ( 111) |   0.8775 |  0.87745 |   0.8606 |  0.86055\n",
            "Warsaw_Pact       |     146 |   0.43 +  24.46 =  24.89 ms |    97.26 % |  61% (  52) |  96% (  94) |   0.8425 |  0.84247 |   0.8803 |  0.88027\n",
            "Materialism       |     203 |   0.44 +   16.0 =  16.44 ms |    99.01 % |  88% (  54) |  97% ( 149) |   0.9507 |  0.95074 |   0.9465 |  0.94652\n",
            "Pub               |     356 |   0.43 +  15.78 =  16.21 ms |    96.35 % |  84% ( 356) |   0% (   0) |   0.8455 |  0.84551 |    0.841 |  0.84104\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|----------|---------\n",
            "Grand Total       |    2850 |   0.42 +  17.48 =   17.9 ms |    96.28 % |  78% (1852) |  94% ( 998) |     0.84 |   0.8433 |   0.8315 |  0.84073\n"
          ]
        }
      ],
      "source": [
        "# with minilm and electra"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2F0MIGorXoWe",
        "outputId": "59dbd0da-9071-4072-b81c-88b65c5d9e9e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Theme             | Queries |  AIT: (ANN) + (QnA) = Total | RPinTopK % | TP % (TotP) | TN % (TotN) | Para Acc | Final PA | F1 Score | Final F1\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|----------|---------\n",
            "IPod              |     326 |   7.89 +  14.91 =   22.8 ms |    96.32 % |  77% ( 326) |   0% (   0) |   0.7791 |  0.77914 |   0.7541 |  0.75406\n",
            "2008_Sichuan_eart |     521 |   0.23 +  16.79 =  17.02 ms |    93.28 % |  79% ( 521) |   0% (   0) |   0.7927 |  0.79271 |   0.7884 |  0.78839\n",
            "Wayback_Machine   |     208 |   0.36 +  17.08 =  17.44 ms |    95.19 % |  62% (  79) |  93% ( 129) |   0.8173 |  0.81731 |   0.8113 |  0.81127\n",
            "Canadian_Armed_Fo |     396 |   0.25 +  19.19 =  19.44 ms |    94.95 % |  70% ( 179) |  94% ( 217) |   0.8409 |  0.84091 |   0.8325 |  0.83255\n",
            "Cardinal_(Catholi |     322 |   0.31 +  16.54 =  16.85 ms |    95.65 % |  61% ( 117) |  96% ( 205) |   0.8354 |   0.8354 |   0.8423 |  0.84232\n",
            "Human_Development |     168 |   0.73 +   20.5 =  21.22 ms |    96.43 % |  65% (  75) |  90% (  93) |   0.7917 |  0.79167 |     0.84 |  0.83995\n",
            "Heresy            |     204 |   0.48 +   18.2 =  18.68 ms |    98.53 % |  80% (  93) |  95% ( 111) |   0.8873 |  0.88725 |   0.8665 |  0.86645\n",
            "Warsaw_Pact       |     146 |   0.89 +  17.28 =  18.17 ms |    95.89 % |  75% (  52) |  97% (  94) |   0.8973 |  0.89726 |   0.9041 |  0.90411\n",
            "Materialism       |     203 |   0.34 +   17.1 =  17.44 ms |    97.04 % |  79% (  54) |  96% ( 149) |   0.9212 |  0.92118 |   0.9088 |  0.90875\n",
            "Pub               |     356 |   0.45 +  18.35 =   18.8 ms |     94.1 % |  83% ( 356) |   0% (   0) |   0.8343 |  0.83427 |   0.8299 |   0.8299\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|----------|---------\n",
            "Grand Total       |    2850 |   1.24 +  17.46 =  18.71 ms |    95.33 % |  76% (1852) |  95% ( 998) |   0.8309 |  0.83971 |   0.8267 |  0.83777\n"
          ]
        }
      ],
      "source": [
        "# with universal sentence encoder and electra"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "30c77b0f-dc8d-4bb8-c9bc-bc5925e59c8e",
        "id": "I7hHjokEnIvo"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Theme             | Queries |  AIT: (ANN) + (QnA) = Total | ansInCtx % | TP % (TotP) | TN % (TotN) | Para Acc | F1 Score | Avg Ctx Size\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|-------------\n",
            "IPod              |     326 |   1.46 +  20.07 =  21.53 ms |    79.45 % |  76% ( 326) |   0% (   0) |   0.7638 |   0.7091 |   202.3\n",
            "2008_Sichuan_eart |     521 |   1.21 +  22.36 =  23.57 ms |    81.19 % |  77% ( 521) |   0% (   0) |   0.7735 |   0.7462 |   212.4\n",
            "Wayback_Machine   |     208 |   1.91 +  22.31 =  24.21 ms |    81.01 % |  64% (  79) |  89% ( 129) |   0.7981 |   0.8011 |   205.5\n",
            "Canadian_Armed_Fo |     396 |   1.19 +  21.55 =  22.73 ms |    74.86 % |  71% ( 179) |  91% ( 217) |   0.8258 |   0.8018 |   222.0\n",
            "Cardinal_(Catholi |     322 |    2.0 +  23.33 =  25.33 ms |     73.5 % |  66% ( 117) |  92% ( 205) |   0.8323 |    0.843 |   216.4\n",
            "Human_Development |     168 |   1.53 +  22.56 =   24.1 ms |    74.67 % |  60% (  75) |  89% (  93) |   0.7619 |   0.7839 |   211.0\n",
            "Heresy            |     204 |   1.64 +  23.25 =  24.89 ms |    90.32 % |  84% (  93) |  89% ( 111) |   0.8725 |   0.8674 |   210.0\n",
            "Warsaw_Pact       |     146 |   1.43 +  22.67 =   24.1 ms |    76.92 % |  67% (  52) |  92% (  94) |   0.8356 |   0.8354 |   231.9\n",
            "Materialism       |     203 |   1.62 +  21.24 =  22.86 ms |    81.48 % |  74% (  54) |  98% ( 149) |   0.9212 |   0.9159 |   179.5\n",
            "Pub               |     356 |   1.96 +  21.08 =  23.04 ms |    85.96 % |  82% ( 356) |   0% (   0) |   0.8258 |   0.8191 |   209.9\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|--------------\n",
            "Grand Total       |    2850 |   1.56 +  21.94 =   23.5 ms |    80.78 % |  75% (1852) |  92% ( 998) |   0.8147 |   0.8013 |   210.6\n"
          ]
        }
      ],
      "source": [
        "# with simcse and electra-best (optimized)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dcNnLJxOPW08"
      },
      "source": [
        "#### Variation of k\n",
        "5 | 6 | **7** | 8 | 9 | 10 | 14 (2 passes)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "q3nxCt197SVs",
        "outputId": "cbfc41f3-4329-41ed-93b7-7da3d6a3d6f0"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Theme             | Queries |  AIT: (ANN) + (QnA) = Total | ansInCtx % | TP % (TotP) | TN % (TotN) | Para Acc | Final PA | F1 Score | Final F1\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|----------|---------\n",
            "IPod              |     326 |   1.14 +  15.67 =   16.8 ms |    93.87 % |  87% ( 326) |   0% (   0) |   0.8742 |  0.87423 |   0.8597 |  0.85967\n",
            "2008_Sichuan_eart |     521 |   0.97 +  18.65 =  19.62 ms |    93.09 % |  86% ( 521) |   0% (   0) |   0.8656 |  0.86564 |   0.8561 |  0.85607\n",
            "Wayback_Machine   |     208 |   1.18 +  18.68 =  19.86 ms |    88.61 % |  73% (  79) |  93% ( 129) |   0.8558 |  0.85577 |   0.8619 |  0.86194\n",
            "Canadian_Armed_Fo |     396 |   0.94 +  21.54 =  22.47 ms |    93.85 % |  88% ( 179) |  92% ( 217) |   0.9066 |  0.90657 |   0.8851 |  0.88513\n",
            "Cardinal_(Catholi |     322 |   1.22 +  22.73 =  23.95 ms |    88.89 % |  79% ( 117) |  93% ( 205) |    0.882 |  0.88199 |   0.8752 |  0.87523\n",
            "Human_Development |     168 |   1.25 +  18.89 =  20.14 ms |    89.33 % |  74% (  75) |  84% (  93) |   0.8036 |  0.80357 |   0.8453 |  0.84526\n",
            "Heresy            |     204 |   1.18 +  18.01 =  19.19 ms |     91.4 % |  87% (  93) |  90% ( 111) |   0.8922 |  0.89216 |   0.8856 |  0.88559\n",
            "Warsaw_Pact       |     146 |   1.07 +  23.98 =  25.05 ms |    96.15 % |  78% (  52) |  94% (  94) |   0.8904 |  0.89041 |   0.9155 |  0.91553\n",
            "Materialism       |     203 |   1.12 +  17.32 =  18.44 ms |    94.44 % |  87% (  54) |  95% ( 149) |    0.936 |  0.93596 |   0.9265 |  0.92653\n",
            "Pub               |     356 |   1.26 +   16.6 =  17.86 ms |    91.01 % |  86% ( 356) |   0% (   0) |   0.8652 |  0.86517 |    0.858 |    0.858\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|----------|---------\n",
            "Grand Total       |    2850 |   1.11 +  19.06 =  20.17 ms |    92.33 % |  85% (1852) |  92% ( 998) |   0.8779 |  0.87715 |   0.8729 |  0.87689\n"
          ]
        }
      ],
      "source": [
        "# with mpnet and electra-best (optimized) k = 7"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "B1uwnPOa-G-C",
        "outputId": "34529826-9450-43e7-9019-203a470002fa"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Theme             | Queries |  AIT: (ANN) + (QnA) = Total | ansInCtx % | TP % (TotP) | TN % (TotN) | Para Acc | Final PA | F1 Score | Final F1\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|----------|---------\n",
            "IPod              |     326 |   1.13 +   35.9 =  37.02 ms |    96.01 % |  82% ( 326) |   0% (   0) |   0.8252 |  0.82515 |   0.8054 |  0.80544\n",
            "2008_Sichuan_eart |     521 |    1.1 +  46.34 =  47.44 ms |    96.35 % |  84% ( 521) |   0% (   0) |   0.8484 |  0.84837 |   0.8314 |  0.83139\n",
            "Wayback_Machine   |     208 |   1.22 +  47.81 =  49.02 ms |    93.67 % |  72% (  79) |  86% ( 129) |   0.8077 |  0.80769 |   0.8096 |  0.80959\n",
            "Canadian_Armed_Fo |     396 |   0.94 +  49.38 =  50.33 ms |    95.53 % |  82% ( 179) |  87% ( 217) |    0.851 |  0.85101 |   0.8351 |  0.83507\n",
            "Cardinal_(Catholi |     322 |   1.18 +  47.67 =  48.85 ms |    93.16 % |  80% ( 117) |  88% ( 205) |   0.8571 |  0.85714 |   0.8619 |  0.86194\n",
            "Human_Development |     168 |   1.27 +  41.27 =  42.54 ms |    93.33 % |  78% (  75) |  84% (  93) |   0.8214 |  0.82143 |   0.8522 |  0.85218\n",
            "Heresy            |     204 |   1.17 +  41.76 =  42.93 ms |     95.7 % |  89% (  93) |  86% ( 111) |   0.8775 |  0.87745 |   0.8695 |  0.86949\n",
            "Warsaw_Pact       |     146 |   1.07 +  53.17 =  54.24 ms |    100.0 % |  78% (  52) |  88% (  94) |   0.8493 |  0.84932 |   0.8718 |  0.87182\n",
            "Materialism       |     203 |   1.14 +  40.25 =  41.39 ms |    98.15 % |  79% (  54) |  94% ( 149) |   0.9064 |   0.9064 |   0.9028 |  0.90285\n",
            "Pub               |     356 |   1.28 +   37.7 =  38.98 ms |    95.51 % |  86% ( 356) |   0% (   0) |    0.868 |  0.86798 |   0.8656 |   0.8656\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|----------|---------\n",
            "Grand Total       |    2850 |   1.14 +  44.03 =  45.17 ms |    95.73 % |  83% (1852) |  88% ( 998) |   0.8512 |  0.85119 |   0.8462 |  0.85054\n"
          ]
        }
      ],
      "source": [
        "# with mpnet and electra-best (optimized) k = 14 num_passes = 2"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "c_R6CDGO9W8-",
        "outputId": "3caed2dc-40c3-4ec0-d46a-22a6d34433c0"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Theme             | Queries |  AIT: (ANN) + (QnA) = Total | ansInCtx % | TP % (TotP) | TN % (TotN) | Para Acc | Final PA | F1 Score | Final F1\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|----------|---------\n",
            "IPod              |     326 |   1.18 +  21.98 =  23.16 ms |    95.71 % |  86% ( 326) |   0% (   0) |    0.862 |  0.86196 |   0.8531 |   0.8531\n",
            "2008_Sichuan_eart |     521 |   0.97 +  28.78 =  29.74 ms |    95.01 % |  85% ( 521) |   0% (   0) |    0.856 |  0.85605 |   0.8454 |  0.84537\n",
            "Wayback_Machine   |     208 |    1.2 +  30.09 =  31.29 ms |    91.14 % |  70% (  79) |  89% ( 129) |   0.8269 |  0.82692 |   0.8368 |  0.83683\n",
            "Canadian_Armed_Fo |     396 |   0.99 +  34.63 =  35.61 ms |    94.97 % |  83% ( 179) |  88% ( 217) |   0.8662 |  0.86616 |   0.8484 |  0.84841\n",
            "Cardinal_(Catholi |     322 |   1.17 +  29.17 =  30.34 ms |     90.6 % |  79% ( 117) |  92% ( 205) |   0.8789 |  0.87888 |   0.8796 |  0.87963\n",
            "Human_Development |     168 |   1.28 +  28.11 =  29.39 ms |    89.33 % |  74% (  75) |  87% (  93) |   0.8155 |  0.81548 |   0.8616 |  0.86163\n",
            "Heresy            |     204 |   1.17 +  25.48 =  26.64 ms |    94.62 % |  88% (  93) |  90% ( 111) |   0.8971 |  0.89706 |   0.8954 |   0.8954\n",
            "Warsaw_Pact       |     146 |   1.05 +  35.86 =  36.91 ms |    98.08 % |  82% (  52) |  89% (  94) |   0.8699 |  0.86986 |   0.8814 |  0.88141\n",
            "Materialism       |     203 |   1.11 +   25.1 =  26.21 ms |    94.44 % |  79% (  54) |  97% ( 149) |    0.931 |  0.93103 |   0.9271 |  0.92707\n",
            "Pub               |     356 |   1.25 +  23.33 =  24.58 ms |    93.54 % |  87% ( 356) |   0% (   0) |   0.8736 |   0.8736 |   0.8617 |  0.86166\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|----------|---------\n",
            "Grand Total       |    2850 |   1.12 +   28.1 =  29.21 ms |    94.22 % |  84% (1852) |  91% ( 998) |   0.8674 |   0.8677 |   0.8642 |  0.86905\n"
          ]
        }
      ],
      "source": [
        "# with mpnet and electra-best (optimized) k = 10"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SIxfAQEX8ysA",
        "outputId": "7f4acd31-e2de-4c79-e85a-ff5e13d445aa"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Theme             | Queries |  AIT: (ANN) + (QnA) = Total | ansInCtx % | TP % (TotP) | TN % (TotN) | Para Acc | Final PA | F1 Score | Final F1\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|----------|---------\n",
            "IPod              |     326 |   1.13 +  19.35 =  20.48 ms |    95.71 % |  85% ( 326) |   0% (   0) |   0.8589 |   0.8589 |   0.8551 |  0.85507\n",
            "2008_Sichuan_eart |     521 |   0.98 +  26.15 =  27.13 ms |    94.82 % |  86% ( 521) |   0% (   0) |   0.8637 |  0.86372 |   0.8548 |  0.85484\n",
            "Wayback_Machine   |     208 |   1.24 +  27.48 =  28.72 ms |    91.14 % |  70% (  79) |  90% ( 129) |   0.8317 |  0.83173 |   0.8432 |  0.84324\n",
            "Canadian_Armed_Fo |     396 |    1.0 +  30.37 =  31.37 ms |    94.41 % |  85% ( 179) |  90% ( 217) |   0.8813 |  0.88131 |   0.8651 |  0.86512\n",
            "Cardinal_(Catholi |     322 |   1.22 +  25.36 =  26.58 ms |    89.74 % |  78% ( 117) |  93% ( 205) |    0.882 |  0.88199 |   0.8843 |  0.88428\n",
            "Human_Development |     168 |   1.29 +  23.74 =  25.03 ms |    89.33 % |  73% (  75) |  84% (  93) |   0.7976 |  0.79762 |   0.8438 |  0.84377\n",
            "Heresy            |     204 |   1.16 +  21.92 =  23.08 ms |    92.47 % |  86% (  93) |  90% ( 111) |   0.8824 |  0.88235 |   0.8807 |  0.88069\n",
            "Warsaw_Pact       |     146 |   1.09 +  34.72 =  35.81 ms |    98.08 % |  82% (  52) |  89% (  94) |   0.8699 |  0.86986 |   0.8787 |  0.87867\n",
            "Materialism       |     203 |   1.16 +  24.57 =  25.73 ms |    94.44 % |  83% (  54) |  96% ( 149) |    0.931 |  0.93103 |   0.9224 |  0.92242\n",
            "Pub               |     356 |   1.29 +  21.39 =  22.68 ms |    93.26 % |  87% ( 356) |   0% (   0) |   0.8736 |   0.8736 |   0.8688 |  0.86878\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|----------|---------\n",
            "Grand Total       |    2850 |   1.14 +  25.25 =  26.39 ms |     93.9 % |  84% (1852) |  91% ( 998) |   0.8691 |  0.86721 |   0.8677 |  0.86969\n"
          ]
        }
      ],
      "source": [
        "# with mpnet and electra-best (optimized) k = 9"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RPX11anA8cXJ",
        "outputId": "21ca23ce-0ae2-4a65-eb5b-a238ab0b2a36"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Theme             | Queries |  AIT: (ANN) + (QnA) = Total | ansInCtx % | TP % (TotP) | TN % (TotN) | Para Acc | Final PA | F1 Score | Final F1\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|----------|---------\n",
            "IPod              |     326 |   1.11 +  17.16 =  18.27 ms |    95.09 % |  86% ( 326) |   0% (   0) |    0.865 |  0.86503 |   0.8661 |  0.86606\n",
            "2008_Sichuan_eart |     521 |   0.96 +  21.43 =  22.39 ms |    93.86 % |  85% ( 521) |   0% (   0) |    0.856 |  0.85605 |   0.8451 |   0.8451\n",
            "Wayback_Machine   |     208 |    1.2 +  22.25 =  23.46 ms |    89.87 % |  73% (  79) |  90% ( 129) |   0.8413 |  0.84135 |   0.8432 |  0.84324\n",
            "Canadian_Armed_Fo |     396 |   1.14 +  28.23 =  29.37 ms |    93.85 % |  88% ( 179) |  91% ( 217) |    0.899 |  0.89899 |   0.8781 |  0.87811\n",
            "Cardinal_(Catholi |     322 |    1.2 +  22.33 =  23.53 ms |    89.74 % |  78% ( 117) |  93% ( 205) |   0.8789 |  0.87888 |   0.8778 |  0.87779\n",
            "Human_Development |     168 |   1.26 +  20.75 =  22.02 ms |    89.33 % |  73% (  75) |  84% (  93) |   0.7976 |  0.79762 |   0.8429 |  0.84292\n",
            "Heresy            |     204 |   1.19 +  20.01 =  21.21 ms |    92.47 % |  86% (  93) |  91% ( 111) |   0.8922 |  0.89216 |   0.8891 |  0.88909\n",
            "Warsaw_Pact       |     146 |   1.05 +  28.21 =  29.26 ms |    96.15 % |  82% (  52) |  93% (  94) |   0.8973 |  0.89726 |   0.9026 |  0.90264\n",
            "Materialism       |     203 |   1.12 +  19.13 =  20.25 ms |    94.44 % |  87% (  54) |  96% ( 149) |   0.9409 |  0.94089 |    0.931 |  0.93105\n",
            "Pub               |     356 |   1.26 +  18.56 =  19.81 ms |    91.85 % |  86% ( 356) |   0% (   0) |    0.868 |  0.86798 |   0.8678 |  0.86783\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|----------|---------\n",
            "Grand Total       |    2850 |   1.14 +  21.73 =  22.87 ms |    93.09 % |  84% (1852) |  92% ( 998) |   0.8733 |  0.87362 |   0.8706 |  0.87438\n"
          ]
        }
      ],
      "source": [
        "# with mpnet and electra-best (optimized) k = 8"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HStg9dEy7_Hx",
        "outputId": "c49db1da-03d7-4a46-ea63-80ade849f8a4"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Theme             | Queries |  AIT: (ANN) + (QnA) = Total | ansInCtx % | TP % (TotP) | TN % (TotN) | Para Acc | Final PA | F1 Score | Final F1\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|----------|---------\n",
            "IPod              |     326 |    1.2 +  15.17 =  16.37 ms |    90.49 % |  85% ( 326) |   0% (   0) |   0.8528 |  0.85276 |   0.8311 |  0.83106\n",
            "2008_Sichuan_eart |     521 |   0.97 +  15.97 =  16.93 ms |    91.75 % |  86% ( 521) |   0% (   0) |   0.8656 |  0.86564 |    0.844 |  0.84402\n",
            "Wayback_Machine   |     208 |   1.17 +   15.9 =  17.07 ms |    86.08 % |  73% (  79) |  91% ( 129) |   0.8462 |  0.84615 |   0.8523 |  0.85232\n",
            "Canadian_Armed_Fo |     396 |   0.93 +  18.49 =  19.41 ms |     93.3 % |  89% ( 179) |  92% ( 217) |   0.9116 |  0.91162 |   0.8865 |  0.88646\n",
            "Cardinal_(Catholi |     322 |   1.18 +  16.75 =  17.93 ms |    86.32 % |  77% ( 117) |  94% ( 205) |   0.8851 |  0.88509 |   0.8814 |  0.88144\n",
            "Human_Development |     168 |   1.28 +  15.61 =  16.89 ms |    85.33 % |  73% (  75) |  81% (  93) |   0.7798 |  0.77976 |   0.8247 |  0.82473\n",
            "Heresy            |     204 |    1.2 +  15.59 =  16.79 ms |     91.4 % |  87% (  93) |  88% ( 111) |   0.8775 |  0.87745 |   0.8709 |  0.87089\n",
            "Warsaw_Pact       |     146 |   1.05 +  20.15 =   21.2 ms |    96.15 % |  80% (  52) |  92% (  94) |   0.8836 |  0.88356 |   0.9046 |  0.90457\n",
            "Materialism       |     203 |   1.12 +  17.81 =  18.93 ms |    94.44 % |  88% (  54) |  96% ( 149) |   0.9458 |  0.94581 |   0.9347 |  0.93474\n",
            "Pub               |     356 |   1.77 +  18.37 =  20.14 ms |    90.45 % |  86% ( 356) |   0% (   0) |    0.868 |  0.86798 |   0.8606 |  0.86063\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|----------|---------\n",
            "Grand Total       |    2850 |   1.18 +  16.91 =  18.08 ms |    90.77 % |  84% (1852) |  91% ( 998) |    0.874 |  0.87158 |   0.8657 |  0.86908\n"
          ]
        }
      ],
      "source": [
        "# with mpnet and electra-best (optimized) k = 6"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OKptJlDU7m6V",
        "outputId": "c7459785-01ce-4343-b0b5-422303282637"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Theme             | Queries |  AIT: (ANN) + (QnA) = Total | ansInCtx % | TP % (TotP) | TN % (TotN) | Para Acc | Final PA | F1 Score | Final F1\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|----------|---------\n",
            "IPod              |     326 |    1.1 +  13.75 =  14.85 ms |    88.34 % |  84% ( 326) |   0% (   0) |   0.8436 |  0.84356 |   0.8207 |  0.82069\n",
            "2008_Sichuan_eart |     521 |   1.15 +  14.81 =  15.96 ms |     90.4 % |  85% ( 521) |   0% (   0) |   0.8599 |  0.85988 |   0.8352 |  0.83515\n",
            "Wayback_Machine   |     208 |   1.17 +  14.52 =  15.69 ms |    86.08 % |  75% (  79) |  89% ( 129) |   0.8413 |  0.84135 |   0.8496 |  0.84963\n",
            "Canadian_Armed_Fo |     396 |   0.98 +  16.19 =  17.17 ms |    92.74 % |  88% ( 179) |  92% ( 217) |    0.904 |  0.90404 |   0.8818 |  0.88177\n",
            "Cardinal_(Catholi |     322 |   1.15 +  13.84 =  14.99 ms |    82.91 % |  76% ( 117) |  94% ( 205) |   0.8789 |  0.87888 |   0.8747 |  0.87469\n",
            "Human_Development |     168 |   1.26 +  13.16 =  14.42 ms |    82.67 % |  72% (  75) |  82% (  93) |   0.7798 |  0.77976 |   0.8265 |  0.82649\n",
            "Heresy            |     204 |   1.17 +  12.97 =  14.14 ms |    88.17 % |  87% (  93) |  87% ( 111) |   0.8725 |  0.87255 |   0.8597 |  0.85968\n",
            "Warsaw_Pact       |     146 |   1.08 +  16.91 =   18.0 ms |    94.23 % |  76% (  52) |  96% (  94) |   0.8973 |  0.89726 |    0.924 |  0.92397\n",
            "Materialism       |     203 |   1.13 +  13.01 =  14.15 ms |    94.44 % |  88% (  54) |  95% ( 149) |    0.936 |  0.93596 |   0.9249 |  0.92489\n",
            "Pub               |     356 |   1.28 +  12.66 =  13.94 ms |    88.76 % |  85% ( 356) |   0% (   0) |   0.8596 |  0.85955 |   0.8516 |  0.85163\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|----------|---------\n",
            "Grand Total       |    2850 |   1.14 +  14.23 =  15.37 ms |    89.09 % |  84% (1852) |  91% ( 998) |   0.8684 |  0.86728 |   0.8597 |  0.86486\n"
          ]
        }
      ],
      "source": [
        "# with mpnet and electra-best (optimized) k = 5"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s48sHYjMZPgg"
      },
      "source": [
        "#### Context Length Threshold\n",
        "170 | 190 | 200 | **205** | 210 | 220"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "sA0HdzxjZfIk",
        "outputId": "d7ae48de-ca2f-4ca5-ef83-e20c704c27c6"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Theme             | Queries |  AIT: (ANN) + (QnA) = Total | ansInCtx % | TP % (TotP) | TN % (TotN) | Para Acc | F1 Score | Avg Ctx Size\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|-------------\n",
            "IPod              |     326 |   1.25 +  20.06 =  21.32 ms |     95.4 % |  88% ( 326) |   0% (   0) |   0.8804 |   0.8638 |   181.4\n",
            "2008_Sichuan_eart |     521 |   1.11 +   21.0 =  22.12 ms |    93.28 % |  87% ( 521) |   0% (   0) |   0.8714 |     0.85 |   184.6\n",
            "Wayback_Machine   |     208 |   1.34 +   20.5 =  21.85 ms |    88.61 % |  72% (  79) |  92% ( 129) |   0.8462 |   0.8608 |   184.9\n",
            "Canadian_Armed_Fo |     396 |   1.05 +  20.28 =  21.33 ms |     93.3 % |  88% ( 179) |  93% ( 217) |   0.9091 |   0.8879 |   188.8\n",
            "Cardinal_(Catholi |     322 |   1.32 +  20.63 =  21.95 ms |    86.32 % |  78% ( 117) |  93% ( 205) |    0.882 |   0.8812 |   189.4\n",
            "Human_Development |     168 |    1.4 +  20.14 =  21.55 ms |    86.67 % |  73% (  75) |  84% (  93) |   0.7976 |   0.8373 |   182.2\n",
            "Heresy            |     204 |   1.38 +   20.8 =  22.17 ms |     91.4 % |  88% (  93) |  90% ( 111) |   0.8971 |   0.8856 |   184.2\n",
            "Warsaw_Pact       |     146 |   1.16 +  19.61 =  20.77 ms |    92.31 % |  75% (  52) |  95% (  94) |   0.8836 |   0.9114 |   194.2\n",
            "Materialism       |     203 |   1.18 +  21.84 =  23.02 ms |    94.44 % |  81% (  54) |  97% ( 149) |    0.936 |   0.9249 |   183.3\n",
            "Pub               |     356 |   1.37 +  18.17 =  19.54 ms |    91.85 % |  88% ( 356) |   0% (   0) |   0.8848 |   0.8622 |   183.3\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|--------------\n",
            "Grand Total       |    2850 |   1.24 +  20.28 =  21.52 ms |    92.39 % |  85% (1852) |  93% ( 998) |   0.8814 |    0.873 |   185.4\n"
          ]
        }
      ],
      "source": [
        "# with mpnet and electra-best (optimized) context_length_threshold = 170"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5nwLWwfvZfIl",
        "outputId": "85c94df8-fffa-4e67-a0a2-c559c7589df4"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Theme             | Queries |  AIT: (ANN) + (QnA) = Total | ansInCtx % | TP % (TotP) | TN % (TotN) | Para Acc | F1 Score | Avg Ctx Size\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|-------------\n",
            "IPod              |     326 |   1.15 +  19.83 =  20.98 ms |    95.71 % |  88% ( 326) |   0% (   0) |   0.8804 |   0.8584 |   197.0\n",
            "2008_Sichuan_eart |     521 |    1.0 +  21.42 =  22.42 ms |    93.67 % |  87% ( 521) |   0% (   0) |   0.8733 |   0.8545 |   204.1\n",
            "Wayback_Machine   |     208 |   1.28 +  20.88 =  22.16 ms |    88.61 % |  72% (  79) |  90% ( 129) |   0.8365 |   0.8443 |   203.9\n",
            "Canadian_Armed_Fo |     396 |   1.05 +   21.0 =  22.05 ms |    93.85 % |  87% ( 179) |  92% ( 217) |    0.904 |   0.8846 |   208.7\n",
            "Cardinal_(Catholi |     322 |   1.31 +  21.25 =  22.56 ms |    88.03 % |  79% ( 117) |  93% ( 205) |   0.8851 |   0.8847 |   208.6\n",
            "Human_Development |     168 |    1.4 +  20.69 =  22.09 ms |     88.0 % |  74% (  75) |  86% (  93) |   0.8095 |   0.8504 |   199.4\n",
            "Heresy            |     204 |    1.3 +  21.21 =  22.51 ms |     91.4 % |  88% (  93) |  91% ( 111) |    0.902 |   0.8891 |   201.8\n",
            "Warsaw_Pact       |     146 |   1.09 +  20.44 =  21.53 ms |    94.23 % |  76% (  52) |  94% (  94) |   0.8836 |   0.9114 |   210.8\n",
            "Materialism       |     203 |   1.17 +  21.89 =  23.07 ms |    94.44 % |  81% (  54) |  97% ( 149) |    0.936 |   0.9261 |   199.3\n",
            "Pub               |     356 |    1.3 +  20.03 =  21.32 ms |    92.13 % |  88% ( 356) |   0% (   0) |   0.8876 |   0.8646 |   201.9\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|--------------\n",
            "Grand Total       |    2850 |   1.18 +  20.87 =  22.05 ms |    92.87 % |  85% (1852) |  92% ( 998) |   0.8821 |   0.8733 |   203.7\n"
          ]
        }
      ],
      "source": [
        "# with mpnet and electra-best (optimized) context_length_threshold = 190"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HCS1idEwZfIl",
        "outputId": "539dad95-c25a-4d99-8df3-8b0dec592c71"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Theme             | Queries |  AIT: (ANN) + (QnA) = Total | ansInCtx % | TP % (TotP) | TN % (TotN) | Para Acc | F1 Score | Avg Ctx Size\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|-------------\n",
            "IPod              |     326 |   1.16 +  20.08 =  21.24 ms |    95.71 % |  88% ( 326) |   0% (   0) |   0.8834 |   0.8607 |   202.9\n",
            "2008_Sichuan_eart |     521 |    1.0 +  21.62 =  22.62 ms |    93.67 % |  87% ( 521) |   0% (   0) |   0.8733 |   0.8552 |   211.9\n",
            "Wayback_Machine   |     208 |   1.26 +  21.68 =  22.93 ms |    88.61 % |  72% (  79) |  91% ( 129) |   0.8413 |   0.8491 |   212.4\n",
            "Canadian_Armed_Fo |     396 |   1.03 +  21.72 =  22.75 ms |    93.85 % |  87% ( 179) |  92% ( 217) |    0.904 |   0.8843 |   217.2\n",
            "Cardinal_(Catholi |     322 |   1.33 +  21.94 =  23.27 ms |    88.03 % |  79% ( 117) |  93% ( 205) |   0.8851 |   0.8847 |   216.1\n",
            "Human_Development |     168 |   1.41 +  21.57 =  22.98 ms |     88.0 % |  74% (  75) |  86% (  93) |   0.8095 |   0.8504 |   208.3\n",
            "Heresy            |     204 |    1.3 +  21.88 =  23.18 ms |    92.47 % |  89% (  93) |  90% ( 111) |    0.902 |   0.8891 |   209.3\n",
            "Warsaw_Pact       |     146 |   1.12 +  21.44 =  22.55 ms |    96.15 % |  78% (  52) |  95% (  94) |   0.8973 |    0.924 |   223.1\n",
            "Materialism       |     203 |   1.19 +  22.96 =  24.15 ms |    94.44 % |  81% (  54) |  97% ( 149) |    0.936 |   0.9261 |   207.4\n",
            "Pub               |     356 |   1.31 +  20.38 =  21.69 ms |    92.42 % |  89% ( 356) |   0% (   0) |   0.8904 |    0.867 |   209.6\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|--------------\n",
            "Grand Total       |    2850 |   1.18 +  21.45 =  22.63 ms |    93.03 % |  85% (1852) |  92% ( 998) |   0.8839 |    0.875 |   211.7\n"
          ]
        }
      ],
      "source": [
        "# with mpnet and electra-best (optimized) context_length_threshold = 200"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "q5mDIXKzZfIl",
        "outputId": "3b465930-51dc-487c-c3db-315cf7be7a76"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Theme             | Queries |  AIT: (ANN) + (QnA) = Total | ansInCtx % | TP % (TotP) | TN % (TotN) | Para Acc | F1 Score | Avg Ctx Size\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|-------------\n",
            "IPod              |     326 |   1.23 +   22.2 =  23.43 ms |    95.71 % |  88% ( 326) |   0% (   0) |   0.8834 |   0.8607 |   205.4\n",
            "2008_Sichuan_eart |     521 |   1.05 +  22.56 =  23.61 ms |    93.67 % |  87% ( 521) |   0% (   0) |   0.8733 |   0.8538 |   216.9\n",
            "Wayback_Machine   |     208 |   1.34 +  22.44 =  23.78 ms |    89.87 % |  72% (  79) |  91% ( 129) |   0.8413 |   0.8491 |   216.5\n",
            "Canadian_Armed_Fo |     396 |   1.02 +  23.17 =   24.2 ms |    93.85 % |  87% ( 179) |  92% ( 217) |    0.904 |   0.8868 |   221.9\n",
            "Cardinal_(Catholi |     322 |   1.36 +  26.46 =  27.82 ms |    88.89 % |  80% ( 117) |  93% ( 205) |   0.8882 |   0.8878 |   220.6\n",
            "Human_Development |     168 |   1.48 +  23.63 =   25.1 ms |    89.33 % |  76% (  75) |  86% (  93) |   0.8155 |   0.8563 |   213.0\n",
            "Heresy            |     204 |   1.24 +  21.86 =  23.09 ms |    92.47 % |  89% (  93) |  90% ( 111) |    0.902 |   0.8905 |   214.2\n",
            "Warsaw_Pact       |     146 |    1.1 +  22.64 =  23.74 ms |    96.15 % |  78% (  52) |  96% (  94) |   0.9041 |   0.9308 |   226.6\n",
            "Materialism       |     203 |   1.29 +   24.8 =  26.09 ms |    94.44 % |  81% (  54) |  97% ( 149) |    0.936 |   0.9261 |   210.6\n",
            "Pub               |     356 |    1.3 +  20.67 =  21.97 ms |    92.42 % |  88% ( 356) |   0% (   0) |   0.8876 |    0.867 |   213.2\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|--------------\n",
            "Grand Total       |    2850 |   1.21 +  22.98 =  24.19 ms |     93.2 % |  85% (1852) |  93% ( 998) |   0.8846 |   0.8762 |   215.8\n"
          ]
        }
      ],
      "source": [
        "# with mpnet and electra-best (optimized) context_length_threshold = 205"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "R_0Dk3E1ZfIl",
        "outputId": "b3b74d2e-629a-4eb7-9a2b-21fa93421dc7"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Theme             | Queries |  AIT: (ANN) + (QnA) = Total | ansInCtx % | TP % (TotP) | TN % (TotN) | Para Acc | F1 Score | Avg Ctx Size\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|-------------\n",
            "IPod              |     326 |   1.18 +  20.43 =  21.62 ms |    95.71 % |  88% ( 326) |   0% (   0) |   0.8834 |   0.8607 |   207.1\n",
            "2008_Sichuan_eart |     521 |   1.01 +  22.84 =  23.85 ms |    93.86 % |  86% ( 521) |   0% (   0) |   0.8695 |   0.8507 |   221.2\n",
            "Wayback_Machine   |     208 |   1.27 +   22.6 =  23.87 ms |    89.87 % |  69% (  79) |  91% ( 129) |   0.8317 |   0.8395 |   221.5\n",
            "Canadian_Armed_Fo |     396 |   1.04 +  22.83 =  23.87 ms |    93.85 % |  88% ( 179) |  92% ( 217) |   0.9066 |   0.8874 |   227.1\n",
            "Cardinal_(Catholi |     322 |   1.28 +  22.64 =  23.92 ms |    88.89 % |  80% ( 117) |  93% ( 205) |   0.8882 |   0.8878 |   224.8\n",
            "Human_Development |     168 |   1.41 +  21.98 =  23.39 ms |    89.33 % |  76% (  75) |  86% (  93) |   0.8155 |   0.8563 |   215.1\n",
            "Heresy            |     204 |   1.24 +   22.5 =  23.75 ms |    93.55 % |  89% (  93) |  90% ( 111) |    0.902 |   0.8954 |   218.0\n",
            "Warsaw_Pact       |     146 |   1.11 +  21.83 =  22.94 ms |    96.15 % |  78% (  52) |  96% (  94) |   0.9041 |   0.9308 |   230.6\n",
            "Materialism       |     203 |   1.16 +  23.67 =  24.83 ms |    94.44 % |  81% (  54) |  97% ( 149) |    0.936 |   0.9261 |   213.1\n",
            "Pub               |     356 |   1.31 +   20.8 =   22.1 ms |    92.98 % |  88% ( 356) |   0% (   0) |   0.8876 |    0.867 |   216.3\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|--------------\n",
            "Grand Total       |    2850 |   1.18 +   22.2 =  23.38 ms |    93.41 % |  85% (1852) |  93% ( 998) |   0.8835 |   0.8754 |   219.5\n"
          ]
        }
      ],
      "source": [
        "# with mpnet and electra-best (optimized) context_length_threshold = 210"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SqPkY7oAZfIl",
        "outputId": "ef00dc28-f132-4a3d-f75e-7e25f66aad2a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Theme             | Queries |  AIT: (ANN) + (QnA) = Total | ansInCtx % | TP % (TotP) | TN % (TotN) | Para Acc | F1 Score | Avg Ctx Size\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|-------------\n",
            "IPod              |     326 |   1.23 +  24.66 =  25.88 ms |    95.71 % |  87% ( 326) |   0% (   0) |   0.8773 |   0.8561 |   211.3\n",
            "2008_Sichuan_eart |     521 |   1.54 +  28.74 =  30.28 ms |    94.24 % |  87% ( 521) |   0% (   0) |   0.8714 |   0.8539 |   228.1\n",
            "Wayback_Machine   |     208 |   1.36 +  24.82 =  26.18 ms |    89.87 % |  70% (  79) |  90% ( 129) |   0.8317 |   0.8384 |   230.9\n",
            "Canadian_Armed_Fo |     396 |   1.05 +  23.57 =  24.62 ms |    93.85 % |  88% ( 179) |  92% ( 217) |    0.904 |   0.8852 |   238.3\n",
            "Cardinal_(Catholi |     322 |   1.24 +  22.86 =   24.1 ms |    89.74 % |  80% ( 117) |  93% ( 205) |   0.8882 |   0.8847 |   231.2\n",
            "Human_Development |     168 |    1.4 +  22.39 =   23.8 ms |    89.33 % |  76% (  75) |  86% (  93) |   0.8155 |   0.8563 |   222.4\n",
            "Heresy            |     204 |   1.24 +  23.07 =  24.31 ms |    93.55 % |  89% (  93) |  90% ( 111) |    0.902 |   0.8954 |   224.6\n",
            "Warsaw_Pact       |     146 |   1.11 +  22.51 =  23.62 ms |    96.15 % |  78% (  52) |  95% (  94) |   0.8973 |   0.9251 |   240.7\n",
            "Materialism       |     203 |   1.21 +  24.28 =  25.49 ms |    94.44 % |  81% (  54) |  97% ( 149) |    0.936 |   0.9261 |   216.8\n",
            "Pub               |     356 |   1.32 +  21.31 =  22.63 ms |    93.26 % |  89% ( 356) |   0% (   0) |   0.8904 |   0.8698 |   222.7\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|--------------\n",
            "Grand Total       |    2850 |   1.29 +  24.26 =  25.55 ms |    93.63 % |  85% (1852) |  92% ( 998) |   0.8828 |   0.8747 |   226.7\n"
          ]
        }
      ],
      "source": [
        "# with mpnet and electra-best (optimized) context_length_threshold = 220"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vXLsYXshHhdR"
      },
      "source": [
        "### Current Best Performance"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1403da33-fb00-4906-d151-53e5062ba523",
        "id": "1rFoprCbyMcb"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Theme             | Queries |  AIT: (ANN) + (QnA) = Total | ansInCtx % | TP % (TotP) | TN % (TotN) | Para Acc | F1 Score | Avg Ctx Size\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|-------------\n",
            "IPod              |     326 |   1.15 +  21.24 =  22.39 ms |    96.93 % |  88% ( 326) |   0% (   0) |   0.8804 |   0.8562 |   210.0\n",
            "2008_Sichuan_eart |     521 |   1.01 +  22.77 =  23.78 ms |    95.39 % |  89% ( 521) |   0% (   0) |   0.8944 |   0.8743 |   217.8\n",
            "Wayback_Machine   |     208 |   1.29 +  21.89 =  23.18 ms |    92.41 % |  74% (  79) |  92% ( 129) |   0.8558 |   0.8528 |   217.7\n",
            "Canadian_Armed_Fo |     396 |   1.02 +  22.63 =  23.64 ms |    95.53 % |  89% ( 179) |  90% ( 217) |   0.9015 |   0.8809 |   224.0\n",
            "Cardinal_(Catholi |     322 |   1.23 +  21.77 =   23.0 ms |     90.6 % |  82% ( 117) |  94% ( 205) |   0.8975 |   0.8956 |   224.0\n",
            "Human_Development |     168 |   1.35 +  22.26 =  23.62 ms |    89.33 % |  74% (  75) |  88% (  93) |   0.8214 |   0.8605 |   213.4\n",
            "Heresy            |     204 |    1.2 +  21.57 =  22.77 ms |    96.77 % |  91% (  93) |  86% ( 111) |   0.8873 |   0.8842 |   213.9\n",
            "Warsaw_Pact       |     146 |   1.07 +  20.84 =  21.91 ms |    94.23 % |  80% (  52) |  95% (  94) |   0.9041 |   0.9267 |   225.6\n",
            "Materialism       |     203 |   1.15 +   24.5 =  25.65 ms |     96.3 % |  87% (  54) |  97% ( 149) |   0.9507 |   0.9327 |   213.5\n",
            "Pub               |     356 |   1.28 +  21.34 =  22.63 ms |    95.51 % |  89% ( 356) |   0% (   0) |   0.8961 |   0.8839 |   218.3\n",
            "Web_browser       |     192 |   1.01 +   22.1 =   23.1 ms |    100.0 % |  95% (  72) |  92% ( 120) |   0.9375 |   0.9436 |   214.5\n",
            "Catalan_language  |     340 |   0.96 +  23.92 =  24.88 ms |    92.94 % |  78% ( 340) |   0% (   0) |   0.7882 |   0.8241 |   210.0\n",
            "Paper             |     371 |   1.13 +  20.77 =   21.9 ms |    94.44 % |  87% ( 108) |  96% ( 263) |   0.9407 |   0.9354 |   206.9\n",
            "Adult_contemporar |     216 |   1.52 +   22.1 =  23.62 ms |    93.52 % |  83% ( 216) |   0% (   0) |   0.8333 |   0.8461 |   222.7\n",
            "Nanjing           |     312 |    1.1 +  21.98 =  23.09 ms |    97.44 % |  93% ( 312) |   0% (   0) |   0.9327 |    0.921 |   217.6\n",
            "Dialect           |     373 |   1.28 +  22.05 =  23.33 ms |    95.15 % |  90% ( 165) |  94% ( 208) |   0.9276 |   0.9158 |   220.9\n",
            "Southampton       |     426 |   1.29 +  20.26 =  21.55 ms |    97.89 % |  94% ( 426) |   0% (   0) |   0.9437 |   0.9529 |   212.6\n",
            "The_Times         |     210 |   1.49 +  26.53 =  28.02 ms |    96.19 % |  91% ( 210) |   0% (   0) |   0.9143 |   0.9295 |   220.2\n",
            "Immunology        |      85 |   1.21 +  21.79 =   23.0 ms |    94.12 % |  91% (  85) |   0% (   0) |   0.9176 |   0.8223 |   209.3\n",
            "Imamah_(Shia_doct |      64 |   1.26 +  24.12 =  25.38 ms |    95.31 % |  95% (  64) |   0% (   0) |   0.9531 |     0.93 |   221.0\n",
            "Grape             |      48 |   1.55 +  22.34 =   23.9 ms |    95.83 % |  89% (  48) |   0% (   0) |   0.8958 |   0.8507 |   217.1\n",
            "United_States_dol |     319 |   1.14 +  22.32 =  23.45 ms |    97.48 % |  93% ( 159) |  93% ( 160) |    0.931 |   0.9229 |   217.9\n",
            "Everton_F.C.      |     230 |   1.23 +  20.14 =  21.37 ms |    97.35 % |  88% ( 113) |  88% ( 117) |   0.8826 |   0.8672 |   211.3\n",
            "Hard_rock         |     249 |   1.13 +  23.04 =  24.17 ms |    95.97 % |  91% ( 124) |  88% ( 125) |   0.8956 |   0.8879 |   222.6\n",
            "Great_Plains      |     108 |   1.32 +  20.53 =  21.85 ms |    97.87 % |  97% (  47) |  91% (  61) |   0.9444 |   0.9383 |   216.3\n",
            "Biodiversity      |     302 |   1.08 +  19.65 =  20.73 ms |    93.71 % |  90% ( 143) |  93% ( 159) |   0.9205 |   0.9082 |   199.4\n",
            "Federal_Bureau_of |     441 |   1.07 +  21.59 =  22.67 ms |    94.57 % |  92% ( 221) |  93% ( 220) |   0.9297 |   0.9094 |   219.8\n",
            "Mary_(mother_of_J |     366 |   1.19 +   22.6 =  23.79 ms |    97.19 % |  90% ( 178) |  90% ( 188) |   0.9071 |   0.9031 |   221.4\n",
            "Anti-aircraft_war |     340 |   1.13 +  21.22 =  22.35 ms |    92.94 % |  92% ( 340) |   0% (   0) |   0.9265 |   0.8965 |   214.9\n",
            "Sanskrit          |     275 |   1.07 +  22.15 =  23.22 ms |     91.3 % |  91% ( 138) |  97% ( 137) |   0.9455 |   0.9092 |   205.7\n",
            "Order_of_the_Brit |      63 |   1.23 +  20.42 =  21.66 ms |    95.24 % |  92% (  63) |   0% (   0) |   0.9206 |     0.82 |   222.7\n",
            "Elizabeth_II      |     431 |   1.11 +   21.2 =  22.31 ms |    95.05 % |  88% ( 222) |  80% ( 209) |   0.8445 |   0.8387 |   218.3\n",
            "Capital_punishmen |     422 |   1.16 +  20.69 =  21.85 ms |    96.21 % |  88% ( 211) |  92% ( 211) |   0.9076 |    0.915 |   218.7\n",
            "Age_of_Enlightenm |     347 |   1.58 +  21.86 =  23.44 ms |    95.68 % |  93% ( 347) |   0% (   0) |   0.9395 |   0.9075 |   217.5\n",
            "Athanasius_of_Ale |     377 |   0.99 +   23.1 =  24.09 ms |    84.77 % |  74% ( 197) |  85% ( 180) |   0.7984 |   0.7749 |   214.2\n",
            "Freemasonry       |     444 |   1.17 +  21.45 =  22.62 ms |    91.14 % |  87% ( 271) |  97% ( 173) |   0.9122 |   0.8935 |   219.0\n",
            "Montevideo        |     247 |   1.16 +  21.64 =   22.8 ms |    96.36 % |  90% ( 247) |   0% (   0) |   0.9028 |   0.8964 |   215.4\n",
            "Poultry           |     310 |   1.14 +  21.56 =   22.7 ms |    85.71 % |  69% ( 140) |  94% ( 170) |   0.8323 |   0.7386 |   216.0\n",
            "Clothing          |     203 |   1.13 +  19.73 =  20.86 ms |    97.14 % |  95% ( 105) |  96% (  98) |   0.9606 |   0.9448 |   190.9\n",
            "Department_store  |     378 |   1.11 +  20.94 =  22.05 ms |    90.45 % |  82% ( 178) |  94% ( 200) |   0.8862 |   0.8787 |   210.3\n",
            "Marvel_Comics     |     247 |    1.1 +  22.71 =  23.81 ms |    92.68 % |  80% ( 123) |  85% ( 124) |     0.83 |   0.8347 |   223.8\n",
            "Alloy             |     198 |   1.01 +  19.41 =  20.42 ms |    95.83 % |  90% (  96) |  94% ( 102) |   0.9242 |   0.9241 |   200.5\n",
            "Electric_motor    |     363 |    1.1 +   21.6 =   22.7 ms |    91.26 % |  85% ( 183) |  94% ( 180) |   0.8981 |   0.8631 |   216.3\n",
            "Nutrition         |     275 |   1.44 +  21.85 =  23.29 ms |    94.91 % |  92% ( 275) |   0% (   0) |   0.9273 |   0.8953 |   212.5\n",
            "Chinese_character |     168 |   1.02 +  23.81 =  24.82 ms |    91.67 % |  87% ( 168) |   0% (   0) |    0.875 |   0.8268 |   217.7\n",
            "Bermuda           |     434 |   1.04 +   20.2 =  21.24 ms |    92.98 % |  90% ( 228) |  91% ( 206) |   0.9101 |    0.873 |   209.0\n",
            "Utrecht           |     210 |   0.95 +  19.83 =  20.78 ms |    94.23 % |  90% ( 104) |  91% ( 106) |   0.9095 |   0.7821 |   198.6\n",
            "Capacitor         |     364 |   1.48 +  22.14 =  23.62 ms |    90.27 % |  85% ( 185) |  97% ( 179) |   0.9148 |   0.8858 |   203.8\n",
            "Comcast           |     266 |   1.14 +   21.1 =  22.25 ms |    96.95 % |  80% ( 131) |  94% ( 135) |   0.8759 |    0.873 |   209.7\n",
            "Tuberculosis      |     298 |   1.24 +  19.91 =  21.14 ms |    94.04 % |  86% ( 151) |  91% ( 147) |   0.8893 |   0.8889 |   190.6\n",
            "North_Carolina    |     297 |   1.13 +  21.36 =   22.5 ms |    97.64 % |  91% ( 297) |   0% (   0) |   0.9192 |    0.918 |   213.3\n",
            "Heian_period      |     205 |   1.14 +  20.83 =  21.97 ms |    96.88 % |  87% (  96) |  88% ( 109) |   0.8829 |   0.8917 |   212.3\n",
            "On_the_Origin_of_ |     261 |   1.31 +  21.89 =   23.2 ms |    88.51 % |  85% ( 261) |   0% (   0) |   0.8544 |   0.7753 |   221.3\n",
            "Political_party   |     245 |   1.07 +  20.47 =  21.54 ms |    95.45 % |  90% ( 110) |  94% ( 135) |   0.9265 |    0.898 |   219.4\n",
            "Tibet             |     322 |   1.12 +  22.39 =  23.51 ms |    96.18 % |  93% ( 157) |  91% ( 165) |   0.9255 |   0.9223 |   211.2\n",
            "Oklahoma          |     271 |   0.96 +  22.27 =  23.23 ms |    98.52 % |  94% ( 271) |   0% (   0) |   0.9446 |   0.9311 |   218.1\n",
            "European_Central_ |     253 |   1.21 +  21.57 =  22.78 ms |    92.91 % |  88% ( 127) |  97% ( 126) |   0.9289 |   0.8845 |   219.6\n",
            "Group_(mathematic |     259 |   1.16 +  20.47 =  21.63 ms |     95.5 % |  92% ( 111) |  89% ( 148) |   0.9112 |   0.8883 |   213.9\n",
            "God               |     293 |   1.19 +  23.02 =  24.21 ms |    93.06 % |  86% ( 173) |  85% ( 120) |   0.8635 |   0.7992 |   213.1\n",
            "Karl_Popper       |     391 |   1.24 +  22.14 =  23.38 ms |    77.78 % |  72% ( 180) |  92% ( 211) |   0.8338 |   0.8358 |   222.1\n",
            "Planck_constant   |     265 |   1.29 +  22.55 =  23.85 ms |    94.23 % |  91% ( 156) |  99% ( 109) |   0.9434 |   0.8943 |   221.8\n",
            "Zinc              |     344 |   1.11 +  21.49 =   22.6 ms |    96.22 % |  89% ( 185) |  93% ( 159) |   0.9128 |   0.8913 |   195.6\n",
            "Appalachian_Mount |     228 |   1.14 +  25.92 =  27.06 ms |    93.33 % |  86% (  60) |  91% ( 168) |   0.9035 |   0.8883 |   221.5\n",
            "Post-punk         |     352 |   1.12 +  23.09 =  24.22 ms |    88.79 % |  82% ( 214) |  94% ( 138) |    0.875 |   0.8568 |   222.9\n",
            "Richard_Feynman   |     304 |   1.21 +  23.22 =  24.43 ms |    90.07 % |  84% ( 151) |  96% ( 153) |   0.9046 |   0.8981 |   220.6\n",
            "Muammar_Gaddafi   |     485 |   1.18 +  22.17 =  23.35 ms |    88.45 % |  80% ( 485) |   0% (   0) |   0.8062 |   0.7751 |   212.3\n",
            "Neptune           |     399 |   1.05 +  23.02 =  24.07 ms |    91.38 % |  85% ( 174) |  93% ( 225) |   0.9023 |   0.8926 |   214.8\n",
            "Endangered_Specie |     274 |    1.3 +  22.62 =  23.92 ms |    90.91 % |  85% ( 121) |  88% ( 153) |   0.8759 |   0.8294 |   221.7\n",
            "Han_dynasty       |     323 |   1.16 +  22.05 =  23.21 ms |    96.28 % |  91% ( 323) |   0% (   0) |   0.9195 |   0.8983 |   215.5\n",
            "Transistor        |     216 |   1.15 +  23.24 =  24.39 ms |    96.33 % |  88% ( 109) |  75% ( 107) |   0.8241 |   0.7956 |   214.8\n",
            "51st_state        |     293 |   1.15 +  21.27 =  22.42 ms |    95.52 % |  89% ( 134) |  90% ( 159) |    0.901 |    0.875 |   219.6\n",
            "Antenna_(radio)   |     302 |   1.15 +  20.35 =  21.49 ms |    85.43 % |  74% ( 302) |   0% (   0) |    0.745 |   0.7188 |   217.1\n",
            "Washington_Univer |     341 |   1.23 +  20.43 =  21.66 ms |    96.47 % |  92% ( 170) |  88% ( 171) |   0.9062 |   0.8723 |   209.6\n",
            "Central_Intellige |      65 |   1.31 +  21.87 =  23.17 ms |    96.92 % |  96% (  65) |   0% (   0) |   0.9692 |   0.9794 |   221.5\n",
            "Armenia           |     241 |   1.06 +  21.11 =  22.17 ms |    95.85 % |  91% ( 241) |   0% (   0) |    0.917 |    0.882 |   213.3\n",
            "Muslim_world      |     241 |    1.2 +   25.2 =   26.4 ms |    98.44 % |  96% ( 128) |  92% ( 113) |   0.9419 |   0.9188 |   217.8\n",
            "Kievan_Rus%27     |     317 |   1.16 +  22.95 =  24.11 ms |    96.95 % |  91% ( 131) |  92% ( 186) |   0.9211 |   0.9002 |   219.0\n",
            "Immaculate_Concep |     197 |   1.39 +  22.47 =  23.87 ms |    67.31 % |  54% ( 104) |  90% (  93) |   0.7157 |   0.6523 |   228.7\n",
            "Egypt             |     247 |   1.08 +  20.73 =  21.81 ms |    98.38 % |  96% ( 247) |   0% (   0) |   0.9676 |   0.9259 |   211.3\n",
            "Protestantism     |     369 |   1.03 +  20.62 =  21.65 ms |    94.31 % |  90% ( 369) |   0% (   0) |   0.9024 |   0.8672 |   211.0\n",
            "Richmond,_Virgini |     291 |   1.08 +  21.77 =  22.85 ms |    98.97 % |  93% ( 291) |   0% (   0) |   0.9347 |   0.9553 |   213.4\n",
            "Normans           |     208 |   0.98 +  21.17 =  22.15 ms |    93.75 % |  82% (  96) |  68% ( 112) |     0.75 |   0.7121 |   215.0\n",
            "Computational_com |     418 |   1.51 +  20.47 =  21.98 ms |    96.45 % |  79% ( 197) |  86% ( 221) |   0.8349 |   0.8114 |   210.1\n",
            "Southern_Californ |     324 |   1.15 +  20.73 =  21.88 ms |    97.22 % |  85% ( 180) |  54% ( 144) |   0.7191 |   0.6862 |   214.2\n",
            "Sky_(United_Kingd |     210 |   1.32 +  22.98 =   24.3 ms |    100.0 % |  88% ( 108) |  93% ( 102) |   0.9095 |   0.8906 |   216.6\n",
            "Victoria_(Austral |     247 |   0.96 +  19.35 =  20.31 ms |    100.0 % |  94% ( 124) |  62% ( 123) |   0.7854 |   0.7297 |   197.7\n",
            "Huguenot          |     424 |   1.14 +  22.34 =  23.47 ms |    87.56 % |  74% ( 217) |  68% ( 207) |   0.7146 |   0.6975 |   215.1\n",
            "Steam_engine      |     444 |   1.24 +  20.78 =  22.02 ms |    98.14 % |  91% ( 215) |  53% ( 229) |   0.7185 |   0.7057 |   213.9\n",
            "Oxygen            |     415 |    1.2 +  21.86 =  23.06 ms |    97.93 % |  80% ( 241) |  64% ( 174) |   0.7422 |   0.6985 |   212.5\n",
            "1973_oil_crisis   |     255 |   1.19 +  21.83 =  23.02 ms |    100.0 % |  89% ( 106) |  60% ( 149) |   0.7255 |   0.6928 |   214.0\n",
            "European_Union_la |     421 |   1.29 +  20.82 =   22.1 ms |     97.4 % |  93% ( 231) |  95% ( 190) |    0.943 |   0.9159 |   221.7\n",
            "Amazon_rainforest |     284 |   1.24 +  21.48 =  22.72 ms |    96.17 % |  91% ( 183) |  81% ( 101) |   0.8768 |   0.8393 |   217.8\n",
            "Ctenophora        |     325 |   1.21 +  25.68 =  26.89 ms |    92.35 % |  79% ( 170) |  64% ( 155) |   0.7231 |   0.7154 |   220.7\n",
            "Fresno,_Californi |     275 |   1.01 +  22.59 =   23.6 ms |    100.0 % |  94% ( 136) |  58% ( 139) |   0.7636 |   0.7469 |   216.1\n",
            "Packet_switching  |     373 |   0.96 +  21.89 =  22.85 ms |     87.5 % |  78% ( 104) |  54% ( 269) |   0.6113 |   0.5655 |   211.2\n",
            "Black_Death       |     219 |    1.1 +  21.33 =  22.43 ms |    92.59 % |  84% ( 108) |  74% ( 111) |   0.7945 |   0.7614 |   217.6\n",
            "Geology           |     362 |   1.16 +  21.46 =  22.63 ms |    98.28 % |  91% ( 116) |  74% ( 246) |   0.7983 |   0.7779 |   213.2\n",
            "Pharmacy          |     244 |   1.21 +  22.91 =  24.11 ms |    92.97 % |  85% ( 128) |  94% ( 116) |   0.8975 |   0.8437 |   217.2\n",
            "Civil_disobedienc |     327 |   1.21 +  23.52 =  24.72 ms |    79.19 % |  58% ( 197) |  73% ( 130) |   0.6453 |   0.5989 |   222.7\n",
            "Construction      |     232 |   1.18 +  20.25 =  21.43 ms |    97.96 % |  97% (  98) |  67% ( 134) |    0.806 |   0.7891 |   208.6\n",
            "Private_school    |     240 |   1.15 +  22.32 =  23.47 ms |    97.35 % |  84% ( 113) |  77% ( 127) |   0.8042 |   0.7745 |   216.9\n",
            "Harvard_Universit |     262 |   1.19 +  22.71 =   23.9 ms |    97.32 % |  84% ( 112) |  81% ( 150) |   0.8282 |    0.827 |   227.4\n",
            "Jacksonville,_Flo |     192 |   1.07 +  20.56 =  21.63 ms |    97.92 % |  89% (  96) |  69% (  96) |   0.7969 |   0.7814 |   212.2\n",
            "Economic_inequali |     515 |   1.25 +   20.6 =  21.85 ms |     93.6 % |  85% ( 297) |  86% ( 218) |   0.8602 |   0.8169 |   218.6\n",
            "University_of_Chi |     356 |   1.18 +  24.86 =  26.04 ms |    97.16 % |  89% ( 176) |  71% ( 180) |   0.8006 |   0.8016 |   231.0\n",
            "Yuan_dynasty      |     445 |    1.0 +  22.19 =  23.19 ms |    93.42 % |  82% ( 228) |  72% ( 217) |    0.773 |    0.751 |   207.7\n",
            "Immune_system     |     458 |   1.19 +   21.7 =  22.89 ms |    97.21 % |  91% ( 215) |  89% ( 243) |   0.9017 |   0.8843 |   203.1\n",
            "Intergovernmental |     222 |   1.09 +  21.09 =  22.18 ms |    97.12 % |  84% ( 104) |  72% ( 118) |   0.7838 |   0.7456 |   222.9\n",
            "Prime_number      |     308 |   1.52 +  21.64 =  23.17 ms |    98.69 % |  87% ( 153) |  69% ( 155) |   0.7825 |   0.7711 |   214.1\n",
            "Rhine             |     498 |   1.13 +  23.89 =  25.02 ms |    92.41 % |  77% ( 290) |  79% ( 208) |   0.7851 |   0.7563 |   217.6\n",
            "Scottish_Parliame |     357 |   1.19 +  20.27 =  21.46 ms |    97.88 % |  87% ( 189) |  94% ( 168) |   0.9048 |   0.8731 |   217.9\n",
            "Islamism          |     384 |   1.26 +  23.13 =  24.39 ms |    97.46 % |  92% ( 197) |  89% ( 187) |   0.9089 |   0.8748 |   221.8\n",
            "Imperialism       |     364 |   1.09 +  20.45 =  21.54 ms |    95.19 % |  83% ( 187) |  83% ( 177) |   0.8352 |   0.8028 |   213.8\n",
            "Warsaw            |     486 |   1.16 +  21.08 =  22.24 ms |    97.93 % |  92% ( 242) |  54% ( 244) |   0.7346 |   0.7217 |   207.1\n",
            "French_and_Indian |     402 |   1.04 +  21.14 =  22.18 ms |    91.28 % |  84% ( 172) |  79% ( 230) |   0.8184 |     0.78 |   218.6\n",
            "Force             |     377 |   1.15 +  20.57 =  21.71 ms |    96.04 % |  79% ( 202) |  89% ( 175) |   0.8408 |   0.8094 |   214.2\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|--------------\n",
            "Grand Total       |   34926 |   1.17 +  21.81 =  22.98 ms |    94.26 % |  87% (20857) |  84% (14069) |   0.8622 |   0.8403 |   215.0\n"
          ]
        }
      ],
      "source": [
        "# with mpnet - finetuned and electra-best (optimized) - k = 10, context_length_threshold = 205 - new themes"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KQ2pN8vCdGYG",
        "outputId": "6317e27b-8d69-415b-a24d-7ac1c960813c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Theme             | Queries |  AIT: (ANN) + (QnA) = Total | ansInCtx % | TP % (TotP) | TN % (TotN) | Para Acc | F1 Score | Avg Ctx Size\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|-------------\n",
            "IPod              |     326 |   1.67 +  23.22 =  24.89 ms |    95.71 % |  88% ( 326) |   0% (   0) |   0.8834 |   0.8607 |   205.4\n",
            "2008_Sichuan_eart |     521 |   1.14 +  24.29 =  25.43 ms |    93.67 % |  87% ( 521) |   0% (   0) |   0.8733 |   0.8538 |   216.9\n",
            "Wayback_Machine   |     208 |   1.31 +  23.79 =  25.11 ms |    89.87 % |  72% (  79) |  91% ( 129) |   0.8413 |   0.8491 |   216.5\n",
            "Canadian_Armed_Fo |     396 |   1.06 +  24.08 =  25.15 ms |    93.85 % |  87% ( 179) |  92% ( 217) |    0.904 |   0.8868 |   221.9\n",
            "Cardinal_(Catholi |     322 |   1.32 +  23.71 =  25.03 ms |    88.89 % |  80% ( 117) |  93% ( 205) |   0.8882 |   0.8878 |   220.6\n",
            "Human_Development |     168 |   1.35 +  21.69 =  23.04 ms |    89.33 % |  76% (  75) |  86% (  93) |   0.8155 |   0.8563 |   213.0\n",
            "Heresy            |     204 |   1.23 +  21.98 =  23.21 ms |    92.47 % |  89% (  93) |  90% ( 111) |    0.902 |   0.8905 |   214.2\n",
            "Warsaw_Pact       |     146 |   1.18 +  23.53 =  24.72 ms |    96.15 % |  78% (  52) |  96% (  94) |   0.9041 |   0.9308 |   226.6\n",
            "Materialism       |     203 |   1.22 +  25.59 =  26.81 ms |    94.44 % |  81% (  54) |  97% ( 149) |    0.936 |   0.9261 |   210.6\n",
            "Pub               |     356 |   1.31 +   21.7 =  23.01 ms |    92.42 % |  88% ( 356) |   0% (   0) |   0.8876 |    0.867 |   213.2\n",
            "Web_browser       |     192 |   1.07 +  23.78 =  24.85 ms |    98.61 % |  91% (  72) |  93% ( 120) |   0.9271 |   0.9332 |   213.7\n",
            "Catalan_language  |     340 |   1.03 +  23.85 =  24.88 ms |    93.24 % |  77% ( 340) |   0% (   0) |   0.7765 |   0.8147 |   203.2\n",
            "Paper             |     371 |   1.12 +  21.02 =  22.13 ms |    93.52 % |  89% ( 108) |  96% ( 263) |   0.9488 |   0.9419 |   199.7\n",
            "Adult_contemporar |     216 |   1.61 +  24.23 =  25.84 ms |    93.06 % |  83% ( 216) |   0% (   0) |   0.8333 |   0.8585 |   220.2\n",
            "Nanjing           |     312 |   1.46 +  27.93 =  29.39 ms |    92.95 % |  90% ( 312) |   0% (   0) |   0.9071 |   0.8757 |   216.8\n",
            "Dialect           |     373 |    1.9 +  23.67 =  25.57 ms |    95.15 % |  90% ( 165) |  93% ( 208) |   0.9223 |   0.9137 |   217.6\n",
            "Southampton       |     426 |   1.32 +  20.28 =  21.59 ms |    92.96 % |  88% ( 426) |   0% (   0) |   0.8873 |   0.8985 |   207.3\n",
            "The_Times         |     210 |    1.5 +  23.02 =  24.53 ms |    93.33 % |  89% ( 210) |   0% (   0) |   0.8905 |   0.8831 |   219.9\n",
            "Immunology        |      85 |   1.16 +  22.38 =  23.53 ms |    92.94 % |  87% (  85) |   0% (   0) |   0.8706 |   0.7865 |   209.2\n",
            "Imamah_(Shia_doct |      64 |   1.39 +  24.76 =  26.15 ms |    93.75 % |  92% (  64) |   0% (   0) |   0.9219 |   0.9137 |   220.7\n",
            "Grape             |      48 |   1.63 +  22.48 =   24.1 ms |    97.92 % |  91% (  48) |   0% (   0) |   0.9167 |   0.8681 |   212.4\n",
            "United_States_dol |     319 |   1.17 +  22.02 =  23.19 ms |    94.97 % |  91% ( 159) |  90% ( 160) |   0.9091 |   0.9052 |   217.7\n",
            "Everton_F.C.      |     230 |   1.27 +  20.16 =  21.43 ms |    90.27 % |  82% ( 113) |  88% ( 117) |   0.8522 |    0.836 |   205.2\n",
            "Hard_rock         |     249 |   1.22 +  22.92 =  24.14 ms |    96.77 % |  91% ( 124) |  86% ( 125) |   0.8916 |   0.8812 |   224.0\n",
            "Great_Plains      |     108 |   1.34 +  20.76 =   22.1 ms |    97.87 % |  97% (  47) |  90% (  61) |   0.9352 |   0.9265 |   216.1\n",
            "Biodiversity      |     302 |   1.14 +  19.79 =  20.93 ms |    91.61 % |  91% ( 143) |  93% ( 159) |   0.9238 |   0.8986 |   195.1\n",
            "Federal_Bureau_of |     441 |   1.12 +   21.4 =  22.52 ms |    92.76 % |  91% ( 221) |  94% ( 220) |   0.9274 |    0.912 |   220.6\n",
            "Mary_(mother_of_J |     366 |   1.22 +  22.33 =  23.55 ms |    95.51 % |  88% ( 178) |  90% ( 188) |   0.8989 |   0.8938 |   220.6\n",
            "Anti-aircraft_war |     340 |   1.18 +  21.55 =  22.73 ms |    92.65 % |  91% ( 340) |   0% (   0) |   0.9176 |   0.8966 |   212.3\n",
            "Sanskrit          |     275 |    1.1 +  21.96 =  23.06 ms |     91.3 % |  89% ( 138) |  97% ( 137) |   0.9309 |   0.9018 |   206.3\n",
            "Order_of_the_Brit |      63 |   1.29 +  24.63 =  25.92 ms |    95.24 % |  90% (  63) |   0% (   0) |   0.9048 |   0.8142 |   222.3\n",
            "Elizabeth_II      |     431 |   1.16 +  21.32 =  22.49 ms |    90.99 % |  86% ( 222) |  81% ( 209) |   0.8399 |   0.8349 |   218.7\n",
            "Capital_punishmen |     422 |   1.21 +  20.85 =  22.06 ms |    95.73 % |  89% ( 211) |  93% ( 211) |   0.9123 |   0.9205 |   217.6\n",
            "Age_of_Enlightenm |     347 |   1.57 +  21.65 =  23.22 ms |    92.51 % |  92% ( 347) |   0% (   0) |    0.928 |   0.8902 |   217.9\n",
            "Athanasius_of_Ale |     377 |   1.01 +  22.43 =  23.44 ms |    83.76 % |  74% ( 197) |  91% ( 180) |   0.8249 |   0.8113 |   208.2\n",
            "Freemasonry       |     444 |   1.22 +  21.39 =   22.6 ms |    89.67 % |  86% ( 271) |  98% ( 173) |   0.9144 |   0.8928 |   215.1\n",
            "Montevideo        |     247 |   1.16 +  21.41 =  22.57 ms |    93.52 % |  90% ( 247) |   0% (   0) |   0.9069 |   0.8881 |   209.2\n",
            "Poultry           |     310 |   1.19 +   21.6 =  22.79 ms |    82.14 % |  64% ( 140) |  94% ( 170) |   0.8097 |   0.7285 |   215.9\n",
            "Clothing          |     203 |    1.1 +  20.28 =  21.38 ms |    97.14 % |  97% ( 105) |  94% (  98) |   0.9606 |   0.9365 |   186.9\n",
            "Department_store  |     378 |   1.12 +  20.23 =  21.35 ms |    86.52 % |  82% ( 178) |  95% ( 200) |   0.8915 |   0.8771 |   200.5\n",
            "Marvel_Comics     |     247 |   1.09 +  22.24 =  23.33 ms |    85.37 % |  78% ( 123) |  82% ( 124) |   0.8057 |   0.7998 |   222.5\n",
            "Alloy             |     198 |   1.05 +  18.96 =  20.01 ms |    95.83 % |  90% (  96) |  89% ( 102) |    0.899 |   0.8988 |   190.9\n",
            "Electric_motor    |     363 |   1.13 +  21.13 =  22.26 ms |    90.16 % |  83% ( 183) |  92% ( 180) |   0.8788 |   0.8503 |   214.1\n",
            "Nutrition         |     275 |   1.44 +  22.16 =   23.6 ms |    94.55 % |  92% ( 275) |   0% (   0) |     0.92 |   0.8808 |   210.3\n",
            "Chinese_character |     168 |   0.97 +  23.97 =  24.94 ms |    91.07 % |  86% ( 168) |   0% (   0) |    0.869 |   0.8207 |   216.0\n",
            "Bermuda           |     434 |   1.08 +  20.43 =  21.51 ms |    89.91 % |  89% ( 228) |  90% ( 206) |   0.8963 |   0.8612 |   209.0\n",
            "Utrecht           |     210 |   0.92 +   19.5 =  20.42 ms |    89.42 % |  84% ( 104) |  87% ( 106) |   0.8619 |   0.7485 |   192.1\n",
            "Capacitor         |     364 |   1.54 +  22.05 =  23.59 ms |    87.57 % |  86% ( 185) |  97% ( 179) |   0.9176 |   0.8752 |   200.5\n",
            "Comcast           |     266 |   1.12 +  20.79 =  21.91 ms |     91.6 % |  79% ( 131) |  91% ( 135) |   0.8571 |   0.8567 |   204.1\n",
            "Tuberculosis      |     298 |   1.27 +  19.95 =  21.22 ms |    88.08 % |  80% ( 151) |  91% ( 147) |   0.8557 |   0.8598 |   188.3\n",
            "North_Carolina    |     297 |   1.12 +  20.93 =  22.05 ms |    94.28 % |  90% ( 297) |   0% (   0) |   0.9057 |   0.8976 |   209.0\n",
            "Heian_period      |     205 |   1.12 +  20.99 =   22.1 ms |    92.71 % |  87% (  96) |  84% ( 109) |   0.8585 |   0.8608 |   208.2\n",
            "On_the_Origin_of_ |     261 |   1.29 +   21.0 =   22.3 ms |    79.31 % |  76% ( 261) |   0% (   0) |   0.7663 |   0.6847 |   218.8\n",
            "Political_party   |     245 |   1.12 +  20.76 =  21.88 ms |    94.55 % |  90% ( 110) |  95% ( 135) |   0.9306 |   0.9024 |   218.0\n",
            "Tibet             |     322 |   1.14 +  21.83 =  22.97 ms |    93.63 % |  91% ( 157) |  89% ( 165) |   0.9037 |   0.8913 |   208.5\n",
            "Oklahoma          |     271 |   0.98 +  22.52 =   23.5 ms |    97.05 % |  94% ( 271) |   0% (   0) |   0.9446 |   0.9177 |   215.0\n",
            "European_Central_ |     253 |   1.25 +  20.98 =  22.23 ms |    88.98 % |  84% ( 127) |  96% ( 126) |   0.9051 |    0.879 |   218.8\n",
            "Group_(mathematic |     259 |   1.17 +  21.16 =  22.33 ms |    94.59 % |  90% ( 111) |  89% ( 148) |   0.9035 |   0.8908 |   214.8\n",
            "God               |     293 |   1.24 +  21.94 =  23.18 ms |    92.49 % |  84% ( 173) |  84% ( 120) |    0.843 |   0.7883 |   207.6\n",
            "Karl_Popper       |     391 |   1.29 +  22.59 =  23.88 ms |    71.67 % |  66% ( 180) |  90% ( 211) |   0.7954 |   0.7945 |   222.0\n",
            "Planck_constant   |     265 |   1.29 +  22.15 =  23.44 ms |    92.95 % |  91% ( 156) |  97% ( 109) |   0.9396 |   0.8862 |   220.5\n",
            "Zinc              |     344 |   1.16 +  19.93 =  21.09 ms |    91.89 % |  86% ( 185) |  94% ( 159) |   0.9041 |   0.8875 |   183.7\n",
            "Appalachian_Mount |     228 |    1.1 +  25.91 =  27.01 ms |    88.33 % |  85% (  60) |  89% ( 168) |   0.8816 |   0.8701 |   220.2\n",
            "Post-punk         |     352 |   1.16 +  22.88 =  24.04 ms |    83.64 % |  77% ( 214) |  97% ( 138) |   0.8551 |   0.8474 |   221.6\n",
            "Richard_Feynman   |     304 |   1.21 +  23.23 =  24.44 ms |    82.78 % |  78% ( 151) |  96% ( 153) |   0.8783 |    0.868 |   214.6\n",
            "Muammar_Gaddafi   |     485 |   1.18 +  21.94 =  23.12 ms |    80.41 % |  74% ( 485) |   0% (   0) |   0.7485 |   0.7066 |   205.3\n",
            "Neptune           |     399 |   1.08 +  21.26 =  22.34 ms |    87.93 % |  82% ( 174) |  92% ( 225) |   0.8772 |   0.8589 |   211.5\n",
            "Endangered_Specie |     274 |   1.16 +  21.12 =  22.28 ms |    85.95 % |  80% ( 121) |  90% ( 153) |   0.8613 |   0.8283 |   222.0\n",
            "Han_dynasty       |     323 |   1.13 +  21.91 =  23.04 ms |    95.36 % |  90% ( 323) |   0% (   0) |   0.9009 |   0.8772 |   214.3\n",
            "Transistor        |     216 |   1.16 +  23.89 =  25.05 ms |    95.41 % |  90% ( 109) |  80% ( 107) |   0.8565 |   0.8265 |   213.1\n",
            "51st_state        |     293 |   1.17 +  20.79 =  21.96 ms |    95.52 % |  89% ( 134) |  91% ( 159) |   0.9078 |    0.883 |   217.3\n",
            "Antenna_(radio)   |     302 |   1.18 +  20.56 =  21.74 ms |    79.47 % |  71% ( 302) |   0% (   0) |   0.7119 |   0.6853 |   214.8\n",
            "Washington_Univer |     341 |   1.23 +  21.08 =  22.31 ms |    91.76 % |  89% ( 170) |  88% ( 171) |   0.8915 |   0.8557 |   209.1\n",
            "Central_Intellige |      65 |    1.3 +  21.81 =  23.11 ms |    95.38 % |  95% (  65) |   0% (   0) |   0.9538 |   0.9517 |   216.7\n",
            "Armenia           |     241 |   1.08 +  20.76 =  21.84 ms |    92.53 % |  88% ( 241) |   0% (   0) |   0.8838 |   0.8572 |   208.2\n",
            "Muslim_world      |     241 |   1.24 +   25.0 =  26.24 ms |    97.66 % |  94% ( 128) |  92% ( 113) |   0.9336 |   0.9113 |   215.7\n",
            "Kievan_Rus%27     |     317 |   1.17 +  22.74 =  23.91 ms |    93.13 % |  89% ( 131) |  89% ( 186) |   0.8927 |   0.8832 |   217.7\n",
            "Immaculate_Concep |     197 |   1.45 +  23.83 =  25.28 ms |    60.58 % |  45% ( 104) |  89% (  93) |   0.6599 |   0.6244 |   233.5\n",
            "Egypt             |     247 |   1.12 +  20.23 =  21.35 ms |    95.95 % |  95% ( 247) |   0% (   0) |   0.9514 |   0.9213 |   204.9\n",
            "Protestantism     |     369 |   1.01 +  20.29 =   21.3 ms |    91.33 % |  87% ( 369) |   0% (   0) |    0.878 |   0.8477 |   205.1\n",
            "Richmond,_Virgini |     291 |   1.11 +  21.13 =  22.24 ms |    97.94 % |  93% ( 291) |   0% (   0) |   0.9381 |   0.9537 |   211.8\n",
            "Normans           |     208 |   1.03 +  21.37 =   22.4 ms |    90.62 % |  79% (  96) |  66% ( 112) |   0.7212 |   0.6845 |   212.6\n",
            "Computational_com |     418 |   1.56 +  20.28 =  21.84 ms |     93.4 % |  76% ( 197) |  85% ( 221) |   0.8158 |   0.7952 |   208.9\n",
            "Southern_Californ |     324 |   1.19 +  21.12 =  22.31 ms |    96.11 % |  85% ( 180) |  56% ( 144) |   0.7222 |   0.6866 |   211.1\n",
            "Sky_(United_Kingd |     210 |   1.39 +   22.2 =  23.58 ms |    100.0 % |  91% ( 108) |  91% ( 102) |   0.9143 |   0.8958 |   215.1\n",
            "Victoria_(Austral |     247 |   0.98 +   19.2 =  20.18 ms |    100.0 % |  92% ( 124) |  63% ( 123) |   0.7814 |   0.7353 |   193.4\n",
            "Huguenot          |     424 |   1.14 +  22.16 =   23.3 ms |    83.87 % |  74% ( 217) |  69% ( 207) |   0.7217 |   0.7013 |   213.0\n",
            "Steam_engine      |     444 |   1.27 +  20.61 =  21.88 ms |    97.21 % |  89% ( 215) |  56% ( 229) |   0.7252 |   0.7105 |   213.1\n",
            "Oxygen            |     415 |   1.19 +  21.86 =  23.05 ms |    95.85 % |  78% ( 241) |  63% ( 174) |   0.7229 |   0.6803 |   209.4\n",
            "1973_oil_crisis   |     255 |   1.22 +  21.24 =  22.45 ms |    98.11 % |  86% ( 106) |  58% ( 149) |    0.702 |   0.6706 |   209.4\n",
            "European_Union_la |     421 |   1.31 +  21.09 =  22.41 ms |    96.54 % |  90% ( 231) |  92% ( 190) |   0.9169 |   0.8957 |   221.8\n",
            "Amazon_rainforest |     284 |   1.25 +  21.36 =  22.61 ms |    93.99 % |  87% ( 183) |  81% ( 101) |   0.8521 |   0.8068 |   218.0\n",
            "Ctenophora        |     325 |   1.19 +  24.73 =  25.92 ms |    91.18 % |  78% ( 170) |  65% ( 155) |   0.7262 |   0.7167 |   220.4\n",
            "Fresno,_Californi |     275 |   1.04 +  23.25 =   24.3 ms |    98.53 % |  95% ( 136) |  63% ( 139) |   0.7927 |   0.7731 |   217.7\n",
            "Packet_switching  |     373 |   0.97 +  20.99 =  21.96 ms |    85.58 % |  78% ( 104) |  51% ( 269) |   0.5925 |   0.5419 |   209.6\n",
            "Black_Death       |     219 |   1.12 +  22.02 =  23.14 ms |    91.67 % |  80% ( 108) |  68% ( 111) |   0.7443 |   0.7147 |   218.1\n",
            "Geology           |     362 |   1.21 +  20.89 =  22.11 ms |    95.69 % |  88% ( 116) |  73% ( 246) |   0.7818 |   0.7641 |   211.1\n",
            "Pharmacy          |     244 |   1.25 +  22.81 =  24.05 ms |    92.19 % |  89% ( 128) |  94% ( 116) |   0.9221 |   0.8525 |   212.4\n",
            "Civil_disobedienc |     327 |   1.26 +  22.81 =  24.07 ms |    77.66 % |  59% ( 197) |  73% ( 130) |   0.6483 |   0.6038 |   221.9\n",
            "Construction      |     232 |   1.21 +  19.98 =  21.18 ms |    95.92 % |  96% (  98) |  64% ( 134) |   0.7845 |   0.7584 |   202.5\n",
            "Private_school    |     240 |    1.2 +  21.97 =  23.18 ms |    96.46 % |  84% ( 113) |  76% ( 127) |      0.8 |   0.7613 |   212.5\n",
            "Harvard_Universit |     262 |   1.19 +  23.49 =  24.68 ms |    94.64 % |  83% ( 112) |  86% ( 150) |   0.8511 |   0.8463 |   224.6\n",
            "Jacksonville,_Flo |     192 |   1.09 +   20.5 =  21.58 ms |    91.67 % |  84% (  96) |  70% (  96) |    0.776 |   0.7395 |   211.2\n",
            "Economic_inequali |     515 |   1.25 +  20.68 =  21.93 ms |    92.93 % |  83% ( 297) |  87% ( 218) |   0.8544 |   0.8143 |   217.3\n",
            "University_of_Chi |     356 |    1.2 +  24.73 =  25.93 ms |    96.59 % |  87% ( 176) |  73% ( 180) |   0.8062 |   0.8044 |   232.7\n",
            "Yuan_dynasty      |     445 |   1.01 +  21.27 =  22.28 ms |    89.04 % |  82% ( 228) |  70% ( 217) |   0.7663 |   0.7281 |   201.7\n",
            "Immune_system     |     458 |   1.22 +  20.98 =   22.2 ms |    96.74 % |  90% ( 215) |  88% ( 243) |   0.8974 |   0.8833 |   195.9\n",
            "Intergovernmental |     222 |   1.12 +  21.49 =  22.61 ms |    96.15 % |  83% ( 104) |  70% ( 118) |   0.7658 |   0.7389 |   221.8\n",
            "Prime_number      |     308 |   1.58 +  21.45 =  23.02 ms |    98.69 % |  88% ( 153) |  67% ( 155) |    0.776 |   0.7667 |   211.6\n",
            "Rhine             |     498 |    1.1 +  23.99 =  25.09 ms |    89.66 % |  72% ( 290) |  79% ( 208) |    0.755 |   0.7333 |   215.7\n",
            "Scottish_Parliame |     357 |   1.22 +  20.28 =   21.5 ms |    96.83 % |  86% ( 189) |  94% ( 168) |    0.902 |   0.8677 |   216.7\n",
            "Islamism          |     384 |   1.28 +  22.74 =  24.03 ms |    96.95 % |  93% ( 197) |  87% ( 187) |   0.9062 |   0.8722 |   221.1\n",
            "Imperialism       |     364 |   1.16 +   20.4 =  21.56 ms |    94.12 % |  81% ( 187) |  83% ( 177) |   0.8269 |   0.7916 |   211.5\n",
            "Warsaw            |     486 |   1.14 +  21.13 =  22.27 ms |    95.87 % |  91% ( 242) |  54% ( 244) |   0.7284 |   0.7104 |   203.0\n",
            "French_and_Indian |     402 |   1.05 +  20.68 =  21.74 ms |    86.05 % |  80% ( 172) |  79% ( 230) |    0.796 |   0.7631 |   217.1\n",
            "Force             |     377 |   1.17 +  20.36 =  21.52 ms |    95.05 % |  78% ( 202) |  90% ( 175) |   0.8408 |   0.8192 |   212.2\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|--------------\n",
            "Grand Total       |   34926 |   1.21 +  21.84 =  23.05 ms |    91.76 % |  85% (20857) |  84% (14069) |   0.8501 |   0.8281 |   212.4\n"
          ]
        }
      ],
      "source": [
        "# with mpnet and electra-best (optimized) - k = 10, context_length_threshold = 205 - new themes"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5ddb55d3-7e83-4e1b-ba3b-12b0630a2dc4",
        "id": "b7CqcBpLDNmo"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Theme             | Queries |  AIT: (ANN) + (QnA) = Total | ansInCtx % | TP % (TotP) | TN % (TotN) | Para Acc | F1 Score | Avg Ctx Size\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|-------------\n",
            "Beyoncé           |     228 |   1.25 +   22.4 =  23.65 ms |    95.61 % |  85% ( 228) |   0% (   0) |   0.8596 |   0.8765 |   220.2\n",
            "Frédéric_Chopin   |     203 |   1.23 +  22.49 =  23.72 ms |    88.18 % |  85% ( 203) |   0% (   0) |   0.8522 |    0.836 |   214.5\n",
            "Sino-Tibetan_rela |      98 |   1.34 +  24.68 =  26.02 ms |    92.86 % |  85% (  98) |   0% (   0) |   0.8571 |   0.8492 |   224.3\n",
            "The_Legend_of_Zel |     125 |   1.32 +  22.82 =  24.14 ms |    96.05 % |  82% (  76) |  91% (  49) |    0.864 |   0.8541 |   214.4\n",
            "Spectre_(2015_fil |     142 |   1.25 +  23.15 =   24.4 ms |     95.4 % |  82% (  87) |  87% (  55) |   0.8451 |   0.8627 |   217.2\n",
            "New_York_City     |     253 |   1.28 +  21.95 =  23.23 ms |     99.6 % |  92% ( 253) |   0% (   0) |   0.9289 |     0.93 |   220.5\n",
            "To_Kill_a_Mocking |      62 |   1.52 +   23.7 =  25.22 ms |    93.55 % |  82% (  62) |   0% (   0) |   0.8226 |   0.8643 |   220.7\n",
            "Solar_energy      |      73 |   1.54 +   21.8 =  23.34 ms |    94.52 % |  93% (  73) |   0% (   0) |   0.9315 |    0.903 |   215.7\n",
            "Kanye_West        |     148 |   1.25 +  21.74 =  22.99 ms |    92.57 % |  77% ( 148) |   0% (   0) |    0.777 |   0.8294 |   217.6\n",
            "Buddhism          |     189 |   1.17 +  23.98 =  25.16 ms |    91.01 % |  86% ( 189) |   0% (   0) |   0.8677 |    0.824 |   215.0\n",
            "American_Idol     |     265 |   1.13 +   20.3 =  21.43 ms |     91.7 % |  74% ( 265) |   0% (   0) |   0.7434 |   0.7882 |   210.5\n",
            "Dog               |     117 |   1.39 +  20.59 =  21.97 ms |    97.44 % |  97% ( 117) |   0% (   0) |   0.9744 |   0.9181 |   210.8\n",
            "2008_Summer_Olymp |     227 |    1.1 +  21.44 =  22.54 ms |    89.87 % |  80% ( 227) |   0% (   0) |   0.8062 |   0.7855 |   215.4\n",
            "Genome            |      56 |   1.26 +  21.18 =  22.45 ms |    95.83 % |  91% (  24) | 100% (  32) |   0.9643 |    0.925 |   207.6\n",
            "Comprehensive_sch |      52 |   1.41 +  20.26 =  21.67 ms |    100.0 % |  87% (  16) | 100% (  36) |   0.9615 |   0.9712 |   216.7\n",
            "Republic_of_the_C |     111 |   1.18 +  20.51 =  21.69 ms |    96.43 % |  82% (  56) |  96% (  55) |   0.8919 |   0.8879 |   207.5\n",
            "Prime_minister    |      55 |   1.25 +  20.59 =  21.83 ms |    100.0 % |  88% (  26) |  93% (  29) |   0.9091 |   0.9091 |   222.2\n",
            "Institute_of_tech |      55 |   1.49 +  21.08 =  22.56 ms |    96.36 % |  83% (  55) |   0% (   0) |   0.8364 |   0.8933 |   207.5\n",
            "Dutch_Republic    |      62 |   1.29 +  20.84 =  22.12 ms |    100.0 % |  95% (  24) |  94% (  38) |   0.9516 |    0.894 |   218.1\n",
            "Symbiosis         |      51 |   1.71 +  26.77 =  28.48 ms |    93.75 % |  75% (  16) |  91% (  35) |   0.8627 |   0.8562 |   209.7\n",
            "Iranian_languages |      40 |   1.44 +  21.55 =  22.99 ms |    94.74 % |  73% (  19) |  90% (  21) |    0.825 |      0.8 |   198.9\n",
            "Lighting          |      42 |   1.54 +  19.87 =   21.4 ms |    95.24 % |  92% (  42) |   0% (   0) |   0.9286 |   0.8283 |   196.4\n",
            "Separation_of_pow |      69 |   1.36 +  20.34 =   21.7 ms |    83.33 % |  70% (  24) |  93% (  45) |   0.8551 |   0.8435 |   217.8\n",
            "Architecture      |     105 |   1.13 +   20.8 =  21.93 ms |    94.74 % |  87% (  57) |  95% (  48) |   0.9143 |   0.9101 |   220.9\n",
            "Southern_Europe   |      66 |   1.26 +  18.69 =  19.95 ms |    100.0 % |  94% (  34) |  90% (  32) |   0.9242 |   0.9185 |   202.3\n",
            "BBC_Television    |      65 |   1.46 +  20.44 =   21.9 ms |    94.29 % |  80% (  35) |  93% (  30) |   0.8615 |   0.8857 |   221.3\n",
            "Arnold_Schwarzene |      56 |   1.41 +   21.9 =  23.32 ms |    96.43 % |  87% (  56) |   0% (   0) |    0.875 |   0.9226 |   218.0\n",
            "Plymouth          |     122 |   1.11 +  20.77 =  21.88 ms |    99.18 % |  93% ( 122) |   0% (   0) |   0.9344 |   0.9292 |   211.5\n",
            "Christian         |      62 |   1.38 +  27.26 =  28.64 ms |     95.0 % |  90% (  20) |  80% (  42) |   0.8387 |   0.8011 |   211.2\n",
            "Sony_Music_Entert |      62 |   1.15 +  20.31 =  21.46 ms |     91.3 % |  78% (  23) |  97% (  39) |   0.9032 |   0.8468 |   212.6\n",
            "Oklahoma_City     |      63 |   1.26 +  21.85 =  23.11 ms |    98.41 % |  96% (  63) |   0% (   0) |   0.9683 |    0.919 |   214.8\n",
            "Hunter-gatherer   |      97 |   1.31 +  22.21 =  23.52 ms |    84.21 % |  73% (  38) |  96% (  59) |   0.8763 |    0.852 |   216.4\n",
            "United_Nations_Po |      68 |   1.46 +  22.89 =  24.35 ms |    97.14 % |  91% (  35) |  84% (  33) |   0.8824 |   0.8766 |   223.9\n",
            "Russian_Soviet_Fe |     103 |   1.38 +   24.3 =  25.68 ms |    98.28 % |  87% (  58) |  91% (  45) |   0.8932 |   0.9158 |   217.6\n",
            "Alexander_Graham_ |     105 |   1.25 +  21.41 =  22.66 ms |    95.24 % |  86% ( 105) |   0% (   0) |   0.8667 |   0.8405 |   221.4\n",
            "Internet_service_ |      51 |   1.14 +  21.16 =   22.3 ms |    100.0 % |  88% (  27) | 100% (  24) |   0.9412 |   0.9057 |   210.7\n",
            "Comics            |     102 |    1.3 +  21.35 =  22.65 ms |    95.83 % |  81% (  48) |  94% (  54) |   0.8824 |   0.9134 |   211.9\n",
            "Saint_Helena      |      93 |   1.08 +  20.26 =  21.34 ms |    100.0 % |  95% (  93) |   0% (   0) |    0.957 |   0.9361 |   213.3\n",
            "Aspirated_consona |      77 |    1.3 +   27.7 =   29.0 ms |    93.33 % |  80% (  30) |  85% (  47) |   0.8312 |   0.8258 |   206.3\n",
            "Hydrogen          |      77 |   1.22 +  22.96 =  24.18 ms |     96.1 % |  94% (  77) |   0% (   0) |   0.9481 |   0.8571 |   206.9\n",
            "Space_Race        |      67 |   1.47 +  21.75 =  23.22 ms |    98.51 % |  85% (  67) |   0% (   0) |   0.8507 |   0.8539 |   218.0\n",
            "BeiDou_Navigation |      82 |   1.23 +  22.77 =   24.0 ms |    97.83 % |  84% (  46) |  97% (  36) |   0.9024 |   0.8772 |   212.8\n",
            "Canon_law         |      61 |   1.16 +  21.82 =  22.99 ms |    96.77 % |  80% (  31) |  93% (  30) |   0.8689 |   0.8634 |   228.7\n",
            "Communications_in |      79 |   1.16 +  20.35 =  21.51 ms |    100.0 % |  97% (  35) |  90% (  44) |   0.9367 |   0.9175 |   202.5\n",
            "Boston            |     110 |   1.07 +  23.29 =  24.36 ms |    98.18 % |  98% ( 110) |   0% (   0) |   0.9818 |   0.9735 |   218.7\n",
            "Universal_Studios |     115 |   1.14 +  24.14 =  25.28 ms |    96.36 % |  87% (  55) |  90% (  60) |    0.887 |   0.9043 |   217.1\n",
            "Estonian_language |      69 |   1.14 +  21.46 =  22.59 ms |    94.12 % |  85% (  34) |  85% (  35) |   0.8551 |   0.8754 |   209.4\n",
            "Daylight_saving_t |      93 |   1.95 +  21.38 =  23.34 ms |    94.62 % |  88% (  93) |   0% (   0) |   0.8817 |   0.8692 |   220.7\n",
            "Royal_Institute_o |      81 |   1.25 +  20.66 =  21.91 ms |    95.24 % |  90% (  42) |  92% (  39) |   0.9136 |   0.9095 |   222.5\n",
            "National_Archives |      66 |   1.22 +  20.36 =  21.59 ms |    100.0 % |  96% (  32) |  94% (  34) |   0.9545 |   0.9606 |   215.5\n",
            "Tristan_da_Cunha  |      60 |   1.12 +  21.44 =  22.56 ms |    100.0 % | 100% (  10) |  94% (  50) |     0.95 |    0.935 |   215.0\n",
            "University_of_Kan |      98 |   1.47 +  22.62 =  24.09 ms |    95.56 % |  86% (  45) |  86% (  53) |   0.8673 |   0.8724 |   218.1\n",
            "Arena_Football_Le |      81 |   1.38 +  20.44 =  21.82 ms |    95.06 % |  85% (  81) |   0% (   0) |   0.8519 |   0.8593 |   219.7\n",
            "Bern              |      47 |    1.2 +  24.26 =  25.46 ms |    100.0 % |  89% (  47) |   0% (   0) |   0.8936 |   0.8028 |   211.4\n",
            "Westminster_Abbey |     130 |   1.04 +  20.29 =  21.33 ms |    95.16 % |  91% (  62) |  95% (  68) |   0.9385 |   0.9252 |   218.7\n",
            "Political_corrupt |      51 |   1.39 +  21.28 =  22.67 ms |    98.04 % | 100% (  51) |   0% (   0) |      1.0 |   0.9789 |   220.8\n",
            "Classical_music   |      97 |   1.36 +  23.36 =  24.73 ms |    94.85 % |  91% (  97) |   0% (   0) |   0.9175 |   0.8996 |   218.4\n",
            "Slavs             |     167 |   1.39 +  24.74 =  26.13 ms |    95.12 % |  97% (  82) |  91% (  85) |   0.9461 |   0.9184 |   216.0\n",
            "Treaty            |      87 |   1.72 +  20.16 =  21.88 ms |     97.7 % |  95% (  87) |   0% (   0) |    0.954 |   0.9346 |   209.3\n",
            "Josip_Broz_Tito   |     104 |   1.24 +  21.46 =   22.7 ms |    97.12 % |  94% ( 104) |   0% (   0) |   0.9423 |    0.899 |   213.3\n",
            "Marshall_Islands  |      83 |   1.31 +  21.37 =  22.68 ms |    97.59 % |  93% (  83) |   0% (   0) |   0.9398 |   0.9394 |   212.2\n",
            "Szlachta          |      84 |   1.17 +   25.4 =  26.56 ms |    95.24 % |  91% (  84) |   0% (   0) |   0.9167 |   0.8313 |   215.7\n",
            "Virgil            |      65 |   1.36 +  23.65 =  25.01 ms |    100.0 % |  92% (  25) |  80% (  40) |   0.8462 |   0.8615 |   222.3\n",
            "Alps              |      92 |   1.22 +  21.37 =  22.59 ms |    100.0 % |  97% (  92) |   0% (   0) |   0.9783 |   0.9813 |   219.5\n",
            "Gene              |      92 |   1.27 +  20.35 =  21.62 ms |    95.65 % |  94% (  92) |   0% (   0) |   0.9457 |   0.8719 |   208.5\n",
            "Guinea-Bissau     |      82 |   1.13 +   20.3 =  21.43 ms |    98.78 % |  92% (  82) |   0% (   0) |   0.9268 |   0.9195 |   188.6\n",
            "List_of_numbered_ |      68 |   1.46 +  22.41 =  23.88 ms |    97.06 % |  91% (  68) |   0% (   0) |   0.9118 |   0.9342 |   208.8\n",
            "Brain             |      78 |   1.32 +  23.11 =  24.43 ms |    100.0 % |  92% (  78) |   0% (   0) |   0.9231 |    0.895 |   216.1\n",
            "Near_East         |      59 |   1.63 +  20.46 =  22.09 ms |    93.22 % |  94% (  59) |   0% (   0) |   0.9492 |   0.9153 |   207.4\n",
            "Zhejiang          |     116 |   1.09 +   20.8 =  21.89 ms |    98.25 % | 100% (  57) |  98% (  59) |   0.9914 |   0.9791 |   208.6\n",
            "Ministry_of_Defen |      79 |   1.13 +  20.77 =  21.89 ms |    100.0 % |  97% (  38) |  90% (  41) |   0.9367 |   0.9269 |   224.0\n",
            "High-definition_t |     116 |   1.54 +  23.85 =   25.4 ms |    96.97 % |  86% (  66) |  92% (  50) |   0.8879 |   0.8793 |   218.7\n",
            "Wood              |      98 |   1.58 +   20.8 =  22.37 ms |    96.94 % |  88% (  98) |   0% (   0) |   0.8878 |   0.8759 |   196.5\n",
            "Somalis           |      83 |   1.55 +  21.95 =   23.5 ms |     98.8 % |  93% (  83) |   0% (   0) |   0.9398 |   0.9229 |   209.8\n",
            "Middle_Ages       |     118 |   1.19 +  21.82 =  23.01 ms |    99.15 % |  95% ( 118) |   0% (   0) |   0.9576 |   0.9317 |   217.3\n",
            "Phonology         |      53 |   1.24 +   23.0 =  24.24 ms |    100.0 % |  95% (  23) |  96% (  30) |   0.9623 |   0.9623 |   214.8\n",
            "Computer          |      68 |    1.5 +  20.04 =  21.54 ms |    97.06 % |  94% (  68) |   0% (   0) |   0.9412 |    0.892 |   200.7\n",
            "Black_people      |      87 |   1.07 +  22.59 =  23.67 ms |     95.4 % |  94% (  87) |   0% (   0) |   0.9425 |   0.8526 |   217.5\n",
            "New_Delhi         |      76 |   1.48 +  22.29 =  23.77 ms |    94.74 % |  93% (  76) |   0% (   0) |   0.9342 |   0.9265 |   215.7\n",
            "Bird_migration    |      59 |   1.06 +  19.15 =  20.21 ms |    93.22 % |  93% (  59) |   0% (   0) |   0.9322 |   0.8694 |   191.5\n",
            "Atlantic_City,_Ne |      79 |   1.44 +  22.38 =  23.83 ms |    98.73 % |  87% (  79) |   0% (   0) |   0.8734 |   0.8598 |   212.9\n",
            "MP3               |     107 |   1.32 +  22.33 =  23.64 ms |    92.52 % |  86% ( 107) |   0% (   0) |   0.8692 |   0.8318 |   213.6\n",
            "House_music       |     123 |   1.19 +  23.45 =  24.64 ms |    88.89 % |  83% (  54) |  91% (  69) |    0.878 |   0.8636 |   222.2\n",
            "Letter_case       |      13 |   2.16 +  27.27 =  29.44 ms |    100.0 % |  84% (  13) |   0% (   0) |   0.8462 |   0.8034 |   231.7\n",
            "Chihuahua_(state) |     115 |    1.1 +  22.02 =  23.12 ms |    98.26 % |  93% ( 115) |   0% (   0) |   0.9391 |   0.9246 |   218.2\n",
            "Pitch_(music)     |      14 |   1.83 +  21.12 =  22.94 ms |    100.0 % | 100% (  14) |   0% (   0) |      1.0 |      1.0 |   216.3\n",
            "England_national_ |      32 |   1.72 +  20.99 =  22.71 ms |    94.12 % |  88% (  17) | 100% (  15) |   0.9375 |   0.9688 |   212.9\n",
            "Houston           |     143 |   1.04 +  20.41 =  21.45 ms |    98.72 % |  96% (  78) |  92% (  65) |   0.9441 |    0.919 |   211.3\n",
            "Copper            |      82 |   1.08 +  21.28 =  22.36 ms |    95.74 % |  91% (  47) | 100% (  35) |   0.9512 |   0.9393 |   206.6\n",
            "Identity_(social_ |      51 |   1.81 +  23.18 =  24.98 ms |    95.65 % |  91% (  23) |  85% (  28) |   0.8824 |    0.851 |   221.1\n",
            "Himachal_Pradesh  |      64 |   1.13 +  23.24 =  24.37 ms |    93.55 % |  83% (  31) |  78% (  33) |   0.8125 |   0.7978 |   188.1\n",
            "Communication     |      27 |   1.63 +  20.17 =   21.8 ms |    92.31 % |  92% (  13) |  92% (  14) |   0.9259 |   0.8889 |   197.6\n",
            "Computer_security |     122 |    1.3 +  20.66 =  21.96 ms |    95.83 % | 100% (  48) |  85% (  74) |   0.9098 |   0.8944 |   220.7\n",
            "Orthodox_Judaism  |      69 |   1.39 +  21.27 =  22.66 ms |    90.62 % |  90% (  32) |  94% (  37) |   0.9275 |    0.913 |   219.7\n",
            "Animal            |      26 |   2.56 +  23.31 =  25.87 ms |    96.15 % |  88% (  26) |   0% (   0) |   0.8846 |    0.859 |   188.9\n",
            "Beer              |      90 |   1.28 +  21.56 =  22.83 ms |    100.0 % |  91% (  47) |  97% (  43) |   0.9444 |   0.9669 |   210.6\n",
            "Race_and_ethnicit |      14 |   2.23 +  21.47 =   23.7 ms |    100.0 % |  92% (  14) |   0% (   0) |   0.9286 |      1.0 |   218.0\n",
            "Imperial_College_ |      96 |   1.17 +  22.55 =  23.73 ms |    96.36 % |  92% (  55) |  87% (  41) |   0.9062 |    0.906 |   221.3\n",
            "Hanover           |      53 |   1.09 +  22.97 =  24.06 ms |    100.0 % | 100% (  29) |  95% (  24) |   0.9811 |   0.9407 |   205.4\n",
            "Emotion           |     111 |   1.39 +  21.65 =  23.04 ms |    98.41 % |  93% (  63) |  95% (  48) |   0.9459 |   0.9483 |   212.8\n",
            "Old_English       |      49 |   1.52 +  22.21 =  23.73 ms |     96.3 % |  96% (  27) |  81% (  22) |    0.898 |    0.898 |   220.6\n",
            "Aircraft_carrier  |      89 |   1.51 +  20.84 =  22.35 ms |    94.12 % |  82% (  51) |  94% (  38) |   0.8764 |   0.8958 |   219.9\n",
            "Federal_Aviation_ |      41 |   1.76 +  20.44 =   22.2 ms |    100.0 % | 100% (  25) |  81% (  16) |   0.9268 |   0.9024 |   222.2\n",
            "Lancashire        |      59 |    1.1 +  21.75 =  22.85 ms |    100.0 % |  96% (  30) |  82% (  29) |   0.8983 |   0.8927 |   215.1\n",
            "Mesozoic          |      38 |   1.53 +  21.55 =  23.08 ms |    94.12 % |  82% (  17) |  85% (  21) |   0.8421 |   0.7696 |   196.3\n",
            "Videoconferencing |      57 |   1.29 +  24.93 =  26.22 ms |    80.77 % |  80% (  26) |  93% (  31) |   0.8772 |   0.8491 |   221.1\n",
            "Gregorian_calenda |      66 |   1.46 +  21.46 =  22.92 ms |    94.59 % |  86% (  37) |  89% (  29) |   0.8788 |   0.8696 |   222.9\n",
            "Xbox_360          |      68 |   1.29 +  20.44 =  21.74 ms |    92.86 % |  85% (  28) |  90% (  40) |   0.8824 |   0.8911 |   219.3\n",
            "Military_history_ |     121 |   1.09 +  20.37 =  21.46 ms |    92.73 % |  90% (  55) |  81% (  66) |   0.8595 |   0.8492 |   219.2\n",
            "Infrared          |      57 |   1.59 +  20.52 =  22.11 ms |    92.86 % |  85% (  28) |  89% (  29) |   0.8772 |   0.8596 |   214.5\n",
            "ASCII             |      72 |   1.27 +   22.5 =  23.77 ms |     96.3 % |  85% (  27) |  93% (  45) |   0.9028 |   0.8941 |   218.8\n",
            "Digestion         |      29 |    1.4 +  19.97 =  21.37 ms |    100.0 % | 100% (  29) |   0% (   0) |      1.0 |   0.9204 |   177.9\n",
            "Gymnastics        |      75 |   1.13 +  21.01 =  22.14 ms |    92.31 % |  97% (  39) |  80% (  36) |   0.8933 |   0.8056 |   204.3\n",
            "FC_Barcelona      |      75 |   1.25 +  21.47 =  22.72 ms |     96.0 % |  88% (  75) |   0% (   0) |     0.88 |    0.914 |   217.7\n",
            "Melbourne         |      81 |   1.35 +  20.55 =   21.9 ms |    98.77 % |  97% (  81) |   0% (   0) |   0.9753 |   0.9598 |   217.1\n",
            "John,_King_of_Eng |      70 |   1.11 +  20.71 =  21.82 ms |    97.14 % |  94% (  70) |   0% (   0) |   0.9429 |   0.9429 |   218.2\n",
            "Macintosh         |     155 |    1.3 +  21.66 =  22.97 ms |    96.43 % |  94% (  84) |  94% (  71) |   0.9419 |     0.91 |   219.6\n",
            "Valencia          |      80 |   1.06 +  23.52 =  24.58 ms |     95.0 % |  93% (  80) |   0% (   0) |   0.9375 |   0.9196 |   225.3\n",
            "General_Electric  |      50 |   1.48 +  22.86 =  24.34 ms |    100.0 % | 100% (  24) |  92% (  26) |     0.96 |     0.96 |   218.7\n",
            "United_States_Arm |      82 |   1.15 +  20.81 =  21.95 ms |    97.73 % |  95% (  44) |  94% (  38) |   0.9512 |    0.939 |   217.5\n",
            "Franco-Prussian_W |      93 |   1.23 +  20.44 =  21.67 ms |    97.85 % |  93% (  93) |   0% (   0) |   0.9355 |   0.9103 |   216.3\n",
            "Adolescence       |     105 |   1.76 +   20.3 =  22.06 ms |    97.14 % |  96% ( 105) |   0% (   0) |   0.9619 |   0.8992 |   214.2\n",
            "Antarctica        |     156 |   1.03 +   22.8 =  23.83 ms |    100.0 % |  94% (  57) |  92% (  99) |   0.9359 |   0.9189 |   214.4\n",
            "Eritrea           |     119 |   1.29 +  20.26 =  21.55 ms |    96.92 % |  98% (  65) |  87% (  54) |   0.9328 |   0.8932 |   204.0\n",
            "Uranium           |     142 |   1.33 +  23.57 =   24.9 ms |     95.0 % |  86% (  60) |  97% (  82) |   0.9296 |    0.939 |   211.6\n",
            "Circadian_rhythm  |      55 |   1.35 +  21.48 =  22.83 ms |    96.43 % | 100% (  28) |  92% (  27) |   0.9636 |    0.955 |   203.2\n",
            "Sexual_orientatio |      91 |   1.15 +   21.7 =  22.85 ms |    85.71 % |  86% (  91) |   0% (   0) |   0.8681 |   0.6888 |   217.0\n",
            "Dell              |     135 |   1.24 +  22.39 =  23.63 ms |    93.65 % |  87% (  63) |  87% (  72) |   0.8741 |    0.879 |   220.7\n",
            "Nintendo_Entertai |     103 |    1.1 +  20.96 =  22.07 ms |    98.15 % |  90% (  54) |  95% (  49) |    0.932 |   0.9521 |   216.0\n",
            "Seattle           |      72 |   1.14 +  20.69 =  21.83 ms |    100.0 % |  95% (  72) |   0% (   0) |   0.9583 |   0.9514 |   217.3\n",
            "Memory            |      96 |   1.29 +  20.75 =  22.05 ms |    75.68 % |  70% (  37) |  89% (  59) |   0.8229 |   0.8148 |   205.8\n",
            "Multiracial_Ameri |     140 |   1.28 +  21.64 =  22.92 ms |    93.67 % |  87% (  79) |  98% (  61) |   0.9214 |   0.8787 |   214.5\n",
            "Ashkenazi_Jews    |      59 |   1.83 +  24.68 =  26.51 ms |    100.0 % | 100% (  59) |   0% (   0) |      1.0 |   0.9432 |   228.4\n",
            "Pharmaceutical_in |     183 |    1.1 +  22.66 =  23.76 ms |    93.81 % |  92% ( 113) |  92% (  70) |    0.929 |   0.8869 |   221.8\n",
            "Umayyad_Caliphate |      90 |   1.14 +  23.27 =  24.41 ms |    94.44 % |  88% (  36) |  88% (  54) |   0.8889 |   0.8897 |   215.4\n",
            "Asphalt           |      84 |   1.16 +  23.91 =  25.07 ms |    86.36 % |  77% (  44) |  87% (  40) |   0.8214 |   0.7937 |   214.5\n",
            "Queen_Victoria    |     274 |   1.14 +  20.53 =  21.67 ms |    92.27 % |  87% ( 207) |  97% (  67) |   0.8978 |   0.8659 |   210.8\n",
            "Israel            |      96 |   0.95 +  20.69 =  21.64 ms |    97.92 % |  96% (  96) |   0% (   0) |   0.9688 |   0.9194 |   210.2\n",
            "Hellenistic_perio |     111 |   1.22 +  23.95 =  25.16 ms |     97.3 % |  96% ( 111) |   0% (   0) |    0.964 |   0.9595 |   217.5\n",
            "Bill_%26_Melinda_ |      58 |    1.1 +  22.36 =  23.46 ms |    92.31 % |  76% (  26) |  87% (  32) |   0.8276 |   0.6984 |   222.6\n",
            "Dutch_language    |      90 |    1.6 +  22.87 =  24.47 ms |    93.33 % |  90% (  90) |   0% (   0) |      0.9 |   0.8399 |   215.6\n",
            "Buckingham_Palace |     136 |   1.01 +  20.33 =  21.34 ms |    94.38 % |  93% (  89) |  97% (  47) |   0.9485 |   0.9236 |   218.2\n",
            "Incandescent_ligh |     105 |   1.44 +  20.84 =  22.28 ms |    91.23 % |  84% (  57) |  93% (  48) |   0.8857 |   0.8646 |   209.5\n",
            "Arsenal_F.C.      |      96 |   1.11 +  21.35 =  22.47 ms |    95.83 % |  81% (  48) |  85% (  48) |   0.8333 |   0.8285 |   216.6\n",
            "Chicago_Cubs      |      68 |   1.62 +  23.11 =  24.73 ms |    95.59 % |  95% (  68) |   0% (   0) |   0.9559 |   0.9578 |   221.0\n",
            "Korean_War        |     109 |   1.27 +   21.7 =  22.97 ms |    92.66 % |  84% ( 109) |   0% (   0) |    0.844 |   0.8644 |   219.3\n",
            "Copyright_infring |     114 |   1.28 +  21.16 =  22.44 ms |     98.0 % |  94% (  50) |  98% (  64) |   0.9649 |   0.9559 |   221.1\n",
            "Greece            |     108 |   1.05 +  22.75 =  23.81 ms |    99.07 % |  91% ( 108) |   0% (   0) |   0.9167 |   0.9074 |   222.3\n",
            "Royal_Dutch_Shell |      78 |   1.31 +  22.37 =  23.67 ms |    86.84 % |  86% (  38) |  90% (  40) |   0.8846 |   0.8682 |   222.2\n",
            "Mammal            |      67 |   1.61 +  23.95 =  25.55 ms |    100.0 % |  93% (  32) |  68% (  35) |    0.806 |   0.7473 |   211.4\n",
            "East_India_Compan |      97 |   1.46 +  21.15 =  22.61 ms |    96.55 % |  89% (  58) |  94% (  39) |   0.9175 |   0.9105 |   218.5\n",
            "Hokkien           |      70 |   1.24 +   25.4 =  26.64 ms |    96.43 % |  89% (  28) |  78% (  42) |   0.8286 |   0.8381 |   214.5\n",
            "Professional_wres |      79 |   1.25 +  20.41 =  21.66 ms |    91.14 % |  84% (  79) |   0% (   0) |   0.8481 |   0.7704 |   222.4\n",
            "Film_speed        |      62 |   1.27 +  23.28 =  24.56 ms |     88.0 % |  96% (  25) |  89% (  37) |   0.9194 |   0.8591 |   224.3\n",
            "Mexico_City       |     111 |   1.06 +  24.27 =  25.33 ms |     95.5 % |  89% ( 111) |   0% (   0) |   0.8919 |   0.8613 |   218.8\n",
            "Napoleon          |     121 |   1.57 +  20.46 =  22.04 ms |    97.52 % |  93% ( 121) |   0% (   0) |   0.9339 |   0.9559 |   210.9\n",
            "Germans           |     104 |   1.18 +  21.03 =  22.21 ms |    98.57 % |  97% (  70) |  94% (  34) |   0.9615 |   0.9011 |   219.5\n",
            "Southeast_Asia    |      66 |    1.4 +   20.4 =   21.8 ms |    100.0 % |  96% (  28) |  94% (  38) |   0.9545 |   0.9419 |   213.1\n",
            "Brigham_Young_Uni |     108 |   1.47 +   20.5 =  21.97 ms |    94.29 % |  91% (  70) |  97% (  38) |   0.9352 |   0.9354 |   209.2\n",
            "Intellectual_prop |      64 |   1.94 +  22.93 =  24.87 ms |    100.0 % | 100% (  32) |  90% (  32) |   0.9531 |   0.9385 |   225.8\n",
            "Florida           |      91 |   0.98 +  20.51 =  21.49 ms |    89.74 % |  84% (  39) |  98% (  52) |   0.9231 |   0.8402 |   207.5\n",
            "Queen_(band)      |      77 |   1.14 +  20.44 =  21.58 ms |    88.31 % |  75% (  77) |   0% (   0) |   0.7532 |   0.7987 |   215.0\n",
            "Presbyterianism   |      98 |   1.27 +  20.64 =  21.91 ms |    93.75 % |  87% (  32) |  96% (  66) |   0.9388 |   0.9259 |   219.4\n",
            "Thuringia         |      93 |   1.04 +  24.06 =  25.09 ms |    93.75 % |  81% (  48) |  91% (  45) |   0.8602 |    0.824 |   214.2\n",
            "Predation         |      68 |   1.62 +  21.33 =  22.95 ms |    91.89 % |  83% (  37) |  90% (  31) |   0.8676 |   0.8539 |   211.0\n",
            "British_Empire    |      81 |   1.07 +  21.02 =  22.09 ms |    98.77 % |  93% (  81) |   0% (   0) |   0.9383 |   0.9095 |   219.4\n",
            "Botany            |      63 |    1.0 +   21.8 =   22.8 ms |    92.06 % |  84% (  63) |   0% (   0) |   0.8413 |   0.8276 |   212.5\n",
            "Madonna_(entertai |     105 |   1.24 +  20.33 =  21.57 ms |    94.29 % |  77% ( 105) |   0% (   0) |   0.7714 |   0.8043 |   205.1\n",
            "Law_of_the_United |      99 |   1.53 +  20.36 =  21.89 ms |    100.0 % |  96% (  59) |  92% (  40) |   0.9495 |   0.9153 |   224.7\n",
            "Myanmar           |     101 |   1.29 +  22.88 =  24.16 ms |    83.17 % |  76% ( 101) |   0% (   0) |   0.7624 |    0.607 |   211.3\n",
            "Jews              |      82 |   1.48 +  23.34 =  24.82 ms |    97.06 % |  97% (  34) |  93% (  48) |   0.9512 |   0.9181 |   222.6\n",
            "Cotton            |     107 |   1.27 +  20.52 =  21.79 ms |    98.21 % |  85% (  56) |  90% (  51) |   0.8785 |   0.8697 |   214.5\n",
            "Data_compression  |      68 |   1.37 +  20.33 =   21.7 ms |     96.3 % |  92% (  27) |  92% (  41) |   0.9265 |   0.9127 |   206.5\n",
            "The_Sun_(United_K |      79 |   1.17 +  20.84 =  22.01 ms |     96.2 % |  93% (  79) |   0% (   0) |   0.9367 |    0.902 |   225.1\n",
            "Pesticide         |      79 |   1.29 +  21.44 =  22.73 ms |    83.78 % |  81% (  37) |  95% (  42) |   0.8861 |   0.8769 |   196.3\n",
            "Somerset          |     102 |   1.09 +  22.09 =  23.18 ms |    95.45 % |  84% (  44) |  96% (  58) |   0.9118 |   0.8047 |   214.0\n",
            "Yale_University   |     139 |   1.09 +  21.39 =  22.48 ms |    97.18 % |  91% (  71) |  98% (  68) |   0.9496 |   0.9254 |   221.7\n",
            "Late_Middle_Ages  |     121 |   1.35 +  20.59 =  21.94 ms |    100.0 % |  98% (  58) |  93% (  63) |   0.9587 |   0.9575 |   218.0\n",
            "Ann_Arbor,_Michig |      63 |   1.16 +  20.94 =   22.1 ms |    95.83 % |  91% (  24) |  84% (  39) |    0.873 |   0.8651 |   209.8\n",
            "Gothic_architectu |     113 |   1.26 +  21.97 =  23.23 ms |    91.18 % |  89% (  68) |  95% (  45) |   0.9204 |   0.8808 |   221.5\n",
            "Cubism            |      93 |   1.48 +  25.44 =  26.92 ms |    94.44 % |  97% (  36) |  92% (  57) |   0.9462 |   0.8967 |   224.3\n",
            "Political_philoso |      59 |   1.27 +  21.35 =  22.62 ms |    100.0 % |  93% (  16) |  88% (  43) |   0.8983 |   0.9153 |   215.7\n",
            "Norfolk_Island    |      89 |    1.3 +  21.26 =  22.55 ms |    93.75 % |  87% (  48) |  97% (  41) |   0.9213 |   0.9086 |   218.6\n",
            "Edmund_Burke      |     124 |   1.06 +  20.76 =  21.82 ms |    83.33 % |  81% (  60) |  89% (  64) |   0.8548 |   0.8385 |   221.9\n",
            "Samoa             |      73 |   1.43 +  21.74 =  23.17 ms |    100.0 % | 100% (  36) |  97% (  37) |   0.9863 |   0.9817 |   205.8\n",
            "Pope_Paul_VI      |      74 |   1.42 +  22.21 =  23.63 ms |    89.19 % |  81% (  74) |   0% (   0) |   0.8108 |   0.7718 |   221.0\n",
            "Switzerland       |     110 |    1.3 +   21.8 =  23.11 ms |    99.09 % |  95% ( 110) |   0% (   0) |   0.9545 |   0.9612 |   213.1\n",
            "Mali              |      61 |   1.21 +  20.49 =   21.7 ms |    100.0 % |  97% (  35) | 100% (  26) |   0.9836 |   0.9591 |   210.2\n",
            "Raleigh,_North_Ca |      98 |   0.91 +  20.69 =   21.6 ms |    100.0 % |  88% (  50) |  83% (  48) |   0.8571 |    0.804 |   212.1\n",
            "Crimean_War       |      71 |   1.32 +  20.44 =  21.76 ms |    94.37 % |  94% (  71) |   0% (   0) |   0.9437 |   0.9117 |   212.0\n",
            "Nonprofit_organiz |      64 |   1.33 +  21.14 =  22.46 ms |    91.18 % |  91% (  34) |  90% (  30) |   0.9062 |   0.8227 |   215.6\n",
            "Literature        |     102 |   1.38 +  22.32 =   23.7 ms |    96.97 % |  93% (  33) |  95% (  69) |    0.951 |   0.9359 |   217.8\n",
            "Avicenna          |     120 |   1.15 +  23.86 =  25.01 ms |    85.37 % |  82% (  41) |  94% (  79) |   0.9083 |   0.8979 |   216.1\n",
            "Nigeria           |      74 |   1.08 +   21.5 =  22.58 ms |     97.3 % |  87% (  74) |   0% (   0) |   0.8784 |   0.9005 |   212.3\n",
            "Molotov%E2%80%93R |     111 |   1.41 +  21.41 =  22.82 ms |    79.17 % |  75% (  48) |  92% (  63) |   0.8468 |   0.8153 |   220.7\n",
            "History_of_scienc |     107 |   1.09 +  22.97 =  24.06 ms |    99.07 % |  93% ( 107) |   0% (   0) |   0.9346 |   0.8983 |   218.4\n",
            "Digimon           |      56 |   1.32 +  24.52 =  25.84 ms |     75.0 % |  75% (  32) |  79% (  24) |   0.7679 |    0.736 |   220.4\n",
            "Glacier           |      54 |    1.2 +  20.34 =  21.53 ms |    100.0 % |  95% (  24) |  96% (  30) |    0.963 |   0.9523 |   197.4\n",
            "Affirmative_actio |     134 |   1.29 +   20.3 =  21.59 ms |    90.77 % |  86% (  65) |  97% (  69) |   0.9179 |   0.8977 |   219.1\n",
            "FA_Cup            |      86 |   1.02 +  20.48 =   21.5 ms |    80.43 % |  67% (  46) |  82% (  40) |   0.7442 |   0.6609 |   219.6\n",
            "New_Haven,_Connec |     167 |   1.42 +  22.76 =  24.18 ms |     98.8 % |  97% ( 167) |   0% (   0) |   0.9701 |   0.9078 |   219.8\n",
            "Alsace            |      67 |   1.33 +  21.76 =  23.09 ms |    97.22 % |  88% (  36) |  87% (  31) |   0.8806 |   0.8568 |   221.2\n",
            "Carnival          |      67 |   1.32 +   21.0 =  22.31 ms |    89.55 % |  83% (  67) |   0% (   0) |   0.8358 |   0.8068 |   203.7\n",
            "Baptists          |      57 |   1.65 +  20.43 =  22.08 ms |    100.0 % | 100% (  29) | 100% (  28) |      1.0 |   0.9848 |   206.3\n",
            "Child_labour      |      76 |   1.13 +  19.67 =   20.8 ms |    90.79 % |  86% (  76) |   0% (   0) |   0.8684 |   0.8011 |   210.1\n",
            "Dissolution_of_th |      78 |   1.14 +  22.22 =  23.36 ms |    93.59 % |  84% (  78) |   0% (   0) |   0.8462 |   0.8237 |   220.5\n",
            "Crucifixion_of_Je |      97 |   1.32 +  23.07 =  24.39 ms |    88.46 % |  76% (  52) |  86% (  45) |   0.8144 |   0.8069 |   220.9\n",
            "Supreme_court     |      69 |   1.32 +  20.95 =  22.27 ms |    85.37 % |  75% (  41) | 100% (  28) |   0.8551 |   0.8679 |   218.1\n",
            "Textual_criticism |      81 |   1.23 +   20.7 =  21.92 ms |    81.08 % |  62% (  37) |  86% (  44) |   0.7531 |    0.696 |   220.5\n",
            "Gramophone_record |     119 |    1.1 +  21.01 =  22.11 ms |    84.03 % |  83% ( 119) |   0% (   0) |   0.8319 |   0.7348 |   220.2\n",
            "Turner_Classic_Mo |      58 |   1.32 +  22.84 =  24.16 ms |    100.0 % |  88% (  27) |  90% (  31) |   0.8966 |   0.9138 |   230.3\n",
            "Hindu_philosophy  |      85 |   1.11 +   26.2 =  27.31 ms |    95.12 % |  87% (  41) |  90% (  44) |   0.8941 |   0.8988 |   203.2\n",
            "A_cappella        |      80 |   1.32 +  23.69 =  25.01 ms |    96.43 % |  96% (  28) |  90% (  52) |    0.925 |   0.9196 |   218.9\n",
            "Dominican_Order   |     136 |   1.12 +  20.44 =  21.56 ms |    84.51 % |  74% (  71) |  90% (  65) |   0.8235 |   0.8027 |   216.0\n",
            "Eton_College      |     125 |   1.07 +   21.8 =  22.88 ms |    90.91 % |  92% (  55) |  92% (  70) |    0.928 |    0.904 |   222.3\n",
            "Cork_(city)       |     121 |   1.04 +  20.37 =  21.41 ms |    92.68 % |  90% (  41) |  87% (  80) |   0.8843 |   0.8825 |   206.5\n",
            "Galicia_(Spain)   |      46 |   1.32 +   28.6 =  29.92 ms |    93.48 % |  82% (  46) |   0% (   0) |   0.8261 |   0.8225 |   224.0\n",
            "USB               |      62 |   1.51 +  23.84 =  25.36 ms |    93.55 % |  95% (  62) |   0% (   0) |   0.9516 |   0.9126 |   216.1\n",
            "Sichuan           |      95 |   1.14 +  21.13 =  22.27 ms |    100.0 % | 100% (  27) |  94% (  68) |   0.9579 |   0.9358 |   213.5\n",
            "Unicode           |      92 |    1.1 +  22.72 =  23.82 ms |    89.58 % |  91% (  48) |  90% (  44) |    0.913 |   0.8891 |   218.3\n",
            "Detroit           |     106 |   1.03 +  20.86 =  21.89 ms |    100.0 % |  95% ( 106) |   0% (   0) |   0.9528 |   0.9572 |   212.4\n",
            "London            |     104 |    1.4 +  22.15 =  23.55 ms |    100.0 % |  94% ( 104) |   0% (   0) |   0.9423 |   0.9399 |   219.3\n",
            "Culture           |      45 |   1.56 +   23.8 =  25.36 ms |    86.96 % |  73% (  23) | 100% (  22) |   0.8667 |   0.8804 |   218.8\n",
            "Sahara            |      76 |    1.1 +  21.31 =  22.41 ms |    92.59 % |  81% (  27) |  75% (  49) |   0.7763 |   0.7577 |   214.8\n",
            "Rule_of_law       |      60 |   1.31 +  20.82 =  22.13 ms |    96.88 % |  96% (  32) |  89% (  28) |   0.9333 |   0.9144 |   223.5\n",
            "Exhibition_game   |      45 |    1.2 +  20.41 =  21.61 ms |    100.0 % |  95% (  22) |  91% (  23) |   0.9333 |   0.9333 |   221.7\n",
            "Northwestern_Univ |     133 |   1.37 +   22.6 =  23.97 ms |    95.24 % |  95% (  63) |  97% (  70) |   0.9624 |   0.9622 |   219.6\n",
            "Strasbourg        |     104 |   1.06 +  24.19 =  25.25 ms |    100.0 % |  91% (  58) |  89% (  46) |   0.9038 |   0.8942 |   221.3\n",
            "History_of_India  |     117 |   1.24 +  22.04 =  23.28 ms |    96.58 % |  90% ( 117) |   0% (   0) |    0.906 |    0.894 |   218.8\n",
            "Gamal_Abdel_Nasse |     137 |   1.15 +  21.06 =  22.21 ms |    86.13 % |  71% ( 137) |   0% (   0) |   0.7153 |   0.7224 |   213.9\n",
            "Pope_John_XXIII   |      66 |   1.26 +  20.47 =  21.73 ms |    100.0 % | 100% (  31) |  88% (  35) |   0.9394 |   0.9302 |   219.5\n",
            "Time              |     121 |   1.21 +  21.22 =  22.43 ms |    97.37 % |  94% (  38) |  93% (  83) |   0.9421 |   0.9268 |   217.7\n",
            "St._John%27s,_New |      80 |   1.25 +  23.69 =  24.94 ms |    98.04 % |  90% (  51) |  82% (  29) |    0.875 |   0.8792 |   221.1\n",
            "John_von_Neumann  |      55 |   1.46 +  21.83 =   23.3 ms |    87.27 % |  83% (  55) |   0% (   0) |   0.8364 |   0.7415 |   222.8\n",
            "PlayStation_3     |     166 |   1.44 +   21.1 =  22.54 ms |    98.57 % |  91% (  70) |  94% (  96) |   0.9337 |   0.9228 |   219.2\n",
            "Royal_assent      |     112 |   1.25 +  20.64 =  21.89 ms |    90.32 % |  79% (  62) |  92% (  50) |   0.8482 |   0.8227 |   222.4\n",
            "Central_African_R |      62 |   1.16 +  22.63 =  23.79 ms |    93.75 % |  84% (  32) |  96% (  30) |   0.9032 |   0.9038 |   221.6\n",
            "Asthma            |      76 |   1.46 +  21.95 =  23.41 ms |    92.11 % |  84% (  38) |  94% (  38) |   0.8947 |    0.855 |   192.4\n",
            "LaserDisc         |      61 |    1.3 +  22.87 =  24.18 ms |     91.8 % |  86% (  61) |   0% (   0) |   0.8689 |   0.8429 |   222.0\n",
            "George_VI         |      62 |   1.04 +  20.46 =  21.51 ms |    100.0 % |  96% (  30) |  93% (  32) |   0.9516 |   0.9447 |   217.8\n",
            "Federalism        |     101 |   1.02 +  20.41 =  21.43 ms |    93.48 % |  93% (  46) |  94% (  55) |   0.9406 |   0.8226 |   214.9\n",
            "Annelid           |      74 |   1.24 +  24.47 =  25.71 ms |    97.62 % |  88% (  42) |  96% (  32) |   0.9189 |   0.9334 |   214.9\n",
            "War_on_Terror     |     104 |   1.18 +  21.85 =  23.03 ms |    94.59 % |  89% (  37) |  94% (  67) |   0.9231 |   0.9127 |   220.7\n",
            "Labour_Party_(UK) |      90 |   0.99 +  20.86 =  21.84 ms |    88.46 % |  80% (  26) |  92% (  64) |   0.8889 |   0.8687 |   221.9\n",
            "Estonia           |      93 |   1.24 +  20.89 =  22.13 ms |    96.77 % |  90% (  93) |   0% (   0) |   0.9032 |   0.9116 |   208.8\n",
            "Alaska            |     133 |   1.27 +  20.22 =   21.5 ms |    98.46 % |  95% (  65) | 100% (  68) |   0.9774 |   0.9678 |   210.7\n",
            "Mandolin          |     132 |   1.07 +  22.16 =  23.24 ms |    80.33 % |  77% (  61) |  94% (  71) |   0.8636 |    0.795 |   214.5\n",
            "Insect            |     112 |   1.32 +   23.7 =  25.02 ms |    96.43 % |  92% ( 112) |   0% (   0) |   0.9286 |   0.9116 |   201.6\n",
            "Race_(human_categ |     104 |   1.32 +  21.24 =  22.57 ms |    97.12 % |  96% ( 104) |   0% (   0) |   0.9615 |   0.9162 |   221.8\n",
            "Paris             |     124 |   1.19 +  24.02 =  25.21 ms |    97.58 % |  96% ( 124) |   0% (   0) |   0.9677 |   0.9144 |   223.3\n",
            "Apollo            |      66 |   1.48 +  34.85 =  36.33 ms |    93.94 % |  90% (  66) |   0% (   0) |   0.9091 |   0.8606 |   216.2\n",
            "United_States_pre |      72 |    1.4 +  21.37 =  22.77 ms |    97.06 % |  88% (  34) |  92% (  38) |   0.9028 |   0.8986 |   215.0\n",
            "Liberal_Party_of_ |      77 |   1.36 +  22.59 =  23.95 ms |    93.55 % |  93% (  31) |  93% (  46) |   0.9351 |   0.9329 |   224.2\n",
            "Samurai           |      74 |   1.24 +  21.94 =  23.19 ms |    94.59 % |  89% (  74) |   0% (   0) |   0.8919 |   0.8698 |   211.5\n",
            "Software_testing  |      60 |   1.42 +  20.28 =  21.69 ms |    90.32 % |  74% (  31) |  89% (  29) |   0.8167 |   0.7703 |   214.5\n",
            "States_of_Germany |      79 |   1.41 +  24.12 =  25.53 ms |    93.55 % |  90% (  31) |  95% (  48) |   0.9367 |   0.8987 |   222.2\n",
            "Glass             |      74 |   1.18 +   20.0 =  21.18 ms |    93.75 % |  87% (  32) |  92% (  42) |   0.9054 |   0.8716 |   199.2\n",
            "Renewable_energy_ |      95 |   1.37 +  22.25 =  23.62 ms |    100.0 % |  98% (  52) |  95% (  43) |   0.9684 |   0.9461 |   218.0\n",
            "Palermo           |      68 |   1.32 +  22.46 =  23.78 ms |    91.18 % |  88% (  34) |  91% (  34) |   0.8971 |   0.8649 |   202.1\n",
            "Green             |      72 |   1.49 +  21.24 =  22.73 ms |    100.0 % | 100% (  36) |  94% (  36) |   0.9722 |   0.9663 |   212.8\n",
            "Neoclassical_arch |      61 |   1.33 +  22.44 =  23.77 ms |    94.29 % |  77% (  35) |  92% (  26) |   0.8361 |   0.8607 |   222.0\n",
            "Serbo-Croatian    |      53 |   1.36 +  24.96 =  26.32 ms |    100.0 % | 100% (  16) |  91% (  37) |   0.9434 |   0.9413 |   222.2\n",
            "CBC_Television    |      58 |   1.36 +  21.93 =  23.28 ms |    89.66 % |  86% (  29) |  89% (  29) |   0.8793 |   0.8667 |   226.1\n",
            "IBM               |      74 |   1.53 +  22.24 =  23.77 ms |    95.12 % |  92% (  41) | 100% (  33) |   0.9595 |   0.9302 |   216.3\n",
            "Energy            |      80 |   1.62 +  21.92 =  23.54 ms |    96.97 % |  96% (  33) | 100% (  47) |   0.9875 |   0.9786 |   219.9\n",
            "East_Prussia      |      60 |   1.28 +  23.05 =  24.32 ms |    90.48 % |  90% (  21) |  76% (  39) |   0.8167 |   0.8082 |   215.8\n",
            "Ottoman_Empire    |     124 |   1.17 +   21.3 =  22.47 ms |    99.19 % |  95% ( 124) |   0% (   0) |   0.9597 |   0.9377 |   217.6\n",
            "Philosophy_of_spa |      63 |    1.4 +  21.04 =  22.44 ms |    96.97 % |  96% (  33) | 100% (  30) |   0.9841 |   0.9577 |   221.7\n",
            "Neolithic         |      55 |   1.09 +  21.44 =  22.52 ms |    96.77 % |  90% (  31) |  95% (  24) |   0.9273 |   0.9345 |   211.2\n",
            "Friedrich_Hayek   |      79 |   1.38 +  22.58 =  23.96 ms |    83.54 % |  75% (  79) |   0% (   0) |   0.7595 |   0.7379 |   222.6\n",
            "Diarrhea          |      54 |    1.3 +  21.43 =  22.73 ms |    76.19 % |  71% (  21) |  87% (  33) |   0.8148 |   0.7771 |   199.5\n",
            "Madrasa           |      99 |   1.49 +  23.92 =  25.41 ms |    80.39 % |  68% (  51) |  97% (  48) |   0.8283 |   0.8139 |   216.7\n",
            "Miami             |     129 |   1.35 +   21.4 =  22.75 ms |    100.0 % |  96% (  63) |  92% (  66) |   0.9457 |   0.9468 |   214.6\n",
            "Philadelphia      |      59 |   1.04 +  21.87 =  22.91 ms |    100.0 % |  96% (  59) |   0% (   0) |   0.9661 |    0.916 |   210.9\n",
            "John_Kerry        |      71 |   1.07 +  20.78 =  21.85 ms |    92.96 % |  90% (  71) |   0% (   0) |   0.9014 |    0.883 |   220.7\n",
            "Rajasthan         |      67 |   1.27 +  22.16 =  23.42 ms |    100.0 % |  97% (  34) |  90% (  33) |   0.9403 |   0.9147 |   195.1\n",
            "Guam              |      74 |   1.34 +  22.39 =  23.73 ms |    91.43 % |  88% (  35) |  89% (  39) |   0.8919 |   0.8648 |   201.6\n",
            "Empiricism        |      68 |   1.47 +  24.19 =  25.67 ms |    90.32 % |  93% (  31) |  94% (  37) |   0.9412 |   0.9118 |   219.9\n",
            "Idealism          |      79 |   1.17 +   23.1 =  24.27 ms |     95.0 % |  87% (  40) |  97% (  39) |   0.9241 |   0.9249 |   217.2\n",
            "Czech_language    |      93 |   1.31 +  21.84 =  23.15 ms |    97.83 % |  93% (  46) |  93% (  47) |   0.9355 |   0.9297 |   210.0\n",
            "Education         |      83 |   1.22 +  20.05 =  21.27 ms |    87.88 % |  87% (  33) |  94% (  50) |   0.9157 |    0.894 |   207.3\n",
            "Tennessee         |      60 |   1.27 +  20.89 =  22.16 ms |    100.0 % |  96% (  60) |   0% (   0) |   0.9667 |     0.93 |   215.3\n",
            "Canadian_football |      63 |   1.43 +  22.83 =  24.25 ms |    96.43 % |  89% (  28) |  91% (  35) |   0.9048 |   0.9365 |   221.6\n",
            "Seven_Years%27_Wa |      82 |   1.14 +  21.61 =  22.75 ms |    81.71 % |  75% (  82) |   0% (   0) |   0.7561 |   0.5967 |   216.8\n",
            "Cyprus            |      76 |   1.12 +  22.74 =  23.86 ms |    98.68 % |  90% (  76) |   0% (   0) |   0.9079 |   0.8611 |   218.8\n",
            "Steven_Spielberg  |     148 |   1.16 +  20.81 =  21.97 ms |    92.11 % |  82% (  76) |  70% (  72) |   0.7703 |   0.7599 |   216.0\n",
            "Elevator          |      81 |   1.33 +  20.26 =  21.59 ms |    93.83 % |  88% (  81) |   0% (   0) |   0.8889 |   0.8257 |   210.4\n",
            "Railway_electrifi |      81 |   1.27 +  21.43 =   22.7 ms |    85.11 % |  89% (  47) |  97% (  34) |   0.9259 |   0.8261 |   220.6\n",
            "Spanish_language_ |      70 |   1.33 +  21.95 =  23.27 ms |     92.0 % |  88% (  25) |  75% (  45) |      0.8 |   0.6659 |   221.8\n",
            "Charleston,_South |     147 |   1.24 +  20.62 =  21.86 ms |    100.0 % |  94% (  78) |  97% (  69) |   0.9592 |   0.9573 |   214.8\n",
            "The_Blitz         |     120 |   1.18 +  20.35 =  21.54 ms |     97.5 % |  91% ( 120) |   0% (   0) |   0.9167 |   0.8551 |   212.4\n",
            "Vacuum            |      80 |   1.16 +  20.33 =  21.49 ms |    97.22 % |  97% (  36) |  93% (  44) |     0.95 |   0.9204 |   206.3\n",
            "Quran             |     125 |   1.31 +  24.44 =  25.75 ms |    90.57 % |  84% (  53) |  93% (  72) |    0.896 |   0.8651 |   214.8\n",
            "Geography_of_the_ |      47 |    1.2 +  20.71 =  21.91 ms |    100.0 % |  94% (  19) |  96% (  28) |   0.9574 |   0.9397 |   217.8\n",
            "Compact_disc      |      80 |   1.26 +  20.26 =  21.53 ms |    91.43 % |  88% (  35) |  73% (  45) |      0.8 |   0.8003 |   210.6\n",
            "Modern_history    |     114 |   1.05 +  20.35 =   21.4 ms |    98.25 % |  95% ( 114) |   0% (   0) |   0.9561 |   0.9365 |   215.1\n",
            "Flowering_plant   |      79 |   1.26 +  23.73 =  24.99 ms |    97.67 % |  88% (  43) |  91% (  36) |   0.8987 |   0.8785 |   214.1\n",
            "Hyderabad         |      81 |   1.41 +  24.44 =  25.85 ms |    100.0 % |  87% (  81) |   0% (   0) |   0.8765 |   0.9116 |   217.5\n",
            "Santa_Monica,_Cal |      68 |   1.64 +  22.06 =   23.7 ms |    100.0 % |  97% (  38) |  80% (  30) |   0.8971 |   0.8971 |   205.0\n",
            "Pain              |      91 |   1.12 +  21.97 =  23.09 ms |    96.15 % |  84% (  52) |  94% (  39) |   0.8901 |    0.879 |   218.5\n",
            "Database          |      71 |   1.19 +   20.3 =   21.5 ms |    88.57 % |  85% (  35) |  94% (  36) |   0.9014 |   0.8617 |   210.6\n",
            "Tucson,_Arizona   |     111 |   1.04 +  20.95 =  21.99 ms |     98.2 % |  93% ( 111) |   0% (   0) |   0.9369 |    0.889 |   211.7\n",
            "Bacteria          |      66 |   1.27 +  22.08 =  23.35 ms |    93.94 % |  89% (  66) |   0% (   0) |   0.8939 |   0.8717 |   204.1\n",
            "Printed_circuit_b |     105 |   1.45 +  22.06 =  23.52 ms |    98.21 % |  92% (  56) |  95% (  49) |   0.9429 |   0.9339 |   198.2\n",
            "Greeks            |     123 |    1.4 +  21.71 =  23.11 ms |    82.76 % |  55% (  58) |  95% (  65) |   0.7642 |   0.6863 |   217.1\n",
            "Premier_League    |     179 |   1.38 +  20.42 =   21.8 ms |    98.32 % |  90% ( 119) | 100% (  60) |   0.9385 |    0.855 |   220.9\n",
            "Roman_Republic    |     118 |   1.16 +  20.43 =  21.59 ms |    93.22 % |  88% ( 118) |   0% (   0) |   0.8898 |    0.833 |   195.2\n",
            "Pacific_War       |     134 |   1.32 +  21.91 =  23.24 ms |    95.52 % |  91% ( 134) |   0% (   0) |   0.9179 |   0.8818 |   215.8\n",
            "San_Diego         |     147 |   1.22 +  22.56 =  23.78 ms |    97.18 % |  90% (  71) |  93% (  76) |   0.9184 |   0.9216 |   217.3\n",
            "Iran              |      92 |   1.44 +  21.78 =  23.22 ms |    100.0 % |  96% (  92) |   0% (   0) |   0.9674 |   0.9475 |   220.3\n",
            "British_Isles     |     115 |   1.32 +  21.94 =  23.26 ms |    92.59 % |  96% (  54) |  93% (  61) |   0.9478 |   0.8984 |   216.2\n",
            "Association_footb |      67 |   1.37 +  20.91 =  22.28 ms |    100.0 % |  97% (  38) | 100% (  29) |   0.9851 |   0.9851 |   213.4\n",
            "Georgian_architec |      49 |   1.31 +   20.9 =  22.21 ms |     96.3 % |  96% (  27) |  90% (  22) |   0.9388 |   0.9209 |   219.3\n",
            "Liberia           |      81 |   1.11 +  20.48 =  21.59 ms |    97.73 % |  95% (  44) |  86% (  37) |   0.9136 |   0.9023 |   210.7\n",
            "Alfred_North_Whit |     149 |   1.36 +  20.48 =  21.84 ms |    94.06 % |  87% ( 101) |  97% (  48) |    0.906 |   0.8478 |   214.0\n",
            "Antibiotics       |      88 |    1.1 +  22.16 =  23.27 ms |    87.23 % |  87% (  47) |  85% (  41) |   0.8636 |     0.83 |   210.5\n",
            "Windows_8         |     141 |   1.13 +  22.47 =  23.61 ms |    96.61 % |  84% (  59) |  95% (  82) |   0.9078 |   0.8886 |   221.4\n",
            "Swaziland         |      59 |   1.51 +  20.81 =  22.32 ms |     88.0 % |  92% (  25) |  76% (  34) |   0.8305 |   0.7891 |   202.5\n",
            "Translation       |      80 |    1.4 +  22.95 =  24.35 ms |    93.18 % |  95% (  44) |  94% (  36) |     0.95 |   0.9217 |   220.6\n",
            "Airport           |      65 |   1.41 +  18.15 =  19.56 ms |    97.06 % |  97% (  34) |  80% (  31) |   0.8923 |   0.8703 |   192.5\n",
            "Super_Nintendo_En |      75 |    1.2 +  22.32 =  23.52 ms |    93.55 % |  90% (  31) |  93% (  44) |     0.92 |   0.8966 |   223.8\n",
            "Sumer             |      80 |   1.15 +  22.08 =  23.24 ms |     92.0 % |  90% (  50) |  90% (  30) |      0.9 |   0.8707 |   218.2\n",
            "Tuvalu            |      94 |   1.27 +  24.29 =  25.56 ms |    94.68 % |  89% (  94) |   0% (   0) |   0.8936 |   0.8809 |   219.0\n",
            "Namibia           |     109 |   1.16 +  21.34 =   22.5 ms |    100.0 % |  90% (  55) |  87% (  54) |   0.8899 |   0.8777 |   213.9\n",
            "Russian_language  |     101 |   1.33 +  24.67 =   26.0 ms |    96.08 % |  84% (  51) |  96% (  50) |    0.901 |   0.8709 |   214.9\n",
            "United_States_Air |      77 |    1.2 +  21.84 =  23.04 ms |    94.81 % |  96% (  77) |   0% (   0) |    0.961 |    0.922 |   220.3\n",
            "Light-emitting_di |     122 |   1.33 +  22.01 =  23.34 ms |     87.5 % |  85% (  56) |  95% (  66) |   0.9098 |    0.892 |   209.7\n",
            "Great_power       |      53 |   1.33 +  21.83 =  23.16 ms |    100.0 % |  90% (  22) |  77% (  31) |   0.8302 |   0.8152 |   222.5\n",
            "Bird              |      63 |    1.3 +  22.08 =  23.38 ms |    98.41 % |  96% (  63) |   0% (   0) |   0.9683 |    0.958 |   202.0\n",
            "Qing_dynasty      |      97 |   1.04 +  21.11 =  22.15 ms |    89.69 % |  70% (  97) |   0% (   0) |    0.701 |   0.6923 |   220.8\n",
            "Indigenous_people |      84 |   1.25 +  23.45 =   24.7 ms |    97.62 % |  94% (  84) |   0% (   0) |   0.9405 |   0.9192 |   217.8\n",
            "Red               |     130 |   1.33 +   20.3 =  21.63 ms |    94.52 % |  90% (  73) |  87% (  57) |   0.8923 |   0.8703 |   211.1\n",
            "Mosaic            |      86 |   1.49 +  23.46 =  24.95 ms |    88.37 % |  87% (  86) |   0% (   0) |   0.8721 |   0.8411 |   218.0\n",
            "University        |     100 |    1.4 +   20.3 =  21.71 ms |    89.47 % |  86% (  38) |  93% (  62) |     0.91 |   0.9067 |   219.4\n",
            "Religion_in_ancie |     122 |   1.12 +  21.51 =  22.62 ms |    90.16 % |  81% ( 122) |   0% (   0) |   0.8197 |   0.8182 |   218.8\n",
            "YouTube           |      89 |   1.32 +  20.67 =  21.99 ms |    94.87 % |  89% (  39) |  94% (  50) |   0.9213 |    0.903 |   219.5\n",
            "Separation_of_chu |     130 |   1.34 +  22.13 =  23.47 ms |    100.0 % |  95% (  65) | 100% (  65) |   0.9769 |   0.9685 |   226.9\n",
            "Bras%C3%ADlia     |      65 |    1.3 +  22.14 =  23.44 ms |    100.0 % |  93% (  32) |  81% (  33) |   0.8769 |   0.8575 |   202.4\n",
            "Economy_of_Greece |      90 |   1.25 +  22.14 =  23.39 ms |    94.74 % |  97% (  38) |  92% (  52) |   0.9444 |   0.9321 |   219.7\n",
            "Party_leaders_of_ |      79 |   1.32 +  21.49 =  22.81 ms |    88.24 % |  76% (  34) |  93% (  45) |   0.8608 |    0.819 |   227.3\n",
            "Armenians         |      66 |   1.32 +  22.06 =  23.38 ms |    96.77 % |  93% (  31) |  94% (  35) |   0.9394 |   0.9404 |   220.6\n",
            "Jehovah%27s_Witne |     128 |   1.37 +  22.62 =  23.99 ms |    96.83 % |  95% (  63) |  93% (  65) |   0.9453 |   0.9447 |   218.8\n",
            "Dwight_D._Eisenho |     124 |   1.29 +  21.37 =  22.66 ms |    94.35 % |  89% ( 124) |   0% (   0) |   0.8952 |   0.9125 |   218.9\n",
            "The_Bronx         |      85 |   1.06 +  23.13 =  24.19 ms |    97.65 % |  94% (  85) |   0% (   0) |   0.9412 |    0.912 |   217.3\n",
            "Financial_crisis_ |     111 |   1.62 +  22.09 =  23.71 ms |     95.5 % |  94% ( 111) |   0% (   0) |   0.9459 |   0.9356 |   221.6\n",
            "Portugal          |     119 |   1.23 +   22.6 =  23.83 ms |    97.48 % |  94% ( 119) |   0% (   0) |   0.9412 |   0.8984 |   218.8\n",
            "Humanism          |      73 |   1.34 +  23.71 =  25.05 ms |    77.42 % |  67% (  31) |  97% (  42) |   0.8493 |    0.803 |   222.3\n",
            "Geological_histor |      85 |   1.29 +  22.27 =  23.56 ms |    92.16 % |  74% (  51) |  94% (  34) |   0.8235 |   0.8373 |   207.9\n",
            "Police            |      89 |   1.07 +  20.81 =  21.88 ms |    95.24 % |  90% (  42) |  95% (  47) |   0.9326 |   0.8984 |   224.1\n",
            "Genocide          |      86 |   1.52 +  21.35 =  22.88 ms |     95.0 % |  85% (  40) |  93% (  46) |   0.8953 |   0.8941 |   228.7\n",
            "Saint_Barth%C3%A9 |      65 |   1.26 +  21.54 =   22.8 ms |    97.14 % |  97% (  35) |  83% (  30) |   0.9077 |   0.8933 |   203.8\n",
            "Tajikistan        |      83 |   1.06 +  26.13 =  27.19 ms |    100.0 % |  97% (  39) |  97% (  44) |   0.9759 |   0.8847 |   217.6\n",
            "University_of_Not |      71 |   1.43 +  21.71 =  23.14 ms |    94.37 % |  91% (  71) |   0% (   0) |   0.9155 |   0.8991 |   210.8\n",
            "Anthropology      |     127 |   1.18 +  21.07 =  22.25 ms |     97.3 % |  95% (  74) |  88% (  53) |   0.9291 |   0.8927 |   216.4\n",
            "Montana           |      40 |   1.28 +  20.97 =  22.24 ms |    100.0 % |  97% (  40) |   0% (   0) |    0.975 |   0.9524 |   209.8\n",
            "Punjab,_Pakistan  |      73 |   1.08 +  21.27 =  22.34 ms |    100.0 % |  91% (  34) |  92% (  39) |   0.9178 |   0.9148 |   203.6\n",
            "Infection         |      90 |   1.18 +  20.38 =  21.56 ms |    100.0 % | 100% (  39) |  98% (  51) |   0.9889 |   0.9742 |   207.7\n",
            "Hunting           |     155 |   1.09 +  22.36 =  23.45 ms |    95.83 % |  96% (  96) |  94% (  59) |   0.9613 |   0.9385 |   215.9\n",
            "Kathmandu         |      82 |   1.23 +  20.75 =  21.99 ms |    100.0 % |  92% (  82) |   0% (   0) |   0.9268 |   0.9329 |   186.8\n",
            "Myocardial_infarc |      33 |   1.38 +  23.13 =  24.52 ms |      0.0 % |   0% (   0) |  93% (  33) |   0.9394 |   0.9394 |   211.6\n",
            "Matter            |      37 |    1.4 +  22.42 =  23.82 ms |      0.0 % |   0% (   0) |  81% (  37) |   0.8108 |   0.8108 |   216.9\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|--------------\n",
            "Grand Total       |   32401 |   1.27 +  21.86 =  23.13 ms |    94.86 % |  89% (21918) |  92% (10483) |   0.9032 |   0.8822 |   215.3\n"
          ]
        }
      ],
      "source": [
        "# with mpnet - finetuned and electra-best (optimized) - k = 10, context_length_threshold = 205 - old themes"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-4hlF13_jPa9",
        "outputId": "7d8ed8ae-9366-4ed0-f3d9-d01e74b6aa15"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Theme             | Queries |  AIT: (ANN) + (QnA) = Total | ansInCtx % | TP % (TotP) | TN % (TotN) | Para Acc | F1 Score | Avg Ctx Size\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|-------------\n",
            "Beyoncé           |     228 |    1.2 +  20.84 =  22.04 ms |     94.3 % |  88% ( 228) |   0% (   0) |    0.886 |   0.8858 |   216.2\n",
            "Frédéric_Chopin   |     203 |    1.2 +  21.97 =  23.17 ms |     80.3 % |  75% ( 203) |   0% (   0) |   0.7537 |   0.7463 |   210.2\n",
            "Sino-Tibetan_rela |      98 |   1.29 +  23.09 =  24.38 ms |     89.8 % |  83% (  98) |   0% (   0) |   0.8367 |   0.8378 |   220.7\n",
            "The_Legend_of_Zel |     125 |   1.25 +  20.76 =  22.01 ms |    94.74 % |  85% (  76) |  91% (  49) |     0.88 |   0.8594 |   213.4\n",
            "Spectre_(2015_fil |     142 |   1.07 +  21.54 =  22.62 ms |    94.25 % |  81% (  87) |  90% (  55) |   0.8521 |    0.864 |   214.5\n",
            "New_York_City     |     253 |   1.17 +  21.32 =  22.49 ms |    99.21 % |  91% ( 253) |   0% (   0) |    0.917 |   0.9103 |   220.6\n",
            "To_Kill_a_Mocking |      62 |   1.47 +  25.86 =  27.33 ms |    90.32 % |  79% (  62) |   0% (   0) |   0.7903 |   0.7823 |   219.1\n",
            "Solar_energy      |      73 |   1.49 +  20.73 =  22.21 ms |    93.15 % |  94% (  73) |   0% (   0) |   0.9452 |   0.8945 |   214.6\n",
            "Kanye_West        |     148 |   1.26 +  22.23 =  23.49 ms |    89.19 % |  75% ( 148) |   0% (   0) |     0.75 |   0.7968 |   216.2\n",
            "Buddhism          |     189 |   1.23 +  23.89 =  25.12 ms |    88.36 % |  83% ( 189) |   0% (   0) |    0.836 |   0.8107 |   210.0\n",
            "American_Idol     |     265 |   1.13 +  20.73 =  21.86 ms |    91.32 % |  73% ( 265) |   0% (   0) |   0.7321 |   0.7806 |   208.7\n",
            "Dog               |     117 |   1.54 +  21.16 =   22.7 ms |    97.44 % |  95% ( 117) |   0% (   0) |   0.9573 |   0.9052 |   205.5\n",
            "2008_Summer_Olymp |     227 |   1.24 +  22.27 =  23.51 ms |     85.9 % |  78% ( 227) |   0% (   0) |   0.7841 |   0.7729 |   211.8\n",
            "Genome            |      56 |   1.36 +  23.32 =  24.68 ms |    91.67 % |  95% (  24) |  93% (  32) |   0.9464 |   0.8893 |   205.6\n",
            "Comprehensive_sch |      52 |   1.64 +  21.33 =  22.97 ms |    93.75 % |  87% (  16) | 100% (  36) |   0.9615 |   0.9665 |   217.9\n",
            "Republic_of_the_C |     111 |    1.3 +  21.65 =  22.95 ms |    94.64 % |  82% (  56) |  98% (  55) |   0.9009 |   0.9176 |   206.4\n",
            "Prime_minister    |      55 |   1.31 +  21.14 =  22.45 ms |    100.0 % |  88% (  26) |  93% (  29) |   0.9091 |   0.8939 |   219.3\n",
            "Institute_of_tech |      55 |   1.67 +  22.92 =  24.59 ms |    98.18 % |  92% (  55) |   0% (   0) |   0.9273 |   0.9301 |   211.6\n",
            "Dutch_Republic    |      62 |   1.28 +  21.75 =  23.03 ms |    95.83 % |  91% (  24) |  94% (  38) |   0.9355 |   0.8927 |   215.5\n",
            "Symbiosis         |      51 |   1.76 +  24.39 =  26.14 ms |    81.25 % |  75% (  16) |  91% (  35) |   0.8627 |   0.8562 |   203.7\n",
            "Iranian_languages |      40 |   1.36 +  21.01 =  22.37 ms |    84.21 % |  78% (  19) |  95% (  21) |    0.875 |    0.825 |   198.8\n",
            "Lighting          |      42 |    1.5 +  19.44 =  20.94 ms |    95.24 % |  92% (  42) |   0% (   0) |   0.9286 |   0.8283 |   196.4\n",
            "Separation_of_pow |      69 |   1.39 +  20.45 =  21.84 ms |     87.5 % |  66% (  24) |  93% (  45) |   0.8406 |   0.8338 |   209.2\n",
            "Architecture      |     105 |   1.22 +  21.34 =  22.55 ms |    96.49 % |  92% (  57) |  97% (  48) |   0.9524 |   0.9333 |   214.6\n",
            "Southern_Europe   |      66 |   1.35 +  18.53 =  19.88 ms |    100.0 % |  94% (  34) |  93% (  32) |   0.9394 |   0.9336 |   193.5\n",
            "BBC_Television    |      65 |   1.56 +  21.03 =   22.6 ms |    94.29 % |  80% (  35) |  90% (  30) |   0.8462 |   0.8434 |   221.3\n",
            "Arnold_Schwarzene |      56 |   1.57 +  22.59 =  24.16 ms |    89.29 % |  83% (  56) |   0% (   0) |   0.8393 |    0.869 |   219.8\n",
            "Plymouth          |     122 |   1.15 +  20.19 =  21.34 ms |    98.36 % |  92% ( 122) |   0% (   0) |   0.9262 |    0.915 |   208.3\n",
            "Christian         |      62 |   1.45 +  25.95 =   27.4 ms |     90.0 % |  85% (  20) |  85% (  42) |   0.8548 |   0.8387 |   207.2\n",
            "Sony_Music_Entert |      62 |   1.07 +  20.48 =  21.55 ms |     91.3 % |  78% (  23) | 100% (  39) |   0.9194 |   0.8737 |   213.9\n",
            "Oklahoma_City     |      63 |   1.25 +  22.06 =  23.32 ms |    98.41 % |  98% (  63) |   0% (   0) |   0.9841 |   0.9079 |   213.8\n",
            "Hunter-gatherer   |      97 |   1.33 +  22.49 =  23.82 ms |    81.58 % |  71% (  38) |  94% (  59) |   0.8557 |    0.841 |   215.1\n",
            "United_Nations_Po |      68 |   1.63 +  22.76 =  24.38 ms |    91.43 % |  88% (  35) |  84% (  33) |   0.8676 |    0.859 |   224.2\n",
            "Russian_Soviet_Fe |     103 |    1.4 +  23.28 =  24.68 ms |    94.83 % |  81% (  58) |  93% (  45) |   0.8641 |   0.8931 |   220.2\n",
            "Alexander_Graham_ |     105 |   1.11 +   21.5 =  22.61 ms |    91.43 % |  85% ( 105) |   0% (   0) |   0.8571 |   0.8143 |   220.4\n",
            "Internet_service_ |      51 |   1.04 +  21.22 =  22.25 ms |    100.0 % |  92% (  27) | 100% (  24) |   0.9608 |   0.9317 |   209.3\n",
            "Comics            |     102 |   1.36 +  21.13 =  22.49 ms |    89.58 % |  81% (  48) |  92% (  54) |   0.8725 |   0.8864 |   210.9\n",
            "Saint_Helena      |      93 |   1.05 +  19.97 =  21.02 ms |    100.0 % |  95% (  93) |   0% (   0) |    0.957 |   0.9379 |   209.9\n",
            "Aspirated_consona |      77 |    1.3 +  27.62 =  28.92 ms |     90.0 % |  80% (  30) |  82% (  47) |   0.8182 |     0.81 |   202.8\n",
            "Hydrogen          |      77 |   1.21 +  21.14 =  22.35 ms |    93.51 % |  90% (  77) |   0% (   0) |   0.9091 |   0.8252 |   203.9\n",
            "Space_Race        |      67 |   1.41 +  21.56 =  22.97 ms |    92.54 % |  88% (  67) |   0% (   0) |   0.8806 |   0.8675 |   217.3\n",
            "BeiDou_Navigation |      82 |   1.32 +  23.98 =  25.31 ms |    97.83 % |  84% (  46) |  97% (  36) |   0.9024 |   0.8885 |   214.0\n",
            "Canon_law         |      61 |   1.24 +  22.54 =  23.78 ms |    90.32 % |  83% (  31) |  90% (  30) |   0.8689 |   0.8689 |   223.4\n",
            "Communications_in |      79 |   1.17 +  19.98 =  21.15 ms |    100.0 % |  94% (  35) |  93% (  44) |   0.9367 |   0.9143 |   197.9\n",
            "Boston            |     110 |   1.15 +  22.17 =  23.32 ms |    97.27 % |  97% ( 110) |   0% (   0) |   0.9727 |   0.9566 |   214.9\n",
            "Universal_Studios |     115 |   1.15 +  24.15 =   25.3 ms |    94.55 % |  83% (  55) |  90% (  60) |   0.8696 |    0.887 |   216.2\n",
            "Estonian_language |      69 |   1.26 +  21.33 =  22.59 ms |    94.12 % |  79% (  34) |  85% (  35) |   0.8261 |   0.8464 |   203.7\n",
            "Daylight_saving_t |      93 |   2.11 +  24.07 =  26.17 ms |    92.47 % |  87% (  93) |   0% (   0) |    0.871 |   0.8692 |   219.5\n",
            "Royal_Institute_o |      81 |   1.35 +  25.92 =  27.27 ms |    83.33 % |  78% (  42) |  94% (  39) |   0.8642 |   0.8642 |   225.5\n",
            "National_Archives |      66 |   1.27 +  21.24 =  22.52 ms |    100.0 % |  93% (  32) |  94% (  34) |   0.9394 |   0.9455 |   216.7\n",
            "Tristan_da_Cunha  |      60 |    1.2 +  21.24 =  22.44 ms |     90.0 % |  90% (  10) |  92% (  50) |   0.9167 |   0.9083 |   216.0\n",
            "University_of_Kan |      98 |   1.48 +  21.64 =  23.12 ms |    86.67 % |  77% (  45) |  88% (  53) |   0.8367 |   0.8469 |   214.3\n",
            "Arena_Football_Le |      81 |    1.4 +  20.45 =  21.86 ms |    92.59 % |  81% (  81) |   0% (   0) |   0.8148 |   0.8078 |   215.8\n",
            "Bern              |      47 |   1.21 +  27.05 =  28.26 ms |    97.87 % |  91% (  47) |   0% (   0) |   0.9149 |   0.8156 |   213.0\n",
            "Westminster_Abbey |     130 |   1.01 +  21.14 =  22.16 ms |    90.32 % |  87% (  62) |  97% (  68) |   0.9231 |   0.9086 |   220.1\n",
            "Political_corrupt |      51 |   1.55 +  21.69 =  23.24 ms |    96.08 % |  98% (  51) |   0% (   0) |   0.9804 |   0.9593 |   218.5\n",
            "Classical_music   |      97 |    1.5 +  21.97 =  23.47 ms |    94.85 % |  90% (  97) |   0% (   0) |   0.9072 |   0.8922 |   214.2\n",
            "Slavs             |     167 |   1.43 +  24.51 =  25.94 ms |    96.34 % |  98% (  82) |  91% (  85) |   0.9521 |   0.9226 |   210.6\n",
            "Treaty            |      87 |   1.81 +  20.29 =  22.11 ms |     95.4 % |  93% (  87) |   0% (   0) |    0.931 |   0.9154 |   206.5\n",
            "Josip_Broz_Tito   |     104 |   1.28 +   21.4 =  22.68 ms |    95.19 % |  90% ( 104) |   0% (   0) |   0.9038 |    0.891 |   206.1\n",
            "Marshall_Islands  |      83 |   1.37 +  20.85 =  22.22 ms |    89.16 % |  84% (  83) |   0% (   0) |   0.8434 |   0.8655 |   211.9\n",
            "Szlachta          |      84 |   1.18 +  23.83 =  25.01 ms |    90.48 % |  91% (  84) |   0% (   0) |   0.9167 |   0.8438 |   212.3\n",
            "Virgil            |      65 |   1.38 +  24.04 =  25.42 ms |     96.0 % |  84% (  25) |  77% (  40) |      0.8 |   0.8154 |   219.0\n",
            "Alps              |      92 |   1.26 +  21.59 =  22.86 ms |    98.91 % |  96% (  92) |   0% (   0) |   0.9674 |   0.9693 |   217.1\n",
            "Gene              |      92 |   1.31 +  20.64 =  21.95 ms |    95.65 % |  95% (  92) |   0% (   0) |   0.9565 |   0.8706 |   205.5\n",
            "Guinea-Bissau     |      82 |   1.14 +  20.58 =  21.73 ms |    96.34 % |  90% (  82) |   0% (   0) |   0.9024 |   0.8811 |   186.4\n",
            "List_of_numbered_ |      68 |    1.4 +  21.06 =  22.46 ms |    95.59 % |  91% (  68) |   0% (   0) |   0.9118 |   0.9265 |   203.6\n",
            "Brain             |      78 |   1.28 +  22.17 =  23.45 ms |    97.44 % |  92% (  78) |   0% (   0) |   0.9231 |   0.9004 |   215.6\n",
            "Near_East         |      59 |   1.78 +  19.97 =  21.75 ms |    93.22 % |  96% (  59) |   0% (   0) |   0.9661 |   0.9238 |   199.6\n",
            "Zhejiang          |     116 |   1.13 +  20.83 =  21.97 ms |    96.49 % |  98% (  57) |  98% (  59) |   0.9828 |   0.9704 |   204.9\n",
            "Ministry_of_Defen |      79 |   1.14 +  20.87 =  22.01 ms |    97.37 % |  94% (  38) |  90% (  41) |   0.9241 |   0.9156 |   218.7\n",
            "High-definition_t |     116 |   1.56 +  24.24 =   25.8 ms |    98.48 % |  89% (  66) |  94% (  50) |   0.9138 |   0.8897 |   220.6\n",
            "Wood              |      98 |   1.52 +  19.75 =  21.28 ms |    93.88 % |  83% (  98) |   0% (   0) |   0.8367 |    0.835 |   184.9\n",
            "Somalis           |      83 |   1.58 +  21.69 =  23.27 ms |     98.8 % |  96% (  83) |   0% (   0) |   0.9639 |    0.943 |   206.6\n",
            "Middle_Ages       |     118 |   1.26 +   21.7 =  22.96 ms |    98.31 % |  93% ( 118) |   0% (   0) |   0.9322 |   0.9176 |   215.7\n",
            "Phonology         |      53 |    1.3 +  23.55 =  24.85 ms |    95.65 % |  82% (  23) |  96% (  30) |   0.9057 |   0.8868 |   217.1\n",
            "Computer          |      68 |   1.46 +  20.26 =  21.73 ms |    97.06 % |  97% (  68) |   0% (   0) |   0.9706 |   0.9157 |   200.9\n",
            "Black_people      |      87 |   1.07 +  21.01 =  22.09 ms |     95.4 % |  93% (  87) |   0% (   0) |    0.931 |   0.8555 |   215.2\n",
            "New_Delhi         |      76 |   1.56 +  21.32 =  22.87 ms |    90.79 % |  92% (  76) |   0% (   0) |   0.9211 |   0.9086 |   209.5\n",
            "Bird_migration    |      59 |   1.07 +  18.32 =  19.39 ms |    91.53 % |  91% (  59) |   0% (   0) |   0.9153 |   0.8382 |   181.5\n",
            "Atlantic_City,_Ne |      79 |   1.47 +  22.45 =  23.92 ms |    100.0 % |  88% (  79) |   0% (   0) |   0.8861 |   0.8636 |   214.2\n",
            "MP3               |     107 |   1.35 +  22.76 =  24.11 ms |    92.52 % |  90% ( 107) |   0% (   0) |   0.9065 |   0.8668 |   213.2\n",
            "House_music       |     123 |   1.24 +   22.0 =  23.24 ms |    85.19 % |  83% (  54) |  88% (  69) |   0.8618 |   0.8544 |   217.5\n",
            "Letter_case       |      13 |   2.04 +  23.98 =  26.02 ms |    100.0 % |  92% (  13) |   0% (   0) |   0.9231 |   0.8803 |   221.5\n",
            "Chihuahua_(state) |     115 |   1.09 +  22.38 =  23.47 ms |    95.65 % |  93% ( 115) |   0% (   0) |   0.9304 |   0.9108 |   217.6\n",
            "Pitch_(music)     |      14 |    1.9 +  20.85 =  22.74 ms |    100.0 % | 100% (  14) |   0% (   0) |      1.0 |      1.0 |   216.3\n",
            "England_national_ |      32 |   1.96 +  20.51 =  22.47 ms |    94.12 % |  88% (  17) |  86% (  15) |    0.875 |    0.875 |   209.9\n",
            "Houston           |     143 |   1.15 +  20.83 =  21.98 ms |    96.15 % |  96% (  78) |  90% (  65) |   0.9371 |    0.912 |   210.4\n",
            "Copper            |      82 |   1.13 +   21.4 =  22.54 ms |    93.62 % |  89% (  47) |  97% (  35) |   0.9268 |   0.9207 |   202.8\n",
            "Identity_(social_ |      51 |   1.91 +  21.84 =  23.75 ms |    95.65 % |  95% (  23) |  89% (  28) |   0.9216 |   0.8758 |   218.4\n",
            "Himachal_Pradesh  |      64 |   1.18 +  23.26 =  24.44 ms |    93.55 % |  83% (  31) |  84% (  33) |   0.8438 |   0.8311 |   185.7\n",
            "Communication     |      27 |   1.63 +  19.79 =  21.42 ms |    84.62 % |  84% (  13) |  85% (  14) |   0.8519 |   0.8148 |   191.6\n",
            "Computer_security |     122 |   1.33 +  21.64 =  22.97 ms |    93.75 % |  97% (  48) |  83% (  74) |   0.8934 |   0.8773 |   221.1\n",
            "Orthodox_Judaism  |      69 |   1.45 +  22.37 =  23.81 ms |     87.5 % |  87% (  32) |  97% (  37) |   0.9275 |    0.913 |   220.7\n",
            "Animal            |      26 |   2.71 +  23.89 =   26.6 ms |    96.15 % |  88% (  26) |   0% (   0) |   0.8846 |   0.8803 |   190.0\n",
            "Beer              |      90 |   1.38 +  21.19 =  22.56 ms |    97.87 % |  89% (  47) |  97% (  43) |   0.9333 |   0.9614 |   202.9\n",
            "Race_and_ethnicit |      14 |   2.18 +  20.89 =  23.06 ms |    100.0 % |  85% (  14) |   0% (   0) |   0.8571 |   0.9286 |   215.1\n",
            "Imperial_College_ |      96 |   1.26 +  21.99 =  23.25 ms |    92.73 % |  89% (  55) |  87% (  41) |   0.8854 |   0.8852 |   222.8\n",
            "Hanover           |      53 |    1.1 +  23.58 =  24.68 ms |     93.1 % | 100% (  29) |  95% (  24) |   0.9811 |   0.9124 |   206.2\n",
            "Emotion           |     111 |   1.44 +  20.93 =  22.37 ms |    98.41 % |  93% (  63) |  95% (  48) |   0.9459 |   0.9438 |   206.6\n",
            "Old_English       |      49 |   1.63 +  23.83 =  25.46 ms |     96.3 % |  96% (  27) |  90% (  22) |   0.9388 |   0.9388 |   218.2\n",
            "Aircraft_carrier  |      89 |   1.51 +  22.64 =  24.15 ms |     90.2 % |  76% (  51) |  97% (  38) |   0.8539 |   0.8734 |   217.5\n",
            "Federal_Aviation_ |      41 |   1.84 +  20.64 =  22.48 ms |    100.0 % | 100% (  25) |  81% (  16) |   0.9268 |   0.9268 |   217.3\n",
            "Lancashire        |      59 |   1.16 +  22.65 =   23.8 ms |    100.0 % | 100% (  30) |  86% (  29) |   0.9322 |   0.9266 |   216.8\n",
            "Mesozoic          |      38 |   1.55 +  20.68 =  22.23 ms |    88.24 % |  76% (  17) |  76% (  21) |   0.7632 |    0.689 |   188.1\n",
            "Videoconferencing |      57 |   1.27 +   23.5 =  24.77 ms |    76.92 % |  80% (  26) |  90% (  31) |   0.8596 |   0.8296 |   221.0\n",
            "Gregorian_calenda |      66 |    1.4 +   21.1 =  22.49 ms |    91.89 % |  91% (  37) |  89% (  29) |   0.9091 |   0.8681 |   223.5\n",
            "Xbox_360          |      68 |   1.44 +  20.61 =  22.05 ms |    92.86 % |  89% (  28) |  87% (  40) |   0.8824 |   0.8767 |   215.8\n",
            "Military_history_ |     121 |   1.17 +  20.35 =  21.52 ms |    92.73 % |  87% (  55) |  80% (  66) |   0.8347 |   0.8289 |   217.3\n",
            "Infrared          |      57 |   1.71 +  20.94 =  22.64 ms |    92.86 % |  92% (  28) |  93% (  29) |   0.9298 |   0.9123 |   207.9\n",
            "ASCII             |      72 |    1.4 +  23.34 =  24.74 ms |     96.3 % |  85% (  27) |  93% (  45) |   0.9028 |   0.8941 |   219.8\n",
            "Digestion         |      29 |   1.49 +  19.26 =  20.75 ms |    100.0 % |  96% (  29) |   0% (   0) |   0.9655 |   0.8703 |   171.9\n",
            "Gymnastics        |      75 |    1.1 +  20.37 =  21.47 ms |    89.74 % |  94% (  39) |  77% (  36) |   0.8667 |   0.7987 |   201.5\n",
            "FC_Barcelona      |      75 |   1.34 +   21.9 =  23.24 ms |    94.67 % |  86% (  75) |   0% (   0) |   0.8667 |   0.9007 |   217.9\n",
            "Melbourne         |      81 |   1.41 +  21.52 =  22.93 ms |    97.53 % |  96% (  81) |   0% (   0) |    0.963 |   0.9651 |   216.5\n",
            "John,_King_of_Eng |      70 |   1.13 +  20.91 =  22.04 ms |    88.57 % |  87% (  70) |   0% (   0) |   0.8714 |   0.8766 |   218.6\n",
            "Macintosh         |     155 |   1.33 +  22.63 =  23.96 ms |    92.86 % |  89% (  84) |  91% (  71) |   0.9032 |   0.8808 |   220.1\n",
            "Valencia          |      80 |   1.02 +  22.86 =  23.88 ms |     87.5 % |  88% (  80) |   0% (   0) |   0.8875 |   0.8551 |   222.9\n",
            "General_Electric  |      50 |   1.51 +  23.39 =  24.91 ms |    100.0 % |  95% (  24) |  88% (  26) |     0.92 |      0.9 |   220.1\n",
            "United_States_Arm |      82 |   1.17 +  21.94 =  23.11 ms |    97.73 % |  93% (  44) |  92% (  38) |   0.9268 |   0.9146 |   219.8\n",
            "Franco-Prussian_W |      93 |   1.32 +   21.5 =  22.81 ms |    96.77 % |  90% (  93) |   0% (   0) |   0.9032 |   0.8974 |   216.8\n",
            "Adolescence       |     105 |   1.81 +  20.55 =  22.37 ms |    97.14 % |  99% ( 105) |   0% (   0) |   0.9905 |   0.9062 |   210.2\n",
            "Antarctica        |     156 |   1.08 +  21.63 =  22.71 ms |    91.23 % |  85% (  57) |  92% (  99) |   0.9038 |     0.89 |   210.1\n",
            "Eritrea           |     119 |   1.37 +  20.48 =  21.85 ms |    96.92 % |  96% (  65) |  83% (  54) |   0.9076 |   0.8554 |   202.1\n",
            "Uranium           |     142 |   1.42 +  23.55 =  24.97 ms |    93.33 % |  85% (  60) |  97% (  82) |   0.9225 |   0.9347 |   209.8\n",
            "Circadian_rhythm  |      55 |   1.46 +  20.96 =  22.42 ms |    96.43 % | 100% (  28) |  88% (  27) |   0.9455 |   0.9321 |   198.6\n",
            "Sexual_orientatio |      91 |   1.25 +  20.83 =  22.08 ms |    81.32 % |  83% (  91) |   0% (   0) |   0.8352 |   0.6713 |   214.0\n",
            "Dell              |     135 |   1.15 +  22.01 =  23.16 ms |     87.3 % |  82% (  63) |  88% (  72) |   0.8593 |   0.8568 |   220.3\n",
            "Nintendo_Entertai |     103 |    1.2 +  20.98 =  22.18 ms |    98.15 % |  88% (  54) |  93% (  49) |   0.9126 |   0.9327 |   211.5\n",
            "Seattle           |      72 |    1.2 +  21.88 =  23.08 ms |    93.06 % |  91% (  72) |   0% (   0) |   0.9167 |   0.8921 |   218.6\n",
            "Memory            |      96 |    1.4 +  20.72 =  22.12 ms |    75.68 % |  64% (  37) |  86% (  59) |   0.7812 |   0.7785 |   202.1\n",
            "Multiracial_Ameri |     140 |   1.37 +  20.63 =   22.0 ms |    91.14 % |  83% (  79) | 100% (  61) |   0.9071 |   0.8918 |   211.8\n",
            "Ashkenazi_Jews    |      59 |   1.94 +  24.36 =   26.3 ms |    100.0 % | 100% (  59) |   0% (   0) |      1.0 |   0.9379 |   224.9\n",
            "Pharmaceutical_in |     183 |   1.11 +  23.03 =  24.15 ms |    95.58 % |  93% ( 113) |  94% (  70) |   0.9399 |   0.8903 |   219.9\n",
            "Umayyad_Caliphate |      90 |   1.19 +  23.68 =  24.87 ms |    91.67 % |  88% (  36) |  87% (  54) |   0.8778 |   0.8675 |   208.3\n",
            "Asphalt           |      84 |   1.21 +  22.76 =  23.97 ms |    79.55 % |  77% (  44) |  90% (  40) |   0.8333 |   0.8197 |   211.3\n",
            "Queen_Victoria    |     274 |   1.18 +   20.6 =  21.78 ms |    85.99 % |  81% ( 207) |  97% (  67) |   0.8504 |   0.8236 |   208.3\n",
            "Israel            |      96 |   0.95 +  20.98 =  21.93 ms |    97.92 % |  95% (  96) |   0% (   0) |   0.9583 |   0.9013 |   208.3\n",
            "Hellenistic_perio |     111 |   1.28 +  22.73 =   24.0 ms |     96.4 % |  93% ( 111) |   0% (   0) |   0.9369 |   0.9489 |   212.3\n",
            "Bill_%26_Melinda_ |      58 |   1.11 +  21.66 =  22.77 ms |    92.31 % |  76% (  26) |  87% (  32) |   0.8276 |    0.686 |   219.1\n",
            "Dutch_language    |      90 |   1.62 +  22.82 =  24.44 ms |    87.78 % |  85% (  90) |   0% (   0) |   0.8556 |   0.7942 |   213.0\n",
            "Buckingham_Palace |     136 |   1.09 +  20.35 =  21.44 ms |    92.13 % |  94% (  89) |  97% (  47) |   0.9559 |   0.9222 |   217.9\n",
            "Incandescent_ligh |     105 |    1.5 +  20.99 =  22.49 ms |    91.23 % |  89% (  57) |  97% (  48) |   0.9333 |   0.8926 |   206.1\n",
            "Arsenal_F.C.      |      96 |   1.12 +  20.69 =  21.81 ms |     87.5 % |  81% (  48) |  87% (  48) |   0.8438 |   0.8337 |   213.9\n",
            "Chicago_Cubs      |      68 |    1.5 +  21.99 =   23.5 ms |    97.06 % |  98% (  68) |   0% (   0) |   0.9853 |   0.9775 |   215.1\n",
            "Korean_War        |     109 |   1.31 +  21.92 =  23.23 ms |    89.91 % |  78% ( 109) |   0% (   0) |    0.789 |   0.7855 |   215.9\n",
            "Copyright_infring |     114 |   1.34 +  21.47 =  22.81 ms |     96.0 % |  94% (  50) |  96% (  64) |   0.9561 |   0.9471 |   217.8\n",
            "Greece            |     108 |   1.06 +   23.0 =  24.06 ms |    97.22 % |  94% ( 108) |   0% (   0) |   0.9444 |   0.9228 |   222.0\n",
            "Royal_Dutch_Shell |      78 |   1.38 +  21.57 =  22.95 ms |    86.84 % |  86% (  38) |  95% (  40) |   0.9103 |   0.8887 |   225.1\n",
            "Mammal            |      67 |   1.58 +  23.78 =  25.36 ms |    96.88 % |  93% (  32) |  74% (  35) |   0.8358 |   0.7821 |   211.9\n",
            "East_India_Compan |      97 |   1.44 +  22.19 =  23.64 ms |     93.1 % |  93% (  58) |  97% (  39) |   0.9485 |   0.9198 |   220.7\n",
            "Hokkien           |      70 |   1.29 +  25.12 =  26.41 ms |    92.86 % |  89% (  28) |  71% (  42) |   0.7857 |   0.7829 |   209.1\n",
            "Professional_wres |      79 |   1.35 +  20.71 =  22.06 ms |    87.34 % |  84% (  79) |   0% (   0) |   0.8481 |   0.7688 |   220.5\n",
            "Film_speed        |      62 |   1.33 +  23.39 =  24.72 ms |     84.0 % |  92% (  25) |  91% (  37) |   0.9194 |   0.8441 |   222.3\n",
            "Mexico_City       |     111 |   1.07 +  22.72 =  23.79 ms |    88.29 % |  82% ( 111) |   0% (   0) |   0.8288 |   0.8084 |   216.8\n",
            "Napoleon          |     121 |   1.59 +  20.36 =  21.95 ms |    94.21 % |  90% ( 121) |   0% (   0) |   0.9008 |    0.916 |   207.2\n",
            "Germans           |     104 |   1.18 +  21.57 =  22.75 ms |    98.57 % |  94% (  70) |  94% (  34) |   0.9423 |   0.8841 |   218.2\n",
            "Southeast_Asia    |      66 |   1.54 +  19.92 =  21.46 ms |    96.43 % |  96% (  28) |  94% (  38) |   0.9545 |   0.9419 |   206.2\n",
            "Brigham_Young_Uni |     108 |   1.45 +  20.35 =   21.8 ms |    88.57 % |  82% (  70) |  94% (  38) |   0.8704 |   0.8668 |   205.3\n",
            "Intellectual_prop |      64 |   1.94 +  21.39 =  23.34 ms |    96.88 % |  96% (  32) |  90% (  32) |   0.9375 |   0.9229 |   223.0\n",
            "Florida           |      91 |   0.94 +  20.43 =  21.37 ms |    82.05 % |  76% (  39) | 100% (  52) |   0.9011 |   0.8326 |   203.7\n",
            "Queen_(band)      |      77 |    1.2 +  20.64 =  21.84 ms |    80.52 % |  66% (  77) |   0% (   0) |   0.6623 |   0.7396 |   216.0\n",
            "Presbyterianism   |      98 |   1.38 +  20.43 =  21.81 ms |    93.75 % |  84% (  32) |  95% (  66) |   0.9184 |   0.9065 |   213.5\n",
            "Thuringia         |      93 |   1.06 +   24.1 =  25.16 ms |    89.58 % |  81% (  48) |  91% (  45) |   0.8602 |   0.8074 |   208.1\n",
            "Predation         |      68 |   1.65 +  20.48 =  22.13 ms |    86.49 % |  83% (  37) |  83% (  31) |   0.8382 |   0.8247 |   207.3\n",
            "British_Empire    |      81 |   1.06 +  20.48 =  21.54 ms |     96.3 % |  95% (  81) |   0% (   0) |   0.9506 |   0.9177 |   220.0\n",
            "Botany            |      63 |   1.02 +  21.35 =  22.37 ms |    88.89 % |  85% (  63) |   0% (   0) |   0.8571 |    0.807 |   204.8\n",
            "Madonna_(entertai |     105 |   1.28 +   20.4 =  21.68 ms |    90.48 % |  77% ( 105) |   0% (   0) |   0.7714 |   0.7984 |   204.0\n",
            "Law_of_the_United |      99 |   1.53 +  21.07 =   22.6 ms |    98.31 % |  96% (  59) |  90% (  40) |   0.9394 |   0.8942 |   225.8\n",
            "Myanmar           |     101 |   1.41 +  21.66 =  23.07 ms |     80.2 % |  77% ( 101) |   0% (   0) |   0.7723 |      0.6 |   207.2\n",
            "Jews              |      82 |   1.47 +  22.63 =  24.11 ms |    97.06 % |  94% (  34) |  95% (  48) |   0.9512 |   0.9069 |   221.0\n",
            "Cotton            |     107 |   1.33 +  20.59 =  21.92 ms |    92.86 % |  83% (  56) |  92% (  51) |   0.8785 |   0.8548 |   211.8\n",
            "Data_compression  |      68 |   1.39 +  20.86 =  22.24 ms |    88.89 % |  81% (  27) |  85% (  41) |   0.8382 |   0.8245 |   208.6\n",
            "The_Sun_(United_K |      79 |   1.22 +  21.03 =  22.25 ms |    94.94 % |  91% (  79) |   0% (   0) |   0.9114 |   0.8732 |   225.0\n",
            "Pesticide         |      79 |   1.41 +  20.44 =  21.85 ms |    78.38 % |  75% (  37) |  97% (  42) |   0.8734 |    0.855 |   184.5\n",
            "Somerset          |     102 |   1.05 +  21.62 =  22.67 ms |    90.91 % |  79% (  44) |  98% (  58) |    0.902 |   0.8128 |   214.0\n",
            "Yale_University   |     139 |   1.19 +  23.01 =   24.2 ms |    94.37 % |  92% (  71) |  98% (  68) |   0.9568 |   0.9135 |   219.8\n",
            "Late_Middle_Ages  |     121 |   1.41 +  20.46 =  21.87 ms |    100.0 % | 100% (  58) |  95% (  63) |   0.9752 |   0.9658 |   216.9\n",
            "Ann_Arbor,_Michig |      63 |   1.26 +  20.91 =  22.17 ms |    91.67 % |  83% (  24) |  84% (  39) |   0.8413 |   0.8519 |   206.5\n",
            "Gothic_architectu |     113 |   1.38 +  20.53 =  21.92 ms |    88.24 % |  89% (  68) |  95% (  45) |   0.9204 |   0.8634 |   220.4\n",
            "Cubism            |      93 |   1.46 +   24.2 =  25.66 ms |    91.67 % |  94% (  36) |  98% (  57) |   0.9677 |   0.9198 |   219.4\n",
            "Political_philoso |      59 |    1.3 +  22.06 =  23.36 ms |    100.0 % |  93% (  16) |  88% (  43) |   0.8983 |   0.9153 |   215.8\n",
            "Norfolk_Island    |      89 |   1.34 +  21.78 =  23.12 ms |    89.58 % |  81% (  48) |  97% (  41) |   0.8876 |   0.8834 |   221.0\n",
            "Edmund_Burke      |     124 |   1.07 +  21.16 =  22.23 ms |    76.67 % |  73% (  60) |  90% (  64) |   0.8226 |   0.8062 |   220.0\n",
            "Samoa             |      73 |    1.5 +  21.09 =  22.58 ms |    100.0 % | 100% (  36) |  97% (  37) |   0.9863 |   0.9817 |   204.3\n",
            "Pope_Paul_VI      |      74 |   1.41 +  21.58 =  22.99 ms |    85.14 % |  81% (  74) |   0% (   0) |   0.8108 |   0.7641 |   220.4\n",
            "Switzerland       |     110 |   1.39 +  21.94 =  23.33 ms |    97.27 % |  93% ( 110) |   0% (   0) |   0.9364 |   0.9418 |   211.3\n",
            "Mali              |      61 |   1.26 +  20.57 =  21.83 ms |    97.14 % |  94% (  35) | 100% (  26) |   0.9672 |    0.946 |   209.0\n",
            "Raleigh,_North_Ca |      98 |   0.93 +  21.32 =  22.25 ms |     88.0 % |  90% (  50) |  81% (  48) |   0.8571 |   0.7942 |   209.5\n",
            "Crimean_War       |      71 |   1.33 +  19.94 =  21.28 ms |    94.37 % |  94% (  71) |   0% (   0) |   0.9437 |   0.8965 |   205.1\n",
            "Nonprofit_organiz |      64 |   1.38 +  20.62 =   22.0 ms |    85.29 % |  88% (  34) |  93% (  30) |   0.9062 |   0.8255 |   211.6\n",
            "Literature        |     102 |   1.44 +  20.95 =  22.39 ms |    96.97 % |  90% (  33) |  95% (  69) |   0.9412 |   0.9283 |   218.2\n",
            "Avicenna          |     120 |   1.23 +   24.1 =  25.33 ms |    73.17 % |  68% (  41) |  91% (  79) |   0.8333 |    0.815 |   212.3\n",
            "Nigeria           |      74 |   1.16 +  21.74 =   22.9 ms |    91.89 % |  79% (  74) |   0% (   0) |   0.7973 |   0.8036 |   214.5\n",
            "Molotov%E2%80%93R |     111 |    1.5 +  21.77 =  23.27 ms |    77.08 % |  75% (  48) |  90% (  63) |   0.8378 |   0.8032 |   221.4\n",
            "History_of_scienc |     107 |   1.11 +  21.88 =  22.99 ms |    99.07 % |  94% ( 107) |   0% (   0) |   0.9439 |   0.9119 |   218.4\n",
            "Digimon           |      56 |   1.24 +  23.85 =  25.09 ms |    71.88 % |  78% (  32) |  79% (  24) |   0.7857 |   0.7364 |   217.2\n",
            "Glacier           |      54 |   1.18 +  20.01 =  21.19 ms |    100.0 % | 100% (  24) |  96% (  30) |   0.9815 |   0.9687 |   189.1\n",
            "Affirmative_actio |     134 |   1.35 +  21.15 =   22.5 ms |    90.77 % |  83% (  65) |  95% (  69) |   0.8955 |   0.8926 |   220.2\n",
            "FA_Cup            |      86 |   1.03 +  21.44 =  22.47 ms |    82.61 % |  71% (  46) |  85% (  40) |   0.7791 |   0.6826 |   224.0\n",
            "New_Haven,_Connec |     167 |   1.51 +  22.37 =  23.88 ms |    95.21 % |  92% ( 167) |   0% (   0) |   0.9222 |   0.8615 |   221.1\n",
            "Alsace            |      67 |    1.3 +  21.22 =  22.52 ms |    88.89 % |  80% (  36) |  77% (  31) |    0.791 |   0.7671 |   218.4\n",
            "Carnival          |      67 |   1.36 +  21.56 =  22.92 ms |    88.06 % |  88% (  67) |   0% (   0) |   0.8806 |   0.8367 |   203.0\n",
            "Baptists          |      57 |   1.68 +  20.22 =   21.9 ms |    100.0 % |  96% (  29) | 100% (  28) |   0.9825 |   0.9848 |   203.0\n",
            "Child_labour      |      76 |   1.18 +  20.04 =  21.22 ms |    86.84 % |  88% (  76) |   0% (   0) |   0.8816 |   0.8001 |   209.1\n",
            "Dissolution_of_th |      78 |   1.18 +  22.94 =  24.11 ms |    92.31 % |  85% (  78) |   0% (   0) |    0.859 |    0.828 |   219.3\n",
            "Crucifixion_of_Je |      97 |    1.3 +  21.96 =  23.26 ms |    88.46 % |  75% (  52) |  95% (  45) |   0.8454 |    0.835 |   220.1\n",
            "Supreme_court     |      69 |    1.2 +   20.6 =  21.79 ms |    78.05 % |  75% (  41) |  96% (  28) |   0.8406 |   0.8244 |   217.9\n",
            "Textual_criticism |      81 |   1.35 +  21.35 =  22.71 ms |    75.68 % |  67% (  37) |  88% (  44) |   0.7901 |   0.7134 |   219.6\n",
            "Gramophone_record |     119 |    1.1 +  21.64 =  22.74 ms |    83.19 % |  84% ( 119) |   0% (   0) |   0.8403 |   0.7365 |   219.8\n",
            "Turner_Classic_Mo |      58 |   1.38 +  23.69 =  25.07 ms |     96.3 % |  92% (  27) |  93% (  31) |    0.931 |    0.931 |   230.0\n",
            "Hindu_philosophy  |      85 |    1.1 +  25.33 =  26.43 ms |    92.68 % |  92% (  41) |  90% (  44) |   0.9176 |   0.9105 |   201.6\n",
            "A_cappella        |      80 |   1.29 +  22.86 =  24.15 ms |    92.86 % |  92% (  28) |  94% (  52) |   0.9375 |   0.9321 |   217.5\n",
            "Dominican_Order   |     136 |   1.16 +  20.94 =   22.1 ms |    73.24 % |  71% (  71) |  89% (  65) |   0.8015 |   0.7733 |   211.9\n",
            "Eton_College      |     125 |   1.16 +  22.89 =  24.05 ms |    87.27 % |  85% (  55) |  94% (  70) |    0.904 |    0.872 |   223.3\n",
            "Cork_(city)       |     121 |   1.07 +  20.52 =  21.59 ms |    85.37 % |  87% (  41) |  90% (  80) |   0.8926 |   0.8712 |   205.2\n",
            "Galicia_(Spain)   |      46 |   1.32 +   26.3 =  27.62 ms |     91.3 % |  82% (  46) |   0% (   0) |   0.8261 |   0.8068 |   221.6\n",
            "USB               |      62 |   1.52 +  22.43 =  23.95 ms |    95.16 % |  96% (  62) |   0% (   0) |   0.9677 |    0.933 |   211.4\n",
            "Sichuan           |      95 |   1.13 +  21.41 =  22.53 ms |    100.0 % | 100% (  27) |  92% (  68) |   0.9474 |   0.9339 |   210.5\n",
            "Unicode           |      92 |   1.17 +   22.3 =  23.47 ms |    89.58 % |  91% (  48) |  93% (  44) |   0.9239 |   0.9013 |   215.8\n",
            "Detroit           |     106 |   1.04 +  21.38 =  22.42 ms |    94.34 % |  89% ( 106) |   0% (   0) |   0.8962 |    0.895 |   211.9\n",
            "London            |     104 |   1.48 +  21.46 =  22.94 ms |    100.0 % |  96% ( 104) |   0% (   0) |   0.9615 |   0.9733 |   216.3\n",
            "Culture           |      45 |   1.66 +  21.81 =  23.47 ms |    82.61 % |  73% (  23) | 100% (  22) |   0.8667 |   0.8522 |   212.2\n",
            "Sahara            |      76 |   1.13 +  21.19 =  22.31 ms |    92.59 % |  85% (  27) |  77% (  49) |   0.8026 |   0.7719 |   210.1\n",
            "Rule_of_law       |      60 |   1.28 +  21.63 =  22.91 ms |    96.88 % |  96% (  32) |  89% (  28) |   0.9333 |   0.9144 |   227.7\n",
            "Exhibition_game   |      45 |   1.26 +   20.7 =  21.96 ms |    95.45 % |  86% (  22) |  91% (  23) |   0.8889 |   0.8889 |   217.6\n",
            "Northwestern_Univ |     133 |   1.38 +  23.35 =  24.73 ms |    96.83 % |  95% (  63) |  95% (  70) |   0.9549 |   0.9427 |   221.1\n",
            "Strasbourg        |     104 |   1.16 +  24.24 =   25.4 ms |    100.0 % |  96% (  58) |  91% (  46) |   0.9423 |   0.9295 |   224.4\n",
            "History_of_India  |     117 |   1.24 +  22.11 =  23.35 ms |    92.31 % |  87% ( 117) |   0% (   0) |   0.8718 |    0.863 |   216.4\n",
            "Gamal_Abdel_Nasse |     137 |   1.19 +   20.8 =  21.99 ms |     78.1 % |  71% ( 137) |   0% (   0) |   0.7153 |   0.7057 |   208.2\n",
            "Pope_John_XXIII   |      66 |   1.27 +  21.35 =  22.62 ms |    93.55 % |  93% (  31) |  91% (  35) |   0.9242 |   0.9151 |   220.2\n",
            "Time              |     121 |   1.23 +  20.79 =  22.02 ms |    94.74 % |  92% (  38) |  97% (  83) |   0.9587 |   0.9503 |   218.8\n",
            "St._John%27s,_New |      80 |    1.3 +  22.41 =  23.71 ms |    94.12 % |  88% (  51) |  82% (  29) |   0.8625 |   0.8667 |   220.6\n",
            "John_von_Neumann  |      55 |    1.5 +  21.72 =  23.22 ms |    78.18 % |  78% (  55) |   0% (   0) |   0.7818 |   0.6532 |   218.6\n",
            "PlayStation_3     |     166 |   1.54 +  21.17 =  22.71 ms |    95.71 % |  84% (  70) |  93% (  96) |   0.8976 |   0.8957 |   217.2\n",
            "Royal_assent      |     112 |    1.3 +  20.66 =  21.95 ms |    85.48 % |  80% (  62) |  92% (  50) |   0.8571 |   0.8093 |   223.1\n",
            "Central_African_R |      62 |   1.23 +  22.47 =  23.71 ms |    93.75 % |  87% (  32) |  90% (  30) |   0.8871 |   0.8877 |   222.8\n",
            "Asthma            |      76 |   1.44 +  20.14 =  21.58 ms |    92.11 % |  86% (  38) |  94% (  38) |   0.9079 |   0.8712 |   185.0\n",
            "LaserDisc         |      61 |   1.36 +   22.9 =  24.26 ms |    90.16 % |  81% (  61) |   0% (   0) |   0.8197 |   0.8139 |   219.3\n",
            "George_VI         |      62 |   1.09 +  20.57 =  21.67 ms |    100.0 % |  96% (  30) |  90% (  32) |   0.9355 |   0.9189 |   219.0\n",
            "Federalism        |     101 |   1.07 +  20.35 =  21.42 ms |    93.48 % |  89% (  46) |  94% (  55) |   0.9208 |   0.8285 |   210.5\n",
            "Annelid           |      74 |   1.26 +  24.55 =  25.81 ms |    95.24 % |  85% (  42) |  96% (  32) |   0.9054 |   0.9161 |   210.7\n",
            "War_on_Terror     |     104 |   1.25 +  22.07 =  23.31 ms |    94.59 % |  89% (  37) |  92% (  67) |   0.9135 |   0.9012 |   220.3\n",
            "Labour_Party_(UK) |      90 |   0.99 +   20.4 =  21.39 ms |    84.62 % |  80% (  26) |  92% (  64) |   0.8889 |   0.8742 |   221.5\n",
            "Estonia           |      93 |   1.25 +  20.78 =  22.03 ms |     95.7 % |  88% (  93) |   0% (   0) |   0.8817 |   0.8844 |   207.7\n",
            "Alaska            |     133 |   1.41 +  20.74 =  22.15 ms |    96.92 % |  93% (  65) |  98% (  68) |   0.9624 |   0.9528 |   209.3\n",
            "Mandolin          |     132 |   1.09 +  22.19 =  23.28 ms |    75.41 % |  75% (  61) |  94% (  71) |   0.8561 |    0.797 |   210.7\n",
            "Insect            |     112 |   1.26 +  21.62 =  22.87 ms |    96.43 % |  91% ( 112) |   0% (   0) |   0.9196 |   0.8932 |   192.4\n",
            "Race_(human_categ |     104 |   1.37 +  20.64 =  22.01 ms |    92.31 % |  90% ( 104) |   0% (   0) |   0.9038 |   0.8699 |   217.4\n",
            "Paris             |     124 |   1.24 +  24.08 =  25.33 ms |    97.58 % |  95% ( 124) |   0% (   0) |   0.9597 |   0.9079 |   219.9\n",
            "Apollo            |      66 |    1.6 +   36.2 =   37.8 ms |    90.91 % |  92% (  66) |   0% (   0) |   0.9242 |   0.8737 |   213.2\n",
            "United_States_pre |      72 |   1.34 +  20.95 =  22.29 ms |    97.06 % |  88% (  34) |  92% (  38) |   0.9028 |   0.8832 |   214.1\n",
            "Liberal_Party_of_ |      77 |   1.35 +  21.43 =  22.78 ms |    93.55 % |  93% (  31) |  91% (  46) |   0.9221 |   0.9199 |   224.4\n",
            "Samurai           |      74 |   1.07 +  21.11 =  22.18 ms |    89.19 % |  85% (  74) |   0% (   0) |   0.8514 |   0.8127 |   207.8\n",
            "Software_testing  |      60 |   1.56 +  20.09 =  21.65 ms |    90.32 % |  80% (  31) | 100% (  29) |      0.9 |   0.8462 |   210.7\n",
            "States_of_Germany |      79 |   1.41 +  25.92 =  27.33 ms |    90.32 % |  90% (  31) |  93% (  48) |   0.9241 |   0.8734 |   226.2\n",
            "Glass             |      74 |   1.17 +  19.76 =  20.93 ms |    90.62 % |  90% (  32) |  80% (  42) |   0.8514 |   0.8243 |   191.3\n",
            "Renewable_energy_ |      95 |   1.44 +  21.09 =  22.54 ms |    98.08 % |  96% (  52) |  93% (  43) |   0.9474 |   0.9251 |   215.3\n",
            "Palermo           |      68 |   1.24 +  20.83 =  22.07 ms |    85.29 % |  85% (  34) |  82% (  34) |   0.8382 |   0.8061 |   193.4\n",
            "Green             |      72 |   1.43 +   21.0 =  22.44 ms |    100.0 % |  97% (  36) |  91% (  36) |   0.9444 |   0.9306 |   211.3\n",
            "Neoclassical_arch |      61 |   1.34 +   23.3 =  24.64 ms |    88.57 % |  82% (  35) |  92% (  26) |   0.8689 |   0.8825 |   223.3\n",
            "Serbo-Croatian    |      53 |   1.38 +   25.4 =  26.78 ms |    93.75 % |  93% (  16) |  89% (  37) |   0.9057 |   0.8973 |   220.7\n",
            "CBC_Television    |      58 |   1.46 +  22.09 =  23.54 ms |    86.21 % |  72% (  29) |  86% (  29) |   0.7931 |    0.795 |   222.0\n",
            "IBM               |      74 |   1.56 +  22.88 =  24.44 ms |    92.68 % |  90% (  41) |  96% (  33) |   0.9324 |   0.9167 |   214.0\n",
            "Energy            |      80 |   1.77 +  20.59 =  22.36 ms |    96.97 % |  93% (  33) | 100% (  47) |    0.975 |   0.9661 |   217.4\n",
            "East_Prussia      |      60 |    1.2 +  21.72 =  22.92 ms |    85.71 % |  80% (  21) |  82% (  39) |   0.8167 |   0.8082 |   216.4\n",
            "Ottoman_Empire    |     124 |   1.27 +  21.75 =  23.02 ms |    95.97 % |  93% ( 124) |   0% (   0) |   0.9355 |   0.9128 |   217.7\n",
            "Philosophy_of_spa |      63 |   1.49 +  21.65 =  23.14 ms |    96.97 % |  96% (  33) | 100% (  30) |   0.9841 |    0.963 |   222.4\n",
            "Neolithic         |      55 |    1.1 +  23.72 =  24.82 ms |    96.77 % |  87% (  31) |  87% (  24) |   0.8727 |   0.8739 |   211.9\n",
            "Friedrich_Hayek   |      79 |   1.41 +  22.88 =  24.29 ms |    70.89 % |  65% (  79) |   0% (   0) |   0.6582 |   0.6227 |   222.4\n",
            "Diarrhea          |      54 |   1.37 +  20.14 =  21.51 ms |    76.19 % |  71% (  21) |  84% (  33) |   0.7963 |   0.7491 |   191.0\n",
            "Madrasa           |      99 |   1.44 +  22.81 =  24.25 ms |    74.51 % |  70% (  51) |  95% (  48) |   0.8283 |   0.8074 |   215.4\n",
            "Miami             |     129 |   1.37 +  22.32 =   23.7 ms |    95.24 % |  92% (  63) |  96% (  66) |   0.9457 |   0.9468 |   215.3\n",
            "Philadelphia      |      59 |   1.16 +  23.31 =  24.48 ms |    100.0 % |  98% (  59) |   0% (   0) |   0.9831 |   0.9281 |   212.2\n",
            "John_Kerry        |      71 |   1.11 +  21.02 =  22.14 ms |    87.32 % |  84% (  71) |   0% (   0) |   0.8451 |   0.8218 |   222.1\n",
            "Rajasthan         |      67 |   1.33 +   21.5 =  22.83 ms |    97.06 % |  94% (  34) |  87% (  33) |   0.9104 |   0.8851 |   185.4\n",
            "Guam              |      74 |   1.32 +  21.62 =  22.94 ms |    85.71 % |  74% (  35) |  89% (  39) |   0.8243 |    0.822 |   197.7\n",
            "Empiricism        |      68 |   1.28 +  23.53 =  24.81 ms |    90.32 % |  93% (  31) |  94% (  37) |   0.9412 |   0.9265 |   217.8\n",
            "Idealism          |      79 |   1.27 +   23.2 =  24.47 ms |     87.5 % |  77% (  40) |  97% (  39) |   0.8734 |   0.8743 |   216.5\n",
            "Czech_language    |      93 |   1.26 +  21.75 =  23.01 ms |    93.48 % |  91% (  46) |  91% (  47) |    0.914 |   0.9104 |   206.6\n",
            "Education         |      83 |   1.35 +  19.67 =  21.02 ms |    90.91 % |  87% (  33) |  92% (  50) |   0.9036 |   0.8955 |   202.8\n",
            "Tennessee         |      60 |   1.37 +  21.25 =  22.62 ms |     95.0 % |  95% (  60) |   0% (   0) |     0.95 |   0.9056 |   213.7\n",
            "Canadian_football |      63 |   1.45 +  21.85 =   23.3 ms |    92.86 % |  82% (  28) |  91% (  35) |    0.873 |   0.9206 |   221.1\n",
            "Seven_Years%27_Wa |      82 |   1.19 +  20.73 =  21.91 ms |    75.61 % |  70% (  82) |   0% (   0) |   0.7073 |   0.5844 |   216.1\n",
            "Cyprus            |      76 |   1.17 +  23.72 =  24.89 ms |    96.05 % |  90% (  76) |   0% (   0) |   0.9079 |   0.8515 |   219.0\n",
            "Steven_Spielberg  |     148 |   1.22 +  20.81 =  22.03 ms |    90.79 % |  81% (  76) |  73% (  72) |    0.777 |   0.7655 |   211.6\n",
            "Elevator          |      81 |   1.36 +  19.69 =  21.05 ms |    91.36 % |  87% (  81) |   0% (   0) |   0.8765 |   0.8263 |   203.8\n",
            "Railway_electrifi |      81 |   1.17 +  20.44 =  21.61 ms |    85.11 % |  91% (  47) |  97% (  34) |   0.9383 |    0.849 |   218.7\n",
            "Spanish_language_ |      70 |   1.29 +  21.33 =  22.61 ms |     92.0 % |  76% (  25) |  75% (  45) |   0.7571 |   0.6589 |   222.5\n",
            "Charleston,_South |     147 |   1.33 +  20.78 =   22.1 ms |    98.72 % |  94% (  78) |  97% (  69) |   0.9592 |   0.9629 |   214.8\n",
            "The_Blitz         |     120 |   1.32 +  20.49 =   21.8 ms |    91.67 % |  86% ( 120) |   0% (   0) |   0.8667 |   0.8214 |   210.5\n",
            "Vacuum            |      80 |   1.28 +  20.23 =   21.5 ms |    94.44 % |  94% (  36) |  90% (  44) |    0.925 |   0.8954 |   203.8\n",
            "Quran             |     125 |   1.39 +   23.0 =  24.39 ms |    84.91 % |  83% (  53) |  91% (  72) |     0.88 |   0.8651 |   214.0\n",
            "Geography_of_the_ |      47 |   1.21 +  20.95 =  22.17 ms |    100.0 % | 100% (  19) |  96% (  28) |   0.9787 |   0.9397 |   215.7\n",
            "Compact_disc      |      80 |   1.31 +  20.37 =  21.69 ms |    91.43 % |  88% (  35) |  77% (  45) |    0.825 |   0.8236 |   210.5\n",
            "Modern_history    |     114 |   1.09 +  20.37 =  21.47 ms |    97.37 % |  92% ( 114) |   0% (   0) |   0.9298 |   0.8889 |   213.4\n",
            "Flowering_plant   |      79 |   1.37 +  23.53 =  24.91 ms |    97.67 % |  90% (  43) |  94% (  36) |   0.9241 |   0.9114 |   212.7\n",
            "Hyderabad         |      81 |   1.41 +  24.21 =  25.61 ms |    97.53 % |  87% (  81) |   0% (   0) |   0.8765 |   0.9128 |   219.9\n",
            "Santa_Monica,_Cal |      68 |   1.59 +  20.99 =  22.59 ms |    100.0 % |  97% (  38) |  80% (  30) |   0.8971 |   0.8971 |   204.3\n",
            "Pain              |      91 |    1.2 +   21.5 =   22.7 ms |    90.38 % |  88% (  52) |  92% (  39) |   0.9011 |   0.8718 |   212.7\n",
            "Database          |      71 |   1.09 +  20.46 =  21.56 ms |    88.57 % |  91% (  35) |  94% (  36) |   0.9296 |   0.8734 |   208.0\n",
            "Tucson,_Arizona   |     111 |   1.09 +  20.78 =  21.87 ms |    91.89 % |  87% ( 111) |   0% (   0) |   0.8739 |   0.8314 |   203.8\n",
            "Bacteria          |      66 |   1.31 +  21.61 =  22.92 ms |    92.42 % |  87% (  66) |   0% (   0) |   0.8788 |   0.8554 |   197.8\n",
            "Printed_circuit_b |     105 |   1.48 +   21.0 =  22.47 ms |    98.21 % |  83% (  56) |  93% (  49) |   0.8857 |    0.892 |   197.0\n",
            "Greeks            |     123 |   1.39 +  21.78 =  23.17 ms |    79.31 % |  60% (  58) |  95% (  65) |   0.7886 |   0.7082 |   214.6\n",
            "Premier_League    |     179 |   1.39 +   20.6 =  21.99 ms |    98.32 % |  91% ( 119) |  98% (  60) |   0.9385 |   0.8539 |   220.5\n",
            "Roman_Republic    |     118 |    1.2 +  19.58 =  20.78 ms |    91.53 % |  88% ( 118) |   0% (   0) |   0.8898 |   0.8341 |   189.2\n",
            "Pacific_War       |     134 |    1.3 +  20.91 =  22.21 ms |    91.79 % |  86% ( 134) |   0% (   0) |   0.8657 |   0.8438 |   212.2\n",
            "San_Diego         |     147 |   1.28 +  23.33 =  24.61 ms |    91.55 % |  88% (  71) |  97% (  76) |    0.932 |   0.9255 |   215.9\n",
            "Iran              |      92 |   1.57 +  22.13 =   23.7 ms |    98.91 % |  94% (  92) |   0% (   0) |   0.9457 |   0.9221 |   218.2\n",
            "British_Isles     |     115 |    1.4 +  21.16 =  22.56 ms |    88.89 % |  92% (  54) |  90% (  61) |    0.913 |   0.8739 |   214.6\n",
            "Association_footb |      67 |   1.29 +  20.48 =  21.77 ms |    100.0 % |  97% (  38) | 100% (  29) |   0.9851 |   0.9851 |   213.7\n",
            "Georgian_architec |      49 |   1.24 +  20.89 =  22.13 ms |     96.3 % |  96% (  27) |  86% (  22) |   0.9184 |   0.9034 |   220.1\n",
            "Liberia           |      81 |   1.14 +  20.39 =  21.53 ms |    88.64 % |  90% (  44) |  91% (  37) |   0.9136 |   0.8879 |   209.3\n",
            "Alfred_North_Whit |     149 |   1.37 +  20.43 =   21.8 ms |    83.17 % |  80% ( 101) | 100% (  48) |   0.8658 |   0.8014 |   210.2\n",
            "Antibiotics       |      88 |   1.19 +  22.78 =  23.97 ms |    91.49 % |  89% (  47) |  85% (  41) |    0.875 |   0.8538 |   209.4\n",
            "Windows_8         |     141 |   1.13 +  21.03 =  22.16 ms |    93.22 % |  84% (  59) |  95% (  82) |   0.9078 |   0.8838 |   221.9\n",
            "Swaziland         |      59 |   1.58 +  20.53 =  22.11 ms |     80.0 % |  84% (  25) |  70% (  34) |   0.7627 |   0.7213 |   197.4\n",
            "Translation       |      80 |   1.54 +  23.36 =   24.9 ms |    93.18 % |  95% (  44) |  94% (  36) |     0.95 |   0.9392 |   220.8\n",
            "Airport           |      65 |   1.48 +  18.16 =  19.64 ms |    94.12 % |  94% (  34) |  83% (  31) |   0.8923 |   0.8665 |   186.7\n",
            "Super_Nintendo_En |      75 |   1.31 +  22.73 =  24.04 ms |    93.55 % |  93% (  31) |  97% (  44) |     0.96 |   0.9256 |   224.0\n",
            "Sumer             |      80 |   1.11 +  22.27 =  23.38 ms |     92.0 % |  92% (  50) |  96% (  30) |   0.9375 |   0.8999 |   214.3\n",
            "Tuvalu            |      94 |   1.25 +  23.56 =  24.81 ms |    86.17 % |  85% (  94) |   0% (   0) |   0.8511 |    0.817 |   219.3\n",
            "Namibia           |     109 |   1.16 +  21.13 =  22.29 ms |    87.27 % |  80% (  55) |  88% (  54) |    0.844 |   0.8379 |   211.7\n",
            "Russian_language  |     101 |   1.36 +  24.62 =  25.97 ms |    84.31 % |  84% (  51) |  94% (  50) |   0.8911 |   0.8694 |   212.4\n",
            "United_States_Air |      77 |   1.22 +  22.81 =  24.03 ms |    94.81 % |  94% (  77) |   0% (   0) |   0.9481 |   0.9171 |   220.6\n",
            "Light-emitting_di |     122 |   1.41 +  21.83 =  23.24 ms |    83.93 % |  78% (  56) |  93% (  66) |   0.8689 |   0.8488 |   206.0\n",
            "Great_power       |      53 |   1.26 +  21.04 =   22.3 ms |    100.0 % |  90% (  22) |  83% (  31) |   0.8679 |   0.8324 |   219.0\n",
            "Bird              |      63 |   1.34 +  23.36 =   24.7 ms |    93.65 % |  95% (  63) |   0% (   0) |   0.9524 |   0.9103 |   190.9\n",
            "Qing_dynasty      |      97 |   2.84 +  30.89 =  33.73 ms |    87.63 % |  75% (  97) |   0% (   0) |   0.7526 |   0.7447 |   219.9\n",
            "Indigenous_people |      84 |   1.28 +  25.77 =  27.05 ms |    98.81 % |  96% (  84) |   0% (   0) |   0.9643 |   0.9475 |   215.4\n",
            "Red               |     130 |   1.51 +  23.32 =  24.82 ms |    94.52 % |  89% (  73) |  82% (  57) |   0.8615 |   0.8416 |   206.7\n",
            "Mosaic            |      86 |   1.35 +  24.16 =  25.51 ms |    83.72 % |  82% (  86) |   0% (   0) |   0.8256 |   0.8004 |   216.5\n",
            "University        |     100 |   1.52 +   25.0 =  26.53 ms |    89.47 % |  86% (  38) |  90% (  62) |     0.89 |   0.8867 |   216.5\n",
            "Religion_in_ancie |     122 |   1.27 +  24.48 =  25.75 ms |    84.43 % |  78% ( 122) |   0% (   0) |   0.7869 |   0.7768 |   217.8\n",
            "YouTube           |      89 |    1.4 +  23.34 =  24.74 ms |    94.87 % |  92% (  39) |  92% (  50) |   0.9213 |    0.912 |   214.7\n",
            "Separation_of_chu |     130 |   1.45 +  23.92 =  25.37 ms |    98.46 % |  95% (  65) | 100% (  65) |   0.9769 |   0.9639 |   223.2\n",
            "Bras%C3%ADlia     |      65 |   2.29 +  22.26 =  24.54 ms |    90.62 % |  81% (  32) |  81% (  33) |   0.8154 |   0.7926 |   200.9\n",
            "Economy_of_Greece |      90 |   1.43 +  24.32 =  25.76 ms |    94.74 % |  94% (  38) |  96% (  52) |   0.9556 |   0.9517 |   217.2\n",
            "Party_leaders_of_ |      79 |   1.24 +  27.03 =  28.27 ms |    76.47 % |  61% (  34) |  93% (  45) |   0.7975 |   0.7722 |   225.9\n",
            "Armenians         |      66 |   1.53 +  23.81 =  25.34 ms |    96.77 % |  93% (  31) |  88% (  35) |   0.9091 |   0.9177 |   212.1\n",
            "Jehovah%27s_Witne |     128 |   1.45 +  24.89 =  26.34 ms |    92.06 % |  92% (  63) |  96% (  65) |   0.9453 |   0.9399 |   219.1\n",
            "Dwight_D._Eisenho |     124 |   1.31 +  24.02 =  25.33 ms |    90.32 % |  84% ( 124) |   0% (   0) |   0.8468 |   0.8384 |   218.4\n",
            "The_Bronx         |      85 |   1.21 +  26.84 =  28.05 ms |    95.29 % |  92% (  85) |   0% (   0) |   0.9294 |   0.9218 |   220.0\n",
            "Financial_crisis_ |     111 |    1.7 +  25.07 =  26.77 ms |    94.59 % |  93% ( 111) |   0% (   0) |   0.9369 |    0.924 |   223.6\n",
            "Portugal          |     119 |   1.35 +  24.88 =  26.23 ms |    94.12 % |  94% ( 119) |   0% (   0) |   0.9412 |   0.8702 |   215.8\n",
            "Humanism          |      73 |    1.4 +  24.83 =  26.23 ms |    74.19 % |  67% (  31) |  95% (  42) |   0.8356 |   0.7849 |   221.1\n",
            "Geological_histor |      85 |   1.34 +  27.83 =  29.17 ms |    94.12 % |  74% (  51) |  94% (  34) |   0.8235 |   0.8392 |   202.5\n",
            "Police            |      89 |   1.26 +  23.93 =  25.19 ms |    95.24 % |  95% (  42) |  95% (  47) |   0.9551 |   0.8869 |   221.8\n",
            "Genocide          |      86 |   1.58 +  23.71 =   25.3 ms |     95.0 % |  85% (  40) |  97% (  46) |   0.9186 |   0.9147 |   228.7\n",
            "Saint_Barth%C3%A9 |      65 |   1.31 +   23.8 =  25.11 ms |    88.57 % |  94% (  35) |  93% (  30) |   0.9385 |    0.881 |   196.8\n",
            "Tajikistan        |      83 |   1.07 +  26.22 =  27.29 ms |    97.44 % |  89% (  39) |  97% (  44) |   0.9398 |   0.8615 |   210.9\n",
            "University_of_Not |      71 |    1.4 +  20.78 =  22.18 ms |    87.32 % |  87% (  71) |   0% (   0) |   0.8732 |   0.8169 |   205.2\n",
            "Anthropology      |     127 |   1.16 +  20.95 =  22.11 ms |    93.24 % |  90% (  74) |  84% (  53) |   0.8819 |    0.859 |   209.2\n",
            "Montana           |      40 |   1.38 +  20.97 =  22.35 ms |     95.0 % |  97% (  40) |   0% (   0) |    0.975 |   0.9524 |   205.3\n",
            "Punjab,_Pakistan  |      73 |   1.14 +   23.2 =  24.34 ms |    97.06 % |  97% (  34) |  97% (  39) |   0.9726 |   0.9452 |   197.8\n",
            "Infection         |      90 |   1.23 +  22.52 =  23.75 ms |    100.0 % | 100% (  39) |  98% (  51) |   0.9889 |   0.9742 |   207.4\n",
            "Hunting           |     155 |   1.19 +  25.94 =  27.13 ms |    95.83 % |  97% (  96) |  93% (  59) |   0.9613 |    0.938 |   213.0\n",
            "Kathmandu         |      82 |   1.67 +  25.65 =  27.32 ms |    96.34 % |  91% (  82) |   0% (   0) |   0.9146 |   0.8923 |   179.3\n",
            "Myocardial_infarc |      33 |   2.01 +  28.43 =  30.44 ms |      0.0 % |   0% (   0) |  96% (  33) |   0.9697 |   0.9697 |   208.2\n",
            "Matter            |      37 |   2.63 +  33.73 =  36.37 ms |      0.0 % |   0% (   0) |  78% (  37) |   0.7838 |   0.7838 |   216.3\n",
            "------------------|---------|-----------------------------|------------|-------------|-------------|----------|----------|--------------\n",
            "Grand Total       |   32401 |   1.32 +  22.03 =  23.35 ms |    92.08 % |  87% (21918) |  91% (10483) |   0.8913 |   0.8684 |   213.1\n"
          ]
        }
      ],
      "source": [
        "# with mpnet and electra-best (optimized) - context_length_threshold = 205 - old themes"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [
        "U38_GxU5ZWFh",
        "JyKtVXm661x3",
        "UzHw72nd6_wA",
        "BTEq0Khe7DM0",
        "Uxnd1xuWqVeA",
        "xccnbbAf7J1o",
        "raCs2XgbZpTG",
        "W7Jcd0sw7TOf",
        "XNJggLTqUSrg",
        "DDxE7DH90sss",
        "_4eigUBJO3eO",
        "uFTXHe2dOqOR",
        "dcNnLJxOPW08"
      ],
      "provenance": [],
      "include_colab_link": true
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU",
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "5e46ca6cebaa497bb3542c85689fa117": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_9c07e959cb5d4e258c6068ac8f115937",
              "IPY_MODEL_665352ce795348028e9873845109bcc3",
              "IPY_MODEL_50636167e4484b8b9dbcc313c3fb22bb"
            ],
            "layout": "IPY_MODEL_ea1a9a9253d3427b9fccbc21082476e2"
          }
        },
        "9c07e959cb5d4e258c6068ac8f115937": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d1333a25055d4c179e352bbd18329d56",
            "placeholder": "​",
            "style": "IPY_MODEL_7cb95e77b0554fe8aa182823dda95199",
            "value": "Downloading (…)okenizer_config.json: 100%"
          }
        },
        "665352ce795348028e9873845109bcc3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ebfa6f7b16e643eaba0070455a42c407",
            "max": 252,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_ac4e90976a65403e9d0e15eb5ab772b0",
            "value": 252
          }
        },
        "50636167e4484b8b9dbcc313c3fb22bb": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_cb4fe34398864c8193747a1ca2c40d4e",
            "placeholder": "​",
            "style": "IPY_MODEL_75e802335a78483091ac3f15b428070a",
            "value": " 252/252 [00:00&lt;00:00, 4.48kB/s]"
          }
        },
        "ea1a9a9253d3427b9fccbc21082476e2": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d1333a25055d4c179e352bbd18329d56": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7cb95e77b0554fe8aa182823dda95199": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "ebfa6f7b16e643eaba0070455a42c407": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ac4e90976a65403e9d0e15eb5ab772b0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "cb4fe34398864c8193747a1ca2c40d4e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "75e802335a78483091ac3f15b428070a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "8fdb9e2851374dbd867a6ab0924a77ee": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_cc2e0a13ae0d47f8a47db163b3a4d3cb",
              "IPY_MODEL_4c3213c235db43fb81c09882882cd450",
              "IPY_MODEL_847580e6e30c489b92141d08e2de6d37"
            ],
            "layout": "IPY_MODEL_9fdf189253744c72bd42b5907b1949f3"
          }
        },
        "cc2e0a13ae0d47f8a47db163b3a4d3cb": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_9a9c74ebbd334ce998a33d59643f1240",
            "placeholder": "​",
            "style": "IPY_MODEL_efb04f5e8cb741d0afebc0f48f9cfa2c",
            "value": "Downloading (…)lve/main/config.json: 100%"
          }
        },
        "4c3213c235db43fb81c09882882cd450": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d30fc6bf02e9407db50fba22ae2fee4d",
            "max": 689,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_3ca0a533b4b94904b141dce121b85566",
            "value": 689
          }
        },
        "847580e6e30c489b92141d08e2de6d37": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_cc0beb05f7244dd2a1c5023444837846",
            "placeholder": "​",
            "style": "IPY_MODEL_674fa0d39deb4879b6dce947051689aa",
            "value": " 689/689 [00:00&lt;00:00, 10.1kB/s]"
          }
        },
        "9fdf189253744c72bd42b5907b1949f3": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9a9c74ebbd334ce998a33d59643f1240": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "efb04f5e8cb741d0afebc0f48f9cfa2c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "d30fc6bf02e9407db50fba22ae2fee4d": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3ca0a533b4b94904b141dce121b85566": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "cc0beb05f7244dd2a1c5023444837846": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "674fa0d39deb4879b6dce947051689aa": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "ea971bb214894a919bd86597c746cbe3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_6e915df9688044aba21a1907b8a9f96f",
              "IPY_MODEL_3eb9929610e94362a31abf6f2ec38a12",
              "IPY_MODEL_13ed22d90c1b4107acc2f8dc5bf9d363"
            ],
            "layout": "IPY_MODEL_fd29c20e9249400089610ad7ed4add24"
          }
        },
        "6e915df9688044aba21a1907b8a9f96f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_aa58fed4497a4aff8e9bb7316e7a9785",
            "placeholder": "​",
            "style": "IPY_MODEL_e214f60e233d4d04bed4569542cfb072",
            "value": "Downloading (…)solve/main/vocab.txt: 100%"
          }
        },
        "3eb9929610e94362a31abf6f2ec38a12": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_9630eac4580d40c78b05bd473126b9db",
            "max": 231508,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_c51453b6361e43e897a78b1b5c0cedc2",
            "value": 231508
          }
        },
        "13ed22d90c1b4107acc2f8dc5bf9d363": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_eb6db8b1f5fc41fba752a7400cb2cb59",
            "placeholder": "​",
            "style": "IPY_MODEL_e25723644cab48c0bffe5dff7a5a6459",
            "value": " 232k/232k [00:00&lt;00:00, 259kB/s]"
          }
        },
        "fd29c20e9249400089610ad7ed4add24": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "aa58fed4497a4aff8e9bb7316e7a9785": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e214f60e233d4d04bed4569542cfb072": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "9630eac4580d40c78b05bd473126b9db": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c51453b6361e43e897a78b1b5c0cedc2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "eb6db8b1f5fc41fba752a7400cb2cb59": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e25723644cab48c0bffe5dff7a5a6459": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "93fc6edb6c9a4922a15f7dd6ff0172da": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_979cf28032aa404cb11ce79fcd662954",
              "IPY_MODEL_189bb73c5b024388a10d2ff17de27134",
              "IPY_MODEL_9f11c6b8f2144e57864618aa73050669"
            ],
            "layout": "IPY_MODEL_2effa6df7cd44b7386d43b9a1153b67b"
          }
        },
        "979cf28032aa404cb11ce79fcd662954": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_32d7320c36e047f194a63bc7348dd486",
            "placeholder": "​",
            "style": "IPY_MODEL_e6217fa25df94b11bcad04ace5943027",
            "value": "Downloading (…)cial_tokens_map.json: 100%"
          }
        },
        "189bb73c5b024388a10d2ff17de27134": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e85e8399e02b4a5783bbe622a9231b9c",
            "max": 112,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_eb0f9afaa66c4bb8bd3d1a5992e2b5d5",
            "value": 112
          }
        },
        "9f11c6b8f2144e57864618aa73050669": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_c7ac9ad92d5e4d62a5857c8ee3656093",
            "placeholder": "​",
            "style": "IPY_MODEL_ac95d86d816641afaf273b31f2040a13",
            "value": " 112/112 [00:00&lt;00:00, 2.36kB/s]"
          }
        },
        "2effa6df7cd44b7386d43b9a1153b67b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "32d7320c36e047f194a63bc7348dd486": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e6217fa25df94b11bcad04ace5943027": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "e85e8399e02b4a5783bbe622a9231b9c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "eb0f9afaa66c4bb8bd3d1a5992e2b5d5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "c7ac9ad92d5e4d62a5857c8ee3656093": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ac95d86d816641afaf273b31f2040a13": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "f75c2adcf7f54d5c98a4febc130fd0ad": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_b1bae9234b664cf4814dcfb8bc83ee13",
              "IPY_MODEL_53ba79f7ba804105b387b0860ef05cca",
              "IPY_MODEL_dc7ed138503542d4a6ee6583f2c15123"
            ],
            "layout": "IPY_MODEL_5ccc7d6ec71842b1bf8e772dc46434a6"
          }
        },
        "b1bae9234b664cf4814dcfb8bc83ee13": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_da9963741e684cd4a680903328053ae3",
            "placeholder": "​",
            "style": "IPY_MODEL_440e641feb82497dbf87415b49e6ed0c",
            "value": "Downloading (…)&quot;pytorch_model.bin&quot;;: 100%"
          }
        },
        "53ba79f7ba804105b387b0860ef05cca": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_bd6974c44f1149b1bec7c1934970c1af",
            "max": 437998343,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_3b2337a8da274229bdc482832294a298",
            "value": 437998343
          }
        },
        "dc7ed138503542d4a6ee6583f2c15123": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_8de015da5a874636a88a040226991336",
            "placeholder": "​",
            "style": "IPY_MODEL_f6e0cfeade5d4d69aeb4567708814ee7",
            "value": " 438M/438M [00:03&lt;00:00, 71.9MB/s]"
          }
        },
        "5ccc7d6ec71842b1bf8e772dc46434a6": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "da9963741e684cd4a680903328053ae3": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "440e641feb82497dbf87415b49e6ed0c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "bd6974c44f1149b1bec7c1934970c1af": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3b2337a8da274229bdc482832294a298": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "8de015da5a874636a88a040226991336": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f6e0cfeade5d4d69aeb4567708814ee7": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "a45951df22984efdbe00b4d65daa4dc6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_855090a3f45d4eebb987cbbd2bf271bc",
              "IPY_MODEL_06e1376dab834540a042a897dc61683a",
              "IPY_MODEL_2c4fecbc235d4b929479e5ce587e72ab"
            ],
            "layout": "IPY_MODEL_9eba878120744c22b7c8374a01335689"
          }
        },
        "855090a3f45d4eebb987cbbd2bf271bc": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_4a12b088d0bb4d2d84a6417fe3c00b03",
            "placeholder": "​",
            "style": "IPY_MODEL_1f56d944546b4c16aac4b511a05286d4",
            "value": "Downloading (…)okenizer_config.json: 100%"
          }
        },
        "06e1376dab834540a042a897dc61683a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_07e61381ff0a4e6ebb6dd1b068c38c2e",
            "max": 549,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_95827f07fecc473f8b0743bd52bfa49a",
            "value": 549
          }
        },
        "2c4fecbc235d4b929479e5ce587e72ab": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_0c4f2cd3070248949da77b13ea04f43e",
            "placeholder": "​",
            "style": "IPY_MODEL_a0fc0bed83524a63a284ad6277a53a46",
            "value": " 549/549 [00:00&lt;00:00, 22.9kB/s]"
          }
        },
        "9eba878120744c22b7c8374a01335689": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "4a12b088d0bb4d2d84a6417fe3c00b03": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1f56d944546b4c16aac4b511a05286d4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "07e61381ff0a4e6ebb6dd1b068c38c2e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "95827f07fecc473f8b0743bd52bfa49a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "0c4f2cd3070248949da77b13ea04f43e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a0fc0bed83524a63a284ad6277a53a46": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "4e18146581d342dc9aea70c73b9059fd": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_26bf94bc382f4b8aa47734030fc6d789",
              "IPY_MODEL_6ef8ac763cc74a3ab76758db98196195",
              "IPY_MODEL_2badb4ab562b4378acf6f1d03b2983e3"
            ],
            "layout": "IPY_MODEL_dfaf25d204c84b91bcc7629db9f17f80"
          }
        },
        "26bf94bc382f4b8aa47734030fc6d789": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_15f0a57013de4dc5854b81c36365beb0",
            "placeholder": "​",
            "style": "IPY_MODEL_2cae8ec3495f43d5be727a5e82f192c5",
            "value": "Downloading (…)solve/main/vocab.txt: 100%"
          }
        },
        "6ef8ac763cc74a3ab76758db98196195": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_b721da0ecb1047c0a89d165b83b9ea05",
            "max": 231508,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_c89d7f6e1eaa435f85c0a21988af0895",
            "value": 231508
          }
        },
        "2badb4ab562b4378acf6f1d03b2983e3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_c70ead75d16241a1a802507ec26e6f45",
            "placeholder": "​",
            "style": "IPY_MODEL_17c0422b6280417081c2b4ee0d5eb7bc",
            "value": " 232k/232k [00:00&lt;00:00, 263kB/s]"
          }
        },
        "dfaf25d204c84b91bcc7629db9f17f80": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "15f0a57013de4dc5854b81c36365beb0": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2cae8ec3495f43d5be727a5e82f192c5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "b721da0ecb1047c0a89d165b83b9ea05": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c89d7f6e1eaa435f85c0a21988af0895": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "c70ead75d16241a1a802507ec26e6f45": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "17c0422b6280417081c2b4ee0d5eb7bc": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "56008c0784d948d790939ab3ad2416fc": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_6a73776ef54243f5bc34ef43673440e3",
              "IPY_MODEL_7480a8f7d4e34f909f068d209df08266",
              "IPY_MODEL_821d39b2154d4619b96570233b05efb3"
            ],
            "layout": "IPY_MODEL_7a7b5180f61144e3bd6cca311d9f1261"
          }
        },
        "6a73776ef54243f5bc34ef43673440e3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_7b12523306a2406881d7a49f175a90d9",
            "placeholder": "​",
            "style": "IPY_MODEL_b30a4fcd572c4e6eb7fb8b4cc49df921",
            "value": "Downloading (…)/main/tokenizer.json: 100%"
          }
        },
        "7480a8f7d4e34f909f068d209df08266": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_2f7e5fc0c0cf497faa278a005c0d9fe1",
            "max": 466247,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_57afc18618274219a57e66aa040a0aa1",
            "value": 466247
          }
        },
        "821d39b2154d4619b96570233b05efb3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d02486d7710840dcb756709f0b426ecb",
            "placeholder": "​",
            "style": "IPY_MODEL_841b3278610448559ca0547ce9299b62",
            "value": " 466k/466k [00:01&lt;00:00, 421kB/s]"
          }
        },
        "7a7b5180f61144e3bd6cca311d9f1261": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7b12523306a2406881d7a49f175a90d9": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b30a4fcd572c4e6eb7fb8b4cc49df921": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "2f7e5fc0c0cf497faa278a005c0d9fe1": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "57afc18618274219a57e66aa040a0aa1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "d02486d7710840dcb756709f0b426ecb": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "841b3278610448559ca0547ce9299b62": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "c88ff834bce04877a8263fc9aa8c9658": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_8ef4eba9a1614c0c930ee261532ef6ee",
              "IPY_MODEL_b2f75b902e114860871d56e94da64e13",
              "IPY_MODEL_f71958b283ea4183bb5bc3b6b7388fb2"
            ],
            "layout": "IPY_MODEL_d5128c7a02e74386920bc377ce190da4"
          }
        },
        "8ef4eba9a1614c0c930ee261532ef6ee": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_a39af2f41f744091abbb834921b91bd4",
            "placeholder": "​",
            "style": "IPY_MODEL_d9e1ed5f431b4b6ebeb58801ad180b9f",
            "value": "Downloading (…)cial_tokens_map.json: 100%"
          }
        },
        "b2f75b902e114860871d56e94da64e13": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_682e6725dfa0430d82182b8443042bde",
            "max": 112,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_227bce91d5604a7c938b5bdfaaab89f5",
            "value": 112
          }
        },
        "f71958b283ea4183bb5bc3b6b7388fb2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_a026aab1f317479fa790edbb51603fda",
            "placeholder": "​",
            "style": "IPY_MODEL_6bb7e8237d1a4b22a43721e50663e00a",
            "value": " 112/112 [00:00&lt;00:00, 6.49kB/s]"
          }
        },
        "d5128c7a02e74386920bc377ce190da4": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a39af2f41f744091abbb834921b91bd4": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d9e1ed5f431b4b6ebeb58801ad180b9f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "682e6725dfa0430d82182b8443042bde": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "227bce91d5604a7c938b5bdfaaab89f5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "a026aab1f317479fa790edbb51603fda": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6bb7e8237d1a4b22a43721e50663e00a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "65e5a635331f483ea00b389342aef4a7": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_ef9409853ab0400981b156aaf773f2a2",
              "IPY_MODEL_b3b5662e1c5544ac8343ae68707a6914",
              "IPY_MODEL_ad31c2c720d24a4ca5c926fd60f32875"
            ],
            "layout": "IPY_MODEL_7d2b9b0d591d4cdcb9365adcfeb31d51"
          }
        },
        "ef9409853ab0400981b156aaf773f2a2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f68adc2e9af742a89fbc42e6997ce123",
            "placeholder": "​",
            "style": "IPY_MODEL_833e0e3e5de24f8daca1da0c8b79ae68",
            "value": "Downloading (…)lve/main/config.json: 100%"
          }
        },
        "b3b5662e1c5544ac8343ae68707a6914": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f41cf80f0b674031acdbbea6dc12fe6e",
            "max": 783,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_1fe5f85c0f9d445599f0a42850c1916f",
            "value": 783
          }
        },
        "ad31c2c720d24a4ca5c926fd60f32875": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_7bf3729a79174620be770f06a4f44e03",
            "placeholder": "​",
            "style": "IPY_MODEL_ec529a8662a14d01bba58ebc4913b7c9",
            "value": " 783/783 [00:00&lt;00:00, 42.1kB/s]"
          }
        },
        "7d2b9b0d591d4cdcb9365adcfeb31d51": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f68adc2e9af742a89fbc42e6997ce123": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "833e0e3e5de24f8daca1da0c8b79ae68": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "f41cf80f0b674031acdbbea6dc12fe6e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1fe5f85c0f9d445599f0a42850c1916f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "7bf3729a79174620be770f06a4f44e03": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ec529a8662a14d01bba58ebc4913b7c9": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "44665a4b5de54ad4b76d2195d5f931bf": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_541b159c4252417fb820f104c8c448a6",
              "IPY_MODEL_6316c1d751184feba6b4cbf206125e77",
              "IPY_MODEL_ed6aab02518b491abc4a1c8d1d26ec72"
            ],
            "layout": "IPY_MODEL_65de4b3151fa4ee1a148c98f0dd26b0d"
          }
        },
        "541b159c4252417fb820f104c8c448a6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_715d912540ee4e8b837a6933fdf74033",
            "placeholder": "​",
            "style": "IPY_MODEL_899d25fd97474279be41d8ca127e5ed1",
            "value": "Downloading (…)&quot;pytorch_model.bin&quot;;: 100%"
          }
        },
        "6316c1d751184feba6b4cbf206125e77": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_91d136f49faa4fbf98e89b7501c1fcf1",
            "max": 435654577,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_d4b42a978220400cafb0c87fe4cdbd9d",
            "value": 435654577
          }
        },
        "ed6aab02518b491abc4a1c8d1d26ec72": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d32888b6ba354a5082e46456ce3645f6",
            "placeholder": "​",
            "style": "IPY_MODEL_7779a29f6b474020854c6e5a3e038a14",
            "value": " 436M/436M [00:26&lt;00:00, 16.9MB/s]"
          }
        },
        "65de4b3151fa4ee1a148c98f0dd26b0d": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "715d912540ee4e8b837a6933fdf74033": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "899d25fd97474279be41d8ca127e5ed1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "91d136f49faa4fbf98e89b7501c1fcf1": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d4b42a978220400cafb0c87fe4cdbd9d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "d32888b6ba354a5082e46456ce3645f6": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7779a29f6b474020854c6e5a3e038a14": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}