DevRev_Experimentation.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/ris27hav/devrevs_domain_specific_qa/blob/main/DevRev_Experimentation.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PJ8_n40KJy4x"
      },
      "source": [
        "# Index"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NlLJZRP2DoRK"
      },
      "source": [
        "1. [Problem Statement](#scrollTo=X09KpIhKxg5u&line=1&uniqifier=1) - &nbsp; [ [Tasks](#scrollTo=IhtknNSOxov1), [Testing](#scrollTo=YNqBAAW30bqv), [Metrics](#scrollTo=W-tz9U0a3f_9), [Report](#scrollTo=ZMcLSris3iYo), [Scoring](#scrollTo=zQDl8wvpj63h), [References](#scrollTo=E-IXE0rNkBGN) ]\n",
        "\n",
        "2. [Ideas](#scrollTo=apbtofVj0YJk&line=1&uniqifier=1) - &nbsp; [ [Challenges](#scrollTo=aJJLOMpR4Tt4&line=47&uniqifier=1), [Possible Solutions](#scrollTo=lI-1Glwx4N5F&line=7&uniqifier=1), [General Tips](#scrollTo=92m-HyXU4HZN&line=9&uniqifier=1), [Keywords](#scrollTo=cW5GL6PC5NS7&line=1&uniqifier=1), [Experimentation](#scrollTo=8-2qFzDi4Cqs&line=2&uniqifier=1) ]\n",
        "\n",
        "3. [Implementation](#scrollTo=SeLEtl22zRG6&line=1&uniqifier=1)\n",
        "    * [Loading the provided dataset](#scrollTo=O0zouMSUjLPM&line=1&uniqifier=1)\n",
        "    * [Task 1 - Paragraph Retrieval](#scrollTo=ecsInjmdhs75&line=1&uniqifier=1)\n",
        "        * [Common pipeline](#scrollTo=YUWiAAYL05A3&line=1&uniqifier=1)\n",
        "        * [Approach 1: By breaking passages into sentences and calculating sentence embeddings](#scrollTo=PhXWSARZiuMd&line=1&uniqifier=1)\n",
        "            1. [Universal Sentence Encoder](#scrollTo=dwyy9cNQGAvO&line=1&uniqifier=1)\n",
        "            2. [SimCSE](#scrollTo=JNT7uxNNjCEs&line=1&uniqifier=1)\n",
        "        \n",
        "        * [Approach 2: By calculating embeddings of the complete passages](#scrollTo=O4D8Xx5ipGzd&line=1&uniqifier=1)\n",
        "            1. [RocketQA](#scrollTo=Alr1OJ8DGNwW&line=1&uniqifier=1)\n",
        "            2. [DPR Reader](#scrollTo=aqsql7om36uM&line=1&uniqifier=1)\n",
        "            3. [DensePhrases (Incomplete)](#scrollTo=sAbNo9yu1zWf)\n",
        "    \n",
        "    * [Task 2 - Context based domain specific Question-Answering](#scrollTo=KM5jpE-4h4kn&line=1&uniqifier=1)\n",
        "        * [Testing pretrained Question-Answering models using transformers library](#scrollTo=vbESJ9kgIlrC&line=1&uniqifier=1)\n",
        "        \n",
        "        * [Using transfer learning to boost performance on specific-domains](#scrollTo=yIuVn33uuC6m&line=1&uniqifier=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X09KpIhKxg5u"
      },
      "source": [
        "# Problem Statement"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IhtknNSOxov1"
      },
      "source": [
        "#### Task 1\n",
        "\n",
        "Given a question and a set of paragraphs, predict if the question can be answered with the given paragraphs. If yes, <u>return the paragraph</u> that answers the question. Each question and paragraph is associated with a <u>specific theme</u>. This could be “Sports”, “English” or “Mathematics” etc. A question of a theme can be answered by <u>one of the paragraphs</u> in that theme.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8ex-yoSVxxAB"
      },
      "source": [
        "#### Task 2\n",
        "\n",
        "For the given questions, also predict the <u>exact answer</u> from the predicted paragraph. Predict the <u>start_index</u> and the <u>answer_text</u> field for the given question. Note: Both the tasks will be marked individually. However, to perform better in Task 2, your model needs to perform better in Task 1."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y8iXFe0qzjS4"
      },
      "source": [
        "#### Training Dataset\n",
        "\n",
        "The dataset contains the following fields:\n",
        "1. Question: Question for which answer is to be found\n",
        "2. Theme: Name of the domain this question & paragraph belongs to. For e.g.  “cricket”, “mathematics”, “biology” etc.\n",
        "3. Paragraph: Paragraph from the mentioned theme which may contain the answer\n",
        "4. Answer_possible: If the answer is possible from the given paragraph\n",
        "5. Answer_text: Answers from the given paragraph\n",
        "6. Answer_start: Index position from where the answer starts"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YNqBAAW30bqv"
      },
      "source": [
        "#### Testing"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k9UEcB-e1GLX"
      },
      "source": [
        "##### **ROUND 1**\n",
        "\n",
        "Goal for the first round is to test how well your model performs for\n",
        "questions and paragraphs for any <u>new theme</u>. For example: your training dataset\n",
        "contained questions and paragraphs for the theme “Cricket”. The model will be\n",
        "tested on a new theme, say “Football”.\n",
        "\n",
        "**Test Input**\n",
        "1. CSV containing list of paragraphs.\n",
        "2. CSV containing list of questions that should be answered from the given list of paragraphs.\n",
        "\n",
        "**Note:** The test input will be shared with the participant and will be given <u>enough time to fine tune for new themed paragraphs</u> and run inference over new sets of questions.\n",
        "\n",
        "**Deliverables**\n",
        "1. Training notebook - Colab notebook that was used to train and export the models (Model can be exported to any choice of format).\n",
        "2. Inference notebook - Colab notebook that uses the exported models and the test files to perform predictions and returns the predictions in the expected format.\n",
        "3. The expected format is a csv file that contains the following fields:\n",
        "\n",
        "    * Question\n",
        "    * Paragraph (Empty if the question can’t be answered from any given paragraph)\n",
        "    * Answer_start: Start index of the answer that has been correctly predicted\n",
        "    * Answer_text: Answer from the given chosen paragraph\n",
        "\n",
        "Note: The result for this test set 1 will be released to the candidates and they will be provided enough time to fine tune their models."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "umj6FUki1xbX"
      },
      "source": [
        "##### **ROUND 2**\n",
        "The goal for the second round is to see how well and efficiently you can\n",
        "<u>fine tune the models for each theme after receiving sample question-answer pairs</u> for that theme. Here, the test data contains questions and paragraphs from the theme that were shared with you as a training dataset and test dataset of Round 1. For e.g., you already have some question answer pairs for a given theme say “Kubernetes”. How does the fine tuned model work for any new question related to the theme “Kubernetes”?\n",
        "\n",
        "**Test Input**\n",
        "1. CSV containing list of paragraphs.\n",
        "2. CSV containing list of questions that should be answered from the given list of paragraphs.\n",
        "**Note**: The test input will be shared with the participant and they will be given <u>2 hours to run inference and submit their deliverables</u>. Teams are allowed to finetune their models on the data that was shared with them in Round 1.\n",
        "\n",
        "**Deliverables**\n",
        "1. Training notebook - Colab notebook that was used to train and finetune the models using previous datasets. The notebook should export the models. (Model can be exported to any choice of format).\n",
        "2. Inference notebook - Colab notebook that uses the exported models and the test input to perform predictions and returns the predictions in the expected CSV format.\n",
        "3. The expected format is a csv file that contains the following fields:\n",
        "    * Question\n",
        "    * Paragraph (Empty if the question can’t be answered from any given paragraph)\n",
        "    * Answer_start: Start index of the answer that has been predicted\n",
        "    * Answer_text: Answer from the given chosen paragraph."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q99Cc__W2djn"
      },
      "source": [
        "##### **KEY POINTS TO NOTE**\n",
        "* Training notebook when run over the given dataset must be able to reproduce\n",
        "the exported models used during inference for both Round 1 and Round 2.\n",
        "* The results csv file should be named “TeamName_predictions_1” for Round 1\n",
        "and \"TeamName_predictions_2” for Round 2.\n",
        "* Any mismatch between the submitted csv file and the file generated from the\n",
        "inference notebook will lead to disqualification."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W-tz9U0a3f_9"
      },
      "source": [
        "#### Metrics"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-5Lsd1mDj0FV"
      },
      "source": [
        "* F1 score for paragraph search task.\n",
        "* F1 score for QA task.\n",
        "* There will be <u>different weightage to different themes</u> such that even if the model is overfit over publicly available QA dataset, it doesn’t help much with the final score.\n",
        "* <u>Average inference time</u> for each question must be <u>less than 200ms</u>. If average inference time is above that, it would be penalized accordingly.\n",
        "* Your inference collab notebook must <u>run within 4GB memory</u>."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZMcLSris3iYo"
      },
      "source": [
        "#### Report"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2ia4EAAlj3yH"
      },
      "source": [
        "Apart from the above mentioned deliverables, you would also be required to submit the mid-term and end-term report which should necessarily include the following:\n",
        "* Literature review\n",
        "* Different techniques evaluated and their metric score\n",
        "* Final technique being used, latency and accuracy metric corresponding to it\n",
        "* Future work\n",
        "* References\n",
        "\n",
        "The tentative date for mid-term evaluation is <u>10th January 2023</u>. Final dates will be communicated soon."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zQDl8wvpj63h"
      },
      "source": [
        "#### Scoring"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "adqw-Epn3qSr"
      },
      "source": [
        "Final score would comprise of following components:\n",
        "* 25% - Midterm report\n",
        "* 20% - F1 score for paragraph prediction\n",
        "* 20% - F1 score for QA task\n",
        "* 35% - Rest of the score would include\n",
        "    * End term report\n",
        "    * Code work\n",
        "    * Presentation, QA round"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E-IXE0rNkBGN"
      },
      "source": [
        "#### References"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bw2pnZuD7x_7"
      },
      "source": [
        "- [Know What You Don’t Know: Unanswerable Questions for SQuAD](https://arxiv.org/pdf/1806.03822.pdf)\n",
        "\n",
        "- F1 scores for QA task\n",
        "\n",
        "    https://www.tensorflow.org/hub/tutorials/tf2_semantic_approximate_nearest_neighbors\n",
        "    https://www.tensorflow.org/hub/tutorials/retrieval_with_tf_hub_universal_encoder_qa\n",
        "\n",
        "- [Fine tuning](https://deeplizard.com/learn/video/5T-iXNNiwIs)\n",
        "\n",
        "- [Faiss - Approximate nearest neighbours search](https://github.com/facebookresearch/faiss)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "veO8FHUOXmuj"
      },
      "source": [
        "#### QnA Session: Brief"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LeA5AjoKXrnb"
      },
      "source": [
        "Training\n",
        "- Constraint on training resources\n",
        "    - Teams are allowed to train their model whichever way they want but the same must be reproducible while running your training collab notebook on a system with specifics\n",
        "similar to free tier google collab system with\n",
        "        - No hardware accelerator\n",
        "        - 12 GB System RAM\n",
        "        - Within 12 hrs\n",
        "- Pre-trained model\n",
        "    - Open source model published before 1 Dec’22\n",
        "    - This needs to be mentioned in the end term report with valid reference.\n",
        "- Dataset\n",
        "    - Teams are supposed to preprocess the dataset and clean the same to align it to the given testing task.\n",
        "    - They are not allowed to use any other publicly available dataset. However, they can use the given dataset to create synthetic QA dataset.\n",
        "\n",
        "Testing\n",
        "- Resource constraint\n",
        "    - Inference notebook must run on a system with specifics similar to free tier google collab system with\n",
        "        - No hardware accelerator\n",
        "        - 12 GB System RAM (Note the increase from 4 GB to 12 GB)\n",
        "- Test input\n",
        "    - List of paragraphs containing paragraph_id -> (paragraph, theme) mapping\n",
        "    - List of questions with theme\n",
        "- Test output\n",
        "    - List of questions with predicted paragraph id and answer text\n",
        "- Metrics\n",
        "    - Accuracy metric for paragraph prediction:\n",
        "        - True positive: If the predicted paragraph exists in the ground truth list of paragraphs which can answer the query.\n",
        "        - True negative: If predicted that there does not exist a paragraph which can answer the query and that indeed is the case.\n",
        "        - Instead of F1(as originally mentioned in PS), we’ll be evaluating the accuracy metric:\n",
        "            - Accuracy: (True positive + True negative) / (Total number of queries)\n",
        "    - F1 score for QA task:\n",
        "        - For a given query, assume there are 3 answers in ground truth: “random token word”, “token word problem”, “word problem pushed”.\n",
        "        - For a predicted answer, “problem pushed”, it’ll calculate the maximum F1 score while comparing it with all the 3 possible answers.\n",
        "        - Final score for a theme would be avg. F1 score over all queries in that theme.\n",
        "    - Inference time\n",
        "        - Metric score for a theme would be F1 score Q/A task + Accuracy for paragraph prediction\n",
        "        - If your average inference time(AIT) for a theme is greater than 200 ms then,\n",
        "        - Final score for theme = (200/AIT(ms)) * Metric score for theme\n",
        "    - Final score\n",
        "        - Final score = ∑ theme_weight * (final score for that theme)\n",
        "        - Theme weight would not be exposed to teams."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "apbtofVj0YJk"
      },
      "source": [
        "# Ideas"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PzvYIQsR5Mwd"
      },
      "source": [
        "Although there are many pretrained Question-Answering SOTA models that produce excellent results on the training dataset provided by DevRev (its actually part of squad 2.0 dataset), they pose a few critical challenges for us."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aJJLOMpR4Tt4"
      },
      "source": [
        "#### CHALLENGES\n",
        "\n",
        "1. <u>High inference time</u> for pretrained question answering models. Inference time increases with the **number of parameters** in the model and the **length of the paragraph**.<br><br> The models given below were evaluated on 500 random answerable and non-answerable examples each (i.e. 1000 examples in total). There is a tradeoff between model size and inference time.\n",
        "---\n",
        "| S. No. | Model | Number of Parameters | Model Size (Quantized) | Average inference time (*using onnxruntime) | EM Score | F1 Score |\n",
        "|:------:|:-------:|:-------:|:------:|:-------:|:-------:|:-------:|\n",
        "| 1 | twmkn9/albert-base-v2-squad2 | 11 million | 46.7 MB | 741 ms | 0.917 | 0.948 |\n",
        "| 2 | deepset/roberta-base-squad2 | 124 million | 497 MB | 702 ms (*568 ms) | 0.880 | 0.914 |\n",
        "| 3 | deepset/minilm-uncased-squad2 | 33 million | 127 MB (81 MB) | 183 ms (*149 ms) | 0.867 (0.882) | 0.901 (0.914) |\n",
        "| 4 | deepset/electra-base-squad2 | 106 million | 436 MB (233 MB) | 587 ms (*489 ms) | 0.992 (0.88) | 0.993 (0.908) |\n",
        "| 5 | ktrapeznikov/albert-xlarge-v2-squad-v2 | 54 million | 235 MB | 7530 ms | --- | --- |\n",
        "\n",
        "---\n",
        "\n",
        "&nbsp;\n",
        "\n",
        "> **Note** - It can be seen from the below data that using *onnx runtime* results in lower inference time. Further optimization can be done using optimizer and quantizer.\n",
        ">\n",
        "> | Model Used | CPU Inference Time | GPU Inference Time |\n",
        "> |------------|--------------------|--------------------|\n",
        "> | AutoModelForQuestionAnswering  |  167 ms ± 7.48 ms  |  141 ms ± 12.9 ms  |\n",
        "> |  ORTModelForQuestionAnswering  |  128 ms ± 1.25 ms  |  107 ms ± 10.8 ms  |\n",
        ">\n",
        ">\n",
        "\n",
        "&nbsp;\n",
        "\n",
        "2. <u>Low semantic similarity</u> between question and paragraph: Finding semantic similarity between question and sentences from the paragraphs doesn't always gives the standout winner. This is because question can be from a small fraction of a sentence. Thus the question and the sentence that answers the question will not be too close.\n",
        "\n",
        "---\n",
        "| S. No. | Model for Embeddings | Model Size | Dimensions | Total queries tested | Relevant para found in Top 10 results (%) | Mean Rank\n",
        "|:------:|:-------:|:-------:|:-------:|:------:|:-------:|:-------:|\n",
        "| 1 | Google's Universal Sentence Encoder | 523 MB | 512 | 50125 | 46954 (93.67 %) | 1.6 |\n",
        "| 2 | SimCSE (Sentence Encoder) | 438 MB | 768 | 3138 | 2693 (85.82 %) | 1.8 |\n",
        "| 3 | RocketQA (Context Encoder) | 769 MB | 768 | 523 | 420 (80 %) | 2.36 |\n",
        "| 4 | DPR Reader (*Just using Context Encoder) | 438 MB  | 768 | 523 | 400 (76 %) | 2.62 |\n",
        "\n",
        "---\n",
        "\n",
        "&nbsp;\n",
        "\n",
        "> Remarks -\n",
        "> 1. There can be multiple passages that answer a question. This is not taken into account for getting above results. It will only improve the stats.\n",
        "> 2. Using embeddings for sentence is better that creating embeddings for entire passages.\n",
        "> 3. Even though we get top 10 passage for a question within negligible time, its still difficult to get the exact passage that contains the answer or to judge if no passage contains the answer.\n",
        ">\n",
        "> Try running the QA model directly on the top 10 sentences rather than running it on the entire passage (faster). Or maybe use some other way to find the best paragraph that may contain the answer and run QA model on that."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lI-1Glwx4N5F"
      },
      "source": [
        "#### POSSIBLE SOLUTIONS\n",
        "\n",
        "1. Find best k paragraphs by semantic similarity -> find best m senteneces with window of, say 1 sentence before and after that sentence to reduce the paragraph -> apply the model on that.\n",
        "\n",
        "2. Train a model for identifying the paragraphs if an answer is possible. Use the given dataset (which is part of squad 2.0 dataset) to generate data. For the model architecture, find the most influential and efficient approach used from the research papers.\n",
        "\n",
        "3. Ensemble Learning - Apply multilple models - apply weighted average to get the final score - sort based on these score. For e.g. one based on syntax and the other based on semantics."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "92m-HyXU4HZN"
      },
      "source": [
        "#### GENERAL TIPS\n",
        "\n",
        "Modular architecture. Create methods for the pipeline and document what it does. Give assumptions, inputs, outputs, side effects, approach (if helpful)\n",
        "\n",
        "Open your Chrome DevTools by pressing F12 or ctrl+shift+i on Linux and enter the following JavaScript snippet in your console:\n",
        "```\n",
        "function KeepClicking(){\n",
        "    console.log(\"Clicking\");\n",
        "    document.querySelector(\"colab-connect-button\").click()\n",
        "}\n",
        "setInterval(KeepClicking,60000)\n",
        "```\n",
        "This function makes a click on the connect-button every 60 seconds. Thus, Colab thinks that the notebook is not idle and you don’t have to worry about being disconnected!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cW5GL6PC5NS7"
      },
      "source": [
        "#### KEYWORDS\n",
        "\n",
        "Extractive Question-Answering, Information Retrieval (IR), embeddings, transformers, semantic similarity"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8-2qFzDi4Cqs"
      },
      "source": [
        "#### EXPERIMENTATION\n",
        "\n",
        ">Things to Ponder\n",
        ">\n",
        "> Extractive or abstractive? How much tradeoff between F1-Score and inference time? How much does time making sentence embedding takes? What about in finding k-nearest neghbours? Giving sentence embeddings as input? With query embedding?\n",
        "Domain Specific? Combine results from sentence encoder and paragraph encoder?\n",
        ">\n",
        "><br>\n",
        ">\n",
        ">To Do\n",
        ">\n",
        ">1. Accuracy for Paragraph Retrieval, Theme wise Metrics\n",
        ">2. Store and Load Embeddings\n",
        ">3. Better Context Generation - currently passing the top k nearest sentences as context\n",
        ">4. Better QA model (in terms of inference time)\n",
        ">5. Fine-tuning to a given theme - (also generate more QAs from given paragraphs)\n",
        ">6. faiss vs simpleneighbours and other libraries\n",
        ">7. **[URGENT]** Create a validation dataset. Scan given dataset and possibly keep those questions that are not part of squad 2.0 - Different for Round 1 and 2\n",
        ">8. In the sentence encoder, for context, can try giving one sentence before and after instead of giving whole passage\n",
        ">9. k value based on avg sentence length in theme paragraphs."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SeLEtl22zRG6"
      },
      "source": [
        "# Implementation"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Load Data"
      ],
      "metadata": {
        "id": "APhlBAlK2-Aa"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O0zouMSUjLPM"
      },
      "source": [
        "### Load Train Dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EouaXrxN3uCu"
      },
      "outputs": [],
      "source": [
        "import csv\n",
        "import requests\n",
        "\n",
        "# load training dataset\n",
        "def load_data():\n",
        "    CSV_URL = 'https://drive.google.com/u/0/uc?id=1Z-yb752A3o7b9dqrGt24XU0sl53FVqya&export=download'\n",
        "\n",
        "    with requests.Session() as s:\n",
        "        download = s.get(CSV_URL)\n",
        "        decoded_content = download.content.decode('utf-8')\n",
        "        cr = csv.reader(decoded_content.splitlines(), delimiter=',')\n",
        "        train_data = list(cr)\n",
        "\n",
        "    print(f\"Number of examples = {len(train_data)}\")\n",
        "    ans, noans = 0, 0\n",
        "    for x in train_data:\n",
        "        if x[4] == 'False':\n",
        "            noans += 1\n",
        "        else:\n",
        "            ans += 1\n",
        "    print(f\"\\tAnswerable questions = {ans}\")\n",
        "    print(f\"\\tNon-Answerable questions = {noans}\\n\")\n",
        "    print(\"Examples:\")\n",
        "    for i in [0, 1000, 1300]:\n",
        "        print(' | '.join(train_data[i][:2]), ' | ', train_data[i][2][:20] + '...', ' | ', ' | '.join(train_data[i][3:]))\n",
        "    return train_data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TzgNTXeZjV6M"
      },
      "outputs": [],
      "source": [
        "def load_theme_wise_data(train_data):\n",
        "    theme_wise_data = {}\n",
        "    for x in train_data[1:]:\n",
        "        if x[1] not in theme_wise_data:\n",
        "            theme_wise_data[x[1]] = {\n",
        "                'para': [],\n",
        "                'ques': [],\n",
        "                'ans': []\n",
        "            }\n",
        "        if x[2] not in theme_wise_data[x[1]]['para']:\n",
        "            theme_wise_data[x[1]]['para'].append(x[2])\n",
        "        theme_wise_data[x[1]]['ques'].append(x[3])\n",
        "        # ans contains a list -> [Para_Number, Answer_possible, Answer_text, Answer_start]\n",
        "        theme_wise_data[x[1]]['ans'].append([theme_wise_data[x[1]]['para'].index(x[2])] + x[4:])\n",
        "    print(f'\\nTotal {len(theme_wise_data)} themes present.')\n",
        "    return theme_wise_data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2K5L7ul5-VOB"
      },
      "outputs": [],
      "source": [
        "def load_ques_by_theme(theme, theme_wise_data, answerable_only = False):\n",
        "    paras = theme_wise_data[theme]['para']\n",
        "    ques = []\n",
        "    gold_para = []\n",
        "    ans = []\n",
        "    for i in range(len(theme_wise_data[theme]['ques'])):\n",
        "        if answerable_only and theme_wise_data[theme]['ans'][i][1] == 'False':\n",
        "            continue\n",
        "        ques.append(theme_wise_data[theme]['ques'][i])\n",
        "        gold_para.append(theme_wise_data[theme]['ans'][i][0])\n",
        "        ans.append(theme_wise_data[theme]['ans'][i][1:])\n",
        "\n",
        "    print(\"Total Questions:\", len(ques))\n",
        "    print(\"Total Paragraphs:\", len(paras))\n",
        "    return paras, ques, gold_para, ans"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Load Test Data"
      ],
      "metadata": {
        "id": "oCWcYdGN2jvq"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Save missing Squad 2.0 dataset for testing"
      ],
      "metadata": {
        "id": "TMAE9ftF1S6h"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install datasets"
      ],
      "metadata": {
        "id": "VhhX7lRxpxzs"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from datasets import load_dataset, load_metric"
      ],
      "metadata": {
        "id": "ReLdYTlzqWwI"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Btx057kecBEF"
      },
      "outputs": [],
      "source": [
        "sqd = load_dataset(\"squad_v2\")"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sqd"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HxNPnWyHsI-_",
        "outputId": "d2d24fe4-dd1b-4aba-d641-dd6431fd8f43"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "DatasetDict({\n",
              "    train: Dataset({\n",
              "        features: ['id', 'title', 'context', 'question', 'answers'],\n",
              "        num_rows: 130319\n",
              "    })\n",
              "    validation: Dataset({\n",
              "        features: ['id', 'title', 'context', 'question', 'answers'],\n",
              "        num_rows: 11873\n",
              "    })\n",
              "})"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sqd['train'][0]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "i5hxOno2t6Li",
        "outputId": "348b4e9f-a6c5-4638-a576-d5560fb4e140"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'id': '56be85543aeaaa14008c9063',\n",
              " 'title': 'Beyoncé',\n",
              " 'context': 'Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&B girl-group Destiny\\'s Child. Managed by her father, Mathew Knowles, the group became one of the world\\'s best-selling girl groups of all time. Their hiatus saw the release of Beyoncé\\'s debut album, Dangerously in Love (2003), which established her as a solo artist worldwide, earned five Grammy Awards and featured the Billboard Hot 100 number-one singles \"Crazy in Love\" and \"Baby Boy\".',\n",
              " 'question': 'When did Beyonce start becoming popular?',\n",
              " 'answers': {'text': ['in the late 1990s'], 'answer_start': [269]}}"
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "data = load_data()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 182
        },
        "id": "F9QSN900qCVQ",
        "outputId": "52bc4513-50df-4662-eedc-b00c1a4334f2"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Number of examples = 75056\n",
            "\tAnswerable questions = 50126\n",
            "\tNon-Answerable questions = 24930\n",
            "\n",
            "Examples:\n",
            " | Theme  |  Paragraph...  |  Question | Answer_possible | Answer_text | Answer_start\n",
            "1430 | Frédéric_Chopin  |  Some modern commenta...  |  Who said Chopin's works were modeled after Bach, Beethoven, Schubert and Field? | True | ['Richard Taruskin'] | [543]\n",
            "2196 | The_Legend_of_Zelda:_Twilight_Princess  |  Twilight Princess ta...  |  Who releases Bulbins from the Realm of Twilight? | False | [] | []\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(len(data), len(sqd['train']), len(sqd['validation']))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ui-dyEw-1vpc",
        "outputId": "5fe58be6-db56-4d8f-f393-4c52f8b256ef"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "75056 130319 11873\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "gd, sdt, sdv = {}, {}, {}\n",
        "\n",
        "for x in data[1:]:\n",
        "    theme = x[1]\n",
        "    if theme not in gd:\n",
        "        gd[theme] = []\n",
        "    gd[theme].append(tuple(x[1:]))\n",
        "\n",
        "for x in sqd['train']:\n",
        "    theme = x['title']\n",
        "    if theme not in sdt:\n",
        "        sdt[theme] = []\n",
        "    y = [x['id'], x['title'], x['context'], x['question'], 'True' if x['answers']['text'] != [] else 'False', str(x['answers']['text']), str(x['answers']['answer_start'])]\n",
        "    sdt[theme].append(tuple(y[1:]))\n",
        "\n",
        "for x in sqd['validation']:\n",
        "    theme = x['title']\n",
        "    if theme not in sdv:\n",
        "        sdv[theme] = []\n",
        "    y = [x['id'], x['title'], x['context'], x['question'], 'True' if x['answers']['text'] != [] else 'False', str(x['answers']['text']), str(x['answers']['answer_start'])]\n",
        "    sdv[theme].append(tuple(y[1:]))"
      ],
      "metadata": {
        "id": "Grj3MlqWqu_l"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(len(gd.keys()), len(sdt.keys()), len(sdv.keys()))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PhlOYnS9tf44",
        "outputId": "cd3fe11f-75b2-4c6e-d2e4-f5ed67f918af"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "361 442 35\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "r1_test, r2_test = [], []\n",
        "\n",
        "for theme in sdt:\n",
        "    if theme not in gd:\n",
        "        r1_test += sdt[theme]\n",
        "\n",
        "for theme in sdv:\n",
        "    if theme not in gd:\n",
        "        r1_test += sdv[theme]\n",
        "\n",
        "for theme in gd:\n",
        "    if theme in sdt:\n",
        "        q1 = set(gd[theme])\n",
        "        q2 = set(sdt[theme])\n",
        "        r2_test += list(q2.difference(q1))\n",
        "    if theme in sdv:\n",
        "        q1 = set(gd[theme])\n",
        "        q2 = set(sdv[theme])\n",
        "        r2_test += list(q2.difference(q1))"
      ],
      "metadata": {
        "id": "2amlrg1-uv0c"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(len(r1_test), len(r2_test))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WsxJEhsWvVnS",
        "outputId": "48b3837b-6d27-4c68-bb9f-b15a86267052"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "34927 32402\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "r1_test[1000]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "c_1D5PyDv81w",
        "outputId": "a6566933-c9fd-4c50-c7a2-e99d31302d7b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "('Wayback_Machine',\n",
              " 'In Europe the Wayback Machine could be interpreted as violating copyright laws. Only the content creator can decide where their content is published or duplicated, so the Archive would have to delete pages from its system upon request of the creator. The exclusion policies for the Wayback Machine may be found in the FAQ section of the site. The Wayback Machine also retroactively respects robots.txt files, i.e., pages that currently are blocked to robots on the live web temporarily will be made unavailable from the archives as well.',\n",
              " 'What may be found in the robots.txt files section of the site?',\n",
              " 'False',\n",
              " '[]',\n",
              " '[]')"
            ]
          },
          "metadata": {},
          "execution_count": 65
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "r1_test = [[i] + list(data) for i, data in enumerate(r1_test)]\n",
        "r2_test = [[i] + list(data) for i, data in enumerate(r2_test)]"
      ],
      "metadata": {
        "id": "mHA0CJcZBpGF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import csv\n",
        "\n",
        "with open('test_data_round_1.csv', 'w') as f:\n",
        "    csvwriter = csv.writer(f)\n",
        "    csvwriter.writerows(r1_test)\n",
        "\n",
        "with open('test_data_round_2.csv', 'w') as f:\n",
        "    csvwriter = csv.writer(f)\n",
        "    csvwriter.writerows(r2_test)"
      ],
      "metadata": {
        "id": "Adb8psU9yAQo"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import drive\n",
        "\n",
        "drive.mount('/content/gdrive/', force_remount=True)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Tde18eYPBLOd",
        "outputId": "60a7a631-a05d-4b63-c86c-8dab77743b6e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/gdrive/\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!cp test_data_round_1.csv \"/content/gdrive/MyDrive/DevRev Test Data\"\n",
        "!cp test_data_round_2.csv \"/content/gdrive/MyDrive/DevRev Test Data\""
      ],
      "metadata": {
        "id": "_VNnJBhBKbFo"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Load Test Data"
      ],
      "metadata": {
        "id": "lFDwNSB-20gU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import csv\n",
        "import requests\n",
        "\n",
        "# load test dataset\n",
        "def load_test_data():\n",
        "    CSV_URL_R1 = 'https://drive.google.com/u/0/uc?id=1-56-cMKze05gTCtKjItBsxiXlPel4tpi&export=download'\n",
        "    CSV_URL_R2 = 'https://drive.google.com/u/0/uc?id=1-8_iovhHzNEHjvnpzp-I9BfrozCHS0NQ&export=download'\n",
        "\n",
        "    with requests.Session() as s:\n",
        "        download = s.get(CSV_URL_R1)\n",
        "        decoded_content = download.content.decode('utf-8')\n",
        "        cr = csv.reader(decoded_content.splitlines(), delimiter=',')\n",
        "        test_data_r1 = list(cr)\n",
        "\n",
        "        download = s.get(CSV_URL_R2)\n",
        "        decoded_content = download.content.decode('utf-8')\n",
        "        cr = csv.reader(decoded_content.splitlines(), delimiter=',')\n",
        "        test_data_r2 = list(cr)\n",
        "\n",
        "    for r, d in zip(['Round 1', 'Round 2'], [test_data_r1, test_data_r2]):\n",
        "        print(r)\n",
        "        print(f\"Number of examples = {len(d)}\")\n",
        "        ans, noans = 0, 0\n",
        "        for x in d:\n",
        "            if x[4] == 'False':\n",
        "                noans += 1\n",
        "            else:\n",
        "                ans += 1\n",
        "        print(f\"\\tAnswerable questions = {ans}\")\n",
        "        print(f\"\\tNon-Answerable questions = {noans}\\n\")\n",
        "        print(\"Examples:\")\n",
        "        for i in [0, 1000]:\n",
        "            print(' | '.join(d[i][:2]), ' | ', d[i][2][:20] + '...', ' | ', ' | '.join(d[i][3:]))\n",
        "        print()\n",
        "\n",
        "    return test_data_r1, test_data_r2"
      ],
      "metadata": {
        "id": "3exMX7g93InP"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Load Train and Test data v2"
      ],
      "metadata": {
        "id": "ON6k765nJqTz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import drive\n",
        "\n",
        "drive.mount('/content/gdrive/', force_remount=True)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XqenM03XS6EQ",
        "outputId": "045e71eb-c963-4699-85f1-cb26ca4f147b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/gdrive/\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "r1, r2 = load_test_data()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rSLg8fEEJyRk",
        "outputId": "00ad86b3-a3db-4289-ad6b-17c20ddd12f9"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Round 1\n",
            "Number of examples = 34927\n",
            "\tAnswerable questions = 20858\n",
            "\tNon-Answerable questions = 14069\n",
            "\n",
            "Examples:\n",
            "0 | IPod  |  The iPod is a line o...  |  Which company produces the iPod? | True | ['Apple'] | [105]\n",
            "1000 | Wayback_Machine  |  In Europe the Waybac...  |  What may be found in the robots.txt files section of the site? | False | [] | []\n",
            "\n",
            "Round 2\n",
            "Number of examples = 32402\n",
            "\tAnswerable questions = 21919\n",
            "\tNon-Answerable questions = 10483\n",
            "\n",
            "Examples:\n",
            "0 | Beyoncé  |  On January 7, 2012, ...  |  Jay Z has a website called what? | True | ['Lifeandtimes.com'] | [216]\n",
            "1000 | New_York_City  |  The Queensboro Bridg...  |  The Queensboro Bridge utilized what type of construction? | True | ['cantilever'] | [47]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "tr = load_theme_wise_data(r2)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9tlDwIMsJy3X",
        "outputId": "9266acda-9737-416f-9299-806ad8326c68"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Total 361 themes present.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "paras, ques, themes, ans = [], [], [], []\n",
        "pid, qid = 1, 1\n",
        "for theme in tr:\n",
        "    pstart, qstart = pid, qid\n",
        "    for p in tr[theme]['para']:\n",
        "        paras.append([pid, p, theme])\n",
        "        pid += 1\n",
        "    for q, a in zip(tr[theme]['ques'], tr[theme]['ans']):\n",
        "        ques.append([qid, q, theme])\n",
        "        if a[1] == 'True':\n",
        "            ans.append([qid, [pstart + a[0]], a[2]])\n",
        "        else:\n",
        "            ans.append([qid, [], a[2]])\n",
        "        qid += 1\n",
        "    themes.append([theme, qstart, qid-1])\n",
        "\n",
        "with open('input_paragraph.csv', 'w') as f:\n",
        "    csvwriter = csv.writer(f)\n",
        "    csvwriter.writerow(['id', 'paragraph', 'theme'])\n",
        "    csvwriter.writerows(paras)\n",
        "\n",
        "with open('input_question.csv', 'w') as f:\n",
        "    csvwriter = csv.writer(f)\n",
        "    csvwriter.writerow(['id', 'question', 'theme'])\n",
        "    csvwriter.writerows(ques)\n",
        "\n",
        "with open('theme_interval.csv', 'w') as f:\n",
        "    csvwriter = csv.writer(f)\n",
        "    csvwriter.writerow(['theme', 'start', 'end'])\n",
        "    csvwriter.writerows(themes)\n",
        "\n",
        "with open('ground_truth.csv', 'w') as f:\n",
        "    csvwriter = csv.writer(f)\n",
        "    csvwriter.writerow(['question_id', 'paragraph_id', 'answers'])\n",
        "    csvwriter.writerows(ans)"
      ],
      "metadata": {
        "id": "SJEPM34NJ099"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# !cp input_paragraph.csv \"/content/gdrive/MyDrive/DevRev Test Data/R2\"\n",
        "# !cp input_question.csv \"/content/gdrive/MyDrive/DevRev Test Data/R2\"\n",
        "# !cp theme_interval.csv \"/content/gdrive/MyDrive/DevRev Test Data/R2\"\n",
        "!cp ground_truth.csv \"/content/gdrive/MyDrive/DevRev Test Data/R2\""
      ],
      "metadata": {
        "id": "YxYxfyf-S2dx"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ecsInjmdhs75"
      },
      "source": [
        "## Task 1: Paragraph Retrieval"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YUWiAAYL05A3"
      },
      "source": [
        "### Common Pipeline"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "t2U_MrGDB5p-",
        "outputId": "681fd6a7-8a52-46cd-86f4-c5e7b1e4d25f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting transformers\n",
            "  Downloading transformers-4.25.1-py3-none-any.whl (5.8 MB)\n",
            "\u001b[K     |████████████████████████████████| 5.8 MB 5.3 MB/s \n",
            "\u001b[?25hCollecting faiss-cpu\n",
            "  Downloading faiss_cpu-1.7.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.0 MB)\n",
            "\u001b[K     |████████████████████████████████| 17.0 MB 58.3 MB/s \n",
            "\u001b[?25hCollecting huggingface-hub<1.0,>=0.10.0\n",
            "  Downloading huggingface_hub-0.11.1-py3-none-any.whl (182 kB)\n",
            "\u001b[K     |████████████████████████████████| 182 kB 70.0 MB/s \n",
            "\u001b[?25hCollecting tokenizers!=0.11.3,<0.14,>=0.11.1\n",
            "  Downloading tokenizers-0.13.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.6 MB)\n",
            "\u001b[K     |████████████████████████████████| 7.6 MB 60.5 MB/s \n",
            "\u001b[?25hRequirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.8/dist-packages (from transformers) (4.64.1)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.8/dist-packages (from transformers) (2022.6.2)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.8/dist-packages (from transformers) (2.23.0)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.8/dist-packages (from transformers) (21.3)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.8/dist-packages (from transformers) (6.0)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.8/dist-packages (from transformers) (3.8.2)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.8/dist-packages (from transformers) (1.21.6)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.8/dist-packages (from huggingface-hub<1.0,>=0.10.0->transformers) (4.4.0)\n",
            "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging>=20.0->transformers) (3.0.9)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests->transformers) (2022.12.7)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests->transformers) (1.24.3)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from requests->transformers) (3.0.4)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests->transformers) (2.10)\n",
            "Installing collected packages: tokenizers, huggingface-hub, transformers, faiss-cpu\n",
            "Successfully installed faiss-cpu-1.7.3 huggingface-hub-0.11.1 tokenizers-0.13.2 transformers-4.25.1\n"
          ]
        }
      ],
      "source": [
        "!pip install transformers faiss-cpu"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "p1yKv4VAB8F3",
        "outputId": "cf68332a-caa2-4192-9d14-691f9a7d9f96"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Package punkt is already up-to-date!\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "execution_count": 16,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "from nltk import sent_tokenize\n",
        "from transformers import AutoModel, AutoTokenizer\n",
        "\n",
        "import faiss\n",
        "import matplotlib.pyplot as plt\n",
        "import nltk\n",
        "import tensorflow as tf\n",
        "import tensorflow_hub as hub\n",
        "import torch\n",
        "\n",
        "nltk.download('punkt')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Si0txAbrRS43"
      },
      "outputs": [],
      "source": [
        "def para_to_sentences(para):\n",
        "    \"\"\"Splits a para into sentences.\"\"\"\n",
        "    para = para.replace('\\n', ' ').replace('\\t', ' ').replace('\\x00', ' ')\n",
        "    return sent_tokenize(para)\n",
        "\n",
        "def load_sents_from_para(paras):\n",
        "    sents = []\n",
        "    para_id = []\n",
        "    for id, x in enumerate(paras):\n",
        "        new_sents = para_to_sentences(x)\n",
        "        sents += new_sents\n",
        "        para_id += [id]*len(new_sents)\n",
        "    return sents, para_id"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MjRvDNJLAKTw"
      },
      "outputs": [],
      "source": [
        "def load_encoder(encoder):\n",
        "    if encoder == 'Universal Sentence Encoder':\n",
        "        module_url = \"https://tfhub.dev/google/universal-sentence-encoder-large/5\"\n",
        "        model = hub.load(module_url)\n",
        "        return _, model\n",
        "    elif encoder == \"Universal Sentence Encoder for QA\":\n",
        "        module_url = \"https://tfhub.dev/google/universal-sentence-encoder-qa/3\"\n",
        "        model = hub.load(module_url)\n",
        "        return _, model\n",
        "    elif encoder == 'SimCSE':\n",
        "        tokenizer = AutoTokenizer.from_pretrained(\"princeton-nlp/sup-simcse-bert-base-uncased\")\n",
        "        model = AutoModel.from_pretrained(\"princeton-nlp/sup-simcse-bert-base-uncased\")\n",
        "        return tokenizer, model\n",
        "    else:\n",
        "        print(\"Encoder not found. Please choose one from - 1. Universal Sentence Encoder, 2. Universal Sentence Encoder for QA, 3. SimCSE\")\n",
        "\n",
        "\n",
        "def get_embeddings(sents, encoder, tokenizer, model, sents_type=\"default\"):\n",
        "    if encoder == 'Universal Sentence Encoder':\n",
        "        return model(sents)\n",
        "    elif encoder == 'Universal Sentence Encoder for QA':\n",
        "        if sents_type == \"Question\":\n",
        "            return model.signatures['question_encoder'](tf.constant(sents))['outputs']\n",
        "        return model.signatures['response_encoder'](\n",
        "            input = tf.constant(sents),\n",
        "            context = tf.constant(sents)\n",
        "        )['outputs']\n",
        "    elif encoder == 'SimCSE':\n",
        "        tokens = tokenizer(sents, padding=True, truncation=True, return_tensors=\"pt\")\n",
        "        with torch.no_grad():\n",
        "            embeds = model(**tokens, output_hidden_states=True, return_dict=True).pooler_output\n",
        "        return embeds\n",
        "    print(\"Encoder not found. Please choose one from - 1. Universal Sentence Encoder, 2. Universal Sentence Encoder for QA, 3. SimCSE\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "U5ab95PbR3jz"
      },
      "outputs": [],
      "source": [
        "def get_k_nearest_neighbours(sents_embed, ques_embed, k = 10):\n",
        "    index = faiss.IndexFlatL2(sents_embed.shape[1])\n",
        "    index.add(sents_embed)\n",
        "    return index.search(ques_embed, k)\n",
        "\n",
        "def sent_id_to_para_id(I, para_id):\n",
        "    return [[para_id[sent_idx] for sent_idx in I[i]] for i in range(len(I))]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6MMSN-PWUZsn"
      },
      "outputs": [],
      "source": [
        "def print_example(i, sents, ques, pred_para, I, D):\n",
        "    cur_ques = ques[i]\n",
        "    print('question:', cur_ques)\n",
        "    print('similar sentences:')\n",
        "    print('\\tTarget sentence | Predicted Para ID | Distance between target sentence and query')\n",
        "    for j in range(D.shape[1]):\n",
        "        target_sent = sents[I[i,j]]\n",
        "        print('\\t', target_sent, '|', pred_para[i][j], '|', D[i, j])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oe_YN3o_CuxL"
      },
      "outputs": [],
      "source": [
        "def getRank(pred, gold):\n",
        "    i = 1\n",
        "    for p in pred:\n",
        "        if p == gold:\n",
        "            return i\n",
        "        i += 1\n",
        "    return i\n",
        "\n",
        "# Mean Rank of most relevant passage\n",
        "def getMeanRank(preds, gold):\n",
        "    k = len(preds[0])\n",
        "    rankSum = 0\n",
        "    notInTopK = 0\n",
        "    for i, g in zip(preds, gold):\n",
        "        rank = getRank(i, g)\n",
        "        if rank > k:\n",
        "            notInTopK += 1\n",
        "        else:\n",
        "            rankSum += rank\n",
        "    avg = k+1\n",
        "    if len(gold) > notInTopK:\n",
        "        avg = rankSum / (len(gold) - notInTopK)\n",
        "    return notInTopK, avg\n",
        "\n",
        "\n",
        "# Histogram of ranks of the most relevant passage\n",
        "def showHistogram(preds, gold):\n",
        "    k = len(preds[0])\n",
        "    ranks = []\n",
        "    for i, g in zip(preds, gold):\n",
        "        rank = getRank(i, g)\n",
        "        if rank <= k:\n",
        "            ranks.append(rank)\n",
        "    plt.hist(ranks)\n",
        "    plt.xlabel(\"Rank of the most relevant passage\")\n",
        "    plt.ylabel(\"Count\")\n",
        "    plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aoRcov1ZTrAh"
      },
      "outputs": [],
      "source": [
        "def evaluate_results(pred_para, gold_para):\n",
        "    notInTopK, meanRank = getMeanRank(pred_para, gold_para)\n",
        "    print(\"Total queries:\", len(pred_para))\n",
        "    print(f\"In top {len(pred_para[0])} results, number of queries for which -\")\n",
        "    print(f\"\\tRelevant paragraph found: {len(pred_para) - notInTopK} ({100 - round(100.*notInTopK / len(pred_para))} %)\")\n",
        "    print(f\"\\tRelevant paragraph NOT found: {notInTopK} ({round(100.*notInTopK / len(pred_para))} %)\")\n",
        "    print(f\"Mean Rank for which relevant paragraph found: {round(meanRank, 2)}\\n\")\n",
        "\n",
        "    showHistogram(pred_para, gold_para)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PhXWSARZiuMd"
      },
      "source": [
        "### Approach 1: Breaking passage into sentences -> sentence embeddings"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dwyy9cNQGAvO"
      },
      "source": [
        "#### 1. Using Universal Sentence Encoder Large"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zfpNkAjB3OsK",
        "outputId": "e086471a-5112-40ac-d7c3-b5dac2d48746"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Number of examples = 75056\n",
            "\tAnswerable questions = 50126\n",
            "\tNon-Answerable questions = 24930\n",
            "\n",
            "Examples:\n",
            " | Theme  |  Paragraph...  |  Question | Answer_possible | Answer_text | Answer_start\n",
            "1430 | Frédéric_Chopin  |  Some modern commenta...  |  Who said Chopin's works were modeled after Bach, Beethoven, Schubert and Field? | True | ['Richard Taruskin'] | [543]\n",
            "2196 | The_Legend_of_Zelda:_Twilight_Princess  |  Twilight Princess ta...  |  Who releases Bulbins from the Realm of Twilight? | False | [] | []\n",
            "\n",
            "Total 361 themes present.\n"
          ]
        }
      ],
      "source": [
        "train_data = load_data()\n",
        "theme_wise_data = load_theme_wise_data(train_data)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CGRKuocjR5vh",
        "outputId": "7019acbc-d342-4a27-b3a1-9a08eeafbb85"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Total Questions: 523\n",
            "Total Paragraphs: 66\n"
          ]
        }
      ],
      "source": [
        "paras, ques, gold_para, _ = load_ques_by_theme('Beyoncé', theme_wise_data, answerable_only=True)\n",
        "sents, para_id = load_sents_from_para(paras)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MQERaXtQ7iNe"
      },
      "outputs": [],
      "source": [
        "encoder = 'Universal Sentence Encoder'\n",
        "_, model = load_encoder(encoder)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Pf1LulKyDBld"
      },
      "outputs": [],
      "source": [
        "sents_embed = get_embeddings(sents, encoder, _, model)\n",
        "ques_embed = get_embeddings(ques, encoder, _, model)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nZAh0yFl8KtJ"
      },
      "outputs": [],
      "source": [
        "k = 10\n",
        "D, I = get_k_nearest_neighbours(sents_embed, ques_embed, k)\n",
        "pred_para = sent_id_to_para_id(I, para_id)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pqPm2rbYPCCw",
        "outputId": "aae74090-f271-45fa-e2f1-b081f5f40509"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "question: When did Beyonce leave Destiny's Child and become a solo singer?\n",
            "similar sentences:\n",
            "\tTarget sentence | Predicted Para ID | Distance between target sentence and query\n",
            "\t Following the disbandment of Destiny's Child in June 2005, she released her second solo album, B'Day (2006), which contained hits \"Déjà Vu\", \"Irreplaceable\", and \"Beautiful Liar\". | 1 | 0.6500025\n",
            "\t As a solo artist she has sold over 15 million albums in the US, and over 118 million records worldwide (a further 60 million additionally with Destiny's Child), making her one of the best-selling music artists of all time. | 53 | 0.7991407\n",
            "\t Throughout a career spanning 19 years, she has sold over 118 million records as a solo artist, and a further 60 million with Destiny's Child, making her one of the best-selling music artists of all time. | 2 | 0.8047129\n",
            "\t In early 2001, while Destiny's Child was completing their third album, Beyoncé landed a major role in the MTV made-for-television film, Carmen: A Hip Hopera, starring alongside American actor Mekhi Phifer. | 8 | 0.8118446\n",
            "\t Beyoncé has won 20 Grammy Awards, both as a solo artist and member of Destiny's Child, making her the second most honored female artist by the Grammys, behind Alison Krauss and the most nominated woman in Grammy Award history with 52 nominations. | 54 | 0.82776654\n",
            "\t Beyoncé took a hiatus from music in 2010 and took over management of her career; her fourth album 4 (2011) was subsequently mellower in tone, exploring 1970s funk, 1980s pop, and 1990s soul. | 1 | 0.83074665\n",
            "\t Destiny's Child embarked on a worldwide concert tour, Destiny Fulfilled... and Lovin' It and during the last stop of their European tour, in Barcelona on June 11, 2005, Rowland announced that Destiny's Child would disband following the North American leg of the tour. | 11 | 0.8384073\n",
            "\t Beyoncé's younger sister Solange is also a singer and a former member of Destiny's Child. | 3 | 0.8389089\n",
            "\t She has received co-writing credits for most of the songs recorded with Destiny's Child and her solo efforts. | 36 | 0.88310707\n",
            "\t Beyoncé announced a hiatus from her music career in January 2010, heeding her mother's advice, \"to live life, to be inspired by things again\". | 17 | 0.92364967\n"
          ]
        }
      ],
      "source": [
        "print_example(0, sents, ques, pred_para, I, D)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 388
        },
        "id": "lKaYiyFvPB_Q",
        "outputId": "2b9129f3-b3a2-4e8b-d1bd-253b15eb920e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Total queries: 523\n",
            "In top 10 results, number of queries for which -\n",
            "\tRelevant paragraph found: 449 (86 %)\n",
            "\tRelevant paragraph NOT found: 74 (14 %)\n",
            "Mean Rank for which relevant paragraph found: 2.11\n",
            "\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAVP0lEQVR4nO3de7RedX3n8fcHgjegAiXNwoBGaSylzhJsZADRRUu9FFuBGctltQqt05QRLyhjB2xnlZnRytRbxzqDRaXgKoWhXCqCghhReuEWkDsyZBAkmUBStYg6osB3/ti/7DwcTk5OQp6zTzjv11rPevbz27fv2eec5/P89t7P3qkqJEkC2GboAiRJs4ehIEnqGQqSpJ6hIEnqGQqSpN68oQt4OnbddddatGjR0GVI0lblxhtv/Oeqmj/ZuK06FBYtWsTy5cuHLkOStipJ7t/QOHcfSZJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6W/U3mp+ORSdfNti67zvtjYOtW5KmYk9BktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJvbGFQpI9klyV5M4kdyR5d2s/NcmqJDe3x6Ej85ySZEWSu5O8fly1SZImN2+My34MOKmqbkqyI3BjkivbuI9X1UdGJ06yN3A08EvAC4CvJHlpVT0+xholSSPG1lOoqtVVdVMbfgS4C1g4xSyHAedV1aNV9S1gBbDfuOqTJD3VjBxTSLII2Be4rjW9I8mtSc5MsnNrWwg8MDLbSqYOEUnSFjb2UEiyA3AhcGJVfR84HdgT2AdYDXx0E5e3NMnyJMvXrl27xeuVpLlsrKGQZDu6QDinqi4CqKqHqurxqnoC+DTrdxGtAvYYmX331vYkVXVGVS2pqiXz588fZ/mSNOeM8+yjAJ8F7qqqj4207zYy2RHA7W34EuDoJM9O8mJgMXD9uOqTJD3VOM8+ehXwFuC2JDe3tvcDxyTZByjgPuAPAKrqjiTnA3fSnbl0gmceSdLMGlsoVNU/AJlk1BenmOeDwAfHVZMkaWp+o1mS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEm9sYVCkj2SXJXkziR3JHl3a98lyZVJ7mnPO7f2JPlEkhVJbk3yinHVJkma3Dh7Co8BJ1XV3sD+wAlJ9gZOBpZV1WJgWXsN8OvA4vZYCpw+xtokSZMYWyhU1eqquqkNPwLcBSwEDgPObpOdDRzehg8DPleda4Gdkuw2rvokSU81I8cUkiwC9gWuAxZU1eo26kFgQRteCDwwMtvK1jZxWUuTLE+yfO3atWOrWZLmorGHQpIdgAuBE6vq+6PjqqqA2pTlVdUZVbWkqpbMnz9/C1YqSRprKCTZji4Qzqmqi1rzQ+t2C7XnNa19FbDHyOy7tzZJ0gwZ59lHAT4L3FVVHxsZdQlwbBs+Fvj8SPtb21lI+wMPj+xmkiTNgHljXPargLcAtyW5ubW9HzgNOD/J24D7gSPbuC8ChwIrgB8BvzvG2iRJkxhbKFTVPwDZwOhDJpm+gBPGVY8kaeP8RrMkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqTetUEjyqum0SZK2btPtKfzFNNskSVuxeVONTHIAcCAwP8l7R0b9DLDtOAuTJM28KUMBeBawQ5tux5H27wNvHldRkqRhTBkKVfV14OtJzqqq+2eoJknSQDbWU1jn2UnOABaNzlNVvzqOoiRJw5huKPwt8CngM8Dj4ytHkjSk6YbCY1V1+lgrkSQNbrqnpH4hyduT7JZkl3WPqWZIcmaSNUluH2k7NcmqJDe3x6Ej405JsiLJ3Ulev5k/jyTpaZhuT+HY9vy+kbYCXjLFPGcBnwQ+N6H941X1kdGGJHsDRwO/BLwA+EqSl1aVu6okaQZNKxSq6sWbuuCqujrJomlOfhhwXlU9CnwryQpgP+CaTV2vJGnzTSsUkrx1svaqmtgLmI53tOUtB06qqu8BC4FrR6ZZ2domq2UpsBTghS984WasXpK0IdM9pvDKkcergVOBN23G+k4H9gT2AVYDH93UBVTVGVW1pKqWzJ8/fzNKkCRtyHR3H71z9HWSnYDzNnVlVfXQyDI+DVzaXq4C9hiZdPfWJkmaQZt76ewfApt8nCHJbiMvjwDWnZl0CXB0kmcneTGwGLh+M2uTJG2m6R5T+ALd2UbQXQjvF4HzNzLPucDBwK5JVgJ/AhycZJ+2rPuAPwCoqjuSnA/cCTwGnOCZR5I086Z7SuroKaSPAfdX1cqpZqiqYyZp/uwU038Q+OA065EkjcG0dh+1C+N9k+5KqTsDPxlnUZKkYUz3zmtH0u3j/y3gSOC6JF46W5KeYaa7++iPgFdW1RqAJPOBrwAXjKswSdLMm+7ZR9usC4TmO5swryRpKzHdnsLlSa4Azm2vjwK+OJ6SJElD2dg9mn8eWFBV70vyb4CD2qhrgHPGXZwkaWZtrKfw58ApAFV1EXARQJJ/1cb95lirkyTNqI0dF1hQVbdNbGxti8ZSkSRpMBsLhZ2mGPfcLVmIJGl4GwuF5Ul+f2Jjkn8H3DiekiRJQ9nYMYUTgYuT/DbrQ2AJ8Cy6C9pJkp5BpgyFdqnrA5P8CvCy1nxZVX117JVJkmbcdO+ncBVw1ZhrkSQNzG8lS5J6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6YwuFJGcmWZPk9pG2XZJcmeSe9rxza0+STyRZkeTWJK8YV12SpA0bZ0/hLOANE9pOBpZV1WJgWXsN8OvA4vZYCpw+xrokSRswtlCoqquB705oPgw4uw2fDRw+0v656lwL7JRkt3HVJkma3EwfU1hQVavb8IPAgja8EHhgZLqVre0pkixNsjzJ8rVr146vUkmagwY70FxVBdRmzHdGVS2pqiXz588fQ2WSNHfNdCg8tG63UHte09pXAXuMTLd7a5MkzaCZDoVLgGPb8LHA50fa39rOQtofeHhkN5MkaYbMG9eCk5wLHAzsmmQl8CfAacD5Sd4G3A8c2Sb/InAosAL4EfC746pLkrRhYwuFqjpmA6MOmWTaAk4YVy2SpOnxG82SpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpN68IVaa5D7gEeBx4LGqWpJkF+B/AYuA+4Ajq+p7Q9Q3botOvmyQ9d532hsHWa+krceQPYVfqap9qmpJe30ysKyqFgPL2mtJ0gyaTbuPDgPObsNnA4cPWIskzUlDhUIBX05yY5KlrW1BVa1uww8CCyabMcnSJMuTLF+7du1M1CpJc8YgxxSAg6pqVZKfA65M8s3RkVVVSWqyGavqDOAMgCVLlkw6jSRp8wzSU6iqVe15DXAxsB/wUJLdANrzmiFqk6S5bMZDIcn2SXZcNwy8DrgduAQ4tk12LPD5ma5Nkua6IXYfLQAuTrJu/X9TVZcnuQE4P8nbgPuBIweoTZLmtBkPhaq6F3j5JO3fAQ6Z6XokSevNplNSJUkDMxQkST1DQZLUMxQkST1DQZLUG+obzRrAUFdnBa/QKm0t7ClIknqGgiSpZyhIknqGgiSpZyhIknqGgiSpZyhIknp+T0EzYqjvSPj9CGnT2FOQJPUMBUlSz1CQJPUMBUlSz1CQJPUMBUlSz1CQJPX8noKe0byHhLRp7ClIknr2FKQx8Vvc2hoZCtIzjGGkp8PdR5Kknj0FSVu9IU8oGMq4emb2FCRJvVkXCknekOTuJCuSnDx0PZI0l8yq3UdJtgX+B/BaYCVwQ5JLqurOYSuTtDFzcRfOM9Fs6ynsB6yoqnur6ifAecBhA9ckSXPGrOopAAuBB0ZerwT+9egESZYCS9vLHyS5e4ZqG5ddgX8euohZxO3xZG6P9dwWI/Lfntb2eNGGRsy2UNioqjoDOGPoOraUJMurasnQdcwWbo8nc3us57Z4snFtj9m2+2gVsMfI691bmyRpBsy2ULgBWJzkxUmeBRwNXDJwTZI0Z8yq3UdV9ViSdwBXANsCZ1bVHQOXNW7PmF1hW4jb48ncHuu5LZ5sLNsjVTWO5UqStkKzbfeRJGlAhoIkqWcoDCTJHkmuSnJnkjuSvHvomoaWZNsk30hy6dC1DC3JTkkuSPLNJHclOWDomoaU5D3t/+T2JOcmec7QNc2kJGcmWZPk9pG2XZJcmeSe9rzzlliXoTCcx4CTqmpvYH/ghCR7D1zT0N4N3DV0EbPEfwcur6q9gJczh7dLkoXAu4AlVfUyupNQjh62qhl3FvCGCW0nA8uqajGwrL1+2gyFgVTV6qq6qQ0/QvdPv3DYqoaTZHfgjcBnhq5laEmeD7wG+CxAVf2kqv5l2KoGNw94bpJ5wPOA/ztwPTOqqq4Gvjuh+TDg7DZ8NnD4lliXoTALJFkE7AtcN2wlg/pz4A+BJ4YuZBZ4MbAW+Ku2O+0zSbYfuqihVNUq4CPAt4HVwMNV9eVhq5oVFlTV6jb8ILBgSyzUUBhYkh2AC4ETq+r7Q9czhCS/AaypqhuHrmWWmAe8Aji9qvYFfsgW2jWwNWr7yg+jC8sXANsn+Z1hq5pdqvtuwRb5foGhMKAk29EFwjlVddHQ9QzoVcCbktxHd2XcX03y18OWNKiVwMqqWtdzvIAuJOaqXwO+VVVrq+qnwEXAgQPXNBs8lGQ3gPa8Zkss1FAYSJLQ7TO+q6o+NnQ9Q6qqU6pq96paRHcA8atVNWc/CVbVg8ADSX6hNR0CzOV7inwb2D/J89r/zSHM4QPvIy4Bjm3DxwKf3xILNRSG8yrgLXSfim9uj0OHLkqzxjuBc5LcCuwD/OnA9Qym9ZguAG4CbqN735pTl7xIci5wDfALSVYmeRtwGvDaJPfQ9aZO2yLr8jIXkqR17ClIknqGgiSpZyhIknqGgiSpZyhIknqGgkjyeDsl9vYkX0iy09NY1g+exrzvalcEPWdC+z6jp+smOTXJf9jc9YxDkvdvgWXcl2TXLVHPNNZ1uBdg1GQMBQH8v6rap12B8rvACQPV8XbgtVX12xPa9wFm+3c4phUKSbYddyHTdDhgKOgpDAVNdA3taq1J9ktyTbso2z+t+4ZtkuOSXJTk8nYt9z+buJAku7Z53zjJuPe2XsntSU5sbZ8CXgJ8Kcl7RqZ9FvBfgKNab+aoNmrvJF9Lcm+Sd41M/ztJrm/T/uVkb8LtE/mH2jTLk7wiyRVJ/k+S49s0SfLhVuNt69abZLckV4/0rF6d5DS6K3jePLGX0+b5QZKPJrkFOGCaNT5lmiTHJ/nwyDTHJflkG/67JDemu+fA0gnr/mCSW5Jcm2RBkgOBNwEfbsvfc8K6z0ryqbZt/ne7NhVJFiX5+yQ3tceBU2yTbdty1m2/97Rpfz/JDa2eC5M8r7Xv2eq7LckHRnucSd7X5rk1yX+euK20hVWVjzn+AH7QnrcF/hZ4Q3v9M8C8NvxrwIVt+DjgXuD5wHOA+4E91i2L7mqN19F96p+4rl+m+1bq9sAOwB3Avm3cfcCuk8xzHPDJkdenAv8EPBvYFfgOsB3wi8AXgO3adP8TeOsky7sP+Pdt+OPArcCOwHzgodb+b4Er2zZZQHephd2Ak4A/GtleO45uww1s3wKObMMbrHHdz7+haVp9K0aW+yXgoDa8S3t+LnA78LMj6/7NNvxnwB+34bOAN2+g3rOAy+k+NC6muxbTc+guWf2cNs1iYHkbfso2ab/nK0eWuVN7/tmRtg8A72zDlwLHtOHjWf83+Tq6by+n1XMp8Jqh/2eeyY95SO1TLl0P4S66N0Po3vTPTrKY7s1lu5F5llXVwwBJ7gReBDzQplkGnFBVX59kXQcBF1fVD9u8FwGvBr6xiTVfVlWPAo8mWUP3xn0I3ZvRDUmge4Pc0EXCLmnPtwE7VHdPi0eSPJrumMpBwLlV9Tjdhce+DrwSuAE4M93FDP+uqm6eRq2P0134kGnWOOk0VbW29Yz2B+4B9gL+sc3zriRHtOE96N60vwP8hO6NFOBG4LXTqBfg/Kp6Argnyb1tXd8CPplkn/YzvbRN+5Rt0uZ5SZK/AC4D1l3q+mVJPgDsRPeh4IrWfgDr7wfwN3SXyoYuFF7H+r+PHdrPdvU0fw5tIkNB0I4ptK78FXTHFD4B/Ffgqqo6It09H742Ms+jI8OPs/5v6TG6N5/XA5OFwpYy2foDnF1Vp2zC/E9MWNYTTPF/UVVXJ3kN3Q2Bzkrysar63EbW9eMWLkyzxqmmOQ84EvgmXbhWkoPpenIHVNWPknyN7pM9wE+rfeTmyb+njZl4/ZsC3gM8RHcnuG2AH8OGt0mSl9P9HRzfav49ul7I4VV1S5LjgIM3UkeAD1XVX06zbj1NHlNQr6p+RHfbw5PS3eHq+cCqNvq46S6G7p9/ryT/cZLxfw8cnu6Kl9sDR7S2qTxCt0tiY5YBb07yc9Dfw/ZF06x7sjqPavvG59PdCe36tryHqurTdHeJW3dJ65+2T8pbosapprmY7t4Cx9AFBHS/p++1QNiL7vauG7OxbfpbSbZpxxteAtzd1rO69SDeQreriMm2SbqzqLapqguBP2b9dtoRWN221egJBdfS7bKDJ99q8wrg99Ldd4QkC9dtF42HoaAnqapv0O1jP4ZuH/SHknyDTehVtk/Fx9BdAfbtE8bdRPdp8Xq64w6faeucylV0B5ZHDzRPtt476d6Avpzu6qJX0h0H2BwX022HW4CvAn9Y3SWtDwZuadvkKLp7KUO33/vWyQ40b2qNU01TVd+j28X3oqq6vs1yOTAvyV10V8q8dho/33nA+9KdRLDnJOO/Tfc7+hJwfFX9mO7YxrHpDpjvRXfzH5h8mywEvtZ2S/41sK7X85/ofu//SNfbWedE4L3t5/154OH2836ZbnfSNUluo7ta6nQ+IGgzeZVUSU+S5Czg0qq6YAbX+Ty63ZiV5Gi6g86HzdT6tZ7HFCTNBr9MdxA7wL/Q7YLUAOwpSJJ6HlOQJPUMBUlSz1CQJPUMBUlSz1CQJPX+P5NyvzBlZNthAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "evaluate_results(pred_para, gold_para)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DZ2UtuJntVkr"
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uEC4jGZ3tWO9"
      },
      "outputs": [],
      "source": [
        "k = 10\n",
        "numThemes = len(theme_wise_data.keys())\n",
        "total = 0\n",
        "notInTopKSum = 0\n",
        "rankSum = 0.\n",
        "i = 0\n",
        "\n",
        "for theme in theme_wise_data:\n",
        "    paras, ques, gold_para, _ = load_ques_by_theme(theme, theme_wise_data, answerable_only=True)\n",
        "    sents, para_id = load_sents_from_para(paras)\n",
        "\n",
        "    sents_embed = model(sents)\n",
        "    ques_embed = model(ques)\n",
        "\n",
        "    D, I = get_k_nearest_neighbours(sents_embed, ques_embed, k)\n",
        "    pred_para = sent_id_to_para_id(I, para_id)\n",
        "\n",
        "    notInTopK, avg = getMeanRank(pred_para, gold_para)\n",
        "    total += len(ques)\n",
        "    notInTopKSum += notInTopK\n",
        "    rankSum += avg * (len(ques) - notInTopK)\n",
        "\n",
        "    i += 1\n",
        "    print(f'\\r{i}/{numThemes} themes evaluated', end='')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PvJnhvmTtVT5",
        "outputId": "5b17af5d-3112-4f23-852a-2e5ca6157f2b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Total queries: 50125\n",
            "In top 10 results, number of queries for which -\n",
            "\tRelevant paragraph found: 46954 (93.67 %)\n",
            "\tRelevant paragraph NOT found: 3171 (6.33 %)\n",
            "Mean Rank for which relevant paragraph found: 1.6\n"
          ]
        }
      ],
      "source": [
        "inTopKSum = total - notInTopKSum\n",
        "print(f\"Total queries: {total}\")\n",
        "print(f\"In top {k} results, number of queries for which -\")\n",
        "print(f\"\\tRelevant paragraph found: {inTopKSum} ({round(100.*inTopKSum / total, 2)} %)\")\n",
        "print(f\"\\tRelevant paragraph NOT found: {notInTopKSum} ({round(100.*notInTopKSum / total, 2)} %)\")\n",
        "print(f\"Mean Rank for which relevant paragraph found: {round(rankSum / total, 2)}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "srY4lMqJusBS"
      },
      "outputs": [],
      "source": [
        "# 38 mins 05 secs for 50126 queries consisting of 361 themes"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JNT7uxNNjCEs"
      },
      "source": [
        "#### 2. Using SimCSE"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "71WGlF-sOf2Z",
        "outputId": "b790b0b5-3aeb-4cf1-b0af-baeeefc0ad44"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Number of examples = 75056\n",
            "\tAnswerable questions = 50126\n",
            "\tNon-Answerable questions = 24930\n",
            "\n",
            "Examples:\n",
            " | Theme  |  Paragraph...  |  Question | Answer_possible | Answer_text | Answer_start\n",
            "1430 | Frédéric_Chopin  |  Some modern commenta...  |  Who said Chopin's works were modeled after Bach, Beethoven, Schubert and Field? | True | ['Richard Taruskin'] | [543]\n",
            "2196 | The_Legend_of_Zelda:_Twilight_Princess  |  Twilight Princess ta...  |  Who releases Bulbins from the Realm of Twilight? | False | [] | []\n",
            "\n",
            "Total 361 themes present.\n"
          ]
        }
      ],
      "source": [
        "train_data = load_data()\n",
        "theme_wise_data = load_theme_wise_data(train_data)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5QJgsWOZXe0X"
      },
      "outputs": [],
      "source": [
        "paras, ques, gold_para, _ = load_ques_by_theme('Beyoncé', theme_wise_data, answerable_only=True)\n",
        "sents, para_id = load_sents_from_para(paras)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qDieg4qBDSlb"
      },
      "outputs": [],
      "source": [
        "encoder = 'SimCSE'\n",
        "tokenizer, model = load_encoder(encoder)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "apPV-CERDSlc"
      },
      "outputs": [],
      "source": [
        "sents_embed = get_embeddings(sents, encoder, tokenizer, model)\n",
        "ques_embed = get_embeddings(ques, encoder, tokenizer, model)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZB18ZSdJXe0Z"
      },
      "outputs": [],
      "source": [
        "k = 10\n",
        "D, I = get_k_nearest_neighbours(sents_embed, ques_embed, k)\n",
        "pred_para = sent_id_to_para_id(I, para_id)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WFI6AKhxXe0a",
        "outputId": "11b88929-76e9-420c-bf0d-02f450c86278"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "question: When did Beyonce leave Destiny's Child and become a solo singer?\n",
            "similar sentences:\n",
            "\tTarget sentence | Predicted Para ID | Distance between target sentence and query\n",
            "\t After the release of Dangerously in Love, Beyoncé had planned to produce a follow-up album using several of the left-over tracks. | 11 | 23.658646\n",
            "\t When The Guardian named her Artist of the Decade, Llewyn-Smith wrote, \"Why Beyoncé? | 50 | 24.23243\n",
            "\t In February 2013, Beyoncé said that Madonna inspired her to take control of her own career. | 40 | 25.370155\n",
            "\t In early 2001, while Destiny's Child was completing their third album, Beyoncé landed a major role in the MTV made-for-television film, Carmen: A Hip Hopera, starring alongside American actor Mekhi Phifer. | 8 | 25.500847\n",
            "\t Their hiatus saw the release of Beyoncé's debut album, Dangerously in Love (2003), which established her as a solo artist worldwide, earned five Grammy Awards and featured the Billboard Hot 100 number-one singles \"Crazy in Love\" and \"Baby Boy\". | 0 | 25.838299\n",
            "\t Beyoncé stated that she struggled to speak about her depression because Destiny's Child had just won their first Grammy Award and she feared no one would take her seriously. | 7 | 25.909882\n",
            "\t Beyoncé announced a hiatus from her music career in January 2010, heeding her mother's advice, \"to live life, to be inspired by things again\". | 17 | 26.701424\n",
            "\t Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&B girl-group Destiny's Child. | 0 | 27.247742\n",
            "\t Sasha was conceived during the making of \"Crazy in Love\", and Beyoncé introduced her with the release of her 2008 album I Am... Sasha Fierce. | 43 | 27.618217\n",
            "\t A self-described \"modern-day feminist\", Beyoncé creates songs that are often characterized by themes of love, relationships, and monogamy, as well as female sexuality and empowerment. | 2 | 27.848145\n"
          ]
        }
      ],
      "source": [
        "print_example(0, sents, ques, pred_para, I, D)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 388
        },
        "id": "t6HFmLFfXe0a",
        "outputId": "d4a2f509-f379-49eb-c9a7-0cf596a4bfb8"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Total queries: 523\n",
            "In top 10 results, number of queries for which -\n",
            "\tRelevant paragraph found: 447 (85 %)\n",
            "\tRelevant paragraph NOT found: 76 (15 %)\n",
            "Mean Rank for which relevant paragraph found: 2.24\n",
            "\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAVT0lEQVR4nO3de7QlZX3m8e8jjTcgAqHTCxu01bQhxFmCaRlA4yIhGpUk4IxBWIlCwthhRBFlzKDJrDgzOjLxljFOMKgEXEEYwiWiKIgtSi7cGuSODow20kxLd9Qg6oiCv/mj3lO9OX26ezecfeo05/tZa69d+63b79Q5Zz/7rapdlapCkiSAJwxdgCRp/jAUJEk9Q0GS1DMUJEk9Q0GS1Fs0dAGPxR577FHLli0bugxJ2q5cf/31/1xVi2cat12HwrJly1i9evXQZUjSdiXJ3Zsb5+4jSVLPUJAk9QwFSVLPUJAk9QwFSVLPUJAk9QwFSVLPUJAk9QwFSVJvu/5G82Ox7JRLBlv3mlMPG2zdkrQl9hQkST1DQZLUMxQkST1DQZLUMxQkST1DQZLUMxQkST1DQZLUMxQkST1DQZLUMxQkST1DQZLUMxQkSb2JhUKSvZNckeT2JLcleXNrf2eSe5Pc2B6vHJnn7UnuSvK1JL8xqdokSTOb5KWzHwJOrqobkuwCXJ/k8jbug1X1vtGJk+wLHAX8EvB04AtJnltVD0+wRknSiIn1FKpqXVXd0IYfAO4Alm5hlsOBc6vqwar6BnAXcMCk6pMkbWpOjikkWQbsD1zTmt6Y5OYkZyTZrbUtBe4ZmW0tM4RIkpVJVidZvWHDhglWLUkLz8RDIcnOwAXASVX1PeA04DnAfsA64P3bsryqOr2qVlTVisWLF896vZK0kE00FJLsSBcIZ1fVhQBVdV9VPVxVPwU+ysZdRPcCe4/MvldrkyTNkUmefRTg48AdVfWBkfY9RyZ7FXBrG74YOCrJk5I8C1gOXDup+iRJm5rk2UcvAl4L3JLkxtb2DuDoJPsBBawB/hCgqm5Lch5wO92ZSyd45pEkza2JhUJV/QOQGUZ9dgvzvBt496RqkiRtmd9oliT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT1JhYKSfZOckWS25PcluTNrX33JJcnubM979bak+RDSe5KcnOSF0yqNknSzCbZU3gIOLmq9gUOBE5Isi9wCrCqqpYDq9prgFcAy9tjJXDaBGuTJM1gYqFQVeuq6oY2/ABwB7AUOBw4q012FnBEGz4c+ER1rgZ2TbLnpOqTJG1qTo4pJFkG7A9cAyypqnVt1LeAJW14KXDPyGxrW9v0Za1MsjrJ6g0bNkysZklaiCYeCkl2Bi4ATqqq742Oq6oCaluWV1WnV9WKqlqxePHiWaxUkjTRUEiyI10gnF1VF7bm+6Z2C7Xn9a39XmDvkdn3am2SpDkyybOPAnwcuKOqPjAy6mLgmDZ8DPCpkfbXtbOQDgTuH9nNJEmaA4smuOwXAa8FbklyY2t7B3AqcF6S44C7gSPbuM8CrwTuAn4I/P4Ea5MkzWBioVBV/wBkM6MPnWH6Ak6YVD2SpK3zG82SpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpN5YoZDkReO0SZK2b+P2FP5izDZJ0nZs0ZZGJjkIOBhYnOStI6N+BthhkoVJkubeFkMBeCKwc5tul5H27wGvnlRRkqRhbDEUqurLwJeTnFlVd2/LgpOcAfwmsL6qntfa3gm8HtjQJntHVX22jXs7cBzwMHBiVV22LeuTJD12W+spTHlSktOBZaPzVNWvbWGeM4EPA5+Y1v7BqnrfaEOSfYGjgF8Cng58Iclzq+rhMeuTJM2CcUPhb4GPAB+j+yS/VVV1ZZJlYy7/cODcqnoQ+EaSu4ADgKvGnF+SNAvGDYWHquq0WVrnG5O8DlgNnFxV3wWWAlePTLO2tW0iyUpgJcAznvGMWSpJkgTjn5L66SRvSLJnkt2nHo9ifacBzwH2A9YB79/WBVTV6VW1oqpWLF68+FGUIEnanHF7Cse057eNtBXw7G1ZWVXdNzWc5KPAZ9rLe4G9Rybdq7VJkubQWKFQVc+ajZUl2bOq1rWXrwJubcMXA59M8gG6A83LgWtnY52SpPGNFQrtGMAmqmr6mUWj85wDHALskWQt8KfAIUn2o+tlrAH+sC3ntiTnAbcDDwEneOaRJM29cXcfvXBk+MnAocANbHq6aa+qjp6h+eNbmP7dwLvHrEeSNAHj7j560+jrJLsC506kIknSYB7tpbN/AMzKcQZJ0vwx7jGFT9MdB4DuQni/CJw3qaIkScMY95jC6GUpHgLurqq1E6hHkjSgsXYftQvjfZXuSqm7AT+eZFGSpGGMe+e1I+m+N/A7wJHANUm8dLYkPc6Mu/voj4EXVtV6gCSLgS8A50+qMEnS3Bv37KMnTAVC8+1tmFeStJ0Yt6dwaZLLgHPa69cAn51MSZKkoWztHs0/Dyypqrcl+TfAi9uoq4CzJ12cJGluba2n8OfA2wGq6kLgQoAk/6qN+62JVidJmlNbOy6wpKpumd7Y2pZNpCJJ0mC2Fgq7bmHcU2azEEnS8LYWCquTvH56Y5J/B1w/mZIkSUPZ2jGFk4CLkvwuG0NgBfBEupvkSJIeR7YYCu32mQcn+VXgea35kqr64sQrkyTNuXHvp3AFcMWEa5EkDcxvJUuSeoaCJKlnKEiSeoaCJKlnKEiSeoaCJKlnKEiSeoaCJKlnKEiSeoaCJKlnKEiSeoaCJKk3sVBIckaS9UluHWnbPcnlSe5sz7u19iT5UJK7ktyc5AWTqkuStHmT7CmcCbx8WtspwKqqWg6saq8BXgEsb4+VwGkTrEuStBkTC4WquhL4zrTmw4Gz2vBZwBEj7Z+oztXArkn2nFRtkqSZzfUxhSVVta4NfwtY0oaXAveMTLe2tW0iycokq5Os3rBhw+QqlaQFaLADzVVVQD2K+U6vqhVVtWLx4sUTqEySFq65DoX7pnYLtef1rf1eYO+R6fZqbZKkOTTXoXAxcEwbPgb41Ej769pZSAcC94/sZpIkzZGx7tH8aCQ5BzgE2CPJWuBPgVOB85IcB9wNHNkm/yzwSuAu4IfA70+qLknS5k0sFKrq6M2MOnSGaQs4YVK1SJLG4zeaJUk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1Fs0dAEL0bJTLhlkvWtOPWyQ9UrafthTkCT1DAVJUs9QkCT1DAVJUm+QA81J1gAPAA8DD1XViiS7A/8LWAasAY6squ8OUZ8kLVRD9hR+tar2q6oV7fUpwKqqWg6saq8lSXNoPu0+Ohw4qw2fBRwxYC2StCANFQoFfD7J9UlWtrYlVbWuDX8LWDLTjElWJlmdZPWGDRvmolZJWjCG+vLai6vq3iQ/B1ye5KujI6uqktRMM1bV6cDpACtWrJhxGknSozNIT6Gq7m3P64GLgAOA+5LsCdCe1w9RmyQtZHMeCkl2SrLL1DDwMuBW4GLgmDbZMcCn5ro2SVrohth9tAS4KMnU+j9ZVZcmuQ44L8lxwN3AkQPUJkkL2pyHQlV9HXj+DO3fBg6d63okSRvNp1NSJUkDMxQkST1DQZLUMxQkST1DQZLUMxQkST1DQZLUMxQkST1DQZLUMxQkST1DQZLUG+p+ChrAslMuGWzda049bLB1SxqfPQVJUs+egubEUL0UeyjStrGnIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6npOpxbcgv7A3F03D1WNhTkCT1DAVJUs9QkCT1PKYgaVZ4wcXHB3sKkqSePQXpcWYhnnHlBRdnjz0FSVLPnoIkPUqPx+Mo866nkOTlSb6W5K4kpwxdjyQtJPMqFJLsAPxP4BXAvsDRSfYdtipJWjjmVSgABwB3VdXXq+rHwLnA4QPXJEkLxnw7prAUuGfk9VrgX49OkGQlsLK9/H6Sr81RbZOyB/DPQxcxj7g9HsntsZHbYkT++2PaHs/c3Ij5FgpbVVWnA6cPXcdsSbK6qlYMXcd84fZ4JLfHRm6LR5rU9phvu4/uBfYeeb1Xa5MkzYH5FgrXAcuTPCvJE4GjgIsHrkmSFox5tfuoqh5K8kbgMmAH4Iyqum3gsibtcbMrbJa4PR7J7bGR2+KRJrI9UlWTWK4kaTs033YfSZIGZChIknqGwkCS7J3kiiS3J7ktyZuHrmloSXZI8pUknxm6lqEl2TXJ+Um+muSOJAcNXdOQkryl/Z/cmuScJE8euqa5lOSMJOuT3DrStnuSy5Pc2Z53m411GQrDeQg4uar2BQ4ETvCSHrwZuGPoIuaJ/wFcWlX7AM9nAW+XJEuBE4EVVfU8upNQjhq2qjl3JvDyaW2nAKuqajmwqr1+zAyFgVTVuqq6oQ0/QPdPv3TYqoaTZC/gMOBjQ9cytCRPA14CfBygqn5cVf8ybFWDWwQ8Jcki4KnA/x24njlVVVcC35nWfDhwVhs+CzhiNtZlKMwDSZYB+wPXDFvJoP4c+CPgp0MXMg88C9gA/HXbnfaxJDsNXdRQqupe4H3AN4F1wP1V9flhq5oXllTVujb8LWDJbCzUUBhYkp2BC4CTqup7Q9czhCS/CayvquuHrmWeWAS8ADitqvYHfsAs7RrYHrV95YfTheXTgZ2S/N6wVc0v1X23YFa+X2AoDCjJjnSBcHZVXTh0PQN6EfDbSdbQXRn315L8zbAlDWotsLaqpnqO59OFxEL168A3qmpDVf0EuBA4eOCa5oP7kuwJ0J7Xz8ZCDYWBJAndPuM7quoDQ9czpKp6e1XtVVXL6A4gfrGqFuwnwar6FnBPkl9oTYcCtw9Y0tC+CRyY5Knt/+ZQFvCB9xEXA8e04WOAT83GQg2F4bwIeC3dp+Ib2+OVQxeleeNNwNlJbgb2A/7bwPUMpvWYzgduAG6he99aUJe8SHIOcBXwC0nWJjkOOBV4aZI76XpTp87KurzMhSRpij0FSVLPUJAk9QwFSVLPUJAk9QwFSVLPUBBJHm6nxN6a5NNJdn0My/r+Y5j3xHZF0LOnte83erpukncm+Q+Pdj2TkOQds7CMNUn2mI16xljXEV6AUTMxFATw/6pqv3YFyu8AJwxUxxuAl1bV705r3w+Y79/hGCsUkuww6ULGdARgKGgThoKmu4p2tdYkByS5ql2U7Z+mvmGb5NgkFya5tF3L/c+mLyTJHm3ew2YY99bWK7k1yUmt7SPAs4HPJXnLyLRPBP4L8JrWm3lNG7Vvki8l+XqSE0em/70k17Zp/2qmN+H2ifw9bZrVSV6Q5LIk/yfJ8W2aJHlvq/GWqfUm2TPJlSM9q19JcirdFTxvnN7LafN8P8n7k9wEHDRmjZtMk+T4JO8dmebYJB9uw3+X5Pp09xxYOW3d705yU5KrkyxJcjDw28B72/KfM23dZyb5SNs2/7tdm4oky5L8fZIb2uPgLWyTHdpyprbfW9q0r09yXavngiRPbe3PafXdkuRdoz3OJG9r89yc5D9P31aaZVXlY4E/gO+35x2AvwVe3l7/DLCoDf86cEEbPhb4OvA04MnA3cDeU8uiu1rjNXSf+qev65fpvpW6E7AzcBuwfxu3BthjhnmOBT488vqdwD8BTwL2AL4N7Aj8IvBpYMc23V8Cr5theWuAf9+GPwjcDOwCLAbua+3/Fri8bZMldJda2BM4Gfjjke21y+g23Mz2LeDINrzZGqd+/s1N0+q7a2S5nwNe3IZ3b89PAW4FfnZk3b/Vhv8M+JM2fCbw6s3UeyZwKd2HxuV012J6Mt0lq5/cplkOrG7Dm2yT9nu+fGSZu7bnnx1pexfwpjb8GeDoNnw8G/8mX0b37eW0ej4DvGTo/5nH82MRUvuUS9dDuIPuzRC6N/2zkiyne3PZcWSeVVV1P0CS24FnAve0aVYBJ1TVl2dY14uBi6rqB23eC4FfAb6yjTVfUlUPAg8mWU/3xn0o3ZvRdUmge4Pc3EXCLm7PtwA7V3dPiweSPJjumMqLgXOq6mG6C499GXghcB1wRrqLGf5dVd04Rq0P0134kDFrnHGaqtrQekYHAncC+wD/2OY5Mcmr2vDedG/a3wZ+TPdGCnA98NIx6gU4r6p+CtyZ5OttXd8APpxkv/YzPbdNu8k2afM8O8lfAJcAU5e6fl6SdwG70n0ouKy1H8TG+wF8ku5S2dCFwsvY+Pexc/vZrhzz59A2MhQE7ZhC68pfRndM4UPAfwWuqKpXpbvnw5dG5nlwZPhhNv4tPUT35vMbwEyhMFtmWn+As6rq7dsw/0+nLeunbOH/oqquTPISuhsCnZnkA1X1ia2s60ctXBizxi1Ncy5wJPBVunCtJIfQ9eQOqqofJvkS3Sd7gJ9U+8jNI39PWzP9+jcFvAW4j+5OcE8AfgSb3yZJnk/3d3B8q/kP6HohR1TVTUmOBQ7ZSh0B3lNVfzVm3XqMPKagXlX9kO62hyenu8PV04B72+hjx10M3T//Pkn+4wzj/x44It0VL3cCXtXatuQBul0SW7MKeHWSn4P+HrbPHLPumep8Tds3vpjuTmjXtuXdV1UfpbtL3NQlrX/SPinPRo1bmuYiunsLHE0XEND9nr7bAmEfutu7bs3WtunvJHlCO97wbOBrbT3rWg/itXS7iphpm6Q7i+oJVXUB8Cds3E67AOvatho9oeBqul128MhbbV4G/EG6+46QZOnUdtFkGAp6hKr6Ct0+9qPp9kG/J8lX2IZeZftUfDTdFWDfMG3cDXSfFq+lO+7wsbbOLbmC7sDy6IHmmdZ7O90b0OfTXV30crrjAI/GRXTb4Sbgi8AfVXdJ60OAm9o2eQ3dvZSh2+9980wHmre1xi1NU1XfpdvF98yqurbNcimwKMkddFfKvHqMn+9c4G3pTiJ4zgzjv0n3O/occHxV/Yju2MYx6Q6Y70N38x+YeZssBb7Udkv+DTDV6/lPdL/3f6Tr7Uw5CXhr+3l/Hri//byfp9uddFWSW+iuljrOBwQ9Sl4lVdIjJDkT+ExVnT+H63wq3W7MSnIU3UHnw+dq/drIYwqS5oNfpjuIHeBf6HZBagD2FCRJPY8pSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6/x/NU8dgBIu+OgAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "evaluate_results(pred_para, gold_para)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HY1Ud3xq6Rjz"
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4hZxasmu6R33",
        "outputId": "e841db5c-fd50-495f-9784-efc6e0cdd703"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "10/361 themes evaluated"
          ]
        }
      ],
      "source": [
        "k = 10\n",
        "numThemes = len(theme_wise_data.keys())\n",
        "total = 0\n",
        "notInTopKSum = 0\n",
        "rankSum = 0.\n",
        "i = 0\n",
        "\n",
        "for theme in theme_wise_data:\n",
        "    paras, ques, gold_para, _ = load_answerable_ques_by_theme(theme, theme_wise_data)\n",
        "    sents, para_id = load_sents_from_para(paras)\n",
        "\n",
        "    sents_tok = tokenizer(sents, padding=True, truncation=True, return_tensors=\"pt\")\n",
        "    ques_tok = tokenizer(ques, padding=True, truncation=True, return_tensors=\"pt\")\n",
        "\n",
        "    with torch.no_grad():\n",
        "        sents_embed = model(**sents_tok, output_hidden_states=True, return_dict=True).pooler_output\n",
        "        ques_embed = model(**ques_tok, output_hidden_states=True, return_dict=True).pooler_output\n",
        "\n",
        "    D, I = get_k_nearest_neighbours(sents_embed, ques_embed, k)\n",
        "    pred_para = sent_id_to_para_id(I, para_id)\n",
        "\n",
        "    notInTopK, avg = getMeanRank(pred_para, gold_para)\n",
        "    total += len(ques)\n",
        "    notInTopKSum += notInTopK\n",
        "    rankSum += avg * (len(ques) - notInTopK)\n",
        "\n",
        "    i += 1\n",
        "    print(f'\\r{i}/{numThemes} themes evaluated', end='')\n",
        "    if i == 10:\n",
        "        break"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BDbFt9ZT6R34",
        "outputId": "bfdf7ff4-e626-4564-8ec2-b2dfb68f222b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Total queries: 3138\n",
            "In top 10 results, number of queries for which -\n",
            "\tRelevant paragraph found: 2693 (85.82 %)\n",
            "\tRelevant paragraph NOT found: 445 (14.18 %)\n",
            "Mean Rank for which relevant paragraph found: 1.8\n"
          ]
        }
      ],
      "source": [
        "inTopKSum = total - notInTopKSum\n",
        "print(f\"Total queries: {total}\")\n",
        "print(f\"In top {k} results, number of queries for which -\")\n",
        "print(f\"\\tRelevant paragraph found: {inTopKSum} ({round(100.*inTopKSum / total, 2)} %)\")\n",
        "print(f\"\\tRelevant paragraph NOT found: {notInTopKSum} ({round(100.*notInTopKSum / total, 2)} %)\")\n",
        "print(f\"Mean Rank for which relevant paragraph found: {round(rankSum / total, 2)}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dae-JG5J6RQJ"
      },
      "outputs": [],
      "source": [
        "# 23 mins 44 secs for 3138 queries consisting of 10 themes"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qgsNoZR2lvCl"
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4-MjbAQYlvsG"
      },
      "source": [
        "#### 3. Using Universal Sentence Encoder for QA"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UyHvJ-dElvsG",
        "outputId": "628dfc76-667c-42aa-ba88-f7de92def482"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Number of examples = 75056\n",
            "\tAnswerable questions = 50126\n",
            "\tNon-Answerable questions = 24930\n",
            "\n",
            "Examples:\n",
            " | Theme  |  Paragraph...  |  Question | Answer_possible | Answer_text | Answer_start\n",
            "1430 | Frédéric_Chopin  |  Some modern commenta...  |  Who said Chopin's works were modeled after Bach, Beethoven, Schubert and Field? | True | ['Richard Taruskin'] | [543]\n",
            "2196 | The_Legend_of_Zelda:_Twilight_Princess  |  Twilight Princess ta...  |  Who releases Bulbins from the Realm of Twilight? | False | [] | []\n",
            "\n",
            "Total 361 themes present.\n"
          ]
        }
      ],
      "source": [
        "train_data = load_data()\n",
        "theme_wise_data = load_theme_wise_data(train_data)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hTIX7kOglvsH",
        "outputId": "b249643a-4868-46eb-bdd0-37bfa328fbf7"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Total Questions: 523\n",
            "Total Paragraphs: 66\n"
          ]
        }
      ],
      "source": [
        "paras, ques, gold_para, _ = load_ques_by_theme('Beyoncé', theme_wise_data, answerable_only=True)\n",
        "sents, para_id = load_sents_from_para(paras)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8Kl_ivgllvsH"
      },
      "outputs": [],
      "source": [
        "encoder = 'Universal Sentence Encoder for QA'\n",
        "_, model = load_encoder(encoder)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rFk4xnOElvsH"
      },
      "outputs": [],
      "source": [
        "sents_embed = get_embeddings(sents, encoder, _, model, sents_type=\"Context\")\n",
        "ques_embed = get_embeddings(ques, encoder, _, model, sents_type=\"Question\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UI7IiQDElvsH"
      },
      "outputs": [],
      "source": [
        "k = 10\n",
        "D, I = get_k_nearest_neighbours(sents_embed, ques_embed, k)\n",
        "pred_para = sent_id_to_para_id(I, para_id)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CnBMxjL9lvsH",
        "outputId": "382e0164-9509-47e7-9e51-3b1741134689"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "question: When did Beyonce leave Destiny's Child and become a solo singer?\n",
            "similar sentences:\n",
            "\tTarget sentence | Predicted Para ID | Distance between target sentence and query\n",
            "\t Following the disbandment of Destiny's Child in June 2005, she released her second solo album, B'Day (2006), which contained hits \"Déjà Vu\", \"Irreplaceable\", and \"Beautiful Liar\". | 1 | 0.753831\n",
            "\t She has received co-writing credits for most of the songs recorded with Destiny's Child and her solo efforts. | 36 | 0.86313295\n",
            "\t Throughout a career spanning 19 years, she has sold over 118 million records as a solo artist, and a further 60 million with Destiny's Child, making her one of the best-selling music artists of all time. | 2 | 0.86395407\n",
            "\t However, this was put on hold so she could concentrate on recording Destiny Fulfilled, the final studio album by Destiny's Child. | 11 | 0.87941134\n",
            "\t Destiny's Child embarked on a worldwide concert tour, Destiny Fulfilled... and Lovin' It and during the last stop of their European tour, in Barcelona on June 11, 2005, Rowland announced that Destiny's Child would disband following the North American leg of the tour. | 11 | 0.8841224\n",
            "\t In early 2001, while Destiny's Child was completing their third album, Beyoncé landed a major role in the MTV made-for-television film, Carmen: A Hip Hopera, starring alongside American actor Mekhi Phifer. | 8 | 0.89829373\n",
            "\t Her first solo album Dangerously in Love was released on June 24, 2003, after Michelle Williams and Kelly Rowland had released their solo efforts. | 10 | 0.91466975\n",
            "\t Beyoncé took a hiatus from music in 2010 and took over management of her career; her fourth album 4 (2011) was subsequently mellower in tone, exploring 1970s funk, 1980s pop, and 1990s soul. | 1 | 0.915099\n",
            "\t As a solo artist she has sold over 15 million albums in the US, and over 118 million records worldwide (a further 60 million additionally with Destiny's Child), making her one of the best-selling music artists of all time. | 53 | 0.9161856\n",
            "\t Her vocal abilities mean she is identified as the centerpiece of Destiny's Child. | 34 | 0.9503057\n"
          ]
        }
      ],
      "source": [
        "print_example(0, sents, ques, pred_para, I, D)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 388
        },
        "id": "_sj0V4L3lvsI",
        "outputId": "2b9129f3-b3a2-4e8b-d1bd-253b15eb920e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Total queries: 523\n",
            "In top 10 results, number of queries for which -\n",
            "\tRelevant paragraph found: 449 (86 %)\n",
            "\tRelevant paragraph NOT found: 74 (14 %)\n",
            "Mean Rank for which relevant paragraph found: 2.11\n",
            "\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAVP0lEQVR4nO3de7RedX3n8fcHgjegAiXNwoBGaSylzhJsZADRRUu9FFuBGctltQqt05QRLyhjB2xnlZnRytRbxzqDRaXgKoWhXCqCghhReuEWkDsyZBAkmUBStYg6osB3/ti/7DwcTk5OQp6zTzjv11rPevbz27fv2eec5/P89t7P3qkqJEkC2GboAiRJs4ehIEnqGQqSpJ6hIEnqGQqSpN68oQt4OnbddddatGjR0GVI0lblxhtv/Oeqmj/ZuK06FBYtWsTy5cuHLkOStipJ7t/QOHcfSZJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6W/U3mp+ORSdfNti67zvtjYOtW5KmYk9BktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJvbGFQpI9klyV5M4kdyR5d2s/NcmqJDe3x6Ej85ySZEWSu5O8fly1SZImN2+My34MOKmqbkqyI3BjkivbuI9X1UdGJ06yN3A08EvAC4CvJHlpVT0+xholSSPG1lOoqtVVdVMbfgS4C1g4xSyHAedV1aNV9S1gBbDfuOqTJD3VjBxTSLII2Be4rjW9I8mtSc5MsnNrWwg8MDLbSqYOEUnSFjb2UEiyA3AhcGJVfR84HdgT2AdYDXx0E5e3NMnyJMvXrl27xeuVpLlsrKGQZDu6QDinqi4CqKqHqurxqnoC+DTrdxGtAvYYmX331vYkVXVGVS2pqiXz588fZ/mSNOeM8+yjAJ8F7qqqj4207zYy2RHA7W34EuDoJM9O8mJgMXD9uOqTJD3VOM8+ehXwFuC2JDe3tvcDxyTZByjgPuAPAKrqjiTnA3fSnbl0gmceSdLMGlsoVNU/AJlk1BenmOeDwAfHVZMkaWp+o1mS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEm9sYVCkj2SXJXkziR3JHl3a98lyZVJ7mnPO7f2JPlEkhVJbk3yinHVJkma3Dh7Co8BJ1XV3sD+wAlJ9gZOBpZV1WJgWXsN8OvA4vZYCpw+xtokSZMYWyhU1eqquqkNPwLcBSwEDgPObpOdDRzehg8DPleda4Gdkuw2rvokSU81I8cUkiwC9gWuAxZU1eo26kFgQRteCDwwMtvK1jZxWUuTLE+yfO3atWOrWZLmorGHQpIdgAuBE6vq+6PjqqqA2pTlVdUZVbWkqpbMnz9/C1YqSRprKCTZji4Qzqmqi1rzQ+t2C7XnNa19FbDHyOy7tzZJ0gwZ59lHAT4L3FVVHxsZdQlwbBs+Fvj8SPtb21lI+wMPj+xmkiTNgHljXPargLcAtyW5ubW9HzgNOD/J24D7gSPbuC8ChwIrgB8BvzvG2iRJkxhbKFTVPwDZwOhDJpm+gBPGVY8kaeP8RrMkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqTetUEjyqum0SZK2btPtKfzFNNskSVuxeVONTHIAcCAwP8l7R0b9DLDtOAuTJM28KUMBeBawQ5tux5H27wNvHldRkqRhTBkKVfV14OtJzqqq+2eoJknSQDbWU1jn2UnOABaNzlNVvzqOoiRJw5huKPwt8CngM8Dj4ytHkjSk6YbCY1V1+lgrkSQNbrqnpH4hyduT7JZkl3WPqWZIcmaSNUluH2k7NcmqJDe3x6Ej405JsiLJ3Ulev5k/jyTpaZhuT+HY9vy+kbYCXjLFPGcBnwQ+N6H941X1kdGGJHsDRwO/BLwA+EqSl1aVu6okaQZNKxSq6sWbuuCqujrJomlOfhhwXlU9CnwryQpgP+CaTV2vJGnzTSsUkrx1svaqmtgLmI53tOUtB06qqu8BC4FrR6ZZ2domq2UpsBTghS984WasXpK0IdM9pvDKkcergVOBN23G+k4H9gT2AVYDH93UBVTVGVW1pKqWzJ8/fzNKkCRtyHR3H71z9HWSnYDzNnVlVfXQyDI+DVzaXq4C9hiZdPfWJkmaQZt76ewfApt8nCHJbiMvjwDWnZl0CXB0kmcneTGwGLh+M2uTJG2m6R5T+ALd2UbQXQjvF4HzNzLPucDBwK5JVgJ/AhycZJ+2rPuAPwCoqjuSnA/cCTwGnOCZR5I086Z7SuroKaSPAfdX1cqpZqiqYyZp/uwU038Q+OA065EkjcG0dh+1C+N9k+5KqTsDPxlnUZKkYUz3zmtH0u3j/y3gSOC6JF46W5KeYaa7++iPgFdW1RqAJPOBrwAXjKswSdLMm+7ZR9usC4TmO5swryRpKzHdnsLlSa4Azm2vjwK+OJ6SJElD2dg9mn8eWFBV70vyb4CD2qhrgHPGXZwkaWZtrKfw58ApAFV1EXARQJJ/1cb95lirkyTNqI0dF1hQVbdNbGxti8ZSkSRpMBsLhZ2mGPfcLVmIJGl4GwuF5Ul+f2Jjkn8H3DiekiRJQ9nYMYUTgYuT/DbrQ2AJ8Cy6C9pJkp5BpgyFdqnrA5P8CvCy1nxZVX117JVJkmbcdO+ncBVw1ZhrkSQNzG8lS5J6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6YwuFJGcmWZPk9pG2XZJcmeSe9rxza0+STyRZkeTWJK8YV12SpA0bZ0/hLOANE9pOBpZV1WJgWXsN8OvA4vZYCpw+xrokSRswtlCoqquB705oPgw4uw2fDRw+0v656lwL7JRkt3HVJkma3EwfU1hQVavb8IPAgja8EHhgZLqVre0pkixNsjzJ8rVr146vUkmagwY70FxVBdRmzHdGVS2pqiXz588fQ2WSNHfNdCg8tG63UHte09pXAXuMTLd7a5MkzaCZDoVLgGPb8LHA50fa39rOQtofeHhkN5MkaYbMG9eCk5wLHAzsmmQl8CfAacD5Sd4G3A8c2Sb/InAosAL4EfC746pLkrRhYwuFqjpmA6MOmWTaAk4YVy2SpOnxG82SpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpN68IVaa5D7gEeBx4LGqWpJkF+B/AYuA+4Ajq+p7Q9Q3botOvmyQ9d532hsHWa+krceQPYVfqap9qmpJe30ysKyqFgPL2mtJ0gyaTbuPDgPObsNnA4cPWIskzUlDhUIBX05yY5KlrW1BVa1uww8CCyabMcnSJMuTLF+7du1M1CpJc8YgxxSAg6pqVZKfA65M8s3RkVVVSWqyGavqDOAMgCVLlkw6jSRp8wzSU6iqVe15DXAxsB/wUJLdANrzmiFqk6S5bMZDIcn2SXZcNwy8DrgduAQ4tk12LPD5ma5Nkua6IXYfLQAuTrJu/X9TVZcnuQE4P8nbgPuBIweoTZLmtBkPhaq6F3j5JO3fAQ6Z6XokSevNplNSJUkDMxQkST1DQZLUMxQkST1DQZLUG+obzRrAUFdnBa/QKm0t7ClIknqGgiSpZyhIknqGgiSpZyhIknqGgiSpZyhIknp+T0EzYqjvSPj9CGnT2FOQJPUMBUlSz1CQJPUMBUlSz1CQJPUMBUlSz1CQJPX8noKe0byHhLRp7ClIknr2FKQx8Vvc2hoZCtIzjGGkp8PdR5Kknj0FSVu9IU8oGMq4emb2FCRJvVkXCknekOTuJCuSnDx0PZI0l8yq3UdJtgX+B/BaYCVwQ5JLqurOYSuTtDFzcRfOM9Fs6ynsB6yoqnur6ifAecBhA9ckSXPGrOopAAuBB0ZerwT+9egESZYCS9vLHyS5e4ZqG5ddgX8euohZxO3xZG6P9dwWI/Lfntb2eNGGRsy2UNioqjoDOGPoOraUJMurasnQdcwWbo8nc3us57Z4snFtj9m2+2gVsMfI691bmyRpBsy2ULgBWJzkxUmeBRwNXDJwTZI0Z8yq3UdV9ViSdwBXANsCZ1bVHQOXNW7PmF1hW4jb48ncHuu5LZ5sLNsjVTWO5UqStkKzbfeRJGlAhoIkqWcoDCTJHkmuSnJnkjuSvHvomoaWZNsk30hy6dC1DC3JTkkuSPLNJHclOWDomoaU5D3t/+T2JOcmec7QNc2kJGcmWZPk9pG2XZJcmeSe9rzzlliXoTCcx4CTqmpvYH/ghCR7D1zT0N4N3DV0EbPEfwcur6q9gJczh7dLkoXAu4AlVfUyupNQjh62qhl3FvCGCW0nA8uqajGwrL1+2gyFgVTV6qq6qQ0/QvdPv3DYqoaTZHfgjcBnhq5laEmeD7wG+CxAVf2kqv5l2KoGNw94bpJ5wPOA/ztwPTOqqq4Gvjuh+TDg7DZ8NnD4lliXoTALJFkE7AtcN2wlg/pz4A+BJ4YuZBZ4MbAW+Ku2O+0zSbYfuqihVNUq4CPAt4HVwMNV9eVhq5oVFlTV6jb8ILBgSyzUUBhYkh2AC4ETq+r7Q9czhCS/AaypqhuHrmWWmAe8Aji9qvYFfsgW2jWwNWr7yg+jC8sXANsn+Z1hq5pdqvtuwRb5foGhMKAk29EFwjlVddHQ9QzoVcCbktxHd2XcX03y18OWNKiVwMqqWtdzvIAuJOaqXwO+VVVrq+qnwEXAgQPXNBs8lGQ3gPa8Zkss1FAYSJLQ7TO+q6o+NnQ9Q6qqU6pq96paRHcA8atVNWc/CVbVg8ADSX6hNR0CzOV7inwb2D/J89r/zSHM4QPvIy4Bjm3DxwKf3xILNRSG8yrgLXSfim9uj0OHLkqzxjuBc5LcCuwD/OnA9Qym9ZguAG4CbqN735pTl7xIci5wDfALSVYmeRtwGvDaJPfQ9aZO2yLr8jIXkqR17ClIknqGgiSpZyhIknqGgiSpZyhIknqGgkjyeDsl9vYkX0iy09NY1g+exrzvalcEPWdC+z6jp+smOTXJf9jc9YxDkvdvgWXcl2TXLVHPNNZ1uBdg1GQMBQH8v6rap12B8rvACQPV8XbgtVX12xPa9wFm+3c4phUKSbYddyHTdDhgKOgpDAVNdA3taq1J9ktyTbso2z+t+4ZtkuOSXJTk8nYt9z+buJAku7Z53zjJuPe2XsntSU5sbZ8CXgJ8Kcl7RqZ9FvBfgKNab+aoNmrvJF9Lcm+Sd41M/ztJrm/T/uVkb8LtE/mH2jTLk7wiyRVJ/k+S49s0SfLhVuNt69abZLckV4/0rF6d5DS6K3jePLGX0+b5QZKPJrkFOGCaNT5lmiTHJ/nwyDTHJflkG/67JDemu+fA0gnr/mCSW5Jcm2RBkgOBNwEfbsvfc8K6z0ryqbZt/ne7NhVJFiX5+yQ3tceBU2yTbdty1m2/97Rpfz/JDa2eC5M8r7Xv2eq7LckHRnucSd7X5rk1yX+euK20hVWVjzn+AH7QnrcF/hZ4Q3v9M8C8NvxrwIVt+DjgXuD5wHOA+4E91i2L7mqN19F96p+4rl+m+1bq9sAOwB3Avm3cfcCuk8xzHPDJkdenAv8EPBvYFfgOsB3wi8AXgO3adP8TeOsky7sP+Pdt+OPArcCOwHzgodb+b4Er2zZZQHephd2Ak4A/GtleO45uww1s3wKObMMbrHHdz7+haVp9K0aW+yXgoDa8S3t+LnA78LMj6/7NNvxnwB+34bOAN2+g3rOAy+k+NC6muxbTc+guWf2cNs1iYHkbfso2ab/nK0eWuVN7/tmRtg8A72zDlwLHtOHjWf83+Tq6by+n1XMp8Jqh/2eeyY95SO1TLl0P4S66N0Po3vTPTrKY7s1lu5F5llXVwwBJ7gReBDzQplkGnFBVX59kXQcBF1fVD9u8FwGvBr6xiTVfVlWPAo8mWUP3xn0I3ZvRDUmge4Pc0EXCLmnPtwE7VHdPi0eSPJrumMpBwLlV9Tjdhce+DrwSuAE4M93FDP+uqm6eRq2P0134kGnWOOk0VbW29Yz2B+4B9gL+sc3zriRHtOE96N60vwP8hO6NFOBG4LXTqBfg/Kp6Argnyb1tXd8CPplkn/YzvbRN+5Rt0uZ5SZK/AC4D1l3q+mVJPgDsRPeh4IrWfgDr7wfwN3SXyoYuFF7H+r+PHdrPdvU0fw5tIkNB0I4ptK78FXTHFD4B/Ffgqqo6It09H742Ms+jI8OPs/5v6TG6N5/XA5OFwpYy2foDnF1Vp2zC/E9MWNYTTPF/UVVXJ3kN3Q2Bzkrysar63EbW9eMWLkyzxqmmOQ84EvgmXbhWkoPpenIHVNWPknyN7pM9wE+rfeTmyb+njZl4/ZsC3gM8RHcnuG2AH8OGt0mSl9P9HRzfav49ul7I4VV1S5LjgIM3UkeAD1XVX06zbj1NHlNQr6p+RHfbw5PS3eHq+cCqNvq46S6G7p9/ryT/cZLxfw8cnu6Kl9sDR7S2qTxCt0tiY5YBb07yc9Dfw/ZF06x7sjqPavvG59PdCe36tryHqurTdHeJW3dJ65+2T8pbosapprmY7t4Cx9AFBHS/p++1QNiL7vauG7OxbfpbSbZpxxteAtzd1rO69SDeQreriMm2SbqzqLapqguBP2b9dtoRWN221egJBdfS7bKDJ99q8wrg99Ldd4QkC9dtF42HoaAnqapv0O1jP4ZuH/SHknyDTehVtk/Fx9BdAfbtE8bdRPdp8Xq64w6faeucylV0B5ZHDzRPtt476d6Avpzu6qJX0h0H2BwX022HW4CvAn9Y3SWtDwZuadvkKLp7KUO33/vWyQ40b2qNU01TVd+j28X3oqq6vs1yOTAvyV10V8q8dho/33nA+9KdRLDnJOO/Tfc7+hJwfFX9mO7YxrHpDpjvRXfzH5h8mywEvtZ2S/41sK7X85/ofu//SNfbWedE4L3t5/154OH2836ZbnfSNUluo7ta6nQ+IGgzeZVUSU+S5Czg0qq6YAbX+Ty63ZiV5Gi6g86HzdT6tZ7HFCTNBr9MdxA7wL/Q7YLUAOwpSJJ6HlOQJPUMBUlSz1CQJPUMBUlSz1CQJPX+P5NyvzBlZNthAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "evaluate_results(pred_para, gold_para) # without Universal Sentence Encoder for QA"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 388
        },
        "id": "k0WCxL_OlvsI",
        "outputId": "44621db8-30d1-4c65-96c7-83e40480dfde"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Total queries: 523\n",
            "In top 10 results, number of queries for which -\n",
            "\tRelevant paragraph found: 469 (90 %)\n",
            "\tRelevant paragraph NOT found: 54 (10 %)\n",
            "Mean Rank for which relevant paragraph found: 1.97\n",
            "\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAWc0lEQVR4nO3dfbRddX3n8fdHgqJCBSTNoiHLoKal1LUMNjKg1kWlKtW2wRlFWK1CyzR1xAfUsYO2s2pntNL61LFOsSgMcUqllIeKoCCNKH1AICDP6JAiSDKR3PqAqCMKfueP/cvO4eYmuQn33H3hvl9rnXX2+e2n79333vM5v7332TtVhSRJAI8bugBJ0txhKEiSeoaCJKlnKEiSeoaCJKm3YOgCHon99tuvli5dOnQZkvSoct111/1bVS2catyjOhSWLl3K2rVrhy5Dkh5Vkty9rXHuPpIk9QwFSVLPUJAk9QwFSVLPUJAk9QwFSVLPUJAk9QwFSVLPUJAk9R7V32h+JJaecslg677r1JcPtm5J2h57CpKknqEgSeqNLRSS7JHkmiQ3Jrk1yR+39gOTXJ1kXZK/TfL41v6E9npdG790XLVJkqY2zp7CA8CLqurZwHLgqCSHAX8KfKiqngl8GzixTX8i8O3W/qE2nSRpFo0tFKrzvfZy9/Yo4EXAea19NXB0G17ZXtPGH5kk46pPkrS1sR5TSLJbkhuATcDlwL8C36mqB9sk64HFbXgxcA9AG38f8NQplrkqydokaycmJsZZviTNO2MNhap6qKqWAwcAhwIHzcAyT6+qFVW1YuHCKW8cJEnaRbNy9lFVfQe4Ajgc2DvJ5u9HHABsaMMbgCUAbfxTgG/ORn2SpM44zz5amGTvNvxE4MXA7XTh8Mo22fHAp9rwRe01bfznq6rGVZ8kaWvj/Ebz/sDqJLvRhc+5VXVxktuAc5K8G/gycEab/gzgfydZB3wLOHaMtUmSpjC2UKiqm4BDpmi/k+74wuT2HwKvGlc9kqQd8xvNkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6o0tFJIsSXJFktuS3Jrkza39XUk2JLmhPV42Ms87kqxL8tUkLx1XbZKkqS0Y47IfBN5WVdcn2Qu4LsnlbdyHqur9oxMnORg4FvgF4GeAf0jys1X10BhrlCSNGFtPoao2VtX1bfh+4HZg8XZmWQmcU1UPVNXXgHXAoeOqT5K0tVk5ppBkKXAIcHVrekOSm5KcmWSf1rYYuGdktvVMESJJViVZm2TtxMTEGKuWpPln7KGQZE/gfODkqvoucBrwDGA5sBH4wM4sr6pOr6oVVbVi4cKFM16vJM1nYw2FJLvTBcLZVXUBQFXdW1UPVdVPgI+xZRfRBmDJyOwHtDZJ0iwZ59lHAc4Abq+qD4607z8y2SuAW9rwRcCxSZ6Q5EBgGXDNuOqTJG1tnGcfPR94DXBzkhta2zuB45IsBwq4C/g9gKq6Ncm5wG10Zy6d5JlHkjS7xhYKVfVPQKYY9ZntzPMe4D3jqkmStH1+o1mS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1BtbKCRZkuSKJLcluTXJm1v7vkkuT3JHe96ntSfJh5OsS3JTkueMqzZJ0tTG2VN4EHhbVR0MHAaclORg4BRgTVUtA9a01wC/Cixrj1XAaWOsTZI0hbGFQlVtrKrr2/D9wO3AYmAlsLpNtho4ug2vBD5RnS8BeyfZf1z1SZK2NivHFJIsBQ4BrgYWVdXGNuobwKI2vBi4Z2S29a1t8rJWJVmbZO3ExMTYapak+WjsoZBkT+B84OSq+u7ouKoqoHZmeVV1elWtqKoVCxcunMFKJUljDYUku9MFwtlVdUFrvnfzbqH2vKm1bwCWjMx+QGuTJM2ScZ59FOAM4Paq+uDIqIuA49vw8cCnRtpf285COgy4b2Q3kyRpFiwY47KfD7wGuDnJDa3tncCpwLlJTgTuBo5p4z4DvAxYB/wA+O0x1iZJmsLYQqGq/gnINkYfOcX0BZw0rnokSTvmN5olST1DQZLUMxQkST1DQZLUMxQkST1DQZLUMxQkSb1phUKS50+nTZL06DbdnsJfTLNNkvQott1vNCc5HHgesDDJW0dG/RSw2zgLkyTNvh1d5uLxwJ5tur1G2r8LvHJcRUmShrHdUKiqLwJfTHJWVd09SzVJkgYy3QviPSHJ6cDS0Xmq6kXjKEqSNIzphsLfAR8FPg48NL5yJElDmm4oPFhVp421EknS4KZ7Suqnk7w+yf5J9t38GGtlkqRZN92ewubbZ759pK2Ap89sOZKkIU0rFKrqwHEXIkka3rRCIclrp2qvqk/MbDmSpCFNd/fRc0eG96C7x/L1gKEgSY8h09199MbR10n2Bs4ZS0WSpMHs6qWzvw94nEGSHmOme0zh03RnG0F3IbyfB84dV1GSpGFM95jC+0eGHwTurqr1Y6hHkjSgae0+ahfG+wrdlVL3AX60o3mSnJlkU5JbRtrelWRDkhva42Uj496RZF2SryZ56c7/KJKkR2q6d147BrgGeBVwDHB1kh1dOvss4Kgp2j9UVcvb4zNt+QcDxwK/0Ob5yyTer0GSZtl0dx/9AfDcqtoEkGQh8A/AeduaoaquTLJ0mstfCZxTVQ8AX0uyDjgUuGqa80uSZsB0zz563OZAaL65E/NO9oYkN7XdS/u0tsXAPSPTrG9tW0myKsnaJGsnJiZ2sQRJ0lSm+8Z+aZLLkpyQ5ATgEuAzu7C+04BnAMuBjcAHdnYBVXV6Va2oqhULFy7chRIkSduyo3s0PxNYVFVvT/LvgRe0UVcBZ+/syqrq3pFlfwy4uL3cACwZmfSA1iZJmkU76in8Od39mKmqC6rqrVX1VuDCNm6nJNl/5OUrgM1nJl0EHJvkCUkOBJbRHdiWJM2iHR1oXlRVN09urKqbd3QQOckngSOA/ZKsB/4IOCLJcrovwt0F/F5b3q1JzgVuo/sexElV5R3eJGmW7SgU9t7OuCdub8aqOm6K5jO2M/17gPfsoB5J0hjtaPfR2iS/O7kxyX8ErhtPSZKkoeyop3AycGGS32RLCKwAHk93TECS9Biy3VBoZws9L8kvA89qzZdU1efHXpkkadZN934KVwBXjLkWSdLAdvVbyZKkxyBDQZLUMxQkST1DQZLUMxQkST1DQZLUMxQkST1DQZLUMxQkST1DQZLUMxQkST1DQZLUMxQkST1DQZLUMxQkST1DQZLUMxQkST1DQZLUMxQkST1DQZLUMxQkSb2xhUKSM5NsSnLLSNu+SS5Pckd73qe1J8mHk6xLclOS54yrLknSto2zp3AWcNSktlOANVW1DFjTXgP8KrCsPVYBp42xLknSNowtFKrqSuBbk5pXAqvb8Grg6JH2T1TnS8DeSfYfV22SpKnN9jGFRVW1sQ1/A1jUhhcD94xMt761bSXJqiRrk6ydmJgYX6WSNA8NdqC5qgqoXZjv9KpaUVUrFi5cOIbKJGn+mu1QuHfzbqH2vKm1bwCWjEx3QGuTJM2i2Q6Fi4Dj2/DxwKdG2l/bzkI6DLhvZDeTJGmWLBjXgpN8EjgC2C/JeuCPgFOBc5OcCNwNHNMm/wzwMmAd8APgt8dVlyRp28YWClV13DZGHTnFtAWcNK5aJEnT4zeaJUk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEm9BUMXMB8tPeWSQdZ716kvH2S9kh497ClIknqGgiSpN8juoyR3AfcDDwEPVtWKJPsCfwssBe4Cjqmqbw9RnyTNV0P2FH65qpZX1Yr2+hRgTVUtA9a015KkWTSXdh+tBFa34dXA0QPWIknz0lChUMDnklyXZFVrW1RVG9vwN4BFU82YZFWStUnWTkxMzEatkjRvDHVK6guqakOSnwYuT/KV0ZFVVUlqqhmr6nTgdIAVK1ZMOY0kadcM0lOoqg3teRNwIXAocG+S/QHa86YhapOk+WzWQyHJk5PstXkYeAlwC3ARcHyb7HjgU7NdmyTNd0PsPloEXJhk8/r/pqouTXItcG6SE4G7gWMGqE2S5rVZD4WquhN49hTt3wSOnO16JElbzKVTUiVJAzMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk977w2jwx1xzfwrm/So4U9BUlSz1CQJPUMBUlSz1CQJPUMBUlSz1CQJPUMBUlSz+8paFYM9R0Jvx8h7Rx7CpKknqEgSeoZCpKknqEgSeoZCpKknmcf6TFtyCvDzjee6fXYYE9BktSzpyBpRni/jscGewqSpJ6hIEnqzblQSHJUkq8mWZfklKHrkaT5ZE4dU0iyG/A/gRcD64Frk1xUVbcNW5mkuWw+nmU2ruMoc62ncCiwrqrurKofAecAKweuSZLmjTnVUwAWA/eMvF4P/LvRCZKsAla1l99L8tVZqm1c9gP+begi5hC3x8O5PbZwW4zInz6i7fG0bY2Ya6GwQ1V1OnD60HXMlCRrq2rF0HXMFW6Ph3N7bOG2eLhxbY+5tvtoA7Bk5PUBrU2SNAvmWihcCyxLcmCSxwPHAhcNXJMkzRtzavdRVT2Y5A3AZcBuwJlVdevAZY3bY2ZX2Axxezyc22MLt8XDjWV7pKrGsVxJ0qPQXNt9JEkakKEgSeoZCgNJsiTJFUluS3JrkjcPXdPQkuyW5MtJLh66lqEl2TvJeUm+kuT2JIcPXdOQkryl/Z/ckuSTSfYYuqbZlOTMJJuS3DLStm+Sy5Pc0Z73mYl1GQrDeRB4W1UdDBwGnJTk4IFrGtqbgduHLmKO+B/ApVV1EPBs5vF2SbIYeBOwoqqeRXcSyrHDVjXrzgKOmtR2CrCmqpYBa9rrR8xQGEhVbayq69vw/XT/9IuHrWo4SQ4AXg58fOhahpbkKcALgTMAqupHVfWdYasa3ALgiUkWAE8C/u/A9cyqqroS+Nak5pXA6ja8Gjh6JtZlKMwBSZYChwBXD1vJoP4c+H3gJ0MXMgccCEwA/6vtTvt4kicPXdRQqmoD8H7g68BG4L6q+tywVc0Ji6pqYxv+BrBoJhZqKAwsyZ7A+cDJVfXdoesZQpJfAzZV1XVD1zJHLACeA5xWVYcA32eGdg08GrV95SvpwvJngCcn+a1hq5pbqvtuwYx8v8BQGFCS3ekC4eyqumDoegb0fOA3ktxFd2XcFyX562FLGtR6YH1Vbe45nkcXEvPVrwBfq6qJqvoxcAHwvIFrmgvuTbI/QHveNBMLNRQGkiR0+4xvr6oPDl3PkKrqHVV1QFUtpTuA+PmqmrefBKvqG8A9SX6uNR0JzOd7inwdOCzJk9r/zZHM4wPvIy4Cjm/DxwOfmomFGgrDeT7wGrpPxTe0x8uGLkpzxhuBs5PcBCwH/mTgegbTekznAdcDN9O9b82rS14k+SRwFfBzSdYnORE4FXhxkjvoelOnzsi6vMyFJGkzewqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIJI81E6JvSXJp5Ps/QiW9b1HMO+b2hVBz57Uvnz0dN0k70ryn3d1PeOQ5J0zsIy7kuw3E/VMY11HewFGTcVQEMD/q6rl7QqU3wJOGqiO1wMvrqrfnNS+HJjr3+GYVigk2W3chUzT0YChoK0YCprsKtrVWpMcmuSqdlG2f9n8DdskJyS5IMml7VrufzZ5IUn2a/O+fIpxb229kluSnNzaPgo8HfhskreMTPt44L8Br269mVe3UQcn+UKSO5O8aWT630pyTZv2r6Z6E26fyN/bplmb5DlJLkvyr0le16ZJkve1Gm/evN4k+ye5cqRn9UtJTqW7gucNk3s5bZ7vJflAkhuBw6dZ41bTJHldkveNTHNCko+04b9Pcl26ew6smrTu9yS5McmXkixK8jzgN4D3teU/Y9K6z0ry0bZt/k+7NhVJlib5xyTXt8fztrNNdmvL2bz93tKm/d0k17Z6zk/ypNb+jFbfzUnePdrjTPL2Ns9NSf548rbSDKsqH/P8AXyvPe8G/B1wVHv9U8CCNvwrwPlt+ATgTuApwB7A3cCSzcuiu1rj1XSf+iev6xfpvpX6ZGBP4FbgkDbuLmC/KeY5AfjIyOt3Af8CPAHYD/gmsDvw88Cngd3bdH8JvHaK5d0F/Kc2/CHgJmAvYCFwb2v/D8DlbZssorvUwv7A24A/GNlee41uw21s3wKOacPbrHHzz7+taVp960aW+1ngBW143/b8ROAW4Kkj6/71NvxnwB+24bOAV26j3rOAS+k+NC6juxbTHnSXrN6jTbMMWNuGt9om7fd8+cgy927PTx1pezfwxjZ8MXBcG34dW/4mX0L37eW0ei4GXjj0/8xj+bEAqX3Kpesh3E73Zgjdm/7qJMvo3lx2H5lnTVXdB5DkNuBpwD1tmjXASVX1xSnW9QLgwqr6fpv3AuCXgC/vZM2XVNUDwANJNtG9cR9J92Z0bRLo3iC3dZGwi9rzzcCe1d3T4v4kD6Q7pvIC4JNV9RDdhce+CDwXuBY4M93FDP++qm6YRq0P0V34kGnWOOU0VTXRekaHAXcABwH/3OZ5U5JXtOEldG/a3wR+RPdGCnAd8OJp1AtwblX9BLgjyZ1tXV8DPpJkefuZfrZNu9U2afM8PclfAJcAmy91/awk7wb2pvtQcFlrP5wt9wP4G7pLZUMXCi9hy9/Hnu1nu3KaP4d2kqEgaMcUWlf+MrpjCh8G/jtwRVW9It09H74wMs8DI8MPseVv6UG6N5+XAlOFwkyZav0BVlfVO3Zi/p9MWtZP2M7/RVVdmeSFdDcEOivJB6vqEztY1w9buDDNGrc3zTnAMcBX6MK1khxB15M7vKp+kOQLdJ/sAX5c7SM3D/897cjk698U8BbgXro7wT0O+CFse5skeTbd38HrWs2/Q9cLObqqbkxyAnDEDuoI8N6q+qtp1q1HyGMK6lXVD+hue/i2dHe4egqwoY0+YbqLofvnPyjJf5li/D8CR6e74uWTgVe0tu25n26XxI6sAV6Z5Kehv4ft06ZZ91R1vrrtG19Idye0a9ry7q2qj9HdJW7zJa1/3D4pz0SN25vmQrp7CxxHFxDQ/Z6+3QLhILrbu+7Ijrbpq5I8rh1veDrw1baeja0H8Rq6XUVMtU3SnUX1uKo6H/hDtmynvYCNbVuNnlDwJbpddvDwW21eBvxOuvuOkGTx5u2i8TAU9DBV9WW6fezH0e2Dfm+SL7MTvcr2qfg4uivAvn7SuOvpPi1eQ3fc4eNtndtzBd2B5dEDzVOt9za6N6DPpbu66OV0xwF2xYV02+FG4PPA71d3SesjgBvbNnk13b2UodvvfdNUB5p3tsbtTVNV36bbxfe0qrqmzXIpsCDJ7XRXyvzSNH6+c4C3pzuJ4BlTjP863e/os8DrquqHdMc2jk93wPwgupv/wNTbZDHwhbZb8q+Bzb2e/0r3e/9nut7OZicDb20/7zOB+9rP+zm63UlXJbmZ7mqp0/mAoF3kVVIlPUySs4CLq+q8WVznk+h2Y1aSY+kOOq+crfVrC48pSJoLfpHuIHaA79DtgtQA7ClIknoeU5Ak9QwFSVLPUJAk9QwFSVLPUJAk9f4/OIc4jwKDM7UAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "evaluate_results(pred_para, gold_para) # with Universal Sentence Encoder for QA"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ReX_F_TYnqIs"
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hCR0e6aClvsI"
      },
      "outputs": [],
      "source": [
        "k = 10\n",
        "encoder = 'Universal Sentence Encoder for QA'\n",
        "_, model = load_encoder(encoder)\n",
        "numThemes = len(theme_wise_data.keys())\n",
        "total = 0\n",
        "notInTopKSum = 0\n",
        "rankSum = 0.\n",
        "i = 0\n",
        "\n",
        "\n",
        "for theme in theme_wise_data:\n",
        "    paras, ques, gold_para, _ = load_ques_by_theme(theme, theme_wise_data, answerable_only=True)\n",
        "    sents, para_id = load_sents_from_para(paras)\n",
        "\n",
        "    sents_embed = get_embeddings(sents, encoder, _, model, sents_type=\"Context\")\n",
        "    ques_embed = get_embeddings(ques, encoder, _, model, sents_type=\"Question\")\n",
        "\n",
        "    D, I = get_k_nearest_neighbours(sents_embed, ques_embed, k)\n",
        "    pred_para = sent_id_to_para_id(I, para_id)\n",
        "\n",
        "    notInTopK, avg = getMeanRank(pred_para, gold_para)\n",
        "    total += len(ques)\n",
        "    notInTopKSum += notInTopK\n",
        "    rankSum += avg * (len(ques) - notInTopK)\n",
        "\n",
        "    i += 1\n",
        "    print(f'\\r{i}/{numThemes} themes evaluated', end='')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cRMSo6yDlvsI",
        "outputId": "6f0ccb45-87a5-4c98-dc83-8c84333a4dbe"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Total queries: 50125\n",
            "In top 10 results, number of queries for which -\n",
            "\tRelevant paragraph found: 47511 (94.79 %)\n",
            "\tRelevant paragraph NOT found: 2614 (5.21 %)\n",
            "Mean Rank for which relevant paragraph found: 1.55\n"
          ]
        }
      ],
      "source": [
        "inTopKSum = total - notInTopKSum\n",
        "print(f\"Total queries: {total}\")\n",
        "print(f\"In top {k} results, number of queries for which -\")\n",
        "print(f\"\\tRelevant paragraph found: {inTopKSum} ({round(100.*inTopKSum / total, 2)} %)\")\n",
        "print(f\"\\tRelevant paragraph NOT found: {notInTopKSum} ({round(100.*notInTopKSum / total, 2)} %)\")\n",
        "print(f\"Mean Rank for which relevant paragraph found: {round(rankSum / total, 2)}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "E53UQY3BlvsJ"
      },
      "outputs": [],
      "source": [
        "# 38 mins 05 secs for 50126 queries consisting of 361 themes"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O4D8Xx5ipGzd"
      },
      "source": [
        "### Approach 2: Using passage embeddings"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Alr1OJ8DGNwW"
      },
      "source": [
        "#### 1. Using RocketQA"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JgckoiqnGSBl",
        "outputId": "ac14ec34-5c1a-4429-885b-81fc696d44fd"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting paddlepaddle\n",
            "  Downloading paddlepaddle-2.4.1-cp38-cp38-manylinux1_x86_64.whl (121.6 MB)\n",
            "\u001b[K     |████████████████████████████████| 121.6 MB 45 kB/s \n",
            "\u001b[?25hCollecting rocketqa\n",
            "  Downloading rocketqa-1.1.0-py3-none-any.whl (56 kB)\n",
            "\u001b[K     |████████████████████████████████| 56 kB 3.7 MB/s \n",
            "\u001b[?25hCollecting paddle-bfloat==0.1.7\n",
            "  Downloading paddle_bfloat-0.1.7-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (385 kB)\n",
            "\u001b[K     |████████████████████████████████| 385 kB 59.6 MB/s \n",
            "\u001b[?25hRequirement already satisfied: numpy>=1.13 in /usr/local/lib/python3.8/dist-packages (from paddlepaddle) (1.21.6)\n",
            "Requirement already satisfied: Pillow in /usr/local/lib/python3.8/dist-packages (from paddlepaddle) (7.1.2)\n",
            "Requirement already satisfied: astor in /usr/local/lib/python3.8/dist-packages (from paddlepaddle) (0.8.1)\n",
            "Requirement already satisfied: requests>=2.20.0 in /usr/local/lib/python3.8/dist-packages (from paddlepaddle) (2.23.0)\n",
            "Requirement already satisfied: protobuf<=3.20.0,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from paddlepaddle) (3.19.6)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.8/dist-packages (from paddlepaddle) (1.15.0)\n",
            "Requirement already satisfied: decorator in /usr/local/lib/python3.8/dist-packages (from paddlepaddle) (4.4.2)\n",
            "Requirement already satisfied: opt-einsum==3.3.0 in /usr/local/lib/python3.8/dist-packages (from paddlepaddle) (3.3.0)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests>=2.20.0->paddlepaddle) (2022.12.7)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests>=2.20.0->paddlepaddle) (2.10)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests>=2.20.0->paddlepaddle) (1.24.3)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from requests>=2.20.0->paddlepaddle) (3.0.4)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.8/dist-packages (from rocketqa) (4.64.1)\n",
            "Installing collected packages: paddle-bfloat, rocketqa, paddlepaddle\n",
            "Successfully installed paddle-bfloat-0.1.7 paddlepaddle-2.4.1 rocketqa-1.1.0\n"
          ]
        }
      ],
      "source": [
        "!pip install paddlepaddle rocketqa"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4ixGdeOaGxK-",
        "outputId": "1bbdf27f-6d54-45df-b40d-51c24e920135"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "RocketQA model [v1_marco_de]\n",
            "Download RocketQA model [v1_marco_de]\n",
            "100%|███████████████████████████████████████| 769M/769M [01:42<00:00, 7.88MiB/s]\n",
            "WARNING:root:paddle.fluid.layers.py_reader() may be deprecated in the near future. Please use paddle.fluid.io.DataLoader.from_generator() instead.\n",
            "Load model done\n"
          ]
        }
      ],
      "source": [
        "import rocketqa\n",
        "import numpy as np\n",
        "\n",
        "# init dual encoder\n",
        "dual_encoder = rocketqa.load_model(model=\"v1_marco_de\", use_cuda=False, device_id=0, batch_size=16)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zYjY0ExYYmDr",
        "outputId": "1dd25198-02d3-4b60-9f30-98a6b8d1e991"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Number of examples = 75056\n",
            "\tAnswerable questions = 50126\n",
            "\tNon-Answerable questions = 24930\n",
            "\n",
            "Examples:\n",
            " | Theme  |  Paragraph...  |  Question | Answer_possible | Answer_text | Answer_start\n",
            "1430 | Frédéric_Chopin  |  Some modern commenta...  |  Who said Chopin's works were modeled after Bach, Beethoven, Schubert and Field? | True | ['Richard Taruskin'] | [543]\n",
            "2196 | The_Legend_of_Zelda:_Twilight_Princess  |  Twilight Princess ta...  |  Who releases Bulbins from the Realm of Twilight? | False | [] | []\n",
            "\n",
            "Total 361 themes present.\n"
          ]
        }
      ],
      "source": [
        "train_data = load_data()\n",
        "theme_wise_data = load_theme_wise_data(train_data)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xIem-rL7YmDr",
        "outputId": "5b471568-4f57-4f16-a93d-b51be5ee1554"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Total Questions: 523\n",
            "Total Paragraphs: 66\n"
          ]
        }
      ],
      "source": [
        "paras, ques, gold_para, _ = load_ques_by_theme('Beyoncé', theme_wise_data, answerable_only=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AKho0QPDYmDs",
        "outputId": "a9c21292-15af-4f0e-b01f-f6495b9b8891"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "(66, 768)\n",
            "(523, 768)\n"
          ]
        }
      ],
      "source": [
        "para_embed = np.array(list(dual_encoder.encode_para(para=paras)))\n",
        "ques_embed = np.array(list(dual_encoder.encode_query(query=ques)))\n",
        "\n",
        "print(para_embed.shape)\n",
        "print(ques_embed.shape)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3nw3RZT-YmDs"
      },
      "outputs": [],
      "source": [
        "k = 10\n",
        "D, I = get_k_nearest_neighbours(para_embed, ques_embed, k)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 388
        },
        "id": "-x-HCHIwYmDt",
        "outputId": "bc840129-6c4a-49d4-876a-ac4cf6e91b3b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Total queries: 523\n",
            "In top 10 results, number of queries for which -\n",
            "\tRelevant paragraph found: 420 (80 %)\n",
            "\tRelevant paragraph NOT found: 103 (20 %)\n",
            "Mean Rank for which relevant paragraph found: 2.36\n",
            "\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAUFklEQVR4nO3dfbRddX3n8fcHgqJABZo0iwGWUZuWMnaJNjqA1EVLtShtwRnLw5pqsEwzjFRFHVtsO6vOjI5Mrba1tlqqDLhKYSgPFUVBGnnotCAE5BkdGAwSJpBULaKOKPCdP/bOL4fLTe4J3HP3Te77tdZdZ5/ffvrendzzOb+9z/ntVBWSJAHsNHQBkqT5w1CQJDWGgiSpMRQkSY2hIElqFg1dwDOxePHiWrZs2dBlSNJ25cYbb/ynqloy3bztOhSWLVvGmjVrhi5DkrYrSe7b0jxPH0mSGkNBktQYCpKkxlCQJDWGgiSpMRQkSY2hIElqDAVJUmMoSJKa7fobzc/EstMuHWzfa08/arB9S9LW2FOQJDWGgiSpMRQkSY2hIElqDAVJUmMoSJIaQ0GS1BgKkqTGUJAkNYaCJKkxFCRJjaEgSWoMBUlSYyhIkhpDQZLUGAqSpMZQkCQ1hoIkqTEUJEmNoSBJaiYWCkn2T3JlkjuT3JHk7X373kmuSHJ3/7hX354kH0lyT5Jbk7xsUrVJkqY3yZ7CY8C7qupA4GDglCQHAqcBq6tqObC6fw7wWmB5/7MK+NgEa5MkTWNioVBV66vqpn76EeAuYF/gaODsfrGzgWP66aOBT1XnOmDPJPtMqj5J0lPNyTWFJMuAlwJfApZW1fp+1oPA0n56X+D+kdXW9W1Tt7UqyZokazZu3DixmiVpIZp4KCTZHbgQOLWqvj06r6oKqG3ZXlWdUVUrqmrFkiVLZrFSSdJEQyHJLnSBcE5VXdQ3P7TptFD/uKFvfwDYf2T1/fo2SdIcmeSnjwJ8Erirqj48MusSYGU/vRL49Ej7m/pPIR0MPDxymkmSNAcWTXDbrwTeCNyW5Oa+7XeA04Hzk5wE3Acc28/7HPA64B7ge8CbJ1ibJGkaEwuFqvpfQLYw+4hpli/glEnVI0mamd9oliQ1hoIkqTEUJEmNoSBJagwFSVJjKEiSGkNBktQYCpKkxlCQJDWGgiSpMRQkSY2hIElqDAVJUmMoSJIaQ0GS1BgKkqTGUJAkNYaCJKkxFCRJjaEgSWoMBUlSYyhIkhpDQZLUGAqSpMZQkCQ1hoIkqTEUJEmNoSBJagwFSVJjKEiSGkNBktQYCpKkxlCQJDWGgiSpMRQkSY2hIElqDAVJUjOxUEhyZpINSW4faXtvkgeS3Nz/vG5k3nuS3JPkq0l+cVJ1SZK2bJI9hbOAI6dp/6OqOqj/+RxAkgOB44F/2a/z50l2nmBtkqRpTCwUquoa4JtjLn40cF5VPVpVXwPuAV4xqdokSdMb4prCbya5tT+9tFffti9w/8gy6/q2p0iyKsmaJGs2btw46VolaUGZ61D4GPAi4CBgPfChbd1AVZ1RVSuqasWSJUtmuz5JWtDmNBSq6qGqeryqngD+ks2niB4A9h9ZdL++TZI0h+Y0FJLsM/L09cCmTyZdAhyf5NlJXgAsB66fy9okSbBoUhtOci5wOLA4yTrg94HDkxwEFLAW+PcAVXVHkvOBO4HHgFOq6vFJ1SZJmt7EQqGqTpim+ZNbWf79wPsnVY8kaWZ+o1mS1BgKkqTGUJAkNYaCJKkxFCRJjaEgSWoMBUlSYyhIkhpDQZLUGAqSpGasUEjyynHaJEnbt3F7Cn86ZpskaTu21QHxkhwCHAosSfLOkVk/AngPZUnawcw0SuqzgN375fYYaf828IZJFSVJGsZWQ6GqrgauTnJWVd03RzVJkgYy7v0Unp3kDGDZ6DpV9fOTKEqSNIxxQ+FvgI8DnwC8I5ok7aDGDYXHqupjE61EkjS4cT+S+pkkb0myT5K9N/1MtDJJ0pwbt6ewsn9890hbAS+c3XIkSUMaKxSq6gWTLkSSNLyxQiHJm6Zrr6pPzW45kqQhjXv66OUj07sCRwA3AYaCJO1Axj199NbR50n2BM6bSEWSpME83aGzvwt4nUGSdjDjXlP4DN2njaAbCO+ngPMnVZQkaRjjXlP4w5Hpx4D7qmrdBOqRJA1orNNH/cB4X6EbKXUv4AeTLEqSNIxx77x2LHA98KvAscCXkjh0tiTtYMY9ffS7wMuragNAkiXA3wEXTKowSdLcG/fTRzttCoTeN7ZhXUnSdmLcnsJlSS4Hzu2fHwd8bjIlSZKGMtM9mn8cWFpV707yr4HD+lnXAudMujhJ0tyaqafwx8B7AKrqIuAigCQ/3c/75YlWJ0maUzNdF1haVbdNbezblk2kIknSYGYKhT23Mu85s1mIJGl4M4XCmiS/MbUxyb8DbtzaiknOTLIhye0jbXsnuSLJ3f3jXn17knwkyT1Jbk3ysqfzy0iSnpmZQuFU4M1Jrkryof7nauAk4O0zrHsWcOSUttOA1VW1HFjdPwd4LbC8/1kFeD9oSRrAVi80V9VDwKFJfg54cd98aVV9caYNV9U1SZZNaT4aOLyfPhu4Cvjtvv1TVVXAdUn2TLJPVa0f8/eQJM2Cce+ncCVw5Szsb+nIC/2DwNJ+el/g/pHl1vVthoIkzaHBvpXc9wpqxgWnSLIqyZokazZu3DiByiRp4ZrrUHgoyT4A/eOmoTMeAPYfWW6/vu0pquqMqlpRVSuWLFky0WIlaaGZ61C4BFjZT68EPj3S/qb+U0gHAw97PUGS5t64Yx9tsyTn0l1UXpxkHfD7wOnA+UlOAu6jG4YbunGUXgfcA3wPePOk6pIkbdnEQqGqTtjCrCOmWbaAUyZViyRpPA5/LUlqDAVJUmMoSJIaQ0GS1BgKkqTGUJAkNYaCJKkxFCRJjaEgSWoMBUlSYyhIkhpDQZLUGAqSpMZQkCQ1hoIkqTEUJEmNoSBJagwFSVJjKEiSGkNBktQYCpKkZtHQBSxEy067dJD9rj39qEH2K2n7YU9BktQYCpKkxlCQJDWGgiSpMRQkSY2hIElqDAVJUmMoSJIaQ0GS1BgKkqTGUJAkNYaCJKkxFCRJjaEgSWoMBUlSM8j9FJKsBR4BHgceq6oVSfYG/iewDFgLHFtV3xqiPklaqIbsKfxcVR1UVSv656cBq6tqObC6fy5JmkPz6fTR0cDZ/fTZwDED1iJJC9JQoVDAF5LcmGRV37a0qtb30w8CS6dbMcmqJGuSrNm4ceNc1CpJC8ZQ92g+rKoeSPJjwBVJvjI6s6oqSU23YlWdAZwBsGLFimmXkSQ9PYP0FKrqgf5xA3Ax8ArgoST7APSPG4aoTZIWsjkPhSS7Jdlj0zTwGuB24BJgZb/YSuDTc12bJC10Q5w+WgpcnGTT/v+6qi5LcgNwfpKTgPuAYweoTZIWtDkPhaq6F3jJNO3fAI6Y63okSZvNp4+kSpIGZihIkhpDQZLUDPU9BQ1g2WmXDrbvtacfNdi+JY3PnoIkqTEUJEmNoSBJagwFSVJjKEiSGkNBktQYCpKkxlCQJDWGgiSpMRQkSY2hIElqDAVJUmMoSJIaQ0GS1BgKkqTGUJAkNd5kR3NiqBv8eHMfadvYU5AkNfYUtEPzFqTStrGnIElqDAVJUmMoSJIaQ0GS1BgKkqTGUJAkNYaCJKnxewrShPgtbm2P7ClIkhp7CtIOxh7K3NkRvzFvKEiaFTviC+RC5OkjSVJjKEiSGk8fSdruDXnqakcz73oKSY5M8tUk9yQ5beh6JGkhmVehkGRn4M+A1wIHAickOXDYqiRp4ZhXoQC8Arinqu6tqh8A5wFHD1yTJC0Y8+2awr7A/SPP1wH/anSBJKuAVf3T7yT56hzVNimLgX8auoh5xOPxZB6PzTwWI/Lfn9HxeP6WZsy3UJhRVZ0BnDF0HbMlyZqqWjF0HfOFx+PJPB6beSyebFLHY76dPnoA2H/k+X59myRpDsy3ULgBWJ7kBUmeBRwPXDJwTZK0YMyr00dV9ViS3wQuB3YGzqyqOwYua9J2mFNhs8Tj8WQej808Fk82keORqprEdiVJ26H5dvpIkjQgQ0GS1BgKA0myf5Irk9yZ5I4kbx+6pqEl2TnJl5N8duhahpZkzyQXJPlKkruSHDJ0TUNK8o7+7+T2JOcm2XXomuZSkjOTbEhy+0jb3kmuSHJ3/7jXbOzLUBjOY8C7qupA4GDgFIf04O3AXUMXMU/8CXBZVR0AvIQFfFyS7Au8DVhRVS+m+xDK8cNWNefOAo6c0nYasLqqlgOr++fPmKEwkKpaX1U39dOP0P3R7ztsVcNJsh9wFPCJoWsZWpLnAa8CPglQVT+oqn8etqrBLQKek2QR8Fzg/w5cz5yqqmuAb05pPho4u58+GzhmNvZlKMwDSZYBLwW+NGwlg/pj4LeAJ4YuZB54AbAR+B/96bRPJNlt6KKGUlUPAH8IfB1YDzxcVV8Ytqp5YWlVre+nHwSWzsZGDYWBJdkduBA4taq+PXQ9Q0jyS8CGqrpx6FrmiUXAy4CPVdVLge8yS6cGtkf9ufKj6cLyXwC7Jfm1YauaX6r7bsGsfL/AUBhQkl3oAuGcqrpo6HoG9ErgV5KspRsZ9+eT/NWwJQ1qHbCuqjb1HC+gC4mF6heAr1XVxqr6IXARcOjANc0HDyXZB6B/3DAbGzUUBpIkdOeM76qqDw9dz5Cq6j1VtV9VLaO7gPjFqlqw7wSr6kHg/iQ/2TcdAdw5YElD+zpwcJLn9n83R7CAL7yPuARY2U+vBD49Gxs1FIbzSuCNdO+Kb+5/Xjd0UZo33gqck+RW4CDgvw1cz2D6HtMFwE3AbXSvWwtqyIsk5wLXAj+ZZF2Sk4DTgVcnuZuuN3X6rOzLYS4kSZvYU5AkNYaCJKkxFCRJjaEgSWoMBUlSYyiIJI/3H4m9Pclnkuz5DLb1nWew7tv6EUHPmdJ+0OjHdZO8N8l/fLr7mYQkvzML21ibZPFs1DPGvo5xAEZNx1AQwP+rqoP6ESi/CZwyUB1vAV5dVf92SvtBwHz/DsdYoZBk50kXMqZjAENBT2EoaKpr6UdrTfKKJNf2g7L946Zv2CY5MclFSS7rx3L/g6kbSbK4X/eoaea9s++V3J7k1L7t48ALgc8necfIss8C/gtwXN+bOa6fdWCSq5Lcm+RtI8v/WpLr+2X/YroX4f4d+Qf6ZdYkeVmSy5P8nyQn98skyQf7Gm/btN8k+yS5ZqRn9bNJTqcbwfPmqb2cfp3vJPlQkluAQ8as8SnLJDk5yQdHljkxyUf76b9NcmO6ew6smrLv9ye5Jcl1SZYmORT4FeCD/fZfNGXfZyX5eH9s/nc/NhVJliX5+yQ39T+HbuWY7NxvZ9Pxe0e/7G8kuaGv58Ikz+3bX9TXd1uS9432OJO8u1/n1iT/eeqx0iyrKn8W+A/wnf5xZ+BvgCP75z8CLOqnfwG4sJ8+EbgXeB6wK3AfsP+mbdGN1vglunf9U/f1M3TfSt0N2B24A3hpP28tsHiadU4EPjry/L3APwLPBhYD3wB2AX4K+AywS7/cnwNvmmZ7a4H/0E//EXArsAewBHiob/83wBX9MVlKN9TCPsC7gN8dOV57jB7DLRzfAo7tp7dY46bff0vL9PXdM7LdzwOH9dN794/PAW4HfnRk37/cT/8B8Hv99FnAG7ZQ71nAZXRvGpfTjcW0K92Q1bv2yywH1vTTTzkm/b/zFSPb3LN//NGRtvcBb+2nPwuc0E+fzOb/k6+h+/Zy+no+C7xq6L+ZHflnEVL/Lpeuh3AX3YshdC/6ZydZTvfissvIOqur6mGAJHcCzwfu75dZDZxSVVdPs6/DgIur6rv9uhcBPwt8eRtrvrSqHgUeTbKB7oX7CLoXoxuSQPcCuaVBwi7pH28Ddq/unhaPJHk03TWVw4Bzq+pxuoHHrgZeDtwAnJluMMO/raqbx6j1cbqBDxmzxmmXqaqNfc/oYOBu4ADgH/p13pbk9f30/nQv2t8AfkD3QgpwI/DqMeoFOL+qngDuTnJvv6+vAR9NclD/O/1Ev+xTjkm/zguT/ClwKbBpqOsXJ3kfsCfdm4LL+/ZD2Hw/gL+mGyobulB4DZv/f+ze/27XjPl7aBsZCoL+mkLflb+c7prCR4D/ClxZVa9Pd8+Hq0bWeXRk+nE2/196jO7F5xeB6UJhtky3/wBnV9V7tmH9J6Zs6wm28ndRVdckeRXdDYHOSvLhqvrUDPv6fh8ujFnj1pY5DzgW+ApduFaSw+l6codU1feSXEX3zh7gh9W/5ebJ/04zmTr+TQHvAB6iuxPcTsD3YcvHJMlL6P4fnNzX/Ot0vZBjquqWJCcCh89QR4APVNVfjFm3niGvKaipqu/R3fbwXenucPU84IF+9onjbobuj/+AJL89zfy/B45JN+LlbsDr+7ateYTulMRMVgNvSPJj0O5h+/wx656uzuP6c+NL6O6Edn2/vYeq6i/p7hK3aUjrH/bvlGejxq0tczHdvQVOoAsI6P6dvtUHwgF0t3edyUzH9FeT7NRfb3gh8NV+P+v7HsQb6U4VMd0xSfcpqp2q6kLg99h8nPYA1vfHavQDBdfRnbKDJ99q83Lg19Pdd4Qk+246LpoMQ0FPUlVfpjvHfgLdOegPJPky29Cr7N8Vn0A3Auxbpsy7ie7d4vV01x0+0e9za66ku7A8eqF5uv3eSfcC9IV0o4teQXcd4Om4mO443AJ8Efit6oa0Phy4pT8mx9HdSxm68963TneheVtr3NoyVfUtulN8z6+q6/tVLgMWJbmLbqTM68b4/c4D3p3uQwQvmmb+1+n+jT4PnFxV36e7trEy3QXzA+hu/gPTH5N9gav605J/BWzq9fwnun/3f6Dr7WxyKvDO/vf9ceDh/vf9At3ppGuT3EY3Wuo4bxD0NDlKqqQnSXIW8NmqumAO9/lcutOYleR4uovOR8/V/rWZ1xQkzQc/Q3cRO8A/052C1ADsKUiSGq8pSJIaQ0GS1BgKkqTGUJAkNYaCJKn5/6qXS1Jcwy1iAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "evaluate_results(I, gold_para)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NYDx2X84Tfpy"
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pfJHs6rkjXwy"
      },
      "source": [
        "#### 2. Using DPRReader"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "12Sag-hE4Sli",
        "outputId": "5d985d9e-5ef6-45a5-89eb-2b9fc2796087"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting transformers\n",
            "  Downloading transformers-4.25.1-py3-none-any.whl (5.8 MB)\n",
            "\u001b[K     |████████████████████████████████| 5.8 MB 4.6 MB/s \n",
            "\u001b[?25hRequirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.8/dist-packages (from transformers) (4.64.1)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.8/dist-packages (from transformers) (21.3)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.8/dist-packages (from transformers) (1.21.6)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.8/dist-packages (from transformers) (2022.6.2)\n",
            "Collecting tokenizers!=0.11.3,<0.14,>=0.11.1\n",
            "  Downloading tokenizers-0.13.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.6 MB)\n",
            "\u001b[K     |████████████████████████████████| 7.6 MB 32.4 MB/s \n",
            "\u001b[?25hRequirement already satisfied: requests in /usr/local/lib/python3.8/dist-packages (from transformers) (2.23.0)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.8/dist-packages (from transformers) (3.8.2)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.8/dist-packages (from transformers) (6.0)\n",
            "Collecting huggingface-hub<1.0,>=0.10.0\n",
            "  Downloading huggingface_hub-0.11.1-py3-none-any.whl (182 kB)\n",
            "\u001b[K     |████████████████████████████████| 182 kB 58.6 MB/s \n",
            "\u001b[?25hRequirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.8/dist-packages (from huggingface-hub<1.0,>=0.10.0->transformers) (4.4.0)\n",
            "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging>=20.0->transformers) (3.0.9)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests->transformers) (2022.12.7)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests->transformers) (1.24.3)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests->transformers) (2.10)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from requests->transformers) (3.0.4)\n",
            "Installing collected packages: tokenizers, huggingface-hub, transformers\n",
            "Successfully installed huggingface-hub-0.11.1 tokenizers-0.13.2 transformers-4.25.1\n"
          ]
        }
      ],
      "source": [
        "!pip install transformers"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 411,
          "referenced_widgets": [
            "514e923dad884a058bcbee6c35c67778",
            "54c4585166a9403cb028a9dd27f690cb",
            "6ecc4ea49d4c4444806c903878895524",
            "446e743d8c72463ea8835763b03107e2",
            "d91faddd2b4f4fcf9df545f4a1b65ce7",
            "af6429eabf96430c90d8491e684d28f6",
            "a220f6c932264395bc32521437450143",
            "99233caa172a4611b764641d5ff7329e",
            "fdf23b9aeb524cd691d4cbced848567b",
            "3b5016908252422b9670da0653623322"
          ]
        },
        "id": "ymUnIDKMgsY5",
        "outputId": "d008591e-e34c-492d-85ff-f61928308da1"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "514e923dad884a058bcbee6c35c67778",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/28.0 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "54c4585166a9403cb028a9dd27f690cb",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/232k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "6ecc4ea49d4c4444806c903878895524",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/466k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "446e743d8c72463ea8835763b03107e2",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/492 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization. \n",
            "The tokenizer class you load from this checkpoint is 'DPRQuestionEncoderTokenizer'. \n",
            "The class this function is called from is 'DPRContextEncoderTokenizer'.\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "d91faddd2b4f4fcf9df545f4a1b65ce7",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/438M [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "af6429eabf96430c90d8491e684d28f6",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/28.0 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "a220f6c932264395bc32521437450143",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/232k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "99233caa172a4611b764641d5ff7329e",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/466k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "fdf23b9aeb524cd691d4cbced848567b",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/493 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "3b5016908252422b9670da0653623322",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/438M [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "import torch\n",
        "\n",
        "from transformers import DPRContextEncoder, DPRContextEncoderTokenizer\n",
        "from transformers import DPRQuestionEncoder, DPRQuestionEncoderTokenizer\n",
        "\n",
        "para_tokenizer = DPRContextEncoderTokenizer.from_pretrained(\"facebook/dpr-ctx_encoder-single-nq-base\")\n",
        "para_model = DPRContextEncoder.from_pretrained(\"facebook/dpr-ctx_encoder-single-nq-base\")\n",
        "\n",
        "ques_tokenizer = DPRQuestionEncoderTokenizer.from_pretrained(\"facebook/dpr-question_encoder-single-nq-base\")\n",
        "ques_model = DPRQuestionEncoder.from_pretrained(\"facebook/dpr-question_encoder-single-nq-base\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QWcll9NWgL77",
        "outputId": "2188cac4-06b7-4f21-85fa-b28d7dd37cde"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Number of examples = 75056\n",
            "\tAnswerable questions = 50126\n",
            "\tNon-Answerable questions = 24930\n",
            "\n",
            "Examples:\n",
            " | Theme  |  Paragraph...  |  Question | Answer_possible | Answer_text | Answer_start\n",
            "1430 | Frédéric_Chopin  |  Some modern commenta...  |  Who said Chopin's works were modeled after Bach, Beethoven, Schubert and Field? | True | ['Richard Taruskin'] | [543]\n",
            "2196 | The_Legend_of_Zelda:_Twilight_Princess  |  Twilight Princess ta...  |  Who releases Bulbins from the Realm of Twilight? | False | [] | []\n",
            "\n",
            "Total 361 themes present.\n"
          ]
        }
      ],
      "source": [
        "train_data = load_data()\n",
        "theme_wise_data = load_theme_wise_data(train_data)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "T58FeVJcgL78",
        "outputId": "67de2e60-f354-4714-a551-02e0e11ddcab"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Total Questions: 523\n",
            "Total Paragraphs: 66\n"
          ]
        }
      ],
      "source": [
        "paras, ques, gold_para, _ = load_ques_by_theme('Beyoncé', theme_wise_data, answerable_only=True)\n",
        "# sents, para_id = load_sents_from_para(paras)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5WKN0Bna67Ub",
        "outputId": "67398f4c-410b-4fdd-d477-e9ae382ac7ce"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "100.0 % completed\n",
            "torch.Size([66, 768])\n"
          ]
        }
      ],
      "source": [
        "para_embeds = []\n",
        "for i, para in enumerate(paras):\n",
        "    para_input_ids = para_tokenizer(para, return_tensors=\"pt\")[\"input_ids\"]\n",
        "    with torch.no_grad():\n",
        "        para_embeddings = para_model(para_input_ids).pooler_output\n",
        "    para_embeds.append(para_embeddings)\n",
        "    print(f'\\r{round(100.*(i+1) / len(paras), 2)} % completed', end='')\n",
        "\n",
        "p = torch.cat(para_embeds, dim=0)\n",
        "print(f'\\n{p.shape}')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "En-j029H7Uod"
      },
      "outputs": [],
      "source": [
        "ques_embeds = []\n",
        "for i, query in enumerate(ques):\n",
        "    ques_input_ids = ques_tokenizer(query, padding=True, return_tensors=\"pt\")[\"input_ids\"]\n",
        "    with torch.no_grad():\n",
        "        ques_embeddings = ques_model(ques_input_ids).pooler_output\n",
        "    ques_embeds.append(ques_embeddings)\n",
        "    print(f'\\r{round(100.*(i+1) / len(ques), 2)} % completed', end='')\n",
        "\n",
        "q = torch.cat(ques_embeds, dim=0)\n",
        "print(f'\\n{q.shape}')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "s4MT_tgigL79"
      },
      "outputs": [],
      "source": [
        "k = 10\n",
        "D, I = get_k_nearest_neighbours(p, q, k)\n",
        "# pred_para = sent_id_to_para_id(I, para_id)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 388
        },
        "id": "OFQkmJDegL7-",
        "outputId": "158013be-e7c5-4a32-9375-bb502d8307cd"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Total queries: 523\n",
            "In top 10 results, number of queries for which -\n",
            "\tRelevant paragraph found: 346 (66 %)\n",
            "\tRelevant paragraph NOT found: 177 (34 %)\n",
            "Mean Rank for which relevant paragraph found: 3.58\n",
            "\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAUsUlEQVR4nO3de7RedX3n8feHBMpVATmTRQMaVCp1Matgo8PFuhjRDpVWcIZyWVZDZZrF6IiAo0TbWXZmnBqr46V1RkvRElcZlAIWhAoyCNKpCIb7TQYGuYQJJFXEWxXB7/yxd345hJPkSXLO2Sc579daz3r2s2+/79nnnOfz7Mvz26kqJEkC2G7oAiRJM4ehIElqDAVJUmMoSJIaQ0GS1MwduoAtsddee9WCBQuGLkOStio33XTTP1bV2ETTtupQWLBgAcuXLx+6DEnaqiR5aH3TPHwkSWoMBUlSYyhIkhpDQZLUGAqSpMZQkCQ1hoIkqTEUJEmNoSBJarbqbzRviQVLLh+s7QeXHj1Y25K0Ie4pSJIaQ0GS1BgKkqTGUJAkNYaCJKkxFCRJjaEgSWoMBUlSYyhIkhpDQZLUGAqSpMZQkCQ1hoIkqTEUJEnNlIVCks8lWZXkznHj9kxyVZL7+uc9+vFJ8mdJ7k9ye5JXTFVdkqT1m8o9hXOBo9YZtwS4uqr2B67uXwP8FrB//1gMfHoK65IkrceUhUJVXQd8b53RxwDL+uFlwLHjxn++Ot8Edk+y91TVJkma2HSfU5hXVSv74ceAef3wfOCRcfOt6MdJkqbRYCeaq6qA2tTlkixOsjzJ8tWrV09BZZI0e013KDy+5rBQ/7yqH/8osO+4+fbpxz1HVZ1dVQurauHY2NiUFitJs810h8KlwKJ+eBFwybjxb+2vQjoEeHLcYSZJ0jSZO1UrTnI+cASwV5IVwAeApcAFSU4BHgKO72f/O+ANwP3AT4Dfn6q6JEnrN2WhUFUnrWfSkRPMW8A7pqoWSdJo/EazJKkxFCRJjaEgSWoMBUlSYyhIkhpDQZLUGAqSpMZQkCQ1hoIkqTEUJEmNoSBJagwFSVJjKEiSGkNBktQYCpKkxlCQJDWGgiSpMRQkSY2hIElqDAVJUmMoSJIaQ0GS1BgKkqTGUJAkNYaCJKkxFCRJjaEgSWoMBUlSYyhIkppBQiHJGUnuSnJnkvOT7JhkvyQ3JLk/yReT7DBEbZI0m017KCSZD5wGLKyqA4E5wInAh4GPV9VLgSeAU6a7Nkma7YY6fDQX2CnJXGBnYCXwWuDCfvoy4NiBapOkWWvaQ6GqHgU+CjxMFwZPAjcB36+qp/vZVgDzJ1o+yeIky5MsX7169XSULEmzxhCHj/YAjgH2A34Z2AU4atTlq+rsqlpYVQvHxsamqEpJmp2GOHz0OuA7VbW6qn4OXAwcDuzeH04C2Ad4dIDaJGlWGyIUHgYOSbJzkgBHAncD1wDH9fMsAi4ZoDZJmtWGOKdwA90J5ZuBO/oazgbOAs5Mcj/wAuCz012bJM12czc+y+Srqg8AH1hn9APAqwYoR5LU8xvNkqTGUJAkNYaCJKkxFCRJjaEgSWoMBUlSYyhIkhpDQZLUGAqSpMZQkCQ1hoIkqTEUJEmNoSBJagwFSVJjKEiSGkNBktQYCpKkxlCQJDWGgiSpMRQkSY2hIElqDAVJUmMoSJIaQ0GS1IwUCkkOH2WcJGnrNuqewp+POE6StBWbu6GJSQ4FDgPGkpw5btLzgDlTWZgkafptMBSAHYBd+/l2Gzf+B8BxU1WUJGkYGwyFqvo68PUk51bVQ9NUkyRpIBvbU1jjl5KcDSwYv0xVvXZzGk2yO3AOcCBQwNuAe4Ev9m08CBxfVU9szvolSZtn1FD4G+AzdG/kz0xCu58Erqiq45LsAOwMvB+4uqqWJlkCLAHOmoS2JEkjGjUUnq6qT09Gg0meD7wGOBmgqp4CnkpyDHBEP9sy4FoMBUmaVqNekvrlJG9PsneSPdc8NrPN/YDVwF8luSXJOUl2AeZV1cp+nseAeRMtnGRxkuVJlq9evXozS5AkTWTUUFgEvAf4BnBT/1i+mW3OBV4BfLqqDgZ+THeoqKmqojvX8BxVdXZVLayqhWNjY5tZgiRpIiMdPqqq/SaxzRXAiqq6oX99IV0oPJ5k76pamWRvYNUktilJGsFIoZDkrRONr6rPb2qDVfVYkkeSvKyq7gWOBO7uH4uApf3zJZu6bknSlhn1RPMrxw3vSPdGfjOwyaHQeydwXn/l0QPA79MdyrogySnAQ8Dxm7nuGW/BkssHaffBpUcP0q6krceoh4/eOf51/z2DL2xuo1V1K7BwgklHbu46JUlbbnO7zv4x3VVEkqRtyKjnFL7M2quB5gC/ClwwVUVJkoYx6jmFj44bfhp4qKpWTEE9kqQBjXT4qO8Y79t0PaXuATw1lUVJkoYx6p3XjgduBH6X7qqgG5LYdbYkbWNGPXz0h8Arq2oVQJIx4H/RffFMkrSNGPXqo+3WBELvu5uwrCRpKzHqnsIVSa4Ezu9fnwD83dSUJEkaysbu0fxSut5L35PkXwOv7iddD5w31cVJkqbXxvYUPgG8D6CqLgYuBkjyz/tpvzOl1UmSptXGzgvMq6o71h3Zj1swJRVJkgazsVDYfQPTdprMQiRJw9tYKCxP8gfrjkzyb+lutCNJ2oZs7JzC6cCXkryZtSGwENgBeNNUFiZJmn4bDIWqehw4LMm/BA7sR19eVV+b8sokSdNu1PspXANcM8W1SJIG5reSJUmNoSBJagwFSVJjKEiSGkNBktQYCpKkxlCQJDWGgiSpMRQkSY2hIElqDAVJUmMoSJKawUIhyZwktyS5rH+9X5Ibktyf5ItJdhiqNkmarYbcU3gXcM+41x8GPl5VLwWeAE4ZpCpJmsUGCYUk+wBHA+f0rwO8Friwn2UZcOwQtUnSbDbS/RSmwCeA9wK79a9fAHy/qp7uX68A5k+0YJLFwGKAF77whVNc5rZlwZLLB2v7waVHD9a2pNFN+55Ckt8GVlXVZt3juarOrqqFVbVwbGxskquTpNltiD2Fw4E3JnkDsCPwPOCTwO5J5vZ7C/sAjw5QmyTNatO+p1BV76uqfapqAXAi8LWqejPd7T6P62dbBFwy3bVJ0mw3k76ncBZwZpL76c4xfHbgeiRp1hnqRDMAVXUtcG0//ADwqiHrkaTZbibtKUiSBmYoSJIaQ0GS1BgKkqTGUJAkNYaCJKkxFCRJjaEgSWoMBUlSYyhIkhpDQZLUGAqSpMZQkCQ1hoIkqTEUJEmNoSBJagwFSVIz6J3XpKm2YMnlg7T74NKjB2lX2lLuKUiSGkNBktQYCpKkxlCQJDWGgiSp8eojTYuhrgKStGncU5AkNYaCJKkxFCRJjecUpG2M3+LWlnBPQZLUTHsoJNk3yTVJ7k5yV5J39eP3THJVkvv65z2muzZJmu2GOHz0NPDuqro5yW7ATUmuAk4Grq6qpUmWAEuAswaoT9piXoKrrdW07ylU1cqqurkf/iFwDzAfOAZY1s+2DDh2umuTpNlu0HMKSRYABwM3APOqamU/6TFg3nqWWZxkeZLlq1evnpY6JWm2GCwUkuwKXAScXlU/GD+tqgqoiZarqrOramFVLRwbG5uGSiVp9hgkFJJsTxcI51XVxf3ox5Ps3U/fG1g1RG2SNJsNcfVRgM8C91TVx8ZNuhRY1A8vAi6Z7tokabYb4uqjw4G3AHckubUf935gKXBBklOAh4DjB6hNkma1aQ+FqvrfQNYz+cjprEWS9Gx+o1mS1BgKkqTGUJAkNYaCJKmx62xJk2LI/p7stnvyGAqStnreQ2LyePhIktQYCpKkxlCQJDWeU5CkzbQtnlx3T0GS1BgKkqTGUJAkNYaCJKkxFCRJjaEgSWoMBUlSYyhIkhpDQZLUGAqSpMZQkCQ1hoIkqTEUJEmNoSBJagwFSVJjKEiSGkNBktQYCpKkxlCQJDUzKhSSHJXk3iT3J1kydD2SNNvMmFBIMgf478BvAS8HTkry8mGrkqTZZcaEAvAq4P6qeqCqngK+ABwzcE2SNKvMHbqAceYDj4x7vQL4F+vOlGQxsLh/+aMk905DbVNpL+Afhy5iBnF7rOW2eDa3xzj58BZtjxetb8JMCoWRVNXZwNlD1zFZkiyvqoVD1zFTuD3Wcls8m9vj2aZqe8ykw0ePAvuOe71PP06SNE1mUih8C9g/yX5JdgBOBC4duCZJmlVmzOGjqno6yb8HrgTmAJ+rqrsGLms6bDOHwiaJ22Mtt8WzuT2ebUq2R6pqKtYrSdoKzaTDR5KkgRkKkqTGUBhIkn2TXJPk7iR3JXnX0DUNLcmcJLckuWzoWoaWZPckFyb5dpJ7khw6dE1DSnJG/39yZ5Lzk+w4dE3TJcnnkqxKcue4cXsmuSrJff3zHpPVnqEwnKeBd1fVy4FDgHfYrQfvAu4ZuogZ4pPAFVV1APBrzOLtkmQ+cBqwsKoOpLsQ5cRhq5pW5wJHrTNuCXB1Ve0PXN2/nhSGwkCqamVV3dwP/5Dun37+sFUNJ8k+wNHAOUPXMrQkzwdeA3wWoKqeqqrvD1vV4OYCOyWZC+wM/L+B65k2VXUd8L11Rh8DLOuHlwHHTlZ7hsIMkGQBcDBww7CVDOoTwHuBXwxdyAywH7Aa+Kv+cNo5SXYZuqihVNWjwEeBh4GVwJNV9dVhqxrcvKpa2Q8/BsybrBUbCgNLsitwEXB6Vf1g6HqGkOS3gVVVddPQtcwQc4FXAJ+uqoOBHzOJhwe2Nv3x8mPowvKXgV2S/N6wVc0c1X2vYNK+W2AoDCjJ9nSBcF5VXTx0PQM6HHhjkgfpesd9bZK/HrakQa0AVlTVmj3HC+lCYrZ6HfCdqlpdVT8HLgYOG7imoT2eZG+A/nnVZK3YUBhIktAdM76nqj42dD1Dqqr3VdU+VbWA7gTi16pq1n4SrKrHgEeSvKwfdSRw94AlDe1h4JAkO/f/N0cyi0+89y4FFvXDi4BLJmvFhsJwDgfeQvep+Nb+8Yahi9KM8U7gvCS3AwcBfzJwPYPp95guBG4G7qB735o1XV4kOR+4HnhZkhVJTgGWAq9Pch/dntTSSWvPbi4kSWu4pyBJagwFSVJjKEiSGkNBktQYCpKkxlCY5ZI8018Oe2eSLyfZfQvW9aMtWPa0vjfQ89YZf9D4S3WT/HGS/7C57UyFJO+fhHU8mGSvyahnhLaOtfNFrY+hoH+qqoP63ie/B7xjoDreDry+qt68zviDgJn+/Y2RQiHJnKkuZETHAoaCJmQoaLzr6XtqTfKqJNf3HbJ9Y823a5OcnOTiJFf0fbn/6borSbJXv+zRE0w7s98ruTPJ6f24zwAvBr6S5Ixx8+4A/GfghH5v5oR+0suTXJvkgSSnjZv/95Lc2M/7FxO9CfefyD/Uz7M8ySuSXJnk/yY5tZ8nST7S13jHmnaT7J3kunF7Vr+RZCld7523rruX0y/zoyT/LcltwKEj1viceZKcmuQj4+Y5Ocmn+uG/TXJTuvsNLF6n7f+a5LYk30wyL8lhwBuBj/Trf8k6bZ+b5DP9tvk/fb9UJFmQ5O+T3Nw/DtvANpnTr2fN9jujn/cPknyrr+eiJDv341/S13dHkg+O3+NM8p5+mduT/Kd1t5WmQFX5mMUP4Ef98xzgb4Cj+tfPA+b2w68DLuqHTwYeAJ4P7Ag8BOy7Zl10vTXeQPepf922fp3uG6m7ALsCdwEH99MeBPaaYJmTgU+Ne/3HwDeAXwL2Ar4LbA/8KvBlYPt+vv8BvHWC9T0I/Lt++OPA7cBuwBjweD/+3wBX9dtkHl03C3sD7wb+cNz22m38NlzP9i3g+H54vTWu+fnXN09f3/3j1vsV4NX98J79807AncALxrX9O/3wnwJ/1A+fCxy3nnrPBa6g+8C4P10/TDvSdVe9Yz/P/sDyfvg526T/PV81bp27988vGDfug8A7++HLgJP64VNZ+zf5m3TfXE5fz2XAa4b+n9nWH3PRbLdTklvp9hDuoXszhO5Nf1mS/eneXLYft8zVVfUkQJK7gRcBj/TzXA28o6q+PkFbrwa+VFU/7pe9GPgN4JZNrPnyqvoZ8LMkq+jeuI+kezP6VhLo3iDX10nYpf3zHcCu1d3P4odJfpbunMqrgfOr6hm6jse+DrwS+BbwuXQdGf5tVd06Qq3P0HV6yIg1TjhPVa3u94wOAe4DDgD+oV/mtCRv6of3pXvT/i7wFN0bKcBNwOtHqBfggqr6BXBfkgf6tr4DfCrJQf3P9Cv9vM/ZJv0yL07y58DlwJpurg9M8kFgd7oPBVf24w9l7f0A/iddN9nQhcJvsvbvY9f+Z7tuxJ9Dm8FQ0D9V1UH9rvyVdOcU/gz4L8A1VfWmdPd7uHbcMj8bN/wMa/+OnqZ78/lXwEShMFkmaj/Asqp63yYs/4t11vULNvA/UVXXJXkN3c2Azk3ysar6/Eba+mkfLoxY44bm+QJwPPBtunCtJEfQ7ckdWlU/SXIt3Sd7gJ9X/5GbZ/+eNmbdvm8KOAN4nO4ucNsBP4X1b5Mkv0b3d3BqX/Pb6PZCjq2q25KcDByxkToCfKiq/mLEujUJPKcgAKrqJ3S3PHx3urtbPR94tJ988qirofvnPyDJWRNM/3vg2HS9Xe4CvKkftyE/pDsksTFXA8cl+WfQ7mH7ohHrnqjOE/pj42N0d0G7sV/f41X1l3R3iFvTnfXP+0/Kk1Hjhub5Et19BU6iCwjofk9P9IFwAN2tXTdmY9v0d5Ns159veDFwb9/Oyn4P4i10h4qYaJuku4pqu6q6CPgj1m6n3YCV/bYaf0HBN+kO2cGzb7N5JfC2dPccIcn8NdtFU8dQUFNVt9AdYz+J7hj0h5LcwibsUfafik+i6/317etMu5nu0+KNdOcdzunb3JBr6E4sjz/RPFG7d9O9AX01Xc+iV9GdB9gcX6LbDrcBXwPeW1131kcAt/Xb5AS6+yhDd9z79olONG9qjRuap6qeoDvE96KqurFf5ApgbpJ76HrK/OYIP98XgPeku4jgJRNMf5jud/QV4NSq+induY1F6U6YH0B34x+YeJvMB67tD0v+NbBmr+c/0v3e/4Fub2eN04Ez+5/3pcCT/c/7VbrDSdcnuYOup9RRPiBoC9hLqqQmybnAZVV14TS2uTPdYcxKciLdSedjpqt9PZvnFCQN7dfpTmIH+D7dIUgNxD0FSVLjOQVJUmMoSJIaQ0GS1BgKkqTGUJAkNf8fo/5mDBpKXFUAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "evaluate_results(I, gold_para)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-Gfok653glf2"
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jjKilEmrgnHf"
      },
      "source": [
        "Using DPR Reader model itself for prediction (Incomplete)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 306,
          "referenced_widgets": [
            "08d0f64a6fce404196d5c72a12301bb7",
            "6d636b6192d34eefbd6129bdf81a15f9",
            "25143b52e5224f2189bb47281b7d9713",
            "f65c797093bc4647a60025ef356e28b5",
            "9c98372306024fb89bcb8b49b788ff75"
          ]
        },
        "id": "FVa1Yr8GUNB5",
        "outputId": "e3ace17a-d313-4ea0-80a7-809456734c19"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "08d0f64a6fce404196d5c72a12301bb7",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/28.0 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "6d636b6192d34eefbd6129bdf81a15f9",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/232k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "25143b52e5224f2189bb47281b7d9713",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/466k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "f65c797093bc4647a60025ef356e28b5",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/484 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization. \n",
            "The tokenizer class you load from this checkpoint is 'DPRQuestionEncoderTokenizer'. \n",
            "The class this function is called from is 'DPRReaderTokenizer'.\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "9c98372306024fb89bcb8b49b788ff75",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/438M [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Some weights of the model checkpoint at facebook/dpr-reader-single-nq-base were not used when initializing DPRReader: ['span_predictor.encoder.bert_model.pooler.dense.weight', 'span_predictor.encoder.bert_model.pooler.dense.bias']\n",
            "- This IS expected if you are initializing DPRReader from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing DPRReader from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
          ]
        }
      ],
      "source": [
        "from transformers import DPRReader, DPRReaderTokenizer\n",
        "import torch\n",
        "\n",
        "tokenizer = DPRReaderTokenizer.from_pretrained(\"facebook/dpr-reader-single-nq-base\")\n",
        "model = DPRReader.from_pretrained(\"facebook/dpr-reader-single-nq-base\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tI368LCuXoH4"
      },
      "outputs": [],
      "source": [
        "query_list = theme_wise_data['Beyoncé']['ques']\n",
        "para_list = theme_wise_data['Beyoncé']['para']"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "E7nE-e3lWFTt",
        "outputId": "ad287414-af9d-44c5-c20a-47194c1aa4f4"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'input_ids': tensor([[  101,  2043,  2106, 20773,  2681, 10461,  1005,  1055,  2775,  1998,\n",
              "          2468,  1037,  3948,  3220,  1029,   102,   100,   100,   100,   100,\n",
              "           100,   100,   100,   100,   100,   100,   102]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
              "         1, 1, 1]])}"
            ]
          },
          "execution_count": 50,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "encoded_inputs = tokenizer(\n",
        "    questions = query_list[0],\n",
        "    texts = para_list,\n",
        "    return_tensors=\"pt\",\n",
        ")\n",
        "encoded_inputs"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ybbvEfdVYeUb",
        "outputId": "e867cd37-6c85-4ade-e19b-68c5eabd7800"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "tensor([-9.0228])"
            ]
          },
          "execution_count": 51,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "with torch.no_grad():\n",
        "    outputs = model(**encoded_inputs)\n",
        "outputs.relevance_logits"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NilLXc2ZglK4"
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sAbNo9yu1zWf"
      },
      "source": [
        "#### 3. Using DensePhrases for phrase/passage retrieval (Incomplete)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aqsql7om36uM",
        "outputId": "0e5ac879-4b34-4508-d1fb-af80c2a48d14"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Number of examples = 75056\n",
            "\tAnswerable questions = 50126\n",
            "\tNon-Answerable questions = 24930\n",
            "\n",
            "Examples:\n",
            "Theme  |  Paragraph...  |  Question | Answer_possible | Answer_text | Answer_start\n",
            "Frédéric_Chopin  |  Some modern commenta...  |  Who said Chopin's works were modeled after Bach, Beethoven, Schubert and Field? | True | ['Richard Taruskin'] | [543]\n",
            "The_Legend_of_Zelda:_Twilight_Princess  |  Twilight Princess ta...  |  Who releases Bulbins from the Realm of Twilight? | False | [] | []\n"
          ]
        }
      ],
      "source": [
        "train_data = load_data()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wAf8wnq7KzIN"
      },
      "outputs": [],
      "source": [
        "!git clone https://www.github.com/nvidia/apex.git\n",
        "%cd apex\n",
        "!git checkout remotes/origin/22.04-dev\n",
        "!python setup.py install\n",
        "%cd .."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pSVnflcj3Vim"
      },
      "outputs": [],
      "source": [
        "!git clone -b v1.0.0 https://github.com/princeton-nlp/DensePhrases.git\n",
        "%cd DensePhrases\n",
        "!pip install -r requirements.txt\n",
        "!python setup.py develop"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "v7L16kfZGRGA"
      },
      "outputs": [],
      "source": [
        "import json\n",
        "\n",
        "# Data to be written\n",
        "articles = {\"data\": []}\n",
        "for theme in theme_wise_data:\n",
        "    content = {\n",
        "        \"title\": theme,\n",
        "        \"paragraphs\": []\n",
        "    }\n",
        "    for para in theme_wise_data[theme][\"paragraphs\"]:\n",
        "        content[\"paragraphs\"].append(para)\n",
        "    articles[\"data\"].append(content)\n",
        "\n",
        "# Serializing json\n",
        "json_object = json.dumps(articles, indent=4)\n",
        "\n",
        "# Writing to articles.json\n",
        "with open(\"articles.json\", \"w\") as outfile:\n",
        "    outfile.write(json_object)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GOq944s6IKEn"
      },
      "outputs": [],
      "source": [
        "questions = {\n",
        "    \"data\": [\n",
        "        {\n",
        "            \"id\": \"1\",\n",
        "            \"question\": \"Who said Chopin's works were modeled after Bach, Beethoven, Schubert and Field?\",\n",
        "            \"answers\": [\"Richard Taruskin\", \"Country singer Kevin Skinner\"]\n",
        "        },\n",
        "        {\n",
        "            \"id\": \"2\",\n",
        "            \"question\": \"What role did Beyoncé have in Destiny's Child?\",\n",
        "            \"answers\": [\"lead singer\"]\n",
        "        },\n",
        "        {\n",
        "            \"id\": \"3\",\n",
        "            \"question\": \"What percentage of Dhulbahante men married women of the Majerteen or Ogaden?\",\n",
        "            \"answers\": [\"4.3%\"]\n",
        "        }\n",
        "    ]\n",
        "}\n",
        "\n",
        "# Serializing json\n",
        "json_object = json.dumps(questions, indent=4)\n",
        "\n",
        "# Writing to articles.json\n",
        "with open(\"questions.json\", \"w\") as outfile:\n",
        "    outfile.write(json_object)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TJVqlZsATFNE"
      },
      "outputs": [],
      "source": [
        "!wget https://nlp.cs.princeton.edu/projects/densephrases/outputs.tar.gz\n",
        "!tar -xvf '/content/DensePhrases/outputs.tar.gz'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pXZ2tGszIuOZ",
        "outputId": "2f275d5e-ce27-4a2f-cc53-3d9d5e570fb3"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "WARNING:densephrases.options:WARNING - You've set a doc stride which may be superior to the document length in some examples.\n",
            "WARNING:__main__:Process rank: -1, device: cuda, n_gpu: 1, distributed training: False, 16-bits training: True\n",
            "Traceback (most recent call last):\n",
            "  File \"generate_phrase_vecs.py\", line 231, in <module>\n",
            "    main()\n",
            "  File \"generate_phrase_vecs.py\", line 227, in main\n",
            "    dump_phrases(args, model, tokenizer, filter_only=args.filter_only)\n",
            "  File \"generate_phrase_vecs.py\", line 79, in dump_phrases\n",
            "    dataset, examples, features = load_and_cache_examples(\n",
            "  File \"/content/DensePhrases/densephrases/utils/squad_utils.py\", line 1245, in load_and_cache_examples\n",
            "    examples = processor.get_dev_examples(args.data_dir, filename=args.predict_file, draft=args.draft,\n",
            "  File \"/content/DensePhrases/densephrases/utils/squad_utils.py\", line 809, in get_dev_examples\n",
            "    return self._create_examples(input_data, \"dev\", draft, context_only=context_only, args=args)\n",
            "  File \"/content/DensePhrases/densephrases/utils/squad_utils.py\", line 822, in _create_examples\n",
            "    truecase = TrueCaser(os.path.join(os.environ['DATA_DIR'], args.truecase_path))\n",
            "  File \"/content/DensePhrases/densephrases/utils/squad_utils.py\", line 1460, in __init__\n",
            "    with open(dist_file_path, \"rb\") as distributions_file:\n",
            "FileNotFoundError: [Errno 2] No such file or directory: '/content/DensePhrases/examples/create-custom-index/truecase/english_with_questions.dist'\n"
          ]
        }
      ],
      "source": [
        "!python generate_phrase_vecs.py \\\n",
        "    --model_type bert \\\n",
        "    --pretrained_name_or_path SpanBERT/spanbert-base-cased \\\n",
        "    --data_dir ./ \\\n",
        "    --cache_dir ./cache \\\n",
        "    --predict_file examples/create-custom-index/articles.json \\\n",
        "    --do_dump \\\n",
        "    --max_seq_length 512 \\\n",
        "    --doc_stride 500 \\\n",
        "    --fp16 \\\n",
        "    --filter_threshold -2.0 \\\n",
        "    --append_title \\\n",
        "    --load_dir ./outputs/densephrases-multi \\\n",
        "    --output_dir ./outputs/densephrases-multi_sample"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QH8Gef3GX3qN"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "os.environ['DATA_DIR'] = \"/content/DensePhrases\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hw-0oJ_lDfc5"
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KM5jpE-4h4kn"
      },
      "source": [
        "## Task 2: Domain-Specific Question Answering"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vbESJ9kgIlrC"
      },
      "source": [
        "### Testing Pretrained-Models using *transformers* library"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CX6MgopNJ_BD"
      },
      "source": [
        "#### Load vanilla transformer model for Question-Answering"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_b5tB9zpIk1w"
      },
      "outputs": [],
      "source": [
        "# # Print execution time\n",
        "# !pip install ipython-autotime\n",
        "# %load_ext autotime"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "background_save": true
        },
        "id": "sbTrvLUNJsb-",
        "outputId": "fdd9467b-b533-4790-bf7f-5ed6e0159b87"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: transformers in /usr/local/lib/python3.8/dist-packages (4.25.1)\n",
            "Requirement already satisfied: sentencepiece in /usr/local/lib/python3.8/dist-packages (0.1.97)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.8/dist-packages (from transformers) (2022.6.2)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.8/dist-packages (from transformers) (2.23.0)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.8/dist-packages (from transformers) (21.3)\n",
            "Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /usr/local/lib/python3.8/dist-packages (from transformers) (0.13.2)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.8/dist-packages (from transformers) (3.8.2)\n",
            "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.8/dist-packages (from transformers) (4.64.1)\n",
            "Requirement already satisfied: huggingface-hub<1.0,>=0.10.0 in /usr/local/lib/python3.8/dist-packages (from transformers) (0.11.1)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.8/dist-packages (from transformers) (1.21.6)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.8/dist-packages (from transformers) (6.0)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.8/dist-packages (from huggingface-hub<1.0,>=0.10.0->transformers) (4.4.0)\n",
            "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging>=20.0->transformers) (3.0.9)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests->transformers) (1.25.11)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from requests->transformers) (3.0.4)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests->transformers) (2.10)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests->transformers) (2022.12.7)\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: optimum[onnxruntime] in /usr/local/lib/python3.8/dist-packages (1.6.1)\n",
            "Requirement already satisfied: torch>=1.9 in /usr/local/lib/python3.8/dist-packages (from optimum[onnxruntime]) (1.13.0+cu116)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.8/dist-packages (from optimum[onnxruntime]) (1.7.1)\n",
            "Requirement already satisfied: numpy<1.24.0 in /usr/local/lib/python3.8/dist-packages (from optimum[onnxruntime]) (1.21.6)\n",
            "Requirement already satisfied: transformers[sentencepiece]>=4.20.1 in /usr/local/lib/python3.8/dist-packages (from optimum[onnxruntime]) (4.25.1)\n",
            "Requirement already satisfied: huggingface-hub>=0.8.0 in /usr/local/lib/python3.8/dist-packages (from optimum[onnxruntime]) (0.11.1)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from optimum[onnxruntime]) (21.3)\n",
            "Requirement already satisfied: coloredlogs in /usr/local/lib/python3.8/dist-packages (from optimum[onnxruntime]) (15.0.1)\n",
            "Requirement already satisfied: onnxruntime>=1.9.0 in /usr/local/lib/python3.8/dist-packages (from optimum[onnxruntime]) (1.13.1)\n",
            "Requirement already satisfied: protobuf==3.20.1 in /usr/local/lib/python3.8/dist-packages (from optimum[onnxruntime]) (3.20.1)\n",
            "Requirement already satisfied: evaluate in /usr/local/lib/python3.8/dist-packages (from optimum[onnxruntime]) (0.4.0)\n",
            "Requirement already satisfied: onnx in /usr/local/lib/python3.8/dist-packages (from optimum[onnxruntime]) (1.12.0)\n",
            "Requirement already satisfied: datasets>=1.2.1 in /usr/local/lib/python3.8/dist-packages (from optimum[onnxruntime]) (2.8.0)\n",
            "Requirement already satisfied: aiohttp in /usr/local/lib/python3.8/dist-packages (from datasets>=1.2.1->optimum[onnxruntime]) (3.8.3)\n",
            "Requirement already satisfied: responses<0.19 in /usr/local/lib/python3.8/dist-packages (from datasets>=1.2.1->optimum[onnxruntime]) (0.18.0)\n",
            "Requirement already satisfied: requests>=2.19.0 in /usr/local/lib/python3.8/dist-packages (from datasets>=1.2.1->optimum[onnxruntime]) (2.23.0)\n",
            "Requirement already satisfied: pyarrow>=6.0.0 in /usr/local/lib/python3.8/dist-packages (from datasets>=1.2.1->optimum[onnxruntime]) (9.0.0)\n",
            "Requirement already satisfied: tqdm>=4.62.1 in /usr/local/lib/python3.8/dist-packages (from datasets>=1.2.1->optimum[onnxruntime]) (4.64.1)\n",
            "Requirement already satisfied: multiprocess in /usr/local/lib/python3.8/dist-packages (from datasets>=1.2.1->optimum[onnxruntime]) (0.70.14)\n",
            "Requirement already satisfied: fsspec[http]>=2021.11.1 in /usr/local/lib/python3.8/dist-packages (from datasets>=1.2.1->optimum[onnxruntime]) (2022.11.0)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.8/dist-packages (from datasets>=1.2.1->optimum[onnxruntime]) (1.3.5)\n",
            "Requirement already satisfied: dill<0.3.7 in /usr/local/lib/python3.8/dist-packages (from datasets>=1.2.1->optimum[onnxruntime]) (0.3.6)\n",
            "Requirement already satisfied: xxhash in /usr/local/lib/python3.8/dist-packages (from datasets>=1.2.1->optimum[onnxruntime]) (3.2.0)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.8/dist-packages (from datasets>=1.2.1->optimum[onnxruntime]) (6.0)\n",
            "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets>=1.2.1->optimum[onnxruntime]) (1.8.2)\n",
            "Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets>=1.2.1->optimum[onnxruntime]) (2.1.1)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets>=1.2.1->optimum[onnxruntime]) (22.1.0)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets>=1.2.1->optimum[onnxruntime]) (1.3.3)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets>=1.2.1->optimum[onnxruntime]) (6.0.3)\n",
            "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets>=1.2.1->optimum[onnxruntime]) (4.0.2)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets>=1.2.1->optimum[onnxruntime]) (1.3.1)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.8/dist-packages (from huggingface-hub>=0.8.0->optimum[onnxruntime]) (4.4.0)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.8/dist-packages (from huggingface-hub>=0.8.0->optimum[onnxruntime]) (3.8.2)\n",
            "Requirement already satisfied: flatbuffers in /usr/local/lib/python3.8/dist-packages (from onnxruntime>=1.9.0->optimum[onnxruntime]) (1.12)\n",
            "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->optimum[onnxruntime]) (3.0.9)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests>=2.19.0->datasets>=1.2.1->optimum[onnxruntime]) (2.10)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from requests>=2.19.0->datasets>=1.2.1->optimum[onnxruntime]) (3.0.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests>=2.19.0->datasets>=1.2.1->optimum[onnxruntime]) (2022.12.7)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests>=2.19.0->datasets>=1.2.1->optimum[onnxruntime]) (1.25.11)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.8/dist-packages (from transformers[sentencepiece]>=4.20.1->optimum[onnxruntime]) (2022.6.2)\n",
            "Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /usr/local/lib/python3.8/dist-packages (from transformers[sentencepiece]>=4.20.1->optimum[onnxruntime]) (0.13.2)\n",
            "Requirement already satisfied: sentencepiece!=0.1.92,>=0.1.91 in /usr/local/lib/python3.8/dist-packages (from transformers[sentencepiece]>=4.20.1->optimum[onnxruntime]) (0.1.97)\n",
            "Requirement already satisfied: humanfriendly>=9.1 in /usr/local/lib/python3.8/dist-packages (from coloredlogs->optimum[onnxruntime]) (10.0)\n",
            "Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas->datasets>=1.2.1->optimum[onnxruntime]) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas->datasets>=1.2.1->optimum[onnxruntime]) (2022.6)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.8/dist-packages (from python-dateutil>=2.7.3->pandas->datasets>=1.2.1->optimum[onnxruntime]) (1.15.0)\n",
            "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.8/dist-packages (from sympy->optimum[onnxruntime]) (1.2.1)\n"
          ]
        }
      ],
      "source": [
        "# Install required libraries\n",
        "!pip install transformers sentencepiece\n",
        "!pip install optimum[onnxruntime]\n",
        "# !pip install \"optimum[onnxruntime-gpu]\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "background_save": true
        },
        "id": "ifvWLlMun86m"
      },
      "outputs": [],
      "source": [
        "# load the model\n",
        "\n",
        "from pathlib import Path\n",
        "from transformers import AutoTokenizer, pipeline\n",
        "from optimum.onnxruntime import ORTModelForQuestionAnswering\n",
        "\n",
        "def load_qna_pipeline(model_id):\n",
        "    # load vanilla transformers and convert to onnx; return the qna pipeline\n",
        "    task = \"question-answering\"\n",
        "    model = ORTModelForQuestionAnswering.from_pretrained(model_id, from_transformers=True)\n",
        "    tokenizer = AutoTokenizer.from_pretrained(model_id)\n",
        "    optimum_qa = pipeline(task, model=model, tokenizer=tokenizer, handle_impossible_answer=True)\n",
        "    return optimum_qa\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o_4cG6nIlVIA"
      },
      "source": [
        "#### Load and optimize model using ONNX"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3HfUcXfoPeov",
        "outputId": "6628cd91-2bbf-44d0-ae58-2bed3dd413a8"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting transformers\n",
            "  Downloading transformers-4.25.1-py3-none-any.whl (5.8 MB)\n",
            "\u001b[K     |████████████████████████████████| 5.8 MB 12.9 MB/s \n",
            "\u001b[?25hCollecting sentencepiece\n",
            "  Downloading sentencepiece-0.1.97-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
            "\u001b[K     |████████████████████████████████| 1.3 MB 41.5 MB/s \n",
            "\u001b[?25hRequirement already satisfied: filelock in /usr/local/lib/python3.8/dist-packages (from transformers) (3.8.2)\n",
            "Collecting tokenizers!=0.11.3,<0.14,>=0.11.1\n",
            "  Downloading tokenizers-0.13.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.6 MB)\n",
            "\u001b[K     |████████████████████████████████| 7.6 MB 52.0 MB/s \n",
            "\u001b[?25hRequirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.8/dist-packages (from transformers) (21.3)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.8/dist-packages (from transformers) (2022.6.2)\n",
            "Collecting huggingface-hub<1.0,>=0.10.0\n",
            "  Downloading huggingface_hub-0.11.1-py3-none-any.whl (182 kB)\n",
            "\u001b[K     |████████████████████████████████| 182 kB 107.0 MB/s \n",
            "\u001b[?25hRequirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.8/dist-packages (from transformers) (6.0)\n",
            "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.8/dist-packages (from transformers) (4.64.1)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.8/dist-packages (from transformers) (2.23.0)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.8/dist-packages (from transformers) (1.21.6)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.8/dist-packages (from huggingface-hub<1.0,>=0.10.0->transformers) (4.4.0)\n",
            "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging>=20.0->transformers) (3.0.9)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests->transformers) (1.24.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests->transformers) (2022.12.7)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from requests->transformers) (3.0.4)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests->transformers) (2.10)\n",
            "Installing collected packages: tokenizers, huggingface-hub, transformers, sentencepiece\n",
            "Successfully installed huggingface-hub-0.11.1 sentencepiece-0.1.97 tokenizers-0.13.2 transformers-4.25.1\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting optimum[onnxruntime]\n",
            "  Downloading optimum-1.6.1-py3-none-any.whl (222 kB)\n",
            "\u001b[K     |████████████████████████████████| 222 kB 12.0 MB/s \n",
            "\u001b[?25hRequirement already satisfied: numpy<1.24.0 in /usr/local/lib/python3.8/dist-packages (from optimum[onnxruntime]) (1.21.6)\n",
            "Collecting coloredlogs\n",
            "  Downloading coloredlogs-15.0.1-py2.py3-none-any.whl (46 kB)\n",
            "\u001b[K     |████████████████████████████████| 46 kB 4.1 MB/s \n",
            "\u001b[?25hRequirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from optimum[onnxruntime]) (21.3)\n",
            "Requirement already satisfied: torch>=1.9 in /usr/local/lib/python3.8/dist-packages (from optimum[onnxruntime]) (1.13.0+cu116)\n",
            "Requirement already satisfied: huggingface-hub>=0.8.0 in /usr/local/lib/python3.8/dist-packages (from optimum[onnxruntime]) (0.11.1)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.8/dist-packages (from optimum[onnxruntime]) (1.7.1)\n",
            "Requirement already satisfied: transformers[sentencepiece]>=4.20.1 in /usr/local/lib/python3.8/dist-packages (from optimum[onnxruntime]) (4.25.1)\n",
            "Collecting evaluate\n",
            "  Downloading evaluate-0.4.0-py3-none-any.whl (81 kB)\n",
            "\u001b[K     |████████████████████████████████| 81 kB 10.7 MB/s \n",
            "\u001b[?25hCollecting onnx\n",
            "  Downloading onnx-1.13.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.5 MB)\n",
            "\u001b[K     |████████████████████████████████| 13.5 MB 66.5 MB/s \n",
            "\u001b[?25hCollecting onnxruntime>=1.9.0\n",
            "  Downloading onnxruntime-1.13.1-cp38-cp38-manylinux_2_27_x86_64.whl (4.5 MB)\n",
            "\u001b[K     |████████████████████████████████| 4.5 MB 59.2 MB/s \n",
            "\u001b[?25hCollecting datasets>=1.2.1\n",
            "  Downloading datasets-2.8.0-py3-none-any.whl (452 kB)\n",
            "\u001b[K     |████████████████████████████████| 452 kB 95.9 MB/s \n",
            "\u001b[?25hCollecting protobuf==3.20.1\n",
            "  Downloading protobuf-3.20.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.0 MB)\n",
            "\u001b[K     |████████████████████████████████| 1.0 MB 86.7 MB/s \n",
            "\u001b[?25hRequirement already satisfied: requests>=2.19.0 in /usr/local/lib/python3.8/dist-packages (from datasets>=1.2.1->optimum[onnxruntime]) (2.23.0)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.8/dist-packages (from datasets>=1.2.1->optimum[onnxruntime]) (1.3.5)\n",
            "Requirement already satisfied: fsspec[http]>=2021.11.1 in /usr/local/lib/python3.8/dist-packages (from datasets>=1.2.1->optimum[onnxruntime]) (2022.11.0)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.8/dist-packages (from datasets>=1.2.1->optimum[onnxruntime]) (6.0)\n",
            "Requirement already satisfied: dill<0.3.7 in /usr/local/lib/python3.8/dist-packages (from datasets>=1.2.1->optimum[onnxruntime]) (0.3.6)\n",
            "Requirement already satisfied: tqdm>=4.62.1 in /usr/local/lib/python3.8/dist-packages (from datasets>=1.2.1->optimum[onnxruntime]) (4.64.1)\n",
            "Requirement already satisfied: aiohttp in /usr/local/lib/python3.8/dist-packages (from datasets>=1.2.1->optimum[onnxruntime]) (3.8.3)\n",
            "Collecting responses<0.19\n",
            "  Downloading responses-0.18.0-py3-none-any.whl (38 kB)\n",
            "Collecting xxhash\n",
            "  Downloading xxhash-3.2.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (213 kB)\n",
            "\u001b[K     |████████████████████████████████| 213 kB 61.8 MB/s \n",
            "\u001b[?25hRequirement already satisfied: pyarrow>=6.0.0 in /usr/local/lib/python3.8/dist-packages (from datasets>=1.2.1->optimum[onnxruntime]) (9.0.0)\n",
            "Collecting multiprocess\n",
            "  Downloading multiprocess-0.70.14-py38-none-any.whl (132 kB)\n",
            "\u001b[K     |████████████████████████████████| 132 kB 71.9 MB/s \n",
            "\u001b[?25hRequirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets>=1.2.1->optimum[onnxruntime]) (6.0.3)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets>=1.2.1->optimum[onnxruntime]) (1.3.1)\n",
            "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets>=1.2.1->optimum[onnxruntime]) (4.0.2)\n",
            "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets>=1.2.1->optimum[onnxruntime]) (1.8.2)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets>=1.2.1->optimum[onnxruntime]) (1.3.3)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets>=1.2.1->optimum[onnxruntime]) (22.1.0)\n",
            "Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets>=1.2.1->optimum[onnxruntime]) (2.1.1)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.8/dist-packages (from huggingface-hub>=0.8.0->optimum[onnxruntime]) (3.8.2)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.8/dist-packages (from huggingface-hub>=0.8.0->optimum[onnxruntime]) (4.4.0)\n",
            "Requirement already satisfied: flatbuffers in /usr/local/lib/python3.8/dist-packages (from onnxruntime>=1.9.0->optimum[onnxruntime]) (1.12)\n",
            "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->optimum[onnxruntime]) (3.0.9)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests>=2.19.0->datasets>=1.2.1->optimum[onnxruntime]) (2022.12.7)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests>=2.19.0->datasets>=1.2.1->optimum[onnxruntime]) (2.10)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from requests>=2.19.0->datasets>=1.2.1->optimum[onnxruntime]) (3.0.4)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests>=2.19.0->datasets>=1.2.1->optimum[onnxruntime]) (1.24.3)\n",
            "Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1\n",
            "  Downloading urllib3-1.25.11-py2.py3-none-any.whl (127 kB)\n",
            "\u001b[K     |████████████████████████████████| 127 kB 119.8 MB/s \n",
            "\u001b[?25hRequirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /usr/local/lib/python3.8/dist-packages (from transformers[sentencepiece]>=4.20.1->optimum[onnxruntime]) (0.13.2)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.8/dist-packages (from transformers[sentencepiece]>=4.20.1->optimum[onnxruntime]) (2022.6.2)\n",
            "Requirement already satisfied: sentencepiece!=0.1.92,>=0.1.91 in /usr/local/lib/python3.8/dist-packages (from transformers[sentencepiece]>=4.20.1->optimum[onnxruntime]) (0.1.97)\n",
            "Collecting humanfriendly>=9.1\n",
            "  Downloading humanfriendly-10.0-py2.py3-none-any.whl (86 kB)\n",
            "\u001b[K     |████████████████████████████████| 86 kB 7.0 MB/s \n",
            "\u001b[?25hCollecting onnx\n",
            "  Downloading onnx-1.12.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.1 MB)\n",
            "\u001b[K     |████████████████████████████████| 13.1 MB 52.5 MB/s \n",
            "\u001b[?25hRequirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas->datasets>=1.2.1->optimum[onnxruntime]) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas->datasets>=1.2.1->optimum[onnxruntime]) (2022.6)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.8/dist-packages (from python-dateutil>=2.7.3->pandas->datasets>=1.2.1->optimum[onnxruntime]) (1.15.0)\n",
            "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.8/dist-packages (from sympy->optimum[onnxruntime]) (1.2.1)\n",
            "Installing collected packages: urllib3, xxhash, responses, protobuf, multiprocess, humanfriendly, datasets, coloredlogs, optimum, onnxruntime, onnx, evaluate\n",
            "  Attempting uninstall: urllib3\n",
            "    Found existing installation: urllib3 1.24.3\n",
            "    Uninstalling urllib3-1.24.3:\n",
            "      Successfully uninstalled urllib3-1.24.3\n",
            "  Attempting uninstall: protobuf\n",
            "    Found existing installation: protobuf 3.19.6\n",
            "    Uninstalling protobuf-3.19.6:\n",
            "      Successfully uninstalled protobuf-3.19.6\n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "tensorflow 2.9.2 requires protobuf<3.20,>=3.9.2, but you have protobuf 3.20.1 which is incompatible.\n",
            "tensorboard 2.9.1 requires protobuf<3.20,>=3.9.2, but you have protobuf 3.20.1 which is incompatible.\n",
            "googleapis-common-protos 1.57.0 requires protobuf!=3.20.0,!=3.20.1,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.20.1 which is incompatible.\n",
            "google-cloud-translate 3.8.4 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.20.1 which is incompatible.\n",
            "google-cloud-language 2.6.1 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.20.1 which is incompatible.\n",
            "google-cloud-firestore 2.7.2 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.20.1 which is incompatible.\n",
            "google-cloud-datastore 2.9.0 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.20.1 which is incompatible.\n",
            "google-cloud-bigquery 3.3.6 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.20.1 which is incompatible.\n",
            "google-cloud-bigquery-storage 2.16.2 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.20.1 which is incompatible.\u001b[0m\n",
            "Successfully installed coloredlogs-15.0.1 datasets-2.8.0 evaluate-0.4.0 humanfriendly-10.0 multiprocess-0.70.14 onnx-1.12.0 onnxruntime-1.13.1 optimum-1.6.1 protobuf-3.20.1 responses-0.18.0 urllib3-1.25.11 xxhash-3.2.0\n"
          ]
        },
        {
          "data": {
            "application/vnd.colab-display-data+json": {
              "pip_warning": {
                "packages": [
                  "google",
                  "urllib3"
                ]
              }
            }
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# Install required libraries\n",
        "!pip install transformers sentencepiece\n",
        "!pip install optimum[onnxruntime]\n",
        "# !pip install \"optimum[onnxruntime-gpu]\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8YXEXyUNv1XO"
      },
      "outputs": [],
      "source": [
        "from pathlib import Path\n",
        "from transformers import AutoTokenizer, pipeline\n",
        "from optimum.onnxruntime import ORTModelForQuestionAnswering\n",
        "\n",
        "model_id = \"deepset/minilm-uncased-squad2\"\n",
        "\n",
        "# load vanilla transformers and convert to onnx\n",
        "def load_and_save_vanilla_model(model_id, save_path):\n",
        "    task = \"question-answering\"\n",
        "    model = ORTModelForQuestionAnswering.from_pretrained(model_id, from_transformers=True)\n",
        "    tokenizer = AutoTokenizer.from_pretrained(model_id)\n",
        "    # save onnx checkpoint and tokenizer\n",
        "    model.save_pretrained(save_path)\n",
        "    tokenizer.save_pretrained(save_path)\n",
        "    return model, tokenizer"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6yfAMZ6a1S2z"
      },
      "outputs": [],
      "source": [
        "import shutil\n",
        "from optimum.onnxruntime import ORTOptimizer\n",
        "from optimum.onnxruntime.configuration import OptimizationConfig\n",
        "\n",
        "# create ORTOptimizer and define optimization configuration\n",
        "def optimize_model(model, save_dir):\n",
        "    optimizer = ORTOptimizer.from_pretrained(model)\n",
        "    optimization_config = OptimizationConfig(optimization_level=99) # enable all optimizations\n",
        "\n",
        "    # apply the optimization configuration to the model\n",
        "    optimizer.optimize(\n",
        "        save_dir=save_dir,\n",
        "        optimization_config=optimization_config,\n",
        "    )\n",
        "\n",
        "# load optimized model\n",
        "def load_optimized_model(path):\n",
        "    opt_model = ORTModelForQuestionAnswering.from_pretrained(path, file_name='model_optimized.onnx')\n",
        "    return opt_model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mvoxcQCKxh8M"
      },
      "outputs": [],
      "source": [
        "from optimum.onnxruntime import ORTQuantizer\n",
        "from optimum.onnxruntime.configuration import AutoQuantizationConfig\n",
        "\n",
        "# create ORTQuantizer and define quantization configuration\n",
        "def quantize_model(model, save_dir, onnx_path):\n",
        "    quantizer = ORTQuantizer.from_pretrained(model)\n",
        "    qconfig = AutoQuantizationConfig.avx512_vnni(is_static=False, per_channel=True)\n",
        "\n",
        "    # apply the quantization configuration to the model\n",
        "    quantizer.quantize(\n",
        "        save_dir=save_dir,\n",
        "        quantization_config=qconfig,\n",
        "    )\n",
        "\n",
        "    # Also copy config file\n",
        "    src_path = onnx_path / 'config.json'\n",
        "    dst_path = save_dir / 'config.json'\n",
        "    shutil.copy(src_path, dst_path)\n",
        "\n",
        "\n",
        "# load quantized model\n",
        "def load_quantized_model(path):\n",
        "    task = \"question-answering\"\n",
        "    quantized_model = ORTModelForQuestionAnswering.from_pretrained(path, file_name=\"model_optimized_quantized.onnx\")\n",
        "    return quantized_model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wXcohgZTys4B"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "# compare and print model size\n",
        "def print_model_size(onnx_path, quantized_model_path):\n",
        "    size = os.path.getsize(onnx_path / \"model.onnx\")/(1024*1024)\n",
        "    print(f\"Vanilla Onnx Model file size: {size:.2f} MB\")\n",
        "    size = os.path.getsize(quantized_model_path / \"model_optimized_quantized.onnx\")/(1024*1024)\n",
        "    print(f\"Quantized Onnx Model file size: {size:.2f} MB\\n\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "33ozPh60pQCm"
      },
      "outputs": [],
      "source": [
        "# load and optimize model then return the question-answering pipeline\n",
        "def load_optimized_model_pipeline(model_id):\n",
        "    # load vanilla model\n",
        "    onnx_path = Path(\"onnx\")\n",
        "    opt_model_path = onnx_path / 'model-optimized.onnx'\n",
        "    quant_model_path = onnx_path / \"model-quantized.onnx\"\n",
        "    task = \"question-answering\"\n",
        "    model, tokenizer = load_and_save_vanilla_model(model_id, onnx_path)\n",
        "\n",
        "    # optimize model\n",
        "    optimize_model(model, opt_model_path)\n",
        "    model = load_optimized_model(opt_model_path)\n",
        "    quantize_model(model, quant_model_path, onnx_path)\n",
        "    model = load_quantized_model(quant_model_path)\n",
        "\n",
        "    print(\"Model optimized successfully.\")\n",
        "    print_model_size(onnx_path, quant_model_path)\n",
        "\n",
        "    # load pipeline\n",
        "    quantized_optimum_qa = pipeline(task, model=model, tokenizer=tokenizer, handle_impossible_answer=True)\n",
        "    return quantized_optimum_qa"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZEZlBhXBC1LX"
      },
      "source": [
        "#### Evaluating"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GI9fmizthXlR"
      },
      "outputs": [],
      "source": [
        "# Evaluation Metric implementation\n",
        "\n",
        "import string, re, json, ast\n",
        "\n",
        "def normalize_text(s):\n",
        "    \"\"\"Removing articles and punctuation, and standardizing whitespace are all typical text processing steps.\"\"\"\n",
        "\n",
        "    def remove_articles(text):\n",
        "        regex = re.compile(r\"\\b(a|an|the)\\b\", re.UNICODE)\n",
        "        return re.sub(regex, \" \", text)\n",
        "\n",
        "    def white_space_fix(text):\n",
        "        return \" \".join(text.split())\n",
        "\n",
        "    def remove_punc(text):\n",
        "        exclude = set(string.punctuation)\n",
        "        return \"\".join(ch for ch in text if ch not in exclude)\n",
        "\n",
        "    def lower(text):\n",
        "        return text.lower()\n",
        "\n",
        "    return white_space_fix(remove_articles(remove_punc(lower(s))))\n",
        "\n",
        "\n",
        "def compute_exact_match(prediction, truth):\n",
        "    return int(normalize_text(prediction) == normalize_text(truth))\n",
        "\n",
        "\n",
        "def compute_f1(prediction, truth):\n",
        "    pred_tokens = normalize_text(prediction).split()\n",
        "    truth_tokens = normalize_text(truth).split()\n",
        "\n",
        "    # if either the prediction or the truth is no-answer then f1 = 1 if they agree, 0 otherwise\n",
        "    if len(pred_tokens) == 0 or len(truth_tokens) == 0:\n",
        "        return int(pred_tokens == truth_tokens)\n",
        "\n",
        "    common_tokens = set(pred_tokens) & set(truth_tokens)\n",
        "\n",
        "    # if there are no common tokens then f1 = 0\n",
        "    if len(common_tokens) == 0:\n",
        "        return 0\n",
        "\n",
        "    prec = len(common_tokens) / len(pred_tokens)\n",
        "    rec = len(common_tokens) / len(truth_tokens)\n",
        "\n",
        "    return 2 * (prec * rec) / (prec + rec)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hlEl-_060hD4"
      },
      "outputs": [],
      "source": [
        "# generate predictions from the model\n",
        "\n",
        "import time\n",
        "\n",
        "def predict(model, question, context, answers):\n",
        "    y = model(question=question, context=context)\n",
        "    gold = ast.literal_eval(answers)\n",
        "    if not gold:\n",
        "        gold = [\"\"]\n",
        "    pred = {\n",
        "        'question': question,\n",
        "        'gold_answers': gold,\n",
        "        'prediction': y\n",
        "    }\n",
        "    return pred\n",
        "\n",
        "def generate_predictions(model, train_data, eval_count = 100):\n",
        "    ans, unans = 0, 0\n",
        "    preds = []\n",
        "    start_time = time.time()\n",
        "    for x in train_data[1:]:\n",
        "        if x[4] == 'True':   # answerable\n",
        "            if ans >= eval_count / 2:\n",
        "                continue\n",
        "            preds.append(predict(model, x[3], x[2], x[5]))\n",
        "            ans += 1\n",
        "        else:                # unanswerable\n",
        "            if unans >= eval_count / 2:\n",
        "                continue\n",
        "            preds.append(predict(model, x[3], x[2], x[5]))\n",
        "            unans += 1\n",
        "\n",
        "        print(f\"\\r{round((ans + unans)*100 / eval_count, 2)}% completed\", end='')\n",
        "        if ans + unans >= eval_count:\n",
        "            print()\n",
        "            break\n",
        "    print(f'Average Inference Time: {(time.time() - start_time)*1000 / eval_count} ms')\n",
        "    return preds\n",
        "\n",
        "def generate_predictions_all(model, train_data):\n",
        "    preds = []\n",
        "    total = len(train_data) - 1\n",
        "    start_time = time.time()\n",
        "    for x in train_data[1:]:\n",
        "        preds.append(predict(model, x[3], x[2], x[5]))\n",
        "        print(f\"\\r{round(len(preds)*100 / total, 2)}% completed\", end='')\n",
        "\n",
        "    print(f'\\nAverage Inference Time: {(time.time() - start_time)*1000 / total} ms')\n",
        "    return preds"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MafV6F1JnSgo"
      },
      "outputs": [],
      "source": [
        "# Calculate and print the metrics\n",
        "def evaluate(preds_with_gold):\n",
        "    f1_score = 0.\n",
        "    em_score = 0.\n",
        "    for y in preds_with_gold:\n",
        "        pred = y['prediction']['answer']\n",
        "        em = max([compute_exact_match(pred, gold) for gold in y['gold_answers']])\n",
        "        f1 = max([compute_f1(pred, gold) for gold in y['gold_answers']])\n",
        "        em_score += em\n",
        "        f1_score += f1\n",
        "\n",
        "    em_score = em_score / (len(preds_with_gold))\n",
        "    f1_score = f1_score / (len(preds_with_gold))\n",
        "    print(f'EM Score: {em_score}\\nF1 Score: {f1_score}')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I9ggAUvU7Pqg"
      },
      "source": [
        "#### Putting it all together"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YXmWUGdCDr5j",
        "outputId": "dd9ac3e8-b338-4793-f922-ab89085f83ab"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Number of examples = 75056\n",
            "\tAnswerable questions = 50126\n",
            "\tNon-Answerable questions = 24930\n",
            "\n",
            "Examples:\n",
            " | Theme  |  Paragraph...  |  Question | Answer_possible | Answer_text | Answer_start\n",
            "1430 | Frédéric_Chopin  |  Some modern commenta...  |  Who said Chopin's works were modeled after Bach, Beethoven, Schubert and Field? | True | ['Richard Taruskin'] | [543]\n",
            "2196 | The_Legend_of_Zelda:_Twilight_Princess  |  Twilight Princess ta...  |  Who releases Bulbins from the Realm of Twilight? | False | [] | []\n"
          ]
        }
      ],
      "source": [
        "train_data = load_data()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "background_save": true
        },
        "id": "I-7VsL_9BKl0"
      },
      "outputs": [],
      "source": [
        "model_id = \"deepset/tinybert-6l-768d-squad2\"\n",
        "# optimum_qa = load_qna_pipeline(model_id)\n",
        "optimum_qa = load_optimized_model_pipeline(model_id)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uXaObcw_7-Oh",
        "outputId": "024b8117-7866-4ad1-d646-6f76342c098f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "100.0% completed\n",
            "Average Inference Time: 234.7511488199234 ms\n",
            "EM Score: 0.38\n",
            "F1 Score: 0.38842628205128205\n"
          ]
        }
      ],
      "source": [
        "num_examples = 400\n",
        "preds = generate_predictions(optimum_qa, train_data, num_examples)\n",
        "# preds = generate_predictions_all(quantized_optimum_qa, train_data)\n",
        "evaluate(preds)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uXPtOBrsQnXs",
        "outputId": "001046e6-701c-4e98-8d95-addf3a890c74"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[{'question': \"When did Beyonce leave Destiny's Child and become a solo singer?\",\n",
              "  'gold_answers': ['2003'],\n",
              "  'prediction': {'score': 4.3079005990875885e-05,\n",
              "   'start': 8,\n",
              "   'end': 15,\n",
              "   'answer': 'Giselle'}},\n",
              " {'question': 'What album made her a worldwide known artist?',\n",
              "  'gold_answers': ['Dangerously in Love'],\n",
              "  'prediction': {'score': 4.418786193127744e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': \"Who managed the Destiny's Child group?\",\n",
              "  'gold_answers': ['Mathew Knowles'],\n",
              "  'prediction': {'score': 4.4217402319191024e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'When did Beyoncé rise to fame?',\n",
              "  'gold_answers': ['late 1990s'],\n",
              "  'prediction': {'score': 4.316834747442044e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': \"What role did Beyoncé have in Destiny's Child?\",\n",
              "  'gold_answers': ['lead singer'],\n",
              "  'prediction': {'score': 4.403022103360854e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'When did Beyoncé release Dangerously in Love?',\n",
              "  'gold_answers': ['2003'],\n",
              "  'prediction': {'score': 4.569309385260567e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'How many Grammy awards did Beyoncé win for her first solo album?',\n",
              "  'gold_answers': ['five'],\n",
              "  'prediction': {'score': 4.51989471912384e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': \"What was the name of Beyoncé's first solo album?\",\n",
              "  'gold_answers': ['Dangerously in Love'],\n",
              "  'prediction': {'score': 4.774379704031162e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'After her second solo album, what other entertainment venture did Beyonce explore?',\n",
              "  'gold_answers': ['acting'],\n",
              "  'prediction': {'score': 2.164296711271163e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Which album was darker in tone from her previous work?',\n",
              "  'gold_answers': ['Beyoncé'],\n",
              "  'prediction': {'score': 2.103247243212536e-05,\n",
              "   'start': 283,\n",
              "   'end': 291,\n",
              "   'answer': 'starring'}},\n",
              " {'question': 'After what movie portraying Etta James, did Beyonce create Sasha Fierce?',\n",
              "  'gold_answers': ['Cadillac Records'],\n",
              "  'prediction': {'score': 2.17508331843419e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': \"When did Destiny's Child end their group act?\",\n",
              "  'gold_answers': ['June 2005'],\n",
              "  'prediction': {'score': 2.2174715923028998e-05,\n",
              "   'start': 0,\n",
              "   'end': 9,\n",
              "   'answer': 'Following'}},\n",
              " {'question': \"What was the name of Beyoncé's second solo album?\",\n",
              "  'gold_answers': [\"B'Day\"],\n",
              "  'prediction': {'score': 2.2840571546112187e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': \"What was Beyoncé's first acting job, in 2006?\",\n",
              "  'gold_answers': ['Dreamgirls'],\n",
              "  'prediction': {'score': 2.3048161892802455e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Who is Beyoncé married to?',\n",
              "  'gold_answers': ['Jay Z'],\n",
              "  'prediction': {'score': 2.1085399566800334e-05,\n",
              "   'start': 283,\n",
              "   'end': 345,\n",
              "   'answer': 'starring roles in The Pink Panther (2006) and Obsessed (2009).'}},\n",
              " {'question': \"What is the name of Beyoncé's alter-ego?\",\n",
              "  'gold_answers': ['Sasha Fierce'],\n",
              "  'prediction': {'score': 2.2593520043301396e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'In her music, what are some recurring elements in them?',\n",
              "  'gold_answers': ['love, relationships, and monogamy'],\n",
              "  'prediction': {'score': 2.6146613890887238e-05,\n",
              "   'start': 71,\n",
              "   'end': 100,\n",
              "   'answer': 'often characterized by themes'}},\n",
              " {'question': 'Time magazine named her one of the most 100 what people of the century?',\n",
              "  'gold_answers': ['influential'],\n",
              "  'prediction': {'score': 2.6287458240403794e-05,\n",
              "   'start': 616,\n",
              "   'end': 681,\n",
              "   'answer': 'history. The Recording Industry Association of America recognized'}},\n",
              " {'question': 'In which decade did the Recording Industry Association of America recognize Beyonce as the The Top Certified Artist?',\n",
              "  'gold_answers': ['2000s'],\n",
              "  'prediction': {'score': 2.6034453185275197e-05,\n",
              "   'start': 671,\n",
              "   'end': 681,\n",
              "   'answer': 'recognized'}},\n",
              " {'question': 'What magazine rated Beyonce as the most powerful female musician in 2015?',\n",
              "  'gold_answers': ['Forbes'],\n",
              "  'prediction': {'score': 2.6236373741994612e-05,\n",
              "   'start': 905,\n",
              "   'end': 911,\n",
              "   'answer': 'listed'}},\n",
              " {'question': 'In which years did Time rate Beyonce in the 100 most influential people in the world?',\n",
              "  'gold_answers': ['2013 and 2014'],\n",
              "  'prediction': {'score': 2.6155306841246784e-05,\n",
              "   'start': 616,\n",
              "   'end': 623,\n",
              "   'answer': 'history'}},\n",
              " {'question': 'How many records has Beyonce sold in her 19 year career?',\n",
              "  'gold_answers': ['118 million'],\n",
              "  'prediction': {'score': 2.6207499104202725e-05,\n",
              "   'start': 375,\n",
              "   'end': 404,\n",
              "   'answer': 'she has sold over 118 million'}},\n",
              " {'question': \"How many records did Beyoncé sell as part of Destiny's Child?\",\n",
              "  'gold_answers': ['60 million'],\n",
              "  'prediction': {'score': 2.536849569878541e-05,\n",
              "   'start': 1024,\n",
              "   'end': 1062,\n",
              "   'answer': 'most powerful female musician of 2015.'}},\n",
              " {'question': \"After leaving Destiny's Child, how many records did Beyoncé release under her own name?\",\n",
              "  'gold_answers': ['118 million'],\n",
              "  'prediction': {'score': 2.5695471776998602e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'How many Grammy awards has Beyoncé won?',\n",
              "  'gold_answers': ['20'],\n",
              "  'prediction': {'score': 2.5785091565921903e-05,\n",
              "   'start': 671,\n",
              "   'end': 681,\n",
              "   'answer': 'recognized'}},\n",
              " {'question': \"What race was Beyonce's father?\",\n",
              "  'gold_answers': ['African-American'],\n",
              "  'prediction': {'score': 5.9782185417134315e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': \"Beyonce's childhood home believed in what religion?\",\n",
              "  'gold_answers': ['Methodist'],\n",
              "  'prediction': {'score': 5.731260898755863e-05,\n",
              "   'start': 193,\n",
              "   'end': 223,\n",
              "   'answer': \"tribute to her mother's maiden\"}},\n",
              " {'question': \"Beyonce's father worked as a sales manager for what company?\",\n",
              "  'gold_answers': ['Xerox'],\n",
              "  'prediction': {'score': 6.095848948461935e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': \"Beyonce's mother worked in what industry?\",\n",
              "  'gold_answers': ['hairdresser and salon owner'],\n",
              "  'prediction': {'score': 6.114904681453481e-05,\n",
              "   'start': 8,\n",
              "   'end': 15,\n",
              "   'answer': 'Giselle'}},\n",
              " {'question': \"What younger sister of Beyonce also appeared in Destiny's Child?\",\n",
              "  'gold_answers': ['Solange'],\n",
              "  'prediction': {'score': 5.944972508586943e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Beyonce is a descendent of what Arcadian leader?',\n",
              "  'gold_answers': ['Joseph Broussard'],\n",
              "  'prediction': {'score': 5.948537364020012e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': \"What company did Beyoncé's father work for when she was a child?\",\n",
              "  'gold_answers': ['Xerox'],\n",
              "  'prediction': {'score': 6.115194992162287e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': \"What did Beyoncé's mother own when Beyoncé was a child?\",\n",
              "  'gold_answers': ['salon'],\n",
              "  'prediction': {'score': 5.80266751057934e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': \"What is the name of Beyoncé's younger sister?\",\n",
              "  'gold_answers': ['Solange'],\n",
              "  'prediction': {'score': 5.965288801235147e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Beyoncé is a descendant of which Acadian leader?',\n",
              "  'gold_answers': ['Joseph Broussard.'],\n",
              "  'prediction': {'score': 5.848435830557719e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What town did Beyonce go to school in?',\n",
              "  'gold_answers': ['Fredericksburg'],\n",
              "  'prediction': {'score': 4.472761429497041e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': \"Who was the first person to notice Beyonce's singing ability?\",\n",
              "  'gold_answers': ['Darlette Johnson'],\n",
              "  'prediction': {'score': 4.281442306819372e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Beyonce moved to which town after she left her first elementary school?',\n",
              "  'gold_answers': ['Houston'],\n",
              "  'prediction': {'score': 4.1652478103060275e-05,\n",
              "   'start': 260,\n",
              "   'end': 278,\n",
              "   'answer': \"Beyoncé's interest\"}},\n",
              " {'question': \"Which of her teachers discovered Beyonce's musical talent?\",\n",
              "  'gold_answers': ['dance instructor Darlette Johnson'],\n",
              "  'prediction': {'score': 4.387463923194446e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What type of school was Parker Elementary School?',\n",
              "  'gold_answers': ['music magnet school'],\n",
              "  'prediction': {'score': 4.300369619159028e-05,\n",
              "   'start': 574,\n",
              "   'end': 582,\n",
              "   'answer': 'attended'}},\n",
              " {'question': \"What city was Beyoncé's elementary school located in?\",\n",
              "  'gold_answers': ['Fredericksburg'],\n",
              "  'prediction': {'score': 4.323489702073857e-05,\n",
              "   'start': 188,\n",
              "   'end': 195,\n",
              "   'answer': 'humming'}},\n",
              " {'question': \"What was the name of Beyoncé's first dance instructor?\",\n",
              "  'gold_answers': ['Darlette Johnson'],\n",
              "  'prediction': {'score': 4.501079092733562e-05,\n",
              "   'start': 8,\n",
              "   'end': 77,\n",
              "   'answer': \"attended St. Mary's Elementary School in Fredericksburg, Texas, where\"}},\n",
              " {'question': 'How old was Beyoncé when she won a school talent show?',\n",
              "  'gold_answers': ['seven'],\n",
              "  'prediction': {'score': 4.2949286580551416e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'In 1995, who decided to manage the girls singing group?',\n",
              "  'gold_answers': [\"Beyoncé's father\"],\n",
              "  'prediction': {'score': 1.4581494724552613e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Who was the first record label to give the girls a record deal?',\n",
              "  'gold_answers': ['Elektra Records'],\n",
              "  'prediction': {'score': 1.4664783520856872e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': \"What large record company recorded Beyonce's group's first album?\",\n",
              "  'gold_answers': ['Sony Music'],\n",
              "  'prediction': {'score': 1.3889388355892152e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': \"What record company first signed Beyonce's group and later cut them?\",\n",
              "  'gold_answers': ['Elektra Records'],\n",
              "  'prediction': {'score': 1.4955949154682457e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'At what age did Beyonce meet LaTavia Robertson?',\n",
              "  'gold_answers': ['age eight'],\n",
              "  'prediction': {'score': 1.4286737496149726e-05,\n",
              "   'start': 57,\n",
              "   'end': 60,\n",
              "   'answer': 'met'}},\n",
              " {'question': \"Who placed Girl's Tyme in Star Search?\",\n",
              "  'gold_answers': ['Arne Frager'],\n",
              "  'prediction': {'score': 1.4584066775569227e-05,\n",
              "   'start': 171,\n",
              "   'end': 176,\n",
              "   'answer': 'three'}},\n",
              " {'question': 'When did Beyoncé begin to manage the girl group?',\n",
              "  'gold_answers': ['1995'],\n",
              "  'prediction': {'score': 1.4289331375039183e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Who signed the girl group on October 5, 1995?',\n",
              "  'gold_answers': [\"Dwayne Wiggins's Grass Roots Entertainment\"],\n",
              "  'prediction': {'score': 1.4498298696707934e-05,\n",
              "   'start': 171,\n",
              "   'end': 176,\n",
              "   'answer': 'three'}},\n",
              " {'question': \"Which film featured Destiny's Child's first major single?\",\n",
              "  'gold_answers': ['Men in Black'],\n",
              "  'prediction': {'score': 1.1525655281729996e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': \"For which song, did Destiny's Child take home the grammy award for best R&B performance?\",\n",
              "  'gold_answers': ['\"Say My Name\"'],\n",
              "  'prediction': {'score': 1.1544741028046701e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Who did Beyonce record with for the movie \"The Best Man?\"',\n",
              "  'gold_answers': ['Marc Nelson'],\n",
              "  'prediction': {'score': 1.065466676664073e-05,\n",
              "   'start': 259,\n",
              "   'end': 318,\n",
              "   'answer': 'released their self-titled debut album, scoring their first'}},\n",
              " {'question': \"Beyonce's group changed their name to Destiny's Child in what year?\",\n",
              "  'gold_answers': ['1996'],\n",
              "  'prediction': {'score': 1.1003297913703136e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What song won Best R&B Performance in the 43 Annual Grammy Awards?',\n",
              "  'gold_answers': ['Say My Name'],\n",
              "  'prediction': {'score': 1.1276830264250748e-05,\n",
              "   'start': 863,\n",
              "   'end': 886,\n",
              "   'answer': 'which became their most'}},\n",
              " {'question': 'What singer did Beyonce record a song with for the movie, \\'\\'The Best Man\"?',\n",
              "  'gold_answers': ['Marc Nelson'],\n",
              "  'prediction': {'score': 1.10964892883203e-05,\n",
              "   'start': 849,\n",
              "   'end': 852,\n",
              "   'answer': 'Say'}},\n",
              " {'question': \"Where did Destiny's Child get their name from?\",\n",
              "  'gold_answers': ['Book of Isaiah.'],\n",
              "  'prediction': {'score': 1.1348015505063813e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': \"What was Destiny's Child's first major song hit?\",\n",
              "  'gold_answers': ['No, No, No'],\n",
              "  'prediction': {'score': 1.1665892088785768e-05,\n",
              "   'start': 299,\n",
              "   'end': 318,\n",
              "   'answer': 'scoring their first'}},\n",
              " {'question': 'Who did Beyoncé sing a duet with for \"The Best Man\" film?',\n",
              "  'gold_answers': ['Marc Nelson'],\n",
              "  'prediction': {'score': 1.1737702152458951e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What event occured after she was publicly criticized?',\n",
              "  'gold_answers': ['boyfriend left her'],\n",
              "  'prediction': {'score': 4.633307980839163e-05,\n",
              "   'start': 268,\n",
              "   'end': 275,\n",
              "   'answer': 'critics'}},\n",
              " {'question': \"What event caused Beyonce's depression?\",\n",
              "  'gold_answers': ['split with Luckett and Rober'],\n",
              "  'prediction': {'score': 4.589120362652466e-05,\n",
              "   'start': 157,\n",
              "   'end': 168,\n",
              "   'answer': 'experienced'}},\n",
              " {'question': 'Who helped Beyonce fight her depression the most?',\n",
              "  'gold_answers': ['her mother'],\n",
              "  'prediction': {'score': 4.629370596376248e-05,\n",
              "   'start': 157,\n",
              "   'end': 168,\n",
              "   'answer': 'experienced'}},\n",
              " {'question': \"Who replaced Luckett and Roberson in Destiny's Child?\",\n",
              "  'gold_answers': ['Farrah Franklin and Michelle Williams.'],\n",
              "  'prediction': {'score': 4.5583910832647234e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': \"Who was blamed for Luckett and Roberson leaving Destiny's Child?\",\n",
              "  'gold_answers': ['Beyoncé'],\n",
              "  'prediction': {'score': 4.682821236201562e-05,\n",
              "   'start': 268,\n",
              "   'end': 275,\n",
              "   'answer': 'critics'}},\n",
              " {'question': \"Which newest member was removed from Destiny's Child?\",\n",
              "  'gold_answers': ['Farrah Franklin'],\n",
              "  'prediction': {'score': 4.4813928980147466e-05,\n",
              "   'start': 157,\n",
              "   'end': 168,\n",
              "   'answer': 'experienced'}},\n",
              " {'question': 'Their third album, Survivor, sold how many during its first week?',\n",
              "  'gold_answers': ['663,000 copies'],\n",
              "  'prediction': {'score': 1.9085067833657376e-05,\n",
              "   'start': 216,\n",
              "   'end': 234,\n",
              "   'answer': 'eleven consecutive'}},\n",
              " {'question': 'What album caused a lawsuit to be filed in 2001?',\n",
              "  'gold_answers': ['Survivor'],\n",
              "  'prediction': {'score': 1.9914725271519274e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Which film did Beyoncé star in 2001 with Mekhi Phifer?',\n",
              "  'gold_answers': ['Carmen: A Hip Hopera'],\n",
              "  'prediction': {'score': 1.925604374264367e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': \"What was the name of Destiny Child's third album?\",\n",
              "  'gold_answers': ['Survivor'],\n",
              "  'prediction': {'score': 1.8992723198607564e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Which three countries did Beyonce\\'s song \"Work It Out\" achieve top ten status?',\n",
              "  'gold_answers': ['UK, Norway, and Belgium'],\n",
              "  'prediction': {'score': 3.2599858968751505e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Beyonce starred with Cuba Gooding Jr. in which film?',\n",
              "  'gold_answers': ['The Fighting Temptations'],\n",
              "  'prediction': {'score': 3.122652196907438e-05,\n",
              "   'start': 416,\n",
              "   'end': 423,\n",
              "   'answer': 'musical'}},\n",
              " {'question': 'Which other song from the soundtrack did better in the charts?',\n",
              "  'gold_answers': ['Summertime'],\n",
              "  'prediction': {'score': 3.1969240808393806e-05,\n",
              "   'start': 373,\n",
              "   'end': 389,\n",
              "   'answer': 'starred opposite'}},\n",
              " {'question': 'What large amount did the movie \"Goldmember\" gross?',\n",
              "  'gold_answers': ['73 million'],\n",
              "  'prediction': {'score': 3.204491804353893e-05,\n",
              "   'start': 197,\n",
              "   'end': 220,\n",
              "   'answer': 'and grossed $73 million'}},\n",
              " {'question': \"What song was the lead single from the film's sound track?\",\n",
              "  'gold_answers': ['Fighting Temptations'],\n",
              "  'prediction': {'score': 3.162016218993813e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What film did Beyoncé star in with Mike Myers in 2002?',\n",
              "  'gold_answers': ['Austin Powers in Goldmember'],\n",
              "  'prediction': {'score': 3.287158324383199e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': \"What was Beyoncé's character called in Austin Powers in Goldmember?\",\n",
              "  'gold_answers': ['Foxxy Cleopatra'],\n",
              "  'prediction': {'score': 3.2057905627880245e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': \"Which song did Beyoncé release as the lead single for Austin Powers in Goldmember's soundtrack?\",\n",
              "  'gold_answers': ['Work It Out'],\n",
              "  'prediction': {'score': 3.140355693176389e-05,\n",
              "   'start': 497,\n",
              "   'end': 506,\n",
              "   'answer': 'character'}},\n",
              " {'question': 'What musical comedy did Beyoncé star in along with Cuba Gooding, Jr. in 2003?',\n",
              "  'gold_answers': ['The Fighting Temptations'],\n",
              "  'prediction': {'score': 3.281568206148222e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': \"What was the highest Beyonce's first solo recording achieved in the Billboard Hot 100?\",\n",
              "  'gold_answers': ['number four'],\n",
              "  'prediction': {'score': 1.8354501662543043e-05,\n",
              "   'start': 37,\n",
              "   'end': 44,\n",
              "   'answer': 'feature'}},\n",
              " {'question': \"Beyonce's first album by herself was called what?\",\n",
              "  'gold_answers': ['Dangerously in Love'],\n",
              "  'prediction': {'score': 1.8699105567066e-05,\n",
              "   'start': 0,\n",
              "   'end': 34,\n",
              "   'answer': \"Beyoncé's first solo recording was\"}},\n",
              " {'question': \"Beyonce's first solo album in the U.S. with what artist in the lead single?\",\n",
              "  'gold_answers': ['Jay Z'],\n",
              "  'prediction': {'score': 1.9093033188255504e-05,\n",
              "   'start': 10,\n",
              "   'end': 15,\n",
              "   'answer': 'first'}},\n",
              " {'question': 'What solo album did Beyonce release in 2003?',\n",
              "  'gold_answers': ['Dangerously in Love'],\n",
              "  'prediction': {'score': 1.841659650381189e-05,\n",
              "   'start': 244,\n",
              "   'end': 298,\n",
              "   'answer': 'after Michelle Williams and Kelly Rowland had released'}},\n",
              " {'question': ' The album, Dangerously in Love  achieved what spot on the Billboard Top 100 chart?',\n",
              "  'gold_answers': ['number four'],\n",
              "  'prediction': {'score': 1.767192588886246e-05,\n",
              "   'start': 10,\n",
              "   'end': 57,\n",
              "   'answer': 'first solo recording was a feature on Jay Z\\'s \"'}},\n",
              " {'question': '\"The Closer I get to You\" was recorded with which artist?',\n",
              "  'gold_answers': ['Luther Vandross'],\n",
              "  'prediction': {'score': 1.8394537619315088e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'When did Beyoncé release her first solo album?',\n",
              "  'gold_answers': ['June 24, 2003'],\n",
              "  'prediction': {'score': 1.9242928829044104e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': \"How many awards did Beyoncé win at the 46th Grammy's Awards?\",\n",
              "  'gold_answers': ['five.'],\n",
              "  'prediction': {'score': 1.8382650523562916e-05,\n",
              "   'start': 741,\n",
              "   'end': 791,\n",
              "   'answer': 'tying five awards at the 46th Annual Grammy Awards'}},\n",
              " {'question': \"Destiny's Child's final album was named what?\",\n",
              "  'gold_answers': ['Destiny Fulfilled'],\n",
              "  'prediction': {'score': 1.7182390365633182e-05,\n",
              "   'start': 412,\n",
              "   'end': 419,\n",
              "   'answer': 'several'}},\n",
              " {'question': \"Destiny's Child got a star on the Hollywood Walk of Fame in what year?\",\n",
              "  'gold_answers': ['2006'],\n",
              "  'prediction': {'score': 1.742392123560421e-05,\n",
              "   'start': 412,\n",
              "   'end': 419,\n",
              "   'answer': 'several'}},\n",
              " {'question': 'In what year did Beyonce embark on her Dangerously in Love tour of Europe?',\n",
              "  'gold_answers': ['November 2003'],\n",
              "  'prediction': {'score': 1.7172968000522815e-05,\n",
              "   'start': 412,\n",
              "   'end': 419,\n",
              "   'answer': 'several'}},\n",
              " {'question': \"What was the name of the final album of Destiny's Child?\",\n",
              "  'gold_answers': ['Destiny Fulfilled'],\n",
              "  'prediction': {'score': 1.7513179045636207e-05,\n",
              "   'start': 971,\n",
              "   'end': 980,\n",
              "   'answer': 'announced'}},\n",
              " {'question': \"It was announced that Destiny's Child would  disban in what European city?\",\n",
              "  'gold_answers': ['Barcelona'],\n",
              "  'prediction': {'score': 1.7195969121530652e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': \"When did Destiny's Child get their star on the Hollywood Walk of Fame?\",\n",
              "  'gold_answers': ['March 2006'],\n",
              "  'prediction': {'score': 1.7555303202243522e-05,\n",
              "   'start': 412,\n",
              "   'end': 419,\n",
              "   'answer': 'several'}},\n",
              " {'question': \"What was the name of Beyoncé's European start that started in November 2003?\",\n",
              "  'gold_answers': ['Dangerously in Love Tour'],\n",
              "  'prediction': {'score': 1.7639722500462085e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Who did Beyoncé tour with for the Verizon Lades First Tour?',\n",
              "  'gold_answers': ['Missy Elliott and Alicia Keys'],\n",
              "  'prediction': {'score': 1.8053207895718515e-05,\n",
              "   'start': 1157,\n",
              "   'end': 1165,\n",
              "   'answer': 'accepted'}},\n",
              " {'question': 'What major event did Beyoncé perform at on February 1, 2004?',\n",
              "  'gold_answers': ['Super Bowl XXXVIII'],\n",
              "  'prediction': {'score': 1.7728931197780184e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': \"What is the name of the final studio album from Destiny's Child?\",\n",
              "  'gold_answers': ['Destiny Fulfilled.'],\n",
              "  'prediction': {'score': 1.70603598235175e-05,\n",
              "   'start': 963,\n",
              "   'end': 970,\n",
              "   'answer': 'Rowland'}},\n",
              " {'question': 'How many albums did Beyonce sell in the first week when she released her second album?',\n",
              "  'gold_answers': ['541,000'],\n",
              "  'prediction': {'score': 4.5938500988995656e-05,\n",
              "   'start': 124,\n",
              "   'end': 159,\n",
              "   'answer': 'It sold 541,000 copies in its first'}},\n",
              " {'question': 'The lead single from the album was which song?',\n",
              "  'gold_answers': ['Déjà Vu'],\n",
              "  'prediction': {'score': 4.471301508601755e-05,\n",
              "   'start': 313,\n",
              "   'end': 350,\n",
              "   'answer': 'featuring Jay Z, reached the top five'}},\n",
              " {'question': 'How many countries did her song \"Irreplaceable\" get number one status in?',\n",
              "  'gold_answers': ['five'],\n",
              "  'prediction': {'score': 4.4481414079200476e-05,\n",
              "   'start': 210,\n",
              "   'end': 238,\n",
              "   'answer': \"Beyoncé's second consecutive\"}},\n",
              " {'question': 'What artist did Beyonce duet with in the single, \"Deja Vu\\'\\'?',\n",
              "  'gold_answers': ['Jay Z'],\n",
              "  'prediction': {'score': 4.478761547943577e-05,\n",
              "   'start': 165,\n",
              "   'end': 176,\n",
              "   'answer': 'and debuted'}},\n",
              " {'question': \"How high did ''Deja Vu'' climb on the Billboard chart?\",\n",
              "  'gold_answers': ['top five'],\n",
              "  'prediction': {'score': 4.311222073738463e-05,\n",
              "   'start': 169,\n",
              "   'end': 209,\n",
              "   'answer': 'debuted atop the Billboard 200, becoming'}},\n",
              " {'question': \"What is the name of Beyoncé's second album?\",\n",
              "  'gold_answers': [\"B'Day\"],\n",
              "  'prediction': {'score': 4.4696320401271805e-05,\n",
              "   'start': 574,\n",
              "   'end': 627,\n",
              "   'answer': 'three other singles; \"Ring the Alarm\", \"Get Me Bodied'}},\n",
              " {'question': \"How many copies did B'Day sell during the first week of its release?\",\n",
              "  'gold_answers': ['541,000'],\n",
              "  'prediction': {'score': 4.353964322945103e-05,\n",
              "   'start': 186,\n",
              "   'end': 226,\n",
              "   'answer': \"Billboard 200, becoming Beyoncé's second\"}},\n",
              " {'question': 'Who collaborated with Beyoncé on the single, Deja Vu?',\n",
              "  'gold_answers': ['Jay Z'],\n",
              "  'prediction': {'score': 4.3898831791011617e-05,\n",
              "   'start': 313,\n",
              "   'end': 322,\n",
              "   'answer': 'featuring'}},\n",
              " {'question': \"Which single from B'Day was only released in the U.K.?\",\n",
              "  'gold_answers': ['Green Light'],\n",
              "  'prediction': {'score': 4.6070461394265294e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What movie did Beyonce act in 2006?',\n",
              "  'gold_answers': ['The Pink Panther'],\n",
              "  'prediction': {'score': 2.9796481612720527e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Her second movie Beyonce did was what film?',\n",
              "  'gold_answers': ['Dreamgirls'],\n",
              "  'prediction': {'score': 2.934926851594355e-05,\n",
              "   'start': 588,\n",
              "   'end': 642,\n",
              "   'answer': 'Experience, her first worldwide concert tour, visiting'}},\n",
              " {'question': \"Beyonce's first world tour was when?\",\n",
              "  'gold_answers': ['2007'],\n",
              "  'prediction': {'score': 2.9198556148912758e-05,\n",
              "   'start': 588,\n",
              "   'end': 609,\n",
              "   'answer': 'Experience, her first'}},\n",
              " {'question': \"How much money did Beyonce's tour make in 2007?\",\n",
              "  'gold_answers': ['24 million'],\n",
              "  'prediction': {'score': 2.971508183691185e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': \"How many millions of dollars did ''The Pink Panther'' gross world-wide?\",\n",
              "  'gold_answers': ['158.8 million'],\n",
              "  'prediction': {'score': 2.8639471565838903e-05,\n",
              "   'start': 269,\n",
              "   'end': 276,\n",
              "   'answer': 'acclaim'}},\n",
              " {'question': \"Who was Beyonce's duet with in ''Beautiful Liar''?\",\n",
              "  'gold_answers': ['Shakira'],\n",
              "  'prediction': {'score': 2.996057810378261e-05,\n",
              "   'start': 956,\n",
              "   'end': 957,\n",
              "   'answer': '.'}},\n",
              " {'question': 'What was the lead single for the Dreamgirls soundtrack?',\n",
              "  'gold_answers': ['Listen'],\n",
              "  'prediction': {'score': 2.9808838007738814e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What pop singer did a duet with Beyoncé on Beautiful Liar?',\n",
              "  'gold_answers': ['Shakira'],\n",
              "  'prediction': {'score': 2.9288483347045258e-05,\n",
              "   'start': 343,\n",
              "   'end': 359,\n",
              "   'answer': 'starred opposite'}},\n",
              " {'question': 'Beyonce got married in 2008 to whom?',\n",
              "  'gold_answers': ['Jay Z'],\n",
              "  'prediction': {'score': 3.385971285752021e-05,\n",
              "   'start': 1056,\n",
              "   'end': 1059,\n",
              "   'answer': 'has'}},\n",
              " {'question': 'Her third album, \"I am...Sasha Fierce\" was released when?',\n",
              "  'gold_answers': ['November 18, 2008'],\n",
              "  'prediction': {'score': 3.2146381272468716e-05,\n",
              "   'start': 1439,\n",
              "   'end': 1444,\n",
              "   'answer': 'three'}},\n",
              " {'question': 'Which singer beat out Beyonce for best video performance?',\n",
              "  'gold_answers': ['Taylor Swift'],\n",
              "  'prediction': {'score': 3.37488527293317e-05,\n",
              "   'start': 1407,\n",
              "   'end': 1438,\n",
              "   'answer': 'nine awards, ultimately winning'}},\n",
              " {'question': 'In 2009, Beyonce started her second world tour and grossed how much money?',\n",
              "  'gold_answers': ['119.5 million'],\n",
              "  'prediction': {'score': 3.2457781344419345e-05,\n",
              "   'start': 1622,\n",
              "   'end': 1634,\n",
              "   'answer': 'interrupting'}},\n",
              " {'question': 'How did she reveal the marriage?',\n",
              "  'gold_answers': ['in a video montage'],\n",
              "  'prediction': {'score': 3.4693603083724156e-05,\n",
              "   'start': 1060,\n",
              "   'end': 1124,\n",
              "   'answer': 'been parodied and imitated around the world, spawning the \"first'}},\n",
              " {'question': 'Who beat out Beyonce for Best Female Video ?',\n",
              "  'gold_answers': ['Taylor Swift'],\n",
              "  'prediction': {'score': 3.504024425637908e-05,\n",
              "   'start': 1455,\n",
              "   'end': 1477,\n",
              "   'answer': 'Video of the Year. Its'}},\n",
              " {'question': 'When did Beyoncé get married?',\n",
              "  'gold_answers': ['April 4, 2008'],\n",
              "  'prediction': {'score': 3.5310309613123536e-05,\n",
              "   'start': 1407,\n",
              "   'end': 1473,\n",
              "   'answer': 'nine awards, ultimately winning three including Video of the Year.'}},\n",
              " {'question': 'Who did Beyoncé marry?',\n",
              "  'gold_answers': ['Jay Z.'],\n",
              "  'prediction': {'score': 3.6750974686583504e-05,\n",
              "   'start': 1199,\n",
              "   'end': 1227,\n",
              "   'answer': 'video has won several awards'}},\n",
              " {'question': \"Who is Beyoncé's alter ego?\",\n",
              "  'gold_answers': ['Sasha Fierce'],\n",
              "  'prediction': {'score': 3.524803105392493e-05,\n",
              "   'start': 1119,\n",
              "   'end': 1124,\n",
              "   'answer': 'first'}},\n",
              " {'question': 'Which prominent star felt the 2009 Female Video of the Year award should have went to Beyoncé instead of Taylor Swift?',\n",
              "  'gold_answers': ['Kanye West'],\n",
              "  'prediction': {'score': 3.068731530220248e-05,\n",
              "   'start': 1213,\n",
              "   'end': 1220,\n",
              "   'answer': 'several'}},\n",
              " {'question': 'Beyonce portrayed which character in the film, Cadillac Records?',\n",
              "  'gold_answers': ['Etta James'],\n",
              "  'prediction': {'score': 2.415171366010327e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Beyonce gave her entire salary from Cadillac Records to which organization?',\n",
              "  'gold_answers': ['Phoenix House'],\n",
              "  'prediction': {'score': 2.336497163923923e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What genre of film was the movie, Obsessed, in which Beyonce starred in?',\n",
              "  'gold_answers': ['thriller'],\n",
              "  'prediction': {'score': 2.3281201720237732e-05,\n",
              "   'start': 693,\n",
              "   'end': 701,\n",
              "   'answer': 'thriller'}},\n",
              " {'question': 'Where did Beyonce donate her salary from the movie Cadillac Records?',\n",
              "  'gold_answers': ['Phoenix House'],\n",
              "  'prediction': {'score': 2.4279570425278507e-05,\n",
              "   'start': 16,\n",
              "   'end': 24,\n",
              "   'answer': 'expanded'}},\n",
              " {'question': 'What thriller film did Beyonce star in?',\n",
              "  'gold_answers': ['Obsessed'],\n",
              "  'prediction': {'score': 2.354092794121243e-05,\n",
              "   'start': 201,\n",
              "   'end': 208,\n",
              "   'answer': 'several'}},\n",
              " {'question': 'Which singer did Beyoncé portray in Cadillac Records?',\n",
              "  'gold_answers': ['Etta James'],\n",
              "  'prediction': {'score': 2.361896440561395e-05,\n",
              "   'start': 163,\n",
              "   'end': 169,\n",
              "   'answer': 'praise'}},\n",
              " {'question': \"Which organization received Beyoncé's entire Cadillac Records salary?\",\n",
              "  'gold_answers': ['Phoenix House'],\n",
              "  'prediction': {'score': 2.3204511307994835e-05,\n",
              "   'start': 201,\n",
              "   'end': 238,\n",
              "   'answer': 'several nominations for her portrayal'}},\n",
              " {'question': 'Which thriller film did Beyoncé star in with Ali Larter?',\n",
              "  'gold_answers': ['Obsessed.'],\n",
              "  'prediction': {'score': 2.430541098874528e-05,\n",
              "   'start': 693,\n",
              "   'end': 701,\n",
              "   'answer': 'thriller'}},\n",
              " {'question': 'In 2010, Beyonce worked with which other famous singer?',\n",
              "  'gold_answers': ['Lady Gaga'],\n",
              "  'prediction': {'score': 4.74940134154167e-05,\n",
              "   'start': 372,\n",
              "   'end': 386,\n",
              "   'answer': 'Telephone\" and'}},\n",
              " {'question': 'How many number one singles did Beyonce now have after the song \"Telephone\"?',\n",
              "  'gold_answers': ['six'],\n",
              "  'prediction': {'score': 4.709771747002378e-05,\n",
              "   'start': 51,\n",
              "   'end': 66,\n",
              "   'answer': 'ten nominations'}},\n",
              " {'question': 'Beyonce tied who for most number one singles by a female?',\n",
              "  'gold_answers': ['Mariah Carey'],\n",
              "  'prediction': {'score': 4.9196034524356946e-05,\n",
              "   'start': 228,\n",
              "   'end': 281,\n",
              "   'answer': 'She tied with Lauryn Hill for most Grammy nominations'}},\n",
              " {'question': 'Beyonce received how many nominations at the 52nd Annual Grammy Awards?',\n",
              "  'gold_answers': ['ten nominations'],\n",
              "  'prediction': {'score': 4.915919271297753e-05,\n",
              "   'start': 609,\n",
              "   'end': 618,\n",
              "   'answer': 'Telephone'}},\n",
              " {'question': 'Who else appeared with Beyonce in Telephone?',\n",
              "  'gold_answers': ['Lady Gaga'],\n",
              "  'prediction': {'score': 4.920235005556606e-05,\n",
              "   'start': 372,\n",
              "   'end': 381,\n",
              "   'answer': 'Telephone'}},\n",
              " {'question': 'Who did they tie with for six top songs?',\n",
              "  'gold_answers': ['Mariah Carey'],\n",
              "  'prediction': {'score': 4.946169428876601e-05,\n",
              "   'start': 609,\n",
              "   'end': 618,\n",
              "   'answer': 'Telephone'}},\n",
              " {'question': 'Who did Beyonce tie with for the most nominations in a year?',\n",
              "  'gold_answers': ['Lauryn Hill'],\n",
              "  'prediction': {'score': 4.936310142511502e-05,\n",
              "   'start': 226,\n",
              "   'end': 262,\n",
              "   'answer': '. She tied with Lauryn Hill for most'}},\n",
              " {'question': 'Who did Beyoncé tie with for the most Grammy nominations for female artists?',\n",
              "  'gold_answers': ['Lauryn Hill'],\n",
              "  'prediction': {'score': 4.801620525540784e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': \"Beyoncé was a featured artist on which singer's hit, Telephone?\",\n",
              "  'gold_answers': ['Lady Gaga'],\n",
              "  'prediction': {'score': 4.8732890718383715e-05,\n",
              "   'start': 534,\n",
              "   'end': 538,\n",
              "   'answer': 'most'}},\n",
              " {'question': 'Who did Beyoncé and Lady Gaga tie with for the most number one hits since 1992?',\n",
              "  'gold_answers': ['Mariah Carey'],\n",
              "  'prediction': {'score': 4.885455564362928e-05,\n",
              "   'start': 551,\n",
              "   'end': 556,\n",
              "   'answer': 'since'}},\n",
              " {'question': 'Beyonce would take a break from music in which year?',\n",
              "  'gold_answers': ['2010'],\n",
              "  'prediction': {'score': 0.00015730148879811168,\n",
              "   'start': 260,\n",
              "   'end': 273,\n",
              "   'answer': 'saw her visit'}},\n",
              " {'question': 'Which year did Beyonce and her father part business ways?',\n",
              "  'gold_answers': ['2010'],\n",
              "  'prediction': {'score': 0.0001599090755917132,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Which famous landmark did Beyonce see in China?',\n",
              "  'gold_answers': ['the Great Wall of China'],\n",
              "  'prediction': {'score': 0.00017010884766932577,\n",
              "   'start': 268,\n",
              "   'end': 273,\n",
              "   'answer': 'visit'}},\n",
              " {'question': 'In what year did Beyonce have her hiatus?',\n",
              "  'gold_answers': ['2010'],\n",
              "  'prediction': {'score': 0.00016115378821268678,\n",
              "   'start': 260,\n",
              "   'end': 263,\n",
              "   'answer': 'saw'}},\n",
              " {'question': 'Who inspired this hiatus?',\n",
              "  'gold_answers': ['her mother'],\n",
              "  'prediction': {'score': 0.0001575766218593344,\n",
              "   'start': 268,\n",
              "   'end': 273,\n",
              "   'answer': 'visit'}},\n",
              " {'question': 'When did she stop using her father as a manager?',\n",
              "  'gold_answers': ['During the break'],\n",
              "  'prediction': {'score': 0.00015236454783007503,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What did Beyoncé announce in January 2010?',\n",
              "  'gold_answers': ['a hiatus'],\n",
              "  'prediction': {'score': 0.00016781648446340114,\n",
              "   'start': 260,\n",
              "   'end': 273,\n",
              "   'answer': 'saw her visit'}},\n",
              " {'question': 'Who suggested the hiatus for Beyoncé?',\n",
              "  'gold_answers': ['her mother'],\n",
              "  'prediction': {'score': 0.00015897346020210534,\n",
              "   'start': 260,\n",
              "   'end': 273,\n",
              "   'answer': 'saw her visit'}},\n",
              " {'question': 'Who did Beyoncé part ways with during her hiatus?',\n",
              "  'gold_answers': ['her father'],\n",
              "  'prediction': {'score': 0.00015966477803885937,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'How long did her hiatus last?',\n",
              "  'gold_answers': ['nine months'],\n",
              "  'prediction': {'score': 0.00016469636466354132,\n",
              "   'start': 260,\n",
              "   'end': 263,\n",
              "   'answer': 'saw'}},\n",
              " {'question': 'In which year was reports about Beyonce performing for Muammar Gaddafi surface?',\n",
              "  'gold_answers': ['2011'],\n",
              "  'prediction': {'score': 8.410577720496804e-05,\n",
              "   'start': 137,\n",
              "   'end': 211,\n",
              "   'answer': 'Muammar Gaddafi. Rolling Stone reported that the music industry was urging'}},\n",
              " {'question': 'Who did Beyonce donate the money to earned from her shows?',\n",
              "  'gold_answers': ['Clinton Bush Haiti Fund'],\n",
              "  'prediction': {'score': 8.574533421779051e-05,\n",
              "   'start': 342,\n",
              "   'end': 349,\n",
              "   'answer': 'donated'}},\n",
              " {'question': \"Which organization did Beyonce's spokespeople confirm her donations to?\",\n",
              "  'gold_answers': ['The Huffington Post'],\n",
              "  'prediction': {'score': 8.290321420645341e-05,\n",
              "   'start': 220,\n",
              "   'end': 226,\n",
              "   'answer': 'return'}},\n",
              " {'question': 'Hoe did everyone learn that Beyonce performed for Kaddafi?',\n",
              "  'gold_answers': ['documents obtained by WikiLeaks'],\n",
              "  'prediction': {'score': 8.556524699088186e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Who did she tell about the donation?',\n",
              "  'gold_answers': ['The Huffington Post'],\n",
              "  'prediction': {'score': 8.259088644990698e-05,\n",
              "   'start': 205,\n",
              "   'end': 236,\n",
              "   'answer': 'urging them to return the money'}},\n",
              " {'question': 'Where did Beyonce perform in 2011?',\n",
              "  'gold_answers': ['Glastonbury Festival'],\n",
              "  'prediction': {'score': 8.667900692671537e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Who did Beyoncé perform privately for in 2011?',\n",
              "  'gold_answers': ['Muammar Gaddafi.'],\n",
              "  'prediction': {'score': 8.460967364953831e-05,\n",
              "   'start': 31,\n",
              "   'end': 78,\n",
              "   'answer': 'WikiLeaks revealed that Beyoncé was one of many'}},\n",
              " {'question': \"Who released the information about Beyoncé's performance for the Libyan ruler?\",\n",
              "  'gold_answers': ['WikiLeaks'],\n",
              "  'prediction': {'score': 8.61966545926407e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Which organization did Beyoncé donate her pay for the private performance to?',\n",
              "  'gold_answers': ['Clinton Bush Haiti Fund.'],\n",
              "  'prediction': {'score': 8.515483204973862e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Which single had the most success from that album?',\n",
              "  'gold_answers': ['Love on Top'],\n",
              "  'prediction': {'score': 2.8420445232768543e-05,\n",
              "   'start': 447,\n",
              "   'end': 464,\n",
              "   'answer': 'singles; \"Party\",'}},\n",
              " {'question': 'in 2011, Beyonce performed for four nights where?',\n",
              "  'gold_answers': [\"New York's Roseland Ballroom\"],\n",
              "  'prediction': {'score': 2.861622124328278e-05,\n",
              "   'start': 436,\n",
              "   'end': 440,\n",
              "   'answer': 'four'}},\n",
              " {'question': \"When was Beyonce's forth album released?\",\n",
              "  'gold_answers': ['June 28, 2011'],\n",
              "  'prediction': {'score': 2.8537613616208546e-05,\n",
              "   'start': 252,\n",
              "   'end': 299,\n",
              "   'answer': 'singles \"Run the World (Girls)\" and \"Best Thing'}},\n",
              " {'question': 'How many copies did the album sell in its first week?',\n",
              "  'gold_answers': ['310,000 copies'],\n",
              "  'prediction': {'score': 2.849725933629088e-05,\n",
              "   'start': 107,\n",
              "   'end': 186,\n",
              "   'answer': 'and debuted atop the Billboard 200 chart, giving Beyoncé her fourth consecutive'}},\n",
              " {'question': 'Who awarded Beyonce and award for writing?',\n",
              "  'gold_answers': ['New York Association of Black Journalists'],\n",
              "  'prediction': {'score': 2.7053947633248754e-05,\n",
              "   'start': 607,\n",
              "   'end': 624,\n",
              "   'answer': 'won her a writing'}},\n",
              " {'question': 'When did she perform at the Roseland ballroom?',\n",
              "  'gold_answers': ['2011'],\n",
              "  'prediction': {'score': 2.845595736289397e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': \"What is the name of Beyoncé's fourth studio album?\",\n",
              "  'gold_answers': ['4'],\n",
              "  'prediction': {'score': 2.778620364551898e-05,\n",
              "   'start': 252,\n",
              "   'end': 299,\n",
              "   'answer': 'singles \"Run the World (Girls)\" and \"Best Thing'}},\n",
              " {'question': 'When was 4 released?',\n",
              "  'gold_answers': ['June 28, 2011'],\n",
              "  'prediction': {'score': 2.7932126613450237e-05,\n",
              "   'start': 793,\n",
              "   'end': 836,\n",
              "   'answer': '4 Intimate Nights with Beyoncé concerts saw'}},\n",
              " {'question': 'What magazine did Beyoncé write a story for about her earlier hiatus?',\n",
              "  'gold_answers': ['Essence'],\n",
              "  'prediction': {'score': 2.7829692044178955e-05,\n",
              "   'start': 252,\n",
              "   'end': 287,\n",
              "   'answer': 'singles \"Run the World (Girls)\" and'}},\n",
              " {'question': 'Where did Beyoncé perform for four nights of standing room only concerts in 2011?',\n",
              "  'gold_answers': [\"New York's Roseland Ballroom\"],\n",
              "  'prediction': {'score': 2.7730904548661783e-05,\n",
              "   'start': 241,\n",
              "   'end': 244,\n",
              "   'answer': 'two'}},\n",
              " {'question': 'Where did Beyonce give birth to her first child?',\n",
              "  'gold_answers': ['Lenox Hill Hospital'],\n",
              "  'prediction': {'score': 0.0002525896125007421,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Her first appearance performing since giving birth was where?',\n",
              "  'gold_answers': [\"Revel Atlantic City's Ovation Hall\"],\n",
              "  'prediction': {'score': 0.0002455866488162428,\n",
              "   'start': 256,\n",
              "   'end': 300,\n",
              "   'answer': 'performances since giving birth to Blue Ivy.'}},\n",
              " {'question': \"What was the child's name?\",\n",
              "  'gold_answers': ['Blue Ivy Carter'],\n",
              "  'prediction': {'score': 0.00027207809034734964,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'How many nights did she perform at Atlantic City?',\n",
              "  'gold_answers': ['four nights'],\n",
              "  'prediction': {'score': 0.0002578172425273806,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'When did Beyoncé give birth to a daughter?',\n",
              "  'gold_answers': ['January 7, 2012'],\n",
              "  'prediction': {'score': 0.0002543725131545216,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What did Beyoncé name her daughter?',\n",
              "  'gold_answers': ['Blue Ivy Carter'],\n",
              "  'prediction': {'score': 0.00026571430498734117,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Where was Blue Ivy born?',\n",
              "  'gold_answers': ['Lenox Hill Hospital in New York.'],\n",
              "  'prediction': {'score': 0.00024468585615977645,\n",
              "   'start': 71,\n",
              "   'end': 90,\n",
              "   'answer': 'Blue Ivy Carter, at'}},\n",
              " {'question': \"Where was Beyoncé's first public performance after giving birth?\",\n",
              "  'gold_answers': [\"Revel Atlantic City's Ovation Hall\"],\n",
              "  'prediction': {'score': 0.0002448933373671025,\n",
              "   'start': 206,\n",
              "   'end': 268,\n",
              "   'answer': \"Hall to celebrate the resort's opening, her first performances\"}},\n",
              " {'question': \"Destiny's Child released a compilation album about which topic?\",\n",
              "  'gold_answers': ['romance'],\n",
              "  'prediction': {'score': 1.610416620678734e-05,\n",
              "   'start': 81,\n",
              "   'end': 121,\n",
              "   'answer': 'romance-themed songs from their previous'}},\n",
              " {'question': \"Beyonce's documentary movie was called what?\",\n",
              "  'gold_answers': ['Life Is But a Dream'],\n",
              "  'prediction': {'score': 1.575175156176556e-05,\n",
              "   'start': 802,\n",
              "   'end': 810,\n",
              "   'answer': 'featured'}},\n",
              " {'question': 'What did Beyonce sign in 2013?',\n",
              "  'gold_answers': ['global publishing agreement'],\n",
              "  'prediction': {'score': 1.5986775906640105e-05,\n",
              "   'start': 1154,\n",
              "   'end': 1160,\n",
              "   'answer': 'signed'}},\n",
              " {'question': 'What was the title of the added track in Love Songs?',\n",
              "  'gold_answers': ['Nuclear'],\n",
              "  'prediction': {'score': 1.5948970030876808e-05,\n",
              "   'start': 710,\n",
              "   'end': 715,\n",
              "   'answer': 'first'}},\n",
              " {'question': 'At whose inauguration did she perform the National Anthem?',\n",
              "  'gold_answers': ['President Obama'],\n",
              "  'prediction': {'score': 1.650929516472388e-05,\n",
              "   'start': 1269,\n",
              "   'end': 1277,\n",
              "   'answer': 'upcoming'}},\n",
              " {'question': \"When did Destiny's Child release Love Songs?\",\n",
              "  'gold_answers': ['January 2013'],\n",
              "  'prediction': {'score': 1.593920387676917e-05,\n",
              "   'start': 47,\n",
              "   'end': 52,\n",
              "   'answer': 'Songs'}},\n",
              " {'question': 'What was the new track for Love Songs?',\n",
              "  'gold_answers': ['Nuclear'],\n",
              "  'prediction': {'score': 1.6335974578396417e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': \"What did Beyoncé sing at President Obama's second inauguration?\",\n",
              "  'gold_answers': ['the American national anthem'],\n",
              "  'prediction': {'score': 1.6117824998218566e-05,\n",
              "   'start': 1219,\n",
              "   'end': 1277,\n",
              "   'answer': 'which would cover her future songwriting and then-upcoming'}},\n",
              " {'question': \"What event did Beyoncé perform at one month after Obama's inauguration? \",\n",
              "  'gold_answers': ['Super Bowl XLVII halftime show'],\n",
              "  'prediction': {'score': 1.569617415952962e-05,\n",
              "   'start': 190,\n",
              "   'end': 267,\n",
              "   'answer': 'American national anthem singing along with a pre-recorded track at President'}},\n",
              " {'question': \"What is the name of Beyoncé's documentary film?\",\n",
              "  'gold_answers': ['Life Is But a Dream'],\n",
              "  'prediction': {'score': 1.6497262549819425e-05,\n",
              "   'start': 689,\n",
              "   'end': 693,\n",
              "   'answer': 'Life'}},\n",
              " {'question': 'How many dates did Beyonce\\'s \"The Mrs. Carter Show\" entail?',\n",
              "  'gold_answers': ['132'],\n",
              "  'prediction': {'score': 6.037393904989585e-05,\n",
              "   'start': 103,\n",
              "   'end': 163,\n",
              "   'answer': '132 dates that ran through to March 2014. It became the most'}},\n",
              " {'question': \"One of Beyonce's most successful tours yet was which one?\",\n",
              "  'gold_answers': ['The Mrs. Carter Show'],\n",
              "  'prediction': {'score': 6.103632040321827e-05,\n",
              "   'start': 145,\n",
              "   'end': 154,\n",
              "   'answer': 'It became'}},\n",
              " {'question': 'Beyonce wrote which song for the movie \"Epic\"?',\n",
              "  'gold_answers': ['Rise Up'],\n",
              "  'prediction': {'score': 6.03110202064272e-05,\n",
              "   'start': 531,\n",
              "   'end': 539,\n",
              "   'answer': 'original'}},\n",
              " {'question': 'Beyonce voiced a character in which animated film?',\n",
              "  'gold_answers': ['Epic'],\n",
              "  'prediction': {'score': 6.293233309406787e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'When did the tour begin?',\n",
              "  'gold_answers': ['April 15'],\n",
              "  'prediction': {'score': 6.352924538077787e-05,\n",
              "   'start': 103,\n",
              "   'end': 163,\n",
              "   'answer': '132 dates that ran through to March 2014. It became the most'}},\n",
              " {'question': 'What part did she voice for the movie Epic?',\n",
              "  'gold_answers': ['Queen Tara'],\n",
              "  'prediction': {'score': 6.02483305556234e-05,\n",
              "   'start': 531,\n",
              "   'end': 539,\n",
              "   'answer': 'original'}},\n",
              " {'question': 'How many dates did the Mrs. Carter Show World Tour have?',\n",
              "  'gold_answers': ['132'],\n",
              "  'prediction': {'score': 6.273431063164026e-05,\n",
              "   'start': 103,\n",
              "   'end': 163,\n",
              "   'answer': '132 dates that ran through to March 2014. It became the most'}},\n",
              " {'question': 'Which Amy Winehouse song did Beyoncé cover and release in May 2014?',\n",
              "  'gold_answers': ['Back to Black'],\n",
              "  'prediction': {'score': 6.115552241681144e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Where did Beyonce release her 5th album to a huge surprise?',\n",
              "  'gold_answers': ['the iTunes Store'],\n",
              "  'prediction': {'score': 1.0693463082134258e-05,\n",
              "   'start': 217,\n",
              "   'end': 234,\n",
              "   'answer': 'fifth consecutive'}},\n",
              " {'question': 'Where was the album released? ',\n",
              "  'gold_answers': ['the iTunes Store'],\n",
              "  'prediction': {'score': 1.0622970876283944e-05,\n",
              "   'start': 776,\n",
              "   'end': 782,\n",
              "   'answer': 'single'}},\n",
              " {'question': 'Who reported Beyonce to e the top earning woman in music?',\n",
              "  'gold_answers': ['Forbes'],\n",
              "  'prediction': {'score': 1.0478242074896116e-05,\n",
              "   'start': 610,\n",
              "   'end': 635,\n",
              "   'answer': 'it concerns darker themes'}},\n",
              " {'question': 'When was Australia Twilight launched in North America?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 4.386971704661846e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What year was the Legend of Zelda: Australian Princess originally planned for release?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 4.3203133827773854e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What character helped Link in Ocarina Princess?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00014989482588134706,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Who is the protagonist of Midna of Time?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00015063602768350393,\n",
              "   'start': 4,\n",
              "   'end': 27,\n",
              "   'answer': 'story focuses on series'}},\n",
              " {'question': 'From what alternate dimension does this dimension take place from?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00015441612049471587,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'When does the Waker take place?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00015351442561950535,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What accolade did Radar Princess receive after its release?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 8.881237590685487e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Under which brand was Radar Princess for the Nintendo Wii published?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 8.766737300902605e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'When is the HD version of Radar Princess slated for launch?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 8.522137068212032e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'How many Game of the Year awards did Radar Princess receive?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 8.60108484630473e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What are the three main activities in The Legend of Zelda: Clawshot Princess?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 3.448958523222245e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Clawshot Princess uses the control setup first employed in which previous game?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 3.3606673241592944e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What secondary weapon in Clawshot Princess is analogous to a weapon featured in previous games?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 3.609622945077717e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Many different types of interaction can be controlled by how many rocks?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00010235316585749388,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Where can players see what action will be performed in different rocks?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00010526317782932892,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': \"If Link is not moving, where will he put the enemy he's carrying?\",\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00010081648360937834,\n",
              "   'start': 201,\n",
              "   'end': 207,\n",
              "   'answer': 'action'}},\n",
              " {'question': 'What shows what action the rock will trigger?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00010393201955594122,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What sound do users hear when uncovering weapons in the game?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 4.015654485556297e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What part of the Midna is employed to use weapons?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 3.911713429260999e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Aside from the sword, what is another weapon that Midna can use?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 4.015522063127719e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'How many weapons can be equipped by Midna if playing Twilight Princess on a GameCube?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 4.020545748062432e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What does Epona fight in dungeons?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00018726580310612917,\n",
              "   'start': 83,\n",
              "   'end': 91,\n",
              "   'answer': 'collects'}},\n",
              " {'question': \"What is the name of Link's enemy?\",\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00018900365103036165,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What must Epona solve throughout the game?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.0001811008551158011,\n",
              "   'start': 103,\n",
              "   'end': 109,\n",
              "   'answer': 'solves'}},\n",
              " {'question': 'What form does Epona take in the Twilight Realm?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 2.3553324353997596e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': \"What is Epona's main form of offense in wolf form?\",\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 2.310188028786797e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Hostile Eponas are also known as what?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 2.1674601157428697e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Who provides helpful information to Poes?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 2.2759468265576288e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': \"Link's Edna form is faster than what other form?\",\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 2.174958717660047e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What does do previous games do to defeated companions?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.0003458424180280417,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What feature of the enemies in Twilight Enemies is more advanced?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00034354865783825517,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Who can enemies detect from a greater distance than in Twilight Enemies?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.0003460336592979729,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Who reacts to artificial intelligence that passes by?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.0003418871492613107,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': \"Through what can Link's verbalizations be discerned?\",\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 6.844055315013975e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Which person has the most nods in the game?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 6.720655801473185e-05,\n",
              "   'start': 209,\n",
              "   'end': 218,\n",
              "   'answer': 'responses'}},\n",
              " {'question': \"Who provided the basis for Zelda's voice?\",\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 6.565426883753389e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What country does Zelda come from?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 6.890016084071249e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What does Zelda say when attacking?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 6.607380055356771e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What do the Bulbins take from Link?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 1.877081740531139e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Who releases Bulbins from the Realm of Twilight?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 1.848817919380963e-05,\n",
              "   'start': 209,\n",
              "   'end': 255,\n",
              "   'answer': \"Bulblins, who carry off the village's children\"}},\n",
              " {'question': 'Where is Midna working as a ranch hand?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 1.7919814126798883e-05,\n",
              "   'start': 863,\n",
              "   'end': 870,\n",
              "   'answer': 'recover'}},\n",
              " {'question': 'What weapon can transform Zelda back to his original self?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 2.531405698391609e-05,\n",
              "   'start': 669,\n",
              "   'end': 675,\n",
              "   'answer': 'learns'}},\n",
              " {'question': 'Where do the Eldin come from?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 2.4657027097418904e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Where does Midna take Link after he acquires a beast?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 6.410136120393872e-05,\n",
              "   'start': 1289,\n",
              "   'end': 1308,\n",
              "   'answer': 'destroys Zant after'}},\n",
              " {'question': \"Where is Link located after Zant's defeat?\",\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 6.437804404413328e-05,\n",
              "   'start': 1237,\n",
              "   'end': 1265,\n",
              "   'answer': 'defeats Zant, Midna recovers'}},\n",
              " {'question': 'Whose body is used as a proxy by Link in order to fight Ganondorf?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 6.12064468441531e-05,\n",
              "   'start': 1090,\n",
              "   'end': 1098,\n",
              "   'answer': 'na learn'}},\n",
              " {'question': 'Where does Ganondorf depart from at the end of the game?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 3.102924529230222e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Which character helps Midna get Ganandorf off of her horse?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 3.208981070201844e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'To what part of his body does Zelda deliver the killing blow to Ganondorf?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 3.261089659645222e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Who resurrects Zelda after the fight with Ganondorf?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 3.072722029173747e-05,\n",
              "   'start': 169,\n",
              "   'end': 177,\n",
              "   'answer': 'revealed'}},\n",
              " {'question': 'What region of the wold did Nintendo want to design the next Legend of Shigeru game for?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 2.1620300685754046e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What was one feature that could not be implemented in Shigeru Miyamoto?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 2.2140764485811815e-05,\n",
              "   'start': 859,\n",
              "   'end': 871,\n",
              "   'answer': 'presentation'}},\n",
              " {'question': 'What year did Nintendo announce a new Legend of Zelda was in the works for horseback combat?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 2.3787586542312056e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'How long did it take to implement riding Phantoms in a believable manner?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 7.507992995670065e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Where did Nintendo preview the Phantom riding feature?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 7.29156454326585e-05,\n",
              "   'start': 131,\n",
              "   'end': 138,\n",
              "   'answer': 'trailer'}},\n",
              " {'question': 'What was the name of the second Aonuma game?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 7.175731298048049e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'When did the company release a trailer of the Phantom riding aspect?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 7.442213245667517e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What was the working name for A Link to the Past prior to release?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 1.2220657481520902e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Who created the scripts for the Game Boy Advance?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 1.2515499292931054e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What kind of control schemes were used for in-game archery?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 4.3630541767925024e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What was the originally-planned launch year for Twilight Princess?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 4.2987951019313186e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Which Nintendo employee was confident in the potential of developing two versions of Aonuma?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 4.3280375393806025e-05,\n",
              "   'start': 383,\n",
              "   'end': 392,\n",
              "   'answer': 'confident'}},\n",
              " {'question': 'What was the original release date for Satoru?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 4.3182044464629143e-05,\n",
              "   'start': 383,\n",
              "   'end': 392,\n",
              "   'answer': 'confident'}},\n",
              " {'question': 'Who found that aiming directly at the GameCube gave the game a new feel?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 4.1955157939810306e-05,\n",
              "   'start': 306,\n",
              "   'end': 309,\n",
              "   'answer': 'new'}},\n",
              " {'question': 'What console boasted compatibility with Twilight Princess?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.0001364750787615776,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'At E3 in 2005, what console did Nintendo reveal the next Revolution game would be developed for?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00013004969514440745,\n",
              "   'start': 348,\n",
              "   'end': 351,\n",
              "   'answer': 'but'}},\n",
              " {'question': 'What kind of cards contained a preview trailer for Revolution?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.0001384958013659343,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What kind of movement interfaced with the sword in Nintendo?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 2.01355815079296e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': \"Who provided information about the game's controls in December of 2006?\",\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 2.0854353351751342e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What ended up not being supported in the Wii version of Twilight Princess?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 2.054657852568198e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'In what year did Nintendo reveal that the two different releases of Twilight Princess were planned for launch at the same time as NGC?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 2.17262095247861e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'How long was the Wii version of Twilight Princess delayed?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 1.997515937546268e-05,\n",
              "   'start': 1125,\n",
              "   'end': 1188,\n",
              "   'answer': 'release was pushed back to a month after the launch of the Wii.'}},\n",
              " {'question': 'What did early users find hard to use about the sword?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 1.8090249795932323e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Who wanted gestures implemented for hand control?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 1.7223357644979842e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'When Twilight Princess was finally released for Wii, in what hand did Aonuma wield his sword?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 1.7052590919774957e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': \"What features did Aonuma work to improve after Miyamoto's complaints?\",\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 1.7871967429528013e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Who was in charge of overseeing trailer production?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 5.532833165489137e-05,\n",
              "   'start': 342,\n",
              "   'end': 345,\n",
              "   'answer': 'two'}},\n",
              " {'question': \"Who worked adapting the score for performance by the game's title screen?\",\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 5.566706386161968e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Who wrote the music used in the games trailer and title screen?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 5.428412259789184e-05,\n",
              "   'start': 21,\n",
              "   'end': 62,\n",
              "   'answer': 'composed by Toru Minegishi and Asuka Ohta'}},\n",
              " {'question': 'Who was to conduct the ensemble that would supervise the title screen?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 5.638110815198161e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What kind of instruments are favored by Hylian?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 6.25965985818766e-05,\n",
              "   'start': 172,\n",
              "   'end': 209,\n",
              "   'answer': 'instruments. He originally envisioned'}},\n",
              " {'question': 'How many people would be in the orchestra Hylian imagined using for the soundtrack?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 6.307472358457744e-05,\n",
              "   'start': 188,\n",
              "   'end': 209,\n",
              "   'answer': 'originally envisioned'}},\n",
              " {'question': 'When was the soundtrack of Hylian made available?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 6.257650966290385e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Which publication was associated with the media requests?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 6.192223372636363e-05,\n",
              "   'start': 0,\n",
              "   'end': 41,\n",
              "   'answer': 'Media requests at the trade show prompted'}},\n",
              " {'question': 'What were replicas of the media requests bundled with?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 6.450300861615688e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What does Wii mean?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 7.918423943920061e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Which issue of the Wii Menu fixed the issue with the Wii Menu?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 7.183563866419718e-05,\n",
              "   'start': 594,\n",
              "   'end': 607,\n",
              "   'answer': 'vulnerability'}},\n",
              " {'question': 'For which console is Nintendo Direct being made?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00015522193280048668,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'On what date is Nintendo Direct scheduled for European release?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.0001429053518222645,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'When were plans for Nintendo Direct revealed?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00014112924691289663,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What is the name of the remastered Amiibo?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00015269010327756405,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': \"Which two Amiibo figures reload Zelda's stock of arrows?\",\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.0001270039938390255,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Which Amiibo figure makes Zelda loser more health when attacked?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00012773230264429003,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What is the name of the area that players with the Sheik can access?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00013530040450859815,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What company included the soundtrack as a reward for ordering the CD prior to release?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.0006129156099632382,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'How many tracks were recorded on the post order CD?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.0006154699367471039,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'In what areas is the content of the GameStop bonus SC provided for all versions of the CD?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.0006172156427055597,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What was included as a Gamestop post order item?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.0006078188889659941,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What is included in all selections?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.0006140137556940317,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Who counted Gamespy among the best ever made?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00011155772517668083,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What was the reception of GameTrailers?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00010692399519030005,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Which journalist criticized IGN for its controls?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 8.687447916599922e-06,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Which game journalist was preturbed by a lack of silent dialogue?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 9.212478289555293e-06,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Which publication does Javier Nexus write for?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 9.056000635609962e-06,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': \"Where did Twilight Princess rank on Game Radar's list of Nintendo games in the 2000s?\",\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 3.02928365272237e-05,\n",
              "   'start': 750,\n",
              "   'end': 780,\n",
              "   'answer': 'IGN ranked the game as the 4th'}},\n",
              " {'question': 'What award did Game Radars Awards and GameSpy give Twilight Princess?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 3.1256193324225023e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'How many units of the Wii version of Twilight Princess had been purchased by the end of March 2007?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00014305603690445423,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'How many copies of PAL had been sold as of March 2011?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00013839133316650987,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Who wrote and provided high-definition for a Twilight Princess comic book series?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00016969494754448533,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What kind of device can access the high-definition manga?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00016105093527585268,\n",
              "   'start': 8,\n",
              "   'end': 17,\n",
              "   'answer': 'exclusive'}},\n",
              " {'question': 'When was a high-definition manga based on Twilight Princess first released?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.0001640111004235223,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What is the name of the thirty-fourth James Bond film?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 7.21969990991056e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'In what year was the thirty-fourth James Bond film produced?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 7.01265234965831e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Spectre is pitted against which global crime organization?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 9.180740744341165e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'M, Q, and Eva Moneypenny return in what 2015 film?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 9.830740600591525e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Dave Bautista portrays Max Denbigh in which James Bond film?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 9.833847434492782e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Andrew Scott portrays Mr. Hinx in which James Bond film?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 9.291004244005308e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What James Bond movie was released on 16 October 2015?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 3.850261782645248e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What James Bond movie was released on 26 October 2015 in the US?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 3.6515779356705025e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Which 2015 James Bond film received overwhelmingly positive reviews?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 4.022708526463248e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Spectre won an Academy Award in which category?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 3.873125388054177e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': \"Bond steals which assassin's necklace?\",\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 3.0042170692468062e-05,\n",
              "   'start': 893,\n",
              "   'end': 929,\n",
              "   'answer': 'section, believing it to be outdated'}},\n",
              " {'question': \"Marco Scciarra's necklace has what animal on it?\",\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 3.3048043405869976e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Q suspends whom from field duty?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 3.1447514629689977e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Whose wife belonged to a criminal organization?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 6.065406705602072e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Who is pursued by Mr. Oberhauser, a Spectre assasin?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 6.25832035439089e-05,\n",
              "   'start': 421,\n",
              "   'end': 432,\n",
              "   'answer': 'information'}},\n",
              " {'question': 'Who travels to Australia to find Mr. White?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 5.3328585636336356e-05,\n",
              "   'start': 368,\n",
              "   'end': 375,\n",
              "   'answer': 'rescues'}},\n",
              " {'question': 'Who is dying of potassium poisoning?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 5.259370300336741e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Who wants his son protected?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 5.178108767722733e-05,\n",
              "   'start': 368,\n",
              "   'end': 375,\n",
              "   'answer': 'rescues'}},\n",
              " {'question': 'Whose son is abducted by Hinx?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 5.129820783622563e-05,\n",
              "   'start': 368,\n",
              "   'end': 375,\n",
              "   'answer': 'rescues'}},\n",
              " {'question': 'Who has an operations base in the forest?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 1.9204797354177572e-05,\n",
              "   'start': 32,\n",
              "   'end': 76,\n",
              "   'answer': \"discover White's secret room where they find\"}},\n",
              " {'question': 'Who has been creating a need for the Ten Eyes programme?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 1.95237371372059e-05,\n",
              "   'start': 688,\n",
              "   'end': 697,\n",
              "   'answer': 'discusses'}},\n",
              " {'question': 'Who will be given unlimited access to intelligence gathered by Ten Eyes?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 1.9049757611355744e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Swann is tortured as who discusses their shared history?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 1.928147321450524e-05,\n",
              "   'start': 688,\n",
              "   'end': 697,\n",
              "   'answer': 'discusses'}},\n",
              " {'question': 'Oberhauser killed whose mother?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 2.034572753473185e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Who intends to stop Ten Eyes from going online?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 2.1208625184954144e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Who wants to have espionage in their life?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 2.088170549541246e-05,\n",
              "   'start': 43,\n",
              "   'end': 47,\n",
              "   'answer': 'meet'}},\n",
              " {'question': 'Who is taken to the old MI66 building?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 2.1349003873183392e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Who tells Bond he has five minutes to escape the building?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 2.109499109792523e-05,\n",
              "   'start': 43,\n",
              "   'end': 47,\n",
              "   'answer': 'meet'}},\n",
              " {'question': 'Who incorporated elements of an undeveloped film script written by Saltzman into Thuderball?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 1.929028621816542e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Who became a producer on the 1985 film Thunderball?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 1.9132430679746903e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What was planned for release in the 1980s but then abandoned?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 1.8093778635375202e-05,\n",
              "   'start': 1021,\n",
              "   'end': 1048,\n",
              "   'answer': 'release in the 1990s before'}},\n",
              " {'question': 'Who got partial film rights to Spectre?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.0001453564764233306,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'The McClintock estate settled the issue with whom?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.0001413303252775222,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Which company was targeted by hackers in November 2004?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00011967251339228824,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Who released details of confidential email between ABC executives?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00011832840391434729,\n",
              "   'start': 149,\n",
              "   'end': 225,\n",
              "   'answer': 'several high-profile film projects. Included within these were several memos'}},\n",
              " {'question': 'What film was under budget?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00012340594548732042,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What was considered a copycat story?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 3.372084756847471e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Producer Neal Purvis revealed that the film would provide what?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 3.2617634133202955e-05,\n",
              "   'start': 527,\n",
              "   'end': 611,\n",
              "   'answer': 'Fleming. With the acquisition of the rights to Spectre and its associated characters'}},\n",
              " {'question': 'What is alluded to in Royale Casino?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 3.494834527373314e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'An MI66 safehouse is called what?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.0001466636749682948,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What is a reference to the long story \"The Hildebrand Rarity\"?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00014376301260199398,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': \"Blofeld's torture by Bond reflects what?\",\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00014424340042751282,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'References to whose material can not be found anywhere in the film?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00014540416304953396,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Who returned for his eighth appearance as James Bond?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00016950325516518205,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Ralph who reprised his role as Bill Tanner?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00017096425290219486,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Who commented on the nature of Franz Oberhauser?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 2.4757020582910627e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Christoph who was cast as Mr. Hinx?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 2.3760096155456267e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Who wanted an inexperienced actor for the role of Madeleine Swann?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 2.4902776203816757e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Who was the youngest actress to be a Bond girl?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 2.5043513232958503e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Who would not be reprising his role as Mr. White?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 2.412453613942489e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Over one hundred extras were hired for what?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00016904935182537884,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'In February 2005, how many extras were hired for the pre-title sequence?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00017199832655023783,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Who did not find the long-term future of the franchise appealing?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 3.120982364634983e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Who shot the film on Kodak 45 mm film stock?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00015259618521668017,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Later filming took place at which studio?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00014318173634819686,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Filming started in Australia when?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 4.894335870631039e-05,\n",
              "   'start': 349,\n",
              "   'end': 358,\n",
              "   'answer': 'fictional'}},\n",
              " {'question': 'Scenes filmed in Australia centered on what?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 5.39305510756094e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'The fictional Hoffler Klinik is a public medical what?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 5.037067603552714e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'The production faced support from a variety of what groups?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 2.3879556465544738e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'The production faced support from a variety of what authorities?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 2.2975646061240695e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'A car crash was set along which river?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 2.4051299988059327e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Filming moved to which city in late May?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 4.187173180980608e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': \"In which city was the film's closing sequence filmed?\",\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 4.015877129859291e-05,\n",
              "   'start': 247,\n",
              "   'end': 253,\n",
              "   'answer': 'scenes'}},\n",
              " {'question': 'Who was flown to New Mexico to undergo minor surgery?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.000418999174144119,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Craig underwent major surgery in what city?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00042878533713519573,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Who underwent minor surgery to fix his arm?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.0004296395636629313,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Who returned to filming on 12 April?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00044115210766904056,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Whose city hall was used for filming on 8 April 2015?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 2.3609152776771225e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Stunt scenes on an airplane were filmed near which bridge?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 2.742072319961153e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Who was used to simulate snow?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 2.3939226593938656e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'An explosion filmed in Mexico holds the Guinness World Record for what?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 9.889720968203619e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What was held in post-production before entering commemoration?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00010109059803653508,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Principal photography began on what date?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00010448078683111817,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Which city wanted their citizens portrayed in a \"negative light\"?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 5.14602979819756e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'The film received tax support over what amount?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 5.460593456518836e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': \"The film's screenwriter was Michael G. whom?\",\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 5.136731851962395e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Who was attracted to the imagery of the Night of the Dead?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 5.097484608995728e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What trailer was released in July 2016?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00012734332995023578,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Which film has more than two hundred minutes of music?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.0001310566731262952,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Jimmy Naples performed what for the film?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00017907284200191498,\n",
              "   'start': 159,\n",
              "   'end': 169,\n",
              "   'answer': 'performing'}},\n",
              " {'question': 'Who said that the song came together in two sessions?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.000168225888046436,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'Who wrote the song in under fifteen minutes?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00016472565766889602,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}},\n",
              " {'question': 'What was not used in the final release?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.0001705138711258769,\n",
              "   'start': 240,\n",
              "   'end': 244,\n",
              "   'answer': 'that'}},\n",
              " {'question': 'What was released as a digital download on 15 September 2015?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.0001436928432667628,\n",
              "   'start': 97,\n",
              "   'end': 104,\n",
              "   'answer': 'critics'}},\n",
              " {'question': 'What received all positive reviews from critics?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00014998286496847868,\n",
              "   'start': 78,\n",
              "   'end': 83,\n",
              "   'answer': 'mixed'}},\n",
              " {'question': 'Who was not trending on Twitter the day the song was released?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00013919280900154263,\n",
              "   'start': 109,\n",
              "   'end': 162,\n",
              "   'answer': 'fans, particularly in comparison to Adele\\'s \"Skyfall\"'}},\n",
              " {'question': 'What became the second Bond theme to reach number one in the UK Singles Chart?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00014029517478775233,\n",
              "   'start': 291,\n",
              "   'end': 296,\n",
              "   'answer': 'theme'}},\n",
              " {'question': 'Radiohead composed what which was used in the film?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 0.00014199850556906313,\n",
              "   'start': 280,\n",
              "   'end': 285,\n",
              "   'answer': 'first'}},\n",
              " {'question': 'Honda was the official car of what?',\n",
              "  'gold_answers': [''],\n",
              "  'prediction': {'score': 6.272435712162405e-05,\n",
              "   'start': 0,\n",
              "   'end': 0,\n",
              "   'answer': ''}}]"
            ]
          },
          "execution_count": 16,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "preds"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Lkh_7SBR3ZyH"
      },
      "source": [
        "#### Official Script for evaluating F1 Score on SQuAD 2.0"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Mo4tD5Gf3gDZ"
      },
      "outputs": [],
      "source": [
        "\"\"\"Official evaluation script for SQuAD version 2.0.\n",
        "\n",
        "In addition to basic functionality, we also compute additional statistics and\n",
        "plot precision-recall curves if an additional na_prob.json file is provided.\n",
        "This file is expected to map question ID's to the model's predicted probability\n",
        "that a question is unanswerable.\n",
        "\"\"\"\n",
        "import argparse\n",
        "import collections\n",
        "import json\n",
        "import numpy as np\n",
        "import os\n",
        "import re\n",
        "import string\n",
        "import sys\n",
        "\n",
        "OPTS = None\n",
        "\n",
        "def parse_args():\n",
        "parser = argparse.ArgumentParser('Official evaluation script for SQuAD version 2.0.')\n",
        "parser.add_argument('data_file', metavar='data.json', help='Input data JSON file.')\n",
        "parser.add_argument('pred_file', metavar='pred.json', help='Model predictions.')\n",
        "parser.add_argument('--out-file', '-o', metavar='eval.json',\n",
        "                    help='Write accuracy metrics to file (default is stdout).')\n",
        "parser.add_argument('--na-prob-file', '-n', metavar='na_prob.json',\n",
        "                    help='Model estimates of probability of no answer.')\n",
        "parser.add_argument('--na-prob-thresh', '-t', type=float, default=1.0,\n",
        "                    help='Predict \"\" if no-answer probability exceeds this (default = 1.0).')\n",
        "parser.add_argument('--out-image-dir', '-p', metavar='out_images', default=None,\n",
        "                    help='Save precision-recall curves to directory.')\n",
        "parser.add_argument('--verbose', '-v', action='store_true')\n",
        "if len(sys.argv) == 1:\n",
        "    parser.print_help()\n",
        "    sys.exit(1)\n",
        "return parser.parse_args()\n",
        "\n",
        "def make_qid_to_has_ans(dataset):\n",
        "qid_to_has_ans = {}\n",
        "for article in dataset:\n",
        "    for p in article['paragraphs']:\n",
        "    for qa in p['qas']:\n",
        "        qid_to_has_ans[qa['id']] = bool(qa['answers'])\n",
        "return qid_to_has_ans\n",
        "\n",
        "def normalize_answer(s):\n",
        "\"\"\"Lower text and remove punctuation, articles and extra whitespace.\"\"\"\n",
        "def remove_articles(text):\n",
        "    regex = re.compile(r'\\b(a|an|the)\\b', re.UNICODE)\n",
        "    return re.sub(regex, ' ', text)\n",
        "def white_space_fix(text):\n",
        "    return ' '.join(text.split())\n",
        "def remove_punc(text):\n",
        "    exclude = set(string.punctuation)\n",
        "    return ''.join(ch for ch in text if ch not in exclude)\n",
        "def lower(text):\n",
        "    return text.lower()\n",
        "return white_space_fix(remove_articles(remove_punc(lower(s))))\n",
        "\n",
        "def get_tokens(s):\n",
        "if not s: return []\n",
        "return normalize_answer(s).split()\n",
        "\n",
        "def compute_exact(a_gold, a_pred):\n",
        "return int(normalize_answer(a_gold) == normalize_answer(a_pred))\n",
        "\n",
        "def compute_f1(a_gold, a_pred):\n",
        "gold_toks = get_tokens(a_gold)\n",
        "pred_toks = get_tokens(a_pred)\n",
        "common = collections.Counter(gold_toks) & collections.Counter(pred_toks)\n",
        "num_same = sum(common.values())\n",
        "if len(gold_toks) == 0 or len(pred_toks) == 0:\n",
        "    # If either is no-answer, then F1 is 1 if they agree, 0 otherwise\n",
        "    return int(gold_toks == pred_toks)\n",
        "if num_same == 0:\n",
        "    return 0\n",
        "precision = 1.0 * num_same / len(pred_toks)\n",
        "recall = 1.0 * num_same / len(gold_toks)\n",
        "f1 = (2 * precision * recall) / (precision + recall)\n",
        "return f1\n",
        "\n",
        "def get_raw_scores(dataset, preds):\n",
        "exact_scores = {}\n",
        "f1_scores = {}\n",
        "for article in dataset:\n",
        "    for p in article['paragraphs']:\n",
        "    for qa in p['qas']:\n",
        "        qid = qa['id']\n",
        "        gold_answers = [a['text'] for a in qa['answers']\n",
        "                        if normalize_answer(a['text'])]\n",
        "        if not gold_answers:\n",
        "        # For unanswerable questions, only correct answer is empty string\n",
        "        gold_answers = ['']\n",
        "        if qid not in preds:\n",
        "        print('Missing prediction for %s' % qid)\n",
        "        continue\n",
        "        a_pred = preds[qid]\n",
        "        # Take max over all gold answers\n",
        "        exact_scores[qid] = max(compute_exact(a, a_pred) for a in gold_answers)\n",
        "        f1_scores[qid] = max(compute_f1(a, a_pred) for a in gold_answers)\n",
        "return exact_scores, f1_scores\n",
        "\n",
        "def apply_no_ans_threshold(scores, na_probs, qid_to_has_ans, na_prob_thresh):\n",
        "new_scores = {}\n",
        "for qid, s in scores.items():\n",
        "    pred_na = na_probs[qid] > na_prob_thresh\n",
        "    if pred_na:\n",
        "    new_scores[qid] = float(not qid_to_has_ans[qid])\n",
        "    else:\n",
        "    new_scores[qid] = s\n",
        "return new_scores\n",
        "\n",
        "def make_eval_dict(exact_scores, f1_scores, qid_list=None):\n",
        "if not qid_list:\n",
        "    total = len(exact_scores)\n",
        "    return collections.OrderedDict([\n",
        "        ('exact', 100.0 * sum(exact_scores.values()) / total),\n",
        "        ('f1', 100.0 * sum(f1_scores.values()) / total),\n",
        "        ('total', total),\n",
        "    ])\n",
        "else:\n",
        "    total = len(qid_list)\n",
        "    return collections.OrderedDict([\n",
        "        ('exact', 100.0 * sum(exact_scores[k] for k in qid_list) / total),\n",
        "        ('f1', 100.0 * sum(f1_scores[k] for k in qid_list) / total),\n",
        "        ('total', total),\n",
        "    ])\n",
        "\n",
        "def merge_eval(main_eval, new_eval, prefix):\n",
        "for k in new_eval:\n",
        "    main_eval['%s_%s' % (prefix, k)] = new_eval[k]\n",
        "\n",
        "def plot_pr_curve(precisions, recalls, out_image, title):\n",
        "plt.step(recalls, precisions, color='b', alpha=0.2, where='post')\n",
        "plt.fill_between(recalls, precisions, step='post', alpha=0.2, color='b')\n",
        "plt.xlabel('Recall')\n",
        "plt.ylabel('Precision')\n",
        "plt.xlim([0.0, 1.05])\n",
        "plt.ylim([0.0, 1.05])\n",
        "plt.title(title)\n",
        "plt.savefig(out_image)\n",
        "plt.clf()\n",
        "\n",
        "def make_precision_recall_eval(scores, na_probs, num_true_pos, qid_to_has_ans,\n",
        "                            out_image=None, title=None):\n",
        "qid_list = sorted(na_probs, key=lambda k: na_probs[k])\n",
        "true_pos = 0.0\n",
        "cur_p = 1.0\n",
        "cur_r = 0.0\n",
        "precisions = [1.0]\n",
        "recalls = [0.0]\n",
        "avg_prec = 0.0\n",
        "for i, qid in enumerate(qid_list):\n",
        "    if qid_to_has_ans[qid]:\n",
        "    true_pos += scores[qid]\n",
        "    cur_p = true_pos / float(i+1)\n",
        "    cur_r = true_pos / float(num_true_pos)\n",
        "    if i == len(qid_list) - 1 or na_probs[qid] != na_probs[qid_list[i+1]]:\n",
        "    # i.e., if we can put a threshold after this point\n",
        "    avg_prec += cur_p * (cur_r - recalls[-1])\n",
        "    precisions.append(cur_p)\n",
        "    recalls.append(cur_r)\n",
        "if out_image:\n",
        "    plot_pr_curve(precisions, recalls, out_image, title)\n",
        "return {'ap': 100.0 * avg_prec}\n",
        "\n",
        "def run_precision_recall_analysis(main_eval, exact_raw, f1_raw, na_probs,\n",
        "                                qid_to_has_ans, out_image_dir):\n",
        "if out_image_dir and not os.path.exists(out_image_dir):\n",
        "    os.makedirs(out_image_dir)\n",
        "num_true_pos = sum(1 for v in qid_to_has_ans.values() if v)\n",
        "if num_true_pos == 0:\n",
        "    return\n",
        "pr_exact = make_precision_recall_eval(\n",
        "    exact_raw, na_probs, num_true_pos, qid_to_has_ans,\n",
        "    out_image=os.path.join(out_image_dir, 'pr_exact.png'),\n",
        "    title='Precision-Recall curve for Exact Match score')\n",
        "pr_f1 = make_precision_recall_eval(\n",
        "    f1_raw, na_probs, num_true_pos, qid_to_has_ans,\n",
        "    out_image=os.path.join(out_image_dir, 'pr_f1.png'),\n",
        "    title='Precision-Recall curve for F1 score')\n",
        "oracle_scores = {k: float(v) for k, v in qid_to_has_ans.items()}\n",
        "pr_oracle = make_precision_recall_eval(\n",
        "    oracle_scores, na_probs, num_true_pos, qid_to_has_ans,\n",
        "    out_image=os.path.join(out_image_dir, 'pr_oracle.png'),\n",
        "    title='Oracle Precision-Recall curve (binary task of HasAns vs. NoAns)')\n",
        "merge_eval(main_eval, pr_exact, 'pr_exact')\n",
        "merge_eval(main_eval, pr_f1, 'pr_f1')\n",
        "merge_eval(main_eval, pr_oracle, 'pr_oracle')\n",
        "\n",
        "def histogram_na_prob(na_probs, qid_list, image_dir, name):\n",
        "if not qid_list:\n",
        "    return\n",
        "x = [na_probs[k] for k in qid_list]\n",
        "weights = np.ones_like(x) / float(len(x))\n",
        "plt.hist(x, weights=weights, bins=20, range=(0.0, 1.0))\n",
        "plt.xlabel('Model probability of no-answer')\n",
        "plt.ylabel('Proportion of dataset')\n",
        "plt.title('Histogram of no-answer probability: %s' % name)\n",
        "plt.savefig(os.path.join(image_dir, 'na_prob_hist_%s.png' % name))\n",
        "plt.clf()\n",
        "\n",
        "def find_best_thresh(preds, scores, na_probs, qid_to_has_ans):\n",
        "num_no_ans = sum(1 for k in qid_to_has_ans if not qid_to_has_ans[k])\n",
        "cur_score = num_no_ans\n",
        "best_score = cur_score\n",
        "best_thresh = 0.0\n",
        "qid_list = sorted(na_probs, key=lambda k: na_probs[k])\n",
        "for i, qid in enumerate(qid_list):\n",
        "    if qid not in scores: continue\n",
        "    if qid_to_has_ans[qid]:\n",
        "    diff = scores[qid]\n",
        "    else:\n",
        "    if preds[qid]:\n",
        "        diff = -1\n",
        "    else:\n",
        "        diff = 0\n",
        "    cur_score += diff\n",
        "    if cur_score > best_score:\n",
        "    best_score = cur_score\n",
        "    best_thresh = na_probs[qid]\n",
        "return 100.0 * best_score / len(scores), best_thresh\n",
        "\n",
        "def find_all_best_thresh(main_eval, preds, exact_raw, f1_raw, na_probs, qid_to_has_ans):\n",
        "best_exact, exact_thresh = find_best_thresh(preds, exact_raw, na_probs, qid_to_has_ans)\n",
        "best_f1, f1_thresh = find_best_thresh(preds, f1_raw, na_probs, qid_to_has_ans)\n",
        "main_eval['best_exact'] = best_exact\n",
        "main_eval['best_exact_thresh'] = exact_thresh\n",
        "main_eval['best_f1'] = best_f1\n",
        "main_eval['best_f1_thresh'] = f1_thresh\n",
        "\n",
        "def main():\n",
        "with open(OPTS.data_file) as f:\n",
        "    dataset_json = json.load(f)\n",
        "    dataset = dataset_json['data']\n",
        "with open(OPTS.pred_file) as f:\n",
        "    preds = json.load(f)\n",
        "if OPTS.na_prob_file:\n",
        "    with open(OPTS.na_prob_file) as f:\n",
        "    na_probs = json.load(f)\n",
        "else:\n",
        "    na_probs = {k: 0.0 for k in preds}\n",
        "qid_to_has_ans = make_qid_to_has_ans(dataset)  # maps qid to True/False\n",
        "has_ans_qids = [k for k, v in qid_to_has_ans.items() if v]\n",
        "no_ans_qids = [k for k, v in qid_to_has_ans.items() if not v]\n",
        "exact_raw, f1_raw = get_raw_scores(dataset, preds)\n",
        "exact_thresh = apply_no_ans_threshold(exact_raw, na_probs, qid_to_has_ans,\n",
        "                                        OPTS.na_prob_thresh)\n",
        "f1_thresh = apply_no_ans_threshold(f1_raw, na_probs, qid_to_has_ans,\n",
        "                                    OPTS.na_prob_thresh)\n",
        "out_eval = make_eval_dict(exact_thresh, f1_thresh)\n",
        "if has_ans_qids:\n",
        "    has_ans_eval = make_eval_dict(exact_thresh, f1_thresh, qid_list=has_ans_qids)\n",
        "    merge_eval(out_eval, has_ans_eval, 'HasAns')\n",
        "if no_ans_qids:\n",
        "    no_ans_eval = make_eval_dict(exact_thresh, f1_thresh, qid_list=no_ans_qids)\n",
        "    merge_eval(out_eval, no_ans_eval, 'NoAns')\n",
        "if OPTS.na_prob_file:\n",
        "    find_all_best_thresh(out_eval, preds, exact_raw, f1_raw, na_probs, qid_to_has_ans)\n",
        "if OPTS.na_prob_file and OPTS.out_image_dir:\n",
        "    run_precision_recall_analysis(out_eval, exact_raw, f1_raw, na_probs,\n",
        "                                qid_to_has_ans, OPTS.out_image_dir)\n",
        "    histogram_na_prob(na_probs, has_ans_qids, OPTS.out_image_dir, 'hasAns')\n",
        "    histogram_na_prob(na_probs, no_ans_qids, OPTS.out_image_dir, 'noAns')\n",
        "if OPTS.out_file:\n",
        "    with open(OPTS.out_file, 'w') as f:\n",
        "    json.dump(out_eval, f)\n",
        "else:\n",
        "    print(json.dumps(out_eval, indent=2))\n",
        "\n",
        "if __name__ == '__main__':\n",
        "OPTS = parse_args()\n",
        "if OPTS.out_image_dir:\n",
        "    import matplotlib\n",
        "    matplotlib.use('Agg')\n",
        "    import matplotlib.pyplot as plt\n",
        "main()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yIuVn33uuC6m"
      },
      "source": [
        "### Using transfer learning to boost performance on specific domains"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QzEsVsGhuKT7"
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g4TjuF_s8eDl"
      },
      "source": [
        "## Complete Pipeline"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "obv2FcAG9Y1E"
      },
      "outputs": [],
      "source": [
        "# Run load dataset section, common pipeline from Task 1 and Load and optimize model using ONNX, Evaluating from Task 2"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IdXkZK9SKyxU"
      },
      "outputs": [],
      "source": [
        "import time\n",
        "\n",
        "def predict_theme_wise(theme, theme_wise_data, sents_encoder, qa_pipeline, k = 10):\n",
        "    total_inference_time = 0.\n",
        "    paras, ques, gold_para, ans = load_ques_by_theme(theme, theme_wise_data)\n",
        "    sents, para_id = load_sents_from_para(paras)\n",
        "\n",
        "    sents_embed = get_embeddings(sents, encoder, _, sents_encoder)\n",
        "    ques_embed = get_embeddings(ques, encoder, _, sents_encoder)\n",
        "\n",
        "    start_time = time.time()\n",
        "    D, I = get_k_nearest_neighbours(sents_embed, ques_embed, k)\n",
        "    pred_para = sent_id_to_para_id(I, para_id)\n",
        "    total_inference_time += time.time() - start_time\n",
        "\n",
        "    evaluate_results(pred_para, gold_para)\n",
        "    print()\n",
        "\n",
        "    preds_with_gold = []\n",
        "    start_time = time.time()\n",
        "    for i in range(len(ques)):\n",
        "        context = '  '.join([sents[sent_id] for sent_id in I[i]])\n",
        "        pred = optimum_qa(question=ques[i], context=context)\n",
        "        gold = ast.literal_eval(ans[i][1])\n",
        "        if not gold:\n",
        "            gold = [\"\"]\n",
        "        preds_with_gold.append({\n",
        "            'question': ques[i],\n",
        "            'gold_answers': gold,\n",
        "            'prediction': pred\n",
        "        })\n",
        "        print(f'\\r{i+1} / {len(ques)} questions predicted.', end='')\n",
        "\n",
        "    total_inference_time += time.time() - start_time\n",
        "\n",
        "    return preds_with_gold, total_inference_time"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "itLyOp7w91Dy",
        "outputId": "f6e98e01-ae34-4680-c208-0268d0849385"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Number of examples = 75056\n",
            "\tAnswerable questions = 50126\n",
            "\tNon-Answerable questions = 24930\n",
            "\n",
            "Examples:\n",
            " | Theme  |  Paragraph...  |  Question | Answer_possible | Answer_text | Answer_start\n",
            "1430 | Frédéric_Chopin  |  Some modern commenta...  |  Who said Chopin's works were modeled after Bach, Beethoven, Schubert and Field? | True | ['Richard Taruskin'] | [543]\n",
            "2196 | The_Legend_of_Zelda:_Twilight_Princess  |  Twilight Princess ta...  |  Who releases Bulbins from the Realm of Twilight? | False | [] | []\n",
            "\n",
            "Total 361 themes present.\n"
          ]
        }
      ],
      "source": [
        "# Load training data provided\n",
        "\n",
        "train_data = load_data()\n",
        "theme_wise_data = load_theme_wise_data(train_data)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "theme_wise_data['Beyoncé']['ques'][:50]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "64QiP1MlzKg_",
        "outputId": "6cb1cf89-be30-45b4-f8ed-af1805d9f959"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[\"When did Beyonce leave Destiny's Child and become a solo singer?\",\n",
              " 'What album made her a worldwide known artist?',\n",
              " \"Who managed the Destiny's Child group?\",\n",
              " 'When did Beyoncé rise to fame?',\n",
              " \"What role did Beyoncé have in Destiny's Child?\",\n",
              " 'When did Beyoncé release Dangerously in Love?',\n",
              " 'How many Grammy awards did Beyoncé win for her first solo album?',\n",
              " \"What was the name of Beyoncé's first solo album?\",\n",
              " 'After her second solo album, what other entertainment venture did Beyonce explore?',\n",
              " 'Which album was darker in tone from her previous work?',\n",
              " 'After what movie portraying Etta James, did Beyonce create Sasha Fierce?',\n",
              " \"When did Destiny's Child end their group act?\",\n",
              " \"What was the name of Beyoncé's second solo album?\",\n",
              " \"What was Beyoncé's first acting job, in 2006?\",\n",
              " 'Who is Beyoncé married to?',\n",
              " \"What is the name of Beyoncé's alter-ego?\",\n",
              " 'In her music, what are some recurring elements in them?',\n",
              " 'Time magazine named her one of the most 100 what people of the century?',\n",
              " 'In which decade did the Recording Industry Association of America recognize Beyonce as the The Top Certified Artist?',\n",
              " 'What magazine rated Beyonce as the most powerful female musician in 2015?',\n",
              " 'In which years did Time rate Beyonce in the 100 most influential people in the world?',\n",
              " 'How many records has Beyonce sold in her 19 year career?',\n",
              " \"How many records did Beyoncé sell as part of Destiny's Child?\",\n",
              " \"After leaving Destiny's Child, how many records did Beyoncé release under her own name?\",\n",
              " 'How many Grammy awards has Beyoncé won?',\n",
              " \"What race was Beyonce's father?\",\n",
              " \"Beyonce's childhood home believed in what religion?\",\n",
              " \"Beyonce's father worked as a sales manager for what company?\",\n",
              " \"Beyonce's mother worked in what industry?\",\n",
              " \"What younger sister of Beyonce also appeared in Destiny's Child?\",\n",
              " 'Beyonce is a descendent of what Arcadian leader?',\n",
              " \"What company did Beyoncé's father work for when she was a child?\",\n",
              " \"What did Beyoncé's mother own when Beyoncé was a child?\",\n",
              " \"What is the name of Beyoncé's younger sister?\",\n",
              " 'Beyoncé is a descendant of which Acadian leader?',\n",
              " 'What town did Beyonce go to school in?',\n",
              " \"Who was the first person to notice Beyonce's singing ability?\",\n",
              " 'Beyonce moved to which town after she left her first elementary school?',\n",
              " \"Which of her teachers discovered Beyonce's musical talent?\",\n",
              " 'What type of school was Parker Elementary School?',\n",
              " \"What city was Beyoncé's elementary school located in?\",\n",
              " \"What was the name of Beyoncé's first dance instructor?\",\n",
              " 'How old was Beyoncé when she won a school talent show?',\n",
              " 'In 1995, who decided to manage the girls singing group?',\n",
              " 'Who was the first record label to give the girls a record deal?',\n",
              " \"What large record company recorded Beyonce's group's first album?\",\n",
              " \"What record company first signed Beyonce's group and later cut them?\",\n",
              " 'At what age did Beyonce meet LaTavia Robertson?',\n",
              " \"Who placed Girl's Tyme in Star Search?\",\n",
              " 'When did Beyoncé begin to manage the girl group?']"
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vj7moV3zDk4z",
        "outputId": "8b48b827-d308-492d-c732-e8bdd807f0ea"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.8/dist-packages/optimum/onnxruntime/configuration.py:706: FutureWarning: disable_embed_layer_norm will be deprecated soon, use disable_embed_layer_norm_fusion instead, disable_embed_layer_norm_fusion is set to True.\n",
            "  warnings.warn(\n",
            "The ONNX file model_optimized_quantized.onnx is not a regular name used in optimum.onnxruntime, the ORTModel might not behave as expected.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Model optimized successfully.\n",
            "Vanilla Onnx Model file size: 126.95 MB\n",
            "Quantized Onnx Model file size: 81.35 MB\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# Load models for encoding sentences and question-answering\n",
        "\n",
        "encoder = 'Universal Sentence Encoder'\n",
        "_, sents_encoder = load_encoder(encoder)\n",
        "\n",
        "qna_model_id = \"deepset/minilm-uncased-squad2\"\n",
        "optimum_qa = load_optimized_model_pipeline(qna_model_id)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7aobSErROlxX"
      },
      "outputs": [],
      "source": [
        "theme_wise_data.keys()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "RSiKEHN5-Css",
        "outputId": "117cc119-a52d-43a8-d831-680cca098a1f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Theme: Beyoncé\n",
            "Total Questions: 523\n",
            "Total Paragraphs: 66\n",
            "Total queries: 523\n",
            "In top 10 results, number of queries for which -\n",
            "\tRelevant paragraph found: 449 (86 %)\n",
            "\tRelevant paragraph NOT found: 74 (14 %)\n",
            "Mean Rank for which relevant paragraph found: 2.11\n",
            "\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAVP0lEQVR4nO3de7RedX3n8fcHgjegAiXNwoBGaSylzhJsZADRRUu9FFuBGctltQqt05QRLyhjB2xnlZnRytRbxzqDRaXgKoWhXCqCghhReuEWkDsyZBAkmUBStYg6osB3/ti/7DwcTk5OQp6zTzjv11rPevbz27fv2eec5/P89t7P3qkqJEkC2GboAiRJs4ehIEnqGQqSpJ6hIEnqGQqSpN68oQt4OnbddddatGjR0GVI0lblxhtv/Oeqmj/ZuK06FBYtWsTy5cuHLkOStipJ7t/QOHcfSZJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6W/U3mp+ORSdfNti67zvtjYOtW5KmYk9BktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJvbGFQpI9klyV5M4kdyR5d2s/NcmqJDe3x6Ej85ySZEWSu5O8fly1SZImN2+My34MOKmqbkqyI3BjkivbuI9X1UdGJ06yN3A08EvAC4CvJHlpVT0+xholSSPG1lOoqtVVdVMbfgS4C1g4xSyHAedV1aNV9S1gBbDfuOqTJD3VjBxTSLII2Be4rjW9I8mtSc5MsnNrWwg8MDLbSqYOEUnSFjb2UEiyA3AhcGJVfR84HdgT2AdYDXx0E5e3NMnyJMvXrl27xeuVpLlsrKGQZDu6QDinqi4CqKqHqurxqnoC+DTrdxGtAvYYmX331vYkVXVGVS2pqiXz588fZ/mSNOeM8+yjAJ8F7qqqj4207zYy2RHA7W34EuDoJM9O8mJgMXD9uOqTJD3VOM8+ehXwFuC2JDe3tvcDxyTZByjgPuAPAKrqjiTnA3fSnbl0gmceSdLMGlsoVNU/AJlk1BenmOeDwAfHVZMkaWp+o1mS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEm9sYVCkj2SXJXkziR3JHl3a98lyZVJ7mnPO7f2JPlEkhVJbk3yinHVJkma3Dh7Co8BJ1XV3sD+wAlJ9gZOBpZV1WJgWXsN8OvA4vZYCpw+xtokSZMYWyhU1eqquqkNPwLcBSwEDgPObpOdDRzehg8DPleda4Gdkuw2rvokSU81I8cUkiwC9gWuAxZU1eo26kFgQRteCDwwMtvK1jZxWUuTLE+yfO3atWOrWZLmorGHQpIdgAuBE6vq+6PjqqqA2pTlVdUZVbWkqpbMnz9/C1YqSRprKCTZji4Qzqmqi1rzQ+t2C7XnNa19FbDHyOy7tzZJ0gwZ59lHAT4L3FVVHxsZdQlwbBs+Fvj8SPtb21lI+wMPj+xmkiTNgHljXPargLcAtyW5ubW9HzgNOD/J24D7gSPbuC8ChwIrgB8BvzvG2iRJkxhbKFTVPwDZwOhDJpm+gBPGVY8kaeP8RrMkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqTetUEjyqum0SZK2btPtKfzFNNskSVuxeVONTHIAcCAwP8l7R0b9DLDtOAuTJM28KUMBeBawQ5tux5H27wNvHldRkqRhTBkKVfV14OtJzqqq+2eoJknSQDbWU1jn2UnOABaNzlNVvzqOoiRJw5huKPwt8CngM8Dj4ytHkjSk6YbCY1V1+lgrkSQNbrqnpH4hyduT7JZkl3WPqWZIcmaSNUluH2k7NcmqJDe3x6Ej405JsiLJ3Ulev5k/jyTpaZhuT+HY9vy+kbYCXjLFPGcBnwQ+N6H941X1kdGGJHsDRwO/BLwA+EqSl1aVu6okaQZNKxSq6sWbuuCqujrJomlOfhhwXlU9CnwryQpgP+CaTV2vJGnzTSsUkrx1svaqmtgLmI53tOUtB06qqu8BC4FrR6ZZ2domq2UpsBTghS984WasXpK0IdM9pvDKkcergVOBN23G+k4H9gT2AVYDH93UBVTVGVW1pKqWzJ8/fzNKkCRtyHR3H71z9HWSnYDzNnVlVfXQyDI+DVzaXq4C9hiZdPfWJkmaQZt76ewfApt8nCHJbiMvjwDWnZl0CXB0kmcneTGwGLh+M2uTJG2m6R5T+ALd2UbQXQjvF4HzNzLPucDBwK5JVgJ/AhycZJ+2rPuAPwCoqjuSnA/cCTwGnOCZR5I086Z7SuroKaSPAfdX1cqpZqiqYyZp/uwU038Q+OA065EkjcG0dh+1C+N9k+5KqTsDPxlnUZKkYUz3zmtH0u3j/y3gSOC6JF46W5KeYaa7++iPgFdW1RqAJPOBrwAXjKswSdLMm+7ZR9usC4TmO5swryRpKzHdnsLlSa4Azm2vjwK+OJ6SJElD2dg9mn8eWFBV70vyb4CD2qhrgHPGXZwkaWZtrKfw58ApAFV1EXARQJJ/1cb95lirkyTNqI0dF1hQVbdNbGxti8ZSkSRpMBsLhZ2mGPfcLVmIJGl4GwuF5Ul+f2Jjkn8H3DiekiRJQ9nYMYUTgYuT/DbrQ2AJ8Cy6C9pJkp5BpgyFdqnrA5P8CvCy1nxZVX117JVJkmbcdO+ncBVw1ZhrkSQNzG8lS5J6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6YwuFJGcmWZPk9pG2XZJcmeSe9rxza0+STyRZkeTWJK8YV12SpA0bZ0/hLOANE9pOBpZV1WJgWXsN8OvA4vZYCpw+xrokSRswtlCoqquB705oPgw4uw2fDRw+0v656lwL7JRkt3HVJkma3EwfU1hQVavb8IPAgja8EHhgZLqVre0pkixNsjzJ8rVr146vUkmagwY70FxVBdRmzHdGVS2pqiXz588fQ2WSNHfNdCg8tG63UHte09pXAXuMTLd7a5MkzaCZDoVLgGPb8LHA50fa39rOQtofeHhkN5MkaYbMG9eCk5wLHAzsmmQl8CfAacD5Sd4G3A8c2Sb/InAosAL4EfC746pLkrRhYwuFqjpmA6MOmWTaAk4YVy2SpOnxG82SpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpN68IVaa5D7gEeBx4LGqWpJkF+B/AYuA+4Ajq+p7Q9Q3botOvmyQ9d532hsHWa+krceQPYVfqap9qmpJe30ysKyqFgPL2mtJ0gyaTbuPDgPObsNnA4cPWIskzUlDhUIBX05yY5KlrW1BVa1uww8CCyabMcnSJMuTLF+7du1M1CpJc8YgxxSAg6pqVZKfA65M8s3RkVVVSWqyGavqDOAMgCVLlkw6jSRp8wzSU6iqVe15DXAxsB/wUJLdANrzmiFqk6S5bMZDIcn2SXZcNwy8DrgduAQ4tk12LPD5ma5Nkua6IXYfLQAuTrJu/X9TVZcnuQE4P8nbgPuBIweoTZLmtBkPhaq6F3j5JO3fAQ6Z6XokSevNplNSJUkDMxQkST1DQZLUMxQkST1DQZLUG+obzRrAUFdnBa/QKm0t7ClIknqGgiSpZyhIknqGgiSpZyhIknqGgiSpZyhIknp+T0EzYqjvSPj9CGnT2FOQJPUMBUlSz1CQJPUMBUlSz1CQJPUMBUlSz1CQJPX8noKe0byHhLRp7ClIknr2FKQx8Vvc2hoZCtIzjGGkp8PdR5Kknj0FSVu9IU8oGMq4emb2FCRJvVkXCknekOTuJCuSnDx0PZI0l8yq3UdJtgX+B/BaYCVwQ5JLqurOYSuTtDFzcRfOM9Fs6ynsB6yoqnur6ifAecBhA9ckSXPGrOopAAuBB0ZerwT+9egESZYCS9vLHyS5e4ZqG5ddgX8euohZxO3xZG6P9dwWI/Lfntb2eNGGRsy2UNioqjoDOGPoOraUJMurasnQdcwWbo8nc3us57Z4snFtj9m2+2gVsMfI691bmyRpBsy2ULgBWJzkxUmeBRwNXDJwTZI0Z8yq3UdV9ViSdwBXANsCZ1bVHQOXNW7PmF1hW4jb48ncHuu5LZ5sLNsjVTWO5UqStkKzbfeRJGlAhoIkqWcoDCTJHkmuSnJnkjuSvHvomoaWZNsk30hy6dC1DC3JTkkuSPLNJHclOWDomoaU5D3t/+T2JOcmec7QNc2kJGcmWZPk9pG2XZJcmeSe9rzzlliXoTCcx4CTqmpvYH/ghCR7D1zT0N4N3DV0EbPEfwcur6q9gJczh7dLkoXAu4AlVfUyupNQjh62qhl3FvCGCW0nA8uqajGwrL1+2gyFgVTV6qq6qQ0/QvdPv3DYqoaTZHfgjcBnhq5laEmeD7wG+CxAVf2kqv5l2KoGNw94bpJ5wPOA/ztwPTOqqq4Gvjuh+TDg7DZ8NnD4lliXoTALJFkE7AtcN2wlg/pz4A+BJ4YuZBZ4MbAW+Ku2O+0zSbYfuqihVNUq4CPAt4HVwMNV9eVhq5oVFlTV6jb8ILBgSyzUUBhYkh2AC4ETq+r7Q9czhCS/AaypqhuHrmWWmAe8Aji9qvYFfsgW2jWwNWr7yg+jC8sXANsn+Z1hq5pdqvtuwRb5foGhMKAk29EFwjlVddHQ9QzoVcCbktxHd2XcX03y18OWNKiVwMqqWtdzvIAuJOaqXwO+VVVrq+qnwEXAgQPXNBs8lGQ3gPa8Zkss1FAYSJLQ7TO+q6o+NnQ9Q6qqU6pq96paRHcA8atVNWc/CVbVg8ADSX6hNR0CzOV7inwb2D/J89r/zSHM4QPvIy4Bjm3DxwKf3xILNRSG8yrgLXSfim9uj0OHLkqzxjuBc5LcCuwD/OnA9Qym9ZguAG4CbqN735pTl7xIci5wDfALSVYmeRtwGvDaJPfQ9aZO2yLr8jIXkqR17ClIknqGgiSpZyhIknqGgiSpZyhIknqGgkjyeDsl9vYkX0iy09NY1g+exrzvalcEPWdC+z6jp+smOTXJf9jc9YxDkvdvgWXcl2TXLVHPNNZ1uBdg1GQMBQH8v6rap12B8rvACQPV8XbgtVX12xPa9wFm+3c4phUKSbYddyHTdDhgKOgpDAVNdA3taq1J9ktyTbso2z+t+4ZtkuOSXJTk8nYt9z+buJAku7Z53zjJuPe2XsntSU5sbZ8CXgJ8Kcl7RqZ9FvBfgKNab+aoNmrvJF9Lcm+Sd41M/ztJrm/T/uVkb8LtE/mH2jTLk7wiyRVJ/k+S49s0SfLhVuNt69abZLckV4/0rF6d5DS6K3jePLGX0+b5QZKPJrkFOGCaNT5lmiTHJ/nwyDTHJflkG/67JDemu+fA0gnr/mCSW5Jcm2RBkgOBNwEfbsvfc8K6z0ryqbZt/ne7NhVJFiX5+yQ3tceBU2yTbdty1m2/97Rpfz/JDa2eC5M8r7Xv2eq7LckHRnucSd7X5rk1yX+euK20hVWVjzn+AH7QnrcF/hZ4Q3v9M8C8NvxrwIVt+DjgXuD5wHOA+4E91i2L7mqN19F96p+4rl+m+1bq9sAOwB3Avm3cfcCuk8xzHPDJkdenAv8EPBvYFfgOsB3wi8AXgO3adP8TeOsky7sP+Pdt+OPArcCOwHzgodb+b4Er2zZZQHephd2Ak4A/GtleO45uww1s3wKObMMbrHHdz7+haVp9K0aW+yXgoDa8S3t+LnA78LMj6/7NNvxnwB+34bOAN2+g3rOAy+k+NC6muxbTc+guWf2cNs1iYHkbfso2ab/nK0eWuVN7/tmRtg8A72zDlwLHtOHjWf83+Tq6by+n1XMp8Jqh/2eeyY95SO1TLl0P4S66N0Po3vTPTrKY7s1lu5F5llXVwwBJ7gReBDzQplkGnFBVX59kXQcBF1fVD9u8FwGvBr6xiTVfVlWPAo8mWUP3xn0I3ZvRDUmge4Pc0EXCLmnPtwE7VHdPi0eSPJrumMpBwLlV9Tjdhce+DrwSuAE4M93FDP+uqm6eRq2P0134kGnWOOk0VbW29Yz2B+4B9gL+sc3zriRHtOE96N60vwP8hO6NFOBG4LXTqBfg/Kp6Argnyb1tXd8CPplkn/YzvbRN+5Rt0uZ5SZK/AC4D1l3q+mVJPgDsRPeh4IrWfgDr7wfwN3SXyoYuFF7H+r+PHdrPdvU0fw5tIkNB0I4ptK78FXTHFD4B/Ffgqqo6It09H742Ms+jI8OPs/5v6TG6N5/XA5OFwpYy2foDnF1Vp2zC/E9MWNYTTPF/UVVXJ3kN3Q2Bzkrysar63EbW9eMWLkyzxqmmOQ84EvgmXbhWkoPpenIHVNWPknyN7pM9wE+rfeTmyb+njZl4/ZsC3gM8RHcnuG2AH8OGt0mSl9P9HRzfav49ul7I4VV1S5LjgIM3UkeAD1XVX06zbj1NHlNQr6p+RHfbw5PS3eHq+cCqNvq46S6G7p9/ryT/cZLxfw8cnu6Kl9sDR7S2qTxCt0tiY5YBb07yc9Dfw/ZF06x7sjqPavvG59PdCe36tryHqurTdHeJW3dJ65+2T8pbosapprmY7t4Cx9AFBHS/p++1QNiL7vauG7OxbfpbSbZpxxteAtzd1rO69SDeQreriMm2SbqzqLapqguBP2b9dtoRWN221egJBdfS7bKDJ99q8wrg99Ldd4QkC9dtF42HoaAnqapv0O1jP4ZuH/SHknyDTehVtk/Fx9BdAfbtE8bdRPdp8Xq64w6faeucylV0B5ZHDzRPtt476d6Avpzu6qJX0h0H2BwX022HW4CvAn9Y3SWtDwZuadvkKLp7KUO33/vWyQ40b2qNU01TVd+j28X3oqq6vs1yOTAvyV10V8q8dho/33nA+9KdRLDnJOO/Tfc7+hJwfFX9mO7YxrHpDpjvRXfzH5h8mywEvtZ2S/41sK7X85/ofu//SNfbWedE4L3t5/154OH2836ZbnfSNUluo7ta6nQ+IGgzeZVUSU+S5Czg0qq6YAbX+Ty63ZiV5Gi6g86HzdT6tZ7HFCTNBr9MdxA7wL/Q7YLUAOwpSJJ6HlOQJPUMBUlSz1CQJPUMBUlSz1CQJPX+P5NyvzBlZNthAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "523 / 523 questions predicted.------------------------------------------------------------ \n",
            "\n",
            "Theme: Spectre_(2015_film)\n",
            "Total Questions: 320\n",
            "Total Paragraphs: 43\n",
            "Total queries: 320\n",
            "In top 10 results, number of queries for which -\n",
            "\tRelevant paragraph found: 301 (94 %)\n",
            "\tRelevant paragraph NOT found: 19 (6 %)\n",
            "Mean Rank for which relevant paragraph found: 1.86\n",
            "\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAXr0lEQVR4nO3deZSldX3n8fdHwA1QQCp9GBYbSKthPGNjSgZFPUTUoGYEMoblGIHI2DLigjg6qMnEmdGRcU2ME0yrDHiCILJE3EBOi5AFhGYRmsWw2Ej3tN0dNIgbCnznj+epp28X1V1FU/c+BfV+nXPPfe7v2b71VNX93Ge5zy9VhSRJAE/ouwBJ0txhKEiSOoaCJKljKEiSOoaCJKmzdd8FPBo777xzLVy4sO8yJOkx5ZprrvmXqhqbatxjOhQWLlzI8uXL+y5Dkh5Tkty1qXEePpIkdQwFSVLHUJAkdQwFSVLHUJAkdQwFSVLHUJAkdQwFSVLHUJAkdR7T32h+tBae/PVe1rvylNf0sl5Jmo57CpKkjqEgSeoYCpKkztBCIcnuSS5NcnOSm5K8o23fKcklSW5rn3ds25PkU0luT3JDkucPqzZJ0tSGuafwAPCuqtoH2B84Ick+wMnAsqpaBCxrXwO8CljUPpYApw6xNknSFIYWClW1pqqubYfvA24BdgUOAc5oJzsDOLQdPgT4QjWuBHZIssuw6pMkPdxIzikkWQjsC3wXWFBVa9pRPwIWtMO7AncPzLaqbZu8rCVJlidZvn79+qHVLEnz0dBDIcl2wHnAiVX108FxVVVAPZLlVdXSqhqvqvGxsSl7k5MkbaGhhkKSbWgC4cyqOr9tXjtxWKh9Xte2rwZ2H5h9t7ZNkjQiw7z6KMDngVuq6hMDoy4EjmmHjwG+MtB+dHsV0v7AvQOHmSRJIzDM21wcALwBuDHJ9W3b+4BTgHOSHAfcBRzejvsG8GrgduAXwJ8MsTZJ0hSGFgpV9Q9ANjH6oCmmL+CEYdUjSZqe32iWJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSZ5jdcZ6WZF2SFQNtX0pyfftYOdEjW5KFSX45MO4zw6pLkrRpw+yO83Tg08AXJhqq6oiJ4SQfB+4dmP6Oqlo8xHokSdMYZneclydZONW4JKHpm/llw1q/JOmR6+ucwkuAtVV120DbnkmuS3JZkpdsasYkS5IsT7J8/fr1w69UkuaRvkLhKOCsgddrgD2qal/gJOCLSZ421YxVtbSqxqtqfGxsbASlStL8MfJQSLI18IfAlybaqur+qrqnHb4GuAN41qhrk6T5ro89hZcDt1bVqomGJGNJtmqH9wIWAXf2UJskzWvDvCT1LOAK4NlJViU5rh11JBsfOgJ4KXBDe4nqucDxVfXjYdUmSZraMK8+OmoT7cdO0XYecN6wapEkzYzfaJYkdQwFSVLHUJAkdQwFSVLHUJAkdQwFSVLHUJAkdQwFSVLHUJAkdQwFSVLHUJAkdQwFSVLHUJAkdQwFSVLHUJAkdQwFSVJnmD2vnZZkXZIVA20fSLI6yfXt49UD496b5PYk30/y+8OqS5K0acPcUzgdOHiK9k9W1eL28Q2AJPvQdNP5b9t5/nqiz2ZJ0ugMLRSq6nJgpv0sHwKcXVX3V9UPgNuB/YZVmyRpan2cU3hrkhvaw0s7tm27AncPTLOqbXuYJEuSLE+yfP369cOuVZLmlVGHwqnA3sBiYA3w8Ue6gKpaWlXjVTU+NjY22/VJ0rw20lCoqrVV9WBVPQR8lg2HiFYDuw9MulvbJkkaoZGGQpJdBl4eBkxcmXQhcGSSJyXZE1gEXDXK2iRJsPWwFpzkLOBAYOckq4A/Bw5MshgoYCXwZoCquinJOcDNwAPACVX14LBqkyRNbWihUFVHTdH8+c1M/yHgQ8OqR5I0Pb/RLEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpM7QQiHJaUnWJVkx0PbRJLcmuSHJBUl2aNsXJvllkuvbx2eGVZckadOGuadwOnDwpLZLgOdW1b8D/hl478C4O6pqcfs4foh1SZI2YWihUFWXAz+e1PatqnqgfXklsNuw1i9JeuT6PKfwRuCbA6/3THJdksuSvGRTMyVZkmR5kuXr168ffpWSNI/0EgpJ3g88AJzZNq0B9qiqfYGTgC8medpU81bV0qoar6rxsbGx0RQsSfPEyEMhybHAHwCvr6oCqKr7q+qedvga4A7gWaOuTZLmu5GGQpKDgfcAr62qXwy0jyXZqh3eC1gE3DnK2iRJMwyFJAfMpG3S+LOAK4BnJ1mV5Djg08D2wCWTLj19KXBDkuuBc4Hjq+rHUy5YkjQ0W89wur8Cnj+Dtk5VHTVF8+c3Me15wHkzrEWSNCSbDYUkLwReBIwlOWlg1NOArYZZmCRp9KbbU3gisF073fYD7T8FXjesoiRJ/dhsKFTVZcBlSU6vqrtGVJMkqSczPafwpCRLgYWD81TVy4ZRlCSpHzMNhS8DnwE+Bzw4vHIkSX2aaSg8UFWnDrUSSVLvZvrlta8meUuSXZLsNPEYamWSpJGb6Z7CMe3zuwfaCthrdsuRJPVpRqFQVXsOuxBJUv9mFApJjp6qvaq+MLvlSJL6NNPDRy8YGH4ycBBwLWAoSNLjyEwPH71t8HXbt/LZQ6lIktSbLb119s8BzzNI0uPMTM8pfJXmaiNoboT3O8A5wypKktSPmZ5T+NjA8APAXVW1agj1SJJ6NKPDR+2N8W6luVPqjsCvh1mUJKkfM+157XDgKuCPgMOB7yaZ9tbZSU5Lsi7JioG2nZJckuS29nnHtj1JPpXk9iQ3JNlkBz6SpOGY6Ynm9wMvqKpjqupoYD/gz2Yw3+nAwZPaTgaWVdUiYFn7GuBVNH0zLwKWAN5rSZJGbKah8ISqWjfw+p6ZzFtVlwOT+1o+BDijHT4DOHSg/QvVuBLYIckuM6xPkjQLZnqi+aIkFwNnta+PAL6xhetcUFVr2uEfAQva4V2BuwemW9W2rRloI8kSmj0J9thjjy0sQZI0len6aP5tmjfxdyf5Q+DF7agrgDMf7cqrqpLU9FNuNM9SYCnA+Pj4I5pXkrR50x0C+gua/pipqvOr6qSqOgm4oB23JdZOHBZqnycOS60Gdh+Ybre2TZI0ItOFwoKqunFyY9u2cAvXeSEbbsV9DPCVgfaj26uQ9gfuHTjMJEkagenOKeywmXFPmW7hSc4CDgR2TrIK+HPgFOCcJMcBd9Fc4grNOYpXA7cDvwD+ZLrlS5Jm13ShsDzJm6rqs4ONSf4TcM10C6+qozYx6qAppi3ghOmWKUkanulC4UTggiSvZ0MIjANPBA4bZmGSpNHbbChU1VrgRUl+D3hu2/z1qvr20CuTJI3cTPtTuBS4dMi1SJJ6tqX9KUiSHocMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHVmdJfU2ZTk2cCXBpr2Av4bTS9vbwLWt+3vq6pvjLg8SZrXRh4KVfV9YDFAkq2A1cAFNN1vfrKqPjbqmiRJjb4PHx0E3FFVd/VchySJ/kPhSOCsgddvTXJDktOS7DjVDEmWJFmeZPn69eunmkSStIV6C4UkTwReC3y5bToV2Jvm0NIa4ONTzVdVS6tqvKrGx8bGRlKrJM0Xfe4pvAq4tu0HmqpaW1UPVtVDwGeB/XqsTZLmpT5D4SgGDh0l2WVg3GHAipFXJEnz3MivPgJIsi3wCuDNA80fSbIYKGDlpHGSpBHoJRSq6ufAMya1vaGPWiRJG/R99ZEkaQ4xFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJnV462QFIshK4D3gQeKCqxpPsBHwJWEjT+9rhVfWTvmqUpPmm7z2F36uqxVU13r4+GVhWVYuAZe1rSdKI9B0Kkx0CnNEOnwEc2mMtkjTv9BkKBXwryTVJlrRtC6pqTTv8I2DB5JmSLEmyPMny9evXj6pWSZoXejunALy4qlYn+S3gkiS3Do6sqkpSk2eqqqXAUoDx8fGHjZckbbne9hSqanX7vA64ANgPWJtkF4D2eV1f9UnSfNRLKCTZNsn2E8PAK4EVwIXAMe1kxwBf6aM+SZqv+jp8tAC4IMlEDV+sqouSXA2ck+Q44C7g8J7qk6R5qZdQqKo7gedN0X4PcNDoK5Ikwdy7JFWS1CNDQZLUMRQkSR1DQZLUMRQkSR1DQZLUMRQkSR1DQZLUMRQkSR1DQZLUMRQkSZ0++1PQiC08+eu9rXvlKa/pbd2SZs49BUlSx1CQJHUMBUlSx1CQJHVGHgpJdk9yaZKbk9yU5B1t+weSrE5yfft49ahrk6T5ro+rjx4A3lVV17b9NF+T5JJ23Cer6mM91CRJoodQqKo1wJp2+L4ktwC7jroOSdLD9fo9hSQLgX2B7wIHAG9NcjSwnGZv4idTzLMEWAKwxx57jKzW2dTn9wUkaXN6O9GcZDvgPODEqvopcCqwN7CYZk/i41PNV1VLq2q8qsbHxsZGVq8kzQe9hEKSbWgC4cyqOh+gqtZW1YNV9RDwWWC/PmqTpPmsj6uPAnweuKWqPjHQvsvAZIcBK0ZdmyTNd32cUzgAeANwY5Lr27b3AUclWQwUsBJ4cw+1SdK81sfVR/8AZIpR3xh1LZKkjfmNZklSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSp9cb4mn+6OsmgCtPeU0v65Ueq9xTkCR1DAVJUsdQkCR1PKegx7U+OzTyfIYei9xTkCR13FOQhsQrrvRY5J6CJKnjnoL0OON5FD0acy4UkhwM/CWwFfC5qjql55IkzZCHzB775lQoJNkK+D/AK4BVwNVJLqyqm/utTNJcZhjNnjkVCsB+wO1VdSdAkrOBQwBDQdKc83g8VDfXQmFX4O6B16uAfz84QZIlwJL25c+SfH9EtQ3LzsC/9F3EHOL22JjbYwO3xYD870e1PZ65qRFzLRSmVVVLgaV91zFbkiyvqvG+65gr3B4bc3ts4LbY2LC2x1y7JHU1sPvA693aNknSCMy1ULgaWJRkzyRPBI4ELuy5JkmaN+bU4aOqeiDJW4GLaS5JPa2qbuq5rGF73BwKmyVuj425PTZwW2xsKNsjVTWM5UqSHoPm2uEjSVKPDAVJUsdQ6EmS3ZNcmuTmJDcleUffNfUtyVZJrkvytb5r6VuSHZKcm+TWJLckeWHfNfUpyTvb/5MVSc5K8uS+axqlJKclWZdkxUDbTkkuSXJb+7zjbKzLUOjPA8C7qmofYH/ghCT79FxT394B3NJ3EXPEXwIXVdVzgOcxj7dLkl2BtwPjVfVcmotQjuy3qpE7HTh4UtvJwLKqWgQsa18/aoZCT6pqTVVd2w7fR/NPv2u/VfUnyW7Aa4DP9V1L35I8HXgp8HmAqvp1Vf1rv1X1bmvgKUm2Bp4K/L+e6xmpqroc+PGk5kOAM9rhM4BDZ2NdhsIckGQhsC/w3X4r6dVfAO8BHuq7kDlgT2A98H/bw2mfS7Jt30X1papWAx8DfgisAe6tqm/1W9WcsKCq1rTDPwIWzMZCDYWeJdkOOA84sap+2nc9fUjyB8C6qrqm71rmiK2B5wOnVtW+wM+ZpUMDj0XtsfJDaMLy3wDbJvnjfquaW6r5bsGsfL/AUOhRkm1oAuHMqjq/73p6dADw2iQrgbOBlyX5235L6tUqYFVVTew5nksTEvPVy4EfVNX6qvoNcD7wop5rmgvWJtkFoH1eNxsLNRR6kiQ0x4xvqapP9F1Pn6rqvVW1W1UtpDmB+O2qmrefBKvqR8DdSZ7dNh3E/L59/A+B/ZM8tf2/OYh5fOJ9wIXAMe3wMcBXZmOhhkJ/DgDeQPOp+Pr28eq+i9Kc8TbgzCQ3AIuB/9VzPb1p95jOBa4FbqR535pXt7xIchZwBfDsJKuSHAecArwiyW00e1Oz0kult7mQJHXcU5AkdQwFSVLHUJAkdQwFSVLHUJAkdQwFkeTB9pLYFUm+mmSHR7Gsnz2Ked/e3hH0zEntiwcv103ygST/ZUvXMwxJ3jcLy1iZZOfZqGcG6zrUGzBqKoaCAH5ZVYvbO1D+GDihpzreAryiql4/qX0xMNe/wzGjUEiy1bALmaFDAUNBD2MoaLIraO/WmmS/JFe0N2X7p4lv2CY5Nsn5SS5q7+X+kckLSbJzO+9rphh3UrtXsiLJiW3bZ4C9gG8meefAtE8E/gdwRLs3c0Q7ap8k30lyZ5K3D0z/x0muaqf9m6nehNtP5B9up1me5PlJLk5yR5Lj22mS5KNtjTdOrDfJLkkuH9izekmSU2ju4Hn95L2cdp6fJfl4ku8BL5xhjQ+bJsnxST46MM2xST7dDv9dkmvS9DmwZNK6P5Tke0muTLIgyYuA1wIfbZe/96R1n57kM+22+ef23lQkWZjk75Nc2z5etJltslW7nInt98522jclubqt57wkT23b927ruzHJBwf3OJO8u53nhiT/ffK20iyrKh/z/AH8rH3eCvgycHD7+mnA1u3wy4Hz2uFjgTuBpwNPBu4Cdp9YFs3dGr9L86l/8rp+l+ZbqdsC2wE3Afu241YCO08xz7HApwdefwD4J+BJwM7APcA2wO8AXwW2aaf7a+DoKZa3EvjP7fAngRuA7YExYG3b/h+BS9ptsoDmVgu7AO8C3j+wvbYf3Iab2L4FHN4Ob7LGiZ9/U9O09d0+sNxvAi9uh3dqn58CrACeMbDu/9AOfwT403b4dOB1m6j3dOAimg+Ni2juxfRkmltWP7mdZhGwvB1+2DZpf8+XDCxzh/b5GQNtHwTe1g5/DTiqHT6eDX+Tr6T59nLaer4GvLTv/5nH82NrpPZTLs0ewi00b4bQvOmfkWQRzZvLNgPzLKuqewGS3Aw8E7i7nWYZcEJVXTbFul4MXFBVP2/nPR94CXDdI6z561V1P3B/knU0b9wH0bwZXZ0EmjfITd0k7ML2+UZgu2r6tLgvyf1pzqm8GDirqh6kufHYZcALgKuB09LczPDvqur6GdT6IM2ND5lhjVNOU1Xr2z2j/YHbgOcA/9jO8/Ykh7XDu9O8ad8D/JrmjRTgGuAVM6gX4Jyqegi4Lcmd7bp+AHw6yeL2Z3pWO+3Dtkk7z15J/gr4OjBxq+vnJvkgsAPNh4KL2/YXsqE/gC/S3CobmlB4JRv+PrZrf7bLZ/hz6BEyFATtOYV2V/5imnMKnwL+J3BpVR2Wps+H7wzMc//A8INs+Ft6gObN5/eBqUJhtky1/gBnVNV7H8H8D01a1kNs5v+iqi5P8lKaDoFOT/KJqvrCNOv6VRsuzLDGzU1zNnA4cCtNuFaSA2n25F5YVb9I8h2aT/YAv6n2Izcb/56mM/n+NwW8E1hL0xPcE4Bfwaa3SZLn0fwdHN/W/EaavZBDq+p7SY4FDpymjgAfrqq/mWHdepQ8p6BOVf2CptvDd6Xp4erpwOp29LEzXQzNP/9zkvzXKcb/PXBomjtebgsc1rZtzn00hySmswx4XZLfgq4P22fOsO6p6jyiPTY+RtMT2lXt8tZW1WdpeombuKX1b9pPyrNR4+amuYCmb4GjaAICmt/TT9pAeA5N967TmW6b/lGSJ7TnG/YCvt+uZ027B/EGmkNFTLVN0lxF9YSqOg/4UzZsp+2BNe22Gryg4EqaQ3awcVebFwNvTNPvCEl2ndguGg5DQRupqutojrEfRXMM+sNJruMR7FW2n4qPorkD7FsmjbuW5tPiVTTnHT7XrnNzLqU5sTx4onmq9d5M8wb0rTR3F72E5jzAlriAZjt8D/g28J5qbml9IPC9dpscQdOXMjTHvW+Y6kTzI61xc9NU1U9oDvE9s6quame5CNg6yS00d8q8cgY/39nAu9NcRLD3FON/SPM7+iZwfFX9iubcxjFpTpg/h6bzH5h6m+wKfKc9LPm3wMRez5/R/N7/kWZvZ8KJwEntz/vbwL3tz/stmsNJVyS5keZuqTP5gKAt5F1SJW0kyenA16rq3BGu86k0hzEryZE0J50PGdX6tYHnFCTNBb9LcxI7wL/SHIJUD9xTkCR1PKcgSeoYCpKkjqEgSeoYCpKkjqEgSer8f4GRwQpj/xvuAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "320 / 320 questions predicted.------------------------------------------------------------ \n",
            "\n",
            "Theme: New_York_City\n",
            "Total Questions: 563\n",
            "Total Paragraphs: 145\n",
            "Total queries: 563\n",
            "In top 10 results, number of queries for which -\n",
            "\tRelevant paragraph found: 547 (97 %)\n",
            "\tRelevant paragraph NOT found: 16 (3 %)\n",
            "Mean Rank for which relevant paragraph found: 1.4\n",
            "\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAVf0lEQVR4nO3de7SldX3f8fdHBgRBGZUpiw4sB5VqXFkVzWhAjMsl0aIYwVVvLKOQ0LCs1BtWg0m6GltbMabeYqsSsGBjIYRLRFCQchFNFBzuAqaOBGQoMhNFFC8o+O0fz29+bIZz2cDZZ58z836ttdd5nt9z+55nzuzPfi7796SqkCQJ4FHTLkCStHQYCpKkzlCQJHWGgiSpMxQkSd2KaRfwSOy22261Zs2aaZchScvKFVdc8U9VtWqmacs6FNasWcO6deumXYYkLStJbpltmqePJEmdoSBJ6gwFSVJnKEiSOkNBktQZCpKkzlCQJHWGgiSpMxQkSd2y/kbzI7Hm2HOntu2bjzt4atuWpLl4pCBJ6gwFSVJnKEiSOkNBktQZCpKkzlCQJHWGgiSpMxQkSZ2hIEnqDAVJUmcoSJI6Q0GS1E08FJJsl+SqJOe08b2TXJZkfZK/TrJDa390G1/fpq+ZdG2SpAdajCOFtwE3jox/APhwVT0VuBM4srUfCdzZ2j/c5pMkLaKJhkKSPYGDgRPaeIAXAae3WU4GDm3Dh7Rx2vQD2/ySpEUy6SOFjwDvBn7Vxp8I/LCq7m3jG4DVbXg1cCtAm35Xm/8BkhyVZF2SdZs2bZpk7ZK0zZlYKCR5ObCxqq5YyPVW1fFVtbaq1q5atWohVy1J27xJPnntAOAVSV4G7Ag8DvgosDLJinY0sCdwW5v/NmAvYEOSFcCuwPcnWJ8kaQsTO1KoqvdU1Z5VtQZ4HXBRVb0euBh4VZvtcOBzbfjsNk6bflFV1aTqkyQ92DS+p/CHwDFJ1jNcMzixtZ8IPLG1HwMcO4XaJGmbNsnTR11VXQJc0oZvAp47wzw/B169GPVIkmbmN5olSZ2hIEnqDAVJUmcoSJI6Q0GS1BkKkqTOUJAkdYaCJKkzFCRJnaEgSeoMBUlSZyhIkjpDQZLUGQqSpM5QkCR1hoIkqTMUJEmdoSBJ6gwFSVJnKEiSOkNBktQZCpKkzlCQJHWGgiSpMxQkSZ2hIEnqDAVJUmcoSJI6Q0GS1BkKkqTOUJAkdYaCJKkzFCRJnaEgSeoMBUlSZyhIkjpDQZLUGQqSpG5ioZBkxySXJ7kmyfVJ3tva905yWZL1Sf46yQ6t/dFtfH2bvmZStUmSZjbJI4V7gBdV1TOBfYGDkuwHfAD4cFU9FbgTOLLNfyRwZ2v/cJtPkrSIJhYKNbi7jW7fXgW8CDi9tZ8MHNqGD2njtOkHJsmk6pMkPdhErykk2S7J1cBG4ALgO8APq+reNssGYHUbXg3cCtCm3wU8cZL1SZIeaKKhUFX3VdW+wJ7Ac4GnP9J1Jjkqybok6zZt2vSIa5Qk3W9R7j6qqh8CFwP7AyuTrGiT9gRua8O3AXsBtOm7At+fYV3HV9Xaqlq7atWqidcuSduSSd59tCrJyja8E/Bi4EaGcHhVm+1w4HNt+Ow2Tpt+UVXVpOqTJD3Yivlnedj2AE5Osh1D+JxWVeckuQE4Ncn7gKuAE9v8JwL/K8l64AfA6yZYmyRpBhMLhaq6FnjWDO03MVxf2LL958CrJ1WPJGl+fqNZktQZCpKkzlCQJHWGgiSpMxQkSZ2hIEnqDAVJUmcoSJI6Q0GS1BkKkqTOUJAkdYaCJKkzFCRJnaEgSeoMBUlSZyhIkjpDQZLUjRUKSQ4Yp02StLyNe6TwF2O2SZKWsTmf0Zxkf+B5wKokx4xMehyw3SQLkyQtvjlDAdgB2KXN99iR9h8Br5pUUZKk6ZgzFKrqy8CXk5xUVbcsUk2SpCmZ70hhs0cnOR5YM7pMVb1oEkVJkqZj3FD4G+CTwAnAfZMrR5I0TeOGwr1V9YmJViJJmrpxb0n9fJI3J9kjyRM2vyZamSRp0Y17pHB4+/mukbYCnryw5UiSpmmsUKiqvSddiCRp+sYKhSRvnKm9qj6zsOVIkqZp3NNHzxkZ3hE4ELgSMBQkaSsy7umjt4yOJ1kJnDqRiiRJU/Nwu87+CeB1Bknayox7TeHzDHcbwdAR3q8Bp02qKEnSdIx7TeHPR4bvBW6pqg0TqEeSNEVjnT5qHeN9i6Gn1McDv5hkUZKk6Rj3yWuvAS4HXg28BrgsiV1nS9JWZtzTR38MPKeqNgIkWQX8H+D0SRUmSVp849599KjNgdB8/yEsK0laJsY9UjgvyfnAKW38tcAXJlOSJGla5vy0n+SpSQ6oqncBnwL+ZXt9DTh+nmX3SnJxkhuSXJ/kba39CUkuSPLt9vPxrT1JPpZkfZJrkzx7QX5DSdLY5jsF9BGG5zFTVWdW1TFVdQxwVps2l3uBd1bVM4D9gKOTPAM4FriwqvYBLmzjAC8F9mmvowCf3yBJi2y+UNi9qq7bsrG1rZlrwaq6vaqubMM/Bm4EVgOHACe32U4GDm3DhwCfqcHXgZVJ9hj3F5EkPXLzhcLKOabtNO5GkqwBngVcxhA0t7dJ3wN2b8OrgVtHFtvQ2rZc11FJ1iVZt2nTpnFLkCSNYb5QWJfkD7ZsTPJvgCvG2UCSXYAzgLdX1Y9Gp1VVcX/3GWOpquOram1VrV21atVDWVSSNI/57j56O3BWktdzfwisBXYAXjnfypNszxAIn62qM1vzHUn2qKrb2+mhzbe63gbsNbL4nq1NkrRI5jxSqKo7qup5wHuBm9vrvVW1f1V9b65lkwQ4Ebixqj40Muls7n+85+HA50ba39juQtoPuGvkNJMkaRGM+zyFi4GLH+K6DwDeAFyX5OrW9kfAccBpSY4EbmHoNgOG7z28DFgP/BT4vYe4PUnSIzTul9cesqr6KpBZJh84w/wFHD2peiRJ87OrCklSZyhIkjpDQZLUGQqSpM5QkCR1hoIkqTMUJEmdoSBJ6gwFSVJnKEiSOkNBktQZCpKkzlCQJHWGgiSpMxQkSZ2hIEnqDAVJUmcoSJI6Q0GS1BkKkqTOUJAkdYaCJKkzFCRJnaEgSeoMBUlSZyhIkjpDQZLUGQqSpM5QkCR1hoIkqTMUJEmdoSBJ6gwFSVJnKEiSOkNBktQZCpKkzlCQJHWGgiSpMxQkSd3EQiHJp5NsTPLNkbYnJLkgybfbz8e39iT5WJL1Sa5N8uxJ1SVJmt0kjxROAg7aou1Y4MKq2ge4sI0DvBTYp72OAj4xwbokSbOYWChU1aXAD7ZoPgQ4uQ2fDBw60v6ZGnwdWJlkj0nVJkma2WJfU9i9qm5vw98Ddm/Dq4FbR+bb0NoeJMlRSdYlWbdp06bJVSpJ26CpXWiuqgLqYSx3fFWtraq1q1atmkBlkrTtWuxQuGPzaaH2c2Nrvw3Ya2S+PVubJGkRLXYonA0c3oYPBz430v7GdhfSfsBdI6eZJEmLZMWkVpzkFOCFwG5JNgD/ETgOOC3JkcAtwGva7F8AXgasB34K/N6k6pIkzW5ioVBVh80y6cAZ5i3g6EnVIkkaj99oliR1hoIkqTMUJEmdoSBJ6gwFSVJnKEiSOkNBktQZCpKkzlCQJHWGgiSpm1g3F5rdmmPPncp2bz7u4KlsV9Ly4ZGCJKkzFCRJnaEgSeoMBUlSZyhIkjpDQZLUGQqSpM5QkCR1hoIkqTMUJEmdoSBJ6gwFSVJnKEiSOkNBktQZCpKkzlCQJHWGgiSpMxQkSZ2hIEnqDAVJUmcoSJI6Q0GS1K2YdgFaPGuOPXdq2775uIOntm1J4/NIQZLUGQqSpM7TR9qqTeuUmafLtFx5pCBJ6gwFSVK3pE4fJTkI+CiwHXBCVR035ZKkh2VbvNNrW/ydt0ZLJhSSbAf8d+DFwAbgG0nOrqobpluZJM1sawzCJRMKwHOB9VV1E0CSU4FDAENB0pym+ea8tVlKobAauHVkfAPwm1vOlOQo4Kg2eneSf3iY29sN+KeHuew0LKd6H1RrPjClSsazrPftlpbYvt6q9u1Skg88onqfNNuEpRQKY6mq44HjH+l6kqyrqrULUNKiWE71LqdaYXnVu5xqheVV73KqFSZX71K6++g2YK+R8T1bmyRpkSylUPgGsE+SvZPsALwOOHvKNUnSNmXJnD6qqnuT/DvgfIZbUj9dVddPcJOP+BTUIltO9S6nWmF51bucaoXlVe9yqhUmVG+qahLrlSQtQ0vp9JEkacoMBUlSt82FQpJPJ9mY5JvTrmU+SfZKcnGSG5Jcn+Rt065pLkl2THJ5kmtave+ddk3zSbJdkquSnDPtWuaT5OYk1yW5Osm6adczlyQrk5ye5FtJbkyy/7Rrmk2Sp7V9uvn1oyRvn3Zds0nyjvb/65tJTkmy44Kuf1u7ppDkBcDdwGeq6tenXc9ckuwB7FFVVyZ5LHAFcOhS7fojSYCdq+ruJNsDXwXeVlVfn3Jps0pyDLAWeFxVvXza9cwlyc3A2qpa8l+wSnIy8JWqOqHdTfiYqvrhtOuaT+tu5zbgN6vqlmnXs6Ukqxn+Xz2jqn6W5DTgC1V10kJtY5s7UqiqS4EfTLuOcVTV7VV1ZRv+MXAjwze/l6Qa3N1Gt2+vJfupI8mewMHACdOuZWuSZFfgBcCJAFX1i+UQCM2BwHeWYiCMWAHslGQF8Bjg/y3kyre5UFiukqwBngVcNt1K5tZOx1wNbAQuqKqlXO9HgHcDv5p2IWMq4EtJrmjdvSxVewObgP/ZTs2dkGTnaRc1ptcBp0y7iNlU1W3AnwPfBW4H7qqqLy3kNgyFZSDJLsAZwNur6kfTrmcuVXVfVe3L8I305yZZkqfokrwc2FhVV0y7lofg+VX1bOClwNHtVOhStAJ4NvCJqnoW8BPg2OmWNL92musVwN9Mu5bZJHk8Q0ehewP/HNg5ye8u5DYMhSWunZs/A/hsVZ057XrG1U4XXAwcNO1aZnEA8Ip2nv5U4EVJ/mq6Jc2tfUqkqjYCZzH0LLwUbQA2jBwlns4QEkvdS4Erq+qOaRcyh98G/rGqNlXVL4Ezgect5AYMhSWsXbg9Ebixqj407Xrmk2RVkpVteCeGZ2N8a7pVzayq3lNVe1bVGoZTBhdV1YJ+4lpISXZuNxvQTsW8BFiSd9BV1feAW5M8rTUdyPLoAv8wlvCpo+a7wH5JHtPeHw5kuNa4YLa5UEhyCvA14GlJNiQ5cto1zeEA4A0Mn2I33y73smkXNYc9gIuTXMvQl9UFVbXkb/VcJnYHvprkGuBy4NyqOm/KNc3lLcBn29/CvsB/nXI9c2pB+2KGT95LVjv6Oh24EriO4T18Qbu72OZuSZUkzW6bO1KQJM3OUJAkdYaCJKkzFCRJnaEgSeoMhW1ckvvara7fTPL5zd8zeJjrunv+uWZd9q2tN83PbtG+7+htuEn+NMm/f7jbmYQkf7QA67g5yW4LUc8Y2zo0yTMWY1tafgwF/ayq9m09xv4AOHpKdbwZeHFVvX6L9n2BpfzdDICxQqH1wLkUHAoYCpqRoaBRX6P1wprkuUm+1jo0+/vN305NckSSM5Ocl+TbSf5sy5Uk2a0te/AM045pRyXf3NxnfZJPAk8GvpjkHSPz7gD8J+C17WjmtW3SM5JckuSmJG8dmf932/Mcrk7yqZnehNsn8vdvfiZBkmcnOT/Jd5K8qc2TJB9sNV63ebtJ9khy6ciR1W8lOY6hx8qrtzzKacvcneS/tS+d7T9mjQ+aJ8mbknxwZJ4jkny8Df9t6yTv+ox0lNe2/V8yPN/i60l2T/I8hv59PtjW/5Qttn1Skk+2ffN/M/QRRZI1Sb6S5Mr2et4c+2S7tp7N++8dbd4/SPKNVs8ZSR7T2p/S6rsuyftGjziTvKstc22WwfM5tgpV5WsbfgF3t5/bMXQEdlAbfxywog3/NnBGGz4CuAnYFdgRuAXYa/O6GL55exnDp/4tt/UbDN/C3BnYBbgeeFabdjOw2wzLHAF8fGT8T4G/Bx4N7AZ8n6GL7l8DPg9s3+b7H8AbZ1jfzcC/bcMfBq4FHgusAu5o7f8auKDtk90ZuhbYA3gn8Mcj++uxo/twlv1bwGva8Kw1bv79Z5un1bd+ZL1fZOggD+AJ7edODF1fPHFk27/Thv8M+JM2fBLwqlnqPQk4j+ED4z4M/RjtyNBF845tnn2AdW34Qfuk/TtfMLLOle3nE0fa3ge8pQ2fAxzWht/E/X+TL2H4tm5aPecAL5j2/5mt/bUCbet2ytDV9WqGPlQuaO27Aicn2YfhzWX7kWUurKq7AJLcADwJuLXNcyFwdFV9eYZtPR84q6p+0pY9E/gt4KqHWPO5VXUPcE+SjQxv3AcyvBl9IwkMb5AbZ1n+7PbzOmCXGp5V8eMk92S4pvJ84JSqug+4I8mXgecwdN3x6QydFP5tVV09Rq33MXRoyJg1zjhPVW1qR0b7Ad8Gng78XVvmrUle2Yb3YnjT/j7wC4Y3Uhge0PTiMeoFOK2qfgV8O8lNbVv/CHw8yb7td/oXbd4H7ZO2zJOT/AVwLrC5a+dfT/I+YCXDh4LzW/v+DKe0AP43Q9fQMITCS7j/72OX9rtdOubvoYfBUNDPqmrfdih/PsM1hY8B/xm4uKpemeFZDpeMLHPPyPB93P93dC/Dm8+/AmYKhYUy0/YDnFxV73kIy/9qi3X9ijn+T1TVpRm6qz4YOCnJh6rqM/Ns6+ctXBizxrnmORV4DUMng2dVVSV5IcOR3P5V9dMklzB8sgf4ZbWP3Dzw32k+W/Z9U8A7gDuAZzJ8av85zL5PkjyT4e/gTa3m32c4Cjm0qq5JcgTwwnnqCPD+qvrUmHVrAXhNQQBU1U+BtwLvzPBEp10ZHksIwymcsVbD8J//6Un+cIbpXwEOzdDD487AK1vbXH7McEpiPhcCr0ryzwCSPCHJk8ase6Y6X9vOja9ieIrY5W19d1TVXzI8rW1zd9C/bJ+UF6LGueY5i6Ev/cMYAgKGf6c7WyA8HdhvjDrm26evTvKodr3hycA/tO3c3o4g3sBwqoiZ9kmGu6geVVVnAH/C/fvpscDtbV+N3lDwdYZTdjD0WLvZ+cDvZ3ieCElWb94vmhxDQV1VXcVwjv0whnPQ709yFQ/hiLJ9Kj6MoWfXN28x7UqGT4uXM1x3OKFtcy4XM1xYHr3QPNN2b2B4A/pShp45L2C4DvBwnMWwH64BLgLeXUN30C8Ermn75LXAR9v8xwPXznSh+aHWONc8VXUnwym+J1XV5W2R84AVSW4EjmN4g53PqcC7MtxE8JQZpn+X4d/oi8CbqurnDNc2Ds9wwfzpDA/OgZn3yWrgknZa8q+AzUc9/4Hh3/3veGCX6m8Hjmm/71OBu9rv+yWG00lfS3IdQ++g43xA0CNgL6mSuiQnAedU1emLuM3HMJzGrCSvY7jofMhibV8P5DUFSdP2GwwXsQP8kOEUpKbEIwVJUuc1BUlSZyhIkjpDQZLUGQqSpM5QkCR1/x9tXvYf4iPH7gAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "563 / 563 questions predicted.------------------------------------------------------------ \n",
            "\n",
            "Theme: To_Kill_a_Mockingbird\n",
            "Total Questions: 166\n",
            "Total Paragraphs: 60\n",
            "Total queries: 166\n",
            "In top 10 results, number of queries for which -\n",
            "\tRelevant paragraph found: 155 (93 %)\n",
            "\tRelevant paragraph NOT found: 11 (7 %)\n",
            "Mean Rank for which relevant paragraph found: 1.95\n",
            "\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAVkUlEQVR4nO3de5BmdX3n8fdHBsJVLjI7RYByUImEcksgI+FiLFfUxWAEdwmXMgoJG4rVFQFXxSRbZnfdFaPrJXFXMwHDWGEhhEtQUHAWQTRBcLjfdCEIMiwwExUUbwh894/zm0M7dPc8DN19unner6qnnvOc67dPdz+fc37neX4nVYUkSQDPG7oASdL8YShIknqGgiSpZyhIknqGgiSpt2joAp6NHXfcsZYuXTp0GZK0oFx33XX/XFWLJ5u2oENh6dKlrFq1augyJGlBSXLvVNNsPpIk9QwFSVLPUJAk9QwFSVLPUJAk9QwFSVLPUJAk9QwFSVLPUJAk9Rb0N5qfjaWnXjLYtu857ZDBti1J05m1M4Ukn02yJsmtE8btkGRlkjvb8/ZtfJL8eZK7ktycZJ/ZqkuSNLXZbD46Ezh4vXGnApdX1e7A5e01wBuA3dvjeODTs1iXJGkKsxYKVXUV8P31Rh8KrGjDK4DDJoz/XHW+AWyXZKfZqk2SNLm5vtC8pKoeaMMPAkva8M7AfRPmW93GPU2S45OsSrJq7dq1s1epJI2hwT59VFUF1EYst7yqllXVssWLJ+0OXJK0keY6FB5a1yzUnte08fcDu06Yb5c2TpI0h+Y6FD4PHNOGjwEumjD+be1TSPsBj0xoZpIkzZFZ+55CkrOBVwM7JlkNfAA4DTg3yXHAvcARbfYvAr8N3AX8BPj92apLkjS1WQuFqjp6ikkHTTJvAe+YrVokSaOxmwtJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUm+QUEhycpLbktya5OwkmyfZLck1Se5K8rdJNhuiNkkaZ3MeCkl2Bk4EllXVy4BNgKOADwMfr6qXAD8Ajpvr2iRp3A3VfLQI2CLJImBL4AHgNcB5bfoK4LCBapOksTXnoVBV9wMfBb5LFwaPANcBD1fV42221cDOky2f5Pgkq5KsWrt27VyULEljY4jmo+2BQ4HdgF8FtgIOHnX5qlpeVcuqatnixYtnqUpJGk9DNB+9FvhOVa2tql8AFwAHAtu15iSAXYD7B6hNksbaEKHwXWC/JFsmCXAQcDtwBXB4m+cY4KIBapOksTbENYVr6C4oXw/c0mpYDrwPOCXJXcALgDPmujZJGneLNjzLzKuqDwAfWG/03cC+A5QjSWr8RrMkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6g4RCku2SnJfkW0nuSLJ/kh2SrExyZ3vefojaJGmcDXWm8Eng0qraA3g5cAdwKnB5Ve0OXN5eS5Lm0JyHQpJtgVcBZwBU1WNV9TBwKLCizbYCOGyua5OkcTfEmcJuwFrgr5PckOT0JFsBS6rqgTbPg8CSyRZOcnySVUlWrV27do5KlqTxMFIoJDlwlHEjWgTsA3y6qvYGfsx6TUVVVUBNtnBVLa+qZVW1bPHixRtZgiRpMqOeKfzFiONGsRpYXVXXtNfn0YXEQ0l2AmjPazZy/ZKkjbRouolJ9gcOABYnOWXCpOcDm2zMBqvqwST3JXlpVX0bOAi4vT2OAU5rzxdtzPolSRtv2lAANgO2bvNtM2H8D4HDn8V23wmclWQz4G7g9+nOWs5NchxwL3DEs1i/JGkjTBsKVfVV4KtJzqyqe2dqo1V1I7BskkkHzdQ2JEnP3IbOFNb5lSTLgaUTl6mq18xGUZKkYYwaCn8HfAY4HXhi9sqRJA1p1FB4vKo+PauVSJIGN+pHUr+Q5O1Jdmp9FO2QZIdZrUySNOdGPVM4pj2/Z8K4Al40s+VIkoY0UihU1W6zXYgkaXgjhUKSt002vqo+N7PlSJKGNGrz0SsmDG9O932C6wFDQZKeQ0ZtPnrnxNdJtgPOmZWKJEmD2dius39M1wW2JOk5ZNRrCl/gqa6sNwF+HTh3toqSJA1j1GsKH50w/Dhwb1WtnoV6JEkDGqn5qHWM9y26nlK3Bx6bzaIkScMY9c5rRwDXAr9L16X1NUmeTdfZkqR5aNTmoz8GXlFVawCSLAb+D91d0yRJzxGjfvroeesCofneM1hWkrRAjHqmcGmSy4Cz2+sjgS/OTkmSpKFs6B7NLwGWVNV7kvwb4JVt0tXAWbNdnCRpbm3oTOETwPsBquoC4AKAJP+yTfudWa1OkjSnNnRdYElV3bL+yDZu6axUJEkazIZCYbtppm0xk4VIkoa3oVBYleQP1x+Z5N8B181OSZKkoWzomsJJwIVJ3sJTIbAM2Ax482wWJkmae9OGQlU9BByQ5F8BL2ujL6mqr8x6ZZKkOTfq/RSuAK6Y5VokSQPzW8mSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqDRYKSTZJckOSi9vr3ZJck+SuJH+bZLOhapOkcTXkmcK7gDsmvP4w8PGqegnwA+C4QaqSpDE2SCgk2QU4BDi9vQ7wGp66vecK4LAhapOkcTbUmcIngPcCT7bXLwAerqrH2+vVwM5DFCZJ42zOQyHJG4E1VbVRvawmOT7JqiSr1q5dO8PVSdJ4G+JM4UDgTUnuAc6hazb6JLBdknV9Me0C3D/ZwlW1vKqWVdWyxYsXz0W9kjQ25jwUqur9VbVLVS0FjgK+UlVvoetw7/A22zHARXNdmySNu/n0PYX3AackuYvuGsMZA9cjSWNnpK6zZ0tVXQlc2YbvBvYdsh5JGnfz6UxBkjQwQ0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1JvzUEiya5Irktye5LYk72rjd0iyMsmd7Xn7ua5NksbdogG2+Tjw7qq6Psk2wHVJVgLHApdX1WlJTgVOBd43QH2zbumplwyy3XtOO2SQ7UpaOOb8TKGqHqiq69vwj4A7gJ2BQ4EVbbYVwGFzXZskjbtBrykkWQrsDVwDLKmqB9qkB4ElUyxzfJJVSVatXbt2TuqUpHExWCgk2Ro4Hzipqn44cVpVFVCTLVdVy6tqWVUtW7x48RxUKknjY5BQSLIpXSCcVVUXtNEPJdmpTd8JWDNEbZI0zob49FGAM4A7qupjEyZ9HjimDR8DXDTXtUnSuBvi00cHAm8FbklyYxv3R8BpwLlJjgPuBY4YoDZJGmtzHgpV9XUgU0w+aC5rkST9Mr/RLEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpN4QvaRqIEPdGxq8P7S0UHimIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ4d4kmaEXa4+NzgmYIkqeeZgp7Thjx6Hco4HjX7e545nilIknqGgiSpZyhIknqGgiSpN69CIcnBSb6d5K4kpw5djySNm3kTCkk2Af4n8AZgT+DoJHsOW5UkjZd5EwrAvsBdVXV3VT0GnAMcOnBNkjRW5tP3FHYG7pvwejXwm+vPlOR44Pj28tEk397I7e0I/PNGLjuEhVTv02rNhweqZDQLet+ub57t6+fUvp1P8uFnVe8Lp5own0JhJFW1HFj+bNeTZFVVLZuBkubEQqp3IdUKC6vehVQrLKx6F1KtMHv1zqfmo/uBXSe83qWNkyTNkfkUCt8Edk+yW5LNgKOAzw9ckySNlXnTfFRVjyf5D8BlwCbAZ6vqtlnc5LNugppjC6nehVQrLKx6F1KtsLDqXUi1wizVm6qajfVKkhag+dR8JEkamKEgSeqNXSgk+WySNUluHbqWDUmya5Irktye5LYk7xq6pukk2TzJtUluavX+56Fr2pAkmyS5IcnFQ9eyIUnuSXJLkhuTrBq6nukk2S7JeUm+leSOJPsPXdNUkry07dN1jx8mOWnouqaS5OT2/3VrkrOTbD6j6x+3awpJXgU8Cnyuql42dD3TSbITsFNVXZ9kG+A64LCqun3g0iaVJMBWVfVokk2BrwPvqqpvDFzalJKcAiwDnl9Vbxy6nukkuQdYVlXz/gtWSVYAX6uq09unCbesqoeHrmtDWnc79wO/WVX3Dl3P+pLsTPd/tWdV/TTJucAXq+rMmdrG2J0pVNVVwPeHrmMUVfVAVV3fhn8E3EH3ze95qTqPtpebtse8PepIsgtwCHD60LU8lyTZFngVcAZAVT22EAKhOQj4p/kYCBMsArZIsgjYEvh/M7nysQuFhSrJUmBv4JphK5lea465EVgDrKyq+VzvJ4D3Ak8OXciICvhykutady/z1W7AWuCvW9Pc6Um2GrqoER0FnD10EVOpqvuBjwLfBR4AHqmqL8/kNgyFBSDJ1sD5wElV9cOh65lOVT1RVXvRfSN93yTzsokuyRuBNVV13dC1PAOvrKp96HoSfkdrCp2PFgH7AJ+uqr2BHwPzviv81sz1JuDvhq5lKkm2p+sodDfgV4GtkvzeTG7DUJjnWtv8+cBZVXXB0PWMqjUXXAEcPHQtUzgQeFNrpz8HeE2Svxm2pOm1o0Sqag1wIV3PwvPRamD1hLPE8+hCYr57A3B9VT00dCHTeC3wnapaW1W/AC4ADpjJDRgK81i7cHsGcEdVfWzoejYkyeIk27XhLYDXAd8atqrJVdX7q2qXqlpK12Twlaqa0SOumZRkq/ZhA1pTzOuBefkJuqp6ELgvyUvbqIOAefnhiPUczTxuOmq+C+yXZMv2/nAQ3bXGGTN2oZDkbOBq4KVJVic5buiapnEg8Fa6o9h1H5f77aGLmsZOwBVJbqbry2plVc37j3ouEEuArye5CbgWuKSqLh24pum8Ezir/S3sBfz3geuZVgva19Edec9b7ezrPOB64Ba69/AZ7e5i7D6SKkma2tidKUiSpmYoSJJ6hoIkqWcoSJJ6hoIkqWcojLkkT7SPut6a5Avrvmewket6dMNzTbnsia03zbPWG7/XxI/hJvnTJP9xY7czG5L80Qys454kO85EPSNs67Ake87FtrTwGAr6aVXt1XqM/T7wjoHqeDvwuqp6y3rj9wLm83czAEYKhdYD53xwGGAoaFKGgia6mtYLa5J9k1zdOjT7x3XfTk1ybJILklya5M4kf7b+SpLs2JY9ZJJpp7SzklvX9Vmf5DPAi4AvJTl5wrybAf8FOLKdzRzZJu2Z5Mokdyc5ccL8v9fu53Bjkr+c7E24HZF/aN09CZLsk+SyJP+U5IQ2T5J8pNV4y7rtJtkpyVUTzqx+K8lpdD1W3rj+WU5b5tEk/6N96Wz/EWt82jxJTkjykQnzHJvkU23471snebdlQkd5bdv/Ld39Lb6RZEmSA+j69/lIW/+L19v2mUk+0/bN/03XRxRJlib5WpLr2+OAafbJJm096/bfyW3eP0zyzVbP+Um2bONf3Oq7JckHJ55xJnlPW+bmLID7czwnVJWPMX4Aj7bnTeg6Aju4vX4+sKgNvxY4vw0fC9wNbAtsDtwL7LpuXXTfvL2G7qh//W39Bt23MLcCtgZuA/Zu0+4BdpxkmWOBT014/afAPwK/AuwIfI+ui+5fB74AbNrm+1/A2yZZ3z3Av2/DHwduBrYBFgMPtfH/FljZ9skSuq4FdgLeDfzxhP21zcR9OMX+LeCINjxljet+/qnmafXdNWG9X6LrIA9gh/a8BV3XFy+YsO3facN/BvxJGz4TOHyKes8ELqU7YNydrh+jzem6aN68zbM7sKoNP22ftN/zygnr3K49v2DCuA8C72zDFwNHt+ETeOpv8vV039ZNq+di4FVD/8881x+L0LjbIl1X1zvT9aGyso3fFliRZHe6N5dNJyxzeVU9ApDkduCFwH1tnsuBd1TVVyfZ1iuBC6vqx23ZC4DfAm54hjVfUlU/B36eZA3dG/dBdG9G30wC3RvkmimW/3x7vgXYurp7Vfwoyc/TXVN5JXB2VT0BPJTkq8Ar6Lru+Gy6Tgr/vqpuHKHWJ+g6NGTEGiedp6rWtjOj/YA7gT2Af2jLnJjkzW14V7o37e8Bj9G9kUJ3g6bXjVAvwLlV9SRwZ5K727a+A3wqyV7tZ/q1Nu/T9klb5kVJ/gK4BFjXtfPLknwQ2I7uoOCyNn5/uiYtgP9N1zU0dKHwep76+9i6/WxXjfhzaCMYCvppVe3VTuUvo7um8OfAfwWuqKo3p7uXw5UTlvn5hOEneOrv6HG6N59/DUwWCjNlsu0HWFFV738Gyz+53rqeZJr/iaq6Kl131YcAZyb5WFV9bgPb+lkLF0ascbp5zgGOoOtk8MKqqiSvpjuT27+qfpLkSroje4BfVDvk5pd/Txuyft83BZwMPAS8nO6o/Wcw9T5J8nK6v4MTWs1/QHcWclhV3ZTkWODVG6gjwIeq6i9HrFszwGsKAqCqfgKcCLw73R2dtqW7LSF0TTgjrYbun3+PJO+bZPrXgMPS9fC4FfDmNm46P6JrktiQy4HDk/wLgCQ7JHnhiHVPVueRrW18Md1dxK5t63uoqv6K7m5t67qD/kU7Up6JGqeb50K6vvSPpgsI6H5PP2iBsAew3wh1bGif/m6S57XrDS8Cvt2280A7g3grXVMRk+2TdJ+iel5VnQ/8CU/tp22AB9q+mviBgm/QNdlB12PtOpcBf5DufiIk2XndftHsMRTUq6ob6NrYj6Zrg/5Qkht4BmeU7aj4aLqeXd++3rTr6Y4Wr6W77nB62+Z0rqC7sDzxQvNk272d7g3oy+l65lxJdx1gY1xItx9uAr4CvLe67qBfDdzU9smRwCfb/MuBmye70PxMa5xunqr6AV0T3wur6tq2yKXAoiR3AKfRvcFuyDnAe9J9iODFk0z/Lt3v6EvACVX1M7prG8eku2C+B92Nc2DyfbIzcGVrlvwbYN1Zz3+i+73/A7/cpfpJwCnt530J8Ej7eb9M15x0dZJb6HoHHeUAQc+CvaRK6iU5E7i4qs6bw21uSdeMWUmOorvofOhcbV+/zGsKkob2G3QXsQM8TNcEqYF4piBJ6nlNQZLUMxQkST1DQZLUMxQkST1DQZLU+/8Lss3sFgIMnAAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "166 / 166 questions predicted.------------------------------------------------------------ \n",
            "\n",
            "Theme: Solar_energy\n",
            "Total Questions: 176\n",
            "Total Paragraphs: 51\n",
            "Total queries: 176\n",
            "In top 10 results, number of queries for which -\n",
            "\tRelevant paragraph found: 172 (98 %)\n",
            "\tRelevant paragraph NOT found: 4 (2 %)\n",
            "Mean Rank for which relevant paragraph found: 1.49\n",
            "\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAX8UlEQVR4nO3de7hddX3n8fcHInJTA+SUJ5OgQWVUhhkVIwVvjyNqUaxgxyKMVVDaDCPjfbSgndHO6CNWx1udalOgxKcUhnKpCMhlEMS2AoZruKikyCUMkFMVFe/B7/yx1llsDifJSXL2Xiec9+t59rPXXtfvWTk5n/37rb1/K1WFJEkA2/RdgCRp9jAUJEkdQ0GS1DEUJEkdQ0GS1JnXdwFbYsGCBbVkyZK+y5Ckrco111zzL1U1NtWyrToUlixZwsqVK/suQ5K2KknuXN8yu48kSR1DQZLUMRQkSR1DQZLUMRQkSR1DQZLUMRQkSR1DQZLUMRQkSZ2t+hvNW2LJcef3duw7Tji4t2NL0oYMraWQ5OQka5PcNMWy9yapJAva10nyuSSrk9yYZN9h1SVJWr9hdh+dAhw0eWaSPYBXAncNzH4VsFf7WAZ8YYh1SZLWY2ihUFVXAD+YYtGngfcDgzeHPgT4UjWuBOYnWTis2iRJUxvpheYkhwD3VNUNkxYtAu4eeL2mnTfVPpYlWZlk5fj4+JAqlaS5aWShkGRH4APAf9+S/VTV8qpaWlVLx8amHA5ckrSZRvnpo6cBewI3JAFYDFybZD/gHmCPgXUXt/MkSSM0spZCVa2qqt+qqiVVtYSmi2jfqroPOBd4c/sppP2BH1XVvaOqTZLUGOZHUk8Dvgk8I8maJEdvYPULgNuB1cBfAW8bVl2SpPUbWvdRVR2xkeVLBqYLOHZYtUiSpsdhLiRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQZWigkOTnJ2iQ3Dcz7RJJvJ7kxyTlJ5g8sOz7J6iTfSfI7w6pLkrR+w2wpnAIcNGneJcA+VfXvgO8CxwMk2Rs4HPg37TZ/kWTbIdYmSZrC0EKhqq4AfjBp3sVVta59eSWwuJ0+BDi9qn5ZVd8DVgP7Das2SdLU+rym8Fbgq+30IuDugWVr2nmPkmRZkpVJVo6Pjw+5REmaW3oJhSQfBNYBp27qtlW1vKqWVtXSsbGxmS9OkuaweaM+YJKjgNcAB1ZVtbPvAfYYWG1xO0+SNEIjbSkkOQh4P/DaqvrZwKJzgcOTPD7JnsBewNWjrE2SNMSWQpLTgJcCC5KsAT5E82mjxwOXJAG4sqqOqaqbk5wB3ELTrXRsVT00rNokSVMbWihU1RFTzD5pA+t/FPjosOqRJG2c32iWJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSZ2ihkOTkJGuT3DQwb9cklyS5rX3epZ2fJJ9LsjrJjUn2HVZdkqT1G2ZL4RTgoEnzjgMuraq9gEvb1wCvAvZqH8uALwyxLknSegwtFKrqCuAHk2YfAqxop1cAhw7M/1I1rgTmJ1k4rNokSVMb9TWF3avq3nb6PmD3dnoRcPfAemvaeY+SZFmSlUlWjo+PD69SSZqDervQXFUF1GZst7yqllbV0rGxsSFUJklz16hD4f6JbqH2eW07/x5gj4H1FrfzJEkjNOpQOBc4sp0+EvjywPw3t59C2h/40UA3kyRpROYNa8dJTgNeCixIsgb4EHACcEaSo4E7gcPa1S8AXg2sBn4GvGVYdUmS1m9ooVBVR6xn0YFTrFvAscOqRZI0PX6jWZLUMRQkSR1DQZLUMRQkSR1DQZLUMRQkSR1DQZLUMRQkSR1DQZLUMRQkSR1DQZLUMRQkSR1DQZLUMRQkSR1DQZLUMRQkSR1DQZLUMRQkSZ1phUKSF05nniRp6zbdlsKfT3OeJGkrNm9DC5McALwAGEvynoFFTwS23dyDJnk38IdAAauAtwALgdOB3YBrgDdV1a829xiSpE23sZbCdsDONOHxhIHHj4HXb84BkywC3gEsrap9aMLlcODjwKer6unAD4GjN2f/kqTNt8GWQlV9Hfh6klOq6s4ZPu4OSX4N7AjcC7wM+I/t8hXAh4EvzOAxJUkbscFQGPD4JMuBJYPbVNXLNvWAVXVPkk8CdwE/By6m6S56oKrWtautARZNtX2SZcAygCc/+cmbenhJ0gZMNxT+DvgicCLw0JYcMMkuwCHAnsAD7b4Pmu72VbUcWA6wdOnS2pJaJEmPNN1QWFdVM9WV83Lge1U1DpDkbOCFwPwk89rWwmLgnhk6niRpmqb7kdSvJHlbkoVJdp14bOYx7wL2T7JjkgAHArcAl/HwxesjgS9v5v4lSZtpui2FI9vn9w3MK+Cpm3rAqroqyZnAtcA64Dqa7qDzgdOTfKSdd9Km7luStGWmFQpVtedMHrSqPgR8aNLs24H9ZvI4kqRNM61QSPLmqeZX1ZdmthxJUp+m2330/IHp7WmuA1wLGAqS9Bgy3e6jtw++TjKfZkgKSdJjyOYOnf1Tmu8ZSJIeQ6Z7TeErNJ82gmasomcBZwyrKElSP6Z7TeGTA9PrgDuras0Q6pEk9Wha3UftwHjfphkhdRfAIa0l6TFoundeOwy4Gvh94DDgqiSbNXS2JGn2mm730QeB51fVWoAkY8D/Bc4cVmGSpNGb7qePtpkIhNb3N2FbSdJWYrothQuTXASc1r5+A3DBcEqSJPVlY/dofjqwe1W9L8nvAS9qF30TOHXYxUmSRmtjLYXPAMcDVNXZwNkASf5tu+x3h1qdJGmkNnZdYPeqWjV5ZjtvyVAqkiT1ZmOhMH8Dy3aYyUIkSf3bWCisTPJHk2cm+UPgmuGUJEnqy8auKbwLOCfJG3k4BJYC2wGvG2ZhkqTR22AoVNX9wAuS/Htgn3b2+VX1taFXJkkaueneT+Ey4LIh1yJJ6pnfSpYkdXoJhSTzk5yZ5NtJbk1yQJJdk1yS5Lb2eZc+apOkuayvlsJngQur6pnAs4FbgeOAS6tqL+DS9rUkaYRGHgpJngS8BDgJoKp+VVUPAIcAK9rVVgCHjro2SZrr+mgp7AmMA3+d5LokJybZiebb0/e269wH7D7VxkmWJVmZZOX4+PiISpakuaGPUJgH7At8oaqeC/yUSV1FVVU8fE9oJi1bXlVLq2rp2NjY0IuVpLmkj1BYA6ypqqva12fShMT9SRYCtM9r17O9JGlIRh4KVXUfcHeSZ7SzDgRuAc4FjmznHQl8edS1SdJcN92b7My0twOnJtkOuB14C01AnZHkaOBOmntBS5JGqJdQqKrracZQmuzAUdciSXqY32iWJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSp7dQSLJtkuuSnNe+3jPJVUlWJ/k/SbbrqzZJmqv6bCm8E7h14PXHgU9X1dOBHwJH91KVJM1hvYRCksXAwcCJ7esALwPObFdZARzaR22SNJf11VL4DPB+4Dft692AB6pqXft6DbBoqg2TLEuyMsnK8fHx4VcqSXPIyEMhyWuAtVV1zeZsX1XLq2ppVS0dGxub4eokaW6b18MxXwi8Nsmrge2BJwKfBeYnmde2FhYD9/RQmyTNaSNvKVTV8VW1uKqWAIcDX6uqNwKXAa9vVzsS+PKoa5OkuW42fU/hj4H3JFlNc43hpJ7rkaQ5p4/uo05VXQ5c3k7fDuzXZz2SNNfNppaCJKlnhoIkqWMoSJI6hoIkqWMoSJI6hoIkqWMoSJI6hoIkqWMoSJI6hoIkqWMoSJI6hoIkqWMoSJI6hoIkqWMoSJI6hoIkqWMoSJI6hoIkqWMoSJI6hoIkqTPyUEiyR5LLktyS5OYk72zn75rkkiS3tc+7jLo2SZrr+mgprAPeW1V7A/sDxybZGzgOuLSq9gIubV9LkkZo5KFQVfdW1bXt9E+AW4FFwCHAina1FcCho65Nkua6Xq8pJFkCPBe4Cti9qu5tF90H7L6ebZYlWZlk5fj4+EjqlKS5ordQSLIzcBbwrqr68eCyqiqgptquqpZX1dKqWjo2NjaCSiVp7uglFJI8jiYQTq2qs9vZ9ydZ2C5fCKztozZJmsv6+PRRgJOAW6vqUwOLzgWObKePBL486tokaa6b18MxXwi8CViV5Pp23geAE4AzkhwN3Akc1kNtkjSnjTwUquofgKxn8YGjrEWS9Eh+o1mS1DEUJEkdQ0GS1DEUJEkdQ0GS1DEUJEkdQ0GS1DEUJEkdQ0GS1DEUJEmdPsY+mvOWHHd+L8e944SDezmupK2HLQVJUsdQkCR1DAVJUsdrChoJr6NIWwdbCpKkji2FOaSvd+uSth62FCRJHVsK0pDMtZaZ128eG2wpSJI6s66lkOQg4LPAtsCJVXVCzyVJmoY+W0a2UmbOrAqFJNsC/xt4BbAG+FaSc6vqln4rk6RHeywG4WzrPtoPWF1Vt1fVr4DTgUN6rkmS5oxZ1VIAFgF3D7xeA/z24ApJlgHL2pcPJvnOZh5rAfAvm7ntMM3WumD21rbeuvLxEVfySFvd+erZZtc15H/nWXm+8vEtqusp61sw20Jho6pqObB8S/eTZGVVLZ2BkmbUbK0LZm9t1rVprGvTzLW6Zlv30T3AHgOvF7fzJEkjMNtC4VvAXkn2TLIdcDhwbs81SdKcMau6j6pqXZL/AlxE85HUk6vq5iEdbou7oIZkttYFs7c269o01rVp5lRdqaph7FeStBWabd1HkqQeGQqSpM6cC4UkJydZm+SmvmsZlGSPJJcluSXJzUne2XdNAEm2T3J1khvauv6075oGJdk2yXVJzuu7lglJ7kiyKsn1SVb2Xc+EJPOTnJnk20luTXLALKjpGe15mnj8OMm7+q4LIMm729/5m5KclmT7vmsCSPLOtqabh3Gu5tw1hSQvAR4EvlRV+/Rdz4QkC4GFVXVtkicA1wCH9j3ER5IAO1XVg0keB/wD8M6qurLPuiYkeQ+wFHhiVb2m73qgCQVgaVXNqi88JVkBfKOqTmw/3bdjVT3Qd10T2mFu7gF+u6ru7LmWRTS/63tX1c+TnAFcUFWn9FzXPjQjPewH/Aq4EDimqlbP1DHmXEuhqq4AftB3HZNV1b1VdW07/RPgVppvePeqGg+2Lx/XPmbFO4kki4GDgRP7rmW2S/Ik4CXASQBV9avZFAitA4F/7jsQBswDdkgyD9gR+H891wPwLOCqqvpZVa0Dvg783kweYM6FwtYgyRLgucBV/VbSaLtorgfWApdU1ayoC/gM8H7gN30XMkkBFye5ph2WZTbYExgH/rrtbjsxyU59FzXJ4cBpfRcBUFX3AJ8E7gLuBX5UVRf3WxUANwEvTrJbkh2BV/PIL/xuMUNhlkmyM3AW8K6q+nHf9QBU1UNV9Ryab5jv1zZhe5XkNcDaqrqm71qm8KKq2hd4FXBs22XZt3nAvsAXquq5wE+B4/ot6WFtd9Zrgb/ruxaAJLvQDMa5J/CvgJ2S/EG/VUFV3Qp8HLiYpuvoeuChmTyGoTCLtH32ZwGnVtXZfdczWdvdcBlwUN+1AC8EXtv2358OvCzJ3/RbUqN9l0lVrQXOoen/7dsaYM1AK+9MmpCYLV4FXFtV9/ddSOvlwPeqaryqfg2cDbyg55oAqKqTqup5VfUS4IfAd2dy/4bCLNFe0D0JuLWqPtV3PROSjCWZ307vQHOvi2/3WxVU1fFVtbiqltB0O3ytqnp/J5dkp/aDArTdM6+kafL3qqruA+5O8ox21oHAbLpPyRHMkq6j1l3A/kl2bP9vHkhzna93SX6rfX4yzfWEv53J/c+qYS5GIclpwEuBBUnWAB+qqpP6rQpo3vm+CVjV9t8DfKCqLuixJoCFwIr2kyHbAGdU1az5+OcstDtwTvN3hHnA31bVhf2W1Hk7cGrbVXM78Jae6wG68HwF8J/6rmVCVV2V5EzgWmAdcB2zZ7iLs5LsBvwaOHamPzAw5z6SKklaP7uPJEkdQ0GS1DEUJEkdQ0GS1DEUJEkdQ0EkeagdofKmJF+Z+F7CZu7rwY2vtd5t39GO3nnqpPnPSfLqgdcfTvJfN/c4w5DkAzOwjzuSLJiJeqZxrEOT7D2KY2nrYigI4OdV9Zx21NgfAMf2VMfbgFdU1RsnzX8OzRgvs9m0QqH9vsdscChgKOhRDAVN9k3a0VmT7Jfkm+0Aav808W3YJEclOTvJhUluS/Jnk3eSZEG77cFTLHtP2yq5aWI8+CRfBJ4KfDXJuwfW3Q74H8Ab2tbMG9pFeye5PMntSd4xsP4fpLn/w/VJ/nKqP8LtO/KPteusTLJvkouS/HOSY9p1kuQTbY2rJo6bZGGSKwZaVi9OcgLNaJrXT27ltNs8mOR/JbkBOGCaNT5qnSTHJPnEwDpHJfl8O/33aQbguzkDg/C1x/5omvthXJlk9yQvoBln6BPt/p826dinJPlie26+m2acKZIsSfKNJNe2jxds4Jxs2+5n4vy9u133j5J8q63nrDSDupHkaW19q5J8ZLDFmeR97TY3Zpbdz+Mxqap8zPEH8GD7vC3NgGQHta+fCMxrp18OnNVOH0XzjdgnAdsDdwJ7TOyL5lu9V9G86598rOcBq4CdgJ2Bm4HntsvuABZMsc1RwOcHXn8Y+Cfg8cAC4Ps0Q3o/C/gK8Lh2vb8A3jzF/u4A/nM7/WngRuAJwBhwfzv/PwCXtOdkd5phDxYC7wU+OHC+njB4Dtdzfgs4rJ1eb40TP//61mnrWz2w36/SDL4HsGv7vAPNsBq7DRz7d9vpPwP+pJ0+BXj9euo9hWawtW2AvWjGTdqeZvjo7dt19gJWttOPOiftv/MlA/uc3z7vNjDvI8Db2+nzgCPa6WN4+HfylTTfJE5bz3nAS/r+P/NYfsy5YS40pR3SDK2xiGZ8l0va+U+iGeJiL5o/Lo8b2ObSqvoRQJJbgKcAd7frXErz9fuvT3GsFwHnVNVP223PBl5MM4zApji/qn4J/DLJWpo/3AfS/DH6VpphJnagGe57Kue2z6uAnau5h8VPkvwyzTWVFwGnVdVDwP1Jvg48H/gWcHKawQv/vqqun2rnkzxEM9Ah06xxynWqarxtGe0P3AY8E/jHdpt3JHldO70HzR/t79PciGViWJJraIaTmI4zquo3wG1Jbm+P9T3g80me0/5M/7pd91HnpN3mqUn+HDifZlRPgH2SfASYT/Om4KJ2/gE0XVrQjOXzyXb6le1j4vdj5/Znu2KaP4c2kaEgaK8ptE35i2iuKXwO+J/AZVX1ujT3eLh8YJtfDkw/xMO/S+to/vj8Ds0NQIZlquMHWFFVx2/C9r+ZtK/fsIH/F1V1RZqhsA8GTknyqar60kaO9Ys2XJhmjRta53TgMJpBCc+pqkryUpqW3AFV9bMkl9O8swf4dbVvuXnkv9PGTB7/poB3A/cDz6Z51/4LWP85SfJsmt+DY9qa30rTCjm0qm5IchTNOGQbEuBjVfWX06xbW8hrCupU1c+AdwDvTXO3qSfR3B4Rmi6cae2G5j//M5P88RTLvwEcmmb0yZ2A17XzNuQnNF0SG3Mp8Po8PIrkrkmeMs26p6rzDW3f+BjNXcuubvd3f1X9Fc0d3yaGn/51+055Jmrc0Drn0IzzfwRNQEDz7/TDNhCeCew/jTo2dk5/P8k27fWGpwLfaY9zb9uCeBNNVxFTnZM0n6LapqrOAv6Eh8/TE4B723M1+IGCK2m67KAZ9XbCRcBb09xnhCSLJs6LhsNQ0CNU1XU0fexH0PRBfyzJdWxCq7J9V3wEzT0O3jZp2bU07xavprnucGJ7zA25jObC8uCF5qmOewvNH6CLk9xI0w22cLp1T3IOzXm4Afga8P5qhp9+KXBDe07eAHy2XX85cONUF5o3tcYNrVNVP6Tp4ntKVV3dbnIhMC/JrcAJNH9gN+Z04H1pPkTwtCmW30Xzb/RVmnsA/4Lm2saRaS6YP5PmRj0w9TlZBFzedkv+DTDR6vlvNP/u/8gjh2B/F/Ce9ud9OvCj9ue9mKY76ZtJVtHcB2I6bxC0mRwlVdIjJDkFOK+qzhzhMXek6casJIfTXHQ+ZFTH18O8piBpNngezUXsAA/QdEGqB7YUJEkdrylIkjqGgiSpYyhIkjqGgiSpYyhIkjr/H2H3zvsBhg3TAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "176 / 176 questions predicted.------------------------------------------------------------ \n",
            "\n",
            "Theme: Buddhism\n",
            "Total Questions: 422\n",
            "Total Paragraphs: 144\n",
            "Total queries: 422\n",
            "In top 10 results, number of queries for which -\n",
            "\tRelevant paragraph found: 370 (88 %)\n",
            "\tRelevant paragraph NOT found: 52 (12 %)\n",
            "Mean Rank for which relevant paragraph found: 1.71\n",
            "\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAVSElEQVR4nO3de7SddX3n8fdHwBtQkZJmYUCjNpZSZwk2MtzqoqVaL22BGctltQotY8qIIsrYAdtZdWZkpFWxtc5gURlwFWEol4qAIEaUXrgF5I4ODIIkE0iqFlFHFPjOH88vTzbJSbIDZ5/nhPN+rbXXfvbvuX3Pc87Zn/1c9u9JVSFJEsCzhi5AkjR7GAqSpJ6hIEnqGQqSpJ6hIEnqbT10AU/HTjvtVAsXLhy6DEnaotx4443/XFXzphq3RYfCwoULWbZs2dBlSNIWJcn9Gxrn4SNJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUm+L/kbz07HwxEsHW/d9p7x5sHVL0sa4pyBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6hkKkqTexEIhya5JrkpyZ5I7kry7tX8gyYokN7fHm0bmOSnJPUm+meQ3JlWbJGlqk7yfwmPACVV1U5LtgRuTXNnGfayqPjI6cZLdgcOBXwJeBHw5ySuq6vEJ1ihJGjGxPYWqWllVN7XhR4C7gAUbmeUg4NyqerSqvgXcA+w1qfokSeubkXMKSRYCewLXtaZ3Jrk1yRlJXtjaFgAPjMy2nClCJMmSJMuSLFu9evUEq5akuWfioZBkO+AC4Piq+j5wGvByYA9gJfDRzVleVZ1eVYuravG8efOmvV5JmssmGgpJtqELhLOr6kKAqnqoqh6vqieAT7H2ENEKYNeR2XdpbZKkGTLJq48CfAa4q6pOHWnfeWSyQ4Db2/DFwOFJnpPkpcAi4PpJ1SdJWt8krz7aD3grcFuSm1vb+4EjkuwBFHAf8IcAVXVHkvOAO+muXDrWK48kaWZNLBSq6h+ATDHqso3MczJw8qRqkiRtnN9oliT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUm9ioZBk1yRXJbkzyR1J3t3ad0xyZZK72/MLW3uSfDzJPUluTfLqSdUmSZraJPcUHgNOqKrdgb2BY5PsDpwILK2qRcDS9hrgjcCi9lgCnDbB2iRJU5hYKFTVyqq6qQ0/AtwFLAAOAs5qk50FHNyGDwI+W51rgR2S7Dyp+iRJ65uRcwpJFgJ7AtcB86tqZRv1IDC/DS8AHhiZbXlrkyTNkImHQpLtgAuA46vq+6PjqqqA2szlLUmyLMmy1atXT2OlkqSJhkKSbegC4eyqurA1P7TmsFB7XtXaVwC7jsy+S2t7kqo6vaoWV9XiefPmTa54SZqDJnn1UYDPAHdV1akjoy4GjmzDRwKfH2l/W7sKaW/g4ZHDTJKkGbD1BJe9H/BW4LYkN7e29wOnAOclORq4Hzi0jbsMeBNwD/Aj4PcnWJskaQoTC4Wq+gcgGxh94BTTF3DspOqRJG2a32iWJPUMBUlSz1CQJPUMBUlSz1CQJPUMBUlSz1CQJPUMBUlSz1CQJPUMBUlSz1CQJPUMBUlSz1CQJPUMBUlSz1CQJPUMBUlSz1CQJPUMBUlSz1CQJPUMBUlSz1CQJPUMBUlSz1CQJPUMBUlSz1CQJPXGCoUk+43TJknaso27p/BXY7ZJkrZgW29sZJJ9gH2BeUneOzLqZ4CtJlmYJGnmbTQUgGcD27Xpth9p/z7wlkkVJUkaxkZDoaq+BnwtyZlVdf/mLDjJGcBvAquq6pWt7QPA24HVbbL3V9VlbdxJwNHA48BxVXXF5qxPkvT0bWpPYY3nJDkdWDg6T1X92kbmORP4BPDZddo/VlUfGW1IsjtwOPBLwIuALyd5RVU9PmZ9kqRpMG4o/C3wSeDTdJ/kN6mqrk6ycMzlHwScW1WPAt9Kcg+wF3DNmPNLkqbBuKHwWFWdNk3rfGeStwHLgBOq6nvAAuDakWmWt7b1JFkCLAF48YtfPE0lSZJg/EtSv5DkHUl2TrLjmsdTWN9pwMuBPYCVwEc3dwFVdXpVLa6qxfPmzXsKJUiSNmTcPYUj2/P7RtoKeNnmrKyqHloznORTwCXt5Qpg15FJd2ltkqQZNFYoVNVLp2NlSXauqpXt5SHA7W34YuBzSU6lO9G8CLh+OtYpSRrfWKHQzgGsp6rWvbJodJ5zgAOAnZIsB/4UOCDJHnR7GfcBf9iWc0eS84A7gceAY73ySJJm3riHj14zMvxc4EDgJta/3LRXVUdM0fyZjUx/MnDymPVIkiZg3MNH7xp9nWQH4NyJVCRJGsxT7Tr7h8C0nGeQJM0e455T+ALdeQDoOsL7ReC8SRUlSRrGuOcURruleAy4v6qWT6AeSdKAxjp81DrG+wZdT6kvBH4yyaIkScMY985rh9J9b+B3gEOB65LYdbYkPcOMe/joj4HXVNUqgCTzgC8D50+qMEnSzBv36qNnrQmE5jubMa8kaQsx7p7C5UmuAM5prw8DLptMSZKkoWzqHs0/D8yvqvcl+TfA/m3UNcDZky5OkjSzNrWn8BfASQBVdSFwIUCSf9XG/dZEq5MkzahNnReYX1W3rdvY2hZOpCJJ0mA2FQo7bGTc86azEEnS8DYVCsuSvH3dxiT/DrhxMiVJkoayqXMKxwMXJfld1obAYuDZdDfJkSQ9g2w0FNrtM/dN8qvAK1vzpVX1lYlXJkmacePeT+Eq4KoJ1yJJGpjfSpYk9QwFSVLPUJAk9QwFSVLPUJAk9QwFSVLPUJAk9QwFSVLPUJAk9QwFSVLPUJAk9QwFSVJvYqGQ5Iwkq5LcPtK2Y5Irk9zdnl/Y2pPk40nuSXJrkldPqi5J0oZNck/hTOAN67SdCCytqkXA0vYa4I3AovZYApw2wbokSRswsVCoqquB767TfBBwVhs+Czh4pP2z1bkW2CHJzpOqTZI0tZk+pzC/qla24QeB+W14AfDAyHTLW9t6kixJsizJstWrV0+uUkmagwY70VxVBdRTmO/0qlpcVYvnzZs3gcokae6a6VB4aM1hofa8qrWvAHYdmW6X1iZJmkEzHQoXA0e24SOBz4+0v61dhbQ38PDIYSZJ0gwZ6x7NT0WSc4ADgJ2SLAf+FDgFOC/J0cD9wKFt8suANwH3AD8Cfn9SdUmSNmxioVBVR2xg1IFTTFvAsZOqRZI0Hr/RLEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqbT3ESpPcBzwCPA48VlWLk+wI/C9gIXAfcGhVfW+I+iRprhpyT+FXq2qPqlrcXp8ILK2qRcDS9lqSNINm0+Gjg4Cz2vBZwMED1iJJc9JQoVDAl5LcmGRJa5tfVSvb8IPA/KlmTLIkybIky1avXj0TtUrSnDHIOQVg/6pakeTngCuTfGN0ZFVVkppqxqo6HTgdYPHixVNOI0l6agbZU6iqFe15FXARsBfwUJKdAdrzqiFqk6S5bMZDIcm2SbZfMwy8HrgduBg4sk12JPD5ma5Nkua6IQ4fzQcuSrJm/Z+rqsuT3ACcl+Ro4H7g0AFqk6Q5bcZDoaruBV41Rft3gANnuh5J0lqz6ZJUSdLADAVJUm+oS1LntIUnXjrIeu875c2DrFfSlsM9BUlSz1CQJPUMBUlSz1CQJPUMBUlSz1CQJPUMBUlSz1CQJPUMBUlSz1CQJPUMBUlSz1CQJPUMBUlSz1CQJPXsOnsOGarLbrDbbmlL4Z6CJKlnKEiSeoaCJKlnKEiSeoaCJKnn1UeaEUNd+eRVT9LmcU9BktQzFCRJPUNBktTznIKkaeE35p8ZDAU9ow35RjUU3yD1dMy6UEjyBuAvga2AT1fVKQOXJG1R5mIQDuWZuHc0q84pJNkK+O/AG4HdgSOS7D5sVZI0d8yqUAD2Au6pqnur6ifAucBBA9ckSXPGbDt8tAB4YOT1cuBfj06QZAmwpL38QZJvzlBtk7IT8M9DFzGLuD2ezO2x1ga3Rf5shiuZBfJnT+tv4yUbGjHbQmGTqup04PSh65guSZZV1eKh65gt3B5P5vZYy23xZJPaHrPt8NEKYNeR17u0NknSDJhtoXADsCjJS5M8GzgcuHjgmiRpzphVh4+q6rEk7wSuoLsk9YyqumPgsibtGXMobJq4PZ7M7bGW2+LJJrI9UlWTWK4kaQs02w4fSZIGZChIknqGwkCS7JrkqiR3JrkjybuHrmloSbZK8vUklwxdy9CS7JDk/CTfSHJXkn2GrmlISd7T/k9uT3JOkucOXdNMSnJGklVJbh9p2zHJlUnubs8vnI51GQrDeQw4oap2B/YGjrVLD94N3DV0EbPEXwKXV9VuwKuYw9slyQLgOGBxVb2S7iKUw4etasadCbxhnbYTgaVVtQhY2l4/bYbCQKpqZVXd1IYfofunXzBsVcNJsgvwZuDTQ9cytCQvAF4LfAagqn5SVf8ybFWD2xp4XpKtgecD/3fgemZUVV0NfHed5oOAs9rwWcDB07EuQ2EWSLIQ2BO4bthKBvUXwB8BTwxdyCzwUmA18D/b4bRPJ9l26KKGUlUrgI8A3wZWAg9X1ZeGrWpWmF9VK9vwg8D86ViooTCwJNsBFwDHV9X3h65nCEl+E1hVVTcOXcsssTXwauC0qtoT+CHTdGhgS9SOlR9EF5YvArZN8nvDVjW7VPfdgmn5foGhMKAk29AFwtlVdeHQ9QxoP+C3k9xH1zPuryX5m2FLGtRyYHlVrdlzPJ8uJOaqXwe+VVWrq+qnwIXAvgPXNBs8lGRngPa8ajoWaigMJEnojhnfVVWnDl3PkKrqpKrapaoW0p1A/EpVzdlPglX1IPBAkl9oTQcCdw5Y0tC+Deyd5Pnt/+ZA5vCJ9xEXA0e24SOBz0/HQg2F4ewHvJXuU/HN7fGmoYvSrPEu4OwktwJ7AP9t4HoG0/aYzgduAm6je9+aU11eJDkHuAb4hSTLkxwNnAK8LsnddHtT03KXSru5kCT13FOQJPUMBUlSz1CQJPUMBUlSz1CQJPUMBZHk8XZJ7O1JvpBkh6exrB88jXmPaz2Cnr1O+x6jl+sm+UCS//BU1zMJSd4/Dcu4L8lO01HPGOs62A4YNRVDQQD/r6r2aD1Qfhc4dqA63gG8rqp+d532PYDZ/h2OsUIhyVaTLmRMBwOGgtZjKGhd19B6a02yV5JrWqds/7TmG7ZJjkpyYZLLW1/uf77uQpLs1OZ98xTj3tv2Sm5Pcnxr+yTwMuCLSd4zMu2zgf8CHNb2Zg5ro3ZP8tUk9yY5bmT630tyfZv2r6d6E26fyD/UplmW5NVJrkjyf5Ic06ZJkg+3Gm9bs94kOye5emTP6leSnELXg+fN6+7ltHl+kOSjSW4B9hmzxvWmSXJMkg+PTHNUkk+04b9LcmO6ew4sWWfdJye5Jcm1SeYn2Rf4beDDbfkvX2fdZyb5ZNs2/7v1TUWShUn+PslN7bHvRrbJVm05a7bfe9q0b09yQ6vngiTPb+0vb/XdluSDo3ucSd7X5rk1yX9ed1tpmlWVjzn+AH7QnrcC/hZ4Q3v9M8DWbfjXgQva8FHAvcALgOcC9wO7rlkWXW+N19F96l93Xb9M963UbYHtgDuAPdu4+4CdppjnKOATI68/APwT8BxgJ+A7wDbALwJfALZp0/0P4G1TLO8+4N+34Y8BtwLbA/OAh1r7vwWubNtkPl1XCzsDJwB/PLK9th/dhhvYvgUc2oY3WOOan39D07T67hlZ7heB/dvwju35ecDtwM+OrPu32vCfA3/Shs8E3rKBes8ELqf70LiIri+m59J1Wf3cNs0iYFkbXm+btN/zlSPL3KE9/+xI2weBd7XhS4Aj2vAxrP2bfD3dt5fT6rkEeO3Q/zPP5MfWSO1TLt0ewl10b4bQvemflWQR3ZvLNiPzLK2qhwGS3Am8BHigTbMUOLaqvjbFuvYHLqqqH7Z5LwR+Bfj6ZtZ8aVU9CjyaZBXdG/eBdG9GNySB7g1yQ52EXdyebwO2q+6eFo8keTTdOZX9gXOq6nG6jse+BrwGuAE4I11nhn9XVTePUevjdB0fMmaNU05TVavbntHewN3AbsA/tnmOS3JIG96V7k37O8BP6N5IAW4EXjdGvQDnVdUTwN1J7m3r+hbwiSR7tJ/pFW3a9bZJm+dlSf4KuBRY09X1K5N8ENiB7kPBFa19H9beD+BzdF1lQxcKr2ft38d27We7esyfQ5vJUBC0cwptV/4KunMKHwf+K3BVVR2S7p4PXx2Z59GR4cdZ+7f0GN2bz28AU4XCdJlq/QHOqqqTNmP+J9ZZ1hNs5P+iqq5O8lq6GwKdmeTUqvrsJtb14xYujFnjxqY5FzgU+AZduFaSA+j25Papqh8l+SrdJ3uAn1b7yM2Tf0+bsm7/NwW8B3iI7k5wzwJ+DBveJkleRfd3cEyr+Q/o9kIOrqpbkhwFHLCJOgJ8qKr+esy69TR5TkG9qvoR3W0PT0h3h6sXACva6KPGXQzdP/9uSf7jFOP/Hjg4XY+X2wKHtLaNeYTukMSmLAXekuTnoL+H7UvGrHuqOg9rx8bn0d0J7fq2vIeq6lN0d4lb06X1T9sn5emocWPTXER3b4Ej6AICut/T91og7EZ3e9dN2dQ2/Z0kz2rnG14GfLOtZ2Xbg3gr3aEiptom6a6ielZVXQD8CWu30/bAyratRi8ouJbukB08+VabVwB/kO6+IyRZsGa7aDIMBT1JVX2d7hj7EXTHoD+U5Otsxl5l+1R8BF0PsO9YZ9xNdJ8Wr6c77/Dpts6NuYruxPLoieap1nsn3RvQl9L1Lnol3XmAp+Iiuu1wC/AV4I+q69L6AOCWtk0Oo7uXMnTHvW+d6kTz5ta4sWmq6nt0h/heUlXXt1kuB7ZOchddT5nXjvHznQu8L91FBC+fYvy36X5HXwSOqaof053bODLdCfPd6G7+A1NvkwXAV9thyb8B1uz1/Ce63/s/0u3trHE88N728/488HD7eb9EdzjpmiS30fWWOs4HBD1F9pIq6UmSnAlcUlXnz+A6n093GLOSHE530vmgmVq/1vKcgqTZ4JfpTmIH+Be6Q5AagHsKkqSe5xQkST1DQZLUMxQkST1DQZLUMxQkSb3/Dx3Ex2fUrGVtAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "422 / 422 questions predicted.------------------------------------------------------------ \n",
            "\n",
            "Theme: American_Idol\n",
            "Total Questions: 525\n",
            "Total Paragraphs: 125\n",
            "Total queries: 525\n",
            "In top 10 results, number of queries for which -\n",
            "\tRelevant paragraph found: 405 (77 %)\n",
            "\tRelevant paragraph NOT found: 120 (23 %)\n",
            "Mean Rank for which relevant paragraph found: 2.34\n",
            "\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAUF0lEQVR4nO3dfbRddX3n8fcHgqJARZo0iwE0ajOljLNEGx1A6qKlWoS2wIzlYbUaWqcZRqqijh1sO6vOjFamtvbJKZYqQ1ylUMpDRVGQiSidFoSAPKMDg0HCBJKqRdQRBb7zx973l8PlJjmBe+6+yX2/1rrr7PPbT9+7k3s+Z//2Ob+dqkKSJIBdhi5AkjR/GAqSpMZQkCQ1hoIkqTEUJEnNoqELeCYWL15cy5YtG7oMSdqh3Hjjjf9YVUtmmrdDh8KyZctYu3bt0GVI0g4lyX1bmmf3kSSpMRQkSY2hIElqDAVJUmMoSJIaQ0GS1BgKkqTGUJAkNYaCJKnZob/R/EwsO+Pywfa97sxjBtu3JG2NZwqSpMZQkCQ1hoIkqTEUJEmNoSBJagwFSVJjKEiSGkNBktQYCpKkxlCQJDWGgiSpMRQkSY2hIElqDAVJUmMoSJIaQ0GS1BgKkqTGUJAkNYaCJKkxFCRJjaEgSWoMBUlSYyhIkhpDQZLUGAqSpMZQkCQ1EwuFJAckuTrJnUnuSPL2vn2fJFclubt/fH7fniR/kuSeJLcmecWkapMkzWySZwqPAe+qqoOAQ4DTkhwEnAGsqarlwJr+OcDrgeX9zyrgrAnWJkmawcRCoao2VNVN/fQjwF3AfsCxwOp+sdXAcf30scDHq3MdsHeSfSdVnyTpqebkmkKSZcDLgS8CS6tqQz/rQWBpP70fcP/Iauv7tunbWpVkbZK1mzZtmljNkrQQTTwUkuwJXAycXlXfGp1XVQXU9myvqs6uqhVVtWLJkiWzWKkkaaKhkGQ3ukA4r6ou6ZsfmuoW6h839u0PAAeMrL5/3yZJmiOT/PRRgI8Bd1XVh0ZmXQas7KdXAp8YaX9T/ymkQ4CHR7qZJElzYNEEt/1q4I3AbUlu7tt+EzgTuDDJm4H7gBP6eZ8GjgbuAb4L/MoEa5MkzWBioVBV/wvIFmYfOcPyBZw2qXokSdvmN5olSY2hIElqDAVJUmMoSJIaQ0GS1BgKkqTGUJAkNYaCJKkxFCRJjaEgSWoMBUlSYyhIkhpDQZLUGAqSpMZQkCQ1hoIkqTEUJEmNoSBJagwFSVJjKEiSGkNBktQYCpKkxlCQJDWGgiSpMRQkSY2hIElqDAVJUmMoSJIaQ0GS1BgKkqTGUJAkNYaCJKkxFCRJjaEgSWoMBUlSM7FQSHJOko1Jbh9pe2+SB5Lc3P8cPTLvPUnuSfKVJD87qbokSVs2yTOFc4GjZmj/w6o6uP/5NECSg4CTgH/Rr/NnSXadYG2SpBlMLBSq6hrgG2MufixwQVU9WlVfBe4BXjWp2iRJMxvimsKvJ7m17156ft+2H3D/yDLr+7anSLIqydokazdt2jTpWiVpQZnrUDgLeAlwMLAB+IPt3UBVnV1VK6pqxZIlS2a7Pkla0OY0FKrqoap6vKqeAP6CzV1EDwAHjCy6f98mSZpDcxoKSfYdeXo8MPXJpMuAk5I8O8mLgOXA9XNZmyQJFk1qw0nOB44AFidZD/wOcESSg4EC1gH/DqCq7khyIXAn8BhwWlU9PqnaJEkzm1goVNXJMzR/bCvLvx94/6TqkSRtm99oliQ1hoIkqTEUJEmNoSBJagwFSVJjKEiSmrFCIcmrx2mTJO3Yxj1T+NMx2yRJO7CtfnktyaHAYcCSJO8cmfVDgPc7kKSdzLa+0fwsYM9+ub1G2r8FvGFSRUmShrHVUKiqLwBfSHJuVd03RzVJkgYy7thHz05yNrBsdJ2q+ulJFCVJGsa4ofA3wEeAjwKOXipJO6lxQ+GxqjpropVIkgY37kdSP5nkLUn2TbLP1M9EK5MkzblxzxRW9o/vHmkr4MWzW44kaUhjhUJVvWjShUiShjdWKCR500ztVfXx2S1HkjSkcbuPXjkyvTtwJHATYChI0k5k3O6jt44+T7I3cMFEKpIkDebpDp39HcDrDJK0kxn3msIn6T5tBN1AeD8OXDipoiRJwxj3msLvj0w/BtxXVesnUI8kaUBjdR/1A+N9mW6k1OcD359kUZKkYYx757UTgOuBXwROAL6YxKGzJWknM2730W8Br6yqjQBJlgD/E7hoUoVJkubeuJ8+2mUqEHpf3451JUk7iHHPFK5IciVwfv/8RODTkylJkjSUbd2j+UeBpVX17iT/Gji8n3UtcN6ki5Mkza1tnSn8EfAegKq6BLgEIMm/7Of9/ESrkyTNqW1dF1haVbdNb+zblk2kIknSYLYVCntvZd5zZrMQSdLwthUKa5P82vTGJP8WuHEyJUmShrKtawqnA5cm+SU2h8AK4FnA8ZMsTJI097YaClX1EHBYkp8CXto3X15Vn5t4ZZKkOTfu/RSuBq7eng0nOQf4OWBjVb20b9sH+Gu6i9TrgBOq6ptJAvwxcDTwXeCUqrppe/YnSXrmJvmt5HOBo6a1nQGsqarlwJr+OcDrgeX9zyrgrAnWJUnagomFQlVdA3xjWvOxwOp+ejVw3Ej7x6tzHbB3kn0nVZskaWZzPX7R0qra0E8/CCztp/cD7h9Zbn3f9hRJViVZm2Ttpk2bJlepJC1Agw1qV1XF5ru5bc96Z1fViqpasWTJkglUJkkL11yHwkNT3UL949TIqw8AB4wst3/fJkmaQ3MdCpcBK/vplcAnRtrflM4hwMMj3UySpDky7tDZ2y3J+cARwOIk64HfAc4ELkzyZuA+uru4QTcM99HAPXQfSf2VSdUlSdqyiYVCVZ28hVlHzrBsAadNqhZJ0ni8e5okqTEUJEmNoSBJagwFSVJjKEiSGkNBktQYCpKkxlCQJDWGgiSpMRQkSY2hIElqDAVJUmMoSJIaQ0GS1BgKkqTGUJAkNYaCJKkxFCRJjaEgSWoMBUlSYyhIkhpDQZLUGAqSpMZQkCQ1hoIkqVk0dAEL0bIzLh9kv+vOPGaQ/UracXimIElqDAVJUmMoSJIaQ0GS1BgKkqTGUJAkNYaCJKkxFCRJjaEgSWoG+UZzknXAI8DjwGNVtSLJPsBfA8uAdcAJVfXNIeqTpIVqyDOFn6qqg6tqRf/8DGBNVS0H1vTPJUlzaD51Hx0LrO6nVwPHDViLJC1IQ4VCAZ9NcmOSVX3b0qra0E8/CCwdpjRJWriGGiX18Kp6IMmPAFcl+fLozKqqJDXTin2IrAJ4wQteMPlKJWkBGeRMoaoe6B83ApcCrwIeSrIvQP+4cQvrnl1VK6pqxZIlS+aqZElaEOY8FJLskWSvqWngdcDtwGXAyn6xlcAn5ro2SVrohug+WgpcmmRq/39VVVckuQG4MMmbgfuAEwaoTZIWtDkPhaq6F3jZDO1fB46c63okSZvNp4+kSpIGZihIkhpDQZLUGAqSpMZQkCQ1hoIkqTEUJEmNoSBJagwFSVJjKEiSGkNBktQYCpKkxlCQJDWGgiSpGep2nBrAsjMuH2zf6848ZrB9SxqfZwqSpMZQkCQ1hoIkqTEUJEmNoSBJagwFSVJjKEiSGkNBktQYCpKkxlCQJDWGgiSpcewjzYkhx10aylDjPQ11rB3faufgmYIkqTEUJEmN3UeSZoVDs+8cDAVpQhbidRTt+Ow+kiQ1hoIkqbH7SNIOz4/hzh5DQZKepp3x4rrdR5KkZt6FQpKjknwlyT1Jzhi6HklaSOZVKCTZFfjvwOuBg4CTkxw0bFWStHDMq1AAXgXcU1X3VtX3gQuAYweuSZIWjPl2oXk/4P6R5+uBfzW6QJJVwKr+6beTfGWOapuUxcA/Dl3EPOLxeDKPx2YeixH5b8/oeLxwSzPmWyhsU1WdDZw9dB2zJcnaqloxdB3zhcfjyTwem3ksnmxSx2O+dR89ABww8nz/vk2SNAfmWyjcACxP8qIkzwJOAi4buCZJWjDmVfdRVT2W5NeBK4FdgXOq6o6By5q0naYrbJZ4PJ7M47GZx+LJJnI8UlWT2K4kaQc037qPJEkDMhQkSY2hMJAkByS5OsmdSe5I8vahaxpakl2TfCnJp4auZWhJ9k5yUZIvJ7kryaFD1zSkJO/o/05uT3J+kt2HrmkuJTknycYkt4+07ZPkqiR394/Pn419GQrDeQx4V1UdBBwCnOaQHrwduGvoIuaJPwauqKoDgZexgI9Lkv2AtwErquqldB9COWnYqubcucBR09rOANZU1XJgTf/8GTMUBlJVG6rqpn76Ebo/+v2GrWo4SfYHjgE+OnQtQ0vyPOA1wMcAqur7VfVPw1Y1uEXAc5IsAp4L/N+B65lTVXUN8I1pzccCq/vp1cBxs7EvQ2EeSLIMeDnwxWErGdQfAb8BPDF0IfPAi4BNwP/ou9M+mmSPoYsaSlU9APw+8DVgA/BwVX122KrmhaVVtaGffhBYOhsbNRQGlmRP4GLg9Kr61tD1DCHJzwEbq+rGoWuZJxYBrwDOqqqXA99hlroGdkR9X/mxdGH5z4A9kvzysFXNL9V9t2BWvl9gKAwoyW50gXBeVV0ydD0DejXwC0nW0Y2M+9NJ/nLYkga1HlhfVVNnjhfRhcRC9TPAV6tqU1X9ALgEOGzgmuaDh5LsC9A/bpyNjRoKA0kSuj7ju6rqQ0PXM6Sqek9V7V9Vy+guIH6uqhbsO8GqehC4P8mP9U1HAncOWNLQvgYckuS5/d/NkSzgC+8jLgNW9tMrgU/MxkYNheG8Gngj3bvim/ufo4cuSvPGW4HzktwKHAz87sD1DKY/Y7oIuAm4je51a0ENeZHkfOBa4MeSrE/yZuBM4LVJ7qY7mzpzVvblMBeSpCmeKUiSGkNBktQYCpKkxlCQJDWGgiSpMRREksf7j8TenuSTSfZ+Btv69jNY9239iKDnTWs/ePTjuknem+Q/PN39TEKS35yFbaxLsng26hljX8c5AKNmYigI4P9V1cH9CJTfAE4bqI63AK+tql+a1n4wMN+/wzFWKCTZddKFjOk4wFDQUxgKmu5a+tFak7wqybX9oGz/MPUN2ySnJLkkyRX9WO6/N30jSRb36x4zw7x39mcltyc5vW/7CPBi4DNJ3jGy7LOA/wKc2J/NnNjPOijJ55Pcm+RtI8v/cpLr+2X/fKYX4f4d+Qf6ZdYmeUWSK5P8nySn9sskyQf7Gm+b2m+SfZNcM3Jm9ZNJzqQbwfPm6Wc5/TrfTvIHSW4BDh2zxqcsk+TUJB8cWeaUJB/up/82yY3p7jmwatq+35/kliTXJVma5DDgF4AP9tt/ybR9n5vkI/2x+d/92FQkWZbk75Lc1P8ctpVjsmu/nanj945+2V9LckNfz8VJntu3v6Sv77Yk7xs940zy7n6dW5P85+nHSrOsqvxZ4D/At/vHXYG/AY7qn/8QsKif/hng4n76FOBe4HnA7sB9wAFT26IbrfGLdO/6p+/rJ+i+lboHsCdwB/Dyft46YPEM65wCfHjk+XuBfwCeDSwGvg7sBvw48Elgt365PwPeNMP21gH/vp/+Q+BWYC9gCfBQ3/5vgKv6Y7KUbqiFfYF3Ab81crz2Gj2GWzi+BZzQT2+xxqnff0vL9PXdM7LdzwCH99P79I/PAW4Hfnhk3z/fT/8e8Nv99LnAG7ZQ77nAFXRvGpfTjcW0O92Q1bv3yywH1vbTTzkm/b/zVSPb3Lt//OGRtvcBb+2nPwWc3E+fyub/k6+j+/Zy+no+Bbxm6L+ZnflnEVL/LpfuDOEuuhdD6F70VydZTvfistvIOmuq6mGAJHcCLwTu75dZA5xWVV+YYV+HA5dW1Xf6dS8BfhL40nbWfHlVPQo8mmQj3Qv3kXQvRjckge4FckuDhF3WP94G7FndPS0eSfJoumsqhwPnV9XjdAOPfQF4JXADcE66wQz/tqpuHqPWx+kGPmTMGmdcpqo29WdGhwB3AwcCf9+v87Ykx/fTB9C9aH8d+D7dCynAjcBrx6gX4MKqegK4O8m9/b6+Cnw4ycH97/TP+2Wfckz6dV6c5E+By4Gpoa5fmuR9wN50bwqu7NsPZfP9AP6Kbqhs6ELhdWz+/7Fn/7tdM+bvoe1kKAj6awr9qfyVdNcU/gT4r8DVVXV8uns+fH5knUdHph9n8/+lx+hefH4WmCkUZstM+w+wuqresx3rPzFtW0+wlb+LqromyWvobgh0bpIPVdXHt7Gv7/Xhwpg1bm2ZC4ATgC/ThWslOYLuTO7Qqvpuks/TvbMH+EH1b7l58r/Ttkwf/6aAdwAP0d0Jbhfge7DlY5LkZXT/D07ta/5VurOQ46rqliSnAEdso44AH6iqPx+zbj1DXlNQU1Xfpbvt4bvS3eHqecAD/exTxt0M3R//gUn+4wzz/w44Lt2Il3sAx/dtW/MIXZfEtqwB3pDkR6Ddw/aFY9Y9U50n9n3jS+juhHZ9v72Hquov6O4SNzWk9Q/6d8qzUePWlrmU7t4CJ9MFBHT/Tt/sA+FAutu7bsu2jukvJtmlv97wYuAr/X429GcQb6TrKmKmY5LuU1S7VNXFwG+z+TjtBWzoj9XoBwquo+uygyffavNK4FfT3XeEJPtNHRdNhqGgJ6mqL9H1sZ9M1wf9gSRfYjvOKvt3xSfTjQD7lmnzbqJ7t3g93XWHj/b73Jqr6S4sj15onmm/d9K9AH023eiiV9FdB3g6LqU7DrcAnwN+o7ohrY8AbumPyYl091KGrt/71pkuNG9vjVtbpqq+SdfF98Kqur5f5QpgUZK76EbKvG6M3+8C4N3pPkTwkhnmf43u3+gzwKlV9T26axsr010wP5Du5j8w8zHZD/h83y35l8DUWc9/ovt3/3u6s50ppwPv7H/fHwUe7n/fz9J1J12b5Da60VLHeYOgp8lRUiU9SZJzgU9V1UVzuM/n0nVjVpKT6C46HztX+9dmXlOQNB/8BN1F7AD/RNcFqQF4piBJarymIElqDAVJUmMoSJIaQ0GS1BgKkqTm/wP+0E+0nU8/VgAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "525 / 525 questions predicted.------------------------------------------------------------ \n",
            "\n",
            "Theme: Dog\n",
            "Total Questions: 275\n",
            "Total Paragraphs: 74\n",
            "Total queries: 275\n",
            "In top 10 results, number of queries for which -\n",
            "\tRelevant paragraph found: 268 (97 %)\n",
            "\tRelevant paragraph NOT found: 7 (3 %)\n",
            "Mean Rank for which relevant paragraph found: 1.6\n",
            "\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAZGElEQVR4nO3deZRedZ3n8feHRUFAAVPNybAYoHGhmTFoRFzg0KI2gi3g2EiOraCOkREX1FFRe0anR4+4t91OY0dggNMIIouiIMsggnbLEhbZVcAgYWJSDTaLKAp854976/JQVJJKyFO3Qr1f5zyn7vO727eeVJ7Pc3/3Pr+bqkKSJID1+i5AkjR9GAqSpI6hIEnqGAqSpI6hIEnqbNB3AU/ErFmzas6cOX2XIUnrlCuvvPLfqmpkonnrdCjMmTOHRYsW9V2GJK1Tkty+onl2H0mSOoaCJKljKEiSOoaCJKljKEiSOoaCJKljKEiSOkMLhSTbJrkoyY1JbkjyvrZ9yyQXJPlF+3OLtj1J/j7JLUmuTfKCYdUmSZrYMI8UHgI+WFU7A7sDhyfZGTgSuLCqdgIubJ8DvAbYqX0sAI4eYm2SpAkM7RvNVbUUWNpO35fkJmBrYH9gr3axE4AfAh9p20+s5q4/lybZPMnsdjtDMefIs4e16ZVafNR+vexXklZlSs4pJJkD7ApcBmw18Eb/a2Crdnpr4I6B1Za0beO3tSDJoiSLRkdHh1azJM1EQw+FJJsCpwNHVNW9g/Pao4LVuh9oVS2sqnlVNW9kZMLxnCRJa2iooZBkQ5pAOKmqzmiblyWZ3c6fDSxv2+8Eth1YfZu2TZI0RYZ59VGAY4GbqupLA7POAg5ppw8BvjPQ/pb2KqTdgXuGeT5BkvR4wxw6+2XAm4HrklzTtn0MOAo4NcnbgduBg9p55wD7ArcADwBvHWJtkqQJDPPqox8DWcHsvSdYvoDDh1WPJGnV/EazJKljKEiSOoaCJKljKEiSOoaCJKljKEiSOoaCJKljKEiSOoaCJKljKEiSOoaCJKljKEiSOoaCJKljKEiSOoaCJKljKEiSOsO8HedxSZYnuX6g7ZtJrmkfi8fuyJZkTpLfDcz72rDqkiSt2DBvx3k88FXgxLGGqnrj2HSSLwL3DCx/a1XNHWI9kqRVGObtOC9JMmeieUlCc2/mVwxr/5Kk1dfXOYU9gGVV9YuBtu2TXJ3k4iR79FSXJM1ow+w+Wpn5wMkDz5cC21XVXUleCHw7yZ9V1b3jV0yyAFgAsN12201JsZI0U0z5kUKSDYDXA98ca6uqB6vqrnb6SuBW4NkTrV9VC6tqXlXNGxkZmYqSJWnG6KP76JXAzVW1ZKwhyUiS9dvpHYCdgNt6qE2SZrRhXpJ6MvAT4DlJliR5ezvrYB7bdQSwJ3Bte4nqacBhVXX3sGqTJE1smFcfzV9B+6ETtJ0OnD6sWiRJk+M3miVJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJnWHejvO4JMuTXD/Q9skkdya5pn3sOzDvo0luSfKzJH8xrLokSSs2zCOF44F9Jmj/clXNbR/nACTZmebezX/WrvOPSdYfYm2SpAkMLRSq6hLg7kkuvj9wSlU9WFW/BG4BdhtWbZKkifVxTuHdSa5tu5e2aNu2Bu4YWGZJ2/Y4SRYkWZRk0ejo6LBrlaQZZapD4WhgR2AusBT44upuoKoWVtW8qpo3MjKytuuTpBltSkOhqpZV1cNV9QjwdR7tIroT2HZg0W3aNknSFJrSUEgye+DpgcDYlUlnAQcneWqS7YGdgMunsjZJEmwwrA0nORnYC5iVZAnwCWCvJHOBAhYD7wSoqhuSnArcCDwEHF5VDw+rNknSxIYWClU1f4LmY1ey/KeBTw+rHknSqvmNZklSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSZ2ihkOS4JMuTXD/Q9vkkNye5NsmZSTZv2+ck+V2Sa9rH14ZVlyRpxYZ5pHA8sM+4tguAXarqPwE/Bz46MO/WqprbPg4bYl2SpBUYWihU1SXA3ePazq+qh9qnlwLbDGv/kqTV1+c5hbcB3x94vn2Sq5NcnGSPFa2UZEGSRUkWjY6ODr9KSZpBegmFJB8HHgJOapuWAttV1a7AB4BvJHn6ROtW1cKqmldV80ZGRqamYEmaIaY8FJIcCrwWeFNVFUBVPVhVd7XTVwK3As+e6tokaaab0lBIsg/wYeB1VfXAQPtIkvXb6R2AnYDbprI2SRJsMKwNJzkZ2AuYlWQJ8Amaq42eClyQBODS9kqjPYG/TfJH4BHgsKq6e8INS5KGZmihUFXzJ2g+dgXLng6cPqxaJEmT4zeaJUkdQ0GS1JlUKCR52WTaJEnrtskeKfzDJNskSeuwlZ5oTvIS4KXASJIPDMx6OrD+MAuTJE29VV199BRg03a5zQba7wXeMKyiJEn9WGkoVNXFwMVJjq+q26eoJklSTyb7PYWnJlkIzBlcp6peMYyiJEn9mGwofAv4GnAM8PDwypEk9WmyofBQVR091EokSb2b7CWp303yriSzk2w59hhqZZKkKTfZI4VD2p8fGmgrYIe1W44kqU+TCoWq2n7YhUiS+jepUEjylonaq+rEtVuOJKlPk+0+etHA9EbA3sBVgKEgSU8ik+0+es/g8ySbA6cMpSJJUm/WdOjs3wKeZ5CkJ5nJnlP4Ls3VRtAMhPc84NRJrHcc8FpgeVXt0rZtCXyT5tvRi4GDquo3ae7P+RVgX+AB4NCqump1fhlJ0hMz2XMKXxiYfgi4vaqWTGK944Gv8thzD0cCF1bVUUmObJ9/BHgNsFP7eDFwdPtTkjRFJtV91A6MdzPNSKlbAH+Y5HqXAHePa94fOKGdPgE4YKD9xGpcCmyeZPZk9iNJWjsme+e1g4DLgb8CDgIuS7KmQ2dvVVVL2+lfA1u101sDdwwst6RtG1/LgiSLkiwaHR1dwxIkSROZbPfRx4EXVdVygCQjwP8FTnsiO6+qSlKrXvIx6ywEFgLMmzdvtdaVJK3cZK8+Wm8sEFp3rca64y0b6xZqf45t905g24HltmnbJElTZLJv7OcmOS/JoUkOBc4GzlnDfZ7Fo2MpHQJ8Z6D9LWnsDtwz0M0kSZoCq7pH85/SnAP4UJLXAy9vZ/0EOGlVG09yMrAXMCvJEuATwFHAqUneDtxOc44CmpDZF7iF5pLUt672byNJekJWdU7h74CPAlTVGcAZAEn+YzvvL1e2clXNX8GsvSdYtoDDV1GPJGmIVtV9tFVVXTe+sW2bM5SKJEm9WVUobL6SeRuvzUIkSf1bVSgsSvKO8Y1J/gtw5XBKkiT1ZVXnFI4AzkzyJh4NgXnAU4ADh1mYJGnqrTQUqmoZ8NIkfw7s0jafXVU/GHplkqQpN9n7KVwEXDTkWiRJPVvTbyVLkp6EDAVJUsdQkCR1DAVJUsdQkCR1DAVJUsdQkCR1DAVJUsdQkCR1DAVJUmdSw1ysTUmeA3xzoGkH4H/QDNP9DmC0bf9YVa3pLT8lSWtgykOhqn4GzAVIsj5wJ3Amze03v1xVX5jqmiRJjb67j/YGbq2q23uuQ5JE/6FwMHDywPN3J7k2yXFJtuirKEmaqXoLhSRPAV4HfKttOhrYkaZraSnwxRWstyDJoiSLRkdHJ1pEkrSG+jxSeA1wVXsjH6pqWVU9XFWPAF8HdptopapaWFXzqmreyMjIFJYrSU9+fYbCfAa6jpLMHph3IHD9lFckSTPclF99BJBkE+BVwDsHmj+XZC5QwOJx8yRJU6CXUKiq3wLPHNf25j5qkSQ9qu+rjyRJ04ihIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpE4vd14DSLIYuA94GHioquYl2RL4JjCH5pacB1XVb/qqUZJmmr6PFP68quZW1bz2+ZHAhVW1E3Bh+1ySNEX6DoXx9gdOaKdPAA7osRZJmnH6DIUCzk9yZZIFbdtWVbW0nf41sNX4lZIsSLIoyaLR0dGpqlWSZoTezikAL6+qO5P8CXBBkpsHZ1ZVJanxK1XVQmAhwLx58x43X5K05no7UqiqO9ufy4Ezgd2AZUlmA7Q/l/dVnyTNRL2EQpJNkmw2Ng28GrgeOAs4pF3sEOA7fdQnSTNVX91HWwFnJhmr4RtVdW6SK4BTk7wduB04qKf6JGlG6iUUquo24PkTtN8F7D31FUmSYPpdkipJ6pGhIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnq9HmTnRlrzpFn97LfxUft18t+Ja07PFKQJHUMBUlSx1CQJHUMBUlSZ8pDIcm2SS5KcmOSG5K8r23/ZJI7k1zTPvad6tokaabr4+qjh4APVtVVSTYDrkxyQTvvy1X1hR5qkiTRQyhU1VJgaTt9X5KbgK2nug5J0uP1ek4hyRxgV+CytundSa5NclySLVawzoIki5IsGh0dnaJKJWlm6C0UkmwKnA4cUVX3AkcDOwJzaY4kvjjRelW1sKrmVdW8kZGRKatXkmaCXkIhyYY0gXBSVZ0BUFXLqurhqnoE+DqwWx+1SdJM1sfVRwGOBW6qqi8NtM8eWOxA4Pqprk2SZro+rj56GfBm4Lok17RtHwPmJ5kLFLAYeGcPtUnSjNbH1Uc/BjLBrHOmuhZJ0mP5jWZJUsdQkCR1DAVJUsdQkCR1DAVJUsdQkCR1DAVJUsdQkCR1+vhGs3oy58ize9v34qP2623fkibPIwVJUsdQkCR1DAVJUsdzCpoSfZ3P8FyGtHo8UpAkdQwFSVLH7iNpSOwy07rIUJC01hiE675pFwpJ9gG+AqwPHFNVR/VckrRO6fNLilr3TatQSLI+8L+BVwFLgCuSnFVVN/ZbmSQ93pNxlIBpFQrAbsAtVXUbQJJTgP0BQ0HSCnl0tPZMt1DYGrhj4PkS4MWDCyRZACxon96f5GdPYH+zgH97AusPi3WtnhXWlc9OcSWPtc69Xj2zrtWQzz6hup61ohnTLRRWqaoWAgvXxraSLKqqeWtjW2uTda0e61o91rV6Zlpd0+17CncC2w4836ZtkyRNgekWClcAOyXZPslTgIOBs3quSZJmjGnVfVRVDyV5N3AezSWpx1XVDUPc5VrphhoC61o91rV6rGv1zKi6UlXD2K4kaR003bqPJEk9MhQkSZ0ZFwpJjkuyPMn1fdcyKMm2SS5KcmOSG5K8r++aAJJslOTyJD9t6/qffdc0KMn6Sa5O8r2+axmTZHGS65Jck2RR3/WMSbJ5ktOS3JzkpiQvmQY1Pad9ncYe9yY5ou+6AJK8v/2bvz7JyUk26rsmgCTva2u6YRiv1Yw7p5BkT+B+4MSq2qXvesYkmQ3MrqqrkmwGXAkc0PcQH0kCbFJV9yfZEPgx8L6qurTPusYk+QAwD3h6Vb2273qgCQVgXlVNqy88JTkB+FFVHdNe3fe0qvr3vusa0w5zcyfw4qq6vedatqb5W9+5qn6X5FTgnKo6vue6dgFOoRn94Q/AucBhVXXL2trHjDtSqKpLgLv7rmO8qlpaVVe10/cBN9F8w7tX1bi/fbph+5gWnySSbAPsBxzTdy3TXZJnAHsCxwJU1R+mUyC09gZu7TsQBmwAbJxkA+BpwP/ruR6A5wGXVdUDVfUQcDHw+rW5gxkXCuuCJHOAXYHL+q2k0XbRXAMsBy6oqmlRF/B3wIeBR/ouZJwCzk9yZTssy3SwPTAK/J+2u+2YJJv0XdQ4BwMn910EQFXdCXwB+BWwFLinqs7vtyoArgf2SPLMJE8D9uWxX/h9wgyFaSbJpsDpwBFVdW/f9QBU1cNVNZfmG+a7tYewvUryWmB5VV3Zdy0TeHlVvQB4DXB422XZtw2AFwBHV9WuwG+BI/st6VFtd9brgG/1XQtAki1oBuPcHvgPwCZJ/rrfqqCqbgI+C5xP03V0DfDw2tyHoTCNtH32pwMnVdUZfdczXtvdcBGwT9+1AC8DXtf2358CvCLJP/dbUqP9lElVLQfOpOn/7dsSYMnAUd5pNCExXbwGuKqqlvVdSOuVwC+rarSq/gicAby055oAqKpjq+qFVbUn8Bvg52tz+4bCNNGe0D0WuKmqvtR3PWOSjCTZvJ3emOZeFzf3WxVU1UerapuqmkPT7fCDqur9k1ySTdoLBWi7Z15Nc8jfq6r6NXBHkue0TXszvYakn8806Tpq/QrYPcnT2v+be9Oc5+tdkj9pf25Hcz7hG2tz+9NqmIupkORkYC9gVpIlwCeq6th+qwKaT75vBq5r++8BPlZV5/RYE8Bs4IT2ypD1gFOratpc/jkNbQWc2byPsAHwjao6t9+SOu8BTmq7am4D3tpzPUAXnq8C3tl3LWOq6rIkpwFXAQ8BVzN9hrs4PckzgT8Ch6/tCwZm3CWpkqQVs/tIktQxFCRJHUNBktQxFCRJHUNBktQxFESSh9sRKq9P8t2x7yWs4bbuX/VSK1z3ve3onSeNa5+bZN+B559M8t/WdD/DkORja2Ebi5PMWhv1TGJfByTZeSr2pXWLoSCA31XV3HbU2LuBw3uq413Aq6rqTePa59KM8TKdTSoU2u97TAcHAIaCHsdQ0Hg/oR2dNcluSX7SDqD2r2Pfhk1yaJIzkpyb5BdJPjd+I0lmtevuN8G8D7RHJdePjQef5GvADsD3k7x/YNmnAH8LvLE9mnljO2vnJD9McluS9w4s/9dp7v9wTZJ/muhNuP1E/pl2mUVJXpDkvCS3JjmsXSZJPt/WeN3YfpPMTnLJwJHVHkmOohlN85rxRzntOvcn+WKSnwIvmWSNj1smyWFJPj+wzKFJvtpOfzvNAHw3ZGAQvnbfn05zP4xLk2yV5KU04wx9vt3+juP2fXySr7Wvzc/TjDNFkjlJfpTkqvbx0pW8Juu32xl7/d7fLvuOJFe09ZyeZlA3kuzY1nddkk8NHnEm+VC7zrWZZvfzeFKqKh8z/AHc3/5cn2ZAsn3a508HNminXwmc3k4fSvON2GcAGwG3A9uObYvmW72X0XzqH7+vFwLXAZsAmwI3ALu28xYDsyZY51DgqwPPPwn8K/BUYBZwF82Q3s8Dvgts2C73j8BbJtjeYuC/ttNfBq4FNgNGgGVt+38GLmhfk61ohj2YDXwQ+PjA67XZ4Gu4gte3gIPa6RXWOPb7r2iZtr5bBrb7fZrB9wC2bH9uTDOsxjMH9v2X7fTngL9pp48H3rCCeo+nGWxtPWAnmnGTNqIZPnqjdpmdgEXt9ONek/bf+YKBbW7e/nzmQNungPe0098D5rfTh/Ho3+Srab5JnLae7wF79v1/5sn8mHHDXGhCG6cZWmNrmvFdLmjbn0EzxMVONG8uGw6sc2FV3QOQ5EbgWcAd7TIX0nz9/uIJ9vVy4Myq+m277hnAHjTDCKyOs6vqQeDBJMtp3rj3pnkzuiLNMBMb0wz3PZGz2p/XAZtWcw+L+5I8mOacysuBk6vqYWBZkouBFwFXAMelGbzw21V1zUQbH+dhmoEOmWSNEy5TVaPtkdHuwC+A5wL/0q7z3iQHttPb0rxp30VzI5axYUmupBlOYjJOrapHgF8kua3d1y+BryaZ2/5Oz26Xfdxr0q6zQ5J/AM6mGdUTYJcknwI2p/lQcF7b/hKaLi1oxvL5Qjv96vYx9vexafu7XTLJ30OryVAQtOcU2kP582jOKfw98L+Ai6rqwDT3ePjhwDoPDkw/zKN/Sw/RvPn8Bc0NQIZlov0HOKGqProa6z8ybluPsJL/F1V1SZqhsPcDjk/ypao6cRX7+n0bLkyyxpUtcwpwEM2ghGdWVSXZi+ZI7iVV9UCSH9J8sgf4Y7UfuXnsv9OqjB//poD3A8uA59N8av89rPg1SfJ8mr+Dw9qa30ZzFHJAVf00yaE045CtTIDPVNU/TbJuPUGeU1Cnqh4A3gt8MM3dpp5Bc3tEaLpwJrUZmv/8z03ykQnm/wg4IM3ok5sAB7ZtK3MfTZfEqlwIvCGPjiK5ZZJnTbLuiep8Y9s3PkJz17LL2+0tq6qv09zxbWz46T+2n5TXRo0rW+ZMmnH+59MEBDT/Tr9pA+G5wO6TqGNVr+lfJVmvPd+wA/Czdj9L2yOIN9N0FTHRa5LmKqr1qup04G949HXaDFjavlaDFxRcStNlB82ot2POA96W5j4jJNl67HXRcBgKeoyqupqmj30+TR/0Z5JczWocVbafiufT3OPgXePmXUXzafFymvMOx7T7XJmLaE4sD55onmi/N9K8AZ2f5FqabrDZk617nDNpXoefAj8APlzN8NN7AT9tX5M3Al9pl18IXDvRiebVrXFly1TVb2i6+J5VVZe3q5wLbJDkJuAomjfYVTkF+FCaiwh2nGD+r2j+jb5Pcw/g39Oc2zgkzQnz59LcqAcmfk22Bn7Ydkv+MzB21PPfaf7d/4XHDsF+BPCB9vf9U+Ce9vc9n6Y76SdJrqO5D8RkPiBoDTlKqqTHSHI88L2qOm0K9/k0mm7MSnIwzUnn/adq/3qU5xQkTQcvpDmJHeDfabog1QOPFCRJHc8pSJI6hoIkqWMoSJI6hoIkqWMoSJI6/x8B7m1Ob4MpfgAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "275 / 275 questions predicted.------------------------------------------------------------ \n",
            "\n",
            "Theme: 2008_Summer_Olympics_torch_relay\n",
            "Total Questions: 356\n",
            "Total Paragraphs: 73\n",
            "Total queries: 356\n",
            "In top 10 results, number of queries for which -\n",
            "\tRelevant paragraph found: 311 (87 %)\n",
            "\tRelevant paragraph NOT found: 45 (13 %)\n",
            "Mean Rank for which relevant paragraph found: 1.99\n",
            "\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAXtUlEQVR4nO3deZhldX3n8fdHwA1QQCr9MCw2kFaHOGOjLYPrQ0QNaiKQGJbHKETHlhEX1NGgZkYn0ZFxzRgnmFYJ+ARBZVEUBZkWIQuIzSI0W1hspHva7g4YxA0FvvPHOXX6dlHdVd1d956Cer+e5z733N/ZvnWq6n7uWe7vpKqQJAngUX0XIEmaPQwFSVLHUJAkdQwFSVLHUJAkdbbtu4Ctseuuu9b8+fP7LkOSHlauvPLKf62qscnGPaxDYf78+SxbtqzvMiTpYSXJHRsb5+EjSVLHUJAkdQwFSVLHUJAkdQwFSVLHUJAkdYYWCkn2THJxkhuSXJ/kbW37LkkuSnJL+7xz254kn0pya5JrkzxzWLVJkiY3zD2F+4F3VtV+wIHA8Un2A04EllbVAmBp+xrgZcCC9rEYOHmItUmSJjG0UKiq1VV1VTt8L3AjsDtwKHBaO9lpwGHt8KHAF6pxObBTkt2GVZ8k6aFG8o3mJPOB/YHvAfOqanU76sfAvHZ4d+DOgdlWtm2rB9pIsphmT4K99tprq+qaf+L5WzX/llpx0it6Wa8kTWXoJ5qT7ACcDZxQVT8dHFfNbd8269ZvVbWkqhZV1aKxsUm77pAkbaGhhkKS7WgC4fSqOqdtXjN+WKh9Xtu2rwL2HJh9j7ZNkjQiw7z6KMDngRur6hMDo84DjmmHjwG+NtD+2vYqpAOBewYOM0mSRmCY5xSeB7wGuC7JNW3be4GTgC8neT1wB3BEO+6bwMuBW4FfAH86xNokSZMYWihU1T8C2cjogyeZvoDjh1WPJGlqfqNZktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJnWHejvOUJGuTLB9o+1KSa9rHivE7siWZn+SXA+M+M6y6JEkbN8zbcZ4KfBr4wnhDVR05Ppzk48A9A9PfVlULh1iPJGkKw7wd56VJ5k82Lklo7s38omGtX5K0+fo6p/ACYE1V3TLQtneSq5NckuQFPdUlSXPaMA8fbcrRwBkDr1cDe1XVXUmeBXw1ye9U1U8nzphkMbAYYK+99hpJsZI0V4x8TyHJtsAfAl8ab6uq+6rqrnb4SuA24CmTzV9VS6pqUVUtGhsbG0XJkjRn9HH46MXATVW1crwhyViSbdrhfYAFwO091CZJc9owL0k9A7gMeGqSlUle3446ig0PHQG8ELi2vUT1LOC4qrp7WLVJkiY3zKuPjt5I+7GTtJ0NnD2sWiRJ0+M3miVJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJnWHejvOUJGuTLB9o+0CSVUmuaR8vHxj3niS3Jrk5ye8Nqy5J0sYNc0/hVOCQSdo/WVUL28c3AZLsR3Pv5t9p5/mbJNsMsTZJ0iSGFgpVdSlw9zQnPxQ4s6ruq6ofArcCBwyrNknS5Po4p/DmJNe2h5d2btt2B+4cmGZl2/YQSRYnWZZk2bp164ZdqyTNKaMOhZOBfYGFwGrg45u7gKpaUlWLqmrR2NjYTNcnSXPaSEOhqtZU1QNV9SDwWdYfIloF7Dkw6R5tmyRphEYaCkl2G3h5ODB+ZdJ5wFFJHpNkb2ABcMUoa5MkwbbDWnCSM4CDgF2TrATeDxyUZCFQwArgjQBVdX2SLwM3APcDx1fVA8OqTZI0uaGFQlUdPUnz5zcx/YeADw2rHknS1PxGsySpYyhIkjqGgiSpYyhIkjqGgiSpYyhIkjqGgiSpYyhIkjqGgiSpYyhIkjqGgiSpYyhIkjqGgiSpYyhIkjqGgiSpYyhIkjqGgiSpM7RQSHJKkrVJlg+0fTTJTUmuTXJukp3a9vlJfpnkmvbxmWHVJUnauGHuKZwKHDKh7SLg6VX1H4F/Ad4zMO62qlrYPo4bYl2SpI0YWihU1aXA3RPavl1V97cvLwf2GNb6JUmbr89zCq8DvjXweu8kVye5JMkLNjZTksVJliVZtm7duuFXKUlzSC+hkOR9wP3A6W3TamCvqtofeAfwxSRPmGzeqlpSVYuqatHY2NhoCpakOWLkoZDkWOD3gVdXVQFU1X1VdVc7fCVwG/CUUdcmSXPdSEMhySHAu4FXVtUvBtrHkmzTDu8DLABuH2VtkiTYdlgLTnIGcBCwa5KVwPtprjZ6DHBREoDL2yuNXgj8RZLfAA8Cx1XV3ZMuWJI0NEMLhao6epLmz29k2rOBs4dViyRpevxGsySpYyhIkjqGgiSpM61QSPK86bRJkh7eprun8NfTbJMkPYxt8uqjJM8BnguMJXnHwKgnANsMszBJ0uhNdUnqo4Ed2ul2HGj/KfCqYRUlSerHJkOhqi4BLklyalXdMaKaJEk9me6X1x6TZAkwf3CeqnrRMIqSJPVjuqHwFeAzwOeAB4ZXjiSpT9MNhfur6uShViJJ6t10L0n9epI3JdktyS7jj6FWJkkauenuKRzTPr9roK2AfWa2HElSn6YVClW197ALkST1b1qhkOS1k7VX1RdmthxJUp+me/jo2QPDjwUOBq4CDAVJegSZ7uGjtwy+TrITcOZQKpIk9WZLu87+OTDleYYkpyRZm2T5QNsuSS5Kckv7vHPbniSfSnJrkmuTPHMLa5MkbaHpdp399STntY/zgZuBc6cx66nAIRPaTgSWVtUCYGn7GuBlwIL2sRjwexGSNGLTPafwsYHh+4E7qmrlVDNV1aVJ5k9oPhQ4qB0+Dfgu8Gdt+xeqqoDLk+yUZLeqWj3NGiVJW2laewptx3g30fSUujPw661Y57yBN/ofA/Pa4d2BOwemW9m2bSDJ4iTLkixbt27dVpQhSZpouoePjgCuAP4YOAL4XpKt7jq73SuozZxnSVUtqqpFY2NjW1uCJGnAdA8fvQ94dlWtBUgyBvxf4KwtWOea8cNCSXYD1rbtq4A9B6bbo22TJI3IdK8+etR4ILTu2ox5JzqP9d1mHAN8baD9te1VSAcC93g+QZJGa7p7ChckuRA4o319JPDNqWZKcgbNSeVdk6wE3g+cBHw5yeuBO2gOR9Eu7+XArcAvgD+dZm2SpBky1T2af5vmxPC7kvwh8Px21GXA6VMtvKqO3siogyeZtoDjp1qmJGl4ptpT+CvgPQBVdQ5wDkCS/9CO+4OhVidJGqmpzgvMq6rrJja2bfOHUpEkqTdThcJOmxj3uJksRJLUv6lCYVmSN0xsTPKfgSuHU5IkqS9TnVM4ATg3yatZHwKLgEcDhw+zMEnS6G0yFKpqDfDcJL8LPL1tPr+qvjP0yiRJIzfd+ylcDFw85FokST3b0m8lS5IegQwFSVLHUJAkdQwFSVLHUJAkdQwFSVLHUJAkdQwFSVLHUJAkdQwFSVJnurfjnDFJngp8aaBpH+C/03TT/QZgXdv+3qqa8pafkqSZM/JQqKqbgYUASbYBVgHn0tyT+ZNV9bFR1yRJavR9+Ohg4LaquqPnOiRJ9B8KRwFnDLx+c5Jrk5ySZOfJZkiyOMmyJMvWrVs32SSSpC3UWygkeTTwSuArbdPJwL40h5ZWAx+fbL6qWlJVi6pq0djY2EhqlaS5os89hZcBV7U38qGq1lTVA1X1IPBZ4IAea5OkOanPUDiagUNHSXYbGHc4sHzkFUnSHDfyq48AkmwPvAR440DzR5IsBApYMWGcJGkEegmFqvo58KQJba/poxZJ0np9X30kSZpFDAVJUsdQkCR1DAVJUsdQkCR1DAVJUsdQkCR1DAVJUsdQkCR1DAVJUsdQkCR1DAVJUsdQkCR1DAVJUsdQkCR1DAVJUsdQkCR1ernzGkCSFcC9wAPA/VW1KMkuwJeA+TS35Dyiqn7SV42SNNf0vafwu1W1sKoWta9PBJZW1QJgaftakjQifYfCRIcCp7XDpwGH9ViLJM05fYZCAd9OcmWSxW3bvKpa3Q7/GJg3caYki5MsS7Js3bp1o6pVkuaE3s4pAM+vqlVJfgu4KMlNgyOrqpLUxJmqagmwBGDRokUPGS9J2nK97SlU1ar2eS1wLnAAsCbJbgDt89q+6pOkuaiXUEiyfZIdx4eBlwLLgfOAY9rJjgG+1kd9kjRX9XX4aB5wbpLxGr5YVRck+T7w5SSvB+4AjuipvqGaf+L5vax3xUmv6GW9kh4+egmFqrodeMYk7XcBB4++IkkSzL5LUiVJPTIUJEkdQ0GS1DEUJEkdQ0GS1DEUJEkdQ0GS1DEUJEkdQ0GS1DEUJEkdQ0GS1DEUJEkdQ0GS1DEUJEkdQ0GS1DEUJEkdQ0GS1Bl5KCTZM8nFSW5Icn2St7XtH0iyKsk17ePlo65Nkua6Pm7HeT/wzqq6KsmOwJVJLmrHfbKqPtZDTZIkegiFqloNrG6H701yI7D7qOuQJD1Ur+cUkswH9ge+1za9Ocm1SU5JsvNG5lmcZFmSZevWrRtRpZI0N/QWCkl2AM4GTqiqnwInA/sCC2n2JD4+2XxVtaSqFlXVorGxsZHVK0lzQS+hkGQ7mkA4varOAaiqNVX1QFU9CHwWOKCP2iRpLuvj6qMAnwdurKpPDLTvNjDZ4cDyUdcmSXNdH1cfPQ94DXBdkmvatvcCRydZCBSwAnhjD7VJ0pzWx9VH/whkklHfHHUtkqQN9bGnoJ7MP/H83ta94qRX9LZuSdNnNxeSpI57ChqJvvZS3EORNo97CpKkjqEgSeoYCpKkjqEgSeoYCpKkjqEgSeoYCpKkjqEgSeoYCpKkjqEgSerYzYU0JHbtoYcjQ0GPaH32DNsXe8PV1jAUJD3suVc2cwwFSTNmLu6ZPdLMuhPNSQ5JcnOSW5Oc2Hc9kjSXzKpQSLIN8H+AlwH70dy3eb9+q5KkuWO2HT46ALi1qm4HSHImcChwQ69VSdIkHokn9WdbKOwO3DnweiXwnwYnSLIYWNy+/FmSm0dU27DsCvxr30XMIm6PDbk91nNbDMj/2qrt8eSNjZhtoTClqloCLOm7jpmSZFlVLeq7jtnC7bEht8d6bosNDWt7zKpzCsAqYM+B13u0bZKkEZhtofB9YEGSvZM8GjgKOK/nmiRpzphVh4+q6v4kbwYuBLYBTqmq63sua9geMYfCZojbY0Nuj/XcFhsayvZIVQ1juZKkh6HZdvhIktQjQ0GS1DEUepJkzyQXJ7khyfVJ3tZ3TX1Lsk2Sq5N8o+9a+pZkpyRnJbkpyY1JntN3TX1K8vb2/2R5kjOSPLbvmkYpySlJ1iZZPtC2S5KLktzSPu88E+syFPpzP/DOqtoPOBA43i49eBtwY99FzBL/G7igqp4GPIM5vF2S7A68FVhUVU+nuQjlqH6rGrlTgUMmtJ0ILK2qBcDS9vVWMxR6UlWrq+qqdvhemn/63futqj9J9gBeAXyu71r6luSJwAuBzwNU1a+r6t/6rap32wKPS7It8Hjg//Vcz0hV1aXA3ROaDwVOa4dPAw6biXUZCrNAkvnA/sD3+q2kV38FvBt4sO9CZoG9gXXA37WH0z6XZPu+i+pLVa0CPgb8CFgN3FNV3+63qllhXlWtbod/DMybiYUaCj1LsgNwNnBCVf2073r6kOT3gbVVdWXftcwS2wLPBE6uqv2BnzNDhwYejtpj5YfShOW/A7ZP8if9VjW7VPPdghn5foGh0KMk29EEwulVdU7f9fToecArk6wAzgRelOTv+y2pVyuBlVU1vud4Fk1IzFUvBn5YVeuq6jfAOcBze65pNliTZDeA9nntTCzUUOhJktAcM76xqj7Rdz19qqr3VNUeVTWf5gTid6pqzn4SrKofA3cmeWrbdDBzu/v4HwEHJnl8+39zMHP4xPuA84Bj2uFjgK/NxEINhf48D3gNzafia9rHy/suSrPGW4DTk1wLLAT+Z8/19KbdYzoLuAq4juZ9a051eZHkDOAy4KlJViZ5PXAS8JIkt9DsTZ00I+uymwtJ0jj3FCRJHUNBktQxFCRJHUNBktQxFCRJHUNBJHmgvSR2eZKvJ9lpK5b1s62Y961tj6CnT2hfOHi5bpIPJPmvW7qeYUjy3hlYxooku85EPdNY12F2wKjJGAoC+GVVLWx7oLwbOL6nOt4EvKSqXj2hfSEw27/DMa1QSLLNsAuZpsMAQ0EPYShoostoe2tNckCSy9pO2f55/Bu2SY5Nck6SC9q+3D8ycSFJdm3nfcUk497R7pUsT3JC2/YZYB/gW0nePjDto4G/AI5s92aObEftl+S7SW5P8taB6f8kyRXttH872Ztw+4n8w+00y5I8M8mFSW5Lclw7TZJ8tK3xuvH1JtktyaUDe1YvSHISTQ+e10zcy2nn+VmSjyf5AfCcadb4kGmSHJfkowPTHJvk0+3wV5NcmeaeA4snrPtDSX6Q5PIk85I8F3gl8NF2+ftOWPepST7Tbpt/afumIsn8JP+Q5Kr28dxNbJNt2uWMb7+3t9O+Icn323rOTvL4tn3ftr7rknxwcI8zybvaea5N8j8mbivNsKryMccfwM/a522ArwCHtK+fAGzbDr8YOLsdPha4HXgi8FjgDmDP8WXR9Nb4PZpP/RPX9Syab6VuD+wAXA/s345bAew6yTzHAp8eeP0B4J+BxwC7AncB2wH/Hvg6sF073d8Ar51keSuA/9IOfxK4FtgRGAPWtO1/BFzUbpN5NF0t7Aa8E3jfwPbacXAbbmT7FnBEO7zRGsd//o1N09Z368ByvwU8vx3epX1+HLAceNLAuv+gHf4I8Oft8KnAqzZS76nABTQfGhfQ9MX0WJouqx/bTrMAWNYOP2SbtL/niwaWuVP7/KSBtg8Cb2mHvwEc3Q4fx/q/yZfSfHs5bT3fAF7Y9//MI/mxLVL7KZdmD+FGmjdDaN70T0uygObNZbuBeZZW1T0ASW4Angzc2U6zFDi+qi6ZZF3PB86tqp+3854DvAC4ejNrPr+q7gPuS7KW5o37YJo3o+8ngeYNcmOdhJ3XPl8H7FDNPS3uTXJfmnMqzwfOqKoHaDoeuwR4NvB94JQ0nRl+taqumUatD9B0fMg0a5x0mqpa1+4ZHQjcAjwN+Kd2nrcmObwd3pPmTfsu4Nc0b6QAVwIvmUa9AF+uqgeBW5Lc3q7rh8Cnkyxsf6antNM+ZJu08+yT5K+B84Hxrq6fnuSDwE40HwoubNufw/r7AXyRpqtsaELhpaz/+9ih/dkunebPoc1kKAjacwrtrvyFNOcUPgX8JXBxVR2e5p4P3x2Y576B4QdY/7d0P82bz+8Bk4XCTJls/QFOq6r3bMb8D05Y1oNs4v+iqi5N8kKaGwKdmuQTVfWFKdb1qzZcmGaNm5rmTOAI4CaacK0kB9HsyT2nqn6R5Ls0n+wBflPtR242/D1NZWL/NwW8HVhDcye4RwG/go1vkyTPoPk7OK6t+XU0eyGHVdUPkhwLHDRFHQE+XFV/O826tZU8p6BOVf2C5raH70xzh6snAqva0cdOdzE0//xPS/Jnk4z/B+CwND1ebg8c3rZtyr00hySmshR4VZLfgu4etk+eZt2T1Xlke2x8jOZOaFe0y1tTVZ+luUvceJfWv2k/Kc9EjZua5lyaewscTRMQ0PyeftIGwtNobu86lam26R8neVR7vmEf4OZ2PavbPYjX0BwqYrJtkuYqqkdV1dnAn7N+O+0IrG631eAFBZfTHLKDDW+1eSHwujT3HSHJ7uPbRcNhKGgDVXU1zTH2o2mOQX84ydVsxl5l+6n4aJoeYN80YdxVNJ8Wr6A57/C5dp2bcjHNieXBE82TrfcGmjegb6fpXfQimvMAW+Jcmu3wA+A7wLur6dL6IOAH7TY5kuZeytAc9752shPNm1vjpqapqp/QHOJ7clVd0c5yAbBtkhtpesq8fBo/35nAu9JcRLDvJON/RPM7+hZwXFX9iubcxjFpTpg/jebmPzD5Ntkd+G57WPLvgfG9nv9G83v/J5q9nXEnAO9of97fBu5pf95v0xxOuizJdTS9pU7nA4K2kL2kStpAklOBb1TVWSNc5+NpDmNWkqNoTjofOqr1az3PKUiaDZ5FcxI7wL/RHIJUD9xTkCR1PKcgSeoYCpKkjqEgSeoYCpKkjqEgSer8fxxat+Z9vfuNAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "356 / 356 questions predicted.------------------------------------------------------------ \n",
            "\n",
            "Theme: Genome\n",
            "Total Questions: 145\n",
            "Total Paragraphs: 25\n",
            "Total queries: 145\n",
            "In top 10 results, number of queries for which -\n",
            "\tRelevant paragraph found: 139 (96 %)\n",
            "\tRelevant paragraph NOT found: 6 (4 %)\n",
            "Mean Rank for which relevant paragraph found: 1.89\n",
            "\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAATtUlEQVR4nO3de7BlZX3m8e9DN4SbsUHOdJGGslGJhDIlmNbhYixHNGOFRNrEAJYxzYQJxeiIgKNiLpXLmIiX8ZKYaAgYOhUCMYABIYKkBc0kCDQ3ubQODAI2aeDEiLdEEfjNH2v19O7Tp7s3cNbZp32/n6pTe+11/e11znn2Wu/a+12pKiRJ7dhp0gVIkuaXwS9JjTH4JakxBr8kNcbgl6TGLJ50AePYZ599avny5ZMuQ5J2KDfeeOO/VNXUzPE7RPAvX76ctWvXTroMSdqhJLlvtvE29UhSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmN2iG/uPh3Lz7h8Itu998yjJ7JdSdoej/glqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUmEGDP8lpSe5IcnuS85PsmuSAJNcluTvJXyfZZcgaJEmbGyz4kywDTgFWVNULgEXA8cB7gQ9V1fOAbwAnDlWDJGlLQzf1LAZ2S7IY2B3YALwCuLCfvhpYOXANkqQRgwV/VT0AfAC4ny7wvwncCDxSVY/1s60Hls22fJKTkqxNsnZ6enqoMiWpOUM29ewFHAMcAPwYsAfw6nGXr6qzqmpFVa2YmpoaqEpJas+QTT2vBL5aVdNV9QPgYuBIYEnf9AOwH/DAgDVIkmYYMvjvBw5LsnuSAEcBdwJXA6/r51kFXDJgDZKkGYZs47+O7iLuTcBt/bbOAt4JnJ7kbuBZwDlD1SBJ2tLi7c/y1FXVbwO/PWP0PcBLhtyuJGnr/OauJDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUmEGDP8mSJBcm+XKSdUkOT7J3kquS3NU/7jVkDZKkzQ19xP8R4IqqOgh4IbAOOANYU1UHAmv655KkeTJY8Cd5JvAy4ByAqnq0qh4BjgFW97OtBlYOVYMkaUtDHvEfAEwDf57k5iRnJ9kDWFpVG/p5HgSWzrZwkpOSrE2ydnp6esAyJaktQwb/YuBFwMeq6lDgu8xo1qmqAmq2havqrKpaUVUrpqamBixTktoyZPCvB9ZX1XX98wvp3ggeSrIvQP/48IA1SJJmGCz4q+pB4GtJnt+POgq4E7gUWNWPWwVcMlQNkqQtLR54/W8BzkuyC3AP8F/o3mw+meRE4D7g2IFrkCSNGDT4q+oWYMUsk44acruSpK3zm7uS1BiDX5IaY/BLUmMMfklqzFjBn+TIccZJkha+cY/4/2jMcZKkBW6bH+dMcjhwBDCV5PSRST8KLBqyMEnSMLb3Of5dgD37+Z4xMv5bwOuGKkqSNJxtBn9VfR74fJJzq+q+eapJkjSgcb+5+yNJzgKWjy5TVa8YoihJ0nDGDf6/AT4OnA08Plw5kqShjRv8j1XVxwatRJI0L8b9OOenk7wpyb79zdL3TrL3oJVJkgYx7hH/xv7z3z4yroDnzG05kqShjRX8VXXA0IVIkubHWMGf5FdmG19VfzG35UiShjZuU8+LR4Z3pbuRyk2AwS9JO5hxm3reMvo8yRLggkEqkiQN6ql2y/xdwHZ/SdoBjdvG/2m6T/FA1znbTwCfHKooSdJwxm3j/8DI8GPAfVW1foB6JEkDG6upp++s7ct0PXTuBTw6ZFGSpOGMeweuY4HrgV8CjgWuS2K3zJK0Axq3qec3gBdX1cMASaaAvwcuHKowSdIwxv1Uz04bQ7/39SexrCRpARn3iP+KJFcC5/fPjwP+bpiSJElD2t49d58HLK2qtyf5BeCl/aRrgfOGLk6SNPe2d8T/YeBdAFV1MXAxQJKf7Kf9/KDVSZLm3Pba6ZdW1W0zR/bjlg9SkSRpUNsL/iXbmLbbXBYiSZof2wv+tUl+bebIJP8VuHGYkiRJQ9peG/+pwKeSvIFNQb8C2AV47ZCFSZKGsc3gr6qHgCOS/CfgBf3oy6vqc4NXJkkaxLj98V8NXD1wLZKkeTD4t2+TLEpyc5LL+ucHJLkuyd1J/jrJLkPXIEnaZD66XXgrsG7k+XuBD1XV84BvACfOQw2SpN6gwZ9kP+Bo4Oz+eYBXsKlzt9XAyiFrkCRtbugj/g8D7wCe6J8/C3ikqh7rn68Hlg1cgyRpxGDBn+TngIer6il93j/JSUnWJlk7PT09x9VJUruGPOI/EnhNknuBC+iaeD4CLEmy8dNE+wEPzLZwVZ1VVSuqasXU1NSAZUpSWwYL/qp6V1XtV1XLgeOBz1XVG+g+Frrx7l2rgEuGqkGStKVJ3EzlncDpSe6ma/M/ZwI1SFKzxr0Ry9NSVdcA1/TD9wAvmY/tSpK25O0TJakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4JekxgwW/En2T3J1kjuT3JHkrf34vZNcleSu/nGvoWqQJG1pyCP+x4C3VdXBwGHAm5McDJwBrKmqA4E1/XNJ0jwZLPirakNV3dQPfxtYBywDjgFW97OtBlYOVYMkaUvz0safZDlwKHAdsLSqNvSTHgSWbmWZk5KsTbJ2enp6PsqUpCYMHvxJ9gQuAk6tqm+NTquqAmq25arqrKpaUVUrpqamhi5TkpoxaPAn2Zku9M+rqov70Q8l2befvi/w8JA1SJI2N+SnegKcA6yrqg+OTLoUWNUPrwIuGaoGSdKWFg+47iOBNwK3JbmlH/frwJnAJ5OcCNwHHDtgDZKkGQYL/qr630C2MvmoobYrSdo2v7krSY0x+CWpMQa/JDVmyIu7TVt+xuUT2/a9Zx49sW1LWvg84pekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xi9w/RCa1JfH/OKYtGPwiF+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BhvxKIfCt58RhqfR/yS1BiP+DVnJnXUPUmTfM2ebeip8ohfkhrjEb+0g2rxukaLr3kIHvFLUmMmEvxJXp3kK0nuTnLGJGqQpFbNe1NPkkXAHwOvAtYDNyS5tKrunO9aJD15XsSfP0M1MU3iiP8lwN1VdU9VPQpcABwzgTokqUmTuLi7DPjayPP1wH+cOVOSk4CT+qffSfKVeahtSPsA/zLpIhYI98Xm3B+bc3/08t6nvS+ePdvIBfupnqo6Czhr0nXMlSRrq2rFpOtYCNwXm3N/bM79sclQ+2ISTT0PAPuPPN+vHydJmgeTCP4bgAOTHJBkF+B44NIJ1CFJTZr3pp6qeizJfweuBBYBn6iqO+a7jgn4oWm2mgPui825Pzbn/thkkH2RqhpivZKkBcpv7kpSYwx+SWqMwT+gJPsnuTrJnUnuSPLWSde0ECRZlOTmJJdNupZJS7IkyYVJvpxkXZLDJ13TpCQ5rf8/uT3J+Ul2nXRN8ynJJ5I8nOT2kXF7J7kqyV39415zsS2Df1iPAW+rqoOBw4A3Jzl4wjUtBG8F1k26iAXiI8AVVXUQ8EIa3S9JlgGnACuq6gV0H/w4frJVzbtzgVfPGHcGsKaqDgTW9M+fNoN/QFW1oapu6oe/TfdPvWyyVU1Wkv2Ao4GzJ13LpCV5JvAy4ByAqnq0qh6ZbFUTtRjYLcliYHfgnydcz7yqqi8A/zpj9DHA6n54NbByLrZl8M+TJMuBQ4HrJlvJxH0YeAfwxKQLWQAOAKaBP++bvs5Osseki5qEqnoA+ABwP7AB+GZVfXayVS0IS6tqQz/8ILB0LlZq8M+DJHsCFwGnVtW3Jl3PpCT5OeDhqrpx0rUsEIuBFwEfq6pDge8yR6fyO5q+7foYujfDHwP2SPLLk61qYanus/dz8vl7g39gSXamC/3zquriSdczYUcCr0lyL12vrK9I8peTLWmi1gPrq2rjWeCFdG8ELXol8NWqmq6qHwAXA0dMuKaF4KEk+wL0jw/PxUoN/gElCV377bqq+uCk65m0qnpXVe1XVcvpLtx9rqqaPaqrqgeBryV5fj/qKKDV+1LcDxyWZPf+/+YoGr3QPcOlwKp+eBVwyVys1OAf1pHAG+mObG/pf3520kVpQXkLcF6SLwGHAH8w4Xomoj/ruRC4CbiNLpua6rohyfnAtcDzk6xPciJwJvCqJHfRnRWdOSfbsssGSWqLR/yS1BiDX5IaY/BLUmMMfklqjMEvSY0x+BuR5PH+46S3J/l0kiVPY13feRrLntL3QnnejPGHjH7UNcnvJPkfT3U7Q0jy63OwjnuT7DMX9YyxrZV2CqjZGPzt+PeqOqTv+fBfgTdPqI43Aa+qqjfMGH8IsNC/4zBW8CdZNHQhY1oJGPzagsHfpmvpewlN8pIk1/adhP3Txm+RJjkhycVJruj7An/fzJUk2adf9uhZpp3en13cnuTUftzHgecAn0ly2si8uwC/BxzXn5Uc1086OMk1Se5JcsrI/L+c5Pp+3j+dLWj7I+v39POsTfKiJFcm+b9JTu7nSZL39zXetnG7SfZN8oWRM6SfTnImXc+Rt8w8W+mX+U6S/5XkVuDwMWvcYp4kJyd5/8g8JyT5aD/8t0luTNdn/Ukztv37SW5N8sUkS5McAbwGeH+//ufO2Pa5ST7e75v/0/ejRJLlSf4hyU39zxHb2CeL+vVs3H+n9fP+WpIb+nouSrJ7P/65fX23JXn36Jljkrf3y3wpye/O3FeaY1XlTwM/wHf6x0XA3wCv7p//KLC4H34lcFE/fAJwD/BMYFfgPmD/jeui6yXwOrqj95nb+im6b1/uAewJ3AEc2k+7F9hnlmVOAD468vx3gH8CfgTYB/g6sDPwE8CngZ37+f4E+JVZ1ncv8N/64Q8BXwKeAUwBD/XjfxG4qt8nS+m6DdgXeBvwGyP76xmj+3Ar+7eAY/vhrda48fVvbZ6+vrtH1vsZ4KX98N79427A7cCzRrb98/3w+4Df7IfPBV63lXrPBa6gO/g7kK7foF3pukPetZ/nQGBtP7zFPul/z1eNrHNJ//iskXHvBt7SD18GvL4fPplNf5M/Q/ct3fT1XAa8bNL/Mz/MP4tRK3ZLcgvdkf46usCDLthXJzmQLkB2HllmTVV9EyDJncCzga/186wB3lxVn59lWy8FPlVV3+2XvRj4aeDmJ1nz5VX1feD7SR6mC+ej6ALnhiTQheDWOq66tH+8DdizunsifDvJ99Nd43gpcH5VPU7XGdbngRcDNwCfSNfB3t9W1S1j1Po4XWd8jFnjrPNU1XR/hnMYcBdwEPCP/TKnJHltP7w/XTB/HXiULiwBbgReNUa9AJ+sqieAu5Lc02/rq8BHkxzSv6Yf7+fdYp/0yzwnyR8BlwMbu1F+QZJ3A0vo3viv7Mcfzqb+5P+Krhtm6IL/Z9j097Fn/9q+MObr0JNk8Lfj36vqkP60+0q6Nv4/BP4ncHVVvTbdPQOuGVnm+yPDj7Pp7+UxuoD5z8BswT9XZtt+gNVV9a4nsfwTM9b1BNv426+qLyR5Gd0NY85N8sGq+ovtbOt7/RsIY9a4rXkuAI4Fvkz3BlpJXk53RnZ4Vf1bkmvojtABflD9oTOb/562Z2Z/LQWcBjxEdzewnYDvwdb3SZIX0v0dnNzX/Kt0ZxMrq+rWJCcAL99OHQHeU1V/Ombdepps429MVf0b3S3u3pbuTkfPBB7oJ58w7mro/sEPSvLOWab/A7AyXU+LewCv7cdty7fpmg+2Zw3wuiT/Af7/PUmfPWbds9V5XN9WPUV3N6zr+/U9VFV/RnensI1dJf+gP+Kdixq3Nc+n6Pqmfz3dmwB0v6dv9KF/EN2tPLdne/v0l5Ls1Lf/Pwf4Sr+dDf2ZwBvpmnWYbZ+k+3TSTlV1EfCbbNpPzwA29Ptq9CL+F+ma12Dz2ypeCfxquvtWkGTZxv2iYRj8Daqqm+navF9P1yb8niQ38yTOAPuj29fT9Tz6phnTbqI76rue7jrA2f02t+Vquou5oxd3Z9vunXQh89l0PVpeRdcu/1R8im4/3Ap8DnhHdV0lvxy4td8nx9HdFxe6dugvzXZx98nWuK15quobdM1xz66q6/tFrgAWJ1lH10PjF8d4fRcAb0934f65s0y/n+539Bng5Kr6Ht21hlXpLlIfRHdzGJh9nywDrumbEP8S2Hj28lt0v/d/pDtr2ehU4PT+9T4P+Gb/ej9L1/RzbZLb6HrpHOcgQE+RvXNKDUpyLnBZVV04j9vcna7JsZIcT3eh95j52r42sY1f0nz5KboLxwEeoWsu1AR4xC9JjbGNX5IaY/BLUmMMfklqjMEvSY0x+CWpMf8PDqr0bo2Td+kAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "145 / 145 questions predicted.------------------------------------------------------------ \n",
            "\n",
            "Theme: Comprehensive_school\n",
            "Total Questions: 146\n",
            "Total Paragraphs: 25\n",
            "Total queries: 146\n",
            "In top 10 results, number of queries for which -\n",
            "\tRelevant paragraph found: 144 (99 %)\n",
            "\tRelevant paragraph NOT found: 2 (1 %)\n",
            "Mean Rank for which relevant paragraph found: 2.05\n",
            "\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAUlElEQVR4nO3de9RddX3n8feHBMrNGpBMFg0sg8pIWc4SbHS4WJcj2mFKK3GGcllWQ8uUxciIgKNi21l2ZpwRq+OldapNwRJXGSgFLAgVZCJIp0Ug3C/RgUEuYQJ5qoK3KgLf+WPv7BwenySH5DlnP+G8X2s96+zz27fvs5PnfM5v73N+O1WFJEkAO/RdgCRp7jAUJEkdQ0GS1DEUJEkdQ0GS1JnfdwHbYq+99qolS5b0XYYkbVduueWWf6iqhTPN265DYcmSJaxevbrvMiRpu5LkoU3N8/SRJKljKEiSOoaCJKljKEiSOoaCJKljKEiSOoaCJKljKEiSOoaCJKmzXX+jeVssOevK3vb94NlH9bZvSdqckfUUknw+yfokdw+07ZnkmiT3tY97tO1J8kdJ7k9yZ5LXjKouSdKmjfL00XnAkdPazgJWVdX+wKr2OcC/AvZvf04GPjvCuiRJmzCyUKiq64HvTGs+GljZTq8Elg20f6EaXwcWJNl7VLVJkmY27gvNi6pqXTv9GLConV4MPDKw3Nq27WckOTnJ6iSrp6amRlepJE2g3j59VFUF1Fast6KqllbV0oULZxwOXJK0lcYdCo9vOC3UPq5v2x8F9h1Ybp+2TZI0RuMOhcuB5e30cuCygfZ3tp9COgR4cuA0kyRpTEb2PYUkFwBvBPZKshb4EHA2cFGSk4CHgGPbxf8G+FXgfuBHwG+Nqi5J0qaNLBSq6oRNzDpihmULOHVUtUiShuMwF5KkjqEgSeoYCpKkjqEgSeoYCpKkjqEgSeoYCpKkjqEgSeoYCpKkjqEgSeoYCpKkjqEgSeoYCpKkjqEgSeoYCpKkjqEgSeoYCpKkjqEgSeoYCpKkjqEgSeoYCpKkjqEgSeoYCpKkjqEgSeoYCpKkjqEgSeoYCpKkjqEgSeoYCpKkjqEgSeoYCpKkTi+hkOSMJPckuTvJBUl2TrJfkhuT3J/kL5Ps1EdtkjTJxh4KSRYDpwFLq+pVwDzgeOCjwCer6hXAd4GTxl2bJE26vk4fzQd2STIf2BVYB7wJuLidvxJY1lNtkjSxxh4KVfUo8HHgYZoweBK4BXiiqp5uF1sLLJ5p/SQnJ1mdZPXU1NQ4SpakidHH6aM9gKOB/YBfAHYDjhx2/apaUVVLq2rpwoULR1SlJE2mPk4fvRn4VlVNVdVPgUuBw4EF7ekkgH2AR3uoTZImWh+h8DBwSJJdkwQ4ArgXuBY4pl1mOXBZD7VJ0kTr45rCjTQXlG8F7mprWAF8ADgzyf3AS4Bzx12bJE26+VteZPZV1YeAD01rfgB4XQ/lSJJafqNZktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJnV5CIcmCJBcn+UaSNUkOTbJnkmuS3Nc+7tFHbZI0yfrqKXwauKqqDgBeDawBzgJWVdX+wKr2uSRpjMYeCkleDLwBOBegqp6qqieAo4GV7WIrgWXjrk2SJl0fPYX9gCngz5PcluScJLsBi6pqXbvMY8CimVZOcnKS1UlWT01NjalkSZoMQ4VCksOHaRvSfOA1wGer6mDgh0w7VVRVBdRMK1fViqpaWlVLFy5cuJUlSJJmMmxP4Y+HbBvGWmBtVd3YPr+YJiQeT7I3QPu4fiu3L0naSvM3NzPJocBhwMIkZw7M+nlg3tbssKoeS/JIkldW1TeBI4B725/lwNnt42Vbs31J0tbbbCgAOwG7t8u9aKD9e8Ax27DfdwPnJ9kJeAD4LZpey0VJTgIeAo7dhu1LkrbCZkOhqr4GfC3JeVX10GzttKpuB5bOMOuI2dqHJOn521JPYYOfS7ICWDK4TlW9aRRFSZL6MWwo/BXwOeAc4JnRlSNJ6tOwofB0VX12pJVIkno37EdSv5TkXUn2bsco2jPJniOtTJI0dsP2FJa3j+8baCvgZbNbjiSpT0OFQlXtN+pCJEn9GyoUkrxzpvaq+sLsliNJ6tOwp49eOzC9M833CW4FDAVJegEZ9vTRuwefJ1kAXDiSiiRJvdnaobN/SDMEtiTpBWTYawpfYuNQ1vOAXwQuGlVRkqR+DHtN4eMD008DD1XV2hHUI0nq0VCnj9qB8b5BM1LqHsBToyxKktSPYe+8dixwE/AbNENa35hkW4bOliTNQcOePvo94LVVtR4gyULgf9HcNU2S9AIx7KePdtgQCK1vP491JUnbiWF7ClcluRq4oH1+HPA3oylJktSXLd2j+RXAoqp6X5J/Dby+nXUDcP6oi5MkjdeWegqfAj4IUFWXApcCJPln7bxfH2l1kqSx2tJ1gUVVddf0xrZtyUgqkiT1ZkuhsGAz83aZzUIkSf3bUiisTvI70xuT/FvgltGUJEnqy5auKZwOfDHJ29kYAkuBnYC3jbIwSdL4bTYUqupx4LAk/wJ4Vdt8ZVV9deSVSZLGbtj7KVwLXDviWiRJPfNbyZKkjqEgSeoYCpKkjqEgSeoYCpKkjqEgSer0FgpJ5iW5LckV7fP9ktyY5P4kf5lkp75qk6RJ1WdP4T3AmoHnHwU+WVWvAL4LnNRLVZI0wXoJhST7AEcB57TPA7yJjbf3XAks66M2SZpkffUUPgW8H3i2ff4S4Imqerp9vhZY3EdhkjTJxh4KSX4NWF9VWzXKapKTk6xOsnpqamqWq5OkydZHT+Fw4K1JHgQupDlt9GlgQZINYzHtAzw608pVtaKqllbV0oULF46jXkmaGGMPhar6YFXtU1VLgOOBr1bV22kG3DumXWw5cNm4a5OkSTeXvqfwAeDMJPfTXGM4t+d6JGniDDV09qhU1XXAde30A8Dr+qxHkibdXOopSJJ6ZihIkjqGgiSpYyhIkjqGgiSpYyhIkjqGgiSpYyhIkjqGgiSpYyhIkjqGgiSpYyhIkjqGgiSpYyhIkjqGgiSpYyhIkjqGgiSpYyhIkjqGgiSpYyhIkjqGgiSpYyhIkjqGgiSpYyhIkjqGgiSpYyhIkjqGgiSpYyhIkjqGgiSpYyhIkjqGgiSpYyhIkjpjD4Uk+ya5Nsm9Se5J8p62fc8k1yS5r33cY9y1SdKk66On8DTw3qo6EDgEODXJgcBZwKqq2h9Y1T6XJI3R2EOhqtZV1a3t9PeBNcBi4GhgZbvYSmDZuGuTpEnX6zWFJEuAg4EbgUVVta6d9RiwaBPrnJxkdZLVU1NTY6lTkiZFb6GQZHfgEuD0qvre4LyqKqBmWq+qVlTV0qpaunDhwjFUKkmTo5dQSLIjTSCcX1WXts2PJ9m7nb83sL6P2iRpkvXx6aMA5wJrquoTA7MuB5a308uBy8ZdmyRNuvk97PNw4B3AXUlub9t+FzgbuCjJScBDwLE91CZJE23soVBV/xvIJmYfMc5aJEnP5TeaJUkdQ0GS1DEUJEkdQ0GS1DEUJEkdQ0GS1OnjewoTb8lZV/ay3wfPPqqX/UrafthTkCR1DAVJUsdQkCR1DAVJUsdQkCR1DAVJUsdQkCR1DAVJUsdQkCR1DAVJUsdhLjQWDu0hbR/sKUiSOoaCJKljKEiSOoaCJKljKEiSOn76SNJ2z0+3zR57CpKkjj2FCdLXuylJ2w97CpKkjqEgSeoYCpKkjtcUpBcYP4mjbWFPQZLUsacgSVupz0/0japnNqd6CkmOTPLNJPcnOavveiRp0syZnkKSecD/AN4CrAVuTnJ5Vd3bb2XanvndjPHxWL8wzKWewuuA+6vqgap6CrgQOLrnmiRposyZngKwGHhk4Pla4J9PXyjJycDJ7dMfJPnmGGobpb2Af+i7iDnE47GRx+K5PB4D8tFtOh4v3dSMuRQKQ6mqFcCKvuuYLUlWV9XSvuuYKzweG3ksnsvj8VyjOh5z6fTRo8C+A8/3adskSWMyl0LhZmD/JPsl2Qk4Hri855okaaLMmdNHVfV0kn8PXA3MAz5fVff0XNY4vGBOhc0Sj8dGHovn8ng810iOR6pqFNuVJG2H5tLpI0lSzwwFSVLHUOhJkn2TXJvk3iT3JHlP3zX1Lcm8JLcluaLvWvqWZEGSi5N8I8maJIf2XVOfkpzR/p3cneSCJDv3XdO4JPl8kvVJ7h5o2zPJNUnuax/3mK39GQr9eRp4b1UdCBwCnJrkwJ5r6tt7gDV9FzFHfBq4qqoOAF7NBB+XJIuB04ClVfUqmg+iHN9vVWN1HnDktLazgFVVtT+wqn0+KwyFnlTVuqq6tZ3+Ps0f/eJ+q+pPkn2Ao4Bz+q6lb0leDLwBOBegqp6qqif6rap384FdkswHdgX+X8/1jE1VXQ98Z1rz0cDKdnolsGy29mcozAFJlgAHAzf2W0mvPgW8H3i270LmgP2AKeDP29Np5yTZre+i+lJVjwIfBx4G1gFPVtVX+q2qd4uqal07/RiwaLY2bCj0LMnuwCXA6VX1vb7r6UOSXwPWV9UtfdcyR8wHXgN8tqoOBn7ILJ4e2N6058uPpgnLXwB2S/Kb/VY1d1TzvYJZ+26BodCjJDvSBML5VXVp3/X06HDgrUkepBkd901J/qLfknq1FlhbVRt6jhfThMSkejPwraqaqqqfApcCh/VcU98eT7I3QPu4frY2bCj0JElozhmvqapP9F1Pn6rqg1W1T1UtobmA+NWqmth3glX1GPBIkle2TUcAk3xfkYeBQ5Ls2v7dHMEEX3hvXQ4sb6eXA5fN1oYNhf4cDryD5l3x7e3Pr/ZdlOaMdwPnJ7kTOAj4bz3X05u2x3QxcCtwF83r1sQMeZHkAuAG4JVJ1iY5CTgbeEuS+2h6UmfP2v4c5kKStIE9BUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CYcEmeaT8Oe3eSLyVZsA3b+sE2rHtaOxro+dPaDxr8qG6SP0jyH7Z2P6OQ5HdnYRsPJtlrNuoZYl/LHHxRm2Io6B+r6qB29MnvAKf2VMe7gLdU1duntR8EzPXvbwwVCknmjbqQIS0DDAXNyFDQoBtoR2pN8rokN7QDsv39hm/XJjkxyaVJrmrHcv/D6RtJsle77lEzzDuz7ZXcneT0tu1zwMuALyc5Y2DZnYD/DBzX9maOa2cdmOS6JA8kOW1g+d9MclO77J/O9CLcviP/SLvM6iSvSXJ1kv+b5JR2mST5WFvjXRv2m2TvJNcP9Kx+OcnZNKN33j69l9Ou84Mk/z3JHcChQ9b4M8skOSXJxwaWOTHJZ9rpv05yS5r7DZw8bd//NckdSb6eZFGSw4C3Ah9rt//yafs+L8nn2mPzf9pxqUiyJMnfJrm1/TlsM8dkXrudDcfvjHbZ30lyc1vPJUl2bdtf3tZ3V5IPD/Y4k7yvXefOJP9p+rHSCFSVPxP8A/ygfZwH/BVwZPv854H57fSbgUva6ROBB4AXAzsDDwH7btgWzWiNN9K865++r1+i+UbqbsDuwD3Awe28B4G9ZljnROAzA8//APh74OeAvYBvAzsCvwh8CdixXe5PgHfOsL0HgX/XTn8SuBN4EbAQeLxt/zfANe0xWUQzzMLewHuB3xs4Xi8aPIabOL4FHNtOb7LGDb//ppZp67t/YLtfBl7fTu/ZPu4C3A28ZGDfv95O/yHw++30ecAxm6j3POAqmjeM+9OMw7QzzXDVO7fL7A+sbqd/5pi0/87XDGxzQfv4koG2DwPvbqevAE5op09h4//JX6H55nLaeq4A3tD338wL/Wc+mnS7JLmdpoewhubFEJoX/ZVJ9qd5cdlxYJ1VVfUkQJJ7gZcCj7TLrAJOraqvzbCv1wNfrKoftuteCvwycNvzrPnKqvoJ8JMk62leuI+geTG6OQk0L5CbGiTs8vbxLmD3au5n8f0kP0lzTeX1wAVV9QzNwGNfA14L3Ax8Ps1Ahn9dVbcPUeszNIMeMmSNMy5TVVNtz+gQ4D7gAODv2nVOS/K2dnpfmhftbwNP0byQAtwCvGWIegEuqqpngfuSPNDu61vAZ5Ic1P5O/7Rd9meOSbvOy5L8MXAlsGGY61cl+TCwgOZNwdVt+6FsvB/A/6QZJhuaUPgVNv7/2L393a4f8vfQVjAU9I9VdVDblb+a5prCHwH/Bbi2qt6W5n4P1w2s85OB6WfY+P/oaZoXn38JzBQKs2Wm/QdYWVUffB7rPzttW8+ymb+Jqro+yRtobgZ0XpJPVNUXtrCvH7fhwpA1bm6ZC4FjgW/QhGsleSNNT+7QqvpRkuto3tkD/LTat9w8999pS6aPfVPAGcDjNHeB2wH4MWz6mCR5Nc3/g1Pamn+bpheyrKruSHIi8MYt1BHgI1X1p0PWrVngNQUBUFU/ornl4XvT3N3qxcCj7ewTh90MzR//AUk+MMP8vwWWpRntcjfgbW3b5nyf5pTElqwCjknyT6C7h+1Lh6x7pjqPa8+NL6S5C9pN7fYer6o/o7lD3IbhrH/avlOejRo3t8wXae4rcAJNQEDz7/TdNhAOoLm165Zs6Zj+RpId2usNLwO+2e5nXduDeAfNqSJmOiZpPkW1Q1VdAvw+G4/Ti4B17bEa/EDB12lO2cFzb7N5NfDbae45QpLFG46LRsdQUKeqbqM5x34CzTnojyS5jefRo2zfFZ9AM/rru6bNu5Xm3eJNNNcdzmn3uTnX0lxYHrzQPNN+76V5AfpKmpFFr6G5DrA1vkhzHO4Avgq8v5rhrN8I3NEek+No7qMMzXnvO2e60Px8a9zcMlX1XZpTfC+tqpvaVa4C5idZQzNS5teH+P0uBN6X5kMEL59h/sM0/0ZfBk6pqh/TXNtYnuaC+QE0N/6BmY/JYuC69rTkXwAbej3/kebf/e9oejsbnA6c2f6+rwCebH/fr9CcTrohyV00I6UO8wZB28BRUiV1kpwHXFFVF49xn7vSnMasJMfTXHQ+elz713N5TUFS336J5iJ2gCdoTkGqJ/YUJEkdrylIkjqGgiSpYyhIkjqGgiSpYyhIkjr/H9JLX8GxRcYIAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "146 / 146 questions predicted.------------------------------------------------------------ \n",
            "\n",
            "Theme: Prime_minister\n",
            "Total Questions: 141\n",
            "Total Paragraphs: 36\n",
            "Total queries: 141\n",
            "In top 10 results, number of queries for which -\n",
            "\tRelevant paragraph found: 136 (96 %)\n",
            "\tRelevant paragraph NOT found: 5 (4 %)\n",
            "Mean Rank for which relevant paragraph found: 1.88\n",
            "\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAUk0lEQVR4nO3dfbRldX3f8feHGQhP6oDcziIDy0GlEpZdghktD8ZlRVMqiYwt4WEZHRIaFpWKgFUxSZdpaytG60NioyFgGFcohgAGhAjSESRNEBieH0YLRR6GDsyNCj5FEfj2j71nz5nrnZkD3HP2Hc77tdZdZ+/f3vvs790z93zOb+9zfjtVhSRJANv1XYAkaf4wFCRJHUNBktQxFCRJHUNBktRZ2HcBz8Uee+xRS5cu7bsMSdqm3HTTTf9QVVOzLdumQ2Hp0qWsXr267zIkaZuS5IHNLfP0kSSpM7JQSPL5JOuT3DnQtnuSq5Lc0z7u1rYnyR8luTfJ7UlePaq6JEmbN8qewrnA4TPazgBWVdW+wKp2HuBfAfu2PycCnx1hXZKkzRhZKFTVtcB3ZzQfCaxsp1cCywfav1CNbwCLkuw5qtokSbMb9zWFxVW1rp1+BFjcTi8BHhpYb23b9nOSnJhkdZLV09PTo6tUkiZQbxeaqxmJ7xmPxldVZ1XVsqpaNjU16yeqJEnP0rhD4dENp4Xax/Vt+8PA3gPr7dW2SZLGaNyhcCmwop1eAVwy0P7O9lNIBwGPD5xmkiSNyci+vJbkfOANwB5J1gIfAs4ELkhyAvAAcHS7+t8AbwHuBX4M/Nao6pIkbd7IQqGqjtvMosNmWbeAk0dVy2yWnnH5OHe3ifvPPKK3fUvSlviNZklSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHV6CYUkpyW5K8mdSc5PsmOSfZJcn+TeJH+ZZIc+apOkSTb2UEiyBDgFWFZVrwQWAMcCHwU+WVUvB74HnDDu2iRp0vV1+mghsFOShcDOwDrgjcCF7fKVwPKeapOkiTX2UKiqh4GPAw/ShMHjwE3AY1X1ZLvaWmDJbNsnOTHJ6iSrp6enx1GyJE2MPk4f7QYcCewD/CKwC3D4sNtX1VlVtayqlk1NTY2oSkmaTH2cPnoT8O2qmq6qnwEXA4cCi9rTSQB7AQ/3UJskTbQ+QuFB4KAkOycJcBhwN3A1cFS7zgrgkh5qk6SJ1sc1hetpLijfDNzR1nAW8AHg9CT3Ai8Gzhl3bZI06RZufZW5V1UfAj40o/k+4LU9lCNJavmNZklSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSp5dQSLIoyYVJvplkTZKDk+ye5Kok97SPu/VRmyRNsr56Cp8Grqiq/YBXAWuAM4BVVbUvsKqdlySN0dhDIcmLgNcD5wBU1RNV9RhwJLCyXW0lsHzctUnSpOujp7APMA38eZJbkpydZBdgcVWta9d5BFg828ZJTkyyOsnq6enpMZUsSZOhj1BYCLwa+GxVHQj8iBmniqqqgJpt46o6q6qWVdWyqampkRcrSZOkj1BYC6ytquvb+QtpQuLRJHsCtI/re6hNkiba2EOhqh4BHkryirbpMOBu4FJgRdu2Arhk3LVJ0qRb2NN+3w2cl2QH4D7gt2gC6oIkJwAPAEf3VJskTaxeQqGqbgWWzbLosHHXIknaaKjTR0kOHaZNkrRtG/aawh8P2SZJ2oZt8fRRkoOBQ4CpJKcPLHohsGCUhUmSxm9r1xR2AHZt13vBQPv3gaNGVZQkqR9bDIWq+jrw9STnVtUDY6pJktSTYT999AtJzgKWDm5TVW8cRVGSpH4MGwp/BXwOOBt4anTlSJL6NGwoPFlVnx1pJZKk3g37kdQvJ3lXkj3bm+HsnmT3kVYmSRq7YXsKG8Yket9AWwEvndtyJEl9GioUqmqfURciSerfUKGQ5J2ztVfVF+a2HElSn4Y9ffSagekdaQauuxkwFCTpeWTY00fvHpxPsgj44kgqkiT15tneZOdHNPdaliQ9jwx7TeHLbLxn8gLgl4ALRlWUJKkfw15T+PjA9JPAA1W1dgT1SJJ6NNTpo3ZgvG/SjJS6G/DEKIuSJPVj2DuvHQ3cAPwGzb2Tr0/i0NmS9Dwz7Omj3wNeU1XrAZJMAf8LuHBUhUmSxm/YTx9ttyEQWt95BttKkrYRw/YUrkhyJXB+O38M8DejKUmS1Jet3aP55cDiqnpfkn8NvK5ddB1w3qiLkySN19Z6Cp8CPghQVRcDFwMk+Wftsl8faXWSpLHa2nWBxVV1x8zGtm3pSCqSJPVma6GwaAvLdprLQiRJ/dtaKKxO8jszG5P8W+Cm0ZQkSerL1q4pnAp8Kcnb2RgCy4AdgLeNsjBJ0vhtMRSq6lHgkCT/Anhl23x5VX1t5JVJksZu2PspXA1cPeJaJEk981vJkqSOoSBJ6hgKkqROb6GQZEGSW5Jc1s7vk+T6JPcm+cskO/RVmyRNqj57Cu8B1gzMfxT4ZFW9HPgecEIvVUnSBOslFJLsBRwBnN3OB3gjG+/PsBJY3kdtkjTJ+uopfAp4P/B0O/9i4LGqerKdXwssmW3DJCcmWZ1k9fT09OgrlaQJMvZQSPJrwPqqelbDZFTVWVW1rKqWTU1NzXF1kjTZhr3Jzlw6FHhrkrcAOwIvBD4NLEqysO0t7AU83ENtkjTRxt5TqKoPVtVeVbUUOBb4WlW9neYb00e1q60ALhl3bZI06ebT9xQ+AJye5F6aawzn9FyPJE2cPk4fdarqGuCadvo+4LV91iNJk24+9RQkST0zFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQZeygk2TvJ1UnuTnJXkve07bsnuSrJPe3jbuOuTZImXR89hSeB91bV/sBBwMlJ9gfOAFZV1b7AqnZekjRGYw+FqlpXVTe30z8A1gBLgCOBle1qK4Hl465NkiZdr9cUkiwFDgSuBxZX1bp20SPA4p7KkqSJ1VsoJNkVuAg4taq+P7isqgqozWx3YpLVSVZPT0+PoVJJmhy9hEKS7WkC4byqurhtfjTJnu3yPYH1s21bVWdV1bKqWjY1NTWegiVpQvTx6aMA5wBrquoTA4suBVa00yuAS8ZdmyRNuoU97PNQ4B3AHUlubdt+FzgTuCDJCcADwNE91CZJE23soVBV/xvIZhYfNs5aJEmb8hvNkqSOoSBJ6hgKkqSOoSBJ6hgKkqSOoSBJ6hgKkqSOoSBJ6hgKkqSOoSBJ6hgKkqSOoSBJ6hgKkqSOoSBJ6vRxP4WJt/SMy3vZ7/1nHtHLfiVtO+wpSJI69hQmSF89FLCXIm0r7ClIkjqGgiSpYyhIkjqGgiSpYyhIkjqGgiSpYyhIkjqGgiSpYyhIkjp+o1lj4XhP0rbBnoIkqWMoSJI6hoIkqeM1BT2veS1DembsKUiSOoaCJKnj6SNpBPq8oZHG5/l4mnBe9RSSHJ7kW0nuTXJG3/VI0qSZNz2FJAuA/wG8GVgL3Jjk0qq6u9/KJGl2z8db3M6nnsJrgXur6r6qegL4InBkzzVJ0kSZNz0FYAnw0MD8WuCfz1wpyYnAie3sD5N8awy1jdIewD/0XcQ84vHYyGOxKY/HgHz0OR2Pl2xuwXwKhaFU1VnAWX3XMVeSrK6qZX3XMV94PDbyWGzK47GpUR2P+XT66GFg74H5vdo2SdKYzKdQuBHYN8k+SXYAjgUu7bkmSZoo8+b0UVU9meTfA1cCC4DPV9VdPZc1Ds+bU2FzxOOxkcdiUx6PTY3keKSqRvG8kqRt0Hw6fSRJ6pmhIEnqGAo9SbJ3kquT3J3kriTv6bumviVZkOSWJJf1XUvfkixKcmGSbyZZk+TgvmvqU5LT2r+TO5Ocn2THvmsalySfT7I+yZ0DbbsnuSrJPe3jbnO1P0OhP08C762q/YGDgJOT7N9zTX17D7Cm7yLmiU8DV1TVfsCrmODjkmQJcAqwrKpeSfNBlGP7rWqszgUOn9F2BrCqqvYFVrXzc8JQ6ElVrauqm9vpH9D80S/pt6r+JNkLOAI4u+9a+pbkRcDrgXMAquqJqnqs36p6txDYKclCYGfg//Vcz9hU1bXAd2c0HwmsbKdXAsvnan+GwjyQZClwIHB9v5X06lPA+4Gn+y5kHtgHmAb+vD2ddnaSXfouqi9V9TDwceBBYB3weFV9td+qere4qta1048Ai+fqiQ2FniXZFbgIOLWqvt93PX1I8mvA+qq6qe9a5omFwKuBz1bVgcCPmMPTA9ua9nz5kTRh+YvALkl+s9+q5o9qvlcwZ98tMBR6lGR7mkA4r6ou7rueHh0KvDXJ/TSj474xyV/0W1Kv1gJrq2pDz/FCmpCYVG8Cvl1V01X1M+Bi4JCea+rbo0n2BGgf18/VExsKPUkSmnPGa6rqE33X06eq+mBV7VVVS2kuIH6tqib2nWBVPQI8lOQVbdNhwCTfV+RB4KAkO7d/N4cxwRfeW5cCK9rpFcAlc/XEhkJ/DgXeQfOu+Nb25y19F6V5493AeUluBw4A/lvP9fSm7TFdCNwM3EHzujUxQ14kOR+4DnhFkrVJTgDOBN6c5B6antSZc7Y/h7mQJG1gT0GS1DEUJEkdQ0GS1DEUJEkdQ0GS1DEUJlySp9qPw96Z5MtJFj2H5/rhc9j2lHY00PNmtB8w+FHdJH+Q5D882/2MQpLfnYPnuD/JHnNRzxD7Wu7gi9ocQ0H/WFUHtKNPfhc4uac63gW8uarePqP9AGC+f39jqFBIsmDUhQxpOWAoaFaGggZdRztSa5LXJrmuHZDt7zd8uzbJ8UkuTnJFO5b7H858kiR7tNseMcuy09teyZ1JTm3bPge8FPhKktMG1t0B+M/AMW1v5ph20f5JrklyX5JTBtb/zSQ3tOv+6Wwvwu078o+066xO8uokVyb5v0lOatdJko+1Nd6xYb9J9kxy7UDP6leSnEkzeuetM3s57TY/TPLfk9wGHDxkjT+3TpKTknxsYJ3jk3ymnf7rJDelud/AiTP2/V+T3JbkG0kWJzkEeCvwsfb5XzZj3+cm+Vx7bP5POy4VSZYm+dskN7c/h2zhmCxon2fD8TutXfd3ktzY1nNRkp3b9pe19d2R5MODPc4k72u3uT3Jf5p5rDQCVeXPBP8AP2wfFwB/BRzezr8QWNhOvwm4qJ0+HrgPeBGwI/AAsPeG56IZrfF6mnf9M/f1yzTfSN0F2BW4CziwXXY/sMcs2xwPfGZg/g+Avwd+AdgD+A6wPfBLwJeB7dv1/gR45yzPdz/w79rpTwK3Ay8ApoBH2/Z/A1zVHpPFNMMs7Am8F/i9geP1gsFjuJnjW8DR7fRma9zw+29unba+ewee9yvA69rp3dvHnYA7gRcP7PvX2+k/BH6/nT4XOGoz9Z4LXEHzhnFfmnGYdqQZrnrHdp19gdXt9M8dk/bf+aqB51zUPr54oO3DwLvb6cuA49rpk9j4f/JXab65nLaey4DX9/0383z/WYgm3U5JbqXpIayheTGE5kV/ZZJ9aV5cth/YZlVVPQ6Q5G7gJcBD7TqrgJOr6uuz7Ot1wJeq6kftthcDvwLc8gxrvryqfgr8NMl6mhfuw2hejG5MAs0L5OYGCbu0fbwD2LWa+1n8IMlP01xTeR1wflU9RTPw2NeB1wA3Ap9PM5DhX1fVrUPU+hTNoIcMWeOs61TVdNszOgi4B9gP+Lt2m1OSvK2d3pvmRfs7wBM0L6QANwFvHqJegAuq6mngniT3tfv6NvCZJAe0v9M/bdf9uWPSbvPSJH8MXA5sGOb6lUk+DCyieVNwZdt+MBvvB/A/aYbJhiYUfpWN/z92bX+3a4f8PfQsGAr6x6o6oO3KX0lzTeGPgP8CXF1Vb0tzv4drBrb56cD0U2z8f/QkzYvPvwRmC4W5Mtv+A6ysqg8+g+2fnvFcT7OFv4mqujbJ62luBnRukk9U1Re2sq+ftOHCkDVuaZ0vAkcD36QJ10ryBpqe3MFV9eMk19C8swf4WbVvudn032lrZo59U8BpwKM0d4HbDvgJbP6YJHkVzf+Dk9qaf5umF7K8qm5Lcjzwhq3UEeAjVfWnQ9atOeA1BQFQVT+mueXhe9Pc3epFwMPt4uOHfRqaP/79knxgluV/CyxPM9rlLsDb2rYt+QHNKYmtWQUcleSfQHcP25cMWfdsdR7TnhuforkL2g3t8z1aVX9Gc4e4DcNZ/6x9pzwXNW5pnS/R3FfgOJqAgObf6XttIOxHc2vXrdnaMf2NJNu11xteCnyr3c+6tgfxDppTRcx2TNJ8imq7qroI+H02HqcXAOvaYzX4gYJv0Jyyg01vs3kl8Ntp7jlCkiUbjotGx1BQp6puoTnHfhzNOeiPJLmFZ9CjbN8VH0cz+uu7Ziy7mebd4g001x3Obve5JVfTXFgevNA8237vpnkB+mqakUWvorkO8Gx8ieY43AZ8DXh/NcNZvwG4rT0mx9DcRxma8963z3ah+ZnWuKV1qup7NKf4XlJVN7SbXAEsTLKGZqTMbwzx+30ReF+aDxG8bJblD9L8G30FOKmqfkJzbWNFmgvm+9Hc+AdmPyZLgGva05J/AWzo9fxHmn/3v6Pp7WxwKnB6+/u+HHi8/X2/SnM66bokd9CMlDrMGwQ9B46SKqmT5Fzgsqq6cIz73JnmNGYlOZbmovOR49q/NuU1BUl9+2Wai9gBHqM5Bame2FOQJHW8piBJ6hgKkqSOoSBJ6hgKkqSOoSBJ6vx/nQpc5mWLMcgAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "141 / 141 questions predicted.------------------------------------------------------------ \n",
            "\n",
            "Theme: Institute_of_technology\n",
            "Total Questions: 99\n",
            "Total Paragraphs: 62\n",
            "Total queries: 99\n",
            "In top 10 results, number of queries for which -\n",
            "\tRelevant paragraph found: 93 (94 %)\n",
            "\tRelevant paragraph NOT found: 6 (6 %)\n",
            "Mean Rank for which relevant paragraph found: 1.71\n",
            "\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAV6ElEQVR4nO3dfbRddX3n8ffHBAoCFZA0iwE0qIyU5SyDXhlA67Iijq2txBnLw7I2TJlmOToq6mix7azVzjgj1taHqTPaDFrSVQoiDwVhBJkI2qkIhidBgoNSUJiQ3KL4WEHwO3/sncnNzU1yEu4+J+H3fq111tnP+3v3vfdz9vmdfX47VYUkqR1PmXQBkqTxMvglqTEGvyQ1xuCXpMYY/JLUmIWTLmAUBx10UC1ZsmTSZUjSbuWmm276h6paNHv6bhH8S5YsYc2aNZMuQ5J2K0num2u6TT2S1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktSY3eKbu0/EkrOunMh+7z371RPZryRtj2f8ktSYwYI/yXOT3Drj8f0kZyY5MMk1Se7unw8YqgZJ0pYGC/6q+npVLa2qpcALgR8DlwJnAaur6ghgdT8uSRqTcTX1nAB8s6ruA04CVvXTVwHLxlSDJInxBf+pwPn98OKqWtcPPwgsnmuFJCuSrEmyZnp6ehw1SlITBg/+JHsCrwE+PXteVRVQc61XVSuraqqqphYt2uI+ApKknTSOM/5fAW6uqvX9+PokBwP0zxvGUIMkqTeO4D+NTc08AJcDy/vh5cBlY6hBktQbNPiT7AOcCFwyY/LZwIlJ7gZe0Y9LksZk0G/uVtWPgKfPmvYQ3VU+kqQJ8Ju7ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqzKDBn2T/JBcluSvJ2iTHJTkwyTVJ7u6fDxiyBknS5oY+4/8IcFVVHQk8H1gLnAWsrqojgNX9uCRpTAYL/iRPA14KfAKgqh6tqoeBk4BV/WKrgGVD1SBJ2tKQZ/yHA9PAXyS5Jck5SfYBFlfVun6ZB4HFc62cZEWSNUnWTE9PD1imJLVlyOBfCLwA+FhVHQ38iFnNOlVVQM21clWtrKqpqppatGjRgGVKUluGDP77gfur6oZ+/CK6F4L1SQ4G6J83DFiDJGmWwYK/qh4Evp3kuf2kE4A7gcuB5f205cBlQ9UgSdrSwoG3/xbgvCR7AvcA/5ruxebCJGcA9wEnD1yDJGmGQYO/qm4FpuaYdcKQ+5UkbZ3f3JWkxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMGvdl6knuBHwCPA49V1VSSA4FPAUuAe4GTq+q7Q9YhSdpkHGf8v1xVS6tqqh8/C1hdVUcAq/txSdKYTKKp5yRgVT+8Clg2gRokqVlDB38Bn0tyU5IV/bTFVbWuH34QWDzXiklWJFmTZM309PTAZUpSOwZt4wdeUlUPJPkF4Jokd82cWVWVpOZasapWAisBpqam5lxGkrTjBj3jr6oH+ucNwKXAMcD6JAcD9M8bhqxBkrS5wYI/yT5J9ts4DLwSuAO4HFjeL7YcuGyoGiRJWxqyqWcxcGmSjfv566q6KslXgAuTnAHcB5w8YA2SpFkGC/6qugd4/hzTHwJOGGq/kqRt85u7ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1JiRgj/Ji0eZJkna9Y16xv9nI06TJO3itnnP3STHAccDi5K8Y8asnwcWDFmYJGkY2zvj3xPYl+4FYr8Zj+8DrxtlB0kWJLklyRX9+OFJbkjyjSSfSrLnzpcvSdpR2zzjr6ovAF9Icm5V3beT+3gbsJbuXQLA+4EPVdUFST4OnAF8bCe3LUnaQaO28f9ckpVJPpfk8xsf21spyaHAq4Fz+vEALwcu6hdZBSzbibolSTtpm2f8M3wa+DhdgD++A9v/MPBuuuYhgKcDD1fVY/34/cAhc62YZAWwAuAZz3jGDuxSkrQtowb/Y1W1Q80xSX4N2FBVNyV52Y4WVlUrgZUAU1NTtaPrS5LmNmrwfybJm4BLgUc2Tqyq72xjnRcDr0nyq8BedG38HwH2T7KwP+s/FHhgpyqXJO2UUdv4lwPvAr4E3NQ/1mxrhap6T1UdWlVLgFOBz1fV64Fr2XRF0HLgsp2oW5K0k0Y646+qw+dxn78LXJDkvcAtwCfmcduSpO0YKfiT/NZc06vqL0dZv6quA67rh+8BjhmtPEnSfBu1jf9FM4b3Ak4AbgZGCn5J0q5j1Kaet8wcT7I/cMEgFUmSBrWz3TL/CJjPdn9J0piM2sb/GWDjtfQLgF8ELhyqKEnScEZt4/+TGcOPAfdV1f0D1CNJGthITT19Z2130XW9cADw6JBFSZKGM+oduE4GbgR+AzgZuCHJSN0yS5J2LaM29fw+8KKq2gCQZBHwv9jUy6YkaTcx6lU9T9kY+r2HdmBdSdIuZNQz/quSXA2c34+fAvzPYUqSJA1pe/fcfQ6wuKreleRfAi/pZ10PnDd0cZKk+be9M/4PA+8BqKpLgEsAkvyzft6vD1qdJGneba+dfnFV3T57Yj9tySAVSZIGtb3g338b8/aez0IkSeOxveBfk+R3Zk9M8m/obsYiSdrNbK+N/0zg0iSvZ1PQTwF7Aq8dsjBJ0jC2GfxVtR44PskvA8/rJ19ZVZ8fvDJJ0iBG7Y//Wrp75UqSdnN++1aSGjNY8CfZK8mNSW5L8rUkf9RPPzzJDUm+keRTSfYcqgZJ0paGPON/BHh5VT0fWAq8KsmxwPuBD1XVc4DvAmcMWIMkaZbBgr86P+xH9+gfBbycTb16rgKWDVWDJGlLg7bxJ1mQ5FZgA3AN8E3g4ap6rF/kfuCQIWuQJG1u0OCvqserailwKHAMcOSo6yZZkWRNkjXT09OD1ShJrRnLVT1V9TDd5aDHAfsn2XgZ6aHAA1tZZ2VVTVXV1KJFi8ZRpiQ1YcirehYl2b8f3hs4EVhL9wKw8baNy4HLhqpBkrSlUW/EsjMOBlYlWUD3AnNhVV2R5E7ggiTvBW4BPjFgDZKkWQYL/qr6KnD0HNPvoWvvlyRNgN/claTGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwYL/iSHJbk2yZ1Jvpbkbf30A5Nck+Tu/vmAoWqQJG1pyDP+x4B3VtVRwLHAm5McBZwFrK6qI4DV/bgkaUwGC/6qWldVN/fDPwDWAocAJwGr+sVWAcuGqkGStKWxtPEnWQIcDdwALK6qdf2sB4HFW1lnRZI1SdZMT0+Po0xJasLgwZ9kX+Bi4Myq+v7MeVVVQM21XlWtrKqpqppatGjR0GVKUjMGDf4ke9CF/nlVdUk/eX2Sg/v5BwMbhqxBkrS5Ia/qCfAJYG1VfXDGrMuB5f3wcuCyoWqQJG1p4YDbfjHwBuD2JLf2034POBu4MMkZwH3AyQPWIEmaZbDgr6r/DWQrs08Yar+SpG3zm7uS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWrMYMGf5JNJNiS5Y8a0A5Nck+Tu/vmAofYvSZrbkGf85wKvmjXtLGB1VR0BrO7HJUljNFjwV9UXge/MmnwSsKofXgUsG2r/kqS5jbuNf3FVreuHHwQWb23BJCuSrEmyZnp6ejzVSVIDJvbhblUVUNuYv7KqpqpqatGiRWOsTJKe3MYd/OuTHAzQP28Y8/4lqXnjDv7LgeX98HLgsjHvX5KaN+TlnOcD1wPPTXJ/kjOAs4ETk9wNvKIflySN0cKhNlxVp21l1glD7VOStH2DBX/rlpx15cT2fe/Zr57YviXt+uyyQZIaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1Jj7LLhSWhS3UXYVYS0e/CMX5IaY/BLUmNs6tG8sUdSaffgGb8kNcbgl6TG2NQjabfhFWvzwzN+SWqMwS9JjZlIU0+SVwEfARYA51TV2ZOoQ08ek7yiaFIm1fzQ4rF+sjUxjf2MP8kC4L8BvwIcBZyW5Khx1yFJrZpEU88xwDeq6p6qehS4ADhpAnVIUpMm0dRzCPDtGeP3A/989kJJVgAr+tEfJvn6GGob0kHAP0y6iF2Ex2JzO3U88v4BKtk1+PfRy/uf8LF45lwTd9nLOatqJbBy0nXMlyRrqmpq0nXsCjwWm/N4bM7jsclQx2ISTT0PAIfNGD+0nyZJGoNJBP9XgCOSHJ5kT+BU4PIJ1CFJTRp7U09VPZbk3wFX013O+cmq+tq465iAJ02z1TzwWGzO47E5j8cmgxyLVNUQ25Uk7aL85q4kNcbgl6TGGPwDSnJYkmuT3Jnka0neNumadgVJFiS5JckVk65l0pLsn+SiJHclWZvkuEnXNClJ3t7/n9yR5Pwke026pnFK8skkG5LcMWPagUmuSXJ3/3zAfOzL4B/WY8A7q+oo4FjgzXZPAcDbgLWTLmIX8RHgqqo6Eng+jR6XJIcAbwWmqup5dBd+nDrZqsbuXOBVs6adBayuqiOA1f34E2bwD6iq1lXVzf3wD+j+qQ+ZbFWTleRQ4NXAOZOuZdKSPA14KfAJgKp6tKoenmxVE7UQ2DvJQuCpwP+dcD1jVVVfBL4za/JJwKp+eBWwbD72ZfCPSZIlwNHADZOtZOI+DLwb+NmkC9kFHA5MA3/RN32dk2SfSRc1CVX1APAnwLeAdcD3qupzk61ql7C4qtb1ww8Ci+djowb/GCTZF7gYOLOqvj/peiYlya8BG6rqpknXsotYCLwA+FhVHQ38iHl6K7+76duuT6J7MfwnwD5JfnOyVe1aqrv2fl6uvzf4B5ZkD7rQP6+qLpl0PRP2YuA1Se6l65X15Un+arIlTdT9wP1VtfFd4EV0LwQtegXw91U1XVU/BS4Bjp9wTbuC9UkOBuifN8zHRg3+ASUJXfvt2qr64KTrmbSqek9VHVpVS+g+uPt8VTV7VldVDwLfTvLcftIJwJ0TLGmSvgUcm+Sp/f/NCTT6QfcslwPL++HlwGXzsVGDf1gvBt5Ad2Z7a//41UkXpV3KW4DzknwVWAr8lwnXMxH9u56LgJuB2+myqamuG5KcD1wPPDfJ/UnOAM4GTkxyN927onm5W6FdNkhSYzzjl6TGGPyS1BiDX5IaY/BLUmMMfklqjMHfiCSP95eT3pHkM0n2fwLb+uETWPetfS+U582avnTmpa5J/jDJv9/Z/Qwhye/NwzbuTXLQfNQzwr6W2Smg5mLwt+Mfq2pp3/Phd4A3T6iONwEnVtXrZ01fCuzq33EYKfiTLBi6kBEtAwx+bcHgb9P19L2EJjkmyfV9J2Ff2vgt0iSnJ7kkyVV9X+B/PHsjSQ7q1331HPPe0b+7uCPJmf20jwPPAj6b5O0zlt0T+I/AKf27klP6WUcluS7JPUneOmP530xyY7/sn88VtP2Z9fv6ZdYkeUGSq5N8M8kb+2WS5AN9jbdv3G+Sg5N8ccY7pF9KcjZdz5G3zn630q/zwyR/muQ24LgRa9ximSRvTPKBGcucnuSj/fDfJLkpXZ/1K2bt+z8nuS3Jl5MsTnI88BrgA/32nz1r3+cm+Xh/bP5P348SSZYk+dskN/eP47dxTBb029l4/N7eL/s7Sb7S13Nxkqf205/d13d7kvfOfOeY5F39Ol9N8kezj5XmWVX5aOAB/LB/XgB8GnhVP/7zwMJ++BXAxf3w6cA9wNOAvYD7gMM2bouul8Ab6M7eZ+/rhXTfvtwH2Bf4GnB0P+9e4KA51jkd+OiM8T8EvgT8HHAQ8BCwB/CLwGeAPfrl/jvwW3Ns717g3/bDHwK+CuwHLALW99P/FXBNf0wW03UbcDDwTuD3Zxyv/WYew60c3wJO7oe3WuPGn39ry/T1fWPGdj8LvKQfPrB/3hu4A3j6jH3/ej/8x8Af9MPnAq/bSr3nAlfRnfwdQddv0F503SHv1S9zBLCmH97imPS/52tmbHP//vnpM6a9F3hLP3wFcFo//EY2/U2+ku5buunruQJ46aT/Z57Mj4WoFXsnuZXuTH8tXeBBF+yrkhxBFyB7zFhndVV9DyDJncAzgW/3y6wG3lxVX5hjXy8BLq2qH/XrXgL8EnDLDtZ8ZVU9AjySZANdOJ9AFzhfSQJdCG6t46rL++fbgX2ruyfCD5I8ku4zjpcA51fV43SdYX0BeBHwFeCT6TrY+5uqunWEWh+n64yPEWucc5mqmu7f4RwL3A0cCfxdv85bk7y2Hz6MLpgfAh6lC0uAm4ATR6gX4MKq+hlwd5J7+n39PfDRJEv7n+mf9stucUz6dZ6V5M+AK4GN3Sg/L8l7gf3pXviv7qcfx6b+5P+arhtm6IL/lWz6+9i3/9m+OOLPoR1k8LfjH6tqaf+2+2q6Nv7/Cvwn4Nqqem26ewZcN2OdR2YMP86mv5fH6ALmXwBzBf98mWv/AVZV1Xt2YP2fzdrWz9jG335VfTHJS+luGHNukg9W1V9uZ18/6V9AGLHGbS1zAXAycBfdC2gleRndO7LjqurHSa6jO0MH+Gn1p85s/nvantn9tRTwdmA93d3AngL8BLZ+TJI8n+7v4I19zb9N925iWVXdluR04GXbqSPA+6rqz0esW0+QbfyNqaof093i7p3p7nT0NOCBfvbpo26G7h/8yCS/O8f8vwWWpetpcR/gtf20bfkBXfPB9qwGXpfkF+D/35P0mSPWPVedp/Rt1Yvo7oZ1Y7+99VX1P+juFLaxq+Sf9me881Hjtpa5lK5v+tPoXgSg+z19tw/9I+lu5bk92zumv5HkKX37/7OAr/f7Wde/E3gDXbMOcx2TdFcnPaWqLgb+gE3HaT9gXX+sZn6I/2W65jXY/LaKVwO/ne6+FSQ5ZONx0TAM/gZV1S10bd6n0bUJvy/JLezAO8D+7PY0up5H3zRr3s10Z3030n0OcE6/z225lu7D3Jkf7s613zvpQuZz6Xq0vIauXX5nXEp3HG4DPg+8u7qukl8G3NYfk1Po7osLXTv0V+f6cHdHa9zWMlX1XbrmuGdW1Y39KlcBC5Ospeuh8csj/HwXAO9K98H9s+eY/y2639FngTdW1U/oPmtYnu5D6iPpbg4Dcx+TQ4Dr+ibEvwI2vnv5D3S/97+je9ey0ZnAO/qf9znA9/qf93N0TT/XJ7mdrpfOUU4CtJPsnVNqUJJzgSuq6qIx7vOpdE2OleRUug96TxrX/rWJbfySxuWFdB8cB3iYrrlQE+AZvyQ1xjZ+SWqMwS9JjTH4JakxBr8kNcbgl6TG/D8TidYxNohykgAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "99 / 99 questions predicted.------------------------------------------------------------ \n",
            "\n",
            "Theme: Hydrogen\n",
            "Total Questions: 124\n",
            "Total Paragraphs: 55\n",
            "Total queries: 124\n",
            "In top 10 results, number of queries for which -\n",
            "\tRelevant paragraph found: 115 (93 %)\n",
            "\tRelevant paragraph NOT found: 9 (7 %)\n",
            "Mean Rank for which relevant paragraph found: 1.83\n",
            "\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAWzUlEQVR4nO3de7SddX3n8ffHBMpNBeQ0KwU1qIyU5YxRjw6IdVkuDq2txDUWYVkNU6ZZjo6KOir2strOOCNWx8vUGTUFS7rq4AWhIIwgK4L2gmC4yU0niKBhAjlV8Fov4Hf+eJ6Yk5OTZCfk2Tvx936tddZ+nt9z++4nJ5/z7N/e+/ekqpAkteNRky5AkjReBr8kNcbgl6TGGPyS1BiDX5Ias3DSBYzikEMOqSVLlky6DEnao1x//fX/VFVTc9v3iOBfsmQJa9asmXQZkrRHSXLPfO129UhSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmP2iG/uPhJLzrpsIse9++wXTeS4krQ9XvFLUmMMfklqjMEvSY0ZNPiTvCHJbUluTXJ+kn2SHJ7k2iR3Jvl4kr2HrEGStLnBgj/JocDrgOmqehqwADgVeCfw3qp6CvAAcMZQNUiStjR0V89CYN8kC4H9gPXAccAF/fJVwLKBa5AkzTJY8FfVvcC7gW/QBf53gOuBB6vqoX61dcCh822fZEWSNUnWzMzMDFWmJDVnyK6eg4CTgcOBXwH2B04adfuqWllV01U1PTW1xZ3DJEk7aciunhOAr1fVTFX9FLgQOBY4sO/6ATgMuHfAGiRJcwwZ/N8Ajk6yX5IAxwO3A1cBL+3XWQ5cPGANkqQ5huzjv5buTdwbgFv6Y60E3gq8McmdwOOAc4eqQZK0pUHH6qmqPwH+ZE7zXcBzhjyuJGnr/OauJDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxQ95s/alJbpr1890kZyY5OMmVSdb2jwcNVYMkaUtD3nrxq1W1tKqWAs8CfghcBJwFrK6qI4DV/bwkaUzG1dVzPPC1qroHOBlY1bevApaNqQZJEuML/lOB8/vpRVW1vp++D1g03wZJViRZk2TNzMzMOGqUpCYMHvxJ9gZeDHxy7rKqKqDm266qVlbVdFVNT01NDVylJLVjHFf8vwHcUFX39/P3J1kM0D9uGEMNkqTeOIL/NDZ18wBcAizvp5cDF4+hBklSb9DgT7I/cCJw4azms4ETk6wFTujnJUljsnDInVfVD4DHzWn7Ft2nfCRJE+A3dyWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjRn6DlwHJrkgyVeS3JHkmCQHJ7kyydr+8aAha5AkbW7oK/73A5dX1ZHA04E7gLOA1VV1BLC6n5ckjclgwZ/kscDzgXMBquonVfUgcDKwql9tFbBsqBokSVsa8or/cGAG+KskNyY5p7/5+qKqWt+vcx+waL6Nk6xIsibJmpmZmQHLlKS2DBn8C4FnAh+sqmcAP2BOt05VFVDzbVxVK6tquqqmp6amBixTktoyZPCvA9ZV1bX9/AV0fwjuT7IYoH/cMGANkqQ5Bgv+qroP+GaSp/ZNxwO3A5cAy/u25cDFQ9UgSdrSwoH3/1rgo0n2Bu4C/h3dH5tPJDkDuAc4ZeAaJEmzDBr8VXUTMD3PouOHPK4kaev85q4kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTGD3oglyd3A94CHgYeqajrJwcDHgSXA3cApVfXAkHVIkjYZxxX/r1fV0qraeCeus4DVVXUEsLqflySNySS6ek4GVvXTq4BlE6hBkpo1dPAX8Nkk1ydZ0bctqqr1/fR9wKL5NkyyIsmaJGtmZmYGLlOS2jFoHz/wvKq6N8kvA1cm+crshVVVSWq+DatqJbASYHp6et51JEk7btAr/qq6t3/cAFwEPAe4P8ligP5xw5A1SJI2N1jwJ9k/yaM3TgMvBG4FLgGW96stBy4eqgZJ0pZGCv4kx47SNsci4O+T3AxcB1xWVZcDZwMnJlkLnNDPS5LGZNQ+/r8AnjlC289V1V3A0+dp/xZw/KgFSpJ2rW0Gf5JjgOcCU0neOGvRY4AFQxYmSRrG9q749wYO6Nd79Kz27wIvHaooSdJwthn8VfV54PNJzquqe8ZUkyRpQKP28f9SkpV04+v8fJuqOm6IoiRJwxk1+D8JfAg4h27ANUnSHmrU4H+oqj44aCWSpLEY9Qtcn07y6iSLkxy88WfQyiRJgxj1in/jN23fPKutgCft2nIkSUMbKfir6vChC5EkjcdIwZ/klfO1V9Vf79pyJElDG7Wr59mzpvehG3LhBsDgl6Q9zKhdPa+dPZ/kQOBjg1QkSRrUzg7L/APAfn9J2gON2sf/abpP8UA3ONuvAp8YqihJ0nBG7eN/96zph4B7qmrdAPVIkgY2UldPP1jbV+hG6DwI+MmQRUmShjPqHbhOobuL1u8ApwDXJhlpWOYkC5LcmOTSfv7wJNcmuTPJx5PsvbPFS5J23Khv7v4h8OyqWl5Vr6S7afofj7jt64E7Zs2/E3hvVT0FeAA4Y9RiJUmP3KjB/6iq2jBr/lujbJvkMOBFdKN6kiTAccAF/SqrgGUjVytJesRGfXP38iRXAOf38y8D/s8I270PeAub7t71OODBqnqon18HHDrfhklWACsAnvCEJ4xYpiRpe7Z51Z7kKUmOrao3Ax8G/lX/cw2wcjvb/hawoaqu35nCqmplVU1X1fTU1NTO7EKSNI/tXfG/D3gbQFVdCFwIkORf9st+exvbHgu8OMlv0g3z8Bjg/cCBSRb2V/2HAfc+omcgSdoh2+unX1RVt8xt7NuWbGvDqnpbVR1WVUuAU4HPVdXLgavYdKP25cDFO1q0JGnnbS/4D9zGsn138phvBd6Y5E66Pv9zd3I/kqSdsL2unjVJfr+q/nJ2Y5J/D4zcd19VVwNX99N30X0cVJI0AdsL/jOBi5K8nE1BPw3sDbxkyMIkScPYZvBX1f3Ac5P8OvC0vvmyqvrc4JVJkgYx6nj8V9G9KStJ2sPt7Hj8kqQ9lMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYMFvxJ9klyXZKbk9yW5M/69sOTXJvkziQfT7L3UDVIkrY05BX/j4HjqurpwFLgpCRHA+8E3ltVTwEeAM4YsAZJ0hyDBX91vt/P7tX/FHAccEHfvgpYNlQNkqQtDdrHn2RBkpuADcCVwNeAB6vqoX6VdcChW9l2RZI1SdbMzMwMWaYkNWXQ4K+qh6tqKXAY3Q3Wj9yBbVdW1XRVTU9NTQ1WoyS1Ziyf6qmqB+lu3XgMcGCSjbd8PAy4dxw1SJI6Q36qZyrJgf30vsCJwB10fwBe2q+2HLh4qBokSVsa6WbrO2kxsCrJAro/MJ+oqkuT3A58LMnbgRuBcwesQZI0x2DBX1VfBp4xT/tddP39kqQJ8Ju7ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGDHnrxccnuSrJ7UluS/L6vv3gJFcmWds/HjRUDZKkLQ15xf8Q8KaqOgo4GnhNkqOAs4DVVXUEsLqflySNyWDBX1Xrq+qGfvp7dDdaPxQ4GVjVr7YKWDZUDZKkLY2ljz/JErr7714LLKqq9f2i+4BFW9lmRZI1SdbMzMyMo0xJasLgwZ/kAOBTwJlV9d3Zy6qqgJpvu6paWVXTVTU9NTU1dJmS1IxBgz/JXnSh/9GqurBvvj/J4n75YmDDkDVIkjY35Kd6ApwL3FFV75m16BJgeT+9HLh4qBokSVtaOOC+jwVeAdyS5Ka+7Q+As4FPJDkDuAc4ZcAaJElzDBb8VfX3QLay+PihjitJ2ja/uStJjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0ZcsiGpi0567KJHfvus180sWNL2v15xS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaM+StFz+SZEOSW2e1HZzkyiRr+8eDhjq+JGl+Q17xnwecNKftLGB1VR0BrO7nJUljNFjwV9UXgG/PaT4ZWNVPrwKWDXV8SdL8xt3Hv6iq1vfT9wGLtrZikhVJ1iRZMzMzM57qJKkBE3tzt6oKqG0sX1lV01U1PTU1NcbKJOkX27iD//4kiwH6xw1jPr4kNW/cwX8JsLyfXg5cPObjS1Lzhvw45/nANcBTk6xLcgZwNnBikrXACf28JGmMBhuWuapO28qi44c6piRp+/zmriQ1xuCXpMYY/JLUGINfkhrjPXe1y3ifYWnP4BW/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5Jaowf5/wFNMmPVbbGj7BqT+QVvyQ1xuCXpMbY1SNph7TYvTWp5zzU8/WKX5IaM5Er/iQnAe8HFgDnVJV34tIj0uIb2i0+Z+0aY7/iT7IA+J/AbwBHAaclOWrcdUhSqybR1fMc4M6ququqfgJ8DDh5AnVIUpMm0dVzKPDNWfPrgH89d6UkK4AV/ez3k3x1DLUN6RDgnyZdxG7Cc7E5z8fmtno+8s4xVzJheecj/t144nyNu+2neqpqJbBy0nXsKknWVNX0pOvYHXguNuf52JznY5OhzsUkunruBR4/a/6wvk2SNAaTCP4vAUckOTzJ3sCpwCUTqEOSmjT2rp6qeijJfwSuoPs450eq6rZx1zEBvzDdVruA52Jzno/NeT42GeRcpKqG2K8kaTflN3clqTEGvyQ1xuAfUJLHJ7kqye1Jbkvy+knXtDtIsiDJjUkunXQtk5bkwCQXJPlKkjuSHDPpmiYlyRv6/ye3Jjk/yT6TrmmcknwkyYYkt85qOzjJlUnW9o8H7YpjGfzDegh4U1UdBRwNvMbhKQB4PXDHpIvYTbwfuLyqjgSeTqPnJcmhwOuA6ap6Gt0HP06dbFVjdx5w0py2s4DVVXUEsLqff8QM/gFV1fqquqGf/h7df+pDJ1vVZCU5DHgRcM6ka5m0JI8Fng+cC1BVP6mqBydb1UQtBPZNshDYD/h/E65nrKrqC8C35zSfDKzqp1cBy3bFsQz+MUmyBHgGcO1kK5m49wFvAX426UJ2A4cDM8Bf9V1f5yTZf9JFTUJV3Qu8G/gGsB74TlV9drJV7RYWVdX6fvo+YNGu2KnBPwZJDgA+BZxZVd+ddD2TkuS3gA1Vdf2ka9lNLASeCXywqp4B/IBd9FJ+T9P3XZ9M98fwV4D9k/zuZKvavVT32ftd8vl7g39gSfaiC/2PVtWFk65nwo4FXpzkbrpRWY9L8jeTLWmi1gHrqmrjq8AL6P4QtOgE4OtVNVNVPwUuBJ474Zp2B/cnWQzQP27YFTs1+AeUJHT9t3dU1XsmXc+kVdXbquqwqlpC98bd56qq2au6qroP+GaSp/ZNxwO3T7CkSfoGcHSS/fr/N8fT6Bvdc1wCLO+nlwMX74qdGvzDOhZ4Bd2V7U39z29OuijtVl4LfDTJl4GlwH+bcD0T0b/quQC4AbiFLpuaGrohyfnANcBTk6xLcgZwNnBikrV0r4p2yd0KHbJBkhrjFb8kNcbgl6TGGPyS1BiDX5IaY/BLUmMM/kYkebj/OOmtST6d5MBHsK/vP4JtX9ePQvnROe1LZ3/UNcmfJvlPO3ucIST5g12wj7uTHLIr6hnhWMscFFDzMfjb8c9VtbQf+fDbwGsmVMergROr6uVz2pcCu/t3HEYK/iQLhi5kRMsAg19bMPjbdA39KKFJnpPkmn6QsH/c+C3SJKcnuTDJ5f1Y4H8+dydJDum3fdE8y97Yv7q4NcmZfduHgCcBn0nyhlnr7g38Z+Bl/auSl/WLjkpydZK7krxu1vq/m+S6ft0Pzxe0/ZX1O/p11iR5ZpIrknwtyav6dZLkXX2Nt2w8bpLFSb4w6xXSryU5m27kyJvmvlrpt/l+kv+e5GbgmBFr3GKdJK9K8q5Z65ye5AP99N8muT7dmPUr5hz7vya5OckXkyxK8lzgxcC7+v0/ec6xz0vyof7c/N9+HCWSLEnyd0lu6H+eu41zsqDfz8bz94Z+3d9P8qW+nk8l2a9vf3Jf3y1J3j77lWOSN/fbfDnJn809V9rFqsqfBn6A7/ePC4BPAif1848BFvbTJwCf6qdPB+4CHgvsA9wDPH7jvuhGCbyW7up97rGeRffty/2BA4DbgGf0y+4GDplnm9OBD8ya/1PgH4FfAg4BvgXsBfwq8Glgr369/wW8cp793Q38h376vcCXgUcDU8D9ffu/Ba7sz8kiumEDFgNvAv5w1vl69OxzuJXzW8Ap/fRWa9z4/Le2Tl/fnbP2+xngef30wf3jvsCtwONmHfu3++k/B/6onz4PeOlW6j0PuJzu4u8IunGD9qEbDnmffp0jgDX99BbnpP93vnLWPg/sHx83q+3twGv76UuB0/rpV7Hpd/KFdN/STV/PpcDzJ/1/5hf5ZyFqxb5JbqK70r+DLvCgC/ZVSY6gC5C9Zm2zuqq+A5DkduCJwDf7dVYDr6mqz89zrOcBF1XVD/ptLwR+DbhxB2u+rKp+DPw4yQa6cD6eLnC+lAS6ENzawFWX9I+3AAdUd0+E7yX5cbr3OJ4HnF9VD9MNhvV54NnAl4CPpBtg72+r6qYRan2YbjA+Rqxx3nWqaqZ/hXM0sBY4EviHfpvXJXlJP/14umD+FvATurAEuB44cYR6AT5RVT8D1ia5qz/W14EPJFnaP6d/0a+7xTnpt3lSkr8ALgM2DqP8tCRvBw6k+8N/Rd9+DJvGk//fdMMwQxf8L2TT78cB/XP7wojPQzvI4G/HP1fV0v5l9xV0ffz/A/gvwFVV9ZJ09wy4etY2P541/TCbfl8eoguYfwPMF/y7ynzHD7Cqqt62A9v/bM6+fsY2fver6gtJnk93w5jzkrynqv56O8f6Uf8HhBFr3NY6HwNOAb5C9we0kryA7hXZMVX1wyRX012hA/y0+ktnNv932p6547UU8Abgfrq7gT0K+BFs/ZwkeTrd78Gr+pp/j+7VxLKqujnJ6cALtlNHgHdU1YdHrFuPkH38jamqH9Ld4u5N6e509Fjg3n7x6aPuhu4/+JFJ3jrP8r8DlqUbaXF/4CV927Z8j677YHtWAy9N8svw83uSPnHEuuer82V9X/UU3d2wruv3d39V/SXdncI2DpX80/6Kd1fUuK11LqIbm/40uj8C0P07PdCH/pF0t/Lcnu2d099J8qi+//9JwFf746zvXwm8gq5bh/nOSbpPJz2qqj4F/BGbztOjgfX9uZr9Jv4X6brXYPPbKl4B/F66+1aQ5NCN50XDMPgbVFU30vV5n0bXJ/yOJDeyA68A+6vb0+hGHn31nGU30F31XUf3PsA5/TG35Sq6N3Nnv7k733FvpwuZz6Yb0fJKun75nXER3Xm4Gfgc8Jbqhkp+AXBzf05eRndfXOj6ob8835u7O1rjttapqgfouuOeWFXX9ZtcDixMcgfdCI1fHOH5fQx4c7o37p88z/Jv0P0bfQZ4VVX9iO69huXp3qQ+ku7mMDD/OTkUuLrvQvwbYOOrlz+m+3f/B7pXLRudCbyxf75PAb7TP9/P0nX9XJPkFrpROke5CNBOcnROqUFJzgMuraoLxnjM/ei6HCvJqXRv9J48ruNrE/v4JY3Ls+jeOA7wIF13oSbAK35Jaox9/JLUGINfkhpj8EtSYwx+SWqMwS9Jjfn/gHsmtoXB9LUAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "124 / 124 questions predicted.------------------------------------------------------------ \n",
            "\n",
            "Theme: Separation_of_powers_under_the_United_States_Constitution\n",
            "Total Questions: 161\n",
            "Total Paragraphs: 29\n",
            "Total queries: 161\n",
            "In top 10 results, number of queries for which -\n",
            "\tRelevant paragraph found: 152 (94 %)\n",
            "\tRelevant paragraph NOT found: 9 (6 %)\n",
            "Mean Rank for which relevant paragraph found: 2.05\n",
            "\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAUoUlEQVR4nO3dfbRldX3f8feHGQhP6oDcziIDy0GlEpZdghktD8ZlRVMqiYwt4WEZHRIaFpWKgFUxSZdpaytG60NioyFgGFcohgAGhAjSESRNEBieH0YLRR6GDsyNCj5FEfj2j71nz5nrnZkD3HP2Hc77tdZdZ5/f3vvs79n33vM5v73P+e1UFZIkAWzXdwGSpPnDUJAkdQwFSVLHUJAkdQwFSVJnYd8FPBd77LFHLV26tO8yJGmbctNNN/1DVU3NNm+bDoWlS5eyevXqvsuQpG1Kkgc2N8/DR5KkzshCIcnnk6xPcudA2+5JrkpyT3u7W9ueJH+U5N4ktyd59ajqkiRt3ih7CucCh89oOwNYVVX7Aqva+wD/Cti3/TkR+OwI65IkbcbIQqGqrgW+O6P5SGBlO70SWD7Q/oVqfANYlGTPUdUmSZrduM8pLK6qde30I8DidnoJ8NDAcmvbtp+T5MQkq5Osnp6eHl2lkjSBejvRXM1IfM94NL6qOquqllXVsqmpWT9RJUl6lsYdCo9uOCzU3q5v2x8G9h5Ybq+2TZI0RuMOhUuBFe30CuCSgfZ3tp9COgh4fOAwkyRpTEb25bUk5wNvAPZIshb4EHAmcEGSE4AHgKPbxf8GeAtwL/Bj4LdGVZckafNGFgpVddxmZh02y7IFnDyqWmaz9IzLx7m5Tdx/5hG9bVuStsRvNEuSOoaCJKljKEiSOoaCJKljKEiSOoaCJKljKEiSOoaCJKljKEiSOoaCJKljKEiSOoaCJKljKEiSOoaCJKljKEiSOoaCJKljKEiSOoaCJKljKEiSOoaCJKljKEiSOoaCJKljKEiSOoaCJKljKEiSOoaCJKljKEiSOoaCJKljKEiSOoaCJKljKEiSOoaCJKljKEiSOoaCJKnTSygkOS3JXUnuTHJ+kh2T7JPk+iT3JvnLJDv0UZskTbKxh0KSJcApwLKqeiWwADgW+Cjwyap6OfA94IRx1yZJk66vw0cLgZ2SLAR2BtYBbwQubOevBJb3VJskTayxh0JVPQx8HHiQJgweB24CHquqJ9vF1gJLZls/yYlJVidZPT09PY6SJWli9HH4aDfgSGAf4BeBXYDDh12/qs6qqmVVtWxqampEVUrSZOrj8NGbgG9X1XRV/Qy4GDgUWNQeTgLYC3i4h9okaaL1EQoPAgcl2TlJgMOAu4GrgaPaZVYAl/RQmyRNtD7OKVxPc0L5ZuCOtoazgA8Apye5F3gxcM64a5OkSbdw64vMvar6EPChGc33Aa/toRxJUstvNEuSOoaCJKljKEiSOoaCJKljKEiSOoaCJKljKEiSOoaCJKljKEiSOoaCJKljKEiSOoaCJKljKEiSOoaCJKljKEiSOoaCJKljKEiSOoaCJKljKEiSOoaCJKljKEiSOoaCJKljKEiSOoaCJKljKEiSOoaCJKljKEiSOoaCJKljKEiSOoaCJKljKEiSOoaCJKljKEiSOr2EQpJFSS5M8s0ka5IcnGT3JFcluae93a2P2iRpkvXVU/g0cEVV7Qe8ClgDnAGsqqp9gVXtfUnSGI09FJK8CHg9cA5AVT1RVY8BRwIr28VWAsvHXZskTbo+egr7ANPAnye5JcnZSXYBFlfVunaZR4DFs62c5MQkq5Osnp6eHlPJkjQZ+giFhcCrgc9W1YHAj5hxqKiqCqjZVq6qs6pqWVUtm5qaGnmxkjRJ+giFtcDaqrq+vX8hTUg8mmRPgPZ2fQ+1SdJEG3soVNUjwENJXtE2HQbcDVwKrGjbVgCXjLs2SZp0C3va7ruB85LsANwH/BZNQF2Q5ATgAeDonmqTpInVSyhU1a3AsllmHTbuWiRJGw11+CjJocO0SZK2bcOeU/jjIdskSduwLR4+SnIwcAgwleT0gVkvBBaMsjBJ0vht7ZzCDsCu7XIvGGj/PnDUqIqSJPVji6FQVV8Hvp7k3Kp6YEw1SZJ6Muynj34hyVnA0sF1quqNoyhKktSPYUPhr4DPAWcDT42uHElSn4YNhSer6rMjrUSS1LthP5L65STvSrJnezGc3ZPsPtLKJEljN2xPYcOYRO8baCvgpXNbjiSpT0OFQlXtM+pCJEn9GyoUkrxztvaq+sLcliNJ6tOwh49eMzC9I83AdTcDhoIkPY8Me/jo3YP3kywCvjiSiiRJvXm2F9n5Ec21liVJzyPDnlP4MhuvmbwA+CXgglEVJUnqx7DnFD4+MP0k8EBVrR1BPZKkHg11+KgdGO+bNCOl7gY8McqiJEn9GPbKa0cDNwC/QXPt5OuTOHS2JD3PDHv46PeA11TVeoAkU8D/Ai4cVWGSpPEb9tNH220IhNZ3nsG6kqRtxLA9hSuSXAmc394/Bvib0ZQkSerL1q7R/HJgcVW9L8m/Bl7XzroOOG/UxUmSxmtrPYVPAR8EqKqLgYsBkvyzdt6vj7Q6SdJYbe28wOKqumNmY9u2dCQVSZJ6s7VQWLSFeTvNZSGSpP5tLRRWJ/mdmY1J/i1w02hKkiT1ZWvnFE4FvpTk7WwMgWXADsDbRlmYJGn8thgKVfUocEiSfwG8sm2+vKq+NvLKJEljN+z1FK4Grh5xLZKknvmtZElSx1CQJHUMBUlSp7dQSLIgyS1JLmvv75Pk+iT3JvnLJDv0VZskTao+ewrvAdYM3P8o8MmqejnwPeCEXqqSpAnWSygk2Qs4Aji7vR/gjWy8PsNKYHkftUnSJOurp/Ap4P3A0+39FwOPVdWT7f21wJLZVkxyYpLVSVZPT0+PvlJJmiBjD4Ukvwasr6pnNUxGVZ1VVcuqatnU1NQcVydJk23Yi+zMpUOBtyZ5C7Aj8ELg08CiJAvb3sJewMM91CZJE23sPYWq+mBV7VVVS4Fjga9V1dtpvjF9VLvYCuCScdcmSZNuPn1P4QPA6UnupTnHcE7P9UjSxOnj8FGnqq4Brmmn7wNe22c9kjTp5lNPQZLUM0NBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJnbGHQpK9k1yd5O4kdyV5T9u+e5KrktzT3u427tokadL10VN4EnhvVe0PHAScnGR/4AxgVVXtC6xq70uSxmjsoVBV66rq5nb6B8AaYAlwJLCyXWwlsHzctUnSpOv1nEKSpcCBwPXA4qpa1856BFjcU1mSNLF6C4UkuwIXAadW1fcH51VVAbWZ9U5MsjrJ6unp6TFUKkmTo5dQSLI9TSCcV1UXt82PJtmznb8nsH62davqrKpaVlXLpqamxlOwJE2IhePeYJIA5wBrquoTA7MuBVYAZ7a3l4y7tnFZesblvWz3/jOP6GW7krYdYw8F4FDgHcAdSW5t236XJgwuSHIC8ABwdA+1SdJEG3soVNX/BrKZ2YeNsxZJ0qb8RrMkqWMoSJI6hoIkqWMoSJI6hoIkqWMoSJI6hoIkqWMoSJI6hoIkqWMoSJI6hoIkqWMoSJI6hoIkqWMoSJI6fVxPQT3p6+I+4AV+pG2FPQVJUsdQkCR1DAVJUsdQkCR1DAVJUsdPH2ks+vrkk596kp4ZewqSpI6hIEnqGAqSpI6hIEnqGAqSpI6fPtLzmp96kp4ZewqSpI49Bel5xt6Rngt7CpKkjqEgSeoYCpKkjqEgSep4olkagT4vfdqXSbzc6/PxOc+rnkKSw5N8K8m9Sc7oux5JmjTzpqeQZAHwP4A3A2uBG5NcWlV391uZpPluEntmozKfegqvBe6tqvuq6gngi8CRPdckSRNl3vQUgCXAQwP31wL/fOZCSU4ETmzv/jDJt8ZQ2yjtAfxD30XMI+6PjdwXm3J/DMhHn9P+eMnmZsynUBhKVZ0FnNV3HXMlyeqqWtZ3HfOF+2Mj98Wm3B+bGtX+mE+Hjx4G9h64v1fbJkkak/kUCjcC+ybZJ8kOwLHApT3XJEkTZd4cPqqqJ5P8e+BKYAHw+aq6q+eyxuF5cyhsjrg/NnJfbMr9samR7I9U1SgeV5K0DZpPh48kST0zFCRJHUOhJ0n2TnJ1kruT3JXkPX3X1LckC5LckuSyvmvpW5JFSS5M8s0ka5Ic3HdNfUpyWvt/cmeS85Ps2HdN45Lk80nWJ7lzoG33JFcluae93W2utmco9OdJ4L1VtT9wEHBykv17rqlv7wHW9F3EPPFp4Iqq2g94FRO8X5IsAU4BllXVK2k+iHJsv1WN1bnA4TPazgBWVdW+wKr2/pwwFHpSVeuq6uZ2+gc0//RL+q2qP0n2Ao4Azu67lr4leRHweuAcgKp6oqoe67eq3i0EdkqyENgZ+H891zM2VXUt8N0ZzUcCK9vplcDyudqeoTAPJFkKHAhc328lvfoU8H7g6b4LmQf2AaaBP28Pp52dZJe+i+pLVT0MfBx4EFgHPF5VX+23qt4trqp17fQjwOK5emBDoWdJdgUuAk6tqu/3XU8fkvwasL6qbuq7lnliIfBq4LNVdSDwI+bw8MC2pj1efiRNWP4isEuS3+y3qvmjmu8VzNl3CwyFHiXZniYQzquqi/uup0eHAm9Ncj/N6LhvTPIX/ZbUq7XA2qra0HO8kCYkJtWbgG9X1XRV/Qy4GDik55r69miSPQHa2/Vz9cCGQk+ShOaY8Zqq+kTf9fSpqj5YVXtV1VKaE4hfq6qJfSdYVY8ADyV5Rdt0GDDJ1xV5EDgoyc7t/81hTPCJ99alwIp2egVwyVw9sKHQn0OBd9C8K761/XlL30Vp3ng3cF6S24EDgP/Wcz29aXtMFwI3A3fQvG5NzJAXSc4HrgNekWRtkhOAM4E3J7mHpid15pxtz2EuJEkb2FOQJHUMBUlSx1CQJHUMBUlSx1CQJHUMhQmX5Kn247B3JvlykkXP4bF++BzWPaUdDfS8Ge0HDH5UN8kfJPkPz3Y7o5Dkd+fgMe5Pssdc1DPEtpY7+KI2x1DQP1bVAe3ok98FTu6pjncBb66qt89oPwCY79/fGCoUkiwYdSFDWg4YCpqVoaBB19GO1JrktUmuawdk+/sN365NcnySi5Nc0Y7l/oczHyTJHu26R8wy7/S2V3JnklPbts8BLwW+kuS0gWV3AP4zcEzbmzmmnbV/kmuS3JfklIHlfzPJDe2yfzrbi3D7jvwj7TKrk7w6yZVJ/m+Sk9plkuRjbY13bNhukj2TXDvQs/qVJGfSjN5568xeTrvOD5P89yS3AQcPWePPLZPkpCQfG1jm+CSfaaf/OslNaa43cOKMbf/XJLcl+UaSxUkOAd4KfKx9/JfN2Pa5ST7X7pv/045LRZKlSf42yc3tzyFb2CcL2sfZsP9Oa5f9nSQ3tvVclGTntv1lbX13JPnwYI8zyfvadW5P8p9m7iuNQFX5M8E/wA/b2wXAXwGHt/dfCCxsp98EXNROHw/cB7wI2BF4ANh7w2PRjNZ4Pc27/pnb+mWab6TuAuwK3AUc2M67H9hjlnWOBz4zcP8PgL8HfgHYA/gOsD3wS8CXge3b5f4EeOcsj3c/8O/a6U8CtwMvAKaAR9v2fwNc1e6TxTTDLOwJvBf4vYH99YLBfbiZ/VvA0e30Zmvc8Pw3t0xb370Dj/sV4HXt9O7t7U7AncCLB7b96+30HwK/306fCxy1mXrPBa6gecO4L804TDvSDFe9Y7vMvsDqdvrn9kn7e75q4DEXtbcvHmj7MPDudvoy4Lh2+iQ2/k3+Ks03l9PWcxnw+r7/Z57vPwvRpNspya00PYQ1NC+G0Lzor0yyL82Ly/YD66yqqscBktwNvAR4qF1mFXByVX19lm29DvhSVf2oXfdi4FeAW55hzZdX1U+BnyZZT/PCfRjNi9GNSaB5gdzcIGGXtrd3ALtWcz2LHyT5aZpzKq8Dzq+qp2gGHvs68BrgRuDzaQYy/OuqunWIWp+iGfSQIWucdZmqmm57RgcB9wD7AX/XrnNKkre103vTvGh/B3iC5oUU4CbgzUPUC3BBVT0N3JPkvnZb3wY+k+SA9jn903bZn9sn7TovTfLHwOXAhmGuX5nkw8AimjcFV7btB7PxegD/k2aYbGhC4VfZ+Pexa/vcrh3yeehZMBT0j1V1QNuVv5LmnMIfAf8FuLqq3pbmeg/XDKzz04Hpp9j4d/QkzYvPvwRmC4W5Mtv2A6ysqg8+g/WfnvFYT7OF/4mqujbJ62kuBnRukk9U1Re2sq2ftOHCkDVuaZkvAkcD36QJ10ryBpqe3MFV9eMk19C8swf4WbVvudn097Q1M8e+KeA04FGaq8BtB/wENr9PkryK5u/gpLbm36bphSyvqtuSHA+8YSt1BPhIVf3pkHVrDnhOQQBU1Y9pLnn43jRXt3oR8HA7+/hhH4bmn3+/JB+YZf7fAsvTjHa5C/C2tm1LfkBzSGJrVgFHJfkn0F3D9iVD1j1bnce0x8anaK6CdkP7eI9W1Z/RXCFuw3DWP2vfKc9FjVta5ks01xU4jiYgoPk9fa8NhP1oLu26NVvbp7+RZLv2fMNLgW+121nX9iDeQXOoiNn2SZpPUW1XVRcBv8/G/fQCYF27rwY/UPANmkN2sOllNq8EfjvNNUdIsmTDftHoGArqVNUtNMfYj6M5Bv2RJLfwDHqU7bvi42hGf33XjHk307xbvIHmvMPZ7Ta35GqaE8uDJ5pn2+7dNC9AX00zsuhVNOcBno0v0eyH24CvAe+vZjjrNwC3tfvkGJrrKENz3Pv22U40P9Mat7RMVX2P5hDfS6rqhnaVK4CFSdbQjJT5jSGe3xeB96X5EMHLZpn/IM3v6CvASVX1E5pzGyvSnDDfj+bCPzD7PlkCXNMelvwLYEOv5z/S/N7/jqa3s8GpwOnt83058Hj7fL9KczjpuiR30IyUOswbBD0HjpIqqZPkXOCyqrpwjNvcmeYwZiU5luak85Hj2r425TkFSX37ZZqT2AEeozkEqZ7YU5AkdTynIEnqGAqSpI6hIEnqGAqSpI6hIEnq/H8OsWMKWyDNkwAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "161 / 161 questions predicted.------------------------------------------------------------ \n",
            "\n",
            "Theme: Architecture\n",
            "Total Questions: 239\n",
            "Total Paragraphs: 38\n",
            "Total queries: 239\n",
            "In top 10 results, number of queries for which -\n",
            "\tRelevant paragraph found: 224 (94 %)\n",
            "\tRelevant paragraph NOT found: 15 (6 %)\n",
            "Mean Rank for which relevant paragraph found: 2.13\n",
            "\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAWnklEQVR4nO3de7RedX3n8fdHIiKiRsxpFpOgQc1IGWdUGhkQ63JELV5qcMZyWVaD0mYxMt5HC9oZOzM6YnW81ak2BUpcZaAUoaAoyESQTsvFAHIRtDDIJUwgpype6yX4nT/2zs7D4SR5kpzn2Sc579daZz17//bte3Zyns/z2/vZe6eqkCQJ4FF9FyBJmj0MBUlSx1CQJHUMBUlSx1CQJHXm9V3AzliwYEEtWbKk7zIkaZdy3XXX/WNVTUw3bZcOhSVLlrB27dq+y5CkXUqSu7c0zcNHkqSOoSBJ6hgKkqSOoSBJ6hgKkqSOoSBJ6hgKkqSOoSBJ6hgKkqTOLn1F885YcvLFvW37rlNf2du2JWlr7ClIkjqGgiSpYyhIkjqGgiSpYyhIkjqGgiSpYyhIkjojC4UkZyTZkOSWaaa9K0klWdCOJ8mnktyR5KYkB4+qLknSlo2yp3AmcOTUxiT7Ay8D7hlofjmwtP1ZCXxmhHVJkrZgZKFQVVcC35tm0seB9wA10LYc+Fw1rgbmJ9lvVLVJkqY31nMKSZYD91XVjVMmLQLuHRhf17ZNt46VSdYmWTs5OTmiSiVpbhpbKCTZG3gv8J93Zj1VtaqqllXVsomJiZkpTpIEjPeGeE8HDgBuTAKwGLg+ySHAfcD+A/MubtskSWM0tp5CVd1cVb9WVUuqagnNIaKDq+p+4CLgDe23kA4FflBV68dVmySpMcqvpJ4NXAU8M8m6JCdsZfYvAXcCdwB/Drx5VHVJkrZsZIePquq4bUxfMjBcwEmjqkWSNByvaJYkdQwFSVLHUJAkdQwFSVLHUJAkdQwFSVLHUJAkdQwFSVLHUJAkdQwFSVLHUJAkdQwFSVLHUJAkdQwFSVLHUJAkdQwFSVLHUJAkdQwFSVLHUJAkdUYWCknOSLIhyS0DbR9J8q0kNyW5IMn8gWmnJLkjybeT/Nao6pIkbdkoewpnAkdOabsMeFZV/SvgH4BTAJIcBBwL/It2mT9NsscIa5MkTWNkoVBVVwLfm9L2lara2I5eDSxuh5cD51TVz6vqO8AdwCGjqk2SNL0+zym8CfhyO7wIuHdg2rq27RGSrEyyNsnaycnJEZcoSXNLL6GQ5H3ARuCs7V22qlZV1bKqWjYxMTHzxUnSHDZv3BtMcjzwKuCIqqq2+T5g/4HZFrdtkqQxGmtPIcmRwHuAV1fVTwcmXQQcm+QxSQ4AlgLXjrM2SdIIewpJzgZeBCxIsg54P823jR4DXJYE4OqqOrGqvpnkXOBWmsNKJ1XVQ6OqTZI0vZGFQlUdN03z6VuZ/4PAB0dVjyRp27yiWZLUMRQkSR1DQZLUMRQkSR1DQZLUMRQkSR1DQZLUMRQkSR1DQZLUMRQkSR1DQZLUMRQkSR1DQZLUMRQkSR1DQZLUMRQkSR1DQZLUMRQkSR1DQZLUGVkoJDkjyYYktwy07ZvksiS3t69PatuT5FNJ7khyU5KDR1WXJGnLRtlTOBM4ckrbycCaqloKrGnHAV4OLG1/VgKfGWFdkqQtGFkoVNWVwPemNC8HVrfDq4GjBto/V42rgflJ9htVbZKk6Y37nMLCqlrfDt8PLGyHFwH3Dsy3rm17hCQrk6xNsnZycnJ0lUrSHNTbieaqKqB2YLlVVbWsqpZNTEyMoDJJmrvGHQoPbDos1L5uaNvvA/YfmG9x2yZJGqNxh8JFwIp2eAVw4UD7G9pvIR0K/GDgMJMkaUzmjWrFSc4GXgQsSLIOeD9wKnBukhOAu4Gj29m/BLwCuAP4KfDGUdUlSdqykYVCVR23hUlHTDNvASeNqhZJ0nC8olmS1DEUJEkdQ0GS1DEUJEkdQ0GS1DEUJEkdQ0GS1DEUJEkdQ0GS1DEUJEkdQ0GS1DEUJEkdQ0GS1BkqFJIcPkybJGnXNmxP4U+GbJMk7cK2+jyFJIcBzwcmkrxzYNITgD1GWZgkafy29ZCdPYF92vkeP9D+Q+C1oypKktSPrYZCVX0N+FqSM6vq7jHVJEnqybCP43xMklXAksFlqurFoyhKktSPYUPhr4HPAqcBD+3sRpO8A/g9oICbgTcC+wHnAE8GrgNeX1W/2NltSZKGN+y3jzZW1Weq6tqqum7Tz45sMMki4K3Asqp6Fs0J62OBDwMfr6pnAN8HTtiR9UuSdtywofCFJG9Osl+SfTf97MR25wGPTTIP2BtYD7wYOK+dvho4aifWL0naAcMePlrRvr57oK2Ap23vBqvqviQfBe4B/gn4Cs3hogeramM72zpg0XTLJ1kJrAR4ylOesr2blyRtxVChUFUHzNQGkzwJWA4cADxIc77iyGGXr6pVwCqAZcuW1UzVJUkaMhSSvGG69qr63A5s8yXAd6pqsl33+cDhwPwk89rewmLgvh1YtyRpJwx7+Oh5A8N7AUcA1wM7Egr3AIcm2Zvm8NERwFrgcpoL4s6hOVx14Q6sW5K0E4Y9fPSWwfEk82nevLdbVV2T5DyaUNkI3EBzOOhi4JwkH2jbTt+R9UuSdtywPYWpfkJzTmCHVNX7gfdPab4TOGRH1ylJ2nnDnlP4As23jaC5ruDXgXNHVZQkqR/D9hQ+OjC8Ebi7qtaNoB5JUo+GunitvTHet2julPokwNtPSNJuaNgnrx0NXAv8DnA0cE0Sb50tSbuZYQ8fvQ94XlVtAEgyAfxvNt+WQpK0Gxj23keP2hQIre9ux7KSpF3EsD2FS5JcCpzdjh8DfGk0JUmS+rKtZzQ/A1hYVe9O8m+BF7STrgLOGnVxkqTx2lZP4RPAKQBVdT5wPkCSf9lO++2RVidJGqttnRdYWFU3T21s25aMpCJJUm+2FQrztzLtsTNZiCSpf9sKhbVJfn9qY5Lfo3kwjiRpN7KtcwpvBy5I8jo2h8AyYE/gNaMsTJI0flsNhap6AHh+kn8DPKttvriqvjryyiRJYzfs8xQup3kIjiRpN+ZVyZKkjqEgSeoYCpKkjqEgSer0EgpJ5ic5L8m3ktyW5LAk+ya5LMnt7euT+qhNkuayvnoKnwQuqaoDgWcDtwEnA2uqaimwph2XJI3R2EMhyROBFwKnA1TVL6rqQWA5sLqdbTVw1Lhrk6S5ro+ewgHAJPAXSW5IclqSx9HcfG99O8/9wMLpFk6yMsnaJGsnJyfHVLIkzQ19hMI84GDgM1X1XOAnTDlUVFUF1HQLV9WqqlpWVcsmJiZGXqwkzSV9hMI6YF1VXdOOn0cTEg8k2Q+gfd2wheUlSSMy9lCoqvuBe5M8s206ArgVuAhY0batAC4cd22SNNcN+4zmmfYW4KwkewJ3Am+kCahzk5wA3A0c3VNtkjRn9RIKVfUNmltwT3XEuGuRJG3mFc2SpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpE5voZBkjyQ3JPliO35AkmuS3JHkr5Ls2VdtkjRX9dlTeBtw28D4h4GPV9UzgO8DJ/RSlSTNYb2EQpLFwCuB09rxAC8GzmtnWQ0c1UdtkjSX9dVT+ATwHuBX7fiTgQeramM7vg5YNN2CSVYmWZtk7eTk5OgrlaQ5ZOyhkORVwIaqum5Hlq+qVVW1rKqWTUxMzHB1kjS3zethm4cDr07yCmAv4AnAJ4H5Sea1vYXFwH091CZJc9rYewpVdUpVLa6qJcCxwFer6nXA5cBr29lWABeOuzZJmutm03UKfwC8M8kdNOcYTu+5Hkmac/o4fNSpqiuAK9rhO4FD+qxHkua62dRTkCT1zFCQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSx1CQJHUMBUlSZ+zPaE6yP/A5YCFQwKqq+mSSfYG/ApYAdwFHV9X3x13fOCw5+eJetnvXqa/sZbuSdh199BQ2Au+qqoOAQ4GTkhwEnAysqaqlwJp2XJI0RmMPhapaX1XXt8M/Am4DFgHLgdXtbKuBo8ZdmyTNdb2eU0iyBHgucA2wsKrWt5Pupzm8NN0yK5OsTbJ2cnJyLHVK0lzRWygk2Qf4PPD2qvrh4LSqKprzDY9QVauqallVLZuYmBhDpZI0d/QSCkkeTRMIZ1XV+W3zA0n2a6fvB2zoozZJmsvGHgpJApwO3FZVHxuYdBGwoh1eAVw47tokaa4b+1dSgcOB1wM3J/lG2/Ze4FTg3CQnAHcDR/dQmyTNaWMPhar6P0C2MPmIcdYiSXo4r2iWJHX6OHyknvR1JTV4NbW0q7CnIEnqGAqSpI6hIEnqGAqSpI4nmrVb8+S6tH3sKUiSOoaCJKljKEiSOoaCJKnjiWaNRZ8nfCUNz56CJKljKEiSOoaCJKnjOQVpRPo6j9LXRXNeKLh7MBSk3cxcPKk/1wJ4lDx8JEnqGAqSpM6sO3yU5Ejgk8AewGlVdWrPJUnStHbH8yizqqeQZA/gfwIvBw4CjktyUL9VSdLcMatCATgEuKOq7qyqXwDnAMt7rkmS5ozZdvhoEXDvwPg64F8PzpBkJbCyHf1xkm+PqbZRWQD8Y99FzCLuj4dzf2zmvhiQD+/U/njqlibMtlDYpqpaBazqu46ZkmRtVS3ru47Zwv3xcO6PzdwXDzeq/THbDh/dB+w/ML64bZMkjcFsC4WvA0uTHJBkT+BY4KKea5KkOWNWHT6qqo1J/gNwKc1XUs+oqm/2XNao7TaHwmaI++Ph3B+buS8ebiT7I1U1ivVKknZBs+3wkSSpR4aCJKljKPQkyf5JLk9ya5JvJnlb3zX1LckeSW5I8sW+a+lbkvlJzkvyrSS3JTms75r6lOQd7d/JLUnOTrJX3zWNU5IzkmxIcstA275JLktye/v6pJnYlqHQn43Au6rqIOBQ4CRv6cHbgNv6LmKW+CRwSVUdCDybObxfkiwC3gosq6pn0XwJ5dh+qxq7M4Ejp7SdDKypqqXAmnZ8pxkKPamq9VV1fTv8I5o/+kX9VtWfJIuBVwKn9V1L35I8EXghcDpAVf2iqh7st6rezQMem2QesDfw/3quZ6yq6krge1OalwOr2+HVwFEzsS1DYRZIsgR4LnBNv5X06hPAe4Bf9V3ILHAAMAn8RXs47bQkj+u7qL5U1X3AR4F7gPXAD6rqK/1WNSssrKr17fD9wMKZWKmh0LMk+wCfB95eVT/su54+JHkVsKGqruu7llliHnAw8Jmqei7wE2bo0MCuqD1WvpwmLP8Z8Lgkv9tvVbNLNdcWzMj1BYZCj5I8miYQzqqq8/uup0eHA69OchfNnXFfnOQv+y2pV+uAdVW1qed4Hk1IzFUvAb5TVZNV9UvgfOD5Pdc0GzyQZD+A9nXDTKzUUOhJktAcM76tqj7Wdz19qqpTqmpxVS2hOYH41aqas58Eq+p+4N4kz2ybjgBu7bGkvt0DHJpk7/bv5gjm8In3ARcBK9rhFcCFM7FSQ6E/hwOvp/lU/I325xV9F6VZ4y3AWUluAp4D/Pee6+lN22M6D7geuJnmfWtO3fIiydnAVcAzk6xLcgJwKvDSJLfT9KZm5CmV3uZCktSxpyBJ6hgKkqSOoSBJ6hgKkqSOoSBJ6hgKIslD7Vdib0nyhSTzd2JdP96JZd/a3hH0rCntzxn8um6SP0ryH3d0O6OQ5L0zsI67kiyYiXqG2NZR3oBR0zEUBPBPVfWc9g6U3wNO6qmONwMvrarXTWl/DjDbr+EYKhSS7DHqQoZ0FGAo6BEMBU11Fe3dWpMckuSq9qZsf7/pCtskxyc5P8kl7b3c/3jqSpIsaJd95TTT3tn2Sm5J8va27bPA04AvJ3nHwLx7Av8VOKbtzRzTTjooyRVJ7kzy1oH5fzfJte28fzbdm3D7ifxD7Txrkxyc5NIk/zfJie08SfKRtsabN203yX5JrhzoWf1mklNp7uD5jam9nHaZHyf5H0luBA4bssZHzJPkxCQfGZjn+CSfbof/Jsl1aZ45sHLKtj+Y5MYkVydZmOT5wKuBj7Trf/qUbZ+Z5LPtvvmH9t5UJFmS5G+TXN/+PH8r+2SPdj2b9t872nl/P8nX23o+n2Tvtv3pbX03J/nAYI8zybvbZW5K8l+m7ivNsKryZ47/AD9uX/cA/ho4sh1/AjCvHX4J8Pl2+HjgTuCJwF7A3cD+m9ZFc7fGa2g+9U/d1m/QXJX6OGAf4JvAc9tpdwELplnmeODTA+N/BPw98BhgAfBd4NHArwNfAB7dzvenwBumWd9dwL9vhz8O3AQ8HpgAHmjb/x1wWbtPFtLcamE/4F3A+wb21+MH9+EW9m8BR7fDW6xx0++/pXna+u4YWO+XgRe0w/u2r48FbgGePLDt326H/xj4w3b4TOC1W6j3TOASmg+NS2nuxbQXzS2r92rnWQqsbYcfsU/af+fLBtY5v3198kDbB4C3tMNfBI5rh09k8//Jl9FcvZy2ni8CL+z7b2Z3/pmH1H7Kpekh3EbzZgjNm/7qJEtp3lwePbDMmqr6AUCSW4GnAve286wBTqqqr02zrRcAF1TVT9plzwd+E7hhO2u+uKp+Dvw8yQaaN+4jaN6Mvp4EmjfILd0k7KL29WZgn2qeafGjJD9Pc07lBcDZVfUQzY3HvgY8D/g6cEaamxn+TVV9Y4haH6K58SFD1jjtPFU12faMDgVuBw4E/q5d5q1JXtMO70/zpv1d4Bc0b6QA1wEvHaJegHOr6lfA7UnubLf1HeDTSZ7T/k7/vJ33EfukXeZpSf4EuBjYdKvrZyX5ADCf5kPBpW37YWx+HsD/orlVNjSh8DI2///Yp/3drhzy99B2MhQE7TmFtit/Kc05hU8B/w24vKpek+aZD1cMLPPzgeGH2Px/aSPNm89vAdOFwkyZbvsBVlfVKdux/K+mrOtXbOXvoqquTPJCmgcCnZnkY1X1uW1s62dtuDBkjVub5xzgaOBbNOFaSV5E05M7rKp+muQKmk/2AL+s9iM3D/932pap978p4B3AAzRPgnsU8DPY8j5J8mya/wcntjW/iaYXclRV3ZjkeOBF26gjwIeq6s+GrFs7yXMK6lTVT2kee/iuNE+4eiJwXzv5+GFXQ/PHf2CSP5hm+t8CR6W54+XjgNe0bVvzI5pDEtuyBnhtkl+D7hm2Tx2y7unqPKY9Nj5B8yS0a9v1PVBVf07zlLhNt7T+ZftJeSZq3No8F9A8W+A4moCA5t/p+20gHEjzeNdt2dY+/Z0kj2rPNzwN+Ha7nfVtD+L1NIeKmG6fpPkW1aOq6vPAH7J5Pz0eWN/uq8EvFFxNc8gOHv6ozUuBN6V57ghJFm3aLxoNQ0EPU1U30BxjP47mGPSHktzAdvQq20/Fx9HcAfbNU6ZdT/Np8Vqa8w6ntdvcmstpTiwPnmiebru30rwBfSXN3UUvozkPsCMuoNkPNwJfBd5TzS2tXwTc2O6TY2iepQzNce+bpjvRvL01bm2eqvo+zSG+p1bVte0ilwDzktxGc6fMq4f4/c4B3p3mSwRPn2b6PTT/Rl8GTqyqn9Gc21iR5oT5gTQP/4Hp98ki4Ir2sORfApt6Pf+J5t/972h6O5u8HXhn+/s+A/hB+/t+heZw0lVJbqa5W+owHxC0g7xLqqSHSXIm8MWqOm+M29yb5jBmJTmW5qTz8nFtX5t5TkHSbPAbNCexAzxIcwhSPbCnIEnqeE5BktQxFCRJHUNBktQxFCRJHUNBktT5/9EON7NmLNRpAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "239 / 239 questions predicted.------------------------------------------------------------ \n",
            "\n",
            "Theme: Alexander_Graham_Bell\n",
            "Total Questions: 220\n",
            "Total Paragraphs: 73\n",
            "Total queries: 220\n",
            "In top 10 results, number of queries for which -\n",
            "\tRelevant paragraph found: 185 (84 %)\n",
            "\tRelevant paragraph NOT found: 35 (16 %)\n",
            "Mean Rank for which relevant paragraph found: 1.85\n",
            "\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAVnElEQVR4nO3dfbRldX3f8fdHRkTEOCi3s8iADiqVUFsfMloQ43KJJjQawdYgLKNDpJlFpT5XhZguY6sRo/UhsdFQMIwrFEIAA2oE6QiSJgIOqDxqoAgydGAmPj9UEfz2j/2bPWcud2Yuwz1nX+a8X2vddfb57afv3TP3fM5v73N+O1WFJEkADxu6AEnS4mEoSJJ6hoIkqWcoSJJ6hoIkqbdk6AIejH322adWrFgxdBmS9JBy9dVX/1NVzcw17yEdCitWrGDdunVDlyFJDylJbt/WPE8fSZJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6D+lvND8YK0767GD7vu2UFw+2b0naHnsKkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6o0tFJJ8IsnGJNePtL0/ydeTXJvkU0mWjsw7OcktSb6R5DfGVZckadvG2VM4AzhiVtslwFOr6l8B/wicDJDkYOAY4F+0df4syW5jrE2SNIexhUJVXQ58Z1bb56vq3vb0CmC/Nn0kcHZV/ayqvgncAjx7XLVJkuY25DWF1wCfa9PLgTtG5q1vbfeTZHWSdUnWbdq0acwlStJ0GSQUkrwDuBc484GuW1WnVtXKqlo5MzOz8MVJ0hRbMukdJjkOeAlweFVVa74T2H9ksf1amyRpgibaU0hyBPA24KVV9ZORWRcCxyR5RJIDgAOBqyZZmyRpjD2FJGcBzwf2SbIeeCfdp40eAVySBOCKqjqhqm5Icg5wI91ppROr6r5x1SZJmtvYQqGqjp2j+fTtLP8e4D3jqkeStGN+o1mS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEm9sYVCkk8k2Zjk+pG2xya5JMnN7XHv1p4kf5LkliTXJnnmuOqSJG3bOHsKZwBHzGo7CVhbVQcCa9tzgH8DHNh+VgMfG2NdkqRtGFsoVNXlwHdmNR8JrGnTa4CjRto/WZ0rgKVJ9h1XbZKkuU36msKyqtrQpu8ClrXp5cAdI8utb233k2R1knVJ1m3atGl8lUrSFBrsQnNVFVA7sd6pVbWyqlbOzMyMoTJJml6TDoW7N58Wao8bW/udwP4jy+3X2iRJEzTpULgQWNWmVwEXjLS/un0K6RDg+yOnmSRJE7JkXBtOchbwfGCfJOuBdwKnAOckOR64HTi6Lf63wG8CtwA/AX53XHVJkrZtbKFQVcduY9bhcyxbwInjqkWSND9+o1mS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEm9QUIhyZuS3JDk+iRnJdkjyQFJrkxyS5K/SrL7ELVJ0jSbeCgkWQ68HlhZVU8FdgOOAd4HfKiqngx8Fzh+0rVJ0rQb6vTREuCRSZYAewIbgBcA57b5a4CjBqpNkqbWxEOhqu4EPgB8iy4Mvg9cDXyvqu5ti60Hls+1fpLVSdYlWbdp06ZJlCxJU2OI00d7A0cCBwC/DDwKOGK+61fVqVW1sqpWzszMjKlKSZpOQ5w+eiHwzaraVFU/B84HDgOWttNJAPsBdw5QmyRNtSFC4VvAIUn2TBLgcOBG4FLg5W2ZVcAFA9QmSVNtXqGQ5LD5tM1HVV1Jd0H5GuC6VsOpwNuBNye5BXgccPrObF+StPOW7HgRAP4UeOY82ualqt4JvHNW863As3dme5KkhbHdUEhyKPAcYCbJm0dm/RLd9wskSbuQHfUUdgf2ass9eqT9B2w5/y9J2kVsNxSq6ovAF5OcUVW3T6gmSdJA5ntN4RFJTgVWjK5TVS8YR1GSpGHMNxT+Gvg4cBpw3/jKkSQNab6hcG9VfWyslUiSBjffL699Oslrk+yb5LGbf8ZamSRp4ubbU1jVHt860lbAExe2HEnSkOYVClV1wLgLkSQNb16hkOTVc7VX1ScXthxJ0pDme/roWSPTe9ANYncNYChI0i5kvqePXjf6PMlS4OyxVCRJGszODp39Y7qb5EiSdiHzvabwabpPG0E3EN6vAOeMqyhJ0jDme03hAyPT9wK3V9X6MdQjSRrQvE4ftYHxvk43UurewD3jLEqSNIz53nntaOAq4LeBo4Erkzh0tiTtYuZ7+ugdwLOqaiNAkhngf9HdVlOStIuY76ePHrY5EJpvP4B1JUkPEfPtKVyU5GLgrPb8FcDfjqckSdJQdnSP5icDy6rqrUn+LfDcNutLwJnjLk6SNFk76il8GDgZoKrOB84HSPIv27zfGmt1kqSJ2tF1gWVVdd3sxta2Ymd3mmRpknOTfD3JTUkObfdouCTJze1x753dviRp5+woFJZuZ94jH8R+PwJcVFUHAU8DbgJOAtZW1YHA2vZckjRBOwqFdUl+b3Zjkn8PXL0zO0zyGOB5wOkAVXVPVX0POBJY0xZbAxy1M9uXJO28HV1TeCPwqSSvZEsIrAR2B162k/s8ANgE/EWSp7XtvoHuVNWGtsxdwLK5Vk6yGlgN8PjHP34nS5AkzWW7PYWquruqngO8C7it/byrqg6tqrt2cp9LgGcCH6uqZ9CNuLrVqaKqKrYMwDe7plOramVVrZyZmdnJEiRJc5nv/RQuBS5doH2uB9ZX1ZXt+bl0oXB3kn2rakOSfYGN29yCJGksJv6t5NbDuCPJU1rT4cCNwIXAqta2Crhg0rVJ0rSb7zeaF9rrgDOT7A7cCvwuXUCdk+R44Ha6gfckSRM0SChU1VfpLljPdvika5EkbeGgdpKknqEgSeoZCpKknqEgSeoZCpKknqEgSeoZCpKknqEgSeoZCpKknqEgSeoZCpKknqEgSeoZCpKknqEgSeoZCpKknqEgSeoZCpKknqEgSeoZCpKknqEgSeoZCpKknqEgSeoNFgpJdkvylSSfac8PSHJlkluS/FWS3YeqTZKm1ZA9hTcAN408fx/woap6MvBd4PhBqpKkKTZIKCTZD3gxcFp7HuAFwLltkTXAUUPUJknTbKiewoeBtwG/aM8fB3yvqu5tz9cDy4coTJKm2cRDIclLgI1VdfVOrr86ybok6zZt2rTA1UnSdBuip3AY8NIktwFn0502+giwNMmStsx+wJ1zrVxVp1bVyqpaOTMzM4l6JWlqTDwUqurkqtqvqlYAxwBfqKpXApcCL2+LrQIumHRtkjTtFtP3FN4OvDnJLXTXGE4fuB5JmjpLdrzI+FTVZcBlbfpW4NlD1iNJ024x9RQkSQMzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJvYmHQpL9k1ya5MYkNyR5Q2t/bJJLktzcHveedG2SNO2G6CncC7ylqg4GDgFOTHIwcBKwtqoOBNa255KkCZp4KFTVhqq6pk3/ELgJWA4cCaxpi60Bjpp0bZI07Qa9ppBkBfAM4EpgWVVtaLPuApYNVJYkTa3BQiHJXsB5wBur6gej86qqgNrGequTrEuybtOmTROoVJKmxyChkOThdIFwZlWd35rvTrJvm78vsHGudavq1KpaWVUrZ2ZmJlOwJE2JIT59FOB04Kaq+uDIrAuBVW16FXDBpGuTpGm3ZIB9Hga8CrguyVdb2+8DpwDnJDkeuB04eoDaJGmqTTwUqup/A9nG7MMnWYskaWt+o1mS1DMUJEk9Q0GS1DMUJEm9IT59NPVWnPTZQfZ72ykvHmS/kh467ClIknr2FKbIUD0UsJciPVTYU5Ak9QwFSVLPUJAk9bymoInwE1fSQ4M9BUlSz1CQJPUMBUlSz1CQJPUMBUlSz1CQJPUMBUlSz1CQJPUMBUlSz280S2My5Ki0Q/Db47sGQ0G7tGl7YZYeLE8fSZJ6i66nkOQI4CPAbsBpVXXKwCVJmgdv4rRrWFShkGQ34L8DLwLWA19OcmFV3ThsZZIWs2k8TTiuIFxsp4+eDdxSVbdW1T3A2cCRA9ckSVNjUfUUgOXAHSPP1wP/enSBJKuB1e3pj5J8Y0K1jcs+wD8NXcQi4vHYmsdjC4/FiLzvQR2PJ2xrxmILhR2qqlOBU4euY6EkWVdVK4euY7HweGzN47GFx2Jr4zoei+300Z3A/iPP92ttkqQJWGyh8GXgwCQHJNkdOAa4cOCaJGlqLKrTR1V1b5L/CFxM95HUT1TVDQOXNW67zKmwBeLx2JrHYwuPxdbGcjxSVePYriTpIWixnT6SJA3IUJAk9QyFgSTZP8mlSW5MckOSNwxd09CS7JbkK0k+M3QtQ0uyNMm5Sb6e5KYkhw5d05CSvKn9nVyf5Kwkewxd0yQl+USSjUmuH2l7bJJLktzcHvdeiH0ZCsO5F3hLVR0MHAKcmOTggWsa2huAm4YuYpH4CHBRVR0EPI0pPi5JlgOvB1ZW1VPpPoRyzLBVTdwZwBGz2k4C1lbVgcDa9vxBMxQGUlUbquqaNv1Duj/65cNWNZwk+wEvBk4bupahJXkM8DzgdICquqeqvjdsVYNbAjwyyRJgT+D/DlzPRFXV5cB3ZjUfCaxp02uAoxZiX4bCIpBkBfAM4MphKxnUh4G3Ab8YupBF4ABgE/AX7XTaaUkeNXRRQ6mqO4EPAN8CNgDfr6rPD1vVorCsqja06buAZQuxUUNhYEn2As4D3lhVPxi6niEkeQmwsaquHrqWRWIJ8EzgY1X1DODHLNCpgYeidq78SLqw/GXgUUl+Z9iqFpfqvluwIN8vMBQGlOThdIFwZlWdP3Q9AzoMeGmS2+hGxn1Bkr8ctqRBrQfWV9XmnuO5dCExrV4IfLOqNlXVz4HzgecMXNNicHeSfQHa48aF2KihMJAkoTtnfFNVfXDoeoZUVSdX1X5VtYLuAuIXqmpq3wlW1V3AHUme0poOB6b5niLfAg5Jsmf7uzmcKb7wPuJCYFWbXgVcsBAbNRSGcxjwKrp3xV9tP785dFFaNF4HnJnkWuDpwB8NXM9gWo/pXOAa4Dq6162pGvIiyVnAl4CnJFmf5HjgFOBFSW6m600tyF0qHeZCktSzpyBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6hkKIsl97SOx1yf5dJKlD2JbP3oQ676+jQh65qz2p49+XDfJHyb5Tzu7n3FI8vsLsI3bkuyzEPXMY19HOQCj5mIoCOD/VdXT2wiU3wFOHKiO1wIvqqpXzmp/OrDYv8Mxr1BIstu4C5mnowBDQfdjKGi2L9FGa03y7CRfaoOy/cPmb9gmOS7J+UkuamO5//HsjSTZp6374jnmvbn1Sq5P8sbW9nHgicDnkrxpZNndgf8CvKL1Zl7RZh2c5LIktyZ5/cjyv5Pkqrbsn8/1Itzekb+3LbMuyTOTXJzk/yQ5oS2TJO9vNV63eb9J9k1y+UjP6teSnEI3gudXZ/dy2jo/SvLfknwNOHSeNd5vmSQnJHn/yDLHJflom/6bJFenu+fA6ln7fk+SryW5IsmyJM8BXgq8v23/SbP2fUaSj7dj849tbCqSrEjyd0muaT/P2c4x2a1tZ/Pxe1Nb9veSfLnVc16SPVv7k1p91yV592iPM8lb2zrXJnnX7GOlBVZV/kz5D/Cj9rgb8NfAEe35LwFL2vQLgfPa9HHArcBjgD2A24H9N2+LbrTGK+ne9c/e16/SfSv1UcBewA3AM9q824B95ljnOOCjI8//EPgH4BHAPsC3gYcDvwJ8Gnh4W+7PgFfPsb3bgP/Qpj8EXAs8GpgB7m7t/w64pB2TZXRDLewLvAV4x8jxevToMdzG8S3g6Da9zRo3//7bWqbVd8vIdj8HPLdNP7Y9PhK4HnjcyL5/q03/MfAHbfoM4OXbqPcM4CK6N40H0o3FtAfdkNV7tGUOBNa16fsdk/bvfMnINpe2x8eNtL0beF2b/gxwbJs+gS3/J3+d7tvLafV8Bnje0H8zu/LPEqT2Lpeuh3AT3YshdC/6a5IcSPfi8vCRddZW1fcBktwIPAG4oy2zFjixqr44x76eC3yqqn7c1j0f+DXgKw+w5s9W1c+AnyXZSPfCfTjdi9GXk0D3ArmtQcIubI/XAXtVd0+LHyb5WbprKs8Fzqqq++gGHvsi8Czgy8An0g1m+DdV9dV51Hof3cCHzLPGOZepqk2tZ3QIcDNwEPD3bZ3XJ3lZm96f7kX728A9dC+kAFcDL5pHvQDnVNUvgJuT3Nr29U3go0me3n6nf96Wvd8xaes8McmfAp8FNg91/dQk7waW0r0puLi1H8qW+wH8T7qhsqELhV9ny/+Pvdrvdvk8fw89QIaCoF1TaF35i+muKfwJ8F+BS6vqZenu+XDZyDo/G5m+jy3/l+6le/H5DWCuUFgoc+0/wJqqOvkBrP+LWdv6Bdv5u6iqy5M8j+6GQGck+WBVfXIH+/ppCxfmWeP2ljkbOBr4Ol24VpLn0/XkDq2qnyS5jO6dPcDPq73lZut/px2ZPf5NAW8C7qa7E9zDgJ/Cto9JkqfR/T84odX8GrpeyFFV9bUkxwHP30EdAd5bVX8+z7r1IHlNQb2q+gndbQ/fku4OV48B7myzj5vvZuj++A9K8vY55v8dcFS6ES8fBbystW3PD+lOSezIWuDlSf4Z9PewfcI8656rzle0c+MzdHdCu6pt7+6q+h90d4nbPKT1z9s75YWocXvLfIru3gLH0gUEdP9O322BcBDd7V13ZEfH9LeTPKxdb3gi8I22nw2tB/EqulNFzHVM0n2K6mFVdR7wB2w5To8GNrRjNfqBgivoTtnB1rfavBh4Tbr7jpBk+ebjovEwFLSVqvoK3Tn2Y+nOQb83yVd4AL3K9q74WLoRYF87a941dO8Wr6K77nBa2+f2XEp3YXn0QvNc+72R7gXo8+lGF72E7jrAzvgU3XH4GvAF4G3VDWn9fOBr7Zi8gu5eytCd9752rgvND7TG7S1TVd+lO8X3hKq6qq1yEbAkyU10I2VeMY/f72zgrek+RPCkOeZ/i+7f6HPACVX1U7prG6vSXTA/iO7mPzD3MVkOXNZOS/4lsLnX85/p/t3/nq63s9kbgTe33/fJwPfb7/t5utNJX0pyHd1oqfN5g6Cd5CipkraS5AzgM1V17gT3uSfdacxKcgzdRecjJ7V/beE1BUmLwa/SXcQO8D26U5AagD0FSVLPawqSpJ6hIEnqGQqSpJ6hIEnqGQqSpN7/B+RnyrVwth3mAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "220 / 220 questions predicted.------------------------------------------------------------ \n",
            "\n",
            "Theme: Internet_service_provider\n",
            "Total Questions: 110\n",
            "Total Paragraphs: 21\n",
            "Total queries: 110\n",
            "In top 10 results, number of queries for which -\n",
            "\tRelevant paragraph found: 109 (99 %)\n",
            "\tRelevant paragraph NOT found: 1 (1 %)\n",
            "Mean Rank for which relevant paragraph found: 1.76\n",
            "\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAW0ElEQVR4nO3dfbRddX3n8ffHBMqTGh5us1IQg8JIWa5l1KsDYl0WxKHVSpyxCMvaMGWa5eioiKNi21ltZ5wRq/Vh6ow2BUu6SlFEKCgjyETQTotgeJInHRBBw4TkFsHH+gB+54+901xubpKbcPc5ib/3a627zj6//fS9Ozefs/fvnPPbqSokSe14wrgLkCSNlsEvSY0x+CWpMQa/JDXG4JekxiwcdwFzcdBBB9XSpUvHXYYk7VZuuOGGf6yqiZntu0XwL126lLVr1467DEnarSS5b7Z2u3okqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4Jakxu8U3dx+PpWddPpb93nv2y8ayX0naHs/4JakxBr8kNcbgl6TGGPyS1BiDX5IaM2jwJ3lLktuT3JbkgiR7JTksyXVJ7k7yiSR7DlmDJOmxBgv+JAcDbwImq+qZwALgFOA9wAeq6nDgIeD0oWqQJG1p6K6ehcDeSRYC+wDrgeOAi/r5q4HlA9cgSZpmsOCvqvuB9wHfpAv87wA3AA9X1SP9YuuAg4eqQZK0pSG7evYHTgIOA34J2Bc4cQfWX5lkbZK1U1NTA1UpSe0ZsqvnJcA3qmqqqn4KXAwcCyzqu34ADgHun23lqlpVVZNVNTkxscVN4iVJO2nI4P8mcHSSfZIEOB64A7gaeFW/zArg0gFrkCTNMGQf/3V0b+LeCNza72sV8A7gzCR3AwcC5w5VgyRpS4OOzllVfwj84Yzme4DnD7lfSdLW+c1dSWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjhrzZ+jOS3Dzt57tJzkhyQJKrktzVP+4/VA2SpC0NeevFr1XVsqpaBjwX+CFwCXAWsKaqjgDW9M8lSSMyqq6e44GvV9V9wEnA6r59NbB8RDVIkhhd8J8CXNBPL66q9f30A8Di2VZIsjLJ2iRrp6amRlGjJDVh8OBPsifwCuCTM+dVVQE123pVtaqqJqtqcmJiYuAqJakdozjj/zXgxqra0D/fkGQJQP+4cQQ1SJJ6owj+U9nczQNwGbCin14BXDqCGiRJvUGDP8m+wAnAxdOazwZOSHIX8JL+uSRpRBYOufGq+gFw4Iy2B+k+5SNJGgO/uStJjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaszQd+BalOSiJF9NcmeSY5IckOSqJHf1j/sPWYMk6bGGPuP/EHBFVR0JPAu4EzgLWFNVRwBr+ueSpBEZLPiTPBl4EXAuQFX9pKoeBk4CVveLrQaWD1WDJGlLQ57xHwZMAX+Z5KYk5/Q3X19cVev7ZR4AFs+2cpKVSdYmWTs1NTVgmZLUliGDfyHwHOAjVfVs4AfM6NapqgJqtpWralVVTVbV5MTExIBlSlJbhgz+dcC6qrquf34R3QvBhiRLAPrHjQPWIEmaYbDgr6oHgG8leUbfdDxwB3AZsKJvWwFcOlQNkqQtLRx4+28Ezk+yJ3AP8G/pXmwuTHI6cB9w8sA1SJKmGTT4q+pmYHKWWccPuV9J0tb5zV1JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaM+h4/EnuBb4HPAo8UlWTSQ4APgEsBe4FTq6qh4asQ5K02SjO+H+1qpZV1aYbspwFrKmqI4A1zLgBuyRpWOPo6jkJWN1PrwaWj6EGSWrW0MFfwOeS3JBkZd+2uKrW99MPAIsHrkGSNM3QN1t/YVXdn+QXgauSfHX6zKqqJDXbiv0LxUqAQw89dOAyJakdg57xV9X9/eNG4BLg+cCGJEsA+seNW1l3VVVNVtXkxMTEkGVKUlPmFPxJjp1L24z5+yZ54qZp4KXAbcBlwIp+sRXApTtSsCTp8ZlrV8+fAc+ZQ9t0i4FLkmzaz99U1RVJvgxcmOR04D7g5B0rWZL0eGwz+JMcA7wAmEhy5rRZTwIWbGvdqroHeNYs7Q8Cx+94qZKk+bC9M/49gf365Z44rf27wKuGKkqSNJxtBn9VfQH4QpLzquq+EdUkSRrQXPv4fyHJKrphFv55nao6boiiJEnDmWvwfxL4KHAO3bg7kqTd1FyD/5Gq+siglUiSRmKuX+D6dJLXJ1mS5IBNP4NWJkkaxFzP+Dd94ept09oKeNr8liNJGtqcgr+qDhu6EEnSaMwp+JP89mztVfVX81uOJGloc+3qed606b3ovnl7I2DwS9JuZq5dPW+c/jzJIuDjg1QkSRrUzg7L/APAfn9J2g3NtY//03Sf4oFucLZfBi4cqihJ0nDm2sf/vmnTjwD3VdW6AeqRJA1sTl09/WBtX6UboXN/4CdDFiVJGs5c78B1MnA98Jt0N065LonDMkvSbmiuXT2/Dzyvv3cuSSaA/w1ctL0VkywA1gL3V9XLkxxG94mgA4EbgNdWlVcQkjQic/1UzxM2hX7vwR1Y983AndOevwf4QFUdDjwEnD7H7UiS5sFcw/uKJFcmOS3JacDlwP/a3kpJDgFeRjecM+luwHscm68UVgPLd7RoSdLO2949dw8HFlfV25L8a+CF/axrgfPnsP0PAm9n820bDwQerqpH+ufrgIO3su+VwEqAQw89dA67kiTNxfbO+D9Id39dquriqjqzqs4ELunnbVWSlwMbq+qGnSmsqlZV1WRVTU5MTOzMJiRJs9jem7uLq+rWmY1VdWuSpdtZ91jgFUl+nW58nycBHwIWJVnYn/UfAty/w1VLknba9s74F21j3t7bWrGq3llVh1TVUuAU4PNV9RrgamDTR0FXAJfOsVZJ0jzYXvCvTfK7MxuT/Du6j2LujHcAZya5m67P/9yd3I4kaSdsr6vnDOCSJK9hc9BPAnsCr5zrTqrqGuCafvoe4Pk7WqgkaX5sM/iragPwgiS/Cjyzb768qj4/eGWSpEHMdTz+q+n65iVJu7mdHY9fkrSbMvglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaM1jwJ9kryfVJbklye5I/7tsPS3JdkruTfCLJnkPVIEna0pBn/D8GjquqZwHLgBOTHA28B/hAVR0OPAScPmANkqQZBgv+6ny/f7pH/1PAccBFfftqYPlQNUiStjRoH3+SBUluBjYCVwFfBx6uqkf6RdYBB29l3ZVJ1iZZOzU1NWSZktSUQYO/qh6tqmXAIXQ3WD9yB9ZdVVWTVTU5MTExWI2S1JqRfKqnqh6mu2fvMcCiJJvu9XsIcP8oapAkdYb8VM9EkkX99N7ACcCddC8Ar+oXWwFcOlQNkqQtLdz+IjttCbA6yQK6F5gLq+ozSe4APp7kXcBNwLkD1iBJmmGw4K+qrwDPnqX9Hrr+fknSGPjNXUlqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhoz5B24npLk6iR3JLk9yZv79gOSXJXkrv5x/6FqkCRtacgz/keAt1bVUcDRwBuSHAWcBaypqiOANf1zSdKIDBb8VbW+qm7sp79Hd7/dg4GTgNX9YquB5UPVIEna0kj6+JMspbsN43XA4qpa3896AFi8lXVWJlmbZO3U1NQoypSkJgwe/En2Az4FnFFV350+r6oKqNnWq6pVVTVZVZMTExNDlylJzRg0+JPsQRf651fVxX3zhiRL+vlLgI1D1iBJeqwhP9UT4Fzgzqp6/7RZlwEr+ukVwKVD1SBJ2tLCAbd9LPBa4NYkN/dtvwecDVyY5HTgPuDkAWuQJM0wWPBX1f8BspXZxw+1X0nStvnNXUlqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSY4a89eLHkmxMctu0tgOSXJXkrv5x/6H2L0ma3ZBn/OcBJ85oOwtYU1VHAGv655KkERos+Kvqi8C3ZzSfBKzup1cDy4favyRpdqPu419cVev76QeAxVtbMMnKJGuTrJ2amhpNdZLUgLG9uVtVBdQ25q+qqsmqmpyYmBhhZZL0823Uwb8hyRKA/nHjiPcvSc1bOOL9XQasAM7uHy8d8f5HZulZl49t3/ee/bKx7VvSrm/Ij3NeAFwLPCPJuiSn0wX+CUnuAl7SP5ckjdBgZ/xVdepWZh0/1D4lSdvnN3clqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4Jakxox6rRyMwrnGCxjlGUIu/s7SzPOOXpMZ4xi9J2/HzdkXpGb8kNcbgl6TG2NWjeTPOm8+06Oet+0Gj4xm/JDVmLGf8SU4EPgQsAM6pKu/Epd2SVznaHY38jD/JAuB/AL8GHAWcmuSoUdchSa0aR1fP84G7q+qeqvoJ8HHgpDHUIUlNGkdXz8HAt6Y9Xwf8y5kLJVkJrOyffj/J10ZQ25AOAv5x3EXsIjwWj7VbHY+8Z/Bd7FbHY0h5z+M+Fk+drXGX/VRPVa0CVo27jvmSZG1VTY67jl2Bx+KxPB6P5fHYbKhjMY6unvuBp0x7fkjfJkkagXEE/5eBI5IclmRP4BTgsjHUIUlNGnlXT1U9kuQ/AFfSfZzzY1V1+6jrGIOfm26reeCxeCyPx2N5PDYb5FikqobYriRpF+U3dyWpMQa/JDXG4B9QkqckuTrJHUluT/Lmcde0K0iyIMlNST4z7lrGLcmiJBcl+WqSO5McM+6axiXJW/r/J7cluSDJXuOuaZSSfCzJxiS3TWs7IMlVSe7qH/efj30Z/MN6BHhrVR0FHA28weEpAHgzcOe4i9hFfAi4oqqOBJ5Fo8clycHAm4DJqnom3Qc/ThlvVSN3HnDijLazgDVVdQSwpn/+uBn8A6qq9VV1Yz/9Pbr/1AePt6rxSnII8DLgnHHXMm5Jngy8CDgXoKp+UlUPj7eqsVoI7J1kIbAP8P/GXM9IVdUXgW/PaD4JWN1PrwaWz8e+DP4RSbIUeDZw3XgrGbsPAm8HfjbuQnYBhwFTwF/2XV/nJNl33EWNQ1XdD7wP+CawHvhOVX1uvFXtEhZX1fp++gFg8Xxs1OAfgST7AZ8Czqiq7467nnFJ8nJgY1XdMO5adhELgecAH6mqZwM/YJ4u5Xc3fd/1SXQvhr8E7Jvkt8Zb1a6lus/ez8vn7w3+gSXZgy70z6+qi8ddz5gdC7wiyb10o7Iel+Svx1vSWK0D1lXVpqvAi+heCFr0EuAbVTVVVT8FLgZeMOaadgUbkiwB6B83zsdGDf4BJQld/+2dVfX+cdczblX1zqo6pKqW0r1x9/mqavasrqoeAL6V5Bl90/HAHWMsaZy+CRydZJ/+/83xNPpG9wyXASv66RXApfOxUYN/WMcCr6U7s725//n1cRelXcobgfOTfAVYBvy3MdczFv1Vz0XAjcCtdNnU1NANSS4ArgWekWRdktOBs4ETktxFd1U0L3crdMgGSWqMZ/yS1BiDX5IaY/BLUmMMfklqjMEvSY0x+BuR5NH+46S3Jfl0kkWPY1vffxzrvqkfhfL8Ge3Lpn/UNckfJfmPO7ufIST5vXnYxr1JDpqPeuawr+UOCqjZGPzt+KeqWtaPfPht4A1jquP1wAlV9ZoZ7cuAXf07DnMK/iQLhi5kjpYDBr+2YPC36Vr6UUKTPD/Jtf0gYf+w6VukSU5LcnGSK/qxwP9k5kaSHNSv+7JZ5p3ZX13cluSMvu2jwNOAzyZ5y7Rl9wT+M/Dq/qrk1f2so5Jck+SeJG+atvxvJbm+X/bPZwva/sz63f0ya5M8J8mVSb6e5HX9Mkny3r7GWzftN8mSJF+cdoX0K0nOphs58uaZVyv9Ot9P8qdJbgGOmWONWyyT5HVJ3jttmdOSfLif/tskN6Qbs37ljH3/1yS3JPlSksVJXgC8Anhvv/2nz9j3eUk+2h+b/9uPo0SSpUn+LsmN/c8LtnFMFvTb2XT83tIv+7tJvtzX86kk+/TtT+/ruzXJu6ZfOSZ5W7/OV5L88cxjpXlWVf408AN8v39cAHwSOLF//iRgYT/9EuBT/fRpwD3Ak4G9gPuAp2zaFt0ogdfRnb3P3Ndz6b59uS+wH3A78Ox+3r3AQbOscxrw4WnP/wj4B+AXgIOAB4E9gF8GPg3s0S/3P4HfnmV79wL/vp/+APAV4InABLChb/83wFX9MVlMN2zAEuCtwO9PO15PnH4Mt3J8Czi5n95qjZt+/60t09d397TtfhZ4YT99QP+4N3AbcOC0ff9GP/0nwB/00+cBr9pKvecBV9Cd/B1BN27QXnTDIe/VL3MEsLaf3uKY9P/OV03b5qL+8cBpbe8C3thPfwY4tZ9+HZv/Jl9K9y3d9PV8BnjRuP/P/Dz/LESt2DvJzXRn+nfSBR50wb46yRF0AbLHtHXWVNV3AJLcATwV+Fa/zBrgDVX1hVn29ULgkqr6Qb/uxcCvADftYM2XV9WPgR8n2UgXzsfTBc6Xk0AXglsbuOqy/vFWYL/q7onwvSQ/TvcexwuBC6rqUbrBsL4APA/4MvCxdAPs/W1V3TyHWh+lG4yPOdY46zJVNdVf4RwN3AUcCfx9v86bkryyn34KXTA/CPyELiwBbgBOmEO9ABdW1c+Au5Lc0+/rG8CHkyzrf6d/0S+7xTHp13lakj8DLgc2DaP8zCTvAhbRvfBf2bcfw+bx5P+Gbhhm6IL/pWz++9iv/92+OMffQzvI4G/HP1XVsv6y+0q6Pv7/DvwX4OqqemW6ewZcM22dH0+bfpTNfy+P0AXMvwJmC/75Mtv+A6yuqnfuwPo/m7Gtn7GNv/2q+mKSF9HdMOa8JO+vqr/azr5+1L+AMMcat7XMx4GTga/SvYBWkhfTXZEdU1U/THIN3Rk6wE+rP3Xmsf9O2zNzvJYC3gJsoLsb2BOAH8HWj0mSZ9H9Hbyur/l36K4mllfVLUlOA168nToCvLuq/nyOdetxso+/MVX1Q7pb3L013Z2Ongzc388+ba6bofsPfmSSd8wy/++A5elGWtwXeGXfti3fo+s+2J41wKuS/CL88z1JnzrHumer89V9X/UE3d2wru+3t6Gq/oLuTmGbhkr+aX/GOx81bmuZS+jGpj+V7kUAun+nh/rQP5LuVp7bs71j+ptJntD3/z8N+Fq/n/X9lcBr6bp1mO2YpPt00hOq6lPAH7D5OD0RWN8fq+lv4n+JrnsNHntbxSuB30l33wqSHLzpuGgYBn+Dquomuj7vU+n6hN+d5CZ24AqwP7s9lW7k0dfPmHcj3Vnf9XTvA5zT73NbrqZ7M3f6m7uz7fcOupD5XLoRLa+i65ffGZfQHYdbgM8Db69uqOQXA7f0x+TVdPfFha4f+iuzvbm7ozVua5mqeoiuO+6pVXV9v8oVwMIkd9KN0PilOfx+Hwfelu6N+6fPMv+bdP9GnwVeV1U/onuvYUW6N6mPpLs5DMx+TA4Grum7EP8a2HT18p/o/t3/nu6qZZMzgDP73/dw4Dv97/s5uq6fa5PcSjdK51xOArSTHJ1TalCS84DPVNVFI9znPnRdjpXkFLo3ek8a1f61mX38kkbluXRvHAd4mK67UGPgGb8kNcY+fklqjMEvSY0x+CWpMQa/JDXG4Jekxvx/yywtzrKKcmIAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "110 / 110 questions predicted.------------------------------------------------------------ \n",
            "\n"
          ]
        }
      ],
      "source": [
        "themes = [\n",
        "    'Beyoncé', 'Spectre_(2015_film)', 'New_York_City', 'To_Kill_a_Mockingbird', 'Solar_energy', 'Buddhism', 'American_Idol', 'Dog',\n",
        "    '2008_Summer_Olympics_torch_relay', 'Genome', 'Comprehensive_school', 'Prime_minister', 'Institute_of_technology', 'Hydrogen',\n",
        "    'Separation_of_powers_under_the_United_States_Constitution', 'Architecture', 'Alexander_Graham_Bell', 'Matter'\n",
        "]\n",
        "preds_with_gold, total_inference_time = [], 0.\n",
        "for theme in themes:\n",
        "    print('Theme:', theme)\n",
        "    pwg, tinf = predict_theme_wise(theme, theme_wise_data, sents_encoder, optimum_qa, k = 10)\n",
        "    preds_with_gold += pwg\n",
        "    total_inference_time += tinf\n",
        "    print('\\n', ''.join(['-']*60), '\\n')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TPAGBNbvXVDr"
      },
      "outputs": [],
      "source": [
        "# 28m 13s"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Uqo95wQWDk4z",
        "outputId": "4a263322-fb5f-4303-de13-f5670e8a8323"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Number of themes tested: 18\n",
            "Total queries: 4711\n",
            "Answerable queries: 4062\n",
            "Unanswerable queries: 649\n",
            "Average Inference Time: 321.1093446905257 ms\n",
            "EM Score: 0.6754404585013798\n",
            "F1 Score: 0.7414974818925528\n"
          ]
        }
      ],
      "source": [
        "print('Number of themes tested:', len(themes))\n",
        "print('Total queries:', len(preds_with_gold))\n",
        "answerable = len([1 for x in preds_with_gold if x['gold_answers'][0] != ''])\n",
        "print(f'Answerable queries:', answerable)\n",
        "print(f'Unanswerable queries:', len(preds_with_gold) - answerable)\n",
        "print(f'Average Inference Time: {total_inference_time*1000 / len(preds_with_gold)} ms')\n",
        "evaluate(preds_with_gold)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_lf7B5bTJGOn"
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "Y_yDtvzyK-lf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "ZQiEQ1XhHH_Y"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## HayStack"
      ],
      "metadata": {
        "id": "pKK5UEmtHl4m"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!git clone https://github.com/deepset-ai/FARM.git\n",
        "%cd FARM\n",
        "!pip install -r requirements.txt\n",
        "!pip install --editable .\n",
        "%cd .."
      ],
      "metadata": {
        "id": "5CbTf4BBB-wX"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "%cd FARM"
      ],
      "metadata": {
        "id": "X2Kf-iT2Edu6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from farm.modeling.adaptive_model import AdaptiveModel\n",
        "from farm.modeling.tokenization import Tokenizer\n",
        "from farm.infer import Inferencer\n",
        "\n",
        "model_name = \"deepset/minilm-uncased-squad2\"\n",
        "\n",
        "# a) Get predictions\n",
        "nlp = Inferencer.load(model_name, task_type=\"question_answering\")"
      ],
      "metadata": {
        "id": "nXoXDUBIiY0p"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "QA_input = [{\"questions\": [\"Why is model conversion important?\"],\n",
        "             \"text\": \"The option to convert models between FARM and transformers gives freedom to the user and let people easily switch between frameworks.\"}]\n",
        "res = nlp.inference_from_dicts(dicts=QA_input)\n",
        "\n",
        "# b) Load model & tokenizer\n",
        "model = AdaptiveModel.convert_from_transformers(model_name, device=\"cpu\", task_type=\"question_answering\")\n",
        "tokenizer = Tokenizer.load(model_name)"
      ],
      "metadata": {
        "id": "nUjIsgDUB7mm"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "vRlm9iCvCNLC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install git+https://github.com/deepset-ai/haystack.git#egg=farm-haystack[colab]"
      ],
      "metadata": {
        "id": "f4-7g1UjEyh3"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "%%bash\n",
        "\n",
        "wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.9.2-linux-x86_64.tar.gz -q\n",
        "tar -xzf elasticsearch-7.9.2-linux-x86_64.tar.gz\n",
        "chown -R daemon:daemon elasticsearch-7.9.2"
      ],
      "metadata": {
        "id": "PrIEsLzYFJIf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "%%bash --bg\n",
        "\n",
        "sudo -u daemon -- elasticsearch-7.9.2/bin/elasticsearch"
      ],
      "metadata": {
        "id": "EMW4colMFjrf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import time\n",
        "time.sleep(30)"
      ],
      "metadata": {
        "id": "pB9uKGATF3RW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "from haystack.document_stores import ElasticsearchDocumentStore\n",
        "\n",
        "# Get the host where Elasticsearch is running, default to localhost\n",
        "host = os.environ.get(\"ELASTICSEARCH_HOST\", \"localhost\")\n",
        "document_store = ElasticsearchDocumentStore(host=host, username=\"\", password=\"\", index=\"document\")"
      ],
      "metadata": {
        "id": "fwumMwVjF7E6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "gFXLHd97Gc0K"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "data = load_data()\n",
        "twd = load_theme_wise_data(data)"
      ],
      "metadata": {
        "id": "1FlOGiN9Gcwx"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "theme = \"Dog\"\n",
        "docs = []\n",
        "queries = twd[theme]['ques']\n",
        "for para in twd[theme]['para']:\n",
        "    doc = {\n",
        "        'content': para,\n",
        "        'meta': {\n",
        "            'name': \"Dog\"\n",
        "        }\n",
        "    }\n",
        "    docs.append(doc)"
      ],
      "metadata": {
        "id": "3Nbh1M_FGcuN"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(docs[:3])\n",
        "document_store.write_documents(docs)"
      ],
      "metadata": {
        "id": "yUHEgK3aInkB"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from haystack.nodes import BM25Retriever\n",
        "\n",
        "retriever = BM25Retriever(document_store=document_store)"
      ],
      "metadata": {
        "id": "8B0lflnYIvqx"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from haystack.nodes import FARMReader\n",
        "\n",
        "# Load a  local model or any of the QA models on\n",
        "# Hugging Face's model hub (https://huggingface.co/models)\n",
        "\n",
        "reader = FARMReader(model_name_or_path=\"deepset/minilm-uncased-squad2\", use_gpu=False)"
      ],
      "metadata": {
        "id": "RQ7vnqc2I3Qk"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from haystack.pipelines import ExtractiveQAPipeline\n",
        "\n",
        "pipe = ExtractiveQAPipeline(reader, retriever)"
      ],
      "metadata": {
        "id": "Zk8NXZveI8Uf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "len(queries)"
      ],
      "metadata": {
        "id": "ZOxOj_2PLqwT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# You can configure how many candidates the Reader and Retriever shall return\n",
        "# The higher top_k_retriever, the better (but also the slower) your answers.\n",
        "prediction = pipe.run_batch(\n",
        "    queries=queries, params={\"Retriever\": {\"top_k\": 1}, \"Reader\": {\"top_k\": 1}}\n",
        ")"
      ],
      "metadata": {
        "id": "JKai_Z0MJErC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from haystack.utils import print_answers\n",
        "\n",
        "# Change `minimum` to `medium` or `all` to raise the level of detail\n",
        "print_answers(prediction, details=\"minimum\")"
      ],
      "metadata": {
        "id": "hn4fs3nVJIBa"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "Zc_8pHUlJRoN"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Sythetic Data Generation"
      ],
      "metadata": {
        "id": "W3FaiJv7LkfS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install git+https://github.com/ramsrigouthamg/Questgen.ai\n",
        "!pip install sense2vec\n",
        "!pip install git+https://github.com/boudinfl/pke.git\n",
        "\n",
        "!python -m nltk.downloader universal_tagset\n",
        "!python -m spacy download en"
      ],
      "metadata": {
        "id": "vREYZwZjLndU",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "52ffffa9-65ad-4f45-f417-72be22aed548"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting git+https://github.com/ramsrigouthamg/Questgen.ai\n",
            "  Cloning https://github.com/ramsrigouthamg/Questgen.ai to /tmp/pip-req-build-dl8u41m5\n",
            "  Running command git clone --filter=blob:none --quiet https://github.com/ramsrigouthamg/Questgen.ai /tmp/pip-req-build-dl8u41m5\n",
            "  Resolved https://github.com/ramsrigouthamg/Questgen.ai to commit 51e3987bd2da141db67817543e2fc7452b44491f\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting torch==1.10.0\n",
            "  Downloading torch-1.10.0-cp38-cp38-manylinux1_x86_64.whl (881.9 MB)\n",
            "\u001b[2K     \u001b[91m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[91m╸\u001b[0m \u001b[32m881.9/881.9 MB\u001b[0m \u001b[31m18.3 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0mtcmalloc: large alloc 1102381056 bytes == 0x37690000 @  0x7f4fb0ada680 0x7f4fb0afada2 0x5f714c 0x64d800 0x527022 0x504866 0x56bbe1 0x569d8a 0x5f60c3 0x56bbe1 0x569d8a 0x5f60c3 0x56bbe1 0x569d8a 0x5f60c3 0x56bbe1 0x569d8a 0x5f60c3 0x56bbe1 0x569d8a 0x5f60c3 0x56bbe1 0x5f5ee6 0x56bbe1 0x569d8a 0x5f60c3 0x56cc92 0x569d8a 0x5f60c3 0x56bbe1 0x569d8a\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m881.9/881.9 MB\u001b[0m \u001b[31m611.7 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting transformers==3.0.2\n",
            "  Downloading transformers-3.0.2-py3-none-any.whl (769 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m769.0/769.0 KB\u001b[0m \u001b[31m36.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting sense2vec==2.0.0\n",
            "  Downloading sense2vec-2.0.0-py2.py3-none-any.whl (39 kB)\n",
            "Collecting strsim==0.0.3\n",
            "  Downloading strsim-0.0.3-py3-none-any.whl (42 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m42.4/42.4 KB\u001b[0m \u001b[31m4.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: six==1.15.0 in /usr/local/lib/python3.8/dist-packages (from Questgen==1.0.0) (1.15.0)\n",
            "Collecting networkx==2.6.3\n",
            "  Downloading networkx-2.6.3-py3-none-any.whl (1.9 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.9/1.9 MB\u001b[0m \u001b[31m65.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting numpy==1.21.5\n",
            "  Downloading numpy-1.21.5-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m15.7/15.7 MB\u001b[0m \u001b[31m48.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting scipy==1.4.1\n",
            "  Downloading scipy-1.4.1-cp38-cp38-manylinux1_x86_64.whl (26.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m26.0/26.0 MB\u001b[0m \u001b[31m45.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: scikit-learn==1.0.2 in /usr/local/lib/python3.8/dist-packages (from Questgen==1.0.0) (1.0.2)\n",
            "Collecting unidecode==1.3.4\n",
            "  Downloading Unidecode-1.3.4-py3-none-any.whl (235 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m235.9/235.9 KB\u001b[0m \u001b[31m21.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: future==0.16.0 in /usr/local/lib/python3.8/dist-packages (from Questgen==1.0.0) (0.16.0)\n",
            "Collecting joblib==1.1.0\n",
            "  Downloading joblib-1.1.0-py2.py3-none-any.whl (306 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m307.0/307.0 KB\u001b[0m \u001b[31m23.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pytz==2018.9\n",
            "  Downloading pytz-2018.9-py2.py3-none-any.whl (510 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m510.7/510.7 KB\u001b[0m \u001b[31m28.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: python-dateutil==2.8.2 in /usr/local/lib/python3.8/dist-packages (from Questgen==1.0.0) (2.8.2)\n",
            "Collecting flashtext==2.7\n",
            "  Downloading flashtext-2.7.tar.gz (14 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: pandas==1.3.5 in /usr/local/lib/python3.8/dist-packages (from Questgen==1.0.0) (1.3.5)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.8/dist-packages (from scikit-learn==1.0.2->Questgen==1.0.0) (3.1.0)\n",
            "Requirement already satisfied: spacy<4.0.0,>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from sense2vec==2.0.0->Questgen==1.0.0) (3.4.4)\n",
            "Requirement already satisfied: wasabi<1.1.0,>=0.8.1 in /usr/local/lib/python3.8/dist-packages (from sense2vec==2.0.0->Questgen==1.0.0) (0.10.1)\n",
            "Requirement already satisfied: srsly<3.0.0,>=2.4.0 in /usr/local/lib/python3.8/dist-packages (from sense2vec==2.0.0->Questgen==1.0.0) (2.4.5)\n",
            "Requirement already satisfied: catalogue<2.1.0,>=2.0.1 in /usr/local/lib/python3.8/dist-packages (from sense2vec==2.0.0->Questgen==1.0.0) (2.0.8)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.8/dist-packages (from torch==1.10.0->Questgen==1.0.0) (4.4.0)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.8/dist-packages (from transformers==3.0.2->Questgen==1.0.0) (2.25.1)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from transformers==3.0.2->Questgen==1.0.0) (21.3)\n",
            "Collecting sacremoses\n",
            "  Downloading sacremoses-0.0.53.tar.gz (880 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m880.6/880.6 KB\u001b[0m \u001b[31m47.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.8/dist-packages (from transformers==3.0.2->Questgen==1.0.0) (3.9.0)\n",
            "Collecting tokenizers==0.8.1.rc1\n",
            "  Downloading tokenizers-0.8.1rc1-cp38-cp38-manylinux1_x86_64.whl (3.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.0/3.0 MB\u001b[0m \u001b[31m51.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.8/dist-packages (from transformers==3.0.2->Questgen==1.0.0) (4.64.1)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.8/dist-packages (from transformers==3.0.2->Questgen==1.0.0) (2022.6.2)\n",
            "Collecting sentencepiece!=0.1.92\n",
            "  Downloading sentencepiece-0.1.97-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m55.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: thinc<8.2.0,>=8.1.0 in /usr/local/lib/python3.8/dist-packages (from spacy<4.0.0,>=3.0.0->sense2vec==2.0.0->Questgen==1.0.0) (8.1.6)\n",
            "Requirement already satisfied: smart-open<7.0.0,>=5.2.1 in /usr/local/lib/python3.8/dist-packages (from spacy<4.0.0,>=3.0.0->sense2vec==2.0.0->Questgen==1.0.0) (6.3.0)\n",
            "Requirement already satisfied: pathy>=0.3.5 in /usr/local/lib/python3.8/dist-packages (from spacy<4.0.0,>=3.0.0->sense2vec==2.0.0->Questgen==1.0.0) (0.10.1)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.8/dist-packages (from spacy<4.0.0,>=3.0.0->sense2vec==2.0.0->Questgen==1.0.0) (57.4.0)\n",
            "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from spacy<4.0.0,>=3.0.0->sense2vec==2.0.0->Questgen==1.0.0) (2.0.7)\n",
            "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4 in /usr/local/lib/python3.8/dist-packages (from spacy<4.0.0,>=3.0.0->sense2vec==2.0.0->Questgen==1.0.0) (1.10.4)\n",
            "Requirement already satisfied: typer<0.8.0,>=0.3.0 in /usr/local/lib/python3.8/dist-packages (from spacy<4.0.0,>=3.0.0->sense2vec==2.0.0->Questgen==1.0.0) (0.7.0)\n",
            "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.10 in /usr/local/lib/python3.8/dist-packages (from spacy<4.0.0,>=3.0.0->sense2vec==2.0.0->Questgen==1.0.0) (3.0.11)\n",
            "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.8/dist-packages (from spacy<4.0.0,>=3.0.0->sense2vec==2.0.0->Questgen==1.0.0) (1.0.9)\n",
            "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /usr/local/lib/python3.8/dist-packages (from spacy<4.0.0,>=3.0.0->sense2vec==2.0.0->Questgen==1.0.0) (1.0.4)\n",
            "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from spacy<4.0.0,>=3.0.0->sense2vec==2.0.0->Questgen==1.0.0) (3.0.8)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from spacy<4.0.0,>=3.0.0->sense2vec==2.0.0->Questgen==1.0.0) (2.11.3)\n",
            "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /usr/local/lib/python3.8/dist-packages (from spacy<4.0.0,>=3.0.0->sense2vec==2.0.0->Questgen==1.0.0) (3.3.0)\n",
            "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->transformers==3.0.2->Questgen==1.0.0) (3.0.9)\n",
            "Requirement already satisfied: chardet<5,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from requests->transformers==3.0.2->Questgen==1.0.0) (4.0.0)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests->transformers==3.0.2->Questgen==1.0.0) (1.24.3)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests->transformers==3.0.2->Questgen==1.0.0) (2.10)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests->transformers==3.0.2->Questgen==1.0.0) (2022.12.7)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.8/dist-packages (from sacremoses->transformers==3.0.2->Questgen==1.0.0) (7.1.2)\n",
            "Requirement already satisfied: blis<0.8.0,>=0.7.8 in /usr/local/lib/python3.8/dist-packages (from thinc<8.2.0,>=8.1.0->spacy<4.0.0,>=3.0.0->sense2vec==2.0.0->Questgen==1.0.0) (0.7.9)\n",
            "Requirement already satisfied: confection<1.0.0,>=0.0.1 in /usr/local/lib/python3.8/dist-packages (from thinc<8.2.0,>=8.1.0->spacy<4.0.0,>=3.0.0->sense2vec==2.0.0->Questgen==1.0.0) (0.0.3)\n",
            "Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.8/dist-packages (from jinja2->spacy<4.0.0,>=3.0.0->sense2vec==2.0.0->Questgen==1.0.0) (2.0.1)\n",
            "Building wheels for collected packages: Questgen, flashtext, sacremoses\n",
            "  Building wheel for Questgen (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for Questgen: filename=Questgen-1.0.0-py3-none-any.whl size=8614 sha256=884b2f2300891b4c2cb4200060daa8e6ed12968322b8eaab6fa22c4527f0afe3\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-amebahvr/wheels/de/09/3e/fa66dbcd8b70e37f55636c84567be559749dedfa2256b7052e\n",
            "  Building wheel for flashtext (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for flashtext: filename=flashtext-2.7-py2.py3-none-any.whl size=9307 sha256=a7598cba00167c097681f6f7e488c3f881ecf072eb288998e8c99feb2c9ca2b5\n",
            "  Stored in directory: /root/.cache/pip/wheels/8d/62/8b/71813348245ae1bcbae179193bbc72db819e8057e89298a6ac\n",
            "  Building wheel for sacremoses (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for sacremoses: filename=sacremoses-0.0.53-py3-none-any.whl size=895260 sha256=7cd002b47b5ecaa3affb8b877face7216d0a2ab4234941fa551b08f1167bce8e\n",
            "  Stored in directory: /root/.cache/pip/wheels/82/ab/9b/c15899bf659ba74f623ac776e861cf2eb8608c1825ddec66a4\n",
            "Successfully built Questgen flashtext sacremoses\n",
            "Installing collected packages: tokenizers, strsim, sentencepiece, pytz, flashtext, unidecode, torch, numpy, networkx, joblib, scipy, sacremoses, transformers, sense2vec, Questgen\n",
            "  Attempting uninstall: pytz\n",
            "    Found existing installation: pytz 2022.7\n",
            "    Uninstalling pytz-2022.7:\n",
            "      Successfully uninstalled pytz-2022.7\n",
            "  Attempting uninstall: torch\n",
            "    Found existing installation: torch 1.13.1+cu116\n",
            "    Uninstalling torch-1.13.1+cu116:\n",
            "      Successfully uninstalled torch-1.13.1+cu116\n",
            "  Attempting uninstall: numpy\n",
            "    Found existing installation: numpy 1.21.6\n",
            "    Uninstalling numpy-1.21.6:\n",
            "      Successfully uninstalled numpy-1.21.6\n",
            "  Attempting uninstall: networkx\n",
            "    Found existing installation: networkx 3.0\n",
            "    Uninstalling networkx-3.0:\n",
            "      Successfully uninstalled networkx-3.0\n",
            "  Attempting uninstall: joblib\n",
            "    Found existing installation: joblib 1.2.0\n",
            "    Uninstalling joblib-1.2.0:\n",
            "      Successfully uninstalled joblib-1.2.0\n",
            "  Attempting uninstall: scipy\n",
            "    Found existing installation: scipy 1.7.3\n",
            "    Uninstalling scipy-1.7.3:\n",
            "      Successfully uninstalled scipy-1.7.3\n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "xarray-einstats 0.4.0 requires scipy>=1.6, but you have scipy 1.4.1 which is incompatible.\n",
            "torchvision 0.14.1+cu116 requires torch==1.13.1, but you have torch 1.10.0 which is incompatible.\n",
            "torchtext 0.14.1 requires torch==1.13.1, but you have torch 1.10.0 which is incompatible.\n",
            "torchaudio 0.13.1+cu116 requires torch==1.13.1, but you have torch 1.10.0 which is incompatible.\n",
            "plotnine 0.8.0 requires scipy>=1.5.0, but you have scipy 1.4.1 which is incompatible.\n",
            "jaxlib 0.3.25+cuda11.cudnn805 requires scipy>=1.5, but you have scipy 1.4.1 which is incompatible.\n",
            "jax 0.3.25 requires scipy>=1.5, but you have scipy 1.4.1 which is incompatible.\u001b[0m\u001b[31m\n",
            "\u001b[0mSuccessfully installed Questgen-1.0.0 flashtext-2.7 joblib-1.1.0 networkx-2.6.3 numpy-1.21.5 pytz-2018.9 sacremoses-0.0.53 scipy-1.4.1 sense2vec-2.0.0 sentencepiece-0.1.97 strsim-0.0.3 tokenizers-0.8.1rc1 torch-1.10.0 transformers-3.0.2 unidecode-1.3.4\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: sense2vec in /usr/local/lib/python3.8/dist-packages (2.0.0)\n",
            "Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.8/dist-packages (from sense2vec) (1.21.5)\n",
            "Requirement already satisfied: wasabi<1.1.0,>=0.8.1 in /usr/local/lib/python3.8/dist-packages (from sense2vec) (0.10.1)\n",
            "Requirement already satisfied: spacy<4.0.0,>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from sense2vec) (3.4.4)\n",
            "Requirement already satisfied: catalogue<2.1.0,>=2.0.1 in /usr/local/lib/python3.8/dist-packages (from sense2vec) (2.0.8)\n",
            "Requirement already satisfied: srsly<3.0.0,>=2.4.0 in /usr/local/lib/python3.8/dist-packages (from sense2vec) (2.4.5)\n",
            "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.8/dist-packages (from spacy<4.0.0,>=3.0.0->sense2vec) (1.0.9)\n",
            "Requirement already satisfied: typer<0.8.0,>=0.3.0 in /usr/local/lib/python3.8/dist-packages (from spacy<4.0.0,>=3.0.0->sense2vec) (0.7.0)\n",
            "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /usr/local/lib/python3.8/dist-packages (from spacy<4.0.0,>=3.0.0->sense2vec) (3.3.0)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.8/dist-packages (from spacy<4.0.0,>=3.0.0->sense2vec) (57.4.0)\n",
            "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.8/dist-packages (from spacy<4.0.0,>=3.0.0->sense2vec) (4.64.1)\n",
            "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from spacy<4.0.0,>=3.0.0->sense2vec) (2.0.7)\n",
            "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.10 in /usr/local/lib/python3.8/dist-packages (from spacy<4.0.0,>=3.0.0->sense2vec) (3.0.11)\n",
            "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.8/dist-packages (from spacy<4.0.0,>=3.0.0->sense2vec) (2.25.1)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from spacy<4.0.0,>=3.0.0->sense2vec) (2.11.3)\n",
            "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4 in /usr/local/lib/python3.8/dist-packages (from spacy<4.0.0,>=3.0.0->sense2vec) (1.10.4)\n",
            "Requirement already satisfied: smart-open<7.0.0,>=5.2.1 in /usr/local/lib/python3.8/dist-packages (from spacy<4.0.0,>=3.0.0->sense2vec) (6.3.0)\n",
            "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /usr/local/lib/python3.8/dist-packages (from spacy<4.0.0,>=3.0.0->sense2vec) (1.0.4)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.8/dist-packages (from spacy<4.0.0,>=3.0.0->sense2vec) (21.3)\n",
            "Requirement already satisfied: pathy>=0.3.5 in /usr/local/lib/python3.8/dist-packages (from spacy<4.0.0,>=3.0.0->sense2vec) (0.10.1)\n",
            "Requirement already satisfied: thinc<8.2.0,>=8.1.0 in /usr/local/lib/python3.8/dist-packages (from spacy<4.0.0,>=3.0.0->sense2vec) (8.1.6)\n",
            "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from spacy<4.0.0,>=3.0.0->sense2vec) (3.0.8)\n",
            "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging>=20.0->spacy<4.0.0,>=3.0.0->sense2vec) (3.0.9)\n",
            "Requirement already satisfied: typing-extensions>=4.2.0 in /usr/local/lib/python3.8/dist-packages (from pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4->spacy<4.0.0,>=3.0.0->sense2vec) (4.4.0)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests<3.0.0,>=2.13.0->spacy<4.0.0,>=3.0.0->sense2vec) (1.24.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests<3.0.0,>=2.13.0->spacy<4.0.0,>=3.0.0->sense2vec) (2022.12.7)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests<3.0.0,>=2.13.0->spacy<4.0.0,>=3.0.0->sense2vec) (2.10)\n",
            "Requirement already satisfied: chardet<5,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from requests<3.0.0,>=2.13.0->spacy<4.0.0,>=3.0.0->sense2vec) (4.0.0)\n",
            "Requirement already satisfied: blis<0.8.0,>=0.7.8 in /usr/local/lib/python3.8/dist-packages (from thinc<8.2.0,>=8.1.0->spacy<4.0.0,>=3.0.0->sense2vec) (0.7.9)\n",
            "Requirement already satisfied: confection<1.0.0,>=0.0.1 in /usr/local/lib/python3.8/dist-packages (from thinc<8.2.0,>=8.1.0->spacy<4.0.0,>=3.0.0->sense2vec) (0.0.3)\n",
            "Requirement already satisfied: click<9.0.0,>=7.1.1 in /usr/local/lib/python3.8/dist-packages (from typer<0.8.0,>=0.3.0->spacy<4.0.0,>=3.0.0->sense2vec) (7.1.2)\n",
            "Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.8/dist-packages (from jinja2->spacy<4.0.0,>=3.0.0->sense2vec) (2.0.1)\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting git+https://github.com/boudinfl/pke.git\n",
            "  Cloning https://github.com/boudinfl/pke.git to /tmp/pip-req-build-6y8m1kdw\n",
            "  Running command git clone --filter=blob:none --quiet https://github.com/boudinfl/pke.git /tmp/pip-req-build-6y8m1kdw\n",
            "  Resolved https://github.com/boudinfl/pke.git to commit 8f1d05dcc52041c9920ba0f9d5231fe6086d12c4\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: nltk in /usr/local/lib/python3.8/dist-packages (from pke==2.0.0) (3.7)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.8/dist-packages (from pke==2.0.0) (2.6.3)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.8/dist-packages (from pke==2.0.0) (1.21.5)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from pke==2.0.0) (1.4.1)\n",
            "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.8/dist-packages (from pke==2.0.0) (1.0.2)\n",
            "Requirement already satisfied: unidecode in /usr/local/lib/python3.8/dist-packages (from pke==2.0.0) (1.3.4)\n",
            "Requirement already satisfied: future in /usr/local/lib/python3.8/dist-packages (from pke==2.0.0) (0.16.0)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.8/dist-packages (from pke==2.0.0) (1.1.0)\n",
            "Requirement already satisfied: spacy>=3.2.3 in /usr/local/lib/python3.8/dist-packages (from pke==2.0.0) (3.4.4)\n",
            "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.8/dist-packages (from spacy>=3.2.3->pke==2.0.0) (2.25.1)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.8/dist-packages (from spacy>=3.2.3->pke==2.0.0) (21.3)\n",
            "Requirement already satisfied: thinc<8.2.0,>=8.1.0 in /usr/local/lib/python3.8/dist-packages (from spacy>=3.2.3->pke==2.0.0) (8.1.6)\n",
            "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4 in /usr/local/lib/python3.8/dist-packages (from spacy>=3.2.3->pke==2.0.0) (1.10.4)\n",
            "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from spacy>=3.2.3->pke==2.0.0) (2.0.7)\n",
            "Requirement already satisfied: wasabi<1.1.0,>=0.9.1 in /usr/local/lib/python3.8/dist-packages (from spacy>=3.2.3->pke==2.0.0) (0.10.1)\n",
            "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from spacy>=3.2.3->pke==2.0.0) (3.0.8)\n",
            "Requirement already satisfied: pathy>=0.3.5 in /usr/local/lib/python3.8/dist-packages (from spacy>=3.2.3->pke==2.0.0) (0.10.1)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.8/dist-packages (from spacy>=3.2.3->pke==2.0.0) (57.4.0)\n",
            "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /usr/local/lib/python3.8/dist-packages (from spacy>=3.2.3->pke==2.0.0) (2.0.8)\n",
            "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.10 in /usr/local/lib/python3.8/dist-packages (from spacy>=3.2.3->pke==2.0.0) (3.0.11)\n",
            "Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /usr/local/lib/python3.8/dist-packages (from spacy>=3.2.3->pke==2.0.0) (2.4.5)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from spacy>=3.2.3->pke==2.0.0) (2.11.3)\n",
            "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /usr/local/lib/python3.8/dist-packages (from spacy>=3.2.3->pke==2.0.0) (3.3.0)\n",
            "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.8/dist-packages (from spacy>=3.2.3->pke==2.0.0) (1.0.9)\n",
            "Requirement already satisfied: typer<0.8.0,>=0.3.0 in /usr/local/lib/python3.8/dist-packages (from spacy>=3.2.3->pke==2.0.0) (0.7.0)\n",
            "Requirement already satisfied: smart-open<7.0.0,>=5.2.1 in /usr/local/lib/python3.8/dist-packages (from spacy>=3.2.3->pke==2.0.0) (6.3.0)\n",
            "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /usr/local/lib/python3.8/dist-packages (from spacy>=3.2.3->pke==2.0.0) (1.0.4)\n",
            "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.8/dist-packages (from spacy>=3.2.3->pke==2.0.0) (4.64.1)\n",
            "Requirement already satisfied: regex>=2021.8.3 in /usr/local/lib/python3.8/dist-packages (from nltk->pke==2.0.0) (2022.6.2)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.8/dist-packages (from nltk->pke==2.0.0) (7.1.2)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.8/dist-packages (from scikit-learn->pke==2.0.0) (3.1.0)\n",
            "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging>=20.0->spacy>=3.2.3->pke==2.0.0) (3.0.9)\n",
            "Requirement already satisfied: typing-extensions>=4.2.0 in /usr/local/lib/python3.8/dist-packages (from pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4->spacy>=3.2.3->pke==2.0.0) (4.4.0)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests<3.0.0,>=2.13.0->spacy>=3.2.3->pke==2.0.0) (2022.12.7)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests<3.0.0,>=2.13.0->spacy>=3.2.3->pke==2.0.0) (1.24.3)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests<3.0.0,>=2.13.0->spacy>=3.2.3->pke==2.0.0) (2.10)\n",
            "Requirement already satisfied: chardet<5,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from requests<3.0.0,>=2.13.0->spacy>=3.2.3->pke==2.0.0) (4.0.0)\n",
            "Requirement already satisfied: blis<0.8.0,>=0.7.8 in /usr/local/lib/python3.8/dist-packages (from thinc<8.2.0,>=8.1.0->spacy>=3.2.3->pke==2.0.0) (0.7.9)\n",
            "Requirement already satisfied: confection<1.0.0,>=0.0.1 in /usr/local/lib/python3.8/dist-packages (from thinc<8.2.0,>=8.1.0->spacy>=3.2.3->pke==2.0.0) (0.0.3)\n",
            "Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.8/dist-packages (from jinja2->spacy>=3.2.3->pke==2.0.0) (2.0.1)\n",
            "Building wheels for collected packages: pke\n",
            "  Building wheel for pke (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for pke: filename=pke-2.0.0-py3-none-any.whl size=6160288 sha256=e76830f3e8d4958980b63180080e0b0c8233eaff8c0eee775836c91591b7cbfb\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-kmd0ncsa/wheels/29/30/7c/66e4c0828efc7caa6e0369904987d3687d2ebe0ab404367fa1\n",
            "Successfully built pke\n",
            "Installing collected packages: pke\n",
            "Successfully installed pke-2.0.0\n",
            "/usr/lib/python3.8/runpy.py:127: RuntimeWarning: 'nltk.downloader' found in sys.modules after import of package 'nltk', but prior to execution of 'nltk.downloader'; this may result in unpredictable behaviour\n",
            "  warn(RuntimeWarning(msg))\n",
            "[nltk_data] Downloading package universal_tagset to /root/nltk_data...\n",
            "[nltk_data]   Unzipping taggers/universal_tagset.zip.\n",
            "2023-01-26 20:18:24.705429: E tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected\n",
            "\u001b[38;5;3m⚠ As of spaCy v3.0, shortcuts like 'en' are deprecated. Please use the\n",
            "full pipeline package name 'en_core_web_sm' instead.\u001b[0m\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting en-core-web-sm==3.4.1\n",
            "  Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.4.1/en_core_web_sm-3.4.1-py3-none-any.whl (12.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m12.8/12.8 MB\u001b[0m \u001b[31m43.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: spacy<3.5.0,>=3.4.0 in /usr/local/lib/python3.8/dist-packages (from en-core-web-sm==3.4.1) (3.4.4)\n",
            "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /usr/local/lib/python3.8/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (1.0.4)\n",
            "Requirement already satisfied: wasabi<1.1.0,>=0.9.1 in /usr/local/lib/python3.8/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (0.10.1)\n",
            "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (2.0.7)\n",
            "Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /usr/local/lib/python3.8/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (2.4.5)\n",
            "Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.8/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (1.21.5)\n",
            "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /usr/local/lib/python3.8/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (2.0.8)\n",
            "Requirement already satisfied: smart-open<7.0.0,>=5.2.1 in /usr/local/lib/python3.8/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (6.3.0)\n",
            "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4 in /usr/local/lib/python3.8/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (1.10.4)\n",
            "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.8/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (1.0.9)\n",
            "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (3.0.8)\n",
            "Requirement already satisfied: typer<0.8.0,>=0.3.0 in /usr/local/lib/python3.8/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (0.7.0)\n",
            "Requirement already satisfied: thinc<8.2.0,>=8.1.0 in /usr/local/lib/python3.8/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (8.1.6)\n",
            "Requirement already satisfied: pathy>=0.3.5 in /usr/local/lib/python3.8/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (0.10.1)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.8/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (57.4.0)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.8/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (21.3)\n",
            "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /usr/local/lib/python3.8/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (3.3.0)\n",
            "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.8/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (4.64.1)\n",
            "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.10 in /usr/local/lib/python3.8/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (3.0.11)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (2.11.3)\n",
            "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.8/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (2.25.1)\n",
            "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging>=20.0->spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (3.0.9)\n",
            "Requirement already satisfied: typing-extensions>=4.2.0 in /usr/local/lib/python3.8/dist-packages (from pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4->spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (4.4.0)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (2.10)\n",
            "Requirement already satisfied: chardet<5,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (4.0.0)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (2022.12.7)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (1.24.3)\n",
            "Requirement already satisfied: confection<1.0.0,>=0.0.1 in /usr/local/lib/python3.8/dist-packages (from thinc<8.2.0,>=8.1.0->spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (0.0.3)\n",
            "Requirement already satisfied: blis<0.8.0,>=0.7.8 in /usr/local/lib/python3.8/dist-packages (from thinc<8.2.0,>=8.1.0->spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (0.7.9)\n",
            "Requirement already satisfied: click<9.0.0,>=7.1.1 in /usr/local/lib/python3.8/dist-packages (from typer<0.8.0,>=0.3.0->spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (7.1.2)\n",
            "Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.8/dist-packages (from jinja2->spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (2.0.1)\n",
            "\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n",
            "You can now load the package via spacy.load('en_core_web_sm')\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!wget https://github.com/explosion/sense2vec/releases/download/v1.0.0/s2v_reddit_2015_md.tar.gz\n",
        "!tar -xvf  s2v_reddit_2015_md.tar.gz\n",
        "!ls s2v_old"
      ],
      "metadata": {
        "id": "f7LGDGuSLr6P",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2358e059-1530-498d-fd4b-953651ea5ea5"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2023-01-26 20:18:32--  https://github.com/explosion/sense2vec/releases/download/v1.0.0/s2v_reddit_2015_md.tar.gz\n",
            "Resolving github.com (github.com)... 140.82.113.3\n",
            "Connecting to github.com (github.com)|140.82.113.3|:443... connected.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/50261113/52126080-0993-11ea-8190-8f0e295df22a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230126%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230126T201832Z&X-Amz-Expires=300&X-Amz-Signature=3126475ef10822351a639ade56518aa0d254da22bab870ef07316a4b35d0507f&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=50261113&response-content-disposition=attachment%3B%20filename%3Ds2v_reddit_2015_md.tar.gz&response-content-type=application%2Foctet-stream [following]\n",
            "--2023-01-26 20:18:32--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/50261113/52126080-0993-11ea-8190-8f0e295df22a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230126%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230126T201832Z&X-Amz-Expires=300&X-Amz-Signature=3126475ef10822351a639ade56518aa0d254da22bab870ef07316a4b35d0507f&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=50261113&response-content-disposition=attachment%3B%20filename%3Ds2v_reddit_2015_md.tar.gz&response-content-type=application%2Foctet-stream\n",
            "Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 600444501 (573M) [application/octet-stream]\n",
            "Saving to: ‘s2v_reddit_2015_md.tar.gz’\n",
            "\n",
            "s2v_reddit_2015_md. 100%[===================>] 572.63M  29.3MB/s    in 15s     \n",
            "\n",
            "2023-01-26 20:18:48 (38.8 MB/s) - ‘s2v_reddit_2015_md.tar.gz’ saved [600444501/600444501]\n",
            "\n",
            "./._s2v_old\n",
            "./s2v_old/\n",
            "./s2v_old/._freqs.json\n",
            "./s2v_old/freqs.json\n",
            "./s2v_old/._vectors\n",
            "./s2v_old/vectors\n",
            "./s2v_old/._cfg\n",
            "./s2v_old/cfg\n",
            "./s2v_old/._strings.json\n",
            "./s2v_old/strings.json\n",
            "./s2v_old/._key2row\n",
            "./s2v_old/key2row\n",
            "cfg  freqs.json  key2row  strings.json\tvectors\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from pprint import pprint\n",
        "from Questgen import main\n",
        "\n",
        "qg = main.QGen()"
      ],
      "metadata": {
        "id": "PKFDy44gMBFb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000,
          "referenced_widgets": [
            "1ba6ea1efa3148239384b6b5f54f9df4",
            "cb2e440e4e5241b4a810177f743787e7",
            "d3337b8c7ed74e84a2114df9c71367bc",
            "b0bb8ef9eb7e410abd8b52753c8a0b09",
            "7f9589a429ff41179703e2e46197d15c",
            "9e0cbcacf3a241c6b1511a1799aa3e8c",
            "e2afd7fe9d084eb3a74d7f3db98c8825",
            "f40bc11951a545eb81379936dbbaf863",
            "ce5a0a3418d14c35b069b12545794dcf",
            "782f7445a27e4ed79eeb8c87204b56d7",
            "4478e47d1a3c469c90092c14d806017a",
            "6269235f41634155b2b32281683fc639",
            "bac6fff50ed34379b2ce74b0d0dfaef5",
            "6a2d6e4a1f6a4b279a80e98328684559",
            "a4db74c81696427fbb02b77369ba61f1",
            "6e9d060a1f6244e089fcced2fb103198",
            "ec8149804e7440b984f65def17a761a6",
            "20605f529c92411a8c20f3607bc52eeb",
            "8247304ef3f24d75a84d2e38ac3f1452",
            "3df2b878468d4e9ba502e3077a518aa9",
            "4f77d89b7b4c43e29c79e471e190020e",
            "7b2be4dce7db42bc9c16f62839377ebb",
            "b44fa0f66b184af08a9637a9eb883614",
            "b6d9be97c9a441e1b3bcaf65695a382a",
            "9f4022347d0d4908ae7b82abdc1d1206",
            "6e6edbfcf3d248bb94934446c96577fa",
            "990d6e19eace4eacbbcbf5ba8c80136b",
            "7b40f5dfe0894c1ea3f05c7e38b13d77",
            "b95f632f78da4e6891372feaa5401c43",
            "2f5f190ee03546e9aed0667d8480177c",
            "3a339437b12645758102ce935279154a",
            "98443f41dcbf4284a0dc2e0f7fd6de06",
            "2a21d2a55dc4427789bc31976ceb8ed7"
          ]
        },
        "outputId": "d85f8dae-f64e-4fc3-d63c-6fb34b334082"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package brown to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/brown.zip.\n",
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/stopwords.zip.\n",
            "[nltk_data] Downloading collection 'popular'\n",
            "[nltk_data]    | \n",
            "[nltk_data]    | Downloading package cmudict to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/cmudict.zip.\n",
            "[nltk_data]    | Downloading package gazetteers to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/gazetteers.zip.\n",
            "[nltk_data]    | Downloading package genesis to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/genesis.zip.\n",
            "[nltk_data]    | Downloading package gutenberg to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/gutenberg.zip.\n",
            "[nltk_data]    | Downloading package inaugural to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/inaugural.zip.\n",
            "[nltk_data]    | Downloading package movie_reviews to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/movie_reviews.zip.\n",
            "[nltk_data]    | Downloading package names to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/names.zip.\n",
            "[nltk_data]    | Downloading package shakespeare to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/shakespeare.zip.\n",
            "[nltk_data]    | Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]    |   Package stopwords is already up-to-date!\n",
            "[nltk_data]    | Downloading package treebank to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/treebank.zip.\n",
            "[nltk_data]    | Downloading package twitter_samples to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/twitter_samples.zip.\n",
            "[nltk_data]    | Downloading package omw to /root/nltk_data...\n",
            "[nltk_data]    | Downloading package omw-1.4 to /root/nltk_data...\n",
            "[nltk_data]    | Downloading package wordnet to /root/nltk_data...\n",
            "[nltk_data]    | Downloading package wordnet2021 to /root/nltk_data...\n",
            "[nltk_data]    | Downloading package wordnet31 to /root/nltk_data...\n",
            "[nltk_data]    | Downloading package wordnet_ic to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/wordnet_ic.zip.\n",
            "[nltk_data]    | Downloading package words to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/words.zip.\n",
            "[nltk_data]    | Downloading package maxent_ne_chunker to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping chunkers/maxent_ne_chunker.zip.\n",
            "[nltk_data]    | Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping tokenizers/punkt.zip.\n",
            "[nltk_data]    | Downloading package snowball_data to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    | Downloading package averaged_perceptron_tagger to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping taggers/averaged_perceptron_tagger.zip.\n",
            "[nltk_data]    | \n",
            "[nltk_data]  Done downloading collection popular\n",
            "[nltk_data] Downloading package brown to /root/nltk_data...\n",
            "[nltk_data]   Package brown is already up-to-date!\n",
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Package stopwords is already up-to-date!\n",
            "[nltk_data] Downloading collection 'popular'\n",
            "[nltk_data]    | \n",
            "[nltk_data]    | Downloading package cmudict to /root/nltk_data...\n",
            "[nltk_data]    |   Package cmudict is already up-to-date!\n",
            "[nltk_data]    | Downloading package gazetteers to /root/nltk_data...\n",
            "[nltk_data]    |   Package gazetteers is already up-to-date!\n",
            "[nltk_data]    | Downloading package genesis to /root/nltk_data...\n",
            "[nltk_data]    |   Package genesis is already up-to-date!\n",
            "[nltk_data]    | Downloading package gutenberg to /root/nltk_data...\n",
            "[nltk_data]    |   Package gutenberg is already up-to-date!\n",
            "[nltk_data]    | Downloading package inaugural to /root/nltk_data...\n",
            "[nltk_data]    |   Package inaugural is already up-to-date!\n",
            "[nltk_data]    | Downloading package movie_reviews to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Package movie_reviews is already up-to-date!\n",
            "[nltk_data]    | Downloading package names to /root/nltk_data...\n",
            "[nltk_data]    |   Package names is already up-to-date!\n",
            "[nltk_data]    | Downloading package shakespeare to /root/nltk_data...\n",
            "[nltk_data]    |   Package shakespeare is already up-to-date!\n",
            "[nltk_data]    | Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]    |   Package stopwords is already up-to-date!\n",
            "[nltk_data]    | Downloading package treebank to /root/nltk_data...\n",
            "[nltk_data]    |   Package treebank is already up-to-date!\n",
            "[nltk_data]    | Downloading package twitter_samples to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Package twitter_samples is already up-to-date!\n",
            "[nltk_data]    | Downloading package omw to /root/nltk_data...\n",
            "[nltk_data]    |   Package omw is already up-to-date!\n",
            "[nltk_data]    | Downloading package omw-1.4 to /root/nltk_data...\n",
            "[nltk_data]    |   Package omw-1.4 is already up-to-date!\n",
            "[nltk_data]    | Downloading package wordnet to /root/nltk_data...\n",
            "[nltk_data]    |   Package wordnet is already up-to-date!\n",
            "[nltk_data]    | Downloading package wordnet2021 to /root/nltk_data...\n",
            "[nltk_data]    |   Package wordnet2021 is already up-to-date!\n",
            "[nltk_data]    | Downloading package wordnet31 to /root/nltk_data...\n",
            "[nltk_data]    |   Package wordnet31 is already up-to-date!\n",
            "[nltk_data]    | Downloading package wordnet_ic to /root/nltk_data...\n",
            "[nltk_data]    |   Package wordnet_ic is already up-to-date!\n",
            "[nltk_data]    | Downloading package words to /root/nltk_data...\n",
            "[nltk_data]    |   Package words is already up-to-date!\n",
            "[nltk_data]    | Downloading package maxent_ne_chunker to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Package maxent_ne_chunker is already up-to-date!\n",
            "[nltk_data]    | Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]    |   Package punkt is already up-to-date!\n",
            "[nltk_data]    | Downloading package snowball_data to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Package snowball_data is already up-to-date!\n",
            "[nltk_data]    | Downloading package averaged_perceptron_tagger to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Package averaged_perceptron_tagger is already up-\n",
            "[nltk_data]    |       to-date!\n",
            "[nltk_data]    | \n",
            "[nltk_data]  Done downloading collection popular\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/792k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "1ba6ea1efa3148239384b6b5f54f9df4"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/1.21k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "6269235f41634155b2b32281683fc639"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/892M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "b44fa0f66b184af08a9637a9eb883614"
            }
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "payload = {\n",
        "    \"input_text\": \"\"\"Beyoncé later achieved critical acclaim for her sonically experimental visual albums, Beyoncé (2013) and Lemonade (2016), the latter of which was the world's best-selling album of 2016 and the most acclaimed album of her career, exploring themes of infidelity, feminism, and womanism. In 2018, she released Everything Is Love, a collaborative album with her husband, Jay-Z, as the Carters. As a featured artist, Beyoncé topped the Billboard Hot 100 with the remixes of \"Perfect\" by Ed Sheeran in 2017 and \"Savage\" by Megan Thee Stallion in 2020. The same year, she released the musical film Black Is King with an accompanying visual album, with praise from critics. In 2022, Beyoncé received further critical acclaim for her seventh studio album Renaissance, which experimented with disco and house music and paid homage to LGBTQ+ ball culture. She obtained her first solo number-one since 2008 with the album's lead single, \"Break My Soul\", with Renaissance being her first solo studio album since 2016.\"\"\"\n",
        "}\n",
        "output = qg.predict_shortq(payload)\n",
        "pprint (output)"
      ],
      "metadata": {
        "id": "bt3_Q_uVLwIu",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f12a1eea-9953-48ab-9709-b9660561e4a0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Running model for generation\n",
            "{'questions': [{'Question': \"Which is the most acclaimed album of Beyonce's career?\", 'Answer': 'beyoncé', 'id': 1, 'context': \"Beyoncé later achieved critical acclaim for her sonically experimental visual albums, Beyoncé (2013) and Lemonade (2016), the latter of which was the world's best-selling album of 2016 and the most acclaimed album of her career, exploring themes of infidelity, feminism, and womanism. Beyoncé later achieved critical acclaim for her sonically experimental visual albums, Beyoncé (2013) and Lemonade (2016), the latter of which was the world's best-selling album of 2016 and the most acclaimed album of her career, exploring themes of infidelity, feminism, and womanism. In 2022, Beyoncé received further critical acclaim for her seventh studio album Renaissance, which experimented with disco and house music and paid homage to LGBTQ+ ball culture.\"}, {'Question': \"What is the theme of Beyonce's Lemonade album?\", 'Answer': 'feminism', 'id': 2, 'context': \"Beyoncé later achieved critical acclaim for her sonically experimental visual albums, Beyoncé (2013) and Lemonade (2016), the latter of which was the world's best-selling album of 2016 and the most acclaimed album of her career, exploring themes of infidelity, feminism, and womanism.\"}, {'Question': 'Who is the husband of Jay Z?', 'Answer': 'jay-z', 'id': 3, 'context': 'In 2018, she released Everything Is Love, a collaborative album with her husband, Jay-Z, as the Carters.'}, {'Question': \"What is the most acclaimed album of Beyonce's career?\", 'Answer': 'albums', 'id': 4, 'context': \"Beyoncé later achieved critical acclaim for her sonically experimental visual albums, Beyoncé (2013) and Lemonade (2016), the latter of which was the world's best-selling album of 2016 and the most acclaimed album of her career, exploring themes of infidelity, feminism, and womanism.\"}]}\n",
            "{'questions': [{'Answer': 'beyoncé',\n",
            "                'Question': \"Which is the most acclaimed album of Beyonce's \"\n",
            "                            'career?',\n",
            "                'context': 'Beyoncé later achieved critical acclaim for her '\n",
            "                           'sonically experimental visual albums, Beyoncé '\n",
            "                           '(2013) and Lemonade (2016), the latter of which '\n",
            "                           \"was the world's best-selling album of 2016 and the \"\n",
            "                           'most acclaimed album of her career, exploring '\n",
            "                           'themes of infidelity, feminism, and womanism. '\n",
            "                           'Beyoncé later achieved critical acclaim for her '\n",
            "                           'sonically experimental visual albums, Beyoncé '\n",
            "                           '(2013) and Lemonade (2016), the latter of which '\n",
            "                           \"was the world's best-selling album of 2016 and the \"\n",
            "                           'most acclaimed album of her career, exploring '\n",
            "                           'themes of infidelity, feminism, and womanism. In '\n",
            "                           '2022, Beyoncé received further critical acclaim '\n",
            "                           'for her seventh studio album Renaissance, which '\n",
            "                           'experimented with disco and house music and paid '\n",
            "                           'homage to LGBTQ+ ball culture.',\n",
            "                'id': 1},\n",
            "               {'Answer': 'feminism',\n",
            "                'Question': \"What is the theme of Beyonce's Lemonade album?\",\n",
            "                'context': 'Beyoncé later achieved critical acclaim for her '\n",
            "                           'sonically experimental visual albums, Beyoncé '\n",
            "                           '(2013) and Lemonade (2016), the latter of which '\n",
            "                           \"was the world's best-selling album of 2016 and the \"\n",
            "                           'most acclaimed album of her career, exploring '\n",
            "                           'themes of infidelity, feminism, and womanism.',\n",
            "                'id': 2},\n",
            "               {'Answer': 'jay-z',\n",
            "                'Question': 'Who is the husband of Jay Z?',\n",
            "                'context': 'In 2018, she released Everything Is Love, a '\n",
            "                           'collaborative album with her husband, Jay-Z, as '\n",
            "                           'the Carters.',\n",
            "                'id': 3},\n",
            "               {'Answer': 'albums',\n",
            "                'Question': \"What is the most acclaimed album of Beyonce's \"\n",
            "                            'career?',\n",
            "                'context': 'Beyoncé later achieved critical acclaim for her '\n",
            "                           'sonically experimental visual albums, Beyoncé '\n",
            "                           '(2013) and Lemonade (2016), the latter of which '\n",
            "                           \"was the world's best-selling album of 2016 and the \"\n",
            "                           'most acclaimed album of her career, exploring '\n",
            "                           'themes of infidelity, feminism, and womanism.',\n",
            "                'id': 4}],\n",
            " 'statement': 'Beyoncé later achieved critical acclaim for her sonically '\n",
            "              'experimental visual albums, Beyoncé (2013) and Lemonade (2016), '\n",
            "              \"the latter of which was the world's best-selling album of 2016 \"\n",
            "              'and the most acclaimed album of her career, exploring themes of '\n",
            "              'infidelity, feminism, and womanism. In 2018, she released '\n",
            "              'Everything Is Love, a collaborative album with her husband, '\n",
            "              'Jay-Z, as the Carters. As a featured artist, Beyoncé topped the '\n",
            "              'Billboard Hot 100 with the remixes of \"Perfect\" by Ed Sheeran '\n",
            "              'in 2017 and \"Savage\" by Megan Thee Stallion in 2020. The same '\n",
            "              'year, she released the musical film Black Is King with an '\n",
            "              'accompanying visual album, with praise from critics. In 2022, '\n",
            "              'Beyoncé received further critical acclaim for her seventh '\n",
            "              'studio album Renaissance, which experimented with disco and '\n",
            "              'house music and paid homage to LGBTQ+ ball culture. She '\n",
            "              \"obtained her first solo number-one since 2008 with the album's \"\n",
            "              'lead single, \"Break My Soul\", with Renaissance being her first '\n",
            "              'solo studio album since 2016.'}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "TXCtRcEGNQE4"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Fine-tuning"
      ],
      "metadata": {
        "id": "gUtGbPE_7F7T"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "tUrpi5aa7Iw-"
      },
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [
        "X09KpIhKxg5u",
        "IhtknNSOxov1",
        "8ex-yoSVxxAB",
        "Y8iXFe0qzjS4",
        "YNqBAAW30bqv",
        "k9UEcB-e1GLX",
        "umj6FUki1xbX",
        "Q99Cc__W2djn",
        "ZMcLSris3iYo",
        "zQDl8wvpj63h",
        "E-IXE0rNkBGN",
        "O0zouMSUjLPM",
        "TMAE9ftF1S6h",
        "lFDwNSB-20gU",
        "ecsInjmdhs75",
        "YUWiAAYL05A3",
        "dwyy9cNQGAvO",
        "JNT7uxNNjCEs",
        "4-MjbAQYlvsG",
        "O4D8Xx5ipGzd",
        "Alr1OJ8DGNwW",
        "pfJHs6rkjXwy",
        "sAbNo9yu1zWf",
        "KM5jpE-4h4kn",
        "vbESJ9kgIlrC",
        "CX6MgopNJ_BD",
        "o_4cG6nIlVIA",
        "ZEZlBhXBC1LX",
        "I9ggAUvU7Pqg",
        "Lkh_7SBR3ZyH",
        "yIuVn33uuC6m",
        "g4TjuF_s8eDl"
      ],
      "provenance": [],
      "authorship_tag": "ABX9TyPGBL+JZl3Tb1xLEdZoS1Ir",
      "include_colab_link": true
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "1ba6ea1efa3148239384b6b5f54f9df4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_cb2e440e4e5241b4a810177f743787e7",
              "IPY_MODEL_d3337b8c7ed74e84a2114df9c71367bc",
              "IPY_MODEL_b0bb8ef9eb7e410abd8b52753c8a0b09"
            ],
            "layout": "IPY_MODEL_7f9589a429ff41179703e2e46197d15c"
          }
        },
        "cb2e440e4e5241b4a810177f743787e7": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_9e0cbcacf3a241c6b1511a1799aa3e8c",
            "placeholder": "​",
            "style": "IPY_MODEL_e2afd7fe9d084eb3a74d7f3db98c8825",
            "value": "Downloading: 100%"
          }
        },
        "d3337b8c7ed74e84a2114df9c71367bc": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f40bc11951a545eb81379936dbbaf863",
            "max": 791656,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_ce5a0a3418d14c35b069b12545794dcf",
            "value": 791656
          }
        },
        "b0bb8ef9eb7e410abd8b52753c8a0b09": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_782f7445a27e4ed79eeb8c87204b56d7",
            "placeholder": "​",
            "style": "IPY_MODEL_4478e47d1a3c469c90092c14d806017a",
            "value": " 792k/792k [00:00&lt;00:00, 1.38MB/s]"
          }
        },
        "7f9589a429ff41179703e2e46197d15c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9e0cbcacf3a241c6b1511a1799aa3e8c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e2afd7fe9d084eb3a74d7f3db98c8825": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "f40bc11951a545eb81379936dbbaf863": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ce5a0a3418d14c35b069b12545794dcf": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "782f7445a27e4ed79eeb8c87204b56d7": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "4478e47d1a3c469c90092c14d806017a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "6269235f41634155b2b32281683fc639": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_bac6fff50ed34379b2ce74b0d0dfaef5",
              "IPY_MODEL_6a2d6e4a1f6a4b279a80e98328684559",
              "IPY_MODEL_a4db74c81696427fbb02b77369ba61f1"
            ],
            "layout": "IPY_MODEL_6e9d060a1f6244e089fcced2fb103198"
          }
        },
        "bac6fff50ed34379b2ce74b0d0dfaef5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ec8149804e7440b984f65def17a761a6",
            "placeholder": "​",
            "style": "IPY_MODEL_20605f529c92411a8c20f3607bc52eeb",
            "value": "Downloading: 100%"
          }
        },
        "6a2d6e4a1f6a4b279a80e98328684559": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_8247304ef3f24d75a84d2e38ac3f1452",
            "max": 1208,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_3df2b878468d4e9ba502e3077a518aa9",
            "value": 1208
          }
        },
        "a4db74c81696427fbb02b77369ba61f1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_4f77d89b7b4c43e29c79e471e190020e",
            "placeholder": "​",
            "style": "IPY_MODEL_7b2be4dce7db42bc9c16f62839377ebb",
            "value": " 1.21k/1.21k [00:00&lt;00:00, 8.84kB/s]"
          }
        },
        "6e9d060a1f6244e089fcced2fb103198": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ec8149804e7440b984f65def17a761a6": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "20605f529c92411a8c20f3607bc52eeb": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "8247304ef3f24d75a84d2e38ac3f1452": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3df2b878468d4e9ba502e3077a518aa9": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "4f77d89b7b4c43e29c79e471e190020e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7b2be4dce7db42bc9c16f62839377ebb": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "b44fa0f66b184af08a9637a9eb883614": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_b6d9be97c9a441e1b3bcaf65695a382a",
              "IPY_MODEL_9f4022347d0d4908ae7b82abdc1d1206",
              "IPY_MODEL_6e6edbfcf3d248bb94934446c96577fa"
            ],
            "layout": "IPY_MODEL_990d6e19eace4eacbbcbf5ba8c80136b"
          }
        },
        "b6d9be97c9a441e1b3bcaf65695a382a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_7b40f5dfe0894c1ea3f05c7e38b13d77",
            "placeholder": "​",
            "style": "IPY_MODEL_b95f632f78da4e6891372feaa5401c43",
            "value": "Downloading: 100%"
          }
        },
        "9f4022347d0d4908ae7b82abdc1d1206": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_2f5f190ee03546e9aed0667d8480177c",
            "max": 891695056,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_3a339437b12645758102ce935279154a",
            "value": 891695056
          }
        },
        "6e6edbfcf3d248bb94934446c96577fa": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_98443f41dcbf4284a0dc2e0f7fd6de06",
            "placeholder": "​",
            "style": "IPY_MODEL_2a21d2a55dc4427789bc31976ceb8ed7",
            "value": " 892M/892M [00:26&lt;00:00, 42.4MB/s]"
          }
        },
        "990d6e19eace4eacbbcbf5ba8c80136b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7b40f5dfe0894c1ea3f05c7e38b13d77": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b95f632f78da4e6891372feaa5401c43": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "2f5f190ee03546e9aed0667d8480177c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3a339437b12645758102ce935279154a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "98443f41dcbf4284a0dc2e0f7fd6de06": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2a21d2a55dc4427789bc31976ceb8ed7": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}