Credibility Indicator Variance - Simplified.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Credibility Indicator Variance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Author: Ewen Dai <br>\n",
    "Completed: 28 July 2019 <br>\n",
    "Contact Info: ewendai@berkeley.edu <br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Definitions\n",
    "\n",
    "- Assignment: The specialist assignments that each article goes through. (Ex: Language, Reasoning, Probability, Evidence, etc)\n",
    "- Article: The piece of written work that we are analyzing. Identified by an article number.\n",
    "- Task: Each run of an assignment on an article\n",
    "\n",
    "### Variance Calculation Method\n",
    "\n",
    "#### Variance of a single question of a task\n",
    "Find the count of each provided answer and take the variance of the given numbers. For instance, Q7 has three options. 4 responses indicate the first option, 5 responses indicate the second option, and 1 response indicate the third option. The variance of Q7 would be `np.var([4, 5, 1])`, which is `2.889`.\n",
    "\n",
    "##### Suggested Calculation Method\n",
    "1. One-hot-encode answers.\n",
    "2. Sum each column of the one-hot-encoded data.\n",
    "3. Take the variance of the column sums.\n",
    "\n",
    "#### Variance of a single question of an article\n",
    "The mean of the variances of each task from the article.\n",
    "\n",
    "#### Variance of an article\n",
    "The weighted average of the variances of each question where weights are based on the \"importance\" of each question, which is decided separately. If there are no specified weights, assume questions are weighted equally."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## General Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`numVariance` is designed to find the variance of numerical questions/data. Expected format of numerical data is that they are already one-hot-encoded.\n",
    "\n",
    "Inputs: \n",
    "* `df` = df with columns ['quiz_task_uuid', 'article_number'] and columns of the questions (e.g. ['T1.Q2.A1', 'T1.Q2.A2', 'T1.Q2.A3', 'T1.Q3.A1', 'T1.Q3.A2', 'T1.Q3.A3', 'T1.Q3.A4', 'T1.Q3.A5', ...])\n",
    "* `columns` = list of column names containing question data as strings\n",
    "* `questions` = list of question names (e.g. ['T1.Q2', 'T1.Q3', ...])\n",
    "\n",
    "Output:\n",
    "* dictionary of dictionaries: keys of \"outer\" dictionary are task numbers, keys of \"inner\" dictionary are question numbers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 167,
   "metadata": {},
   "outputs": [],
   "source": [
    "def numVariance(df, columns, questions):\n",
    "    if len(columns) == 0 or len(questions) == 0:\n",
    "        return {}\n",
    "    \n",
    "    articleNums = df[\"article_number\"].unique()\n",
    "    quizTaskUUID = df[\"quiz_task_uuid\"].unique()\n",
    "\n",
    "    dictionary = {}\n",
    "    for i in articleNums:\n",
    "        dictionary[i] = {}\n",
    "        for j in quizTaskUUID:\n",
    "            dfTemp = df[(df[\"article_number\"] == i) & (df[\"quiz_task_uuid\"] == j)]\n",
    "            for q in questions:\n",
    "                if not q in dictionary[i]:\n",
    "                    dictionary[i][q] = []\n",
    "                if not dfTemp.empty:\n",
    "                    tempList = []\n",
    "                    for c in columns:\n",
    "                        if c.startswith(q):\n",
    "                            tempList.append(c)\n",
    "\n",
    "                    counts = []\n",
    "                    for t in tempList:\n",
    "                        counts.append(sum(dfTemp[t]))\n",
    "                dictionary[i][q].append(np.var(counts))\n",
    "        for q in questions:\n",
    "            dictionary[i][q] = np.mean(dictionary[i][q])\n",
    "    return dictionary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`textVariance` is designed to find the variance of text-based questions/data. Expected format of data is that all responses to one question is in one column.\n",
    "\n",
    "Inputs: \n",
    "* `df` = df with columns ['quiz_task_uuid', 'article_number'] and columns of the questions (e.g. ['T1.Q4', 'T1.Q5', 'T1.Q6', 'T1.Q7', 'T1.Q8' ...])\n",
    "* `columns` = list of column names containing question data as strings\n",
    "* `questions` = list of question names. should be the same as `columns` (e.g. ['T1.Q4', 'T1.Q5', ...])\n",
    "\n",
    "Output:\n",
    "* dictionary of dictionaries: keys of \"outer\" dictionary are task numbers, keys of \"inner\" dictionary are question numbers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {},
   "outputs": [],
   "source": [
    "def textVariance(df, questions):\n",
    "    if len(questions) == 0:\n",
    "        return {}\n",
    "    \n",
    "    articleNums = df[\"article_number\"].unique()\n",
    "    quizTaskUUID = df[\"quiz_task_uuid\"].unique()\n",
    "    \n",
    "    dictionary = {}\n",
    "    for i in articleNums:\n",
    "        dictionary[i] = {}\n",
    "        for j in quizTaskUUID:\n",
    "            dfTemp = df[(df[\"article_number\"] == i) & (df[\"quiz_task_uuid\"] == j)]\n",
    "            for q in questions:\n",
    "                if not q in dictionary[i]:\n",
    "                    dictionary[i][q] = []\n",
    "                if not dfTemp.empty:\n",
    "                    dictionary[i][q].append(pd.get_dummies(dfTemp[q]).sum().values)\n",
    "        for q in questions:\n",
    "            dictionary[i][q] = np.mean([np.var(arr) for arr in dictionary[i][q]])\n",
    "    return dictionary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`finalVariance` is designed to find the total variance of a task\n",
    "\n",
    "Assumptions:\n",
    "* Questions are either numerical or text, and cannot be both\n",
    "* Questions are numbered in order, with no questions having no data\n",
    "\n",
    "Inputs: \n",
    "* `articles` = list of article numbers\n",
    "* `numDict` = dictionary of dictionaries containing calculated variances of numerical data\n",
    "* `textDict` = dictionary of dictionaries containing calculated variances of text-based data\n",
    "* `weights` = list of weights for each question. len(weights) must equal the number of questions as provided in `numDict` and `textDict`. `weights` is assumed to correspond to questions, in order. For unspecified weights, use `np.ones(numQuestions).tolist()`\n",
    "* `questions` = list of all questions in this task\n",
    "\n",
    "Output: \n",
    "* dictionary: keys are task numbers and corresponding values are the calculated variances"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 256,
   "metadata": {},
   "outputs": [],
   "source": [
    "def finalVariance(articles, numDict, textDict, weights, questions):\n",
    "    dictionary = {}\n",
    "    for i in articles:\n",
    "        if len(numDict) == 0 and len(textDict) == 0:\n",
    "            return {}\n",
    "        elif len(numDict) == 0:\n",
    "            lengths = [len(v) for v in textDict.values()]\n",
    "            if all(x == lengths[0] for x in lengths): \n",
    "                if lengths[0] == len(weights) and len(weights) == len(questions):\n",
    "                    dictionary[i] = np.mean([w*variance for w, variance in zip(weights, textDict[i].values())])\n",
    "                else:\n",
    "                    raise ValueError(\"Number of weight(s) do not equal the number of question(s) or given weights/questions is incorrect.\")\n",
    "            else:\n",
    "                raise ValueError(\"Given textDict does not have variance for each question for each task\")\n",
    "        elif len(textDict) == 0:\n",
    "            lengths = [len(v) for v in numDict.values()]\n",
    "            if all(x == lengths[0] for x in lengths): \n",
    "                if lengths[0] == len(weights) and len(weights) == len(questions):\n",
    "                    dictionary[i] = np.mean([w*variance for w, variance in zip(weights, numDict[i].values())])\n",
    "                else:\n",
    "                    raise ValueError(\"Number of weight(s) do not equal the number of question(s) or given weights/questions is incorrect.\")\n",
    "            else:\n",
    "                raise ValueError(\"Given numDict does not have variance for each question for each task\")\n",
    "        else: # both len(numDict) and len(textDict) > 0\n",
    "            lengthText = [len(v) for v in textDict.values()]\n",
    "            lengthNum = [len(v) for v in numDict.values()]\n",
    "            if all(x == lengthText[0] for x in lengthText) and all(x == lengthNum[0] for x in lengthNum): \n",
    "                if lengthText[0] + lengthNum[0] == len(weights) and len(weights) == len(questions):\n",
    "                    combinedDict = {}\n",
    "                    # data is pulled from textDict[i] and numDict[i]\n",
    "                    for q in questions:\n",
    "                        if q in textDict[i]:\n",
    "                            # if the variance does not exist, we currently set to 0 for ease of calculation.\n",
    "                            # will change to not considering the nan value in the future.\n",
    "                            if np.isnan(textDict[i][q]):\n",
    "                                textDict[i][q] = 0\n",
    "                            combinedDict[q] = textDict[i][q]\n",
    "                        else:\n",
    "                            if np.isnan(numDict[i][q]):\n",
    "                                numDict[i][q] = 0\n",
    "                            combinedDict[q] = numDict[i][q]\n",
    "                    dictionary[i] = np.mean([w*variance for w, variance in zip(weights, combinedDict.values())])\n",
    "                else:\n",
    "                    raise ValueError(\"Number of weight(s) do not equal the number of question(s). numDict has info on \", \n",
    "                                     lengthNum[0], \" question(s) and textDict has info on \", lengthText[0], \n",
    "                                    \" questions. There are \", len(questions), \n",
    "                                     \" questions total. Alternatively, given weights/questions may be incorrect.\")\n",
    "            else:\n",
    "                raise ValueError(\"Given dictionary/dictionaries do(es) not have variance for each question for each task\")\n",
    "    return dictionary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Argument Relevance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 257,
   "metadata": {},
   "outputs": [],
   "source": [
    "argRel = pd.read_csv('demo1/Demo1ArgRel3-2018-09-01T2239-DataHuntAnswers.csv')\n",
    "argRel = argRel.loc[:, ['quiz_task_uuid', 'article_number', 'quiz_taskrun_uuid', \n",
    "                        'contributor_uuid', 'T1.Q1']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 258,
   "metadata": {},
   "outputs": [],
   "source": [
    "articles = argRel[\"article_number\"].unique()\n",
    "numDict = {}\n",
    "textDict = textVariance(argRelSimp, ['T1.Q1'])\n",
    "weights = np.ones(1).tolist()\n",
    "questions = ['T1.Q1']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 259,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{1721: 0.125, 1712: 0.041666666666666664, 1737: 0.0}"
      ]
     },
     "execution_count": 259,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "argRelVariances = finalVariance(articles, numDict, textDict, weights, questions)\n",
    "argRelVariances"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evidence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 260,
   "metadata": {},
   "outputs": [],
   "source": [
    "evidence = pd.read_csv('demo1/Demo1Evi-2018-09-01T2239-DataHuntAnswers.csv')\n",
    "evidence = evidence.drop(columns=['schema_namespace', 'schema_sha256', 'task_url', \n",
    "                                                    'article_batch_name', 'article_filename', 'article_sha256',\n",
    "                                                    'quiz_taskrun_uuid', 'contributor_uuid', 'created',\n",
    "                                                    'finish_time', 'elapsed_seconds', 'final_queue'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 261,
   "metadata": {},
   "outputs": [],
   "source": [
    "numericaldf = evidence.loc[:, ['quiz_task_uuid', 'article_number', 'T1.Q2.A1', 'T1.Q2.A2', 'T1.Q2.A3', 'T1.Q3.A1', 'T1.Q3.A2', 'T1.Q3.A3', 'T1.Q3.A4', \n",
    "           'T1.Q3.A5', 'T1.Q3.A6', 'T1.Q3.A7', 'T1.Q3.A8', 'T1.Q3.A9']]\n",
    "columns = ['T1.Q2.A1', 'T1.Q2.A2', 'T1.Q2.A3', 'T1.Q3.A1', 'T1.Q3.A2', 'T1.Q3.A3', 'T1.Q3.A4', \n",
    "           'T1.Q3.A5', 'T1.Q3.A6', 'T1.Q3.A7', 'T1.Q3.A8', 'T1.Q3.A9']\n",
    "questions = ['T1.Q2', 'T1.Q3']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 262,
   "metadata": {},
   "outputs": [],
   "source": [
    "numDict = numVariance(numericaldf, columns, questions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 263,
   "metadata": {},
   "outputs": [],
   "source": [
    "textdf = evidence.loc[:, ['quiz_task_uuid', 'article_number', 'T1.Q4', 'T1.Q5', 'T1.Q6', 'T1.Q7', 'T1.Q8', 'T1.Q9', 'T1.Q11', \n",
    "                                   'T1.Q12', 'T1.Q13', 'T1.Q14', 'T1.Q15', 'T1.Q16']]\n",
    "questions = ['T1.Q4', 'T1.Q5', 'T1.Q6', 'T1.Q7', 'T1.Q8', 'T1.Q9', 'T1.Q11', \n",
    "                                   'T1.Q12', 'T1.Q13', 'T1.Q14', 'T1.Q15', 'T1.Q16']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 264,
   "metadata": {},
   "outputs": [],
   "source": [
    "textDict = textVariance(textdf, questions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 265,
   "metadata": {},
   "outputs": [],
   "source": [
    "articles = evidence[\"article_number\"].unique()\n",
    "weights = np.ones(14).tolist()\n",
    "questions = ['T1.Q2', 'T1.Q3', 'T1.Q4', 'T1.Q5', 'T1.Q6', 'T1.Q7', 'T1.Q8', 'T1.Q9',\n",
    "             'T1.Q11', 'T1.Q12', 'T1.Q13', 'T1.Q14', 'T1.Q15', 'T1.Q16']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 266,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{1721: 0.5960055359592397, 1712: 0.7523105893592004, 1737: 0.614108735057809}"
      ]
     },
     "execution_count": 266,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evidenceVariances = finalVariance(articles, numDict, textDict, weights, questions)\n",
    "evidenceVariances"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Probability"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 267,
   "metadata": {},
   "outputs": [],
   "source": [
    "prob = pd.read_csv('demo1/Demo1Prob-2018-09-01T2240-DataHuntAnswers.csv')\n",
    "prob = prob.drop(columns=['schema_namespace', 'schema_sha256', 'task_url', \n",
    "                                                    'article_batch_name', 'article_filename', 'article_sha256',\n",
    "                                                    'quiz_taskrun_uuid', 'contributor_uuid', 'created',\n",
    "                                                    'finish_time', 'elapsed_seconds', 'final_queue'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 268,
   "metadata": {},
   "outputs": [],
   "source": [
    "numericaldf = prob.loc[:, ['quiz_task_uuid', 'article_number', 'T1.Q12.A1', 'T1.Q12.A2', 'T1.Q12.A3', 'T1.Q12.A4']]\n",
    "columns = ['T1.Q12.A1', 'T1.Q12.A2', 'T1.Q12.A3', 'T1.Q12.A4']\n",
    "questions = ['T1.Q12']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 269,
   "metadata": {},
   "outputs": [],
   "source": [
    "numDict = numVariance(numericaldf, columns, questions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 270,
   "metadata": {},
   "outputs": [],
   "source": [
    "textdf = prob.loc[:, ['quiz_task_uuid', 'article_number', 'T1.Q1', 'T1.Q2', 'T1.Q4', 'T1.Q5', \n",
    "                      'T1.Q6', 'T1.Q7', 'T1.Q8', 'T1.Q9', 'T1.Q10', 'T1.Q11', 'T1.Q13', 'T1.Q14']]\n",
    "questions = ['T1.Q1', 'T1.Q2', 'T1.Q4', 'T1.Q5', 'T1.Q6', 'T1.Q7', 'T1.Q8', 'T1.Q9', \n",
    "             'T1.Q10', 'T1.Q11', 'T1.Q13', 'T1.Q14']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 271,
   "metadata": {},
   "outputs": [],
   "source": [
    "textDict = textVariance(textdf, questions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 272,
   "metadata": {},
   "outputs": [],
   "source": [
    "articles = prob[\"article_number\"].unique()\n",
    "weights = np.ones(13).tolist()\n",
    "questions = ['T1.Q1', 'T1.Q2', 'T1.Q4', 'T1.Q5', 'T1.Q6', 'T1.Q7', 'T1.Q8', 'T1.Q9', \n",
    "             'T1.Q10', 'T1.Q11', 'T1.Q12', 'T1.Q13', 'T1.Q14']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 273,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{1737: 0.9082051282051282, 1712: 0.6904700854700855, 1721: 0.9754273504273503}"
      ]
     },
     "execution_count": 273,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "probVariances = finalVariance(articles, numDict, textDict, weights, questions)\n",
    "probVariances"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quote Source"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 274,
   "metadata": {},
   "outputs": [],
   "source": [
    "quosour = pd.read_csv('demo1/Demo1QuoSour-2018-09-01T2240-DataHuntAnswers.csv')\n",
    "quosour = quosour.drop(columns=['schema_namespace', 'schema_sha256', 'task_url', \n",
    "                                                    'article_batch_name', 'article_filename', 'article_sha256',\n",
    "                                                    'quiz_taskrun_uuid', 'contributor_uuid', 'created',\n",
    "                                                    'finish_time', 'elapsed_seconds', 'final_queue'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 275,
   "metadata": {},
   "outputs": [],
   "source": [
    "numericaldf = quosour.loc[:, ['quiz_task_uuid', 'article_number', 'T1.Q1.A1', 'T1.Q1.A2', 'T1.Q1.A3', 'T1.Q1.A4', \n",
    "                              'T1.Q1.A5', 'T1.Q1.A6', 'T1.Q1.A7', 'T1.Q1.A8', 'T1.Q1.A9', 'T1.Q2.A1', 'T1.Q2.A2', \n",
    "                              'T1.Q3.A1', 'T1.Q3.A2', 'T1.Q3.A3', 'T1.Q3.A4', 'T1.Q3.A5']]\n",
    "columns = ['T1.Q1.A1', 'T1.Q1.A2', 'T1.Q1.A3', 'T1.Q1.A4', 'T1.Q1.A5', 'T1.Q1.A6', 'T1.Q1.A7', 'T1.Q1.A8', \n",
    "           'T1.Q1.A9', 'T1.Q2.A1', 'T1.Q2.A2', 'T1.Q3.A1', 'T1.Q3.A2', 'T1.Q3.A3', 'T1.Q3.A4', 'T1.Q3.A5']\n",
    "questions = ['T1.Q1', 'T1.Q2', 'T1.Q3']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 276,
   "metadata": {},
   "outputs": [],
   "source": [
    "numDict = numVariance(numericaldf, columns, questions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 277,
   "metadata": {},
   "outputs": [],
   "source": [
    "textdf = quosour.loc[:, ['quiz_task_uuid', 'article_number', 'T1.Q4']]\n",
    "questions = ['T1.Q4']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 278,
   "metadata": {},
   "outputs": [],
   "source": [
    "textDict = textVariance(textdf, questions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 279,
   "metadata": {},
   "outputs": [],
   "source": [
    "articles = quosour[\"article_number\"].unique()\n",
    "weights = np.ones(4).tolist()\n",
    "questions = ['T1.Q1', 'T1.Q2', 'T1.Q3', 'T1.Q4']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 280,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{1712: 0.7030632716049383, 1721: 0.411820987654321, 1737: 0.22109567901234572}"
      ]
     },
     "execution_count": 280,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "quosourVariances = finalVariance(articles, numDict, textDict, weights, questions)\n",
    "quosourVariances"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reasoning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 281,
   "metadata": {},
   "outputs": [],
   "source": [
    "reas = pd.read_csv('demo1/Demo1Reas-2018-09-01T2241-DataHuntAnswers.csv')\n",
    "reas = reas.drop(columns=['schema_namespace', 'schema_sha256', 'task_url', \n",
    "                                                    'article_batch_name', 'article_filename', 'article_sha256',\n",
    "                                                    'quiz_taskrun_uuid', 'contributor_uuid', 'created',\n",
    "                                                    'finish_time', 'elapsed_seconds', 'final_queue'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 282,
   "metadata": {},
   "outputs": [],
   "source": [
    "numericaldf = reas.loc[:, ['quiz_task_uuid', 'article_number', 'T1.Q1.A1', 'T1.Q1.A2', 'T1.Q1.A3', 'T1.Q1.A4', \n",
    "                              'T1.Q1.A5', 'T1.Q2.A1', 'T1.Q2.A2', 'T1.Q2.A3', 'T1.Q2.A4', 'T1.Q2.A5', 'T1.Q2.A6',\n",
    "                              'T1.Q3.A1', 'T1.Q3.A2', 'T1.Q3.A3', 'T1.Q3.A4', 'T1.Q3.A5', 'T1.Q3.A6', 'T1.Q3.A7',\n",
    "                              'T1.Q6.A1', 'T1.Q6.A2', 'T1.Q6.A3', 'T1.Q6.A4', 'T1.Q6.A5', 'T1.Q6.A6', 'T1.Q6.A7', \n",
    "                              'T1.Q6.A8', 'T1.Q6.A9']]\n",
    "columns = ['T1.Q1.A1', 'T1.Q1.A2', 'T1.Q1.A3', 'T1.Q1.A4', 'T1.Q1.A5', 'T1.Q2.A1', 'T1.Q2.A2', 'T1.Q2.A3', \n",
    "           'T1.Q2.A4', 'T1.Q2.A5', 'T1.Q2.A6', 'T1.Q3.A1', 'T1.Q3.A2', 'T1.Q3.A3', 'T1.Q3.A4', 'T1.Q3.A5', \n",
    "           'T1.Q3.A6', 'T1.Q3.A7', 'T1.Q6.A1', 'T1.Q6.A2', 'T1.Q6.A3', 'T1.Q6.A4', 'T1.Q6.A5', 'T1.Q6.A6', \n",
    "           'T1.Q6.A7', 'T1.Q6.A8', 'T1.Q6.A9']\n",
    "questions = ['T1.Q1', 'T1.Q2', 'T1.Q3', 'T1.Q6']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 283,
   "metadata": {},
   "outputs": [],
   "source": [
    "numDict = numVariance(numericaldf, columns, questions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 284,
   "metadata": {},
   "outputs": [],
   "source": [
    "textdf = reas.loc[:, ['quiz_task_uuid', 'article_number', 'T1.Q4', 'T1.Q5', 'T1.Q7', 'T1.Q8', 'T1.Q9', 'T1.Q10']]\n",
    "questions = ['T1.Q4', 'T1.Q5', 'T1.Q7', 'T1.Q8', 'T1.Q9', 'T1.Q10']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 285,
   "metadata": {},
   "outputs": [],
   "source": [
    "textDict = textVariance(textdf, questions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 286,
   "metadata": {},
   "outputs": [],
   "source": [
    "articles = reas[\"article_number\"].unique()\n",
    "weights = np.ones(10).tolist()\n",
    "questions = ['T1.Q1', 'T1.Q2', 'T1.Q3', 'T1.Q4', 'T1.Q5',  'T1.Q6', 'T1.Q7', 'T1.Q8', 'T1.Q9', 'T1.Q10']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 287,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{1712: 0.6269523389602755}"
      ]
     },
     "execution_count": 287,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reasVariances = finalVariance(articles, numDict, textDict, weights, questions)\n",
    "reasVariances"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Language"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 288,
   "metadata": {},
   "outputs": [],
   "source": [
    "lang = pd.read_csv('demo1/DemoLang-2018-09-01T2240-DataHuntAnswers.csv')\n",
    "lang = lang.drop(columns=['schema_namespace', 'schema_sha256', 'task_url', \n",
    "                                                    'article_batch_name', 'article_filename', 'article_sha256',\n",
    "                                                    'quiz_taskrun_uuid', 'contributor_uuid', 'created',\n",
    "                                                    'finish_time', 'elapsed_seconds', 'final_queue'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 289,
   "metadata": {},
   "outputs": [],
   "source": [
    "numericaldf = lang.loc[:, ['quiz_task_uuid', 'article_number', 'T1.Q1.A1', 'T1.Q1.A2', 'T1.Q1.A3', 'T1.Q1.A4', \n",
    "                           'T1.Q1.A5', 'T1.Q1.A6', 'T1.Q1.A7', 'T1.Q1.A8', 'T1.Q1.A9', 'T1.Q1.A10', 'T1.Q1.A11', \n",
    "                           'T1.Q1.A12', 'T1.Q1.A13', 'T1.Q6.A1', 'T1.Q6.A2', 'T1.Q6.A3', 'T1.Q6.A4', 'T1.Q6.A5', \n",
    "                           'T1.Q6.A6', 'T1.Q6.A7', 'T1.Q6.A8', 'T1.Q12.A1', 'T1.Q12.A2', 'T1.Q12.A3', 'T1.Q12.A4']]\n",
    "columns = ['T1.Q1.A1', 'T1.Q1.A2', 'T1.Q1.A3', 'T1.Q1.A4', 'T1.Q1.A5', 'T1.Q1.A6', 'T1.Q1.A7', 'T1.Q1.A8', \n",
    "           'T1.Q1.A9', 'T1.Q1.A10', 'T1.Q1.A11', 'T1.Q1.A12', 'T1.Q1.A13', 'T1.Q6.A1', 'T1.Q6.A2', 'T1.Q6.A3', \n",
    "           'T1.Q6.A4', 'T1.Q6.A5', 'T1.Q6.A6', 'T1.Q6.A7', 'T1.Q6.A8', 'T1.Q12.A1', 'T1.Q12.A2', 'T1.Q12.A3',\n",
    "           'T1.Q12.A4']\n",
    "questions = ['T1.Q1', 'T1.Q6', 'T1.Q12']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 290,
   "metadata": {},
   "outputs": [],
   "source": [
    "numDict = numVariance(numericaldf, columns, questions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 291,
   "metadata": {},
   "outputs": [],
   "source": [
    "textdf = lang.loc[:, ['quiz_task_uuid', 'article_number', 'T1.Q2', 'T1.Q3', 'T1.Q5', 'T1.Q7', \n",
    "                      'T1.Q9', 'T1.Q10', 'T1.Q11', 'T1.Q13', 'T1.Q15']]\n",
    "questions = ['T1.Q2', 'T1.Q3', 'T1.Q5', 'T1.Q7', 'T1.Q9', 'T1.Q10', 'T1.Q11', 'T1.Q13', 'T1.Q15']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 292,
   "metadata": {},
   "outputs": [],
   "source": [
    "textDict = textVariance(textdf, questions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 293,
   "metadata": {},
   "outputs": [],
   "source": [
    "articles = lang[\"article_number\"].unique()\n",
    "weights = np.ones(12).tolist()\n",
    "questions = ['T1.Q1', 'T1.Q2', 'T1.Q3', 'T1.Q5', 'T1.Q6', 'T1.Q7', 'T1.Q9', 'T1.Q10', 'T1.Q11', 'T1.Q12', 'T1.Q13', 'T1.Q15']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 294,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{1712: 1.4098379829873124, 1737: 1.1954882136678202}"
      ]
     },
     "execution_count": 294,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "langVariances = finalVariance(articles, numDict, textDict, weights, questions)\n",
    "langVariances"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}