notebooks/reproduce_results_given_params.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook shows how to reproduce the KNN based discovery results on  [Unsupervised discovery of sign terms by K-nearest neighbours approach (ECCV '20)](https://link.springer.com/chapter/10.1007/978-3-030-66096-3_22  \"Link to paper\").  \n",
    "The Bayesian parameter search step is not performed in this notebook, instead the saved optimum parameters for each cross validation fold are used to re-run the experiments and obtain results. If you also want to do Bayesian parameter search using `scikit-optimize`, refer to [related notebook](./notebooks/run_crossval_exp.ipynb).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "ename": "",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31mFailed to start the Kernel. \n",
      "Kernel pytorch (Python 3.7.6) is not usable. Check the Jupyter output tab for more information. \n",
      "View Jupyter <a href='command:jupyter.viewOutput'>log</a> for further details."
     ]
    }
   ],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "# import matplotlib.pyplot as plt\n",
    "from os.path import join\n",
    "import pandas as pd\n",
    "import os\n",
    "from copy import deepcopy\n",
    "\n",
    "from utils.feature_loaders import load_feats\n",
    "from utils.helper_fncs import load_json\n",
    "from utils.db_utils import get_seq_names\n",
    "from run.pipeline import run_exp\n",
    "from run.optim import run_for_set\n",
    "from utils.feature_loaders import load_feats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**sets**  \n",
    "There are 9 signers in the Phoenix dataset, where the amount of data for each signer is highly imbalanced. Therefore the  \n",
    "signers are grouped into 3 sets (named `A,B,C`) so that each set has similar amount of data. The assignments and the total  \n",
    "number of sequences for each set are given in the cell below. \n",
    "\n",
    "**cross validation scheme**  \n",
    "For each fold, parameter tune is performed on devset and using those optimum parameters, report results on remaining test sets.  \n",
    "Since test sets would contain different signers, this result can be viewed as test on unseen signers. At the end, compute  \n",
    "a weighted average of the results for *dev* and *test* sets, weighted by total number of files in each set. Because the  \n",
    "amount of data is very few, the test results are reported as average of cross validation rather than a single test set. \n",
    "\n",
    "| dev set |test sets|\n",
    "|  :----: | :----: |\n",
    "| A | B,C |  \n",
    "| B | A,C |  \n",
    "| C | A,B |\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'CVroot': '/home/korhan/Desktop/knn_utd/data/CVfolds',\n",
       " 'CVset': 'A',\n",
       " 'clustering': {'cost_thr': 0.01, 'method': 'pairwise', 'olapthr_m': 0.5},\n",
       " 'disc_method': 'knn',\n",
       " 'disc': {'a': 3,\n",
       "  'dim_fix': 6,\n",
       "  'emb_type': 'gauss_kernel',\n",
       "  'k': 150,\n",
       "  'lmax': 15,\n",
       "  'lmin': 2,\n",
       "  'metric': 'L2',\n",
       "  'norm': False,\n",
       "  'olapthr_m': 0.2,\n",
       "  'pca': '',\n",
       "  'r': 0.2,\n",
       "  's': 0.6,\n",
       "  'seg_type': 'uniform',\n",
       "  'top_delta': 0.05,\n",
       "  'use_gpu': True},\n",
       " 'eval': {'tderunfile': '/home/korhan/Desktop/knn_utd/run_tde.sh',\n",
       "  'TDEROOT': '/home/korhan/Desktop/tez/tdev2/tdev2',\n",
       "  'TDESOURCE': '/home/korhan/miniconda3/etc/profile.d/conda.sh',\n",
       "  'njobs': 2,\n",
       "  'dataset': 'phoenix',\n",
       "  'config_file': '/home/korhan/Desktop/knn_utd/config/config_phoenix.json'},\n",
       " 'exp_root': '/home/korhan/Desktop/knn_utd/data/results/',\n",
       " 'feats_root': '/home/korhan/Desktop/data/features/',\n",
       " 'featype': 'c3right',\n",
       " 'coverage': {'patience': 30, 'covth': 10, 'covmargin': 1},\n",
       " 'tune': {'keys': ['r', 's'],\n",
       "  'chkpoint_root': '/home/korhan/Desktop/knn_utd/data/checkpoints',\n",
       "  'minimizer': 'gp',\n",
       "  'n_split': 1,\n",
       "  'stop_margin': 0.01,\n",
       "  'stopper_patience': 5}}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cvsets = set(['A','B','C'])\n",
    "signers_per_set = {'A': [4,8,9], 'B':[2,5,7], 'C':[1,3,6]}\n",
    "nfolder_perset = {'A':1705,'B':1991,'C':1975}\n",
    "\n",
    "alg_type = 'knn'\n",
    "params = load_json(full_path='./config/{}.json'.format(alg_type))\n",
    "params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# disable comment for choosing {openpose/deephand} features\n",
    "# params['featype'] = 'op100'\n",
    "params['featype'] = 'c3_right_PCA40W'\n",
    "\n",
    "params_default = deepcopy(params)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>devset</th>\n",
       "      <th>set</th>\n",
       "      <th>ned</th>\n",
       "      <th>coverage</th>\n",
       "      <th>coverageNS</th>\n",
       "      <th>coverageNS_f</th>\n",
       "      <th>grouping_F</th>\n",
       "      <th>grouping_P</th>\n",
       "      <th>grouping_R</th>\n",
       "      <th>...</th>\n",
       "      <th>lmin</th>\n",
       "      <th>metric</th>\n",
       "      <th>norm</th>\n",
       "      <th>olapthr_m</th>\n",
       "      <th>pca</th>\n",
       "      <th>seg_type</th>\n",
       "      <th>top_delta</th>\n",
       "      <th>use_gpu</th>\n",
       "      <th>r</th>\n",
       "      <th>s</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>A</td>\n",
       "      <td>A</td>\n",
       "      <td>37.59</td>\n",
       "      <td>8.52</td>\n",
       "      <td>9.32</td>\n",
       "      <td>11.62</td>\n",
       "      <td>54.36</td>\n",
       "      <td>53.19</td>\n",
       "      <td>55.68</td>\n",
       "      <td>...</td>\n",
       "      <td>2</td>\n",
       "      <td>L2</td>\n",
       "      <td>False</td>\n",
       "      <td>0.2</td>\n",
       "      <td></td>\n",
       "      <td>uniform</td>\n",
       "      <td>0.05</td>\n",
       "      <td>True</td>\n",
       "      <td>0.2100</td>\n",
       "      <td>0.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>A</td>\n",
       "      <td>B</td>\n",
       "      <td>43.99</td>\n",
       "      <td>9.74</td>\n",
       "      <td>10.45</td>\n",
       "      <td>14.28</td>\n",
       "      <td>53.77</td>\n",
       "      <td>52.14</td>\n",
       "      <td>55.62</td>\n",
       "      <td>...</td>\n",
       "      <td>2</td>\n",
       "      <td>L2</td>\n",
       "      <td>False</td>\n",
       "      <td>0.2</td>\n",
       "      <td></td>\n",
       "      <td>uniform</td>\n",
       "      <td>0.05</td>\n",
       "      <td>True</td>\n",
       "      <td>0.2100</td>\n",
       "      <td>0.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>8</td>\n",
       "      <td>A</td>\n",
       "      <td>C</td>\n",
       "      <td>41.08</td>\n",
       "      <td>8.65</td>\n",
       "      <td>9.83</td>\n",
       "      <td>12.15</td>\n",
       "      <td>49.73</td>\n",
       "      <td>48.19</td>\n",
       "      <td>51.47</td>\n",
       "      <td>...</td>\n",
       "      <td>2</td>\n",
       "      <td>L2</td>\n",
       "      <td>False</td>\n",
       "      <td>0.2</td>\n",
       "      <td></td>\n",
       "      <td>uniform</td>\n",
       "      <td>0.05</td>\n",
       "      <td>True</td>\n",
       "      <td>0.2100</td>\n",
       "      <td>0.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>12</td>\n",
       "      <td>B</td>\n",
       "      <td>A</td>\n",
       "      <td>44.73</td>\n",
       "      <td>9.05</td>\n",
       "      <td>9.92</td>\n",
       "      <td>13.05</td>\n",
       "      <td>50.52</td>\n",
       "      <td>50.34</td>\n",
       "      <td>50.81</td>\n",
       "      <td>...</td>\n",
       "      <td>2</td>\n",
       "      <td>L2</td>\n",
       "      <td>False</td>\n",
       "      <td>0.2</td>\n",
       "      <td></td>\n",
       "      <td>uniform</td>\n",
       "      <td>0.06</td>\n",
       "      <td>True</td>\n",
       "      <td>0.2176</td>\n",
       "      <td>0.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>16</td>\n",
       "      <td>B</td>\n",
       "      <td>B</td>\n",
       "      <td>42.95</td>\n",
       "      <td>8.71</td>\n",
       "      <td>9.34</td>\n",
       "      <td>12.74</td>\n",
       "      <td>55.10</td>\n",
       "      <td>54.19</td>\n",
       "      <td>56.15</td>\n",
       "      <td>...</td>\n",
       "      <td>2</td>\n",
       "      <td>L2</td>\n",
       "      <td>False</td>\n",
       "      <td>0.2</td>\n",
       "      <td></td>\n",
       "      <td>uniform</td>\n",
       "      <td>0.05</td>\n",
       "      <td>True</td>\n",
       "      <td>0.2176</td>\n",
       "      <td>0.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>20</td>\n",
       "      <td>B</td>\n",
       "      <td>C</td>\n",
       "      <td>43.11</td>\n",
       "      <td>8.67</td>\n",
       "      <td>9.88</td>\n",
       "      <td>11.99</td>\n",
       "      <td>46.13</td>\n",
       "      <td>45.38</td>\n",
       "      <td>47.01</td>\n",
       "      <td>...</td>\n",
       "      <td>2</td>\n",
       "      <td>L2</td>\n",
       "      <td>False</td>\n",
       "      <td>0.2</td>\n",
       "      <td></td>\n",
       "      <td>uniform</td>\n",
       "      <td>0.05</td>\n",
       "      <td>True</td>\n",
       "      <td>0.2176</td>\n",
       "      <td>0.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>24</td>\n",
       "      <td>C</td>\n",
       "      <td>A</td>\n",
       "      <td>44.73</td>\n",
       "      <td>9.05</td>\n",
       "      <td>9.92</td>\n",
       "      <td>13.05</td>\n",
       "      <td>50.52</td>\n",
       "      <td>50.34</td>\n",
       "      <td>50.81</td>\n",
       "      <td>...</td>\n",
       "      <td>2</td>\n",
       "      <td>L2</td>\n",
       "      <td>False</td>\n",
       "      <td>0.2</td>\n",
       "      <td></td>\n",
       "      <td>uniform</td>\n",
       "      <td>0.06</td>\n",
       "      <td>True</td>\n",
       "      <td>0.2176</td>\n",
       "      <td>0.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>28</td>\n",
       "      <td>C</td>\n",
       "      <td>B</td>\n",
       "      <td>42.95</td>\n",
       "      <td>8.71</td>\n",
       "      <td>9.34</td>\n",
       "      <td>12.74</td>\n",
       "      <td>55.10</td>\n",
       "      <td>54.19</td>\n",
       "      <td>56.15</td>\n",
       "      <td>...</td>\n",
       "      <td>2</td>\n",
       "      <td>L2</td>\n",
       "      <td>False</td>\n",
       "      <td>0.2</td>\n",
       "      <td></td>\n",
       "      <td>uniform</td>\n",
       "      <td>0.05</td>\n",
       "      <td>True</td>\n",
       "      <td>0.2176</td>\n",
       "      <td>0.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>32</td>\n",
       "      <td>C</td>\n",
       "      <td>C</td>\n",
       "      <td>43.11</td>\n",
       "      <td>8.67</td>\n",
       "      <td>9.88</td>\n",
       "      <td>11.99</td>\n",
       "      <td>46.13</td>\n",
       "      <td>45.38</td>\n",
       "      <td>47.01</td>\n",
       "      <td>...</td>\n",
       "      <td>2</td>\n",
       "      <td>L2</td>\n",
       "      <td>False</td>\n",
       "      <td>0.2</td>\n",
       "      <td></td>\n",
       "      <td>uniform</td>\n",
       "      <td>0.05</td>\n",
       "      <td>True</td>\n",
       "      <td>0.2176</td>\n",
       "      <td>0.6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>9 rows × 38 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    Unnamed: 0 devset set    ned  coverage  coverageNS  coverageNS_f  \\\n",
       "0            0      A   A  37.59      8.52        9.32         11.62   \n",
       "4            4      A   B  43.99      9.74       10.45         14.28   \n",
       "8            8      A   C  41.08      8.65        9.83         12.15   \n",
       "12          12      B   A  44.73      9.05        9.92         13.05   \n",
       "16          16      B   B  42.95      8.71        9.34         12.74   \n",
       "20          20      B   C  43.11      8.67        9.88         11.99   \n",
       "24          24      C   A  44.73      9.05        9.92         13.05   \n",
       "28          28      C   B  42.95      8.71        9.34         12.74   \n",
       "32          32      C   C  43.11      8.67        9.88         11.99   \n",
       "\n",
       "    grouping_F  grouping_P  grouping_R  ...  lmin  metric   norm  olapthr_m  \\\n",
       "0        54.36       53.19       55.68  ...     2      L2  False        0.2   \n",
       "4        53.77       52.14       55.62  ...     2      L2  False        0.2   \n",
       "8        49.73       48.19       51.47  ...     2      L2  False        0.2   \n",
       "12       50.52       50.34       50.81  ...     2      L2  False        0.2   \n",
       "16       55.10       54.19       56.15  ...     2      L2  False        0.2   \n",
       "20       46.13       45.38       47.01  ...     2      L2  False        0.2   \n",
       "24       50.52       50.34       50.81  ...     2      L2  False        0.2   \n",
       "28       55.10       54.19       56.15  ...     2      L2  False        0.2   \n",
       "32       46.13       45.38       47.01  ...     2      L2  False        0.2   \n",
       "\n",
       "    pca  seg_type  top_delta  use_gpu       r    s  \n",
       "0         uniform       0.05     True  0.2100  0.6  \n",
       "4         uniform       0.05     True  0.2100  0.6  \n",
       "8         uniform       0.05     True  0.2100  0.6  \n",
       "12        uniform       0.06     True  0.2176  0.6  \n",
       "16        uniform       0.05     True  0.2176  0.6  \n",
       "20        uniform       0.05     True  0.2176  0.6  \n",
       "24        uniform       0.06     True  0.2176  0.6  \n",
       "28        uniform       0.05     True  0.2176  0.6  \n",
       "32        uniform       0.05     True  0.2176  0.6  \n",
       "\n",
       "[9 rows x 38 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "if params['featype']=='c3_right_PCA40W':\n",
    "    params['exp_var'] = 40\n",
    "    saved_results = pd.read_csv('./data/paper_results/knn_c3_right_PCA40_l6-45_dim_fix_r_s.csv', keep_default_na=False)\n",
    "if params['featype']=='op100':\n",
    "    saved_results = pd.read_csv('./data/paper_results/knn_op100_l6_lmax_dim_fix_r_s.csv', keep_default_na=False)\n",
    "\n",
    "saved_results = saved_results.iloc[::4] # eliminate single signer results\n",
    "saved_results #[report_cols[:-1]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "knn_A_c3_right_PCA40W_a3_dim_fix4_emb_typegauss_kernel_k150_lmax15_lmin2_metricL2_normFalse_olapthr_m0.2_pca_r0.21_s0.6_seg_typeuniform_top_delta0.05_use_gpuTrue\n",
      "0.01\n",
      "Computing Embeddings\n",
      "Building index of size 872529x160\n",
      "Searching index\n",
      "Selecting good pairs\n",
      "*** found 3507 matches ***\n",
      "*** pairwise clustering ***\n",
      "*** post disc completed, found 68 segments from 34 clusters ***\n",
      "/home/korhan/miniconda3/etc/profile.d/conda.sh\n",
      "/home/korhan/Desktop/knn_utd/data/results/knn_A_c3_right_PCA40W_a3_dim_fix4_emb_typegauss_kernel_k150_lmax15_lmin2_metricL2_normFalse_olapthr_m0.2_pca_r0.21_s0.6_seg_typeuniform_top_delta0.05_use_gpuTrue/postpairwise_cost0.01_olap0.5 phoenix sdtw /home/korhan/Desktop/knn_utd/data/results/knn_A_c3_right_PCA40W_a3_dim_fix4_emb_typegauss_kernel_k150_lmax15_lmin2_metricL2_normFalse_olapthr_m0.2_pca_r0.21_s0.6_seg_typeuniform_top_delta0.05_use_gpuTrue/postpairwise_cost0.01_olap0.5/scores.json\n",
      "Reading gold\n",
      "*** Config file read, ovth 50.0 ***\n",
      "/home/korhan/Desktop/knn_utd/config/config_phoenix.json\n",
      "Generating discovered -class- file\n",
      "Reading discovered classes\n",
      "Discovered Class file read\n",
      "\n",
      "2 unique intervals, 0 clusters with 0 nodes found\n",
      "Computing scores..\n",
      "Computing Boundary...\n",
      "Computing Grouping...\n",
      "Number of grouping jobs: 2\n",
      "Computing Token and Type...\n",
      "Computing Coverage...\n",
      "Computing Coverage...\n",
      "Computing Coverage No Single...\n",
      "*** Config file read, ovth 50.0 ***\n",
      "Computing NED...\n",
      "*** Config file read, ovth 50.0 ***\n",
      "scores saved to  /home/korhan/Desktop/knn_utd/data/results/knn_A_c3_right_PCA40W_a3_dim_fix4_emb_typegauss_kernel_k150_lmax15_lmin2_metricL2_normFalse_olapthr_m0.2_pca_r0.21_s0.6_seg_typeuniform_top_delta0.05_use_gpuTrue/postpairwise_cost0.01_olap0.5/scores.json\n",
      "*** Coverage: 0.0100, NED: 100.00\n",
      "trial 0 th:0.01000 cov:0.01000 err:-9.99000\n",
      "0.99\n",
      "*** Matches already discovered !!! ***\n",
      "*** found 3507 matches ***\n",
      "*** pairwise clustering ***\n",
      "*** post disc completed, found 5524 segments from 2762 clusters ***\n",
      "/home/korhan/miniconda3/etc/profile.d/conda.sh\n",
      "/home/korhan/Desktop/knn_utd/data/results/knn_A_c3_right_PCA40W_a3_dim_fix4_emb_typegauss_kernel_k150_lmax15_lmin2_metricL2_normFalse_olapthr_m0.2_pca_r0.21_s0.6_seg_typeuniform_top_delta0.05_use_gpuTrue/postpairwise_cost0.99_olap0.5 phoenix sdtw /home/korhan/Desktop/knn_utd/data/results/knn_A_c3_right_PCA40W_a3_dim_fix4_emb_typegauss_kernel_k150_lmax15_lmin2_metricL2_normFalse_olapthr_m0.2_pca_r0.21_s0.6_seg_typeuniform_top_delta0.05_use_gpuTrue/postpairwise_cost0.99_olap0.5/scores.json\n",
      "Reading gold\n",
      "*** Config file read, ovth 50.0 ***\n",
      "/home/korhan/Desktop/knn_utd/config/config_phoenix.json\n",
      "Generating discovered -class- file\n",
      "Reading discovered classes\n",
      "Discovered Class file read\n",
      "\n",
      "2176 unique intervals, 1145 clusters with 2290 nodes found\n",
      "Computing scores..\n",
      "Computing Boundary...\n",
      "Computing Grouping...\n",
      "Number of grouping jobs: 2\n",
      "Computing Token and Type...\n",
      "Computing Coverage...\n",
      "Computing Coverage...\n",
      "Computing Coverage No Single...\n",
      "*** Config file read, ovth 50.0 ***\n",
      "Computing NED...\n",
      "*** Config file read, ovth 50.0 ***\n",
      "scores saved to  /home/korhan/Desktop/knn_utd/data/results/knn_A_c3_right_PCA40W_a3_dim_fix4_emb_typegauss_kernel_k150_lmax15_lmin2_metricL2_normFalse_olapthr_m0.2_pca_r0.21_s0.6_seg_typeuniform_top_delta0.05_use_gpuTrue/postpairwise_cost0.99_olap0.5/scores.json\n",
      "*** Coverage: 9.3200, NED: 37.59\n",
      "trial 1 th:0.99000 cov:9.32000 err:-0.68000\n",
      "knn_B_c3_right_PCA40W_a3_dim_fix4_emb_typegauss_kernel_k150_lmax15_lmin2_metricL2_normFalse_olapthr_m0.2_pca_r0.21_s0.6_seg_typeuniform_top_delta0.05_use_gpuTrue\n",
      "0.01\n",
      "Computing Embeddings\n",
      "Building index of size 1016793x160\n",
      "Searching index\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m/tmp/ipykernel_14989/930040426.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     15\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     16\u001b[0m     \u001b[0;31m# re-run to get full stats\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 17\u001b[0;31m     \u001b[0mstats\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mrun_for_set\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msetx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mload_feats\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mparams\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0msetx\u001b[0m \u001b[0;32min\u001b[0m \u001b[0msorted\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcvsets\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     18\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     19\u001b[0m     \u001b[0mall_stats\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mdevset\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mstats\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/tmp/ipykernel_14989/930040426.py\u001b[0m in \u001b[0;36m<listcomp>\u001b[0;34m(.0)\u001b[0m\n\u001b[1;32m     15\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     16\u001b[0m     \u001b[0;31m# re-run to get full stats\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 17\u001b[0;31m     \u001b[0mstats\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mrun_for_set\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msetx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mload_feats\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mparams\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0msetx\u001b[0m \u001b[0;32min\u001b[0m \u001b[0msorted\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcvsets\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     18\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     19\u001b[0m     \u001b[0mall_stats\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mdevset\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mstats\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Desktop/knn_utd/runexp/optim.py\u001b[0m in \u001b[0;36mrun_for_set\u001b[0;34m(cvset, feat_loader, params, signer_independent)\u001b[0m\n\u001b[1;32m     22\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     23\u001b[0m         \u001b[0mtmp_feats\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mfeats_dict\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mkey\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mseq_names\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 24\u001b[0;31m         \u001b[0mscores\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mparams\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrun_exp\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtmp_feats\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mparams\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgenname\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     25\u001b[0m         \u001b[0;32mwhile\u001b[0m \u001b[0mscores\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'ned'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m>\u001b[0m \u001b[0;36m99\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     26\u001b[0m             \u001b[0mparams\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'disc'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'top_delta'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;36m0.01\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Desktop/knn_utd/runexp/pipeline.py\u001b[0m in \u001b[0;36mrun_exp\u001b[0;34m(feats_dict, params, genname)\u001b[0m\n\u001b[1;32m    205\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    206\u001b[0m         matches_df, nodes_df, clusters_list, scores, pars = run_fixed_coverage(\n\u001b[0;32m--> 207\u001b[0;31m                                                                 tmp_dict, params)\n\u001b[0m\u001b[1;32m    208\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mexc\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    209\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Desktop/knn_utd/runexp/pipeline.py\u001b[0m in \u001b[0;36mrun_fixed_coverage\u001b[0;34m(feats_dict, pars, covth, covmargin)\u001b[0m\n\u001b[1;32m    159\u001b[0m         \u001b[0mpars\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'clustering'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'cost_thr'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mth\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    160\u001b[0m         \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpars\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'clustering'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'cost_thr'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 161\u001b[0;31m         \u001b[0mmatches_df\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnodes_df\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mclusters_list\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mscores\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdiscovery_pipeline\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfeats_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpars\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    162\u001b[0m         \u001b[0mnmatch\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmatches_df\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    163\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Desktop/knn_utd/runexp/pipeline.py\u001b[0m in \u001b[0;36mdiscovery_pipeline\u001b[0;34m(feats_dict, params)\u001b[0m\n\u001b[1;32m     99\u001b[0m     \u001b[0;34m''' combine discovery, clustering and evaluation in a single function '''\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    100\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 101\u001b[0;31m     \u001b[0mmatches_df\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mseq_names\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrun_matches_discovery\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfeats_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mparams\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    102\u001b[0m     \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'*** found {} matches ***'\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmatches_df\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    103\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Desktop/knn_utd/runexp/pipeline.py\u001b[0m in \u001b[0;36mrun_matches_discovery\u001b[0;34m(feats_dict, params)\u001b[0m\n\u001b[1;32m     49\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mparams\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'disc_method'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m'knn'\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     50\u001b[0m             \u001b[0mknndisc\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mKnnDiscovery\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfeats_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mparams\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 51\u001b[0;31m             \u001b[0mmatches_df\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mknndisc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     52\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     53\u001b[0m         \u001b[0;31m# limit max # of matches\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Desktop/knn_utd/knn/discoverer.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m     83\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     84\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 85\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun_disc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     86\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpost_disc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     87\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Desktop/knn_utd/knn/discoverer.py\u001b[0m in \u001b[0;36mrun_disc\u001b[0;34m(self, new_disc_params)\u001b[0m\n\u001b[1;32m     64\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     65\u001b[0m         \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Searching index'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 66\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mD\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mI\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mindex\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msearch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mparams\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'disc'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'k'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     67\u001b[0m         \u001b[0;31m# self.expname = gen_expname(params['disc'])\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     68\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Desktop/knn_utd/knn/faissWrapper.py\u001b[0m in \u001b[0;36msearch\u001b[0;34m(self, X, k, Xq)\u001b[0m\n\u001b[1;32m     57\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mXq\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mXq\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     58\u001b[0m         \u001b[0;31m# search Xq in index\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 59\u001b[0;31m         \u001b[0mD\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mI\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mindex\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msearch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mXq\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mk\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     60\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     61\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mD\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mI\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/pytorch/lib/python3.7/site-packages/faiss/__init__.py\u001b[0m in \u001b[0;36mreplacement_search\u001b[0;34m(self, x, k, D, I)\u001b[0m\n\u001b[1;32m    142\u001b[0m             \u001b[0;32massert\u001b[0m \u001b[0mI\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mk\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    143\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 144\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msearch_c\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mswig_ptr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mk\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mswig_ptr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mD\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mswig_ptr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mI\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    145\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mD\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mI\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    146\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/pytorch/lib/python3.7/site-packages/faiss/swigfaiss.py\u001b[0m in \u001b[0;36msearch\u001b[0;34m(self, n, x, k, distances, labels)\u001b[0m\n\u001b[1;32m   5226\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   5227\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0msearch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mk\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdistances\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlabels\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 5228\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0m_swigfaiss\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mGpuIndex_search\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mk\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdistances\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlabels\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   5229\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   5230\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mcompute_residual\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mresidual\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "# load optiumum parameters for \n",
    "\n",
    "all_stats = dict()\n",
    "\n",
    "for devset in sorted(cvsets):    \n",
    "    \n",
    "    # load related parameters\n",
    "    expinfo = saved_results[saved_results['devset']==devset].iloc[0]\n",
    "    # set params from experiment info\n",
    "    params['CVset'] = devset\n",
    "    for key,value in params['disc'].items():\n",
    "        if key in expinfo.keys():\n",
    "            params['disc'][key] = expinfo[key]\n",
    "\n",
    "    # re-run to get full stats\n",
    "    stats = [run_for_set(setx, load_feats, params) for setx in sorted(cvsets)]\n",
    "    \n",
    "    all_stats[devset] = stats\n",
    "\n",
    "    # re-set params\n",
    "    params = params_default.copy()\n",
    "\n",
    "    # break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ned</th>\n",
       "      <th>length_avg</th>\n",
       "      <th>n_clus</th>\n",
       "      <th>n_node</th>\n",
       "      <th>grouping_P</th>\n",
       "      <th>grouping_R</th>\n",
       "      <th>grouping_F</th>\n",
       "      <th>coverageNS</th>\n",
       "      <th>nfile</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>37.59</td>\n",
       "      <td>12.525163</td>\n",
       "      <td>1145</td>\n",
       "      <td>2290</td>\n",
       "      <td>53.19</td>\n",
       "      <td>55.68</td>\n",
       "      <td>54.36</td>\n",
       "      <td>9.32</td>\n",
       "      <td>1705</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>43.99</td>\n",
       "      <td>14.188693</td>\n",
       "      <td>1651</td>\n",
       "      <td>3302</td>\n",
       "      <td>52.14</td>\n",
       "      <td>55.62</td>\n",
       "      <td>53.77</td>\n",
       "      <td>10.45</td>\n",
       "      <td>1991</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>41.08</td>\n",
       "      <td>13.312050</td>\n",
       "      <td>1224</td>\n",
       "      <td>2448</td>\n",
       "      <td>48.19</td>\n",
       "      <td>51.47</td>\n",
       "      <td>49.73</td>\n",
       "      <td>9.83</td>\n",
       "      <td>1975</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>44.73</td>\n",
       "      <td>11.944089</td>\n",
       "      <td>1292</td>\n",
       "      <td>2584</td>\n",
       "      <td>50.34</td>\n",
       "      <td>50.81</td>\n",
       "      <td>50.52</td>\n",
       "      <td>9.92</td>\n",
       "      <td>1705</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>42.95</td>\n",
       "      <td>13.250463</td>\n",
       "      <td>1427</td>\n",
       "      <td>2854</td>\n",
       "      <td>54.19</td>\n",
       "      <td>56.15</td>\n",
       "      <td>55.10</td>\n",
       "      <td>9.34</td>\n",
       "      <td>1991</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>43.11</td>\n",
       "      <td>12.589120</td>\n",
       "      <td>1248</td>\n",
       "      <td>2496</td>\n",
       "      <td>45.38</td>\n",
       "      <td>47.01</td>\n",
       "      <td>46.13</td>\n",
       "      <td>9.88</td>\n",
       "      <td>1975</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>44.73</td>\n",
       "      <td>11.944089</td>\n",
       "      <td>1292</td>\n",
       "      <td>2584</td>\n",
       "      <td>50.34</td>\n",
       "      <td>50.81</td>\n",
       "      <td>50.52</td>\n",
       "      <td>9.92</td>\n",
       "      <td>1705</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>42.95</td>\n",
       "      <td>13.250463</td>\n",
       "      <td>1427</td>\n",
       "      <td>2854</td>\n",
       "      <td>54.19</td>\n",
       "      <td>56.15</td>\n",
       "      <td>55.10</td>\n",
       "      <td>9.34</td>\n",
       "      <td>1991</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>43.11</td>\n",
       "      <td>12.589120</td>\n",
       "      <td>1248</td>\n",
       "      <td>2496</td>\n",
       "      <td>45.38</td>\n",
       "      <td>47.01</td>\n",
       "      <td>46.13</td>\n",
       "      <td>9.88</td>\n",
       "      <td>1975</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     ned  length_avg  n_clus  n_node  grouping_P  grouping_R  grouping_F  \\\n",
       "0  37.59   12.525163    1145    2290       53.19       55.68       54.36   \n",
       "1  43.99   14.188693    1651    3302       52.14       55.62       53.77   \n",
       "2  41.08   13.312050    1224    2448       48.19       51.47       49.73   \n",
       "3  44.73   11.944089    1292    2584       50.34       50.81       50.52   \n",
       "4  42.95   13.250463    1427    2854       54.19       56.15       55.10   \n",
       "5  43.11   12.589120    1248    2496       45.38       47.01       46.13   \n",
       "6  44.73   11.944089    1292    2584       50.34       50.81       50.52   \n",
       "7  42.95   13.250463    1427    2854       54.19       56.15       55.10   \n",
       "8  43.11   12.589120    1248    2496       45.38       47.01       46.13   \n",
       "\n",
       "   coverageNS  nfile  \n",
       "0        9.32   1705  \n",
       "1       10.45   1991  \n",
       "2        9.83   1975  \n",
       "3        9.92   1705  \n",
       "4        9.34   1991  \n",
       "5        9.88   1975  \n",
       "6        9.92   1705  \n",
       "7        9.34   1991  \n",
       "8        9.88   1975  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# type of scores to report results\n",
    "report_cols = ['ned','length_avg', 'n_clus', 'n_node',\n",
    "              'grouping_P', 'grouping_R', 'grouping_F', \n",
    "              'coverageNS']\n",
    "\n",
    "# concat dataframes in 'all_stats' dict               \n",
    "dflist = []\n",
    "for devset, stats in all_stats.items():\n",
    "    dflist.append(pd.DataFrame([{**{'devset':devset},**{'set':key}, **item} \n",
    "                                for sublist in stats for key,item in sublist.items()]))\n",
    "    \n",
    "df = pd.concat(dflist, ignore_index=True)\n",
    "df['nfile'] = df.set.apply(lambda x: nfolder_perset[x])\n",
    "\n",
    "df[report_cols] #.to_csv('../results/cv/{}_{}.csv'.format(params['disc_method'], params['featype']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "seperate the results into two dataframes, for **devset** and **testset** results\n",
    "for obtaining development and test scores. \n",
    "\n",
    "\n",
    "| devset | testset | df |\n",
    "| :----: | :----:  | :----: |\n",
    "| A | A | dev_df |\n",
    "| B | B | dev_df |\n",
    "| C | C | dev_df |\n",
    "| A | B,C | test_df |\n",
    "| B | A,C | test_df |\n",
    "| C | A,B | test_df |\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "devs_ms = []\n",
    "tests_ms = []\n",
    "\n",
    "for devset in ['A','B','C']:\n",
    "\n",
    "    fold_df = df.loc[df.devset.isin([devset])]\n",
    "    dev_df = fold_df.loc[fold_df.set.isin([devset])][report_cols+['set']]\n",
    "    test_df = fold_df.loc[~fold_df.set.isin([devset])][report_cols+['set']]\n",
    "\n",
    "    # DEV\n",
    "    fold = dev_df\n",
    "    fold_ms = fold.loc[[len(x)==1 for x in fold.set]]    \n",
    "    devs_ms.append(fold_ms)\n",
    "    \n",
    "    # TEST\n",
    "    fold = test_df\n",
    "    fold_ms = fold.loc[[len(x)==1 for x in fold.set]]\n",
    "    tests_ms.append(fold_ms)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ned</th>\n",
       "      <th>length_avg</th>\n",
       "      <th>n_clus</th>\n",
       "      <th>n_node</th>\n",
       "      <th>grouping_P</th>\n",
       "      <th>grouping_R</th>\n",
       "      <th>grouping_F</th>\n",
       "      <th>coverageNS</th>\n",
       "      <th>nfile</th>\n",
       "      <th>set</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>37.59</td>\n",
       "      <td>12.525163</td>\n",
       "      <td>1145</td>\n",
       "      <td>2290</td>\n",
       "      <td>53.19</td>\n",
       "      <td>55.68</td>\n",
       "      <td>54.36</td>\n",
       "      <td>9.32</td>\n",
       "      <td>1705</td>\n",
       "      <td>A</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     ned  length_avg  n_clus  n_node  grouping_P  grouping_R  grouping_F  \\\n",
       "0  37.59   12.525163    1145    2290       53.19       55.68       54.36   \n",
       "\n",
       "   coverageNS  nfile set  \n",
       "0        9.32   1705   A  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "devs_ms[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>exp</th>\n",
       "      <th>ned</th>\n",
       "      <th>length_avg</th>\n",
       "      <th>n_clus</th>\n",
       "      <th>n_node</th>\n",
       "      <th>grouping_P</th>\n",
       "      <th>grouping_R</th>\n",
       "      <th>grouping_F</th>\n",
       "      <th>coverageNS</th>\n",
       "      <th>nfile</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Dev MS</td>\n",
       "      <td>48.555274</td>\n",
       "      <td>8.955494</td>\n",
       "      <td>1090.555458</td>\n",
       "      <td>2181.110915</td>\n",
       "      <td>43.677896</td>\n",
       "      <td>39.975927</td>\n",
       "      <td>41.519275</td>\n",
       "      <td>9.612768</td>\n",
       "      <td>1899.441192</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Test MS</td>\n",
       "      <td>50.721803</td>\n",
       "      <td>9.254117</td>\n",
       "      <td>1207.087639</td>\n",
       "      <td>2414.175278</td>\n",
       "      <td>43.052922</td>\n",
       "      <td>39.704803</td>\n",
       "      <td>41.028422</td>\n",
       "      <td>10.051700</td>\n",
       "      <td>1899.441192</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       exp        ned  length_avg       n_clus       n_node  grouping_P  \\\n",
       "0   Dev MS  48.555274    8.955494  1090.555458  2181.110915   43.677896   \n",
       "0  Test MS  50.721803    9.254117  1207.087639  2414.175278   43.052922   \n",
       "\n",
       "   grouping_R  grouping_F  coverageNS        nfile  \n",
       "0   39.975927   41.519275    9.612768  1899.441192  \n",
       "0   39.704803   41.028422   10.051700  1899.441192  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def averege(df_list, report_cols):\n",
    "    ''' computes average scores weighted by number of files '''\n",
    "\n",
    "    fold_df = pd.concat(df_list)\n",
    "    tmp = pd.DataFrame(\n",
    "        ((fold_df[report_cols].values * fold_df.nfile.values[:,None]).sum(0) / fold_df.nfile.sum())[None,:],\n",
    "        columns=report_cols)\n",
    "    return tmp\n",
    "\n",
    "keys = ['Dev MS'  ,'Test MS']\n",
    "\n",
    "final = []\n",
    "\n",
    "for k,df_list in enumerate([devs_ms , tests_ms]):\n",
    "    avg_df = averege(df_list, report_cols)\n",
    "    avg_df['exp'] = keys[k]\n",
    "    \n",
    "    final.append(avg_df)\n",
    "    \n",
    "    # display(avg_df)\n",
    "\n",
    "finaldf = pd.concat(final)[['exp'] + report_cols]\n",
    "\n",
    "finaldf"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "a6c383b80ed2235b8763a392a237e8a8299efed4a3e0c15005348d1d9366f620"
  },
  "kernelspec": {
   "display_name": "Python 3.7.6 64-bit ('pytorch': conda)",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}