02_Data_preparation.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# default_exp data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data preparation\n",
    "\n",
    "> Downloading the data and developing the machinery for feeding it to our models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `fastai` library provides the very flexible mechanism of the DataBlock API. This should generally be our main goto tool when working with data in non-standard formats.\n",
    "\n",
    "As fastai v2 is a new version of the library and we have little experience with it, we decided to first drop down to using the mid-level API. This is to ensure we have full control over data processing and to learn how to write custom transforms (we will need this to run some of the experiments we have planned). Once we complete the deepdive into how data is handled by the `fastai` library, hopefully we will be able to utilize the lessons we learn and use the higher level DataBlock API.\n",
    "\n",
    "Let's download the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2020-03-12 16:17:11--  https://static-content.springer.com/esm/art%3A10.1038%2Fs41598-019-48909-4/MediaObjects/41598_2019_48909_MOESM2_ESM.xlsx\n",
      "Resolving static-content.springer.com (static-content.springer.com)... 151.101.192.95, 151.101.128.95, 151.101.64.95, ...\n",
      "Connecting to static-content.springer.com (static-content.springer.com)|151.101.192.95|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 867711 (847K) [application/octet-stream]\n",
      "Saving to: ‘data/Dominicana.xlsx’\n",
      "\n",
      "data/Dominicana.xls 100%[===================>] 847.37K  2.59MB/s    in 0.3s    \n",
      "\n",
      "2020-03-12 16:17:12 (2.59 MB/s) - ‘data/Dominicana.xlsx’ saved [867711/867711]\n",
      "\n",
      "--2020-03-12 16:17:12--  https://static-content.springer.com/esm/art%3A10.1038%2Fs41598-019-48909-4/MediaObjects/41598_2019_48909_MOESM3_ESM.xlsx\n",
      "Resolving static-content.springer.com (static-content.springer.com)... 151.101.192.95, 151.101.128.95, 151.101.64.95, ...\n",
      "Connecting to static-content.springer.com (static-content.springer.com)|151.101.192.95|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 1753245 (1.7M) [application/octet-stream]\n",
      "Saving to: ‘data/ETP.xlsx’\n",
      "\n",
      "data/ETP.xlsx       100%[===================>]   1.67M  4.08MB/s    in 0.4s    \n",
      "\n",
      "2020-03-12 16:17:12 (4.08 MB/s) - ‘data/ETP.xlsx’ saved [1753245/1753245]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget https://static-content.springer.com/esm/art%3A10.1038%2Fs41598-019-48909-4/MediaObjects/41598_2019_48909_MOESM2_ESM.xlsx -O data/Dominicana.xlsx\n",
    "!wget https://static-content.springer.com/esm/art%3A10.1038%2Fs41598-019-48909-4/MediaObjects/41598_2019_48909_MOESM3_ESM.xlsx -O data/ETP.xlsx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## First look at the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "from fastai2.data.all import *\n",
    "\n",
    "dominicana = pd.read_excel('data/Dominicana.xlsx')\n",
    "etp = pd.read_excel('data/ETP.xlsx')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And this is what the data looks like. It contains the ICI information (independent variables) as well as labels, such as Coda type or Clan membership."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>codaNUM2018</th>\n",
       "      <th>Date</th>\n",
       "      <th>nClicks</th>\n",
       "      <th>Duration</th>\n",
       "      <th>ICI1</th>\n",
       "      <th>ICI2</th>\n",
       "      <th>ICI3</th>\n",
       "      <th>ICI4</th>\n",
       "      <th>ICI5</th>\n",
       "      <th>ICI6</th>\n",
       "      <th>ICI7</th>\n",
       "      <th>ICI8</th>\n",
       "      <th>ICI9</th>\n",
       "      <th>CodaType</th>\n",
       "      <th>Clan</th>\n",
       "      <th>Unit</th>\n",
       "      <th>UnitNum</th>\n",
       "      <th>IDN</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>2005-03-04 00:00:00</td>\n",
       "      <td>5</td>\n",
       "      <td>1.188</td>\n",
       "      <td>0.293</td>\n",
       "      <td>0.282</td>\n",
       "      <td>0.298</td>\n",
       "      <td>0.315</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5R3</td>\n",
       "      <td>EC1</td>\n",
       "      <td>A</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>2005-03-04 00:00:00</td>\n",
       "      <td>5</td>\n",
       "      <td>1.125</td>\n",
       "      <td>0.287</td>\n",
       "      <td>0.265</td>\n",
       "      <td>0.299</td>\n",
       "      <td>0.274</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5R3</td>\n",
       "      <td>EC1</td>\n",
       "      <td>A</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>2005-03-04 00:00:00</td>\n",
       "      <td>5</td>\n",
       "      <td>1.090</td>\n",
       "      <td>0.264</td>\n",
       "      <td>0.253</td>\n",
       "      <td>0.297</td>\n",
       "      <td>0.276</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5R3</td>\n",
       "      <td>EC1</td>\n",
       "      <td>A</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>2005-03-04 00:00:00</td>\n",
       "      <td>5</td>\n",
       "      <td>1.090</td>\n",
       "      <td>0.269</td>\n",
       "      <td>0.265</td>\n",
       "      <td>0.271</td>\n",
       "      <td>0.285</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5R3</td>\n",
       "      <td>EC1</td>\n",
       "      <td>A</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>2005-03-04 00:00:00</td>\n",
       "      <td>5</td>\n",
       "      <td>1.101</td>\n",
       "      <td>0.273</td>\n",
       "      <td>0.267</td>\n",
       "      <td>0.266</td>\n",
       "      <td>0.295</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5R3</td>\n",
       "      <td>EC1</td>\n",
       "      <td>A</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   codaNUM2018                 Date  nClicks  Duration   ICI1   ICI2   ICI3  \\\n",
       "0            1  2005-03-04 00:00:00        5     1.188  0.293  0.282  0.298   \n",
       "1            2  2005-03-04 00:00:00        5     1.125  0.287  0.265  0.299   \n",
       "2            3  2005-03-04 00:00:00        5     1.090  0.264  0.253  0.297   \n",
       "3            4  2005-03-04 00:00:00        5     1.090  0.269  0.265  0.271   \n",
       "4            5  2005-03-04 00:00:00        5     1.101  0.273  0.267  0.266   \n",
       "\n",
       "    ICI4  ICI5  ICI6  ICI7  ICI8  ICI9 CodaType Clan Unit  UnitNum IDN  \n",
       "0  0.315   0.0   0.0   0.0   0.0   0.0      5R3  EC1    A        1   0  \n",
       "1  0.274   0.0   0.0   0.0   0.0   0.0      5R3  EC1    A        1   0  \n",
       "2  0.276   0.0   0.0   0.0   0.0   0.0      5R3  EC1    A        1   0  \n",
       "3  0.285   0.0   0.0   0.0   0.0   0.0      5R3  EC1    A        1   0  \n",
       "4  0.295   0.0   0.0   0.0   0.0   0.0      5R3  EC1    A        1   0  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dominicana.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CodaNum</th>\n",
       "      <th>FullDateTime</th>\n",
       "      <th>ClanNum</th>\n",
       "      <th>ClanName</th>\n",
       "      <th>NumClicks</th>\n",
       "      <th>TotalDur</th>\n",
       "      <th>ICI1</th>\n",
       "      <th>ICI2</th>\n",
       "      <th>ICI3</th>\n",
       "      <th>ICI4</th>\n",
       "      <th>ICI5</th>\n",
       "      <th>ICI6</th>\n",
       "      <th>ICI7</th>\n",
       "      <th>ICI8</th>\n",
       "      <th>ICI9</th>\n",
       "      <th>ICI10</th>\n",
       "      <th>ICI11</th>\n",
       "      <th>Coda Type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>02/23/1985 12:03:01</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Regular</td>\n",
       "      <td>5</td>\n",
       "      <td>0.531</td>\n",
       "      <td>0.141</td>\n",
       "      <td>0.127</td>\n",
       "      <td>0.126</td>\n",
       "      <td>0.138</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>599.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>02/23/1985 12:03:03</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Regular</td>\n",
       "      <td>5</td>\n",
       "      <td>0.526</td>\n",
       "      <td>0.132</td>\n",
       "      <td>0.128</td>\n",
       "      <td>0.124</td>\n",
       "      <td>0.143</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>599.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>02/23/1985 12:03:06</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Regular</td>\n",
       "      <td>5</td>\n",
       "      <td>0.491</td>\n",
       "      <td>0.134</td>\n",
       "      <td>0.129</td>\n",
       "      <td>0.118</td>\n",
       "      <td>0.109</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>599.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>02/23/1985 12:03:07</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Regular</td>\n",
       "      <td>8</td>\n",
       "      <td>1.411</td>\n",
       "      <td>0.204</td>\n",
       "      <td>0.194</td>\n",
       "      <td>0.218</td>\n",
       "      <td>0.202</td>\n",
       "      <td>0.207</td>\n",
       "      <td>0.206</td>\n",
       "      <td>0.18</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>899.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>02/23/1985 12:03:09</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Regular</td>\n",
       "      <td>5</td>\n",
       "      <td>0.528</td>\n",
       "      <td>0.133</td>\n",
       "      <td>0.129</td>\n",
       "      <td>0.130</td>\n",
       "      <td>0.137</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>599.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   CodaNum         FullDateTime  ClanNum ClanName  NumClicks  TotalDur   ICI1  \\\n",
       "0        1  02/23/1985 12:03:01      1.0  Regular          5     0.531  0.141   \n",
       "1        2  02/23/1985 12:03:03      1.0  Regular          5     0.526  0.132   \n",
       "2        3  02/23/1985 12:03:06      1.0  Regular          5     0.491  0.134   \n",
       "3        4  02/23/1985 12:03:07      1.0  Regular          8     1.411  0.204   \n",
       "4        5  02/23/1985 12:03:09      1.0  Regular          5     0.528  0.133   \n",
       "\n",
       "    ICI2   ICI3   ICI4   ICI5   ICI6  ICI7  ICI8  ICI9  ICI10  ICI11  \\\n",
       "0  0.127  0.126  0.138  0.000  0.000  0.00   0.0   0.0    0.0    0.0   \n",
       "1  0.128  0.124  0.143  0.000  0.000  0.00   0.0   0.0    0.0    0.0   \n",
       "2  0.129  0.118  0.109  0.000  0.000  0.00   0.0   0.0    0.0    0.0   \n",
       "3  0.194  0.218  0.202  0.207  0.206  0.18   0.0   0.0    0.0    0.0   \n",
       "4  0.129  0.130  0.137  0.000  0.000  0.00   0.0   0.0    0.0    0.0   \n",
       "\n",
       "   Coda Type  \n",
       "0      599.0  \n",
       "1      599.0  \n",
       "2      599.0  \n",
       "3      899.0  \n",
       "4      599.0  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "etp.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data for pretraining"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's construct our dataset step by step. A `TfmdLists` object is able to read the rows of our DataFrame and treat them as items (`item` is the name for an example in the `fastai` parlance)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "tfmd_lists = TfmdLists(dominicana, [noop])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "codaNUM2018                      1\n",
       "Date           2005-03-04 00:00:00\n",
       "nClicks                          5\n",
       "Duration                     1.188\n",
       "ICI1                         0.293\n",
       "ICI2                         0.282\n",
       "ICI3                         0.298\n",
       "ICI4                         0.315\n",
       "ICI5                             0\n",
       "ICI6                             0\n",
       "ICI7                             0\n",
       "ICI8                             0\n",
       "ICI9                             0\n",
       "CodaType                       5R3\n",
       "Clan                           EC1\n",
       "Unit                             A\n",
       "UnitNum                          1\n",
       "IDN                              0\n",
       "Name: 0, dtype: object"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfmd_lists[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(8719, 8719)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(tfmd_lists), dominicana.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking good. Let's see if we can extract the ICI information from a single row and package it in a way that would be suitable for our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "def get_independent_vars(row, start_col=4, n_vals=9):\n",
    "    vals = [v for v in row[start_col:(start_col+n_vals)].values if v != 0]\n",
    "    return np.pad(vals, (n_vals - len(vals), 0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.   , 0.   , 0.   , 0.   , 0.   , 0.293, 0.282, 0.298, 0.315])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_independent_vars(dominicana.iloc[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This looks good. Can we use this in `TfmdLists`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "tfmd_lists = TfmdLists(dominicana, [get_independent_vars])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.   , 0.   , 0.   , 0.   , 0.   , 0.293, 0.282, 0.298, 0.315])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfmd_lists[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the pretraining, we can go directly from this representation to the targets (the target being the last ICI)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "def independent_vars_to_targs(ary): return ary[-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "tfmd_lists = TfmdLists(dominicana, [get_independent_vars, independent_vars_to_targs])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.315"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfmd_lists[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now need to make sure that the independent variables, our train data, doesn't contain the targets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "def drop_last_value(ary): return ary[:-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We would like each example to be represented as a tuple of `(independent_variables, targets)`. In order to arrive at this representation, we can run two transformation pipelines in parallel.\n",
    "\n",
    "One transformation pipeline will give us the independent variables:\n",
    "\n",
    "```TfmdLists(dominicana, [get_independent_vars, drop_last_value])```\n",
    "\n",
    "and the other will give us the dependent variable, our target:\n",
    "\n",
    "```TfmdLists(dominicana, [get_independent_vars, independent_vars_to_targs])```\n",
    "\n",
    "The fastai class that can wrap multiple transformation pipelines is called `Datasets`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(#8719) [(array([0.   , 0.   , 0.   , 0.   , 0.   , 0.293, 0.282, 0.298]), 0.315),(array([0.   , 0.   , 0.   , 0.   , 0.   , 0.287, 0.265, 0.299]), 0.274),(array([0.   , 0.   , 0.   , 0.   , 0.   , 0.264, 0.253, 0.297]), 0.276),(array([0.   , 0.   , 0.   , 0.   , 0.   , 0.269, 0.265, 0.271]), 0.285),(array([0.   , 0.   , 0.   , 0.   , 0.   , 0.273, 0.267, 0.266]), 0.295),(array([0.   , 0.   , 0.   , 0.   , 0.   , 0.278, 0.27 , 0.269]), 0.271),(array([0.   , 0.   , 0.   , 0.   , 0.   , 0.269, 0.293, 0.287]), 0.289),(array([0.   , 0.   , 0.   , 0.   , 0.   , 0.279, 0.277, 0.28 ]), 0.294),(array([0.   , 0.   , 0.   , 0.   , 0.   , 0.284, 0.278, 0.28 ]), 0.29),(array([0.   , 0.   , 0.   , 0.   , 0.   , 0.297, 0.277, 0.275]), 0.283)...]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "datasets = Datasets(dominicana, [[get_independent_vars, drop_last_value], [get_independent_vars, independent_vars_to_targs]]); datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([0.   , 0.   , 0.   , 0.   , 0.   , 0.293, 0.282, 0.298]), 0.315)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "datasets[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is looking good. We have the data for pretraining ready. But what about actual training? Here we will need labels transformed in a way suitable for our model to learn from."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data for training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We specify the first pipeline as follows:\n",
    "```TfmdLists(dominicana, [get_independent_vars])```\n",
    "\n",
    "For the second pipeline however, we will need new functionality we have not developed yet. We would like to be able to specify a set of labels as targets (this could be clan membership or coda type for instance)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "def get_target(row, col_name): return row[col_name]\n",
    "get_clan_membership = partialler(get_target, col_name='Clan')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "tfmd_lists = TfmdLists(dominicana, [get_clan_membership])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'EC1'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfmd_lists[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['EC1', 'EC2'], dtype=object)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dominicana.Clan.unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now extract the clan name as a string, but this is not a representation we can train our model on. We need to go from string labels to a set of indexes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorCategory(0)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Categorize(vocab=['EC1', 'EC2'])('EC1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This does the trick!\n",
    "\n",
    "Let's now pull all this into a `Datasets` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(#8719) [(array([0.   , 0.   , 0.   , 0.   , 0.   , 0.293, 0.282, 0.298, 0.315]), TensorCategory(0)),(array([0.   , 0.   , 0.   , 0.   , 0.   , 0.287, 0.265, 0.299, 0.274]), TensorCategory(0)),(array([0.   , 0.   , 0.   , 0.   , 0.   , 0.264, 0.253, 0.297, 0.276]), TensorCategory(0)),(array([0.   , 0.   , 0.   , 0.   , 0.   , 0.269, 0.265, 0.271, 0.285]), TensorCategory(0)),(array([0.   , 0.   , 0.   , 0.   , 0.   , 0.273, 0.267, 0.266, 0.295]), TensorCategory(0)),(array([0.   , 0.   , 0.   , 0.   , 0.   , 0.278, 0.27 , 0.269, 0.271]), TensorCategory(0)),(array([0.   , 0.   , 0.   , 0.   , 0.   , 0.269, 0.293, 0.287, 0.289]), TensorCategory(0)),(array([0.   , 0.   , 0.   , 0.   , 0.   , 0.279, 0.277, 0.28 , 0.294]), TensorCategory(0)),(array([0.   , 0.   , 0.   , 0.   , 0.   , 0.284, 0.278, 0.28 , 0.29 ]), TensorCategory(0)),(array([0.   , 0.   , 0.   , 0.   , 0.   , 0.297, 0.277, 0.275, 0.283]), TensorCategory(0))...]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "datasets = Datasets(dominicana, [[get_independent_vars], [get_clan_membership, Categorize]]); datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `Datasets` class can work with the `Categorize` transform to initialize it without us having to explicitly pass the vocab (it creates the vocab from the data we provide it)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(#2) ['EC1','EC2']"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "datasets.tfms[1][-1].vocab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have everything we need on the data side to reproduce the RNN experiments from the paper. Let us now see if we can use the DataBlock API to nicely package it all up."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using the DataBlock API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the pretraining, we can use all the data we have across the two datasets (the Dominica and Eastern Tropical Pacific (ETP) datasets). Let's concatenate them together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# export\n",
    "pd.set_option('display.max_columns', None)\n",
    "merged_datasets = pd.concat((etp, dominicana)).fillna(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CodaNum</th>\n",
       "      <th>FullDateTime</th>\n",
       "      <th>ClanNum</th>\n",
       "      <th>ClanName</th>\n",
       "      <th>NumClicks</th>\n",
       "      <th>TotalDur</th>\n",
       "      <th>ICI1</th>\n",
       "      <th>ICI2</th>\n",
       "      <th>ICI3</th>\n",
       "      <th>ICI4</th>\n",
       "      <th>ICI5</th>\n",
       "      <th>ICI6</th>\n",
       "      <th>ICI7</th>\n",
       "      <th>ICI8</th>\n",
       "      <th>ICI9</th>\n",
       "      <th>ICI10</th>\n",
       "      <th>ICI11</th>\n",
       "      <th>Coda Type</th>\n",
       "      <th>codaNUM2018</th>\n",
       "      <th>Date</th>\n",
       "      <th>nClicks</th>\n",
       "      <th>Duration</th>\n",
       "      <th>CodaType</th>\n",
       "      <th>Clan</th>\n",
       "      <th>Unit</th>\n",
       "      <th>UnitNum</th>\n",
       "      <th>IDN</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>02/23/1985 12:03:01</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Regular</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.531</td>\n",
       "      <td>0.141</td>\n",
       "      <td>0.127</td>\n",
       "      <td>0.126</td>\n",
       "      <td>0.138</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>599.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.0</td>\n",
       "      <td>02/23/1985 12:03:03</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Regular</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.526</td>\n",
       "      <td>0.132</td>\n",
       "      <td>0.128</td>\n",
       "      <td>0.124</td>\n",
       "      <td>0.143</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>599.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3.0</td>\n",
       "      <td>02/23/1985 12:03:06</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Regular</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.491</td>\n",
       "      <td>0.134</td>\n",
       "      <td>0.129</td>\n",
       "      <td>0.118</td>\n",
       "      <td>0.109</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>599.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4.0</td>\n",
       "      <td>02/23/1985 12:03:07</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Regular</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1.411</td>\n",
       "      <td>0.204</td>\n",
       "      <td>0.194</td>\n",
       "      <td>0.218</td>\n",
       "      <td>0.202</td>\n",
       "      <td>0.207</td>\n",
       "      <td>0.206</td>\n",
       "      <td>0.18</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>899.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5.0</td>\n",
       "      <td>02/23/1985 12:03:09</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Regular</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.528</td>\n",
       "      <td>0.133</td>\n",
       "      <td>0.129</td>\n",
       "      <td>0.130</td>\n",
       "      <td>0.137</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>599.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   CodaNum         FullDateTime  ClanNum ClanName  NumClicks  TotalDur   ICI1  \\\n",
       "0      1.0  02/23/1985 12:03:01      1.0  Regular        5.0     0.531  0.141   \n",
       "1      2.0  02/23/1985 12:03:03      1.0  Regular        5.0     0.526  0.132   \n",
       "2      3.0  02/23/1985 12:03:06      1.0  Regular        5.0     0.491  0.134   \n",
       "3      4.0  02/23/1985 12:03:07      1.0  Regular        8.0     1.411  0.204   \n",
       "4      5.0  02/23/1985 12:03:09      1.0  Regular        5.0     0.528  0.133   \n",
       "\n",
       "    ICI2   ICI3   ICI4   ICI5   ICI6  ICI7  ICI8  ICI9  ICI10  ICI11  \\\n",
       "0  0.127  0.126  0.138  0.000  0.000  0.00   0.0   0.0    0.0    0.0   \n",
       "1  0.128  0.124  0.143  0.000  0.000  0.00   0.0   0.0    0.0    0.0   \n",
       "2  0.129  0.118  0.109  0.000  0.000  0.00   0.0   0.0    0.0    0.0   \n",
       "3  0.194  0.218  0.202  0.207  0.206  0.18   0.0   0.0    0.0    0.0   \n",
       "4  0.129  0.130  0.137  0.000  0.000  0.00   0.0   0.0    0.0    0.0   \n",
       "\n",
       "   Coda Type  codaNUM2018 Date  nClicks  Duration CodaType Clan Unit  UnitNum  \\\n",
       "0      599.0          0.0    0      0.0       0.0        0    0    0      0.0   \n",
       "1      599.0          0.0    0      0.0       0.0        0    0    0      0.0   \n",
       "2      599.0          0.0    0      0.0       0.0        0    0    0      0.0   \n",
       "3      899.0          0.0    0      0.0       0.0        0    0    0      0.0   \n",
       "4      599.0          0.0    0      0.0       0.0        0    0    0      0.0   \n",
       "\n",
       "  IDN  \n",
       "0   0  \n",
       "1   0  \n",
       "2   0  \n",
       "3   0  \n",
       "4   0  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "merged_datasets.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let us craft a DataBlock that will read in the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "get_ETP_independent_vars = partial(get_independent_vars, start_col=5, n_vals=11) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<function fastai2.torch_core.trainable_params(m)>"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainable_params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "dblock_pretrain = DataBlock(\n",
    "    get_x = (get_ETP_independent_vars, drop_last_value),\n",
    "    get_y = (get_ETP_independent_vars, independent_vars_to_targs),\n",
    "    splitter=TrainTestSplitter(test_size=0.1, random_state=42) # having a validation set is crucial for any task,\n",
    ")                                                              # including pretraining!\n",
    "\n",
    "datasets_pretrain = dblock_pretrain.datasets(merged_datasets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(#25714) [(array([0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.531, 0.141, 0.127,\n",
       "       0.126]), 0.138),(array([0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.526, 0.132, 0.128,\n",
       "       0.124]), 0.143),(array([0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.491, 0.134, 0.129,\n",
       "       0.118]), 0.109),(array([0.   , 0.   , 0.   , 1.411, 0.204, 0.194, 0.218, 0.202, 0.207,\n",
       "       0.206]), 0.18),(array([0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.528, 0.133, 0.129,\n",
       "       0.13 ]), 0.137),(array([0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.538, 0.138, 0.133,\n",
       "       0.127]), 0.14),(array([0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.534, 0.131, 0.137,\n",
       "       0.128]), 0.137),(array([0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.549, 0.148, 0.135,\n",
       "       0.127]), 0.139),(array([0.   , 0.   , 0.   , 1.032, 0.148, 0.147, 0.139, 0.142, 0.149,\n",
       "       0.144]), 0.162),(array([0.   , 0.   , 0.   , 0.   , 0.   , 0.709, 0.148, 0.133, 0.132,\n",
       "       0.135]), 0.16)...]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "datasets_pretrain"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is looking good. As for the train data, situation is a bit more complex - we need to align how we create our datasets with the paper.\n",
    "\n",
    "It seems that due to lack of data whale identification was only evaluated on train set. Since this gives us little insights into how the model would generalize to unseen data, let us not include this in our analysis.\n",
    "\n",
    "With regards to the \"coda type classification\" task, the paper reports training on 23 coda types from the Dominicana dataset and 43 coda types from the ETP dataset. The authors were very kind to share their [code on github](https://github.com/dgruber212/Sperm_Whale_Machine_Learning/blob/master/RNNClassifier.py) and we can align how we create our datasets with them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>codaNUM2018</th>\n",
       "      <th>Date</th>\n",
       "      <th>nClicks</th>\n",
       "      <th>Duration</th>\n",
       "      <th>ICI1</th>\n",
       "      <th>ICI2</th>\n",
       "      <th>ICI3</th>\n",
       "      <th>ICI4</th>\n",
       "      <th>ICI5</th>\n",
       "      <th>ICI6</th>\n",
       "      <th>ICI7</th>\n",
       "      <th>ICI8</th>\n",
       "      <th>ICI9</th>\n",
       "      <th>CodaType</th>\n",
       "      <th>Clan</th>\n",
       "      <th>Unit</th>\n",
       "      <th>UnitNum</th>\n",
       "      <th>IDN</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>2005-03-04 00:00:00</td>\n",
       "      <td>5</td>\n",
       "      <td>1.188</td>\n",
       "      <td>0.293</td>\n",
       "      <td>0.282</td>\n",
       "      <td>0.298</td>\n",
       "      <td>0.315</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5R3</td>\n",
       "      <td>EC1</td>\n",
       "      <td>A</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>2005-03-04 00:00:00</td>\n",
       "      <td>5</td>\n",
       "      <td>1.125</td>\n",
       "      <td>0.287</td>\n",
       "      <td>0.265</td>\n",
       "      <td>0.299</td>\n",
       "      <td>0.274</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5R3</td>\n",
       "      <td>EC1</td>\n",
       "      <td>A</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>2005-03-04 00:00:00</td>\n",
       "      <td>5</td>\n",
       "      <td>1.090</td>\n",
       "      <td>0.264</td>\n",
       "      <td>0.253</td>\n",
       "      <td>0.297</td>\n",
       "      <td>0.276</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5R3</td>\n",
       "      <td>EC1</td>\n",
       "      <td>A</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>2005-03-04 00:00:00</td>\n",
       "      <td>5</td>\n",
       "      <td>1.090</td>\n",
       "      <td>0.269</td>\n",
       "      <td>0.265</td>\n",
       "      <td>0.271</td>\n",
       "      <td>0.285</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5R3</td>\n",
       "      <td>EC1</td>\n",
       "      <td>A</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>2005-03-04 00:00:00</td>\n",
       "      <td>5</td>\n",
       "      <td>1.101</td>\n",
       "      <td>0.273</td>\n",
       "      <td>0.267</td>\n",
       "      <td>0.266</td>\n",
       "      <td>0.295</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5R3</td>\n",
       "      <td>EC1</td>\n",
       "      <td>A</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   codaNUM2018                 Date  nClicks  Duration   ICI1   ICI2   ICI3  \\\n",
       "0            1  2005-03-04 00:00:00        5     1.188  0.293  0.282  0.298   \n",
       "1            2  2005-03-04 00:00:00        5     1.125  0.287  0.265  0.299   \n",
       "2            3  2005-03-04 00:00:00        5     1.090  0.264  0.253  0.297   \n",
       "3            4  2005-03-04 00:00:00        5     1.090  0.269  0.265  0.271   \n",
       "4            5  2005-03-04 00:00:00        5     1.101  0.273  0.267  0.266   \n",
       "\n",
       "    ICI4  ICI5  ICI6  ICI7  ICI8  ICI9 CodaType Clan Unit  UnitNum IDN  \n",
       "0  0.315   0.0   0.0   0.0   0.0   0.0      5R3  EC1    A        1   0  \n",
       "1  0.274   0.0   0.0   0.0   0.0   0.0      5R3  EC1    A        1   0  \n",
       "2  0.276   0.0   0.0   0.0   0.0   0.0      5R3  EC1    A        1   0  \n",
       "3  0.285   0.0   0.0   0.0   0.0   0.0      5R3  EC1    A        1   0  \n",
       "4  0.295   0.0   0.0   0.0   0.0   0.0      5R3  EC1    A        1   0  "
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dominicana.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "mask = dominicana.CodaType.isin(['1-NOISE', '2-NOISE','3-NOISE','4-NOISE','5-NOISE','6-NOISE','7-NOISE','8-NOISE','9-NOISE','10-NOISE','10i','10R'])\n",
    "dominicana_clean = dominicana[~mask]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(8032, 18)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dominicana_clean.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "23"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dominicana_clean.CodaType.nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the ETP dataset, unfortunately the preprocessing code is not in the repository. Based on the information in the paper, we were unable to infer how exactly the data was processed.\n",
    "\n",
    "We will only use the Dominicana dataset for our experiments. This might actually be advantageous - the intent behind this repository is open the reaserch field to a broader audience and to evaluate the performance of Random Forests. As such, narrowing down the scope of our inquiry in terms of data we run our experiments on might be beneficial.\n",
    "\n",
    "We will therefore focus on two tasks - performing classification on the Dominicana dataset for the clan identification and coda identification task.\n",
    "\n",
    "For the clan identification task, the authors report balancing the classes to 949 per class. Let us carry out this procedure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "EC1    7770\n",
       "EC2     949\n",
       "Name: Clan, dtype: int64"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dominicana.Clan.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(949, 18)"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dominicana[dominicana.Clan == 'EC1'].sample(n=949).shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "dominicana_clan = pd.concat(\n",
    "    (\n",
    "        dominicana[dominicana.Clan == 'EC1'].sample(n=949, random_state=42),\n",
    "        dominicana[dominicana.Clan == 'EC2']\n",
    "    )\n",
    ")    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1898, 18)"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dominicana_clan.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now construct our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "dblock_train = DataBlock(\n",
    "    get_x = get_independent_vars,\n",
    "    get_y = (get_clan_membership, Categorize),\n",
    "    splitter = TrainTestSplitter(test_size=0.1, random_state=42, stratify=dominicana_clan.Clan.factorize()[0])\n",
    ")\n",
    "\n",
    "datasets_clan = dblock_train.datasets(dominicana_clan)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(#1898) [(array([0.    , 0.    , 0.    , 0.4812, 0.2679, 0.2377, 0.1766, 0.1265,\n",
       "       0.1412]), TensorCategory(0)),(array([0.   , 0.   , 0.   , 0.   , 0.   , 0.117, 0.097, 0.078, 0.069]), TensorCategory(0)),(array([0.    , 0.    , 0.    , 0.    , 0.    , 0.2199, 0.2457, 0.1611,\n",
       "       0.1799]), TensorCategory(0)),(array([0.       , 0.       , 0.       , 0.       , 0.       , 0.2730917,\n",
       "       0.2577083, 0.1612834, 0.1407083]), TensorCategory(0)),(array([0.   , 0.   , 0.   , 0.   , 0.   , 0.085, 0.106, 0.074, 0.073]), TensorCategory(0)),(array([0.       , 0.       , 0.       , 0.       , 0.       , 0.369    ,\n",
       "       0.3160333, 0.1682917, 0.1581833]), TensorCategory(0)),(array([0.       , 0.       , 0.       , 0.422875 , 0.3855917, 0.1168333,\n",
       "       0.116425 , 0.1389583, 0.1752667]), TensorCategory(0)),(array([0.   , 0.   , 0.   , 0.   , 0.   , 0.083, 0.097, 0.073, 0.074]), TensorCategory(0)),(array([0.    , 0.    , 0.    , 0.    , 0.    , 0.    , 0.261 , 0.1877,\n",
       "       0.1174]), TensorCategory(0)),(array([0.    , 0.    , 0.    , 0.    , 0.    , 0.3706, 0.4403, 0.1895,\n",
       "       0.1945]), TensorCategory(0))...]"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "datasets_clan"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now construct a dataset for classifying coda types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "get_coda_type = partialler(get_target, col_name='CodaType')\n",
    "\n",
    "dblock_train = DataBlock(\n",
    "    get_x = get_independent_vars,\n",
    "    get_y = (get_coda_type, Categorize),\n",
    "    splitter = TrainTestSplitter(test_size=0.1, random_state=42, stratify=dominicana_clean.CodaType.factorize()[0])\n",
    ")\n",
    "\n",
    "datasets_coda = dblock_train.datasets(dominicana_clean)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(#23) ['1+1+3','1+31','1+32','2+3','3D','3R','4D','4R1','4R2','5R1'...]"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "datasets_coda.vocab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now we are ready to start training!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}