notebooks/03-process.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Interatively run array ephys workflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook walks you through the steps in detail to run the ephys workflow.\n",
    "\n",
    "+ If you need a more automatic approach to run the workflow, refer to [03-automate](03-automate.ipynb)\n",
    "+ The workflow requires neuropixels meta file and kilosort output data. If you haven't configure the paths, refer to [01-configure](01-configure.ipynb)\n",
    "+ To overview the schema structures, refer to [02-workflow-structure](02-workflow-structure.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's will change the directory to the package root to load configuration and also import relevant schemas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "if os.path.basename(os.getcwd()) == \"notebooks\": os.chdir(\"..\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2022-09-15 09:12:40,845][INFO]: Connecting cbroz@tutorial-db.datajoint.io:3306\n",
      "[2022-09-15 09:12:41,306][INFO]: Connected cbroz@tutorial-db.datajoint.io:3306\n"
     ]
    }
   ],
   "source": [
    "import datajoint as dj\n",
    "from workflow_array_ephys.pipeline import lab, subject, session, probe, ephys"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ingestion of metadata: subjects, sessions, probes, and probe insertions\n",
    "\n",
    "The first step to run through the workflow is to insert metadata into the following tables:\n",
    "\n",
    "+ subject.Subject: an animal subject for experiments\n",
    "+ session.Session: an experimental session performed on an animal subject\n",
    "+ session.SessionDirectory: directory to the data for a given experimental session\n",
    "+ probe.Probe: probe information\n",
    "+ ephys.ProbeInsertion: probe insertion into an animal subject during a given experimental session"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"200pt\" height=\"185pt\" viewBox=\"0.00 0.00 199.50 185.00\">\n<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 181)\">\n<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-181 195.5,-181 195.5,4 -4,4\"/>\n<!-- subject.Subject -->\n<g id=\"node1\" class=\"node\">\n<title>subject.Subject</title>\n<g id=\"a_node1\"><a xlink:title=\"subject              \r------------------------------\rsex                  \rsubject_birth_date   \rsubject_description=&quot;&quot; \r\">\n<polygon fill=\"#00ff00\" fill-opacity=\"0.188235\" stroke=\"#00ff00\" stroke-opacity=\"0.188235\" points=\"101,-177 3,-177 3,-142 101,-142 101,-177\"/>\n<text text-anchor=\"start\" x=\"11\" y=\"-157.4\" font-family=\"arial\" text-decoration=\"underline\" font-size=\"12.00\" fill=\"darkgreen\">subject.Subject</text>\n</a>\n</g>\n</g>\n<!-- session.Session -->\n<g id=\"node2\" class=\"node\">\n<title>session.Session</title>\n<g id=\"a_node2\"><a xlink:title=\"→ subject.Subject\rsession_datetime     \r\">\n<polygon fill=\"#00ff00\" fill-opacity=\"0.188235\" stroke=\"#00ff00\" stroke-opacity=\"0.188235\" points=\"104,-106 0,-106 0,-71 104,-71 104,-106\"/>\n<text text-anchor=\"start\" x=\"8\" y=\"-86.4\" font-family=\"arial\" text-decoration=\"underline\" font-size=\"12.00\" fill=\"darkgreen\">session.Session</text>\n</a>\n</g>\n</g>\n<!-- subject.Subject&#45;&gt;session.Session -->\n<g id=\"edge1\" class=\"edge\">\n<title>subject.Subject-&gt;session.Session</title>\n<path fill=\"none\" stroke=\"#000000\" stroke-width=\"0.75\" stroke-opacity=\"0.250980\" d=\"M52,-141.8C52,-130.95 52,-116.87 52,-106.05\"/>\n</g>\n<!-- ephys.ProbeInsertion -->\n<g id=\"node3\" class=\"node\">\n<title>ephys.ProbeInsertion</title>\n<g id=\"a_node3\"><a xlink:title=\"→ session.Session\rinsertion_number     \r------------------------------\r→ probe.Probe\r\">\n<polygon fill=\"#00ff00\" fill-opacity=\"0.188235\" stroke=\"#00ff00\" stroke-opacity=\"0.188235\" points=\"169,-35 39,-35 39,0 169,0 169,-35\"/>\n<text text-anchor=\"start\" x=\"47\" y=\"-15.4\" font-family=\"arial\" text-decoration=\"underline\" font-size=\"12.00\" fill=\"darkgreen\">ephys.ProbeInsertion</text>\n</a>\n</g>\n</g>\n<!-- session.Session&#45;&gt;ephys.ProbeInsertion -->\n<g id=\"edge2\" class=\"edge\">\n<title>session.Session-&gt;ephys.ProbeInsertion</title>\n<path fill=\"none\" stroke=\"#000000\" stroke-width=\"0.75\" stroke-opacity=\"0.250980\" d=\"M64.59,-70.8C72.76,-59.95 83.38,-45.87 91.53,-35.05\"/>\n</g>\n<!-- probe.Probe -->\n<g id=\"node4\" class=\"node\">\n<title>probe.Probe</title>\n<g id=\"a_node4\"><a xlink:title=\"probe                \r------------------------------\r→ probe.ProbeType\rprobe_comment=&quot;&quot;     \r\">\n<polygon fill=\"#000000\" fill-opacity=\"0.125490\" stroke=\"transparent\" points=\"191.5,-106 122.5,-106 122.5,-71 191.5,-71 191.5,-106\"/>\n<text text-anchor=\"start\" x=\"130.5\" y=\"-87\" font-family=\"arial\" text-decoration=\"underline\" font-size=\"10.00\">probe.Probe</text>\n</a>\n</g>\n</g>\n<!-- probe.Probe&#45;&gt;ephys.ProbeInsertion -->\n<g id=\"edge3\" class=\"edge\">\n<title>probe.Probe-&gt;ephys.ProbeInsertion</title>\n<path fill=\"none\" stroke=\"#000000\" stroke-width=\"0.75\" stroke-dasharray=\"5,2\" stroke-opacity=\"0.250980\" d=\"M144.17,-70.8C135.84,-59.95 125.02,-45.87 116.71,-35.05\"/>\n</g>\n</g>\n</svg>",
      "text/plain": [
       "<datajoint.diagram.Diagram at 0x10b161fd0>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(\n",
    "    dj.Diagram(subject.Subject)\n",
    "    + dj.Diagram(session.Session)\n",
    "    + dj.Diagram(probe.Probe)\n",
    "    + dj.Diagram(ephys.ProbeInsertion)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our example dataset is for subject6, session1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ingest Subject"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "# Animal Subject\n",
       "subject              : varchar(8)                   # \n",
       "---\n",
       "sex                  : enum('M','F','U')            # \n",
       "subject_birth_date   : date                         # \n",
       "subject_description=\"\" : varchar(1024)                # "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "subject.Subject.heading"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    \n",
       "    <style type=\"text/css\">\n",
       "        .Relation{\n",
       "            border-collapse:collapse;\n",
       "        }\n",
       "        .Relation th{\n",
       "            background: #A0A0A0; color: #ffffff; padding:4px; border:#f0e0e0 1px solid;\n",
       "            font-weight: normal; font-family: monospace; font-size: 100%;\n",
       "        }\n",
       "        .Relation td{\n",
       "            padding:4px; border:#f0e0e0 1px solid; font-size:100%;\n",
       "        }\n",
       "        .Relation tr:nth-child(odd){\n",
       "            background: #ffffff;\n",
       "        }\n",
       "        .Relation tr:nth-child(even){\n",
       "            background: #f3f1ff;\n",
       "        }\n",
       "        /* Tooltip container */\n",
       "        .djtooltip {\n",
       "        }\n",
       "        /* Tooltip text */\n",
       "        .djtooltip .djtooltiptext {\n",
       "            visibility: hidden;\n",
       "            width: 120px;\n",
       "            background-color: black;\n",
       "            color: #fff;\n",
       "            text-align: center;\n",
       "            padding: 5px 0;\n",
       "            border-radius: 6px;\n",
       "            /* Position the tooltip text - see examples below! */\n",
       "            position: absolute;\n",
       "            z-index: 1;\n",
       "        }\n",
       "        #primary {\n",
       "            font-weight: bold;\n",
       "            color: black;\n",
       "        }\n",
       "        #nonprimary {\n",
       "            font-weight: normal;\n",
       "            color: white;\n",
       "        }\n",
       "\n",
       "        /* Show the tooltip text when you mouse over the tooltip container */\n",
       "        .djtooltip:hover .djtooltiptext {\n",
       "            visibility: visible;\n",
       "        }\n",
       "    </style>\n",
       "    \n",
       "    <b>Animal Subject</b>\n",
       "        <div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "        <table border=\"1\" class=\"Relation\">\n",
       "            <thead> <tr style=\"text-align: right;\"> <th> <div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">subject</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">sex</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">subject_birth_date</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">subject_description</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div> </th> </tr> </thead>\n",
       "            <tbody> <tr> <td>subject5</td>\n",
       "<td>F</td>\n",
       "<td>2020-01-01</td>\n",
       "<td>rich</td></tr><tr><td>subject6</td>\n",
       "<td>M</td>\n",
       "<td>2020-01-03</td>\n",
       "<td>hneih_E105</td> </tr> </tbody>\n",
       "        </table>\n",
       "        \n",
       "        <p>Total: 2</p></div>\n",
       "        "
      ],
      "text/plain": [
       "*subject     sex     subject_birth_ subject_descri\n",
       "+----------+ +-----+ +------------+ +------------+\n",
       "subject5     F       2020-01-01     rich          \n",
       "subject6     M       2020-01-03     hneih_E105    \n",
       " (Total: 2)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# insert entries with insert1() or insert(), with all required attributes specified in a dictionary\n",
    "subject.Subject.insert1(\n",
    "    dict(subject=\"subject6\", sex=\"M\", subject_birth_date=\"2020-01-04\"),\n",
    "    skip_duplicates=True,\n",
    ")  # skip_duplicates avoids error when inserting entries with duplicated primary keys\n",
    "subject.Subject()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ingest Session"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-> subject.Subject\n",
      "session_datetime     : datetime                     \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'-> subject.Subject\\nsession_datetime     : datetime                     \\n'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "session.Session.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "# \n",
       "subject              : varchar(8)                   # \n",
       "session_datetime     : datetime                     # "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "session.Session.heading"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    \n",
       "    <style type=\"text/css\">\n",
       "        .Relation{\n",
       "            border-collapse:collapse;\n",
       "        }\n",
       "        .Relation th{\n",
       "            background: #A0A0A0; color: #ffffff; padding:4px; border:#f0e0e0 1px solid;\n",
       "            font-weight: normal; font-family: monospace; font-size: 100%;\n",
       "        }\n",
       "        .Relation td{\n",
       "            padding:4px; border:#f0e0e0 1px solid; font-size:100%;\n",
       "        }\n",
       "        .Relation tr:nth-child(odd){\n",
       "            background: #ffffff;\n",
       "        }\n",
       "        .Relation tr:nth-child(even){\n",
       "            background: #f3f1ff;\n",
       "        }\n",
       "        /* Tooltip container */\n",
       "        .djtooltip {\n",
       "        }\n",
       "        /* Tooltip text */\n",
       "        .djtooltip .djtooltiptext {\n",
       "            visibility: hidden;\n",
       "            width: 120px;\n",
       "            background-color: black;\n",
       "            color: #fff;\n",
       "            text-align: center;\n",
       "            padding: 5px 0;\n",
       "            border-radius: 6px;\n",
       "            /* Position the tooltip text - see examples below! */\n",
       "            position: absolute;\n",
       "            z-index: 1;\n",
       "        }\n",
       "        #primary {\n",
       "            font-weight: bold;\n",
       "            color: black;\n",
       "        }\n",
       "        #nonprimary {\n",
       "            font-weight: normal;\n",
       "            color: white;\n",
       "        }\n",
       "\n",
       "        /* Show the tooltip text when you mouse over the tooltip container */\n",
       "        .djtooltip:hover .djtooltiptext {\n",
       "            visibility: visible;\n",
       "        }\n",
       "    </style>\n",
       "    \n",
       "    <b></b>\n",
       "        <div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "        <table border=\"1\" class=\"Relation\">\n",
       "            <thead> <tr style=\"text-align: right;\"> <th> <div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">subject</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">session_datetime</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div> </th> </tr> </thead>\n",
       "            <tbody> <tr> <td>subject5</td>\n",
       "<td>2018-07-03 20:32:28</td></tr><tr><td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td> </tr> </tbody>\n",
       "        </table>\n",
       "        \n",
       "        <p>Total: 2</p></div>\n",
       "        "
      ],
      "text/plain": [
       "*subject     *session_datet\n",
       "+----------+ +------------+\n",
       "subject5     2018-07-03 20:\n",
       "subject6     2021-01-15 11:\n",
       " (Total: 2)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "session_key = dict(subject=\"subject6\", session_datetime=\"2021-01-15 11:16:38\")\n",
    "session.Session.insert1(session_key, skip_duplicates=True)\n",
    "session.Session()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ingest SessionDirectory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-> session.Session\n",
      "---\n",
      "session_dir          : varchar(256)                 # Path to the data directory for a session\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'-> session.Session\\n---\\nsession_dir          : varchar(256)                 # Path to the data directory for a session\\n'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "session.SessionDirectory.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "# \n",
       "subject              : varchar(8)                   # \n",
       "session_datetime     : datetime                     # \n",
       "---\n",
       "session_dir          : varchar(256)                 # Path to the data directory for a session"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "session.SessionDirectory.heading"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    \n",
       "    <style type=\"text/css\">\n",
       "        .Relation{\n",
       "            border-collapse:collapse;\n",
       "        }\n",
       "        .Relation th{\n",
       "            background: #A0A0A0; color: #ffffff; padding:4px; border:#f0e0e0 1px solid;\n",
       "            font-weight: normal; font-family: monospace; font-size: 100%;\n",
       "        }\n",
       "        .Relation td{\n",
       "            padding:4px; border:#f0e0e0 1px solid; font-size:100%;\n",
       "        }\n",
       "        .Relation tr:nth-child(odd){\n",
       "            background: #ffffff;\n",
       "        }\n",
       "        .Relation tr:nth-child(even){\n",
       "            background: #f3f1ff;\n",
       "        }\n",
       "        /* Tooltip container */\n",
       "        .djtooltip {\n",
       "        }\n",
       "        /* Tooltip text */\n",
       "        .djtooltip .djtooltiptext {\n",
       "            visibility: hidden;\n",
       "            width: 120px;\n",
       "            background-color: black;\n",
       "            color: #fff;\n",
       "            text-align: center;\n",
       "            padding: 5px 0;\n",
       "            border-radius: 6px;\n",
       "            /* Position the tooltip text - see examples below! */\n",
       "            position: absolute;\n",
       "            z-index: 1;\n",
       "        }\n",
       "        #primary {\n",
       "            font-weight: bold;\n",
       "            color: black;\n",
       "        }\n",
       "        #nonprimary {\n",
       "            font-weight: normal;\n",
       "            color: white;\n",
       "        }\n",
       "\n",
       "        /* Show the tooltip text when you mouse over the tooltip container */\n",
       "        .djtooltip:hover .djtooltiptext {\n",
       "            visibility: visible;\n",
       "        }\n",
       "    </style>\n",
       "    \n",
       "    <b></b>\n",
       "        <div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "        <table border=\"1\" class=\"Relation\">\n",
       "            <thead> <tr style=\"text-align: right;\"> <th> <div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">subject</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">session_datetime</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">session_dir</p>\n",
       "                            <span class=\"djtooltiptext\">Path to the data directory for a session</span>\n",
       "                        </div> </th> </tr> </thead>\n",
       "            <tbody> <tr> <td>subject5</td>\n",
       "<td>2018-07-03 20:32:28</td>\n",
       "<td>subject5/session1</td></tr><tr><td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>subject6/session1</td> </tr> </tbody>\n",
       "        </table>\n",
       "        \n",
       "        <p>Total: 2</p></div>\n",
       "        "
      ],
      "text/plain": [
       "*subject     *session_datet session_dir   \n",
       "+----------+ +------------+ +------------+\n",
       "subject5     2018-07-03 20: subject5/sessi\n",
       "subject6     2021-01-15 11: subject6/sessi\n",
       " (Total: 2)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "session.SessionDirectory.insert1(\n",
    "    dict(\n",
    "        subject=\"subject6\",\n",
    "        session_datetime=\"2021-01-15 11:16:38\",\n",
    "        session_dir=\"subject6/session1\",\n",
    "    ),\n",
    "    skip_duplicates=True,\n",
    ")\n",
    "session.SessionDirectory()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**:\n",
    "\n",
    "the `session_dir` needs to be:\n",
    "+ a directory **relative to** the `ephys_root_path` in the configuration file, refer to [01-configure](01-configure.ipynb) for more information.\n",
    "+ a directory in POSIX format (Unix/Linux), with `/`, the difference in Operating system will be taken care of by the elements."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ingest Probe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "# Represent a physical probe with unique identification\n",
       "probe                : varchar(32)                  # unique identifier for this model of probe (e.g. serial number)\n",
       "---\n",
       "probe_type           : varchar(32)                  # e.g. neuropixels_1.0\n",
       "probe_comment=\"\"     : varchar(1000)                # "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "probe.Probe.heading"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    \n",
       "    <style type=\"text/css\">\n",
       "        .Relation{\n",
       "            border-collapse:collapse;\n",
       "        }\n",
       "        .Relation th{\n",
       "            background: #A0A0A0; color: #ffffff; padding:4px; border:#f0e0e0 1px solid;\n",
       "            font-weight: normal; font-family: monospace; font-size: 100%;\n",
       "        }\n",
       "        .Relation td{\n",
       "            padding:4px; border:#f0e0e0 1px solid; font-size:100%;\n",
       "        }\n",
       "        .Relation tr:nth-child(odd){\n",
       "            background: #ffffff;\n",
       "        }\n",
       "        .Relation tr:nth-child(even){\n",
       "            background: #f3f1ff;\n",
       "        }\n",
       "        /* Tooltip container */\n",
       "        .djtooltip {\n",
       "        }\n",
       "        /* Tooltip text */\n",
       "        .djtooltip .djtooltiptext {\n",
       "            visibility: hidden;\n",
       "            width: 120px;\n",
       "            background-color: black;\n",
       "            color: #fff;\n",
       "            text-align: center;\n",
       "            padding: 5px 0;\n",
       "            border-radius: 6px;\n",
       "            /* Position the tooltip text - see examples below! */\n",
       "            position: absolute;\n",
       "            z-index: 1;\n",
       "        }\n",
       "        #primary {\n",
       "            font-weight: bold;\n",
       "            color: black;\n",
       "        }\n",
       "        #nonprimary {\n",
       "            font-weight: normal;\n",
       "            color: white;\n",
       "        }\n",
       "\n",
       "        /* Show the tooltip text when you mouse over the tooltip container */\n",
       "        .djtooltip:hover .djtooltiptext {\n",
       "            visibility: visible;\n",
       "        }\n",
       "    </style>\n",
       "    \n",
       "    <b>Represent a physical probe with unique identification</b>\n",
       "        <div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "        <table border=\"1\" class=\"Relation\">\n",
       "            <thead> <tr style=\"text-align: right;\"> <th> <div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">probe</p>\n",
       "                            <span class=\"djtooltiptext\">unique identifier for this model of probe (e.g. serial number)</span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">probe_type</p>\n",
       "                            <span class=\"djtooltiptext\">e.g. neuropixels_1.0</span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">probe_comment</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div> </th> </tr> </thead>\n",
       "            <tbody> <tr> <td>17131311651</td>\n",
       "<td>neuropixels 1.0 - 3B</td>\n",
       "<td></td></tr><tr><td>262716621</td>\n",
       "<td>neuropixels 1.0 - 3A</td>\n",
       "<td></td></tr><tr><td>714000838</td>\n",
       "<td>neuropixels 1.0 - 3A</td>\n",
       "<td></td> </tr> </tbody>\n",
       "        </table>\n",
       "        \n",
       "        <p>Total: 3</p></div>\n",
       "        "
      ],
      "text/plain": [
       "*probe         probe_type     probe_comment \n",
       "+------------+ +------------+ +------------+\n",
       "17131311651    neuropixels 1.               \n",
       "262716621      neuropixels 1.               \n",
       "714000838      neuropixels 1.               \n",
       " (Total: 3)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "probe.Probe.insert1(\n",
    "    dict(probe=\"17131311651\", probe_type=\"neuropixels 1.0 - 3B\"), skip_duplicates=True\n",
    ")  # this info could be achieve from neuropixels meta file.\n",
    "probe.Probe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ingest ProbeInsertion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# Probe insertion implanted into an animal for a given session.\n",
      "-> session.Session\n",
      "insertion_number     : tinyint unsigned             \n",
      "---\n",
      "-> probe.Probe\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'# Probe insertion implanted into an animal for a given session.\\n-> session.Session\\ninsertion_number     : tinyint unsigned             \\n---\\n-> probe.Probe\\n'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ephys.ProbeInsertion.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "# Probe insertion implanted into an animal for a given session.\n",
       "subject              : varchar(8)                   # \n",
       "session_datetime     : datetime                     # \n",
       "insertion_number     : tinyint unsigned             # \n",
       "---\n",
       "probe                : varchar(32)                  # unique identifier for this model of probe (e.g. serial number)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ephys.ProbeInsertion.heading"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    \n",
       "    <style type=\"text/css\">\n",
       "        .Relation{\n",
       "            border-collapse:collapse;\n",
       "        }\n",
       "        .Relation th{\n",
       "            background: #A0A0A0; color: #ffffff; padding:4px; border:#f0e0e0 1px solid;\n",
       "            font-weight: normal; font-family: monospace; font-size: 100%;\n",
       "        }\n",
       "        .Relation td{\n",
       "            padding:4px; border:#f0e0e0 1px solid; font-size:100%;\n",
       "        }\n",
       "        .Relation tr:nth-child(odd){\n",
       "            background: #ffffff;\n",
       "        }\n",
       "        .Relation tr:nth-child(even){\n",
       "            background: #f3f1ff;\n",
       "        }\n",
       "        /* Tooltip container */\n",
       "        .djtooltip {\n",
       "        }\n",
       "        /* Tooltip text */\n",
       "        .djtooltip .djtooltiptext {\n",
       "            visibility: hidden;\n",
       "            width: 120px;\n",
       "            background-color: black;\n",
       "            color: #fff;\n",
       "            text-align: center;\n",
       "            padding: 5px 0;\n",
       "            border-radius: 6px;\n",
       "            /* Position the tooltip text - see examples below! */\n",
       "            position: absolute;\n",
       "            z-index: 1;\n",
       "        }\n",
       "        #primary {\n",
       "            font-weight: bold;\n",
       "            color: black;\n",
       "        }\n",
       "        #nonprimary {\n",
       "            font-weight: normal;\n",
       "            color: white;\n",
       "        }\n",
       "\n",
       "        /* Show the tooltip text when you mouse over the tooltip container */\n",
       "        .djtooltip:hover .djtooltiptext {\n",
       "            visibility: visible;\n",
       "        }\n",
       "    </style>\n",
       "    \n",
       "    <b>Probe insertion implanted into an animal for a given session.</b>\n",
       "        <div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "        <table border=\"1\" class=\"Relation\">\n",
       "            <thead> <tr style=\"text-align: right;\"> <th> <div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">subject</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">session_datetime</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">insertion_number</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">probe</p>\n",
       "                            <span class=\"djtooltiptext\">unique identifier for this model of probe (e.g. serial number)</span>\n",
       "                        </div> </th> </tr> </thead>\n",
       "            <tbody> <tr> <td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>17131311651</td> </tr> </tbody>\n",
       "        </table>\n",
       "        \n",
       "        <p>Total: 1</p></div>\n",
       "        "
      ],
      "text/plain": [
       "*subject     *session_datet *insertion_num probe         \n",
       "+----------+ +------------+ +------------+ +------------+\n",
       "subject6     2021-01-15 11: 0              17131311651   \n",
       " (Total: 1)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ephys.ProbeInsertion.insert1(\n",
    "    dict(\n",
    "        subject=\"subject6\",\n",
    "        session_datetime=\"2021-01-15 11:16:38\",\n",
    "        insertion_number=0,\n",
    "        probe=\"17131311651\",\n",
    "    ),\n",
    "    skip_duplicates=True,\n",
    ")  # probe, subject, session_datetime needs to follow the restrictions of foreign keys.\n",
    "ephys.ProbeInsertion()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Automate this manual step\n",
    "\n",
    "In this workflow, these manual steps could be automated by:\n",
    "1. Insert entries in files `/user_data/subjects.csv` and `/user_data/session.csv`\n",
    "2. Extract user-specified information from `/user_data/subjects.csv` and `/user_data/sessions.csv` and insert to Subject and Session tables by running:\n",
    "```\n",
    "from workflow_array_ephys.ingest import ingest_subjects, ingest_sessions\n",
    "ingest_subjects()\n",
    "ingest_sessions()\n",
    "```\n",
    "`ingest_sessions` also extracts probe and probe insertion information automatically from the meta files.\n",
    "\n",
    "This is the regular routine for daily data processing, illustrated in notebook [04-automate](04-automate[optional].ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Populate EphysRecording"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are ready to populate EphysRecording, a table for entries of ephys recording in a particular session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"360pt\" height=\"240pt\" viewBox=\"0.00 0.00 360.12 240.00\">\n<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 236)\">\n<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-236 356.12,-236 356.12,4 -4,4\"/>\n<!-- probe.ElectrodeConfig.Electrode -->\n<g id=\"node1\" class=\"node\">\n<title>probe.ElectrodeConfig.Electrode</title>\n<g id=\"a_node1\"><a xlink:title=\"→ probe.ElectrodeConfig\r→ probe.ProbeType.Electrode\r\">\n<polygon fill=\"transparent\" stroke=\"transparent\" points=\"158,-82 0,-82 0,-63 158,-63 158,-82\"/>\n<text text-anchor=\"middle\" x=\"79\" y=\"-70\" font-family=\"arial\" font-size=\"10.00\">probe.ElectrodeConfig.Electrode</text>\n</a>\n</g>\n</g>\n<!-- ephys.EphysRecording.EphysFile -->\n<g id=\"node2\" class=\"node\">\n<title>ephys.EphysRecording.EphysFile</title>\n<g id=\"a_node2\"><a xlink:title=\"→ ephys.EphysRecording\rfile_path            \r\">\n<polygon fill=\"transparent\" stroke=\"transparent\" points=\"346,-19 182,-19 182,0 346,0 346,-19\"/>\n<text text-anchor=\"start\" x=\"190\" y=\"-8\" font-family=\"arial\" text-decoration=\"underline\" font-size=\"10.00\">ephys.EphysRecording.EphysFile</text>\n</a>\n</g>\n</g>\n<!-- ephys.EphysRecording -->\n<g id=\"node3\" class=\"node\">\n<title>ephys.EphysRecording</title>\n<g id=\"a_node3\"><a xlink:title=\"→ ephys.ProbeInsertion\r------------------------------\r→ probe.ElectrodeConfig\r→ ephys.AcquisitionSoftware\rsampling_rate        \rrecording_datetime   \rrecording_duration   \r\">\n<ellipse fill=\"#00007f\" fill-opacity=\"0.250980\" stroke=\"#00007f\" stroke-opacity=\"0.250980\" cx=\"264\" cy=\"-72.5\" rx=\"88.25\" ry=\"17.5\"/>\n<text text-anchor=\"middle\" x=\"264\" y=\"-69.4\" font-family=\"arial\" font-size=\"12.00\" fill=\"#00007f\" fill-opacity=\"0.627451\">ephys.EphysRecording</text>\n</a>\n</g>\n</g>\n<!-- ephys.EphysRecording&#45;&gt;ephys.EphysRecording.EphysFile -->\n<g id=\"edge1\" class=\"edge\">\n<title>ephys.EphysRecording-&gt;ephys.EphysRecording.EphysFile</title>\n<path fill=\"none\" stroke=\"#000000\" stroke-width=\"0.75\" stroke-opacity=\"0.250980\" d=\"M264,-54.83C264,-43.26 264,-28.32 264,-19.08\"/>\n</g>\n<!-- ephys.ProbeInsertion -->\n<g id=\"node4\" class=\"node\">\n<title>ephys.ProbeInsertion</title>\n<g id=\"a_node4\"><a xlink:title=\"→ session.Session\rinsertion_number     \r------------------------------\r→ probe.Probe\r\">\n<polygon fill=\"#00ff00\" fill-opacity=\"0.188235\" stroke=\"#00ff00\" stroke-opacity=\"0.188235\" points=\"329,-161 199,-161 199,-126 329,-126 329,-161\"/>\n<text text-anchor=\"middle\" x=\"264\" y=\"-140.4\" font-family=\"arial\" font-size=\"12.00\" fill=\"darkgreen\">ephys.ProbeInsertion</text>\n</a>\n</g>\n</g>\n<!-- ephys.ProbeInsertion&#45;&gt;ephys.EphysRecording -->\n<g id=\"edge2\" class=\"edge\">\n<title>ephys.ProbeInsertion-&gt;ephys.EphysRecording</title>\n<path fill=\"none\" stroke=\"#000000\" stroke-width=\"2\" stroke-opacity=\"0.250980\" d=\"M264,-125.8C264,-114.95 264,-100.87 264,-90.05\"/>\n</g>\n<!-- probe.ElectrodeConfig -->\n<g id=\"node5\" class=\"node\">\n<title>probe.ElectrodeConfig</title>\n<g id=\"a_node5\"><a xlink:title=\"electrode_config_hash \r------------------------------\r→ probe.ProbeType\relectrode_config_name \r\">\n<polygon fill=\"#000000\" fill-opacity=\"0.125490\" stroke=\"transparent\" points=\"135.5,-161 22.5,-161 22.5,-126 135.5,-126 135.5,-161\"/>\n<text text-anchor=\"start\" x=\"30.5\" y=\"-142\" font-family=\"arial\" text-decoration=\"underline\" font-size=\"10.00\">probe.ElectrodeConfig</text>\n</a>\n</g>\n</g>\n<!-- probe.ElectrodeConfig&#45;&gt;probe.ElectrodeConfig.Electrode -->\n<g id=\"edge3\" class=\"edge\">\n<title>probe.ElectrodeConfig-&gt;probe.ElectrodeConfig.Electrode</title>\n<path fill=\"none\" stroke=\"#000000\" stroke-width=\"0.75\" stroke-opacity=\"0.250980\" d=\"M79,-125.8C79,-112.08 79,-93.19 79,-82.27\"/>\n</g>\n<!-- probe.ElectrodeConfig&#45;&gt;ephys.EphysRecording -->\n<g id=\"edge4\" class=\"edge\">\n<title>probe.ElectrodeConfig-&gt;ephys.EphysRecording</title>\n<path fill=\"none\" stroke=\"#000000\" stroke-width=\"0.75\" stroke-dasharray=\"5,2\" stroke-opacity=\"0.250980\" d=\"M123.31,-125.97C154.1,-114.49 194.76,-99.32 224.45,-88.25\"/>\n</g>\n<!-- session.Session -->\n<g id=\"node6\" class=\"node\">\n<title>session.Session</title>\n<g id=\"a_node6\"><a xlink:title=\"→ subject.Subject\rsession_datetime     \r\">\n<polygon fill=\"#00ff00\" fill-opacity=\"0.188235\" stroke=\"#00ff00\" stroke-opacity=\"0.188235\" points=\"316,-232 212,-232 212,-197 316,-197 316,-232\"/>\n<text text-anchor=\"start\" x=\"220\" y=\"-212.4\" font-family=\"arial\" text-decoration=\"underline\" font-size=\"12.00\" fill=\"darkgreen\">session.Session</text>\n</a>\n</g>\n</g>\n<!-- session.Session&#45;&gt;ephys.ProbeInsertion -->\n<g id=\"edge5\" class=\"edge\">\n<title>session.Session-&gt;ephys.ProbeInsertion</title>\n<path fill=\"none\" stroke=\"#000000\" stroke-width=\"0.75\" stroke-opacity=\"0.250980\" d=\"M264,-196.8C264,-185.95 264,-171.87 264,-161.05\"/>\n</g>\n</g>\n</svg>",
      "text/plain": [
       "<datajoint.diagram.Diagram at 0x11fd9afd0>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(\n",
    "    dj.Diagram(session.Session)\n",
    "    + (dj.Diagram(probe.ElectrodeConfig) + 1)\n",
    "    + dj.Diagram(ephys.EphysRecording)\n",
    "    + dj.Diagram(ephys.EphysRecording.EphysFile)\n",
    ")\n",
    "# +1 means plotting 1 more layer of the child tables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "EphysRecording: 100%|████████████████████████████████████████████████████████████████████| 1/1 [00:02<00:00,  2.84s/it]\n"
     ]
    }
   ],
   "source": [
    "# The first argument specify a particular session to populate\n",
    "ephys.EphysRecording.populate(session_key, display_progress=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Populate EphysRecording extracts the following information from .ap.meta file from SpikeGLX:\n",
    "\n",
    "1. **probe.EelectrodeConfig**: this procedure detects new ElectrodeConfig, i.e. which 384 electrodes out of the total 960 on the probe were used in this ephys session, and save the results into the table `probe.EelectrodeConfig`. Each entry in table `ephys.EphysRecording` specifies which ElectrodeConfig is used in a particular ephys session."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this ephys session we just populated, Electrodes 0-383 was used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    \n",
       "    <style type=\"text/css\">\n",
       "        .Relation{\n",
       "            border-collapse:collapse;\n",
       "        }\n",
       "        .Relation th{\n",
       "            background: #A0A0A0; color: #ffffff; padding:4px; border:#f0e0e0 1px solid;\n",
       "            font-weight: normal; font-family: monospace; font-size: 100%;\n",
       "        }\n",
       "        .Relation td{\n",
       "            padding:4px; border:#f0e0e0 1px solid; font-size:100%;\n",
       "        }\n",
       "        .Relation tr:nth-child(odd){\n",
       "            background: #ffffff;\n",
       "        }\n",
       "        .Relation tr:nth-child(even){\n",
       "            background: #f3f1ff;\n",
       "        }\n",
       "        /* Tooltip container */\n",
       "        .djtooltip {\n",
       "        }\n",
       "        /* Tooltip text */\n",
       "        .djtooltip .djtooltiptext {\n",
       "            visibility: hidden;\n",
       "            width: 120px;\n",
       "            background-color: black;\n",
       "            color: #fff;\n",
       "            text-align: center;\n",
       "            padding: 5px 0;\n",
       "            border-radius: 6px;\n",
       "            /* Position the tooltip text - see examples below! */\n",
       "            position: absolute;\n",
       "            z-index: 1;\n",
       "        }\n",
       "        #primary {\n",
       "            font-weight: bold;\n",
       "            color: black;\n",
       "        }\n",
       "        #nonprimary {\n",
       "            font-weight: normal;\n",
       "            color: white;\n",
       "        }\n",
       "\n",
       "        /* Show the tooltip text when you mouse over the tooltip container */\n",
       "        .djtooltip:hover .djtooltiptext {\n",
       "            visibility: visible;\n",
       "        }\n",
       "    </style>\n",
       "    \n",
       "    <b>The electrode configuration setting on a given probe</b>\n",
       "        <div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "        <table border=\"1\" class=\"Relation\">\n",
       "            <thead> <tr style=\"text-align: right;\"> <th> <div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">electrode_config_hash</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">probe_type</p>\n",
       "                            <span class=\"djtooltiptext\">e.g. neuropixels_1.0</span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">electrode_config_name</p>\n",
       "                            <span class=\"djtooltiptext\">user friendly name</span>\n",
       "                        </div> </th> </tr> </thead>\n",
       "            <tbody> <tr> <td>6f6d8149-1603-9d2c-f884-0fdf995ec8b3</td>\n",
       "<td>neuropixels 1.0 - 3B</td>\n",
       "<td>0-383</td> </tr> </tbody>\n",
       "        </table>\n",
       "        \n",
       "        <p>Total: 1</p></div>\n",
       "        "
      ],
      "text/plain": [
       "*electrode_con probe_type     electrode_conf\n",
       "+------------+ +------------+ +------------+\n",
       "6f6d8149-1603- neuropixels 1. 0-383         \n",
       " (Total: 1)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "probe.ElectrodeConfig()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    \n",
       "    <style type=\"text/css\">\n",
       "        .Relation{\n",
       "            border-collapse:collapse;\n",
       "        }\n",
       "        .Relation th{\n",
       "            background: #A0A0A0; color: #ffffff; padding:4px; border:#f0e0e0 1px solid;\n",
       "            font-weight: normal; font-family: monospace; font-size: 100%;\n",
       "        }\n",
       "        .Relation td{\n",
       "            padding:4px; border:#f0e0e0 1px solid; font-size:100%;\n",
       "        }\n",
       "        .Relation tr:nth-child(odd){\n",
       "            background: #ffffff;\n",
       "        }\n",
       "        .Relation tr:nth-child(even){\n",
       "            background: #f3f1ff;\n",
       "        }\n",
       "        /* Tooltip container */\n",
       "        .djtooltip {\n",
       "        }\n",
       "        /* Tooltip text */\n",
       "        .djtooltip .djtooltiptext {\n",
       "            visibility: hidden;\n",
       "            width: 120px;\n",
       "            background-color: black;\n",
       "            color: #fff;\n",
       "            text-align: center;\n",
       "            padding: 5px 0;\n",
       "            border-radius: 6px;\n",
       "            /* Position the tooltip text - see examples below! */\n",
       "            position: absolute;\n",
       "            z-index: 1;\n",
       "        }\n",
       "        #primary {\n",
       "            font-weight: bold;\n",
       "            color: black;\n",
       "        }\n",
       "        #nonprimary {\n",
       "            font-weight: normal;\n",
       "            color: white;\n",
       "        }\n",
       "\n",
       "        /* Show the tooltip text when you mouse over the tooltip container */\n",
       "        .djtooltip:hover .djtooltiptext {\n",
       "            visibility: visible;\n",
       "        }\n",
       "    </style>\n",
       "    \n",
       "    <b>Electrodes selected for recording</b>\n",
       "        <div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "        <table border=\"1\" class=\"Relation\">\n",
       "            <thead> <tr style=\"text-align: right;\"> <th> <div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">electrode_config_hash</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">probe_type</p>\n",
       "                            <span class=\"djtooltiptext\">e.g. neuropixels_1.0</span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">electrode</p>\n",
       "                            <span class=\"djtooltiptext\">electrode index, starts at 0</span>\n",
       "                        </div> </th> </tr> </thead>\n",
       "            <tbody> <tr> <td>6f6d8149-1603-9d2c-f884-0fdf995ec8b3</td>\n",
       "<td>neuropixels 1.0 - 3B</td>\n",
       "<td>0</td></tr><tr><td>6f6d8149-1603-9d2c-f884-0fdf995ec8b3</td>\n",
       "<td>neuropixels 1.0 - 3B</td>\n",
       "<td>1</td></tr><tr><td>6f6d8149-1603-9d2c-f884-0fdf995ec8b3</td>\n",
       "<td>neuropixels 1.0 - 3B</td>\n",
       "<td>2</td></tr><tr><td>6f6d8149-1603-9d2c-f884-0fdf995ec8b3</td>\n",
       "<td>neuropixels 1.0 - 3B</td>\n",
       "<td>3</td></tr><tr><td>6f6d8149-1603-9d2c-f884-0fdf995ec8b3</td>\n",
       "<td>neuropixels 1.0 - 3B</td>\n",
       "<td>4</td></tr><tr><td>6f6d8149-1603-9d2c-f884-0fdf995ec8b3</td>\n",
       "<td>neuropixels 1.0 - 3B</td>\n",
       "<td>5</td></tr><tr><td>6f6d8149-1603-9d2c-f884-0fdf995ec8b3</td>\n",
       "<td>neuropixels 1.0 - 3B</td>\n",
       "<td>6</td> </tr> </tbody>\n",
       "        </table>\n",
       "        <p>...</p>\n",
       "        <p>Total: 384</p></div>\n",
       "        "
      ],
      "text/plain": [
       "*electrode_con *probe_type    *electrode   \n",
       "+------------+ +------------+ +-----------+\n",
       "6f6d8149-1603- neuropixels 1. 0            \n",
       "6f6d8149-1603- neuropixels 1. 1            \n",
       "6f6d8149-1603- neuropixels 1. 2            \n",
       "6f6d8149-1603- neuropixels 1. 3            \n",
       "6f6d8149-1603- neuropixels 1. 4            \n",
       "6f6d8149-1603- neuropixels 1. 5            \n",
       "6f6d8149-1603- neuropixels 1. 6            \n",
       "   ...\n",
       " (Total: 384)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "probe.ElectrodeConfig.Electrode()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. **ephys.EphysRecording**: note here that it refers to a particular electrode_config identified with a hash."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    \n",
       "    <style type=\"text/css\">\n",
       "        .Relation{\n",
       "            border-collapse:collapse;\n",
       "        }\n",
       "        .Relation th{\n",
       "            background: #A0A0A0; color: #ffffff; padding:4px; border:#f0e0e0 1px solid;\n",
       "            font-weight: normal; font-family: monospace; font-size: 100%;\n",
       "        }\n",
       "        .Relation td{\n",
       "            padding:4px; border:#f0e0e0 1px solid; font-size:100%;\n",
       "        }\n",
       "        .Relation tr:nth-child(odd){\n",
       "            background: #ffffff;\n",
       "        }\n",
       "        .Relation tr:nth-child(even){\n",
       "            background: #f3f1ff;\n",
       "        }\n",
       "        /* Tooltip container */\n",
       "        .djtooltip {\n",
       "        }\n",
       "        /* Tooltip text */\n",
       "        .djtooltip .djtooltiptext {\n",
       "            visibility: hidden;\n",
       "            width: 120px;\n",
       "            background-color: black;\n",
       "            color: #fff;\n",
       "            text-align: center;\n",
       "            padding: 5px 0;\n",
       "            border-radius: 6px;\n",
       "            /* Position the tooltip text - see examples below! */\n",
       "            position: absolute;\n",
       "            z-index: 1;\n",
       "        }\n",
       "        #primary {\n",
       "            font-weight: bold;\n",
       "            color: black;\n",
       "        }\n",
       "        #nonprimary {\n",
       "            font-weight: normal;\n",
       "            color: white;\n",
       "        }\n",
       "\n",
       "        /* Show the tooltip text when you mouse over the tooltip container */\n",
       "        .djtooltip:hover .djtooltiptext {\n",
       "            visibility: visible;\n",
       "        }\n",
       "    </style>\n",
       "    \n",
       "    <b>Ephys recording from a probe insertion for a given session.</b>\n",
       "        <div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "        <table border=\"1\" class=\"Relation\">\n",
       "            <thead> <tr style=\"text-align: right;\"> <th> <div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">subject</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">session_datetime</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">insertion_number</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">electrode_config_hash</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">acq_software</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">sampling_rate</p>\n",
       "                            <span class=\"djtooltiptext\">(Hz)</span>\n",
       "                        </div> </th> </tr> </thead>\n",
       "            <tbody> <tr> <td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>6f6d8149-1603-9d2c-f884-0fdf995ec8b3</td>\n",
       "<td>SpikeGLX</td>\n",
       "<td>30000.6</td> </tr> </tbody>\n",
       "        </table>\n",
       "        \n",
       "        <p>Total: 1</p></div>\n",
       "        "
      ],
      "text/plain": [
       "*subject     *session_datet *insertion_num electrode_conf acq_software   sampling_rate \n",
       "+----------+ +------------+ +------------+ +------------+ +------------+ +------------+\n",
       "subject6     2021-01-15 11: 0              6f6d8149-1603- SpikeGLX       30000.6       \n",
       " (Total: 1)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ephys.EphysRecording() & session_key"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. **ephys_element.EphysRecording.EphysFile**\n",
    "\n",
    "The table `EphysFile` only saves the meta file from the recording"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    \n",
       "    <style type=\"text/css\">\n",
       "        .Relation{\n",
       "            border-collapse:collapse;\n",
       "        }\n",
       "        .Relation th{\n",
       "            background: #A0A0A0; color: #ffffff; padding:4px; border:#f0e0e0 1px solid;\n",
       "            font-weight: normal; font-family: monospace; font-size: 100%;\n",
       "        }\n",
       "        .Relation td{\n",
       "            padding:4px; border:#f0e0e0 1px solid; font-size:100%;\n",
       "        }\n",
       "        .Relation tr:nth-child(odd){\n",
       "            background: #ffffff;\n",
       "        }\n",
       "        .Relation tr:nth-child(even){\n",
       "            background: #f3f1ff;\n",
       "        }\n",
       "        /* Tooltip container */\n",
       "        .djtooltip {\n",
       "        }\n",
       "        /* Tooltip text */\n",
       "        .djtooltip .djtooltiptext {\n",
       "            visibility: hidden;\n",
       "            width: 120px;\n",
       "            background-color: black;\n",
       "            color: #fff;\n",
       "            text-align: center;\n",
       "            padding: 5px 0;\n",
       "            border-radius: 6px;\n",
       "            /* Position the tooltip text - see examples below! */\n",
       "            position: absolute;\n",
       "            z-index: 1;\n",
       "        }\n",
       "        #primary {\n",
       "            font-weight: bold;\n",
       "            color: black;\n",
       "        }\n",
       "        #nonprimary {\n",
       "            font-weight: normal;\n",
       "            color: white;\n",
       "        }\n",
       "\n",
       "        /* Show the tooltip text when you mouse over the tooltip container */\n",
       "        .djtooltip:hover .djtooltiptext {\n",
       "            visibility: visible;\n",
       "        }\n",
       "    </style>\n",
       "    \n",
       "    <b>Paths of files of a given EphysRecording round.</b>\n",
       "        <div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "        <table border=\"1\" class=\"Relation\">\n",
       "            <thead> <tr style=\"text-align: right;\"> <th> <div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">subject</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">session_datetime</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">insertion_number</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">file_path</p>\n",
       "                            <span class=\"djtooltiptext\">filepath relative to root data directory</span>\n",
       "                        </div> </th> </tr> </thead>\n",
       "            <tbody> <tr> <td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>subject6/session1/towersTask_g0_imec0/towersTask_g0_t0.imec0.ap.meta</td> </tr> </tbody>\n",
       "        </table>\n",
       "        \n",
       "        <p>Total: 1</p></div>\n",
       "        "
      ],
      "text/plain": [
       "*subject     *session_datet *insertion_num *file_path    \n",
       "+----------+ +------------+ +------------+ +------------+\n",
       "subject6     2021-01-15 11: 0              subject6/sessi\n",
       " (Total: 1)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ephys.EphysRecording.EphysFile() & session_key"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create ClusteringTask and run/validate Clustering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": "<svg height=\"185pt\" viewBox=\"0.00 0.00 330.13 185.00\" width=\"330pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n<g class=\"graph\" id=\"graph0\" transform=\"scale(1 1) rotate(0) translate(4 181)\">\n<title>%3</title>\n<polygon fill=\"white\" points=\"-4,4 -4,-181 326.125,-181 326.125,4 -4,4\" stroke=\"none\"/>\n<!-- ephys.ClusteringTask -->\n<g class=\"node\" id=\"node1\"><title>ephys.ClusteringTask</title>\n<g id=\"a_node1\"><a xlink:title=\"→ ephys.EphysRecording\r\n→ ephys.ClusteringParamSet\r\n------------------------------\r\nclustering_output_dir \r\ntask_mode=&quot;load&quot;     \r\n\">\n<polygon fill=\"#00ff00\" fill-opacity=\"0.188235\" points=\"234.625,-106 106.625,-106 106.625,-71 234.625,-71 234.625,-106\" stroke=\"#00ff00\" stroke-opacity=\"0.188235\"/>\n<text fill=\"darkgreen\" font-family=\"arial\" font-size=\"12.00\" text-anchor=\"middle\" x=\"170.625\" y=\"-85.4\">ephys.ClusteringTask</text>\n</a>\n</g>\n</g>\n<!-- ephys.Clustering -->\n<g class=\"node\" id=\"node3\"><title>ephys.Clustering</title>\n<g id=\"a_node3\"><a xlink:title=\"→ ephys.ClusteringTask\r\n------------------------------\r\nclustering_time      \r\npackage_version=&quot;&quot;   \r\n\">\n<ellipse cx=\"170.625\" cy=\"-17.5\" fill=\"#00007f\" fill-opacity=\"0.250980\" rx=\"64.2502\" ry=\"17.5\" stroke=\"#00007f\" stroke-opacity=\"0.250980\"/>\n<text fill=\"#00007f\" font-family=\"arial\" font-size=\"12.00\" text-anchor=\"middle\" x=\"170.625\" y=\"-14.4\">ephys.Clustering</text>\n</a>\n</g>\n</g>\n<!-- ephys.ClusteringTask&#45;&gt;ephys.Clustering -->\n<g class=\"edge\" id=\"edge1\"><title>ephys.ClusteringTask-&gt;ephys.Clustering</title>\n<path d=\"M170.625,-70.797C170.625,-59.9485 170.625,-45.8669 170.625,-35.0492\" fill=\"none\" stroke=\"#000000\" stroke-opacity=\"0.250980\" stroke-width=\"2\"/>\n</g>\n<!-- ephys.EphysRecording -->\n<g class=\"node\" id=\"node2\"><title>ephys.EphysRecording</title>\n<g id=\"a_node2\"><a xlink:title=\"→ ephys.ProbeInsertion\r\n------------------------------\r\n→ probe.ElectrodeConfig\r\n→ ephys.AcquisitionSoftware\r\nsampling_rate        \r\n\">\n<ellipse cx=\"85.625\" cy=\"-159.5\" fill=\"#00007f\" fill-opacity=\"0.250980\" rx=\"85.7502\" ry=\"17.5\" stroke=\"#00007f\" stroke-opacity=\"0.250980\"/>\n<text fill=\"#00007f\" font-family=\"arial\" font-size=\"12.00\" text-anchor=\"middle\" x=\"85.625\" y=\"-156.4\">ephys.EphysRecording</text>\n</a>\n</g>\n</g>\n<!-- ephys.EphysRecording&#45;&gt;ephys.ClusteringTask -->\n<g class=\"edge\" id=\"edge2\"><title>ephys.EphysRecording-&gt;ephys.ClusteringTask</title>\n<path d=\"M105.769,-142.148C119.122,-131.308 136.592,-117.127 150.035,-106.214\" fill=\"none\" stroke=\"#000000\" stroke-opacity=\"0.250980\" stroke-width=\"0.75\"/>\n</g>\n<!-- ephys.ClusteringParamSet -->\n<g class=\"node\" id=\"node4\"><title>ephys.ClusteringParamSet</title>\n<g id=\"a_node4\"><a xlink:title=\"paramset_idx         \r\n------------------------------\r\n→ ephys.ClusteringMethod\r\nparamset_desc        \r\nparam_set_hash       \r\nparams               \r\nUNIQUE INDEX (param_set_hash)\r\n\">\n<polygon fill=\"#000000\" fill-opacity=\"0.125490\" points=\"322.125,-177 189.125,-177 189.125,-142 322.125,-142 322.125,-177\" stroke=\"none\"/>\n<text font-family=\"arial\" font-size=\"10.00\" text-anchor=\"start\" text-decoration=\"underline\" x=\"197.125\" y=\"-158\">ephys.ClusteringParamSet</text>\n</a>\n</g>\n</g>\n<!-- ephys.ClusteringParamSet&#45;&gt;ephys.ClusteringTask -->\n<g class=\"edge\" id=\"edge3\"><title>ephys.ClusteringParamSet-&gt;ephys.ClusteringTask</title>\n<path d=\"M235.049,-141.797C221.685,-130.949 204.338,-116.867 191.012,-106.049\" fill=\"none\" stroke=\"#000000\" stroke-opacity=\"0.250980\" stroke-width=\"0.75\"/>\n</g>\n</g>\n</svg>",
      "text/plain": [
       "<datajoint.diagram.Diagram at 0x1ffb83e9648>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(\n",
    "    dj.Diagram(ephys.EphysRecording)\n",
    "    + ephys.ClusteringParamSet\n",
    "    + ephys.ClusteringTask\n",
    "    + ephys.Clustering\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next major table in the ephys pipeline is the `ClusteringTask`.\n",
    "\n",
    "+ An entry in `ClusteringTask` indicates a set of clustering results generated from Kilosort2 outside `workflow-array-ephys` are ready be ingested. In a future release, an entry in `ClusteringTask` can also indicate a new Kilosort2 clustering job is ready to be triggered.\n",
    "\n",
    "+ The `ClusteringTask` table depends on the table `ClusteringParamSet`, which are the parameters of the clustering task and needed to be ingested first."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A method of the class `ClusteringParamSet` called `insert_new_params` helps on the insertion of a parameter set and ensures the inserted one is not duplicated with existing parameter sets in the database.\n",
    "\n",
    "The following parameters' values are set based on [Kilosort StandardConfig file](https://github.com/MouseLand/Kilosort/tree/main/configFiles)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    \n",
       "    <style type=\"text/css\">\n",
       "        .Relation{\n",
       "            border-collapse:collapse;\n",
       "        }\n",
       "        .Relation th{\n",
       "            background: #A0A0A0; color: #ffffff; padding:4px; border:#f0e0e0 1px solid;\n",
       "            font-weight: normal; font-family: monospace; font-size: 100%;\n",
       "        }\n",
       "        .Relation td{\n",
       "            padding:4px; border:#f0e0e0 1px solid; font-size:100%;\n",
       "        }\n",
       "        .Relation tr:nth-child(odd){\n",
       "            background: #ffffff;\n",
       "        }\n",
       "        .Relation tr:nth-child(even){\n",
       "            background: #f3f1ff;\n",
       "        }\n",
       "        /* Tooltip container */\n",
       "        .djtooltip {\n",
       "        }\n",
       "        /* Tooltip text */\n",
       "        .djtooltip .djtooltiptext {\n",
       "            visibility: hidden;\n",
       "            width: 120px;\n",
       "            background-color: black;\n",
       "            color: #fff;\n",
       "            text-align: center;\n",
       "            padding: 5px 0;\n",
       "            border-radius: 6px;\n",
       "            /* Position the tooltip text - see examples below! */\n",
       "            position: absolute;\n",
       "            z-index: 1;\n",
       "        }\n",
       "        #primary {\n",
       "            font-weight: bold;\n",
       "            color: black;\n",
       "        }\n",
       "        #nonprimary {\n",
       "            font-weight: normal;\n",
       "            color: white;\n",
       "        }\n",
       "\n",
       "        /* Show the tooltip text when you mouse over the tooltip container */\n",
       "        .djtooltip:hover .djtooltiptext {\n",
       "            visibility: visible;\n",
       "        }\n",
       "    </style>\n",
       "    \n",
       "    <b>Parameter set to be used in a clustering procedure</b>\n",
       "        <div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "        <table border=\"1\" class=\"Relation\">\n",
       "            <thead> <tr style=\"text-align: right;\"> <th> <div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">paramset_idx</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">clustering_method</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">paramset_desc</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">param_set_hash</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">params</p>\n",
       "                            <span class=\"djtooltiptext\">dictionary of all applicable parameters</span>\n",
       "                        </div> </th> </tr> </thead>\n",
       "            <tbody> <tr> <td>0</td>\n",
       "<td>kilosort2</td>\n",
       "<td>Spike sorting using Kilosort2</td>\n",
       "<td>de78cee1-526f-319e-b6d5-8a2ba04963d8</td>\n",
       "<td>=BLOB=</td> </tr> </tbody>\n",
       "        </table>\n",
       "        \n",
       "        <p>Total: 1</p></div>\n",
       "        "
      ],
      "text/plain": [
       "*paramset_idx  clustering_met paramset_desc  param_set_hash params    \n",
       "+------------+ +------------+ +------------+ +------------+ +--------+\n",
       "0              kilosort2      Spike sorting  de78cee1-526f- =BLOB=    \n",
       " (Total: 1)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# insert clustering task manually\n",
    "params_ks = {\n",
    "    \"fs\": 30000,\n",
    "    \"fshigh\": 150,\n",
    "    \"minfr_goodchannels\": 0.1,\n",
    "    \"Th\": [10, 4],\n",
    "    \"lam\": 10,\n",
    "    \"AUCsplit\": 0.9,\n",
    "    \"minFR\": 0.02,\n",
    "    \"momentum\": [20, 400],\n",
    "    \"sigmaMask\": 30,\n",
    "    \"ThPr\": 8,\n",
    "    \"spkTh\": -6,\n",
    "    \"reorder\": 1,\n",
    "    \"nskip\": 25,\n",
    "    \"GPU\": 1,\n",
    "    \"Nfilt\": 1024,\n",
    "    \"nfilt_factor\": 4,\n",
    "    \"ntbuff\": 64,\n",
    "    \"whiteningRange\": 32,\n",
    "    \"nSkipCov\": 25,\n",
    "    \"scaleproc\": 200,\n",
    "    \"nPCs\": 3,\n",
    "    \"useRAM\": 0,\n",
    "}\n",
    "ephys.ClusteringParamSet.insert_new_params(\n",
    "    clustering_method=\"kilosort2\",\n",
    "    paramset_idx=0,\n",
    "    params=params_ks,\n",
    "    paramset_desc=\"Spike sorting using Kilosort2\",\n",
    ")\n",
    "ephys.ClusteringParamSet()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are then able to insert an entry into the `ClusteringTask` table. One important field of the table is `clustering_output_dir`, which specifies the Kilosort2 output directory for the later processing.\n",
    "**Note**: this output dir is a relative path to be combined with `ephys_root_directory` in the config file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# Manual table for defining a clustering task ready to be run\n",
      "-> ephys.EphysRecording\n",
      "-> ephys.ClusteringParamSet\n",
      "---\n",
      "clustering_output_dir : varchar(255)                 # clustering output directory relative to the clustering root data directory\n",
      "task_mode=\"load\"     : enum('load','trigger')       # 'load': load computed analysis results, 'trigger': trigger computation\n",
      "\n"
     ]
    }
   ],
   "source": [
    "ephys.ClusteringTask.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "# Manual table for defining a clustering task ready to be run\n",
       "subject              : varchar(32)                  # \n",
       "session_datetime     : datetime(3)                  # \n",
       "insertion_number     : tinyint unsigned             # \n",
       "paramset_idx         : smallint                     # \n",
       "---\n",
       "clustering_output_dir : varchar(255)                 # clustering output directory relative to the clustering root data directory\n",
       "task_mode=\"load\"     : enum('load','trigger')       # 'load': load computed analysis results, 'trigger': trigger computation"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ephys.ClusteringTask.heading"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "ephys.ClusteringTask.insert1(\n",
    "    dict(\n",
    "        session_key,\n",
    "        insertion_number=0,\n",
    "        paramset_idx=0,\n",
    "        clustering_output_dir=\"subject6/session1/towersTask_g0_imec0\",\n",
    "    ),\n",
    "    skip_duplicates=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    \n",
       "    <style type=\"text/css\">\n",
       "        .Relation{\n",
       "            border-collapse:collapse;\n",
       "        }\n",
       "        .Relation th{\n",
       "            background: #A0A0A0; color: #ffffff; padding:4px; border:#f0e0e0 1px solid;\n",
       "            font-weight: normal; font-family: monospace; font-size: 100%;\n",
       "        }\n",
       "        .Relation td{\n",
       "            padding:4px; border:#f0e0e0 1px solid; font-size:100%;\n",
       "        }\n",
       "        .Relation tr:nth-child(odd){\n",
       "            background: #ffffff;\n",
       "        }\n",
       "        .Relation tr:nth-child(even){\n",
       "            background: #f3f1ff;\n",
       "        }\n",
       "        /* Tooltip container */\n",
       "        .djtooltip {\n",
       "        }\n",
       "        /* Tooltip text */\n",
       "        .djtooltip .djtooltiptext {\n",
       "            visibility: hidden;\n",
       "            width: 120px;\n",
       "            background-color: black;\n",
       "            color: #fff;\n",
       "            text-align: center;\n",
       "            padding: 5px 0;\n",
       "            border-radius: 6px;\n",
       "            /* Position the tooltip text - see examples below! */\n",
       "            position: absolute;\n",
       "            z-index: 1;\n",
       "        }\n",
       "        #primary {\n",
       "            font-weight: bold;\n",
       "            color: black;\n",
       "        }\n",
       "        #nonprimary {\n",
       "            font-weight: normal;\n",
       "            color: white;\n",
       "        }\n",
       "\n",
       "        /* Show the tooltip text when you mouse over the tooltip container */\n",
       "        .djtooltip:hover .djtooltiptext {\n",
       "            visibility: visible;\n",
       "        }\n",
       "    </style>\n",
       "    \n",
       "    <b>Manual table for defining a clustering task ready to be run</b>\n",
       "        <div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "        <table border=\"1\" class=\"Relation\">\n",
       "            <thead> <tr style=\"text-align: right;\"> <th> <div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">subject</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">session_datetime</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">insertion_number</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">paramset_idx</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">clustering_output_dir</p>\n",
       "                            <span class=\"djtooltiptext\">clustering output directory relative to the clustering root data directory</span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">task_mode</p>\n",
       "                            <span class=\"djtooltiptext\">'load': load computed analysis results, 'trigger': trigger computation</span>\n",
       "                        </div> </th> </tr> </thead>\n",
       "            <tbody> <tr> <td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>0</td>\n",
       "<td>subject6/session1/towersTask_g0_imec0</td>\n",
       "<td>load</td> </tr> </tbody>\n",
       "        </table>\n",
       "        \n",
       "        <p>Total: 1</p></div>\n",
       "        "
      ],
      "text/plain": [
       "*subject     *session_datet *insertion_num *paramset_idx  clustering_out task_mode    \n",
       "+----------+ +------------+ +------------+ +------------+ +------------+ +-----------+\n",
       "subject6     2021-01-15 11: 0              0              subject6/sessi load         \n",
       " (Total: 1)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ephys.ClusteringTask() & session_key"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are then able to populate the clustering results. The `Clustering` table now validates the Kilosort2 outcomes before ingesting the spike sorted results. In a future release of `element-array-ephys`, this table may be used to trigger a Kilosort2 process. A record in the `Clustering` indicates that Kilosort2 job is done successfully and the results are ready to be processed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Clustering: 100%|████████████████████████████████████████████████████████████████████████| 1/1 [00:02<00:00,  2.75s/it]\n"
     ]
    }
   ],
   "source": [
    "ephys.Clustering.populate(display_progress=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    \n",
       "    <style type=\"text/css\">\n",
       "        .Relation{\n",
       "            border-collapse:collapse;\n",
       "        }\n",
       "        .Relation th{\n",
       "            background: #A0A0A0; color: #ffffff; padding:4px; border:#f0e0e0 1px solid;\n",
       "            font-weight: normal; font-family: monospace; font-size: 100%;\n",
       "        }\n",
       "        .Relation td{\n",
       "            padding:4px; border:#f0e0e0 1px solid; font-size:100%;\n",
       "        }\n",
       "        .Relation tr:nth-child(odd){\n",
       "            background: #ffffff;\n",
       "        }\n",
       "        .Relation tr:nth-child(even){\n",
       "            background: #f3f1ff;\n",
       "        }\n",
       "        /* Tooltip container */\n",
       "        .djtooltip {\n",
       "        }\n",
       "        /* Tooltip text */\n",
       "        .djtooltip .djtooltiptext {\n",
       "            visibility: hidden;\n",
       "            width: 120px;\n",
       "            background-color: black;\n",
       "            color: #fff;\n",
       "            text-align: center;\n",
       "            padding: 5px 0;\n",
       "            border-radius: 6px;\n",
       "            /* Position the tooltip text - see examples below! */\n",
       "            position: absolute;\n",
       "            z-index: 1;\n",
       "        }\n",
       "        #primary {\n",
       "            font-weight: bold;\n",
       "            color: black;\n",
       "        }\n",
       "        #nonprimary {\n",
       "            font-weight: normal;\n",
       "            color: white;\n",
       "        }\n",
       "\n",
       "        /* Show the tooltip text when you mouse over the tooltip container */\n",
       "        .djtooltip:hover .djtooltiptext {\n",
       "            visibility: visible;\n",
       "        }\n",
       "    </style>\n",
       "    \n",
       "    <b>Clustering Procedure</b>\n",
       "        <div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "        <table border=\"1\" class=\"Relation\">\n",
       "            <thead> <tr style=\"text-align: right;\"> <th> <div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">subject</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">session_datetime</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">insertion_number</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">paramset_idx</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">clustering_time</p>\n",
       "                            <span class=\"djtooltiptext\">time of generation of this set of clustering results</span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">package_version</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div> </th> </tr> </thead>\n",
       "            <tbody> <tr> <td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>0</td>\n",
       "<td>2021-02-22 23:07:16</td>\n",
       "<td></td> </tr> </tbody>\n",
       "        </table>\n",
       "        \n",
       "        <p>Total: 1</p></div>\n",
       "        "
      ],
      "text/plain": [
       "*subject     *session_datet *insertion_num *paramset_idx  clustering_tim package_versio\n",
       "+----------+ +------------+ +------------+ +------------+ +------------+ +------------+\n",
       "subject6     2021-01-15 11: 0              0              2021-02-22 23:               \n",
       " (Total: 1)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ephys.Clustering() & session_key"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import clustering results and manually curated results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now ready to ingest the clustering results (spike times etc.) into the database. For the `no-curation` mode, these clustering results directly from Kilosort2. For more information on manual curation, please visit the [electrophysiology description page](https://elements.datajoint.org/description/array_ephys/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"194pt\" height=\"240pt\" viewBox=\"0.00 0.00 194.25 240.00\">\n<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 236)\">\n<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-236 190.25,-236 190.25,4 -4,4\"/>\n<!-- ephys.Clustering -->\n<g id=\"node1\" class=\"node\">\n<title>ephys.Clustering</title>\n<g id=\"a_node1\"><a xlink:title=\"→ ephys.ClusteringTask\r------------------------------\rclustering_time      \rpackage_version=&quot;&quot;   \r\">\n<ellipse fill=\"#00007f\" fill-opacity=\"0.250980\" stroke=\"#00007f\" stroke-opacity=\"0.250980\" cx=\"93.12\" cy=\"-143.5\" rx=\"66.75\" ry=\"17.5\"/>\n<text text-anchor=\"middle\" x=\"93.12\" y=\"-140.4\" font-family=\"arial\" font-size=\"12.00\" fill=\"#00007f\" fill-opacity=\"0.627451\">ephys.Clustering</text>\n</a>\n</g>\n</g>\n<!-- ephys.CuratedClustering -->\n<g id=\"node3\" class=\"node\">\n<title>ephys.CuratedClustering</title>\n<g id=\"a_node3\"><a xlink:title=\"→ ephys.Clustering\r\">\n<ellipse fill=\"#00007f\" fill-opacity=\"0.250980\" stroke=\"#00007f\" stroke-opacity=\"0.250980\" cx=\"93.12\" cy=\"-72.5\" rx=\"93.25\" ry=\"17.5\"/>\n<text text-anchor=\"middle\" x=\"93.12\" y=\"-69.4\" font-family=\"arial\" font-size=\"12.00\" fill=\"#00007f\" fill-opacity=\"0.627451\">ephys.CuratedClustering</text>\n</a>\n</g>\n</g>\n<!-- ephys.Clustering&#45;&gt;ephys.CuratedClustering -->\n<g id=\"edge1\" class=\"edge\">\n<title>ephys.Clustering-&gt;ephys.CuratedClustering</title>\n<path fill=\"none\" stroke=\"#000000\" stroke-width=\"2\" stroke-opacity=\"0.250980\" d=\"M93.12,-125.8C93.12,-114.95 93.12,-100.87 93.12,-90.05\"/>\n</g>\n<!-- ephys.CuratedClustering.Unit -->\n<g id=\"node2\" class=\"node\">\n<title>ephys.CuratedClustering.Unit</title>\n<g id=\"a_node2\"><a xlink:title=\"→ ephys.CuratedClustering\runit                 \r------------------------------\r→ probe.ElectrodeConfig.Electrode\r→ ephys.ClusterQualityLabel\rspike_count          \rspike_times          \rspike_sites          \rspike_depths         \r\">\n<polygon fill=\"transparent\" stroke=\"transparent\" points=\"166.12,-19 20.12,-19 20.12,0 166.12,0 166.12,-19\"/>\n<text text-anchor=\"start\" x=\"28.12\" y=\"-8\" font-family=\"arial\" text-decoration=\"underline\" font-size=\"10.00\">ephys.CuratedClustering.Unit</text>\n</a>\n</g>\n</g>\n<!-- ephys.CuratedClustering&#45;&gt;ephys.CuratedClustering.Unit -->\n<g id=\"edge2\" class=\"edge\">\n<title>ephys.CuratedClustering-&gt;ephys.CuratedClustering.Unit</title>\n<path fill=\"none\" stroke=\"#000000\" stroke-width=\"0.75\" stroke-opacity=\"0.250980\" d=\"M93.12,-54.83C93.12,-43.26 93.12,-28.32 93.12,-19.08\"/>\n</g>\n<!-- ephys.ClusteringTask -->\n<g id=\"node4\" class=\"node\">\n<title>ephys.ClusteringTask</title>\n<g id=\"a_node4\"><a xlink:title=\"→ ephys.EphysRecording\r→ ephys.ClusteringParamSet\r------------------------------\rclustering_output_dir=&quot;&quot; \rtask_mode=&quot;load&quot;     \r\">\n<polygon fill=\"#00ff00\" fill-opacity=\"0.188235\" stroke=\"#00ff00\" stroke-opacity=\"0.188235\" points=\"158.62,-232 27.62,-232 27.62,-197 158.62,-197 158.62,-232\"/>\n<text text-anchor=\"middle\" x=\"93.12\" y=\"-211.4\" font-family=\"arial\" font-size=\"12.00\" fill=\"darkgreen\">ephys.ClusteringTask</text>\n</a>\n</g>\n</g>\n<!-- ephys.ClusteringTask&#45;&gt;ephys.Clustering -->\n<g id=\"edge3\" class=\"edge\">\n<title>ephys.ClusteringTask-&gt;ephys.Clustering</title>\n<path fill=\"none\" stroke=\"#000000\" stroke-width=\"2\" stroke-opacity=\"0.250980\" d=\"M93.12,-196.8C93.12,-185.95 93.12,-171.87 93.12,-161.05\"/>\n</g>\n</g>\n</svg>",
      "text/plain": [
       "<datajoint.diagram.Diagram at 0x10b161fa0>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(\n",
    "    dj.Diagram(ephys.ClusteringTask)\n",
    "    + dj.Diagram(ephys.Clustering)\n",
    "    + dj.Diagram(ephys.CuratedClustering)\n",
    "    + dj.Diagram(ephys.CuratedClustering.Unit)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we populate `CuratedClustering`, ingesting the output of Kilosort2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "ephys.CuratedClustering.populate(session_key, display_progress=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The part table `CuratedClustering.Unit` contains the spike sorted units"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    \n",
       "    <style type=\"text/css\">\n",
       "        .Relation{\n",
       "            border-collapse:collapse;\n",
       "        }\n",
       "        .Relation th{\n",
       "            background: #A0A0A0; color: #ffffff; padding:4px; border:#f0e0e0 1px solid;\n",
       "            font-weight: normal; font-family: monospace; font-size: 100%;\n",
       "        }\n",
       "        .Relation td{\n",
       "            padding:4px; border:#f0e0e0 1px solid; font-size:100%;\n",
       "        }\n",
       "        .Relation tr:nth-child(odd){\n",
       "            background: #ffffff;\n",
       "        }\n",
       "        .Relation tr:nth-child(even){\n",
       "            background: #f3f1ff;\n",
       "        }\n",
       "        /* Tooltip container */\n",
       "        .djtooltip {\n",
       "        }\n",
       "        /* Tooltip text */\n",
       "        .djtooltip .djtooltiptext {\n",
       "            visibility: hidden;\n",
       "            width: 120px;\n",
       "            background-color: black;\n",
       "            color: #fff;\n",
       "            text-align: center;\n",
       "            padding: 5px 0;\n",
       "            border-radius: 6px;\n",
       "            /* Position the tooltip text - see examples below! */\n",
       "            position: absolute;\n",
       "            z-index: 1;\n",
       "        }\n",
       "        #primary {\n",
       "            font-weight: bold;\n",
       "            color: black;\n",
       "        }\n",
       "        #nonprimary {\n",
       "            font-weight: normal;\n",
       "            color: white;\n",
       "        }\n",
       "\n",
       "        /* Show the tooltip text when you mouse over the tooltip container */\n",
       "        .djtooltip:hover .djtooltiptext {\n",
       "            visibility: visible;\n",
       "        }\n",
       "    </style>\n",
       "    \n",
       "    <b>Properties of a given unit from a round of clustering (and curation)</b>\n",
       "        <div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "        <table border=\"1\" class=\"Relation\">\n",
       "            <thead> <tr style=\"text-align: right;\"> <th> <div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">subject</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">session_datetime</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">insertion_number</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">paramset_idx</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">unit</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">electrode_config_hash</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">probe_type</p>\n",
       "                            <span class=\"djtooltiptext\">e.g. neuropixels_1.0</span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">electrode</p>\n",
       "                            <span class=\"djtooltiptext\">electrode index, starts at 0</span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">cluster_quality_label</p>\n",
       "                            <span class=\"djtooltiptext\">cluster quality type - e.g. 'good', 'MUA', 'noise', etc.</span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">spike_count</p>\n",
       "                            <span class=\"djtooltiptext\">how many spikes in this recording for this unit</span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">spike_times</p>\n",
       "                            <span class=\"djtooltiptext\">(s) spike times of this unit, relative to the start of the EphysRecording</span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">spike_sites</p>\n",
       "                            <span class=\"djtooltiptext\">array of electrode associated with each spike</span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">spike_depths</p>\n",
       "                            <span class=\"djtooltiptext\">(um) array of depths associated with each spike, relative to the (0, 0) of the probe</span>\n",
       "                        </div> </th> </tr> </thead>\n",
       "            <tbody> <tr> <td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>0</td>\n",
       "<td>0</td>\n",
       "<td>6f6d8149-1603-9d2c-f884-0fdf995ec8b3</td>\n",
       "<td>neuropixels 1.0 - 3B</td>\n",
       "<td>0</td>\n",
       "<td>mua</td>\n",
       "<td>1227</td>\n",
       "<td>=BLOB=</td>\n",
       "<td>=BLOB=</td>\n",
       "<td>=BLOB=</td></tr><tr><td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>0</td>\n",
       "<td>1</td>\n",
       "<td>6f6d8149-1603-9d2c-f884-0fdf995ec8b3</td>\n",
       "<td>neuropixels 1.0 - 3B</td>\n",
       "<td>0</td>\n",
       "<td>mua</td>\n",
       "<td>9390</td>\n",
       "<td>=BLOB=</td>\n",
       "<td>=BLOB=</td>\n",
       "<td>=BLOB=</td></tr><tr><td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>0</td>\n",
       "<td>2</td>\n",
       "<td>6f6d8149-1603-9d2c-f884-0fdf995ec8b3</td>\n",
       "<td>neuropixels 1.0 - 3B</td>\n",
       "<td>2</td>\n",
       "<td>mua</td>\n",
       "<td>34</td>\n",
       "<td>=BLOB=</td>\n",
       "<td>=BLOB=</td>\n",
       "<td>=BLOB=</td></tr><tr><td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>0</td>\n",
       "<td>3</td>\n",
       "<td>6f6d8149-1603-9d2c-f884-0fdf995ec8b3</td>\n",
       "<td>neuropixels 1.0 - 3B</td>\n",
       "<td>4</td>\n",
       "<td>mua</td>\n",
       "<td>4393</td>\n",
       "<td>=BLOB=</td>\n",
       "<td>=BLOB=</td>\n",
       "<td>=BLOB=</td></tr><tr><td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>0</td>\n",
       "<td>4</td>\n",
       "<td>6f6d8149-1603-9d2c-f884-0fdf995ec8b3</td>\n",
       "<td>neuropixels 1.0 - 3B</td>\n",
       "<td>5</td>\n",
       "<td>mua</td>\n",
       "<td>5535</td>\n",
       "<td>=BLOB=</td>\n",
       "<td>=BLOB=</td>\n",
       "<td>=BLOB=</td></tr><tr><td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>0</td>\n",
       "<td>5</td>\n",
       "<td>6f6d8149-1603-9d2c-f884-0fdf995ec8b3</td>\n",
       "<td>neuropixels 1.0 - 3B</td>\n",
       "<td>6</td>\n",
       "<td>mua</td>\n",
       "<td>20133</td>\n",
       "<td>=BLOB=</td>\n",
       "<td>=BLOB=</td>\n",
       "<td>=BLOB=</td></tr><tr><td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>0</td>\n",
       "<td>6</td>\n",
       "<td>6f6d8149-1603-9d2c-f884-0fdf995ec8b3</td>\n",
       "<td>neuropixels 1.0 - 3B</td>\n",
       "<td>6</td>\n",
       "<td>good</td>\n",
       "<td>8519</td>\n",
       "<td>=BLOB=</td>\n",
       "<td>=BLOB=</td>\n",
       "<td>=BLOB=</td></tr><tr><td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>0</td>\n",
       "<td>7</td>\n",
       "<td>6f6d8149-1603-9d2c-f884-0fdf995ec8b3</td>\n",
       "<td>neuropixels 1.0 - 3B</td>\n",
       "<td>8</td>\n",
       "<td>good</td>\n",
       "<td>10475</td>\n",
       "<td>=BLOB=</td>\n",
       "<td>=BLOB=</td>\n",
       "<td>=BLOB=</td></tr><tr><td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>0</td>\n",
       "<td>8</td>\n",
       "<td>6f6d8149-1603-9d2c-f884-0fdf995ec8b3</td>\n",
       "<td>neuropixels 1.0 - 3B</td>\n",
       "<td>9</td>\n",
       "<td>mua</td>\n",
       "<td>212</td>\n",
       "<td>=BLOB=</td>\n",
       "<td>=BLOB=</td>\n",
       "<td>=BLOB=</td></tr><tr><td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>0</td>\n",
       "<td>9</td>\n",
       "<td>6f6d8149-1603-9d2c-f884-0fdf995ec8b3</td>\n",
       "<td>neuropixels 1.0 - 3B</td>\n",
       "<td>11</td>\n",
       "<td>good</td>\n",
       "<td>30</td>\n",
       "<td>=BLOB=</td>\n",
       "<td>=BLOB=</td>\n",
       "<td>=BLOB=</td> </tr> </tbody>\n",
       "        </table>\n",
       "        <p>...</p>\n",
       "        <p>Total: 150</p></div>\n",
       "        "
      ],
      "text/plain": [
       "*subject     *session_datet *insertion_num *paramset_idx  *unit    electrode_conf probe_type     electrode     cluster_qualit spike_count    spike_time spike_site spike_dept\n",
       "+----------+ +------------+ +------------+ +------------+ +------+ +------------+ +------------+ +-----------+ +------------+ +------------+ +--------+ +--------+ +--------+\n",
       "subject6     2021-01-15 11: 0              0              0        6f6d8149-1603- neuropixels 1. 0             mua            1227           =BLOB=     =BLOB=     =BLOB=    \n",
       "subject6     2021-01-15 11: 0              0              1        6f6d8149-1603- neuropixels 1. 0             mua            9390           =BLOB=     =BLOB=     =BLOB=    \n",
       "subject6     2021-01-15 11: 0              0              2        6f6d8149-1603- neuropixels 1. 2             mua            34             =BLOB=     =BLOB=     =BLOB=    \n",
       "subject6     2021-01-15 11: 0              0              3        6f6d8149-1603- neuropixels 1. 4             mua            4393           =BLOB=     =BLOB=     =BLOB=    \n",
       "subject6     2021-01-15 11: 0              0              4        6f6d8149-1603- neuropixels 1. 5             mua            5535           =BLOB=     =BLOB=     =BLOB=    \n",
       "subject6     2021-01-15 11: 0              0              5        6f6d8149-1603- neuropixels 1. 6             mua            20133          =BLOB=     =BLOB=     =BLOB=    \n",
       "subject6     2021-01-15 11: 0              0              6        6f6d8149-1603- neuropixels 1. 6             good           8519           =BLOB=     =BLOB=     =BLOB=    \n",
       "subject6     2021-01-15 11: 0              0              7        6f6d8149-1603- neuropixels 1. 8             good           10475          =BLOB=     =BLOB=     =BLOB=    \n",
       "subject6     2021-01-15 11: 0              0              8        6f6d8149-1603- neuropixels 1. 9             mua            212            =BLOB=     =BLOB=     =BLOB=    \n",
       "subject6     2021-01-15 11: 0              0              9        6f6d8149-1603- neuropixels 1. 11            good           30             =BLOB=     =BLOB=     =BLOB=    \n",
       "   ...\n",
       " (Total: 150)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ephys.CuratedClustering.Unit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Populate LFP\n",
    "+ `LFP`: Mean local field potential across different electrodes.\n",
    "+ `LFP.Electrode`: Local field potential of a given electrode."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "title": "LFP and LFP.Electrode: By populating LFP, LFP of every other 9 electrode on the probe will be saved into table `ephys_element.LFP.Electrode` and an average LFP saved into table `ephys_element.LFP`"
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": "<svg height=\"169pt\" viewBox=\"0.00 0.00 179.25 169.00\" width=\"179pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n<g class=\"graph\" id=\"graph0\" transform=\"scale(1 1) rotate(0) translate(4 165)\">\n<title>%3</title>\n<polygon fill=\"white\" points=\"-4,4 -4,-165 175.25,-165 175.25,4 -4,4\" stroke=\"none\"/>\n<!-- ephys.LFP.Electrode -->\n<g class=\"node\" id=\"node1\"><title>ephys.LFP.Electrode</title>\n<g id=\"a_node1\"><a xlink:title=\"→ ephys.LFP\r\n→ probe.ElectrodeConfig.Electrode\r\n------------------------------\r\nlfp                  \r\n\">\n<polygon fill=\"none\" points=\"139.625,-19 31.625,-19 31.625,-0 139.625,-0 139.625,-19\" stroke=\"none\"/>\n<text font-family=\"arial\" font-size=\"10.00\" text-anchor=\"middle\" x=\"85.625\" y=\"-7\">ephys.LFP.Electrode</text>\n</a>\n</g>\n</g>\n<!-- ephys.EphysRecording -->\n<g class=\"node\" id=\"node2\"><title>ephys.EphysRecording</title>\n<g id=\"a_node2\"><a xlink:title=\"→ ephys.ProbeInsertion\r\n------------------------------\r\n→ probe.ElectrodeConfig\r\n→ ephys.AcquisitionSoftware\r\nsampling_rate        \r\n\">\n<ellipse cx=\"85.625\" cy=\"-143.5\" fill=\"#00007f\" fill-opacity=\"0.250980\" rx=\"85.7502\" ry=\"17.5\" stroke=\"#00007f\" stroke-opacity=\"0.250980\"/>\n<text fill=\"#00007f\" font-family=\"arial\" font-size=\"12.00\" text-anchor=\"middle\" x=\"85.625\" y=\"-140.4\">ephys.EphysRecording</text>\n</a>\n</g>\n</g>\n<!-- ephys.LFP -->\n<g class=\"node\" id=\"node3\"><title>ephys.LFP</title>\n<g id=\"a_node3\"><a xlink:title=\"→ ephys.EphysRecording\r\n------------------------------\r\nlfp_sampling_rate    \r\nlfp_time_stamps      \r\nlfp_mean             \r\n\">\n<ellipse cx=\"85.625\" cy=\"-72.5\" fill=\"#00007f\" fill-opacity=\"0.250980\" rx=\"45.7503\" ry=\"17.5\" stroke=\"#00007f\" stroke-opacity=\"0.250980\"/>\n<text fill=\"#00007f\" font-family=\"arial\" font-size=\"12.00\" text-anchor=\"middle\" x=\"85.625\" y=\"-69.4\">ephys.LFP</text>\n</a>\n</g>\n</g>\n<!-- ephys.EphysRecording&#45;&gt;ephys.LFP -->\n<g class=\"edge\" id=\"edge1\"><title>ephys.EphysRecording-&gt;ephys.LFP</title>\n<path d=\"M85.625,-125.797C85.625,-114.949 85.625,-100.867 85.625,-90.0492\" fill=\"none\" stroke=\"#000000\" stroke-opacity=\"0.250980\" stroke-width=\"2\"/>\n</g>\n<!-- ephys.LFP&#45;&gt;ephys.LFP.Electrode -->\n<g class=\"edge\" id=\"edge2\"><title>ephys.LFP-&gt;ephys.LFP.Electrode</title>\n<path d=\"M85.625,-54.8315C85.625,-43.2612 85.625,-28.3217 85.625,-19.0782\" fill=\"none\" stroke=\"#000000\" stroke-opacity=\"0.250980\" stroke-width=\"0.75\"/>\n</g>\n</g>\n</svg>",
      "text/plain": [
       "<datajoint.diagram.Diagram at 0x1ffb7d0f648>"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(\n",
    "    dj.Diagram(ephys.EphysRecording)\n",
    "    + dj.Diagram(ephys.LFP)\n",
    "    + dj.Diagram(ephys.LFP.Electrode)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "LFP: 100%|██████████████████████████████████████████████████████████████████████████████| 1/1 [02:31<00:00, 151.23s/it]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    \n",
       "    <style type=\"text/css\">\n",
       "        .Relation{\n",
       "            border-collapse:collapse;\n",
       "        }\n",
       "        .Relation th{\n",
       "            background: #A0A0A0; color: #ffffff; padding:4px; border:#f0e0e0 1px solid;\n",
       "            font-weight: normal; font-family: monospace; font-size: 100%;\n",
       "        }\n",
       "        .Relation td{\n",
       "            padding:4px; border:#f0e0e0 1px solid; font-size:100%;\n",
       "        }\n",
       "        .Relation tr:nth-child(odd){\n",
       "            background: #ffffff;\n",
       "        }\n",
       "        .Relation tr:nth-child(even){\n",
       "            background: #f3f1ff;\n",
       "        }\n",
       "        /* Tooltip container */\n",
       "        .djtooltip {\n",
       "        }\n",
       "        /* Tooltip text */\n",
       "        .djtooltip .djtooltiptext {\n",
       "            visibility: hidden;\n",
       "            width: 120px;\n",
       "            background-color: black;\n",
       "            color: #fff;\n",
       "            text-align: center;\n",
       "            padding: 5px 0;\n",
       "            border-radius: 6px;\n",
       "            /* Position the tooltip text - see examples below! */\n",
       "            position: absolute;\n",
       "            z-index: 1;\n",
       "        }\n",
       "        #primary {\n",
       "            font-weight: bold;\n",
       "            color: black;\n",
       "        }\n",
       "        #nonprimary {\n",
       "            font-weight: normal;\n",
       "            color: white;\n",
       "        }\n",
       "\n",
       "        /* Show the tooltip text when you mouse over the tooltip container */\n",
       "        .djtooltip:hover .djtooltiptext {\n",
       "            visibility: visible;\n",
       "        }\n",
       "    </style>\n",
       "    \n",
       "    <b>Acquired local field potential (LFP) from a given Ephys recording.</b>\n",
       "        <div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "        <table border=\"1\" class=\"Relation\">\n",
       "            <thead> <tr style=\"text-align: right;\"> <th> <div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">subject</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">session_datetime</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">insertion_number</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">lfp_sampling_rate</p>\n",
       "                            <span class=\"djtooltiptext\">(Hz)</span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">lfp_time_stamps</p>\n",
       "                            <span class=\"djtooltiptext\">(s) timestamps with respect to the start of the recording (recording_timestamp)</span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">lfp_mean</p>\n",
       "                            <span class=\"djtooltiptext\">(uV) mean of LFP across electrodes - shape (time,)</span>\n",
       "                        </div> </th> </tr> </thead>\n",
       "            <tbody> <tr> <td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>2500.05</td>\n",
       "<td>=BLOB=</td>\n",
       "<td>=BLOB=</td> </tr> </tbody>\n",
       "        </table>\n",
       "        \n",
       "        <p>Total: 1</p></div>\n",
       "        "
      ],
      "text/plain": [
       "*subject     *session_datet *insertion_num lfp_sampling_r lfp_time_s lfp_mean  \n",
       "+----------+ +------------+ +------------+ +------------+ +--------+ +--------+\n",
       "subject6     2021-01-15 11: 0              2500.05        =BLOB=     =BLOB=    \n",
       " (Total: 1)"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Takes a few minutes to populate\n",
    "ephys.LFP.populate(session_key, display_progress=True)\n",
    "ephys.LFP & session_key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    \n",
       "    <style type=\"text/css\">\n",
       "        .Relation{\n",
       "            border-collapse:collapse;\n",
       "        }\n",
       "        .Relation th{\n",
       "            background: #A0A0A0; color: #ffffff; padding:4px; border:#f0e0e0 1px solid;\n",
       "            font-weight: normal; font-family: monospace; font-size: 100%;\n",
       "        }\n",
       "        .Relation td{\n",
       "            padding:4px; border:#f0e0e0 1px solid; font-size:100%;\n",
       "        }\n",
       "        .Relation tr:nth-child(odd){\n",
       "            background: #ffffff;\n",
       "        }\n",
       "        .Relation tr:nth-child(even){\n",
       "            background: #f3f1ff;\n",
       "        }\n",
       "        /* Tooltip container */\n",
       "        .djtooltip {\n",
       "        }\n",
       "        /* Tooltip text */\n",
       "        .djtooltip .djtooltiptext {\n",
       "            visibility: hidden;\n",
       "            width: 120px;\n",
       "            background-color: black;\n",
       "            color: #fff;\n",
       "            text-align: center;\n",
       "            padding: 5px 0;\n",
       "            border-radius: 6px;\n",
       "            /* Position the tooltip text - see examples below! */\n",
       "            position: absolute;\n",
       "            z-index: 1;\n",
       "        }\n",
       "        #primary {\n",
       "            font-weight: bold;\n",
       "            color: black;\n",
       "        }\n",
       "        #nonprimary {\n",
       "            font-weight: normal;\n",
       "            color: white;\n",
       "        }\n",
       "\n",
       "        /* Show the tooltip text when you mouse over the tooltip container */\n",
       "        .djtooltip:hover .djtooltiptext {\n",
       "            visibility: visible;\n",
       "        }\n",
       "    </style>\n",
       "    \n",
       "    <b></b>\n",
       "        <div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "        <table border=\"1\" class=\"Relation\">\n",
       "            <thead> <tr style=\"text-align: right;\"> <th> <div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">subject</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">session_datetime</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">insertion_number</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">electrode_config_hash</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">probe_type</p>\n",
       "                            <span class=\"djtooltiptext\">e.g. neuropixels_1.0</span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">electrode</p>\n",
       "                            <span class=\"djtooltiptext\">electrode index, starts at 0</span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">lfp</p>\n",
       "                            <span class=\"djtooltiptext\">(uV) recorded lfp at this electrode</span>\n",
       "                        </div> </th> </tr> </thead>\n",
       "            <tbody> <tr> <td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>6f6d8149-1603-9d2c-f884-0fdf995ec8b3</td>\n",
       "<td>neuropixels 1.0 - 3B</td>\n",
       "<td>5</td>\n",
       "<td>=BLOB=</td></tr><tr><td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>6f6d8149-1603-9d2c-f884-0fdf995ec8b3</td>\n",
       "<td>neuropixels 1.0 - 3B</td>\n",
       "<td>14</td>\n",
       "<td>=BLOB=</td></tr><tr><td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>6f6d8149-1603-9d2c-f884-0fdf995ec8b3</td>\n",
       "<td>neuropixels 1.0 - 3B</td>\n",
       "<td>23</td>\n",
       "<td>=BLOB=</td></tr><tr><td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>6f6d8149-1603-9d2c-f884-0fdf995ec8b3</td>\n",
       "<td>neuropixels 1.0 - 3B</td>\n",
       "<td>32</td>\n",
       "<td>=BLOB=</td></tr><tr><td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>6f6d8149-1603-9d2c-f884-0fdf995ec8b3</td>\n",
       "<td>neuropixels 1.0 - 3B</td>\n",
       "<td>41</td>\n",
       "<td>=BLOB=</td></tr><tr><td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>6f6d8149-1603-9d2c-f884-0fdf995ec8b3</td>\n",
       "<td>neuropixels 1.0 - 3B</td>\n",
       "<td>50</td>\n",
       "<td>=BLOB=</td></tr><tr><td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>6f6d8149-1603-9d2c-f884-0fdf995ec8b3</td>\n",
       "<td>neuropixels 1.0 - 3B</td>\n",
       "<td>59</td>\n",
       "<td>=BLOB=</td></tr><tr><td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>6f6d8149-1603-9d2c-f884-0fdf995ec8b3</td>\n",
       "<td>neuropixels 1.0 - 3B</td>\n",
       "<td>68</td>\n",
       "<td>=BLOB=</td></tr><tr><td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>6f6d8149-1603-9d2c-f884-0fdf995ec8b3</td>\n",
       "<td>neuropixels 1.0 - 3B</td>\n",
       "<td>77</td>\n",
       "<td>=BLOB=</td></tr><tr><td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>6f6d8149-1603-9d2c-f884-0fdf995ec8b3</td>\n",
       "<td>neuropixels 1.0 - 3B</td>\n",
       "<td>86</td>\n",
       "<td>=BLOB=</td> </tr> </tbody>\n",
       "        </table>\n",
       "        <p>...</p>\n",
       "        <p>Total: 43</p></div>\n",
       "        "
      ],
      "text/plain": [
       "*subject     *session_datet *insertion_num *electrode_con *probe_type    *electrode    lfp       \n",
       "+----------+ +------------+ +------------+ +------------+ +------------+ +-----------+ +--------+\n",
       "subject6     2021-01-15 11: 0              6f6d8149-1603- neuropixels 1. 5             =BLOB=    \n",
       "subject6     2021-01-15 11: 0              6f6d8149-1603- neuropixels 1. 14            =BLOB=    \n",
       "subject6     2021-01-15 11: 0              6f6d8149-1603- neuropixels 1. 23            =BLOB=    \n",
       "subject6     2021-01-15 11: 0              6f6d8149-1603- neuropixels 1. 32            =BLOB=    \n",
       "subject6     2021-01-15 11: 0              6f6d8149-1603- neuropixels 1. 41            =BLOB=    \n",
       "subject6     2021-01-15 11: 0              6f6d8149-1603- neuropixels 1. 50            =BLOB=    \n",
       "subject6     2021-01-15 11: 0              6f6d8149-1603- neuropixels 1. 59            =BLOB=    \n",
       "subject6     2021-01-15 11: 0              6f6d8149-1603- neuropixels 1. 68            =BLOB=    \n",
       "subject6     2021-01-15 11: 0              6f6d8149-1603- neuropixels 1. 77            =BLOB=    \n",
       "subject6     2021-01-15 11: 0              6f6d8149-1603- neuropixels 1. 86            =BLOB=    \n",
       "   ...\n",
       " (Total: 43)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ephys.LFP.Electrode & session_key"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Populate Spike Waveform"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The current workflow also contain tables to save spike waveforms:\n",
    "+ `WaveformSet`: a table to drive the processing of all spikes waveforms resulting from a CuratedClustering.\n",
    "+ `WaveformSet.Waveform`: mean waveform across spikes for a given unit and electrode.\n",
    "+ `WaveformSet.PeakWaveform`: mean waveform across spikes for a given unit at the electrode with peak spike amplitude."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": "<svg height=\"169pt\" viewBox=\"0.00 0.00 356.63 169.00\" width=\"357pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n<g class=\"graph\" id=\"graph0\" transform=\"scale(1 1) rotate(0) translate(4 165)\">\n<title>%3</title>\n<polygon fill=\"white\" points=\"-4,4 -4,-165 352.625,-165 352.625,4 -4,4\" stroke=\"none\"/>\n<!-- ephys.WaveformSet -->\n<g class=\"node\" id=\"node1\"><title>ephys.WaveformSet</title>\n<g id=\"a_node1\"><a xlink:title=\"→ ephys.CuratedClustering\r\n\">\n<ellipse cx=\"92\" cy=\"-72.5\" fill=\"#00007f\" fill-opacity=\"0.250980\" rx=\"76.0008\" ry=\"17.5\" stroke=\"#00007f\" stroke-opacity=\"0.250980\"/>\n<text fill=\"#00007f\" font-family=\"arial\" font-size=\"12.00\" text-anchor=\"middle\" x=\"92\" y=\"-69.4\">ephys.WaveformSet</text>\n</a>\n</g>\n</g>\n<!-- ephys.WaveformSet.PeakWaveform -->\n<g class=\"node\" id=\"node2\"><title>ephys.WaveformSet.PeakWaveform</title>\n<g id=\"a_node2\"><a xlink:title=\"→ ephys.WaveformSet\r\n→ ephys.CuratedClustering.Unit\r\n------------------------------\r\npeak_electrode_waveform \r\n\">\n<polygon fill=\"none\" points=\"172,-19 0,-19 0,-0 172,-0 172,-19\" stroke=\"none\"/>\n<text font-family=\"arial\" font-size=\"10.00\" text-anchor=\"middle\" x=\"86\" y=\"-7\">ephys.WaveformSet.PeakWaveform</text>\n</a>\n</g>\n</g>\n<!-- ephys.WaveformSet&#45;&gt;ephys.WaveformSet.PeakWaveform -->\n<g class=\"edge\" id=\"edge1\"><title>ephys.WaveformSet-&gt;ephys.WaveformSet.PeakWaveform</title>\n<path d=\"M90.3605,-54.8315C89.2224,-43.2612 87.753,-28.3217 86.8438,-19.0782\" fill=\"none\" stroke=\"#000000\" stroke-opacity=\"0.250980\" stroke-width=\"0.75\"/>\n</g>\n<!-- ephys.WaveformSet.Waveform -->\n<g class=\"node\" id=\"node5\"><title>ephys.WaveformSet.Waveform</title>\n<g id=\"a_node5\"><a xlink:title=\"→ ephys.WaveformSet\r\n→ ephys.CuratedClustering.Unit\r\n→ probe.ElectrodeConfig.Electrode\r\n------------------------------\r\nwaveform_mean        \r\nwaveforms=null       \r\n\">\n<polygon fill=\"none\" points=\"339.5,-19 190.5,-19 190.5,-0 339.5,-0 339.5,-19\" stroke=\"none\"/>\n<text font-family=\"arial\" font-size=\"10.00\" text-anchor=\"middle\" x=\"265\" y=\"-7\">ephys.WaveformSet.Waveform</text>\n</a>\n</g>\n</g>\n<!-- ephys.WaveformSet&#45;&gt;ephys.WaveformSet.Waveform -->\n<g class=\"edge\" id=\"edge2\"><title>ephys.WaveformSet-&gt;ephys.WaveformSet.Waveform</title>\n<path d=\"M131.69,-57.5054C165.208,-45.6868 212.145,-29.1366 240.454,-19.1548\" fill=\"none\" stroke=\"#000000\" stroke-opacity=\"0.250980\" stroke-width=\"0.75\"/>\n</g>\n<!-- ephys.CuratedClustering.Unit -->\n<g class=\"node\" id=\"node3\"><title>ephys.CuratedClustering.Unit</title>\n<g id=\"a_node3\"><a xlink:title=\"→ ephys.CuratedClustering\r\nunit                 \r\n------------------------------\r\n→ probe.ElectrodeConfig.Electrode\r\n→ ephys.ClusterQualityLabel\r\nspike_count          \r\nspike_times          \r\nspike_sites          \r\nspike_depths         \r\n\">\n<polygon fill=\"none\" points=\"329.5,-82 186.5,-82 186.5,-63 329.5,-63 329.5,-82\" stroke=\"none\"/>\n<text font-family=\"arial\" font-size=\"10.00\" text-anchor=\"start\" text-decoration=\"underline\" x=\"194.5\" y=\"-71\">ephys.CuratedClustering.Unit</text>\n</a>\n</g>\n</g>\n<!-- ephys.CuratedClustering.Unit&#45;&gt;ephys.WaveformSet.PeakWaveform -->\n<g class=\"edge\" id=\"edge3\"><title>ephys.CuratedClustering.Unit-&gt;ephys.WaveformSet.PeakWaveform</title>\n<path d=\"M233.721,-62.8896C200.941,-51.2638 142.872,-30.6698 110.155,-19.0666\" fill=\"none\" stroke=\"#000000\" stroke-opacity=\"0.250980\" stroke-width=\"2\"/>\n</g>\n<!-- ephys.CuratedClustering.Unit&#45;&gt;ephys.WaveformSet.Waveform -->\n<g class=\"edge\" id=\"edge4\"><title>ephys.CuratedClustering.Unit-&gt;ephys.WaveformSet.Waveform</title>\n<path d=\"M259.003,-62.7602C260.334,-51.1641 262.669,-30.8115 263.999,-19.2236\" fill=\"none\" stroke=\"#000000\" stroke-opacity=\"0.250980\" stroke-width=\"0.75\"/>\n</g>\n<!-- ephys.CuratedClustering -->\n<g class=\"node\" id=\"node4\"><title>ephys.CuratedClustering</title>\n<g id=\"a_node4\"><a xlink:title=\"→ ephys.Curation\r\n\">\n<ellipse cx=\"258\" cy=\"-143.5\" fill=\"#00007f\" fill-opacity=\"0.250980\" rx=\"90.7502\" ry=\"17.5\" stroke=\"#00007f\" stroke-opacity=\"0.250980\"/>\n<text fill=\"#00007f\" font-family=\"arial\" font-size=\"12.00\" text-anchor=\"middle\" x=\"258\" y=\"-140.4\">ephys.CuratedClustering</text>\n</a>\n</g>\n</g>\n<!-- ephys.CuratedClustering&#45;&gt;ephys.WaveformSet -->\n<g class=\"edge\" id=\"edge5\"><title>ephys.CuratedClustering-&gt;ephys.WaveformSet</title>\n<path d=\"M221.571,-127.358C193.511,-115.695 154.954,-99.6674 127.206,-88.1338\" fill=\"none\" stroke=\"#000000\" stroke-opacity=\"0.250980\" stroke-width=\"2\"/>\n</g>\n<!-- ephys.CuratedClustering&#45;&gt;ephys.CuratedClustering.Unit -->\n<g class=\"edge\" id=\"edge6\"><title>ephys.CuratedClustering-&gt;ephys.CuratedClustering.Unit</title>\n<path d=\"M258,-125.797C258,-112.077 258,-93.1854 258,-82.2652\" fill=\"none\" stroke=\"#000000\" stroke-opacity=\"0.250980\" stroke-width=\"0.75\"/>\n</g>\n</g>\n</svg>",
      "text/plain": [
       "<datajoint.diagram.Diagram at 0x1ffb7bd0408>"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dj.Diagram(ephys.CuratedClustering) + dj.Diagram(ephys.WaveformSet) + 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "title": "The `probe_element.EelectrodeConfig` table conains the configuration information of the electrodes used, i.e. which 384 electrodes out of the total 960 on the probe were used in this ephys session, while the table `ephys_element.EphysRecording` specify which ElectrodeConfig is used in a particular ephys session."
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WaveformSet: 100%|█████████████████████████████████████████████████████████████████████| 1/1 [28:14<00:00, 1694.99s/it]\n"
     ]
    }
   ],
   "source": [
    "# Takes ~1h to populate for the test dataset\n",
    "ephys.WaveformSet.populate(session_key, display_progress=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    \n",
       "    <style type=\"text/css\">\n",
       "        .Relation{\n",
       "            border-collapse:collapse;\n",
       "        }\n",
       "        .Relation th{\n",
       "            background: #A0A0A0; color: #ffffff; padding:4px; border:#f0e0e0 1px solid;\n",
       "            font-weight: normal; font-family: monospace; font-size: 100%;\n",
       "        }\n",
       "        .Relation td{\n",
       "            padding:4px; border:#f0e0e0 1px solid; font-size:100%;\n",
       "        }\n",
       "        .Relation tr:nth-child(odd){\n",
       "            background: #ffffff;\n",
       "        }\n",
       "        .Relation tr:nth-child(even){\n",
       "            background: #f3f1ff;\n",
       "        }\n",
       "        /* Tooltip container */\n",
       "        .djtooltip {\n",
       "        }\n",
       "        /* Tooltip text */\n",
       "        .djtooltip .djtooltiptext {\n",
       "            visibility: hidden;\n",
       "            width: 120px;\n",
       "            background-color: black;\n",
       "            color: #fff;\n",
       "            text-align: center;\n",
       "            padding: 5px 0;\n",
       "            border-radius: 6px;\n",
       "            /* Position the tooltip text - see examples below! */\n",
       "            position: absolute;\n",
       "            z-index: 1;\n",
       "        }\n",
       "        #primary {\n",
       "            font-weight: bold;\n",
       "            color: black;\n",
       "        }\n",
       "        #nonprimary {\n",
       "            font-weight: normal;\n",
       "            color: white;\n",
       "        }\n",
       "\n",
       "        /* Show the tooltip text when you mouse over the tooltip container */\n",
       "        .djtooltip:hover .djtooltiptext {\n",
       "            visibility: visible;\n",
       "        }\n",
       "    </style>\n",
       "    \n",
       "    <b>A set of spike waveforms for units out of a given CuratedClustering</b>\n",
       "        <div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "        <table border=\"1\" class=\"Relation\">\n",
       "            <thead> <tr style=\"text-align: right;\"> <th> <div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">subject</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">session_datetime</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">insertion_number</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">paramset_idx</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">curation_id</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div> </th> </tr> </thead>\n",
       "            <tbody> <tr> <td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>0</td>\n",
       "<td>1</td> </tr> </tbody>\n",
       "        </table>\n",
       "        \n",
       "        <p>Total: 1</p></div>\n",
       "        "
      ],
      "text/plain": [
       "*subject     *session_datet *insertion_num *paramset_idx  *curation_id  \n",
       "+----------+ +------------+ +------------+ +------------+ +------------+\n",
       "subject6     2021-01-15 11: 0              0              1             \n",
       " (Total: 1)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ephys.WaveformSet & session_key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    \n",
       "    <style type=\"text/css\">\n",
       "        .Relation{\n",
       "            border-collapse:collapse;\n",
       "        }\n",
       "        .Relation th{\n",
       "            background: #A0A0A0; color: #ffffff; padding:4px; border:#f0e0e0 1px solid;\n",
       "            font-weight: normal; font-family: monospace; font-size: 100%;\n",
       "        }\n",
       "        .Relation td{\n",
       "            padding:4px; border:#f0e0e0 1px solid; font-size:100%;\n",
       "        }\n",
       "        .Relation tr:nth-child(odd){\n",
       "            background: #ffffff;\n",
       "        }\n",
       "        .Relation tr:nth-child(even){\n",
       "            background: #f3f1ff;\n",
       "        }\n",
       "        /* Tooltip container */\n",
       "        .djtooltip {\n",
       "        }\n",
       "        /* Tooltip text */\n",
       "        .djtooltip .djtooltiptext {\n",
       "            visibility: hidden;\n",
       "            width: 120px;\n",
       "            background-color: black;\n",
       "            color: #fff;\n",
       "            text-align: center;\n",
       "            padding: 5px 0;\n",
       "            border-radius: 6px;\n",
       "            /* Position the tooltip text - see examples below! */\n",
       "            position: absolute;\n",
       "            z-index: 1;\n",
       "        }\n",
       "        #primary {\n",
       "            font-weight: bold;\n",
       "            color: black;\n",
       "        }\n",
       "        #nonprimary {\n",
       "            font-weight: normal;\n",
       "            color: white;\n",
       "        }\n",
       "\n",
       "        /* Show the tooltip text when you mouse over the tooltip container */\n",
       "        .djtooltip:hover .djtooltiptext {\n",
       "            visibility: visible;\n",
       "        }\n",
       "    </style>\n",
       "    \n",
       "    <b>Spike waveforms and their mean across spikes for the given unit</b>\n",
       "        <div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "        <table border=\"1\" class=\"Relation\">\n",
       "            <thead> <tr style=\"text-align: right;\"> <th> <div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">subject</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">session_datetime</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">insertion_number</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">paramset_idx</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">curation_id</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">unit</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">electrode_config_hash</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">probe_type</p>\n",
       "                            <span class=\"djtooltiptext\">e.g. neuropixels_1.0</span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">electrode</p>\n",
       "                            <span class=\"djtooltiptext\">electrode index, starts at 0</span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">waveform_mean</p>\n",
       "                            <span class=\"djtooltiptext\">(uV) mean waveform across spikes of the given unit</span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">waveforms</p>\n",
       "                            <span class=\"djtooltiptext\">(uV) (spike x sample) waveforms of a sampling of spikes at the given electrode for the given unit</span>\n",
       "                        </div> </th> </tr> </thead>\n",
       "            <tbody> <tr> <td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>0</td>\n",
       "<td>1</td>\n",
       "<td>0</td>\n",
       "<td>6f6d8149-1603-9d2c-f884-0fdf995ec8b3</td>\n",
       "<td>neuropixels 1.0 - 3B</td>\n",
       "<td>0</td>\n",
       "<td>=BLOB=</td>\n",
       "<td>=BLOB=</td></tr><tr><td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>0</td>\n",
       "<td>1</td>\n",
       "<td>0</td>\n",
       "<td>6f6d8149-1603-9d2c-f884-0fdf995ec8b3</td>\n",
       "<td>neuropixels 1.0 - 3B</td>\n",
       "<td>1</td>\n",
       "<td>=BLOB=</td>\n",
       "<td>=BLOB=</td></tr><tr><td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>0</td>\n",
       "<td>1</td>\n",
       "<td>0</td>\n",
       "<td>6f6d8149-1603-9d2c-f884-0fdf995ec8b3</td>\n",
       "<td>neuropixels 1.0 - 3B</td>\n",
       "<td>2</td>\n",
       "<td>=BLOB=</td>\n",
       "<td>=BLOB=</td></tr><tr><td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>0</td>\n",
       "<td>1</td>\n",
       "<td>0</td>\n",
       "<td>6f6d8149-1603-9d2c-f884-0fdf995ec8b3</td>\n",
       "<td>neuropixels 1.0 - 3B</td>\n",
       "<td>3</td>\n",
       "<td>=BLOB=</td>\n",
       "<td>=BLOB=</td></tr><tr><td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>0</td>\n",
       "<td>1</td>\n",
       "<td>0</td>\n",
       "<td>6f6d8149-1603-9d2c-f884-0fdf995ec8b3</td>\n",
       "<td>neuropixels 1.0 - 3B</td>\n",
       "<td>4</td>\n",
       "<td>=BLOB=</td>\n",
       "<td>=BLOB=</td></tr><tr><td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>0</td>\n",
       "<td>1</td>\n",
       "<td>0</td>\n",
       "<td>6f6d8149-1603-9d2c-f884-0fdf995ec8b3</td>\n",
       "<td>neuropixels 1.0 - 3B</td>\n",
       "<td>5</td>\n",
       "<td>=BLOB=</td>\n",
       "<td>=BLOB=</td></tr><tr><td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>0</td>\n",
       "<td>1</td>\n",
       "<td>0</td>\n",
       "<td>6f6d8149-1603-9d2c-f884-0fdf995ec8b3</td>\n",
       "<td>neuropixels 1.0 - 3B</td>\n",
       "<td>6</td>\n",
       "<td>=BLOB=</td>\n",
       "<td>=BLOB=</td> </tr> </tbody>\n",
       "        </table>\n",
       "        <p>...</p>\n",
       "        <p>Total: 57450</p></div>\n",
       "        "
      ],
      "text/plain": [
       "*subject     *session_datet *insertion_num *paramset_idx  *curation_id   *unit    *electrode_con *probe_type    *electrode    waveform_m waveforms \n",
       "+----------+ +------------+ +------------+ +------------+ +------------+ +------+ +------------+ +------------+ +-----------+ +--------+ +--------+\n",
       "subject6     2021-01-15 11: 0              0              1              0        6f6d8149-1603- neuropixels 1. 0             =BLOB=     =BLOB=    \n",
       "subject6     2021-01-15 11: 0              0              1              0        6f6d8149-1603- neuropixels 1. 1             =BLOB=     =BLOB=    \n",
       "subject6     2021-01-15 11: 0              0              1              0        6f6d8149-1603- neuropixels 1. 2             =BLOB=     =BLOB=    \n",
       "subject6     2021-01-15 11: 0              0              1              0        6f6d8149-1603- neuropixels 1. 3             =BLOB=     =BLOB=    \n",
       "subject6     2021-01-15 11: 0              0              1              0        6f6d8149-1603- neuropixels 1. 4             =BLOB=     =BLOB=    \n",
       "subject6     2021-01-15 11: 0              0              1              0        6f6d8149-1603- neuropixels 1. 5             =BLOB=     =BLOB=    \n",
       "subject6     2021-01-15 11: 0              0              1              0        6f6d8149-1603- neuropixels 1. 6             =BLOB=     =BLOB=    \n",
       "   ...\n",
       " (Total: 57450)"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ephys.WaveformSet.Waveform & session_key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    \n",
       "    <style type=\"text/css\">\n",
       "        .Relation{\n",
       "            border-collapse:collapse;\n",
       "        }\n",
       "        .Relation th{\n",
       "            background: #A0A0A0; color: #ffffff; padding:4px; border:#f0e0e0 1px solid;\n",
       "            font-weight: normal; font-family: monospace; font-size: 100%;\n",
       "        }\n",
       "        .Relation td{\n",
       "            padding:4px; border:#f0e0e0 1px solid; font-size:100%;\n",
       "        }\n",
       "        .Relation tr:nth-child(odd){\n",
       "            background: #ffffff;\n",
       "        }\n",
       "        .Relation tr:nth-child(even){\n",
       "            background: #f3f1ff;\n",
       "        }\n",
       "        /* Tooltip container */\n",
       "        .djtooltip {\n",
       "        }\n",
       "        /* Tooltip text */\n",
       "        .djtooltip .djtooltiptext {\n",
       "            visibility: hidden;\n",
       "            width: 120px;\n",
       "            background-color: black;\n",
       "            color: #fff;\n",
       "            text-align: center;\n",
       "            padding: 5px 0;\n",
       "            border-radius: 6px;\n",
       "            /* Position the tooltip text - see examples below! */\n",
       "            position: absolute;\n",
       "            z-index: 1;\n",
       "        }\n",
       "        #primary {\n",
       "            font-weight: bold;\n",
       "            color: black;\n",
       "        }\n",
       "        #nonprimary {\n",
       "            font-weight: normal;\n",
       "            color: white;\n",
       "        }\n",
       "\n",
       "        /* Show the tooltip text when you mouse over the tooltip container */\n",
       "        .djtooltip:hover .djtooltiptext {\n",
       "            visibility: visible;\n",
       "        }\n",
       "    </style>\n",
       "    \n",
       "    <b>Mean waveform across spikes for a given unit at its representative electrode</b>\n",
       "        <div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "        <table border=\"1\" class=\"Relation\">\n",
       "            <thead> <tr style=\"text-align: right;\"> <th> <div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">subject</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">session_datetime</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">insertion_number</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">paramset_idx</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">curation_id</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">unit</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">peak_electrode_waveform</p>\n",
       "                            <span class=\"djtooltiptext\">(uV) mean waveform for a given unit at its representative electrode</span>\n",
       "                        </div> </th> </tr> </thead>\n",
       "            <tbody> <tr> <td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>0</td>\n",
       "<td>1</td>\n",
       "<td>0</td>\n",
       "<td>=BLOB=</td></tr><tr><td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>0</td>\n",
       "<td>1</td>\n",
       "<td>1</td>\n",
       "<td>=BLOB=</td></tr><tr><td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>0</td>\n",
       "<td>1</td>\n",
       "<td>2</td>\n",
       "<td>=BLOB=</td></tr><tr><td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>0</td>\n",
       "<td>1</td>\n",
       "<td>3</td>\n",
       "<td>=BLOB=</td></tr><tr><td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>0</td>\n",
       "<td>1</td>\n",
       "<td>4</td>\n",
       "<td>=BLOB=</td></tr><tr><td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>0</td>\n",
       "<td>1</td>\n",
       "<td>5</td>\n",
       "<td>=BLOB=</td></tr><tr><td>subject6</td>\n",
       "<td>2021-01-15 11:16:38</td>\n",
       "<td>0</td>\n",
       "<td>0</td>\n",
       "<td>1</td>\n",
       "<td>6</td>\n",
       "<td>=BLOB=</td> </tr> </tbody>\n",
       "        </table>\n",
       "        <p>...</p>\n",
       "        <p>Total: 150</p></div>\n",
       "        "
      ],
      "text/plain": [
       "*subject     *session_datet *insertion_num *paramset_idx  *curation_id   *unit    peak_elect\n",
       "+----------+ +------------+ +------------+ +------------+ +------------+ +------+ +--------+\n",
       "subject6     2021-01-15 11: 0              0              1              0        =BLOB=    \n",
       "subject6     2021-01-15 11: 0              0              1              1        =BLOB=    \n",
       "subject6     2021-01-15 11: 0              0              1              2        =BLOB=    \n",
       "subject6     2021-01-15 11: 0              0              1              3        =BLOB=    \n",
       "subject6     2021-01-15 11: 0              0              1              4        =BLOB=    \n",
       "subject6     2021-01-15 11: 0              0              1              5        =BLOB=    \n",
       "subject6     2021-01-15 11: 0              0              1              6        =BLOB=    \n",
       "   ...\n",
       " (Total: 150)"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ephys.WaveformSet.PeakWaveform & session_key"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary and next step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook walks through the detailed steps running the workflow.\n",
    "\n",
    "+ For an more automated way running the workflow, refer to [04-automate](04-automate-optional.ipynb)\n",
    "+ In the next notebook [05-explore](05-explore.ipynb), we will introduce DataJoint methods to explore and visualize the ingested data."
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "encoding": "# -*- coding: utf-8 -*-",
   "formats": "ipynb,py_scripts//py"
  },
  "kernelspec": {
   "display_name": "Python 3.9.13 ('ele')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  },
  "vscode": {
   "interpreter": {
    "hash": "d00c4ad21a7027bf1726d6ae3a9a6ef39c8838928eca5a3d5f51f3eb68720410"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}