Merge branch 'master' of github.com:Yambottle/playground_tutorial

.gitignore

@@ -1,12 +1,14 @@
+# OSX and python intermediate files
 __pycache__/
 .DS_Store
 ~*
 *~
 *.pyc
 .ipynb_checkpoints/
-dj_local_conf.json
-
-.idea/
+debug.log
 
+# DataJoint and setup files
+dj_local_con*.json
 julia/setup.jl
-debug.log
+.idea/
+temp*

00-Getting started.ipynb → 00-Getting_started/00-Getting started.ipynb

@@ -11,7 +11,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Congratulations! If you are reading this, then you have successfully connected to the workshop JupyterHub and opened up your very first workshop notebook!\n",
+    "Congratulations! If you are reading this, then you have successfully connected to the workshop JupyterHub and opened up your very first workshop notebook! These notebooks are oranized with numbers to highlight what should come next. First, we'll walk through the startup process. Next, you'll see an introduction to DataJoint, and how to structure a data pipeline. The Calcium Imaging and Electrophysiology folders show example data structures and analysis techniques from common experimental techniques. We've included some fill-in-the-blank sections for you to code yourself. If you want to see completed notebooks, see the folder of completed exercises.\n",
     "\n",
     "Before we start, remember that all of the edits you make in these tutorial notebooks are ***not persistent*** - they will be reset to the original content everytime you restart the server. However, you can easily download the notebooks that you're interested in keeping the changes."
    ]
@@ -316,18 +316,18 @@
     "The next step in the tutorial is to:\n",
     "1. learn the basic of the DataJoint framework with [01-DataJoint Basics](./01-DataJoint%20Basics%20-%20Interactive.ipynb)\n",
     "2. data ingestion in a pipeline, with practical examples in 2 domains:\n",
-    "    + [02-electrophysiology](./electrophysiology/02-Imported%20Tables%20-%20Interactive.ipynb)\n",
-    "    + [02-calcium imaging](./calcium_imaging/02-Imported%20Tables%20-%20Interactive.ipynb)\n",
+    "    + [02-electrophysiology](/02-Electrophysiology/02-Imported%20Tables%20-%20Interactive.ipynb)\n",
+    "    + [02-calcium imaging](./01-Calcium_Imaging/02-Imported%20Tables%20-%20Interactive.ipynb)\n",
     "3. more advanced topics on built-in automatic computation of the pipeline with DataJoint, also with practical examples from:\n",
-    "    + [03-electrophysiology](./electrophysiology/03-Computed%20Table%2C%20Lookup%20Table%2C%20and%20Part%20Table%20-%20Completed.ipynb)\n",
-    "    + [03-calcium imaging](./calcium_imaging/03-Computed%20Table%2C%20Lookup%20Table%2C%20and%20Part%20Table%20-%20Completed.ipynb)"
+    "    + [03-electrophysiology](/02-Electrophysiology/03-Computed%20Table%2C%20Lookup%20Table%2C%20and%20Part%20Table%20-%20Completed.ipynb)\n",
+    "    + [03-calcium imaging](./01-Calcium_Imaging/03-Computed%20Table%2C%20Lookup%20Table%2C%20and%20Part%20Table%20-%20Completed.ipynb)"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "As you will find out that these tutorial notebooks are meant to be interactive. There will be empty cells meant for you to write the code yourself, as mini exercises. To access the notebooks with complete solutions, visit this [folder](./completed)."
+    "As you will find out that these tutorial notebooks are meant to be interactive. There will be empty cells meant for you to write the code yourself, as mini exercises. To access the notebooks with complete solutions, visit this [folder](../03-Completed_Exercises)."
    ]
   }
  ],

01-DataJoint Basics - Interactive.ipynb → 00-Getting_started/01-DataJoint Basics - Interactive.ipynb

@@ -555,7 +555,7 @@
    "source": [
     "Note that, to uniquely identify an experimental session (or simply a **session**), we need to know the mouse that the session was about. In other words, a session cannot existing without a corresponding mouse! \n",
     "\n",
-    "With **mouse** already represented as a table in our pipeline, we say that the session **depends on** the mouse! We would graphically represent this in an **entity relationship diagram (ERD)** by drawing the line between two tables, with the one below (**session**) dependeing on the one above (**mouse**)."
+    "With **mouse** already represented as a table in our pipeline, we say that the session **depends on** the mouse! We could graphically represent this in an **entity relationship diagram (ERD)** by drawing the line between two tables, with the one below (**session**) dependeing on the one above (**mouse**)."
    ]
   },
   {
@@ -590,7 +590,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "You can actually generate the entity relationship diagram (ERD) on the fly by calling `dj.ERD` with the schema object"
+    "You can actually generate something similar to an entity relationship diagram (ERD) on the fly by calling `dj.Diagram` with the schema object. Many of the symbols and features are the same as the ERD standard."
    ]
   },
   {
@@ -599,7 +599,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "dj.ERD(schema)"
+    "dj.Diagram(schema)"
    ]
   },
   {
@@ -1535,8 +1535,8 @@
     "In the next session, we are going to extend our data pipeline with tables to represent **imported data** and define new tables to **compute and hold analysis results**.\n",
     "\n",
     "We will use both ephys and calcium imaging as example pipelines:\n",
-    "+ [02-electrophysiology](./electrophysiology/02-Imported%20Tables%20-%20Interactive.ipynb)\n",
-    "+ [02-calcium imaging](./calcium_imaging/02-Imported%20Tables%20-%20Interactive.ipynb)"
+    "+ [02-electrophysiology](../02-Electrophysiology/02-Imported%20Tables%20-%20Interactive.ipynb)\n",
+    "+ [02-calcium imaging](../01-Calcium_Imaging/02-Imported%20Tables%20-%20Interactive.ipynb)"
    ]
   },
   {

01-DataJoint Basics - Interactive.py → 00-Getting_started/01-DataJoint Basics - Interactive.py

@@ -42,13 +42,13 @@
 #
 # Let's go ahead and build together a pipeline from scratch to better understand what a data pipeline is all about.
 
-# # Building our first pipeline: 
+# # Building our first pipeline:
 
 # Let's build a pipeline to collect, store and process data and analysis for our hypothetical single electrode recording or calcium imaging recording in mice. To help us understand the project better, here is a brief description:
 
 # > * Your lab houses many mice, and each mouse is identified by a unique ID. You also want to keep track of information about each mouse such as their date of birth, and gender.
 # > * As a hard working neuroscientist, you perform experiments every day, sometimes working with more than one mouse in a day! However, on any given day, a mouse undergoes at most one recording session.
-# > * For each experimental session, you would like to record what mouse you worked with and when you performed the experiment. You would also like to keep track of other helpful information such as the experimental setup you worked on.  
+# > * For each experimental session, you would like to record what mouse you worked with and when you performed the experiment. You would also like to keep track of other helpful information such as the experimental setup you worked on.
 #
 # > * In a session of electrophysiology
 # >> * you record electrical activity from a single neuron. You use recording equipment that produces separate data files for each neuron you recorded.
@@ -64,7 +64,7 @@
 
 # > * Your lab houses many **mice**, and each mouse is identified by a unique ID. You also want to keep track of information about each mouse such as their date of birth, and gender.
 # > * As a hard working neuroscientist, you perform experiments every day, sometimes working with more than one mouse in a day! However, on an any given day, a mouse undergoes at most one recording session.
-# > * For each **experimental session**, you would like to record what mouse you worked with and when you performed the experiment. You would also like to keep track of other helpful information such as the experimental setup you worked on.  
+# > * For each **experimental session**, you would like to record what mouse you worked with and when you performed the experiment. You would also like to keep track of other helpful information such as the experimental setup you worked on.
 #
 # > * In a session of electrophysiology
 # >> * you record electrical activity from a **single neuron**. You use recording equipment that produces separate data files for each neuron you recorded.
@@ -92,7 +92,7 @@
 
 # In the current notebook, we will design the tables for mouse and experimental sessions, the rest of the pipeline will be designed in the subdirectory `electrophysiology` and `calcium_imaging`
 
-# In DataJoint data pipeline, we represent these **entities** as **tables**. Different *kinds* of entities become distinct tables, and each row of the table is a single example (instance) of the category of entity. 
+# In DataJoint data pipeline, we represent these **entities** as **tables**. Different *kinds* of entities become distinct tables, and each row of the table is a single example (instance) of the category of entity.
 #
 # For example, if we have a `Mouse` table, then each row in the mouse table represents a single mouse!
 
@@ -229,7 +229,7 @@ class Mouse(dj.Manual):
 
 # ## Create tables with dependencies
 
-# Congratulations! We have successfully created your first table! We are now ready to tackle and include other **entities** in the project into our data pipeline. 
+# Congratulations! We have successfully created your first table! We are now ready to tackle and include other **entities** in the project into our data pipeline.
 #
 # Let's now take a look at representing an **experimental session**.
 
@@ -245,11 +245,11 @@ class Mouse(dj.Manual):
 #
 # to uniquely identify a single experimental session.
 
-# Note that, to uniquely identify an experimental session (or simply a **session**), we need to know the mouse that the session was about. In other words, a session cannot existing without a corresponding mouse! 
+# Note that, to uniquely identify an experimental session (or simply a **session**), we need to know the mouse that the session was about. In other words, a session cannot existing without a corresponding mouse!
 #
-# With **mouse** already represented as a table in our pipeline, we say that the session **depends on** the mouse! We would graphically represent this in an **entity relationship diagram (ERD)** by drawing the line between two tables, with the one below (**session**) dependeing on the one above (**mouse**).
+# With **mouse** already represented as a table in our pipeline, we say that the session **depends on** the mouse! We could graphically represent this in an **entity relationship diagram (ERD)** by drawing the line between two tables, with the one below (**session**) dependeing on the one above (**mouse**).
 
-# Thus we will need both **mouse** and a new attribute **session_date** to uniquely identify a single session. 
+# Thus we will need both **mouse** and a new attribute **session_date** to uniquely identify a single session.
 #
 # Remember that a **mouse** is already uniquely identified by its primary key - **mouse_id**. In DataJoint, you can declare that **session** depends on the mouse, and DataJoint will automatically include the mouse's primary key (`mouse_id`) as part of the session's primary key, along side any additional attribute(s) you specificy.
 
@@ -266,9 +266,9 @@ class Session(dj.Manual):
     """
 
 
-# You can actually generate the entity relationship diagram (ERD) on the fly by calling `dj.ERD` with the schema object
+# You can actually generate something similar to an entity relationship diagram (ERD) on the fly by calling `dj.Diagram` with the schema object. Many of the symbols and features are the same as the ERD standard.
 
-dj.ERD(schema)
+dj.Diagram(schema)
 
 # Let's try inserting a few sessions manually.
 
@@ -397,17 +397,17 @@ class Session(dj.Manual):
 # ENTER YOUR CODE
 
 
-# In computer science/math lingo, DataJoint operations are said to **satisfy closure property**. Practically speaking, this means that the result of a query can immediately be used in another query, allowing you to build more complex queries from simpler ones. 
+# In computer science/math lingo, DataJoint operations are said to **satisfy closure property**. Practically speaking, this means that the result of a query can immediately be used in another query, allowing you to build more complex queries from simpler ones.
 
 # ### Restriction one table with another
 
 # All mice that have a session
 
-Mouse & Session 
+Mouse & Session
 
 # ### Combining restrictions
 
-# All the above queries could be combined 
+# All the above queries could be combined
 
 # Male mice that had a session
 
@@ -452,7 +452,7 @@ class Session(dj.Manual):
 Mouse * Session & 'session_date > "2017-05-19"'
 
 # ## Projection .proj(): focus on attributes of interest
-# Beside restriction (`&`) and join (`*`) operations, DataJoint offers another type of operation: projection (`.proj()`). Projection is used to select attributes (columns) from a table, to rename them, or to create new calculated attributes. 
+# Beside restriction (`&`) and join (`*`) operations, DataJoint offers another type of operation: projection (`.proj()`). Projection is used to select attributes (columns) from a table, to rename them, or to create new calculated attributes.
 
 # From the ***Mouse*** table, suppose we want to focus only on the `sex` attribute and ignore the others, this can be done as:
 
@@ -556,7 +556,7 @@ class Session(dj.Manual):
 # In the next session, we are going to extend our data pipeline with tables to represent **imported data** and define new tables to **compute and hold analysis results**.
 #
 # We will use both ephys and calcium imaging as example pipelines:
-# + [02-electrophysiology](./electrophysiology/02-Imported%20Tables%20-%20Interactive.ipynb)
-# + [02-calcium imaging](./calcium_imaging/02-Imported%20Tables%20-%20Interactive.ipynb)
+# + [02-calcium imaging](.../01-Calcium_Imaging/02-Imported%20Tables%20-%20Interactive.ipynb)
+# + [02-electrophysiology](../02-Electrophysiology/02-Imported%20Tables%20-%20Interactive.ipynb)
 
 

calcium_imaging/02-Imported Tables - Interactive.ipynb → 01-Calcium_Imaging/02-Imported Tables - Interactive.ipynb

@@ -67,8 +67,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import sys\n",
-    "sys.path.append(\"..\")\n",
+    "import os\n",
+    "if os.path.basename(os.getcwd())!='notebooks': os.chdir('/home/notebooks')\n",
     "from tutorial_pipeline.mouse_session import schema, Mouse, Session"
    ]
   },
@@ -142,7 +142,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "This table is dependent on the table `Session`, inheriting its primary key attributes, with an additional primary key attribute `scan_idx`. One session could contain multiple scans, which is another example of **one-to-many** relationship. We could take a look at the ERD again."
+    "This table is dependent on the table `Session`, inheriting its primary key attributes, with an additional primary key attribute `scan_idx`. One session could contain multiple scans, which is another example of **one-to-many** relationship. We could take a look at the Diagram again."
    ]
   },
   {
@@ -151,7 +151,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "dj.ERD(schema)"
+    "dj.Diagram(schema)"
    ]
   },
   {
@@ -296,7 +296,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "dj.ERD(schema)"
+    "dj.Diagram(schema)"
    ]
   },
   {
@@ -333,9 +333,9 @@
    "source": [
     "In DataJoint, the tier of the table indicates **the nature of the data and the data source for the table**. So far we have encountered two table tiers: `Manual` and `Imported`, and we will encounter the two other major tiers in this session. \n",
     "\n",
-    "DataJoint tables in `Manual` tier, or simply **Manual tables** indicate that its contents are **manually** entered by either experimenters or a recording system, and its content **do not depend on external data files or other tables**. This is the most basic table type you will encounter, especially as the tables at the beggining of the pipeline. In the ERD, `Manual` tables are depicted by green rectangles.\n",
+    "DataJoint tables in `Manual` tier, or simply **Manual tables** indicate that its contents are **manually** entered by either experimenters or a recording system, and its content **do not depend on external data files or other tables**. This is the most basic table type you will encounter, especially as the tables at the beggining of the pipeline. In the Diagram, `Manual` tables are depicted by green rectangles.\n",
     "\n",
-    "On the other hand, **Imported tables** are understood to pull data (or *import* data) from external data files, and come equipped with functionalities to perform this importing process automatically, as we will see shortly! In the ERD, `Imported` tables are depicted by blue ellipses."
+    "On the other hand, **Imported tables** are understood to pull data (or *import* data) from external data files, and come equipped with functionalities to perform this importing process automatically, as we will see shortly! In the Diagram, `Imported` tables are depicted by blue ellipses."
    ]
   },
   {
@@ -344,7 +344,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "dj.ERD(schema)"
+    "dj.Diagram(schema)"
    ]
   },
   {
@@ -676,7 +676,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "dj.ERD(schema)"
+    "dj.Diagram(schema)"
    ]
   },
   {

calcium_imaging/03-Computed Table, Lookup Table, and Part Table - Interactive.ipynb → 01-Calcium_Imaging/03-Computed Table, Lookup Table, and Part Table - Interactive.ipynb

@@ -11,7 +11,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Welcome back! In this session, we are going to continue working with the pipeline for the mouse electrophysiology example. \n",
+    "Welcome back! In this session, we are going to continue working with the pipeline for the mouse calcium imaging example. \n",
     "\n",
     "In this session, we will learn to:\n",
     "\n",
@@ -69,8 +69,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import sys\n",
-    "sys.path.append(\"..\")\n",
+    "import os\n",
+    "if os.path.basename(os.getcwd())!='notebooks': os.chdir('/home/notebooks')\n",
     "from tutorial_pipeline.imaging import schema, Mouse, Session, Scan, AverageFrame"
    ]
   },
@@ -406,7 +406,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Let's define `Param` table to hold different parameter configuration for our spike detection algorithm. We are going to define this table as a `Lookup` table, rather than a `Manual` table. By now, you know that `Lookup` must be yet another **table tier** in DataJoint. `Lookup` tables are depicted by gray boxes in the ERD.\n",
+    "Let's define `Param` table to hold different parameter configuration for our spike detection algorithm. We are going to define this table as a `Lookup` table, rather than a `Manual` table. By now, you know that `Lookup` must be yet another **table tier** in DataJoint. `Lookup` tables are depicted by gray boxes in the Diagram.\n",
     "\n",
     "This tier indicates that the table will contain information:\n",
     "* that will be referenced by other tables\n",
@@ -435,7 +435,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "dj.ERD(schema)"
+    "dj.Diagram(schema)"
    ]
   },
   {
@@ -501,7 +501,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "dj.ERD(schema)"
+    "dj.Diagram(schema)"
    ]
   },
   {
@@ -1006,14 +1006,14 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "dj.ERD(schema)"
+    "dj.Diagram(schema)"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "We have covered most of the building elements of data pipeline design. Using these elements, we could design more sophiscated pipelines that facillitates your experimental recordings and data analyses."
+    "We have covered most of the building elements of data pipeline design. Using these elements, we could design more sophiscated pipelines that facillitates your experimental recordings and data analyses. To see these concepts in the context of another methodology, see the next folder on [Electrophysiology](../02-Electrophysiology)"
    ]
   }
  ],