Merge pull request #3 from yambottle/update-to-v8

Update to v8

.devcontainer/Dockerfile

@@ -7,9 +7,9 @@ RUN \
     # MICrONS dependencies
     pip3 install git+https://github.com/AllenInstitute/em_coregistration.git@phase3 && \
     pip3 install seaborn ipykernel && \
-    pip3 install git+https://github.com/yambottle/microns_phase3_nda.git
+    pip3 install git+https://github.com/datajoint/microns_phase3_nda.git
 
-ENV DJ_HOST tutorial-db.datajoint.io
+ENV DJ_HOST db.datajoint.com
 ENV DJ_USER microns
-ENV DJ_PASS microns2021
+ENV DJ_PASS microns2023
 ENV DATABASE_PREFIX microns_

.gitignore

@@ -1,3 +1,5 @@
+export_notebooks/*
+
 # Byte-compiled / optimized / DLL files
 __pycache__/
 *.py[cod]

README.md

@@ -1,5 +1,9 @@
 # microns_phase3_nda
-nda schema for MICrONS phase3. For more on the MICrONS project please see: [MICrONS Explorer](https://www.microns-explorer.org/)
+This repository contains the Python functions and utilities required to interact with the functional data for the MICrONS project. The database containing this functional data is called `nda` (short for neural data access).
+
+For more on the MICrONS project please see: [MICrONS Explorer](https://www.microns-explorer.org/)
+
+The current version of this repository and database is v8 (Semantic version: 0.8.0).
 
 ## Interactive Environment
 Here are some options that provide a great experience:
@@ -21,54 +25,31 @@ Here are some options that provide a great experience:
 You will know your environment has finished loading once you either see a terminal open related to `Running postStartCommand` with a final message: `Done` or the `README.md` is opened in `Preview`.
 
 ## Technical documentation
-Technical documentation on the functional data can be found [here](https://www.microns-explorer.org/cortical-mm3#f-data).
-
-## Installation Instructions
-This package requires access to the functional data. To download the SQL database get started on the microns-nda-access repo [here](https://github.com/cajal/microns-nda-access).
+Technical documentation on the functional data can be found on the MICrONS project website [here](https://www.microns-explorer.org/cortical-mm3#f-data).
 
-This package requires PyTorch. Windows users may need to install from the PyTorch website: [PyTorch installation](https://pytorch.org/get-started/locally/)
+Download the nda v8 database technical documentation, which includes a changelog from the v7 database [here](https://bossdb-open-data.s3.amazonaws.com/iarpa_microns/minnie/functional_data/two_photon_processed_data_and_metadata/database_v8/two_photon_processed_data_and_metadata_technical_documentation_v8.pdf).
 
-This package requires the em_coregistration package from the Allen Institute:
+Separate from the data contained in the nda database, the MICrONS project website [here](https://www.microns-explorer.org/cortical-mm3#f-data) contains technical docs and access instructions for:
 
-```bash
-pip3 install git+https://github.com/AllenInstitute/em_coregistration.git@phase3
-```
+1. Raster- and motion-corrected functional scan tiffs.
+2. Raster- and motion-corrected two-photon structural stack tiff
+3. Stitched and temporally aligned stimuli for each scan
 
-Install this package:
-
-```bash
-pip3 install git+https://github.com/cajal/microns_phase3_nda.git
-```
-
-## Import Instructions
-
-Import datajoint. Configuration instructions: https://docs.datajoint.io/python/setup/01-Install-and-Connect.html
+## Installation Instructions
+This package requires access to the functional database. To download the SQL database and the pre-built Docker access images start with the `microns-nda-access` repo [here](https://github.com/cajal/microns-nda-access).
 
-```python
-import datajoint as dj
-```
+Once inside your properly configured environment run the tutorials below:
 
-In a jupyter notebook:
+## Tutorials:
 
-```python
-from phase3 import nda, func, utils
-```
+[Using DataJoint to Access Functional Data Tutorial](tutorial_notebooks/Using_DataJoint_to_Access_Functional_Data.ipynb)
 
-## Using the schema
+[Matched Cell Functional Data Tutorial](tutorial_notebooks/Matched_Cell_Functional_Data.ipynb)
 
-To view schema ERD:
-```python
-dj.ERD(nda)
-```
+## nda table descriptions
 
 ![nda](images/nda_erd.png)
 
-## Tutorial: 
-
-[Using DataJoint to Access Functional Data Tutorial](notebooks/Using_DataJoint_to_Access_Functional_Data.ipynb) 
-
-## nda table descriptions
-
 **nda.Scan:** Information on completed scans. Cajal Pipeline: [meso.ScanInfo](https://github.com/cajal/pipeline/blob/6a8342bf3edb07f5653c61024742258295cd8014/python/pipeline/meso.py#L29)
 
 **nda.ScanInclude:** Scans determined suitable for analysis. 
@@ -83,7 +64,7 @@ dj.ERD(nda)
 
 **nda.Treadmill:** Treadmill velocities low-pass filtered with a hamming window to the scan frame rate then linearly interpolated to scan frame times.
 
-**nda.FrameTimes:** Timestamps of scan frames in seconds relative to the start of the scan for the first pixel of the first imaging field.
+**nda.ScanTimes:** Timestamps of scan frames in seconds relative to the start of the scan for the first pixel of the first imaging field.
 
 **nda.Stimulus:** For each scan, contains the movie aligned to activity traces in `nda.Activity`.
 
@@ -95,6 +76,10 @@ dj.ERD(nda)
 
 **nda.Trippy:** Detailed information for the Trippy stimulus.
 
+**nda.RasterCorrection:** Raster phase correction applied to each scan field.
+
+**nda.MotionCorrection:** Motion correction applied to each scan frame.
+
 **nda.MeanIntensity:** Mean intensity of imaging field over time. Cajal Pipeline: [meso.Quality.MeanIntensity](https://github.com/cajal/pipeline/blob/fa202ee43437a67d55719e8ae9769ee9937581d0/python/pipeline/meso.py#L173)
 
 **nda.SummaryImages:** Reference images of the scan field. Cajal Pipeline: [meso.SummaryImages](https://github.com/cajal/pipeline/blob/921a920478c73687dd78b863fcd05e12bbf1e197/python/pipeline/meso.py#L571)
@@ -111,9 +96,11 @@ dj.ERD(nda)
 
 **nda.ScanUnit:** Unit_id assignment that is unique across the entire scan. Includes info about each unit. Cajal Pipeline: [meso.ScanSet.Unit / meso.ScanSet.UnitInfo](https://github.com/cajal/pipeline/blob/921a920478c73687dd78b863fcd05e12bbf1e197/python/pipeline/meso.py#L1341)
 
+**nda.UnitHash:** Assigns hash and semantic string to each unique session - scan\_idx - unit\_id triplet.\\
+
 **nda.Activity:** Deconvolved spike trace from the fluorescence trace. Cajal Pipeline: [meso.Activity.Trace](https://github.com/cajal/pipeline/blob/921a920478c73687dd78b863fcd05e12bbf1e197/python/pipeline/meso.py#L1501)
 
-**nda.StackUnit:** Unit coordinates in stack reference frame after field registration. `np_x, np_y, np_z` should be used for transformation to EM space using Coregistration. [meso.StackCoordinates.UnitInfo](https://github.com/cajal/pipeline/blob/921a920478c73687dd78b863fcd05e12bbf1e197/python/pipeline/meso.py#L1672)
+**nda.StackUnit:** Unit coordinates in stack reference frame after field registration. `stack_x, stack_y, stack_z` should be used for transformation to EM space using Coregistration. [meso.StackCoordinates.UnitInfo](https://github.com/cajal/pipeline/blob/921a920478c73687dd78b863fcd05e12bbf1e197/python/pipeline/meso.py#L1672)
 
 **nda.AreaMembership:** Visual area labels for all units.
 

microns_phase3/__init__.py

@@ -0,0 +1,12 @@
+from .version import __version__
+
+import datajoint as dj
+
+dj.config['enable_python_native_blobs'] = True
+dj.errors._switch_filepath_types(True)
+dj.errors._switch_adapted_types(True)
+
+# try:
+#     import coregister.solve as cs
+# except ModuleNotFoundError:
+#     raise ModuleNotFoundError('Coregistration package missing. Run "pip3 install git+https://github.com/AllenInstitute/em_coregistration.git@phase3"')