Fix combatiblity with read only caches.

Create function for writing to last manifest file.
Add specific version to account for write only caches.
Add local cache option to notebooks.
Add versioning and changelog.

intro.md

@@ -58,6 +58,10 @@ Available notebooks:
   (Kimberly Siletti)**
 
 ## Release Notes
+* **[abc_atlas_access (v0.110)]**
+  * Fixed compatibility issue with read only local caches specifically for
+    fuse style file mounts such as those used on CodeOcean.
+  * Added local cache example to notebooks.
 * **[abc_atlas_access (v0.1.0)]**
   * Fixed issue with loading 10X mouse data with the `get_gene_data` function.
 * **[Spring 2024 Public Beta (version 20240330)]**

notebooks/10x_snRNASeq_tutorial_part_1.ipynb

@@ -36,6 +36,8 @@
    "source": [
     "We will interact with the data using the **AbcProjectCache**. This cache object tracks which data has been downloaded and serves the path to the requsted data on disk. For metadata, the cache can also directly serve a up a Pandas Dataframe. See the ``getting_started`` notebook for more details on using the cache including installing it if it has not already been.\n",
     "\n",
+    "The commented section that calls ``from_local_cache`` can be used instead when a download of the data already exists on disk or running on CodeOcean with the attached Allen Brain Cell Atlas or similar mounting of the AWS s3 bucket as a directory through [s3fs-fuse](https://github.com/s3fs-fuse/s3fs-fuse) or similar.\n",
+    "\n",
     "**Change the download_base variable to where you have downloaded the data in your system.**"
    ]
   },
@@ -58,6 +60,10 @@
    "source": [
     "download_base = Path('../../abc_download_root')\n",
     "abc_cache = AbcProjectCache.from_s3_cache(download_base)\n",
+    "\n",
+    "# download_base = Path('../../data/abc_atlas') # Path to the already downloaded data or s3fs-fuse mount.\n",
+    "# abc_cache = AbcProjectCache.from_local_cache(download_base)\n",
+    "\n",
     "abc_cache.current_manifest"
    ]
   },
@@ -2421,7 +2427,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.8"
+   "version": "3.11.9"
   }
  },
  "nbformat": 4,

notebooks/10x_snRNASeq_tutorial_part_2a.ipynb

@@ -35,6 +35,8 @@
    "source": [
     "We will interact with the data using the **AbcProjectCache**. This cache object tracks which data has been downloaded and serves the path to the requsted data on disk. For metadata, the cache can also directly serve a up a Pandas Dataframe. See the ``getting_started`` notebook for more details on using the cache including installing it if it has not already been.\n",
     "\n",
+    "The commented section that calls ``from_local_cache`` can be used instead when a download of the data already exists on disk or running on CodeOcean with the attached Allen Brain Cell Atlas or similar mounting of the AWS s3 bucket as a directory through [s3fs-fuse](https://github.com/s3fs-fuse/s3fs-fuse) or similar.\n",
+    "\n",
     "**Change the download_base variable to where you have downloaded the data in your system.**"
    ]
   },
@@ -57,6 +59,10 @@
    "source": [
     "download_base = Path('../../abc_download_root')\n",
     "abc_cache = AbcProjectCache.from_s3_cache(download_base)\n",
+    "\n",
+    "# download_base = Path('../../data/abc_atlas') # Path to the already downloaded data or s3fs-fuse mount.\n",
+    "# abc_cache = AbcProjectCache.from_local_cache(download_base)\n",
+    "\n",
     "abc_cache.current_manifest"
    ]
   },
@@ -554,7 +560,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.8"
+   "version": "3.11.9"
   }
  },
  "nbformat": 4,

notebooks/10x_snRNASeq_tutorial_part_2b.ipynb

@@ -34,6 +34,8 @@
    "source": [
     "We will interact with the data using the **AbcProjectCache**. This cache object tracks which data has been downloaded and serves the path to the requsted data on disk. For metadata, the cache can also directly serve a up a Pandas Dataframe. See the ``getting_started`` notebook for more details on using the cache including installing it if it has not already been.\n",
     "\n",
+    "The commented section that calls ``from_local_cache`` can be used instead when a download of the data already exists on disk or running on CodeOcean with the attached Allen Brain Cell Atlas or similar mounting of the AWS s3 bucket as a directory through [s3fs-fuse](https://github.com/s3fs-fuse/s3fs-fuse) or similar.\n",
+    "\n",
     "**Change the download_base variable to where you have downloaded the data in your system.**"
    ]
   },
@@ -56,6 +58,10 @@
    "source": [
     "download_base = Path('../../abc_download_root')\n",
     "abc_cache = AbcProjectCache.from_s3_cache(download_base)\n",
+    "\n",
+    "# download_base = Path('../../data/abc_atlas') # Path to the already downloaded data or s3fs-fuse mount.\n",
+    "# abc_cache = AbcProjectCache.from_local_cache(download_base)\n",
+    "\n",
     "abc_cache.current_manifest"
    ]
   },
@@ -590,7 +596,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.8"
+   "version": "3.11.9"
   }
  },
  "nbformat": 4,

notebooks/WHB-10x_snRNASeq_tutorial_part_1.ipynb

@@ -36,7 +36,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "We will interact with the data using the **AbcProjectCache**. This cache object tracks which data has been downloaded and serves the path to the requsted data on disk. For metadata, the cache can also directly serve a up a Pandas Dataframe from the stored csv file. See the ``getting_started`` notebook for more details on using the cache including installing it if it has not already been.\n",
+    "We will interact with the data using the **AbcProjectCache**. This cache object tracks which data has been downloaded and serves the path to the requsted data on disk. For metadata, the cache can also directly serve a up a Pandas Dataframe. See the ``getting_started`` notebook for more details on using the cache including installing it if it has not already been.\n",
+    "\n",
+    "The commented section that calls ``from_local_cache`` can be used instead when a download of the data already exists on disk or running on CodeOcean with the attached Allen Brain Cell Atlas or similar mounting of the AWS s3 bucket as a directory through [s3fs-fuse](https://github.com/s3fs-fuse/s3fs-fuse) or similar.\n",
     "\n",
     "**Change the download_base variable to where you have downloaded the data in your system.**"
    ]
@@ -60,6 +62,10 @@
    "source": [
     "download_base = Path('../../abc_download_root')\n",
     "abc_cache = AbcProjectCache.from_s3_cache(download_base)\n",
+    "\n",
+    "# download_base = Path('../../data/abc_atlas') # Path to the already downloaded data or s3fs-fuse mount.\n",
+    "# abc_cache = AbcProjectCache.from_local_cache(download_base)\n",
+    "\n",
     "abc_cache.current_manifest"
    ]
   },
@@ -1346,7 +1352,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.8"
+   "version": "3.11.9"
   }
  },
  "nbformat": 4,

notebooks/WHB-10x_snRNASeq_tutorial_part_2.ipynb

@@ -34,6 +34,8 @@
    "source": [
     "We will interact with the data using the **AbcProjectCache**. This cache object tracks which data has been downloaded and serves the path to the requsted data on disk. For metadata, the cache can also directly serve a up a Pandas Dataframe. See the ``getting_started`` notebook for more details on using the cache including installing it if it has not already been.\n",
     "\n",
+    "The commented section that calls ``from_local_cache`` can be used instead when a download of the data already exists on disk or running on CodeOcean with the attached Allen Brain Cell Atlas or similar mounting of the AWS s3 bucket as a directory through [s3fs-fuse](https://github.com/s3fs-fuse/s3fs-fuse) or similar.\n",
+    "\n",
     "**Change the download_base variable to where you have downloaded the data in your system.**"
    ]
   },
@@ -56,6 +58,10 @@
    "source": [
     "download_base = Path('../../abc_download_root')\n",
     "abc_cache = AbcProjectCache.from_s3_cache(download_base)\n",
+    "\n",
+    "# download_base = Path('../../data/abc_atlas') # Path to the already downloaded data or s3fs-fuse mount.\n",
+    "# abc_cache = AbcProjectCache.from_local_cache(download_base)\n",
+    "\n",
     "abc_cache.current_manifest"
    ]
   },
@@ -1061,7 +1067,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.8"
+   "version": "3.11.9"
   }
  },
  "nbformat": 4,

notebooks/WHB_cluster_annotation_tutorial.ipynb

@@ -34,6 +34,8 @@
    "source": [
     "We will interact with the data using the **AbcProjectCache**. This cache object tracks which data has been downloaded and serves the path to the requsted data on disk. For metadata, the cache can also directly serve a up a Pandas Dataframe. See the ``getting_started`` notebook for more details on using the cache including installing it if it has not already been.\n",
     "\n",
+    "The commented section that calls ``from_local_cache`` can be used instead when a download of the data already exists on disk or running on CodeOcean with the attached Allen Brain Cell Atlas or similar mounting of the AWS s3 bucket as a directory through [s3fs-fuse](https://github.com/s3fs-fuse/s3fs-fuse) or similar.\n",
+    "\n",
     "**Change the download_base variable to where you have downloaded the data in your system.**"
    ]
   },
@@ -56,6 +58,10 @@
    "source": [
     "download_base = Path('../../abc_download_root')\n",
     "abc_cache = AbcProjectCache.from_s3_cache(download_base)\n",
+    "\n",
+    "# download_base = Path('../../data/abc_atlas') # Path to the already downloaded data or s3fs-fuse mount.\n",
+    "# abc_cache = AbcProjectCache.from_local_cache(download_base)\n",
+    "\n",
     "abc_cache.current_manifest"
    ]
   },
@@ -1605,7 +1611,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.8"
+   "version": "3.11.9"
   }
  },
  "nbformat": 4,

notebooks/ccf_and_parcellation_annotation_tutorial.ipynb

@@ -38,6 +38,8 @@
    "source": [
     "We will interact with the data using the **AbcProjectCache**. This cache object tracks which data has been downloaded and serves the path to the requsted data on disk. For metadata, the cache can also directly serve a up a Pandas Dataframe. See the ``getting_started`` notebook for more details on using the cache including installing it if it has not already been.\n",
     "\n",
+    "The commented section that calls ``from_local_cache`` can be used instead when a download of the data already exists on disk or running on CodeOcean with the attached Allen Brain Cell Atlas or similar mounting of the AWS s3 bucket as a directory through [s3fs-fuse](https://github.com/s3fs-fuse/s3fs-fuse) or similar.\n",
+    "\n",
     "**Change the download_base variable to where you have downloaded the data in your system.**"
    ]
   },
@@ -58,8 +60,12 @@
     }
    ],
    "source": [
-    "download_base = pathlib.Path('../../abc_download_root')\n",
+    "download_base = Path('../../abc_download_root')\n",
     "abc_cache = AbcProjectCache.from_s3_cache(download_base)\n",
+    "\n",
+    "# download_base = Path('../../data/abc_atlas') # Path to the already downloaded data or s3fs-fuse mount.\n",
+    "# abc_cache = AbcProjectCache.from_local_cache(download_base)\n",
+    "\n",
     "abc_cache.current_manifest"
    ]
   },
@@ -2659,7 +2665,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.8"
+   "version": "3.11.9"
   }
  },
  "nbformat": 4,

notebooks/cluster_annotation_tutorial.ipynb

@@ -36,6 +36,8 @@
    "source": [
     "We will interact with the data using the **AbcProjectCache**. This cache object tracks which data has been downloaded and serves the path to the requsted data on disk. For metadata, the cache can also directly serve a up a Pandas Dataframe. See the ``getting_started`` notebook for more details on using the cache including installing it if it has not already been.\n",
     "\n",
+    "The commented section that calls ``from_local_cache`` can be used instead when a download of the data already exists on disk or running on CodeOcean with the attached Allen Brain Cell Atlas or similar mounting of the AWS s3 bucket as a directory through [s3fs-fuse](https://github.com/s3fs-fuse/s3fs-fuse) or similar.\n",
+    "\n",
     "**Change the download_base variable to where you have downloaded the data in your system.**"
    ]
   },
@@ -56,8 +58,12 @@
     }
    ],
    "source": [
-    "download_base = pathlib.Path('../../abc_download_root')\n",
+    "download_base = Path('../../abc_download_root')\n",
     "abc_cache = AbcProjectCache.from_s3_cache(download_base)\n",
+    "\n",
+    "# download_base = Path('../../data/abc_atlas') # Path to the already downloaded data or s3fs-fuse mount.\n",
+    "# abc_cache = AbcProjectCache.from_local_cache(download_base)\n",
+    "\n",
     "abc_cache.current_manifest"
    ]
   },
@@ -1917,7 +1923,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.8"
+   "version": "3.11.9"
   }
  },
  "nbformat": 4,

notebooks/cluster_groups_and_embeddings_tutorial.ipynb

@@ -31,6 +31,8 @@
    "source": [
     "We will interact with the data using the **AbcProjectCache**. This cache object tracks which data has been downloaded and serves the path to the requsted data on disk. For metadata, the cache can also directly serve a up a Pandas Dataframe. See the ``getting_started`` notebook for more details on using the cache including installing it if it has not already been.\n",
     "\n",
+    "The commented section that calls ``from_local_cache`` can be used instead when a download of the data already exists on disk or running on CodeOcean with the attached Allen Brain Cell Atlas or similar mounting of the AWS s3 bucket as a directory through [s3fs-fuse](https://github.com/s3fs-fuse/s3fs-fuse) or similar.\n",
+    "\n",
     "**Change the download_base variable to where you have downloaded the data in your system.**"
    ]
   },
@@ -51,8 +53,12 @@
     }
    ],
    "source": [
-    "download_base = pathlib.Path('../../abc_download_root')\n",
+    "download_base = Path('../../abc_download_root')\n",
     "abc_cache = AbcProjectCache.from_s3_cache(download_base)\n",
+    "\n",
+    "# download_base = Path('../../data/abc_atlas') # Path to the already downloaded data or s3fs-fuse mount.\n",
+    "# abc_cache = AbcProjectCache.from_local_cache(download_base)\n",
+    "\n",
     "abc_cache.current_manifest"
    ]
   },
@@ -1379,7 +1385,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.8"
+   "version": "3.11.9"
   }
  },
  "nbformat": 4,

notebooks/cluster_neighborhood_gallery.ipynb

@@ -31,6 +31,8 @@
    "source": [
     "We will interact with the data using the **AbcProjectCache**. This cache object tracks which data has been downloaded and serves the path to the requsted data on disk. For metadata, the cache can also directly serve a up a Pandas Dataframe. See the ``getting_started`` notebook for more details on using the cache including installing it if it has not already been.\n",
     "\n",
+    "The commented section that calls ``from_local_cache`` can be used instead when a download of the data already exists on disk or running on CodeOcean with the attached Allen Brain Cell Atlas or similar mounting of the AWS s3 bucket as a directory through [s3fs-fuse](https://github.com/s3fs-fuse/s3fs-fuse) or similar.\n",
+    "\n",
     "**Change the download_base variable to where you have downloaded the data in your system.**"
    ]
   },
@@ -51,8 +53,12 @@
     }
    ],
    "source": [
-    "download_base = pathlib.Path('../../abc_download_root')\n",
+    "download_base = Path('../../abc_download_root')\n",
     "abc_cache = AbcProjectCache.from_s3_cache(download_base)\n",
+    "\n",
+    "# download_base = Path('../../data/abc_atlas') # Path to the already downloaded data or s3fs-fuse mount.\n",
+    "# abc_cache = AbcProjectCache.from_local_cache(download_base)\n",
+    "\n",
     "abc_cache.current_manifest"
    ]
   },
@@ -878,7 +884,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.8"
+   "version": "3.11.9"
   }
  },
  "nbformat": 4,

notebooks/general_accessing_10x_snRNASeq_tutorial.ipynb

@@ -33,7 +33,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "We will interact with the data using the **AbcProjectCache**. This cache object tracks which data has been downloaded and serves the path to the requsted data on disk. For metadata, the cache can also directly serve up a Pandas Dataframe. See the ``getting_started`` notebook for more details on using the cache including installing it if it has not already been.\n",
+    "We will interact with the data using the **AbcProjectCache**. This cache object tracks which data has been downloaded and serves the path to the requsted data on disk. For metadata, the cache can also directly serve a up a Pandas Dataframe. See the ``getting_started`` notebook for more details on using the cache including installing it if it has not already been.\n",
+    "\n",
+    "The commented section that calls ``from_local_cache`` can be used instead when a download of the data already exists on disk or running on CodeOcean with the attached Allen Brain Cell Atlas or similar mounting of the AWS s3 bucket as a directory through [s3fs-fuse](https://github.com/s3fs-fuse/s3fs-fuse) or similar.\n",
     "\n",
     "**Change the download_base variable to where you have downloaded the data in your system.**"
    ]
@@ -57,6 +59,10 @@
    "source": [
     "download_base = Path('../../abc_download_root')\n",
     "abc_cache = AbcProjectCache.from_s3_cache(download_base)\n",
+    "\n",
+    "# download_base = Path('../../data/abc_atlas') # Path to the already downloaded data or s3fs-fuse mount.\n",
+    "# abc_cache = AbcProjectCache.from_local_cache(download_base)\n",
+    "\n",
     "abc_cache.current_manifest"
    ]
   },

notebooks/getting_started.ipynb

@@ -144,7 +144,9 @@
     "\n",
     "Below we show how to setup up the cache to download from S3, how to list and switch to a different data release, and additionally how to list the directories available, their size, and the files in that directory.\n",
     "\n",
-    "Setup the **AbcProjectCache** object by specifying a directory and calling ``from_s3_cache`` as shown below. We also print what version of the manifest is being currently loaded by the cache."
+    "Setup the **AbcProjectCache** object by specifying a directory and calling ``from_s3_cache`` as shown below. We also print what version of the manifest is being currently loaded by the cache.\n",
+    "\n",
+    "The commented section that calls ``from_local_cache`` can be used instead when a download of the data already exists on disk or running on CodeOcean with the attached Allen Brain Cell Atlas or similar mounting of the AWS s3 bucket as a directory through [s3fs-fuse](https://github.com/s3fs-fuse/s3fs-fuse) or similar. Using the cache from a s3fs mount means that all files will be streamed over the internet instead of saved to the local disk."
    ]
   },
   {
@@ -166,6 +168,10 @@
    "source": [
     "download_base = Path('../../abc_download_root') # Path to where you would like to write the downloaded data.\n",
     "abc_cache = AbcProjectCache.from_s3_cache(download_base)\n",
+    "\n",
+    "# download_base = Path('../../data/abc_atlas') # Path to the already downloaded data or s3fs-fuse mount.\n",
+    "# abc_cache = AbcProjectCache.from_local_cache(download_base)\n",
+    "\n",
     "abc_cache.current_manifest"
    ]
   },