Merge branch 'u/morriscb/debugForS3FS'

intro.md

@@ -58,6 +58,10 @@ Available notebooks:
   (Kimberly Siletti)**
 
 ## Release Notes
+* **[abc_atlas_access (v0.110)]**
+  * Fixed compatibility issue with read only local caches specifically for
+    fuse style file mounts such as those used on CodeOcean.
+  * Added local cache example to notebooks.
 * **[abc_atlas_access (v0.1.0)]**
   * Fixed issue with loading 10X mouse data with the `get_gene_data` function.
 * **[Spring 2024 Public Beta (version 20240330)]**

notebooks/10x_snRNASeq_tutorial_part_1.ipynb

@@ -36,6 +36,8 @@
    "source": [
     "We will interact with the data using the **AbcProjectCache**. This cache object tracks which data has been downloaded and serves the path to the requsted data on disk. For metadata, the cache can also directly serve a up a Pandas Dataframe. See the ``getting_started`` notebook for more details on using the cache including installing it if it has not already been.\n",
     "\n",
+    "The commented section that calls ``from_local_cache`` can be used instead when a download of the data already exists on disk or running on CodeOcean with the attached Allen Brain Cell Atlas or similar mounting of the AWS s3 bucket as a directory through [s3fs-fuse](https://github.com/s3fs-fuse/s3fs-fuse) or similar.\n",
+    "\n",
     "**Change the download_base variable to where you have downloaded the data in your system.**"
    ]
   },
@@ -58,6 +60,10 @@
    "source": [
     "download_base = Path('../../abc_download_root')\n",
     "abc_cache = AbcProjectCache.from_s3_cache(download_base)\n",
+    "\n",
+    "# download_base = Path('../../data/abc_atlas') # Path to the already downloaded data or s3fs-fuse mount.\n",
+    "# abc_cache = AbcProjectCache.from_local_cache(download_base)\n",
+    "\n",
     "abc_cache.current_manifest"
    ]
   },
@@ -2421,7 +2427,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.8"
+   "version": "3.11.9"
   }
  },
  "nbformat": 4,

notebooks/10x_snRNASeq_tutorial_part_2a.ipynb

@@ -35,6 +35,8 @@
    "source": [
     "We will interact with the data using the **AbcProjectCache**. This cache object tracks which data has been downloaded and serves the path to the requsted data on disk. For metadata, the cache can also directly serve a up a Pandas Dataframe. See the ``getting_started`` notebook for more details on using the cache including installing it if it has not already been.\n",
     "\n",
+    "The commented section that calls ``from_local_cache`` can be used instead when a download of the data already exists on disk or running on CodeOcean with the attached Allen Brain Cell Atlas or similar mounting of the AWS s3 bucket as a directory through [s3fs-fuse](https://github.com/s3fs-fuse/s3fs-fuse) or similar.\n",
+    "\n",
     "**Change the download_base variable to where you have downloaded the data in your system.**"
    ]
   },
@@ -57,6 +59,10 @@
    "source": [
     "download_base = Path('../../abc_download_root')\n",
     "abc_cache = AbcProjectCache.from_s3_cache(download_base)\n",
+    "\n",
+    "# download_base = Path('../../data/abc_atlas') # Path to the already downloaded data or s3fs-fuse mount.\n",
+    "# abc_cache = AbcProjectCache.from_local_cache(download_base)\n",
+    "\n",
     "abc_cache.current_manifest"
    ]
   },
@@ -554,7 +560,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.8"
+   "version": "3.11.9"
   }
  },
  "nbformat": 4,

notebooks/10x_snRNASeq_tutorial_part_2b.ipynb

@@ -34,6 +34,8 @@
    "source": [
     "We will interact with the data using the **AbcProjectCache**. This cache object tracks which data has been downloaded and serves the path to the requsted data on disk. For metadata, the cache can also directly serve a up a Pandas Dataframe. See the ``getting_started`` notebook for more details on using the cache including installing it if it has not already been.\n",
     "\n",
+    "The commented section that calls ``from_local_cache`` can be used instead when a download of the data already exists on disk or running on CodeOcean with the attached Allen Brain Cell Atlas or similar mounting of the AWS s3 bucket as a directory through [s3fs-fuse](https://github.com/s3fs-fuse/s3fs-fuse) or similar.\n",
+    "\n",
     "**Change the download_base variable to where you have downloaded the data in your system.**"
    ]
   },
@@ -56,6 +58,10 @@
    "source": [
     "download_base = Path('../../abc_download_root')\n",
     "abc_cache = AbcProjectCache.from_s3_cache(download_base)\n",
+    "\n",
+    "# download_base = Path('../../data/abc_atlas') # Path to the already downloaded data or s3fs-fuse mount.\n",
+    "# abc_cache = AbcProjectCache.from_local_cache(download_base)\n",
+    "\n",
     "abc_cache.current_manifest"
    ]
   },
@@ -590,7 +596,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.8"
+   "version": "3.11.9"
   }
  },
  "nbformat": 4,

notebooks/WHB-10x_snRNASeq_tutorial_part_1.ipynb

@@ -36,7 +36,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "We will interact with the data using the **AbcProjectCache**. This cache object tracks which data has been downloaded and serves the path to the requsted data on disk. For metadata, the cache can also directly serve a up a Pandas Dataframe from the stored csv file. See the ``getting_started`` notebook for more details on using the cache including installing it if it has not already been.\n",
+    "We will interact with the data using the **AbcProjectCache**. This cache object tracks which data has been downloaded and serves the path to the requsted data on disk. For metadata, the cache can also directly serve a up a Pandas Dataframe. See the ``getting_started`` notebook for more details on using the cache including installing it if it has not already been.\n",
+    "\n",
+    "The commented section that calls ``from_local_cache`` can be used instead when a download of the data already exists on disk or running on CodeOcean with the attached Allen Brain Cell Atlas or similar mounting of the AWS s3 bucket as a directory through [s3fs-fuse](https://github.com/s3fs-fuse/s3fs-fuse) or similar.\n",
     "\n",
     "**Change the download_base variable to where you have downloaded the data in your system.**"
    ]
@@ -60,6 +62,10 @@
    "source": [
     "download_base = Path('../../abc_download_root')\n",
     "abc_cache = AbcProjectCache.from_s3_cache(download_base)\n",
+    "\n",
+    "# download_base = Path('../../data/abc_atlas') # Path to the already downloaded data or s3fs-fuse mount.\n",
+    "# abc_cache = AbcProjectCache.from_local_cache(download_base)\n",
+    "\n",
     "abc_cache.current_manifest"
    ]
   },
@@ -1346,7 +1352,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.8"
+   "version": "3.11.9"
   }
  },
  "nbformat": 4,

notebooks/WHB-10x_snRNASeq_tutorial_part_2.ipynb

@@ -34,6 +34,8 @@
    "source": [
     "We will interact with the data using the **AbcProjectCache**. This cache object tracks which data has been downloaded and serves the path to the requsted data on disk. For metadata, the cache can also directly serve a up a Pandas Dataframe. See the ``getting_started`` notebook for more details on using the cache including installing it if it has not already been.\n",
     "\n",
+    "The commented section that calls ``from_local_cache`` can be used instead when a download of the data already exists on disk or running on CodeOcean with the attached Allen Brain Cell Atlas or similar mounting of the AWS s3 bucket as a directory through [s3fs-fuse](https://github.com/s3fs-fuse/s3fs-fuse) or similar.\n",
+    "\n",
     "**Change the download_base variable to where you have downloaded the data in your system.**"
    ]
   },
@@ -56,6 +58,10 @@
    "source": [
     "download_base = Path('../../abc_download_root')\n",
     "abc_cache = AbcProjectCache.from_s3_cache(download_base)\n",
+    "\n",
+    "# download_base = Path('../../data/abc_atlas') # Path to the already downloaded data or s3fs-fuse mount.\n",
+    "# abc_cache = AbcProjectCache.from_local_cache(download_base)\n",
+    "\n",
     "abc_cache.current_manifest"
    ]
   },
@@ -1061,7 +1067,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.8"
+   "version": "3.11.9"
   }
  },
  "nbformat": 4,

notebooks/WHB_cluster_annotation_tutorial.ipynb

@@ -34,6 +34,8 @@
    "source": [
     "We will interact with the data using the **AbcProjectCache**. This cache object tracks which data has been downloaded and serves the path to the requsted data on disk. For metadata, the cache can also directly serve a up a Pandas Dataframe. See the ``getting_started`` notebook for more details on using the cache including installing it if it has not already been.\n",
     "\n",
+    "The commented section that calls ``from_local_cache`` can be used instead when a download of the data already exists on disk or running on CodeOcean with the attached Allen Brain Cell Atlas or similar mounting of the AWS s3 bucket as a directory through [s3fs-fuse](https://github.com/s3fs-fuse/s3fs-fuse) or similar.\n",
+    "\n",
     "**Change the download_base variable to where you have downloaded the data in your system.**"
    ]
   },
@@ -56,6 +58,10 @@
    "source": [
     "download_base = Path('../../abc_download_root')\n",
     "abc_cache = AbcProjectCache.from_s3_cache(download_base)\n",
+    "\n",
+    "# download_base = Path('../../data/abc_atlas') # Path to the already downloaded data or s3fs-fuse mount.\n",
+    "# abc_cache = AbcProjectCache.from_local_cache(download_base)\n",
+    "\n",
     "abc_cache.current_manifest"
    ]
   },
@@ -1605,7 +1611,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.8"
+   "version": "3.11.9"
   }
  },
  "nbformat": 4,

notebooks/ccf_and_parcellation_annotation_tutorial.ipynb

@@ -38,6 +38,8 @@
    "source": [
     "We will interact with the data using the **AbcProjectCache**. This cache object tracks which data has been downloaded and serves the path to the requsted data on disk. For metadata, the cache can also directly serve a up a Pandas Dataframe. See the ``getting_started`` notebook for more details on using the cache including installing it if it has not already been.\n",
     "\n",
+    "The commented section that calls ``from_local_cache`` can be used instead when a download of the data already exists on disk or running on CodeOcean with the attached Allen Brain Cell Atlas or similar mounting of the AWS s3 bucket as a directory through [s3fs-fuse](https://github.com/s3fs-fuse/s3fs-fuse) or similar.\n",
+    "\n",
     "**Change the download_base variable to where you have downloaded the data in your system.**"
    ]
   },
@@ -58,8 +60,12 @@
     }
    ],
    "source": [
-    "download_base = pathlib.Path('../../abc_download_root')\n",
+    "download_base = Path('../../abc_download_root')\n",
     "abc_cache = AbcProjectCache.from_s3_cache(download_base)\n",
+    "\n",
+    "# download_base = Path('../../data/abc_atlas') # Path to the already downloaded data or s3fs-fuse mount.\n",
+    "# abc_cache = AbcProjectCache.from_local_cache(download_base)\n",
+    "\n",
     "abc_cache.current_manifest"
    ]
   },
@@ -2659,7 +2665,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.8"
+   "version": "3.11.9"
   }
  },
  "nbformat": 4,

notebooks/cluster_annotation_tutorial.ipynb

@@ -36,6 +36,8 @@
    "source": [
     "We will interact with the data using the **AbcProjectCache**. This cache object tracks which data has been downloaded and serves the path to the requsted data on disk. For metadata, the cache can also directly serve a up a Pandas Dataframe. See the ``getting_started`` notebook for more details on using the cache including installing it if it has not already been.\n",
     "\n",
+    "The commented section that calls ``from_local_cache`` can be used instead when a download of the data already exists on disk or running on CodeOcean with the attached Allen Brain Cell Atlas or similar mounting of the AWS s3 bucket as a directory through [s3fs-fuse](https://github.com/s3fs-fuse/s3fs-fuse) or similar.\n",
+    "\n",
     "**Change the download_base variable to where you have downloaded the data in your system.**"
    ]
   },
@@ -56,8 +58,12 @@
     }
    ],
    "source": [
-    "download_base = pathlib.Path('../../abc_download_root')\n",
+    "download_base = Path('../../abc_download_root')\n",
     "abc_cache = AbcProjectCache.from_s3_cache(download_base)\n",
+    "\n",
+    "# download_base = Path('../../data/abc_atlas') # Path to the already downloaded data or s3fs-fuse mount.\n",
+    "# abc_cache = AbcProjectCache.from_local_cache(download_base)\n",
+    "\n",
     "abc_cache.current_manifest"
    ]
   },
@@ -1917,7 +1923,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.8"
+   "version": "3.11.9"
   }
  },
  "nbformat": 4,

notebooks/cluster_groups_and_embeddings_tutorial.ipynb

@@ -31,6 +31,8 @@
    "source": [
     "We will interact with the data using the **AbcProjectCache**. This cache object tracks which data has been downloaded and serves the path to the requsted data on disk. For metadata, the cache can also directly serve a up a Pandas Dataframe. See the ``getting_started`` notebook for more details on using the cache including installing it if it has not already been.\n",
     "\n",
+    "The commented section that calls ``from_local_cache`` can be used instead when a download of the data already exists on disk or running on CodeOcean with the attached Allen Brain Cell Atlas or similar mounting of the AWS s3 bucket as a directory through [s3fs-fuse](https://github.com/s3fs-fuse/s3fs-fuse) or similar.\n",
+    "\n",
     "**Change the download_base variable to where you have downloaded the data in your system.**"
    ]
   },
@@ -51,8 +53,12 @@
     }
    ],
    "source": [
-    "download_base = pathlib.Path('../../abc_download_root')\n",
+    "download_base = Path('../../abc_download_root')\n",
     "abc_cache = AbcProjectCache.from_s3_cache(download_base)\n",
+    "\n",
+    "# download_base = Path('../../data/abc_atlas') # Path to the already downloaded data or s3fs-fuse mount.\n",
+    "# abc_cache = AbcProjectCache.from_local_cache(download_base)\n",
+    "\n",
     "abc_cache.current_manifest"
    ]
   },
@@ -1379,7 +1385,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.8"
+   "version": "3.11.9"
   }
  },
  "nbformat": 4,

notebooks/cluster_neighborhood_gallery.ipynb

@@ -31,6 +31,8 @@
    "source": [
     "We will interact with the data using the **AbcProjectCache**. This cache object tracks which data has been downloaded and serves the path to the requsted data on disk. For metadata, the cache can also directly serve a up a Pandas Dataframe. See the ``getting_started`` notebook for more details on using the cache including installing it if it has not already been.\n",
     "\n",
+    "The commented section that calls ``from_local_cache`` can be used instead when a download of the data already exists on disk or running on CodeOcean with the attached Allen Brain Cell Atlas or similar mounting of the AWS s3 bucket as a directory through [s3fs-fuse](https://github.com/s3fs-fuse/s3fs-fuse) or similar.\n",
+    "\n",
     "**Change the download_base variable to where you have downloaded the data in your system.**"
    ]
   },
@@ -51,8 +53,12 @@
     }
    ],
    "source": [
-    "download_base = pathlib.Path('../../abc_download_root')\n",
+    "download_base = Path('../../abc_download_root')\n",
     "abc_cache = AbcProjectCache.from_s3_cache(download_base)\n",
+    "\n",
+    "# download_base = Path('../../data/abc_atlas') # Path to the already downloaded data or s3fs-fuse mount.\n",
+    "# abc_cache = AbcProjectCache.from_local_cache(download_base)\n",
+    "\n",
     "abc_cache.current_manifest"
    ]
   },
@@ -878,7 +884,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.8"
+   "version": "3.11.9"
   }
  },
  "nbformat": 4,

notebooks/general_accessing_10x_snRNASeq_tutorial.ipynb

@@ -33,7 +33,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "We will interact with the data using the **AbcProjectCache**. This cache object tracks which data has been downloaded and serves the path to the requsted data on disk. For metadata, the cache can also directly serve up a Pandas Dataframe. See the ``getting_started`` notebook for more details on using the cache including installing it if it has not already been.\n",
+    "We will interact with the data using the **AbcProjectCache**. This cache object tracks which data has been downloaded and serves the path to the requsted data on disk. For metadata, the cache can also directly serve a up a Pandas Dataframe. See the ``getting_started`` notebook for more details on using the cache including installing it if it has not already been.\n",
+    "\n",
+    "The commented section that calls ``from_local_cache`` can be used instead when a download of the data already exists on disk or running on CodeOcean with the attached Allen Brain Cell Atlas or similar mounting of the AWS s3 bucket as a directory through [s3fs-fuse](https://github.com/s3fs-fuse/s3fs-fuse) or similar.\n",
     "\n",
     "**Change the download_base variable to where you have downloaded the data in your system.**"
    ]
@@ -57,6 +59,10 @@
    "source": [
     "download_base = Path('../../abc_download_root')\n",
     "abc_cache = AbcProjectCache.from_s3_cache(download_base)\n",
+    "\n",
+    "# download_base = Path('../../data/abc_atlas') # Path to the already downloaded data or s3fs-fuse mount.\n",
+    "# abc_cache = AbcProjectCache.from_local_cache(download_base)\n",
+    "\n",
     "abc_cache.current_manifest"
    ]
   },

notebooks/getting_started.ipynb

@@ -144,7 +144,9 @@
     "\n",
     "Below we show how to setup up the cache to download from S3, how to list and switch to a different data release, and additionally how to list the directories available, their size, and the files in that directory.\n",
     "\n",
-    "Setup the **AbcProjectCache** object by specifying a directory and calling ``from_s3_cache`` as shown below. We also print what version of the manifest is being currently loaded by the cache."
+    "Setup the **AbcProjectCache** object by specifying a directory and calling ``from_s3_cache`` as shown below. We also print what version of the manifest is being currently loaded by the cache.\n",
+    "\n",
+    "The commented section that calls ``from_local_cache`` can be used instead when a download of the data already exists on disk or running on CodeOcean with the attached Allen Brain Cell Atlas or similar mounting of the AWS s3 bucket as a directory through [s3fs-fuse](https://github.com/s3fs-fuse/s3fs-fuse) or similar. Using the cache from a s3fs mount means that all files will be streamed over the internet instead of saved to the local disk."
    ]
   },
   {
@@ -166,6 +168,10 @@
    "source": [
     "download_base = Path('../../abc_download_root') # Path to where you would like to write the downloaded data.\n",
     "abc_cache = AbcProjectCache.from_s3_cache(download_base)\n",
+    "\n",
+    "# download_base = Path('../../data/abc_atlas') # Path to the already downloaded data or s3fs-fuse mount.\n",
+    "# abc_cache = AbcProjectCache.from_local_cache(download_base)\n",
+    "\n",
     "abc_cache.current_manifest"
    ]
   },