python/how-tos/How_to_find_and_access_EMIT_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to: Find and Access EMIT Data\n",
    "\n",
    "**Summary**  \n",
    "\n",
    "There are currently 3 ways to find EMIT data:\n",
    "\n",
    "1. [EarthData Search](https://search.earthdata.nasa.gov/search)\n",
    "2. [NASA's CMR API](https://www.earthdata.nasa.gov/eosdis/science-system-description/eosdis-components/cmr)\n",
    "3. [Visions Open Access Data Portal](https://earth.jpl.nasa.gov/emit/data/data-portal/coverage-and-forecasts/)\n",
    "\n",
    "This notebook will explain how to access Earth Surface Minteral Dust Source Investigation (EMIT) data programmaticly using the [earthaccess python library](https://github.com/nsidc/earthaccess). `earthaccess` is an easy to use library that reduces finding and downloading or streaming data over https or s3 to only a few lines of code. `earthaccess` searches NASA's Common Metadata Repository (CMR), a metadata system that catalogs Earth Science data and associated metadata records, then can be used to download granules or generate lists granule search result URLs.\n",
    "\n",
    "**Requirements:**\n",
    "+ A NASA [Earthdata Login](https://urs.earthdata.nasa.gov/) account is required to download EMIT data   \n",
    "+ Selected the `emit_tutorials` environment as the kernel for this notebook.\n",
    "  + For instructions on setting up the environment, follow the the `setup_instructions.md` included in the `/setup/` folder of the repository.  \n",
    "\n",
    "**Learning Objectives**  \n",
    "- How to search and access EMIT data using `earthaccess`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import the required packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import earthaccess\n",
    "import pandas as pd\n",
    "import geopandas as gp\n",
    "import xarray as xr\n",
    "import sys\n",
    "sys.path.append('../modules/')\n",
    "from emit_tools import emit_xarray\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Authentication\n",
    "\n",
    "Login to your NASA Earthdata account and create a `.netrc` file using the `login` function from the `earthaccess` library. If you do not have an Earthdata Account, you can create one [here](https://urs.earthdata.nasa.gov/home). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "earthaccess.login(persist=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Searching for Collections\n",
    "\n",
    "If we want to see the available EMIT collections, we can \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Query = earthaccess.collection_query().keyword('emit').provider('LPCLOUD')\n",
    "print(f'Collections found: {Query.hits()}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can retrieve metadata for these collections, and then the shortnames so we can search for granules. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find Collections\n",
    "collections = Query.fields(['ShortName']).get()\n",
    "# Retrieve Collection Short-names\n",
    "[product['short-name'] for product in [collection.summary() for collection in collections]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you print the `collections` object you can explore all of the json metadata."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Searching for Granules\n",
    "\n",
    "A granule can be thought of as a unique spatiotemporal grouping within a collection. To search for granules, we simply use the `search_data` function from `earthaccess` and provide the arguments for our search. Its possible to specify search products using several criteria shown in the table below:\n",
    "\n",
    "|dataset origin and location|spatio temporal parameters|dataset metadata parameters|\n",
    "|:---|:---|:---|\n",
    "|archive_center|bounding_box|concept_id\n",
    "|data_center|temporal|entry_title\n",
    "|daac|point|keyword\n",
    "|provider|polygon|version\n",
    "|cloud_hosted|line|short_name\n",
    "\n",
    "### Point Search\n",
    "\n",
    "In this case, we specify the `shortname`, `point`, and `temporal`, as well as `count`, which limits the maximum number of results returned. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# POINT\n",
    "results = earthaccess.search_data(\n",
    "    short_name='EMITL2ARFL',\n",
    "    point=(-62.1123,-39.89402),\n",
    "    temporal=('2022-09-03','2022-09-04'),\n",
    "    count=100\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bounding Box Search\n",
    "\n",
    "You can also use a bounding box to search. To do this we will first open a geojson file containing our region of interest (ROI) then simplify it to a bounding box by getting the bounds and putting them into a tuple. We will use the `total_bounds` property to get the bounding box of our ROI, and add that to a python tuple, which is the expected data type for the bounding_box parameter `earthaccess` `search_data`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "geojson = gp.read_file('../../data/isla_gaviota.geojson')\n",
    "geojson.geometry"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bbox = tuple(list(geojson.total_bounds))\n",
    "bbox"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can search for granules using the a bounding box."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Search Example using Bounding Box\n",
    "results = earthaccess.search_data(\n",
    "    short_name='EMITL2ARFL',\n",
    "    bounding_box=bbox,\n",
    "    temporal=('2022-09-03','2022-09-04'),\n",
    "    count=100\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Polygon Search\n",
    "\n",
    "A polygon can also be used to search. For a simple polygon without holes we can take the geojson we opened and grab the coordinates of the exterior ring and place them in a list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "polygon = list(geojson.geometry[0].exterior.coords)\n",
    "polygon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this list of coordinate pairs we can use the `polygon` parameter for our search. \n",
    "> Note that we overwrote the `results` object, because for all 3 types spatial search, the `results` are the same for this example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Search Example using a Polygon\n",
    "results = earthaccess.search_data(\n",
    "    short_name='EMITL2ARFL',\n",
    "    polygon=polygon,\n",
    "    temporal=('2022-09-03','2022-09-04'),\n",
    "    count=100\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Working with Search Results\n",
    "\n",
    "After we've gotten results from our search using `earthaccess` we can view the results in a table and view assets for each granule in the list. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.json_normalize(results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After we have our results, there are 2 ways we an work with the data:\n",
    "\n",
    "1. Download\n",
    "2. Access in place / Stream the data. \n",
    "\n",
    "To download the data we can simply use the download function. This will retrieve all assets associated with a granule, and is nice if you plan to work with the data in this way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# earthaccess.download(results, '../../data/')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to stream the data or further filter the assets for download we want to first create a list of URLs nested by granule using list comprehesion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "emit_results_urls = [granule.data_links() for granule in results]\n",
    "emit_results_urls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can also split these into results for specific assets or filter out an asset using the following. In this example, we only want to access or download reflectance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filtered_asset_links = []\n",
    "# Pick Desired Assets (leave _ on RFL to distinguish from RFLUNC)\n",
    "desired_assets = ['RFL_'] # Add more or do individually for reflectance, reflectance uncertainty, or mask\n",
    "# Step through each sublist (granule) and filter based on desired assets.\n",
    "for n, granule in enumerate(emit_results_urls):\n",
    "    for url in granule: \n",
    "        asset_name = url.split('/')[-1]\n",
    "        if any(asset in asset_name for asset in desired_assets):\n",
    "            filtered_asset_links.append(url)\n",
    "filtered_asset_links"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After we have our filtered list, we can stream the reflectance asset or download it. Start an https session then open it to stream the data, or download to save the file.\n",
    "\n",
    "#### Stream Data\n",
    "\n",
    "This may take a while to load the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get Https Session using Earthdata Login Info\n",
    "fs = earthaccess.get_fsspec_https_session()\n",
    "# Retrieve granule asset ID from URL (to maintain existing naming convention)\n",
    "url = filtered_asset_links[0]\n",
    "granule_asset_id = url.split('/')[-1]\n",
    "# Define Local Filepath\n",
    "fp = fs.open(url)\n",
    "# Open with `emit_xarray` function\n",
    "ds = emit_xarray(fp)\n",
    "ds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Download Filtered "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get requests https Session using Earthdata Login Info\n",
    "fs = earthaccess.get_requests_https_session()\n",
    "# Retrieve granule asset ID from URL (to maintain existing naming convention)\n",
    "for url in filtered_asset_links:\n",
    "    granule_asset_id = url.split('/')[-1]\n",
    "    # Define Local Filepath\n",
    "    fp = f'../../data/{granule_asset_id}'\n",
    "    # Download the Granule Asset if it doesn't exist\n",
    "    if not os.path.isfile(fp):\n",
    "        with fs.get(url,stream=True) as src:\n",
    "            with open(fp,'wb') as dst:\n",
    "                for chunk in src.iter_content(chunk_size=64*1024*1024):\n",
    "                    dst.write(chunk)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contact Info:  \n",
    "\n",
    "Email: LPDAAC@usgs.gov  \n",
    "Voice: +1-866-573-3222  \n",
    "Organization: Land Processes Distributed Active Archive Center (LP DAAC)¹  \n",
    "Website: <https://lpdaac.usgs.gov/>  \n",
    "Date last modified: 07-03-2023  \n",
    "\n",
    "¹Work performed under USGS contract G15PD00467 for NASA contract NNG14HH33I. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  },
  "vscode": {
   "interpreter": {
    "hash": "3292b2aceff7d39327a7519422d4180a7c9b133202090f26e797e3dd8f2c7877"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}