03_time_series_extract.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9e262b60-a61c-4fb6-adfb-c00d1ebda073",
   "metadata": {},
   "source": [
    "<font face=\"Calibri\" size=\"2\"> <i>eSBAE - Notebook Series - Part 3, version 0.4, March 2023. Andreas Vollrath, UN-Food and Agricultural Organization, Rome</i>\n",
    "</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d64fe01-50b8-4280-9629-1445ad541e45",
   "metadata": {},
   "source": [
    "![title](images/header.png)\n",
    "\n",
    "# III - eSBAE Time-Series Extraction\n",
    "### Extract various time-series data for large sets of points from Google Earth Engine\n",
    "-------\n",
    "\n",
    "This notebook takes you through the process of extracting time-series for a set of points using [Google's Earth Engine](https://earthengine.google.com/). The script is optimized to deal with thousands of points and will use parallelization to efficiently extract the information from the platform.\n",
    "\n",
    "**You will need**:\n",
    "- a valid Earth Engine account ([sign up here](https://code.earthengine.google.com/register))\n",
    "- an uploaded table of points (Feature Collection) \n",
    "- the table needs a unique point identifier (Point ID)\n",
    "\n",
    "**You should be aware, that:** \n",
    "\n",
    "- As a SEPAL user: this notebook does **not need huge resources**, as processing is done on the platform. A **m2 instance** is best suited.  \n",
    "- The extraction can take up to days (>100000 points). If you are on SEPAL, make use of the **\"keep instance running\"** option within the user report dashboard. However, **do not forget** to shut down your machine once processing finished. \n",
    "- A logfile is created within your tmp-folder. Interruption of connectivity to the SEPAL server may lead to block the output of the Jupyter notebook. **This does not mean the processing stopped.** You can see in esbae_log_(time) if the processing is still on going. \n",
    "- You can restart the kernel and execute all cells, and extraction will **start where it stopped**. This is also valid, if your instance has been shut down before processing was completely finished."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f1f9d9a-8a0d-4d85-b134-f0bda3727152",
   "metadata": {},
   "source": [
    "### 1 - Import libraries (*only execute this cell*)\n",
    "\n",
    "This cell will provide us with the functionality we need for running the subsequent cells of the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a75e78de-8687-4632-8586-9ca964b31045",
   "metadata": {},
   "outputs": [],
   "source": [
    "# initialize EE    \n",
    "import ee\n",
    "try:\n",
    "    ee.Initialize(opt_url='https://earthengine-highvolume.googleapis.com')\n",
    "except:\n",
    "    ee.Authenticate()\n",
    "    ee.Initialize(opt_url='https://earthengine-highvolume.googleapis.com')\n",
    "    \n",
    "from sampling_handler import TimeSeriesExtraction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23c04e79-5bfe-48d9-8412-c17707d7c5e9",
   "metadata": {},
   "source": [
    "### 2 - Basic Input Variables\n",
    "\n",
    "Here a so called class instance is initialized. The class instance needs some parameters to be set and is written into the *esbae* variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "22c8d21f-1d84-44b2-9b44-3458bc7ff444",
   "metadata": {},
   "outputs": [],
   "source": [
    "esbae = TimeSeriesExtraction(\n",
    "     # your project name that you use fo all of the notebooks\n",
    "    project_name  = 'my_first_esbae_project',\n",
    "    \n",
    "    # your start and end date. \n",
    "    # NOTE that this should go further back to the past than the \n",
    "    # envisaged monitoing period for calibration purposes\n",
    "    ts_start      = '2015-01-01',      # YYYY-MM-DD format\n",
    "    ts_end        = '2023-01-01',        # YYYY-MM-DD format\n",
    "    \n",
    "    # satellite platform (for now only Landsat is supported)\n",
    "    satellite     = 'Landsat',\n",
    "    \n",
    "    # at what resolution in metres you want to extract (shall conform with forest definition MMU)\n",
    "    scale         = 70, # pixel size in metres\n",
    "    \n",
    "    # wether the TS will be extracted on a bounding box with diameter scale with original scale (e.g 30m for Landsat) of the underlying data (True), \n",
    "    # or if the underlying data is rescaled to the scale (False)\n",
    "    # setting it to True might be more accurate, but tends to be slower\n",
    "    bounds_reduce = False,\n",
    "    \n",
    "    # bands\n",
    "    bands         =  [\n",
    "        'green', 'red', 'nir', 'swir1', 'swir2',   # reflectance bands\n",
    "        'ndfi', #'ndmi', 'ndvi',                    # indices\n",
    "        'brightness', 'greenness', 'wetness'       # Tasseled Cap \n",
    "    ], \n",
    "    # This is in case you haven't run notebook 1 and 2, and want to directly start from here\n",
    "    #aoi = ee.FeatureCollection(ee.FeatureCollection('users/username/my_points').geometry().convexHull(100))\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d4dc5b2-3d18-429e-ae4a-5b16f677448a",
   "metadata": {},
   "source": [
    "### 3 - Landsat parameters\n",
    "\n",
    "Here you can select, which satellites you want to include from the Landsat mission.\n",
    "In addition you can select the BRDF correction and a filter for maximum cloud cover. Note that the bands parameter is already set in the initialization and will be taken from the class attribute. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "94d1fb79-897c-4a86-9b02-16e2c7d85758",
   "metadata": {},
   "outputs": [],
   "source": [
    "# landsat related parameters\n",
    "lsat_params = {\n",
    "    'l9': True,\n",
    "    'l8': True,\n",
    "    'l7': True,\n",
    "    'l5': True,\n",
    "    'l4': True,\n",
    "    'brdf': True,\n",
    "    'bands': esbae.bands,\n",
    "    'max_cc': 75    # percent\n",
    "} \n",
    "\n",
    "# apply the basic configuration set in the cell above\n",
    "esbae.lsat_params = lsat_params"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf83cb8d-3f49-4bdf-ae77-25b1b13a6de3",
   "metadata": {},
   "source": [
    "### 4 - Processing parameters\n",
    "\n",
    "Here you can refine the parallelization options. For efficient extraction, the time-series extraction is done on chunks of data, defined by squared grids of given sizes. The routine will check how many points are in each chunk. If this is below the max_points_per_chunk, it will go on and process those points. Otherwise it will try to process those points at a lower grid size level. Some optimized settings are given below, comment and uncomment as appropriate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4c99611d-a405-49c1-941f-4700cda2e61c",
   "metadata": {},
   "outputs": [],
   "source": [
    "esbae.workers = 10                   # this defines how many parallel requests will be send to EarthEngine at a time\n",
    "esbae.max_points_per_chunk = 100     # this defines the maximum amount of points as send per request to Earth Engine at a time\n",
    "\n",
    "# this defines the chunk sizes (in degree) to create the requests\n",
    "#esbae.grid_size_levels = [0.1, 0.075, 0.05]   # optimized for 1km systematic grid\n",
    "esbae.grid_size_levels = [0.2, 0.15, 0.1]    # optimized for 2km systematic grid\n",
    "#esbae.grid_size_levels = [0.4, 0.3, 0.2]     # optimized for 4km systematic grid"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6ae865d-43bf-495b-b6ce-e8be7fd12884",
   "metadata": {},
   "source": [
    "### 5 - Set a custom grid \n",
    "\n",
    "This step is only necessary if you skipped notebook 2. You then need to define an Earth Engine feature collection as well as the unique point identifier. Uncomment the lines by removing the #"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "786e8355-4b7e-4f47-8a30-62ad5d8321c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "#esbae.sample_asset = 'users/username/my_already_existing_points'\n",
    "#esbae.pid = 'my_unique_point_id'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58ed4350-c397-43eb-ab7e-fe795a24c831",
   "metadata": {},
   "source": [
    "### 4 - Check for already processed data (optional)\n",
    "\n",
    "This is useful for large points sizes and when the connection to Sepal gets interrupted. Usually processing will continue, but it is not straightforward to track progress. \n",
    "You can instead restart the kernel, execute all cells and see if processing has been finished with the following line of code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad269ed4-9e28-4d4b-a629-d05513d41376",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "esbae.check_if_completed()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d5c28df-5381-413c-924d-23830b97b87e",
   "metadata": {},
   "source": [
    "### 5 - Run the time-series data extraction *(only execute this)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01ccf187-bba1-4e1a-a493-d646f51b9fc3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "esbae.get_time_series_data()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": " (venv) Sample Based Area Estimation",
   "language": "python",
   "name": "venv-esbae_notebooks"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "toc-autonumbering": false
 },
 "nbformat": 4,
 "nbformat_minor": 5
}