Data_exploration_template_tabular.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Template to get started with data exploration\n",
    "\n",
    "The other notebooks show the results of existing analysis. Most of the resulting code has been moved from the notebooks to the associated python modules, in order to showcase the results. But that makes it harder to experiment with the data and come up with new analyses. This is particularly true because the current data structures that store the data are a little complicated. Maybe after we switch to xarrays in the future, we will no longer need this!\n",
    "\n",
    "But for now, users can use this exploration template and plug in their code/analyses here. And finally, they can put the code into a module for re-use elsewhere"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set up the dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for reading and validating data\n",
    "import emeval.input.spec_details as eisd\n",
    "import emeval.input.phone_view as eipv\n",
    "import emeval.input.eval_view as eiev\n",
    "from emeval.input.tabularize import tabularize_pv_map"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Visualization helpers\n",
    "import emeval.viz.phone_view as ezpv\n",
    "import emeval.viz.eval_view as ezev\n",
    "import emeval.viz.geojson as ezgj"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For plots\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For maps\n",
    "import folium\n",
    "import branca.element as bre"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For easier debugging while working on modules\n",
    "import importlib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import arrow\n",
    "from IPython.display import display"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The spec\n",
    "\n",
    "The spec defines what experiments were done, and over which time ranges. Once the experiment is complete, most of the structure is read back from the data, but we use the spec to validate that it all worked correctly. The spec also contains the ground truth for the legs. Here, we read the spec for the trip to UC Berkeley."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "AUTHOR_EMAIL = \"shankari@eecs.berkeley.edu\"\n",
    "\n",
    "# If using ServerSpecDetails, data can alternatively be retrieved as such:\n",
    "# DATASTORE_LOC = \"http://localhost:8080\"\n",
    "# sd = eisd.ServerSpecDetails(DATASTORE_LOC, AUTHOR_EMAIL, \"train_bus_ebike_mtv_ucb\")\n",
    "\n",
    "# You must run `cd bin/ && python dump_data_to_file.py --spec-id train_bus_ebike_mtv_ucb`\n",
    "# before using this notebook!\n",
    "\n",
    "DATASTORE_LOC = \"bin/data/\"\n",
    "sd = eisd.FileSpecDetails(DATASTORE_LOC, AUTHOR_EMAIL, \"train_bus_ebike_mtv_ucb\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The views\n",
    "\n",
    "There are two main views for the data - the phone view and the evaluation view. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Phone view\n",
    "\n",
    "In the phone view, the phone is primary, and then there is a tree that you can traverse to get the data that you want. Traversing that tree typically involves nested for loops; here's an example of loading the phone view and traversing it. You can replace the print statements with real code. When you are ready to check this in, please move the function to one of the python modules so that we can invoke it more generally"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pv = eipv.PhoneView(sd)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `tabularize_pv_map` function in `emeval.input.tabularize` can be used to convert a phone view tree into a series of dataframes for more intuitive querying. The dataframes are organized by the operating systems of the test phones and the type of data they hold."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pv_dfs = tabularize_pv_map(pv.map())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for phone_os, df_map in pv_dfs.items():\n",
    "    for df_label, df in df_map.items():\n",
    "        print(f\"{phone_os=}, {df_label=}\")\n",
    "        print(df.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Eval view\n",
    "\n",
    "In the eval view, the experiment is primary, and then there is a similar tree that you can traverse to get the data that you want. Traversing that tree typically involves nested for loops; here's an example of manipulating the eval view and traversing it. You can replace the print statements with real code. When you are ready to check this in, please move the function to one of the python modules so that we can invoke it more generally"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ev = eiev.EvaluationView()\n",
    "ev.from_view_eval_trips(pv, \"\", \"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see these evaluation trips have labels such as `HAHFDC_0`, or `HAMFDC_0`. What do these mean?\n",
    "\n",
    "`HA` or `MA` refer to high accuracy or medium accuracy, respectively. Trips with `HA` will tend to favor GPS utilization and result in high power consumption.\n",
    "\n",
    "`HF` or `MF` refer to high frequency or medium accuracy, respectively. Trips with `HF` will sense and process more often, and are likely to have higher spatiotemporal accuracy (e.g. will hug corners more accurately), albeit with higher power consumption."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for phone_os, eval_map in ev.map(\"evaluation\").items():\n",
    "    print(15 * \"=*\")\n",
    "    print(phone_os, eval_map.keys())\n",
    "    for (curr_calibrate, curr_calibrate_trip_map) in eval_map.items():\n",
    "        print(4 * ' ', 15 * \"-*\")\n",
    "        print(4 * ' ', curr_calibrate, curr_calibrate_trip_map.keys())\n",
    "        for trip_id, trip_map in curr_calibrate_trip_map.items():\n",
    "            print(8 * ' ', 30 * \"=\")\n",
    "            print(8 * ' ', trip_id, trip_map.keys())\n",
    "            for run_label, tr in trip_map.items():\n",
    "                print(12 * ' ', 30 * \"=\")\n",
    "                print(12 * ' ', run_label, tr.keys())\n",
    "                for sr in tr[\"evaluation_section_ranges\"]:\n",
    "                    print(16 * ' ', 30 * \"~\")\n",
    "                    print(16 * ' ',sr[\"trip_id\"], sr.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ground truth\n",
    "\n",
    "The ground truth is stored in the spec, and we can retrieve it from there. Once we have retrieved the trip, there are many possible analyses using them. Please see `get_concat_trajectories` for an example.\n",
    "\n",
    "We can confirm that there are valid timestamp (`ts`) values for each trip range:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for phone_os, df_map in pv_dfs.items():\n",
    "    for df_label, df in df_map.items():\n",
    "        print(\"-\"*10, f\"{phone_os=}, {df_label=}\", \"-\"*10)\n",
    "        if \"trip_range_id\" in df.columns:\n",
    "            tr_ids = df[\"trip_range_id_base\"].dropna().unique()\n",
    "            for tr_id in tr_ids:\n",
    "                tr_id_rows = df.query(f\"trip_range_id_base == '{tr_id}'\")[[\"trip_range_id_base\", \"trip_range_gt_start_ts\", \"trip_range_gt_end_ts\"]].drop_duplicates()\n",
    "                for _, _, gt_start_ts, gt_end_ts in (tr_id_rows.itertuples()):\n",
    "                    assert not df.query(f\"{gt_start_ts} < ts < {gt_end_ts}\")[\"ts\"].isnull().values.any()\n",
    "        print(\"Has valid `ts` values for each trip range ✅ \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As well as each section range:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for phone_os, df_map in pv_dfs.items():\n",
    "    for df_label, df in df_map.items():\n",
    "        print(\"-\"*10, f\"{phone_os=}, {df_label=}\", \"-\"*10)\n",
    "        if \"section_range_id\" in df.columns:\n",
    "            sr_ids = df[\"section_range_id_base\"].dropna().unique()\n",
    "            for sr_id in sr_ids:\n",
    "                sr_id_rows = df.query(f\"section_range_id_base == '{sr_id}'\")[[\"section_range_id_base\", \"section_range_gt_start_ts\", \"section_range_gt_end_ts\"]].drop_duplicates()\n",
    "                for _, _, gt_start_ts, gt_end_ts in (sr_id_rows.itertuples()):\n",
    "                    assert not df.query(f\"{gt_start_ts} < ts < {gt_end_ts}\")[\"ts\"].isnull().values.any()\n",
    "        print(\"Has valid `ts` values for each section range ✅ \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### For trips\n",
    "\n",
    "Using the phone view dataframes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tr_subset = [\"trip_range_id_base\", \"trip_range_gt_start_ts\", \"trip_range_gt_end_ts\"]\n",
    "for phone_os, df_map in pv_dfs.items():\n",
    "    for df_label, df in df_map.items():\n",
    "        print(\"-\"*10, f\"{phone_os=}, {df_label=}\", \"-\"*10)\n",
    "        if \"trip_range_id\" in df.columns:\n",
    "            tr_info = df[tr_subset].drop_duplicates().dropna()\n",
    "            for _, tr_id_base, gt_start_ts, gt_end_ts in tr_info.itertuples():\n",
    "                print(f\"{tr_id_base=}, {gt_start_ts=}, {gt_end_ts=}\")\n",
    "                gt_trip = sd.get_ground_truth_for_trip(tr_id_base, gt_start_ts, gt_end_ts)\n",
    "                print(eisd.SpecDetails.get_concat_trajectories(gt_trip)[\"properties\"], end=\"\\n\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### For sections\n",
    "\n",
    "Same as above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sr_subset = [\"trip_range_id_base\", \"section_range_gt_start_ts\", \"section_range_gt_end_ts\"]\n",
    "for phone_os, df_map in pv_dfs.items():\n",
    "    for df_label, df in df_map.items():\n",
    "        print(\"-\"*10, f\"{phone_os=}, {df_label=}\", \"-\"*10)\n",
    "        if \"section_range_id\" in df.columns:\n",
    "            sr_info = df[sr_subset].drop_duplicates().dropna()\n",
    "            for _, tr_id_base, gt_start_ts, gt_end_ts in sr_info.itertuples():\n",
    "                print(f\"{tr_id_base=}, {gt_start_ts=}, {gt_end_ts=}\")\n",
    "                gt_trip = sd.get_ground_truth_for_trip(tr_id_base, gt_start_ts, gt_end_ts)\n",
    "                print(eisd.SpecDetails.get_concat_trajectories(gt_trip)[\"properties\"], end=\"\\n\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Work with a single trip\n",
    "\n",
    "You can also work with the details of a single trip - here, we look at the battery drain across phones for the third repetition. Code inspired by `plot_all_power_drain`, located in `emeval.viz.phone_view`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ifig, ax = plt.subplots(ncols=1, nrows=1, figsize=(10,5))\n",
    "for phone_os, df_map in pv_dfs.items():\n",
    "    battery_df = df_map[\"battery_df\"]\n",
    "    for i, phone_label in enumerate(battery_df[\"phone_label\"].unique()):\n",
    "        third_rep = battery_df.query(\n",
    "            f\"phone_label == '{phone_label}' and range_type == 'evaluation' and range_index == 2\")\n",
    "        role = third_rep[\"role\"].unique()[0]\n",
    "        third_rep.plot(ax=ax, x=\"hr\", y=\"battery_level_pct\", label=f\"{phone_label} ({role})\", ylim=(0, 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Work with a single leg\n",
    "\n",
    "You can also work with the details of a single leg. This is not likely to be useful for power estimates because there are so few points, but it is going to be easier to work with trajectory estimates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bart_leg = pv_dfs[\"ios\"][\"location_df\"].query(\n",
    "    \"phone_label == 'ucb-sdb-ios-1'\"\n",
    "    \"and range_type == 'evaluation'\"\n",
    "    \"and range_index == 2\"\n",
    "    \"and trip_range_index == 0\"\n",
    "    \"and section_range_index == 5\"\n",
    ")\n",
    "\n",
    "bart_leg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gt_subset = [\"trip_range_id_base\", \"section_range_id_base\", \"section_range_gt_start_ts\", \"section_range_gt_end_ts\"]\n",
    "\n",
    "tr_id_base, sr_id_base, gt_start_ts, gt_end_ts = tuple(bart_leg[gt_subset].drop_duplicates().iloc[0].values)\n",
    "\n",
    "gt_leg = sd.get_ground_truth_for_leg(tr_id_base, sr_id_base, gt_start_ts, gt_end_ts)\n",
    "\n",
    "gt_leg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Display the leg\n",
    "\n",
    "Note the layer control on the map that allows you to toggle the lines separately"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "curr_map = folium.Map()\n",
    "gt_leg_gj = sd.get_geojson_for_leg(gt_leg)\n",
    "sensed_section_gj = ezgj.get_geojson_for_loc_df(bart_leg)\n",
    "gt_leg_gj_feature = folium.GeoJson(gt_leg_gj, name=\"ground_truth\")\n",
    "sensed_leg_gj_feature = folium.GeoJson(sensed_section_gj, name=\"sensed_values\")\n",
    "curr_map.add_child(gt_leg_gj_feature)\n",
    "curr_map.add_child(sensed_leg_gj_feature)\n",
    "curr_map.fit_bounds(sensed_leg_gj_feature.get_bounds())\n",
    "folium.LayerControl().add_to(curr_map)\n",
    "curr_map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Display the leg with points\n",
    "\n",
    "In this case, the points are in a separate layer so they can be toggled indepdendently of the underlying lines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "curr_map = folium.Map()\n",
    "gt_leg_gj = sd.get_geojson_for_leg(gt_leg)\n",
    "sensed_section_gj = ezgj.get_geojson_for_loc_df(bart_leg)\n",
    "gt_leg_gj_feature = folium.GeoJson(gt_leg_gj, name=\"ground_truth\")\n",
    "gt_leg_gj_points = ezgj.get_point_markers(gt_leg_gj[2], name=\"ground_truth_points\", color=\"green\")\n",
    "sensed_leg_gj_feature = folium.GeoJson(sensed_section_gj, name=\"sensed_values\")\n",
    "sensed_leg_gj_points = ezgj.get_point_markers(sensed_section_gj, name=\"sensed_points\", color=\"red\")\n",
    "curr_map.add_child(gt_leg_gj_feature)\n",
    "curr_map.add_child(gt_leg_gj_points)\n",
    "curr_map.add_child(sensed_leg_gj_feature)\n",
    "curr_map.add_child(sensed_leg_gj_points)\n",
    "curr_map.fit_bounds(sensed_leg_gj_feature.get_bounds())\n",
    "folium.LayerControl().add_to(curr_map)\n",
    "curr_map"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}