Altair.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Altair"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Handy hints "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some Jupyter shortcut keys:\n",
    "\n",
    "- Esc : get into Command mode (leaves Edit mode)\n",
    "- Enter : edit a cell (puts you in Edit mode)\n",
    "- h : see help (see all commands)\n",
    "- Shift+Enter or Ctrl+Enter : run the code in the cell\n",
    "- a / b : add new cell above/below\n",
    "- m : turn current cell into a Markdown cell\n",
    "- y : turn current cell into a Code cell\n",
    "- 1-6 : turn current cell into a heading (a type of Markdown cell)\n",
    "\n",
    "You can get help on Python objects and functions with `help()` or the `?` operator.\n",
    "\n",
    "In general, we are using plotting libraries that return objects encapsulating the plot. You can check the type of these returned objects with `type()`. \n",
    "\n",
    "Some of the plotting libraries we use need to communicate a lot of data to the browser. Current versions of Jupyter shouldn't have a problem, but if you are running an older version, you may need to launch this notebook with a higher data rate limit: `jupyter notebook --NotebookApp.iopub_data_rate_limit=10000000`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Altair is not bundled with Anaconda - you may need\n",
    "\n",
    "`conda install -c conda-forge altair vega`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-01-15T00:55:16.420432Z",
     "start_time": "2018-01-15T00:55:16.416622Z"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-01-15T00:55:16.929054Z",
     "start_time": "2018-01-15T00:55:16.903001Z"
    }
   },
   "outputs": [],
   "source": [
    "# We may want to use some colours etc from other libraries\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import altair as alt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this workshop we're going to load in two main datasets:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### LTEE data "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This data is from the [Long Term Evolution Experiment (LTEE)](http://myxo.css.msu.edu/ecoli/). This experiment has been running for over 30 years and over 50,000 E. coli generations, and is still ongoing. Twelve separate populations of E. coli have been propagated for the life of the experiment. Every 500 generations, each population is cloned and stored, allowing researchers to study evolutionary behaviour over the long term, and to \"rewind and replay\" alternate evolutionary trajectories by propagating from an earlier clone. \n",
    "\n",
    "Several interesting events have occurred during the experiment. Some populations have spontaneously developed hypermutator phenotypes. In addition, around generation 31,000 one population, Ara-3, spontaneously developed a rare and novel Cit+ mutation, giving it the ability to metabolise citrate in the substrate.\n",
    "\n",
    "There have been many publications from this experiment. A handful:\n",
    "\n",
    "- [Blount et al 2008: Historical contingency and the evolution of a key innovation in an experimental population of Escherichia coli](https://www.pnas.org/content/105/23/7899) - on the spontaneous development of citrate metabolisation and on potentiating mutations\n",
    "- [Tenaillon et al 2016: Tempo and mode of genome evolution in a 50,000-generation experiment](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4988878) - various investigations by sequencing and variant-calling over 50,000 generations of clones, including discussion of hypermutator phenotypes and genetic drift vs natural selection.\n",
    "\n",
    "Sequence data from clones is available, but for this workshop we'll just be using some published tabular data.\n",
    "\n",
    "A version of this dataset is also used by the [Data Carpentry lessons on Genomics](https://datacarpentry.org/genomics-workshop/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lesson we'll use a large flat file containing both sample metadata on each clone, and information on observed mutations in their genomes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you have done the pandas and data tidying workshop and saved the final file,\n",
    "# you can try reading that file in here if you prefer\n",
    "ltee = pd.read_csv('ltee_merged.csv',\n",
    "                   index_col = 'Strain ID')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### House sales data "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This data represents house sales in Seattle in 2014 and 2015. We have information on the houses themselves: location, size, quality, view, and whether the house is tagged as 'waterfront' or not. We also have information on the date and price of each sale."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-01-14T23:56:53.195524Z",
     "start_time": "2018-01-14T23:56:53.140978Z"
    }
   },
   "outputs": [],
   "source": [
    "sales = pd.read_csv(\"housing-data-10000.csv\", \n",
    "                    usecols=['id','date','price','lat','long', 'zipcode',\n",
    "                             'waterfront','view','grade','sqft_living'],\n",
    "                    parse_dates=['date'], \n",
    "                    dtype={'zipcode': 'category',\n",
    "                           'waterfront': 'bool'})\n",
    "\n",
    "# We'll downsample in order to make plots a bit simpler and smaller to store\n",
    "# In particular Altair defaults to a limit of 5000 observations\n",
    "sales = sales.sample(3000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Demo data "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll read in the classic `iris` and `mtcars` datasets for demo purposes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris = pd.read_csv('iris.csv')\n",
    "cars = pd.read_csv('mtcars.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Altair "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Altair](https://altair-viz.github.io) is a library for creating interactive plots. \n",
    "\n",
    "Altair is built around the [Vega-Lite](https://vega.github.io/vega-lite/) schema, a \"visualisation grammar\". Altair plots are specified in Python, then converted behind the scenes to a declarative JSON structure that follows the Vega-Lite schema, which can then be rendered by a Javascript library.\n",
    "\n",
    "Altair works very well with Pandas - in fact, it usually expects data to be in a Pandas DataFrame."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Simple interactive plotting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's a simple example of an Altair plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame({\n",
    "    'Time': [1,2,3,4,5],\n",
    "    'Projected': [2,5,10,17,26],\n",
    "    'Actual': [1,4,9,11,9]\n",
    "})\n",
    "\n",
    "chart = alt.Chart(df)\n",
    "\n",
    "chart.mark_line(color='blue').encode(x='Time', y='Projected')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here `mark_line()` tells Altair we want to draw a line, and `encode()` is used to assign our variables (columns of the DataFrame) to possible encoding channels of the line. We've just used `x` and `y`. Notice that we set the colour in `mark_line()`; if we'd wanted to set the colour to encode some variable, we would have set it in `encode()`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Simple pan-and-zoom interactivity can be added with `.interactive()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chart = alt.Chart(df)\n",
    "\n",
    "chart.mark_line(color='blue').encode(x='Time', y='Projected').interactive()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we wanted to produce the chart we produced earlier, with scatter points, we can use `alt.layer()`, which takes the components as arguments, or just the `+` operator: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-01-15T00:55:27.518001Z",
     "start_time": "2018-01-15T00:55:27.491207Z"
    },
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "df = pd.DataFrame({\n",
    "    'Time': [1,2,3,4,5],\n",
    "    'Projected': [2,5,10,17,26],\n",
    "    'Actual': [1,4,9,11,9],\n",
    "})\n",
    "\n",
    "chart = alt.Chart(df)\n",
    "line = chart.mark_line(color='blue').encode(x='Time', y='Projected')\n",
    "points = chart.mark_point(color='red').encode(x='Time',y='Actual')\n",
    "\n",
    "# or alt.layer(line, points).interactive()\n",
    "(line + points).interactive()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise: \n",
    "\n",
    "Recreate, in Altair, the scatter plot of house sales with x-coordinates given by `longitude` and y-coordinates given by `latitude`. You can use `.mark_point()` or `.mark_circle()`. One issue you will discover, if you use the `x` and `y` encodings, is that Altair and Vega-lite include the axis zero by default. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data types "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Altair recognises four fundamental [data types](https://altair-viz.github.io/user_guide/encoding.html#encoding-data-types):\n",
    "\n",
    "| Data Type | Shorthand Code | Description |\n",
    "|-|-|-|\n",
    "| quantitative | Q | a continuous real-valued quantity |\n",
    "| ordinal | O | a discrete ordered quantity |\n",
    "| nominal | N | a discrete unordered category |\n",
    "| temporal | T | a time or date value |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "alt.Chart(sales.sample(10))\\\n",
    "    .mark_bar()\\\n",
    "    .encode(x='date:N', y='price:Q')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# What kind of sequencing experiments have been done on the LTEE data?\n",
    "\n",
    "chart = alt.Chart(data=ltee)\n",
    "\n",
    "chart.mark_point()\\\n",
    "    .encode(x=alt.X('Sequencing Depth', \n",
    "                    scale=alt.Scale(type='log')),\n",
    "            y='Read Type',\n",
    "            color='Read Length:N')  # try N, O, Q"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise:\n",
    "\n",
    "* Colour your \"geographical\" scatter plot of house sales according to whether the property is or is not a waterfront property. What is the appropriate data type?\n",
    "* Colour according to price (if you prefer, try a log scale). What is the appropriate data type?\n",
    "* Colour according to view. What is the appropriate data type?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Marks and encodings "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Altair has various marks used to draw different kinds of plots. For instance:\n",
    "\n",
    "* `mark_point()` : points on a scatter plot\n",
    "* `mark_bar()` : rectangular bars, in e.g. a bar plot or histogram\n",
    "* `mark_area()` : filled/shaded areas\n",
    "\n",
    "The full list of marks can be found at [https://altair-viz.github.io/user_guide/marks.html](https://altair-viz.github.io/user_guide/marks.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can assign variables of the DataFrame to different encoding channels. For instance:\n",
    "\n",
    "* `x` and `y` : position of mark (scatterpoint position, top of bar-chart bar, etc)\n",
    "* `color` : colour of the mark (colour of point, bar, shaded area etc)\n",
    "* `size` : size of the mark (point size, bar width, etc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can find a list of encodings at [https://altair-viz.github.io/user_guide/encoding.html](https://altair-viz.github.io/user_guide/encoding.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Demo: marks from common base\n",
    "base = alt.Chart(sales.sample(10))\\\n",
    "    .encode(x='date:T', y='price:Q') \n",
    "\n",
    "base.mark_point(color='red') | base.mark_bar()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Demo: box plot\n",
    "base = alt.Chart(ltee)\n",
    "\n",
    "base.mark_boxplot().encode(x='Population', \n",
    "                           y='Total Mutations')\n",
    "# Log scale works here\n",
    "# Color also works here (sort of!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Aggregations "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Many common plot types involve summarising the data:\n",
    "\n",
    "* histograms involve putting some continuous variable into bins (_binning_), and then _counting_ how many samples are in each bin\n",
    "* bar charts or point plots usually involve plotting the _mean_ of some variable, and possibly error bars based on _confidence intervals_\n",
    "* box plots usually involve finding the _mean_, _quartiles_, _min_ and _max_ of some variable (although as of v3 Altair has a built-in boxplot mark type).\n",
    "\n",
    "Combining aggregation functions with different kind of marks is very powerful. \n",
    "\n",
    "Most aggregations have a shorthand string form (e.g. encode `y='count()'`) and a longer form (e.g encode `y=alt.Y(aggregate='count', type='quantitative')`).\n",
    "\n",
    "Binning is done using an explicit axis object like `alt.X()` or `alt.Y()`, with `bin=True` or `bin=alt.Bin()`.\n",
    "\n",
    "You can find a list of aggregation functions at [https://altair-viz.github.io/user_guide/encoding.html#binning-and-aggregation](https://altair-viz.github.io/user_guide/encoding.html#binning-and-aggregation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Demo: Histogram of iris petal lengths"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Demo: LTEE heatmap"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise:\n",
    "1. Using `mark_bar()`, create a histogram of total mutation counts from the `ltee` data.\n",
    "2. Try the same plot but using `.mark_area()`.\n",
    "3. On the original histogram, try setting the `color` encoding to the `Mutator` field. Do you get a stacked or layered histogram by default? (This can be controlled using the `stack` parameter to `alt.Y()`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Interactivity "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hover text can be added to a plot simply by setting the `tooltip` encoding."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise: \n",
    "\n",
    "On your scatter plot of house location, set the hover info to display the sale price."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Basic pan-and-zoom interactivity is set with `.interactive()`. This is actually short for:\n",
    "\n",
    "```\n",
    "chart.add_selection(\n",
    "    alt.selection_interval(bind='scales')\n",
    ")\n",
    "```\n",
    "\n",
    "This means we've added a behaviour where dragging the mouse (selecting an interval) causes the plot to zoom and pan (bind='scales'). We can create custom interactive behaviour with [bindings, selections, and conditions](https://altair-viz.github.io/user_guide/interactions.html).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can define particular kinds of selections with Altair functions:\n",
    "\n",
    "* `selection_interval()` allows us to select everything in a region of the plot by dragging over it\n",
    "* `selection_single()` allows us to select a single element by clicking on it\n",
    "* `selection_multi()` allows us to select multiple elements by holding shift and clicking\n",
    "\n",
    "`add_selection()` is used to the defined selection object to a plot.\n",
    "\n",
    "`alt.condition()` is used to change plot encodings conditional on whether an element is currently selected or not. This is what makes the plots interactive."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise:\n",
    "\n",
    "You're given the below plot, where we've added a selection that changes the colour of the points in `area_price_chart` when selected. Edit the `location_chart` so that when points are selected in `area_price_chart`, they are _also_ highlighted in `location_chart`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-07-16T19:10:31.343996Z",
     "start_time": "2018-07-16T19:10:30.056261Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "selected_houses = alt.selection_interval()\n",
    "\n",
    "area_price_chart = alt.Chart(data=sales)\\\n",
    "    .mark_circle(size=20, opacity=0.5)\\\n",
    "    .encode(\n",
    "        x='sqft_living:Q',\n",
    "        y=alt.Y('price:Q', scale=alt.Scale(type='log')),\n",
    "        color=alt.condition(selected_houses,\n",
    "                            alt.value('red'),\n",
    "                            alt.value('lightgrey')))\\\n",
    "    .add_selection(selected_houses)\n",
    "\n",
    "location_chart = alt.Chart(data=sales)\\\n",
    "    .mark_circle(size=20)\\\n",
    "    .encode(\n",
    "        alt.X('long', scale=alt.Scale(zero=False)),\n",
    "        alt.Y('lat', scale=alt.Scale(zero=False))\n",
    "        )\n",
    "\n",
    "area_price_chart | location_chart"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {
    "height": "138px",
    "width": "252px"
   },
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}