s1_probability.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Introduction to Machine learning and Probability\n",
    "\n",
    "## Machine Learning and Adaptive Intelligence\n",
    "\n",
    "### Mauricio Álvarez \n",
    "\n",
    "### Based on slides by Neil D. Lawrence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Course Text\n",
    "\n",
    "<img src=\"diagrams/a-first-course-in-machine-learning-2ed.jpg\" width=\"300\" height=\"40\" align=center>\n",
    "\n",
    "Simon Rogers and Mark Girolami, *A First Course in Machine Learning*, Chapman and Hall/CRC Press, 2nd Edition, 2016.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Course Text\n",
    "\n",
    "<img src=\"diagrams/978-0-387-31073-2.png\" width=\"340\" height=\"40\" align=center>\n",
    "\n",
    "Christopher Bishop, *Pattern Recognition and Machine Learning*, Springer-Verlag, 2006."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Course Text\n",
    "\n",
    "<img src=\"diagrams/ml-prob-per.jpg\" width=\"340\" height=\"40\" align=center>\n",
    "\n",
    "Kevin Murphy, *Machine Learning: A Probabilistic Perspective*, MIT Press, 2012."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Course Text\n",
    "\n",
    "<img src=\"diagrams/hands-on-scikit-tflow.jpg\" width=\"340\" height=\"40\" align=center>\n",
    "\n",
    "Aurélien Géron, *Hands-On Machine Learning with Scikit-Learn and TensorFlow*, O′Reilly, 2017."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Machine Learning or Statistical Learning\n",
    "\n",
    "- We would like to come up with a **model** that help us to solve a **prediction** problem.\n",
    "\n",
    "\n",
    "- The model is built using a **dataset**.\n",
    "\n",
    "\n",
    "- The ultimate goal is to extract knowlegde from data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Handwritten digit recognition\n",
    "\n",
    "<img src=\"diagrams/mnist_sample.jpeg\" width=\"400\" height=\"40\" align=center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Face detection and face recognition\n",
    "\n",
    "<img src=\"diagrams/face_detection_recognition.png\" width=\"900\" height=\"40\" align=center>\n",
    "\n",
    "Taken from Murphy (2012)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Predicting the age of a person \n",
    "\n",
    "Predicting the age of a person looking at a particular YouTube video\n",
    "\n",
    "<img src=\"diagrams/youtube.png\" width=\"300\" height=\"40\" align=center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Stock market\n",
    "\n",
    "<img src=\"diagrams/stockMarket.jpg\" width=\"400\" height=\"40\" align=center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Clustering\n",
    "\n",
    "<img src=\"diagrams/clustering.png\" width=\"900\" height=\"40\" align=center>\n",
    "\n",
    "Taken from Murphy (2012)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Autoclass\n",
    "\n",
    "<img src=\"diagrams/outerspace.jpg\" width=\"300\" height=\"40\" align=center>\n",
    "<img src=\"diagrams/autoclass.gif\" width=\"400\" height=\"50\" align=center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Recommendation systems\n",
    "\n",
    "<img src=\"diagrams/rec_sys.png\" width=\"1000\" height=\"40\" align=center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Basic definitions\n",
    "\n",
    "- Handwritten digit recognition\n",
    "\n",
    "<img src=\"diagrams/mnist_sample.jpeg\" width=\"250\" height=\"40\" align=center>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Variability."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Each image can be transformed into a vector $\\mathbf{x}$ (feature extraction)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- An instance is made of a the pair $(\\mathbf{x}, y)$, where $y$ is the label of the image."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Objective: find a function $f(\\mathbf{x}, \\mathbf{w})$ that allows predictions. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Basic definitions\n",
    "\n",
    "- **Training set**: a set of $N$ images and their labels, $(\\mathbf{x}_1, y_1), \\ldots, (\\mathbf{x}_N, y_N)$, to fit the predictive model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- **Estimation or training phase**: process of getting the values of $\\mathbf{w}$ of the function $f(\\mathbf{x}, \\mathbf{w})$ that best fit the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- **Generalisation**: ability to correctly predict the label of new images $\\mathbf{x}_*$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Supervised and unsupervised learning\n",
    "\n",
    "- Supervised learning:\n",
    "    - Variable $y$ is discrete: *classification*.\n",
    "    - Variable $y$ is continuous: *regression*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Unsupervised learning: from the set of instances $(\\mathbf{x}_1, y_1), \\ldots, (\\mathbf{x}_N, y_N)$ we only have       access to $\\mathbf{x}_1,\\ldots, \\mathbf{x}_N$.\n",
    "\n",
    "    - Find similar groups: *clustering*.\n",
    "    - Find a probability function for $\\mathbf{x}$: *density estimation*.\n",
    "    - Find a lower dimensionality representation for $\\mathbf{x}$: *dimensionality reduction and feature      \n",
    "      selection*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Other types of learning: semi-supervised learning, active learning, multi-task learning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Example: Olympic 100m Data\n",
    "\n",
    "-  Gold medal times for Olympic 100 m runners since 1896.\n",
    "\n",
    "<img src=\"diagrams/100m_final_start.jpg\" width=\"300\" height=\"40\" align=center>\n",
    "\n",
    "Image from Wikimedia Commons <http://bit.ly/191adDC>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Example: Olympic 100m Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<img src=\"diagrams/male100.jpeg\" width=\"500\" height=\"40\" align=center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Model\n",
    "\n",
    "- We will use a linear model $y = f(x)$, where $y$ is the time in seconds and $x$ the year of the competition."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- The linear model is given as\n",
    "\n",
    "$$ \n",
    "y = w_1 x + w_0, \n",
    "$$\n",
    "\n",
    "  where $w_0$ is the intercept and $w_1$ is the slope."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Objective function\n",
    "\n",
    "- We use an objetive function to estimate the parameters $w_0$ and $w_1$ that best fit the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- In this example, we use a a least squares objective function\n",
    "\n",
    "$$\n",
    "E(w_0, w_1) = \\sum_{\\forall \\; i} (y_i - f(x_i))^2 = \\sum_{\\forall \\; i} [y_i - (w_1 x_i + w_0)]^2.\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Minimising the error we get the solution as $w_0=3.64$ and $w_1 = -1.34$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Data and model\n",
    "\n",
    "<img src=\"diagrams/male100pluspred.jpeg\" width=\"500\" height=\"40\" align=center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Predictions\n",
    "\n",
    "- We can now use this model for making predictions.\n",
    "\n",
    "\n",
    "- For example, what does the model predict for $2012$?\n",
    "\n",
    "\n",
    "- If we say $x=2012$, then \n",
    "\n",
    "$$\n",
    "y = f(x) = f(2012) = w_1 x + w_0 = -1.34 \\times 2012 + 3.64 = 9.59.\n",
    "$$\n",
    "\n",
    "- The actual value was $9.63$.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Probability Review\n",
    "\n",
    "-   We are interested in trials which result in two random variables,\n",
    "    $X$ and $Y$, each of which has an ‘outcome’denoted by $x$ or $y$.\n",
    "\n",
    "-   We summarise the notation and terminology for these distributions in\n",
    "    the following table.\n",
    "       \n",
    "Terminology | Mathematical notation | Description\n",
    "------|:-------------:|:-------------:|\n",
    "joint | $P(X=x, Y=y)$ | probability that X=x *and* Y=y\n",
    "marginal | $P(X=x)$ | probability  that X=x *regardless of* Y\n",
    "conditional | $P(X=x$ &vert; $Y=y)$ | probability that X=x *given that* Y=y\n",
    "\n",
    "<center>The different basic probability\n",
    "  distributions.</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# A Pictorial Definition of Probability\n",
    "\n",
    "<img src=\"diagrams/prob_diagram.svg\" width=\"580\" height=\"40\" align=center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Different Distributions\n",
    "\n",
    "- Definition of probability distributions.\n",
    "\n",
    "Terminology      |  Definition                                              |         Probability Notation\n",
    ":-----------------:|:----------------------------------------------------------:|:------------------------------:                                                                                                                               \n",
    "  Joint Probability      | $\\lim_{N\\rightarrow\\infty}\\frac{n_{X=3,Y=4}}{N}$ | $P\\left(X=3,Y=4\\right)$\n",
    "  Marginal Probability |  $\\lim_{N\\rightarrow\\infty}\\frac{n_{X=5}}{N}$    | $P\\left(X=5\\right)$\n",
    " Conditional Probability | $\\lim_{N\\rightarrow\\infty}\\frac{n_{X=3,Y=4}}{n_{Y=4}}$ |  $P\\left(X=3\\right.$ &vert; $\\left. Y=4\\right)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Notational Details\n",
    "\n",
    "-   Typically we should write out\n",
    "    $P\\left(X=x,Y=y\\right)$.\n",
    "\n",
    "-   In practice, we often use $P\\left(x,y\\right)$.\n",
    "\n",
    "-   This looks very much like we might write a multivariate function,\n",
    "    *e.g.*\n",
    "    $f\\left(x,y\\right)=\\frac{x}{y}$.\n",
    "\n",
    "    -   For a multivariate function though,\n",
    "        $f\\left(x,y\\right)\\neq f\\left(y,x\\right)$.\n",
    "\n",
    "    -   However\n",
    "        $P\\left(x,y\\right)=P\\left(y,x\\right)$\n",
    "        because\n",
    "        $P\\left(X=x,Y=y\\right)=P\\left(Y=y,X=x\\right)$.\n",
    "\n",
    "-   We now quickly review the ‘rules of probability’."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Normalization\n",
    "\n",
    "*All* distributions are normalized. This is clear from the fact that\n",
    "$\\sum_{x}n_{x}=N$, which gives\n",
    "\n",
    "$$\\sum_{x}P\\left(x\\right)={\\lim_{N\\rightarrow\\infty}}\\frac{\\sum_{x}n_{x}}{N}={\\lim_{N\\rightarrow\\infty}}\\frac{N}{N}=1.$$\n",
    "\n",
    "A similar result can be derived for the marginal and conditional\n",
    "distributions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# The Sum Rule\n",
    "\n",
    "Ignoring the limit in our definitions:\n",
    "\n",
    "-   The marginal probability $P\\left(y\\right)$ is\n",
    "    ${\\lim_{N\\rightarrow\\infty}}\\frac{n_{y}}{N}$ .\n",
    "\n",
    "\n",
    "-   The joint distribution $P\\left(x,y\\right)$ is\n",
    "    ${\\lim_{N\\rightarrow\\infty}}\\frac{n_{x,y}}{N}$.\n",
    "\n",
    "\n",
    "-   $n_{y}=\\sum_{x}n_{x,y}$ so\n",
    "\n",
    "    $${\\lim_{N\\rightarrow\\infty}}\\frac{n_{y}}{N}={\\lim_{N\\rightarrow\\infty}}\\sum_{x}\\frac{n_{x,y}}{N},$$\n",
    "\n",
    "   in other words\n",
    "   \n",
    "   $$P\\left(y\\right)=\\sum_{x}P\\left(x,y\\right).$$\n",
    "   \n",
    "   This is known as the sum rule of probability."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# The Product Rule\n",
    "\n",
    "-   $P\\left(x|y\\right)$ is\n",
    "    $${\\lim_{N\\rightarrow\\infty}}\\frac{n_{x,y}}{n_{y}}.$$\n",
    "\n",
    "-   $P\\left(x,y\\right)$ is\n",
    "    $${\\lim_{N\\rightarrow\\infty}}\\frac{n_{x,y}}{N}={\\lim_{N\\rightarrow\\infty}}\\frac{n_{x,y}}{n_{y}}\\frac{n_{y}}{N}$$\n",
    "    \n",
    "    or in other words\n",
    "  \n",
    "    $$P\\left(x,y\\right)=P\\left(x|y\\right)P\\left(y\\right).$$\n",
    "    \n",
    "    This is known as the product rule of probability."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Bayes’ Rule\n",
    "\n",
    "From the product rule,\n",
    "    $$P\\left(y,x\\right)=P\\left(x,y\\right)=P\\left(x|y\\right)P\\left(y\\right),$$\n",
    "    \n",
    "so\n",
    "    $$P\\left(y|x\\right)P\\left(x\\right)=P\\left(x|y\\right)P\\left(y\\right)$$\n",
    "    \n",
    "which leads to Bayes’ rule,\n",
    "    $$\n",
    "P\\left(y|x\\right)=\\frac{P\\left(x|y\\right)P\\left(y\\right)}{P\\left(x\\right)}.\n",
    "    $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Bayes’ Theorem Example\n",
    "\n",
    "\n",
    "There are two barrels in front of you. Barrel One contains 20 apples\n",
    "    and 4 oranges. Barrel Two other contains 4 apples and 8 oranges. You\n",
    "    choose a barrel randomly and select a fruit. It is an apple. What is\n",
    "    the probability that the barrel was Barrel One?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Bayes’ Theorem Example: Answer I\n",
    "\n",
    " We are given that: \n",
    "         \\begin{aligned}\n",
    "          P(\\text{F}=\\text{A}|\\text{B}=1) = & 20/24 \\\\\n",
    "          P(\\text{F}=\\text{A}|\\text{B}=2) = & 4/12 \\\\\n",
    "          P(\\text{B}=1) = & 0.5 \\\\\n",
    "          P(\\text{B}=2) = & 0.5\n",
    "        \\end{aligned}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Bayes’ Theorem Example: Answer II\n",
    "\n",
    "-   We use the sum rule to compute: \n",
    "\n",
    "\\begin{aligned}\n",
    "P(\\text{F}=\\text{A}) = & P(\\text{F}=\\text{A}|\\text{B}=1)P(\\text{B}=1) \\\\\n",
    "                       & + P(\\text{F}=\\text{A}|\\text{B}=2)P(\\text{B}=2)\\\\\n",
    "                     = & 20/24\\times 0.5 + 4/12 \\times 0.5 = 7/12\n",
    "\\end{aligned}\n",
    "\n",
    "-   And Bayes’ theorem tells us that: \n",
    "\n",
    "\\begin{aligned}\n",
    "P(\\text{B}=1|\\text{F}=\\text{A}) = & \\frac{P(\\text{F} = \\text{A}|\\text{B}=1)P(\\text{B}=1)}{P(\\text{F}=\\text{A})}\\\\ \n",
    "                                = & \\frac{20/24 \\times 0.5}{7/12} = 5/7\n",
    "\\end{aligned}\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Reading & Exercises\n",
    "\n",
    "Before next week, review the example on Bayes Theorem\n",
    "-   Read and *understand* Bishop on probability distributions: page\n",
    "    12–17 (Section 1.2).\n",
    "-   Complete Exercise 1.3 in Bishop."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Expectation Computation Example\n",
    "\n",
    "-   Consider the following distribution.\n",
    "\n",
    "$y$        |  1  |  2  |  3  |  4\n",
    "-----------------------------|-----|-----|-----|-----\n",
    "$P\\left(y\\right)$ |  0.3|  0.2|  0.1|  0.4\n",
    "\n",
    "\n",
    "-   What is the mean of the distribution?\n",
    "\n",
    "\n",
    "-   What is the standard deviation of the distribution?\n",
    "\n",
    "\n",
    "-   Are the mean and standard deviation representative of the\n",
    "    distribution form?\n",
    "\n",
    "\n",
    "-   What is the expected value of $-\\log P(y)$?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Expectations Example: Answer\n",
    "\n",
    "-   We are given that:\n",
    "\n",
    "$y$        |   1   |   2   |   3   |   4\n",
    "---------------------|-------|-------|-------|-------\n",
    "$P\\left(y\\right)$ |  0.3  |  0.2  |  0.1  |  0.4\n",
    "$y^2$       |   1   |   4   |   9   |  16\n",
    "  $-\\log(P(y))$   | 1.204 | 1.609 | 2.302 | 0.916\n",
    "\n",
    "\n",
    "-   Mean:\n",
    "    $1\\times 0.3 + 2\\times 0.2 + 3 \\times 0.1 + 4 \\times 0.4 = 2.6$\n",
    "\n",
    "\n",
    "-   Second moment:\n",
    "    $1 \\times 0.3 + 4 \\times 0.2 + 9 \\times 0.1 + 16 \\times 0.4 = 8.4$\n",
    "\n",
    "\n",
    "-   Variance: $8.4 - 2.6\\times 2.6 = 1.64$\n",
    "\n",
    "\n",
    "-   Standard deviation: $\\sqrt{1.64} = 1.2806$\n",
    "\n",
    "\n",
    "-   Expectation $-\\log(P(y))$:\n",
    "    $0.3\\times 1.204 + 0.2\\times 1.609 + 0.1\\times 2.302 +0.4\\times 0.916 = 1.280$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Sample Based Approximation Example\n",
    "\n",
    "-   You are given the following values samples of heights of students,\n",
    "\n",
    "$i$       |   1  |    2 |  3   |   4  |   5  |    6\n",
    "-----------------|------|------|------|------|------|------\n",
    "$y_i$ |  1.76|  1.73| 1.79 | 1.81 | 1.85 |  1.80\n",
    "\n",
    "\n",
    "-   What is the sample mean?\n",
    "\n",
    "-   What is the sample variance?\n",
    "\n",
    "-   Can you compute sample approximation expected value of\n",
    "    $-\\log P(y)$?\n",
    "\n",
    "-   Actually these “data” were sampled from a Gaussian with mean 1.7 and\n",
    "    standard deviation 0.15. Are your estimates close to the real\n",
    "    values? If not why not?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Sample Based Approximation Example: Answer\n",
    "\n",
    "-   We can compute:\n",
    "\n",
    "$i$        |    1    |    2    |    3    |    4    |    5    |    6\n",
    "-------------------|--------|--------|--------|--------|--------|--------\n",
    "$y_i$  |   1.76  |   1.73  |   1.79  |   1.81  |   1.85  |   1.80\n",
    "$y^2_i$ |  3.0976 |  2.9929 |  3.2041 |  3.2761 |  3.4225 |  3.2400\n",
    "\n",
    "\n",
    "-   Mean: $\\frac{1.76 + 1.73 + 1.79 + 1.81 + 1.85 + 1.80}{6} = 1.79$\n",
    "\n",
    "\n",
    "-   Second moment:\n",
    "    $ \\frac{3.0976 + 2.9929 + 3.2041 + 3.2761 + 3.4225 + 3.2400}{6} = 3.2055$\n",
    "\n",
    "\n",
    "-   Variance: $3.2055 - 1.79\\times1.79 = 1.43\\times 10^{-3}$\n",
    "\n",
    "\n",
    "-   Standard deviation: $0.0379$\n",
    "\n",
    "\n",
    "-   No, you can’t compute it. You don’t have access to\n",
    "    $P(y)$ directly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Reading\n",
    "\n",
    "-   See probability review at end of slides for reminders.\n",
    "\n",
    "\n",
    "-   Read and *understand* Rogers and Girolami (2016) on:\n",
    "\n",
    "    1.  Section 2.2 (Random Variables and Probability).\n",
    "    2.  Section 2.4 (Continuous Random Variables - Density functions).\n",
    "    3.  Section 2.5.1 (the Uniform density function).\n",
    "    4.  Section 2.5.3 (the Gaussian density function).\n",
    "\n",
    "\n",
    "-   For other material in Bishop (2006) read:\n",
    "\n",
    "    1.  Probability densities: Section 1.2.1 (Pages 17–19).\n",
    "    2.  Expectations and Covariances: Section 1.2.2 (Pages 19–20).\n",
    "    3.  The Gaussian density: Section 1.2.4 (Pages 24–28) (don’t worry\n",
    "        about material on bias).\n",
    "    4.  For material on information theory and KL divergence try Section\n",
    "        1.6 of Bishop (2006) (pg 48 onwards).\n",
    "        \n",
    "\n",
    "-   If you are unfamiliar with probabilities you should complete exercises 1.7, 1.8, 1.9 in Bishop (2006)."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}