solutions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Representing and comparing probabilities with kernels - practical\n",
    "\n",
    "Notebook prepared by [Heiko Strathmann](http://herrstrathmann.de/) with additional code by [Dougal Sutherland](http://www.gatsby.ucl.ac.uk/~dougals/) and Kacper Chwialkowski.\n",
    "Feel free to email any questions you have.\n",
    "\n",
    "In this notebook, we will cover implementation details and applications for three types of kernel hypothesis tests:\n",
    "    \n",
    "* Two-sampling testing using the Maximum Mean Discrepancy (MMD)\n",
    "* Independence testing using the Hilbert-Schmidt Independence Criterion (HSIC)\n",
    "* Goodness-of-fit testing using the Kernel Stein Discrepancy (KSD)\n",
    "\n",
    "On the fly, we will introduce basic concepts of statistical hypothesis testing.\n",
    "\n",
    "The notebook contains a number of `# IMPLEMENT` comments, where you are supposed to insert missing code yourself.\n",
    "We also prepared a solution notebook that contains the code we envisioned.\n",
    "\n",
    "Cells are meant to be executed in order (i.e. cells might depend on previous ones).\n",
    "\n",
    "## Dependencies\n",
    "The notebook was written using Python 3.6, but we made an effort to keep it backwards compatible to older versions down to 2.7 (hopefully).\n",
    "\n",
    "We encourage you to use a `conda` environment so you don't modify your system's Python setup (especially due to the problems that come with install TensorFlow and Shogun).\n",
    "\n",
    "The packages you need are:\n",
    "* A standard scipy setup covers most content: `numpy`, `scipy`, `matplotlib`, and of yource `jupyter`\n",
    "* `tensorflow` for gradient-based optimization of test power on the MNIST-GAN example. Note we recommend to install this using `conda install -c conda-forge tensorflow `\n",
    "* Optional: `scikit-learn` for a cross-validation splitting iterator (can implement yourself or use another lib)\n",
    "* Optional: `seaborn`, `pandas` for convenient visualization in some examples\n",
    "* Optional: `tqdm` for a fancy progress bar\n",
    "* Optional: Shogun if you want to run a faster MMD test in the MNIST-GAN example: `conda install -c conda-forge shogun` (do this first as the package currently uses openblas and not MKL, so some of the other packages need to be re-installed).\n",
    "\n",
    "This is an example how to set this environment up and run the notebook server\n",
    "```\n",
    "conda create --name ds3_kernels  python=3.6\n",
    "source activate ds3_kernels\n",
    "conda install -c conda-forge tensorflow \n",
    "conda install -c conda-forge shogun\n",
    "conda install jupyter matplotlib scikit-learn seaborn pandas tqdm\n",
    "jupyter notebook\n",
    "```\n",
    "\n",
    "See the Anaconda/TensorFlow/Shogun installation pages for more details.\n",
    "\n",
    "\n",
    "## Additional files\n",
    "All files can be found at https://github.com/karlnapf/ds3_kernel_testing.\n",
    "They include\n",
    "\n",
    "* This notebook and the solution notebook (we encourage you to not look at this before the practical has finished).\n",
    "* **Provided Python modules**: We provide a few external modules that need to be in the `PYTHONPATH` of your notebook server (just put the files in the same folder as this notebook and start the server from this folder).\n",
    "To see whether this worked, try ```import mmd```, and if it doesn't work, make sure that the file is present in one of the paths when you execute `import sys; print(sys.path)`.\n",
    "* **Provided data**: Put those into the same folder as this notebook\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "try:\n",
    "    from tqdm import tqdm_notebooks as tqdm\n",
    "except ImportError:\n",
    "    tqdm = lambda x: x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data (don't give it to students)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# mean shifted Gaussians\n",
    "N = 1000\n",
    "np.random.seed(0)\n",
    "X = np.random.randn(N)\n",
    "Y = np.random.randn(N) + 0.2\n",
    "np.savez(\"simple.npz\", X=X.astype(np.float32), Y=Y.astype(np.float32))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# gaussian vs laplace with same mean / variance\n",
    "from scipy.stats import laplace, norm\n",
    "\n",
    "def sample_gaussian_vs_laplace(n=220, mu=0.0, sigma2=1, b=np.sqrt(0.5)):    \n",
    "    # sample from both distributions\n",
    "    X=norm.rvs(size=n)*np.sqrt(sigma2)+mu\n",
    "    Y=laplace.rvs(size=n, loc=mu, scale=b)\n",
    "    \n",
    "    return X,Y\n",
    "\n",
    "np.random.seed(2)\n",
    "N=300\n",
    "X, Y = sample_gaussian_vs_laplace(n=N)\n",
    "np.savez(\"almost_simple.npz\", X=X.astype(np.float32), Y=Y.astype(np.float32))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sample_blobs(n, ratio, rows=5, cols=5, sep=10, rs=None):\n",
    "    # ratio is eigenvalue ratio\n",
    "    correlation = (ratio - 1) / (ratio + 1)\n",
    "\n",
    "    # generate within-blob variation\n",
    "    mu = np.zeros(2)\n",
    "    sigma = np.eye(2)\n",
    "    X = np.random.multivariate_normal(mu, sigma, size=n)\n",
    "\n",
    "    corr_sigma = np.array([[1, correlation], [correlation, 1]])\n",
    "    Y = np.random.multivariate_normal(mu, corr_sigma, size=n)\n",
    "\n",
    "    # assign to blobs\n",
    "    X[:, 0] += np.random.randint(rows, size=n) * sep\n",
    "    X[:, 1] += np.random.randint(cols, size=n) * sep\n",
    "    Y[:, 0] += np.random.randint(rows, size=n) * sep\n",
    "    Y[:, 1] += np.random.randint(cols, size=n) * sep\n",
    "\n",
    "    return X, Y\n",
    "\n",
    "np.random.seed(0)\n",
    "N=500\n",
    "ratio = 0.01\n",
    "rows_cols = 10\n",
    "X, Y = sample_blobs(n=N, rows=rows_cols, cols=rows_cols, ratio=ratio)\n",
    "np.savez(\"blobs.npz\", X=X.astype(np.float32), Y=Y.astype(np.float32))\n",
    "X, Y = sample_blobs(n=N, rows=rows_cols, cols=rows_cols, ratio=ratio)\n",
    "np.savez(\"blobs2.npz\", X=X.astype(np.float32), Y=Y.astype(np.float32))\n",
    "X, Y = sample_blobs(n=N, rows=1, cols=1, ratio=ratio)\n",
    "np.savez(\"blobs_zoomed.npz\", X=X.astype(np.float32), Y=Y.astype(np.float32))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(0)\n",
    "N=400\n",
    "X = np.random.randn(N)\n",
    "Y = np.random.randn(N)\n",
    "np.savez(\"hsic1.npz\", X=X.astype(np.float32), Y=Y.astype(np.float32))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sample_hsic(n, angle, sigma=0.2, offset=1):\n",
    "    n4 = int(n/4)\n",
    "    N = np.random.randn(n4, 2)*sigma\n",
    "    S = np.random.randn(n4, 2)*sigma\n",
    "    E = np.random.randn(n4, 2)*sigma\n",
    "    W = np.random.randn(n4, 2)*sigma\n",
    "    \n",
    "    N[:,1] += offset\n",
    "    S[:,1] -= offset\n",
    "    W[:,0] -= offset\n",
    "    E[:,0] += offset\n",
    "    \n",
    "    R = np.array([[np.cos(angle), -np.sin(angle)],[np.sin(angle), np.cos(angle)]])\n",
    "    A = R.dot(np.vstack((N,S,W,E)).T).T\n",
    "    \n",
    "    return A[:,0], A[:,1]\n",
    "\n",
    "N=200\n",
    "np.random.seed(0)\n",
    "X,Y = sample_hsic(n=N, angle=np.pi/12)\n",
    "np.savez(\"hsic2.npz\", X=X.astype(np.float32), Y=Y.astype(np.float32))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Two-sample testing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us begin with some simple data $X\\sim P_X, Y\\sim P_Y$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAADvZJREFUeJzt3X+snmV9x/H3R2RuURcRDqSWuhpTFnHZijkhXVwWCKJIFot/sEAWbZxJNYEMiEsETIbbQnDZlM1lI6uDWBMUSdBIMrZZWRdnQtUD6RAo1Aax1Da04lSIiUvhuz/O3Xmkp+f53ec8F+9X8uS57+u57vv+nlI+5+r13D9SVUiS2vWKaRcgSZosg16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUuFdOuwCAM844o9avXz/tMiRppjz44IM/rKq5Xv1WRdCvX7+ehYWFaZchSTMlyff76efUjSQ1rmfQJ1mXZGeSPUkeTXJN1/7xJD9Isrt7XbpkmxuS7EvyRJJ3TfIHkCStrJ+pm6PAR6rqoSSvBR5MsqP77Naq+pulnZOcC1wBvBV4A/C1JOdU1QvjLFyS1J+eI/qqOlRVD3XLzwF7gLUrbLIZuKuqfl5V3wP2AeePo1hJ0uAGmqNPsh44D/hm13R1koeT3JHktK5tLfD0ks0OsMwvhiRbkywkWThy5MjAhUuS+tN30Cd5DXAPcG1V/RS4DXgzsBE4BHzyWNdlNj/u6SZVta2q5qtqfm6u59lBkqQh9RX0SU5lMeTvrKovAVTVM1X1QlW9CHyGX0zPHADWLdn8bODg+EqWJA2in7NuAtwO7KmqTy1pX7Ok23uBR7rle4ErkrwqyZuADcC3xleyJGkQ/Zx183bgfcB3kuzu2m4ErkyykcVpmaeADwFU1aNJ7gYeY/GMnas840aSpqdn0FfVN1h+3v2+Fba5Gbh5hLqk/u28ZTrHvfCG6RxXGpBXxkpS4wx6SWqcQS9JjTPoJalxq+I2xdKkPfDks2Pf566je0/42XUXnzP240nDckQvSY0z6CWpcQa9JDXOOXppSJv2bzvxhztPn8xBvUhLQ3BEL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DjvXqmpuXXHiZ/QNIhN+8f/9CipJY7oJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDWuZ9AnWZdkZ5I9SR5Nck3X/vokO5J8t3s/rWtPkk8n2Zfk4SRvm/QPIUk6sX5G9EeBj1TVW4BNwFVJzgWuB+6vqg3A/d06wLuBDd1rK3Db2KuWJPWtZ9BX1aGqeqhbfg7YA6wFNgPbu27bgcu65c3A52rRLuB1SdaMvXJJUl8GmqNPsh44D/gmcFZVHYLFXwbAmV23tcDTSzY70LVJkqag76BP8hrgHuDaqvrpSl2Xaatl9rc1yUKShSNHjvRbhiRpQH0FfZJTWQz5O6vqS13zM8emZLr3w137AWDdks3PBg6+dJ9Vta2q5qtqfm5ubtj6JUk99HPWTYDbgT1V9aklH90LbOmWtwBfWdL+/u7sm03AT45N8UiSTr5+njD1duB9wHeS7O7abgQ+Adyd5IPAfuDy7rP7gEuBfcDPgA+MtWJJ0kB6Bn1VfYPl590BLlqmfwFXjViXJGlMvDJWkhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGtfPowSl/u28pe+um/Y/O8FCJB3jiF6SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqc97qRJuCBJydzH59dR/ce13bdxedM5FhqhyN6SWqcQS9JjTPoJalxPYM+yR1JDid5ZEnbx5P8IMnu7nXpks9uSLIvyRNJ3jWpwiVJ/elnRP9Z4JJl2m+tqo3d6z6AJOcCVwBv7bb5xySnjKtYSdLgegZ9VX0d+FGf+9sM3FVVP6+q7wH7gPNHqE+SNKJR5uivTvJwN7VzWte2Fnh6SZ8DXZskaUqGDfrbgDcDG4FDwCe79izTt5bbQZKtSRaSLBw5cmTIMiRJvQwV9FX1TFW9UFUvAp/hF9MzB4B1S7qeDRw8wT62VdV8Vc3Pzc0NU4YkqQ9DBX2SNUtW3wscOyPnXuCKJK9K8iZgA/Ct0UqUJI2i5y0QknwBuAA4I8kB4CbggiQbWZyWeQr4EEBVPZrkbuAx4ChwVVW9MJnSNW637jj+8vpBbdo/mUv/JQ2vZ9BX1ZXLNN++Qv+bgZtHKUqSND5eGStJjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnA8Hl2bIpv3bjm/cefpkD3rhDZPdvybOEb0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNa5n0Ce5I8nhJI8saXt9kh1Jvtu9n9a1J8mnk+xL8nCSt02yeElSb/2M6D8LXPKStuuB+6tqA3B/tw7wbmBD99oK3DaeMiVJw+oZ9FX1deBHL2neDGzvlrcDly1p/1wt2gW8LsmacRUrSRrcsHP0Z1XVIYDu/cyufS3w9JJ+B7q24yTZmmQhycKRI0eGLEOS1Mu4v4zNMm21XMeq2lZV81U1Pzc3N+YyJEnHDBv0zxybkuneD3ftB4B1S/qdDRwcvjxJ0qiGDfp7gS3d8hbgK0va39+dfbMJ+MmxKR5J0nS8sleHJF8ALgDOSHIAuAn4BHB3kg8C+4HLu+73AZcC+4CfAR+YQM2SpAH0DPqquvIEH120TN8Crhq1KEnS+HhlrCQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWpcz9MrNcN23jJQ9037n51QIZqkB56c7H+3XUf3/tL6dRefM9Hjafwc0UtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnLdAkLSiTfu3/XLDztMnf9ALb5j8MV5GHNFLUuMMeklqnFM3q9itO/b27rQC70YpCRzRS1LzDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxI93ULMlTwHPAC8DRqppP8nrgi8B64CngD6vqf0YrU5I0rHGM6C+sqo1VNd+tXw/cX1UbgPu7dUnSlExi6mYzsL1b3g5cNoFjSJL6NGrQF/DVJA8m2dq1nVVVhwC69zNHPIYkaQSjPnjk7VV1MMmZwI4kj/e7YfeLYSvAG9/4xhHLkCSdyEgj+qo62L0fBr4MnA88k2QNQPd++ATbbquq+aqan5ubG6UMSdIKhg76JK9O8tpjy8A7gUeAe4EtXbctwFdGLVKSNLxRpm7OAr6c5Nh+Pl9V/5bk28DdST4I7AcuH71MSdKwhg76qnoS+J1l2p8FLhqlKEnS+HhlrCQ1zqCXpMYZ9JLUOINekhpn0EtS40a9MlbSy8wDTz478WPsOroXgOsuPmfix3o5cEQvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1znvdnCw7bxl4k037J39PEUntc0QvSY0z6CWpcU7dDOjWHXuH2s5pGKl/m/ZvW1zYefrJPfCFN5zc450kjuglqXEGvSQ1zqCXpMY5Ry9p1ToZjy1c6ncvPKmHO2kc0UtS4wx6SWqcQS9JjTPoJalxM/9l7LAXMEnSy4UjeklqnEEvSY2b+ambQf3/PTQk6WViYkGf5BLg74BTgH+uqk9M6liSNBZDPDdiZCfhRmoTmbpJcgrwD8C7gXOBK5OcO4ljSZJWNqk5+vOBfVX1ZFX9L3AXsHlCx5IkrWBSQb8WeHrJ+oGuTZJ0kk1qjj7LtNUvdUi2Alu71eeTPDGG454B/HAM+5kGa5+eWa5/lmuH2a5/TLXfOMrGv9FPp0kF/QFg3ZL1s4GDSztU1TZgrKfAJFmoqvlx7vNksfbpmeX6Z7l2mO36Z6n2SU3dfBvYkORNSX4FuAK4d0LHkiStYCIj+qo6muRq4N9ZPL3yjqp6dBLHkiStbGLn0VfVfcB9k9r/Cczy1VDWPj2zXP8s1w6zXf/M1J6q6t1LkjSzvNeNJDWuqaBP8pdJHk6yO8lXk7xh2jUNIslfJ3m8+xm+nOR1066pX0kuT/JokheTzMSZCEkuSfJEkn1Jrp92PYNIckeSw0kemXYtg0qyLsnOJHu6vzPXTLumQST51STfSvLfXf1/Pu2aemlq6ibJr1fVT7vlPwHOraoPT7msviV5J/Af3ZfZfwVQVR+dcll9SfIW4EXgn4A/raqFKZe0ou42HXuBi1k8HfjbwJVV9dhUC+tTkt8Hngc+V1W/Ne16BpFkDbCmqh5K8lrgQeCyGfqzD/Dqqno+yanAN4BrqmrXlEs7oaZG9MdCvvNqXnKR1mpXVV+tqqPd6i4Wrz+YCVW1p6rGcdHbyTLTt+moqq8DP5p2HcOoqkNV9VC3/Bywhxm6cr4WPd+tntq9VnXWNBX0AEluTvI08EfAn027nhH8MfCv0y6iYd6mYxVIsh44D/jmdCsZTJJTkuwGDgM7qmpV1z9zQZ/ka0keWea1GaCqPlZV64A7gaunW+3xetXf9fkYcJTFn2HV6Kf2GdLzNh2arCSvAe4Brn3Jv8ZXvap6oao2sviv7vOTrOrps5l78EhVvaPPrp8H/gW4aYLlDKxX/Um2AH8AXFSr7AuUAf7sZ0HP23Rocrq57XuAO6vqS9OuZ1hV9eMk/wlcAqzaL8ZnbkS/kiQblqy+B3h8WrUMo3tYy0eB91TVz6ZdT+O8TceUdF9m3g7sqapPTbueQSWZO3ZGXJJfA97BKs+a1s66uQf4TRbP/vg+8OGq+sF0q+pfkn3Aq4Bnu6Zds3LWUJL3An8PzAE/BnZX1bumW9XKklwK/C2/uE3HzVMuqW9JvgBcwOIdFJ8Bbqqq26daVJ+S/B7wX8B3WPx/FeDG7mr6VS/JbwPbWfx78wrg7qr6i+lWtbKmgl6SdLympm4kSccz6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJatz/AShKy+z42qtCAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data = np.load(\"simple.npz\")\n",
    "X = data[\"X\"]\n",
    "Y = data[\"Y\"]\n",
    "plt.hist(X, alpha=0.5)\n",
    "plt.hist(Y, alpha=0.5);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Question: do X and Y come from the same distribution?\n",
    "Or in other words: can we reject the null hypothesis that $P_X=P_Y$?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The simplest of all test statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by comparing the means of the samples.\n",
    "Clearly, if the distributions are the same, the mean would be zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean differencce: -0.25887364\n"
     ]
    }
   ],
   "source": [
    "def simple_statistic(X,Y, squared=False):\n",
    "    assert X.ndim == Y.ndim == 1\n",
    "    \n",
    "    # IMPLEMENT: compute mean difference of X and Y\n",
    "    result = np.mean(X) - np.mean(Y)\n",
    "    \n",
    "    if squared:\n",
    "        result *= result\n",
    "    return result\n",
    "\n",
    "my_statistic = simple_statistic(X,Y)\n",
    "print(\"Mean differencce:\", my_statistic)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Significant?\n",
    "What would it be if distributions were the same? What if they were different?\n",
    "\n",
    "Let's assume that $P_X=P_Y$ and look at the distribution of the test statistic (null distribution).\n",
    "For that, we can merge the samples and shuffle them, after which they have the same \"joint\" distribution.\n",
    "We can then re-compute the statistic for multiple permutations to get a feeling for how the statistic is distributed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def two_sample_permutation_test(test_statistic, X, Y, num_permutations, prog_bar=True):\n",
    "    assert X.ndim == Y.ndim\n",
    "    \n",
    "    statistics = np.zeros(num_permutations)\n",
    "    \n",
    "    range_ = range(num_permutations)\n",
    "    if prog_bar:\n",
    "        range_ = tqdm(range_)\n",
    "    for i in range_:\n",
    "        # concatenate samples\n",
    "        if X.ndim == 1:\n",
    "            Z = np.hstack((X,Y))\n",
    "        elif X.ndim == 2:\n",
    "            Z = np.vstack((X,Y))\n",
    "            \n",
    "        # IMPLEMENT: permute samples and compute test statistic\n",
    "        perm_inds = np.random.permutation(len(Z))\n",
    "        Z = Z[perm_inds]\n",
    "        X_ = Z[:len(X)]\n",
    "        Y_ = Z[len(X):]\n",
    "        my_test_statistic = test_statistic(X_, Y_)\n",
    "        statistics[i] = my_test_statistic\n",
    "    return statistics\n",
    "\n",
    "num_permutations = 200\n",
    "statistics = two_sample_permutation_test(simple_statistic, X, Y, num_permutations)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have an idea about what to expect from the statistic in case where the distributions are equal.\n",
    "How likely is our particular value of that statistic?\n",
    "We can compare it against the say 95% quantiles of the null distribution.\n",
    "If our statistic lies outside those, we can reject the null with a confidence of 0.05."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAEKCAYAAADn+anLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xt0VPX57/H3w82AIiaAHpR6AqegSIRBg0WwRxAVFKq2YNUjipXKqr+f1XoH8VaFLnShIsWWolLB/uRSELSCigjUSkUMiMjNclcEucSCIqJCn/PH3olDmJ1MLpMZwue1Fmtm9v7u7zzZGfKZfftuc3dEREQSqZXuAkREJHMpJEREJJJCQkREIikkREQkkkJCREQiKSRERCSSQkJERCIpJEREJJJCQkREItVJdwHJaNKkiefm5pZ/wY8+Ch5POaVK65GaQx8RqcmfgcWLF+9096aV6eOwCInc3FwKCgrKv2DXrsHj/PlVWY7UIPqISE3+DJjZpsr2od1NIiISSSEhIiKRFBIiIhLpsDgmkch3333H5s2b2bdvX3SjBx4IHletqp6iJC2ysrJo3rw5devWTXcpIjXOYRsSmzdvpmHDhuTm5mJmiRvVCjeUauJpCwKAu1NYWMjmzZtp0aJFussRqXEO291N+/bto3HjxtEBIUcEM6Nx48alb1GKSIUdtiEBKCAE0OdAJJUO65AQEZHUUkhUwpNPPkleXh5t27Zl5MiRxdMffPBBTjrpJGKxGLFYjFmzZgGwYMEC2rVrR8eOHVm7di0Au3btokePHqTzXuMzZsxg5cqVxa/vv/9+5syZA0DXrl0rdiGjiNQIh+2B63Rbvnw5Tz/9NIsWLaJevXr07NmTXr160apVKwBuvfVW7rjjjoOWeeyxx5g2bRobN27kj3/8I4899hgPP/ww99xzT1p3mcyYMYPevXtz2mmnAfDQQw+lrRZJr9xBM6u0v43De1Vpf1L9tCVRQatWraJTp040aNCAOnXqcO655zJ9+vRSl6lbty5ff/01e/fupW7duqxbt45PP/2Uc889N3KZ1157jVNPPZVzzjmHm2++md69ewPB1sqIESOK2+Xl5bFx40YALrvsMs4880zatm3L2LFji9scc8wxDBkyhPbt29OpUye2bdvGP//5T15++WXuvPNOYrEY69at47rrrmPq1KmH1DJ79mzOPvtszjjjDC6//HL27NkDwKBBgzjttNNo167dIcEoIoe3GrEl8ZvfwNKlCWbs/UHw2KD8fcZiELcH6RB5eXkMGTKEwsJC6tevz6xZs8jPzy+eP3r0aCZMmEB+fj6PPfYY2dnZDB48mIEDB1K/fn2ef/557rjjDh5++OHI99i3bx833HADc+fO5Yc//CFXXHFFUrWPGzeOnJwcvv76azp27EifPn1o3LgxX331FZ06dWLYsGHcddddPP3009x7771ccskl9O7dm759+0b2uXPnToYOHcqcOXM4+uijeeSRR3j88ce56aabmD59OqtXr8bM2LVrV1I1isjhQVsSFdSmTRvuvvtuLrjgAnr27En79u2pUyfI3BtvvJF169axdOlSmjVrxu233w5ALBZj4cKFzJs3j/Xr13PiiSfi7lxxxRX069ePbdu2HfQeq1evpkWLFrRq1Qozo1+/fknVNmrUqOKthU8++YQ1a9YAUK9eveItkTPPPLN4yyMZCxcuZOXKlXTp0oVYLMb48ePZtGkTxx57LFlZWfzyl7/kxRdfpEGDCiSyiGSsGrElEfmN/6NPgscUXUw3YMAABgwYAMA999xD8+bNATjhhBOK29xwww3Ff5iLuDtDhw5l8uTJ3HTTTfz2t79l48aNjBo1imHDhh3UNupYRZ06dfjPf/5T/LroOoH58+czZ84c3nnnHRo0aEDXrl2L59WtW7e4v9q1a7N///6kf1Z354ILLmDixImHzFu0aBFvvvkmkyZNYvTo0cydOzfpfkUks2lLohK2b98OwMcff8yLL77IVVddBcDWrVuL20yfPp28vLyDlhs/fjy9evUiOzubvXv3UqtWLWrVqsXevXsPanfqqaeyYcMG1q1bB3DQH+jc3FyWLFkCwJIlS9iwYQMAu3fvJjs7mwYNGrB69WoWLlxY5s/RsGFDvvzyy1LbdOrUiQULFhSflbV3717+9a9/sWfPHnbv3s3FF1/MyJEjWZpwv5+IHK5qxJZEuvTp04fCwkLq1q3LU089RXZ2NgB33XUXS5cuxczIzc3lT3/6U/Eye/fuZfz48cyePRuA2267jT59+lCvXr1DvqVnZWUxduxYevXqRZMmTTjnnHNYvnx58XtPmDCBWCxGx44dad26NQA9e/ZkzJgxtGvXjlNOOYVOnTqV+XNceeWV3HDDDYwaNSrhAWuApk2b8txzz3HVVVfxzTffADB06FAaNmzIpZdeyr59+3B3nnjiiXKuRRHJZJbO8/OTlZ+f7yXP1V+1ahVt2rQpfcEadsup+fPnM2LECF555ZV0l5Jxkvo8JFCTbzhTEUfiKbA1+TNgZovdPb/sltG0u0lERCJpd9NhpGvXrnQt+tojIlINtCUhIiKRFBIiIhJJISEiIpEUEiIiEkkhUUnTp0/HzFi9enWZbZ977jm2bNlS4feaP3/+IVdvAyxdurR4OPLy2rVrF3/4wx/KtczIkSMPufAvmXYXX3xxqWM7lbe9iKSeQqKSJk6cyDnnnMOkSZPKbFvZkIhyuITErFmzOO6446qsvYiknkKiEvbs2cOCBQt49tlnDwmJRx99lNNPP5327dszaNAgpk6dSkFBAVdffTWxWIyvv/6a3Nxcdu7cCUBBQUHx6a2LFi2ic+fOdOjQgc6dO/NR0UWBCXz77bfcf//9TJ48mVgsxuTJk/nqq6+4/vrr6dixIx06dOCll14CYMWKFZx11lnEYjHatWvHmjVrGDRoEOvWrSMWi3HnnXce1PdXX31Fr169aN++PXl5eUyePJlRo0axZcsWunXrRrdu3YBgQMP8/Hzatm3LAw88AJCwXdHPm2y/8etnwoQJtGvXjvbt23PNNddU5tcmIuVQM66TiBorvOhbaUVGJi1rrHCCm/X07NmT1q1bk5OTw5IlSzjjjDN49dVXmTFjBu+++y4NGjTg888/Jycnh9GjRzNixIiDhhRP5NRTT+Wtt96iTp06zJkzh3vuuYdp06YlbFuvXj0eeughCgoKGD16NBAMNnjeeecxbtw4du3axVlnncX555/PmDFjuOWWW7j66qv59ttvOXDgAMOHD2f58uUJx1x67bXXOPHEE5k5M7gKd/fu3TRq1IjHH3+cefPm0aRJEwCGDRtGTk4OBw4coHv37ixbtoybb775kHbl7bfIihUrGDZsGAsWLKBJkyZ8/vnnpa4/Eak6Kd2SMLONZvahmS01s4JwWo6ZvWFma8LH7FTWkEoTJ07kyiuvBILxj4rGXpozZw6/+MUviofNzsnJKVe/u3fv5vLLLycvL49bb72VFStWlGv52bNnM3z4cGKxWPEosB9//DFnn302v/vd73jkkUfYtGkT9evXL7Wf008/nTlz5nD33Xfzj3/8g0aNGiVsN2XKFM444ww6dOjAihUrDroVamX6LTJ37lz69u1bHB7lXZ8iUnHVsSXRzd13xr0eBLzp7sPNbFD4+u5KvUPUN/4Ujt1UWFjI3LlzWb58OWbGgQMHMDMeffRR3D2p25HGD/ddNJw3wH333Ue3bt2YPn06GzduLPdV1u7OtGnTOKXEz92mTRt+9KMfMXPmTHr06MEzzzxDy5YtI/tp3bo1ixcvZtasWQwePJgLL7yQ+++//6A2GzZsYMSIEbz33ntkZ2dz3XXXHfSzVLTfkj9POm/vKnIkS8cxiUuB8eHz8cBlaaih0qZOncq1117Lpk2b2LhxI5988gktWrTg7bff5sILL2TcuHHFB2GLdo+UHJI7NzeXxYsXAxy0O2n37t2cdNJJQHCwuywl++3Rowe///3vKRq88f333wdg/fr1tGzZkptvvplLLrmEZcuWlTpM+JYtW2jQoAH9+vXjjjvuKB6aPH6ZL774gqOPPppGjRqxbds2Xn311ci6ytNvvO7duzNlyhQKCwsBtLtJpBqlOiQcmG1mi81sYDjtBHffChA+Hp/iGlJi4sSJ/PSnPz1oWp8+fXjhhRfo2bMnl1xyCfn5+cRiseJ7UV933XX86le/Kj5w/cADD3DLLbfw4x//mNq1axf3c9dddzF48GC6dOnCgQMHyqylW7durFy5svjA9X333cd3331Hu3btyMvL47777gNg8uTJ5OXlEYvFWL16Nddeey2NGzemS5cu5OXlHXLg+sMPPyw+0D1s2DDuvfdeAAYOHMhFF11Et27daN++PR06dKBt27Zcf/31dOnSpXj5+Hbl7Tde27ZtGTJkCOeeey7t27fntttuK3OdiEjVSOlQ4WZ2ortvMbPjgTeAXwMvu/txcW3+7e6HHJcIQ2UgwMknn3zmpk2bDpp/JA4VLtE0VHjV0FDhNUvGDxXu7lvCx+3AdOAsYJuZNQMIH7dHLDvW3fPdPb9p06apLFNERCKkLCTM7Ggza1j0HLgQWA68DPQPm/UHXkpVDSIiUjmpPLvpBGB6eFZKHeAFd3/NzN4DppjZAOBj4PKKvoHOehGAw+HuiiKHq5SFhLuvB9onmF4IdK9s/1lZWRQWFtK4cWMFxRHM3SksLCQrKyvdpYjUSIftFdfNmzdn8+bN7NixI7rRZ58Fj+G1CFIzZWVl0bx583SXIVIjHbYhUbduXVq0aFF6oxtvDB5r4mkLIiLVQAP8iYhIJIWEiIhEUkiIiEgkhYSIiERSSIiISCSFhIiIRFJIiIhIJIWEiIhEUkiIiEgkhYSIiERSSIiISKTDduwmEan6O8mJlKQtCRERiaSQEBGRSAoJERGJpJAQEZFICgkREYmkkBARkUgKCRERiaSQEBGRSAoJERGJpJAQEZFICgkREYmkkBARkUgKCRERiaSQEBGRSCkPCTOrbWbvm9kr4esWZvauma0xs8lmVi/VNYiISMVUx5bELcCquNePAE+4eyvg38CAaqhBREQqIKUhYWbNgV7AM+FrA84DpoZNxgOXpbIGERGpuFRvSYwE7gL+E75uDOxy9/3h683ASSmuQUREKihlIWFmvYHt7r44fnKCph6x/EAzKzCzgh07dqSkRhERKV0qtyS6AJeY2UZgEsFuppHAcWZWdG/t5sCWRAu7+1h3z3f3/KZNm6awTBERiZKykHD3we7e3N1zgSuBue5+NTAP6Bs26w+8lKoaRESkctJxncTdwG1mtpbgGMWzaahBRESSUKfsJpXn7vOB+eHz9cBZ1fG+IiJSObriWkREIikkREQkkkJCREQiKSRERCSSQkJERCIpJEREJJJCQkREIikkREQkkkJCREQiKSRERCSSQkJERCIpJEREJJJCQkREIikkREQkkkJCREQiKSRERCSSQkJERCIpJEREJJJCQkREIpU7JMws28zapaIYERHJLEmFhJnNN7NjzSwH+AD4s5k9ntrSREQk3ZLdkmjk7l8APwP+7O5nAuenriwREckEyYZEHTNrBvwceCWF9YiISAZJNiR+C7wOrHX398ysJbAmdWWJiEgmqJNku63uXnyw2t3X65iESPnlDpqZ7hJEyiXZLYnfJzlNRERqkFK3JMzsbKAz0NTMboubdSxQO5WFiYhI+pW1u6kecEzYrmHc9C+AvqkqSkREMkOpIeHufwf+bmbPufum8nRsZlnAW8BR4ftMdfcHzKwFMAnIAZYA17j7txWqXkREUirZA9dHmdlYIDd+GXc/r5RlvgHOc/c9ZlYXeNvMXgVuA55w90lmNgYYAPyxQtWLiEhKJRsSfwXGAM8AB5JZwN0d2BO+rBv+c+A84P+F08cDD6KQEBHJSMmGxH53L/cfcjOrDSwGfgg8BawDdrn7/rDJZuCk8vYrIiLVI9lTYP9mZv9lZs3MLKfoX1kLufsBd48BzYGzgDaJmiVa1swGmlmBmRXs2LEjyTJFRKQqJbsl0T98vDNumgMtk1nY3XeZ2XygE3CcmdUJtyaaA1silhkLjAXIz89PGCQiIpJaSYWEu7cob8dm1hT4LgyI+gQDAj4CzCM4fXYSQfi8VN6+RUSkeiQVEmZ2baLp7j6hlMWaAePD4xK1gCnu/oqZrQQmmdlQ4H3g2XLWLCIi1STZ3U0d455nAd0JrnGIDAl3XwZ0SDB9PcHxCRERyXDJ7m76dfxrM2sEPJ+SikREJGNU9B7Xe4FWVVmIiIhknmSPSfyN709VrU1wKuuUVBUlIjVDVQ+NvnF4ryrtT8qW7DGJEXHP9wOb3H1zCuoREZEMktTupnCgv9UEI8FmAxqQT0TkCJBUSJjZz4FFwOUE97l+18w0VLiISA2X7O6mIUBHd98OxRfKzQGmpqowERFJv2TPbqpVFBChwnIsKyIih6lktyReM7PXgYnh6yuAWakpSUREMkVZ97j+IXCCu99pZj8DzgEMeAf4n2qoT0RE0qisXUYjgS8B3P1Fd7/N3W8l2IoYmeriREQkvcoKidxwDKaDuHsBwa1MRUSkBisrJLJKmVe/KgsREZHMU1ZIvGdmN5ScaGYDCG5LKiIiNVhZZzf9BphuZlfzfSjkA/WAn6ayMBERSb9SQ8LdtwGdzawbkBdOnunuc1NemYiIpF2y95OYR3DbUREROYLoqmkREYmkkBARkUgKCRERiaSQEBGRSAoJERGJpJAQEZFICgkREYmkkBARkUgKCRERiaSQEBGRSAoJERGJlLKQMLMfmNk8M1tlZivM7JZweo6ZvWFma8LH7FTVICIilZPKLYn9wO3u3gboBPy3mZ0GDALedPdWwJvhaxERyUApCwl33+ruS8LnXwKrgJOAS4HxYbPxwGWpqkFERCqnWo5JmFku0AF4FzjB3bdCECTA8dVRg4iIlF/KQ8LMjgGmAb9x9y/KsdxAMysws4IdO3akrkAREYmU0pAws7oEAfE/7v5iOHmbmTUL5zcDtida1t3Hunu+u+c3bdo0lWWKiEiEVJ7dZMCzwCp3fzxu1stA//B5f+ClVNUgIiKVk9TtSyuoC3AN8KGZLQ2n3QMMB6aY2QDgY+DyFNYgIiKVkLKQcPe3AYuY3T1V7ysiIlVHV1yLiEgkhYSIiERSSIiISCSFhIiIRFJIiIhIJIWEiIhEUkiIiEgkhYSIiERSSIiISCSFhIiIRFJIiIhIJIWEiIhEUkiIiEgkhYSIiERSSIiISCSFhIiIRFJIiIhIJIWEiIhEUkiIiEgkhYSIiERSSIiISKQ66S5AJJPlDpqZ7hJE0kpbEiIiEkkhISIikbS7SUQOG1W9+2/j8F5V2l9NpC0JERGJpJAQEZFICgkREYmUspAws3Fmtt3MlsdNyzGzN8xsTfiYnar3FxGRykvllsRzQM8S0wYBb7p7K+DN8LWIiGSolIWEu78FfF5i8qXA+PD5eOCyVL2/iIhUXnUfkzjB3bcChI/HV/P7i4hIOWTsgWszG2hmBWZWsGPHjnSXIyJyRKrukNhmZs0AwsftUQ3dfay757t7ftOmTautQBER+V51h8TLQP/weX/gpWp+fxERKYdUngI7EXgHOMXMNpvZAGA4cIGZrQEuCF+LiEiGStnYTe5+VcSs7ql6TxERqVoZe+BaRETSTyEhIiKRFBIiIhJJISEiIpEUEiIiEkkhISIikRQSIiISSSEhIiKRFBIiIhJJISEiIpEUEiIiEkkhISIikRQSIiISKWWjwIqkQ+6gmeVq/9n6TuFyC1NRjmS43EEzq/QzsHF4r0r3kWm0JSEiIpEUEiIiEkkhISIikRQSIiISSSEhIiKRFBIiIhJJISEiIpEUEiIiEkkhISIikRQSIiISScNySFqVdxgNkUxW1Z/nTBjmQ1sSIiISSSEhIiKRFBIiIhIpLSFhZj3N7CMzW2tmg9JRg4iIlK3aQ8LMagNPARcBpwFXmdlp1V2HiIiULR1bEmcBa919vbt/C0wCLk1DHSIiUoZ0hMRJwCdxrzeH00REJMOk4zoJSzDND2lkNhAYGL7cY2YfVfwdE70lAE2AnRXuN7VUW8VUqLZNj6SgkkPVuPVWTaqltkp8BlJWn1X+c3lKZTtIR0hsBn4Q97o5sKVkI3cfC4xNZSFmVuDu+al8j4pSbRWj2ipGtVVcJtdnZgWV7SMdu5veA1qZWQszqwdcCbychjpERKQM1b4l4e77zewm4HWgNjDO3VdUdx0iIlK2tIzd5O6zgFnpeO8SUro7q5JUW8WotopRbRWXyfVVujZzP+SYsYiICKBhOUREpBQ1PiTMLMfM3jCzNeFjdoI2MTN7x8xWmNkyM7sibt5zZrbBzJaG/2IZVFsLM3s3XH5yeCJAtdUWtnvNzHaZ2Sslpqd1vZVRWyast/5hmzVm1j9u+vxwyJqi9XZ8FdRU6jA4ZnZUuB7WhuslN27e4HD6R2bWo7K1VFVtZpZrZl/Hracxaajt/5rZEjPbb2Z9S8xL+PvNkNoOxK23sk8acvca/Q94FBgUPh8EPJKgTWugVfj8RGArcFz4+jmgb4bWNgW4Mnw+BrixOmsL53UHfgK8UmJ6WtdbGbWldb0BOcD68DE7fJ4dzpsP5FdhPbWBdUBLoB7wAXBaiTb/BYwJn18JTA6fnxa2PwpoEfZTO0NqywWWp+LzVY7acoF2wIT4z3ppv9901xbO21Oe96vxWxIEQ36MD5+PBy4r2cDd/+Xua8LnW4DtQNNMrs3MDDgPmFra8qmsLazpTeDLKnzfZFS4tgxZbz2AN9z9c3f/N/AG0LMKa4iXzDA48TVPBbqH6+lSYJK7f+PuG4C1YX+ZUFuqlVmbu29092XAf0osm+rfb2VqK7cjISROcPetAOFjqZvvZnYWQTqvi5s8LNzV84SZHZUhtTUGdrn7/nB2VQ9vUq7aImTEeishE9ZbWUPT/DncFXBfFfxBTGYYnOI24XrZTbCeUj2ETmVqA2hhZu+b2d/N7MdVWFeytaVi2eroP8vMCsxsoZmV+QWpRty+1MzmAP8rwawh5eynGfA80N/dixJ4MPAZwR/nscDdwEPpri3ij0e5TlWrqtoiZMR6S9R1gmnVvd5Kq+Fqd//UzBoC04BrCHYZVFQyP29Um0qvqzJUpratwMnuXmhmZwIzzKytu39RjbWlYtnq6P9kd99iZi2BuWb2obuvi2pcI0LC3c+Pmmdm28ysmbtvDf/Qbo9odywwE7jX3RfG9b01fPqNmf0ZuCNDatsJHGdmdcJvWAmHN0l1baX0nfb1FiET1ttmoGvc6+YExyJw90/Dxy/N7AWCXQuVCYlkhsEparPZzOoAjYDPk1y2Mipcmwc7178BcPfFZraO4PhdpYehKEdtpS3btcSy86ukqu/7r/DvJdxtjbuvN7P5QAcO3nNykCNhd9PLQNHZBf2Bl0o2sODslunABHf/a4l5zcJHI9i/vDwTagv/k8wD+pa2fCprK02611uUDFlvrwMXmlm2BWc/XQi8bmZ1zKwJgJnVBXpT+fWWzDA48TX3BeaG6+ll4MrwDKMWQCtgUSXrqZLazKypBfemIfxG3IrgAHF11hYl4e83E2oLazoqfN4E6AKsLHWhqjrinqn/CPZfvgmsCR9zwun5wDPh837Ad8DSuH+xcN5c4EOC/6x/AY7JoNpaEvynXQv8FTiqOmsLX/8D2AF8TfANp0cmrLcyasuE9XZ9+P5rgV+E044GFgPLgBXAk1TB2UTAxcC/CL4tDgmnPQRcEj7PCtfD2nC9tIxbdki43EfARVW1nipbG9AnXEcfAEuAn6Shto7h5+oroBBYUdrvNxNqAzqH/y8/CB8HlPVeuuJaREQiHQm7m0REpIIUEiIiEkkhISIikRQSIiISSSEhIiKRFBKSdmbWOG5Uys/M7NO410mP0Gpm15tZoiuho9qfZ2adytvOzP7bzK6uqvaVEV5bsSsVfYtADbniWg5v7l4IxADM7EGCUSpHVKCr6wnOmf8syfbnEVyBvbA87dz9qSpuL5KxtCUhGc2CcfkXhVsVfzCzWuG35+fN7EMzW25mN1twn40YMDnRFoiZ3WpmK83sAzP7i5n9H+CXwJ1h+85mdqkF9yx438xmm9nxEe2GmtlvytFvfPvWZjY3bL/E4u7dEM5/zMwGxr0eama3mNmx4XJLLBg0sXeCdXW+mc2Iez3GzPqFzztaMBDeYjN71cxOqIrfj9R82pKQjGVmecBPgc7uvt/MxhIMQbAOaOLup4ftjnP3XWb2a+Amd1+aoLu7gP/t7t/GtX8G2OnuI8N+soGX3d3N7FfA7e5+d4J2F5ez3/j2E4EH3f1vZpbFoV/UJgHD+f7exJcD3QiuGr/UgzGdjgcWAK+QhHAYhicJrsbdGe76ehgYWPqSIgoJyWznEwwvUGDBoLf1CYZIfh04xcyeBGYBs5PoawXwFzN7CZgR0eZkYEp4XOMogmEPqqJfoDiEmrj73wDcfV/JNu7+npn9IPym3xz4zIMRO+sBj5jZOQT3CPhBOPZOMscj2gBtgTnheqxNMGSDSJkUEpLJDBjn7vcdMsOsHXARcDPBOD5lfSvuAZxLcHOWe8OtlJKeAn7n7rPM7HyCO8uVJZl+4yUzDs40gp8pl2DLAuBaghFQzwi3qjYTjGsUbz8Hb5kUzTdgmbtX9T0X5AigYxKSyeYAP7fvR0ZtbGYnm1lTwDwYFfcB4Iyw/ZdAw5KdWDBaaHN3nwvcSXDXwQYJ2jcCPrXg63b8fYkr2y8AHtylbKeZ/SRcPsvMGiT4uScR7Fb7GUFgFNW2PQyIC0h8k5lNQFszqxdutZwXTl8JnGTBTasI57dNsLzIIRQSkrHc/UPgtwS7SZYR7FY6gWAs/bfMbCnwNHBPuMifgWcSHLiuA7wQ9rGE4L7TXxIM4/3z8EB1Z+BBgmHZ/w5si1u+ZLvy9hvvauD2cJm3SXCbXHf/IJy+wd2L7kfxPNDZzAoIjlOsSbDcBoJdXh8S3INiSTj9G4Jhth83sw+A94EflVxeJBGNAisiIpG0JSEiIpEUEiIiEkkhISIikRQSIiISSSEhIiKRFBIiIhJJISEiIpEUEiIiEun3z24WAAAABklEQVT/A0TKVg2JjJJPAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_permutation_samples(null_samples, statistic=None):\n",
    "    plt.hist(null_samples)\n",
    "    plt.axvline(x=np.percentile(null_samples, 2.5), c='b')\n",
    "    legend = [\"95% quantiles\"]\n",
    "    if statistic is not None:\n",
    "        plt.axvline(x=statistic, c='r')\n",
    "        legend += [\"Actual test statistic\"]\n",
    "    plt.legend(legend)\n",
    "    plt.axvline(x=np.percentile(null_samples, 97.5), c='b')\n",
    "    plt.xlabel(\"Test statistic value\")\n",
    "    plt.ylabel(\"Counts\")\n",
    "    \n",
    "plot_permutation_samples(statistics, my_statistic)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that, depending on the test statistic, there might be multiple ways to compute the null distribution.\n",
    "Not all of them work via sampling it directly, which also can be costly.\n",
    "For example, if we just use the mean difference as a test statistic, and then plug in two Gaussian distributed random variables, we know that the distribution of the statistic is also Gaussian (see the above plot). \n",
    "We might be even able to analytically work out the parameters of this distribution.\n",
    "\n",
    "In case you haven't realized, this is basically a [t-test](https://en.wikipedia.org/wiki/Student%27s_t-test) where we assume that we know that both distributions have unit variance and we can distinguish them via solely looking at their mean.\n",
    "As it will be useful for kernel based test statistics later, we will also look at the distribution of a squared test statistic, in order to make it strictly positive (and zero in expectation if both distributions are the same).\n",
    "Naturally, the average of squared Gaussian random variables has a chi-square distribution -- something that can be used for the kernel tests that we introduce later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEKCAYAAAAFJbKyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XuYFOWd9vHvLQMixgMnXQXdwY1HRhhwMCjmFTziYdUsuuqrUaORy90YjcYDiqeoeIkvnghJXFQiZhPERVGjaAwiceMGdUBEUIwcdcQgjguKaBTye//omkkzDEzXzPR0z3B/rmuu6ap6qurXTTN3P1XVTykiMDMzy9U2hS7AzMxaFweHmZml4uAwM7NUHBxmZpaKg8PMzFJxcJiZWSoODjMzS8XBYWZmqTg4zMwslZJCF9AU3bp1i9LS0tTrvfNO5ve++zZvPfnUGms2szxp4h+E2bNnfxwR3Ru7+1YdHKWlpVRWVqZeb/DgzO+ZM5u1nLxqjTWbWZ408Q+CpOVN2b0PVZmZWSp5Cw5JEyR9JGl+PcuukBSSuiXTkjRW0iJJ8yT1z1ddZmbWNPnscTwEDK07U9IewNHAe1mzjwP2Tn6GA7/IY11mZtYEeQuOiHgJ+KSeRXcDVwHZ47mfDDwcGbOAnSXtlq/azMys8Vr0HIekk4APIuKNOot6AO9nTVcl8+rbxnBJlZIqV61aladKzcxsc1osOCR1AkYCN9S3uJ559d5hKiLGR0RFRFR0797oq8nMzKyRWvJy3H8CegFvSALoCcyRdDCZHsYeWW17AitasDYzM8tRi/U4IuLNiNglIkojopRMWPSPiL8ATwHnJFdXDQTWRMSHLVWbmZnlLp+X404C/gTsK6lK0gVbaD4NWAIsAu4H/j1fdZmZWdPk7VBVRJzZwPLSrMcB/CBftdRn1pJqSkfMarDdsttPaIFqzMxaD39z3MzMUnFwmJlZKg4OMzNLxcFhZmapODjMzCwVB4eZmaXi4DAzs1QcHGZmloqDw8zMUnFwmJlZKg4OMzNLxcFhZmapODjMzCwVB4eZmaXi4DAzs1QcHGZmloqDw8zMUnFwmJlZKg4OMzNLxcFhZmapODjMzCyVvAWHpAmSPpI0P2ve/5O0UNI8SVMl7Zy17BpJiyS9I+nYfNVlZmZNk88ex0PA0Drzfg+URUQf4M/ANQCSDgDOAHon6/xcUrs81mZmZo2Ut+CIiJeAT+rMez4i1ieTs4CeyeOTgUci4q8RsRRYBBycr9rMzKzxCnmO43zg2eRxD+D9rGVVybxNSBouqVJS5apVq/JcopmZ1VWQ4JA0ElgP/LpmVj3Nor51I2J8RFREREX37t3zVaKZmW1GSUvvUNK5wInAkRFREw5VwB5ZzXoCK1q6NjMza1iL9jgkDQWuBk6KiHVZi54CzpC0raRewN7Aqy1Zm5mZ5SZvPQ5Jk4DBQDdJVcCNZK6i2hb4vSSAWRFxUUQskPQo8BaZQ1g/iIgN+arNzMwaL2/BERFn1jP7wS20HwWMylc9ZmbWPPzNcTMzS8XBYWZmqTg4zMwsFQeHmZml4uAwM7NUHBxmZpaKg8PMzFJxcJiZWSoODjMzS8XBYWZmqTg4zMwsFQeHmZml4uAwM7NUHBxmZpaKg8PMzFJxcJiZWSoODjMzS8XBYWZmqTg4zMwsFQeHmZml4uAwM7NU8hYckiZI+kjS/Kx5XST9XtK7ye/OyXxJGitpkaR5kvrnqy4zM2uafPY4HgKG1pk3AnghIvYGXkimAY4D9k5+hgO/yGNdZmbWBHkLjoh4CfikzuyTgYnJ44nAKVnzH46MWcDOknbLV21mZtZ4LX2OY9eI+BAg+b1LMr8H8H5Wu6pknpmZFZliOTmueuZFvQ2l4ZIqJVWuWrUqz2WZmVldLR0cK2sOQSW/P0rmVwF7ZLXrCayobwMRMT4iKiKionv37nkt1szMNtXSwfEUcG7y+Fzgyaz55yRXVw0E1tQc0jIzs+JSkq8NS5oEDAa6SaoCbgRuBx6VdAHwHnBa0nwacDywCFgHfC9fdZmZWdPkLTgi4szNLDqynrYB/CBftZiZWfMplpPjZmbWSjg4zMwsFQeHmZml4uAwM7NUHBxmZpaKg8PMzFJxcJiZWSoODjMzS8XBYWZmqTg4zMwsFQeHmZml4uAwM7NUHBxmZpaKg8PMzFJxcJiZWSoODjMzS8XBYWZmqTg4zMwsFQeHmZml4uAwM7NUHBxmZpZKQYJD0mWSFkiaL2mSpI6Sekl6RdK7kiZL6lCI2szMbMtSB4ekzpL6NHaHknoAlwAVEVEGtAPOAEYDd0fE3sD/Ahc0dh9mZpY/OQWHpJmSdpTUBXgD+KWku5qw3xJgO0klQCfgQ+AIYEqyfCJwShO2b2ZmeZJrj2OniPgU+BfglxFxEHBUY3YYER8AY4D3yATGGmA2sDoi1ifNqoAejdm+mZnlV67BUSJpN+BfgaebskNJnYGTgV7A7sD2wHH1NI3NrD9cUqWkylWrVjWlFDMza4Rcg+MnwO+ARRHxmqS9gHcbuc+jgKURsSoivgYeBw4Fdk4OXQH0BFbUt3JEjI+Iioio6N69eyNLMDOzxippuAkAH0ZE7QnxiFjShHMc7wEDJXUCvgCOBCqBF4FTgUeAc4EnG7l9MzPLo1x7HD/NcV6DIuIVMifB5wBvJjWMB64GLpe0COgKPNiY7ZuZWX5tscch6RAyh5G6S7o8a9GOZC6jbZSIuBG4sc7sJcDBjd2mmZm1jIYOVXUAvpG02yFr/qdkDiuZmdlWZovBERF/AP4g6aGIWN5CNZmZWRHL9eT4tpLGA6XZ60TEEfkoyszMileuwfFfwH3AA8CG/JVjZmbFLtfgWB8Rv8hrJWZm1irkejnubyX9u6TdJHWp+clrZWZmVpRy7XGcm/y+MmteAHs1bzlmZlbscgqOiOiV70LMzKx1yCk4JJ1T3/yIeLh5yzEzs2KX66GqAVmPO5IZX2oO4OAwM9vK5Hqo6ofZ05J2An6Vl4rMzKyoNfae4+uAvZuzEDMzax1yPcfxW/5+Y6V2wP7Ao/kqyszMileu5zjGZD1eDyyPiKo81GNmZkUup0NVyWCHC8mMkNsZ+CqfRZmZWfHKKTgk/SvwKnAamfuOvyLJw6qbmW2Fcj1UNRIYEBEfAUjqDkwncyc/MzPbiuR6VdU2NaGRqE6xrpmZtSG59jiek/Q7YFIyfTowLT8lmZlZMWvonuPfBHaNiCsl/QtwGCDgT8CvW6A+MzMrMg0dbroH+AwgIh6PiMsj4jIyvY178l2cmZkVn4aCozQi5tWdGRGVZG4j2yiSdpY0RdJCSW9LOiS5x8fvJb2b/O7c2O2bmVn+NBQcHbewbLsm7Pde4LmI2A/oC7wNjABeiIi9gReSaTMzKzINBcdrki6sO1PSBcDsxuxQ0o7A/wEeBIiIryJiNXAyMDFpNhE4pTHbNzOz/GroqqofAVMlncXfg6IC6AB8p5H73AtYBfxSUt9ku5eSOQn/IUBEfChpl0Zu38zM8miLwRERK4FDJQ0BypLZz0TEjCbusz/ww4h4RdK9pDgsJWk4MBxgzz33bEIZZmbWGLnej+NF4MVm2mcVUBURryTTU8gEx0pJuyW9jd2Aj+pbOSLGA+MBKioqor42ZmaWPy3+7e+I+AvwvqR9k1lHAm8BTwHnJvPOBZ5s6drMzKxhuX5zvLn9EPi1pA7AEuB7ZELs0eTE+3tkBlQ0M7MiU5DgiIi5ZE6y13VkS9diZmbpeKBCMzNLxcFhZmapODjMzCwVB4eZmaXi4DAzs1QcHGZmloqDw8zMUnFwmJlZKg4OMzNLxcFhZmapODjMzCwVB4eZmaXi4DAzs1QcHGZmloqDw8zMUnFwmJlZKg4OMzNLxcFhZmapODjMzCwVB4eZmaXi4DAzs1QcHGZmlkrBgkNSO0mvS3o6me4l6RVJ70qaLKlDoWozM7PNK2SP41Lg7azp0cDdEbE38L/ABQWpyszMtqggwSGpJ3AC8EAyLeAIYErSZCJwSiFqMzOzLStUj+Me4Crgb8l0V2B1RKxPpquAHvWtKGm4pEpJlatWrcp/pWZmtpEWDw5JJwIfRcTs7Nn1NI361o+I8RFREREV3bt3z0uNZma2eSUF2Ocg4CRJxwMdgR3J9EB2llSS9Dp6AisKUJuZmTWgxXscEXFNRPSMiFLgDGBGRJwFvAicmjQ7F3iypWszM7OGFdP3OK4GLpe0iMw5jwcLXI+ZmdWjEIeqakXETGBm8ngJcHAh6zEzs4YVU4/DzMxaAQeHmZml4uAwM7NUHBxmZpaKg8PMzFJxcJiZWSoODjMzS8XBYWZmqRT0C4CtQemIZ3Jqt+z2E/JciZlZcXCPw8zMUnFwmJlZKg4OMzNLxcFhZmapODjMzCwVB4eZmaXi4DAzs1QcHGZmloq/AGjN6uuvv6aqqoovv/yy0KVYEejYsSM9e/akffv2hS7FmpGDw5pVVVUVO+ywA6WlpUgqdDlWQBFBdXU1VVVV9OrVq9DlWDPyoSprVl9++SVdu3Z1aBiS6Nq1q3ufbZCDw5qdQ8Nq+L3QNrV4cEjaQ9KLkt6WtEDSpcn8LpJ+L+nd5Hfnlq7N2oZ7772XsrIyevfuzT333FM7/6abbqJHjx6Ul5dTXl7OtGnTAHj55Zfp06cPAwYMYNGiRQCsXr2aY489logoyHMAeOKJJ3jrrbdqp2+44QamT58OwODBg6msrCxUabaVK0SPYz3w44jYHxgI/EDSAcAI4IWI2Bt4IZk2S2X+/Pncf//9vPrqq7zxxhs8/fTTvPvuu7XLL7vsMubOncvcuXM5/vjjAbjzzjt57LHHuO222/jFL34BwC233MK1115b0E/MdYPj5ptv5qijjipYPWY1Wjw4IuLDiJiTPP4MeBvoAZwMTEyaTQROaenarPV7++23GThwIJ06daKkpITDDz+cqVOnbnGd9u3b88UXX7Bu3Trat2/P4sWL+eCDDzj88MM3u85zzz3Hfvvtx2GHHcYll1zCiSeeCGR6NWPGjKltV1ZWxrJlywA45ZRTOOigg+jduzfjx4+vbfONb3yDkSNH0rdvXwYOHMjKlSv5n//5H5566imuvPJKysvLWbx4Meeddx5TpkzZpJbnn3+eQw45hP79+3Paaaexdu1aAEaMGMEBBxxAnz59uOKKK3J+Dc0aUtCrqiSVAv2AV4BdI+JDyISLpF0KWJo1gx/9CObObd5tlpdD1tGnTZSVlTFy5Eiqq6vZbrvtmDZtGhUVFbXLx40bx8MPP0xFRQV33nknnTt35pprrmH48OFst912/OpXv+KKK67glltu2ew+vvzySy688EJmzJjBN7/5TU4//fScap8wYQJdunThiy++YMCAAQwbNoyuXbvy+eefM3DgQEaNGsVVV13F/fffz3XXXcdJJ53EiSeeyKmnnrrZbX788cfceuutTJ8+ne23357Ro0dz1113cfHFFzN16lQWLlyIJFavXp1TjWa5KNjJcUnfAB4DfhQRn6ZYb7ikSkmVq1atyl+B1irtv//+XH311Rx99NEMHTqUvn37UlKS+Xz0b//2byxevJi5c+ey22678eMf/xiA8vJyZs2axYsvvsiSJUvYfffdiQhOP/10zj77bFauXLnRPhYuXEivXr3Ye++9kcTZZ5+dU21jx46t7VW8//77tYfQOnToUNtjOeigg2p7KLmYNWsWb731FoMGDaK8vJyJEyeyfPlydtxxRzp27Mj3v/99Hn/8cTp16pTzNs0aUpAeh6T2ZELj1xHxeDJ7paTdkt7GbsBH9a0bEeOB8QAVFRWFO3NpDdpSzyCfLrjgAi644AIArr32Wnr27AnArrvuWtvmwgsvrP1jXSMiuPXWW5k8eTIXX3wxP/nJT1i2bBljx45l1KhRG7Xd3LmPkpIS/va3v9VO11yKOnPmTKZPn86f/vQnOnXqxODBg2uXtW/fvnZ77dq1Y/369Tk/14jg6KOPZtKkSZsse/XVV3nhhRd45JFHGDduHDNmzMh5u2ZbUoirqgQ8CLwdEXdlLXoKODd5fC7wZEvXZm3DRx9lPnO89957PP7445x55pkAfPjhh7Vtpk6dSllZ2UbrTZw4kRNOOIHOnTuzbt06ttlmG7bZZhvWrVu3Ubv99tuPpUuXsnjxYoCN/miXlpYyZ84cAObMmcPSpUsBWLNmDZ07d6ZTp04sXLiQWbNmNfg8dthhBz777LMtthk4cCAvv/xy7dVg69at489//jNr165lzZo1HH/88dxzzz3Mbe5jhrZVK0SPYxDwXeBNSTXv5muB24FHJV0AvAecVoDaGs33Ji8ew4YNo7q6mvbt2/Ozn/2Mzp0zV3ZfddVVzJ07F0mUlpbyH//xH7XrrFu3jokTJ/L8888DcPnllzNs2DA6dOiwyaf5jh07Mn78eE444QS6devGYYcdxvz582v3/fDDD1NeXs6AAQPYZ599ABg6dCj33Xcfffr0Yd9992XgwIENPo8zzjiDCy+8kLFjx9Z7Uhyge/fuPPTQQ5x55pn89a9/BeDWW29lhx124OSTT+bLL78kIrj77rtTvopmm6dCXqfeVBUVFdGYa9kHD4ZZS6r5h//b8Ke+5tbY4Bg8OPN75sxmKyUv3n77bfbff/9Cl9GiZs6cyZgxY3j66acLXUpR2hrfE3nXxD8IkmZHREXDLevnb46bmVkqHuTQrIkGDx7M4JpPgGZbAfc4zMwsFQeHmZml4uAwM7NUHBxmZpaKg8PapKlTpyKJhQsXNtj2oYceYsWKFY3e18yZMzf5FjrA3Llza4duT2v16tX8/Oc/T7XOPffcs8mXFXNpd/zxx29xLKu07a3tc3BYmzRp0iQOO+wwHnnkkQbbNjU4Nqe1BMe0adPYeeedm629tX0ODmtz1q5dy8svv8yDDz64SXDccccdHHjggfTt25cRI0YwZcoUKisrOeussygvL+eLL76gtLSUjz/+GIDKysraS21fffVVDj30UPr168ehhx7KO++8s9kavvrqK2644QYmT55MeXk5kydP5vPPP+f8889nwIAB9OvXjyefzIyqs2DBAg4++GDKy8vp06cP7777LiNGjGDx4sWUl5dz5ZVXbrTtzz//nBNOOIG+fftSVlbG5MmTGTt2LCtWrGDIkCEMGTIEyAzqWFFRQe/evbnxxhsB6m1X83xz3W726/Pwww/Tp08f+vbty3e/+92m/LNZK+LvcVj+FGJcdTI3QBo6dCj77LMPXbp0Yc6cOfTv359nn32WJ554gldeeYVOnTrxySef0KVLF8aNG8eYMWM2Gn69Pvvttx8vvfQSJSUlTJ8+nWuvvZbHHnus3rYdOnTg5ptvprKyknHjxgGZARePOOIIJkyYwOrVqzn44IM56qijuO+++7j00ks566yz+Oqrr9iwYQO333478+fPr3eMqeeee47dd9+dZ57JDHOzZs0adtppJ+666y5efPFFunXrBsCoUaPo0qULGzZs4Mgjj2TevHlccsklm7RLu90aCxYsYNSoUbz88st069aNTz75ZIuvn7Ud7nFYmzNp0iTOOOMMIDPeU81YU9OnT+d73/te7RDjXbp0SbXdNWvWcNppp1FWVsZll13GggULUq3//PPPc/vtt1NeXl47Ou57773HIYccwm233cbo0aNZvnw522233Ra3c+CBBzJ9+nSuvvpq/vu//5uddtqp3naPPvoo/fv3p1+/fixYsGCjuwk2Zbs1ZsyYwamnnlobKGlfT2u93OOw/CnAuOrV1dXMmDGD+fPnI4kNGzYgiTvuuIOIyOlWsNlDo9cMfQ5w/fXXM2TIEKZOncqyZctSf1s8InjsscfYd999N5q///77861vfYtnnnmGY489lgceeIC99tprs9vZZ599mD17NtOmTeOaa67hmGOO4YYbbtiozdKlSxkzZgyvvfYanTt35rzzztvouTR2u3WfTyFvrWuF4x6HtSlTpkzhnHPOYfny5Sxbtoz333+fXr168cc//pFjjjmGCRMm1J7orTm0Unf48tLSUmbPng2w0aGoNWvW0KNHDyBzQr0hdbd77LHH8tOf/pSagUVff/11AJYsWcJee+3FJZdcwkknncS8efO2OKT6ihUr6NSpE2effTZXXHFF7TDu2et8+umnbL/99uy0006sXLmSZ599drN1pdlutiOPPJJHH32U6upqAB+q2oo4OKxNmTRpEt/5znc2mjds2DB+85vfMHToUE466SQqKiooLy+vvTf4eeedx0UXXVR7cvzGG2/k0ksv5dvf/jbt2rWr3c5VV13FNddcw6BBg9iwYUODtQwZMoS33nqr9uT49ddfz9dff02fPn0oKyvj+uuvB2Dy5MmUlZVRXl7OwoULOeecc+jatSuDBg2irKxsk5Pjb775Zu3J9FGjRnHdddcBMHz4cI477jiGDBlC37596devH7179+b8889n0KBBtetnt0u73Wy9e/dm5MiRHH744fTt25fLL7+8wdfE2gYPq16k6g6/7mHVrbXyeyIPPKy6mZm1Jg4OMzNLxcFhZmapODis2bXm82bWvPxeaJv8PY4iVTrimY2m/7JkYDJ/4xP6jb2Heb507NiR6upqunbt6mv8t3IRQXV1NR07dix0KdbMHBzWrHr27ElVVRWrVq0qdClWBDp27EjPnj0LXYY1s6ILDklDgXuBdsADEXF7gUuyFNq3b0+vXr0KXYaZ5VFRBYekdsDPgKOBKuA1SU9FxJYH2dmK1T2ktTnFdkjLzFqvogoO4GBgUUQsAZD0CHAy4OBopRxsZm1PsV1V1QN4P2u6KplnZmZFoth6HPVdhrPR9XyShgPDk8m1kjZ/N50t67Z8NB83ct2CWT6abpC+bo3OQzG5a7DmAte3OY16rQusNdYMrbPuwtec/srFmpr/sSm7LbbgqAL2yJruCWx0T8+IGA+Mb+qOJFU2ZayWQmmNdbfGmqF11t0aa4bWWffWXHOxHap6DdhbUi9JHYAzgKcKXJOZmWUpqh5HRKyXdDHwOzKX406IiHS3WTMzs7wqquAAiIhpwLQW2FWTD3cVSGusuzXWDK2z7tZYM7TOurfamlv1/TjMzKzlFds5DjMzK3JtMjgkDZX0jqRFkkbUs3xbSZOT5a9IKs1adk0y/x1JxxZ7zZK6SnpR0lpJ41qq3mao+2hJsyW9mfw+ohXUfLCkucnPG5K+U3fdYqw7a/meyfvkimKvWVKppC+yXu/7ir3mZFkfSX+StCB5b7fYCI9NeK3Pynqd50r6m6TyLe4sItrUD5mT6ouBvYAOwBvAAXXa/DtwX/L4DGBy8viApP22QK9kO+2KvObtgcOAi4Bxrei17gfsnjwuAz5oBTV3AkqSx7sBH9VMF3PdWcsfA/4LuKLYawZKgfkt+X5uhppLgHlA32S6a0v8/Wiu90cy/0BgSUP7a4s9jtphSyLiK6Bm2JJsJwMTk8dTgCMlKZn/SET8NSKWAouS7RVtzRHxeUT8EfiyBeqsqyl1vx4RNd/RWQB0lLRtkde8LiLWJ/M7UufLqXnWlPc1kk4BlpB5rVtKk2oukKbUfAwwLyLeAIiI6ojY0ArqznYmMKmhnbXF4Mhl2JLaNskfgjVkPh0UasiTptRcSM1V9zDg9Yj4a57qrLeeRKqaJX1L0gLgTeCirCDJt0bXLWl74GrgJy1QZ731JNK+P3pJel3SHyR9O9/F1q0nkabmfYCQ9DtJcyRd1QL1blJTorH/F08nh+Aoustxm0GDw5ZsoU0u6+ZDU2oupCbXLak3MJrMp7WW0KSaI+IVoLek/YGJkp6NiJbo7TWl7p8Ad0fE2hb+MN+Umj8E9oyIakkHAU9I6h0RnzZ3kTnWk0ubEjKHjQcA64AXJM2OiBeat8R6Ncf/xW8B6yJifkM7a4s9jgaHLcluI6kE2An4JMd186EpNRdSk+qW1BOYCpwTEYvzXm2dehKNeq0j4m3gczLnZ1pCU+r+FnCHpGXAj4Brlfmibb41uubkcHE1QETMJnP8fp+8V9z0vx9/iIiPI2Idme+j9c97xXVqSjTmfX0GOfQ2gDZ5cryEzLHcXvz9JFHvOm1+wMYniR5NHvdm45PjS2iZk+ONrjlr+Xm0/MnxprzWOyfth7Wimnvx95Pj/0jmP2a3Yq+7TpubaLmT4015rbvX/N8jc8L3A6BLkdfcGZhDchEFMB04odhf62R6GzLBsldO+2uJJ9XSP8DxwJ/JfEoZmcy7GTgpedyRzNUli4BXs18sYGSy3jvAca2k5mVkPjmsTf7xDyj2uoHryHxin5v1s0uR1/xdMieX5yZ/IE5pLe/rrG3cRAsFRxNf62HJa/1G8lr/c7HXnCw7O6l7PnBHa3l/AIOBWbnuy98cNzOzVNriOQ4zM8sjB4eZmaXi4DAzs1QcHGZmloqDw8zMUnFwWNFSZuTfmhE7/yLpg6zpDim2c76kf0jR/ghJA9O2k/QDSWc1V/umkFQiaXU+tm3WFoccsTYiMt8cLgeQdBOwNiLGNGJT55P5LsBfcmx/BPAxMCtNu4j4WTO3NytK7nFYqyTpXEmvJr2Pn0vaJvmU/avkPgjzJV0i6XQy4TO5vp6KpMskvaXM/TX+U9I/Ad8HrkzaHyrp5OT+Ba9Lel7SLptpd6ukH6XYbnb7fSTNSNrPqblXQladd0oanjV9q6RLJe2YrDdH0jxJJ9bzWh0l6Yms6fsknZ08HpAMIjhb0rOSdm2Ofx9r29zjsFZHUhnwHeDQiFgvaTyZIRQWkxkC5MCk3c4RsVrSD4GLI2JuPZu7CvjHiPgqq/0DwMcRcU+ync7AUxERki4CfhwRV9fT7viU281uPwm4KSJ+q8zNf+p+qHsEuJ2/3zP6NGAI8AVwckR8JmkX4GXg6Rxfx22Be8l8s/jj5LDZLcDwLa9pWzsHh7VGR5EZgbQyGe11OzLDRf8O2FfSvWQGmHs+h20tAP5T0pPAE5tpsyfwaHKeZFsywzo0x3aB2mDqFhG/BYh6RtuNiNck7ZH0CHoCf4mIFUkParSkw4C/AXtI6gbkcn5jfzLjs01PXsd2ZIasMdsiB4e1RgImRMT1myyQ+gDHAZeQGe+ooU/PxwKHk7nJzXVXlgP6AAABfUlEQVRJb6aunwG3RcQ0SUcBm9yWs5HbzZbL2D+PkXlOpWR6IADnkBnltH/S+6oiMyZRtvVs3IOpWS4yNx5qqXtdWBvhcxzWGk0H/jX5ZF1z9dWekroDioj/Am7k70NafwbsUHcjktoBPSNiBnAlmRFZO9XTfifgA2U+lp+bNb+p2wUgIv4X+FjSPyfrd5TUqZ7n/QiZQ3L/QiZEamr7KAmNo6n/xmPLydxDpEPSu6m5v/tbQA9JByf77aDM/VHMtsjBYa1ORLxJ5uZE0yXNI3NIalcy9xp4SdJc4H7g2mSVXwIP1HNyvAT4TbKNOcDoiPgMeJJMML0u6VAyI8pOBf4ArMxav267tNvNdhbw42SdP5IJm7rP+41k/tKI+CiZ/SvgUEmVZM57vFvPekvJHC57E3g4qYnI3HHxVOAuSW8Ar5O5d4fZFnl0XDMzS8U9DjMzS8XBYWZmqTg4zMwsFQeHmZml4uAwM7NUHBxmZpaKg8PMzFJxcJiZWSr/H9rNdlhLqZLhAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# IMPLEMENT: use squared test statistic\n",
    "simple_statistic_squared = lambda X,Y: simple_statistic(X,Y,squared=True)\n",
    "\n",
    "# visualise test\n",
    "statistics = two_sample_permutation_test(simple_statistic_squared, X, Y, num_permutations)\n",
    "my_statistic = simple_statistic_squared(X,Y)\n",
    "plot_permutation_samples(statistics, my_statistic)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Alternative distribution via explicit simulation\n",
    "\n",
    "As a caveat, please note that the test statistic itself has a distribution under the alternative hypothesis ($P_X \\neq P_Y$), and that our computed test statistic is only a single sample from this distribution.\n",
    "In practice, it is unfortunately not possible to look at the distribution of the test under the alternative, as this would require a way to generate more data.\n",
    "We can try this for a synthetic example where we have infinite data though."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtcAAAEKCAYAAADZxnkxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XucVXW9//HXRy4hhsotj0KdwV94Y4JBB8NLP0FMUTxqR01ME9PkUeeYt7yA1zIp7YdGZmVkJnYK8aCgJZoiUCdPaICE3AxBVMIQMUhFMvD7+2MW0wADs2dm7dkz8Ho+Hvux9177u777s7+uznnzne9aK1JKSJIkSWq83UpdgCRJkrSzMFxLkiRJOTFcS5IkSTkxXEuSJEk5MVxLkiRJOTFcS5IkSTkxXEuSJEk5MVxLkiRJOTFcS5IkSTlpXeoCGqNLly6prKysyb/3xRerng88sLl1tnNyiCRJUqnNnj37zZRS17ratehwXVZWxqxZs5r8ewcMqHqeMaO5dbZzcogkSVKpRcQrhbRzWYgkSZKUE8O1JEmSlBPDtSRJkpSTFr3mWpIkqbH+8Y9/sGLFCjZs2FDqUtQMtGvXju7du9OmTZsG7W+4liRJu7QVK1bQoUMHysrKiIhSl6MSSimxZs0aVqxYQY8ePRrUh8tCJEnSLm3Dhg107tzZYC0igs6dOzfqrxiGa0mStMszWGuzxh4LhmtJkiQpJ4ZrSZKkEvvud79LeXk5vXr1YsyYMdXbv/a1r9GtWzcqKiqoqKhgypQpADzzzDP07t2bfv368dJLLwGwdu1aTjjhBFJKJfkNAJMnT2bhwoXV72+88UamTp0KwIABA0py87+m5gmNDTRz2RrKRswsqO3yW4cUuRpJktRSzZ8/nx//+Mc899xztG3blsGDBzNkyBB69uwJwOWXX86VV165xT633347Dz30EMuXL+eHP/wht99+O9/4xje49tprS7rEZfLkyZx88skccsghANx8880lq6VUijZzHRH3RsQbETG/xrb/FxGLI2JeREyKiL1rfDYyIl6KiBcj4oRi1SVJktScLFq0iP79+9O+fXtat27NMcccw6RJk3a4T5s2bXjvvfdYv349bdq0YenSpfz5z3/mmGOO2e4+TzzxBAcddBBHH300l1xyCSeffDJQNTs+evTo6nbl5eUsX74cgNNOO43DDjuMXr16MXbs2Oo2H/7wh7nuuuvo06cP/fv3Z9WqVfzv//4vjz76KFdddRUVFRUsXbqU888/n4kTJ25Ty5NPPskRRxzBoYceyplnnsk777wDwIgRIzjkkEPo3bv3Nv+gaCmKOXN9H3AXcH+NbU8BI1NKGyPiNmAkcE1EHAIMBXoB+wFTI+KAlNKmItYnSZK0hcsug7lz8+2zogJqrPTYRnl5Oddddx1r1qxh9913Z8qUKVRWVlZ/ftddd3H//fdTWVnJ7bffTseOHRk5ciTDhw9n991352c/+xlXXnkl3/jGN7b7HRs2bOCiiy5i2rRpfPzjH+ess84qqPZ7772XTp068d5779GvXz9OP/10OnfuzLvvvkv//v0ZNWoUV199NT/+8Y+5/vrrOeWUUzj55JM544wzttvnm2++yS233MLUqVPZY489uO2227jjjju4+OKLmTRpEosXLyYiWLt2bUE1NjdFm7lOKf0WeGurbU+mlDZmb2cC3bPXpwIPpJT+nlJ6GXgJOLxYtUmSJDUXBx98MNdccw2f/vSnGTx4MH369KF166r5zy9/+cssXbqUuXPnsu+++/LVr34VgIqKCmbOnMn06dNZtmwZ++23HyklzjrrLM4991xWrVq1xXcsXryYHj160LNnTyKCc889t6Da7rzzzurZ6ddee40lS5YA0LZt2+qZ78MOO6x6prsQM2fOZOHChRx11FFUVFQwbtw4XnnlFfbcc0/atWvHF7/4RR5++GHat29fcJ/NSSnXXF8ATMhed6MqbG+2ItsmSZLUZHY0w1xMF154IRdeeCEA1157Ld27V80/7rPPPtVtLrrooupAu1lKiVtuuYUJEyZw8cUX8/Wvf53ly5dz5513MmrUqC3abm8tduvWrfnggw+q32++xvOMGTOYOnUqv//972nfvj0DBgyo/qxNmzbV/bVq1YqNGzdu2/F2pJT49Kc/zfjx47f57LnnnuPpp5/mgQce4K677mLatGkF99tclORqIRFxHbAR+PnmTbU0q/VU14gYHhGzImLW6tWri1WiJElSk3njjTcAePXVV3n44Yc5++yzAXj99der20yaNIny8vIt9hs3bhxDhgyhY8eOrF+/nt12243ddtuN9evXb9HuoIMO4uWXX2bp0qUAWwTbsrIy5syZA8CcOXN4+eWXAVi3bh0dO3akffv2LF68mJkz676QQ4cOHXj77bd32KZ///4888wz1Vc5Wb9+PX/605945513WLduHSeddBJjxoxhbt7rc5pIk89cR8Qw4GRgUPrntWJWAB+t0aw7sLK2/VNKY4GxAJWVlaW71owkSVJOTj/9dNasWUObNm34/ve/T8eOHQG4+uqrmTt3LhFBWVkZP/rRj6r3Wb9+PePGjePJJ58E4IorruD000+nbdu228wKt2vXjrFjxzJkyBC6dOnC0Ucfzfz586u/+/7776eiooJ+/fpxwAEHADB48GDuvvtuevfuzYEHHkj//v3r/B1Dhw7loosu4s4776z1REaArl27ct9993H22Wfz97//HYBbbrmFDh06cOqpp7JhwwZSSnznO9+p5yg2D1HMayFGRBnwq5RSefZ+MHAHcExKaXWNdr2AX1C1zno/4GmgZ10nNFZWVqZSXC9xwICqS/H9y+dyuBTfgAFVzzNmNLasnZZDJEkqpkWLFnHwwQeXuowmNWPGDEaPHs2vfvWrUpfSLNV2TETE7JRS5XZ2qVa0meuIGA8MALpExArgJqquDvIh4Klsnc7MlNKXUkoLIuJBYCFVy0X+0yuFSJIkqaUpWrhOKZ1dy+af7KD9KGDU9j6XJElSPgYMGMCAzX8aVq68/bkkSZKUE8O1JEmSlBPDtSRJkpQTw7UkSZKUE8O1JElSMzBp0iQigsWLF9fZ9r777mPlylpvCVKQGTNmbHO3R4C5c+cyZcqUBvW5du1afvCDH9RrnzFjxmxzw5tC2p100kmsXbs2t/Z5MlxLkiQ1A+PHj+foo4/mgQceqLNtY8P19rSUcD1lyhT23nvv3NrnyXAtSZJUYu+88w7PPPMMP/nJT7YJ19/+9rf5xCc+QZ8+fRgxYgQTJ05k1qxZnHPOOVRUVPDee+9RVlbGm2++CcCsWbOqL7P33HPPceSRR9K3b1+OPPJIXnzxxe3W8P7773PjjTcyYcIEKioqmDBhAu+++y4XXHAB/fr1o2/fvjzyyCMALFiwgMMPP5yKigp69+7NkiVLGDFiBEuXLqWiooKrrrpqi77fffddhgwZQp8+fSgvL2fChAnceeedrFy5koEDBzJw4EAAvvzlL1NZWUmvXr246aabAGptt/n3FtpvzfG5//776d27N3369OHzn/98Y/6z1arJb38uSZLUbF12Gcydm2+fFRUwZswOm0yePJnBgwdzwAEH0KlTJ+bMmcOhhx7K448/zuTJk3n22Wdp3749b731Fp06deKuu+5i9OjRVFbu+IaBBx10EL/97W9p3bo1U6dO5dprr+Whhx6qtW3btm25+eabmTVrFnfddRcA1157Lcceeyz33nsva9eu5fDDD+e4447j7rvv5tJLL+Wcc87h/fffZ9OmTdx6663Mnz+fubWM3xNPPMF+++3HY489BsC6devYa6+9uOOOO5g+fTpdunQBYNSoUXTq1IlNmzYxaNAg5s2bxyWXXLJNu/r2u9mCBQsYNWoUzzzzDF26dOGtt97a4fg1hDPXkiRJJTZ+/HiGDh0KwNChQxk/fjwAU6dO5Qtf+ALt27cHoFOnTvXqd926dZx55pmUl5dz+eWXs2DBgnrt/+STT3LrrbdSUVHBgAED2LBhA6+++ipHHHEE3/zmN7ntttt45ZVX2H333XfYzyc+8QmmTp3KNddcw//8z/+w11571druwQcf5NBDD6Vv374sWLCAhQsX5tLvZtOmTeOMM86oDt31Hc9COHMtSZK0WR0zzMWwZs0apk2bxvz584kINm3aRETw7W9/m5QSEVFnH61bt+aDDz4AYMOGDdXbb7jhBgYOHMikSZNYvnx5ve/KmFLioYce4sADD9xi+8EHH8wnP/lJHnvsMU444QTuuece9t9//+32c8ABBzB79mymTJnCyJEjOf7447nxxhu3aPPyyy8zevRo/vCHP9CxY0fOP//8LX5LQ/vd+vcUMp6N4cy1JElSCU2cOJHzzjuPV155heXLl/Paa6/Ro0cPfve733H88cdz7733Vp+ct3kZQ4cOHXj77ber+ygrK2P27NkAWyz7WLduHd26dQOqToKsy9b9nnDCCXzve98jpQTA888/D8CyZcvYf//9ueSSSzjllFOYN2/eNvvWtHLlStq3b8+5557LlVdeyZw5c7b5vr/97W/sscce7LXXXqxatYrHH398u3XVp9+aBg0axIMPPsiaNWsAXBYiSZK0sxk/fjyf+cxntth2+umn84tf/ILBgwdzyimnUFlZSUVFBaNHjwbg/PPP50tf+lL1CY033XQTl156KZ/61Kdo1apVdT9XX301I0eO5KijjmLTpk111jJw4EAWLlxYfULjDTfcwD/+8Q969+5NeXk5N9xwAwATJkygvLyciooKFi9ezHnnnUfnzp056qijKC8v3+aExhdeeKH6BMhRo0Zx/fXXAzB8+HBOPPFEBg4cSJ8+fejbty+9evXiggsu4Kijjqrev2a7+vZbU69evbjuuus45phj6NOnD1dccUWdY1JfsflfIi1RZWVlmjVrVpN/74ABMHPZGv7lczMLar/81iE77gxgxozGlrXTcogkScW0aNEiDj744FKXoWaktmMiImanlHZ8BinOXEuSJEm5MVxLkiRJOTFcS5KkXV5LXiarfDX2WDBcS5KkXVq7du1Ys2aNAVuklFizZg3t2rVrcB9e51qSJO3SunfvzooVK1i9enWpS1Ez0K5dO7p3797g/Q3XkiRpl9amTRt69OhR6jK0k3BZiCRJkpQTw7UkSZKUE8O1JEmSlBPDtSRJkpQTw7UkSZKUE8O1JEmSlJOiheuIuDci3oiI+TW2dYqIpyJiSfbcMdseEXFnRLwUEfMi4tBi1SVJkiQVSzFnru8DBm+1bQTwdEqpJ/B09h7gRKBn9hgO/LCIdUmSJElFUbRwnVL6LfDWVptPBcZlr8cBp9XYfn+qMhPYOyL2LVZtkiRJUjE09ZrrfVJKrwNkzx/JtncDXqvRbkW2bRsRMTwiZkXELG9TKkmSpOakuZzQGLVsS7U1TCmNTSlVppQqu3btWuSyJEmSpMI1dbhetXm5R/b8RrZ9BfDRGu26AyubuDZJkiSpUZo6XD8KDMteDwMeqbH9vOyqIf2BdZuXj0iSJEktRetidRwR44EBQJeIWAHcBNwKPBgRFwKvAmdmzacAJwEvAeuBLxSrLkmSJKlYihauU0pnb+ejQbW0TcB/FqsWSZIkqSk0lxMaJUmSpBbPcC1JkiTlxHAtSZIk5cRwLUmSJOXEcC1JkiTlxHAtSZIk5cRwLUmSJOXEcC1JkiTlxHAtSZIk5cRwLUmSJOXEcC1JkiTlxHAtSZIk5cRwLUmSJOXEcC1JkiTlxHAtSZIk5cRwLUmSJOXEcC1JkiTlxHAtSZIk5cRwLUmSJOXEcC1JkiTlxHAtSZIk5cRwLUmSJOXEcC1JkiTlxHAtSZIk5aQk4ToiLo+IBRExPyLGR0S7iOgREc9GxJKImBARbUtRmyRJktRQTR6uI6IbcAlQmVIqB1oBQ4HbgO+klHoCfwUubOraJEmSpMYo1bKQ1sDuEdEaaA+8DhwLTMw+HwecVqLaJEmSpAZp8nCdUvozMBp4lapQvQ6YDaxNKW3Mmq0AujV1bZIkSVJjlGJZSEfgVKAHsB+wB3BiLU3TdvYfHhGzImLW6tWri1eoJEmSVE+lWBZyHPBySml1SukfwMPAkcDe2TIRgO7Aytp2TimNTSlVppQqu3bt2jQVS5IkSQUoRbh+FegfEe0jIoBBwEJgOnBG1mYY8EgJapMkSZIarBRrrp+l6sTFOcALWQ1jgWuAKyLiJaAz8JOmrk2SJElqjNZ1N8lfSukm4KatNi8DDi9BOZIkSVIuvEOjJEmSlBPDtSRJkpQTw7UkSZKUE8O1JEmSlBPDtSRJkpQTw7UkSZKUk3qH64joGBG9i1GMJEmS1JIVFK4jYkZE7BkRnYA/Aj+NiDuKW5okSZLUshQ6c71XSulvwL8DP00pHQYcV7yyJEmSpJan0HDdOiL2BT4L/KqI9UiSJEktVqHh+uvAr4GXUkp/iIj9gSXFK0uSJElqeVoX2O71lFL1SYwppWWuuZYkSZK2VOjM9fcK3CZJkiTtsnY4cx0RRwBHAl0j4ooaH+0JtCpmYZIkSVJLU9eykLbAh7N2HWps/xtwRrGKkiRJklqiHYbrlNJvgN9ExH0ppVeaqCZJkiSpRSr0hMYPRcRYoKzmPimlY4tRlCRJktQSFRqu/xu4G7gH2FS8ciRJkqSWq9BwvTGl9MOiViJJkiS1cIVeiu+XEfEfEbFvRHTa/ChqZZIkSVILU+jM9bDs+aoa2xKwf77lSJIkSS1XQeE6pdSj2IVIkiRJLV1B4Toizqtte0rp/nzLkSRJklquQpeF9Kvxuh0wCJgDGK4lSZKkTKHLQr5S831E7AX8rCgVSZIkSS1UoVcL2dp6oGdDvzQi9o6IiRGxOCIWRcQR2RVInoqIJdlzx4b2L0mSJJVCQeE6In4ZEY9mj8eAF4FHGvG93wWeSCkdBPQBFgEjgKdTSj2Bp7P3kiRJUotR6Jrr0TVebwReSSmtaMgXRsSewP8FzgdIKb0PvB8RpwIDsmbjgBnANQ35DkmSJKkUCpq5Tin9BlgMdAA6Au834jv3B1YDP42I5yPinojYA9gnpfR69n2vAx9pxHdIkiRJTa7QZSGfBZ4DzgQ+CzwbEWc08DtbA4cCP0wp9QXepR5LQCJieETMiohZq1evbmAJkiRJUv4KPaHxOqBfSmlYSuk84HDghgZ+5wpgRUrp2ez9RKrC9qqI2Bcge36jtp1TSmNTSpUppcquXbs2sARJkiQpf4WG691SSjXD7pp67LuFlNJfgNci4sBs0yBgIfAo/7zN+jAad8KkJEmS1OQKPaHxiYj4NTA+e38WMKUR3/sV4OcR0RZYBnyBqrD+YERcCLxK1RIUSZIkqcXYYbiOiI9TdaLhVRHx78DRQAC/B37e0C9NKc0FKmv5aFBD+5QkSZJKra6lHWOAtwFSSg+nlK5IKV1O1az1mGIXJ0mSJLUkdYXrspTSvK03ppRmAWVFqUiSJElqoeoK1+128NnueRYiSZIktXR1hes/RMRFW2/MTjqcXZySJEmSpJaprquFXAZMiohz+GeYrgTaAp8pZmGSJElSS7PDcJ1SWgUcGREDgfJs82MppWlFr0ySJElqYQq6znVKaTowvci1SJIkSS1aoTeRkbQ9079V6gqazsCRpa5AkqRmrUG3MJckSZK0LcO1JEmSlBPDtSRJkpQTw7UkSZKUE8O1JEmSlBPDtSRJkpQTw7UkSZKUE8O1JEmSlBPDtSRJkpQTw7UkSZKUE29/rvzlfTvwtZ/L+v1Fvv1KkiTlzJlrSZIkKSeGa0mSJCknhmtJkiQpJ4ZrSZIkKSeGa0mSJCknhmtJkiQpJyUL1xHRKiKej4hfZe97RMSzEbEkIiZERNtS1SZJkiQ1RClnri8FFtV4fxvwnZRST+CvwIUlqUqSJElqoJKE64joDgwB7sneB3AsMDFrMg44rRS1SZIkSQ1VqpnrMcDVwAfZ+87A2pTSxuz9CqBbbTtGxPCImBURs1avXl38SiVJkqQCNXm4joiTgTdSSrNrbq6laapt/5TS2JRSZUqpsmvXrkWpUZIkSWqI1iX4zqOAUyLiJKAdsCdVM9l7R0TrbPa6O7CyBLVJkiRJDdbkM9cppZEppe4ppTJgKDAtpXQOMB04I2s2DHikqWuTJEmSGqM5Xef6GuCKiHiJqjXYPylxPZIkSVK9lGJZSLWU0gxgRvZ6GXB4KeuRJEmSGqM5zVxLkiRJLZrhWpIkScqJ4VqSJEnKieFakiRJyonhWpIkScqJ4VqSJEnKSUkvxSephZn+rVJX0LQGjix1BZKkFsaZa0mSJCknhmtJkiQpJ4ZrSZIkKSeGa0mSJCknhmtJkiQpJ4ZrSZIkKSeGa0mSJCknhmtJkiQpJ4ZrSZIkKSeGa0mSJCknhmtJkiQpJ61LXcCuoGzEY9v97IFlawAYWqPN8luHFL0mSZIk5c+Za0mSJCknhmtJkiQpJ4ZrSZIkKSeGa0mSJCknhmtJkiQpJ00eriPioxExPSIWRcSCiLg0294pIp6KiCXZc8emrk2SJElqjFJcim8j8NWU0pyI6ADMjoingPOBp1NKt0bECGAEcE0J6iu5HV26rzZeuk+SJKl5aPKZ65TS6ymlOdnrt4FFQDfgVGBc1mwccFpT1yZJkiQ1RknXXEdEGdAXeBbYJ6X0OlQFcOAjpatMkiRJqr+S3aExIj4MPARcllL6W0QUut9wYDjAxz72seIVKEnTv1XqCprWwJGlrkCSWrySzFxHRBuqgvXPU0oPZ5tXRcS+2ef7Am/Utm9KaWxKqTKlVNm1a9emKViSJEkqQCmuFhLAT4BFKaU7anz0KDAsez0MeKSpa5MkSZIaoxTLQo4CPg+8EBFzs23XArcCD0bEhcCrwJklqE2SJElqsCYP1yml3wHbW2A9qClrkSRJkvJUshMadyWXtZ643c+6x+o629Rp+ryG7ytJkqTcePtzSZIkKSeGa0mSJCknhmtJkiQpJ665liRV8aY5ktRozlxLkiRJOTFcS5IkSTkxXEuSJEk5MVxLkiRJOTFcS5IkSTkxXEuSJEk5MVxLkiRJOTFcS5IkSTkxXEuSJEk58Q6NO4ExT/+pXu0vG3RAkSqRJEnatTlzLUmSJOXEcC1JkiTlxGUhuyCXkUgSMP1bpa6g6QwcWeoKpF2GM9eSJElSTpy5Vp2c6ZYkSSqMM9eSJElSTgzXkiRJUk5cFqKdmktaJIld6+RN8AROlZQz15IkSVJOnLmWJElqyXalv0y0gL9KNLtwHRGDge8CrYB7Ukq3lrgkSZLUkuxKYVPNTrNaFhIRrYDvAycChwBnR8Qhpa1KkiRJKkxzm7k+HHgppbQMICIeAE4FFpa0KtVLfU8irMuKv75XlH4lSZLy1qxmroFuwGs13q/ItkmSJEnNXqSUSl1DtYg4EzghpfTF7P3ngcNTSl+p0WY4MDx7eyDwYpMXWqUL8GaJvntX5riXjmNfGo57aTjupePYl4bjXrd/TSl1ratRc1sWsgL4aI333YGVNRuklMYCY5uyqNpExKyUUmWp69jVOO6l49iXhuNeGo576Tj2peG456e5LQv5A9AzInpERFtgKPBoiWuSJEmSCtKsZq5TShsj4mLg11Rdiu/elNKCEpclSZIkFaRZhWuAlNIUYEqp6yhAyZem7KIc99Jx7EvDcS8Nx710HPvScNxz0qxOaJQkSZJasua25lqSJElqsQzXW4mIwRHxYkS8FBEjavn8QxExIfv82Ygoq/HZyGz7ixFxQlPWvTNo6NhHRFlEvBcRc7PH3U1de0tWwLj/34iYExEbI+KMrT4bFhFLssewpqt659DIsd9U45j3xO96KGDcr4iIhRExLyKejoh/rfGZx3wDNXLcPd4boYCx/1JEvJCN7+9q3h3bbNMAKSUf2YOqkyiXAvsDbYE/Aods1eY/gLuz10OBCdnrQ7L2HwJ6ZP20KvVvaimPRo59GTC/1L+hJT4KHPcyoDdwP3BGje2dgGXZc8fsdcdS/6aW8mjM2GefvVPq39ASHwWO+0Cgffb6yzX+b43HfAnGPXvv8V7csd+zxutTgCey12abBjycud5S9e3XU0rvA5tvv17TqcC47PVEYFBERLb9gZTS31NKLwMvZf2pMI0ZezVcneOeUlqeUpoHfLDVvicAT6WU3kop/RV4ChjcFEXvJBoz9mq4QsZ9ekppffZ2JlX3XACP+cZozLircQoZ+7/VeLsHsPmEPLNNAxiut1TI7der26SUNgLrgM4F7qvta8zYA/SIiOcj4jcR8aliF7sTacxx6zHfOI0dv3YRMSsiZkbEafmWtlOr77hfCDzewH31T40Zd/B4b4yCxj4i/jMilgLfBi6pz77aUrO7FF+J1TYLuvXlVLbXppB9tX2NGfvXgY+llNZExGHA5IjotdW/xFW7xhy3HvON09jx+1hKaWVE7A9Mi4gXUkpLc6ptZ1bwuEfEuUAlcEx999U2GjPu4PHeGAWNfUrp+8D3I+JzwPXAsEL31Zacud5Snbdfr9kmIloDewFvFbivtq/BY5/9uWoNQEppNlVrwg4oesU7h8Yctx7zjdOo8UsprcyelwEzgL55FrcTK2jcI+I44DrglJTS3+uzr2rVmHH3eG+c+h63DwCb/zrgMd8AhustFXL79Uep+tccwBnAtFS16v9RYGh2RYseQE/guSaqe2fQ4LGPiK4R0Qogm9XoSdWJRqpbIeO+Pb8Gjo+IjhHRETg+26bCNHjsszH/UPa6C3AUsLBole5c6hz3iOgL/IiqgPdGjY885huuwePu8d5ohYx9zxpvhwBLstdmm4Yo9RmVze0BnAT8iarZz+uybTdT9T92gHbAf1O1qP85YP8a+16X7fcicGKpf0tLezR07IHTgQVUndE8B/i3Uv+WlvQoYNz7UTV78S6wBlhQY98Lsv8eLwFfKPVvaWmPho49cCTwQnbMvwBcWOrf0pIeBYz7VGAVMDd7PFpjX4/5Jh53j/cmGfvvZv9/dC4wHehVY1+zTT0f3qFRkiRJyonLQiRJkqScGK4lSZKknBiuJUmSpJwYriVJkqScGK4lSZKknBiuJakBIqJzRMzNHn+JiD/XeN+2Hv1cEBH/Uo9J1gMBAAADo0lEQVT2x0ZE//q2y25tfE5e7RsjIlpHxNpi9C1JpebtzyWpAVLVXUErACLia8A7KaXRDejqAqquz/6XAtsfC7wJzKxPu1R1a+M820uSauHMtSTlLCKGRcRz2Sz2DyJit2y29mcR8UJEzI+ISyLiLKoC+oTaZrwj4vKIWBgRf4yI/4qI/wN8Ebgqa39kRJwaEc9GxPMR8WREfGQ77W6JiMvq0W/N9gdExLSs/ZyIKNuqztsjYniN97dExKURsWe235yImBcRJ9cyVsdFxOQa7++OiHOz1/0i4jcRMTsiHo+IffL47yNJxeTMtSTlKCLKgc8AR6aUNkbEWKpuN7wU6JJS+kTWbu+U0tqI+ApwcUppbi3dXQ38a0rp/Rrt7wHeTCmNyfrpSNWd7FJEfAn4akrpmlranVTPfmu2Hw98LaX0y4hox7YTMw8AtwJjs/dnAgOB94BTU0pvR8RHgGeAXxU4jh+i6q5xp6SU3syWqHwDGL7jPSWptAzXkpSv46i6bfmsiADYHXgN+DVwYER8F5gCPFlAXwuA/4qIR4DJ22nzMeDBbN32h6i6xXEe/QLV4b1LSumXACmlDVu3SSn9ISI+ms0sdwf+klJamc3E3xYRRwMfAB+NiC5AIeutDwZ6AVOzcWxF1a3gJalZM1xLUr4CuDeldMM2H0T0Bk4ELgFOp+5Z2BOAY4BTgeuzWfGtfR/4ZkppSkQcB4wooMZC+q0pFdDnQ1T9pjKqZrIBzgP2Ag7NZvFXAO222m8jW86Eb/48gHkppU8V8N2S1Gy45lqS8jUV+Gw2Q7v5qiIfi4iuQKSU/hu4CTg0a/820GHrTiKiFdA9pTQNuAroCrSvpf1ewJ+janp3WI3tje0XgJTSX4E3I+Lfsv3bRUT7Wn73A1Qtf/l3qoL25treyIL1p4Futez3CtArItpms+THZtsXAt0i4vDse9tGRK9a9pekZsVwLUk5Sim9AHydquUM86ha/rEP8FHgtxExF/gxcG22y0+Be2o5obE18IusjznAbSmlt4FHqArvz0fEkcDXgEnAb4BVNfbful19+63pHOCr2T6/oyqQb/27/5htfzml9Ea2+WfAkRExi6p12Etq2e9lqpamvADcn9VESunvwBnAHRHxR+B54JNb7y9JzU2kVMhf+yRJkiTVxZlrSZIkKSeGa0mSJCknhmtJkiQpJ4ZrSZIkKSeGa0mSJCknhmtJkiQpJ4ZrSZIkKSeGa0mSJCkn/x9JGLd/rQ2HnwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "np.random.seed(0)\n",
    "\n",
    "# we make a model where the null distribution is false\n",
    "shift = .3\n",
    "N = 200\n",
    "X = np.random.randn(N)\n",
    "Y = np.random.randn(N)+shift\n",
    "\n",
    "statistics_null = two_sample_permutation_test(simple_statistic_squared, X, Y, num_permutations)\n",
    "my_statistic = simple_statistic_squared(X,Y) # this is a single sample from the alternative\n",
    "\n",
    "statistics_alt = np.zeros(num_permutations)\n",
    "for i in tqdm(range(num_permutations)):\n",
    "    # IMPLEMENT: generate more data from the alternative\n",
    "    X = np.random.randn(N)\n",
    "    Y = np.random.randn(N)+shift\n",
    "    \n",
    "    statistics_alt[i] = simple_statistic_squared(X,Y)\n",
    "    \n",
    "plt.figure(figsize=(12,4))\n",
    "plot_permutation_samples(statistics_null, my_statistic)    \n",
    "plt.hist(statistics_alt, alpha=0.5);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, even though most of the alternative distribution far larger than the null, we can be unlucky: there are datasets for which the test would not reject the null simply by chance -- we only have a single draw from the alternative.\n",
    "With this fixed seed, it turns out that we have been unlucky and would have not rejected the null hypothesis even it is not true.\n",
    "You could play with the sample size and observe how this impacts the position of the alternative distribution (and as such the test power).\n",
    "\n",
    "Let's try the test on another dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Means: -0.05499658 0.027213123\n",
      "Standard deviations: 1.046988 0.9609663\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAEDFJREFUeJzt3X+sX3V9x/Hna1TEnwHhYlhbVlyqmyP+IDekjGQRkK0ooWyRBOK00y7NMtwQXYSORLIsRoybuGWbSyeMmjGUoQaiOO2whiyhaEFEoPxoalauoL2K+GMmuup7f9xTc6m39/Z+z/f2++2H5yO5+X7P53zO+bxT2hef+/mec76pKiRJ7fqVURcgSVpaBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcctGXQDACSecUKtWrRp1GZJ0RLnnnnu+U1UTC/Ubi6BftWoVO3bsGHUZknRESfI/h9LPpRtJapxBL0mNM+glqXEGvSQ1bsGgT3J9kr1JHphj318kqSQndNtJ8vdJdiW5P8lpS1G0JOnQHcqM/gZg7YGNSVYC5wJ7ZjWfB6zufjYCH+lfoiSpjwWDvqruBJ6aY9e1wHuA2V9RtQ74WM3YDhyb5KShVCpJGshAa/RJLgC+WVVfO2DXcuDxWdtTXdtc59iYZEeSHdPT04OUIUk6BIsO+iTPB64C3jvX7jna5vxS2qraXFWTVTU5MbHgjV2SpAENcmfsrwOnAF9LArACuDfJ6czM4FfO6rsCeKJvkRLb3j+6sc/aNLqxpSFY9Iy+qr5eVSdW1aqqWsVMuJ9WVd8CbgPe2l19swb4flU9OdySJUmLcSiXV94E3AW8IslUkg3zdL8d2A3sAv4F+NOhVClJGtiCSzdVdckC+1fNel/Apf3LkiQNi3fGSlLjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQsGfZLrk+xN8sCstg8meTjJ/Uk+neTYWfs2JdmV5JEkv7dUhUuSDs2hzOhvANYe0LYVOLWqXgU8CmwCSPJK4GLgt7pj/inJUUOrVpK0aAsGfVXdCTx1QNsXqmpft7kdWNG9Xwd8vKp+UlXfAHYBpw+xXknSIg1jjf7twOe698uBx2ftm+raJEkj0ivok1wF7ANu3N80R7c6yLEbk+xIsmN6erpPGZKkeQwc9EnWA+cDb66q/WE+Bayc1W0F8MRcx1fV5qqarKrJiYmJQcuQJC1goKBPsha4Arigqn48a9dtwMVJnpvkFGA18OX+ZUqSBrVsoQ5JbgJeB5yQZAq4mpmrbJ4LbE0CsL2q/qSqHkxyM/AQM0s6l1bVz5aqeEnSwhYM+qq6ZI7m6+bp/z7gfX2KkiQNj3fGSlLjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDVuwaBPcn2SvUkemNX2kiRbkzzWvR7XtSfJ3yfZleT+JKctZfGSpIUdyoz+BmDtAW1XAndU1Wrgjm4b4DxgdfezEfjIcMqUJA1qwaCvqjuBpw5oXgds6d5vAS6c1f6xmrEdODbJScMqVpK0eIOu0b+0qp4E6F5P7NqXA4/P6jfVtUmSRmTYH8Zmjraas2OyMcmOJDump6eHXIYkab9Bg/7b+5dkute9XfsUsHJWvxXAE3OdoKo2V9VkVU1OTEwMWIYkaSGDBv1twPru/Xrg1lntb+2uvlkDfH//Eo8kaTSWLdQhyU3A64ATkkwBVwPXADcn2QDsAS7qut8OvAHYBfwYeNsS1CxJWoQFg76qLjnIrnPm6FvApX2LkiQNj3fGSlLjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDWuV9AnuTzJg0keSHJTkmOSnJLk7iSPJflEkqOHVawkafEGDvoky4E/Byar6lTgKOBi4APAtVW1GvgesGEYhUqSBtN36WYZ8Lwky4DnA08CZwO3dPu3ABf2HEOS1MPAQV9V3wT+BtjDTMB/H7gHeLqq9nXdpoDlfYuUJA1u2aAHJjkOWAecAjwN/Adw3hxd6yDHbwQ2Apx88smDlqHGXLv10Tnb1+z57tDHOuNlxw/9nNI46rN083rgG1U1XVX/B3wK+G3g2G4pB2AF8MRcB1fV5qqarKrJiYmJHmVIkubTJ+j3AGuSPD9JgHOAh4BtwJu6PuuBW/uVKEnqo88a/d3MfOh6L/D17lybgSuAdyXZBRwPXDeEOiVJAxp4jR6gqq4Grj6geTdwep/zSpKGxztjJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY3rdcOUdCS7a/ehPSht+765H7S2GJef+/Le55AG5Yxekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY3rFfRJjk1yS5KHk+xMckaSlyTZmuSx7vW4YRUrSVq8vjP6vwP+s6p+A3g1sBO4ErijqlYDd3TbkqQRGTjok7wY+B3gOoCq+mlVPQ2sA7Z03bYAF/YtUpI0uD4z+pcB08C/Jvlqko8meQHw0qp6EqB7PXGug5NsTLIjyY7p6ekeZUiS5tMn6JcBpwEfqarXAv/LIpZpqmpzVU1W1eTExESPMiRJ8+kT9FPAVFXd3W3fwkzwfzvJSQDd695+JUqS+hj4qwSr6ltJHk/yiqp6BDgHeKj7WQ9c073eOpRKpRFZs2dz/5NsO37xx5y1qf+4Ev2/M/bPgBuTHA3sBt7GzG8JNyfZAOwBLuo5hiSph15BX1X3AZNz7Dqnz3klScPjnbGS1DiDXpIaZ9BLUuMMeklqnEEvSY3re3mlnm22vX9JT79mz3eX9PzSs5EzeklqnEEvSY0z6CWpcQa9JDXOoJekxnnVjRZ07dZHf/Heq2KkI48zeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjvGFKOgzu2r34G82273t04U5zuPzclw90nNrVe0af5KgkX03ymW77lCR3J3ksySeSHN2/TEnSoIaxdHMZsHPW9geAa6tqNfA9YMMQxpAkDahX0CdZAbwR+Gi3HeBs4Jauyxbgwj5jSJL66Tuj/zDwHuDn3fbxwNNVta/bngKWz3Vgko1JdiTZMT093bMMSdLBDBz0Sc4H9lbVPbOb5+hacx1fVZurarKqJicmJgYtQ5K0gD5X3ZwJXJDkDcAxwIuZmeEfm2RZN6tfATzRv0xJ0qAGntFX1aaqWlFVq4CLgS9W1ZuBbcCbum7rgVt7VylJGthS3DB1BfCuJLuYWbO/bgnGkCQdoqHcMFVVXwK+1L3fDZw+jPNKkvrzEQiS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWrcwEGfZGWSbUl2JnkwyWVd+0uSbE3yWPd63PDKlSQt1rIex+4D3l1V9yZ5EXBPkq3AHwF3VNU1Sa4ErgSu6F+qZrt266OjLkHSEWLgGX1VPVlV93bvfwjsBJYD64AtXbctwIV9i5QkDW4oa/RJVgGvBe4GXlpVT8LM/wyAE4cxhiRpMH2WbgBI8kLgk8A7q+oHSQ71uI3ARoCTTz65bxlSc9bs2TzYgduO7zfwWZv6Ha+x02tGn+Q5zIT8jVX1qa7520lO6vafBOyd69iq2lxVk1U1OTEx0acMSdI8+lx1E+A6YGdVfWjWrtuA9d379cCtg5cnSeqrz9LNmcBbgK8nua9r+0vgGuDmJBuAPcBF/UqUJPUxcNBX1X8DB1uQP2fQ80qShss7YyWpcQa9JDWu9+WVksbLXbu/2+v47fsO/a7ry899ea+xdHg4o5ekxhn0ktQ4l26GzIeNSRo3zuglqXEGvSQ1zqWbI9DAD7uS9KzkjF6SGmfQS1LjDHpJapxBL0mNM+glqXFedSPpGRZ1VVffry08kF9juCSc0UtS4wx6SWqcSzeSBtb3kcgHmu8RyT4SeXDO6CWpcc7oJY2NeT8IHvYHv7M1/iHwkR/0294/6gqeYc2e4f4qK0l9LdnSTZK1SR5JsivJlUs1jiRpfksyo09yFPCPwLnAFPCVJLdV1UNLMZ4k9TLKlYHDsGy0VEs3pwO7qmo3QJKPA+sAg17SQIZ9hc+4OOOspR9jqZZulgOPz9qe6tokSYfZUs3oM0dbPaNDshHY2G3+KMkjS1TLoE4AvjPqIuZhfYMb59rA+voa5/p+ubY//ts+5/u1Q+m0VEE/Bayctb0CeGJ2h6raDIztVyUl2VFVk6Ou42Csb3DjXBtYX1/jXN+oaluqpZuvAKuTnJLkaOBi4LYlGkuSNI8lmdFX1b4k7wA+DxwFXF9VDy7FWJKk+S3ZDVNVdTtw+1Kd/zAY22WljvUNbpxrA+vra5zrG0ltqaqFe0mSjlg+1EySGmfQzyPJXye5P8l9Sb6Q5FdHXdNsST6Y5OGuxk8nOXbUNe2X5KIkDyb5eZKxuQJinB/NkeT6JHuTPDDqWg6UZGWSbUl2dv9dLxt1TbMlOSbJl5N8ravvr0Zd01ySHJXkq0k+czjHNejn98GqelVVvQb4DPDeURd0gK3AqVX1KuBRYJwewfcA8AfAnaMuZL9Zj+Y4D3glcEmSV462qme4AVg76iIOYh/w7qr6TWANcOmY/dn9BDi7ql4NvAZYm2TNiGuay2XAzsM9qEE/j6r6wazNF3DATV+jVlVfqKp93eZ2Zu5XGAtVtbOqxu0muF88mqOqfgrsfzTHWKiqO4GnRl3HXKrqyaq6t3v/Q2bCamzudq8ZP+o2n9P9jNW/1yQrgDcCHz3cYxv0C0jyviSPA29m/Gb0s70d+NyoixhzPppjCJKsAl4L3D3aSp6pWxa5D9gLbK2qsaoP+DDwHuDnh3vgZ33QJ/mvJA/M8bMOoKquqqqVwI3AO8atvq7PVcz8an3juNU2ZhZ8NIfml+SFwCeBdx7wG+/IVdXPumXWFcDpSU4ddU37JTkf2FtV94xi/CP/i0d6qqrXH2LXfwc+C1y9hOX8koXqS7IeOB84pw7ztbKL+LMbFws+mkMHl+Q5zIT8jVX1qVHXczBV9XSSLzHzece4fLB9JnBBkjcAxwAvTvJvVfWHh2PwZ/2Mfj5JVs/avAB4eFS1zCXJWuAK4IKq+vGo6zkC+GiOASUJcB2ws6o+NOp6DpRkYv9VZ0meB7yeMfr3WlWbqmpFVa1i5u/dFw9XyINBv5BruqWI+4HfZeYT83HyD8CLgK3dJaD/POqC9kvy+0mmgDOAzyb5/Khr6j643v9ojp3AzeP0aI4kNwF3Aa9IMpVkw6hrmuVM4C3A2d3ftfu62em4OAnY1v1b/Qoza/SH9RLGceadsZLUOGf0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMb9P4qCFxL8Tx8mAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XmUFOXZ/vHvLYuAIg4w+qKEDLwBRUZodCAI5ie4gUJcgkY9rtFIYl7jviC4KznoQUWC0aASwETEgKgR3BCISkQERGRTZFMEQcaAIqKC9++PqhmbYZaeoau7oa7POXO6u+p5qu6pWa6upZ8yd0dEROJrr2wXICIi2aUgEBGJOQWBiEjMKQhERGJOQSAiEnMKAhGRmFMQiIjEnIJARCTmFAQiIjFXO9sFpKJp06ZeUFCQ7TJS9sEHweMhh2S3jrTYo74ZkdTtCb/6c+bM2eDu+VW12y2CoKCggNmzZ2e7jJR17x48Tp+ezSrSZI/6ZkRStyf86pvZqlTa6dCQiEjMKQhERGJOQSAiEnO7xTkCEdnR999/z+rVq9m6dWu2S9lj3XZb8Lh4cXbrSEW9evVo3rw5derUqVF/BYHIbmj16tU0bNiQgoICzCzb5eyR9gqPl+T6VUPuTnFxMatXr6Zly5Y1WoYODYnshrZu3UqTJk0UAoKZ0aRJk13aO1QQiOymFAJSYld/FxQEIiIxpyAQkRp58MEHKSwspF27dgwdOrR0+u23387BBx9MIpEgkUgwefJkAGbMmEH79u3p1KkTH330EQAbN26kZ8+eZPPe6c8++yyLFi0qfX3rrbcyZcoUAM4/v/tu9WHWmtLJYpFdUNB/UlbW++gpzWiblTUHFixYwKOPPsqsWbOoW7cuvXr1onfv3rRu3RqAq6++muuuu26HPvfddx8TJkxg5cqVPPzww9x3333cddddDBgwIKuHuZ599ln69OnDYYcdBsCdd94J/DjERBxoj0BEqm3x4sV06dKFBg0aULt2bY455hgmTpxYaZ86derwzTffsGXLFurUqcOyZcv49NNPOeaYYyrs89JLL3HooYdy9NFHc8UVV9CnTx8g2OsYMmRIabvCwkJWrlwJwGmnncaRRx5Ju3btGDFiRGmbfffdl4EDB9KhQwe6dOnCunXr+M9//sPzzz/P9ddfTyKRYNmyZVx00UWMHz9+p1peeeUVjjrqKI444gjOPPNMNm/eDED//v057LDDaN++/U7ht7uIbI/AzOoBrwN7h+sZ7+63mdko4BhgU9j0InefF1UdInu6q66CeWn+C0okIOloz04KCwsZOHAgxcXF1K9fn8mTJ1NUVFQ6f/jw4YwZM4aioiLuu+8+8vLyuOmmm+jXrx/169fniSee4LrrruOuu+6qcB1bt27l0ksvZerUqfzsZz/jrLPOSqn2kSNH0rhxY7755hs6depE3759adKkCV9//TVdunRh0KBB3HDDDTz66KPcfPPNnHLKKfTp04czzjijwmVu2LCBu+++mylTprDPPvtwzz33cP/993P55ZczceJElixZgpmxcePGlGrMNVHuEXwLHOvuHYAE0MvMuoTzrnf3RPilEBDZzbRt25Ybb7yRE044gV69etGhQwdq1w7eV1522WUsW7aMefPm0axZM6699loAEokEM2fOZNq0aSxfvpyDDjoId+ess87ivPPOY926dTusY8mSJbRs2ZLWrVtjZpx33nkp1TZs2LDSd/2ffPIJS5cuBaBu3bqlexRHHnlk6R5EKmbOnMmiRYvo1q0biUSC0aNHs2rVKvbbbz/q1avHb3/7W5555hkaNGiQ8jJzSWR7BB6c/dkcvqwTfmXvjJDIHqqyd+5RuuSSS7jkkksAGDBgAM2bNwfgwAMPLG1z6aWXlv7zLeHu3H333YwbN47LL7+cO+64g5UrVzJs2DAGDRq0Q9uKzh3Url2bH374ofR1yTX006dPZ8qUKbz11ls0aNCA7t27l86rU6dO6fJq1arFtm3bUv5e3Z0TTjiBsWPH7jRv1qxZvPbaazz11FMMHz6cqVOnprzcXBHpOQIzq2Vm84D1wKvu/nY4a5CZzTezB8xs7wr69jOz2WY2+/PPP4+yTBGpgfXr1wPw8ccf88wzz3DOOecAsHbt2tI2EydOpLCwcId+o0ePpnfv3uTl5bFlyxb22msv9tprL7Zs2bJDu0MPPZQVK1awbNkygB3+CRcUFDB37lwA5s6dy4oVKwDYtGkTeXl5NGjQgCVLljBz5swqv4+GDRvy1VdfVdqmS5cuzJgxo/Rqpy1btvDhhx+yefNmNm3axMknn8zQoUOZl+5jdBkS6VVD7r4dSJjZ/sBEMysEbgI+A+oCI4AbgTvL6TsinE9RUZH2JERyTN++fSkuLqZOnTo89NBD5OXlAXDDDTcwb948zIyCggL++te/lvbZsmULo0eP5pVXXgHgmmuuoW/fvtStW3end9v16tVjxIgR9O7dm6ZNm3L00UezYMGC0nWPGTOGRCJBp06daNOmDQC9evXikUceoX379hxyyCF06dKFqpx99tlceumlDBs2rNyTxAD5+fmMGjWKc845h2+//RaAu+++m4YNG3LqqaeydetW3J0HHnigmlsxN1imrt81s9uAr919SNK07sB17t6nwo4EQbA7Xcu7J9zQotQe9c2kXzYvHz2h6xFZWXe2TJ8+nSFDhvDCCy9kZH272x3KFi9eTNu2O15UbGZz3L2ogi6lIjs0ZGb54Z4AZlYfOB5YYmbNwmkGnAYsiKoGERGpWpSHhpoBo82sFkHgPO3uL5jZVDPLBwyYB/w+whpEZA/RvXt3upfsoUpaRXnV0HygYznTj41qnSIiUn36ZLGISMwpCEREYk5BICIScwoCEamxiRMnYmYsWbKkyrajRo1izZo1NV7X9OnTd/qUMsC8efNKh7quro0bN/KXv/ylWn2GDh2604ffUml38sknVzoWUXXbp5OCQERqbOzYsRx99NE89dRTVbbd1SCoyO4SBJMnT2b//fdPW/t0UhCISI1s3ryZGTNm8Pjjj+8UBPfeey+HH344HTp0oH///owfP57Zs2dz7rnnkkgk+OabbygoKGDDhg0AzJ49u/TS0FmzZtG1a1c6duxI165d+aCSGwN899133HrrrYwbN45EIsG4ceP4+uuvufjii+nUqRMdO3bkueeeA2DhwoV07tyZRCJB+/btWbp0Kf3792fZsmUkEgmuv/76HZa9ZcvX/O53venQoQOFhYWMGzeOYcOGsWbNGnr06EGPHj2AYJC9oqIi2rVrx2233QZQbruS7/frr7+md++ql5u8fcaMGUP79u3p0KED559//q782MqlG9PIHiFbn/DNCdkYh5rghi69evWiTZs2NG7cmLlz53LEEUfw4osv8uyzz/L222/ToEEDvvjiCxo3bszw4cMZMmTIDsNVl+fQQw/l9ddfp3bt2kyZMoUBAwYwYcKEctvWrVuXO++8k9mzZzN8+HAgGADv2GOPZeTIkWzcuJHOnTtz/PHH88gjj3DllVdy7rnn8t1337F9+3YGDx7MggULyh0j6I03XuKAAw5i+vTgd2vTpk00atSI+++/n2nTptG0aVMABg0aROPGjdm+fTvHHXcc8+fP54orrtipXYmXXnqJgw46iEmTKl9uiYULFzJo0CBmzJhB06ZN+eKLLyrdfjWhPQIRqZGxY8dy9tlnA8F4PSVjBU2ZMoXf/OY3pUMyN27cuFrL3bRpE2eeeSaFhYVcffXVLFy4sFr9X3nlFQYPHkwikSgdffTjjz/mqKOO4k9/+hP33HMPq1aton79+pUup02bw/nPf6Zw44038sYbb9CoUaNy2z399NMcccQRdOzYkYULF+5w28vyHH744UyZUvVyS0ydOpUzzjijNCCquz1ToT0Ckd1dFsahLi4uZurUqSxYsAAzY/v27ZgZ9957L+6e0q0nk4eSLhkqGuCWW26hR48eTJw4kZUrV1b708TuzoQJEzikzCBBbdu25ec//zmTJk2iZ8+ePPbYY7Rq1arC5bRs2YYJE+awdOlkbrrpJk488URuvfXWHdqsWLGCIUOG8M4775CXl8dFF120w/dSnjZt2jBnzhwmT654uWW/n6hv5ak9AhGptvHjx3PBBRewatUqVq5cySeffELLli158803OfHEExk5cmTpic+SQxllh3suKChgzpw5ADsc+tm0aRMHH3wwEJxgrkrZ5fbs2ZM///nPlAyo+e677wKwfPlyWrVqxRVXXMEpp5zC/PnzKx2Cet26NdSv34DzzjuP6667rnTY6+Q+X375Jfvssw+NGjVi3bp1vPjiixXWVWLNmjU0aFD5cpMdd9xxPP300xQXFwPo0JCI5IaxY8dy+umn7zCtb9++PPnkk/Tq1YtTTjmFoqIiEolE6b2FL7roIn7/+9+Xniy+7bbbuPLKK/nFL35BrVq1Spdzww03cNNNN9GtWze2b99eZS09evRg0aJFpSeLb7nlFr7//nvat29PYWEht9xyCwDjxo2jsLCQRCLBkiVLuOCCC2jSpAndunWjsLBwp5PFH374PmeeGZxcHjRoEDfffDMA/fr146STTqJHjx506NCBjh070q5dOy6++GK6detW2j+5XbL333+/9KR1RctN1q5dOwYOHMgxxxxDhw4duOaaa6rcJtWVsWGod4WGoc6i3eSbidvJ4jgOQ51pcRqGWucIJG3i9s9YZE+hQ0MiIjGnIBDZDTnO7nBYVzJjV38XFAQiu6FVG7+nuLhYYSC4O8XFxdSrV6/Gy9A5ApHd0J/f/i/Ht/2Kzz//PNul7LE++yx4DD/qkNPq1atH8+bNa9xfQSCyG/ry2x9o2bJltsvYo112WfCY4xfMpYUODYmIxJyCQEQk5iILAjOrZ2azzOw9M1toZneE01ua2dtmttTMxplZ3ahqEBGRqkW5R/AtcKy7dwASQC8z6wLcAzzg7q2B/wKXRFiDiIhUIbIg8MDm8GWd8MuBY4Hx4fTRwGlR1SAiIlWL9ByBmdUys3nAeuBVYBmw0d23hU1WAwdX0Lefmc02s9m6RE5EJDqRBoG7b3f3BNAc6Ay0La9ZBX1HuHuRuxfl5+dHWaaISKxl5Kohd98ITAe6APubWcnnF5oD6b+btYiIpCzKq4byzWz/8Hl94HhgMTANOCNsdiHwXFQ1iIhI1aL8ZHEzYLSZ1SIInKfd/QUzWwQ8ZWZ3A+8Cj0dYg4iIVCGyIHD3+UDHcqYvJzhfICIiOUCfLBYRiTkFgYhIzCkIRERiTkEgIhJzCgIRkZhTEIiIxJyCQEQk5hQEIiIxpyAQEYk5BYGISMwpCEREYk5BICIScwoCEZGYUxCIiMScgkBEJOYUBCIiMacgEBGJOQWBiEjMKQhERGIusiAws5+Y2TQzW2xmC83synD67Wb2qZnNC79OjqoGERGpWmQ3rwe2Ade6+1wzawjMMbNXw3kPuPuQCNctIiIpiiwI3H0tsDZ8/pWZLQYOjmp9IiJSMxk5R2BmBUBH4O1w0uVmNt/MRppZXiZqEBGR8kV5aAgAM9sXmABc5e5fmtnDwF2Ah4/3AReX068f0A+gRYsWUZcpstsp6D8pK+tdObh3VtYr0Yl0j8DM6hCEwD/c/RkAd1/n7tvd/QfgUaBzeX3dfYS7F7l7UX5+fpRliojEWpRXDRnwOLDY3e9Pmt4sqdnpwIKoahARkapFeWioG3A+8L6ZzQunDQDOMbMEwaGhlcDvIqxBRESqEOVVQ28CVs6syVGtU0REqk+fLBYRibnIrxqSzEvn1SRPLS8G4OwsXaEiItHTHoGISMwpCEREYk5BICIScwoCEZGYUxCIiMScgkBEJOYUBCIiMacgEBGJOQWBiEjMKQhERGJOQSAiEnMKAhGRmFMQiIjEXLWDwMzyzKx9FMWIiEjmpRQEZjbdzPYzs8bAe8DfzOz+qvqJiEjuS3WPoJG7fwn8Cvibux8JHB9dWSIikimpBkHt8KbzvwZeiLAeERHJsFSD4A7gZeAjd3/HzFoBS6MrS0REMiXVIFjr7u3d/Q8A7r4cqPQcgZn9xMymmdliM1toZleG0xub2atmtjR8zNu1b0FERHZFqkHw5xSnJdsGXOvubYEuwP+Z2WFAf+A1d28NvBa+FhGRLKn05vVmdhTQFcg3s2uSZu0H1Kqsr7uvBdaGz78ys8XAwcCpQPew2WhgOnBjDWoXEZE0qDQIgLrAvmG7hknTvwTOSHUlZlYAdATeBg4MQwJ3X2tmB1SjXhERSbNKg8Dd/w3828xGufuqmqzAzPYFJgBXufuXZpZqv35AP4AWLVrUZNUiIpKCqvYISuxtZiOAguQ+7n5sZZ3MrA5BCPzD3Z8JJ68zs2bh3kAzYH15fd19BDACoKioyFOsU0REqinVIPgn8AjwGLA9lQ4WvPV/HFjs7slXGD0PXAgMDh+fS7laERFJu1SDYJu7P1zNZXcDzgfeN7N54bQBBAHwtJldAnwMnFnN5YqISBqlGgT/MrM/ABOBb0smuvsXFXVw9zeBik4IHJdyhSIiEqlUg+DC8PH6pGkOtEpvOSIikmkpBYG7t4y6EBERyY6UgsDMLihvuruPSW85IiKSaakeGuqU9LwewTH+uYCCQERkN5fqoaE/Jr82s0bAE5FUJCIiGVXTexZvAVqnsxAREcmOVM8R/IvgKiEIBptrCzwdVVEikrsK+k/K2rpXDu6dtXXvyVI9RzAk6fk2YJW7r46gHhERybCUDg2Fg88tIRiBNA/4LsqiREQkc1IKAjP7NTCLYDiIXwNvm1nKw1CLiEjuSvXQ0ECgk7uvBzCzfGAKMD6qwkREJDNSvWpor5IQCBVXo6+IiOSwVPcIXjKzl4Gx4euzgMnRlCQiIplU1T2Lf0Zwa8nrzexXwNEEI4q+BfwjA/WJiEjEqjq8MxT4CsDdn3H3a9z9aoK9gaFRFyciItGrKggK3H1+2YnuPpvgtpUiIrKbqyoI6lUyr346CxERkeyoKgjeMbNLy04MbzM5J5qSREQkk6q6augqYKKZncuP//iLgLrA6VEWJiIimVFpELj7OqCrmfUACsPJk9x9auSViYhIRqR6P4JpwLTqLNjMRgJ9gPXuXhhOux24FPg8bDbA3fV5BBGRLIry08GjgF7lTH/A3RPhl0JARCTLIgsCd38d+CKq5YuISHpkY7ygy81svpmNNLO8LKxfRESSZDoIHgb+F0gAa4H7KmpoZv3MbLaZzf78888raiYiIrsoo0Hg7uvcfbu7/wA8CnSupO0Idy9y96L8/PzMFSkiEjMZDQIza5b08nRgQSbXLyIiO0t1GOpqM7OxQHegqZmtBm4DuptZAnBgJfC7qNYvIiKpiSwI3P2cciY/HtX6RESkZnSXMRGRmFMQiIjEnIJARCTmFAQiIjGnIBARiTkFgYhIzCkIRERiTkEgIhJzCgIRkZhTEIiIxJyCQEQk5hQEIiIxF9mgcwIF/SdluwQRkSppj0BEJOYUBCIiMacgEBGJOQWBiEjMKQhERGJOQSAiEnMKAhGRmIssCMxspJmtN7MFSdMam9mrZrY0fMyLav0iIpKaKPcIRgG9ykzrD7zm7q2B18LXIiKSRZEFgbu/DnxRZvKpwOjw+WjgtKjWLyIiqcn0EBMHuvtaAHdfa2YHVNTQzPoB/QBatGiRofJEJJdlctiWz5Z3Cdc5k5WDe2dsvdmQsyeL3X2Euxe5e1F+fn62yxER2WNlOgjWmVkzgPBxfYbXLyIiZWQ6CJ4HLgyfXwg8l+H1i4hIGVFePjoWeAs4xMxWm9klwGDgBDNbCpwQvhYRkSyK7GSxu59TwazjolqniIhUX86eLBYRkcxQEIiIxJyCQEQk5hQEIiIxpyAQEYk5BYGISMwpCEREYk5BICIScwoCEZGYUxCIiMScgkBEJOYUBCIiMacgEBGJOQWBiEjMKQhERGJOQSAiEnMKAhGRmIvsDmW5oqD/pIyv87PlXQD4ny4ZX7WISLVpj0BEJOayskdgZiuBr4DtwDZ3L8pGHSIikt1DQz3cfUMW1y8iIujQkIhI7GUrCBx4xczmmFm/LNUgIiJk79BQN3dfY2YHAK+a2RJ3fz25QRgQ/QBatGiRjRpFRGIhK3sE7r4mfFwPTAQ6l9NmhLsXuXtRfn5+pksUEYmNjAeBme1jZg1LngMnAgsyXYeIiASycWjoQGCimZWs/0l3fykLdYiICFkIAndfDnTI9HpFRKR8unxURCTm9vixhkREdlU2xiwrsXJw78jXoT0CEZGYUxCIiMScgkBEJOYUBCIiMacgEBGJOQWBiEjMKQhERGJOQSAiEnMKAhGRmFMQiIjEnIJARCTmFAQiIjGnIBARiTkFgYhIzCkIRERiTkEgIhJzCgIRkZhTEIiIxFxWgsDMepnZB2b2kZn1z0YNIiISyHgQmFkt4CHgJOAw4BwzOyzTdYiISCAbewSdgY/cfbm7fwc8BZyahTpERITsBMHBwCdJr1eH00REJAvM3TO7QrMzgZ7u/tvw9flAZ3f/Y5l2/YB+4ctDgA/KWVxTYEOE5daU6qq+XK1NdVVPrtYFuVtblHX91N3zq2pUO6KVV2Y18JOk182BNWUbufsIYERlCzKz2e5elN7ydp3qqr5crU11VU+u1gW5W1su1JWNQ0PvAK3NrKWZ1QXOBp7PQh0iIkIW9gjcfZuZXQ68DNQCRrr7wkzXISIigWwcGsLdJwOT07CoSg8dZZHqqr5crU11VU+u1gW5W1vW68r4yWIREcktGmJCRCTmcj4IzKyxmb1qZkvDx7xy2iTM7C0zW2hm883srKR5Lc3s7bD/uPAEdUbqCtu9ZGYbzeyFMtNHmdkKM5sXfiVypK5sb68LwzZLzezCpOnTw2FJSrbXAWmoqdKhTsxs73AbfBRuk4KkeTeF0z8ws567Wks66jKzAjP7JmkbPZLhuv6fmc01s21mdkaZeeX+XHOgru1J2yvtF62kUNs1ZrYo/L/1mpn9NGleZNtsJ+6e01/AvUD/8Hl/4J5y2rQBWofPDwLWAvuHr58Gzg6fPwJclqm6wnnHAb8EXigzfRRwRja2VxV1ZW17AY2B5eFjXvg8L5w3HShK43aqBSwDWgF1gfeAw8q0+QPwSPj8bGBc+PywsP3eQMtwObVyoK4CYEG6f6eqUVcB0B4Yk/y7XdnPNZt1hfM2R7G9qlFbD6BB+PyypJ9lZNusvK+c3yMgGH5idPh8NHBa2Qbu/qG7Lw2frwHWA/lmZsCxwPjK+kdVV1jPa8BXaVpnKmpcVw5sr57Aq+7+hbv/F3gV6JWm9ZeVylAnyTWPB44Lt9GpwFPu/q27rwA+CpeX7bqiVGVd7r7S3ecDP5TpG+XPdVfqiloqtU1z9y3hy5kEn6uCzP4t7BZBcKC7rwUIHys9JGBmnQnSdxnQBNjo7tvC2ekczqJadVVgULhL+ICZ7Z0DdWV7e1U1/Mjfwl34W9Lwjy+VoU5K24TbZBPBNopymJRdqQugpZm9a2b/NrNfpKmmVOuKom/Uy65nZrPNbKaZpetNT4nq1nYJ8GIN++6SrFw+WpaZTQH+p5xZA6u5nGbAE8CF7v5DBf8sUr5MKl11VeAm4DOC0BoB3AjcmeW6sr29Klv/ue7+qZk1BCYA5xPs6tdUKt9rRW12aTtVYVfqWgu0cPdiMzsSeNbM2rn7lxmqK4q+US+7hbuvMbNWwFQze9/dl2W6NjM7DygCjqlu33TIiSBw9+Mrmmdm68ysmbuvDf/Rr6+g3X7AJOBmd58ZTt4A7G9mtcN3TuUOZxFlXZUse2349Fsz+xtwXQ7Ule3ttRronvS6OcG5Adz90/DxKzN7kmC3e1eCIJWhTkrarDaz2kAj4IsU+2a8Lg8OLn8L4O5zzGwZwfmz2Rmqq7K+3cv0nZ6GmkqWXeOfRXgoGXdfbmbTgY4ERxMyVpuZHU/wZukYd/82qW/3Mn2np6munewOh4aeB0rOmF8IPFe2gQVXtkwExrj7P0umh38Y04AzKusfVV2VCf8ZlhyXPw1YkO26cmB7vQycaGZ5FlxVdCLwspnVNrOmAGZWB+jDrm+vVIY6Sa75DGBquI2eB84Or95pCbQGZu1iPbtcl5nlW3C/D8J3uK0JTjJmqq6KlPtzzXZdYT17h8+bAt2ARWmqK6XazKwj8FfgFHdPfnMU5TbbWVRnodP1RXDs8zVgafjYOJxeBDwWPj8P+B6Yl/SVCOe1Ivgj/Qj4J7B3puoKX78BfA58Q5DyPcPpU4H3Cf6h/R3YN0fqyvb2ujhc90fAb8Jp+wBzgPnAQuBB0nCVDnAy8CHBO8CB4bQ7Cf4oAeqF2+CjcJu0Suo7MOz3AXBSmn/na1QX0DfcPu8Bc4FfZriuTuHv0tdAMbCwsp9rtusCuoZ/g++Fj5eks64Ua5sCrOPH/1vPZ2Kblf3SJ4tFRGJudzg0JCIiEVIQiIjEnIJARCTmFAQiIjGnIBARiTkFgWSdmTVJGgHyMzP7NOl1yqOfmtnFZlbeJ5sran+smXWpbjsz+z8zOzdd7XdF+BmLjVEsW+IjJz5ZLPHm7sVAAsDMbicYEXJIDRZ1McH185+l2P5Ygk9Tz6xOO3d/KM3tRbJKewSS08Ix2WeFewd/MbO9wnfBT5jZ+2a2wMyusOAeFAlgXHl7EmZ2tQXjvr9nZn83s/8FfgtcH7bvamanWjC+/7tm9oqZHVBBu7vN7KpqLDe5fRszmxq2n2tJ9zgI599nZv2SXt9tZlea2X5hv7kWDFTYp5xtdbyZPZv0+hELxrDBzDpZMBDdHDN70cwOTMfPR/YM2iOQnGVmhcDpQFd332ZmIwg+pr8MaOruh4ft9nf3jWb2R+Byd59XzuJuAH7q7t8ltX8M2ODuQ8Pl5BF8stPN7PfAte5+YzntTq7mcpPbjwVud/d/mVk9dn4z9hQwmB/vY3smwZj13wCnejDW0gHADOAFUhAOo/AgwadZN4SHqe4C+lXeU+JCQSC57HiC4QFmWzCQbH2CoXlfBg4xsweBycArKSxrIfB3M3sOeLaCNi2Ap8PzDHsTDA2QjuUCpUHT1N3/BeDuW8u2cfd3zOwn4Tv25sBnHoyOWRe4x8yOJhhX/yfh+Djf+iFJAAABq0lEQVSpnB9oC7QDpoTbsRbBkAsigIJAcpsBI939lp1mmLUHTgKuIBhjp6p3tz0Jhvg9Fbg53Nso6yHgT+4+2YIRIXe6tWANl5sslTFdJhB8TwUEewgAFxCMMnpEuHe0mmDMoWTb2HEPo2S+AfPdPZ33J5A9iM4RSC6bAvzafhx5tImZtTCzfMA8GGn2NuCIsP1XQMOyC7FgRM7m7j4VuB7IBxqU074R8KkFb5uT7xG7q8sFwIM7TW0ws1+G/euZWYNyvu+nCA6B/YogFEpqWx+GwAmUf5OSVUA7M6sb7n0cG05fBBxswU2bCOe3K6e/xJSCQHKWu78P3EFwSGM+wSGgAwnGeH/dzOYBjwIDwi5/Ax4r52RxbeDJcBlzCe6X/BXBUNi/Dk8OdwVuJxjO/N8EI0KWKNuuustNdi5wbdjnTYLwKPt9vxdOX+E/Dk38BNDVzGYTnDdYWk6/FQSHp94nuFfD3HD6twTDVd9vZu8B7wI/L9tf4kujj4qIxJz2CEREYk5BICIScwoCEZGYUxCIiMScgkBEJOYUBCIiMacgEBGJOQWBiEjM/X9tnKtcO9bShAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data = np.load(\"almost_simple.npz\")\n",
    "X = data[\"X\"]\n",
    "Y = data[\"Y\"]\n",
    "plt.hist(X, alpha=0.5)\n",
    "plt.hist(Y, alpha=0.5)\n",
    "\n",
    "# IMPLEMENT: print mean and std deviation of data\n",
    "print(\"Means:\", np.mean(X), np.mean(Y))\n",
    "print(\"Standard deviations:\", np.std(X), np.std(Y))\n",
    "\n",
    "my_statistic = simple_statistic(X,Y)\n",
    "statistics = two_sample_permutation_test(simple_statistic, X, Y, num_permutations)\n",
    "\n",
    "plt.figure()\n",
    "plot_permutation_samples(statistics, my_statistic)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now for the squared test statistic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAHmBJREFUeJzt3XmYFOW59/HvLYsDBnFY9CDEDJygAiMz4IAI5AiiguLrEkDxdY1EXnOiuCGru5ALvFCRYGJQieBJWAKCRtFwEEgiCeCABFnDNuCIYRkDCkgUcr9/dM04QA30LN3VM/w+1zXXdFU/VXX348ivn6rup8zdEREROdopURcgIiKpSQEhIiKhFBAiIhJKASEiIqEUECIiEkoBISIioRQQIiISSgEhIiKhFBAiIhKqetQFlEeDBg08IyOj1NutXx/7fd55FVtP0lT6FyAnG/3JppZly5btdveGJ2pXqQMiIyOD3NzcUm/XpUvs98KFFVpO8lT6FyAnG/3JphYz2xpPO51iEhGRUAoIEREJpYAQEZFQlfoahKSeb775hvz8fA4ePBh1KZIC0tLSaNKkCVAj6lKkDBQQUqHy8/OpU6cOGRkZmFnU5UiE3J2CggLy8/OBplGXI2WgU0xSoQ4ePEj9+vUVDoKZUb9+fY0mKzEFhFQ4hYMU0t9C5aaAEBGRUAoIqXJeeOEFMjMzadWqFWPHji1a/8QTT9C4cWOys7PJzs5mzpw5ACxatIjWrVvTrl07Nm7cCMCePXvo3r07Ud6zffbs2axZs6Zo+bHHHmPevHkAdOnSpUxfEhUpjZP2IvXizQVkDFkcybHzRvWM5Lgng1WrVvHyyy+zdOlSatasSY8ePejZsyfNmzcH4IEHHmDgwIFHbPPss88yc+ZM8vLy+OUvf8mzzz7L008/zbBhwyI9RTJ79myuvvpqWrZsCcBTTz0VWS1yctIIQqqUtWvX0qFDB2rXrk316tW55JJLmDVr1nG3qVGjBl999RUHDhygRo0abNq0iU8//ZRLLrmkxG3ee+89zj//fDp37syAAQO4+uqrgdgoZcyYMUXtMjMzycvLA+C6667jwgsvpFWrVkyYMKGozXe+8x2GDx9OVlYWHTp0YMeOHfzlL3/hrbfe4uGHHyY7O5tNmzZxxx13MGPGjGNqmTt3LhdffDFt27alT58+7Nu3D4AhQ4bQsmVLWrdufUwoisTjpB1BSOLdfz+sWFGx+8zOhmJnjY6RmZnJ8OHDKSgooFatWsyZM4ecnJyi58ePH8/kyZPJycnh2WefJT09naFDh9K/f39q1arF66+/zsCBA3n66adLPMbBgwe56667mD9/Pt///ve58cYb46p94sSJ1KtXj6+++op27drRq1cv6tevz/79++nQoQMjR45k0KBBvPzyyzzyyCNcc801XH311fTu3bvEfe7evZsRI0Ywb948TjvtNEaPHs1zzz3HPffcw6xZs1i3bh1mxp49e+KqUaQ4jSCkSmnRogWDBw/m8ssvp0ePHmRlZVG9eux90E9+8hM2bdrEihUraNSoEQ899BAA2dnZLF68mAULFrB582bOPvts3J0bb7yRW265hR07dhxxjHXr1tG0aVOaN2+OmXHLLbfEVdu4ceOKRgmffPIJGzZsAKBmzZpFI5ALL7ywaMQRj8WLF7NmzRo6depEdnY2kyZNYuvWrZx++umkpaXx4x//mDfeeIPatWvHvU+RQhpBSMIc751+IvXr149+/foBMGzYsOCbvHDWWWcVtbnrrruK/lEu5O6MGDGCadOmcc899/Dkk0+Sl5fHuHHjGDly5BFtS7o2Ub16df79738XLRd+B2DhwoXMmzePv/71r9SuXZsuXboUPVejRo2i/VWrVo1Dhw7F/Vrdncsvv5wpU6Yc89zSpUt5//33mTp1KuPHj2f+/Plx71cENIKQKmjnzp0AbNu2jTfeeIObbroJgM8++6yozaxZs8jMzDxiu0mTJtGzZ0/S09M5cOAAp5xyCqeccgoHDhw4ot3555/Pli1b2LRpE8AR/zhnZGSwfPlyAJYvX86WLVsA2Lt3L+np6dSuXZt169axePGJPyBRp04dvvzyy+O26dChA4sWLSr69NWBAwf4+9//zr59+9i7dy9XXXUVY8eOZUVFn+uTk4JGEFLl9OrVi4KCAmrUqMGLL75Ieno6AIMGDWLFihWYGRkZGfzqV78q2ubAgQNMmjSJuXPnAvDggw/Sq1cvatasecy787S0NCZMmEDPnj1p0KABnTt3ZtWqVUXHnjx5MtnZ2bRr145zzz0XgB49evDSSy/RunVrzjvvPDp06HDC19G3b1/uuusuxo0bF3pxGqBhw4a89tpr3HTTTfzrX/8CYMSIEdSpU4drr72WgwcP4u48//zzpexFEbAoP+ddXjk5OV7WGwYt3lzAf/zfSvox1xS++8ratWtp0aJF1GUk1cKFCxkzZgxvv/121KWkpLVr1/KTn8T+JlLwT/akZGbL3D3nRO10iklERELpFJNIOXXp0oUuhaM6kSpEIwgREQmlgBARkVAKCBERCaWAEBGRUAoIqZJmzZqFmbFu3boTtn3ttdfYvn17mY+1cOHCY76VDbBixYqiKcVLa8+ePfziF78o1TZjx4495kt98bS76qqrjjtXU2nbS9WhgJAqacqUKXTu3JmpU6eesG15A6IklSUg5syZwxlnnFFh7aXqUEBIlbNv3z4WLVrEq6++ekxAPPPMM1xwwQVkZWUxZMgQZsyYQW5uLjfffDPZ2dl89dVXZGRksHv3bgByc3OLPsK6dOlSOnbsSJs2bejYsSPr168vsYavv/6axx57jGnTppGdnc20adPYv38/d955J+3ataNNmza8+eabAKxevZr27duTnZ1N69at2bBhA0OGDGHTpk1kZ2fz8MMPH7Hv/fv307NnT7KyssjMzGTatGmMGzeO7du307VrV7p27QrEJifMycmhVatWPP744wCh7Qpfb7z7Ld4/kydPpnXr1mRlZXHrrbeW5z+bpCB9D0ISJ4r5vondaKdHjx6ce+651KtXj+XLl9O2bVveffddZs+ezZIlS6hduzaff/459erVY/z48YwZM+aIacHDnH/++fzpT3+ievXqzJs3j2HDhjFz5szQtjVr1uSpp54iNzeX8ePHA7GJAy+99FImTpzInj17aN++PZdddhkvvfQS9913HzfffDNff/01hw8fZtSoUaxatSp0DqX33nuPs88+m3feeQeIzfNUt25dnnvuORYsWECDBg0AGDlyJPXq1ePw4cN069aNlStXMmDAgGPalXa/hVavXs3IkSNZtGgRDRo04PPPPz9u/0nloxGEVDlTpkyhb9++QGw+o8K5lObNm8ePfvSjoqmv69WrV6r97t27lz59+pCZmckDDzzA6tWrS7X93LlzGTVqFNnZ2UWzuW7bto2LL76Yn/3sZ4wePZqtW7dSq1at4+7nggsuYN68eQwePJg///nP1K1bN7Td9OnTadu2LW3atGH16tVH3L60PPstNH/+fHr37l0UHKXtT0l9GkFI4kQw33dBQQHz589n1apVmBmHDx/GzHjmmWdw97huIVp8yu7CKbkBHn30Ubp27cqsWbPIy8sr9ben3Z2ZM2dy3nnnHbG+RYsWXHTRRbzzzjt0796dV155hWbNmpW4n3PPPZdly5YxZ84chg4dyhVXXMFjjz12RJstW7YwZswYPvzwQ9LT07njjjuOeC1l3e/RryfKW7JK4mkEIVXKjBkzuO2229i6dSt5eXl88sknNG3alA8++IArrriCiRMnFl1wLTwlcvS02hkZGSxbtgzgiFNIe/fupXHjxkDswvaJHL3f7t278/Of/5zCCTI/+ugjADZv3kyzZs0YMGAA11xzDStXrjzuVN/bt2+ndu3a3HLLLQwcOLBoevHi23zxxRecdtpp1K1blx07dvDuu++WWFdp9ltct27dmD59OgUFBQA6xVQFJSwgzGyime00s1XF1tUzs/81sw3B7/RgvZnZODPbaGYrzaxtouqSqm3KlClcf/31R6zr1asXv/3tb+nRowfXXHMNOTk5ZGdnF907+o477uDuu+8uukj9+OOPc9999/GDH/yAatWqFe1n0KBBDB06lE6dOnH48OET1tK1a1fWrFlTdJH60Ucf5ZtvvqF169ZkZmby6KOPAjBt2jQyMzPJzs5m3bp13HbbbdSvX59OnTqRmZl5zEXqjz/+uOii9siRI3nkkUcA6N+/P1deeSVdu3YlKyuLNm3a0KpVK+688046depUtH3xdqXdb3GtWrVi+PDhXHLJJWRlZfHggw+esE+kcknYdN9m9l/APmCyu2cG654BPnf3UWY2BEh398FmdhVwL3AVcBHwgrtfdKJjaLrvheUtpcKdjNN9y/Fpuu/UE/l03+7+J+DoMee1wKTg8STgumLrJ3vMYuAMM2uUqNpEROTEkn0N4ix3/wwg+H1msL4x8EmxdvnBOhERiUiqXKQO+yhE6LkvM+tvZrlmlrtr164ElyVlUZnvUigVS38LlVuyA2JH4amj4PfOYH0+8N1i7ZoAoXMfuPsEd89x95yGDRsmtFgpvbS0NAoKCvQPg+DuFBQUkJaWFnUpUkbJ/h7EW8DtwKjg95vF1t9jZlOJXaTeW3gqSiqXJk2akJ+fj0Z3ArE3DE2aNIm6DCmjhAWEmU0BugANzCwfeJxYMEw3s37ANqBP0HwOsU8wbQQOAD9KVF2SWDVq1KBp06ZRlyEiFSBhAeHuN5XwVLeQtg78NFG1iIhI6aXKRWoREUkxCggREQmlgBARkVAKCBERCaWAEBGRUAoIEREJpYAQEZFQCggREQmlgBARkVAKCBERCaWAEBGRUAoIEREJpYAQEZFQCggREQmlgBARkVAKCBERCaWAEBGRUAoIEREJpYAQEZFQCggREQmlgBARkVAKCBERCaWAEBGRUAoIEREJpYAQEZFQCggREQmlgBARkVAKCBERCRVJQJjZA2a22sxWmdkUM0szs6ZmtsTMNpjZNDOrGUVtIiISk/SAMLPGwAAgx90zgWpAX2A08Ly7Nwf+CfRLdm0iIvKtqE4xVQdqmVl1oDbwGXApMCN4fhJwXUS1iYgIEQSEu38KjAG2EQuGvcAyYI+7Hwqa5QONw7Y3s/5mlmtmubt27UpGySIiJ6UoTjGlA9cCTYGzgdOAK0Oaetj27j7B3XPcPadhw4aJK1RE5CQXxSmmy4At7r7L3b8B3gA6AmcEp5wAmgDbI6hNREQCUQTENqCDmdU2MwO6AWuABUDvoM3twJsR1CYiIoEorkEsIXYxejnwcVDDBGAw8KCZbQTqA68muzYREflW9RM3qXju/jjw+FGrNwPtIyhHRERC6JvUIiISSgEhIiKhFBAiIhJKASEiIqEUECIiEkoBISIioRQQIiISSgEhIiKhFBAiIhJKASEiIqEUECIiEkoBISIioRQQIiISSgEhIiKhFBAiIhJKASEiIqFKHRBmlm5mrRNRjIiIpI64AsLMFprZ6WZWD/gb8Gszey6xpYmISJTiHUHUdfcvgB8Cv3b3C4HLEleWiIhELd6AqG5mjYAbgLcTWI+IiKSIeAPiSeAPwEZ3/9DMmgEbEleWiIhErXqc7T5z96IL0+6+WdcgRESqtnhHED+Pc52IiFQRxx1BmNnFQEegoZk9WOyp04FqiSxMRESidaJTTDWB7wTt6hRb/wXQO1FFiYhI9I4bEO7+R+CPZvaau29NUk0iIpIC4r1IfaqZTQAyim/j7pcmoigREYlevAHxO+Al4BXgcHkPamZnBPvKBBy4E1gPTCMWQnnADe7+z/IeS0REyibeTzEdcvdfuvtSd19W+FOO474AvOfu5wNZwFpgCPC+uzcH3g+WRUQkIvEGxO/N7L/NrJGZ1Sv8KcsBzex04L+AVwHc/Wt33wNcC0wKmk0CrivL/kVEpGLEe4rp9uD3w8XWOdCsDMdsBuwiNuFfFrAMuA84y90/A3D3z8zszDLsW0REKkhcAeHuTSv4mG2Be919iZm9QClOJ5lZf6A/wDnnnFOBZYmISHFxBYSZ3Ra23t0nl+GY+UC+uy8JlmcQC4gdZtYoGD00AnaWcMwJwASAnJwcL8PxRUQkDvGeYmpX7HEa0A1YDpQ6INz9H2b2iZmd5+7rg32tCX5uB0YFv98s7b5FRKTixHuK6d7iy2ZWF3i9HMe9F/iNmdUENgM/InbBfLqZ9QO2AX3KsX8RESmneEcQRzsANC/rQd19BZAT8lS3su5TREQqVrzXIH5P7FNLEJukrwUwPVFFiYhI9OIdQYwp9vgQsNXd8xNQj4iIpIi4vigXTNq3jtiMrunA14ksSkREohdXQJjZDcBSYheObwCWmJmm+xYRqcLiPcU0HGjn7jsBzKwhMI/YdxhERKQKincuplMKwyFQUIptRUSkEop3BPGemf0BmBIs3wjMSUxJIiKSCk50T+rvE5tE72Ez+yHQGTDgr8BvklCfiIhE5ESnicYCXwK4+xvu/qC7P0Bs9DA20cWJiEh0ThQQGe6+8uiV7p5L7M5vIiJSRZ0oINKO81ytiixERERSy4kC4kMzu+volcGEeuW55aiIiKS4E32K6X5glpndzLeBkAPUBK5PZGEiIhKt4waEu+8AOppZVyAzWP2Ou89PeGUiIhKpeO8HsQBYkOBaREQkhejb0CIiEkoBISIioRQQIiISSgEhIiKhFBAiIhJKASEiIqEUECIiEkoBISIioRQQIiISSgEhIiKhFBAiIhJKASEiIqEiCwgzq2ZmH5nZ28FyUzNbYmYbzGyamdWMqjYREYl2BHEfsLbY8mjgeXdvDvwT6BdJVSIiAkQUEGbWBOgJvBIsG3ApMCNoMgm4LoraREQkJqoRxFhgEPDvYLk+sMfdDwXL+UDjKAoTEZGYpAeEmV0N7HT34ve0tpCmXsL2/c0s18xyd+3alZAaRUQkmhFEJ+AaM8sDphI7tTQWOMPMCu9w1wTYHraxu09w9xx3z2nYsGEy6hUROSklPSDcfai7N3H3DKAvMN/dbyZ2S9PeQbPbgTeTXZuIiHwrlb4HMRh40Mw2Ersm8WrE9YiInNSqn7hJ4rj7QmBh8Hgz0D7KekRE5FupNIIQEZEUooAQEZFQCggREQmlgBARkVAKCBERCaWAEBGRUAoIEREJpYAQEZFQCggREQmlgBARkVAKCBERCaWAEBGRUAoIEREJpYAQEZFQCggREQmlgBARkVAKCBERCaWAEBGRUAoIEREJFek9qSW5Moa8E9mx80b1jOzYIlI2GkGIiEgojSAiUN538lM3FwDQN8IRgYhUfRpBiIhIKAWEiIiEUkCIiEgoBYSIiIRSQIiISCgFhIiIhEp6QJjZd81sgZmtNbPVZnZfsL6emf2vmW0IfqcnuzYREflWFCOIQ8BD7t4C6AD81MxaAkOA9929OfB+sCwiIhFJekC4+2fuvjx4/CWwFmgMXAtMCppNAq5Ldm0iIvKtSK9BmFkG0AZYApzl7p9BLESAM0vYpr+Z5ZpZ7q5du5JVqojISSeygDCz7wAzgfvd/Yt4t3P3Ce6e4+45DRs2TFyBIiInuUgCwsxqEAuH37j7G8HqHWbWKHi+EbAzitpERCQmik8xGfAqsNbdnyv21FvA7cHj24E3k12biIh8K4rZXDsBtwIfm9mKYN0wYBQw3cz6AduAPhHUJiIigaQHhLt/AFgJT3dLZi0iIlIyfZNaRERCKSBERCSUAkJEREIpIEREJJQCQkREQikgREQklAJCRERCKSBERCSUAkJEREIpIEREJJQCQkREQikgREQklAJCRERCRTHdt0jSZAx5J5Lj5o3qGclxRSqSRhAiIhJKASEiIqEUECIiEkoBISIioXSRWpIiqovFIlJ2GkGIiEgoBYSIiIRSQIiISChdgxBJgCivuehLelJRNIIQEZFQCggREQmlU0wiVUwqfqT4H5s7AJAxZHFC9h/VabWqfipRIwgREQmVUiMIM+sBvABUA15x91ERlyQilUAqjpqqgpQZQZhZNeBF4EqgJXCTmbWMtioRkZNXygQE0B7Y6O6b3f1rYCpwbcQ1iYictFIpIBoDnxRbzg/WiYhIBMzdo64BADPrA3R39x8Hy7cC7d393qPa9Qf6B4vnAevLeMgGwO4yblvVqW9Kpr4pmfqmZKnWN99z94YnapRKF6nzge8WW24CbD+6kbtPACaU92BmluvuOeXdT1WkvimZ+qZk6puSVda+SaVTTB8Czc2sqZnVBPoCb0Vck4jISStlRhDufsjM7gH+QOxjrhPdfXXEZYmInLRSJiAA3H0OMCdJhyv3aaoqTH1TMvVNydQ3JauUfZMyF6lFRCS1pNI1CBERSSFVMiDMrIeZrTezjWY2JOT5U81sWvD8EjPLKPbc0GD9ejPrnsy6k6GsfWNm9c1sgZntM7Pxya47GcrRN5eb2TIz+zj4fWmya0+0cvRNezNbEfz8zcyuT3btiVSef2uC588J/p8amKyaS8Xdq9QPsQvcm4BmQE3gb0DLo9r8N/BS8LgvMC143DJofyrQNNhPtahfU4r0zWlAZ+BuYHzUryXF+qYNcHbwOBP4NOrXk0J9UxuoHjxuBOwsXK7sP+Xpl2LPzwR+BwyM+vWE/VTFEUQ8U3ZcC0wKHs8AupmZBeunuvu/3H0LsDHYX1VR5r5x9/3u/gFwMHnlJlV5+uYjdy/8zs5qIM3MTk1K1clRnr454O6HgvVpQFW66Fmef2sws+uAzcT+ZlJSVQyIeKbsKGoT/PHuBerHuW1lVp6+qeoqqm96AR+5+78SVGcUytU3ZnaRma0GPgbuLhYYlV2Z+8XMTgMGA08moc4yq4oBYSHrjn7XUlKbeLatzMrTN1VdufvGzFoBo4H/V4F1pYJy9Y27L3H3VkA7YKiZpVVwfVEpT788CTzv7vsqvKoKVBUDIp4pO4ramFl1oC7weZzbVmbl6Zuqrlx9Y2ZNgFnAbe6+KeHVJleF/N24+1pgP7HrNFVBefrlIuAZM8sD7geGBV8UTilVMSDimbLjLeD24HFvYL7Hrhi9BfQNPnnQFGgOLE1S3clQnr6p6srcN2Z2BvAOMNTdFyWt4uQpT980Df5hxMy+R2yCzbzklJ1wZe4Xd/+Bu2e4ewYwFviZu6fepwOjvkqeiB/gKuDvxD5hMDxY9xRwTfA4jdgnBzYSC4BmxbYdHmy3Hrgy6teSYn2TR+zdzz5i74xaJrv+VOwb4BFi74xXFPs5M+rXkyJ9cyuxi7ArgOXAdVG/llTol6P28QQp+ikmfZNaRERCVcVTTCIiUgEUECIiEkoBISIioRQQIiISSgEhIiKhFBCSsoIZZAtnAv2HmX1abLlmKfZzp5n9RynaX2pmHUrbzsx+amY3V1T78jCz6ma2JxH7lpNHSt1RTqQ4dy8AsgHM7Algn7uPKcOu7iT2Gfx/xNn+UmA3sLg07dz9xQpuLxIpjSCkUjKz281saTCa+IWZnRK8a349uC/DKjMbYGY3EguZaWEjDzN7wMzWBPcq+B8z+0/gx8DDQfuOZnZtMJf/R2Y218zOLKHdCDO7vxT7Ld7+XDObH7RfHnLfgGfNrH+x5RFmdp+ZnR5st9zMVprZ1SF9dZmZzS62/JKZ3RI8bmdmf7TYfSzeNbOzKuK/j1QNGkFIpWNmmcD1QEd3P2RmE4hNc7AJaODuFwTtznD3PWZ2L3CPu68I2d0g4Hvu/nWx9q8Au919bLCfdOAtd3czuxt4yN0Hh7S7qpT7Ld5+CvCEu//eYpPZHf3mbSowim/vbdwH6Ap8BVzr7l+a2ZnAIuDtOPvxVOAFYt/63R2c7noa6H/8LeVkoYCQyugyYjOD5lpsav1axKZU/gNwnpm9AMwB5saxr9XA/5jZm8DsEtqcA0wPrmOcSmxqhYrYL1AUQA3c/fcA7n7MPTfc/UMz+27wDr8J8A933x6MiEabWWfg38B3zawBEM/1hxZAK2Be0I/ViE2hIgIoIKRyMmCiuz96zBNmrYErgQHE7s1wonfD3YFLiN3Y5ZFgdHK0F4lNpjbHzC4Djrm1ZBn3W1w8c97MJPaaMoiNKABuIzZDaNtgNJVPbP6f4g5x5Iik8HkDVrr7D+I4tpyEdA1CKqN5wA3BO+XCTzudY2YNAXP33wGPA22D9l8CdY7eiZlVA5q4+3zgYaAhsVtkHt2+LvCpxd5m315sfXn3C4C7/xPYbWb/J9g+zcxqh7zuqcROpf2QWFgU1rYzCIfLCb/B1VaglZnVDEYrhffMXgM0NrP2wXFrWuyeFiKAAkIqIXf/mNgNV+aZ2Upip5LOIjbv/p/MbAXwMjAs2OTXwCshF6mrA78N9rEcGO3uXwJvEgugj8ysI7HZNmcBfwR2FNv+6Hal3W9xNwMPBdt8QCxUjn7dfwvWb3H3ncHq14GOZpZL7LrEhpDtthA7zfUxMDmoCY/d9a438JyZ/Q34iNh9CkQANJuriIiE0whCRERCKSBERCSUAkJEREIpIEREJJQCQkREQikgREQklAJCRERCKSBERCTU/wfVcY6zZTJwJgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# IMPLEMENT: visualize test for squared statistic\n",
    "simple_statistic_squared = lambda X,Y: simple_statistic(X,Y,squared=True)\n",
    "statistics = two_sample_permutation_test(simple_statistic_squared, X, Y, num_permutations)\n",
    "my_statistic = simple_statistic_squared(X,Y)\n",
    "plot_permutation_samples(statistics, my_statistic)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, both test statistics essentially have no chance in distinguishing this data.\n",
    "(althogh keep in mind we could have been unlucky as outlined above).\n",
    "For now, let's just assume we only have a single dataset to compute the test."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quadratic time MMD\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have already seen the quadratic time MMD: it was the squared difference of the sample means above, if we had used a linear kernel (where the feature space embedding is the identity function).\n",
    "This test wasn't able to distinguish the above distributions, but let's try a different kernel.\n",
    "\n",
    "We will use the unbiased statistic here, defined as\n",
    "$$ \n",
    "\\frac{1}{n(n-1)}\\sum_{i=1}^n\\sum_{j\\neq i}^m k(x_i,x_j) + \\frac{1}{m(m-1)}\\sum_{i=1}^n\\sum_{j\\neq i}^mk(y_i,y_j) -2\\frac{1}{nm}\\sum_{i=1}^n \\sum_{j=1}^m k(x_i,y_j)\n",
    "$$\n",
    "See the slides for more details."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Kernels\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first need to implement a kernel function.\n",
    "Though we could use a framework to do this for us, it is quite instructive to do this ourselves (at least once).\n",
    "\n",
    "We will start with a Gaussian kernel\n",
    "$$\n",
    "k(x,y) = \\exp \\left( - \\frac{||x-y||_2^2}{\\sigma}\\right)\n",
    "$$\n",
    "with some bandwidth parameter $\\sigma$ (note there are many different parametrizations!).\n",
    "\n",
    "This kernel is translation invariant, and like many other kernels, it is defined in terms of pairwise (Euclidean) distances between the data.\n",
    "Let's implement it this way so we can easily try other kernels that are translation invariant."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.spatial.distance import squareform, pdist, cdist\n",
    "\n",
    "def sq_distances(X,Y=None):\n",
    "    \"\"\"\n",
    "    If Y=None, then this computes the distance between X and itself\n",
    "    \"\"\"\n",
    "    assert(X.ndim==2)\n",
    "\n",
    "    # IMPLEMENT: compute pairwise distance matrix. Don't use explicit loops, but the above scipy functions\n",
    "    # if X=Y, use more efficient pdist call which exploits symmetry\n",
    "    if Y is None:\n",
    "        sq_dists = squareform(pdist(X, 'sqeuclidean'))\n",
    "    else:\n",
    "        assert(Y.ndim==2)\n",
    "        assert(X.shape[1]==Y.shape[1])\n",
    "        sq_dists = cdist(X, Y, 'sqeuclidean')\n",
    "\n",
    "    return sq_dists\n",
    "\n",
    "def gauss_kernel(X, Y=None, sigma=1.0):\n",
    "    \"\"\"\n",
    "    Computes the standard Gaussian kernel k(x,y)=exp(- ||x-y||**2 / (2 * sigma**2))\n",
    "\n",
    "    X - 2d array, samples on left hand side\n",
    "    Y - 2d array, samples on right hand side, can be None in which case they are replaced by X\n",
    "    \n",
    "    returns: kernel matrix\n",
    "    \"\"\"\n",
    "\n",
    "    # IMPLEMENT: compute squared distances and kernel matrix\n",
    "    sq_dists = sq_distances(X,Y)\n",
    "    K = np.exp(-sq_dists / (2 * sigma**2))\n",
    "    return K\n",
    "\n",
    "# IMPLEMENT\n",
    "def linear_kernel(X, Y):\n",
    "    return np.dot(X, Y.T)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the kernel at hand, we can implement the quadratic time MMD statistic as a function of the kernel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def quadratic_time_mmd(X,Y,kernel):\n",
    "    assert X.ndim == Y.ndim == 2\n",
    "    K_XX = kernel(X,X)\n",
    "    K_XY = kernel(X,Y)\n",
    "    K_YY = kernel(Y,Y)\n",
    "       \n",
    "    n = len(K_XX)\n",
    "    m = len(K_YY)\n",
    "    \n",
    "    # IMPLEMENT: unbiased MMD statistic (could also use biased, doesn't matter if we use permutation tests)\n",
    "    np.fill_diagonal(K_XX, 0)\n",
    "    np.fill_diagonal(K_YY, 0)\n",
    "    mmd = np.sum(K_XX) / (n*(n-1))  + np.sum(K_YY) / (m*(m-1))  - 2*np.sum(K_XY)/(n*m)\n",
    "    return mmd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAH29JREFUeJzt3XuUFNW99vHvTy4ZMIgzgL4IyRk8QQVGpsHBIJBXQBEiHDRBoy6MGI28uXg03kG8C1ngQkMIJgaViJ4TwICgEVREIIlExAERuRkEBkWUyxiQi6jg7/2jasYG5tIzTHXPWM9nrV7dXb1r195TDU/vqu5d5u6IiEh8HZPpBoiISGYpCEREYk5BICIScwoCEZGYUxCIiMScgkBEJOYUBCIiMacgEBGJOQWBiEjM1c90A1LRvHlzz83Njaz+d94J7k89NbJNSA3TPpOvvRp4ky9dunSHu7eorFydCILc3FwKCwsjq79nz+B+4cLINiE1TPtMvvZq4E1uZptSKadDQyIiMacgEBGJOQWBiEjM1YlzBCJyqC+++ILNmzezf//+TDdFonL33cH9mjWVFs3KyqJ169Y0aNCgWptSEIjUQZs3b6ZJkybk5uZiZplujkThmPCATSXfGnJ3iouL2bx5M23atKnepqq1lohk1P79+2nWrJlCQDAzmjVrdlSjQwWBSB2lEJASR/teUBCIiMScgkBEquW3v/0teXl5dOjQgXHjxpUuv+eee2jVqhWJRIJEIsGcOXMAWLRoER07dqRLly68++67AOzcuZO+ffuSyWunz5o1i9WrV5c+v+uuu5g3bx4APXv2jPTHrLWFThZHIHfY7GqtVzS6fw23RCQaK1eu5NFHH2XJkiU0bNiQfv360b9/f9q2bQvADTfcwM0333zIOg8++CAzZsygqKiIP/zhDzz44IPcf//93H777Rk9zDVr1iwGDBhA+/btAbjvvvsy1pZM0YhARKpszZo1dO3alcaNG1O/fn3OPvtsZs6cWeE6DRo04NNPP2Xfvn00aNCA9evX88EHH3D22WeXu86LL77IaaedRo8ePbjuuusYMGAAEIw6xo4dW1ouLy+PoqIiAC688ELOOOMMOnTowMSJE0vLfPOb32TEiBHk5+fTtWtXtm7dyj//+U+ee+45brnlFhKJBOvXr+fKK69k+vTpR7Rl7ty5nHXWWXTu3JmLL76YPXv2ADBs2DDat29Px44djwi/ukIjApE67le/guXLa7bORAKSjvYcIS8vjxEjRlBcXEyjRo2YM2cOBQUFpa9PmDCBJ598koKCAh588EGys7MZPnw4Q4cOpVGjRjz11FPcfPPN3H///eVuY//+/VxzzTXMnz+f73znO1xyySUptX3SpEnk5OTw6aef0qVLFwYNGkSzZs3Yu3cvXbt2ZdSoUdx66608+uij3HHHHQwcOJABAwZw0UUXlVvnjh07GDlyJPPmzePYY49lzJgxPPTQQ1x77bXMnDmTtWvXYmbs3LkzpTbWNhoRiEiVtWvXjttuu40+ffrQr18/8vPzqV8/+Fz585//nPXr17N8+XJatmzJTTfdBEAikWDx4sUsWLCADRs2cNJJJ+HuXHLJJVx++eVs3br1kG2sXbuWNm3a0LZtW8yMyy+/PKW2jR8/vvRT//vvv8+6desAaNiwYemI4owzzigdQaRi8eLFrF69mu7du5NIJJg8eTKbNm3iuOOOIysri5/+9Kc888wzNG7cOOU6axONCETquIo+uUfp6quv5uqrrwbg9ttvp3Xr1gCceOKJpWWuueaa0v98S7g7I0eOZNq0aVx77bXce++9FBUVMX78eEaNGnVI2fLOHdSvX58vv/yy9HnJd+gXLlzIvHnzeO2112jcuDE9e/Ysfa1Bgwal9dWrV48DBw6k3Fd3p0+fPkyZMuWI15YsWcIrr7zC1KlTmTBhAvPnz0+53tpCIwIRqZZt27YB8N577/HMM89w2WWXAfDhhx+Wlpk5cyZ5eXmHrDd58mT69+9PdnY2+/bt45hjjuGYY45h3759h5Q77bTT2LhxI+vXrwc45D/h3Nxcli1bBsCyZcvYuHEjALt27SI7O5vGjRuzdu1aFi9eXGk/mjRpwu7duyss07VrVxYtWlT6bad9+/bxr3/9iz179rBr1y7OP/98xo0bx/KaPkaXJhoRiEi1DBo0iOLiYho0aMDDDz9MdnY2ALfeeivLly/HzMjNzeWPf/xj6Tr79u1j8uTJzJ07F4Abb7yRQYMG0bBhwyM+bWdlZTFx4kT69+9P8+bN6dGjBytXrizd9pNPPkkikaBLly6ccsopAPTr149HHnmEjh07cuqpp9K1a9dK+3HppZdyzTXXMH78+DJPEgO0aNGCJ554gssuu4zPPvsMgJEjR9KkSRMuuOAC9u/fj7vzm9/8pop/xdrBMvn93VQVFBR4Xbowjb4+Gr24X5hmzZo1tGvXLtPNSKuFCxcyduxYnn/++Uw3JT2qeIWyst4TZrbU3QvKWaVUpCMCMysCdgMHgQPuXmBmOcA0IBcoAn7k7v+Osh0iIlK+dJwj6OXuiaRUGga84u5tgVfC5yIiFerZs2d8RgNplomTxRcAk8PHk4ELM9AGEREJRR0EDsw1s6VmNjRcdqK7fwgQ3p9Q1opmNtTMCs2scPv27RE3U0QkvqL+1lB3d99iZicAL5vZ2lRXdPeJwEQIThZH1UARkbiLdETg7lvC+23ATOBMYKuZtQQI77dF2QYREalYZEFgZseaWZOSx8B5wErgOWBIWGwI8GxUbRCRaM2cORMzY+3aygf7TzzxBFu2bKn2thYuXHjEr5QBli9fXjrVdVXt3LmT3//+91VaZ9y4cUf8+C2Vcueff36FcxEdUX7oUHZ+8kmV2lZdUY4ITgReNbO3gCXAbHd/ERgN9DGzdUCf8LmI1EFTpkyhR48eTJ06tdKyRxsE5akrQTBnzhyOP/741MtPnMjxxx1XpbZVV2RB4O4b3D0/vHVw91Hh8mJ3P8fd24b3H0fVBhGJzp49e1i0aBGPP/74EUHwwAMPcPrpp5Ofn8+wYcOYPn06hYWFDB48mEQiwaeffkpubi47duwAoLCwkJ7hrwSXLFlCt27d6NSpE926deOdkh9WleHzzz/nrrvuYtq0aSQSCaZNm8bevXu56qqr6NKlC506deLZZ4ODDqtWreLMM88kkUjQsWNH1q1bx7Bhw1i/fj2JRIJbbrnlkLr37t1L//79yc/PJy8vj2nTpjF+/Hi2bNlCr1696NWrFxBMsldQUECHDh24++67AcosV9LfVOvN7d2bHf8OfmL15JNP0rFjR/Lz8/nxj398NLutTJpiQqSuy8Q81AQXdOnXrx+nnHIKOTk5LFu2jM6dO/PCCy8wa9YsXn/9dRo3bszHH39MTk4OEyZMYOzYsYdMV12W0047jb///e/Ur1+fefPmcfvttzNjxowyyzZs2JD77ruPwsJCJkyYAAQT4PXu3ZtJkyaxc+dOzjzzTM4991weeeQRrr/+egYPHsznn3/OwYMHGT16NCtXrixzjqAXX3yRk046idmzg5kCdu3aRdOmTXnooYdYsGABzZs3B2DUqFHk5ORw8OBBzjnnHFasWMF11113RLmq1lti1apVjBo1ikWLFtG8eXM+/rjmPztr0jkRqZYpU6Zw6aWXAsF8PSVzBc2bN4+f/OQnpVMy5+TkVKneXbt2cfHFF5OXl8cNN9zAqlWrqrT+3LlzGT16NIlEonT20ffee4+zzjqLX//614wZM4ZNmzbRqFGjCus5/fTTmTdvHrfddhv/+Mc/aNq0aZnlnn76aTp37kynTp1YtWrVIZe9PJp6S8yfP5+LLrqoNCCq+vdMhUYEInVdBuahLi4uZv78+axcuRIz4+DBg5gZDzzwAO6e0qUnk6eSLpkqGuDOO++kV69ezJw5k6KiotJDRqlyd2bMmMGph83R065dO7773e8ye/Zs+vbty2OPPcbJJ59cbj2nnHIKS5cuZc6cOQwfPpzzzjuPu+6665AyGzduZOzYsbzxxhtkZ2dz5ZVXHtKX6tZ7eH+ivpSnRgQiUmXTp0/niiuuYNOmTRQVFfH+++/Tpk0bXn31Vc477zwmTZpUeuKz5FDG4dM95+bmsnTpUoBDDv3s2rWLVq1aAcEJ5socXm/fvn353e9+R8mEmm+++SYAGzZs4OSTT+a6665j4MCBrFixosIpqLds2ULjxo25/PLLufnmm0unvU5e55NPPuHYY4+ladOmbN26lRdeeKHcdlWl3mTnnHMOTz/9NMXFxQA6NCQitcOUKVP4wQ9+cMiyQYMG8ec//5l+/foxcOBACgoKSCQSpdcWvvLKK/nZz35WerL47rvv5vrrr+d73/se9erVK63n1ltvZfjw4XTv3p2DBw9W2pZevXqxevXq0pPFd955J1988QUdO3YkLy+PO++8E4Bp06aRl5dHIpFg7dq1XHHFFTRr1ozu3buTl5d3xMnit99+u/Tk8qhRo7jjjjsAGDp0KN///vfp1asX+fn5dOrUiQ4dOnDVVVfRvXv30vWTy1W13mQdOnRgxIgRnH322eTn53PjjTdW+jepKk1Djaahros0DXX8pqGOnTROQ60RgYhIzCkIRERiTkEgUkfVhcO6kh5H+15QEIjUQVlZWRQXFysMBHenuLiYrKysateh3xGI1EGtW7dm8+bN6FodX2MffRTch7+1qEhWVhatW7eu9qYUBCJ1UIMGDWjTpk2mmyFR+vnPg/s0fDVOh4ZERGJOQSAiEnMKAhGRmFMQiIjEnIJARCTmFAQiIjGnIBARiTkFgYhIzCkIRERiTkEgIhJzCgIRkZhTEIiIxJyCQEQk5hQEIiIxpyAQEYk5BYGISMwpCEREYk5BICISc5EHgZnVM7M3zez58HkbM3vdzNaZ2TQzaxh1G0REpHzpGBFcD6xJej4G+I27twX+DVydhjaIiEg5Ig0CM2sN9AceC58b0BuYHhaZDFwYZRtERKRiUY8IxgG3Al+Gz5sBO939QPh8M9Aq4jaIiEgFIgsCMxsAbHP3pcmLyyjq5aw/1MwKzaxw+/btkbRRRESiHRF0BwaaWREwleCQ0DjgeDOrH5ZpDWwpa2V3n+juBe5e0KJFiwibKSISb5EFgbsPd/fW7p4LXArMd/fBwALgorDYEODZqNogIiKVq195kRp3GzDVzEYCbwKPZ6ANlcodNjvTTRARSYu0BIG7LwQWho83AGemY7siIlI5/bJYRCTmFAQiIjGnIBARiTkFgYhIzCkIRERiTkEgIhJzCgIRkZhTEIiIxJyCQEQk5hQEIiIxl4m5hqQc1Z3fqGh0/xpuiYjEiUYEIiIxpyAQEYk5BYGISMwpCEREYk5BICIScwoCEZGYUxCIiMScgkBEJOYUBCIiMacgEBGJOQWBiEjMKQhERGJOQSAiEnMKAhGRmFMQiIjEnIJARCTmqhwEZpZtZh2jaIyIiKRfSkFgZgvN7DgzywHeAv5kZg9F2zQREUmHVEcETd39E+CHwJ/c/Qzg3OiaJSIi6ZJqENQ3s5bAj4DnU1nBzLLMbImZvWVmq8zs3nB5GzN73czWmdk0M2tYzbaLiEgNSDUI7gVeAt519zfM7GRgXSXrfAb0dvd8IAH0M7OuwBjgN+7eFvg3cHX1mi4iIjUh1SD40N07uvsvANx9A1DhOQIP7AmfNghvDvQGpofLJwMXVrnVIiJSY1INgt+luOwQZlbPzJYD24CXgfXATnc/EBbZDLRKsQ0iIhKB+hW9aGZnAd2AFmZ2Y9JLxwH1Kqvc3Q8CCTM7HpgJtCurWDnbHgoMBfj2t79d2aZERKSaKhsRNAS+SRAYTZJunwAXpboRd98JLAS6AsebWUkAtQa2lLPORHcvcPeCFi1apLopERGpogpHBO7+N+BvZvaEu2+qSsVm1gL4wt13mlkjgq+bjgEWEITIVGAI8Gy1Wi4iIjWiwiBI8g0zmwjkJq/j7r0rWKclMNnM6hGMPJ529+fNbDUw1cxGAm8Cj1er5SIiUiNSDYK/AI8AjwEHU1nB3VcAncpYvgE4M9UGiohItFINggPu/odIWyIiIhmR6tdH/2pmvzCzlmaWU3KLtGUiIpIWqY4IhoT3tyQtc+Dkmm2OiIikW0pB4O5tom6IiIhkRkpBYGZXlLXc3Z+s2eaIiEi6pXpoqEvS4yzgHGAZoCAQEanjUj009N/Jz82sKfBUJC0SEZG0qu41i/cBbWuyISIikhmpniP4K19NDlePYPK4p6NqlIiIpE+q5wjGJj0+AGxy980RtEdERNIspUND4eRzawlmHs0GPo+yUSIikj4pBYGZ/QhYAlxMcN3i180s5WmoRUSk9kr10NAIoIu7b4PSKabn8dUlJ0VEpI5K9VtDx5SEQKi4CuuKiEgtluqI4EUzewmYEj6/BJgTTZNERCSdKrtm8XeAE939FjP7IdADMOA14H/T0D4REYlYZYd3xgG7Adz9GXe/0d1vIBgNjIu6cSIiEr3KgiA3vNLYIdy9kOCylSIiUsdVFgRZFbzWqCYbIiIimVFZELxhZtccvtDMrgaWRtMkERFJp8q+NfQrYKaZDear//gLgIbAD6JsmIiIpEeFQeDuW4FuZtYLyAsXz3b3+ZG3TERE0iLV6xEsABZE3Bapptxhs6u1XtHo/jXcEhGpi/TrYBGRmFMQiIjEnIJARCTmFAQiIjGnIBARiTkFgYhIzCkIRERiLrIgMLNvmdkCM1tjZqvM7PpweY6ZvWxm68L77KjaICIilYtyRHAAuMnd2wFdgV+aWXtgGPCKu7cFXgmfi4hIhkQWBO7+obsvCx/vBtYArYALgMlhscnAhVG1QUREKpeWcwRmlgt0Al4nuOLZhxCEBXBCOtogIiJlizwIzOybwAzgV+7+SRXWG2pmhWZWuH379ugaKCISc5EGgZk1IAiB/3X3Z8LFW82sZfh6S2BbWeu6+0R3L3D3ghYtWkTZTBGRWIvyW0MGPA6scfeHkl56DhgSPh4CPBtVG0REpHIpTUNdTd2BHwNvm9nycNntwGjg6fAqZ+8BF0fYBhERqURkQeDurwJWzsvnRLVdERGpGv2yWEQk5hQEIiIxpyAQEYk5BYGISMwpCEREYk5BICIScwoCEZGYUxCIiMScgkBEJOYUBCIiMacgEBGJOQWBiEjMKQhERGJOQSAiEnNRXo+gVsgdNrvSMh9t6BqWXRx1c2qVVP42ZSka3b+GWyIimaQRgYhIzCkIRERiTkEgIhJzCgIRkZhTEIiIxJyCQEQk5hQEIiIxpyAQEYk5BYGISMwpCEREYk5BICIScwoCEZGYUxCIiMScgkBEJOYUBCIiMRdZEJjZJDPbZmYrk5blmNnLZrYuvM+OavsiIpKaKEcETwD9Dls2DHjF3dsCr4TPRUQkgyILAnf/O/DxYYsvACaHjycDF0a1fRERSU26zxGc6O4fAoT3J6R5+yIicphae7LYzIaaWaGZFW7fvj3TzRER+dpKdxBsNbOWAOH9tvIKuvtEdy9w94IWLVqkrYEiInGT7iB4DhgSPh4CPJvm7YuIyGGi/ProFOA14FQz22xmVwOjgT5mtg7oEz4XEZEMqh9Vxe5+WTkvnRPVNqX2yx02u1rrFY3uX8MtEZEStfZksYiIpIeCQEQk5hQEIiIxpyAQEYk5BYGISMwpCEREYk5BICIScwoCEZGYUxCIiMScgkBEJOYUBCIiMacgEBGJOQWBiEjMKQhERGJOQSAiEnMKAhGRmFMQiIjEnIJARCTmIrtUpUhNOvwSlx9t6BouX1zherrEpUjlNCIQEYk5BYGISMwpCEREYk7nCKTKDj9eX5tVt606tyBxohGBiEjMKQhERGJOQSAiEnMKAhGRmFMQiIjEnIJARCTmFAQiIjGXkSAws35m9o6ZvWtmwzLRBhERCaQ9CMysHvAw8H2gPXCZmbVPdztERCSQiRHBmcC77r7B3T8HpgIXZKAdIiJCZoKgFfB+0vPN4TIREcmATMw1ZGUs8yMKmQ0FhoZP95jZO5XU2xzYcTQN2zTmaNauUUfdl1ok0r5Etc+s/Hq1b2qnr1NfILk/VtZ/mSn7j1QKZSIINgPfSnreGthyeCF3nwhMTLVSMyt094Kjb17mqS+119epP+pL7ZXu/mTi0NAbQFsza2NmDYFLgecy0A4RESEDIwJ3P2Bm1wIvAfWASe6+Kt3tEBGRQEauR+Duc4A5NVxtyoeR6gD1pfb6OvVHfam90tofcz/iPK2IiMSIppgQEYm5Wh0EZpZjZi+b2brwPrucckPCMuvMbEjS8jPM7O1wKovxZsH3sMzsHjP7wMyWh7fzI+xDhdNpmNk3zGxa+PrrZpab9NrwcPk7ZtY31TqjFFF/isL9tNzMCtPTk+r3xcyamdkCM9tjZhMOW6fM91wd7cvCsM6SfycnpKMv4bar258+ZrY03AdLzax30jp1bd9U1Jea3TfuXmtvwAPAsPDxMGBMGWVygA3hfXb4ODt8bQlwFsFvF14Avh8uvwe4OQ3trwesB04GGgJvAe0PK/ML4JHw8aXAtPBx+7D8N4A2YT31UqmzLvUnfK0IaJ7m99bR9OVYoAfwM2DCYeuU+Z6ro31ZCBSkc7/UQH86ASeFj/OAD+rwvqmoLzW6b2r1iIBg6onJ4ePJwIVllOkLvOzuH7v7v4GXgX5m1hI4zt1f8+Av92Q560cplek0kvs4HTgn/KRyATDV3T9z943Au2F9mZyiI4r+ZEq1++Lue939VWB/cuEMvudqvC8ZdjT9edPdS36XtArICj9x18V9U2ZfomhkbQ+CE939Q4DwvqzhT3lTVrQKHx++vMS1ZrbCzCZZOYecakAq02mUlnH3A8AuoFkF62Zyio4o+gPBL8vnhsPfoaTH0fSlojores9FJYq+lPhTeOjhznQdSqHm+jMIeNPdP6Pu75vkvpSosX2T8SAws3lmtrKMW6qfcsubsqKiqSz+APwnkAA+BB6scsNTk8p0GlVtf0pTdEQkiv4AdHf3zgQz0v7SzP5v9ZuYsqPpy9HUGYUo+gIw2N1PB74X3n5cjbZVx1H3x8w6AGOA/1eFOqMQRV+ghvdNxoPA3c9197wybs8CW8MhXcmwe1sZVZQ3ZcXm8PHhy3H3re5+0N2/BB4lukMUqUynUVrGzOoDTYGPK1g3pSk6IhJFfygZ/rr7NmAm6TlkdDR9qajOMt9zEYuiL7j7B+H9buDPpO9Q3lH1x8xaE7yPrnD39Unl69y+KacvNb5vMh4ElXgOKPkW0BDg2TLKvAScZ2bZ4SGe84CXwkNJu82sazhsuqJk/ZJwCf0AWBlR+1OZTiO5jxcB88NjmM8Bl4bHN9sAbQlOdmVyio4a74+ZHWtmTQDM7FiC/RfV/qipvpSpovdcxGq8L2ZW38yah48bAANIz36Bo+iPmR0PzAaGu/uiksJ1cd+U15dI9k1NnXWO4kZwnOwVYF14nxMuLwAeSyp3FcHJx3eBnyQtLwj/QOuBCXz1A7qngLeBFeFOaBlhH84H/hW2YUS47D5gYPg4C/hL2PYlwMlJ644I13uHpG84lFVnGvdJjfaH4NsUb4W3Vensz1H2pYjgU9segk907St6z9W1vhB8m2hp+G9kFfBbwm951eb+AHcAe4HlSbcT6uK+Ka8vUewb/bJYRCTmavuhIRERiZiCQEQk5hQEIiIxpyAQEYk5BYGISMwpCCTjLJgBs2QWxY/s0JlhG1ahnqvM7P9UoXxvM+ta1XJm9kszG1xT5Y9G+J3ynVHULfGRkSuUiSRz92KC6T4ws3uAPe4+thpVXQUsAz5KsXxvYAewuCrl3P3hGi4vklEaEUitZsG1JpaEo4Pfm9kx4afgpyyYp32lmV1nZpcQhMm0skYSZnaDma02s7fM7H/M7D+BnwK3hOW7mdkFFswH/6aZzTWzE8opN9LMflWFepPLn2Jm88Pyyyzpeg3h6w9a0sR74brXm9lx4XrLLJgscUAZf6tzzWxW0vNHzOzy8HEXM/ubBRP7vWBmJ9bE/pGvB40IpNYyszyCKUC6ufsBM5tI8BP99QTXLzg9LHe8u+80s/8GrnX35WVUdyvwH+7+eVL5x4Ad7j4urCcbeM7d3cx+Btzk7reVUe78KtabXH4KcI+7/9XMsjjyw9hUYDRfXbP2YqAX8ClwgbvvtuAiJIuA51P8O36D4NenA919R3iY6n4gXTO9Si2nIJDa7FygC1AYTA9DI4Lpel8CTjWz3wJzgLkp1LUK+B8zexaYVU6ZbwNPh+cZvkEwLUBN1AuUBk1zd/8rgLsfcQ0Ad3/DzL4VfmJvDXzk7lvCEc4YM+sBfAl8K5xvJpXzA+2ADsC88O9Yj0OnZJaYUxBIbWbAJHe/84gXzDoSTFt9HcFc7ZV9uu0LnE1wEZA7wtHG4R4Gfu3uc8zsXIKr4lUmlXqTpTKnywyCPuUSjBAgmCStKdA5HB1tJpijJtkBDh1hlLxuwAp3/14K25YY0jkCqc3mAT9KmmmxmZl928xaEEwY9hfgbqBzWH430OTwSsysHtDa3ecDtwAtgMZllG8KfGDBx+YhScuPtl4APLiC3g4z+69w/Swza1xGv6cSHAL7IUEolLRtWxgCfSj7oiqbgA5m1jAcfZRc43Y10MrMzgy329CCOe5FAAWB1GLu/jZwL8EhjRUEh4BOJJi7/e9mtpzgehK3h6v8CXisjJPF9YE/h3UsI7j29W6CaYh/FJ4c7kZwLeuZwN+ArUnrH16uqvUmGwzcFK7zKkF4HN7vt8LlGz24RgMEM+Z2M7NCgvMG68pYbyPB4am3CS7FuCxc/hnB9MYPmdlbwJvAdw9fX+JLs4+KiMScRgQiIjGnIBARiTkFgYhIzCkIRERiTkEgIhJzCgIRkZhTEIiIxJyCQEQk5v4/ap4vOUqwwmYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "my_kernel = lambda X,Y : gauss_kernel(X,Y,sigma=0.3)\n",
    "my_mmd = lambda X,Y : quadratic_time_mmd(X[:,np.newaxis],Y[:,np.newaxis], my_kernel)\n",
    "\n",
    "statistics = two_sample_permutation_test(my_mmd, X, Y, num_permutations)\n",
    "my_statistic = my_mmd(X,Y)\n",
    "\n",
    "plot_permutation_samples(statistics, my_statistic)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This seems much better: the Gaussian kernel allows us to very clearly distinguish the two distributions.\n",
    "\n",
    "Now let us try the MMD with a linear kernel and compare it to the above.\n",
    "In particular, compare (and think about) the relationship to the simple squared mean difference statistic we computed above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAH2JJREFUeJzt3Xt4VOW59/HvLQEDVpFDdCOxDe4iAhECBopAtyAqKGzQohVfD6BUXrtr8Swnz0ov8EVFitWiUsHdciiIWkVLEaiVFjEgVU6Ws0Ysh1hQQFTo/f4xK2GABZkcZtaE/D7XlSsza5615n6SwG+edXiWuTsiIiKHOi7qAkREJD0pIEREJJQCQkREQikgREQklAJCRERCKSBERCSUAkJEREIpIEREJJQCQkREQmVEXUBFNGzY0HNycsq17kcfxb43a1Z59SRFlSlUykq/WonKkiVLtrt7VmntqnRA5OTkUFBQUK51u3SJfV+woNLKSY4qU6iUlX61EhUz25RIO+1iEhGRUAoIEREJpYAQEZFQVfoYhIgc7ttvv6WwsJC9e/dGXYpELDMzk+zsbGrWrFmu9RUQIseYwsJCTjzxRHJycjCzqMuRiLg7RUVFFBYW0qRJk3JtQ7uYRI4xe/fupUGDBgqHas7MaNCgQYVGkgoIkWOQwkGg4n8HCggREQmlgBCRSvXkk0+Sm5tLy5YtGTt2bMnyBx54gMaNG5OXl0deXh6zZ88GYOHChbRq1Yp27dqxdu1aAHbs2EH37t1x90j6APDyyy+zcuXKkuf33Xcfc+fOBaBLly7lvki3Kqm2B6kXrS8CIGfoopS/98ZRPVP+niKpsHz5cp599lkWL15MrVq16NGjBz179qRp06YA3Hbbbdx5550HrfPYY48xc+ZMNm7cyNNPP81jjz3Gww8/zPDhwyPdVfbyyy/Tq1cvWrRoAcBDDz0UWS1R0QhCRCrNqlWr6NChA3Xq1CEjI4PzzjuPWbNmHXWdmjVr8tVXX7Fnzx5q1qzJunXr+PTTTznvvPOOuM6bb77JWWedRefOnRk8eDC9evUCYqOUMWPGlLTLzc1l48aNAFx66aWcc845tGzZkgkTJpS0+c53vsOIESNo3bo1HTp0YMuWLfz1r3/l1Vdf5a677iIvL49169YxYMAAZsyYcVgtc+bM4dxzz6Vt27ZcccUV7Nq1C4ChQ4fSokULWrVqdVgoVhXVdgQhUh3ceissW1a528zLg7g9RwfJzc1lxIgRFBUVUbt2bWbPnk1+fn7J6+PHj2fy5Mnk5+fz2GOPUa9ePYYNG8agQYOoXbs2L774InfeeScPP/zwEd9/79693HjjjcybN4/vf//7XHnllQnVPXHiROrXr89XX31Fu3bt6Nu3Lw0aNGD37t106NCBkSNHcvfdd/Pss89yzz330Lt3b3r16sXll19+xG1u376dRx55hLlz53LCCScwevRoHn/8cW6++WZmzZrF6tWrMTN27NiRUI3pRiMIEak0zZs3Z8iQIVx44YX06NGD1q1bk5ER+xz605/+lHXr1rFs2TIaNWrEHXfcAUBeXh6LFi1i/vz5rF+/ntNOOw1358orr+Saa65hy5YtB73H6tWradKkCU2bNsXMuOaaaxKqbdy4cSWjhE8++YQ1a9YAUKtWrZIRyDnnnFMy4kjEokWLWLlyJZ06dSIvL49JkyaxadMmTjrpJDIzM/nJT37CSy+9RJ06dRLeZjrRCELkGHakT/rJNHDgQAYOHAjA8OHDyc7OBuDUU08taXPjjTeW/KdczN155JFHmDZtGjfffDMPPvggGzduZNy4cYwcOfKgtkc6NpGRkcG///3vkufF1wAsWLCAuXPn8re//Y06derQpUuXktdq1qxZsr0aNWqwb9++hPvq7lx44YVMmTLlsNcWL17MW2+9xdSpUxk/fjzz5s1LeLvpQiMIEalUW7duBeDjjz/mpZde4qqrrgLgs88+K2kza9YscnNzD1pv0qRJ9OzZk3r16rFnzx6OO+44jjvuOPbs2XNQu7POOosNGzawbt06gIP+c87JyWHp0qUALF26lA0bNgCwc+dO6tWrR506dVi9ejWLFpV+csqJJ57Il19+edQ2HTp0YOHChSVnX+3Zs4d//OMf7Nq1i507d3LJJZcwduxYllX2fr4USdoIwswmAr2Are6eGyz7f8B/A98A64Dr3X1H8NowYCCwHxjs7n9MVm0ikjx9+/alqKiImjVr8tRTT1GvXj0A7r77bpYtW4aZkZOTw69//euSdfbs2cOkSZOYM2cOALfffjt9+/alVq1ah306z8zMZMKECfTs2ZOGDRvSuXNnli9fXvLekydPJi8vj3bt2nHmmWcC0KNHD5555hlatWpFs2bN6NChQ6n96NevHzfeeCPjxo0LPTgNkJWVxQsvvMBVV13F119/DcAjjzzCiSeeSJ8+fdi7dy/uzhNPPFHGn2J6sGSdZ2xm/wXsAibHBcRFwDx332dmowHcfYiZtQCmAO2B04C5wJnuvv9o75Gfn+/lPRc587ux01z/4/+k+WmuuqvMMStZv9pVq1bRvHnzyt1oGluwYAFjxozhtddei7qUtBT292BmS9w9/wirlEjaLiZ3fxv4/JBlc9y9eAffIiA7eNwHmOruX7v7BmAtsbAQEZGIRHkM4gbgjeBxY+CTuNcKg2UiIkfVpUsXjR6SJJKAMLMRwD7gt8WLQpqF7vsys0FmVmBmBdu2bUtWiSIi1V7KA8LM+hM7eH21HzgAUgicHtcsG9gctr67T3D3fHfPz8rKSm6xIiLVWEoDwsx6AEOA3u4ef+7aq0A/MzvezJoATYHFqaxNREQOlszTXKcAXYCGZlYI3A8MA44H/hRcmLLI3W9y9xVmNh1YSWzX089KO4NJRESSK5lnMV3l7o3cvaa7Z7v78+7+fXc/3d3zgq+b4tqPdPf/dPdm7v7G0bYtIult1qxZmBmrV68ute0LL7zA5s2he5QTsmDBgsOuygZYtmxZyZTiZbVjxw5+9atflWmdsWPHHnZRXyLtLrnkkqPO1VTW9pVJV1KLSKWbMmUKnTt3ZurUqaW2rWhAHElVCYjZs2dz8sknV1r7yqSAEJFKtWvXLhYuXMjzzz9/WEA8+uijnH322bRu3ZqhQ4cyY8YMCgoKuPrqq8nLy+Orr74iJyeH7du3A1BQUECX4IrCxYsX07FjR9q0aUPHjh356KOPjljDN998w3333ce0adPIy8tj2rRp7N69mxtuuIF27drRpk0bXnnlFQBWrFhB+/btycvLo1WrVqxZs4ahQ4eybt068vLyuOuuuw7a9u7du+nZsyetW7cmNzeXadOmMW7cODZv3kzXrl3p2rUrEJucMD8/n5YtW3L//fcDhLYr7m+i243/+UyePJlWrVrRunVrrr322or82kJpsj6RY1mq5/smdqOdHj16cOaZZ1K/fn2WLl1K27ZteeONN3j55Zd59913qVOnDp9//jn169dn/PjxjBkz5qBpwcOcddZZvP3222RkZDB37lyGDx/OzJkzQ9vWqlWLhx56iIKCAsaPHw/EJg48//zzmThxIjt27KB9+/ZccMEFPPPMM9xyyy1cffXVfPPNN+zfv59Ro0axfPny0DmU3nzzTU477TRef/11IDbPU926dXn88ceZP38+DRs2BGDkyJHUr1+f/fv3061bNz744AMGDx58WLuybrfYihUrGDlyJAsXLqRhw4Z8/vlB1yVXCo0gRKRSTZkyhX79+gGx+YyK51KaO3cu119/fcnU1/Xr1y/Tdnfu3MkVV1xBbm4ut912GytWrCjT+nPmzGHUqFHk5eWVzOb68ccfc+655/KLX/yC0aNHs2nTJmrXrn3U7Zx99tnMnTuXIUOG8Je//IW6deuGtps+fTpt27alTZs2rFix4qDbl1Zku8XmzZvH5ZdfXhIcZf15JkIjCJFjWYrn+y4qKmLevHksX74cM2P//v2YGY8++ijuntAtROOn7C6ekhvg3nvvpWvXrsyaNYuNGzeW7HpKlLszc+ZMmjVrdtDy5s2b84Mf/IDXX3+d7t2789xzz3HGGWcccTtnnnkmS5YsYfbs2QwbNoyLLrqI++6776A2GzZsYMyYMbz33nvUq1ePAQMGHNSX8m730P4k+5asGkGISKWZMWMG1113HZs2bWLjxo188sknNGnShHfeeYeLLrqIiRMnlhxwLd4lcui02jk5OSxZsgTgoF1IO3fupHHj2Aw8L7zwQqm1HLrd7t2788tf/pLi63Pff/99ANavX88ZZ5zB4MGD6d27Nx988MFRp/revHkzderU4ZprruHOO+8smV48fp0vvviCE044gbp167JlyxbeeOPAiZlH2nYi243XrVs3pk+fTlFRbOJR7WISkbQ2ZcoULrvssoOW9e3bl9/97nf06NGD3r17k5+fT15eXsm9owcMGMBNN91UcpD6/vvv55ZbbuGHP/whNWrUKNnO3XffzbBhw+jUqRP795d+mVTXrl1ZuXJlyUHqe++9l2+//ZZWrVqRm5vLvffeC8C0adPIzc0lLy+P1atXc91119GgQQM6depEbm7uYQepP/zww5KD2iNHjuSee+4BYNCgQVx88cV07dqV1q1b06ZNG1q2bMkNN9xAp06dStaPb1fW7cZr2bIlI0aM4LzzzqN169bcfvvtpf5Myipp032ngqb7lqpM031LKqTldN8iIlK1KSBERCSUAkLkGFSVdx1L5ano34ECQuQYk5mZSVFRkUKimnN3ioqKyMzMLPc2dB2EyDEmOzubwsJCdEMtyczMJDs7u/SGR6CAEDnG1KxZkyZNmkRdhhwDtItJRERCKSBERCSUAkJEREIpIEREJJQCQkREQikgREQklAJCRERCKSBERCSUAkJEREIpIEREJFTSAsLMJprZVjNbHresvpn9yczWBN/rBcvNzMaZ2Voz+8DM2iarLhERSUwyRxAvAD0OWTYUeMvdmwJvBc8BLgaaBl+DgKeTWJeIiCQgaQHh7m8Dh95Fuw8wKXg8Cbg0bvlkj1kEnGxmjZJVm4iIlC7VxyBOdffPAILvpwTLGwOfxLUrDJYdxswGmVmBmRVoOmMRkeRJl4PUFrIs9G4n7j7B3fPdPT8rKyvJZYmIVF+pDogtxbuOgu9bg+WFwOlx7bKBzSmuTURE4qQ6IF4F+geP+wOvxC2/LjibqQOws3hXlIiIRCNpd5QzsylAF6ChmRUC9wOjgOlmNhD4GLgiaD4buARYC+wBrk9WXSIikpikBYS7X3WEl7qFtHXgZ8mqRUREyi5dDlKLiEiaUUCIiEgoBYSIiIRSQIiISCgFhIiIhFJAiIhIKAWEiIiEUkCIiEgoBYSIiIRSQIiISCgFhIiIhFJAiIhIKAWEiIiEUkCIiEgoBYSIiIRSQIiISCgFhIiIhFJAiIhIKAWEiIiEUkCIiEgoBYSIiIRSQIiISCgFhIiIhFJAiIhIqEgCwsxuM7MVZrbczKaYWaaZNTGzd81sjZlNM7NaUdQmIiIxKQ8IM2sMDAby3T0XqAH0A0YDT7h7U+BfwMBU1yYiIgdEtYspA6htZhlAHeAz4HxgRvD6JODSiGoTEREiCAh3/xQYA3xMLBh2AkuAHe6+L2hWCDQOW9/MBplZgZkVbNu2LRUli4hUS1HsYqoH9AGaAKcBJwAXhzT1sPXdfYK757t7flZWVvIKFRGp5qLYxXQBsMHdt7n7t8BLQEfg5GCXE0A2sDmC2kREJBBFQHwMdDCzOmZmQDdgJTAfuDxo0x94JYLaREQkEMUxiHeJHYxeCnwY1DABGALcbmZrgQbA86muTUREDsgovUnlc/f7gfsPWbweaB9BOSIiEkJXUouISCgFhIiIhCpzQJhZPTNrlYxiREQkfSQUEGa2wMxOMrP6wN+B35jZ48ktTUREopToCKKuu38B/Aj4jbufQ+x6BhEROUYlGhAZZtYI+DHwWhLrERGRNJFoQDwI/BFY6+7vmdkZwJrklSUiIlFL9DqIz9y95MC0u6/XMQgRkWNboiOIXya4TEREjhFHHUGY2bnEJtLLMrPb4146idiNfkRE5BhV2i6mWsB3gnYnxi3/ggMT64mIyDHoqAHh7n8G/mxmL7j7phTVJCIiaSDRg9THm9kEICd+HXc/PxlFiYhI9BINiN8DzwDPAfuTV46IiKSLRANin7s/ndRKREQkrSR6musfzOx/zKyRmdUv/kpqZSIiEqlERxD9g+93xS1z4IzKLUdERNJFQgHh7k2SXYiIiKSXhALCzK4LW+7ukyu3HBERSReJ7mJqF/c4E+gGLAUUECIix6hEdzH9PP65mdUFXkxKRSIikhbKe0/qPUDTyixERETSS6LHIP5A7KwliE3S1xyYnqyiREQkeokegxgT93gfsMndC5NQj4iIpImEdjEFk/atJjajaz3gm4q8qZmdbGYzzGy1ma0ys3ODi+/+ZGZrgu/1KvIeIiJSMQkFhJn9GFgMXEHsvtTvmllFpvt+EnjT3c8CWgOrgKHAW+7eFHgreC4iIhFJdBfTCKCdu28FMLMsYC4wo6xvaGYnAf8FDABw92+Ab8ysD9AlaDYJWAAMKev2RUSkciR6FtNxxeEQKCrDuoc6A9gG/MbM3jez58zsBOBUd/8MIPh+StjKZjbIzArMrGDbtm3lLEFEREqT6H/yb5rZH81sgJkNAF4HZpfzPTOAtsDT7t4G2E0Zdie5+wR3z3f3/KysrHKWICIipSntntTfJ/bJ/i4z+xHQGTDgb8Bvy/mehUChu78bPJ9BLCC2mFkjd//MzBoBW4+4BRERSbrSRhBjgS8B3P0ld7/d3W8jNnoYW543dPd/Ap+YWbNgUTdgJfAqB2aN7Q+8Up7ti4hI5SjtIHWOu39w6EJ3LzCznAq878+B35pZLWA9cD2xsJpuZgOBj4mdMSUiIhEpLSAyj/Ja7fK+qbsvA/JDXupW3m2KiEjlKm0X03tmduOhC4NP+UuSU5KIiKSD0kYQtwKzzOxqDgRCPlALuCyZhYmISLSOGhDuvgXoaGZdgdxg8evuPi/plYmISKQSvR/EfGB+kmsREZE0Ut6roUVE5BingBARkVAKCBERCaWAEBGRUAoIEREJpYAQEZFQCggREQmlgBARkVAKCBERCaWAEBGRUAoIEREJpYAQEZFQCggREQmlgBARkVAKCBERCaWAEBGRUAoIEREJpYAQEZFQCggREQkVWUCYWQ0ze9/MXgueNzGzd81sjZlNM7NaUdUmIiLRjiBuAVbFPR8NPOHuTYF/AQMjqUpERICIAsLMsoGewHPBcwPOB2YETSYBl0ZRm4iIxEQ1ghgL3A38O3jeANjh7vuC54VA4ygKExGRmJQHhJn1Ara6+5L4xSFN/QjrDzKzAjMr2LZtW1JqFBGRaEYQnYDeZrYRmEps19JY4GQzywjaZAObw1Z29wnunu/u+VlZWamoV0SkWkp5QLj7MHfPdvccoB8wz92vBuYDlwfN+gOvpLo2ERE5IJ2ugxgC3G5ma4kdk3g+4npERKq1jNKbJI+7LwAWBI/XA+2jrEdERA5IpxGEiIikEQWEiIiEinQXU3WVM/T1hNtOXV8EQL8yrHMkG0f1rPA2RKT60AhCRERCKSBERCSUAkJEREIpIEREJJQCQkREQikgREQklAJCRERCKSBERCSUAkJEREIpIEREJJQCQkREQikgREQklAJCRERCKSBERCSUAkJEREIpIEREJJQCQkREQikgREQklAJCRERCKSBERCSUAkJEREKlPCDM7HQzm29mq8xshZndEiyvb2Z/MrM1wfd6qa5NREQOiGIEsQ+4w92bAx2An5lZC2Ao8Ja7NwXeCp6LiEhEUh4Q7v6Zuy8NHn8JrAIaA32ASUGzScClqa5NREQOiPQYhJnlAG2Ad4FT3f0ziIUIcEp0lYmISGQBYWbfAWYCt7r7F2VYb5CZFZhZwbZt25JXoIhINRdJQJhZTWLh8Ft3fylYvMXMGgWvNwK2hq3r7hPcPd/d87OyslJTsIhINRTFWUwGPA+scvfH4156FegfPO4PvJLq2kRE5ICMCN6zE3At8KGZLQuWDQdGAdPNbCDwMXBFBLWJiEgg5QHh7u8AdoSXu6WyFhEROTJdSS0iIqEUECIiEkoBISIioRQQIiISSgEhIiKhojjNVSKSM/T1yN5746iekb23iJSPRhAiIhJKASEiIqG0i0lSIqrdW9q1JVJ+GkGIiEgoBYSIiIRSQIiISCgFhIiIhFJAiIhIKAWEiIiEUkCIiEgoBYSIiIRSQIiISCgFhIiIhFJAiIhIKAWEiIiE0mR9ckyL8h4Ypfnn+g4A5AxdVKnb1QSFUlk0ghARkVAaQYgcYzS1ulSWtBtBmFkPM/vIzNaa2dCo6xERqa7SKiDMrAbwFHAx0AK4ysxaRFuViEj1lG67mNoDa919PYCZTQX6ACsjrUpEJESUJ0GkYpdeWo0ggMbAJ3HPC4NlIiKSYubuUddQwsyuALq7+0+C59cC7d3953FtBgGDgqfNgI9SXmjqNQS2R11ERNT36qs69z/Zff+eu2eV1ijddjEVAqfHPc8GNsc3cPcJwIRUFhU1Mytw9/yo64iC+l49+w7Vu//p0vd028X0HtDUzJqYWS2gH/BqxDWJiFRLaTWCcPd9ZnYz8EegBjDR3VdEXJaISLWUVgEB4O6zgdlR15FmqtUutUOo79VXde5/WvQ9rQ5Si4hI+ki3YxAiIpImFBARK21qETM73symBa+/a2Y5ca8NC5Z/ZGbdU1l3ZShv382sgZnNN7NdZjY+1XVXhgr0/UIzW2JmHwbfz0917RVVgb63N7NlwdffzeyyVNdeURX59x68/t3g7/7OlBTs7vqK6IvYgfh1wBlALeDvQItD2vwP8EzwuB8wLXjcImh/PNAk2E6NqPuUor6fAHQGbgLGR92XFPe9DXBa8DgX+DTq/qSw73WAjOBxI2Br8fOq8FWRvse9PhP4PXBnKmrWCCJaJVOLuPs3QPHUIvH6AJOCxzOAbmZmwfKp7v61u28A1gbbqyrK3Xd33+3u7wB7U1dupapI39939+Jrg1YAmWZ2fEqqrhwV6fsed98XLM8EqtoB1Ir8e8fMLgXWE/u9p4QCIlqJTC1S0ib4x7ETaJDguumsIn2v6iqr732B99396yTVmQwV6ruZ/cDMVgAfAjfFBUZVUO6+m9kJwBDgwRTUWUIBES0LWXbop6IjtUlk3XRWkb5XdRXuu5m1BEYD/7cS60qFCvXd3d9195ZAO2CYmWVWcn3JVJG+Pwg84e67Kr2qo1BARKvUqUXi25hZBlAX+DzBddNZRfpe1VWo72aWDcwCrnP3dUmvtnJVyu/d3VcBu4kdh6kqKtL3HwCPmtlG4FZgeHBRcVIpIKKVyNQirwL9g8eXA/M8drTqVaBfcNZDE6ApsDhFdVeGivS9qit3383sZOB1YJi7L0xZxZWnIn1vEvyniZl9j9hknRtTU3alKHff3f2H7p7j7jnAWOAX7p78M/iiPrJf3b+AS4B/EDu7YUSw7CGgd/A4k9hZC2uJBcAZceuOCNb7CLg46r6kuO8biX2y2kXsU1eLVNcfRd+Be4h9cl4W93VK1P1JUd+vJXaAdhmwFLg06r6kqu+HbOMBUnQWk66kFhGRUNrFJCIioRQQIiISSgEhIiKhFBAiIhJKASEiIqEUEJK2gllbi2fv/KeZfRr3vFYZtnODmf1HGdqfb2YdytrOzH5mZldXVvuKMLMMM9uRjG1L9ZF2d5QTKebuRUAegJk9AOxy9zHl2NQNxM6b/2eC7c8HtgOLytLO3Z+q5PYikdIIQqokM+tvZouD0cSvzOy44FPzi8G9Epab2WAzu5JYyEwLG3mY2W1mtjK4v8D/mtl/Aj8B7gradzSzPsHc/O+b2RwzO+UI7R4xs1vLsN349mea2byg/dKQ+wA8ZmaD4p4/Yma3mNlJwXpLzewDM+sV8rO6wMxejnv+jJldEzxuZ2Z/tti9Jd4ws1Mr4/cjxwaNIKTKMbNc4DKgo7vvM7MJxKYtWAc0dPezg3Ynu/sOM/s5cLO7LwvZ3N3A99z9m7j2zwHb3X1ssJ16wKvu7mZ2E3CHuw8JaXdJGbcb334K8IC7/8FiE9Ad+uFtKjCKA/cqvgLoCnwF9HH3L83sFGAh8FqCP8fjgSeJXcW7Pdjd9TAw6OhrSnWhgJCq6AJis3kWWGyq/NrEpkj+I9DMzJ4EZgNzEtjWCuB/zewV4OUjtPkuMD04jnE8sakSKmO7QEkANXT3PwC4+2H3uXD398zs9OATfjbwT3ffHIyIRptZZ+DfwOlm1hBI5PhDc6AlMDf4OdYgNm2JCKCAkKrJgInufu9hL5i1Ai4GBhO7X0Jpn4a7A+cRu1HLPcHo5FBPEZscbbaZXQAcdqvIcm43XiJz3swk1qccYiMKgOuIzfjZNhhNFRKbzyfePg4ekRS/bsAH7v7DBN5bqiEdg5CqaC7w4+CTcvHZTt81syzA3P33wP1A26D9l8CJh27EzGoA2e4+D7gLyCJ2W8tD29cFPrXYx+z+ccsrul0A3P1fwHYz++9g/UwzqxPS76nEdqX9iFhYFNe2NQiHCwm/adQmoKWZ1QpGK8X3sV4JNDaz9sH71rLYfSZEAAWEVEHu/iGxG6jMNbMPiO1KOpXYPPpvm9ky4FlgeLDKb4DnQg5SZwC/C7axFBjt7l8CrxALoPfNrCOx2TNnAX8GtsStf2i7sm433tXAHcE67xALlUP7/fdg+QZ33xosfhHoaGYFxI5LrAlZbwOx3VwfApODmvDYneguBx43s78D7xO774AIgGZzFRGRcBpBiIhIKAWEiIiEUkCIiEgoBYSIiIRSQIiISCgFhIiIhFJAiIhIKAWEiIiE+v9JWwjBmLDX3gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# IMPLEMENT: visualize test using linear kernel\n",
    "my_kernel = lambda X,Y : linear_kernel(X,Y)\n",
    "my_mmd = lambda X,Y : quadratic_time_mmd(X[:,np.newaxis],Y[:,np.newaxis], my_kernel)\n",
    "\n",
    "statistics = two_sample_permutation_test(my_mmd, X, Y, num_permutations)\n",
    "my_statistic = my_mmd(X,Y)\n",
    "\n",
    "plot_permutation_samples(statistics, my_statistic)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is almost the same result as we had for doing the simple squared statistic above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# IMPLEMENT (optional): try mix and matching kernels and data sources\n",
    "# e.g. try a periodic kernel on data that differs in frequency space"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Alternative methods to estimate the null distribution\n",
    "\n",
    "There are multiple other way to get our hands on the distribution of the MMD test statistic under the null hypothesis.\n",
    "\n",
    "We know that asymptotically, the distribution is an infinite sum of Chi-square variables -- intractable unfortunately.\n",
    "Techniques to approximate it include\n",
    "\n",
    "* moment matching using a Gamma distribution: fast, but doesn't result in a consistent test\n",
    "* a spectral approximation: use Eigenvalues of the kernel matrix to sample from the null distribution in closed form. Costly (cubic!) for large sample sets.\n",
    "* wild-bootstrap: a technique used for correlated samples that is similar to permuting\n",
    "* linear time statistics: for those, one can often show that the null distribution is Gaussian and its variance can be estimated in closed form.\n",
    "\n",
    "We do not cover those here since they require some more details.\n",
    "The permutation test, while potentially slower, is easy to understand and works for most applications (if implemented well).\n",
    "\n",
    "Some resources:\n",
    "* [a Shogun notebook](http://www.shogun-toolbox.org/notebook/latest/mmd_two_sample_testing.html), old but contains many null approximation methods\n",
    "* Linear time tests based on analytic function comparisons (see notes at end of section)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MMD witness function in the RKHS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One nice feature of the MMD is that we can see where the density functions are different via querying the (empirical) MMD witness function.\n",
    "This is just the difference of the empirical mean embeddings in the RKHS and thus computable (remember the MMD is the RKHS norm of this witness function).\n",
    "See the slides for details.\n",
    "\n",
    "$$f^*(\\cdot)=\\hat \\mu_X(\\cdot) - \\hat \\mu_Y(\\cdot) $$\n",
    "\n",
    "Let's define a grid and evaluate the witness function on it to see where X and Y differ most."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/heiko/software/anaconda3/envs/ds3_kernels/lib/python3.6/site-packages/matplotlib/axes/_axes.py:6462: UserWarning: The 'normed' kwarg is deprecated, and has been replaced by the 'density' kwarg.\n",
      "  warnings.warn(\"The 'normed' kwarg is deprecated, and has been \"\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAD8CAYAAACfF6SlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAH65JREFUeJzt3Xl8VfWd//HX597khmwkkIRFSAIIsgvaFNs6tuLSolZotU5pbWuXGWbG8pjOo9MFuzgt7Uz76zad+am/3zgdZ7ro8Kutjqi0aF3q3hIUlVUjCIQQCCELhMDN8vn9kWivMQjknuTccN7PxyOP3HPu957vhyXv+833fM+55u6IiEi0xMIuQEREhp7CX0QkghT+IiIRpPAXEYkghb+ISAQp/EVEIkjhLyISQQp/EZEIUviLiERQVtgFHE9paalPmjQp7DJERIaV9evXH3D3shO1y9jwnzRpEtXV1WGXISIyrJjZzpNpp2kfEZEIUviLiESQwl9EJIIU/iIiEaTwFxGJIIW/iEgEBRL+ZrbIzLaZWY2ZrThOmz83s81mtsnM7giiXxERGZi01/mbWRy4GbgUqAXWmdlqd9+c0mYacANwvrs3mdmYdPsVEZGBC+IirwVAjbtvBzCzVcASYHNKm78Ebnb3JgB33x9AvxJxt2y4JbS+r59/fWh9iwQhiGmfCcDulO3a3n2pzgLOMrMnzewZM1sUQL8iIjJAQYz8rZ993k8/04ALgYnA42Y2x92b33Ags2XAMoCKiooAShMRkf4EMfKvBcpTticCdf20ucfdO9x9B7CNnjeDN3D3W929yt2ryspOeF8iEREZoCDCfx0wzcwmm1kCWAqs7tPmf4CFAGZWSs800PYA+hYRkQFIO/zdvRNYDqwFtgC/dPdNZrbSzBb3NlsLNJrZZuAR4Ivu3phu3yIiMjCB3NLZ3dcAa/rsuzHlsQOf7/0SEZGQ6QpfEZEIUviLiESQwl9EJIIU/iIiEaTwFxGJIIW/iEgEKfxFRCJI4S8iEkEKfxGRCFL4i4hEkMJfRCSCFP4iIhGk8BcRiSCFv4hIBCn8RUQiSOEvIhJBCn8RkQhS+IuIRFAg4W9mi8xsm5nVmNmKfp7/pJk1mNmG3q+/CKJfEREZmLQ/w9fM4sDNwKVALbDOzFa7++Y+Tf+fuy9Ptz8REUlfECP/BUCNu2939ySwClgSwHFFRGSQBBH+E4DdKdu1vfv6utrMXjCzX5lZeX8HMrNlZlZtZtUNDQ0BlCYiIv0JIvytn33eZ/teYJK7nw38Dvhpfwdy91vdvcrdq8rKygIoTURE+hNE+NcCqSP5iUBdagN3b3T3Y72b/w68LYB+RURkgIII/3XANDObbGYJYCmwOrWBmY1P2VwMbAmgXxERGaC0V/u4e6eZLQfWAnHgNnffZGYrgWp3Xw38rZktBjqBg8An0+1XREQGLu3wB3D3NcCaPvtuTHl8A3BDEH2JiEj6dIWviEgEKfxFRCJI4S8iEkEKfxGRCFL4i4hEkMJfRCSCFP4iIhGk8BcRiSCFv4hIBCn8RUQiSOEvIhJBCn8RkQhS+IuIRJDCX0QkghT+IiIRpPAXEYkghb+ISAQFEv5mtsjMtplZjZmteIt2HzIzN7OqIPoVEZGBSTv8zSwO3AxcBswCPmJms/ppVwj8LfCHdPsUEZH0BDHyXwDUuPt2d08Cq4Al/bT7FvA94GgAfYqISBqCCP8JwO6U7drefa8zs3OAcne/L4D+REQkTUGEv/Wzz19/0iwG/DPw9yc8kNkyM6s2s+qGhoYAShMRkf5kBXCMWqA8ZXsiUJeyXQjMAR41M4BxwGozW+zu1akHcvdbgVsBqqqqHJHXPPKdN+9rfiH4fiZfEPwxRTJQECP/dcA0M5tsZglgKbD6tSfdvcXdS919krtPAp4B3hT8IiIydNIOf3fvBJYDa4EtwC/dfZOZrTSzxekeX0REghfEtA/uvgZY02ffjcdpe2EQfYqIyMDpCl8RkQhS+IuIRJDCX0QkghT+IiIRpPAXEYkghb+ISAQp/EVEIkjhLyISQYFc5CVy2tjx+Mm1a2pJv6+FN6R/DJEB0shfRCSCFP4iIhGk8BcRiSCFv4hIBCn8RUQiSOEvIhJBCn8RkQhS+IuIRJDCX0QkggIJfzNbZGbbzKzGzFb08/xfm9mLZrbBzJ4ws1lB9CsiIgOTdvibWRy4GbgMmAV8pJ9wv8Pd57r7fOB7wI/S7VdERAYuiJH/AqDG3be7exJYBSxJbeDurSmb+YAH0K+IiAxQEDd2mwDsTtmuBc7r28jMPgt8HkgAF/V3IDNbBiwDqKioCKA0ERHpTxAjf+tn35tG9u5+s7ufCXwZ+Fp/B3L3W929yt2rysrKAihNRET6E0T41wLlKdsTgbq3aL8K+EAA/YqIyAAFEf7rgGlmNtnMEsBSYHVqAzOblrJ5BfByAP2KiMgApT3n7+6dZrYcWAvEgdvcfZOZrQSq3X01sNzMLgE6gCbgunT7FRGRgQvkk7zcfQ2wps++G1Mefy6IfkREJBi6wldEJIIU/iIiEaTwFxGJIIW/iEgEKfxFRCJI4S8iEkEKfxGRCApknb9I1NzS/EL6B9lwyym/5Pr516ffrwga+YuIRJLCX0QkgjTtIwP3yHfCrkBEBkgjfxGRCFL4i4hEkKZ9JG2BrHwRkSGlkb+ISAQp/EVEIkjhLyISQYGEv5ktMrNtZlZjZiv6ef7zZrbZzF4ws4fMrDKIfkVEZGDSPuFrZnHgZuBSoBZYZ2ar3X1zSrPngCp3P2JmfwN8D/hwun2LDGs7Hj/11zS1DKyvhTcM7HVy2gpi5L8AqHH37e6eBFYBS1IbuPsj7n6kd/MZYGIA/YqIyAAFEf4TgN0p27W9+47nM8BvAuhXREQGKIh1/tbPPu+3odnHgCrgPcd5fhmwDKCioiKA0kREpD9BjPxrgfKU7YlAXd9GZnYJ8FVgsbsf6+9A7n6ru1e5e1VZWVkApYmISH+CCP91wDQzm2xmCWApsDq1gZmdA/wbPcG/P4A+RUQkDWmHv7t3AsuBtcAW4JfuvsnMVprZ4t5m3wcKgDvNbIOZrT7O4UREZAgEcm8fd18DrOmz78aUx5cE0Y+IiARDV/iKiESQwl9EJIIU/iIiEaTwFxGJIIW/iEgEKfxFRCJI4S8iEkEKfxGRCFL4i4hEkMJfRCSCFP4iIhGk8BcRiSCFv4hIBCn8RUQiSOEvIhJBCn8RkQhS+IuIRJDCX0QkggIJfzNbZGbbzKzGzFb08/y7zexZM+s0sw8F0aeIiAxc2p/ha2Zx4GbgUqAWWGdmq919c0qzXcAngS+k25+cwCPfCbsCERkGgvgA9wVAjbtvBzCzVcAS4PXwd/dXe5/rDqA/ERFJUxDTPhOA3Snbtb37TpmZLTOzajOrbmhoCKA0ERHpTxDhb/3s84EcyN1vdfcqd68qKytLsywRETmeIMK/FihP2Z4I1AVwXBERGSRBhP86YJqZTTazBLAUWB3AcUVEZJCkfcLX3TvNbDmwFogDt7n7JjNbCVS7+2ozeztwNzAKuNLMvunus9PtWyRqbml+YWAv3HBL2n1fP//6tI8hmSOI1T64+xpgTZ99N6Y8XkfPdJCIiGQAXeErIhJBgYz8ZXjpdmg4lkVDMotuN7ocujG6vWeZVmmik4m5HSRiA1q0JZlox+PpH6Op5eTaLbwh/b5k0Cn8T2NHu4yNrblsaMlje1sOu9sT7GlPUHs0m2T3W//SZzhnjOigIi9JZe4xphUc471jWijP6xii6kVkMCn8TyMH25I8XV/E+uY8nm3OY1NrLh3eE/Kjsjspz00yo7CdS8e0MDG3g7KcDrLMiVtP2McNzGDf0Sx2tuew60iCnUcSPNhQxKo9WXxr2xnMLmznsrEtLBrbwtSCYyH/iUVkoBT+Q2EQ77fzaluCBxtG8uD+kVQ35dNNJTmxbs4e2c6nKw9wbvERzi0+QllOZ1r97DqS4Lf7RvKbfUX8oGYcP6gZx9T8o3yq8gDdBUbMNEUkMpwo/IehmsM53L23mLX7iqhpGwHAjIJ2PjtlPwvLDjFnZHvg8/UVeUmWTT7AsskHqD+axdp9Rdy1dxRf3TyRktwrWFjxLJOL6rD+rvcWkYyj8B8mmpJx7qsv5ld1o3i+JY+4Oe8YdZiPlTdy8ZhWynOHbi5+3IhOrqts5BMVjazdP5IVW0r41baLmDSyjgsrn2VMXvOQ1SIiA6Pwz2DdDo8dKGBVbQkPNRTS4TFmFLTztel1LB7fzJg0p3LSZQaLxrbycvZTPLfvLJ7aM5efvng588bUsLByPdmxrlDri4LO7hjNRwtpOVZATlaSwsQRCrKPENdKLTkBhX8GOpiMc+ee0dy+ezS72nMoTXTwiYpGrj6jiVkjjx73dQO++jNN8RhUjd/K7LLtPLVnLuvrZ7KvbRQfPOv3FCSOX6+cGnfY0XIGrzRPoKm9kINHR9KazOfN91Z08rPbKUwcoSS3lcqR9VQW7aUw0R5G2ZKhFP4Z5LnmXH6+u4T76otJdsdYMOowX5xWz/vGtg6LNfe5WUkurlxPeeF+7n/lfH6+6TKuOutRxuY3hV3asOYO25sn8OSeudS3lZKIJxk9opUJhQ3MHfEKo0YcoijnMMmubA4l82hN5nMomcehZB47msez6cAUAEpym5nU+0ZQWVSv38wiTuEfsmS3saa+iP/cVcrzLXkUxLtYOuEg15Y3Mr1weC6lPGv0bopy1nLXSxdyx+b38f4zn2Da6Nqwyxp23OGV5gk8teds6ttKKMo5xKLJTzO7dPtJT+u4Q8ORYl5tHc+rLeN5fv9U1u+bQXasg6mjaplZ8iqTivaSFdPnLEWNwj8kDceyuH33aG7fXUJDMpsp+Uf51sw9fPCMJgqyhv8P4tj8Jj4++7fc/dJ7uPvl9/Ce8udYMH6zVgOdpNZjedzz8rvZ21ZK8QBC/zVmMCa/mTH5zSwYv4XO7hi7W8ey7WAFLzVVsKVxMjnxJNNG7WZGyatUjqzX+YKIUPgPIXfY0JLLz3aVcl99ER0eY2FpK5+s3M0FJYeJnWbBWJBoZ+msB/nN9nfy+93ncqRjBAsrnw27rIzXcKSIO7deTEd31oBD/3iyYt1MLt7L5OK9XNr9R3a2jmdrYyUvN5Wz8cCZ5GYdZcboncwsfZUJBQ16sz6NKfyHwNEu4776Yn62q4QXWnumdq4tP8gnKg4wJT8ZdnmDKjvWxZVnPkFe1lHW1c+iMHGEqvFbwy4rY+1uLeOuly4kO9bFR2c9QNkgLpuNx5wpxXVMKa6js/sP7Gg+g82Nk3jxwJk8t386IxOHmVGyk+mjdzIu/+DJLygI4PbRqXQr6cGh8B9ER5Kd3PRwDauenMnBjiymnmZTOyfLDC6qXM/hZB4P73obhYkjTC/ZFXZZGeflgxO5t+bPGJnTxjUzHqYop23I+s6KdTNtdC3TRtdyrCuLmqZytjROorp+Jn/cO5uRicNMG72bs0btYkLhgbe+ojuIm8ileqsbyukmcgOm8B9EOVlx7n9xL1Wj2riuopF3jT4c2V+jY+ZcMfVJ2rZczH2vnE9+op2JhQ1hl5Uxnt8/lQd2LGBcfiNXT3+UvOzwTvbnxDuZXbqD2aU7aO9MUNM0kZcOVrBh31msr59JfnY7ZxbXMqW4jsqR9eRk6WZ/w5G5Z+bJnaqqKq+urg67jLQd7ehixBPfG5K+wlrnfyraOxL8YvMi2jtyuHb2WkpyW8MuKXTr66fz0M63M6VoD4unPUYinplLMI91ZbG9eQIvHaxgR/N4kt0JjG4mFB5gUlEdk4vqGJvfNLT3eZp8waAefjhOOZnZenevOlG7QEb+ZrYI+Bd6PsbxJ+7+3T7P5wA/A94GNAIfdvdXg+g7043IjoddQkbJzU5yzfSHuH3zIu7cehEfm72WgghffLSrdQwP73wb00btYvHUxzN6pU1OvJOZJTuZWbKTrm6j7nApO1rOYEfLGTxRO58naueTiCWZUHiAiYX7mVC4n/EFjbqeIEOlHf5mFgduBi4FaoF1Zrba3TenNPsM0OTuU81sKfC/gA+n2/dbuSXgk05pGQYj8qFUPKKNq6c/wn9vvpRfb7uQa2evjeQ688PJXO6tuYDiEYe5fMrTGR38fcVjTvnIBspHNvDu8udp68hhZ8t4ag+NofZQGY/XzgcgZl2MzWtibH4j4/IPMjb/IKW5LcQj+O+daYIY+S8Aatx9O4CZrQKWAKnhvwT4Ru/jXwE3mZl5ps45yaAbl3+QK6c+wV0vLeT3u8/h4sr1YZc0pLrdWF3zZyS7svnzGb8b9vPm+dnHmFX6KrNKXwWgvTPBnkNl1B4aw97DJWxunMyG/dOBnjeEstxmSvNaKBnRQkluC6NzWxk14lDG3Ro8rEHkUEw3BRH+E4DdKdu1wHnHa+PunWbWApQABwLoX4apqaP2cO7Yrayvn8nkojqmFO8Nu6Qh89ju+dQeGssVZz5BWd5JfjziMJKblWTqqD1MHbUH6LnGpflYAfvaRr/+tbNl7Ou3noCeN4XinMMU5bQxMqeNkYk/Pc7Pbicv6xiJeMcbF00EvbLoJLlDt8foxnAHx8CNnreu1wr0P911yV573PPdLLXlm9/wOuf+NVnxwf2I9SDCv7/1K33/NCfTBjNbBiwDqKioSKuojDpRk0m1ZJhPd3Sx5KYneXTP5ay45gJKC3LCLmnQ/XZjPd/7w3o+9o4Kvv2Bwfugn+Hg0NEOXmloo2b/YWr2H2bXwTZqm9qpbWqncf+br4FJxGOUFCQYnZ+gKDebvEQWeYk4eYk4uYk4udlxsmI9H0kXM4iZYfSETWdXNx3dTkdnN53dTkdXN8c6e786unof937v6Hmc7Op5nOzqpqOz5/WdXd10D/IvKI81PMOdf/2uQe0jiPCvBcpTticCdcdpU2tmWUARcLDvgdz9VuBW6FntE0BtkuFGZMf514+cw5U3PcEX73ye2z75duw0Xg+740AbX7zzeeZNLOLr758VdjmhKxyRzfzyYuaXF7/puSPJTuqae98IDidpbDtGY1uSg4eTNLYlaW3voPlIO0eSnRxJdtGe7OJIRxfd7hxvQjk7bmTHY2TFer7nZMXIyY6/4XtBThYl+TFysnq2E71f2fEYWXEjOxYjHjOy40YsZsTNsNfeaFLebF6b1X6tlm733v3g9NR4vJnv8UW56f/lnkAQ4b8OmGZmk4E9wFLgo33arAauA54GPgQ8rPl+ec30cYV85bIZfOPezfzs6Z1c965JYZc0KI52dPE3v1hPPG7cfO255GRpJdhbyUtkMXVMIVPHFJ7ya733DcDpCV0D4jE7rQcWpyrtSSV37wSWA2uBLcAv3X2Tma00s8W9zf4DKDGzGuDzwIp0+5XTy3XvmsTC6WX845otbKs/FHY5g+KWR19ha/0hfnjNPCaOygu7nNOaWe+ovHeEnxWPKfj70EVekjEOHD7Goh8/Rkl+DvcsP/+0ukbi5X2HuPxfH+eKueP58dJzwi5HTmMne5HX4J5OFjkFpQU5/OCaeWzbd4gfPfhS2OUEprvbueGuF8nPyeJrmueXDKHwl4xy4fQxfGRBOT95fDsb95weSyD/e90uqnc28dXLZ0ZiNZMMDwp/yTgrLptJSUEOX/71C3R2De8rQfe1HuW7a7byziklfOhtE8MuR+R1Cn/JOEW52axcPJtNda38xxM7wi4nLd+8dxPHurr5p6vm6oSjZBSFv2SkRXPGcemssfzz715iV+ORsMsZkAc372PNi/V87uJpTC7ND7sckTdQ+EtGMjO+tWQOWbEYX7n7xeNeDJOpDh/r5MZ7NjJ9bCF/ecGUE79AZIgp/CVjjSsawZcvm8ETNQf49bN7wi7nlPzwgW3Utx7ln66aSyJLP2aSefS/UjLatQsqqKocxbfv38yBw+F9utWp2FzXyk+fepWPLqjgbZWjwi5HpF8Kf8losZjxnavm0nask5X3bj7xC0LW3e3ceM9GivMSfPF908MuR+S4FP6S8aaNLeSzC6ey+vk6Htm2P+xy3tJdz+2hemcTKxbNoDgvEXY5Isel8Jdh4W8uPJOpYwr42t0baTvWGXY5/Wpp7+A7a7ZwbkWx1vRLxlP4y7CQkxXnO1fNZU9ze8be+uFHD2yj6UiSlUvmEItpTb9kNoW/DBtvnzSaa8+r4D+f3MHzu5vDLucNNu5p4efP7OTj76hkzoSisMsROSGFvwwrX75sBqUFOay460U6MuTWD6+d5B2Vl+Dz79VJXhkeFP4yrIwckc3KJbPZsjdzbv3wq/W1PLurmRsun0lRbnbY5YicFIW/DDuL5oznvbPG8s8PvsTOxrZQaznYluS7v91KVeUorjpnQqi1iJwKhb8MSyuXzCERD//WD1//n40cOtrBtz+ok7wyvCj8ZVgaVzSCL102gydrGvll9e5Qarj3+Truf3Evf3fJWcwYNzKUGkQGKq3wN7PRZvagmb3c+73fa9nN7Ldm1mxm96XTn0iqaxdU8M4pJXxj9WZq9g/t5/7uP3SUr9+zkXnlxfzVu3XjNhl+0h35rwAecvdpwEMc/4PZvw98PM2+RN4gFjN+vHQ+uYk4y+94jqMdXUPSr7vzlbs20p7s4ofXzCMrrl+gZfhJ93/tEuCnvY9/Cnygv0bu/hAwtEMziYSxI0fwwz+fx9b6Q3z7/qG5989dz+7hd1v28cX3TWfqmIIh6VMkaOmG/1h33wvQ+31M+iWJnJqF08ew7N1T+MUzu/jNi3sHta+9Le18495NvH3SKD51/uRB7UtkMGWdqIGZ/Q4Y189TXw26GDNbBiwDqKioCPrwchr7wnun84cdB/nSr19gzoQiykfnBd6Hu/OlX71AZ5fzg2vmEdfqHhnGTjjyd/dL3H1OP1/3APvMbDxA7/e0brno7re6e5W7V5WVlaVzKImYRFaM/730HHD421XPDcrVv//11Ks8/vIBbrh8BpUl+lhGGd7SnfZZDVzX+/g64J40jycyYBUleXz36rN5blcz31+7LdBj3/VsLSvv28wlM8fwsfMqAz22SBjSDf/vApea2cvApb3bmFmVmf3ktUZm9jhwJ3CxmdWa2fvS7FekX1ecPZ5rz6vg1se28/21WwO5AOy+F+r4wp3P864zS7jpo+fqYi45LZxwzv+tuHsjcHE/+6uBv0jZviCdfkROxcolc+h25+ZHXqHxcJJvf2DOgJdjPrCpns+t2kBV5Wj+/RNVjMiOB1ytSDjSCn+RTBSPGf/0wbmU5Odw0yM1HGxL8q8fOeeUg/vRbftZfsdzzJ1QxG2fejt5Cf24yOlDV6fIacnM+ML7pvMPV87igc37uO62P9J6tOOkX/9UzQH+6ufrmTa2gJ9+egEFOQp+Ob0o/OW09qnzJ/MvS+ezfmcTH/63Z1i/8yBd3f2fB3B3/rjjIJ9b9RzX/ecfmVSSz88/c55u0yynJQ1n5LS3ZP4EivMSXP+L9Vz9f56mOC+b95xVxsLpY3jPWWXEYsZdz9Zyxx928fL+wxSOyOLa8ypZftFURufrQ9jl9GRh3g73rVRVVXl1dXXYZchppOVIB4+93MAj2/bz+20NNLYliRlkxWMkO7uZV17MtedVcOXZZ5Cb0IldGZ7MbL27V52onUb+EhlFedlcOe8Mrpx3Bt3dzot7Wnh4635aj3Zw9bkT9dm7EikKf4mkWMyYV17MvPLisEsRCYVO+IqIRJDCX0QkghT+IiIRpPAXEYkghb+ISAQp/EVEIkjhLyISQQp/EZEIytjbO5hZA7Az7Dr6KAUOhF3EW1B96cnk+jK5NlB96Qqyvkp3P+Hn4GZs+GciM6s+mXtmhEX1pSeT68vk2kD1pSuM+jTtIyISQQp/EZEIUvifmlvDLuAEVF96Mrm+TK4NVF+6hrw+zfmLiESQRv4iIhGk8D9FZvYtM3vBzDaY2QNmdkbYNaUys++b2dbeGu82s4y6Yb2ZXWNmm8ys28wyYvWFmS0ys21mVmNmK8KuJ5WZ3WZm+81sY9i19MfMys3sETPb0vvv+rmwa0plZiPM7I9m9nxvfd8Mu6a+zCxuZs+Z2X1D2a/C/9R9393Pdvf5wH3AjWEX1MeDwBx3Pxt4Cbgh5Hr62ghcBTwWdiHQ84MH3AxcBswCPmJms8Kt6g3+C1gUdhFvoRP4e3efCbwD+GyG/f0dAy5y93nAfGCRmb0j5Jr6+hywZag7VfifIndvTdnMBzLqpIm7P+Dunb2bzwATw6ynL3ff4u7bwq4jxQKgxt23u3sSWAUsCbmm17n7Y8DBsOs4Hnff6+7P9j4+RE+ITQi3qj/xHod7N7N7vzLmZ9bMJgJXAD8Z6r4V/gNgZv9oZruBa8m8kX+qTwO/CbuIDDcB2J2yXUsGhddwYmaTgHOAP4RbyRv1TqtsAPYDD7p7JtX3Y+BLQPdQd6zw74eZ/c7MNvbztQTA3b/q7uXA7cDyTKuvt81X6fmV/PZMrC+DWD/7MmZkOFyYWQHwa+Dv+vx2HDp37+qdpp0ILDCzOWHXBGBm7wf2u/v6MPrXB7j3w90vOcmmdwD3A/8wiOW8yYnqM7PrgPcDF3sIa3lP4e8vE9QC5SnbE4G6kGoZlswsm57gv93d7wq7nuNx92Yze5SecyiZcAL9fGCxmV0OjABGmtkv3P1jQ9G5Rv6nyMympWwuBraGVUt/zGwR8GVgsbsfCbueYWAdMM3MJptZAlgKrA65pmHDzAz4D2CLu/8o7Hr6MrOy11a8mVkucAkZ8jPr7je4+0R3n0TP/7uHhyr4QeE/EN/tncJ4AXgvPWfqM8lNQCHwYO9y1P8bdkGpzOyDZlYLvBO438zWhllP78nx5cBaek5W/tLdN4VZUyoz+2/gaWC6mdWa2WfCrqmP84GPAxf1/n/b0DuSzRTjgUd6f17X0TPnP6RLKjOVrvAVEYkgjfxFRCJI4S8iEkEKfxGRCFL4i4hEkMJfRCSCFP4iIhGk8BcRiSCFv4hIBP1/0OgNveZ0LjAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "grid = np.linspace(np.min(X), np.max(X))\n",
    "\n",
    "my_kernel = lambda X,Y : gauss_kernel(X,Y, sigma=0.5)\n",
    "\n",
    "# IMPLEMENT: evaluate MMD witness function on grid\n",
    "phi_X = np.mean(my_kernel(X[:,np.newaxis], grid[:,np.newaxis]), axis=0)\n",
    "phi_Y = np.mean(my_kernel(Y[:,np.newaxis], grid[:,np.newaxis]), axis=0)\n",
    "witness = phi_X-phi_Y\n",
    "\n",
    "plt.plot(grid, witness)\n",
    "\n",
    "plt.hist(X, alpha=0.5, normed=True)\n",
    "plt.hist(Y, alpha=0.5, normed=True);\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, the witness function is positive where X as a higher density than Y, and negative vice versa.\n",
    "It is zero where both densities match.\n",
    "Intuitively, the RKHS norm of this function can only be zero if the densities match everywhere, and it grows as the densities differ on more and more points in their support.\n",
    "Of course, this kind of visualization only works in few dimensions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notes on efficient implementations\n",
    "Note that the above implemention of the MMD, and in particular the permutation test, is extremelty inefficient since the kernel matrix is re-computed in every iteration of the permutation test (and the data is copied around for concatenation as well).\n",
    "We could precompute it once, but then we still would be required to permute it multiple times.\n",
    "\n",
    "We have written a multi-threaded version in C++ that does not permute the matrix, but simply traverses it multiple times, which makes the algorithm L2 cache friendly and therefore hundreds of times faster.\n",
    "See:\n",
    "* http://shogun.ml for the toolbox\n",
    "* http://shogun.ml/examples/latest/examples/statistical_testing/quadratic_time_mmd.html for an API example\n",
    "* http://www.shogun-toolbox.org/notebook/latest/mmd_two_sample_testing.html for a complex demo in Python\n",
    "* https://arxiv.org/abs/1611.04488 for a paper with performance details\n",
    "\n",
    "We will use Shogun for a larger example below.\n",
    "You could time our handwritten implementation against it, or you can tune the Python implementation and try to beat it ;)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Kernel learning (aka which kernel parameter to choose?)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have seen that the Gaussian kernel above increased the power of the test.\n",
    "But what happened if we chose a wrong kernel parameter?\n",
    "Let's pick a bad one and see."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAH0BJREFUeJzt3Xt0VOXZ9/HvJQEDFjGB6ItQn8AjKhJJ0IAIdgmeQKFqi1Z9QaVaeWtr8YwgHit0oQ9aitoqWgr6tBwqglZQaQS0pSICInKyHASNKEgsKCAqeL1/7J04CTlMQmYmyf591sqamT33vvc1wzC/ufeeube5OyIiEl2HpLoAERFJLQWBiEjEKQhERCJOQSAiEnEKAhGRiFMQiIhEnIJARCTiFAQiIhGnIBARibi0VBcQj1atWnl2dnaqy6iz3nsvuDz++NTWIXWMXhg10pCetqVLl25396yq2tWLIMjOzmbJkiWpLqPO6tUruFywIJVVSJ2jF0aNNKSnzcw2x9NOu4ZERCJOQSAiEnEKAhGRiKsXxwjK880331BYWMjevXtTXUrK3XNPcLlmTWrrSJX09HTatm1L48aNU12KSL1Ub4OgsLCQ5s2bk52djZmlupyUOiQc1zWEbzlUl7tTVFREYWEh7dq1S3U5IvVSvd01tHfvXlq2bBn5EIg6M6Nly5YaGYochHobBIBCQAC9DkQOVr0OAhEROXgKgoPwu9/9jpycHDp16sS4ceNKlt977720adOGvLw88vLymDNnDgALFy6kc+fOdO3alfXr1wOwY8cO+vTpQyrPHT1r1ixWr15dcvvuu++moKAAgF69eunHfCINXL09WJxqK1eu5Mknn2Tx4sU0adKEvn370q9fPzp06ADATTfdxK233lpqnYceeogZM2awadMm/vCHP/DQQw9x//33c8cdd6R098asWbPo378/J554IgC//vWvU1ZLfZM9fHbKtr1pTL+UbVsaFo0IamjNmjV0796dZs2akZaWxhlnnMHMmTMrXadx48Z8+eWX7Nmzh8aNG7NhwwY++ugjzjjjjArXefnllznhhBM4/fTTGTp0KP379weCUcfYsWNL2v3whzls2rQJgIsuuohTTjmFTp06MWHChJI23/ve9xg5ciS5ubl0796drVu38q9//YsXXniB2267jby8PDZs2MDgwYN59tlnD6hl7ty5nHbaaZx88slccskl7Nq1C4Dhw4dz4okn0rlz5wPCT0TqvgYxIrjxRli+vHb7zMuDmL09B8jJyWHkyJEUFRXRtGlT5syZQ35+fsn9jz76KE8//TT5+fk89NBDZGRkMGLECIYMGULTpk155plnuPXWW7n//vsr3MbevXu59tprmTdvHsceeyyXXnppXLVPnDiRzMxMvvzyS7p27cqAAQNo2bIlu3fvpnv37owePZphw4bx5JNPcuedd3LBBRfQv39/Lr744gr73L59O6NGjaKgoIDDDjuMBx54gIcffpjrr7+emTNnsnbtWsyMHTt2xFWjiNQdGhHUUMeOHbn99ts555xz6Nu3L7m5uaSlBbl63XXXsWHDBpYvX07r1q255ZZbAMjLy2PRokXMnz+fjRs3cvTRR+PuXHrppQwaNIitW7eW2sbatWtp164dHTp0wMwYNGhQXLWNHz++5FP/hx9+yLp16wBo0qRJyYjilFNOKRlBxGPRokWsXr2anj17kpeXx+TJk9m8eTOHH3446enp/OxnP+O5556jWbNmcfcpInVDgxgRVPbJPZGuueYarrnmGgDuuOMO2rZtC8BRRx1V0ubaa68tefMt5u6MGjWKadOmcf3113PfffexadMmxo8fz+jRo0u1rejYQVpaGt9++23J7a++Cr5Hv2DBAgoKCnjjjTdo1qwZvXr1KvmOfePGjUv6a9SoEfv27Yv7sbo755xzDlOmTDngvsWLF/Pqq68ydepUHn30UebNmxd3vyKSehoRHIRt27YB8MEHH/Dcc89x+eWXA/Dxxx+XtJk5cyY5OTml1ps8eTL9+vUjIyODPXv2cMghh3DIIYewZ8+eUu1OOOEE3n//fTZs2ABQ6k04OzubZcuWAbBq1TIKC98HYOfOnWRkZNCsWTPWrl3LokWLqnwczZs354svvqi0Tffu3Vm4cGHJt5327NnDv//9b3bt2sXOnTs5//zzGTduHMtrex+diCRcgxgRpMqAAQMoKiqicePGPPbYY2RkZAAwbNgwli9fjpmRnZ3NE088UbLOnj17mDx5MnPnzgXg5ptvZsCAATRp0uSAT9vp6elMmDCBfv360apVK04//XRWrlxZsu2nn36avLw8jjuuK9nZxwHQt29fHn/8cTp37szxxx9P9+7dq3wcl112Gddeey3jx48v9yAxQFZWFpMmTeLyyy/nq6++AmDUqFE0b96cCy+8kL179+Lu/Pa3v63msygiqWap/P56vPLz873sd9nXrFlDx44dU1RRaixYsICxY8fy4osvllrekM6oVFOpej3U6a+PNqQzrCRRQ3razGypu+dX1U67hkREIk67huqRXr160av444qISC3RiEBEJOIUBCIiEacgEBGJOAWBiEjEKQgO0syZMzEz1q5dW2XbSZMmsWXLlhpva8GCBQf8ShlgzZrlvPbanBr1uWPHDn7/+99Xa51x48Yd8OO3eNqdf/75lc5FVN32IlI7Eh4EZtbIzN42sxfD2+3M7E0zW2dm08ysSaJrSKQpU6Zw+umnM3Xq1CrbHmwQVKS+BMGcOXM44ogjaq29iNSOZIwIbgDWxNx+APitu3cA/gNck4QaEmLXrl0sXLiQP/7xjwcEwYMPPshJJ51Ebm4uw4cP59lnn2XJkiUMHDiQvLw8vvzyS7Kzs9m+fTsAS5YsKflq6OLFi+nRowddunShR48evFf8i7FyfP311zzyyN289NI08vLymDZtGrt37+bqq6+ma9eudOnSheeffx6AVatW0a1bN/Ly8ujcuTPr1q1j+PDhbNiwgby8PG677bZSfe/evZt+/fqRm5tLTk4O06ZNY/z48WzZsoXevXvTu3dvIJhkLz8/n06dOnHPPfcAlNuu+PHG22/s8/P000/TuXNncnNzueKKKw7mn01Eykjo7wjMrC3QDxgN3GzBjGdnAv83bDIZuBf4w0FtKBXzUBOc0KVv374cd9xxZGZmsmzZMk4++WReeuklZs2axZtvvkmzZs347LPPyMzM5NFHH2Xs2LGlpqsuzwknnMDrr79OWloaBQUF3HHHHcyYMaPctk2aNOFXv/o1K1cu4c9/fhQIJsA788wzmThxIjt27KBbt26cffbZPP7449xwww0MHDiQr7/+mv379zNmzBhWrlxZ7hxBL7/8MkcffTSzZwe/nt25cyctWrTg4YcfZv78+bRq1QqA0aNHk5mZyf79+znrrLNYsWIFQ4cOPaBddfsttmrVKkaPHs3ChQtp1aoVn332WaXPn4hUT6JHBOOAYUDxNJktgR3uXjztZSHQJsE1JMyUKVO47LLLgGC+nuK5ggoKCvjpT39aMiVzZmZmtfrduXMnl1xyCTk5Odx0002sWrWqWuvPnTuXMWPGkJeXVzL76AcffMBpp53Gb37zGx544AE2b95M06ZNK+3npJNOoqCggNtvv51//OMftGjRotx206dP5+STT6ZLly6sWrWq1GkvD6bfYvPmzePiiy8uCYjqPp8iUrmEjQjMrD+wzd2Xmlmv4sXlNC13siMzGwIMATjmmGMq31gK5qEuKipi3rx5rFy5EjNj//79mBkPPvgg7h7XqSdjp5Iunioa4K677qJ3797MnDmTTZs2VfvXxO7OjBkzOL7M5EMdO3bk1FNPZfbs2fTp04ennnqK9u3bV9jPcccdx9KlS5kzZw4jRozg3HPP5e677y7V5v3332fs2LG89dZbZGRkMHjw4FKPpab9ln08qTyVp0hDl8gRQU/gAjPbBEwl2CU0DjjCzIoDqC1Q7tFTd5/g7vnunp+VlZXAMmvm2Wef5corr2Tz5s1s2rSJDz/8kHbt2vHPf/6Tc889l4kTJ5Yc+CzelVF2uufs7GyWLl0KUGrXz86dO2nTJhgoTZo0qcpaDjusObt3f9dvnz59eOSRRyieUPDtt98GYOPGjbRv356hQ4dywQUXsGLFikqnoN6yZQvNmjVj0KBB3HrrrSXTXseu8/nnn3PYYYfRokULtm7dyksvvVSyfkV9x9NvrLPOOovp06dTVFQEoF1DIrUsYUHg7iPcva27ZwOXAfPcfSAwHyg+J+JVwPOJqiGRpkyZwo9+9KNSywYMGMBf/vIX+vbtywUXXEB+fj55eXkl5xYePHgwP//5z0sOFt9zzz3ccMMN/OAHP6BRo0Yl/QwbNowRI0bQs2dP9u/fX2Utp57amw0bVpccLL7rrrv45ptv6Ny5Mzk5Odx1110ATJs2jZycHPLy8li7di1XXnklLVu2pGfPnuTk5BxwsPjdd98tObg8evRo7rzzTgCGDBnCeeedR+/evcnNzaVLly506tSJq6++mp49e5asH9uuuv3G6tSpEyNHjuSMM84gNzeXm2++ucrnRETil5RpqMNdQ7e6e38za08wQsgE3gYGuftXla2vaagrp2moNQ11uRrSfMpJ1JCetninoU7K7KPuvgBYEF7fCHRLxnZFRKRq+mWxiEjE1esgqA9nV5PE0+tA5ODU2yBIT0+nqKhIbwIR5+4UFRWRnp6e6lJE6q16e4aytm3bUlhYyKeffprqUlLuk0+Cy2+/rbxdQ5Wenk7btm1TXYZIvVVvg6Bx48a0a9cu1WXUCdddF1w2hG85iEjy1dtdQyIiUjsUBCIiEacgEBGJOAWBiEjE1duDxVL31OnpFkSkQhoRiIhEnIJARCTiFAQiIhGnIBARiTgFgYhIxCkIREQiTkEgIhJxCgIRkYhTEIiIRJyCQEQk4hQEIiIRpyAQEYk4BYGISMQpCEREIk5BICIScQoCEZGIUxCIiESczlAmDUIqz44mUt9pRCAiEnEKAhGRiFMQiIhEnIJARCTiFAQiIhGnIBARiTgFgYhIxCkIREQiTkEgIhJxCgIRkYhLWBCYWbqZLTazd8xslZndFy5vZ2Zvmtk6M5tmZk0SVYOIiFQtkSOCr4Az3T0XyAP6mll34AHgt+7eAfgPcE0CaxARkSokLAg8sCu82Tj8c+BM4Nlw+WTgokTVICIiVUvoMQIza2Rmy4FtwN+BDcAOd98XNikE2iSyBhERqVxCp6F29/1AnpkdAcwEOpbXrLx1zWwIMATgmGOOSViNIvVVVVNvT91YBMBltTxF96Yx/Wq1P0m9pHxryN13AAuA7sARZlYcQG2BLRWsM8Hd8909PysrKxlliohEUiK/NZQVjgQws6bA2cAaYD5wcdjsKuD5RNUgIiJVS+SuodbAZDNrRBA40939RTNbDUw1s1HA28AfE1iDiIhUIWFB4O4rgC7lLN8IdEvUdkVEpHr0y2IRkYhTEIiIRJyCQEQk4hQEIiIRpyAQEYk4BYGISMQpCEREIk5BICIScQoCEZGIUxCIiEScgkBEJOIUBCIiEVftIDCzDDPrnIhiREQk+eIKAjNbYGaHm1km8A7wJzN7OLGliYhIMsQ7Imjh7p8DPwb+5O6nEJxoRkRE6rl4gyDNzFoDPwFeTGA9IiKSZPEGwX3AK8B6d3/LzNoD6xJXloiIJEu8Zyj72N1LDhC7+0YdIxARaRjiHRE8EucyERGpZyodEZjZaUAPIMvMbo6563CgUSILExGR5Khq11AT4Hthu+Yxyz8HLk5UUSIikjyVBoG7vwa8ZmaT3H1zkmoSEZEkivdg8aFmNgHIjl3H3c9MRFEiIpI88QbBX4HHgaeA/YkrR2pD9vDZqS5BROqReINgn7v/IaGViIhISsT79dG/mdkvzKy1mWUW/yW0MhERSYp4RwRXhZe3xSxzoH3tliMiIskWVxC4e7tEFyIiIqkRVxCY2ZXlLXf3p2u3HBERSbZ4dw11jbmeDpwFLAMUBCIi9Vy8u4Z+FXvbzFoAzySkIhERSaqanrN4D9ChNgsREZHUiPcYwd8IviUEwWRzHYHpiSpKRESSJ95jBGNjru8DNrt7YQLqERGRJItr11A4+dxaghlIM4CvE1mUiIgkT1xBYGY/ARYDlxCct/hNM9M01CIiDUC8u4ZGAl3dfRuAmWUBBcCziSpMRESSI95vDR1SHAKhomqsKyIidVi8b+Yvm9krZjbYzAYDs4E5la1gZt83s/lmtsbMVpnZDeHyTDP7u5mtCy8zDu4hiIjIwag0CMzsWDPr6e63AU8AnYFc4A1gQhV97wNucfeOQHfgl2Z2IjAceNXdOwCvhrdFRCRFqhoRjAO+AHD359z9Zne/iWA0MK6yFd39Y3dfFl7/AlgDtAEuBCaHzSYDF9W8fBEROVhVBUG2u68ou9DdlxCctjIuZpYNdAHeBI5y94/Dfj4Gjoy3HxERqX1VBUF6Jfc1jWcDZvY9YAZwo7t/Hm9hZjbEzJaY2ZJPP/003tVERKSaqgqCt8zs2rILzewaYGlVnZtZY4IQ+LO7Pxcu3mpmrcP7WwPbylvX3Se4e76752dlZVW1KRERqaGqfkdwIzDTzAby3Rt/PtAE+FFlK5qZAX8E1rj7wzF3vUBwxrMx4eXzNahbRERqSaVB4O5bgR5m1hvICRfPdvd5cfTdE7gCeNfMlofL7iAIgOnhqOIDgl8ri4hIisR7PoL5wPzqdOzu/wSsgrvPqk5fIiKSOPp1sIhIxCkIREQiTkEgIhJxCgIRkYhTEIiIRJyCQEQk4hQEIiIRpyAQEYk4BYGISMQpCEREIk5BICIScQoCEZGIUxCIiEScgkBEJOIUBCIiEacgEBGJOAWBiEjEKQhERCIurlNViogUyx4+O2Xb3jSmX8q23ZBpRCAiEnEKAhGRiFMQiIhEnIJARCTiFAQiIhGnIBARiTgFgYhIxCkIREQiTkEgIhJxCgIRkYhTEIiIRJyCQEQk4hQEIiIRpyAQEYk4BYGISMQpCEREIk5BICIScQoCEZGIUxCIiERcwoLAzCaa2TYzWxmzLNPM/m5m68LLjERtX0RE4pPIEcEkoG+ZZcOBV929A/BqeFtERFIoYUHg7q8Dn5VZfCEwObw+GbgoUdsXEZH4JPsYwVHu/jFAeHlkkrcvIiJl1NmDxWY2xMyWmNmSTz/9NNXliIg0WMkOgq1m1hogvNxWUUN3n+Du+e6en5WVlbQCRUSiJtlB8AJwVXj9KuD5JG9fRETKSOTXR6cAbwDHm1mhmV0DjAHOMbN1wDnhbRERSaG0RHXs7pdXcNdZidqmiIhUX509WCwiIsmhIBARiTgFgYhIxCkIREQiTkEgIhJxCgIRkYhTEIiIRJyCQEQk4hQEIiIRpyAQEYk4BYGISMQpCEREIk5BICIScQoCEZGIUxCIiEScgkBEJOIUBCIiEacgEBGJuISdqjLqsofPTtq2PtnYPdzmoqRtUyQVkvH/qi79f9o0pl9StqMRgYhIxCkIREQiTkEgIhJxCgIRkYhTEIiIRJyCQEQk4hQEIiIRpyAQEYk4BYGISMQpCEREIk5BICIScQoCEZGIUxCIiERcg599NJmzgIqI1EcaEYiIRJyCQEQk4hQEIiIRpyAQEYk4BYGISMSlJAjMrK+ZvWdm681seCpqEBGRQNKDwMwaAY8B5wEnApeb2YnJrkNERAKpGBF0A9a7+0Z3/xqYClyYgjpERITUBEEb4MOY24XhMhERSQFz9+Ru0OwSoI+7/yy8fQXQzd1/VabdEGBIePN44L2kFhpoBWxPwXZrSvUmXn2rWfUmXl2u+b/cPauqRqmYYqIQ+H7M7bbAlrKN3H0CMCFZRZXHzJa4e34qa6gO1Zt49a1m1Zt49bHmslKxa+gtoIOZtTOzJsBlwAspqENEREjBiMDd95nZ9cArQCNgoruvSnYdIiISSMnso+4+B5iTim1XU0p3TdWA6k28+laz6k28+lhzKUk/WCwiInWLppgQEYm4SASBmWWa2d/NbF14mVFBu6vCNuvM7KqY5aeY2bvhlBjjzcyq6tfMepnZcjNbZWav1Yeaw/u7mtl+M7u4LtdrZgPNbEX49y8zy42zzkqnNzGzQ81sWnj/m2aWHXPfiHD5e2bWp6o+wy9EvBnWPi38ckS1JLneP4fLV5rZRDNrXN16k11zzP2PmNmuul6vBUab2b/NbI2ZDa1JzbXO3Rv8H/AgMDy8Phx4oJw2mcDG8DIjvJ4R3rcYOA0w4CXgvMr6BY4AVgPHhLePrOs1h7cbAfMIjt9cXJfrBXrErHse8GYcNTYCNgDtgSbAO8CJZdr8Ang8vH4ZMC28fmLY/lCgXdhPo8r6BKYDl4XXHweuq+Zzmux6zw+ffwOmVLfeVNQcrpcPPAPsquv1Aj8FngYOqel7QyL+Ul5AUh5k8GO01uH11sB75bS5HHgi5vYT4bLWwNry2lXUb/jCGVWfag5v3wj8EphE9YMg6fXGtM8APoqjxtOAV2JujwBGlGnzCnBaeD2N4IdCVrZtcbuK+gzX2Q6klbftOJ/TpNVbzrZvAkbX4HWb1JoJ3nTnh6+NmgRBsutdDBxb3ToT/ReJXUPAUe7+MUB4eWQ5bSqa+qJNeL3s8sr6PQ7IMLMFZrbUzK6s6zWbWRvgRwSfXGsi2c9xrGsIRhFViWd6k5I27r4P2Am0rKL28pa3BHaEfVS0rbpUb4lwl9AVwMvVrDcVNV8PvFD8GqkH9f43cKmZLTGzl8ysQw3rrlUN5uT1ZlYA/J9y7hoZbxflLPNKllcmDTgFOAtoCrxhZovc/d+lNli3ah4H3O7u+8Pd8wdurG7VW1xTb4IgOP0gth9Pm4qWl/dh6qAeUxy1xNOmuvXG+j3wurv/o8oKD5S0ms3saOASoFd1Coyzlnja1OQ5PhTY6+75ZvZjYCLwgzhrTZgGEwTufnZF95nZVjNr7e4fm1lrYFs5zQop/YJqCywIl7cts7x4SoyK+i0Etrv7bmC3mb0O5AKlgqCO1ZwPTA1DoBVwvpntc/dZdbRezKwz8BTB8YSiimors/2qpjcpblNoZmlAC+CzKtYtb/l24AgzSws/RZY7lUodqhcAM7sHyAL+XzVrTUXNXYBjgfXh67aZma1392PraL3Ffc0Ir88E/lSNWhMn1fumkvEH/A+lDzg+WE6bTOB9gv3NGeH1zPC+t4DufHcg8/zK+gU6Aq8SBG0zYCWQU5drLtPvJKp/jCDZz/ExwHqgRzVqTCM4QN2O7w7idSrT5peUPjA4PbzeidIHBjcS7J+usE/gr5Q+WPyLaj6nya73Z8C/gKYH8X8tqTWX6bcmxwiS/RyPAa4Or/cC3qrpc12bfykvICkPMtif9yqwLrwsfvPJB56KaXc1wZvLeuCnMcvzCd7MNwCP8t0P8crtN7zvNoJvDq0EbqwPNcesO4nqB0FS6yUYCfwHWB7+LYmzzvMJRmYbgJHhsl8DF4TX0wnewNcTHNhrH7PuyHC99wi/1VRRn+Hy9mEf68M+D63B6yCZ9e4LlxU/p3fX8P9b0mous91qB0EKnuMjgNnAu8AbQG5Naq7tP/2yWEQk4qLyrSEREamAgkBEJOIUBCIiEacgEBGJOAWBiEjEKQgk5cyspQUztS43s0/M7KOY23HP2GlmV5tZeb98rqj9mWbWvbrtzOyXZjawttofDDNLM7MdiehboqPB/LJY6i8PfhWcB2Bm9xJ8H3xsDbq6GlgGfBJn+zMJfgG8qDrt3P2xWm4vklIaEUidZsH5CxaHo4Pfm9kh4afgZyw4f8FKMxtqZpcShMm08kYSZnaTma02s3fM7H/N7L8Jfkl7W9i+h5ldGM43/7aZzTWzIytoN8rMbqxGv7HtjzOzeWH7ZbFz24f3P2RmQ2JujzKzG8zs8HC9ZRacg6F/Oc/V2WY2K+b242Y2KLze1cxeCydBfMnMjqqNfx9pGDQikDrLzHIIZkTt4e77zGwCwU/8NwCt3P2ksN0R7r7DzH4FXO/uy8vpbhjwX+7+dUz7pwjmhBoX9pNBMJOlm9nPgVvc/fZy2p1fzX5j208B7nX3v5lZOgd+GJtKMA1B8XlwLwF6A18CF7r7F2Z2JLAQeDHO5/FQ4HcEv5TdHu6muh8YUvmaEhUKAqnLzga6AkvCScWaEkzv+wpwvJn9juAkOnPj6GsV8L9m9jwwq4I2xwDTw+MMh1JmksCD6BcoCZpW7v43AHffW7aNu79lZt8PP7G3BT5x9y3hCOcBMzsd+Bb4vpm1AuI5PtCRYF6cgvB5bETpab8l4hQEUpcZMNHd7zrgjmDm0fOAocAAqv502wc4A7gQuDMcbZT1GPAbd59jZmcTTHJXlXj6jRXPnC4zCB5TNsEIAeBKglkvTw5HR4UEc+DE2kfpEUbx/QascPeUT3csdZOOEUhdVgD8JPzkW/ztomPMLItgUrq/AvcAJ4ftvwCal+3EzBoBbd19HsFkgFkEs8KWbd8C+MiCj81XxSw/2H4BcPf/ANvN7Ifh+ulm1qycxz2VYBfYj/luyuIWwLYwBM6h/JPcbAY6mVmTcPRxZrh8NdDGzLqF221iZp3KWV8iSkEgdZa7vwvcR7BLYwXBLqCjCOZ6f93MlgNPAneEq/wJeKqcg8VpwF/CPpYRnPf4C+B5gqB528x6APcSzBH/GrA1Zv2y7arbb6yBwC3hOv8kCI+yj/udcPn77l58/oVngB5mtoTguMG6ctZ7n2D31LsE58VdFi7/CrgYeNjM3gHeBk4tu75El2YfFRGJOI0IREQiTkEgIhJxCgIRkYhTEIiIRJyCQEQk4hQEIiIRpyAQEYk4BYGISMT9f596T7s+u09GAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "my_kernel = lambda X,Y : gauss_kernel(X,Y,sigma=0.001)\n",
    "my_mmd = lambda X,Y : quadratic_time_mmd(X[:,np.newaxis],Y[:,np.newaxis], my_kernel)\n",
    "\n",
    "statistics = two_sample_permutation_test(my_mmd, X, Y, num_permutations)\n",
    "my_statistic = my_mmd(X,Y)\n",
    "plot_permutation_samples(statistics, my_statistic)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output is quite different, the performance of kernel tests indeed crucially depends on the chosen parameters.\n",
    "How to chose those in practice?\n",
    "\n",
    "For simple datasets, there is a heuristic rule to chose the parameter of a Gaussian kernel: using the median of the pairwise distances of the data.\n",
    "See https://arxiv.org/abs/1707.07269 for details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6585612327494343\n"
     ]
    }
   ],
   "source": [
    "def gaussian_kernel_median_heuristic(Z):\n",
    "    # IMPLEMENT: compute the median of the pairwise distances in Z\n",
    "    # (not taking zero distance between identical samples (diagonal) into account)\n",
    "    sq_dists = sq_distances(Z)\n",
    "    np.fill_diagonal(sq_dists, np.nan)\n",
    "    sq_dists = np.ravel(sq_dists)\n",
    "    sq_dists = sq_dists[~np.isnan(sq_dists)]\n",
    "    median_dist = np.median(np.sqrt(sq_dists))\n",
    "    \n",
    "    return np.sqrt(median_dist/2.0) # our kernel uses a bandwidth of 2*(sigma**2)\n",
    "\n",
    "sigma_median = gaussian_kernel_median_heuristic(np.vstack((X[:,np.newaxis],Y[:,np.newaxis])))\n",
    "print(sigma_median)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we could probably get away with sub-sampling the data to compute pair-wise distances to speed up computing the heuristic, as it stabilizes quickly.\n",
    "In addition, we should probably re-use the pairwise distances when computing the kernel matrix; again, see Shogun.\n",
    "\n",
    "Let's try the heuristic for the statistical test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAHydJREFUeJzt3XmUVNW99vHvj8kGg8jQchGS2/AGFWjpBhuCQF4ZHFB4wQSJelExElnJvUbjhAxOUcgCFxokmBhUInojQ0DQCBqCQIxExAZRmQwCjSLI0AoKiAr5vX+c050Guunq4VR1c57PWrWq6tQ+u/buqtVP7TPsY+6OiIjEV41UN0BERFJLQSAiEnMKAhGRmFMQiIjEnIJARCTmFAQiIjGnIBARiTkFgYhIzCkIRERirlaqG5CIJk2aeEZGRoXref/94P7ssytclVRR+ozlpFEJX+aVK1fucff00spViyDIyMggNze3wvX06BHcL11a4aqkitJnLCeNSvgym9nWRMpp05CISMwpCEREYk5BICISc9ViH4GIHO2bb75h27ZtHDp0KNVNkajcd19wv359qUXT0tJo0aIFtWvXLtdbKQhEqqFt27ZRv359MjIyMLNUN0eiUCPcYFPKUUPuTn5+Ptu2baNly5ble6tyrSUiKXXo0CEaN26sEBDMjMaNG1dodKggEKmmFAJSoKLfBQWBiEjMRRoEZna6mc02sw1mtt7MzjezRmb2VzPbGN43jLINIhKNRx99lMzMTNq1a8fEiRMLl99///00b96c7OxssrOzWbBgAQDLli2jffv2dOrUiQ8++ACAvXv3cskll5DKa6fPmzePdevWFT6/9957WbRoEQA9evSolJNZq7qodxY/Crzi7leYWR2gHjAKeNXdx5nZCGAEcFdUDcgYMb/w8Sebu4TLlpe5nrxxfSutTSLV3Zo1a3jiiSdYsWIFderUoU+fPvTt25fWrVsDcOutt3LHHXcctc7DDz/MnDlzyMvL43e/+x0PP/wwDz74IKNGjUrpZq558+bRr18/2rZtC8ADDzyQsrakSmQjAjM7Dfi/wFMA7v61u+8FBgDTwmLTgMujaoOIRGP9+vV06dKFevXqUatWLS644ALmzp17wnVq167Nl19+ycGDB6lduzabNm3i448/5oILLihxnVdeeYVzzjmH7t27c/PNN9OvXz8gGHVMmDChsFxmZiZ5eXkAXH755Zx33nm0a9eOKVOmFJb51re+xejRo8nKyqJLly7s3LmTf/zjH7z44ovceeedZGdns2nTJq6//npmz559XFsWLlzI+eefT8eOHRk0aBD79+8HYMSIEbRt25b27dsfF37VRZQjglbAbuAPZpYFrARuAZq6+w4Ad99hZmdE2AaRk94vfgGrV1dundnZUGRrz3EyMzMZPXo0+fn51K1blwULFpCTk1P4+uTJk3nmmWfIycnh4YcfpmHDhowcOZJhw4ZRt25dnn32We644w4efPDBEt/j0KFD3HjjjSxevJjvfve7XHnllQm1ferUqTRq1Igvv/ySTp06MXDgQBo3bsyBAwfo0qULY8eOZfjw4TzxxBPcfffd9O/fn379+nHFFVeUWOeePXsYM2YMixYt4tRTT2X8+PE88sgj3HTTTcydO5cNGzZgZuzduzehNlY1Ue4jqAV0BH7n7h2AAwSbgRJiZsPMLNfMcnfv3h1VG0WkHNq0acNdd93FRRddRJ8+fcjKyqJWreB35c9+9jM2bdrE6tWradasGbfffjsA2dnZLF++nCVLlrB582bOPPNM3J0rr7ySa665hp07dx71Hhs2bKBly5a0bt0aM+Oaa65JqG2TJk0q/NX/0UcfsXHjRgDq1KlTOKI477zzCkcQiVi+fDnr1q2jW7duZGdnM23aNLZu3cppp51GWloaP/nJT3j++eepV69ewnVWJVGOCLYB29z9zfD5bIIg2GlmzcLRQDNgV3Eru/sUYApATk5O6vYkiVRxJ/rlHqWhQ4cydOhQAEaNGkWLFi0AaNq0aWGZG2+8sfCfbwF3Z8yYMcycOZObbrqJX/7yl+Tl5TFp0iTGjh17VNmS9h3UqlWLf/3rX4XPC46hX7p0KYsWLeKNN96gXr169OjRo/C12rVrF9ZXs2ZNDh8+nHBf3Z2LLrqI6dOnH/faihUrePXVV5kxYwaTJ09m8eLFCddbVUQ2InD3T4CPzKzgtLjewDrgRWBIuGwI8EJUbRCR6OzaFfyG+/DDD3n++ee5+uqrAdixY0dhmblz55KZmXnUetOmTaNv3740bNiQgwcPUqNGDWrUqMHBgwePKnfOOeewZcsWNm3aBHDUP+GMjAxWrVoFwKpVq9iyZQsA+/bto2HDhtSrV48NGzawfHnpB4bUr1+fL7744oRlunTpwrJlywqPdjp48CD//Oc/2b9/P/v27eOyyy5j4sSJrK7sbXRJEvVRQz8H/hgeMbQZ+DFB+Mwys6HAh8CgiNsgIhEYOHAg+fn51K5dm8cee4yGDYMjwYcPH87q1asxMzIyMvj9739fuM7BgweZNm0aCxcuBOC2225j4MCB1KlT57hf22lpaUyZMoW+ffvSpEkTunfvzpo1awrf+5lnniE7O5tOnTpx1llnAdCnTx8ef/xx2rdvz9lnn02XLl1K7cdVV13FjTfeyKRJk4rdSQyQnp7O008/zdVXX81XX30FwJgxY6hfvz4DBgzg0KFDuDu//vWvy/hXrBoslcfvJionJ8fLeyzvUYePPhd8Kf7jv3T46MkqLhemWb9+PW3atEl1M5Jq6dKlTJgwgZdeeinVTUmOMl6hrLjvhJmtdPecElYppDOLRURiTrOPiki10KNHD3oUDPmkUmlEICIScwoCEZGYUxCIiMScgkBEJOYUBCJSbnPnzsXM2LBhQ6lln376abZv317u91q6dOlxZykDrF69unCq67Lau3cvv/3tb8u0zsSJE487+S2RcpdddtkJ5yI6rvywYez9/PMyta28FAQiUm7Tp0+ne/fuzJgxo9SyFQ2CklSXIFiwYAGnn3564uWnTOH0004rU9vKS0EgIuWyf/9+li1bxlNPPXVcEDz00EOce+65ZGVlMWLECGbPnk1ubi6DBw8mOzubL7/8koyMDPbs2QNAbm5u4aGhK1asoGvXrnTo0IGuXbvyfsGJVcX4+uuvuffee5k5cybZ2dnMnDmTAwcOcMMNN9CpUyc6dOjACy8Es9isXbuWzp07k52dTfv27dm4cSMjRoxg06ZNZGdnc+eddx5V94EDB+jbty9ZWVlkZmYyc+ZMJk2axPbt2+nZsyc9e/YEgkn2cnJyaNeuHffddx9AseUK+ptovRm9erHns88AeOaZZ2jfvj1ZWVlce+21FfnYiqXzCESqu1TMQ01wQZc+ffpw1lln0ahRI1atWkXHjh15+eWXmTdvHm+++Sb16tXj008/pVGjRkyePJkJEyYcNV11cc455xxee+01atWqxaJFixg1ahRz5swptmydOnV44IEHyM3NZfLkyUAwAV6vXr2YOnUqe/fupXPnzlx44YU8/vjj3HLLLQwePJivv/6aI0eOMG7cONasWVPsHEGvvPIKZ555JvPnB7MT7Nu3jwYNGvDII4+wZMkSmjRpAsDYsWNp1KgRR44coXfv3rz77rvcfPPNx5Ura70F1q5dy9ixY1m2bBlNmjTh008/PeHfrzw0IhCRcpk+fTpXXXUVEMzXUzBX0KJFi/jxj39cOCVzo0aNylTvvn37GDRoEJmZmdx6662sXbu2TOsvXLiQcePGkZ2dXTj76Icffsj555/Pr371K8aPH8/WrVupW7fuCes599xzWbRoEXfddRd///vfadCgQbHlZs2aRceOHenQoQNr16496rKXFam3wOLFi7niiisKA6Ksf89EaEQgUt2lYB7q/Px8Fi9ezJo1azAzjhw5gpnx0EMP4e4JXXqy6FTSBVNFA9xzzz307NmTuXPnkpeXV+azid2dOXPmcPYxc/S0adOG733ve8yfP59LLrmEJ598klatWpVYz1lnncXKlStZsGABI0eO5OKLL+bee+89qsyWLVuYMGECb731Fg0bNuT6668/qi/lrffY/kR9KU+NCESkzGbPns11113H1q1bycvL46OPPqJly5a8/vrrXHzxxUydOrVwx2fBpoxjp3vOyMhg5cqVAEdt+tm3bx/NmzcHgh3MpTm23ksuuYTf/OY3FEyo+fbbbwOwefNmWrVqxc0330z//v159913TzgF9fbt26lXrx7XXHMNd9xxR+G010XX+fzzzzn11FNp0KABO3fu5OWXXy6xXWWpt6jevXsza9Ys8vPzAbRpSESqhunTp/ODH/zgqGUDBw7kueeeo0+fPvTv35+cnByys7MLry18/fXX89Of/rRwZ/F9993HLbfcwve//31q1qxZWM/w4cMZOXIk3bp148iRI6W2pWfPnqxbt65wZ/E999zDN998Q/v27cnMzOSee+4BYObMmWRmZpKdnc2GDRu47rrraNy4Md26dSMzM/O4ncXvvfde4c7lsWPHcvfddwMwbNgwLr30Unr27ElWVhYdOnSgXbt23HDDDXTr1q1w/aLlylpvUe3atWP06NFccMEFZGVlcdttt5X6NykrTUOdIE1DXT1oGmo5aWgaahERSRYFgYhIzCkIRKqp6rBZV5Kjot8FBYFINZSWlkZ+fr7CQHB38vPzSUtLK3cdOo9ApBpq0aIF27ZtY/fu3aluikTlk0+C+/BcixNJS0ujRYsW5X4rBYFINVS7dm1atmyZ6mZIlH72s+A+CYfAadOQiEjMKQhERGJOQSAiEnMKAhGRmIt0Z7GZ5QFfAEeAw+6eY2aNgJlABpAH/MjdP4uyHSIiUrJkHDXU0933FHk+AnjV3ceZ2Yjw+V1JaEeFFJ2zqCI0Z5GIVDWp2DQ0AJgWPp4GXJ6CNoiISCjqIHBgoZmtNLNh4bKm7r4DILw/I+I2iIjICUS9aaibu283szOAv5rZhkRXDINjGMB3vvOdqNonIhJ7kY4I3H17eL8LmAt0BnaaWTOA8H5XCetOcfccd89JT0+PspkiIrEWWRCY2almVr/gMXAxsAZ4ERgSFhsCvBBVG0REpHRRbhpqCswNL7pcC3jO3V8xs7eAWWY2FPgQGBRhG0REpBSRBYG7bwayilmeD/SO6n1FRKRsdGaxiEjMKQhERGJOQSAiEnMKAhGRmFMQiIjEnIJARCTmFAQiIjGnIBARiTkFgYhIzCkIRERiTkEgIhJzCgIRkZhTEIiIxJyCQEQk5hQEIiIxpyAQEYk5BYGISMwpCEREYk5BICIScwoCEZGYUxCIiMScgkBEJOZqpboBcZMxYn6F68gb17cSWiIiEtCIQEQk5hQEIiIxF3kQmFlNM3vbzF4Kn7c0szfNbKOZzTSzOlG3QURESpaMEcEtwPoiz8cDv3b31sBnwNAktEFEREoQaRCYWQugL/Bk+NyAXsDssMg04PIo2yAiIicW9YhgIjAc+Ff4vDGw190Ph8+3Ac0jboOIiJxAZEFgZv2AXe6+sujiYop6CesPM7NcM8vdvXt3JG0UEZFoRwTdgP5mlgfMINgkNBE43cwKzl9oAWwvbmV3n+LuOe6ek56eHmEzRUTiLbIgcPeR7t7C3TOAq4DF7j4YWAJcERYbArwQVRtERKR0qTiP4C7gNjP7gGCfwVMpaIOIiISSMsWEuy8FloaPNwOdk/G+IiJSOp1ZLCIScwoCEZGYUxCIiMScgkBEJOYUBCIiMacgEBGJOQWBiEjMlTkIzKyhmbWPojEiIpJ8CQWBmS01s9PMrBHwDvAHM3sk2qaJiEgyJDoiaODunwM/BP7g7ucBF0bXLBERSZZEg6CWmTUDfgS8FGF7REQkyRINgl8CfwE+cPe3zKwVsDG6ZomISLIkOuncDncv3EHs7pu1j0BE5OSQ6IjgNwkuExGRauaEIwIzOx/oCqSb2W1FXjoNqBllw0REJDlK2zRUB/hWWK5+keWf8++rjImISDV2wiBw978BfzOzp919a5LaJCIiSZTozuJTzGwKkFF0HXfvFUWjREQkeRINgj8BjwNPAkeia46IiCRbokFw2N1/F2lLREQkJRI9fPTPZvbfZtbMzBoV3CJtmYiIJEWiI4Ih4f2dRZY50KpymyMiIsmWUBC4e8uoGyIiIqmRUBCY2XXFLXf3Zyq3OSIikmyJbhrqVORxGtAbWAUoCEREqrlENw39vOhzM2sAPHuidcwsDXgNOCV8n9nufp+ZtQRmAI0IwuRad/+6HG0XEZFKUN5rFh8EWpdS5iugl7tnAdlAHzPrAowHfu3urYHPgKHlbIOIiFSCRPcR/JngKCEIJptrA8w60Tru7sD+8Gnt8OZAL+C/wuXTgPsBnaMgIpIiie4jmFDk8WFgq7tvK20lM6sJrAS+CzwGbAL2uvvhsMg2oHnizRURkcqW0KahcPK5DQQzkDYEEtqm7+5H3D0baAF0JhhJHFesuHXNbJiZ5ZpZ7u7duxN5OxERKYeEgsDMfgSsAAYRXLf4TTNLeBpqd98LLAW6AKebWcFIpAWwvYR1prh7jrvnpKenJ/pWIiJSRoluGhoNdHL3XQBmlg4sAmaXtEJY5ht332tmdYELCXYULyG4lsEMgjOWXyh/80VEpKISDYIaBSEQyqf00UQzYFq4n6AGMMvdXzKzdcAMMxsDvA08VdZGi4hI5Uk0CF4xs78A08PnVwILTrSCu78LdChm+WaC/QUiIlIFlHbN4u8CTd39TjP7IdAdMOAN4I9JaJ+IiESstM07E4EvANz9eXe/zd1vJRgNTIy6cSIiEr3SgiAj3MRzFHfPJbhspYiIVHOlBUHaCV6rW5kNERGR1CgtCN4ysxuPXWhmQwnOGBYRkWqutKOGfgHMNbPB/Psffw5QB/hBlA0TEZHkOGEQuPtOoKuZ9QQyw8Xz3X1x5C0TEZGkSPR6BEsIzggWEZGTTHmvRyAiIicJBYGISMwpCEREYk5BICIScwoCEZGYUxCIiMScgkBEJOYUBCIiMacgEBGJOQWBiEjMKQhERGJOQSAiEnMKAhGRmFMQiIjEnIJARCTmFAQiIjGnIBARibnIgsDMvm1mS8xsvZmtNbNbwuWNzOyvZrYxvG8YVRtERKR0UY4IDgO3u3sboAvwP2bWFhgBvOrurYFXw+ciIpIikQWBu+9w91Xh4y+A9UBzYAAwLSw2Dbg8qjaIiEjpkrKPwMwygA7Am0BTd98BQVgAZySjDSIiUrzIg8DMvgXMAX7h7p+XYb1hZpZrZrm7d++OroEiIjEXaRCYWW2CEPijuz8fLt5pZs3C15sBu4pb192nuHuOu+ekp6dH2UwRkViL8qghA54C1rv7I0VeehEYEj4eArwQVRtERKR0tSKsuxtwLfCema0Ol40CxgGzzGwo8CEwKMI2iIhIKSILAnd/HbASXu4d1fuKiEjZ6MxiEZGYUxCIiMScgkBEJOYUBCIiMacgEBGJOQWBiEjMKQhERGJOQSAiEnMKAhGRmFMQiIjEnIJARCTmFAQiIjGnIBARiTkFgYhIzCkIRERiLsoL00hEMkbMT3UTjpI3rm+qmyAiFaARgYhIzCkIRERiTkEgIhJzCgIRkZhTEIiIxJyCQEQk5hQEIiIxpyAQEYk5BYGISMxFFgRmNtXMdpnZmiLLGpnZX81sY3jfMKr3FxGRxEQ5Inga6HPMshHAq+7eGng1fC4iIikUWRC4+2vAp8csHgBMCx9PAy6P6v1FRCQxyd5H0NTddwCE92ck+f1FROQYVXZnsZkNM7NcM8vdvXt3qpsjInLSSnYQ7DSzZgDh/a6SCrr7FHfPcfec9PT0pDVQRCRukh0ELwJDwsdDgBeS/P4iInKMKA8fnQ68AZxtZtvMbCgwDrjIzDYCF4XPRUQkhSK7Qpm7X13CS72jek8RESm7KruzWEREkkNBICISc7p4vVRYxoj5lVJP3ri+lVKPiJSNRgQiIjGnIBARiTkFgYhIzCkIRERiTkEgIhJzCgIRkZhTEIiIxJyCQEQk5hQEIiIxpzOL5aSzfHM+GSOWV6gOneUscaIRgYhIzCkIRERiTpuGpMqojMnrPtncpRJaIhIvGhGIiMScgkBEJOYUBCIiMacgEBGJOQWBiEjMKQhERGJOQSAiEnMKAhGRmFMQiIjEXErOLDazPsCjQE3gSXcfl4p2iEStMs6WBk2CJ9FK+ojAzGoCjwGXAm2Bq82sbbLbISIigVRsGuoMfODum939a2AGMCAF7RAREVITBM2Bj4o83xYuExGRFEjFPgIrZpkfV8hsGDAsfLrfzN6vrAZsHV9ZNVWaJsCeVDciRSLpe0U/Y4v+O1KmfiehPcmk73tZWHH/MhP2n4kUSkUQbAO+XeR5C2D7sYXcfQowJVmNSiUzy3X3nFS3IxXi2ve49hvU96rY91RsGnoLaG1mLc2sDnAV8GIK2iEiIqRgRODuh83sJuAvBIePTnX3tcluh4iIBFJyHoG7LwAWpOK9q6hYbAIrQVz7Htd+g/pe5Zj7cftpRUQkRjTFhIhIzCkIKpmZ9TGz983sAzMbUczrp5jZzPD1N80so8hrI8Pl75vZJYnWWVVE1Pc8M3vPzFabWW5yelJ25e27mTU2syVmtt/MJh+zznlh3z8ws0lmFTuOMCoR9X1pWOfq8HZGcnpTNhXo+0VmtjL8fFeaWa8i6yT/c3d33SrpRrDzexPQCqgDvAO0PabMfwOPh4+vAmaGj9uG5U8BWob11Eykzqpwi6Lv4Wt5QJNU9y/Cvp8KdAd+Ckw+Zp0VwPkE5968DFya6r4mse9LgZxU9y/CvncAzgwfZwIfp/Jz14igciUyfcYAYFr4eDbQO0z8AcAMd//K3bcAH4T1VZcpOaLoe3VR7r67+wF3fx04VLSwmTUDTnP3Nzz47/AMcHmkvSifSu97NVKRvr/t7gXnT60F0sLRQ0o+dwVB5Upk+ozCMu5+GNgHND7ButVlSo4o+g7BWecLw+HzMKqmivT9RHVuK6XOqiCKvhf4Q7hZ6J4qulmssvo+EHjb3b8iRZ97Sg4fPYklMn1GSWVKWl5cWFfFQ72i6DtAN3ffHm4j/quZbXD31yrQzihUpO8VqbMqiKLvAIPd/WMzqw/MAa4l+HVclVS472bWDhgPXFyGOiudRgSVK5HpMwrLmFktoAHw6QnWTWhKjiogir5TMHx2913AXKrmJqOK9P1EdbYopc6qIIq+4+4fh/dfAM9xEn7uZtaC4Dt9nbtvKlI+6Z+7gqByJTJ9xovAkPDxFcDicFvgi8BV4XbClkBrgp1G1WVKjkrvu5mdGv4ixMxOJfjVtCYJfSmrivS9WO6+A/jCzLqEm0WuA16o/KZXWKX33cxqmVmT8HFtoB8n2eduZqcD84GR7r6soHDKPvdU73k/2W7AZcA/CY4mGB0uewDoHz5OA/5EsEN0BdCqyLqjw/Xep8iRAsXVWRVvld13gqMx3glva0/ivucR/ErcT/CLsG24PIfgH+AmYDLhCaBV7VbZfSc4mmgl8G74uT9KeBRZVbuVt+/A3cABYHWR2xmp+tx1ZrGISMxp05CISMwpCEREYk5BICIScwoCEZGYUxCIiMScgkBSLpyFsmCWyU/M7OMiz+uUoZ4bzOw/ylC+l5l1KWs5M/sfMxtcWeUrIjzmfm8UdUt8aIoJSTl3zweyAczsfmC/u08oR1U3AKuATxIs3wvYAywvSzl3f6ySy4uklEYEUqWZ2RAzWxGODn5rZjXCX8HPhnO2rzGzm83sSoIwmVncSMLMbjWzdWb2jpn9r5n9H+AnwJ1h+a5mNiCcM/5tM1toZmeUUG6Mmf2iDPUWLX+WmS0Oy6+yItdkCF9/uOjkeuG6t5jZaeF6q8zsXTPrV8zf6kIzm1fk+eNmdk34uJOZ/c2CyfteNrOmlfH5yMlBIwKpsswsE/gB0NXdD5vZFILT+DcRXKPg3LDc6e6+18x+Dtzk7quLqW448J/u/nWR8k8Ce9x9YlhPQ+BFd3cz+ylwu7vfVUy5y8pYb9Hy04H73f3PZpbG8T/GZgDj+Pe1bQcBPYEvgQHu/oUFE/AtA15K8O94CsHZuf3dfU+4mepBoKrO5ipJpiCQquxCoBOQG0y7Ql2CKX3/ApxtZo8CC4CFCdS1FvhfM3sBmFdCme8As8L9DKcQTB1QGfUChUHTxN3/DODux83D7+5vmdm3w1/sLYBPPJh9tQ4w3sy6A/8Cvh3Ox5PI/oE2QDtgUfh3rMnRUx1LzCkIpCozYKq733PcC2btgUuBmwnmcy/t1+0lwAUEFwq5OxxtHOsx4FfuvsDMLgQSuSxoIvUWlcicLnMI+pRBMEKAYPKxBkDHcHS0jWAem6IOc/QIo+B1A9519+8n8N4SQ9pHIFXZIuBH9u+ZKBub2XfMLJ1gIq4/AfcBHcPyXwD1j63EzGoCLdx9MXAnkA7UK6Z8A+DjcNbHIUWWV7ReANz9M2CPmf2/cP00M6tXTL9nEGwC+yFBKBS0bVcYAhdR/MVKtgLtzKxOOPoouA7uOqC5mXUO37eOBfPgiwAKAqnC3P094JcEmzTeJdgE1JRgfvfXzGw18AQwKlzlD8CTxewsrgU8F9axChjvwTz3LxAEzdtm1hW4n2B++L8BO4usf2y5stZb1GDg9nCd1wnC49h+vxMu3+LBdRgAngW6mlkuwX6DjcWst4Vg89R7BBdxWRUu/4pgCuRHzOwd4G3ge8euL/Gl2UdFRGJOIwIRkZhTEIiIxJyCQEQk5hQEIiIxpyAQEYk5BYGISMwpCEREYk5BICISc/8fZrViCccQcHAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "my_kernel = lambda X,Y : gauss_kernel(X,Y,sigma=sigma_median)\n",
    "my_mmd = lambda X,Y : quadratic_time_mmd(X[:,np.newaxis],Y[:,np.newaxis], my_kernel)\n",
    "\n",
    "statistics = two_sample_permutation_test(my_mmd, X, Y, num_permutations)\n",
    "my_statistic = my_mmd(X,Y)\n",
    "plot_permutation_samples(statistics, my_statistic)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This worked great!\n",
    "But it doesn't always.\n",
    "\n",
    "Let's try the median heuristic on the following dataset, referred to as Gaussian blobs, see https://papers.nips.cc/paper/4727-optimal-kernel-choice-for-large-scale-two-sample-tests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAEICAYAAABLdt/UAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzsvXt8VNd57/1dc9OMRmgkISEh2DC2wQFjY2Mgk7ihTuocH5/YsZPabpo29senPbgnveQkrZ23r1vKIZzmTe22SXN5eV+IT1w7TdoEN8XBCXXsxA6u44kgxmAZbARsGJAQEpJGt5Hmts8fS5vZGs1do/v6fT58xMzsvdfat9961u951vMIwzBQUFBQUJg/sM10BxQUFBQUygtF7AoKCgrzDIrYFRQUFOYZFLErKCgozDMoYldQUFCYZ1DErqCgoDDPoIhdQSEDhBBPCSH+V47fDSHEqgKO85AQ4tXy9k5BITcUsSvMGYyR5DEhxLAQ4qIQYpcQomam+6WgMNugiF1hTkAI8WfA3wCPAj7gfcBK4CdCCNdM9k1BYbZBEbvCrIcQohrYAfyJYRgHDMOIGYahA7+FJPdPjW33P4UQ3xNCPC2EGBBCtAohNlmO0yyEeFYI0SWEOCOE+EyepuuFED8ZO9YrQoiVWfrnG2uzSwhxVgjxl0II2/hNxNeEEGEhxAkhxG2TuR4KCvmgiF1hLuAWwA38q/VLwzAGgR8D/8ny9d3APwM1wHPA1wHGiPaHwJvAMuA24LNCiP+co93fBXYC9cAR4J+ybPc15CziauBW4EHgv1p+DwCnx46zHfhXIURdrhNWUJgMFLErzAXUA92GYcQz/NYx9ruJVw3D+JFhGAngGeDGse83Aw2GYXzBMIyoYRingT3Ab+do93nDMH5uGMYo8BfA+4UQmnUDIYQd+ATwfxuGMTA2k/g74AHLZpeAr4zNNP4FeAe4s7BTV1AoHo6Z7oCCQgHoRsoijgzkvnTsdxMXLf8fBtxCCAdSsmkWQvRZfrcDB3O0GzL/YxjGoBCiB2i2fo8cVFzAWct3Z5GzAhMXjPHZ9s6OHUdBYUqgLHaFuYBfAKPAb1q/FEJ4gf8CvFTAMULAGcMwaiz/FhmG8ZEc+1yxzoUQVUAd0J62TTcQQw4cJlYAFyyflwkhRNrv6cdRUCgbFLErzHoYhhFGOk+/JoS4QwjhFEL4ge8D55GSSz78EugXQvxfQgiPEMIuhLheCLE5xz4fEUJ8YCzqZicQNAzDaq0zJvl8D/hrIcSiMQfrnwLftmy2BPjMWL/vB9YCPyrk3BUUSoEidoU5AcMwHgceA/4W6AeCSCv8tjENPN/+CeCjwE3AGaSl/U2k0zMbvoN0dvYAG5HO1Ez4E2AI6SB9dWy//235PQisHmvzr4H7DMO4nK/PCgqlQqhCGwoKCgrzC8piV1BQUJhnUMSuoKCgMM+giF1BQUFhnkERu4KCgsI8w4wsUKqvrzf8fv9MNK2goKAwZ3H48OFuwzAa8m03I8Tu9/s5dOjQTDStoKCgMGchhDibfyslxSgoKCjMO0ya2IUQbiHEL4UQb46lSd1Rjo4pKCgoKJSGckgxo8BvjCVJcgKvCiF+bBjG62U4toKCgoJCkZg0sY9lrRsc++gc+6eWsyooKCjMEMqisY8lVDqCzDv9E8Mwghm2eVgIcUgIcairq6sczSooKCgoZEBZiN0wjIRhGDcBy4H3CiGuz7DNbsMwNhmGsamhIW+0joKCgoJCiShrVIxhGH3Ay8Ad5TzuFYRCcPCg/FvI9woKswjqMVWYLkxaYxdCNAAxwzD6hBAe4MPIavLlRSgEO3ZAPA4OB2zfDpqW/XsFhVkE9ZgqTCfKYbEvBX4mhDgKtCA19v1lOO546Lp8K/x++VfXc3+voDCLoB5ThelEOaJijgIbytCX3PD7pamj6/KvmZIg2/cKCpNAKCQfKb+/PJa1ekwVphMzUmhj06ZNRkkpBbK9beV+CxUWNKZKNlGPqcJkIYQ4bBjGpnzbzUiumJKhaZnfiGzfK8xZzCQJWmUTXZf/ytEH9ZgqTBfmFrErLAjMtKNRySYKcx2K2BVmHabKYi4UmiYHEyWbKMxVKGJXmHWYDRazkk0U5jIUsSvMOiiLWUFhclDEXgpUeMOUQ1nMCgqlQxF7sZhpz55CWaHGaIX5CEXsxWKmPXsKZcNCHKPVQLYwoIi9WMwGz94MYj4Rw0IboxfiQLZQoYi9WCxgz958I4ZCx+j5MpgttIFsIUMReynI59mbL0yQhvlGDIWM0fNpMFvgk80FBUXs5cZ8YoI0zEdiyDdGz6fBbAFPNhccFLGXG/OJCdKwEIkhfTBzuWSxjLl6/iqMdGFAEXu5MR/NWgsWGjFYBzOXC/bsmZeTMYV5BkXs5cZCNGvnOczB7ODBeTsZU5hnUMReKnI5SBeaWbtAMM8nYwrzCIrYS8E8dpAqZIeajCnMFShiLwXz2EGqkBtqMqYwF1COYtYLD2pOrqCgMIuhLPZSUMqcfJ4uWpoqmJfL5YJoVF02BYVioIi9VBQzJ1eafFEwL1c4DEePwvr14PMtnMumbACFyUJJMdMBqyYfj8vPM4BQSIbshUIz0nzBMC9XVdX4vzN02aYV5qD25JPy73Tcq7nyXCgUjklb7EIIDXgaaAKSwG7DMP5hssedV5gFmvxcmjSYlyscln8HB6XFPhWXbbZZx9Ptl59Lz4VC4SiHFBMH/swwjF8JIRYBh4UQPzEM4+0yHHv+4I475N9AYEbenLkUyJO+2nOqNPbZSGrTbQPMpedCoXBMmtgNw+gAOsb+PyCEOA4sAxSxw0T2CARmxEycBZOGojAdYYWzkdSmO1Z+rj0XCoWhrM5TIYQf2AAEM/z2MPAwwIoVK8rZ7OxGOnsEg3DgQOlmYr5wkSyDhlpcMxGzldSyDWpTYQ+o52J+omzELoSoAp4FPmsYRn/674Zh7AZ2A2zatMkoV7uzHunsAZLUa2vhxAlJ9MVG12QLF8mjLajFNeMxl0htKmUj9VzMP5SF2IUQTiSp/5NhGP9ajmPONpRsLaWzB8DevbB/v/z/s89O1N2zNZYeLiIEtLXBU0/BQw/NTm1hlmMmSa2QZ8rcprNT3VqFwlGOqBgBPAkcNwzj7yffpdmHSVtL6exx333Q1wdr10Jv7/i3NBSCRx+VVrnPB088kfrNGi4C8Npr0N8v93/7bfjkJ6G7G4aGpi6MRKEskkghz5R1m9FRMIzZJxspzE6Uw2L/NeAB4JgQ4sjYd48ZhvGjMhx7VqDshnAgIHX23t6Jb2kwCIcPg9crrXGrVGO1/ltb4Vvfgupqqbd3dMDu3eDxQCQCjzwyL026mQ5PLJckUsgzlb7NXXdBY+P0n/tMX3OF4lGOqJhXAVGGvsxalN3JZhJ0cIKPubB9NU124uWX5SAQi8HixZLU162THY1GJ9nJ8mGyxGD1F890oYtyDfKFPFPp28xEpOxsDAlVyA+VUqAATJmTzYyOOXAg9cYEArBpk7Tma2vl52ydeuKJ1OCgaZL1ZtlcfbLEYN2/u3v82DUTOrPLVR61K9MzlT4AFlpseyqt6VIGMmXhzzwUsReIsjvZsr0xmgaPP17Ym5HeqebmcfvNxAuW3uZkLVzr/kNDUmWaqbErFJJjZ7nULuvtyzYA5nrupsOaLna2qiz82QFF7DOFXG9MqaOIZb+CXrAyM3+mNicrY1n39/nggQfg3DnYvHnmZJipULuKGQCnM1Km2NmqCsyaHVDEPlMop75jJWgAXae90088rmV/wXIwf6l8n+ml3rJlcqeZnl7A1NiPHJETlOmciXR2yuiUqZgxFDoAWm/byIiMeJ3qGUwxdsZsXfS10KCIfSYxGX0nk0fRjIlzu1k36qDB2I6ua6kXzMrYWUyrjHxPYUyf7aWerIxl7j9TxaSt18QwZHRKuR2ZhY7z5YyUUStZ5y8Usc9GZLDAx70lmTyKy5fLuHaPBz74Qap0nUfu0nm3UVrtGmmMvXXreBZ2ueDgwQmWfnswhHagMNFUI8Rf36Gj46c5oM243lsupJNpY+PUEFYhA2C5ImWmayWrcqTODBSxzzZkWpXido9/+9I9ij09UpswV6O2toLPR2PAT6P5Mh3Ux7NTNJpR45hg6aMXZiaP9bsxHqfR4YDAdqD0NzkTIWiaHI9aWqZXY59peSH9WpTDIp4OLVw5UmcOithnG8w3rrZ2nAU+7u1L9yh++MPwgx/Ilaznz8scMmvXjj9upji9DBqH1dK/xhWiMdQpxdx8rFZGpshGCGZUynRr7DMpL+SKlpkMpmOwmorBQ80ACoMi9tkGv18S6f79EyzwcaJ1ev6ZlhYZ+263y/QCp06l4uMhe5ye6RW0kHejq5fGl3fCoUNygBEiv7BcRqbIRgiZvje3n+zip2IjS6eqnfR99u2TGSTKHbs/HYNVuQcPNQMoHIrYZwrZ3nRNG59L5vx5uPVWuOeeidtZP5tvaWenHBTS2S8elzr8iROy7eZmubhp715IJKScc9ttksQ/+1lJ9IOD8Fu/JffPJyyXkSmshDAyIk8pFJpIFC4XfP7zqbVcjz9e+uKnqSSKUtpJT+QJ5U//U/a1GRmOX87BQ4VSFg5F7DOBfG+6NZeMzzeR1DPBfEtDIblvupnU0wMvvii/e/ppmVWytxfeeUfmpRkYgOPH5f8vXYLVqyWxHzsGN9xQGKOUiSlMQggGZTf3709NPqxEEQzKSYXXKycoxWRAhukjilLascbMQ+axfS6gnIPHTPs65hIUsc8E8r3pkzV10svwhUJSWzcMmfK3v1/+W7JESjOjo1BRIc1Dm006Vs+fl9t+7GPwqU9NO6OY0ktFxcS4eLMrpaTasWK6iKKUdtLdKOmkvhC1ZhVKWTgUsc8ECnnTCzF10t/uTGX4QG7jdkvTdnBQknd7u/zO5ZL7xuNw+bLMFplIyL+rVklTuNwokJXyXaZAADZuTGU4zpZWJ1fz00EUpRBSrn0KlXamm/yno72plo/mCxSxzwTKYXoEg7Bzp3SI+nypOMBMnjaXS6YAjkalhT44KP9GozLh2B/+Id2XwXj6aaqcMTxLl0Jd3dRk2ypCcM53mTRN5kEr5jJman7LlkmeUwFtlnKrs5FYIU7k6XY0Ksfm7IIi9plCEabHBGIIhSSpHz8uSX3p0hTJZ/K0RaMyBLKqCt54Q+Zur66WBD8yQrsW4Mt7YYVTBtX85v/QaHpuijJFFik457tMxVpw0+2ASye8rVszl6otBpmcyOmkOhURRLmgHJuzC4rYZzkyWkK6nrLUw2EpqSxblt3T5veDz8dQR5hYogqvw43TAdTUwJ//OaEQ3HV4Bz5vnPCQg7PR7TRZFy+ZrJD+ppZiik6hsF1IdwptPtexijltK+G1tsrxt75+clZt+kwmE6kWQv7lJF7l2JxdUMQ+y5HREhojaq65RkoqDz8Mzz2X3dOmaVy8eysX/2gno7aluOx1XHX/Zmruvx0CAfx7D9JGHB0/Dehytak2pk/s2MFQOE5/xIGxbTvNgZTncvixnQwmPFQu9VH1eIFMMUUesEKlgExLAA4ezJ6xIf1YxUoOVsKLROR4XA6rNn2mkk6qhZC/9ZwmezuUY3N2QRH7LEdGSyjTW7RhQ863qufNEK7oII7FlQwNVXDW/0FqxryNjQE/3k0OBnt1qmodVGkuGd9+/DjDHWFe6lhHbVjn2GNB7v20TqPmYvixnQy0HCdm89Bzop76HwWp/4MC3+Yp8IAVIgVYCWzLluwkbR6rtlaqXdYwSutvJ07kD7G03iozc0OxVm0+4s1GqvnI3zx2uSx55dicPVDEPsuR1RJKf4tyvVXBIFc/9xWMnjcxegQD7nqMFa5x+1Y9vp0qk32+/OUrJfdE1E6jEzyL7Nx8+lmMb1aAo5vhQYOozUND5ByeaB+DT+/l1OJA0cm/yhVJ4ffLxUzBYOaFPJkILNtg4PfLCND9++W+Tz8t/wYC4xcGg4yzz5eIy3pr0mqh5EUxM5FCBxhr20obn59QxD4HMClLaMzR6m4/Q8JjJ1ZZQ/VVK/DUZqkSEQpJ3d7rBUC4a2iPrcfXH+Ka8C+pqtkECQ9eZw+DdgcRm5cjTXdgvxDntSd1Th/QCrb6yh1JIYQM1Rdi4m/BIFy4AGvWyHVZmXRoczDQNLj3Xrn4d9kyeOUV+OY3U4ukrAuDzWMV2u9i7mW5Uwpkaltp4/MTitjnO0xHa20t9oEB7I44rGwa/wZbGba3Vy5eCofB4cC9ciV39Pwc59u/wBYfxfb8KXj/+/HUeWm8NoFxtp9rl4S5cNHONYs66Q6H0PXCrPZyWovmYqb3vU86KfftS7kaQiFpWZ84IRfS3nxzZh3a2ra5+LetTWrjy5fLvur6+IXBhZJhsTOT6UgpAEobn69QxD7fYTpa166V+V7uvx/uvHP8G2wVjv/jP6SZevXVMk+MYeD56lehwiH/xWKSQevq8Nx8M7S0kKiqJhw6x9r/+Cb+imfRXI8DWl42K6e1aB6rtTVFhEeOjJdcvF5JxkND+Y+naTI08bHHwOmUVvvGjfkHhEwoZWZi9rmcKQVypSdShD6/UBZiF0L8b+Au4JJhGNeX45izCXN2+bbZ8XzB0yYr/upX0jxdtQricbqMxRjff5ba3n6cAwMkk0kQNhK/PIwzsEkyZFsbi+p6WNvfysji5TiNU7hCQWimoADu9OwHpcIk23375GerdOH3y9OKROTYVleXiuDMRbjRqPy8fr10ot53X3bHZC6UMjOxDno+n8w/ny3qtBCoBUQLC+Wy2J8Cvg48XabjzRrM2ReimJqmVvPU4YBXXqH7qo3s2QPvv1yBrfoOfi38zxiRUSLOakaHnDhXb6bmqlrZlhDYjx3DGwtL5yvkDeAOoY3rnhmNMtmQu3vukZZ6eujftm3jF+rmCwGEFLn29kopptCUBekoZWaSKZpmMs+gcpIuLJSF2A3D+LkQwl+OY802zNkXIkvHs62EvLYzSuOYeRp54zi7L9/Hj7oDrB08wMqRCwy76rDFB6gwRkjGHZxffCM1mxul2Dw8LL2WkYhcuqppMvzD4ZBpDs6elQnHzL48/zyjJ+Ks6NhMcnOgbAt3ILtMEghIck+vvpSLcMulP5dynGzladP9B4VCOUkXFpTGngez5YUoWg7K0vF0knjsMWkRXu308zdeB1W9vfRVLeecN8CwofF3bOcj9UEernuWS29dYlnvW1xadA3XPP8PEKyShH7hghSwzQRioZBk0rvvhj/+Y0gm5XcejyT+xx9nZTTJH/UmeSbydSKVd5v+3Qlx46Ugk0ySrfpSrkRb1pj3dBR7P4qNhslUnralRTp/h4dT/oNionGUk3ThYNqIXQjxMPAwwIoVK6ar2UljNrwQJclBWTpu5fueHjh9WkoTbUMaH/mz7dy5Tsdw+Ynv0bimEiLNGrdt06hvDmB7ah+x/W78Qx24zujwbgwWLZK53IeHpckdi6X6cO6c9DyaOsvq1bB4MRw9ijMxSg2DPHTpS9z6xQ08/l3tSmz43r1FaO4ZGDYYTFnm1gSXmWZe2QaBXNc70+9mG+V4RtL7Go1Kcv/c52Rizo4OqKwsfvaonKQLB9NG7IZh7AZ2A2zatMmYrnbLgZl+IUqWgzJ03Mr3b78Nf/u3qd8i9Rps0WgGvkCIrhadhs3+sTQCGnUP3QMv/wBCF2V+mr4+6UCtqJDWusslE5KZbW7eLJkvFJLbfPKT8vs9e2BwELvbRc3KWjbU6nzqVlh2WmfRDX7OxLXCzjEDwwbbNR58MPXV00+nFhYVOvPKd73Tfw8GpSJVLj9Mpr7qugxWGhmRIZCRSGl5bkrFnA0gWKBQUswcQLnlIJPv/X54+eVUabkrzsFQiOY9O2iOx+GIA5otTFVVJc3GSESSeWWllFqWL5fk7XZLwXzbNnnAp59Omc9mOb5PflJWc2pqkgOBy8UHf/x5VlzohXNJ2pq3sLr3diCPtzIDA7/wssbgoOxuV5ds2rT+M828MhFW+vV2ucbnk0n/Hcrrh8nWV2t6oG3bMrcxFc7+ORtAsIBRrnDH7wIfBOqFEOeB7YZhPFmOYytMnRykabJO6ITjZjNZdV2OAPffL9P/2u0ydjASkcWuf/hDOHlSagc7d8KuXZLMb7yRi50gHnuU6ncPYxjA2usY+ugDnEfjqu+9QM3R13mPw4H9os7a+FEcj34bGp/OHYqSxrDtLj8tLVIZevttaGiQ44n1fPNJKuY2+SJS0hOJZapGONl7kx5rXsgzMBXO/jkbQLCAUa6omE+W4zgK2TFVclDG42abIpjfm2XzHn5YEr3fLysyfeELUp6x2+HMGfj616G1lWHDQ+exCLbIIJf7vXg8ED42yk+HYeOlPZzva6Ny8DK2qkrsAliyWGoOY+Z2RhkgQ4z+KV2jrk6uqzp2TEaORKNy00zXzrouy3Tamt+bTtODBwvT5qfaD1OoFDIVzv7ZEkCgUDiUFDMPUHb9M5N5aDZy992we7eMcHnuuZQJGwxKiUYIWcDj3Xfhq1+FxYtJVC7BSC7F5rDjSQ7hSYJOLUND4CPM2cgSGpLVxGLVNDmHiHUP4aiqwLl5c2armhB8/vMpDenxx0HT8CO3icflwtmjR2U3sq2P8vsnJvt69lmpKJltFUpq5jEns4goG4qRQqZidjcbAggUioMi9jmOKdM/rSaptZHubknqaVmpurqhKu7EZXdiB9mZaBSiUdyOMIhmvr32i9jfeoPNNSfprF5NhcfF4pNHqYnGsVW6eP7qz9Bpa+Yq+zl6q1fwiVCU7jeCXH0hinuNn6O9Gu3BENrxp+AXv5CFQk6duhIfaSWgzk5J2FnWR105teuuk1EmGzbIXDK9vTLfjHlqW7YURmrplQrLGSlTiBSSPriXm3xnOoBAoTgoYp/jmBb909rI0BDDPRH6Xx/L3e73EwrBP/5IY+ugwDuSpBJgNIoNg3gsweBiP44/3MbHfM1cs/vLeI8GcfTHSGh+eq5dxam+JVQag0Rjgmb7OWyrVnDj4ecQXw9zfegoI6Pr6Tvm4/Larax7dg+0n5SFt+NxGTff3X0lxlHbvBltS4BgUH49NJS5wAWkkmx1dEg93ueTE4506zwfqZmVCt96S2ryq1eXN1Im36xBOTcV0qGIfY5jMvpnwRKOpZFBh4+veB/BFovSa/j5DDI00RaL0l2/ltHzx3GKUWwYRBuaOCOu5sVFn+bNXwT4080HcZ1+F1f4EnaHDWeojeZ163A0Gwyci/Jfu/6GoWGB/d0oFxavp6rKi9EXxp3oxu2E9RdfwB47z5CvGVesBVtXF3anE77zHSnHCAEOBxefeJo9zwXweCYWmLKGD8bjMpinq0vmg3noIXm6ptZeKHRdrtMaGJCTlJGR1LhT6ICb617kk0KUc1MhHYrYy4SZivMtdbl6MCgXArndBVh5lkZaO/207dfw+6FLT7X7nM/PwJDAKwR9Pj+ukTCDi67mklhFXZOL9x36Oq+9GuFTZ44jkqMkhcBWVcXl2+7nW6+tY5X9ZT4w/BaeVRrO9jOsqbmI/fUQ9FzmPYTpsTVQnewj0dlGsn+ISNLDqGsRvqWVuLq7pVi+ejXoOvFnvks1zSxfp10J5Ml0jawFM95+O3W6pqVt5l/Pd039fjmmVFVJfd5ch1XogFuIxZ1r1uD3jy8y4horgAUTF3qpePSFAUXsZcBMT4VLWa5+/ryUp++6q8BiEWONNIfAkRbap2nwmSc03n1mG5e/vhO7zcDlGcF97/28+atm7v/RoxjxOJWJQWIVi4jHK3AbEWzV1Zyr28Dx+gC1TS6Sp7+NozOEp9IOVy8nfvwtEgjsxBlKVjAyEGPI68IT7cVmNyAZJzEQAa1eOmx1HXp7qbt8ko9d2MG/sR2HT8uoO2ta5oIZULz1q2kTk4wFAvJfISSaKTqnmOenvV3OOhIJSep//dfyOACbNl3xLc/4c6owfVDEXgbMpamw2de1ayWxnzghVzQWKuFkmyFoGmiPBbh4/TYcX9pJZW0dlZcP8eD6lbjejhNv1Igdb4NkjIS9Co+IwfLlLPvxHhhs5id1AS598Gn+ePULVJ5sgbffRiTigB1BEjcjNNHBor5zOIgTj9tJ2H0MVS7GWOKnsqZGslokQuWvb2Zjq47vVp26ezQZRXNQn8Cw2QpmlCJtBQIybD/TdckG03p2ucZH5xSTUsHU93VdDigg5aexAljjBu2Zfk7VbGH6oIi9DJhLcb5mX3t7pTV3773F50LPNUNoqo3CtfVX2KNmdQO4gc6TxJa4Of2JHSy9fAz7xZOEr93Myf062jKdNyIadz7cTM25WnhjEC5dQpDEQZI4TipEHF+iExejGAgqSOCJ9RO7mKTFeRO3RI/gXLlMekJbW/H6fNx4jx/IbqZmG6RKDe0rZeZkduvWW+XswVq6r5Bj6XpqlhAOS6u/sTFlsZvLDGBmn1M1W5heKGIvA+ZSnO+U9jUUknGGo6NX2ONi8wbsTeuoHuqkYkUj7/mT26H9eti5k9E3WrHHHVy/pBPnUJCm3XvA6JCrWkdHsblcJBMJhN3Boio7rsEoREEASWyMuqpxuGwsu3yMmBucGzbI8BZruaFsK4ws16MYPTvbaRd7PXVdEnFVlfxbXy9nTsWW2+vslNtbUw2YmRtg/KA9k89psbMFZd1PDorYy4S5FOebHqI+2RcoFIL2YIirntlBcjTOIpeB9967aNcCPLNT55aLdfT6bua2Ch2vGQfo8bCot4eKhJfmw/tZE+vGt8qAS2elJzKZBJsNW0UFrkQC18AlSCSIO53EcSBcLjpqbiDirObd5lu5t/4VyYo+3/hk5VNkplpllFxFMExHNYzPNNnbKxdQmfs98kjp5fYMAz7xiYkkngkz9ZwWcxuUdT95KGKfByiVnMvxApnHWNqms+FXceLL/SyL6dRebuSnFzXOGPBhnwPCOv0RB15IFfPsCrK0Lsr55X7quoewd52WqX4dDpkOOB6XzDk8LPWGvj4cxHG4bLDhepp/979yuj7AhwIaXu7MfBGmwEwtYL3Wle0+/3k4dEh+XrtW5kxzu+V+q1bJ+iODg6kyfIUOuOkWcGPj7Ca/Ym7DTPt3HhUDAAAgAElEQVQC5gMUsc9xTIac0+WAYl4gk3g6O2Xbw0v8xHCwLKoTSTjY8z0/Q3VwtE2DVdu5qlnnmm1+WQv1wAGGWnWOnPERiQhGT7ZCdYSjDbdxu7cTZ2019PfLIPMVK2QKyv5+mYPGMK5kmGxY10jDFrPDOUxRTZOhI/v2jU/SnuF8CuH+tPVaRCKZLVHTMjcdmZ2dMmfamjWp/QwjVarP2pd8+eA7O2WIY6Z2Z6uMUehsYS75rGYrFLHPcUzGunG5xssBZrnSfLASz8iIVE7OGRqH6rfz68t1QnY/Q3Ua69bJ7a+/VeOeezSazX5t307LUzrbWvwscbXz33p2ErN7WNr7DkP1K6mpc0hz1utNBYafPClN25ERuRJI16V+3tk5XoOwaiRjyWEuvtGO77//Ds7kCA6vWy5ospB7sYOjlXh8PimjZKoV7vdL5+WpU/LzihWpAhl2O3z0ozLePd15neueWvsqhAxXTT/96ZIxyjmApB8rn3U/Wwev2QJF7HMck7FuolG54rKqKiUHFIJ04rnrLikFuFwa0ah2RXc2iS+9PmcIjYNonBfgd+gIw6BisJuVsVNULL8aHHZ63v8R3jA2YE9EWfObLpq+9pfw2msy0ZjPJys1ffWr8qRvvhm+8Q0ABj+/g+GOMLWhozhXLmMkPMKFriXUdHYQcbipDPdi//73xxG7eT7ra0OMnNBpD/rRNC0reWQjnlBofN52TZMx5OkaezAok4394hey++kTiFz3NJ8EM10yRjkHkFzpk6e67fkKRezzAHfcIf8WG7bo90uOjMcnygGZYDWGrcSTqd3m5sykaL6UHR0yeKZxUS8fEK9SMRzFbsSxNdxE5MhxTr6xj67oIb5Rv5316+Hv+oZxjzlU6e+XljzIzy0t9P3DU7xtrCX2ehybvYrr2oeoudiCIxbjhlgSGwmc0VGw2Rh55XXe3BuiOSATh/n90DASYuP+HTiIs+5ZB+3adnbs0YjH5W+P3KfTGEidTDrx5CKn9Guj67L72chX02Q2ypYWaeWbC6fMvqaTvnUAmsxAX6ocNdkBpNhjKQ0+PxSxz2Gkk0mumhSZUIxDK1OB5Uzyg/XY2Zx+ZuItpxOWRs/hqK/BWeeFc+eIHm/j8mVoSy5jSewCN8eDODsbicScuCsqUh1oapK6OZDoH6DrH5/HnXiZ4dEq6ptdOJMjiNFhhBAIDIbtXioSIySranjnoo+XntQ5fUC7QsCP3Kdj9MXxrPVT1atzqkUnHtdYXysJ3+iLw4Hs5mExZJOPfM3C2+GwlMrWrx+fMdI6kMPEAcUcFDZvLj2uPpsVnG1wn4wOXuxgpDT4/FDEPkcRCklfYDicOSKjUFgJOFMRaBNW4mpthRdekFEe6RZjvvb9fuk0DIelvtxds5lEm1ua70uWcP5DD9D1w1/wa5dfIZaAOy49zam6j+D2uWQHkklZku/RR+Fb34LOTmI9g7hjg9TQRSTp54feT/Mhzc0tp76NzYAkEBUenLY4A6MuFve1sdLXCxcOXpFdGgN+Sdy9OjgcNGz24zgCIyd0HEjCpzf7RS6EbKzXqZCkXlVV4/+mZ4w0UxZY78tTT8m8NxUVcOSInDkVutCpkNTAxQzuhaLYwKW5tG5kpqCIfQ7CfMFMiw4Kk1IyHcd8Odrb4cEHJb8mk7L40d13p7b1+6Xf8uWXZXGklhZpcVtD+ArROzVtfF6VLl+Agc8+jfecHFGczQEOn9rL5qPf5N3B5WwxXmHJxShhI4G49nrcq1bIUJJYTLJXfT2O/iHcI4MMVdThrnFz18aLXG1vxNbukR7GkREMh5MR52L0yhvxjnSz6aePs95VxdXP1ELgcUJotN+xHT86Cc3PqajG1q1AyM+6Zx1UjRE+fn/GgSwf2WSyiLdsyXyNzGt98aI81cHBVLqAdPI1B5TWVvksdHXBhQtF5ACytJlpYLKeazr5R6PZz6FYFBtfP5fWjcwEFLHPQZgvmBl1Yl1oWSjSI1u8XhmCl0hIIvnSl2TxCesxhZDW9uioDNtzuWTYeUUF3HJL4USSnlelSQtgFq7WgI99MUDv5w6w9vQF3CPwZnQtQ5fOU+3ysPqaSrz2ESLf/j6cPIXD68K5fCm+gVEcVbUsC7fh7H9FdrKhAYaGMBCcatzCys5fsLLvCIvccZwDQxjLNZxvnqDvH57iq+cfosutMTqqYRjWgUqjKiAZu93l57WgljUrZi6yKVYXFkK2ccMNqcVHMLG2qqnHf/e7Mtz/2mslsR8/LqNF8y0EyhWJkslCzychKSt6dkAR+xxBNgdZpqiTQmASjcMBr74qF8r09srf3G4ZpmclH9Phd8stcOmS9F86HFcWiLJ/P2zcWB69szmgwZe382+PBbn22LNUDfZCo4/nVj3Cr6+P4urrpOKVb3Ft3wDi8gj2i+B6XwDXSASWrpIjXmurLLoN2N98k/cM9nKhdzlJh50O9yKuj/8cR2wYenowXnyRe8Nvc3bzvfywM8DlSo01aywEvEUjhFZUVsx0kitGFzavtVnJyRr5kol8TT2+rU3OnjZulJkrcznTC4lEyWShZ5uVqEiV2QVF7HMAmV6ayWqMfr80al98UZJ0TQ3cdJMkiJUrYenS8eRjEpNZOu7WW+W0/+hRObgcOiRznWTqe6awwHwk0BzQ2PhFjS//aYDkaZ2OUT91Do23W8HREeL3+p8iWrGI0XglVV4D98qVcqoRiaR0CZBsNDyMEXEy6q2j2hXB0XeemHsRdpcLnE4qGnz4Tx/C82ofDZ4DvHT1VlzBKA0+P36/7JhJcvmyYpopBNJrpxajC+caBLKRb67ZW6Z7UMgMIlM/MkUD6XpqodpURqqoGUHhUMQ+B5DpJdyyJf/LmwuaJjM7dnTI5e2RiNz37/8+s0Ms21T985+Hl16SFYP6+qQM8MQTmafyJsGlO/z27cs864hGwb1ao/a9GgMnZDbKU6eg9maNb5zaxqOjO/FUG1xX0ZYSoh95BFpaGBqGXrGcxuD3cRoxquwutD6DytgAMbsH+yKgpwdGR6l87SWcvjriH1jDqqHzvG9kJ/1GPVXCQVX7VtCjXOPy43BoObNimud74QIMHg/xW+/VOWP40XUta+hjtvtXyiCQcc1AlntQSGSOrud2kGZaqDZVkSpqRlAcFLHPARTyEpby0Jv5yBsaUpkBc4VMphOTpkkL8c03pVzg9Y5PTZDNKkx3+IGM4Ejvt3WWsGwZ3H67lB16e8G1JcDlW3fxXoI46ZbpEceYtp1m2r52hJrB47j7BbViGOfwJRaPHdfhEthGAWHIsJGODpweF8v7j0unbF0dlctrZZbJxx4DTaN5ZIS/ufFDdMYXU3N7gOZmxgLM/Vc6bYZyNkZD/G7nDpb9NMwWT4SmB7Zh+hDSkcvCL3SALqVsXq79Cn2e0o9tLlTLNRiVanWr2PXiUBZiF0LcAfwDYAe+aRjGl8pxXAWJUl/eQlDq4iaQhPT978sV/uFwKg+KOfBYCTwSSaUsMM9n3z752ZTE0y33TOdtLnxyuYAQeJ49ABXjA/lDIXhz6R2srOrGfihG9fAb2G02GWJiGDiG+0kiwC6wnzwpnQQdHdLR4HBIneXgQelJrqiAq66CYJDFr77KYo8HWtYyJLwMjFbIgt6PS+ZzueDwYVh9UacyGmap7RQr3WE8u3fS3riLU1Ft3P2zWvhtbcVHspjINQgUI+uYKPR5Sj92vmdoMlb3ZGPXF5qMM2liF0LYgW8A/wk4D7QIIZ4zDOPt3HsqFINSX95sCAZlyKFhyCl0Pgsx/cVIr9yzbBnceacsCG0l5q1bU6GNe/ak4qo1TZL4kSMydPLYMRnVkW65Z5olgCSIqy/oxNviXH+XH99YjHl7OzTs2sH6s3EqGOX48ttYPdQFbQOSUQAwEBiIxNhHp1NeBJDTD5tNSjtNTdIBcegQsUiMqLsap9OFca6T0921nKp/Hw2ndFYFdRoBd4vO7T4XDHSyJNlDVSLMiNNHd7eHJx/T0TUtoyS1Zo0k9kIiWTLdi1woJeY71/NUqmQEkzNASjkPa59nSsaZqQGlHBb7e4E2wzBOAwgh/hm4B1DEPk0o9qE3Sfmtt6S1vWiR/LxrV/YIj/QXQ9fHV+7x+8eTuoloVKokmV5mk/g/97mU0Wwmycp1DiZBuNf4ibc5GP1VKyyKcLHXxTO7dW7R4wy6atkQ3s9Nzg5cQ31ylc/AAImELNQhSCCsnTRz6cbjUk8wE475fITfexuh54+y9NJxhmxOomtuYMjpxY/OSGyE0AutiF3/L8uJ85cnj9Iysp4wVZxx+BEjdfTpPp7Hz41XycOnS1K9vYVFsmS7F4WQeyGRO9bts+XCKSanSzqyzeAKRTFtWTFTMs5MDijlIPZlQMjy+TwZBEUhxMPAwwArVqwoQ7MKVhTz0JukbLNJK9nnk5+zPfCZXgy/X+5nrdxjWvLFhPlFozK0srtbOmCbm/NbrOYxj/ZqDK/dyuYhOfVwfmknFe6HqfQ58HUex+EAt7YEzp2ExYux1deT7OiEaBRbwkAkk/JAXi/89m/LacPFi6mVQckkuFyE/Lcy6LzAUJ1GfDjKiU3/je+d3MC6oSC3JJ/F/cN9JIfaOLt4A6s9cSqqqqh0G7xQ+QB9rkaOR/y806HRcQA+8IHU+ZVihWa6F+b3xViF+Ugn0/M0WYLMNYObSkxWxikVM+kXKAexiwzfGRO+MIzdwG6ATZs2TfhdoTCUY2rn98u0sbGYVCBiMfk52wOfLewt34KWQsL8XC4pQ0Sjkksffrg4eeHaziiV/+IhevwUru4wW2y7eX7DNqo8IXzspWEoQaXdLi1wjwfHhhtlnGZHh/xOCDlN2LBBjiwgnaaLFsmpxvLlrDj1U87EB0gOj1I5Gqbxud24N+wi1gfLF/WhR5ehjbbhjVzC5nTQ4Bpk2OmjZ1WAy5dh8SWd9y6F+FKN++7LLC2ZBJ3v3NOt3t5e+Md/LM4qzJeOItszVg6CzDWDmypMRsaZDGZqQIHyEPt5wHqplgPtZTiuQhrKNbXTNDnt7+uTmu7580wgnPTtM70Y6VZdrgiMbMfOlDrYmvo2l1ygaUDIz/CuCAPnwgzafAw4PVy1LMrzi+7jfCLAje0/4hNL2nE7x1ZS1dbKUezSJRIVlSQRiISBo6cHfvUr6ck0R5nubujpoRq4LtnGqF0wUL2YUZuHD3mDrL70DPXDx/CMnuBszQ0cW3knjXcarAr3Eu6Os/2WN+C55zh6Ng5RB//etJ1AQBtXLk/TcpfWy3QvrFbv7t3ZKzhlQr50FLmesXIQ5EyRXakyzmTbnKmcNuUg9hZgtRDiKuAC8NvA75ThuAppKOfUzgx1jMcluefLDFnIi1HKS2tKOiaRWJfrb92an/RCaPxiwzaWH9lJ36iH/riPZw/7cV4tz2vjwe9B/Aw01cHixUT6Y1xesoEG8Q6MDGHY7AyHDWyX49SY5fgMQ+rr0SgJVwWjJy+QWNRAdzd02K/m/ICPvrbLNIffxtlYTe1APyO/vpb7O39I5YvDcPgwtbW18JMk3HAD773jWmLHTnDjB4OMoI0rl7dihYwmKiaRm9XqNS33Qq+5+QxlW9CU7xmbLEHOJNnNBGZiQIEyELthGHEhxB8D/44Md/zfhmG0TrpnChNQbmtnMqGOpSTCygRzn2BQRoUcPZoilZaW3JWEgkE5EAwMBDg5vIuVhs5Ft58lNRr0wNDJIA0Dp3Hah+F0HyN2Nyd6fdjePUE4qnE1fdiNJO74ECFbMzVm6b0xJA2DaMyGrSPEkKOOrtqNhG75BD8bCnB3U5DFXXGchgFOwYp3XpRToOFhOSiA1OgvXqT63Xfl51eeJVgfIHYaNg7rXKr0k0hoRREzTFyUlKmCUyFySqYFTdNhURcrPykUj7LEsRuG8SPgR+U4lkJ2mNPwYnNtp2OyedzzTddL6deBAyl5YHhYSt8rVsjwx3SSMds387Zs2ABDdRonbBp2OyxB+kOjHZBIGCRt0uKIjgocsREWGWEiIxF6WEzYUU+9o4+GRRF5Ir/7uzIbGmOhoEaSWIWXNxvvgFicUwONxJdp3Hx3O86fCRgeJp4wGDBq8DpGcfW1S5Y9f152etMmSfgbNkBvL7Ung9x//ADxSJw4Dl7xb+cPtmlFpb/NN4BORk6ZDotarSKdeqiVp3MIZsKneLy4XNvpmKykY+5fWyut7GBwci+mVR4YHpZBKYsWydKkn/vcRGvU3N7M23LpklxHtGSJJPT775dpDg7ZApwR1+CKnaB65XJclQ6az73NoKimPtlNhTOJ27hIpcPAY++XYY433QS//CUAQtgIO5fiiYZZMXScJSvdrFjXStX9fpqiUYbXbuTScBWXj1/CdkngidezqqIdR0WFPJFkEl5/XXaqoQF8PuJxWFIXp9vrZ/FlnQd/XScQmHjx8jnJcw2gk5FTpiPueiajRRYKFLHPIRTyQhTyYk52uu33ywRi+/fLz3v3libnZOrPyIjMDW+zSdI2Y7ut03aXS/o1h4bguutkrWunU/pEa2ulcbx/P7wb0fj/VnyR/9G/kxVLPSyx92D3Cbz9nSQZJZG0U2mM4BpNENu1m8FnX6KiqYnKBx6A117DZrNRf6EDIxrhmr5D2EUNVf/0Dn2/+BFv/d5f0NPmY3QwTs/wUvo/eDfL2n7OVRdP4OjvlqtW7XZ5Yg0NV8Tsmnbo/vYBmkZ0DJ+DVbdPvPiTtWjT76/LNd4hnQ3WdkdHU/lwoLxkP5PRIgsFitjLhMlaOuUg5EIJYbLTbU2TL31fn1w1Wcoy+Gz92b8ffv5zKcUYhpSdrFWDTIeqxyNzeHm90oGo66kl+dFoqpjHiCfAv9h38ch9Omguhv74MeJHjxN3eqkZvYTDZUMkE/T1QlcsQfRShNWVXrzr1zPaehLbaBRbpRv78BCJ/gFGEm4cPb/gwrndvKHdQ+PqKKeCl7ntl9+h3hHG7kASeV+fjCPt6ZHMOqadNWtge2Irgz9toeo3NtPUzATWTR/Ag8HS/Ba6zpXC4oUMEtaZ2P798hT27pX3oqKifLLJQnOgzgQUsZcBk7WwykHIxZbKm6y33oyq6e3NbnUVu/Rd02SFpooKSUgjI3J1bGVl6pxMh+q6dVLliMUyL8nXNGsxD41GTYYZftn1RT5i28miaBdeekgkbLiNGI74CBFfE/9Y8zl+c30U341+fviFN/jv/BH2EYNa2whGEmxGnKp4P5u79nNT908YrGzk9tEw9XSTeP8HcLR5ZKdqaqTV/t73yhHqhRfkBdE0mp4bY9rvvArfncia1gF8dJSshT0KuZ4HDxZfi/X4cfl5zRqZNscwUrnhyyWbzFS0yEKBIvYyoFyadSH7Z3ohSi2VV+oso9iUrsUQ0u23y1Kmw8OSHyMRSdrmOW3enHKo1tZK0jl/XjpaP/5x+MhHcsfZh5oDfKF9F1pHkIfsT9NU2U9jXYzvux/gqc476Y5q6C1QeSCE0V7LN+r/J1tsr7HmRif17ccIn+ikJtLBgHcZywffZnH4IrakDJGsePnHcPPN8Hu/Jzv28suS3A8floVIHY7x8Y3BoNxu7VrJoGPOCusA3tkpredSn61cs7z0+2+NUNq7Vw7aPt/UpuNVmBooYi8DyqFZT2b/fLHJmVAq8ebaL1d9zEIJKRCQZd6++104eVISeWvr+HMyMzz6/VKP37lTcmVLiyT2bDBj5oev0ziY0EhoAVYaOrf9vp9//Y7GhS6o80LFpRCfOrsDVyTMdQO/pMI+StWoB/fmG/D8xp0knnqShlgnBoJE0oYw5PLrRDyB8e4pHDfdJE/kzjvlNCoUkh7hWIyR3iGGIh7c6Hh9PjmCmc6KZ5+94qwwiTYUmlgOz3ovzIVO2Xwc2WZ5ufK+aFrKr2G2p2STuQVF7GVAOTTryeyfLzY5E0ol3mz7FVsfMxcCAUneO3ZkPierJa7r45eo59Kjrde5txd279Y47dFo/YGUOpqaoPJyiC2xfdQ7wkQ9gtpwFwZ2YpdjiK5+Rh/cwteP38bacy9w1fmfs3LwLWpGOzGAmKhgaMSNeKGFOpNpN2+WlUfa20kaMHJxhDe19YTERv7T33+E5lAQvvlNWLuWofO9tO3TqbtHyxueaBY5MRc6bdyYKnCSjkyzvGIjZ/JF0bhc2Wdv+bDQUupOBxSxlwmT1Qwns38pA0Ops4Rs+6UTRa76mFYUm2EwV39GRjIXrMiETKs337s0xMe7d7BxWRjf2aNcNGoxhJ0EduzxUYY7B2jvdtFT0Ux4wwf5ft3tLAqHuL7zJbZc+GccFQ7idhftDZupsza0dKnMLBmJ4hntY2Pn86ztfIWLb9xE853SWTF0vpfDRx38G37686QuNq93b690HkOqwIn5W77noByRKekS4Pr1chAuxsc0mZmjGgyyQxH7PEGxA0Ops4Rs+2Uiinx9KlTWyXWMYvTo9HA+w0iR+sMPw8pzOqsq43jXrYNWCNvXc/y5JSyhi6bRsyRrmrj2x1/hE8cMosLNKoeDRU9sp709QNvOdupGL9Jf2UTjhuZUoy6XbLi3FztyoVQPVdiIs7KrBbQ/hq1b6fhuC7+6vILrq3TeCoNZTi8b/H7pYzh1Sn72+WRT1lJ1udIAaxrcfTf89KfwG79RGjnquiT1oSHZXlXV+LTEhR4j2www2/1XC5zyQxH7AkYps4TJWthW5JN1wuHiSvbl0qMztff+98MPfyhDJ597Dr6w1Y/3SErTqtv6ELuGH2Jt2z5u6n2FulXrqL30OutXCC5dtYaGS61UntvHhqYmhm6ppbtqA9cO6nijOqGQhq7L7JONHg8ANiAJNAyfwWhejvP2zbLTX/kKy8908DutIc6cuplVbh/XuLYD4y9i+rV//PHxGnumcMUDBzITXzAIjz4qt//xj+XarGJXILtc0lIfHZVtXbo0sQh6PmQyCPIR92SDFRYCFLErFIxScnjnQi5ZJxyW1mg4nLsIiBX5Bpf09hYvHq/Pn4pqNFsO0Kxp/Pk34Ojz9xB94giXD+sMGLVcf73BysFWaDsKrwB2O16HwGvo4HPQ7vJfuU4NI37+V9JOxVgfbECi2seZ3/o8nuYAWnAv0dcPkYwa1CW74GqBtz6ON6oDKdO13eVnxx4tb5ELs7IfyGCbbGsMzLBRc0BsaSme2K2ZOS9dgv/yXwrz71iR6Z7lC9FUC5zyQxG7QsEot6WUS9aJRCSp5ysCkumYuUJFre1BBgtf0wghre2xj+jrNH5w83Ypkwz64RMQuGgp2Krr4yo5n9K1K9epvxUur7uV5gs6xOMkEYRsK/n3Q/WcezTEp1d2U3MpQtTuoToK1ZFO3L7VE0xX0e2g2rOd6nVa1mtvDVd89tncaww2b5a/hULy7+bN+a9tOswoo3hcWurFkrq13+YAc/CgnAnkIu7JBhssBChiVygY+Sylcjm0NC21ctQsv1eMVZarH+nEX0ixEL8f+n0aP49rOHzQHAC4Z3yGMouY7Ud+1d8a4mNHd1CzKixJf9EiBhOV9A0uZUvffkb0vRitEeLCQUUywknvjdhWfpgbtt4+3nStraVGP86qSJBf6lpOK9UarmjKNJkQCMDTT6cSyhVrrZttTcZBnr5NelRVriibyQYrzHcIw5KmdLqwadMm45AZp6Uwp5DtJS3FoZVvn0IHCut2kPmYhR7r4EF48snUrOT3fx+2bMmyf46DhkLQs+8gq155Eu86/5Vg/C5HE+/+3X66vH6u6Q6iaQb/0bOWxWd/RUI4sNfXcePKHjyful8mvfniF+UCJ6eToRsCvPXgEzQHcjtWC7m2hVzLchBnof3Idt0VxkMIcdgwjE35tlMWu0JZkE+mSSdfM4olHJYarRmuV6xmn04cd9yRuSZooSSXbVZi7UvqXDS0LZkPpBFCa+oEx2gqGH/zZhreeINFvpNE4jruq5x4XEk2jLzLgHMQj8fA3nseW1cH8VPv4Nh8syzXl0yC14u3IkGgUc8+UlmCyntaolSH/Tmlm3zXshzRJoXKd36/jKwJBoufoSlMhCL2BYByWWG5XvxcMk2mMEO3W1aha22VmRnd7uKr1sNE4oCJ/SjGN5BPXiiI/KwbGYbU3zUNvvxlCAZxd3fj9nikznTNNSwJHcI7GqNiOIKRNIg53HRHqqk/10lFjUeunDJDhLKFjcC4oPJVq9bzsTYf/8Z2HD7tym65noNM18n8vtRnpxhHpxBjOfDF+EuptPTioYh9nqOcVlgugsxFiNb9Xn9dvrhLl8KLL0pjtKICVq2Smmq+c8kXPx8IjF8Ob25XTBRFrplCQYOEGdZTVSVzxTQ2ypMLh1OpfJ1OmQzHMHAMD1DldBCzOYknbCQr3NiSMfq9jTQs9cpVSG53qtJ3prARkN+NBZN7l1SxsTKO71a5khXyy16dndJqNq+TNS6+1GenUB1e1+VzYE02lq/PCtmhiH2eo5yRLPmsr2yEaN3PTNx14oT8rqZGGqJC5CbcXLlNshXatvarXFEUBVmgZoC32VmXS+ZI8Pkk0cdi8p/bfeXk7SIJwuCyZynvNN3K+Yab+fDf3wm0w2OPyYHgySflAKFpVzoxOOqgtdOPpkGzwyEHD4cDBgfx+nzceI8f8oQQWtcN9PTIQiV33lm+Z6cQSS3TdVXx6qVDEfs8RzljfkslyExhhmZIXjyeWoSULh1D6v/5ZguFxLhbQ+pKJfiCroE1wHtwMEXGTzwBzz8P3/62PBnTmn/3XYhGsSdiNHp7qR3+GUb9AN7mO+Hr/yqdp1VVchXQ2bMyQ+TatQzo3Xyr524O/ousm7rz4e1sqNUnJm4Jhbi2U6dhxI+eIaomfd3A/v2S2M1nx1ydW4pUNtnrquLVS4Mi9nmOcsf8lhpmlr6fGZKXLdTQqsVPNqmYiUJC6grRdPNeA2uAt9UTqGky7n3pUpn0/PRpGTKZSMgpSzKJY1PpvQ0AABlaSURBVHgQx5J6GO2FZ56Br31N6iP9/fKCnD0r933hBdw2F/eLn9GyvJFfxQNs262xa1daxMzYSTfG42wXDlrv2k7zWCk+c4Dz+zOvG9iyRV4jM+x0z57SyzEWgkzPiBmXr1AcFLHPEKbTKZT+wswWh1R6vzJp8WvWyO8LTSqWC9bjt7ZKwqqvn+h/nLSmm2s0NVm0u1vKMcmkJGzTazg6KvWQG2+EU6dIIMDlQYxGsAEMDFwpu2fzLcI5OMLVl1toXRHA44H2YAjN2q7lpKt0nQBBOoM6f7vXT5dbu3Ke6esGzHJ6nZ3jV+dOlxxizRppVtDKlh5BYSIUsc8AZjKJ0VS2PdkBI5MWb7XQM4ccFt6WtVZqJCJJLJP/sSwkls2s1zTJon/4h9JEjsXk90LIDjmdMjvXtm1ceuENvM/sxZ6I4USQsNuxY0gHrGFgHx6kalEtdv8KrrkKbhwNcsOunVDnkRft3nvH6fFmOSajz81dpxwcvms7R3u1K9a5WXHK5ZIBPOGw7I7XO71yiPUZ7e6Wl6WQqmAKKShinwHMpFNoqtouNnFXJmTS4jORd6mLocxaqWY2x+eem0hYhVQbytavghEIyE5v3ZqKlPF4JKN6vVJ2Ad659m7e2fT3fKj9GegfZGnVAJXdIam3NzWBYeBesYI/rX6Ok7c2subbj+E5c0L+bmbmWrYspTmNpb/0rPXjOKXjfCPI2iqda1x+IFXcY+9eKet7vXIQ/LM/k8Q6XTM86zNqDsJKZy8OkyJ2IcT9wP8E1gLvNQxDLSctAOV0aM6Gts16qx0d8l84LAM5Pv3p7GljsyGTzpqOUgYncx/T8qutzayW5Ks2NDKSv7hzQbOJ2lqZpev8eUnoLpc86C23yO/27eM9nhUsCv2A/qSXpkQHrF4Fq5fLsJX6+is5iqt0nQ0nvwdtrbKDZk27NWtk8Y6WKHX3bEHzy/SXvl6dDf4eruvZhctbh3ePD5qzj4719dO7CtT6jPp88MgjpRfxWKiYrMX+FvCbwP9fhr4sGJTboTmTbVst9WPHwGaTBuPp07Iw0FToon6/NEhff13yYyGDU6YBLZNakuk760BilinNVty54NmE3y+t7gsXpByzerW02s+fl+ZyKMSSc+eotbkY8dbheM8qPB+3pE8MhaRpHQxKveTkSVlmD+QgsWQJkTeO89a5Wv4Fs3iHhjbmjazctYvKPh2MXmCpHJnHjn2LFuJ3V+i0xf0k1mpy5jWNjpmZfD/mCyZF7IZhHAcQ1qViCgWh0OiSYt6nYopTlOtlsVrCw8PSWHS5JPHmShs7WZi+xkJTHU2GLKzafL7iznlnE9ab9MQT4xOqAzz1lKx3190N4TDOJUtwJsJQ2SyzdZnOAEhdgKEhuSBg6VJ5fI8HLl4kGqvESNawfDkcNe/DljGHal2dvDk9PbKtykoZobN1K8179vBoXZz+iAPjs9tphml3CpXzGV2ImDaNXQjxMPAwwIoVK6ar2TmNbNZftjQhM+GQtVrCS5fCX/2V7EuutLGTNf50XYZBmhEzhQ4c2cgiX7Ueqza/bZsM+csWgpdT6hq7SUPhMdLctp3m++4bf4CGBmnBOxxSe29slFrIww/LjphOjI9+NJUv99IlSfK1tZKkXS4YHKSy8jJ2ljNyQsexTKYVIBiEl1+WI+/SpVLjWLIkpVGNJWr3rvPj1XWI6qCjVgrNMeQldiHEi0BThp/+wjCMfYU2ZBjGbmA3yOyOBfdwHqJQYstk/UFmAp8ph2wmSzh9WT+kYqaz9d+K9OuT/rlQP0Ep6WKzVesxec9Me5AtBC/nzEDXGQrHeemUn9qwzms7dX7HGnceCjH0o5exxR04BiM4AwH4gz+gXQvQ1aKzuiNMZccpSe69vXDxotzP4ZCVrb/zHSnLDAyAw4HTiLLumgjJ3/fzQAC09iA8+KDU4UdGQNOILFlJsq0NPK14l8pEZePSEefzKivMSuQldsMwPjwdHVkoKMayzkRg2Qh8Jh2yubTqQrIv5tKot26Vhmr69conqxR6nfMNiMXcA+u5m8e2fsbvpz/ioDasU+lz0Onxj9u3M6jTdtzNYN3Hae4/QdO9v89I4D527ABHBzwYjHCdCFPZ6MPldDLSoNFXdxWLxCDeSAQ0jcElV+F46QAs13A3L6Zy2zYCYwuS2NciCT0ahYEB4qfPcrB5Cz5RSVvvrXzor+6Ri5eamwvzKivMWqhwx2lGMZZ1NgLLllZ2Nr576ecLuQeg9O3NEm7p1ytfTHuh1znfgFjMPTCRdVDRNIxt23ltp06nx0//WNbFK+eOnzgOGl299HiWEUfDtu8gjg4/r57VeGd4G48ld7LI7eHa2lEunhti8OIgIxU+rl2xmar/OMKbh+LYvVt413MrH/loPfXNlsLamzfLBVGDg1BRQdzhprHnBPGmZRxedg8roprU09NH6tmyok2hYEw23PHjwNeABuB5IcQRwzD+c1l6Nk9RrGWdKfwvG4HPlMMp13uffr5WmcblmmjVpm+fTRmwtp2JRPNdZ2uf8w2IxdwDyD2oNAc0fmeXlnHf5oDGVzdupzask3S6+OwrexDxOB8/5uBwdDtvugP8P4t38SFvkNrhvQw4K6hxRTiw6hFEbQDXvc280qez0tfLDa/uJvldD7z8bGqhUigEn/wkvPQSNDWRcFTzH4P30lYXoN+ncY0rBAfTOjaTq+kUSsZko2J+APygTH1ZECiHZT2bIgYKKXCd7XwLzdaYSRkwkY1Ec7Wbqc/FxmnnugeFzAKyDSCfeUJD1zWu7TxI1f44vMfPhmGdX7uo03VZ1mNNLNapdbt5x/ATDevUj4RY+3YIw4CfODXWvbqbZb1vUXNKwOmEXFwQCslUBE6nDFd68EG8gQAfa4d3X9BJ2Nup/soeqJglzhuFSUFJMTOA2UTMk0Uh732+2PD0/TJZyKWQaLb9poOr7rhD/i15gVbIDwfkiS1Z6uDzf+XnAyG5zS2anyV7HNxWqTPYM8JG2zNU/u3bAPzZkhVEGgx8QwPYu4flwoL166W27vHI5aSxmIy2Aaq/sgPnoTi+SDfHHR6u/fg6fL16aiqVnqTdvMBKnpnVUMQ+g5gP70apTttyOXtLmQHlanuy9yR9NlBKkWhgwok1axr3XTmWBs3b8eo63s5OuRLM6YRoFE9yGI9HEPNWMequxkkc58CAjA+Nx2VEzeLFV3Swwd44XV4/q51DiP4IkeM6vuWO8VU2hJBVoMxRKtc0zXIBQ2SWnBSmHorYZwhzWbpMJ79y5GifzLkXOwPSNBlt09IiNfxyysllnQ3kOjFr2NEzz8gVqwDJJN2f/gsufO0HjDo8GDY71/zOfdTfpElnxfe+JxcnfeUrcMMN1Fw+yXsu6oTdTbx48za2PhiFgH/iiTQ25vdMWy7g4KiDrxrbx2WQnCvP93yAIvYZwlyVLnNVMioWM+nsNUMojxxJ5RifzD2xppmd1rBTTYMHHpCW+JIlYBhc8FzLPwZ2cX2VzluDfj5+ncaWAKmCH7W18G//Bj/7Ge7+fla73PRrLlb8RTONAcsJZ6uykW3KY7mAg6/r1Aod75rCi2krlA+K2GcIMxl3XgiySRLTMSBNtURV7rUAhRTwmFIEAqkVUw4HDZv99B/R+Hlcw2Gp83HlBI8fT61uTSRwJOPUdbwNoSA0k7r4d98NX/qSHAi+/GW4776UHJNhutXu8iO6HVQP6VTVOug1/HTps/P5nu9QxD5DmK1x55BbkvD7pS8tGBxfHGg62i4XshF4qfckfaCIRqc3G2ImPX57c6aQUi1Vkujpp+GXv5Rx7Tab1OgvX4ZHH00lYh8aks7T/n753ZkzcNVVMseNVQo6eJB2l5+/2qNR7dlOY0TngW1+PtMsrfVrXCGadR3wz64HfR5DEfsMYrZGx+SzyoVIJeGa7rbLgXKvBZgVs6+0jpv/nThIjm0XCKTqrzocMm+MYaQSsXd3y1zuPp/MQDk4CIsWyUHhqafgoYfGNSC6HVR7tlO9TuO4rnEqCls00JjDzqQ5DEXsChOQi6h0XaYNz5a2dirbLifKOajO1tlXzkFS0wjd+d9pr78TPzqNAf/4zGYOh8y/XFcnR/CuLvn95ctygdPZs+PyQ1QP6TRGdI6nF8ueq86kOQ5F7AoTNO1cRDVVxFvMStBCz2M6MRtnX/nCOqUhreFwaHxBC8l0Atddl8q5/NnPyg27u+HHP5ZJxxYvhg0bZBIyuNKA1+fggUf8nEr3LcyK6czCgzAKTWhdRmzatMk4dEgVW5oNCIVSsqrPl5JP8+2TTqCTIdX/0969x8hVlnEc/z60tNQurgWqsGXKmrUx2oaoFCEoFgJqo0g1aKIxEcVAGmPUPxTERje1IRFIjAn+I6QkmoDGUE0NXiJ4qTXatSupYG0lxUwzLQoF2mKBsjvw+Mc74146tzNz5tzm90madtrp7HNmzjznPc97i6OuntZerlmfi9Asvp07YevW8Pen7Z7gc0e3cM6KJWGp4HonKczdOurKK0P5ZmoqdKjecUd4Tr2YX78InHPO3JlZWX+TcsTM/urua9s9Ty32ATcxMVNWPXAgPG733ZvfOu01qcZxt96vO/52c3GyXj5udidRb0g/v7fC9Y9tYfi0fXByGMbGZsas79w59009++ywIcf09MwOJ/UXv/nmsKXVs8+G8s2ll87tZM3aG1Nwp6UdgOTf7KRarc6MwuhUHHfr/brjb3VsvR53I7VBJlQqvb9WK/Vy2/Xrypy/agknFw0z/VxtE4/6m1ffOmrv3vCmQpjBeskl4ff6AZfLoTSzYEF43sKF4Raw1RuS1IEOKLXYB9wll8DateF7uWxZd1Pge02qcXQ+9qsDs9WxxX0xSfoOoFSCBRePsveuYc6YGmNo6jne8K4PcTaEW7ctW0LL/KWXwo7SIyNhvPz8Ax4dDT3qJ06EMk212nosbB5udXJOiX3AlUqhVJp2Uo3jbr0fd/zthkbGeTFJYwBJpQJ/O289Fww9w+v37uCsHX+GR38fttsrl0OCHhsLT5yaaj77asmSsFfh9HSYCfvBDzYPXiNl+k6JXTKbVLOikyVb4pD4AJJKhdXbNlM9XGXpS88wvXAJS88/Fx77U5i0NDwcauZmYULTWWfNtLBhZr/DiQk4dmxmtMzq1WEBsJ1NLngaKdN3SuySGXGPtsmbxMfDl8sMLa6y5ppRXn7kBc585TmW7NgTWtOvvBI6S6enwyzUffvCCo9Hj4ZEXl/C4OWXwwzVJ56A/fthbIynHz/K/bfv/P8uUadUWmoH+tREmTKjjFCiBIP1YfeZErtkQqOyKwxeKTbRO59ay3n4aBlWDcPaq2D79jDj9De/CUl78WJYswZ27AiJe8WK8H/rpZRdu0KLft26kOxfeIGlm7/CFXYhC88e5r6xccrl0inHVKHE5l+Vwmf7K/jmjRVG7mkx/EgJPxIldomsH9+zRmVXUCm2r+bfIgBMToYlgBcuhMsuCwn90KHQw37ddTO96/VO1GXLQgfr4cOhzr5yJYv+c4TpRUNMHa/yhpfKjI6e+qHN/7yP7C4z0mYp4IG5usdAiV0i6df3rFnZVaXYPqvdItQv1mM3jjNSmYAHHggf8kUXzV3VsW7+BWFiArZtg2qV0xcv5B1vOsFxG2bs66OMNBlHv/xkhUUTZZYPj7L84lHY03opYF3dO6fEPkDiaGn363vWrL6cxTVYimbuxbrE+HiJUn3H8WZv/PyaUX1hsdos1NdMTfGaRYtgqgwVTnmNEhXGbTMnvMqQLWRoZLzxh62O1q4osQ+IuFra/fyeNaovF2W0TZbLxA0v1pd38cbP/rDaTdndvp2h6nGGLl0980Mvv7zxCaCre2RK7BkQx5e+3WvE1dLW9yy6rJeJu75Ytzrpmp1w9Tfj+HF49NHw3HYL+xfl6p4gJfaUJbUAVpwtbX3Posl6mbiri3W7k67ZCVd/M1avDo/XrYMNG7L1hhRAT4ndzO4EPgRMAU8An3H3Y3EENiiSWgBLLe305KFMHPli3e6ka3bCzX4zhoeV1Puk1xb7Q8Ct7l41s9uBW4Fbeg9rcCS5AJZa2unI0kU1tlp/o5Ou0cL+cbQwstxBkVGxrcduZh8BPurun2z3XK3HPlcSNXaR2Gv9s0866E9HQtY7KBLW6XrscS7bewPwyxYB3WRmk2Y2eaS+zZYA4TxtNCAg6deQbOtkpdtWz4l9meHZJ10/1jCG/r1uwbUtxZjZw8C5Df5pk7tvrz1nE1AF7mv2Ou5+N3A3hBZ7V9GKDKhOGq7d9mfGol8vnocOigxqm9jd/epW/25m1wPXAFd5GvvsiQyATjrIu+3PjEWpxJM3jnNkd5nlF48yEteLZ6mDIkd6HRWzntBZus7dX4wnJBGZr5OGayfP6VcHeqUCm+8phc2x98D4SIw/R73+kfU6Kua7wGLgITMD2OXuG3uOSkTm6KThmmbjNutj9QdNT4nd3d8UVyAi0lonDde0GrcqhWeLZp6KhkpKz3q6W9AJGDsl9gHXbiSFvnPSqa7uFrodp67JHy0psQ+4VrVRzQ2RvuumOJ/UAks5FucEJcmhVrVRzQ2RvuumOB/HiTn/NSYm2s/+yhG12Adcq9qoOsSk77opzse9wNLJk2H3p8WLC9N6j22tmCi0Vkx+FLgMKTkz51wkxhr7U0/Bgw/OlIM++9mwVEIGdbpWjFrs0pLmhkgWnFoSL4VdnnpRP7krlZnNueffAeS0ZaPELiKZ19cJUM3KQTnuYFViF5HM63t/T6Nb0xxPp1ViF5HM63r7vl7KKDkePaDELiK5EKm/p1kZJUqyz/HKkkrsIlI8jcooEL1mntPRA5qgJCLF06iMMkAz7tRiF5HiaVZGyWnNPColdhEppvlllBzXzKNSYheRwZHTmnlUqrFL1yqVQq2bJFIYarFLV/IwKS+ns8FFeqbELl3J+qS8PFx4RPpFpRjpStYn5Q3QyDaRU6jFLl3J+gCDrF94RPpJib1Akq4pZ3mAQdYvPCL91FNiN7MtwAbgVeBp4NPu/mQcgUk0SdeU89AxmeULj0g/9Vpjv9PdL3T3twEPAt+IISbpQpI15fpFZOvW8LuGO4pkS0+J3d2fn/VwKZD8PnsCJFtTVsekSLb1XGM3s9uATwHHgSt7jki6kmRNWR2TItnWdjNrM3sYOLfBP21y9+2znncrcIa7jzd5nZuAmwBWrlx50cGDB7sOWtKXhxq7SNF0upl128Qe4QdeAPzc3de0e+7atWt9cnIylp8rIjIoOk3sPdXYzWzVrIfXAvt7eT0REeldrzX2b5nZmwnDHQ8CG3sPSUREetFTYnf36+IKRERE4qG1YkRECkaJXUSkYJTYRUQKRoldRKRglNhFRJKS0H6SWrZXRCQJCS7Bqha7iEgSElw9T4ldRCQJCa6ep1KMiEgSElyCVYldRCQpCW3rpVKMiEjBKLGLiBSMEruISMEosYuIFIwSu4hIwSixi4gUTGx7nkb6oWZHCDsuxekc4JmYXzNJeY8f8n8MeY8fdAxZ0M/4L3D35e2elEpi7wczm+xkk9esynv8kP9jyHv8oGPIgizEr1KMiEjBKLGLiBRMkRL73WkH0KO8xw/5P4a8xw86hixIPf7C1NhFRCQoUotdRERQYhcRKZzCJHYz22Jmj5rZHjP7tZmNpB1TVGZ2p5ntrx3HT83sdWnHFJWZfczM9prZq2aWmyFrZrbezP5pZgfM7KtpxxOVmd1rZk+b2d/TjqUbZlYys9+Z2b7a+fPFtGOKyszOMLO/mNnfasewObVYilJjN7PXuvvztT9/AXiru29MOaxIzOx9wG/dvWpmtwO4+y0phxWJmb0FeBX4HvBld59MOaS2zGwB8DjwXuAQsBv4hLv/I9XAIjCz9wAngB+4+5q044nKzM4DznP3R8zsTOCvwIdz9hkYsNTdT5jZ6cAfgS+6+66kYylMi72e1GuWArm7Yrn7r929Wnu4Czg/zXi64e773P2faccR0TuBA+7+L3efAn4EbEg5pkjc/Q/Ac2nH0S13/7e7P1L783+BfcCKdKOKxoMTtYen136lkocKk9gBzOw2M6sAnwS+kXY8PboB+GXaQQyIFUBl1uND5CypFImZjQJvBybSjSQ6M1tgZnuAp4GH3D2VY8hVYjezh83s7w1+bQBw903uXgLuAz6fbrSNtTuG2nM2AVXCcWROJ8eQM9bg73J3x1cEZjYEbAO+NO8uPBfc/RV3fxvhbvudZpZKWSxXe566+9UdPvV+4OfAeB/D6Uq7YzCz64FrgKs8ox0gET6HvDgEzN6I8nzgyZRiGVi1uvQ24D53/0na8fTC3Y+Z2e+B9UDiHdq5arG3YmarZj28FtifVizdMrP1wC3Ate7+YtrxDJDdwCoze6OZLQI+Dvws5ZgGSq3jcSuwz92/nXY83TCz5fWRbGa2BLialPJQkUbFbAPeTBiRcRDY6O6H040qGjM7ACwGnq391a4cjuz5CHAXsBw4Buxx9/enG1V7ZvYB4DvAAuBed78t5ZAiMbMfAlcQlox9Chh3962pBhWBmb0b2Ak8RvgOA3zN3X+RXlTRmNmFwPcJ59BpwI/d/ZupxFKUxC4iIkFhSjEiIhIosYuIFIwSu4hIwSixi4gUjBK7iEjBKLGLiBSMEruISMH8D2zidkWvZoEHAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEICAYAAABPgw/pAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzsvXt8VNW99//eyZ5JhoQMuREIWbC5iFwUi4Db1nq8PQfb4oF6xPbX9ng5x+qxrcfWPsVa+uPEPPF4Wmhry9GHPlJPEa32KNbiActjtWpjq0OIYiAEgcAmG3Ihl0lCkknmkv38sWfIEMJFmT25sN6vF68wmcmstdde67u+67vW/n4Uy7KQSCQSycgnZagrIJFIJJLEIA26RCKRjBKkQZdIJJJRgjToEolEMkqQBl0ikUhGCdKgSyQSyShBGnTJqEBRlA2KojwS/f+1iqIcOcNnLUVRZpzmvTsVRXnnHMt8S1GUr3+yGkskiUcadMmIImpE/YqipA11XSSS4YY06JIRg6IoGnA1YAFLh7QyEskwRBp0yUjiduA9YANwx3l+1xcURTmoKEqzoihrFEUZdCwoivIZRVHKFUVpj/78zICPTFcUZXv0/c2KouRE/y5dUZRnFUVpURSlLfq3BedZZ4nkjEiDLhlJ3A78JvrvxvM0kDcDC4HLgWXAPw38QNQ4bwXWArnAz4CtiqLkDqjTPwGFQDj6WbAnHC8gon97LxA4j/pKJGdFGnTJiEBRlM8CU4AXLMuqAGqAr57HV/7YsqxWy7JqgZ8DXxnkM0uA/ZZlPWNZVtiyrOeBvcDfxX3mGcuydluW1QWsAr6kKEoqEMI25DMsy4pYllVhWVbHedRXIjkr0qBLRgp3AK9ZltUcff0c5xd2MeP+fxjbwx5IYfQ9Bnx20hm+xwXkAc8A/xf4raIodYqirFYUxXUe9ZVIzoo06JJhj6IoHuBLwDWKojQoitIAPABcpijKZZ/wa0Xc/ycDdYN8pg57VcCAzx49w/eEgGbLskKWZZVYljUH+AxwE3Z4RiJxDGnQJSOBLwIRYA7wqei/2UAZn9xIrlAUJVtRFAF8G/ivQT7zKjBTUZSvKoqiKory5WgdtsR95h8URZmjKMoY4H8BmyzLiiiKcp2iKJdGwy8d2IY+8gnrKpGcE9KgS0YCdwC/tiyr1rKshtg/4HHga4qiqJ/gOzcDFcBO7I3PpwZ+wLKsFmzP+n8CLcCDwE1xYR+wQysbgAYgHbg/+vsJwCZsY14NvA08+wnqKZGcM4oUuJBIJJLRgfTQJRKJZJQgDbpEIpGMEqRBl0gkklGCNOgSiUQySvgkpwM+MXl5eZamacksUiKRSEY8FRUVzZZl5Z/tc0k16JqmsWPHjmQWKZFIJCMeRVEGPrE8KDLkIpFIJKMEadAlEolklCANukQikYwSpEGXSCSSUYI06BKJRDJKkAZdIpFIRgnSoEvOC9OEsjL7p0QiGVqSeg5dch6YJhgGaBoIcbZPJwXThJISCIdBVaG4OHlVG4bN4TxDeNFnKnrge6P23oyAC5MGPYE4dr9jlrO9HQIBWLUKdD2BBXzMukQv0jAE4bB9vYZh/0tGPzdNWLHCbg6vF9asSfL4GoqBfQ6zp1PVOlPRA9+7+25Yv97BSX6ojOpQei8fg5Fp0E0TfD77/7qe/BtrGOB2QzB4omM5er8Nw7ZeNTX2z9JSWLcu+R0q/iJ7e7lk8jyW7Qqzq2kx6kQdTUvOePP5oKICMjLgwAH7ddKaYsCNrru7mJqgcN6+GAax2bOryuDwBh+5sw0KdM3x/hdXdP/EjUmjz+CNao32dsHcufbvy8sH+Wyi2iV+Jg+H4W/+BhYvBl13vt8N2ghi2DntI8+gmyY8+CDEUggsWABr1mAinG/YeE+5shLmzbNHzy23UNmsc+SIYPZs8PsT3JE1zfbMYy6px5M8dzieWKfOzoaXXiL7979naZ/FF/b9jOP/+hMC3DUSnJjzI64Net79AN83N1A27U46vMLZ69U0UFW6qgx2V/Rg7XgJv5pGxkKVzNXF+HyCo0dh1qzE979o0RiG/XO626Trvgc5/u4hru3tokf5PJ1HriB/is6iRYKdO/s/m9DUTbGZHOwCdu6E9es5Vvw4Je8udbbfDWwETcM0Ye0Kk+x2g1e8GvevEUPe30eeQTcM8PsJujIIBkE93ED3hs2srVpGU7pw1pDEBnNmpv1TUQi+8y4d5QfJ7J5Kl3sNW2oECxcmuCMLYYdZSkttY+71EnOHG30GBhoIEb9gcIZYp66uhs5OCIVIjURIDfaS9sh3eacpn6NHlzpiVOLRdVi40C4jOzvJ0SdNg54eePllUptauUo5zIzePfx69moMw8EBLQQUF3Ngs8H2pkaubtuCgUZ3fRXBDZt5s3wZBw4IDhywfZxE9r9o0SccpkKfj6533mFyayOphPg61XTsmkBf5tXkFa6msDixzlXMC57dAnkA3d1gWRAKEenuIbSqhInjgvTM09nVJpxZsQ1oBBPB735hcuOfHqRwjJ+eiJtjpUs4LnIZt1inUB8ayz7iDHqdW6OrJZv0QzW4U8LQ2EHQepubWnby1jXFvHvEoRsK/QatvR1UlZ79h4nUt9KujkGEK7hb30pleC7XXaMhEl0BXbfDLLGRAhy7r4RdFWHCisqvlLuZPS1IzwQHPYVYp/b5oKUF3n//xFuR7gBT161gQXo9b1d/gUlXisROagOq8Z3vwGuvwfjxzpRxxsKXL4dDh7BcY+huctPT4CerwEDTHB7EQpCzTFD7jkn70W1MClXhaarkwFH4u56dfGr+UnoP1DJ73iKESOwsJ0T/mGraCu7OHlxYWCj0AYTDtO88iLXVh7g3cf0vPpSU36Pz8JyFZDQeAr+fSG+IQMQNjcf47LGnqN+3jb9MLGbTJuFMJDbaCD4frFwJ03f6WOLfQSTFxSxqCT79HpFUN7XrL4cXnxgSo5768MMPJ62wJ5988uF77rnnE/+9acKKUi/rP9TZ2TuLtkyBJyOVlEvm0n2klW37plF1fAptbbb983oTWHmwv3D+fLj4Yli+nOp9qfRV76UnJQN3qJtxbYdY1P02881XcF0yC4qKEl/+lCng9dL4x0p2/bqCqi6NnC6TK46/wZTAR8xoKCM4Zz5FcxJ98XF1mDMHLruM3jIfir8VFACFtHAXV1h/5crIX5l0/SxC44tISUn8fTBNeOgh2LIF3n3XXjA4cr9Px9ixUF5On3mE3s4QDZ7p/LngVnLVDrxGJZljHbjoKF4vzNa9mHnz6W3v4dCeHip65pLf+hHXGRuY07WdWR9tJvXKKxLf/6LsNsbS/aaP3J6jYEVQUHApESwLaD9OxvWJuxmVlXaURdOgrtPLlFt1im6+guaLdDqq66gL5aL2hahwfYYxVjfjp2WgdrQwJiOFmQsTfw9iEd/du2Fy5x6ut94gLTVCltVOihVBVfrICDTTUngp+dfMSVi5JSUl9Q8//PCTZ/vciPLQo9EWunIEfwgJ9iomF6eUcJllkD5NRY1o3DTf2eV+vKvy4acKyXxlD7kpftLSMphJLWN7ukndFbKn8A0bHIt/GGjgUpmaYuAKB+gIeTC7NC5ur6Lgvc2gL3M0gG0W6qy9bAuLPM/y2QMbyKGVUCCEGuxhSt9erGdK+c3BdY7Eln0+e3HQ3W2/rq8f5H47uVslBKxZw/4NPt54A1pn6Ox/HVIeKeGAJ3wiru1k+7+8Q9BoLmN5aCczMwzyevykulLImCNwNZn27qRDsahCXbD22v9kjrGVXP8B0sMdFHXt40jefK7KSezgGxi6LtQFJoKSp69GnbMYy+fjhu5NZAT9qPSyoGoj6SlB1EezqZu/OuFesmHYUc/sbChv1/lQXcCMzAYKIseho4uIBRZQ2FZt98EkB9VHlEHXNLsha2ogKwu80wTjv11MRrZBulsjbb3A73dgM2YQTBO2fCiozVvNhB6DGwr3ML+xlNTePnC5IBJxdOOyUBes1YtJbzCob3Fzc/N6PpNWRWFLJZ53umHvNkePNxoGNKUL9iz9AR/8+Xq+XLmSycE9pIW7IDubFMXihq7NvMEyR2LLarTnhsOQmjrgfifjiJkQjL1T8OFhOHIEisJl5GaFqXNrdPoNMh2897GtnIKFgh8dKOZiy6Ag18/PXStIbTDp6lM5PnkRExwp3b6s+9cIDONeNA1S60yU0hKmePxkeBM7+AbG74WwH2QLh6FokeCtLoFxTEcvMPAcquKrDT9Fzcggta2G1td8CTfomhZdJc2G1gLB8RvWkD/TwO32E/jxL4j4u/B2HsNVW2n3wSSfDBhRBl0IWL164IlFAQgKgeLC5B0hMgxITwd9uWDvXoG2TMP1h1ft9aHLBRMmODqr9A8qgdsNmx4rZN6hzaRFupndWQ9Nzh5vjPecmsfo+PUNfEHZyvQdL5JdkM6EgwcI7XmbL6btZLq7GEhcHXQdrrwSGhpsY/7oowMu8TRHzBJN/JbCmxs1AtUq+SGDzGxnPYpY24fDMPMGwYKFgsWLob2xgDd+VE5N9iJqX9Epnu/cOIiPqSMErCt2bPCdVBYn970JE0CZKKhNE8wLNuJqhlAEUoH8s+r7fLK6nDzB2PYHwDN/Pp7Nm+HttzlxjjPJp9FGlEGHU2/uOb3nwPI71qn8fpg0CeYtEbDk8aSej4+/3sI1gjrfMi5atw234fzxxviO7XbD+vWC34XvJf/aJfxg7mbS3nubgvFzyes0yAgaJNKgRyMep7+lgxwxc4rYPdB1QZ2vGA2DTH2wSiW2zIFeK0BZmc4fZ9rPA4SN5NoSE4GBQCORd3pwBl4/2P+f7tbx/tsCehrbSS/wkrHEmdXpae2MELBsGc6d2zw7imVZSSts4cKFVtIl6Bxcfg+3hwoAe0KJP96YpCXfSW3BMHiqbljeHGcZqocZh9VDlMPhvjtQB0VRKizLWnjWz416g15WBk891b/8vusuuPrq5NYh2YzSTi05O0PR7BfiEEs252rQR1zI5WOTxOX3sOFMcakLqQ4XIEPR7BfiEBuujH4PHaS3KJE4jBxiziI99HiktyiROIocYsMDKXAhkUgkowRp0CUSiWSUIA26RCKRjBKkQZdIJJLzZLho614Ym6ISieSCYChO2wzRs3yDIg16gpDHtpKAbORhSTJvy2BlxRQpm5vtNCppac4/sRpf5osv2nXyemH69KERE4sx8gz6gDs6HMZ40h59Hg4XO1QMFMq+5x479eaAtnC0iYZD3xsufSBajzq3xoM/FyfUo1avdtaIDhxn0K9IGQjYv7/5ZmdTaMekTSsq7DIVxTbmsa45lA9WjSyDHr2jx+rDNPlVjt5VzAvviiHPIZGU5H7DKGGGaUKdz0SjX6TYceKFslta4L77+lUtom3haBMN+PKGpXfz3JNBGj3aiZzvsWo6Zmvj69DTYysn6Xpy9HRPU48eU+VITTHHxwlqapwV7I4fZynlPnp/Ws7xWYvw+3UyMuwkpx0dtuBJUZFzhjXWFePLzM21+9xNNzlT5rlyTgZdUZQHgK9j527fBfwjMBH4LZADvA/cZllW0KF62hgGx+rD/Nd7GoVBg+dXGmTMhWumGOxu15zVdByMqJcy3a3R2yt4771+pzHhnGXWSJbjFhPGvamihAOEcU1speOiRaQvW8yEpQ6Ke8YLZaelEeoO0taskNl0BE/UisQGWmam/TOhE2t8+1dV4fpRKZ9pzMPr6eG1scv53S903jsqnF3ux4t0b9kCbW10PfMS/915C5XpOkcU4WQK/FPq0aFmEzm8l6uPb6VencuBsIaTuRbdbjvEMbHWx/0Vt5NXFYZQmG+rN/KG/3LeTF/CpMsEX/+6s8lOYznRO6tN5vX4KCoC8TmdLR/aNmDHjqHzt85q0BVFmQTcD8yxLCugKMoLwP8HfAF4zLKs3yqK8kvgLmCdo7XVNJr8KoVBg7RMlZ6wm2+Wf4uJ7zcyJ62AtNuewPnknVHiQgDZ9a1c13MrZZlLCFkO9qLBEmaYJnteNVj9gkZXjnB8U8YwILvdwJsRpqlNxf3Xt8jaUUHvpmdpeGajc0ZdiBNC2d3dFp3le2H7dtpTXVhPbmQMkOnWqay0V2yWBXv2JHCCi2//QAA8Hlr6spl6aAuLaOO4sY3fphYjPiOwLIdWabE67N1rvy4qImXL69zYU82CcC6PZj7KAw/oPPaYw0Zd0+hq7aH7rS14gwH+JfghByPz6cmZyGyR2Nz3MUwT1q+3Nx4nG+WMGxvGyvSSWl3JDX2/5iqe44beV9mkPIGuO+vYCQE/e8Ak9N0VZB+swHMsTNOvCqmN3I7/yi/QlC6GLI5+riEXFfAoihICxgD1wPXAV6PvPw08jNMGXQiCDxXz0rcM9oc09PatLIq8RdhSESnV/O7HW8mbf29yGjLqDgYrq0k1DL5EJX+jvcqT+U84slIwEdR9rpixLQbHczUKEaT6THpWllBbFebz3SobJhfDbGc7k6bBb10a9c0qU9p3oWBLAqb7G+h95nmYX+hc4VGh7P2bDd7vruKzzZupUyZxReXrsGYNhZlTuW7GGlozBOXl8PLLthJbQia4uCTcDX43Vd9Zz8T2vUTC0Fo4m/Gqn6yjBn/5iyAz0/YmE068osZLL8GBA7iPN1PQ28qkSDXrur/IT4OPUFp6l1O6JifqcXjRciLvNjCu5zDjrGZEz34ypo4hN2hgmokPAcUWJ3PnQm3LIjobVNL2GWT29dGHizRCTFFqGdduJGWlXhg0IKsdsl2EzCbGdTZyO4d598hb/OG61c4Lhp+Gsxp0y7KOKoryE6AWCACvARVAm2VZ4ejHjgCTBvt7RVHuAe4BmDx58nlXOG++4NJvCio2gOhqgi4IKW7SlDAZXU34fEnaM9I0ulsDBA83MyYUQkElr/Z9ZhT5En4z+xcDgspKwbx5oL4EFx8zuGpvmL09GlqKQVargacVZjYaYGqONUBrhuA304v5dNtWbjmwhnR/A2NCfjJb9jsvuyVs5fsdr2nkH97BpMB+XKFWOD6GnGMVzJvm47/7BKpqy4QlcnMsJuJQVQe/6Svkiiwf1/S+hCfgp0NV6cjRWPopk7xOw25/J1Tf+xU1YMMGUquq8PT4SSFCgdVAqf9fWNuaj2EsdbT/j1usc+TxDYzr6yOc6iY9JQiBAHVuzZF9jNjipKoKKo/q+MVGlux/jL+NbEbp60Ohj04rg54CLTmbkrG4S0cHVm+QoOKmz+1hWt9B/j79VYQxB9CS7qafS8glG1gGTAXagBeBzw/y0UHTNlqW9STwJNjZFj9xTek3bEeOQDAIO/MX0xL4NWlWL+3uPP46djFHHzO5yGXQM0Hj/jUOztRC8NGtqxj3wTfx9B1HIQXVo3LrrZCf4DJj3klmZv/P+nqoiWjcmK0ijhuoY1QmTXXzUGYJ47eEYZszgVzDsI+FZV8u+O/qe5n4uflccfB5Mlv24/mbRUmT3TIRlFjFfJkNFHKYbNy41RBf+hKMz4U3N5rk7zVQvVpCJtj4/UjTtMs/Pk6wHZ3vLTdQZ2jM+R0s3VnCGFeYuS+poDs7sXHnnbBtG6k+H32ARSopSh+Xd/wJTVvqTLlRCnVB66OrqFtZSqpioboUJjy0ipqgcOSAQGxxsnmz/dpTpLPav4bcrgATQ4dJz83k+F0/5aHbkrSPFpPN2rqVwLpn6K6qZVywETWlh7nv/xjWLhiSQ+nnEnL5H8Ahy7KaABRF+R3wGWCcoihq1EsvAuqcq6ZNzLDNnm0fdqifpLN24vPcObecJm0RR7cWcvOuEjyuMGMOq9T5iu1onkMue94Snadf/d/cvH0lqUQQV0xgnAOyVzHvpL3d/tnZae+LhSzBi/nFZBUYLLpV4/t5hm3MHTxuo2nQ22vvyQG8ME7nyu8V4llfkrSE2IYBoRCEJgj+GLyTBa17mDfWT+60bPK+oLMck8+9VEKnP0ymopLJ+cd14/dEu7rs37lckD1bcPm37e8uf7EM1QrTlKGhJEMDTgj42c/gnntI2b0bgNQ0F5964HrGJ8GGXHKXTt0l62gqN8hfpDFBF4RM53Kjxyu8+f0w6UqB/9rHmZ5nn7aaeJq2duzAgBBw7714lyyho3QD7X96A8+U8aQbFf3e1zDUFK0FrlQUZQx2yOUGYAfwJrAc+6TLHcBmpyoZI17Hc+FCuOUW0HUdIXTKyuCi18rwuMIc6tOYGzKY3uKDkm2OHfUTAu54QqfOtwENg4zTHeE7zx4Vr6Hodturk34tRYGmRb0SE9szd9CwCmG3e1sbzJpl34uaoKBwMJFLh4itdg8cgIMINl2xmjm3GxBr/7IyMtPCZF6pJWxii98T9Xrhe9/rvw/RIunI0cgpUElvN+gIqGQkY+2v6/bs+uyzpBw4QMqyZYxf6qx3Hk+hLiiMCy0N1PtMdFc49fsFZ5qsk3LaVwjEqjuh77DtdR2Jel1eb9IPpZ9LDN2nKMom7KOJYeAD7BDKVuC3iqI8Ev3dU05WFM7cWTQNeiZojDmsMjdkMG+BSl4ujh8Qt0OaZ+hUCepRp8s3fdLvnB5NUXQdtm2zjfmJeSOJCbHjVrs0NcHixYICfUBnSLCbeLam1TTo8Np7CwUBg+mrBvmQUwgBP/hBcso6B5zuCh/n+5PyjEisUoN5XUmOoY8qxaJTHniBoX8YZ5QKLg71A4tnnSeHoIJD3SaSUxlGz+OdFxekYtGg3nISQwGDMkoFF5PokA/KWT2vIajgULeJ5FSStGgdNowqgz4oQz3KLrQelSRG6TwpcYChNgHJZPQb9OHAhdSjkoScJyWSU5EGXTJikfOkRHIyUrFIIpFIRgnSoEskEskoQRp0iUQiGSVIgy6RSCSjhJFj0IeLrLZk2CG7hsRxBulkw7HfjYxTLvGPe7W2wqJFsHhxEqRZJMOd0fIkoGQYM0gnMxHDst+NDA899ligqsJbb8Evfwm3324n+pcMC4bKW4l/YjSW3M5xhtI1M00aN5Xh22Sesfjh6D2eN8Ookw1JvzsHRoaHHnsscNcu+7UQ0NAAzz8PhQ4q5JyN4aACPwwYSi856U+MnuZik3LvTZPOB0s4sCNMGJWnZ9/N3bcHTxHqdvp+fJxrTVi7DGEnq3NrKM0qWV0GGV67k02vM5ndbNDYpaF6Rbwi5JDagJFh0GOPBW7daqfZa2iwU/3tjyrk3H03BIPUuTVqgiI5jRmnKUogQMM9qyh5RR+aJdgQJ6Kq85lMO2qQPkuj0p94CbwzXV6sa5xIyobG+eY+P1MdZjYaFAxIImMiWLHC7gpeLzzwgEPJ9gyDTr+db/2iYBU3b1+JuzYCuanw6KMnQpBOZhiMt6uTjvr4XG45wXmLmHmbfkoZCbXBp7soh/u+aULJekGWx86iedv3NAqBwvUl3OcJ02scJc2bTkbpRBpuumvobECUkWHQ4UQyeebPtz3z/fvtWHpVFZSW0uXJ40Clyu/nFdPhFc43ZkxivqYG2ttx/aiUrEnryJorkiXaYzNgYnFS9j02dtxuW7A3HIYZrT6+1VpKyhEP3Qe81C8oxu0WlJUlZowNNArRuRu3u3/l/Vm3D33dSohEYNMEe9JPoNccX4f8Ho1iRSUzbklQudVEecegL0vDFxCsXAk5OQ7cDk0jM1tl3F6D3o5WRGAfWT3dsL/HDkGuWYM5fymNjdDTYw+NQCCx+qYxu3pZj4+lb91OSl+YvpdUfvnmRu79z5ONemyIZGbaP89rTAy2FIu/Mb29cMst1Ak9oU5d7Hqz5gqqDUFNEAqNMgiHyVB7yfjgTbvfvQHZm15h8qc307dIp6rKVldatiy5Rn3kGPQYug6FhXQ+WELnewZZoQBjcjw0Z2qk91ZxQ9dm3mCZ80KxmgaBAKHWdnoVD2pPJzNafWw3RHKTRcVPLC0t9HzzAarvfoy8Jad6TOdD/NhpbrbV1z9dZHLd66WkWdVclumhy5PLxMt8rF4vEualxDtm0bkbjwfef98eRwKTmYGV5HeVk5rmgsOHwedL6KbVyc6hoOqmYvQC48RNnv1iCcs7wozp7OV3KbfQ2KpT4xe0t9v1TZhgsxB0fKeYt44ZZNdX8bVDj0A4ZFvvw4cJ3/stnl5YwIEcne5uW1kpJ8eefBMVmYzZVXdlOSl9YZrSBBPCJoVHy2neCiJcbjtauo7bDZWV/ffgvCaWwZL3lNmGlexs2LKFnkP11Ndu4EWxiqZpOqtXn/81Dx7Si/5y+3bo6zvxWVdHC9ceeJLnKKTygF3wzp3J9dRHnkHH1nNcaxWTrRj0Zbj5jrqevGNVuP2VWFXwxfSdTHefv+zYGRGChntW0XrvSvI6DtLX0suXrtzEgpt0CvUk6RrCiYmFlhYiHZ0cb66DR0pZ+9a6hGqqDpRgCwSgZ69BWPWgpnlw19fizmoj+PomLrHcXJReS4WyCMM4v4klfkAFArYxz8y0bZjHAzMw6AtFiKS4SO3rs7XpSGzYYeCgLtQFJgKfD/Kqy5iTHiZ9YjaXmlsoGlNPp7mBNWmrMAp0PJ7ErtZqggJDCLLmadRu+gNZ/rdJVRTIyqInlELRntfwXh5ke4dGMF0wd64z2p77nlmE61GV8b0mIVRSMj1csuZ2IGq9N24kGNSZN8++X52d9srqvAuPu4hYbHucUY0nFKKnrpncJpNbOkv5/xvX4fOdf/8fPAlc9JfPPgsPP3ziwlKwuML1AeLwN8ietIq0uXpyV+uMUINuGNCULsiYZYc3qm4qRG/YjAJMGD+XvE6DjKCBowYd2J+ts3PmN/jckac46J7FZSl+23NL5hpLCHtd/8ADBHvq6FJycGd5mNzgo3WzgVimJaQ+g0mwYWrMfsmLuz4Putrgc59j7MF6bvPdRyjFxeUpKop/I/DJYw4DhWDWr7cXJOnptsE+gEZn5gQU6zBYIViwAHT9tJtW51uH2Pc8+CDs2AETQhoPp6hcVVBNWkeY8VnNRLpMvh8q5dcT1xE+z7LjMU1OhFMq/YL6qx5n5ZRncW9cSyjVTadfpbCrnOPHarhtnMrmTxVjOLBiFAK4TcfXthGlohxrwSK+PLYc14Zwf1y7vBxtmY7Xa9+nRKuxxce2ZwR8/OOkdaTWGHT9xEJfAAAgAElEQVSmeOlN8VAUNkjU+B80CVxMJWr8eHjkEfsiw2HSCVLUWc2X20vZmLvuvPvex2VEGvTBPCZYxpidO5kSNsCbnJiHpsErE3Tqjm4jM+QnM3uIEnPrOjz2GNbKUppqPEQCKlfUbmLG2+mwMzG7M4N6Krqwle19Pti0CcJhxvT4cWWn0JUnyGgzcdWWcz4GPVZ2rPqFhf3G3Y6hC6aJNbjM6BHWaMA6tmnVEVCxvldMYQKuP/YVZWX2nnxGBhxHsGFcMRNv8DHnrXV25cZ7uXSih2983iBnWWJWSfEhL0WBm24CXRdkix/A31/P4efLKXtPZW7HexwLaiwoMvjX2w32FST+kIBpwooVUFGhAzoLx8NVX8EejGZUJXrRIkdTHMfHtrcbgk9fKbj4xVI693mIWF5yLteS85jKXXfZz8T4fHZszTBw5XiZOdHDHdck7v6fKyPSoJ9xGZTE0x5CwP1rBHW+YjQMMk8nEp0MdJ0xG9Yx2WfQUt3IlMotZMzVErrePq2nIoRtSA0D/H5cK1Yw7rgJafbATiTxdegfsMKeXGJEY6sZczUyDAMSvFrTNDtsW1Njvw7NEoy9U8BicSLIP8br5bJlWsKKHRhCKiiIuxe6Tlqhzp4VJtMqdjBVMcibqJKpaxTEPpPA0yCxbZuMDPu132+vVids3Ajl/TF0cC7F8UCnLm+Jzpgl65jqM1DQeCiZYc/YRQrn7v+5Mqo0RSVRhvrxSZ/vlIGdVJJw/abZ/1ybrsd9vUPH6M7lkk7R1I19IMHt0e+h268XLiQhG5CfpB7D7rkPhyp1rpqi0qCPVoZlb08io/D6P/ElOSBUftoJTeIIF6RItCSOC13OZxRe/ye+JAcepx2FzTsqkAZdIhntOLk7KRlWSIMukVwISJf6gmBkZFuUSCQSyVmRBl0ikUhGCdKgSyQSyShBGnSJRCIZJUiDLjk7o1L+RiIZfchTLpIzM9RPnUokZ2EUPkP2iZEe+sfhDJ7qYG+NCsd2uIonDhVDrCc65B1qQB2Gukoxf2PtWvjGN4aPzPBQtcuo8NCTped4Ok91sLfAQcf2DBec8LY4nVLMGZ77dvR+RL88qXKD8WUP1WolvuyeHli+PPnP3MfXobWVtpmLeHrfYg7k6CepSSXznsQShVVX20nCVq6EDRuG1lMfym5yTgZdUZRxwK+ASwAL+CfgI+C/AA0wgC9ZluV3pJYDidNCazSDrN2kYUSEswpsp1FMME1baqq9nZPEBMAhXcczDGxHOtKZkoGDnX88KvkWXz1HFPGiX36sPsyuXSovX1rM2LHwveXGKULJjjBQPmnzZli0iEYziIHmrLBJrOyoOg9tbfDSS3DLLckz7LE6qCqRN98i9c8V3Bl5lle/vJE/tuuUlkJeXhKNmGkys9EgtU6jttbO+37woO1rOFV2zPRMd5sUmoM7NU5qup6Nc/XQfwFssyxruaIobmAMsBJ4w7KsHymK8hDwEPB9h+rZT1T5vLu+nWyzEqVoHjfWevlpRjH7onqO3/iGA318EE+1zmfyXKlBW7cb1/4gDd0a6sT+hPaOqNEPNrC3bYNiW8wgYRqO8ZwuGThAezuNPoN9hjjRgeOkVhMrwWYYdLWHeX2/RprfIHOPjxvZhtUWhm1JsCKxPlBVZWurdXcTeuw/aAjOwG0pPH35Ku54IrHSf6eUvXev/bqoCN5+Gw4dgpQUeOghWLrUgYJPrUPvjl1EeqApo4AxgUbGvvsagam2OlOsD/h8Dq+ao5N7QThMabCXseot1OTqHAoLqqvttxNdbsxZyWo3+XLFg+RZO3CrnOLUOJA655w5q0FXFCUL+BvgTgDLsoJAUFGUZcC10Y89DbxFEgx6o8/gwI4wvaFMZh0LE9YyIRRmapOPrFSDPRUaP/mJ4JJLEjy+B/FUldISrt3VzrTOSvZ75uH1e8n+134xBUfSZ0R7S+CDvVgB6MsuIvPIEfD5cAuROA3HM5Uflwy8y+Vl7f9xk960iYwMuO77OoGAPbF4vSRWgk3T6AioTAwatLlVurshrITpULNJr6rGV+pjuxAsXuzQKi3WBzZvtl9nZhLeUcXErj2kpQS5uXwldb4NgEj8fY+V7fPZnvmRI/YK7aOP7Bt+771QVwdLljg3qUXrcPgXW8ne/+9M7D5AxIL54XIuvdnH1peDNFZp9KqCTZtsVSnHvPXo5N4RUJlc+zrfTq3H6NzGIynFvP66YM+exKf0jflSl2QaeHr9BD0ZuDOwPZe4GUwIMWSpc87FQ58GNAG/VhTlMqAC+DZQYFlWPYBlWfWKoowf7I8VRbkHuAdg8uTJ511hA41AWMVqaycQVDF2dqJNT+Xmgy/R0JzGYkvlsepi6nNF4pc6AzzVLE+YUFomkdYwfbmZaJPCJ0nfOZI+Qwjq7i7m9yt9XKk8Q+HWt0nLBdemTbivFXxpYpDu8Rq1ljh/DcfTlM/q1Sdi6O/tE1z56M+5NLgDy4KUHy+g9PtrWPWkwONJsPSYEDTcU8y2Hxk0pGv09oLHeInMt7bQ2gPt2zfxnEdnwwbBc885aNSXLbPVf9vbcRFkbE8H4RQ3+X01uPb7KNmWOJHsU8qOiYn4fPSW/DspLW2keDykdnTAr39th8KcXKkIgefb9/LyW7Bo16/Zl3Epk7vauey5Uu7LyaMjoPL+TcW8+J5wNORQ59YwK3qY0fAOkVAHU0UzSno+otVgf7ugri7xoZeY5727XWNuWjbuSA10QXfYRee6l8jISSPDa990IZKrVBTjXAy6ClwO/ItlWT5FUX6BHV45JyzLehJ4Eux86J+olnEU6oJV04rpqjLw97kZpwa5PNLI3xZs4cMejaKwwaSQgd/vsJafppHhVbn8onasHpX5F3WSEW+9HNwZrAkKtgvbYEbee4qOubOYGTnCxS+W8rmjeYSPqmxZUIymOeipRa8ptbSMrD4/PakZ9EUg19/A/NrNPLlqWcI3LU0TnnhF0D7J3i+56UtQ+dtbyN7fxl/Csxgb8TPTbeDrEjz/fOKU7k8hbrWmzp8P//G/6UvLIlsJ0Rzuj592VJkJ1XWNL99E8PIEN189eB9qV5CxrhQCUy+l80gYxWdQ4KA1EQImfX0JDY/sYGpWGHdHgM6Ih/FRhaiFeQYvq8LRkENNULBz8nJygg2obc3kdQfIzg7QkO5AYVH6b7tAuFfjNn00NcOLL8Klxhb8fo0bphu2StYQ7cqei0E/AhyxLCt2IGgTtkFvVBRlYtQ7nwgcc6qS8QgB33hU8MADgqNHITcX8iaYZLZvY5ppEFRV2rI0fviQw20avbtjYgKX8dv7Dm9zaxr09sJ/7ta5L7iNgt1+iqYHGJPj4ZJ5Gr3vV3HZpA2M880GnN0wm7lY49D6bDKba3C5wuR0dcDbb1O4cyeFCb7u2JI3tvmclwe+CTrHDm8jL9VPV1hlb49Gdxj277dvgWPOamxS0zTUykrUaIxp3GIddb1tzL9YWcIMwgnTdY3HMOD9SUtJ/0oBuTte40pXOUZFmDAqWzZp3O/wPum8JYK1bxWT3W7Q53LznYz1J4LGBbpGse5syEHT4LdZOuM7tjHWyqcgJUDaN1cx6Y/94T4nVmj9vowAXbC3DA68YaL7t0G7QUdAJSN2EmwIYi5nNeiWZTUoimIqinKxZVkfATcAe6L/7gB+FP252dGaxhHVRI7J9xH2Crz/WsysnQbbj2msXSySo3x2upiKw9vcQtiHG55qE+ydVEzNUQPvMjfzd6zHe6QKzAqo3QGvqKds2CSaQl3Ai6tpe81HYXs1rsOVJx/3SfBEFr/ZpOu2UHKdr5jCFoOKFo3P1ggaGmz1O6dPGNhjVjD9gTUUBg3QNAqFoLgQWjcbzCCccF3XGLG2+GtYR12ok77Q5MPNBumzNJr8DoQbByCEradrRDfDMyk8yYAJnC//utsFv28vRi8weN3SuHmmYM3i5NpRTYMOr+A304spCBhMX6XZbwzRucVzPeXyL8BvoidcDgL/iP1Q0guKotwF1AK3OlPFwdH1EyLbaBoUCkGhLpiTzEqcjiRsc+u6fbjlUFigThLkLQGWFNobdk1N9ukXSPBxl8Ep1O22P7Eycei6T6fTIIQA7Hs/02dP9FVVCY7fD+DkRZiguFjE1Qc7zLLTuT5w6h694OUdgrA/eScrTvZnnNgwOjP2GBBUhMWJa3Zk3+oMxIdhNE1QKDghUj4U5xalpqhTJGHJNWgRpnnGc+KOM4TPYTt6Bn4A5yTTmeS2uBAfgR+W1+xAyFVqig41SXAVBi1iwCmUpD9NmGwXKY6BMXZHTvlEOadFWJLbYgibfsgYltd8uqVkEpAGfTQyLHu58yTzgY4hHLOSkcAQjUFp0CWjhmQb2Qt03pQMY6RBl4wqpJGVXMjI9LkSiUQySpAGXSKRSEYJ0qBLJBLJKEEadIlEIhklSIM+3Blqja/hgmwHieSsyFMuCcKRJ9akQLNNMh8BlUhGMNJDTwAxe/PUU/bPhDmRUqDZJl4GqbraTtYyhJ76kC0WLvRVyoV+/efAyPTQzyJSnGwcS654pkcffT4oL7fTCibbW012Ag1Nsz3zeBkkxzXOBsc0Ye0Kk+x2g1e8GvevscseSpFypzltzqBktv85XH/SqhRXkOmEOtV5MPIMumnCffdBRQW4XLYxS2byqYF1MQymuzVUJxL6n+7RR58PvvIVW4IsPR2efz55Rn2QgeV4pxbCDrPE8iWnptoybGlpoKrU3V2ccDGNwTBNeGODyY2+EvK8YdoPqFRuLbazHDptZwd4DTEd1+x9PtJ3lZN5/SImLE18HxjUjjIEIbCo5FxzpkZe+6kiEkmb7+IKOuZXWdVZTFeOwOsdHhHRkWfQfT7bmHd12eK4DQ3JldWOEXdjC3t7+fdrbuFgnp545fe4Rx9jjsGlm19j7LFmwu4xqO3HSH3ySQfleQYQZ1i6qgzKNxhs/DNkdxj0FGg89IRD0lvx+ZIbG22B7Ggdnik1qM4Tjg7k2O2eeMAgqyVMq0djaqiKrPLNZHUsI2uucDZTatxqrbNX5SebNNRjPv657Ct4U7rp+2UK79z/S6b8y9KElj/o6hPDQSXwwalzaxyoVFHCBodVlRlujUI4MSjqGjXCYWdl74ATDdKerVH5usGYkI+8MT4mTIA6nx5N5Tx0jDyDDrZnnpJCpDdEx/FUArGbm0xiPT07G7ZsIb+tjfxJ20AvBhJ/U+M9kM/sHs/XeoHeHlIiQUIf7CHyrRUcveg6si/KJX+Jg2GoqGE5Vm7wwS6VF6rdfH7vg0xI89NWlU3l1tWIex0qOza5maadDN6wFWIOWRqKAvX1djr4ZcsSf/mxML53vMYYr8p0dxVabyWhBvhy9Ttsb1pO1wQdTRPOLP3jVmtVjRpNWwR/599AXuQYrr4ISl+Y/F+WsLZ2PvevSdykOnjUTzttCKzOrTmyWqoJCn4/r5hLMg12d2rcHBQUxg2Kub0q+VYxVVW2PKEjAukAmkZnr0rdXw3USC9fCmxkSms1noYQ3p9PB/HokG7YjziDXid0UqfppPkbOFqfyksTHqV2va0Sk8yU33WNGlP9KqnV1WSFwTVrFvj9jrkG8Z7Snw99gUuzX2W6tZ+e7uO0Z88l809vkPv6O4RUD12vLiDjCYfCUMIWqf7FAwZ/UTQuPebjU5EdhMIZTIrUUPGWD3OJM156v6EUiKhxa/BrvLlC0FtlNz/Y+s3/627zhIpQIirjdkNlJbwfFuxwFfP0/9iM6zC4ioq4fM8WZgTaSFO20VFXzL+ud1YkutAEdRs0K+NRLAvFitBHCu4xbrLbDVvzMkFlDh71GxACU1XYuJFAUwdth1N5c/6j1E7UE3rtMWWgP4cFaky4JG5QZBoG//hpgwf/29baXb/emUWriWCtVUy6x8BPI3ekPYXS5yI3pQXXob3wP/8n3HgjLF48JIZ9RBl004SS9YKsnDX0tRkcnaJRNNNWaUlW1KX/BJ2g7oNirs/0cWNkE/OP+G3Fb4dytsZ7SpFCwbPex5nV4eOK2k1MbjlKal+ISEYWvRE3rsZ2x4RqTRM2lwt2Zwu6e8AyfaSkgKLY7xsG/F8HtDxPjZEKxNWCzjKYN8+OwO3ZA+PHQ1a7iVJaAnmJs6rBoF1OZiZ0dgqqZiwjbe9Oxn2wF48KuZ+ZDX4/NeWG40v/mJGt832B0OpNpO79kOPhNBrTp+D3agnvgoMmPIsPgVVVEfq3H0NrB1N6Q9xe+V3ebLmN4xvy4M7ErBZPVWgCX6PG3F6VzOjy4XiuRl6es0JBhgFGRJA5VdCXYpLVvomiug7UrrC9p7NzJxw8CM8+Cxs3Jt2ojyiDHpuQrSLB5h2CUAg8W2xRnmRIbsXXQVFgV5ug0S14K0WndJ7BtXdqjs0qg0mO2erjOsf/5KP1588wuWsPaSkh0guc0V6LPw5+4ADMmAFtHp3DNQtID7TTnOKl+xL9xAlLx2O5wn7t9dqfSUuDzk6YFTDI8iT22FGsnNiE8sxbgjc8xcwI+Lhjzktk+G3tt/xFGupO53Oy20ZWgP6f4PNxvBmUPJ37E72Hc/ZK0LynEaU1hDvYR6QvBW/jPpa1PkJmowf2LLQFVxJk1GMRN3tyF2R0FXPrHIOZizUKCwXqNmfbPrZSi0kPdq9Zg1q3FV58EVpaoLkZJk+2ZSDLy6VBPxMxL7W62v55ww1w5AgsX568cEusDseO2a/dbmhQBM2zhROh85MY6CnZnVuwukagLtSZ3OBj+XKY8A/OxNDjFYEArrkGJkwQbPmvNUxVDF7YrpFxRFBUlPjBdLoTnPETndtte9LT3RoZ6xOrdBFfTmxPNksTbDcEC27S0QuMk0Sik3aULdop8oF8h4s6HTW5Oq6cBUxrr6AvGMbtBndBFu4MtyNhyJO2r/4iOBgSTDps3x+n8+GfvFKDzmwBS++FJUvg1Vfhxz+2jbmq2keKk8yIMuixQeXzwaZN9k0tKkruJBhfh40b7RucnT10+yCxzl20SLDPEDR+FmY6ux+KYdje6rJl9u9jQr2TrrQnVyceDTiTeMWpIQEBhYkf2YPsyaKqtkh2fBkXWk72Ql2w9qrHmdzgwxtqZvmYV3Eb1dAVsgdHgmf3eMcOILZ95fNBQYGzE2n8Su0kEXIh4J//GT71qaF7PoQRLBI9HMRhh0sdkvm8yWDXPBzaIdlciNd8Jk5qD5x/8C/2bOGmTfajGD09dhg0+miCo+NgKO79uYpEj1iDLulHGhfJhUqs78c9moBhwF13wdVXD3HlEsi5GvQRFXKRDM6FtsSXSGKcLgyWrEMSww1p0CUSyYjnTHssFxLSoEskklGBXKnK9LkSiUQyapAGXSKRSEYJ0qBLJBLJKEEadIlEIhklSIMukZwNKX0mGSHIUy6SEUdSH6SSQt2SEcTIMujDTEv0QidmWGNJsZJhYOt8Js+VGjR6NDq84iT76oihd0wwVnK+xN9vkGfQYSQZdNOEFSts+TmAhYlLy3k+dWr0GRhoiZeeO32RQz6nmSZUbjUpf9Gg13JzeH+QrHka4YnCWQfWtPOcf6Y6jNfTw9t5y0/IfjnmSJ9JqPv01UyKaHSy+148dT6TpnKD/EV2+ckmXqz7ty6N1gyRlDwuw51zNuiKoqQCO4CjlmXdpCjKVOC3QA7wPnCbZVlBZ6pJvwZYRob9OpZeLWrd6oSeFKHgE5gmzf/0IOaHfkJp2ay9anVCpb9OUyQPPgixdDgLFsD3v2qSUpucgRWbTN7caHLT+yUsadrDrL5q9qbMpnv/HDaPKU6oWs4pGHae84Anmxm1W/jb5nomr9sAYhVGUD/hSFdVJVCK7myPIA4wrNA/sfT2wi23ODDxmiadD5aw970w3SGVjZcXO6flOggfvGJi3l2CFQrTmKnS+HgxndlJHHvYE8pNFSV4M8LUN6v8Znox2nUO67rGMxw8q0H4OB76t4FqICv6+sfAY5Zl/VZRlF8CdwHrEly/fmJ5Kw8csF+npdn5aysrCXd2EwoV8Pq0f+Oji5eyxiH1tXiaX/XR+9cdeIIZ5Co1jNvnw+cTjnpmhmHPY7E5LWKYHLuvhNxgPW29fo6vfIiLVyxNfMH0h5KPHoVxuwymhA8yI7iDVMJ8inJqjwWY0epD0xxseE0jw6tydd5eaA6R623BbRyB0lIy71lHc7Ogqam/i+zcmSBv7TSPINb5THpWlnBoX5igZRvW624X8VKztLXZOUYS6jUaBs31YT5s1xB9BvXvGrz6quCf/zlB338GTBPW/9BAbwpTq2hoHQa//6FB9wJnRboHMrbF4HggzD6XxnjVQEQM3ntPOJGt9xTqfCbqygcZd3AHbhXbs0qG0TkHzumUi6IoRcAS4FfR1wpwPbAp+pGngS86UcETCGE32r//u/3vttugowNaW0lpbaawYw/f3HUvoXd8JyZOJzl2DCygr8/2xvbvh5fXmpQ9WsbaFaYjByI0zTYUXV32v0khg3GBei5q+itT296n4If/xLGnXkl8wfSHkmfNgiOqhqfzGCn0oSgppBJmcrCGf+xZh6hzsPGj3rLnvrvwzJ2OOxwAr5cuy8NrTxp4PPaEN2OGLcIRU05yAtOEZ0oNPtodZvsxjXGth/jCn77LmNdfOSVXd8LroWl0h1UKAgYd3SrVAY0XXrANjdOncQwDaiIaYVQ0DEKWSk1EY162ybSjZXYdHMbng9UvaIQVlZwOgxmih3yrkfweE6eTx8bue8NePw3tGQRdGXbkwDCGxWmoc/XQfw48CIyNvs4F2izLCkdfHwEmDfaHiqLcA9wDMHny5E9eUzjZUzJNW9OwtxcshQguFPq4qvs18qqDYGqOzpjjFutUP74AQu20RLzUpQrurChhxpQwgZBKna/YlghLIELY2waxCWu6W8PztQY8fcdJwcId8tP9bz+k7pL5CQ8/xULJfj9MulLQNO4BxNP30xeOoETAGjcOz5ED8MAD8NhjziX3j/UBIeheWUpnxENnj5fGHI1PF5n4mwy62ty4fUHyvZpjKwbDgEaPRkqaih4qY0GwjCAu3BtfY+F/PMOfvUvZtMlur0Rn/zMRPJFTzBGPwZ5ujZRJgovSTVJXrgB3u72STZDHOHA/QNMgbYZgtVGMiBi0Z2tcOgkWbClBJczUZ1R8FDsW1zdNW5u62hA0eov5rNtHduNLzA9u4Zqcbfwm39mwX+y+K9nZuI/XEO4At9drnwyI6TMGAraI9nAUiVYU5SbgmGVZFYqiXBv79SAfHXRutCzrSeBJsPOhf8J6nooQ8OijcN99sHs3VigVS3VzdXo5cyproMTZ9V+kUPDcZWs4tt1gX1BjwVgDtz/MvqDGVMVAw8AJTbqT5zTBG5OWo320m1RCgIK7tZHfr/SxXSR2CXyqpuld/Kwqnyk7NzMr+CET25rITeskVamzR9y6dY5OqGahztr8dWS3Gxx1adBrG5UxoXYuCVXQmz8Zl3siGazGifugaVCXKni4r5jvpXyX3oiL7vR8PEoL43f9ieU/X4quO7M5ahjQlSPI/3tBYBvMmAjzenzkHKwAj8teuW7dCvfee17lnG6j+YknYOtWQVOTYPFi0MwyrF+FCRdp7H/b4O2nDA5uszfIY/VNVBsYBng89py165jA1WWwIC2N/SGNSzAoKDSY7gbKElhoHJoGHV7Br2evZkaBj1tvhTFL9P49vpoa+2cSxsBgnIuHfhWwVFGULwDp2DH0nwPjFEVRo156EVDnXDVPg67D735Hytat9O5vol3JZ87h98iYqzl+xMwwbKNesFzwwTYITYJsl8r0IoO8iSqZuuZIuQPrUHnpP3Ct/3dMbKpESUujL30MkYgzp+ziJ5OyMngzcyn105dySZePHzQ9wFh3HZ6cHHvEObwzZRjQlC7ImCXoMuDO6WVcHAwzVlXw/LkZjycF9hy1lzMO1EMIW27vV22Co7NvQ3nzNbKtFlLVFLj++hOfcaIJYqulcNgWcbjlFri2BVw/CkNDi/3Giy/aOpfnUYHTndgUYsBcUajBNpWGIwZhVNJnaYSjZxa2bUvsyaPYVtrEifZ+Tn2aRiCkMstjML5Q5Y573ExYHzcL3X13Qs/U9js2Ak0T5Md/ZSBgG3OvNyljYDDOatAty/oB8AOAqIf+PcuyvqYoyovAcuyTLncAmx2s5+mJ9q5xwDjThJIdSclyHz+oFiyAeQsFOZcVo2UbSTsMG/MWNi58nOUfrGTa5Ah9BROp9eg0Gc42QfwetQ+d5xY9xg/7SiHHM0Bs0bny408TzlysMeGwCgfi1LtDIUfroOu2wXo/vBSue4Y7J/8J97LrYakzG9MxBj14Y+rwwjTYu9feaMnJOW+Dcs4nNqMVUnwGWzZpNPnt1SEk/gh/7No3R61Nfb3gpy3FXDXJ4NuPaUwIGpx03Km0FPLyEnqecdCJWgg7zFJa2r+EGAKVjY8lQRdn0G9SFGUa/ccWPwD+wbKs3jP9fVIk6JL4GOFAXcOhOAMbu9zpbpPCoAGahomzp23iyz7p5BbJ1cI75VbHKrRxo23MExhLPuc6DCU+38kGJQGd8eNe38CHfZx6yDYWDjolZB0fJ2puttti7tzk6dI51CGkpmiSKCuDp54avVqGI5JhZWWTzDC7dierc9rvjn+Eef36UZG2QWqKJolP8CChxGkuZOmaYXbtTlbntN8d/0Zh4bCa4JxGGvTzRGoZSiTDmGE2wTmNNOgJ4ALrMxKJZJgi86FLJBLJKEEadIlEIhklSIMukUgkowRp0CWDMwwSDUkkko+H3BQ9F4bZ2V7HkbJrEsmIRBr0s3EhGrfBknjEfn+hTGoSyQhEhlzORpxx6z5Uz5Hv/pSGV5KQcH0oGfi0VCw16Nq18I1vkJSE88MRGYY6iaQ3h2z/szLyPPRkhz+ixq27rByr7B3crt1EXvsDDc9sZMLS5Oc7Tsr1D3xaapikBtui+/UAACAASURBVB1ShstKbZiE/5LeHPEFOqbt9zHw+aC8nIbJi9ifrQ/17TjByDLoQzGoosat9bs/xe3aTXf+VDKaDML/50kImic6VbKEgVmxoj9FZ1ziqYSXP/BpqXNIDZpMWxNfVl0dlJfDokUOagoMCEM1+gz2GcnV0TwpI1VrK9x663mnyP2knC61ruMFOqrtd474fHD77YR6w1h+lTc/u5HaifqwiMaOLIMevant2Rq971ehbthMzuJFCc13PChCoN72FSKv/YGMJoOMYAsp5a8RqHiD3qLpVPz9o6yv1J3PuOjzQUWFLSp64IAtYjB3LnVujZL1wrl57kypQU2Tpq0+9u+H3+zX6cpxXlsyfl5vbYXdu0FR7GvfuNEhox4XhursVfnJJo1IxKQgYHDbKucFuoETK6VgZTUcriW08yN49S0ynliTtAybMc6Wwyjhk3uswHhtP79/SHKOU14Ovb30usahhttYYJVzMKwPSVUGMrIMuqbR2auy7+UqilorqT/YzdgN/4Hr8nkJSxd6OiYs1Wl4ZiO9zzyPu/o9Oo90MLa7gZ7mvXgOlnI0ax0LbxbJ62OhkC1i8N57KM0qk627KRofZHe75owEl67bYZb4UWqadH1rBdaffMwIh7g5dQF/+crjHAoLR9sgFgHKzITaWnsFftFFthEpL3fIoMeFoaoaNSL/BV+rKaGjNUznN3tpuvsW8pc4HALQNLpbAwQO+0kJqTR1ZHHc1475c4M/7rc1Zl/xaty/xjkJthhnymHkyEI6VmAsX3W8tl98dkWnnTuAyZPB78cTbkEJpbC/Z/KJraaysqGNho0sgy4EVbcUs+/gZsaMgY5gJqHAXlyZmfYI37ABZs92LLY2YakO8wtpvHMFaTXvkNoXpk3JJuL2kN9lcORdWJhnMN2t4YTsGboOCxfanTk31xYx0DSymqpYWlnKcXceM1SV6e5iZ8ofGIYxDIK1DXiCHShKH5f0+DBe30DnlXc6ouUZG7d+P1RW2gYjHIbUVPu9IsvkepfhnJ5s9PoLTSjYUEZHa5iD/mymN27haEkbY/7wEhmfv8YWVEhwH7SvXRC8YRU5O1fy/9o7//Coqjv/v04yGTKZkGFiQkjw4NUASkFcQBzb2sUfu20XKLGVbrddf3QfxNrar9Vn1bY8D5vN5qnbyq5W1/3yrYqlat12C22hsctaf5b+GgNCgyEoCV64ISRAfkxIMslkMvf7x52RSUgA9d6ZZHJez8OTzA9yPveccz/ncz7n3PMu6m7CLQYxun1s+x83n26u4uILndOzPW3DaYc16hlGhkHHNp2CkEbBfGlvOiZRYFzbr8WtYQRh/tYqsrpDmPvq4PKFeEudC+4MAzqO+Jm75Bo8uSZZ/YLrPufno0thyyMG/lDqBtXRmFgOHSgLSLZcXMGs3XvJi4bIyXXB8eOWUkttLeTkWBJCTgkbSEnLNzawdd8LfLr7ZxyPFjJwykeswM2aI1Vc7Q1xQbVDIrFSWirRyWc96zpeEWbuQg+6VyPvuE62oUMqUgCaRq43GzM2SCSWRT69fLTjBW5qDuJvWQ/SvusfqVswezZMnw49PXDDDZB7wmDlbx+g6Bed8JrfqieH7igp4Zb1Gq1fc1He1kAsBm/3zqT8Dy/BG69ZI8ySJfD447bYMHw9MIBn8WY8+4JEIlDvDVBxiY6r2dKzvTRaT3nDNjAqbF3bOa+oO/6l2aEoN9a5+CWVuHzS/iOlpcRAUlUFlxzdSf+BKCb5XNoe5a26fK4gynQHpoiJOigIadx4tJQlC6N4y1xcUaERDBqs3F2Fzxsl1OjcoHouJpxDlxLu3iBpCVbibdd5y3Rz0YlaCnt7rYUSsKJ1B+f8PX7J28vuJCxW0FtviRR/cq5O8Rsh3EYTRDrga1+DO+6A5cvttWO0s57dbmLff5Ljf9QJR1387gdu7mYnJQHN2bmflHgefpDW29dxSm+nyGzHl9XDlHdPwLp11hZHmyLV5EW43l5rjdY0rUzbihUgg0F4Zpe1vtDU5JiWaIKygGTP2kp+/c9BVvZvYUakmeysfhjstxL6u3fbZsPIBcjlX5GApL0dTr0KbUNwZZGLq4rqmXGijpw6oGovLWsrbVtbOa9F0PiXvPM1lvTVUzZzG3lfrKDMgXZI2JN7mUbfPheegRADUReDHT3s2efjcrdGmUNlFsyX/JJKfMt0rqjQQEq04E4aiaKjUYxzIvHnYsI5dEh0JElVldVZi/vL+Gff7/EePWp9wWE9v4SeZntU0n+5JF/Au1H4y1iY3L4O6OmywsjqanjtNeeixSTn/vqyMp7bqZPtcfOZ4JOEB6OwIwU7gQIBYk9t5jfrgizbv5GiXh1XvgcOHYKnnrJtJ0LyIpzPB/fdNyJdmoat8QtXSJ59TfJ8a4BFg0ECA4/AO/usWWJOjm3ljFyATB4jly+3UjHl7krKarfB67wnuXaiVicalbbsRBltEfSM6D/xpfp6vI11zM4DntwLZfb3wURRdZ2SpsWV5BzVOdLqxu+NkDNHIy8ibXfow+rAJymskO/57JKAhvdKFz2dOvn+1IjEj8aEdOgwMmKQvLX6IQIkC1w658RGLghZ9khKOteT8917oSlsTbsLClK2Eh8ukuzySK4xd+IiSt90DaJ6SsouC0g+s1ly8gXJ1J9V4x7qgaMD1nqGTdd/TiGRQMBKcyS2Vjq2f3G4TRs2nFaAz22R1sxkaAhmzLDNhrNd++kxXUJZBezd+57XLV6q4do73AnbZQOMloKRwxWcE1qeDvTB4fZIWlok1dXQ7aA+82h1cHoRVJL/UCX5aX5GYMI69JERQ1lgtBUa5xi5IGT9HoCSR6yb+tAhayeK358SXbrEeulgi4bfdHGRqadUE09KkHcGYMVGK9WwdevwnQh2lTFWE5/2rim9oYbZJAPWwrwDNpz12pO/lORxyqSk0kYFtmQbdu4cIwUjJVQMH1ic6oPJ9kh55iYsJ8scfU0htT5oNCa0SPQ4eWjuTBLq85DSp9kS9VHuNiiL6OmtmHHbOAo7OOciaYa3f6rF4c9XJHpCO3SFQpE+Mtxnn5VUP7R+vg59wqZcFApFehkHGYa0cc41nTShHLpCoVB8AMbjgKaOz1UoFIoMQTl0hUKhyBCUQ1coFIoMQTl0hUKhyBCUQx/vKNmtcYtqGsV4Q+1yGc+MF9kzxRmopkkfk3n/+7mYeBH6ZAqLkg+siUat1+lkPNT9eLCB8dc0qSZdzZAYSDdtsn6OLH+cdI+0MbEi9HESFqUsQjiXzlcqGVn3a9emRh1mNBtCIev8XCfOnD9PO+a26RT3W+pQ6W6aVJPO2/Bsx/imyq7xPEM4p0MXQkjgGWAGEAOeME3zUSFEIfBTQAN04G9N0+x0zlTSoEx7JintzOPpcbTkuq+vt44GLipK6R3dFtQRB0NMa2/CHQ5ZNmzc+F7ZKRPqrqqiJBqlUrioX1lJWSA96jTpQtehIGSwIF93TvJwDJJjnI4O63Rqt/s9ESPH3cM4iSnH5Hwi9Cjwj6ZpvimEmArsFkL8Bvgy8LJpmt8VQnwL+BbwTedMZexDmVN4ENa5Oo3tTiX+OJphgJ5OvcLkug+HLbHouHPv2LyNg/MqHHVshgGPbdH4+6YwZncI/ywfbo/Havu4HJmjQtkJkjpAvq4TKNExkCnXkkx0e89Jg4AIUnQBKTsIrtxtcGNdFSIaRcOFvrOStjaZkuITMc5rzxrUPK7zv3/WeOmHsPGbOuV/oeFySUcntInm9/stkbQXXgCJgfeEztxPpkgs/Cyc06GbpnkMOBb//ZQQogGYCVQA18a/9iPgNZx26KMdSHz//QwGdxMND8Lscjz//qCj03BNg/5+62by+YYLw7a0WEGjx2OvZvVoUQFY1ZDfaZB1RKd4qcOdKbnuE/J39fUMvllHUy305+zlsSWVjmkp6jqcyJW88en1LN1RjSjyMN3lsgSDc3MRJ10UeCqH6Vgm/p+tjnZEUNHi1njsfktL8vUcN2tvjTiuFGUY8MAD0PxHgwdOPkCEXQwWQ07AQenFJMoiOr6FURrCGt1v6PzqP3SMbAiU6NxWqbFolbPlSwwW11SRG4qSn9NPZFAgnp5C2eUu/mVtJU0R6djgmrj/a2qs+7Hjzwb3dldhDoXQHw3T8b31LFiThjRgnPeVQxdCaMAiLH2YkrizxzTNY0KI6WP8nzuAOwBmzZr1YWy1GHEoc7g1REdnDgV97Qy+eQD+cR2ee+yTPhsNISz5s3AYHnkEcnPB22GQc1SnvU2jPU8yZ459U76Rs4Jg0BICch0zuPa3VUzzRmn3uuB5a/rvWOohue7LyujYvI32g/BuZD5z3Tr+kO7Y9DvhR/8QDXDwmo38w3U6J4+3cVFdDV5No6BXZ3ZHkK6gTrFPw+2WzkyNRwQVRhBW7q6iKCdEYXMd2W2zoVg4mt9PCGXPdun4szoJm14GsiHaGuLdzUEumKfbOqiMpkxkulx07dXp6nHRnuXmzoEq3CeidKwd4ETLTRSvcDBc13WmeqI0RjUWDwTxmCYn86+mN6RTFtEp+4RzA4qUsHq1pXZZUACDr+jkD4WQkSYKIiFC31zHyehXKVqeumOzkzlvhy6EyAe2AveYptkthDiv/2ea5hPAE2Adn/tBjBwTTeP4oA9PuBsXUXpc+XgaD1nivJs3O3JT6TpMmQJXXw1/+pO1LnhVqcG8l6qIDURZ3O3iQVcl+/olbrc9ZY7MNIHlqMr6dIb6o7yTozGzT+fwz4K49wb51X9DY2GAbp90bmEIyWP1FXwqtBdXu87JC1x0XqY5tjg4fIIgefhJSUHI4Ma6HSxBx+sa4DbvFk5Fcpna18/hF1dTEArYrzyfMCb+xxJakh2RfKYPDeA7fhBODp2R37cTTbOm/I1Rjc6Yn0toInsADhzKYUDfQi9DTC0Pk/fgh+//Ywk51N9UyevHdP54TEPEI44Ot59Aew3R/9cFr2+Fm25yJrDSNHI8Lq4s1smK+DjVIxAHdHbnu5jtgJboSAIBK6AKheB4nkbOqTD5ZojBHA+l4UOIp5+CWnukF98v5+XQhRA5WM78x6Zp/jz+dpsQojQenZcCx50yciwMJA9dsIFpBS+wvOdnTMsdoijnsKXnaRj0ratm31c32prbTXaufr8lWNzyB50rRJTmfI28Pp3FhTqhyyWRiD1ljpZp2rEDGsIa5biQMZ3sWD+zXn+WvJ/s57PdcGLWEjbP2+BYxJxIgdTdWEnnHp3yGzTu/rKzi2MJP7pzp3UzDeVLnp8dF+ud0Ya3pgav3w81NVzc3cWNx3Y4pzwfJ6El2XcsxLS+GDmxAfBfYOXdHFq0l9KSqQ0GJVknH8Itgrx7HF56AZa1/pTCriY4cOai8QdhrDWjsoDk2A6JvxiOD4JodrEguwERg4jXZzXSsWO2acoOQ0pa76jkpe/qHIpptLbC316l866p8dmIpMwwrOT2iRPwyU/aHtSNDC6at67n+NPV5IsezIEB3JfPg2hqpCdHcj67XASwCWgwTfPhpI+2A7cB343/3OaIhWdB16G3UOK++U42vrmCr10ZJGevpUMV8fiob/Lw8iadQzvsi1RHppK//31ozdUQbhfXaToNQy4KLtfIK7V3UWbkUZ2VldYN/esfVFLYrVPmauOvc54iu8DLYBjMjhAlYR1Nc6ZDJYv0umZL7vhy6vqu2w11dYmoUfIPSyWUGZbzOHAAAM+ieSwp7sS3TKewwsGBRiZpSXZ2whNPnF5EcXAv4+n+IAFJUxBqfmiw+Phmpg+GMEt8tgwqY+2cTdwHwSA8G5FsjFSyoC/ImoJnWNL0O+jrtgKr4mLbHZthwH9ul4RmSjo6oPAK2J1nbR8tdxtw113W9hewZurPP++IU09cUiAQoOWmjXS9GGTqrq14o/ZKL74fzidC/zhwC7BPCLE3/t46LEf+30KINcAR4PPOmDg2ic7W2Qm5cyTyHgktEqqr6TrloW/AR+5lmu2DZXKkOGUKaNdJauoruW2ZzuVLLcVxp3c8JGwIBGRc9d3A8/2tcLSJ0gKYcomPW9ZrlDlkQzp3VEYisHAh5OdDT4/1epiHieuZen0urqjQwGnbku/uRYvSUimRCExfIvlDeD2+3dXklXrw2jConK2dpTw9U73k85KGBom5AHJ2xZ15OGz9s9mxJWYNCQ3qlSuhpMQqpkzXoa3Ncgxut7WCWVvr+PMKZQFpbUowAmndZnw+u1x+B4yVML/BXnPeH6N2NhmAjRsxgzo1WzROdDr34Mew6MUnKayQlEkcz+ElMyxSs+bh5AAlKdhDlq4D/jXNCoCj0RGB8OlRLn03VZoqJVEnTd4AP122kftW62DTwujZLik5qLrwQrjo8wHo2mFF5omHv2yuj5GzhuFdXbO8e0OD1UGKi2HpUlvLPyvpuiniZLSmaCoeNBnPT41lMqrezyRddXJGuSkw5KxFOJxDTwdKJFqhUCgyhPN16BPvcC6FQqFQjIpy6AqFQpEhKIeuUCgUGYJy6AqFQpEhKIeuUCgUGYJy6AqFQpEhKIeuUCjGNxNIVy7dpk4sCTqFQjG5GO8SQUmMB1NVhK7IKNIdITmOYViiHlu2nPUina6HlNXzBFLjHg+mqgh9gqEeeR+bYNAZxagziDdCi1ujKWKde58SveyEVFHiaeslS2i5d8MZCj1OR4otQYPnq3XaPJqjZ+4DZx7ckiwRJs8u5pLqe2U8aLpPOIc+rhxais+MGA9TumEEg1BbS+uspRz0B1LbJiM6gmHAunXW6bl+P8ybZ/9x1IZhObP5W6sQ3SFO7Avz69nr+fnRAAsXWoPI2rUOOvfEEb1eLwCdh0M8eq9OyA8XC51b1msMlUm2bbPOi0+cRmhrPRgGorqKjzVEmeYZ4LWim6h7IYA+36ETRkeTPoxGYWCA9suX8auXiziZK5kiItbponEZRqcH90RfmH3w11zQfhDmzEEuX87atZLaWpg163SEnsp7dEI59FEdGqkViR5mzF13wSuvWAY9/DD827/BmjWOFTfqjcrwaDFlTjUYhFtvZTDUS15nP11z1vDYgq87pik6jOSO0N8Pq1dTdzLA229LTp2y5MESx6naXeQlR3WyGkLMFk2UdIb4wsFqdmZtJD9fEoprShQVOTTgJqSKmpoYjELdcR8F/X/izt7bEVNcZN8+g6cvfpA/DgWoq4O+Pii1+Vx+dJ0CT5Swx8+cIzVc13aMgX/ZzAuL1nOkNOBMkJF8XnVCofkXvyD3xZ18ri+HbLegOX8efE2wZ+16IosCVFdbBy76fFBebt+glhDnfvUZgzV7vk5e68vEYmGyXC76Nz3HMzOeZ1ebFWAsWmTVfyoDrwnl0Eeqp7QEDeSW++H3v7ekg6SEr34VVqwYVoOORPW6DkeOwMDAexEDd99N5zvHeWfpzbaqJCWcSShkCTtAvKO6rQ96Q1Ea61z8cmGl81PgBLW1DIZ64eRJvEMRPnXgUWaGDtASfBzpdOHJ0us1NdDVxcU9O5jaVUmPSxKNwg032FsHiSJzL9Nw7QvjiobodvvoMz3Miun09EjCYSsiHKnuYxunpYrY9Sf44/YQ97b+H6aYYUQEYg0Hufnw7bxd+hTZ2QE6O+Gf/sn+QcV0uVjoaiA/N0pu7knC7QafqavmB2x0TCErUTYuFzQ0EB2I0jFYwFA0Ql6khxmD+3G3R8j6p3X8q9xMn0/i81n3jF1Hsifuw+Zm8L+lUxhtRcSGIBaDSATXnl0UTH2B/YN30t8P+/dDXl5qhYsm1KLoGTkqdPrf0Rk61kasqwv27bPOX77//vdWaxKNsGmT9dOORRzDgGCbRn+O12rMOLG+PnIe+R7m/Q/w2P2GbQtGyQf6L1wIy5ZZo35ZxPrgZL6GiEZZkK+nbDGmddZS+roGEUNRTARR3PgGWtFIQeGJjhBXJ2LePPJc1vUXFcH06TB3rjNF1nVKXli8nuz58yi+upwZl/r4m69qfOEL8O/LtrO24R6m/Xa7czlUKTECq3m0eTXFrfsQZozEbZw9NMiM3oN85cg65uUbzJyJbTKIiUXQYIukyqzkZ9Nu53D2JbgiYTqiPo51eOiu023T0R2VRPrl9tvpvGQxLgaZ6hnClW3iHzyBOxZmetc7XNO0md4DBqWlVurNriPZE/fhvHnQ7NI4FpuBMIesD7OyMLOymDXQyDXmTmYJwyl9j7MyoSL0kYIWLXs0Qk2DXDJkVWpMQFZ2tjUsx4fFsTQRPyinZ/uS2aUPc//MLzLFePe9z12xQaabLfhDum3RSvJA5vNBRUXiGqwPikI6h10u3urRcDmrfPYeB/0B3l78IF/c802yImGyXFB6xQzyAvHCnVzsGEWdqKjUxfQCjdkRK3C3ezljeN8LkMdG2oI6G7donHhHsvjV7Xw2eAuzh2Jcd2gTPUvX4edmnJBL0nUYGoI3p13P509tIodBBCCyssj25DLVPcSVRTrHbNJSTc5wnTwJHo/Ee61kq1dyy+FqQi4P3jwfBXM02waQMYmnXwZlgF+uCzI0BKU97/CZI/+X7l4PU8NtLIu+xILsw7ReXckNNurcDhPz+KikY9njhP/8OFN+8oT1QY6HRbGDXNC9CXf2AHuLb6LijoDzM9YkJpRDh9PpNMOA9U9ISj2VPNi7lgIzhIt4cj0hY2MYzG3TKe7X0HV7lIuSB4hGAuxf918seuRWePddME1ME8IDLjp9mm2OdUwZsPgHXl1nttsSyE3liv6PPrKGFv8CLj3yIrM/Op3SNcvJSzSO06u3I9SJ8jWNbyEdXTAfLkYjeUeXnMi1yrv0jVeIRWPkFPrIbmvD/6tnob3J9ms3DEthraMD3sxZheviH7N68Mf8hfgzUyNd5OTmMnvhDFbeqlFm05JScp/v7bWiTl0HV2mArq9sZOcT1o6XqINi3CMpC0g+s9lq73K3wZRH6ijY9y69hwZpK12Mn04WzdMpsbHuz7wPJfA9+PLnoLaWnJwcLn/pjxSG/czYVcOnLujCs30HLEpdEn3COfQEum7lK3eXreIr2SV8Nv9FPnNjNtOWzj0dnlVVURKNUilc1K+stCWvPTLtU7QiAIuegXXryOrtBZeXyM0PcvcKe3OJYypbxT8oI/XSd1aQHGDr1gBTIuB6EirLQNo9LTqXIfG/LUntjoLkvlA483r+qvmH0N5ufbhw4enNyDYZlTxOer3xWWrhKn7kW0X5WoOphrU5wBsIELCxIkbOEO+7L3knj6RokbMD6VgMk1/csIGpwSBZz27lyoFO8v0u8hOzRUfKTCIQsP4ZBt7aWuaeaoCpwKLLrHA+hUn0CevQExqK5eUQLgswd32AacnT7MSKuKaRr+sESnTb9BVH1THdvBl0HY+msSjt+ylTg5RWPUyZMsJ3j4cNuSlgeF9YRfaeZ62tSIYBubm2X3vyOAlw661J4shSQsCZfjfmDDHp87R3+bgR3kAAbzr1ZBOpwC1bLGee4v4/YR36uTqZk7v8R+3A46JXp55Rq/mcjZM5DGt2uQpWrXJs/eDs4sjOMmG6d7oNHZEKTHX/z2xN0XH1FFLmoqo5dai6npycr6bohI3Qz4t0j9aTBFXNqUPVteJsTKh96AqFQqEYG+XQFQqFIkNQDl2hUCgyBOXQFQqFIkNQDl2hUCgyBOXQFQqFIkNQDl2hsIGMl75TTAgyex+6QpEC0qkkpR40Sj3juc4nvEMfz5WryEBG6XCpPIss2YRkRbaUDSQj9FTPdt85dm++DxvsZtzJQI5gQjv0tFbu9u2W/Nz111vndzAJB5fJdsHGaYWotg4XDZ+vZOEKy6H091tnMvkcPo/+zLPJHdIOPUvh56OQ5di9aRhw//2EW0MYR3z8YskGWrIlq1en5mybxOC90G/Qf0CnJail9Lzzc/GhHLoQ4tPAo0A28JRpmt+1xaoRjOY3DANe3mxQ2qjjX6RR1ylTd0rl9u1wyy0QizH01CYOf2kdoZU388B/SI4ehZkz4emnnbEl1SrnY/7NZF28cNiShXFYJPv8jXMIXac3FOV/GjTEEZ3f1AX5zZM6y7+mEQ5LOjpwVrGH4bOB3BMGvqM6WX1u5okI5W4NJwQ1RhZuKWTpBESQpqOWU2PEWfSOzVqCQdi9myHTy6yTjVweDvKrtyWtrdaBp053Q02D4n6DJTVVuIiy4NkB4KbUnpR2Fj6wQxdCZAP/Cfw10AzUCiG2m6a53y7jYPSRHuCx+w0+Fayi5ESI4rfDuBbcwdw2Pxia8xX7yisQixH1+jCPtzHlp8/y7pYm3umqpCVb0tgIzz0H3/62vcXGgxNCISsS3LBh+ABnt389a5Sl61ZhTU28p468cWNK5749D1TR0xkl3++i+55K56ffmkZ32IWvUyfCADd0bWEwlEvvXf0U5a6mvyzA/g5JMOicDYkTF7vrDb7UWMX8C0N49uwm66JZDH1nBsFbN9iqZzta4UUhnVZzgIve2MKFOblc/KyLDVsqOZEr3+snTp+g7HaDGYO33oJQH2RlWf3U6W4oJdy3WsfsimL6/YidNYRDXXh27BhzGpLKuOPDROhXAY2maR4CEEL8BKgAbHXoo430ALmtOgVmiMKsJor727mi/uvk/DQAO3zO516uv56hp39IrKMdAbRfMJsL9IPcYm7mudiXaRaWU7ebeHCC1wuNjQxzHCdfCHL1rlpeDy/lzWjAlo59tiirxa0x5WiYgo4QOYU+a+6fwoP824I6jbuinPBq+N/WeXWdji6ls6k3KTHXV/L7dTrNu9u4JlpDJ35WUINnoItQbAePF1biZJScOJm4Y5vObKJ4hYCekwy+G2Fgbz2Hm5/jycXfdqYOkhSyLqlvY+iXNXjmaQzsqWdh6zZOfKyCuk5rQCspgbVrk4UwbLIhEIArr2ToWCcHOvzs6AjQ1Q/TThl8fIZOnqk5K1QNlAQ0jj/r4sjLDZh9sOfYBgyHBAAAClpJREFUZcynE0/wTIWkVKeFP4xDnwkkb9JqBs6ICYUQdwB3AMyaNet9FzLaSN/SAr89onHt8TDTB0OYhVMgKwL5+barxIyGsWgVvwg8y0cObmNq6ztc2FLHRWYHksPMj+1nvfshKipSOP0KBlmw4VZmHu2lYnCQx2c+yH7Pmg9dDWNFWYYBVU9KZvnXs8qoZm6phzynk8cj0NGI4kJD59igi6YhjXLN+VxyWUCyZrNkc7WB+OEO5kUtoep3suYhszq5oVwn4JDQRAIpQVZosNcFjcdhaIisk8fJi5l86p3/4EDp9eh6wLFBDSkp1gzYtQOa6xk06rhsCGbV7KVdW8tbGyP8oVAbM7/+oct/6CGCm3Xuq9doHJDMjBl8M1pFYWeU6W/1c+n+1aA5lwIxkKzvqSRPBFmRvYW+5k7e6naxZaPGN9wGi/z6e6NYqhfMP4xDF6O8d8bh6qZpPgE8AdZ56O+3kNG0EnQdpi+RvNyxHvMP1UyNmVwYaiTW0IP/Yh/5cT1Rp+Y5ug5vzlxFx8dX4d2xhb8zNhAK5dHX76aITj51qU5JiWTnTnuLjwcndHaOEEKurSVnoJcLzHZisSj3tq3j560w/7UwuJd+4NzLWDoViU4aWxrgmbyN3LZM54oKLWXRuWFYN9Wr8yqZOahzdJbGYJ4cdXrvRDeQEj65RvIvf6qk4ECQG4e2Mj2nE22Oi8CDGmWpqIZkdZyqKjjYSG9sKgNDOZR31qJpDq9nJMrfto0cYO6F8xl4s545HdU0dRWR1+nix+WVzkTLUnKgWHI4BrEYaOi4RZSeHD+B9hq8/9UFtWOnQD4sug69hZK6mZLalgAzB3V6CjT6G+H416voDUTx+qxwXNNkSsW7PoxDb2b43PJCoOXDmTM6I8+ATsjPvXkqwM6SjaxcoPPGXjeaGSHL1PjSHpjxRBUFntMV65RyTLEWoHD6xQzV7iYrNkhE+hmSGtXVUFRk7zRLSrjnHqithaVLk/7m0qVE+wcR0ShZ7hxyPTFubViHqyUfnnPBM898KKc+0vZhkbtPUlghHV2LSyZ5CtufJ7luteSW+KWNdNwtQYPnqy0BYzujRcOwtgvmXSo56JY8eSrA4kKd3Rdp3J0Sbx4n3jjHQ26y7r6LIZFFJCuXK9YsTc3YKiVUVMDevfg6dZgapjffQ6epQUinJKyjafYbYhjw+uuQlwd9fXA8R8M15OIK7wGyBqB9+jwujDqn5Zksf9nhl5zotxZlr52yE3fEWjT2RnXQdeQnZErFuz6MQ68F5gghLgaOAn8HfMkWq87BcOk+yWshSaMHZn0MjjTDT767k5VtUTp9GjeU65bGoI01OTxylXjZwIkXgmz/GTQWBmgOSzwe+6dZCUcSjcLevVBWZv1doyzAb+Y9yOdq1zEkcvC5Irhyc6C4GI4cgRdftHXpP50KcyOnsCUlp8sfuQtHVFfxsYYoeT57o8WEDfPnQ08PmEWSvKvlmW2dotWwt+eu4tVrS1hi1rJbLOW6uSncbZTcGdxuvE8+yQ15Ot1hF+XrnZmtJHRsV6+GPXtgyRLJ7w5WcrI7yFVZW1lgOqvlObL/t7RYi7E5pkZ2o4uiHh18p8tPpSjJB3bopmlGhRBfB/4Xa9vi06Zp1ttm2TlIlu4LBmHrVisVEQ5DyK+R1++CkNWxvA407PBGkhTfKVmyCGK1cO0sa2ej3dOssfJxug6/LV9D9NIFePbVctUyD5du+Q7sj69P79plORebB7V07NI6790Tuk6BxxrUCenM7ggyt023ZRdUsg0+Hwgxij0pXA3TNDhSGuBQNJAeTe7kzlBWhlfXrXvOwet1uaz7ffZsa9ZqbZuUSHcAb0R3fBBNvmQprQ0Iui4pd1empPyxyBhN0ZFPzxWEDErCOres1yhzeJEqUX7y/evECv9YPmLU93/9A2sz/OWXWx+sWQOf+IQ9hqSZ8wp8kx6C6enoJz9f4PVPsc25JtsAo9izcyds2nR69HW4/ifjM16T6XonnaboiCABXZdomkzNAhVnRs+RiP3371ipjlHfX77cSrYnvHzKwzbnOK/ZQdIWO29bG9TU2JoDG2nDGX/O6Y3Y57An05ls13u+ZIxDTyYdjZ2q+3esazvj/XQmuscLiUoxDNixI3VbDRJlT/b6V6ScjEm5jAcm2zRwQqEaRzGBmXQpl/GAmgaOY1TjKCYBSuBCoVAoMgTl0BUKhSJDUA5doVAoMgTl0BUKhSJDUA5doVAoMgTl0BUKhSJDSOk+dCHECeCwA3+6CDjpwN+dKEz26wdVB+r6M/v6LzJNs/hcX0qpQ3cKIcSu89l0n6lM9usHVQfq+if39SdQKReFQqHIEJRDVygUigwhUxz6E+k2IM1M9usHVQfq+hWZkUNXKBQKReZE6AqFQjHpUQ5doVAoMoQJ79CFEJ8WQrwthGgUQnwr3fY4jRBCCiFeFUI0CCHqhRDfiL9fKIT4jRDiYPynP922OokQIlsIsUcIURN/fbEQIhi//p8KIdzpttEphBDThBBbhBAH4v3go5Op/YUQ98b7/ltCiP8SQuROpvY/GxPaoQshsoH/BP4G+AjwRSHER9JrleNEgX80TXMecDVwV/yavwW8bJrmHODl+OtM5htAQ9Lr7wGPxK+/E1iTFqtSw6PADtM0LwOuwKqHSdH+QoiZwN3AlaZpLsASqP87Jlf7j8mEdujAVUCjaZqHTNOMAD8BKtJsk6OYpnnMNM0347+fwrqZZ2Jd94/iX/sRcGN6LHQeIcSFwArgqfhrAVwPbIl/JWOvXwhRAPwlsAnANM2IaZpdTKL2xxLm8QghXEAecIxJ0v7nYqI79JmAkfS6Of7epEAIoQGLgCBQYprmMbCcPjA9fZY5zveBB4BY/PUFQJdpmtH460zuB5cAJ4AfxlNOTwkhvEyS9jdN8yjwb8ARLEceAnYzedr/rEx0hy5GeW9S7MMUQuQDW4F7TNPsTrc9qUIIsRI4bprm7uS3R/lqpvYDF7AY2Gia5iKglwxNr4xGfG2gArgYKAO8WCnXkWRq+5+Vie7Qm4FkocgLgZY02ZIyhBA5WM78x6Zp/jz+dpsQojT+eSlwPF32OczHgVVCCB0rxXY9VsQ+LT4Fh8zuB81As2mawfjrLVgOfrK0/18B75qmecI0zUHg58DHmDztf1YmukOvBebEV7jdWIsj29Nsk6PE88WbgAbTNB9O+mg7cFv899uAbam2LRWYpvlt0zQvNE1Tw2rvV0zT/HvgVWB1/GuZfP2tgCGEuDT+1g3AfiZJ+2OlWq4WQuTF74XE9U+K9j8XE/5JUSHEcqwILRt42jTN76TZJEcRQlwD7AT2cTqHvA4rj/7fwCysTv950zQ70mJkihBCXAvcZ5rmSiHEJVgReyGwB7jZNM2BdNrnFEKIv8BaEHYDh4B/wArOJkX7CyGqgC9g7fjaA9yOlTOfFO1/Nia8Q1coFAqFxURPuSgUCoUijnLoCoVCkSEoh65QKBQZgnLoCoVCkSEoh65QKBQZgnLoCoVCkSEoh65QKBQZwv8HYTxJ6ApRaHUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data = np.load(\"blobs_zoomed.npz\")\n",
    "X = data[\"X\"]\n",
    "Y = data[\"Y\"]\n",
    "plt.plot(X[:,0], X[:,1], 'b.', alpha=0.5)\n",
    "plt.plot(Y[:,0], Y[:,1], 'r.', alpha=0.5)\n",
    "plt.title(\"One blob\")\n",
    "\n",
    "plt.figure()\n",
    "data = np.load(\"blobs.npz\")\n",
    "X = data[\"X\"]\n",
    "Y = data[\"Y\"]\n",
    "plt.plot(X[:,0], X[:,1], 'b.', alpha=0.5)\n",
    "plt.plot(Y[:,0], Y[:,1], 'r.', alpha=0.5);\n",
    "plt.title(\"All blobs\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clearly, the two distributions are very different.\n",
    "On the grid, the scale at which they differ, however, is somewhat invisible from the median heuristic, which looks at global scaling of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "median bw/sigma: 51.60189981691304 / 5.079463545341823\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XmYVNW57/HvK4MtiqSBjgcl3sYEBWmh0IYgmCvgAALX4aBHvOIQDZwkx+M8AM4KedALSgiJBg0BPQnDAVGjaBSBGEkQG2wRBINMCUKYPKCIqOB7/9i726Kp7q5qeld1u3+f56mnqlattfZbRdFvrbX3XtvcHRERia/Dch2AiIjklhKBiEjMKRGIiMScEoGISMwpEYiIxJwSgYhIzCkRiIjEnBKBiEjMKRGIiMRcw1wHkI6WLVt6YWFhrsPI2PvvB/cnnZTbOGqs3r8ByQZ9TequJUuWbHf3gurq1YtEUFhYSElJSa7DyFjPnsH9ggW5jOIQ1Ps3INmgr0ndZWYb0qmnqSERkZhTIhARiTklAhGRmKsX+whE5EBffvklGzduZO/evbkOhXvvDe5XrsxtHHGWl5dH69atadSoUY3aKxGI1EMbN26kadOmFBYWYmY5jeWwcF5BRw3lhruzY8cONm7cSJs2bWrUh6aGROqhvXv30qJFi5wnAck9M6NFixaHNDpUIhCpp5QEpMyhfheUCEREYk6JQERq5Oc//zlFRUUMGNCBKVPGlZffd999HHfccSQSCRKJBHPmzAFg4cKFdOzYkS5duvDBBx8AsHPnTvr06UMur53+7LPP8t5775U/v+eee5g7dy4APXv2rJcns2ZKO4ul1hQOezFn214/un/Oth1Hy5cv54knnmDx4sWsW9eYIUP6snp1f9q2bQvATTfdxK233npAm7FjxzJr1izWr1/PY489xtixY3nwwQcZMWJETqe5nn32WQYMGMDJJ58MwAMPPJCzWHJFIwIRydjKlSvp1q0bTZo0oWHDhnTpciazZ8+usk2jRo347LPP2LNnD40aNWLNmjV8+OGHnHnmmZW2efnll2nXrh1nnHEG119/PQMGDACCUceYMWPK6xUVFbF+/XoALrzwQk477TQ6dOjAxIkTy+scddRR3HnnnXTq1Ilu3bqxZcsW/vKXv/D8889z2223kUgkWLNmDVdffTUzZ848KJZXXnmF008/nVNPPZVLLrmE3bt3AzBs2DBOPvlkOnbseFDyqy80IhCp5268EUpLa7fPRALGjav89aKiIu6880527NjBZ58dwZ/+NIcf/KC4/PUJEybw1FNPUVxczNixY8nPz2f48OEMHTqUI444gqeffppbb72VBx98sNJt7N27lyFDhjBv3jy+973vcemll6YV+6RJk2jevDmfffYZXbp0YeDAgbRo0YJPP/2Ubt26MWrUKG6//XaeeOIJ7rrrLs4//3wGDBjAxRdfXGmf27dvZ+TIkcydO5cjjzyShx56iEceeYTrrruO2bNns2rVKsyMnTt3phVjXaMRgYhkrH379txxxx2cc845DBnSl3btOtGwYfC78ic/+Qlr1qyhtLSUVq1accsttwCQSCRYtGgR8+fPZ+3atRx77LG4O5deeimDBw9my5YtB2xj1apVtGnThrZt22JmDB48OK3Yxo8fX/6r/x//+AerV68GoHHjxuUjitNOO618BJGORYsW8d5779GjRw8SiQRTpkxhw4YNHH300eTl5fGjH/2IZ555hiZNmqTdZ12iEYFIPVfVL/coXXvttVx77bW8/z488sgI2rZtDcAxxxxTXmfIkCHlf3zLuDsjR45k+vTpXHfdddx///2sX7+e8ePHM2rUqAPqVrbvoGHDhnz11Vflz8uOoV+wYAFz587lr3/9K02aNKFnz57lrzVq1Ki8vwYNGrBv376036u7c8455zB16tSDXlu8eDGvvfYa06ZNY8KECcybNy/tfusKjQhEpEa2bt0KwKZNf+fVV5/hsssuA2Dz5s3ldWbPnk1RUdEB7aZMmUL//v3Jz89nz549HHbYYRx22GHs2bPngHrt2rVj3bp1rFmzBuCAP8KFhYUsXboUgKVLl7Ju3ToAdu3aRX5+Pk2aNGHVqlUsWrSo2vfRtGlTPvnkkyrrdOvWjYULF5Yf7bRnzx7+9re/sXv3bnbt2kW/fv0YN24cpbU9R5clkY8IzKwBUAJ86O4DzKwNMA1oDiwFrnD3L6KOQ0Rq18CBA9mxYwdffdWIe+75Jfn5+QDcfvvtlJaWYmYUFhby61//urzNnj17mDJlCq+88goAN998MwMHDqRx48YH/drOy8tj4sSJ9O/fn5YtW3LGGWewfPny8m0/9dRTJBIJunTpwoknnghA3759efzxx+nYsSMnnXQS3bp1q/Z9DBo0iCFDhjB+/PiUO4kBCgoKmDx5Mpdddhmff/45ACNHjqRp06ZccMEF7N27F3fn0UcfzfBTrBss6uN3zexmoBg4OkwEM4Bn3H2amT0OvOPuj1XVR3FxsdfHY3nr/QU7MnwDOnw0e1auXEn79u1zHQaQvSuULViwgDFjxvDCCy9Eu6F6KtV3wsyWuHtxJU3KRTo1ZGatgf7Ak+FzA3oDZWl3CnBhlDGIiEjVop4aGgfcDjQNn7cAdrp72V6ajcBxEccgIt8APXv2pGfZKFVqVWSJwMwGAFvdfYmZ9SwrTlE15dyUmQ0FhgIcf/zxkcQo3xy5mpaK25SUfDNFOTXUAzjfzNYT7BzuTTBC+JaZlSWg1sCmVI3dfaK7F7t7cUFBQYRhiojEW2SJwN2Hu3trdy8EBgHz3P1yYD5QdgrfVcBzUcUgIiLVy8V5BHcAN5vZBwT7DH6TgxhERCSUlUTg7gvcfUD4eK27d3X377n7Je7+eTZiEJHaN3v2bNq1M9auXVVt3cmTJ7NpU8qZ4LQsWLDgoLOUAUpLS8uXus7Uzp07+dWvfpVRm3Hjxh108ls69fr161flWkSZ1q9NWmLiG6i2dpxOW7sDgEE5PD9A6rapU6dy2mln8OKL0zjvvPuqrDt58mSKioo49thjazWG0tJSSkpK6NevX8ZtyxLBT3/607TbjBs3jsGDB1e7rlDFetUlq0zr1yYtMSEiNbJ7924WLlzIyJG/Yc6caQe89vDDD3PKKafQqVMnhg0bxsyZMykpKeHyyy8nkUjw2WefUVhYyPbt2wEoKSkpPzR08eLFdO/enc6dO9O9e3feLztjLYUvvviCe+65h+nTp5NIJJg+fTqffvop11xzDV26dKFz584891ywG3LFihV07dqVRCJBx44dWb16NcOGDWPNmjUkEgluu+22A/r+9NNP6d+/P506daKoqIjp06czfvx4Nm3aRK9evejVqxcQLLJXXFxMhw4duPfeewFS1it7v+n2m/z5PPXUU3Ts2JFOnTpxxRVXHMo/W0oaEYjUd7lYh5rggi59+/alTZsTadasOUuXLuXUU0/lpZde4tlnn+XNN9+kSZMmfPTRRzRv3pwJEyYwZswYiourPtG1Xbt2vP766zRs2JC5c+cyYsQIZs2albJu48aNeeCBBygpKWHChAkAjBgxgt69ezNp0iR27txJ165dOfvss3n88ce54YYbuPzyy/niiy/Yv38/o0ePZvny5SnXCHr55Zc59thjefHFYES8a9cumjVrxiOPPML8+fNp2bIlAKNGjaJ58+bs37+fs846i2XLlnH99dcfVC/TfsusWLGCUaNGsXDhQlq2bMlHH31U5edXExoRiEiNTJ06lUGDBgHQr9+g8rWC5s6dyw9/+MPyKY7mzZtn1O+uXbu45JJLKCoq4qabbmLFihUZtX/llVcYPXo0iUSifPXRv//975x++un87Gc/46GHHmLDhg0cccQRVfZzyimnMHfuXO644w7+/Oc/06xZs5T1ZsyYwamnnkrnzp1ZsWLFAZe9PJR+y8ybN4+LL764PEFk+nmmQyMCkfouB+tQ79ixg3nz5rF8+XL27TO++mo/jRoZDz/8MO6e1qUnk5eSLlsqGuDuu++mV69ezJ49m/Xr12d8NrG7M2vWLE6qsPhR+/bt+f73v8+LL75Inz59ePLJJznhhBMq7efEE09kyZIlzJkzh+HDh3Puuedyzz33HFBn3bp1jBkzhrfeeov8/HyuvvrqA95LTfut+H6ivpSnRgQikrGZM2dy5ZVXsmHDBubNW8+CBf+gTZs2vPHGG5x77rlMmjSp/AiYsqmMiss9FxYWsmTJEoADpn527drFcccFK89Mnjy52lgq9tunTx9+8YtfULag5ttvvw3A2rVrOeGEE7j++us5//zzWbZsWZVLUG/atIkmTZowePBgbr311vJlr5PbfPzxxxx55JE0a9aMLVu28NJLL1UaVyb9JjvrrLOYMWMGO3YEB29oakhE6oSpU6dy0UUXHVA2cOBAfv/739O3b1/OP/98iouLSSQS5dcWvvrqq/nxj39cvrP43nvv5YYbbuAHP/gBDRo0KO/n9ttvZ/jw4fTo0YP9+/dXG0uvXr147733yncW33333Xz55Zd07NiRoqIi7r77bgCmT59OUVERiUSCVatWceWVV9KiRQt69OhBUVHRQTuL33333fKdy6NGjeKuu+4CYOjQoZx33nn06tWLTp060blzZzp06MA111xDjx49ytsn18u032QdOnTgzjvv5Mwzz6RTp07cfPPN1X4mmYp8GeraoGWoM1Nrh4/+fhgAg/7v6Frp75soV2sNxXEZaqlanV2GWkRE6j4lAhGRmFMiEKmn6sO0rmTHoX4XlAhE6qG8vDx27NihZCC4Ozt27CAvL6/Gfeg8ApF6qHXr1mzcuJFt27blOhT++c/gPjwlQHIgLy+P1q1b17i9EoFIPdSoUSPatGmT6zAA+MlPgvtsHx0ntUdTQyIiMadEICISc5ElAjPLM7PFZvaOma0ws/vD8slmts7MSsNbIqoYRESkelHuI/gc6O3uu82sEfCGmZUtxHGbu8+McNsiIpKmyBKBB8e17Q6fNgpvOtZNRKSOiXQfgZk1MLNSYCvwqru/Gb40ysyWmdmjZnZ4JW2HmlmJmZXUhUPkRES+qSJNBO6+390TQGugq5kVAcOBdkAXoDlwRyVtJ7p7sbsXFxQURBmmiEisZeWoIXffCSwA+rr7Zg98DvwW6JqNGEREJLUojxoqMLNvhY+PAM4GVplZq7DMgAuB5VHFICIi1YvyqKFWwBQza0CQcGa4+wtmNs/MCgADSoEfRxiDiIhUI8qjhpYBnVOU945qmyIikjmdWSwiEnNKBCIiMadEICISc0oEIiIxp0QgIhJzSgQiIjGnRCAiEnNKBCIiMadEICISc0oEIiIxp0QgIhJzSgQiIjGnRCAiEnNKBCIiMadEICISc1FeoSzPzBab2TtmtsLM7g/L25jZm2a22symm1njqGIQEZHqRTki+Bzo7e6dgATQ18y6AQ8Bj7p7W+B/gGsjjEFERKoRWSIIL1C/O3zaKLw50BuYGZZPIbhusYiI5Eik+wjMrIGZlQJbgVeBNcBOd98XVtkIHBdlDCIiUrUoL16Pu+8HEmb2LWA20D5VtVRtzWwoMBTg+OOPjyxGkUNROOzFnG17/ej+Odu2fLNk5aghd98JLAC6Ad8ys7IE1BrYVEmbie5e7O7FBQUF2QhTRCSWojxqqCAcCWBmRwBnAyuB+cDFYbWrgOeiikFERKoX5dRQK2CKmTUgSDgz3P0FM3sPmGZmI4G3gd9EGIOIiFQjskTg7suAzinK1wJdo9quiIhkRmcWi4jEnBKBiEjMKRGIiMScEoGISMwpEYiIxJwSgYhIzCkRiIjEnBKBiEjMKRGIiMScEoGISMwpEYiIxJwSgYhIzCkRiIjEnBKBiEjMKRGIiMScEoGISMxFeanK75jZfDNbaWYrzOyGsPw+M/vQzErDW7+oYhARkepFeanKfcAt7r7UzJoCS8zs1fC1R919TITbFhGRNEV5qcrNwObw8SdmthI4LqrtiYhIzWQ8NWRm+WbWMcM2hQTXL34zLLrOzJaZ2SQzy880BhERqT1pJQIzW2BmR5tZc+Ad4Ldm9kiabY8CZgE3uvvHwGPAd4EEwYhhbCXthppZiZmVbNu2LZ1NiYhIDaQ7ImgW/hH/V+C37n4acHZ1jcysEUES+J27PwPg7lvcfb+7fwU8AXRN1dbdJ7p7sbsXFxQUpBmmiIhkKt1E0NDMWgH/BryQTgMzM+A3wEp3fySpvFVStYuA5WnGICIiEUh3Z/H9wB+BN9z9LTM7AVhdTZsewBXAu2ZWGpaNAC4zswTgwHrg3zOOWkREak26iWCzu5fvIHb3tdXtI3D3NwBL8dKcDOITEZGIpTs19Is0y0REpJ6pckRgZqcD3YECM7s56aWjgQZRBiYiItlR3dRQY+CosF7TpPKPgYujCkpERLKnykTg7n8C/mRmk919Q5ZiEhGRLEp3Z/HhZjYRKExu4+69owhKRESyJ91E8N/A48CTwP7owhERkWxLNxHsc/fHIo1ERERyIt3DR/9gZj81s1Zm1rzsFmlkIiKSFemOCK4K729LKnPghNoNR0REsi2tRODubaIOREREciOtRGBmV6Yqd/enajccERHJtnSnhrokPc4DzgKWAkoEIiL1XLpTQ/+Z/NzMmgFPRxKRiIhkVcaXqgztAdrWZiAiIpIb6e4j+APBUUIQLDbXHpgRVVAiIpI96e4jGJP0eB+wwd03RhCPiIhkWVpTQ+Hic6sIViDNB76oro2ZfcfM5pvZSjNbYWY3hOXNzexVM1sd3ucfyhsQEZFDk1YiMLN/AxYDlxBct/hNM6tuGep9wC3u3h7oBvyHmZ0MDANec/e2wGvhcxERyZF0p4buBLq4+1YAMysA5gIzK2vg7puBzeHjT8xsJXAccAHQM6w2BVgA3FGD2EVEpBake9TQYWVJILQjg7aYWSHQGXgTOCZMEmXJ4tuVtBlqZiVmVrJt27Z0NyUiIhlKd0Twspn9EZgaPr+UNC9Cb2ZHAbOAG939Y7NU17M/mLtPBCYCFBcXezXVRUSkhqq7ZvH3CH7B32Zm/wqcARjwV+B31XVuZo0IksDv3P2ZsHiLmbVy981m1grYWnkPIiISteqmd8YBnwC4+zPufrO730QwGhhXVUMLfvr/Bljp7o8kvfQ8X69mehXwXE0CFxGR2lHd1FChuy+rWOjuJeG8f1V6AFcA75pZaVg2AhgNzDCza4G/ExyJJCIiOVJdIsir4rUjqmro7m8QTCOlclY12xWRahQOezEn210/un9OtivRqW5q6C0zG1KxMPw1vySakEREJJuqGxHcCMw2s8v5+g9/MdAYuCjKwEREJDuqTATuvgXobma9gKKw+EV3nxd5ZCIikhXpXo9gPjA/4lhERCQH0j2hTGpg0dodABQOW5TjSEREKlfTC9OIiMg3hBKBiEjMKRGIiMScEoGISMwpEYiIxJwSgYhIzCkRiIjEnBKBiEjMKRGIiMScEoGISMxFlgjMbJKZbTWz5Ull95nZh2ZWGt76RbV9ERFJT5QjgslA3xTlj7p7IrzNiXD7IiKShsgSgbu/DnwUVf8iIlI7crGP4DozWxZOHeXnYPsiIpIk24ngMeC7QALYDIytrKKZDTWzEjMr2bZtW7biExGJnawmAnff4u773f0r4AmgaxV1J7p7sbsXFxQUZC9IEZGYyWoiMLNWSU8vApZXVldERLIjsiuUmdlUoCfQ0sw2AvcCPc0sATiwHvj3qLYvIiLpiSwRuPtlKYp/E9X2RESkZnRmsYhIzCkRiIjEnBKBiEjMKRGIiMScEoGISMwpEYiIxJwSgYhIzCkRiIjEnBKBiEjMKRGIiMScEoGISMwpEYiIxJwSgYhIzCkRiIjEnBKBiEjMKRGIiMRcZInAzCaZ2VYzW55U1tzMXjWz1eF9flTbFxGR9EQ5IpgM9K1QNgx4zd3bAq+Fz0VEJIciSwTu/jrwUYXiC4Ap4eMpwIVRbV9ERNKT7X0Ex7j7ZoDw/tuVVTSzoWZWYmYl27Zty1qAIiJxU2d3Frv7RHcvdvfigoKCXIcjIvKNle1EsMXMWgGE91uzvH0REakg24ngeeCq8PFVwHNZ3r6IiFQQ5eGjU4G/AieZ2UYzuxYYDZxjZquBc8LnIiKSQw2j6tjdL6vkpbOi2qaIiGSuzu4sFhGR7FAiEBGJOSUCEZGYUyIQEYk5JQIRkZhTIhARiTklAhGRmFMiEBGJOSUCEZGYUyIQEYk5JQIRkZhTIhARiTklAhGRmFMiEBGJOSUCEZGYi+x6BFUxs/XAJ8B+YJ+7F+ciDhERyVEiCPVy9+053L6IiKCpIRGR2MtVInDgFTNbYmZDcxSDiIiQu6mhHu6+ycy+DbxqZqvc/fXkCmGCGApw/PHH13hDhcNePKRAD023HG5bJBoV/0/9c223sHxR5NteP7p/5NuIo5yMCNx9U3i/FZgNdE1RZ6K7F7t7cUFBQbZDFBGJjawnAjM70syalj0GzgWWZzsOEREJ5GJq6BhgtpmVbf/37v5yDuIQERFykAjcfS3QKdvbFRGR1HT4qIhIzCkRiIjEnBKBiEjMKRGIiMScEoGISMwpEYiIxJwSgYhIzCkRiIjEnBKBiEjMKRGIiMRcLq9QJiJSL+RyOftsLL2tEYGISMwpEYiIxJwSgYhIzCkRiIjEnBKBiEjM5SQRmFlfM3vfzD4ws2G5iEFERAK5uGZxA+CXwHnAycBlZnZytuMQEZFALkYEXYEP3H2tu38BTAMuyEEcIiJCbhLBccA/kp5vDMtERCQHzN2zu0GzS4A+7v6j8PkVQFd3/88K9YYCQ8OnJwHvZzXQQ9MS2J7rIDJUH2OG+hm3Ys6e+hh3bcb8v9y9oLpKuVhiYiPwnaTnrYFNFSu5+0RgYraCqk1mVuLuxbmOIxP1MWaon3Er5uypj3HnIuZcTA29BbQ1szZm1hgYBDyfgzhERIQcjAjcfZ+ZXQf8EWgATHL3FdmOQ0REAjlZfdTd5wBzcrHtLKmPU1r1MWaon3Er5uypj3FnPeas7ywWEZG6RUtMiIjEnBJBNcysuZm9amarw/v8SupdFdZZbWZXJZWfZmbvhstpjDczC8svMbMVZvaVmRVX6Gt4WP99M+tTh2JO2a+Z9TSzXWZWGt7uySDWKpcbMbPDzWx6+PqbZlaY9FrKz6myPsMDFN4M458eHqyQsSzHPNnM1iV9tomaxBxh3JPMbKuZLa/QV1rfwToW831m9mHSZ92vLsRsZt8xs/lmttKCvxk3JNWvlc8Zd9etihvwMDAsfDwMeChFnebA2vA+P3ycH762GDgdMOAl4LywvD3B+RELgOKkvk4G3gEOB9oAa4AGdSTmlP0CPYEXavDZNgjf3wlA4/B9n1yhzk+Bx8PHg4DpVX1OVfUJzAAGhY8fB35SD2KeDFxcC9/jWo87fO1/A6cCyzP9DtbBmO8Dbq1rnzPQCjg1rNMU+FvS9+OQP2d314ggDRcAU8LHU4ALU9TpA7zq7h+5+/8ArwJ9zawVcLS7/9WDf6mnytq7+0p3T3WS3AXANHf/3N3XAR8QLMuR85jT7DcT6Sw3krzNmcBZ4Qilss8pZZ9hm95hH4cSf9ZirkFs2Y4bd38d+CjF9mrju5LtmGtDrcfs7pvdfWkY+yfASr5ejaFW/k8qEVTvGHffDBDefztFncqWzTgufFyxvCq1sQRHVDFX1e/pZvaOmb1kZh3SjDOd91pex933AbuAFtXEn6q8BbAz7KOybdW1mMuMMrNlZvaomR1eg5ijirsq6XwH61rMANeFn/WkGk6zRBpzOI3UGXgzLKqNz1mJAMDM5prZ8hS3dH+VWYoyr6K8Jn0dWKluxbyU4FT2TsAvgGcPMYZ06tRWeaayGTPAcKAd0IVgGu+O9MI8SBRxRy3bMT8GfBdIAJuBsdUFmEJkMZvZUcAs4EZ3/7gGsVUqJ+cR1DXufnZlr5nZFjNr5e6bw2mTrSmqbSSYJy/TmmDuf2P4OLn8oOU0UvSVzhIcuYg5Zb/JX0p3n2NmvzKzlu5e3Xop6bzXsjobzawh0IxgWF9V21Tl24FvmVnD8FdYOv8WuY657FcewOdm9lvg1hrEHGXclUnnO1inYnb3LWWPzewJ4IW6ErOZNSJIAr9z92eS6tTG56ydxdXdgP/HgTtjHk5RpzmwjmCna374uHn42ltAN77e8dqvQtsFHLizuAMH7jBaS+Y7iyOJubJ+gX/h63NSugJ/L3teTZwNw/fXhq93rHWoUOc/OHDH2oyqPqeq+gT+mwN3Fv+0Bt+HbMfcKrw3YBwwuobf41qPO6ldIQfveK32O1gHY26V9Pgmgvn6nMcc/ts/BYyryf/1tOKuSaM43Qjm7l4DVof3ZX8si4Enk+pdQ7Bz5wPgh0nlxcBygiMAJvD1H8yLCH4BfA5sAf6Y1ObOsP77hEfs1JGYK+v3OmBF+CVeBHTPINZ+BEdBrAHuDMseAM4PH+cR/AH/gOBophOq+5xS9RmWnxD28UHY5+E1/E5kM+Z5wLvhv8d/AUcdwnc5irinEkyjfBl+n6+t6rtSx2N+OvyslxGsf9aqLsQMnEEwRbQMKA1vZT/OauVz1pnFIiIxp53FIiIxp0QgIhJzSgQiIjGnRCAiEnNKBCIiMadEIDlnZi2SVnz8Z4UVINNeIdTMrjGzf8mgfm8z65ZpPTP7DzO7vLbqHwoza2hmO6PoW+JDZxZLzrn7DoLT+jGz+4Dd7j6mBl1dQ7DcxT/TrN+b4IzjRZnUc/df1nJ9kZzSiEDqNAuumbA4HB38yswOC38FP23BNROWm9n1ZnYpQTKZnmokYWY3mdl74cJ4/2Vm3wV+BNwW1u9uZheE68O/bWavmNm3K6k30sxuzKDf5Ponmtm8sP5SS1qLPnx9rJkNTXo+0sxuMLOjw3ZLw0XRBqT4rM42s2eTnj9uZoPDx13M7E9mtsSChQGPqY1/H/lm0IhA6iwzKyI4A7u7u+8zs4kEp+SvAVq6+ylhvW+5+04z+0/gOncvTdHd7QQL432RVP9JYLu7jwv7yQeed3c3sx8Dt7j7HSnq9cuw3+T6U4H73P0PZpbHwT/GpgGj+fq6tZcAvYDPgAti9LTFAAACK0lEQVTc/RMz+zawkDTXwrFgxdKfE5zZuj2cpnoQGFp1S4kLJQKpy84mWHWzxIKLpB1BsEzvH4GTzOznwBzglTT6WgH8l5k9R+Wrox4PzAj3MxxOsExAbfQLlCealu7+BwB331uxjru/ZcEVqY4hWHTsn+6+KRzhPGRmZwBfAd8xs5ZAOvsH2hOsYzM3/BwbcOBS4xJzSgRSlxkwyd3vPugFs47AecD1wECq/3XbBziT4EIed4WjjYp+CfzMgxVUzyZYxKs66fSbLJ01XWYRvKdCghECwJUEq1SeGo6ONhKsWZNsHweOMMpeN2CZu/8gjW1LDGkfgdRlc4F/C3/5lh1ddLyZFRAshPffwL0Elx0E+ITgUn4HMLMGQGt3nwfcBhQATVLUbwZ8aMHP5quSyg+1XwA8uBLcdjP7P2H7PDNrkuJ9TyOYAvtXgqRQFtvWMAmcQ+qLrGwAOphZ43D00Tssfw84zsy6htttbOlfPEhiQIlA6ix3fxe4n2BKYxnBFNAxBGu2v25mpcATwIiwyW+BJ1PsLG4I/D7sYynBdV0/AZ4jSDRvm1l3gmvWzgb+RLAibJmK9TLtN9nlwC1hmzcIkkfF9/1OWL7O3cvWl38a6G5mJQT7DVanaLeOYHrqXYJli8sub/g5cDHwiJm9A7wNfL9ie4kvrT4qIhJzGhGIiMScEoGISMwpEYiIxJwSgYhIzCkRiIjEnBKBiEjMKRGIiMScEoGISMz9f+vCOiw7avscAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sigma_median = gaussian_kernel_median_heuristic(np.vstack((X,Y)))\n",
    "print(\"median bw/sigma:\", 2*(sigma_median**2), \"/\", sigma_median)\n",
    "\n",
    "# IMPLEMENT: visualize test using median heuristic\n",
    "my_kernel = lambda X,Y : gauss_kernel(X,Y,sigma=sigma_median)\n",
    "my_mmd = lambda X,Y : quadratic_time_mmd(X,Y, my_kernel)\n",
    "\n",
    "statistics = two_sample_permutation_test(my_mmd, X, Y, num_permutations)\n",
    "my_statistic = my_mmd(X,Y)\n",
    "\n",
    "plot_permutation_samples(statistics, my_statistic)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Somehow, the median distance does not really represent the scale on which the two distribution are differing here (they clearly do!).\n",
    "We have figure out how to fix that though."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using cross-validation to select a single kernel parameter\n",
    "\n",
    "As a modern ML scientist, your first reflex might be to just try a bunch of different kernel parameters, and compute a cross-validation estimate of the test power.\n",
    "This involves computing the test for the \"training partitions\" of every fold, and average the number of rejections.\n",
    "Let's start with that.\n",
    "Note that we have to use a different \"training\" dataset to learn the kernel -- otherwise the test would over-fit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/heiko/software/anaconda3/envs/ds3_kernels/lib/python3.6/site-packages/sklearn/cross_validation.py:41: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.\n",
      "  \"This module will be removed in 0.20.\", DeprecationWarning)\n"
     ]
    }
   ],
   "source": [
    "# Let's start with few values\n",
    "log_sigmas = np.array([-5, 0., 10])\n",
    "\n",
    "num_folds = 5\n",
    "data = np.load(\"blobs2.npz\")\n",
    "X_learn = data[\"X\"]\n",
    "Y_learn = data[\"Y\"]\n",
    "\n",
    "from sklearn.cross_validation import KFold\n",
    "\n",
    "test_powers = np.zeros(len(log_sigmas))\n",
    "np.random.seed(0)\n",
    "\n",
    "for i, log_sigma in tqdm(list(enumerate(log_sigmas))):\n",
    "    # IMPLEMENT: define kernel and mmd\n",
    "    my_kernel = lambda X,Y : gauss_kernel(X,Y,sigma=10**log_sigma)\n",
    "    my_mmd = lambda X,Y : quadratic_time_mmd(X,Y, my_kernel)\n",
    "\n",
    "    rejects = []\n",
    "    for train, _ in  KFold(len(X_learn), n_folds=5, shuffle=True):\n",
    "        # IMPLEMENT: split data into train and test folds\n",
    "        X_learn_train = X_learn[train]\n",
    "        Y_learn_train = Y_learn[train]\n",
    "        \n",
    "        # IMPLEMENT: compute test statistic and null distribution, use prog_bar=False\n",
    "        statistics = two_sample_permutation_test(my_mmd, X_learn_train, Y_learn_train, num_permutations, prog_bar=False)\n",
    "        my_statistic = my_mmd(X_learn_train, Y_learn_train)\n",
    "                \n",
    "        # IMPLEMENT: compute p-value as the quantile of the test statistic in the null distribution\n",
    "        p_value = np.mean(my_statistic <= np.sort(statistics))\n",
    "        rejects += [p_value<0.05]\n",
    "    \n",
    "    test_powers[i] = np.mean(rejects)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd4VGXax/HvPZOEkIReQpUaFKSHEqwgoLgWXJUqICIEXbGxRd3ivuu+6767rigqqxQpFgzqrrusDQGxE0pQlCJJCL13SIC0ud8/ZsjGGJIBcnImyf25rrmYc+Y5Z34MYe48pzyPqCrGGGMMgMftAMYYY0KHFQVjjDEFrCgYY4wpYEXBGGNMASsKxhhjClhRMMYYU8CKgjHGmAJWFIwxxhSwomCMMaZAmNsBzlX9+vW1ZcuWbscokJWVRXR0tNsxzirU80HoZwz1fBD6GUM9H1T+jCkpKQdVtUGpDVW1Qj3i4+M1lCxbtsztCCUK9XyqoZ8x1POphn7GUM+nWvkzAqs1iO9YO3xkjDGmgBUFY4wxBawoGGOMKWBFwRhjTAErCsYYYwo4WhREZJCIbBKRdBF59CxthorIBhFZLyLzncxjjDGmZI7dpyAiXmAaMBDYCawSkYWquqFQmzjgMeByVT0iIg2dymOMMaZ0TvYUegHpqpqhqjlAEjC4SJsJwDRVPQKgqvsdzGNC0Pvf7WH/SZ/bMYwxAaIOzdEsIrcDg1R1fGB5NNBbVScVavMvIBW4HPAC/6OqHxazr0QgESA2NjY+KSnJkcznIzMzk5iYGLdjnFUo50s7ks+fVpymRrjySK8omtUIzVNcofwZnhHqGUM9H1T+jP369UtR1R6ltXNymAspZl3RChQGxAF9gWbA5yLSUVWP/mAj1RnADIAePXpo3759yzzs+frkk08IpTxFhXK+6TOSqR/jIz83l6e/zuPVu3vTsWktt2P9SCh/hmeEesZQzweW8QwnfzXbCTQvtNwM2F1Mm3+raq6qbgE24S8SppL7Kv0gyzMO8bO+bXmsdyRREWGMnJnM2h1HS9/YGOMYJ4vCKiBORFqJSAQwHFhYpM2/gH4AIlIfaAdkOJjJhABV5enFqTSqGcnI3hfRMMrDgokJ1I6KYNSsFaRsO+x2RGOqLMeKgqrmAZOARcBG4E1VXS8iT4jIzYFmi4BDIrIBWAb8UlUPOZXJhIZPUw+Qsu0Ik65pS2S4F4BmdaJYMDGB+jWqMebllazIsB8DY9zg6Jk9VX1fVdupahtV/VNg3eOqujDwXFV1sqp2UNVOqho6Z5CNI1SVKYtTaVanOkN7NP/Ba41rVWdBYgKNa1dn7JxVfJl+0KWUxlRdoXm5h6m0Fm/Yx7c7j/FA/zgiwn7849ewZiRJiQm0qBfFuLmr+DT1gAspjam6rCiYcuPz+XsJrepHc2u3pmdtVz+mGm9MSKBtwxgmzFvNkg37yjGlMVWbFQVTbj5Yt5fv957gwf5xhHlL/tGrEx3B/PEJtG9cg3teS+HDdXvKKaUxVZsVBVMu8n3KM0tSiWsYw01dmgS1Ta2ocF4d35suzWtz3/yv+c/aolc0G2PKmhUFUy4Wrt1F+v5MHh7YDq+nuPsai1czMpx543oR36IODyZ9zT/X7HQwpTHGioJxXG6+j2eXpNGhcU0GXdronLePqRbG3Lt60qdNPX7+1lreXLXDgZTGGLCiYMrBP9fsZNuhk0we2A7POfQSCouKCOPlO3tyVVwDfvWPb3k1eVsZpzTGgBUF47DsvHyeW5pOl+a16d/+wkZGjwz3MmNMPAPaN+R3/1rH7C+2lFFKY8wZVhSMo95ctYNdR0/x84HtEDm/XkJh1cK8/P2OeK7v2Ign3t3A9E83l0FKY8wZVhSMY07n5vPCsnR6tqzDlXH1y2y/EWEenh/RjZu6NOHPH3zP80vTymzfxlR1Tg6dbaq415K3se94NlOHdyuTXkJhYV4Pzw7rSrhXeHpxKrn5Ph4uo96IMVWZFQXjiJM5ebz06WYub1uPhNb1HHkPr0f42+1diPB6eO7jdLLzfTw66BIrDMZcACsKxhHzvtrGwcwcpg+82NH38XiEJ3/aiTCvMP3TDHLzlN/d2N4KgzHnyYqCKXMnTucy/bPN9Lu4AfEt6jj+fh6P8MfBHYnwepn95RZy83384eZLz/vyV2OqMisKpszN/mIrR0/mMtnhXkJhIsLvbmxPRJiHlz7dTG6+jyd/2skKgzHnyIqCKVNHT+Yw6/MMrrs0lk7Nyne+ZRHhkUEXExHm4bmlaeTk+3jq9i7nNKyGMVWdFQVTpmZ+nkFmTh4PD2znyvuLCJMHtiPcc+aqJGXK0C6ElzIqqzHGz4qCKTOHMrOZ8+VWbujUmEsa1XQ1y/2BSXz+/MH35OX7mDq8W7GT+hhjfsj+l5gy89Knmzmdm89DA9zpJRQ18eo2PH5jBz5Yt5efvZ5Cdl6+25GMCXlWFEyZ2H/8NK8s38Yt3ZrStmGM23EKjLuiFf97S0eWbNxP4ispnM61wmBMSawomDLx9082k+9THuwf53aUHxmV0IK/3taZz9IOcPe8VZzMyXM7kjEhy4qCuWC7jp5i/ortDOnRjBb1ot2OU6yhPZvz9JAuLN98iLFzVpGZbYXBmOJYUTAX7IWP0wGYdE3o9RIKu7V7M6YO70bKtiOMeXkFx0/nuh3JmJBjRcFckO2HTvLW6h2M6NWcprWrux2nVDd1acK0kd34btcxRs9awbGTVhiMKczRoiAig0Rkk4iki8ijxbw+VkQOiMg3gcd4J/OYsjd1aRpej3Bfv7ZuRwnaoI6NeWlUPBv3nGDkrGQOZ+W4HcmYkOFYURARLzANuB7oAIwQkQ7FNF2gql0Dj1lO5TFlb/OBTN75eidj+rSgYc1It+Ock/7tY5l5Zw/S92cycmYyBzOz3Y5kTEhwsqfQC0hX1QxVzQGSgMEOvp8pZ88uSSMy3Ms9V7dxO8p5ubpdA2aP7cnWQ1kMn5HM/uOn3Y5kjOtEVZ3ZscjtwCBVHR9YHg30VtVJhdqMBf4MHABSgYdVdUcx+0oEEgFiY2Pjk5KSHMl8PjIzM4mJCZ3r8otyKt+OEz4e//IUN7QO5/Z2ERe0L7c/w02H83km5TS1qgmP9IqkbuQPf1dyO18wQj1jqOeDyp+xX79+Karao9SGqurIAxgCzCq0PBp4vkibekC1wPN7gI9L2298fLyGkmXLlrkdoURO5Ut8ZZV2fPxDPZKVfcH7CoXPcPXWw9rx8Q/1yr98rDsOZ/3gtVDIV5pQzxjq+VQrf0ZgtQbx3e3k4aOdQPNCy82A3UUK0iFVPXMwdyYQ72AeU0bW7TrGovX7uPvKVtSOurBeQqiIb1GH18b35ujJHIZNT2bboSy3IxnjCieLwiogTkRaiUgEMBxYWLiBiDQutHgzsNHBPKaMTFmcSu2ocMZd0crtKGWqS/PazJ+QwMmcPIZNT2bzgUy3IxlT7hwrCqqaB0wCFuH/sn9TVdeLyBMicnOg2QMisl5E1gIPAGOdymPKRsq2I3z8/X4Sr2pNzchwt+OUuY5Na/FGYgJ5Ph/DZySTtu+E25GMKVeO3qegqu+rajtVbaOqfwqse1xVFwaeP6aql6pqF1Xtp6rfO5nHXLhnFqdSLzqCO/u0dDuKYy5pVJOkxAQEGD4jmR0nfG5HMqbc2B3NJmjJGYf4Iv0g9/ZtQ3S1yj0VR9uGNVgwsQ8RYR7+b+Up1u065nYkY8qFFQUTFFVlykepxNasxqiEFm7HKRet6kfz5sQ+RHqFkTOT+WbHUbcjGeM4KwomKF+kH2Tl1sPc168tkeFet+OUm+Z1o3isdyS1oyIYNWsFq7cedjuSMY6yomBKpar87aNUmtauzrCezUvfoJKpX93DmxP70LBGNcbMXklyxiG3IxnjGCsKplQff7+ftTuOcv81bakWVnV6CYU1qhVJ0sQEmtauztg5K/ki7aDbkYxxhBUFUyKfT3n6o1Ra1IvitvhmbsdxVcMakSQlJtCyXjTj5q1i2ab9bkcypsxZUTAlWrR+Lxv2HOfB/nGEe+3HpV5MNd6YkEC72BgmvpLC4g373I5kTJmy/+XmrPJ9yjNLUmnTIJrBXZu6HSdk1ImO4PXxCbRvUpN7X0vhg+/2uB3JmDJjRcGc1bvf7iZ1XyYPD2yH1yNuxwkptaqH89rdvejavDaT3viaf3+zy+1IxpQJKwqmWHn5Pp5dksYljWrwk46NS9+gCqoRGc68cb3o2bIODy/4hrdTdrodyZgLFlRREJEEERkTeF5PRC5yNpZx2ztf72LLwSwmD2yHx3oJZxVdLYw5Y3txWZv6/PLttSSt3O52JGMuSKlFQUR+C/we+G1gVSQw38lQxl05eT6mLk2jc7NaDOwQ63ackFc9wsusO3twdbsGPPrP73hl+Va3Ixlz3oLpKdwO/ATIAlDVXUBNJ0MZd72VsoOdR07x8MB2iFgvIRiR4V6mj45nYIdYHv/3emZ9nuF2JGPOSzBFITswa48CiEiUs5GMm07n5vP80nTiW9Shb7sGbsepUKqFefn7Hd35SadG/O97G3nxk81uRzLmnAUz1OU/RWQaUEtE7gLuBuY4G8u45Y2V29l7/DRThnaxXsJ5CPd6eG54N8K9a/nLh9+Tm+/jgf5xbscyJmilFgVV/YuIXA/kAF2AP6nqB44nM+XuVE4+05Ztpk/relzWtr7bcSqsMK+HKUO7EubxMGVxKjl5Pn5+rR2KMxVDqUVBRJ5U1V8DHxSzzlQiryzfysHMbF4a1d3tKBWe1yM8dXtnIsKEF5alk5Pv47HrL7HCYEJeMOcUBhWz7oayDmLclZmdx0ufbuaqdg3o0bKu23EqBY9H+NMtnRjTpwUzPsvgD//ZgP/0nDGh66w9BRGZCNwDtBORNYVeqgGkOB3MlK85X2zhyMlcfj6wndtRKhWPR/jDzZcS4fUw64st5Ob7+OPgjnbvhwlZJR0+ehNYCvwZeLTQ+hOqasNDViLHTuUy8/MMBrSPpUvz2m7HqXREhN/c0J7wMA8vfrKZ3Hwff761sw0dYkLSWYuCqh4BjgBDAESkLv4b18JEpImq7i6fiMZpL3+ewfHTeUy2XoJjRIRfXXcxEV4PU5emkZuvPHV7Z8Js5FkTYoI50fwT4FmgGXAIaAKkAZc4G82Uh8NZObz8xRZu6NSYDk3snkQniQgPD2xHRJiHpxZtIjffxzPDutqQ5CakBHOfwpPA5cBHqtpNRAYCtzkby5SX6Z9t5mRuPg8NsGvpy8t9/doS4fXwp/c3kpvv4/kR3YkIs8JgQkMwP4l5qnoA8IiIqOpiIKhrFkVkkIhsEpF0EXm0hHa3i4iKSI8gc5sysP/EaeZ9tZVbujYlLraG23GqlAlXteZ/burAovX7uPe1FE7n5rsdyRgguKJwTESigS+AV0TkacBX2kYi4gWmAdcDHYARItKhmHY1gAeAFecS3Fw4/0lP5UG749YVYy9vxZ9+2pGl3+9nwiurrTCYkBBMUbgFyAYeAj4BdgE3BbFdLyBdVTNUNQdIAgYX0+6PwF+B08EENmVjz7FTvL5iO7d3b0bL+tFux6my7ujdgr/e3pkv0g9y15xVnMzJczuSqeJKLQqqekJV84Bw4C1gFv4iUZqmwI5CyzsD6wqISDeguaq+G3RiUyZe+DgdVeX+/m3djlLlDe3RnClDu7BiyyHGzl5FZrYVBuMeKe0OSxEZj/+3+Xz8h40EUFUtcaIdERkCXKeq4wPLo4Feqnp/YNkDfAyMVdWtIvIJ8AtVXV3MvhKBRIDY2Nj4pKSkc/pLOikzM5OYmBi3Y5xVcfkOnPTx6OenuLpZGGMureZSsv+qiJ+hE1buyeOlb7NpXcvD5PhIosKDv4/BPsMLV9kz9uvXL0VVSz9vq6olPvBfftqwtHbFbNcHWFRo+THgsULLtYCDwNbA4zSwG+hR0n7j4+M1lCxbtsztCCUqLt8v3vxG437zvu45eqr8AxWjIn6GTvnguz3a9tfv6U3Pf65HsrKD3s4+wwtX2TMCqzWI7+5gzilkAMfPqST5rQLiRKSViEQAw4GFhYrRMVWtr6otVbUlkAzcrMX0FEzZ2XIwi39+vYtRvVvQqFak23FMEYM6NuKlUfF8v+cEI2eu4HBWjtuRTBUTTFF4FPhSRKaJyJQzj9I2Uv95iEnAImAj8KaqrheRJ0Tk5guLbc7X1CWpRHg93Nu3jdtRzFn0bx/LrDt7sPlAJsNnLOfAiWBO4RlTNoIpCi8BXwLfAOsLPUqlqu+rajtVbaOqfwqse1xVFxbTtq/1EpyVuu8E/167mzsva0mDGu6fSzBnd1W7BswZ25Mdh08xfMZy9h23i/NM+QimKPhU9QFVnamqL595OJ7MlLlnl6QSHRHGxKtaux3FBOGytvWZN64Xe4+dZtj05ew+esrtSKYKCKYoLBWRcSLSQERqnnk4nsyUqfW7j/H+d3sZd0Ur6kRHuB3HBKlXq7q8Or43hzJzGDZjOTsOn3Q7kqnkgikKdwJ/ANbw30NH65wMZcreM4tTqRkZxt1XtHI7ijlH3S+qw+sTenP8VB7Dpi9n68EstyOZSiyYm9eaF/Mo8R4FE1q+2XGUJRv3k3hVa2pVD3c7jjkPnZvVZv6E3pzO8zFsxnI2H8h0O5KppM5aFETk6sCfNxf3KL+I5kI9/dEm6kZHMPZy6yVUZJc2qcUbExLI98Gw6cmk7jvhdiRTCZXUUxgY+HNIMY/bHc5lykjqkXw+TzvIPVe3JqZaMCOlm1B2caMaJCUm4BEYPiOZDbvP5xYiY86upJnXfht4+htV3V74NRGxw0cVgKryj9QcGtSoxuiElm7HMWWkbcMYFkzsw8iZyYyYmcxrd/d2O5KpRII50fyvINeZEPPV5kNsOuLjvr5tqB7hdTuOKUOt6kfz5sQ+1IgMY+SsZNKP2rDbpmyUdE6hnYgMBmoVOZ8wCv9czSaEqar/XEKkMLyXdewqo+Z1o1gwsQ91oyP426rTrNp62O1IphIoqadwKf5zB7X54fmEy4CJzkczF+KTTQdYs/0oN7cJJzLcegmVVdPa1XlzYh9qRwp3zl7J8s2H3I5kKriSzim8A7wjIleo6hflmMlcIFVlyuJUmtetzhVNgx9+2VRMsTUjeaxXdaZt8HDX3JXMHNODK+MauB3LVFDBnFO4IXAXc5iILBKRfSIy0vFk5rx9tGEf3+06xoP92xHmsaJQFdSqJrwxIYFW9WO4e95qln2/3+1IpoIKpihcr6rHgRuB/UBH4BFHU5nz5vMpUz5KpXX9aG7p2sTtOKYc1YupxhsTenNxbA0SX13NovV73Y5kKqBgisKZW2B/AryhqgeAkqdrM65577s9bNp3ggcHxBHmDeaf11QmtaMieG18bzo2rcV9r6/hvW/3uB3JVDDBfGu8LyLrgN7AYhGpT3BzNJtylpfv45klqVwcW4ObOlsvoaqqVT2cV+/uTbeLanP/G2v49ze73I5kKpBgxj76JXANEK+qufinzbzV6WDm3P37m91kHMji4YFxeOxcQpUWUy2MeeN60btVPR5a8A1vrd7hdiRTQZRaFESkOjAOeD6wqhHQ2clQ5tzl5vuYujSNS5vU5LpLG7kdx4SAqIgwZo/tyRVt6/PLt79l/ortpW9kqrxgDh/NDrS7MrC8G3jSsUTmvLydspPth0/y82vbIWK9BONXPcLLzDE9uOaShvz6ne+Y99VWtyOZEBdMUYhT1SeBXABVPQnYt04Iyc7L5/mlaXRtXpt+Fzd0O44JMZHhXl4aFc+1HWL5/cL1zPo8w+1IJoQFUxRyRCSSwBVHItIKyHE0lTknC1btYPex0/zi2outl2CKFRHmYdod3bmhU2P+972NTFuW7nYkE6KCGUv5CeBDoJmIzAOuBu52NJUJ2uncfF74OJ1erepyedt6bscxISzc62Hq8K6Ee4WnFm0iN9/Hg/3j7BcJ8wOlFgVV/VBEUvCPeSTAL1XVbpcMEa8lb2P/iWyeH9HN/nObUoV5PTw9tCthXg/PLkkjJ8/HL6+zHqb5r6BmXQncsPZvh7OYc5SVncffP9nMlXH16d3aegkmOF6P8NfbOhMR5uHvn2wmN9/Hr3/S3gqDAYIsCiY0zf1qK4ezcpg8sJ3bUUwF4/EIf7qlIxFeDzM/30JuvvL7mzpYYTBBnWg+byIySEQ2iUi6iDxazOv3iMh3IvKNiHwhIh2czFOZHD+dy4zPMuh/SUO6XVTH7TimAhIRfn9TByZc2Yq5X23l1++sw+ezEWyquhJ7CiLiBdaoapdz3XFg22n453reCawSkYWquqFQs/mq+lKg/c3AFGDQub5XVfTy51s4diqXh62XYC6AiPDrn7T3X520zH8o6S+3dcZrd8RXWSUWBVXNF5ENItJUVc91AJVeQLqqZgCISBIwGCgoCoHRV8+IxgbaC8qRrBxmf7GF6zs2omPTWm7HMRWciPCLay8mwuvlmSWp5Ob7eHpIFxtQsYoK5pxCfWCjiCwHss6sVNXSxj9qChQecGUn/kH1fkBE7gMmAxH4x1gypZjxeQaZOXnWSzBlRkR4cEAc4WHCXz/cRF6+8uzwroRbYahyRLXkX85FpH9x61V1aSnbDQGuU9XxgeXRQC9Vvf8s7UcG2t9ZzGuJQCJAbGxsfFJSUomZy1NmZiYxMTHl9n7Hs5VffHaS7g293NOl9Kmyyzvf+Qj1jKGeD8o244dbcknalEP3hl7u7VqN8DI4lFTVPkOnXEjGfv36pahqj1IbqmqpD6AZ0C/wPBKIDmKbPsCiQsuPAY+V0N4DHCttv/Hx8RpKli1bVq7v98f/rNdWj76rm/efCKp9eec7H6GeMdTzqZZ9xrlfbtEWj7yrY2ev0FM5eRe8v6r4GTrhQjICqzWI7/tgRkkdBywEZgVWXURw9yysAuJEpJWIRADDA/spvO+4Qos3AGlB7LfK2nf8NK8mb+PW7s1o3SC0f6MxFdudl7XkyZ924pPUA0x4ZTWncvLdjmTKSTAHDB8AEoDjAKqaCsSWtpGq5gGTgEXARuBNVV0vIk8ErjQCmCQi60XkG/znFX506Mj817Rl6eT7lAf7x5Xe2JgLNLL3Rfz1ts58kX6Qu+auJCs7z+1IphwEc6L5tKrmnLmpJXCpaVBU9X3g/SLrHi/0/MFg91XV7TxykjdWbmdoz+Y0rxvldhxTRQzp0ZyIMA+T31zL2DkrmT22JzUiw0vf0FRYwfQUvhSRXwGRItIPWAC862wsU9QLH6cjCJP6tXU7iqliBndtynPDu/H19qOMfnklx07luh3JOCiYovAr4ATwPfAgsBT4jZOhzA9tPZjFWyk7Gdn7IprUru52HFMF3dC5MX+/ozvrdx9j1KwVHD1po+dXVsEUhcuB2ar6U1W9RVVfVFWf08HMfz23NI1wr/Czfm3cjmKqsGsvbcSM0T3YtO8Ew2ckcygz2+1IxgHBFIV7gPWBsYmeFJHrRaSm08GMX/r+E/zrm12M6dOShjVKvy/BGCf1u6QhL9/Zg62Hshg+I5n9J067HcmUsVKLgqqOVNW2wB3APmA6cMjpYMbvmSVpVA/3MvGq1m5HMQaAK+MaMGdsL3YdPcXwGcnsPWaFoTIJ5j6F4SIyDZgPXAe8hA1HUS427jnOe9/u4a7LW1EvpprbcYwp0KdNPeaN68X+49kMm7GcXUdPuR3JlJFgDh+9CPTAXwwSVfVJVf3c2VgG4JnFqdSIDGPCldZLMKGnZ8u6vHp3Lw5n5TBs+nJ2HD7pdiRTBoIpCnXxjztUG3haRJJFZI6zscy3O4/y0YZ9TLiyNbWi7LpwE5q6XVSH+eMTyMzOY+j05Ww5mFX6RiakBVMUooCG+O9ibgTUwz+iqXHQlMWp1I4K567LW7odxZgSdWpWi/njE8jO8zFs+nLS92e6HclcgGCKwkpgCJAKjFXVOFW9w9lYVVvKtsN8sukA91zdxu4eNRVChyY1SUpMwKcwfMZyNu094XYkc56CufroUlVNBN4C9jofyTz9USr1YyIY06eF21GMCVq72BosmJiA1yMMn7Gc9buPuR3JnIdgrj7qICKr8I9gullEVthcys75avNBvtp8iJ/1bUtURDBDUxkTOto0iGFBYh+qh3sZOXMF3+486nYkc46COXw0A/i1qjZT1Sb4h7iY4WysqklVmfJRKo1qRjKy90VuxzHmvLSsH82CiX2oWT2MO2auIGXbEbcjmXMQTFGooaqLzyyo6hKghnORqq7P0g6yetsR7rumLZHhQQ9Ga0zIaV43igWJfagXE8GYl1ew6bDNx1BRBFMUtorIYyLSLPB4FNjmdLCqxt9L2ETT2tUZ1qO523GMuWBNaldnwcQ+NKoVydMpp/kq/aDbkUwQgikK44Dm+OdF+CDw/C4nQ1VFSzbuZ+3OYzzYP46IMJss3VQOsTUjSUrsQ8Pqwl1zV/Fp6gG3I5lSBHP10SFV/Rn+OZd7qep9qmpjH5Uhn0+ZsjiVlvWiuLV7U7fjGFOmGtSoxiO9qtOmQQwT5q1m6cZ9bkcyJQjm6qPuIvI1sAlIE5EUEenufLSq44N1e9m45zgPDWhHmNd6CabyqREhzJ/Qm0sa1+Ce11L4cJ1d3R6qgvkGmgNMDlx91Az4eWCdKQP5PuWZJanENYzhpi5N3I5jjGNqR0Xw2vjedGpai/vmr+Hdb3e7HckUI5iikKWqy84sqOongN3HXkb+s3Y36fszeXhgO7wecTuOMY6qGRnOK3f3Jv6iOjzwxte88/VOtyOZIoIpCitEZJqIXCEil4vIc8AyEeksIp2dDliZ5eX7eHZJKu0b12TQpY3cjmNMuYipFsbccT1JaF2PyW+u5c1VO9yOZAoJ5pbZHoE/ixaAqwEFrirTRFXIP9fsYuuhk8wa0wOP9RJMFRIVEcbssT1JfDWFX/3jW3LyfYxKsGFdQkGpRUFVryyPIFVNTp6PqUvT6NK8Nv3bN3Q7jjHlLjLcy4zR8dz3+hp++6915Ob7uOvyVm7HqvLsUheXLFi9g11HTzF5YDtErJdgqqbIcC8vjornuktj+cN/NjDjs81uR6ryHC0KIjJIRDaJSHrgTuiir08WkQ0i8q2ILBUZGKNUAAAXNElEQVSRKtF/PJ2bzwsfp9GzZR2uiqvvdhxjXBUR5uGFkd25sXNjnnz/e174OM3tSFVaMPcp/OgQU3HrimnjBaYB1wMdgBHFjK76NdBDVTsDbwN/DSZ0Rff6iu3sO57N5IEXWy/BGCDc6+HZYV25tVtT/vZRKlMWp6KqbseqkoKdZCeYdUX1AtJVNUNVc4AkYHDhBqq6TFXPTOyaDDQLYr8V2smcPF78JJ3L2tSjT5t6bscxJmSEeT08NaQLQ3s047mlafx10SYrDC4462/8ItIQaAxUF5FOwJlfaWvin6KzNE2Bwtea7QR6l9D+bvxjK1Vq877axsHMHKaPbud2FGNCjtcj/N+tnQn3enjxk83k5Pn47Q3trUddjuRslVhE7sI/GF5X/Id5zvyrnADmqOpbJe5YZAhwnaqODyyPxj920v3FtB0FTAKuVtXsYl5PBBIBYmNj45OSkoL725WDzMxMYmJigmp7Kk/5xacnaVPLy+QekQ4n8zuXfG4J9Yyhng9CP+O55lNV5n+fw+JtefS/KIw72kfgcbgwhPpnCBeWsV+/fimq2qPUhqpa4gMYWlqbs2zXB1hUaPkx4LFi2g0ANgINg9lvfHy8hpJly5YF3XbqklRt8ci7+u2Oo84FKuJc8rkl1DOGej7V0M94Pvl8Pp8++d4GbfHIu/rI22s1P99X9sEKCfXPUPXCMgKrNYjv2GDOKTQUkZoAIvKSiKwUkf5BbLcKiBORViISAQwHFhZuICLdgOnAzaq6P4h9VljHTuYy8/MMru0QS6dmtdyOY0zIExEevf4S7r+mLUmrdvDLt78l32fnGJwWzB3Niar6gohci/9E8L34p+OML2kjVc0TkUnAIsALzFbV9SLyBP6KtRB4CogB3gocM9yuqjef/18ndM38PIMTp/N4eKCdSzAmWCLCz6+9mHCvhymLU8nN9zFlaBcbTdhBwRSFM6X5evznElJEJKh/EVV9H//kPIXXPV7o+YBgg1ZkhzKzmfPlFm7s3Jj2jWu6HceYCueB/nGEez385cPvyfP5mDq8G+FWGBwRTFFYKyLvA+2A34hIDP8tFCYI0z/L4FRuPg8NsF6CMefr3r5tCPcK//veRnLy1jDtjm5UC7O5zMtaMKX2LuB/8F85dBKIxH/5qAnC/uOneWX5Vm7p1pS2DUP7ygZjQt34K1vzx8GXsmTjPia+msLp3Hy3I1U6wUzHmQ+0xn8uAaB6MNsZv79/spncfOXB/nFuRzGmUhjdpyX/d2snPk09wPh5qzmVY4WhLAUzzMULQD9gVGBVFvCSk6Eqi91HTzF/xXaGxDejRb1ot+MYU2kM73URf7u9C19tPsjYOSvJys5zO1KlEcxv/Jep6kTgNICqHgYiHE1VSbywLB2A+62XYEyZuy2+Gc8M68rqbUe4c/ZKTpzOdTtSpRBMUcgNXG2kACJSD/A5mqoS2H7oJG+u2sHwXs1pWru623GMqZQGd23KCyO68c2Oo4x6eSXHTlphuFBnLQqFRkKdBvwDaCAifwC+AP5SDtkqtOc+TsPrEe7r19btKMZUatd3asyLo+LZuPs4I2clcyQrx+1IFVpJPYWVAKr6CvBb4G/AEWCIqobO4EMhaPOBTP65ZiejE1oQW7N8xjgypiob2CGWGWPiSdufyYiZyRzM/NEQaiZIJRWFgtGnVHW9qk5V1WdVdV055KrQpi5JIzLcyz1927gdxZgqo+/FDZl9Z0+2HspixIxk9h8/7XakCqmkm9caiMjks72oqlMcyFPhbdp7gv98u5t7r25D/Zhqbscxpkq5Iq4+c+/qxbi5qxg+I5n5ExJoVMt66+eipJ6CF/+4RDXO8jDFeGZxKjERYSRe1drtKMZUSQmt6/HKuF7sP5HN0OnL2XnkZOkbmQIl9RT2qOoT5ZakEli36xgfrt/LQwPiqB1lV+0a45YeLevy2vjejHl5BcOmJ/PGhAQuqhfM3GAmqHMKJjhTFqdSq3o4465o5XYUY6q8rs1rM39CAlk5eQydvpwtB7PcjlQhlFQUgpkzwQSs2X6Ej7/fT+JVrakZGe52HGMM0LFpLd6YkEBuvo+h05eTvv+E25FC3lmLQuDOZROkZxanUi86grGXtXQ7ijGmkPaNa5KUmADAsOnJfL/3uMuJQpsNbFcGVmQc4vO0g9zbtw3R1YIZjdwYU57iYmuwIDGBcK+HETOSWbfrmNuRQpYVhQukqjy9OJWGNaoxKqGF23GMMWfRukEMCyYmEBURxsiZyXyz46jbkUKSFYULtOGQj5VbDjPpmrZEhtuEH8aEshb1olkwMYFaUeGMmrWClG12lLwoKwoXQFX5R1oOTWpFMqxnc7fjGGOC0KxOFG9O7EODGtUY/fJKVmQccjtSSLGicAGWbdpPxjEfD/SPs2kBjalAGteqzoLEBJrUrs6dc1byZfpBtyOFDCsK50lVefqjVBpUF26Lb+Z2HGPMOWpYM5KkxARa1otm3NxVfHvAJuoBKwrnbdH6vazffZxb2oYT7rWP0ZiKqH5MNd6YkEDbhjE8tyabJRv2uR3JdfZtdh7yfcqUxam0aRBNnyZ2CaoxFVmd6Ajmj0+geU0P97yWwofr9rgdyVVWFM7Du9/uJnVfJg8NaIdHbDQQYyq6WlHh/LJHJF2a1+a++V+zcO1utyO5xtGiICKDRGSTiKSLyKPFvH6ViKwRkTwRud3JLGUlL9/H1CVpXNKoBjd0aux2HGNMGYkKF+aN60V8izo8lPQ1/0jZ6XYkVzhWFETEi38qz+uBDsAIEelQpNl2YCww36kcZe2dr3eRcTCLhwe2w+OxXoIxlUlMtTDm3tWTPm3q8Yu317Jg1Xa3I5U7J3sKvYB0Vc1Q1RwgCRhcuIGqblXVbwGfgznKTG6+j+c+TqNT01pc2yHW7TjGGAdERYTx8p09uSquAY/84zteTd7mdqRy5WRRaArsKLS8M7Cuwnpr9U52HD7F5GvbIXYuwZhKKzLcy4wx8Qxo35Df/Wsds7/Y4nakciOq6syORYYA16nq+MDyaKCXqt5fTNu5wLuq+vZZ9pUIJALExsbGJyUlOZK5JDn5yqOfn6JupPCb3pEFRSEzM5OYmJhyzxOsUM8HoZ8x1PNB6GcM9XxQfMY8n/LS2mxW78tnaLtwftLa3cmzLuRz7NevX4qq9ii1oao68gD6AIsKLT8GPHaWtnOB24PZb3x8vLphzhcZ2uKRd/XLtAM/WL9s2TJX8gQr1POphn7GUM+nGvoZQz2f6tkz5ubl66T5a7TFI+/q1CWp5RuqiAv5HIHVGsR3rJMX2a8C4kSkFbALGA6MdPD9HHMqJ58Xlm0moXVdLmtb3+04xphyFOb18OywroR7hCmLU8nN9zF5YOU9hOxYUVDVPBGZBCwCvMBsVV0vIk/gr1gLRaQn8A5QB7hJRP6gqpc6lel8vZq8lYOZ2bw4qrvbUYwxLvB6hKeGdCHc6+H5j9PJyffx6KBLKmVhcPR2XFV9H3i/yLrHCz1fBYT0wEGZ2Xm89GkGV7VrQM+Wdd2OY4xxidcj/PnWToSHCdM/zSAnz8fjN3aodIXBxmgoxdwvt3A4K4fJA9u5HcUY4zKPR/jj4I5EeL3M/nILufk+nri5Y6W6Z8mKQgmOncplxmcZDGgfS9fmtd2OY4wJASLC725sT0SYh5c+3UxunvLkrZ3wVpLCYEWhBC9/nsHx03nWSzDG/ICI8Migi4nwCs99nE5uvo+/3t6ZsEowYrIVhbM4kpXD7C+38pNOjejQpKbbcYwxIUZEmHztxYR7PTy9OJVcnzJlaJcKP5S+FYWzmP5ZBlk5eTw8wHoJxpizu79/HBFhHv78wffk5vl4bkQ3IsIqbmGouMkddOBENvO+2srgLk2Ii63hdhxjTIibeHUbHr+xAx+u38vPXk8hOy/f7UjnzYpCMV78ZDM5+T4etF6CMSZI465oxR9v6ciSjfuZ8EoKp3MrZmGwolDE3mOneW3FNm7r3pRW9aPdjmOMqUBGJ7TgL7d14vO0A4ybu4qTORVv3mcrCkW8sCwNVeX+a+LcjmKMqYCG9byIp4d0ITnjEGPnrCIzu2IVBisKhew4fJIFq3YwrGdzmteNcjuOMaaCurV7M6YO70bKtiOMeXkFx0/nuh0paFYUCnn+4zREhEn9rJdgjLkwN3VpwrSR3fhu1zFGz1rBsZMVozBYUQjYejCLf6zZxajeLWhUK9LtOMaYSmBQx8a8eEc8G/ecYMTMZA5n5bgdqVRWFAKmLk0jwuvh3r5t3I5ijKlEBnSIZcaYeDYfyGTEjGQOZma7HalEVhSAtH0n+Nc3uxhzWQsa1KjmdhxjTCXT9+KGzB7bk22Hsxg+I5n9x0+7HemsrCgAzy5JIzoijHuusl6CMcYZl7etz7y7erHn6CmGzUhmz7FTbkcqVpUvCut3H+O97/Yw7vKW1Il2d/5VY0zl1rt1PV65uzcHT2QzdPpydhw+6XakH6nyReGZxWnUjAzj7itbux3FGFMFxLeow2vje3PsZC7DZySz7VCW25F+oEoXhbU7jrJk4z4Sr2pNrerhbscxxlQRXZrXZv6EBE7m5DF0+nI2H8h0O1KBKl0Unl6cSp2ocMZe3srtKMaYKqZj01q8kZhAvk8ZNj2ZtH0n3I4EVOGisGrrYT5LPcC9fdsQU81GEDfGlL9LGtUkKTEBj8DwGcls3HPc7UhVtyg8/dEmGtSoxuiElm5HMcZUYW0b1mDBxD5EhHkYMTOZdbuOuZqnShaFr9IPkpxxmJ/1bUP1CK/bcYwxVVyr+tEsSOxDdEQYI2Ym8/X2I65lqXJFQVV5enEqjWtFMqLXRW7HMcYYAC6qF8WCiQnUiYpg9MsrWb31sCs5qlxR+CT1ACnbjjDpmrZEhlsvwRgTOprVieLNiX1oWKMaY2avZPnmQ+WewdGiICKDRGSTiKSLyKPFvF5NRBYEXl8hIi2dzKOqTPkoleZ1qzMkvrmTb2WMMeelUa1IkiYm0LR2de6au5Iv0g6W6/s7VhRExAtMA64HOgAjRKRDkWZ3A0dUtS3wDPAXp/IAfLRhH9/tOsYD18RV6Im1jTGVW8MakSQlJtCyXjTj5q1i2ff7y+29nfxm7AWkq2qGquYAScDgIm0GA/MCz98G+ouIOBHG51OeWZxK6/rR/LRbUyfewhhjyky9mGq8MSGBdrExJL66mo/W7y2X93WyKDQFdhRa3hlYV2wbVc0DjgH1nAjz/ro9fL/3BA8OiCPMa70EY0zoqxMdwevjE+jQpBY/e30Nq/c6P7WnqKozOxYZAlynquMDy6OBXqp6f6E26wNtdgaWNwfaHCqyr0QgESA2NjY+KSnpnPOsPZDHpzvymNStGp4y7IxkZmYSExNTZvsra6GeD0I/Y6jng9DPGOr5ILQznspTXlqbzXVN8+jQ6Pwy9uvXL0VVe5TaUFUdeQB9gEWFlh8DHivSZhHQJ/A8DDhIoFCd7REfH6+hZNmyZW5HKFGo51MN/Yyhnk819DOGej7Vyp8RWK1BfHc7eRxlFRAnIq1EJAIYDiws0mYhcGfg+e3Ax4HwxhhjXODYoD+qmicik/D3BrzAbFVdLyJP4K9YC4GXgVdFJB04jL9wGGOMcYmjI8Gp6vvA+0XWPV7o+WlgiJMZjDHGBM8uwzHGGFPAioIxxpgCVhSMMcYUsKJgjDGmgBUFY4wxBRy7o9kpInIA2OZ2jkLq47/pLlSFej4I/Yyhng9CP2Oo54PKn7GFqjYorVGFKwqhRkRWazC3jrsk1PNB6GcM9XwQ+hlDPR9YxjPs8JExxpgCVhSMMcYUsKJw4Wa4HaAUoZ4PQj9jqOeD0M8Y6vnAMgJ2TsEYY0wh1lMwxhhTwIpCGRGRX4iIikh9t7MUJSJPicj3IvKtiLwjIrXdzgQgIoNEZJOIpIvIo27nKUpEmovIMhHZKCLrReRBtzMVR0S8IvK1iLzrdpbiiEhtEXk78DO4UUT6uJ2pKBF5OPBvvE5E3hCRSJfzzBaR/SKyrtC6uiKyWETSAn/WceK9rSiUARFpDgwEtrud5SwWAx1VtTOQin/CI1eJiBeYBlwPdABGiEgHd1P9SB7wc1VtDyQA94VgRoAHgY1uhyjBVOBDVb0E6EKIZRWRpsADQA9V7Yh/qH+3h/GfCwwqsu5RYKmqxgFLA8tlzopC2XgG+BUQkidoVPUj9c+BDZAMNHMzT0AvIF1VM1Q1B0gCBruc6QdUdY+qrgk8P4H/y6zoPOOuEpFmwA3ALLezFEdEagJX4Z87BVXNUdWj7qYqVhhQXUTCgChgt5thVPUz/HPMFDYYmBd4Pg+4xYn3tqJwgUTkZmCXqq51O0uQxgEfuB0C/5frjkLLOwmxL9zCRKQl0A1Y4W6SH3kW/y8kPreDnEVr4AAwJ3CIa5aIRLsdqjBV3QX8DX9Pfw9wTFU/cjdVsWJVdQ/4f2EBGjrxJlYUgiAiSwLHGos+BgO/AR4vbR8uZzzT5jf4D4m87l7SAlLMupDsaYlIDPAP4CFVPe52njNE5EZgv6qmuJ2lBGFAd+BFVe0GZOHQYY/zFTg2PxhoBTQBokVklLup3OPozGuVhaoOKG69iHTC/4O0VkTAf1hmjYj0UtW95RjxrBnPEJE7gRuB/iEyD/ZOoHmh5Wa43GUvjoiE4y8Ir6vqP93OU8TlwM0i8hMgEqgpIq+paih9oe0EdqrqmR7W24RYUQAGAFtU9QCAiPwTuAx4zdVUP7ZPRBqr6h4RaQzsd+JNrKdwAVT1O1VtqKotVbUl/v8A3cu7IJRGRAYBjwA3q+pJt/MErALiRKSViETgP7G30OVMPyD+Sv8ysFFVp7idpyhVfUxVmwV+9oYDH4dYQSDwf2GHiFwcWNUf2OBipOJsBxJEJCrwb96fEDsZHrAQuDPw/E7g3068ifUUqoYXgGrA4kCPJllV73EzkKrmicgkYBH+qz1mq+p6NzMV43JgNPCdiHwTWPfrwNzjJnj3A68Hin8GcJfLeX5AVVeIyNvAGvyHV7/G5bubReQNoC9QX0R2Ar8H/g94U0Tuxl/IHJnf3u5oNsYYU8AOHxljjClgRcEYY0wBKwrGGGMKWFEwxhhTwIqCMcaYAlYUTJUhIpnl9D6zQnTgPGNKZZekmipDRDJVNcbtHMaEMuspmCpNRFqIyNLAXBNLReSiwPo2IpIsIqtE5IniehkiEi0i74nI2sA4U8MC6z8RkR6B53eLSGpg3UwReSGwfq6IvBiYryFDRK4OjKG/UUTmFnqPF0VkdWCs/z+Uy4diqjQrCqaqewF4JTDXxOvAc4H1U4GpqtqTs4/JNAjYrapdAuPwf1j4RRFpAvwO/1wMA4FLimxfB7gGeBj4D/4h2C8FOolI10Cb36hqD6AzcLWIdD7vv6kxQbCiYKq6PsD8wPNXgSsKrX8r8Hx+0Y0CvgMGiMhfRORKVT1W5PVewKeqelhVcwvt74z/BAYn/A7YFxhLywesB1oG2gwVkTX4h164FP+ERMY4xoqCMT8U9Ek2VU0F4vF/qf9ZRIoOoV7c8OCFZQf+9BV6fmY5TERaAb/AP7JtZ+A9/KOhGuMYKwqmqvuK/069eAfwReB5MnBb4HmxUzMGDg+dVNXX8E/S0r1Ik5X4D/nUCczodVvRfZSiJv75B46JSCz+qUuNcZSNkmqqkqjAiJNnTME/N+9sEfkl/hnCzozg+RDwmoj8HP9v6EUPDQF0Ap4SER+QC9xb+EVV3SUiT+KfrW03/iGji9tPsVR1rYh8jf9wUgbwZbDbGnO+7JJUY4ohIlHAKVVVERkOjFDVc55DWkRiVDUz0FN4B/8Q4e+UdV5jyor1FIwpXjzwQmDSlaP457Y+H/8jIgPwnwv4CPhXGeUzxhHWUzDGGFPATjQbY4wpYEXBGGNMASsKxhhjClhRMMYYU8CKgjHGmAJWFIwxxhT4fzdMlF+JlOaOAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(log_sigmas, test_powers)\n",
    "plt.xlabel(\"Log sigma\")\n",
    "plt.ylabel(\"Test power estimate\")\n",
    "plt.grid(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This seems to give us an idea of which kernel parameter is better.\n",
    "Unfortunately, this approach is quite costly and noisy.\n",
    "In reality, we would also want to use more folds, repeat the x-validation, and try out more kernel parameters (play with it and you will see).\n",
    "\n",
    "We could try one of the faster approximations for the null distribution mentioned above, for example the moment matching Gamma approach.\n",
    "This would speed up the procedure, however, the Gamma approximation has its limits in how accurate it is, especially for extreme values of the kernel.\n",
    "\n",
    "Instead, we will do something better."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optimal kernel via maximising the ratio of mmd and its variance, see https://arxiv.org/abs/1611.04488"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have seen that estimating the test power using cross-validation is a bit messy; it is also not differentiable.\n",
    "\n",
    "Luckily, there is a closed form criterion that acts as a surrogate for the test power: the MMD divided by its variance.\n",
    "You have seen in the slides that maximising this directly maximises the power of the statistical test -- without ever computing the test.\n",
    "\n",
    "Long story short, you can\n",
    "\n",
    "1. Pick a training set (not the one you use for performing the test, to avoid overfitting)\n",
    "2. Choose the kernel that maximises the value of the MMD, divided by its standard variance\n",
    "3. Enjoy a test that has higher power"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As implementing the variance of the MMD estimator is slightly more complicated, we have provided a TensorFlow implementation for you (so we can later automagically differentiate it).\n",
    "Feel free to examine the imported code in `mmd.py`.\n",
    "\n",
    "(Note that the provided implementation uses a biased MMD statistic, and as such produces slightly different numbers)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MMD and ratio: (0.003497325, 7.8510857)\n"
     ]
    }
   ],
   "source": [
    "# provided implementation\n",
    "import tensorflow as tf\n",
    "import mmd\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    init = tf.global_variables_initializer()\n",
    "    sess.run(init)\n",
    "    print(\"MMD and ratio:\", sess.run(mmd.rbf_mmd2_and_ratio(X,Y, sigma=sigma_median)))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As seen above, the most naive way to select a kernel to to come up with a number of baseline kernels, then simply compute the ratio of MMD and standard deviation for all of them, and finally to pick the kernel with the largest ratio.\n",
    "Remember it is important to do this on a separate \"training\" dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "best sigma: 0.21544346900318834\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEjCAYAAAA/ugbCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8FeXZ//HPlY2EhH0JEPZVVGR1AxdQ3Ddw17rU2lptrVZ9bLXtr619+tRa61qXVutCtRVrW3fFNeBaFQQEBAERlB1ElrCFkOv3x0wwxORkEjLnZPm+X6955Zw5M3O+OeK5MnPfc9/m7oiISNOVluoAIiKSWioEIiJNnAqBiEgTp0IgItLEqRCIiDRxKgQiIk2cCoGISBOnQiAi0sRlJHrRzLoCZwOHAl2ArcBs4HngRXcvjT2hiIjEyqq6s9jMHgIKgOeAqcBqIBvoD4wBhgPXufsbyYkqIiJxSFQI9nX32VXuaJYFdHf3hXGFExGR+FXZRuDus80s3cwereL1YhUBEZGGL2FjsbvvBDqEf/2LiEgjlLCxOLQYeNvMngE2l61091vjCiUiIskTpRAsD5c0oEW8cUREJNmqbCz+xoZmue6+ufotRUSkIan2hjIzO9jMPgbmhs8Hm9k9sScTEZGkiHJn8e3AMcCXAO4+EzgszlAiIpI8kYaYcPcvKqzaGUMWERFJgSiNxV+Y2UjAw26kVxBeJhIRkYav2sZiM2sP3AGMBQx4GbjS3b+MP56IiMQtSiHIdvdtScojIiJJFqUQLARWAW8CbwBvu/uGJGQTEZEkiHQfgZl1JxiKehRwPLDe3YfEnE1ERJKg2sbicE6CUQSFYDAwB3gr5lwiIpIkUS4NlQIfAL9z96eTkkpERJImSiEYDBxCcBNZd2ABMMXdH4g/noiIxC1qG0EeQTE4FDgPcHfvGW80ERFJhihtBFOBZsA7BG0Dh7n7kriDiYhIckS5NNTB3dckKY+IiCRZlLGGis3sVjObGi63mFmr2JOJiEhSRCkEDwKbgDPDZSPwUJyhREQkeaJcGppR8eaxytaJiEjDFOWMYKuZHVL2xMxGAVvjiyQiIskU5YxgCDABaEUw+ug64EJ3/yj+eIH27dt7z549a7Xv5s2byc3NrdtAdUC5aka5aka5aqax5po2bdpad+9Q7YbuHmkBWgIto25fl8vw4cO9tgoLC2u9b5yUq2aUq2aUq2Yaay5gqkf4jo0yZ3E7M7sTmAwUmtkdZtau1iVKRETqlShtBBOBNcBpwOnh48fjDCUiIskTZarKtu7+v+We/9bMxsUVSEREkivKGUGhmZ1tZmnhcibwfNzBREQkOaIUgu8D/wC2h8tE4Goz22RmG+MMJyIi8av20pC7t0hGEBERSY0oZwQiItKIqRBIUi39aguP/HcJxSWlqY4iIqEovYZE9tjKDdu4q3ABj3/wBTt2Os3S0zhz/26pjiUiRDgjMLNHoqwTqczqTdu44dk5HHZzIRPf/4IzRnSje9vmPDl9WaqjiUgoyhnBPuWfmFk6MDyeONJYfFm0nfveWMSEdxezY6dz6tACrjiyH93aNuf2V+dzx2sLWLFhK51b5aQ6qkiTV2UhMLPrgZ8BOeW6iRpQDNyXhGzSAK3fUsz9by7i4bcXs2XHTsYNCQpAr/ZfD5w1bkgBt7+6gGdmLOf7h/dJYVoRgQSFwN1vBG40sxvd/fokZpIGaOO2HTz41mc88OZnbNpewgn7deaqsf3o2/GbvY97ts9laPfWPDl9mQqBSD0Q5dLQc2aW6+6bzew8YBhwh2sCewE2by/h4XcWc98bi9iwdQfH7JPPj8f2Z2Dnlgn3GzekgF89M4d5KzeyV6fE24pIvKJ0H70X2GJmg4GfAEuAv8WaSuq9rcU7ue+NTzn0D4Xc/NInDO/Rhud+dAh/OX9EtUUA4MT9OpOeZjw1fXkS0opIIlHOCErc3c3sFIIzgQfM7MK4g0n9tG3HTv7x3ufcM/lT1hZt59B+7bnqqP4M696mRsdpl9eMw/t34OkZy/jJMQNIS7OYEotIdaIUgk1hw/H5wKFhr6HMeGNJfVNcUsrjU7/g7tcXsnLjNg7q3ZZ7vjWMA3q1rfUxxw0t4PV5q3nvs3Uc3EdTXIikSpRCcBZwLvAdd19pZt2Bm+ONJfXFjp2l/HvaUv70+kKWrd/K8B5tuPXMwYzs236Pj33UwHxys9J5avoyFQKRFIoy6NxKM/s30C9ctRZ4MtZUknI7S52npi/jztcXsOTLLQzu2orfnTqIw/q1x6xuLuPkZKVzzL6deGH2Cm44ZR+yM9Pr5LgiUjPVFgIz+x5wCdAW6AMUAH8Gjow3mqRCqTvPzFzO7a/OZ9GazezduSV/vWAERw7sWGcFoLzxQwv4z4fLKJy3muMGda7z44tI9aJcGvohcADwHoC7LzCzjrGmkqQrLXVemrOS3769lWVF0xmQ34I/nzeMo/fuFGtD7sg+7enQohlPTl+mQiCSIlEKwXZ3Ly77a9DMMgCPNZUkjbvz6tzV3PbKfD5esZFOucad5wzlxEGdk9KTJz3NOHlwFx55dwnrtxTTunlW7O8pIruLch/BFDMrG2riKOAJ4Nl4Y0nc3J3Jn6xm3N1v872/TWVzcQm3njmY/xuVw8mDuyS1O+f4oQUU7yzlhVkrk/aeIvK1KGcE1wEXA7MIpq18wd3vjzWVxOqdhWu55ZX5TFvyFQWtc7jptEGcOqwrmelpTJ68MOl59unSkj4dcnlq+jLOPbB70t9fpKmLUgh+5O53ALu+/M3synCdNAA7S521RduZt3IT905eyH8XraNTy2z+d9y+nDWiG1kZqZ2fyMwYP7SAP748n6VfbaFrm+YpzSPS1EQpBBcCFb/0v13JOkmRjdt2sHz91nDZ9vXjDcHjlRu2UVIaNOu0z2vGr07am3MO6F6vumueMiQoBE/PWM4Px/RNdRyRJiXRMNTnENxI1svMnin3Ugvgy7iDSaC4pJSVG7axbP1WVmwIvuCXrd+26/Hy9dso2l6y2z4ZaUZ+y2wKWucwokcbOrfOoUvrHApaZ3Nw7/bkZNWfAlCmW9vm7N+zDU9OX8YPRveJpauqiFQu0RnBO8AKoD1wS7n1m4CPor5BOCTFVGCZu59oZr2AiQT3JXwInO/uxTUN3hiUljpfbi5mefglvyz8a77847VF2/EKfbTa5WbRuXU2PdvlMrJPe7q0zqZL6xw6t8qhoHUOHVo0I70Bjt1zypACfvHUbOYs38i+Ba1SHUekyUg0H8ESgpFGD97D97gSmAuUDUl5E3Cbu080sz8TNETfu4fvUSfcnZ2lTkmpU+rBz507nZ3h+vJLSYXnwTallFSyfUmpsy78wi+7dPPpyi2sf3XSNyZxz8lM3/XFvteAjsEXfOvgr/vOrYL19emSTl06YVBnbnh2Dk/PWKZCIJJEsU5eb2ZdgROA/wOutuB8/wiCS04AE4BfE1MhuP4/syicvYWs9wsrfGE7JTtLKXUoKS2ltDT8GfPdEWkGnVoGX+a9WqYxZEAPurQKLtt0aZ1Nl1Y5tG6e2WQvi7TJzWL0gI48PWM51x03sEGe1Yg0ROYVrzvU5cHN/gXcSNCu8D8Ejcz/dfe+4evdgBfdfd9K9r2EYGgL8vPzh0+cOLHG7//comI+W1dMVmYGaWakGaQbpKUFN1CkW9BjJd2CL+k0g/TwtWAb2237Xful2dfbl9u34mvl17fIMlo3s11fbkVFReTl5dX4d4pbqnO9v7KEe2Zs59oR2ezT/uszn1Tnqopy1Yxy1cye5hozZsw0dx9R7YbuXu0CZAH7hktmxH1OBO4JH48GngM6AAvLbdMNmFXdsYYPH+61VVhYWOt946RcldtaXOL7/nKSX/PPGbutT3WuqihXzShXzexpLmCqR/i+rrYDuZmNBhYAdwP3APPN7LAIxWgUcLKZLSZoHD4CuB1oHQ5TAdAV0BRVskt2ZjrHDerEpNkr2Vq8M9VxRJqEKHcS3QIc7e6Hu/thwDHAbdXt5O7Xu3tXd+8JnA287u7fAgqB08PNLgSerlVyabTGDSmgaHsJr85dleooIk1ClEKQ6e6flD1x9/ns2QxlPyVoOF4ItAMe2INjSSN0YO92dGqZzVPTl6U6ikiTEKXX0FQzewB4JHz+LWBaTd7E3ScDk8PHiwiGtRapVHqaccqQLjzw1mes21xM21yNSCoSpyhnBJcBc4ArCO4J+Jhg8DmR2IwbWkBJqfP8R2pCEolblEJwqbvf6u6nuvt4d7+NoDiIxGZg55YMyG/Bk7o8JBK7KIXgwkrWfbuOc4h8w7ihBXz4+Xo+/3JLqqOINGpVFgIzO8fMniUcdK7cUogGnZMkOGVIFwCemqGzApE4xT7onEhtdWmdw4G92vLU9GUMGq7ZUUXikoxB50RqbfzQAq77zyw+25jNmFSHEWmkUjs1lUg1jhvUmaz0NN5dXlL9xiJSKyoEUq+1ysnkiL068t6KEkp2lla/g4jUmAqB1HvjhhawsRjeWrg21VFEGqVEU1XOAiproTPA3X2/2FKJlDNmrw40z4CnZyxn9ICOqY4j0ugk6jV0YtJSiCTQLCOdAzplMGn2Sn47roTcZrHOpyTS5FR5acjdlyRakhlS5OAuGWzdsZNXPtaIpCJ1Lcp8BAeZ2QdmVmRmxWa208w2JiOcSJl+bdIoaJ2jm8tEYhClsfgu4ByCyWlygO8Cf4ozlEhFaRaMSPrmgrWs2bQ91XFEGpVIvYbcfSGQ7u473f0h0L09knzjhhaws9R5TiOSitSpKIVgi5llATPM7A9mdhWQG3MukW/on9+CvTu31IQ1InUsSiE4P9zucmAzwYTzp8UZSqQq44cWMHPpBhatKUp1FJFGo9pCEPYS2ubuG939Bne/OrxUJJJ0Jw3ughk8NUOXh0TqSpReQ6PM7BUzm29mi8qWZIQTqahTq2xG9mnHU9OX4a4RSUXqQpRLQw8AtwKHAPuXW0RSYtyQAj5ft4XpX6xPdRSRRiFKIdjg7i+6+2p3/7JsiT2ZSBWO3bcTzTLS1GgsUkeiFIJCM7vZzA42s2FlS+zJRKrQIjuTsXvn8+zM5ezQiKQieyzKoC0Hhj9HlFvnwBF1H0ckmvFDCnj+oxW8MX8NRw7MT3UckQat2kLg7rp5TOqdw/p3oE3zTJ6asVyFQGQPRek11MrMbjWzqeFyi5m1SkY4kapkZaRxwn6deXnOSjZt25HqOCINWpQ2ggcJJqw/M1w2Ag/FGUokivFDC9heUspLczQiqcieiFII+rj7r9x9UbjcAPSOO5hIdYZ1b0O3tjk8rRFJRfZIlEKw1cwOKXtiZqOArfFFEonGzBg3pIC3F65l1cZtqY4j0mBFKQSXAXeb2WIzW0IwLPWl8cYSieaUIQWUOjw7U0NOiNRWlLGGZrj7YGA/YJC7D3X3mfFHE6le34557Ne1lSasEdkDiSavP8/dHzWzqyusB8Ddb405m0gk44YU8JvnPmbBqk30y2+R6jgiDU6iM4KyOQdaVLLkxZxLJLITB3cmzdBZgUgtVXlG4O5/CR++6u5vl38tbDAWqRc6tsjmkH4deGr6cq45agBpaZbqSCINSpTG4srmJ9acxVKvjB/ahWXrtzLt869SHUWkwUnURnAwMBLoUKGdoCWQHncwkZo4eu9O5GTO5snpy9i/Z9tUxxFpUBKdEWQRtAVksHv7wEbg9PijiUSX2yyDo/fJ5/mPVlBcohFJRWoiURvBFDN7i6DL6A1JzCRSK+OGFvD0jOVM/mQ1R+/TKdVxRBqMhG0E7r4TqNV5tpllm9n7ZjbTzOaY2Q3h+l5m9p6ZLTCzx80sqzbHF6no0L7taZebpd5DIjUUpbF4upk9Y2bnm9mpZUuE/bYDR4Q3ow0BjjWzg4CbgNvcvR/wFXBxrdOLlJORnsZJg7vw6tzVbNiqEUlFoopSCNoCXxJMRHNSuJxY3U4eKAqfZoZL2YQ2/wrXTwDG1TCzSJXGDS2guKSUSbNXpDqKSIMRZWKai2p7cDNLB6YBfYG7gU+B9e5eEm6yFCio7fFFKhrctRW92ufy1PTlnLV/91THEWkQzN0Tb2DWleC+gVEEf9G/BVzp7ksjv4lZa+BJ4JfAQ+7eN1zfDXjB3QdVss8lwCUA+fn5wydOnBj17XZTVFREXl79uxFauWqmJrmeWljM0wt38MfDc2iXE+WkNzm5kkm5aqax5hozZsw0dx9R7YbunnABXgEuIjh7yAC+DbxS3X6VHOdXwLXAWiAjXHcw8FJ1+w4fPtxrq7CwsNb7xkm5aqYmuT5bU+Q9fvqc3zt5YXyBQo3h80om5aqZPc0FTPUI389R/lzq4O4PuXtJuDwMdKhuJzPrEJ4JYGY5wFhgLlDI1/chXAg8HSGDSGQ92+cytHtrnpqu3kMiUUQpBGvN7DwzSw+X8wgaj6vTGSg0s4+ADwjOIp4DfgpcbWYLgXbAA7UNL1KVcUMKmLdyE3NXbEx1FJF6L0oh+A7BXMUrw+X0cF1C7v6RB3MX7Ofu+7r7b8L1i9z9AHfv6+5nuPv2PfkFRCpz4n6dSU8z3VMgEkGUiWk+d/eT3b1DuIxz9yXJCCdSW+3ymnF4/w48M2M5paWJO0SINHXVFgIz621mz5rZGjNbbWZPm5kmr5d675QhXVixYRvvfbYu1VFE6rUol4b+AfyT4Jp/F+AJ4LE4Q4nUhaP37kRuVroajUWqEaUQmLs/Uq7X0KME9xOI1Gs5Wekcs28nXpi1gm07dqY6jki9FaUQFJrZdWbW08x6mNlPgOfNrK2ZaeB3qdfGDy1g0/YSCuetTnUUkXqr2iEmgLPCn9+vsP47BGcGai+Qemtkn/Z0aNGMJ6cv47hBnVMdR6ReijLWUK9kBBGJQ3qacfLgLvzt3cWs31JM6+Ya9Vykoii9hjLN7Aoz+1e4XG5mmckIJ1IXxg8tYMdO54VZK1MdRaReitJGcC8wHLgnXIaH60QahH26tKRPh1z1HhKpQpQ2gv09mFymzOtmNjOuQCJ1zcwYP7SAP748ny/WbaFb2+apjiRSr0Q5I9hpZn3KnoQ3k6kvnjQopwwJpr14ZubyFCcRqX+iFIJrCbqQTjazKcDrwDXxxhKpW93aNmdEjzY8OX1Z2bDoIhJKWAjMLA3YCvQDrgiXAe5emIRsInVq3NACFq4uYs5yjUgqUl7CQuDupcAt7r49HE10pkYLlYbqhEGdyUw3NRqLVBDl0tDLZnaamVnsaURi1CY3i9EDOvLMzOXs1IikIrtEKQRXEww0V2xmG81sk5np3FoapHFDCli9aTvvfhplbiWRpiHKfAQt3D3N3TPdvWX4vGUywonUtSMHdqRFswye1OUhkV2inBFgZqea2a1mdouZjYs7lEhcsjPTOW5QJ16as5KtxeoFLQLRhpi4B7gUmAXMBi41s7vjDiYSl3FDCijaXsKrc1elOopIvRDlzuLDgX097HxtZhMIioJIg3Rg73Z0apnNU9OXcdLgLqmOI5JyUS4NfQJ0L/e8G/BRPHFE4peeZpwypAtT5q9h6VdbUh1HJOWiFIJ2wNzwzuLJwMdABzN7xsyeiTWdSEwuGNmTtDTj1pfnpzqKSMpFuTT0y9hTiCRZQescLhrVk/veWMR3DunFvgWtUh1JJGWqLARmZh6YkmibeGKJxO8Ho/vy+AdfcNOkeTxy8YGpjiOSMokuDRWa2Y/MrHz7AGaWZWZHhI3GF8YbTyQ+rXIyuXxMX95csJY35q9JdRyRlElUCI4lGG76MTNbbmYfm9kiYAFwDnCbuz+chIwisTn/4B50bZPDjS/O07AT0mRVWQjcfZu73+Puo4AewJHAMHfv4e7fc/cZSUspEpNmGelce8wA5q7YqMHopMmKdGexu+9w9xXuvj7uQCLJdtJ+XRhU0IpbXv6EbTt0t7E0PZEKgUhjlpZmXH/8XizfsI2H31mc6jgiSadCIAKM7NOeMQM6cHfhQr7aXJzqOCJJFXXQuR5mNjZ8nGNmLeKNJZJ81x03kM3bS7i7cGGqo4gkVZRB574H/Av4S7iqK/BUnKFEUmFApxacPrwrf3t3CV+s09AT0nREOSP4ITAK2Ajg7guAjnGGEkmVq47qT1oa/PHlT1IdRSRpohSC7e6+66KpmWUA6nAtjVLnVjlcfEgvnp6xnFlLN6Q6jkhSRCkEU8zsZ0COmR1FMG3ls/HGEkmd7x/eh7a5WfzuhbmEo6+LNGpRCsF1wBqCOQi+D7wA/CLOUCKp1DI7kyuO6Mu7i75ksoaekCYgSiHIAR509zPc/XTgwXCdSKN17oE96NGuOb9/QUNPSOMXpRC8xu5f/DnAq/HEEakfsjLS+Mkxe/HJqk38+8OlqY4jEqsohSDb3YvKnoSPm1e3k5l1M7NCM5trZnPM7MpwfVsze8XMFoQ/29Q+vkh8jh/UicHdWnPry/M10b00alEKwWYzG1b2xMyGA1sj7FcCXOPuA4GDgB+a2d4EbQ6vuXs/grON62oeWyR+ZsbPjtuLlRu38eDbn6U6jkhsohSCHwNPmNmbZvYm8DhweXU7hYPUfRg+3gTMBQqAU4AJ4WYTgHG1CS6SDAf2bsfYgfncO/lTvizanuo4IrGwKN3jzCwTGAAYMM/dd9ToTcx6Am8A+wKfu3vrcq995e7fuDxkZpcAlwDk5+cPnzhxYk3ecpeioiLy8vJqtW+clKtmUplreVEpP39rK2N7ZPCtgc3qTa5ElKtmGmuuMWPGTHP3EdVu6O7VLsBI4FzggrIlyn7hvnnANODU8Pn6Cq9/Vd0xhg8f7rVVWFhY633jpFw1k+pc1/17pve5/nn/bE3RbutTnasqylUzjTUXMNUjfE9HGWvoEeCPwCHA/uFSfYVh15nEv4G/u/t/wtWrzKxz+HpnYHWUY4mk0o/H9iczPY2bNfSENEJVTl5fzghg77C6RBZObP8AMNfdby330jMEcx3/Pvz5dE2OK5IK+S2z+d6hvbjz9YV895CvGNpdnd2k8YjSWDwb6FSLY48CzgeOMLMZ4XI8QQE4yswWAEeFz0XqvUsO70O73CxufHGehp6QRiXKGUF74GMzex/Y1W3C3U9OtJO7v0XQuFyZIyMnFKkn8ppl8OOx/fh/T8/htbmrGbt3fqojidSJKIXg13GHEGkozj6gOw++vZjfT5rH6AEdUh1HpE5UWwjcfUoygog0BJnpafz02AFc+uiHPDFtKZ1THUikDkTpNXSQmX1gZkVmVmxmO81sYzLCidRHx+zTiWHdW3PrK/PZXqK2Amn4ojQW3wWcAywgGHDuu+E6kSbJzPjZ8QNZs2k7kxbX6N5KkXop0uT17r4QSHf3ne7+EDA61lQi9dyInm05Zp98XvxsB2s2aegJadiiFIItZpYFzDCzP5jZVUBuzLlE6r2fHLsXxaVw52sLUh1FZI9EKQTnA+kEA81tBroBp8UZSqQh6NMhj9FdM3js/c9ZtKao+h1E6qlqC4G7L3H3re6+0d1vcPerw0tFIk3eKX2zyMpI4+aXNPSENFxVFgIz+2f4c5aZfVRxSV5EkfqrVTPj+4f14cXZK5m25KtUxxGplUT3EVwZ/jwxGUFEGqrvHtqLR99bwo0vzOWJSw8mGGZLpOGo8ozA3VeYWTrwQHh5aLcliRlF6rXcZhlcNbY/U5d8xcsfr0p1HJEaS9hG4O47CXoNtUpSHpEG6cwRXenTIZebXpzHjp2lqY4jUiNReg1tA2aZ2QNmdmfZEncwkYYkIz2N644byKK1m3n8gy9SHUekRqIMOvd8uIhIAmMHduSAnm25/dX5jBtaQF6zKP97iaRelEHnJlS3jYgEQ09cf/xejL/nHe5/YxFXHdU/1ZFEIoky6Fw/M/uXmX1sZovKlmSEE2lohnZvwwmDOnP/m4tYvXFbquOIRBKljeAh4F6gBBgD/A14JM5QIg3ZtccMoLiklNs19IQ0EFEKQY67vwZY2HX018AR8cYSabh6ts/lvIN68PgHX7Bw9aZUxxGpVqReQ2aWBiwws8vNbDzQMeZcIg3aj47oS05mOjdN0tATUv9FKQQ/BpoDVwDDgfOAC+MMJdLQtctrxmWj+/DKx6t4/7N1qY4jklCUQlDi7kXuvtTdL3L309z9v7EnE2ngvjOqF/ktm/G7F+birpnMpP6KUghuNbN5Zva/ZrZP7IlEGomcrHSuOWoAM75Yz4uzV6Y6jkiVogxDPYZgRrI1wH3haKS/iDuYSGNw2vCu9M/P4w+T5lFcoqEnpH6KOlXlSne/E7gUmAH8MtZUIo1Eeppx3XF7sfjLLTz2/uepjiNSqSg3lA00s1+b2WyCSevfAbrGnkykkRgzoCMH9W7LHa8tYNM2TXYv9U/UG8q+Ao5298Pd/V53Xx1zLpFGw8y4/riBrNtczF+m6KZ8qX+itBEc5O53uPvyZAQSaYwGd2vNSYO78Ne3FrFyg4aekPolUhuBiOy5a48ewM5S57ZX5qc6ishuVAhEkqR7u+acf1BPnpj2BfNXaegJqT9UCESS6EdH9CW3WQY3vTgv1VFEdqlyPgIzexao8nZIdz85lkQijVib3Cx+MLovN02ax7uffsnBfdqlOpJIwjOCPwK3AJ8BW4H7w6UImB1/NJHG6aJRPencKpsbX5xLaamGnpDUq7IQuPsUd58CDHX3s9z92XA5FzgkeRFFGpfszHSuOXoAHy3dwPOzVqQ6jkikNoIOZta77ImZ9QI6xBdJpPEbP7SAvTq14A8vzWN7yc5Ux5EmLkohuAqYbGaTzWwyUEgwNLWI1FJ6mnH98QP5Yt1WHnl3SarjSBMXZfL6SWbWD9grXDXP3bfHG0uk8TusX3sO7dee/3thLh8sXsdlo/sypFvrVMeSJqjaQhAaDvQMtx9sZrj732JLJdIEmBl3nTOMv761iAnvLOalOasY2acdl43uwyF922NmqY4oTUSUQeceIehBdAiwf7iMiLDfg2a2OhysrmxdWzN7xcwWhD/b7EF2kQavVfNMrjl6AO9cfyQ/P34gn64p4vwH3ueku97i+Y9WsFO9iiQJopwRjAD29ppPsfQwwWil5c8crgNec/ffm9l14fOf1vC4Io1OXrMMvndYby4Y2YOnpi8e4F+UAAATq0lEQVTjL1MW8cN/fEjPds35/uF9OHVYAc0y0lMdUxqpKI3Fs4FONT2wu78BVJys9RRgQvh4AjCupscVacyaZaRz1v7deeXqw7n3W8NokZ3J9f+ZxaE3FfKXKZ9qGGuJhVX3h76ZFQJDgPeBXY3EUe4sNrOewHPuvm/4fL27ty73+lfuXunlITO7BLgEID8/f/jEiROre7tKFRUVkZeXV6t946RcNdNUc7k7c9eV8tyiYj7+spScDDiyeyZH98ikZbOq2xCa6udVW40115gxY6a5e7WX8nH3hAtweGVLdfuF+/YEZpd7vr7C619FOc7w4cO9tgoLC2u9b5yUq2aUy33G51/5pY9M9Z7XPef9f/6C/+LJWf75l5tTnqsmlKtm9jQXMNUjfMdG6T46pValqHKrzKyzu68ws86AJrgRiWhwt9bce95wPl1TxH1TFjHxg8/5x/ufc+J+nbn08D4M7Nwy1RGlgYrSa+ggM/vAzIrMrNjMdprZxlq+3zPAheHjC4Gna3kckSarT4c8bjp9P978yRF8Z1RPXv14Fcfd8SYXPfQ+739WsVlOpHpReg3dBZwNPEHQg+gCoF91O5nZY8BooL2ZLQV+Bfwe+KeZXQx8DpxRu9gi0qlVNj8/YW9+OKYvj7y7hIfeWcyZf3mXET3acEi7Eg4rddLSdC+CVC/SDWXuvtDM0t19J/CQmb0TYZ9zqnjpyJoEFJHEWjfP4kdH9uO7h/bmn1O/4L43FnH7ku28uOxNLh3dmxP360JmuqYekapF+dexxcyygBlm9gczuwrIjTmXiNRQTlY6F47syeRrR3PJfs1wnKsen8nomycz4Z3FbC3W4HZSuSiF4Pxwu8uBzUA34LQ4Q4lI7WWmpzGySwaTrjyMBy4cQadW2fzqmTmMuul1/vTaAjZs0b0IsrsovYbKhkbcBtwQbxwRqStpacaRA/M5cmA+Hyxex72TP+WWV+bz5ymfcu6B3bn4kN50apWd6phSD0QddE5EGrD9e7Zl/2+3Ze6Kjfxlyqc8+PZiHn5nMacO7colh/emT4f6dzOVJI9akESakIGdW3L72UOZ/D+jOXv/7jw1Yxljb53CZY9OY87yDamOJymiQiDSBHVr25z/Hbcvb/30CH4wug9vLVzLiX96i589OYt1m4tTHU+SLGEhMLMLzexDM9scLlPN7IJkhROReHVo0Yxrj9mLt356BN8e2ZPHP/iC0TcXMuGdxZTsLE11PEmSKgtB+IX/Y+AaoAtQAPwEuFLFQKRxaZWTya9O2ocXrzyUQV1b8atn5nDCnW/xzqdrUx1NkiDRGcEPgPHuXujuG9x9vbu/TtB19AfJiSciydQ/vwWPXnwgfz5vGJuLSzj3/vf44d8/ZOlXW1IdTWKUqBC0dPfFFVeG6zS6lUgjZWYcu29nXr36cK4+qj+vzVvFkbdM4fZX57Nth25Ka4wSFYKttXxNRBqB7Mx0rjiyH69dM5qxe+dz+6sLOPKWKbw4a0XZMPLSSCQqBAPN7KNKllnAXskKKCKpVdA6h7vPHcZj3zuIFtkZXPb3D/nWX9/jk5WbUh1N6kiiG8oGJi2FiNR7B/dpx3M/OoR/vP85t7w8n+PvfJPzD+rBVWP706p5ZqrjyR6oshCUG1pCRASAjPQ0Lji4Jyfu14VbXv6ECe8u5pmZy7n2mAGcOaIb6Rr2ukFK1H10k5ltLLdsKv8zmSFFpH5pm5vF/40fxLOXH0KfDrlc/59ZnHL3W0xdrIlxGqJEbQSvAR8DvwX2dfcW7t6y7Gdy4olIfbZvQSv++f2DuePsIazdVMzpf36Xqx6fwaqN21IdTWqgykLg7uOAY4A1wP1mNsXMfmBmbZOWTkTqPTPjlCEFvHbN4Vw+pi/Pf7SCMX+czD2TF7K9RN1NG4KEQ0yEN5I9BBwH/Bn4DfDtJOQSkQYmt1kG/3PMAF65+jBG9W3PHyZ9wtG3vcFrc1epu2k9V91YQyPN7E/Ah8AogjuNb01KMhFpkHq0y+X+C0Yw4TsHkJ5mXDxhKhc9/AGfrilKdTSpQqLG4sXAPcAy4BLgQWCzmQ0zs2HJiSciDdXh/Tsw6crD+MUJA5m6+CuOvf0NHv+kmE3bNENafZPoPoLFgBO0ExxT4TUHjogpk4g0ElkZaXz30N6cPKQLN0/6hCemLWXqLVO47ti9GD+0gDR1N60XEt1HMDqJOUSkEevYIpubzxjMwMy1PL0sm2uemMmj7y3h1yftw+BurVMdr8mrshCY2amJdnT3/9R9HBFpzHq3TufJk0fy7w+XctOkTxh3z9ucMbwr1x6zFx1aNEt1vCYr0aWhfwEzwgWg/DmcAyoEIlJjaWnGGSO6ccy+nfjTawt46O3FvDhrJVeO7ceFI3uSma6JE5MtUSE4DTgL2A94GnjM3RcmJZWINHotszP5+Ql7c9b+3fnNcx/z2+fnMvGDLzh1WAH9O7ZgQKcWFLTOUTtCEiRqI3gSeNLMcoFTgFvMrB3wc3efkqyAItK49e2Yx4SL9ue1uav5/aR5/GHSJ7tey8lMp19+Hv06tqB/fh7981vQLz+PgtY5mKlA1JVEZwRltgEbgI1AdyA71kQi0uSYGWP3zmfs3vls2LqDhas3MX9VEfNXbWLBqiLeXLCGf3+4dNf2uVnp9M1vQf+OeQzo1IJ++UGh6NQyWwWiFhI1Fo8BzgEOAF4F7nD3qckKJiJNU6ucTIb3aMvwHruPZrN+S3G54hAUisJPVvPEtK8LRIvsDPp1LDtz+PosomOLZioQCSQ6I3gN+Ah4C2gGXFB+0np3vyLmbCIiu7RunsUBvdpyQK/dC8S6zcW7FYf5qzbx8sermPjBF7u2aZWTGRSITsFZRFmhaJ+XpQJB4kJwUdJSiIjUUtvcLA7q3Y6Derfbbf3aou3MX7WJ+Ss3MX91EQtWbeL5j1bwj61f39ncpnkm/fJbkFeync+bLd7VFtEur2l1ZU3UWDwhmUFEROpS+7xmtM9rxsg+7Xetc3fWbNr+9SWmsC3i3RUlvP70nF3btWmeSU5m+q7nlZ01lF9V9tjK9bLf7fVKjrPbEe2b68yMLZu3MHHQFrq3a17dr7tHojQWi4g0CmZGx5bZdGyZzSH9vi4QhYWFDBx2cHAGsWoTi9ZuZkdJKRDcNAVQfgBVZ7cnu20XbOsVX66w/ze33W181vDJ6jVbycqI/74KFQIRafLMjE6tsunUKpvD+ndIdZxdJk+eTKdW8XfUTDT66E3hzzNiTyEiIimT6JzjeDPLBK5PVhgREUm+RJeGJgFrgdxwsnojuHJlgGveYhGRxiHRnMXXunsr4Pnyk9Zr8noRkcal2uZodz/FzPLN7MRw2eOWFDM71sw+MbOFZnbdnh5PRERqr9pCEDYWvw+cAZwJvG9mp9f2Dc0sHbgbOA7YGzjHzPau7fFERGTPROk++gtgf3dfDRCeEbxKMF9BbRwALHT3ReHxJhKMbvpxLY8nIiJ7wMrf+FDpBmaz3H1QuedpwMzy62r0hsHZxLHu/t3w+fnAge5+eYXtLgEuAcjPzx8+ceLE2rwdRUVF5OXl1WrfOClXzShXzShXzTTWXGPGjJnm7iOq2y7KGcEkM3sJeCx8fhbwQq2TVbizOvSNauTu9wH3AZjZmjFjxiyp5fu1J+j9VN8oV80oV80oV8001lw9omxUbSFw92vD+YsPIfgSvy+ctKa2lgLdyj3vCiyvJkOtG6jNbGqUiphsylUzylUzylUzTT1XpCEmwonq62qO4g+AfmbWC1gGnA2cW0fHFhGRGkr6WEPuXmJmlwMvAenAg+4+p5rdREQkJikZdM7dX2DP2hlq4r4kvU9NKVfNKFfNKFfNNOlc1fYaAjCzLKB/+PQTd9+RaHsREWk4onQfHQ1MABYTNBZ3Ay509zfiDiciIvGLUgimAee6+yfh8/7AY+4+PAn5REQkZlGmvsksKwIA7j4fyIwv0p4xs5vNbJ6ZfWRmT5pZ6yq2S+p4R2Z2hpnNMbNSM6uyO5iZLTazWWY2w8ym1qNcyf682prZK2a2IPzZportdoaf1QwzeybGPAl/fzNrZmaPh6+/Z2Y948pSw1zfNrM15T6j7yYp14NmttrMZlfxupnZnWHuj8xsWD3INNrMNpT7rH4Zd6bwfbuZWaGZzQ3/X7yykm3i/bzcPeECPAg8AIwOl/uBh6rbL1ULcDSQET6+Cbipkm3SgU+B3kAWMBPYO+ZcA4EBwGRgRILtFgPtk/h5VZsrRZ/XH4DrwsfXVfbfMXytKAmfUbW/P/AD4M/h47OBx+tJrm8DdyXr31O59z0MGAbMruL144EXCS43HwS8Vw8yjQaeS8Fn1RkYFj5uAcyv5L9jrJ9XlDOCy4A5wBXAlQRjAl0aYb+UcPeX3b0kfPpfghvWKto13pG7FwNl4x3FmWuulzuzqi8i5kr65xUef0L4eAIwLub3SyTK718+77+AI80qmfE8+blSwoM2xHUJNjkF+JsH/gu0NrPOKc6UEu6+wt0/DB9vAuYCBRU2i/XzijIM9XZ3v9XdT3X38e5+m7tvr6sAMfsOQRWtqAD4otzzpXzzg08VB142s2nheEv1QSo+r3x3XwHB/yhAxyq2yzazqWb2XzOLq1hE+f13bRP+IbIBaBdTnprkAjgtvJzwLzPrVsnrqVBf/x882MxmmtmLZrZPst88vKQ4FHivwkuxfl5V3kdgZv909zPNbBaVjwW0X12FqCkzexXoVMlLP3f3p8Ntfg6UAH+v7BCVrKu+H20d5IpglLsvN7OOwCtmNs/3sIdWHeRK+udVg8N0Dz+v3sDrFgyS+OmeZqsgyu8fy2dUjSjv+SxB547tZnYpwVnLETHniiIVn1d1PgR6uHuRmR0PPAX0S9abm1ke8G/gx+6+seLLlexSZ59XohvKyhosTqyrN6sr7j420etmdiFB7iM9vMBWQY3HO6qLXBGPsTz8udrMniQ4/d+jQlAHuZL+eZnZKjPr7O4rwlPg1VUco+zzWmRmkwn+mqrrQhDl9y/bZqmZZQCtiP8yRLW53P3Lck/vJ2g3qw9i+Te1J8p/+br7C2Z2j5m1d/fYB6OzYH74fwN/92BIn4pi/bwSTVW5Inz4A3dfUn4haBirl8zsWOCnwMnuvqWKzXaNd2TBzXJnA7H1OInKzHLNrEXZY4KG70p7OCRZKj6vZ4ALw8cXAt84czGzNmbWLHzcHhhFPPNaRPn9y+c9HXi9ij9CkpqrwnXkkwmuP9cHzwAXhL1hDgI2lPvOSQkz61TWrmNmBxB8P36ZeK86eV8j6JAz191vrWKzeD+vCC3aH1ay7qO6bLGuywVYSHAtbUa4lPXk6AK8UG674wla5z8luEQSd67xBFV9O7AKeKliLoLeHzPDZU59yZWiz6sd8BqwIPzZNlw/Avhr+HgkMCv8vGYBF8eY5xu/P/Abgj84ALKBJ8J/f+8DveP+jCLmujH8tzQTKAT2SlKux4AVwI7w39fFBJ1MLg1fN4KZCj8N/9tV2ZMuiZkuL/dZ/RcYmaTP6hCCyzwflfveOj6Zn1eVN5SZ2WUEf/n3ZvdT7RbA2+5+XqU7iohIg5KoELQC2hD8RVH+RpVN7l7vumCJiEjtRBp0DiDsxZJd9tzdP48rlIiIJE+19xGY2UlmtgD4DJhCcOdrZX3zRUSkAYpyZ/FvCW5pnu/uvYAjgbdjTSUiIkkTpRDs8KAvcpqZpbl7ITAk5lwiIpIkUQrB+vCOtzeAv5vZHQR37IrUe2ZWlKT3+auZ7Z2M9xKpa1HmI8gFthIUjW8R3DH5d9/9jkWResnMitw9L9U5ROqzKIPObXb3UncvcfcJBDc1HBt/NJF4mFkPM3stHIjtNTPrHq7vEw5e94GZ/aays4nw7u/nw4HJZpvZWeH6yRbO52BmF5vZ/HDd/WZ2V7j+YTO7Nxx7fpGZHW7BGPlzzezhcu9xbziQ3hwzuyEpH4o0aVUWAjNraWbXm9ldZnZ0eGvz5cAi4MzkRRSpc3cRDOm7H8GghHeG6+8A7nD3/al6HJdjgeXuPtjd9wUmlX/RzLoA/4+gg8VRwF4V9m9DMOjbVQQDwt0G7AMMMrOytrefu/sIYD/gcDNL2QCP0jQkOiN4hGDCklnAd4GXgTOAU9y9Xox5LlJLBwP/CB8/QnCLf9n6J8LH/6i4U2gWMNbMbjKzQ919Q4XXDwCmuPs6d99R7nhlnvXgeuwsYJW7z3L3UoKhDXqG25xpZh8C0wmKhNoeJFaJRh/t7e6DIGgIA9YSDPm7KSnJRJIn8uBw7j7fzIYTjAVzo5m97O6/KbdJdZPRlM3lUVrucdnzDDPrBfwPsL+7fxVeMspGJEaJzgh2lD1w953AZyoC0ki8QzBSJwQdIN4KH/8XOC18fHbFnWDXpZ8t7v4o8EeCqQ/Le5/gck6bcDjq0yoeoxotgc3ABjPLB46r4f4iNZbojGCwmZWNz21ATvjcAHf3lrGnE9lzzc1sabnntxJMu/qgmV0LrAEuCl/7MfComV0DPE8wy1hFg4CbzayU4I+ly8q/6O7LzOx3BDNMLScYFruy41TK3Wea2XSCS0WL0M2bkgSRxxoSaezMrDmw1d3dzM4GzqlNe5iZ5Xkwy1UG8CTwoLs/Wdd5RepKojMCkaZmOHBXOFHIeoI5r2vj12Y2luDa/ssEUx6K1Fs6IxARaeKiDDEhIiKNmAqBiEgTp0IgItLEqRCIiDRxKgQiIk3c/wcrS+yDlvlgowAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data = np.load(\"blobs2.npz\")\n",
    "X_learn = data[\"X\"]\n",
    "Y_learn = data[\"Y\"]\n",
    "\n",
    "# sigma = 10**log_sigma\n",
    "log_sigmas = np.linspace(-2, 2,10)\n",
    "                         \n",
    "ratios = np.empty(len(log_sigmas))\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    init = tf.global_variables_initializer()\n",
    "    sess.run(init)\n",
    "    \n",
    "    # IMPLEMENT: compute ratio for every sigma\n",
    "    for i, log_sigma in tqdm(list(enumerate(log_sigmas))):\n",
    "        current_ratio = sess.run(mmd.rbf_mmd2_and_ratio(X_learn,Y_learn, sigma=10**log_sigma))[1]\n",
    "        ratios[i] = current_ratio\n",
    "\n",
    "plt.plot(log_sigmas, ratios)\n",
    "plt.grid(True)\n",
    "plt.xlabel(\"Log sigma\")\n",
    "plt.ylabel(\"Ratio of MMD and variance (proportional to test power)\")\n",
    "\n",
    "print(\"best sigma:\", 10**log_sigmas[np.argmax(ratios)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This was much faster, even for more parameters to try.\n",
    "\n",
    "The curve looks very much like those obtained when cross-validating kernel-parameters for kernel machines (like a support vector machine): you make the model/test more and more complex (bandwidth smaller), until it hits a sweet spot (peak), and after which it starts overfitting (ratio decreasing).\n",
    "Let's try out how it worked."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAH05JREFUeJzt3Xt0VOXd9vHvD5IYUMAE0AdJfQKPqEAkAwaLYJeABxB40RaturCiUnlr64NVkaNnoQt8USliaxEpYFuEgqAVVBqB2lIRAyJyspyCIgoSBUVEBe/3j72zmYRJMhMyh8D1WWvWzOy5971/mZnMNfsw9zbnHCIiIgB1kl2AiIikDoWCiIgEFAoiIhJQKIiISEChICIiAYWCiIgEFAoiIhJQKIiISEChICIigbRkFxCNJk2auNzc3IQv9/33vetzzkn4oiVJ9JpLyqiBN+PKlSv3OOeaxjJPrQiF3NxcioqKEr7crl2966VLE75oSRK95pIyauDNaGbbY51Hm49ERCSgUBARkYBCQUREArVin4KIlPXdd9+xY8cODh48mOxSJF4eeMC73rChyqaZmZnk5OSQnp5+zItVKIjUQjt27KBBgwbk5uZiZskuR+Khjr8hp4qjj5xzlJSUsGPHDlq0aHHsiz3mHkQk4Q4ePEjjxo0VCIKZ0bhx4xpba1QoiNRSCgQpVZPvBYWCiIgEFAoiUi2//e1vycvLo23btkyYMCGY/uCDD9K8eXNCoRChUIiFCxcCsGzZMtq1a0fHjh3ZvHkzAHv37qVHjx4k81zx8+fPZ/369cH9+++/n8LCQgC6du2alB/OJpN2NJ8AcocvOKb5i8f2rqFK5Hixdu1annnmGVasWEFGRgY9e/akd+/etGrVCoA777yTIUOGlJnnscceY+7cuRQXF/P73/+exx57jEceeYSRI0cmdVPY/Pnz6dOnD23atAHg4YcfTlotqUBrCiISsw0bNtCpUyfq169PWloaF198MfPmzat0nvT0dL7++msOHDhAeno6W7Zs4aOPPuLiiy+ucJ5XX32Vc889l4suuojBgwfTp08fwFsbGT9+fNAuLy+P4uJiAK666irOP/982rZty+TJk4M2p5xyCqNGjSI/P59OnTqxa9cu/v3vf/PSSy9xzz33EAqF2LJlCzfddBNz5sw5qpZFixZx4YUX0qFDB6655hr2798PwPDhw2nTpg3t2rU7KghrI60piNRyv/41rF5ds32GQhC2RegoeXl5jBo1ipKSEurVq8fChQspKCgIHp80aRIzZsygoKCAxx57jKysLEaMGMGgQYOoV68ezz33HEOGDOGRRx6pcBkHDx7k1ltvZfHixZx11llce+21UdU+depUsrOz+frrr+nYsSP9+vWjcePGfPXVV3Tq1IkxY8YwdOhQnnnmGe6991769u1Lnz59uPrqqyvsc8+ePYwePZrCwkJOPvlkxo0bx+OPP87tt9/OvHnz2LhxI2bG3r17o6oxlWlNQURi1rp1a4YNG8Zll11Gz549yc/PJy3N+4552223sWXLFlavXk2zZs24++67AQiFQixfvpwlS5awdetWzjjjDJxzXHvttdxwww3s2rWrzDI2btxIixYtaNWqFWbGDTfcEFVtEydODNYGPvzwQzZt2gRARkZGsKZx/vnnB2sW0Vi+fDnr16+nS5cuhEIhpk+fzvbt22nYsCGZmZn8/Oc/54UXXqB+/fpR95mqtKYgUstV9o0+ngYOHMjAgQMBGDlyJDk5OQCcfvrpQZtbb701+CAu5Zxj9OjRzJo1i9tvv52HHnqI4uJiJk6cyJgxY8q0rWhfQ1paGt9//31wv/QY/aVLl1JYWMibb75J/fr16dq1a/BYenp60F/dunU5dOhQ1H+rc47LLruMmTNnHvXYihUreP3113n++eeZNGkSixcvjrrfVKQ1BRGplt27dwPwwQcf8MILL3D99dcD8PHHHwdt5s2bR15eXpn5pk+fTu/evcnKyuLAgQPUqVOHOnXqcODAgTLtzj33XLZt28aWLVsAynwg5+bmsmrVKgBWrVrFtm3bANi3bx9ZWVnUr1+fjRs3snz58ir/jgYNGvDll19W2qZTp04sW7YsOGrqwIED/Oc//2H//v3s27ePXr16MWHCBFbX9Ha8JNCagohUS79+/SgpKSE9PZ2nnnqKrKwsAIYOHcrq1asxM3Jzc/nDH/4QzHPgwAGmT5/OokWLALjrrrvo168fGRkZR30Lz8zMZPLkyfTu3ZsmTZpw0UUXsXbt2mDZM2bMIBQK0bFjR84++2wAevbsydNPP027du0455xz6NSpU5V/x3XXXcett97KxIkTI+5gBmjatCnTpk3j+uuv55tvvgFg9OjRNGjQgCuvvJKDBw/inOOJJ56I8VlMPZbM44OjVVBQ4HSSnerTIanRqy2v+YYNG2jdunWyy0iopUuXMn78eF5++eVkl5IYMZ55LdJ7wsxWOucKKpglIm0+EhGRgDYfiUit0LVrV7qWrspJ3GhNQUREAgoFEREJKBRERCSgUBARkYBCQUSqbd68eZgZGzdurLLttGnT2LlzZ7WXtXTp0qN+HQ2wevXqYHjuWO3du5ff/e53Mc0zYcKEo35oF027Xr16VTo20lHtBw1i7xdfxFRbTVAoiEi1zZw5k4suuojnn3++yrbHGgoVqS2hsHDhQk499dTo20+ezKkNG8ZUW01QKIhItezfv59ly5bx7LPPHhUKjz76KOeddx75+fkMHz6cOXPmUFRURP/+/QmFQnz99dfk5uayZ88eAIqKioLDTVesWEHnzp1p3749nTt35v3SH3FF8O2333L//fcza9YsQqEQs2bN4quvvuKWW26hY8eOtG/fnhdffBGAdevWccEFFxAKhWjXrh2bNm1i+PDhbNmyhVAoxD333FOm76+++orevXuTn59PXl4es2bNYuLEiezcuZNu3brRrVs3wBsAsKCggLZt2/LAAw8ARGxX+vdG229u9+7s+fxzAGbMmEG7du3Iz8/nZz/72bG8bFXS7xREartkjJ2Nd3Kanj17cvbZZ5Odnc2qVavo0KEDr7zyCvPnz+ett96ifv36fPbZZ2RnZzNp0iTGjx9fZojtSM4991zeeOMN0tLSKCwsZOTIkcydOzdi24yMDB5++GGKioqYNGkS4A3O1717d6ZOncrevXu54IILuPTSS3n66ae544476N+/P99++y2HDx9m7NixrF27NuKYRa+++ipnnHEGCxZ4IwLs27ePRo0a8fjjj7NkyRKaNGkCwJgxY8jOzubw4cNccsklrFmzhsGDBx/VLtZ+S61bt44xY8awbNkymjRpwmeffVbp83estKYgItUyc+ZMrrvuOsAbP6h07KLCwkJuvvnmYBjp7OzsmPrdt28f11xzDXl5edx5552sW7cupvkXLVrE2LFjCYVCwSipH3zwARdeeCG/+c1vGDduHNu3b6devXqV9nPeeedRWFjIsGHD+Oc//0mjRo0itps9ezYdOnSgffv2rFu3rsypPY+l31KLFy/m6quvDsIi1uczVlpTEKntkjB2dklJCYsXL2bt2rWYGYcPH8bMePTRR3HORXV6zfDhr0uHtwa477776NatG/PmzaO4uDjmXzE755g7dy7nlBszqHXr1vzwhz9kwYIF9OjRgylTptCyZcsK+zn77LNZuXIlCxcuZMSIEVx++eXcf//9Zdps27aN8ePH8/bbb5OVlcVNN91U5m+pbr/l/55Enq5UawoiErM5c+Zw4403sn37doqLi/nwww9p0aIF//rXv7j88suZOnVqsNO0dHNH+SGqc3NzWblyJUCZzUP79u2jefPmgLdzuirl++3RowdPPvkkpYN9vvPOOwBs3bqVli1bMnjwYPr27cuaNWsqHTZ7586d1K9fnxtuuIEhQ4YEQ3WHz/PFF19w8skn06hRI3bt2sUrr7xSYV2x9BvukksuYfbs2ZSUlABo85GIpJ6ZM2fy4x//uMy0fv368Ze//IWePXvSt29fCgoKCIVCwbmUb7rpJn7xi18EO5ofeOAB7rjjDn70ox9Rt27doJ+hQ4cyYsQIunTpwuHDh6uspVu3bqxfvz7Y0Xzffffx3Xff0a5dO/Ly8rjvvvsAmDVrFnl5eYRCITZu3MiNN95I48aN6dKlC3l5eUftaH7vvfeCHdNjxozh3nvvBWDQoEFcccUVdOvWjfz8fNq3b0/btm255ZZb6NKlSzB/eLtY+w3Xtm1bRo0axcUXX0x+fj533XVXlc/JsdDQ2ZWoLcMoV0VDZ0evtrzmJ+LQ2Sec43XobDOra2bvmNnL/v0WZvaWmW0ys1lmlhHvGkREJDqJ2Hx0B7Ah7P444AnnXCvgc2BgAmoQEZEoxDUUzCwH6A1M8e8b0B0oPefddOCqeNYgcryqDZt+JTFq8r0Q7zWFCcBQ4Hv/fmNgr3PukH9/B9A8zjWIHHcyMzMpKSlRMAjOOUpKSsjMzKyR/uL2OwUz6wPsds6tNLOupZMjNI34rjazQcAggDPPPDMuNYrUVjk5OezYsYNPP/002aVIvHzyiXf9/feVt8P7kpCTk1Mji43nj9e6AH3NrBeQCTTEW3M41czS/LWFHCDiCFnOucnAZPCOPopjnSK1Tnp6Oi1atEh2GRJPt93mXSf4ULi4bT5yzo1wzuU453KB64DFzrn+wBLgar/ZAODFeNUgIiKxScaP14YBd5nZZrx9DM8moQYREYkgIWMfOeeWAkv921uBCxKxXBERiY2GuRARkYBCQUREAgoFEREJKBRERCSgUBARkYBCQUREAgoFEREJKBRERCSgUBARkYBCQUREAgoFEREJKBRERCSgUBARkYBCQUREAgoFEREJKBRERCSgUBARkUBCzrwmtVvu8AXHNH/x2N41VImIxJvWFEREJKBQEBGRgEJBREQCCgUREQkoFEREJKBQEBGRgEJBREQCCgUREQkoFEREJKBQEBGRgEJBREQCCgUREQkoFEREJKBQEBGRgEJBREQCCgUREQkoFEREJKAzr9UCx3rmMxGRaMVtTcHMMs1shZm9a2brzOwhf3oLM3vLzDaZ2Swzy4hXDSIiEpt4bj76BujunMsHQkBPM+sEjAOecM61Aj4HBsaxBhERiUHcQsF59vt30/2LA7oDc/zp04Gr4lWDiIjEJq47ms2srpmtBnYDfwe2AHudc4f8JjuA5vGsQUREohfXUHDOHXbOhYAc4AKgdaRmkeY1s0FmVmRmRZ9++mk8yxQREV9CDkl1zu0FlgKdgFPNrPSopxxgZwXzTHbOFTjnCpo2bZqIMkVETnjxPPqoqZmd6t+uB1wKbACWAFf7zQYAL8arBhERiU08f6fQDJhuZnXxwme2c+5lM1sPPG9mo4F3gGfjWIOIiMQgbqHgnFsDtI8wfSve/gUREUkxGuZCREQCCgUREQkoFEREJKBQEBGRgEZJlbg71lFei8f2rqFKRKQqWlMQEZFAzKFgZllm1i4exYiISHJFFQpmttTMGppZNvAu8Eczezy+pYmISKJFu6bQyDn3BfAT4I/OufPxhq0QEZHjSLShkGZmzYCfAi/HsR4REUmiaEPhIeA1YLNz7m0zawlsil9ZIiKSDNEekvqxcy7Yueyc26p9CiIix59o1xSejHKaiIjUYpWuKZjZhUBnoKmZ3RX2UEOgbjwLExGRxKtq81EGcIrfrkHY9C84cqIcERE5TlQaCs65fwD/MLNpzrntCapJRESSJNodzSeZ2WQgN3we51z3eBQlIiLJEW0o/BV4GpgCHI5fOSIikkzRhsIh59zv41qJiIgkXbSHpP7NzH5pZs3MLLv0EtfKREQk4aJdUxjgX98TNs0BLWu2HBERSaaoQsE51yLehYiISPJFFQpmdmOk6c65GTVbjoiIJFO0m486ht3OBC4BVgEKBRGR40i0m4/+N/y+mTUCnotLRSIikjTVPUfzAaBVTRYiIiLJF+0+hb/hHW0E3kB4rYHZ8SpKRESSI9p9CuPDbh8CtjvndsShHhERSaKoNh/5A+NtxBspNQv4Np5FiYhIckQVCmb2U2AFcA3eeZrfMjMNnS0icpyJdvPRKKCjc243gJk1BQqBOfEqTEREEi/ao4/qlAaCrySGeUVEpJaIdk3hVTN7DZjp378WWBifkkREJFmqOkfzWcDpzrl7zOwnwEWAAW8Cf05AfSIikkBVbQKaAHwJ4Jx7wTl3l3PuTry1hAnxLk5ERBKrqlDIdc6tKT/ROVeEd2pOERE5jlQVCpmVPFavshnN7AdmtsTMNpjZOjO7w5+ebWZ/N7NN/nVWrEWLiEh8VBUKb5vZreUnmtlAYGUV8x4C7nbOtQY6Ab8yszbAcOB151wr4HX/voiIpICqjj76NTDPzPpzJAQKgAzgx5XN6Jz7GPjYv/2lmW0AmgNXAl39ZtOBpcCwatQuIiI1rNJQcM7tAjqbWTcgz5+8wDm3OJaFmFku0B54C+9optKw+NjMTou1aBERiY9oz6ewBFhSnQWY2SnAXODXzrkvzCza+QYBgwDOPPPM6ixaRERiFNdfJZtZOl4g/Nk594I/eZeZNfMfbwbsjjSvc26yc67AOVfQtGnTeJYpIiK+uIWCeasEzwIbnHOPhz30EjDAvz0AeDFeNYiISGyiHeaiOroAPwPeM7PV/rSRwFhgtn8E0wd4I6+KiEgKiFsoOOf+hTckRiSXxGu5IiJSfRrpVEREAgoFEREJKBRERCSgUBARkYBCQUREAgoFEREJKBRERCSgUBARkUA8f9EsQO7wBckuodY71ueweGzvGqpE5PinNQUREQkoFEREJKBQEBGRgEJBREQCCgUREQkoFEREJKBQEBGRgEJBREQCCgUREQkoFEREJKBQEBGRgEJBREQCCgUREQkoFEREJKBQEBGRgEJBREQCCgUREQkoFEREJKBQEBGRgEJBREQCCgUREQkoFEREJKBQEBGRgEJBREQCCgUREQkoFEREJJCW7AJS3fKtJeQOX57sMkREEiJuawpmNtXMdpvZ2rBp2Wb2dzPb5F9nxWv5IiISu3huPpoG9Cw3bTjwunOuFfC6f19ERFJE3ELBOfcG8Fm5yVcC0/3b04Gr4rV8ERGJXaJ3NJ/unPsYwL8+LcHLFxGRSqTs0UdmNsjMisys6NNPP012OSIiJ4REh8IuM2sG4F/vrqihc26yc67AOVfQtGnThBUoInIiS3QovAQM8G8PAF5M8PJFRKQS8TwkdSbwJnCOme0ws4HAWOAyM9sEXObfFxGRFBG3H685566v4KFL4rVMERE5Nim7o1lERBJPoSAiIgGFgoiIBBQKIiISUCiIiEhAoSAiIgGFgoiIBBQKIiISUCiIiEhAoSAiIgGFgoiIBBQKIiISUCiIiEggbqOkporc4QuqPe8nWzvVYCUiIqlPawoiIhJQKIiISEChICIigeN+n4JILPuVSvcj5Q5fXmZ68djeNVqTSKrSmoKIiAQUCiIiElAoiIhIQKEgIiIB7WgWicKx/AgStKNaag+tKYiISEChICIiAYWCiIgEFAoiIhJQKIiISEChICIiAYWCiIgEFAoiIhJQKIiISEChICIiAYWCiIgEFAoiIhJQKIiISCApoWBmPc3sfTPbbGbDk1GDiIgcLeGhYGZ1gaeAK4A2wPVm1ibRdYiIyNGSsaZwAbDZObfVOfct8DxwZRLqEBGRcpIRCs2BD8Pu7/CniYhIkiXjzGsWYZo7qpHZIGCQf3e/mb0fwzKaAHuqUVtE28fVVE9ADddWw1SbL8bXvMrarGbfQ7HQa1o9qVObHfWRGUtt/x3r4pIRCjuAH4TdzwF2lm/knJsMTK7OAsysyDlXUL3y4ku1VY9qqx7VVj0ncm3J2Hz0NtDKzFqYWQZwHfBSEuoQEZFyEr6m4Jw7ZGa3A68BdYGpzrl1ia5DRESOlozNRzjnFgIL47iIam12ShDVVj2qrXpUW/WcsLWZc0ft4xURkROUhrkQEZEjnHMpewGygb8Dm/zrrAraDfDbbAIGhE0/H3gP2AxM5MiaUcR+gf7AGv/ybyA/hWo7F3gT+AYYUsGyegLv+30Oj/D4ScAs//G3gNywx0b4098HelTVJ9DC72OT32dGFa9lImu73Z/mgCZRvM8SWduf/elrgalAegrV9izwLt77fw5wSqrUFvb4k8D+FHtNpwHbgNX+JZRCtRkwBvgPsAEYXOVzV1WDZF6AR0v/QGA4MC5Cm2xgq3+d5d8u/SBdAVzoPzGvAFdU1i/QOWzeK4C3Uqi204CO/gt8VCjg7bTfArQEMvD+uduUa/NL4Gn/9nXALP92G7/9SXgf9lv8/irsE5gNXOfffhq4rZLnKtG1tQdygWKqCIUk1NbLf80NmJliz1vDsH4fJ8IHVrJq8+crAJ6jilBIwvM2Dbg6ys+0RNd2MzADqFP6OVJljdH8Icm64CVfM/92M+D9CG2uB/4Qdv8P/rRmwMZI7aLsNwv4KNVqAx4kcihcCLwWdn8EMKJcm9eAC/3baXg/gLHybUvbVdSnP88eIC3SspNZW7k+i6k6FJJSmz/9TmBMqtXmz/97YFiq1Ib3wbcE73+iqlBIdG3TiD4UEl3bCuCsaGorvaT6PoXTnXMfA/jXp0VoU9GwGc392+WnR9vvQLxv8KlYWyTRDB8StHHOHQL2AY2rqDPS9MbAXr+PipaVrNpilZTazCwd+BnwairVZmZ/BD7B21z5ZArVdjvwUun/RhWS8ZqOMbM1ZvaEmZ2UQrX9D3CtmRWZ2Stm1qqS2oAkHZIazswKgf+K8NCoaLuIMM1VMj2amrrhhcJuM1ubSrVVY1nHUk+kLw3VqT+RtcUqWbX9DnjDOffPVKrNOXezP5Lxk8C1wB+TXZuZnQFcA3StoJak1eZfj8AL0gy8w0WHAQ+nSG0nAQedcwVm9hO8/Vg/qqA2IAVCwTl3aUWPmdkuM2vmnPvYzJoBuyM020HZN0sOsNSfnlNueulwGhX2a2btgCl42/j/k0q1VSGa4UNK2+wwszSgEfBZFfNGmr4HONXM0vxvMhGHKklSbbFKeG1m9gDQFPi/qVYbgHPusJnNAu6h4lBIZG3tgbOAzeaNA1TfzDY7585KgdoIW3v5xl/TGlJBXQmvzZ9nrn97HhW/nkfEsq0p0Rfg/1F2p+ujEdpk4+35z/Iv24Bs/7G3gU4c2Znbq7J+gTPx9t53TrXawvp8kMj7FNLwdmS34MjOprbl2vyKsjuwZvu321J2B9ZWvG24FfYJ/JWyO5p/WclzldDawvospup9Col+3n6Od2RbvSjeYwmrDe99eJY/rwHjgfGpUFuEZVe1TyHRr2mzsOdtAjA2hWobC9zi3+4KvF3l+66qBsm84G1Hex3vsMfXOfKBWgBMCWt3C96H+Wbg5rDpBXiH/m0BJnHksM+K+p0CfM6RQ8uKUqi2/8JL/S+Avf7thuVq6oV36NkWYJQ/7WGgr387E+/DfDPeDqiWYfOO8ud7H/9IqIr69Ke39PvY7Pd5UhWvZSJrG+w/P4fwvjFNSaHaDvnTSt9j96dCbXibIJbhHSa9Fu/Q2YapUFuE5UZzSGoiX9PFYc/bn6j6UN5E1nYqsMCv700qOcy+9KJfNIuISCDVjz4SEZEEUiiIiEhAoSAiIgGFgoiIBBQKIiISUChI0plZYzNb7V8+MbOPwu5nxNDPLWYW6dfxFbXvbmadYm1nZr8ys/411f5YmFmame2NR99yYkr6L5pFnHMlQAjAzB7EOw59fDW6ugVYhTfkQDS64/06e3ks7ZxzT9Vwe5GUoTUFSWlmNsDMVvhrDb8zszr+t+PnzOw9M1trZoPN7Fq8YJkVaQ3DzO40s/Vm9q6Z/cnM/gfv18X3+O07m9mVZvaWmb1jZovM7LQK2o02s1/H0G94+7PNbLHffpWZ5Zar8zEzGxR2f7SZ3WFmDf35VvkDr/WJ8Fxdambzw+4/bWY3+Lc7mtk/zGylPzDa6TXx+sjxR2sKkrLMLA/4Md6wI4fMbDLez/634A1fcZ7f7lTn3F4z+1/gdufc6gjdDQX+2zn3bVj7KcAe59wEv58svJE4nZn9ArjbOTcsQrteMfYb3n4m8KBz7m9mlsnRX8yexxuaoPQ8vNcA3YCvgSudc1+a2Wl4vz5+Ocrn8STgt3i/mN3jb8p6BBhU+ZxyIlIoSCq7FO/EQkX+QGj18IYIfg04x8x+CywEFkXR1zrgT2b2IjC/gjZnArP9/RIn4Q0bUBP9AkHoNHHO/Q3AOXewfBvn3Ntm9gP/m3wO8Ilzbqe/5jPOzC4Cvgd+YGZN8IY8qUprvHFzCv3nsS5lh24XCSgUJJUZMNU5d99RD3ij2V6BN85RP6r+1tsDuBi4ErjXXwsp7yngN865hWZ2Kd6AhFWJpt9w0YwrMxfvb8rFW3MAuBFvtMwO/lrTDrwxcsIdouyaR+njBqxxzlU6ZLIIaJ+CpLZC4Kf+N+LSo5TONLOmeAMI/hV4AOjgt/8SaFC+E/POD5DjnFuMNxx0U6B+hPaNgI/M+zo9IGz6sfYLgHPuc2CPmf0ff/5MM6sf4e9+Hm8z2U84MuxxI2C3HwiXEfmEQtuBtmaW4a+VdPenrweam9kF/nIzzKxthPlFFAqSupxz7wEP4W32WIO3meh0vLHj3zCz1cAzwEh/lj8CUyLsaE4D/uL3sQrvvNdfAi/ihc47ZtYZb1jyecA/gF1h85dvF2u/4foDd/vz/AsvSMr/3e/607c550rPp/Ec0NnMivD2M2yKMN82vE1Y7+Gdl3eVP/0b4GrgcTN7F3gH+GH5+UUAjZIqIiJHaE1BREQCCgUREQkoFEREJKBQEBGRgEJBREQCCgUREQkoFEREJKBQEBGRwP8HvSHmvX9j3y8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sigma_opt = 0.215\n",
    "\n",
    "# IMPLEMENT: visualise test for best sigma\n",
    "my_kernel = lambda X,Y : gauss_kernel(X,Y,sigma=sigma_opt)\n",
    "my_mmd = lambda X,Y : quadratic_time_mmd(X,Y, my_kernel)\n",
    "\n",
    "statistics = two_sample_permutation_test(my_mmd, X, Y, num_permutations)\n",
    "my_statistic = my_mmd(X,Y)\n",
    "plot_permutation_samples(statistics, my_statistic)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Victory!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gradient descent\n",
    "Now, what if we have more than one kernel parameter?\n",
    "Trying out all the different combinations is not the best idea.\n",
    "Instead, we can take a deep learning artisan's mindset and do gradient descent.\n",
    "That is, we maximize the ratio of MMD to standard deviation as a function of the kernel parameters (differentiable!), we could even do this in mini-batches.\n",
    "\n",
    "The implementation of the ratio is quite tricky (you might have checked the imported code).\n",
    "Differentiating it would be a nightmare, but luckily in 2018 we have tools at hand that allow us to do this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GANs\n",
    "\n",
    "Moving to the context of training GANs (Generative adversarial networks).\n",
    "Assume someone came to you with a new GAN, which generates some realistic looking images, supposedly indistinguishable from the images used to train the GAN.\n",
    "\n",
    "Let's assume that person said (https://arxiv.org/abs/1606.03498)\n",
    "> On MTurk, annotators were able to distinguish\n",
    "samples in 52.4% of cases (2000 votes total), where 50% would be obtained by random\n",
    "guessing. Similarly, researchers in our institution were not able to find any artifacts that would allow\n",
    "them to distinguish samples.\n",
    "\n",
    "Let's see what the kernel MMD thinks about this.\n",
    "We will use a slightly different kernel than before: an ARD kernel.\n",
    "This one that scales each of the $D$ input dimensions by a parameter, and then uses a standard Gaussian kernel (unit bandwidth) on top of that.\n",
    "$$ k(x,y) = \\exp\\left(-\\sum_d^D (x_d/\\sigma_d - y_d/\\sigma_d)^2\\right) $$\n",
    "Again, we have implemented this for you in `mmd.py`, feel free to see how.\n",
    "\n",
    "We begin with visualizing the data, and splitting it into train and test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Image dimensions: 28x28=784\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsMAAADWCAYAAAA0CI9DAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAADmZJREFUeJzt3V+odWlZAPDnGb/Mhmmcxj/BNImCFw1U1EWBpCVZEWRGIirNQAUTA95EfxAUFDFJISiYbppA6MK/Jc2FhRGjV6KOM1eWSEQi5tgMjjI5YzPojG8XZx+/4/n2n7X3Xmvttdbz+8Ew5/Dtvdfaz3rXe571vO96V7bWAgAAKrru1DsAAACnIhkGAKAsyTAAAGVJhgEAKEsyDABAWZJhAADKkgwDbJCZz8rMJzLzRafel0Nl5lcy85Wn3g+AqZIMA4uxSlzP//tuZj554ffb9/281tozrbUbWmtfHmJ/ATi9K6feAYC+tNZuOP85M78UEXe21u7b9PrMvNJae3qMfQNgmlSGgTIy812Z+eHM/GBmPh4Rd2TmyzLzM5n5WGb+T2benZk/sHr9lcxsmfni1e/vW/37xzLz8cz8dGa+ZMO2rs/MD2Tm11ef/dnMfP7q3+7MzC+sPuO/MvPOC+/7lcz8Uma+JTO/lplfzczfzMxXZ+Z/ZuY3MvPNa77TP6w+78HM/KkN+3RdZr51tc1HM/NDmfkju/YXYMkkw0A1vx0RH4iI50bEhyPi6Yj4w4h4fkT8QkT8ekTcteX9vxMRb4uImyPiyxHxZxte9/sRcX1E3BoRz4uIN0XEU6t/eyQifiMiboyIP4iIv87Mn77w3lvjrH++ZfX5742IN0bEz0bEKyPinZfmMb929Z1ujoiPRMS9mblu5O+PV9v9xdU2vhURd3fYX4DFkgwD1XyytfbR1tp3W2tPttYeaK3d31p7urX2xYj424j4pS3v/0hr7cHW2nci4v0R8TMbXvedOEuwX7qae/xga+2JiIjV9r/YznwiIj4eEa+48N6nIuI9q218KCJeEBF/1Vp7orX2uYj4j4i4mDzf31q7d/X6v4izJPvn1uzTXRHx1tbaQ621pyLiHRHx+sy8btv+AiyZZBio5r8v/pKZP5GZ/5yZD2fmNyPinXGWFG7y8IWf/y8ibtjwur+LiPsi4u8z86HMfM95tXY15eH+1ZSHxyLi1y5t89HW2jOrn59c/f+RC//+5KXtfu87rd73UJxVlS97UUR8dDUN4rGI+LeIaBHxwm37C7BkkmGgmnbp93si4t/jrCJ6Y0S8PSLy6I209u3W2jtaa7dFxMvjbHrG7Zn5Q3E2leHdEfGjrbWbIuJfj9zmj5//sKry/lhEfHXN674SEb/aWrvpwn/Paa09vGl/j9gngFmQDAPV/XBE/G9EfCszb4vt84U7y8xfzsyfXCWn34yzaQjPRMQPRsSzI+JrEfFMZr46Il515OZ+PjN/a3Xj359GxOMR8cCa1/1NRPz5+XzjzHxhZr5mx/4CLJpkGKjuTyLid+Msgbwnzm6q68MtEfGPcZZYfj7OpiB8sLX2WET8UUTcGxHfiIjXRcQ/HbmteyPijtXnvSEiXrthybi/jIh/iYiPr1bT+FRcnVu8dn+P3C+AycvWLo8YAjAXmfmuiLi1tfZ7p94XgDlSGQYAoCzJMAAAZZkmAQBAWSrDAACUJRkGAKAsyTAAAGVJhgEAKEsyDABAWZJhAADKkgwDAFCWZBgAgLIkwwAAlHVlzI1lpsfdbdFay33fI6bbHRLTCHHdRVvtn7Y6DG21f9rqMLTV/nWNqcowAABlSYYBAChLMgwAQFmSYQAAypIMAwBQlmQYAICyRl1ajWVprduKLpkHrcIDADA4lWEAAMpSGWajrpXffT5HlXgY5zEW3+12tWnxA6hHZRgAgLIkwwAAlGWaBNfoa3oEsDymPU2PaVLfb93fMLFhG5VhAADKUhlmL/tcXa+7OlfB6I8KfndiNW0qefvb1r9G1IvfKc/xqu13Sd9bZRgAgLIkwwAAlDXLaRLWCh1fXzF1bBjLIcOmlYeZdxlyGPo81qazsK9tbWbIc3jTdpfebyz1HFUZBgCgLMkwAABlzWaaxD6l+bHL+EsbFjnm+yx1CGXqltYGjzHEY8TPiTOndqppAXMiDv3r2q/OdaqZyjAAAGXNpjJ87Pq2Q5rrlRDzpPp+rbFiUm2d7LGq49o0+zjlqE2lEaNjz8s59ZcqwwAAlCUZBgCgrNlMk9jH2GsLzmEIYEjWfWbu1rXRykP3+rlpqdwWz4nBfM1hKqnKMAAAZS2yMsw4XKlzase0wV0Vim1PRZtDpWPq9B/90QaHYRm7a+363rvO66neVKcyDABAWZJhAADKMk2iAzeTXLXP0GbVGA1BGzzOMbG6+N4lT5mYQhubc/w4rWPbr2k72y393FQZBgCgLJXhLVwp7m/pV4/Mh7Y4TfpVDrFrhGadIdpahX5l2wjYUr+/yjAAAGVJhgEAKMs0iT0tdYhgF0ObzMUpz9GlDyUyHk/23GyIJ0ZuW1e8mn1jsYSYqQwDAFCWZBgAgLJMk7hkCeX+sVUerhvSprZYPd5TPkerH5tDiRvHGrINaZ9XHfKsgTmsz64yDABAWSrDHUzhquUUplyBg7FUPg8Oqd5UjhfzNYUnME7NIWs7zzVmKsMAAJQlGQYAoCzTJLjG0odDYJNDhviXch7sGhI1/WEc1hdmirbdDLerTXbtW07ZtlWGAQAoS2V4xeT5bsRkeCpw4xPz79f3eS6+TJF2ub9j+4apPulPZRgAgLIkwwAAlFV6msTUyvSs54a+ZX+3MfR9rjse+zlkvVI4Bed2TSrDAACUVa4ybNma46jqcGqWAJsfx+Q4/i6xNOv68UOeeNkXlWEAAMqSDAMAUFa5aRKbGIaal6UdL8PI07W0tnYKbqDbTkzGI9bTs27t4bGnTKgMAwBQlmQYAICyykyT8Ljl7vaNyyHDTmK/mxjt1vejPcV8GIammSN5Qx0qwwAAlLXoyrBqxDhcKXNq69qgNcWnTfyZItXg09l0o+35z0MeB5VhAADKkgwDAFDWoqdJwFwYhuufmE6T43KtdcPD4jSMXWtei3tNKsMAAJSlMgzAoFTbuhOr8Yj1dI391EqVYQAAypIMAwBQVrlpEoZFAADmYYy8TWUYAICyFl0ZVgUGAGAblWEAAMqSDAMAUFaOsX4bAABMkcowAABlSYYBAChLMgwAQFmSYQAAypIMAwBQlmQYAICyJMMAAJQlGQYAoCzJMAAAZUmGAQAoSzIMAEBZkmEAAMqSDAMAUJZkGACAsiTDAACUJRkGAKAsyTAAAGVdGXNjmdnG3N7ctNZy3/eI6XaHxDRCXHfRVvunrQ5DW+2ftjoMbbV/XWOqMgwAQFmSYQAAypIMAwBQlmQYAICyJMMAAJQlGQYAoKxRl1YDGEtr21ccyjxodSgAFkZlGACAslSGe3ZejVJ1Yija2HbbKsJiBrUZMerfrph2dcrYqwwDAFCWZBgAgLJMk7ikr3L/us8x/MJlx7S3ru/V7sSAeTukn9DmOVZf+dBFU22XKsMAAJRVrjI8xJUO7GPsNnhxe1O9Kj+WkRiWou/RSefBfir0l7vs2waXECeVYQAAypIMAwBQVrlpEozHeq+HOSY2u4a3DAEyBDd4Hcf0venQLrdbanxUhgEAKEsyDABAWYuZJjHE8O9ShwP6MsYauRE1jsMQbdbQK0Mbsg+ocN4fs1a485uhnLe3Sm1MZRgAgLJmXxmuuB7eKVS6QlyKXVf3S1iHVLsc3xAV3UrHsa8nR3atFrtpdj/idVWlkUaVYQAAypIMAwBQ1iynSSy9XH9qx8R317DSuqH5rmvjzn3IyiODxzHkOs1Dbnvq+l43XD++3pBtyBQAWE9lGACAsmZTGbYMz/T0vRyYStEwKt0Ecai+4rKUUYxzY/e7S4nbOmOce851+rTrhsyu5nBeqwwDAFCWZBgAgLJmM01ikzmU3+dgrOFQUyI4pSGG+JbYlvu+Wa7rZ1c1xN8xfS2H6vvJiHO4eVxlGACAsiZfGXZVOy2nOB5Tu4LcxxyuiOfmkJj29dQvDuMGaJi3rudo1754an8bVYYBAChLMgwAQFmTnybBNIw9PcKwKWPS3oZhmtt22t30OCbDxmDbTXenfEKiyjAAAGVJhgEAKGv20ySGXA+zkq7rBY69D3AoKxgMw9SH7sQKdlu3JvbYUyZUhgEAKGv2leFthrgqr1RNOuS7Ws91elSn2FdfTy/b9jn6AOCii33CuirxkH2GyjAAAGVJhgEAKGvy0ySmcGMX/TE0Oh1LPxabhtz6sOnzlhbTY7+Pvho4xLr+e8ib6lSGAQAoa/KV4XW6XhEMUZUYYyL33Kj+TJPjctWupXsuv24TN4IxF9rqYU75FDROR2UYAICyJMMAAJQ1y2kSXe1zA01f62pWYk3h6an61LWuQ5v7fH9PtxxGpdjt+htk2h1Mg8owAABlLboyvA8V4W72iZNqx7CqVub7qrYdcs4vLZZD0JeuN8boo9izFGPfAKoyDABAWZJhAADKMk1iT4ZJOZahzP5sG3ruK87O+eOJ4emI/Rn9bndjT1GYwo3fKsMAAJQlGQYAoKwy0ySOuZPXMFPdlQuOtc9a131vr5J139u5zhSt6xOOHZY2BeB4zv3txm5jYx8PlWEAAMoqUxk+t+lqw5OADidm3fS9zqi4byc+TF3XG0APGWHS/ncTo/WGXBN7qjFXGQYAoCzJMAAAZZWbJrHJVEv3p+SmjGFoayzNpr5CW+9m1w2gU1iHde7EaH+VYqYyDABAWSrDHKzSVSPAmPSvMB6VYQAAypIMAwBQlmkSbLRubUtDd8BlYz9pEaBPKsMAAJSlMkwnKsJAF/oKYG5UhgEAKEsyDABAWelmBwAAqlIZBgCgLMkwAABlSYYBAChLMgwAQFmSYQAAypIMAwBQlmQYAICyJMMAAJQlGQYAoCzJMAAAZUmGAQAoSzIMAEBZkmEAAMqSDAMAUJZkGACAsiTDAACUJRkGAKAsyTAAAGVJhgEAKEsyDABAWZJhAADKkgwDAFCWZBgAgLL+H5DRJTeZhZtAAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x216 with 16 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsMAAADWCAYAAAA0CI9DAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAADxRJREFUeJzt3W/IdGlZAPDr2taVcl03s5JMDaL/UhaVWGRBECSCSBob27LWl9QPhZkJQX+pD+YXiwj8EPshFCyNRPuDRmSIFlLtp/5Alqslm63rtlst4sbdh5mH99nnnWfmzMw5M+fc1+8HLzzvPDPznLnmnjPXue7rPidbawEAABXdcu4NAACAc5EMAwBQlmQYAICyJMMAAJQlGQYAoCzJMAAAZUmGATqUma/OzA+dezsA5k4yDHQtM+/KzL/OzP/JzE+vf35dZuaV+/1SZrbM/M4rt796ffsbr9z+b5n5fSd4CQBMSDIMdCsz3xARvxERb4mIZ0fEl0fEayLiuyPitkv3y4i4JyIejoh7NzzVwxHxpsy8Y+ptBuC0JMNAlzLzGRHxKxHxutbau1prj7WVv2ut3d1a+9ylu39PRHxFRPxURNyVmbddebp/iIiPRMTrB/7tl2bm32fmY5n575n5M+vbvzgz35eZ/5mZn13//JWXHvcXmfmrmfnhzPzvzHxvZn5JZr49Mx/NzI9m5lddun/LzJ/MzH/JzIcy8y2ZuXG/nplfn5kfyMyHM/OfMvOHd20vQAWSYaBXL46Ip0bEewbc996IeG9EvHP9/5dtuM/PR8TrM/OZA57vdyLiJ1prT4+IF0TEn69vvyUi7ouI50fE8yLi8Yj4rSuPvStWVernRMRXxyoJvy8inhmrpPwXr9z/FRHx7RHxbRHx8oj48asbk5lPi4gPRMQ7IuLLIuJHIuK3M/ObdmwvQPckw0CvnhURD7XWnri4YV1xfSQzH8/Ml6xv+6KIeFVEvKO19vmIeFdsaJVord0fEe+PiDcN+Nufj4hvzMw7Wmufba397fo5PtNae3dr7X9ba49FxK9FxPdeeex9rbWPtdb+KyL+JCI+1lr7s/Xr+P2I+NYr939za+3h1tonIuKtsUp0r3pZRHy8tXZfa+2J9fa8OyJeuW17ASqQDAO9+kxEPCszb724obX2Xa21O9e/u9j/vSIinoiIP17//+0R8YOZ+aUbnvMXIuK1mfnsHX/7hyLipRHxQGZ+MDNfHLFKvDPzbZn5QGY+GhF/GRF3ZuYXXHrsf1z6+fEN/7/9yt/65KWfH4hVu8dVz4+IF60PBB7JzEci4u5Y9VFfu70AFUiGgV59JCI+F6vWgW3ujVWC+YnMfDBW1denxIYKa2vtHyPiDyLi57Y9YWvto621l8eqJeEPI+L31r96Q0R8XUS8qLV2R0S8ZH173vwsgz330s/Pi4hPbbjPJyPig621Oy/9u7219tod2wvQPckw0KXW2iMR8cux6o19ZWbenpm3ZOYLI+JpERGZ+ZyI+P5YtRG8cP3vWyLizbH5rBKxfs4fi4g7N/0yM2/LzLsz8xnrtotHI+L/1r9+eqyqu4+se4+v9v8e4o3rhXnPjdUCwHduuM/7IuJrM/OezHzK+t93ZOY37NhegO5JhoFutdZ+PSJ+OiJ+NiI+HauWg7fFqu/3w7FaqHZ/a+39rbUHL/5FxG9GxDdn5gs2POe/RsTvxjqhvsY9EfHxdSvEayLiR9e3vzUivjAiHoqIv4qIPz3+VcZ7IuJvIuL+iPijWC2Gu7rNj0XED8Rqcd6nIuLBWCX8T92xvQDdy9baubcBgANkZouIr2mt/fO5twVgqVSGAQAoSzIMAEBZ2iQAAChLZRgAgLIkwwAAlCUZBgCgLMkwAABlSYYBAChLMgwAQFmSYQAAypIMAwBQlmQYAICybj3lH8tMl7vborWW+z5GTLc7JKYR4rqLsTo+Y3Uaxur4jNVpGKvjGxpTlWEAAMqSDAMAUJZkGACAsk7aMwxA31q7uYUx86AWU4CTUBkGAKAsyTAAAGWVa5PYNIUXYRoPAGAs1+VbEfPLuVSGAQAoq8vK8LajkX0eM7cjF2AaPv/HOWSfy3DGJ3N2bM41h7GsMgwAQFmSYQAAyuqmTWJomf5yOX7XYy5+P4cSPjA9n/X97NqHiufhhn4/RYgzyzaHXEtlGACAsrqpDG+y6yhj0+8tBDkfp73jlHzWmaNjFiPZV47LwsXtxt6HnnO2Q2UYAICyJMMAAJTVZZvEMeX1TQvsLFQ4vZ7jvGvqzaIk5uiQRcqMY1dMfVcdb+j41pIy3JJaUVWGAQAoSzIMAEBZ3bRJmLJYLtNO+00dWeHMqWiNWC771ScbayxXb0nptY1PZRgAgLK6qQwzLdXI8Rwbt7kuQKAfKsLzckwVsmIFc8rxu2tRWJUYX9bDa1YZBgCgLMkwAABlaZPgWlO0RpjiP97FeyCWjElrxDzs+nz73G92zoVdFT4TvbdKqgwDAFCWyjCD9HQECOzPPuC09rkq5a7H92xbbKrE4JR6janKMAAAZUmGAQAoS5sENxl7gUavV6w5FwtoOJVzfjarn7t1H+LzZGPFQwvGzY79XA5dAH7qz7/KMAAAZakMX6HqdhpVj6qZJ+NxWsfuV3s/rdMmvov247Sf46sUE5VhAADKkgwDAFBW120SY5X4e5+OG1OlaRVYuus+r8fs88a6kt3QBTa97Z/tQw8zdMGV+N6gfekGlWEAAMqSDAMAUFaXbRJTnid3qVMA+9h0HkAtJ9Cf6y75O7QFYayWiKGP6XWKe5/p5l5jMKYpY9TDd9g5L+895T7jGCrDAACU1U1l+NjqraPtm22qGh1SKerhSHqOxJU5Mi45pX1nEA6puBvTx1lCfqUyDABAWZJhAADKWnybxDHl90Me2+u5LXep9nqBze1nWqGYO+PxZqdaNHdMTnbO901lGACAshZfGb4w9Ihi11HL0AVi1U63ts0UV7ECTuu606xtu21Kx+6rl0CV/bwqjLELuz7fu/R+ujqVYQAAypIMAwBQVjdtEruMNR0y9Opscyj7T63q6z6HJZynkX5sWyy3ySFtY8Y0c+Q77IbeWyMuUxkGAKCsbirDp77W9rHN6ABzt+/VvYb8ft+/B2Pyfb3/zM9Yf2/OVIYBAChLMgwAQFndtEkMNUW5XssEp7SEKSf6tat1wvgcbtuC7F3fK+K8n0rnFB5qrNylh9ipDAMAUJZkGACAshbfJjG38vzctmdspuvo0aapaYYTt/G5zP30xPKGQ2LRU1uoyjAAAGUtvjIMverpqBu4nsVy07APnVZP41JlGACAsiTDAACUpU2Cg/U0RUI9xi9zZFweb1t7hPiyicowAABlSYYBAChLMgwAQFmSYQAAyrKAjr1YfHA6Yg0wDvtTtlEZBgCgLJVhAKAbqsDsS2UYAICyJMMAAJSV267UAgAAPVMZBgCgLMkwAABlSYYBAChLMgwAQFmSYQAAypIMAwBQlmQYAICyJMMAAJQlGQYAoCzJMAAAZUmGAQAoSzIMAEBZkmEAAMqSDAMAUJZkGACAsiTDAACUJRkGAKCsW0/5xzKznfLvLU1rLfd9jJhud0hMI8R1F2N1fMbqNIzV8Rmr0zBWxzc0pirDAACUJRkGAKAsyTAAAGWdtGcYGK617a1gmQe17QEAl6gMAwBQlmQYAICytEkwOdP9wDab9hH2C8CpqAwDAFCWyjCT2VURZjNxowLVYGAuVIYBAChLMgwAQFnaJBjV0Cl+06FPZpHhcY5tLRHf09EeAcyNyjAAAGWpDA9wuZKhgnHDIdU48Xsyi+WOM1b8tj2PMXs81WCmNuW+1Fjtn8owAABlSYYBAChLm8QVu6Zaqk73aYkYj8Vy4xkrVtpVpiGuy3Lxfi1pH3SKMaZV8jBLypdUhgEAKEsyDABAWdok1vZdTX75/kuaCtiHcwYD+9IGNA9VWlQuxtNYrQxV4jaGQ2I111YclWEAAMoqVxk+tmpRoeoxh4pwj9V257KdFxWg0zG+9zf1+OztPent9czVITnQEva1KsMAAJQlGQYAoKwybRI9TruP6ZzxWcIUylSMwdPaZ6x5b4a7Lq5iuL+x94feg2G0sd2sQlvoBZVhAADK6royPFa1s9ejo3NWc445JUvEcmN+SnNYCHlOqsDTM+N2nGOqwOK8n8ozkGPqddypDAMAUJZkGACAsrpskzhFe8SSpwosljudKV/vWM+95PaTfWOwtNc3R9U+w6dmjB7m1Pvaau/TFC2mc6IyDABAWd1UhncdgYx1hFLtaHAMU1/1D4ZachV8zsRyOFXG8zgmxtW+g455vUuNlcowAABlSYYBACirmzaJy1MgruBzs3NMXYy1APHivkudfhnbvucP3iduF/ddypgfup2bYmC6mlMy3qYlluPZ97ujh+9mlWEAAMqSDAMAUFY3bRKXHTN1WskU00q9Xrr63E51aeVe359Nr0vrxH7EBuo65jtoCbmWyjAAAGV1WRkeS+9Vj6Wcc3UJR5XndszR+Jzf+ykdUy3e9TysGIPbHbJvqxqrc6j+3VNprKkMAwBQlmQYAICyyrVJmPLcbG4tExbrHKb6tN6xXBr8MMfGZWnntz7EpnO3VjgXeG/E/ThzjZ/KMAAAZZWpDFevCE95hb5DqhqsbKuAi+v8bHpv5jarcg6HvO6qY3ZTrPaZkag6xk6l6risTmUYAICyJMMAAJTVdZuERVib7ZqGn9s00ZLfs0PaU04V/yXHda6qLW4a+nrntk9ZAjGbjyqf58pUhgEAKKvLyrCK8DBTLqo7dDt6dsjCuDH/Lkzl2DFtjN7g++s8VOJrUxkGAKAsyTAAAGV12SbB/g5ZAGPq7jBTLDbyXkxnn/ej5/fhkDafnuMxJvvXwx0Tu+rXH+AGlWEAAMqSDAMAUFY3bRKmmaYllqcj1vs79nK1VpIPt+ksNMbscKbmz0v82URlGACAsrqpDF/myA5qmfKc2fYn1xOb4baNS3Ec19B9gLhzQWUYAICyJMMAAJS1+DYJC1+Ay0x9MhcWa01v6PmvxZptVIYBAChrkZVhp1EDAC7IAeZnSTP3KsMAAJQlGQYAoKxFtklcZmoEgDny/QTLoDIMAEBZi6wMO9oGAJivJeVqKsMAAJQlGQYAoKxc0nngAABgTCrDAACUJRkGAKAsyTAAAGVJhgEAKEsyDABAWZJhAADKkgwDAFCWZBgAgLIkwwAAlCUZBgCgLMkwAABlSYYBAChLMgwAQFmSYQAAypIMAwBQlmQYAICyJMMAAJQlGQYAoCzJMAAAZUmGAQAoSzIMAEBZkmEAAMqSDAMAUNb/A51pewHVcq7ZAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x216 with 16 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "X = np.load(\"samples-to-compare.npz\")[\"target\"]\n",
    "# before fitting the model, we binarized the images\n",
    "X[X<0.5]=0\n",
    "X[X>=0.5]=1\n",
    "\n",
    "Y = np.load(\"samples-to-compare.npz\")[\"model\"]\n",
    "\n",
    "plt.figure(figsize=(12,3))\n",
    "plt.suptitle(\"Train samples\")\n",
    "\n",
    "for i in np.arange(0,16):\n",
    "    plt.subplot(2,8,i+1)\n",
    "    plt.imshow(X[i],cmap='gray')\n",
    "    plt.axis('off')\n",
    "plt.figure(figsize=(12,3))\n",
    "plt.suptitle(\"GAN samples\")\n",
    "for i in np.arange(0,16):\n",
    "    plt.subplot(2,8,i+1)\n",
    "    plt.imshow(Y[i],cmap='gray')\n",
    "    plt.axis('off')\n",
    "\n",
    "# split off a training set, reshape to vectors\n",
    "N = len(X)\n",
    "assert len(X) == len(Y)\n",
    "size = X.shape[1]\n",
    "size2 = size**2\n",
    "assert size2 == X.shape[2]**2\n",
    "half = int(N/2)\n",
    "X_learn = X[half:].reshape(half,size2)\n",
    "Y_learn = Y[half:].reshape(half,size2)\n",
    "X = X[:half].reshape(half,size2)\n",
    "Y = Y[:half].reshape(half,size2)\n",
    "\n",
    "print(\"Image dimensions: %dx%d=%d\" % (size, size, size2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have some code to compute the ratio we want to optimize, and make TensorFlow's Adam optimizer do the rest for us, i.e optimizing the bandwidth variables, $28^2$ of them.\n",
    "\n",
    "Note: this is a very simplistic implementation that might suffer from all sorts of problems.\n",
    "In practice, we would pay more careful attention to the optimization scheme."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "start, ratio: 0.1268358\n",
      "step, ratio: 0.1310806\n",
      "step, ratio: 0.14533705\n",
      "step, ratio: 0.17821728\n",
      "step, ratio: 0.24888371\n",
      "step, ratio: 0.39294672\n",
      "step, ratio: 0.6644053\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-32-0699506fef2d>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     18\u001b[0m     \u001b[0;32mfor\u001b[0m \u001b[0mstep\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m20\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     19\u001b[0m         \u001b[0;31m# IMPLEMENT: run optimizer and update best found sigmas so far\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 20\u001b[0;31m         \u001b[0msess\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mminimize\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     21\u001b[0m         \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"step,\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"ratio:\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msess\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mratio\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     22\u001b[0m         \u001b[0mbest_sigmas\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msess\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msigmas\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/software/anaconda3/envs/ds3_kernels/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m    898\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    899\u001b[0m       result = self._run(None, fetches, feed_dict, options_ptr,\n\u001b[0;32m--> 900\u001b[0;31m                          run_metadata_ptr)\n\u001b[0m\u001b[1;32m    901\u001b[0m       \u001b[0;32mif\u001b[0m \u001b[0mrun_metadata\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    902\u001b[0m         \u001b[0mproto_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTF_GetBuffer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrun_metadata_ptr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/software/anaconda3/envs/ds3_kernels/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run\u001b[0;34m(self, handle, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m   1133\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mfinal_fetches\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mfinal_targets\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mhandle\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mfeed_dict_tensor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1134\u001b[0m       results = self._do_run(handle, final_targets, final_fetches,\n\u001b[0;32m-> 1135\u001b[0;31m                              feed_dict_tensor, options, run_metadata)\n\u001b[0m\u001b[1;32m   1136\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1137\u001b[0m       \u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/software/anaconda3/envs/ds3_kernels/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_run\u001b[0;34m(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m   1314\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mhandle\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1315\u001b[0m       return self._do_call(_run_fn, feeds, fetches, targets, options,\n\u001b[0;32m-> 1316\u001b[0;31m                            run_metadata)\n\u001b[0m\u001b[1;32m   1317\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1318\u001b[0m       \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_do_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_prun_fn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeeds\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetches\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/software/anaconda3/envs/ds3_kernels/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_call\u001b[0;34m(self, fn, *args)\u001b[0m\n\u001b[1;32m   1320\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0m_do_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1321\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1322\u001b[0;31m       \u001b[0;32mreturn\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1323\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0merrors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mOpError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1324\u001b[0m       \u001b[0mmessage\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcompat\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mas_text\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmessage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/software/anaconda3/envs/ds3_kernels/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run_fn\u001b[0;34m(feed_dict, fetch_list, target_list, options, run_metadata)\u001b[0m\n\u001b[1;32m   1305\u001b[0m       \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_extend_graph\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1306\u001b[0m       return self._call_tf_sessionrun(\n\u001b[0;32m-> 1307\u001b[0;31m           options, feed_dict, fetch_list, target_list, run_metadata)\n\u001b[0m\u001b[1;32m   1308\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1309\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_prun_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/software/anaconda3/envs/ds3_kernels/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_call_tf_sessionrun\u001b[0;34m(self, options, feed_dict, fetch_list, target_list, run_metadata)\u001b[0m\n\u001b[1;32m   1407\u001b[0m       return tf_session.TF_SessionRun_wrapper(\n\u001b[1;32m   1408\u001b[0m           \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_session\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moptions\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtarget_list\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1409\u001b[0;31m           run_metadata)\n\u001b[0m\u001b[1;32m   1410\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1411\u001b[0m       \u001b[0;32mwith\u001b[0m \u001b[0merrors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mraise_exception_on_not_ok_status\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mstatus\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "sigmas = tf.Variable(np.ones(X_learn.shape[1]), dtype=np.float32, name=\"sigmas\")\n",
    "\n",
    "# provided function. note we are using the \"training\" dataset here.\n",
    "ratio = mmd.ard_mmd2_and_ratio(X_learn,Y_learn, bws=sigmas)[1]\n",
    "\n",
    "# IMPLEMENT: set up adam optimizer tensorflow op to maximise the ratio\n",
    "learning_rate = 0.5\n",
    "optimizer = tf.train.AdamOptimizer(learning_rate)\n",
    "minimize = optimizer.minimize(-ratio)\n",
    "\n",
    "init = tf.global_variables_initializer()\n",
    "\n",
    "with tf.Session() as sess:  \n",
    "    sess.run(init)\n",
    "    \n",
    "    print(\"start,\", \"ratio:\", sess.run(ratio))\n",
    "    \n",
    "    for step in range(20):  \n",
    "        # IMPLEMENT: run optimizer and update best found sigmas so far\n",
    "        sess.run(minimize)\n",
    "        print(\"step,\", \"ratio:\", sess.run(ratio))\n",
    "        best_sigmas = sess.run(sigmas)\n",
    "    print(\"done, ratio:\", sess.run(ratio))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we now have learnt one bandwidth per pixel, we can visualise the scaled data to see what the resulting test focusses on.\n",
    "Note that we do this on the actual test data, while we learned the weights on the training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# IMPLEMENT: scale images according to sigmas\n",
    "X_scaled = (X/best_sigmas)\n",
    "Y_scaled = (Y/best_sigmas)\n",
    "\n",
    "plt.suptitle(\"Train samples\")\n",
    "plt.figure(figsize=(12,3))\n",
    "for i in np.arange(0,16):\n",
    "    plt.subplot(2,8,i+1)\n",
    "    plt.imshow(X_scaled[i].reshape(size, size),cmap='gray')\n",
    "    plt.axis('off')\n",
    "plt.figure(figsize=(12,3))\n",
    "plt.suptitle(\"GAN samples\")\n",
    "for i in np.arange(0,16):\n",
    "    plt.subplot(2,8,i+1)\n",
    "    plt.imshow(Y_scaled[i].reshape(size, size),cmap='gray')\n",
    "    plt.axis('off')\n",
    "plt.figure()\n",
    "plt.title(\"ARD weights\")\n",
    "plt.imshow(best_sigmas.reshape(size,size), 'magma')\n",
    "plt.colorbar()\n",
    "plt.axis('off');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, there seem to be pixel level artifacts that the test focuses on.\n",
    "As it turns out, focusing on those allows the test to fully distinguish training samples from GAN samples.\n",
    "\n",
    "Since our above implementation for the MMD permutation test is very inefficient, we don't use it here to perform the full test -- it would take too long.\n",
    "You can use the below Shogun snippet to perform the test much more quickly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import shogun as sg\n",
    "sg.get_global_parallel().set_num_threads(4)\n",
    "\n",
    "sigma = 1\n",
    "kernel_width = 2 * sigma**2\n",
    "mmd = sg.QuadraticTimeMMD()\n",
    "mmd.set_p(sg.RealFeatures(X_scaled.T.astype(np.float64))) # shogun uses column major format\n",
    "mmd.set_q(sg.RealFeatures(Y_scaled.T.astype(np.float64)))\n",
    "mmd.set_kernel(sg.GaussianKernel(10, kernel_width))\n",
    "\n",
    "mmd.set_num_null_samples(num_permutations)\n",
    "statistics = mmd.sample_null()\n",
    "my_statistic = mmd.compute_statistic()\n",
    "plot_permutation_samples(statistics, my_statistic)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# IMPLEMENT (optional): how does this compare to all weights being one.\n",
    "sigma = 1\n",
    "kernel_width = 2 * sigma**2\n",
    "mmd = sg.QuadraticTimeMMD()\n",
    "mmd.set_p(sg.RealFeatures(X.T.astype(np.float64)))\n",
    "mmd.set_q(sg.RealFeatures(Y.T.astype(np.float64)))\n",
    "mmd.set_kernel(sg.GaussianKernel(10, kernel_width))\n",
    "\n",
    "mmd.set_num_null_samples(num_permutations)\n",
    "statistics = mmd.sample_null()\n",
    "my_statistic = mmd.compute_statistic()\n",
    "plot_permutation_samples(statistics, my_statistic)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we see that the unscaled features already allow the test to distinguish the distribution quite well.\n",
    "The scaled version above, however, is a much more powerful test."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Notes\n",
    "\n",
    "Have much more data? Try the linear time versions of the tests\n",
    "* https://arxiv.org/abs/1605.06796\n",
    "* https://arxiv.org/abs/1506.04725\n",
    "\n",
    "\n",
    "Need faster implementations (of the quadratic time MMD tests)\n",
    "* use Shogun!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Independence testing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's move on to independence testing.\n",
    "Here, the data we observe is in pairs $(x_i,y_i)\\sim P_{XY}$..\n",
    "The null hypothesis is that the two sets of samples come from independent distributions: $P_{XY} = P_X\\cdot P_Y$.\n",
    "\n",
    "We will use the HSIC, which you can think of as the MMD between the joint distribution $P_{XY}$ and the product of the marginal distributions $P_X P_Y$. \n",
    "Clearly, if those are different, then the samples cannot be independent.\n",
    "\n",
    "See the slides for more details."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quadratic time test statistic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's kick of with implementing the quadratic time version of the HSIC statistic.\n",
    "\n",
    "$$ \\frac{1}{n^2}\\text{Trace}(KL) $$\n",
    "\n",
    "where $K$ and $L$ are centered kernel matrices of the samples from the two distribution respectively.\n",
    "(Note that one can center a kernel matrix via multiplying it with a centering matrix $H=I-\\frac{1}{n}\\mathbf{1}\\mathbf{1}^\\top$.)\n",
    "\n",
    "As our kernel will quite expensive to be computed, we here write a version that accepts precomputed kernel matrices rather than a kernel function, and we will permute this kernel matrix for the permutation test.\n",
    "We implement the test statistic as a function of kernel matrices, and with an option to permute the matrices before computing the statistic (for approximating the null distribution).\n",
    "\n",
    "Note that the permutation for dependence testing is different from two-sample testing: in two sample testing, we could just permute the joint kernel matrix (as there is a kernel between $X$ and $Y$).\n",
    "For dependence testing, however, the kernel only is defined between $X$ and itself and $Y$ and itself.\n",
    "\"Breaking\" any potential structure therefore means that we need to shuffle the samples within $X$ and within $Y$ so that any potential pairs $(x_i,y_i)$ do not occur together anymore."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def HSIC(K_XX, K_YY, permute=False):\n",
    "    assert len(K_XX) == len(K_YY)\n",
    "    N = len(K_XX)\n",
    "    if permute:\n",
    "        # IMPLEMENT: permute the kernel matrices for X and Y\n",
    "        inds_X = np.random.permutation(len(K_XX))\n",
    "        inds_Y = np.random.permutation(len(K_YY))\n",
    "        K_XX_ = K_XX[inds_X, :][:, inds_X]\n",
    "        K_YY_ = K_YY[inds_Y, :][:, inds_Y]\n",
    "    else:\n",
    "        K_XX_ = K_XX\n",
    "        K_YY_ = K_YY\n",
    "\n",
    "    # IMPLEMENT: HSIC statistic\n",
    "    H = np.eye(N) - 1.0/N\n",
    "    statistic = np.trace(K_XX_.dot(H).dot(K_YY_.dot(H))) / (N**2)\n",
    "    return statistic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start with a simple dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "data = np.load(\"hsic2.npz\")\n",
    "X = data[\"X\"]\n",
    "Y = data[\"Y\"]\n",
    "plt.plot(X,Y,'.')\n",
    "plt.axis('equal')\n",
    "plt.grid(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clearly, there is some very simple dependence going on.\n",
    "The first thing a statistician in the field would do is of course to check for correlation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# IMPLEMENT: produce a seaborn pairplot to show the correlation between X and Y\n",
    "print(\"Correlation between X and Y:\", np.corrcoef(X,Y)[0,1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The correlation is very low.\n",
    "A kernel test for independence is more powerful, even with a bandwidth parameter chosen by the median heuristic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# IMPLEMENT: compute kernel matrices using the median heuristic\n",
    "sigma_median = gaussian_kernel_median_heuristic(np.vstack((X[:,np.newaxis],Y[:,np.newaxis])))\n",
    "my_kernel = lambda X,Y : gauss_kernel(X[:,np.newaxis],Y[:,np.newaxis],sigma=sigma_median)\n",
    "K_XX = my_kernel(X,X)\n",
    "K_YY = my_kernel(Y,Y)\n",
    "K_XY = my_kernel(X,Y)\n",
    "\n",
    "statistics = np.empty(num_permutations)\n",
    "for i in tqdm(range(num_permutations)):\n",
    "    # IMPLEMENT: compute HSIC statistic with permuted matrices\n",
    "    statistics[i] = HSIC(K_XX, K_YY, permute=True)\n",
    "    \n",
    "# IMPLEMENT: visualise test\n",
    "my_statistic = HSIC(K_XX, K_YY)\n",
    "plot_permutation_samples(statistics, my_statistic)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Victory once again."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## European parliament documents translations and string kernels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now do a slightly more elaborate test that involves some very mild NLP.\n",
    "More precisely we will analyze dependence between documents.\n",
    "We will use transcripts of the Canadian parliament's house debates, downloaded from [here](https://www.isi.edu/natural-language/download/hansard/).\n",
    "Those consist of pairs of French and English transcripts.\n",
    "\n",
    "Our question here is whether we can detect this supposedly strong dependence structure using the kernel HSIC.\n",
    "Note that this approach does not rely on attempting to translate the documents, but rather on comparing within-document structure. \n",
    "HSIC compares the self-similarity within the English documents with self-similarity of the French ones."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In a first step, we load the data, and store the transcript of each session into lists X and Y, for English and French respectively.\n",
    "We don't do any sophisticated preprocessing here, but feel free to make this more elaborate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# download english and french stopwords (optional)\n",
    "try:\n",
    "    import nltk\n",
    "    from nltk.corpus import stopwords\n",
    "    nltk.download('stopwords');\n",
    "    stop_english = set(stopwords.words(\"english\"))\n",
    "    stop_french = set(stopwords.words(\"french\"))\n",
    "except Exception:\n",
    "    stop_english = set()\n",
    "    stop_french = set()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from glob import glob\n",
    "import re\n",
    "\n",
    "X = []\n",
    "Y = []\n",
    "for fname in tqdm(sorted(list(glob(\"transcripts/hansard.36.1.house.debates.*.*\")))):\n",
    "    is_french = fname.endswith(\"f\")\n",
    "    is_english = fname.endswith(\"e\")\n",
    "    \n",
    "    if not (is_english or is_french):\n",
    "        continue\n",
    "        \n",
    "    with open(fname, \"r\", encoding='latin-1') as f:\n",
    "        # some very very simple preprocessing, could be more elaborate with special characters\n",
    "        content = \"\".join(f.readlines())\n",
    "        if is_english:\n",
    "            letters_only = re.sub(\"[^a-zA-Z]\", \" \", content)\n",
    "        elif is_french:\n",
    "            letters_only = re.sub(\"[^a-zA-Z\\'ÁÀÂÉÈÊéèáàêâúùûÚÙÛçîô']\", \" \", content) \n",
    "        words = letters_only.lower().split()\n",
    "        \n",
    "        if is_english:\n",
    "            stops = stop_english\n",
    "        elif is_french:\n",
    "            stops = stop_french\n",
    "        meaningful_words = [w for w in words if not w in stops]\n",
    "        \n",
    "        result = \" \".join(meaningful_words)\n",
    "        \n",
    "        if fname.endswith(\"e\"):\n",
    "            X += [result]\n",
    "        else:\n",
    "            Y += [result]\n",
    "N = len(X)\n",
    "assert N == len(Y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at some excerpts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(X[0][-100:])\n",
    "print(Y[0][-100:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we are going to construct a string kernel, a  \"bag of words\" kernel between documents $s$ and $t$,\n",
    "\n",
    "$$\n",
    "k(s,t) = \\frac{1}{|\\mathcal{W}|} \\phi(s)^\\top \\phi(t)\n",
    "$$\n",
    "\n",
    "where $\\mathcal{W}$ consists of all words on all considered documents, and each element of $\\phi(x)\\in\\mathbb{N}^{|\\mathcal{W}|}$ contains the number of times that a particular word $w\\in\\mathcal{W}$ appears in $x$.\n",
    "\n",
    "Naturally, the kernel value will be larger, if a word appears in both documents many times.\n",
    "\n",
    "Efficient implementations of string kernels are rare (Shogun has many!), and often based on low-level dynamic programming concepts.\n",
    "Instead, we will here explicitly embed the documents into a feature space and compute (gram) matrix of inner products manually. \n",
    "Note that most string kernel implementations are much more efficient in computing the kernel since they don't do the feature space mapping explicitly.\n",
    "\n",
    "Let's start by building a document embedding mechanism using scikit-learn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "# IMPLEMENT: fit a vectorizer for the joint set of documents\n",
    "vectorizer = CountVectorizer(analyzer = \"word\",\n",
    "                             decode_error = 'replace',\n",
    "                             tokenizer = None,    \n",
    "                             preprocessor = None, \n",
    "                             stop_words = None,   \n",
    "                             max_features = 50000)\n",
    "vectorizer.fit(X+Y),"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can look at associated words for each feature space dimension."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "feature_names = np.array(vectorizer.get_feature_names())\n",
    "np.random.seed(0)\n",
    "list(feature_names[np.random.permutation(len(feature_names))[-10:]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's define the bag of words kernel, as a product of embedded documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def bag_of_words_kernel(fitted_vectoriser, X,Y):\n",
    "    \"\"\"\n",
    "    Computes the gram matrix between the features of X and Y, where the features are computed\n",
    "    using the provided (fitted) sklearn transformer.\n",
    "    \n",
    "    The kernel is k(x,y) = 1/dim(x) * \\phi(x) \\cdot \\phi(y)\n",
    "    \"\"\"\n",
    "    # IMPLEMENT: transform X and Y documents\n",
    "    X_feats = fitted_vectoriser.transform(X)\n",
    "    if X is Y :\n",
    "        Y_feats = X_feats\n",
    "    else:\n",
    "        Y_feats = fitted_vectoriser.transform(Y)\n",
    "    \n",
    "    # IMPLEMENT: compute (dense) gram matrix of pariwise inner products of feature vectors\n",
    "    K = X_feats.dot(Y_feats.T).todense() / len(vectorizer.get_feature_names())\n",
    "    return K"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at the kernel matrices (takes some time to compute)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# IMPLEMENT: compute kernel matrices between feature vectors using the bag of words kernel\n",
    "my_kernel = lambda X,Y: bag_of_words_kernel(vectorizer, X,Y)\n",
    "K_XX = my_kernel(X,X)\n",
    "K_YY = my_kernel(Y,Y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(12,4))\n",
    "plt.subplot(121)\n",
    "plt.imshow(K_XX)\n",
    "plt.subplot(122)\n",
    "plt.imshow(K_YY)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is a lot of shared (self-similarity) structure between X and Y.\n",
    "The HSIC test quantifies this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# IMPLEMENT: visualise HSIC test for given kernel matrices\n",
    "statistics = np.empty(num_permutations)\n",
    "for i in tqdm(range(num_permutations)):\n",
    "    statistics[i] = HSIC(K_XX, K_YY, permute=True)\n",
    "my_statistic = HSIC(K_XX, K_YY)\n",
    "plot_permutation_samples(statistics, my_statistic)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A clear reject of the null hypothesis that X and Y are independent."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Notes\n",
    "\n",
    "As for the MMD, there are various linear time tests available for the HSIC, in particular https://arxiv.org/abs/1610.04782."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Goodness of fit testing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final topic is testing whether a given set of samples comes from a specified distribution.\n",
    "This problem is slightly different from two-sample and independence testing as we are only given a single set of samples, and the reference distribution will be only available through the gradient of the log density (we cannot sample from it).\n",
    "\n",
    "But there are also similarities, i.e. the test statistic is the RKHS norm of a witness function, we can evaluate the witness function, there is a simple kernel-based quadratic time statistic, etc.\n",
    "\n",
    "Unfortunately, as we cannot sample from the reference distribution, we cannot permute our samples in any way to approximate the null distribution.\n",
    "We will use a slightly more sophisticated (still sampling based) way, called the wild bootstrap.\n",
    "This method was originally developed to approximate null distributions for two-sample tests of correlated samples, however, it turns out that we can also utilise the technique for cases where we cannot permute samples.\n",
    "\n",
    "We start with a simple example where we provide you with code to compute test statistic and p-value.\n",
    "The reest of his section is intentionally kept more sparse, encouraging faster participants to implement more advanced concepts on their own.\n",
    "\n",
    "As usual, see the slides for details.\n",
    "\n",
    "The two main references are\n",
    " * https://arxiv.org/abs/1602.02964\n",
    " * https://arxiv.org/abs/1705.07673 (NIPS 2017 best paper award)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing samples of a student-t against a Gaussian\n",
    "\n",
    "We will begin with a simple synthetic example: given a number of samples from a student-t distribution, can we reject the null hypothesis that those samples came from a Gaussian.\n",
    "How does the test change as we make the student-t look more and more like a Gaussian?\n",
    "\n",
    "Remember, we will only use the gradient of the log-density of the reference Gaussian."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def grad_log_normal(x):\n",
    "    \"\"\"\n",
    "    Returns the gradient of the log Gaussian density (zero mean, std-dev)\n",
    "    \"\"\"\n",
    "    \n",
    "    # IMPLEMENT:\n",
    "    grad = -x\n",
    "    \n",
    "    return  grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Distributions of p-values\n",
    "\n",
    "In the beginning of this notebook, we saw that every time a statistical test is computed, we only receive a single test statistic.\n",
    "This is a single sample from the alternative distribution, and leads to a single p-value  (remember the example where we explicitly sampled from the alternative distribution?).\n",
    "\n",
    "Again, this number clearly depends on the data used for the test, a different set of test data will lead to a different test statistic and p-value (while the estimated null distribution should be stable).\n",
    "Under the null hypothesis of any test, the distribution of p-values is uniform, since we just take a sample from the very same distribution and then compute its quantile.\n",
    "Long story short: we never know whether we just have been \"lucky\" when computing the test.\n",
    "\n",
    "In order to get a feeling for this effect, we will now run the goodness-of-fit test multiple times and plot the distribution over p-values.\n",
    "We will perform this experiment using samples from a [student-t distribution](https://en.wikipedia.org/wiki/Student%27s_t-distribution), and test the null hypothesis that these samples are distributed as a standard Gaussian.\n",
    "As the degrees of freedom are increasing, the samples look more and more Gaussian, and we will quantify the distribution of p-values along the way.\n",
    "\n",
    "Since we need to compute the test many times to sample the distribution of p-values, the quadratic time statistic would take a long time.\n",
    "This is why we have provided an implementation of a linear-time goodness-of-fit test using the kernel Stein discrepancy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import itertools\n",
    "from ksd import GaussianSteinTest\n",
    "\n",
    "# parameter for the linear time test (see paper if interested)\n",
    "num_random_freq=2\n",
    "\n",
    "N = 500\n",
    "\n",
    "degrees_of_freedom = range(1, 10, 2)\n",
    "num_repetitions = 200\n",
    "results = []\n",
    "\n",
    "for dof, mc in tqdm(list(itertools.product(degrees_of_freedom, range(num_repetitions)))):\n",
    "    # IMPLEMENT: sample from student t with given degrees of freedom (dof)\n",
    "    X = np.random.standard_t(dof,N)\n",
    "    \n",
    "    # this is how we run the linear time goodness-of-fit test\n",
    "    pvalue = GaussianSteinTest(grad_log_normal,num_random_freq).compute_pvalue(X)\n",
    "    results += [[dof, pvalue]]\n",
    "\n",
    "for mc in tqdm(range(num_repetitions)):\n",
    "        # IMPLEMENT: run the test to sample from the null distribution\n",
    "        X = np.random.randn(N)\n",
    "        pvalue = GaussianSteinTest(grad_log_normal,num_random_freq).compute_pvalue(X)\n",
    "        \n",
    "        results += [[np.Inf, pvalue]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame(results, columns=[\"dof\",\"p_value\"])\n",
    "sns.boxplot(y=\"p_value\",x=\"dof\",data=df);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that for a low number of degrees of freedom, the p-values tend to be mostly small (i.e. the test rejects the null hypothesis more often).\n",
    "As we make the distribution look more like a Gaussian, the p-values increase.\n",
    "Eventually, the student-t becomes a Gaussian for an infinite number of degrees of freedom, i.e. we are sampling from the (unitform!) distribution of p-values under the null hypothesis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quadratic time test statistic\n",
    "\n",
    "Let's implement the quadratic time statistic to get a feeling for what quantities are needed: the gradient of the target log-density, the kernel, and the gradient of the kernel with respect to left and right arguments (individually and both).\n",
    "We will use a standard Gaussian kernel with unit scaling, for simplicity in 1D.\n",
    "The first step is to implement kernel gradients (we could do this using an autodiff framework!).\n",
    "\n",
    "See https://arxiv.org/abs/1602.02964 or the slides for the definition of the test statistic. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def k( x, y):\n",
    "    return np.exp(-np.dot(x - y,x - y))\n",
    "\n",
    "def g1k(x, y):\n",
    "    # IMPLEMENT: gradient wrt left argument\n",
    "    return -2.0 *  k(x, y) * (x - y)\n",
    "\n",
    "def g2k(x, y):\n",
    "    # IMPLEMENT: gradient wrt right argument\n",
    "    return -g1k(x, y)\n",
    "\n",
    "def gk(x, y):\n",
    "    # IMPLEMENT: gradient wrt to both arguments\n",
    "    return 2.0 * k(x, y) * ( - 2 * (x - y) ** 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from ksd import GaussianQuadraticTest\n",
    "\n",
    "X = np.random.standard_t(1,N)\n",
    "kernel_bw = 1.0\n",
    "# reference implementation: the function returns N*statistic. Nevermind the typo :)\n",
    "print(\"Reference\", GaussianQuadraticTest(grad_log_normal, scaling=kernel_bw).get_statistic_multiple(X)[1]/N)\n",
    "\n",
    "# all elements that the statistic sums over\n",
    "U_matrix = np.zeros((N, N))\n",
    "\n",
    "for i in tqdm(range(N)):\n",
    "    for j in range(N):\n",
    "        # IMPLEMENT: implement the quadratic time KSD test statistic (needs gradient of log-density and the gradient of the Gaussian kernel)\n",
    "        # hint: use four terms\n",
    "        x1 = X[i]\n",
    "        x2 = X[j]\n",
    "        a = grad_log_normal(x1) * grad_log_normal(x2) * k(x1, x2)\n",
    "        b = grad_log_normal(x2) * g1k(x1, x2)\n",
    "        c = grad_log_normal(x1) * g2k(x1, x2)\n",
    "        d = gk(x1, x2)\n",
    "        U_matrix[i, j] = a + b + c + d\n",
    "    \n",
    "stat = np.mean(U_matrix)\n",
    "print(\"My implementation:\", stat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you like, try to visualize the witness function of the kernel Stein discrepancy, just as we visualized the witness function of the MMD above (see slides for details)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Advanced exercise\n",
    "\n",
    "Let's see if we can quantify convergence of an MCMC chain using the goodness-of-fit test.\n",
    "For simplification, we will thin the MCMC samples so that they are not correlated (see the original KSD paper on how to run the test on correlated samples).\n",
    "This experiment loosely follows one in the KSD paper.\n",
    "\n",
    "In this open-ended las task you will:\n",
    "\n",
    "* Run an MCMC chain using stochastic gradient Langevin dynamics with a large step-size. See https://www.ics.uci.edu/~welling/publications/papers/stoclangevin_v6.pdf\n",
    "* Target a standard Gaussian distribution in 1D (or 2D if you like)\n",
    "* After the sampler is completed, thin the MCMC samples so that they are almost i.i.d.\n",
    "* Run the goodness-of-fit test to quantify at what step-size of the sampler the samples reach a good quality.\n",
    "\n",
    "Good luck :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The end"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}