8.2.2dropoutRegression_v2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center>\n",
    "    <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-DL0110EN-SkillsNetwork/Template/module%201/images/IDSNlogo.png\" width=\"300\" alt=\"cognitiveclass.ai logo\"  />\n",
    "</center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>Using Dropout in Regression</h1>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Objective for this Notebook<h3>    \n",
    "<h5> 1. Create the Model and Cost Function the PyTorch way.</h5>\n",
    "<h5> 2. Learn Batch Gradient Descent </h5>    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Table of Contents</h2>\n",
    "<p>In this lab, you will see how adding dropout to your model will decrease overfitting.</p>\n",
    "\n",
    "<ul>\n",
    "    <li><a href=\"#Makeup_Data\">Make Some Data</a></li>\n",
    "    <li><a href=\"#Model_Cost\">Create the Model and Cost Function the PyTorch way</a></li>\n",
    "    <li><a href=\"#BGD\">Batch Gradient Descent</a></li>\n",
    "</ul>\n",
    "<p>Estimated Time Needed: <strong>20 min</strong></p>\n",
    "\n",
    "<hr>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Preparation</h2>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll need the following libraries\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch._C.Generator at 0x7f8a9d2260f0>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Import the libraries we need for the lab\n",
    "\n",
    "import torch\n",
    "import matplotlib.pyplot as plt\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import numpy as np\n",
    "from torch.utils.data import Dataset, DataLoader\n",
    "\n",
    "torch.manual_seed(0) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--Empty Space for separating topics-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 id=\"Makeup_Data\">Make Some Data</h2>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create polynomial dataset class: \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create Data object\n",
    "\n",
    "class Data(Dataset):\n",
    "    \n",
    "    # Constructor\n",
    "    def __init__(self, N_SAMPLES=40, noise_std=1, train=True):\n",
    "        self.x = torch.linspace(-1, 1, N_SAMPLES).view(-1, 1)\n",
    "        self.f = self.x ** 2\n",
    "        if train != True:\n",
    "            torch.manual_seed(1)\n",
    "            self.y = self.f + noise_std * torch.randn(self.f.size())\n",
    "            self.y = self.y.view(-1, 1)\n",
    "            torch.manual_seed(0)\n",
    "        else:\n",
    "            self.y = self.f + noise_std * torch.randn(self.f.size())\n",
    "            self.y = self.y.view(-1, 1)\n",
    "            \n",
    "    # Getter\n",
    "    def __getitem__(self, index):    \n",
    "        return self.x[index], self.y[index]\n",
    "    \n",
    "    # Get Length\n",
    "    def __len__(self):\n",
    "        return self.len\n",
    "    \n",
    "    # Plot the data\n",
    "    def plot(self):\n",
    "        plt.figure(figsize = (6.1, 10))\n",
    "        plt.scatter(self.x.numpy(), self.y.numpy(), label=\"Samples\")\n",
    "        plt.plot(self.x.numpy(), self.f.numpy() ,label=\"True Function\", color='orange')\n",
    "        plt.xlabel(\"x\")\n",
    "        plt.ylabel(\"y\")\n",
    "        plt.xlim((-1, 1))\n",
    "        plt.ylim((-2, 2.5))\n",
    "        plt.legend(loc=\"best\")\n",
    "        plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a dataset object:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZsAAAJRCAYAAACTAxq3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAA9jUlEQVR4nO3deZhU1Z3/8feh2ZrNVlxpQDAibiCYFhdMgivKGGFcoo4xJpMJWUYTMxMmMslkMsaM/oZkEjNZTWI0iaMmEyRGiRjFPS6sgoi4ICrduKEgaCPb+f1xq7GBpumm69at5f16nnq66tate78U1f2pe+6554QYI5IkpalT1gVIksqfYSNJSp1hI0lKnWEjSUqdYSNJSp1hI0lKXWZhE0IYEEK4N4SwOISwKITwpRbWGRNCWB1CmJ+7fSOLWiVJHdM5w31vBP45xjg3hNAbmBNC+EuM8alt1nswxnhGBvVJkvIksyObGOOKGOPc3P01wGKgNqt6JEnpKYpzNiGEQcBI4LEWnj42hPBECOHPIYTDCluZJCkfsmxGAyCE0Av4A3BZjPHtbZ6eC+wfY1wbQhgHTAOG7GA7E4GJAD179vzgwQcfnF7RklQh5syZ80aMca+ObidkOTZaCKELcDswI8b4321YfxlQF2N8o7X16urq4uzZs/NTpCRVsBDCnBhjXUe3k2VvtAD8Eli8o6AJIeybW48QwiiSelcWrkpJUj5k2Yw2GrgIWBhCmJ9b9q/AQIAY40+Bc4DPhxA2Ao3A+dFhqiWp5GQWNjHGh4Cwk3V+CPywMBVJktKSeQcBSZVlw4YNLF++nHXr1mVdiprp3r07/fv3p0uXLqls37CRVFDLly+nd+/eDBo0iNwpWWUsxsjKlStZvnw5gwcPTmUfRXGdjaTKsW7dOvr27WvQFJEQAn379k31aNOwkVRwBk3xSfv/xLCRVFFWrlzJiBEjGDFiBPvuuy+1tbVbHq9fvz4v+xgzZgxDhw7dst3/+7//y8t2AebPn8/06dO3PL7tttu4+uqr87b9tHjORlJF6du3L/Pnzwfgm9/8Jr169eIrX/nKluc3btxI584d/9N44403UlfX4WshtzN//nxmz57NuHHjADjzzDM588wz876ffPPIRlLF++QnP8k//dM/ccIJJ/DVr36Vb37zm3znO9/Z8vzhhx/OsmXLAPjtb3/LqFGjGDFiBJ/97GfZtGlTm/fR/AinV69eANx3332MGTOGc845h4MPPpgLL7yQpssJZ82axXHHHccRRxzBqFGjWL16Nd/4xje45ZZbGDFiBLfccgvXX389l1xyCQAvvvgiJ510EsOHD+ekk07ipZde2rLvL37xixx33HEccMABeT3SaivDRpKAZ555hrvvvpvvfve7O1xn8eLF3HLLLTz88MPMnz+fqqoqbrzxxhbXvfDCC7c0o61c2frAJ/PmzeP73/8+Tz31FEuXLuXhhx9m/fr1nHfeeVxzzTU88cQT3H333fTs2ZMrrriC8847j/nz53PeeedttZ1LLrmET3ziEyxYsIALL7yQL37xi1ueW7FiBQ899BC33347l19+eTvemfywGU1SduZcBm/Nz+82dx8BH/x+u1927rnnUlVV1eo699xzD3PmzOGoo44CoLGxkb333rvFddvTjDZq1Cj69+8PwIgRI1i2bBm77bYb++2335Z99enTZ6fbeeSRR5g6dSoAF110Ef/yL/+y5bkJEybQqVMnDj30UF599dU21ZVPho0kAT179txyv3PnzmzevHnL46YuwTFGLr74Yq666qp2b7/5NmOMW3VG6Nat25b7VVVVbNy4kRhjh3uINX99831kMeqXYSMpO7twBFIIgwYN4vbbbwdg7ty5vPDCCwCcdNJJjB8/ni9/+cvsvffevPnmm6xZs4b999+/TducM2cOH/vYx/jjH//Ihg0bWl3/4IMPpqGhgVmzZnHUUUexZs0aqqur6d27N2vWrGnxNccddxw333wzF110ETfeeCPHH398O//l6fGcjSRt4+yzz+bNN99kxIgR/OQnP+Gggw4C4NBDD+XKK6/k1FNPZfjw4ZxyyimsWLGiTdv8zGc+w/3338+oUaN47LHHtjqSaknXrl255ZZbuPTSSzniiCM45ZRTWLduHSeccAJPPfXUlg4Czf3gBz/gV7/6FcOHD+c3v/kN11xzza69ASnIdD6btDifjVS8Fi9ezCGHHJJ1GWpBS/83JT+fjSSpchg2kqTUGTaSpNQZNpKk1Bk2kqTUGTaSpNQZNpIq0re//W0OO+wwhg8fzogRI3jsscdS29eYMWOo9MsxHEFAUsV55JFHuP3225k7dy7dunXjjTfeyNtcNmqZRzaSKs6KFSvYc889t4wXtueee9KvXz+uuOIKjjrqKA4//HAmTpy4ZQyxMWPG8OUvf5kPf/jDHHLIIcyaNYuzzjqLIUOG8PWvfx2AZcuWcfDBB3PxxRczfPhwzjnnHN59993t9n3XXXdx7LHHcuSRR3Luueeydu1aAC6//HIOPfRQhg8fvtX8OuXCsJFU1KbNq2f01TMZfPkdjL56JtPm1Xd4m6eeeiovv/wyBx10EF/4whe4//77gWSI/lmzZvHkk0/S2Ni4ZXw0SIaPeeCBB/jc5z7H+PHj+dGPfsSTTz7J9ddfv2UKgSVLljBx4kQWLFhAnz59+PGPf7zVft944w2uvPJK7r77bubOnUtdXR3//d//zZtvvsmtt97KokWLWLBgwZYAKyeGjaSiNW1ePZOnLqR+VSMRqF/VyOSpCzscOL169WLOnDlce+217LXXXpx33nlcf/313HvvvRx99NEMGzaMmTNnsmjRoi2vaZoNc9iwYRx22GHst99+dOvWjQMOOICXX34ZgAEDBjB69GgAPv7xj/PQQw9ttd9HH32Up556itGjRzNixAhuuOEGXnzxRfr06UP37t35h3/4B6ZOnUqPHj069O8rRp6zkVS0psxYQuOGrWfCbNywiSkzljBhZG2Htl1VVcWYMWMYM2YMw4YN42c/+xkLFixg9uzZDBgwgG9+85tbphaA94fo79Sp01bD9Xfq1ImNGzcCbDclwLaPY4yccsop3HTTTdvV8/jjj3PPPfdw880388Mf/pCZM2d26N9XbDyykVS0GlY1tmt5Wy1ZsoRnn312y+P58+czdOhQIDl/s3bt2l2aOvmll17ikUceAeCmm27aboj/Y445hocffpjnnnsOgHfffZdnnnmGtWvXsnr1asaNG8f3v/995s+fv4v/suLlkY2kotWvppr6FoKlX011h7a7du1aLr30UlatWkXnzp058MADufbaa6mpqWHYsGEMGjRoywyZ7XHIIYdwww038NnPfpYhQ4bw+c9/fqvn99prL66//nouuOAC3nvvPQCuvPJKevfuzfjx41m3bh0xRr73ve916N9XjJxiQFJBtWeKgaZzNs2b0qq7VHHVWcM63IyWb8uWLeOMM87gySefzLqUXZbmFAMe2UgqWk2BMmXGEhpWNdKvpppJY4cWXdBo5wwbSUVtwsjakgiXQYMGlfRRTdrsICBJSp1hI6ngyvFccalL+//EsJFUUN27d2flypUGThGJMbJy5Uq6d++e2j48ZyOpoPr378/y5ct5/fXXsy5FzXTv3p3+/funtn3DRlJBdenShcGDB2ddhgrMZjRJUuoMG0lS6gwbSVLqDBtJUuoMG0lS6gwbSVLqDBtJUuoMG0lS6gwbSVLqDBtJUuoMG0lS6gwbSVLqDBtJUuoMG0lS6gwbSVLqDBtJUuoMG0lS6jILmxDCgBDCvSGExSGERSGEL7WwTggh/CCE8FwIYUEI4cgsapUkdUyW00JvBP45xjg3hNAbmBNC+EuM8alm65wODMndjgZ+kvspSSohmR3ZxBhXxBjn5u6vARYDtdusNh74dUw8CtSEEPYrcKmSpA4qinM2IYRBwEjgsW2eqgVebvZ4OdsHkiSpyGUeNiGEXsAfgMtijG9v+3QLL4k72M7EEMLsEMLs119/Pd9lSpI6INOwCSF0IQmaG2OMU1tYZTkwoNnj/kBDS9uKMV4bY6yLMdbttdde+S9WkrTLsuyNFoBfAotjjP+9g9VuAz6R65V2DLA6xriiYEVKkvIiy95oo4GLgIUhhPm5Zf8KDASIMf4UmA6MA54D3gU+VfgyJUkdlVnYxBgfouVzMs3XicA/FqYiSVJaMu8gIEkqf4aNJCl1ho0kKXWGjSQpdYaNJCl1WXZ9llRA0+bVM2XGEhpWNdKvpppJY4cyYaSjP6kwDBupAkybV8/kqQtp3LAJgPpVjUyeuhDAwFFB2IwmVYApM5ZsCZomjRs2MWXGkowqUqUxbKQK0LCqsV3LpXwzbKQK0K+mul3LpXwzbKQKMGnsUKq7VG21rLpLFZPGDs2oIlUaOwhIFaCpE4C90ZQVw0aqEBNG1houyozNaJKk1Bk2kqTUGTaSpNQZNpKk1Bk2kqTUGTaSpNQZNpKk1Bk2kqTUGTaSpNQZNpKk1Bk2kqTUGTaSpNQZNpKk1Bk2kqTUGTaSpNQZNpKk1Bk2kqTUGTaSpNQZNpKk1Bk2kqTUGTaSpNQZNpKk1Bk2kqTUGTaSpNQZNpKk1HXOugBJKnbT5tUzZcYSGlY10q+mmkljhzJhZG3WZZUUw0aSWjFtXj2Tpy6kccMmAOpXNTJ56kIAA6cdbEaTpFZMmbFkS9A0adywiSkzlmRUUWkybCSpFQ2rGtu1XC0zbCSpFf1qqtu1XC0zbCSpFZPGDqW6S9VWy6q7VDFp7NCMKipNdhCQpFY0dQKwN1rHGDaStBMTRtYaLh1kM5okKXWGjSQpdYaNJCl1ho0kKXWGjSQpdZmGTQjhuhDCayGEJ3fw/JgQwuoQwvzc7RuFrlGS1HFZd32+Hvgh8OtW1nkwxnhGYcqRJKUh0yObGOMDwJtZ1iBJSl/WRzZtcWwI4QmgAfhKjHFR1gXtjHNfSNLWij1s5gL7xxjXhhDGAdOAIS2tGEKYCEwEGDhwYMEK3JZzX0jS9oq6N1qM8e0Y49rc/elAlxDCnjtY99oYY12MsW6vvfYqaJ3NOfeFJG2vqMMmhLBvCCHk7o8iqXdltlW1zrkvJGl7mTajhRBuAsYAe4YQlgP/DnQBiDH+FDgH+HwIYSPQCJwfY4wZldsm/WqqqW8hWJz7QlIlyzRsYowX7OT5H5J0jS4Zk8YO3eqcDTj3hSQVeweBkuPcF5K0PcMmBc59IUlbK+oOApKk8mDYSJJSZ9hIklJn2EiSUmfYSJJSZ9hIklJn2EiSUmfYSJJS50WdKirOBSSVJ8NGRcO5gKTyZTOaioZzAUnly7BR0XAuIKl8GTYqGjua88e5gKTSZ9ioaEwaO5TqLlVbLXMuIKk82EFARcO5gKTyZdioqDgXkFSebEaTJKXOsJEkpc6wkSSlzrCRJKXOsJEkpc7eaBlxwElJlcSwyYADTkqqNIZNBlobcNKwkfLHFoTiYdhkwAEnpfTZglBc7CCQAQeclNLnlBXFxbDJgANOSumzBaG42IyWAQec7Bjb4dUW/WqqqW8hWGxByIZhkxEHnNw1tsOrrSaNHbrVZwVsQciSzWgqKbbDq60mjKzlqrOGUVtTTQBqa6q56qxhfinJiEc2Kim2w6s9bEEoHh7ZqKTYk08qTYaNSoo9+aTSZDOaSoo9+aTSZNio5NgOL5Uem9EkSakzbCRJqTNsJEmpM2wkSakzbCRJqTNsJEmpM2wkSakzbCRJqTNsJEmpM2wkSakzbCRJqTNsJEmpM2wkSakzbCRJqcs0bEII14UQXgshPLmD50MI4QchhOdCCAtCCEcWukZJUsdlfWRzPXBaK8+fDgzJ3SYCPylATZKkPMs0bGKMDwBvtrLKeODXMfEoUBNC2K8w1UmS8iXrI5udqQVebvZ4eW7ZTsSUypGkCtIwI2+bKvawCS0sazFJQggTQwizQwizG1c+k3JZklTm3nkJHj4/b5sr9rBZDgxo9rg/0NDSijHGa2OMdTHGuuqwFpb/qSAFSlLZ2bwR/nohxI1522Sxh81twCdyvdKOAVbHGFfs9FVV1fDYp+DdFnNJktSaRd+G1x+Co36ct01m3fX5JuARYGgIYXkI4dMhhM+FED6XW2U6sBR4Dvg58IU2bbjXAbCxER75BMTNaZQuSeXptYfgyStg0Mdh8EV522znvG1pF8QYL9jJ8xH4x3ZvuKo7fPAaePwzsHgKHPrVXS1RkirH+reS5rOeg+CoH+V108XejLbrPvBpGHAOPPF1eOPxrKuRpOIWIzz+WWhsgONugi598rr58g2bEODoa6G6H/z172DD21lXJEnFa+l18NLvYfi3YM9Red98+YYNQNfd4bgb4Z0XYFb7W+MkqSKsfhpmfxH2OREO/ZdUdlHeYQOw9/Fw2L/Bst/CC7/NuhpJKi6b3oO/XgCdq+HY30BIJxbKP2wADv867DUaZn0e1jyfdTWSVDzmT4a35sPRv4Ie/VLbTWWETafOSXNaqErO32zekHVFkpS9hjthyfdgyD9C/4+muqvKCBuAnvvD0T+HlY/Dgm9kXY0kZavxVXj0YqgZBiOnpL67ygkbgIHnwgf+AZ76f0miS1Il2rwJHvl40kv3uJuS8zUpq6ywgeRiz5rDkzf6nZd3vr4klZsnr4BX7oa6H0HNYQXZZeWFTececPzvkx4YD30MNq3PuiJJKpyGGfDkt+CAT8IH/r5gu628sAHoMxSO/iWsfBTmO5SNpArxzsvwyIVJ605dfoej2ZnKDBuA/T8GB10KS74PL/0h62okKV2b1sPD5yU/j/+/pJWngDIdiDNzI7+T9E579FNQMxz6DMm6IkkFNm1ePVNmLKFhVSP9aqqZNHYoE0a2YULgUjP/q/DGI3D876DPQQXffeUe2QBUdU3e+E5d4KFzkmkJJFWMafPqmTx1IfWrGolA/apGJk9dyLR59VmXll8v/SFpxTno0qRXbgYqO2wAeg6E434LqxbAnEuzrkZSAU2ZsYTGDZu2Wta4YRNTZizJqKIUvP0sPPb30PfopDUnI4YNQL/T4bCvwfO/hKXXZ12NpAJpWNVya8aOlpecjY1Jq03onLTiVHXNrBTDpsmw/4B9ToBZX4BVC7OuRlIB9Ktp+WLGHS0vOXMuTVptjvtt0oqTIcOmSacqOO5/oWsNPHi2898oL6bNq2f01TMZfPkdjL56ZvmdCyhxk8YOpbpL1VbLqrtUMWns0IwqyqOl1yetNYd9LWm9yZhh01z1vjD6Zli7FB79dDJznbSLKubkcwmbMLKWq84aRm1NNQGoranmqrOGlX5vtLeeSFpp9jkhabUpAiGW4R/Uurq6OHv27F3fwOLvwLxJcMR/wmGT81eYKsroq2dS30Lbf21NNQ9ffmIGFakirHsDZhyVjG5/2uzkS3QHhBDmxBjrOlpWZV9nsyMH/zO8ORee+Fpy/U3t32RdkUpQ2Z98VvHZvDG5cLNxBZzyYIeDJp9sRmtJCHD0L2D3Ecn8N2+XUTdIFUzZn3xW8Zn3FXh1Joz6GfQ9KutqtmLY7EjnHvDhW6FTV3hgPKxfnXVFKjFlffJZxWfpDbDkGhj6JTjg4qyr2Y5h05qe+ydjCK15Hv76cYibs65IJaRsTz6r+LzxODz+WdjnxEwv3GyNHQTa4pkfwexL4LCvwxHfyt92JamjGl+BOz+YtMKMnQXd98zr5u0gUEhDvgBvzYNFVybncQaenXVFkpTMy/Xg2bB+FZz6SN6DJp9sRmuLEJK5H/oek8zZ7QgDkrIWI8y+FN74Kxx7Pew+POuKWmXYtFVVN/jQH6BLH7h/PLy3MuuKJFWy534Kz/8cDp2c2UjO7WHYtEePfvChW6GxHh46L+nTLkmF9toDMPuL0G8cDC+N88iGTXvteTQc9VN49R6Y+09ZVyOp0qx9AR48B3odkIzn2Klq568pAnYQ2BUf+FRy3mbJ96D3EBjqPDiSCmD9KrhvHMSN8JHboOtuWVfUZobNrho5Bd5ZCnMvg16DofaMrCuSVM42rU96nq19Hk64C/qU1sXBhs2u6lQFx90Id38EHj4fTn4Q9hiZdVXbqZj51VU0/MylIEaY9flkKJpjboB9xmRdUbt5zqYjOveEj/wJuu4B958B7y7PuqKtOMS9Cs3PXEqeuhqWXgeH/xsc8Imsq9klhk1HVe8HH7kdNqyB+85IfhaJiphfXUXFz1wKXrwFnvhX2P/vimZuml1h2OTD7sPh+N/D6ieTJrUi6RLtEPcqND9zefb6I/DIxbDX8XDML5MLzEuUYZMv/cYmoww0TIc5lxXFLJ8Oca9C8zOXR2uXwgNnQo8ByfV9Vd2zrqhDDJt8GvJZOOQr8OyPYMkPsq7GIe5VcH7mOmbavHpGXz2TI752Cy/+YQzrN26EMdOLesyztrI3Wr6N+H/JlARzv5x0ie5/ZmalNPUAsmeQCsXP3K5r6lyxceM6fj342+xXtYJPP/+fnP18DyYUX0fXdnOKgTRsfBfuHgOrF8HJ9xXdjHmSis/oq2fSsOodvtP/+5y9x0wue+mfmbbqBGprqnn48hMzqytfUwzYjJaGzj2Sq3u77wP3nQ6rn866IklFrmHVu0ze71ecvcdMvvvKhUxbdUJueXl0rjBs0lK9L5x4F4QquPfUorsGR8WjqZ1+8OV3MPrqmV6TUqG+MuA2Ju51K9e/cQb/89r5W5aXS+cKwyZNvQ+EMXfChtUw89SSnpbAP4jp8CJIAfD8dfzj7j/n9tVj+I+GiUDSxbmcOlcYNmnbYyR8+I9JN8b7zoCN72RdUbv5BzE9XgQplv8RHv8M7DeWTUdfR7+angSgtqaaq84aVjadK+yNVgj7jIHRN8NDZycD6X34NqjqmnVVbdbaH8Ry+UXYVR0dB8yLICvcaw8kc2PtcRR86A+M79yT8UcOzrqqVHhkUygDJsCoa2HFDHj0kxA3Z11Rm/kHsWX5OOLzIsjy0e6m5rfmw/0fTealGXNHMtZiGTNstpHquYkPfBqOuApevKloRhloC/8gtiwfTWBeBFke2v3FY83zcO9p0GU3OGEGdOtb0HqzYNg0U5BzE4d+FQ7+J3jmf2DRt/O33RT5B7Fl+TjimzCylqvOGkZtTXVZttNXinZ98WhckfRQjRuTeWl6DihQldnynE0zBTk3EUIy8dq612HBv0G3PWHI5/Kz7ZR4VXjL+tVUU99CsLT3iG/CyNqKfy9LXZu/eKxfBfeeDutehRNnwm4Hp19ckTBsminYuYnQKRnBdf2bMOsLUNWj6Oeo8A/i9iaNHcrkqQu3+oLiEV9latMXjw1vJ0Hz9lPwkTtgz1EFrDB7NqM1U9BzE526JNMS7HMiPPYpWPa/+d+HUmUTmJrstKl5w1q4bxy8ORtG/w72OyWDKrPlkU0zBf+m2rk6GdbmvnHwyEVJAA08N519KRUe8Ql20tS88R24/2/gjUeTSyAGTMi22IxkGjYhhNOAa4Aq4Bcxxqu3eX4M8EfghdyiqTHGK9KqJ5NzE517JDN93nc6PHxBMrzNgLPS25+kVLT4xWPju3D/mfD6Q3DsjTDwnGyKKwKZjfocQqgCngFOAZYDs4ALYoxPNVtnDPCVGOMZ7dl25qM+74oNa+DesbByFnzoD5lOTbCrOnqBo1RWNq1LguaVu+HYX8Pgjxdkt/n+PSyHUZ9HAc/FGJfGGNcDNwPjM6wnW116w5g/wx5HwkPnQP30rCtqF4e0kZrZ9B48cFYSNMdcV9CgKdbfwyzDphZ4udnj5bll2zo2hPBECOHPIYTDClNaRrrmLvDabRg8eBasuCvritrMMb6knE3r4cFzYMWfk1FDDvhkwXZdzL+HWYZNaGHZtm16c4H9Y4xHAP8DTNvhxkKYGEKYHUKY/frrr+evykLrWgMn/gX6HAwPjIdX7sm6ojZxSBsJ2LwBHj4fGm6Ho34MB/5DQXdfzL+HWYbNcqD5pbP9gYbmK8QY344xrs3dnw50CSG0OBl3jPHaGGNdjLFur732Sqvmwui2B5x4N/Qekoyd9Op9WVe0Uw5po4q3eQP89UJYfit88Acw5PMFL6GYfw+zDJtZwJAQwuAQQlfgfOC25iuEEPYNIYTc/VEk9ZbupDDt0X3PJHB6DU66RjfcmXVFrXJIG1W0TevgoXPhpd/DyO/C0EszKaOYfw8zC5sY40bgEmAGsBj4XYxxUQjhcyGEpvFbzgGeDCE8AfwAOD9m1X0uC933hpPugz5D4YEz4aU/ZF3RDnmBoyrWxneSFojlf4S6H8Ih/5RZKcX8e5hZ1+c0lWTX59asXwX3/Q2sfBSOvg4OuDjriiRBRfxulkPXZ7VV1xo48S7Y+4RkLpxnfpR1RZLWvQ73nABvzkqGoCnDoMknw6ZUdO4JY26H/uNh9iWw6Oqdv0ZSOt6th7s/DG8/ncy8O/DsrCsqeoZNKanqngzeuf/fwROTYf6/lswEbFLZWLsU/vKhJHBOmAH9Tsu6opLgQJylplOXZOiLLr3gqatg4xr44DXJtAVSiSjZoY1WPwUzT05GCDjpHuh7VNYVlQzDphR1qoKjfgqde8PT34WNa2HUz6GT/50qfk1DqjRd6d40pApQ3IHz5txk/MJQBSffDzWHZ11RSfHrcKlqmvFz2BWw9PpkeJuN72RdlbRTxTykyg6t+AvcPSaZ6PDkBw2aXWDYlLIQYNi/Qd2PoOGO5Jeh8ZWsq5JaVcxDqrTo+V8lF1b3GgSnPgx9hmRdUUkybMrBQV+AD01L2pPvOhZWL866ImmHinlIla3ECAv+HR77e9hnTHJE06N/1lWVLMOmXPT/aNKOvOlduOs4eO2BrCuSWlTMQ6pssWl9ck3bk1ckozaPmZ6Myq5dZtiUk751cOqjUL0vzDwFlt2UdUXSdop5SBUA1q9OZs594dfJOdGjr0t6gapD7L5UbnoNhlMehgf/Fv76d/DOi3DoV5PzO1KRaHEK5WLwzsvJ+Zm3n4ZjrndUgDzyyKYcddsDTrgL9j8/ufhz1udg88asq5KK21vz4a5j4N2X4IQ7DZo888imXFV1g+NuhJ6D4Kmrk29sx98MXfpkXZlUfBr+DA99LBmH8JSHoGZY1hWVHY9sylnoBCOuSi4AfeUumHE0vF3E1zJIhRYjLLoqGbm594HJOU+DJhWGTSUY8tlkIrb33oAZo2D5n7KuSMrehrXJ0cwT/wr7n5ec6+xRhOeRyoRhUyn2GQOnzYZeByYTsS28AuLmrKuSsrHmueSatOVTk5E4jvtf6Nwj66rKmmFTSXrun7RHD/o4LPx3ePBs2PB21lVJhdVwJ9x5FDQ2wJg74ZCv2FuzAAybStO5Ohk1+sjvQ/2fcudxnsm6Kil9MSbzQN03DnoOTI709zsl66oqhr3RdkHJDo/eJAQ4+Euw+/CkzXrGUUnPtdozsq5MSseGtcmwMy/9Prkk4OhfJBMSqmA8smmnpuHR61c1Enl/ePRp8+qzLq399jkhdx7nA3B/7jzO5k07f51USt5+Jjk/8/Ifmp2fMWgKzbBpp5IcHr01PfdPeuE0nce595RkBsIUTJtXz+irZzL48jsYffXM0gxolY4YYekNcOeRnp8pAoZNO5Xc8Oht0bkajr0BjvkVvPEY/PmIvHePLqsjQhW/DW/DXz+eDKa5Rx2MW+D5mYwZNu1UMsOjt1cIyei2p8+FHgOS7tGzvwSb1uVl82V3RKji9cbj8OeR8NItMPxbcOI9Xj9TBHYaNiGES0IIuxeimFJQEsOjd0SfoclV1EMvg2d+ADOOgdVPd3izZXlEqOISN8NT/wV/GZ2MBXjy/XD415Np1JW5thzZ7AvMCiH8LoRwWgiV3eBZ9MOj50NVN/jg9+Ajt0NjPdz5QXj+l0kb+C4q2yNCFYfGV+De02D+V6H/eBg3H/YanXVVaibENvwByQXMqcCngDrgd8AvY4zPp1verqmrq4uzZ8/Ouozy8G4DPHIRvDoTBp4Ho36aDFbYTk3nbJo3pVV3qSq/oFbhNdwJj16cnKf54DXwgc/YCSCPQghzYox1Hd1Om87ZxCSRXsndNgK7A/8XQvivjhagItejXzJdwRFXwcv/B3ccDvV3tHszFXFEqMJavxoem5hMdNZtbxg7Gw6caNAUqZ0e2YQQvghcDLwB/AKYFmPcEELoBDwbY/xA+mW2j0c2KVk5Cx79FKxeBIMuSprauvXNuipVovo74PHPwroVcPA/w7D/SHpVKu/ydWTTlhEE9gTOijG+2HxhjHFzCMFLzitJ36PgtDmw6D+T2yszoO7HMPDsrCtTpXhvJcy5DJb9FnY7DD58a/K5rCClOoJJm87ZlBqPbArgrSeSo5y35sGAc6Duh1C9T9ZVqZy99AeY/QV470047F+TW1W3rKsqqCzOfRb0nI20nd2PgLGPwRH/CfW3wR2Hwgs3dqjHmtSixlfhwXPhoXOgun8yxNLw/6i4oIHSvl7NsNGu69QFDpsMp8+H3gfBIx9Pxlh758WdvlTaqbgZlv46+SJTf1vSSWXsY8kXnQpVyterGTZFrGTGEtvtkGSenCO/B6/eA7cfDAv+HTa+m3VlKlUrZ8Fdo5MuzX2GJl9oDrscOlX2QPWlfL2aYVOkSm4ssU5VcPBlcMbTUDsenrwiCZ0Xb7FpTW3X+Co8+vfJ9OXvvABHX5d8kdntkKwrKwqlPIKJYVOkSrZttudAOP5mOPmBpFv0w+fDPWPgrflZV6Zitmk9LP4O/GlI0tPskEnw0WfgA5+C4J+pJqV8vVplH5MWsVJumwVg7w8lF9kt/SU88bVkyJsPfAaGXwnd98y6OhWThj8n3ZnXPAP9xiXNsX0OyrqqojVhZG1JhMu2/MpQpEq5bXaLTlXJFd0ffQYOuhSe/0XyzfXpa2DTe1lXp6ytfhruOyOZphngI3fAmDsMmjJl2BSpUm6b3U7X3eGD30/mFOl7FMy9DP50EDz3c9i8IevqVGhvPwt/vQimHwavPZDMnjluIdSOy7oypciwKVKl3Da7Q7sdCifMSMZaq94PHp8IfxoKz/8qGRJe5W3tC8nJ/zsOSaZoPvif4cznk9kzq7pmXZ1S5ggCykaMSVv9wm/Am3Og14Ew7N9h/wucf6TcvPMSLPo2PH8dhCoY8nk49HJHnCgRjiCg0hZC0mwydhZ8eBp07pFMZTD98Fx36c1ZV6iOerceZv0j/OlAWHo9HPhZOHNpMoCrQVNxDBtlK4RksqvT58Hxv0+6uT58PtxxGDz7My8MLUWrFsJj/wC3fQCeuxYO+BR89Fk46ofJlBWqSIaNikPoBAPPgdMXwHE3QVUPmPU5mNYf5l8O77ycdYVqTdwMy/8E95wM04fDsv+FAz6Z9EQc9bPk+itVNM/ZqDjFCK8/DEu+D8tvBUIyuvTBl8Gex2RcnLbYsCZpIlvyA1j7HPToDwddklxT1W2PrKtTHhRyPhup8EKAvY9PbmuXwTM/TK7TeekW6Hs0DL0MBpxlL6asrH3h/f+TDW9D32PgiCuT/5NOXbKuTkXIIxuVjg1r4YUbYMk1sObZZDicgefD4Iug7yinA07b+tXJ1OAv/AZeux9CZxh4Lgz9Eux5dNbVKSX5OrIxbFR64mZYMQOW3gD1f4RN65IpDgZfBIM+Dr0GZV1h+di8AVbcBS/8Ohnmv/l7fcAnk2YzlTXDphWGTQVZvzq5QPCF38Br9yXL9vpQ8sdw4LnQtSbL6kpTjMm1Ty/8Bl68Cd57PTmK3P8CGHRRMgqER5EVw7BphWFTod55EZbdmPyRfPtp6NQV9h4DtX+TDPDY+8CsKyxem9bBq/dDwx3QMB3WPp+8f7VnJsG932meH6tQhk0rDJsK1/TN/MWbkj+eb+emZeh9UBI6tX+THP1U4LTCW3nnpSRYGqbDK/fApnehqjvscxL0PzN3ZLh71lUqY4ZNKwwbbWXN8+//UX31Xtj8HnTuBfueDPuNhT2Pg90OK/9hct57E1Y+lpzcb5ieXHwJ0HMQ9PubJIT3HgOdi2Nk8Wnz6pkyYwkNqxrpV1PNpLFDS3tswBJVFmETQjgNuAaoAn4RY7x6m+dD7vlxwLvAJ2OMc3e2XcNGO7TxnSRw6u9IjnrezV0s2rlXci6i7zHJdTx9jy7tIVU2b0zCZOWj8EbutuaZ5LnQOZlvqF+uebHPwUV3DqZpptrmEwhWd6kq/cFoS1DJX2cTQqgCfgScAiwHZoUQbosxPtVstdOBIbnb0cBPcj+lXdO5J9SekdxiTM5NvPHY+3+UF0+BmBuBuuegJHh2HwG9hyS3XgcWzTd/IPk3rHsl6Qr+9jOwZgmsfBxWzk6axQC67w17HpsMG7PnMbBHHXTplW3dO9HaTLWGTWnK8qLOUcBzMcalACGEm4HxQPOwGQ/8OiaHX4+GEGpCCPvFGFcUvlyVnRCSTgO9D4TBFybLNjbCW/OS4Fn5aDKKwYs3b/26HgPeD5+mW/d9kx5b3faALrvlbyrjTe/B+jfhvZXJ7Z1lSbCseTY5UlnzHGxc+/76nbrC7iPhwM+8f5TWc/+iO3LZmZKfqVbbyTJsaoHmA14tZ/ujlpbWqQUMG6WjczXsdVxya7Lh7dyRQ/M/8s/CS79PgmBboRN03SO5desLXfsmXbB3FkCb1iWBsn5lcn5l/cqk2W+77VdBz8FJyO314eRnn4OSnz0GlsW5p3411dS3ECwlNVOttpJl2LT0VWvbE0htWSdZMYSJwESAgQMd9E951KUP7PHB5Lat995Mji7ee+39gHhv5dZHI431sHoRO/jovq9T1yScqvtDzRHvh1W3vu/f7zEAeg0u+yFhJo0d2uI5m5KcqVZAtmGzHBjQ7HF/oGEX1gEgxngtcC0kHQTyV6bUim57QLdRWVdRdprOy9gbrXxkGTazgCEhhMFAPXA+8HfbrHMbcEnufM7RwGrP10iVYcLIWsOljGQWNjHGjSGES4AZJF2fr4sxLgohfC73/E+B6STdnp8j6fr8qazqlSTtukynGIgxTicJlObLftrsfgT+sdB1SZLyy5k6JUmpM2wkSakzbCRJqTNsJEmpM2wkSakzbCRJqTNsJEmpK8uwWVi/mtFXz2TavPqsS5EkUaZhA1C/qpHJUxcaOJJUBMo2bOD9yZYkSdkq67ABJ1uSpGJQ9mHjZEuSlL1MB+JMm5MtSe0zbV69c8goFWUbNrX+okjtMm1e/VazYzZ1sgH8PVKHlWUz2rDa3Xj48hP9BZHaYcqMJVtNwwx2slH+lGXYSGq/HXWmsZON8qFsm9FUuTzvsGv61VRT30Kw2MlG+eCRjcpK03mH+lWNRLy4tz0mjR1KdZeqrZbZyUb5YtiorHjeYddNGFnLVWcNo7ammkDSyeaqs4Z5VKi8sBlNZcXzDh0zYWSt4aJUeGSjsrKj8wued5CyZdiorHjeQSpONqOprDQ1AdkbTSouFRU2domtDJ53kIpPxYSNQ3FIUnYq5pyNXWIlKTsVEzZ2iZWk7FRM2NglVpKyUzFhY5dYScpOxXQQsEusJGWnYsIG7BIrSVmpqLCRypnXkamYGTZSGfA6MhW7iukgIJUzryNTsTNspDLgdWQqdoaNVAa8jkzFzrCRyoDXkanY2UFAKgNeR6ZiZ9hIZcLryFTMbEaTJKXOsJEkpc6wkSSlzrCRJKXOsJEkpc6wkSSlzrCRJKXO62wklSynVSgdho2kkuS0CqXFZjRJJclpFUqLYSOpJDmtQmkxbCSVJKdVKC2GjaSS5LQKpSWTDgIhhD2AW4BBwDLgYzHGt1pYbxmwBtgEbIwx1hWuSknFzGkVSkuIMRZ+pyH8F/BmjPHqEMLlwO4xxq+2sN4yoC7G+EZ7tl9XVxdnz56dn2IlqYKFEObk44t+Vs1o44EbcvdvACZkVIckqQCyCpt9YowrAHI/997BehG4K4QwJ4QwsWDVSZLyKrVzNiGEu4F9W3jqa+3YzOgYY0MIYW/gLyGEp2OMD+xgfxOBiQADBw5sd72SpPSkFjYxxpN39FwI4dUQwn4xxhUhhP2A13awjYbcz9dCCLcCo4AWwybGeC1wLSTnbDpavyQpf7JqRrsNuDh3/2Lgj9uuEELoGULo3XQfOBV4smAVSpLyJquwuRo4JYTwLHBK7jEhhH4hhOm5dfYBHgohPAE8DtwRY7wzk2olSR2SyXU2McaVwEktLG8AxuXuLwWOKHBpkqQUOIKAJCl1ho0kKXWGjSQpdYaNJCl1ho0kKXWGjSQpdYaNJCl1ho0kKXWGjSQpdZmMICBJlWbavPqKnlXUsJGklE2bV8/kqQtp3LAJgPpVjUyeuhCgYgLHsJHULpX+DX1XTJmxZEvQNGncsIkpM5ZUzHtn2EhqM7+h75qGVY3tWl6O7CAgqc1a+4auHetXU92u5WmbNq+e0VfPZPDldzD66plMm1ef+j4NG0lt5jf0XTNp7FCqu1Rttay6SxWTxg4teC1NR6f1qxqJvH90mnbgGDaS2qzYvqGXigkja7nqrGHU1lQTgNqaaq46a1gmTY9ZHZ16zkZSm00aO3SrczaQ3Tf0UjNhZG1RnNfK6ujUIxtJbVZM39C1a7I6OvXIRlK7FMs3dO2arI5ODRtJqiBNXxQKfa2UYSNJFSaLo1PP2UiSUmfYSJJSZ9hIklJn2EiSUmfYSJJSZ9hIklJn2EiSUmfYSJJSZ9hIklJn2EiSUmfYSJJSZ9hIklJn2EiSUmfYSJJSZ9hIklJn2EiSUmfYSJJSZ9hIklJn2EiSUmfYSJJSZ9hIklJn2EiSUmfYSJJSZ9hIklJn2EiSUmfYSJJSZ9hIklJn2EiSUtc56wKUvWnz6pkyYwkNqxrpV1PNpLFDmTCyNuuyJJURw6bCTZtXz+SpC2ncsAmA+lWNTJ66EMDAkZQ3mTSjhRDODSEsCiFsDiHUtbLeaSGEJSGE50IIlxeyxkoxZcaSLUHTpHHDJqbMWJJRRZLKUVbnbJ4EzgIe2NEKIYQq4EfA6cChwAUhhEMLU17laFjV2K7lkrQrMgmbGOPiGOPOvjqPAp6LMS6NMa4HbgbGp19dZelXU92u5ZK0K4q5N1ot8HKzx8tzy5RHk8YOpbpL1VbLqrtUMWns0IwqklSOUusgEEK4G9i3hae+FmP8Y1s20cKy2Mr+JgITAQYOHNimGvV+JwB7o0lKU2phE2M8uYObWA4MaPa4P9DQyv6uBa4FqKur22EoaXsTRtYaLpJSVczNaLOAISGEwSGErsD5wG0Z1yRJ2gVZdX3+2xDCcuBY4I4Qwozc8n4hhOkAMcaNwCXADGAx8LsY46Is6pUkdUwmF3XGGG8Fbm1heQMwrtnj6cD0ApYmSUpBMTejSZLKhGEjSUqdYSNJSp1hI0lKnWEjSUqdYSNJSp1hI0lKnWEjSUqdYSNJSp1hI0lKnWEjSUqdYSNJSp1hI0lKnWEjSUqdYSNJSp1hI0lKnWEjSUqdYSNJSp1hI0lKnWEjSUqdYSNJSp1hI0lKnWEjSUqdYSNJSp1hI0lKnWEjSUqdYSNJSp1hI0lKnWEjSUqdYSNJSp1hI0lKXeesC5CyMG1ePVNmLKFhVSP9aqqZNHYoE0bWZl2WVLYMG1WcafPqmTx1IY0bNgFQv6qRyVMXAhg4UkpsRlPFmTJjyZagadK4YRNTZizJqCKp/Bk2qjgNqxrbtVxSxxk2qjj9aqrbtVxSxxk2qjiTxg6lukvVVsuqu1QxaezQjCqSyp8dBFRxmjoB2BtNKhzDRhVpwshaw0UqIJvRJEmpM2wkSakzbCRJqTNsJEmpM2wkSakzbCRJqTNsJEmpM2wkSakzbCRJqTNsJEmpM2wkSakzbCRJqcskbEII54YQFoUQNocQ6lpZb1kIYWEIYX4IYXYha5Qk5U9Woz4/CZwF/KwN654QY3wj5XokSSnKJGxijIsBQghZ7F6SVGDFfs4mAneFEOaEECZmXYwkadekdmQTQrgb2LeFp74WY/xjGzczOsbYEELYG/hLCOHpGOMDO9jfRGAiwMCBA3epZklSOlILmxjjyXnYRkPu52shhFuBUUCLYRNjvBa4FqCuri52dN+SpPwp2ma0EELPEELvpvvAqSQdCyRJJSaTDgIhhL8F/gfYC7gjhDA/xjg2hNAP+EWMcRywD3BrrhNBZ+B/Y4x3ZlGvlLVp8+qZMmMJDasa6VdTzaSxQ5kwsjbrsqQ2y6o32q3ArS0sbwDG5e4vBY4ocGlS0Zk2r57JUxfSuGETAPWrGpk8dSGAgaOSUbTNaJISU2Ys2RI0TRo3bGLKjCUZVSS1n2EjFbmGVY3tWi4VI8NGKnL9aqrbtVwqRoaNVOQmjR1KdZeqrZZVd6li0tihGVUktV9WY6NJaqOmTgD2RlMpM2ykEjBhZK3hopJmM5okKXWGjSQpdYaNJCl1ho0kKXWGjSQpdYaNJCl1ho0kKXWGjSQpdV7UKe2Ac8hI+WPYSC1wDhkpv2xGk1rgHDJSfnlkU+Js6kmHc8hI+eWRTQlrauqpX9VI5P2mnmnz6rMureQ5h4yUX4ZNCbOpJz3OISPll81oJcymnvQ4h4yUX4ZNCetXU019C8FiU09+OIeMlD82o5Uwm3oklQqPbEqYTT2SSoVhU+Js6pFUCmxGkySlzrCRJKXOsJEkpc6wkSSlzrCRJKXOsJEkpc6wkSSlzrCRJKXOsJEkpc6wkSSlzrCRJKXOsJEkpc6wkSSlzrCRJKXOsJEkpc6wkSSlzrCRJKXOsJEkpc6wkSSlzrCRJKXOsJEkpc6wkSSlzrCRJKXOsJEkpc6wkSSlzrCRJKXOsJEkpS6TsAkhTAkhPB1CWBBCuDWEULOD9U4LISwJITwXQri8wGVKkvIkqyObvwCHxxiHA88Ak7ddIYRQBfwIOB04FLgghHBoQauUJOVFJmETY7wrxrgx9/BRoH8Lq40CnosxLo0xrgduBsYXqkZJUv4Uwzmbvwf+3MLyWuDlZo+X55ZJkkpM57Q2HEK4G9i3hae+FmP8Y26drwEbgRtb2kQLy2Ir+5sITMw9fC+E8GT7Ks7EnsAbWRfRRqVSa6nUCaVTa6nUCaVTa6nUCTA0HxtJLWxijCe39nwI4WLgDOCkGGNLIbIcGNDscX+goZX9XQtcm9v27BhjXbuLLrBSqRNKp9ZSqRNKp9ZSqRNKp9ZSqROSWvOxnax6o50GfBU4M8b47g5WmwUMCSEMDiF0Bc4HbitUjZKk/MnqnM0Pgd7AX0II80MIPwUIIfQLIUwHyHUguASYASwGfhdjXJRRvZKkDkitGa01McYDd7C8ARjX7PF0YPou7OLaXSyt0EqlTiidWkulTiidWkulTiidWkulTshTraHl0yWSJOVPMXR9liSVuZIMmxDCuSGERSGEzSGEHfbo2NFwNyGEPUIIfwkhPJv7uXuKte50XyGEoblzV023t0MIl+We+2YIob7Zc+O220mB6syttyyEsDBXy+z2vr5QtYYQBoQQ7g0hLM59Vr7U7LlU39OdDbMUEj/IPb8ghHBkW1+bb22o9cJcjQtCCH8NIRzR7LkWPwsZ1TkmhLC62f/pN9r62gxqndSszidDCJtCCHvknivke3pdCOG1sIPLRPL+OY0xltwNOISk7/d9QN0O1qkCngcOALoCTwCH5p77L+Dy3P3Lgf+XYq3t2leu7leA/XOPvwl8pQDvaZvqBJYBe3b035l2rcB+wJG5+71JhkVq+v9P7T1t7XPXbJ1xJBcyB+AY4LG2vjaDWo8Dds/dP72p1tY+CxnVOQa4fVdeW+hat1n/o8DMQr+nuX19GDgSeHIHz+f1c1qSRzYxxsUxxiU7Wa214W7GAzfk7t8ATEil0F3b10nA8zHGF1OsqSUdfU+K6j2NMa6IMc7N3V9D0qOxECNQtGWYpfHAr2PiUaAmhLBfG19b0FpjjH+NMb6Ve7ijoaXS1pH3peje021cANyUYj07FGN8AHizlVXy+jktybBpo9aGu9knxrgCkj9KwN4p1tHefZ3P9h++S3KHsdel2DzV1jojcFcIYU5IRm1o7+sLWSsAIYRBwEjgsWaL03pP2zLM0o7WKfQQTe3d36fZemipHX0W8q2tdR4bQngihPDnEMJh7XxtvrR5fyGEHsBpwB+aLS7Ue9oWef2cZtL1uS1CG4a72dkmWliWSte71mpt53a6Amey9SjYPwG+RVL7t4Dvkownl1Wdo2OMDSGEvUmuk3o69w0pr/L4nvYi+WW+LMb4dm5x3t7TlnbZwrJtP3c7Wqdgn9md1LH9iiGcQBI2xzdbXJDPQhvrnEvS9Lw2dw5uGjCkja/Np/bs76PAwzHG5kcXhXpP2yKvn9OiDZu4k+Fu2qC14W5eDSHsF2NckTssfK0jO2qt1hBCe/Z1OjA3xvhqs21vuR9C+Dlwe5Z1xuRaKGKMr4UQbiU5pH6AInxPQwhdSILmxhjj1Gbbztt72oK2DLO0o3W6tuG1+dSmIaFCCMOBXwCnxxhXNi1v5bNQ8DqbfZEgxjg9hPDjEMKebXltoWttZrtWjAK+p22R189pOTejtTbczW3Axbn7FwNtOVLaVe3Z13btt7k/pk3+FkhrgNGd1hlC6BlC6N10Hzi1WT1F9Z6GEALwS2BxjPG/t3kuzfe0LcMs3QZ8Itfb5xhgda45sNBDNO10fyGEgcBU4KIY4zPNlrf2Wciizn1z/+eEEEaR/G1b2ZbXFrrWXI27AR+h2We3wO9pW+T3c1qIXg/5vpH8gVgOvAe8CszILe8HTG+23jiSXkjPkzS/NS3vC9wDPJv7uUeKtba4rxZq7UHyy7HbNq//DbAQWJD7D90vqzpJep88kbstKub3lKS5J+bet/m527hCvKctfe6AzwGfy90PJBMDPp+ro66116Z5a0OtvwDeavYezt7ZZyGjOi/J1fEESUeG44r1Pc09/iRw8zavK/R7ehOwAthA8vf002l+Th1BQJKUunJuRpMkFQnDRpKUOsNGkpQ6w0aSlDrDRpKUOsNGkpQ6w0aSlDrDRiqgEMJRuQFAu+euGF8UQjg867qktHlRp1RgIYQrge5ANbA8xnhVxiVJqTNspALLjSc1C1hHMqzKpoxLklJnM5pUeHsAvUhmEO2ecS1SQXhkIxVYCOE2ktkNB5MMAnpJxiVJqSva+WykchRC+ASwMcb4vyGEKuCvIYQTY4wzs65NSpNHNpKk1HnORpKUOsNGkpQ6w0aSlDrDRpKUOsNGkpQ6w0aSlDrDRpKUOsNGkpS6/w+/NtfgVHMNLgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 439.2x720 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Create the dataset object and plot the dataset\n",
    "\n",
    "data_set = Data()\n",
    "data_set.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get some validation data: \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create validation dataset object\n",
    "\n",
    "validation_set = Data(train=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--Empty Space for separating topics-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 id=\"Model_Cost\">Create the Model, Optimizer, and Total Loss Function (Cost)</h2>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a custom module with three layers. <code>in_size</code> is the size of the input features, <code>n_hidden</code> is the size of the layers, and <code>out_size</code> is the size. <code>p</code> is dropout probability. The default is 0 which is no dropout.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the class for model\n",
    "\n",
    "class Net(nn.Module):\n",
    "    \n",
    "    # Constructor\n",
    "    def __init__(self, in_size, n_hidden, out_size, p=0):\n",
    "        super(Net, self).__init__()\n",
    "        self.drop = nn.Dropout(p=p)\n",
    "        self.linear1 = nn.Linear(in_size, n_hidden)\n",
    "        self.linear2 = nn.Linear(n_hidden, n_hidden)\n",
    "        self.linear3 = nn.Linear(n_hidden, out_size)\n",
    "        \n",
    "    def forward(self, x):\n",
    "        x = F.relu(self.drop(self.linear1(x)))\n",
    "        x = F.relu(self.drop(self.linear2(x)))\n",
    "        x = self.linear3(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create two model objects: <code>model</code> had no dropout, and <code>model_drop </cdoe> has a dropout probability of 0.5:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the model objects\n",
    "\n",
    "model = Net(1, 300, 1)\n",
    "model_drop = Net(1, 300, 1, p=0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--Empty Space for separating topics-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 id=\"Train\">Train the Model via Mini-Batch Gradient Descent</h2>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set the model using dropout to training mode; this is the default mode, but it's good practice. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Net(\n",
       "  (drop): Dropout(p=0.5, inplace=False)\n",
       "  (linear1): Linear(in_features=1, out_features=300, bias=True)\n",
       "  (linear2): Linear(in_features=300, out_features=300, bias=True)\n",
       "  (linear3): Linear(in_features=300, out_features=1, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Set the model to train mode\n",
    "\n",
    "model_drop.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train the model by using the Adam optimizer. See the unit on other optimizers. Use the mean square loss: \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the optimizer and criterion function\n",
    "\n",
    "optimizer_ofit = torch.optim.Adam(model.parameters(), lr=0.01)\n",
    "optimizer_drop = torch.optim.Adam(model_drop.parameters(), lr=0.01)\n",
    "criterion = torch.nn.MSELoss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initialize a dictionary that stores the training and validation loss for each model:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the dict to contain the loss results\n",
    "\n",
    "LOSS={}\n",
    "LOSS['training data no dropout']=[]\n",
    "LOSS['validation data no dropout']=[]\n",
    "LOSS['training data dropout']=[]\n",
    "LOSS['validation data dropout']=[]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run 500 iterations of batch gradient descent: \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train the model\n",
    "\n",
    "epochs = 20\n",
    "\n",
    "def train_model(epochs):\n",
    "    for epoch in range(epochs):\n",
    "        yhat = model(data_set.x)\n",
    "        yhat_drop = model_drop(data_set.x)\n",
    "        loss = criterion(yhat, data_set.y)\n",
    "        loss_drop = criterion(yhat_drop, data_set.y)\n",
    "\n",
    "        #store the loss for  both the training and validation  data for both models \n",
    "        LOSS['training data no dropout'].append(loss.item())\n",
    "        LOSS['validation data no dropout'].append(criterion(model(validation_set.x), validation_set.y).item())\n",
    "        LOSS['training data dropout'].append(loss_drop.item())\n",
    "        model_drop.eval()\n",
    "        LOSS['validation data dropout'].append(criterion(model_drop(validation_set.x), validation_set.y).item())\n",
    "        model_drop.train()\n",
    "\n",
    "        optimizer_ofit.zero_grad()\n",
    "        optimizer_drop.zero_grad()\n",
    "        loss.backward()\n",
    "        loss_drop.backward()\n",
    "        optimizer_ofit.step()\n",
    "        optimizer_drop.step()\n",
    "        \n",
    "train_model(epochs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set the model with dropout to evaluation mode:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Net(\n",
       "  (drop): Dropout(p=0.5, inplace=False)\n",
       "  (linear1): Linear(in_features=1, out_features=300, bias=True)\n",
       "  (linear2): Linear(in_features=300, out_features=300, bias=True)\n",
       "  (linear3): Linear(in_features=300, out_features=1, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Set the model with dropout to evaluation mode\n",
    "\n",
    "model_drop.eval()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make a prediction by using both models: \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make the prediction\n",
    "\n",
    "yhat = model(data_set.x)\n",
    "yhat_drop = model_drop(data_set.x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot predictions of both models. Compare them to the training points and the true function: \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the predictions for both models\n",
    "\n",
    "plt.figure(figsize=(6.1, 10))\n",
    "\n",
    "plt.scatter(data_set.x.numpy(), data_set.y.numpy(), label=\"Samples\")\n",
    "plt.plot(data_set.x.numpy(), data_set.f.numpy(), label=\"True function\", color='orange')\n",
    "plt.plot(data_set.x.numpy(), yhat.detach().numpy(), label='no dropout', c='r')\n",
    "plt.plot(data_set.x.numpy(), yhat_drop.detach().numpy(), label=\"dropout\", c ='g')\n",
    "\n",
    "plt.xlabel(\"x\")\n",
    "plt.ylabel(\"y\")\n",
    "plt.xlim((-1, 1))\n",
    "plt.ylim((-2, 2.5))\n",
    "plt.legend(loc = \"best\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see that the model using dropout does better at tracking the function that generated the data. We use the log to make the difference more apparent \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot out the loss for training and validation data on both models:  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the loss\n",
    "\n",
    "plt.figure(figsize=(6.1, 10))\n",
    "for key, value in LOSS.items():\n",
    "    plt.plot(np.log(np.array(value)), label=key)\n",
    "    plt.legend()\n",
    "    plt.xlabel(\"iterations\")\n",
    "    plt.ylabel(\"Log of cost or total loss\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You see that the model without dropout performs better on the training data, but it performs worse on the validation data. This suggests overfitting.  However, the model using dropout performs better on the validation data, but worse on the training data. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://dataplatform.cloud.ibm.com/registration/stepone?context=cpdaas&apps=data_science_experience,watson_machine_learning\"><img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-DL0110EN-SkillsNetwork/Template/module%201/images/Watson_Studio.png\"/></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--Empty Space for separating topics-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>About the Authors:</h2> \n",
    "\n",
    "<a href=\"https://www.linkedin.com/in/joseph-s-50398b136/\">Joseph Santarcangelo</a> has a PhD in Electrical Engineering, his research focused on using machine learning, signal processing, and computer vision to determine how videos impact human cognition. Joseph has been working for IBM since he completed his PhD. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other contributors: <a href=\"https://www.linkedin.com/in/michelleccarey/\">Michelle Carey</a>, <a href=\"www.linkedin.com/in/jiahui-mavis-zhou-a4537814a\">Mavis Zhou</a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Change Log\n",
    "\n",
    "| Date (YYYY-MM-DD) | Version | Changed By | Change Description                                          |\n",
    "| ----------------- | ------- | ---------- | ----------------------------------------------------------- |\n",
    "| 2020-09-23        | 2.0     | Srishti    | Migrated Lab to Markdown and added to course repo in GitLab |\n",
    "\n",
    "<hr>\n",
    "\n",
    "## <h3 align=\"center\"> © IBM Corporation 2020. All rights reserved. <h3/>\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python",
   "language": "python",
   "name": "conda-env-python-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}