8.3.3.He_Initialization_v2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center>\n",
    "    <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-DL0110EN-SkillsNetwork/Template/module%201/images/IDSNlogo.png\" width=\"300\" alt=\"cognitiveclass.ai logo\"  />\n",
    "</center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>Test Uniform, Default and He Initialization on MNIST Dataset with Relu Activation</h1>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Objective for this Notebook<h3>    \n",
    "<h5> 1. Learn how to Define Several Neural Network, Criterion function, Optimizer.</h5>\n",
    "<h5> 2. Test Uniform, Default and He Initialization </h5>   \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Table of Contents</h2>\n",
    "<p>In this lab, you will test the Uniform Initialization, Default Initialization and He Initialization on the MNIST dataset with Relu Activation</p>\n",
    "\n",
    "<ul>\n",
    "    <li><a href=\"#Model\">Neural Network Module and Training Function</a></li>\n",
    "    <li><a href=\"#Makeup_Data\">Make Some Data</a></li>\n",
    "    <li><a href=\"#Cost\">Define Several Neural Network, Criterion function, Optimizer</a></li>\n",
    "    <li><a href=\"#Train\">Test Uniform, Default and He Initialization</a></li>\n",
    "    <li><a href=\"#Result\">Analyze Results</a></li>\n",
    "</ul>\n",
    "<p>Estimated Time Needed: <strong>25 min</strong></p>\n",
    "\n",
    "<hr>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Preparation</h2>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll need the following libraries:  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch._C.Generator at 0x7f5394d050d0>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Import the libraries we need to use in this lab\n",
    "\n",
    "# Using the following line code to install the torchvision library\n",
    "# !conda install -y torchvision\n",
    "\n",
    "import torch \n",
    "import torch.nn as nn\n",
    "import torchvision.transforms as transforms\n",
    "import torchvision.datasets as dsets\n",
    "import torch.nn.functional as F\n",
    "import matplotlib.pylab as plt\n",
    "import numpy as np\n",
    "\n",
    "torch.manual_seed(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--Empty Space for separating topics-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 id=\"Model\">Neural Network Module and Training Function</h2> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the neural network module or class with He Initialization\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the class for neural network model with He Initialization\n",
    "\n",
    "class Net_He(nn.Module):\n",
    "    \n",
    "    # Constructor\n",
    "    def __init__(self, Layers):\n",
    "        super(Net_He, self).__init__()\n",
    "        self.hidden = nn.ModuleList()\n",
    "\n",
    "        for input_size, output_size in zip(Layers, Layers[1:]):\n",
    "            linear = nn.Linear(input_size, output_size)\n",
    "            torch.nn.init.kaiming_uniform_(linear.weight, nonlinearity='relu')\n",
    "            self.hidden.append(linear)\n",
    "\n",
    "    # Prediction\n",
    "    def forward(self, x):\n",
    "        L = len(self.hidden)\n",
    "        for (l, linear_transform) in zip(range(L), self.hidden):\n",
    "            if l < L - 1:\n",
    "                x = F.relu(linear_transform(x))\n",
    "            else:\n",
    "                x = linear_transform(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the class or neural network with Uniform Initialization\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the class for neural network model with Uniform Initialization\n",
    "\n",
    "class Net_Uniform(nn.Module):\n",
    "    \n",
    "    # Constructor\n",
    "    def __init__(self, Layers):\n",
    "        super(Net_Uniform, self).__init__()\n",
    "        self.hidden = nn.ModuleList()\n",
    "\n",
    "        for input_size, output_size in zip(Layers, Layers[1:]):\n",
    "            linear = nn.Linear(input_size,output_size)\n",
    "            linear.weight.data.uniform_(0, 1)\n",
    "            self.hidden.append(linear)\n",
    "    \n",
    "    # Prediction\n",
    "    def forward(self, x):\n",
    "        L = len(self.hidden)\n",
    "        for (l, linear_transform) in zip(range(L), self.hidden):\n",
    "            if l < L - 1:\n",
    "                x = F.relu(linear_transform(x))\n",
    "            else:\n",
    "                x = linear_transform(x)\n",
    "                \n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Class or Neural Network with PyTorch Default Initialization\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the class for neural network model with PyTorch Default Initialization\n",
    "\n",
    "class Net(nn.Module):\n",
    "    \n",
    "    # Constructor\n",
    "    def __init__(self, Layers):\n",
    "        super(Net, self).__init__()\n",
    "        self.hidden = nn.ModuleList()\n",
    "\n",
    "        for input_size, output_size in zip(Layers, Layers[1:]):\n",
    "            linear = nn.Linear(input_size, output_size)\n",
    "            self.hidden.append(linear)\n",
    "        \n",
    "    def forward(self, x):\n",
    "        L=len(self.hidden)\n",
    "        for (l, linear_transform) in zip(range(L), self.hidden):\n",
    "            if l < L - 1:\n",
    "                x = F.relu(linear_transform(x))\n",
    "            else:\n",
    "                x = linear_transform(x)\n",
    "                \n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define a function to train the model, in this case the function returns a Python dictionary to store the training loss and accuracy on the validation data \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define function to  train model\n",
    "\n",
    "def train(model, criterion, train_loader, validation_loader, optimizer, epochs = 100):\n",
    "    i = 0\n",
    "    loss_accuracy = {'training_loss': [], 'validation_accuracy': []}  \n",
    "    \n",
    "    #n_epochs\n",
    "    for epoch in range(epochs):\n",
    "        for i, (x, y) in enumerate(train_loader):\n",
    "            optimizer.zero_grad()\n",
    "            z = model(x.view(-1, 28 * 28))\n",
    "            loss = criterion(z, y)\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "            loss_accuracy['training_loss'].append(loss.data.item())\n",
    "        \n",
    "        correct = 0\n",
    "        for x, y in validation_loader:\n",
    "            yhat = model(x.view(-1, 28 * 28))\n",
    "            _, label = torch.max(yhat, 1)\n",
    "            correct += (label == y).sum().item()\n",
    "        accuracy = 100 * (correct / len(validation_dataset))\n",
    "        loss_accuracy['validation_accuracy'].append(accuracy)\n",
    "    \n",
    "    return loss_accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--Empty Space for separating topics-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 id=\"Makeup_Data\">Make some Data</h2> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the training dataset by setting the parameters <code>train </code> to <code>True</code> and convert it to a tensor  by placing a transform object int the argument <code>transform</code>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the training dataset\n",
    "\n",
    "train_dataset = dsets.MNIST(root='./data', train=True, download=True, transform=transforms.ToTensor())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the testing dataset by setting the parameters train  <code>False</code> and convert it to a tensor  by placing a transform object int the argument <code>transform</code>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the validation dataset\n",
    "\n",
    "validation_dataset = dsets.MNIST(root='./data', train=False, download=True, transform=transforms.ToTensor())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the training-data loader and the validation-data loader object \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the data loader for training and validation\n",
    "\n",
    "train_loader = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=2000, shuffle=True)\n",
    "validation_loader = torch.utils.data.DataLoader(dataset=validation_dataset, batch_size=5000, shuffle=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--Empty Space for separating topics-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 id=\"Cost\">Define Neural Network, Criterion function, Optimizer and Train the Model</h2> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the criterion function  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the criterion function\n",
    "\n",
    "criterion = nn.CrossEntropyLoss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a list that contains layer size \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the parameters\n",
    "\n",
    "input_dim = 28 * 28\n",
    "output_dim = 10\n",
    "layers = [input_dim, 100, 200, 100, output_dim]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--Empty Space for separating topics-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 id=\"Train\">Test PyTorch Default Initialization, Xavier Initialization and Uniform Initialization</h2> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train the network using PyTorch Default Initialization\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train the model with the default initialization\n",
    "\n",
    "model = Net(layers)\n",
    "learning_rate = 0.01\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)\n",
    "training_results = train(model, criterion, train_loader,validation_loader, optimizer, epochs=30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train the network using He Initialization function\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train the model with the He initialization\n",
    "\n",
    "model_He = Net_He(layers)\n",
    "optimizer = torch.optim.SGD(model_He.parameters(), lr=learning_rate)\n",
    "training_results_He = train(model_He, criterion, train_loader, validation_loader, optimizer, epochs=30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train the network using Uniform Initialization function\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train the model with the Uniform initialization\n",
    "\n",
    "model_Uniform = Net_Uniform(layers)\n",
    "optimizer = torch.optim.SGD(model_Uniform.parameters(), lr=learning_rate)\n",
    "training_results_Uniform = train(model_Uniform, criterion, train_loader, validation_loader, optimizer, epochs=30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--Empty Space for separating topics-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 id=\"Result\">Analyze Results</h2> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compare the training loss for each activation \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x12a3d4cc0>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaEAAAEWCAYAAADPZygPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XuYFdWd7vHvSzfSRPDCzSDogAajeENtlIwmYSIH0ONAHo3jZYx4G2KOJupkjJiT84gmmWBi4mjMoBzF25ioMfFIDBGRxFGTqIAajRIjQSIdVFoaEUQIl9/5o1bjpu3u3d12ddH0+3me/eyqVatqrare3b9eq9aupYjAzMysCN2KroCZmXVdDkJmZlYYByEzMyuMg5CZmRXGQcjMzArjIGRmZoVxELIuSdKNkv5Pe+dtZR2GSApJle197CbKe1HS6I4oq4ny95a0VlJFUXWw7Y/8PSHrbCQtBc6LiEeKrsuHIWkI8CrQPSI2dXDZU4GPRcQZOZaxlB3g52T5ckvIdjgd1bLoynyNrb04CFmnIulOYG/g56lr56sl3VrnSnoN+FXK+xNJb0haLekxSQeWHOc2Sd9My6Ml1Uj6iqQVkl6XdHYb8/aV9HNJ70iaL+mbkp5o4bntKWmWpDpJiyX9S8m2IyUtSMd9U9L3U3qVpP+StFLS26nMPZo4/lJJYySNB74GnJKu4e/T9l0l3ZLO6a+p7hVp21mSfiPpWkl1wFRJ+0r6VSr7LUl3SdqtBT+nyhac71RJ90q6Q9Ka1JVYXbL9slTHNZJelnRsS66xbX8chKxTiYjPA68B/xgRvSLiOyWbPw0cAIxL678EhgEDgGeAu5o59EeBXYFBwLnADyXt3oa8PwTeTXkmpVdL/RioAfYEPgf8e8kf1+uA6yJiF2Bf4N6UPinVZS+gL3A+8F5zhUTEQ8C/A/eka3ho2nQ7sAn4GHAYMBY4r2TXo4AlZNfzW4CAb6f6HpDqMDWV0dzPqSXnCzABuBvYDZgF3AAg6ePAhcDIiOhN9vNe2tw52/bLQch2JFMj4t2IeA8gImZGxJqI2ED2x/FQSbs2se9G4KqI2BgRs4G1wMdbkze1Gk4CroiIdRHxEtkf9rIk7QUcA1wWEesj4jngZuDzJWV+TFK/iFgbEU+WpPclu7+zOSIWRsQ7LSmzQfl7AMcBF6druAK4Fji1JNvyiPhBRGyKiPciYnFEzI2IDRFRC3yf7B+B9jhfgCciYnZEbAbuBOqD5WagBzBcUveIWBoRf27tOdv2wUHIdiTL6hckVUiaJunPkt7h/f+U+zWx78oGgwPWAb1ambc/UFlajwbLzdkTqIuINSVpfyFrbUHW4toP+GPqcjshpd8JzAHulrRc0nckdW9hmaX+DugOvJ669d4GbiJr9TR6LpIGSLo7dYu9A/wXTV/fhsqdL8AbJcvrgCpJlRGxGLiY7B+LFakOe7awXNvOOAhZZ9TUkM7S9NOBicAYsu6qISld+VWLWrLurMElaXu1cN/lQB9JvUvS9gb+ChARr0TEaWRB4WrgPkk7p9bYlRExHPh74ATgzBaU1/AaLgM2AP0iYrf02iUiDmxmn2+ntENSN+EZbHt9mxt62+z5lq18xI8i4hiy4Blk18Q6IQch64zeBPYpk6c32R/VlcBHyO6B5Cp1G/2M7Kb9RyTtT8sCAhGxDPgt8O002OAQstbPXQCSzpDUPyK2AG+n3TZL+gdJB6euwHfIuuc2t6DIN4Ehkrql8l8HHga+J2kXSd3SwIPmutd6k3VFvi1pEHBpI2U0+nMqd77NkfRxSZ+R1ANYT3YPrCXnbNshByHrjL4NfD11G/1bE3nuIOve+SvwEvBkE/na24VkLa83yLrKfkwWDFviNLIW23LgfrJ7S3PTtvHAi5LWkg1SODUi1pMNgLiPLAAtAv6brFusnJ+k95WSnknLZwI7kV2vVem4A5s5xpXA4cBq4BdkAbhUuZ9Tc+fbnB7ANOAtsus8gGy0n3VC/rKqWY4kXQ18NCJaM0rOrMtwS8isHUnaX9IhyhxJ1sV0f9H1Mtte+VvPZu2rN1kX3J7ACuB7wAOF1shsO+buODMzK4y748zMrDDujiujX79+MWTIkKKrYWbWqSxcuPCtiOhfLp+DUBlDhgxhwYIFRVfDzKxTkfSXluRzd5yZmRXGQcjMzArjIGRmZoXxPSEzswY2btxITU0N69evL7oq272qqioGDx5M9+5teXi7g5CZ2QfU1NTQu3dvhgwZgpTng9c7t4hg5cqV1NTUMHTo0DYdw91xZmYNrF+/nr59+zoAlSGJvn37fqgWo4OQmVkjHIBa5sNeJwehHP1iyS94d+O7RVfDzGy75SCUk0UrFzHl8SlM/e3UoqtiZp1Qr17bzi5/2223ceGFFxZUm/w4COVk3aZ1AKxYt6LgmpiZbb8chMzMOpna2lpOOukkRo4cyciRI/nNb35TdJXazEO0c+IpMsx2DFf+/EVeWv5Oux5z+J67cMU/Hthsnvfee48RI0ZsXa+rq2PChAkAXHTRRVxyySUcc8wxvPbaa4wbN45Fixa1ax07ioOQmdl2qGfPnjz33HNb12+77batD1N+5JFHeOmll7Zue+edd1izZg29e/fu8Hp+WA5CZmbNKNdiKcKWLVv43e9+R8+ePYuuyofme0JmZp3M2LFjueGGG7aul7aYOptcg5CkpZJekPScpAUprY+kuZJeSe+7p3RJul7SYknPSzq85DiTUv5XJE0qST8iHX9x2ldtLcPMrLO4/vrrWbBgAYcccgjDhw/nxhtvLLpKbdYR3XH/EBFvlaxPAeZFxDRJU9L6ZcBxwLD0OgqYDhwlqQ9wBVANBLBQ0qyIWJXyTAaeBGYD44FftraMPE/ezKwt1q5du836WWedxVlnnQVAv379uOeeewqoVfsrojtuInB7Wr4d+GxJ+h2ReRLYTdJAYBwwNyLqUuCZC4xP23aJiN9FNhTtjgbHak0ZZmZWgLyDUAAPS1ooaXJK2yMiXgdI7wNS+iBgWcm+NSmtufSaRtLbUsY2JE2WtEDSgtra2lacrpmZtUbe3XFHR8RySQOAuZL+2Ezexp6CF21Ib06L9omIGcAMgOrqan/hx8wsJ7m2hCJieXpfAdwPHAm8Wd8Flt7rn2tTA+xVsvtgYHmZ9MGNpNOGMszMrAC5BSFJO0vqXb8MjAX+AMwC6ke4TQIeSMuzgDPTCLZRwOrUlTYHGCtp9zTKbSwwJ21bI2lUGhV3ZoNjtaaMdhdlG2VmZpZnd9wewP1p1HQl8KOIeEjSfOBeSecCrwEnp/yzgeOBxcA64GyAiKiT9A1gfsp3VUTUpeUvArcBPclGxf0ypU9rTRlmZlaM3IJQRCwBDm0kfSVwbCPpAVzQxLFmAjMbSV8AHNQeZeTFE2OZWVtUVFRw8MEHs3HjRiorK5k0aRIXX3wx3bo134F16aWXMnv2bI4//ni++93vtrrcXr16sXbtWpYuXcpvf/tbTj/99LaeQov4sT1mZtuh0mfHrVixgtNPP53Vq1dz5ZVXNrvfTTfdRG1tLT169PhQ5S9dupQf/ehHuQchP7bHzGw7N2DAAGbMmMENN9xARLB582YuvfRSRo4cySGHHMJNN90EwIQJE3j33Xc56qijuOeee/j5z3/OUUcdxWGHHcaYMWN48803AZg6dSrXXHPN1uMfdNBBLF26dJsyp0yZwuOPP86IESO49tprczs3t4TMzJrzyynwxgvte8yPHgzHTWvVLvvssw9btmxhxYoVPPDAA+y6667Mnz+fDRs2cPTRRzN27FhmzZpFr169tragVq1axZNPPokkbr75Zr7zne/wve99r0XlTZs2jWuuuYYHH3yw1afXGg5CZmadRP08ZQ8//DDPP/889913HwCrV6/mlVdeYejQodvkr6mp4ZRTTuH111/nb3/72we2bw8chMzMmtPKFktelixZQkVFBQMGDCAi+MEPfsC4ceOa3edLX/oS//qv/8qECRN49NFHmTp1KgCVlZVs2bJla77169fnWfVm+Z6Qmdl2rra2lvPPP58LL7wQSYwbN47p06ezceNGAP70pz/x7rvvfmC/1atXM2hQ9mSy22+/fWv6kCFDeOaZZwB45plnePXVVz+wb+/evVmzZk0ep7MNByEzs+1Q/fTeBx54IGPGjGHs2LFcccUVAJx33nkMHz6cww8/nIMOOogvfOELbNq06QPHmDp1KieffDKf/OQn6dev39b0k046ibq6OkaMGMH06dPZb7/9PrDvIYccQmVlJYceemiuAxNU38dojauuro76KXVbY/4b8zlnzjlU71HNreNvzaFmZpaXRYsWccABBxRdjU6jseslaWFEVJfb1y0hMzMrjIOQmZkVxkHIzMwK4yBkZmaFcRAyM7PCOAjlxKMOzczKcxAyM9sOLV26lIMO2nammoYPHm1owYIFfPnLXwZgw4YNjBkzhhEjRnDPPffkWtcPw4/tMTPbQVRXV1NdnX0159lnn2Xjxo1bH2baEps3b6aioiKv6jXKLSEzs05m9OjRXHbZZRx55JHst99+PP744wA8+uijnHDCCaxYsYIzzjiD5557jhEjRvDnP/+ZefPmcdhhh3HwwQdzzjnnsGHDBiB7hM9VV13FMcccw09+8hNGjx7NJZdcwqc+9SkOOOAA5s+fz4knnsiwYcP4+te/3u7n4paQmVkzrn76av5Y98d2Peb+ffbnsiMv+1DH2LRpE08//TSzZ8/myiuv5JFHHtm6bcCAAdx8881bp2JYv349o0ePZt68eey3336ceeaZTJ8+nYsvvhiAqqoqnnjiCQBuvPFGdtppJx577DGuu+46Jk6cyMKFC+nTpw/77rsvl1xyCX379v1QdS/lllDOPL23mbVFU3876tNPPPFEAI444ogPTEjX0Msvv8zQoUO3PiNu0qRJPPbYY1u3n3LKKdvknzBhAgAHH3wwBx54IAMHDqRHjx7ss88+LFu2rE3n0xS3hMzMmvFhWyxt1bdvX1atWrVNWl1d3dY5geqn766oqGj04aWlyo3W3XnnnbdZrz92t27dtpkmvFu3bmXLai23hHISeIi2mbVdr169GDhwIPPmzQOyAPTQQw9xzDHHtPpY+++/P0uXLmXx4sUA3HnnnXz6059u1/q2lVtCZmbbqTvuuIMLLriAr3zlKwBcccUV7Lvvvq0+TlVVFbfeeisnn3wymzZtYuTIkZx//vntXd028VQOZbR1KoenXn+K8x4+j5EfHcnMcTNzqJmZ5cVTObSOp3IwM7NOyUHIzMwK4yBkZtYI36pomQ97nRyEzMwaqKqqYuXKlQ5EZUQEK1eupKqqqs3H8Og4M7MGBg8eTE1NDbW1tUVXZbtXVVXF4MGD27x/7kFIUgWwAPhrRJwgaShwN9AHeAb4fET8TVIP4A7gCGAlcEpELE3HuBw4F9gMfDki5qT08cB1QAVwc0RMS+mtLsPMrF737t23finU8tUR3XEXAYtK1q8Gro2IYcAqsuBCel8VER8Drk35kDQcOBU4EBgP/KekihTcfggcBwwHTkt5W11GHuq/rCr82B4zs6bkGoQkDQb+J3BzWhfwGeC+lOV24LNpeWJaJ20/NuWfCNwdERsi4lVgMXBkei2OiCUR8Teyls/ENpZhZmYFyLsl9B/AV4Etab0v8HZE1D98qAYYlJYHAcsA0vbVKf/W9Ab7NJXeljK2IWmypAWSFrhP2MwsP7kFIUknACsiYmFpciNZo8y29kovV/77CREzIqI6Iqr79+/fyC5mZtYe8hyYcDQwQdLxQBWwC1nLaDdJlaklMhhYnvLXAHsBNZIqgV2BupL0eqX7NJb+VhvKMDOzAuTWEoqIyyNicEQMIRtY8KuI+Gfg18DnUrZJwANpeVZaJ23/VWSD9GcBp0rqkUa9DQOeBuYDwyQNlbRTKmNW2qe1ZZiZWQGK+J7QZcDdkr4JPAvcktJvAe6UtJisdXIqQES8KOle4CVgE3BBRGwGkHQhMIdsiPbMiHixLWWYmVkxOiQIRcSjwKNpeQnZyLaGedYDJzex/7eAbzWSPhuY3Uh6q8swM7OO58f2mJlZYRyEclJ/q8lfVjUza5qDkJmZFcZByMzMCuMgZGZmhXEQMjOzwjgImZlZYRyEzMysMA5CefMIbTOzJjkImZlZYRyEzMysMA5CZmZWGAehnMQH58ozM7MGHITMzKwwDkJmZlYYByEzMyuMg5CZmRXGQShnnk/IzKxpDkJ58eA4M7OyHITMzKwwDkI58feEzMzKcxAyM7PCOAjlxC0hM7PyHIRyEuEgZGZWjoNQzjxE28ysaQ5COXF3nJlZeQ5CZmZWGAchMzMrjIOQmZkVJrcgJKlK0tOSfi/pRUlXpvShkp6S9IqkeyTtlNJ7pPXFafuQkmNdntJfljSuJH18SlssaUpJeqvLaG8eHWdmVl6eLaENwGci4lBgBDBe0ijgauDaiBgGrALOTfnPBVZFxMeAa1M+JA0HTgUOBMYD/ympQlIF8EPgOGA4cFrKS2vLMDOzYuQWhCKzNq12T68APgPcl9JvBz6bliemddL2YyUppd8dERsi4lVgMXBkei2OiCUR8TfgbmBi2qe1ZbQ7j44zMysv13tCqcXyHLACmAv8GXg7IjalLDXAoLQ8CFgGkLavBvqWpjfYp6n0vm0oo2G9J0taIGlBbW1t207ezMzKyjUIRcTmiBgBDCZruRzQWLb03liLJNoxvbkytk2ImBER1RFR3b9//0Z2Kc/3hMzMyuuQ0XER8TbwKDAK2E1SZdo0GFielmuAvQDS9l2ButL0Bvs0lf5WG8pod/XdcTn19pmZ7RDyHB3XX9JuabknMAZYBPwa+FzKNgl4IC3PSuuk7b+KrDkxCzg1jWwbCgwDngbmA8PSSLidyAYvzEr7tLYMMzMrQGX5LG02ELg9jWLrBtwbEQ9Kegm4W9I3gWeBW1L+W4A7JS0ma52cChARL0q6F3gJ2ARcEBGbASRdCMwBKoCZEfFiOtZlrSkjDx6YYGZWXm5BKCKeBw5rJH0J2f2hhunrgZObONa3gG81kj4bmN0eZZiZWcfzExPy4oaQmVlZDkJmZlaYFgUhSRdJ2kWZWyQ9I2ls3pXrzHxPyMysvJa2hM6JiHeAsUB/4GxgWm61MjOzLqGlQaj+yy7HA7dGxO9p/IuflrglZGZWXkuD0EJJD5MFoTmSegNb8qtW51f/9SNP721m1rSWDtE+l+xJ2EsiYp2kPmRdcmZmZm3W0pbQJ4CXI+JtSWcAXyd7+Kc1wd1xZmbltTQITQfWSToU+CrwF+CO3GplZmZdQkuD0Kb0jLWJwHURcR3QO79qdX5uCZmZldfSe0JrJF0OfB74ZHoeXPf8qmVmZl1BS1tCp5BN131ORLxBNjncd3Or1Y7ADSEzs7JaFIRS4LkL2FXSCcD6iPA9ITMz+1Ba+tiefyKbw+dk4J+ApyR9rvm9ujbfEzIzK6+l94T+NzAyIlZANmEd8AhwX14VMzOzHV9L7wl1qw9AycpW7NslecJWM7PyWtoSekjSHODHaf0UGplMzt63tTvOT+0xM2tSi4JQRFwq6STgaLI/qzMi4v5ca2ZmZju8Fk/vHRE/BX6aY112KB6YYGZWXrNBSNIaGv/Gi4CIiF1yqZWZmXUJzQahiPCjedrIAxPMzMrzCDczMyuMg5CZmRXGQcjMzArjIJQTj44zMyvPQShn8rdVzcya5CCUE4+OMzMrz0EoJ+6OMzMrz0HIzMwKk1sQkrSXpF9LWiTpRUkXpfQ+kuZKeiW9757SJel6SYslPS/p8JJjTUr5X5E0qST9CEkvpH2ul6S2ltHe3B1nZlZeni2hTcBXIuIAYBRwgaThwBRgXkQMA+aldYDjgGHpNRmYDllAAa4AjgKOBK6oDyopz+SS/can9FaVYWZmxcgtCEXE6xHxTFpeAywCBgETgdtTttuBz6blicAdkXkS2E3SQGAcMDci6iJiFTAXGJ+27RIRv4us2XFHg2O1pgwzMytAh9wTkjQEOAx4CtgjIl6HLFABA1K2QcCykt1qUlpz6TWNpNOGMhrWd7KkBZIW1NbWtuZUzcysFXIPQpJ6kU0BcXFEvNNc1kbSog3pzVanJftExIyIqI6I6v79+5c5ZOM8Os7MrLxcg5Ck7mQB6K6I+FlKfrO+Cyy9108bXgPsVbL7YGB5mfTBjaS3pYzc+MuqZmZNy3N0nIBbgEUR8f2STbOA+hFuk4AHStLPTCPYRgGrU1faHGCspN3TgISxwJy0bY2kUamsMxscqzVltDuPjjMzK6/FM6u2wdHA54EXJD2X0r4GTAPulXQu8Bpwcto2GzgeWAysA84GiIg6Sd8A5qd8V0VEXVr+InAb0BP4ZXrR2jLMzKwYuQWhiHiCxu/BABzbSP4ALmjiWDOBmY2kLwAOaiR9ZWvLaG++J2RmVp6fmJATByEzs/IchMzMrDAOQjnxwAQzs/IchHKWHmdnZmaNcBAyM7PCOAiZmVlhHIRy4ntCZmblOQiZmVlhHIRy4u8JmZmV5yCUEwchM7PyHITMzKwwDkI58cAEM7PyHIRy5vmEzMya5iCUE98TMjMrz0HIzMwK4yBkZmaFcRAyM7PCOAjlxKPjzMzKcxAyM7PCOAjlxKPjzMzKcxDKSX13nL8nZGbWNAchMzMrjINQTtwdZ2ZWnoOQmZkVxkHIzMwK4yBkZmaFcRDKib+samZWnoOQmZkVJrcgJGmmpBWS/lCS1kfSXEmvpPfdU7okXS9psaTnJR1ess+klP8VSZNK0o+Q9ELa53pJamsZefDoODOz8vJsCd0GjG+QNgWYFxHDgHlpHeA4YFh6TQamQxZQgCuAo4AjgSvqg0rKM7lkv/FtKcPMzIqTWxCKiMeAugbJE4Hb0/LtwGdL0u+IzJPAbpIGAuOAuRFRFxGrgLnA+LRtl4j4XWQ3X+5ocKzWlJGLM4afwR4f2YOqyqq8ijAz6/Q6+p7QHhHxOkB6H5DSBwHLSvLVpLTm0msaSW9LGR8gabKkBZIW1NbWtuoE63Xv1p2elT3btK+ZWVexvQxMaOwBa9GG9LaU8cHEiBkRUR0R1f379y9zWDMza6uODkJv1neBpfcVKb0G2Ksk32BgeZn0wY2kt6WMXHmAgplZ0zo6CM0C6ke4TQIeKEk/M41gGwWsTl1pc4CxknZPAxLGAnPStjWSRqVRcWc2OFZryjAzs4JU5nVgST8GRgP9JNWQjXKbBtwr6VzgNeDklH02cDywGFgHnA0QEXWSvgHMT/muioj6wQ5fJBuB1xP4ZXrR2jLylEaNm5lZE3ILQhFxWhObjm0kbwAXNHGcmcDMRtIXAAc1kr6ytWXkyU9OMDNr2vYyMGGH5AntzMya5yCUMw9MMDNrmoOQmZkVxkEoR+6OMzNrnoOQmZkVxkEoR5I8Os7MrBkOQmZmVhgHITMzK4yDUM48RNvMrGkOQjnyY3vMzJrnIJQzD0wwM2uag5CZmRXGQShHQr4nZGbWDAchMzMrjINQjvzYHjOz5jkI5czdcWZmTXMQypGHaJuZNc9BKG9uCJmZNclByMzMCuMglCMP0TYza56DkJmZFcZBKGduCZmZNc1ByMzMCuMglCMP0TYza56DUM78FG0zs6Y5CJmZWWEchHLkIdpmZs1zEDIzs8I4COXILSEzs+Z1uSAkabyklyUtljSl6PqYmXVlXSoISaoAfggcBwwHTpM0PMfy8jq0mdkOobLoCnSwI4HFEbEEQNLdwETgpfYu6Mn//Bfe6/4Sf6rczLibDwXYOsWdQ5OZdQbHb96HL33h/lzL6GpBaBCwrGS9BjiqYSZJk4HJAHvvvXebCxu/dicWVm3a5q5QyPeIzKxz+IjyDxFdLQg11gj5QFSIiBnADIDq6uo2RY1R/+v/MqotO5qZdSFd6p4QWctnr5L1wcDygupiZtbldbUgNB8YJmmopJ2AU4FZBdfJzKzL6lLdcRGxSdKFwBygApgZES8WXC0zsy6rSwUhgIiYDcwuuh5mZtb1uuPMzGw74iBkZmaFcRAyM7PCOAiZmVlh5Jk/myepFvhLG3fvB7zVjtXp7Hw9tuXr8T5fi23tCNfj7yKif7lMDkI5krQgIqqLrsf2wtdjW74e7/O12FZXuh7ujjMzs8I4CJmZWWEchPI1o+gKbGd8Pbbl6/E+X4ttdZnr4XtCZmZWGLeEzMysMA5CZmZWGAehnEgaL+llSYslTSm6PnmTtJekX0taJOlFSRel9D6S5kp6Jb3vntIl6fp0fZ6XdHixZ5APSRWSnpX0YFofKumpdD3uSVOKIKlHWl+ctg8pst55kLSbpPsk/TF9Tj7RVT8fki5Jvyd/kPRjSVVd9bPhIJQDSRXAD4HjgOHAaZKGF1ur3G0CvhIRBwCjgAvSOU8B5kXEMGBeWofs2gxLr8nA9I6vcoe4CFhUsn41cG26HquAc1P6ucCqiPgYcG3Kt6O5DngoIvYHDiW7Ll3u8yFpEPBloDoiDiKbVuZUuupnIyL8aucX8AlgTsn65cDlRderg6/BA8D/AF4GBqa0gcDLafkm4LSS/Fvz7Sgvspl75wGfAR4km17+LaCy4eeEbI6rT6TlypRPRZ9DO16LXYBXG55TV/x8AIOAZUCf9LN+EBjXVT8bbgnlo/5DVq8mpXUJqbvgMOApYI+IeB0gvQ9I2brCNfoP4KvAlrTeF3g7Ijal9dJz3no90vbVKf+OYh+gFrg1dU/eLGlnuuDnIyL+ClwDvAa8TvazXkgX/Ww4COVDjaR1ibHwknoBPwUujoh3msvaSNoOc40knQCsiIiFpcmNZI0WbNsRVAKHA9Mj4jDgXd7vemvMDns90n2vicBQYE9gZ7Lux4a6xGfDQSgfNcBeJeuDgeUF1aXDSOpOFoDuioifpeQ3JQ1M2wcCK1L6jn6NjgYmSFoK3E3WJfcfwG6S6mc0Lj3nrdcjbd8VqOvICuesBqiJiKfS+n1kQakrfj7GAK9GRG1EbAR+Bvw9XfSz4SCUj/nAsDTaZSeym46zCq5TriQJuAVYFBHfL9k0C5iUlieR3SuqTz8zjYIaBayu75bZEUTE5RExOCKGkP38fxUR/wz8GvhcytaedUC2AAACwklEQVTwetRfp8+l/DvMf7sR8QawTNLHU9KxwEt0zc/Ha8AoSR9Jvzf116JLfjb8xIScSDqe7D/fCmBmRHyr4CrlStIxwOPAC7x/D+RrZPeF7gX2JvvlOzki6tIv3w3AeGAdcHZELOjwincASaOBf4uIEyTtQ9Yy6gM8C5wRERskVQF3kt1LqwNOjYglRdU5D5JGADcDOwFLgLPJ/hHucp8PSVcCp5CNKn0WOI/s3k+X+2w4CJmZWWHcHWdmZoVxEDIzs8I4CJmZWWEchMzMrDAOQmZmVhgHIbMOIOm36X2IpNPb+dhfa6wss87AQ7TNOlDpd4ZasU9FRGxuZvvaiOjVHvUz62huCZl1AElr0+I04JOSnktzylRI+q6k+WnenC+k/KOVzc/0I7IvACPp/0lamOahmZzSpgE90/HuKi0rPW3gu2nOmhcknVJy7EdL5va5K3051KzDVZbPYmbtaAolLaEUTFZHxEhJPYDfSHo45T0SOCgiXk3r56SnCfQE5kv6aURMkXRhRIxopKwTgRFkc/f0S/s8lrYdBhxI9nyy35A96+6J9j9ds+a5JWRWrLFkz0h7juwRR33JJnIDeLokAAF8WdLvgSfJHmg5jOYdA/w4IjZHxJvAfwMjS45dExFbgOeAIe1yNmat5JaQWbEEfCki5myTmN07erfB+hiyyc3WSXoUqGrBsZuyoWR5M/5bYAVxS8isY60BepeszwG+mKbBQNJ+abK3hnYlm+J5naT9yaZQr7exfv8GHgNOSfed+gOfAp5ul7Mwayf+78esYz0PbErdarcB15F1hT2TBgfUAp9tZL+HgPMlPU821fWTJdtmAM9LeiZNF1HvfrJpon9PNgnaVyPijRTEzLYLHqJtZmaFcXecmZkVxkHIzMwK4yBkZmaFcRAyM7PCOAiZmVlhHITMzKwwDkJmZlaY/w83/cmm54oP5wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the loss\n",
    "\n",
    "plt.plot(training_results_He['training_loss'], label='He')\n",
    "plt.plot(training_results['training_loss'], label='Default')\n",
    "plt.plot(training_results_Uniform['training_loss'], label='Uniform')\n",
    "plt.ylabel('loss')\n",
    "plt.xlabel('iteration ') \n",
    "plt.title('training loss iterations')\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compare the validation loss for each model  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xd8VfX9+PHXO3snQMIKIiBThqhhVFFx4ajFXfcXrBatWhX9Wm2/to6On7ZWbdWqqFVsteDGWrUCddYBEaMoIwwRIpgEQkJC5k3evz/OSQgj5GacnHtz38/H4z7OyL3nvC9Xz/uczxRVxRhjTOSK8jsAY4wx/rJEYIwxEc4SgTHGRDhLBMYYE+EsERhjTISzRGCMMRHOEoExxkQ4SwTGGBPhLBEYY0yEi/E7gGBkZmbqoEGD/A7DGGPCyqeffrpVVbNae19YJIJBgwaRm5vrdxjGGBNWROSbYN5nRUPGGBPhLBEYY0yEs0RgjDERzhKBMcZEOEsExhgT4TxNBCJynYh8KSJficj17r6eIrJQRNa4yx5exmCMMWb/PEsEIjIG+DEwETgEOE1EhgG3AItVdRiw2N02xhjjEy/7EYwCPlbVSgAReRc4EzgdmOq+Zy7wDnCzh3EYY0yXU1UCDUptoMF51TvLmkADNYF6Z1nXQHWgnpo6d5+7rK7b9Z5LjxxMz+Q4T2P1MhF8CfxWRHoBVcCpQC7QR1W3AKjqFhHp7WEMxpgIV1ffQFVdPVW19VTWOsuqugDVdQ1U1dZT7V54q+rqqXHfVx2op6p210W6zr2I19U7F/SawB77ml3sa5qtd3RKeBE4fXz/8E0EqrpSRO4GFgIVwOdAINjPi8gsYBbAwIEDPYnRGBM66uobqKgOUFEToLw6QHl1HRU1zvauC3g9lbUBqmobqKrbc/+e73HW6+rbfjWOjhISY6NJiI0iPiaa2GghLiaKuJgoYqOjiIuOIiU+hrgkd9vdHx/r/C3efW+c+zdnO7rpGPHuKyE22l2PJj62+fau84qIB//au/N0iAlVfQJ4AkBEfgcUAIUi0s99GugHFLXw2TnAHICcnJwO5lVjTEeoKtV1DeysDbCzJsDOmvqm9craeudiXRNgZ239rjviQAO19fX7LBqpDTh34BXVAXZUB6ioqaO6riGoWKIEkuJiSIyLJjE2mqS46Kb1HkmxJMbFkBgb1eJ7kuKiSYh11hPd9YSYaBLiopr2x0ZHVoNKTxOBiPRW1SIRGQicBXwPGAzMAO5ylwu8jMEY41zIawINTXfJ1e4ddGllHdsraymtrGV7ZR2llXXueuO2syyvrqOhDbdjjXfCze+KG9cb75p7JcdxYK9kUuJjSE2IITU+hpSEmKbtlPhYUhNiSI6PISlu18U8LjqqS+6SI4nXg8696NYR1AFXq+p2EbkLeE5ELgM2Aud6HIMxYU1V2VEVoLSq1rlQV9VRVlVHWeWu7dJKd19VLRU1zoW+um7XRT/Yu+3kuGgykuLokRxLj6Q4DuiZRI+kWNISYkmKjyYlPoakuBhS4qNJinMu0snx0STH7bpgx0VHERVlF+pw4nXR0FH72LcNON7L8xoT6lSVipoAWytqKS6vYWtFDcXlNbuvV+za3l85d1JcNBmJsaQnxZGeGEN2RqJT5BETtavowy3vTozdVSySEBtNRpJzwe+RFEt6UizxMdFd+K9gQkVYDENtTLioqq1nS1mVewGvZWtFTdOruLy22XoNNYG979KjBHqlxJOVEk9WajzDeqeSlRpPZkocPZLiSE+MJSPJeaUnOttxMZFVnm06nyUCY4JU36AUlVezubSKzaWNyyo2l+1a315Zt9fnogR6JseRmRJPZko8gzOTyUzZtZ2VuuvVIymOaCtWMV3MEoGJaHsW0Tiv6l3bze7gi8prqN+jxjQ1IYb+6Yn0z0hg/AEZ9M9IpF96gnsX77x6JtvF3YQ2SwSmW2toUIrKa/i2tJKC7VVNr29Lq/h2eyWbS6upqqvf63PRUUKv5LimC/rwPqn0SYunf0ai80pPpF9GAmkJsT58K2M6lyUCE7bq6hsoLq+hcEc1hTtqKCqvblrfXOpc7LeUVlNbv3tZfM/kOLIzEhneJ5WpI3rTJ22PIpoUp4jGWr6YSGGJwIS02kADq78r5/OCUlZs2cF3Zbsu9tt21uzVhT86SuidGk+ftATGZqdzyph+ZPdIZECPRAZkJJLdI5GkOPvP3pjm7P8IEzIaGpT1W3fyRUEpn28q5fOCMlZs2UGt27omPTGWAT0S6ZOWwLgB6fROTaBPWgJ90pwLf++0eHolx1t5vDFtZInA+EJV2VhSyVebd/BFQRlfFJSyvKCM8hpnOKqkuGjGZKcz84hBjBuQziEDMhjQI9F6lBrjAUsExnOB+gbWFlfw1bc7+GrzDr7aXMaKzTuaLvqx0cKofmmcfmh/xg3IYPwBGRyUlWJ39sZ0EUsEplOpOsU7H6/fxpffOhf9Vd+VNxXvJMRGNV30R/dPZ3T/NIb3SSUh1nq0GuMXSwSmw7ZV1PDfddv4YE0xH6zZyuayagDSEmIY3T+dGd87sOmiP8Tu9I0JOZYITJtV19WzdEMJH6zZyvtrtrJiyw7AufAfcVAmVx2byZFDMxnUK8nK9I0JA5YITFC+3rqTRSsKeTe/mCUbSqgNNBAbLRw2sAf/O204U4ZlMTY73e72jQlDlgjMPjU0KHkFpSxcUcjCFYWsLaoAYHifFC6ZfCBThmYycXBPkuPtPyFjwp39X2yaVNfV8+G6rSxcUciilUUUl9cQHSVMGtyTiyYN5IRRfTigZ5LfYRpjOpnXM5TNBi4HFFgOXAr0A+YBPYFlwCWqWutlHKZllbUB3lj+HW+t+I738rdSVVdPSnwMx4zI4sRRfTh2RG/Sk2w8HWO6M88SgYhkA9cCB6tqlYg8B5wPnArcp6rzROQR4DLgYa/iMPu2YetO/vbxNzyXu4ny6gB90xI4+/BsTjy4L5OH9LQJSoyJIF4XDcUAiSJSByQBW4DjgAvdv88FbscSQZdoaFDezS9m7kcbeGd1MTFRwilj+3HJ5AOZMKiHtfAxJkJ5lghU9VsRuQdnXuIq4C3gU6BUVQPu2wqAbK9iMI6yyjqe/3QTT3/0DRtLKslKjef6E4Zx4cSB9E5L8Ds8Y4zPvCwa6gGcDgwGSoHngVP28dZ9TsYqIrOAWQADBw70KMrubcXmHfzt4w28/Nm3VNc1MGFQD246aQQnje5r0xsaY5p4WTR0AvC1qhYDiMhLwBFAhojEuE8FA4DN+/qwqs4B5gDk5OS0PHO32UvuhhLuXZjPh+u2kRAbxZmHZnPJ5EEc3D/N79CMMSHIy0SwEZgsIkk4RUPHA7nA28A5OC2HZgALPIwhony+qZQ/LsznvfxiMlPi+cWpIzkvZ6C1+jHG7JeXdQSfiMgLOE1EA8BnOHf4/wLmichv3H1PeBVDpPhqcxn3Lcxn0coieiTF8vNTRvI/3xtEYpy1/DHGtM7TVkOqehtw2x671wMTvTxvpMgvLOf+Rfm8vvw70hJi+N9pw5l55GBSrLevMaYN7IoRhtYXV/CnxWt49fPNJMfFcO3xw7hsymDSE60IyBjTdpYIwkhZVR2/eW0FLy4rID4mmiuPOYhZRw2hR3Kc36EZY8KYJYIwsbm0iplPLuHrrTv50ZGDuXLqQWSmxPsdljGmG7BEEAZWfbeDmX9dys6aAHMvncgRQzP9DskY041YIghxH67byhVPf0pyfAzPXfk9RvWzvgDGmM5liSCEvfr5Zm58Lo/Bmck8delE+mck+h2SMaYbskQQglSVx95fz+9eX8WkwT2Zc0mOdQozxnjGEkGIqW9Qfv3aCp76cAPfH9ePe394iA0JbYzxlCWCEFJdV8/s+Xm88eV3XDZlMP936iiibA5gY4zHLBGEiNLKWn78dC5LN2zn1u+P4vKjhvgdkjEmQlgiCAEF2yuZ+eRSNm6r5MELD+W0cf39DskYE0EsEfisvLqO/3liCcUVNTx92UQmD+nld0jGmAhjicBHqsotLy7nm5JKnr18EpMsCRhjfGDTVPlo7ocb+NfyLdx00ghLAsYY31gi8MlnG7fz29dXcsKo3syyimFjjI8sEfhg+85arn5mGX3SEvjjueOtiagxxleeJQIRGSEiec1eO0TkehHpKSILRWSNu+zhVQyhqKFBmf1cHlsrann4osOtx7AxxneeJQJVXa2q41V1PHA4UAm8DNwCLFbVYcBidzti/OWdtbyzuphf/uBgxg5I9zscY4zpsqKh44F1qvoNcDow190/Fziji2Lw3YfrtnLvwnymH9KfiycN9DscY4wBui4RnA/8w13vo6pbANxl7319QERmiUiuiOQWFxd3UZjeKdxRzbX/+IwhWSn8v7PGImL1AsaY0OB5IhCROGA68HxbPqeqc1Q1R1VzsrKyvAmuiwTqG/jpPz5jZ009D190GMk2ubwxJoS0mghEZEwHz3EKsExVC93tQhHp5x67H1DUweOHvHveymfJ1yX87qwxDOuT6nc4xhizm2CeCB4RkSUicpWIZLTjHBewq1gI4FVghrs+A1jQjmOGjcUrC3nk3XVcMHEgZx46wO9wjDFmL60mAlWdAlwEHADkisizInJiMAcXkSTgROClZrvvAk4UkTXu3+5qc9RhYlNJJbPn5zG6fxq3/eBgv8Mxxph9CqqwWlXXiMitQC7wZ+BQcWo7f6GqL+3nc5VArz32bcNpRdSt1QTqufrZZSjw8EWHkxBrk8sYY0JTMHUE40TkPmAlcBzwA1Ud5a7f53F8YevpD7/hi4Iy/nDOOAb2SvI7HGOMaVEwTwQPAo/h3P1XNe5U1c3uU4LZQ2llLQ/8Zw1HD8/i5DH9/A7HGGP2K5hEcCpQpar1ACISBSSoaqWq/s3T6MLUQ2+vpbwmwM9PGel3KMYY06pgWg0tAhKbbSe5+8w+bCqpZO6H33DOYQMY1S/N73CMMaZVwSSCBFWtaNxw163QuwV/+PdqoqLghmnD/Q7FGGOCEkwi2CkihzVuiMjhQNV+3h+xvigo5dXPN3PZlMH0S09s/QPGGBMCgqkjuB54XkQ2u9v9gPO8Cyk8qSq/e30lPZPjuOKYg/wOxxhjgtZqIlDVpSIyEhgBCLBKVes8jyzMvL26iI/Xl3DH9NGkJdgcA8aY8BHs6GcjgIOBBJzOZKjq096FFV4C9Q38v9dXMTgzmQtteGljTJhpNRGIyG3AVJxE8DrOIHIfAJYIXM9/WsCaogoevugwYqNt9k9jTHgJ5qp1Ds6QEN+p6qXAIUC8p1GFkcraAPcuzOfwA3tw8pi+fodjjDFtFkwiqFLVBiAgImk4w0YP8Tas8PHYe19TXF7DL04daZPNGGPCUjB1BLnu8NOPAZ8CFcAST6MKE8XlNTz63jpOHt2Xww/s6Xc4xhjTLvtNBO4Io/9PVUtx5iV4E0hT1S+6JLoQd/+ifGoDDfzs5BF+h2KMMe2236IhVVXglWbbGywJONYWVTBv6SYunDSQIVkpfodjjDHtFkwdwcciMqE9BxeRDBF5QURWichKEfmeiPQUkYUissZd9mjPsf1295urSIyN5rrjh/kdijHGdEgwieBY4CMRWSciX4jIchEJ9qngT8CbqjoSp7XRSuAWYLGqDgMWu9thZcnXJSxcUchPph5ErxRrQGWMCW/BVBaf0p4Duy2MjgZmAqhqLVArIqfj9EsAmAu8A9zcnnP4QVX57esr6ZuWwI+OHOx3OMYY02HBPBFoC6/WDAGKgSdF5DMReVxEkoE+qroFwF32blfkPvnX8i18vqmUG6YNJzHOpp80xoS/YJ4I/oVz4RecISYGA6uB0UEc+zDgp6r6iYj8iTYUA4nILGAWwMCBoTFsg6ry4H/WMrxPCmcfNsDvcIwxplMEM+jc2Obb7pDUVwRx7AKgQFU/cbdfwEkEhSLST1W3iEg/nA5q+zrvHGAOQE5OTjBPIJ7L21TKqu/K+e2ZY4iOss5jXamuro6CggKqq6v9DiXkJSQkMGDAAGJjbfBDE5xgB51roqrLgmlFpKrficgmERmhqqtxhqlY4b5mAHe5ywVtjcEv85ZsIjE2mumH9Pc7lIhTUFBAamoqgwYNsh7c+6GqbNu2jYKCAgYPtjosE5xgBp27odlmFE5xT3GQx/8p8IyIxAHrgUvdYzwnIpcBG4Fz2xSxTypqAvzzi8384JB+pNow012uurrakkAQRIRevXpRXBzs/6LGBPdEkNpsPYBTZ/BiMAdX1TwgZx9/Oj6Yz4eSV/M2U1lbz/kTQ6O+IhJZEgiO/TuZtgqmjuCOrggk1M1bupERfVI59IAMv0MxPklJSaGiomn6bp566ilyc3N58MEHfYzKmI5rtfmo2/s3o9l2DxH5t7dhhZavNpfxRUEZ5088wO62jDHdTjD9CLLcQecAUNXthFnb/46at2QTcTFRnHlott+hmBBVXFzM2WefzYQJE5gwYQL//e9//Q7JmKAFU0dQLyIDVXUjgIgcSHAdyrqFqtp6Xsn7llPH9CUjKc7vcIyPqqqqGD9+fNN2SUkJ06dPB+C6665j9uzZTJkyhY0bN3LSSSexcuVKv0I1pk2CSQT/B3wgIu+620fjdvSKBP9avoXy6oBVEoeQO/75FSs27+jUYx7cP43bfrD/PpKJiYnk5eU1bTfWEQAsWrSIFStWNP1tx44dlJeXk5qautdxjAk1wVQWv+l2IpuM07t4tqpu9TyyEDFvyUaGZCYzabBNPGNa1tDQwEcffURiYqLfoRjTZsH0IzgT+I+qvuZuZ4jIGar6SisfDXtrCsvJ/WY7Pz/FpqEMJa3dufth2rRpPPjgg9x0000A5OXl7VaMZEwoC6ay+DZVLWvccCuOb/MupNAxb+kmYqOFsw+3cYXM/v35z38mNzeXcePGcfDBB/PII4/4HZIxQQumjmBfyaLNQ1OEm5pAPS8tK+DEg/uQaXMOGNitDwHAzJkzmTlzJgCZmZnMnz/fh6iM6bhgnghyReReETlIRIaIyH04k9h3a//+qpDtlXWcP8EqiY0x3VswieCnQC0wH3geqAau9jKoUDBvyUYG9EhkytBMv0MxxhhPBdNqaCdhOJ1kR3yzbScfrtvGjScOJ8qGmzbGdHPBtBrKAn6GMxFNQuN+VT3Ow7h8NW/pJqIEzs05wO9QjDHGc8EUDT0DrMKZmewOYAOw1MOYfFVX38DzuQUcN7I3fdMTWv+AMcaEuWASQS9VfQKoU9V3VfVHOJ3LuqXFK4vYWlFjlcTGGP+VbuqS0wSTCOrc5RYR+b6IHAoE1bBeRDaIyHIRyRORXHdfT3dE0zXuskc7Y/fEvKUb6ZMWz9QRWX6HYkJMdHQ048ePZ/To0RxyyCHce++9NDQ0tPq5m266idGjRzd1NmurlJQUADZs2MCzzz7brmOYMFNXDW/+HB44DLZ84fnpgukP8BsRSQduBB4A0oDZbTjHsXsMSXELsFhV7xKRW9ztm9twPM98W1rFu/nFXHPsUGKig8mRJpI0H2uoqKiICy+8kLKyMu64Y/9Tdjz66KMUFxcTH9+x/iiNieDCCy/s0HFMiCtaCS9eDoVfwsQrIHOY56ds9Wqnqq+papmqfqmqx6rq4ar6agfOeTow112fC5zRgWN1queWOo9hP7RKYtOK3r17M2fOHB588EFUlfr6em666SYmTJjAuHHjePTRRwGYPn06O3fuZNKkScyfP59//vOfTJo0iUMPPZQTTjiBwsJCAG6//XbuueeepuOPGTOGDRs27HbOW265hffff5/x48dz3333ddl3NV1EFT6ZA3OmQkUhXPg8nPp7iPV+/Cqvewgr8JaIKPCoqs4B+qjqFgBV3SIiITG3QX2D8nzuJqYMzeSAnkl+h2PCwJAhQ2hoaKCoqIgFCxaQnp7O0qVLqamp4cgjj2TatGm8+uqrpKSkND1JbN++nY8//hgR4fHHH+f3v/89f/zjH4M631133cU999zDa6+95uXXMn6oKIYFV8Gat2DYNDj9IUjpukuj14ngSFXd7F7sF4rIqmA/KCKzcIe7HjjQ+4rb9/KL2VxWza2nHez5uUwHvXELfLe8c4/ZdyycclebP6bqTM3x1ltv8cUXX/DCCy8AUFZWxpo1axg8ePBu7y8oKOC8885jy5Yt1NbW7vV3E4HWLIRXfgLVO+CU38PEWdDFg1x6WhCuqpvdZRHwMjARKBSRfgDusqiFz85R1RxVzcnK8r7i9h9LNtIrOY4TRvXx/Fyme1i/fj3R0dH07t0bVeWBBx4gLy+PvLw8vv76a6ZNm7bXZ376059yzTXXsHz5ch599FGqq6sBiImJ2a3iuXG/6cbqquH1n8Ez50ByFsx6ByZd0eVJAILrUBYPnA0Mav5+Vb2zlc8lA1GqWu6uTwPuBF4FZgB3ucsF7Q2+sxSX17B4VRGXTxlMXIxVEoe8dty5d7bi4mKuvPJKrrnmGkSEk046iYcffpjjjjuO2NhY8vPzyc7OJjk5ebfPlZWVkZ3tTHk6d+7cpv2DBg1qKvJZtmwZX3/99V7nTE1Npby83MNvZbpM4VdOhXDRCph0JZxwB8T6128pmKKhBUAZzkBzNW04dh/gZXcc/xjgWXeSm6XAcyJyGbAROLdtIXe+zzZup75BOWlMX79DMSGscarKuro6YmJiuOSSS7jhhhsAuPzyy9mwYQOHHXYYqkpWVhavvLL3lB2333475557LtnZ2UyePLnpgn/22Wfz9NNPM378eCZMmMDw4cP3+uy4ceOIiYnhkEMOYebMmcye3ZbGeyYkNNTDJ4/CotshIQ0uegGGneh3VEhjGWeLbxD5UlXHdFE8+5STk6ONUwJ64cH/rOGet/L56o6TSI7v9iNsh6WVK1cyatQov8MIG/bvFYIKv4JXr4Vvc2HYSW6FsLfF3iLyqarmtPa+YK56H4rIWFXt5Nq50JFfWEF2RqIlAWNM56urgnd/Dx/+GRLS4azHYOy5vtQFtCSYK98UYKaIfI1TNCSAquo4TyPrQvmF5Qzvk+J3GMaY7mb9u/Da9VCyHsZfBNN+A0mhN/95MIngFM+j8FGgvoH1xTs5xoaUMMZ0lsoSeOuXkPd36DEY/mcBDJnqd1QtCmY+gm9E5BDgKHfX+6r6ubdhdZ0N2yqprW9geO9Uv0MxxoQ7VfjyRXjjZqguhSk3wDE/65LewR0RTPPR64AfAy+5u/4uInNU9QFPI+siawqd5njD+1giMMZ0wPZv4F83wNpFkH04/GAB9PW1nU3QgikaugyY5M5UhojcDXyEMwBd2MsvrEAEhva2OgJjTDs0Ngn9z68BgZPvhok/hqhovyMLWjC9pwSob7Zd7+7rFvILyxnYM4nEuPD50Yw/NmzYwJgxu9/h7TlY3J5yc3O59tprAaipqeGEE05g/PjxzJ8/39NYTRcpWgV/PQn+/XMYdBRc/QlMvjKskgAE90TwJPCJiLzsbp8BPOFdSF0rv7CcYVY/YDySk5NDTo7TjPuzzz6jrq6uaQC6YNTX1xMdHV4XlYhQXwcf3A/v/R7iUuCsx2HsOSHVJLQtghmG+l7gUqAE2A5cqqr3ex1YV6gNNPD11p2M6GvFQqZjpk6dys0338zEiRMZPnw477//PgDvvPMOp512GkVFRVx88cXk5eUxfvx41q1bx+LFizn00EMZO3YsP/rRj6ipcTruDxo0iDvvvJMpU6bw/PPPM3XqVGbPns3RRx/NqFGjWLp0KWeddRbDhg3j1ltv9fNrR6bNeTDnWHj7NzDyNLh6CYwLrX4BbdViIhCRNHfZE2ee4r8DfwO+cfeFva+37iTQoFZRbDpFIBBgyZIl3H///XtNVtO7d28ef/xxjjrqKPLy8sjOzmbmzJnMnz+f5cuXEwgEePjhh5ven5CQwAcffMD5558PQFxcHO+99x5XXnklp59+Og899BBffvklTz31FNu2bevS7xmx6qpg4W3w2HGwsxjOfxbOfdLz3sFdYX9FQ88Cp+GMMdR8HApxt4d4GFeXyHdbDFnRUHi5e8ndrCoJekTzoIzsOZKbJ+5/ojxp4Y6vcf9ZZ50FwOGHH77XpDJ7Wr16NYMHD24aU2jGjBk89NBDXH/99QCcd955u71/+vTpAIwdO5bRo0fTr18/wJkTYdOmTfTq1Wu/5zMd9M1H8Oo1sG0tHHqJ0zEsMcPvqDpNi4lAVU9zl912wPQ1heVECQzJSm79zSbi9erVi+3bt++2r6SkpGlOgcapKKOjowkEAvs9VmtjfO05amnjsaOionab8jIqKqrVc5kOqCmHRXfA0scg48CQ7xjWXsH0I1isqse3ti8crS4sZ1BmMgmxVhkXTlq7c/dKSkoK/fr1Y/HixRx//PGUlJTw5ptvct111/Hkk0+26VgjR45kw4YNrF27lqFDh/K3v/2NY445xqPITZupwspX4c1fwI5vYdJP4PhfQlz3vGlsMRGISAKQBGSKSA92NRlNA/p3QWyeW1NYYfUDpk2efvpprr76am688UYAbrvtNg466KA2HychIYEnn3ySc889l0AgwIQJE7jyyis7O1zTHtvWwes3wbrF0GeMUw9wwES/o/JUi8NQuz2Kr8e56H/LrkSwA3hMVR/skgjxZhjq6rp6Dv7Vm1xz7FBumDaiU49tOp8Nq9w29u/VDrWV8MG98N8/QUwCHPt/MOFyiA7fUYk7PAy1qv4J+JOI/LS7DCfR3PrinTQoDO9rTwTGRLzVb8AbP4PSjTDuPDjxTkiNnImqghl07gERGQMcDCQ02/90MCcQkWggF/hWVU8TkcHAPKAnsAy4RFVr2xN8R+TbGEPGmO0bnAHi8t+ErJEw818waIrfUXW5VjuUichtOOMKPQAcC/wemN6Gc1wHrGy2fTdwn6oOw+mgdlkbjtVp8gvLiYkSBvXqnpU/xpj9qKt2Jot5aBJ8/T6c+Gu48oOITAIQ3FhD5wDHA9+p6qXAIUD8/j/iEJEBwPeBx91tAY4DXnDfMhdnyIoul19YweDMZJusPoy01uTSOOzfaT9UYeVr8PD34O3fwohT4JqlcOS1EB3rd3S+CaYWpEpVG0Qk4PY2LiL4zmT3Az8DGstfegGlqtrlQuUjAAAWLElEQVTY8LkAyN7XB0VkFjALYODAgUGeLnj5heWMHZDe6cc13khISGDbtm306tWrxY5dxkkC27ZtIyEhofU3R5qNH8PCX8GmTyBzOFzyChx0rN9RhYRgEkGuiGQAj+H0Mq4AlrT2IRE5DShS1U9FZGrj7n28dZ+3L6o6B5gDTquhIOIMWlVtPZu2V3L2YQM687DGQwMGDKCgoIDi4mK/Qwl5CQkJDBhg/203KV4Ni26H1a9DSl/4wZ9g/MVh3RqoswVTWXyVu/qIiLwJpKnqF0Ec+0hguoicilPJnIbzhJAhIjHuU8EAYHP7Qm+/tUUVqGLzFIeR2NjYph68xgRlxxZ453fw2d8hNhmOuxUmX9VtO4V1xP46lB22v7+p6rL9HVhVfw783H3/VOB/VfUiEXkep95hHjADWNCOuDukqcWQNR01pvupLnP6Anz0F2gIwMQr4OibINnGY2rJ/p4I/uguE4Ac4HOcop1xwCdAe6vXbwbmichvgM/wYW6D/MJy4qKjOLBnUlef2hjjlUANLH0C3vsDVJXA2HOdTmE97UmyNfvrUHYsgIjMA2ap6nJ3ewzwv205iaq+A7zjrq8HfO2vnV9YzpCsZGKircWQMWGvsgQ+fRKWPA7lm51B4U64A/qP9zuysBFMbcnIxiQAoKpfikhY/wvnF1aQM6iH32EYYzqi8Cv45BH44jkIVMPgY+D0B2Fo2I+H2eWCSQQrReRxnIlpFLiY3TuIhZWKmgDfllZxYZ/Ob5JqjPFYQ73TC/iTR+Dr95wxgcadB5OuhD4H+x1d2AomEVwK/ASnhzDAe8DDLb89tK1pmozGWgwZEzaqy5zWP0vmOMNCpA2AE26Hw2ZAUreYMNFXwTQfrQbuc19hb01hBWBjDBkTFr5bDsuehrxnobYCDpjsJICRP7B+AJ1of81Hn1PVH4rIcvbR6UtVx3kamUdWF5aTEBvFAdZiyJjQtGMLLH8ePp8HRV9BVCyMORsmXwn9D/U7um5pfym1sSjotK4IpKvkF5YztHcK0VE2TIExIaN2J6z6l3PxX/82aANk58Cp9zhJwIp/PLW/5qNb3OU3XReO99YUVnDEQdaxxBjfNTTAhvfhi/mwYoFT9JM+EI660akAzhzmd4QRY39FQ+XsexwgAVRV0zyLyiNlVXV8t6PaehQb4wdVKNsEBUth01JY+U/YUQBxqTD6TDjkfBh4BERZ/56utr8ngm53tVzTNBmNtRgyxnM1FbD5M+fCX5DrLHcWOX+LSYBBR8GJd8DI70Nsor+xRrigq91FpDe7z1C20ZOIPJTvthga1rvb5TjTVvV1UPI1bP/aKZKoq4a6SqdjUl3Vrleg2Xp0LCT2dMqrE3tCYo9d60k9nGVCOkTiMNlV22HrWiheBd/mOhf+ohVOWT9Az4PgoONgQI7z6jMmosf/DzWtJgIRmY4z7lB/nLkIDsTpUDba29A6X35hOclx0WRn2N1HxKjdCVvXwNZ8ZzjirauhOB9K1kNDXcufi4p17lJjE52719hEJ3lUlUBVKS2Mng4SDYkZzmiXse7nYhJ3HWu37QSIT9sjoTRbxiaFVlIJ1DjJc9ta2LbGufA3rldu2/W+hHTIPty50x8wwVm3yt6QFswTwa+BycAiVT1URI4FLvA2LG/kF5YztE8qUdZiqPvauha+fNGZfGRrvlMm3UiinQHIMkfAyFOdZa+hEJ+694V6f23UG+qdDk6VJU5i2HNZtX33p4q6SuepY2fx3k8ageqWzxMdv/vTR2PyiGlcJuxKUs2X0bGAOElEonatN+1z16FZfDvd9Z1QW+nsq6vctb7jW2di98Y7fIDk3k6F7sjvQ69hzr9l5nDoOcTK+cNMMImgTlW3iUiUiESp6tsicrfnkXkgv7CCY0dk+R2G6Ww7tsBXLzltzzd/BohT9HDAJDjsf5yLU9YIp3giJq7j54uKdi7QnXGXG6hxEkdLSaVy+66nkMqtThFWoGr3ZX1Nx+No1Pi0EpfsPJE0rvc/FMb+0L3YD3WWCTbDX3cRTCIoFZEUnKElnhGRIiDQymdCzvadtWytqLEexd1FVanT6mT5c87k4yj0Gw/TfgtjzoK0/n5HGJyYeEjt67zaq6HBSQaNTxj1dYA6rXQal83X0V139rGJbjFWonPhtzv5iBRMIjgdqAJmAxcB6cCdXgblBZuMphuoq4L8fzt3/mvegvpapxjimJth7DmR2+48KgqiEq3ljWm3YBLBLOB5VS0A5gZ7YBFJwHmKiHfP84Kq3iYig3FmJ+sJLAMuUdXaNkfeRvnWdDR8qULuE7DoTqgpg5Q+MOFy5+Lf/7DQqlA1JgwFkwjSgH+LSAnOBfwFVS0M4nM1wHGqWiEiscAHIvIGcANwn6rOE5FHgMvogtFM8wsrSI2PoW9aQutvNqGjrAAWXOMMOzBkKkyZ7bQ/j4r2OzJjuo1WCwRV9Q5VHQ1cjdOE9F0RWRTE51RVK9zNWPelwHHAC+7+ucAZ7Qm8rfILyxneNxWxu8fwoOqMOPmX78GmJfD9e+GSV5xkYEnAmE7VlnFci4DvgG1A72A+ICLRwKfAUOAhYB1QqqqNlc0FQHYbYmgXVSW/sJyTx3SgQs50nYoi+Of1sPpfMPB7cMZfnLoAY4wngulQ9hPgPCAL507+x6q6IpiDq2o9MF5EMoCXgVH7elsL552FUz/BwIEdm01sa0Ut2yvrrEdxOFixAF6b7QxPMO03MPkqewIwxmPBPBEcCFyvqnntPYmqlorIOzgd0zJEJMZ9KhgAbG7hM3OAOQA5OTktdOMMTuMYQyOsxVDoqtoOr9/ktAjqNx7OfBR6j/Q7KmMiQjB1BLe0JwmISJb7JICIJAIn4AxN8TZwjvu2GcCCth67rRpbDA2zFkOhac1Cpy7gq5dh6i/g8kWWBIzpQl7O9dYPmOvWE0QBz6nqayKyApgnIr8BPgOe8DAGAFYXVpCRFEtWSrzXpzJtUVMBb/0ffPoUZI2CC+ZB//F+R2VMxPEsEajqF8Be88qp6npgolfn3Zc1heUM720thkJKQS689GNnELMjroXjbnV62Rpjuly370/e2GJoeF8rFgoJ9QF45254YpozFMLM12Dary0JGOMjL4uGQkLhjhp2VAdsjKFQsG0dvHyFM0HJuPPg1D/YwGXGhIBunwiaKoq7U9NRVWcoZK3fY9mwa1sbnCGJEzL8H0hMFZY9DW/+3Bne+ewnnOEhjDEhIWISQdiPMVSyHv77Z2ei77rK4D8n0ZCcCclZzZZ7rmdBWrYzhk9nJ42dW+HVa53OYYOPhjMehvQBnXsOY0yHREQiyEyJo1e4thja8gX8936naWVUDIw9F9IPcDpZSZS7jN5j6e6vq3ImQ9lZ7FyQd26F7bnOsrZ873NFxUJ6tnP89AHNXge4r2xnbPpg5b8FC66G6lJneOjJV/n/dGKM2UsEJIKK8CsWUoVvPoQP7oO1CyEuFY74qXMh7ci49c3VVbnJodgZ0mFHgTPAW1kBlG5yxvgv37z7jFQAcSnOjF6Ny+av5vtK1sFnf4feB8MlL0PfMZ0TtzGm03XrRKCqrC2q4JzDw6QooqEB1vzbSQCbPoGkTDjul86Qy4kZnXuu2ETIOMB5taQ+AOVbnOkeywqcqQorS6BmhzP1Yk2589q51VnWutsNAUBg8tVw/K+caRWNMSGrWyeCzWXVVNQEQr9HcX0dfPmSkwCKV0L6QDj1Hhh/EcQl+RdXdEzryWJPqs4sWQ31EB/i/+7GGKCbJ4L87xorikO4aChQC387A775r9O79sw5zlSL0bF+R9Y+IjZTljFhpnsngsYWQ6FcR/DGTU4SOO1+OGyGVaYaY7pcN08EFfRJiyc9KUTvrpc+4YyzM2U25FzqdzTGmAjVrW8/UxNiOOKgTL/D2LdvPoQ3fgZDT3QqhI0xxifd+ong9umj/Q5h30o3wfxLoMcgOPtxm3jFGOOrbv1EEJJqK2H+RVBfC+f/o/ObhRpjTBt16yeCkKMK/7zW6S18wTzIGu53RMYY490TgYgcICJvi8hKEflKRK5z9/cUkYUissZd9vAqhpDz4QPOVIzH3QojTvY7GmOMAbwtGgoAN6rqKJy5iq8WkYOBW4DFqjoMWOxud39rF8Gi2+DgM+CoG/2OxhhjmniWCFR1i6ouc9fLceYrzgZOB+a6b5sLnOFVDCFj2zp44UfOuDtn/MXpdGWMMSGiSyqLRWQQzrSVnwB9VHULOMkC6N0VMfimegf84wJnZNDzn2nb6J3GGNMFPE8EIpICvAhcr6o72vC5WSKSKyK5xcXF3gXopYYGZ0aubWvhh3Od5qLGGBNiPE0EIhKLkwSeUdWX3N2FItLP/Xs/oGhfn1XVOaqao6o5WVlZXobpnXfvgtWvw0m/cyZlMcaYEORlqyEBngBWquq9zf70KjDDXZ8BLPAqBl/lvwXv3g3jL4ZJV/gdjTHGtMjLfgRHApcAy0Ukz933C+Au4DkRuQzYCJzrYQz+qCiCBVdBnzHw/T9a5bAxJqR5lghU9QOgpSvg8V6d13eq8MpVzgQtM/5pk7IYY0Ke9SzubJ886kwveeo90HuU39EYY0yrbKyhzlT4FSz8FQw/2Zle0hhjwoAlgs5SVwUvXAYJ6TD9QasXMMaEDSsa6iwLf+XMN3zxi5ASps1djTERyZ4IOkP+v2HJHJh8FQw9we9ojDGmTSwRdFR5odNKqM8YOP42v6Mxxpg2s0TQEQ0NTn+B2go4+wlrKmqMCUtWR9ARSx51hpc+9R7oPdLvaIwxpl3siaC9vvvSbSp6ijUVNcaENUsE7VFXBS9eBok94HRrKmqMCW9WNNQeb/0SilfBxS9Bcqbf0RhjTIfYE0FbrXodlj4Gk6+God13yCRjTOSwRBAsVWfy+ecugb5j4QRrKmqM6R6saCgYlSVOX4H8N2DUdJj+AMTE+x2VMcZ0CksErdm0xJl4vvw7OOUPMPHHVjlsjOlWLBG0RBU+ehAW3Q5p2XDZW5B9mN9RGWNMp/Nyqsq/ikiRiHzZbF9PEVkoImvcZQ+vzt8hlSXwjwvgrVthxClwxXuWBIwx3ZaXlcVPASfvse8WYLGqDgMWu9uhZdNSePRop8fwKX+AH/4NEjP8jsoYYzzj5VSV74nIoD12nw5MddfnAu8AN3sVw91L7mZVyargP7DjW9i+AVLjYcixsP0j+PdHXoVnjDH7NbLnSG6e6NklsklX1xH0UdUtAKq6RUR6t/RGEZkFzAIYOHCgN9E0BJxewnWVsHMrVG2HpF6QOQyirPrEGBMZQvZqp6pzgDkAOTk52p5jNGXSnVudnsDFq53XVndZvmXXm2OTnb4BE2dZqyBjTETp6kRQKCL93KeBfkCRp2d7bTasWACV23bti0uBzOEwZCpkjYCskc52j0EQFe1pOMYYE4q6OhG8CswA7nKXCzw9W/oAGHmac7HPGu4s07Ltjt8YY5rxLBGIyD9wKoYzRaQAuA0nATwnIpcBG4FzvTo/AEfd6OnhjTGmO/Cy1dAFLfzJRmozxpgQYoPOGWNMhLNEYIwxEc4SgTHGRDhLBMYYE+EsERhjTISzRGCMMRHOEoExxkQ4UW3XMD5dSkSKgW/a+fFMYGsnhhMKutt3su8T+rrbd+pu3wf2/Z0OVNWs1j4YFomgI0QkV1Vz/I6jM3W372TfJ/R1t+/U3b4PdOw7WdGQMcZEOEsExhgT4SIhEczxOwAPdLfvZN8n9HW379Tdvg904Dt1+zoCY4wx+xcJTwTGGGP2o1snAhE5WURWi8haEbnF73g6SkQ2iMhyEckTkVy/42kPEfmriBSJyJfN9vUUkYUissZd9vAzxrZo4fvcLiLfur9Tnoic6meMbSEiB4jI2yKyUkS+EpHr3P3h/Bu19J3C8ncSkQQRWSIin7vf5w53/2AR+cT9jeaLSFzQx+yuRUMiEg3kAycCBcBS4AJVXeFrYB0gIhuAHFUN2/bPInI0UAE8rapj3H2/B0pU9S43YfdQ1Zv9jDNYLXyf24EKVb3Hz9jaw51Ctp+qLhORVOBT4AxgJuH7G7X0nX5IGP5OIiJAsqpWiEgs8AFwHXAD8JKqzhORR4DPVfXhYI7ZnZ8IJgJrVXW9qtYC84DTfY4p4qnqe0DJHrtPB+a663Nx/icNCy18n7ClqltUdZm7Xg6sBLIJ79+ope8UltRR4W7Gui8FjgNecPe36TfqzokgG9jUbLuAMP7xXQq8JSKfisgsv4PpRH1UdQs4/9MCvX2OpzNcIyJfuEVHYVOM0pyIDAIOBT6hm/xGe3wnCNPfSUSiRSQPKAIWAuuAUlUNuG9p0/WuOyeCfc1QH+7lYEeq6mHAKcDVbrGECT0PAwcB44EtwB/9DaftRCQFeBG4XlV3+B1PZ9jHdwrb30lV61V1PDAAp/Rj1L7eFuzxunMiKAAOaLY9ANjsUyydQlU3u8si4GWc/wC6g0K3HLexPLfI53g6RFUL3f9RG4DHCLPfyS13fhF4RlVfcneH9W+0r+8U7r8TgKqWAu8Ak4EMEWmch75N17vunAiWAsPcmvQ44HzgVZ9jajcRSXYruhCRZGAa8OX+PxU2XgVmuOszgAU+xtJhjRdM15mE0e/kVkQ+AaxU1Xub/Slsf6OWvlO4/k4ikiUiGe56InACTr3H28A57tva9Bt121ZDAG5zsPuBaOCvqvpbn0NqNxEZgvMUABADPBuO30dE/gFMxRkpsRC4DXgFeA4YCGwEzlXVsKiAbeH7TMUpblBgA3BFY/l6qBORKcD7wHKgwd39C5wy9XD9jVr6ThcQhr+TiIzDqQyOxrmZf05V73SvEfOAnsBnwMWqWhPUMbtzIjDGGNO67lw0ZIwxJgiWCIwxJsJZIjDGmAhnicAYYyKcJQJjjIlwlgiM6WQiMlVEXvM7DmOCZYnAGGMinCUCE5FE5GJ3TPc8EXnUHbYcEakQkT+KyDIRWSwiWe7+8SLysTtA2cuNA5SJyFARWeSODb9MRA5yT5EiIi+IyCoRecbt3YqI3CUiK9zjhNXwx6b7skRgIo6IjALOwxnEbzxQD1zk/jkZWOYO7vcuTk9hgKeBm1V1HE4P1cb9zwAPqeohwBE4g5eBM8Ll9cDBwBDgSBHpiTOUwWj3OL/x7lsaEzxLBCYSHQ8cDix1h/I9HudiDc4QBPPd9b8DU0QkHchQ1Xfd/XOBo92xn7JV9WUAVa1W1Ur3PUtUtcAd0CwPGATsAKqBx0XkLKDxvcb4Kqb1txjT7QgwV1V/HsR79zcGy76GOm/UfIyXeiBGVQMiMhEn8ZwPXIMzmYgxvrInAhOJFgPniEhvaJqP90D3b1HsGsHxQuADVS0DtovIUe7+S4B33THtC0TkDPc48SKS1NJJ3fHw01X1dZxio/Gd/cWMaQ97IjARR1VXiMitOLO9RQF1wNXAN8BOYLSIfAqU4dQlgDOs7yPuhX49cKm7/xLgURG50z3Oufs5dSqwQEQScJ4mZnfuNzOmfWz0UWOaEZEKVU3xOw5jupIVDRljTISzJwJjjIlw9kRgjDERzhKBMcZEOEsExhgT4SwRGGNMhLNEYIwxEc4SgTHGRLj/D1fcSg8yzK5QAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the accuracy\n",
    "\n",
    "plt.plot(training_results_He['validation_accuracy'], label='He')\n",
    "plt.plot(training_results['validation_accuracy'], label='Default')\n",
    "plt.plot(training_results_Uniform['validation_accuracy'], label='Uniform') \n",
    "plt.ylabel('validation accuracy')\n",
    "plt.xlabel('epochs ')   \n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://dataplatform.cloud.ibm.com/registration/stepone?context=cpdaas&apps=data_science_experience,watson_machine_learning\"><img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-DL0110EN-SkillsNetwork/Template/module%201/images/Watson_Studio.png\"/></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--Empty Space for separating topics-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>About the Authors:</h2> \n",
    "\n",
    "<a href=\"https://www.linkedin.com/in/joseph-s-50398b136/\">Joseph Santarcangelo</a> has a PhD in Electrical Engineering, his research focused on using machine learning, signal processing, and computer vision to determine how videos impact human cognition. Joseph has been working for IBM since he completed his PhD. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other contributors: <a href=\"https://www.linkedin.com/in/michelleccarey/\">Michelle Carey</a>, <a href=\"www.linkedin.com/in/jiahui-mavis-zhou-a4537814a\">Mavis Zhou</a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Change Log\n",
    "\n",
    "| Date (YYYY-MM-DD) | Version | Changed By | Change Description                                          |\n",
    "| ----------------- | ------- | ---------- | ----------------------------------------------------------- |\n",
    "| 2020-09-23        | 2.0     | Srishti    | Migrated Lab to Markdown and added to course repo in GitLab |\n",
    "\n",
    "<hr>\n",
    "\n",
    "## <h3 align=\"center\"> © IBM Corporation 2020. All rights reserved. <h3/>\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python",
   "language": "python",
   "name": "conda-env-python-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}