9.4.3CNN_Small_Image_batch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center>\n",
    "    <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-DL0110EN-SkillsNetwork/Template/module%201/images/IDSNlogo.png\" width=\"300\" alt=\"cognitiveclass.ai logo\"  />\n",
    "</center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>Convolutional Neural Network with Batch-Normalization </h1> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Objective for this Notebook<h3>    \n",
    "<h5> 1. Learn how to compare a Convolutional Neural Network using Batch Normalization with a regular Convolutional Neural Network  to classify handwritten digits from the MNIST database..</h5>     \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Table of Contents</h2>\n",
    "<b>This lab takes a long time to run so the results are given. You can run the notebook your self but it may take a long time.</b>\n",
    "<p>In this lab, we will compare a Convolutional Neural Network using Batch Normalization with a regular Convolutional Neural Network  to classify handwritten digits from the MNIST database. We will reshape the images to make them faster to process. </p>\n",
    "\n",
    "<ul>\n",
    "    <li><a href=\"#read_me\">Read me Batch Norm for Convolution Operation  </a></li>\n",
    "<li><a href=\"#Makeup_Data\">Get Some Data</a></li>\n",
    "<li><a href=\"#CNN\">Two Types of Convolutional Neural Network</a></li>\n",
    "<li><a href=\"#Train\">Define Criterion function, Optimizer and Train the Model</a></li>\n",
    "<li><a href=\"#Result\">Analyze Results</a></li>\n",
    "</ul>\n",
    "<p>Estimated Time Needed: <strong>25 min</strong></p>\n",
    "\n",
    "<hr>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 id=\"read_me\">Read me Batch Norm for Convolution Operation  </h2>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like a fully connected network, we create a <code>BatchNorm2d</code> object, but we apply it to the 2D convolution object. First, we create objects <code>Conv2d</code> object; we require the number of output channels, specified by the variable <code>OUT</code>.  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<code>self.cnn1 = nn.Conv2d(in_channels=1, out_channels=OUT, kernel_size=5, padding=2) </code>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then create a Batch Norm  object for 2D convolution as follows:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<code>self.conv1_bn = nn.BatchNorm2d(OUT)</code>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The parameter out is the number of channels in the output. We can then apply batch norm  after  the convolution operation :\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<code>x = self.cnn1(x)</code>\n",
    "\n",
    "<p></p>\n",
    " <code> x=self.conv1_bn(x)</code>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--Empty Space for separating topics-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Preparation</h2>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# Import the libraries we need to use in this lab\n",
    "\n",
    "# Using the following line code to install the torchvision library\n",
    "# !conda install -y torchvision\n",
    "\n",
    "import torch \n",
    "import torch.nn as nn\n",
    "import torchvision.transforms as transforms\n",
    "import torchvision.datasets as dsets\n",
    "import matplotlib.pylab as plt\n",
    "import numpy as np\n",
    "def show_data(data_sample):\n",
    "    plt.imshow(data_sample[0].numpy().reshape(IMAGE_SIZE, IMAGE_SIZE), cmap='gray')\n",
    "    plt.title('y = '+ str(data_sample[1].item()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 id=\"Makeup_Data\">Get the Data</h2> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we create a transform to resize the image and convert it to a tensor :\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "IMAGE_SIZE = 16\n",
    "\n",
    "composed = transforms.Compose([transforms.Resize((IMAGE_SIZE, IMAGE_SIZE)), transforms.ToTensor()])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the training dataset by setting the parameters <code>train </code> to <code>True</code>. We use the transform defined above.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "train_dataset = dsets.MNIST(root='./data', train=True, download=True, transform=composed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the testing dataset by setting the parameters train  <code>False</code>.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make the validating \n",
    "\n",
    "validation_dataset = dsets.MNIST(root='./data', train=False, download=True, transform=composed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see the data type is long.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'torch.LongTensor'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Show the data type for each element in dataset\n",
    "\n",
    "train_dataset[0][1].type()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each element in the rectangular tensor corresponds to a number representing a pixel intensity as demonstrated by the following image.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/DL0110EN/notebook_images%20/chapter%206/6.2.1imagenet.png\" width=\"550\" alt=\"MNIST data image\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print out the fourth label \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(1)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The label for the fourth data element\n",
    "\n",
    "train_dataset[3][1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the fourth sample \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAEICAYAAACQ6CLfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAD3xJREFUeJzt3X+wVOV9x/H3hx9qMTZgRTRCowbHmVZTZBgxiaOZUi1SRtKZOIWpFoMzTKa11bapQZlpnPafpmlpWlrjoNLSBDFTf0TGhkbGJOPUqUTk5yXXCFIjKAEsrahpBy9++8ceMsu6e7l7ft299/m8Znb2x3mefb6cvR/O2bN79lFEYGbpGTPcBZjZ8HD4zRLl8JslyuE3S5TDb5Yoh98sUQ6/WaIcfitM0p9L2ilpQNK9w12PDY3Db2XYA9wF/OtwF2JD5/CPYpL+RNJjLY+tlPTVMseJiDURsQF4u8zntWo5/KPbN4C5kiYCSBoH/Bbw9XaNJT0l6X86XJ6qsW6rwbjhLsCqExEHJD0L3AQ8AMwF3oyIFzu0n19nfTa8vOUf/dYAN2e3b6bDVt/S4/CPft8CPi7pMmA+sLZTQ0kbJL3T4bKhtoqtFt7tH+Ui4v8kPQo8DPwgIl4bpO0NecaQNB4YS2NjMk7SGcB7EXE8z/NZPbzlT8Ma4HKq2+V/APhfYBGwPLt9S0VjWUnkH/MY/ST9IvAScF5EHB3ueqw3eMs/ykkaA/wR8IiDb838nn8Uk3QmcBD4MY2P+cx+xrv9Zonybr9Zomrd7Zfk3QyzikWEhtLOW36zRDn8Zoly+M0SVSj8kuZK+pGkPZKWlVWUmVUv90d9ksYCLwPXAfuBF4BFEfHDQfr4gJ9Zxeo44HclsCci9kbEMeARYEGB5zOzGhUJ/wXAvqb7+7PHTiJpqaTNkjYXGMvMSlbkc/52uxYf2K2PiFXAKvBuv1kvKbLl3w9Ma7o/FXijWDlmVpci4X8BuETSRZJOAxYC68spy8yqlnu3PyIGJN0OfIfGr7isjohdpVVmZpWq9aw+v+c3q56/229mg3L4zRLl8JslyuE3S5TDb5Yoh98sUQ6/WaIcfrNEOfxmiXL4zRLl8JslyuE3S5TDb5Yoh98sUZ6l13rGlClTauu3d+/eXGO98847ufr1Im/5zRLl8JslyuE3S1Tu8EuaJul7kvol7ZJ0R5mFmVm1ihzwGwD+OCK2SDoLeFHSxsGm6zKz3pF7yx8RByJiS3b7baCfNjP2mFlvKuWjPkkXAlcAm9osWwosLWMcMytP4fBL+hDwGHBnRBxtXe7pusx6U6Gj/ZLG0wj+2oh4vJySzKwORY72C3gI6I+IFeWVZGZ1KLLl/xRwC/CrkrZll3kl1WVmFSsyV9+/036abjMbAfwNP7NEeaJOq8SkSZO67nPfffflGmv27Nld97nllltyjfXcc8/l6lcnT9RpZoNy+M0S5fCbJcrhN0uUw2+WKIffLFEOv1miHH6zRDn8Zoly+M0S5fCbJcrhN0uUp+uyQY0Zk2/7MHPmzK77XH311bnG2rp1a9d9Dh8+nGus0cRbfrNEOfxmiXL4zRJVOPySxkraKumpMgoys3qUseW/g8ZsPWY2ghT93f6pwG8AD5ZTjpnVpeiW/6vAXcD7JdRiZjUqMmnHfOBQRLx4inZLJW2WtDnvWGZWvqKTdtwo6VXgERqTd3yjtVFErIqIWRExq8BYZlayIlN03x0RUyPiQmAh8N2IuLm0ysysUv6c3yxRpXy3PyK+D3y/jOcys3p4y2+WKE/XZYOaPn16rn6rV6/uus/kyZNzjbVo0aKu++zYsSPXWO+/3/ufanu6LjMblMNvliiH3yxRDr9Zohx+s0Q5/GaJcvjNEuXwmyXK4TdLlMNvliiH3yxRDr9Zohx+s0R5rr5ETJgwIVe/m266KVe/GTNmdN3nnnvuyTVWX19f131Gwtl5VfOW3yxRDr9Zohx+s0QVnbFnoqRHJb0kqV/SJ8oqzMyqVfSA398C/xYRn5V0GpDvqJKZ1S53+CX9PHANcCtARBwDjpVTlplVrchu/8XAYeAfsym6H5R0ZmsjT9dl1puKhH8cMBP4WkRcAbwLLGtt5Om6zHpTkfDvB/ZHxKbs/qM0/jMwsxGgyFx9PwH2Sbo0e2gO8MNSqjKzyhU92v/7wNrsSP9e4HPFSzKzOhQKf0RsA/xe3mwE8ok9I9CYMd2/W7v22mtzjbVkyZJc/TZs2NB1n3Xr1uUaa2BgIFe/1PnrvWaJcvjNEuXwmyXK4TdLlMNvliiH3yxRDr9Zohx+s0Q5/GaJcvjNEuXwmyXK4TdLlMNvliif1TcCTZ8+ves+d999d66xIiJXvxUrVnTd58iRI7nGsny85TdLlMNvliiH3yxRRafr+kNJuyT1SVon6YyyCjOzauUOv6QLgD8AZkXEZcBYYGFZhZlZtYru9o8Dfk7SOBrz9L1RvCQzq0OR3+1/Hfgr4DXgAPBWRDzd2s7TdZn1piK7/ZOABcBFwEeAMyXd3NrO03WZ9aYiu/2/BvxnRByOiPeAx4FPllOWmVWtSPhfA66SNEGSaEzX1V9OWWZWtSLv+TfRmJxzC7Aze65VJdVlZhUrOl3Xl4AvlVSLmdXI3/AzS5TP6htG48blW/233npr131mz56da6yHH344V7++vr6u++Q9g9Dy8ZbfLFEOv1miHH6zRDn8Zoly+M0S5fCbJcrhN0uUw2+WKIffLFEOv1miHH6zRDn8ZonyiT3DaMaMGbn63XbbbV332b59e66xVq5cmavfu+++m6uf1cdbfrNEOfxmiXL4zRJ1yvBLWi3pkKS+psfOlrRR0u7selK1ZZpZ2Yay5f8nYG7LY8uAZyLiEuCZ7L6ZjSCnDH9EPAscaXl4AbAmu70G+EzJdZlZxfJ+1DclIg4ARMQBSed2aihpKbA05zhmVpHKP+ePiFVkv+cvyb/QaNYj8h7tPyjpfIDs+lB5JZlZHfKGfz2wOLu9GHiynHLMrC5D+ahvHfAfwKWS9ku6DfgL4DpJu4HrsvtmNoKc8j1/RCzqsGhOybWYWY38DT+zRPmsvhKcfvrpufrNnz8/V79zz+34yWpH999/f66xdu3alauf9T5v+c0S5fCbJcrhN0uUw2+WKIffLFEOv1miHH6zRDn8Zoly+M0S5fCbJcrhN0uUw2+WKJ/YU4LJkyfn6pf3xJ7du3d33efJJ/P93sqxY8dy9bPe5y2/WaIcfrNEOfxmico7XddXJL0kaYekJyRNrLZMMytb3um6NgKXRcTHgZeBu0uuy8wqlmu6roh4OiIGsrvPA1MrqM3MKlTGe/4lwIZOCyUtlbRZ0uYSxjKzkhT6nF/ScmAAWNupjafrMutNucMvaTEwH5gTEQ612QiTK/yS5gJfBK6NiJ+WW5KZ1SHvdF1/D5wFbJS0TVK+H4U3s2GTd7quhyqoxcxq5G/4mSXKZ/WVQFKufmPHjs3Vb9++fV336e/vzzWWj+WOXt7ymyXK4TdLlMNvliiH3yxRDr9Zohx+s0Q5/GaJcvjNEuXwmyXK4TdLlMNvliiH3yxRDr9ZolTnWVuj9Tf8xo8fn6vf5ZdfnqvfmDHd/5+9devWXGMdP348Vz8bPhExpNNMveU3S5TDb5aoXNN1NS37gqSQdE415ZlZVfJO14WkacB1wGsl12RmNcg1XVfmb4C7gFF5EM9stMv7u/03Aq9HxPZT/X6dpKXA0jzjmFl1ug6/pAnAcuD6obT3dF1mvSnP0f6PARcB2yW9SmOG3i2SziuzMDOrVtdb/ojYCZx74n72H8CsiHizxLrMrGJ5p+sysxEu73RdzcsvLK0aM6uNv+Fnliif2GM2yvjEHjMblMNvliiH3yxRDr9Zohx+s0Q5/GaJcvjNEuXwmyXK4TdLlMNvliiH3yxRDr9Zohx+s0Tl+gHPAt4Eftxh2TnZ8uHmOk7mOk7W63V8dKhPUOspvYORtDkiZrkO1+E66qnDu/1miXL4zRLVS+FfNdwFZFzHyVzHyUZNHT3znt/M6tVLW34zq5HDb5aoWsMvaa6kH0naI2lZm+WnS/pmtnyTpAsrqGGapO9J6pe0S9Idbdp8WtJbkrZllz8tu46msV6VtDMbZ3Ob5ZL0d9k62SFpZsnjX9r079wm6aikO1vaVLY+JK2WdEhSX9NjZ0vaKGl3dj2pQ9/FWZvdkhZXUMdXJL2UrfcnJE3s0HfQ17CEOu6V9HrT+p/Xoe+g+fqAiKjlAowFXgEuBk4DtgO/1NLmd4H7s9sLgW9WUMf5wMzs9lnAy23q+DTwVE3r5VXgnEGWzwM2AAKuAjZV/Br9BPhoXesDuAaYCfQ1PfaXwLLs9jLgy236nQ3sza4nZbcnlVzH9cC47PaX29UxlNewhDruBb4whNdu0Hy1Xurc8l8J7ImIvRFxDHgEWNDSZgGwJrv9KDBHp5oDvEsRcSAitmS33wb6gQvKHKNkC4B/jobngYmSzq9orDnAKxHR6VuYpYuIZ4EjLQ83/x2sAT7TpuuvAxsj4khE/DewEZhbZh0R8XREDGR3n6cxKW2lOqyPoRhKvk5SZ/gvAPY13d/PB0P3szbZSn8L+IWqCsreVlwBbGqz+BOStkvaIOmXq6oBCOBpSS9KWtpm+VDWW1kWAus6LKtrfQBMiYgD0PjPmqaJYZvUuV4AltDYA2vnVK9hGW7P3n6s7vA2qOv1UWf4223BWz9nHEqbUkj6EPAYcGdEHG1ZvIXGru+vACuBb1VRQ+ZTETETuAH4PUnXtJbapk/p60TSacCNwL+0WVzn+hiqOv9WlgMDwNoOTU71Ghb1NeBjwAzgAPDX7cps89ig66PO8O8HpjXdnwq80amNpHHAh8m3CzQoSeNpBH9tRDzeujwijkbEO9ntbwPjJZ1Tdh3Z87+RXR8CnqCx+9ZsKOutDDcAWyLiYJsaa1sfmYMn3tpk14fatKllvWQHEucDvx3Zm+tWQ3gNC4mIgxFxPCLeBx7o8Pxdr486w/8CcImki7KtzEJgfUub9cCJo7afBb7baYXnlR1DeAjoj4gVHdqcd+JYg6Qraayn/yqzjuy5z5R01onbNA4w9bU0Ww/8TnbU/yrgrRO7xCVbRIdd/rrWR5Pmv4PFwJNt2nwHuF7SpGw3+PrssdJImgt8EbgxIn7aoc1QXsOidTQf4/nNDs8/lHydrIwjlF0cyZxH4+j6K8Dy7LE/o7FyAc6gsdu5B/gBcHEFNVxNY3doB7Atu8wDPg98PmtzO7CLxhHT54FPVrQ+Ls7G2J6Nd2KdNNci4B+ydbYTmFVBHRNohPnDTY/Vsj5o/IdzAHiPxtbrNhrHeZ4BdmfXZ2dtZwEPNvVdkv2t7AE+V0Ede2i8jz7xd3Lik6iPAN8e7DUsuY6vZ6/9DhqBPr+1jk75Guzir/eaJcrf8DNLlMNvliiH3yxRDr9Zohx+s0Q5/GaJcvjNEvX/in9jkZH6/F8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# The image for the fourth data element\n",
    "show_data(train_dataset[3])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The fourth sample is a \"1\".\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--Empty Space for separating topics-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 id=\"CNN\">Build a Two Convolutional Neural Network Class</h2>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Build a Convolutional Network class with two Convolutional layers and one fully connected layer. Pre-determine the size of the final output matrix. The parameters in the constructor are the number of output channels for the first and second layer.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CNN(nn.Module):\n",
    "    \n",
    "    # Contructor\n",
    "    def __init__(self, out_1=16, out_2=32):\n",
    "        super(CNN, self).__init__()\n",
    "        self.cnn1 = nn.Conv2d(in_channels=1, out_channels=out_1, kernel_size=5, padding=2)\n",
    "        self.maxpool1=nn.MaxPool2d(kernel_size=2)\n",
    "\n",
    "        self.cnn2 = nn.Conv2d(in_channels=out_1, out_channels=out_2, kernel_size=5, stride=1, padding=2)\n",
    "        self.maxpool2=nn.MaxPool2d(kernel_size=2)\n",
    "        self.fc1 = nn.Linear(out_2 * 4 * 4, 10)\n",
    "    \n",
    "    # Prediction\n",
    "    def forward(self, x):\n",
    "        x = self.cnn1(x)\n",
    "        x = torch.relu(x)\n",
    "        x = self.maxpool1(x)\n",
    "        x = self.cnn2(x)\n",
    "        x = torch.relu(x)\n",
    "        x = self.maxpool2(x)\n",
    "        x = x.view(x.size(0), -1)\n",
    "        x = self.fc1(x)\n",
    "        return x\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Build a Convolutional Network class with two Convolutional layers and one fully connected layer. But we add Batch Norm for the convolutional layers. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CNN_batch(nn.Module):\n",
    "    \n",
    "    # Contructor\n",
    "    def __init__(self, out_1=16, out_2=32,number_of_classes=10):\n",
    "        super(CNN_batch, self).__init__()\n",
    "        self.cnn1 = nn.Conv2d(in_channels=1, out_channels=out_1, kernel_size=5, padding=2)\n",
    "        self.conv1_bn = nn.BatchNorm2d(out_1)\n",
    "\n",
    "        self.maxpool1=nn.MaxPool2d(kernel_size=2)\n",
    "        \n",
    "        self.cnn2 = nn.Conv2d(in_channels=out_1, out_channels=out_2, kernel_size=5, stride=1, padding=2)\n",
    "        self.conv2_bn = nn.BatchNorm2d(out_2)\n",
    "\n",
    "        self.maxpool2=nn.MaxPool2d(kernel_size=2)\n",
    "        self.fc1 = nn.Linear(out_2 * 4 * 4, number_of_classes)\n",
    "        self.bn_fc1 = nn.BatchNorm1d(10)\n",
    "    \n",
    "    # Prediction\n",
    "    def forward(self, x):\n",
    "        x = self.cnn1(x)\n",
    "        x=self.conv1_bn(x)\n",
    "        x = torch.relu(x)\n",
    "        x = self.maxpool1(x)\n",
    "        x = self.cnn2(x)\n",
    "        x=self.conv2_bn(x)\n",
    "        x = torch.relu(x)\n",
    "        x = self.maxpool2(x)\n",
    "        x = x.view(x.size(0), -1)\n",
    "        x = self.fc1(x)\n",
    "        x=self.bn_fc1(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Function to train the model \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_model(model,train_loader,validation_loader,optimizer,n_epochs=4):\n",
    "    \n",
    "    #global variable \n",
    "    N_test=len(validation_dataset)\n",
    "    accuracy_list=[]\n",
    "    loss_list=[]\n",
    "    for epoch in range(n_epochs):\n",
    "        for x, y in train_loader:\n",
    "            model.train()\n",
    "            optimizer.zero_grad()\n",
    "            z = model(x)\n",
    "            loss = criterion(z, y)\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "            loss_list.append(loss.data)\n",
    "\n",
    "        correct=0\n",
    "        #perform a prediction on the validation  data  \n",
    "        for x_test, y_test in validation_loader:\n",
    "            model.eval()\n",
    "            z = model(x_test)\n",
    "            _, yhat = torch.max(z.data, 1)\n",
    "            correct += (yhat == y_test).sum().item()\n",
    "        accuracy = correct / N_test\n",
    "        accuracy_list.append(accuracy)\n",
    "     \n",
    "    return accuracy_list, loss_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 id=\"Train\">Define the Convolutional Neural Network Classifier, Criterion function, Optimizer and Train the Model</h2> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are 16 output channels for the first layer, and 32 output channels for the second layer \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the model object using CNN class\n",
    "model = CNN(out_1=16, out_2=32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the loss function, the optimizer and the dataset loader \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "criterion = nn.CrossEntropyLoss()\n",
    "learning_rate = 0.1\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr = learning_rate)\n",
    "train_loader = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=100)\n",
    "validation_loader = torch.utils.data.DataLoader(dataset=validation_dataset, batch_size=5000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train the model and determine validation accuracy technically test accuracy **(This may take a long time)**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# Train the model\n",
    "accuracy_list_normal, loss_list_normal=train_model(model=model,n_epochs=10,train_loader=train_loader,validation_loader=validation_loader,optimizer=optimizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Repeat the Process for the model with  batch norm \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "model_batch=CNN_batch(out_1=16, out_2=32)\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "learning_rate = 0.1\n",
    "optimizer = torch.optim.SGD(model_batch.parameters(), lr = learning_rate)\n",
    "accuracy_list_batch, loss_list_batch=train_model(model=model_batch,n_epochs=10,train_loader=train_loader,validation_loader=validation_loader,optimizer=optimizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--Empty Space for separating topics-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 id=\"Result\">Analyze Results</h2> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the loss with both networks.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x12a05da20>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEWCAYAAACdaNcBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XeYFFXWwOHfmeCMCoqSRBEGEFGJIrCoK2DCuJhdMYFxxbTuqnyY0NVds+AakV3DGnARRGURxYiAkRHJcUgy5AHJacL5/qjqme6eTjPTPd3VnPd56umuqlt1753pPn3rVtUtUVWMMcakl4xkF8AYY0z8WXA3xpg0ZMHdGGPSkAV3Y4xJQxbcjTEmDVlwN8aYNGTB3ew1RGSZiJyW7HIYUxssuBtjTBqy4G6MMWnIgrvZ64hIjog8KyKr3OlZEclx1zUQkXEisklENorIZBHJcNf9n4isFJGtIrJARE5Nbk2MCS8r2QUwJgnuA7oDnQAFPgLuBx4A7gQKgYZu2u6Aikgb4Fagq6quEpE8ILN2i21M7KzlbvZGVwAPq+o6VV0P/A24yl1XDDQBmqtqsapOVmcAplIgBzhGRLJVdZmqLk5K6Y2JgQV3szc6FFjuN7/cXQbwFFAAfCYiS0RkEICqFgB3AA8B60TkvyJyKMakKAvuZm+0CmjuN9/MXYaqblXVO1W1JfAH4K++vnVVHaGqv3e3VeCJ2i22MbGz4G72Ru8C94tIQxFpAAwG3gYQkXNF5AgREWALTndMqYi0EZFT3BOvu4Cd7jpjUpIFd7M3+juQD8wEZgHT3GUArYEvgG3A98BLqjoRp7/9caAIWAM0Au6t1VIbUwViD+swxpj0Yy13Y4xJQxbcjTEmDVlwN8aYNGTB3Rhj0lDShh9o0KCB5uXlJSt7Y4zxpJ9//rlIVRtGS5e04J6Xl0d+fn6ysjfGGE8SkeXRU1m3jDHGpCUL7sYYk4YsuBtjTBqy8dyNSQPFxcUUFhaya9euZBfFxElubi5NmzYlOzu7WttbcDcmDRQWFlK3bl3y8vJwxjwzXqaqbNiwgcLCQlq0aFGtfVi3jDFpYNeuXdSvX98Ce5oQEerXr1+jIzEL7sakCQvs6aWm/0/vBfc5c2DwYFi3LtklMcaYlOW94D53LjzyCKxfn+ySGGP81KlTJ9lFSIiJEydy7rnnJrsYVea94O5j49AbY6IoLd17H5blveBu/YrGpDRV5e6776Zdu3a0b9+ekSNHArB69Wp69OhBp06daNeuHZMnT6a0tJT+/fuXpx06dGil/fXv35/bb7+dE044gZYtWzJ69OiI+UycOJGTTz6Zyy+/nPbt27Ns2TKOOuoorr/+etq1a8cVV1zBF198wYknnkjr1q356aefAPjpp5844YQTOPbYYznhhBNYsGBBxHqWlpZy11130b59ezp06MDzzz8POEOrPPjgg3Tu3Jn27dszf/58AB566CGuvfZaevXqRcuWLXnuuefi8wcPwy6FNCbN3HEHTJ8e33126gTPPhtb2jFjxjB9+nRmzJhBUVERXbt2pUePHowYMYIzzjiD++67j9LSUnbs2MH06dNZuXIls2fPBmDTpk0h97l69WqmTJnC/Pnz6dOnDxdffHHYfMAJ1LNnz6ZFixYsW7aMgoICRo0axfDhw+natSsjRoxgypQpjB07lkcffZQPP/yQo446ikmTJpGVlcUXX3zBvffey/vvvx+2nsOHD2fp0qX88ssvZGVlsXHjxvJ1DRo0YNq0abz00ks8/fTT/Pvf/wZg/vz5fP3112zdupU2bdowYMCAal/HHo13g7t1yxiTkqZMmULfvn3JzMykcePG9OzZk6lTp9K1a1euvfZaiouLOf/88+nUqRMtW7ZkyZIl3HbbbZxzzjn07t075D7PP/98MjIyOOaYY1i7dm3EfA444AC6desWcH14ixYtaN++PQBt27bl1FNPRUTKW/YAmzdvpl+/fixatAgRobi4OGI9v/jiC2666SayspwwevDBB5evu/DCCwE47rjjGDNmTPnyc845h5ycHHJycmjUqBFr166ladOmVfwLx8Z7wd26ZYyJKNYWdqKEey5zjx49mDRpEh9//DFXXXUVd999N1dffTUzZsxgwoQJvPjii7z33nu89tprlbbNycmptP9Iz3/ef//9w26fkZFRPp+RkUFJSQkADzzwACeffDIffPABy5Yto1evXlHrGe5yRd/+MzMzy/cfXI7gdfHmvT53H2u5G5OSevTowciRIyktLWX9+vVMmjSJbt26sXz5cho1asQNN9zAddddx7Rp0ygqKqKsrIyLLrqIRx55hGnTptU4n+ravHkzhx12GABvvPFG1PS9e/dm2LBh5QHav1smFVjL3RgTVxdccAHff/89HTt2RER48sknOeSQQ/jPf/7DU089RXZ2NnXq1OHNN99k5cqVXHPNNZSVlQHw2GOP1Tgf3wnMqho4cCD9+vVjyJAhnHLKKVHTX3/99SxcuJAOHTqQnZ3NDTfcwK233lqtvBNBIh3aAIjI4cCbwCFAGTBcVf8ZlKYX8BGw1F00RlUfjrTfLl26aLUe1jFmDFx0kXPGqGPHqm9vTBqaN28eRx99dLKLYeIs1P9VRH5W1S7Rto2l5V4C3Kmq00SkLvCziHyuqnOD0k1W1cRf6W8td2OMiSpqn7uqrlbVae77rcA84LBEFywq63M3xpiwqnRCVUTygGOBH0OsPl5EZojIJyLSNsz2N4pIvojkr6/u8AHWcjfGmKhiDu4iUgd4H7hDVbcErZ4GNFfVjsDzwIeh9qGqw1W1i6p2adgw6sO7jTHGVFNMwV1EsnEC+zuqOiZ4vapuUdVt7vvxQLaINIhrSStnmtDdG2OMl0UN7uJcpf8qME9Vh4RJc4ibDhHp5u53QzwL6pdZQnZrjDHpJJaW+4nAVcApIjLdnc4WkZtE5CY3zcXAbBGZATwHXKbRrrGsKWu5G5NSEjXkb//+/csHC4vFsmXLGDFiRNR0eXl5FBUV1aRoKS3qpZCqOgWI2FxW1ReAF+JVqIis5W6MicAX3C+//PJkFyWpbPgBY0xcxXvIX3AG6TrppJM48sgjGTduHOAE8ZNOOonOnTvTuXNnvvvuOwAGDRrE5MmT6dSpE0OHDg07NC/A888/X2loXn+pPqxvJDb8gDHpJslj/iZiyN9ly5bxzTffsHjxYk4++WQKCgpo1KgRn3/+Obm5uSxatIi+ffuSn5/P448/ztNPP13+I/Dyyy9XeWhen1Qf1jcS77bcjTEpKdKQv6+//joPPfQQs2bNom7dugFD/n766acccMABIfd56aWXkpGRQevWrWnZsiXz58+nuLiYG264gfbt23PJJZcwd27wTfOOWIfm9Q39W9NtfcP6NmjQoHxY32TwXsvdx7pljAktyWP+JmLI3+ChdUWEoUOH0rhxY2bMmEFZWRm5ublhy1PVoXlrsm1tDusbifda7tYtY0xKS8SQv6NGjaKsrIzFixezZMkS2rRpw+bNm2nSpAkZGRm89dZb5c9LrVu3Llu3bi3ftiZD86b6sL6RWMvdGBNXiRjyt02bNvTs2ZO1a9cybNgwcnNzufnmm7nooosYNWoUJ598cvkDOjp06EBWVhYdO3akf//+3HbbbdUemjfVh/WNJOqQv4lS7SF/P/4Yzj0XfvoJunaNf8GM8SAb8jc91WTIX+91y/hYy90YY8LyXHDftdvpc0/SOQpjjPEEzwV3313IH31oLXdj/CWri9UkRk3/n54L7gcd7LTck3jjlzEpJzc3lw0bNliATxOqyoYNG8Je3hkLz10tc+aZwIuwa3eyS2JM6mjatCmFhYVU+yE4JuXk5ubStGnTam/vueCemem8CtZCMcYnOzubFi1aJLsYJoV4rlvGdxNTbk6UdMYYsxfzXnB37bevtdyNMSYc7wV3t+W+fbtd6m6MMeF4L7i7iouVMKODGmPMXs97wd1v4LCVK5NYDmOMSWHeC+5+Qgy/bIwxBg8Hd0F54IFkl8IYY1KT94K7X7dMl6jjohljzN7Je8HdJShBjzs0xhjj8l5w92u5d+yYxHIYY0wK815wdwnKjBnJLoUxxqQm7wV3e4aqMcZE5b3gbowxJirPBncbFdIYY8LzXnC3bhljjIkqanAXkcNF5GsRmScic0TkzyHSiIg8JyIFIjJTRDonprh+eVrL3RhjworlYR0lwJ2qOk1E6gI/i8jnqjrXL81ZQGt3+h3wsvsaf9ZyN8aYqKK23FV1tapOc99vBeYBhwUlOw94Ux0/APVEpEncS+tHUIvzxhgTRpUesyciecCxwI9Bqw4DVvjNF7rLVtegbOEKAUCPk2DDb3HfuzHGpIWYT6iKSB3gfeAOVd0SvDrEJpU6xUXkRhHJF5H8mj7Id59sZefOGu3CGGPSVkzBXUSycQL7O6o6JkSSQuBwv/mmwKrgRKo6XFW7qGqXhg0bVqe85S33ffdznsZkjDGmsliulhHgVWCeqg4Jk2wscLV71Ux3YLOqxr9Lxk9mJpSUJDIHY4zxrlj63E8ErgJmich0d9m9QDMAVR0GjAfOBgqAHcA18S9qoKwMteBujDFhRA3uqjqF0H3q/mkUuCVehYrI7ZaxlrsxxoTnvTtUXZmZ1nI3xphwvBfcfS33DGu5G2NMON4L7q4st+WuNgqBMcZU4r3g7tfnDlBWlsSyGGNMivJecHf5grt1zRhjTGXeDe4ZTn+MBXdjjKnMe8E9qFvGgrsxxlTmveDuspa7McaE573gbi13Y4yJynvB3ZWVaS13Y4wJx3vB3W25Z1jL3RhjwvJecHdlWZ+7McaE5b3gbn3uxhgTlfeCu8u6ZYwxJjzPBnfrljHGmPC8F9zdbpksdyT63buTWBZjjElR3gvurpx9nJZ7YWGSC2KMMSnIe8HdbbkvW+bMXpPwB/oZY4z3eC+4u3Zsd1ruW7YkuSDGGJOCvBfcJeLjXI0xxuDF4G6MMSYqzwZ3wZ6vZ4wx4XgvuPu6Zax3xhhjwvJecHeJPRnbGGPC8l5w940K6b2SG2NMrfFsiDzlZKfl3qBBkgtijDEpyHvB3W25Z2c7s0VFSSyLMcakKO8Fd5ddLWOMMeF5L7jbTUzGGBNV1OAuIq+JyDoRmR1mfS8R2Swi091pcPyLGSrf2sjFGGO8KSuGNG8ALwBvRkgzWVXPjUuJYmTdMsYYE17UlruqTgI21kJZYmNNdmOMiSpefe7Hi8gMEflERNqGSyQiN4pIvojkr1+/vmY52k1MxhgTVjyC+zSguap2BJ4HPgyXUFWHq2oXVe3SsGHD6uXmttzLyqq3uTHG7A1qHNxVdYuqbnPfjweyRSThtxaVlljL3RhjwqlxcBeRQ0Sc5rSIdHP3uaGm+42QIQB16zjBPTMzYTkZY4xnRb1aRkTeBXoBDUSkEHgQyAZQ1WHAxcAAESkBdgKXqSawQ9wdVCYzQ+ncGQ49NGE5GWOMZ0UN7qraN8r6F3AulawdvqtlysrIzITS0lrL2RhjPMO7d6iqkpFhwd0YY0LxXnD3jfWrai13Y4wJw3vB3bpljDEmKu8Gd7flbte7G2NMZd4L7tYtY4wxUXkvuPta7kuW2AlVY4wJw3vB3fdcPetzN8aYsGIZ8je11K3rvGZksGYNTJ+e3OIYY0wq8l7LPSPD6ZopKSkP7KtWJbdIxhiTarwX3AGysqCkpHzWumaMMSZQWgR3Y4wxgdIiuNvDmYwxJlBaBHdjjDGBLLgbY0wa8mZwX78exo8vn7VuGWOMCeTN4A6wfHmyS2CMMSnLu8Hdz5dfJrsExhiTWjwd3Fu3dl779UtuOYwxJtV4OrjffXeyS2CMManJ08E9w9OlN8aYxPHewGEAZ50F69fbVTLGGBOGN9u+2dlQXGzB3RhjwvB0cLduGWOMCc2b4dEN7tu3J7sgxhiTmrwb3EtK+OWXZBfEGGNSkzeDe1aWdcsYY0wE3gyPbreMMcaY0NImuG/ZkqSyGGNMCooa3EXkNRFZJyKzw6wXEXlORApEZKaIdI5/MYOECO5duyY8V2OM8YxYWu5vAGdGWH8W0NqdbgRernmxoggxnvvChQnP1RhjPCNqcFfVScDGCEnOA95Uxw9APRFpEq8ChmR97sYYE1E8+twPA1b4zRe6yyoRkRtFJF9E8tevX1/9HN2Wu5Zp9fdhjDFpLB7BPdQgACGjrqoOV9UuqtqlYcOG1c9x6FAADtk0v/r7MMaYNBaP4F4IHO433xRYFYf9hrdtGwD1ty5NaDbGGONV8QjuY4Gr3atmugObVXV1HPYbVUZZaW1kY4wxnhN1yF8ReRfoBTQQkULgQSAbQFWHAeOBs4ECYAdwTaIKW4lan7sxxoQSNbirat8o6xW4JW4lisWTT8LAgaw74IhazdYYY7zCm3eo5uUBUFpiLXdjjAnFm8HdHTGsc6eyJBfEGGNSkzeD+86dAJx5XA2ulTfGmDTmzeD++usA5D7+YJILYowxqcmbwT3LPQ9sQxAYY0xIng7uEjR4mDHGGIc3g3t2tvO6e1dyy2GMMSnKm8HdbblnzpuT5IIYY0xq8mZwl1BjlRljjPHxZnD/y1+SXQJjjElp3gzuhxyS7BIYY0xK82Zwr1Mn5OJt25wem7ffruXyGGNMivFmcG/UKOTiX391Xv/xj1osizHGpCBvBvco7HyrMWZvl1bB3ddyN8aYvV1aBfezznJereVujNnbpVVw95k7FwoLk10KY4xJHs8H9xxCD0Fw+OGwdWstF8YYY1KE54P7/mwPu27HjlosiDHGpBDPB/ccdoddZ8/PNsbsrTwf3BcNtjuWjDEmmOeD+/4LpiW7CMYYk3I8H9wZOTLZJTDGmJTj3eD+88+1nuW8ebDLng9ijPEA7wb3Aw6o1ey2bIFjjoF+/Wo1W2OMqRbvBvfMzFrNbudO53XixFrN1hhjqiWtg7vvUsjHHoPhwxNcngRaswZmzEh2KYwxXuLd4J5RUfRz+R/tmBU26b33wp/+FP8ilJXVTh98q1bQqVPi8zHGpI+YgruInCkiC0SkQEQGhVjfX0TWi8h0d7o+/kUN0rBh+dv/0YdZdEh4lsHuvBP23ReKixObj91pa4ypqqxoCUQkE3gROB0oBKaKyFhVnRuUdKSq3pqAMoaWk1NrWYXz6qvO686dkJ2d3LIYY4y/WFru3YACVV2iqnuA/wLnJbZY8ZHo4Qd83f4lJYnNxxhjqiqW4H4YsMJvvtBdFuwiEZkpIqNF5PC4lC7BZs+Gyy+vfnD2BfeysviVKZWsXeuMjT9pUrJLYoypqliCe6hHXwS3if8H5KlqB+AL4D8hdyRyo4jki0j++vXrq1bSBLjiCnj3XZgzp3rb+x4Kkq7BfcoU5/Wf/0xuOYwxVRdLcC8E/FviTYFV/glUdYOq+oZn/BdwXKgdqepwVe2iql0a+p0QTZTDDoPOnSsvHz/eudu0KmyESWOMl8QS3KcCrUWkhYjsA1wGjPVPICJN/Gb7AFUMnfFRh8pP5/jll4r3c91TwOec49xtWlP2OD9jTKqKGtxVtQS4FZiAE7TfU9U5IvKwiPRxk90uInNEZAZwO9A/UQWO5FnuiLi+bVvYtCn0uj17nCEGwrFAXrsKCqp+dGWMqRD1UkgAVR0PjA9aNtjv/T3APfEtWtUdzMaoaQ46KPTyc8+Fzz8P3/0SqVsmXbtsklmv1q2TXwZjvMy7d6gCtGkTMHsBHzKGC2Le3L8V//nnVc/e15q3AGSMSTXeDu4XXVRp0QV8GPPmv/7qvD7+ePS0obplfEH9qKNiztJTrCvKGO/ydnB/+OG47Oa//w29fNeuih+ASK3zzZvjUowq+eorGDQosTdQ2RGJMd7l7eAeZmTIswJPD1SZqhM8L7sMmjdPzevYTz0VnniiYgiE2nToodCnT/R0XlRaCu+9Zz9sxvu8HdwBrryy0qLxnENnqvekprFj4Z57nOD50UfOsrKyii6Kdesq0qZCt8X27Ynbd7j6rV4N//tf4vJNpqFD4Y9/hHfeSXZJjKmZmK6WSWn77htycSxXzoRyXohRc1RTqyVXW2Xx5bNkSe3kF42qM0Lm/vsnLo+VK53XFLiB2pga8X7LPUzzMoP49aWkWreMf3BP5NFDYaHzOn164vKoiiFDoE4dWLUqetrq2rPHeS0qSlwe4UyZUnGjnTE15f3g3qtXyMUTOJNM4nO2MVRwLysLvOmpsDDxY7D4gnpttdxLS2snn1i9957z6jvJnQhjxjivjz6auDzCOekk50Y7Y+LB+8G9b9+wq+qzIS5ZhArugwYFPoWpTx+4447EBp5Q5Ulkyz0Vzin4q43yhDtKW7Eisec3kqm0FMaNS62uR1Nz3g/uESjCPuwmg5o1QcvKAFXu5xFasxCAl14KTPPbb85rPC5NLC2F3r3hm28Cl9d2y31vFO4HpFkzOO202i1LbXn6afjDH+CDD2o/7yZNnBPYtW3Tpoojwdr2zDPw44+Jzyc9gnvPniEX12MTu8nlI87jSt7iFL6s1u7LyiBjw3oeYTCfczpQuRUX7W7VsjJ44IGK/uKFC2Fr5XHOAOdqlM8/d4Yk9ufbd2213FNVIn/cMiJ8I374IXH5hvPBB3D66YnNw3e0uXp14PJJk2DbtsTmvWZNcoLsVVc5PyqLFtV+3nfdBd27Jz6f9AjujRqFXHwlbwNwLh/zFlfzJaehCP9HDLek+vG13AFyCf1EbF+QffBBmDmzYnlpKfz8M/z0E/z9786HCpyRE049NXAfqk6/6yOPBO7Tfz0EdgfFO7ivXh1Y/r1Nqv1YXnghfPFFYvMI9VyCNWucNpPv81qbCgsTfxJ/+XLnNZ2fT5wewT3MSdXBPBJy+QBertLu/YM7hG45+i4XfOedwF/lf/wDunSpaPX5B+apU519DRkCGzY4rfkpU2D4cGd9uCEP/Ac/E4GnnoJZs6pUpUr27HGORvLyoGPH8OlertqfrkrCjdhZmyK13NNVqKNO35FpMn7oDz8cjj02sXn4/s/B3+VJk1LvQoLqSo+P8o03Vil5c37lQJxI0ozlPMVdSIRLJ4OHF7gnyviXvg/HuHFOSx7gL39xXoM/TM8/D3feCRdcUBHUfUK13EP9sAwcCMeFeDzK9u2xf1Dr1nUuM/RdChjOzTfHtr+qWLLEeeThQQfB66+HT5eoVnVZWUWrNdVa7j610RUVKo90Pb/jq7P/0cqXXzpHK08+mZwyxVt6BPcwwxBEsomDeICHeZe+3MUzdGZapTRNWMX5fOCcSPP71j/xROR9q8Lu3U7gDhZ8GPjnPzuvkyc7LXh/wa3ISF++4uLA5WVlTrAeMCByWefOdY4aQgX14EA3blzkffl8/TUMHhw5za5dsHixU+9WreDSS53ln34aWx6R9hstIG3Z4vx9fv3VuQomL6/iaCiRLffi4sD/f2mpc0d0LAG0NoJ7qHM5e1Nw993XkcjnCNTm3zM9grtIxe2LVfAwD5KD83TAtjgPUhXKOJxfactsfqUZH3Ahy5eWsnZt7PstLobc3MonqABmzIj91vZII1H6+H84/Uen9LXY//Uvp39x9uzQebRtC+3bx1aeP/whtnSnnFJx3iCcvn3hiCOcvwdUfKHCBddff63o2gr3BSksdG5YDr6Syd+OHXDggfDXvzrjBjVr5gR43z0L1W25T5wI778fOc3pp1fcXVtS4vyYn3cejB4dff+RTtSPHBn5CG31aqde/sNai0CPHs77SMG9ppYsgQUL4rOveAp1niGZl9omQnoEd58wQxFEcpzbYv8P/TmSBZSRya80ZzbtyXIvocyklNNPd75djVlH5eeDhxauHzzEcDgh+T5szVjO8XzHqlWVv+S+2+Uh8EvkH2jy8pwAvnkzfP+9c/el/0PBQ/0IxWLHDif/iROdI5Wq8LXQg4NSqOC+dq0TiKPxnfcYOTJ8Gl9f8ttvh14fKv9YDtNPPhkuvjhyGt+lrWPGQHa28yxfIKaGQ7jg/vrrzgB3kX7QfD+KL74YuHzyZOe1ug96r18/9DOK/bVqFdjomDPH+WGvySXDGzY4jYdI5f3lF6deCxeGXh/pyCRS6/q77ypa+OEsXRr+f2rBvSY++aTamzYhdJTLCrrT9UKc2xj3Zxvtcc44ZVDKfsTnLhehjGH8ibZ7nAfALieP7ziRFi3g3nsD0z79dOh9hLq365xz4IQToGFDaNfO+ZLURO/e0LSpE9juvjt0mj17nADep49zsjjYHUFPRgzVegr+gob78sUSpKJ1NwTnP2AA/N//hd9fdfiuJ584MXSePj/7jX0Xrry+gez8f+SD+e9/yJDKX5HqBrqNGwOfURyLK65whtgOdyQZiwEDnG6/LyNc2ez78R7r97TnvDx49lnnfajzDLG03E88seIpYeG0bAmHHBJ6nQX3mgh1ZjFGIwl9N8VQ/oJS8Z8/nBUA/MjvmElH/sozvMa1bKcOeSwN2PZiRqFI+QncWDRhNX9iOMNXn1tp3VNPxbybSr79NnC+QYPq7yt4f6HGRPn+e8jJgfvvd0aRvPhiuO8+p6893BepKofGL79cccLaf1v/L+ytt0LjxpW33RhmXLnglvuwYbGXx19pqXPpayxXAIWq88cfO1dZ+UT7MfJfv26dcyljMFXn5P3ZZwcuDw50+fkwrfIpqBpZtswZ0iEeJ659R1+RTv6H+rssX15xYUOkhkC0fvFdoa+GDuuFF5ybtcLllyjeHxUy2MEHV3tTp8ulsj8xnB/5Xfn8s/yFdTSiLU5Ee4a7ytctpSXi120zEOeYvjWLyKcroDRhNas5NGp5SkudO9l+FzVlhS+/jK0PN5z92M5+7AAakrNzE/PozqW8x5YtHWLa3jdMMjj92lDxpKu1a50v+Pvvhz8sD/WlD7Vs3LiKK3f+9jfn1T9IBW+zbp0TaOvXj1z+WIJOaSm89hpccw1kBX2Dtm1zblLp2dO5aW3x4spXAAV3CYXKM/gSxKoEd9+PmW9ZrEcrvsDTtWvodNFMnQrdujndg0ceGbiuTx+nm7JOncA8/f36K9x2W/R8QgXm1193jiJ9N3z57/+RR5xnJPurbss92Lp1zpHIbbeF396/ThbcayIzE3burFb/eySvcV26VKrYAAATpklEQVTA/AiuCJMytOYsJ4fdnM+H3MUz9GUEzVnOWhrzMeewnkbUpwhBA34cuncP38Ofy052EVhP/1vkr+Qt8unCfI6OuZxT6coxzENVabboS45iAQ/yNw48MPLZwi+/dLp5/LuDwt3RGekEW3DLOdQ+xowJvLJoyRLnUNgXNEN9gXwBL9poj7FcLfPSS3D77c45B9/VTj7XXgujRsErrzjz27Y55WvRomp5hruBbeFC52R9s2aB25aVOUdJoU6O+/YVbgz+ml4KWVLi/HD6/v6ffFI5uPtOmPvueA0VCAcMqDgPEUmoE8DXXhu6vKpOF07w1VvB+zjjDOcad6jadRlXXOHcZNarF3QIav9Mn17582vdMjWVm5vU7LdShzpspQtT6Uo+AKO5hCmcxF08A8C7XM7j3MPrXMs6GtOUFRTRkPU0opDDAQK6ggCu4TVewrm28Qw+ZSf70Z3vw5bjLa5mHsdUqezH4HwLhw6t+mVbDRo4v6s14fsynMV4LuR9jj/euWvXX/Alo61aOcHFd4NVpHE7WrWKnL9/0PFdyePvueecQePAOV9QWhp4LiF44LjRo508I3XvBAe6lSsrn2fw3aXapk3gyWXfiTtVp5vCv59+8GDnJHq0FmmkLgpffbZscX6wQn0m+vZ1zuP4ThiH6i4JPlKL5Ud0woTQy/1/jIYNc05gBqvq0cpnn1V0t1RljB3feavgS5HBuREr+FLkWh0+XFWTMh133HGaSMWPP+2758cT0zoahFz+BldXWnYWH5e/H82F+gWn6OlMUAU9nQnlSX1vgnebSbEq6Czali/LZUelbR7uMKo8j/qsj7k6bZmldzCkSn+CZiwrr9vGjeHLHm564oma/xs2blQ95piqbfPhh4HznTpVPd9XX3U+s5HSnHJKYB0nTVKdM6di/o47wm87dmz4daqq999fMf/VV6HTXH65837yZGf+2msjl/fbbyPXac6cwPWh0j79tLP8X/9Szc933peUVKx/++3K2+ze7aS7++7Ide7Z03n/9deq06eHThPMP+9BgwLTv/hiRbpI+W7cGDmPWAD5qtFjbNQEiZoSHdxVVSeP3xLbtyuNpve4WEE1g4pPIqhms1u7850+yIMB6TMp1vMZU2k/oHolbwYs68freh+PlM9nUKKgOonfB/xQ7GIfdx9llYrYn9f0JW7SHHYGLL+Y91RBR3GR+6msKIdvyqBEv6JXwA+Y/5TFHv2anvp7JoVcn0GJCqWVlmezW7PZraDarp3q8XxbXjffdBAb9AgWJuxft3mzag47y39k/ae6bA75t3znnZrnu3Kl6uDBkdP8+9+B86tXx77vcOvmz3e+o5G2HzIkME1xseqzz/p9lvpX3ubII51tBg4Mv19V1V69Iuetqjp6tPN+1aroZfVtEymdqur69ZXTV5UFd5+jj675N8Bj0238U7vyY/m8//vgqTvfxbzfH+gWML8f29xPkDNf/qF13+zDrvJlvqMF3zSQxxVUf8f3eiw/6904zdIxnB+wj5m005m0U1BtzlJV0CIOVlCtwxZ9grvL8zmChaqgi2gV+ssFOoP2lZavpaHuJltB9fdMUgW9l78HpCnk0IA6Bk/7sl0P5Ldq/9tefFF1EwdoKRKwvDULVEGv41+VtqnqUUa46bGBG/VFBlT6we3Np3oB7yfiI6qgun2783fz/bAGT127Vl523XXR9xvtx2rCBOdIKFKaWbMC5wsLo+f7yy9OyAm3fs4c1TVr/D6P1WTB3d+sWVq2Zav+4x+qG6bMTcwn1SYdyOPam08DlrVjpm5n35Dp19Iw5PI1NKq0LPhHqB4b9UnuUgUdwIsKqi0pKF+/P1vLk9djo37GaeXr/HfdgekB+7yEkaqgX9FLFfRi3nO/JYHbduc7PZL55fOf0rvSvn3TzbygCtqElQHLhVJ9nIHaikUh8wDVPjj9Ph/SJ+yfvim/1uhf92L2nwP+jr4pVHniOU2Z4uThf9QHqo1ZrZ9xmh5MUaVt9tuv5vmKqB5/ZJH257WYt8nNjU+d/Y9mqsuCeyTr1qn26KEbLrohMZ9am2p1WsgRqqDn8UH5sq/opS9xU8j0b3JlyOWvcIPeynOVlvt3RfkW+94cSqHuw67yeV8rNJvd+meGBvwonccH2pl8PYwVehqf6Vb2D9iv730BLd1lZVGDe0++VqXiRyh4UtCXuKnS8ue5RRWna20oTnC/gyF6AJvKu6SC61yVKYed2pylIdcdy89hj/pAy3+07+LJhH1sfD/G/j/QoLqKQ3QcZycs344dVTdSTx/m/mqHLwvuMZp5yu26okFHZ2bxYt0zfY6CVuqCsMkmxemuGcfZAcvy6RwwfxLfVHm/x/NtxPW+4N6SAn2OW/UIFuoZfKL38ndV0B/pqlnsUVD9hpMqlck3ncn4gPl92KVPcWfAsnU00Ct4q3w+XLGm06G8y8yZyrQts7QeFWcNMyjRy3lbD+Q3zWGnvoLToArufvsf5+h29lWIHtyz2a3H823IdYewSkOdn9ifrXoEC7UrPyqozqSdKmh7ZuhhrCg/FxOtzvGY9pCl/+CeascsC+418Nhjqpefslr1lVdUR45M3H/ZJptSYPqCyB3QD/Kg1mWzFpOpw7leFfQd+pavf5Mr9UJGh9zWF0RDTcHnYYKnf3JbwKKLGKXDub78SOpWnlMo0/3Zqjfxkv6ViivkbuYFXU99VZyuQf8dgepcjlIFXcUhqqCfcEbAkV9wcRqxRkH1DoZoT76utN53lBh8tHIEC/VxBuoAXiw/WikhI3Va7sCZwAKgABgUYn0OMNJd/yOQF22fqRzcKykO+hA+9JBuKirWoiLVnW06ln8YbrlF9VDCn3lZeeujSf8i22ST16bJnKgbqVereX7EH/QYZmsrFpUfbfhPa2lYraN73xHa33ig2uEobsEdyAQWAy2BfYAZwDFBaW4GhrnvLwNGRtuvp4K7qn5Hd1XQ6+qNDlxRWqq6a5fu2ePMvvee6tgRW3XyK85FyKWHHqbz3s7XSZ9scxJcdZXqDTfo9pW/6ZxDTtGiHwtUy8pUp07V0uUr9I37FmpJr1NVHw3/Q1Dc+6yA+YUcoeM5M+lfQptssqkKUzXFM7gfD0zwm78HuCcozQTgePd9FlAESKT9ei24j/2oTDs1XqU7d9Zuvl99vCPsh2HKX0dr4YJtunGj6scfq7785BYt+GCmTr9/lG6+6hbVoiJdvqTESTx3rpa1aKHffLhRp7/8nc585bvAnS1erLp2repzz6kuWaJlA24uz3fR8VfqjhNPc+72mDZNVzXuqF8166dfPvOLLnngVa3LZl2Y0zbgg7v2stt0yfvTVEeMKF9WmNtS53OkvnDeZ6qLFqk+8ogWX3iJ5vf4i25r0Ex3dz2hSl+OcFfbRN2u4+nJ/2LbZFM1xRrcxUkbnohcDJypqte781cBv1PVW/3SzHbTFLrzi900RUH7uhG4EaBZs2bHLfc9pdZEtGhBGfUbZtRkTLRqKVy0k5lDv+TslyqPTplIO3coow+7na6v38JR5x8VsG7t2opxYjZscIYSqlcP5+sS6j770lLIzKSkxFkd8aFdqkw493mOnPoOLdYFjmGwdN4usnMzadoim5ISZ5Cs45uvorheQzav2EKDNkEjks2fjzY5lN05B7B62W6atc4Jn3dREYu/XUOr89tT8K+vOeL6Xuza5YyisXOHMuWtpZz+p5aAM2Jjk4y1HFRPWfhdES271ueA1o0r6r5gAcycSUGL09lTmklxTh06dMooX711cxn7ZuwmKycTtmxBizawpf0JHFiykbJSZc3MdezXvCH1DlR+GPo9jU9oRYtuDSn6LZOpk3fRduVnFB/ekjX5hZxwby/nD7plCxQXs/uxIRSd0Iff9m1C0ZpSWvU4jGatcwDYXZLJtoI1aJlSt0UDcqb/yPzBIzhq4jDmPfc5+7RtzfYVG2l/VlNW5a+i4ItldD6vKfsf3ZyPP82k05y32dawJQVjZtLzsTPJPPxQcnKgbOVqMj6fwG9LN7E6J4/NR3dn++4szjgDpEUeuk8Os8ctI6/uBlaVHUJe5gq2fjWVBo86o9otejefstFjqPfnfuy7bhlz35lO056tyG3TjC1Sj51vjiKzdDebCjaw/8E5HDX4UrZm1iN3/nT2bd6IPZdeyfrj+7Cp1XEUbDiI4/I2sN/5vanX7AAWjV/IwbMmsaJJNxofsJMGBT+Q8fQT7FO2mwUfzqPNeYGf7ViJyM+q2iVquhiC+yXAGUHBvZuq3uaXZo6bxj+4d1PVsCOGd+nSRfPz82OqjDHGGEeswT2WgcMKwR3JytEUWBUujYhkAQcCYUbMNsYYk2ixBPepQGsRaSEi++CcMB0blGYs0M99fzHwlUY7JDDGGJMwUcdzV9USEbkV56RpJvCaqs4RkYdxOvbHAq8Cb4lIAU6L/bJEFtoYY0xkMT2sQ1XHA+ODlg32e78LuCS+RTPGGFNd6fmwDmOM2ctZcDfGmDRkwd0YY9KQBXdjjElDUW9iSljGIuuB6t6i2gBniIN0YHVJTelSl3SpB1hdfJqrasNoiZIW3GtCRPJjuUPLC6wuqSld6pIu9QCrS1VZt4wxxqQhC+7GGJOGvBrchye7AHFkdUlN6VKXdKkHWF2qxJN97sYYYyLzasvdGGNMBBbcjTEmDXkuuIvImSKyQEQKRGRQsssTioi8JiLr3CdU+ZYdLCKfi8gi9/Ugd7mIyHNufWaKSGe/bfq56ReJSL9QeSW4HoeLyNciMk9E5ojInz1cl1wR+UlEZrh1+Zu7vIWI/OiWa6Q7rDUikuPOF7jr8/z2dY+7fIGInFHbdXHLkCkiv4jIOI/XY5mIzBKR6SKS7y7z3OfLLUM9ERktIvPd78zxSa1LLM/iS5WJGB7WnQoT0APoDMz2W/YkMMh9Pwh4wn1/NvAJIEB34Ed3+cHAEvf1IPf9QbVcjyZAZ/d9XWAhcIxH6yJAHfd9NvCjW8b3gMvc5cOAAe77kA99d+s/A8gBWrifx8wkfMb+CowAxrnzXq3HMqBB0DLPfb7ccvwHuN59vw9QL5l1qdXKx+GPF/Vh3akyAXkEBvcFQBP3fRNggfv+FaBvcDqgL/CK3/KAdEmq00fA6V6vC7AfMA34Hc5dglnBny/CPPQ9+DPnn64Wy98U+BI4BRjnlstz9XDzXUbl4O65zxdwALAU9yKVVKiL17plDgNW+M0Xusu8oLGqrgZwXxu5y8PVKaXq6h7OH4vT4vVkXdyujOnAOuBznNbqJlUtCVGu8jK76zcD9UmNujwLDATK3Pn6eLMeAAp8JiI/i8iN7jIvfr5aAuuB193usn+LyP4ksS5eC+4hHm+P16/lDFenlKmriNQB3gfuUNUtkZKGWJYydVHVUlXthNPy7QYcHSqZ+5qSdRGRc4F1qvqz/+IQSVO6Hn5OVNXOwFnALSLSI0LaVK5LFk5X7MuqeiywHacbJpyE18VrwT2Wh3WnqrUi0gTAfV3nLg9Xp5Soq4hk4wT2d1R1jLvYk3XxUdVNwEScvs564jzUPbhc4R76nuy6nAj0EZFlwH9xumaexXv1AEBVV7mv64APcH50vfj5KgQKVfVHd340TrBPWl28FtxjeVh3qvJ/iHg/nP5r3/Kr3bPn3YHN7uHbBKC3iBzknmHv7S6rNSIiOM/HnaeqQ/xWebEuDUWknvt+X+A0YB7wNc5D3aFyXUI99H0scJl7FUoLoDXwU+3UAlT1HlVtqqp5OJ//r1T1CjxWDwAR2V9E6vre43wuZuPBz5eqrgFWiEgbd9GpwFySWZfaPoEShxMXZ+NctbEYuC/Z5QlTxneB1UAxzi/xdTj9nF8Ci9zXg920Arzo1mcW0MVvP9cCBe50TRLq8XucQ8KZwHR3OtujdekA/OLWZTYw2F3eEieoFQCjgBx3ea47X+Cub+m3r/vcOi4Azkri56wXFVfLeK4ebplnuNMc3/fZi58vtwydgHz3M/YhztUuSauLDT9gjDFpyGvdMsYYY2Jgwd0YY9KQBXdjjElDFtyNMSYNWXA3xpg0ZMHdeJ6IfOe+5onI5XHe972h8jIm1dmlkCZtiEgv4C5VPbcK22SqammE9dtUtU48ymdMbbKWu/E8Ednmvn0cOMkdG/wv7kBhT4nIVHfM7D+56XuJM079CJwbSBCRD93Bq+b4BrASkceBfd39veOfl3tn4VMiMluc8cj/6LfviX7jer/j3ulrTK3Kip7EGM8YhF/L3Q3Sm1W1q4jkAN+KyGdu2m5AO1Vd6s5fq6ob3aEJporI+6o6SERuVWewsWAX4tyR2BFo4G4zyV13LNAWZ0yQb3HGg5kS/+oaE5613E06640zfsd0nKGK6+OMoQLwk19gB7hdRGYAP+AM3NSayH4PvKvOSJNrgW+Arn77LlTVMpwhG/LiUhtjqsBa7iadCXCbqgYMvOT2zW8Pmj8N52EVO0RkIs6YLNH2Hc5uv/el2PfMJIG13E062YrzOECfCcAAd9hiRORId/TBYAcCv7mB/SicoYB9in3bB5kE/NHt12+I82jFWh1V0ZhIrEVh0slMoMTtXnkD+CdOl8g096TmeuD8ENt9CtwkIjNxRkj8wW/dcGCmiExTZ2hdnw9wHmc3A2fkzIGqusb9cTAm6exSSGOMSUPWLWOMMWnIgrsxxqQhC+7GGJOGLLgbY0wasuBujDFpyIK7McakIQvuxhiThv4fTBFUueYqea4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the loss and accuracy\n",
    "\n",
    "plt.plot(loss_list_normal, 'b',label='loss normal cnn ')\n",
    "plt.plot(loss_list_batch,'r',label='loss batch cnn')\n",
    "plt.xlabel('iteration')\n",
    "plt.title(\"loss\")\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEWCAYAAABmE+CbAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl4VEX28PHvIYQdRJZBJCCrshMkgKKsMyqoo7LogIgyKLjgKKioDP7UF8WVcQVHEWRxUFBEFh0GEYOAoBJkB9lRAqjsEPYk5/2jbkgTAumst5M+n+fpJ9331r19usV7uqpuVYmqYowxxhTyOwBjjDGhwRKCMcYYwBKCMcYYjyUEY4wxgCUEY4wxHksIxhhjAEsIxhhjPJYQTIEmIvNEZL+IFPU7FmNCnSUEU2CJSHWgNaDATXn83oXz8v2MyQmWEExBdifwPTAOuCtwh4gUF5F/icgvInJQRBaKSHFv39UiskhEDojIdhHp7W2fJyL3BJyjt4gsDHitItJfRDYCG71tb3rnOCQiS0WkdUD5CBH5p4hsFpHD3v6qIjJSRP6VJt6ZIjIgp78gYwJZQjAF2Z3ARO9xnYhUCtg3HGgGtALKAY8DySJSDZgFvA1UBKKB5Zl4z1uAlkB97/US7xzlgI+AT0WkmLfvEaAHcD1QBugDHAXGAz1EpBCAiFQA/gx8nIk4jMk0SwimQBKRq4FLgE9UdSmwGbjd21cId/F9WFV3qGqSqi5S1RNAT+BrVf1YVU+p6l5VzUxCeFFV96nqMQBV/Y93jkRV/RdQFLjMK3sP8JSqrldnhVf2R+AgLgkAdAfmqerv2fpSjMmAJQRTUN0FfKWqe7zXH5HabFQBKIZLEmlVPcf2YG0PfCEij4rIOq9Z6gBwgff+Gb3XeOAO7/kdwIfZiMmYoFjHlylwvL6A24AIEfnN21wUKCsiTYBVwHGgFrAizeHbgRbnOPURoETA64vSKXN6+mCvv+AJ3C/9NaqaLCL7AQl4r1rA6nTO8x9gtRdvPWDaOWIyJsdYDcEURLcASbh2/GjvUQ9YANypqsnAB8BrInKx17l7pXdr6kTgLyJym4gUFpHyIhLtnXc50EVESohIbeDuDOIoDSQCu4HCIvI0rq8gxWjgORGpI05jESkPoKrxuP6HD4HPUpqgjMlNlhBMQXQXMFZVf1XV31IewAigp3dL6GO4msISYB/wMlBIVX/FdfI+6m1fDjTxzvs6cBL4HdekMzGDOGbjOqg3AL/gaiWBTUqvAZ8AXwGHgDFA8YD944FGWHORySNiC+QYE5pEpA2u6ai6V6sxJldZDcGYECQikcDDwGhLBiavWEIwJsSISD3gAFAZeMPncEwYsSYjY4wxgNUQjDHGePLVOIQKFSpo9erV/Q7DGGPylaVLl+5R1YoZlctXCaF69erExcX5HYYxxuQrIvJLMOWsycgYYwxgCcEYY4zHEoIxxhjAEoIxxhiPJQRjjDGAJQRjjDEeSwjGGGOAfDYOwRiTjyUnw9GjcOQIJCS4R8rztNuOHAFVKFw4/UdExLn3ZXd/4L6ICBd3YmLqIynpzNfZ3RfsMQ89BBUqZPw9Z4MlBGP8cuoU/PILbN4MW7bA3r15c8E71/6ICBBxF8CUi3JGF+5g9yUkuGRgsqZQIejRwxKCMfnaoUPugp9y0U95vnkz/Pqru/iGkogI96s0M0qWdI9SpVL/lioFlSqdvS29cmm3lSwJJUq4i2BWflXn9P60yTOvk3VEhPsu8kBQCUFEOgJvAhG4+dlfSrP/EtyShBVxq0zd4S0BiIi8DNzgFX1OVSd722sAk4BywE9AL1U9me1PZExeSk6GXbvOfdHfu/fM8hUqQK1a0KoV3HGHe57yqFgx9UKUlxe8wMepU1CkSPoX6fQu4CkX7tySclE0eSLDb1pEIoCRwDVAPLBERGao6tqAYsOBCao6XkQ6AC8CvUTkBuBy3Jq2RYFvRWSWqh7CLVn4uqpOEpF3cevT/jsnP5wxOeLECdi6Nf2L/tatcPx4atmICKhWzV3gu3WDmjVTL/g1a0KZMud+H4DIyNz9LMacRzCptwWwSVW3AIjIJOBmIDAh1AcGes9jgWkB279V1UQgUURWAB1F5FOgA3C7V2488CyWEIwfVGH//jN/2Qde9HfscGVSlCzpLvB168INN5x50a9WzS7qJt8KJiFU4cyFweOBlmnKrAC64pqVOgOlRaS8t/0ZEXkNKAG0xyWS8sABL1GknLNKem8uIv2AfgDVqlULIlxjgpCQAF98AZ9+Ct98AwcOnLn/oovcBb5Dh9Rf94FNOyL+xG1MLgomIaT3Lz/tMmuPASNEpDcwH9gBJKrqVyLSHFgE7AYWA4lBntNtVB0FjAKIiYmx5d1M1h0+DDNnwpQpMGuWa+q56CLXtFOvXuoFv0YNVwswJswEkxDigaoBr6OAnYEFVHUn0AVAREoBXVX1oLdvGDDM2/cRsBHYA5QVkcJeLeGscxqTIw4dckng00/hf/9z/QEXXwx9+8Ktt7rO3YgIv6M0JiQEkxCWAHW8u4J2AN1JbfsHQEQqAPtUNRkYjLvjKKVDuqyq7hWRxkBj4CtVVRGJBbrh7jS6C5ieQ5/JhLuDB1OTwOzZLglUqQL33puaBPLoNj5j8pMME4KqJorIg8Bs3G2nH6jqGhEZCsSp6gygHfCiiCiuyai/d3gksEBce+sh3O2oKf0GTwCTROR5YBkwJuc+lgk7Bw7AjBkuCXz1FZw8CVFRcP/9LglccYUlAWMyIKr5p1k+JiZGbQlNc9r+/WcmgVOnoGpV1ydw663QsqUlAWMAEVmqqjEZlbMRHyZ/2bcPpk93SeDrr10SqFbNzfPSrRu0aGFJwJgssoRgQt/evWcmgcREuOQSePhhVxNo3txuAzUmB1hCMKFp7174/PPUcQKJiVC9Ogwc6JJATIwlAWNymCUEEzr27DkzCSQluQFhjz7qksDll1sSMCYXWUIw/tq9G6ZOdUlg3jyXBGrVgkGDXBJo2tSSgDF5xBKCyRuqblbQ9ethwwb3d9kymD/fzRhapw488YRLAk2aWBIwxgeWEEzOOnw49YIf+HfDBjd/UIrixd3kcIMHuyTQuLElAWN8ZgnBZF5iopv2Oe1Ff/16VwtIIeI6gi+7DK6+2v299FL3t0oVuz3UmBBjCcGkTxX++CP9i/7mzS4ppChf3l3or7vuzIt+rVpQrJh/n8EYkymWEMLdkSOwcWP6zTwHD6aWK1oUateGBg2gS5fUi/6ll7qEYIzJ9ywhhJMjRyA2FubMgTVr3EV/+/Yzy1St6i70PXu6vykX/WrVbFZQYwo4SwgFmaq76M+a5R7ffutm/ixeHBo1grZtz2ziqVPHrZFrjAlLlhAKmqNHXS0gJQls2eK2160LDzwAnTpB69bWtm+MOYslhPxO1fUBpCSAefNcLaBECbf846OPuiRQo4bfkRpjQpwlhPzo6FF34U9JAps3u+2XXebm/+/UCdq0sVqAMSZTLCHkF2lrAcePu76ADh3chG+dOrl5f4wxJossIYSqY8fOrAVs2uS2X3qpWwqyUyfXKWy1AGNMDgkqIYhIR+BN3BKao1X1pTT7L8Gto1wR2IdbKjPe2/cKcANQCJgDPOytqTwPqAwc805zrar+ke1PlJ9t2pSaAGJjU2sB7du7uf87dXKDvYwxJhdkmBBEJAIYCVwDxANLRGSGqq4NKDYcmKCq40WkA/Ai0EtEWgFXAY29cguBtsA873VPVQ3fNTGPHXO3gqYkgY0b3fY6daBfP7j+etcXULy4v3EaY8JCMDWEFsAmVd0CICKTgJuBwIRQHxjoPY8FpnnPFSgGFAEEiAR+z37Y+djmzWfWAo4dc80+7du7ZSCtFmCM8UkwCaEKEDicNR5omabMCqArrlmpM1BaRMqr6mIRiQV24RLCCFVdF3DcWBFJAj4DnldVTfvmItIP6AdQrVq14D5VKFq40DX7/PSTe127NvTtm9oXYLUAY4zPgpluMr05idNeuB8D2orIMlyT0A4gUURqA/WAKFxi6SAibbxjeqpqI6C19+iV3pur6ihVjVHVmIoVKwYRboj5/Xe46y43GGz3bnjjDdc0tHEjvPkmdOxoycAYExKCqSHEA1UDXkcBOwMLqOpOoAuAiJQCuqrqQe/X/feqmuDtmwVcAcxX1R3esYdF5CNc09SEbH6e0JGYCO++C0895cYNDB4MQ4ZAyZJ+R2aMMekKpoawBKgjIjVEpAjQHZgRWEBEKohIyrkG4+44AvgVV3MoLCKRuNrDOu91Be/YSOBGYHX2P06IWLwYmjeHf/zD/V21Cl54wZKBMSakZZgQVDUReBCYDawDPlHVNSIyVERu8oq1A9aLyAagEjDM2z4F2AyswvUzrFDVmUBRYLaIrASW45qY3s+xT+WX3bvh7ruhVSv3/JNP4Kuv3AhiY4wJcZJOP27IiomJ0bi4ELxLNSkJRo2Cf/7TLRM5cCA8/TSUKuV3ZMYYg4gsVdWYjMrZSOXs+vFHN4vo0qXu1tERI6B+fb+jMsbkUydPulnr1651y5ak/P3mG7jootx9b0sIWbV3r+soHj3a/Vf6+GP4299soXhjTFBOnHAX/sCL/tq17gbEpCRXRsQNS2rQwN2bktssIWRWcjKMGQNPPumWmBw4EJ55BsqU8TsyY0wIOn7crUqb9hf/5s2pF/5ChdzQpPr1oWtX97d+fdf9mJd3pVtCyIy4OOjf3zUTtWkDI0dCw4Z+R2WMCQHHjrkLf9pf/Js3u9+R4FahrV3bXTZuu8398q9f381ZGQrzVFpCCMa+fW4MwXvvwZ/+BB9+6NYctuYhY8LO0aPw889nXvTXrnWLE6Zc+AsXdlOSNW4MPXq4i36DBm5b0aL+xn8+lhDOJzkZxo2DJ55wSeGhh+D//T+44AK/IzPG5LLkZFi50j1SLvpr1sDWrW6hQnAX/ksvhaZN3W/ElF/8depAkSL+xp8VlhDOZdky1zy0eDFcdZVrHmrSxO+ojDG5aO9emD3bzT05e7YbTgQQGena82Ni3Ew0Kb/4a9d2+woKSwhpHTgA//d/8M47UL68qyH06uV6fYwxBUpysrtjPGUC4h9+cL/+K1SA665zU43FxLg7fQrShf9cLCGkUIUJE+Dxx2HPHje24LnnoGxZvyMzxuSgvXvdBAKzZsH//udqASJulplnnnETEDdr5jqAw40lBHCNhP37uymqr7jC/Uu5/HK/ozLG5IDkZDfrfGAtIDnZNQBcd51bh+raayE/Tqac08I7IRw8CM8+C2+/DRde6MYX9O5tzUPG5HP79p1ZC/jjD1cLiIlxLcKdOrnn4VgLOJ/wTAiq8NFH8Nhjbr2Ce++FYcOgXDm/IzPGZEFysrsPJKUW8P33blu5cq4foFMnVxuwWsD5hV9CWL3aNQ/Nn+8aDWfOdD8VjDH5yv79Z9YCfvcW542JccuQdOrk/he3WkDwwichHD7sxhC88YYbRzBqlJuq2pqHjMkXkpNh+fLUWsDixam1gOuuS60F/OlPfkeaf4VHQpg8GR55BHbudOsYv/CCu6/MGBPS9u+HOXNSk0BgLWDIEJcEWrSwWkBOCY+E8PnnbkbSqVOhZUu/ozEmZKnCqVNuBdikJPc3vUdu7zt+HBYtcrWApCR3z0dgLaBSJb+/qYIpqIQgIh2BN4EIYLSqvpRm/yW4ZTMrAvuAO1Q13tv3CnADbnW2OcDDqqoi0gwYBxQH/puyPSc+1FlGjXLLV9rPCGPStWEDjB/vhuLEx/sdjZsSokkTN8N8Si2gcHj8fPVVhl+xiEQAI4FrgHhgiYjMUNW1AcWGAxNUdbyIdABeBHqJSCvgKqCxV24hbl3lecC/gX7A97iE0BGYlRMf6iw2NbUxZzl0yK3yOm4cfPed607r2BHuu8+Nyi1c2D0iIlKfp33kxj7r1vNPMDm3BbBJVbcAiMgk4GYgMCHUBwZ6z2OBad5zBYoBRQABIoHfRaQyUEZVF3vnnADcQm4lBGMM4DphY2NdEvjsMzdlc7168PLLboaWypX9jtD4KZiEUAXYHvA6HkjbEL8C6IprVuoMlBaR8qq6WERigV24hDBCVdeJSIx3nsBzVknvzUWkH64mQbVq1YII1xiT1ubNrklo/Hj49Vd3o91dd8Hf/+5uzbSZ3A0ElxDS+6eStq3/MWCEiPQG5gM7gEQRqQ3UA6K8cnNEpA1wLIhzuo2qo4BRADExMbnTx2BMAZSQAJ9+6moD8+e7i/6117rawM035+1KXCZ/CCYhxANVA15HATsDC6jqTqALgIiUArqq6kHv1/33qprg7ZsFXAF8SGqSSPecxpjMS052F/9x42DKFDhyxM3X/8ILrkkoKirDU5gwFkz3zRKgjojUEJEiQHdgRmABEakgIinnGoy74wjgV6CtiBQWkUhch/I6Vd0FHBaRK0REgDuB6TnweYwJS9u2uXGXtWtD+/buDuvbb3edxT//7O7WsWRgMpJhDUFVE0XkQWA27rbTD1R1jYgMBeJUdQbQDnhRRBTXZNTfO3wK0AFYhWsS+p+qzvT23U/qbaezsA5lYzLlyBF34R871nUUi0CHDm7W9s6doUQJvyM0+Y3k1q3/uSEmJkbj4uL8DsMY36i6X/1jx7pbRhMS3OItvXu7JqFLLvE7QhOKRGSpqmY4aZsN9TAmH9i+3Q0aGzcONm1y4yxvu83dJXT11XaXkMkZlhCMCVHHjrlZV8aNg6+/drWDdu3cTJ5du0KpUn5HaAoaSwjGhBBVN5f/uHEwaZIbTVy9Ojz9tBs3UKOG3xGagswSgjHZoOpu9cyJCd6WLnWJYP161yHcrZvrG2jb1qZzMHnDEoIJS3v2wLx57u6cFStSZ/jMykU9J7VuDY8/DrfeCqVL5+y5jcmIJQQTFg4ccAO2YmPhm29g5Uq3vVQpaNbMzX+Y15O4pd1XpYprHjLGL5YQTIGUkAALF6YmgJ9+ck07xYq5u3KGDXP37Ddr5mb2NMZYQjAFxPHjbjGVb75xSeCHH1xzTmQkXHEF/N//uQTQsiUULep3tMaEJksIJl86eRKWLElNAIsWwYkTrhkmJgYGDXJTOFx1lY3YNSZYlhBMvpCU5Jp9UpqAFiyAo0fdgKzoaOjf39UAWre29ZCMySpLCCYkJSfDqlWpCWD+fDh40O2rXx/69HEJoG1bKFfO31iNKSgsIZiQoOruv09pAoqNhb173b7ateFvf3NNQO3awUUX+RqqMQWWJQTjm61bYe7c1FrAb7+57VWrwo03uhpA+/butTEm91lCMHnq11/dlAwffwzLl7ttlSq5i39KAqhZ0yZrM8YPlhBMrvv9d7eU48cfu7uBwN3++dpr0LEj1K1rCcCYUGAJweSKAwfc4i0ff+yag5KToVEjNyCse3dXCzDGhBZLCCbHHDkCM2e6JqFZs9xYgZo13fKNPXpAgwZ+R2iMOZ+gEoKIdATexC2hOVpVX0qz/xLcOsoVgX3AHaoaLyLtgdcDitYFuqvqNBEZh1tj2buZkN6qujw7H8bkvRMnYPZslwSmT3djAy6+2I0L6NHDDRKz5iBj8ocME4KIRAAjgWuAeGCJiMxQ1bUBxYYDE1R1vIh0AF4EeqlqLBDtnaccsAn4KuC4Qao6JWc+iskrSUnuzqBJk+Czz1zzUPnybgnHHj3cXEEREX5HaYzJrGBqCC2ATaq6BUBEJgE3A4EJoT4w0HseC0xL5zzdgFmqejTr4Rq/qLq5giZNcmv5/v67mym0c2eXBP7yF5skzpj8LpiEUAXYHvA6HmiZpswKoCuuWakzUFpEyqvq3oAy3YHX0hw3TESeBuYCT6rqibRvLiL9gH4A1apVCyJck1NU3VoBkya5xy+/uInhbrzRdQzfcAMUL+53lMaYnBLMOkzptQBrmtePAW1FZBmuX2AHcHrpEBGpDDQCZgccMxjXp9AcKAc8kd6bq+ooVY1R1ZiKFSsGEa7Jrg0bYOhQN0VE06YwfLh7Pn48/PEHTJniVvOyZGBMwRJMDSEeCBwrGgXsDCygqjuBLgAiUgroqqoHA4rcBnyuqqcCjtnlPT0hImNxScX4ZPt2mDzZ3Sb600+uI7hNG3j4YXfxr1DB7wiNMbktmISwBKgjIjVwv/y7A7cHFhCRCsA+VU3G/fL/IM05enjbA4+prKq7RESAW4DVWfsIJqt2704dMLZwodvWvDn8619w220QFeVvfMaYvJVhQlDVRBF5ENfcEwF8oKprRGQoEKeqM4B2wIsiosB8oH/K8SJSHVfD+DbNqSeKSEVck9Ry4L5sfxqTIVXXHzB+PHz9tbtjqH59eO451y9Qu7bfERpj/CKqabsDQldMTIzGxcX5HUa+lZQEDz4I774LNWq4BNCjBzRsaGMFjCnIRGSpqsZkVM5GKoeJ48ehZ083ncTjj8OLL0KhYG4pMMaEDUsIYeDAAbjlFvj2W3j9dRgwwO+IjDGhyBJCAbdzJ3TqBOvWwUcfuSYiY4xJjyWEAmz9erjuOrfy2JdfwjXX+B2RMSaUWUIooH78Ea6/3vUTzJsHzZr5HZExJtRZt2IB9L//uZXHypSB776zZGCMCY4lhALmww/hr3+FSy91q5PVqeN3RMaY/MISQgEyfDjceaebcuLbb+Gii/yOyBiTn1hCKACSk+HRR2HQIDflxH//65qLjDEmM6xTOZ87eRL69IGJE90o5DfftAFnxpissYSQjyUkQNeu8NVXbvH6wYNtCgpjTNZZQsindu92C9QsXQqjR8Pdd/sdkTEmv7OEkA9t3eoGnG3fDtOmubuKjDEmuywh5DMrVkDHjnDiBMydC61a+R2RMaagsO7HfGTePHdLaeHCbkEbSwbGmJxkCSGfmDLFNRNFRbkBZ/Xr+x2RMaagCSohiEhHEVkvIptE5Ml09l8iInNFZKWIzBORKG97exFZHvA4LiK3ePtqiMgPIrJRRCaLSJGc/WgFxzvvuPEFMTGwYAFUrZrxMcYYk1kZJgQRiQBGAp2A+kAPEUn7+3Q4MEFVGwNDgRcBVDVWVaNVNRroABwFvvKOeRl4XVXrAPsBu08mDVV4+mno3x9uvBHmzIFy5fyOyhhTUAVTQ2gBbFLVLap6EpgE3JymTH1grvc8Np39AN2AWap6VEQElyCmePvGA7dkNviCLDER7r3XrXV8991upbMSJfyOyhhTkAWTEKoA2wNex3vbAq0AunrPOwOlRaR8mjLdgY+95+WBA6qaeJ5zAiAi/UQkTkTidu/eHUS4+d+xY9CtG7z/PgwZ4v4WtvvBjDG5LJiEkN7YV03z+jGgrYgsA9oCO4CUiz0iUhloBMzOxDndRtVRqhqjqjEVK1YMItz8bf9+uPZamDED3n4bnn/eRh8bY/JGML8744HAbswoYGdgAVXdCXQBEJFSQFdVPRhQ5Dbgc1U95b3eA5QVkcJeLeGsc4aj+Hg3xmDjRpg0yXUkG2NMXgmmhrAEqOPdFVQE1/QzI7CAiFQQkZRzDQY+SHOOHqQ2F6Gqiutr6OZtuguYnvnwC45169y4gl9/dQvcWDIwxuS1DBOC9wv+QVxzzzrgE1VdIyJDReQmr1g7YL2IbAAqAcNSjheR6rgaxrdpTv0E8IiIbML1KYzJ1ifJxxYvhquvdjOXfvutW+3MGGPymrgf6/lDTEyMxsXF+R1GjvryS7j1VqhSBWbPhpo1/Y7IGFPQiMhSVY3JqJyNVPbR2LFw881u1PF331kyMMb4yxKCD1ThpZfcwjYdOkBsLPzpT35HZYwJd5YQ8lhyMgwY4Bazuf12+OILKF3a76iMMcYSQp46cQJ69oS33oKBA+HDD6GIzeBkjAkRNv41D/3tbzB9OrzyCjz2mA04M8aEFksIeWTpUpcMnnsOBg3yOxpjjDmbNRnlkZEjoWRJePBBvyMxxpj0WULIA3v2wEcfQa9eULas39EYY0z6LCHkgTFjXIey1Q6MMaHMEkIuS0pyK561bw8NGvgdjTHGnJslhFz2xRduwjqrHRhjQp0lhFz29ttuDeSbbsq4rDHG+MkSQi5atw7mzoX777cVz4wxoc8SQi4aOdKNRL7nHr8jMcaYjFlCyCWHDsH48dC9O4TByp/GmALAEkIumTABEhKsM9kYk39YQsgFyckwYgS0bAnNm/sdjTHGBCeohCAiHUVkvYhsEpEn09l/iYjMFZGVIjJPRKIC9lUTka9EZJ2IrPWW1ERExonIVhFZ7j2ic+pD+W3uXFi/3moHxpj8JcOEICIRwEigE1Af6CEi9dMUGw5MUNXGwFDgxYB9E4BXVbUe0AL4I2DfIFWN9h7Ls/E5QsqIEa7f4NZb/Y7EGGOCF0wNoQWwSVW3qOpJYBJwc5oy9YG53vPYlP1e4iisqnMAVDVBVY/mSOQhats2mDkT+vWDokX9jsYYY4IXTEKoAmwPeB3vbQu0AujqPe8MlBaR8sClwAERmSoiy0TkVa/GkWKY18z0uoike/kUkX4iEicicbt37w7qQ/npnXegUCG47z6/IzHGmMwJJiGkt4yLpnn9GNBWRJYBbYEdQCJuvYXW3v7mQE2gt3fMYKCut70c8ER6b66qo1Q1RlVjKob4/ZtHj8Lo0dC5M0RFZVzeGGNCSTAJIR6oGvA6CtgZWEBVd6pqF1VtCgzxth30jl3mNTclAtOAy739u9Q5AYzFNU3la5Mmwf791plsjMmfgkkIS4A6IlJDRIoA3YEZgQVEpIKIpJxrMPBBwLEXikjKT/sOwFrvmMreXwFuAVZn54P4TdXNW9SwIbRp43c0xhiTeRkmBO+X/YPAbGAd8ImqrhGRoSKSMmVbO2C9iGwAKgHDvGOTcM1Fc0VkFa756X3vmInetlVABeD5HPtUPli0CJYvh3/8w9ZKNsbkT6KatjsgdMXExGhcXJzfYaSrRw+YNQt27HBLZRpjTKgQkaWqGpNRORupnAN27YIpU6BPH0sGxpj8yxJCDhg1ChIT4YEH/I7EGGOyzhJCNp08Ce++C506Qe3afkdjjDFZZwkhm6ZOhd9+c53JxhiTn1lCyKYRI6BWLbjuOr8jMcaY7LGEkA3LlsF330H//m66CmOMyc/sMpYNI0ZAiRLw97/7HYkxxmSfJYQs2ruWLbACAAAVdklEQVQXPvoIevWCsmX9jsYYY7LPEkIWffABHD/umouMMaYgsISQBUlJbprrtm2hUSO/ozHGmJxhCSELvvzSLYRjs5oaYwoSSwhZMGKEW+/gllv8jsQYY3KOJYRM+vlnmDPHrYhWuLDf0RhjTM6xhJBJ77wDRYpA375+R2KMMTnLEkImHD4M48bBbbfBn/7kdzTGGJOzLCFkwoQJLinYvEXGmIIoqIQgIh1FZL2IbBKRJ9PZf4mIzBWRlSIyT0SiAvZVE5GvRGSdiKwVkere9hoi8oOIbBSRyd7ynCFL1XUmN28OLfL96s/GGHO2DBOCiEQAI4FOQH2gh4jUT1NsODBBVRsDQ4EXA/ZNAF5V1XpAC+APb/vLwOuqWgfYD9ydnQ+S2775xnUo262mxpiCKpgaQgtgk6puUdWTwCTg5jRl6gNzveexKfu9xFFYVecAqGqCqh4VEQE6AFO8Y8YDIX0T59tvQ4UKrv/AGGMKomASQhVge8DreG9boBVAV+95Z6C0iJQHLgUOiMhUEVkmIq96NY7ywAFVTTzPOQEQkX4iEicicbt37w7uU+Wwbdtg5kzo1w+KFfMlBGOMyXXBJARJZ5umef0Y0FZElgFtgR1AIlAYaO3tbw7UBHoHeU63UXWUqsaoakzFihWDCDfnvfuu+3vffb68vTHG5IlgEkI8UDXgdRSwM7CAqu5U1S6q2hQY4m076B27zGtuSgSmAZcDe4CyIlL4XOcMFceOwejRblRy1aoZlzfGmPwqmISwBKjj3RVUBOgOzAgsICIVRCTlXIOBDwKOvVBEUn7adwDWqqri+hq6edvvAqZn/WPknkmT3FTX1plsjCnoMkwI3i/7B4HZwDrgE1VdIyJDReQmr1g7YL2IbAAqAcO8Y5NwzUVzRWQVrqnofe+YJ4BHRGQTrk9hTI59qhyi6jqTGzSAdu38jsYYY3KXuB/r+UNMTIzGxcXl2fstXgytWsG//239B8aY/EtElqpqTEblbKTyeYwYAWXKwB13+B2JMcbkPksI5/Dbb/Dpp2695FKl/I7GGGNynyWEcxg1Ck6dsiUyjTHhwxJCOk6dcmMPOnaEOnX8jsYYY/KGLfGSjs8/h1274P33My5rjDEFhdUQ0jFiBNSs6WoIxhgTLqyGkMaKFbBgAQwfDhERfkdjwsGpU6eIj4/n+PHjfodi8rlixYoRFRVFZGRklo63hJDGiBFQvDj06eN3JCZcxMfHU7p0aapXr46bCNiYzFNV9u7dS3x8PDVq1MjSOazJKMC+fTBxoht3cOGFfkdjwsXx48cpX768JQOTLSJC+fLls1XTtIQQYOxYN5md3Wpq8polA5MTsvvvyBKCJykJRo6E1q2hSRO/ozHGmLxnCcEzaxZs3Qr/+IffkRgT3nr37s2UKVPS3Td8+HDq1q1Lw4YNadKkCRMmTACgXbt2xMSkTtUTFxdHO29Gynnz5iEizJw58/T+G2+8kXnz5uXaZ8ivLCF4RoyAiy926x4YY7ImMTEx40JZ9O677zJnzhx+/PFHVq9ezfz58wmcnPOPP/5g1qxZ6R4bFRXFsGHDci22gsLuMgI2bIDZs2HoUMji3VrG5IgBA2D58pw9Z3Q0vPFG8OVLlSrFww8/zBdffEHx4sWZPn06lSpV4pdffqFPnz7s3r2bihUrMnbsWKpVq0bv3r0pV64cy5Yt4/LLL6d06dJs3bqVXbt2sWHDBl577TW+//57Zs2aRZUqVZg5cyaRkZEMHTqUmTNncuzYMVq1asV777133jbwF154gdjYWMqUKQPABRdcwF133XV6/6BBg3j++efp1KnTWcc2adKEU6dOMWfOHK655prgv4wwYzUEXN9BZCT07et3JMb478iRI1xxxRWsWLGCNm3a8L43ZP/BBx/kzjvvZOXKlfTs2ZOHHnro9DEbNmzg66+/5l//+hcAmzdv5ssvv2T69OnccccdtG/fnlWrVlG8eHG+/PLL0+dbsmQJq1ev5tixY3zxxRfnjOnw4cMcPnyYWrVqnbPMlVdeSdGiRYmNjU13/1NPPcXzzz+f6e8jnIR9DeHwYRg3Dm67DS66yO9oTLjLzC/53FKkSBFuvPFGAJo1a8acOXMAWLx4MVOnTgWgV69ePP7446ePufXWW4kIGMnZqVMnIiMjadSoEUlJSXT0hv03atSIbdu2ARAbG8srr7zC0aNH2bdvHw0aNOCvf/1rujGpalB30KRc9F9++eWz9rVu3RqABQsWZHiecBVUDUFEOorIehHZJCJPprP/EhGZKyIrRWSeiEQF7EsSkeXeY0bA9nEisjVgX3TOfKTM+c9/4NAhWyLTmBSRkZGnL74RERHn7BcIvECXLFnyjH1FixYFoFChQmecr1ChQiQmJnL8+HEeeOABpkyZwqpVq+jbt+95758vU6YMJUuWZMuWLeeNvUOHDhw/fpzvv/8+3f1DhgyxvoTzyDAhiEgEMBLoBNQHeohI/TTFhgMTVLUxMBR4MWDfMVWN9h43pTluUMC+HG45zZiq60xu1gxatszrdzcmf2nVqhWTJk0CYOLEiVx99dVZPlfKxb9ChQokJCSc866iQIMHD6Z///4cOnQIgEOHDjFq1Kizyg0ZMoRXXnkl3XNce+217N+/nxUrVmQ59oIsmCajFsAmVd0CICKTgJuBtQFl6gMDveexwLScDDK3xMbC2rVuQJqNCzLm/N566y369OnDq6++erpTOavKli1L3759adSoEdWrV6d58+YZHnP//feTkJBA8+bNiYyMJDIykkcfffSsctdffz0VK1Y853mGDBnCzTffnOXYC7IM11QWkW5AR1W9x3vdC2ipqg8GlPkI+EFV3xSRLsBnQAVV3SsiicByIBF4SVWneceMA64ETgBzgSdV9cT5YsnpNZW7dIH58yE+HooVy7HTGpMp69ato169en6HYQqI9P495eSayun9dk6bRR4D2orIMqAtsAOXAACqeYHcDrwhIim3CQwG6gLNgXLAE+m+uUg/EYkTkbjdu3cHEW5wfv0Vpk93dxZZMjDGmOASQjxQNeB1FLAzsICq7lTVLqraFBjibTuYss/7uwWYBzT1Xu9S5wQwFtc0dRZVHaWqMaoac75qYGa9+677e999OXZKY4zJ14JJCEuAOiJSQ0SKAN2BGYEFRKSCiKScazDwgbf9QhEpmlIGuAqv70FEKnt/BbgFWJ39jxOc48fdamg33QSXXJJX72qMMaEtw05lVU0UkQeB2UAE8IGqrhGRoUCcqs4A2gEviogC84GU+ULrAe+JSDIu+bykqimd0RNFpCKuSWo5kGe/1SdPhj17bN4iY4wJFNTANFX9L/DfNNueDng+BTjrvjFVXQQ0Osc5O2Qq0hyiCm+/DfXqQfv2fkRgjDGhKexGKv/4Iyxd6qarsFtNjTEmVdjNZfT221C6NPTq5XckxoSmhIQE7r33XmrVqkWDBg1o06YNP/zwA+BGJwfe+z98+HCeffZZAJ599llKlCjBH3/8cXp/qVKlshTDPffcw9q1rnX5hRdeOL1927ZtNGzYMMPjn332WapUqUJ0dDR169bl/vvvJzk5+bzHTJs27fR7nsu8efNOT+txPr1796ZKlSqcOOHupN+zZw/Vq1fP8Di/hVVC+P13+OQT+PvfXVIwxpztnnvuoVy5cmzcuJE1a9Ywbtw49uzZA7gpKaZOnXr6dVoVKlQ4PcFddowePZr69d2ECIEJITMGDhzI8uXLWbt2LatWreLbb789b/lgEkJmRERE8MEHH2Tp2KSkpByLIzPCKiG8/z6cOgUPPOB3JMacw4AB0K5dzj4GDAj67Tdv3swPP/zA888/T6FC7vJQs2ZNbrjhBgAKFy5Mv379eP3119M9vk+fPkyePJl9+/ad8z0++eQTHnnkEQDefPNNatasefq9U6bDaNeuHXFxcTz55JMcO3aM6OhoevbsCbiLZd++fWnQoAHXXnstx44dO+9nOnnyJMePH+dCb6H0999/n+bNm9OkSRO6du3K0aNHWbRoETNmzGDQoEFER0ezefNmNm3axF/+8heaNGnC5ZdfzubNmwFXg+rWrRt169alZ8+eZ6zJEGjAgAG8/vrrZ80FpaoMGjSIhg0b0qhRIyZPngy42kf79u25/fbbT08CWLduXe655x4aNmxIz549+frrr7nqqquoU6cOP/7443k/d1aETUI4dcqNPbj2WrjsMr+jMSY0rVmzhujo6DNmLk2rf//+TJw4kYMHD561r1SpUvTp04c333zznMe3adPm9IyjCxYsoHz58uzYsYOFCxeenpE0xUsvvUTx4sVZvnw5EydOBGDjxo3079+fNWvWULZsWT777LN03+f1118nOjqaypUrc+mllxId7ebP7NKlC0uWLGHFihXUq1ePMWPG0KpVK2666SZeffVVli9fTq1atejZsyf9+/dnxYoVLFq0iMqVKwOwbNky3njjDdauXcuWLVv47rvv0n3/atWqcfXVV/Phhx+esX3q1KksX76cFStW8PXXXzNo0CB27doFwI8//siwYcNO11Q2bdrEww8/zMqVK/n555/56KOPWLhwIcOHD89yzel8wqZTedo02LED/v1vvyMx5jxCYf7rDJQpU4Y777yTt956i+LFi5+1/6GHHiI6OjrdeYYALrroIhISEjh8+DDbt2/n9ttvZ/78+SxYsIAuXbpk+P41atQ4fXFv1qzZ6em00xo4cCCPPfYYp06dolu3bkyaNInu3buzevVqnnrqKQ4cOEBCQgLXXXfdWccePnyYHTt20LlzZwCKBUxn0KJFC6Ki3ITO0dHRbNu27ZwT/f3zn//kpptuOl3DAli4cCE9evQgIiKCSpUq0bZtW5YsWUKZMmVo0aIFNWrUOOOzNmrkbtRs0KABf/7znxGRM6YRz0lhU0MYMQJq1IDrr/c7EmNCV4MGDVixYkWGHbADBgxgzJgxHDly5Kx9ZcuW5fbbb+edd9455/FXXnklY8eO5bLLLqN169YsWLCAxYsXc9VVV2UYY8rU2nD+6blTREZG0rFjR+bPnw+4Dt8RI0awatUqnnnmmXSn3T7fHG+Zef/atWsTHR3NJ598EtS5zzWNOLipwwOnFc+N5UrDIiGsXOkmsXvgAThPTdiYsFerVi1iYmJ45plnTl+4Nm7cyPTp088oV65cOW677TbGjBmT7nkeeeQR3nvvvXNetNq0acPw4cNp06YNTZs2JTY2lqJFi3LBBRecVTYyMpJTp05l+TOpKosWLTq92trhw4epXLkyp06dOt0MBVC6dGkOHz4MuFpQVFQU06a5iZtPnDjB0aNHs/T+Q4YMYfjw4adft2nThsmTJ5OUlMTu3buZP38+LVqkO3NPnguLhDBypJvArk8fvyMxJvSNHj2a3377jdq1a9OoUSP69u3LxRdffFa5Rx999Lx3G3Xu3Pn0bZdptW7dmu3bt9OmTRsiIiKoWrXqOZtd+vXrR+PGjU93KgcrpQ+hYcOGJCYm8oB3N8lzzz1Hy5Ytueaaa6hbt+7p8t27d+fVV1+ladOmbN68mQ8//JC33nqLxo0b06pVK3777bdMvX+KBg0acPnll59+3blzZxo3bkyTJk3o0KEDr7zyCheFyHKNGU5/HUqyOv31yy/DgQPw4osZlzUmr9n01yYnZWf667DoVH4i3Ym1jTHGBAqLJiNjjDEZs4RgTAjIT023JnRl99+RJQRjfFasWDH27t1rScFki6qyd+/eM8ZMZFZY9CEYE8qioqKIj48nJ5eINeGpWLFipwfNZYUlBGN8FhkZecboVGP8Yk1GxhhjAEsIxhhjPJYQjDHGAPlspLKI7AZ+yeLhFYD0x9mHJ/s+Utl3cSb7Ps5UEL6PS1S1YkaF8lVCyA4RiQtm6Ha4sO8jlX0XZ7Lv40zh9H1Yk5ExxhjAEoIxxhhPOCWEUX4HEGLs+0hl38WZ7Ps4U9h8H2HTh2CMMeb8wqmGYIwx5jwsIRhjjAHCJCGISEcRWS8im0TkSb/j8YuIVBWRWBFZJyJrRORhv2MKBSISISLLROQLv2Pxm4iUFZEpIvKz9+/kSr9j8ouIDPT+P1ktIh+LSNanEc0nCnxCEJEIYCTQCagP9BCR+v5G5ZtE4FFVrQdcAfQP4+8i0MPAOr+DCBFvAv9T1bpAE8L0exGRKsBDQIyqNgQigO7+RpX7CnxCAFoAm1R1i6qeBCYBN/scky9UdZeq/uQ9P4z7n72Kv1H5S0SigBuA0X7H4jcRKQO0AcYAqOpJVT3gb1S+KgwUF5HCQAlgp8/x5LpwSAhVgO0Br+MJ84sggIhUB5oCP/gbie/eAB4Hkv0OJATUBHYDY70mtNEiUtLvoPygqjuA4cCvwC7goKp+5W9UuS8cEoKksy2s77UVkVLAZ8AAVT3kdzx+EZEbgT9UdanfsYSIwsDlwL9VtSlwBAjLPjcRuRDXklADuBgoKSJ3+BtV7guHhBAPVA14HUUYVP3ORUQicclgoqpO9Tsen10F3CQi23BNiR1E5D/+huSreCBeVVNqjVNwCSIc/QXYqqq7VfUUMBVo5XNMuS4cEsISoI6I1BCRIriOoRk+x+QLERFc+/A6VX3N73j8pqqDVTVKVavj/l18o6oF/lfguajqb8B2EbnM2/RnYK2PIfnpV+AKESnh/X/zZ8Kgg73AL6Gpqoki8iAwG3enwAequsbnsPxyFdALWCUiy71t/1TV//oYkwkt/wAmej+etgB/9zkeX6jqDyIyBfgJd3feMsJgCgubusIYYwwQHk1GxhhjgmAJwRhjDGAJwRhjjMcSgjHGGMASgjHGGI8lBGMCiEiSiCwPeOTYSF0RqS4iq3PqfMbktAI/DsGYTDqmqtF+B2GMH6yGYEwQRGSbiLwsIj96j9re9ktEZK6IrPT+VvO2VxKRz0VkhfdImfYgQkTe9+bZ/0pEivv2oYxJwxKCMWcqnqbJ6G8B+w6pagtgBG6WVLznE1S1MTAReMvb/hbwrao2wc0HlDI6vg4wUlUbAAeArrn8eYwJmo1UNiaAiCSoaql0tm8DOqjqFm+CwN9UtbyI7AEqq+opb/suVa0gIruBKFU9EXCO6sAcVa3jvX4CiFTV53P/kxmTMashGBM8Pcfzc5VJz4mA50lYP54JIZYQjAne3wL+LvaeLyJ1acWewELv+Vzgfji9ZnOZvArSmKyyXyfGnKl4wEyw4NYXTrn1tKiI/ID7IdXD2/YQ8IGIDMKtNpYyO+jDwCgRuRtXE7gft/KWMSHL+hCMCYLXhxCjqnv8jsWY3GJNRsYYYwCrIRhjjPFYDcEYYwxgCcEYY4zHEoIxxhjAEoIxxhiPJQRjjDEA/H9QJewOcKc6pwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(accuracy_list_normal, 'b',label=' normal CNN')\n",
    "plt.plot(accuracy_list_batch,'r',label=' CNN with Batch Norm')\n",
    "plt.xlabel('Epoch')\n",
    "plt.title(\"Accuracy \")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see the CNN with batch norm performers better, with faster convergence.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://dataplatform.cloud.ibm.com/registration/stepone?context=cpdaas&apps=data_science_experience,watson_machine_learning\"><img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-DL0110EN-SkillsNetwork/Template/module%201/images/Watson_Studio.png\"/></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--Empty Space for separating topics-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>About the Authors:</h2> \n",
    "\n",
    "<a href=\"https://www.linkedin.com/in/joseph-s-50398b136/\">Joseph Santarcangelo</a> has a PhD in Electrical Engineering, his research focused on using machine learning, signal processing, and computer vision to determine how videos impact human cognition. Joseph has been working for IBM since he completed his PhD.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other contributors: <a href=\"https://www.linkedin.com/in/michelleccarey/\">Michelle Carey</a>, <a href=\"www.linkedin.com/in/jiahui-mavis-zhou-a4537814a\">Mavis Zhou</a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thanks to Magnus <a href=\"http://www.hvass-labs.org/\">Erik Hvass Pedersen</a> whose tutorials helped me understand convolutional Neural Network\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Change Log\n",
    "\n",
    "| Date (YYYY-MM-DD) | Version | Changed By | Change Description                                          |\n",
    "| ----------------- | ------- | ---------- | ----------------------------------------------------------- |\n",
    "| 2020-09-23        | 2.0     | Srishti    | Migrated Lab to Markdown and added to course repo in GitLab |\n",
    "\n",
    "<hr>\n",
    "\n",
    "## <h3 align=\"center\"> © IBM Corporation 2020. All rights reserved. <h3/>\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python",
   "language": "python",
   "name": "conda-env-python-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}