8.1.1mist2layer_v2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center>\n",
    "    <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-DL0110EN-SkillsNetwork/Template/module%201/images/IDSNlogo.png\" width=\"300\" alt=\"cognitiveclass.ai logo\"  />\n",
    "</center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>Hidden Layer Deep Network: Sigmoid, Tanh and Relu Activations Functions MNIST Dataset</h1>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Objective for this Notebook<h3>    \n",
    "<h5> 1. Define Several Neural Network, Criterion function, Optimizer.</h5>\n",
    "<h5> 2. Test Sigmoid ,Tanh and Relu. </h5>\n",
    "<h5> 3. Analyse Results. </h5>     \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Table of Contents</h2>\n",
    "<p>In this lab, you will test Sigmoid, Tanh and Relu activation functions on the MNIST dataset with two hidden Layers.</p>\n",
    "\n",
    "<ul>\n",
    "    <li><a href=\"#Model\">Neural Network Module and Training Function</a></li>\n",
    "    <li><a href=\"#Makeup_Data\">Make Some Data</a></li>\n",
    "    <li><a href=\"#Train\">Define Several Neural Network, Criterion function, Optimizer</a></li>\n",
    "    <li><a href=\"#Test\">Test Sigmoid ,Tanh and Relu </a></li>\n",
    "    <li><a href=\"#Result\">Analyse Results</a></li>\n",
    "</ul>\n",
    "<p>Estimated Time Needed: <strong>25 min</strong></p>\n",
    "\n",
    "<hr>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll need the following libraries\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch._C.Generator at 0x7fef3315fcf0>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Import the libraries we need for this lab\n",
    "\n",
    "# Using the following line code to install the torchvision library\n",
    "# !conda install -y torchvision\n",
    "\n",
    "import torch \n",
    "import torch.nn as nn\n",
    "import torchvision.transforms as transforms\n",
    "import torchvision.datasets as dsets\n",
    "import torch.nn.functional as F\n",
    "import matplotlib.pylab as plt\n",
    "import numpy as np\n",
    "torch.manual_seed(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--Empty Space for separating topics-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 id=\"Model\">Neural Network Module and Training Function</h2> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the neural network module or class, with two hidden Layers \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://ibm.box.com/shared/static/5wtclahun0f70qlwkn2kxzh3amnbq4zg.png\" width=\"200\" alt=\"Neural Network Model\" />\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the model class using sigmoid as the activation function\n",
    "\n",
    "class Net(nn.Module):\n",
    "    \n",
    "    # Constructor\n",
    "    def __init__(self, D_in, H1, H2, D_out):\n",
    "        super(Net, self).__init__()\n",
    "        self.linear1 = nn.Linear(D_in, H1)\n",
    "        self.linear2 = nn.Linear(H1, H2)\n",
    "        self.linear3 = nn.Linear(H2, D_out)\n",
    "    \n",
    "    # Prediction\n",
    "    def forward(self,x):\n",
    "        x = torch.sigmoid(self.linear1(x)) \n",
    "        x = torch.sigmoid(self.linear2(x))\n",
    "        x = self.linear3(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the class with the Tanh activation function \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the model class using Tanh as a activation function\n",
    "\n",
    "class NetTanh(nn.Module):\n",
    "    \n",
    "    # Constructor\n",
    "    def __init__(self, D_in, H1, H2, D_out):\n",
    "        super(NetTanh, self).__init__()\n",
    "        self.linear1 = nn.Linear(D_in, H1)\n",
    "        self.linear2 = nn.Linear(H1, H2)\n",
    "        self.linear3 = nn.Linear(H2, D_out)\n",
    "    \n",
    "    # Prediction\n",
    "    def forward(self, x):\n",
    "        x = torch.tanh(self.linear1(x))\n",
    "        x = torch.tanh(self.linear2(x))\n",
    "        x = self.linear3(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the class for the Relu activation function \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the model class using Relu as a activation function\n",
    "\n",
    "class NetRelu(torch.nn.Module):\n",
    "    def __init(self, in_dim, h1_dim, h2_dim, out_dim):\n",
    "        super(Network, self).__init__()\n",
    "        self.linear1 = nn.Linear(in_dim, h1_dim)\n",
    "        self.linear2 = nn.Linear(h1_dim, h2_dim)\n",
    "        self.linear3 = nn.Linaer(h2_dim, out_dim)\n",
    "        \n",
    "    def forward(self, X):\n",
    "        temp = torch.relu(self.linear1(X))\n",
    "        temp = torch.relu(self.linear2(temp))\n",
    "        temp = self.linear3(temp) #since this is a regression network there is no activation at the output layer.\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define a function to  train the model, in this case the function returns a Python dictionary to store the training loss and accuracy on the validation data \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train the model\n",
    "\n",
    "def train(model, criterion, train_loader, validation_loader, optimizer, epochs=100):\n",
    "    i = 0\n",
    "    useful_stuff = {'training_loss': [], 'validation_accuracy': []}  \n",
    "    \n",
    "    for epoch in range(epochs):\n",
    "        for i, (x, y) in enumerate(train_loader):\n",
    "            optimizer.zero_grad()\n",
    "            z = model(x.view(-1, 28 * 28))\n",
    "            loss = criterion(z, y)\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "            useful_stuff['training_loss'].append(loss.data.item())\n",
    "        \n",
    "        correct = 0\n",
    "        for x, y in validation_loader:\n",
    "            z = model(x.view(-1, 28 * 28))\n",
    "            _, label = torch.max(z, 1)\n",
    "            correct += (label == y).sum().item()\n",
    "            \n",
    "        accuracy = 100 * (correct / len(validation_dataset))\n",
    "        print(\"Epoch: {}, Accuracy: {}\".format(epoch, accuracy))    \n",
    "\n",
    "        useful_stuff['validation_accuracy'].append(accuracy)\n",
    "    \n",
    "    return useful_stuff"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--Empty Space for separating topics-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 id=\"Makeup_Data\">Make Some Data</h2> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the training dataset by setting the parameters <code>train</code> to <code>True</code> and convert it to a tensor  by placing a transform object int the argument <code>transform</code>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the training dataset\n",
    "\n",
    "train_dataset = dsets.MNIST(root='./data', train=True, download=True, transform=transforms.ToTensor())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the testing dataset by setting the parameters <code>train</code> to <code>False</code> and convert it to a tensor  by placing a transform object int the argument <code>transform</code>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the validating dataset\n",
    "\n",
    "validation_dataset = dsets.MNIST(root='./data', train=False, download=True, transform=transforms.ToTensor())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the criterion function  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the criterion function\n",
    "\n",
    "criterion = nn.CrossEntropyLoss() #multiclass classification."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the training-data loader and the validation-data loader object \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the training data loader and validation data loader object\n",
    "\n",
    "train_loader = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=32, shuffle=True)\n",
    "validation_loader = torch.utils.data.DataLoader(dataset=validation_dataset, batch_size=32, shuffle=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--Empty Space for separating topics-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 id=\"Train\">Define Neural Network, Criterion function, Optimizer and Train the Model</h2> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create  the model with 100 hidden layers  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the parameters for create the model\n",
    "\n",
    "input_dim = 28 * 28\n",
    "hidden_dim1 = 50\n",
    "hidden_dim2 = 50\n",
    "output_dim = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The epoch number in the video is 35. You can try 10 for now. If you try 35, it may take a long time.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the number of iterations\n",
    "\n",
    "cust_epochs = 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--Empty Space for separating topics-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 id=\"Test\">Test Sigmoid ,Tanh and Relu</h2> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train the network using the Sigmoid activation function\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train the model with sigmoid function\n",
    "\n",
    "learning_rate = 0.01\n",
    "model = Net(input_dim, hidden_dim1, hidden_dim2, output_dim)\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)\n",
    "training_results = train(model, criterion, train_loader, validation_loader, optimizer, epochs=cust_epochs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train the network using the Tanh activation function\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train the model with tanh function\n",
    "\n",
    "learning_rate = 0.01\n",
    "model_Tanh = NetTanh(input_dim, hidden_dim1, hidden_dim2, output_dim)\n",
    "optimizer = torch.optim.Adam(model_Tanh.parameters(), lr=learning_rate)\n",
    "training_results_tanch = train(model_Tanh, criterion, train_loader, validation_loader, optimizer, epochs=cust_epochs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train the network using the Relu activation function\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train the model with relu function\n",
    "\n",
    "learning_rate = 0.01\n",
    "modelRelu = NetRelu(input_dim, hidden_dim1, hidden_dim2, output_dim)\n",
    "optimizer = torch.optim.Adam(modelRelu.parameters(), lr=learning_rate)\n",
    "training_results_relu = train(modelRelu, criterion, train_loader, validation_loader, optimizer, epochs=cust_epochs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--Empty Space for separating topics-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 id=\"Result\">Analyze Results</h2> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compare the training loss for each activation \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compare the training loss\n",
    "\n",
    "plt.plot(training_results_tanch['training_loss'], label='tanh')\n",
    "plt.plot(training_results['training_loss'], label='sigmoid')\n",
    "plt.plot(training_results_relu['training_loss'], label='relu')\n",
    "plt.ylabel('loss')\n",
    "plt.title('training loss iterations')\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compare the validation loss for each model  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compare the validation loss\n",
    "\n",
    "plt.plot(training_results_tanch['validation_accuracy'], label = 'tanh')\n",
    "plt.plot(training_results['validation_accuracy'], label = 'sigmoid')\n",
    "plt.plot(training_results_relu['validation_accuracy'], label = 'relu') \n",
    "plt.ylabel('validation accuracy')\n",
    "plt.xlabel('Iteration')   \n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://dataplatform.cloud.ibm.com/registration/stepone?context=cpdaas&apps=data_science_experience,watson_machine_learning\"><img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-DL0110EN-SkillsNetwork/Template/module%201/images/Watson_Studio.png\"/></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--Empty Space for separating topics-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>About the Authors:</h2> \n",
    "\n",
    "<a href=\"https://www.linkedin.com/in/joseph-s-50398b136/\">Joseph Santarcangelo</a> has a PhD in Electrical Engineering, his research focused on using machine learning, signal processing, and computer vision to determine how videos impact human cognition. Joseph has been working for IBM since he completed his PhD. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other contributors: <a href=\"https://www.linkedin.com/in/michelleccarey/\">Michelle Carey</a>, <a href=\"www.linkedin.com/in/jiahui-mavis-zhou-a4537814a\">Mavis Zhou</a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Change Log\n",
    "\n",
    "| Date (YYYY-MM-DD) | Version | Changed By | Change Description                                          |\n",
    "| ----------------- | ------- | ---------- | ----------------------------------------------------------- |\n",
    "| 2020-09-23        | 2.0     | Srishti    | Migrated Lab to Markdown and added to course repo in GitLab |\n",
    "\n",
    "<hr>\n",
    "\n",
    "## <h3 align=\"center\"> © IBM Corporation 2020. All rights reserved. <h3/>\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python",
   "language": "python",
   "name": "conda-env-python-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}