CIFAR10-ResidualNet.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PhShOHkoCHQ4"
   },
   "source": [
    "# CE-40719: Deep Learning\n",
    "## HW3 - CNN / CNN Case Studies / CNN Applications\n",
    "(18 points)\n",
    "\n",
    "#### Name: Sadroddin Barikbin\n",
    "#### Student No.: 98208824"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "E8SJVCBZxJQB"
   },
   "source": [
    "In this exercise we are going to implement a simple residual netwrok architecture to classify images from Cifar10 dataset. Here we give you a suggestion for architecture but you are allowed to  make changes and experiment to get better results. Explain your ideas or reference the papers that you take the ideas from. You are allowed to use out-of-the-box pytorch modules in `torch.nn`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8bd7O4pSrpL1"
   },
   "source": [
    "## Architecture\n",
    "\n",
    "_All convolution layers have `3 * 3` kernel,  `padding=1`, batch normalization and relu activation_\n",
    "\n",
    "__ResidualBlock:__ _in_channel, out_channel, stride_\n",
    "\n",
    "- Conv(in_channel, out_channel, stride)\n",
    "- Conv(out_channel, out_channel, stride=1)\n",
    "\n",
    "`*` This block has a residual connection. To match dimmensions of output and residual use `1 * 1` convolution and stride. \n",
    "\n",
    "__ResidualLayer:__ _in_channel, out_channel, stride_\n",
    "\n",
    "- ResidualBlock(in_channel, out_channel, stride)\n",
    "- ResidualBlock(out_channel, out_channel, stride=1)\n",
    "\n",
    "__ResidualNetwork__:\n",
    "- Conv(3, 64, stride=1)\n",
    "- ResidualLayer(64, 64, stride=1)\n",
    "- ResidualLayer(64, 128, stride=2)\n",
    "- ResidualLayer(128, 256, stride=2)\n",
    "- ResidualLayer(256, 512, stride=2)\n",
    "- AveragePool(4, 4)\n",
    "- Linear(512, 10)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "7z8_RIQFnwPx"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch import nn,optim\n",
    "from torch.nn import functional as F\n",
    "import torchvision\n",
    "import torchvision.transforms as transforms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "UjUjOCpQqVav"
   },
   "outputs": [],
   "source": [
    "class ResidualBlock(nn.Module):\n",
    "  def __init__(self,in_channel,out_channel,stride):\n",
    "    super(ResidualBlock,self).__init__()\n",
    "    self.conv1=nn.Conv2d(in_channel,out_channel,3,stride=stride,padding=1)\n",
    "    self.batch1=nn.BatchNorm2d(out_channel)\n",
    "    self.conv2=nn.Conv2d(out_channel,out_channel,3,padding=1)\n",
    "    self.batch2=nn.BatchNorm2d(out_channel)\n",
    "    self.reshaper=nn.Conv2d(in_channel,out_channel,1,stride=stride)\n",
    "  def forward(self,x):\n",
    "    y=F.relu(self.batch1(self.conv1(x)))\n",
    "    y=F.relu(self.batch2(self.conv2(y)))\n",
    "    return self.reshaper(x)+y\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Ix6fA875tb-v"
   },
   "outputs": [],
   "source": [
    "class ResidualLayer(nn.Module):\n",
    "  def __init__(self,in_channel,out_channel,stride):\n",
    "    super(ResidualLayer,self).__init__()\n",
    "    self.block1=ResidualBlock(in_channel,out_channel,stride)\n",
    "    self.block2=ResidualBlock(out_channel,out_channel,1)\n",
    "  def forward(self,x):\n",
    "    return self.block2(self.block1(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "2QPDVsQXKokZ"
   },
   "outputs": [],
   "source": [
    "class ResidualNetwork(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(ResidualNetwork, self).__init__()\n",
    "        self.conv=nn.Conv2d(3,64,3,padding=1)\n",
    "        self.res1=ResidualLayer(64,64,1)\n",
    "        self.res2=ResidualLayer(64,128,2)\n",
    "        self.res3=ResidualLayer(128,256,2)\n",
    "        self.res4=ResidualLayer(256,512,2)\n",
    "        self.pool=nn.AvgPool2d(4)\n",
    "        self.line=nn.Linear(512,10)\n",
    "    def forward(self, x):\n",
    "        return self.line(torch.flatten(self.pool(self.res4(self.res3(self.res2(self.res1(self.conv(x)))))),1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 3104,
     "status": "ok",
     "timestamp": 1586182285213,
     "user": {
      "displayName": "sadra barikbein",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiCq0YqNHWfcZyly3I_AMTkvS8_tO8PZOQpGtkK=s64",
      "userId": "07513140462530453530"
     },
     "user_tz": -270
    },
    "id": "z5NuweLooE9h",
    "outputId": "0ce60556-f0c2-48bb-aae1-8a83395be4a9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files already downloaded and verified\n",
      "Files already downloaded and verified\n"
     ]
    }
   ],
   "source": [
    "batch_size_train = 256\n",
    "batch_size_test = 256\n",
    "\n",
    "transform=transforms.Compose([transforms.ToTensor(),transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])\n",
    "\n",
    "trainset = torchvision.datasets.CIFAR10(root='./data', train=True,download=True, transform=transform)\n",
    "trainloader = torch.utils.data.DataLoader(trainset, batch_size=batch_size_train)\n",
    "\n",
    "testset = torchvision.datasets.CIFAR10(root='./data', train=False,download=True, transform=transform)\n",
    "testloader=torch.utils.data.DataLoader(testset, batch_size=batch_size_test)\n",
    "classes = ('plane', 'car', 'bird', 'cat','deer', 'dog', 'frog', 'horse', 'ship', 'truck')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1272,
     "status": "ok",
     "timestamp": 1586181486939,
     "user": {
      "displayName": "sadra barikbein",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiCq0YqNHWfcZyly3I_AMTkvS8_tO8PZOQpGtkK=s64",
      "userId": "07513140462530453530"
     },
     "user_tz": -270
    },
    "id": "axL44Q_ivmV2",
    "outputId": "56a88f49-78f9-4d9a-c0aa-78dd9f6a6307"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cuda:0\n"
     ]
    }
   ],
   "source": [
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "print(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "cxxwF5VvvuCe"
   },
   "outputs": [],
   "source": [
    "num_epoches = 20\n",
    "model=ResidualNetwork()\n",
    "model.to(device)\n",
    "#optimizer=optim.Adam(model.parameters(),lr=0.1,weight_decay=0.3)\n",
    "optimizer = optim.SGD(model.parameters(), lr=0.1, momentum=0.9)\n",
    "criterion = nn.CrossEntropyLoss()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 986120,
     "status": "ok",
     "timestamp": 1586184484190,
     "user": {
      "displayName": "sadra barikbein",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiCq0YqNHWfcZyly3I_AMTkvS8_tO8PZOQpGtkK=s64",
      "userId": "07513140462530453530"
     },
     "user_tz": -270
    },
    "id": "dwdY8LIawIgw",
    "outputId": "6d137ed8-80ee-4242-84fc-8e4bc1a8479f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EPOCH  1:\n",
      "\titeration  100: training_loss = 1.808029, test_loss = 1.535358\n",
      "test_accuracy = 0.522800\n",
      "  >> plane      : 0.448000\n",
      "  >> car        : 0.866000\n",
      "  >> bird       : 0.271000\n",
      "  >> cat        : 0.329000\n",
      "  >> deer       : 0.304000\n",
      "  >> dog        : 0.501000\n",
      "  >> frog       : 0.640000\n",
      "  >> horse      : 0.713000\n",
      "  >> ship       : 0.620000\n",
      "  >> truck      : 0.536000\n",
      "EPOCH  2:\n",
      "\titeration  100: training_loss = 1.174139, test_loss = 1.059011\n",
      "test_accuracy = 0.661100\n",
      "  >> plane      : 0.677000\n",
      "  >> car        : 0.885000\n",
      "  >> bird       : 0.359000\n",
      "  >> cat        : 0.653000\n",
      "  >> deer       : 0.514000\n",
      "  >> dog        : 0.423000\n",
      "  >> frog       : 0.780000\n",
      "  >> horse      : 0.735000\n",
      "  >> ship       : 0.839000\n",
      "  >> truck      : 0.746000\n",
      "EPOCH  3:\n",
      "\titeration  100: training_loss = 0.875948, test_loss = 0.867437\n",
      "test_accuracy = 0.727300\n",
      "  >> plane      : 0.745000\n",
      "  >> car        : 0.899000\n",
      "  >> bird       : 0.501000\n",
      "  >> cat        : 0.697000\n",
      "  >> deer       : 0.644000\n",
      "  >> dog        : 0.481000\n",
      "  >> frog       : 0.848000\n",
      "  >> horse      : 0.763000\n",
      "  >> ship       : 0.846000\n",
      "  >> truck      : 0.849000\n",
      "EPOCH  4:\n",
      "\titeration  100: training_loss = 0.681306, test_loss = 0.710963\n",
      "test_accuracy = 0.757100\n",
      "  >> plane      : 0.728000\n",
      "  >> car        : 0.880000\n",
      "  >> bird       : 0.564000\n",
      "  >> cat        : 0.710000\n",
      "  >> deer       : 0.716000\n",
      "  >> dog        : 0.608000\n",
      "  >> frog       : 0.794000\n",
      "  >> horse      : 0.795000\n",
      "  >> ship       : 0.889000\n",
      "  >> truck      : 0.887000\n",
      "EPOCH  5:\n",
      "\titeration  100: training_loss = 0.557348, test_loss = 0.667391\n",
      "test_accuracy = 0.757000\n",
      "  >> plane      : 0.774000\n",
      "  >> car        : 0.885000\n",
      "  >> bird       : 0.542000\n",
      "  >> cat        : 0.623000\n",
      "  >> deer       : 0.714000\n",
      "  >> dog        : 0.685000\n",
      "  >> frog       : 0.813000\n",
      "  >> horse      : 0.798000\n",
      "  >> ship       : 0.839000\n",
      "  >> truck      : 0.897000\n",
      "EPOCH  6:\n",
      "\titeration  100: training_loss = 0.464217, test_loss = 0.681494\n",
      "test_accuracy = 0.769500\n",
      "  >> plane      : 0.772000\n",
      "  >> car        : 0.922000\n",
      "  >> bird       : 0.588000\n",
      "  >> cat        : 0.596000\n",
      "  >> deer       : 0.762000\n",
      "  >> dog        : 0.692000\n",
      "  >> frog       : 0.824000\n",
      "  >> horse      : 0.825000\n",
      "  >> ship       : 0.870000\n",
      "  >> truck      : 0.844000\n",
      "EPOCH  7:\n",
      "\titeration  100: training_loss = 0.402697, test_loss = 0.670921\n",
      "test_accuracy = 0.756700\n",
      "  >> plane      : 0.707000\n",
      "  >> car        : 0.914000\n",
      "  >> bird       : 0.645000\n",
      "  >> cat        : 0.642000\n",
      "  >> deer       : 0.722000\n",
      "  >> dog        : 0.649000\n",
      "  >> frog       : 0.807000\n",
      "  >> horse      : 0.820000\n",
      "  >> ship       : 0.880000\n",
      "  >> truck      : 0.781000\n",
      "EPOCH  8:\n",
      "\titeration  100: training_loss = 0.343445, test_loss = 0.713275\n",
      "test_accuracy = 0.766400\n",
      "  >> plane      : 0.801000\n",
      "  >> car        : 0.844000\n",
      "  >> bird       : 0.677000\n",
      "  >> cat        : 0.667000\n",
      "  >> deer       : 0.733000\n",
      "  >> dog        : 0.566000\n",
      "  >> frog       : 0.829000\n",
      "  >> horse      : 0.810000\n",
      "  >> ship       : 0.861000\n",
      "  >> truck      : 0.876000\n",
      "EPOCH  9:\n",
      "\titeration  100: training_loss = 0.296253, test_loss = 0.753111\n",
      "test_accuracy = 0.779800\n",
      "  >> plane      : 0.816000\n",
      "  >> car        : 0.878000\n",
      "  >> bird       : 0.699000\n",
      "  >> cat        : 0.668000\n",
      "  >> deer       : 0.751000\n",
      "  >> dog        : 0.602000\n",
      "  >> frog       : 0.853000\n",
      "  >> horse      : 0.804000\n",
      "  >> ship       : 0.874000\n",
      "  >> truck      : 0.853000\n",
      "EPOCH 10:\n",
      "\titeration  100: training_loss = 0.233959, test_loss = 0.805950\n",
      "test_accuracy = 0.780600\n",
      "  >> plane      : 0.750000\n",
      "  >> car        : 0.863000\n",
      "  >> bird       : 0.708000\n",
      "  >> cat        : 0.650000\n",
      "  >> deer       : 0.726000\n",
      "  >> dog        : 0.655000\n",
      "  >> frog       : 0.844000\n",
      "  >> horse      : 0.830000\n",
      "  >> ship       : 0.882000\n",
      "  >> truck      : 0.898000\n",
      "EPOCH 11:\n",
      "\titeration  100: training_loss = 0.197821, test_loss = 0.918351\n",
      "test_accuracy = 0.782700\n",
      "  >> plane      : 0.841000\n",
      "  >> car        : 0.855000\n",
      "  >> bird       : 0.673000\n",
      "  >> cat        : 0.672000\n",
      "  >> deer       : 0.739000\n",
      "  >> dog        : 0.675000\n",
      "  >> frog       : 0.836000\n",
      "  >> horse      : 0.785000\n",
      "  >> ship       : 0.843000\n",
      "  >> truck      : 0.908000\n",
      "EPOCH 12:\n",
      "\titeration  100: training_loss = 0.155092, test_loss = 0.844681\n",
      "test_accuracy = 0.803700\n",
      "  >> plane      : 0.875000\n",
      "  >> car        : 0.874000\n",
      "  >> bird       : 0.654000\n",
      "  >> cat        : 0.674000\n",
      "  >> deer       : 0.753000\n",
      "  >> dog        : 0.739000\n",
      "  >> frog       : 0.818000\n",
      "  >> horse      : 0.852000\n",
      "  >> ship       : 0.892000\n",
      "  >> truck      : 0.906000\n",
      "EPOCH 13:\n",
      "\titeration  100: training_loss = 0.125262, test_loss = 0.843449\n",
      "test_accuracy = 0.802900\n",
      "  >> plane      : 0.860000\n",
      "  >> car        : 0.883000\n",
      "  >> bird       : 0.694000\n",
      "  >> cat        : 0.676000\n",
      "  >> deer       : 0.731000\n",
      "  >> dog        : 0.723000\n",
      "  >> frog       : 0.841000\n",
      "  >> horse      : 0.824000\n",
      "  >> ship       : 0.902000\n",
      "  >> truck      : 0.895000\n",
      "EPOCH 14:\n",
      "\titeration  100: training_loss = 0.099562, test_loss = 0.865179\n",
      "test_accuracy = 0.808400\n",
      "  >> plane      : 0.884000\n",
      "  >> car        : 0.884000\n",
      "  >> bird       : 0.656000\n",
      "  >> cat        : 0.725000\n",
      "  >> deer       : 0.745000\n",
      "  >> dog        : 0.703000\n",
      "  >> frog       : 0.855000\n",
      "  >> horse      : 0.830000\n",
      "  >> ship       : 0.899000\n",
      "  >> truck      : 0.903000\n",
      "EPOCH 15:\n",
      "\titeration  100: training_loss = 0.072966, test_loss = 0.929028\n",
      "test_accuracy = 0.808500\n",
      "  >> plane      : 0.834000\n",
      "  >> car        : 0.894000\n",
      "  >> bird       : 0.739000\n",
      "  >> cat        : 0.656000\n",
      "  >> deer       : 0.729000\n",
      "  >> dog        : 0.753000\n",
      "  >> frog       : 0.856000\n",
      "  >> horse      : 0.829000\n",
      "  >> ship       : 0.910000\n",
      "  >> truck      : 0.885000\n",
      "EPOCH 16:\n",
      "\titeration  100: training_loss = 0.069101, test_loss = 0.932766\n",
      "test_accuracy = 0.806800\n",
      "  >> plane      : 0.811000\n",
      "  >> car        : 0.846000\n",
      "  >> bird       : 0.734000\n",
      "  >> cat        : 0.720000\n",
      "  >> deer       : 0.753000\n",
      "  >> dog        : 0.706000\n",
      "  >> frog       : 0.840000\n",
      "  >> horse      : 0.846000\n",
      "  >> ship       : 0.918000\n",
      "  >> truck      : 0.894000\n",
      "EPOCH 17:\n",
      "\titeration  100: training_loss = 0.055957, test_loss = 0.951800\n",
      "test_accuracy = 0.821000\n",
      "  >> plane      : 0.839000\n",
      "  >> car        : 0.882000\n",
      "  >> bird       : 0.762000\n",
      "  >> cat        : 0.734000\n",
      "  >> deer       : 0.769000\n",
      "  >> dog        : 0.692000\n",
      "  >> frog       : 0.852000\n",
      "  >> horse      : 0.825000\n",
      "  >> ship       : 0.925000\n",
      "  >> truck      : 0.930000\n",
      "EPOCH 18:\n",
      "\titeration  100: training_loss = 0.044153, test_loss = 0.871612\n",
      "test_accuracy = 0.805700\n",
      "  >> plane      : 0.852000\n",
      "  >> car        : 0.863000\n",
      "  >> bird       : 0.701000\n",
      "  >> cat        : 0.691000\n",
      "  >> deer       : 0.795000\n",
      "  >> dog        : 0.683000\n",
      "  >> frog       : 0.844000\n",
      "  >> horse      : 0.840000\n",
      "  >> ship       : 0.909000\n",
      "  >> truck      : 0.879000\n",
      "EPOCH 19:\n",
      "\titeration  100: training_loss = 0.037427, test_loss = 0.990997\n",
      "test_accuracy = 0.827500\n",
      "  >> plane      : 0.891000\n",
      "  >> car        : 0.903000\n",
      "  >> bird       : 0.760000\n",
      "  >> cat        : 0.685000\n",
      "  >> deer       : 0.807000\n",
      "  >> dog        : 0.724000\n",
      "  >> frog       : 0.858000\n",
      "  >> horse      : 0.862000\n",
      "  >> ship       : 0.898000\n",
      "  >> truck      : 0.887000\n",
      "EPOCH 20:\n",
      "\titeration  100: training_loss = 0.033433, test_loss = 0.971912\n",
      "test_accuracy = 0.823500\n",
      "  >> plane      : 0.859000\n",
      "  >> car        : 0.895000\n",
      "  >> bird       : 0.737000\n",
      "  >> cat        : 0.665000\n",
      "  >> deer       : 0.788000\n",
      "  >> dog        : 0.761000\n",
      "  >> frog       : 0.869000\n",
      "  >> horse      : 0.856000\n",
      "  >> ship       : 0.907000\n",
      "  >> truck      : 0.898000\n"
     ]
    }
   ],
   "source": [
    "for epoch in range(num_epoches):\n",
    "        print('EPOCH {:2d}:'.format(epoch + 1))\n",
    "        running_loss = 0.\n",
    "        for i, (x, y) in enumerate(trainloader):\n",
    "            inputs, labels = x.to(device), y.to(device)\n",
    "            optimizer.zero_grad()\n",
    "            out= model(inputs)\n",
    "            loss = criterion(out, labels)\n",
    "            loss.backward()\n",
    "            optimizer.step()  \n",
    "            running_loss += loss.item()\n",
    "            if i % 100 == 99:\n",
    "                test_loss = 0.\n",
    "                with torch.no_grad():\n",
    "                    for data in testloader:\n",
    "                        images, labels = data[0].to(device), data[1].to(device)\n",
    "                        out = model(images)\n",
    "                        loss = F.cross_entropy(out, labels)\n",
    "                        test_loss+=loss.item()\n",
    "                print('\\titeration {:4d}: training_loss = {:5f}, test_loss = {:5f}'.format(i + 1, running_loss/100, test_loss/len(testloader)))\n",
    "                running_loss = 0.\n",
    "        with torch.no_grad():\n",
    "            correct = 0\n",
    "            total = 0\n",
    "            class_correct = [0.] * 10\n",
    "            class_total = [0.] * 10\n",
    "            for data in testloader:\n",
    "                images, labels = data[0].to(device), data[1].to(device)\n",
    "                out= model(images)\n",
    "                _, predicted = torch.max(out.data, 1)\n",
    "                total += labels.size(0)\n",
    "                correct += (predicted == labels).sum().item()                \n",
    "                for i in range(len(labels)):\n",
    "                    class_total[labels[i]]+=1\n",
    "                    if predicted[i]==labels[i]:\n",
    "                        class_correct[labels[i]]+=1\n",
    "        print('test_accuracy = {:5f}'.format(correct / total))\n",
    "        for i in range(10):\n",
    "            print('  >> {:11s}: {:5f}'.format(classes[i], class_correct[i]/class_total[i]))\n",
    "        torch.save(model.state_dict(), './model.pth')\n",
    "        torch.save(optimizer.state_dict(), './optimizer.pth')"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "HW3_Part3.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}