Lab8_ObjectDetectionAndSegmentation.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Lab8_ObjectDetectionAndSegmentation.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "REaV1GblZovs"
      },
      "source": [
        "#Object detection and segmentation\n",
        "In this Lab you will learn about\n",
        "\n",
        "- Fully convolutional networks\n",
        "- Localization\n",
        "- Object detection with a YOLO-like network\n",
        "- Image segmentation with U-Net\n",
        "\n",
        "We will not have time to reproduce exactly the network architectures covered in Lecture 8. Instead this Lab serves is to give your some intuition about fundamental properties of these networks.\n",
        "\n",
        "**Before we start - remember to set runtime to GPU**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n4XtfFsFfW3s"
      },
      "source": [
        "**NOTE:** In case you have trouble running Keras/TensorFlow in Colab, try one of the following:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EQTL2goufaNV"
      },
      "source": [
        "# Try this\n",
        "#!pip install --upgrade tensorflow==1.8.0\n",
        "\n",
        "# ... or this\n",
        "#%tensorflow_version 1.x\n",
        "\n",
        "# Check TensorFlow version\n",
        "#import tensorflow as tf\n",
        "#print(tf.__version__)"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qc8RJkIJaxap"
      },
      "source": [
        "##1. Download the MNIST dataset\n",
        "As usual:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "r-_9ZwQkiGQg"
      },
      "source": [
        "from __future__ import print_function\n",
        "import keras\n",
        "from keras.datasets import mnist\n",
        "from keras import backend as K\n",
        "\n",
        "num_classes = 10\n",
        "\n",
        "# input image dimensions\n",
        "img_rows, img_cols = 28, 28\n",
        "\n",
        "# the data, split between train and test sets\n",
        "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
        "\n",
        "if K.image_data_format() == 'channels_first':\n",
        "    x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)\n",
        "    x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)\n",
        "    input_shape = (1, img_rows, img_cols)\n",
        "else:\n",
        "    x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)\n",
        "    x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)\n",
        "    input_shape = (img_rows, img_cols, 1)\n",
        "\n",
        "# Pre-process inputs\n",
        "x_train = x_train.astype('float32')\n",
        "x_test = x_test.astype('float32')\n",
        "x_train /= 255\n",
        "x_test /= 255\n",
        "\n",
        "# Convert class indices to one-hot vectors\n",
        "y_train = keras.utils.to_categorical(y_train, num_classes)\n",
        "y_test = keras.utils.to_categorical(y_test, num_classes)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IwMozk31fnTL"
      },
      "source": [
        "Print some stats:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NTvEhwcQfqcH"
      },
      "source": [
        "# Input shape: 28 x 28 x 1 = image with one color channel\n",
        "print('input_shape :',input_shape)\n",
        "\n",
        "print('x_train shape:', x_train.shape)\n",
        "print(x_train.shape[0], 'train samples')\n",
        "print(x_test.shape[0], 'test samples')\n",
        "\n",
        "# to_categorical converts class indices to one-hot vectors\n",
        "print('y_train shape:', y_train.shape)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dkOorGdZf4L6"
      },
      "source": [
        "##2. Train a simple CNN classifier\n",
        "Set up CNN:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wvHAmMda0uNE"
      },
      "source": [
        "from __future__ import print_function\n",
        "import keras\n",
        "from keras.layers import Input, Conv2D, MaxPooling2D, Dense, Flatten\n",
        "from keras.models import Model\n",
        "\n",
        "inputs = Input(shape=(28, 28, 1))\n",
        "\n",
        "# Encoder (convolutional base)\n",
        "x = Conv2D(8, kernel_size=(3, 3), padding='same', activation='relu')(inputs)\n",
        "x = MaxPooling2D((2, 2))(x)\n",
        "x = Conv2D(16, kernel_size=(3, 3), padding='same', activation='relu')(x)\n",
        "x = MaxPooling2D((2, 2))(x)\n",
        "x = Conv2D(32, kernel_size=(3, 3), padding='same', activation='relu')(x)\n",
        "encoded = Flatten()(x)\n",
        "\n",
        "# Decoder (2 fully connected layers)\n",
        "x = Dense(64, activation='relu')(encoded)\n",
        "predictions = Dense(num_classes,activation='softmax')(x)\n",
        "\n",
        "# This creates a callable model that includes the Input layer and the prediction layer\n",
        "model = Model(inputs=inputs, outputs=predictions)\n",
        "\n",
        "model.summary()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "44dU5LYWrMXY"
      },
      "source": [
        "Train the model for four epochs:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tB02sLW7mbX6"
      },
      "source": [
        "batch_size = 32\n",
        "epochs = 4\n",
        "\n",
        "model.compile(loss='categorical_crossentropy',\n",
        "              optimizer='SGD',\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "history = model.fit(x_train, y_train,\n",
        "                    batch_size=batch_size,\n",
        "                    epochs=epochs,\n",
        "                    verbose=1,\n",
        "                    validation_data=(x_test, y_test))\n",
        "\n",
        "score = model.evaluate(x_test, y_test, verbose=0)\n",
        "print('Test loss:', score[0])\n",
        "print('Test accuracy:', score[1])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xs9A2gSKhmOM"
      },
      "source": [
        "##3. Task 1: Simple sliding window\n",
        "Sliding window  (i.e., repeating the same operation on an image at many different locations) is a core operation in many computer vision tasks.\n",
        "\n",
        "![alt text](https://miro.medium.com/max/576/1*Mj8WKVKf_RpiAsX3SC1ZdQ.png)\n",
        "\n",
        "To perform sliding window efficiently - and to handle input images of varying shape - most modern CNNs are *fully convolutional*. The term is not uniquely defined, but at its core it addresses the issue that fully connected layers (or dense layers) require that the input vector has a fixed, predetermined shape. There are basically two solutions to this problem:\n",
        "\n",
        "1. Perform some sort of pooling operation on the last feature map of the convolutional encoder, such that the input to the fully connected layers has a fixed shape, regardless of the shape of the input image. For instance, *global average pooling* takes an NxNxC volume and reduces it to a C-dimensional vector by averaging the NxN valus of each channel. (see [Task 2.4 of Lab 7](https://github.com/aivclab/dlcourse/blob/master/Lab7_Solution.ipynb)).\n",
        "2. Convert the fully connected layers into convolutional layers. This allows the network to handle images of arbitrary shape. If the shape of the last feature map is NxNxC, the first step of replacing the fully connected layers with convolutional layers is to perform convolution with K different NxN sized filters to produce an output volume of shape 1x1xK. Performing successive 1x1 convolutions on this volume \"mimics\" the traditional fully connected layers. \n",
        "\n",
        "The main difference between the two approaches above is that in *1.* the output shape is always the same, whereas in *2.* the output shape increases as the shape of the input image increases. This is because *2.* performs a kind of *sliding window* operation. We will implement this later."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nkQUJdrN3UT4"
      },
      "source": [
        "###Create a larger test image to perform sliding window on\n",
        "To motivate why we need fully convolutional networks to perform sliding window effeciently, lets first consider simple sliding window, where each window is run through a CNN.\n",
        "\n",
        "The CNN that we trained above takes input images of size 28 x 28 and produces a 10-dimensional output vector of class probabilities. Now, suppose that the input image is three times as large (i.e., it has size 84 x 84), but the digits have the same absolute scale/shape as before. Let's generate and display such a test image using zero padding:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fTAL7qdze1vA"
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "\n",
        "# Change to select another test image\n",
        "i = 0\n",
        "\n",
        "# Dimensions are [batch_size, height, width, channels]\n",
        "img = np.reshape(x_train[i,:],(1,28,28,1))\n",
        "\n",
        "# Pad with zeros to obtain image of size (28 + 2*padsize) = 84 if padsize = 28\n",
        "padsize = 28\n",
        "img_large = np.pad(img,((0,0),(padsize,padsize),(padsize,padsize),(0,0)))\n",
        "print('Original shape',img.shape)\n",
        "print('Larger image shape',img_large.shape)\n",
        "\n",
        "# Display\n",
        "plt.subplot(1,2,1); plt.imshow(img.squeeze(),cmap='gray'); plt.title('Original (28 x 28)');\n",
        "plt.subplot(1,2,2); plt.imshow(img_large.squeeze(),cmap='gray'); plt.title('Larger image (84 x 84)');"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wGon_HY0e0lT"
      },
      "source": [
        "###Run sliding window\n",
        "We should be able to detect the digit in the larger image (`img_large`) by running our current CNN model over the image in a sliding window fashion. \n",
        "The input image is 84 x 84 pixels, and the network accepts images of shape 28x28. With a stride of 1, how many times can we run the network over the input image along each spatial dimension? Call this number `N`, insert it in the code block below, and run it. \n",
        "\n",
        "Please ask me (Henrik) if the task is unclear, or if you get stuck.\n",
        "\n",
        "**Note** that it will take a while to run the code block below - that's the whole point :-)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "u_RK8Uh7gp-1"
      },
      "source": [
        "import time\n",
        "\n",
        "# Initialize result array\n",
        "N = ???\n",
        "result = np.zeros((N,N,num_classes))\n",
        "\n",
        "# Perform sliding window\n",
        "start = time.time()\n",
        "for x in range(N):\n",
        "  for y in range(N):\n",
        "    cropout = img_large[:,x:x+28,y:y+28,:]\n",
        "    cropout = np.reshape(cropout,(1,28,28,1))\n",
        "    result[x,y,:] = model.predict(cropout)\n",
        "\n",
        "elapsed = time.time() - start\n",
        "print('Elapsed time',elapsed)\n",
        "\n",
        "# Show result\n",
        "plt.figure(figsize=(20,6))\n",
        "for c in range(num_classes):\n",
        "  plt.subplot(1,num_classes,c+1)\n",
        "  plt.imshow(result[:,:,c],vmin=0.,vmax=1.,cmap='gray')\n",
        "  plt.title(str(c))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uhAe0K9ZjOZ4"
      },
      "source": [
        "### Questions\n",
        "1. What are we looking at here? That is, explain what the output images (`result`) show.\n",
        "2. How long time did it take to run sliding window this way?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "obC47RsqjpJb"
      },
      "source": [
        "##4. Task 2: Efficient sliding window with FCN\n",
        "Sliding window as implemented above takes several seconds, even for this tiny image. To make the sliding window operation more efficient, we can convert our model to a fully convolutional network (FCN) by converting the fully connected layers to convolutional layers.\n",
        "\n",
        "In summary, the procedure involves the following steps:\n",
        "\n",
        "1. First set up an FCN equivalent to the CNN that we used above.\n",
        "2. Train the FCN on the 28 x 28 training images (`x_train`).\n",
        "3. Modify the FCN architecture such that it takes input images of arbitrary input shape (and test it on the 84 x 84 image).\n",
        "\n",
        "The latter is a two-step procedure, where we first modify the FCN architecture a little bit (by removing a flatten layer), then copy the weights from the trained network.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kX7DLQeq3eWs"
      },
      "source": [
        "###Step 1 - set up FCN\n",
        "As a first step, convert our fully connected model into an FCN by replacing the fully connected layers with convolution layers. You can do this by filling in the missing code below (see comments to get help):\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0z7HuSBbhnzX"
      },
      "source": [
        "from keras.activations import softmax\n",
        "\n",
        "def depthwiseSoftmax(x):\n",
        "    return softmax(x,axis=3)\n",
        "\n",
        "def FCN(inputs,train_mode=False):\n",
        "  # Encoder (convolutional base)\n",
        "  x = Conv2D(filters=8, kernel_size=(3, 3), padding='same', activation='relu')(inputs)\n",
        "  x = MaxPooling2D((2, 2))(x)\n",
        "  x = Conv2D(filters=16, kernel_size=(3, 3), padding='same', activation='relu')(x)\n",
        "  x = MaxPooling2D((2, 2))(x)\n",
        "  x = Conv2D(filters=32, kernel_size=(3, 3), padding='same', activation='relu')(x)\n",
        "\n",
        "  # 1. At this point x has shape (None, 7, 7, 32)  \n",
        "  x = Conv2D(filters=???, kernel_size=???, padding=???, activation='relu')(x)\n",
        "\n",
        "  # 2. At this point x has shape (None, 1, 1, 64)\n",
        "  predictions = Conv2D(filters=???, kernel_size=???, padding=???, activation=depthwiseSoftmax, padding='same')(x)\n",
        "\n",
        "  # 3. At this point predictions has shape (None, 1, 1, num_classes)\n",
        "\n",
        "  # The flatten operation is required during training only\n",
        "  if train_mode:\n",
        "    predictions = Flatten()(predictions)\n",
        "    # 4. At this point \"predictions\" has shape (None, num_classes)\n",
        "\n",
        "  fcn = Model(inputs=inputs, outputs=predictions)\n",
        "\n",
        "  return fcn\n",
        "\n",
        "fcn = FCN(inputs=inputs,train_mode=True)\n",
        "fcn.summary()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iQyWluqPmIMY"
      },
      "source": [
        "###Step 2 - train FCN\n",
        "Unfortunately, it is not so straight forward to transfer the learned weights from the existing CNN (`model`) to the FCN (`fcn`). So we need to quickly train it:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_7az3y5YkBUk"
      },
      "source": [
        "epochs = 4\n",
        "\n",
        "fcn.compile(loss='categorical_crossentropy',\n",
        "            optimizer='SGD',\n",
        "            metrics=['accuracy'])\n",
        "\n",
        "history = fcn.fit(x_train, y_train,\n",
        "                  batch_size=batch_size,\n",
        "                  epochs=epochs,\n",
        "                  verbose=1,\n",
        "                  validation_data=(x_test, y_test))\n",
        "\n",
        "score = model.evaluate(x_test, y_test, verbose=0)\n",
        "print('Test loss:', score[0])\n",
        "print('Test accuracy:', score[1])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Byntwzvimju7"
      },
      "source": [
        "####Question\n",
        "1. Why is the `predictions = Flatten()(x)` operation requires during training, but not during test?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kSpRrkOxCBzj"
      },
      "source": [
        "###Step 3.1 - make FCN accept images of arbitrary shape\n",
        "Now, that we have trained our FCN, we can modify it to take input images of arbitrary shape. This is a two-step procedure. First set input height/width to `None` and remove the `Flatten` operation by setting `train_mode=False`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CLSuvt1enmqZ"
      },
      "source": [
        "# Tell Keras that height/width can be arbitrary\n",
        "large_input_shape = ((None,None,1))\n",
        "large_inputs = Input(large_input_shape)\n",
        "\n",
        "fcn_large = FCN(inputs=large_inputs,train_mode=False)\n",
        "fcn_large.summary()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rG3fhdF1ClvK"
      },
      "source": [
        "Next, copy the weights from the trained model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ysgdheE3oNGp",
        "outputId": "6ef57598-f124-40f6-bb79-a2d96d6f3b05",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "# copy weights from one model to another model    \n",
        "# tested in Keras 1.x    \n",
        "def copyModel2Model(model_source,model_target):        \n",
        "    for l_tg,l_sr in zip(model_target.layers,model_source.layers):\n",
        "        wk0 = l_sr.get_weights()\n",
        "        l_tg.set_weights(wk0)\n",
        "    print(\"model source was copied into model target\")\n",
        "\n",
        "copyModel2Model(fcn,fcn_large)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "model source was copied into model target\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u-pK5gHDC4IZ"
      },
      "source": [
        "Let's verify that we get the same results with `fcn` and `fcn_large`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oAvE_d3jpuSZ"
      },
      "source": [
        "# FCN that always expects 28 x 28 input\n",
        "result = fcn.predict(img)\n",
        "print('result',result)\n",
        "print('result.shape',result.shape)\n",
        "\n",
        "# FCN that accepts arbitrary input shape (but fed with 28x28 image here)\n",
        "result_large = fcn_large.predict(img)\n",
        "print(result_large)\n",
        "print('result_large.shape',result_large.shape)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dSE5tkcYF5Ha"
      },
      "source": [
        "####Questions:\n",
        "1. Is there any sliding window going on here (when running `fcn_large`)?\n",
        "2. Why are the output shapes `result.shape` and `result_large.shape` different?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Dq6B7cQF4BYK"
      },
      "source": [
        "###Test speed (FCN vs simple slinding window)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T7u1FXbwDvBW"
      },
      "source": [
        "Finally, let's verify that `fcn_large` performs sliding window much faster than the simple sliding window above by processing the 84 x 84 image, `img_large`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zoJeYn0ODV8y"
      },
      "source": [
        "start = time.time()\n",
        "result_large = fcn_large.predict(img_large)\n",
        "elapsed = time.time() - start\n",
        "print(result_large.shape)\n",
        "print(elapsed)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2XHDZ-Jpmb9-"
      },
      "source": [
        "Display the results:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9CqDXqgDrI-L"
      },
      "source": [
        "# Show input image\n",
        "plt.figure()\n",
        "plt.imshow(img_large.squeeze(),cmap='gray')\n",
        "\n",
        "# Show result\n",
        "plt.figure(figsize=(20,6))\n",
        "for c in range(num_classes):\n",
        "  plt.subplot(1,num_classes,c+1)\n",
        "  plt.imshow(result_large[:,:,:,c].squeeze(),vmin=0.,vmax=1.,cmap='gray')\n",
        "  plt.title(str(c))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q5J15TLJGtBj"
      },
      "source": [
        "###Questions:\n",
        "1. With simple sliding window, the output shape of `results` was NxNx10 = 56x56x10. However, when using FCN for sliding window, the output shape is 15x15x10. Why?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mapzf8Y0Hv87"
      },
      "source": [
        "##5. Task 3: Simple localization\n",
        "The goal of localization is to find the bounding box of a single object in an image. This is a regression problem, where we seek to predict the corners of the bounding box (or eqivalently one corner + height and width of the bbox). Either way, the output of the regression is 4 numbers.\n",
        "\n",
        "Localization can be performed in the *class agnostic* way, where we use the same regressor for all classes, or the *class specific* way. In the latter case, the output of the regression is C x 4 dimensional, where C is the number of classes.\n",
        "\n",
        "Here we will focus on the simpler class agnostic approach."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WY27xHHl4QgG"
      },
      "source": [
        "###Train and test data\n",
        "First, let's create train and test data for this task."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mKwFuZyXH06g"
      },
      "source": [
        "num_train = x_train.shape[0]\n",
        "y_train_loc = np.zeros((num_train,4)) # Bounding box corners\n",
        "for i in range(num_train):\n",
        "  \n",
        "  # Bounding box corners coordinates\n",
        "  tmp = x_train[i,:,:,:]\n",
        "  ix = np.where(tmp>0.1)\n",
        "  rows = ix[0]\n",
        "  cols = ix[1]  \n",
        "  y_train_loc[i,0] = np.min(rows) # Upper left corner - row coordinate\n",
        "  y_train_loc[i,1] = np.min(cols) # Upper left corner - column coordinate\n",
        "  y_train_loc[i,2] = np.max(rows) # Lower right corner - row coordinate\n",
        "  y_train_loc[i,3] = np.max(cols) # Lower right corner - column coordinate\n",
        "\n",
        "num_test = x_test.shape[0]\n",
        "y_test_loc = np.zeros((num_test,4))\n",
        "for i in range(num_test):\n",
        "  \n",
        "  # Bounding box corners coordinates\n",
        "  tmp = x_test[i,:,:,:]\n",
        "  ix = np.where(tmp>0.1)\n",
        "  rows = ix[0]\n",
        "  cols = ix[1]  \n",
        "  y_test_loc[i,0] = np.min(rows)\n",
        "  y_test_loc[i,1] = np.min(cols)\n",
        "  y_test_loc[i,2] = np.max(rows)\n",
        "  y_test_loc[i,3] = np.max(cols)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0OkxIsIELPks"
      },
      "source": [
        "Show some examples:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5a4rlUSmIoQH"
      },
      "source": [
        "import cv2\n",
        "plt.figure(figsize=(20,4))\n",
        "for i in range(10):\n",
        "  img = np.tile(x_train[i,:,:,:],(1,1,3))\n",
        "  xmin = int(y_train_loc[i,0]) # row\n",
        "  ymin = int(y_train_loc[i,1]) # col\n",
        "  xmax = int(y_train_loc[i,2]) # row\n",
        "  ymax = int(y_train_loc[i,3]) # col\n",
        "  cv2.rectangle(img,(ymin,xmin),(ymax,xmax),(0,1,0),1)\n",
        "\n",
        "  ax = plt.subplot(2,5,i+1)\n",
        "  plt.imshow(img)\n",
        "  ax.get_xaxis().set_visible(False)\n",
        "  ax.get_yaxis().set_visible(False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tmKcaOfuncAP"
      },
      "source": [
        "###Implement Localization network\n",
        "Your task is to implement and train a CNN that predicts the bounding box coordinates: `(ymin,xmin),(ymax,xmax)`. You can use the CNN architecture that we used and modify the last layer only. It should output 4 real-valued numbers.\n",
        "\n",
        "**Note:** the last layer has no activation function. Why? Because this is a regression problem, where the output values are not necessarily limited to stay within a pre-deterrmined range."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Qp7eoL-GNEcY"
      },
      "source": [
        "inputs = Input(shape=(28, 28, 1))\n",
        "\n",
        "# Encoder (convolutional base)\n",
        "x = Conv2D(8, kernel_size=(3, 3), padding='same', activation='relu')(inputs)\n",
        "x = MaxPooling2D((2, 2))(x)\n",
        "x = Conv2D(16, kernel_size=(3, 3), padding='same', activation='relu')(x)\n",
        "x = MaxPooling2D((2, 2))(x)\n",
        "x = Conv2D(32, kernel_size=(3, 3), padding='same', activation='relu')(x)\n",
        "encoded = Flatten()(x)\n",
        "\n",
        "# Decoder (2 fully connected layers)\n",
        "x = Dense(64, activation='relu')(encoded)\n",
        "predictions = Dense(???)(x)\n",
        "\n",
        "# This creates a callable model that includes the Input layer and the prediction layer\n",
        "localizer = Model(inputs=inputs, outputs=predictions)\n",
        "\n",
        "localizer.summary()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "icAs0SMroy7j"
      },
      "source": [
        "###Training\n",
        "Recall that localization is a regression problem and not a classification problem. For regression, we usually want to minimize the L2 loss or the *mean squared error* (mse) as it is called in Keras. The procedure for training the localizer is more or less the same as for the classifier, excpet that we need to change the loss to `mse`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GxC7-G6hNjvf"
      },
      "source": [
        "batch_size = 32\n",
        "epochs = 4\n",
        "\n",
        "localizer.compile(loss='mse',\n",
        "                  optimizer='Adam')\n",
        "\n",
        "history = localizer.fit(x_train, y_train_loc,\n",
        "                    batch_size=batch_size,\n",
        "                    epochs=epochs,\n",
        "                    verbose=1,\n",
        "                    validation_data=(x_test, y_test_loc))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E5kwfG5KprU7"
      },
      "source": [
        "###Model evaluation\n",
        "Display test results (do not expect perfect results...):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hBfFvfQ8OWTw"
      },
      "source": [
        "# Predict bounding boxes for one test batch\n",
        "result = localizer.predict(x_test[0:batch_size,:])\n",
        "\n",
        "plt.figure(figsize=(20,4))\n",
        "for i in range(10):\n",
        "\n",
        "  img = np.tile(x_test[i,:,:,:],(1,1,3))\n",
        "  xmin = int(result[i,0]) # row\n",
        "  ymin = int(result[i,1]) # col\n",
        "  xmax = int(result[i,2]) # row\n",
        "  ymax = int(result[i,3]) # col\n",
        "  cv2.rectangle(img,(ymin,xmin),(ymax,xmax),(0,1,0),1)\n",
        "\n",
        "  ax = plt.subplot(2,5,i+1)\n",
        "  plt.imshow(img)\n",
        "  ax.get_xaxis().set_visible(False)\n",
        "  ax.get_yaxis().set_visible(False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NV6vSV3WCQZd"
      },
      "source": [
        "##6. Task 4: Object detection\n",
        "Classification + Localization predicts one class label and one bounding box per image. This is problematic if the image contains multiple objects. Object detection is about detecting and classifying multiple objects in images. Object detection networks output the corner coordinates of the bounding box of each detect object, along with a class label.\n",
        "\n",
        "![alt text](https://miro.medium.com/max/576/1*Mj8WKVKf_RpiAsX3SC1ZdQ.png)\n",
        "\n",
        "There are many ways to implement object detection with CNNs. You may want to take a look at this 3-part tutorial:\n",
        "- [A Step-by-Step Introduction to the Basic Object Detection Algorithms (Part 1)](https://www.analyticsvidhya.com/blog/2018/10/a-step-by-step-introduction-to-the-basic-object-detection-algorithms-part-1/)\n",
        "- [A Practical Implementation of the Faster R-CNN Algorithm for Object Detection (Part 2 – with Python codes)](https://www.analyticsvidhya.com/blog/2018/11/implementation-faster-r-cnn-python-object-detection/)\n",
        "- [A Practical Guide to Object Detection using the Popular YOLO Framework – Part III (with Python codes)](https://www.analyticsvidhya.com/blog/2018/12/practical-guide-object-detection-yolo-framewor-python/?utm_source=blog&utm_medium=implementation-faster-r-cnn-python-object-detection)\n",
        "\n",
        "The R-CNN family of techniques we saw in Lecture 8 primarily use regions to localize the objects within the image. In short\n",
        "- **R-CNN** feeds each region proposal through a CNN to predict the class and bounding box, which is really slow\n",
        "- **Fast R-CNN** speeds up R-CNN by running the image through a fully convolutional CNN once, then extracts regions from the resulting feature volume to predict classes and bounding boxes.\n",
        "- **Faster R-CNN** speeds up Fast R-CNN by integrating the region proposal step into the network (instead of running selective search).\n",
        "\n",
        "The R-CNN family of techniques does not look at the entire image, only at the parts of the images which have a higher chance of containing an object. The **YOLO** framework (You Only Look Once) on the other hand, deals with object detection in a different way. It takes the entire image in a single instance and predicts the bounding box coordinates and class probabilities for these boxes. The biggest advantage of using YOLO is its superb speed – it’s incredibly fast and can process 45 frames per second. YOLO also understands generalized object representation.\n",
        "\n",
        "![alt text](https://miro.medium.com/max/1152/1*m8p5lhWdFDdapEFa2zUtIA.jpeg)\n",
        "\n",
        "The basic idea of our simple YOLO-like object detector below is as follows:\n",
        "\n",
        "- The input image is divided into a grid (7-by-7 in the paper).\n",
        "- Each grid cell can be the center of one object, or no object. The object can be bigger or smaller than the cell, it doesn't matter; the important thing to consider is whether a cell represents the center of an object or not.\n",
        "- If a cell is the center of an object, we want our model to know during training (and predict it at test time). For this purpose, we introduce a concept called *confidence*: The confidence is 1 if the cell represents an objects, and otherwise it is 0. In the figure above, the confidence - as predicted by the model - is represented by the thickness of the black boxes in the top-most image.\n",
        "- If the confidence is 1, it means there is an object, so we want to predict its bounding box (relative to the center of the grid cell).\n",
        "- Also, we want to predict the class label of the object. To be more precise, we want our model to predict the class probabilities (for all classes).\n",
        "\n",
        "**Example:**\n",
        "If the input image has shape 64x64, the output of the YOLO-like model will have shape 2x2x15:\n",
        "- 1 output per cell for the confidence (is there an object or not?)\n",
        "- 4 outputs per cell corresponding to the upper left and lower right coordinates of the bounding box\n",
        "- 10 outputs per cell corresponding to the class probabalities (assuming we have 10 classes).\n",
        "\n",
        "This totals 15 outputs per cell.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rHkjYyA09Slk"
      },
      "source": [
        "###Set up YOLO-like network\n",
        "Your first task is to fill in the empty spots below (marked with ???), then run the code block to set up the object detection network."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DBaW6otN9dHO"
      },
      "source": [
        "from keras.layers import concatenate\n",
        "from keras.activations import softmax\n",
        "\n",
        "def depthwiseSoftmax(x):\n",
        "    return softmax(x,axis=3)\n",
        "\n",
        "input_img = Input(shape=(64, 64, 1))\n",
        "\n",
        "x = Conv2D(8, 33, activation='relu', padding='same')(input_img) #nb_filter, nb_row, nb_col\n",
        "x = MaxPooling2D((2, 2), padding='same')(x)\n",
        "x = Conv2D(16, 3, activation='relu', padding='same')(x)\n",
        "x = MaxPooling2D((2, 2), padding='same')(x)\n",
        "x = Conv2D(32, 3, activation='relu', padding='same')(x)\n",
        "encoded = MaxPooling2D((2, 2), padding='same')(x)\n",
        "\n",
        "# Notice that all three outputs use the same encoder\n",
        "\n",
        "# 1. This predicts whether there is an object in a cell or not\n",
        "x = Conv2D(4, 3, activation='relu', padding='same')(encoded)\n",
        "x = MaxPooling2D((2, 2), padding='same')(x)\n",
        "x = Conv2D(4, 3, activation='relu', padding='same')(x)\n",
        "x = MaxPooling2D((2, 2), padding='same')(x)\n",
        "confidence = Conv2D(???, 1, activation='sigmoid', padding='same')(x)\n",
        "\n",
        "# 2. This predicts the bounding box coordinates for each cell\n",
        "x = Conv2D(4, 3, activation='relu', padding='same')(encoded)\n",
        "x = MaxPooling2D((2, 2), padding='same')(x)\n",
        "x = Conv2D(4, 3, activation='relu', padding='same')(x)\n",
        "x = MaxPooling2D((2, 2), padding='same')(x)\n",
        "box = Conv2D(???, 1, padding='same')(x)\n",
        "\n",
        "# 3. This predicts the class probabilities for each cell\n",
        "x = Conv2D(16, 3, activation='relu', padding='same')(encoded)\n",
        "x = MaxPooling2D((2, 2), padding='same')(x)\n",
        "x = Conv2D(16, 3, activation='relu', padding='same')(x)\n",
        "x = MaxPooling2D((2, 2), padding='same')(x)\n",
        "classes = Conv2D(???, 1, activation=depthwiseSoftmax, padding='same')(x)\n",
        "\n",
        "# Merge output\n",
        "merged = concatenate([confidence, box, classes])\n",
        "\n",
        "objdet = Model(input_img, merged)\n",
        "objdet.compile(optimizer='Adam', loss='mse')\n",
        "\n",
        "objdet.summary()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VtLlialYGZVm"
      },
      "source": [
        "###Training data\n",
        "Let's generate some training data for our object detector.\n",
        "\n",
        "The training images (`x_train_obj`) will be 64x64, where two of the four quadrants will contain one handwritten digit. This just serves to illustrate that we can teach a network to detect and classify more than one digit per input image.\n",
        "\n",
        "The output is (`y_train_obj`) is 2x2x15 as explained above:\n",
        "- 1 output per cell for the confidence (is there an object or not?)\n",
        "- 4 outputs per cell corresponding to the upper left and lower right coordinates of the bounding box\n",
        "- 10 outputs per cell corresponding to the class probabalities (assuming we have 10 classes).\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "s8EGlKP8Gd28"
      },
      "source": [
        "x_train_obj = np.zeros((5000,64,64,1))\n",
        "y_train_obj = np.zeros((5000,2,2,15))\n",
        "\n",
        "for i in range(5000):\n",
        "  \n",
        "  ## 1 \n",
        "  q = np.random.randint(0,2) # 1st or 2nd image quadrant?\n",
        "  \n",
        "  # Random image\n",
        "  rand_ix = np.random.randint(0,x_train.shape[0])\n",
        "  x_off_start = np.random.randint(0,5)\n",
        "  y_off_start = np.random.randint(0,5)\n",
        "  x_train_obj[i,0+x_off_start:28+x_off_start,32*q+y_off_start:32*q+28+y_off_start,:] = x_train[rand_ix,:,:,:]\n",
        "  \n",
        "  # Set confidence to 1\n",
        "  y_train_obj[i,0,q,0] = 1\n",
        "\n",
        "  # Class label\n",
        "  label = np.argmax(y_train[rand_ix,:])\n",
        "  y_train_obj[i,0,q,5+label] = 1\n",
        "\n",
        "  # Bounding box corners coordinates\n",
        "  tmp = x_train[rand_ix,:,:,:]\n",
        "  ix = np.where(tmp>0.1)\n",
        "  rows = ix[0]\n",
        "  cols = ix[1]  \n",
        "  y_train_obj[i,0,q,1] = np.min(rows) + x_off_start\n",
        "  y_train_obj[i,0,q,2] = np.min(cols) + y_off_start\n",
        "  y_train_obj[i,0,q,3] = np.max(rows) + x_off_start\n",
        "  y_train_obj[i,0,q,4] = np.max(cols) + y_off_start\n",
        "\n",
        "  ## 2\n",
        "  q = np.random.randint(0,2) # 3rd or 4rd image quadrant?\n",
        "  \n",
        "  # Random image\n",
        "  rand_ix = np.random.randint(0,x_train.shape[0])\n",
        "  x_off_start = np.random.randint(0,5)\n",
        "  y_off_start = np.random.randint(0,5)\n",
        "  x_train_obj[i,32+x_off_start:32+28+x_off_start,32*q+y_off_start:32*q+28+y_off_start,:] = x_train[rand_ix,:,:,:]\n",
        "  \n",
        "  # Set confidence to 1\n",
        "  y_train_obj[i,1,q,0] = 1\n",
        "\n",
        "  # Class label\n",
        "  label = np.argmax(y_train[rand_ix,:])\n",
        "  y_train_obj[i,1,q,5+label] = 1\n",
        "\n",
        "  # Bounding box corners coordinates\n",
        "  tmp = x_train[rand_ix,:,:,:]\n",
        "  ix = np.where(tmp>0.1)\n",
        "  rows = ix[0]\n",
        "  cols = ix[1]  \n",
        "  y_train_obj[i,1,q,1] = np.min(rows) + x_off_start\n",
        "  y_train_obj[i,1,q,2] = np.min(cols) + y_off_start\n",
        "  y_train_obj[i,1,q,3] = np.max(rows) + x_off_start\n",
        "  y_train_obj[i,1,q,4] = np.max(cols) + y_off_start"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DLAKnAV0KkSC"
      },
      "source": [
        "Display example outputs"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3Usjz_hbI7k4"
      },
      "source": [
        "plt.figure(figsize=(20,4))\n",
        "for k in range(10):\n",
        "  result = np.tile(x_train_obj[k,:,:,:],(1,1,3))\n",
        "\n",
        "  for i in range(2):\n",
        "    for j in range(2):\n",
        "      object_present = np.round(y_train_obj[k,i,j,0])\n",
        "      if object_present:\n",
        "        class_index = np.argmax(y_train_obj[k,i,j,5:])\n",
        "        xmin = int(y_train_obj[k,i,j,1] + i*32) # row\n",
        "        ymin = int(y_train_obj[k,i,j,2] + j*32) # col\n",
        "        xmax = int(y_train_obj[k,i,j,3] + i*32) # row\n",
        "        ymax = int(y_train_obj[k,i,j,4] + j*32) # col\n",
        "        cv2.rectangle(result,(ymin,xmin),(ymax,xmax),(0,255,0),1)\n",
        "        cv2.putText(result, str(class_index), (ymin-10, xmin+10), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0),lineType=cv2.LINE_AA)\n",
        "  ax = plt.subplot(2,5,k+1)\n",
        "  plt.imshow(result)\n",
        "  ax.get_xaxis().set_visible(False)\n",
        "  ax.get_yaxis().set_visible(False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q3x93OVGOLvO"
      },
      "source": [
        "###Training\n",
        "We are ready to start training the model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "osyiFWyhWUri"
      },
      "source": [
        "objdet.fit(x_train_obj, y_train_obj, epochs=50, batch_size=128,shuffle=True,verbose=1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TveS0ULsOR75"
      },
      "source": [
        "###Model evaluation\n",
        "The code below shows displays the results for 10 example images. For each cell in each image, it predicts the confidence, the bounding box and the class label.\n",
        "\n",
        "However, the confidence is not used for anything, so all boxes and class labels are shown, regardless of whethter the confidence is high or low.\n",
        "\n",
        "**Your task** is to fix the code, such that bounding boxes and class labels are only shown if the model is confident that there is on object in a given cell. Fill in the `???` (there are multiple solutions)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c1kETljRYh7f"
      },
      "source": [
        "out = objdet.predict(x_train_obj[0:10,:,:,:])\n",
        "\n",
        "plt.figure(figsize=(20,4))\n",
        "for k in range(10):\n",
        "  result = np.tile(x_train_obj[k,:,:,:],(1,1,3))\n",
        "\n",
        "  for i in range(2):\n",
        "    for j in range(2):\n",
        "      confidence = out[k,i,j,0]\n",
        "      object_present = ???\n",
        "      if object_present:\n",
        "        class_index = np.argmax(out[k,i,j,5:])\n",
        "        xmin = int(out[k,i,j,1] + i*32) # row\n",
        "        ymin = int(out[k,i,j,2] + j*32) # col\n",
        "        xmax = int(out[k,i,j,3] + i*32) # row\n",
        "        ymax = int(out[k,i,j,4] + j*32) # col\n",
        "        cv2.rectangle(result,(ymin,xmin),(ymax,xmax),(0,255,0),1)\n",
        "        cv2.putText(result, str(class_index), (ymin-10, xmin+10), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0),lineType=cv2.LINE_AA)\n",
        "  ax = plt.subplot(2,5,k+1)\n",
        "  plt.imshow(result)\n",
        "  ax.get_xaxis().set_visible(False)\n",
        "  ax.get_yaxis().set_visible(False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sK3z-27PBVLn"
      },
      "source": [
        "**Note:** Do not expect perfect results. Our object detection network is over-simplified compared to state-of-the-art. Or more precisely, the loss function is far from ideal, so we are optimizing *the wrong objective*, so to say."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QJzrjPFn455A"
      },
      "source": [
        "##7. Task 5: Image segmentation with U-Net"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_zhuKLxG6wPG"
      },
      "source": [
        "###Representing the task\n",
        "Image segmentation is a form of image-to-image transformation. It outputs a softmax classification per pixel. So if the input image has size 64x64, and there are 10 classes, the output will have shape 64x64x10. That is, for each pixel the network outputs a vector of class probabilities:\n",
        "\n",
        "![alt text](https://miro.medium.com/max/498/1*P1ooLjeSwhxeJGyFawCvaQ.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iWU_knJFHgtF"
      },
      "source": [
        "###Training data\n",
        "We will create training images (x_train_seg) of size 64x64, where two of the four quadrants will contain one handwritten digit. This is to illustrate that we can teach a network to identify and segment more than one digit per input image.\n",
        "\n",
        "The target output (`y_train_seg`) will be 64x64x10, with a one-hot vector for each pixel indicating the correct class."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5jS8Dwk6H2Qr"
      },
      "source": [
        "x_train_seg = np.zeros((5000,64,64,1))\n",
        "y_train_seg = np.zeros((5000,64,64,10))\n",
        "\n",
        "for i in range(5000):\n",
        "  \n",
        "  ## 1\n",
        "  q = np.random.randint(0,2) # 1st or 2nd image quadrant?\n",
        "  rand_ix = np.random.randint(0,x_train.shape[0])\n",
        "  x_off_start = np.random.randint(0,5)\n",
        "  y_off_start = np.random.randint(0,5)\n",
        "  x_train_seg[i,0+x_off_start:28+x_off_start,32*q+y_off_start:32*q+28+y_off_start,:] = x_train[rand_ix,:,:,:]\n",
        "\n",
        "  # Mask\n",
        "  tmp = x_train[rand_ix,:,:,:]\n",
        "  ix = np.where(tmp>0.1)\n",
        "  tmp = np.zeros(tmp.shape)\n",
        "  tmp[ix] = 1\n",
        "  label = np.argmax(y_train[rand_ix,:])\n",
        "  y_train_seg[i,0+x_off_start:28+x_off_start,32*q+y_off_start:32*q+28+y_off_start,label] = tmp.squeeze()\n",
        "  \n",
        "  ## 2\n",
        "  q = np.random.randint(0,2) #  3rd or 4th image quadrant?\n",
        "  rand_ix = np.random.randint(0,x_train.shape[0])\n",
        "  x_off_start = np.random.randint(0,5)\n",
        "  y_off_start = np.random.randint(0,5)\n",
        "  x_train_seg[i,32+x_off_start:32+28+x_off_start,32*q+y_off_start:32*q+28+y_off_start,:] = x_train[rand_ix,:,:,:]\n",
        "\n",
        "  # Mask\n",
        "  tmp = x_train[rand_ix,:,:,:]\n",
        "  ix = np.where(tmp>0.1)\n",
        "  tmp = np.zeros(tmp.shape)\n",
        "  tmp[ix] = 1\n",
        "  label = np.argmax(y_train[rand_ix,:])\n",
        "  y_train_seg[i,32+x_off_start:32+28+x_off_start,32*q+y_off_start:32*q+28+y_off_start,label] = tmp.squeeze()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eDCApMqM7Sqj"
      },
      "source": [
        "Show examples:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sNh7RfpjM9rl"
      },
      "source": [
        "# Show examples of input to output mappings\n",
        "for ex in range(5):\n",
        "  plt.figure(figsize=(20,6))\n",
        "  rand_ix = np.random.randint(0,5000)\n",
        "  ax = plt.subplot(1,11,1)\n",
        "  plt.imshow(x_train_seg[rand_ix,:,:,:].squeeze())\n",
        "  plt.title('Input image')\n",
        "  for i in range(10):\n",
        "    ax = plt.subplot(1,11,i+2)\n",
        "    plt.imshow(y_train_seg[rand_ix,:,:,i].squeeze())\n",
        "    plt.title(\"Out class \"+str(i))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rvyONqJ27jSl"
      },
      "source": [
        "**Your task** is to implement a function `show_segmentation` that takes a 64x64x10 volume of class probabilities as input (e.g., `y_train_seg`) and displays a 64x64 image, where the pixel intensity at a given location is the class index with the highest probability."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "22kPniSl8Mh-",
        "outputId": "23e13e84-5401-4bc6-e10f-616dd2c69566",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 203
        }
      },
      "source": [
        "def show_segmentation(x):\n",
        "  plt.imshow(???,vmin=0,vmax=9)\n",
        "  \n",
        "plt.subplot(121);\n",
        "plt.imshow(x_train_seg[0,:,:,:].squeeze(),cmap='gray')\n",
        "plt.subplot(122)\n",
        "show_segmentation(y_train_seg[0,:,:,:].squeeze())"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAC6CAYAAAC3HRZZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAATx0lEQVR4nO3dfZBU5ZXH8e8ReY0SIGYpBCpINFKm\nTHSLRFitTQSzhSy+K2Isg4qOf+gWwZCgkioraoGQKsnb1upEomhlF1mMC2qIaxBqY2KBoDEqiiCG\nMAQlsQBf/hAHz/7R916amR7mdve9t/sZfp8qam4/t7vvcTweD899eczdERGR8BzV6ABERKQ2KuAi\nIoFSARcRCZQKuIhIoFTARUQCpQIuIhKougq4mU0ys81mttXMbskqKJFGU25LCKzW68DNrBfwBvAN\noA14HrjC3TdlF55I8ZTbEop6OvCvAlvdfZu77weWAhdkE5ZIQym3JQhH1/HZ4cCOstdtwBmH+4CZ\n6bZPyZW7WwZfU1Vu97G+3o9PZXBYkcreZ8/f3f2zHcfrKeCpmFkL0JL3cUSKVJ7X/RjAGTaxwRFJ\nT/ZbX7690ng9Uyg7gZFlr0dEY4dw91Z3H+vuY+s4lkiRus3t8rzuTd9CgxOJ1VPAnwdOMrMTzKwP\nMA1YmU1YIg2l3JYg1DyF4u7tZnYT8BTQC/iFu7+aWWQiDaLcllDUNQfu7r8Gfp1RLCJNQ7ktIdCd\nmCIigVIBFxEJlAq4iEigVMBFRAKlAi4iEigVcBGRQKmAi4gESgVcRCRQKuAiIoFSARcRCZQKuIhI\noFTARUQCpQIuIhIoFXARkUCpgIuIBKrbAm5mvzCz3Wb2StnYEDN72sy2RD8H5xumSPaU2xK6NB34\ng8CkDmO3AKvd/SRgdfRaJDQPotyWgHW7Io+7/5+ZjeowfAHw9Wh7CbAWmJNhXNJEBgwYAMDtt9+e\njL3++usAPPjgg8mYu6f+zsGDDza255xzDgCPPfZYMtbe3l5TrNVQbgvA9h/8U6exz93+h8y+N4vv\n6kqtc+BD3X1XtP02MDSjeEQaTbktwahrTUwAd3cz67L1MrMWoKXe40jjzJ49G4Dvfve7ydi+ffsA\nWL58eTL2/vvvp/7O73//+8n2rFmzAHj11YPrBp966qm1BZuhw+V2eV73Y0ChcUk2KnXeeXxn+VjW\n3XitHfg7ZjYMIPq5u6s3unuru49197E1HkukSKlyuzyve9O30ABFYrUW8JXA9Gh7OrAim3BEGk65\nLcHodgrFzP6L0kmd48ysDbgduBtYZmYzgO3A1DyDlMb42te+BsDll18OwJ49e5J9l1xyCVDdtEm5\nz3/+853G3njjjZq+q1bK7SNTHlMnjZLmKpQrutg1MeNYRAql3JbQ1X0SU3qGPn36ADBv3rxk7Kab\nbgLAzAC49957k31r166t6TjXXHMNAOeff34yduDAAQBWrNBsheQjbddd60nGRnX1upVeRCRQ6sAF\ngOOPPx6Am2++ORnbsmULANdeey0Av//972v+/mOPPRaAuXPndtq3dOlSAB566KGav1+kkjSdcZ43\n2uT9/erARUQCpQIuIhIoTaEcweLLBOHQZ5rE6p06iZ+hAvDwww8DMHr0aAA+/PDDZN+iRYtq+n6R\nSvI+YVnLsfKiDlxEJFDqwI8g8eWA1113HQA//OEPk31HHVX6f/nMmTOTsfXr19d0nL59S7eWjxo1\nKhkrv2wQ4Mknn0y2X3jhhZqOI1KuqM67yA6/O+rARUQCpQIuIhIoTaEcQeKTkvfddx9w6LNNpkyZ\nAsCzzz7b6XPDhw8HYMSIEZ329e/fP9n+yle+AsBFF10EwLhx47qMZd26dVXFLtKVWq/1bvQJyCyo\nAxcRCZQ68B5q4MCBACxevDgZO++88w55z09+8pNkO+6uK3XG8RMH45OTAG+99RYAra2tyVjcjZ98\n8sldxhXf3fnII4+k+KcQqaza7rkndNuVqAMXEQmUOvAeav78+cDB53ZXEi+VBvDBBx8A8MwzzyRj\nd955JwDbt28HDj41EGD//v2dvu/ll18G4LLLLgMOXbg4dsUVpSe4/vWvf03xTyFyqCw76Wov82uG\n56p01G0HbmYjzWyNmW0ys1fNbGY0PsTMnjazLdHPzv+1ijQx5baELs0USjvwHXc/BRgH3GhmpwC3\nAKvd/SRgdfRaJCTKbQlamhV5dgG7ou33zew1YDhwAaXlqACWAGuBOblEKVWLnzlSzr20wPof/lD6\na95Pf/rTZN+yZcvqPmZ8eeIXv/jFTvviFefffPPNuo+TFeV2z1fU806KnjqJVTUHbmajgNOBdcDQ\n6D8AgLeBoV18pgVoqT1EkfxVm9vled2PAR13ixQidQE3s2OAR4Fvu/t78XM1ANzdzcwrfc7dW4HW\n6Dsqvkeyd/311wNw3HHHJWNxB/7SSy8VHk+8BNu+ffsKP3Z3asnt8rweaEOU102gUV1wI6W6jNDM\nelNK8F+6+6+i4XfMbFi0fxiwO58QRfKj3JaQpbkKxYDFwGvufk/ZrpXA9Gh7OqAVaSUoym0JXZop\nlDOBq4CXzeyP0dhtwN3AMjObAWwHpuYTotSira3tkJ+NsHPnzmR71qxZDYvjMJTbgWm2aZJGx5Pm\nKpRnAeti98RswxEpjnJbQqdb6SUzq1atYtWqVRX3tbe3097eXnBEIj2bCriISKD0LBTJTMebdG67\n7bYGRSJyZFAHLiISKBVwEZFAaQpFMhMv8jB58mQAnnrqqUaGI5KLRl86WE4duIhIoNSBS2biG3fK\nb+ARCV0zddwdqQMXEQmUCriISKBUwEVEAqUCLiISKBVwEZFAqYCLiARKBVxEJFBpVuTpZ2brzewl\nM3vVzH4QjZ9gZuvMbKuZPWJmffIPVyQ7ym0JXZoO/CNggrt/GTgNmGRm44AFwCJ3PxHYA8zIL0yR\nXCi3JWjdFnAv+SB62Tv648AEYHk0vgS4MJcIRXKi3JbQpV2Vvle0ZuBu4GngTWCvu8dLrLQBw7v4\nbIuZbTCzDVkELJKlWnO7PK8/5qPiAhYpk6qAu/sBdz8NGAF8FRiT9gDu3uruY919bI0xiuSm1twu\nz+ve9M01RpGuVHUVirvvBdYA44FBZhY/DGsEoCcYSbCU2xKiNFehfNbMBkXb/YFvAK9RSvZLo7dN\nB1bkFaRIHpTbEro0j5MdBiwxs16UCv4yd3/CzDYBS83sLuBFYHGOcYrkQbktQeu2gLv7n4DTK4xv\nozRnKBIk5baETndiiogESgVcRCRQKuAiIoFSARcRCZQKuIhIoFTARUQCpQIuIhKoNDfyiABw9tln\nJ9tr1qxpYCQi2dn36xOT7U9P3trASKqnDlxEJFDqwKVLCxYsAGDmzJkA9OrVK9n3u9/9DoAJEyYU\nH5hIHco77q72hdKJqwMXEQmUCriISKA0hSKH2LhxY7L9pS99CYCjjir9f37ZsmXJvrvuuqvYwETq\ncLhpk5CpAxcRCVTqDjx6ZvIGYKe7TzGzE4ClwGeAjcBV7r4/nzAlD2PGHFw9LD5ReeqppyZjcef9\n6KOPAjB37txk37Zt24oIMXfK656nnm47lJOXsWo68JmUViuJLQAWufuJwB5gRpaBiRREeS3BSrsq\n/QjgX4H7o9cGTACWR29ZAlyYR4AieVFeS+jSTqH8CPgecGz0+jPAXndvj163AcMzjk1yduONNybb\nLS0tnfbPmzcPgDvuuAOAjz/+uJjAiqO8PoKFNl1SSZpFjacAu919Y3fv7eLzLWa2wcw21PJ5kTxk\nmdcf81HG0Ymkk6YDPxM438wmA/2AgcCPgUFmdnTUrYwAdlb6sLu3Aq0AZuaZRC11WbhwIQA33HBD\np31/+ctfku3W1lagR3bekGFeD7Qhyusm0FMvFTycbjtwd7/V3Ue4+yhgGvCMu18JrAEujd42HViR\nW5QiGVNeS09Qz408c4ClZnYX8CKwOJuQJC9f+MIXgIOdd/mzTeLO+7zzzkvGduzYkdmxx44dC8Do\n0aMB6NOnT7Jvzpw5AFx++eXJ2KZNmzI7dpWU14GptvPOcu67kceGKgu4u68F1kbb24CvZhqNSAMo\nryVUuhNTRCRQehbKEWTWrFkAHHPMMZ32xScsX3nllbqPc+655wJw9913J2PDh5euxhs8eHCXn/vZ\nz36WbOsxtZKlLKYumvEkqTpwEZFAqQPv4aZPn55sX3fddYfsW7lyZbJ9zz33VPW9cSd98cUXJ2Px\nydH4ROXhuu1ymzdvBuDnP/95VTHIkSttN1xr592M3XYl6sBFRAKlAi4iEihNofRw48aNS7bjx8PG\n1q9fn2x/9FH3t4OXT4lcddVVACxatKiqeOK7OlevXp2Mxc9h2bmz4k2PIoXJY+okz2euqAMXEQmU\nOvAeauTIkQBce+21nfbFJy/TnriMO+8HHnggGSu/YzONrVtLXci9994LVN+5i0C6DrnajjfLrrvo\nJxyqAxcRCZQ68B7q6quvBuDoozv/K45vsOlu3rtj511t112+7Nrs2bMBePzxx6v6DpG8hNx5x9SB\ni4gESgVcRCRQmkLpob75zW92uS8+oVjJoEGDku1vfetbQPVTJxs3lha5Kb9Ls62trarvEKlW2mmM\nnjB1ElMHLiISqFQduJn9GXgfOAC0u/tYMxsCPAKMAv4MTHX3PfmEKdUaM2YMAJ988klVn7v//vuT\n7Ysuuqiqz3bsvEPoupXbR4ae1HWXq6YDP9vdT3P3sdHrW4DV7n4SsDp6LRIi5bYEqZ4plAuAJdH2\nEuDC+sMRaQrKbQlC2pOYDvxvtKr8fdGK3EPdfVe0/21gaB4BSvbi67vffffdZGzKlCkAnHPOOVV9\n1wsvvJBshzR1Uka53UPE0yTlUxw9deoklraAn+XuO83sH4Cnzez18p3u7tF/AJ2YWQvQUmecInmp\nKbfL87ofA4qJVKSDVAXc3XdGP3eb2WOUFn19x8yGufsuMxsG7O7is61AK0BXRV6y99577wGVl09b\nsWIFAM8991wydtlll3X5/lj5nZsvvvgiAFOnTk3GQnyaYK25XZ7XA22I8rqJZP1EwWbsvGPdzoGb\n2afM7Nh4G/gX4BVgJRAv9zIdWJFXkCJ5UG5L6NJ04EOBx8wsfv9/uvtvzOx5YJmZzQC2A1MP8x1S\nsDvuuAOAhQsXdtoXX2IY/0zrzjvvTLbnz59fR3RNQ7kth2jmbruSbgu4u28Dvlxh/F1gYh5BiRRB\nuS2h052YIiKBMvfizr/oJGbxli5dmmxPmjQJgL59+wKH3qV54MABAPr375+MdVyCrfzywIkTSw3q\n4Z6r0gjubkUfc6AN8TNMDXuR8l41vtmmUn7ryzeW3WiWUAcuIhIoPY2wh5s2bVqnsbPOOguAvXv3\nJmM7duwA4NJLL03Gxo8fD8CVV14JHLz8EGD//v3ZByuSUqUOuVJXHr8v7469UdSBi4gESnPg0qNo\nDlx6Is2Bi4j0MCrgIiKBUgEXEQmUCriISKBUwEVEAqUCLiISKBVwEZFAqYCLiARKBVxEJFCpCriZ\nDTKz5Wb2upm9ZmbjzWyImT1tZluin4PzDlYka8ptCVnaDvzHwG/cfQylB+C/BtwCrHb3k4DV0WuR\n0Ci3JVhp1sT8NPDPwGIAd9/v7nuBC4Al0duWABfmFaRIHpTbEro0HfgJwN+AB8zsRTO7P1oAdqi7\n74re8zal9QU7MbMWM9tgZhuyCVkkMzXndnlef8xHBYYsclCaAn408I/Af7j76cCHdPgrpZceaVjx\nSYPu3uruYys9SUukwWrO7fK87k3fQoIV6ShNAW8D2tx9XfR6OaWkf8fMhgFEP3fnE6JIbpTbErRu\nC7i7vw3sMLOTo6GJwCZgJTA9GpsOrKjwcZGmpdyW0KVdUu3fgF+aWR9gG3ANpeK/zMxmANuBqfmE\nKJIr5bYEK1UBd/c/ApXmsLUMiQRNuS0h052YIiKBUgEXEQmUCriISKCKXpX+b5Sutf17YQfN3nGE\nG3/IsUP38X/O3T9bVDCxKK+3E/bvN+TYIez408ReMbcLLeAAZrYh5Jt6Qo4/5Nih+eNv9vgOJ+TY\nIez464ldUygiIoFSARcRCVQjCnhrA46ZpZDjDzl2aP74mz2+wwk5dgg7/ppjL3wOXEREsqEpFBGR\nQBVawM1skpltNrOtZtbUq5yY2UgzW2Nmm8zsVTObGY0Hs9yWmfWKnnP9RPT6BDNbF/3+H4me/9GU\nQlrqLKS8BuV2o2WZ24UVcDPrBfw7cC5wCnCFmZ1S1PFr0A58x91PAcYBN0bxhrTc1kxKS4TFFgCL\n3P1EYA8woyFRpRPEUmcB5jUotxstu9x290L+AOOBp8pe3wrcWtTxM4h/BfANYDMwLBobBmxudGxd\nxDsiSoQJwBOAUbpZ4OhK/z6a6Q/waeAtonM0ZeNN97sPPa+jmJXbxcWeaW4XOYUyHNhR9rotGmt6\nZjYKOB1YR8ql5JrAj4DvAZ9Erz8D7HX39uh1M//+61rGr2DB5jUotxsg09zWScxumNkxwKPAt939\nvfJ9XvrfZdNdxmNmU4Dd7r6x0bHUqK5l/CQd5XZDZJrbRRbwncDIstcjorGmZWa9KSX4L939V9Fw\nCMttnQmcb2Z/BpZS+qvmj4FBZhY/A76Zf/8hLXUWXF6DcruBMs3tIgv488BJ0dniPsA0SktXNSUz\nM2Ax8Jq731O2q+mX23L3W919hLuPovR7fsbdrwTWAJdGb2vK2CG4pc6CymtQbjdS5rld8AT+ZOAN\n4E1gbqNPKHQT61mU/hrzJ+CP0Z/JlObbVgNbgN8CQxodazf/HF8Hnoi2RwPrga3AfwN9Gx3fYeI+\nDdgQ/f7/BxjcrL/7kPI6ile53di4M8tt3YkpIhIoncQUEQmUCriISKBUwEVEAqUCLiISKBVwEZFA\nqYCLiARKBVxEJFAq4CIigfp/5EFMvzQ1NgwAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sVfoiPrt_210"
      },
      "source": [
        "###U-Net for image segmentation\n",
        "See if you can implement a U-Net inspired by this one (not necessarily identical to), except that it should operate on 64x64 images instead of 128x128 images:\n",
        "\n",
        "![alt text](https://miro.medium.com/max/937/1*yzbjioOqZDYbO6yHMVpXVQ.jpeg)\n",
        "\n",
        "Recall that\n",
        "- 2@Conv layers means that two consecutive Convolution Layers are applied\n",
        "- c1, c2, ... c9 are the output tensors of Convolutional Layers\n",
        "- p1, p2, p3 and p4 are the output tensors of Max Pooling Layers\n",
        "- u6, u7, u8 and u9 are the output tensors of up-sampling (transposed convolutional is called `Conv2DTranspose` in Keras) layers.\n",
        "\n",
        "**IMPORTANT NOTE** In the figure above, the indices of the encoder go from 1 to 5 (e.g., c1, c2, ..., c5), and the indices of the decoder go from 6 to 9 (e.g., u6, u7, ..., u9). Because we are working on images with half the with and height, our indices go from **1 to 4 in the encoder** and **5 to 8 in the decoder**.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "a_Yw6_PkkWkU"
      },
      "source": [
        "from keras.layers import concatenate, Conv2DTranspose\n",
        "\n",
        "def depthwiseSoftmax(x):\n",
        "    return softmax(x,axis=3)\n",
        "\n",
        "inputs = Input(shape=(64, 64, 1))\n",
        "\n",
        "# Encoder\n",
        "c1 = Conv2D(8, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(inputs)\n",
        "c1 = Conv2D(8, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(c1)\n",
        "p1 = MaxPooling2D(pool_size=(2, 2))(c1)\n",
        "c2 = ???\n",
        "p2 = ???\n",
        "c3 = ???\n",
        "p3 = ???\n",
        "c4 = ???\n",
        "\n",
        "# Decoder\n",
        "u5 = Conv2DTranspose(32, (3, 3), strides = (2, 2), padding = 'same')(c4)\n",
        "u5 = concatenate([c3,u5], axis = 3)\n",
        "c5 = Conv2D(32, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(u5)\n",
        "c5 = Conv2D(32, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(c5)\n",
        "u6 = ???\n",
        "c6 = ???\n",
        "u7 = ???\n",
        "c7 = ???\n",
        "\n",
        "# Perform softmax on each pixel, so axis should be 3 because output has shape: batch_size x 64 x 64 x num_classes\n",
        "c8 = Conv2D(num_classes, 1, activation = depthwiseSoftmax)(c7)\n",
        "\n",
        "model = Model(inputs, c8)\n",
        "model.summary()\n",
        "model.compile(optimizer='Adam', loss = 'binary_crossentropy')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "By1_TT4pOrpu"
      },
      "source": [
        "###Training"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iu1bZx4J65lM"
      },
      "source": [
        "model.fit(x_train_seg, y_train_seg, epochs=30, batch_size=64,shuffle=True,verbose=1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ECy9tSPLOuG2"
      },
      "source": [
        "###Model evaluation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vY7F1-seFTBs"
      },
      "source": [
        "# Pick 4 random examples\n",
        "rand_ix = np.random.randint(0,5000,4)\n",
        "out = model.predict(x_train_seg[rand_ix,:,:,:])\n",
        "ref = y_train_seg[rand_ix,:,:,:].squeeze()\n",
        "for k in range(4):\n",
        "  plt.figure(figsize=(20,4))\n",
        "  plt.subplot(1,3,1)\n",
        "  plt.imshow(x_train_seg[rand_ix[k],:,:,:].squeeze(),cmap='gray')\n",
        "  plt.title('Input image')\n",
        "  ax.get_xaxis().set_visible(False)\n",
        "  ax.get_yaxis().set_visible(False)\n",
        "\n",
        "  plt.subplot(1,3,2)\n",
        "  show_segmentation(ref[k,:,:,:].squeeze())\n",
        "  plt.title('Ground truth')\n",
        "  ax.get_xaxis().set_visible(False)\n",
        "  ax.get_yaxis().set_visible(False)\n",
        "\n",
        "  plt.subplot(1,3,3)\n",
        "  show_segmentation(out[k,:,:,:].squeeze())\n",
        "  plt.title('Predicted')\n",
        "  ax.get_xaxis().set_visible(False)\n",
        "  ax.get_yaxis().set_visible(False)"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}