04_build_your_first_convolutional_neural_network.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%HTML\n",
    "<link rel=\"stylesheet\" type=\"text/css\" href=\"//fonts.googleapis.com/css?family=Quicksand:300\" />\n",
    "<link rel=\"stylesheet\" type=\"text/css\" href=\"custom.css\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.environ['KERAS_BACKEND'] = 'tensorflow'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import random\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "from keras import datasets\n",
    "from keras import layers\n",
    "from keras.models import Sequential\n",
    "from keras.utils import to_categorical\n",
    "\n",
    "from utils import plot_training_summary\n",
    "from utils import TimeSummary\n",
    "from utils import set_seed\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.rcParams['figure.figsize'] = 15, 6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Build your first convolutional neural network\n",
    "\n",
    "- Rodrigo Agundez\n",
    "- Amsterdam @ Booking.com\n",
    "- Friday May 25th, 2018\n",
    "\n",
    "![footer_logo](images/logo.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "**Goal**\n",
    "\n",
    "- Build a convolutional neural network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Questions\n",
    "\n",
    "This Notebook contains some questions labeled __Homework__.\n",
    "Ignore them now but do try to answer them later if you want to get more in-depth understanding."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like many other libraries, `keras` includes some standard datasets to play around with.\n",
    "The follow a specific [API](https://www.tensorflow.org/programmers_guide/datasets) that make them iteract nicely with TensorFlow. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the MNIST dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(X_train, y_train), (X_test, y_test) = datasets.mnist.load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you've tried this Notebook and made sense of the model, load the CIFAR10 dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# (X_train, y_train), (X_test, y_test) = datasets.cifar10.load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get a summary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(f\"Train:\\tX shape:{X_train.shape}\\tY shape:{y_train.shape}\\tType (X, y): ({X_train.dtype}, {y_train.dtype})\\tX values (max, min): ({X_train.min()}, {X_train.max()})\")\n",
    "print(f\"Test:\\tX shape:{X_test.shape}\\tY shape:{y_test.shape}\\tType (X, y): ({X_test.dtype}, {y_test.dtype})\\tX values (max, min): ({X_test.min()}, {X_test.max()})\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> #### Homework: Dataset summary\n",
    ">\n",
    "- How many training examples do we have?\n",
    "- How many color channels does each picture have?\n",
    "- What will the input size to the DNN be?\n",
    "- What will the output size of the DNN be? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show some example images:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "[ax.imshow(random.choice(X_train), cmap='gray') for ax in plt.subplots(1, 6)[1]];"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocessing\n",
    "\n",
    "- Since we are using a convolutional network we do not need to flatten the array into 1D.\n",
    "- rescale pixel values between 0 and 1\n",
    "- Input type should be float\n",
    "- There are 10 classes so in order to compute the cross entropy loss function we need to one-hot encoded vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if len(X_train.shape) != 4:\n",
    "    X_train = np.expand_dims(X_train, axis=3)\n",
    "if len(X_test.shape) != 4:\n",
    "    X_test = np.expand_dims(X_test, axis=3)\n",
    "\n",
    "X_train, X_test = X_train.astype('float') / 255, X_test.astype('float') / 255\n",
    "y_train_onehot, y_test_onehot = to_categorical(y_train), to_categorical(y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the resulting dimensions and types:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Train:\\tX shape:{X_train.shape}\\tY shape:{y_train_onehot.shape}\\tType (X, y): ({X_train.dtype}, {y_train_onehot.dtype})\\tX values (min, max): ({X_train.min()}, {X_train.max()})\")\n",
    "print(f\"Test:\\tX shape:{X_test.shape}\\tY shape:{y_test_onehot.shape}\\tType (X, y): ({X_test.dtype}, {y_test_onehot.dtype})\\tX values (min, max): ({X_test.min()}, {X_test.max()})\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> #### Homework: Model input & output\n",
    "- Are the input and output sizes correct from what you thought?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise (solution in `utils.py`)\n",
    "\n",
    "Construct a model with the instructions:\n",
    "\n",
    "- inputs are normalized using `BatchNormaliation` followed by a `Dropout` layer with a rate of 0.3\n",
    "- then add a [2D convolutional layer](https://keras.io/layers/convolutional/) with a kernel of 3x3\n",
    "- output from the covolutional layer goes through a [`MaxPooling` layer](https://keras.io/layers/pooling/)\n",
    "- then `Flatten` the output and add a `Dropout` layer with a rate of 0.3\n",
    "- connect the output to a `Dense` layer\n",
    "- followed by a `BatchNormalization` and `relu` activation function\n",
    "- then a `DropoutLayer`\n",
    "- and finally connect to the output layer with an `softmax` activation function\n",
    "\n",
    "doen forget to:\n",
    "- use the `relu` activation function or others\n",
    "- use `Dropout` layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "def make_CNN_model():\n",
    "    model = Sequential()\n",
    "    # input layer transformation (BatchNormalization + Dropout)\n",
    "    \n",
    "    # convolutional layer (Conv2D + MaxPooling2D + Flatten + Dropout)\n",
    "    \n",
    "    # fully connected layer (Dense + BatchNormalization + Activation + Dropout)\n",
    "    \n",
    "    # output layer (Dense + BatchNormalization + Activation)\n",
    "\n",
    "    return model\n",
    "\n",
    "make_CNN_model().summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question:** can you explain where are the number of parameters for each layer coming from?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "callback"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "time_summary = TimeSummary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "set_seed(123) # for reproducibility\n",
    "\n",
    "model = make_CNN_model()\n",
    "model.compile(\n",
    "    optimizer='Adam',\n",
    "    loss='categorical_crossentropy',\n",
    "    metrics=['accuracy']\n",
    ")\n",
    "summary = model.fit(\n",
    "    X_train, y_train_onehot,\n",
    "    batch_size=5000,\n",
    "    epochs=5,\n",
    "    validation_split=0.2,\n",
    "    verbose=1,\n",
    "    callbacks=[time_summary]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "score = model.evaluate(X_test, y_test_onehot, verbose=0)\n",
    "print('Test loss:', score[0])\n",
    "print('Test accuracy:', score[1])\n",
    "plot_training_summary(summary, time_summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# [Next: Transfer learning for image classification](05_transfer_learning_for_image_classification.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:pydata]",
   "language": "python",
   "name": "conda-env-pydata-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}