Neural_Networks_Lab_Crash_Course_AI_5.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Neural Networks Lab: Crash Course AI #5",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/CrisFiss/Colab-Notebooks/blob/master/Neural_Networks_Lab_Crash_Course_AI_5.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n-PuJl-rHxW1",
        "colab_type": "text"
      },
      "source": [
        "Neural networks can perform really powerful tasks. We're going to use them to do something relatively simple, but this framework can be expanded to solve more complex problems.\n",
        "\n",
        "In this exercise, we're going to use a dataset of handwritten letters called EMNIST. This dataset is a subset of a larger dataset composed of handwritten letters and numbers that was orginally collected from census bureau employees and high school students. \n",
        "\n",
        "Because different people write letters differently, programming a neural network to recognize handwritten letters and convert them to typed text is a non-trivial task. For example, think about all of the different ways you've seen people write the letter \"J.\"\n",
        "\n",
        "**STEP 1**\n",
        "\n",
        "The first thing we're going to do is import the EMNIST libraries so that we have access to this training set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bGP5woydMEMa",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# STEP 1.1\n",
        "\n",
        "!git clone https://github.com/sorki/python-mnist\n",
        "!./python-mnist/get_data.sh\n",
        "!pip3 install emnist\n",
        "from emnist import extract_training_samples\n",
        "\n",
        "print(\"Imported the EMNIST libraries we need!\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tsAUNl0jQ24H",
        "colab_type": "text"
      },
      "source": [
        "Now, we want to grab the data that we want to analyze from these libraries.  In this case, we want to grab the EMNIST \"letters\" dataset, which is made up of 145,600 28x28 pixel (or 784 pixels in total) images of letters. Each pixel is a grayscale value between 0 and 255. To make it easier for the neural network to process, we divide each value by 255 to get to a number between 0 and 1 for each pixel in each image. Performing a transformation like this to make the data easier to process in a machine learning method is called pre-processing.  Different pre-processing steps may be required for different types of data. \n",
        "\n",
        "This dataset also includes the known labels for these images. That's what our neural network is going to learn from. In this case, we'll use the first 60,000 pictures as our training set and then we'll use the following 10,000 pictures as our testing set (and use them to ask the network to check how well it has learned). This is a bit of an arbitrary decision, but based on our experience, this is large enough of a sample (more will just take longer to process but won't improve accuracy of the model) --- but don't take our word for it, you can play around with the training and testing sample on your own. Just remember to always keep the two samples separate.\n",
        "\n",
        "When you run this cell, it might take a little bit of time -- remember that this code is essentially downloading 70,000 pictures!\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vDj9tyEAMVn5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# STEP 1.2\n",
        "\n",
        "# Grab the data from the OpenML website\n",
        "# X will be our images and y will be the labels\n",
        "X, y = extract_training_samples('letters')\n",
        "\n",
        "# Make sure that every pixel in all of the images is a value between 0 and 1\n",
        "X = X / 255.\n",
        "\n",
        "# Use the first 60000 instances as training and the next 10000 as testing\n",
        "X_train, X_test = X[:60000], X[60000:70000]\n",
        "y_train, y_test = y[:60000], y[60000:70000]\n",
        "\n",
        "# There is one other thing we need to do, we need to\n",
        "# record the number of samples in each dataset and the number of pixels in each image\n",
        "X_train = X_train.reshape(60000,784)\n",
        "X_test = X_test.reshape(10000,784)\n",
        "\n",
        "print(\"Extracted our samples and divided our training and testing data sets\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-7xwZ3sGQ2R5",
        "colab_type": "text"
      },
      "source": [
        "To make sure the data downloaded, let's take a look at some of the pictures and labels in the set.  You can change the img_index value to different numbers and run the code as many times as you want to look at the label and picture for different training instances. Some letters are pretty clear (like, 8888 is definitely a \"Y\"), but others not as much (like, is 1234 a loopy \"L\" or a \"P\" or maybe even an \"E\" at a weird angle?) It's easy to see why this is such a hard task!\n",
        "\n",
        "But the important thing is that we know that the data downloaded correctly, so we're ready for the next steps.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DWcb4sxAP2lT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# STEP 1.3\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "img_index = 14000 # <<<<<  You can update this value to look at other images\n",
        "img = X_train[img_index]\n",
        "print(\"Image Label: \" + str(chr(y_train[img_index]+96)))\n",
        "plt.imshow(img.reshape((28,28)))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gfT7zKZkUyYn",
        "colab_type": "text"
      },
      "source": [
        "**STEP 2**\n",
        "\n",
        "Now that we've looked at the data, let's build a neural network that will be able to take the picture as input and output the corresponding letter.  In this case, we're going to stick to a multi-layer perceptron neural network, or MLP. \n",
        "\n",
        "Rather than trying to write all of it from scratch, we're going to import and use an existing library that provides some MLP tools for us. For a lot of AI problems, we can use existing libraries and tweak them to do something new rather than trying to write everything from scratch.\n",
        "\n",
        "Just to try this out, we'll create a neural network -- a MLP classifier -- with one hidden layer that has 50 neurons in it. We'll have it run through the training data 20 times (that's ```max_iter=20```) so it doesn't take too long. \n",
        "\n",
        "We also set a variety of other learning parameters to create the MLP.  You may recognize some of these parameters, like learning rate.  By changing any of these parameters, we can influence how the MLP performs in training and testing, but it's a good idea to try a default approach and then improve on it later."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Rx-OxVm6Mjj2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# STEP 2.1\n",
        "\n",
        "# These two lines import the ML libraries we need\n",
        "from sklearn.datasets import fetch_openml\n",
        "from sklearn.neural_network import MLPClassifier\n",
        "\n",
        "# This creates our first MLP with 1 hidden layer with 50 neurons and sets it to run through the data 20 times\n",
        "mlp1 = MLPClassifier(hidden_layer_sizes=(50,), max_iter=20, alpha=1e-4,\n",
        "                    solver='sgd', verbose=10, tol=1e-4, random_state=1,\n",
        "                    learning_rate_init=.1)\n",
        "\n",
        "print(\"Created our first MLP network\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lIBxv5cfWD9M",
        "colab_type": "text"
      },
      "source": [
        "**STEP 3**\n",
        "\n",
        "Now that we've created the MLP, let's actually train it and test its ability to recognize some handwritten letters!  Go ahead and \"fit\" command to see how it will perform -- it might take a minute. As it's learning, it will print status update information about the learning process. You can see the loss value decreasing over each iteration, which means it's getting better at recognizing the training set!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DcyaMwrmWCk8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# STEP 3.1\n",
        "\n",
        "mlp1.fit(X_train, y_train)\n",
        "print(\"Training set score: %f\" % mlp1.score(X_train, y_train))\n",
        "print(\"Test set score: %f\" % mlp1.score(X_test, y_test))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nSKEzW7YWT-l",
        "colab_type": "text"
      },
      "source": [
        "You can try running this a couple of times and you'll see that the accuracy and loss will be in the same ballpark but slightly different each time. These differences are due to the fact that the neuron weights are randomly initialized in each run. But if you didn't change anything in the MLP structure above, you're probably seeing something like 89% accuracy on the training dataset and 84% accuracy on the testing dataset.\n",
        "\n",
        "That's actually pretty good -- it means that the network is guessing correctly 84% of time on letters it's never seen before! It learned! But let's see if we can make it even better. We can explore the dataset to figure out where things went wrong."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0Ux7zg-iYAhv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# STEP 3.2\n",
        "\n",
        "# First let's initialize a list with all the predicted values from the training set\n",
        "y_pred = mlp1.predict(X_test)\n",
        "\n",
        "# Now let's visualize the errors between the predictions and the actual labels using a confusion matrix\n",
        "from sklearn.metrics import confusion_matrix\n",
        "cm = confusion_matrix(y_test, y_pred)\n",
        "plt.matshow(cm)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dWe9nrfdaPBq",
        "colab_type": "text"
      },
      "source": [
        "The color of each cell in the confusion matrix represents the number of elements in that cell -- a brighter color means more elements. The rows are the correct value and the columns are the predicted value. The numbers on the axes represent the 26 letters in the alphabet (though EMNIST just represents letters as numbers, so 0=A, 1=B, etc.), so the brightness of cell (0,0) represents the number of times that our network correctly predicted that an \"A\" is an \"A.\"  The diagonal line of cells shows when the predicted value is correct. It's good to see a bright line! \n",
        "\n",
        "If any of the cells off the diagonal are particularly bright, we can investigate more.\n",
        "\n",
        "For example, \"I\" and \"L\" may be easy to confuse (\"U\" and \"V\" are also good ones to try!), so let's look at some of the cases where that happened. You can change the code and run it as many times as you want to look at some examples of these mistakes."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kGNJFFZmfS4I",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# STEP 3.3\n",
        "\n",
        "# You can change this to any letters that you think the neural network may have confused...\n",
        "predicted_letter = 'l'\n",
        "actual_letter = 'i'\n",
        "\n",
        "\n",
        "# This code counts all mistakes for the letters above\n",
        "mistake_list = []\n",
        "for i in range(len(y_test)):\n",
        "  if (y_test[i] == (ord(actual_letter) - 96) and y_pred[i] == (ord(predicted_letter) - 96)):\n",
        "    mistake_list.append(i)\n",
        "print(\"There were \" + str(len(mistake_list)) + \" times that the letter \" + actual_letter + \" was predicted to be the letter \" + predicted_letter + \".\")\n",
        "\n",
        "# Once we know how many mistakes were made, we can change this to see an image of a particular one\n",
        "mistake_to_show = 4 # <<< e.g., change this to 3 if you want to see the 4th mistake\n",
        "\n",
        "# This code checks that the number mistake you asked for can be shown and if so, displays an image of it\n",
        "if (len(mistake_list)> mistake_to_show):\n",
        "  img = X_test[mistake_list[mistake_to_show]]\n",
        "  plt.imshow(img.reshape((28,28)))\n",
        "else:\n",
        "  print(\"Couldn't show mistake number \" + str(mistake_to_show + 1) + \" because there were only \" + str(len(mistake_list)) + \" mistakes to show!\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LAOTwscofh0X",
        "colab_type": "text"
      },
      "source": [
        "So we can sort of see why the MLP neural network made the mistake here. Even as a human, I probably wouldn't know what some of these letters are unless I could see them in a word and guess from context. Neural networks have a hard job!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-QRU60vfX4XJ",
        "colab_type": "text"
      },
      "source": [
        "On EMNIST, the state-of-the-art machine learning techniques get greater than 96% accuracy, so I think we can also get better than 84%. Let's make a new network called mlp2 and try some things out.\n",
        "\n",
        "One thing we can try is having more hidden layers and more neurons in the hidden layers. For example, to add another hidden layer of 50 neurons, you can do the following: ```hidden_layer_sizes=(50,50,)```. \n",
        "\n",
        "The other thing we can try is training for more epochs (or iterations). For example, we can change our ```max_iter=30```. \n",
        "\n",
        "You can play around with the structure to see what happens, but with more layers and more epochs, it takes a longer time for the network to train! So let's just start with 5 layers of 100 neurons each, trained for 50 epochs."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WiAUTXZrW2as",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# STEP 3.4\n",
        "\n",
        "# Change some of the values in the below statement and re-run to see how they \n",
        "# affect performance!\n",
        "mlp2 = MLPClassifier(hidden_layer_sizes=(100,100,100,100,100,), max_iter=50, alpha=1e-4,\n",
        "                    solver='sgd', verbose=10, tol=1e-4, random_state=1,\n",
        "                    learning_rate_init=.1)\n",
        "mlp2.fit(X_train, y_train)\n",
        "print(\"Training set score: %f\" % mlp2.score(X_train, y_train))\n",
        "print(\"Test set score: %f\" % mlp2.score(X_test, y_test))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M_VRX5lEWcwe",
        "colab_type": "text"
      },
      "source": [
        "So that's a fair amount of improvement! By having more layers and more training epochs, we got an extra 5% or so of accuracy (remember, this might vary slightly because there is some randomness in the process). That might not seem like a lot, but eliminating 1 in 20 mistakes is actually a pretty big deal!\n",
        "\n",
        "You can keep playing with the neural network structure in this block and see if you can get even better accuracy. But this should also be good enough for now, so we're ready to apply it to our scanned letters.\n",
        "\n",
        "**STEP 4**\n",
        "\n",
        "First, we have to get the scanned letter dataset that we uploaded and saved on GitHub. You can see the raw scanned data set here: https://github.com/crash-course-ai/lab1-neural-networks/tree/master/letters. \n",
        "\n",
        "But, as you can see, those images are HUGE. So we've also done a bit of preprocessing to avoid having to download and process quite so much data. We've changed the size of every image to 128x128 pixels.The other thing you may notice is that the EMNIST dataset uses a dark background with light strokes, but our original scans have a white background with dark strokes. So, we also went ahead and inverted those colors. \n",
        "\n",
        "This modified dataset is what we'll use in our code and it is also available in the GitHub here: https://github.com/crash-course-ai/lab1-neural-networks/tree/master/letters_mod"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Bs1-wuqGHYRm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# STEP 4.1\n",
        "\n",
        "# Pulls the scanned data set from GitHub\n",
        "!git clone https://github.com/crash-course-ai/lab1-neural-networks.git\n",
        "!git pull\n",
        "!ls lab1-neural-networks/letters_mod\n",
        "!cd /content/lab1-neural-networks/letters_mod\n",
        "!pwd\n",
        "\n",
        "# Puts all the data in the \"files\" variable\n",
        "import os\n",
        "path, dirs, files = next(os.walk(\"/content/lab1-neural-networks/letters_mod/\"))\n",
        "files.sort()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8jot1iIMAC83",
        "colab_type": "text"
      },
      "source": [
        "Here we'll read each image and add it to a list to hold the handwritten story. We'll also print this image to make sure we read it correctly -- feel free to change this index to see different letters."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "K0hPrlEvW5ZY",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# STEP 4.2\n",
        "\n",
        "# These libraries let us import the letters, resize them, and print them out\n",
        "import cv2\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "# This code processes all the scanned images and adds them to the handwritten_story\n",
        "handwritten_story = []\n",
        "for i in range(len(files)):\n",
        "  img = cv2.imread(\"/content/lab1-neural-networks/letters_mod/\"+files[i],cv2.IMREAD_GRAYSCALE)\n",
        "  handwritten_story.append(img)\n",
        "\n",
        "print(\"Imported the scanned images.\")\n",
        "\n",
        "plt.imshow(handwritten_story[4])  #<--- Change this index to see different letters\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L85HdxpOoM0t",
        "colab_type": "text"
      },
      "source": [
        "Well, to the naked eye this looks similar enough to the EMNIST letters and these even seem like they're MORE clear and bright. So, let's try actually getting the story!\n",
        "\n",
        "We're going to put the handwritten story into our trained MLP and see how it does."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2TPLm8eYoq38",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# STEP 4.3\n",
        "\n",
        "# These are libraries we need to do some math on the image\n",
        "# to be able to give it to the MLP in the right format and to resize it to 28x28 pixels\n",
        "import numpy\n",
        "import cv2\n",
        "\n",
        "typed_story = \"\"\n",
        "for letter in handwritten_story:\n",
        "    letter = cv2.resize(letter, (28,28), interpolation = cv2.INTER_CUBIC)\n",
        "    single_item_array = (numpy.array(letter)).reshape(1,784)\n",
        "    prediction = mlp2.predict(single_item_array)\n",
        "    typed_story = typed_story + str(chr(prediction[0]+96))\n",
        "    \n",
        "print(\"Conversion to typed story complete!\")\n",
        "print(typed_story)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FOVBUJagqLus",
        "colab_type": "text"
      },
      "source": [
        "And... that makes no sense, so what are we doing wrong?\n",
        "\n",
        "Well, first of all, John-Green-bot's handwritten story had some empty spaces between words. We never actually trained our model on empty spaces, just the 26 letters, so it wouldn't be able to detect these.\n",
        "\n",
        "But blank pages should be easy to detect. After all, unlike handwritten letters, all blank images should be exactly the same. So, we'll just check each image to see if it's a blank space. And if it is, we'll add a space to our story. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "U8BGJtQRq1xi",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# STEP 4.4\n",
        "\n",
        "# This is a library we need to do some math on the image to be able to give it to the MLP in the right format\n",
        "import numpy\n",
        "\n",
        "typed_story = \"\"\n",
        "for letter in handwritten_story:\n",
        "  letter = cv2.resize(letter, (28,28), interpolation = cv2.INTER_CUBIC)\n",
        "    \n",
        "  #this bit of code checks to see if the image is just a blank space by looking at the color of all the pixels summed\n",
        "  total_pixel_value = 0\n",
        "  for j in range(28):\n",
        "    for k in range(28):\n",
        "      total_pixel_value += letter[j,k]\n",
        "  if total_pixel_value < 20:\n",
        "    typed_story = typed_story + \" \"\n",
        "  else:         #if it NOT a blank, it actually runs the prediction algorithm on it\n",
        "    single_item_array = (numpy.array(letter)).reshape(1,784)\n",
        "    prediction = mlp2.predict(single_item_array)\n",
        "    typed_story = typed_story + str(chr(prediction[0]+96))\n",
        "    \n",
        "print(\"Conversion to typed story complete!\")\n",
        "print(typed_story)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zZ3RdoEarHyM",
        "colab_type": "text"
      },
      "source": [
        "This looks better. There are separate words, and the first word is \"The.\" But there's still something going wrong.\n",
        "\n",
        "The papers we scanned were really big compared to the handwritten samples that were used to train EMNIST. We resized them, but that doesn't seem to be enough. So we should try processing these images in the same way that EMNIST did.\n",
        "\n",
        "So, let's take a look at the orginal EMNIST paper... The paper (https://arxiv.org/abs/1702.05373v1) describes the steps we need to take to process these images in one of the figures. Essentially, we need to make the strokes more blurry, we need to put the letter at the very center of the image and crop out the rest, and we need to resize each one to be 28x28 pixels.\n",
        "\n",
        "We'll do this processing and then print out one letter to see how it turned out. As always, we can change the index in the code to see a different letter."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RQ1SqowzKeHG",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# STEP 4.5\n",
        "\n",
        "# These steps process the scanned images to be in the same format and have the same properties as the EMNIST images\n",
        "# They are described by the EMNIST authors in detail here: https://arxiv.org/abs/1702.05373v1\n",
        "processed_story = []\n",
        "\n",
        "for img in handwritten_story:\n",
        "  #step 1: Apply Gaussian blur filter\n",
        "  img = cv2.GaussianBlur(img, (7,7), 0)\n",
        "  \n",
        "  #steps 2 and 3: Extract the Region of Interest in the image and center in square\n",
        "  points = cv2.findNonZero(img)\n",
        "  x, y, w, h = cv2.boundingRect(points)\n",
        "  if (w > 0 and h > 0):\n",
        "    if w > h:\n",
        "      y = y - (w-h)//2\n",
        "      img = img[y:y+w, x:x+w]\n",
        "    else:\n",
        "      x = x - (h-w)//2\n",
        "      img = img[y:y+h, x:x+h]\n",
        "     \n",
        "  #step 4: Resize and resample to be 28 x 28 pixels\n",
        "  img = cv2.resize(img, (28,28), interpolation = cv2.INTER_CUBIC)\n",
        "  \n",
        "  #step 5: Normalize pixels and reshape before adding to the new story array\n",
        "  img = img/255\n",
        "  img = img.reshape((28,28))\n",
        "  processed_story.append(img)\n",
        "\n",
        "print(\"Processed the scanned images.\")\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "plt.imshow(processed_story[4]) #<<< change this index if you want to see a different letter from the story"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FhobxHEoKqXt",
        "colab_type": "text"
      },
      "source": [
        "So even though the letter looks less clear now to our human eyes, it actually does look much more similar to the letters in the EMNIST dataset, which is good for our neural network. The edges of the letter are kind of fuzzy and it's centered in the square.\n",
        "\n",
        "Okay, so now we can try the processing the story with our neural network one more time!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wFH4kBA9ar-O",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# STEP 4.6\n",
        "\n",
        "# This is a library we need to do some math on the image to be able to give it to the MLP in the right format\n",
        "import numpy\n",
        "\n",
        "typed_story = \"\"\n",
        "for letter in processed_story:\n",
        "  #this bit of code checks to see if the image is just a blank space by looking at the color of all the pixels summed\n",
        "  total_pixel_value = 0\n",
        "  for j in range(28):\n",
        "    for k in range(28):\n",
        "      total_pixel_value += letter[j,k]\n",
        "  if total_pixel_value < 20:\n",
        "    typed_story = typed_story + \" \"\n",
        "  else:         #if it NOT a blank, it actually runs the prediction algorithm on it\n",
        "    single_item_array = (numpy.array(letter)).reshape(1,784)\n",
        "    prediction = mlp2.predict(single_item_array)\n",
        "    typed_story = typed_story + str(chr(prediction[0]+96))\n",
        "    \n",
        "print(\"Conversion to typed story complete!\")\n",
        "print(typed_story)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fWADSaLh_uho",
        "colab_type": "text"
      },
      "source": [
        "We can definitely pick out more of the words! There are still some mistakes because our neural network accuracy wasn't 100%, so we would expect to see about the same rate of error here (maybe even a bit more, since these letters were originally created at a very different size). But, looking at the context and knowing which letters are likely to be mistaken for one another, we can still read the story.\n",
        "\n",
        "Looks like it starts: \n",
        "\"The Fault in Our Power Supplies” \n",
        "“I fell in love the way your battery dies, slowly and then all at once”\n",
        "\n",
        "No matter how many times you run the predict code above, it's going to make the exact same mistakes in the final printed string. That's because once a network is trained, it's not really random anymore -- it's always applying the same math to the pixels to make its prediction. To get a different prediction, you have to rerun the code above where you trained mlp2.\n",
        "\n",
        "We hope that you mess around with some of this code on your own and try to get better accuracy than we got on the scanned story!"
      ]
    }
  ]
}