deep-learning-with-python-notebooks/3.6-classifying-newswires.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "原文代码作者：François Chollet\n",
    "\n",
    "github：https://github.com/fchollet/deep-learning-with-python-notebooks\n",
    "\n",
    "中文注释制作：黄海广\n",
    "\n",
    "github：https://github.com/fengdu78\n",
    "\n",
    "代码全部测试通过。\n",
    "\n",
    "配置环境：keras 2.2.1（原文是2.0.8，运行结果一致），tensorflow 1.8，python 3.6，\n",
    "\n",
    "主机：显卡：一块1080ti；内存：32g（注：绝大部分代码不需要GPU）\n",
    "![公众号](data/gongzhong.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'2.2.1'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import keras\n",
    "keras.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classifying newswires: a multi-class classification example\n",
    "# 新闻分类：多分类问题\n",
    "\n",
    "This notebook contains the code samples found in Chapter 3, Section 5 of [Deep Learning with Python](https://www.manning.com/books/deep-learning-with-python?a_aid=keras&a_bid=76564dff). Note that the original text features far more content, in particular further explanations and figures: in this notebook, you will only find source code and related comments.\n",
    "\n",
    "----\n",
    "\n",
    "In the previous section we saw how to classify vector inputs into two mutually exclusive classes using a densely-connected neural network. \n",
    "But what happens when you have more than two classes? \n",
    "\n",
    "In this section, we will build a network to classify Reuters newswires into 46 different mutually-exclusive topics. Since we have many \n",
    "classes, this problem is an instance of \"multi-class classification\", and since each data point should be classified into only one \n",
    "category, the problem is more specifically an instance of \"single-label, multi-class classification\". If each data point could have \n",
    "belonged to multiple categories (in our case, topics) then we would be facing a \"multi-label, multi-class classification\" problem.\n",
    "\n",
    "\n",
    "上一节中，我们介绍了如何用密集连接的神经网络将向量输入划分为两个互斥的类别。但如果类别不止两个，要怎么做？\n",
    "\n",
    "本节你会构建一个网络，将路透社新闻划分为 46 个互斥的主题。因为有多个类别，所以这是多分类（multiclass classification）问题的一个例子。因为每个数据点只能划分到一个类别，所以更具体地说，这是单标签、多分类（single-label, multiclass classification）问题的一个例子。如果每个数据点可以划分到多个类别（主题），那它就是一个多标签、多分类（multilabel, multiclass classification）问题。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Reuters dataset\n",
    "\n",
    "\n",
    "We will be working with the _Reuters dataset_, a set of short newswires and their topics, published by Reuters in 1986. It's a very simple, \n",
    "widely used toy dataset for text classification. There are 46 different topics; some topics are more represented than others, but each \n",
    "topic has at least 10 examples in the training set.\n",
    "\n",
    "Like IMDB and MNIST, the Reuters dataset comes packaged as part of Keras. Let's take a look right away:\n",
    "\n",
    "## 路透社数据集\n",
    "\n",
    "本节使用路透社数据集，它包含许多短新闻及其对应的主题，由路透社在 1986 年发布。它是一个简单的、广泛使用的文本分类数据集。它包括 46 个不同的主题：某些主题的样本更多，但训练集中每个主题都有至少 10 个样本。\n",
    "\n",
    "与 IMDB 和 MNIST 类似，路透社数据集也内置为 Keras 的一部分。我们来看一下。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.datasets import reuters\n",
    "\n",
    "(train_data, train_labels), (test_data, test_labels) = reuters.load_data(num_words=10000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Like with the IMDB dataset, the argument `num_words=10000` restricts the data to the 10,000 most frequently occurring words found in the \n",
    "data.\n",
    "\n",
    "We have 8,982 training examples and 2,246 test examples:\n",
    "\n",
    "与 IMDB 数据集一样，参数 num_words=10000 将数据限定为前 10 000 个最常出现的单词。 我们有 8982 个训练样本和 2246 个测试样本。（这里下载可能会失败几次，不翻墙可以下的）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8982"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(train_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2246"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As with the IMDB reviews, each example is a list of integers (word indices):\n",
    "\n",
    "与 IMDB 评论一样，每个样本都是一个整数列表（表示单词索引）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1,\n",
       " 245,\n",
       " 273,\n",
       " 207,\n",
       " 156,\n",
       " 53,\n",
       " 74,\n",
       " 160,\n",
       " 26,\n",
       " 14,\n",
       " 46,\n",
       " 296,\n",
       " 26,\n",
       " 39,\n",
       " 74,\n",
       " 2979,\n",
       " 3554,\n",
       " 14,\n",
       " 46,\n",
       " 4689,\n",
       " 4329,\n",
       " 86,\n",
       " 61,\n",
       " 3499,\n",
       " 4795,\n",
       " 14,\n",
       " 61,\n",
       " 451,\n",
       " 4329,\n",
       " 17,\n",
       " 12]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data[10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's how you can decode it back to words, in case you are curious:\n",
    "\n",
    "如果好奇的话，你可以用下列代码将索引解码为单词。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_index = reuters.get_word_index()\n",
    "reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])\n",
    "# Note that our indices were offset by 3（注 意，索引减去了 3）\n",
    "# because 0, 1 and 2 are reserved indices for \"padding\", \"start of sequence\", and \"unknown\".\n",
    "#（因为 0、1、2 是 为“padding”（ 填 充 ）、“start of sequence”（序列开始）、“unknown”（未知词）分别保留的索引）\n",
    "decoded_newswire = ' '.join([reverse_word_index.get(i - 3, '?') for i in train_data[0]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'? ? ? said as a result of its december acquisition of space co it expects earnings per share in 1987 of 1 15 to 1 30 dlrs per share up from 70 cts in 1986 the company said pretax net should rise to nine to 10 mln dlrs from six mln dlrs in 1986 and rental operation revenues to 19 to 22 mln dlrs from 12 5 mln dlrs it said cash flow per share this year should be 2 50 to three dlrs reuter 3'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "decoded_newswire"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The label associated with an example is an integer between 0 and 45: a topic index.\n",
    "\n",
    "样本对应的标签是一个 0~45 范围内的整数，即话题索引编号。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_labels[10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing the data\n",
    "\n",
    "We can vectorize the data with the exact same code as in our previous example:\n",
    "## 准备数据\n",
    "你可以使用与上一个例子相同的代码将数据向量化。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def vectorize_sequences(sequences, dimension=10000):\n",
    "    results = np.zeros((len(sequences), dimension))\n",
    "    for i, sequence in enumerate(sequences):\n",
    "        results[i, sequence] = 1.\n",
    "    return results\n",
    "\n",
    "# Our vectorized training data（将训练数据向量化）\n",
    "x_train = vectorize_sequences(train_data)\n",
    "# Our vectorized test data（将测试数据向量化）\n",
    "x_test = vectorize_sequences(test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "To vectorize the labels, there are two possibilities: we could just cast the label list as an integer tensor, or we could use a \"one-hot\" \n",
    "encoding. One-hot encoding is a widely used format for categorical data, also called \"categorical encoding\". \n",
    "For a more detailed explanation of one-hot encoding, you can refer to Chapter 6, Section 1. \n",
    "In our case, one-hot encoding of our labels consists in embedding each label as an all-zero vector with a 1 in the place of the label index, e.g.:\n",
    "\n",
    "将标签向量化有两种方法：你可以将标签列表转换为整数张量，或者使用 one-hot 编码。\n",
    "\n",
    "one-hot 编码是分类数据广泛使用的一种格式，也叫分类编码（categorical encoding）。6.1 节给出了 one-hot 编码的详细解释。在这个例子中，标签的 one-hot 编码就是将每个标签表示为全零向量，只有标签索引对应的元素为 1。其代码实现如下。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def to_one_hot(labels, dimension=46):\n",
    "    results = np.zeros((len(labels), dimension))\n",
    "    for i, label in enumerate(labels):\n",
    "        results[i, label] = 1.\n",
    "    return results\n",
    "\n",
    "# Our vectorized training labels（将训练标签向量化）\n",
    "one_hot_train_labels = to_one_hot(train_labels)\n",
    "# Our vectorized test labels（将测试标签向量化）\n",
    "one_hot_test_labels = to_one_hot(test_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that there is a built-in way to do this in Keras, which you have already seen in action in our MNIST example:\n",
    "\n",
    "注意，Keras 内置方法可以实现这个操作，你在 MNIST 例子中已经见过这种方法。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.utils.np_utils import to_categorical\n",
    "\n",
    "one_hot_train_labels = to_categorical(train_labels)\n",
    "one_hot_test_labels = to_categorical(test_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building our network\n",
    "\n",
    "\n",
    "This topic classification problem looks very similar to our previous movie review classification problem: in both cases, we are trying to \n",
    "classify short snippets of text. There is however a new constraint here: the number of output classes has gone from 2 to 46, i.e. the \n",
    "dimensionality of the output space is much larger. \n",
    "\n",
    "In a stack of `Dense` layers like what we were using, each layer can only access information present in the output of the previous layer. \n",
    "If one layer drops some information relevant to the classification problem, this information can never be recovered by later layers: each \n",
    "layer can potentially become an \"information bottleneck\". In our previous example, we were using 16-dimensional intermediate layers, but a \n",
    "16-dimensional space may be too limited to learn to separate 46 different classes: such small layers may act as information bottlenecks, \n",
    "permanently dropping relevant information.\n",
    "\n",
    "For this reason we will use larger layers. Let's go with 64 units:\n",
    "## 构建网络\n",
    "\n",
    "这个主题分类问题与前面的电影评论分类问题类似，两个例子都是试图对简短的文本片段进行分类。但这个问题有一个新的约束条件：输出类别的数量从 2 个变为 46 个。输出空间的维度要大得多。\n",
    "\n",
    "对于前面用过的 Dense 层的堆叠，每层只能访问上一层输出的信息。如果某一层丢失了与分类问题相关的一些信息，那么这些信息无法被后面的层找回，也就是说，每一层都可能成为信息瓶颈。上一个例子使用了 16 维的中间层，但对这个例子来说 16 维空间可能太小了，无法学会区分 46 个不同的类别。这种维度较小的层可能成为信息瓶颈，永久地丢失相关信息。\n",
    "\n",
    "出于这个原因，下面将使用维度更大的层，包含 64 个单元。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras import models\n",
    "from keras import layers\n",
    "\n",
    "model = models.Sequential()\n",
    "model.add(layers.Dense(64, activation='relu', input_shape=(10000,)))\n",
    "model.add(layers.Dense(64, activation='relu'))\n",
    "model.add(layers.Dense(46, activation='softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "There are two other things you should note about this architecture:\n",
    "\n",
    "* We are ending the network with a `Dense` layer of size 46. This means that for each input sample, our network will output a \n",
    "46-dimensional vector. Each entry in this vector (each dimension) will encode a different output class.\n",
    "* The last layer uses a `softmax` activation. You have already seen this pattern in the MNIST example. It means that the network will \n",
    "output a _probability distribution_ over the 46 different output classes, i.e. for every input sample, the network will produce a \n",
    "46-dimensional output vector where `output[i]` is the probability that the sample belongs to class `i`. The 46 scores will sum to 1.\n",
    "\n",
    "The best loss function to use in this case is `categorical_crossentropy`. It measures the distance between two probability distributions: \n",
    "in our case, between the probability distribution output by our network, and the true distribution of the labels. By minimizing the \n",
    "distance between these two distributions, we train our network to output something as close as possible to the true labels.\n",
    "\n",
    "关于这个架构还应该注意另外两点。\n",
    "\n",
    "* 网络的最后一层是大小为 46 的 Dense 层。这意味着，对于每个输入样本，网络都会输出一个 46 维向量。这个向量的每个元素（即每个维度）代表不同的输出类别。\n",
    "\n",
    "* 最后一层使用了 softmax 激活。你在 MNIST 例子中见过这种用法。网络将输出在 46 个不同输出类别上的概率分布——对于每一个输入样本，网络都会输出一个 46 维向量，其中 output[i] 是样本属于第 i 个类别的概率。46 个概率的总和为 1。\n",
    "\n",
    "对于这个例子，最好的损失函数是 `categorical_crossentropy`（分类交叉熵）。它用于衡量两个概率分布之间的距离，这里两个概率分布分别是网络输出的概率分布和标签的真实分布。通过将这两个分布的距离最小化，训练网络可使输出结果尽可能接近真实标签。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer='rmsprop',\n",
    "              loss='categorical_crossentropy',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Validating our approach\n",
    "\n",
    "Let's set apart 1,000 samples in our training data to use as a validation set:\n",
    "\n",
    "## 验证你的方法\n",
    "我们在训练数据中留出 1000 个样本作为验证集。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_val = x_train[:1000]\n",
    "partial_x_train = x_train[1000:]\n",
    "\n",
    "y_val = one_hot_train_labels[:1000]\n",
    "partial_y_train = one_hot_train_labels[1000:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's train our network for 20 epochs:\n",
    "\n",
    "现在开始训练网络，共 20 个轮次。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 7982 samples, validate on 1000 samples\n",
      "Epoch 1/20\n",
      "7982/7982 [==============================] - 1s 157us/step - loss: 2.5322 - acc: 0.4955 - val_loss: 1.7208 - val_acc: 0.6120\n",
      "Epoch 2/20\n",
      "7982/7982 [==============================] - 1s 120us/step - loss: 1.4452 - acc: 0.6879 - val_loss: 1.3459 - val_acc: 0.7060\n",
      "Epoch 3/20\n",
      "7982/7982 [==============================] - 1s 122us/step - loss: 1.0953 - acc: 0.7651 - val_loss: 1.1708 - val_acc: 0.7430\n",
      "Epoch 4/20\n",
      "7982/7982 [==============================] - 1s 121us/step - loss: 0.8697 - acc: 0.8165 - val_loss: 1.0793 - val_acc: 0.7590\n",
      "Epoch 5/20\n",
      "7982/7982 [==============================] - 1s 121us/step - loss: 0.7034 - acc: 0.8472 - val_loss: 0.9844 - val_acc: 0.7810\n",
      "Epoch 6/20\n",
      "7982/7982 [==============================] - 1s 122us/step - loss: 0.5667 - acc: 0.8802 - val_loss: 0.9411 - val_acc: 0.8040\n",
      "Epoch 7/20\n",
      "7982/7982 [==============================] - 1s 121us/step - loss: 0.4581 - acc: 0.9048 - val_loss: 0.9083 - val_acc: 0.8020\n",
      "Epoch 8/20\n",
      "7982/7982 [==============================] - 1s 122us/step - loss: 0.3695 - acc: 0.9231 - val_loss: 0.9363 - val_acc: 0.7890\n",
      "Epoch 9/20\n",
      "7982/7982 [==============================] - 1s 122us/step - loss: 0.3032 - acc: 0.9315 - val_loss: 0.8917 - val_acc: 0.8090\n",
      "Epoch 10/20\n",
      "7982/7982 [==============================] - 1s 123us/step - loss: 0.2537 - acc: 0.9414 - val_loss: 0.9071 - val_acc: 0.8110\n",
      "Epoch 11/20\n",
      "7982/7982 [==============================] - 1s 124us/step - loss: 0.2187 - acc: 0.9471 - val_loss: 0.9177 - val_acc: 0.8130\n",
      "Epoch 12/20\n",
      "7982/7982 [==============================] - 1s 127us/step - loss: 0.1873 - acc: 0.9508 - val_loss: 0.9027 - val_acc: 0.8130\n",
      "Epoch 13/20\n",
      "7982/7982 [==============================] - 1s 126us/step - loss: 0.1703 - acc: 0.9521 - val_loss: 0.9323 - val_acc: 0.8110\n",
      "Epoch 14/20\n",
      "7982/7982 [==============================] - 1s 126us/step - loss: 0.1536 - acc: 0.9554 - val_loss: 0.9689 - val_acc: 0.8050\n",
      "Epoch 15/20\n",
      "7982/7982 [==============================] - 1s 122us/step - loss: 0.1390 - acc: 0.9560 - val_loss: 0.9686 - val_acc: 0.8150\n",
      "Epoch 16/20\n",
      "7982/7982 [==============================] - 1s 129us/step - loss: 0.1313 - acc: 0.9560 - val_loss: 1.0220 - val_acc: 0.8060\n",
      "Epoch 17/20\n",
      "7982/7982 [==============================] - 1s 123us/step - loss: 0.1217 - acc: 0.9579 - val_loss: 1.0254 - val_acc: 0.7970\n",
      "Epoch 18/20\n",
      "7982/7982 [==============================] - 1s 126us/step - loss: 0.1198 - acc: 0.9582 - val_loss: 1.0430 - val_acc: 0.8060\n",
      "Epoch 19/20\n",
      "7982/7982 [==============================] - 1s 125us/step - loss: 0.1138 - acc: 0.9597 - val_loss: 1.0955 - val_acc: 0.7970\n",
      "Epoch 20/20\n",
      "7982/7982 [==============================] - 1s 125us/step - loss: 0.1111 - acc: 0.9593 - val_loss: 1.0674 - val_acc: 0.8020\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(partial_x_train,\n",
    "                    partial_y_train,\n",
    "                    epochs=20,\n",
    "                    batch_size=512,\n",
    "                    validation_data=(x_val, y_val))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's display its loss and accuracy curves:\n",
    "\n",
    "我们来绘制损失曲线和精度曲线："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xl8VOXZ//HPBUQQWQWsCErArQIGiCmiqOD6c9+rIq7VorZWrU/7k8e9Kq27FuVni1VqS9wefVxqXYqVFlc0ICCIFFTQCEJA2QSVwPX74z4zGcIkmZCczCT5vl+v85qZc+5z5pqTybnm3Pd97mPujoiICECLbAcgIiK5Q0lBRESSlBRERCRJSUFERJKUFEREJElJQUREkpQUpF6ZWUszW2tmu9Rn2Wwys93MrN77bpvZYWa2MOX1PDM7MJOyW/FefzKzq7d2/Wq2e4uZ/bm+tyvZ0yrbAUh2mdnalJdtge+AjdHri9y9uDbbc/eNQLv6LtscuPue9bEdM7sQOMvdh6ds+8L62LY0fUoKzZy7Jw/K0S/RC9391arKm1krdy9viNhEpOGp+kiqFVUPPGFmj5nZGuAsM9vPzN4xs5VmtsTMxppZXlS+lZm5meVHrydGy18yszVm9raZ9a5t2Wj5UWb2HzNbZWb3mdmbZnZeFXFnEuNFZrbAzL42s7Ep67Y0s3vMbIWZfQwcWc3+udbMHq80b5yZ3R09v9DM5kaf5+PoV3xV2yo1s+HR87Zm9tcotjnAPmne95Nou3PM7Pho/t7A/cCBUdXc8pR9e2PK+hdHn32FmT1rZt0z2Tc1MbMTo3hWmtlrZrZnyrKrzWyxma02s49SPusQM5sezV9qZndk+n4SA3fXpAl3B1gIHFZp3i3A98BxhB8R2wI/AvYlnGn2Af4DXBqVbwU4kB+9nggsB4qAPOAJYOJWlN0BWAOcEC27EtgAnFfFZ8kkxueAjkA+8FXiswOXAnOAnkAXYEr4V0n7Pn2AtcB2KdteBhRFr4+LyhhwCLAeKIiWHQYsTNlWKTA8en4n8C+gM9AL+LBS2dOA7tHf5Mwohh9Eyy4E/lUpzonAjdHzI6IYBwJtgP8HvJbJvknz+W8B/hw93yuK45Dob3R1tN/zgH7AImDHqGxvoE/0/D1gRPS8PbBvtv8XmvOkMwXJxBvu/jd33+Tu6939PXef6u7l7v4JMB4YVs36T7l7ibtvAIoJB6Palj0WmOHuz0XL7iEkkLQyjPF37r7K3RcSDsCJ9zoNuMfdS919BXBrNe/zCTCbkKwADgdWuntJtPxv7v6JB68B/wTSNiZXchpwi7t/7e6LCL/+U9/3SXdfEv1NHiUk9KIMtgswEviTu89w92+B0cAwM+uZUqaqfVOdM4Dn3f216G90K9CBkJzLCQmoX1QF+Wm07yAk993NrIu7r3H3qRl+DomBkoJk4vPUF2b2QzP7u5l9aWargZuArtWs/2XK83VU37hcVdmdUuNwdyf8sk4rwxgzei/CL9zqPAqMiJ6fSUhmiTiONbOpZvaVma0k/Eqvbl8ldK8uBjM7z8xmRtU0K4EfZrhdCJ8vuT13Xw18DfRIKVObv1lV291E+Bv1cPd5wH8R/g7LourIHaOi5wN9gXlm9q6ZHZ3h55AYKClIJip3x/wj4dfxbu7eAbieUD0SpyWE6hwAzMzY/CBWWV1iXALsnPK6pi6zTwCHRb+0TyAkCcxsW+Ap4HeEqp1OwD8yjOPLqmIwsz7AA8AlQJdoux+lbLem7rOLCVVSie21J1RTfZFBXLXZbgvC3+wLAHef6O5DCVVHLQn7BXef5+5nEKoI7wKeNrM2dYxFtpKSgmyN9sAq4Bsz2wu4qAHe8wWg0MyOM7NWwOVAt5hifBK4wsx6mFkX4KrqCrv7UuANYAIwz93nR4taA9sAZcBGMzsWOLQWMVxtZp0sXMdxacqydoQDfxkhP15IOFNIWAr0TDSsp/EYcIGZFZhZa8LB+XV3r/LMqxYxH29mw6P3/jWhHWiqme1lZgdH77c+mjYSPsDZZtY1OrNYFX22TXWMRbaSkoJsjf8CziX8w/+R8Es5VtGB93TgbmAFsCvwPuG6ivqO8QFC3f8HhEbQpzJY51FCw/GjKTGvBH4JPENorD2VkNwycQPhjGUh8BLwl5TtzgLGAu9GZX4IpNbDTwLmA0vNLLUaKLH+y4RqnGei9XchtDPUibvPIezzBwgJ60jg+Kh9oTVwO6Ed6EvCmcm10apHA3Mt9G67Ezjd3b+vazyydSxUzYo0LmbWklBdcaq7v57teESaCp0pSKNhZkeaWceoCuI6Qo+Wd7MclkiToqQgjckBwCeEKogjgRPdvarqIxHZCqo+EhGRJJ0piIhIUqMbEK9r166en5+f7TBERBqVadOmLXf36rpxA40wKeTn51NSUpLtMEREGhUzq+nKfEDVRyIikkJJQUREkpQUREQkqdG1KYhIw9qwYQOlpaV8++232Q5FMtCmTRt69uxJXl5VQ19VT0lBRKpVWlpK+/btyc/PJwxOK7nK3VmxYgWlpaX07t275hXSaBbVR8XFkJ8PLVqEx+Ja3YpepHn79ttv6dKlixJCI2BmdOnSpU5ndU3+TKG4GEaNgnXrwutFi8JrgJF1HhdSpHlQQmg86vq3avJnCtdcU5EQEtatC/NFRGRzsSUFM9vZzCab2Vwzm2Nml6cpM9zMVpnZjGi6vr7j+Oyz2s0XkdyyYsUKBg4cyMCBA9lxxx3p0aNH8vX332d224Xzzz+fefPmVVtm3LhxFNdT3fIBBxzAjBkz6mVbDS3O6qNy4L/cfXp0u79pZjbJ3T+sVO51dz82riB22SVUGaWbLyL1r7g4nIl/9ln4Pxszpm5VtV26dEkeYG+88UbatWvHr371q83KuDvuTosW6X/nTpgwocb3+fnPf771QTYhsZ0puPsSd58ePV8DzKX6e+rGYswYaNt283lt24b5IlK/Em14ixaBe0UbXhydOxYsWED//v25+OKLKSwsZMmSJYwaNYqioiL69evHTTfdlCyb+OVeXl5Op06dGD16NAMGDGC//fZj2bJlAFx77bXce++9yfKjR49m8ODB7Lnnnrz11lsAfPPNN5xyyikMGDCAESNGUFRUVOMZwcSJE9l7773p378/V199NQDl5eWcffbZyfljx44F4J577qFv374MGDCAs846q973WSYapE3BzPKBQWx+y8CE/cxsppm9ZGb9qlh/lJmVmFlJWVlZrd575EgYPx569QKz8Dh+vBqZReLQ0G14H374IRdccAHvv/8+PXr04NZbb6WkpISZM2cyadIkPvywcsUErFq1imHDhjFz5kz2228/Hn744bTbdnfeffdd7rjjjmSCue+++9hxxx2ZOXMmo0eP5v333682vtLSUq699lomT57M+++/z5tvvskLL7zAtGnTWL58OR988AGzZ8/mnHPOAeD2229nxowZzJw5k/vvv7+Oe2frxJ4UzKwd8DRwhbuvrrR4OtDL3QcA9wHPptuGu4939yJ3L+rWrcZB/rYwciQsXAibNoVHJQSReDR0G96uu+7Kj370o+Trxx57jMLCQgoLC5k7d27apLDtttty1FFHAbDPPvuwcOHCtNs++eSTtyjzxhtvcMYZZwAwYMAA+vVL+zs2aerUqRxyyCF07dqVvLw8zjzzTKZMmcJuu+3GvHnzuPzyy3nllVfo2LEjAP369eOss86iuLh4qy8+q6tYk4KZ5RESQrG7/2/l5e6+2t3XRs9fBPLMrGucMYlIfKpqq4urDW+77bZLPp8/fz6///3vee2115g1axZHHnlk2v7622yzTfJ5y5YtKS8vT7vt1q1bb1Gmtjclq6p8ly5dmDVrFgcccABjx47loosuAuCVV17h4osv5t1336WoqIiNGzfW6v3qQ5y9jwx4CJjr7ndXUWbHqBxmNjiKZ0VcMYlIvLLZhrd69Wrat29Phw4dWLJkCa+88kq9v8cBBxzAk08+CcAHH3yQ9kwk1ZAhQ5g8eTIrVqygvLycxx9/nGHDhlFWVoa78+Mf/5jf/OY3TJ8+nY0bN1JaWsohhxzCHXfcQVlZGesq18U1gDh7Hw0FzgY+MLNES8zVwC4A7v4H4FTgEjMrB9YDZ7juDyrSaCWqZuuz91GmCgsL6du3L/3796dPnz4MHTq03t/jF7/4Beeccw4FBQUUFhbSv3//ZNVPOj179uSmm25i+PDhuDvHHXccxxxzDNOnT+eCCy7A3TEzbrvtNsrLyznzzDNZs2YNmzZt4qqrrqJ9+/b1/hlq0uju0VxUVOS6yY5Iw5k7dy577bVXtsPICeXl5ZSXl9OmTRvmz5/PEUccwfz582nVKrcGh0j3NzOzae5eVNO6ufVJRERy2Nq1azn00EMpLy/H3fnjH/+YcwmhrprWpxERiVGnTp2YNm1atsOIVZMf+0hERDKnpCAiIklKCiIikqSkICIiSUoKIpLThg8fvsWFaPfeey8/+9nPql2vXbt2ACxevJhTTz21ym3X1MX93nvv3ewisqOPPpqVK1dmEnq1brzxRu688846b6e+KSmISE4bMWIEjz/++GbzHn/8cUaMGJHR+jvttBNPPfXUVr9/5aTw4osv0qlTp63eXq5TUhCRnHbqqafywgsv8N133wGwcOFCFi9ezAEHHJC8bqCwsJC9996b5557bov1Fy5cSP/+/QFYv349Z5xxBgUFBZx++umsX78+We6SSy5JDrt9ww03ADB27FgWL17MwQcfzMEHHwxAfn4+y5cvB+Duu++mf//+9O/fPzns9sKFC9lrr7346U9/Sr9+/TjiiCM2e590ZsyYwZAhQygoKOCkk07i66+/Tr5/3759KSgoSA7E9+9//zt5k6FBgwaxZs2ard636eg6BRHJ2BVXQH3fUGzgQIiOp2l16dKFwYMH8/LLL3PCCSfw+OOPc/rpp2NmtGnThmeeeYYOHTqwfPlyhgwZwvHHH1/lfYofeOAB2rZty6xZs5g1axaFhYXJZWPGjGH77bdn48aNHHroocyaNYvLLruMu+++m8mTJ9O16+ZjdU6bNo0JEyYwdepU3J19992XYcOG0blzZ+bPn89jjz3Ggw8+yGmnncbTTz9d7f0RzjnnHO677z6GDRvG9ddfz29+8xvuvfdebr31Vj799FNat26drLK68847GTduHEOHDmXt2rW0adOmFnu7ZjpTEJGcl1qFlFp15O5cffXVFBQUcNhhh/HFF1+wdOnSKrczZcqU5MG5oKCAgoKC5LInn3ySwsJCBg0axJw5c2oc7O6NN97gpJNOYrvttqNdu3acfPLJvP766wD07t2bgQMHAtUPzw3h/g4rV65k2LBhAJx77rlMmTIlGePIkSOZOHFi8srpoUOHcuWVVzJ27FhWrlxZ71dU60xBRDJW3S/6OJ144olceeWVTJ8+nfXr1yd/4RcXF1NWVsa0adPIy8sjPz8/7XDZqdKdRXz66afceeedvPfee3Tu3Jnzzjuvxu1UN25cYthtCENv11R9VJW///3vTJkyheeff56bb76ZOXPmMHr0aI455hhefPFFhgwZwquvvsoPf/jDrdp+OjpTEJGc165dO4YPH85PfvKTzRqYV61axQ477EBeXh6TJ09mUbobsqc46KCDKI7uDTp79mxmzZoFhGG3t9tuOzp27MjSpUt56aWXkuu0b98+bb39QQcdxLPPPsu6dev45ptveOaZZzjwwANr/dk6duxI586dk2cZf/3rXxk2bBibNm3i888/5+CDD+b2229n5cqVrF27lo8//pi9996bq666iqKiIj766KNav2d1dKYgIo3CiBEjOPnkkzfriTRy5EiOO+44ioqKGDhwYI2/mC+55BLOP/98CgoKGDhwIIMHDwbCXdQGDRpEv379thh2e9SoURx11FF0796dyZMnJ+cXFhZy3nnnJbdx4YUXMmjQoGqriqryyCOPcPHFF7Nu3Tr69OnDhAkT2LhxI2eddRarVq3C3fnlL39Jp06duO6665g8eTItW7akb9++ybvI1RcNnS0i1dLQ2Y1PXYbOVvWRiIgkKSmIiEiSkoKI1KixVTM3Z3X9WykpiEi12rRpw4oVK5QYGgF3Z8WKFXW6oE29j0SkWj179qS0tJSysrJshyIZaNOmDT179tzq9ZUURKRaeXl59O7dO9thSANR9ZGIiCQpKYiISJKSgoiIJCkpiIhIkpKCiIgkKSmIiEiSkoKIiCQpKYiISJKSgoiIJCkpiIhIkpKCiIgkxZYUzGxnM5tsZnPNbI6ZXZ6mjJnZWDNbYGazzKwwrnhERKRmcQ6IVw78l7tPN7P2wDQzm+TuH6aUOQrYPZr2BR6IHkVEJAtiO1Nw9yXuPj16vgaYC/SoVOwE4C8evAN0MrPuccUkIiLVa5A2BTPLBwYBUyst6gF8nvK6lC0TB2Y2ysxKzKxEY7qLiMQn9qRgZu2Ap4Er3H115cVpVtni9k7uPt7di9y9qFu3bnGEKSIixJwUzCyPkBCK3f1/0xQpBXZOed0TWBxnTCIiUrU4ex8Z8BAw193vrqLY88A5US+kIcAqd18SV0wiIlK9OHsfDQXOBj4wsxnRvKuBXQDc/Q/Ai8DRwAJgHXB+jPGIiEgNYksK7v4G6dsMUss48PO4YhARkdrRFc0iIpKkpCAiIklKCiIikqSkICIiSUoKIiKSpKQgIiJJSgoiIpKkpCAiIklKCiIikqSkICIiSUoKIiKSpKQgIiJJSgoiIpKkpCAiIklKCiIiktRsksKGDfDMM+Bb3AFaREQSmk1SeOQROPlkePXVbEciIpK7mk1SOPts2GUXuPpqnS2IiFSl2SSF1q3hxhuhpASefTbb0YiI5KZmkxQgnC3suSdcey1s3JjtaEREck+zSgqtWsHNN8OHH8Kjj2Y7GhGR3NOskgLAKafAoEGhKun777MdjYhIbml2SaFFCxgzBj75BB5+ONvRiIjklmaXFACOPBIOOABuugnWr892NCIiuaNZJgUz+O1vYckSGDcu29GIiOSOZpkUAA48MJwx/O53sHp1tqMREckNzTYpANxyC3z1Fdx9d7YjERHJDc06KeyzD5x6Ktx1Fyxfnu1oRESyr1knBQiNzevWwa23ZjsSEZHsa/ZJYa+94Jxz4P77obQ029GIiGRXs08KADfcAJs2hTYGEZHmLLakYGYPm9kyM5tdxfLhZrbKzGZE0/VxxVKT/Hy46CJ46CFYsCBbUYiIZF+cZwp/Bo6soczr7j4wmm6KMZYaXXMN5OWF4S9ERJqr2JKCu08Bvopr+/Vtxx3h8svDQHkffJDtaEREsiPbbQr7mdlMM3vJzPplORZ+/Wvo0AGuuy7bkYiIZEc2k8J0oJe7DwDuA6q89Y2ZjTKzEjMrKSsriy2g7bcPieG552Dq1NjeRkQkZ2UtKbj7andfGz1/Ecgzs65VlB3v7kXuXtStW7dY47r8cujWLbQxJBQXh8boFi3CY3FxrCGIiGRNq2y9sZntCCx1dzezwYQEtSJb8SS0axcSwhVXwD//CV9+CaNGhQvcABYtCq8BRo7MXpwiInEwj+ku9mb2GDAc6AosBW4A8gDc/Q9mdilwCVAOrAeudPe3atpuUVGRl5SUxBJzwrffwh57wE47hZFUP/tsyzK9esHChbGGISJSb8xsmrsX1VgurqQQl4ZIChCuWbjwwqqXm4UL3kREGoNMk0K2ex/lrHPPhd13D9cupLPLLg0bj4hIQ8goKZjZrmbWOno+3MwuM7NO8YaWXa1awc03w4YNsM02my9r2zbc0lNEpKnJ9EzhaWCjme0GPAT0Bh6NLaoc8eMfw4AB0KlTODMwC20J48erkVlEmqZMk8Imdy8HTgLudfdfAt3jCys3tGgRBslbtiz0SNq0KTQuKyGISFOVaVLYYGYjgHOBF6J5VdS2Ny3HHAP77Rfuu7B+fbajERGJV6ZJ4XxgP2CMu39qZr2BifGFlTvM4Le/hS++gAceyHY0IiLxqnWXVDPrDOzs7rPiCal6DdUltbIjjoDp02H27DB4nohIY1KvXVLN7F9m1sHMtgdmAhPMrFnd7v7220P10f77w3/+k+1oRETikWn1UUd3Xw2cDExw932Aw+ILK/cMHAiTJ8OaNTB0qAbME5GmKdOk0MrMugOnUdHQ3OwMHgxvvx2G1z74YHih2e4JEWmqMk0KNwGvAB+7+3tm1geYH19YuWu33eCtt6BfPzjxRPjTn7IdkYhI/ckoKbj7/7h7gbtfEr3+xN1PiTe03PWDH4SqpMMOg5/+NHRXbWRDSImIpJVpQ3NPM3vGzJaZ2VIze9rMesYdXC5r1w7+9rcwRtINN8BFF0F5ebajEhGpm0yrjyYAzwM7AT2Av0XzmrW8PJgwAa6+Gh58EE4+ueK+CyIijVGmSaGbu09w9/Jo+jMQ7y3QGgmzMDjeuHGh4fmQQ2D58mxHJSKydTJNCsvN7CwzaxlNZ5EDd0nLJT/7GTz9NMycGbqsfvpptiMSEam9TJPCTwjdUb8ElgCnEoa+kBQnnQSvvgplZeEit/ffz3ZEIiK1k2nvo8/c/Xh37+buO7j7iYQL2aSSoUPhzTfDPRgOOggmTcp2RCIimavLndeurLcompi99grXMvTpA0cfDRObxdCBItIU1CUpWL1F0QT16AFTpsCBB8LZZ4exk3Qtg4jkurokBR3iatCxI7z0EpxxBlx1FVx2me7JICK5rdqkYGZrzGx1mmkN4ZoFqUHr1lBcDFdeCfffD/n5oQvr119nOzIRkS1VmxTcvb27d0gztXf3Vg0VZGPXogXcdRf861+wzz5w7bXhns+/+lW4eY+ISK6oS/WR1NKwYfDiizBjBhx/PNxzD/TuDRdcAB99lO3oRESUFLJiwIBQpbRgAYwaBY8+Cn37hmEydJ8GEckmJYUGUFwc2hJatAiPxcVhfu/eoZ1h0SK45ppQvTRkCAwfDi+/rN5KItLwlBRiVlwczgYWLQoH+UWLwutEYgDYYQe4+eaw7K67whnEUUfBoEHw2GMafVWksXOHefPgs89g06ZsR1M980b2c7SoqMhLSkqyHUbG8vPDwb6yXr1g4cL063z/fahSuu220NbQu3dolD7/fNh22zijFZH6snhxGPZm0qTw+OWXYX6bNrDrrrD77uGmXbvvXjHttFOoUYiDmU1z96IayykpxKtFi/TVQGY1/2LYtCncs+HWW+Gdd6BbN/jlL+HnPw+3BBWR3LFmDfz73xWJ4MMPw/xu3eDQQ8O0aRPMn18xffwxfPddxTa23bYiYVROGjvtFI4bW0tJIUdszZlCZe7w+uvw29/CK69Ap07hQrjLL4ftt6/PaEUkU+Xl8N57IQFMmhR+uJWXhzOBgw6Cww8Pd2csKKj61/+mTVBaunmiWLCgImF8/31F2bZtYfRouO66rYtXSSFHJNoUUm++07YtjB8PI0fWfnslJeHit2efDXd/+9nPwoVxP/hB/cUsIltyh//8p6I6aPJkWL06/HrfZ5+QAA4/PIyQ3KZN3d9v40b4/PPNE8WwYXDCCVu3PSWFHFJcHHoXffZZuGhtzJitSwipZs8OZw5PPBFGZP3pT+HXv4add66fmEWaiw0bwnD3S5fCsmXhsfK0bFloIygrC+v07l1xJnDIIdClS3Y/QyaynhTM7GHgWGCZu/dPs9yA3wNHA+uA89x9ek3bbYxJIU7z54c2h7/8JfxiOe+8cIrZp0+2IxOpP5s2hU4XK1eGg3h5+ZaP6ealLvv+e1ixYssD/4oqbhfWpk04A0+diopCMmiM/1+5kBQOAtYCf6kiKRwN/IKQFPYFfu/u+9a0XSWF9BYtCiOxPvRQ+Ac480z47/8Ow3iLNEZLloSqmn/8I1TXLF1a9222b7/5QX6HHbY88Cemdu3q1rCba7KeFKIg8oEXqkgKfwT+5e6PRa/nAcPdfUl121RSqN6SJeFahwceCCOynnJKqLoaODDbkYlUb9260KEikQg++CDM79atoqpmp50gLw9atdryMd28dGWaq0yTQjZ3UQ/g85TXpdG8LZKCmY0CRgHssssuDRJcY9W9O9x5Z6hCuvdeuO8+eOopOPbYcOYwZEh8/aBFamPTJpg1qyIJvP566J65zTbhPiS33QZHHFF97x2pf9lMCulOzNKetrj7eGA8hDOFOINqKrp2hVtuCRe9jRsXBt8bOhQ6dw6JYcgQ2G8/GDw43PdBpCGkVglNmhTq9wH69w/X3xx+eOjO2bZtduNszrKZFEqB1L4yPYHFWYqlyerUKVQfXX55OGN4883QnzoxtpJZGIwvkSSGDAntEPpl1ry5h942M2aEaebM0OMtcZOo1Lr2xPPKj5Xnff99xbU5O+wQEsARR1RUC0luyGabwjHApVQ0NI9198E1bVNtCvVj1apw4c3bb4ck8c478NVXYVmHDrDvvhVJYt99dZFcU7ZhQ+jZM3NmRRKYMWPzXjm77hqqcdq3D68Th42qHtPNMwvjeR1+uKqEsiHrDc1m9hgwHOgKLAVuAPIA3P0PUZfU+4EjCV1Sz3f3Go/2SgrxcA/dWxNJ4u23Q0NfYiiOPfcMyaGgAPr1C1PPng3XO+Orr8INiXbdVVULdbFqVcXBP/E4e3bFlbNt2oSqnIEDK6a999awKk1B1pNCXJQUGs7ateFsIpEk3n13826B7duHqqdEkkg839pksWJFxZWblR8Tty9t1Sr82tx//9BGsv/+0KNH/XzepiIxdMK8eeEK3MTjRx9tPuRKt26bH/wHDoQ99mjePXSaMiUFicXy5WGgrzlzwpR4nmgwhPCrsm/fzRNGv37h4P3VV1se8BPPU+9bbRau/k4dFKx793D28tZbIUEl6rd32WXzJFFQEP+BzT10oVy9OgyEtnp1zdM334R906VL6AhQ+THxPC8vsxhWrtzywD9vXtifiX0Dob/9HnuEs729965IADvu2LT64Uv1lBSkQS1fvnmSSEyJYQEgdDVMHeAr3YE/8dinD7RuXfX7bdgQqj/efDMkiTffrLjfddu2oaorkSiGDAm9rqqybl2Iv6qprCw8rlgRql8SB/lMxsXPywu9uzp0gO22Cwlk+fJwFlaV1MSRmjQ6dqxPqAXtAAAMlElEQVQ4A5g3b/N927JlGHphzz3DlEgCe+wRkqkO/qKk0ITEMXZSQykrq0gUn3wSepkkDvy9e9fPwGEJn39ekSTeeivUl2/cGJb17RuGKCgv3/Jgn/qrOpVZxQG5W7fwPHGAz2Rq377qxPbddyHJJGLJ5PnateFK29QDfuKxT5+QdEWqoqTQRNT3KKvNyTffhDaRRKJ4//2w7xK/wBNTt25bzuvaNXTnbdky25+iQnm56vtl6ykpNBH1cT8GEZFMk4J6Cue4zz6r3XwRkbpQUshxVQ31pCGgRCQOSgo5bsyYLS/Wats2zBcRqW9KCjlu5MjQqNyrV+gN06uXGplFJD7qy9AIjBypJCAiDUNnCiIikqSkICIiSUoKIiKSpKQgIiJJSgoiIpKkpCAiIklKCs1AcXEYQ6lFi/BYXJztiEQkV+k6hSau8iirixaF16BrH0RkSzpTaOKuuWbzYbchvL7mmuzEIyK5TUmhidMoqyJSG0oKTZxGWRWR2lBSaOI0yqqI1IaSQhOnUVZFpDbU+6gZ0CirIpIpnSmIiEiSkoKIiCQpKYiISJKSgmREQ2WINA9qaJYaaagMkeZDZwpSIw2VIdJ8KClIjTRUhkjzoaQgNdJQGSLNR6xJwcyONLN5ZrbAzEanWX6emZWZ2YxoujDOeGTraKgMkeYjtqRgZi2BccBRQF9ghJn1TVP0CXcfGE1/iise2XoaKkOk+Yiz99FgYIG7fwJgZo8DJwAfxvieEhMNlSHSPMRZfdQD+DzldWk0r7JTzGyWmT1lZjun25CZjTKzEjMrKSsriyNWEREh3qRgaeZ5pdd/A/LdvQB4FXgk3Ybcfby7F7l7Ubdu3eo5TGkIuvhNpHGIMymUAqm//HsCi1MLuPsKd/8uevkgsE+M8UiWJC5+W7QI3CsuflNiEMk9cSaF94Ddzay3mW0DnAE8n1rAzLqnvDwemBtjPJIluvhNpPGIraHZ3cvN7FLgFaAl8LC7zzGzm4ASd38euMzMjgfKga+A8+KKR7JHF7+JNB7mXrmaP7cVFRV5SUlJtsOQWsjPD1VGlfXqBQsXNnQ0Is2TmU1z96KayumKZomdLn4TaTyUFCR2uvhNpPFQUpAGMXJkqCratCk81jYhqEurSMPQ/RQk5+l+DiINR2cKkvPUpVWk4SgpSM5Tl1aRhqOkIDlP93MQaThKCpLz6qNLqxqqRTKjpCA5r65dWjX2kkjmdEWzNHm6olpEVzSLJKmhWiRzSgrS5NVHQ7XaJKS5UFKQJq+uDdVqk5DmRElBmry6NlTr4jlpTpQUpFmoy9hL9dEmoeonaSyUFERqUNc2CVU/SWOipCBSg7q2Saj6SRoTJQWRGtS1TULVT9KYKCmIZKAubRK5UP2kpCKZUlIQiVm2q5+UVKQ2lBREYpbt6qdcSCrSeCgpiDSAbFY/ZTupQN3PNHSm0nCUFERyXF2rn7KdVOp6ppEL1V/NKim5e6Oa9tlnHxdpbiZOdO/Vy90sPE6cWLt127Z1D4fUMLVtm/k2evXafN3E1KtX41i/rp+/rusntrG1f7/6WN/dHSjxDI6xWT/I13ZSUhCpvWwmFbP0B3WzhllfSSnINCnofgoiUqPi4tCG8NlnodppzJjM20Xqej+Luq7fokU4lFZmFtp44l4/258/QfdTEJF6U5eG8rq2iWS7TSXbbTINfT8QJQURiVVdu+TWdf3mnpRqLZM6plya1KYgIrWVzYZetSnETG0KItLY1KVNpj7Wh8zbFJQURESaATU0i4hIrcWaFMzsSDObZ2YLzGx0muWtzeyJaPlUM8uPMx4REalebEnBzFoC44CjgL7ACDPrW6nYBcDX7r4bcA9wW1zxiIhIzeI8UxgMLHD3T9z9e+Bx4IRKZU4AHomePwUcamYWY0wiIlKNOJNCD+DzlNel0by0Zdy9HFgFdKm8ITMbZWYlZlZSVlYWU7giItIqxm2n+8VfuatTJmVw9/HAeAAzKzOzNBd954SuwPJsB1GNXI8Pcj9GxVc3iq9u6hJfr0wKxZkUSoGdU173BBZXUabUzFoBHYGvqtuou3erzyDrk5mVZNLlK1tyPT7I/RgVX90ovrppiPjirD56D9jdzHqb2TbAGcDzlco8D5wbPT8VeM0b24UTIiJNSGxnCu5ebmaXAq8ALYGH3X2Omd1EuNz6eeAh4K9mtoBwhnBGXPGIiEjN4qw+wt1fBF6sNO/6lOffAj+OM4YGNj7bAdQg1+OD3I9R8dWN4qub2ONrdMNciIhIfDTMhYiIJCkpiIhIkpJCLZnZzmY22czmmtkcM7s8TZnhZrbKzGZE0/XpthVjjAvN7IPovbcYUtaCsdGYU7PMrLABY9szZb/MMLPVZnZFpTINvv/M7GEzW2Zms1PmbW9mk8xsfvTYuYp1z43KzDezc9OViSm+O8zso+hv+IyZdapi3Wq/DzHGd6OZfZHydzy6inWrHSMtxvieSIltoZnNqGLdWPdfVceUrH3/MrnpgqaKCegOFEbP2wP/AfpWKjMceCGLMS4Eulaz/GjgJcLFg0OAqVmKsyXwJdAr2/sPOAgoBGanzLsdGB09Hw3clma97YFPosfO0fPODRTfEUCr6Plt6eLL5PsQY3w3Ar/K4DvwMdAH2AaYWfn/Ka74Ki2/C7g+G/uvqmNKtr5/OlOoJXdf4u7To+drgLlsOXxHrjsB+IsH7wCdzKx7FuI4FPjY3bN+hbq7T2HLCydTx+Z6BDgxzar/B5jk7l+5+9fAJODIhojP3f/hYXgYgHcIF4hmRRX7LxOZjJFWZ9XFF423dhrwWH2/byaqOaZk5funpFAH0VDfg4CpaRbvZ2YzzewlM+vXoIGFoUL+YWbTzGxUmuWZjEvVEM6g6n/EbO6/hB+4+xII/7jADmnK5Mq+/Anh7C+dmr4Pcbo0qt56uIrqj1zYfwcCS919fhXLG2z/VTqmZOX7p6SwlcysHfA0cIW7r660eDqhSmQAcB/wbAOHN9TdCwnDlv/czA6qtDyjMafiFF3lfjzwP2kWZ3v/1UYu7MtrgHKguIoiNX0f4vIAsCswEFhCqKKpLOv7DxhB9WcJDbL/ajimVLlamnl12n9KClvBzPIIf7xid//fysvdfbW7r42evwjkmVnXhorP3RdHj8uAZwin6KkyGZcqbkcB0919aeUF2d5/KZYmqtWix2VpymR1X0YNi8cCIz2qZK4sg+9DLNx9qbtvdPdNwINVvG+2918r4GTgiarKNMT+q+KYkpXvn5JCLUX1jw8Bc9397irK7BiVw8wGE/bzigaKbzsza594TmiMnF2p2PPAOVEvpCHAqsRpagOq8tdZNvdfJaljc50LPJemzCvAEWbWOaoeOSKaFzszOxK4Cjje3ddVUSaT70Nc8aW2U51UxftmMkZanA4DPnL30nQLG2L/VXNMyc73L64W9aY6AQcQTs9mATOi6WjgYuDiqMylwBxCT4p3gP0bML4+0fvOjGK4JpqfGp8R7or3MfABUNTA+7At4SDfMWVeVvcfIUEtATYQfn1dQLi3xz+B+dHj9lHZIuBPKev+BFgQTec3YHwLCPXJie/hH6KyOwEvVvd9aKD4/hp9v2YRDnDdK8cXvT6a0OPm44aML5r/58T3LqVsg+6/ao4pWfn+aZgLERFJUvWRiIgkKSmIiEiSkoKIiCQpKYiISJKSgoiIJCkpiETMbKNtPoJrvY3YaWb5qSN0iuSqWG/HKdLIrHf3gdkOQiSbdKYgUoNoPP3bzOzdaNotmt/LzP4ZDfj2TzPbJZr/Awv3N5gZTftHm2ppZg9GY+b/w8y2jcpfZmYfRtt5PEsfUwRQUhBJtW2l6qPTU5atdvfBwP3AvdG8+wlDkBcQBqMbG80fC/zbw4B+hYQrYQF2B8a5ez9gJXBKNH80MCjazsVxfTiRTOiKZpGIma1193Zp5i8EDnH3T6KBy7509y5mtpwwdMOGaP4Sd+9qZmVAT3f/LmUb+YRx73ePXl8F5Ln7LWb2MrCWMBrssx4NBiiSDTpTEMmMV/G8qjLpfJfyfCMVbXrHEMai2geYFo3cKZIVSgoimTk95fHt6PlbhFE9AUYCb0TP/wlcAmBmLc2sQ1UbNbMWwM7uPhn4v0AnYIuzFZGGol8kIhW2tc1v3v6yuye6pbY2s6mEH1IjonmXAQ+b2a+BMuD8aP7lwHgzu4BwRnAJYYTOdFoCE82sI2H02nvcfWW9fSKRWlKbgkgNojaFIndfnu1YROKm6iMREUnSmYKIiCTpTEFERJKUFEREJElJQUREkpQUREQkSUlBRESS/j/SX1An4H9ifQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x24087396e48>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "\n",
    "epochs = range(1, len(loss) + 1)\n",
    "\n",
    "plt.plot(epochs, loss, 'bo', label='Training loss')\n",
    "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
    "plt.title('Training and validation loss')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3XmcFNW5//HPwyYgCAi4gSwadwKI4yBXMGiMAaMSUaMEb1Q0qBGJuSY3KN5ojCTG5HqNxp/X0cQsjqKGYNDrEiXEJURlUAcEIiAijiAgIrLK9vz+ONVNT9sz07NUd8/M9/169au7q05VP13TU0+dc6pOmbsjIiIC0CLfAYiISOFQUhARkSQlBRERSVJSEBGRJCUFERFJUlIQEZEkJQX5HDNraWabzKxXQ5bNJzP7gpk1+PnXZnaqmS1Pef+2mQ3LpmwdPut+M7u+rsuLZKNVvgOQ+jOzTSlv2wOfAbui95e7e2lt1ufuu4AODV22OXD3IxpiPWZ2GXChuw9PWfdlDbFukeooKTQB7p7cKUdHope5+/NVlTezVu6+MxexidREv8fCouajZsDMbjGzR8zsYTPbCFxoZkPM7BUz+8TMVpnZnWbWOirfyszczPpE7x+M5j9tZhvN7J9m1re2ZaP5I81ssZltMLO7zOwfZnZxFXFnE+PlZrbUzNab2Z0py7Y0s/8xs3Vm9g4woprtc4OZTU2bdreZ3R69vszMFkXf553oKL6qdVWY2fDodXsz+2MU2wLguAyfuyxa7wIzOyua/kXg18CwqGnuo5Rte1PK8ldE332dmT1uZgdms21qs50T8ZjZ82b2sZl9aGb/mfI5/xVtk0/NrMzMDsrUVGdmLyf+ztH2fDH6nI+BG8zsMDObFX2Xj6Lt1ill+d7Rd1wbzf+VmbWNYj4qpdyBZrbFzLpW9X2lBu6uRxN6AMuBU9Om3QJsB84kHAi0A44HBhNqi4cAi4EJUflWgAN9ovcPAh8BRUBr4BHgwTqU3Q/YCIyK5v0HsAO4uIrvkk2MfwE6AX2AjxPfHZgALAB6Al2BF8PPPePnHAJsAvZOWfcaoCh6f2ZUxoBTgK1A/2jeqcDylHVVAMOj178E/g50AXoDC9PKfgM4MPqbfDOKYf9o3mXA39PifBC4KXp9WhTjQKAt8P+Av2WzbWq5nTsBq4HvAnsB+wDF0bzrgHLgsOg7DAT2Bb6Qvq2BlxN/5+i77QSuBFoSfo+HA18G2kS/k38Av0z5Pm9F23PvqPyJ0bwSYErK51wLTM/3/2FjfuQ9AD0a+A9adVL4Ww3LfR94LHqdaUf/vyllzwLeqkPZccBLKfMMWEUVSSHLGE9Imf9n4PvR6xcJzWiJeaen76jS1v0K8M3o9UhgcTVlnwSuil5XlxRWpP4tgO+kls2w3reAr0Wva0oKvwd+mjJvH0I/Us+atk0tt/O/A2VVlHsnEW/a9GySwrIaYjgXmBO9HgZ8CLTMUO5E4F3AovdvAqMb+v+qOT3UfNR8vJ/6xsyONLP/i5oDPgVuBrpVs/yHKa+3UH3nclVlD0qNw8N/cUVVK8kyxqw+C3ivmngBHgLGRK+/CSQ7583sDDN7NWo++YRwlF7dtko4sLoYzOxiMyuPmkA+AY7Mcr0Qvl9yfe7+KbAe6JFSJqu/WQ3b+WBgaRUxHExIDHWR/ns8wMweNbMPohh+lxbDcg8nNVTi7v8g1DqGmlk/oBfwf3WMSVCfQnOSfjrmvYQj0y+4+z7AjwhH7nFaRTiSBcDMjMo7sXT1iXEVYWeSUNMps48Ap5pZT0Lz1kNRjO2APwE/IzTtdAb+mmUcH1YVg5kdAtxDaELpGq33Xynrren02ZWEJqnE+joSmqk+yCKudNVt5/eBQ6tYrqp5m6OY2qdMOyCtTPr3+znhrLkvRjFcnBZDbzNrWUUcfwAuJNRqHnX3z6ooJ1lQUmi+OgIbgM1RR93lOfjMJ4FBZnammbUitFN3jynGR4FrzKxH1On4w+oKu/tqQhPHA8Db7r4kmrUXoZ17LbDLzM4gtH1nG8P1ZtbZwnUcE1LmdSDsGNcS8uNlhJpCwmqgZ2qHb5qHgUvNrL+Z7UVIWi+5e5U1r2pUt51nAL3MbIKZtTGzfcysOJp3P3CLmR1qwUAz25eQDD8knNDQ0szGk5LAqolhM7DBzA4mNGEl/BNYB/zUQud9OzM7MWX+HwnNTd8kJAipByWF5uta4CJCx++9hCPlWEU73vOB2wn/5IcCbxCOEBs6xnuAmcB8YA7haL8mDxH6CB5KifkT4HvAdEJn7bmE5JaNGwk1luXA06TssNx9HnAn8FpU5kjg1ZRlnwOWAKvNLLUZKLH8M4RmnunR8r2AsVnGla7K7ezuG4CvAOcQOrYXA1+KZv8CeJywnT8ldPq2jZoFvw1cTzjp4Atp3y2TG4FiQnKaAUxLiWEncAZwFKHWsILwd0jMX074O29399m1/O6SJtE5I5JzUXPASuBcd38p3/FI42VmfyB0Xt+U71gaO128JjllZiMIzQHbCKc07iQcLYvUSdQ/Mwr4Yr5jaQrUfCS5NhRYRmhWGAF8XR2DUldm9jPCtRI/dfcV+Y6nKVDzkYiIJKmmICIiSY2uT6Fbt27ep0+ffIchItKozJ079yN3r+4UcKARJoU+ffpQVlaW7zBERBoVM6vpqn5AzUciIpJCSUFERJKUFEREJElJQUREkpQUREQkSUlBRCRmpaXQpw+0aBGeS0trWqJhl68NJQURafLyuVMuLYXx4+G998A9PI8fn/066rt8reX71m+1fRx33HEuIrn14IPuvXu7m4XnBx9sPMs/+KB7+/buYZcaHu3bZ7+O+i7fu3flZROP3r1zs3wCVdxWNf2R9518bR9KCiK115h3qo19p2yWeXmz3CyfoKQg0oTkc6ee751qY98p5/v7J2SbFNSnIFLg6tumPHkybNlSedqWLWF6NlZUMSB1VdMLbfleVdydu6rpDb38lCnQvn3lae3bh+m5WL62lBREYlbfTs5879TzvVNt7DvlsWOhpAR69waz8FxSEqbnYvlay6Y6UUgPNR9JY1Lfphv3/Ddf5LtPoCG2Yb47ygsB6lMQaRj12SE0RHtwvnfqiXU01rOPJFBSEGkA9d2hNsSZI4WwU5fGL9uk0Ohux1lUVOS6n4LkSp8+oWM3Xe/esHx5/MsnlJaGPoQVK0Jb+pQpMbYpS5NkZnPdvaimcupoFqlGfTtpG+rMkbFjQxLZvTs8KyFIXJQUpMmrz9k/9T3zJednjojUk5KCNGn1Pce/IY70dZQvjYmSgjRp9T3HX0f60tyoo1matBYtQg0hnVk4chdpLtTRLE1GPvsERJobJQUpaIXQJyDSnCgpSEFTn4BIbqlPQQqa+gREGob6FKRJUJ+ASG4pKUhBU5+ASG4pKUhBU5+ASG61yncAIjUZO1ZJQCRXVFOQ2NX3zmMikjuxJgUzG2Fmb5vZUjOblGF+bzObaWbzzOzvZtYzzngk9+p7nYGI5FZsScHMWgJ3AyOBo4ExZnZ0WrFfAn9w9/7AzcDP4opH8qO+1xmISG7FWVMoBpa6+zJ33w5MBUallTkamBm9npVhvjRy9b0fgYjkVpxJoQfwfsr7imhaqnLgnOj12UBHM+saY0ySY7rOQKRxiTMpWIZp6demfh/4kpm9AXwJ+ADY+bkVmY03szIzK1u7dm3DRyqx0XUGIo1LnEmhAjg45X1PYGVqAXdf6e6j3f1YYHI0bUP6ity9xN2L3L2oe/fuMYYsDU3XGYg0LnFepzAHOMzM+hJqABcA30wtYGbdgI/dfTdwHfDbGOORPNF1BiKNR2w1BXffCUwAngUWAY+6+wIzu9nMzoqKDQfeNrPFwP6AGhVERPJIo6SKiDQDGiVVGoyuSBZpPjT2kVQrcUVy4gK0xBXJoH4CkaZINQWplq5IFmlelBSkWroiWaR5UVKQaumKZJHmRUlBqqUrkkWaFyUFqZauSBZpXnT2kdRIVySLNB+qKYiISJKSgoiIJCkpiIhIkpKCiIgkKSmIiEiSkoKIiCQpKTQDGuVURLKl6xSaOI1yKiK1oZpCE6dRTkWkNpQUmjiNcioitaGk0MRplFMRqQ0lhSZOo5yKSG0oKTRxGuVURGpDZx81AxrlVESypZqCiIgkKSmIiEiSkoKIiCQpKYiISJKSgoiIJCkpNAIa0E5EckWnpBY4DWgnIrmkmkKB04B2IpJLqikUOA1o1zTs3h2SuXvd12EWhihp0YgO5VasgOefD4+yMjjiCBg2DE46CQYNgjZt8h2hpFNSKHC9eoUmo0zTJV7uYUf+6aewcWPl59q+3rSp4eLq0AH22Sc8OnbM/nXnznDooeF1XNavh1mz9iSCJUvC9AMOgMGD4V//giefDNPatYMTTtiTJE44AfbeO77YJDtKCgVuypTKfQqgAe3qYvNmeOopWLeu+h18+rTdu2ted6tWn98Rd+0KfftWnrb33vU7yt+9OySXTHGvXl15+q5dVa/noIPgqKPC48gj97w+4IBQG6mNbdtg9uw9SWDu3BBnhw4wfDhcdRWceiocffSeda9ZAy+/DC++CC+9BLfcEpZp1SrUHhJJYuhQ2HffOm8uqSPz+tRn86CoqMjLysryHUZOlZaGPoQVK0INYcoUdTJna/t2uP9+uPnmsONMMMvu6Dqb+W3b1n5nGif3sLNOT3IffwyLF8OiReHxr3+F+QmdOlVOEolH377QsmUos3s3lJeHBPDcc2Gnvm1bmH/CCSEBnHpqqBW0bp1dvJ9+Cv/8554k8dpr8NlnYd4xx4QkkUgUPXs27Laqjc8+g732yt/n15eZzXX3ohrLxZkUzGwE8CugJXC/u9+aNr8X8Hugc1Rmkrs/Vd06m2NSKAS7dsH8+eGfIrEz7NChcNu3d++GqVPhv/4Lli0LO5Uf/zjs9Dp2DEfthbQjzwd3WLlyT5JIJIpFi+DDD/eUa9MGDj887JDnzAm1LQg77EQSOOmkhmuW2rYt9D8kksQ//rEnefXpUzlJHH54fH/Hiorw+YnHW2+Fg7LE5w8bFpJmY/kd5T0pmFlLYDHwFaACmAOMcfeFKWVKgDfc/R4zOxp4yt37VLdeJYXcKyuDK64ITQPpMrVvV3VEvd9+8JWvhLbtuLjD00/D9deHI9oBA+BnP4MRIxrPP28hWL9+T4JIPK9YAcceG5LAKaeEZqhc2LUL5s3bkyReeik0QUH4TQ0dumcnPWBAaIaqLfdQi0pNAu++G+Z17Aj/9m9QVLSnTCJpdutW+fOPPbZun19dXGvXhgObZctCDIcfXrd1ZZsU4uxTKAaWuvuyKKCpwChgYUoZBxLHF52AlTHGI7X0ySeh2eqee2D//eF//zfs0Gtqh1+zpvL0nTv3rLNNGxg5Ei64AM48s2E7FmfPhuuuCzuPQw6Bhx6C888v3NpMIevSBYYMCY98a9ky7GyPPRa++92wo1yypHKS+POfQ9nEDjyxky4uDs176XbtCgcNqUkgkWi6dw/LTpyYOdG4wzvvVP78xx8P8zp0CNss8fmDB4cO9eps2wbLl+/Z8ac/Nm/eU/bOO+ueFLIVZ03hXGCEu18Wvf93YLC7T0gpcyDwV6ALsDdwqrt/7njUzMYD4wF69ep13HuZTseRBuMe+jGuvRY++ggmTAht8p061W1d27aFBPHOO/DYY/DII6HZon17OOssGDMGvvrVurfXvvVWSF4zZoTO0h/9CC69VKc7NieZmnog/AaKi8MO+rjj4O23q2+SGjYsnDZb21rlypWVO8/nzw+//dat4fjjw3pPOCGcKJC+0//gg8rratcuHNRkevTtW3OSqUohNB+dB3w1LSkUu/vVKWX+I4rhv81sCPAboJ+7V3nOh5qP4rVwIXznO/DCC+Eo5557whFaQ9q1K/wDPfww/OlPoY26c2cYPTrUIE4+Obsq+PLlcOON8Mc/hiPEH/4wHEnqtEZZty7s+BNJYu7cPTXW1M7rYcPg4IMb/vPXrw8110SSKCuDHTv2zO/Zc89OPn3Hv//+8TR1ZpsUcPdYHsAQ4NmU99cB16WVWQAcnPJ+GbBfdes97rjjXBre5s3ukya5t2rl3qWL+733uu/aFf/nbt/u/tRT7t/6lnvHju7gvt9+7ldd5f7yy5ljWL3afeJE99at3du2df/BD9w/+ij+WKXx2rTJ/ZVX8vc72bzZ/Z//dF+0yH3r1vzEAJR5NvvubArV5UHor1gG9AXaAOXAMWllngYujl4fRehTsOrWq6TQ8P7yF/fevcOv4eKLw043H7ZscZ82zf2888LOHtwPPjjs9OfOdd+wwf1HP3Lv0MG9ZUv3b3/b/f338xOrSGOTbVKI+5TU04E7CKeb/tbdp5jZzVFwM6Izju4DOhA6nf/T3f9a3TrVfNRwli8PzS0zZoQq9T33hOp0Idi4McT18MPw7LOh6t+6daiCn3ce/OQnoe1XRLKT9z6FuCgp1N/27XD77aHz2AxuugmuuSb7i41y7eOPw9klr78O48aF0/JEpHYK4ZRUKUB//3voSF60CM4+G+64o/DHUdp3X7jssnxHIdI86AzuZuLDD+Hf/z2c2bN1axiU7M9/LvyEICK5paTQxG3fDr/8Zbjg5ZFHwvn8CxbA176W78hEpBCp+agJe+aZ0JG8eHFIArffHv/VkCLSuKmm0AQtXRquFB45MlxV+eST4aGEICI1UVJoQjZtCmP/HHNMuNHJbbeFy/3VVCQi2VJSyIHS0jC+SosW4bm0tGHXnxir6Igj4NZbw1ARixfDD36g8X9EpHbUpxCz0tLKd057773wHhrmRjlz54bRHGfPDufvT5sWBt4SEakL1RRiNnly5VtpQng/eXL91rt2bUguxx8f+hB+8xt49VUlBBGpHyWFmK1YUbvpNdmxA371KzjsMHjgAfje90JT0bhxum+AiNSfdiMxq+risLpcNPb88zBwYBiSYvDgcDeq//7vut3nQEQkEyWFmE2ZEm4mk6p9+zA9Gzt2hBvTnHRSuJXltm3hLk/PPBPuDysi0pCUFGI2diyUlEDv3mHwud69w/uaOpnXrIFbbgk34fjGN8KdpW6/PVyNPGqU7jcsIvHQ2Uc5MHZs9mcazZkDd90VhqTYvh1OOy3cG3nkyHCvWhGROCkpFIDPPgtNRL/+dTiDqEOHcGbRVVfBkUfmOzoRaU6UFPJo5cpQCygpgdWrwzAUd94JF10E++yT7+hEpDlSUsgx93Ch2V13hQvNdu0Kw1BMmBA6knVaqYjkU1ZJwcwOBSrc/TMzGw70B/7g7p/EGVxTsmMHPPhgSAZvvBFOI504Mdzw5tBD8x2diEiQ7XHpNGCXmX0B+A3QF3gotqiamG3b4OtfDxeYbd8emow++CBcY6CEICKFJNvmo93uvtPMzgbucPe7zOyNOANrKrZuhdGjw3UFd98NV16p00lFpHBlmxR2mNkY4CLgzGhagd7mvXBs2RJqCM8/D/ffD5demu+IRESql23z0SXAEGCKu79rZn2BB+MLq/HbvBnOOCMkhAceUEIQkcYhq5qCuy8EJgKYWRego7vfGmdgjdmmTeGMopdfhj/8AS68MN8RiYhkJ6uagpn93cz2MbN9gXLgATO7Pd7QGqeNG2HECPjHP8K9FJQQRKQxybb5qJO7fwqMBh5w9+OAU+MLq3HasAG++tVwVfLDD4c7oImINCbZJoVWZnYg8A3gyRjjabQ++SSMUzRnDjz6KJx3Xr4jEhGpvWyTws3As8A77j7HzA4BlsQXVuPy8cdw6qnhorRp0+Dss/MdkYhI3WTb0fwY8FjK+2XAOXEF1ZisWxcSwsKFMH166GAWEWmssu1o7mlm081sjZmtNrNpZtYz7uAK3dq1cMopsGgRzJihhCAijV+2zUcPADOAg4AewBPRtGZr9Wo4+eRwf+QnnwwdzCIijV22SaG7uz/g7jujx++A7jHGVdBWrYLhw+Hdd+Gpp0LzkYhIU5BtUvjIzC40s5bR40JgXZyBFaoPPggJ4f334emnQ21BRKSpyDYpjCOcjvohsAo4lzD0RbNSURESwsqV8OyzcNJJ+Y5IRKRhZZUU3H2Fu5/l7t3dfT93/zrhQrZm44MP4EtfgjVr4Lnn4MQT8x2RiEjDq899vv6jpgJmNsLM3jazpWY2KcP8/zGzN6PHYjMr2Jv23HJLqCE89xyccEK+oxERiUd9bsdZ7V0BzKwlcDfwFaACmGNmM6LB9QBw9++llL8aOLYe8cRm06YwjtH550Nxcb6jERGJT31qCl7D/GJgqbsvc/ftwFRgVDXlxwAP1yOe2EydGga6u/zyfEciIhKvamsKZraRzDt/A9rVsO4ewPsp7yuAwVV8Tm/CLT7/VsX88cB4gF69etXwsQ3v3nuhXz81G4lI01dtUnD3jvVYd6bmpapqFxcAf3L3XVXEUQKUABQVFdVUQ2lQr78OZWVw5526jaaINH31aT6qSQVwcMr7nsDKKspeQIE2Hd13H7RuDb/4BbRoAX36hP4FEZGmKM6kMAc4zMz6mlkbwo5/RnohMzsC6AL8M8ZY6mTTJvjd72D37nCxmju89x6MH6/EICJNU2xJwd13AhMIQ24vAh519wVmdrOZnZVSdAww1d1z2iyUjalTYds22JXWqLVlC0yenJ+YRETiZAW4L65WUVGRl5WV5eSzjj8+9CdkYhZqECIijYGZzXX3oprKxdl81KglOpi7dMk8Pw8nQYmIxE5JoQolJdC2Ldx6K7RvX3le+/YwZUp+4hIRiZOSQgapVzCPHx8SRO/eocmod+/wfuzYfEcpItLw6jPMRZP18MMhMSSuYB47VklARJoH1RQyKCnRFcwi0jwpKaRJdDBffrmuYBaR5kdJIU1JCbRrBxdemO9IRERyT0khxcaNezqYO3fOdzQiIrmnpJBi6tTQwTx+fL4jERHJDyWFFBoiW0SaOyWFyNy54aEOZhFpzpQUIvfdpw5mERElBdTBLCKSoKSAOphFRBKUFAgdzF/8ojqYRUSafVJIdDCPH68OZhGRZp8UdAWziMgezTopbNwIDz2kDmYRkYRmnRQSQ2Srg1lEJGjWSaGkRB3MIiKpmm1SUAeziMjnNdukoA5mEZHPa5ZJQR3MIiKZNcukkH4PZhERCZplUkh0MA8enO9IREQKS7NLCupgFhGpWrNLCupgFhGpWrNKCupgFhGpXrNKCupgFhGpXrNKCokhstXBLCKSWbNJCnPnwuuv6x7MIiLVaTZJ4ZlnQgfz2LH5jkREpHA1m6QweTIsXqwOZhGR6sSaFMxshJm9bWZLzWxSFWW+YWYLzWyBmT0UZzw9e8a5dhGRxq9VXCs2s5bA3cBXgApgjpnNcPeFKWUOA64DTnT39Wa2X1zxiIhIzeKsKRQDS919mbtvB6YCo9LKfBu4293XA7j7mhjjERGRGsSZFHoA76e8r4impTocONzM/mFmr5jZiEwrMrPxZlZmZmVr166NKVwREYkzKWQ68dPT3rcCDgOGA2OA+83sc13B7l7i7kXuXtS9e/cGD1RERII4k0IFcHDK+57Aygxl/uLuO9z9XeBtQpIQEZE8iDMpzAEOM7O+ZtYGuACYkVbmceBkADPrRmhOWhZjTCIiUo3YkoK77wQmAM8Ci4BH3X2Bmd1sZmdFxZ4F1pnZQmAW8AN3XxdXTCIiUj1zT2/mL2xFRUVeVlaW7zBERBoVM5vr7kU1lWs2VzSLiEjNlBRERCRJSUFERJKUFEREJElJQUREkpQUREQkSUlBRESSlBRERCRJSUFERJKUFEREJElJQUREkmK7HaeINC07duygoqKCbdu25TsUqUbbtm3p2bMnrVu3rtPySgoikpWKigo6duxInz59MMt0Dy3JN3dn3bp1VFRU0Ldv3zqtQ81HIpKVbdu20bVrVyWEAmZmdO3atV61OSUFEcmaEkLhq+/fSElBRESSlBREJBalpdCnD7RoEZ5LS+u3vnXr1jFw4EAGDhzIAQccQI8ePZLvt2/fntU6LrnkEt5+++1qy9x9992U1jfYRkwdzSLS4EpLYfx42LIlvH/vvfAeYOzYuq2za9euvPnmmwDcdNNNdOjQge9///uVyrg77k6LFpmPdx944IEaP+eqq66qW4BNhGoKItLgJk/ekxAStmwJ0xva0qVL6devH1dccQWDBg1i1apVjB8/nqKiIo455hhuvvnmZNmhQ4fy5ptvsnPnTjp37sykSZMYMGAAQ4YMYc2aNQDccMMN3HHHHcnykyZNori4mCOOOILZs2cDsHnzZs455xwGDBjAmDFjKCoqSiasVDfeeCPHH398Mr7E7Y8XL17MKaecwoABAxg0aBDLly8H4Kc//Slf/OIXGTBgAJPj2FhZUFIQkQa3YkXtptfXwoULufTSS3njjTfo0aMHt956K2VlZZSXl/Pcc8+xcOHCzy2zYcMGvvSlL1FeXs6QIUP47W9/m3Hd7s5rr73GL37xi2SCueuuuzjggAMoLy9n0qRJvPHGGxmX/e53v8ucOXOYP38+GzZs4JlnngFgzJgxfO9736O8vJzZs2ez33778cQTT/D000/z2muvUV5ezrXXXttAW6d2lBREpMH16lW76fV16KGHcvzxxyffP/zwwwwaNIhBgwaxaNGijEmhXbt2jBw5EoDjjjsuebSebvTo0Z8r8/LLL3PBBRcAMGDAAI455piMy86cOZPi4mIGDBjACy+8wIIFC1i/fj0fffQRZ555JhAuNmvfvj3PP/8848aNo127dgDsu+++td8QDUBJQUQa3JQp0L595Wnt24fpcdh7772Tr5csWcKvfvUr/va3vzFv3jxGjBiR8bz9Nm3aJF+3bNmSnTt3Zlz3Xnvt9bkyiWag6mzZsoUJEyYwffp05s2bx7hx45JxZDpt1N0L4pRfJQURaXBjx0JJCfTuDWbhuaSk7p3MtfHpp5/SsWNH9tlnH1atWsWzzz7b4J8xdOhQHn30UQDmz5+fsSaydetWWrRoQbdu3di4cSPTpk0DoEuXLnTr1o0nnngCCBcFbtmyhdNOO43f/OY3bN26FYCPP/64wePOhs4+EpFYjB2bmySQbtCgQRx99NH069ePQw45hBNPPLHBP+Pqq68xdne3AAANrElEQVTmW9/6Fv3792fQoEH069ePTp06VSrTtWtXLrroIvr160fv3r0ZPHhwcl5paSmXX345kydPpk2bNkybNo0zzjiD8vJyioqKaN26NWeeeSY/+clPGjz2mlg21aBCUlRU5GVlZfkOQ6TZWbRoEUcddVS+wygIO3fuZOfOnbRt25YlS5Zw2mmnsWTJElq1Kozj7Ex/KzOb6+5FNS1bGN9ARKQR2bRpE1/+8pfZuXMn7s69995bMAmhvprGtxARyaHOnTszd+7cfIcRC3U0i4hIkpKCiIgkKSmIiEiSkoKIiCQpKYhIozB8+PDPXYh2xx138J3vfKfa5Tp06ADAypUrOffcc6tcd02nut9xxx1sSRnl7/TTT+eTTz7JJvRGRUlBRBqFMWPGMHXq1ErTpk6dypgxY7Ja/qCDDuJPf/pTnT8/PSk89dRTdO7cuc7rK1Q6JVVEau2aayDDSNH1MnAgRCNWZ3Tuuedyww038Nlnn7HXXnuxfPlyVq5cydChQ9m0aROjRo1i/fr17Nixg1tuuYVRo0ZVWn758uWcccYZvPXWW2zdupVLLrmEhQsXctRRRyWHlgC48sormTNnDlu3buXcc8/lxz/+MXfeeScrV67k5JNPplu3bsyaNYs+ffpQVlZGt27duP3225OjrF522WVcc801LF++nJEjRzJ06FBmz55Njx49+Mtf/pIc8C7hiSee4JZbbmH79u107dqV0tJS9t9/fzZt2sTVV19NWVkZZsaNN97IOeecwzPPPMP111/Prl276NatGzNnzmy4PwIx1xTMbISZvW1mS81sUob5F5vZWjN7M3pcFmc8ItJ4de3aleLi4uTw01OnTuX888/HzGjbti3Tp0/n9ddfZ9asWVx77bXVDlp3zz330L59e+bNm8fkyZMrXXMwZcoUysrKmDdvHi+88ALz5s1j4sSJHHTQQcyaNYtZs2ZVWtfcuXN54IEHePXVV3nllVe47777kkNpL1myhKuuuooFCxbQuXPn5PhHqYYOHcorr7zCG2+8wQUXXMBtt90GwE9+8hM6derE/PnzmTdvHqeccgpr167l29/+NtOmTaO8vJzHHnus3ts1XWw1BTNrCdwNfAWoAOaY2Qx3Tx856hF3nxBXHCLS8Ko7oo9Toglp1KhRTJ06NXl07u5cf/31vPjii7Ro0YIPPviA1atXc8ABB2Rcz4svvsjEiRMB6N+/P/3790/Oe/TRRykpKWHnzp2sWrWKhQsXVpqf7uWXX+bss89OjtQ6evRoXnrpJc466yz69u3LwIEDgaqH566oqOD8889n1apVbN++nb59+wLw/PPPV2ou69KlC0888QQnnXRSskwcw2vHWVMoBpa6+zJ33w5MBUbVsEwsGvpesSKSH1//+teZOXMmr7/+Olu3bmXQoEFAGGBu7dq1zJ07lzfffJP9998/43DZqTINU/3uu+/yy1/+kpkzZzJv3jy+9rWv1bie6mokiWG3oerhua+++momTJjA/Pnzuffee5Ofl2ko7VwMrx1nUugBvJ/yviKalu4cM5tnZn8ys4MzrcjMxptZmZmVrV27tlZBJO4V+9574L7nXrFKDCKNT4cOHRg+fDjjxo2r1MG8YcMG9ttvP1q3bs2sWbN47733ql3PSSedRGm0E3jrrbeYN28eEIbd3nvvvenUqROrV6/m6aefTi7TsWNHNm7cmHFdjz/+OFu2bGHz5s1Mnz6dYcOGZf2dNmzYQI8eYdf4+9//Pjn9tNNO49e//nXy/fr16xkyZAgvvPAC7777LhDP8NpxJoVM6Sw9pT4B9HH3/sDzwO8/vwi4e4m7F7l7Uffu3WsVRC7vFSsi8RszZgzl5eXJO58BjB07lrKyMoqKiigtLeXII4+sdh1XXnklmzZton///tx2220UFxcD4S5qxx57LMcccwzjxo2rNOz2+PHjGTlyJCeffHKldQ0aNIiLL76Y4uJiBg8ezGWXXcaxxx6b9fe56aabOO+88xg2bBjdunVLTr/hhhtYv349/fr1Y8CAAcyaNYvu3btTUlLC6NGjGTBgAOeff37Wn5Ot2IbONrMhwE3u/tXo/XUA7v6zKsq3BD52906Z5ifUdujsFi1CDeHznwe7d2e9GpFmT0NnNx71GTo7zprCHOAwM+trZm2AC4AZqQXM7MCUt2cBixo6iFzfK1ZEpDGLLSm4+05gAvAsYWf/qLsvMLObzeysqNhEM1tgZuXARODiho4j1/eKFRFpzGK9eM3dnwKeSpv2o5TX1wHXxRlD4naAkyfDihWhhjBlSn5uEyjS2BXKzeWlavXtEmgWVzTn616xIk1J27ZtWbduHV27dlViKFDuzrp162jbtm2d19EskoKI1F/Pnj2pqKigtqeFS261bduWnj171nl5JQURyUrr1q2TV9JK06VRUkVEJElJQUREkpQUREQkKbYrmuNiZmuB6gc2yZ9uwEf5DqIaiq9+Cj0+KPwYFV/91Ce+3u5e4zhBjS4pFDIzK8vmMvJ8UXz1U+jxQeHHqPjqJxfxqflIRESSlBRERCRJSaFhleQ7gBoovvop9Pig8GNUfPUTe3zqUxARkSTVFEREJElJQUREkpQUasnMDjazWWa2KLoXxHczlBluZhvM7M3o8aNM64oxxuVmNj/67M/dps6CO81saXR/7EE5jO2IlO3yppl9ambXpJXJ+fYzs9+a2Rozeytl2r5m9pyZLYmeu1Sx7EVRmSVmdlGOYvuFmf0r+vtNN7POVSxb7W8h5hhvMrMPUv6Op1ex7Agzezv6PU7KYXyPpMS23MzerGLZWLdhVfuUvP3+3F2PWjyAA4FB0euOwGLg6LQyw4En8xjjcqBbNfNPB54m3Ef7BODVPMXZEviQcFFNXrcfcBIwCHgrZdptwKTo9STg5xmW2xdYFj13iV53yUFspwGtotc/zxRbNr+FmGO8Cfh+Fr+Bd4BDgDZAefr/U1zxpc3/b+BH+diGVe1T8vX7U02hltx9lbu/Hr3eSLirXI/8RlVro4A/ePAK0Dnt1qi58mXgHXfP+xXq7v4i8HHa5FHA76PXvwe+nmHRrwLPufvH7r4eeA4YEXds7v5XD3c3BHgFqPtYyQ2giu2XjWJgqbsvc/ftwFTCdm9Q1cVn4eYQ3wAebujPzUY1+5S8/P6UFOrBzPoAxwKvZpg9xMzKzexpMzsmp4GBA381s7lmNj7D/B7A+ynvK8hPYruAqv8R87n9EvZ391UQ/nGB/TKUKYRtOY5Q88ukpt9C3CZETVy/raL5oxC23zBgtbsvqWJ+zrZh2j4lL78/JYU6MrMOwDTgGnf/NG3264QmkQHAXcDjOQ7vRHcfBIwErjKzk9LmZ7ptVk7PTTazNsBZwGMZZud7+9VGXrelmU0GdgKlVRSp6bcQp3uAQ4GBwCpCE026vP8WgTFUX0vIyTasYZ9S5WIZptVr+ykp1IGZtSb88Urd/c/p8939U3ffFL1+CmhtZt1yFZ+7r4ye1wDTCVX0VBXAwSnvewIrcxNd0kjgdXdfnT4j39svxepEs1r0vCZDmbxty6hT8QxgrEcNzOmy+C3Ext1Xu/sud98N3FfFZ+f1t2hmrYDRwCNVlcnFNqxin5KX35+SQi1F7Y+/ARa5++1VlDkgKoeZFRO287ocxbe3mXVMvCZ0SL6VVmwG8K3oLKQTgA2JamoOVXl0ls/tl2YGkDib4yLgLxnKPAucZmZdouaR06JpsTKzEcAPgbPcfUsVZbL5LcQZY2o/1dlVfPYc4DAz6xvVHi8gbPdcORX4l7tXZJqZi21YzT4lP7+/uHrUm+oDGEqons0D3owepwNXAFdEZSYACwhnUrwC/FsO4zsk+tzyKIbJ0fTU+Ay4m3DWx3ygKMfbsD1hJ98pZVpetx8hQa0CdhCOvi4FugIzgSXR875R2SLg/pRlxwFLo8clOYptKaEtOfEb/N+o7EHAU9X9FnK4/f4Y/b7mEXZwB6bHGL0/nXDGzTtxxZgpvmj67xK/u5SyOd2G1exT8vL70zAXIiKSpOYjERFJUlIQEZEkJQUREUlSUhARkSQlBRERSVJSEImY2S6rPIJrg43YaWZ9UkfoFClUrfIdgEgB2eruA/MdhEg+qaYgUoNoPP2fm9lr0eML0fTeZjYzGvBtppn1iqbvb+EeB+XR49+iVbU0s/uiMfP/ambtovITzWxhtJ6pefqaIoCSgkiqdmnNR+enzPvU3YuBXwN3RNN+TRiCvD9hQLo7o+l3Ai94GNBvEOFKWIDDgLvd/RjgE+CcaPok4NhoPVfE9eVEsqErmkUiZrbJ3TtkmL4cOMXdl0UDl33o7l3N7CPC0A07oumr3L2bma0Ferr7Zynr6EMY9/6w6P0PgdbufouZPQNsIowG+7hHgwGK5INqCiLZ8SpeV1Umk89SXu9iT5/e1whjUR0HzI1G7hTJCyUFkeycn/L8z+j1bMKongBjgZej1zOBKwHMrKWZ7VPVSs2sBXCwu88C/hPoDHyutiKSKzoiEdmjnVW+efsz7p44LXUvM3uVcCA1Jpo2Efitmf0AWAtcEk3/LlBiZpcSagRXEkbozKQl8KCZdSKMXvs/7v5Jg30jkVpSn4JIDaI+hSJ3/yjfsYjETc1HIiKSpJqCiIgkqaYgIiJJSgoiIpKkpCAiIklKCiIikqSkICIiSf8f3BjX8cwGlx8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x24087300278>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.clf()   # clear figure(清空图像)\n",
    "\n",
    "acc = history.history['acc']\n",
    "val_acc = history.history['val_acc']\n",
    "\n",
    "plt.plot(epochs, acc, 'bo', label='Training acc')\n",
    "plt.plot(epochs, val_acc, 'b', label='Validation acc')\n",
    "plt.title('Training and validation accuracy')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It seems that the network starts overfitting after 8 epochs. Let's train a new network from scratch for 8 epochs, then let's evaluate it on \n",
    "the test set:\n",
    "\n",
    "网络在训练8 轮后开始过拟合。我们从头开始训练一个新网络，共8个轮次，然后在测试集上评估模型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 7982 samples, validate on 1000 samples\n",
      "Epoch 1/8\n",
      "7982/7982 [==============================] - 1s 152us/step - loss: 2.5398 - acc: 0.5226 - val_loss: 1.6733 - val_acc: 0.6570\n",
      "Epoch 2/8\n",
      "7982/7982 [==============================] - 1s 129us/step - loss: 1.3712 - acc: 0.7121 - val_loss: 1.2758 - val_acc: 0.7210\n",
      "Epoch 3/8\n",
      "7982/7982 [==============================] - 1s 126us/step - loss: 1.0136 - acc: 0.7781 - val_loss: 1.1303 - val_acc: 0.7530\n",
      "Epoch 4/8\n",
      "7982/7982 [==============================] - 1s 129us/step - loss: 0.7976 - acc: 0.8251 - val_loss: 1.0539 - val_acc: 0.7590\n",
      "Epoch 5/8\n",
      "7982/7982 [==============================] - 1s 121us/step - loss: 0.6393 - acc: 0.8624 - val_loss: 0.9754 - val_acc: 0.7920\n",
      "Epoch 6/8\n",
      "7982/7982 [==============================] - 1s 121us/step - loss: 0.5124 - acc: 0.8923 - val_loss: 0.9102 - val_acc: 0.8140\n",
      "Epoch 7/8\n",
      "7982/7982 [==============================] - 1s 123us/step - loss: 0.4123 - acc: 0.9137 - val_loss: 0.8932 - val_acc: 0.8210\n",
      "Epoch 8/8\n",
      "7982/7982 [==============================] - 1s 125us/step - loss: 0.3354 - acc: 0.9288 - val_loss: 0.8732 - val_acc: 0.8260\n",
      "2246/2246 [==============================] - 0s 150us/step\n"
     ]
    }
   ],
   "source": [
    "model = models.Sequential()\n",
    "model.add(layers.Dense(64, activation='relu', input_shape=(10000,)))\n",
    "model.add(layers.Dense(64, activation='relu'))\n",
    "model.add(layers.Dense(46, activation='softmax'))\n",
    "\n",
    "model.compile(optimizer='rmsprop',\n",
    "              loss='categorical_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "model.fit(partial_x_train,\n",
    "          partial_y_train,\n",
    "          epochs=8,\n",
    "          batch_size=512,\n",
    "          validation_data=(x_val, y_val))\n",
    "results = model.evaluate(x_test, one_hot_test_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.9847470230007427, 0.7845057880676759]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Our approach reaches an accuracy of ~78%. With a balanced binary classification problem, the accuracy reached by a purely random classifier \n",
    "would be 50%, but in our case it is closer to 19%, so our results seem pretty good, at least when compared to a random baseline:\n",
    "\n",
    "这种方法可以得到约 78% 的精度。对于平衡的二分类问题，完全随机的分类器能够得到50%的精度。但在这个例子中，完全随机的精度约为19%，所以上述结果相当不错，至少和随机的基准比起来还不错。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.19679430097951914"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import copy\n",
    "\n",
    "test_labels_copy = copy.copy(test_labels)\n",
    "np.random.shuffle(test_labels_copy)\n",
    "float(np.sum(np.array(test_labels) == np.array(test_labels_copy))) / len(test_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating predictions on new data\n",
    "\n",
    "We can verify that the `predict` method of our model instance returns a probability distribution over all 46 topics. Let's generate topic \n",
    "predictions for all of the test data:\n",
    "## 在新数据上生成预测结果\n",
    "\n",
    "你可以验证，模型实例的 predict 方法返回了在 46 个主题上的概率分布。我们对所有测试数据生成主题预测。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = model.predict(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each entry in `predictions` is a vector of length 46:\n",
    "\n",
    "predictions 中的每个元素都是长度为 46 的向量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(46,)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions[0].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The coefficients in this vector sum to 1:\n",
    "\n",
    "这个向量的所有元素总和为 1。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.99999994"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sum(predictions[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The largest entry is the predicted class, i.e. the class with the highest probability:\n",
    "\n",
    "最大的元素就是预测类别，即概率最大的类别。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.argmax(predictions[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A different way to handle the labels and the loss\n",
    "\n",
    "We mentioned earlier that another way to encode the labels would be to cast them as an integer tensor, like such:\n",
    "## 处理标签和损失的另一种方法\n",
    "前面提到了另一种编码标签的方法，就是将其转换为整数张量，如下所示。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train = np.array(train_labels)\n",
    "y_test = np.array(test_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "The only thing it would change is the choice of the loss function. Our previous loss, `categorical_crossentropy`, expects the labels to \n",
    "follow a categorical encoding. With integer labels, we should use `sparse_categorical_crossentropy`:\n",
    "\n",
    "对于这种编码方法，唯一需要改变的是损失函数的选择。对于代码清单 3-21 使用的损失 函数 categorical_crossentropy，标签应该遵循分类编码。对于整数标签，你应该使用 sparse_categorical_crossentropy。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer='rmsprop', loss='sparse_categorical_crossentropy', metrics=['acc'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This new loss function is still mathematically the same as `categorical_crossentropy`; it just has a different interface.\n",
    "\n",
    "这个新的损失函数在数学上与 categorical_crossentropy 完全相同，二者只是接口不同。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## On the importance of having sufficiently large intermediate layers\n",
    "\n",
    "\n",
    "We mentioned earlier that since our final outputs were 46-dimensional, we should avoid intermediate layers with much less than 46 hidden \n",
    "units. Now let's try to see what happens when we introduce an information bottleneck by having intermediate layers significantly less than \n",
    "46-dimensional, e.g. 4-dimensional.\n",
    "## 中间层维度足够大的重要性\n",
    "\n",
    "前面提到，最终输出是 46 维的，因此中间层的隐藏单元个数不应该比 46 小太多。现在来 看一下，如果中间层的维度远远小于 46（比如 4 维），造成了信息瓶颈，那么会发生什么？\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 7982 samples, validate on 1000 samples\n",
      "Epoch 1/20\n",
      "7982/7982 [==============================] - 2s 190us/step - loss: 2.7073 - acc: 0.4411 - val_loss: 2.0155 - val_acc: 0.5930\n",
      "Epoch 2/20\n",
      "7982/7982 [==============================] - 1s 169us/step - loss: 1.7341 - acc: 0.6183 - val_loss: 1.6573 - val_acc: 0.6100\n",
      "Epoch 3/20\n",
      "7982/7982 [==============================] - 1s 170us/step - loss: 1.4678 - acc: 0.6359 - val_loss: 1.5688 - val_acc: 0.6080\n",
      "Epoch 4/20\n",
      "7982/7982 [==============================] - 1s 176us/step - loss: 1.3273 - acc: 0.6446 - val_loss: 1.4833 - val_acc: 0.6170\n",
      "Epoch 5/20\n",
      "7982/7982 [==============================] - 1s 173us/step - loss: 1.2168 - acc: 0.6498 - val_loss: 1.4540 - val_acc: 0.6250\n",
      "Epoch 6/20\n",
      "7982/7982 [==============================] - 1s 168us/step - loss: 1.1265 - acc: 0.6753 - val_loss: 1.4326 - val_acc: 0.6350\n",
      "Epoch 7/20\n",
      "7982/7982 [==============================] - 1s 174us/step - loss: 1.0522 - acc: 0.6944 - val_loss: 1.4520 - val_acc: 0.6420\n",
      "Epoch 8/20\n",
      "7982/7982 [==============================] - 1s 169us/step - loss: 0.9927 - acc: 0.7037 - val_loss: 1.4426 - val_acc: 0.6510\n",
      "Epoch 9/20\n",
      "7982/7982 [==============================] - 1s 164us/step - loss: 0.9425 - acc: 0.7167 - val_loss: 1.4766 - val_acc: 0.6500\n",
      "Epoch 10/20\n",
      "7982/7982 [==============================] - 1s 170us/step - loss: 0.8978 - acc: 0.7429 - val_loss: 1.5227 - val_acc: 0.6480\n",
      "Epoch 11/20\n",
      "7982/7982 [==============================] - 1s 168us/step - loss: 0.8602 - acc: 0.7547 - val_loss: 1.5150 - val_acc: 0.6670\n",
      "Epoch 12/20\n",
      "7982/7982 [==============================] - 1s 171us/step - loss: 0.8261 - acc: 0.7602 - val_loss: 1.5256 - val_acc: 0.6680\n",
      "Epoch 13/20\n",
      "7982/7982 [==============================] - 1s 175us/step - loss: 0.7962 - acc: 0.7677 - val_loss: 1.5739 - val_acc: 0.6660\n",
      "Epoch 14/20\n",
      "7982/7982 [==============================] - 1s 178us/step - loss: 0.7717 - acc: 0.7734 - val_loss: 1.6063 - val_acc: 0.6650\n",
      "Epoch 15/20\n",
      "7982/7982 [==============================] - 1s 173us/step - loss: 0.7448 - acc: 0.7815 - val_loss: 1.6424 - val_acc: 0.6700\n",
      "Epoch 16/20\n",
      "7982/7982 [==============================] - 1s 180us/step - loss: 0.7240 - acc: 0.7880 - val_loss: 1.7068 - val_acc: 0.6670\n",
      "Epoch 17/20\n",
      "7982/7982 [==============================] - 1s 176us/step - loss: 0.7057 - acc: 0.7950 - val_loss: 1.7215 - val_acc: 0.6610\n",
      "Epoch 18/20\n",
      "7982/7982 [==============================] - 1s 168us/step - loss: 0.6848 - acc: 0.8028 - val_loss: 1.7868 - val_acc: 0.6670\n",
      "Epoch 19/20\n",
      "7982/7982 [==============================] - 1s 168us/step - loss: 0.6691 - acc: 0.8077 - val_loss: 1.7905 - val_acc: 0.6710\n",
      "Epoch 20/20\n",
      "7982/7982 [==============================] - 1s 167us/step - loss: 0.6530 - acc: 0.8140 - val_loss: 1.8390 - val_acc: 0.6650\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x2408769cbe0>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = models.Sequential()\n",
    "model.add(layers.Dense(64, activation='relu', input_shape=(10000,)))\n",
    "model.add(layers.Dense(4, activation='relu'))\n",
    "model.add(layers.Dense(46, activation='softmax'))\n",
    "\n",
    "model.compile(optimizer='rmsprop',\n",
    "              loss='categorical_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "model.fit(partial_x_train,\n",
    "          partial_y_train,\n",
    "          epochs=20,\n",
    "          batch_size=128,\n",
    "          validation_data=(x_val, y_val))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Our network now seems to peak at ~71% test accuracy, a 8% absolute drop. This drop is mostly due to the fact that we are now trying to \n",
    "compress a lot of information (enough information to recover the separation hyperplanes of 46 classes) into an intermediate space that is \n",
    "too low-dimensional. The network is able to cram _most_ of the necessary information into these 8-dimensional representations, but not all \n",
    "of it.\n",
    "\n",
    "现在网络的验证精度最大约为 71%，比前面下降了 8%。导致这一下降的主要原因在于，你试图将大量信息（这些信息足够恢复 46 个类别的分割超平面）压缩到维度很小的中间空间。网络能够将大部分必要信息塞入这个四维表示中，但并不是全部信息。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Further experiments\n",
    "\n",
    "* Try using larger or smaller layers: 32 units, 128 units...\n",
    "* We were using two hidden layers. Now try to use a single hidden layer, or three hidden layers.\n",
    "\n",
    "## 进一步的实验\n",
    "* 尝试使用更多或更少的隐藏单元，比如 32 个、128 个等。\n",
    "\n",
    "* 前面使用了两个隐藏层，现在尝试使用一个或三个隐藏层。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wrapping up\n",
    "\n",
    "\n",
    "Here's what you should take away from this example:\n",
    "\n",
    "* If you are trying to classify data points between N classes, your network should end with a `Dense` layer of size N.\n",
    "* In a single-label, multi-class classification problem, your network should end with a `softmax` activation, so that it will output a \n",
    "probability distribution over the N output classes.\n",
    "* _Categorical crossentropy_ is almost always the loss function you should use for such problems. It minimizes the distance between the \n",
    "probability distributions output by the network, and the true distribution of the targets.\n",
    "* There are two ways to handle labels in multi-class classification:\n",
    "    ** Encoding the labels via \"categorical encoding\" (also known as \"one-hot encoding\") and using `categorical_crossentropy` as your loss \n",
    "function.\n",
    "    ** Encoding the labels as integers and using the `sparse_categorical_crossentropy` loss function.\n",
    "* If you need to classify data into a large number of categories, then you should avoid creating information bottlenecks in your network by having \n",
    "intermediate layers that are too small.\n",
    "\n",
    "## 小结\n",
    "下面是你应该从这个例子中学到的要点。\n",
    "\n",
    "* 如果要对 N 个类别的数据点进行分类，网络的最后一层应该是大小为 N 的 Dense 层。\n",
    "* 对于单标签、多分类问题，网络的最后一层应该使用 softmax 激活，这样可以输出在 N个输出类别上的概率分布。\n",
    "* 这种问题的损失函数几乎总是应该使用分类交叉熵。它将网络输出的概率分布与目标的 真实分布之间的距离最小化。\n",
    "* 处理多分类问题的标签有两种方法。\n",
    "* 通过分类编 码（也叫 one-hot 编码）对标签进行编码，然后使用 categorical_crossentropy 作为损失函数。\n",
    "*  将标签编码为整数，然后使用 sparse_categorical_crossentropy 损失函数。\n",
    "* 如果你需要将数据划分到许多类别中，应该避免使用太小的中间层，以免在网络中造成信息瓶颈。\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}