3.5-classifying-newswires.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'2.2.2'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import keras\n",
    "keras.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 뉴스 기사 분류: 다중 분류 문제\n",
    "\n",
    "이 노트북은 [케라스 창시자에게 배우는 딥러닝](https://tensorflow.blog/%EC%BC%80%EB%9D%BC%EC%8A%A4-%EB%94%A5%EB%9F%AC%EB%8B%9D/) 책의 3장 5절의 코드 예제입니다. 책에는 더 많은 내용과 그림이 있습니다. 이 노트북에는 소스 코드에 관련된 설명만 포함합니다.\n",
    "\n",
    "----\n",
    "\n",
    "이전 섹션에서 완전 연결된 신경망을 사용해 벡터 입력을 어떻게 두 개의 클래스로 분류하는지 보았습니다. 두 개 이상의 클래스가 있을 때는 어떻게 해야 할까요?\n",
    "\n",
    "이 절에서 로이터 뉴스를 46개의 상호 배타적인 토픽으로 분류하는 신경망을 만들어 보겠습니다. 클래스가 많기 때문에 이 문제는 다중 분류의 예입니다. 각 데이터 포인트가 정확히 하나의 범주로 분류되기 때문에 좀 더 정확히 말하면 단일 레이블 다중 분류 문제입니다. 각 데이터 포인트가 여러 개의 범주(가령, 토픽)에 속할 수 있다면 이런 문제는 다중 레이블 다중 분류의 문제가 됩니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 로이터 데이터셋\n",
    "\n",
    "1986년에 로이터에서 공개한 짧은 뉴스 기사와 토픽의 집합인 로이터 데이터셋을 사용하겠습니다. 이 데이터셋은 텍스트 분류를 위해 널리 사용되는 간단한 데이터셋입니다. 46개의 토픽이 있으며 어떤 토픽은 다른 것에 비해 데이터가 많습니다. 각 토픽은 훈련 세트에 최소한 10개의 샘플을 가지고 있습니다.\n",
    "\n",
    "IMDB와 MNIST와 마찬가지로 로이터 데이터셋은 케라스에 포함되어 있습니다. 한 번 살펴보죠:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.datasets import reuters\n",
    "\n",
    "(train_data, train_labels), (test_data, test_labels) = reuters.load_data(num_words=10000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "IMDB 데이터셋에서처럼 num_words=10000 매개변수는 데이터에서 가장 자주 등장하는 단어 10,000개로 제한합니다.\n",
    "\n",
    "여기에는 8,982개의 훈련 샘플과 2,246개의 테스트 샘플이 있습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8982"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(train_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2246"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "IMDB 리뷰처럼 각 샘플은 정수 리스트입니다(단어 인덱스):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1,\n",
       " 245,\n",
       " 273,\n",
       " 207,\n",
       " 156,\n",
       " 53,\n",
       " 74,\n",
       " 160,\n",
       " 26,\n",
       " 14,\n",
       " 46,\n",
       " 296,\n",
       " 26,\n",
       " 39,\n",
       " 74,\n",
       " 2979,\n",
       " 3554,\n",
       " 14,\n",
       " 46,\n",
       " 4689,\n",
       " 4329,\n",
       " 86,\n",
       " 61,\n",
       " 3499,\n",
       " 4795,\n",
       " 14,\n",
       " 61,\n",
       " 451,\n",
       " 4329,\n",
       " 17,\n",
       " 12]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data[10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "궁금한 경우를 위해 어떻게 단어로 디코딩하는지 알아보겠습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_index = reuters.get_word_index()\n",
    "reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])\n",
    "# 0, 1, 2는 '패딩', '문서 시작', '사전에 없음'을 위한 인덱스이므로 3을 뺍니다\n",
    "decoded_newswire = ' '.join([reverse_word_index.get(i - 3, '?') for i in train_data[0]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'? ? ? said as a result of its december acquisition of space co it expects earnings per share in 1987 of 1 15 to 1 30 dlrs per share up from 70 cts in 1986 the company said pretax net should rise to nine to 10 mln dlrs from six mln dlrs in 1986 and rental operation revenues to 19 to 22 mln dlrs from 12 5 mln dlrs it said cash flow per share this year should be 2 50 to three dlrs reuter 3'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "decoded_newswire"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "샘플에 연결된 레이블은 토픽의 인덱스로 0과 45 사이의 정수입니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_labels[10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 데이터 준비\n",
    "\n",
    "이전의 예제와 동일한 코드를 사용해서 데이터를 벡터로 변환합니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def vectorize_sequences(sequences, dimension=10000):\n",
    "    results = np.zeros((len(sequences), dimension))\n",
    "    for i, sequence in enumerate(sequences):\n",
    "        results[i, sequence] = 1.\n",
    "    return results\n",
    "\n",
    "# 훈련 데이터 벡터 변환\n",
    "x_train = vectorize_sequences(train_data)\n",
    "# 테스트 데이터 벡터 변환\n",
    "x_test = vectorize_sequences(test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "레이블을 벡터로 바꾸는 방법은 두 가지입니다. 레이블의 리스트를 정수 텐서로 변환하는 것과 원-핫 인코딩을 사용하는 것입니다. 원-핫 인코딩이 범주형 데이터에 널리 사용되기 때문에 범주형 인코딩이라고도 부릅니다. 원-핫 인코딩에 대한 자세한 설명은 6.1절을 참고하세요. 이 경우 레이블의 원-핫 인코딩은 각 레이블의 인덱스 자리는 1이고 나머지는 모두 0인 벡터입니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def to_one_hot(labels, dimension=46):\n",
    "    results = np.zeros((len(labels), dimension))\n",
    "    for i, label in enumerate(labels):\n",
    "        results[i, label] = 1.\n",
    "    return results\n",
    "\n",
    "# 훈련 레이블 벡터 변환\n",
    "one_hot_train_labels = to_one_hot(train_labels)\n",
    "# 테스트 레이블 벡터 변환\n",
    "one_hot_test_labels = to_one_hot(test_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MNIST 예제에서 이미 보았듯이 케라스에는 이를 위한 내장 함수가 있습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.utils.np_utils import to_categorical\n",
    "\n",
    "one_hot_train_labels = to_categorical(train_labels)\n",
    "one_hot_test_labels = to_categorical(test_labels)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 모델 구성\n",
    "\n",
    "이 토픽 분류 문제는 이전의 영화 리뷰 분류 문제와 비슷해 보입니다. 두 경우 모두 짧은 텍스트를 분류하는 것이죠. 여기에서는 새로운 제약 사항이 추가되었습니다. 출력 클래스의 개수가 2에서 46개로 늘어난 점입니다. 출력 공간의 차원이 훨씬 커졌습니다.\n",
    "\n",
    "이전에 사용했던 것처럼 `Dense` 층을 쌓으면 각 층은 이전 층의 출력에서 제공한 정보만 사용할 수 있습니다. 한 층이 분류 문제에 필요한 일부 정보를 누락하면 그 다음 층에서 이를 복원할 방법이 없습니다. 각 층은 잠재적으로 정보의 병목이 될 수 있습니다. 이전 예제에서 16차원을 가진 중간층을 사용했지만 16차원 공간은 46개의 클래스를 구분하기에 너무 제약이 많을 것 같습니다. 이렇게 규모가 작은 층은 유용한 정보를 완전히 잃게 되는 정보의 병목 지점처럼 동작할 수 있습니다.\n",
    "\n",
    "이런 이유로 좀 더 규모가 큰 층을 사용하겠습니다. 64개의 유닛을 사용해 보죠:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras import models\n",
    "from keras import layers\n",
    "\n",
    "model = models.Sequential()\n",
    "model.add(layers.Dense(64, activation='relu', input_shape=(10000,)))\n",
    "model.add(layers.Dense(64, activation='relu'))\n",
    "model.add(layers.Dense(46, activation='softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이 구조에서 주목해야 할 점이 두 가지 있습니다:\n",
    "\n",
    "* 마지막 `Dense` 층의 크기가 46입니다. 각 입력 샘플에 대해서 46차원의 벡터를 출력한다는 뜻입니다. 이 벡터의 각 원소(각 차원)은 각기 다른 출력 클래스가 인코딩된 것입니다.\n",
    "* 마지막 층에 `softmax` 활성화 함수가 사용되었습니다. MNIST 예제에서 이런 방식을 보았습니다. 각 입력 샘플마다 46개의 출력 클래스에 대한 확률 분포를 출력합니다. 즉, 46차원의 출력 벡터를 만들며 `output[i]`는 어떤 샘플이 클래스 `i`에 속할 확률입니다. 46개의 값을 모두 더하면 1이 됩니다.\n",
    "\n",
    "이런 문제에 사용할 최선의 손실 함수는 `categorical_crossentropy`입니다. 이 함수는 두 확률 분포의 사이의 거리를 측정합니다. 여기에서는 네트워크가 출력한 확률 분포와 진짜 레이블의 분포 사이의 거리입니다. 두 분포 사이의 거리를 최소화하면 진짜 레이블에 가능한 가까운 출력을 내도록 모델을 훈련하게 됩니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer='rmsprop',\n",
    "              loss='categorical_crossentropy',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 훈련 검증\n",
    "\n",
    "훈련 데이터에서 1,000개의 샘플을 따로 떼어서 검증 세트로 사용하겠습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_val = x_train[:1000]\n",
    "partial_x_train = x_train[1000:]\n",
    "\n",
    "y_val = one_hot_train_labels[:1000]\n",
    "partial_y_train = one_hot_train_labels[1000:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이제 20번의 에포크로 모델을 훈련시킵니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 7982 samples, validate on 1000 samples\n",
      "Epoch 1/20\n",
      "7982/7982 [==============================] - 1s 151us/step - loss: 2.5322 - acc: 0.4955 - val_loss: 1.7208 - val_acc: 0.6120\n",
      "Epoch 2/20\n",
      "7982/7982 [==============================] - 1s 65us/step - loss: 1.4452 - acc: 0.6879 - val_loss: 1.3459 - val_acc: 0.7060\n",
      "Epoch 3/20\n",
      "7982/7982 [==============================] - 1s 65us/step - loss: 1.0953 - acc: 0.7651 - val_loss: 1.1708 - val_acc: 0.7430\n",
      "Epoch 4/20\n",
      "7982/7982 [==============================] - 1s 65us/step - loss: 0.8697 - acc: 0.8165 - val_loss: 1.0793 - val_acc: 0.7590\n",
      "Epoch 5/20\n",
      "7982/7982 [==============================] - 1s 65us/step - loss: 0.7034 - acc: 0.8472 - val_loss: 0.9844 - val_acc: 0.7810\n",
      "Epoch 6/20\n",
      "7982/7982 [==============================] - 1s 65us/step - loss: 0.5667 - acc: 0.8802 - val_loss: 0.9411 - val_acc: 0.8040\n",
      "Epoch 7/20\n",
      "7982/7982 [==============================] - 1s 65us/step - loss: 0.4581 - acc: 0.9048 - val_loss: 0.9083 - val_acc: 0.8020\n",
      "Epoch 8/20\n",
      "7982/7982 [==============================] - 1s 65us/step - loss: 0.3695 - acc: 0.9231 - val_loss: 0.9363 - val_acc: 0.7890\n",
      "Epoch 9/20\n",
      "7982/7982 [==============================] - 1s 65us/step - loss: 0.3032 - acc: 0.9315 - val_loss: 0.8917 - val_acc: 0.8090\n",
      "Epoch 10/20\n",
      "7982/7982 [==============================] - 1s 65us/step - loss: 0.2537 - acc: 0.9414 - val_loss: 0.9071 - val_acc: 0.8110\n",
      "Epoch 11/20\n",
      "7982/7982 [==============================] - 1s 65us/step - loss: 0.2187 - acc: 0.9471 - val_loss: 0.9177 - val_acc: 0.8130\n",
      "Epoch 12/20\n",
      "7982/7982 [==============================] - 1s 65us/step - loss: 0.1873 - acc: 0.9508 - val_loss: 0.9027 - val_acc: 0.8130\n",
      "Epoch 13/20\n",
      "7982/7982 [==============================] - 1s 65us/step - loss: 0.1703 - acc: 0.9521 - val_loss: 0.9323 - val_acc: 0.8110\n",
      "Epoch 14/20\n",
      "7982/7982 [==============================] - 1s 65us/step - loss: 0.1536 - acc: 0.9554 - val_loss: 0.9689 - val_acc: 0.8050\n",
      "Epoch 15/20\n",
      "7982/7982 [==============================] - 1s 65us/step - loss: 0.1390 - acc: 0.9560 - val_loss: 0.9686 - val_acc: 0.8150\n",
      "Epoch 16/20\n",
      "7982/7982 [==============================] - 1s 65us/step - loss: 0.1313 - acc: 0.9560 - val_loss: 1.0220 - val_acc: 0.8060\n",
      "Epoch 17/20\n",
      "7982/7982 [==============================] - 1s 65us/step - loss: 0.1217 - acc: 0.9579 - val_loss: 1.0254 - val_acc: 0.7970\n",
      "Epoch 18/20\n",
      "7982/7982 [==============================] - 1s 65us/step - loss: 0.1198 - acc: 0.9582 - val_loss: 1.0430 - val_acc: 0.8060\n",
      "Epoch 19/20\n",
      "7982/7982 [==============================] - 1s 65us/step - loss: 0.1138 - acc: 0.9597 - val_loss: 1.0955 - val_acc: 0.7970\n",
      "Epoch 20/20\n",
      "7982/7982 [==============================] - 1s 65us/step - loss: 0.1111 - acc: 0.9593 - val_loss: 1.0674 - val_acc: 0.8020\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(partial_x_train,\n",
    "                    partial_y_train,\n",
    "                    epochs=20,\n",
    "                    batch_size=512,\n",
    "                    validation_data=(x_val, y_val))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "손실과 정확도 곡선을 그려 보죠:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xl8VOXZ//HPxSLIjoAbIAG1KmCEGBELAi6PP7XutSqCC+qDWK1bbeVxq9rSulBF1MelT7Vaomi1ttZira0o2lY0ICCICCpIBDGgbIJK4Pr9cZ8kQ5gkA5nJnEm+79frvObMmXvOXHMymWvu5dzH3B0REZG4aZLtAERERJJRghIRkVhSghIRkVhSghIRkVhSghIRkVhSghIRkVhSgpJYMrOmZrbezPZKZ9lsMrN9zCzt53WY2dFmtjjh/gIzOzyVsjvwWv9nZtft6PNr2O8vzOx36d6v5LZm2Q5AGgYzW59wtxXwDbA5un+xuxdtz/7cfTPQJt1lGwN33y8d+zGzi4CR7j4sYd8XpWPfIqlQgpK0cPeKBBH9Qr/I3f9RXXkza+buZfURm4jkJjXxSb2ImnCeMrMnzWwdMNLMDjOzN81stZktN7OJZtY8Kt/MzNzM8qL7k6LHXzSzdWb2HzPrub1lo8ePM7MPzGyNmd1rZv8ys/OriTuVGC82s0Vm9qWZTUx4blMzu9vMVpnZh8CxNRyfG8xscpVt95vZXdH6RWY2P3o/H0a1m+r2VWJmw6L1Vmb2+yi2ecDBSV73o2i/88zspGj7gcB9wOFR8+nKhGN7c8Lzx0TvfZWZ/cnM9kjl2NTGzE6J4lltZq+Y2X4Jj11nZsvMbK2ZvZ/wXgea2cxo+wozuzPV15OYcnctWtK6AIuBo6ts+wXwLXAi4YfRzsAhwKGEmnwv4APgsqh8M8CBvOj+JGAlUAg0B54CJu1A2V2BdcDJ0WNXA5uA86t5L6nE+GegPZAHfFH+3oHLgHlAN6ATMC38yyV9nV7AeqB1wr4/Bwqj+ydGZQw4EtgI5EePHQ0sTthXCTAsWh8PvAp0BHoA71UpewawR/Q3OTuKYbfosYuAV6vEOQm4OVo/JoqxH9AS+F/glVSOTZL3/wvgd9H6AVEcR0Z/o+ui494c6AMsAXaPyvYEekXrbwPDo/W2wKHZ/l/QUrdFNSipT2+4+1/cfYu7b3T3t919uruXuftHwMPA0Bqe/4y7F7v7JqCI8MW4vWVPAGa5+5+jx+4mJLOkUozxV+6+xt0XE5JB+WudAdzt7iXuvgq4rYbX+QiYS0icAP8FrHb34ujxv7j7Rx68AvwTSDoQooozgF+4+5fuvoRQK0p83afdfXn0N3mC8OOiMIX9AowA/s/dZ7n718BYYKiZdUsoU92xqclZwPPu/kr0N7oNaEf4oVBGSIZ9ombij6NjB+GHxr5m1snd17n79BTfh8SUEpTUp6WJd8xsfzP7q5l9ZmZrgVuBzjU8/7OE9Q3UPDCiurJ7Jsbh7k6ocSSVYowpvRbhl39NngCGR+tnExJreRwnmNl0M/vCzFYTai81Hatye9QUg5mdb2azo6a01cD+Ke4Xwvur2J+7rwW+BLomlNmev1l1+91C+Bt1dfcFwI8Jf4fPoybj3aOio4DewAIze8vMjk/xfUhMKUFJfao6xPohQq1hH3dvB9xEaMLKpOWEJjcAzMzY+gu1qrrEuBzonnC/tmHwTwFHRzWQkwkJCzPbGXgG+BWh+a0D8PcU4/isuhjMrBfwAHAJ0Cna7/sJ+61tSPwyQrNh+f7aEpoSP00hru3ZbxPC3+xTAHef5O6DCM17TQnHBXdf4O5nEZpxfw08a2Yt6xiLZJESlGRTW2AN8JWZHQBcXA+v+QJQYGYnmlkz4AqgS4ZifBq40sy6mlkn4NqaCrv7CuAN4FFggbsvjB5qAewElAKbzewE4KjtiOE6M+tg4TyxyxIea0NIQqWEXH0RoQZVbgXQrXxQSBJPAheaWb6ZtSAkitfdvdoa6XbEfJKZDYte+yeEfsPpZnaAmR0Rvd7GaNlMeAPnmFnnqMa1JnpvW+oYi2SREpRk04+B8whfPg8RahAZFSWBM4G7gFXA3sA7hPO20h3jA4S+oncJHfjPpPCcJwiDHp5IiHk1cBXwHGGgwemERJuKnxFqcouBF4HHE/Y7B5gIvBWV2R9I7Ld5GVgIrDCzxKa68uf/jdDU9lz0/L0I/VJ14u7zCMf8AULyPBY4KeqPagHcQeg3/IxQY7sheurxwHwLo0THA2e6+7d1jUeyx0ITvEjjZGZNCU1Kp7v769mOR0QqqQYljY6ZHWtm7aNmohsJI8PeynJYIlKFEpQ0RoOBjwjNRMcCp7h7dU18IpIlauITEZFYUg1KRERiKecmi+3cubPn5eVlOwwREdlBM2bMWOnuNZ3eAeRggsrLy6O4uDjbYYiIyA4ys9pmVQHUxCciIjGlBCUiIrGkBCUiIrGUc31QItJ4bdq0iZKSEr7++utshyIpaNmyJd26daN58+qmc6yZEpSI5IySkhLatm1LXl4eYSJ6iSt3Z9WqVZSUlNCzZ8/an5BEo2riKyqCvDxo0iTcFhXV9gwRiZOvv/6aTp06KTnlADOjU6dOdartNpoaVFERjB4NGzaE+0uWhPsAI+o8/7KI1Bclp9xR179Vo6lBXX99ZXIqt2FD2C4iIvGTsQRlZt3NbKqZzTezeWZ2RZIyw8xsjZnNipabMhXPJ59s33YRkapWrVpFv3796NevH7vvvjtdu3atuP/tt6ldemrUqFEsWLCgxjL3338/RWnqgxg8eDCzZs1Ky77qWyab+MqAH7v7zOhS0DPM7GV3f69Kudfd/YQMxgHAXnuFZr1k20WkYSoqCq0kn3wS/tfHjatbk36nTp0qvuxvvvlm2rRpwzXXXLNVGXfH3WnSJPnv/0cffbTW17n00kt3PMgGJGM1KHdf7u4zo/V1wHyga6ZerzbjxkGrVltva9UqbBeRhqe833nJEnCv7HfOxOCoRYsW0bdvX8aMGUNBQQHLly9n9OjRFBYW0qdPH2699daKsuU1mrKyMjp06MDYsWM56KCDOOyww/j8888BuOGGG5gwYUJF+bFjxzJgwAD2228//v3vfwPw1Vdf8f3vf5+DDjqI4cOHU1hYWGtNadKkSRx44IH07duX6667DoCysjLOOeeciu0TJ04E4O6776Z3794cdNBBjBw5Mu3HLBX10gdlZnlAf7a+nHS5w8xstpm9aGZ9qnn+aDMrNrPi0tLSHYphxAh4+GHo0QPMwu3DD2uAhEhDVd/9zu+99x4XXngh77zzDl27duW2226juLiY2bNn8/LLL/Pee1Ubj2DNmjUMHTqU2bNnc9hhh/HII48k3be789Zbb3HnnXdWJLt7772X3XffndmzZzN27FjeeeedGuMrKSnhhhtuYOrUqbzzzjv861//4oUXXmDGjBmsXLmSd999l7lz53LuuecCcMcddzBr1ixmz57NfffdV8ejs2MynqDMrA3wLHClu6+t8vBMoIe7HwTcC/wp2T7c/WF3L3T3wi5dap0At1ojRsDixbBlS7hVchJpuOq733nvvffmkEMOqbj/5JNPUlBQQEFBAfPnz0+aoHbeeWeOO+44AA4++GAWL16cdN+nnXbaNmXeeOMNzjrrLAAOOugg+vRJ+vu+wvTp0znyyCPp3LkzzZs35+yzz2batGnss88+LFiwgCuuuIKXXnqJ9u3bA9CnTx9GjhxJUVHRDp9oW1cZTVBm1pyQnIrc/Y9VH3f3te6+PlqfAjQ3s86ZjElEGofq+pcz1e/cunXrivWFCxdyzz338MorrzBnzhyOPfbYpOcD7bTTThXrTZs2paysLOm+W7RosU2Z7b3YbHXlO3XqxJw5cxg8eDATJ07k4osvBuCll15izJgxvPXWWxQWFrJ58+bter10yOQoPgN+C8x397uqKbN7VA4zGxDFsypTMYlI45HNfue1a9fStm1b2rVrx/Lly3nppZfS/hqDBw/m6aefBuDdd99NWkNLNHDgQKZOncqqVasoKytj8uTJDB06lNLSUtydH/zgB9xyyy3MnDmTzZs3U1JSwpFHHsmdd95JaWkpG6q2l9aDTI7iGwScA7xrZuU9d9cBewG4+4PA6cAlZlYGbATOcl2DXkTSoLwJP52j+FJVUFBA79696du3L7169WLQoEFpf40f/ehHnHvuueTn51NQUEDfvn0rmueS6datG7feeivDhg3D3TnxxBP53ve+x8yZM7nwwgtxd8yM22+/nbKyMs4++2zWrVvHli1buPbaa2nbtm3a30NtLNfyQWFhoeuChSKN0/z58znggAOyHUYslJWVUVZWRsuWLVm4cCHHHHMMCxcupFmzeE0QlOxvZmYz3L2wtufG652IiEhK1q9fz1FHHUVZWRnuzkMPPRS75FRXDevdiIg0Eh06dGDGjBnZDiOjGs1cfCIikluUoEREJJaUoEREJJaUoEREJJaUoEREUjRs2LBtTrqdMGECP/zhD2t8Xps2bQBYtmwZp59+erX7ru0UmgkTJmx1wuzxxx/P6tWrUwm9RjfffDPjx4+v837STQlKRCRFw4cPZ/LkyVttmzx5MsOHD0/p+XvuuSfPPPPMDr9+1QQ1ZcoUOnTosMP7izslKBGRFJ1++um88MILfPPNNwAsXryYZcuWMXjw4IrzkgoKCjjwwAP585//vM3zFy9eTN++fQHYuHEjZ511Fvn5+Zx55pls3Lixotwll1xScamOn/3sZwBMnDiRZcuWccQRR3DEEUcAkJeXx8qVKwG466676Nu3L3379q24VMfixYs54IAD+O///m/69OnDMcccs9XrJDNr1iwGDhxIfn4+p556Kl9++WXF6/fu3Zv8/PyKSWpfe+21igs29u/fn3Xr1u3wsU1G50GJSE668kpI94Vi+/WD6Ls9qU6dOjFgwAD+9re/cfLJJzN58mTOPPNMzIyWLVvy3HPP0a5dO1auXMnAgQM56aSTiKYb3cYDDzxAq1atmDNnDnPmzKGgoKDisXHjxrHLLruwefNmjjrqKObMmcPll1/OXXfdxdSpU+ncees5tWfMmMGjjz7K9OnTcXcOPfRQhg4dSseOHVm4cCFPPvkkv/nNbzjjjDN49tlna7y+07nnnsu9997L0KFDuemmm7jllluYMGECt912Gx9//DEtWrSoaFYcP348999/P4MGDWL9+vW0bNlyO4527VSDEhHZDonNfInNe+7OddddR35+PkcffTSffvopK1asqHY/06ZNq0gU+fn55OfnVzz29NNPU1BQQP/+/Zk3b16tE8G+8cYbnHrqqbRu3Zo2bdpw2mmn8frrrwPQs2dP+vXrB9R8SQ8I16davXo1Q4cOBeC8885j2rRpFTGOGDGCSZMmVcxYMWjQIK6++momTpzI6tWr0z6ThWpQIpKTaqrpZNIpp5zC1VdfzcyZM9m4cWNFzaeoqIjS0lJmzJhB8+bNycvLS3qJjUTJalcff/wx48eP5+2336Zjx46cf/75te6npjlVyy/VAeFyHbU18VXnr3/9K9OmTeP555/n5z//OfPmzWPs2LF873vfY8qUKQwcOJB//OMf7L///ju0/2RUgxIR2Q5t2rRh2LBhXHDBBVsNjlizZg277rorzZs3Z+rUqSxZsqTG/QwZMoSi6Przc+fOZc6cOUC4VEfr1q1p3749K1as4MUXX6x4Ttu2bZP28wwZMoQ//elPbNiwga+++ornnnuOww8/fLvfW/v27enYsWNF7ev3v/89Q4cOZcuWLSxdupQjjjiCO+64g9WrV7N+/Xo+/PBDDjzwQK699loKCwt5//33t/s1a6IalIjIdho+fDinnXbaViP6RowYwYknnkhhYSH9+vWrtSZxySWXMGrUKPLz8+nXrx8DBgwAwtVx+/fvT58+fba5VMfo0aM57rjj2GOPPZg6dWrF9oKCAs4///yKfVx00UX079+/xua86jz22GOMGTOGDRs20KtXLx599FE2b97MyJEjWbNmDe7OVVddRYcOHbjxxhuZOnUqTZs2pXfv3hVXB04XXW5DRHKGLreRe+pyuQ018YmISCwpQYmISCwpQYlITsm1bonGrK5/KyUoEckZLVu2ZNWqVUpSOcDdWbVqVZ1O3tUoPhHJGd26daOkpITS0tJshyIpaNmyJd26ddvh5ytBiUjOaN68OT179sx2GFJP1MQnIiKxpAQlIiKxpAQlIiKxpAQlIiKxpAQlIiKxpAQlIiKxpAQlIiKxpAQlIiKxpAQlIiKxpAQlIiKxpAQlIiKxlLEEZWbdzWyqmc03s3lmdkWSMmZmE81skZnNMbOCTMUjIiK5JZOTxZYBP3b3mWbWFphhZi+7+3sJZY4D9o2WQ4EHolsREWnkMlaDcvfl7j4zWl8HzAe6Vil2MvC4B28CHcxsj0zFJCIiuaNe+qDMLA/oD0yv8lBXYGnC/RK2TWKY2WgzKzazYl0HRkSkcch4gjKzNsCzwJXuvrbqw0mess2lMt39YXcvdPfCLl26ZCJMERGJmYwmKDNrTkhORe7+xyRFSoDuCfe7AcsyGZOIiOSGTI7iM+C3wHx3v6uaYs8D50aj+QYCa9x9eaZiEhGR3JHJUXyDgHOAd81sVrTtOmAvAHd/EJgCHA8sAjYAozIYj4iI5JCMJSh3f4PkfUyJZRy4NFMxiIhI7tJMEiIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEkuNKkG5w6RJ8NBD2Y5ERERq06gSlBn84Q9wzTWwfHm2oxERkZo0qgQFMH48fPMN3HBDtiMREZGaNLoEte++cPnl8OijMHNmtqMREZHqNLoEBXDjjdC5M1xxReiXEhGR+GmUCap9e/j5z+GNN+CZZ7IdjYiIJNMoExTARRdBfj785CewcWO2oxERkaoabYJq2hQmTIAlS+Cuu7IdjYiIVNVoExTAEUfAqafCr34Fy5ZlOxoREUnUqBMUwJ13wqZNcN112Y5EREQSNfoEtffecOWV8Nhj8Pbb2Y5GRETKNfoEBXD99bDrriFRadi5iEg8KEEB7drBuHHw73/DU09lOxoREQElqAqjRkG/fvDTn8KGDdmORkRElKAiTZvCPffA0qXw619nOxoREVGCSjBkCJx+Otx2G3z6abajERFp3DKWoMzsETP73MzmVvP4MDNbY2azouWmTMWyPe64AzZvhrFjsx2JiEjjlska1O+AY2sp87q794uWWzMYS8p69oSrrw4XNpw+PdvRiIg0XhlLUO4+DfgiU/vPpP/5H9h9dw07FxHJpmz3QR1mZrPN7EUz61NdITMbbWbFZlZcWlqa8aDatoVf/hLefBOeeCLjLyciIkmYZ7CKYGZ5wAvu3jfJY+2ALe6+3syOB+5x931r22dhYaEXFxenPdaqtmyBAQPgs89gwQJo3TrjLyki0iiY2Qx3L6ytXNZqUO6+1t3XR+tTgOZm1jlb8VTVpEmY7fzTT8N8fYmKiiAvL5TJywv3RUQkvbKWoMxsdzOzaH1AFMuqbMWTzODBcOaZYWTf0qVhW1ERjB4dLtPhHm5Hj1aSEhFJt0wOM38S+A+wn5mVmNmFZjbGzMZERU4H5prZbGAicJZnsr1xB91+e0hE5cPOr79+25kmNmwI20VEJH0y2geVCfXVB5XoxhvhF78Ic/UNGpR8ZJ9Z6LcSEZGaxb4PKpdcey3suSdccQV07568zF571W9MIiINXUoJysz2NrMW0fowM7vczDpkNrT4aNMmXHX37bfh+OOhVautH2/VKsyGLiIi6ZNqDepZYLOZ7QP8FugJNKozhEaODMPOn38eJk6EHj1Cs16PHvDwwzBiRLYjFBFpWJqlWG6Lu5eZ2anABHe/18zeyWRgcVM+7Py734XFi8MiIiKZk2oNapOZDQfOA16ItjXPTEjxddhhMHw4jB8fhpeLiEjmpJqgRgGHAePc/WMz6wlMylxY8XX77aFp79prsx2JiEjDllKCcvf33P1yd3/SzDoCbd39tgzHFkvdu4er7j71FLz4YrajERFpuFIdxfeqmbUzs12A2cCjZnZXZkOLr5/+FHr3hpNOggcfzHY0IiINU6pNfO3dfS1wGvCoux8MHJ25sOKtVatw0u4xx8All8CYMfDtt9mOSkSkYUk1QTUzsz2AM6gcJNGotW8fhpyPHQsPPQRHHw2ff57tqEREGo5UE9StwEvAh+7+tpn1AhZmLqzc0LRpOIH3iSfCSbyHHALvNKrB9yIimZPqIIk/uHu+u18S3f/I3b+f2dByx/Dh8MYbYS6+QYPCAAoREambVAdJdDOz58zsczNbYWbPmlm3TAeXSw4+GIqLoaAAzjorzG6uyWNFRHZcqk18jwLPA3sCXYG/RNskwW67wT//CRddFC4Zf8opsHZttqMSEclNqSaoLu7+qLuXRcvvgC4ZjCtntWgR5ua77z6YMgUGDoSFjb63TkRk+6WaoFaa2UgzaxotI4nZ1W/jxAwuvRT+8Y8wsm/AAPj737MdlYhIbkk1QV1AGGL+GbCccDXcUZkKqqEYNiyM7uveHY47Du6+O/nFDkVEZFupjuL7xN1Pcvcu7r6ru59COGlXatGzZzip95RT4OqrYdQo+PrrbEclIhJ/dbmi7tVpi6KBa9MG/vAHuOUWeOyxULNatizbUYmIxFtdEpSlLYpGoEkTuOkm+OMfYe5cKCyE6dOzHZWISHzVJUGpN2UHnHoq/Oc/0LIlDB0Kv/ud+qVERJKpMUGZ2TozW5tkWUc4J0p2wIEHhsETgwaFPqn8/DA0/auvsh2ZiEh81Jig3L2tu7dLsrR191QvFy9JdOoEL70Ev/0tNGsGF18M3brBNdfARx9lOzoRkeyrSxOf1FGzZnDBBTBzJrz+erh8x4QJsM8+4VpTL7+s5j8RabyUoGLADAYPDpPMLlkS5vGbPj0krN694f77Yd26bEcpIlK/lKCypKgI8vLC6L68vHAfoGtX+PnP4ZNP4PHHoW1buOyy0Px3xRWaNklEGg8lqCwoKoLRo0NtyT3cjh5dmaQgzOl3zjnw1lvw5ptw4onwwAPwne+EWSmmTNFs6SKSulz8vjDPsU6OwsJCLy4uznYYdZKXF5JSVT16wOLF1T/vs8/CaL8HH4Tly0Nf1aWXhpGA7dtnKloRyUWrV4fr1E2bBq+9Fvq6O3YMs9v06rXtbffuoV+8PpjZDHcvrLWcElT9a9Ik+eAHs9R+5Xz7bTjh9957wzRKrVvDeeeFJsDvfCf98YpI/K1cGQZbvfZaSEqzZoXvmebN4dBDw7J2bRgl/PHH4Ufy5s2Vz2/aFPbaK3ny6tUrjDy2NE3PoAQVYztag0pm5syQqJ54IiSuE06Aq66CI45I34dJROLns88qa0evvQbz5oXtLVvCYYeFiQCGDAmX/Nl5522fX1YGJSWVCavq7eefb12+TZvKpPX449Cu3Y7HrgQVY+V9UBs2VG5r1So0340YsWP7XLEi9FH97/9CaSkcdBBceWW4HH2LFumJW0SyZ+nSymQ0bRp88EHY3qZNOOl/yJCQlAoL0/M/v359SFRVk9fSpTBjRqhx7SglqJgrKgrDyT/5JFSrx43b8eSU6OuvQ23q7rvDnH+77QY//CFccgl00SUmRWJn06bQPFdauu2ycmWoycyYUdm60r49HH54ZQ2poKD++o7SJesJysweAU4APnf3vkkeN+Ae4HhgA3C+u8+sbb8NJUFlmnu4/Pxdd8GLL4ZfVCNHhua/Pn2yHZ1IbnAPTWG1LZs2Jd++YUPyxFOefEpLw2CGZMxgl13CD8sDDggJaejQMFVaXWovcRCHBDUEWA88Xk2COh74ESFBHQrc4+6H1rZfJajtN38+3HNPaDfeuDGcAHzVVeG2iU40EOHrr8PpHK+9Bq++GubK3LgxvUOzmzWDzp1Dwkm2VH1sl11yr2aUqqwnqCiIPOCFahLUQ8Cr7v5kdH8BMMzdl9e0TyWoHbdqFTz0ENx3XximfsABYeTfOeeEPjCRxmLjxpCQXn01JKU334Rvvgk/2Pr3h+9+NzSlNWu2/Uvz5pXrLVtWJpz27TVwqVyqCSqb+bkrsDThfkm0rcYEJTuuUye47rowIe3TT4d+qjFjQl/YmDHhnKo99sh2lCLpt2FDuMxNeQ1p+vQw6rVJk9CH86MfheazwYOhQ4dsRyvlspmgkv2WSFqdM7PRwGiAvfbaK5MxNQo77RT6o0aMCOdN3H03/PKXcMcdYTRQYWFYDj4Y9t5bv/ok92zYEM4RLK8hTZ8e+omaNAmf6yuuqExIOsk9vrKZoEqA7gn3uwFJL4Tu7g8DD0No4st8aI2DWRgFNGQILFoUZqh4/fVwXtU334QyHTqEX5jlCauwMJwHoaQldbVpU/jczZ0bzuFZsWLrx8s/Y4mftdrWN2+G2bPDFGGbNoXBBAcfHPpcyxNSXc7fkfqVzQT1PHCZmU0mDJJYU1v/k2TOPvvA+PFhfdOm8IVRXByGtxYXh8uAfPtteLxjx8pkVX7bo4eSliS3eXM4h2bevMpkNHcuvP9++KxBqNkkzlSQ2DVevp5sW7L1/faDq68OCWnQICWkXJbJUXxPAsOAzsAK4GdAcwB3fzAaZn4fcCxhmPkod6919IMGSWTHt9+GL5Xi4srE9e67lV8wnTptnbQOOCA0D+60U3bjlvrjHk7iTExC8+bBe++FQQnl8vLCqQ59+1be7r9/8tkOpGGKxSi+TFCCio9vvglJKjFpzZ0bzv+A0LzSs2f4RVt12W239NW43MM5JeVnuyee+f7xx6GP4ZBDKpc+fRru8N36sG4dfPhh5fLBByERzZu39XXL9tyzMgGVJ6PevcMlZKRxU4KSrPj668rmmwULKpcPPgiPlWvfPnni2nffMDS3qg0btk08idOvfPXV1uV33TUkx7y8MLz+7bdhzZrw2M47h361xKS1zz7xaqLcvDlMNbNuXeVt4nqTJuE8mcSlffv0nNfmHk4gXbRo60T04YdhW2np1uW7dAkERk6UAAANqElEQVSJJzER9ekTYhJJRglKYmXLltD8k5i0FiwIiaykpLKcWejP2m+/0Ne1eHFIQlU70Fu3Dgko2aUD8vLC/GRVX3/RopCoypeZMyuTZseOoXkyMWl17bpj7/Xbb+HLL+GLL0Jy/OKLyvUvv6w+6SSuJzaJpcosvI/yhJW4XnXp2DEktBUrtk4+5evr12+9327dQhLfe+9tF42Ck+2lBNVIZGpOv/r01VehhlU1ea1eHZJNeeJJTEJdutS9xlM+GCQxab37buUlCPbYAwYMCMmqf/+wvWrCSbae2MxVVZMmoYmrTZtwW916bY9v3hxeqzwR1rasXp38Ei/ldtopHNe99942EeXlJa/ViuwoJahGIBOzojd2GzaE6+gkJq3yWaMTlTexdepUWSspX0+2rXy9bdvsNCVu2RKaOKsmrV13DUmoa9fcn99NcocSVCOQzutKSfVWrw79ai1bbp1oNI+hyI7JhamOpI4++WT7tsuO6dAhnOApIvVLvwFzWHWzPmk2KBFpCJSgcti4cdvOQt6qVdguIpLrlKBy2IgRYUBE+TRDPXpogISINBzqg8pxI0YoIYlIw6QalIiIxJISlIiIxJISlIiIxJISlIiIxJISlIiIxJISlIiIxJISlABh4tm8vDC/XF5euC8ikk06D0q2mRV9yZJwH3SOlYhkj2pQwvXXb33JDgj3r78+O/GIiIASlKBZ0UUknpSgRLOii0gsKUGJZkUXkVhSghLNii4isaRRfAJoVnQRiR/VoEREJJaUoEREJJaUoEREJJaUoCStNGWSiKSLBklI2mjKJBFJJ9WgJG00ZZKIpJMSlKSNpkwSkXRSgpK00ZRJIpJOSlCSNpoySUTSKaMJysyONbMFZrbIzMYmefx8Mys1s1nRclEm45HM0pRJIpJOGRvFZ2ZNgfuB/wJKgLfN7Hl3f69K0afc/bJMxSH1S1MmiUi6ZLIGNQBY5O4fufu3wGTg5Ay+noiINCCZTFBdgaUJ90uibVV938zmmNkzZtY92Y7MbLSZFZtZcWlpaSZilRjRyb4iAplNUJZkm1e5/xcgz93zgX8AjyXbkbs/7O6F7l7YpUuXNIcpcVJ+su+SJeBeebKvkpRI45PJBFUCJNaIugHLEgu4+yp3/ya6+xvg4AzGIzlAJ/uKSLlMJqi3gX3NrKeZ7QScBTyfWMDM9ki4exIwP4PxSA7Qyb4iUi5jo/jcvczMLgNeApoCj7j7PDO7FSh29+eBy83sJKAM+AI4P1PxSG7Ya6/QrJdsu4g0LuZetVso3goLC724uDjbYUiGVJ1wFsLJvjqfSqThMLMZ7l5YWznNJCGxopN9RaScLrchsaOTfUUEVIOSBkznU4nkNtWgpEHSxRNFcp9qUNIg6XwqkdynBCUNks6nEsl9SlDSIOniiSK5TwlKGqR0XjxRgy1EskMJShqkdJ1PpclrRbJHM0mI1CAvL/nUSz16wOLF9R2NSMOgmSRE0kCDLUSyRwlKpAbpGmyhfiyR7acEJVKDdAy2UD+WyI5RghKpQToGW+ikYZEdowQlUosRI8KAiC1bwu32jgRMZz+WmgqlMVGCEsmwdPZjqalQGhMlKJEMS9dJw2oqlMZGCUokw9J10rCaCqWxUYISqQd17ceC+DUVKslJpilBieSIODUVprM/TIlOqqMEJZIj4tRUmK7+MA38kJooQYnkkLg0FaarPyydAz/SURNTbS5elKBEGpl0NBWmqz8sXYkuHTWxODZbNvqE6e45tRx88MEuInUzaZJ7jx7uZuF20qTtf36rVu7hqzwsrVpt/3569Nh6H+VLjx71v590xZKuY5Ou/ZTvqy5/73TtoxxQ7Cl832c94WzvogQlEg/p+tJLx5ewWfLkYla/+3CPV9J1T88xTmeydE89Qel6UCKSVUVFoc/pk09CE+G4cdvft5aO63al69pfTZqEr/CqzELfYX3vJ07HppyuByUiOSEdAz/S0a+WrmH86eqfi1M/X7aui6YEJSI5Lx1D8NM1jD9diS5OCTNdyXK7pdIOGKdFfVAiEnfpGlAQl34+9UGlSH1QIiLbJx39fOnYR7lU+6CUoEREpF5pkISIiOS0jCYoMzvWzBaY2SIzG5vk8RZm9lT0+HQzy8tkPCIikjsylqDMrClwP3Ac0BsYbma9qxS7EPjS3fcB7gZuz1Q8IiKSWzJZgxoALHL3j9z9W2AycHKVMicDj0XrzwBHmZllMCYREckRmUxQXYGlCfdLom1Jy7h7GbAG6FR1R2Y22syKzay4tLQ0Q+GKiEicNMvgvpPVhKoOGUylDO7+MPAwgJmVmlmSSTdiqzOwMttBbAfFm1m5Fi/kXsyKN7PSEW+PVAplMkGVAN0T7ncDllVTpsTMmgHtgS9q2qm7d0lnkJlmZsWpDKeMC8WbWbkWL+RezIo3s+oz3kw28b0N7GtmPc1sJ+As4PkqZZ4HzovWTwde8Vw7MUtERDIiYzUody8zs8uAl4CmwCPuPs/MbiVMc/E88Fvg92a2iFBzOitT8YiISG7JZBMf7j4FmFJl200J618DP8hkDDHwcLYD2E6KN7NyLV7IvZgVb2bVW7w5N9WRiIg0DprqSEREYkkJSkREYkkJqo7MrLuZTTWz+WY2z8yuSFJmmJmtMbNZ0XJTsn3VJzNbbGbvRvFsMz28BROjeRLnmFlBNuKMYtkv4djNMrO1ZnZllTJZPcZm9oiZfW5mcxO27WJmL5vZwui2YzXPPS8qs9DMzktWpp7ivdPM3o/+3s+ZWYdqnlvjZ6eeY77ZzD5N+LsfX81za5wXtB7jfSoh1sVmNqua59b7Ma7uuyyrn+NULhqlpfoF2AMoiNbbAh8AvauUGQa8kO1Yq8S0GOhcw+PHAy8STqYeCEzPdsxRXE2Bz4AecTrGwBCgAJibsO0OYGy0Pha4PcnzdgE+im47RusdsxTvMUCzaP32ZPGm8tmp55hvBq5J4TPzIdAL2AmYXfV/tL7irfL4r4Gb4nKMq/suy+bnWDWoOnL35e4+M1pfB8xn2ymdctHJwOMevAl0MLM9sh0UcBTwobvHajYRd5/GtieZJ841+RhwSpKn/j/gZXf/wt2/BF4Gjs1YoJFk8br73z1MOQbwJuHk+tio5hinIpV5QdOupnijOUfPAJ7MdBypquG7LGufYyWoNIouF9IfmJ7k4cPMbLaZvWhmfeo1sOQc+LuZzTCz0UkeT2UuxWw4i+r/qeN2jHdz9+UQ/vmBXZOUietxvoBQg06mts9OfbssapZ8pJrmpzge48OBFe6+sJrHs3qMq3yXZe1zrASVJmbWBngWuNLd11Z5eCahSeog4F7gT/UdXxKD3L2AcDmUS81sSJXHU5onsT5FM5KcBPwhycNxPMapiONxvh4oA4qqKVLbZ6c+PQDsDfQDlhOazaqK3TEGhlNz7Slrx7iW77Jqn5ZkW52PsRJUGphZc8IftMjd/1j1cXdf6+7ro/UpQHMz61zPYVaNaVl0+znwHKEZJFEqcynWt+OAme6+ouoDcTzGwIryZtHo9vMkZWJ1nKPO7ROAER51LlSVwmen3rj7Cnff7O5bgN9UE0vcjnEz4DTgqerKZOsYV/NdlrXPsRJUHUVtyb8F5rv7XdWU2T0qh5kNIBz3VfUX5TbxtDaztuXrhM7xuVWKPQ+cG43mGwisKa/mZ1G1vzrjdowjiXNNngf8OUmZl4BjzKxj1Dx1TLSt3pnZscC1wEnuvqGaMql8dupNlX7RU6uJJZV5QevT0cD77l6S7MFsHeMavsuy9zmuz1EiDXEBBhOqsnOAWdFyPDAGGBOVuQyYRxg99Cbw3SzH3CuKZXYU1/XR9sSYjXBF5A+Bd4HCLMfcipBw2idsi80xJiTO5cAmwq/JCwnXNvsnsDC63SUqWwj8X8JzLwAWRcuoLMa7iNCPUP45fjAquycwpabPThZj/n30+ZxD+CLdo2rM0f3jCaPSPqyvmJPFG23/XfnnNqFs1o9xDd9lWfsca6ojERGJJTXxiYhILClBiYhILClBiYhILClBiYhILClBiYhILClBiaSRmW22rWdeT9vM2WaWlzgztkhDl9FLvos0QhvdvV+2gxBpCFSDEqkH0fV9bjezt6Jln2h7DzP7ZzTZ6T/NbK9o+24Wrsk0O1q+G+2qqZn9Jrpez9/NbOeo/OVm9l60n8lZepsiaaUEJZJeO1dp4jsz4bG17j4AuA+YEG27j3BZk3zC5KwTo+0Tgdc8TH5bQJhRAGBf4H537wOsBr4fbR8L9I/2MyZTb06kPmkmCZE0MrP17t4myfbFwJHu/lE0Iedn7t7JzFYSpufZFG1f7u6dzawU6Obu3yTsI49wzZ19o/vXAs3d/Rdm9jdgPWEW9z95NHGuSC5TDUqk/ng169WVSeabhPXNVPYjf48wd+LBwIxoxmyRnKYEJVJ/zky4/U+0/m/C7NoAI4A3ovV/ApcAmFlTM2tX3U7NrAnQ3d2nAj8FOgDb1OJEco1+ZYmk185mNivh/t/cvXyoeQszm074YTg82nY58IiZ/QQoBUZF268AHjazCwk1pUsIM2Mn0xSYZGbtCbPQ3+3uq9P2jkSyRH1QIvUg6oMqdPeV2Y5FJFeoiU9ERGJJNSgREYkl1aBERCSWlKBERCSWlKBERCSWlKBERCSWlKBERCSW/j9cIGPjQIwpygAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "\n",
    "epochs = range(1, len(loss) + 1)\n",
    "\n",
    "plt.plot(epochs, loss, 'bo', label='Training loss')\n",
    "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
    "plt.title('Training and validation loss')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3XmcFNW5//HPwyabsqpREAaNRmQZHUfQiEaDQTSKu0IwkRAlLpio1yREvFd/RjRRUTR6jYgxJo4L0QuiUUlU4pJEBRRQcAERdATZRLZBZHl+f5yaoafpmemB7unqme/79apXd1Wdqn66pqefPqdOnTJ3R0REJG4a5ToAERGRVJSgREQklpSgREQklpSgREQklpSgREQklpSgREQklpSgJCvMrLGZrTezLpksm0tm9k0zy/h1GWZ2gpktSpj/wMyOSafsTrzWBDO7Zme3F6lLTXIdgMSDma1PmG0JbAK2RvM/dfeS2uzP3bcCrTNdtiFw929lYj9mdiFwvrsfl7DvCzOxb5G6oAQlALh7RYKIfqFf6O4vVFXezJq4+5a6iE2kJvo81k9q4pO0mNmNZva4mT1qZuuA883sKDN73cy+NLOlZnaXmTWNyjcxMzezgmj+4Wj9c2a2zsz+Y2bdals2Wn+SmX1oZmvM7Pdm9i8zG1ZF3OnE+FMzW2Bmq83sroRtG5vZHWa2ysw+AgZWc3yuNbPHkpbdY2a3R88vNLP3ovfzUVS7qWpfpWZ2XPS8pZn9JYptLnB4itddGO13rpkNipb3Au4GjomaT1cmHNvrE7a/OHrvq8xsspntk86xqc1xLo/HzF4wsy/M7HMz+2XC6/x3dEzWmtkMM9s3VXOqmb1W/neOjucr0et8AVxrZgea2bTovayMjlubhO27Ru9xRbT+TjNrHsXcPaHcPmZWZmYdqnq/UkfcXZOmShOwCDghadmNwNfAqYQfNi2AI4C+hJr4/sCHwMiofBPAgYJo/mFgJVAMNAUeBx7eibJ7AeuA06J1VwGbgWFVvJd0YnwKaAMUAF+Uv3dgJDAX6Ax0AF4J/zIpX2d/YD3QKmHfy4HiaP7UqIwB3wU2Ar2jdScAixL2VQocFz2/Dfgn0A7oCsxLKnsusE/0N/lBFMPe0boLgX8mxfkwcH30fEAU46FAc+B/gZfSOTa1PM5tgGXAz4HdgD2APtG6XwOzgQOj93Ao0B74ZvKxBl4r/ztH720LcAnQmPB5PAjoDzSLPif/Am5LeD/vRsezVVT+6GjdeGBMwuv8FzAp1/+HmlwJStOOE1UnqJdq2O5q4K/R81RJ5w8JZQcB7+5E2eHAqwnrDFhKFQkqzRiPTFj/f8DV0fNXCE2d5etOTv7STNr368APoucnAR9WU/YZ4LLoeXUJ6pPEvwVwaWLZFPt9F/h+9LymBPUQcFPCuj0I5x0713RsanmcfwjMqKLcR+XxJi1PJ0EtrCGGs4Hp0fNjgM+BxinKHQ18DFg0Pws4M9P/V5pqP6mJT2rj08QZMzvYzP4WNdmsBW4AOlaz/ecJz8uovmNEVWX3TYzDwzdKaVU7STPGtF4LWFxNvACPAEOi5z8AKjqWmNkpZvZG1MT1JaH2Ut2xKrdPdTGY2TAzmx01U30JHJzmfiG8v4r9uftaYDXQKaFMWn+zGo7zfsCCKmLYj5Ckdkby5/EbZjbRzD6LYvhTUgyLPHTIqcTd/0WojfUzs55AF+BvOxmTZJASlNRGchfr+wi/2L/p7nsA/0Oo0WTTUsIvfADMzKj8hZpsV2JcSvhiK1dTN/jHgRPMrDOhCfKRKMYWwBPAzYTmt7bA39OM4/OqYjCz/YF7Cc1cHaL9vp+w35q6xC8hNBuW7293QlPiZ2nElay64/wpcEAV21W1bkMUU8uEZd9IKpP8/n5H6H3aK4phWFIMXc2scRVx/Bk4n1Dbm+jum6ooJ3VICUp2xe7AGmBDdJL5p3Xwms8ARWZ2qpk1IZzX2DNLMU4ErjCzTtEJ819VV9jdlxGaoR4EPnD3+dGq3QjnRVYAW83sFMK5knRjuMbM2lq4TmxkwrrWhC/pFYRcfSGhBlVuGdA5sbNCkkeBn5hZbzPbjZBAX3X3Kmuk1ajuOE8BupjZSDNrZmZ7mFmfaN0E4EYzO8CCQ82sPSExf07ojNPYzEaQkEyriWEDsMbM9iM0M5b7D7AKuMlCx5MWZnZ0wvq/EJoEf0BIVhIDSlCyK/4LuIDQaeE+Qg0iq6IkcB5wO+EL5wDgbcIv50zHeC/wIvAOMJ1QC6rJI4RzSo8kxPwlcCUwidDR4GxCok3HdYSa3CLgORK+PN19DnAX8GZU5mDgjYRt/wHMB5aZWWJTXfn2zxOa4iZF23cBhqYZV7Iqj7O7rwG+B5xF6JTxIfCdaPWtwGTCcV5L6LDQPGq6vQi4htBh5ptJ7y2V64A+hEQ5BXgyIYYtwClAd0Jt6hPC36F8/SLC3/lrd/93Ld+7ZEn5SUGRvBQ12SwBznb3V3Mdj+QvM/szoePF9bmORQJdqCt5x8wGEppsviJ0U95CqEWI7JTofN5pQK9cxyLbqYlP8lE/YCGh6WcgcLpOasvOMrObCddi3eTun+Q6HtlOTXwiIhJLqkGJiEgs5d05qI4dO3pBQUGuwxARkZ00c+bMle5e3eUhQB4mqIKCAmbMmJHrMEREZCeZWU2jsgBq4hMRkZhSghIRkVhSghIRkVjKu3NQqWzevJnS0lK++uqrXIci1WjevDmdO3emadOqhoYTEdmuXiSo0tJSdt99dwoKCgiDW0vcuDurVq2itLSUbt261byBiDR49aKJ76uvvqJDhw5KTjFmZnTo0EG1XJEcKCmBggJo1Cg8lpTUtEU81IsEBSg55QH9jaShyFRCyMR+SkpgxAhYvBjcw+OIEbXfVy6SXL1JUCKSn+L2ZR6nhJCJ/YweDWVllZeVlYXldR1LreX6nvO1nQ4//HBPNm/evB2W1aWVK1d6YWGhFxYW+t577+377rtvxfymTZvS2sewYcP8/fffr7bM3Xff7Q8//HAmQs6ZXP+tJF4efti9ZUv38LUXppYtw/K63k+mYunatfI+yqeuXXOzH7PU+zGr+1jKATM8je/7nCec2k6ZSFAPPxwOrFl4zOR3/nXXXee33nrrDsu3bdvmW7duzdwL5SklqHjIxP9AJvYRpy/zOCWETO4nE+8rU7GUSzdBNbgmvrqsqi5YsICePXty8cUXU1RUxNKlSxkxYgTFxcX06NGDG264oaJsv379mDVrFlu2bKFt27aMGjWKwsJCjjrqKJYvXw7Atddey7hx4yrKjxo1ij59+vCtb32Lf/873AR0w4YNnHXWWRQWFjJkyBCKi4uZNWvWDrFdd911HHHEERXxhc8MfPjhh3z3u9+lsLCQoqIiFi1aBMBNN91Er169KCwsZHRt2gYkdjLxP5Cp/6NPqri5RVXLs7mfTMXSpUvtlmd7P2PGQMuWlZe1bBmW13UstZZOFovTtKs1qExXVZMl1qDmz5/vZuZvvvlmxfpVq1a5u/vmzZu9X79+PnfuXHd3P/roo/3tt9/2zZs3O+DPPvusu7tfeeWVfvPNN7u7++jRo/2OO+6oKP/LX/7S3d2feuopP/HEE93d/eabb/ZLL73U3d1nzZrljRo18rfffnuHOMvj2LZtmw8ePLji9YqKinzKlCnu7r5x40bfsGGDT5kyxfv16+dlZWWVtt0ZqkHtmrjUWuJU88nUfjIVS5yaLRP3tSufm0zG4q4aVJUy9SspXQcccABHHHFExfyjjz5KUVERRUVFvPfee8ybN2+HbVq0aMFJJ50EwOGHH15Ri0l25pln7lDmtddeY/DgwQAUFhbSo0ePlNu++OKL9OnTh8LCQl5++WXmzp3L6tWrWblyJaeeeioQLqxt2bIlL7zwAsOHD6dFixYAtG/fvvYHQnZZnGotmfo/ysSv+0ztJ1OxDB0K48dD165gFh7Hjw/Lc7Gf8n0tWgTbtoXHXMZSGw0uQdV1VbVVq1YVz+fPn8+dd97JSy+9xJw5cxg4cGDK64KaNWtW8bxx48Zs2bIl5b532223Hcq413wDyrKyMkaOHMmkSZOYM2cOw4cPr4gjVVdwd1cX8V2Uid5hmeiNBZn5H8jU/1GcvszjlBAyvZ9MyEUsDS5BZepX0s5Yu3Ytu+++O3vssQdLly5l6tSpGX+Nfv36MXHiRADeeeedlDW0jRs30qhRIzp27Mi6det48sknAWjXrh0dO3bk6aefBsIF0GVlZQwYMIAHHniAjRs3AvDFF19kPO76LE41H4hXbQPi9WUep4QgDTBB5aqqClBUVMQhhxxCz549ueiiizj66KMz/hqXX345n332Gb1792bs2LH07NmTNm3aVCrToUMHLrjgAnr27MkZZ5xB3759K9aVlJQwduxYevfuTb9+/VixYgWnnHIKAwcOpLi4mEMPPZQ77rgj43HXZ3Gq+UD8ahsiVbF0moTipLi42JNvWPjee+/RvXv3HEUUL1u2bGHLli00b96c+fPnM2DAAObPn0+TJvEYdrEh/q0aNQo1p2Rm4Zd6usprYonJrmVLJQbJP2Y2092LayrX4GpQ9d369es5+uijKSws5KyzzuK+++6LTXLKV7t6/ihONR+RfKJvrnqmbdu2zJw5M9dh1BvJtZby80eQfmIYMyZ1zWdnz9coIUlDoRqUSDUycf5INR+RnaMalEg1MtVzTjUfkdpTDUrqrUxce5SzIV5ERAlK6qdMXXuUy+vmRBo6JagMOO6443a46HbcuHFceuml1W7XunVrAJYsWcLZZ59d5b6Tu9UnGzduHGUJJ0pOPvlkvvzyy3RCr7cyde2Rzh+J5I4SVAYMGTKExx57rNKyxx57jCFDhqS1/b777ssTTzyx06+fnKCeffZZ2rZtu9P7qw8yOeaiRhcQyQ0lqAw4++yzeeaZZ9i0aRMAixYtYsmSJfTr14/169fTv39/ioqK6NWrF0899dQO2y9atIiePXsCYRiiwYMH07t3b84777yK4YUALrnkkopbdVx33XUA3HXXXSxZsoTjjz+e448/HoCCggJWrlwJwO23307Pnj3p2bNnxa06Fi1aRPfu3bnooovo0aMHAwYMqPQ65Z5++mn69u3LYYcdxgknnMCyZcuAcK3Vj3/8Y3r16kXv3r0rhkp6/vnnKSoqorCwkP79+2fk2O4snTsSyX/1rhffFVdAitsf7ZJDD4Xouz2lDh060KdPH55//nlOO+00HnvsMc477zzMjObNmzNp0iT22GMPVq5cyZFHHsmgQYOqHHz13nvvpWXLlsyZM4c5c+ZQVFRUsW7MmDG0b9+erVu30r9/f+bMmcPPfvYzbr/9dqZNm0bHjh0r7WvmzJk8+OCDvPHGG7g7ffv25Tvf+Q7t2rVj/vz5PProo9x///2ce+65PPnkk5x//vmVtu/Xrx+vv/46ZsaECRO45ZZbGDt2LL/5zW9o06YN77zzDgCrV69mxYoVXHTRRbzyyit069Yt5+P1ZfLaIxHJDdWgMiSxmS+xec/dueaaa+jduzcnnHACn332WUVNJJVXXnmlIlH07t2b3r17V6ybOHEiRUVFHHbYYcydOzflQLCJXnvtNc444wxatWpF69atOfPMM3n11VcB6NatG4ceeihQ9S09SktLOfHEE+nVqxe33norc+fOBeCFF17gsssuqyjXrl07Xn/9dY499li6desG5P6WHDp3JJL/6l0NqrqaTjadfvrpXHXVVbz11lts3LixouZTUlLCihUrmDlzJk2bNqWgoCDlLTYSpapdffzxx9x2221Mnz6ddu3aMWzYsBr3U904i+W36oBwu45UTXyXX345V111FYMGDeKf//wn119/fcV+k2OM4y05dO2RSH5TDSpDWrduzXHHHcfw4cMrdY5Ys2YNe+21F02bNmXatGksXry42v0ce+yxlER9od99913mzJkDhFt1tGrVijZt2rBs2TKee+65im1233131q1bl3JfkydPpqysjA0bNjBp0iSOOeaYtN/TmjVr6NSpEwAPPfRQxfIBAwZw9913V8yvXr2ao446ipdffpmPP/4Y2LVbcmTi+iURyX9KUBk0ZMgQZs+eXXFHW4ChQ4cyY8YMiouLKSkp4eCDD652H5dccgnr16+nd+/e3HLLLfTp0wcId8c97LDD6NGjB8OHD690q44RI0Zw0kknVXSSKFdUVMSwYcPo06cPffv25cILL+Swww5L+/1cf/31nHPOORxzzDGVzm9de+21rF69mp49e1JYWMi0adPYc889GT9+PGeeeSaFhYWcd955ab9OokxdvyQi+S+rt9sws4HAnUBjYIK7/zZpfVfgj8CewBfA+e5eWt0+dbuN/FbT36qgICSlZF27hi7eIpL/cn67DTNrDNwDnAQcAgwxs0OSit0G/NndewM3ADdnKx7JD5m8fklE8ls2m/j6AAvcfaG7fw08BpyWVOYQ4MXo+bQU66WB0fVLIlIumwmqE/BpwnxptCzRbOCs6PkZwO5m1iF5R2Y2wsxmmNmMFStWpHyxfLszcEOUzt9IY9+JSLlsJqhUfY6Tv6GuBr5jZm8D3wE+A7bssJH7eHcvdvfiPffcc4edNm/enFWrVilJxZi7s2rVKpo3b15tOV2/JCLlsnkdVCmwX8J8Z2BJYgF3XwKcCWBmrYGz3H1NbV+oc+fOlJaWUlXtSuKhefPmdO7cucZyun5JRCC7CWo6cKCZdSPUjAYDP0gsYGYdgS/cfRvwa0KPvlpr2rRpxQgGIiJSP2Stic/dtwAjganAe8BEd59rZjeY2aCo2HHAB2b2IbA3oDMNIiICZPk6qGxIdR2UiIjkj5xfByUNj4YoEpFMqneDxUpulA9RVH57i/IhikAdHkRk56gGJRmRqVusi4iUU4KSjNAQRSKSaUpQkhEaokhEMk0JSjJCQxSJSKYpQUlGaIgiEck09eKTjNEQRSKSSapBiYhILClBiYhILClBiYhILClBiYhILClBiYhILClBiYhILClBCaCRyEUkfnQdlGgkchGJJdWgRCORi0gsKUGJRiIXkVhSghKNRC4isaQEJRqJXERiSQlKNBK5iMSSevEJoJHIRSR+VIMSEZFYUoISEZFYUoISEZFYUoISEZFYUoISEZFYUoLKcxrkVUTqK3Uzz2Ma5FVE6jPVoPKYBnkV2W7bNnj/fVi5MteRSKaoBpXHNMhr/bVtG2zYAOvXw7p1lR/Xr4fNmzPzOi1bQuvWsPvuOz42axZGFomrLVvg7bfhlVfC9OqrsHp1WPfNb8KRR26feveGpk1zG6/UnhJUHuvSJTTrpVoudccdvv66cgJJTirprEt83LAh1+8KmjSpOnklPrZrB/vvDwceGKY99shOPJs2wZtvbk9I//53OF4ABx0EZ50F3/42rFgBr78OL74IDz8c1jdvDsXFIVn17RseO3fOTpySOUpQeWzMmMrnoECDvGbDf/4DDzwQmo6qSjRbtqS3r0aNtn+xJ37Jd+6c+su/qmXNmu36+3IPn53aJtCVKyvPb9xYeb977x0S1UEHbU9aBx4YajXJgxJXZ8OGcOzLE9Lrr4ckBdCrF1xwARx7bJi+8Y3U76+0NGxXPv3+93DbbWF9p06Va1mHHw4tWuzcsZTsMHfPdQy1Ulxc7DNmzMh1GLFRUhLOOX3ySag5jRmjDhKZ4A7//CfceCO89BK0bRsG0a2uJpHOsubN491stjPKyuCjj+DDD2H+/DCVP1+2rHLZzp13TF4HHRRqYGVl8Npr2xPSzJkh8TdqBEVF8J3vhGTUrx+0b79zsX79NcyeXTlpLVwY1jVpAoWFlZPWAQfk9u+1ahVMnx4+O0VF9SeBmtlMdy+usVw2E5SZDQTuBBoDE9z9t0nruwAPAW2jMqPc/dnq9qkEFW+LF8Pjj4f2/pq+rFu1Cl8+ceIOzz8fEtO//w377AO/+EWoqbZqlevo8s/atduTVnLy+uKL7eUaNQrH3j3UDvv02V47+va3w+clW5Yvhzfe2J6w3nxze9Nhhw7bmwSPPDLE1aZNduLYvBneeady8pw/f/v6Jk3g0EMrx5PrBLqzcp6gzKwx8CHwPaAUmA4Mcfd5CWXGA2+7+71mdgjwrLsXVLdfJah4WrMGbr4Zxo3b3gyTjlatqq59dO0KgwaFL6jGjbMXO4ROCVOmhMQ0c2aojY4aBT/+caj1SOatWlU5cTVtCsccE76Ac1lT2LoV5s7dnrTeeAPmzQvJ0wy6d69cyzrkkJ37fH72WeVkNHPm9ubSvfeGo47anhTXrdtebvr0ygk0MZYjjshsAl2zJvzoTJw++yy03OxKYoxDgjoKuN7dT4zmfw3g7jcnlLkPWOjuv4vKj3X3b1e3XyWoeNm8Odw76vrrw7mJH/0Ibrgh/JPUtpNA8rKFC0OTzF57hUR1xhnQvz/stlvm4t+6Ff7619A0+u674RfpNdfA+edn5jyP1A9r1oTEkJhQVq0K61q3DkmkPEn07Rs+s4k2boS33qq8fWlpWNesWWi+S0w0XbpUnQC2bg0JM3Ff86Kf/WYhYSbuq3v31AnUPdQekxNQ4rRmTeVtmjcPsU2fvmudYeKQoM4GBrr7hdH8D4G+7j4yocw+wN+BdkAr4AR3n5liXyOAEQBdunQ5fHGqrmtSp9zhmWdC89cHH8Bxx8HYseEfLVPWroXnnoNJk+DZZ0PSat0aTj45JKuTT975f5LNm+GRR+Cmm0KTU/fu4VzeeeeFphSR6riH826JSWL27O2dZbp1C8mhXbvQZDhr1o7ryqfCwl3/0fXllzsm0PIm1PIEethhlWtEn3wCX31VeT977BFaLgoKwmPytNdemWlSjEOCOgc4MSlB9XH3yxPKXBXFMDaqQT0A9HT3bVXtVzWo3Hv7bfiv/4Jp08IJ7ltvhVNPzW5b+KZNobPCpEnw1FPhl1+zZqFGdfrpcNppoVkknf386U/w29/CokWhTf/aa0PCi9v5MMkvZWWhlpR4Pmv16h1rV+l8TneVOyxYsL2JsjyBtm+fOvGUT23bZj82SD9B4e5ZmYCjgKkJ878Gfp1UZi6wX8L8QmCv6vZ7+OGHu+TGp5+6X3CBu5l7hw7uv/+9+9df130cW7a4v/qq+1VXuXfrFk6tm7kffbT7bbe5L1iw4zYbNrjfead7p06hfN++7s88475tW93HL5ILcfqsAzM8jTySzRpUE0Inif7AZ4ROEj9w97kJZZ4DHnf3P5lZd+BFoJNXE5RqUHVv/Xr43e9CE97WrXDFFeE8TbZ6M9WGe+j5NGlSmGbPDst79Qq1ou9/P3QXHzs21LqOPRb++79DzSsfez+J1Ac5b+KLgjgZGEfoQv5Hdx9jZjcQsueUqOfe/UBrwIFfuvvfq9unElTd2boV/vjH8IW+bBkMHhx66hUU5Dqyqi1cCJMnh+m110ICAxgwIJxjOvbY3MYnIjFJUNmgBFU3pk6Fq68OPdu+/W24/fbQfp5Pli+Hv/8dvvWt0P1WROIh3QSl/kpSybvvhsQ0dWq4uv+vfw1jnOVjc9hee4Xu4iKSn9RvSYBw8d2IEaHL6xtvhBrTvHlw9tn5mZxEJP+pBtXArV4dOkDceWc453T55eGcU4cOuY5MRBo6JagGauPGMLLzzTeHi/eGDoX/9/9Cs56ISByoia+B2bIF7r8/3PrgV78KHSDefhv+8hclJxGJlxoTlJmNNLN2dRGMZI87PPEE9OgRzjV17Qovvwx/+1s47yQiEjfp1KC+AUw3s4lmNtBMp8wzoaQkXE/UqFF4LCnJ3mu99FLoIn7OOWGcucmT4V//0jVBIhJvNSYod78WOJAwTt4wYL6Z3WRmB2Q5tnqrpCTUYhYvDjWbxYvDfKaT1FtvwYknhlETPv8cHnwQ5swJ49bpZ4aIxF1a56CioYc+j6YthNHHnzCzW7IYW701enTl27RDmB89OjP7X7AgjPpw+OHhHjO33x5G7B42LPv3VRIRyZQae/GZ2c+AC4CVwATgF+6+2cwaAfOBX2Y3xPrnk09qtzxdS5eGezFNmBBG+r722nDRbRzGzBMRqa10upl3BM5090o3YXL3bWZ2SnbCqt+6dAnNeqmW74wvvwy3vBg3Ltzg76c/DcnpG9/YtThFRHIpnSa+Z4EvymfMbHcz6wvg7u9lK7D6bMwYaNmy8rKWLcPydLnDq6+GZrtOncKN9047Dd5/H+6+W8lJRPJfOgnqXmB9wvyGaJnspKFDw23Su3YNnRW6dg3zQ4fWvO3nn8Mtt8DBB4deeP/3f2G8uVmzwh1iD1DXFRGpJ9Jp4rPE+zNFTXsagWIXDR2aXkKCcHHt88/DAw/A00+HIYn69YNf/zp0HW/VKruxiojkQjqJZmHUUaK81nQp4c63kmUffRTux/SnP8GSJWF07quuguHDQw1KRKQ+SydBXQzcBVxLuKngi8CIbAbVkG3cGJrtHngApk0LF/KedFI4r3TKKdC0aa4jFBGpGzUmKHdfDgyug1gatLffDkmppCT0ytt/f7jxRrjgAujcOdfRiYjUvXSug2oO/AToATQvX+7uw7MYV4OwcWNovpswIYz6sNtu4eaAP/kJHHdcqD2JiDRU6XwF/oUwHt+JwMtAZ2BdNoNqCFauhOOPh0svDZ0efv/7cJ6ppAS++10lJxGRdM5BfdPdzzGz09z9ITN7BJia7cDqs0WLwhh5ixfn9y3VRUSyKZ0EtTl6/NLMehLG4yvIWkT13OzZMHAgfPUVvPBC6C4uIiI7SidBjY/uB3UtMAVoDfx3VqOqp6ZNg9NPhz32gNdeC/dmEhGR1KpNUNGAsGvdfTXwCqB7ru6kiRPhhz8Md7J9/nnYb79cRyQiEm/Vnop3923AyDqKpd66665w+4u+fUPNSclJRKRm6fQV+4eZXW1m+5lZ+/Ip65HVA+4wahT8/OehaW/qVGjXLtdRiYjkh3TOQZVf73RZwjJHzX3V2rwZLrwQ/vxnuOSS0I1cNwsUEUlfOiNJdKuLQOqT9evh7LNDjenGG+Gaa9SNXESkttIZSeJHqZa7+58zH07+W74cvv/9MHTRhAlhVAgREam9dJr4jkh43hzoD7wFKEEl+eijcAHukiUweXKrkvnKAAAQNklEQVQY3FVERHZOOk18lyfOm1kbwvBHkmDmTDj55HDvppdegiOPzHVEIiL5bWdGfCsDDsx0IPnsH/8Ig7u2aAH/+peSk4hIJqRzDuppQq89CAntEGBiNoPKJyUlMGwYHHIIPPcc7LtvriMSEakf0jkHdVvC8y3AYncvzVI8eWXsWLj66lB7mjwZ2rTJdUQiIvVHOgnqE2Cpu38FYGYtzKzA3RdlNbIYcw+J6fbb4dxzw7VOu+2W66hEROqXdM5B/RXYljC/NVpWIzMbaGYfmNkCMxuVYv0dZjYrmj40sy/TCzu3pk4Nyemyy+DRR5WcRESyIZ0aVBN3/7p8xt2/NrNmNW1kZo2Be4DvAaXAdDOb4u7zEvZ1ZUL5y4HDahN8rowbB/vsE5KUbiwoIpId6Xy9rjCzQeUzZnYasDKN7foAC9x9YZTgHgNOq6b8EODRNPabU/PmhRrUZZdBsxrTtIiI7Kx0alAXAyVmdnc0XwqkHF0iSSfg04T5UqBvqoJm1hXoBrxUxfoRwAiALl26pPHS2XPXXdC8OYwYkdMwRETqvXQu1P0IONLMWgPm7uvS3Heq0ec8xTKAwcAT7r61ihjGA+MBiouLq9pH1q1aFTpEnH8+7LlnrqIQEWkYamziM7ObzKytu69393Vm1s7Mbkxj36VA4p2POgNLqig7mDxo3hs/HjZuhAMOgIKCcP6poCBcCyUiIpmVzjmok9y9ondddHfdk9PYbjpwoJl1izpVDCbcMr4SM/sW0A74T3oh58bmzXD33dCzJ/zmN7B4cehuvnhxaO5TkhIRyax0ElRjM6voSG1mLYAaO1a7+xbC3XinAu8BE919rpndkNjpgtA54jF3z1nTXTqeeCIMArt8OZSVVV5XVgajR+cmLhGR+spqygtm9ktgEPBgtOjHwBR3vyXLsaVUXFzsM2bMqNPXdA+3a1+zBj78MHUZM9i2LfU6ERHZzsxmuntxTeVqrEFFiehGoDthHL7nga67HGEe+c9/YPr0cOv2rlW88xx3LhQRqXfS6WYO8DlhNIlzgY+BJ7MWUQyNGwdt28KPfhTG2xsxonIzX8uWMGZM7uITEamPqkxQZnYQoWPDEGAV8DihSfD4OootFhYvhiefDGPvtW4NQ4eG5aNHwyefhJrTmDHbl4uISGZUV4N6H3gVONXdFwCY2ZXVlK+X7rknnF+67LLty4YOVUISEcm26s5BnUVo2ptmZvebWX9SX3xbb61fD/ffD2edpXNMIiJ1rcoE5e6T3P084GDgn8CVwN5mdq+ZDaij+HLqoYfgyy/hiityHYmISMOTTi++De5e4u6nEEaDmAXscOuM+mbbNrjzTujTR7dwFxHJhXR78QHg7l8A90VTvfbcczB/frjfkzWohk0RkXjQ3YyqMG4cdOoUzj+JiEjdU4JK4d134YUXYORIaNo019GIiDRMSlApjBsHLVrARRflOhIRkYZLCSrJihXw8MNh1IgOHXIdjYhIw6UEleS++2DTpjDunoiI5I4SVIKvvw4jRwwcCN275zoaEZGGTQkqwcSJ8PnnujBXRCQOlKAi7nDHHaHmNKBBjJMhIhJvtbpQtz577TV46y34wx90Ya6ISByoBhUZNw7at4cf/jDXkYiICChBAfDxxzB5Mvz0p+HmgyIikntKUMDdd0OjRnDppbmOREREyjX4BLVuHUyYAOecA5075zoaEREp1+AT1IMPwtq16louIhI3DTpBbd0Kd90FRx0V7vskIiLx0aAT1N/+Bh99BFdemetIREQkWYNOUOPGwX77wRln5DoSERFJ1mAT1KxZMG0aXH45NNHlyiIisdNgE9Sdd4Zrni68MNeRiIhIKg0yQS1bBo88AsOGQbt2uY5GRERSaZAJ6g9/CLfW0D2fRETiq8ElqE2b4H//F77/fTjooFxHIyIiVWlwCerRR2H5cl2YKyISdw0qQbmHruU9ekD//rmORkREqtOgOlhv2wYXXwx77617PomIxF2DSlCNG4cEJSIi8ZfVJj4zG2hmH5jZAjMbVUWZc81snpnNNbNHshmPiIjkj6zVoMysMXAP8D2gFJhuZlPcfV5CmQOBXwNHu/tqM9srW/GIiEh+yWYNqg+wwN0XuvvXwGPAaUllLgLucffVAO6+PIvxiIhIHslmguoEfJowXxotS3QQcJCZ/cvMXjezgal2ZGYjzGyGmc1YsWJFlsIVEZE4yWaCStVPzpPmmwAHAscBQ4AJZtZ2h43cx7t7sbsX77nnnhkPVERE4iebCaoU2C9hvjOwJEWZp9x9s7t/DHxASFgiItLAZTNBTQcONLNuZtYMGAxMSSozGTgewMw6Epr8FmYxJhERyRNZS1DuvgUYCUwF3gMmuvtcM7vBzAZFxaYCq8xsHjAN+IW7r8pWTCIikj/MPfm0ULwVFxf7jBkzch2GiIjsJDOb6e7FNZVrUGPxiYhI/lCCEhGRWFKCEhGRWFKCEhGRWFKCEhGRWFKCEhGRWFKCEhGRWFKCEhGRWFKCEhGRWFKCEhGRWFKCEhGRWFKCEhGRWFKCEhGRWFKCEhGRWFKCEhGRWFKCEhGRWFKCEhGRWFKCEhGRWFKCEhGRWFKCEhGRWFKCEhGRWFKCEhGRWFKCEhGRWFKCEhGRWFKCEhGRWFKCEhGRWFKCEhGRWFKCEhGRWFKCEhGRWFKCEhGRWFKCEhGRWFKCEhGRWFKCEhGRWFKCEhGRWMpqgjKzgWb2gZktMLNRKdYPM7MVZjYrmi7MZjwiIpI/mmRrx2bWGLgH+B5QCkw3synuPi+p6OPuPjJbcYiISH7KZg2qD7DA3Re6+9fAY8BpWXy9GpWUQEEBNGoUHktKchmNiIhUJ5sJqhPwacJ8abQs2VlmNsfMnjCz/VLtyMxGmNkMM5uxYsWKnQqmpARGjIDFi8E9PI4YoSQlIhJX2UxQlmKZJ80/DRS4e2/gBeChVDty9/HuXuzuxXvuuedOBTN6NJSVVV5WVhaWi4hI/GQzQZUCiTWizsCSxALuvsrdN0Wz9wOHZyuYTz6p3XIREcmtbCao6cCBZtbNzJoBg4EpiQXMbJ+E2UHAe9kKpkuX2i0XEZHcylqCcvctwEhgKiHxTHT3uWZ2g5kNior9zMzmmtls4GfAsGzFM2YMtGxZeVnLlmG5iIjEj7knnxaKt+LiYp8xY8ZObVtSEs45ffJJqDmNGQNDh2Y4QBERqZaZzXT34prKZe06qDgaOlQJSUQkX2ioIxERiSUlKBERiSUlKBERiSUlKBERiSUlKBERiaW862ZuZiuAxbmOI00dgZW5DqKW8i1mxZtd+RYv5F/MDTHeru5e47h1eZeg8omZzUinr3+c5FvMije78i1eyL+YFW/V1MQnIiKxpAQlIiKxpASVXeNzHcBOyLeYFW925Vu8kH8xK94q6ByUiIjEkmpQIiISS0pQIiISS0pQu8jM9jOzaWb2XnRvq5+nKHOcma0xs1nR9D+5iDUppkVm9k4Uzw73L7HgLjNbYGZzzKwoF3FGsXwr4djNMrO1ZnZFUpmcHmMz+6OZLTezdxOWtTezf5jZ/OixXRXbXhCVmW9mF+Qw3lvN7P3o7z3JzNpWsW21n506jvl6M/ss4e9+chXbDjSzD6LP86gcxvt4QqyLzGxWFdvW+TGu6rssp59jd9e0CxOwD1AUPd8d+BA4JKnMccAzuY41KaZFQMdq1p8MPAcYcCTwRq5jjuJqDHxOuNAvNscYOBYoAt5NWHYLMCp6Pgr4XYrt2gMLo8d20fN2OYp3ANAkev67VPGm89mp45ivB65O4zPzEbA/0AyYnfw/WlfxJq0fC/xPXI5xVd9lufwcqwa1i9x9qbu/FT1fR7h7cKfcRpURpwF/9uB1oK2Z7ZProID+wEfuHqvRRNz9FeCLpMWnAQ9Fzx8CTk+x6YnAP9z9C3dfDfwDGJi1QCOp4nX3v3u4EzbA60DnbMdRG1Uc43T0ARa4+0J3/xp4jPC3yarq4jUzA84FHs12HOmq5rssZ59jJagMMrMC4DDgjRSrjzKz2Wb2nJn1qNPAUnPg72Y208xGpFjfCfg0Yb6UeCTewVT9Tx23Y7y3uy+F8M8P7JWiTFyP83BCDTqVmj47dW1k1Cz5xyqan+J4jI8Blrn7/CrW5/QYJ32X5exzrASVIWbWGngSuMLd1yatfovQJFUI/B6YXNfxpXC0uxcBJwGXmdmxSestxTY5vSbBzJoBg4C/plgdx2Ocjjge59HAFqCkiiI1fXbq0r3AAcChwFJCs1my2B1jYAjV155ydoxr+C6rcrMUy3b5GCtBZYCZNSX8QUvc/f+S17v7WndfHz1/FmhqZh3rOMzkmJZEj8uBSYRmkESlwH4J852BJXUTXZVOAt5y92XJK+J4jIFl5c2i0ePyFGVidZyjk9unAEM9OrmQLI3PTp1x92XuvtXdtwH3VxFL3I5xE+BM4PGqyuTqGFfxXZazz7ES1C6K2pIfAN5z99urKPONqBxm1odw3FfVXZQ7xNPKzHYvf044Of5uUrEpwI+i3nxHAmvKq/k5VOWvzrgd48gUoLw30wXAUynKTAUGmFm7qHlqQLSszpnZQOBXwCB3L6uiTDqfnTqTdF70jCpimQ4caGbdolr4YMLfJldOAN5399JUK3N1jKv5Lsvd57gue4nUxwnoR6jKzgFmRdPJwMXAxVGZkcBcQu+h14Fv5zjm/aNYZkdxjY6WJ8ZswD2E3k/vAMU5jrklIeG0SVgWm2NMSJxLgc2EX5M/AToALwLzo8f2UdliYELCtsOBBdH04xzGu4BwHqH8c/yHqOy+wLPVfXZyGPNfos/nHMIX6T7JMUfzJxN6pX1UVzGnijda/qfyz21C2Zwf42q+y3L2OdZQRyIiEktq4hMRkVhSghIRkVhSghIRkVhSghIRkVhSghIRkVhSghLJIDPbapVHXs/YyNlmVpA4MrZIfdck1wGI1DMb3f3QXAchUh+oBiVSB6L7+/zOzN6Mpm9Gy7ua2YvRYKcvmlmXaPneFu7JNDuavh3tqrGZ3R/dr+fvZtYiKv8zM5sX7eexHL1NkYxSghLJrBZJTXznJaxb6+59gLuBcdGyuwm3NelNGJz1rmj5XcDLHga/LSKMKABwIHCPu/cAvgTOipaPAg6L9nNxtt6cSF3SSBIiGWRm6929dYrli4DvuvvCaEDOz929g5mtJAzPszlavtTdO5rZCqCzu29K2EcB4Z47B0bzvwKauvuNZvY8sJ4wivtkjwbOFclnqkGJ1B2v4nlVZVLZlPB8K9vPI3+fMHbi4cDMaMRskbymBCVSd85LePxP9PzfhNG1AYYCr0XPXwQuATCzxma2R1U7NbNGwH7uPg34JdAW2KEWJ5Jv9CtLJLNamNmshPnn3b28q/luZvYG4YfhkGjZz4A/mtkvgBXAj6PlPwfGm9lPCDWlSwgjY6fSGHjYzNoQRqG/w92/zNg7EskRnYMSqQPROahid1+Z61hE8oWa+EREJJZUgxIRkVhSDUpERGJJCUpERGJJCUpERGJJCUpERGJJCUpERGLp/wONU4k37yUlUAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.clf()   # 그래프를 초기화합니다\n",
    "\n",
    "acc = history.history['acc']\n",
    "val_acc = history.history['val_acc']\n",
    "\n",
    "plt.plot(epochs, acc, 'bo', label='Training acc')\n",
    "plt.plot(epochs, val_acc, 'b', label='Validation acc')\n",
    "plt.title('Training and validation accuracy')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이 모델은 9번째 에포크 이후에 과대적합이 시작됩니다. 9번의 에포크로 새로운 모델을 훈련하고 테스트 세트에서 평가하겠습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 7982 samples, validate on 1000 samples\n",
      "Epoch 1/9\n",
      "7982/7982 [==============================] - 1s 86us/step - loss: 2.5398 - acc: 0.5226 - val_loss: 1.6733 - val_acc: 0.6570\n",
      "Epoch 2/9\n",
      "7982/7982 [==============================] - 1s 65us/step - loss: 1.3712 - acc: 0.7121 - val_loss: 1.2758 - val_acc: 0.7210\n",
      "Epoch 3/9\n",
      "7982/7982 [==============================] - 1s 65us/step - loss: 1.0136 - acc: 0.7781 - val_loss: 1.1303 - val_acc: 0.7530\n",
      "Epoch 4/9\n",
      "7982/7982 [==============================] - 1s 65us/step - loss: 0.7976 - acc: 0.8251 - val_loss: 1.0539 - val_acc: 0.7590\n",
      "Epoch 5/9\n",
      "7982/7982 [==============================] - 1s 65us/step - loss: 0.6393 - acc: 0.8624 - val_loss: 0.9754 - val_acc: 0.7920\n",
      "Epoch 6/9\n",
      "7982/7982 [==============================] - 1s 65us/step - loss: 0.5124 - acc: 0.8921 - val_loss: 0.9102 - val_acc: 0.8140\n",
      "Epoch 7/9\n",
      "7982/7982 [==============================] - 1s 65us/step - loss: 0.4124 - acc: 0.9137 - val_loss: 0.8932 - val_acc: 0.8210\n",
      "Epoch 8/9\n",
      "7982/7982 [==============================] - 1s 65us/step - loss: 0.3355 - acc: 0.9290 - val_loss: 0.8732 - val_acc: 0.8260\n",
      "Epoch 9/9\n",
      "7982/7982 [==============================] - 1s 65us/step - loss: 0.2782 - acc: 0.9371 - val_loss: 0.9338 - val_acc: 0.8000\n",
      "2246/2246 [==============================] - 0s 50us/step\n"
     ]
    }
   ],
   "source": [
    "model = models.Sequential()\n",
    "model.add(layers.Dense(64, activation='relu', input_shape=(10000,)))\n",
    "model.add(layers.Dense(64, activation='relu'))\n",
    "model.add(layers.Dense(46, activation='softmax'))\n",
    "\n",
    "model.compile(optimizer='rmsprop',\n",
    "              loss='categorical_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "model.fit(partial_x_train,\n",
    "          partial_y_train,\n",
    "          epochs=9,\n",
    "          batch_size=512,\n",
    "          validation_data=(x_val, y_val))\n",
    "results = model.evaluate(x_test, one_hot_test_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1.022498257544459, 0.7756010686194165]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "대략 78%의 정확도를 달성했습니다. 균형 잡힌 이진 분류 문제에서 완전히 무작위로 분류하면 50%의 정확도를 달성합니다. 이 문제는 불균형한 데이터셋을 사용하므로 무작위로 분류하면 19% 정도를 달성합니다. 여기에 비하면 이 결과는 꽤 좋은 편입니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.182546749777382"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import copy\n",
    "\n",
    "test_labels_copy = copy.copy(test_labels)\n",
    "np.random.shuffle(test_labels_copy)\n",
    "float(np.sum(np.array(test_labels) == np.array(test_labels_copy))) / len(test_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 새로운 데이터에 대해 예측하기\n",
    "\n",
    "모델 인스턴스의 `predict` 메서드는 46개 토픽에 대한 확률 분포를 반환합니다. 테스트 데이터 전체에 대한 토픽을 예측해 보겠습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = model.predict(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`predictions`의 각 항목은 길이가 46인 벡터입니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(46,)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions[0].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이 벡터의 원소 합은 1입니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sum(predictions[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "가장 큰 값이 예측 클래스가 됩니다. 즉, 가장 확률이 높은 클래스입니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.argmax(predictions[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 레이블과 손실을 다루는 다른 방법\n",
    "\n",
    "앞서 언급한 것처럼 레이블을 인코딩하는 다른 방법은 다음과 같이 정수 텐서로 변환하는 것입니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train = np.array(train_labels)\n",
    "y_test = np.array(test_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이 방식을 사용하려면 손실 함수 하나만 바꾸면 됩니다. 코드 3-21에 사용된 손실 함수 `categorical_crossentropy`는 레이블이 범주형 인코딩되어 있을 것이라고 기대합니다. 정수 레이블을 사용할 때는 `sparse_categorical_crossentropy`를 사용해야 합니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer='rmsprop', loss='sparse_categorical_crossentropy', metrics=['acc'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이 손실 함수는 인터페이스만 다를 뿐이고 수학적으로는 `categorical_crossentropy`와 동일합니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 충분히 큰 중간층을 두어야 하는 이유\n",
    "\n",
    "앞서 언급한 것처럼 마지막 출력이 46차원이기 때문에 중간층의 히든 유닛이 46개보다 많이 적어서는 안 됩니다. 46차원보다 훨씬 작은 중간층(예를 들면 4차원)을 두면 정보의 병목이 어떻게 나타나는지 확인해 보겠습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 7982 samples, validate on 1000 samples\n",
      "Epoch 1/20\n",
      "7982/7982 [==============================] - 1s 79us/step - loss: 2.6576 - acc: 0.3773 - val_loss: 1.9684 - val_acc: 0.5280\n",
      "Epoch 2/20\n",
      "7982/7982 [==============================] - 0s 54us/step - loss: 1.6652 - acc: 0.6211 - val_loss: 1.5409 - val_acc: 0.6230\n",
      "Epoch 3/20\n",
      "7982/7982 [==============================] - 0s 54us/step - loss: 1.3324 - acc: 0.6706 - val_loss: 1.3951 - val_acc: 0.6790\n",
      "Epoch 4/20\n",
      "7982/7982 [==============================] - 0s 54us/step - loss: 1.1420 - acc: 0.7269 - val_loss: 1.3235 - val_acc: 0.6970\n",
      "Epoch 5/20\n",
      "7982/7982 [==============================] - 0s 54us/step - loss: 1.0109 - acc: 0.7457 - val_loss: 1.2704 - val_acc: 0.7010\n",
      "Epoch 6/20\n",
      "7982/7982 [==============================] - 0s 54us/step - loss: 0.9098 - acc: 0.7580 - val_loss: 1.2778 - val_acc: 0.7070\n",
      "Epoch 7/20\n",
      "7982/7982 [==============================] - 0s 54us/step - loss: 0.8304 - acc: 0.7742 - val_loss: 1.2595 - val_acc: 0.7080\n",
      "Epoch 8/20\n",
      "7982/7982 [==============================] - 0s 54us/step - loss: 0.7692 - acc: 0.7895 - val_loss: 1.2664 - val_acc: 0.7150\n",
      "Epoch 9/20\n",
      "7982/7982 [==============================] - 0s 54us/step - loss: 0.7130 - acc: 0.8002 - val_loss: 1.2920 - val_acc: 0.7110\n",
      "Epoch 10/20\n",
      "7982/7982 [==============================] - 0s 54us/step - loss: 0.6626 - acc: 0.8109 - val_loss: 1.3244 - val_acc: 0.7020\n",
      "Epoch 11/20\n",
      "7982/7982 [==============================] - 0s 54us/step - loss: 0.6172 - acc: 0.8190 - val_loss: 1.3556 - val_acc: 0.7110\n",
      "Epoch 12/20\n",
      "7982/7982 [==============================] - 0s 54us/step - loss: 0.5787 - acc: 0.8326 - val_loss: 1.3915 - val_acc: 0.7110\n",
      "Epoch 13/20\n",
      "7982/7982 [==============================] - 0s 54us/step - loss: 0.5408 - acc: 0.8444 - val_loss: 1.4515 - val_acc: 0.7080\n",
      "Epoch 14/20\n",
      "7982/7982 [==============================] - 0s 54us/step - loss: 0.5086 - acc: 0.8552 - val_loss: 1.4671 - val_acc: 0.7130\n",
      "Epoch 15/20\n",
      "7982/7982 [==============================] - 0s 54us/step - loss: 0.4773 - acc: 0.8667 - val_loss: 1.5340 - val_acc: 0.7110\n",
      "Epoch 16/20\n",
      "7982/7982 [==============================] - 0s 54us/step - loss: 0.4495 - acc: 0.8801 - val_loss: 1.5469 - val_acc: 0.7050\n",
      "Epoch 17/20\n",
      "7982/7982 [==============================] - 0s 54us/step - loss: 0.4245 - acc: 0.8822 - val_loss: 1.5952 - val_acc: 0.7140\n",
      "Epoch 18/20\n",
      "7982/7982 [==============================] - 0s 54us/step - loss: 0.4036 - acc: 0.8908 - val_loss: 1.6350 - val_acc: 0.7100\n",
      "Epoch 19/20\n",
      "7982/7982 [==============================] - 0s 54us/step - loss: 0.3843 - acc: 0.8943 - val_loss: 1.6489 - val_acc: 0.7090\n",
      "Epoch 20/20\n",
      "7982/7982 [==============================] - 0s 54us/step - loss: 0.3656 - acc: 0.8974 - val_loss: 1.7272 - val_acc: 0.7080\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fb0227154e0>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = models.Sequential()\n",
    "model.add(layers.Dense(64, activation='relu', input_shape=(10000,)))\n",
    "model.add(layers.Dense(4, activation='relu'))\n",
    "model.add(layers.Dense(46, activation='softmax'))\n",
    "\n",
    "model.compile(optimizer='rmsprop',\n",
    "              loss='categorical_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "model.fit(partial_x_train,\n",
    "          partial_y_train,\n",
    "          epochs=20,\n",
    "          batch_size=128,\n",
    "          validation_data=(x_val, y_val))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "검증 정확도의 최고 값은 약 71%로 8% 정도 감소되었습니다. 이런 손실의 대부분 원인은 많은 정보(46개 클래스의 분할 초평면을 복원하기에 충분한 정보)를 중간층의 저차원 표현 공간으로 압축하려고 했기 때문입니다. 이 네트워크는 필요한 정보 대부분을 4차원 표현 안에 구겨 넣었지만 전부는 넣지 못했습니다."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 추가 실험\n",
    "\n",
    "* 더 크거나 작은 층을 사용해 보세요: 32개 유닛, 128개 유닛 등\n",
    "* 여기에서 두 개의 은닉층을 사용했습니다. 한 개의 은닉층이나 세 개의 은닉층을 사용해 보세요."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 정리\n",
    "\n",
    "다음은 이 예제에서 배운 것들입니다.\n",
    "\n",
    "* N개의 클래스로 데이터 포인트를 분류하려면 네트워크의 마지막 `Dense` 층의 크기는 N이어야 합니다.\n",
    "* 단일 레이블, 다중 분류 문제에서는 N개의 클래스에 대한 확률 분포를 출력하기 위해 `softmax` 활성화 함수를 사용해야 합니다.\n",
    "* 이런 문제에는 항상 범주형 크로스엔트로피를 사용해야 합니다. 이 함수는 모델이 출력한 확률 분포와 타깃 분포 사이의 거리를 최소화합니다.\n",
    "* 다중 분류에서 레이블을 다루는 두 가지 방법이 있습니다.\n",
    "    * 레이블을 범주형 인코딩(또는 원-핫 인코딩)으로 인코딩하고 `categorical_crossentropy` 손실 함수를 사용합니다.\n",
    "    * 레이블을 정수로 인코딩하고 `sparse_categorical_crossentropy` 손실 함수를 사용합니다.\n",
    "* 많은 수의 범주를 분류할 때 중간층의 크기가 너무 작아 네트워크에 정보의 병목이 생기지 않도록 해야 합니다."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}