t81_558_class_11_05_english_scratch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_11_05_english_scratch.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 11: Natural Language Processing and Speech Recognition**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), School of Engineering and Applied Science, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 11 Material\n",
    "\n",
    "* Part 11.1: Getting Started with Spacy in Python [[Video]](https://www.youtube.com/watch?v=A5BtU9vXzu8&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_11_01_spacy.ipynb)\n",
    "* Part 11.2: Word2Vec and Text Classification [[Video]](https://www.youtube.com/watch?v=nWxtRlpObIs&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_11_02_word2vec.ipynb)\n",
    "* Part 11.3: What are Embedding Layers in Keras [[Video]](https://www.youtube.com/watch?v=OuNH5kT-aD0&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_11_03_embedding.ipynb)\n",
    "* Part 11.4: Natural Language Processing with Spacy and Keras [[Video]](https://www.youtube.com/watch?v=BKgwjhao5DU&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_11_04_text_nlp.ipynb)\n",
    "* **Part 11.5: Learning English from Scratch with Keras and TensorFlow** [[Video]](https://www.youtube.com/watch?v=Y1khuuSjZzc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN&index=58) [[Notebook]](t81_558_class_11_05_english_scratch.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: not using Google CoLab\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    %tensorflow_version 2.x\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 11.5: Learning English from Scratch with Keras and TensorFlow\n",
    "\n",
    "In this section we will see how a neural network can learn the English language from scratch.  We will make use of a type of neural network called end-to-end memory. [[Cite:sukhbaatar2015end]](https://arxiv.org/abs/1503.08895) We will train this type of neural network on a special dataset that was created by researchers at Facebook to test a neural network's ability to answer questions. [[Cite:sukhbaatar2015end]](https://arxiv.org/pdf/1502.05698.pdf) \n",
    "\n",
    "Other useful links for End-To-End Memory Networks\n",
    "\n",
    "* [bAbI Datasets](https://research.fb.com/downloads/babi/) - The Facebook dataset used to train this network.\n",
    "* [Keras End-To-End Memory Networks](https://github.com/fchollet/keras/blob/master/examples/babi_memnn.py) - Example from Keras author on end-to-end networks.\n",
    "* [Online JavaScript Demo of End-to-End Memory Networks](http://yerevann.com/dmn-ui/#/)\n",
    "\n",
    "## Imports and Utility Functions\n",
    "\n",
    "The following imports are needed to create the end-to-end memory network. Neither Keras nor TensorFlow directly supports End-to-End Memory Networks (yet), so it is necessary to develop them using existing tools.  Several functions are needed to be defined here to read the bAbI dataset that we are using to train.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.models import Sequential, Model\n",
    "from tensorflow.keras.layers import Embedding\n",
    "from tensorflow.keras.layers import Input, Activation, Dense, Permute\n",
    "from tensorflow.keras.layers import Dropout, add, dot, concatenate\n",
    "from tensorflow.keras.layers import LSTM\n",
    "from tensorflow.keras.utils import get_file\n",
    "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
    "from tensorflow.keras.models import load_model\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn import metrics\n",
    "from functools import reduce\n",
    "import pickle\n",
    "import tarfile\n",
    "import numpy as np\n",
    "import re\n",
    "import os\n",
    "import time\n",
    "\n",
    "# Nicely formatted time string\n",
    "def hms_string(sec_elapsed):\n",
    "    h = int(sec_elapsed / (60 * 60))\n",
    "    m = int((sec_elapsed % (60 * 60)) / 60)\n",
    "    s = sec_elapsed % 60\n",
    "    return f\"{h}:{m:>02}:{s:>05.2f}\"\n",
    "\n",
    "\n",
    "def tokenize(sent):\n",
    "    '''Return the tokens of a sentence including punctuation.\n",
    "    >>> tokenize('Bob dropped the apple. Where is the apple?')\n",
    "    ['Bob', 'dropped', 'the', 'apple', '.', 'Where', 'is', 'the', 'apple', '?']\n",
    "    '''\n",
    "    return [x.strip() for x in re.split('(\\W+)', sent) if x.strip()]\n",
    "\n",
    "\n",
    "def parse_stories(lines, only_supporting=False):\n",
    "    '''Parse stories provided in the bAbi tasks format\n",
    "    If only_supporting is true, only the sentences\n",
    "    that support the answer are kept.\n",
    "    '''\n",
    "    data = []\n",
    "    story = []\n",
    "    for line in lines:\n",
    "        line = line.decode('utf-8').strip()\n",
    "        nid, line = line.split(' ', 1)\n",
    "        nid = int(nid)\n",
    "        if nid == 1:\n",
    "            story = []\n",
    "        if '\\t' in line:\n",
    "            q, a, supporting = line.split('\\t')\n",
    "            q = tokenize(q)\n",
    "            substory = None\n",
    "            if only_supporting:\n",
    "                # Only select the related substory\n",
    "                supporting = map(int, supporting.split())\n",
    "                substory = [story[i - 1] for i in supporting]\n",
    "            else:\n",
    "                # Provide all the substories\n",
    "                substory = [x for x in story if x]\n",
    "            data.append((substory, q, a))\n",
    "            story.append('')\n",
    "        else:\n",
    "            sent = tokenize(line)\n",
    "            story.append(sent)\n",
    "    return data\n",
    "\n",
    "\n",
    "def get_stories(f, only_supporting=False, max_length=None):\n",
    "    '''Given a file name, read the file,\n",
    "    retrieve the stories,\n",
    "    and then convert the sentences into a single story.\n",
    "    If max_length is supplied,\n",
    "    any stories longer than max_length tokens will be discarded.\n",
    "    '''\n",
    "    data = parse_stories(f.readlines(), only_supporting=only_supporting)\n",
    "    flatten = lambda data: reduce(lambda x, y: x + y, data)\n",
    "    data = [(flatten(story), q, answer) for story, q, answer in data \\\n",
    "            if not max_length or len(flatten(story)) < max_length]\n",
    "    return data\n",
    "\n",
    "\n",
    "def vectorize_stories(data):\n",
    "    inputs, queries, answers = [], [], []\n",
    "    for story, query, answer in data:\n",
    "        inputs.append([word_idx[w] for w in story])\n",
    "        queries.append([word_idx[w] for w in query])\n",
    "        answers.append(word_idx[answer])\n",
    "    return (pad_sequences(inputs, maxlen=story_maxlen),\n",
    "            pad_sequences(queries, maxlen=query_maxlen),\n",
    "            np.array(answers))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting the Data\n",
    "\n",
    "The data is first downloaded from the Internet if needed. This dataset contains stories and questions about those stories.  The computer is not learning these specific stories.  However, it is learning how to read a story and answer a question about that story.  Consider the first story, \"Mary moved to the bathroom. John went to the hallway.\" the computer is not learning that Mary is in the bathroom or John is in the hallway, this changes per story. Instead, the machine is learning to parse the story and extract information about individual people and their locations.\n",
    "\n",
    "The computer is learning to read, at least in a limited sense."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting stories for the challenge: single_supporting_fact_10k\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    path = get_file('babi-tasks-v1-2.tar.gz', \n",
    "        origin='https://s3.amazonaws.com/text-datasets/\"\\\n",
    "            \"babi_tasks_1-20_v1-2.tar.gz')\n",
    "except:\n",
    "    print('Error downloading dataset, please download it manually:\\n'\n",
    "          '$ wget http://www.thespermwhale.com/jaseweston/babi/tasks_1-20_v1-2.tar.gz\\n'\n",
    "          '$ mv tasks_1-20_v1-2.tar.gz ~/.keras/datasets/babi-tasks-v1-2.tar.gz')\n",
    "    raise\n",
    "tar = tarfile.open(path)\n",
    "\n",
    "challenges = {\n",
    "    # QA1 with 10,000 samples\n",
    "    'single_supporting_fact_10k': \n",
    "        'tasks_1-20_v1-2/en-10k/qa1_single-supporting-fact_{}.txt',\n",
    "    # QA2 with 10,000 samples\n",
    "    'two_supporting_facts_10k': \n",
    "        'tasks_1-20_v1-2/en-10k/qa2_two-supporting-facts_{}.txt',\n",
    "}\n",
    "challenge_type = 'single_supporting_fact_10k'\n",
    "challenge = challenges[challenge_type]\n",
    "\n",
    "print('Extracting stories for the challenge:', challenge_type)\n",
    "train_stories = get_stories(tar.extractfile(challenge.format('train')))\n",
    "test_stories = get_stories(tar.extractfile(challenge.format('test')))\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Story: Mary moved to the bathroom . John went to the hallway .\n",
      "Query: Where is Mary ?\n",
      "Answer: bathroom\n",
      "---\n",
      "Story: Mary moved to the bathroom . John went to the hallway . Daniel went back to the hallway . Sandra moved to the garden .\n",
      "Query: Where is Daniel ?\n",
      "Answer: hallway\n",
      "---\n",
      "Story: Mary moved to the bathroom . John went to the hallway . Daniel went back to the hallway . Sandra moved to the garden . John moved to the office . Sandra journeyed to the bathroom .\n",
      "Query: Where is Daniel ?\n",
      "Answer: hallway\n",
      "---\n",
      "Story: Mary moved to the bathroom . John went to the hallway . Daniel went back to the hallway . Sandra moved to the garden . John moved to the office . Sandra journeyed to the bathroom . Mary moved to the hallway . Daniel travelled to the office .\n",
      "Query: Where is Daniel ?\n",
      "Answer: office\n",
      "---\n",
      "Story: Mary moved to the bathroom . John went to the hallway . Daniel went back to the hallway . Sandra moved to the garden . John moved to the office . Sandra journeyed to the bathroom . Mary moved to the hallway . Daniel travelled to the office . John went back to the garden . John moved to the bedroom .\n",
      "Query: Where is Sandra ?\n",
      "Answer: bathroom\n",
      "---\n"
     ]
    }
   ],
   "source": [
    "# See what the data looks like\n",
    "\n",
    "for i in range(5):\n",
    "    print(\"Story: {}\".format(' '.join(train_stories[i][0])))\n",
    "    print(\"Query: {}\".format(' '.join(train_stories[i][1])))\n",
    "    print(\"Answer: {}\".format(train_stories[i][2]))\n",
    "    print(\"---\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building the Vocabulary\n",
    "\n",
    "This type of neural network can only deal with a set vocabulary.  The words are indexed, and each becomes a number.  Words not in the training vocabulary will not be recognized."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-\n",
      "Vocab size: 22 unique words\n",
      "Story max length: 68 words\n",
      "Query max length: 4 words\n",
      "Number of training stories: 10000\n",
      "Number of test stories: 1000\n",
      "-\n",
      "Here's what a \"story\" tuple looks like (input, query, answer):\n",
      "(['Mary', 'moved', 'to', 'the', 'bathroom', '.', 'John', 'went', 'to', 'the', 'hallway', '.'], ['Where', 'is', 'Mary', '?'], 'bathroom')\n",
      "-\n",
      "(0, '.')\n",
      "(1, '?')\n",
      "(2, 'Daniel')\n",
      "(3, 'John')\n",
      "(4, 'Mary')\n",
      "(5, 'Sandra')\n",
      "(6, 'Where')\n",
      "(7, 'back')\n",
      "(8, 'bathroom')\n",
      "(9, 'bedroom')\n",
      "(10, 'garden')\n",
      "(11, 'hallway')\n",
      "(12, 'is')\n",
      "(13, 'journeyed')\n",
      "(14, 'kitchen')\n",
      "(15, 'moved')\n",
      "(16, 'office')\n",
      "(17, 'the')\n",
      "(18, 'to')\n",
      "(19, 'travelled')\n",
      "(20, 'went')\n"
     ]
    }
   ],
   "source": [
    "vocab = set()\n",
    "for story, q, answer in train_stories + test_stories:\n",
    "    vocab |= set(story + q + [answer])\n",
    "vocab = sorted(vocab)\n",
    "\n",
    "# Reserve 0 for masking via pad_sequences\n",
    "vocab_size = len(vocab) + 1\n",
    "story_maxlen = max(map(len, (x for x, _, _ in train_stories + test_stories)))\n",
    "query_maxlen = max(map(len, (x for _, x, _ in train_stories + test_stories)))\n",
    "\n",
    "print('-')\n",
    "print('Vocab size:', vocab_size, 'unique words')\n",
    "print('Story max length:', story_maxlen, 'words')\n",
    "print('Query max length:', query_maxlen, 'words')\n",
    "print('Number of training stories:', len(train_stories))\n",
    "print('Number of test stories:', len(test_stories))\n",
    "print('-')\n",
    "print('Here\\'s what a \"story\" tuple looks like (input, query, answer):')\n",
    "print(train_stories[0])\n",
    "print('-')\n",
    "\n",
    "\n",
    "for s in list(enumerate(vocab)):\n",
    "    print(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building the Training and Test Data\n",
    "\n",
    "We present the training data to the neural network as a vectorized representation of the sentences.  We replace each word with that word's corresponding vocabulary index.  Additionally, there are two parts to the input (x) data: story and query.  The answer (x) is always a single vocab word number.  We set up this neural network for classification.  Any of the vocab words could potentially be the answer.  Stories can be at most 68 words and questions at most 4.  The program automatically determines both of these limits from the training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vectorizing the word sequences...\n",
      "-\n",
      "inputs: integer tensor of shape (samples, max_length)\n",
      "inputs_train shape: (10000, 68)\n",
      "inputs_test shape: (1000, 68)\n",
      "-\n",
      "queries: integer tensor of shape (samples, max_length)\n",
      "queries_train shape: (10000, 4)\n",
      "queries_test shape: (1000, 4)\n",
      "-\n",
      "answers: binary (1 or 0) tensor of shape (samples, vocab_size)\n",
      "answers_train shape: (10000,)\n",
      "answers_test shape: (1000,)\n",
      "-\n"
     ]
    }
   ],
   "source": [
    "print('Vectorizing the word sequences...')\n",
    "word_idx = dict((c, i + 1) for i, c in enumerate(vocab))\n",
    "inputs_train, queries_train, answers_train \\\n",
    "    = vectorize_stories(train_stories)\n",
    "inputs_test, queries_test, answers_test \\\n",
    "    = vectorize_stories(test_stories)\n",
    "\n",
    "print('-')\n",
    "print('inputs: integer tensor of shape (samples, max_length)')\n",
    "print('inputs_train shape:', inputs_train.shape)\n",
    "print('inputs_test shape:', inputs_test.shape)\n",
    "print('-')\n",
    "print('queries: integer tensor of shape (samples, max_length)')\n",
    "print('queries_train shape:', queries_train.shape)\n",
    "print('queries_test shape:', queries_test.shape)\n",
    "print('-')\n",
    "print('answers: binary (1 or 0) tensor of shape (samples, vocab_size)')\n",
    "print('answers_train shape:', answers_train.shape)\n",
    "print('answers_test shape:', answers_test.shape)\n",
    "print('-')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Story (x): [ 0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0\n",
      "  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0\n",
      "  0  0  0  0  0  0  0  0  5 16 19 18  9  1  4 21 19 18 12  1]\n",
      "Question (x): [ 7 13  5  2]\n",
      "Answer: 9\n"
     ]
    }
   ],
   "source": [
    "# See individual training element.\n",
    "\n",
    "print(\"Story (x): {}\".format(inputs_train[0]))\n",
    "print(\"Question (x): {}\".format(queries_train[0]))\n",
    "print(\"Answer: {}\".format(answers_train[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compile the Neural Network\n",
    "\n",
    "We define the structure of the neural network according to the end-to-end structure defined in the paper discussed previously. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Compiling...\n",
      "Done.\n"
     ]
    }
   ],
   "source": [
    "print('Compiling...')\n",
    "\n",
    "# placeholders\n",
    "input_sequence = Input((story_maxlen,))\n",
    "question = Input((query_maxlen,))\n",
    "\n",
    "# encoders\n",
    "# embed the input sequence into a sequence of vectors\n",
    "input_encoder_m = Sequential()\n",
    "input_encoder_m.add(Embedding(input_dim=vocab_size,\n",
    "                              output_dim=64))\n",
    "input_encoder_m.add(Dropout(0.3))\n",
    "# output: (samples, story_maxlen, embedding_dim)\n",
    "\n",
    "# embed the input into a sequence of vectors of size query_maxlen\n",
    "input_encoder_c = Sequential()\n",
    "input_encoder_c.add(Embedding(input_dim=vocab_size,\n",
    "                              output_dim=query_maxlen))\n",
    "input_encoder_c.add(Dropout(0.3))\n",
    "# output: (samples, story_maxlen, query_maxlen)\n",
    "\n",
    "# embed the question into a sequence of vectors\n",
    "question_encoder = Sequential()\n",
    "question_encoder.add(Embedding(input_dim=vocab_size,\n",
    "                               output_dim=64,\n",
    "                               input_length=query_maxlen))\n",
    "question_encoder.add(Dropout(0.3))\n",
    "# output: (samples, query_maxlen, embedding_dim)\n",
    "\n",
    "# encode input sequence and questions (which are indices)\n",
    "# to sequences of dense vectors\n",
    "input_encoded_m = input_encoder_m(input_sequence)\n",
    "input_encoded_c = input_encoder_c(input_sequence)\n",
    "question_encoded = question_encoder(question)\n",
    "\n",
    "# compute a 'match' between the first input vector sequence\n",
    "# and the question vector sequence\n",
    "# shape: `(samples, story_maxlen, query_maxlen)`\n",
    "match = dot([input_encoded_m, question_encoded], axes=(2, 2))\n",
    "match = Activation('softmax')(match)\n",
    "\n",
    "# add the match matrix with the second input vector sequence\n",
    "response = add([match, input_encoded_c])  # (samples, story_maxlen, query_maxlen)\n",
    "response = Permute((2, 1))(response)  # (samples, query_maxlen, story_maxlen)\n",
    "\n",
    "# concatenate the match matrix with the question vector sequence\n",
    "answer = concatenate([response, question_encoded])\n",
    "\n",
    "# the original paper uses a matrix multiplication for this reduction step.\n",
    "# we choose to use a RNN instead.\n",
    "answer = LSTM(32)(answer)  # (samples, 32)\n",
    "\n",
    "# one regularization layer -- more would probably be needed.\n",
    "answer = Dropout(0.3)(answer)\n",
    "answer = Dense(vocab_size)(answer)  # (samples, vocab_size)\n",
    "# we output a probability distribution over the vocabulary\n",
    "answer = Activation('softmax')(answer)\n",
    "\n",
    "# build the final model\n",
    "model = Model([input_sequence, question], answer)\n",
    "model.compile(optimizer='rmsprop', loss='sparse_categorical_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "print(\"Done.\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the Neural Network\n",
    "\n",
    "It will take some time (probably up to 1/2 hour) to train this network on a CPU.  Once complete, the program saves the network.  If you've previously saved the neural network, you can skip this step and load it in the next step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 10000 samples, validate on 1000 samples\n",
      "Epoch 1/120\n",
      "10000/10000 [==============================] - 8s 833us/sample - loss: 1.9471 - accuracy: 0.1651 - val_loss: 1.7972 - val_accuracy: 0.1810\n",
      "Epoch 2/120\n",
      "10000/10000 [==============================] - 3s 264us/sample - loss: 1.7601 - accuracy: 0.2340 - val_loss: 1.6338 - val_accuracy: 0.3360\n",
      "Epoch 3/120\n",
      "10000/10000 [==============================] - 3s 266us/sample - loss: 1.5809 - accuracy: 0.3713 - val_loss: 1.5326 - val_accuracy: 0.3750\n",
      "Epoch 4/120\n",
      "10000/10000 [==============================] - 3s 269us/sample - loss: 1.5284 - accuracy: 0.3880 - val_loss: 1.4769 - val_accuracy: 0.4120\n",
      "Epoch 5/120\n",
      "10000/10000 [==============================] - 3s 264us/sample - loss: 1.4962 - accuracy: 0.4024 - val_loss: 1.4577 - val_accuracy: 0.4180\n",
      "Epoch 6/120\n",
      "10000/10000 [==============================] - 3s 269us/sample - loss: 1.4661 - accuracy: 0.4143 - val_loss: 1.4012 - val_accuracy: 0.4370\n",
      "Epoch 7/120\n",
      "10000/10000 [==============================] - 3s 272us/sample - loss: 1.4285 - accuracy: 0.4217 - val_loss: 1.3683 - val_accuracy: 0.4490\n",
      "Epoch 8/120\n",
      "10000/10000 [==============================] - 3s 269us/sample - loss: 1.3995 - accuracy: 0.4384 - val_loss: 1.3553 - val_accuracy: 0.4670\n",
      "Epoch 9/120\n",
      "10000/10000 [==============================] - 4s 350us/sample - loss: 1.3811 - accuracy: 0.4521 - val_loss: 1.3246 - val_accuracy: 0.4630\n",
      "Epoch 10/120\n",
      "10000/10000 [==============================] - 4s 356us/sample - loss: 1.3678 - accuracy: 0.4634 - val_loss: 1.3104 - val_accuracy: 0.5000\n",
      "Epoch 11/120\n",
      "10000/10000 [==============================] - 3s 272us/sample - loss: 1.3517 - accuracy: 0.4772 - val_loss: 1.2891 - val_accuracy: 0.5040\n",
      "Epoch 12/120\n",
      "10000/10000 [==============================] - 3s 277us/sample - loss: 1.3149 - accuracy: 0.4913 - val_loss: 1.2603 - val_accuracy: 0.5100\n",
      "Epoch 13/120\n",
      "10000/10000 [==============================] - 3s 275us/sample - loss: 1.2828 - accuracy: 0.5054 - val_loss: 1.2288 - val_accuracy: 0.5180\n",
      "Epoch 14/120\n",
      "10000/10000 [==============================] - 3s 331us/sample - loss: 1.2644 - accuracy: 0.5055 - val_loss: 1.2229 - val_accuracy: 0.5180\n",
      "Epoch 15/120\n",
      "10000/10000 [==============================] - 3s 280us/sample - loss: 1.2501 - accuracy: 0.5024 - val_loss: 1.2035 - val_accuracy: 0.5170\n",
      "Epoch 16/120\n",
      "10000/10000 [==============================] - 3s 286us/sample - loss: 1.2258 - accuracy: 0.5123 - val_loss: 1.2058 - val_accuracy: 0.5130\n",
      "Epoch 17/120\n",
      "10000/10000 [==============================] - 3s 281us/sample - loss: 1.2225 - accuracy: 0.5116 - val_loss: 1.1962 - val_accuracy: 0.5030\n",
      "Epoch 18/120\n",
      "10000/10000 [==============================] - 3s 302us/sample - loss: 1.2115 - accuracy: 0.5135 - val_loss: 1.1884 - val_accuracy: 0.5220\n",
      "Epoch 19/120\n",
      "10000/10000 [==============================] - 3s 287us/sample - loss: 1.1942 - accuracy: 0.5149 - val_loss: 1.1859 - val_accuracy: 0.5200\n",
      "Epoch 20/120\n",
      "10000/10000 [==============================] - 3s 292us/sample - loss: 1.2001 - accuracy: 0.5147 - val_loss: 1.1892 - val_accuracy: 0.5050\n",
      "Epoch 21/120\n",
      "10000/10000 [==============================] - 3s 297us/sample - loss: 1.1891 - accuracy: 0.5201 - val_loss: 1.1724 - val_accuracy: 0.5150\n",
      "Epoch 22/120\n",
      "10000/10000 [==============================] - 3s 308us/sample - loss: 1.1930 - accuracy: 0.5161 - val_loss: 1.1814 - val_accuracy: 0.5190\n",
      "Epoch 23/120\n",
      "10000/10000 [==============================] - 3s 308us/sample - loss: 1.1822 - accuracy: 0.5209 - val_loss: 1.1685 - val_accuracy: 0.5190\n",
      "Epoch 24/120\n",
      "10000/10000 [==============================] - 3s 313us/sample - loss: 1.1704 - accuracy: 0.5208 - val_loss: 1.1781 - val_accuracy: 0.5180\n",
      "Epoch 25/120\n",
      "10000/10000 [==============================] - 3s 308us/sample - loss: 1.1699 - accuracy: 0.5209 - val_loss: 1.1726 - val_accuracy: 0.5100\n",
      "Epoch 26/120\n",
      "10000/10000 [==============================] - 3s 308us/sample - loss: 1.1614 - accuracy: 0.5254 - val_loss: 1.1835 - val_accuracy: 0.5180\n",
      "Epoch 27/120\n",
      "10000/10000 [==============================] - 3s 323us/sample - loss: 1.1583 - accuracy: 0.5244 - val_loss: 1.1643 - val_accuracy: 0.5250\n",
      "Epoch 28/120\n",
      "10000/10000 [==============================] - 3s 316us/sample - loss: 1.1556 - accuracy: 0.5250 - val_loss: 1.1617 - val_accuracy: 0.5240\n",
      "Epoch 29/120\n",
      "10000/10000 [==============================] - 3s 309us/sample - loss: 1.1499 - accuracy: 0.5209 - val_loss: 1.1577 - val_accuracy: 0.5250\n",
      "Epoch 30/120\n",
      "10000/10000 [==============================] - 3s 322us/sample - loss: 1.1466 - accuracy: 0.5268 - val_loss: 1.1662 - val_accuracy: 0.5250\n",
      "Epoch 31/120\n",
      "10000/10000 [==============================] - 3s 343us/sample - loss: 1.1443 - accuracy: 0.5273 - val_loss: 1.1927 - val_accuracy: 0.5040\n",
      "Epoch 32/120\n",
      "10000/10000 [==============================] - 3s 342us/sample - loss: 1.1394 - accuracy: 0.5291 - val_loss: 1.1504 - val_accuracy: 0.5320\n",
      "Epoch 33/120\n",
      "10000/10000 [==============================] - 4s 378us/sample - loss: 1.1304 - accuracy: 0.5326 - val_loss: 1.1810 - val_accuracy: 0.5120\n",
      "Epoch 34/120\n",
      "10000/10000 [==============================] - 4s 392us/sample - loss: 1.1243 - accuracy: 0.5298 - val_loss: 1.1779 - val_accuracy: 0.5210\n",
      "Epoch 35/120\n",
      "10000/10000 [==============================] - 4s 350us/sample - loss: 1.1259 - accuracy: 0.5318 - val_loss: 1.1901 - val_accuracy: 0.5150\n",
      "Epoch 36/120\n",
      "10000/10000 [==============================] - 3s 294us/sample - loss: 1.1253 - accuracy: 0.5340 - val_loss: 1.1512 - val_accuracy: 0.5230\n",
      "Epoch 37/120\n",
      "10000/10000 [==============================] - 3s 284us/sample - loss: 1.1138 - accuracy: 0.5404 - val_loss: 1.1398 - val_accuracy: 0.5260\n",
      "Epoch 38/120\n",
      "10000/10000 [==============================] - 3s 286us/sample - loss: 1.1143 - accuracy: 0.5350 - val_loss: 1.1582 - val_accuracy: 0.5220\n",
      "Epoch 39/120\n",
      "10000/10000 [==============================] - 3s 284us/sample - loss: 1.1049 - accuracy: 0.5406 - val_loss: 1.1536 - val_accuracy: 0.5070\n",
      "Epoch 40/120\n",
      "10000/10000 [==============================] - 3s 284us/sample - loss: 1.1086 - accuracy: 0.5278 - val_loss: 1.1412 - val_accuracy: 0.5220\n",
      "Epoch 41/120\n",
      "10000/10000 [==============================] - 3s 289us/sample - loss: 1.1060 - accuracy: 0.5404 - val_loss: 1.1373 - val_accuracy: 0.5280\n",
      "Epoch 42/120\n",
      "10000/10000 [==============================] - 3s 286us/sample - loss: 1.0858 - accuracy: 0.5395 - val_loss: 1.1531 - val_accuracy: 0.5140\n",
      "Epoch 43/120\n",
      "10000/10000 [==============================] - 3s 284us/sample - loss: 1.0885 - accuracy: 0.5479 - val_loss: 1.1181 - val_accuracy: 0.5280\n",
      "Epoch 44/120\n",
      "10000/10000 [==============================] - 3s 289us/sample - loss: 1.0649 - accuracy: 0.5602 - val_loss: 1.0867 - val_accuracy: 0.5560\n",
      "Epoch 45/120\n",
      "10000/10000 [==============================] - 3s 278us/sample - loss: 1.0314 - accuracy: 0.5738 - val_loss: 1.0485 - val_accuracy: 0.5720\n",
      "Epoch 46/120\n",
      "10000/10000 [==============================] - 3s 283us/sample - loss: 0.9545 - accuracy: 0.6208 - val_loss: 0.9316 - val_accuracy: 0.6370\n",
      "Epoch 47/120\n",
      "10000/10000 [==============================] - 3s 286us/sample - loss: 0.8300 - accuracy: 0.6822 - val_loss: 0.7902 - val_accuracy: 0.7060\n",
      "Epoch 48/120\n",
      "10000/10000 [==============================] - 3s 278us/sample - loss: 0.7298 - accuracy: 0.7356 - val_loss: 0.7444 - val_accuracy: 0.7380\n",
      "Epoch 49/120\n",
      "10000/10000 [==============================] - 3s 278us/sample - loss: 0.6784 - accuracy: 0.7473 - val_loss: 0.6924 - val_accuracy: 0.7370\n",
      "Epoch 50/120\n",
      "10000/10000 [==============================] - 3s 281us/sample - loss: 0.6438 - accuracy: 0.7606 - val_loss: 0.6591 - val_accuracy: 0.7420\n",
      "Epoch 51/120\n",
      "10000/10000 [==============================] - 3s 291us/sample - loss: 0.6240 - accuracy: 0.7696 - val_loss: 0.6443 - val_accuracy: 0.7380\n",
      "Epoch 52/120\n",
      "10000/10000 [==============================] - 3s 281us/sample - loss: 0.5979 - accuracy: 0.7736 - val_loss: 0.6325 - val_accuracy: 0.7370\n",
      "Epoch 53/120\n",
      "10000/10000 [==============================] - 3s 291us/sample - loss: 0.5786 - accuracy: 0.7769 - val_loss: 0.6150 - val_accuracy: 0.7470\n",
      "Epoch 54/120\n",
      "10000/10000 [==============================] - 3s 277us/sample - loss: 0.5599 - accuracy: 0.7846 - val_loss: 0.5735 - val_accuracy: 0.7470\n",
      "Epoch 55/120\n",
      "10000/10000 [==============================] - 3s 270us/sample - loss: 0.5259 - accuracy: 0.7935 - val_loss: 0.5465 - val_accuracy: 0.7570\n",
      "Epoch 56/120\n",
      "10000/10000 [==============================] - 3s 272us/sample - loss: 0.4993 - accuracy: 0.8014 - val_loss: 0.5168 - val_accuracy: 0.7810\n",
      "Epoch 57/120\n",
      "10000/10000 [==============================] - 3s 264us/sample - loss: 0.4705 - accuracy: 0.8115 - val_loss: 0.5037 - val_accuracy: 0.7770\n",
      "Epoch 58/120\n",
      "10000/10000 [==============================] - 3s 267us/sample - loss: 0.4485 - accuracy: 0.8214 - val_loss: 0.4681 - val_accuracy: 0.8030\n",
      "Epoch 59/120\n",
      "10000/10000 [==============================] - 3s 264us/sample - loss: 0.4285 - accuracy: 0.8321 - val_loss: 0.4545 - val_accuracy: 0.8120\n",
      "Epoch 60/120\n",
      "10000/10000 [==============================] - 3s 267us/sample - loss: 0.4117 - accuracy: 0.8368 - val_loss: 0.4293 - val_accuracy: 0.8130\n",
      "Epoch 61/120\n",
      "10000/10000 [==============================] - 3s 261us/sample - loss: 0.3901 - accuracy: 0.8496 - val_loss: 0.4251 - val_accuracy: 0.8250\n",
      "Epoch 62/120\n",
      "10000/10000 [==============================] - 3s 266us/sample - loss: 0.3775 - accuracy: 0.8559 - val_loss: 0.4070 - val_accuracy: 0.8320\n",
      "Epoch 63/120\n",
      "10000/10000 [==============================] - 3s 269us/sample - loss: 0.3665 - accuracy: 0.8602 - val_loss: 0.3714 - val_accuracy: 0.8590\n",
      "Epoch 64/120\n",
      "10000/10000 [==============================] - 3s 272us/sample - loss: 0.3442 - accuracy: 0.8681 - val_loss: 0.3633 - val_accuracy: 0.8610\n",
      "Epoch 65/120\n",
      "10000/10000 [==============================] - 3s 272us/sample - loss: 0.3380 - accuracy: 0.8705 - val_loss: 0.3604 - val_accuracy: 0.8610\n",
      "Epoch 66/120\n",
      "10000/10000 [==============================] - 3s 272us/sample - loss: 0.3243 - accuracy: 0.8746 - val_loss: 0.3458 - val_accuracy: 0.8670\n",
      "Epoch 67/120\n",
      "10000/10000 [==============================] - 3s 273us/sample - loss: 0.3118 - accuracy: 0.8821 - val_loss: 0.3371 - val_accuracy: 0.8730\n",
      "Epoch 68/120\n",
      "10000/10000 [==============================] - 3s 278us/sample - loss: 0.3030 - accuracy: 0.8838 - val_loss: 0.3380 - val_accuracy: 0.8790\n",
      "Epoch 69/120\n",
      "10000/10000 [==============================] - 3s 277us/sample - loss: 0.3028 - accuracy: 0.8867 - val_loss: 0.3296 - val_accuracy: 0.8740\n",
      "Epoch 70/120\n",
      "10000/10000 [==============================] - 3s 269us/sample - loss: 0.2837 - accuracy: 0.8924 - val_loss: 0.3144 - val_accuracy: 0.8740\n",
      "Epoch 71/120\n",
      "10000/10000 [==============================] - 3s 270us/sample - loss: 0.2763 - accuracy: 0.8976 - val_loss: 0.2991 - val_accuracy: 0.8820\n",
      "Epoch 72/120\n",
      "10000/10000 [==============================] - 3s 272us/sample - loss: 0.2646 - accuracy: 0.8994 - val_loss: 0.2856 - val_accuracy: 0.8900\n",
      "Epoch 73/120\n",
      "10000/10000 [==============================] - 3s 275us/sample - loss: 0.2590 - accuracy: 0.9062 - val_loss: 0.3070 - val_accuracy: 0.8780\n",
      "Epoch 74/120\n",
      "10000/10000 [==============================] - 3s 273us/sample - loss: 0.2596 - accuracy: 0.9032 - val_loss: 0.2889 - val_accuracy: 0.8920\n",
      "Epoch 75/120\n",
      "10000/10000 [==============================] - 3s 269us/sample - loss: 0.2421 - accuracy: 0.9112 - val_loss: 0.2715 - val_accuracy: 0.9000\n",
      "Epoch 76/120\n",
      "10000/10000 [==============================] - 3s 270us/sample - loss: 0.2303 - accuracy: 0.9145 - val_loss: 0.2603 - val_accuracy: 0.9000\n",
      "Epoch 77/120\n",
      "10000/10000 [==============================] - 3s 273us/sample - loss: 0.2237 - accuracy: 0.9173 - val_loss: 0.2623 - val_accuracy: 0.9040\n",
      "Epoch 78/120\n",
      "10000/10000 [==============================] - 3s 278us/sample - loss: 0.2122 - accuracy: 0.9217 - val_loss: 0.2779 - val_accuracy: 0.8910\n",
      "Epoch 79/120\n",
      "10000/10000 [==============================] - 3s 273us/sample - loss: 0.2157 - accuracy: 0.9225 - val_loss: 0.2430 - val_accuracy: 0.9140\n",
      "Epoch 80/120\n",
      "10000/10000 [==============================] - 3s 277us/sample - loss: 0.1991 - accuracy: 0.9270 - val_loss: 0.2349 - val_accuracy: 0.9120\n",
      "Epoch 81/120\n",
      "10000/10000 [==============================] - 4s 370us/sample - loss: 0.1949 - accuracy: 0.9275 - val_loss: 0.2400 - val_accuracy: 0.9170\n",
      "Epoch 82/120\n",
      "10000/10000 [==============================] - 3s 298us/sample - loss: 0.1925 - accuracy: 0.9314 - val_loss: 0.2306 - val_accuracy: 0.9180\n",
      "Epoch 83/120\n",
      "10000/10000 [==============================] - 3s 275us/sample - loss: 0.1814 - accuracy: 0.9361 - val_loss: 0.2351 - val_accuracy: 0.9240\n",
      "Epoch 84/120\n",
      "10000/10000 [==============================] - 3s 278us/sample - loss: 0.1749 - accuracy: 0.9364 - val_loss: 0.2158 - val_accuracy: 0.9310\n",
      "Epoch 85/120\n",
      "10000/10000 [==============================] - 3s 277us/sample - loss: 0.1717 - accuracy: 0.9402 - val_loss: 0.2410 - val_accuracy: 0.9160\n",
      "Epoch 86/120\n",
      "10000/10000 [==============================] - 3s 277us/sample - loss: 0.1640 - accuracy: 0.9421 - val_loss: 0.2247 - val_accuracy: 0.9220\n",
      "Epoch 87/120\n",
      "10000/10000 [==============================] - 3s 278us/sample - loss: 0.1629 - accuracy: 0.9420 - val_loss: 0.2318 - val_accuracy: 0.9200\n",
      "Epoch 88/120\n",
      "10000/10000 [==============================] - 3s 287us/sample - loss: 0.1550 - accuracy: 0.9436 - val_loss: 0.1999 - val_accuracy: 0.9350\n",
      "Epoch 89/120\n",
      "10000/10000 [==============================] - 3s 281us/sample - loss: 0.1523 - accuracy: 0.9478 - val_loss: 0.2186 - val_accuracy: 0.9310\n",
      "Epoch 90/120\n",
      "10000/10000 [==============================] - 3s 283us/sample - loss: 0.1495 - accuracy: 0.9449 - val_loss: 0.2027 - val_accuracy: 0.9330\n",
      "Epoch 91/120\n",
      "10000/10000 [==============================] - 3s 278us/sample - loss: 0.1461 - accuracy: 0.9491 - val_loss: 0.1961 - val_accuracy: 0.9390\n",
      "Epoch 92/120\n",
      "10000/10000 [==============================] - 3s 278us/sample - loss: 0.1392 - accuracy: 0.9541 - val_loss: 0.1989 - val_accuracy: 0.9360\n",
      "Epoch 93/120\n",
      "10000/10000 [==============================] - 3s 277us/sample - loss: 0.1371 - accuracy: 0.9539 - val_loss: 0.1953 - val_accuracy: 0.9420\n",
      "Epoch 94/120\n",
      "10000/10000 [==============================] - 3s 278us/sample - loss: 0.1374 - accuracy: 0.9519 - val_loss: 0.1902 - val_accuracy: 0.9440\n",
      "Epoch 95/120\n",
      "10000/10000 [==============================] - 3s 277us/sample - loss: 0.1289 - accuracy: 0.9531 - val_loss: 0.2170 - val_accuracy: 0.9350\n",
      "Epoch 96/120\n",
      "10000/10000 [==============================] - 3s 273us/sample - loss: 0.1243 - accuracy: 0.9565 - val_loss: 0.2341 - val_accuracy: 0.9270\n",
      "Epoch 97/120\n",
      "10000/10000 [==============================] - 3s 289us/sample - loss: 0.1170 - accuracy: 0.9597 - val_loss: 0.2245 - val_accuracy: 0.9300\n",
      "Epoch 98/120\n",
      "10000/10000 [==============================] - 3s 281us/sample - loss: 0.1218 - accuracy: 0.9589 - val_loss: 0.1876 - val_accuracy: 0.9380\n",
      "Epoch 99/120\n",
      "10000/10000 [==============================] - 3s 283us/sample - loss: 0.1210 - accuracy: 0.9583 - val_loss: 0.1957 - val_accuracy: 0.9430\n",
      "Epoch 100/120\n",
      "10000/10000 [==============================] - 3s 280us/sample - loss: 0.1108 - accuracy: 0.9603 - val_loss: 0.2068 - val_accuracy: 0.9350\n",
      "Epoch 101/120\n",
      "10000/10000 [==============================] - 3s 283us/sample - loss: 0.1103 - accuracy: 0.9625 - val_loss: 0.1870 - val_accuracy: 0.9470\n",
      "Epoch 102/120\n",
      "10000/10000 [==============================] - 3s 278us/sample - loss: 0.1112 - accuracy: 0.9608 - val_loss: 0.1912 - val_accuracy: 0.9430\n",
      "Epoch 103/120\n",
      "10000/10000 [==============================] - 3s 281us/sample - loss: 0.1004 - accuracy: 0.9658 - val_loss: 0.1839 - val_accuracy: 0.9360\n",
      "Epoch 104/120\n",
      "10000/10000 [==============================] - 3s 283us/sample - loss: 0.0995 - accuracy: 0.9663 - val_loss: 0.2073 - val_accuracy: 0.9350\n",
      "Epoch 105/120\n",
      "10000/10000 [==============================] - 3s 286us/sample - loss: 0.0960 - accuracy: 0.9663 - val_loss: 0.1979 - val_accuracy: 0.9400\n",
      "Epoch 106/120\n",
      "10000/10000 [==============================] - 3s 283us/sample - loss: 0.0893 - accuracy: 0.9708 - val_loss: 0.1873 - val_accuracy: 0.9350\n",
      "Epoch 107/120\n",
      "10000/10000 [==============================] - 3s 283us/sample - loss: 0.0920 - accuracy: 0.9691 - val_loss: 0.1684 - val_accuracy: 0.9450\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 108/120\n",
      "10000/10000 [==============================] - 3s 267us/sample - loss: 0.0917 - accuracy: 0.9663 - val_loss: 0.3405 - val_accuracy: 0.9030\n",
      "Epoch 109/120\n",
      "10000/10000 [==============================] - 3s 272us/sample - loss: 0.0949 - accuracy: 0.9676 - val_loss: 0.1673 - val_accuracy: 0.9460\n",
      "Epoch 110/120\n",
      "10000/10000 [==============================] - 3s 270us/sample - loss: 0.0890 - accuracy: 0.9697 - val_loss: 0.1612 - val_accuracy: 0.9490\n",
      "Epoch 111/120\n",
      "10000/10000 [==============================] - 3s 264us/sample - loss: 0.0858 - accuracy: 0.9713 - val_loss: 0.1607 - val_accuracy: 0.9500\n",
      "Epoch 112/120\n",
      "10000/10000 [==============================] - 3s 264us/sample - loss: 0.0784 - accuracy: 0.9734 - val_loss: 0.1811 - val_accuracy: 0.9520\n",
      "Epoch 113/120\n",
      "10000/10000 [==============================] - 3s 270us/sample - loss: 0.0843 - accuracy: 0.9710 - val_loss: 0.1714 - val_accuracy: 0.9460\n",
      "Epoch 114/120\n",
      "10000/10000 [==============================] - 3s 267us/sample - loss: 0.0811 - accuracy: 0.9722 - val_loss: 0.1934 - val_accuracy: 0.9390\n",
      "Epoch 115/120\n",
      "10000/10000 [==============================] - 3s 273us/sample - loss: 0.0747 - accuracy: 0.9721 - val_loss: 0.1689 - val_accuracy: 0.9540\n",
      "Epoch 116/120\n",
      "10000/10000 [==============================] - 3s 278us/sample - loss: 0.0805 - accuracy: 0.9708 - val_loss: 0.1870 - val_accuracy: 0.9450\n",
      "Epoch 117/120\n",
      "10000/10000 [==============================] - 3s 273us/sample - loss: 0.0719 - accuracy: 0.9757 - val_loss: 0.1892 - val_accuracy: 0.9500\n",
      "Epoch 118/120\n",
      "10000/10000 [==============================] - 3s 280us/sample - loss: 0.0810 - accuracy: 0.9730 - val_loss: 0.1575 - val_accuracy: 0.9470\n",
      "Epoch 119/120\n",
      "10000/10000 [==============================] - 3s 280us/sample - loss: 0.0705 - accuracy: 0.9760 - val_loss: 0.1762 - val_accuracy: 0.9490\n",
      "Epoch 120/120\n",
      "10000/10000 [==============================] - 3s 283us/sample - loss: 0.0708 - accuracy: 0.9753 - val_loss: 0.1477 - val_accuracy: 0.9560\n",
      "Elapsed time: 0:05:49.06\n"
     ]
    }
   ],
   "source": [
    "start_time = time.time()\n",
    "# train\n",
    "model.fit([inputs_train, queries_train], answers_train,\n",
    "          batch_size=32,\n",
    "          epochs=120,\n",
    "          validation_data=([inputs_test, queries_test], answers_test))\n",
    "\n",
    "# save\n",
    "save_path = \"./data/\"\n",
    "# save entire network to HDF5 (save everything, suggested)\n",
    "model.save(os.path.join(save_path,\"chatbot.h5\"))\n",
    "# save the vocab too, indexes must be the same\n",
    "pickle.dump( vocab, open( os.path.join(save_path,\"vocab.pkl\"), \"wb\" ) )\n",
    "\n",
    "elapsed_time = time.time() - start_time\n",
    "print(\"Elapsed time: {}\".format(hms_string(elapsed_time)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the model, if it exists, load vocab too\n",
    "save_path = \"./data/\"\n",
    "model = load_model(os.path.join(save_path,\"chatbot.h5\"))\n",
    "vocab = pickle.load( open( os.path.join(save_path,\"vocab.pkl\"), \"rb\" ) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate Accuracy\n",
    "\n",
    "We evaluate the accuracy, using the same technique as previous classification networks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[3.2316551e-18 4.5703689e-18 4.2100677e-18 ... 2.7742484e-18\n",
      "  3.3352055e-18 3.6209529e-18]\n",
      " [8.6646321e-15 8.6298309e-15 7.7134844e-15 ... 8.2590549e-15\n",
      "  7.4069846e-15 9.1656156e-15]\n",
      " [1.6307067e-15 2.1523891e-15 2.2016419e-15 ... 1.6251486e-15\n",
      "  2.1003965e-15 2.4207346e-15]\n",
      " ...\n",
      " [2.0678353e-16 2.2777190e-16 2.1688923e-16 ... 2.1682140e-16\n",
      "  2.0552370e-16 1.9355541e-16]\n",
      " [5.6333461e-17 5.9818837e-17 4.9700267e-17 ... 5.3023650e-17\n",
      "  6.0595586e-17 5.4082653e-17]\n",
      " [1.6186161e-09 1.6892178e-09 1.6479546e-09 ... 1.6192523e-09\n",
      "  1.3914115e-09 1.7729884e-09]]\n"
     ]
    }
   ],
   "source": [
    "pred = model.predict([inputs_test, queries_test])\n",
    "# See what the predictions look like, they are just probabilities of each class.\n",
    "print(pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[12  9 15 12 15 12 11 12 17 17 15 11 17 10 10 15 11 15 17 12 12 17 17 12\n",
      " 10 10 10 15 15 15 12 17 15 15  9 17  9 11 15 10  9 11 10 12 11 15 12  9\n",
      " 17 10 10 17 11 11 12 10 15 12 11 12 12 17  9 11 10 15 15  9 17 17 11 12\n",
      " 11 10  9 15 15 15 12 17 11  9 10 15 17 11 11 12 15 15  9  9 12  9  9 15\n",
      " 15 15 10 10 10  9 17 11 11 12 10  9 17 10  9  9 15 12 15 11 17 15  9 17\n",
      " 10  9  9 12 10 17 11 11 11  9 11 12  9 11 17  9 17 15  9  9 17 12 12 17\n",
      " 10 15 15 15 15 15 11 11  9 17 10  9 11 10 17 12 10 10 10 11  9 11 11 11\n",
      " 15 10 17 17 11  9  9 17 15 10 17 17 12 12 12 12 10  9 17 11 15 17 12 15\n",
      " 11 12 12 11 17 15  9 10 12 17 12 10 12  9 15 12  9 12  9 10 12 10  9 10\n",
      "  9 12 12  9  9 12 15 11 10 17 12 11  9  9 17 11 12 17 12 10 10  9 10 11\n",
      " 12 15 11 11 15 15 11 11 11 11 10 17  9  9 10 10 17 17 15 11 11 17 12 17\n",
      " 17  9 11 10  9 10 15 17 17 15  9 17 15 12 17  9  9 11 17 17 11 17  9 17\n",
      " 11  9 15 15 10 15 15 11 11 12 12  9  9  9  9 17 12 11  9 17 17 15 11 11\n",
      " 17 10 11 17 10 17 17 11 11 15 15 11 11  9 10 11 11 10 15 17 15 10 11 17\n",
      " 12 17 12 11 12 11 17 15 11 17 17 12 10 15 10 11 11 10 10  9 17 12 11 12\n",
      " 11  9 11 11 11 11 11 12 12 12 12 12 11 17 17 10 15  9  9 10  9 10 10 17\n",
      " 17 15 17 17 17 15 15 12  9 10 17 17 10 17  9 11 10 11 10 15 17 15 12 12\n",
      " 11 15 10 11  9 12 17 10 10 15 12  9 12 17  9 12 10 15 15 17 10 10 17 15\n",
      " 15 11 17 11  9  9 10 11 10 12  9 11 10 11 10 12 11 15 15 15 15 11  9 15\n",
      " 10 12 11 12  9 17 10 17 11 11 10 10 10 17 12 17 17  9 15 11  9 15 11  9\n",
      " 10 10 12 12 11 10  9 12 10  9 15 10 10 11 11 15 15 10 12 12 15 11  9 12\n",
      "  9 15 10 17 17 10 10 17 10 10 17  9  9 10 12 11 17 17 12 12 11 12 17 11\n",
      " 11 12 11 11 11 17  9 11 10 12 11 10 11 17 12 12 15 11 15 11 12 10 10  9\n",
      " 17 10 15 12 15 10 10 11 11 10 11 15 10  9  9 15 10 17 10 10 15 15 17  9\n",
      " 10 12 10 10 12 12 11 15 10 10 11 12 10 10 17 12 17 11  9  9 17 17 11 10\n",
      " 10 10 15 15 17 11 15 12 17 10 15 17  9 11  9 17 15 12 15 17 11 17 17 15\n",
      "  9 10  9  9 10 10  9 10 10 15 11 11 11 15 12  9 11 11 10 17 11 11 12 17\n",
      " 15 11  9 11 15 15 11  9 15 15 11 15 15 15 11 10 10  9 17 17 17 15 10 10\n",
      " 17  9 15  9 17 12 12  9 15 15 15 17  9 11  9 17  9 12 11  9  9 17 15 11\n",
      " 11 11 15 12 12 12  9  9 17 12 12  9 17 12 11 11 15 15 12 11 12 17 15 17\n",
      " 11 12 12 17 17 11 17  9 11  9 10 10 12 17 12 12  9  9  9 15 11 11 15 17\n",
      "  9 10  9 12 17 15 17 17 12 10 10 11 11 17 15  9 17 10  9 11 12 15  9 15\n",
      " 12 10  9 10 17 17 17 15  9  9  9 17 10 17 17  9 12 17 17 12 12 11 11 11\n",
      "  9 17 11 17 17 17 11 11 12  9 17 10 17 11 10 10 12 12 11 11 17 12 12 17\n",
      " 17 15 11 17 11 11  9 12  9 17  9 10  9 15 11  9  9 15 10  9  9  9 17 12\n",
      " 11 11 17 15 15 10 10 11 11 17 10 12  9 10 10 10 10 10 15 17  9 11 17 15\n",
      " 12 11 10 10 15 12 10 11 12 12 11 11 11 15 12 17 10 12 17 17 12 10 10 12\n",
      " 17 11 11 17 11 12 17 17 11 12 11 17  9  9 11 17 11 11 15 15 17 17 10 10\n",
      " 17 12 11 15 12 11 11 10  9 12 12 15 10  9 17 15 17  9 17 17 10 17 15 12\n",
      " 12 11 10 10 10 17 17 15 12 17  9  9 15 10 12 15 15 15 17 17 15 11 15 10\n",
      " 12  9  9 10 11 11  9  9 12 17  9 15  9 12 12 10 17 17 10 12 15  9 10 17\n",
      " 11 11 11 17  9  9 11 15 11 11 11 15 12 17  9 11]\n"
     ]
    }
   ],
   "source": [
    "# Use argmax to turn those into actual predictions.  The class (word) with the highest\n",
    "# probability is the answer.\n",
    "\n",
    "pred = np.argmax(pred,axis=1)\n",
    "print(pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final accuracy: 0.956\n"
     ]
    }
   ],
   "source": [
    "score = metrics.accuracy_score(answers_test, pred)\n",
    "print(\"Final accuracy: {}\".format(score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adhoc Query\n",
    "\n",
    "You might want to create your own stories and questions.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Remember, I only know these words: ['.', '?', 'Daniel', 'John', 'Mary', 'Sandra', 'Where', 'back', 'bathroom', 'bedroom', 'garden', 'hallway', 'is', 'journeyed', 'kitchen', 'moved', 'office', 'the', 'to', 'travelled', 'went']\n",
      "\n",
      "[5.3319661e-11 5.9570487e-11 5.1706601e-11 5.1733437e-11 5.4510025e-11\n",
      " 5.7622622e-11 4.8129528e-11 5.3163075e-11 5.6659514e-11 8.2092601e-01\n",
      " 2.8999595e-02 1.4757804e-02 3.3609481e-03 5.6475557e-11 5.3110880e-11\n",
      " 1.3155086e-01 5.3828338e-11 4.0473844e-04 5.5407737e-11 5.2006194e-11\n",
      " 5.0534833e-11 5.6617922e-11]\n",
      "Answer: bathroom([9])\n"
     ]
    }
   ],
   "source": [
    "print(\"Remember, I only know these words: {}\".format(vocab))\n",
    "print()\n",
    "story = \"Daniel went to the hallway. Mary went to the bathroom.\"\\\n",
    "    \"Daniel went to the bedroom.\"\n",
    "query = \"Where is Sandra?\"\n",
    "\n",
    "adhoc_stories = (tokenize(story), tokenize(query), '?')\n",
    "\n",
    "adhoc_train, adhoc_query, adhoc_answer = vectorize_stories([adhoc_stories])\n",
    "\n",
    "pred = model.predict([adhoc_train, adhoc_query])\n",
    "print(pred[0])\n",
    "pred = np.argmax(pred,axis=1)\n",
    "print(\"Answer: {}({})\".format(vocab[pred[0]-1],pred))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.7 (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}