4. Sentiment Analysis with Keras & Tensorflow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sentiment analysis \n",
    "\n",
    "# Introduction\n",
    "Analyze & classify sentiment of text data, articles into positive or negative\n",
    "\n",
    "# Objective\n",
    "Sentiment analysis notebooks dives in very depth of various concepts, methods related to text analysis and understand the meaning of it semantically and/or syntactly. They are classified in the following five based notebooks based on different methods & tools used to analyze & classify text.\n",
    "\n",
    "1. Sentiment Analysis with Text Blob, Word Cloud, Count Vectorizer, N-Gram\n",
    "2. Sentiment Analysis using Doc2Vec, N-Gram & Phrase Modelling\n",
    "3. Sentiment Analysis with Chi2 Square & PCA Dimension Reduction\n",
    "4. Sentiment Analysis with Keras & Tensorflow\n",
    "5. Sentiment Analysis with Keras & Tensorflow using Doc2Vec, Pretrained GloVe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cuatro\n",
    "## 4. Sentiment Analysis with Keras & Tensorflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Basic import\n",
    "\n",
    "import re\n",
    "import pandas as pd  \n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "plt.style.use('fivethirtyeight')\n",
    "\n",
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format = 'retina'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from tqdm import tqdm\n",
    "# tqdm.pandas(desc=\"progress-bar\")\n",
    "\n",
    "# from gensim.models import Doc2Vec\n",
    "# from gensim.models.doc2vec import LabeledSentence\n",
    "# from gensim.models.phrases import Phrases, Phraser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from textblob import TextBlob\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.metrics import classification_report, confusion_matrix\n",
    "\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.pipeline import Pipeline\n",
    "from datetime import datetime\n",
    "\n",
    "import multiprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 1600000 entries, 0 to 1599999\n",
      "Data columns (total 3 columns):\n",
      "sentiment        1600000 non-null int64\n",
      "text             1600000 non-null object\n",
      "pre_clean_len    1600000 non-null int64\n",
      "dtypes: int64(2), object(1)\n",
      "memory usage: 48.8+ MB\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentiment</th>\n",
       "      <th>text</th>\n",
       "      <th>pre_clean_len</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>awww that bummer you shoulda got david carr of...</td>\n",
       "      <td>115</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>is upset that he can not update his facebook b...</td>\n",
       "      <td>111</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>dived many times for the ball managed to save ...</td>\n",
       "      <td>89</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>my whole body feels itchy and like its on fire</td>\n",
       "      <td>47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>no it not behaving at all mad why am here beca...</td>\n",
       "      <td>111</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   sentiment                                               text  pre_clean_len\n",
       "0          0  awww that bummer you shoulda got david carr of...            115\n",
       "1          0  is upset that he can not update his facebook b...            111\n",
       "2          0  dived many times for the ball managed to save ...             89\n",
       "3          0     my whole body feels itchy and like its on fire             47\n",
       "4          0  no it not behaving at all mad why am here beca...            111"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read TF dataframe\n",
    "\n",
    "df = pd.read_hdf('./data/redstone.hdf')\n",
    "df.info()\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 1600000 entries, 0 to 1599999\n",
      "Data columns (total 3 columns):\n",
      "sentiment        1600000 non-null int64\n",
      "text             1600000 non-null object\n",
      "pre_clean_len    1600000 non-null int64\n",
      "dtypes: int64(2), object(1)\n",
      "memory usage: 36.6+ MB\n"
     ]
    }
   ],
   "source": [
    "# Santitizing dataframe\n",
    "\n",
    "df.dropna(inplace=True)\n",
    "df.reset_index(drop=True,inplace=True)\n",
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/indiano/miniconda3/envs/tf/lib/python3.5/site-packages/sklearn/cross_validation.py:41: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.\n",
      "  \"This module will be removed in 0.20.\", DeprecationWarning)\n"
     ]
    }
   ],
   "source": [
    "from sklearn import utils\n",
    "from sklearn.cross_validation import train_test_split\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "train = df.text\n",
    "label = df.sentiment\n",
    "SEED = 21\n",
    "\n",
    "# Splitting data into train, test & validation sets\n",
    "x_train, x_val_test, y_train, y_val_test = train_test_split(train, label, test_size=.02, random_state=SEED)\n",
    "\n",
    "x_val, x_test, y_val, y_test = train_test_split(x_val_test, y_val_test, test_size=.5, random_state=SEED)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.series.Series"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set entries.\n",
      "50.00% Negative Entries\n",
      "50.00% Positive Entries\n",
      "\n",
      "Validation set entries.\n",
      "50.01% Negative Entries\n",
      "49.99% Positive Entries\n",
      "\n",
      "Test set entries.\n",
      "50.21% Negative Entries\n",
      "49.79% Positive Entries\n"
     ]
    }
   ],
   "source": [
    "# Quantifying the positive & negative sentiments in the dataset\n",
    "\n",
    "from collections import Counter\n",
    "\n",
    "counter = Counter(y_train)\n",
    "print('Train set entries.')\n",
    "for key in counter:\n",
    "    if key == 0:\n",
    "        print('{:.2f}% Negative Entries'.format( (counter[key]/len(y_train))*100 ))\n",
    "    elif key == 1:\n",
    "        print('{:.2f}% Positive Entries'.format( (counter[key]/len(y_train))*100 ))\n",
    "        \n",
    "counter = Counter(y_val)\n",
    "print('\\nValidation set entries.')\n",
    "for key in counter:\n",
    "    if key == 0:\n",
    "        print('{:.2f}% Negative Entries'.format( (counter[key]/len(y_val))*100 ))\n",
    "    elif key == 1:\n",
    "        print('{:.2f}% Positive Entries'.format( (counter[key]/len(y_val))*100 ))\n",
    "\n",
    "counter = Counter(y_test)\n",
    "print('\\nTest set entries.')\n",
    "for key in counter:\n",
    "    if key == 0:\n",
    "        print('{:.2f}% Negative Entries'.format( (counter[key]/len(y_test))*100 ))\n",
    "    elif key == 1:\n",
    "        print('{:.2f}% Positive Entries'.format( (counter[key]/len(y_test))*100 ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Artificial Neural Networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After experimenting with Logistic Regression, it would be interesting to evaluate the result of neural network classifier. Logistic regression can be thought as a basic neural network with no hidden layer and just one output node."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![title](./images/lr_nn.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TFIDF Vectorizer with Artificial Neural Networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The best performing TFIDF vectors have 100,000 features for (Unigram + Trigram) word tokens with logistic regression. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "tvec = TfidfVectorizer(max_features=100000, ngram_range=(1, 3))\n",
    "tvec = tvec.fit(x_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Transform train  & validation set\n",
    "\n",
    "tf_train = tvec.transform(x_train)\n",
    "tf_val = tvec.transform(x_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 30.7 s, sys: 124 ms, total: 30.8 s\n",
      "Wall time: 31 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# Fitting Logistic Regression classical model\n",
    "clf = LogisticRegression()\n",
    "clf.fit(tf_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8413386479591837"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "0.824"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Train & Validation scores\n",
    "\n",
    "display(clf.score(tf_train, y_train))\n",
    "display(clf.score(tf_val, y_val))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Keras "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "# Basic Keras Import\n",
    "\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout, Flatten\n",
    "from keras.layers.embeddings import Embedding\n",
    "from keras.preprocessing import sequence\n",
    "\n",
    "import numpy as np\n",
    "\n",
    "# Fix the seed\n",
    "np.random.seed(21)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ADAM is an optimization algorithm for updating the parameters and minimizing the cost of the neural network, which is proved to be very effective. It combines two methods of optimization: RMSProp, Momentum. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Keras NN model cannot handle sparse matrix directly. Hence the data has to be either a dense array or matrix, but transforming the whole training data of 1.5 million (TFIDF vectors) into a dense array won't fit into my RAM. \n",
    "An iterable generator object would solve this problem by generating required data on the run which can be achieved by using \"yield\" instead of \"return\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Batch generator\n",
    "\n",
    "def batch_generator(train, label, batch_size):\n",
    "    \n",
    "    # Calculate no of batches\n",
    "    number_of_batches = train.shape[0]/batch_size\n",
    "    \n",
    "    # Data set indices to choose a batch from\n",
    "    batch = np.arange(tf_train.shape[0])\n",
    "    # Starting batch index\n",
    "    batch_idx = 0\n",
    "    while True:\n",
    "        # Selecting batches\n",
    "        train_batch = train[ batch[batch_size*batch_idx:batch_size*(batch_idx+1)], :].toarray()\n",
    "        label_batch = label[ batch[batch_size*batch_idx:batch_size*(batch_idx+1)] ]\n",
    "\n",
    "        #print('\\n{} Batch indices from {} to {} selected.\\n'.format((batch_idx+1), (batch_size*batch_idx), (batch_size*(batch_idx+1))))\n",
    "        \n",
    "        # Generator statement\n",
    "        yield train_batch, label_batch\n",
    "        \n",
    "        # Next batch\n",
    "        batch_idx += 1\n",
    "        # Check if 1 epoch is finished then next batch index should be greater than no of batches\n",
    "        if (batch_idx > number_of_batches):\n",
    "            batch_idx=0\n",
    "            "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Parameters\n",
    "\n",
    "batch_size = 16"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/indiano/.local/lib/python3.5/site-packages/pandas/core/series.py:696: FutureWarning: \n",
      "Passing list-likes to .loc or [] with any missing label will raise\n",
      "KeyError in the future, you can use .reindex() as an alternative.\n",
      "\n",
      "See the documentation here:\n",
      "http://pandas.pydata.org/pandas-docs/stable/indexing.html#deprecate-loc-reindex-listlike\n",
      "  return self.loc[key]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "31156/98000 [========>.....................] - ETA: 21:39 - loss: nan - acc: 1.3440e-04"
     ]
    }
   ],
   "source": [
    "# Create Model\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(64, input_dim=100000, activation='relu'))\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "\n",
    "# Compile Model\n",
    "\n",
    "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "\n",
    "# Fit the model\n",
    "\n",
    "model.fit_generator(generator=batch_generator(tf_train, y_train, batch_size),\n",
    "                    epochs=5, validation_data=(tf_val, y_val),\n",
    "                    steps_per_epoch=tf_train.shape[0]/batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Normalizing inputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's see if normalizing the inputs have any effect on the performance.\n",
    "\n",
    "from sklearn.preprocessing import Normalizer\n",
    "\n",
    "norm = Normalizer().fit(tf_train)\n",
    "\n",
    "tf_train_norm = norm.transform(tf_train)\n",
    "tf_val_norm = norm.transform(tf_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create Model\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(64, input_dim=100000, activation='relu'))\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "\n",
    "# Compile Model\n",
    "\n",
    "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "\n",
    "# Fit the model with normalized inputs\n",
    "\n",
    "model.fit_generator(generator=batch_generator(tf_train_norm, y_train, batch_size),\n",
    "                    epochs=5, validation_data=(tf_val_norm, y_val),\n",
    "                    steps_per_epoch=tf_train.shape[0]/batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TFIDF is already normalized. TF (Term Frequency) in TFIDF isn't the absolute frequency but relative frequency and after multiplying IDF (Inverse Document Frequency) to the relative term frequency value, it further normalizes the values in a cross-document manner."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Dropout"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "According to the research paper \"Improving neural networks by preventing co-adaptation of feature detectors\" by Hinton et al. (2012), a good way to reduce the error on the test set is to average the predictions produced by a very large number of different networks. \n",
    "The standard way to do this is to train many separate networks and apply each of these networks to the test data, but this is computationally expensive during both phase of training and testing. \n",
    "Random dropout makes it possible to train a huge number of different networks in a reasonable time.\n",
    "- https://arxiv.org/pdf/1207.0580.pdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dropout could be thought as the simulation of training many different networks and averaging them by randomly omitting hidden nodes with a certain probability, throughout the training process. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create Model\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(64, input_dim=100000, activation='relu'))\n",
    "model.add(Dropout(0.2))\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "\n",
    "# Compile Model\n",
    "\n",
    "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "\n",
    "# Fit the model\n",
    "\n",
    "model.fit_generator(generator=batch_generator(tf_train, y_train, batch_size),\n",
    "                    epochs=5, validation_data=(tf_val, y_val),\n",
    "                    steps_per_epoch=tf_train.shape[0]/batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dropout has added some generalization to the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Shuffling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By presenting data in the same order during each epoch, there's a possibility that the model learns the parameters which also include noise of the training data. It might eventually lead to overfitting. It can be mitigated by shuffling the order of the data fed to the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Updated Batch Generator with Shuffling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Batch generator updated\n",
    "\n",
    "def batch_generator_shuffle(train, label, batch_size):\n",
    "    \n",
    "    # Calculate no of batches\n",
    "    number_of_batches = train.shape[0]/batch_size\n",
    "    \n",
    "    # Data set indices to choose a batch from\n",
    "    batch = np.arange(tf_train.shape[0])\n",
    "    # Shuffling batch indices\n",
    "    np.random.shuffle(batch)\n",
    "    \n",
    "    # Starting batch index\n",
    "    batch_idx = 0\n",
    "    while True:\n",
    "        # Selecting batches\n",
    "        train_batch = train[ batch[batch_size*batch_idx:batch_size*(batch_idx+1)], :].toarray()\n",
    "        label_batch = label[ batch[batch_size*batch_idx:batch_size*(batch_idx+1)] ]\n",
    "\n",
    "        #print('\\n{} Batch indices from {} to {} selected.\\n'.format((batch_idx+1), (batch_size*batch_idx), (batch_size*(batch_idx+1))))\n",
    "        \n",
    "        # Generator statement\n",
    "        yield train_batch, label_batch\n",
    "        \n",
    "        # Next batch\n",
    "        batch_idx += 1\n",
    "        # Check if 1 epoch is finished then next batch index should be greater than no of batches\n",
    "        if (batch_idx > number_of_batches):\n",
    "            np.random.shuffle(batch)\n",
    "            batch_idx=0\n",
    "            "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Shuffling did improve the model's performance on the validation set. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the \"deeplearning.ai\" course by Andrew Ng, he states that the first thing he would try to improve a neural network model is tweaking the learning rate. \n",
    "Please note that except for the learning rate, the parameters 'beta_1', 'beta_2', and 'epsilon' are set to their default values as presented in the original paper titled\n",
    "\"ADAM: A Method for Stochastic Optimization\" by Kingma and Ba (2015).       \n",
    "- https://arxiv.org/pdf/1412.6980.pdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "import keras\n",
    "\n",
    "# My ADAM with lr 0.005\n",
    "my_adam = keras.optimizers.Adam(lr=0.005, beta_1=0.9, beta_2=0.999, epsilon=1e-8)\n",
    "\n",
    "# Create Model\n",
    "model = Sequential()\n",
    "model.add(Dense(64, input_dim=100000, activation='relu'))\n",
    "model.add(Dropout(0.2))\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "\n",
    "# Compile Model\n",
    "\n",
    "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "\n",
    "# Fit the model\n",
    "\n",
    "model.fit_generator(generator=batch_generator_shuffle(tf_train, y_train, batch_size),\n",
    "                    epochs=5, validation_data=(tf_val, y_val),\n",
    "                    steps_per_epoch=tf_train.shape[0]/batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After trying four different learning rates (0.0005, 0.005, 0.01, 0.1), it seems that none of them outperformed the default learning rate of 0.001."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Increase Hidden Nodes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create Model\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(128, input_dim=100000, activation='relu'))\n",
    "model.add(Dropout(0.2))\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "\n",
    "# Compile Model\n",
    "\n",
    "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "\n",
    "# Fit the model\n",
    "\n",
    "model.fit_generator(generator=batch_generator(tf_train, y_train, batch_size),\n",
    "                    epochs=5, validation_data=(tf_val, y_val),\n",
    "                    steps_per_epoch=tf_train.shape[0]/batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With 128 hidden nodes, the validation accuracy got closer to the performance of logistic regression. Anyhow, Logistic regression took less than a minute to fit and even if the neural network can be improved further, it isn't an efficient way."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As neural network models failed to outperform logistic regression, the probable cause might be high dimensionality and sparse characteristics of the textual data. \n",
    "According to \"An Empirical Evaluation of Supervised Learning in High Dimensions\" by Caruana et al.(2008), logistic regression showed as good performance as neural networks, in some cases outperforms neural networks.   \n",
    "- http://icml2008.cs.helsinki.fi/papers/632.pdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even though the neural network is a more complex version of logistic regression, it doesn't always outperform logistic regression.\n",
    "Sometimes with high dimensional sparse data, logistic regression can deliver good performance with much less computation time than neural network."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}