t81_558_class_14_03_anomaly.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 14: Other Neural Network Techniques**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 14 Video Material\n",
    "\n",
    "* Part 14.1: What is AutoML [[Video]](https://www.youtube.com/watch?v=TFUysIR5AB0&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_14_01_automl.ipynb)\n",
    "* Part 14.2: Using Denoising AutoEncoders in Keras [[Video]](https://www.youtube.com/watch?v=4bTSu6_fucc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_14_02_auto_encode.ipynb)\n",
    "* **Part 14.3: Training an Intrusion Detection System with KDD99** [[Video]](https://www.youtube.com/watch?v=1ySn6h2A68I&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_14_03_anomaly.ipynb)\n",
    "* Part 14.4: Anomaly Detection in Keras [[Video]](https://www.youtube.com/watch?v=VgyKQ5MTDFc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_14_04_ids_kdd99.ipynb)\n",
    "* Part 14.5: The Deep Learning Technologies I am Excited About [[Video]]() [[Notebook]](t81_558_class_14_05_new_tech.ipynb)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 14.3: Anomaly Detection in Keras\n",
    "\n",
    "Anomaly detection is an unsupervised training technique that analyzes the degree to which incoming data is different than data that the data you used to train the neural network. Traditionally, cybersecurity experts have used anomaly detection to ensure network security.  However, you can use anomoly in data science to detect input that you have not trained your neural network for.  \n",
    "\n",
    "There are several data sets that are commonly used to demonstrate anomaly detection. In this part, we will look at the KDD-99 dataset.\n",
    "\n",
    "\n",
    "* [Stratosphere IPS Dataset](https://www.stratosphereips.org/category/dataset.html)\n",
    "* [The ADFA Intrusion Detection Datasets (2013) - for HIDS](https://www.unsw.adfa.edu.au/unsw-canberra-cyber/cybersecurity/ADFA-IDS-Datasets/)\n",
    "* [ITOC CDX (2009)](https://westpoint.edu/centers-and-research/cyber-research-center/data-sets)\n",
    "* [KDD-99 Dataset](http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html)\n",
    "\n",
    "### Read in KDD99 Data Set\n",
    "\n",
    "Although KDD99 dataset is over 20 years old, it is still widely used to demonstrate Intrusion Detection Systems (IDS) and Anomaly detection. KDD99 is the data set used for The Third International Knowledge Discovery and Data Mining Tools Competition, which was held in conjunction with KDD-99 The Fifth International Conference on Knowledge Discovery and Data Mining. The competition task was to build a network intrusion detector, a predictive model capable of distinguishing between \"bad\" connections, called intrusions or attacks, and \"good\" normal connections. This database contains a standard set of data to be audited, including a wide variety of intrusions simulated in a military network environment.\n",
    "\n",
    "The following code reads the KDD99 CSV dataset into a Pandas data frame.  The standard format of KDD99 does not include column names.  Because of that, the program adds them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from http://kdd.ics.uci.edu/databases/kddcup99/kddcup.data_10_percent.gz\n",
      "2146304/2144903 [==============================] - 6s 3us/step\n",
      "/Users/jheaton/.keras/datasets/kddcup.data_10_percent.gz\n",
      "Read 494021 rows.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>duration</th>\n",
       "      <th>protocol_type</th>\n",
       "      <th>service</th>\n",
       "      <th>flag</th>\n",
       "      <th>src_bytes</th>\n",
       "      <th>dst_bytes</th>\n",
       "      <th>land</th>\n",
       "      <th>wrong_fragment</th>\n",
       "      <th>urgent</th>\n",
       "      <th>hot</th>\n",
       "      <th>...</th>\n",
       "      <th>dst_host_srv_count</th>\n",
       "      <th>dst_host_same_srv_rate</th>\n",
       "      <th>dst_host_diff_srv_rate</th>\n",
       "      <th>dst_host_same_src_port_rate</th>\n",
       "      <th>dst_host_srv_diff_host_rate</th>\n",
       "      <th>dst_host_serror_rate</th>\n",
       "      <th>dst_host_srv_serror_rate</th>\n",
       "      <th>dst_host_rerror_rate</th>\n",
       "      <th>dst_host_srv_rerror_rate</th>\n",
       "      <th>outcome</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>SF</td>\n",
       "      <td>181</td>\n",
       "      <td>5450</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>9</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.11</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>normal.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>SF</td>\n",
       "      <td>239</td>\n",
       "      <td>486</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>19</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.05</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>normal.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>SF</td>\n",
       "      <td>235</td>\n",
       "      <td>1337</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>29</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>normal.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>SF</td>\n",
       "      <td>219</td>\n",
       "      <td>1337</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>39</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>normal.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>SF</td>\n",
       "      <td>217</td>\n",
       "      <td>2032</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>49</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.02</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>normal.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 42 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   duration protocol_type service flag  src_bytes  dst_bytes  land  \\\n",
       "0         0           tcp    http   SF        181       5450     0   \n",
       "1         0           tcp    http   SF        239        486     0   \n",
       "2         0           tcp    http   SF        235       1337     0   \n",
       "3         0           tcp    http   SF        219       1337     0   \n",
       "4         0           tcp    http   SF        217       2032     0   \n",
       "\n",
       "   wrong_fragment  urgent  hot  ...  dst_host_srv_count  \\\n",
       "0               0       0    0  ...                   9   \n",
       "1               0       0    0  ...                  19   \n",
       "2               0       0    0  ...                  29   \n",
       "3               0       0    0  ...                  39   \n",
       "4               0       0    0  ...                  49   \n",
       "\n",
       "   dst_host_same_srv_rate  dst_host_diff_srv_rate  \\\n",
       "0                     1.0                     0.0   \n",
       "1                     1.0                     0.0   \n",
       "2                     1.0                     0.0   \n",
       "3                     1.0                     0.0   \n",
       "4                     1.0                     0.0   \n",
       "\n",
       "   dst_host_same_src_port_rate  dst_host_srv_diff_host_rate  \\\n",
       "0                         0.11                          0.0   \n",
       "1                         0.05                          0.0   \n",
       "2                         0.03                          0.0   \n",
       "3                         0.03                          0.0   \n",
       "4                         0.02                          0.0   \n",
       "\n",
       "   dst_host_serror_rate  dst_host_srv_serror_rate  dst_host_rerror_rate  \\\n",
       "0                   0.0                       0.0                   0.0   \n",
       "1                   0.0                       0.0                   0.0   \n",
       "2                   0.0                       0.0                   0.0   \n",
       "3                   0.0                       0.0                   0.0   \n",
       "4                   0.0                       0.0                   0.0   \n",
       "\n",
       "   dst_host_srv_rerror_rate  outcome  \n",
       "0                       0.0  normal.  \n",
       "1                       0.0  normal.  \n",
       "2                       0.0  normal.  \n",
       "3                       0.0  normal.  \n",
       "4                       0.0  normal.  \n",
       "\n",
       "[5 rows x 42 columns]"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "from tensorflow.keras.utils import get_file\n",
    "\n",
    "try:\n",
    "    path = get_file('kddcup.data_10_percent.gz', origin='http://kdd.ics.uci.edu/databases/kddcup99/kddcup.data_10_percent.gz')\n",
    "except:\n",
    "    print('Error downloading')\n",
    "    raise\n",
    "    \n",
    "print(path) \n",
    "\n",
    "# This file is a CSV, just no CSV extension or headers\n",
    "# Download from: http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html\n",
    "df = pd.read_csv(path, header=None)\n",
    "\n",
    "print(\"Read {} rows.\".format(len(df)))\n",
    "# df = df.sample(frac=0.1, replace=False) # Uncomment this line to sample only 10% of the dataset\n",
    "df.dropna(inplace=True,axis=1) # For now, just drop NA's (rows with missing values)\n",
    "\n",
    "# The CSV file has no column heads, so add them\n",
    "df.columns = [\n",
    "    'duration',\n",
    "    'protocol_type',\n",
    "    'service',\n",
    "    'flag',\n",
    "    'src_bytes',\n",
    "    'dst_bytes',\n",
    "    'land',\n",
    "    'wrong_fragment',\n",
    "    'urgent',\n",
    "    'hot',\n",
    "    'num_failed_logins',\n",
    "    'logged_in',\n",
    "    'num_compromised',\n",
    "    'root_shell',\n",
    "    'su_attempted',\n",
    "    'num_root',\n",
    "    'num_file_creations',\n",
    "    'num_shells',\n",
    "    'num_access_files',\n",
    "    'num_outbound_cmds',\n",
    "    'is_host_login',\n",
    "    'is_guest_login',\n",
    "    'count',\n",
    "    'srv_count',\n",
    "    'serror_rate',\n",
    "    'srv_serror_rate',\n",
    "    'rerror_rate',\n",
    "    'srv_rerror_rate',\n",
    "    'same_srv_rate',\n",
    "    'diff_srv_rate',\n",
    "    'srv_diff_host_rate',\n",
    "    'dst_host_count',\n",
    "    'dst_host_srv_count',\n",
    "    'dst_host_same_srv_rate',\n",
    "    'dst_host_diff_srv_rate',\n",
    "    'dst_host_same_src_port_rate',\n",
    "    'dst_host_srv_diff_host_rate',\n",
    "    'dst_host_serror_rate',\n",
    "    'dst_host_srv_serror_rate',\n",
    "    'dst_host_rerror_rate',\n",
    "    'dst_host_srv_rerror_rate',\n",
    "    'outcome'\n",
    "]\n",
    "\n",
    "# display 5 rows\n",
    "pd.set_option('display.max_columns', 7)\n",
    "pd.set_option('display.max_rows', 5)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The KDD99 dataset contains many columns that define the network state over time intervals during which a cyber attack might have taken place.  The column labeled \"outcome\" specifies either \"normal,\" indicating no attack, or the type of attack performed.  The following code displays the counts for each type of attack, as well as \"normal\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "outcome\n",
       "back.                 2203\n",
       "buffer_overflow.        30\n",
       "ftp_write.               8\n",
       "guess_passwd.           53\n",
       "imap.                   12\n",
       "ipsweep.              1247\n",
       "land.                   21\n",
       "loadmodule.              9\n",
       "multihop.                7\n",
       "neptune.            107201\n",
       "nmap.                  231\n",
       "normal.              97278\n",
       "perl.                    3\n",
       "phf.                     4\n",
       "pod.                   264\n",
       "portsweep.            1040\n",
       "rootkit.                10\n",
       "satan.                1589\n",
       "smurf.              280790\n",
       "spy.                     2\n",
       "teardrop.              979\n",
       "warezclient.          1020\n",
       "warezmaster.            20\n",
       "Name: outcome, dtype: int64"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.groupby('outcome')['outcome'].count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocessing \n",
    "\n",
    "Before we can feed the KDD99 data into the neural network we must perform some preprocessing.  We provide the following two functions to assist with preprocessing. The first function converts numeric columns into Z-Scores.  The second function replaces categorical values with dummy variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Encode a numeric column as zscores\n",
    "def encode_numeric_zscore(df, name, mean=None, sd=None):\n",
    "    if mean is None:\n",
    "        mean = df[name].mean()\n",
    "\n",
    "    if sd is None:\n",
    "        sd = df[name].std()\n",
    "\n",
    "    df[name] = (df[name] - mean) / sd\n",
    "    \n",
    "# Encode text values to dummy variables(i.e. [1,0,0],[0,1,0],[0,0,1] for red,green,blue)\n",
    "def encode_text_dummy(df, name):\n",
    "    dummies = pd.get_dummies(df[name])\n",
    "    for x in dummies.columns:\n",
    "        dummy_name = f\"{name}-{x}\"\n",
    "        df[dummy_name] = dummies[x]\n",
    "    df.drop(name, axis=1, inplace=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now use these functions to preprocess each of the columns.  Once the program preprocesses the data we display the results. This code converts all numeric columns to Z-Scores and all textual columns to dummy variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>duration</th>\n",
       "      <th>src_bytes</th>\n",
       "      <th>dst_bytes</th>\n",
       "      <th>wrong_fragment</th>\n",
       "      <th>urgent</th>\n",
       "      <th>hot</th>\n",
       "      <th>num_failed_logins</th>\n",
       "      <th>num_compromised</th>\n",
       "      <th>root_shell</th>\n",
       "      <th>su_attempted</th>\n",
       "      <th>...</th>\n",
       "      <th>flag-S3</th>\n",
       "      <th>flag-SF</th>\n",
       "      <th>flag-SH</th>\n",
       "      <th>land-0</th>\n",
       "      <th>land-1</th>\n",
       "      <th>logged_in-0</th>\n",
       "      <th>logged_in-1</th>\n",
       "      <th>is_host_login-0</th>\n",
       "      <th>is_guest_login-0</th>\n",
       "      <th>is_guest_login-1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.067792</td>\n",
       "      <td>-0.002879</td>\n",
       "      <td>0.138664</td>\n",
       "      <td>-0.04772</td>\n",
       "      <td>-0.002571</td>\n",
       "      <td>-0.044136</td>\n",
       "      <td>-0.009782</td>\n",
       "      <td>-0.005679</td>\n",
       "      <td>-0.010552</td>\n",
       "      <td>-0.004676</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-0.067792</td>\n",
       "      <td>-0.002820</td>\n",
       "      <td>-0.011578</td>\n",
       "      <td>-0.04772</td>\n",
       "      <td>-0.002571</td>\n",
       "      <td>-0.044136</td>\n",
       "      <td>-0.009782</td>\n",
       "      <td>-0.005679</td>\n",
       "      <td>-0.010552</td>\n",
       "      <td>-0.004676</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.067792</td>\n",
       "      <td>-0.002824</td>\n",
       "      <td>0.014179</td>\n",
       "      <td>-0.04772</td>\n",
       "      <td>-0.002571</td>\n",
       "      <td>-0.044136</td>\n",
       "      <td>-0.009782</td>\n",
       "      <td>-0.005679</td>\n",
       "      <td>-0.010552</td>\n",
       "      <td>-0.004676</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.067792</td>\n",
       "      <td>-0.002840</td>\n",
       "      <td>0.014179</td>\n",
       "      <td>-0.04772</td>\n",
       "      <td>-0.002571</td>\n",
       "      <td>-0.044136</td>\n",
       "      <td>-0.009782</td>\n",
       "      <td>-0.005679</td>\n",
       "      <td>-0.010552</td>\n",
       "      <td>-0.004676</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-0.067792</td>\n",
       "      <td>-0.002842</td>\n",
       "      <td>0.035214</td>\n",
       "      <td>-0.04772</td>\n",
       "      <td>-0.002571</td>\n",
       "      <td>-0.044136</td>\n",
       "      <td>-0.009782</td>\n",
       "      <td>-0.005679</td>\n",
       "      <td>-0.010552</td>\n",
       "      <td>-0.004676</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 121 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   duration  src_bytes  dst_bytes  wrong_fragment    urgent       hot  \\\n",
       "0 -0.067792  -0.002879   0.138664        -0.04772 -0.002571 -0.044136   \n",
       "1 -0.067792  -0.002820  -0.011578        -0.04772 -0.002571 -0.044136   \n",
       "2 -0.067792  -0.002824   0.014179        -0.04772 -0.002571 -0.044136   \n",
       "3 -0.067792  -0.002840   0.014179        -0.04772 -0.002571 -0.044136   \n",
       "4 -0.067792  -0.002842   0.035214        -0.04772 -0.002571 -0.044136   \n",
       "\n",
       "   num_failed_logins  num_compromised  root_shell  su_attempted  ...  flag-S3  \\\n",
       "0          -0.009782        -0.005679   -0.010552     -0.004676  ...        0   \n",
       "1          -0.009782        -0.005679   -0.010552     -0.004676  ...        0   \n",
       "2          -0.009782        -0.005679   -0.010552     -0.004676  ...        0   \n",
       "3          -0.009782        -0.005679   -0.010552     -0.004676  ...        0   \n",
       "4          -0.009782        -0.005679   -0.010552     -0.004676  ...        0   \n",
       "\n",
       "   flag-SF  flag-SH  land-0  land-1  logged_in-0  logged_in-1  \\\n",
       "0        1        0       1       0            0            1   \n",
       "1        1        0       1       0            0            1   \n",
       "2        1        0       1       0            0            1   \n",
       "3        1        0       1       0            0            1   \n",
       "4        1        0       1       0            0            1   \n",
       "\n",
       "   is_host_login-0  is_guest_login-0  is_guest_login-1  \n",
       "0                1                 1                 0  \n",
       "1                1                 1                 0  \n",
       "2                1                 1                 0  \n",
       "3                1                 1                 0  \n",
       "4                1                 1                 0  \n",
       "\n",
       "[5 rows x 121 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now encode the feature vector\n",
    "\n",
    "encode_numeric_zscore(df, 'duration')\n",
    "encode_text_dummy(df, 'protocol_type')\n",
    "encode_text_dummy(df, 'service')\n",
    "encode_text_dummy(df, 'flag')\n",
    "encode_numeric_zscore(df, 'src_bytes')\n",
    "encode_numeric_zscore(df, 'dst_bytes')\n",
    "encode_text_dummy(df, 'land')\n",
    "encode_numeric_zscore(df, 'wrong_fragment')\n",
    "encode_numeric_zscore(df, 'urgent')\n",
    "encode_numeric_zscore(df, 'hot')\n",
    "encode_numeric_zscore(df, 'num_failed_logins')\n",
    "encode_text_dummy(df, 'logged_in')\n",
    "encode_numeric_zscore(df, 'num_compromised')\n",
    "encode_numeric_zscore(df, 'root_shell')\n",
    "encode_numeric_zscore(df, 'su_attempted')\n",
    "encode_numeric_zscore(df, 'num_root')\n",
    "encode_numeric_zscore(df, 'num_file_creations')\n",
    "encode_numeric_zscore(df, 'num_shells')\n",
    "encode_numeric_zscore(df, 'num_access_files')\n",
    "encode_numeric_zscore(df, 'num_outbound_cmds')\n",
    "encode_text_dummy(df, 'is_host_login')\n",
    "encode_text_dummy(df, 'is_guest_login')\n",
    "encode_numeric_zscore(df, 'count')\n",
    "encode_numeric_zscore(df, 'srv_count')\n",
    "encode_numeric_zscore(df, 'serror_rate')\n",
    "encode_numeric_zscore(df, 'srv_serror_rate')\n",
    "encode_numeric_zscore(df, 'rerror_rate')\n",
    "encode_numeric_zscore(df, 'srv_rerror_rate')\n",
    "encode_numeric_zscore(df, 'same_srv_rate')\n",
    "encode_numeric_zscore(df, 'diff_srv_rate')\n",
    "encode_numeric_zscore(df, 'srv_diff_host_rate')\n",
    "encode_numeric_zscore(df, 'dst_host_count')\n",
    "encode_numeric_zscore(df, 'dst_host_srv_count')\n",
    "encode_numeric_zscore(df, 'dst_host_same_srv_rate')\n",
    "encode_numeric_zscore(df, 'dst_host_diff_srv_rate')\n",
    "encode_numeric_zscore(df, 'dst_host_same_src_port_rate')\n",
    "encode_numeric_zscore(df, 'dst_host_srv_diff_host_rate')\n",
    "encode_numeric_zscore(df, 'dst_host_serror_rate')\n",
    "encode_numeric_zscore(df, 'dst_host_srv_serror_rate')\n",
    "encode_numeric_zscore(df, 'dst_host_rerror_rate')\n",
    "encode_numeric_zscore(df, 'dst_host_srv_rerror_rate')\n",
    "\n",
    "# display 5 rows\n",
    "\n",
    "df.dropna(inplace=True,axis=1)\n",
    "df[0:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To perform anomaly detection, we divide the data into two groups \"normal\" and the various attacks. The following code divides the data into two dataframes and displays each of these two groups' sizes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Normal count: 97278\n",
      "Attack count: 396743\n"
     ]
    }
   ],
   "source": [
    "normal_mask = df['outcome']=='normal.'\n",
    "attack_mask = df['outcome']!='normal.'\n",
    "\n",
    "df.drop('outcome',axis=1,inplace=True)\n",
    "\n",
    "df_normal = df[normal_mask]\n",
    "df_attack = df[attack_mask]\n",
    "\n",
    "print(f\"Normal count: {len(df_normal)}\")\n",
    "print(f\"Attack count: {len(df_attack)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we convert these two dataframes into Numpy arrays. Keras requires this format for data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is the numeric feature vector, as it goes to the neural net\n",
    "x_normal = df_normal.values\n",
    "x_attack = df_attack.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training the Autoencoder\n",
    "\n",
    "It is important to note that we are not using the outcome column as a label to predict.  This anomaly detection is unsupervised; there is no target (y) value to predict. We will train an autoencoder on the normal data and see how well it can detect that the data not flagged as \"normal\" represents an anomaly. \n",
    "\n",
    "Next, we split the normal data into a 25% test set and a 75% train set.  The program will use the test data to facilitate early stopping."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "x_normal_train, x_normal_test = train_test_split(\n",
    "    x_normal, test_size=0.25, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We display the size of the train and test sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Normal train count: 72958\n",
      "Normal test count: 24320\n"
     ]
    }
   ],
   "source": [
    "print(f\"Normal train count: {len(x_normal_train)}\")\n",
    "print(f\"Normal test count: {len(x_normal_test)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now ready to train the autoencoder on the normal data. The autoencoder will learn to compress the data to a vector of just three numbers. The autoencoder should be able to also decompress with reasonable accuracy. As is typical for autoencoders, we are merely training the neural network to produce the same output values as were fed to the input layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 72958 samples\n",
      "Epoch 1/100\n",
      "72958/72958 [==============================] - 7s 94us/sample - loss: 0.3729\n",
      "Epoch 2/100\n",
      "72958/72958 [==============================] - 6s 79us/sample - loss: 0.3321\n",
      "Epoch 3/100\n",
      "72958/72958 [==============================] - 6s 82us/sample - loss: 0.2998\n",
      "Epoch 4/100\n",
      "72958/72958 [==============================] - 6s 80us/sample - loss: 0.2716\n",
      "Epoch 5/100\n",
      "72958/72958 [==============================] - 6s 80us/sample - loss: 0.2562\n",
      "Epoch 6/100\n",
      "72958/72958 [==============================] - 6s 81us/sample - loss: 0.2473\n",
      "Epoch 7/100\n",
      "72958/72958 [==============================] - 6s 81us/sample - loss: 0.2334\n",
      "Epoch 8/100\n",
      "72958/72958 [==============================] - 6s 83us/sample - loss: 0.2354\n",
      "Epoch 9/100\n",
      "72958/72958 [==============================] - 6s 82us/sample - loss: 0.2181\n",
      "Epoch 10/100\n",
      "72958/72958 [==============================] - 6s 83us/sample - loss: 0.2204\n",
      "Epoch 11/100\n",
      "72958/72958 [==============================] - 6s 83us/sample - loss: 0.2175\n",
      "Epoch 12/100\n",
      "72958/72958 [==============================] - 6s 83us/sample - loss: 0.2085\n",
      "Epoch 13/100\n",
      "72958/72958 [==============================] - 6s 86us/sample - loss: 0.2073\n",
      "Epoch 14/100\n",
      "72958/72958 [==============================] - 6s 84us/sample - loss: 0.2131\n",
      "Epoch 15/100\n",
      "72958/72958 [==============================] - 6s 84us/sample - loss: 0.2084\n",
      "Epoch 16/100\n",
      "72958/72958 [==============================] - 6s 83us/sample - loss: 0.2110\n",
      "Epoch 17/100\n",
      "72958/72958 [==============================] - 6s 82us/sample - loss: 0.2058\n",
      "Epoch 18/100\n",
      "72958/72958 [==============================] - 6s 84us/sample - loss: 0.2002\n",
      "Epoch 19/100\n",
      "72958/72958 [==============================] - 6s 83us/sample - loss: 0.1991\n",
      "Epoch 20/100\n",
      "72958/72958 [==============================] - 7s 100us/sample - loss: 0.2053\n",
      "Epoch 21/100\n",
      "72958/72958 [==============================] - 6s 83us/sample - loss: 0.1943\n",
      "Epoch 22/100\n",
      "72958/72958 [==============================] - 6s 81us/sample - loss: 0.1935\n",
      "Epoch 23/100\n",
      "72958/72958 [==============================] - 6s 80us/sample - loss: 0.1821\n",
      "Epoch 24/100\n",
      "72958/72958 [==============================] - 6s 79us/sample - loss: 0.1808\n",
      "Epoch 25/100\n",
      "72958/72958 [==============================] - 6s 78us/sample - loss: 0.1761\n",
      "Epoch 26/100\n",
      "72958/72958 [==============================] - 6s 79us/sample - loss: 0.1703\n",
      "Epoch 27/100\n",
      "72958/72958 [==============================] - 6s 80us/sample - loss: 0.1760\n",
      "Epoch 28/100\n",
      "72958/72958 [==============================] - 6s 80us/sample - loss: 0.1646\n",
      "Epoch 29/100\n",
      "72958/72958 [==============================] - 6s 79us/sample - loss: 0.1663\n",
      "Epoch 30/100\n",
      "72958/72958 [==============================] - 6s 79us/sample - loss: 0.1594\n",
      "Epoch 31/100\n",
      "72958/72958 [==============================] - 6s 79us/sample - loss: 0.1613\n",
      "Epoch 32/100\n",
      "72958/72958 [==============================] - 6s 80us/sample - loss: 0.1568\n",
      "Epoch 33/100\n",
      "72958/72958 [==============================] - 6s 80us/sample - loss: 0.1685\n",
      "Epoch 34/100\n",
      "72958/72958 [==============================] - 6s 79us/sample - loss: 0.1531\n",
      "Epoch 35/100\n",
      "72958/72958 [==============================] - 6s 80us/sample - loss: 0.1526\n",
      "Epoch 36/100\n",
      "72958/72958 [==============================] - 6s 85us/sample - loss: 0.1552\n",
      "Epoch 37/100\n",
      "72958/72958 [==============================] - 6s 86us/sample - loss: 0.1565\n",
      "Epoch 38/100\n",
      "72958/72958 [==============================] - 6s 88us/sample - loss: 0.1479\n",
      "Epoch 39/100\n",
      "72958/72958 [==============================] - 6s 83us/sample - loss: 0.1494\n",
      "Epoch 40/100\n",
      "72958/72958 [==============================] - 6s 85us/sample - loss: 0.1452\n",
      "Epoch 41/100\n",
      "72958/72958 [==============================] - 6s 84us/sample - loss: 0.1455\n",
      "Epoch 42/100\n",
      "72958/72958 [==============================] - 6s 83us/sample - loss: 0.1425\n",
      "Epoch 43/100\n",
      "72958/72958 [==============================] - 6s 83us/sample - loss: 0.1400\n",
      "Epoch 44/100\n",
      "72958/72958 [==============================] - 6s 81us/sample - loss: 0.1394\n",
      "Epoch 45/100\n",
      "72958/72958 [==============================] - 6s 82us/sample - loss: 0.1406\n",
      "Epoch 46/100\n",
      "72958/72958 [==============================] - 6s 82us/sample - loss: 0.1441\n",
      "Epoch 47/100\n",
      "72958/72958 [==============================] - 6s 83us/sample - loss: 0.1359\n",
      "Epoch 48/100\n",
      "72958/72958 [==============================] - 6s 84us/sample - loss: 0.1363\n",
      "Epoch 49/100\n",
      "72958/72958 [==============================] - 6s 82us/sample - loss: 0.1351\n",
      "Epoch 50/100\n",
      "72958/72958 [==============================] - 6s 82us/sample - loss: 0.1344\n",
      "Epoch 51/100\n",
      "72958/72958 [==============================] - 6s 83us/sample - loss: 0.1330\n",
      "Epoch 52/100\n",
      "72958/72958 [==============================] - 6s 84us/sample - loss: 0.1367\n",
      "Epoch 53/100\n",
      "72958/72958 [==============================] - 6s 82us/sample - loss: 0.1416\n",
      "Epoch 54/100\n",
      "72958/72958 [==============================] - 6s 81us/sample - loss: 0.1332\n",
      "Epoch 55/100\n",
      "72958/72958 [==============================] - 6s 81us/sample - loss: 0.1338\n",
      "Epoch 56/100\n",
      "72958/72958 [==============================] - 6s 81us/sample - loss: 0.1317\n",
      "Epoch 57/100\n",
      "72958/72958 [==============================] - 6s 81us/sample - loss: 0.1333\n",
      "Epoch 58/100\n",
      "72958/72958 [==============================] - 6s 82us/sample - loss: 0.1395\n",
      "Epoch 59/100\n",
      "72958/72958 [==============================] - 6s 81us/sample - loss: 0.1342\n",
      "Epoch 60/100\n",
      "72958/72958 [==============================] - 6s 81us/sample - loss: 0.1311\n",
      "Epoch 61/100\n",
      "72958/72958 [==============================] - 6s 81us/sample - loss: 0.1334\n",
      "Epoch 62/100\n",
      "72958/72958 [==============================] - 6s 81us/sample - loss: 0.1299\n",
      "Epoch 63/100\n",
      "72958/72958 [==============================] - 6s 84us/sample - loss: 0.1315\n",
      "Epoch 64/100\n",
      "72958/72958 [==============================] - 6s 81us/sample - loss: 0.1288\n",
      "Epoch 65/100\n",
      "72958/72958 [==============================] - 6s 81us/sample - loss: 0.1308\n",
      "Epoch 66/100\n",
      "72958/72958 [==============================] - 6s 81us/sample - loss: 0.1369\n",
      "Epoch 67/100\n",
      "72958/72958 [==============================] - 6s 81us/sample - loss: 0.1337\n",
      "Epoch 68/100\n",
      "72958/72958 [==============================] - 6s 82us/sample - loss: 0.1359\n",
      "Epoch 69/100\n",
      "72958/72958 [==============================] - 6s 81us/sample - loss: 0.1314\n",
      "Epoch 70/100\n",
      "72958/72958 [==============================] - 6s 81us/sample - loss: 0.1296\n",
      "Epoch 71/100\n",
      "72958/72958 [==============================] - 6s 81us/sample - loss: 0.1290\n",
      "Epoch 72/100\n",
      "72958/72958 [==============================] - 6s 81us/sample - loss: 0.1307\n",
      "Epoch 73/100\n",
      "72958/72958 [==============================] - 6s 82us/sample - loss: 0.1284\n",
      "Epoch 74/100\n",
      "72958/72958 [==============================] - 6s 81us/sample - loss: 0.1299\n",
      "Epoch 75/100\n",
      "72958/72958 [==============================] - 6s 81us/sample - loss: 0.1296\n",
      "Epoch 76/100\n",
      "72958/72958 [==============================] - 6s 82us/sample - loss: 0.1278\n",
      "Epoch 77/100\n",
      "72958/72958 [==============================] - 6s 81us/sample - loss: 0.1252\n",
      "Epoch 78/100\n",
      "72958/72958 [==============================] - 6s 82us/sample - loss: 0.1260\n",
      "Epoch 79/100\n",
      "72958/72958 [==============================] - 6s 81us/sample - loss: 0.1267\n",
      "Epoch 80/100\n",
      "72958/72958 [==============================] - 6s 81us/sample - loss: 0.1272\n",
      "Epoch 81/100\n",
      "72958/72958 [==============================] - 6s 81us/sample - loss: 0.1253\n",
      "Epoch 82/100\n",
      "72958/72958 [==============================] - 6s 82us/sample - loss: 0.1261\n",
      "Epoch 83/100\n",
      "72958/72958 [==============================] - 6s 83us/sample - loss: 0.1260\n",
      "Epoch 84/100\n",
      "72958/72958 [==============================] - 6s 81us/sample - loss: 0.1307\n",
      "Epoch 85/100\n",
      "72958/72958 [==============================] - 6s 81us/sample - loss: 0.1265\n",
      "Epoch 86/100\n",
      "72958/72958 [==============================] - 6s 81us/sample - loss: 0.1268\n",
      "Epoch 87/100\n",
      "72958/72958 [==============================] - 6s 82us/sample - loss: 0.1259\n",
      "Epoch 88/100\n",
      "72958/72958 [==============================] - 6s 83us/sample - loss: 0.1302\n",
      "Epoch 89/100\n",
      "72958/72958 [==============================] - 6s 81us/sample - loss: 0.1229\n",
      "Epoch 90/100\n",
      "72958/72958 [==============================] - 6s 83us/sample - loss: 0.1243\n",
      "Epoch 91/100\n",
      "72958/72958 [==============================] - 6s 76us/sample - loss: 0.1278\n",
      "Epoch 92/100\n",
      "72958/72958 [==============================] - 6s 76us/sample - loss: 0.1238\n",
      "Epoch 93/100\n",
      "72958/72958 [==============================] - 6s 76us/sample - loss: 0.1230\n",
      "Epoch 94/100\n",
      "72958/72958 [==============================] - 6s 76us/sample - loss: 0.1226\n",
      "Epoch 95/100\n",
      "72958/72958 [==============================] - 6s 75us/sample - loss: 0.1279\n",
      "Epoch 96/100\n",
      "72958/72958 [==============================] - 6s 76us/sample - loss: 0.1224\n",
      "Epoch 97/100\n",
      "72958/72958 [==============================] - 6s 76us/sample - loss: 0.1216\n",
      "Epoch 98/100\n",
      "72958/72958 [==============================] - 6s 76us/sample - loss: 0.1219\n",
      "Epoch 99/100\n",
      "72958/72958 [==============================] - 6s 77us/sample - loss: 0.1269\n",
      "Epoch 100/100\n",
      "72958/72958 [==============================] - 6s 76us/sample - loss: 0.1215\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x645a36e10>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn import metrics\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from IPython.display import display, HTML \n",
    "import tensorflow as tf\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(25, input_dim=x_normal.shape[1], activation='relu'))\n",
    "model.add(Dense(3, activation='relu')) # size to compress to\n",
    "model.add(Dense(25, activation='relu'))\n",
    "model.add(Dense(x_normal.shape[1])) # Multiple output neurons\n",
    "model.compile(loss='mean_squared_error', optimizer='adam')\n",
    "model.fit(x_normal_train,x_normal_train,verbose=1,epochs=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Detecting an Anomaly\n",
    "\n",
    "We are now ready to see if the abnormal data registers as an anomaly. The first two scores show the in-sample and out of sample RMSE errors. Both of these two scores are relatively low at around 0.33 because they resulted from normal data. The much higher 0.76 error occurred from the abnormal data. The autoencoder is not as capable of encoding data that represents an attack. This higher error indicates an anomaly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Out of Sample Score (RMSE): 0.33287804666722703\n",
      "Insample Normal Score (RMSE): 0.34665772844552595\n",
      "Attack Underway Score (RMSE): 0.7674410983524335\n"
     ]
    }
   ],
   "source": [
    "pred = model.predict(x_normal_test)\n",
    "score1 = np.sqrt(metrics.mean_squared_error(pred,x_normal_test))\n",
    "pred = model.predict(x_normal)\n",
    "score2 = np.sqrt(metrics.mean_squared_error(pred,x_normal))\n",
    "pred = model.predict(x_attack)\n",
    "score3 = np.sqrt(metrics.mean_squared_error(pred,x_attack))\n",
    "print(f\"Out of Sample Normal Score (RMSE): {score1}\")\n",
    "print(f\"Insample Normal Score (RMSE): {score2}\")\n",
    "print(f\"Attack Underway Score (RMSE): {score3}\")"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.7 (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}