t81_558_class_08_3_keras_hyperparameters.ipynb

{"cells": [{"cell_type": "markdown", "metadata": {}, "source": ["<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_08_3_keras_hyperparameters.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"]}, {"cell_type": "markdown", "metadata": {}, "source": ["# T81-558: Applications of Deep Neural Networks\n", "**Module 8: Kaggle Data Sets**\n", "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n", "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."]}, {"cell_type": "markdown", "metadata": {}, "source": ["# Module 8 Material\n", "\n", "* Part 8.1: Introduction to Kaggle [[Video]](https://www.youtube.com/watch?v=v4lJBhdCuCU&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_08_1_kaggle_intro.ipynb)\n", "* Part 8.2: Building Ensembles with Scikit-Learn and Keras [[Video]](https://www.youtube.com/watch?v=LQ-9ZRBLasw&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_08_2_keras_ensembles.ipynb)\n", "* **Part 8.3: How Should you Architect Your Keras Neural Network: Hyperparameters** [[Video]](https://www.youtube.com/watch?v=1q9klwSoUQw&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_08_3_keras_hyperparameters.ipynb)\n", "* Part 8.4: Bayesian Hyperparameter Optimization for Keras [[Video]](https://www.youtube.com/watch?v=sXdxyUCCm8s&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_08_4_bayesian_hyperparameter_opt.ipynb)\n", "* Part 8.5: Current Semester's Kaggle [[Video]](https://www.youtube.com/watch?v=PHQt0aUasRg&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_08_5_kaggle_project.ipynb)\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["# Google CoLab Instructions\n", "\n", "The following code ensures that Google CoLab is running the correct version of TensorFlow."]}, {"cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": ["# Startup CoLab\n", "try:\n", "    %tensorflow_version 2.x\n", "    COLAB = True\n", "    print(\"Note: using Google CoLab\")\n", "except:\n", "    print(\"Note: not using Google CoLab\")\n", "    COLAB = False\n", "\n", "\n", "# Nicely formatted time string\n", "def hms_string(sec_elapsed):\n", "    h = int(sec_elapsed / (60 * 60))\n", "    m = int((sec_elapsed % (60 * 60)) / 60)\n", "    s = sec_elapsed % 60\n", "    return \"{}:{:>02}:{:>05.2f}\".format(h, m, s)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["# Part 8.3: How Should you Architect Your Keras Neural Network: Hyperparameters\n", "\n", "* [Guide to choosing Hyperparameters for your Neural Networks](https://towardsdatascience.com/guide-to-choosing-hyperparameters-for-your-neural-networks-38244e87dafe)\n", "\n", "### Number of Hidden Layers and Neuron Counts\n", "\n", "* [Keras Layers](https://keras.io/layers/core/)\n", "\n", "Layer types and when to use them:\n", "\n", "* **Activation** - Layer that simply adds an activation function, the activation function can also be specified as part of a Dense (or other) layer type.\n", "* **ActivityRegularization** Used to add L1/L2 regularization outside of a layer.  L1 and L2 can also be specified as part of a Dense (or other) layer type.\n", "* **Dense** - The original neural network layer type.  Every neuron is connected to the next layer.  The input vector is one-dimensional and placing certain inputs next to each other does not have an effect. \n", "* **Dropout** - Dropout consists in randomly setting a fraction rate of input units to 0 at each update during training time, which helps prevent overfitting.  Dropout only occurs during training.\n", "* **Flatten** - Flattens the input to 1D. Does not affect the batch size.\n", "* **Input** - A Keras tensor is a tensor object from the underlying backend (Theano, TensorFlow or CNTK), which we augment with certain attributes that allow us to build a Keras model just by knowing the inputs and outputs of the model.\n", "* **Lambda** - Wraps arbitrary expression as a Layer object.\n", "* **Masking** - Masks a sequence by using a mask value to skip timesteps.\n", "* **Permute** - Permutes the dimensions of the input according to a given pattern. Useful for e.g. connecting RNNs and convnets together.\n", "* **RepeatVector** - Repeats the input n times.\n", "* **Reshape** - Similar to Numpy reshapes.\n", "* **SpatialDropout1D** - This version performs the same function as Dropout, however it drops entire 1D feature maps instead of individual elements. \n", "* **SpatialDropout2D** - This version performs the same function as Dropout, however it drops entire 2D feature maps instead of individual elements\n", "* **SpatialDropout3D** - This version performs the same function as Dropout, however it drops entire 3D feature maps instead of individual elements. \n", "\n", "\n", "### Activation Functions\n", "\n", "* [Keras Activation Functions](https://keras.io/activations/)\n", "* [Activation Function Cheat Sheets](https://ml-cheatsheet.readthedocs.io/en/latest/activation_functions.html)\n", "\n", "As follows:\n", "\n", "* **softmax** - Used for multi-class classification.  Ensures all output neurons behave as probabilities and sum to 1.0.\n", "* **elu** - Exponential linear unit.  Exponential Linear Unit or its widely known name ELU is a function that tend to converge cost to zero faster and produce more accurate results.  Can produce negative outputs.\n", "* **selu** - Scaled Exponential Linear Unit (SELU), essentially **elu** multiplied by a scaling constant.\n", "* **softplus** - Softplus activation function. $log(exp(x) + 1)$  [Introduced](https://papers.nips.cc/paper/1920-incorporating-second-order-functional-knowledge-for-better-option-pricing.pdf) in 2001.\n", "* **softsign** Softsign activation function. $x / (abs(x) + 1)$ Similar to tanh, but not widely used.\n", "* **relu** - Very popular neural network activation function.  Used for hidden layers, cannot output negative values.  No trainable parameters.\n", "* **tanh** Classic neural network activation function, though often replaced by relu family on modern networks.\n", "* **sigmoid** - Classic neural network activation.  Often used on output layer of a binary classifier.\n", "* **hard_sigmoid** - Less computationally expensive variant of sigmoid.\n", "* **exponential** - Exponential (base e) activation function.\n", "* **linear** - Pass through activation function. Usually used on the output layer of a regression neural network.\n", "\n", "### Advanced Activation Functions\n", "\n", "* [Keras Advanced Activation Functions](https://keras.io/layers/advanced-activations/)\n", "\n", "The advanced activation functions contain parameters that are trained during neural network fitting. As follows:\n", "\n", "* **LeakyReLU** - Leaky version of a Rectified Linear Unit. It allows a small gradient when the unit is not active, controlled by alpha hyperparameter.\n", "* **PReLU** - Parametric Rectified Linear Unit, learns the alpha hyperparameter. \n", "\n", "### Regularization: L1, L2, Dropout\n", "\n", "* [Keras Regularization](https://keras.io/regularizers/)\n", "* [Keras Dropout](https://keras.io/layers/core/)\n", "\n", "### Batch Normalization\n", "\n", "* [Keras Batch Normalization](https://keras.io/layers/normalization/)\n", "\n", "* Ioffe, S., & Szegedy, C. (2015). [Batch normalization: Accelerating deep network training by reducing internal covariate shift](https://arxiv.org/abs/1502.03167). *arXiv preprint arXiv:1502.03167*.\n", "\n", "Normalize the activations of the previous layer at each batch, i.e. applies a transformation that maintains the mean activation close to 0 and the activation standard deviation close to 1.  Can allow learning rate to be larger.\n", "\n", "\n", "### Training Parameters\n", "\n", "* [Keras Optimizers](https://keras.io/optimizers/)\n", "\n", "* **Batch Size** - Usually small, such as 32 or so.\n", "* **Learning Rate**  - Usually small, 1e-3 or so."]}, {"cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": ["import pandas as pd\n", "from scipy.stats import zscore\n", "\n", "# Read the data set\n", "df = pd.read_csv(\n", "    \"https://data.heatonresearch.com/data/t81-558/jh-simple-dataset.csv\",\n", "    na_values=['NA','?'])\n", "\n", "# Generate dummies for job\n", "df = pd.concat([df,pd.get_dummies(df['job'],prefix=\"job\")],axis=1)\n", "df.drop('job', axis=1, inplace=True)\n", "\n", "# Generate dummies for area\n", "df = pd.concat([df,pd.get_dummies(df['area'],prefix=\"area\")],axis=1)\n", "df.drop('area', axis=1, inplace=True)\n", "\n", "# Missing values for income\n", "med = df['income'].median()\n", "df['income'] = df['income'].fillna(med)\n", "\n", "# Standardize ranges\n", "df['income'] = zscore(df['income'])\n", "df['aspect'] = zscore(df['aspect'])\n", "df['save_rate'] = zscore(df['save_rate'])\n", "df['age'] = zscore(df['age'])\n", "df['subscriptions'] = zscore(df['subscriptions'])\n", "\n", "# Convert to numpy - Classification\n", "x_columns = df.columns.drop('product').drop('id')\n", "x = df[x_columns].values\n", "dummies = pd.get_dummies(df['product']) # Classification\n", "products = dummies.columns\n", "y = dummies.values"]}, {"cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["WARNING: Logging before flag parsing goes to stderr.\n", "W0819 15:28:03.170156  1808 training_utils.py:1210] When passing input data as arrays, do not specify `steps_per_epoch`/`steps` argument. Please use `batch_size` instead.\n", "W0819 15:28:03.440148  1808 deprecation.py:323] From C:\\Users\\jheaton\\AppData\\Local\\Continuum\\miniconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow_core\\python\\keras\\optimizer_v2\\optimizer_v2.py:466: BaseResourceVariable.constraint (from tensorflow.python.ops.resource_variable_ops) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Apply a constraint manually following the optimizer update step.\n", "W0819 15:28:26.512756  1808 training_utils.py:1210] When passing input data as arrays, do not specify `steps_per_epoch`/`steps` argument. Please use `batch_size` instead.\n", "W0819 15:28:26.589758  1808 training_utils.py:1210] When passing input data as arrays, do not specify `steps_per_epoch`/`steps` argument. Please use `batch_size` instead.\n", "W0819 15:28:37.886053  1808 training_utils.py:1210] When passing input data as arrays, do not specify `steps_per_epoch`/`steps` argument. Please use `batch_size` instead.\n"]}, {"name": "stdout", "output_type": "stream", "text": ["-0.7551467691035941\n"]}], "source": ["import pandas as pd\n", "import os\n", "import numpy as np\n", "import time\n", "import tensorflow.keras.initializers\n", "import statistics\n", "import tensorflow.keras\n", "from sklearn import metrics\n", "from sklearn.model_selection import StratifiedKFold\n", "from tensorflow.keras.models import Sequential\n", "from tensorflow.keras.layers import Dense, Activation, Dropout\n", "from tensorflow.keras import regularizers\n", "from tensorflow.keras.callbacks import EarlyStopping\n", "from sklearn.model_selection import StratifiedShuffleSplit\n", "from tensorflow.keras.layers import LeakyReLU,PReLU\n", "from tensorflow.keras.optimizers import Adam\n", "\n", "\n", "\n", "def evaluate_network(dropout,lr,neuronPct,neuronShrink):\n", "    SPLITS = 2\n", "\n", "    # Bootstrap\n", "    boot = StratifiedShuffleSplit(n_splits=SPLITS, test_size=0.1)\n", "\n", "    # Track progress\n", "    mean_benchmark = []\n", "    epochs_needed = []\n", "    num = 0\n", "    neuronCount = int(neuronPct * 5000)\n", "\n", "    # Loop through samples\n", "    for train, test in boot.split(x,df['product']):\n", "        start_time = time.time()\n", "        num+=1\n", "\n", "        # Split train and test\n", "        x_train = x[train]\n", "        y_train = y[train]\n", "        x_test = x[test]\n", "        y_test = y[test]\n", "\n", "        # Construct neural network\n", "        # kernel_initializer = tensorflow.keras.initializers.he_uniform(seed=None)\n", "        model = Sequential()\n", "        \n", "        layer = 0\n", "        while neuronCount>25 and layer<10:\n", "            #print(neuronCount)\n", "            if layer==0:\n", "                model.add(Dense(neuronCount, \n", "                    input_dim=x.shape[1], \n", "                    activation=PReLU())) \n", "            else:\n", "                model.add(Dense(neuronCount, activation=PReLU())) \n", "            model.add(Dropout(dropout))\n", "        \n", "            neuronCount = neuronCount * neuronShrink\n", "        \n", "        model.add(Dense(y.shape[1],activation='softmax')) # Output\n", "        model.compile(loss='categorical_crossentropy', optimizer=Adam(lr=lr))\n", "        monitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, \n", "            patience=100, verbose=0, mode='auto', restore_best_weights=True)\n", "\n", "        # Train on the bootstrap sample\n", "        model.fit(x_train,y_train,validation_data=(x_test,y_test),\n", "                  callbacks=[monitor],verbose=0,epochs=1000)\n", "        epochs = monitor.stopped_epoch\n", "        epochs_needed.append(epochs)\n", "\n", "        # Predict on the out of boot (validation)\n", "        pred = model.predict(x_test)\n", "\n", "        # Measure this bootstrap's log loss\n", "        y_compare = np.argmax(y_test,axis=1) # For log loss calculation\n", "        score = metrics.log_loss(y_compare, pred)\n", "        mean_benchmark.append(score)\n", "        m1 = statistics.mean(mean_benchmark)\n", "        m2 = statistics.mean(epochs_needed)\n", "        mdev = statistics.pstdev(mean_benchmark)\n", "\n", "        # Record this iteration\n", "        time_took = time.time() - start_time\n", "\n", "    tensorflow.keras.backend.clear_session()\n", "    return (-m1)\n", "\n", "print(evaluate_network(\n", "    dropout=0.2,\n", "    lr=1e-3,\n", "    neuronPct=0.2,\n", "    neuronShrink=0.2))\n"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": []}], "metadata": {"anaconda-cloud": {}, "kernelspec": {"display_name": "Python 3.6 (tensorflow)", "language": "python", "name": "tensorflow"}, "language_info": {"codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.5"}}, "nbformat": 4, "nbformat_minor": 4}