06. Classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classification\n",
    "Put simply, classification is the task of predicting a label for a given observation. For example: you are given certain physical descriptions of an animal, and your taks is to classify them as either a dog or a cat. Here, we will classify iris flowers.\n",
    "\n",
    "As we will see later, we will use different classifiers and at the end of this notebook, we will compare them. We will define our accuracy function right now to get it out of the way. We will use a simple accuracy function that returns the ratio of the number of correctly classified observations to the total number of predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "findaccuracy (generic function with 1 method)"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "findaccuracy(predictedvals,groundtruthvals) = sum(predictedvals.==groundtruthvals)/length(groundtruthvals)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "using GLMNet\n",
    "using RDatasets\n",
    "using MLBase\n",
    "using Plots\n",
    "using DecisionTree\n",
    "using Distances\n",
    "using NearestNeighbors\n",
    "using Random\n",
    "using LinearAlgebra\n",
    "using DataStructures\n",
    "using LIBSVM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the data first"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>SepalLength</th><th>SepalWidth</th><th>PetalLength</th><th>PetalWidth</th><th>Species</th></tr><tr><th></th><th>Float64</th><th>Float64</th><th>Float64</th><th>Float64</th><th>Cat…</th></tr></thead><tbody><p>150 rows × 5 columns</p><tr><th>1</th><td>5.1</td><td>3.5</td><td>1.4</td><td>0.2</td><td>setosa</td></tr><tr><th>2</th><td>4.9</td><td>3.0</td><td>1.4</td><td>0.2</td><td>setosa</td></tr><tr><th>3</th><td>4.7</td><td>3.2</td><td>1.3</td><td>0.2</td><td>setosa</td></tr><tr><th>4</th><td>4.6</td><td>3.1</td><td>1.5</td><td>0.2</td><td>setosa</td></tr><tr><th>5</th><td>5.0</td><td>3.6</td><td>1.4</td><td>0.2</td><td>setosa</td></tr><tr><th>6</th><td>5.4</td><td>3.9</td><td>1.7</td><td>0.4</td><td>setosa</td></tr><tr><th>7</th><td>4.6</td><td>3.4</td><td>1.4</td><td>0.3</td><td>setosa</td></tr><tr><th>8</th><td>5.0</td><td>3.4</td><td>1.5</td><td>0.2</td><td>setosa</td></tr><tr><th>9</th><td>4.4</td><td>2.9</td><td>1.4</td><td>0.2</td><td>setosa</td></tr><tr><th>10</th><td>4.9</td><td>3.1</td><td>1.5</td><td>0.1</td><td>setosa</td></tr><tr><th>11</th><td>5.4</td><td>3.7</td><td>1.5</td><td>0.2</td><td>setosa</td></tr><tr><th>12</th><td>4.8</td><td>3.4</td><td>1.6</td><td>0.2</td><td>setosa</td></tr><tr><th>13</th><td>4.8</td><td>3.0</td><td>1.4</td><td>0.1</td><td>setosa</td></tr><tr><th>14</th><td>4.3</td><td>3.0</td><td>1.1</td><td>0.1</td><td>setosa</td></tr><tr><th>15</th><td>5.8</td><td>4.0</td><td>1.2</td><td>0.2</td><td>setosa</td></tr><tr><th>16</th><td>5.7</td><td>4.4</td><td>1.5</td><td>0.4</td><td>setosa</td></tr><tr><th>17</th><td>5.4</td><td>3.9</td><td>1.3</td><td>0.4</td><td>setosa</td></tr><tr><th>18</th><td>5.1</td><td>3.5</td><td>1.4</td><td>0.3</td><td>setosa</td></tr><tr><th>19</th><td>5.7</td><td>3.8</td><td>1.7</td><td>0.3</td><td>setosa</td></tr><tr><th>20</th><td>5.1</td><td>3.8</td><td>1.5</td><td>0.3</td><td>setosa</td></tr><tr><th>21</th><td>5.4</td><td>3.4</td><td>1.7</td><td>0.2</td><td>setosa</td></tr><tr><th>22</th><td>5.1</td><td>3.7</td><td>1.5</td><td>0.4</td><td>setosa</td></tr><tr><th>23</th><td>4.6</td><td>3.6</td><td>1.0</td><td>0.2</td><td>setosa</td></tr><tr><th>24</th><td>5.1</td><td>3.3</td><td>1.7</td><td>0.5</td><td>setosa</td></tr><tr><th>25</th><td>4.8</td><td>3.4</td><td>1.9</td><td>0.2</td><td>setosa</td></tr><tr><th>26</th><td>5.0</td><td>3.0</td><td>1.6</td><td>0.2</td><td>setosa</td></tr><tr><th>27</th><td>5.0</td><td>3.4</td><td>1.6</td><td>0.4</td><td>setosa</td></tr><tr><th>28</th><td>5.2</td><td>3.5</td><td>1.5</td><td>0.2</td><td>setosa</td></tr><tr><th>29</th><td>5.2</td><td>3.4</td><td>1.4</td><td>0.2</td><td>setosa</td></tr><tr><th>30</th><td>4.7</td><td>3.2</td><td>1.6</td><td>0.2</td><td>setosa</td></tr><tr><th>&vellip;</th><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccccc}\n",
       "\t& SepalLength & SepalWidth & PetalLength & PetalWidth & Species\\\\\n",
       "\t\\hline\n",
       "\t& Float64 & Float64 & Float64 & Float64 & Cat…\\\\\n",
       "\t\\hline\n",
       "\t1 & 5.1 & 3.5 & 1.4 & 0.2 & setosa \\\\\n",
       "\t2 & 4.9 & 3.0 & 1.4 & 0.2 & setosa \\\\\n",
       "\t3 & 4.7 & 3.2 & 1.3 & 0.2 & setosa \\\\\n",
       "\t4 & 4.6 & 3.1 & 1.5 & 0.2 & setosa \\\\\n",
       "\t5 & 5.0 & 3.6 & 1.4 & 0.2 & setosa \\\\\n",
       "\t6 & 5.4 & 3.9 & 1.7 & 0.4 & setosa \\\\\n",
       "\t7 & 4.6 & 3.4 & 1.4 & 0.3 & setosa \\\\\n",
       "\t8 & 5.0 & 3.4 & 1.5 & 0.2 & setosa \\\\\n",
       "\t9 & 4.4 & 2.9 & 1.4 & 0.2 & setosa \\\\\n",
       "\t10 & 4.9 & 3.1 & 1.5 & 0.1 & setosa \\\\\n",
       "\t11 & 5.4 & 3.7 & 1.5 & 0.2 & setosa \\\\\n",
       "\t12 & 4.8 & 3.4 & 1.6 & 0.2 & setosa \\\\\n",
       "\t13 & 4.8 & 3.0 & 1.4 & 0.1 & setosa \\\\\n",
       "\t14 & 4.3 & 3.0 & 1.1 & 0.1 & setosa \\\\\n",
       "\t15 & 5.8 & 4.0 & 1.2 & 0.2 & setosa \\\\\n",
       "\t16 & 5.7 & 4.4 & 1.5 & 0.4 & setosa \\\\\n",
       "\t17 & 5.4 & 3.9 & 1.3 & 0.4 & setosa \\\\\n",
       "\t18 & 5.1 & 3.5 & 1.4 & 0.3 & setosa \\\\\n",
       "\t19 & 5.7 & 3.8 & 1.7 & 0.3 & setosa \\\\\n",
       "\t20 & 5.1 & 3.8 & 1.5 & 0.3 & setosa \\\\\n",
       "\t21 & 5.4 & 3.4 & 1.7 & 0.2 & setosa \\\\\n",
       "\t22 & 5.1 & 3.7 & 1.5 & 0.4 & setosa \\\\\n",
       "\t23 & 4.6 & 3.6 & 1.0 & 0.2 & setosa \\\\\n",
       "\t24 & 5.1 & 3.3 & 1.7 & 0.5 & setosa \\\\\n",
       "\t25 & 4.8 & 3.4 & 1.9 & 0.2 & setosa \\\\\n",
       "\t26 & 5.0 & 3.0 & 1.6 & 0.2 & setosa \\\\\n",
       "\t27 & 5.0 & 3.4 & 1.6 & 0.4 & setosa \\\\\n",
       "\t28 & 5.2 & 3.5 & 1.5 & 0.2 & setosa \\\\\n",
       "\t29 & 5.2 & 3.4 & 1.4 & 0.2 & setosa \\\\\n",
       "\t30 & 4.7 & 3.2 & 1.6 & 0.2 & setosa \\\\\n",
       "\t$\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m150×5 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m SepalLength \u001b[0m\u001b[1m SepalWidth \u001b[0m\u001b[1m PetalLength \u001b[0m\u001b[1m PetalWidth \u001b[0m\u001b[1m Species   \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Float64     \u001b[0m\u001b[90m Float64    \u001b[0m\u001b[90m Float64     \u001b[0m\u001b[90m Float64    \u001b[0m\u001b[90m Cat…      \u001b[0m\n",
       "─────┼─────────────────────────────────────────────────────────────\n",
       "   1 │         5.1         3.5          1.4         0.2  setosa\n",
       "   2 │         4.9         3.0          1.4         0.2  setosa\n",
       "   3 │         4.7         3.2          1.3         0.2  setosa\n",
       "   4 │         4.6         3.1          1.5         0.2  setosa\n",
       "   5 │         5.0         3.6          1.4         0.2  setosa\n",
       "   6 │         5.4         3.9          1.7         0.4  setosa\n",
       "   7 │         4.6         3.4          1.4         0.3  setosa\n",
       "   8 │         5.0         3.4          1.5         0.2  setosa\n",
       "   9 │         4.4         2.9          1.4         0.2  setosa\n",
       "  10 │         4.9         3.1          1.5         0.1  setosa\n",
       "  11 │         5.4         3.7          1.5         0.2  setosa\n",
       "  ⋮  │      ⋮           ⋮            ⋮           ⋮           ⋮\n",
       " 141 │         6.7         3.1          5.6         2.4  virginica\n",
       " 142 │         6.9         3.1          5.1         2.3  virginica\n",
       " 143 │         5.8         2.7          5.1         1.9  virginica\n",
       " 144 │         6.8         3.2          5.9         2.3  virginica\n",
       " 145 │         6.7         3.3          5.7         2.5  virginica\n",
       " 146 │         6.7         3.0          5.2         2.3  virginica\n",
       " 147 │         6.3         2.5          5.0         1.9  virginica\n",
       " 148 │         6.5         3.0          5.2         2.0  virginica\n",
       " 149 │         6.2         3.4          5.4         2.3  virginica\n",
       " 150 │         5.9         3.0          5.1         1.8  virginica\n",
       "\u001b[36m                                                   129 rows omitted\u001b[0m"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "iris = dataset(\"datasets\", \"iris\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "150-element CategoricalArrays.CategoricalArray{String,1,UInt8}:\n",
       " \"setosa\"\n",
       " \"setosa\"\n",
       " \"setosa\"\n",
       " \"setosa\"\n",
       " \"setosa\"\n",
       " \"setosa\"\n",
       " \"setosa\"\n",
       " \"setosa\"\n",
       " \"setosa\"\n",
       " \"setosa\"\n",
       " \"setosa\"\n",
       " \"setosa\"\n",
       " \"setosa\"\n",
       " ⋮\n",
       " \"virginica\"\n",
       " \"virginica\"\n",
       " \"virginica\"\n",
       " \"virginica\"\n",
       " \"virginica\"\n",
       " \"virginica\"\n",
       " \"virginica\"\n",
       " \"virginica\"\n",
       " \"virginica\"\n",
       " \"virginica\"\n",
       " \"virginica\"\n",
       " \"virginica\""
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = Matrix(iris[:,1:4])\n",
    "irislabels = iris[:,5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "150×4 Matrix{Float64}:\n",
       " 5.1  3.5  1.4  0.2\n",
       " 4.9  3.0  1.4  0.2\n",
       " 4.7  3.2  1.3  0.2\n",
       " 4.6  3.1  1.5  0.2\n",
       " 5.0  3.6  1.4  0.2\n",
       " 5.4  3.9  1.7  0.4\n",
       " 4.6  3.4  1.4  0.3\n",
       " 5.0  3.4  1.5  0.2\n",
       " 4.4  2.9  1.4  0.2\n",
       " 4.9  3.1  1.5  0.1\n",
       " 5.4  3.7  1.5  0.2\n",
       " 4.8  3.4  1.6  0.2\n",
       " 4.8  3.0  1.4  0.1\n",
       " ⋮              \n",
       " 6.0  3.0  4.8  1.8\n",
       " 6.9  3.1  5.4  2.1\n",
       " 6.7  3.1  5.6  2.4\n",
       " 6.9  3.1  5.1  2.3\n",
       " 5.8  2.7  5.1  1.9\n",
       " 6.8  3.2  5.9  2.3\n",
       " 6.7  3.3  5.7  2.5\n",
       " 6.7  3.0  5.2  2.3\n",
       " 6.3  2.5  5.0  1.9\n",
       " 6.5  3.0  5.2  2.0\n",
       " 6.2  3.4  5.4  2.3\n",
       " 5.9  3.0  5.1  1.8"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "150-element Vector{Int64}:\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " ⋮\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "irislabelsmap = labelmap(irislabels)\n",
    "y = labelencode(irislabelsmap, irislabels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In classification, we often want to use some of the data to fit a model, and the rest of the data to validate (commonly known as `training` and `testing` data). We will get this data ready now so that we can easily use it in the rest of this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "perclass_splits (generic function with 1 method)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "function perclass_splits(y,at)\n",
    "    uids = unique(y)\n",
    "    keepids = []\n",
    "    for ui in uids\n",
    "        curids = findall(y.==ui)\n",
    "        rowids = randsubseq(curids, at) \n",
    "        push!(keepids,rowids...)\n",
    "    end\n",
    "    return keepids\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "search: \u001b[0m\u001b[1mr\u001b[22m\u001b[0m\u001b[1ma\u001b[22m\u001b[0m\u001b[1mn\u001b[22m\u001b[0m\u001b[1md\u001b[22m\u001b[0m\u001b[1ms\u001b[22m\u001b[0m\u001b[1mu\u001b[22m\u001b[0m\u001b[1mb\u001b[22m\u001b[0m\u001b[1ms\u001b[22m\u001b[0m\u001b[1me\u001b[22m\u001b[0m\u001b[1mq\u001b[22m \u001b[0m\u001b[1mr\u001b[22m\u001b[0m\u001b[1ma\u001b[22m\u001b[0m\u001b[1mn\u001b[22m\u001b[0m\u001b[1md\u001b[22m\u001b[0m\u001b[1ms\u001b[22m\u001b[0m\u001b[1mu\u001b[22m\u001b[0m\u001b[1mb\u001b[22m\u001b[0m\u001b[1ms\u001b[22m\u001b[0m\u001b[1me\u001b[22m\u001b[0m\u001b[1mq\u001b[22m! \u001b[0m\u001b[1mR\u001b[22m\u001b[0m\u001b[1ma\u001b[22m\u001b[0m\u001b[1mn\u001b[22m\u001b[0m\u001b[1md\u001b[22mom\u001b[0m\u001b[1mS\u001b[22m\u001b[0m\u001b[1mu\u001b[22m\u001b[0m\u001b[1mb\u001b[22m St\u001b[0m\u001b[1mr\u001b[22m\u001b[0m\u001b[1ma\u001b[22mtifiedRa\u001b[0m\u001b[1mn\u001b[22m\u001b[0m\u001b[1md\u001b[22mom\u001b[0m\u001b[1mS\u001b[22m\u001b[0m\u001b[1mu\u001b[22m\u001b[0m\u001b[1mb\u001b[22m\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/latex": [
       "\\begin{verbatim}\n",
       "randsubseq([rng=GLOBAL_RNG,] A, p) -> Vector\n",
       "\\end{verbatim}\n",
       "Return a vector consisting of a random subsequence of the given array \\texttt{A}, where each element of \\texttt{A} is included (in order) with independent probability \\texttt{p}. (Complexity is linear in \\texttt{p*length(A)}, so this function is efficient even if \\texttt{p} is small and \\texttt{A} is large.) Technically, this process is known as \"Bernoulli sampling\" of \\texttt{A}.\n",
       "\n",
       "\\section{Examples}\n",
       "\\begin{verbatim}\n",
       "julia> rng = MersenneTwister(1234);\n",
       "\n",
       "julia> randsubseq(rng, 1:8, 0.3)\n",
       "2-element Vector{Int64}:\n",
       " 7\n",
       " 8\n",
       "\\end{verbatim}\n"
      ],
      "text/markdown": [
       "```\n",
       "randsubseq([rng=GLOBAL_RNG,] A, p) -> Vector\n",
       "```\n",
       "\n",
       "Return a vector consisting of a random subsequence of the given array `A`, where each element of `A` is included (in order) with independent probability `p`. (Complexity is linear in `p*length(A)`, so this function is efficient even if `p` is small and `A` is large.) Technically, this process is known as \"Bernoulli sampling\" of `A`.\n",
       "\n",
       "# Examples\n",
       "\n",
       "```jldoctest\n",
       "julia> rng = MersenneTwister(1234);\n",
       "\n",
       "julia> randsubseq(rng, 1:8, 0.3)\n",
       "2-element Vector{Int64}:\n",
       " 7\n",
       " 8\n",
       "```\n"
      ],
      "text/plain": [
       "\u001b[36m  randsubseq([rng=GLOBAL_RNG,] A, p) -> Vector\u001b[39m\n",
       "\n",
       "  Return a vector consisting of a random subsequence of the given array \u001b[36mA\u001b[39m,\n",
       "  where each element of \u001b[36mA\u001b[39m is included (in order) with independent probability\n",
       "  \u001b[36mp\u001b[39m. (Complexity is linear in \u001b[36mp*length(A)\u001b[39m, so this function is efficient even\n",
       "  if \u001b[36mp\u001b[39m is small and \u001b[36mA\u001b[39m is large.) Technically, this process is known as\n",
       "  \"Bernoulli sampling\" of \u001b[36mA\u001b[39m.\n",
       "\n",
       "\u001b[1m  Examples\u001b[22m\n",
       "\u001b[1m  ≡≡≡≡≡≡≡≡≡≡\u001b[22m\n",
       "\n",
       "\u001b[36m  julia> rng = MersenneTwister(1234);\u001b[39m\n",
       "\u001b[36m  \u001b[39m\n",
       "\u001b[36m  julia> randsubseq(rng, 1:8, 0.3)\u001b[39m\n",
       "\u001b[36m  2-element Vector{Int64}:\u001b[39m\n",
       "\u001b[36m   7\u001b[39m\n",
       "\u001b[36m   8\u001b[39m"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "?randsubseq"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "46-element Vector{Int64}:\n",
       "   2\n",
       "   8\n",
       "   9\n",
       "  12\n",
       "  14\n",
       "  17\n",
       "  20\n",
       "  25\n",
       "  28\n",
       "  29\n",
       "  36\n",
       "  38\n",
       "  39\n",
       "   ⋮\n",
       "  98\n",
       " 105\n",
       " 107\n",
       " 110\n",
       " 111\n",
       " 113\n",
       " 122\n",
       " 128\n",
       " 129\n",
       " 133\n",
       " 144\n",
       " 146"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainids = perclass_splits(y,0.7)\n",
    "testids = setdiff(1:length(y),trainids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will need one more function, and that is the function that will assign classes based on the predicted values when the predicted values are continuous."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "assign_class (generic function with 1 method)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "assign_class(predictedvalue) = argmin(abs.(predictedvalue .- [1,2,3]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🟣 Method 1: Lasso"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Least Squares GLMNet Cross Validation\n",
       "72 models for 4 predictors in 10 folds\n",
       "Best λ 0.001 (mean loss 0.046, std 0.007)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path = glmnet(X[trainids,:], y[trainids])\n",
    "cv = glmnetcv(X[trainids,:], y[trainids])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# choose the best lambda to predict with.\n",
    "path = glmnet(X[trainids,:], y[trainids])\n",
    "cv = glmnetcv(X[trainids,:], y[trainids])\n",
    "mylambda = path.lambda[argmin(cv.meanloss)]\n",
    "\n",
    "path = glmnet(X[trainids,:], y[trainids],lambda=[mylambda]);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "46×1 Matrix{Float64}:\n",
       " 0.956628239318962\n",
       " 0.9680924939488736\n",
       " 1.0274913820485194\n",
       " 1.0270310335343171\n",
       " 0.8996549871193242\n",
       " 0.9363876746151771\n",
       " 0.9951109014647526\n",
       " 1.1205601984214608\n",
       " 0.9388725781539099\n",
       " 0.90915395436343\n",
       " 0.8774788580721995\n",
       " 0.9011511111058725\n",
       " 0.9948572292475704\n",
       " ⋮\n",
       " 2.195783234015177\n",
       " 3.040902937646202\n",
       " 2.631343216737511\n",
       " 3.1687096095329315\n",
       " 2.726291608202368\n",
       " 2.8590000046427835\n",
       " 2.7946995704356583\n",
       " 2.628917580837503\n",
       " 2.948616224528425\n",
       " 2.9953467657101065\n",
       " 3.0742511111785356\n",
       " 2.8728129977638677"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "q = X[testids,:];\n",
    "predictions_lasso = GLMNet.predict(path,q)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9347826086956522"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions_lasso = assign_class.(predictions_lasso)\n",
    "findaccuracy(predictions_lasso,y[testids])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🟣 Method 2: Ridge\n",
    "We will use the same function but set alpha to zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9347826086956522"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# choose the best lambda to predict with.\n",
    "path = glmnet(X[trainids,:], y[trainids],alpha=0);\n",
    "cv = glmnetcv(X[trainids,:], y[trainids],alpha=0)\n",
    "mylambda = path.lambda[argmin(cv.meanloss)]\n",
    "path = glmnet(X[trainids,:], y[trainids],alpha=0,lambda=[mylambda]);\n",
    "q = X[testids,:];\n",
    "predictions_ridge = GLMNet.predict(path,q)\n",
    "predictions_ridge = assign_class.(predictions_ridge)\n",
    "findaccuracy(predictions_ridge,y[testids])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🟣 Method 3: Elastic Net\n",
    "We will use the same function but set alpha to 0.5 (it's the combination of lasso and ridge)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9347826086956522"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# choose the best lambda to predict with.\n",
    "path = glmnet(X[trainids,:], y[trainids],alpha=0.5);\n",
    "cv = glmnetcv(X[trainids,:], y[trainids],alpha=0.5)\n",
    "mylambda = path.lambda[argmin(cv.meanloss)]\n",
    "path = glmnet(X[trainids,:], y[trainids],alpha=0.5,lambda=[mylambda]);\n",
    "q = X[testids,:];\n",
    "predictions_EN = GLMNet.predict(path,q)\n",
    "predictions_EN = assign_class.(predictions_EN)\n",
    "findaccuracy(predictions_EN,y[testids])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🟣 Method 4: Decision Trees\n",
    "We will use the package `DecisionTree`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier\n",
       "max_depth:                2\n",
       "min_samples_leaf:         1\n",
       "min_samples_split:        2\n",
       "min_purity_increase:      0.0\n",
       "pruning_purity_threshold: 1.0\n",
       "n_subfeatures:            0\n",
       "classes:                  [1, 2, 3]\n",
       "root:                     Decision Tree\n",
       "Leaves: 3\n",
       "Depth:  2"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = DecisionTreeClassifier(max_depth=2)\n",
    "DecisionTree.fit!(model, X[trainids,:], y[trainids])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8913043478260869"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "q = X[testids,:];\n",
    "predictions_DT = DecisionTree.predict(model, q)\n",
    "findaccuracy(predictions_DT,y[testids])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🟣 Method 5: Random Forests\n",
    "The `RandomForestClassifier` is available through the `DecisionTree` package as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier\n",
       "n_trees:             20\n",
       "n_subfeatures:       -1\n",
       "partial_sampling:    0.7\n",
       "max_depth:           -1\n",
       "min_samples_leaf:    1\n",
       "min_samples_split:   2\n",
       "min_purity_increase: 0.0\n",
       "classes:             [1, 2, 3]\n",
       "ensemble:            Ensemble of Decision Trees\n",
       "Trees:      20\n",
       "Avg Leaves: 5.6\n",
       "Avg Depth:  4.2"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = RandomForestClassifier(n_trees=20)\n",
    "DecisionTree.fit!(model, X[trainids,:], y[trainids])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9130434782608695"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "q = X[testids,:];\n",
    "predictions_RF = DecisionTree.predict(model, q)\n",
    "findaccuracy(predictions_RF,y[testids])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🟣 Method 6: Using a Nearest Neighbor method\n",
    "We will use the `NearestNeighbors` package here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KDTree{StaticArrays.SVector{4, Float64}, Euclidean, Float64}\n",
       "  Number of points: 104\n",
       "  Dimensions: 4\n",
       "  Metric: Euclidean(0.0)\n",
       "  Reordered: true"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Xtrain = X[trainids,:]\n",
    "ytrain = y[trainids]\n",
    "kdtree = KDTree(Xtrain')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "46×4 Matrix{Float64}:\n",
       " 4.9  3.0  1.4  0.2\n",
       " 5.0  3.4  1.5  0.2\n",
       " 4.4  2.9  1.4  0.2\n",
       " 4.8  3.4  1.6  0.2\n",
       " 4.3  3.0  1.1  0.1\n",
       " 5.4  3.9  1.3  0.4\n",
       " 5.1  3.8  1.5  0.3\n",
       " 4.8  3.4  1.9  0.2\n",
       " 5.2  3.5  1.5  0.2\n",
       " 5.2  3.4  1.4  0.2\n",
       " 5.0  3.2  1.2  0.2\n",
       " 4.9  3.6  1.4  0.1\n",
       " 4.4  3.0  1.3  0.2\n",
       " ⋮              \n",
       " 6.2  2.9  4.3  1.3\n",
       " 6.5  3.0  5.8  2.2\n",
       " 4.9  2.5  4.5  1.7\n",
       " 7.2  3.6  6.1  2.5\n",
       " 6.5  3.2  5.1  2.0\n",
       " 6.8  3.0  5.5  2.1\n",
       " 5.6  2.8  4.9  2.0\n",
       " 6.1  3.0  4.9  1.8\n",
       " 6.4  2.8  5.6  2.1\n",
       " 6.4  2.8  5.6  2.2\n",
       " 6.8  3.2  5.9  2.3\n",
       " 6.7  3.0  5.2  2.3"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "queries = X[testids,:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([[25, 32, 9, 7, 18], [27, 1, 12, 19, 4], [3, 33, 32, 9, 2], [20, 19, 21, 6, 27], [33, 2, 3, 9, 6], [8, 34, 24, 5, 15], [15, 34, 4, 12, 1], [20, 19, 21, 17, 27], [1, 27, 12, 34, 4], [1, 27, 12, 28, 4]  …  [60, 59, 44, 37, 74], [81, 100, 68, 92, 84], [102, 76, 94, 77, 98], [96, 97, 81, 84, 77], [99, 67, 74, 104, 75], [95, 86, 104, 83, 90], [69, 73, 77, 94, 102], [69, 73, 97, 77, 102], [81, 84, 100, 97, 68], [98, 102, 96, 76, 97]], [[0.14142135623730964, 0.14142135623730986, 0.1414213562373099, 0.1732050807568878, 0.22360679774997896], [0.09999999999999964, 0.17320508075688762, 0.1999999999999999, 0.22360679774997902, 0.22360679774997916], [0.2999999999999997, 0.3605551275463988, 0.42426406871192807, 0.42426406871192807, 0.4358898943540674], [0.22360679774997858, 0.2828427124746191, 0.2999999999999998, 0.3000000000000002, 0.31622776601683783], [0.4795831523312718, 0.5000000000000003, 0.519615242270663, 0.58309518948453, 0.6164414002968974], [0.3464101615137753, 0.36055512754639896, 0.3872983346207418, 0.3999999999999999, 0.41231056256176646], [0.14142135623730928, 0.2449489742783178, 0.26457513110645875, 0.31622776601683783, 0.33166247903553986], [0.3741657386773938, 0.412310562561766, 0.4242640687119283, 0.47958315233127186, 0.49999999999999983], [0.14142135623730995, 0.14142135623730995, 0.17320508075688806, 0.22360679774997896, 0.24494897427831802], [0.14142135623730995, 0.14142135623730995, 0.17320508075688806, 0.2645751311064592, 0.2828427124746193]  …  [0.7937253933193769, 0.877496438739212, 0.883176086632784, 0.9433981132056601, 0.9899494936611664], [0.6708203932499366, 0.7071067811865474, 0.7549834435270749, 0.806225774829855, 0.8124038404635958], [0.22360679774997935, 0.374165738677394, 0.46904157598234314, 0.48989794855663593, 0.5099019513592786], [0.17320508075688787, 0.34641016151377513, 0.3605551275463991, 0.374165738677394, 0.42426406871192845], [0.3162277660168375, 0.3162277660168375, 0.33166247903553986, 0.4582575694955842, 0.4898979485566353], [0.14142135623730964, 0.24494897427831838, 0.282842712474618, 0.3605551275463989, 0.4582575694955838], [0.3316624790355402, 0.374165738677394, 0.38729833462074165, 0.4358898943540675, 0.46904157598234253], [0.4242640687119288, 0.43588989435406733, 0.4690415759823429, 0.469041575982343, 0.49999999999999967], [0.22360679774997935, 0.3162277660168376, 0.31622776601683794, 0.346410161513776, 0.4123105625617659], [0.24494897427831822, 0.3605551275463989, 0.360555127546399, 0.374165738677394, 0.42426406871192807]])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "idxs, dists = knn(kdtree, queries', 5, true)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9347826086956522"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c = ytrain[hcat(idxs...)]\n",
    "possible_labels = map(i->counter(c[:,i]),1:size(c,2))\n",
    "predictions_NN = map(i->parse(Int,string(string(argmax(possible_labels[i])))),1:size(c,2))\n",
    "findaccuracy(predictions_NN,y[testids])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🟣 Method 7: Support Vector Machines\n",
    "We will use the `LIBSVM` package here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "104-element Vector{Int64}:\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " ⋮\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Xtrain = X[trainids,:]\n",
    "ytrain = y[trainids]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LIBSVM.SVM{Int64}(SVC, LIBSVM.Kernel.RadialBasis, nothing, 4, 3, [1, 2, 3], Int32[1, 2, 3], Float64[], Int32[], LIBSVM.SupportVectors{Int64, Float64}(35, Int32[4, 15, 16], [1, 1, 1, 1, 2, 2, 2, 2, 2, 2  …  3, 3, 3, 3, 3, 3, 3, 3, 3, 3], [5.7 5.1 … 6.5 5.9; 4.4 3.3 … 3.0 3.0; 1.5 1.7 … 5.2 5.1; 0.4 0.5 … 2.0 1.8], Int32[11, 17, 29, 31, 35, 36, 37, 38, 39, 40  …  86, 87, 89, 90, 95, 98, 99, 101, 102, 104], LIBSVM.SVMNode[LIBSVM.SVMNode(1, 5.7), LIBSVM.SVMNode(1, 5.1), LIBSVM.SVMNode(1, 4.5), LIBSVM.SVMNode(1, 5.1), LIBSVM.SVMNode(1, 6.9), LIBSVM.SVMNode(1, 6.5), LIBSVM.SVMNode(1, 5.7), LIBSVM.SVMNode(1, 6.3), LIBSVM.SVMNode(1, 4.9), LIBSVM.SVMNode(1, 5.0)  …  LIBSVM.SVMNode(1, 6.2), LIBSVM.SVMNode(1, 7.2), LIBSVM.SVMNode(1, 7.9), LIBSVM.SVMNode(1, 6.3), LIBSVM.SVMNode(1, 6.0), LIBSVM.SVMNode(1, 6.9), LIBSVM.SVMNode(1, 5.8), LIBSVM.SVMNode(1, 6.3), LIBSVM.SVMNode(1, 6.5), LIBSVM.SVMNode(1, 5.9)]), 0.0, [0.37125893964830514 0.9086334573221102; 0.192673182555654 0.0; … ; -0.0 -0.7276485813122384; -0.0 -1.0], Float64[], Float64[], [0.04759688891036977, 0.10454494768475925, 0.09360783948273636], 3, 0.25, 200.0, 0.001, 1.0, 0.5, 0.1, true, false)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = svmtrain(Xtrain', ytrain)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9347826086956522"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions_SVM, decision_values = svmpredict(model, X[testids,:]')\n",
    "findaccuracy(predictions_SVM,y[testids])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Putting all the results together:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7×2 Matrix{Any}:\n",
       " \"lasso\"  0.934783\n",
       " \"ridge\"  0.934783\n",
       " \"EN\"     0.934783\n",
       " \"DT\"     0.891304\n",
       " \"RF\"     0.913043\n",
       " \"kNN\"    0.934783\n",
       " \"SVM\"    0.934783"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "overall_accuracies = zeros(7)\n",
    "methods = [\"lasso\",\"ridge\",\"EN\", \"DT\", \"RF\",\"kNN\", \"SVM\"]\n",
    "ytest = y[testids]\n",
    "overall_accuracies[1] = findaccuracy(predictions_lasso,ytest)\n",
    "overall_accuracies[2] = findaccuracy(predictions_ridge,ytest)\n",
    "overall_accuracies[3] = findaccuracy(predictions_EN,ytest)\n",
    "overall_accuracies[4] = findaccuracy(predictions_DT,ytest)\n",
    "overall_accuracies[5] = findaccuracy(predictions_RF,ytest)\n",
    "overall_accuracies[6] = findaccuracy(predictions_NN,ytest)\n",
    "overall_accuracies[7] = findaccuracy(predictions_SVM,ytest)\n",
    "hcat(methods, overall_accuracies)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Finally...\n",
    "After finishing this notebook, you should be able to:\n",
    "- [ ] split your data into training and testing data to test the effectiveness of a certain method\n",
    "- [ ] apply a simple accuracy function to test the effectiveness of a certain method\n",
    "- [ ] run multiple classification algorithms:\n",
    "    - [ ] LASSO\n",
    "    - [ ] Ridge\n",
    "    - [ ] ElasticNet\n",
    "    - [ ] Decision Tree\n",
    "    - [ ] Random Forest\n",
    "    - [ ] Nearest Neighbors\n",
    "    - [ ] Support Vector Machines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 🥳 One cool finding\n",
    "\n",
    "We used multiple methods to run classification on the `iris` dataset which is a dataset of flowers and there are three types of iris flowers in it. We split the data into training and testing and ran our methods. Here is the scoreboard:\n",
    "\n",
    "| method | accuracy score |\n",
    "|---|---|\n",
    "| lasso  |1.0|\n",
    "| ridge  |1.0|\n",
    "| EN     |1.0|\n",
    "| DT     |0.960784|\n",
    "| RF     |0.980392|\n",
    "| kNN    |1.0|\n",
    "| SVM    |1.0|"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 1.6.0",
   "language": "julia",
   "name": "julia-1.6"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}