2.5.2 - Validating a linear regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "For this challenge we are tasked with validating various features for a model that attempts to predict property crime in cities across New York.  We have already cleaned this data and built multiple models in previous exercises.  \n",
    "\n",
    "This notebook looks at 3 features to predice the property crime in New York cities.  We then convert 2 of the features to become categorical and compare the results between the two different approaches.  We seek to answer the questions:  Which features are significant ?  Which features can be left out ?  How effective are the two models in comparison to one another ?\n",
    "\n",
    "The next step would be to try the most effective newly created model on data that we have not encountered yet - such as New York crime data from a previous year or crime data from another state altogether.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing and cleaning the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# importing pnadas modules\n",
    "import pandas as pd\n",
    "from matplotlib import pyplot as plt\n",
    "import numpy as np\n",
    "from sklearn import linear_model\n",
    "import seaborn as sns\n",
    "import statsmodels.formula.api as smf\n",
    "from statsmodels.sandbox.regression.predstd import wls_prediction_std\n",
    "\n",
    "%matplotlib inline\n",
    "#pd.options.display.float_format = '{:.3f}'.format\n",
    "sns.set_style('white')\n",
    "\n",
    "# Suppress annoying harmless error.\n",
    "import warnings\n",
    "warnings.filterwarnings(action=\"ignore\", module=\"scipy\", message=\"^internal gelsd\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Table 8</th>\n",
       "      <th>Unnamed: 1</th>\n",
       "      <th>Unnamed: 2</th>\n",
       "      <th>Unnamed: 3</th>\n",
       "      <th>Unnamed: 4</th>\n",
       "      <th>Unnamed: 5</th>\n",
       "      <th>Unnamed: 6</th>\n",
       "      <th>Unnamed: 7</th>\n",
       "      <th>Unnamed: 8</th>\n",
       "      <th>Unnamed: 9</th>\n",
       "      <th>Unnamed: 10</th>\n",
       "      <th>Unnamed: 11</th>\n",
       "      <th>Unnamed: 12</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>NEW YORK</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Offenses Known to Law Enforcement</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>by City, 2013</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>City</td>\n",
       "      <td>Population</td>\n",
       "      <td>Violent\\ncrime</td>\n",
       "      <td>Murder and\\nnonnegligent\\nmanslaughter</td>\n",
       "      <td>Rape\\n(revised\\ndefinition)1</td>\n",
       "      <td>Rape\\n(legacy\\ndefinition)2</td>\n",
       "      <td>Robbery</td>\n",
       "      <td>Aggravated\\nassault</td>\n",
       "      <td>Property\\ncrime</td>\n",
       "      <td>Burglary</td>\n",
       "      <td>Larceny-\\ntheft</td>\n",
       "      <td>Motor\\nvehicle\\ntheft</td>\n",
       "      <td>Arson3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Adams Village</td>\n",
       "      <td>1861</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>2</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                             Table 8  Unnamed: 1      Unnamed: 2  \\\n",
       "0                           NEW YORK         NaN             NaN   \n",
       "1  Offenses Known to Law Enforcement         NaN             NaN   \n",
       "2                      by City, 2013         NaN             NaN   \n",
       "3                               City  Population  Violent\\ncrime   \n",
       "4                      Adams Village        1861               0   \n",
       "\n",
       "                               Unnamed: 3                    Unnamed: 4  \\\n",
       "0                                     NaN                           NaN   \n",
       "1                                     NaN                           NaN   \n",
       "2                                     NaN                           NaN   \n",
       "3  Murder and\\nnonnegligent\\nmanslaughter  Rape\\n(revised\\ndefinition)1   \n",
       "4                                       0                           NaN   \n",
       "\n",
       "                    Unnamed: 5 Unnamed: 6           Unnamed: 7  \\\n",
       "0                          NaN        NaN                  NaN   \n",
       "1                                     NaN                  NaN   \n",
       "2                          NaN        NaN                  NaN   \n",
       "3  Rape\\n(legacy\\ndefinition)2    Robbery  Aggravated\\nassault   \n",
       "4                            0          0                    0   \n",
       "\n",
       "        Unnamed: 8 Unnamed: 9      Unnamed: 10            Unnamed: 11  \\\n",
       "0              NaN        NaN              NaN                    NaN   \n",
       "1              NaN        NaN              NaN                    NaN   \n",
       "2              NaN        NaN              NaN                    NaN   \n",
       "3  Property\\ncrime   Burglary  Larceny-\\ntheft  Motor\\nvehicle\\ntheft   \n",
       "4               12          2               10                      0   \n",
       "\n",
       "  Unnamed: 12  \n",
       "0         NaN  \n",
       "1         NaN  \n",
       "2         NaN  \n",
       "3      Arson3  \n",
       "4           0  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# importing raw data from excel into a pandas dataframe\n",
    "df_raw = pd.read_excel('table_8_offenses_known_to_law_enforcement_new_york_by_city_2013.xls')\n",
    "# display raw file\n",
    "df_raw.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Drop some rows, rename some columns and square pop to get a pop2 column just in case we want to use this as a feature.   Also, convert data to float."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>city</th>\n",
       "      <th>pop</th>\n",
       "      <th>murder</th>\n",
       "      <th>robbery</th>\n",
       "      <th>pc</th>\n",
       "      <th>pop2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Adams Village</td>\n",
       "      <td>1861.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>3.463321e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Addison Town and Village</td>\n",
       "      <td>2577.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>6.640929e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Akron Village</td>\n",
       "      <td>2846.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>8.099716e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Albany</td>\n",
       "      <td>97956.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>227.0</td>\n",
       "      <td>4090.0</td>\n",
       "      <td>9.595378e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Albion Village</td>\n",
       "      <td>6388.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>223.0</td>\n",
       "      <td>4.080654e+07</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                       city      pop  murder  robbery      pc          pop2\n",
       "0             Adams Village   1861.0     0.0      0.0    12.0  3.463321e+06\n",
       "1  Addison Town and Village   2577.0     0.0      0.0    24.0  6.640929e+06\n",
       "2             Akron Village   2846.0     0.0      0.0    16.0  8.099716e+06\n",
       "3                    Albany  97956.0     8.0    227.0  4090.0  9.595378e+09\n",
       "4            Albion Village   6388.0     0.0      4.0   223.0  4.080654e+07"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# setting the proper heading from the 4th row of raw data\n",
    "headers = df_raw.iloc[3]\n",
    "\n",
    "# exclude rows 1 through 4 (or rows 0, 1, 2, 3)\n",
    "df_ny = pd.DataFrame(df_raw.values[4:], columns=headers)\n",
    "\n",
    "# renaming columns\n",
    "df_ny.columns = ['city', 'pop', 'vc', 'murder', 'rape1', 'rape2', 'robbery', 'aa', 'pc', 'burglary', 'lt', 'mvt', 'arson3']\n",
    "\n",
    "# dropping unnecessary columns\n",
    "df_ny = df_ny.drop(['vc', 'rape1', 'rape2', 'aa', 'burglary', 'lt', 'mvt', 'arson3'], axis=1)\n",
    "\n",
    "# removing last 3 rows of instructional data\n",
    "df_ny = df_ny.drop([348, 349, 350], axis=0)\n",
    "\n",
    "#converting string values to float and creating 'pop2' column which is pop squared\n",
    "df_ny[\"pop\"] = [np.float(x) for x in df_ny[\"pop\"]]\n",
    "df_ny[\"pop2\"] = [np.float(x*x) for x in df_ny[\"pop\"]]\n",
    "df_ny[\"murder\"] = [np.float(x) for x in df_ny[\"murder\"]]\n",
    "df_ny[\"robbery\"] = [np.float(x) for x in df_ny[\"robbery\"]]\n",
    "df_ny[\"pc\"] = [np.float(x) for x in df_ny[\"pc\"]]\n",
    "\n",
    "# display data\n",
    "df_ny.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data exploration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From previous exercises, we know that NYC data is a significant outlier which overwhelms the model, so we are going to drop it for now to build the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0xcc81d70>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAD7CAYAAACIYvgKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAFV9JREFUeJzt3X+M3Pdd5/Hn7HqdTDjHW66md141FBHxLn/kGic5DE3SuJB0LzUiZpFoBAXRCg6Q745A5LZuXQIoUVKZS0tpuKKUEGgpKiQbC1qFNaJtZExag7U5nJC+e/R6zd0aUJN2Y5ds2/V67o+ZTcebmZ0fO7vfmdnnQ4oy85n3zH4/mcz39f1+vp/v91uqVCpIkja3kaIXQJJUPMNAkmQYSJIMA0kShoEkCcNAkoRhIEnCMJAkYRhIkoAtRS9Au3bv3l2ZmJgoejEkaaA89dRTz2bmjlZ1AxMGExMTTE9PF70YkjRQIuJL7dQ5TCRJMgwkSYaBJAnDQJKEYSBJYoBmE0nSZnNkdo7DM8np+QV2jpc5MBns27U+U+wNA0nqQ0dm5zg4fYqFxSUA5uYXODh9CmBdAsFhIknqQ4dn8sUgWLawuMThmVyXv2cYSFIfOj2/0FH7WhkGktSHdo6XO2pfK8NAkvrQgcmgPDZ6QVt5bJQDk7Euf88DyJLUh5YPEjubSJI2uX27JtZt5b9SW2EQEbuB92Tmnoi4EvhtYAn4BvDTmfkvEfFzwM8D54A7M/PjEfFy4KNAGTgNvCUzX2hU2/OeSZLa1vKYQUS8DfgQcHGt6beA/5qZe4Bp4O0R8e+A/wZcC0wCd0fERcCvAh/NzOuBWeDnV6mVJBWknQPIXwCm6p7fmplP1B5vAb4OfB9wPDO/kZnPA/8I/AfgOuAvarWPAjeuUitJKkjLMMjMh4HFuuf/BBARrwX+C/Be4FLg+bq3nQW2r2hv1FbfLkkqSFdTSyPiTcAHgb2Z+WXgDLCtrmQbML+ivVFbfbskqSAdzyaKiDdTPfi7JzO/Ums+AdwVERcDFwHfCzwJHAfeCDwI3AwcW6VWklSQjvYMImIUeD/VrfnpiPh0RPx6Zv5zrf0Y8EngXZn5deBO4NaIOA78APCBVWolSQUpVSqVopehLVNTU5Xp6emiF0OSBkpEnMzMa1rVeTkKSZJhIEkyDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJIEbCl6ASSpG0dm5zg8k5yeX2DneJkDk8G+XRNFL9bAaisMImI38J7M3BMRlwMPAhXgSWB/Zp6PiDuAvcA54LbMPNFJbY/7JWmIHZmd4+D0KRYWlwCYm1/g4PQpAAOhSy2HiSLibcCHgItrTfcChzLzeqAE3BIRVwE3ALuBW4H7uqiVpLYcnskXg2DZwuISh2eyoCUafO0cM/gCMFX3/GrgsdrjR4EbgeuAo5lZycxngC0RsaPDWklqy+n5hY7a1VrLMMjMh4HFuqZSZlZqj88C24FLgefrapbbO6mVpLbsHC931K7WuplNdL7u8TZgHjhTe7yyvZNaSWrLgcmgPDZ6QVt5bJQDk1HQEg2+bsJgNiL21B7fDBwDjgOTETESEZcBI5n5bIe1ktSWfbsmuHvqCibGy5SAifEyd09d4cHjNehmauntwP0RsRV4GngoM5ci4hjwONWA2d9FrSS1bd+uCVf+PVSqVCqtq/rA1NRUZXp6uujFkKSBEhEnM/OaVnWegSxJMgwkSYaBJAnDQJKEYSBJwjCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkoAt3bwpIsaAPwBeBSwBPwecAx4EKsCTwP7MPB8RdwB7a6/flpknIuLyRrVr6okkqWvd7hm8EdiSma8FfgO4C7gXOJSZ1wMl4JaIuAq4AdgN3ArcV3v/S2q774Ikaa26DYPPA1siYgS4FFgErgYeq73+KHAjcB1wNDMrmflM7T07mtRKkgrS1TAR8DWqQ0SfA14O/DDwusys1F4/C2ynGhTP1b1vub3UoFaSVJBu9wx+GZjJzO8BXkP1+MHWute3AfPAmdrjle3nG7RJkgrSbRh8FXi+9vgrwBgwGxF7am03A8eA48BkRIxExGXASGY+26RWklSQboeJ3gs8EBHHqO4RvBP4O+D+iNgKPA08lJlLtZrHqQbP/tr7b19Zu4Y+SJLWqFSpVFpX9YGpqanK9PR00YshSQMlIk5m5jWt6jzpTJJkGEiSDANJEoaBJAnDQJKEYSBJwjCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiS6P5OZ5J64MjsHIdnktPzC+wcL3NgMti3a6LoxdImZBhIBTkyO8fB6VMsLC4BMDe/wMHpUwAGgjacw0RSQQ7P5ItBsGxhcYnDM1nQEmkzMwykgpyeX+ioXVpPhoFUkJ3j5Y7apfVkGEgFOTAZlMdGL2grj41yYDIKWiJtZh5AlgqyfJDY2UTqB4aBVKB9uyZc+asvdB0GEXEQ+BFgK/A7wGPAg0AFeBLYn5nnI+IOYC9wDrgtM09ExOWNatfQD0nSGnR1zCAi9gCvBa4FbgBeCdwLHMrM64EScEtEXFV7fTdwK3Bf7SNeUruGPkiS1qjbA8iTwCngEeDPgY8DV1PdOwB4FLgRuA44mpmVzHwG2BIRO5rUSpIK0u0w0cuB7wR+GPgu4M+Akcys1F4/C2wHLgWeq3vfcnupQa0kqSDdhsFzwOcy85tARsTXqQ4VLdsGzANnao9Xtp9v0CZJKki3w0R/DfyniChFxE7g24C/qh1LALgZOAYcByYjYiQiLqO69/AsMNugVpJUkK72DDLz4xHxOuAE1UDZD3wRuD8itgJPAw9l5lJEHAMer6sDuH1l7dq6IUlai1KlUmld1QempqYq09PTRS+GJA2UiDiZmde0qvNyFJIkw0CSZBhIkjAMJEkYBpIkDANJEoaBJAnDQJKEYSBJwjCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kSsKXoBZDacWR2jsMzyen5BXaOlzkwGezbNVH0YklDY01hEBHfAZwEbgLOAQ8CFeBJYH9mno+IO4C9tddvy8wTEXF5o9q1LIuG15HZOQ5On2JhcQmAufkFDk6fAjAQpB7pepgoIsaA3wUWak33Aocy83qgBNwSEVcBNwC7gVuB+5rVdrscGn6HZ/LFIFi2sLjE4ZksaImk4bOWYwa/CXwQOF17fjXwWO3xo8CNwHXA0cysZOYzwJaI2NGkVmro9PxCR+2SOtdVGETEzwBfzsyZuuZSZlZqj88C24FLgefrapbbG9VKDe0cL3fULqlz3e4ZvBW4KSI+DVwJ/CHwHXWvbwPmgTO1xyvbzzdokxo6MBmUx0YvaCuPjXJgMgpaImn4dBUGmfm6zLwhM/cATwA/DTwaEXtqJTcDx4DjwGREjETEZcBIZj4LzDaolRrat2uCu6euYGK8TAmYGC9z99QVHjyWeqiXU0tvB+6PiK3A08BDmbkUEceAx6kGz/5mtT1cDg2hfbsmXPlL66hUqVRaV/WBqampyvT0dNGLIUkDJSJOZuY1reo8A1mSZBhIkgwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kS3gN5KHh/YElrZRgMuF7fH9hgkTYnw2DArXZ/4E5X4msNFoNEGlweMxhwvbw/8FpuPL8cJHPzC1T4VpAcmZ3reDkkbTzDYMD18v7AawmWtQSJpOI5TDTgDkzGBUM70P79gVcO62wvjzG/sPiSunaCpZd7KJI2nmEw4JbH5Dsdq290fGBstMTYSInF89+6+127wbJzvMxcgxV/N3sokjaeYTAEurk/cKNhncWlCi+7ZIxLtm7p+CDwWvZQtLk58aA/GAYDqBc/nmbDN/MvLDL7q2/oeJm63UNpxRXFcOv11Gh1zzAYML368azHsE43eyircUUx/Ho5NVpr42yiAdOrWTsHJoPy2OgFbf02rOMMpeHnxIP+4Z7BgOnVj2e9hnV6aSNXFA5HFcOJB/3DMBgwvfzx9HpYp9c2akXhcFRxnHjQP7oKg4gYAx4AXgVcBNwJ/APwIFABngT2Z+b5iLgD2AucA27LzBMRcXmj2jX1ZJPo5x9Pr7euN6qvjlsXZxD2UDeLbvcM3gw8l5k/FRH/FpgFngAOZeanI+KDwC0R8SXgBmA38ErgYeA/AveurAUeWWNfNoVGP57Xv3oHh2eSX/7YE4X9mNZj63qjVhSOWxer3/dQN4tuw+BPgYfqnp8DrgYeqz1/FHgDkMDRzKwAz0TElojY0aTWMGhT/Y+nX4Y41mvreiNWFI5bS13OJsrMr2Xm2YjYRjUUDgGl2kof4CywHbgUeL7urcvtjWrVhX6ZcTPIW9eDMLNKWm9dTy2NiFcCnwI+nJkfBerH/LcB88CZ2uOV7Y1q1YV+WQn38oJ5G23frgnunrqCifEyJWBivMzdU1c4dKFNpaswiIhXAEeBt2fmA7Xm2YjYU3t8M3AMOA5MRsRIRFwGjGTms01q1YV+WQkP+tb1vl0THH/HD/LFe/Zy/B0/aBBo0+n2mME7gZcB746Id9fafgl4f0RsBZ4GHsrMpYg4BjxONXj212pvB+6vr+22A5tdv8wuclaINNhKlUqldVUfmJqaqkxPTxe9GH3JE6YkNRMRJzPzmlZ1nnQ2BPpxap4BJQ0Ww0BNdbtC75fprpLaZxiooW5W6Mvh0WjOvmf0Sv3NMNhAaxk6aee9vRya6fQkspXh0cggnHMgbVaGwQZZy9BJO+9tVdNpUHR6/kKj8FhpEM45kDYrw2Cd9WLopNlW+u1/8j+B6sq+1ZnInQZRp5doaLXVP0jnHEibkTe3WUfLW+uNVqrL2hk6aVazVKlwcPoUR2bnVt2S7+aSFZ2eRLbaVr9n9Er9zzBYR+0MnYxfMtbyc1Zb0S6v1Fc7E7lZUKwWUp1eoqFZeLzvTVd6Rq80AAyDdbTaynZZO+f8NVrR1js9v8DrX72D0or25S35ZkFRorr30kwnl2jw+j7SYPOYQQvrffLU/MIi197zyZafe9GWkaZ7GdvLYzx8co76XCkBP3Z19fP+9RvnGr6vAj2d7tmPJ79Jao9hsIqNOnlqtc9tNWWzPDZKqcRLXq8An/j7f+Lhk3NO95TUksNEq2h14PXI7BzX3vNJvusdn+Daez55wZDLoSOnOvpby7ODVn7Wascdlrf+519YbPj6V19YdLqnpLa4Z7CK1WbotNpr+OPP/t+O/95S7QBC/WettuVeAT71uS83nQbaitM9JS1zz2AVzbaaxy8Za7rXcNvHnmDXbxx9ccXerVazhJadnl9oOpNnvNx8ppIHeCXVc8+giSOzc00PvH71hUW+2mRoZvn1Xjg9v8B733TlqscMdo6Xm95LAGh4rwNDQNJKhkGd+rOFS0DRd3qoX9H/2p89xfzChSEzNlp6caW/2kyefruUtJe3lvrPpg+DZgFQdBCMjVy4ov+7L32Fj3zmmQuL2ljIfpvu6eWtpf60qY8ZrLxcRNEBUO/fXLzlgovQ/dHKIAAWz1dWvaREP+rm0hiS1t+mDoN2LhdRlPrpoodnsmlQDdp5Ap1eDVXSxtgUw0Qrx6hf/+odfOpzX+5qOuZGqVBd7n27JlZdUQ7aeQKdXg1V0sYY+j2D+qGgCtUx6o985pm+DoJlv/InT3Bkdm7VawsN2nkCnV4NVdLGGPow6OehoFbOV6rL32gFWgJ+8vsvG7iDrl7QTupPQz9MNOhj0afnF5qeRzCoK9B+m+EkaROEQT/NEOrG8hCRK1BJ66mwMIiIEeB3gNcA3wB+NjP/sZd/4/KDn+jlx224kdLgHROQNJiKPGawD7g4M38AeAfw33v54bvv+kvO9fFuwcob0ax0ydgI9/74le4NSNoQRQ4TXQf8BUBmfiYirunlh//L2W/28uN6qjw2yo9dPfGSew143SBJRSlyz+BS4Pm650sRMdDHMCbGy7zvTVfyf+7Zy0ST6aCjpRJ3T13BnfuucFaNpL5R5Mr3DLCt7vlIZja+TGgfa7Y1f2AyWl4x1IPCkvpFkXsGx4E3AkTE9wOd3RqshVds29qTzxktlS7493h5jJddMtZya9759JIGSZF7Bo8AN0XE31A9nvqWXn74Z991E7vv+suOjx1829ZR7vrR3qy03fKXNCgKC4PMPA/8wnr+jc++66b1/HhJGhpDfzkKSVJrhoEkyTCQJBkGkiQMA0kSA3TV0qeeeurZiPhS0cshSQPmO9spKlUqfXw1N0nShnCYSJJkGEiSDANJEoaBJAnDQJLEAE0t7dRG3GN5vUXELN+6AdAXgd8Ffgs4BxzNzF9v1s/aZcHbqt3QTq0QEbuB92Tmnoi4HHgQqABPAvsz83xE3AHspdqX2zLzRC9qN7Kf8JK+XgX8OfC/ai//j8z82KD3NSLGgAeAVwEXAXcC/7DW5R+gvv4/BvR7HeY9g3W9x/J6i4iLATJzT+2ftwAfBH6C6i1Dd9dWKM362UltISLibcCHgItrTfcChzLzeqqXNb+lttw3ALuBW4H7elG73n1bqUFfrwLurft+PzYkfX0z8Fzt798MfGCtyz9gfR3Y73Vo9wxY53ssb4DXAJdExFGq39OvARdl5hcAImIG+CHg37OinxFxabu1G9ull/gCMAV8uPb8auCx2uNHgTcASXXPpgI8ExFbImJHD2ofWdeevVSjvkZE3EJ1K/I2qv/PDnpf/xR4qO75uSbLNAzfa7O+DuT3Osx7BoN+j+UXgN8EJqne9+H3a23LzgLbadDPWtuZdmqL/G+SmQ8Di3VNpdqPAJov83L7Wms3VIO+ngAOZObrgP8N3MEQ9DUzv5aZZyNiG9UV5aEmyzSsfR3Y73WYw2DQ77H8eeAjmVnJzM9T/R/k2+te3wbM06CfDdqa1vbZf5P6MdBmy7zcvtbaoj2SmSeXHwO7GJK+RsQrgU8BH87MjzZZpmHt68B+r8McBut6j+UN8FZqY/oRsRO4BPjXiPjuiChR3WM4RoN+ZuYZ4Jvt1G5sl1qajYg9tcc3861lnoyIkYi4jGqAPduD2qLNRMT31R7/EHCSIehrRLwCOAq8PTMfqDUP5ffapK8D+70O0rBJp9b1Hssb4PeAByPir6nOIHgr1a2DPwJGqY4rfjYi/pbG/fyFDmr7xe3A/RGxFXgaeCgzlyLiGPA41Y2X/b2o3bAeNfeLwAci4pvAPwP/OTPPDEFf3wm8DHh3RLy71vZLwPuH8Htt1NdfAd43iN+rF6qTJA31MJEkqU2GgSTJMJAkGQaSJAwDSRKGgSQJw0CShGEgSQL+PybkHLlZS4NbAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# drop nyc data\n",
    "df_no_nyc = df_ny.drop([216])\n",
    "\n",
    "# produce new scatterplot\n",
    "plt.scatter(df_no_nyc['pop'], df_no_nyc['pc'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This plot is the same as we have seen in the previous challenge.  Nothing new here.  The property crime for NYC is around 141,000 which we have dropped."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Coefficients: \n",
      " [1.58354338e-02 1.18437047e+02 2.02472175e+00]\n",
      "\n",
      "Intercept: \n",
      " 25.21793454302474\n",
      "\n",
      "R-squared:\n",
      "0.9305586206386044\n"
     ]
    }
   ],
   "source": [
    "# Instantiate and fit our model.\n",
    "regr = linear_model.LinearRegression()\n",
    "Y = df_no_nyc['pc']\n",
    "X = df_no_nyc[['pop','murder','robbery']]\n",
    "regr.fit(X, Y)\n",
    "\n",
    "# Inspect the results.\n",
    "print('\\nCoefficients: \\n', regr.coef_)\n",
    "print('\\nIntercept: \\n', regr.intercept_)\n",
    "print('\\nR-squared:')\n",
    "print(regr.score(X, Y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, we now have an equation that predicts property crime from the input features of population, murder and robbery occurrences by city.  The only data manipulation we have done here is exclude NYC.  \n",
    "\n",
    "Next, let's build a model where we convert the murder and robbery features to **categorical** (i.e. 0 for 0 occurances and 1 for anything greater than 0 occurences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>city</th>\n",
       "      <th>pop</th>\n",
       "      <th>murder</th>\n",
       "      <th>robbery</th>\n",
       "      <th>pc</th>\n",
       "      <th>pop2</th>\n",
       "      <th>murder_c</th>\n",
       "      <th>robbery_c</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Adams Village</td>\n",
       "      <td>1861.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>3.463321e+06</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Addison Town and Village</td>\n",
       "      <td>2577.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>6.640929e+06</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Akron Village</td>\n",
       "      <td>2846.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>8.099716e+06</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Albany</td>\n",
       "      <td>97956.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>227.0</td>\n",
       "      <td>4090.0</td>\n",
       "      <td>9.595378e+09</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Albion Village</td>\n",
       "      <td>6388.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>223.0</td>\n",
       "      <td>4.080654e+07</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                       city      pop  murder  robbery      pc          pop2  \\\n",
       "0             Adams Village   1861.0     0.0      0.0    12.0  3.463321e+06   \n",
       "1  Addison Town and Village   2577.0     0.0      0.0    24.0  6.640929e+06   \n",
       "2             Akron Village   2846.0     0.0      0.0    16.0  8.099716e+06   \n",
       "3                    Albany  97956.0     8.0    227.0  4090.0  9.595378e+09   \n",
       "4            Albion Village   6388.0     0.0      4.0   223.0  4.080654e+07   \n",
       "\n",
       "   murder_c  robbery_c  \n",
       "0         0          0  \n",
       "1         0          0  \n",
       "2         0          0  \n",
       "3         1          1  \n",
       "4         0          1  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# If murders or robberies in greater than 0, convert to 1 else leave at 0\n",
    "df_no_nyc['murder_c'] = df_no_nyc['murder'].apply(lambda x : 1 if x>0 else 0)\n",
    "df_no_nyc['robbery_c'] = df_no_nyc['robbery'].apply(lambda x : 1 if x>0 else 0)\n",
    "df_no_nyc.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Coefficients: \n",
      " [ 3.43860764e-02  1.43473775e+01 -9.42521334e+01]\n",
      "\n",
      "Intercept: \n",
      " -108.42335403478899\n",
      "\n",
      "R-squared:\n",
      "0.7889428000907828\n"
     ]
    }
   ],
   "source": [
    "# Instantiate and fit our model.\n",
    "regr = linear_model.LinearRegression()\n",
    "Y = df_no_nyc['pc']\n",
    "X = df_no_nyc[['pop','murder_c','robbery_c']]\n",
    "regr.fit(X, Y)\n",
    "\n",
    "# Inspect the results.\n",
    "print('\\nCoefficients: \\n', regr.coef_)\n",
    "print('\\nIntercept: \\n', regr.intercept_)\n",
    "print('\\nR-squared:')\n",
    "print(regr.score(X, Y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see we have a different coefficients, intercept and r-squared values from the previous model.  We need to run some tests to see which model is better."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluating the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the first case - where we are keeping the features as is - let's check to see if any of the features can be dropped becuase of irrelevancy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating the model formula\n",
    "# Use a ~ to represent an '=' from the functional form\n",
    "linear_formula = 'pc ~ pop+murder+robbery'\n",
    "\n",
    "# Fit the model to our data using the formula.\n",
    "lm = smf.ols(formula=linear_formula, data=df_no_nyc).fit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the parameters, pvalues and r-squared value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Intercept     25.217935\n",
       "pop            0.015835\n",
       "murder       118.437047\n",
       "robbery        2.024722\n",
       "dtype: float64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lm.params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Intercept    1.771582e-01\n",
       "pop          1.100236e-45\n",
       "murder       4.808789e-14\n",
       "robbery      2.481759e-03\n",
       "dtype: float64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lm.pvalues"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9305586206386043"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lm.rsquared"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This looks great.  All of out p values are less than .05 which likely means they are all adding predictive value to the model.  The r-squared value is still **0.93** and not too close to 1 which makes it less likely we have overfit the data.  \n",
    "\n",
    "Now, let's look at the case where we converted the murder and robbery data to categorical and compare the results with the raw data (excluding NYC of course)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating the model formula\n",
    "# Use a ~ to represent an '=' from the functional form\n",
    "linear_formula = 'pc ~ pop+murder_c+robbery_c'\n",
    "\n",
    "# Fit the model to our data using the formula.\n",
    "lm = smf.ols(formula=linear_formula, data=df_no_nyc).fit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Intercept   -108.423354\n",
       "pop            0.034386\n",
       "murder_c      14.347378\n",
       "robbery_c    -94.252133\n",
       "dtype: float64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lm.params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Intercept     8.205933e-03\n",
       "pop          7.517475e-100\n",
       "murder_c      8.692678e-01\n",
       "robbery_c     9.624259e-02\n",
       "dtype: float64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lm.pvalues"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7889428000907827"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lm.rsquared"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As suspected, here we have a problem.  The p values for murder and robbery are **0.87 and 0.10**, respectively.  They are definitely greater than .05 and can be withdrawn from the model.  \n",
    "\n",
    "Let's withdraw murder and robbery altogether and compare the r-squared value with our r-squared value when leaving all 3 features in the model in their original form."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating the model formula\n",
    "# Use a ~ to represent an '=' from the functional form\n",
    "linear_formula = 'pc ~ pop'\n",
    "\n",
    "# Fit the model to our data using the formula.\n",
    "lm = smf.ols(formula=linear_formula, data=df_no_nyc).fit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Intercept   -155.020489\n",
       "pop            0.033890\n",
       "dtype: float64"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lm.params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Intercept     3.340745e-07\n",
       "pop          5.644411e-118\n",
       "dtype: float64"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lm.pvalues"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7872222002523548"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lm.rsquared"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Discussion and conclusion\n",
    "From our first attempt to build a model in this notebook we got the equation:  **y = 25.2 + .016(pop) + 118.4(murder) + 2.0(robbery)**  \n",
    "\n",
    "This was obtained by first dropping NYC data and then using the three features as is to build the model.  The r-squared value equaled **.931** and all of the features were significant in measuring the variance of the outcome.  \n",
    "\n",
    "Next, we built the model by converting the murder and robbery features to categorical to see if that changed the accuracy of the model.  We got an equation of:  **y = -108.4 + 14.3(pop) + 14.3(murder) -94.3(robbery)**  \n",
    "\n",
    "The r-squared value in this case was **.789** which is much less than our previous **.931**.  We also determined that the 2 categorical features did not add any value to the model becuase their p-values were greater than .05 and we proved this by removing the categorical features and calculated a new r-squared value of **.787** -- practically the same thing as when the 2 features were included in the model.  \n",
    "\n",
    "In this case, converting murder and robbery values to categorical features weakened the predictive power of our model and as it turns out, became insignificant in explaining any variance of the outcome."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}