t_testing.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from scipy.stats import ttest_ind"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hypothesis Testing\n",
    "\n",
    "Definitions:\n",
    "* A _quarter_ is a specific three month period, Q1 is January through March, Q2 is April through June, Q3 is July through September, Q4 is October through December.\n",
    "* A _recession_ is defined as starting with two consecutive quarters of GDP decline, and ending with two consecutive quarters of GDP growth.\n",
    "* A _recession bottom_ is the quarter within a recession which had the lowest GDP.\n",
    "* A _university town_ is a city which has a high percentage of university students compared to the total population of the city.\n",
    "\n",
    "**Hypothesis**: University towns have their mean housing prices less effected by recessions. Our null hypothesis is that both university and non-university towns are affected equally. We will now run a t-test to compare the ratio of the mean price of houses in university towns the quarter before the recession starts compared to the recession bottom. (`price_ratio=quarter_before_recession/recession_bottom`)\n",
    "\n",
    "The following data files is used:\n",
    "* From the [Zillow research data site](http://www.zillow.com/research/data/) there is housing data for the United States. In particular the datafile for [all homes at a city level](http://files.zillowstatic.com/research/public/City/City_Zhvi_AllHomes.csv), ```City_Zhvi_AllHomes.csv```, has median home sale prices at a fine grained level.\n",
    "* From the Wikipedia page on college towns is a list of [university towns in the United States](https://en.wikipedia.org/wiki/List_of_college_towns#College_towns_in_the_United_States) which has been copy and pasted into the file ```university_towns.txt```.\n",
    "* From Bureau of Economic Analysis, US Department of Commerce, the [GDP over time](http://www.bea.gov/national/index.htm#gdp) of the United States in current dollars (use the chained value in 2009 dollars), in quarterly intervals, in the file ```gdplev.xls```. For this assignment, only look at GDP data from the first quarter of 2000 onward.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this dictionary maps state names to two letter acronyms\n",
    "states = {'OH': 'Ohio', 'KY': 'Kentucky', 'AS': 'American Samoa', 'NV': 'Nevada', 'WY': 'Wyoming', 'NA': 'National', 'AL': 'Alabama', 'MD': 'Maryland', 'AK': 'Alaska', 'UT': 'Utah', 'OR': 'Oregon', 'MT': 'Montana', 'IL': 'Illinois', 'TN': 'Tennessee', 'DC': 'District of Columbia', 'VT': 'Vermont', 'ID': 'Idaho', 'AR': 'Arkansas', 'ME': 'Maine', 'WA': 'Washington', 'HI': 'Hawaii', 'WI': 'Wisconsin', 'MI': 'Michigan', 'IN': 'Indiana', 'NJ': 'New Jersey', 'AZ': 'Arizona', 'GU': 'Guam', 'MS': 'Mississippi', 'PR': 'Puerto Rico', 'NC': 'North Carolina', 'TX': 'Texas', 'SD': 'South Dakota', 'MP': 'Northern Mariana Islands', 'IA': 'Iowa', 'MO': 'Missouri', 'CT': 'Connecticut', 'WV': 'West Virginia', 'SC': 'South Carolina', 'LA': 'Louisiana', 'KS': 'Kansas', 'NY': 'New York', 'NE': 'Nebraska', 'OK': 'Oklahoma', 'FL': 'Florida', 'CA': 'California', 'CO': 'Colorado', 'PA': 'Pennsylvania', 'DE': 'Delaware', 'NM': 'New Mexico', 'RI': 'Rhode Island', 'MN': 'Minnesota', 'VI': 'Virgin Islands', 'NH': 'New Hampshire', 'MA': 'Massachusetts', 'GA': 'Georgia', 'ND': 'North Dakota', 'VA': 'Virginia'}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "umich_part_id": "021",
    "umich_partlist_id": "004"
   },
   "outputs": [],
   "source": [
    "def get_list_of_university_towns():\n",
    "    '''Returns a DataFrame of towns and the states they are in from the \n",
    "    university_towns.txt list. The format of the DataFrame should be:\n",
    "    DataFrame( [ [\"Michigan\", \"Ann Arbor\"], [\"Michigan\", \"Yipsilanti\"] ], \n",
    "    columns=[\"State\", \"RegionName\"]  )\n",
    "    \n",
    "    The following cleaning needed to be done:\n",
    "\n",
    "    1. For \"State\", removing characters from \"[\" to the end.\n",
    "    2. For \"RegionName\", when applicable, removing every character from \" (\" to the end.\n",
    "    3. Depending on how you read the data, you may need to remove newline character '\\n'. '''\n",
    "    \n",
    "    uni = pd.read_csv('university_towns.txt',sep=\"\\r\", header=None)\n",
    "    uni['State'] = uni[0].apply(lambda x: x.split('[')[0].strip() if x.count('[')>0 and x.count('(')==0 else np.NaN) \n",
    "    uni['RegionName'] = uni[0].apply(lambda x: x.split('(')[0].strip() if x.count('(')>0 else x)\n",
    "    uni = uni.drop(0,axis=1)\n",
    "    uni['State'] = uni['State'].fillna(method='ffill')\n",
    "    uni = uni.drop(uni[uni['RegionName'].str.contains('[', regex=False)].index)\n",
    "    uni = uni.reset_index(drop=True)\n",
    "\n",
    "    \n",
    "    return uni"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# stateStr = \"\"\"\n",
    "# Ohio, Kentucky, American Samoa, Nevada, Wyoming\n",
    "# ,National, Alabama, Maryland, Alaska, Utah\n",
    "# ,Oregon, Montana, Illinois, Tennessee, District of Columbia\n",
    "# ,Vermont, Idaho, Arkansas, Maine, Washington\n",
    "# ,Hawaii, Wisconsin, Michigan, Indiana, New Jersey\n",
    "# ,Arizona, Guam, Mississippi, Puerto Rico, North Carolina\n",
    "# ,Texas, South Dakota, Northern Mariana Islands, Iowa, Missouri\n",
    "# ,Connecticut, West Virginia, South Carolina, Louisiana, Kansas\n",
    "# ,New York, Nebraska, Oklahoma, Florida, California\n",
    "# ,Colorado, Pennsylvania, Delaware, New Mexico, Rhode Island\n",
    "# ,Minnesota, Virgin Islands, New Hampshire, Massachusetts, Georgia\n",
    "# ,North Dakota, Virginia\n",
    "# \"\"\"\n",
    "# #list of regionName entries string length\n",
    "# regNmLenStr = \"\"\"\n",
    "# 06,08,12,10,10,04,10,08,09,09,05,06,11,06,12,09,08,10,12,06,\n",
    "# 06,06,08,05,09,06,05,06,10,28,06,06,09,06,08,09,10,35,09,15,\n",
    "# 13,10,07,21,08,07,07,07,12,06,14,07,08,16,09,10,11,09,10,06,\n",
    "# 11,05,06,09,10,12,06,06,11,07,08,13,07,11,05,06,06,07,10,08,\n",
    "# 11,08,13,12,06,04,08,10,08,07,12,05,06,09,07,10,16,10,06,12,\n",
    "# 08,07,06,06,06,11,14,11,07,06,06,12,08,10,11,06,10,14,04,11,\n",
    "# 18,07,07,08,09,06,13,11,12,10,07,12,07,04,08,09,09,13,08,10,\n",
    "# 16,09,10,08,06,08,12,07,11,09,07,09,06,12,06,09,07,10,09,10,\n",
    "# 09,06,15,05,10,09,11,12,10,10,09,13,06,09,11,06,11,09,13,37,\n",
    "# 06,13,06,09,49,07,11,12,09,11,11,07,12,10,06,06,09,04,09,15,\n",
    "# 10,12,05,09,08,09,09,07,14,06,07,16,12,09,07,09,06,32,07,08,\n",
    "# 08,06,10,36,09,10,09,06,09,11,09,06,10,07,14,08,07,06,10,09,\n",
    "# 05,11,07,06,08,07,05,07,07,04,06,05,09,04,25,06,07,08,05,08,\n",
    "# 06,05,11,09,07,07,06,13,09,05,16,05,10,09,08,11,06,06,06,10,\n",
    "# 09,07,06,07,10,05,08,07,06,08,06,30,09,07,06,11,07,12,08,09,\n",
    "# 16,12,11,08,06,04,10,10,15,05,11,11,09,08,06,04,10,10,07,09,\n",
    "# 11,08,26,07,13,05,11,03,08,07,06,05,08,13,10,08,08,08,07,07,\n",
    "# 09,05,04,11,11,07,06,10,11,03,04,06,06,08,08,06,10,09,05,11,\n",
    "# 07,09,06,12,13,09,10,11,08,07,07,08,09,10,08,10,08,56,07,12,\n",
    "# 07,16,08,04,10,10,10,10,07,09,08,09,09,10,07,09,09,09,12,14,\n",
    "# 10,29,19,07,11,12,13,13,09,10,12,12,12,08,10,07,10,07,07,08,\n",
    "# 08,08,09,10,09,11,09,07,09,10,11,11,10,09,09,12,09,06,08,07,\n",
    "# 12,09,07,07,06,06,08,06,15,08,06,06,10,10,10,07,05,10,07,11,\n",
    "# 09,12,10,12,04,10,05,05,04,14,07,10,09,07,11,10,10,10,11,15,\n",
    "# 09,14,12,09,09,07,12,04,10,10,06,10,07,28,06,10,08,09,10,10,\n",
    "# 10,13,12,08,10,09,09,07,09,09,07,11,11,13,08,10,07\n",
    "# \"\"\"\n",
    "\n",
    "# df = get_list_of_university_towns()\n",
    "\n",
    "# cols = [\"State\", \"RegionName\"]\n",
    "\n",
    "# print('Shape test: ', \"Passed\" if df.shape ==\n",
    "#       (517, 2) else 'Failed')\n",
    "# print('Index test: ',\n",
    "#       \"Passed\" if df.index.tolist() == list(range(517))\n",
    "#       else 'Failed')\n",
    "\n",
    "# print('Column test: ',\n",
    "#       \"Passed\" if df.columns.tolist() == cols else 'Failed')\n",
    "# print('\\\\n test: ',\n",
    "#       \"Failed\" if any(df[cols[0]].str.contains(\n",
    "#           '\\n')) or any(df[cols[1]].str.contains('\\n'))\n",
    "#       else 'Passed')\n",
    "# print('Trailing whitespace test:',\n",
    "#       \"Failed\" if any(df[cols[0]].str.contains(\n",
    "#           '\\s+$')) or any(df[cols[1]].str.contains(\n",
    "#               '\\s+$'))\n",
    "#       else 'Passed')\n",
    "# print('\"(\" test:',\n",
    "#       \"Failed\" if any(df[cols[0]].str.contains(\n",
    "#           '\\(')) or any(df[cols[1]].str.contains(\n",
    "#               '\\('))\n",
    "#       else 'Passed')\n",
    "# print('\"[\" test:',\n",
    "#       \"Failed\" if any(df[cols[0]].str.contains(\n",
    "#           '\\[')) or any(df[cols[1]].str.contains(\n",
    "#               '\\]'))\n",
    "#       else 'Passed')\n",
    "\n",
    "# states_vlist = [st.strip() for st in stateStr.split(',')]\n",
    "\n",
    "# mismatchedStates = df[~df['State'].isin(\n",
    "#     states_vlist)].loc[:, 'State'].unique()\n",
    "# print('State test: ', \"Passed\" if len(\n",
    "#     mismatchedStates) == 0 else \"Failed\")\n",
    "# if len(mismatchedStates) > 0:\n",
    "#     print()\n",
    "#     print('The following states failed the equality test:')\n",
    "#     print()\n",
    "#     print('\\n'.join(mismatchedStates))\n",
    "\n",
    "# df['expected_length'] = [int(s.strip())\n",
    "#                          for s in regNmLenStr.split(',')\n",
    "#                          if s.strip().isdigit()]\n",
    "# regDiff = df[df['RegionName'].str.len() != df['expected_length']].loc[\n",
    "#     :, ['RegionName', 'expected_length']]\n",
    "# regDiff['actual_length'] = regDiff['RegionName'].str.len()\n",
    "# print('RegionName test: ', \"Passed\" if len(regDiff) ==\n",
    "#       0 else ' \\nMismatching regionNames\\n {}'.format(regDiff))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "umich_part_id": "022",
    "umich_partlist_id": "004"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2008q3'"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def get_recession_start():\n",
    "    '''Returns the year and quarter of the recession start time as a \n",
    "    string value in a format such as 2005q3'''\n",
    "    recess = pd.read_excel('gdplev.xls',header = 0,skiprows = [0,1,2,3,4,6,7]).drop(['Unnamed: 0','Unnamed: 3','Unnamed: 7',\n",
    "                                                                                      'GDP in billions of current dollars',\n",
    "                                                                                      'GDP in billions of chained 2009 dollars',\n",
    "                                                                                      'GDP in billions of current dollars.1'],\n",
    "                                                                                     axis=1)                                                                       \n",
    "\n",
    "    recess = recess.rename(columns={'GDP in billions of chained 2009 dollars.1':'GDP in billions of chained 2009 dollars.',\n",
    "                                'Unnamed: 4':'Quarter'}).drop(recess.iloc[0:212].index).reset_index(drop=True)\n",
    "    GDP_val = list()\n",
    "\n",
    "    for i in range(len(recess['GDP in billions of chained 2009 dollars.'])-2):\n",
    "        if recess['GDP in billions of chained 2009 dollars.'].iloc[i+1]<recess['GDP in billions of chained 2009 dollars.'].iloc[i] and recess['GDP in billions of chained 2009 dollars.'].iloc[i+2]<recess['GDP in billions of chained 2009 dollars.'].iloc[i+1]:\n",
    "            GDP_val.append(recess['GDP in billions of chained 2009 dollars.'].iloc[i+1])\n",
    "        \n",
    "    start_recess = recess[recess['GDP in billions of chained 2009 dollars.']==GDP_val[0]].values[0][0]\n",
    "    \n",
    "    return start_recess\n",
    "get_recession_start()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "umich_part_id": "023",
    "umich_partlist_id": "004"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2009q4'"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def get_recession_end():\n",
    "    recess = pd.read_excel('gdplev.xls',header = 0,skiprows = [0,1,2,3,4,6,7]).drop(['Unnamed: 0','Unnamed: 3',\n",
    "                                                                                    'Unnamed: 7',\n",
    "                                                                                 'GDP in billions of current dollars',\n",
    "                                                                                 'GDP in billions of chained 2009 dollars',\n",
    "                                                                                'GDP in billions of current dollars.1'],\n",
    "                                                                                axis=1)\n",
    "\n",
    "    recess = recess.rename(columns={'GDP in billions of chained 2009 dollars.1':'GDP in billions of chained 2009 dollars.',\n",
    "                                'Unnamed: 4':'Quarter'}).drop(recess.iloc[0:212].index).reset_index(drop=True)\n",
    "    start_recess = get_recession_start()\n",
    "    start_recess_indx = recess[recess['Quarter']==start_recess].index[0]\n",
    "    GDP_val = list()\n",
    "\n",
    "    for i in range(start_recess_indx,len(recess['GDP in billions of chained 2009 dollars.'])-2):\n",
    "        if recess['GDP in billions of chained 2009 dollars.'].iloc[i+1]>recess['GDP in billions of chained 2009 dollars.'].iloc[i] and recess['GDP in billions of chained 2009 dollars.'].iloc[i+2]>recess['GDP in billions of chained 2009 dollars.'].iloc[i+1]:\n",
    "            GDP_val.append(recess['GDP in billions of chained 2009 dollars.'].iloc[i+2])\n",
    "        \n",
    "    end_recess = recess[recess['GDP in billions of chained 2009 dollars.']==GDP_val[0]].values[0][0]\n",
    "      \n",
    "    '''Returns the year and quarter of the recession end time as a \n",
    "    string value in a format such as 2005q3'''\n",
    "       \n",
    "    return end_recess\n",
    "get_recession_end()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "umich_part_id": "024",
    "umich_partlist_id": "004"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2009q2'"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def get_recession_bottom():\n",
    "    recess = pd.read_excel('gdplev.xls',header = 0,skiprows = [0,1,2,3,4,6,7]).drop(['Unnamed: 0','Unnamed: 3',\n",
    "                                                                                    'Unnamed: 7',\n",
    "                                                                                 'GDP in billions of current dollars',\n",
    "                                                                                 'GDP in billions of chained 2009 dollars',\n",
    "                                                                                'GDP in billions of current dollars.1'],\n",
    "                                                                                axis=1)\n",
    "\n",
    "    recess = recess.rename(columns={'GDP in billions of chained 2009 dollars.1':'GDP in billions of chained 2009 dollars.',\n",
    "                                'Unnamed: 4':'Quarter'}).drop(recess.iloc[0:212].index).reset_index(drop=True)\n",
    "    start_recess = get_recession_start()\n",
    "    start_recess_indx = recess[recess['Quarter']==start_recess].index[0]\n",
    "    \n",
    "    end_recess = get_recession_end()\n",
    "    end_recess_indx = recess[recess['Quarter']==end_recess].index[0]\n",
    "    \n",
    "    recess_time = list()\n",
    "    \n",
    "    for i in range(start_recess_indx, end_recess_indx+1):\n",
    "        recess_time.append(recess['GDP in billions of chained 2009 dollars.'].iloc[i])\n",
    "    \n",
    "    recess_bottom_val = min(recess_time)\n",
    "    recess_bottom_Q = recess[recess['GDP in billions of chained 2009 dollars.']==recess_bottom_val].values[0][0]\n",
    "        \n",
    "    \n",
    "    '''Returns the year and quarter of the recession bottom time as a \n",
    "    string value in a format such as 2005q3'''\n",
    "    \n",
    "    return recess_bottom_Q\n",
    "get_recession_bottom()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "scrolled": true,
    "umich_part_id": "025",
    "umich_partlist_id": "004"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>2000q1</th>\n",
       "      <th>2000q2</th>\n",
       "      <th>2000q3</th>\n",
       "      <th>2000q4</th>\n",
       "      <th>2001q1</th>\n",
       "      <th>2001q2</th>\n",
       "      <th>2001q3</th>\n",
       "      <th>2001q4</th>\n",
       "      <th>2002q1</th>\n",
       "      <th>2002q2</th>\n",
       "      <th>...</th>\n",
       "      <th>2014q2</th>\n",
       "      <th>2014q3</th>\n",
       "      <th>2014q4</th>\n",
       "      <th>2015q1</th>\n",
       "      <th>2015q2</th>\n",
       "      <th>2015q3</th>\n",
       "      <th>2015q4</th>\n",
       "      <th>2016q1</th>\n",
       "      <th>2016q2</th>\n",
       "      <th>2016q3</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>State</th>\n",
       "      <th>RegionName</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>New York</th>\n",
       "      <th>New York</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>515466.666667</td>\n",
       "      <td>522800.000000</td>\n",
       "      <td>528066.666667</td>\n",
       "      <td>532266.666667</td>\n",
       "      <td>540800.000000</td>\n",
       "      <td>557200.000000</td>\n",
       "      <td>572833.333333</td>\n",
       "      <td>582866.666667</td>\n",
       "      <td>591633.333333</td>\n",
       "      <td>587200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>California</th>\n",
       "      <th>Los Angeles</th>\n",
       "      <td>207066.666667</td>\n",
       "      <td>214466.666667</td>\n",
       "      <td>220966.666667</td>\n",
       "      <td>226166.666667</td>\n",
       "      <td>233000.000000</td>\n",
       "      <td>239100.000000</td>\n",
       "      <td>245066.666667</td>\n",
       "      <td>253033.333333</td>\n",
       "      <td>261966.666667</td>\n",
       "      <td>272700.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>498033.333333</td>\n",
       "      <td>509066.666667</td>\n",
       "      <td>518866.666667</td>\n",
       "      <td>528800.000000</td>\n",
       "      <td>538166.666667</td>\n",
       "      <td>547266.666667</td>\n",
       "      <td>557733.333333</td>\n",
       "      <td>566033.333333</td>\n",
       "      <td>577466.666667</td>\n",
       "      <td>584050.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Illinois</th>\n",
       "      <th>Chicago</th>\n",
       "      <td>138400.000000</td>\n",
       "      <td>143633.333333</td>\n",
       "      <td>147866.666667</td>\n",
       "      <td>152133.333333</td>\n",
       "      <td>156933.333333</td>\n",
       "      <td>161800.000000</td>\n",
       "      <td>166400.000000</td>\n",
       "      <td>170433.333333</td>\n",
       "      <td>175500.000000</td>\n",
       "      <td>177566.666667</td>\n",
       "      <td>...</td>\n",
       "      <td>192633.333333</td>\n",
       "      <td>195766.666667</td>\n",
       "      <td>201266.666667</td>\n",
       "      <td>201066.666667</td>\n",
       "      <td>206033.333333</td>\n",
       "      <td>208300.000000</td>\n",
       "      <td>207900.000000</td>\n",
       "      <td>206066.666667</td>\n",
       "      <td>208200.000000</td>\n",
       "      <td>212000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pennsylvania</th>\n",
       "      <th>Philadelphia</th>\n",
       "      <td>53000.000000</td>\n",
       "      <td>53633.333333</td>\n",
       "      <td>54133.333333</td>\n",
       "      <td>54700.000000</td>\n",
       "      <td>55333.333333</td>\n",
       "      <td>55533.333333</td>\n",
       "      <td>56266.666667</td>\n",
       "      <td>57533.333333</td>\n",
       "      <td>59133.333333</td>\n",
       "      <td>60733.333333</td>\n",
       "      <td>...</td>\n",
       "      <td>113733.333333</td>\n",
       "      <td>115300.000000</td>\n",
       "      <td>115666.666667</td>\n",
       "      <td>116200.000000</td>\n",
       "      <td>117966.666667</td>\n",
       "      <td>121233.333333</td>\n",
       "      <td>122200.000000</td>\n",
       "      <td>123433.333333</td>\n",
       "      <td>126933.333333</td>\n",
       "      <td>128700.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Arizona</th>\n",
       "      <th>Phoenix</th>\n",
       "      <td>111833.333333</td>\n",
       "      <td>114366.666667</td>\n",
       "      <td>116000.000000</td>\n",
       "      <td>117400.000000</td>\n",
       "      <td>119600.000000</td>\n",
       "      <td>121566.666667</td>\n",
       "      <td>122700.000000</td>\n",
       "      <td>124300.000000</td>\n",
       "      <td>126533.333333</td>\n",
       "      <td>128366.666667</td>\n",
       "      <td>...</td>\n",
       "      <td>164266.666667</td>\n",
       "      <td>165366.666667</td>\n",
       "      <td>168500.000000</td>\n",
       "      <td>171533.333333</td>\n",
       "      <td>174166.666667</td>\n",
       "      <td>179066.666667</td>\n",
       "      <td>183833.333333</td>\n",
       "      <td>187900.000000</td>\n",
       "      <td>191433.333333</td>\n",
       "      <td>195200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Wisconsin</th>\n",
       "      <th>Town of Wrightstown</th>\n",
       "      <td>101766.666667</td>\n",
       "      <td>105400.000000</td>\n",
       "      <td>111366.666667</td>\n",
       "      <td>114866.666667</td>\n",
       "      <td>125966.666667</td>\n",
       "      <td>129900.000000</td>\n",
       "      <td>129900.000000</td>\n",
       "      <td>129433.333333</td>\n",
       "      <td>131900.000000</td>\n",
       "      <td>134200.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>144866.666667</td>\n",
       "      <td>146866.666667</td>\n",
       "      <td>149233.333333</td>\n",
       "      <td>148666.666667</td>\n",
       "      <td>149333.333333</td>\n",
       "      <td>149866.666667</td>\n",
       "      <td>149933.333333</td>\n",
       "      <td>149833.333333</td>\n",
       "      <td>151266.666667</td>\n",
       "      <td>155000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>New York</th>\n",
       "      <th>Urbana</th>\n",
       "      <td>79200.000000</td>\n",
       "      <td>81666.666667</td>\n",
       "      <td>91700.000000</td>\n",
       "      <td>98366.666667</td>\n",
       "      <td>94866.666667</td>\n",
       "      <td>98533.333333</td>\n",
       "      <td>102966.666667</td>\n",
       "      <td>98033.333333</td>\n",
       "      <td>93966.666667</td>\n",
       "      <td>94600.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>132133.333333</td>\n",
       "      <td>137033.333333</td>\n",
       "      <td>140066.666667</td>\n",
       "      <td>141700.000000</td>\n",
       "      <td>137866.666667</td>\n",
       "      <td>136466.666667</td>\n",
       "      <td>136166.666667</td>\n",
       "      <td>138966.666667</td>\n",
       "      <td>144200.000000</td>\n",
       "      <td>143000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Wisconsin</th>\n",
       "      <th>New Denmark</th>\n",
       "      <td>114566.666667</td>\n",
       "      <td>119266.666667</td>\n",
       "      <td>126066.666667</td>\n",
       "      <td>131966.666667</td>\n",
       "      <td>143800.000000</td>\n",
       "      <td>146966.666667</td>\n",
       "      <td>148366.666667</td>\n",
       "      <td>149166.666667</td>\n",
       "      <td>153133.333333</td>\n",
       "      <td>156733.333333</td>\n",
       "      <td>...</td>\n",
       "      <td>174566.666667</td>\n",
       "      <td>181166.666667</td>\n",
       "      <td>186166.666667</td>\n",
       "      <td>187600.000000</td>\n",
       "      <td>188666.666667</td>\n",
       "      <td>188433.333333</td>\n",
       "      <td>188933.333333</td>\n",
       "      <td>191066.666667</td>\n",
       "      <td>192833.333333</td>\n",
       "      <td>197600.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>California</th>\n",
       "      <th>Angels</th>\n",
       "      <td>151000.000000</td>\n",
       "      <td>155900.000000</td>\n",
       "      <td>158100.000000</td>\n",
       "      <td>167466.666667</td>\n",
       "      <td>176833.333333</td>\n",
       "      <td>183766.666667</td>\n",
       "      <td>190233.333333</td>\n",
       "      <td>184566.666667</td>\n",
       "      <td>184033.333333</td>\n",
       "      <td>186133.333333</td>\n",
       "      <td>...</td>\n",
       "      <td>244466.666667</td>\n",
       "      <td>254066.666667</td>\n",
       "      <td>259933.333333</td>\n",
       "      <td>260100.000000</td>\n",
       "      <td>250633.333333</td>\n",
       "      <td>263500.000000</td>\n",
       "      <td>279500.000000</td>\n",
       "      <td>276533.333333</td>\n",
       "      <td>271600.000000</td>\n",
       "      <td>269950.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Wisconsin</th>\n",
       "      <th>Holland</th>\n",
       "      <td>151033.333333</td>\n",
       "      <td>150500.000000</td>\n",
       "      <td>153233.333333</td>\n",
       "      <td>155833.333333</td>\n",
       "      <td>161866.666667</td>\n",
       "      <td>165733.333333</td>\n",
       "      <td>168033.333333</td>\n",
       "      <td>167400.000000</td>\n",
       "      <td>165766.666667</td>\n",
       "      <td>161966.666667</td>\n",
       "      <td>...</td>\n",
       "      <td>201266.666667</td>\n",
       "      <td>201566.666667</td>\n",
       "      <td>201266.666667</td>\n",
       "      <td>206000.000000</td>\n",
       "      <td>207600.000000</td>\n",
       "      <td>212866.666667</td>\n",
       "      <td>217833.333333</td>\n",
       "      <td>221966.666667</td>\n",
       "      <td>228033.333333</td>\n",
       "      <td>234950.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10730 rows × 67 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                         2000q1         2000q2         2000q3  \\\n",
       "State        RegionName                                                         \n",
       "New York     New York                       NaN            NaN            NaN   \n",
       "California   Los Angeles          207066.666667  214466.666667  220966.666667   \n",
       "Illinois     Chicago              138400.000000  143633.333333  147866.666667   \n",
       "Pennsylvania Philadelphia          53000.000000   53633.333333   54133.333333   \n",
       "Arizona      Phoenix              111833.333333  114366.666667  116000.000000   \n",
       "...                                         ...            ...            ...   \n",
       "Wisconsin    Town of Wrightstown  101766.666667  105400.000000  111366.666667   \n",
       "New York     Urbana                79200.000000   81666.666667   91700.000000   \n",
       "Wisconsin    New Denmark          114566.666667  119266.666667  126066.666667   \n",
       "California   Angels               151000.000000  155900.000000  158100.000000   \n",
       "Wisconsin    Holland              151033.333333  150500.000000  153233.333333   \n",
       "\n",
       "                                         2000q4         2001q1         2001q2  \\\n",
       "State        RegionName                                                         \n",
       "New York     New York                       NaN            NaN            NaN   \n",
       "California   Los Angeles          226166.666667  233000.000000  239100.000000   \n",
       "Illinois     Chicago              152133.333333  156933.333333  161800.000000   \n",
       "Pennsylvania Philadelphia          54700.000000   55333.333333   55533.333333   \n",
       "Arizona      Phoenix              117400.000000  119600.000000  121566.666667   \n",
       "...                                         ...            ...            ...   \n",
       "Wisconsin    Town of Wrightstown  114866.666667  125966.666667  129900.000000   \n",
       "New York     Urbana                98366.666667   94866.666667   98533.333333   \n",
       "Wisconsin    New Denmark          131966.666667  143800.000000  146966.666667   \n",
       "California   Angels               167466.666667  176833.333333  183766.666667   \n",
       "Wisconsin    Holland              155833.333333  161866.666667  165733.333333   \n",
       "\n",
       "                                         2001q3         2001q4         2002q1  \\\n",
       "State        RegionName                                                         \n",
       "New York     New York                       NaN            NaN            NaN   \n",
       "California   Los Angeles          245066.666667  253033.333333  261966.666667   \n",
       "Illinois     Chicago              166400.000000  170433.333333  175500.000000   \n",
       "Pennsylvania Philadelphia          56266.666667   57533.333333   59133.333333   \n",
       "Arizona      Phoenix              122700.000000  124300.000000  126533.333333   \n",
       "...                                         ...            ...            ...   \n",
       "Wisconsin    Town of Wrightstown  129900.000000  129433.333333  131900.000000   \n",
       "New York     Urbana               102966.666667   98033.333333   93966.666667   \n",
       "Wisconsin    New Denmark          148366.666667  149166.666667  153133.333333   \n",
       "California   Angels               190233.333333  184566.666667  184033.333333   \n",
       "Wisconsin    Holland              168033.333333  167400.000000  165766.666667   \n",
       "\n",
       "                                         2002q2  ...         2014q2  \\\n",
       "State        RegionName                          ...                  \n",
       "New York     New York                       NaN  ...  515466.666667   \n",
       "California   Los Angeles          272700.000000  ...  498033.333333   \n",
       "Illinois     Chicago              177566.666667  ...  192633.333333   \n",
       "Pennsylvania Philadelphia          60733.333333  ...  113733.333333   \n",
       "Arizona      Phoenix              128366.666667  ...  164266.666667   \n",
       "...                                         ...  ...            ...   \n",
       "Wisconsin    Town of Wrightstown  134200.000000  ...  144866.666667   \n",
       "New York     Urbana                94600.000000  ...  132133.333333   \n",
       "Wisconsin    New Denmark          156733.333333  ...  174566.666667   \n",
       "California   Angels               186133.333333  ...  244466.666667   \n",
       "Wisconsin    Holland              161966.666667  ...  201266.666667   \n",
       "\n",
       "                                         2014q3         2014q4         2015q1  \\\n",
       "State        RegionName                                                         \n",
       "New York     New York             522800.000000  528066.666667  532266.666667   \n",
       "California   Los Angeles          509066.666667  518866.666667  528800.000000   \n",
       "Illinois     Chicago              195766.666667  201266.666667  201066.666667   \n",
       "Pennsylvania Philadelphia         115300.000000  115666.666667  116200.000000   \n",
       "Arizona      Phoenix              165366.666667  168500.000000  171533.333333   \n",
       "...                                         ...            ...            ...   \n",
       "Wisconsin    Town of Wrightstown  146866.666667  149233.333333  148666.666667   \n",
       "New York     Urbana               137033.333333  140066.666667  141700.000000   \n",
       "Wisconsin    New Denmark          181166.666667  186166.666667  187600.000000   \n",
       "California   Angels               254066.666667  259933.333333  260100.000000   \n",
       "Wisconsin    Holland              201566.666667  201266.666667  206000.000000   \n",
       "\n",
       "                                         2015q2         2015q3         2015q4  \\\n",
       "State        RegionName                                                         \n",
       "New York     New York             540800.000000  557200.000000  572833.333333   \n",
       "California   Los Angeles          538166.666667  547266.666667  557733.333333   \n",
       "Illinois     Chicago              206033.333333  208300.000000  207900.000000   \n",
       "Pennsylvania Philadelphia         117966.666667  121233.333333  122200.000000   \n",
       "Arizona      Phoenix              174166.666667  179066.666667  183833.333333   \n",
       "...                                         ...            ...            ...   \n",
       "Wisconsin    Town of Wrightstown  149333.333333  149866.666667  149933.333333   \n",
       "New York     Urbana               137866.666667  136466.666667  136166.666667   \n",
       "Wisconsin    New Denmark          188666.666667  188433.333333  188933.333333   \n",
       "California   Angels               250633.333333  263500.000000  279500.000000   \n",
       "Wisconsin    Holland              207600.000000  212866.666667  217833.333333   \n",
       "\n",
       "                                         2016q1         2016q2    2016q3  \n",
       "State        RegionName                                                   \n",
       "New York     New York             582866.666667  591633.333333  587200.0  \n",
       "California   Los Angeles          566033.333333  577466.666667  584050.0  \n",
       "Illinois     Chicago              206066.666667  208200.000000  212000.0  \n",
       "Pennsylvania Philadelphia         123433.333333  126933.333333  128700.0  \n",
       "Arizona      Phoenix              187900.000000  191433.333333  195200.0  \n",
       "...                                         ...            ...       ...  \n",
       "Wisconsin    Town of Wrightstown  149833.333333  151266.666667  155000.0  \n",
       "New York     Urbana               138966.666667  144200.000000  143000.0  \n",
       "Wisconsin    New Denmark          191066.666667  192833.333333  197600.0  \n",
       "California   Angels               276533.333333  271600.000000  269950.0  \n",
       "Wisconsin    Holland              221966.666667  228033.333333  234950.0  \n",
       "\n",
       "[10730 rows x 67 columns]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def convert_housing_data_to_quarters():\n",
    "    housing_data = pd.read_csv('City_Zhvi_AllHomes.csv')\n",
    "    housing_data.drop(['RegionID','Metro','CountyName','SizeRank'],axis=1, inplace = True)\n",
    "    start_drop = pd.Period('1996-04')\n",
    "    end_drop = pd.Period('1999-12')\n",
    "    drop_range = pd.period_range(start_drop, end_drop, freq = 'M').strftime('%Y-%m')\n",
    "    housing_data = housing_data.drop(drop_range,axis=1)\n",
    "    housing_data['State'] = housing_data['State'].replace(states)\n",
    "\n",
    "    housing_data.set_index(['State','RegionName'],inplace=True)\n",
    "    start_date = pd.Period('2000-01')\n",
    "    end_date = pd.Period('2016-08')\n",
    "    time_span = pd.period_range(start_date,end_date,freq=\"M\")\n",
    "    housing_data.columns = time_span\n",
    "    housing_data = housing_data.resample('Q',axis=1).mean()\n",
    "    housing_data.columns = housing_data.columns.strftime('%Yq%q')\n",
    "    '''Converts the housing data to quarters and returns it as mean \n",
    "    values in a dataframe. This dataframe should be a dataframe with\n",
    "    columns for 2000q1 through 2016q, with a multi-index\n",
    "    in the shape of [\"State\",\"RegionName\"].\n",
    "    \n",
    "    The resulting dataframe is 67x10730.\n",
    "    '''\n",
    "    \n",
    "    return housing_data\n",
    "convert_housing_data_to_quarters()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "umich_part_id": "026",
    "umich_partlist_id": "004"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(True, 0.002724063704761164)"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def run_ttest():\n",
    "    \n",
    "    '''First creates new data showing the decline or growth of housing prices\n",
    "    between the recession start and the recession bottom. Then runs a ttest\n",
    "    comparing the university town values to the non-university towns values, \n",
    "    return whether we fail to reject the null hypothesis (false) or reject the null hypothesis (true)\n",
    "    as well as the p-value of the confidence. \n",
    "    '''\n",
    "    university_towns = get_list_of_university_towns()\n",
    "    housing_data = convert_housing_data_to_quarters()\n",
    "\n",
    "    unis = university_towns.to_records(index=False).tolist()\n",
    "\n",
    "    uni_housing_data = university_towns.join(housing_data,on=['State','RegionName'],how='inner').set_index(['State','RegionName'])\n",
    "    non_uni_housing_data = housing_data[~housing_data.index.isin(unis)]\n",
    "\n",
    "    recession_start = get_recession_start()\n",
    "    recession_bottom = get_recession_bottom()\n",
    "\n",
    "    indx = uni_housing_data.columns.get_loc(recession_start)-1\n",
    "    before_recession = uni_housing_data.columns[indx]\n",
    "\n",
    "    uni_housing_data = uni_housing_data[[before_recession,recession_bottom]]\n",
    "    non_uni_housing_data = non_uni_housing_data[[before_recession,recession_bottom]]\n",
    "\n",
    "    uni_housing_data['Price Ratio'] = uni_housing_data[before_recession]/uni_housing_data[recession_bottom]\n",
    "    non_uni_housing_data['Price Ratio'] = non_uni_housing_data[before_recession]/non_uni_housing_data[recession_bottom]\n",
    "\n",
    "\n",
    "    ttest_statistic,p = ttest_ind(uni_housing_data['Price Ratio'],non_uni_housing_data['Price Ratio'],nan_policy='omit')\n",
    "    different = str()\n",
    "\n",
    "\n",
    "    if p<0.01:\n",
    "        different = True\n",
    "    else:\n",
    "        different = False\n",
    "\n",
    "    \n",
    "\n",
    "    \n",
    "    return (different,p)\n",
    "run_ttest()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "coursera": {
   "course_slug": "python-data-analysis",
   "graded_item_id": "Il9Fx",
   "launcher_item_id": "TeDW0",
   "part_id": "WGlun"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  },
  "umich": {
   "id": "Assignment 4",
   "version": "1.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}