NB2_Conclusion.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# Using Statistical Methods To Find Pay Gap Within The Organization   \r\n",
    "\r\n",
    "By Niladri Ghosh\r\n",
    "\r\n",
    "## Objective\r\n",
    "\r\n",
    "Our main motive in this section is to perform hypothesis test. And the whole process can be summarized into a list of steps; provided below. We begin with a hypothesis (H0) and an alternative hypothesis (H1), our objective is to proove our hypothesis (H0) wrong which implies that alternative hypothesis is true, simple case of if and else.\r\n",
    "\r\n",
    "### Let's begin our Hypothesis Testing.\r\n",
    "\r\n",
    "Before proceeding lets assess the situations. The all of the condition satisfies :\r\n",
    "* Unknown Variance, since sample data and assumed to be equal\r\n",
    "* dependant samples\r\n",
    "\r\n",
    "Therefor we can perform  Student's T Test. Calculate T value , p value to infer the hypothesis.\r\n",
    "\r\n",
    "__Process :__\r\n",
    "\r\n",
    "1. Calucate n (sample size) for both dataset.\r\n",
    "2. Standard Deviation , s<sub>x</sub><sup>2</sup>, s<sub>y</sub><sup>2</sup>\r\n",
    "3. Pooled Variance\r\n",
    "\r\n",
    "   Formula : $$s_{p}^2 = \\frac {(n_{x} - 1)s_{x}^2 + (n_{y} - 1)s_{y}^2}{n_{x} + n_{y} - 2}$$\r\n",
    "\r\n",
    "4. Standard Error \r\n",
    "\r\n",
    "    Formula : $$std. err = \\sqrt { \\frac { s_{p}^2 }{n_{x}} + \\frac { s_{p}^2 }{n_{y}}}$$\r\n",
    "\r\n",
    "5. Degree of freedom \r\n",
    "\r\n",
    "    Formula : $$df = n_{x} + n_{y} -2$$\r\n",
    "\r\n",
    "6. T Statistic Value \r\n",
    "\r\n",
    "    Formula : $$T = \\frac {\\bar d - \\mu_{o}}{std.err} $$\r\n",
    "\r\n",
    "7. Calculate P Value\r\n",
    "\r\n",
    "    * Refer T Table for P value significance.\r\n",
    "\r\n",
    "\r\n",
    "__We choose significance value of 0.05, 95%. α = 0.05__\r\n",
    "\r\n",
    "</br>\r\n",
    "\r\n",
    "\r\n",
    "__*Thumb Rule*__\r\n",
    "\r\n",
    "1. If T score greater than 2, or lower than -2 we can easily reject the null Hypothesis.\r\n",
    "2. P value lower than 0.0000 is extreamly significant, although we consider only upto 3.\r\n",
    "    \r\n",
    "</br>"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "__Getting Started : Load libraries and set options__"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "source": [
    "# import necessary libraries\r\n",
    "import numpy as np\r\n",
    "import pandas as pd\r\n",
    "import math\r\n",
    "from scipy.stats import sem\r\n",
    "from scipy.stats import t\r\n",
    "\r\n",
    "# default='warn'\r\n",
    "pd.options.mode.chained_assignment = None  \r\n",
    "pd.set_option('display.max_columns', None)\r\n",
    "\r\n",
    "import warnings\r\n",
    "warnings.filterwarnings('ignore')"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "__Load Dataset__\r\n",
    "\r\n",
    "First load the supplied CSV file using Pandas read_csv function."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "source": [
    "# read data\r\n",
    "df = pd.read_csv('data/evidence_clean.csv',delimiter=',')"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Check data"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "source": [
    "df.head(10)"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "   Age  Gender Ethnicity  Start_date                      Position    Salary\n",
       "0   51    Male     White  2011-08-15  Software Engineering Manager   56160.0\n",
       "1   30  Female     White  2013-11-11             Software Engineer  116480.0\n",
       "2   31  Female     White  2012-05-11             Software Engineer  102440.0\n",
       "3   34    Male     White  2014-07-07             Software Engineer   99840.0\n",
       "4   38  Female     White  2014-10-11             Software Engineer   99008.0\n",
       "5   27  Female     White  2014-05-05                 Sales Manager  112320.0\n",
       "6   51  Female     White  2014-05-05             Director of Sales  124800.0\n",
       "7   27  Female     White  2013-09-30            Area Sales Manager  118560.0\n",
       "8   43    Male     White  2014-12-05            Area Sales Manager  116480.0\n",
       "9   63    Male     White  2014-12-05            Area Sales Manager  115440.0"
      ],
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Ethnicity</th>\n",
       "      <th>Start_date</th>\n",
       "      <th>Position</th>\n",
       "      <th>Salary</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>51</td>\n",
       "      <td>Male</td>\n",
       "      <td>White</td>\n",
       "      <td>2011-08-15</td>\n",
       "      <td>Software Engineering Manager</td>\n",
       "      <td>56160.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>30</td>\n",
       "      <td>Female</td>\n",
       "      <td>White</td>\n",
       "      <td>2013-11-11</td>\n",
       "      <td>Software Engineer</td>\n",
       "      <td>116480.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>31</td>\n",
       "      <td>Female</td>\n",
       "      <td>White</td>\n",
       "      <td>2012-05-11</td>\n",
       "      <td>Software Engineer</td>\n",
       "      <td>102440.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>34</td>\n",
       "      <td>Male</td>\n",
       "      <td>White</td>\n",
       "      <td>2014-07-07</td>\n",
       "      <td>Software Engineer</td>\n",
       "      <td>99840.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>38</td>\n",
       "      <td>Female</td>\n",
       "      <td>White</td>\n",
       "      <td>2014-10-11</td>\n",
       "      <td>Software Engineer</td>\n",
       "      <td>99008.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>27</td>\n",
       "      <td>Female</td>\n",
       "      <td>White</td>\n",
       "      <td>2014-05-05</td>\n",
       "      <td>Sales Manager</td>\n",
       "      <td>112320.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>51</td>\n",
       "      <td>Female</td>\n",
       "      <td>White</td>\n",
       "      <td>2014-05-05</td>\n",
       "      <td>Director of Sales</td>\n",
       "      <td>124800.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>27</td>\n",
       "      <td>Female</td>\n",
       "      <td>White</td>\n",
       "      <td>2013-09-30</td>\n",
       "      <td>Area Sales Manager</td>\n",
       "      <td>118560.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>43</td>\n",
       "      <td>Male</td>\n",
       "      <td>White</td>\n",
       "      <td>2014-12-05</td>\n",
       "      <td>Area Sales Manager</td>\n",
       "      <td>116480.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>63</td>\n",
       "      <td>Male</td>\n",
       "      <td>White</td>\n",
       "      <td>2014-12-05</td>\n",
       "      <td>Area Sales Manager</td>\n",
       "      <td>115440.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ]
     },
     "metadata": {},
     "execution_count": 3
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Before beginning with our tests we need to assemble a proper set of questions and hypothesis which covers all the posiblities in this particular situation.\r\n",
    "\r\n",
    "__Questions and Hypothesis.__\r\n",
    "\r\n",
    "For each case we consider 2 tail test, because of our hypothesis.\r\n",
    "\r\n",
    "__Q1.__ Is there any gender wage gap in the organisation ?\r\n",
    "\r\n",
    "$$H_{0}  :  \\mu_{male} - \\mu_{female} = 0 \\space , \\space no \\space gender \\space gap$$\r\n",
    "$$H_{1}  :  \\mu_{male} - \\mu_{female} \\neq 0 \\space , \\space gender \\space gap \\space present$$\r\n",
    "\r\n",
    "\r\n",
    "__Q2.__ Is the company discriminating on the basis of race/ethnicity (white vs else) ?\r\n",
    "\r\n",
    "$$H_{0}  :  \\mu_{white} - \\mu_{non-white} = 0 \\space , \\space no \\space racial \\space gap$$\r\n",
    "$$H_{1}  :  \\mu_{white} - \\mu_{non-white} \\neq 0 \\space , \\space racial \\space gap \\space present$$\r\n",
    "\r\n",
    "\r\n",
    "__Q3.__ Is there racial discrimination for the employees above the age of 35 ?\r\n",
    "\r\n",
    "$$H_{0}  :  \\mu_{white-above35} - \\mu_{non-white-above35} = 0 \\space , \\space no \\space racial \\space gap$$\r\n",
    "$$H_{1}  :  \\mu_{white-above35} - \\mu_{non-white-above35} \\neq 0 \\space , \\space racial \\space gap \\space present$$\r\n",
    "\r\n",
    "__Q4.__ Is there gender discrimination for the employees above the age of 35 ?\r\n",
    "\r\n",
    "$$H_{0}  :  \\mu_{male-above35} - \\mu_{female-above35} = 0 \\space , \\space no \\space racial \\space gap$$\r\n",
    "$$H_{1}  :  \\mu_{male-above35} - \\mu_{female-above35} \\neq 0 \\space , \\space racial \\space gap \\space present$$\r\n",
    "\r\n",
    "__Q5.__ Are the top 50 longest working employees facing gender discrimination ?\r\n",
    "\r\n",
    "$$H_{0}  :  \\mu_{male-top50} - \\mu_{female-top50} = 0 \\space , \\space no \\space gender \\space gap$$\r\n",
    "$$H_{1}  :  \\mu_{male-top50} - \\mu_{female-top50} \\neq 0 \\space , \\space gender \\space gap \\space present$$\r\n",
    "\r\n",
    "__Q6.__ Are the top 50 longest working employees facing racial discrimination ?\r\n",
    "\r\n",
    "$$H_{0}  :  \\mu_{white-top50} - \\mu_{non-white-top50} = 0 \\space , \\space no \\space racial \\space gap$$\r\n",
    "$$H_{1}  :  \\mu_{white-top50} - \\mu_{non-white-top50} \\neq 0 \\space , \\space racial \\space gap \\space present$$\r\n",
    "\r\n",
    "__Q7.__ We should check for each department is there gender discrimination ?\r\n",
    "\r\n",
    "$$H_{0}  :  \\mu_{male-department} - \\mu_{female-department} = 0 \\space , \\space no \\space gender \\space gap$$\r\n",
    "$$H_{1}  :  \\mu_{male-department} - \\mu_{female-department} \\neq 0 \\space , \\space gender \\space gap \\space present$$\r\n",
    "\r\n",
    "__Q8.__ We should check for each department is there racial discrimination ?\r\n",
    "\r\n",
    "$$H_{0}  :  \\mu_{white-department} - \\mu_{non-white-department} = 0 \\space , \\space no \\space racial \\space gap$$\r\n",
    "$$H_{1}  :  \\mu_{white-department} - \\mu_{non-white-department} \\neq 0 \\space , \\space racial \\space gap \\space present$$"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "__Create proper dataframes for each hypothesis__"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "source": [
    "# check gender values\r\n",
    "df['Gender'].unique()"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "array(['Male', 'Female'], dtype=object)"
      ]
     },
     "metadata": {},
     "execution_count": 4
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "source": [
    "# create data for male and female -- gender factor\r\n",
    "df_female = df[df['Gender']=='Female']['Salary']\r\n",
    "df_male = df[df['Gender']=='Male']['Salary']"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "source": [
    "# check Ethnicity values\r\n",
    "df['Ethnicity'].unique()"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "array(['White', 'Two or more races', 'Hispanic',\n",
       "       'Black or African American', 'Asian'], dtype=object)"
      ]
     },
     "metadata": {},
     "execution_count": 6
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "source": [
    "# create data for white and non whites -- race factor\r\n",
    "df_white = df[df['Ethnicity']=='White']['Salary']\r\n",
    "df_non_white = df[df['Ethnicity']!='White']['Salary']"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "source": [
    "# create data for employees above 35 years of age. -- racial discrimination\r\n",
    "df_over35_white = df[(df['Age']> 35) & (df['Ethnicity']=='White')]['Salary']\r\n",
    "df_over35_non_white = df[(df['Age']> 35) & (df['Ethnicity']!='White')]['Salary']"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "source": [
    "# create data for white and non whites -- race factor\r\n",
    "df_over35_female = df[(df['Age']> 35) & (df['Gender']=='Female')]['Salary']\r\n",
    "df_over35_male = df[(df['Age']> 35) & (df['Gender']=='Male')]['Salary']"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "source": [
    "# top 50 employees - highest tenure / longest in the organisation - gender and race factor\r\n",
    "df_top50_tenure = df.sort_values(by='Start_date',ascending=True)[:50]"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "source": [
    "# create data for male and female -- gender factor\r\n",
    "df_top50_female = df_top50_tenure[df_top50_tenure['Gender']=='Female']['Salary']\r\n",
    "df_top50_male = df_top50_tenure[df_top50_tenure['Gender']=='Male']['Salary']"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "source": [
    "# create data for male and female -- gender factor\r\n",
    "df_top50_white = df_top50_tenure[df_top50_tenure['Ethnicity']=='White']['Salary']\r\n",
    "df_top50_non_white = df_top50_tenure[df_top50_tenure['Ethnicity']!='White']['Salary']"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "__From the abe given work flow, how to perform hypothesis this, a function has been been created to ease our work__"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "source": [
    "## create function hypothesis test\r\n",
    "def hypothesis_test(df1,df2,alpha,hypo):\r\n",
    "    m1, m2 = df1.mean(), df2.mean()\r\n",
    "    se1, se2 = sem(df1), sem(df2)\r\n",
    "    se = math.sqrt(se1**2.0 + se2**2.0)\r\n",
    "    ts = ((m1 - m2)-hypo) / se\r\n",
    "    deg_free = len(df1) + len(df2) - 2\r\n",
    "    crit_value = t.ppf(1.0 - alpha, deg_free)\r\n",
    "    p = (1.0 - t.cdf(abs(ts), deg_free)) * 2.0\r\n",
    "    result = ''\r\n",
    "    if p < alpha:\r\n",
    "        result = 'Reject the Null Hypothesis'\r\n",
    "    else:\r\n",
    "        result = \"Don't Reject the Null Hypothesis\"\r\n",
    "        \r\n",
    "    if np.isnan(ts):\r\n",
    "        print('Any of the two dataset is either null or has only one item')\r\n",
    "    else:\r\n",
    "        print(\"T Statistic : \",ts)\r\n",
    "        print(\"Degree of Freedom : \",deg_free)\r\n",
    "        print(\"Critical Value : \",crit_value)\r\n",
    "        print(\"P Value : \",p)\r\n",
    "        print(\"\")\r\n",
    "        if result == 'Reject the Null Hypothesis':\r\n",
    "            print('###########################################')\r\n",
    "            print(\"Verdict : \",result)\r\n",
    "            print('###########################################')\r\n",
    "        else:\r\n",
    "            print(\"Verdict : \",result)\r\n",
    "        return ts"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### Q1. Is there any gender wage gap in the organisation ?"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "source": [
    "hypothesis_test(df_male,df_female,0.05,0)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "T Statistic :  1.2508402639016816\n",
      "Degree of Freedom :  172\n",
      "Critical Value :  1.6537609493607581\n",
      "P Value :  0.21269158371105035\n",
      "\n",
      "Verdict :  Don't Reject the Null Hypothesis\n"
     ]
    },
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "1.2508402639016816"
      ]
     },
     "metadata": {},
     "execution_count": 14
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### The result implies our hypothesis was true, that there was no discrimination against gender."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### Q2. Is the company discriminating on the basis of race/ethnicity (white vs else) ?"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "source": [
    "hypothesis_test(df_white,df_non_white,0.05,0)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "T Statistic :  -0.6572076630862012\n",
      "Degree of Freedom :  172\n",
      "Critical Value :  1.6537609493607581\n",
      "P Value :  0.5119258627156795\n",
      "\n",
      "Verdict :  Don't Reject the Null Hypothesis\n"
     ]
    },
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "-0.6572076630862012"
      ]
     },
     "metadata": {},
     "execution_count": 15
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### The result implies our hypothesis was true, that there was is no racial profiling."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### Q3. Is there racial discrimination for the employees above the age of 35 ?"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "source": [
    "hypothesis_test(df_over35_white,df_over35_non_white,0.05,0)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "T Statistic :  0.5469636226703869\n",
      "Degree of Freedom :  90\n",
      "Critical Value :  1.6619610839969403\n",
      "P Value :  0.5857573869489032\n",
      "\n",
      "Verdict :  Don't Reject the Null Hypothesis\n"
     ]
    },
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "0.5469636226703869"
      ]
     },
     "metadata": {},
     "execution_count": 16
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### The result implies our hypothesis was true, that there was is no racial profiling while the employees are abive the age of 35."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### Q4. Is there gender discrimination for the employees above the age of 35 ?"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "source": [
    "hypothesis_test(df_over35_male,df_over35_female,0.05,0)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "T Statistic :  1.4087675289655615\n",
      "Degree of Freedom :  90\n",
      "Critical Value :  1.6619610839969403\n",
      "P Value :  0.16234944888715952\n",
      "\n",
      "Verdict :  Don't Reject the Null Hypothesis\n"
     ]
    },
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "1.4087675289655615"
      ]
     },
     "metadata": {},
     "execution_count": 17
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### The result implies our hypothesis was true, that there is no age discrimination against sex, above the age of 35."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### Q5. Are the top 50 longest working employees facing gender discrimination ?"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "source": [
    "hypothesis_test(df_top50_male,df_top50_female,0.05,0)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "T Statistic :  0.3527540483112269\n",
      "Degree of Freedom :  48\n",
      "Critical Value :  1.6772241953450393\n",
      "P Value :  0.7258171918870246\n",
      "\n",
      "Verdict :  Don't Reject the Null Hypothesis\n"
     ]
    },
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "0.3527540483112269"
      ]
     },
     "metadata": {},
     "execution_count": 18
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### The result implies our hypothesis was true, that there is no gender discrimination on top 50 oldest working employees."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### Q6. Are the top 50 longest working employees facing racial discrimination ?"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "source": [
    "hypothesis_test(df_top50_white,df_top50_non_white,0.05,0)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "T Statistic :  -0.5143007160294015\n",
      "Degree of Freedom :  48\n",
      "Critical Value :  1.6772241953450393\n",
      "P Value :  0.6094016746584257\n",
      "\n",
      "Verdict :  Don't Reject the Null Hypothesis\n"
     ]
    },
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "-0.5143007160294015"
      ]
     },
     "metadata": {},
     "execution_count": 19
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### The result implies our hypothesis was true, that there is no gender discrimination on top 50 oldest working employees."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### Q7. We should check for each position is there gender discrimination ?"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "source": [
    "for i in ['Production Technician I','Area Sales Manager','Production Technician II','Production Manager',\r\n",
    "          'Network Engineer','Database Administrator','Software Engineer','IT Support']:\r\n",
    "    df_fe = df[(df['Gender']=='Female') & (df['Position']==i)]['Salary']\r\n",
    "    df_m = df[(df['Gender']=='Male') & (df['Position']==i)]['Salary']\r\n",
    "    print(\"####################\",i,\"####################\")\r\n",
    "    print(\"\")\r\n",
    "    a = hypothesis_test(df_fe,df_m,0.05,0)\r\n",
    "    print(\"\")\r\n",
    "    print(\"\")\r\n",
    "    print(\"\")"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "#################### Production Technician I ####################\n",
      "\n",
      "T Statistic :  0.9272126831227164\n",
      "Degree of Freedom :  71\n",
      "Critical Value :  1.666599658219398\n",
      "P Value :  0.35695740648515684\n",
      "\n",
      "Verdict :  Don't Reject the Null Hypothesis\n",
      "\n",
      "\n",
      "\n",
      "#################### Area Sales Manager ####################\n",
      "\n",
      "T Statistic :  -1.068419561537325\n",
      "Degree of Freedom :  21\n",
      "Critical Value :  1.7207429028118775\n",
      "P Value :  0.2974597027086754\n",
      "\n",
      "Verdict :  Don't Reject the Null Hypothesis\n",
      "\n",
      "\n",
      "\n",
      "#################### Production Technician II ####################\n",
      "\n",
      "T Statistic :  0.05325231189065007\n",
      "Degree of Freedom :  21\n",
      "Critical Value :  1.7207429028118775\n",
      "P Value :  0.958034218132644\n",
      "\n",
      "Verdict :  Don't Reject the Null Hypothesis\n",
      "\n",
      "\n",
      "\n",
      "#################### Production Manager ####################\n",
      "\n",
      "T Statistic :  -0.5878803211399749\n",
      "Degree of Freedom :  7\n",
      "Critical Value :  1.894578605061305\n",
      "P Value :  0.5750793434683144\n",
      "\n",
      "Verdict :  Don't Reject the Null Hypothesis\n",
      "\n",
      "\n",
      "\n",
      "#################### Network Engineer ####################\n",
      "\n",
      "T Statistic :  -1.6523869783638303\n",
      "Degree of Freedom :  6\n",
      "Critical Value :  1.9431802803927818\n",
      "P Value :  0.14954331295182755\n",
      "\n",
      "Verdict :  Don't Reject the Null Hypothesis\n",
      "\n",
      "\n",
      "\n",
      "#################### Database Administrator ####################\n",
      "\n",
      "T Statistic :  0.22153637581902144\n",
      "Degree of Freedom :  5\n",
      "Critical Value :  2.015048372669157\n",
      "P Value :  0.8334383293098622\n",
      "\n",
      "Verdict :  Don't Reject the Null Hypothesis\n",
      "\n",
      "\n",
      "\n",
      "#################### Software Engineer ####################\n",
      "\n",
      "Any of the two dataset is either null or has only one item\n",
      "\n",
      "\n",
      "\n",
      "#################### IT Support ####################\n",
      "\n",
      "Any of the two dataset is either null or has only one item\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### We don't encounter any bias on the basis of gender in any position."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### Q8. We should check for each position is there racial discrimination ?"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "source": [
    "for i in ['Production Technician I','Area Sales Manager','Production Technician II','Production Manager',\r\n",
    "          'Network Engineer','Database Administrator','Software Engineer','IT Support']:\r\n",
    "    df_w = df[(df['Ethnicity']=='White') & (df['Position']==i)]['Salary']\r\n",
    "    df_nw = df[(df['Ethnicity']!='White') & (df['Position']==i)]['Salary']\r\n",
    "    print(\"####################\",i,\"####################\")\r\n",
    "    print(\"\")\r\n",
    "    a = hypothesis_test(df_w,df_nw,0.05,0)\r\n",
    "    print(\"\")\r\n",
    "    print(\"\")\r\n",
    "    print(\"\")"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "#################### Production Technician I ####################\n",
      "\n",
      "T Statistic :  -0.3774148917203454\n",
      "Degree of Freedom :  71\n",
      "Critical Value :  1.666599658219398\n",
      "P Value :  0.7069911330462606\n",
      "\n",
      "Verdict :  Don't Reject the Null Hypothesis\n",
      "\n",
      "\n",
      "\n",
      "#################### Area Sales Manager ####################\n",
      "\n",
      "T Statistic :  0.4280565206294255\n",
      "Degree of Freedom :  21\n",
      "Critical Value :  1.7207429028118775\n",
      "P Value :  0.6729680218006755\n",
      "\n",
      "Verdict :  Don't Reject the Null Hypothesis\n",
      "\n",
      "\n",
      "\n",
      "#################### Production Technician II ####################\n",
      "\n",
      "T Statistic :  -0.21077316957806802\n",
      "Degree of Freedom :  21\n",
      "Critical Value :  1.7207429028118775\n",
      "P Value :  0.8350960794664801\n",
      "\n",
      "Verdict :  Don't Reject the Null Hypothesis\n",
      "\n",
      "\n",
      "\n",
      "#################### Production Manager ####################\n",
      "\n",
      "T Statistic :  -0.43109245871924246\n",
      "Degree of Freedom :  7\n",
      "Critical Value :  1.894578605061305\n",
      "P Value :  0.6793652591403596\n",
      "\n",
      "Verdict :  Don't Reject the Null Hypothesis\n",
      "\n",
      "\n",
      "\n",
      "#################### Network Engineer ####################\n",
      "\n",
      "Any of the two dataset is either null or has only one item\n",
      "\n",
      "\n",
      "\n",
      "#################### Database Administrator ####################\n",
      "\n",
      "T Statistic :  -0.8483378720863317\n",
      "Degree of Freedom :  5\n",
      "Critical Value :  2.015048372669157\n",
      "P Value :  0.4349711170784749\n",
      "\n",
      "Verdict :  Don't Reject the Null Hypothesis\n",
      "\n",
      "\n",
      "\n",
      "#################### Software Engineer ####################\n",
      "\n",
      "T Statistic :  -2.878928102919357\n",
      "Degree of Freedom :  4\n",
      "Critical Value :  2.13184678133629\n",
      "P Value :  0.04506167868571942\n",
      "\n",
      "###########################################\n",
      "Verdict :  Reject the Null Hypothesis\n",
      "###########################################\n",
      "\n",
      "\n",
      "\n",
      "#################### IT Support ####################\n",
      "\n",
      "T Statistic :  2.4352300778365277\n",
      "Degree of Freedom :  2\n",
      "Critical Value :  2.919985580355516\n",
      "P Value :  0.13524328318045398\n",
      "\n",
      "Verdict :  Don't Reject the Null Hypothesis\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "source": [
    "# check software engineer position\r\n",
    "df[(df['Ethnicity']=='White') & (df['Position']=='Software Engineer')]"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "   Age  Gender Ethnicity  Start_date           Position    Salary\n",
       "1   30  Female     White  2013-11-11  Software Engineer  116480.0\n",
       "2   31  Female     White  2012-05-11  Software Engineer  102440.0\n",
       "3   34    Male     White  2014-07-07  Software Engineer   99840.0\n",
       "4   38  Female     White  2014-10-11  Software Engineer   99008.0"
      ],
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Ethnicity</th>\n",
       "      <th>Start_date</th>\n",
       "      <th>Position</th>\n",
       "      <th>Salary</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>30</td>\n",
       "      <td>Female</td>\n",
       "      <td>White</td>\n",
       "      <td>2013-11-11</td>\n",
       "      <td>Software Engineer</td>\n",
       "      <td>116480.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>31</td>\n",
       "      <td>Female</td>\n",
       "      <td>White</td>\n",
       "      <td>2012-05-11</td>\n",
       "      <td>Software Engineer</td>\n",
       "      <td>102440.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>34</td>\n",
       "      <td>Male</td>\n",
       "      <td>White</td>\n",
       "      <td>2014-07-07</td>\n",
       "      <td>Software Engineer</td>\n",
       "      <td>99840.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>38</td>\n",
       "      <td>Female</td>\n",
       "      <td>White</td>\n",
       "      <td>2014-10-11</td>\n",
       "      <td>Software Engineer</td>\n",
       "      <td>99008.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ]
     },
     "metadata": {},
     "execution_count": 22
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "source": [
    "df[(df['Ethnicity']!='White') & (df['Position']=='Software Engineer')]"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "     Age  Gender                  Ethnicity  Start_date           Position  \\\n",
       "123   38  Female  Black or African American  2012-09-01  Software Engineer   \n",
       "159   30  Female                      Asian  2013-11-11  Software Engineer   \n",
       "\n",
       "       Salary  \n",
       "123  118809.6  \n",
       "159  115460.8  "
      ],
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Ethnicity</th>\n",
       "      <th>Start_date</th>\n",
       "      <th>Position</th>\n",
       "      <th>Salary</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>123</th>\n",
       "      <td>38</td>\n",
       "      <td>Female</td>\n",
       "      <td>Black or African American</td>\n",
       "      <td>2012-09-01</td>\n",
       "      <td>Software Engineer</td>\n",
       "      <td>118809.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>159</th>\n",
       "      <td>30</td>\n",
       "      <td>Female</td>\n",
       "      <td>Asian</td>\n",
       "      <td>2013-11-11</td>\n",
       "      <td>Software Engineer</td>\n",
       "      <td>115460.8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ]
     },
     "metadata": {},
     "execution_count": 23
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### All of the positions does not racially discriminate, except Software Engineer. In this we saw a negative t-statistic, which means the non white folks were provided more than the white people. The p-value was 0.04, which means for 96% of the time it will be true."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Conclusion :\r\n",
    "\r\n",
    "Spark Foundation is a corporation which doesn't discriminate against any standards. It can one of the best places that an individual can work in. On the contrary, in Software Engineer profile, non white people were compensated more than the white people, it could be due to any company policies like relocation or anything, so we cannot directly jump to conclusions. __Our final verdict is Spark Foundation Inc. does not have pay gap within their organisation.__"
   ],
   "metadata": {}
  }
 ],
 "metadata": {
  "orig_nbformat": 4,
  "language_info": {
   "name": "python",
   "version": "3.9.6",
   "mimetype": "text/x-python",
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "pygments_lexer": "ipython3",
   "nbconvert_exporter": "python",
   "file_extension": ".py"
  },
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3.9.6 64-bit"
  },
  "interpreter": {
   "hash": "f9c37c0c0ec68cab32f2d3a83771b1acca5832c232a19ed61fc7a5f670446100"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}