1.4.4 - Evaluate an Experiment Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read the following descriptions of an experiment and its analysis, identify the flaws in each, and describe what you would do to correct them.\n",
    "\n",
    "### 1.  The Sith Lords are concerned that their recruiting slogan, \"Give In to Your Anger,\" isn't very effective. Darth Vader develops an alternative slogan, \"Together We Can Rule the Galaxy.\" They compare the slogans on two groups of 50 captured droids each. In one group, Emperor Palpatine delivers the \"Anger\" slogan. In the other, Darth Vader presents the \"Together\" slogan. 20 droids convert to the Dark Side after hearing Palpatine's slogan, while only 5 droids convert after hearing Vader's. The Sith's data scientist concludes that \"Anger\" is a more effective slogan and should continue to be used.\n",
    "\n",
    "#### Flaws:  \n",
    "* The venue for Palpatine's group could have been much more appealing.  \n",
    "* He could have also threatened them if they didn't convert.  \n",
    "* Maybe he gave them a huge financial reward if they chose to convert.  \n",
    "* The Emperor might have locked them in a room for 3 days straight without food while Vader maybe gave them a 5 minute talk to try to convert them.\n",
    "\n",
    "#### Corrections:  \n",
    "* Use the exact same environment with the same amount of time and same delivery method, same reward system (if applicable).  \n",
    "* Alternate the time of day in which the 2 groups are recruited and alternate who (Palpatine or Vader) gives the recruting message to each of the 2 groups.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.  In the past, the Jedi have had difficulty with public relations. They send two envoys, Jar Jar Binks and Mace Windu, to four friendly and four unfriendly planets respectively, with the goal of promoting favorable feelings toward the Jedi. Upon their return, the envoys learn that Jar Jar was much more effective than Windu: Over 75% of the people surveyed said their attitudes had become more favorable after speaking with Jar Jar, while only 65% said their attitudes had become more favorable after speaking with Windu. This makes Windu angry, because he is sure that he had a better success rate than Jar Jar on every planet. The Jedi choose Jar Jar to be their representative in the future.\n",
    "\n",
    "#### Flaws:  \n",
    "* The friendly planets are probably more susceptible to being influenced by a Jedi envoy.  \n",
    "* The timing could be significant; in other words, if the Jedi establishment just performed a friendly act towards one of the friendly planets, they would probably view the Jedi more favorably.\n",
    "* Maybe Jar Jar was given 3 times as much time and expenses to accomplish his task.\n",
    "* If there were only 10 people on each of the friendly planets compared to 10,000 people on each of the unfriendly planets, the conversion rate for the unfriendly planet is probably more reliable and more significant.  \n",
    "\n",
    "#### Corrections:  \n",
    "* Jar Jar and Windu should be sent at the same time of year/month/day with the same amount of resources.  \n",
    "* They should each be given 2 friendly and 2 unfriendly planets chosen randomly.    \n",
    "* The total population to convert should be similar for each envoy.\n",
    "* They should repeat the experiment one year later switching the groups for each envoy - keeping everything else constant.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.  A company with work sites in five different countries has sent you data on employee satisfaction rates for workers in Human Resources and workers in Information Technology. Most HR workers are concentrated in three of the countries, while IT workers are equally distributed across worksites. The company requests a report on satisfaction for each job type. You calculate average job satisfaction for HR and for IT and present the report.\n",
    "\n",
    "#### Flaws:  \n",
    "* People in the 3 HR countires may be happier in general compared to the population in the 2 other countries.\n",
    "* Working conditions could be vastly different from location to location.\n",
    "* Age group could be vastly different between HR and IT, so the 2 groups may respond differently due to age difference.\n",
    "* Comparing the satisfaction between the 2 groups may not make sense; it might be better to compare the satisfaction within HR and IT based on the different location.\n",
    "* HR could be mostly female.  IT could be mostly or all male.\n",
    "\n",
    "#### Corrections:  \n",
    "* Perform the analyses on just the 3 countries that have both HR and IT.\n",
    "* Make sure to give results that separate departments, country and gender and possibly age.\n",
    "* Maybe further separate the results by a salary range per employee.\n",
    "* Coul also look at (separate/compare) average number of hours worked and/or education level between the 2 groups.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.  When people install the Happy Days Fitness Tracker app, they are asked to \"opt in\" to a data collection scheme where their level of physical activity data is automatically sent to the company for product research purposes. During your interview with the company, they tell you that the app is very effective because after installing the app, the data show that people's activity levels rise steadily.\n",
    "\n",
    "#### Flaws:\n",
    "* Activity rises probably because when they downloaded the app, there were already plannning on raising their activity level.\n",
    "* Those that took the initiative to 'opt in' were probably the ones who would have a rising fitness level compared to the people who did not opt in. \n",
    "\n",
    "#### Corrections:\n",
    "* The study needs to be performed on all users; not just the ones that chose to opt in.\n",
    "* The study needs to be compared to other fitness app activity level rates to obtain context.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.  To prevent cheating, a teacher writes three versions of a test. She stacks the three versions together, first all copies of Version A, then all copies of Version B, then all copies of Version C. As students arrive for the exam, each student takes a test. When grading the test, the teacher finds that students who took Version B scored higher than students who took either Version A or Version C. She concludes from this that Version B is easier, and discards it.\n",
    "\n",
    "#### Flaws:\n",
    "* Maybe the students who received Version B were more prepared and/or better students overall.  Maybe the late comers were worse students.\n",
    "* This doesn't say if the same amount of the different versions were administered.\n",
    "* This may not be fair for the students who took Version B.  They would either have to retake the test or simply not have a grade for it.\n",
    "* Maybe the best students in the class had an honors class that was far away from this class and they would undoubtedly arrive later than most other students.\n",
    "* If there are only 10 or fewer people in the class, then this sample size would be too small and the differences or more likely due to chance.\n",
    "\n",
    "#### Corrections:\n",
    "* The tests could be given randomly after the students sat down at their desks. \n",
    "* The same number of each version should be used.\n",
    "* Class size probably needs to be at least 30 and prefeably more for an accurate measure.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}