p_hacking.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from p_hacking import *\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# P-hacking"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Key Terms / Review\n",
    "Let's review some key terms first!\n",
    "\n",
    "If there is a __correlation__ between variable X and variable Y, that means as variable X increases, variable Y tends to increase as well (__positive correlation__) or tends to decrease (__negative correlation__).\n",
    "\n",
    "<img src=\"https://www.emathzone.com/wp-content/uploads/2014/10/positive-negative-corrrelation.jpg\">\n",
    "\n",
    "Sometimes, a pair of otherwise totally unrelated variables could happen to have a correlation, just by random chance alone. This is what we call a __spurious correlation__, and is an inevitable aspect of our complex world. The way statisticians tell whether a correlation is just due to random chance or is likely to be a real correlation—revealing some underlying connection between the two variables—is through __p-values__.\n",
    "\n",
    "The p-value of a correlation is the probability that the correlation could happen just by random chance. For example, if variables A and B are correlated with $p=0.1$, then there is a 10% chance that this correlation actually arose from random chance—that this is a spurious correlation. If $p=0.01$, then there is only a 1% chance this correlation arose from random chance. The smaller $p$ is, the stronger the correlation.\n",
    "\n",
    "When scientists say \"the correlation is __statistically significant__\", they really mean their p-value is less than, say, 0.05. Different scientists or journals may choose a different p-value cutoff of what counts as statistically significant. If a journal publishes twenty articles, each of which claims a result with $p=0.05$, then on average we should expect about 1 out of those 20 results to be the result of random chance, rather than a true correlation, of no fault of the scientists."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting Started"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tam is running a study where she measures a number of variables of a set of marbles, looking for correlations among the variables. In her first experiment, she collects data on 4 different variables. Run the following box to see the data.\n",
    "\n",
    "Every scatter plot shows the data for two variables. Every bar chart shows the histogram for a single variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "guesser1 = perform_test(num_variables=4, p_value=0.05, true_corr=0.45, sample_size_log=1);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What Tam doesn't know, but the marble factory does, is that there is a pair of variables that are truly correlated with each other in this batch of marbles.\n",
    "\n",
    "**Question 1:** Looking at the graphs, can you tell which correlation is the true correlation? Why or why not?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question 2:** How many pairs of variables have a statistically significant correlation ($p<0.05$)? Can Tam publish a result in a journal that requires $p<0.05$?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Studying more marbles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To improve her study, Tam decides to purchase a much larger batch of marbles. Run the box below to see the data that she collected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "guesser2 = perform_test(num_variables=4, p_value=0.05, true_corr=0.45, sample_size_log=2);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question 3:** Looking at the graphs, can you tell which correlation is the true correlation?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question 4:** Even though the marble factory only ensured one pair of correlated variables, it turns out that there are 2 pairs of variables that have statistically significant correlation ($p<0.05$). Run the box below and choose different pairs of variables. Can you find both pairs of correlated variables?\n",
    "\n",
    "Since only one pair of variables is truly correlated, the other pair must be a spurious correlation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "guesser2();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Studying more pairs of variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Encouraged by the correlations she found, Tam decided to collect more variables from a new batch of marbles, in the hopes that she may uncover other hidden correlations.\n",
    "\n",
    "**Question 5:** As Tam observes the correlations among a larger set of variables, will the number of statisically significant results increase, decrease or stay the same, and why?\n",
    "\n",
    "Run the box below. It may take a minute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "guesser3 = perform_test(num_variables=8, p_value=0.05, true_corr=0.45, sample_size_log=2);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question 6:** Again, the marble factory made sure there is only one pair of variables that are truly correlated (possibly different from before). Can you tell which pair it is just by looking at the graphs?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question 7:** Run the following box and see if your answer to Question 6 is correct!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question 8:** Even though there is only one pair of truly correlated variables, Tam found 4 pairs of variables that are significantly correlated ($p<0.05$). Besides the one true correlation, should she publish the results of the other 3 correlations with $p<0.05$? Why or why not? (Think about the idea of p-hacking.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "guesser3();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Being more careful with statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tam saw more correlations when she increased the number of variables to investigate. This is because each pair of variables could appear correlated by random chance alone (spurious correlation). As she studies more and more pairs, it becomes ever more likely that some of those pairs will show suprious correlation.\n",
    "\n",
    "One way to isolate the true correlation from the spurious correlations is to lower the p-value cutoff from 0.05 to 0.01. Run the box below to see how lowering the p-value cutoff reduces the number of \"statistically significant\" correlations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "guesser4 = perform_test(num_variables=8, p_value=0.01, true_corr=0.45, sample_size_log=2);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are the main takeaways from this module:\n",
    "1. A large number of samples is needed to see statistically significant correlations.\n",
    "2. For the same number of samples, asking more questions (\"Is X correlated with Y?\") makes it more likely that a spurious correlation would appear.\n",
    "3. The more questions you ask, the lower the p-value cutoff must be in order to isolate the true correlation from the spurious ones.\n",
    "4. When publishing a \"statistically significant\" result, you must also disclose how many other questions you asked of the same data set. The more questions you ask of the same data set, the more likely it is that the \"statistically significant\" result is actually a spurious one!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}