R-2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# R Workbook 2: What is the Difference by Year?\n",
    "\n",
    "In previous workbook, we looked at how to bring in a dataset in a Data Frame and explore it. In this workbook, we'll explore more of how we can use R to automate and speed up analysis. In addition, we'll look at using basic visualizations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style = \"color:green\">Motivating Question</span>\n",
    "\n",
    "So far, we've only looked at data from one year. However, part of what makes the LODES data so useful is the availability of the data over many years. The data are available in multiple CSV files. There are many insights we might find from analyzing data over years, so we want to be able to bring them all together. In this notebook, we'll explore how to do that as we try to answer the following questions:\n",
    "\n",
    "**What is the difference in the number of jobs by age group and by year? What are the different trends across years?**\n",
    "\n",
    "To answer these, we'll introduce you to creating functions to make reading datasets easier. Then, we'll use them within loops to automate reading in datasets. Finally, we'll show some basic visualizations with the data that we get to bring it all together."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Starting Out\n",
    "As before, we'll start by loading libraries. We have a few more commands we'll be running in the beginning since we're also going to be doing some visualizations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "library(tidyverse)\n",
    "\n",
    "# Visualization settings to make them look nicer in Jupyter\n",
    "theme_set(theme_gray(base_size = 20))\n",
    "options(repr.plot.width = 12, repr.plot.height = 9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:green\">Creating Functions</span>\n",
    "\n",
    "First, we start off by creating a function. You're familiar with functions in general already, since you've used them to do various things like determining the type of an object (by using `class`) or finding the mean of a list of numbers (using `mean`). Here, we'll walk through creating your own function. This helps simplify your code to make it much more readable if you're going to be doing the same thing many times, as you won't have to copy and paste your code each time you want to do a certain task.\n",
    "\n",
    "Let's start with a very basic function. Suppose we want a function that takes an argument and returns the squared value. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "squarer <- function(x){\n",
    "    y = x^2\n",
    "    return(y)\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "squarer(4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we're creating a function called `squarer` which takes an input, `x`, and outputs the squared value. Let's break it down line by line.\n",
    "\n",
    "    squarer <- function(x){\n",
    "        return()\n",
    "    }\n",
    "\n",
    "The `function` indicates that we're defining a function. This is being assigned to `squarer`, so that is what the name of our function will be. Then, in parentheses, we put in any arguments we want the function to take. If we don't want it to take any arguments, we can just leave it blank. Lastly, we end the line by opening up a set of curly braces.\n",
    "\n",
    "This takes us to the next lines. \n",
    "\n",
    "    y = x^2\n",
    "    return(y) \n",
    "    \n",
    "Here, we have two lines, which assigns the value `x^2` to `y`, then uses the `return` function to output it. \n",
    "\n",
    "Lastly, we have the next cell, which isn't part of the function and is just used to demonstrate calling the function we've just created.\n",
    "    \n",
    "    squarer(4)\n",
    "\n",
    "This just uses the function with the argument `4` to see if it works and gives us a value of 16. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Writing a Function to get CA Data\n",
    "\n",
    "Let's start by writing a function that can take in a year and gets the California Workplace Area Characteristics file as a data frame. We'll first show the complete function, then explain what goes into each part."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_ca_wac <- function(year){\n",
    "    base_url <- 'https://lehd.ces.census.gov/data/lodes/LODES7/'\n",
    "    file_specs <- str_replace_all('ca/wac/ca_wac_S000_JT00_YEAR.csv.gz', 'YEAR', as.character(year))\n",
    "    file_name <- paste(base_url, file_specs, sep = '')\n",
    "    \n",
    "    return(read_csv(file_name, col_types = cols()))\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ca_wac_2015 <- get_ca_wac(2015)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's break down the pieces of the function. First, we define the function using `function`, with the arguments inside parentheses. Then, inside the curly braces, we put the actions that we want the function to take. The first line inside the function is \n",
    "\n",
    "    base_url <- 'https://lehd.ces.census.gov/data/lodes/LODES7/'\n",
    "\n",
    "This creates a string called `base_url` which contains the first part of the URL that we will get our data from. This will be the same no matter which year we want to get our data for. \n",
    "\n",
    "Next, we use \n",
    "\n",
    "    file_specs <- str_replace_all('ca/wac/ca_wac_S000_JT00_YEAR.csv.gz', 'YEAR', as.character(year))\n",
    "\n",
    "This does a few things. First, the `str_replace_all` function replaces all instances of `YEAR` with the year we passed as the argument in the function. Note that we use the `as.character` function to make sure that the year we include is a character, because `str_replace_all` needs to replace with a character.\n",
    "\n",
    "This provides us with two parts to a URL: the `base_url`, as well as the subsequent `file_specs`. We just need to put these together to get the full url, which we do in the next line\n",
    "\n",
    "    file_name <- paste(base_url, file_specs, sep = '')\n",
    "    \n",
    "The `paste` function is used to combine two character objects, with the `sep = ''` argument indicating that we don't want anything between the two. \n",
    "\n",
    "Finally, the last line provides the output.\n",
    "\n",
    "    return(read_csv(file_name, col_types = cols()))\n",
    "    \n",
    "Note that we use the `read_csv` function and return the output of that function directly, and that we actually included an additional argument here:\n",
    "\n",
    "    col_types = cols()\n",
    "    \n",
    "This just specifies that we want to use the default column types instead of specifying the column types explicitly. This is the default behavior already, but doing this avoids displaying the message that pops up about column specifications."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Function for WAC Data from Any State\n",
    "\n",
    "What if we wanted to get WAC data from any state instead of just California? We can adjust the function to allow for this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_wac <- function(state, year){\n",
    "    base_url <- 'https://lehd.ces.census.gov/data/lodes/LODES7/'\n",
    "    file_specs <- str_replace_all('STATE/wac/STATE_wac_S000_JT00_YEAR.csv.gz', 'STATE', state) %>% str_replace_all('YEAR', as.character(year))\n",
    "    file_name <- paste(base_url, file_specs, sep = '')\n",
    "    \n",
    "    return(read_csv(file_name, col_types = cols()))\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df <- get_wac('ca', '2015')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the only thing that changed was the second line inside the function.\n",
    "\n",
    "    file_specs <- str_replace_all('STATE/wac/STATE_wac_S000_JT00_YEAR.csv.gz', 'STATE', state) %>% str_replace_all('YEAR', as.character(year))\n",
    "\n",
    "This new function doesn't do that much more than we did before. Just like we used the `str_replace_all` function before to find all instances of the `STATE`, we can again use it to find all instances of `YEAR` and replace them with the appropriate year. Remember, the `%>%` is the pipe operator, which allows us to use functions in order and takes the expression on the left as the first argument of the expression on the right."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font color = 'red'>Checkpoint 1: Using Functions</font>\n",
    "\n",
    "Try using the functions above to bring in the data for your state. Do you need to make any changes to the functions? Why or why not?\n",
    "\n",
    "Suppose you wanted to adjust the functions so that you can specify whether you want to bring in the Residence Area Characteristics (rac), or the Workplace Area Characteristics (wac), instead of bringing in the wac dataset by default. How would you adjust the code? Try doing it yourself.\n",
    "\n",
    "What about if you wanted to change the function to also work for the Origin-Destination (od) data? How would the function change then? Make sure you look carefully at the file name. \n",
    "\n",
    "**Hint:** You might consider using if-else statements. You can create `if-else` statements using something like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test <- 1\n",
    "if(test < 1){\n",
    "    print(\"The number is less than 1\")\n",
    "}else{\n",
    "    print(\"The number is not less than 1\")    \n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style = \"color:green\">Using Loops</span>\n",
    "\n",
    "Sometimes, we want to run the same code many times over. In these cases, we can use loops so that we don't have to copy and paste the code over and over. To demonstrate how loops work, we'll first look at a basic `for` loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for(i in 1:10){\n",
    "    print(i)\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we are looping through the numbers 1 to 10 and printing them out. Let's break down how each part works.\n",
    "\n",
    "First, consider the first line.\n",
    "\n",
    "    for(i in 1:10){\n",
    "\n",
    "This indicates that we will be looping through the values of 1 to 10, incrementing `i` in each iteration. That is, the code will use `i=1` for one iteration. Then, it will go back and do everything again, except using `i=2`. This keeps going until it hits `i=10`, after which it stops. \n",
    "\n",
    "Notice that the second line is indented. This is a convention that we use to delineate when the loop starts and ends, but is not necessary. All that matters is that the contents of the loop are inside the curly braces. \n",
    "\n",
    "The next line, and everything that comes between the curly braces for loops that are longer, contains everything that we want to do multiple times. Here, we see the following as the only code inside those curly braces.\n",
    "\n",
    "    print(i)\n",
    "\n",
    "This means that it will print the value of `i` every iteration. So, it takes `i=1`, then runs `print(i)`. Then, it sets `i=2` and runs `print(i)`. This is how we get the output that we observed.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using a For Loop to Read In CSV Files\n",
    "\n",
    "Now that we've gone over the basics of how a for loop works, let's apply it to reading in multiple CSV files. We've already created a function that takes a year and reads a CSV file. We want to do this for multiple years automatically, so that we don't need to keep on changing the year and running the code again (if, for example, we want to do this for many years). So, in other words, we want to create a loop that runs the same code multiple times, with only the year changed.\n",
    "\n",
    "Part of our task is a bit easier, since we've already created a function that does what we want. Now, all we need to do is loop through the years we want, calling that function with a different argument for the year.\n",
    "\n",
    "### Using R lists\n",
    "\n",
    "There's one small complication though: how will we automate storage of these Data Frame objects? There's multiple possibilities, but the way we'll do it is using a **list** in R. A **list** is just another way of storing data, but it's more flexible than the data frames that we've been working with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "example_list <- list(1:10, 'example')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "str(example_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can access the elements of the list by using double brackets. So, to get the first element of the list, which would be the vector of numbers from 1 to 10, we can use the following."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "example_list[[1]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using Loops and Functions to Bring in Multiple Datasets\n",
    "We'll start by creating an empty dictionary in which we'll store the Data Frames that we read in. Then, we're going to loop through a few years (here, we'll do 2011 to 2016), calling the `get_wac` function we created earlier to store the appropriate dataset in the dictionary. We'll also make sure to provide the proper key when storing the dataset, so that we can easily access it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wac_all_years <- list()\n",
    "\n",
    "for (i in 1:6){\n",
    "    wac_all_years[[i]] <- get_ca_wac(i+2010)    \n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font color = 'red'>Checkpoint 2: Working with Loops</font>\n",
    "\n",
    "\n",
    "Using what we've learned above, try to apply the same methods to bring in multiple years' worth of data for a different state. Remember to name objects differently so that you don't overwrite anything."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font color = 'green'>Visualizations using `qplot`</font>\n",
    "    \n",
    "Visualizations can help us look at summaries of the data. They can also aid in providing good presentation aids.\n",
    "\n",
    "In this section, we'll look at a few basic types of visualizations: histograms, boxplots, lineplots. In practice, there are many more types of visualizations. Furthermore, there are many different packages that can help you create these different visualizations. While we'll focus only on the basics here with the base R functions as well as `qplot` from the `ggplot2` package, keep in mind that there are many tools available."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Making a Histogram\n",
    "We'll start with a basic histogram. A histogram is useful for looking at the distribution of a numerical variable (that is, a variable that takes numbers as its values as opposed to categories). They can visually represent the spread and center of the variable, as well as provide insights to the shape of the distribution. For example, we can look at the number of **modes**, or peaks, in the data, which wouldn't be captured in numerical summaries such as mean or variance. We can also get a better idea of the **skew** of the data, determining whether most units are grouped on the left or right side (lower or higher values), or whether it's mostly symmetrical. \n",
    "\n",
    "We'll first look at the simplest way to make a graph, then show how to customize it using the plotting parameters as well as Figure and Axes objects. Let's look at a histogram of the total jobs in the 2015 Workplace Area Characteristics for California."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "qplot(df$C000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The existence of outliers means that the graph gets stretched out and it's very hard to tell anything from the resulting histogram. We can restrict the x-axis to get a closer look at the data using `xlim`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "qplot(df$C000, xlim = c(0,500))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Making a Scatterplot\n",
    "\n",
    "When looking at two numerical variables to see if there is a correlation, we might want to use a scatterplot to visualize them. We can simply use the base R `plot` function, passing it the x and y variables as the arguments. \n",
    "\n",
    "Let's look at an example looking at the relationship between number of jobs for two different age groups."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_2011 <- wac_all_years[[1]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot(df_2011$CA01, df_2011$CA02)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style = \"color:green\">Using a Line Chart to Show Jobs by Year for Each Age Group (VIDEO)</span>\n",
    "\n",
    "In this last section, we'll look at a line chart while incorporating all the other methods we've gone over in this notebook. Here, we want to look at the change in number of jobs by year, separating them into each age group so that we can look at the trends for each age group as well as compare the trends across age groups. \n",
    "\n",
    "In order to do this, we'll need to get that data from multiple Data Frames, since we want to combine data from multiple years. We'll put them all into a data frame that aggregates the information, then plot the line chart.  \n",
    "\n",
    "Recall that we've already brought in data from 2011 to 2016 already. We'll use that data for now, but you can go back and change the years to include more. We'll show all of the code, then go over the individual parts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "CA_over_years <- data.frame(Year = rep(NA,6), \n",
    "                           CA01 = rep(NA,6),\n",
    "                           CA02 = rep(NA,6),\n",
    "                           CA03 = rep(NA,6))\n",
    "\n",
    "for (i in 1:6){\n",
    "    CA_sum <- wac_all_years[[i]] %>% select(CA01, CA02, CA03) %>% summarize(Year = i+2010, CA01 = sum(CA01), CA02 = sum(CA02),CA03 = sum(CA03))\n",
    "    CA_over_years[i,] <- CA_sum   \n",
    "}\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we start by initializing an empty Data Frame which we will use to store the data. \n",
    "\n",
    "    CA_over_years <- data.frame(Year = rep(NA,6), \n",
    "                           CA01 = rep(NA,6),\n",
    "                           CA02 = rep(NA,6),\n",
    "                           CA03 = rep(NA,6))\n",
    "\n",
    "The `data.frame` function can be used to create a data frame from scratch, with columns being specified in the arguments. The `rep` function creates a vector consisting of the first argument repeated a number of times equal to the second argument. In this case, `rep(NA,6)` creates a vector of length 6 which contains `NA` for each element of the vector. \n",
    "\n",
    "    for (i in 1:6){\n",
    "        CA_sum <- wac_all_years[[i]] %>% select(CA01, CA02, CA03) %>% summarize(Year = i+2010, CA01 = sum(CA01), CA02 = sum(CA02),CA03 = sum(CA03))\n",
    "        CA_over_years[i,] <- CA_sum   \n",
    "    }\n",
    "    \n",
    "Next, we use a `for` loop, so that we can go through each year and add that year's data to the Data Frame we just created. Let's look at the first line of the `for` loop:\n",
    "\n",
    "    CA_sum <- wac_all_years[[i]] %>% select(CA01, CA02, CA03) %>% summarize(Year = i+2010, CA01 = sum(CA01), CA02 = sum(CA02),CA03 = sum(CA03))\n",
    "   \n",
    "This takes the list of all the years of data and uses the $i^{th}$ Data Frame in that list. Then, it `selects` the age variables, uses `summarize` to get the `Year` as well as the total number of jobs within that age group by summing over all census blocks.\n",
    "\n",
    "This is then stored inside the $i^{th}$ row of the `CA_over_years` Data Frame.\n",
    "\n",
    "Let's take a look at the Data Frame that we've created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "head(CA_over_years)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We actually need to format the data a little bit differently in order to create the graph that we want. For `qplot`, we want three columns: `Year`, `Age Group`, and `Jobs`, with each row representing a Year-Age Group combination. This makes it so that each row only has one value being plotted, and we can just tell `qplot` how to color the lines according to the other variables. \n",
    "\n",
    "To get the data in this format, we will use the `gather` function. We use `-Year` (note the \"-\") to make sure we keep the `Year` variable unchanged, then specify that we want a column for age group and for the number of jobs. The rest of the columns besides `Year` all need to be converted in to this format, which is why we are able to use `-Year` (that is, telling `gather` to use all columns except `Year`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "CA_years_long <- CA_over_years %>% gather(-Year, key = 'Age_Group', value = 'Jobs')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "qplot(Year, Jobs, data = CA_years_long, color = Age_Group, geom = 'line')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:red\">Checkpoint 3: Visualize Your Data</span>\n",
    "\n",
    "Using the methods we've described above, try visualizing data from a different state. What do the visualizations tell you? How are they different from the data from California? How are they the same? Does this make sense?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}