chapter7.Rmd

---
title: Basic Data Wrangling
description: We provide a brief introduction to the dplyr package.
---

## dplyr

```yaml
type: NormalExercise
key: 766d9c4eab
lang: r
xp: 100
skills:
  - 1
```

Load the `dplyr` package and the murders dataset.

```{r}
library(dplyr)
library(dslabs)
data(murders)
```

You can add columns using the `dplyr` function `mutate`. This function is aware of the column names and inside the function you can call them unquoted. Like this:

```{r}
murders <- mutate(murders, population_in_millions = population / 10^6)
```

Note that we can write `population` rather than `murders$population`. The function `mutate` knows we are grabbing columns from `murders`.

`@instructions`
- Use the function `mutate` to add a murders column named `rate` with the per 100,000 murder rate. 
- Make sure you redefine `murders` as done in the example code above.

Remember the murder rate is defined as the total murders divided by the population size times 100,000.

`@hint`
You can define a new function based on existing ones like this:
```{r}
mutate(murders, raw_rate =  total / population) 
```
Note that we use the column names `total` and `population` without having the use the accessor.

`@pre_exercise_code`
```{r}
library(dplyr)
library(dslabs)
data(murders)
```

`@sample_code`
```{r}
# Loading data
library(dslabs)
data(murders)

# Loading dplyr
library(dplyr)

# Redefine murders so that it includes a column named rate with the per 100,000 murder rates

```

`@solution`
```{r}
# Loading data
library(dslabs)
data(murders)

# Loading dplyr
library(dplyr)

# Redefine murders so that it includes a column named rate with the per 100,000 murder rates
murders <- mutate(murders, rate =  total / population * 100000)
```

`@sct`
```{r}
test_error()
test_object("murders", undefined_msg = "Define murder first.", incorrect_msg = "Check the code again.")
success_msg("Awesome! Let's learn another command with mutate.")
```

---

## mutate

```yaml
type: NormalExercise
key: 79bec8ece0
lang: r
xp: 100
skills:
  - 1
```

Note that if `rank(x)` gives you the ranks of `x` from lowest to highest, `rank(-x)` gives you the ranks from highest to lowest.

`@instructions`
- Use the function `mutate` to add a column `rank` containing the rank, from highest to lowest murder rate. Make sure you redefine murders.

`@hint`
Note that `rate` is already defined and that `rank(-rate)` gives you the ranks you want. Now just add that column and name the column `rank`

`@pre_exercise_code`
```{r}
library(dplyr)
library(dslabs)
data(murders)
```

`@sample_code`
```{r}
# Note that if you want ranks from highest to lowest you can take the negative and then compute the ranks 
x <- c(88, 100, 83, 92, 94)
rank(-x)

# Defining rate
rate <-  murders$total/ murders$population * 100000

# Redefine murders to include a column named rank
# with the ranks of rate from highest to lowest

```

`@solution`
```{r}
# Note that if you want ranks from highest to lowest you can take the negative and ten compute the ranks 
x <- c(88, 100, 83, 92, 94)
rank(-x)

# Defining rate
rate <-  murders$total/ murders$population * 100000

# Redefine murders to include a column named rank
# with the ranks of rate from highest to lowest
murders <- mutate(murders, rank = rank(-rate))
```

`@sct`
```{r}
test_error()
test_function("mutate")
test_object("murders", incorrect_msg = "Remember, rank from highest to lowest.")
success_msg("Good job!")
```

---

## select

```yaml
type: NormalExercise
key: 0e7a94d0a3
lang: r
xp: 100
skills:
  - 1
```

With `dplyr` we can use `select` to show only certain columns. For example with this code we would only show the states and population sizes:

```{r}
select(murders, state, population)
```

`@instructions`
- Use `select` to show the state names and abbreviations in `murders`. Just show it, do not define a new object.

`@hint`
You can use select to just show one or more columns like this

```{r}
select(data_frame, column_name_1, column_name_2)
```
The columns you want are `state` and `abb`. The data frame you want is `murders`.

`@pre_exercise_code`
```{r}
library(dplyr)
library(dslabs)
data(murders)
```

`@sample_code`
```{r}
# Load dplyr
library(dplyr)

# Use select to only show state names and abbreviations from murders

```

`@solution`
```{r}
# Load dplyr
library(dplyr)

# Use select to only show state names and abbreviations from murders_south
select(murders, state, abb)
```

`@sct`
```{r}
test_error()
test_function("select")
test_output_contains("select(murders, state, abb)", incorrect_msg = "You need to use select. The columns you want are state and abb. The data you want is `murders`")
success_msg("Now you know how to use select! Let's move on to filter.")
```

---

## filter

```yaml
type: NormalExercise
key: 9a56fa2057
lang: r
xp: 100
skills:
  - 1
```

The `dplyr` function `filter` is used to choose specific rows of the data frame to keep. Unlike `select` which is for columns, `filter` is for rows. For example you can show just the New York row like this:

```{r}
filter(murders, state == "New York")
```
You can use other logical vectors to filter rows.

`@instructions`
- Use `filter` to show the top 5 states with the highest murder rates. After we add murder rate and rank, do not change the murders dataset, just show the result. Note that you can filter based on the `rank` column.

`@hint`
The logical vector you want to filter by is `rank <= 5`.

`@pre_exercise_code`
```{r}
library(dplyr)
library(dslabs)
data(murders)

```

`@sample_code`
```{r}
# Add the necessary columns
murders <- mutate(murders, rate = total/population * 100000, rank = rank(-rate))

# Filter to show the top 5 states with the highest murder rates

```

`@solution`
```{r}
# Add the necessary columns
murders <- mutate(murders, rate = total/population * 100000, rank = rank(-rate))

# Filter to show the top 5 states with the highest murder rates
filter(murders, rank <= 5)
```

`@sct`
```{r}
test_error()
test_function("filter")
test_output_contains("filter(murders, rank <= 5)", incorrect_msg = "Make sure you've written murders in your code.")
success_msg("Great job!")
```

---

## filter with !=

```yaml
type: NormalExercise
key: 846dd29c6c
lang: r
xp: 100
skills:
  - 1
```

We can remove rows using the `!=` operator. For example to remove Florida we would do this:

```{r}
no_florida <- filter(murders, state != "Florida")
```

`@instructions`
- Create a new data frame called `no_south` that removes states from the South region.
- How many states are in this category? You can use the function `nrow` for this.

`@hint`
Use `filter` with the `!=` operator to remove states from the South Region. The logica vector you want is `region!="South". Once you define a new data frame without these states 
you can use `nrow` which gets the number of rows of a data frame.

`@pre_exercise_code`
```{r}
library(dplyr)
library(dslabs)
data(murders)
```

`@sample_code`
```{r}
# Use filter to create a new data frame no_south

# Use nrow() to calculate the number of rows
```

`@solution`
```{r}
# Use filter to create a new data frame no_south
no_south <- filter(murders, region != "South" )

# Number of states (rows) in this category 
nrow(no_south)

```

`@sct`
```{r}
test_error()
test_object("no_south", undefined_msg = "Define no_south first!", incorrect_msg = "The object no_south seems to be incorrectly defined. Make sure you`re excluding the ones not in the South.")
test_function("nrow")
test_output_contains("nrow(no_south)", incorrect_msg = "You need the number of rows for the number of states. You may have constructe the wrong no_south object.")
success_msg("That's great! Let's move to the next exercise.")
```

---

## filter with %in%

```yaml
type: NormalExercise
key: 4e5d7ac60f
lang: r
xp: 100
skills:
  - 1
```

We can also use the `%in%` to filter with `dplyr`. For example you can see the data from New York and Texas like this:

```{r}
filter(murders, state %in% c("New York", "Texas"))
```

`@instructions`
- Create a new data frame called `murders_nw` with only the states from the Northeast and the West. 
- How many states are in this category?

`@hint`
Use filter and the `%in%` operator. The logical vector you want in `filter` is `region %in% c("Northeast", "West")`
You can use `nrow` to quickly get the number of rows of a data frame.

`@pre_exercise_code`
```{r}
library(dplyr)
library(dslabs)
data(murders)
```

`@sample_code`
```{r}
# Create a new data frame called murders_nw with only the states from the northeast and the west

# Number of states (rows) in this category 

```

`@solution`
```{r}
# Create a new data frame called murders_nw with only the states from the northeast and the west
murders_nw <- filter(murders, region %in% c("Northeast", "West"))

# Number of states (rows) in this category
nrow(murders_nw)

```

`@sct`
```{r}
test_error()
test_function("filter")
test_function("nrow")
test_object("murders_nw", undefined_msg = "Define murders_nw first.", incorrect_msg = "Make sure you're using the %in% command.")
test_output_contains("nrow(murders_nw)", incorrect_msg = "You need the number of rows again, same as previous question.")
success_msg("This is great!")
```

---

## filtering by two conditions

```yaml
type: NormalExercise
key: 6e8611adb6
lang: r
xp: 100
skills:
  - 1
```

Suppose you want to live in the Northeast or West **and** want the murder rate to be less than 1. We want to see the data for the states satisfying these options. Note that you can use logical operators with `filter`:

```{r}
filter(murders, population < 5000000 & region == "Northeast")
```

`@instructions`
- Add a murder rate column and a rank column as done before
- Create a table, call it `my_states`, that satisfies both the conditions: it is in the Northeast or West and the murder rate is less than 1. 
- Use select to show only the state name, the rate and the rank

`@hint`
Use the `my_states <- filter(murders, region %in% c("Northeast", "West") & rate < 1)` code. Then use `select`. The columns you want are state, rate and rank.

`@pre_exercise_code`
```{r}
library(dplyr)
library(dslabs)
data(murders)
```

`@sample_code`
```{r}
# add the rate column
murders <- mutate(murders, rate =  total / population * 100000, rank = rank(-rate))

# Create a table, call it my_states, that satisfies both the conditions 

# Use select to show only the state name, the murder rate and the rank
```

`@solution`
```{r}
# add the rate column
murders <- mutate(murders, rate =  total / population * 100000, rank = rank(-rate))

# Create a table, call it `my_states`, that satisfies both the conditions 
my_states <- filter(murders, region %in% c("Northeast", "West") & rate < 1)

# Use select to show only the state name, the murder rate and the rank
select(my_states, state, rate, rank)
```

`@sct`
```{r}
test_error()
test_function("filter")
test_object("my_states", undefined_msg = "Define my_states first!", incorrect_msg = "Use the filter, %in% and < commands.")
test_output_contains("select(my_states, state, rate, rank)", incorrect_msg = "You need the use the select function on `my_states`. You want to see state, rate, and rank.")
success_msg("Now you know how to combine functions and use them to get specific answers. Let's try it one more time!")
```

---

## Using the pipe %>%

```yaml
type: NormalExercise
key: 72305c532e
lang: r
xp: 100
skills:
  - 1
```

The pipe `%>%` can be used to perform operations sequentially 
without having to define intermediate objects. After redefining murder to include rate and rank.

```{r}
library(dplyr)
murders <- mutate(murders, rate =  total / population * 100000, rank = (-rate))
```

in the solution to the previous exercise we did the following:
```{r}
# Created a table 
my_states <- filter(murders, region %in% c("Northeast", "West") & rate < 1)

# Used select to show only the state name, the murder rate and the rank
select(my_states, state, rate, rank)
```

The pipe `%>%` permits us to perform both operation sequentially and without having to define an intermediate variable `my_states`

For example we could have mutated and selected in the same line like this:

```{r}
mutate(murders, rate =  total / population * 100000, rank = rank(-rate)) %>% 
    select(state, rate, rank)
```

Note that `select` no longer has a data frame as the first argument. The first argument is assumed to be the result of the operation conducted right before the  `%>%`

`@instructions`
- Repeat the previous exercise, but now instead of creating a new object, show the result and only include the state, rate, and rank columns in that order.
- Use a pipe `%>%` to do this in just one line.

`@hint`
Use code: `filter(murders, region %in% c("Northeast", "West") & rate < 1) %>% 
    select(state, rate, rank)`

`@pre_exercise_code`
```{r}
library(dplyr)
library(dslabs)
data(murders)
```

`@sample_code`
```{r}
## Define the rate column
murders <- mutate(murders, rate =  total / population * 100000, rank = rank(-rate))

# show the result and only include the state, rate, and rank columns, all in one line, in that order

```

`@solution`
```{r}
## Define the rate and rank column
murders <- mutate(murders, rate =  total / population * 100000, rank = rank(-rate))

# show the result and only include the state, rate, and rank columns, all in one line, in that order
filter(murders, region %in% c("Northeast", "West") & rate < 1) %>%  
   select(state, rate, rank)
   
```

`@sct`
```{r}
test_error()
test_function("filter", incorrect_msg = "Everything needs to be in one line! Use a pipe to help!")
test_output_contains("filter(murders, region %in% c('Northeast', 'West') & rate < 1) %>%  select(state, rate, rank)")
test_pipe(num = 1, absent_msg = "We want you to use a pipe %>%", insuf_msg = "We want you to use a pipe %>%")
success_msg("You're getting a pretty good hang of stuff now!")
```

---

## mutate, filter and select

```yaml
type: NormalExercise
key: 60fa0dbe3a
lang: r
xp: 100
skills:
  - 1
```

Now we will reset murders to the original table by using `data(murders)`.

`@instructions`
Use one line of code to create a new data frame, called `my_states`, that has murder rate and rank columns (with the rank ordered from highest to lowest), considers only states in the Northeast or West which have a murder rate lower than 1, and contain only the state, rate, and rank columns. The line should have four components separated by **three** `%>%` operators: 
- The original dataset `murders`
- A call to `mutate` to add the murder rate and the rank.
- A call to `filter` to keep only the states from the Northeast or West and that have a murder rate below 1.
- A call to `select` that keeps only the columns with the state name, the murder rate, and the rank. 

The line should look something like this `my_states <- murders %>%` mutate something `%>%` filter something `%>%` select something.
Columns in the final data frame MUST be in the order: `state`, `rate`, `rank`.

`@hint`
The following pieces of code should be joined by pipes:
- `mutate(rate =  total / population * 100000, rank = rank(-rate))`
- `filter(region %in% c("Northeast", "West") & rate < 1)`
- `select(state, rate, rank
Make sure you use 3 pipes.

`@pre_exercise_code`
```{r}
library(dplyr)
library(dslabs)
data(murders)
```

`@sample_code`
```{r}
# Create new data frame called my_states (with specifications in the instructions)

```

`@solution`
```{r}
# Create new data frame called my_states (with specifications in the instructions)
my_states <- murders %>% 
    mutate(rate =  total / population * 100000, rank = rank(-rate)) %>%
    filter(region %in% c("Northeast", "West") & rate < 1) %>%
    select(state, rate, rank)
    
```

`@sct`
```{r}
test_error()
test_function("mutate")
test_function("filter")
test_function("select")
test_object("my_states", undefined_msg = "Define my_states!", incorrect_msg = "There is an error with your definition of my_states. Check your spelling and your mutate, filter and select commands.")
test_pipe(num = 3, absent_msg = "We want you to use three pipes %>%", insuf_msg = "We want you to use three pipes %>%")
success_msg("This is absolutely awesome! You now know how to use basic data manipulation techniques in R.")
```

---

## End of Assessment 7

```yaml
type: PureMultipleChoiceExercise
key: 54b0aa0655
lang: r
xp: 50
skills:
  - 1
```

This is the end of the programming assignment for this section. Please DO NOT click through to additional assessments from this page. Please DO answer the question on this page. If you do click through, your scores may NOT be recorded.

Click on "Awesome" to get the "points" for this question and then return to the course on edX.

You can now close this window to go back to the <a href='https://www.edx.org/course/data-science-r-basics-2'>course</a>.

`@hint`
- No hint necessary!

`@possible_answers`
- [Awesome]
- Nope

`@feedback`
- Great! Now navigate back to the course on edX!
- Now navigate back to the course on edX!