In the last section, we imported a clinical cancer dataset and learned to manipulate it using base R (tools included in every R installation).
In this session, we’ll continue to work with the same dataset, but will introduce a set of tools specifically designed for data science in R. By the end of this session, you should be able to:
- install and load packages
- use tidyverse tools to import data and access rows/columns
- combine commands using pipes
- transform and summarize data
Please ensure RStudio is open with your project directory path (e.g.,
~/Desktop/intro_r) listed at the top of your Console. If you do not
see the path to your project directory, go to File -> Open Project and
navigate to the location of your project directory. Alternatively, using
your operating system’s file browser, double click on the
r_intro.Rrpoj file.
Create a new R script called class3.R, and add # Introduction to R: Class 3 as a title.
For this lesson, we’ll be working with a group of R packages called
tidyverse. A package is a group of related functions that help you
accomplish particular tasks. tidyverse
packages have been designed specifically to support tasks related to
data science, such as data manipulation, filtering, and visualization.
The first thing we need to do is install the software:
# install package
install.packages("tidyverse")A few notes about installing packages:
- You only need to perform this installation once per computer, or when updating R or the package.
- If you see red text output in the Console during this installation, don’t be alarmed: this doens’t necessarily indicate a problem. You are seeing a report of the various pieces of software being downloaded and installed.
- If prompted, you should install all packages (say yes or all), as well as yes to compiling any packages
- When the installation is complete (this may take several minutes),
you’ll see the command prompt (
>) in your Console.
Once you have the software installed, you’ll need to load it:
# load library/package
library(tidyverse)## ── Attaching packages ───────────────────────────────────────────────────────────────── tidyverse 1.3.0 ──
## ✓ ggplot2 3.3.2 ✓ purrr 0.3.4
## ✓ tibble 3.0.3 ✓ dplyr 1.0.1
## ✓ tidyr 1.1.1 ✓ stringr 1.4.0
## ✓ readr 1.3.1 ✓ forcats 0.5.0
## ── Conflicts ──────────────────────────────────────────────────────────────────── tidyverse_conflicts() ──
## x dplyr::filter() masks stats::filter()
## x dplyr::lag() masks stats::lag()
Loading packages is similar to opening a software application on your computer; it makes a previously installed set of software available for use. A few notes about loading packages:
- You’ll need to load packages every time you open RStudio (or R restarts)
- Loading
tidyverseloads a collection of packages; these are listed under “Attaching packages” - There are many other packages included as dependencies. If some of
them did not install successfully, you will receive an error at this
step. For this lesson, you can try
library(dplyr), and ask your instructor for help later. - The section in the output above referencing “Conflicts” shows you which functions you just loaded have names identical to packages you already have loaded (in base R). This shouldn’t affect the code we write in this lesson, though it’s useful to know the double colon syntax (::) allows you to reference functions in a different package with same name.
You can check to make sure the new package we’ll be using is available
by executing ?select in the Console, or by searching for that function
in the help panel. You can also look in the “Packages” tab in the same
panel. If the package (in this case, either tidyverse or dplyr) is
present in the list, it’s installed. If the box next to the package name
is checked, it’s loaded. In this lesson, if you receive an error saying
a function isn’t available or recognized, check to make sure the package
is loaded.
The first task we’ll undertake with our newly installed tidyverse
tools is importing our data:
# import data
clinical <- read_csv("data/clinical.csv")## Parsed with column specification:
## cols(
## .default = col_character(),
## age_at_diagnosis = col_double(),
## days_to_death = col_double(),
## days_to_birth = col_double(),
## days_to_last_follow_up = col_double(),
## cigarettes_per_day = col_double(),
## years_smoked = col_double(),
## year_of_birth = col_double(),
## year_of_death = col_double()
## )
## See spec(...) for full column specifications.
Note that this function looks similar to what we used in the last lesson
(read.csv), but the underscore replacing the dot means it’s actually a
different function. In fact, this is the data import function from
tidyverse. The output provided by this function indicates a few key
differences with our import yesterday.
The data import execution includes a description of how each variable
(column) is interpreted. In our data’s case, the numeric data are
col_double and the character data are col_character (not factors!).
We can explore these differences further:
# inspect object
str(clinical)## tibble [6,832 × 20] (S3: spec_tbl_df/tbl_df/tbl/data.frame)
## $ primary_diagnosis : chr [1:6832] "C34.1" "C34.1" "C34.3" "C34.1" ...
## $ tumor_stage : chr [1:6832] "stage ia" "stage ib" "stage ib" "stage ia" ...
## $ age_at_diagnosis : num [1:6832] 24477 26615 28171 27154 29827 ...
## $ vital_status : chr [1:6832] "dead" "dead" "dead" "alive" ...
## $ morphology : chr [1:6832] "8070/3" "8070/3" "8070/3" "8083/3" ...
## $ days_to_death : num [1:6832] 371 136 2304 NA 146 ...
## $ state : chr [1:6832] "live" "live" "live" "live" ...
## $ tissue_or_organ_of_origin : chr [1:6832] "C34.1" "C34.1" "C34.3" "C34.1" ...
## $ days_to_birth : num [1:6832] -24477 -26615 -28171 -27154 -29827 ...
## $ site_of_resection_or_biopsy: chr [1:6832] "C34.1" "C34.1" "C34.3" "C34.1" ...
## $ days_to_last_follow_up : num [1:6832] NA NA 2099 3747 NA ...
## $ cigarettes_per_day : num [1:6832] 10.96 2.19 1.64 1.1 NA ...
## $ years_smoked : num [1:6832] NA NA NA NA NA NA NA NA NA NA ...
## $ gender : chr [1:6832] "male" "male" "female" "male" ...
## $ year_of_birth : num [1:6832] 1936 1931 1927 1930 1923 ...
## $ race : chr [1:6832] "white" "asian" "white" "white" ...
## $ ethnicity : chr [1:6832] "not hispanic or latino" "not hispanic or latino" "not hispanic or latino" "not hispanic or latino" ...
## $ year_of_death : num [1:6832] 2004 2003 NA NA 2004 ...
## $ bcr_patient_barcode : chr [1:6832] "TCGA-18-3406" "TCGA-18-3407" "TCGA-18-3408" "TCGA-18-3409" ...
## $ disease : chr [1:6832] "LUSC" "LUSC" "LUSC" "LUSC" ...
## - attr(*, "spec")=
## .. cols(
## .. primary_diagnosis = col_character(),
## .. tumor_stage = col_character(),
## .. age_at_diagnosis = col_double(),
## .. vital_status = col_character(),
## .. morphology = col_character(),
## .. days_to_death = col_double(),
## .. state = col_character(),
## .. tissue_or_organ_of_origin = col_character(),
## .. days_to_birth = col_double(),
## .. site_of_resection_or_biopsy = col_character(),
## .. days_to_last_follow_up = col_double(),
## .. cigarettes_per_day = col_double(),
## .. years_smoked = col_double(),
## .. gender = col_character(),
## .. year_of_birth = col_double(),
## .. race = col_character(),
## .. ethnicity = col_character(),
## .. year_of_death = col_double(),
## .. bcr_patient_barcode = col_character(),
## .. disease = col_character()
## .. )
You may notice the presence of tbl and related labels in the classes
for this object. tbl stands for
tibble, which is a type of data frame
with specific constraints to ensure better data handling.
If you preview the dataset, it will look the same, and we can interact
with the data in the same way. These assumptions about the data mesh
nicely with the other tools in the tidyverse.
Now that our data are imported, we can explore the tidyverse functions
for extracting parts of the dataset.
First, we can explore selecting certain columns by name:
# selecting columns with tidyverse (dplyr)
sel_columns <- select(clinical, tumor_stage, ethnicity, disease)The syntax for the select function is to specify the dataset first,
then the names of each of the columns you would like to retain in the
output object. If we look at the object, we’ll see it has only three
columns but all rows.
You’ll note that the column headers don’t require quotation marks; this
is a shortcut programmed into tidyverse functions.
As with base R functions, we can also select a range of columns:
# select range of columns
sel_columns2 <- select(clinical, tumor_stage:vital_status)In addition to these approaches, we can also use other helper functions
for selecting columns: starts_with(), ends_with(), and contains()
are examples that assist in extracting columns with headers that meet
certain conditions. For example, using starts_with(tumor) in place of
the column names will give you all columns that start with the word
tumor.
We can use a separate function to extract rows that meet particular conditions:
# select rows conditionally: keep only lung cancer cases
filtered_rows <- filter(clinical, disease == "LUSC") The syntax here is similar to select, and the conditional filters can
be applied in similarly to base R functions.
Create a new object from clinical called race_disease that includes only the race, > ethnicity, and disease columns.
Create a new object from race_disease called race_BRCA that includes only BRCA (from
disease).
The last challenges used an intermediate object to obtain an object with two subsetting methods applied. It’s common in data science to apply more than one requirement for extracting data. If you want to avoid creating an intermediate object, you could nest one command inside the other:
# same task as challenges, but nested commands
race_BRCA2 <- select(filter(clinical, disease == "BRCA"), race, ethnicity, disease)In this case, filter(clinical, disease == "BRCA") becomes the input
for select.
While this is a common approach, especially in base R, it can be difficult for us as coders to read and interpret the code.
One of the most useful features of tidyverse is its inclusion of a
programming method called pipes. This approach can be found in many
programming languages, in part because of its utility: a pipe sends the
output from the lefthand side of the symbol as the input for the
righthand side. In R, pipes are represented as %>%.
We can use pipes to connect the same two data extraction tasks:
# same task as above, but with pipes
piped <- clinical %>%
select(race, ethnicity, disease) %>%
filter(disease == "BRCA")The command above starts by naming the object that will result from this
assignment. The dataset is named as the first input. Because executing
the name of an object sends the object contents as output, this means
the second line receives the object as input. The output from the
select line is sent as input to the filter line. This effectively
demonstrates how pipes can be used to connect multiple commands
together.
Now that we are running code in chunks that span multiple lines, you can see one of the other nice features of RStudio: your cursor can be placed on any line of the multi-line chunk when you execute, and the entire set of code will run together.
These examples also help highlight the importance of style and convention in code formatting. After the first line, the code is indented. While this isn’t necessary for the code to work, it does make it a lot easier to read and understand the code.
Let’s take a look at another example of piped commands:
# extract race, ethinicity, and disease from cases born prior to 1930
piped2 <- clinical %>%
filter(year_of_birth < 1930) %>%
select(race, ethnicity, disease)In the code above, we’re applying a mathematical condition to find
specific rows, and the selecting certain columns. Does the order of
commands differ? We can switch the order of the filter and select
lines to see:
piped3 <- clinical %>%
select(race, ethnicity, disease) %>%
filter(year_of_birth < 1930)The code above should give you an error, because in this case, the order
does matter! The output from the second line does not include the
year_of_birth column, so R is unable to apply the filter in the third
line.
Use pipes to extract the columns
gender,years_smoked, andyear_of_birthfrom the objectclinicalfor only living patients (vital_status) who have smoked fewer than 1cigarettes_per_day.
This lesson so far has mostly shown new ways of accomplishing the same
tasks we learned in the last lesson. tidyverse includes much more
functionality, however, including the ability to mutate columns.
Common tasks for which mutate is useful include unit conversions,
transformation, and creating ratios from among existing columns.
We can use this function to convert the days_to_death column to years:
# convert days to years
clinical_years <- clinical %>%
mutate(years_to_death = days_to_death / 365)The actual conversion works by providing a formula (days_to_death / 365) and the name of the new column (years_to_death). If you
inspect the resulting object, you’ll see years_to_death added as a new
column at the end of the table.
mutateworks by retaining all previous columns and creating new columns as per the formula specified.tidyversealso includestransmute, which drops the existing columns used to calculcate the new columns.
We can use mutate to perform multiple conversions at once:
# convert days to year and months at same time, and we don't always need to assign to object
clinical %>%
mutate(years_to_death = days_to_death / 365,
months_to_death = days_to_death / 30) %>%
glimpse() # preview data output## Rows: 6,832
## Columns: 22
## $ primary_diagnosis <chr> "C34.1", "C34.1", "C34.3", "C34.1", "C34.…
## $ tumor_stage <chr> "stage ia", "stage ib", "stage ib", "stag…
## $ age_at_diagnosis <dbl> 24477, 26615, 28171, 27154, 29827, 23370,…
## $ vital_status <chr> "dead", "dead", "dead", "alive", "dead", …
## $ morphology <chr> "8070/3", "8070/3", "8070/3", "8083/3", "…
## $ days_to_death <dbl> 371, 136, 2304, NA, 146, NA, 345, 716, 28…
## $ state <chr> "live", "live", "live", "live", "live", "…
## $ tissue_or_organ_of_origin <chr> "C34.1", "C34.1", "C34.3", "C34.1", "C34.…
## $ days_to_birth <dbl> -24477, -26615, -28171, -27154, -29827, -…
## $ site_of_resection_or_biopsy <chr> "C34.1", "C34.1", "C34.3", "C34.1", "C34.…
## $ days_to_last_follow_up <dbl> NA, NA, 2099, 3747, NA, 3576, NA, NA, 181…
## $ cigarettes_per_day <dbl> 10.9589041, 2.1917808, 1.6438356, 1.09589…
## $ years_smoked <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, 2…
## $ gender <chr> "male", "male", "female", "male", "male",…
## $ year_of_birth <dbl> 1936, 1931, 1927, 1930, 1923, 1942, 1953,…
## $ race <chr> "white", "asian", "white", "white", "not …
## $ ethnicity <chr> "not hispanic or latino", "not hispanic o…
## $ year_of_death <dbl> 2004, 2003, NA, NA, 2004, NA, 2005, 2006,…
## $ bcr_patient_barcode <chr> "TCGA-18-3406", "TCGA-18-3407", "TCGA-18-…
## $ disease <chr> "LUSC", "LUSC", "LUSC", "LUSC", "LUSC", "…
## $ years_to_death <dbl> 1.0164384, 0.3726027, 6.3123288, NA, 0.40…
## $ months_to_death <dbl> 12.366667, 4.533333, 76.800000, NA, 4.866…
The code above also features a new function, glimpse, that can be
useful when developing new piped code. Note that we did not assign the
output above to a new object; we allowed it to be printed to the
Console. Because this is a large dataset, that type of output can be
unweildy. glimpse allows us to see a preview of the data, including
the two new columns created.
Extract only lung cancer patients (LUSC, from disease) and create a new column called
total_cigrepresenting an estimate of the total number of cigarettes smoked (use columnsyears_smokedandcigarettes_per_day).
Our clinical dataset includes categorical (character) data. One
example is the gender column. We can assess the different categories
available using a base R function:
# show categories in gender
unique(clinical$gender)## [1] "male" "female" NA
tidyverse includes an approach called split-apply-combine that allows
us to:
- split data into groups,
- apply a task for each group,
- combine the results back together into a single table.
We can try out this approach by counting the number of each gender in our dataset:
# count number of individuals of each gender
clinical %>%
group_by(gender) %>%
tally() ## # A tibble: 3 x 2
## gender n
## <chr> <int>
## 1 female 3535
## 2 male 3258
## 3 <NA> 39
group_by is not particularly useful by itself, but powerful together
with a second function like tally. The two columns in the resulting
tibble represent the categories from group_by and the number of cases
for each gender (n).
An additional function for use with group_by is summarize:
# summarize average days to death by gender
clinical %>%
group_by(gender) %>%
summarize(mean_days_to_death = mean(days_to_death, na.rm = TRUE))## `summarise()` ungrouping output (override with `.groups` argument)
## # A tibble: 3 x 2
## gender mean_days_to_death
## <chr> <dbl>
## 1 female 947.
## 2 male 826.
## 3 <NA> NaN
Similar to mutate, we provide summarize with a formula indicating
how we would like the groups to be handled.
In the command above, we use na.rm = TRUE to exclude missing data from
the calculation of mean from days_to_death. We still have NA
reported in the output table, though, because of the NA category in
gender.
We can apply an additional filter to remove this missing data, prior to grouping:
# remove NA
clinical %>%
filter(!is.na(gender)) %>%
group_by(gender) %>%
summarize(mean_days_to_death = mean(days_to_death))## `summarise()` ungrouping output (override with `.groups` argument)
## # A tibble: 2 x 2
## gender mean_days_to_death
## <chr> <dbl>
## 1 female NA
## 2 male NA
Create object called smoke_complete from clinical that contains no missing data for cigarettes per day or age at diagnosis.
How do you save resulting table to file? How would you find this answer?
The solution to the challenges above represent the first of two datasets we’ll be using for data visualization in our next class. Make sure you’ve executed this code to save the filtered data file for use next time:
smoke_complete <- clinical %>%
filter(!is.na(age_at_diagnosis)) %>%
filter(!is.na(cigarettes_per_day))
write_csv(smoke_complete, "data/smoke_complete.csv")The command above uses write_csv, which is the tidyverse method of
saving a csv file. Base R possesses a function, write.csv, that
performs a similar task, but by default includes quotation marks around
cells with character data as well as row names (sequential numbers,
unless otherwise specified).
Create a new object called birth_complete that contains no missing data for year of birth or vital status.
This challenge begins filtering the second of our two datasets for next time. Make sure you include the filter to remove missing data that’s been encoded as “not reported”!
# make sure ALL missing data is removed!
birth_complete <- clinical %>%
filter(!is.na(year_of_birth)) %>%
filter(!is.na(vital_status)) %>%
filter(vital_status != "not reported")We’re going to perform one last manipulation on this second dataset for next time, which will allow us to reduce the total number of cancer types present in this dataset.
First, we’ll need to count how many cases for each cancer type exist in the dataset:
# counting number of records in each cancer
cancer_counts <- clinical %>%
count(disease) %>%
arrange(n) The count function is similar to tally, but doesn’t need to have
group_by applied first. The arrange function added at the end sorts
the table using the column specified. Although this isn’t necessary for
the analysis to proceed, it makes it easier for us to interpret the
results.
Next, we’ll identify which cancer types are represented by at least 500 cases in this dataset:
# get names of frequently occurring cancers
frequent_cancers <- cancer_counts %>%
filter(n >= 500) We can then use this object to filter based on the number of cases:
# extract data from cancers to keep
birth_reduced <- birth_complete %>%
filter(disease %in% frequent_cancers$disease)The new syntax here is %in%, which allows you to compare each entry in
disease from birth_complete to the disease column in
frequent_cancers (remember that frequent_cancers$disease means the
disease column from frequent_cancers). This keeps only cases from
the birth_complete dataset that are from cancers that are frequently
occurring.
Finally, we’ll write the final output to a file:
# save results to file in data/ named birth_reduced
write_csv(birth_reduced, "data/birth_reduced.csv")Extract all tumor stages with more than 200 cases (Hint: also check to see if there are any other missing/ambiguous data!)
In this session, we acquainted ourselves with tidyverse, and learned
some tools for data filtering and manipulation. We covered examples from
many of the main categories of data manipulation tasks; if you’d like
more information on these functions and others available (including
methods of joining multiple tables together), please check out the
dplyr
cheatsheet.
In the next session, we’ll wrap up the course by creating
publication-quality images using ggplot2, a data visualization package
in tidyverse, and the two datasets we filtered in the sections above.
This document is written in R
markdown, which is a method of
formatting text, code, and output to create documents that are sharable
with other people. While this document is intended to serve as a
reference for you to read while typing code into your own script, you
may also be interested in modifying and running code in the original R
markdown file (class3.Rmd in the GitHub repository).
Materials for all lessons in this course include:
- Class 1: R syntax, assigning objects, using functions
- Class 2: Data types and structures; slicing and subsetting data
- Class 3: Data manipulation with
dplyr - Class 4: Data visualization in
ggplot2
Answers to all challenge exercises are available here.
The following exercises all use the same clinical data from this
class.
How many hispanic or latino individuals in clinical are not also white? What are their races?
Create a new column for clinical called age_at_death that calculates
this statistic (in years) from year_of_birth and year_of_death.
dplyr includes several “helpers” that allows selection of columns
meeting particular criteria (described on the first page of the dplyr
cheatsheet near the top of the right hand column:
https://github.com/rstudio/cheatsheets/raw/master/data-transformation.pdf).
Using one of these tools, extract all columns that include “diagnosis”.
How many patients are hispanic or latino patients (column ethnicity), died after the year 2000 (year_of_death), and possess no missing data for cigarettes per day?