Recall in Week 7, you get familiar to R by using the Family Resource Survey data. Today we will keep explore the data by using its categorical variables. As usual we first load the necessary libraries.
+
Some tips to avoid R returning can’t find data errors:
+
Check your working directory by
+
+
getwd()
+
+
[1] "C:/Users/ziye/Documents/GitHub/stats/labs"
+
+
+
Check the relative path of your data folder on your PC/laptop, make sure you know the relative path of your data from your workding directory, returned by getwd().
+
Library knowledge used in today:
+
dplyr: a basic library provides a suite of functions for data manipulation
+
ggplot2: a widely-used data visualisation library to help you create nice plots through layered plotting.
+
tidyverse: a collection of R packages designed for data science, offering a cohesive framework for data manipulation, visualization, and analysis. Containing dyplyr, ggplot2 and other basic libraries.
+
broom: a part of the tidyverse and is designed to convert statistical analysis results into tidy data frames.
3.1.1 Data overview
-
# Load necessary libraries
-library(ggplot2)
+
if(!require("dplyr"))
+install.packages("dplyr")
-
Warning: package 'ggplot2' was built under R version 4.3.2
+
Warning: package 'dplyr' was built under R version 4.3.2
Warning: package 'dplyr' was built under R version 4.3.2
+
Warning: package 'ggplot2' was built under R version 4.3.2
+
library(dplyr)
+library(ggplot2)
-
#or use tidyverse which includes ggplot2, dplyr and other foundamental libraries, remember you need first install the package if you haven’t by using install.packages(“tidyverse”)
+
Or we can use library tidyverse which complies ggplot2, dplyr and other foundamental libraries together already, remember you need first install the package if you haven’t by using install.packages("tidyverse").
Recall in previous weeks, we used the following code to overview the dataset. Familiar yourself again by using them:
-
View(frs_data)
-glimpse(frs_data)
+
View(frs_data)
+glimpse(frs_data)
and also summary() to produce summaries of each variable
-
summary(frs_data)
+
summary(frs_data)
You may notice that for the numeric variables such as hh_income_gross (household gross income) and work_hours(worked hours per week), the summary() offers useful descriptive statistics. While for the qualitative information, such as age_group (age group), highest_qual (Highest educational qualification), marital_status (Marital status) and nssec (Socio-economic status), the summary() function is not that useful by providing mean or median values.
Performing descriptive analysis for categorical variables or qualitative variables, we focus on summarising the frequency and distribution of categories within the variable. This analysis helps understand the composition and diversity of categories in the data, which is especially useful for identifying patterns, common categories, or potential data imbalances.
By using ggplot2, it is easy to create some nice descriptive charts for the categorical variables, such like what you did for the continuous variables last week.
-
ggplot(frs_data, aes(x = highest_qual)) +
-geom_bar(fill="brown",width=0.5) +
-labs(title ="Histogram of Highest Qualification in FRS", x ="Highest Qualification", y ="Count")+#set text info
-theme_classic()#choose theme type, try theme_bw(), theme_minimal() see differences
+
ggplot(frs_data, aes(x = highest_qual)) +
+geom_bar(fill="brown",width=0.5) +
+labs(title ="Histogram of Highest Qualification in FRS", x ="Highest Qualification", y ="Count")+#set text info
+theme_classic()#choose theme type, try theme_bw(), theme_minimal() see differences
-
+
-
ggplot(frs_data, aes(x = health)) +
-geom_bar(fill="skyblue") +
-geom_text(stat ="count", aes(label = ..count..),vjust =-0.3,colour ="grey")+#add text
-labs(title ="Histogram of Health in FRS", x ="Health", y ="Count")+#set text info
-theme_minimal()
+
ggplot(frs_data, aes(x = health)) +
+geom_bar(fill="skyblue") +
+geom_text(stat ="count", aes(label = ..count..),vjust =-0.3,colour ="grey")+#add text
+labs(title ="Histogram of Health in FRS", x ="Health", y ="Count")+#set text info
+theme_minimal()
-
+
-
ggplot(frs_data, aes(x = nssec)) +
-geom_bar(fill ="yellow4") +
-labs(title ="Histogram of NSSEC in FRS", x ="NSSEC", y ="Count") +
-coord_flip()+#Flip the Axes, add a # in front of this line, to make the code in gray and you will see why we would better flip the axes at here
-theme_bw()
+
ggplot(frs_data, aes(x = nssec)) +
+geom_bar(fill ="yellow4") +
+labs(title ="Histogram of NSSEC in FRS", x ="NSSEC", y ="Count") +
+coord_flip()+#Flip the Axes, add a # in front of this line, to make the code in gray and you will see why we would better flip the axes at here
+theme_bw()
-
+
@@ -441,7 +460,7 @@
To calculate the correlation between categorical data, we first use Chi-squared test to assess the independence between pairs of categorical variables, then we use Cramer’s V to measures the strength of association - the correlation coefficents in R.
Pearson’s chi-squared test (χ2) is a statistical test applied to sets of categorical data to evaluate how likely it is that any observed difference between the sets arose by chance. If the p-value is low (typically < 0.05), it suggests a significant association between the two variables.
-
chisq.test(frs_data$health,frs_data$happy)
+
chisq.test(frs_data$health,frs_data$happy)
Warning in chisq.test(frs_data$health, frs_data$happy): Chi-squared
approximation may be incorrect
@@ -457,9 +476,9 @@
If you see a warning message of Chi-squared approximation may be incorrect. This is because some expected frequencies in one or more cells of the cross-tabular (health * happy) are too low. The df means degrees of freedom and it related to the size of the table and the number of categories in each variable. The most important message from the output is the estimated p-value, which shows as p-value < 2.2e-16 (2.2 with 16 decimals move to the left, it is a very small number so written in scientific notation). P-value of the chi-squared test is far smaller than 0.05, so we can say the correlation is statistically significant.
Cramér’s V is a measure of association for categorical (nominal or ordinal) data. It ranges from 0 (no association) to 1 (strong association). The main downside of using Cramer’s V is that no information is provided on whether the correlation is positive or negative. This is not a problem if the variable pair includes a nominal variable but represents an information loss if the both variables being correlated are ordinal.
-
# Install the 'vcd' package if not installed
-if(!require("vcd"))
-install.packages("vcd", repos ="https://cran.r-project.org")
+
# Install the 'vcd' package if not installed
+if(!require("vcd"))
+install.packages("vcd", repos ="https://cran.r-project.org")
Loading required package: vcd
@@ -469,10 +488,10 @@
Loading required package: grid
-
library(vcd)
-
-# Calculate Cramér's V
-assocstats(table(frs_data$health, frs_data$happy))
+
library(vcd)
+
+# Calculate Cramér's V
+assocstats(table(frs_data$health, frs_data$happy))
# load data
-LAcensus <-read.csv("../data/Census2011/UK_DistrictPercentages.csv") # Local authority level
+
# load data
+LAcensus <-read.csv("../data/Census2011/UK_DistrictPercentages.csv") # Local authority level
Using the district-level census dataset “UK_DistrictPercentages.csv”. the variable “Region” (labelled as Government Office Region) is used to explore regional inequality in health.
Familiar yourself with the dataset by using the same codes as last week:
-
#view the data
-View(LAcensus)
-glimpse(LAcensus)
+
#view the data
+View(LAcensus)
+glimpse(LAcensus)
The names() function returns all the column names.
-
names(df)
+
names(df)
The dim() function can merely returns the number of rows and number of columns.
-
dim(LAcensus)
+
dim(LAcensus)
[1] 406 128
There are 406 rows and 130 columns in the dataset. It would be very hard to scan throught the data if we use so many variables altogether. Therefore, we can select several columns to tailor for this practical. You can of course include other variables you are interested in also by their names:
Similar to last week, we build our linear regression model, but also include the Region_label variable into the model.
-
model <-lm(pct_Long_term_ill ~ pct_Males + pct_No_qualifications + pct_Higher_manager_prof + Region_label, data = df)
-
-summary(model)
+
model <-lm(pct_Long_term_ill ~ pct_Males + pct_No_qualifications + pct_Higher_manager_prof + Region_label, data = df)
+
+summary(model)
Call:
@@ -782,13 +801,13 @@
3.2.2Change the baseline category
If you would like to learn about differences in long-term illness between East of England and other regions in the UK, you need to change the baseline category (from London) to the East of England region (with variable name “Region_2”).
-
df$Region_label <-relevel(df$Region_label, ref ="East of England")
+
df$Region_label <-relevel(df$Region_label, ref ="East of England")
The regression model is specified again as follows:
df <- df %>%mutate(New_region_label =if_else(!Region_label %in%c("London","Wales","Scotland","Northern Ireland"), "Other regions in England",Region_label))
+
df <- df %>%mutate(New_region_label =if_else(!Region_label %in%c("London","Wales","Scotland","Northern Ireland"), "Other regions in England",Region_label))
This code may looks a bit complex. You can simply type ?mutate in your console. Now in your right hand Help window, the R studio offers your the explanation of the mutate function. This is a common way you can use R studio to help you learn what the function caate() creates new columns that are functions of existing variables. Therefore, the df %>% mutate() means add a new column into the current dataframe df; the New_region_label in the mutate() function indicates the name of this new column is New_region_label. The right side of the New_region_label = indicates the value we want to assign to the New_region_label in each row.
The right side of New_region_label is
@@ -894,7 +913,7 @@
-
table(df$New_region_label)
+
table(df$New_region_label)
London Northern Ireland Other regions in England
@@ -905,23 +924,23 @@
-
df[,c("Region_label","New_region_label")]
+
df[,c("Region_label","New_region_label")]
Now you will have a new qualitative variable named New_region_label in which the UK is divided into five regions: London, Other regions in England, Wales, Scotland and Northern Ireland.
Based on the newly generated qualitative variableNew_region_label, we can now build our new linear regression model. Don’t forget:
(1) R need to deal with the categorical variables in regression model in the factor type;
