-
Notifications
You must be signed in to change notification settings - Fork 0
/
week_2_lda_and_pca.py
187 lines (126 loc) · 312 KB
/
week_2_lda_and_pca.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
# -*- coding: utf-8 -*-
"""Week 2 LDA and PCA
Automatically generated by Colab.
Original file is located at
https://colab.research.google.com/drive/1lREANBYKqAWFGpXP3ZuNwqiHST19VbRn
# Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA) Dimensionality Reduction
Dimensionality reduction techniques are used to reduce the number of variables in high-dimensional datasets while maintaining the important data. PCA is a technique of feature extraction that reduces the number of variables of the dataset, while still maintaining as much information as possible. LDA is also a form of feature extraction that uses the distinctions between groups to separetwte two or more classes. LDA maximizes the distance bwteen the means of the classes while minimizing the variation within each class to do this. Below are flowcharts of the basic steps to both dimensionality reduction techniques utilized in this project.
### Datasets and Methods
Two different datasets were utilized for this project. For the principal component analysis, a breast cancer dataset was used with the target being the diagnosis of malignant or benign. Only the first four features of this dataset were used in this study. The texture mean and radius mean had small standard deviations across the dataset. The area mean had a fairly large range of numbers and a high standard deviation. For the PCA analysis, the data was standardized prior to analysis. The means were used to calculate the covariance matrix. From the matrix, the eigenvalues and eigenvectors were able to be calculated and sorted based on size. Using these values, the number of principal components were selected and the data was transformed. The values for 1, 2, and 3 principal components are shown below.
For the linear discriminant analysis, an iris dataset containing 3 different species of irises with 150 instances and 4 features was chosen. A class was created. Using the standard formulas, the within-class and between-class scatter matrices were calculated. The within-class matrix was transposed and reshaped. An eigendecomposition was performed to find the eigenvalues and eigenvectors. These values were stored and then the 1 dimensional data was visualized.
"""
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
#import dataset
from google.colab import files
uploaded=files.upload()
df= pd.read_csv("Breast Cancer.csv", usecols=['diagnosis', 'radius_mean', 'texture_mean', 'perimeter_mean','area_mean'])
df
df.describe()
#Seperate diagnosis column
diagnosis = df["diagnosis"].tolist()
X= df.drop("diagnosis", axis= 1)
"""### Principal Component Analysis"""
#Standardize data
X = (X-X.mean())/ X.std(ddof=0)
covariance_matrix= np.cov(X.T)
print (covariance_matrix)
#Find eigen values and vectors
eigenvalues, eigenvectors= np.linalg.eig(covariance_matrix)
#Sort eigen values and vectors from high to low
order = np.argsort(eigenvalues)[::-1]
sort_eigenvalues = eigenvalues[order]
sort_eigenvectors = eigenvectors[ :, order]
print(' Eigenvalues \n', sort_eigenvalues )
print('Eigenvectors \n', sort_eigenvectors )
# Select 1st Principal Component
n_components = 1
eigenvector_sub= sort_eigenvectors[:,0:n_components]
#Transform the data
X_1 = np.dot(eigenvector_sub.transpose(),X.transpose()).transpose()
X_1
# Select 2 Principal Components
n_components = 2
eigenvector_sub= sort_eigenvectors[:,0:n_components]
#Transform the data
X_2 = np.dot(eigenvector_sub.transpose(),X.transpose()).transpose()
df= pd.DataFrame(X_2)
df.to_csv ('PCA_Breast Cancer.csv', index=False)
print(X_2)
#Visualize 2 PCs
plt.scatter(X_2[:, 0], X_2[:, 1])
plt.xlabel('PC1'); plt.xticks([])
plt.ylabel('PC2'); plt.yticks([])
plt.title('2 components')
plt.show()
# Select 3 Principal Components
n_components = 3
eigenvector_sub= sort_eigenvectors[:,0:n_components]
#Transform the data
X_3 = np.dot(eigenvector_sub.transpose(),X.transpose()).transpose()
X_3
"""### Linear Discriminant Analysis"""
#Create LDA class
class LDA:
def __init__(self, n_components):
self.n_components= n_components
self.linear_discrims= None
def fit(self, X, y):
n_features = X.shape[1]
class_labels = np.unique(y)
mean_total = np.mean(X, axis=0)
# Calculate within-class scatter matrix
SW = np.zeros((n_features, n_features))
#Calculate between-class scatter matrix
SB = np.zeros((n_features, n_features))
for c in class_labels:
X_c = X[y == c]
mean_c = np.mean(X_c, axis=0)
#Transpose within-class
SW += (X_c - mean_c).T.dot((X_c - mean_c))
#Reshape
n_c = X_c.shape[0]
mean_diff = (mean_c - mean_total).reshape(n_features, 1)
SB += n_c * (mean_diff).dot(mean_diff.T)
#Perform eigendecomposition
eigendecomp = np.linalg.inv(SW).dot(SB)
# Find eigenvalues and eigenvectors
eigenvalues, eigenvectors = np.linalg.eig(eigendecomp)
# sort eigenvalues high to low
eigenvectors = eigenvectors.T
idxs = np.argsort(abs(eigenvalues))[::-1]
eigenvalues = eigenvalues[idxs]
eigenvectors = eigenvectors[idxs]
# store first n eigenvectors
self.linear_discrims = eigenvectors[0:self.n_components]
def transform(self, X):
# project data
return np.dot(X, self.linear_discrims.T)
if __name__ == "__main__":
# Import iris dataset
import matplotlib.pyplot as plt
from sklearn import datasets
data = datasets.load_iris()
X, y = data.data, data.target
# Project the data into 1D space
lda = LDA(1)
lda.fit(X, y)
X_projected = lda.transform(X)
print("Shape of X:", X.shape)
print("Shape of transformed X:", X_projected.shape)
x1 = X_projected[:, 0]
df= pd.DataFrame(X_projected)
df.to_csv('LDA_Iris.csv', index= False)
print(X_projected)
plt.scatter(
x1, y, edgecolor="none", alpha=0.8, cmap=plt.cm.get_cmap("viridis", 3)
)
plt.xlabel("Linear Discriminant 1")
plt.colorbar()
plt.show()
"""### Results and Conclusions
The principal component analysis performed in this study looked at the first three principal components in the breast cancer dataset. Given the large number of features in this dataset, only the first four were used for this analysis. Given the large standard deviation of the area mean, this feature is not the best to use for this analysis. Principal component analysis was performed for the first 3 principal components. Based on the eigenvalues and eigenvectors for each principal component, it is clear that dropping both components 2 and 3 results in a lot of data being lost. The scatterplot of PC1 and PC2 shows that there is a wide spread on both axes, which means that only using the first principal component would cause a significant amount of information to be lost. Based on the analysis, 3 principal components seems like the best fit.
The linear discriminant analysis on the iris dataset was done with the target being the iris species. Based on the graph of the first linear discriminant, it appears that at least 2 would be ideal given the wide spread of the graph. More research is needed to determine how many is ideal.
"""