Machine Learning (Chapter 16): Linear Discriminant Analysis (LDA)

By


 


Machine Learning (Chapter 16): Linear Discriminant Analysis (LDA)

Linear Discriminant Analysis (LDA) is a powerful technique in machine learning used for classification and dimensionality reduction. It is especially useful when dealing with datasets where you have multiple classes and you want to reduce the number of features while preserving as much of the class discriminatory information as possible.

Mathematical Formulation

The core idea behind LDA is to find a linear combination of features that best separates the classes. The goal is to project the data onto a lower-dimensional space where the classes are as distinct as possible.

Let's break down the mathematics behind LDA:

  1. Compute the Within-Class Scatter Matrix SWS_W: For each class kk, compute the scatter matrix SWkS_{Wk}:

    SWk=∑x∈Dk(x−μk)(x−μk)TS_{Wk} = \sum_{x \in D_k} (x - \mu_k)(x - \mu_k)^T

    where DkD_k is the set of data points in class kk, and μk\mu_k is the mean vector of class kk.

    The total within-class scatter matrix is:

    SW=∑k=1KSWkS_W = \sum_{k=1}^{K} S_{Wk}

    where KK is the number of classes.

  2. Compute the Between-Class Scatter Matrix SBS_B: Compute the scatter matrix between classes as:

    SB=∑k=1Knk(μk−μ)(μk−μ)TS_B = \sum_{k=1}^{K} n_k (\mu_k - \mu)(\mu_k - \mu)^T

    where nkn_k is the number of samples in class kk, μk\mu_k is the mean vector of class kk, and μ\mu is the overall mean vector of all classes.

  3. Solve the Generalized Eigenvalue Problem: To find the linear discriminants, solve the eigenvalue problem:

    SW−1SBw=λwS_W^{-1} S_B \mathbf{w} = \lambda \mathbf{w}

    where w\mathbf{w} represents the eigenvectors (discriminant vectors) and λ\lambda are the corresponding eigenvalues.

  4. Project the Data: Use the top dd eigenvectors (where dd is the number of classes minus one) to project the data onto a dd-dimensional space:

    Y=XWY = X \mathbf{W}

    where XX is the matrix of input features, and W\mathbf{W} is the matrix of top eigenvectors.

Example in Python

Let's apply LDA to a simple dataset using Python. We will use the Iris dataset for demonstration purposes.

python:
import numpy as np
from sklearn.datasets import load_iris
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Load the Iris dataset
data = load_iris()
X = data.data
y = data.target

# Standardize features
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Split data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.3, random_state=42)

# Apply LDA
lda = LinearDiscriminantAnalysis(n_components=2)
X_train_lda = lda.fit_transform(X_train, y_train)
X_test_lda = lda.transform(X_test)

# Train a classifier
from sklearn.linear_model import LogisticRegression
clf = LogisticRegression()
clf.fit(X_train_lda, y_train)

# Predict and evaluate
y_pred = clf.predict(X_test_lda)
accuracy = accuracy_score(y_test, y_pred)
print(f'Accuracy: {accuracy:.2f}')

# Plot the results
import matplotlib.pyplot as plt

plt.figure(figsize=(8, 6))
colors = ['navy', 'turquoise', 'darkorange']
for color, i, target_name in zip(colors, [0, 1, 2], data.target_names):
    plt.scatter(X_train_lda[y_train == i, 0], X_train_lda[y_train == i, 1], color=color, alpha=.8, label=target_name)
plt.xlabel('LD1')
plt.ylabel('LD2')
plt.title('LDA: Iris dataset')
plt.legend(loc='best', shadow=False, scatterpoints=1)
plt.show()

Explanation of the Code

  1. Data Loading and Preprocessing:

    • We load the Iris dataset and standardize the features to have zero mean and unit variance.

  2. Splitting the Data:

    • The dataset is split into training and testing sets.

  3. Applying LDA:

    • We apply LDA to reduce the dimensionality to 2 components.

  4. Training a Classifier:

    • A logistic regression classifier is trained on the reduced feature set.

  5. Evaluation:

    • We evaluate the accuracy of the classifier on the test set.

  6. Plotting:

    • We visualize the results to see how well LDA separates the classes in the reduced dimensional space.

By applying LDA, we can achieve a lower-dimensional representation of the data while maintaining class separability, which is useful for both visualization and improving the performance of machine learning models.

Comments

Post a Comment

Popular posts from this blog

Machine Learning (Chapter 41): The ROC Curve

By
Image
  Machine Learning (Chapter 39): The ROC Curve Introduction to ROC Curve The ROC (Receiver Operating Characteristic) curve is a graphical representation used in binary classification problems to evaluate the performance of a classifier. It plots the True Positive Rate (TPR) against the False Positive Rate (FPR) at various threshold levels. The curve illustrates the trade-offs between sensitivity (recall) and specificity, helping to visualize a classifier’s capability. A perfect classifier would have a curve that reaches the top-left corner of the plot, whereas a random classifier would result in a diagonal line from (0, 0) to (1, 1). Key Definitions Before diving into the mathematics, let's define key metrics: True Positive (TP) : Correctly predicted positives. False Positive (FP) : Incorrectly predicted positives. True Negative (TN) : Correctly predicted negatives. False Negative (FN) : Incorrectly predicted negatives. From these, we calculate: True Positive Rate (TPR) , also kn...
Read more

Machine Learning (Chapter 39): Bootstrapping & Cross-Validation

By
Image
  Machine Learning (Chapter 39): Bootstrapping & Cross-Validation Introduction In the context of machine learning, evaluating the performance of models is a critical step. Two essential techniques widely used for model evaluation and resampling are Bootstrapping and Cross-Validation . Both help estimate the performance of a model by making efficient use of available data, reducing overfitting, and improving generalization. 1. Bootstrapping Bootstrapping is a resampling method where subsets of data are sampled with replacement. The goal is to create multiple training datasets, then build models for each dataset, and assess model variability and performance. Key Concept : When we sample with replacement, some observations may appear multiple times in the bootstrap sample, while others may not appear at all. Mathematical Foundation Given a dataset D = { x 1 , x 2 , … , x n } D = \{x_1, x_2, \dots, x_n\} D = { x 1 ​ , x 2 ​ , … , x n ​ } , a bootstrap sample D b D_b D b ​ is crea...
Read more

Machine Learning (Chapter 40): Class Evaluation Measures

By
Image
  Machine Learning (Chapter 39): Class Evaluation Measures In machine learning, particularly classification problems, evaluation metrics are critical for understanding how well a model performs. These metrics help in comparing models, tuning parameters, and determining which model fits best for a specific problem. In this chapter, we will focus on several widely-used class evaluation measures, along with the corresponding mathematical formulas and practical examples. 1. Accuracy Accuracy measures the proportion of correctly predicted instances out of all instances in the dataset. It is simple but can be misleading for imbalanced datasets. Accuracy = T P + T N T P + T N + F P + F N \text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN} Accuracy = TP + TN + FP + FN TP + TN ​ where: T P TP TP = True Positives T N TN TN = True Negatives F P FP FP = False Positives F N FN FN = False Negatives Example: Let’s calculate accuracy in Python. python Copy code from sklearn.metrics import ac...
Read more