Machine Learning (Chapter 10): Subset Selection

By

 



Machine Learning: Subset Selection (Chapter 10)

Introduction

Subset selection is a crucial technique in machine learning used for feature selection, model simplification, and improving model performance. It involves choosing a subset of relevant features from a larger set to build a more efficient and accurate model. This chapter delves into the mathematical foundations and practical implementations of subset selection.

Mathematical Formulation

Objective Function

The goal of subset selection is to select a subset of features SS from a total set of features FF such that a certain objective function is optimized. Typically, this objective function involves minimizing an error metric while considering the number of features used.

Let:

  • X\mathbf{X} be the matrix of features (with dimensions n×pn \times p), where nn is the number of samples and pp is the number of features.
  • y\mathbf{y} be the vector of target values (with dimensions n×1n \times 1).
  • S⊆{1,2,…,p}S \subseteq \{1, 2, \ldots, p\} be the subset of feature indices selected.
  • XS\mathbf{X}_S be the submatrix of X\mathbf{X} containing only the features in subset SS.

The objective function often used in subset selection is the Mean Squared Error (MSE) of a model, which can be expressed as:

MSE(XS,y)=1n∑i=1n(yi−y^i)2\text{MSE}(\mathbf{X}_S, \mathbf{y}) = \frac{1}{n} \sum_{i=1}^n (y_i - \hat{y}_i)^2

where y^i\hat{y}_i is the predicted value for the ii-th sample using the subset of features SS.

Feature Selection Criteria

Two common criteria for feature selection are:

  1. Forward Selection: Start with an empty set of features and add features one by one, choosing the feature that improves the model performance the most at each step.

  2. Backward Elimination: Start with all features and remove features one by one, choosing the feature whose removal improves the model performance the most at each step.

Example Python Code

Here's an example using Python and the sklearn library to perform subset selection through Recursive Feature Elimination (RFE) on a dataset.

Import Libraries

python:
import numpy as np
import pandas as pd
from sklearn.datasets import load_boston
from sklearn.feature_selection import RFE
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

Load Dataset

python:
# Load the Boston housing dataset
data = load_boston()
X = data.data
y = data.target

# Split the data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

Perform RFE for Feature Selection

python:
# Create a linear regression model
model = LinearRegression()

# Create the RFE model and select the top 5 features
rfe = RFE(model, n_features_to_select=5)
fit = rfe.fit(X_train, y_train)

# Print the selected features
print("Selected features:", fit.support_)
print("Feature ranking:", fit.ranking_)

Evaluate Model Performance

python:
# Fit the model using the selected features
selected_features = np.where(fit.support_)[0]
X_train_selected = X_train[:, selected_features]
X_test_selected = X_test[:, selected_features]

# Train the model
model.fit(X_train_selected, y_train)

# Make predictions
y_pred = model.predict(X_test_selected)

# Calculate and print the Mean Squared Error
mse = mean_squared_error(y_test, y_pred)
print("Mean Squared Error:", mse)

Conclusion

Subset selection is a powerful technique for enhancing machine learning models by focusing on the most relevant features. Through methods like Recursive Feature Elimination, practitioners can effectively reduce the dimensionality of their datasets, leading to more efficient and interpretable models. The combination of mathematical formulation and practical implementation in Python makes subset selection accessible and actionable for real-world machine learning tasks.

Comments

Post a Comment

Popular posts from this blog

Machine Learning (Chapter 41): The ROC Curve

By
Image
  Machine Learning (Chapter 39): The ROC Curve Introduction to ROC Curve The ROC (Receiver Operating Characteristic) curve is a graphical representation used in binary classification problems to evaluate the performance of a classifier. It plots the True Positive Rate (TPR) against the False Positive Rate (FPR) at various threshold levels. The curve illustrates the trade-offs between sensitivity (recall) and specificity, helping to visualize a classifier’s capability. A perfect classifier would have a curve that reaches the top-left corner of the plot, whereas a random classifier would result in a diagonal line from (0, 0) to (1, 1). Key Definitions Before diving into the mathematics, let's define key metrics: True Positive (TP) : Correctly predicted positives. False Positive (FP) : Incorrectly predicted positives. True Negative (TN) : Correctly predicted negatives. False Negative (FN) : Incorrectly predicted negatives. From these, we calculate: True Positive Rate (TPR) , also kn...
Read more

Machine Learning (Chapter 39): Bootstrapping & Cross-Validation

By
Image
  Machine Learning (Chapter 39): Bootstrapping & Cross-Validation Introduction In the context of machine learning, evaluating the performance of models is a critical step. Two essential techniques widely used for model evaluation and resampling are Bootstrapping and Cross-Validation . Both help estimate the performance of a model by making efficient use of available data, reducing overfitting, and improving generalization. 1. Bootstrapping Bootstrapping is a resampling method where subsets of data are sampled with replacement. The goal is to create multiple training datasets, then build models for each dataset, and assess model variability and performance. Key Concept : When we sample with replacement, some observations may appear multiple times in the bootstrap sample, while others may not appear at all. Mathematical Foundation Given a dataset D = { x 1 , x 2 , … , x n } D = \{x_1, x_2, \dots, x_n\} D = { x 1 ​ , x 2 ​ , … , x n ​ } , a bootstrap sample D b D_b D b ​ is crea...
Read more

Machine Learning (Chapter 40): Class Evaluation Measures

By
Image
  Machine Learning (Chapter 39): Class Evaluation Measures In machine learning, particularly classification problems, evaluation metrics are critical for understanding how well a model performs. These metrics help in comparing models, tuning parameters, and determining which model fits best for a specific problem. In this chapter, we will focus on several widely-used class evaluation measures, along with the corresponding mathematical formulas and practical examples. 1. Accuracy Accuracy measures the proportion of correctly predicted instances out of all instances in the dataset. It is simple but can be misleading for imbalanced datasets. Accuracy = T P + T N T P + T N + F P + F N \text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN} Accuracy = TP + TN + FP + FN TP + TN ​ where: T P TP TP = True Positives T N TN TN = True Negatives F P FP FP = False Positives F N FN FN = False Negatives Example: Let’s calculate accuracy in Python. python Copy code from sklearn.metrics import ac...
Read more