Machine Learning (Chapter 9): Multivariate Regression

By

 



Machine Learning (Chapter 9): Multivariate Regression

Introduction

Multivariate regression is an extension of simple linear regression that deals with multiple independent variables. It aims to model the relationship between two or more features and a dependent variable. This chapter explores the concept, mathematical formulation, and practical implementation of multivariate regression.

Mathematical Formulation

In multivariate regression, we predict the dependent variable yy using multiple independent variables x1,x2,…,xnx_1, x_2, \ldots, x_n. The relationship can be described using the following linear equation:

y=β0+β1x1+β2x2+⋯+βnxn+ϵy = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \cdots + \beta_n x_n + \epsilon

where:

  • yy is the dependent variable.
  • β0\beta_0 is the intercept.
  • β1,β2,…,βn\beta_1, \beta_2, \ldots, \beta_n are the coefficients of the independent variables x1,x2,…,xnx_1, x_2, \ldots, x_n.
  • ϵ\epsilon is the error term.

The goal is to estimate the coefficients β0,β1,β2,…,βn\beta_0, \beta_1, \beta_2, \ldots, \beta_n such that the sum of squared errors (or residuals) between the predicted values and the actual values is minimized.

Cost Function

The cost function for multivariate regression, known as the Mean Squared Error (MSE), is given by:

J(β)=12m∑i=1m(hβ(x(i))−y(i))2J(\beta) = \frac{1}{2m} \sum_{i=1}^m (h_{\beta}(x^{(i)}) - y^{(i)})^2

where:

  • mm is the number of training examples.
  • hβ(x(i))h_{\beta}(x^{(i)}) is the hypothesis function (i.e., the predicted value) for the ii-th training example.
  • y(i)y^{(i)} is the actual value for the ii-th training example.

The hypothesis function hβ(x)h_{\beta}(x) in multivariate regression is:

hβ(x)=β0+β1x1+β2x2+⋯+βnxnh_{\beta}(x) = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \cdots + \beta_n x_n

Gradient Descent

To minimize the cost function, we use gradient descent. The update rule for the coefficients is:

βj:=βj−α∂J(β)∂βj\beta_j := \beta_j - \alpha \frac{\partial J(\beta)}{\partial \beta_j}

where α\alpha is the learning rate, and ∂J(β)∂βj\frac{\partial J(\beta)}{\partial \beta_j} is the partial derivative of the cost function with respect to βj\beta_j:

∂J(β)∂βj=1m∑i=1m(hβ(x(i))−y(i))xj(i)\frac{\partial J(\beta)}{\partial \beta_j} = \frac{1}{m} \sum_{i=1}^m (h_{\beta}(x^{(i)}) - y^{(i)}) x_j^{(i)}

Python Implementation

Below is a Python implementation of multivariate regression using scikit-learn and numpy.

python:
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

# Example data
# Features: [x1, x2]
X = np.array([[1, 2], [2, 3], [3, 4], [4, 5]])
# Target variable
y = np.array([3, 5, 7, 9])

# Create and fit the model
model = LinearRegression()
model.fit(X, y)

# Predictions
y_pred = model.predict(X)

# Coefficients
intercept = model.intercept_
coefficients = model.coef_

print(f'Intercept: {intercept}')
print(f'Coefficients: {coefficients}')

# Model performance
mse = mean_squared_error(y, y_pred)
print(f'Mean Squared Error: {mse}')

Explanation

  1. Data Preparation: We define our feature matrix XX and target vector yy.
  2. Model Training: We create an instance of LinearRegression and fit it to our data.
  3. Predictions: We use the model to predict the target values for our input features.
  4. Evaluation: We calculate the Mean Squared Error to assess the model’s performance.

Conclusion

Multivariate regression allows us to model complex relationships between multiple features and a target variable. By understanding the mathematical foundation and applying it through practical implementation, we can make accurate predictions and gain insights from our data.

Comments

Post a Comment

Popular posts from this blog

Machine Learning (Chapter 41): The ROC Curve

By
Image
  Machine Learning (Chapter 39): The ROC Curve Introduction to ROC Curve The ROC (Receiver Operating Characteristic) curve is a graphical representation used in binary classification problems to evaluate the performance of a classifier. It plots the True Positive Rate (TPR) against the False Positive Rate (FPR) at various threshold levels. The curve illustrates the trade-offs between sensitivity (recall) and specificity, helping to visualize a classifier’s capability. A perfect classifier would have a curve that reaches the top-left corner of the plot, whereas a random classifier would result in a diagonal line from (0, 0) to (1, 1). Key Definitions Before diving into the mathematics, let's define key metrics: True Positive (TP) : Correctly predicted positives. False Positive (FP) : Incorrectly predicted positives. True Negative (TN) : Correctly predicted negatives. False Negative (FN) : Incorrectly predicted negatives. From these, we calculate: True Positive Rate (TPR) , also kn...
Read more

Machine Learning (Chapter 39): Bootstrapping & Cross-Validation

By
Image
  Machine Learning (Chapter 39): Bootstrapping & Cross-Validation Introduction In the context of machine learning, evaluating the performance of models is a critical step. Two essential techniques widely used for model evaluation and resampling are Bootstrapping and Cross-Validation . Both help estimate the performance of a model by making efficient use of available data, reducing overfitting, and improving generalization. 1. Bootstrapping Bootstrapping is a resampling method where subsets of data are sampled with replacement. The goal is to create multiple training datasets, then build models for each dataset, and assess model variability and performance. Key Concept : When we sample with replacement, some observations may appear multiple times in the bootstrap sample, while others may not appear at all. Mathematical Foundation Given a dataset D = { x 1 , x 2 , … , x n } D = \{x_1, x_2, \dots, x_n\} D = { x 1 ​ , x 2 ​ , … , x n ​ } , a bootstrap sample D b D_b D b ​ is crea...
Read more

Machine Learning (Chapter 40): Class Evaluation Measures

By
Image
  Machine Learning (Chapter 39): Class Evaluation Measures In machine learning, particularly classification problems, evaluation metrics are critical for understanding how well a model performs. These metrics help in comparing models, tuning parameters, and determining which model fits best for a specific problem. In this chapter, we will focus on several widely-used class evaluation measures, along with the corresponding mathematical formulas and practical examples. 1. Accuracy Accuracy measures the proportion of correctly predicted instances out of all instances in the dataset. It is simple but can be misleading for imbalanced datasets. Accuracy = T P + T N T P + T N + F P + F N \text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN} Accuracy = TP + TN + FP + FN TP + TN ​ where: T P TP TP = True Positives T N TN TN = True Negatives F P FP FP = False Positives F N FN FN = False Negatives Example: Let’s calculate accuracy in Python. python Copy code from sklearn.metrics import ac...
Read more