Machine Learning (Chapter 27): Maximum Likelihood Estimate (MLE)

By


 

Machine Learning (Chapter 27): Maximum Likelihood Estimate (MLE)

Introduction to Maximum Likelihood Estimate (MLE)

Maximum Likelihood Estimation (MLE) is a fundamental statistical method used in machine learning and statistics to estimate the parameters of a probability distribution or statistical model. The core idea behind MLE is to find the parameter values that maximize the likelihood function, which measures how likely it is to observe the given data under different parameter values.

Likelihood Function

Given a dataset D={x1,x2,…,xn}\mathcal{D} = \{x_1, x_2, \dots, x_n\} and a probabilistic model with a parameter θ\theta, the likelihood function L(θ)\mathcal{L}(\theta) is defined as the probability of observing the data given the parameter θ\theta. Mathematically, it is expressed as:

L(θ)=P(D∣θ)=∏i=1nP(xi∣θ)\mathcal{L}(\theta) = P(\mathcal{D}|\theta) = \prod_{i=1}^{n} P(x_i|\theta)

For convenience, the log-likelihood function is often used, which is the natural logarithm of the likelihood function:

log⁡L(θ)=∑i=1nlog⁡P(xi∣θ)\log \mathcal{L}(\theta) = \sum_{i=1}^{n} \log P(x_i|\theta)

Maximum Likelihood Estimation

The goal of MLE is to find the value of θ\theta that maximizes the likelihood function. Formally, this can be expressed as:

θ^=argmaxθ L(θ)=argmaxθ log⁡L(θ)\hat{\theta} = \underset{\theta}{\text{argmax}} \, \mathcal{L}(\theta) = \underset{\theta}{\text{argmax}} \, \log \mathcal{L}(\theta)

Example: MLE for Gaussian Distribution

Let's consider a simple example where we estimate the mean and variance of a Gaussian distribution using MLE. Given a dataset D={x1,x2,…,xn}\mathcal{D} = \{x_1, x_2, \dots, x_n\} that is assumed to be drawn from a Gaussian distribution with mean μ\mu and variance σ2\sigma^2, the probability density function is:

P(x∣μ,σ2)=12πσ2exp⁡(−(x−μ)22σ2)P(x|\mu, \sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x - \mu)^2}{2\sigma^2}\right)

The likelihood function for the dataset is:

L(μ,σ2)=∏i=1n12πσ2exp⁡(−(xi−μ)22σ2)\mathcal{L}(\mu, \sigma^2) = \prod_{i=1}^{n} \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x_i - \mu)^2}{2\sigma^2}\right)

Taking the logarithm of the likelihood function, we get the log-likelihood:

log⁡L(μ,σ2)=−n2log⁡(2πσ2)−12σ2∑i=1n(xi−μ)2\log \mathcal{L}(\mu, \sigma^2) = -\frac{n}{2} \log(2\pi\sigma^2) - \frac{1}{2\sigma^2} \sum_{i=1}^{n} (x_i - \mu)^2

To find the MLE estimates μ^\hat{\mu} and σ^2\hat{\sigma}^2, we take the partial derivatives of the log-likelihood function with respect to μ\mu and σ2\sigma^2, and set them to zero:

∂log⁡L(μ,σ2)∂μ=1σ2∑i=1n(xi−μ)=0\frac{\partial \log \mathcal{L}(\mu, \sigma^2)}{\partial \mu} = \frac{1}{\sigma^2} \sum_{i=1}^{n} (x_i - \mu) = 0 ∂log⁡L(μ,σ2)∂σ2=−n2σ2+12σ4∑i=1n(xi−μ)2=0\frac{\partial \log \mathcal{L}(\mu, \sigma^2)}{\partial \sigma^2} = -\frac{n}{2\sigma^2} + \frac{1}{2\sigma^4} \sum_{i=1}^{n} (x_i - \mu)^2 = 0

Solving these equations, we obtain:

μ^=1n∑i=1nxi\hat{\mu} = \frac{1}{n} \sum_{i=1}^{n} x_i σ^2=1n∑i=1n(xi−μ^)2\hat{\sigma}^2 = \frac{1}{n} \sum_{i=1}^{n} (x_i - \hat{\mu})^2

Python Implementation

Below is a Python implementation of MLE for estimating the parameters of a Gaussian distribution.

python:

import numpy as np

# Sample data
data = np.array([2.3, 2.5, 3.1, 4.0, 4.2, 5.5, 5.7, 6.0])

# MLE estimation of mean
mu_hat = np.mean(data)

# MLE estimation of variance
sigma2_hat = np.var(data, ddof=0)  # ddof=0 for MLE (population variance)

print(f"Estimated mean (MLE): {mu_hat}")
print(f"Estimated variance (MLE): {sigma2_hat}")

Output:

java:

Estimated mean (MLE): 4.1625
Estimated variance (MLE): 1.70859375

Conclusion

Maximum Likelihood Estimation is a powerful and widely-used method for parameter estimation in statistical models. By maximizing the likelihood function, MLE provides estimates of model parameters that best explain the observed data. The example provided demonstrates how MLE can be applied to estimate the mean and variance of a Gaussian distribution, and the corresponding Python implementation illustrates how these concepts can be applied in practice.

This chapter has covered the mathematical foundations and practical application of MLE, setting the stage for more complex models and estimation techniques in the following chapters.

Comments

Post a Comment

Popular posts from this blog

Machine Learning (Chapter 41): The ROC Curve

By
Image
  Machine Learning (Chapter 39): The ROC Curve Introduction to ROC Curve The ROC (Receiver Operating Characteristic) curve is a graphical representation used in binary classification problems to evaluate the performance of a classifier. It plots the True Positive Rate (TPR) against the False Positive Rate (FPR) at various threshold levels. The curve illustrates the trade-offs between sensitivity (recall) and specificity, helping to visualize a classifier’s capability. A perfect classifier would have a curve that reaches the top-left corner of the plot, whereas a random classifier would result in a diagonal line from (0, 0) to (1, 1). Key Definitions Before diving into the mathematics, let's define key metrics: True Positive (TP) : Correctly predicted positives. False Positive (FP) : Incorrectly predicted positives. True Negative (TN) : Correctly predicted negatives. False Negative (FN) : Incorrectly predicted negatives. From these, we calculate: True Positive Rate (TPR) , also kn...
Read more

Machine Learning (Chapter 39): Bootstrapping & Cross-Validation

By
Image
  Machine Learning (Chapter 39): Bootstrapping & Cross-Validation Introduction In the context of machine learning, evaluating the performance of models is a critical step. Two essential techniques widely used for model evaluation and resampling are Bootstrapping and Cross-Validation . Both help estimate the performance of a model by making efficient use of available data, reducing overfitting, and improving generalization. 1. Bootstrapping Bootstrapping is a resampling method where subsets of data are sampled with replacement. The goal is to create multiple training datasets, then build models for each dataset, and assess model variability and performance. Key Concept : When we sample with replacement, some observations may appear multiple times in the bootstrap sample, while others may not appear at all. Mathematical Foundation Given a dataset D = { x 1 , x 2 , … , x n } D = \{x_1, x_2, \dots, x_n\} D = { x 1 ​ , x 2 ​ , … , x n ​ } , a bootstrap sample D b D_b D b ​ is crea...
Read more

Machine Learning (Chapter 40): Class Evaluation Measures

By
Image
  Machine Learning (Chapter 39): Class Evaluation Measures In machine learning, particularly classification problems, evaluation metrics are critical for understanding how well a model performs. These metrics help in comparing models, tuning parameters, and determining which model fits best for a specific problem. In this chapter, we will focus on several widely-used class evaluation measures, along with the corresponding mathematical formulas and practical examples. 1. Accuracy Accuracy measures the proportion of correctly predicted instances out of all instances in the dataset. It is simple but can be misleading for imbalanced datasets. Accuracy = T P + T N T P + T N + F P + F N \text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN} Accuracy = TP + TN + FP + FN TP + TN ​ where: T P TP TP = True Positives T N TN TN = True Negatives F P FP FP = False Positives F N FN FN = False Negatives Example: Let’s calculate accuracy in Python. python Copy code from sklearn.metrics import ac...
Read more