Machine Learning (Chapter 28): Parameter Estimation II - Priors & Maximum A Posteriori (MAP)

By

 



Chapter 28: Parameter Estimation II - Priors & Maximum A Posteriori (MAP)

In the context of machine learning and statistics, parameter estimation is a crucial process. After exploring Maximum Likelihood Estimation (MLE) in previous chapters, we now delve into a more sophisticated technique: Maximum A Posteriori (MAP) Estimation. This method incorporates prior knowledge about the parameters, making it a more general approach than MLE.

1. Introduction to Priors

In Bayesian statistics, a prior represents our beliefs about the parameters before observing any data. It's a probability distribution that reflects our knowledge or assumptions about the parameter's values. Incorporating priors allows us to update our beliefs in light of new evidence, leading to the posterior distribution.

Given a parameter θ\theta and data XX, the prior distribution is denoted as P(θ)P(\theta). The likelihood, P(X∣θ)P(X|\theta), represents the probability of observing the data given the parameter θ\theta.

2. Posterior Distribution

The posterior distribution combines the prior and the likelihood to form the updated belief about θ\theta after observing the data XX. It is given by Bayes' theorem:

P(θ∣X)=P(X∣θ)⋅P(θ)P(X)P(\theta|X) = \frac{P(X|\theta) \cdot P(\theta)}{P(X)}

Where:

  • P(θ∣X)P(\theta|X) is the posterior distribution.
  • P(X∣θ)P(X|\theta) is the likelihood of the data given the parameters.
  • P(θ)P(\theta) is the prior distribution.
  • P(X)P(X) is the marginal likelihood or evidence, a normalizing constant.

3. Maximum A Posteriori (MAP) Estimation

MAP estimation finds the parameter value that maximizes the posterior distribution. Mathematically, MAP is expressed as:

θMAP=argmaxθ P(θ∣X)\theta_{\text{MAP}} = \underset{\theta}{\text{argmax}} \, P(\theta|X)

Using Bayes' theorem, this can be rewritten as:

θMAP=argmaxθ P(X∣θ)⋅P(θ)\theta_{\text{MAP}} = \underset{\theta}{\text{argmax}} \, P(X|\theta) \cdot P(\theta)

Since the evidence P(X)P(X) is constant with respect to θ\theta, it is often ignored in the optimization process, simplifying the expression to:

θMAP=argmaxθ [log⁡P(X∣θ)+log⁡P(θ)]\theta_{\text{MAP}} = \underset{\theta}{\text{argmax}} \, \left[ \log P(X|\theta) + \log P(\theta) \right]

This contrasts with MLE, which only maximizes the likelihood P(X∣θ)P(X|\theta) without considering the prior.

4. Example: MAP Estimation with Python

Let's consider a simple example where we estimate the mean of a Gaussian distribution using MAP.

Problem Setup:

  • Assume we have data X={x1,x2,…,xn}X = \{x_1, x_2, \dots, x_n\} drawn from a Gaussian distribution with an unknown mean μ\mu and a known variance σ2\sigma^2.
  • We assume a Gaussian prior on μ\mu with mean μ0\mu_0 and variance τ2\tau^2.

The likelihood is:

P(X∣μ)=∏i=1n12πσ2exp⁡(−(xi−μ)22σ2)P(X|\mu) = \prod_{i=1}^n \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x_i - \mu)^2}{2\sigma^2}\right)

The prior is:

P(μ)=12πτ2exp⁡(−(μ−μ0)22τ2)P(\mu) = \frac{1}{\sqrt{2\pi\tau^2}} \exp\left(-\frac{(\mu - \mu_0)^2}{2\tau^2}\right)

The posterior (ignoring the normalizing constant) is:

P(μ∣X)∝exp⁡(−12σ2∑i=1n(xi−μ)2−12τ2(μ−μ0)2)P(\mu|X) \propto \exp\left(-\frac{1}{2\sigma^2} \sum_{i=1}^n (x_i - \mu)^2 - \frac{1}{2\tau^2} (\mu - \mu_0)^2 \right)

Maximizing this with respect to μ\mu gives the MAP estimate.

Python Implementation:

python:

import numpy as np

# Given data
X = np.array([5.0, 6.0, 7.0, 8.0, 9.0])  # Sample data points
n = len(X)
sigma2 = 2.0  # Known variance of the likelihood
mu0 = 4.0    # Prior mean
tau2 = 1.0   # Prior variance

# MLE estimate of the mean
mu_mle = np.mean(X)

# MAP estimate of the mean
mu_map = (n * sigma2 * mu0 + tau2 * np.sum(X)) / (n * sigma2 + tau2)

print(f"MLE Estimate: {mu_mle:.2f}")
print(f"MAP Estimate: {mu_map:.2f}")

Output:

python:

MLE Estimate: 7.00
MAP Estimate: 6.83

In this example, the MLE estimate is simply the sample mean, while the MAP estimate incorporates the prior, pulling the estimate slightly towards the prior mean μ0\mu_0.

5. Conclusion

MAP estimation provides a powerful framework for parameter estimation by integrating prior knowledge. This approach is particularly useful when the dataset is small or when incorporating domain knowledge is crucial. In contrast to MLE, which only considers the likelihood, MAP offers a more flexible estimation method by leveraging priors.

Understanding the role of priors and how they influence the posterior distribution is key to effectively applying MAP in practical scenarios. By experimenting with different priors and analyzing their impact, one can gain deeper insights into the underlying parameters of a model.

This chapter provides a foundation for further exploration into Bayesian methods, which are instrumental in many advanced machine learning applications.

Comments

Post a Comment

Popular posts from this blog

Machine Learning (Chapter 41): The ROC Curve

By
Image
  Machine Learning (Chapter 39): The ROC Curve Introduction to ROC Curve The ROC (Receiver Operating Characteristic) curve is a graphical representation used in binary classification problems to evaluate the performance of a classifier. It plots the True Positive Rate (TPR) against the False Positive Rate (FPR) at various threshold levels. The curve illustrates the trade-offs between sensitivity (recall) and specificity, helping to visualize a classifier’s capability. A perfect classifier would have a curve that reaches the top-left corner of the plot, whereas a random classifier would result in a diagonal line from (0, 0) to (1, 1). Key Definitions Before diving into the mathematics, let's define key metrics: True Positive (TP) : Correctly predicted positives. False Positive (FP) : Incorrectly predicted positives. True Negative (TN) : Correctly predicted negatives. False Negative (FN) : Incorrectly predicted negatives. From these, we calculate: True Positive Rate (TPR) , also kn...
Read more

Machine Learning (Chapter 39): Bootstrapping & Cross-Validation

By
Image
  Machine Learning (Chapter 39): Bootstrapping & Cross-Validation Introduction In the context of machine learning, evaluating the performance of models is a critical step. Two essential techniques widely used for model evaluation and resampling are Bootstrapping and Cross-Validation . Both help estimate the performance of a model by making efficient use of available data, reducing overfitting, and improving generalization. 1. Bootstrapping Bootstrapping is a resampling method where subsets of data are sampled with replacement. The goal is to create multiple training datasets, then build models for each dataset, and assess model variability and performance. Key Concept : When we sample with replacement, some observations may appear multiple times in the bootstrap sample, while others may not appear at all. Mathematical Foundation Given a dataset D = { x 1 , x 2 , … , x n } D = \{x_1, x_2, \dots, x_n\} D = { x 1 ​ , x 2 ​ , … , x n ​ } , a bootstrap sample D b D_b D b ​ is crea...
Read more

Machine Learning (Chapter 40): Class Evaluation Measures

By
Image
  Machine Learning (Chapter 39): Class Evaluation Measures In machine learning, particularly classification problems, evaluation metrics are critical for understanding how well a model performs. These metrics help in comparing models, tuning parameters, and determining which model fits best for a specific problem. In this chapter, we will focus on several widely-used class evaluation measures, along with the corresponding mathematical formulas and practical examples. 1. Accuracy Accuracy measures the proportion of correctly predicted instances out of all instances in the dataset. It is simple but can be misleading for imbalanced datasets. Accuracy = T P + T N T P + T N + F P + F N \text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN} Accuracy = TP + TN + FP + FN TP + TN ​ where: T P TP TP = True Positives T N TN TN = True Negatives F P FP FP = False Positives F N FN FN = False Negatives Example: Let’s calculate accuracy in Python. python Copy code from sklearn.metrics import ac...
Read more