MAP vs MLE

The Derivation Bridge

Applying Bayes' Theorem to parameter estimation:

$$P(\theta | X) = \frac{P(X | \theta) P(\theta)}{P(X)}$$ $$\hat{\theta}_{\text{MAP}} = \arg\max_\theta \ln P(\theta | X) = \arg\max_\theta \left[ \ln P(X | \theta) + \ln P(\theta) \right]$$

  MLE Objective:    arg max_θ [ Log-Likelihood: ln P(X | θ) ]
  MAP Objective:    arg max_θ [ Log-Likelihood: ln P(X | θ)  +  Log-Prior: ln P(θ) ]
                                                                     │
                                                            Acts as Regularizer!

How MAP Derives L1 and L2 Regularization

Given Linear Regression $y = X w + \epsilon$ with Gaussian noise $\epsilon \sim \mathcal{N}(0, \sigma^2)$:

1. Gaussian Prior $\implies$ L2 Regularization (Ridge)

Assume prior $w_j \sim \mathcal{N}(0, \tau^2)$:

$$\ln P(w) = -\frac{1}{2\tau^2} \|w\|_2^2 + \text{const}$$ $$\hat{w}_{\text{MAP}} = \arg\min_w \left[ \sum_{i=1}^N (y_i - w^T x_i)^2 + \lambda \|w\|_2^2 \right] \quad \left( \lambda = \frac{\sigma^2}{\tau^2} \right)$$

2. Laplace Prior $\implies$ L1 Regularization (Lasso)

Assume prior $P(w_j) = \frac{1}{2b} \exp\left(-\frac{|w_j|}{b}\right)$:

$$\ln P(w) = -\frac{1}{b} \|w\|_1 + \text{const}$$ $$\hat{w}_{\text{MAP}} = \arg\min_w \left[ \sum_{i=1}^N (y_i - w^T x_i)^2 + \lambda \|w\|_1 \right] \quad \left( \lambda = \frac{\sigma^2}{b} \right)$$

Comparison Matrix

Say this out loud

"MLE maximizes data likelihood P(X|θ) without priors, making it prone to overfitting on small datasets. MAP incorporates a prior P(θ), maximizing log P(X|θ) + log P(θ). MAP bridges statistics and ML regularization: a Gaussian prior over weights derives L2 regularization (Ridge), while a Laplacian prior derives L1 regularization (Lasso). As sample size N grows, MAP converges to MLE."

Follow-ups to expect

• What is Full Bayesian Inference vs MAP? MAP outputs a single point estimate $\hat{\theta}_{\text{MAP}}$ (the mode of the posterior). Full Bayesian inference computes the full posterior distribution $P(\theta|X)$, integrating over all parameters for predictions: $P(y^*|x^*, X) = \int P(y^*|x^*, \theta) P(\theta|X) d\theta$.
• What is Conjugate Prior? A prior $P(\theta)$ is conjugate to likelihood $P(X|\theta)$ if the resulting posterior $P(\theta|X)$ belongs to the same probability distribution family as the prior (e.g. Beta prior + Binomial likelihood = Beta posterior).