Maximum Likelihood

Maximum likelihood estimation chooses model parameters that assign high probability or density to the observed data. For independent observations $x_1,\dots ,x_n$ from $p(x\mid \theta )$,

$$L(\theta )=\prod _{i=1}^{n}p(x_i\mid \theta ),{\hat{\theta }}_{\mathrm{MLE}}=\arg \underset{\theta }{\max }L(\theta ).$$

Here $x_i$ is the $i$th observed data point, $\theta$ is the unknown model parameter, and $p(x_i\mid \theta )$ is the probability mass or density assigned by the model. The product appears because the observations are assumed independent; maximizing it chooses the parameter value that makes the whole observed sample most plausible.

The log-likelihood is usually optimized:

$$\ell (\theta )=\sum _{i=1}^n\log p(x_i\mid \theta ).$$

The logarithm turns the product into a sum without changing the maximizer, which is numerically more stable and easier to differentiate.

This connects probability laws to losses: Gaussian regression uses squared error, while logistic regression uses Bernoulli negative log-likelihood. MAP estimation adds a prior term to the same likelihood.

Worked computation

Suppose a coin is flipped 10 times and produces 7 heads. Under a Bernoulli model with head probability $p$, the likelihood is

$$L(p)=p^7(1-p{)}^3,\ell (p)=7\log p+3\log (1-p).$$

Differentiating gives

$$\frac{d\ell }{dp}=\frac{7}{p}-\frac{3}{1-p}.$$

Setting this derivative to zero yields $7(1-p)=3p$, so $\hat{p}=7/10=0.7$. The fair-coin log-likelihood is $10\log (0.5)\approx -6.9315$, while the fitted value gives $7\log (0.7)+3\log (0.3)\approx -6.1086$. The MLE is preferred because it assigns the observed sequence a higher probability, which appears as a less negative log-likelihood.

Caveats

MLE is a point estimate, so it needs confidence intervals or posterior summaries to express uncertainty. It can overfit flexible models, diverge under separation, and be biased when the likelihood is misspecified.

References

• Maximum likelihood estimation
• SciPy statistics reference

Nav

Section — Probability and Statistics

← Statistical Estimation Maximum A Posteriori Estimation →