Parameter Estimation - Part I

2. Maximum likelihood estimation(MLE)

The working principle of this method is that if the observation process gives rise to certain data, then it is reasonable to assume that the unknown, “true” parameters driving the process are those assigning the highest probability to the observed data. We exemplify this concept in the following two simple examples. In particular, Example 10.1 will be fundamental for the rest of the Unit.

Example 10.1. Coin tossing.
To estimate the probability of getting a "Head" when tossing a coin, a coin has been tossed 5 times obtaining head (H), tail (T), T, T, H. Let $\theta$ the unknown success probability (i.e., the probability of getting a H). The probability of the observed sequence of heads and tails is

P(H,T,T,T,H) = $\theta$ (1- $\theta$) (1- $\theta$) (1- $\theta$) $\theta$ = $\theta$² (1- $\theta$)³

that is, the product of the likelihood of each observation. The following picture shows P(H,T,T,T,H) as a function of $\theta$, i.e. the likelihood function. By the usual derivatives-based techniques it is easy to show that the maximum value of the likelihood is reached for $\theta$ = 2/5= 0.4, the proportion of observed heads.

Example 10.2. Gaussian model.
To estimate the mean tree diameter in a forest, a sample of 10 measures (cm) has been collected:

88.4, 91.5, 83.7, 105.4, 120.5, 97.3, 110.2, 99.5, 80.0, 100.2 .

We assume that the tree diameter follows a Gaussian distribution with unknown mean μ and standard deviation σ.

In analogy with Example 10.1, the likelihood function is the product of the “likelihood” of each collected measure, where the likelihood is given by the Gaussian density:

$$\frac{exp⁡{(-(88.4-μ)^2)⁄(2\sigma^2)}}{\sqrt{(2π\sigma^2)}}×\frac{exp⁡{(-(91.5-μ)^2)⁄(2\sigma^2)}}{\sqrt{(2π\sigma^2)}}×…×\frac{exp⁡{(-(100.2-μ)^2)⁄(2\sigma^2)}}{\sqrt{(2π\sigma^2)}}$$

The two-dimensional likelihood function is shown in the following picture

The ML estimates are given by the sample mean 97.67 cm and the sample standard deviation^{^[1]} 11.74 cm.

[1] $n^{-1} \sum_{i=1,…,n}(x_i-\bar{x})^2 $

The main question is to find the likelihood function for the IFM. In the following sections we will answer to this question in the case of only one snapshot of data. The more general case will be considered in Unit 11.