Parameter Estimation

If we assume that our data is generated by a random variable ${X}$ with an underlying distribution ${P(X)}$ that is defined by a parameter, such as the mean ${\theta}$, then we can try to model the data generation process or make inferences based on an estimation of the parameter.

The better our estimation of the parameters, the better our fit of the distribution gets, and the better we can make guesses about unseen data. For example, if we say our Data is generated by a Gaussian Random Variable with mean ${\theta}$ and variance ${\sigma}$,

we would try something like below:

Frequentist vs Bayesian

Depending on if you believe that the parameters, ${\theta}$, ${\sigma}$ are fixed or Random Variables themselves there are two major approaches to parameter estimation.

Frequentists

If we believe ${\theta}$ is fixed but unknown. Then by the central limit theorem and the law of large numbers, observing the frequency of long experiments, eventually, we should be able to infer ${\theta}$ with a small enough error.

Bayesians

We believe ${\theta}$ is a random variable. Parameters are random variables. This allows us to use Bayes theorem to infer

the posterior. Or more general: .

Maximum Likelihood Estimation

Imagine that have a box with 5 balls inside. Some are blue and some are red. We sample the box three times, each time, we note down the color of the ball and then we put the ball back in the box, shake it and draw again. What is the probability that we draw a blue ball? Out of five balls in the box, ${\theta}$ are blue. So we have a ${\theta}$ in five chance to pick a blue ball:

Continuous Case: The likelihood function values we calculated for each ${\theta}$ as the outcome of 3 consecutive independent Bernoulli trials. Where blue in our trial was 2 and red is 1.

Maximum a Posteriori

So earlier, we assumed that ${\theta}$ was fixed and we tried to find the value for ${\theta}$ that made our observation most likely by maximizing the likelihood, the probability of seeing that outcome given a particular ${\theta}$.

Now we treat ${\theta}$ as a random variable, and we have assume that we have some prior knowledge of it’s distribution. We seek to update our belief (posterior) by the observation using Bayes Theorem.

In MAP estimation we find the value of ${\theta}$ that maximizes the posterior above. Which is simply the likelihood ${P(X | \theta)}$ times the prior (our initial guess) ${P(\theta)}$ divided by the evidence ${P(X)}$. This is one of the most beautiful equations that I know, because the more you think about it, the more it makes sense. But I digress.

In the algebraic sense, this is just a fraction, notice that ${P(X)}$ doesn’t really depend on ${\theta}$. So from the view of a function of ${\theta}$, it looks like this:

where ${c}$ is just some constant. If you aren’t convinced, below is a function which is multiplied by some ${c}$ constant. The red line indicates the maximum.

So we maximize

with respect to ${\theta}$ instead. Now there is an interesting case where ${P(\theta)}$ is constant. We go back to the above logic and find out we can safely ignore it and just maximize ${P(X | \theta)}$ with respect to ${\theta}$.

This should make intuitive sense when thinking bayesian. The posterior = updated confidence in the guess given the observation is the prior confidence in the guess times the probability of the observation given the guess. The stronger we belief in our original guess of ${\theta}$, the more likely If ${P(\theta)}$ is a constant ${c}$. Then it’s a uniform random variable and we think all