Variational Inference

Variational Inference:

 1. Why approximate inference?

Analytical inference ("inference" here refer to computing the posterior distribution, not the one used in the typical deep learning literature, which is a forward pass at test time) is easy when the conjugate priors exist, and hard otherwise. So, in practice, the posterior is approximated. This can be problem at times with the underestimation of the uncertainty.


  
Variational inference (other name for the approximate inference) requires the following steps:
  1. Selection the family of distribution Q (called variational family).
  2. Find the best approximation by minimizing the KL divergence.
In variational inference the selection of Q can lead to different results - bigger Q results in more accuracy but is often difficult to compute (set of all the possible distributions); smaller Q results in less accurate results but is often easy to compute. The true posterior is to lie in Q, but it is often difficult to know. Note that in KL divergence the posterior can be unnormalized distribution, as the evidence or normalizing term does not play any role in optimization of the KL divergence.

 
2. What is the mean field model?

In the mean field variational inference the family of the variational distribution is chosen over the latent variable and are products (factorized in fact) of each local approximates (the distribution over the single latent variables). An example is given in slide 2 where the posterior is a normal distributional with full covariance. With the mean field variational inference the covariance matrix is limited to the diagonal and therefore, the approximation leads to errors.

 See the slide 3 for the formulation of the optimization problem where we have the products of approximating distribution over the each latent variables. Until the convergence the coordinate descent - finding q1 that minimizes the KL divergence, then q2, q3 and so on until the convergence can be reached.

 As shown in slide 4 the analytical solution for this optimization is derived - just follow through the derivation as its not difficult. The final solution can also be found in this slide.


 Follow the example on Ising model in the slide below, as to get more insights.

  

3. Variational Bayes EM.

The EM algorithm involves the optimization of the KL divergence in the E-step. Whenever the full posterior is difficult to compute the variational inference can be used to approximate the true posterior. Note that the full inference is by far the most accurate, but it is the slowest. The accuracy and speed trade off results in the following orders: full inference - mean field approximation - EM algorithm - variational EM - crisp EM.

Comments

Post a Comment

Popular posts from this blog

Notes on "Focal Loss for Dense Object Detection" (RetinaNet)

Image
Focal Loss for Dense Object Detection by Lin et al (2017) The central idea of this paper is a proposal for a new loss function to train one-stage detectors which works effectively for class imbalance problems (typically found in one-stage detectors such as SSD). The authors also introduce RetinaNet based on the proposed loss function which outperforms the state-of-the-art detection algorithms including the two-stage detectors. Focal loss function: pt is defined as p if y = 1 and p-1 otherwise. In this way, CE(pt) is the cross entropy where p is the probability score. As shown below, the cross entropy is denoted blue (gamma = 0). By adding the factor (see the focal loss FL(pt)) the loss function computes smaller value for the well classified examples and puts more focus on hard and misclassified eamples. In this way the easy background examples are dealt with, which enables the training of highly accurate desne object detection algorithms. Just like alpha-cross entropy is the comm
Read more

Introduction to Bayesian methods

Image
Introduction to Bayesian methods: 1. Bayesian way of thinking: A man is running. Why? Some of the possible explanations are (1) this man is in hurry, (2) this man is running, (3) this man just spotted a dragon, and (4) this man always runs. There are three principles in Bayesian way of thinking which we can use to conclude which of these explanations are the most reasonable. These principles are: 1. Use the prior knowledge on the problem - there is no dragon. 2. Choose the answer that explains the data the most - the man is not wearing the runner. 3. Avoid making extra assumptions - crude to think that any man always run. The answer number "1. this man is in hurry" should be chosen following this principles. 2. Review of probability: Checklists are: 1. What is the probability? 2. What are random variables? 3. What is the probability mass function (PMF)? 4. What is the probability density function (PDF)? 5. What does it mean for the random variables X and Y
Read more

Conjugate Priors

Conjugate Priors. 1. Analytical inference: See the Bayes formulae. The problem is that the evidence is difficult to estimate for the analytical inference. One way to get away is the maximum posterior (MAP) and is very easy to compute, but it comes with many problems. 1. Lack of the in-variance to re-parameterization. 2. The MAP cannot be used as the prior. 3. The MAP is an untypical point (the mode). 4. The credible regions may not be computed. 2. Conjugate distributions. Another approach to avoid computing the evidence is called the conjugate distributions. As the likelihood and the evidence is fixed by our model, we can only vary the prior so that it is easier to compute the posterior. I f the posterior distribution p(θ|x) are in the same family as the prior distribution p(θ), the prior and posterior are then called conjugate distributions, and the prior is called a conjugate prior for the likelihood. For example, the Gaussian family is conjugate to itself (or self-
Read more