QLIM Procedure

Marginal Likelihood

The Bayes theorem states that

p left-parenthesis theta vertical-bar bold y right-parenthesis proportional-to pi left-parenthesis theta right-parenthesis upper L left-parenthesis y vertical-bar theta right-parenthesis

where theta is a vector of parameters and pi left-parenthesis theta right-parenthesis is the product of the prior densities that are specified in the PRIOR statement. The term upper L left-parenthesis y vertical-bar theta right-parenthesis is the likelihood that is associated with the MODEL statement. The function pi left-parenthesis theta right-parenthesis upper L left-parenthesis y vertical-bar theta right-parenthesis is the nonnormalized posterior distribution over the parameter vector theta. The normalized posterior distribution (simply, the posterior distribution) is

p left-parenthesis theta vertical-bar bold y right-parenthesis equals StartFraction pi left-parenthesis theta right-parenthesis upper L left-parenthesis y vertical-bar theta right-parenthesis Over integral Underscript theta Endscripts pi left-parenthesis theta right-parenthesis upper L left-parenthesis y vertical-bar theta right-parenthesis d theta EndFraction

The denominator m left-parenthesis y right-parenthesis equals integral Underscript theta Endscripts pi left-parenthesis theta right-parenthesis upper L left-parenthesis y vertical-bar theta right-parenthesis d theta (also called the "marginal likelihood") is a quantity of interest because it represents the probability of the data after the effect of the parameter vector has been averaged out. Because of its interpretation, the marginal likelihood can be used in various applications, including model averaging, variable selection, and model selection.

A natural estimate of the marginal likelihood is provided by the harmonic mean,

m left-parenthesis y right-parenthesis equals left-brace StartFraction 1 Over n EndFraction sigma-summation Underscript i equals 1 Overscript n Endscripts StartFraction 1 Over upper L left-parenthesis y vertical-bar theta Subscript i Baseline right-parenthesis EndFraction right-brace Superscript negative 1

where theta Subscript i is a sample draw from the posterior distribution. In practical applications, this estimator has proven to be unstable.

An alternative and more stable estimator can be obtained with an importance sampling scheme. The auxiliary distribution for the importance sampler can be chosen through the cross entropy theory (Chan and Eisenstat 2015). In particular, given a parametric family of distributions, the auxiliary density function is chosen to be the one closest, in terms of the Kullback-Leibler divergence, to the probability density that would give a zero variance estimate of the marginal likelihood. In practical terms, this is equivalent to the following algorithm:

  1. Choose a parametric family, f left-parenthesis period comma beta right-parenthesis, for the parameters of the model: f left-parenthesis theta vertical-bar beta right-parenthesis.

  2. Evaluate the maximum likelihood estimator of beta by using the posterior samples theta 1 comma ellipsis comma theta Subscript n Baseline as data.

  3. Use f left-parenthesis theta Superscript asterisk Baseline vertical-bar ModifyingAbove beta With caret Subscript m l e Baseline right-parenthesis to generate the importance samples theta 1 Superscript asterisk Baseline comma ellipsis comma theta Subscript n Sub Superscript asterisk Subscript Superscript asterisk.

  4. Estimate the marginal likelihood:

    m left-parenthesis y right-parenthesis equals StartFraction 1 Over n Superscript asterisk Baseline EndFraction sigma-summation Underscript j equals 1 Overscript n Superscript asterisk Baseline Endscripts StartFraction upper L left-parenthesis y vertical-bar theta Subscript j Superscript asterisk Baseline right-parenthesis pi left-parenthesis theta Subscript j Superscript asterisk Baseline right-parenthesis Over f left-parenthesis theta Subscript j Superscript asterisk Baseline vertical-bar ModifyingAbove beta With caret Subscript m l e Baseline right-parenthesis EndFraction

The parametric family for the auxiliary distribution is chosen to be Gaussian. The parameters that are subject to bounds are transformed accordingly:

  • If negative normal infinity less-than theta less-than normal infinity, then p equals theta.

  • If m less-than-or-equal-to theta less-than normal infinity, then q equals log left-parenthesis theta minus m right-parenthesis.

  • If negative normal infinity less-than theta less-than-or-equal-to upper M, then r equals log left-parenthesis upper M minus theta right-parenthesis.

  • If m less-than-or-equal-to theta less-than-or-equal-to upper M, then s equals log left-parenthesis theta minus m right-parenthesis minus log left-parenthesis upper M minus theta right-parenthesis.

Assuming independence for the parameters that are subject to bounds, the auxiliary distribution to generate importance samples is

Start 4 By 1 Matrix 1st Row bold p 2nd Row bold q 3rd Row bold r 4th Row bold s EndMatrix tilde bold upper N left-bracket Start 4 By 1 Matrix 1st Row mu Subscript p Baseline 2nd Row mu Subscript q Baseline 3rd Row mu Subscript r Baseline 4th Row mu Subscript s Baseline EndMatrix comma Start 4 By 4 Matrix 1st Row 1st Column normal upper Sigma Subscript p Baseline 2nd Column 0 3rd Column 0 4th Column 0 2nd Row 1st Column 0 2nd Column normal upper Sigma Subscript q Baseline 3rd Column 0 4th Column 0 3rd Row 1st Column 0 2nd Column 0 3rd Column normal upper Sigma Subscript r Baseline 4th Column 0 4th Row 1st Column 0 2nd Column 0 3rd Column 0 4th Column normal upper Sigma Subscript r Baseline EndMatrix right-bracket

where bold p, bold q, bold r, and bold s are vectors that contain the transformations of the unbounded, bounded-below, bounded-above, and bounded-above-and-below parameters. Also, given the imposed independence structure, normal upper Sigma Subscript p can be a nondiagonal matrix, but normal upper Sigma Subscript q, normal upper Sigma Subscript r, and normal upper Sigma Subscript s are assumed to be diagonal matrices.

Last updated: June 19, 2025