Introduction to Bayesian Analysis Procedures

Using Multiple Chains

An effective strategy to check for convergence is to run multiple chains and check whether they yield similar results. Then the chains must be combined appropriately to compute both multichain posterior summary statistics and multichain MCMC diagnostics. For multichain posterior summary statistics such as the posterior mean, posterior quantiles, posterior intervals, or the posterior covariance and correlation matrices, the chains can be combined into one long chain, and then the statistics can be computed as if there were only one chain. To compute the MCSE of multichain estimates, or to compute multichain convergence diagnostics, you must take more care to account for the different autocorrelation structures in each of the chains.

Multichain Standard Error of the Mean Estimate

Suppose there are m chains, each of n iterations. Then each chain yields an estimate of the posterior mean of the parameter of the form

theta overbar Subscript j Baseline equals StartFraction 1 Over n EndFraction sigma-summation Underscript t equals 1 Overscript n Endscripts theta Superscript j t

where index chains. Based on the central limit theorem, these estimates are independent and have the following asymptotic normal distributions:

theta overbar Subscript j Baseline tilde normal upper N left-parenthesis mu comma sigma squared slash normal upper E normal upper S normal upper S Subscript j Baseline right-parenthesis

where is the true posterior mean, is the true posterior variance, and is the effective sample size of in the jth chain.

The multichain estimate of the posterior mean is the mean of all the iterations over all chains, which is equivalent to taking the mean of single-chain posterior mean estimates. Then the asymptotic distribution of the multichain estimate of the posterior mean is

theta overbar equals StartFraction 1 Over m EndFraction sigma-summation Underscript j equals 1 Overscript m Endscripts theta overbar Subscript j Baseline tilde normal upper N left-parenthesis mu comma StartFraction sigma squared Over m squared EndFraction sigma-summation Underscript j equals 1 Overscript m Endscripts StartFraction 1 Over normal upper E normal upper S normal upper S Subscript j Baseline EndFraction right-parenthesis

Then is estimated using the multichain estimate of the posterior variance, denoted by , and obtained by computing the sample variance, using all chains as one large sample. Estimates of the effective sample sizes are obtained via , where is the estimated autocorrelation time for in the jth chain. This yields

ModifyingAbove normal upper V normal a normal r With caret left-parenthesis theta overbar right-parenthesis equals StartFraction ModifyingAbove sigma With caret squared Over n m EndFraction StartFraction sigma-summation Underscript j equals 1 Overscript m Endscripts tau Subscript j Baseline Over m EndFraction

The multichain MCSE is the square root of this value.

Multichain Effective Sample Size

From the preceding formula for multichain MCSE, multichain estimates of effective sample size and autocorrelation time naturally arise. The multichain estimate of autocorrelation time is simply the mean of the single-chain autocorrelation times:

tau equals StartFraction 1 Over m EndFraction sigma-summation Underscript j equals 1 Overscript m Endscripts tau Subscript j

Given , the multichain estimate of ESS is

normal upper E normal upper S normal upper S equals StartFraction n m Over tau EndFraction

Multichain Autocorrelations

The multichain estimate of autocorrelation time can be equivalently constructed by first computing the mean autocorrelations for each lag, then computing autocorrelation time using the mean autocorrelations. So the multichain estimate of the lag-h autocorrelation is the mean autocorrelation for that lag, given by

rho Subscript h Baseline equals StartFraction 1 Over m EndFraction sigma-summation Underscript j equals 1 Overscript m Endscripts rho Subscript j h

where is the lag-h autocorrelation from the jth chain.

Other Diagnostics

The Geweke, Heidelberger-Welch, and Raftery-Lewis diagnostics are solely single-chain diagnostics and cannot be constructed for multiple chains. They can be used with multiple chains by constructing them for each chain and then by checking whether all chains pass the diagnostic. For example, the Geweke diagnostic and the stationarity test of the Heidelberger-Welch diagnostic can both detect whether any of the chains have not yet converged. However, the half-width test of the Heidelberger-Welch diagnostic and the Raftery-Lewis diagnostic both assume that you will use only a single chain to do posterior inference. This means that these tests will sometimes tell you that you need longer chains than necessary to achieve adequate precision in estimating posterior quantities.

Last updated: July 09, 2026