Computing the Bayesian evidence

Stage 4 project, 2025/26

Supervisor: Darren Wilkinson

Project outline

tl;dr

In the context of Bayesian inference, the (intractable) denominator of Bayes’ Theorem is variously known as the marginal likelihood, the evidence, or model evidence. Knowledge of this normalising constant is of interest for many reasons. In particular, the Bayes factor, often used in the context of Bayesian model comparison, is the ratio of the marginal likelihoods associated with two models. Unfortunately, reliable and efficient computation of the evidence is notoriously difficult. This project will be concerned with surveying some of the available methods and comparing their performance on some challenging statistical models.

A few details

Bayesian inference for parameters \theta given data y is conducted via Bayes’ Theorem, which can be written in the form \pi(\theta|y) = \frac{\pi(\theta)\pi(y|\theta)}{\pi(y)}. The denominator \pi(y) = \int_\Theta \pi(\theta)\pi(y|\theta)\,\rm{d}\theta is the evidence. We can regard this as an expectation wrt the prior distribution, \pi(y) = \mathbb{E}_\Theta[\pi(y|\Theta)], so given a sample \{\theta_i|i=1,\ldots,n\} from the prior, we can use the estimate \hat{\pi}(y) = \frac{1}{n}\sum_{i=1}^n \pi(y|\theta_i). Unfortunately, since the likelihood is typically very concentrated relative to the prior, this estimate is often very unstable. Since we often use MCMC to obtain a (approximate) sample from the posterior distribution, it seems promising to instead compute the evidence as an expectation wrt the posterior. We can do that, since \mathbb{E}_{\Theta|y}\left[\frac{1}{\pi(y|\Theta)}\right] = \int_\Theta \frac{\pi(\theta|y)}{\pi(y|\theta)}\,\rm{d}\theta = \int_\Theta \frac{\pi(\theta)}{\pi(y)}\,\rm{d}\theta = \frac{1}{\pi(y)}, leading to the infamous harmonic mean estimator, \tilde{\pi}(y) = \frac{1}{\displaystyle \frac{1}{n}\sum_{i=1}^n \frac{1}{\pi(y|\theta_i)}}, where now the \theta_i are a sample from the posterior. Unfortunately, the harmonic mean estimator is typically also very unstable.

Fortunately, it turns out that there are more stable estimators of the evidence, and this project will be concerned with exploring some of these.

Potential areas for more in-depth study

  • Stability of the harmonic mean estimator
  • Chib’s estimator using MCMC output
  • Bridge sampling
  • Thermodynamic integration
  • Estimation using sequential Monte Carlo (SMC)
  • Nested sampling
  • Bayesian comparison of (non-nested) models
  • Computing the evidence of generic probabilistic programs

Pre-requisites

You must have taken Bayesian Computation and Modelling III (MATH3421) at Stage 3. You should be comfortable with programming in R and/or Python.

Some relevant resources

Books

Papers