MCMC diagnostics for econometricians

Markov chain Monte Carlo (MCMC) diagnostics assess the numerical approximation to the specified posterior. They do not establish model adequacy, causal identification or decision suitability.

R-hat and chain agreement

Rank-normalised split R-hat compares variation within and between chains. Values near one support chain agreement on the quantities checked. A small value does not prove that chains explored every relevant posterior region.

The runner’s default diagnostic profile warns at R-hat >= 1.01 and fails at R-hat >= 1.05. These are evidence-screening thresholds. Examine parameters, trace plots and model geometry when they trigger. The effective thresholds are retained in diagnostic_gates.resolved.yaml and can be overridden through runner configuration.

Effective sample size

Effective sample size (ESS) measures Monte Carlo information relative to independent posterior draws. It is separate from the number of observations and regression degrees of freedom. Bulk ESS concerns the central posterior; tail ESS concerns tail summaries.

The runner warns when minimum bulk ESS is at most 400 and fails when it is at most 200. The direct diagnostics facade has its own summary thresholds; use its arguments and report to determine the check being applied. No fixed ESS threshold guarantees adequate precision for every posterior quantity.

Inspect Monte Carlo standard errors for the estimands and intervals that matter. Increase draws or improve parameterisation when precision is insufficient. Thinning existing draws discards information and does not repair poor exploration. It can be a storage choice, but is not a general low-ESS remedy.

Divergences

Retained NUTS divergences warn that numerical integration is unreliable in parts of the posterior geometry. Investigate any retained divergence. The runner’s default profile fails when the retained divergence count exceeds zero.

Warmup divergences are separate from the retained-draw diagnostic. Inspect them in the context of adaptation. Persistent post-warmup divergences can require a higher target acceptance rate, reparameterisation or changes to the model. Additional draws alone do not resolve the underlying geometry.

Credible intervals and posterior probabilities

A 94% highest-density interval (HDI) summarises 94% of the posterior mass under the fitted model and prior. Excluding zero does not generally imply a frequentist test with a 6% false-positive rate. A 90% HDI similarly does not guarantee a 10% significance test.

For parameters that permit both signs, an interval entirely above zero supports a positive conditional estimate at the stated interval threshold. If it includes zero, the estimate is inconclusive as to sign. If the interval rules out effects of practical importance, report that narrower conclusion with its specified threshold.

For positive-support media priors, P(beta > 0 | data) is one by construction. It does not establish a positive advertising effect. A posterior probability that a model-implied ROI exceeds a practical threshold can support a conditional assessment, but inherits the response model, prior and identification assumptions. Monte Carlo error also applies to an estimated probability.

Reading a fitted model

  1. Inspect chain agreement, retained divergences, bulk/tail ESS and Monte Carlo error for the quantities you intend to report.
  2. Investigate failed checks before interpreting their posterior summaries.
  3. Assess prior predictive, posterior predictive and holdout evidence.
  4. Review attribution sensitivity and the causal design separately.

Passing computational checks supports using the numerical approximation for further analysis. It does not validate the substantive model.