Preprint
Machine Learning

Bayes Factors

Robert E. Kass(Carnegie Mellon University), Adrian E. Raftery(University of Washington)
June 1, 1995Journal of the American Statistical Association2,346 citations
DOI

2.3k

Citations

0

Influential Citations

Journal of the American Statistical Association

Venue

1995

Year

Abstract

Abstract In a 1935 paper and in his book Theory of Probability, Jeffreys developed a methodology for quantifying the evidence in favor of a scientific theory. The centerpiece was a number, now called the Bayes factor, which is the posterior odds of the null hypothesis when the prior probability on the null is one-half. Although there has been much discussion of Bayesian hypothesis testing in the context of criticism of P-values, less attention has been given to the Bayes factor as a practical tool of applied statistics. In this article we review and discuss the uses of Bayes factors in the context of five scientific applications in genetics, sports, ecology, sociology, and psychology. We emphasize the following points: •From Jeffreys' Bayesian viewpoint, the purpose of hypothesis testing is to evaluate the evidence in favor of a scientific theory.•Bayes factors offer a way of evaluating evidence in favor of a null hypothesis.•Bayes factors provide a way of incorporating external information into the evaluation of evidence about a hypothesis.•Bayes factors are very general and do not require alternative models to be nested.•Several techniques are available for computing Bayes factors, including asymptotic approximations that are easy to compute using the output from standard packages that maximize likelihoods.•In “nonstandard” statistical models that do not satisfy common regularity conditions, it can be technically simpler to calculate Bayes factors than to derive non-Bayesian significance tests.•The Schwarz criterion (or BIC) gives a rough approximation to the logarithm of the Bayes factor, which is easy to use and does not require evaluation of prior distributions.•When one is interested in estimation or prediction, Bayes factors may be converted to weights to be attached to various models so that a composite estimate or prediction may be obtained that takes account of structural or model uncertainty.•Algorithms have been proposed that allow model uncertainty to be taken into account when the class of models initially considered is very large.•Bayes factors are useful for guiding an evolutionary model-building process.•It is important, and feasible, to assess the sensitivity of conclusions to the prior distributions used.

Analysis

Why This Paper Matters

This seminal paper by Kass and Raftery brought Bayes factors from theoretical Bayesian statistics into the mainstream of applied data analysis. At a time when null hypothesis significance testing with P-values dominated scientific practice, the authors provided a coherent alternative that directly quantifies evidence for a hypothesis. The paper's emphasis on practical computation—including the connection to the Schwarz criterion (BIC)—made Bayesian model comparison accessible to practitioners without requiring full prior specification.

The five diverse applications (genetics, sports, ecology, sociology, psychology) demonstrated the method's versatility and helped bridge the gap between Bayesian theory and real-world data analysis. This work has been cited over 2,300 times, reflecting its foundational role in Bayesian statistics and machine learning.

Technical Contributions

  • Bayes factor definition: The ratio of marginal likelihoods under two hypotheses, interpreted as the posterior odds when prior odds are 1.
  • Asymptotic approximation: The Laplace method yields a simple approximation that includes the BIC as a special case, enabling easy computation from standard likelihood outputs.
  • Model averaging: Bayes factors can be converted to model weights for composite estimation, addressing model uncertainty.
  • Non-nested models: Unlike likelihood ratio tests, Bayes factors do not require models to be nested.
  • Large model spaces: Algorithms are discussed for searching over many candidate models.

Results

The paper does not present new experimental results but synthesizes existing applications. Key quantitative points include: the BIC approximation to the log Bayes factor is accurate to O(1) and requires no prior input; in nonstandard models (e.g., with irregular likelihoods), Bayes factors can be easier to compute than frequentist tests. The case studies illustrate how Bayes factors can decisively favor one hypothesis over another (e.g., in genetic linkage analysis).

Significance

This paper fundamentally changed how statisticians and machine learning researchers approach model comparison. It provided a rigorous yet practical framework that influenced subsequent developments in Bayesian nonparametrics, variational Bayes, and automatic model selection. The emphasis on model averaging anticipated modern ensemble methods. Today, Bayes factors remain a gold standard for evidence quantification in scientific inference, and this paper is the definitive reference for their practical application.

bayes-factorshypothesis-testingbayesian-statisticsmodel-selectionbicevidence-quantificationapplied-statistics
Attention Mechanisms

Related Papers