Comparing Frequentist and Bayesian Approaches for Forecasting Binary Inference Performance

Abstract

In this paper, we compare forecasts of the quality of inferences made by an inference enterprise generated from a frequentist perspective and a Bayesian perspective. An inference enterprise (IE) is an organizational entity that uses data, tools, people, and processes to make mission-focused inferences. When evaluating changes to an IE, the quality of the inferences that a new, hypothetical IE makes is uncertain. We can model quality or performance metric—such as recall, precision, and false-positive rate—uncertainty as probability distributions generated either through a frequentist approach or a Bayesian approach. In the frequentist approach, we run several experiments evaluating inference quality and fit a distribution to the results. In the Bayesian approach, we update prior performance beliefs with empirical results. We compare the two approaches in 18 forecast questions and score the two sets of forecasts against ground truth answers. Both approaches forecast similar performance means, but the frequentist approach systematically produces wider confidence intervals. Therefore, the frequentist approach outscores the Bayesian approach in metrics sensitive to interval width.