Measuring calibration of likelihood-ratio systems: A comparison of four metrics, including a new metric devPAV

Highlights

• Four metrics that measure calibration of sets of likelihood ratios are compared.

• Three are taken from the existing literature.

• A new metric devPAV is introduced.

• Comparison is based on simulating Gaussian LLR-distributions.

• DevPAV performs equally well or better than existing methods.

Abstract

Numerical likelihood-ratio (LR) systems aim to calculate evidential strength for forensic evidence evaluation. Calibration of such LR-systems is essential: one does not want to over- or understate the strength of the evidence. Metrics that measure calibration differ in sensitivity to errors in calibration of such systems. In this paper we compare four calibration metrics by a simulation study based on Gaussian Log LR-distributions. Three calibration metrics are taken from the literature (Good, 1985; Royall, 1997; Ramos and Gonzalez-Rodriguez, 2013) [1–3], and a fourth metric is proposed by us. We evaluated these metrics by two performance criteria: differentiation (between well- and ill-calibrated LR-systems) and stability (of the value of the metric for a variety of well-calibrated LR-systems). Two metrics from the literature (the expected values of LR and of 1/LR, and the rate of misleading evidence stronger than 2) do not behave as desired in many simulated conditions. The third one ($C_{\mathit{llr}}^{\mathit{cal}})$ performs better, but our newly proposed method (which we coin devPAV) is shown to behave equally well to clearly better under almost all simulated conditions. On the basis of this work, we recommend to use both devPAV and $C_{\mathit{llr}}^{\mathit{cal}}$ to measure calibration of LR-systems, where the current results indicate that devPAV is the preferred metric. In the future external validity of this comparison study can be extended by simulating non-Gaussian LR-distributions.

Introduction

An increasing majority of forensic scientists is convinced that forensic evidence should be interpreted by a likelihood ratio (LR). An LR is a statistic that discriminates between two propositions, and that can be used in Bayes rule to update prior odds for these propositions to posterior odds. In order to advance forensic science, in the past few decades automated systems that calculate likelihood ratios have been proposed for several evidence modalities, e.g. fingerprints [4], [5], glass [6], [7], [8], XTC [9], gasoline [10], [11], ignitable liquid class in fire debris [12], [13], DNA (see for example [14]), etc.

With the arrival of these automated LR-systems questions about the objective testing of their validity have arisen, and several publications have addressed this issue [3], [14], [15], [16], [17], [18], [19]. A key issue in the testing of validity is the measurement of calibration of the LR-system. It is important that an LR-system is well-calibrated [20]. If not, the value of the LR cannot be trusted and updating by Bayes rule results in misleadingly large or small posterior odds.

An intuitive explanation of what well-calibrated means, is given by [3], [21]. Imagine a weather forecaster who keeps track of his predictions on the probability of rain tomorrow for a sequence of days. Preferably, when examining those days for which his predicted probability of rain was 0.9 say, he would like it to have rained on 90% of these days. When his predicted probability is (nearly) equal to this relative frequency, the weather forecaster is said to be well-calibrated. His probability statements make ‘empirical sense’. A deviation between empirical frequency and prediction may be caused by statistical models or parameter estimates that do not reflect reality well. This is – of course – undesirable. A probability statement of 0.9 that does not mean that it will rain on 90% of such days is very misleading!

In analogy to a dataset of probability statements, a dataset of LR-values may also make ‘empirical sense’ (or not). Consider the general likelihood ratio formula:

$$\mathit{LR}\left(F\right)\mathrm{:}=\frac{P\left(F\mathrm{|}{H}_p\right)}{P\left(F\mathrm{|}{H}_d\right)}=V,$$

Where $F$ denotes evidence¹ and $H_p$ and $H_d$ the prosecution and defense proposition respectively, and $V$ denotes a particular value of the likelihood ratio. Just as the forecaster is well-calibrated if predicted probabilities $p_o$ match empirical frequencies: P(rain|$p_o$ = V) = V, an LR-system ${\mathit{LR}}_o$ is well calibrated if ‘the LR of the LR is the LR’:

$$\mathit{LR}\left({\mathit{LR}}_o=V\right)\mathrm{:}=\frac{P\left({\mathit{LR}}_o=V\mathrm{|}{H}_p\right)}{P\left({\mathit{LR}}_o=V\mathrm{|}{H}_d\right)}=V$$

for any scalar $V$ [1], [22]. As the likelihood ratio is a comprehensive representation of the evidential value, re-calculating the evidential value regarding as evidence this scalar should give the same evidential value. This also gives a handle on how to measure calibration of an LR-system. If an empirical deviation of Eq. (2) is found for a dataset of ${\mathit{LR}}_o$-values, this indicates the ${\mathit{LR}}_o$-system is ill-calibrated. For instance, say we have an ${\mathit{LR}}_o$-system that we apply to a dataset of 100 situations in which $H_d$ is true, and also to 100 situations in which $H_p$ is true. And assume the results are as follows: when $H_d$ is true 90 of the 100 ${\mathit{LR}}_o$-values have a value of 10, erroneously indicating support for $H_p$; when $H_p$ is true, 50 of the 100 ${\mathit{LR}}_o$-values have a value of 10. We then have an ill-calibrated LR-system. For such a system, a deviation from Eq. (2). is found: the LR according to Eq. (2) associated with an LR₀ = 10 would be 0.5/0.9 = 0.56, which is quite different from 10 and more reflective of the fact that this value is observed in 90% of instances when $H_d$ is true.

An LR-system can be ill-calibrated in several ways. In this research, four general types of ill-calibrated LR-sets were investigated by creating them by simulation. The first set has too large LRs. In the second one, all LRs are too small. In the third situation, the LRs are too extreme (i.e. too large for LRs greater than 1, and too small for LRs less than 1) and in the fourth situation too weak (for details see Section 2.2).

From the literature [7], [9], [10] it is known that often a small number of LRs in the dataset are ill-calibrated: they constitute misleading evidence with excessive values. Therefore, in a second simulation experiment, this situation was imitated by adding a misleading LR with excessive value (either under $H_p$ or under $H_d$) to data simulated from a perfectly calibrated LR-system. Together these simulation experiments cover the typical ways in which LR-data can be ill-calibrated.

One of the examined calibration metrics in this study (see Section 2.3) is defined in terms of misleading evidence. This refers to LRs pointing into the wrong direction, i.e. an LR greater than 1 when in fact the defense proposition is true, or an LR less than 1 when in fact the prosecution proposition is true. A relatively large fraction of misleading LRs in a set may be indicative of ill-calibration.

Before embarking on the study, it is important to define criteria that a good metric has to oblige. For this study, we defined two such criteria: differentiation and stability.

Obviously, a metric should differentiate between well- and ill-calibrated LR-systems. But, when one wants to be able to interpret the outcome of a metric it also has to be stable. Stability of results is important so that a typical range of values can be defined for well-calibrated LR-systems. Differentiation is measured by comparing metric outputs for well- and ill-calibrated LRs for the same simulation conditions. Stability is measured by comparing metric outputs for well-calibrated systems under different simulation conditions.

There may be ambiguity in the use of the terms discrimination and differentiation. We use the terms as follows. Discrimination is used in the context of the LR-system itself, in the way an LR-system discriminates between $H_p$ and $H_d$. One summary of the discrimination power of an LR-system is the equal error rate. A second type of discrimination is the differentiation of a calibration metric between well- and ill-calibrated LR-systems. To avoid confusion, in the context of performance of a calibration metric we will not speak of discrimination but of ‘differentiation’ or to ‘differentiate’ between well- and ill-calibrated LR-systems.

The interest of this study is how to best measure calibration of LR-systems. Although several studies have used several metrics to measure calibration, as far as the authors are aware their performance has not yet been compared. In this study, four metrics to measure calibration are compared, three from existing literature [1], [2], [3] and one newly developed method (coined devPAV). We compare them by simulating well- and ill-calibrated LR-datasets and studying how well the metrics could differentiate between these datasets, and how stable the outcome is for well-calibrated LR-sets under different conditions. Finally, we discuss limitations of the comparison study and conclude by recommending some metrics for the measurement of calibration and a further comparison study.