False Positive and False Negative Error Rates

Every binary classification decision produces one of four outcomes. The two-by-two table organises these outcomes by placing the true state of the world (match or non-match) against the decision the method produces (positive or negative). Each cell of the table has a name and a formula.

From this table, four rates follow directly. Sensitivity = TP / (TP + FN): of all true matches, what fraction did the method catch? Specificity = TN / (TN + FP): of all true non-matches, what fraction did the method correctly reject? FPR = FP / (FP + TN) = 1 minus specificity. FNR = FN / (FN + TP) = 1 minus sensitivity. Sensitivity and specificity are properties of the method at a given threshold. They do not change when the base rate of true matches in the population changes.

The table also produces two rates that do depend on base rate. Positive predictive value (PPV) = TP / (TP + FP): of all positive declarations, what fraction are correct? Negative predictive value (NPV) = TN / (TN + FN): of all negative declarations, what fraction are correct? PPV and NPV are the rates that matter most for interpreting a specific case result, but they require knowledge of the prior probability that a match exists, not just the method's intrinsic performance.

Most forensic classification methods produce a continuous similarity or likelihood score rather than a binary output. A fingerprint comparison algorithm produces a similarity score; a handwriting comparison produces a degree-of-agreement score; a speaker recognition system produces a log-likelihood ratio. The classification decision, match or non-match, is made by comparing the score to a fixed threshold. The ROC curve shows what happens to sensitivity and FPR as that threshold is moved.

To construct a ROC curve, a validation dataset of known-source pairs is needed. For each possible threshold value, calculate the sensitivity and FPR the method achieves on that dataset. Plot sensitivity on the y-axis and FPR on the x-axis. Moving the threshold from most stringent (few positives, low FPR, low sensitivity) to most lenient (many positives, high FPR, high sensitivity) traces out the curve from bottom-left to top-right. A method with no discriminating power produces a diagonal line from (0,0) to (1,1). A method with perfect discrimination produces a curve that reaches (0,1), with zero FPR and perfect sensitivity.

The area under the ROC curve (AUC) is a single-number summary of discriminating power. An AUC of 0.5 means the method performs at chance; an AUC of 1.0 means perfect discrimination. Published AUC values for automated fingerprint identification systems in large-scale studies typically exceed 0.99. For some pattern-comparison disciplines, published AUC values are much lower, which directly constrains how strong a forensic opinion can justifiably be. Practitioners should know the AUC from published validation studies for any method they rely on.

Choosing the operating threshold requires a judgment about the relative costs of false positives and false negatives. In a mass casualty identification scenario, missing a true identification (false negative) may be the greater cost, favouring a lower threshold. In a criminal prosecution, a false positive, condemning an innocent person, may be weighted as the greater cost, favouring a higher threshold. The optimal threshold on a ROC curve when FP and FN are treated as equally costly is the point closest to the top-left corner of the plot; that point can shift substantially when the costs are asymmetric. This judgment must be documented in any validation or operational protocol.

Error rates are not inherent constants of a classification method. They are estimates derived from studies applying the method to a reference set of samples with known ground truth. The validity of the estimate depends on how the study was designed. Three criteria determine whether a validation study's error rate estimates can be trusted.

First, sample size must be large enough to produce estimates with useful precision. An FPR estimate of 1 in 100 derived from 100 non-match pairs has a very wide confidence interval: the true rate could plausibly be anywhere from about 0.02% to 5.4% at 95% confidence. An estimate derived from 10,000 pairs has a much narrower interval. Courts and practitioners should routinely ask for the confidence interval around any reported error rate, not just the point estimate.

Second, the sample must be representative of the conditions under which the method will be used. A fingerprint algorithm validated on high-quality laboratory prints may perform very differently on latent prints recovered from rough surfaces at a crime scene. A voice comparison system validated on studio recordings may fail on telephone-quality audio. The match between validation conditions and casework conditions is one of the most frequent criticisms of forensic validation studies.

Third, the study must be blind: the analyst or algorithm must not know the ground truth when making the classification decision. Studies in which analysts knew which pairs were true matches consistently show inflated accuracy compared to blind studies. The 2009 National Research Council report on forensic science in the United States, and the 2016 PCAST report, both identified the absence of adequately blinded, representative validation studies as a systemic weakness across multiple forensic disciplines.

A forensic method's FPR and FNR describe its performance across a validation population. They do not directly tell a court how to interpret a single result. That interpretation requires knowing the prior probability that the hypothesis being tested is true, which is usually called the base rate or prior probability.

Consider a screening programme that tests 10,000 individuals for contact with a controlled substance, using a method with a 1% FPR and 99% sensitivity. In a population where 1% of individuals actually had contact (100 true positives), the method will produce approximately 99 true positives and 99 false positives. Among the 198 positive results, roughly half are incorrect. The PPV is about 50%, despite the method's individually impressive FPR and sensitivity. This is not a flaw in the method; it is a mathematical consequence of combining a low-frequency true condition with a non-zero FPR.

The prosecutor's fallacy is a specific misstatement of this relationship in court. It occurs when a prosecutor or expert presents the false positive rate as if it were the probability that the defendant is innocent. Saying that a match has a one-in-a-million probability of occurring by chance is not the same as saying the defendant has a one-in-a-million chance of being innocent. The correct statement requires incorporating the prior probability of the defendant's guilt, which the court, not the forensic expert, must assess. Conflating these two quantities has contributed to wrongful convictions in multiple documented cases in the United Kingdom and the United States.

The defence fallacy is the mirror error: treating the base rate alone as the probability of guilt, ignoring the strength of the match evidence. Both fallacies arise from the same mistake of failing to apply Bayes' theorem correctly. The likelihood ratio framework, discussed in Role of Statistics in Evidence Evaluation, is the standard tool for combining prior probabilities with match evidence without committing either fallacy.

Admissibility standards in multiple jurisdictions treat known error rate as a formal criterion. In the United States, the Daubert test (Daubert v. Merrell Dow Pharmaceuticals, 1993) asks four questions about scientific evidence: Has it been tested? Has it been subject to peer review? What is the known or potential error rate? Is the underlying methodology generally accepted? The error rate criterion is the one most often inadequately addressed in expert testimony, particularly for pattern-comparison disciplines.

In England and Wales, the Criminal Procedure Rules (CrimPR Part 19) require expert witnesses to state the facts, matters, and assumptions on which their opinion is based, including the limits of their expertise and the reliability of their methodology. The Forensic Science Regulator's Codes of Practice set mandatory accuracy requirements for accredited providers. In India, Section 45 of the Bharatiya Sakshya Adhiniyam 2023 governs the admissibility of expert opinion; courts have increasingly required that experts explain the empirical basis and limitations of their methods, in line with global convergence on disclosure standards.

Best practice for disclosing error rates in an expert report includes: stating the source of the error rate estimate (named study, publication, or laboratory proficiency programme); stating the FPR and FNR separately, with confidence intervals; explaining the conditions of the validation study and whether they match the current case; and stating the operating threshold at which the method was applied. Presenting a single number without confidence intervals, or citing an in-house study without peer review, are both deficiencies that courts in the UK and US have found sufficient to exclude or limit expert testimony.

A forensic opinion is not simply a binary declaration of match or non-match. It carries an implied or explicit claim about how strongly the evidence supports one hypothesis over the other. Error rates constrain the maximum strength that any opinion can justifiably claim, regardless of how impressive the match appears to the examiner.

The likelihood ratio (LR) framework makes this constraint explicit. The LR is the probability of observing the evidence given the prosecution hypothesis divided by the probability of observing it given the defence hypothesis. The false positive rate sets a floor on the denominator of the LR: even if the evidence would be impossible under the defence hypothesis in theory, the empirically observed FPR tells you that the method produces that evidence by error at some measurable rate. A method with an FPR of 1% cannot justifiably support an LR greater than about 100 on that basis alone, regardless of the examiner's subjective confidence. The LR is bounded by the method's measured performance, not by how compelling the match looks.

This constraint has practical consequences. Some traditional forensic disciplines, notably firearms toolmark comparison, latent fingerprint comparison, and bite mark analysis, have been claimed by practitioners to support conclusions of absolute certainty or individualisation. The 2009 NRC report and the 2016 PCAST report both concluded that such claims are not justified by the available empirical data, because the FPR and FNR for these methods have not been estimated from adequately designed validation studies at a scale that would support claims of near-zero error. Reform efforts in the UK, US, and Australia have moved expert reports in these disciplines toward probabilistic language that acknowledges the method's measured or estimated error rate.

For practitioners, the operational implication is that the error rate from the most applicable published validation study should be incorporated into the expert report. If no published study matches the casework conditions, the expert must acknowledge this gap. Silence about error rate is not a neutral position: it implies to a court that the error rate is negligible when the evidence base may not support that implication. See also Numbers in Forensic Conclusions for how probabilistic conclusions are structured and communicated.