History of Statistical Evidence in Courts

Courts received numerical evidence long before modern forensic science. Actuarial tables were used in civil litigation about life expectancy from the eighteenth century. Fingerprint frequency arguments were made from the early twentieth century, initially without any validated population database. The Dreyfus affair in France (1894 to 1906) involved disputed handwriting comparison evidence, and the statistician Henri Poincare testified that the mathematical arguments presented by the prosecution expert were invalid: an early instance of a scientist formally challenging the statistical claims of another expert in court.

In the United States, the use of blood group typing in paternity and criminal cases from the 1920s onwards introduced the idea of expressing forensic conclusions as frequency statements. By the 1950s, serological evidence was routinely presented with probability figures, usually without validation studies for the specific population involved. The pattern was consistent: a new forensic method produced numbers, practitioners presented those numbers to courts, and neither the courts nor the legal profession had the tools to evaluate whether the numbers were well-founded.

The same dynamic appeared in the United Kingdom in the nineteenth-century trials of Florence Maybrick (1889) and in several arsenic-poisoning cases where chemical analyses were presented with implied precision that the underlying methods could not support. The concept that an expert might be technically accurate about a measurement but misleading about its forensic significance was not well developed in legal doctrine until much later.

People v. Collins, decided by the California Supreme Court in 1968, remains the leading case study in how probability can be misused in court. A robbery was committed by a couple matching certain descriptions: a blonde woman with a ponytail, a bearded black man, and a yellow car. The prosecution called a mathematics instructor who assigned probabilities to each characteristic (for example, one in ten for a woman with a ponytail, one in four for a bearded man) and multiplied them together to produce a figure of 1 in 12 million. The prosecutor then argued that the chance of there being another couple in the area matching all these criteria was essentially zero, so the defendants must be the perpetrators.

The California Supreme Court identified several distinct errors. First, the assumed probabilities had no empirical basis; they were invented. Second, even if the probabilities were correct, the product rule applies only to independent events, and it was not established that these physical characteristics are independent of each other in the population (people with certain characteristics tend to appear together). Third, even accepting the 1 in 12 million figure as correct, the argument that the defendants were therefore guilty commits the prosecutor's fallacy: the probability that a random couple shares these features is not the same as the probability that these particular defendants are innocent. The court reversed the conviction.

Collins is also notable for an additional mathematical point raised in the judgment. Even if the 1 in 12 million figure were correct, probability theory shows that when you have a large population to search across and find one matching couple, the probability that the matching couple is the actual perpetrator is substantially less than certainty; there is a meaningful chance that more than one couple in the relevant area matches the description. The court's analysis of this point was an early judicial engagement with the concept that low random-match probability and high probability of guilt are not the same thing.

Sally Clark was a British solicitor convicted in 1999 of murdering her two infant sons, both of whom had died suddenly. The prosecution case included evidence from Sir Roy Meadow, a paediatrician who cited a statistic from the CESDI (Confidential Enquiry into Stillbirths and Deaths in Infancy) report: the chance of a cot death in a family with Clark's socioeconomic profile was approximately 1 in 8,543. He then squared this figure to produce approximately 1 in 73 million, stating this as the probability of two cot deaths occurring by chance in the same family.

Two separate statistical errors combined to produce the figure. The first was the independence assumption: squaring the single-death probability treats the two events as statistically independent. In fact, cot death risk has a familial component. If one child in a family dies of sudden infant death syndrome, the risk for subsequent children is elevated, not the same as the base population risk. The correct calculation, accounting for the known familial correlation, would have produced a much higher probability than 1 in 73 million. The second error was then introduced when the prosecution used this figure: the 1 in 73 million figure was presented to the jury as, in effect, the probability that Clark was innocent. That is the prosecutor's fallacy. The probability of two cot deaths given innocence is not the same as the probability of innocence given two cot deaths. To convert between them requires knowing the prior probability of guilt and the probability of the alternative (murder of two children), neither of which was supplied.

The Royal Statistical Society issued a public statement in October 2001, before Clark's first appeal, noting that the statistical evidence had not been presented in an appropriate way and that the figure had no statistical basis. Clark's first appeal failed, but her second appeal in 2003 succeeded, partly on statistical grounds and partly because of undisclosed microbiological evidence. She was released after serving more than three years. Clark died in 2007. The case resulted in a formal review of other cases in which Meadow had given evidence and in significant changes to judicial directions on statistical evidence in England and Wales.

The prosecutor's fallacy and the defence fallacy are the two canonical errors in courtroom probability. Both arise from the same underlying confusion: the failure to distinguish between a conditional probability and its converse. In formal terms, P(E|H) is not the same as P(H|E), and treating one as the other is called the transposition of the conditional.

The prosecutor's fallacy in its simplest form: a DNA profile has a random match probability of 1 in 1 million. The prosecutor argues: there is only a 1 in 1 million chance that a random person shares this profile, so there is only a 1 in 1 million chance the defendant is innocent. This is wrong. The 1 in 1 million figure is P(matching profile | person is not the source). The probability that the defendant is not the source, P(not the source | matching profile), depends additionally on: how many other people in the relevant population might have been the source, what other evidence exists, and the prior probability before the DNA evidence was considered. In a city of 10 million people, approximately 10 people share the profile. If the defendant was identified solely by database trawl with no other evidence, the probability they are the source given the match is around 1 in 10, not 1 in 1 million.

The defence fallacy runs in the opposite direction: because 10 people in the city share the profile, the evidence against any one of them is weak. This is also wrong. In a case where the suspect pool has been narrowed by geography, opportunity, or other evidence to a small group, the base rate among that group may be very different from the base rate in the general population. The relevant question is the probability of a match within the realistic suspect pool, not within the entire city.

The likelihood ratio framework is the professional response to both fallacies. A likelihood ratio of 1 million means the evidence is 1 million times more probable if the prosecution hypothesis is true than if the defence hypothesis is true. This statement is precise, avoids both fallacies, and leaves the jury to weigh it alongside all other evidence. The European Network of Forensic Science Institutes (ENFSI) Guideline for Evaluative Reporting (2016) requires this format for evaluative conclusions in member laboratories. Similar guidance has been adopted by the Forensic Science Regulator in England and Wales, and by forensic science bodies in Australia and New Zealand. See Role of Statistics in Evidence Evaluation for the full framework.

People v. Collins and R v. Sally Clark are the most widely cited cases, but the same categories of error appear across jurisdictions. In R v. Deen (England, 1994), the Court of Appeal reversed a conviction where a forensic scientist had transposed the conditional in explaining a DNA match, telling the jury that the probability of the defendant being innocent was 1 in 3 million, a statement that confused P(match | innocent) with P(innocent | match). In R v. Adams (England, 1996 and 1998), the Court of Appeal considered whether Bayesian reasoning should be presented to juries using formal probability calculations, ultimately concluding that juries should not be directed to reason according to Bayes' theorem as a formula, but should receive evidence in terms of likelihood ratios with verbal guidance.

In the United States, the National Research Council's 1996 report (NRC II) on DNA evidence in the courts was a landmark response to the perception that DNA statistics were being misused. The report introduced the ceiling principle and later the theta correction for accounting for population substructure in allele frequency calculations, responding directly to defence challenges in cases including People v. Castro (New York, 1989), where a trial court ruled that the DNA results of the prosecution's laboratory were inadmissible because the laboratory had not followed its own protocols. In Australia, the High Court in Festa v. The Queen (2001) addressed the reliability of coincidence evidence where a complainant identified the accused by an unusual vehicle, and statistical arguments about vehicle frequency were central to the appeal.

Germany provides a contrasting institutional model. German courts use a mixed panel of professional and lay judges rather than a pure jury system, and the role of expert witnesses is more closely supervised by the court. Studies of DNA evidence in German criminal proceedings show fewer instances of the prosecutor's fallacy in reported judgments, though they are not absent. The European Court of Human Rights has addressed statistical evidence in several cases concerning the right to a fair trial under Article 6 of the European Convention on Human Rights, holding that inadequate explanation of statistical evidence to a jury can in principle amount to a breach of the right to a fair hearing.

In India, the Bharatiya Sakshya Adhiniyam 2023 governs the admissibility of expert opinion under sections that track the former provisions of the Indian Evidence Act 1872. DNA evidence is now explicitly addressed in the Code, but statistical framework requirements for how probability conclusions should be presented remain underdeveloped in Indian case law compared to the detailed judicial guidance now available in England and Wales or the ENFSI guidelines used across Europe. The Digital Personal Data Protection Act 2023 introduces additional considerations for how population databases used to generate forensic statistics may be assembled and accessed.

The recurring cases produce a clear set of duties for any expert who uses statistical arguments in court. These duties have been codified in professional guidelines, judicial directions, and law commission reports across multiple jurisdictions, and they converge on the same points.

• State assumptions explicitly. Every probability figure rests on assumptions about independence, reference population, and method validity. These must be stated in the report, not left implicit.
• Report in likelihood ratio format. The ENFSI (2016) and Forensic Science Regulator (2020) guidelines require evaluative conclusions to be expressed as likelihood ratios with a verbal scale (for example, LR > 10,000 described as very strong support). This format avoids both the prosecutor's and defence fallacies.
• Stay within your expertise. An expert in paediatric medicine (as in the Clark case) who presents a statistical calculation should have that calculation reviewed by a statistician before trial. The expert should not present statistical conclusions beyond their domain competence.
• Do not conflate conditional probabilities. The expert must not state or imply that P(E|H) equals P(H|E). If asked to comment on the probability of guilt, the expert should explain that this is not within their remit: their role is to report the likelihood ratio, not the posterior probability.
• Disclose the reference database. Any frequency or probability figure must be tied to a named, validated reference database. See Population Databases for Forensic Statistics for what constitutes an adequate database.
• Account for correlated features. The product rule cannot be applied to features that are correlated. Independence must be verified empirically or an adjusted calculation must be used.