Other Statistical Fallacies in Forensic Contexts

Base rate neglect is the failure to incorporate prior probability when updating on new evidence. It is a special case of the broader failure to apply Bayes' theorem correctly. The match statistic for a piece of forensic evidence tells you how likely the evidence is if the suspect is the source. It does not tell you how likely the suspect is the source. To get from one to the other, you need the prior probability that the suspect is the source, which is determined by the investigation as a whole, not by the forensic match alone.

The formal structure is straightforward. Let H1 be the prosecution hypothesis (the suspect is the source) and H2 be the defence hypothesis (someone else is the source). The likelihood ratio (LR) expresses how much more probable the evidence is under H1 than under H2. Bayes' theorem gives: Posterior odds = LR x Prior odds. If the prior odds are very low, even a large LR may produce a posterior probability of guilt that is well below certainty. A DNA match with an LR of one million means that the evidence is one million times more probable if the suspect is the source than if a random unrelated person is the source. But if the suspect was identified by a database trawl through ten million profiles, the prior odds before the DNA evidence were roughly one in ten million, and the posterior odds after the evidence are approximately one in ten. The probability of guilt is around 90%, not near-certainty.

Base rate neglect has been documented in published cases. In the United Kingdom, the Court of Appeal in R v Adams (1996) grappled with the challenge of explaining Bayesian reasoning to a jury. In the United States, the NAS 2009 report on forensic science identified incorrect probabilistic reasoning as a pervasive problem in expert testimony. The Indian Supreme Court and the Delhi High Court have raised concerns about the uncritical acceptance of DNA match probabilities without Bayesian context. The correct response for an expert is to present both the LR and an explicit statement of what prior probability assumptions are required to convert it to a posterior probability, and to leave the specification of the prior to the court or to counsel drawing on all the evidence.

The ecological fallacy is an error in reasoning about reference populations. A forensic database records feature frequencies in a defined sample. The frequency of a DNA profile in that sample is an estimate of how common the profile is in the population from which the sample was drawn. If the individual being evaluated is from a different population, with a different allele frequency distribution, applying the database frequency to that individual is a category error.

DNA random match probabilities are calculated using allele frequencies from reference databases stratified by population group. In the United States, the FBI maintains separate frequency tables for Caucasian, African American, Hispanic, and other groups. In the United Kingdom, the Forensic Science Service developed databases for major ethnic groups present in the population. In India, forensic DNA work has been complicated by the country's genetic diversity: allele frequencies vary significantly across linguistic and caste-defined subpopulations, and a frequency derived from a pan-Indian sample may not apply to an individual from a specific regional or tribal community. The same issue arises in Australia with Aboriginal and Torres Strait Islander populations, and in many African countries where reference databases were built from urban samples that do not represent rural genetic diversity.

The standard forensic correction is to use the most conservative (highest-frequency) database when population membership is uncertain, or to present frequency estimates from multiple reference populations and note the range. The ENFSI DNA Working Group and the FBI Scientific Working Group on DNA Analysis Methods (SWGDAM) both recommend that reports state explicitly which reference population was used and acknowledge population uncertainty where it exists. An expert who states only a single frequency without specifying the reference population has omitted information the court needs to evaluate the match.

A random match probability (RMP) is calculated for a single pairwise comparison between a crime-scene profile and a suspect profile. It answers the question: if I pick one random unrelated person from the relevant population, what is the probability that their profile matches the crime-scene profile at all the tested loci? If the RMP is 1 in 10 million, and the comparison is made between the crime-scene profile and a single named suspect who was identified by independent evidence, that probability is the appropriate statistic.

The problem arises when the crime-scene profile is searched against a database of N profiles. The probability of at least one coincidental match in the database is approximately 1 minus (1 minus RMP) raised to the power N. For large N and small RMP, this simplifies to approximately N times RMP. If the database contains 5 million profiles and the RMP is 1 in 10 million, the probability of at least one adventitious match in the database is approximately 5 in 10 million, or 1 in 2 million. This is the database match probability (DMP), and it is the correct statistic to cite when reporting a cold hit. Citing the single-pair RMP instead overstates the strength of the evidence by a factor of N.

The multiple comparisons problem also appears in non-DNA forensic contexts. Toolmark examination databases, shoeprint databases, and questioned-document feature libraries all involve searches across many stored records. The probability of a coincidental feature match increases with the size of the comparison set in all of these contexts, even when the underlying comparison method is not probabilistic DNA profiling. Analysts should state the size of the comparison set and apply an appropriate correction, or at minimum acknowledge that the single-comparison probability understates the probability of a coincidental match in the database.

The look-elsewhere effect is a form of post-hoc selection bias. An analyst examines a piece of evidence and observes many features. Some features are inconsistent with the suspect sample, and some are consistent. If the analyst reports only the consistent features and presents them as significant matches, without disclosing the inconsistencies or the total number of features examined, the reported significance is inflated. The probability of observing at least one consistent feature by chance increases with the number of features examined, exactly as the probability of at least one database hit increases with the number of profiles searched.

This effect has been identified as a concern in several forensic disciplines. In bite-mark analysis, critics have noted that analysts sometimes select which marks to measure and compare after viewing both the evidence and the suspect, creating the possibility of unconscious cherry-picking of matching features. In fiber and hair analysis, the number of characteristics examined and the proportion that matched have not always been systematically reported. In handwriting examination, the look-elsewhere effect can arise when an examiner selects the letter forms or line quality features that best support a conclusion after reviewing the full document. The 2016 report of the US President's Council of Advisors on Science and Technology (PCAST) on forensic science identified this selective reporting concern as a validity challenge for several pattern-comparison disciplines.

The correct remedy is pre-registration of the analysis plan: the analyst documents which features will be measured and how significance will be assessed before examining the evidence, or at least before comparing the questioned and known samples. In practice, full pre-registration is not always possible in forensic work because the choice of relevant features depends on what is present in the evidence. The minimum acceptable standard is to report all features examined, not only those that matched, and to state explicitly whether the reported findings are the result of an exhaustive examination or a selected subset.

In practice, these errors rarely appear in isolation. A database cold-hit case can simultaneously involve base rate neglect (the low prior probability from the database trawl is ignored), the ecological fallacy (the reference population frequency used to calculate the RMP does not match the suspect's ancestry), and the multiple comparisons problem (the single-pair RMP is reported rather than the database match probability). The compound effect of all three errors can make evidence that provides moderate support for the prosecution hypothesis appear to provide near-conclusive proof.

The look-elsewhere effect interacts with base rate neglect when an analyst searches many features, finds one significant match, and then presents that match without the prior probability context needed to interpret it. If the probability of finding a matching feature by chance across fifty examined characteristics is 1 in 10, and the prior probability that the suspect is the source is also 1 in 10, the posterior probability of guilt given the match is not high. But if the jury hears only that a rare feature matched, without the denominator of the search and without the prior, the error compounds.

Courts have begun to recognise compound probabilistic errors. The UK Court of Appeal in R v T (2010) restricted the use of likelihood ratios in footwear mark evidence on the basis that the underlying database was inadequate and the reference population assumptions were unverified, which combines the ecological fallacy concern with a challenge to the RMP calculation. In Australia, the High Court in Pfennig v R and subsequent cases has required that expert statistical evidence disclose the assumptions underlying frequency estimates. The Indian Evidence Act's successors under the Bharatiya Sakshya Adhiniyam 2023 (sections 45 to 51) continue to vest wide discretion in the court to evaluate expert opinion, which in practice means that judges must be equipped to identify these fallacies without always having the statistical training to detect them independently.

Addressing these fallacies in a forensic report requires explicit disclosure of the information that allows the reader to detect and correct each error independently. This is a disclosure obligation, not a request to argue about the weight of the evidence. The expert's role is to supply the data; the weight is for the court. The minimum disclosures needed to address the four fallacies covered in this topic are set out below.

• Base rate neglect: State the likelihood ratio (or its equivalent) separately from any conclusion about the probability of the proposition. Do not conflate the probability of the evidence given the hypothesis with the probability of the hypothesis given the evidence. Where the suspect was identified by a database search, state that fact explicitly.
• Ecological fallacy: Identify the reference population or database used. State whether the database is an appropriate match for the individual's ancestry, region, or subpopulation. If uncertain, present the range of frequencies across available databases.
• Multiple comparisons: If the match was identified by a database search, state the database size and report the database match probability (approximately N x RMP) rather than the single-pair random match probability. If the single-pair RMP is cited, note that it applies only to the confirmatory comparison, not to the database search process.
• Look-elsewhere effect: Report the total number of features, characteristics, or tests examined, not only those that yielded a match. State explicitly whether the reported comparison features were selected before or after examining the suspect sample.

Regulatory frameworks in multiple jurisdictions now formalise these requirements. The UK Forensic Science Regulator's Codes of Practice (2023 edition) require that evaluative reports state the propositions evaluated, the reference population, and the basis for the likelihood ratio. The ENFSI Guideline for Evaluative Reporting in Forensic Science (2015) sets the same requirements across European member states. The American Academy of Forensic Sciences Standards Board has published standards for DNA reporting that specify disclosure of database search procedures. Under the Bharatiya Sakshya Adhiniyam 2023 in India, expert reports admitted under Section 45 can be challenged through cross-examination, and courts have accepted probabilistic challenges where the expert failed to disclose reference population assumptions. In US federal courts, Rule 702 of the Federal Rules of Evidence, as updated following the 2009 NAS report, requires that expert testimony be based on sufficient facts and reliable methods, which courts have interpreted to include disclosure of the statistical basis for probabilistic claims.