Error, Uncertainty and the Statistics of Forensic Science

An error rate is not an admission of incompetence. It is an inherent property of any real-world measurement process, where reagents, instruments, and analysts all introduce variability. The two varieties are false positives and false negatives. A false positive declares a match where none genuinely exists. A false negative misses a real match. Both occur in every forensic discipline, and their balance is a deliberate design choice: tune a method to be more sensitive and false negatives fall but false positives rise, and vice versa.

Error rates are established through proficiency testing and validation studies. A laboratory submits its analysts to blind samples where the correct answer is known, and the fraction of incorrect calls becomes the observed error rate for that method under those conditions. The 2009 NAS report found that many pattern-comparison disciplines (bitemark, toolmark, hair microscopy, handwriting) lacked any rigorous validation and had no published error rate at all. That was not a minor gap: it meant the courtroom testimony lacked the one number that would let a juror know how much to trust it.

Measurement uncertainty applies across every quantitative forensic discipline: the blood-alcohol concentration from a breathalyser, the elemental composition of a glass fragment, the refractive index of a fibre, the concentration of a drug in a tablet. Any numerical result that comes from an instrument carries an attached band of uncertainty, and that band is not optional to report. The ISO/IEC 17025 standard, which governs accredited forensic laboratories, requires uncertainty to be estimated and reported for all quantitative results.

Uncertainty arises from multiple sources. The instrument itself has a precision limit: two readings of the same sample will not be identical. The reference material used for calibration has its own certified uncertainty. The analyst's technique introduces variability that proficiency studies can quantify. Environmental factors (temperature, humidity, reagent batch) add further spread. Expanded uncertainty is the final reported figure, typically at the 95% confidence level, meaning the true value falls within that range 95 times out of 100 under the same conditions.

The practical significance is this: two samples that look different at face value may be fully consistent once uncertainty is included. Conversely, two samples that look identical may differ if the uncertainty interval is small enough. Without uncertainty estimates, the comparison is meaningless. A blood-alcohol reading of 0.082 and a legal limit of 0.080 cannot be distinguished without knowing whether the instrument's uncertainty is ±0.001 or ±0.005. The number alone is not the answer.

The likelihood ratio is the central tool of modern probabilistic forensic reporting, and its structure is worth understanding precisely. Given a forensic observation E, the analyst proposes two competing hypotheses: Hp (prosecution: the suspect is the source) and Hd (defence: an unknown unrelated person is the source). The LR is then P(E | Hp) divided by P(E | Hd). If a DNA profile occurs in 1 person in 1 billion, the denominator is 10⁻⁹ and a clean match gives a numerator close to 1, producing an LR of approximately one billion: the evidence is one billion times more probable if the suspect is the source than if a random unrelated person is.

The LR is bounded by design: it expresses how much more probable the scientific observation is under one proposition than the other, and does not address whether the suspect committed the act. What the jury does with that LR depends on their prior beliefs about the case, all the other evidence, witness credibility, and motive. The analyst is not supposed to cross into that territory. The moment an expert says 'the evidence proves the defendant was there' rather than 'the evidence is one thousand times more probable if the defendant was there', the expert has stepped outside their role.

• LR > 1: evidence supports the prosecution hypothesis over the defence hypothesis.
• LR = 1: evidence is equally probable under both hypotheses. It provides no discriminating power.
• LR < 1: evidence supports the defence hypothesis. This does not happen often in casework but is logically possible and must be reported honestly if it arises.
• Verbal scale: many laboratories translate LR magnitudes into verbal labels such as 'moderate support', 'strong support', or 'very strong support'. The scale (e.g. the UK Association of Forensic Science Providers scale) is calibrated so that jurors can interpret the strength without needing to process large numbers.

The prosecutor's fallacy is a specific logical error in which P(evidence | innocence) is treated as equivalent to P(innocence | evidence). These two quantities can differ by orders of magnitude, and confusing them can be lethal to fair trial outcomes. The original example comes from DNA: an analyst reports a random match probability of 1 in 1 million. A prosecutor, or the analyst themselves, then tells the jury that there is only a one-in-a-million chance the defendant is innocent. This is wrong. The one-in-a-million figure is the probability of seeing this profile in a randomly chosen unrelated person. The probability of innocence depends on how many people were plausible suspects to begin with, the population of the city, the strength of other evidence, and every other fact in the case.

The mirror image is the defence fallacy: arguing that because many people share a characteristic, it proves nothing about any individual. If a fibre type is worn by 10,000 people in the city, the argument goes, the match is worthless. This overstates the case in the other direction. The matching result does raise the probability that the suspect is the source relative to a prior. It narrows the field. The correct response is the LR framework: calculate exactly how much this evidence shifts the odds, not whether it does so at all.

Both fallacies share the same root: pulling a single probability out of context and treating it as a verdict. The LR framework prevents both errors by making both the numerator and denominator explicit, forcing the audience to compare probabilities rather than take one number as the answer.

The base-rate fallacy is the error of focusing on a match probability while ignoring how many people in the relevant population share the matching characteristic. A DNA test with a random match probability of 1 in 10,000 applied to a city of one million people will statistically match roughly 100 individuals by chance. If the investigation initially had no strong suspect and ran the profile against a database of 500,000 people, a 'cold hit' match might identify someone who is one of those 100 incidental matches rather than the actual source. The cold hit is not worthless, but it is the beginning of an investigation, not the end of one.

Base-rate considerations are not unique to DNA. A shoe-print pattern shared by 5% of footwear sold in a country is meaningless unless the analyst knows that the crime scene has additional features narrowing the relevant population. A particular toolmark that could come from any of 2,000 tools of the same model is much weaker than one that could only come from tools showing a specific wear pattern that appears in roughly 50 examples. The denominator is always part of the calculation.

The transition from binary match language to probabilistic reporting is the most significant change in forensic communication since the introduction of DNA evidence. Binary reports said things like: 'the samples are consistent' or 'the samples match'. Probabilistic reports say instead: 'the DNA findings are approximately 1 billion times more probable if the sample originated from the suspect than if it originated from an unrelated individual.' Both describe the same analytical result. The second statement is longer but scientifically honest about what the result actually proves.

1. Define the hypotheses
   Before calculating anything, the analyst states a clear prosecution hypothesis (Hp) and defence hypothesis (Hd). These are tied to the specific case. For a DNA contact trace, Hp might be 'the suspect touched the item' and Hd 'an unknown unrelated person touched the item'.
2. Calculate or estimate the LR
   Using validated population databases for DNA, or experimentally derived transfer/persistence data for trace, or published frequency data for other features, the analyst computes how probable the observation is under each hypothesis. For complex mixed profiles, probabilistic genotyping software such as STRmix or TrueAllele performs this calculation using Monte Carlo sampling.
3. Apply the verbal scale
   The numerical LR is translated into a verbal statement following a published scale so the report is interpretable by non-statisticians. For example, on the AFSP scale, LRs between 100 and 1,000 might be labelled 'moderate support for the prosecution hypothesis'; LRs above 1 million might be labelled 'extremely strong support'.
4. Report and stop at the source level
   The analyst reports the LR and stops there unless specifically asked to address activity level (how the DNA got there) or offence level (what the presence implies about what happened). Each level requires different data and carries different assumptions. Crossing from source to offence level without acknowledging it is a common source of misleading testimony.

Not every discipline has reached the same point in this transition. DNA profiling has mature probabilistic tools. Fingerprint comparison still uses categorical conclusions ('identification', 'exclusion', 'inconclusive') in many jurisdictions, though several major laboratories are piloting LR-based reporting. Toolmark and bitemark analysis lack the population-frequency data needed to assign a denominator with any confidence, which is why their probative value is so contested.