Cognitive Bias and the 2009 NAS Report

Every forensic decision involves some degree of judgement. Pattern comparisons in fingerprints, tool marks, and handwriting require the analyst to decide how much similarity is enough. Even DNA interpretation at low template levels, where the profile is degraded or mixed, involves analyst choices about peak heights and stochastic effects. These judgement points are where contextual information does its damage.

The cognitive mechanism is well established in general psychology and was documented specifically in forensic contexts by Dror from around 2005 onward. Information that is emotionally or narratively salient, knowing the suspect is accused of a violent crime, or that a previous examiner said it was a match, anchors the analyst's perception. When the physical evidence is ambiguous (as real casework evidence frequently is), that anchor tips the decision.

Contextual bias does not only operate at the comparison stage. It can enter at collection (an officer who knows a suspect is present may search more thoroughly around their territory), at interpretation (a toxicologist who knows the victim was young and healthy may have a prior on accidental overdose), and at reporting (an analyst may choose which findings to emphasise based on the investigative narrative briefed to them).

Between 2005 and 2012, Itiel Dror and colleagues published a series of studies that tested whether forensic analysts were influenced by contextual information. The fingerprint studies became the most discussed, because fingerprint analysis was, and to a large extent still is, presented in courts as a definitive science with no error rate.

In the 2006 study (Dror, Charlton, and Peron), five qualified fingerprint examiners were shown prints they had previously examined and verified as a match (in the Mayfield case or comparable cases). They were given biasing contextual information, such as that the suspect had an alibi, and asked to re-examine the prints. Four of the five changed their original conclusion. The physical comparison had not changed. The information had.

Later studies extended the finding beyond fingerprints. Dror and colleagues showed similar susceptibility in DNA mixture interpretation, in forensic odontology, and in other pattern disciplines. A 2011 study showed that even the order in which exhibits were presented to an examiner could shift conclusions, because early exhibits primed a mental model that was then applied to later ones.

Dror's research suggested that telling analysts to 'just be objective' was unlikely to work. Bias operates below the level of deliberate choice. The structural response is linear sequential unmasking (LSU), a protocol that controls the order and timing of information flow to the analyst.

1. Stage 1: Examine the unknown sample
   The analyst examines and documents the questioned specimen, forming a preliminary assessment of its features, before seeing any reference material. This prevents the reference from anchoring what the analyst perceives in the questioned sample.
2. Stage 2: Examine the reference sample
   The analyst then examines the known reference material independently, documenting its features. Only now are the two assessed against each other, with the analyst's individual characterisations already fixed.
3. Stage 3: Comparison with minimised context
   The comparison is made with only the science-relevant case information available. Case theory, other examiners' conclusions, and suspect background are withheld at this stage.
4. Stage 4: Conclusion, then context
   The analyst reaches and documents a conclusion. Only then is additional case context released if needed for the report. Any change in opinion after this point must be documented and justified, flagging it for review.

LSU has been adopted or endorsed by a number of major forensic organisations. The FBI Laboratory, the UK HOSDB (Home Office Scientific Development Branch), and the Forensic Science Regulator's guidance all reference bias management protocols that incorporate elements of LSU. Implementation in practise remains uneven, particularly in smaller or less-resourced laboratories.

In 2009 the US National Academy of Sciences, commissioned by Congress, published 'Strengthening Forensic Science in the United States: A Path Forward'. The 352-page report was the product of a two-year committee review that included testimony from practitioners, scientists, lawyers, and judges. Its conclusions were direct.

The report called for Congress to establish an independent National Institute of Forensic Science (NIFS), mandatory accreditation for all labs, standardised certification for examiners, and a major research programme to establish the scientific foundations of the existing pattern-matching disciplines. Congress did not create NIFS. Instead, in 2013 the White House established the National Commission on Forensic Science (NCFS) as an advisory body, and the National Institute of Standards and Technology (NIST) expanded its forensic science research programme. NCFS was disbanded in 2017, though NIST's work continued.

In 2016, the President's Council of Advisors on Science and Technology published its own review: 'Forensic Science in Criminal Courts: Ensuring Scientific Validity of Feature-Comparison Methods'. Where the 2009 NAS report was broad and systemic, PCAST focused specifically on the admissibility standard for pattern-matching disciplines, applying the framework of the Supreme Court's Daubert decision (1993), which requires courts to act as gatekeepers for scientific evidence.

• DNA analysis (single-source): foundational validity demonstrated. PCAST endorsed it with established error rates.
• Latent fingerprint analysis: foundational validity 'substantially demonstrated' but with false positive rates that courts should be told about. PCAST identified only two adequate black-box studies at the time.
• Firearms/toolmarks: foundational validity 'not yet established'. Existing studies had methodological limitations.
• Bite marks: foundational validity 'not established'. PCAST found insufficient rigorous testing.
• Hair microscopy: foundational validity 'not established'. The FBI had already acknowledged this in a 2015 review that found examiners had overstated conclusions in 90% of trial transcripts reviewed.

PCAST recommended that courts be explicitly told when a discipline has not demonstrated foundational validity, and that expert testimony in such cases should be appropriately limited. The report met significant pushback from forensic professional organisations and the Department of Justice, but has been cited in admissibility rulings in several US states.

The 2009 NAS and 2016 PCAST reports shifted the conversation about forensic science permanently. They are cited in appellate decisions, relied on in admissibility challenges, quoted in the guidelines of professional bodies, and used as teaching texts in graduate programmes. The academic field of forensic metrology, concerned with error rates and measurement uncertainty, expanded substantially after 2009.

What has changed less is the day-to-day practice of pattern analysis in courts. Hair microscopy testimony largely disappeared in the United States after the FBI's 2015 review and the Innocence Project's work on exonerations involving it. Bite-mark testimony has been restricted in some jurisdictions. For disciplines like firearms and toolmarks, the debate over foundational validity continues, with proponents arguing that practitioner experience constitutes a form of validation that the PCAST framework does not recognise.