Cognitive Bias, Expert Testimony and the 2009 NAS Critique

The NAS committee, chaired by Harry T. Edwards (senior judge, US Court of Appeals for the DC Circuit) and C. Michael Ioannidis (materials scientist), reviewed fire investigation alongside fingerprints, bite marks, hair analysis, and other disciplines. Its fire-investigation findings, reported in Chapter 9, cited several specific categories of concern.

First, the empirical foundation problem. Many pattern indicators used to determine fire origin, including the shape of the burn pattern at the presumed area of origin, alligator charring depth and surface texture, low burn lines running along floor surfaces, and the distribution of calcination on gypsum wallboard, had been applied as heuristics derived from the personal experience of trained investigators. The NAS found that few of these indicators had been subjected to controlled experimental testing to determine their specificity: that is, whether they occurred in accidental fires without any added accelerant. The UL research published between 2004 and 2011, in which NFPA 921-based investigators blind-interpreted fire scene photographs from both arson and accidental fires, found error rates that were significantly higher than the near-zero error rates implied by examiner confidence. In some rounds, examiners disagreed with each other and with the known reference classification at rates above 20 per cent.

Second, the error-rate problem for Daubert. The NAS report explicitly framed this as a Daubert admissibility issue. Under Daubert v. Merrell Dow Pharmaceuticals (1993), admissibility of expert testimony in federal court requires, among other criteria, that the method have a known error rate. If no controlled studies had been conducted to determine the false-positive rate of inferring deliberate fire-setting from pattern indicators alone, then the Daubert criterion was not met, and fire investigation testimony constructed on those indicators was legally vulnerable even if the expert was personally experienced.

Third, the cognitive-bias problem. The NAS noted that fire investigators in most jurisdictions collected scene evidence, interviewed witnesses, reviewed police intelligence about the suspect, and then formed an opinion in the same investigation. This structure creates confirmation bias: an investigator who is told before examining the scene that the building owner is in financial difficulty and has recently increased his insurance will interpret ambiguous burn patterns through that frame. The NAS called for separation of the scene-observation function from the opinion-formation function, and for external check mechanisms.

Fourth, the proficiency-testing problem. The NAS found that proficiency testing for fire investigators was neither mandatory nor systematically conducted. The CTS fire debris scheme (for laboratory analysts) existed, but field fire investigation proficiency testing (where investigators examine test scenes and have their origin-and-cause conclusions checked against known reference) was rarely applied. This meant that error rates could not be empirically measured even if the discipline had wanted to report them.

The NIST experiments most directly relevant to the NAS fire critique were conducted at the Bureau of Alcohol, Tobacco, Firearms and Explosives (ATF) Fire Research Laboratory in Maryland and published through the NIJ grant programme. John DeHaan's laboratory research on burn pattern formation in furnished rooms, published progressively from the early 1990s through the 2000s, demonstrated that patterns previously attributed exclusively to accelerant use could be reproduced in ordinary accidental fires given the right fuel load and ventilation conditions. DeHaan's work formed part of the empirical basis for successive editions of NFPA 921.

The Underwriters Laboratories project, a collaboration between UL's Fire Safety Research Institute and NIST, ran a series of controlled arson-vs-accident test fires in which trained investigators (working under NFPA 921) were blinded to the actual cause and asked to classify each scene. Results published in 2011 showed that even experienced NFPA 921-trained investigators produced false-positive arson classifications in a material fraction of accidental test fires, and false-negative classifications (missing deliberate fires) in a smaller fraction of incendiary test fires. These error rates, while lower than the pre-NFPA 921 era, confirmed that pattern-only origin determination carried residual uncertainty that was not being disclosed to courts.

The BRE (Building Research Establishment) in the UK conducted parallel research on fire patterns in UK residential construction, which differs from US residential construction in materials (brick and block construction is more common in the UK, timber-frame more common in the US). BRE findings, published through the Home Office Scientific Development Branch and the Journal of Fire Sciences, similarly found that ventilation-controlled burning produced floor-level burn patterns and deep char deposits that could be mistaken for accelerant indicators under older interpretive frameworks.

In India, the National Accreditation Board for Laboratories and the Central Forensic Science Laboratory network have not, as of 2025, sponsored published controlled-experiment validation studies on fire pattern indicators in Indian construction types (which include concrete-framed multistorey construction, unreinforced masonry, and vernacular building types absent from US and UK research). This represents a gap: the NAS critique was anchored to US construction and US investigative practice, and its direct applicability to Indian scene patterns in reinforced concrete buildings has not been empirically tested.

Research in cognitive psychology applied to forensic science, led principally by Itiel Dror at University College London and collaborators including Jennifer Mnookin, Simon Cole, and Hired Gun scholars affiliated with the Innocence Project, has identified several bias mechanisms that apply specifically to fire investigation.

Contextual bias (also called task-irrelevant contextual information bias) occurs when information about the case context available to the examiner before or during analysis influences the direction of the result. In fire investigation, the most common form is the investigating officer briefing the fire examiner on suspected motive before the scene examination begins. An examiner who is told that the property owner has a history of insurance fraud and has been seen near the scene in the hours before the fire will approach the scene with a hypothesis that is not purely evidence-driven. Ambiguous patterns, including char depths that could represent either accelerant use or extended burning, will be interpreted in the direction of the pre-existing hypothesis.

Anchoring bias occurs when an investigator's initial judgment about origin or cause, formed early in the scene examination, becomes resistant to revision as conflicting evidence emerges later in the examination. Fire scenes are excavated progressively, and evidence about origin typically appears at different times. An examiner who concludes from early surface-level burn patterns that the fire started in a particular corner may resist revising that conclusion when deeper excavation reveals an electrical fault in an adjacent location.

Expectation bias (observer effect) occurs when the examiner knows what a previous examiner found and allows that finding to constrain their own independent analysis. In multi-examiner cases (for example, where an insurance investigator precedes a police investigator), knowledge of the earlier finding predisposes the second examiner to confirm rather than independently evaluate.

Itiel Dror's controlled studies on fingerprint examiners, published from 2006 onwards, demonstrated that the same fingerprint could be classified differently by the same examiner depending on the contextual information provided, even when the actual print had not changed. Although these studies focused on fingerprints, the underlying bias mechanism is domain-general, and subsequent research by Charman and colleagues extended the finding to fire investigation contexts specifically.

Sequential unmasking, developed by W.C. Thompson and articulated in the forensic context by Itiel Dror, is an examination protocol in which case-relevant contextual information is released to the examiner in a controlled sequence, so that technical-analysis steps are completed before investigative context is revealed. Applied to fire investigation, a sequential-unmasking protocol would proceed as follows: the examiner first conducts and documents all physical scene observations, records measurements, and formulates preliminary hypotheses from scene evidence alone; only after this documentation is complete are police intelligence reports, witness statements, and financial background information provided. The examiner then revisits the documented observations in light of the investigative context, making explicit and auditable any changes to preliminary conclusions.

Blind verification applies a separate analyst (or examination team) to independently re-examine or re-classify an exhibit, without knowledge of the primary analyst's conclusion. In laboratory fire debris analysis, blind verification is operationally feasible: a second analyst re-reads the GC-MS data without seeing the first analyst's classification. In field fire investigation, full blind verification is structurally difficult because the second examiner typically reviews photographs and notes rather than the live scene. The NFPA 1033 working group has been debating whether peer technical review (an examiner reviewing another's work knowing the first examiner's conclusion) can substitute for genuine blind re-examination. The consensus, as of the 2024 NFPA 1033 edition, is that peer technical review satisfies minimum quality requirements but that genuinely blind re-examination is preferred where operationally possible.

The linear ACE-V (Analysis, Comparison, Evaluation, Verification) framework, developed in fingerprint examination and adopted across several forensic disciplines, provides a formal record of the separation between observation (A), comparison against reference (C), and evaluative conclusion (E). Applied to fire investigation, a linear ACE-V requires the examiner to document the Analysis (scene observations and measurements) in a separate, time-stamped record before moving to the Comparison stage (reviewing the evidence against known-pattern templates or background samples) and the Evaluation stage (forming and testing the origin-and-cause hypothesis). Verification (V) is then conducted independently. This structure does not eliminate bias but it creates an audit trail that allows later review of whether contextual information contaminated the technical-analysis stages.

The ENFSI Best Practice Manual for Fire Investigation (third edition, 2021) incorporates sequential unmasking and blind technical review as recommended practice. The ASTM Forensic Science Standards Board has a corresponding discussion document; the 2024 NFPA 921 edition includes commentary on bias mitigation that explicitly references the Dror literature.

The traditional fire investigation courtroom opinion is categorical: "In my opinion, the fire was incendiary in origin." The NAS critique, and subsequent development in the evaluative-reporting literature, calls for probabilistic language that discloses the uncertainty inherent in the opinion. The transition is most advanced in England and Wales, where the FSR Codes of Practice mandate evaluative reporting for DNA and progressively encourage it for other forensic disciplines including fire debris analysis.

The Bayesian evaluative framework expresses the evidential weight of a finding as a likelihood ratio (LR): the probability of the evidence (the observed pattern or residue finding) given the prosecution's proposition (accelerant was added), divided by the probability of the evidence given the defence's proposition (no accelerant was added; the observed pattern resulted from ordinary pyrolysis). An LR greater than 1 supports the prosecution's proposition; an LR less than 1 supports the defence's proposition. The examiner does not assign a probability to guilt or innocence; that is for the jury. The examiner only quantifies how much more or less probable the evidence is under one proposition than the other.

For fire debris GC-MS findings, the LR can be empirically grounded: population-level data from fire debris proficiency testing and OSAC validation studies provide the denominator (how often does a substrate produce the observed chromatographic pattern in the absence of added accelerant?). For field fire pattern evidence (burn shapes, char depth, calcination), the empirical data for LR computation are sparse. This is precisely the NAS critique: without the population-level frequency data that would populate the denominator of the LR, probabilistic opinion language becomes verbal rather than empirical, and the apparent precision of Bayesian reporting conceals a fundamentally unsupported inference.

The UK Court of Appeal in R v. T [2010] EWCA Crim 2439 (a footwear case) and the subsequent discussion in R v. Dlugosz [2013] EWCA Crim 2 warned against LR evidence presented without an adequate empirical database. The Forensic Science Regulator's guidance and the Royal Statistical Society's guidance on expert probability evidence both emphasise that LR reporting with inadequate population data is not inherently safer than categorical reporting: it creates a false veneer of quantitative precision over an essentially subjective judgment.

In India, the Bharatiya Sakshya Adhiniyam 2023 (BSA) Section 39 governing expert evidence does not prescribe a specific reporting language or LR framework. The courts have consistently treated categorical expert opinion as the expected format. Bayesian-framed evidence has appeared in a small number of DNA cases before the Supreme Court and Delhi High Court, but fire investigation opinions in Indian courts remain almost entirely categorical, mirroring the pre-2009 pattern in the US and UK. The transition, if it comes, is likely to lag the UK timeline by a decade or more.

In the United States, the two dominant admissibility standards are Daubert (applied in federal court and a majority of state courts) and Frye (applied in a minority of state courts, including California, Illinois, and New York). Under Daubert, the trial judge assesses whether the methodology is (1) testable, (2) peer-reviewed, (3) generally accepted, and (4) associated with a known error rate. The NAS critique directly weakened fire investigation testimony on criterion (4): if no controlled studies established the error rate of pattern-based incendiary classification, the Daubert error-rate criterion was not satisfied. Following the NAS report, defence challenges to fire investigation testimony using the Daubert framework increased, and several federal and state courts excluded or limited testimony where the challenged indicators (alligator charring, low burn without accelerant) were the primary basis of the incendiary classification.

The Frye standard, older and more permissive, requires only that the methodology be generally accepted within the relevant scientific community. Because NFPA 921 itself is generally accepted, testimony based on NFPA 921 methodology tends to survive Frye challenges more easily. However, the "generally accepted" criterion has its own complexity: NFPA 921 is generally accepted as a framework, but specific pattern indicators within that framework may not be generally accepted as definitive incendiary evidence, and Frye scrutiny at that granularity is available.

In the UK, expert evidence in criminal proceedings is governed by Criminal Procedure Rules Part 19. CrimPR Rule 19.4 requires the expert to provide a statement of the expert's qualifications, the methodology, the limitations of the methodology, and whether the methodology represents currently recognised standards. The "Cairns checklist," developed from the judgment in R v. Dlugosz and the subsequent work of the Law Commission, provides a structured framework for assessing whether expert evidence meets the CrimPR threshold. The FSR Codes of Practice, now statutory under the Forensic Science Regulator Act 2021, intersect with CrimPR Part 19 by providing the regulatory baseline for "currently recognised standards."

In India, the admissibility of expert opinion under BSA 2023 Section 39 (successor to IEA Section 45) rests with the court's assessment of the expert's competence. Unlike Daubert, there is no structured methodology test. The expert is cross-examined on credentials, experience, and method, but the court is not formally required to conduct a pre-admission gatekeeping hearing (the Indian equivalent of a Daubert hearing is not a codified feature of Indian evidence law). Defence challenge of fire investigation opinions in Indian courts therefore focuses on cross-examination of the witness rather than pre-trial exclusion. This gives defence counsel the opportunity to expose methodological weakness but also means that juries (or judges) are exposed to problematic testimony without a pre-screening filter.