Multiple Comparisons Problem
Definition
The inflation of the probability of at least one false-positive result that occurs when many statistical tests or pairwise comparisons are made. In forensic database searches, the probability of a coincidental match across N comparisons is much higher than the single-pair match probability.
- Effect
- Inflates probability of at least one false positive
- Cause
- Many statistical tests or comparisons run together
- Also called
- Multiple testing problem
- Forensic relevance
- DNA or fingerprint database search hits
Common questions
Why is a database cold-hit different from a single suspect comparison in probability terms?+
A cold hit searches one profile against every record in a database, so the chance of a coincidental match rises with the number of comparisons made, and courts have required this database-search probability, not the single-pair random match probability, be stated.
How can an investigator correct for the multiple comparisons problem?+
Statistical corrections such as adjusting the significance threshold by the number of comparisons made, or explicitly reporting the database size alongside the match statistic, are the standard ways to keep the reported probability honest.
Related terms
- Base Rate
- The prior probability of an event or proposition before specific evidence is considered. In forensic inference, the base rate is the probability...
- Bonferroni Correction
- A standard adjustment for multiple comparisons: the significance threshold for any individual test is divided by the total number of tests performed,...
- Database Match Probability
- The adjusted probability that a coincidental match to a crime-scene profile exists somewhere in a database, accounting for the number of profiles...
- Ecological Fallacy
- The error of applying a statistical association observed at the population or group level to an individual, without verifying that the individual...
- Look-Elsewhere Effect
- The inflated apparent significance of a finding that arises when many features, tests, or subsets are examined and only the significant result...