NIST FRVT (Face Recognition Vendor Test)
Definition
Ongoing NIST evaluation of commercial face-recognition algorithms, with the 2019 Demographic Effects report covering 189 algorithms from 99 developers; confirms Gender Shades findings at operational scale and shows false-positive rate differentials of factor 10-100 between demographic groups across some algorithm families.
- Conducted by
- NIST
- 2019 report scope
- 189 algorithms from 99 developers
- Key finding
- Confirmed Gender Shades findings at operational scale
- Disparity range
- False-positive rate differential of factor 10 to 100 between demographic groups in some algorithm families
Common questions
What does 'confirmed Gender Shades findings at operational scale' actually mean?+
The earlier Gender Shades study tested a small number of commercial systems and found higher error rates for darker-skinned individuals and women; FRVT's much larger evaluation of 189 algorithms found the same demographic disparity pattern held across a far broader and more representative set of commercial systems.
Does every face-recognition algorithm show the same level of demographic disparity?+
No, FRVT found substantial variation between algorithm families, with some showing minimal demographic differentials and others showing the 10 to 100 times factor difference, so the specific algorithm and version used in a case matters, not face recognition as a category.
How should this evaluation be used in a legal challenge to face-recognition evidence?+
Counsel or an expert can cite the specific algorithm's demographic performance data from FRVT to argue for or against reliability in a given case, since the report provides algorithm-specific error rates rather than a single industry-wide figure.
Related terms
- AFIS (Automated Fingerprint Identification System)
- An automated system that compares a fingerprint query (typically a latent print from a crime scene) against a reference database to identify...
- Clearview AI
- New York company that built a face-recognition database of approximately 30 billion images scraped from public websites without consent; subject to BIPA...
- Daubert Standard
- The US federal evidentiary standard (Daubert v. Merrell Dow Pharmaceuticals, 1993) requiring that expert testimony be based on scientifically valid methods with...
- Disparate Impact
- A legal and statistical concept describing a policy or practice that has a disproportionate adverse effect on a protected group, regardless of...
- EU AI Act Art 10 (Training Data Requirements)
- Provision requiring that training, validation, and testing datasets for high-risk AI systems be relevant, representative, free of errors, and complete, with appropriate...
- False-Positive Rate (FMR)
- In biometric verification: the probability that a comparison of images of two different individuals is incorrectly declared a match. In forensic identification,...
- Fitzpatrick Scale
- A skin-tone classification system originally developed by dermatologist Thomas B. Fitzpatrick in 1975 (initially covering four types for UV response in phototherapy),...
- Gender Shades (2018)
- Study by Joy Buolamwini and Timnit Gebru published at FAccT 2018 that measured accuracy disparities in three commercial face-recognition APIs across intersectional...
- Robert Williams Case (2020)
- First publicly documented wrongful arrest in the US attributed to a face-recognition false positive: Detroit police arrested Robert Williams, a Black man,...