Bias and Disparate Impact in Face and Fingerprint Matching

Joy Buolamwini's 2018 doctoral research at the MIT Media Lab, conducted with Timnit Gebru (then at Microsoft Research), produced the Gender Shades study, published at the Conference on Fairness, Accountability, and Transparency (FAT*) in February 2018. The study evaluated three commercial face-analysis APIs:

• Microsoft Face API
• IBM Watson Visual Recognition
• Face++ (Megvii)

Each was tested on a benchmark dataset balanced across four intersectional gender-by-skin-tone categories: lighter-skinned males, lighter-skinned females, darker-skinned males, and darker-skinned females, using Fitzpatrick skin-tone scale subgroups III-VI as the measure of skin tone.

The results were stark. Across all three APIs, accuracy on lighter-skinned male faces was dramatically higher than on darker-skinned female faces:

• Microsoft Face API: 99.4% accuracy on lighter-skinned males vs 79.2% on darker-skinned females (93.7% overall)
• IBM Watson: 99.2% on lighter-skinned males vs 65.3% on darker-skinned females
• Face++: similar patterns

The intersectional analysis was the methodological innovation: prior studies had examined gender or skin tone independently, missing the compound disadvantage experienced by darker-skinned women.

Gender Shades produced immediate responses. IBM updated its API; Microsoft issued a similar statement. The 2019 follow-up by Raji and Gebru ("Saving Face: Investigating the Ethical Concerns of Facial Recognition Auditing") found improvement in some but not all metrics, with darker-skinned female faces still showing the highest error rates. The study's broader contribution was methodological: it established that independent third-party auditing of commercial AI systems against demographically balanced benchmarks is both possible and necessary, a framework since adopted by NIST, the EU AI Act's conformity-assessment requirements, and several civil-society AI-auditing initiatives.

The NIST Face Recognition Vendor Test (FRVT) Demographic Effects report, published in December 2019, extended the Gender Shades methodology to a dramatically larger scope. NIST tested 189 algorithms submitted by 99 developers against three large-scale datasets:

• 18.27 million images from approximately 8.49 million subjects (US application database)
• 1.64 million images from 270,000 subjects (border-crossing dataset)
• A smaller set of mugshot-style images

The datasets allowed simultaneous analysis by country of birth, age, and gender.

The FRVT findings confirmed the Gender Shades pattern at scale and extended it in two important directions:

1. False-positive rates (the probability that images of two different people are incorrectly declared a match) were highest for African-American and Asian faces relative to Caucasian faces, with differentials of factor 10 to 100 across the tested algorithms. False positives matter more than false negatives for forensic identification: a false positive means the system presents an innocent person as a candidate match to an investigator.
2. Training-data demographics had the largest single effect on demographic differentials. Algorithms trained on datasets with greater demographic balance showed smaller differentials.

The FRVT report noted that results applied to algorithms as submitted, not current operational deployments, as vendors update products continuously. NIST has continued publishing updated FRVT results and in 2022 introduced the FRVT Morph track, examining algorithm detection of morphed images with demographic analysis. The FRVT framework has been adopted as a reference by the EU AI Act's conformity-assessment provisions, the UK Home Office's interim facial recognition guidance, and the Indian Ministry of Home Affairs' draft framework for police facial recognition systems.

Fingerprint recognition has a longer scientific lineage than face recognition, and the question of whether fingerprint characteristics vary systematically across demographic groups dates to Francis Galton's 1892 monograph "Finger Prints," in which Galton observed apparent differences in the average ridge density of prints from individuals of different ethnic backgrounds. The modern operational question is more specific and more tractable: do contemporary AFIS algorithms produce systematically different false-match rates (FMRs) and false-non-match rates (FNMRs) for latent prints submitted from individuals belonging to different demographic groups?

The most rigorous recent evidence comes from a 2020 paper by Tao, Datta, Hicklin, and colleagues published in Science Advances, which examined 100 million fingerprint comparison decisions from operational FBI data. The study found that automatic fingerprint-matching algorithms produced higher false-positive rates for female subjects compared to male subjects, and for African American subjects compared to other racial groups, for certain query types. The magnitude of the effect was smaller than those found in face recognition by NIST FRVT, but statistically robust. The study attributed the finding primarily to smaller average fingerprint area in female subjects (which reduces the ridge detail available for comparison) and to training-corpus demographics for AFIS algorithms.

A parallel literature exists on latent fingerprint examiners (human experts) rather than automated systems. Research by Bradford, Neumann, and colleagues has examined whether human examiner decisions show systematic demographic effects; results are mixed and contested, but some studies have found that examiners' assessments of ridge clarity (a threshold judgment that affects whether a mark is deemed suitable for comparison) may vary with mark characteristics that are correlated with the source individual's demographic group. This is a more subtle claim than the AFIS finding: it concerns the prior decision of whether to compare, not the comparison outcome itself.

On 9 January 2020, Robert Williams, a Black man living in Farmington Hills, Michigan, was arrested at his home in front of his wife and young daughters by Detroit Police Department detectives. They showed him a still image from a store surveillance camera showing a shoplifter stealing approximately USD 3,800 worth of watches. Williams replied, "I hope you don't think all Black men look alike." The identification had been made by a face-recognition system used by the Michigan State Police, which returned Williams's photo from the state driver's licence database as a candidate match. The officers, apparently relying on the algorithmic output without independent confirmation, applied for and obtained an arrest warrant.

Williams was held for 30 hours before the charges were dropped. The Detroit Police Department acknowledged that the identification had been made by a face-recognition system, in apparent conflict with its own guidance requiring human review and independent corroboration before any arrest on a face-recognition hit. The Williams case was the first known public documentation of a wrongful arrest attributable to a face-recognition false positive in the United States. Two further cases involving Black men became public within 18 months:

• Michael Oliver (Detroit, 2019)
• Nijeer Parks (New Jersey, 2019)

Both involved false-positive identifications from face-recognition systems and inadequate human verification.

The policy response was rapid:

• IBM (June 2020): ceased development and sale of general-purpose face-recognition technology; called on Congress to enact national standards before law-enforcement deployment.
• Microsoft: announced it would not sell face-recognition technology to US police departments until a federal law was enacted.
• Amazon: placed a one-year moratorium on police use of Rekognition (later extended indefinitely for law enforcement).
• Axon: declined to add face recognition to its body camera systems.
• US cities: San Francisco (2019), Boston (2020), and Minneapolis (2021) enacted prohibitions on municipal government use of face recognition.

At the federal level, the Facial Recognition and Biometric Technology Moratorium Act was introduced in Congress in 2020 but did not pass.

In the EU, the European Parliament adopted a resolution in October 2021 calling for a moratorium on law-enforcement face-recognition deployments, citing the NIST FRVT evidence on demographic disparities. The EU AI Act's Article 5 prohibition on real-time remote biometric identification in public spaces reflects a partial legislative response, with narrow exceptions requiring necessity and proportionality at each deployment. For the full regulatory picture, see biometric evidence in court: EU AI Act, DPDP and US statutes.

In India, the Ministry of Home Affairs launched a national procurement process for police facial recognition systems in 2019. Civil society organisations including the Internet Freedom Foundation have challenged the deployment and filed Right to Information requests documenting the absence of an accuracy-testing framework or demographic-effects analysis.

Clearview AI, a New York-based company founded in 2017, built a face-recognition database by scraping photographs from social media platforms, news websites, and other public-facing websites without the consent of the individuals depicted. By 2020, the database contained approximately 3 billion images; by 2024, the company claimed more than 30 billion. Clearview licensed access to its search tool to law-enforcement agencies in the United States, the United Kingdom, Canada, and several other countries, allowing investigators to upload a face image and receive a list of candidate matches drawn from the scraped database, with links to the source pages.

The company's practices triggered simultaneous regulatory and civil-litigation responses in multiple jurisdictions. In the US, Clearview faced a class action under Illinois BIPA (Mutnick v. Clearview AI, N.D. Ill., later consolidated as In Re: Clearview AI Consumer Privacy Litigation) for collecting facial geometry data of Illinois residents without consent, without a published retention policy, and without written releases. A settlement reached in 2022 included a bar on Clearview selling its product to private businesses in Illinois (not to law enforcement) and a commitment not to give free trials to Illinois law-enforcement agencies. The Illinois settlement's limitation to Illinois and its carve-out for law enforcement illustrated the constraints of state-level enforcement on a nationally operating company.

In the EU, the Italian Data Protection Authority (Garante) fined Clearview EUR 20 million in 2022 for violations of GDPR Articles 5 (lawfulness, fairness, transparency), 6 (lawful basis), 9 (special-category data), and 13 (information to data subjects), and ordered deletion of EU residents' data from Clearview's database. The UK Information Commissioner's Office (ICO) issued a GBP 7.5 million fine in 2022 for similar GDPR violations. On appeal in October 2023, the First-Tier Tribunal overturned the fine on jurisdictional grounds; the ICO's counter-appeal to the Upper Tribunal succeeded in October 2025, restoring jurisdiction and remitting the case to the FTT for substantive determination. The fine amount has not been finally settled. Canada's privacy commissioners jointly concluded in 2021 that Clearview's collection was unlawful under PIPEDA (Personal Information Protection and Electronic Documents Act) and that Clearview had failed to obtain meaningful consent. Australia's Privacy Commissioner reached a similar conclusion under the Privacy Act 1988. The enforcement actions collectively establish that scraping biometric data from public-facing websites without consent does not satisfy any of the GDPR Article 9(2) exceptions and is similarly unlawful under equivalent national frameworks.

1. Identify the demographic representation of the training dataset
   Before deploying any face-recognition or fingerprint AFIS system, require the vendor to disclose the demographic composition of the training, validation, and test datasets. A system trained on a corpus that underrepresents the population it will be applied to will show inflated error rates for underrepresented groups. Document this as part of the DPIA under GDPR Art 35 or the risk assessment required for high-risk AI systems under the EU AI Act.
2. Obtain and review published demographic-effects test results
   NIST FRVT results are publicly available for submitted algorithms. Require vendors to disclose their FRVT submission results or equivalent third-party test results disaggregated by age, gender, and race/ethnicity. If no such results exist, treat the system as unvalidated for demographic-differential performance.
3. Establish a human-verification requirement before any investigative action
   The Williams, Oliver, and Parks cases all involved acting on an algorithmic match without adequate independent human verification. Standard operating procedures for law-enforcement face-recognition searches must require at least one independent examiner review of the match before the result is used as a basis for any investigative action, arrest application, or charge.
4. Apply Daubert or equivalent admissibility analysis to demographic-effects evidence
   In any case where biometric evidence forms part of the prosecution, defence counsel should request disclosure of the system's NIST FRVT results or equivalent demographic-effects data. If the system's false-positive rate for the defendant's demographic group is materially higher than for other groups, this is relevant to the reliability assessment under Daubert (US), the Criminal Procedure Rule 33 expert evidence regime (UK), or the Indian Evidence Act s.45 and Bharatiya Sakshya Adhiniyam provisions on expert opinion.
5. Document and audit algorithm updates
   Vendor algorithms change between versions. The NIST FRVT tests a specific algorithm version; an operational deployment may be running a different version with different demographic-effect profiles. Require vendors to notify operational deployments of any algorithm update and to provide updated demographic-effects test results before the new version is used in investigative or evidential contexts.

The admissibility of face-recognition and fingerprint AFIS evidence in criminal proceedings has traditionally been assessed under general expert-evidence frameworks: the Frye general-acceptance standard (US federal courts before 1993 and still some state courts), the Daubert reliability standard (US federal courts under Fed. R. Evid. 702 after Daubert v. Merrell Dow Pharmaceuticals, 1993), the Criminal Procedure Rules Part 33 and Criminal Practice Directions regime (England and Wales), and the Indian Evidence Act section 45 (expert opinion) as amended and re-enacted in the Bharatiya Sakshya Adhiniyam 2023.

Under Daubert, expert testimony must be based on "sufficient facts or data," a "reliable principles and methods," and the application of those methods reliably to the facts of the case. Federal Rule of Evidence 702's 2023 amendment made the preponderance-of-the-evidence standard for reliability more explicit. The demographic-effects evidence from NIST FRVT creates two Daubert challenges. First, a system with a documented elevated false-positive rate for the defendant's demographic group may fail the "reliable application to the facts" prong if the expert witness cannot demonstrate that the specific system's known demographic error profile has been accounted for in the probabilistic assessment of the identification. Second, if the system was not submitted to FRVT or an equivalent test, the proponent may struggle to establish that the system has been "tested" and that its "known or potential rate of error" is within acceptable limits, as required by Daubert.

In England and Wales, the Forensic Science Regulator's Codes of Practice require that any method used in casework meet defined validation standards, including known error rates. The R (Bridges) case found that South Wales Police had failed to comply with the Public Sector Equality Duty because they did not adequately assess whether the facial recognition system risked indirect discrimination, making undisclosed or untested demographic-bias profiles a ground of unlawfulness under the Equality Act 2010; it follows that a system with undisclosed or untested demographic effects would not meet the Forensic Science Regulator's validation requirements for evidential use. The Law Commission's 2011 review of expert evidence recommended that courts apply a reliability assessment, which practitioners have since argued encompasses demographic-effects testing for biometric systems.

Under India's Bharatiya Sakshya Adhiniyam 2023 (replacing the Indian Evidence Act 1872), Section 39 preserves the admissibility of expert opinion where the court needs the opinion of a person specially skilled in science, art, or profession. The reliability of a biometric identification is assessed under Section 57's provision for electronic evidence and the Electronic Records Act requirements. There is no Indian equivalent of the structured Daubert reliability analysis, but the general principle of Section 39 (that an expert's opinion is admissible only insofar as the expert's methodology is sound) provides a basis for challenging a face-recognition identification that relies on a system with known demographic-differential error rates that have not been disclosed to the court.

The EU AI Act's high-risk classification for biometric identification systems requires, under Article 10, that training data be "relevant, representative, free of errors and complete," with "appropriate measures to identify and address possible biases." An AI Act conformity assessment that does not address demographic effects will not satisfy Article 10. For law-enforcement deployments under Article 5's exception structure, the prior judicial or administrative authorisation requirement implies that the authorising body must be informed of the system's known demographic-error profile; authorising a deployment without that information would compromise the necessity and proportionality assessment.