Adversarial Attack
Definition
In AI detection contexts, a forgery specifically constructed to defeat a trained detector by incorporating small perturbations that cause the detector's classifier to output a 'genuine' prediction. A known limitation of neural-network-based forgery detectors operating against motivated, technically sophisticated adversaries.
- Domain
- AI-based forgery detection
- Mechanism
- Small input perturbations
- Target effect
- Classifier flips to a genuine prediction
- Applies to
- Neural-network-based detectors
Common questions
Why are small perturbations enough to fool a trained detector?+
Neural network classifiers learn decision boundaries from training data, and inputs crafted just across that boundary can flip the output even though the change is imperceptible or irrelevant to a human examiner.
What does this mean for relying on an AI detector's verdict alone in court?+
A detector's genuine or fake label should be treated as one input among several, since a motivated adversary with knowledge of the detector's architecture can specifically engineer an image or signature to defeat it.
Related terms
- Automatic Signature Verification (ASV)
- Algorithmic systems that compare a questioned signature against a reference set using extracted biometric features (stroke curvature, velocity profile, pressure distribution). Designed...
- C2PA (Coalition for Content Provenance and Authenticity)
- An open technical standard that embeds cryptographically signed provenance assertions into media files at the point of capture or editing. A C2PA...
- Content-Aware Fill
- An image editing feature (Photoshop from CS5; AI-driven from version 22.0) that replaces a masked region with synthesised pixel content matching the...
- Diffusion Model
- A generative neural network architecture (Ho et al., 2020; Stable Diffusion, Rombach et al., 2022) that learns to reverse a noise-addition process...
- ESDA (Electrostatic Detection Apparatus)
- A non-destructive technique for detecting indented impressions on paper caused by writing pressure. A physical ink signature on paper produces detectable ESDA...
- GAN (Generative Adversarial Network)
- A neural architecture introduced by Goodfellow et al. (2014) consisting of a generator and discriminator trained in opposition. The generator learns to...
- GAN Spectrum Fingerprint
- Periodic artefacts in the Fourier frequency spectrum of images generated by GAN architectures with upsampling layers. First reported by Zhang et al....
- JUMBF (JPEG Universal Metadata Box Format)
- A container format (ISO/IEC 19566-5) used by the C2PA specification to embed provenance manifest data within JPEG image files. The JUMBF box...
- PRNU (Photo Response Non-Uniformity)
- The unique pattern of pixel-level sensitivity variations in a camera sensor, used as a device fingerprint. A genuine camera-original image carries the...