Deepfake Detection Methods: From CNNs to Foundation Models

Before neural classifiers dominated detection research, forensic investigators relied on handcrafted features: specific artifacts expected from a known generation pipeline. These methods are interpretable, do not require a labelled training corpus, and can be validated against first-principles models of image formation. Their limitation is that they target artifacts from a particular generation method, and a generation method that avoids those artifacts evades the detector.

Blending boundary analysis targets the seam between a synthetically generated face and the original video background. GAN-based face swaps must composite the generated face region onto the source video.

Early implementations produced visible colour or texture discontinuities at the jaw, hairline, and ear boundaries. Even when invisible to the naked eye, these boundaries often have statistical properties different from within-region texture, detectable via gradient analysis or illumination consistency checks. See Noise Inconsistency and Lighting Analysis for the underlying techniques applied to splicing detection, which transfer directly to face compositing.

Frequency-domain analysis exploits GAN upsampling artifacts. Neural generators typically use transposed convolutions or nearest-neighbour upsampling to produce full-resolution output from a low-resolution latent code. These operations introduce periodic spectral patterns at frequencies corresponding to the upsampling stride, visible as grid-like peaks in the 2D Fourier spectrum.

Real camera images, shaped by optical blur and sensor noise, do not show these peaks. Frequency-domain detectors compute the 2D DFT of image patches and look for anomalous spectral structure. They are effective against early GANs but diffusion models, which do not use the same upsampling chain, produce different spectral signatures.

The dominant detection paradigm from 2018 to 2022 was the binary CNN classifier: a network trained on pairs of real and manipulated frames to output a probability that the input is fake.

Rossler et al. (2019) demonstrated that XceptionNet, a depthwise-separable CNN designed for image classification, achieves over 99% per-frame accuracy on the FaceForensics++ dataset at low compression when trained on the same manipulation method. This result established the binary CNN approach as the baseline and the dataset as the evaluation standard.

FaceForensics++ contains 1000 original videos sourced from YouTube (with consent), each manipulated by four methods. DeepFakes uses autoencoder-based face identity swap. Face2Face transfers the expression of a source actor onto a target face without swapping identity.

FaceSwap replaces the face region using a 3D model. NeuralTextures refines the output of a 3D-model-based swap with a neural renderer to improve texture fidelity. All four manipulations are provided at three compression levels: raw (lossless), c23 (visually lossless H.264), and c40 (heavy H.264 compression approximating social media re-encoding). Standardised splits allow fair comparison between methods.

The generalisation gap was confirmed almost immediately after FaceForensics++ was released. A classifier trained on DeepFakes and evaluated on FaceSwap achieves accuracy barely above chance, because the learned features are specific to the autoencoder's blending artifacts rather than to forgery in general.

Cross-dataset evaluations, testing on the Celeb-DF, DFDC, or WILD datasets after training on FaceForensics++, showed the same pattern. Several architectural responses have been proposed: disentangled feature learning that separates identity from forgery cues; multi-task networks that simultaneously predict forgery mask and forgery label; and attention mechanisms that force the classifier to focus on specific face regions rather than global texture.

Biological signal methods exploit the observation that generative models produce faces frame-by-frame without modelling the underlying physiology that drives real face video. A real face contains subtle colour variation driven by the cardiac cycle, consistent eye-movement patterns, and head motion correlated with speech articulation. These signals are either absent or incoherent in synthetic video because the generator optimises each frame for visual plausibility, not for temporal physiological consistency.

Remote photoplethysmography (rPPG) measures the pulse-driven change in skin reflectance. Oxyhaemoglobin and deoxyhaemoglobin absorb light at different wavelengths, and the ratio changes with each heartbeat, producing a periodic signal in the green channel of face video at roughly 0.7 to 3 Hz (42 to 180 bpm).

Li et al. (2018) showed that deepfake videos lacked coherent rPPG signals because the autoencoder learned to map from a mean skin colour rather than a temporally varying one. The method requires 10 to 30 seconds of video and is sensitive to lossy compression that removes the subtle colour variation. It performs poorly on videos of subjects with darker skin tones where the reflectance amplitude is lower.

Eye-blink analysis was an early biological approach. Researchers at the University at Albany reported in 2018 that early deepfakes blinked infrequently because training images were sourced predominantly from still photographs with eyes open. Subsequent generation systems were trained to include blinking sequences, closing this specific gap. The general principle, that generators trained on biased data inherit the bias, remains valid; the specific detector must be updated as training data evolves.

Lip synchronisation consistency is particularly relevant for audio-visual deepfakes, where a real audio track is paired with a manipulated video. Cross-modal methods compute a similarity score between the phoneme sequence derived from audio and the lip shape sequence in video. A genuine recording should score high; a mismatched or synthesised video should score lower. The SyncNet architecture from Chung and Zisserman is commonly used as the lip-sync scoring backbone.

Vision transformer (ViT) architectures applied to deepfake detection emerged from 2021 onward. Transformers process images as sequences of patches and use self-attention to model long-range spatial dependencies, making them less likely to overfit to localised artifact patterns than CNNs.

Zhao et al. (2021) proposed a multi-attentional network that forces attention on different face regions simultaneously: the global face structure, local regions where blending boundaries appear, and eye or mouth areas where biological signal methods focus. The multi-attention approach improved cross-dataset generalisation over single-attention baselines.

Foundation models bring a qualitatively different asset: pre-training on hundreds of millions of image-text pairs develops visual representations that are not tied to any specific generation method.

CLIP (Contrastive Language-Image Pre-Training) encodes images into a joint embedding space with text. When fine-tuned on a synthetic media detection task, the CLIP image encoder provides features that reflect high-level semantic consistency rather than low-level GAN artifacts. Studies by Wang et al. and subsequent work showed that CLIP-based detectors transfer better to unseen diffusion-model-generated content than CNN detectors trained on the same data, because the backbone representations are more general.

Video-native temporal transformers, such as TimeSformer and Video Swin Transformer, process sequences of frames jointly and have been applied to deepfake detection to capture temporal inconsistencies across frames that single-frame classifiers miss. A face that is plausible in any individual frame may show unnatural motion trajectories or illumination flicker across a 10-frame window. Temporal models are computationally expensive but address a real failure mode of per-frame classifiers.

The generalisation gap between training and deployment distributions is the most cited limitation in detection literature. A detector achieving 99% accuracy on FaceForensics++ c23 is not a 99%-accurate detector in forensic practice; it is a detector that achieves 99% on one dataset under controlled compression.

Celeb-DF (a higher-quality deepfake dataset released in 2020), the Facebook Deepfake Detection Challenge (DFDC) dataset, and the WildDeepfake dataset have all been used to measure cross-dataset performance, and the results are consistently lower than in-distribution numbers.

Three strategies address the generalisation gap. Data augmentation trains classifiers on augmented images with varied compression, noise, and post-processing to make learned features invariant to these transformations.

Disentangled or invariant feature learning explicitly separates forgery-relevant from forgery-irrelevant features during training, typically using an adversarial training component or a domain-adaptation loss. Ensemble methods combine multiple detectors targeting different artifact types; if each individual detector is fooled by a different adversary, the ensemble is harder to fool simultaneously. None of these fully closes the gap against a generation method that was not represented in training data.

Deepfake detection evidence faces the same admissibility criteria as any scientific expert testimony, applied to a method that courts and legislators are still calibrating. In the United States, the Daubert standard (codified in Federal Rule of Evidence 702) requires that expert methodology be based on sufficient facts or data, be the product of reliable principles and methods, and have been reliably applied to the facts of the case.

A detector with an unknown error rate, or one whose error rate was measured only on FaceForensics++ and is not known for the actual evidence format, is vulnerable to exclusion.

In the United Kingdom, the Criminal Procedure Rules and Criminal Practice Directions set out the duties of expert witnesses and the criteria courts use to assess reliability.

A 2024 Forensic Science Regulator guidance note on AI-assisted forensic tools requires that any AI tool used in evidence preparation have a documented validation study at relevant compression levels and against generation methods representative of current practice. In the European Union, the AI Act (fully applicable from August 2026) classifies AI tools used in criminal investigation as high-risk systems subject to conformity assessment, logging, and human oversight requirements.

In India, the Bharatiya Sakshya Adhiniyam 2023 (which replaced the Indian Evidence Act 1872) governs electronic evidence. Section 63 requires that a certificate accompany electronic records, confirming authenticity and integrity, signed by a person responsible for the device or process.

Courts have accepted expert testimony on digital video authenticity under this framework, though the specific application of detection AI tools to deepfake evidence remains largely untested in published Indian case law as of mid-2026. Practitioners should also consider the Information Technology (Amendment) Act 2008 and the Digital Personal Data Protection Act 2023 when handling video evidence containing biometric facial data.