A new systematic review has identified widespread problems with data independence in forensic artificial intelligence research, raising concerns about whether reported machine-learning performance accurately reflects how systems would behave on unfamiliar evidence.
Published on 18 November 2026 in Frontiers in Analytical Science, the study examined 446 research reports involving artificial intelligence and machine-learning methods applied to complex forensic chemical, material and biological evidence.
The research was conducted by Paolo Bailo, Carlotta Virginia Di Francesco, Giulia Vignali, Riccardo Primavera, Andrea Piccinini and Lorenzo Franceschetti, affiliated with the University of Milan and University of Camerino in Italy.
Artificial intelligence is increasingly being investigated for applications such as forensic DNA analysis, biological identification, chemical evidence classification and interpretation of complex analytical measurements.
However, high accuracy reported during research does not necessarily demonstrate that a model will perform reliably on evidence encountered during actual investigations.
One major concern is data leakage, which occurs when information intended for independent testing influences model development or evaluation.
For example, multiple measurements originating from the same biological specimen may be incorrectly divided between training and testing datasets. This can make a model appear more accurate because it has already encountered closely related information.
The researchers systematically examined 446 reports containing 1,005 eligible evaluation regimes.
They identified source-supported data leakage in 284 reports, representing 63.7% of the reviewed literature.
Another 36 reports showed strong potential concerns, while 85 contained uncertain evaluation boundaries.
Only 41 reports, equivalent to 9.2%, were classified as having no identified leakage concerns under the review's assessment framework.
The researchers also examined external validation, which evaluates models using data outside their original development conditions.
Although 278 reports included external or shifted evaluation, only 94 contained at least one evaluation meeting all five criteria established by the researchers for assessing an unchanged model pipeline.
These findings highlight the importance of separating training and testing information, maintaining independence between related samples and evaluating models under conditions relevant to forensic casework.
However, the review does not establish that 63.7% of forensic AI systems produce incorrect results in criminal investigations.
Its findings concern documented methodological weaknesses in published evaluations, rather than direct measurements of operational error rates.
The review also covered selected complex forensic chemical and biological applications, not every form of forensic artificial intelligence.
Furthermore, some publications contained both problematic and independently evaluated experimental procedures.
The study emphasizes that trustworthy forensic AI requires more than impressive accuracy figures. Transparent validation, independent testing and careful assessment of generalizability are essential before research findings can support claims about real-world forensic performance.