Statistical Comparison and the Reference Population Problem

Forensic comparison always asks a two-part question: are these samples similar, and if so, is that similarity unusual or commonplace? The first part is chemistry and mineralogy. The second part is statistics, and it depends entirely on knowing how the sample compares to what else is out there.

Soil makes the second question genuinely hard. Geochemical composition can vary dramatically over a few metres where a soil boundary crosses a geological contact, a drainage channel, or a fill deposit. That variability gives soil its discriminating power: two samples taken from different locations can often be distinguished. But it also means the reference population that represents 'anywhere else' must be carefully defined and sampled. A poorly designed reference collection can both inflate and deflate the apparent rarity of a match.

Published studies have examined how misclassification rates change with reference-population design. Morgan and Pringle (2012) tested soil discrimination across a range of English landscapes and found that misclassification rates ranged from under 5% in geologically diverse areas to over 20% in geologically uniform ones. Pye and Blott have demonstrated similar variation across different analytical protocols. These are not theoretical concerns. They translate directly into the strength of a court opinion.

A modern soil analysis by inductively coupled plasma mass spectrometry (ICP-MS) can measure thirty or more elements in a single sample. Each element is a dimension, and a case involving a questioned sample and a hundred reference samples is a cloud of points in thirty-dimensional space. No human can visualise that. PCA is the standard tool for collapsing it into something interpretable.

PCA computes new axes, called principal components, that are linear combinations of the original variables and are ordered by how much variance they capture. The first principal component might account for 45% of the total variation in a geochemical dataset, the second another 20%, and so on. By plotting samples along the first two or three components, an analyst can see whether the questioned sample clusters with the reference samples from the suspect's alleged location or sits apart.

LDA goes one step further. Where PCA is unsupervised (it does not know which samples belong to which group), LDA is trained on labelled samples to find the axis that maximally separates known classes. If an analyst has well-characterised samples from several distinct source areas and wants to classify a questioned sample into one of them, LDA provides a principled way to make that assignment along with a posterior probability.

The likelihood ratio is now the preferred reporting framework in forensic science across multiple disciplines, including soil comparison. Its appeal is that it makes explicit what question the science can answer and what question it cannot. The scientist answers: given the observed geochemical and mineralogical data, how much more probable is this degree of similarity under the hypothesis that both samples came from the same location than under the hypothesis that they came from different locations? The court, not the scientist, then weighs that number against the rest of the evidence.

Formally, the LR is the ratio of two probabilities. The numerator is the probability of the observed data if the prosecution's hypothesis is correct (same source). The denominator is the probability of the observed data if the defence hypothesis is correct (different source). The denominator requires sampling the reference population to estimate how often soils from unrelated locations look as similar as the two in question. This is exactly where the reference population design matters most.

LR values in soil cases reported in the literature range widely, from modest values around 10-100 in geologically uniform areas to values exceeding 100,000 in cases where the questioned soil contains a highly distinctive mineralogical assemblage. The ENFSI verbal equivalence scale (discussed in the next topic) provides a way to communicate these magnitudes in plain language without misrepresenting the precision of the estimate.

There is no universal rule for how large a reference population must be or how it should be sampled. The answer depends on the case geography, the geological diversity of the area, and the analytical methods used. A case where the questioned site is in the middle of a geologically uniform glacial plain needs a different approach from one where the site sits on a narrow band of serpentinite surrounded by quite different geology.

1. Define the relevant area
   Starting from the known facts of the case, identify the geographic area that represents plausible alternative sources. This might be a radius around the crime scene, a transit corridor, or an urban catchment area.
2. Sample systematically
   A grid or random stratified sample across the defined area gives a reference collection that reflects the geological diversity within it. Opportunistic sampling biases the denominator toward accessible locations.
3. Analyse identically
   Reference samples must be prepared and analysed using exactly the same protocol as the evidential samples. Differences in preparation introduce analytical variance that is indistinguishable from real geochemical differences.
4. Validate the distribution
   Check that the reference collection captures the range and frequency distribution of the analytical parameters. If the questioned sample falls at the extreme of the reference distribution, the LR estimate in that region is uncertain.

The gap between what is ideal and what is achievable in casework is real. A forensic geologist may receive samples months after an incident, by which time seasonal changes have altered surface chemistry and the ability to sample the exact relevant area may be limited by access restrictions or development. The report must be honest about these limitations and what they mean for the precision of the LR estimate.

One of the Daubert criteria for admissibility of expert scientific testimony is a known or knowable error rate. For geological soil evidence, this means false-positive rates (two samples from different sources classified as matching) and false-negative rates (two samples from the same source classified as non-matching) must be estimated from empirical testing, not asserted from first principles.

Published validation studies provide benchmarks. Pye and Blott conducted blind trials comparing colour, particle size, and geochemical methods across diverse English landscapes and reported correct classification rates of 80-95% depending on method and geology. Morgan and Pringle used a known-source soil set from English agricultural and urban environments and found that multi-element ICP-MS data with LDA gave the lowest false-positive rates, typically below 10% in geologically diverse regions.

• Geologically uniform areas (e.g., glacial till plains) produce higher false-positive rates because soils over a large area look chemically similar.
• Geologically diverse areas (e.g., mineralised terrains, coastal margins with varied inputs) produce lower false-positive rates because even nearby sources differ.
• Method matters: colour matching alone has higher error rates than combined geochemical and mineralogical profiling.
• Seasonal and weather-related variation can shift surface chemistry, increasing apparent false-positive rates when samples are collected at different times.

Statistical outputs from PCA, LDA, or LR calculations are not self-explanatory to a jury or a judge with no statistical training. The forensic geologist's job is to translate them honestly without either overstating the precision of the estimate or understating the weight of genuine evidence.

The LR framework helps because it separates the scientific assessment from the legal conclusion. The scientist says: the observed similarity is X times more probable if both samples came from the same location than if they came from different locations. The court then combines that with everything else it knows about the case. The scientist does not say 'these samples match' as though that settles the question, nor do they say 'it is probable that the suspect was at the scene,' which conflates the LR with a posterior probability.