Cross-Validation
Definition
A validation strategy that repeatedly partitions a labelled corpus into training and test subsets to estimate the method's error rate on unseen data. A key safeguard against over-fitting.
- Purpose
- Estimate a method's error rate on unseen data
- Method
- Repeated partitioning of a labelled corpus into train and test sets
- Guards against
- Over-fitting to the reference corpus
- Applies to
- Stylometry and statistical distance methods
Common questions
Why is over-fitting a specific concern in forensic stylometry?+
A model can appear highly accurate if it is tested on the same texts it learned from, because it may have picked up quirks of that particular corpus rather than genuine authorship signal. Cross-validation exposes this by testing on partitions the model never saw during training.
Does a good cross-validation score guarantee the method will work on a specific court exhibit?+
No. It only shows the method performed well on texts similar to the validation corpus. A questioned document that differs in length, genre, or language register from that corpus can still fall outside the conditions the validation actually covered.
Related terms
- Burrows's Delta
- A distance metric introduced by John Burrows in 2002 that measures how far a text's standardised word-frequency profile deviates from the mean...
- Closed-Corpus Trap
- The error of validating a stylometric method exclusively on texts from authors whose profiles were used to build the model, inflating apparent...
- Cosine Delta
- A variant of Burrows's Delta that uses cosine similarity instead of Manhattan distance. Research by Smith and Aldridge (2011) and later Eder,...
- Principal Component Analysis (PCA)
- A multivariate statistical method used in fire debris research to reduce chromatographic data matrices to principal components that capture major variance. Used...
- Rolling Delta
- Application of the Delta metric to a moving window of text, producing a curve showing how stylistic similarity to each candidate changes...