Closed-Corpus Trap
Definition
The error of validating a stylometric method exclusively on texts from authors whose profiles were used to build the model, inflating apparent accuracy beyond what is achievable on truly unseen authors.
- Field
- Stylometry / authorship analysis
- Error type
- Validation design flaw
- Cause
- Testing only on known-author texts used to build the model
- Consequence
- Inflated accuracy claims
Common questions
Why does testing only on known candidate authors inflate accuracy so much?+
The model only ever has to choose among a fixed, familiar set of authors it was trained on, so it never has to handle an unseen author or the false-positive risk of forcing a match onto the wrong candidate, both of which are common in real casework.
How should a stylometry validation study avoid the closed-corpus trap?+
It should include held-out authors whose writing never appeared in training, and ideally test open-set scenarios where the true author may not be among the candidates at all, so the reported accuracy reflects performance closer to real forensic conditions.
Related terms
- Burrows's Delta
- A distance metric introduced by John Burrows in 2002 that measures how far a text's standardised word-frequency profile deviates from the mean...
- Cosine Delta
- A variant of Burrows's Delta that uses cosine similarity instead of Manhattan distance. Research by Smith and Aldridge (2011) and later Eder,...
- Cross-Validation
- A validation strategy that repeatedly partitions a labelled corpus into training and test subsets to estimate the method's error rate on unseen...
- Principal Component Analysis (PCA)
- A multivariate statistical method used in fire debris research to reduce chromatographic data matrices to principal components that capture major variance. Used...
- Rolling Delta
- Application of the Delta metric to a moving window of text, producing a curve showing how stylistic similarity to each candidate changes...