Audit Sampling Techniques and Working Papers

Statistical sampling gives every item in a defined population a known, nonzero probability of selection. Because selection is random and the sample size is calculated from probability theory, the auditor can make a statement of the form: we are 95 percent confident that the deviation rate in the population does not exceed 6 percent. That kind of precision is required by some frameworks and expected by most certification bodies reviewing SOC 2 or ISO 27001 audit evidence.

The most common statistical method in security auditing is attribute sampling, which tests each item for the presence or absence of a control characteristic. A random sample of 60 access-review records is tested: each record either has a documented supervisor approval or it does not. The number of deviations found is counted, the upper error limit is calculated from published tables, and that limit is compared to the tolerable deviation rate set during planning.

Random number generation for selection can use a random number table, a spreadsheet RAND() function, or purpose-built audit software such as ACL, IDEA, or the open-source equivalents. The selection method must be documented in the working paper. Systematic sampling (selecting every nth item) is sometimes used as a practical substitute for pure random sampling but carries a risk if the population has a periodic pattern that aligns with the interval.

Judgement sampling is appropriate when the population is small enough that statistical projection adds no practical value, when testing is targeted at specific high-risk items rather than the population as a whole, or when time and budget constraints make a full statistical approach impractical. It is also the default when the auditor's goal is to find at least one example of a specific failure pattern rather than to estimate an overall error rate.

Common judgement selection criteria in security audits include: items with the highest value or sensitivity, items processed immediately after a system change or personnel change, items from a period when a known control weakness existed, and items selected to cover the full range of transaction types or system components. A firewall rule audit might use judgement to select the 30 most recently added rules, the 10 rules with the widest open ports, and all rules touching a PCI DSS in-scope segment, rather than a random sample of all 4,000 rules.

For attribute sampling, three inputs drive sample size: the confidence level, the tolerable deviation rate, and the expected deviation rate. A higher confidence level requires a larger sample. A lower tolerable deviation rate requires a larger sample. A higher expected deviation rate also requires a larger sample because the auditor needs more data to distinguish systematic failure from random variation. ISACA's IT Audit Framework and the AICPA's audit sampling guide both publish tables that translate these three inputs into sample sizes without requiring the auditor to compute the underlying hypergeometric or Poisson formula directly.

The population size has almost no effect on sample size when the population exceeds 500 items. This is counterintuitive but mathematically sound: a random sample of 59 items gives essentially the same statistical power whether the population is 1,000 or 1,000,000 items. Population size only matters when the population is very small (under roughly 200 items), in which case the finite population correction factor reduces the required sample.

After selecting and testing the sample, the auditor calculates the upper error limit using published tables or software. If zero exceptions are found in a sample of 59 at 95 percent confidence and a 5 percent tolerable rate, the UEL is approximately 5 percent, which just meets the threshold. If one exception is found, the UEL rises to approximately 8 percent, exceeding the tolerable rate and requiring the auditor to expand testing or report a finding.

A working paper for a sampled audit procedure must contain enough information that a qualified reviewer who was not present can reconstruct exactly what was done and understand why the conclusion follows from the evidence. That standard, sometimes called the knowledgeable third party test, is the benchmark used by ISO 27001 certification bodies, AICPA peer reviewers, and PCI DSS qualified security assessors when reviewing audit documentation.

• Header information: audit name, control or objective being tested, audit period, auditor name, date of testing, and reviewer name and date.
• Population definition: the source of the population (which system, which log file, which date range), the total population count, and any items excluded and why.
• Sampling plan: method chosen (statistical or judgement), confidence level and tolerable rate if statistical, selection rationale if judgement, and sample size with the basis for that size.
• Test procedure: a step-by-step description of what was done to each selected item and what evidence was gathered.
• Results: a listing of items tested with a pass/fail or exception note for each, the total exception count, and any follow-up on individual exceptions.
• Conclusion: the auditor's assessment of control effectiveness based on the results, cross-referenced to the audit criteria (such as the specific ISO 27001 Annex A control or NIST CSF subcategory).

Working papers reference the audit process and must be cross-indexed to the overall audit file so that a reader can trace each finding in the report back to the specific working paper and from there to the original evidence. Electronic working paper tools such as TeamMate+, AuditBoard, or IBM OpenPages maintain this indexing automatically. Manual paper files require the auditor to apply consistent referencing codes.

An exception is any sampled item that does not meet the audit criterion. Finding an exception does not automatically mean reporting a finding: the auditor must determine whether the exception is isolated (an anomaly in an otherwise effective control) or systematic (a pattern indicating the control is not operating as designed). This determination requires investigation: reviewing the circumstances of the exception, checking whether it was already known and reported by management, and, where time permits, testing additional items from the same period.

When the projected error rate exceeds the tolerable deviation rate, the auditor's options are to expand the sample (which may reduce the UEL if no further exceptions are found), to treat the control as not effective and report accordingly, or to identify compensating controls that reduce the risk despite the failure of this specific control. The last option requires additional testing of the compensating control and careful documentation of the compensating-control reliance.

Working paper retention requirements differ by framework. ISO 27001 clause 9.2 requires the organisation to retain documented information as evidence of the audit programme and results but does not specify a minimum period. Most certification bodies expect at least three years of audit records so that surveillance audits and recertification audits can be compared against prior cycles.

India's Digital Personal Data Protection Act 2023 (DPDP Act) requires Data Fiduciaries to implement and demonstrate reasonable security safeguards. The Act does not prescribe audit retention periods directly, but the accountability principle embedded in the Act means that organisations must be able to produce evidence of compliance on request from the Data Protection Board. In practice, organisations subject to the DPDP Act are aligning their audit retention with a minimum of three years, consistent with the anticipated enforcement cycle.

In the United Kingdom, the ICO's Accountability Framework expects organisations to maintain records of their data protection impact assessments and security reviews. In the United States, sector-specific rules apply: HIPAA requires six years, the Gramm-Leach-Bliley Act's Safeguards Rule requires organisations to retain the records of their information security programme, and the FTC has taken enforcement action against organisations that could not produce audit documentation. The general principle across jurisdictions is that the retention period should exceed the longest plausible enforcement limitation period.