Security Metrics and Continuous Monitoring

A security metric is only useful if it is measurable, attributable to a specific control or risk, and actionable: a manager who sees it can decide to do something differently. Metrics that are vague, unmeasurable, or disconnected from any decision are reporting overhead. The discipline of metric design starts by asking: what control or risk does this metric describe, what is the target state, and what threshold triggers a response?

KRIs and KPIs serve different purposes but are often confused. A KRI answers the question: how exposed are we? It measures risk level or trend, and it is most useful when it predicts future problems rather than recording past ones. A high volume of failed authentication attempts against privileged accounts is a KRI for credential-based attack risk: it is rising before a breach, giving the security team time to investigate. A KPI answers the question: how well is our control working? The percentage of privileged accounts with multi-factor authentication enabled is a KPI for the MFA control: it tells you directly how completely the countermeasure is deployed.

Good metric design follows the SMART criteria applied to security: Specific (it measures one control or one risk, not a vague concept), Measurable (the data source and calculation method are defined), Achievable (a realistic target exists), Relevant (it links to a risk in the register or a control objective), and Time-bound (it is measured at a defined frequency). A metric such as 'security is improving' fails all five. A metric such as 'percentage of critical-severity findings from the last penetration test closed within 30 days' passes all five.

NIST SP 800-137 defines Information Security Continuous Monitoring as maintaining ongoing awareness of information security, vulnerabilities, and threats to support organisational risk management decisions. The framework structures a monitoring programme into six steps: define the programme strategy, establish monitoring procedures, implement the programme, analyse the data and report findings, respond to findings, and review and update the programme. The six steps form a cycle, not a one-time project.

Monitoring frequency should match the criticality and volatility of what is being measured. A firewall rule configuration that changes only when an authorised change request is approved needs to be checked after each change and on a periodic schedule to detect unauthorised drift. A vulnerability scanner should run against internet-facing assets more frequently than against an air-gapped development network. The programme design documents what is monitored, at what frequency, by what automated or manual mechanism, and who receives the output.

ISO/IEC 27001:2022 clause 9 (Performance evaluation) requires the organisation to determine what needs to be monitored and measured, the methods for analysis and evaluation, when monitoring and measurement shall be performed, and who shall perform it. Annex A control 8.16 requires monitoring of networks, systems, and applications. These requirements map directly onto the outputs of a well-designed monitoring programme: the programme becomes the primary source of evidence for clause 9 compliance.

The programme also needs to handle the gap between what can be monitored automatically and what requires manual checks. Configuration of some legacy systems cannot be pulled by an automated scanner. Physical controls such as server room access logs may need manual aggregation. A mature programme documents these gaps explicitly and designs compensating manual checks to cover them, rather than leaving them unmonitored because automation is not available.

A SIEM platform performs two core functions: log aggregation and event correlation. Log aggregation collects raw event data from firewalls, endpoint detection and response (EDR) tools, identity providers, cloud control planes, databases, and applications, normalises the data into a common format, and stores it for querying. Event correlation applies rules and statistical models to the aggregated data to identify patterns that indicate a security event: multiple failed logins followed by a successful one from the same source address, for example, or a privileged account active outside business hours in a region where no employees are located.

For auditors, a SIEM generates several categories of evidence. Alert logs show that the monitoring system detected specific events and created a record. Incident response records show that alerts were investigated and resolved within policy timeframes. Report outputs show the volume and type of events over a defined period. Search queries run against the SIEM can answer specific audit questions: 'Were there any privileged account logins outside business hours in the past 90 days, and if so were they all authorised?' A SIEM that cannot answer this kind of query either lacks the log sources or lacks the retention period to cover the audit window.

Common SIEM platforms in enterprise use include Splunk Enterprise Security, Microsoft Sentinel (Azure-native), IBM QRadar, Google Chronicle, and Elastic SIEM. Open-source options such as Wazuh have grown substantially in capability and are widely deployed by smaller organisations and by managed security service providers (MSSPs) serving clients in India, Southeast Asia, and sub-Saharan Africa where per-device licensing costs of commercial platforms are prohibitive. Auditors should evaluate the platform's capabilities against the monitoring requirements, not assume that a named commercial product automatically satisfies those requirements.

A compliance dashboard aggregates metric and control-status data and presents it in a format aligned to one or more regulatory or framework requirements. Modern cloud security platforms include pre-built compliance dashboards for common frameworks: AWS Security Hub maps findings to PCI-DSS, CIS Controls, NIST CSF, and others. Microsoft Defender for Cloud provides a Secure Score and regulatory compliance views aligned to ISO/IEC 27001, SOC 2, and GDPR. Third-party GRC (Governance, Risk, and Compliance) platforms such as ServiceNow GRC, MetricStream, and RSA Archer provide dashboard layers across mixed on-premises and cloud environments.

The value of a compliance dashboard is the speed and breadth of visibility it provides. Instead of assembling evidence manually for each control at audit time, a well-configured dashboard shows control status continuously, flags controls that have drifted out of compliance, and provides drill-down into the underlying evidence. An auditor examining a PCI-DSS audit trail can look at the dashboard to see which requirements show as compliant, then verify a sample of the underlying evidence to confirm that the dashboard's status reflects reality.

Auditors must verify four properties of any compliance dashboard they use as an evidence source. First, coverage: does the dashboard monitor all controls in scope, or only the ones that happen to have automated data feeds? A dashboard that shows 100% compliance on 40% of controls while the remaining 60% are not measured gives a false picture. Second, accuracy: are the underlying data sources reliable, complete, and tamper-evident? Third, threshold governance: are the green/amber/red thresholds formally approved, documented, and reviewed? Fourth, exception management: when a control shows amber or red, is there a documented process for investigation and remediation, and is that process being followed?

Multiple regulatory frameworks contain explicit monitoring requirements. Understanding which requirement applies, and what evidence it calls for, is essential for both compliance teams building monitoring programmes and auditors assessing them.

The CERT-In Directions of April 2022 are worth noting for the detection infrastructure they implicitly require. A six-hour reporting window for a cyber incident from the moment of detection means that automated detection must already be in place: a manual log review process that runs once a day cannot meet a six-hour window. Organisations subject to CERT-In Directions therefore need a functioning SIEM or equivalent automated detection capability as a compliance matter, independent of any other security justification.

The UK NCSC Cyber Essentials Plus certification requires verified monitoring of malware defences and patch status. The EU NIS2 Directive (effective October 2024) requires entities in critical sectors to implement monitoring and detection capabilities proportionate to their risk exposure. These requirements are converging toward a common expectation: organisations operating at meaningful scale must have automated monitoring in place and must be able to produce evidence of its operation on demand.

An auditor assessing a continuous monitoring programme is asking a two-part question: is the programme designed to provide adequate assurance, and is it actually operating as designed? Design adequacy covers whether the metrics are relevant, whether the monitoring frequency is appropriate, and whether the monitoring covers all in-scope assets and controls. Operating effectiveness covers whether the monitoring is actually running, whether alerts are being investigated, and whether the outputs are feeding management decisions.

The audit approach for a monitoring programme typically includes: reviewing the programme documentation (strategy, scope, metrics definitions, thresholds, reporting cadence), examining a sample of metric reports to verify that data was collected and reported on schedule, selecting a sample of alerts or threshold breaches and tracing them through the investigation and response process, verifying that the SIEM or other monitoring platform covers the agreed log sources with no critical gaps, and confirming that retention periods meet applicable regulatory requirements.

A common finding in monitoring programme audits is the gap between alert generation and alert investigation. A SIEM may be generating hundreds of alerts per day, but if the security operations team has only enough capacity to investigate a fraction of them, the monitoring programme is providing false assurance: it looks active but is not delivering the intended detection capability. Auditors look for evidence of alert triage processes, mean-time-to-investigate metrics, and capacity versus alert volume ratios. SOC 2 Type II assessors pay particular attention to this because the opinion covers the operating effectiveness of controls over a period, not just their design.