Detection Sources and Alert Pipelines

A SIEM receives log and event data from sources across the environment: firewalls, operating systems, directory services, web proxies, email gateways, cloud services, and applications. Its first function is normalisation: each source writes logs in its own format, so the SIEM parses incoming data into a common schema, mapping fields such as source IP, destination IP, username, timestamp, and action to consistent field names regardless of which source generated the record. Normalisation is what makes cross-source correlation possible. Without it, a rule that links a firewall block event to a Windows login failure event cannot be written, because those events use different field names and different timestamp formats in their raw form.

Once data is normalised, the SIEM applies correlation rules. A simple rule might fire an alert when more than ten failed login attempts occur from the same source IP within five minutes. A more complex rule might fire when a failed-VPN-authentication event is followed within thirty minutes by a successful authentication from the same account from a geographically distant IP, which then accesses a file server not previously accessed by that account. These multi-step rules catch attack sequences that no single event would reveal individually.

Modern SIEM platforms supplement rule-based correlation with machine-learning anomaly detection. User and Entity Behaviour Analytics (UEBA) builds a baseline of normal behaviour for each user and device, then generates an alert when observed behaviour departs significantly from that baseline, without requiring a rule that anticipates the specific deviation. UEBA is particularly useful for detecting insider threats and novel attack techniques that signature-based rules would miss, but it requires a sufficient history of baseline data and generates more ambiguous alerts that demand more analyst judgement.

An EDR agent runs as a privileged process on each endpoint and captures a continuous stream of telemetry: every process that starts or stops, every file created, modified, or deleted, every network connection opened or closed, and every registry key written. This telemetry is transmitted to the EDR's cloud or on-premises backend, where detection logic runs. The volume of EDR telemetry is large enough that the backend applies its own filtering and aggregation before surfacing alerts.

EDR detection logic operates in two modes. Signature-based detection matches process hashes, file hashes, or command-line strings against a database of known malware. It is fast and reliable for known threats but blind to novel variants and living-off-the-land techniques where attackers use legitimate system tools. Behaviour-based detection looks for patterns: a process that injects code into another process, a script interpreter spawned by an Office document, or a service account opening a large number of files in rapid succession. Behaviour-based detection catches novel techniques but generates more false positives, requiring tuning and analyst judgment.

EDR tools also support live response: an analyst can remotely query a suspicious endpoint for running processes, open network connections, and recently created files without requiring physical access or a separate forensic tool. Some EDR platforms allow the analyst to isolate the endpoint from the network (cutting all connections except the management channel to the EDR backend) directly from the analyst console. This containment capability is important in the early stages of an incident, but the decision to isolate must be made carefully: isolating the wrong system can disrupt operations and, if the attacker has a kill switch, can trigger destructive activity.

Network intrusion detection systems (NIDS) inspect traffic passing through a network segment, typically by receiving a copy of traffic via a switch mirror port or network tap. They compare packet content and connection metadata against a ruleset of known attack signatures. Snort and Suricata are the two most widely deployed open-source NIDS engines; commercial alternatives include Palo Alto Networks Threat Prevention and Cisco Firepower. A NIDS alert identifies the source and destination IP, port, protocol, and the rule that fired, giving an analyst a starting point for investigation.

A network intrusion prevention system (NIPS) is a NIDS placed inline in the traffic path rather than receiving a mirror copy. Because it sits between sender and receiver, it can drop a packet, reset a TCP connection, or block a source IP when a rule fires. This gives the NIPS the ability to stop an attack in real time, but it also means a misconfigured or overly aggressive rule can disrupt legitimate traffic. Most organisations run new rules in detection-only mode for a period of observation before enabling prevention, and they maintain a documented procedure for disabling a prevention rule quickly if it begins blocking legitimate activity.

Network traffic analysis (NTA) tools do not rely on signatures. Instead they establish a baseline of normal communication patterns: which hosts talk to which other hosts, on what protocols, at what volumes and times. Deviations from the baseline, such as an internal host suddenly communicating with an external IP it has never contacted before, or a host sending far more data outbound than it normally does, generate alerts. NTA is particularly effective at detecting command-and-control beaconing and data exfiltration, which may use common protocols and therefore evade signature-based IDS rules.

Cloud environments produce activity logs through provider-specific services. AWS records API calls via CloudTrail, network flow data via VPC Flow Logs, and DNS queries via Route 53 Resolver Query Logs. Microsoft Azure records management-plane activity via Azure Activity Log, data-plane events via Azure Resource Logs, and security signals via Microsoft Defender for Cloud. Google Cloud records admin and data access via Cloud Audit Logs. These logs are the raw material for cloud detection, and they must be explicitly enabled and routed to a SIEM or a cloud-native detection service; they are not automatically available.

Cloud providers also offer managed threat detection services that analyse their own telemetry internally and surface pre-built alerts. AWS GuardDuty processes CloudTrail, VPC Flow Logs, and DNS logs to detect threats such as credential compromise, cryptomining, and unusual API activity. Microsoft Defender for Cloud covers Azure, AWS, and GCP resources and produces security recommendations alongside threat alerts. Google Security Command Center provides centralised visibility and threat detection across GCP. These services require no log export or correlation rule authoring; they apply the provider's detection logic to the same telemetry they already hold.

Containerised and serverless environments present additional detection challenges. Containers are ephemeral: a compromised container may execute malicious code and be destroyed before any log is exported. Runtime security tools such as Falco (open-source) and commercial equivalents intercept kernel system calls from within the container host and generate alerts in real time, before the container terminates. Serverless functions similarly generate short-lived execution environments; function-level logging must be explicitly enabled and must be retained outside the function's execution context to survive its termination.

Threat intelligence feeds supply structured data about known malicious infrastructure and artefacts. A feed entry might identify an IP address as a known command-and-control server, a domain as a phishing infrastructure host, a file hash as a ransomware executable, or a certificate fingerprint as associated with a known threat actor. When a SIEM rule or a network sensor observes traffic to or from a flagged IP, or when an EDR records execution of a file whose hash appears in the feed, a high-confidence alert fires without requiring a behavioural rule that would have anticipated the specific technique.

Feeds come from multiple sources. Commercial threat intelligence providers such as Recorded Future, Mandiant, and CrowdStrike Intelligence aggregate indicators from their own telemetry, from dark web monitoring, and from human intelligence. Open-source feeds include the Abuse.ch collections (MalwareBazaar, URLhaus, ThreatFox), the Cisco Talos IP reputation list, and feeds shared through sector-specific Information Sharing and Analysis Centers (ISACs) in financial services, healthcare, and critical infrastructure. Government feeds include the US CISA Known Exploited Vulnerabilities catalogue and the UK NCSC's threat indicator feeds shared with qualifying organisations.

The STIX/TAXII standard provides a common format (Structured Threat Information Expression) and transport protocol (Trusted Automated eXchange of Indicator Information) for sharing threat intelligence between organisations and platforms. Most modern SIEM platforms and many EDR tools support native STIX/TAXII ingestion, allowing a feed to be pulled automatically on a schedule and applied to detection rules without manual import. The value of a feed degrades quickly as indicators age: an attacker's command-and-control infrastructure changes frequently, so a feed that is updated daily is more useful than one updated weekly.

The path from a raw event to an analyst-ready alert passes through several processing stages. Collection is the first: agents, collectors, and API integrations pull or receive events from sources and forward them to the central platform. Normalisation parses each event into a common schema. Aggregation groups related low-level events before correlation runs; for example, grouping all events from the same source IP and destination host in a five-minute window reduces the number of objects the correlation engine must evaluate. Correlation then applies rules or model-based logic to the aggregated data and produces candidate alerts.

Deduplication removes redundant alerts before they reach the analyst. A single malware execution can trigger the same detection rule multiple times as the same process is logged from multiple sources, or as related child processes fire the same rule. Without deduplication, the analyst sees dozens of identical alerts instead of one. Deduplication logic groups alerts that share the same detection rule, source host, and time window into a single alert with a count. Enrichment then appends context: the asset record for the affected host (its owner, business function, and criticality), geolocation and reputation data for any external IP, and relevant threat intelligence context for any matched IOC.

The output of the pipeline is an alert delivered to a case management or Security Orchestration, Automation, and Response (SOAR) platform, or directly to the analyst queue in the SIEM. Well-designed pipelines also record provenance: which raw events contributed to each alert, what rule or model fired, and what enrichment was applied. This provenance chain is essential for the analyst to understand the full context of an alert and for the incident responder to collect the underlying evidence without retracing the pipeline from scratch. The triage and prioritisation step that follows depends on the quality of the alert the pipeline delivers.