Skip to content

Traffic Analysis and Protocol Dissection

Traffic analysis and protocol dissection are the techniques used to interpret raw packet captures, reconstruct network sessions, and recover transferred files or credentials from PCAP data. This topic covers display filters, stream reassembly, artefact carving from captures, and the detection of anomalous behaviour in network traffic.

Last updated:

Share

Traffic analysis and protocol dissection describe the process of examining a packet capture file to reconstruct what happened on a network: which hosts communicated, which application protocols were in use, what data was transferred, and whether any activity was anomalous. A packet capture (PCAP) file is a time-ordered record of raw network frames. Protocol dissectors, built into tools such as Wireshark, tcpdump, and Zeek, decode the layered headers of each frame to expose IP addresses, ports, protocol fields, and application-layer payloads. From this decoded data, an analyst can reassemble TCP streams into readable conversations, export transferred files, and identify credentials, commands, or malware communications embedded in the traffic.

Network forensics differs from host forensics in one important respect: packets in transit leave no footprint on any single device unless a sensor is already recording them. This means the evidentiary value of a PCAP depends entirely on where and when the sensor was placed. A full-packet capture at a network chokepoint is the richest source, but many investigations work from NetFlow records, firewall logs, or partial captures from intrusion detection sensors. Understanding what each source can and cannot show shapes every analytical decision that follows.

Jurisdictions increasingly treat packet captures as electronic records subject to chain-of-custody requirements. In the United States, network evidence is governed by the Electronic Communications Privacy Act and relevant case law on the third-party doctrine. In the European Union, the e-Evidence Regulation and member-state procedural law define how network intercept data is admitted. India's Bharatiya Sakshya Adhiniyam 2023 and the Information Technology Act 2000 (amended 2008) provide the framework for admissibility of electronic evidence. Regardless of jurisdiction, capturing network traffic without lawful authority is itself an offence in most legal systems, and the analyst must confirm that a valid legal basis exists before collecting or analysing any capture.

By the end of this topic you will be able to:

  • Explain the difference between capture filters and display filters and identify the safer choice for forensic collection.
  • Use stream reassembly to reconstruct a TCP session from individual packets and identify the application-layer content it carried.
  • Describe how to carve files, credentials, and other artefacts from a PCAP file using Wireshark's Export Objects and NetworkMiner.
  • Apply statistical and behavioural indicators to detect anomalous traffic patterns such as beaconing, data exfiltration, and port scanning.
  • Explain the limitations of PCAP analysis when traffic is encrypted and identify the conditions under which TLS session key logging enables decryption.
Key terms
PCAP
Packet capture file. The standard format for storing captured network frames, originally defined by the libpcap library. Each record contains the raw frame bytes, a timestamp, and the captured length. The newer pcapng format extends this with interface metadata and block annotations.
Protocol dissector
A software component in a packet analyser that recognises a specific protocol and parses its header fields into named, readable values. Wireshark includes hundreds of built-in dissectors; analysts can also write custom dissectors in Lua for proprietary or custom protocols.
Stream reassembly
The process of collecting all the TCP segments belonging to a single connection and reordering them by sequence number to reconstruct the complete application-layer payload. Required because TCP segments may arrive out of order or be fragmented across multiple packets.
Display filter
A Wireshark filter expression applied to an already-captured PCAP file to show only packets matching specified criteria. Display filters do not delete non-matching packets, preserving the full capture. Syntax differs from BPF capture filter syntax.
Beaconing
A traffic pattern in which a host contacts the same remote endpoint at regular intervals. Associated with command-and-control communication by malware, which checks in with an attacker's server on a timer. Detected by plotting connection frequency against destination IP or hostname.
Artefact carving
Extracting embedded content from raw data by locating known file signatures (magic bytes) at byte boundaries. In PCAP analysis, this means reassembling the application payload of a session and extracting any complete files whose headers and trailers appear in the stream.

Capture filters vs. display filters

Wireshark and tcpdump both offer two filtering mechanisms that operate at different points in the workflow. Capture filters, written in Berkeley Packet Filter (BPF) syntax, are applied by the capture engine before any packet is written to disk. A packet that does not match the capture filter is discarded immediately and never recorded. Display filters are written in Wireshark's own filter expression language and applied to a PCAP file that already exists; they hide non-matching packets from view but do not remove them from the file.

FeatureCapture filter (BPF)Display filter (Wireshark)
Applied atCollection time, before writingAfter capture, on existing file
SyntaxBPF (e.g. tcp port 80)Wireshark expression (e.g. http)
Effect on discarded packetsPermanently lostHidden, still present in file
Forensic riskEvidence may be excludedNo evidence loss
Performance benefitReduces file size and CPUMinimal; file is already written
Typical forensic useHigh-volume links where full capture is impracticalNavigation and analysis of a complete capture

For forensic investigation, the default is to capture everything and use display filters to navigate. A capture filter that excludes traffic matching no known IOC today may exclude the pivot traffic discovered tomorrow. The exception is high-bandwidth environments where full-packet capture is technically impractical: on a multi-gigabit production link, even short capture windows produce files too large for most tooling. In those cases, a capture filter scoped to specific hosts or ports is acceptable provided the scope is documented and the legal authority covers it.

Stream reassembly and session reconstruction

A TCP connection is identified by its four-tuple: source IP, source port, destination IP, destination port. Packets belonging to the same connection carry the same four-tuple and a sequence number that indicates where each segment fits in the stream. Reassembly involves collecting all matching segments, reordering them by sequence number, and concatenating their payload bytes to recover the application-layer data.

In Wireshark, right-clicking any packet in a TCP session and choosing Follow > TCP Stream opens a window showing the reassembled application data for that connection. Sent data (client to server) is shown in one colour; received data (server to client) in another. The raw stream can be saved to disk as binary or text. This is sufficient to read cleartext HTTP requests and responses, FTP command sessions, SMTP email transactions, and any other protocol carried over TCP without encryption.

UDP does not have a connection concept or guaranteed ordering. For UDP-based protocols such as DNS, DHCP, and many VoIP protocols, the analyst works at the individual datagram level rather than a reassembled stream. Wireshark's protocol dissectors for DNS decode each query and response pair. The Statistics > DNS menu item in Wireshark aggregates DNS queries across the whole capture, listing every queried name with query counts, which is useful for identifying beaconing or DNS tunnelling.

Artefact carving: files, credentials, and documents

Once application-layer payloads are accessible through stream reassembly, any files transferred in cleartext can be recovered. Wireshark's File > Export Objects menu presents a list of files identified within specific protocols: HTTP objects (images, scripts, documents, executables), SMB/SMB2 objects, FTP data objects, TFTP and DICOM objects. Selecting an entry and clicking Save writes the file to disk in its original binary form. The recovered file can then be hashed and examined like any other forensic artefact.

NetworkMiner is a purpose-built network forensics tool that automates file carving from PCAP files across a wider range of protocols. It reconstructs files from HTTP, FTP, TFTP, SMB, and IMAP sessions automatically on import, presenting them in a file browser organised by source host. It also extracts credentials from cleartext authentication exchanges: HTTP Basic Auth headers, FTP USER/PASS sequences, POP3 and IMAP login commands, and SMTP AUTH exchanges.

Credential extraction from cleartext protocols is straightforward because the dissector exposes the relevant fields directly. HTTP Basic Authentication encodes the username and password as Base64 in the Authorization header; decoding the Base64 string recovers the plaintext. FTP authentication appears as explicit USER and PASS commands in the TCP stream. Telnet sessions carry every keystroke in cleartext, including login sequences. These credentials have evidential value in investigations involving account compromise, insider threat, or lateral movement.

Detecting anomalous behaviour in traffic

Anomaly detection in network traffic combines statistical analysis of timing and volume with protocol-level inspection of content. The most common patterns of interest in forensic investigations are port scanning, beaconing, lateral movement, and data exfiltration.

Port scanning produces a characteristic pattern: many SYN packets from a single source to sequential or randomised destination ports on one or more hosts, with corresponding RST or ICMP unreachable responses from closed ports. In Wireshark, filtering for 'tcp.flags.syn == 1 and tcp.flags.ack == 0' isolates SYN packets. A high volume of such packets from one source within a short window identifies the scanner. Nmap's default SYN scan, SYN/ACK scan, and UDP probe patterns each produce recognisable signatures at the packet level.

Beaconing is detected by grouping outbound connections by destination and plotting connection intervals. A host infected with a command-and-control implant will contact its controller at a regular interval, often with small jitter. Tools such as Zeek (formerly Bro) produce connection logs that make this analysis faster than reading raw PCAPs: the conn.log lists every connection with timestamps, making interval analysis a straightforward statistical operation. JA3 and JA3S fingerprints, derived from TLS ClientHello and ServerHello parameters, identify the TLS client library in use and can match known malware families even when the server IP changes.

Data exfiltration leaves two main signatures. Volume-based exfiltration produces an unusually large outbound data transfer from a host that does not normally generate such volumes, visible in NetFlow records or in Wireshark's Statistics > Endpoints and Statistics > Conversations views. Protocol-based exfiltration abuses a permitted protocol, most commonly DNS, to carry data out of a network that blocks most outbound connections. DNS tunnelling produces DNS queries with unusually long, randomly-looking subdomain labels and high query rates to a single authoritative nameserver. The query name itself contains encoded data.

Encrypted traffic: TLS, limitations, and key logging

TLS encrypts application-layer payloads between the TCP handshake and the connection teardown. A PCAP of a TLS session shows the handshake (ClientHello, ServerHello, Certificate, ChangeCipherSpec, Finished) and then opaque encrypted records. Without the session keys, the application data cannot be recovered from the PCAP alone. The RSA private key of the server was historically sufficient to decrypt some older cipher suites, but cipher suites using ephemeral Diffie-Hellman key exchange (ECDHE, DHE) provide forward secrecy: even knowing the private key does not recover session keys.

The SSLKEYLOGFILE mechanism provides a practical path to decryption in controlled environments. When the SSLKEYLOGFILE environment variable is set on a host running a supported browser or application (Chrome, Firefox, and curl all support it), the TLS library writes the session keys for each connection to a file as they are negotiated. Loading this key log file into Wireshark via Edit > Preferences > Protocols > TLS decrypts the matching sessions in any PCAP that captured that traffic. This technique is used during controlled tests and in investigations where the suspect host is under the investigator's control.

Encrypted DNS (DNS over HTTPS, DNS over TLS) removes the previously reliable DNS query visibility. Where an investigation depends on DNS query logs as evidence of contacted domains, the analyst must work from resolver logs if the internal DNS resolver was configured to log, or from endpoint logs (browser history, application logs) rather than network captures. This shift is one reason endpoint forensic data is increasingly complementary to network forensic data rather than a substitute for it.

Check your understanding
Question 1 of 4· 0 answered

An analyst wants to examine a PCAP capture of an incident without risking any loss of packets. Which filtering approach should they use, and at what stage?

Key Takeaways

  • Capture filters discard non-matching packets permanently; display filters hide them without deletion. For forensic work, capture everything and navigate with display filters unless bandwidth makes full capture impractical.
  • Stream reassembly via Follow TCP Stream reconstructs the complete application-layer conversation from individual segments, making cleartext protocols such as HTTP, FTP, and SMTP directly readable and enabling credential and file extraction.
  • Wireshark's Export Objects and NetworkMiner both carve transferred files from PCAP sessions; every carved file should be hashed immediately and linked to the source PCAP hash in the chain-of-custody record.
  • Anomalous traffic patterns including port scanning, beaconing, lateral movement, and DNS tunnelling each have recognisable packet-level signatures that display filters and statistical views can isolate.
  • TLS with forward-secret cipher suites cannot be decrypted from a PCAP alone; the SSLKEYLOGFILE mechanism provides session keys in controlled environments, while TLS metadata (SNI, JA3, certificates) remains available without decryption.
What is the difference between traffic analysis and protocol dissection?
Traffic analysis examines patterns, volumes, timing, and endpoints in captured network data to draw investigative conclusions. Protocol dissection parses the specific fields of a network protocol within each packet, such as HTTP headers, DNS query names, or FTP commands, to extract application-layer content. In practice both tasks run together: a forensic analyst uses a dissector to decode what a protocol contains, then analyses those decoded fields across many packets to identify behaviour patterns.
What does Wireshark's Follow TCP Stream feature do?
Follow TCP Stream reassembles all the TCP segments belonging to a single connection in sequence and displays the combined application-layer payload as a readable stream. It strips the IP and TCP headers and presents only the data exchanged between the two endpoints, making it straightforward to read HTTP conversations, FTP transfers, SMTP emails, and similar cleartext protocols without manually scrolling through individual packets.
Can credentials be recovered from a PCAP file?
Yes, if the credential exchange occurred over an unencrypted protocol. HTTP Basic Authentication transmits a Base64-encoded username and password in the Authorization header. FTP sends credentials as plaintext USER and PASS commands. Telnet and older POP3 or IMAP sessions are entirely cleartext. Wireshark dissectors decode these fields automatically. TLS-encrypted sessions require the session keys, typically from a SSLKEYLOGFILE, before credentials can be recovered.
What is file carving from a PCAP, and how does it work?
File carving from a PCAP extracts files that were transferred over the network by reassembling the application-layer payload of a session and identifying file headers and trailers within it. Tools such as NetworkMiner and Wireshark's Export Objects function identify complete files transferred via HTTP, SMB, FTP, or SMTP and write them to disk. The carved file retains its original binary content and can be hashed and examined like any other forensic artefact.
How are display filters different from capture filters in Wireshark?
Capture filters are applied at collection time using Berkeley Packet Filter (BPF) syntax; they control which packets are written to the capture file and discard the rest permanently. Display filters are applied after capture against an existing PCAP file; they hide packets that do not match without deleting them, so the full capture remains intact. For forensic work, it is safer to capture everything and use display filters to navigate, rather than risk discarding evidence at capture time.

Test yourself on Mobile and Network Forensics with free, timed mocks.

Practice Mobile and Network Forensics questions

Found this useful? Pass it along.

Share

Spotted an error in this page? Report a correction or read our editorial standards.

Your journey to becoming a forensic professional starts here.

Practice with mock tests, learn from structured notes, and get your questions answered by a global forensic community, all in one place.