Malware forensics combines static examination, observed execution, automated sandboxing and memory analysis. These methods provide complementary views of a sample and a potentially affected system; no single result establishes the entire incident or its operator.
Preserve original material and analyse controlled copies with documented tools and containment. Distinguish what code could do, what it did in a test and what independent evidence shows happened on the affected host.
By the end of this topic you will be able to:
- Classify malware by supported capabilities and behaviour.
- Interpret executable structure, strings, hashes and detection rules.
- Plan contained execution and document environmental effects.
- Correlate memory and host evidence and report indicators with confidence limits.
- PE (Portable Executable) header
- Structures describing a Windows executable/object image, including headers, sections and directories. Values require interpretation and can be manipulated.
- ImpHash
- A hash derived from an import representation. It can help cluster files but is neither unique family attribution nor a guaranteed recompilation invariant.
- Entropy (Shannon)
- A statistical measure of byte distribution. High entropy can reflect compression, encryption or other data and is not proof of packing or malware.
- YARA
- A pattern-rule system used for matching selected content or properties. Rule quality, coverage and false-positive assessment matter.
- Volatility 3
- A symbol- and layer-based memory analysis framework. Available plugins and support depend on its version and the captured platform.
- IoC (Indicator of Compromise)
- An observable or detection pattern relevant to possible compromise, with provenance, time, context and confidence. An indicator match alone is not a complete finding.
Malware Types and Family Classification
Virus, worm, trojan, ransomware, remote-access tool, stealer and rootkit describe different behaviour or roles and can overlap. A family label should not replace examination of this sample’s actual capabilities.
Scroll horizontally if needed to see the full table.
“Fileless” is a broad operational label, not proof that no disk artefacts exist. Scripts, registry content, task definitions, logs, paging files and endpoint telemetry can remain even when a payload executes primarily in memory.
Memory may reveal transient code and configuration, while endpoint instrumentation or controlled execution can also observe memory-resident behaviour. It is inaccurate to call a RAM image the only method capable of investigating it.
Use verified published cases only with sources that support the specific claims. A lesson’s practical workflow does not require guessing a ransomware family, institutional capability or undocumented after-action finding.
Static Analysis: PE Header, strings, Hashes, Entropy, YARA
Static examination avoids deliberately executing the sample, but analysis tools themselves process untrusted input. Use an appropriate isolated environment and preserve originals; “no execution” is not “no risk.”
For PE files, inspect architecture, entry point, sections, directories, imports/exports and resources. COFF timestamps can be manipulated or have special meanings in reproducible builds. An unusual date or export name does not alone identify malware or a Windows service.
Imported APIs suggest possible functions, not confirmed runtime actions. Dynamic resolution can hide imports, and legitimate software can use memory or cryptographic APIs. Resources and section names need the same contextual assessment.
Strings tools can expose text under selected encodings; FLOSS additionally attempts extraction of supported obfuscated strings. A path, username-like string or domain can be stale, decoy or unrelated to execution. Preserve its offset and decoding context.
Use SHA-256 for exact-byte identification and suitable similarity methods for their different purposes. Import and fuzzy hashes can yield useful relationships without proving a family or shared actor. Confirm meaningful matches through additional analysis.
Entropy is a clue rather than a fixed packed/unpacked threshold. Compression and legitimate resources can be highly entropic, while packed content need not satisfy one cutoff. Validate packer detections and limitations.
YARA rules should have traceable provenance and be assessed against suitable positive and negative samples. A hit identifies its matched condition; it is not conclusive attribution. Preserve rule/version, offsets and matching data.
Hash each preserved source and relevant derivative. An unpacked or instrumented output may intentionally differ from the original; document that relationship instead of declaring every changed hash invalidates all analysis.
Dynamic Analysis and Sandbox Tiers
Dynamic analysis records behaviour during execution in a defined environment. Procmon, Sysmon, API instrumentation and network capture can expose different activity, subject to configuration, collection loss and supported events.
Use dedicated containment with no unintended production trust, credentials, shared folders or unrestricted network path. A VLAN or VM alone is not a complete isolation guarantee. Retain the baseline, test configuration and operational changes.
CAPE is an automated malware-analysis project derived from Cuckoo; it is not merely a rename of every current Cuckoo deployment. Check actual guest, package, memory and reporting support rather than promise all OS families.
No network, simulated services and controlled external connectivity produce different behaviours. Simulated responses can expose attempted contacts but may not satisfy certificate, protocol or server-side checks. A VPN does not make live execution safe or untraceable.
Public sandbox uploads can disclose sensitive samples and investigation context. Verify access, retention and sharing terms before submission; a password-protected archive is a handling convention, not a containment boundary.
Anti-analysis checks, timeouts, user interaction and unavailable dependencies can suppress behaviour. A quiet run does not prove benignness. Report the tested conditions and observed duration instead of treating a sandbox score as a universal verdict.
Reverse Engineering and Unpacking
Reverse engineering examines code and data relationships to answer focused questions such as configuration decoding or an encryption routine. Disassembly and decompilation are interpretations that need architecture, calling convention and dynamic validation where appropriate.
Ghidra, IDA, Binary Ninja and other tools have different support and licensing. Debuggers such as WinDbg, x64dbg or gdb apply to particular environments; no single debugger is the only viable option for every kernel or dump.
Packing changes how code or data is represented. Identify the actual mechanism, preserve the packed original and document any unpacking. A public unpacker may fail on modified or damaged formats.
Runtime allocation and protection changes can guide inspection, but no universal breakpoint on VirtualAlloc discovers every original entry point. Some code uses different APIs, stages or virtualisation.
Dumped memory may need relocation, imports or metadata reconstructed for further analysis. That reconstruction is a derivative and can remain incomplete; a tool’s “clean PE” output is not necessarily the original disk file.
Keep addresses, scripts, debugger observations and export settings traceable. Separate observed code paths from unexecuted branches and explain the effect of patching or bypassing anti-analysis checks.
Memory Analysis with Volatility 3 and MemProcFS
Acquire memory through a method supported by the actual OS, hardware and access state. Live acquisition changes state and can miss or alter information; it is not an atomic, perfectly clean snapshot of the entire machine.
Verify the collection tool and output destination, avoid unnecessary source writes and record start/end times, errors and hashes. Available formats differ by tool; modern platform protections can limit acquisition.
Volatility 3 separates acquisition from analysis and uses supported layers and symbols. Confirm plugin names against the installed release and inspect warnings. Do not transfer every historical Volatility 2/community plugin name to a current installation.
Process lists, scans, tree relationships, command-line data and network structures can corroborate one another. Discrepancies can reflect exit, reuse, incomplete capture or parsing limitations as well as concealment.
Suspicious executable memory or a malfind result requires validation. Legitimate JIT code and protection choices can resemble injection indicators. No single page permission, process name or parent-child relationship proves malware.
Use a suitable dump mechanism for the identified memory object. DumpFiles operates on supported cached file structures; arbitrary injected process regions require the appropriate process-memory extraction. Recovered bytes can be partial.
MemProcFS provides a filesystem-oriented memory view under its supported features. Such a mounted view is derived analysis, not the original machine’s complete live filesystem or guaranteed record of every action.
Malware Indicators, Persistence and Forensic Reporting
Indicators can include exact hashes, network endpoints, paths, registry values and validated detection patterns. Provide source, first/last observed times where justified, scope and confidence; shared infrastructure and generic names can create false matches.
STIX represents threat-intelligence objects and relationships, while TAXII supports exchange. A STIX bundle does not itself establish provenance, endorsement or automatic publication by an incident-response authority.
Persistence examination distinguishes a configured mechanism, the referenced payload and demonstrated execution. Services, scheduled tasks, startup keys, WMI subscriptions and extensions can all be legitimate under some conditions.
Inspect actual paths, values, security context, timestamps and correlated execution. A broad extension permission or encoded PowerShell argument is a lead, not a unique credential-stealer signature.
For source-specific interpretation, continue to Windows Forensic Artifacts. Match configuration to available activity rather than assume every definition ran.
Periodic connections, generated-looking domains and TLS fingerprints can guide investigation. They are neither mandatory malware features nor unique family identifiers. Validate the relevant configuration and observed network behaviour.
The report should explain receipt, methods, tested environment, findings, affected scope, limitations and response actions. Reporting and disclosure requirements vary by jurisdiction and sector; initial notification may precede a complete analysis.
Keep collected evidence separate from later remediation or test runs. Protect secrets and samples throughout transfer, storage and controlled execution; a familiar archive password does not authorise public sharing.
Check Your Understanding
Sources
- 1.Guide to Integrating Forensic Techniques into Incident Response (2006) · NIST SP 800-86
- 2.Best Practices for Computer Forensic Acquisition, 17-F-002-2.1 · SWGDE
- 3.PE/COFF executable structure · Microsoft
- 4.FLARE Obfuscated String Solver · Mandiant FLARE
- 5.Acquisition separation, symbols and plugin usage · Volatility Foundation
- 6.CAPE automated analysis and configuration · CAPE Sandbox
- 7.Introduction to STIX 2.1 · OASIS CTI documentation