Malware Forensics: Static, Dynamic, Sandbox and Memory Analysis

A malware sample is rarely a single category. A modern banker is also a dropper, a downloader, a credential-stealer and often a coin-miner in a fallback branch. The taxonomy below is still useful because it pins the primary behaviour the analyst should look for first.

The taxonomy guides the first triage. A ransomware sample is run in a network-isolated sandbox with a writable file tree to observe the encryption routine and look for the configuration data that drives the ransom note. An info-stealer is run with realistic browser profiles and credential stores so the data-exfil path lights up. A RAT is run with an outbound C2 that the analyst controls (INetSim or a real handler in a controlled environment). The mismatch between expected family and observed behaviour is the first hint that an automated AV label is wrong.

Static analysis carries no execution risk. Obfuscated, packed, or encrypted samples reveal less than clean binaries, but the analyst still recovers enough to plan dynamic analysis and begin family clustering.

The PE header is the first stop on Windows samples. PEview, PEStudio, CFF Explorer and Detect It Easy (DIE) all parse it. The analyst reads:

• Compilation timestamp in the COFF file header. Often spoofed by malware authors, but a precisely-aligned timestamp at midnight UTC in a year that does not match the campaign is itself a tell.
• Section headers. Standard sections are .text (code), .data, .rdata, .bss, .rsrc. Non-standard names (.UPX0, .UPX1, .nsp1, .vmp0) are packer signatures. A section marked executable that has a virtual size much larger than its raw size on disk is a packer-unpack stub.
• Imports table. The functions the binary asks the loader to resolve. CreateRemoteThread, VirtualAllocEx and WriteProcessMemory together suggest process injection. WinHttpOpen and InternetReadFile suggest HTTP C2. CryptEncrypt, BCryptEncrypt, CryptGenKey together suggest ransomware crypto.
• Exports table. DLLs and some EXEs export functions. An EXE with one named export "ServiceMain" is a Windows service; an EXE with exports "Run" or "Start" matching loader conventions is a stager.
• Resources directory. Icons, manifests, version info, and embedded payloads. Malware regularly hides a second-stage payload in the .rsrc section, sometimes encrypted with a key in .text.

Strings extraction is the second stop. The default strings utility prints printable ASCII; strings -e l prints UTF-16LE which Windows malware uses for most user-visible strings. The FLARE team's FLOSS (FireEye Labs Obfuscated String Solver), developed by Willi Ballenthin and Moritz Raabe at FireEye/Mandiant, goes further, deobfuscating stack strings and simple XOR-encoded strings through symbolic execution. A high-value strings hit is a C2 domain, a ransom-note template, a debug PDB path that contains an attacker username, or an unusual user-agent.

Hashes are next, in three flavours.

• MD5 and SHA-256 are the canonical file hashes. Identical bytes give identical hashes; a one-bit change defeats them. Useful for direct cross-reference against VirusTotal and CERT-In advisories.
• ImpHash hashes the ordered import table. Survives many recompilations because changing imports requires changing code. Clusters builds of the same family.
• Fuzzy hashes (ssdeep, TLSH) compute a hash that matches similar files. Useful when a malware author re-spins the binary with minor changes. TLSH is more robust to large files; ssdeep is the older standard.

Entropy is the fourth stop. Shannon entropy per section of the PE measures information density. A code section sits around 5.5 bits per byte; packed or encrypted sections sit above 7.0. DIE plots entropy by section; a single high-entropy section in an otherwise normal PE is a packer signature. The standard packers (UPX, MPRESS, Themida, VMProtect, ASPack) leave additional fingerprints in the entry point and the section names that DIE matches.

YARA is the fifth stop and the one the analyst writes themselves. A YARA rule is a small declarative file: meta block, strings block (literal, hex with wildcards, or regex), and a condition. A small example rule for a Cobalt Strike default beacon configuration reads:

rule CobaltStrike_DefaultBeacon_Config {
    meta:
        author = "ForensicSpot"
        family = "Cobalt Strike"
        reference = "Sentinel One default profile"
    strings:
        $sleep = { 69 68 69 68 69 68 69 68 }
        $sig1 = "%s as %s\\%s: %d" wide
        $sig2 = "ReflectiveLoader" ascii
    condition:
        uint16(0) == 0x5A4D and 2 of them
}

VirusTotal Retrohunt, Hybrid Analysis and any local malware archive (CFSL Hyderabad maintains one, and most state cyber-forensic labs collect samples through CERT-In feeds) run YARA at scale and surface clusters. The professional rule is: never tender a YARA hit alone as conclusive identification; tender it as one corroborating signal among several.

Dynamic analysis observes behaviour as the sample executes. The standard instrumentation stack on a Windows analyst workstation includes:

• Process Monitor (Procmon, Sysinternals) logs every file, registry, process and thread operation by every process. Filtered to the suspect process tree, it surfaces every dropped file, every registry write (Run keys, services, scheduled tasks), every spawned child process. Procmon traces are the loudest single signal in dynamic analysis.
• Process Hacker is the modern Task Manager replacement. Lists processes with parent-child relationships, command lines, loaded modules, handles, threads, and (with the kernel driver) hidden processes. Right-click on a suspect process reveals strings in memory, which often exposes plaintext C2 URLs that the on-disk binary kept encrypted.
• Wireshark captures network activity (see our Live Packet Capture topic for the capture-side detail).
• INetSim is a Linux service that fakes the internet for the sandbox VM. Responds to DNS, HTTP, HTTPS, SMTP, FTP and IRC with plausible defaults, lets the malware proceed past its connectivity check, and logs every request as the C2 indicator set.
• API Monitor / API hooks instruments user-mode API calls. Useful for tracking process-injection sequences (OpenProcess → VirtualAllocEx → WriteProcessMemory → CreateRemoteThread).
• Sysmon with a sensible config (Olaf Hartong's modular config or SwiftOnSecurity's classic config) writes detailed process, network and file events to the Windows event log, which gives a parallel timeline that the sandbox can export.

The sandbox operates above the analyst workstation tier: a fully isolated VM with snapshot revert, full telemetry, and a controlled network. The current open-source standard is Cuckoo Sandbox (maintained as CAPE Sandbox in modern deployments), which orchestrates VM snapshots, detonates the sample for a configured period, collects Procmon-equivalent telemetry, captures a memory image at completion, and writes a single JSON report covering file, registry, network, and process events.

Sandbox setup has three connectivity tiers. The strictest is no internet: the VM is completely air-gapped, and only the sample's behaviour against the local filesystem and process tree is observed. The next tier is simulated internet via INetSim: the malware sees a "working" internet that records every request without forwarding it; this exposes the C2 domains and URLs the sample tries without giving the attacker any feedback. The loosest tier is live internet through an anonymising VPN: the malware reaches the real C2 server, which is necessary when the sample's later stages will not download unless the C2 sees a real callback. Live-internet detonation is the right call only when the operational gain outweighs the risk of revealing the analyst's interest to the attacker, and is normally done from infrastructure that cannot be attributed back to the analyst's organisation.

Anti-sandbox detection is the cat-and-mouse layer. Modern malware checks for VirtualBox or VMware MAC OUI prefixes, the absence of mouse movement, an unreasonably high CPU clock, a recently-booted machine with no installed applications, debugger-present flags (IsDebuggerPresent, CheckRemoteDebuggerPresent), and timing-side-channel anomalies (RDTSC delta over a sleep). Anti-sandbox bypass is the corresponding sandbox-engineering response: realistic MAC randomisation, simulated user activity, aged VM snapshots with documents and browsing history, debugger hiding.

Static and dynamic analysis answer the surface questions. Deeper questions, the ransomware encryption algorithm, the C2 protocol format, the configuration block decoded by a second stage, require disassembly and debugging.

The disassembler choices in Indian labs:

• IDA Pro is the long-standing commercial standard. Fast disassembly, scriptable in Python (IDAPython), decompiler (Hex-Rays) at extra cost. NFSU and several CFSL labs hold IDA Pro licences.
• Ghidra is the NSA's open-source release. Free, with a built-in decompiler that is competitive with Hex-Rays for many architectures. The de facto default in academic and resource-constrained labs.
• Radare2 / rizin / Cutter are the open-source CLI-and-GUI stack. Powerful, scriptable, smaller user base.
• Binary Ninja is the modern commercial alternative; cheaper than IDA Pro, with a clean MLIL/HLIL pipeline.

The debugger choices:

• x64dbg is the open-source Windows debugger of choice. Plugin ecosystem (Scylla for IAT reconstruction, SwissArmyKnife, ScyllaHide for anti-anti-debug).
• WinDbg ships with the Windows SDK and is the only viable debugger for kernel-mode work and minidump analysis.
• OllyDbg is older; still useful for 32-bit Windows malware that targets a Windows 7 era.
• gdb is the canonical Linux debugger; with pwndbg, peda or gef plugins it is competitive with x64dbg for ELF samples.

Unpacking is the most common reverse-engineering subtask. The simple case is UPX, which has a public unpacker (upx -d). The harder case is custom packers, where the analyst sets a breakpoint at the original entry point of the unpacked code (the standard trick is to break on VirtualAlloc with EXECUTE_READWRITE protection and watch the unpacker write into the allocated region), then dumps the unpacked image and reconstructs the import address table with Scylla. Commercial packers like Themida and VMProtect virtualise the original code into a custom bytecode and present the analyst with a much harder problem; these are normally handled by family-specific scripts and by tracing rather than by unpacking.

1. Identify the packer with DIE
   Detect It Easy fingerprints common packers from entry-point bytes, section names and entropy. UPX, MPRESS, ASPack, Themida, VMProtect, Enigma all have signatures.
2. Try the public unpacker if one exists
   UPX: upx -d. The result is a clean PE that loads in Ghidra without further work. Other packers rarely have a public unpacker.
3. Set a breakpoint on the unpack stub's tail
   In x64dbg, break on VirtualAlloc returning a buffer with PAGE_EXECUTE_READWRITE, then trace into the loop that writes the unpacked code into it. The jump out of that loop is the tail.
4. Dump the unpacked image
   x64dbg's Scylla plugin reads the in-memory PE at the unpacked entry point, rebuilds the IAT (Imports), and writes a clean PE to disk.
5. Re-run static analysis on the unpacked PE
   Strings, imports, YARA all become useful again. Most family identification happens here, not on the packed sample.
6. Document the unpacking steps in the report
   The trial court will not run x64dbg. The report should reproduce the unpacking decisions in plain prose so a non-analyst can follow.

Memory analysis is the only discipline that recovers code that never lands on disk. Fileless malware writes its payload directly into another process's address space; a disk forensic image shows nothing, while a RAM image of the running infection captures the injected code, hollowed processes, and in-memory configuration blocks.

Acquisition first. Windows tools: winpmem (Rekall), DumpIt (Comae), Magnet RAM Capture. Each produces a raw memory dump (.raw or .mem) or a Microsoft crash dump (.dmp). The acquisition should be performed from a verified clean tool on removable media; the output should be written to that same removable media to avoid touching the suspect disk. Linux: AVML (Microsoft, AppArmor-aware); LiME (LKM-based, traditional). macOS: OSXPmem (legacy), with modern macOS being harder due to SIP and lacking maintained tools for the latest releases.

Volatility 3 is the current analysis framework. The Python 3 rewrite of the original Volatility, symbol-driven rather than profile-driven. The relevant plugin set:

MemProcFS is the modern alternative, presenting the memory image as a virtual filesystem. The analyst mounts the dump as a drive letter and walks /proc/, /sys/, /forensic/ paths to read process address spaces, kernel structures, file contents and YARA hits as files. The mental model is friendlier for analysts coming from a system-administration background, and many of the heavy plugins (PE reconstruction, browser-history parsing from process memory) are built in.

Indicators of Compromise are the analysis deliverables that travel beyond the originating investigation. The accepted IoC types and their canonical formats:

• File hashes: MD5, SHA-1, SHA-256, ImpHash, ssdeep, TLSH. Always SHA-256 as the primary; the others corroborate.
• Network indicators: IPv4/IPv6, FQDNs, URLs, URI patterns, JA3/JA3S fingerprints (cross-link to our Live Packet Capture topic).
• Host indicators: registry keys with values, mutex names, named-pipe names, scheduled-task names, service names, file paths.
• Email indicators: sender addresses, subject patterns, attachment hashes.
• YARA rules as a transferable detection artefact.

STIX (Structured Threat Information eXpression) is the data model that wraps these into objects (Indicator, Malware, ThreatActor, AttackPattern, Relationship). TAXII (Trusted Automated eXchange of Intelligence Information) is the transport that carries STIX between organisations. CERT-In publishes advisories that include STIX bundles for the larger campaigns; an Indian SOC consumes these and contributes back when the lab produces new IoCs.

Persistence mechanisms are the host-indicator subset that an investigation almost always documents.

For deeper coverage of registry-side artefacts see our Windows Forensic Artifacts topic.

Network behaviour analysis closes the IoC loop. Modern malware C2 channels characteristically include a Domain Generation Algorithm (DGA) producing daily fallback domains, beacon timing with small jitter, a JA3/JA3S fingerprint, and a recognisable URI shape. DGA analysis compares resolved domains from the host against the family's known generation seed to confirm or rule out attribution. Beacon timing analysis, identifying a periodic inter-flow gap with low variance, is the strongest behavioural signal when payload inspection is impossible. JA3/JA3S fingerprints survive C2 infrastructure rotation, providing a network signature for the family even after per-incident IPs and domains have been changed.

The report that ships to CERT-In has a defined shape: organisation, point of contact, incident type, time first noticed, time of compromise (where determinable), affected systems, malware family if identified, IoCs (hashes, IPs, domains, mutexes), mitigations applied, and supporting artefacts (PCAP, memory image hashes, YARA rules). CFSL Hyderabad maintains a sandbox lab that produces these reports; the cross-link to Cloud Forensics becomes relevant when the infected systems were cloud-hosted and jurisdictional questions on log retrieval enter the picture.

Evidence preservation rules for malware samples specifically: handle the sample only on an analysis VM with no path back to the corporate network or to the analyst's primary workstation; transfer via password-protected ZIP with the password "infected"; never plug a malware-bearing USB into a production machine; never run a sample on a host that has access to the lab's authentication infrastructure; never enable live internet unless the operational case is documented and approved. Anti-forensic techniques that malware uses (timestomping, log clearing, anti-debug, packed payloads) are previewed here and developed in our Anti-Forensic Techniques topic.