Malware Analysis Bootcamp

Malware analysis requires an isolated environment — a dedicated virtual machine with no network access (or a completely isolated network), snapshotted to a clean state so you can restore after each analysis. The VM should have all necessary tools pre-installed and nothing of value (no real credentials, no real data, no production access).

The gold standard analysis VM is FlareVM — a Windows VM with every malware analysis tool pre-installed. REMnux is the Linux equivalent. Both are free. Between them they cover 90% of analysis tasks you will encounter.

Most malware analysis in a SOC or IR context is triage — quickly answering "is this malicious, what family, what does it do, what IOCs does it produce?" in 15 minutes, not 15 hours. The triage workflow is a fixed sequence that gets progressively deeper until the question is answered. If step 1 answers it, stop. If not, proceed to step 2. Most samples are resolved by step 3.

The Portable Executable (PE) format is the file format for Windows executables (.exe, .dll, .sys). Understanding its structure is essential for malware analysis — it tells you how the binary is organised, what code it contains, what data it uses, and crucially, when something is wrong (packed/obfuscated executables have very different PE structure from normal ones).

Every PE file starts with the DOS header (MZ signature), followed by the PE header (machine type, timestamp, number of sections), the optional header (entry point address, image base, subsystem), and then the section table. The sections contain the actual content: .text (code), .data (initialised data), .rdata (read-only data including strings and imports), .rsrc (resources — icons, manifests, embedded files), and sometimes custom sections added by packers.

Process Monitor (Sysinternals, free) is the most powerful Windows system monitoring tool available. It captures every file operation, registry operation, network connection attempt, and process/thread creation in real time. When you run malware under Process Monitor, every action it takes is recorded with full context: which process made the call, what resource it accessed, what the result was, and the complete call stack.

The key technique is filtering. Malware generates hundreds of events per second but most are noise — system processes, background services, Windows update. Filter to show only events from the malware process (or its children). Then look for the high-value event categories: registry writes (persistence), file creates in %APPDATA% or %TEMP% (payload drops), registry Run key modifications (auto-start), and network connections.

Automated sandboxes are the force multiplier of malware analysis — they analyse hundreds of samples per day automatically, producing structured reports that extract behaviour, network IOCs, and ATT&CK technique mappings without analyst time. Every CTI analyst and SOC analyst must know how to read sandbox reports efficiently and how to use sandbox APIs to automate analysis at scale.

Sandbox evasion is common in sophisticated malware — it detects it is running in a VM (by checking CPUID, registry keys for VM software, lack of mouse movement, execution time analysis) and behaves differently or terminates. Knowing sandbox evasion techniques helps you choose the right analysis approach when automated analysis gives no results.

You do not need to become an assembly expert to be a malware analyst. You need to recognise common patterns — function prologues and epilogues, loop structures, conditional branches, API calls, and string construction. Once you can identify these patterns, you can understand what a function does without reading every single instruction.

The x86/x64 architecture uses a register-based model. Registers are fast storage inside the CPU. For 64-bit (x64): RAX, RBX, RCX, RDX (general purpose), RSP (stack pointer), RBP (base pointer), RIP (instruction pointer), RSI/RDI (source/destination index). The calling convention determines how function arguments are passed: in x64 Windows, the first four arguments go in RCX, RDX, R8, R9 — any additional arguments go on the stack. When you see a call instruction, check what is in those registers immediately before the call — that is what arguments the function is receiving.

Ghidra (free, NSA) is the gold standard for malware reverse engineering alongside IDA Pro. Ghidra disassembles binaries, decompiles them to C-like pseudocode, and provides a rich environment for annotating, navigating, and understanding code. The decompiler is especially powerful — even imperfect pseudocode is far easier to read than raw assembly, and it dramatically accelerates analysis of complex functions.

The most important Ghidra skill is renaming and annotating. Start with functions Ghidra names FUN_00401000. As you understand what each function does, rename it: decrypt_config, check_sandbox, inject_shellcode. Add comments at key instructions. As your understanding builds, the binary becomes readable code. This is the foundation of advanced malware analysis.

A packer compresses or encrypts a malware binary and wraps it in a stub that decompresses/decrypts and executes the original code at runtime. The packer defeats static analysis — the binary you see on disk contains only encrypted data, not the actual malware code. To analyse the real malware, you must unpack it: either dump the process memory after the unpacker has run (and the real code is in memory), or trace through the unpacking stub to find where it jumps to the unpacked code (the Original Entry Point, or OEP).

The general unpacking methodology: (1) Load the packed binary in x64dbg. (2) Let the packer stub execute — it will decompress/decrypt the real code into memory. (3) The packer then jumps to the OEP (a jmp eax or push eax; ret at the end of the stub is the classic sign). (4) Set a hardware breakpoint on this jump. (5) When the breakpoint hits, dump the process memory with Scylla. (6) Fix the import table. (7) Analyse the dumped binary.

Script-based malware (PowerShell, VBScript, JScript, Office macros) is the most common initial access vector. Unlike compiled malware, scripts are readable — but attackers heavily obfuscate them using concatenation, encoding, variable substitution, and reflection to evade detection and complicate analysis. The good news: scripts must ultimately execute interpretable code, so deobfuscation always works given enough time and the right approach.

PowerShell obfuscation layers typically include: Base64 encoding (-EncodedCommand flag), character code substitution ([char]65 for 'A'), string concatenation ('I'+'EX' for 'IEX'), variable substitution, format string abuse, and SecureString encoding. Each layer can be peeled off systematically.

Ransomware is the highest-impact malware category because it causes immediate business disruption — encrypted files cannot be accessed until the key is recovered or a backup is restored. Understanding ransomware at the code level answers critical IR questions: Is decryption possible without paying? Was the key generated with a weak PRNG that makes brute-force feasible? Was the key ever transmitted to the C2 (and is it in a network capture)? Is there a flaw in the encryption implementation that allows recovery?

Most ransomware follows a common structure: (1) Generate a symmetric key (AES-256 or ChaCha20) for file encryption. (2) Encrypt each file with that key. (3) Encrypt the symmetric key with the attacker's RSA-2048 or Curve25519 public key (embedded in the binary). (4) Write the encrypted key alongside or inside each encrypted file. (5) Delete originals (Volume Shadow Copies first). The only way to decrypt without paying is: recover the symmetric key from memory before the ransomware deletes it, or find a flaw in the key generation or encryption implementation.

The majority of malware incidents involve commodity malware — tools sold or rented on underground forums, used by many different threat actors. Understanding the major families deeply allows rapid analysis of any variant because they share code, protocol patterns, and IOC signatures.

YARA is the pattern-matching language of malware analysis. A YARA rule describes a malware family using strings (text, hex patterns, regular expressions) and conditions (how many strings must match, file size constraints, PE header conditions). YARA rules are used by every major AV vendor, SIEM, EDR, and sandbox to detect malware families. Writing a YARA rule for a new malware family — or a variant that evades existing signatures — is one of the most valued skills in threat intelligence.

A good YARA rule is specific enough to match the target family (no false negatives) and precise enough not to match legitimate files (no false positives). The balance comes from choosing the right strings: not too generic (common API names appear in thousands of legitimate files), not too specific (single bytes that could change in any recompile). Look for: unique strings that appear only in this family, unique code patterns (specific byte sequences), structural anomalies (section count, section names, import combinations), and configuration data formats.