Malware Analysis and Reverse Engineering

Malware analysis produces actionable indicators: C2 hostnames, mutexes, file paths, registry keys, and behavioral signatures for detection rules. The analyst's job is not to understand every instruction — it is to extract the indicators and capabilities as efficiently as possible. This chapter covers the practical workflow a detection engineer uses when handed a suspicious binary, from triage through full RE to Sigma/KQL rule generation.

Static Analysis Methodology

Dynamic Analysis and Sandbox

// Dynamic analysis: run sample in isolated VM, monitor all activity
// Tools: Process Monitor (ProcMon), Wireshark, Regshot, API Monitor, x64dbg

// ProcMon filters for malware analysis:
// Process Name = WindowsUpdate.dll (or rundll32.exe if invoking DLL)
// Category = File System, Registry, Network
// → Reveals: files written, registry persistence, C2 connections

// Inetsim: fake internet services (HTTP, DNS, SMTP, FTP) for offline sandbox
// Run on separate VM in same network as analysis VM
// inetsim --report-dir /var/inetsim/reports --fakeresolv

// API Monitor hooks:
// Monitor: VirtualAlloc, CreateProcess, WriteFile, RegSetValue, WinHttpSendRequest
// → Shows exact API calls with parameters as malware executes

// Cuckoo sandbox config snippet for network monitoring:
# cuckoo.conf
[routing]
route = inetsim          # route all sample traffic through inetsim
internet = yes           # allow sample to access real internet (use carefully)

# Submit:
# cuckoo submit --timeout 120 WindowsUpdate.dll

// Quick DLL execution for analysis:
// rundll32.exe WindowsUpdate.dll,DllMain  (if exports DllMain)
// rundll32.exe WindowsUpdate.dll,ExportedFunction  (if known export)
// regsvr32.exe WindowsUpdate.dll  (calls DllRegisterServer export)

Disassembly and Decompilation

// IDA Pro / Ghidra / Binary Ninja: static disassembly + decompiler
// Key analysis techniques:

// 1. Finding DllMain / WinMain entrypoint:
//    IDA: Functions window → search "DllMain" or "sub_" with reference from entry
//    Ghidra: Symbol Table → _DllMainCRTStartup → follow to user DllMain

// 2. Identifying encryption:
//    Look for: constant 0x9E3779B9 (MurmurHash/XTEA), 0x61C88647 (RC6),
//              0x67452301 (MD5 init), or large S-box tables (AES)
//    Entropy plugin: FindCrypt (IDA) / Crypto Identifier (Ghidra)

// 3. Resolving obfuscated API calls (hash-based API resolution):
//    Pattern: loop over PEB.Ldr module list → hash each export name → compare
//    Script: IDAPython / Ghidra script to compute FNV-1a hashes of all exports
//            and annotate call sites with resolved names

// IDAPython: resolve FNV-1a hash API calls
import idc, idautils

def fnv1a(s):
    h = 0x811c9dc5
    for c in s.encode():
        h ^= c; h = (h * 0x01000193) & 0xFFFFFFFF
    return h

# Build hash→name table for all loaded modules
HASH_TO_NAME = {}
for m in idautils.Modules():
    for exp in idautils.Entries():
        name = idc.get_name(exp[2])
        if name:
            HASH_TO_NAME[fnv1a(name)] = name

# Scan for PUSH imm32 + CALL (API resolver pattern) and annotate:
for func_ea in idautils.Functions():
    for head in idautils.Heads(func_ea, idc.find_func_end(func_ea)):
        if idc.print_insn_mnem(head) == "push":
            val = idc.get_operand_value(head, 0)
            if val in HASH_TO_NAME:
                idc.set_cmt(head, f"API: {HASH_TO_NAME[val]}", 0)

Unpacking Malware

// Unpacking: let the packer stub run until the original PE is in memory, then dump it
// Method 1: OEP (Original Entry Point) breakpoint
//   - Run in x64dbg; let stub execute
//   - When original PE is mapped in memory: memory map shows new region
//   - Set BP at region's start → execution reaches OEP → dump with Scylla/OllyDump
//   - PE fix: Scylla "Fix Dump" → rebuilds IAT from original imports

// Method 2: Hardware write breakpoint on memory region
//   - After first VirtualAlloc, set hardware BP on allocated region
//   - Packer writes decrypted PE → BP fires at first write
//   - Step through until PE header visible → dump

// Method 3: MiniDump after unpacking
//   - Let malware run in controlled sandbox (Wireshark capturing network)
//   - Attach x64dbg: dump process memory → get mapped PE regions
//   - pe-sieve (hasherezade): scans process for PE modules and dumps them
//   pe-sieve64.exe --pid  --shellcode 1 --out C:\dump

// x64dbg workflow:
// 1. Open DLL: "rundll32.exe WindowsUpdate.dll,#1" in x64dbg
// 2. Run to DllMain → step over packer setup
// 3. Watch Memory Map panel for new RWX allocation
// 4. When new region appears: right-click → Set BP on Write
// 5. Run → BP fires when payload is written → dump with Scylla
// 6. Analyze dumped PE statically (now unpacked)

IOC Extraction Checklist

Detection Engineering

title: Malware Sample — Extracted IOCs from WindowsUpdate.dll Analysis
# Example rule built from analysis findings:
logsource:
  product: windows
  service: sysmon
detection:
  mutex:
    EventID: 17  # PipeEvent (mutex would be in object creation — use process event)
  file_drop:
    EventID: 11
    TargetFilename|contains:
      - '\Roaming\Microsoft\WindowsUpdate.dll'
  registry:
    EventID: 13
    TargetObject|contains: 'CurrentVersion\Run'
    Details|contains: 'WindowsUpdate.dll'
  condition: file_drop or registry
level: critical
tags: [attack.persistence, T1547.001]

-- MDE KQL: hunt for samples of this malware family by imphash
DeviceFileEvents
| where SHA256 in (
    "ABC123...",  // sample hash
    "DEF456..."   // variant hash from VT relations
  )
| project Timestamp, DeviceName, FolderPath, FileName, InitiatingProcessFileName

-- MDE KQL: hunt for C2 connectivity extracted from analysis
DeviceNetworkEvents
| where RemoteUrl has_any ("c2.example[.]com", "update.fakems[.]net")
    or RemoteIP in ("185.220.101.55", "91.195.240.103")
| project Timestamp, DeviceName, InitiatingProcessFileName, RemoteUrl, RemoteIP

Q&A