Chapter 24

Rootkit Detection in Memory

Rootkits operate by subverting the OS's own data structures to hide themselves. Memory forensics detects them by scanning raw memory independently of the OS APIs that rootkits corrupt. This chapter covers DKOM, SSDT hooking, driver hiding, and how Volatility surfaces rootkit indicators.

Scenario

A banking malware sample has been running on a compromised host for 6 weeks before detection. The EDR was deployed 4 weeks ago but never alerted on it — because the rootkit hooked the APIs that EDR uses to enumerate processes and drivers. tasklist shows a clean process list. Device Manager shows no suspicious drivers. But the memory image tells the truth: psscan finds a process not in pslist (DKOM); ssdt reveals hooks on NtOpenProcess and NtQuerySystemInformation (SSDT hooking); modules shows a driver loaded at an unexpected base address not in the legitimate driver list. The rootkit is invisible to the live OS but fully visible in raw memory forensics.

Common Rootkit Techniques and Memory Signatures

Technique	How it works	Detection method
DKOM (Direct Kernel Object Manipulation)	Unlinks the malicious process's EPROCESS structure from the ActiveProcessLinks list — pslist misses it	Compare psscan (raw scan) vs pslist (list walk) — process in psscan but not pslist = DKOM hidden
SSDT hooking	Replaces entries in the System Service Descriptor Table (ntoskrnl function pointers) with attacker code — API calls are intercepted	windows.ssdt — shows any SSDT entries that point outside ntoskrnl/win32k address ranges
Driver hiding (IRP hooking)	Removes the driver's DRIVER_OBJECT from driver list; hooks IRP handlers to intercept disk/network access	Compare modules (list walk) vs driverscan (raw scan) — driver in driverscan but not modules = hidden
IDT hooking	Modifies Interrupt Descriptor Table to intercept interrupts — older technique, less common now	windows.idt plugin — entries outside expected module ranges
SSDT inline patching (hot-patching)	Rewrites the first few bytes of SSDT target functions with JMP to rootkit code (inline hook)	windows.ssdt + manual inspection of function prologues for JMP instructions at start

DKOM Detection

Bashdkom-detection.sh

RAM="/cases/CASE001/RAM.raw"
OUT="/cases/CASE001/memory"

# Get process list via EPROCESS linked list (what OS reports)
vol -f $RAM windows.pslist --output csv > $OUT/pslist.csv

# Get all EPROCESS structures via raw memory scan (bypasses OS)
vol -f $RAM windows.psscan --output csv > $OUT/psscan.csv

# Compare the two sets — missing in pslist = DKOM hidden
python3 << 'EOF'
import csv

def get_pids(filename):
    pids = set()
    with open(filename) as f:
        reader = csv.DictReader(f)
        for row in reader:
            pids.add(row['PID'].strip())
    return pids

pslist_pids = get_pids("/cases/CASE001/memory/pslist.csv")
psscan_pids = get_pids("/cases/CASE001/memory/psscan.csv")

hidden = psscan_pids - pslist_pids
if hidden:
    print("[!] DKOM HIDDEN PROCESSES (in psscan but not pslist):")
    for pid in sorted(hidden):
        print(f"    PID: {pid}")
else:
    print("[OK] No DKOM hidden processes detected")
EOF

SSDT Hook Detection

Bashssdt-analysis.sh

RAM="/cases/CASE001/RAM.raw"

# Enumerate SSDT (System Service Descriptor Table)
# All entries should point to ntoskrnl.exe or win32k.sys
# Any entry pointing elsewhere = hook
vol -f $RAM windows.ssdt

# Normal output (clean system):
# Table 0 (ntoskrnl.exe)
# Entry 0x000 (NtAcceptConnectPort)       ntoskrnl.exe!NtAcceptConnectPort
# Entry 0x001 (NtAccessCheck)             ntoskrnl.exe!NtAccessCheck
# ...

# Hooked output (rootkit):
# Entry 0x007 (NtOpenProcess)             UNKNOWN (hooked!)
#   → Address falls outside ntoskrnl.exe module range
#   → This entry intercepts all process open operations
#   → EDR tools calling NtOpenProcess go through the rootkit's code first

# If SSDT shows unknown addresses, determine what module they belong to:
vol -f $RAM windows.modules | grep -v "ntoskrnl\|win32k"
# Entries outside these two = loaded non-Microsoft kernel modules
# Most should be legitimate (AV drivers, network drivers)
# Ones that are hidden (not in modules but visible in driverscan) = suspicious

Hidden Driver Detection

Bashdriver-analysis.sh

RAM="/cases/CASE001/RAM.raw"
OUT="/cases/CASE001/memory"

# List drivers via kernel module list (DRIVER_OBJECT linked list)
vol -f $RAM windows.modules --output csv > $OUT/modules.csv

# Scan raw memory for DRIVER_OBJECT structures (bypasses hidden list)
vol -f $RAM windows.driverscan --output csv > $OUT/driverscan.csv

# Compare: drivers in driverscan but not modules = hidden drivers
python3 << 'EOF'
import csv

def get_drivers(filename, name_col='Name'):
    drivers = {}
    with open(filename) as f:
        reader = csv.DictReader(f)
        for row in reader:
            name = row.get(name_col, row.get('BaseDllName', '')).strip().lower()
            if name:
                drivers[name] = row
    return drivers

modules  = get_drivers("/cases/CASE001/memory/modules.csv",  name_col='BaseDllName')
drvscan  = get_drivers("/cases/CASE001/memory/driverscan.csv", name_col='Name')

hidden = set(drvscan.keys()) - set(modules.keys())
for d in sorted(hidden):
    print(f"[HIDDEN DRIVER] {d}")
    print(f"  Details: {drvscan[d]}")
EOF

# For each suspicious driver, extract it from memory for static analysis
vol -f $RAM windows.dumpfiles --virtaddr  --output-dir $OUT/drivers/

Rootkit Detection Checklist

Check	Volatility command	Flag for investigation
Hidden processes	Compare pslist vs psscan	Any PID in psscan not in pslist
SSDT hooks	windows.ssdt	Any entry pointing outside ntoskrnl/win32k
Hidden drivers	Compare modules vs driverscan	Any driver in driverscan not in modules
Injected code in kernel	windows.kdbgscan + manual	Executable memory pages in kernel space outside known module ranges
IRP hook	windows.driverirp	IRP handler addresses outside the driver's own address space

Why rootkits don't fully defeat memory forensics

A rootkit's power comes from subverting the OS's own APIs and data structures — it controls what the OS reports back to tools that query it. But Volatility doesn't use the OS API. It reads raw physical memory and reconstructs OS structures independently, using its own offset calculations from the OS version's symbol tables. A rootkit that unlinks its process from ActiveProcessLinks can't remove the EPROCESS structure itself from physical memory — the bytes are still there. A rootkit that hooks SSDT can't remove the original SSDT from the ntoskrnl image in memory. This is the fundamental advantage of memory forensics over live-system analysis: the attacker controls the OS, but they don't control how raw memory bytes are interpreted by an external tool reading a captured image. The exception is hypervisor-level or firmware-level rootkits — but those are rare and require very specific analysis techniques.

Q & A

Q: You find a rootkit but you need to keep the system running for business continuity. Can you still investigate safely?

With a rootkit present, you can't trust anything the live OS tells you — all further live investigation is potentially compromised. The safe path: (1) Take the memory image immediately while the rootkit is still active (you want to capture it, not avoid it). (2) Isolate the system via out-of-band means — network-level isolation (firewall block at the perimeter), not via the OS (the rootkit controls the OS networking). (3) Analysis from that point forward happens on the memory image and any disk images from the isolated system — not via live tools on the rootkit-infected system. If business continuity is critical, your options are: restore from a known-clean backup, bring up a parallel system while the infected one remains isolated for forensics, or accept the risk of operating on a compromised system while documenting the risk. In regulated environments (healthcare, finance, government), an unremoved rootkit on a production system may itself be a reportable security incident — escalate to legal/compliance immediately.