System Calls

How Win32 API calls become kernel operations: ntdll.dll stubs, syscall numbers, the SSDT, EDR hooking in ntdll, and how sophisticated malware bypasses those hooks with direct syscalls

The Windows API Layers

Windows exposes multiple API layers, each at a different abstraction level. From highest to lowest:

  Windows API Hierarchy (user mode)
  ─────────────────────────────────────────────────────────────────
  Layer 4: Win32 Wrappers
  ┌─────────────────────────────────────────────────────────────┐
  │ kernel32.dll, advapi32.dll, shell32.dll, user32.dll        │
  │ CreateFile(), OpenProcess(), CreateRemoteThread()          │
  │ High-level, documented, argument validation, error mapping │
  └──────────────────────────┬──────────────────────────────────┘
                             │ calls
  Layer 3: NT API (Native API)
  ┌──────────────────────────▼──────────────────────────────────┐
  │ ntdll.dll                                                   │
  │ NtCreateFile(), NtOpenProcess(), NtCreateThreadEx()        │
  │ Direct kernel interface; minimal validation; NTSTATUS codes│
  └──────────────────────────┬──────────────────────────────────┘
                             │ SYSCALL instruction
  ═══════════════════════════╪═══════════════ USER/KERNEL BOUNDARY
  Layer 2: System Call Dispatcher
  ┌──────────────────────────▼──────────────────────────────────┐
  │ KiSystemCall64 (in ntoskrnl.exe)                           │
  │ Reads SSN from EAX → looks up in SSDT → calls handler      │
  └──────────────────────────┬──────────────────────────────────┘
                             │
  Layer 1: NT Executive Implementation
  ┌──────────────────────────▼──────────────────────────────────┐
  │ NtCreateFile(), NtOpenProcess() in the Executive           │
  │ Full implementation: access checks, object creation, etc.  │
  └─────────────────────────────────────────────────────────────┘

  

ntdll.dll — The Gateway

ntdll.dll is the lowest-level user-mode DLL. It's the only DLL that's guaranteed to be loaded in every Windows process. Its two primary roles:

1. System call stubs: Contains a stub function for every NT API. Each stub loads a syscall number into EAX and executes SYSCALL to transition to the kernel.
2. Windows runtime support: The loader (LdrLoadDll, process initialization), the heap manager (RtlAllocateHeap), the exception dispatcher, and various runtime library functions.

ntdll vs kernel32

The relationship between kernel32 and ntdll is a wrapper/implementation split. CreateFile() in kernel32 validates arguments, converts the Windows path to an NT path (prepends \\??\\), fills out the correct IO_STATUS_BLOCK structure, and calls NtCreateFile() in ntdll. The ntdll stub doesn't validate anything — it immediately invokes the system call. Malware that calls ntdll directly bypasses kernel32's argument validation and, importantly, bypasses EDR hooks placed on kernel32 functions.

Syscall Numbers (SSNs)

Every NT API function has a corresponding System Service Number (SSN) — an integer index into the SSDT. The SSN tells the kernel dispatcher which kernel function to invoke. SSNs are not stable across Windows versions; they change between major releases and even service packs. This is why malware that hardcodes SSNs breaks on different OS versions.

Syscall Stub Disassembly

Every ntdll syscall stub follows the same pattern. Here's the actual disassembly of NtAllocateVirtualMemory from ntdll:

; NtAllocateVirtualMemory in ntdll.dll (Win10 x64)
; Disassembly from WinDbg: u ntdll!NtAllocateVirtualMemory

NtAllocateVirtualMemory:
    mov     r10, rcx        ; save first argument (ProcessHandle)
                            ; SYSCALL clobbers RCX, so move it to R10 first
    mov     eax, 18h        ; SSN = 0x18 (NtAllocateVirtualMemory on Win10 22H2)
    test    [7FFE0308h], byte ptr 1  ; check SharedUserData.SystemCall flag
    jne     short syscall_alt   ; on very old systems, use INT 0x2E instead
    syscall                 ; transition to kernel mode (Ring 3 → Ring 0)
    ret                     ; SYSRET returns here; return NTSTATUS in RAX
syscall_alt:
    int     2Eh             ; legacy path for pre-Vista compatibility
    ret

That's it — 5 instructions for the actual system call dispatch. The real work happens in the kernel. The stub just needs to: (1) save RCX (which SYSCALL destroys by saving the return RIP into it), (2) load the SSN, (3) execute SYSCALL.

What SYSCALL Does to Registers

The System Service Descriptor Table (SSDT)

The SSDT is an array of kernel function pointers (stored as RVA offsets from the table base on x64). The kernel dispatcher indexes it with the SSN to find the actual kernel implementation to call.

  SSDT Structure (x64, simplified)
  ─────────────────────────────────────────────────────────────────
  KiServiceTable (= KeServiceDescriptorTable.Base):

  Index 0x00: NtMapUserPhysicalPagesScatter  → offset → ntoskrnl+0xAAAAA
  Index 0x01: NtWaitForSingleObject          → offset → ntoskrnl+0xBBBBB
  Index 0x02: NtCallbackReturn               → offset → ntoskrnl+0xCCCCC
  ...
  Index 0x18: NtAllocateVirtualMemory        → offset → ntoskrnl+0xDDDDD
  ...
  Index 0xC1: NtCreateThreadEx               → offset → ntoskrnl+0xEEEEE

  On 64-bit Windows, entries store a right-shifted RVA (the bottom 4 bits
  are used to count the number of stack arguments). The dispatcher
  reconstructs the full address as: KiServiceTable + (entry >> 4)

  

On 32-bit Windows, SSDT hooking was the dominant rootkit technique — replace an entry to redirect system calls to malicious code. On 64-bit Windows, KernelPatchGuard monitors the SSDT and triggers a 0xC0000420 bugcheck if modifications are detected, making SSDT hooking impractical without also defeating PatchGuard.

32-bit Processes on 64-bit Windows (Brief)

When a 32-bit process calls VirtualAllocEx(), it goes through a different path: the 32-bit ntdll (in C:\Windows\SysWOW64\ntdll.dll) executes a 32-bit syscall sequence. The WOW64 layer intercepts this and translates it to a 64-bit system call. This involves the "heaven's gate" technique — a far jump from 32-bit code segment (CS=0x23) to 64-bit code segment (CS=0x33) to reach the 64-bit NTDLL's system call stub. This is covered in depth in Chapter 4.

Direct Syscalls — Bypassing ntdll

EDR products typically hook ntdll functions — they overwrite the first bytes of functions like NtCreateThreadEx with a jump to their monitoring code. Direct syscalls bypass this by placing the syscall instruction inline in the malware's own code, never touching ntdll at all.

; Direct syscall example (x64 MASM) — NtAllocateVirtualMemory
; The malware defines its own syscall stub, hardcoded with the SSN

NtAllocateVirtualMemory_Direct PROC
    mov     r10, rcx
    mov     eax, 18h        ; SSN must be correct for target OS version
    syscall
    ret
NtAllocateVirtualMemory_Direct ENDP

// C implementation using inline assembly (MSVC x64 note: no inline asm in x64 MSVC)
// Use a separate .asm file or a shellcode stub approach

// Hell's Gate — dynamically resolves SSN from ntdll at runtime
// to avoid hardcoding SSNs that break across OS versions
typedef NTSTATUS (NTAPI* pNtAllocateVirtualMemory)(
    HANDLE ProcessHandle,
    PVOID* BaseAddress,
    ULONG_PTR ZeroBits,
    PSIZE_T RegionSize,
    ULONG AllocationType,
    ULONG Protect
);

// Hell's Gate reads the SSN directly from ntdll's stub in memory
WORD GetSyscallNumber(LPCSTR funcName) {
    HMODULE ntdll = GetModuleHandleA("ntdll.dll");
    BYTE* stub = (BYTE*)GetProcAddress(ntdll, funcName);
    // If NOT hooked: mov r10,rcx (4C 8B D1) + mov eax,SSN (B8 xx xx 00 00)
    // SSN is at stub[4] as a WORD
    if (stub[0] == 0x4C && stub[1] == 0x8B && stub[2] == 0xD1 &&
        stub[3] == 0xB8) {
        return *(WORD*)(stub + 4);
    }
    // If hooked (first bytes patched to JMP): Halo's Gate scans nearby stubs
    return 0;  // simplified
}

How EDR Products Hook ntdll

Most EDR products inject a monitoring DLL into every process and hook key ntdll functions by patching their first bytes with a JMP to the EDR's code. This lets the EDR intercept every system call attempt and inspect its arguments before the call reaches the kernel.

  EDR ntdll Hook (inline hook / "trampoline hook")
  ─────────────────────────────────────────────────────────────────
  Before EDR injection (original NtCreateThreadEx stub):
  ┌────────────────────────────────────────────┐
  │ 4C 8B D1          mov r10, rcx             │
  │ B8 C1 00 00 00    mov eax, 0xC1            │
  │ 0F 05             syscall                  │
  │ C3                ret                      │
  └────────────────────────────────────────────┘

  After EDR hooks NtCreateThreadEx:
  ┌────────────────────────────────────────────┐
  │ FF 25 00 00 00 00  jmp [rip+0]             │ ← Overwrites first 14 bytes
  │ AA BB CC DD EE FF 11 22  (EDR hook addr)   │   with absolute indirect JMP
  └─── trampoline ─────────────────────────────┘
       (saved original bytes + jump back)

  Flow:
  Application calls NtCreateThreadEx()
    → JMP to EDR monitoring DLL
    → EDR logs call, inspects arguments
    → EDR decides: allow or block
    → If allow: JMP to trampoline (executes saved original bytes)
    → Trampoline: executes original stub (syscall)
    → Returns to application

  

Detecting EDR Hooks

A process can detect whether its ntdll stubs have been hooked by reading the first bytes of each stub and checking if they match the expected 4C 8B D1 B8 pattern. If the first byte is E9 (relative JMP) or FF 25 (indirect JMP), the stub has been patched by an EDR or another hook.

BOOL IsNtFunctionHooked(LPCSTR funcName) {
    HMODULE ntdll = GetModuleHandleA("ntdll.dll");
    BYTE* func = (BYTE*)GetProcAddress(ntdll, funcName);
    // Expected first bytes of unhooked NT stub:
    //   4C 8B D1 = mov r10, rcx
    //   B8 xx xx = mov eax, SSN
    if (func[0] == 0x4C && func[1] == 0x8B && func[2] == 0xD1) {
        return FALSE;  // not hooked
    }
    if (func[0] == 0xE9 || func[0] == 0xFF) {
        return TRUE;   // hooked (JMP at start)
    }
    return TRUE;  // unexpected — assume hooked
}

Q & A