Chapter 08

x64 Calling Convention

The Microsoft x64 calling convention governs every API call your shellcode makes. Get it wrong and you get crashes that have nothing to do with the instruction that triggered them — the bug is actually three lines back in how you set up the stack. This chapter teaches the convention from the ground up: argument registers, shadow space, stack alignment, volatile versus non-volatile registers, and entry state. By the end you can write and debug any multi-argument Windows API call by hand, and you understand the exact MOVAPS crash that kills most novice shellcode on Windows.

Why the Calling Convention Matters So Much for Shellcode

When the compiler generates code, it handles the calling convention automatically. Every call instruction the compiler emits is surrounded by correct argument placement, stack adjustment, and register saving. When you write shellcode in assembly, you do all of this manually. One mistake — the wrong register for an argument, 8 bytes of missing shadow space, a misaligned stack — produces a crash deep inside a Windows API with no obvious connection to your actual bug.

The canonical example: VirtualAlloc crashes at a MOVAPS instruction 40 instructions into its prologue. The crash is not a bug in VirtualAlloc. It's your stack being 8 bytes off when you called it — the XMM register store instruction requires 16-byte alignment, and you violated that. The error message is "access violation at 0x7FF..." inside kernel32, making it look like a Windows bug when it's entirely your calling convention error.

The MOVAPS crash — what it looks like vs what caused it

  What you see in x64dbg:
  ─────────────────────────────────────────────────────────────────
  Exception: Access violation reading 0x000000001FFE4B58
  At:        KernelBase.dll + 0x1234A: MOVAPS xmmword ptr [rsp+0x10], xmm6
                                       ^^^^^^^^ this is NOT the bug

  Call stack:
    VirtualAlloc  (KernelBase)     ← crash is here
    your_shellcode + 0x48          ← this is the actual bug location
    shellcode_entry

  What actually caused it:
  ─────────────────────────────────────────────────────────────────
  In your_shellcode before the call to VirtualAlloc:
    sub rsp, 0x28    ← you forgot this, or used 0x20 (wrong) instead
                        of 0x28 (correct for proper alignment + shadow)
    call VirtualAlloc   ← RSP was 0x????FFF8 (needs to be 0x????FFF0)
                          → VirtualAlloc stores XMM registers at a
                          misaligned address → MOVAPS fault

Argument Passing — The Four Register Arguments

The first four arguments to any Windows x64 function go in these specific registers, in this specific order. No exceptions. No alternatives. This is the complete rule for the first four arguments:

x64 Windows argument register assignment

  Argument position  │ Integer/Pointer type  │ Floating-point type
  ───────────────────┼───────────────────────┼───────────────────────
  1st argument       │ RCX                   │ XMM0
  2nd argument       │ RDX                   │ XMM1
  3rd argument       │ R8                    │ XMM2
  4th argument       │ R9                    │ XMM3
  5th+ argument      │ on stack (RSP+0x28+)  │ on stack (RSP+0x28+)
  ───────────────────┼───────────────────────┼───────────────────────

  Shadow registers:
  When a function takes floating-point args, the CALLER is also expected
  to place the same value in the integer shadow position:
    XMM0 argument → also in RCX (shadow)
    XMM1 argument → also in RDX (shadow)
  This allows called functions to spill XMM args to integer shadow space.
  For the Windows API functions used in shellcode (all integer/pointer
  args), you never deal with XMM arguments.

  Examples:
  ─────────────────────────────────────────────────────────────────
  VirtualAlloc(lpAddress, dwSize, flAllocationType, flProtect)
    → RCX = lpAddress  (NULL for new allocation)
    → RDX = dwSize     (0x1000 for 4KB page)
    → R8  = flAllocationType (0x3000 = MEM_COMMIT|MEM_RESERVE)
    → R9  = flProtect  (0x40 = PAGE_EXECUTE_READWRITE)

  CreateThread(lpSecurity, dwStack, lpStart, lpParam, dwFlags, lpId)
    → RCX = lpSecurity  (NULL)
    → RDX = dwStack     (0 = default)
    → R8  = lpStart     (address of your shellcode function)
    → R9  = lpParam     (argument to pass to thread, often NULL)
    → [RSP+0x28] = dwFlags  ← 5th arg goes on stack after shadow space
    → [RSP+0x30] = lpId     ← 6th arg

; Example: calling VirtualAlloc(NULL, 0x1000, MEM_COMMIT|MEM_RESERVE, PAGE_EXECUTE_READWRITE)
;
; C prototype: LPVOID VirtualAlloc(LPVOID, SIZE_T, DWORD, DWORD)
;              → all four args fit in registers

    xor     rcx, rcx            ; arg1: lpAddress = NULL
    mov     rdx, 0x1000         ; arg2: dwSize = 4KB
    mov     r8,  0x3000         ; arg3: flAllocationType = MEM_COMMIT | MEM_RESERVE
    mov     r9,  0x40           ; arg4: flProtect = PAGE_EXECUTE_READWRITE
    sub     rsp, 0x28           ; allocate shadow space (32 bytes) + 8 for alignment
    call    qword [rel pVirtualAlloc]
    add     rsp, 0x28           ; clean up shadow space
    ; RAX = allocated memory address, or NULL on failure

; Example: calling CreateThread(NULL, 0, lpStart, NULL, 0, NULL)
;
; C prototype: HANDLE CreateThread(SECURITY_ATTRIBUTES*, SIZE_T, LPTHREAD_START_ROUTINE,
;                                   LPVOID, DWORD, LPDWORD)
; Six arguments: first four in registers, last two on stack

    xor     rcx, rcx            ; arg1: lpThreadAttributes = NULL
    xor     rdx, rdx            ; arg2: dwStackSize = 0 (default)
    lea     r8,  [rel my_func]  ; arg3: lpStartAddress = thread function
    xor     r9,  r9             ; arg4: lpParameter = NULL

    sub     rsp, 0x38           ; 32 bytes shadow + 2 stack args (16 bytes) + 8 alignment
    ; Lay stack args ABOVE the shadow space:
    mov     qword [rsp+0x20], 0     ; arg5: dwCreationFlags = 0 (run immediately)
    mov     qword [rsp+0x28], 0     ; arg6: lpThreadId = NULL
    call    qword [rel pCreateThread]
    add     rsp, 0x38

Register size matters
Using mov ecx, 0 instead of xor rcx, rcx to zero a register for a pointer argument works on x86-64 because writing to a 32-bit register zero-extends to 64 bits. However, using mov cx, 0 (16-bit) does NOT zero-extend — it leaves the upper 48 bits of RCX unchanged. Always use 32-bit or 64-bit operations when setting arguments.

Shadow Space — What It Is and Why It Exists

The Microsoft x64 ABI requires every caller to allocate 32 bytes of "shadow space" (also called "home space" or "spill space") immediately above the return address on the stack, before making any call. This space belongs to the called function — it can use it to spill its register arguments to the stack if it needs those registers for other purposes:

Stack layout immediately before CALL executes

  RSP before "sub rsp, 0x28":
  ┌────────────────────────────────────────────────────┐
  │ [RSP]      = local variables and saved registers   │  ← your stack frame
  └────────────────────────────────────────────────────┘

  After "sub rsp, 0x28":
  ┌────────────────────────────────────────────────────┐ ← RSP+0x28 (original RSP)
  │ [RSP+0x28] = (arg5 goes here if present)          │
  │ [RSP+0x20] = shadow slot for arg4 (R9)            │  ─┐
  │ [RSP+0x18] = shadow slot for arg3 (R8)            │   │ 32 bytes of shadow space
  │ [RSP+0x10] = shadow slot for arg2 (RDX)           │   │ (for the callee to use)
  │ [RSP+0x08] = shadow slot for arg1 (RCX)           │  ─┘
  │ [RSP+0x00] = (CALL pushes return address here)    │  ← RSP when callee starts
  └────────────────────────────────────────────────────┘

  The callee (VirtualAlloc, etc.) may or may not actually USE these slots.
  You must allocate them anyway — it's unconditional.
  The four slots exist for the callee's convenience. Their initial contents
  are undefined — the callee fills them if it needs to.

  Size rule: ALWAYS allocate EXACTLY 32 bytes (0x20) of shadow space.
  Then add: 8 bytes per stack argument beyond the 4th.
  Then add: 0 or 8 bytes for stack alignment (see next section).

The most common mistake: skipping shadow space entirely, or allocating only 8 bytes (one slot). The callee will blindly overwrite the 32 bytes above the return address when it spills arguments. If your local variables are in that range, they get corrupted — silently.

Stack Alignment — The 16-Byte Rule

The Microsoft x64 ABI guarantees that the stack is 16-byte aligned at the point of the CALL instruction — meaning RSP must be a multiple of 16 when the CALL executes. The CALL instruction itself pushes 8 bytes (the return address), so after the CALL, RSP inside the callee is 16-byte aligned at entry (RSP has been aligned before call, then -8, then the callee's prologue may add more).

Tracking stack alignment through function calls

  At shellcode entry point (e.g., injected shellcode's first instruction):
  ─────────────────────────────────────────────────────────────────
  RSP alignment is UNKNOWN when injected by CreateRemoteThread or
  similar methods. You MUST align it yourself before making any calls.

  Safe shellcode prologue:
    and rsp, ~0xF       ; align down to 16-byte boundary
    sub rsp, 8          ; subtract 8 so that after "sub rsp, 0x28" and
                        ; "call" we have the right alignment

  Or more commonly:
    push rax            ; push 8 bytes — if RSP was 16-aligned, now 8-aligned
    and rsp, ~0xF       ; force to 16-aligned
                        ; RSP is now 16-aligned, not 8

  ─────────────────────────────────────────────────────────────────
  When called from a properly set-up function (your own assembly):
  ─────────────────────────────────────────────────────────────────
  At function entry, RSP is 8-byte aligned (because CALL pushed 8 bytes
  and the caller had RSP 16-byte aligned before the CALL).

  A function that saves one non-volatile register (push rbx) makes
  RSP 16-aligned again. Then sub rsp, 0x28 (40 = 0x28 bytes):
    RSP was 16-aligned after push
    sub rsp, 0x28  →  16-aligned - 40 = (16-aligned - 32) - 8
    0x28 = 40 bytes = 32 shadow + 8 alignment padding
    Result: RSP is 16-byte aligned just before CALL ✓

  If you save no registers: RSP starts at -8 (from CALL). Then:
    sub rsp, 0x28  →  -8 - 40 = -48, which is 16-aligned ✓
    (0x28 = 40 bytes achieves alignment because 8 + 40 = 48 = 3×16)

  ─────────────────────────────────────────────────────────────────
  Alignment formula:
  Total_sub = 32 (shadow) + 8*N (stack args beyond 4th) + X (padding)
  where X is chosen so that (8 + Total_sub) mod 16 = 0
  (8 because CALL already pushed 8; we add enough to reach the next 16)
  Common values:
    0 stack args:  X=0,  Total_sub=0x28 (40 = 8+40=48=3×16 ✓)
    1 stack arg:   X=8,  Total_sub=0x38 (add 8 for arg, 8 for pad → 56=3.5×16? NO)
                         Actually: 8+0x38=8+56=64=4×16 ✓
    2 stack args:  X=0,  Total_sub=0x30 (48=3×16, 8+48=56? NO)
    → Recompute: shadow=32 + stack_args×8 + padding, s.t. 8+total ≡ 0 (mod 16)

; Computing the right sub rsp value for any call:
;
; Rule: after sub rsp, N, and the CALL (which pushes 8), RSP must be
;       divisible by 16.
; So:   RSP_entry - 8 (from CALL that called us) - N (our sub) - 8 (CALL we make) ≡ 0 (mod 16)
; Let ENTRY be 0 mod 16 (caller aligned before calling us):
;   0 - 8 - N - 8 ≡ 0 (mod 16)
;   -16 - N ≡ 0 (mod 16)
;   N ≡ 0 (mod 16)  — but this ignores the shadow space minimum of 32!
;
; In practice:
; If you push an ODD number of 8-byte values in your prologue (push rbx,
; or sub rsp with odd multiple of 8), you need sub rsp, 0x28 (40).
; If you push an EVEN number of 8-byte values, you need sub rsp, 0x20 (32)
; BUT 0x20 is also the shadow space minimum, so there's no padding.
;
; Simplest correct rule for shellcode with no prologue saves:
sub rsp, 0x28    ; 40 bytes = 32 shadow + 8 alignment
call target
add rsp, 0x28

; For functions where you also push/pop non-volatile registers:
push rbx         ; -8 from RSP (entry was -8 from call, now -16, i.e., aligned)
push r12         ; -8 → -24 (off by 8 from 16-alignment)
push r13         ; -8 → -32 (16-aligned again)
sub rsp, 0x28    ; -40 → RSP = entry - 72. 72 mod 16 = 8. That's -8 before the CALL, so OK!
                 ; CALL pushes 8 → entry - 80, 80 mod 16 = 0. ✓

Volatile vs Non-Volatile Registers

Registers are split into two categories. Volatile registers can be destroyed by any function call — you cannot rely on their contents surviving across a call. Non-volatile registers must be preserved — if a function uses them, it must save and restore them:

x64 Windows register classification

  VOLATILE — caller-saved (destroyed by API calls, don't rely on them)
  ─────────────────────────────────────────────────────────────────────
  RAX     Return value (and high-64 for 128-bit returns)
  RCX     Argument 1 (destroyed)
  RDX     Argument 2 (destroyed)
  R8      Argument 3 (destroyed)
  R9      Argument 4 (destroyed)
  R10     Scratch register (Win32 syscall uses this for kernel return)
  R11     Scratch register (used in prologue/epilogue by optimizer)
  XMM0–XMM5   Floating-point volatile
  ─────────────────────────────────────────────────────────────────────
  NON-VOLATILE — callee-saved (survive across API calls if you save/restore)
  ─────────────────────────────────────────────────────────────────────
  RBX     General purpose — save/restore across calls
  RBP     Frame pointer (often used as general register in shellcode)
  RDI     String destination (non-volatile in Win x64, unlike System V!)
  RSI     String source (non-volatile in Win x64, unlike System V!)
  R12     Scratch register (non-volatile)
  R13     Scratch register (non-volatile)
  R14     Scratch register (non-volatile)
  R15     Scratch register (non-volatile)
  XMM6–XMM15  Floating-point non-volatile
  ─────────────────────────────────────────────────────────────────────
  NOTE: RDI and RSI are volatile in Linux x64 (System V AMD64 ABI)!
  Shellcode originally written for Linux and ported to Windows must
  save/restore RDI and RSI around Windows API calls.

The practical rule for shellcode: store your persistent values (kernel32 base, function pointers, work-in-progress data) in non-volatile registers (RBX, R12–R15, RBP, RDI, RSI). Make sure you save them at function entry and restore them at exit if you use them. Load API arguments into volatile registers (RCX, RDX, R8, R9) right before the CALL — don't count on them surviving.

; Correct pattern: keep persistent data in non-volatile registers
; Save them at entry, load args right before each call

shellcode_main:
    push    rbp
    mov     rbp, rsp
    push    rbx              ; non-volatile: will hold k32 base
    push    r12              ; non-volatile: will hold VirtualAlloc ptr
    push    r13              ; non-volatile: will hold allocated memory
    push    r14              ; spare
    push    r15              ; spare
    sub     rsp, 0x28        ; shadow space + alignment (5 pushes = 40 bytes = aligned)

    ; Step 1: Find kernel32 (using PEB walk from Ch06)
    ; call find_kernel32 → rax = DllBase
    call    find_kernel32
    mov     rbx, rax         ; save DllBase in non-volatile RBX

    ; Step 2: Resolve VirtualAlloc
    mov     rcx, rbx         ; arg1: DllBase
    lea     rdx, [rel name_VirtualAlloc]   ; arg2: "VirtualAlloc"
    call    get_export
    mov     r12, rax         ; save VirtualAlloc ptr in non-volatile R12

    ; Step 3: Allocate executable memory
    xor     rcx, rcx         ; arg1: lpAddress = NULL
    mov     edx, 0x1000      ; arg2: dwSize = 4KB
    mov     r8d, 0x3000      ; arg3: MEM_COMMIT | MEM_RESERVE
    mov     r9d, 0x40        ; arg4: PAGE_EXECUTE_READWRITE
    call    r12              ; call VirtualAlloc (pointer in R12)
    mov     r13, rax         ; save allocated address in non-volatile R13

    ; Now RBX, R12, R13 all survived the API calls because they're non-volatile
    ; (the convention guarantees any function we called preserved them)

    add     rsp, 0x28
    pop     r15
    pop     r14
    pop     r13
    pop     r12
    pop     rbx
    pop     rbp
    ret

Return Values

Every Windows API returns its result in RAX. For pointer or HANDLE types, the full 64-bit value is in RAX. For 32-bit values (DWORD, BOOL), the value is in EAX (the lower 32 bits of RAX). Failure conventions vary by API:

; After any Windows API call:
; RAX = return value

; Common failure patterns:
; BOOL:   EAX = 0 means failure
;   test  eax, eax     → jz .failed
;   or    test  rax, rax  → same, safe for BOOL
;
; HANDLE: RAX = NULL (0) or INVALID_HANDLE_VALUE (-1 = 0xFFFFFFFFFFFFFFFF)
;   test  rax, rax        → catches NULL
;   cmp   rax, -1         → catches INVALID_HANDLE_VALUE (use separately)
;
; LPVOID (VirtualAlloc): RAX = NULL on failure
;   test  rax, rax        → jz .alloc_failed
;
; Multiple return values: some functions return a structure by pointer
; or use output parameters (e.g., CreateThread's lpThreadId parameter)
; — these go through the output pointer you passed, not through RAX.

; Example: checking VirtualAlloc result
    xor     rcx, rcx
    mov     edx, 0x1000
    mov     r8d, 0x3000
    mov     r9d, 0x40
    call    qword [rel pVirtualAlloc]
    test    rax, rax             ; NULL = failure
    jz      .alloc_failed        ; handle error
    mov     r13, rax             ; success: save the pointer

Entry Register State — What You Can't Assume

When your shellcode is invoked by an injection technique, the register state at entry is NOT a clean slate. Different injection methods leave different register contents. Assuming they're zero is a classic shellcode bug:

  Injection via CreateRemoteThread():
  ─────────────────────────────────────────────────────────────────
  RCX = lpParameter (the parameter you passed to CreateRemoteThread)
         — often your shellcode's address or a config blob
  All other registers: contents from thread startup (ntdll internals)
  RSP: 16-byte aligned at entry? — NOT guaranteed. ALIGN IT.
  Stack size: default 1MB (dwStackSize=0), but top 4KB is guard page
  Action: save RCX immediately if you need it; align RSP.

  Injection via QueueUserAPC() / NtQueueApcThread():
  ─────────────────────────────────────────────────────────────────
  RCX = the ULONG_PTR argument passed to the APC
  RDX, R8: additional APC-specific values
  RSP: NOT guaranteed aligned — ALIGN IT.
  Thread state: the hijacked thread is running and has its own stack
  Action: save RCX immediately; align RSP before any call.

  Injection via Thread Context Hijacking (SetThreadContext):
  ─────────────────────────────────────────────────────────────────
  All registers: whatever they were when the thread was suspended
                 — completely unpredictable
  RSP: wherever the hijacked thread's RSP was
  Action: ALWAYS set your own stack. Don't use the hijacked RSP.
  Pattern: xchg rsp, [rel saved_rsp]  (switch to your own stack)
           or: mov rsp, [rel my_stack_top]

  Injection via process hollowing (NtResumeThread, new process):
  ─────────────────────────────────────────────────────────────────
  Register state follows CreateProcess convention — OS-defined
  RSP: properly aligned by the OS when creating the initial thread
  Action: can trust RSP alignment from new process start, but
          still validate before your first call.

  Safe universal shellcode prologue:
  ─────────────────────────────────────────────────────────────────
  push rax             ; preserve whatever is in RAX (may matter to caller)
  and  rsp, ~0xF       ; align to 16 bytes (truncate, don't add)
  push rax             ; maintain even number of pushes for alignment
  ; ... your code ...
  pop  rax             ; restore
  pop  rax
  ret

Fifth and Sixth Arguments — Stack Placement

The fifth argument goes at [RSP+0x20] and the sixth at [RSP+0x28], after you've done the sub rsp for shadow space. The offsets are from the RSP value before the CALL:

; Calling a 6-argument function: WriteProcessMemory(hProcess, lpBase, lpBuf, nSize, lpWritten)
; Prototype: BOOL WriteProcessMemory(HANDLE hProcess, LPVOID lpBaseAddress,
;                                    LPCVOID lpBuffer, SIZE_T nSize, SIZE_T* lpNumberOfBytesWritten)
; 5 arguments → 4 in registers, 1 on stack

; The stack args must be written AFTER sub rsp, so they're in the right place when
; CALL executes. RSP after sub is the shadow space base.

    mov     rcx, hProcess        ; arg1: process handle
    mov     rdx, lpRemoteBase    ; arg2: destination address in remote process
    lea     r8,  [rel payload]   ; arg3: source buffer in our process
    mov     r9,  payload_size    ; arg4: number of bytes to write

    sub     rsp, 0x30            ; 32 shadow + 8 for 5th arg + 8 pad for alignment
                                 ; (we're saving 0 non-volatile regs before this,
                                 ;  so RSP entry was -8 from the CALL that called us.
                                 ;  -8 - 0x30 = -56. Before our CALL: -56 - 8 = -64.
                                 ;  64 mod 16 = 0. ✓)
    mov     qword [rsp+0x20], 0  ; arg5: lpNumberOfBytesWritten = NULL
    call    qword [rel pWriteProcessMemory]
    add     rsp, 0x30

Complete Worked Example — Multi-Step API Sequence

Putting it all together: allocate memory, copy shellcode into it, and create a thread to execute it. This is the canonical "inject shellcode into self" pattern, which you'll use as a test harness for shellcode development:

; inject_self.asm — allocate RWX memory, copy payload, run it in a thread
; Assumes r12 = pVirtualAlloc, r13 = pCreateThread, r14 = pWaitForSingleObject

BITS 64
section .data
    pVirtualAlloc          dq 0
    pVirtualFree           dq 0
    pCreateThread          dq 0
    pWaitForSingleObject   dq 0

section .text
global inject_self

inject_self:
    push    rbp
    mov     rbp, rsp
    push    rbx              ; will hold allocated buffer address
    push    r12              ; will hold payload size
    push    r13              ; will hold thread handle
    push    r14
    push    r15
    ; 5 pushes (8×5=40) + 8 (CALL push) = 48, so RSP is 48 below original-8 = -56.
    ; Before our sub: RSP is aligned to -8 from where it needs to be for 16-alignment.
    sub     rsp, 0x28        ; shadow + 8 pad → RSP is -56-40=-96. Before our CALL:
                             ; -96-8=-104. 104 mod 16 = 8. WRONG?
                             ; Wait: at entry RSP = X (X was 16-aligned before caller's CALL)
                             ; CALL pushed 8 → RSP = X-8
                             ; push rbp → X-16
                             ; push rbx → X-24  push r12 → X-32  push r13 → X-40
                             ; push r14 → X-48  push r15 → X-56
                             ; sub rsp, 0x28 → X-56-40 = X-96
                             ; CALL pushes 8 → RSP = X-104
                             ; 104 mod 16 = 8. Still not right.
                             ; Fix: use sub rsp, 0x20 (no extra 8, since 5 pushes = 40 bytes,
                             ; and 8+40+32=80, 80 mod 16 = 0, so before CALL: X-80-8=X-88, 88 mod 16=8. No.)
                             ; Let me just use: and rsp, ~0xF after prologue to align.
    and     rsp, ~0xF        ; force alignment after all the saves
    sub     rsp, 0x28        ; shadow space

    ; ── Step 1: VirtualAlloc(NULL, payload_size, MEM_COMMIT|MEM_RESERVE, PAGE_EXECUTE_READWRITE) ──
    xor     rcx, rcx
    lea     rdx, [rel payload_end - payload_start]   ; size (RIP-relative calculation)
    mov     r8d, 0x3000      ; MEM_COMMIT | MEM_RESERVE
    mov     r9d, 0x40        ; PAGE_EXECUTE_READWRITE
    call    qword [rel pVirtualAlloc]
    test    rax, rax
    jz      .fail
    mov     rbx, rax         ; rbx = allocated buffer

    ; ── Step 2: Copy payload into allocation (manual memcpy) ──────────────
    lea     rsi, [rel payload_start]
    mov     rdi, rbx
    lea     rcx, [rel payload_end - payload_start]
.copy_loop:
    movsb                    ; copies [rsi] → [rdi], increments both
    dec     rcx
    jnz     .copy_loop

    ; ── Step 3: CreateThread(NULL, 0, buffer, NULL, 0, NULL) ───────────────
    xor     rcx, rcx
    xor     rdx, rdx
    mov     r8,  rbx         ; lpStartAddress = our allocated shellcode
    xor     r9,  r9
    mov     qword [rsp+0x20], 0     ; dwCreationFlags = 0
    mov     qword [rsp+0x28], 0     ; lpThreadId = NULL
    call    qword [rel pCreateThread]
    test    rax, rax
    jz      .fail
    mov     r13, rax         ; r13 = thread handle

    ; ── Step 4: WaitForSingleObject(hThread, INFINITE) ────────────────────
    mov     rcx, r13
    mov     edx, 0xFFFFFFFF  ; INFINITE
    call    qword [rel pWaitForSingleObject]

.fail:
    add     rsp, 0x28
    pop     r15
    pop     r14
    pop     r13
    pop     r12
    pop     rbx
    pop     rbp
    ret

payload_start:
    ; Your shellcode bytes here (or db 0x90, 0x90, ... as placeholder)
    nop
payload_end:

Debugging Calling Convention Issues

Diagnostic checklist when a Windows API call crashes

  Symptom 1: MOVAPS crash inside a Windows API (most common)
  ─────────────────────────────────────────────────────────────────
  Cause: RSP not 16-byte aligned before CALL
  Diagnosis: Break before the CALL. Check RSP in registers pane.
             RSP mod 16 should be 0 at the CALL instruction
             (because CALL will subtract 8, making it 8-aligned within
              the callee, and the callee adds padding back to 16).
  Actually: At the CALL instruction, RSP should be 0x...0 or 0x...8?
  The ABI says RSP mod 16 == 0 BEFORE the call (at the CALL instruction
  itself), because CALL pushes 8, making callee's RSP mod 16 == 8.
  Fix: add or sub the right number of bytes to get RSP mod 16 == 0.

  Symptom 2: Wrong return value (garbage in RAX)
  ─────────────────────────────────────────────────────────────────
  Cause: You made another function call that overwrote RAX before
         you saved the previous result.
  Fix: Save RAX to a non-volatile register or stack slot immediately
       after the CALL, before any other function calls.

  Symptom 3: Correct first API call, wrong second API call
  ─────────────────────────────────────────────────────────────────
  Cause: You saved results in volatile registers (RCX, RDX, etc.)
         which were overwritten by the next API call's argument setup.
  Fix: Use non-volatile registers (RBX, R12–R15) for persistent data.

  Symptom 4: Function call goes to wrong address
  ─────────────────────────────────────────────────────────────────
  Cause: call [rel pFunc] is reading the function pointer correctly,
         but the pointer table wasn't populated (bootstrap failed silently)
  Diagnosis: Before the CALL, check the memory at [rel pFunc].
             It should hold a valid address in the DLL's .text section.
  Fix: Check bootstrap code, add NULL checks after each resolution.

Questions & Answers

Why is the shadow space 32 bytes? Why not 16 or 64?

Because there are four argument registers (RCX, RDX, R8, R9) and each is 8 bytes wide. The shadow space gives the callee exactly enough room to spill all four argument registers to the stack if it needs those registers for other purposes. This makes the ABI simpler to implement in x64 compilers and allows stack walkers to reliably find argument values in a fixed location regardless of whether the callee actually spilled them. The 32 bytes is the exact minimum to hold all four argument values, chosen to balance stack frame size against caller overhead.

Does Microsoft's RDI/RSI non-volatile convention differ from Linux?

Yes, and this is a notorious gotcha when porting shellcode between platforms. On Linux (System V AMD64 ABI), RDI and RSI are argument registers 1 and 2 (used instead of RCX/RDX) and they're volatile — functions can destroy them. On Windows (Microsoft x64 ABI), RDI and RSI are non-volatile registers that functions must preserve. If you write shellcode using RDI and RSI for persistent storage (which is natural after writing Linux shellcode), you must save/restore them in any Windows function you write. Conversely, any Windows API call you make is guaranteed not to clobber them — so they're safe to use as persistent registers. The confusion usually goes the other way: shellcode that was working on Linux gets ported to Windows and crashes because the argument setup is wrong (Linux uses RDI for arg1, Windows uses RCX).

What happens if I don't sub rsp before a call — is it always a crash?

Not always immediately. If the 32 bytes above your current RSP happen to be safe memory that nothing important uses, the callee may spill its argument registers there without corrupting anything critical, and the call returns successfully. This is why calling convention bugs are so hard to find — they're intermittent, depending on what happens to be on the stack at the moment. On a debug build with a fresh, clean stack, it might work every time. In a deeply nested shellcode call chain with data above on the stack, it corrupts something critical and crashes several calls later. Always allocate shadow space. The 8 bytes of misalignment is slightly more deterministic (MOVAPS faults on certain Windows versions fairly reliably), but shadow space omission can be silent for a while.

How do I call a function with more than 6 arguments?

The same pattern extends: args 5, 6, 7, 8, ... go at [RSP+0x20], [RSP+0x28], [RSP+0x30], [RSP+0x38], ... above the 32-byte shadow. You allocate extra stack space in your sub instruction. The alignment rule still applies: total stack space used (including shadow, extra arg slots, and your own locals) must result in RSP being 16-aligned before each CALL. Practically, Windows APIs used in shellcode rarely need more than 6 arguments. CreateThread, WriteProcessMemory, and NtAllocateVirtualMemory are the common ones that need stack arguments. If you're calling a function with 8+ arguments, you're likely calling a complex API that probably can be replaced with a simpler equivalent for shellcode purposes.

Is there a quick way to check RSP alignment in x64dbg without calculating manually?

Yes: put a breakpoint just before your CALL instruction, then look at the RSP value in the Registers pane. The last hex digit tells you everything. If it's 0 (RSP ends in 0), you're 16-byte aligned ✓. If it ends in 8, you're 8-byte aligned — the CALL will make it 0 inside the callee but the MOVAPS situation requires 16-alignment AT the CALL instruction. Actually the rule is: RSP at the point of the CALL should end in 0 (be divisible by 16). After CALL pushes 8, the callee starts with RSP ending in 8 — and the callee's prologue may push registers to realign. If your RSP at CALL ends in 8, fix it by adding 8 (with add rsp, 8 before the call) or removing one push from your prologue.