Anti Debugging Techniques

SKILL: Anti-Debugging Techniques — Detection & Bypass Playbook

AI LOAD INSTRUCTION: Expert anti-debug techniques across Linux and Windows. Covers ptrace, PEB flags, NtQueryInformationProcess, timing attacks, signal-based detection, TLS callbacks, VEH tricks, and all corresponding bypass methods. Base models often miss the distinction between user-mode and kernel-mode detection and the correct patching strategy for each.

0. RELATED ROUTING

• code-obfuscation-deobfuscation when the binary also uses control flow flattening, VM protection, or string encryption
• vm-and-bytecode-reverse when the anti-debug sits inside a custom VM dispatcher
• symbolic-execution-tools when you want to symbolically skip anti-debug checks entirely

Advanced Reference

Also load ANTI_DEBUG_MATRIX.md when you need:

• Complete cross-reference matrix of technique × OS × detection method × bypass method
• Per-technique reliability ratings and false-positive notes
• Tool compatibility chart (GDB, x64dbg, WinDbg, Frida, ScyllaHide)

Quick bypass picks

1. LINUX ANTI-DEBUG TECHNIQUES

1.1 ptrace(PTRACE_TRACEME)

The classic self-attach: a process calls ptrace(PTRACE_TRACEME, 0, 0, 0). If a debugger is already attached, the call fails (returns -1).

if (ptrace(PTRACE_TRACEME, 0, 0, 0) == -1) {
    exit(1); // debugger detected
}

Bypass methods:

1.2 /proc/self/status — TracerPid

FILE *f = fopen("/proc/self/status", "r");
// parse TracerPid: if non-zero → debugger attached

Bypass: Mount a FUSE filesystem over /proc/self, or LD_PRELOAD hook fopen/fread to filter TracerPid to 0.

1.3 Timing Checks (rdtsc / clock_gettime)

Measures elapsed time between two points; debugger single-stepping causes noticeable delay.

rdtsc
mov ebx, eax       ; save low 32 bits
; ... protected code ...
rdtsc
sub eax, ebx
cmp eax, 0x1000    ; threshold
ja  debugger_detected

Bypass: Set hardware breakpoint after second rdtsc, modify eax to pass the comparison. Or use Frida to replace the timing function.

1.4 Signal-Based Detection (SIGTRAP)

volatile int caught = 0;
void handler(int sig) { caught = 1; }
signal(SIGTRAP, handler);
raise(SIGTRAP);
if (!caught) exit(1); // debugger swallowed the signal

When a debugger is attached, SIGTRAP is consumed by the debugger rather than delivered to the handler. Bypass: In GDB, use handle SIGTRAP nostop pass to forward the signal.

1.5 /proc/self/maps & LD_PRELOAD Detection

Checks for injected libraries or memory regions characteristic of debuggers/instrumentation.

FILE *f = fopen("/proc/self/maps", "r");
while (fgets(buf, sizeof(buf), f)) {
    if (strstr(buf, "frida") || strstr(buf, "LD_PRELOAD"))
        exit(1);
}

Bypass: Hook fopen("/proc/self/maps") to return a filtered version, or rename Frida's agent library.

1.6 Environment Variable Checks

Some protections check for LD_PRELOAD, LINES, COLUMNS (set by GDB's terminal), or debugger-specific env vars.

Bypass: Unset suspicious env vars before launch, or hook getenv().

2. WINDOWS ANTI-DEBUG TECHNIQUES

2.1 IsDebuggerPresent / CheckRemoteDebuggerPresent

if (IsDebuggerPresent()) ExitProcess(1);

BOOL debugged = FALSE;
CheckRemoteDebuggerPresent(GetCurrentProcess(), &debugged);
if (debugged) ExitProcess(1);

Bypass: Hook kernel32!IsDebuggerPresent to return 0, or patch PEB directly.

2.2 PEB Flags

mov rax, gs:[0x60]    ; PEB
movzx eax, byte [rax+0x02]  ; BeingDebugged
test eax, eax
jnz debugger_detected

Bypass: Zero all four fields. ScyllaHide does this automatically.

2.3 NtQueryInformationProcess

Bypass: Hook ntdll!NtQueryInformationProcess to return clean values per info class.

2.4 Hardware Breakpoint Detection

CONTEXT ctx;
ctx.ContextFlags = CONTEXT_DEBUG_REGISTERS;
GetThreadContext(GetCurrentThread(), &ctx);
if (ctx.Dr0 || ctx.Dr1 || ctx.Dr2 || ctx.Dr3)
    ExitProcess(1);

Bypass: Hook GetThreadContext to zero DR0–DR3, or use NtSetInformationThread(ThreadHideFromDebugger) preemptively (ironically, the anti-debug technique itself).

2.5 INT 2D / INT 3 / UD2 Exception Tricks

INT 2D is the kernel debug service interrupt. Without a debugger, it raises STATUS_BREAKPOINT; with a debugger, behavior differs (byte skipping).

xor eax, eax
int 2dh
nop          ; debugger may skip this byte
; ... divergent execution path ...

Bypass: Handle in VEH or patch the interrupt instruction.

2.6 TLS Callbacks

TLS callbacks execute before main() / WinMain(). Anti-debug checks placed here run before the debugger's initial break.

Bypass: In x64dbg, set "Break on TLS Callbacks" option. In WinDbg, use sxe ld to break on module load.

2.7 NtSetInformationThread(ThreadHideFromDebugger)

NtSetInformationThread(GetCurrentThread(), ThreadHideFromDebugger, NULL, 0);

After this call, the thread becomes invisible to the debugger — breakpoints and single-stepping stop working silently.

Bypass: Hook NtSetInformationThread to NOP when ThreadInfoClass == 0x11.