anti-debugging-techniques

SKILL: Anti-Debugging Techniques — Detection & Bypass Playbook

AI LOAD INSTRUCTION: Expert anti-debug techniques across Linux and Windows. Covers ptrace, PEB flags, NtQueryInformationProcess, timing attacks, signal-based detection, TLS callbacks, VEH tricks, and all corresponding bypass methods. Base models often miss the distinction between user-mode and kernel-mode detection and the correct patching strategy for each.

0. RELATED ROUTING

• code-obfuscation-deobfuscation when the binary also uses control flow flattening, VM protection, or string encryption
• vm-and-bytecode-reverse when the anti-debug sits inside a custom VM dispatcher
• symbolic-execution-tools when you want to symbolically skip anti-debug checks entirely

Advanced Reference

Also load ANTI_DEBUG_MATRIX.md when you need:

• Complete cross-reference matrix of technique × OS × detection method × bypass method
• Per-technique reliability ratings and false-positive notes
• Tool compatibility chart (GDB, x64dbg, WinDbg, Frida, ScyllaHide)

Quick bypass picks

1. LINUX ANTI-DEBUG TECHNIQUES

1.1 ptrace(PTRACE_TRACEME)

The classic self-attach: a process calls ptrace(PTRACE_TRACEME, 0, 0, 0). If a debugger is already attached, the call fails (returns -1).

if (ptrace(PTRACE_TRACEME, 0, 0, 0) == -1) {
    exit(1

Bypass methods:

1.2 /proc/self/status — TracerPid

Bypass: Mount a FUSE filesystem over /proc/self, or LD_PRELOAD hook fopen/fread to filter TracerPid to 0.

1.3 Timing Checks (rdtsc / clock_gettime)

Measures elapsed time between two points; debugger single-stepping causes noticeable delay.

Bypass: Set hardware breakpoint after second rdtsc, modify eax to pass the comparison. Or use Frida to replace the timing function.

1.4 Signal-Based Detection (SIGTRAP)

When a debugger is attached, SIGTRAP is consumed by the debugger rather than delivered to the handler. Bypass: In GDB, use handle SIGTRAP nostop pass to forward the signal.

1.5 /proc/self/maps & LD_PRELOAD Detection

Checks for injected libraries or memory regions characteristic of debuggers/instrumentation.

Bypass: Hook fopen("/proc/self/maps") to return a filtered version, or rename Frida's agent library.

1.6 Environment Variable Checks

Some protections check for LD_PRELOAD, LINES, COLUMNS (set by GDB's terminal), or debugger-specific env vars.

Bypass: Unset suspicious env vars before launch, or hook getenv().

2. WINDOWS ANTI-DEBUG TECHNIQUES

2.1 IsDebuggerPresent / CheckRemoteDebuggerPresent

Bypass: Hook kernel32!IsDebuggerPresent to return 0, or patch PEB directly.

2.2 PEB Flags

Bypass: Zero all four fields. ScyllaHide does this automatically.

2.3 NtQueryInformationProcess

Bypass: Hook ntdll!NtQueryInformationProcess to return clean values per info class.

2.4 Hardware Breakpoint Detection

Bypass: Hook GetThreadContext to zero DR0–DR3, or use NtSetInformationThread(ThreadHideFromDebugger) preemptively (ironically, the anti-debug technique itself).

2.5 INT 2D / INT 3 / UD2 Exception Tricks

INT 2D is the kernel debug service interrupt. Without a debugger, it raises STATUS_BREAKPOINT; with a debugger, behavior differs (byte skipping).

Bypass: Handle in VEH or patch the interrupt instruction.

2.6 TLS Callbacks

TLS callbacks execute before main() / WinMain(). Anti-debug checks placed here run before the debugger's initial break.

Bypass: In x64dbg, set "Break on TLS Callbacks" option. In WinDbg, use sxe ld to break on module load.

2.7 NtSetInformationThread(ThreadHideFromDebugger)

After this call, the thread becomes invisible to the debugger — breakpoints and single-stepping stop working silently.

Bypass: Hook NtSetInformationThread to NOP when ThreadInfoClass == 0x11.

2.8 VEH-Based Detection

Registers a Vectored Exception Handler that checks EXCEPTION_RECORD for debugger-specific behavior (single-step flag, guard page violations with debugger semantics).

Bypass: Understand the VEH logic and ensure the exception chain behaves identically to non-debugged execution.

3. ADVANCED MULTI-LAYER TECHNIQUES

3.1 Self-Debugging (fork + ptrace)

The process forks a child that attaches to the parent via ptrace. If an external debugger is already attached, the child's ptrace fails.

Bypass: Patch the fork() return or kill/detach the watchdog child.

3.2 Multi-Process Debugging Detection

Parent and child cooperatively check each other's debug state, creating a mutual-watch pattern.

Bypass: Attach to both processes (GDB follow-fork-mode, or two debugger instances).

3.3 Timing-Based with Multiple Checkpoints

Distributes timing checks across multiple functions, comparing cumulative drift. Single patches fail because the total still exceeds threshold.

Bypass: Frida Interceptor.replace all timing sources (rdtsc, clock_gettime, QueryPerformanceCounter) to return controlled values.

3.4 Nanomite / INT3 Patching

Original conditional jumps are replaced with INT3 (0xCC). A parent debugger process handles each INT3, evaluates the condition, and sets the child's EIP accordingly.

Bypass: Reconstruct the original jump table by tracing all INT3 handlers, then patch the binary.

4. COUNTERMEASURE TOOLS

5. SYSTEMATIC BYPASS METHODOLOGY

6. DECISION TREE

7. CTF & REAL-WORLD PATTERNS

Common CTF Anti-Debug Patterns

Real-World Protections

8. QUICK REFERENCE — BYPASS CHEAT SHEET

Linux One-Liners

Frida Anti-Debug Bypass (Cross-Platform)

x64dbg ScyllaHide Quick Setup

1. Plugins → ScyllaHide → Options
2. Check: PEB BeingDebugged, NtGlobalFlag, HeapFlags
3. Check: NtQueryInformationProcess (all classes)
4. Check: NtSetInformationThread (HideFromDebugger)
5. Check: GetTickCount, QueryPerformanceCounter
6. Apply → restart debugging session