Analyzing Binaries

Analyzing Binaries

Reverse engineering is hypothesis testing against a program you cannot read. The cost of the job is dominated by how much code you look at, so the whole discipline is about narrowing: triage first, find the interesting few percent, then read that carefully.

When to Use

• Understanding what an unknown or undocumented executable does
• Recovering an algorithm, file format, or wire protocol from a binary
• Locating a vulnerability in a closed-source target
• Firmware analysis for embedded and IoT devices
• Reversing challenges in CTFs

When NOT to Use

• Live malware with intent to detonate — use analyzing-malware, which covers containment and safe detonation. Come back here for the disassembly.
• A raw shellcode blob with no headers (position-independent payload, stager, egg-hunter) — use analyzing-shellcode
• Source is available — use auditing-code-for-vulnerabilities
• Android/iOS app assessment as a whole — use testing-mobile-applications; use this skill for the native .so/Mach-O components inside it
• A framework runtime rather than a plain binary — the toolchain is specific and generic RE will not get there: reversing-flutter-apps for libapp.so/Dart snapshots, reversing-unity-il2cpp for libil2cpp.so plus global-metadata.dat, reversing-react-native-apps for Hermes bytecode
• A language runtime with its own symbol recovery — analyzing-go-binaries (pclntab survives stripping), analyzing-rust-binaries (panic strings leak source paths), analyzing-dotnet-assemblies (IL decompiles to C#). Reaching for generic RE on these wastes most of the effort
• A packed or protected executable — use unpacking-protected-binaries first; there is nothing to disassemble until it is dumped
• A whole firmware image to extract before any RE — use analyzing-firmware-images for binwalk, filesystem carving, and cross-architecture emulation; return here for an individual binary inside it

Triage First — Never Open a Disassembler Cold

Every minute here saves an hour in the decompiler.

file target && du -h target
# Architecture, endianness, PIE, stripped or not — all decide your tooling
readelf -hSd target        # ELF: headers, sections, dynamic deps
rabin2 -I target           # radare2's normalized summary of any format
objdump -p target          # PE/ELF imports and load config

# Protections tell you what the author expected
checksec --file=target     # NX, canary, RELRO, PIE, Fortify

# Strings, but read them for structure rather than skimming
strings -n 8 -t x target | less        # ASCII with offsets
strings -e l -n 8 target               # UTF-16LE, essential on Windows

What triage should answer before you disassemble:

# Entropy scan finds packed or embedded-blob regions
binwalk -E target
# Unpack the common case
upx -d target -o target.unpacked

Go and Rust binaries are usually not stripped in the ways that matter. Recover symbols before doing anything else — it changes the job from hours to minutes.

# Go: recover function names and types
GoReSym -t -p target > syms.json    # or the redress / IDAGolangHelper plugins
# Rust: demangle
nm -C target 2>/dev/null | head

Static Analysis Workflow

Pick one tool and go deep; switching tools mid-analysis loses your annotations.

# Ghidra headless: batch import, auto-analyze, run a script
analyzeHeadless /proj MyProj -import target -postScript Decompile.java

# radare2 / rizin interactive
r2 -AA target
# aaa            analyze everything
# afl            list functions, sorted by size — big ones first
# axt @ sym.f    cross-references TO a function (who calls this?)
# pdg @ main     decompile with ghidra plugin (r2ghidra)
# iz / izz       strings in data / whole binary
# /x deadbeef    search for a byte pattern

# Binary Ninja / IDA headless equivalents exist; the workflow is identical

Navigate by evidence, not by address order. The three entry points that find the interesting code fastest:

1. Strings → xrefs. Find a message you saw at runtime, cross-reference it, land in the function that produced it.
2. Imports → xrefs. Cross-reference recv, CreateProcess, fopen, EVP_EncryptInit to find the code that does the thing you care about.
3. Entropy/constants. Crypto constants (AES S-box, SHA-2 round constants, MD5 magic) are recognizable; binwalk, findcrypt, and YARA rules locate them.

Then read outward from that anchor. Rename every function and variable as you work out what it does — a decompiler listing you have annotated is a completely different artifact from a raw one.

Recognizing Structure in Decompiler Output

The decompiler gives you C-shaped noise. What you are looking for:

• Loop with an index into a byte array and an XOR — obfuscation or a homebrew cipher. Extract the key, decode offline.
• A switch on a small integer read from input — command dispatch. This is usually the protocol, and it is the map for everything else.
• memcpy with a length that came from the input — start of a memory safety review; see auditing-code-for-vulnerabilities.
• Repeated [rax + 8*n] accesses on the same base — a struct. Define it in the tool; the listing collapses to readable code.
• A call through a register right after a table load — vtable or callback dispatch. Recover the table to recover the class.

Dynamic Analysis

Static tells you what the code can do; dynamic tells you what it does. Run untrusted binaries only in an isolated VM with no host shares and networking under your control — see analyzing-malware for the containment procedure.

# Syscall and API-level behaviour
strace -f -e trace=network,file,process -o trace.log ./target
ltrace -f ./target
# Windows equivalents: API Monitor, Procmon, drltrace

# Debugging
gdb -q ./target       # with pwndbg/GEF: `checksec`, `vmmap`, `heap`, `telescope`
lldb ./target         # macOS
x64dbg / WinDbg       # Windows

# Instrumentation — the highest-leverage dynamic technique
frida-trace -f ./target -i 'recv*' -i 'EVP_*'
# then edit the generated JS handlers to dump buffers and patch return values

Frida is the fastest route through anti-debugging, custom crypto, and license checks: hook the function after decryption rather than defeating the obfuscation that protects it.

Emulation for firmware and isolated routines:

qemu-arm -L /usr/arm-linux-gnueabi ./target      # user-mode
# Unicorn for a single function: map memory, set registers, run, read result
# angr for symbolic execution when you need an input that reaches a state