Protocol Reverse Engineering

Protocol Reverse Engineering

Overview

This skill supports authorized, evidence-first protocol reverse engineering for defensive security research, interoperability debugging, and protocol documentation.

Use it to move from raw packet captures to a defensible protocol understanding:

• what transport is in use
• whether traffic is cleartext, compressed, or encrypted
• how conversations are segmented
• what fields and message types appear stable across sessions
• what remains unknown and needs validation

This skill focuses on passive analysis and documentation. It does not advise unauthorized interception, encryption bypass, intrusive man-in-the-middle setups, or active tampering unless the user’s scope explicitly authorizes security testing beyond passive observation.

Preserve provenance and raw evidence before trimming, converting, or annotating captures.

When to Use

Use this skill when you need to:

• capture and analyze an unknown or partially documented network protocol
• document message framing, fields, and state transitions from packet evidence
• troubleshoot interoperability issues between a client and server
• determine whether application data is recoverable or blocked by modern encryption
• compare multiple sessions to separate constants, counters, timestamps, lengths, and payload data
• turn packet observations into reusable documentation, field tables, or machine-readable schema notes

Do not use this skill as the primary workflow when the task is actually:

• unauthorized interception or surveillance
• exploit development or offensive protocol manipulation
• active fuzzing of a protocol implementation
• malware traffic analysis that needs a dedicated malware or incident-response workflow
• pure packet forensics where reconstruction and attribution matter more than protocol grammar extraction

Stop and ask for clarification if any of the following are missing:

• authorization to capture or inspect the traffic
• target environment and scope boundaries
• whether passive-only analysis is required
• success criteria for the reverse-engineering output
• available evidence such as pcaps, endpoint logs, or session secrets

Operating Table

Workflow

1. Confirm authorization and define scope

Record the operating boundary before analysis:

• who authorized the work
• what hosts, interfaces, or captures are in scope
• whether analysis is passive-only
• whether the environment includes NAT, proxies, TLS termination, or load balancers
• whether endpoint access, logs, or session secrets are available

Output:

• a short scope statement
• explicit non-goals
• stop conditions for out-of-scope traffic or unexpected sensitive content

2. Preserve raw evidence before analysis

Do this before trimming or filtering:

• keep the original pcap/pcapng untouched
• record capture time window, timezone/clock context, interface, host role, and capture point
• record any BPF capture filter used
• note snaplen and whether packet truncation or drops were reported

Minimum provenance note:

• capture source
• interface or mirror/span point
• filter used at capture time
• file hash if available
• analyst and timestamp

3. Validate capture quality

Before inferring protocol behavior, check whether the capture is trustworthy.

Look for:

• dropped packets
• too-small snaplen causing truncation
• checksum or segmentation offload artifacts
• asymmetric routing or partial visibility
• time skew between capture sources
• incomplete handshakes or missing connection setup

If capture quality is poor, fix collection first when possible. Many “unknown protocol” problems are actually evidence-quality problems.

4. Classify the transport and security layer

Establish the outer structure before studying the payload.

Questions to answer:

• TCP or UDP?
• Single request/response or multiplexed streams?
• Cleartext, compressed, or encrypted?
• TLS present? If so, which version indicators are visible?
• QUIC present over UDP?
• Are SNI, ALPN, certificate metadata, or handshake properties visible even if payload is not?
• Which side acts as client and which as server?

Useful clues:

• destination ports are hints, not proof
• ALPN can indicate higher-level protocols
• repeated length-prefixed records suggest framing
• stable first-byte patterns may indicate message type/version fields

Output:

• protocol classification note
• client/server role assignment
• encryption decision branch

5. Isolate conversations and reconstruct streams

Do not reason from mixed traffic if you can avoid it.

For each candidate conversation:

• isolate by 5-tuple and time window
• follow the TCP stream or equivalent reconstructed flow
• enable or verify reassembly behavior where appropriate
• distinguish application records from transport segmentation
• note retries, resets, retransmissions, and out-of-order delivery

Capture these artifacts:

• conversation ID or stream index
• start/end timestamps
• request/response sequence or bidirectional event list
• any correlation to DNS, process role, or application log entries

6. Infer framing and field structure

Work from repeated observations across multiple messages.

For each message candidate, test whether bytes likely represent:

• magic/version bytes
• message type or opcode
• length fields
• request IDs, correlation IDs, or stream IDs
• flags or capability bits
• counters, sequence numbers, or timestamps
• checksums, MACs, or cryptographic material
• variable-length payloads or nested records

Good practice:

• compare several examples of the same operation
• compare a success case and a failure case
• compare short vs long payloads to validate length assumptions
• record confidence levels rather than forcing certainty too early

Preferred evidence table columns:

7. Handle encrypted protocols correctly

If traffic is protected, document what is visible and what is not.

Allowed, defensible paths include:

• analyzing cleartext protocols directly
• using authorized endpoint-generated session secrets such as key log files where supported
• using server-side material only when it is actually applicable and authorized
• documenting handshake metadata and residual unknowns when decryption is not possible

Important boundary:

• do not assume a private key is enough to decrypt modern TLS traffic
• for TLS 1.3 and many modern deployments, private-key-only decryption is insufficient
• QUIC combines transport and crypto behavior in ways that change normal TCP-era assumptions
• this skill does not cover bypassing encryption controls

If decryption is unavailable, still document:

• handshake metadata
• SNI/ALPN visibility
• certificate or endpoint identity clues
• packet sizes, timing, burst patterns, and session boundaries