Responding To Incidents

Responding to Incidents

Two questions drive every incident: how far did they get, and are they still here. Everything else — the malware, the CVE, the root cause — is supporting detail. Answer those two in order and the response follows.

When to Use

• Suspected or confirmed compromise of a host, account, or cloud tenant
• Forensic analysis of a disk image, memory capture, or log set
• Reconstructing what an attacker did and when
• Scoping blast radius and deciding containment
• Writing a postmortem or a regulator/customer-facing incident narrative

When NOT to Use

• Deep analysis of a recovered sample — use analyzing-malware
• Proactive searching with no known incident — use hunting-threats
• Deciding whether an alert is even an incident yet — use triaging-security-alerts; response begins once triage confirms a true positive
• Building the detections that would have caught it — use engineering-detections
• The incident is in AWS (exposed keys, CloudTrail/GuardDuty) — use investigating-aws-incidents; in Microsoft 365 / Entra — use investigating-m365-entra
• You have a packet capture to work through — use analyzing-network-traffic
• Service outages with no security dimension — this is a security IR skill

Route to a Depth Skill

Specific evidence types have their own procedure skill. This skill sets scope and order; reach for these when a single artifact type becomes the focus.

The Order That Matters

Preparation → Detection & Analysis → Containment → Eradication → Recovery → Lessons Learned
                      ↑__________________|
                    (re-scope after every new finding)

Two rules that are violated constantly and cost the most:

Preserve before you remediate. Rebooting, reimaging, or "just cleaning it up" destroys memory, running process state, and unflushed logs. Once gone, you cannot answer the scoping question, and you will be guessing about whether you got them out.

Do not contain half of it. Partial containment tells the attacker you have noticed and gives them time to re-establish access from the parts you missed. Scope first, then contain everything at once — unless there is active, ongoing damage, in which case stop the damage and accept the trade.

Evidence Acquisition

Order of volatility (RFC 3227 §2.1) — collect top to bottom:

registers, cache
routing table, arp cache, process table, kernel statistics, memory
temporary file systems            ← tmpfs, /dev/shm: staging lives here
disk
remote logging and monitoring data relevant to the system
physical configuration, network topology
archival media

Note that the RFC places memory, the process table, and network state in the same tier rather than ordering them against each other. Modern practice refines that: capture memory first, because the commands you would run to enumerate processes and sockets execute on the box and perturb the memory you have not captured yet. Do not skip the temporary filesystems tier — /dev/shm and tmpfs are ordinary staging locations and they do not survive the reboot that someone will inevitably suggest.

# Memory first, always, on a live suspect host
# Linux
sudo ./avml mem.lime                 # or LiME
# Windows
DumpIt.exe /OUTPUT mem.raw           # or winpmem
# macOS — do NOT reach for osxpmem. Rekall is archived, its last release was
# 2017 and Intel-only, and its kext-based approach is blocked by SIP and
# kext restrictions on Big Sur and later, and on all Apple Silicon. Full-RAM
# capture on a modern Mac realistically needs commercial tooling with the
# required Apple entitlements (e.g. Volexity Surge Collect). If none is
# available, do not stall the response: take process-scoped dumps and a
# comprehensive live-triage collection instead, and record in the incident
# log that full physical memory was not obtainable and why.

# Volatile state before you touch the disk
ps auxwwf; ss -tunap; lsof -n; last -Faiw; w
netstat -anob                        # Windows
Get-NetTCPConnection | Where State -eq Established

# Disk: image, do not analyze in place
sudo dd if=/dev/sda bs=4M conv=noerror,sync status=progress | tee image.dd | sha256sum
sudo ewfacquire /dev/sda             # E01 with built-in hashing, preferred
# Mount read-only, always via a write blocker or loop with `ro`
sudo mount -o ro,noexec,noload,loop image.dd /mnt/evidence

Chain of custody is not paperwork you add later. Record at collection time:

Evidence ID | Source host/serial | Collected by | UTC timestamp | Method/tool+version
SHA-256 at acquisition | SHA-256 at each transfer | Custodian at each handoff | Storage location

Hash immediately, verify after every copy, and never work on the original. If the incident may become litigation or a regulatory matter, involve legal before collection, not after.

Triage Collection

For most incidents, a full disk image per host is too slow. Use targeted collection at scale, then image only the hosts that matter.

# Windows: KAPE with the SANS triage target set
kape.exe --tsource C: --target !SANS_Triage --tdest E:\out --vhdx host01

# Linux/macOS: UAC or a scripted collection
./uac -p full /evidence/host01

# Cloud/EDR: pull the equivalent via API
# - EDR raw telemetry for the window ±7 days
# - Snapshot the volume before terminating any instance
aws ec2 create-snapshot --volume-id vol-xxx --description "IR-<case> preserve"

Artifact Analysis by Question

Go to the artifact that answers your question rather than processing everything.

# Memory analysis — where "are they still here" usually gets answered
vol -f mem.raw windows.pstree
vol -f mem.raw windows.malfind          # injected RWX regions
vol -f mem.raw windows.netscan
vol -f mem.raw windows.cmdline
vol -f mem.raw linux.bash               # recovered shell history

# Filesystem timeline
fls -r -m / image.dd > body.txt && mactime -b body.txt -d > timeline.csv
log2timeline.py --storage-file plaso.db image.dd && psort.py -o dynamic plaso.db > super.csv

Cloud and Identity Incidents

Most modern intrusions run through identity, not malware. Do not stop at the host.

# AWS
aws cloudtrail lookup-events --lookup-attributes AttributeKey=Username,AttributeValue=<user>
# Look for: CreateAccessKey, AttachUserPolicy, AssumeRole chains, ConsoleLogin
# without MFA, GetSecretValue, CreateTrail/StopLogging (anti-forensics)

# Azure / Entra ID
# SigninLogs: impossible travel, legacy auth, unfamiliar device
# AuditLogs: "Add service principal credentials", "Consent to application",
#            "Update conditional access policy", role assignments

# Google Workspace / GCP
# Admin audit: OAuth token grants, mail forwarding rules, delegation changes

Attacker-created OAuth applications, service principal credentials, and mail forwarding rules are the most-missed persistence in cloud incidents. Enumerate them explicitly during eradication.

Timeline Reconstruction

The timeline is the deliverable that everything else supports.

• UTC everywhere. Normalize on ingest and record the source timezone.
• Cite the source artifact for every row. An uncited timeline cannot be defended or re-derived.
• Separate observed from inferred. "Process created (Sysmon E1)" is observed; "attacker pivoted here" is inference. Mark them differently.
• Beware timestomping. $MFT $STANDARD_INFORMATION is trivially forged; $FILE_NAME is not. Disagreement between them is itself a finding.
• Establish the earliest evidence of compromise, then look earlier. The first thing you find is almost never the first thing that happened.

UTC Timestamp        | Host    | Event                                  | Source            | O/I
2026-07-12 03:14:02  | WEB01   | POST /upload.aspx 200, 1.2MB, IP x.x.x.x| IIS log           | O
2026-07-12 03:14:40  | WEB01   | w3wp.exe → cmd.exe → whoami            | Sysmon E1         | O
2026-07-12 03:15:05  | WEB01   | Initial access via upload vuln          | correlation       | I