Investigating AWS Incidents

Investigating AWS Incidents

In AWS an incident is reconstructed from logs the attacker usually could not delete -- CloudTrail, VPC Flow Logs, and the control plane's own record of every API call. The investigation is therefore a log-correlation exercise anchored on the compromised principal: the access key or assumed role, and the sequence of API calls it made. Find the principal, pull every event it generated, and the rest of the intrusion falls out of the timeline.

When to Use

• Exposed or leaked AWS access keys -- committed to a repo, in a pastebin, or flagged by AWS abuse
• Anomalous CloudTrail activity -- API calls from unfamiliar regions, IPs, or user agents
• A GuardDuty finding -- credential exfiltration, tor traffic, crypto-mining, or anomalous IAM behavior
• Unexpected IAM changes -- new users, keys, roles, login profiles, or policy attachments nobody authorized
• Crypto-mining EC2 -- a spike in on-demand instances, GPU families, or an unexpected billing alert
• S3 data exfiltration -- GetObject spikes, buckets made public, or shared snapshots

When NOT to Use

• The incident is in Microsoft 365 / Entra ID, not AWS -- use investigating-m365-entra
• The incident is in Azure resources/subscriptions -- use investigating-azure-incidents
• The incident is in Google Cloud -- use investigating-gcp-incidents
• The general IR process and host-level response -- use responding-to-incidents
• You are the attacker, not the responder -- use exploiting-cloud-platforms
• The pivot is specifically into Kubernetes on EKS -- use attacking-eks-gke-aks
• Proactive hunting with no confirmed incident -- use hunting-threats

First-Hour Triage

Three actions, in order: identify the principal, pull its recent activity, preserve before you contain.

Scope the compromised principal. Whether the report is a leaked key, a GuardDuty finding, or a billing spike, resolve it to a single principal ARN -- an IAM user, a role, or the root account. That ARN is the anchor for everything that follows.

# What is this key/session, and does it still work?
aws sts get-caller-identity                       # if you hold the suspect creds
# Resolve an access key ID to its owner
aws iam get-access-key-last-used --access-key-id AKIA...
# Enumerate the principal's current footprint
aws iam list-access-keys --user-name <user>
aws iam list-attached-user-policies --user-name <user>
aws iam list-user-policies --user-name <user>

Pull recent activity from the principal. Event history (below) is the fastest first look; the S3 log bucket is the source of truth.

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=Username,AttributeValue=<user> \
  --start-time 2026-07-01T00:00:00Z --max-results 200

# By access key -- catches role sessions that lookup-by-username misses
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIA...

Isolate without tipping off -- snapshot then contain. An attacker who sees their key deactivated mid-operation will burn persistence you have not found yet. For anything but active, ongoing damage: preserve evidence first (snapshot volumes, export logs), map persistence, then contain everything at once. If there is live damage -- active mining, active exfil -- stop the damage and accept the trade.

CloudTrail Deep-Dive

Event history vs. the S3 log bucket. The console Event history is searchable but covers only 90 days of management events and drops data events. The CloudTrail S3 log bucket (or CloudTrail Lake) is the authoritative record and the only place with S3/Lambda data events -- query it, not the console, once you are past the first look. Confirm what is actually logged:

aws cloudtrail describe-trails
aws cloudtrail get-trail-status --name <trail>          # IsLogging: true?
aws cloudtrail get-event-selectors --trail-name <trail> # data events on?

userIdentity types tell you what you are looking at:

For AssumedRole, sessionContext.sessionIssuer.arn names the role and sessionContext.attributes.mfaAuthenticated tells you whether MFA was used.

Anomaly fields to grep:

• sourceIPAddress -- external IPs on role credentials that should only run inside the VPC (the IMDS theft signature, below). Correlate against VPC Flow Logs.
• userAgent -- aws-cli/2.x, Boto3, python-requests, or anything with kali on a principal that normally shows console or SDK-from-Lambda agents.
• awsRegion -- calls in regions you do not operate in (attackers spin up mining in ap-* / sa-* to dodge attention). Enumeration touches many regions fast.
• errorCode -- a storm of AccessDenied / UnauthorizedOperation is enumeration: the attacker is mapping what the stolen principal can do.

-- Athena over the CloudTrail S3 bucket: AccessDenied enumeration storm
SELECT eventname, count(*) AS n
FROM cloudtrail_logs
WHERE useridentity.accesskeyid = 'AKIA...'
  AND errorcode = 'AccessDenied'
  AND eventtime > '2026-07-01T00:00:00Z'
GROUP BY eventname ORDER BY n DESC;

Canonical Attacker API Patterns

Grep the timeline for this sequence. It is the shape of nearly every stolen-key intrusion:

• GetCallerIdentity -- often the very first call. The attacker is confirming what the key is and who owns the account.
• Enumeration -- List* / Get* / Describe* across IAM, S3, EC2, RDS, Secrets Manager, often with an AccessDenied storm.
• Privilege escalation -- CreateUser, CreateAccessKey, AttachUserPolicy (watch for AdministratorAccess), PutUserPolicy (inline policy so it does not show in attached-policy lists), CreateLoginProfile (a console password on an API-only account), UpdateLoginProfile.
• iam:PassRole abuse -- passing a high-privilege role to a new EC2 instance, Lambda, or Glue job to inherit its permissions. Look for PassRole paired with RunInstances / CreateFunction.
• Trust-policy tampering -- UpdateAssumeRolePolicy or CreateRole with a trust policy naming an external account (a cross-account backdoor).
• STS chaining -- AssumeRole into a more privileged role, then another, building a chain that launders the original stolen key.

# Pull the IAM mutating calls for the window
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=CreateAccessKey
# Repeat for: CreateUser, AttachUserPolicy, PutUserPolicy, CreateLoginProfile,
#             UpdateAssumeRolePolicy, CreateRole, AssumeRole
# (iam:PassRole is a permission, not an event -- find it in requestParameters of
#  RunInstances / CreateFunction, not via lookup-by-EventName)

Persistence Hunting

Enumerate every mechanism explicitly during eradication -- rotating the one leaked key does nothing about a second one the attacker created.

• New IAM users / keys / roles created inside the incident window.
• A second access key on an existing user -- CreateAccessKey on the victim or another account; a user may have up to two keys.
• Console login profiles on service accounts -- CreateLoginProfile on a principal that never used the console is high-signal.
• Modified role trust policies -- UpdateAssumeRolePolicy adding an external principal or sts:AssumeRole from another account.
• Lambda backdoors -- a new or updated function that mints credentials or re-creates users, often on an EventBridge schedule.
• EventBridge / CloudWatch Events rules invoking attacker Lambda on a cron.
• iam:CreateServiceLinkedRole and other quietly-privileged roles.

# Users and keys created recently
aws iam list-users --query 'Users[?CreateDate>=`2026-07-01`].[UserName,CreateDate]'
for u in $(aws iam list-users --query 'Users[].UserName' --output text); do
  aws iam list-access-keys --user-name "$u" \
    --query 'AccessKeyMetadata[].[UserName,AccessKeyId,CreateDate]' --output text
done

# Roles whose trust policy names an external account
aws iam list-roles --query 'Roles[].[RoleName,AssumeRolePolicyDocument]'

# Recently modified Lambda and the EventBridge rules that fire them
aws lambda list-functions --query 'Functions[].[FunctionName,LastModified]'
aws events list-rules --query 'Rules[].[Name,ScheduleExpression,State]'

IMDS / SSRF Credential Theft Signatures

The classic AWS escalation: an SSRF or foothold on an EC2 instance reads the Instance Metadata Service (http://169.254.169.254/latest/meta-data/iam/...), lifts the instance role's temporary credentials, and uses them elsewhere.

The signature is unmistakable in CloudTrail: the instance role's session credentials appear from a sourceIPAddress that is not the instance. The role is minted for the instance, so any call from an external or unrelated IP means the credentials left the box.

-- Role session creds used from outside the VPC
SELECT eventtime, eventname, sourceipaddress, useragent
FROM cloudtrail_logs
WHERE useridentity.arn LIKE '%assumed-role/<instance-role>%'
  AND sourceipaddress NOT LIKE '10.%'
  AND sourceipaddress NOT LIKE '172.%'
ORDER BY eventtime;

IMDSv1 (a simple GET, no session token) makes this trivial and is itself a finding -- confirm whether the instance enforces IMDSv2 (HttpTokens: required) via aws ec2 describe-instances. Correlate the theft window with VPC Flow Logs for the outbound SSRF and the reuse source.

GuardDuty Finding Triage

GuardDuty is a starting pistol, not the investigation. Each finding type maps to a hypothesis you confirm in CloudTrail:

aws guardduty list-findings --detector-id <id> \
  --finding-criteria '{"Criterion":{"severity":{"Gte":4}}}'
aws guardduty get-findings --detector-id <id> --finding-ids <id>

Findings older than the CloudTrail window still carry the principal and IPs -- pivot on those even when the raw events have aged out of Event history.

Data-Theft Detection

• S3 GetObject spikes -- a jump in GetObject volume/bytes on sensitive buckets, visible only if S3 data events are logged. Check request counts by principal.
• Buckets made public -- PutBucketPolicy / PutBucketAcl granting AllUsers or AuthenticatedUsers, or PutPublicAccessBlock disabling the block.
• Snapshot sharing -- ModifySnapshotAttribute or ModifyImageAttribute adding an external account or all to the volume/AMI (steal data by sharing the snapshot out).
• RDS exfil -- ModifyDBSnapshotAttribute sharing a DB snapshot, or StartExportTask dumping a snapshot to an attacker-controlled S3 bucket.

SELECT eventname, useridentity.arn, sourceipaddress,
       json_extract_scalar(requestparameters, '$.attributeType') AS attr
FROM cloudtrail_logs
WHERE eventname IN ('ModifySnapshotAttribute','ModifyImageAttribute',
                    'ModifyDBSnapshotAttribute','PutBucketPolicy','PutBucketAcl')
  AND eventtime > '2026-07-01T00:00:00Z';