Agent Red Teaming

Agent Red Teaming

Find exploitable control failures without creating uncontrolled harm. Treat written authorization and rules of engagement as prerequisites for execution, not paperwork to complete afterward.

Inputs

Collect:

• Named target owner and explicit authorization for the exact systems to be tested
• Target identifiers, environment, accounts, endpoints, models, versions, and a reproducible configuration digest
• Start/end time, tester identities, source addresses, rate and cost limits, and emergency contact
• In-scope objectives and out-of-scope systems, tenants, data, techniques, and effects
• Agent architecture, tools, privileges, memory, retrieval, handoffs, identities, and external integrations
• Protected assets, security requirements, prior incidents, existing controls, and expected benign tasks
• Approved synthetic data, canary values, test destinations, cleanup plan, and evidence-handling rules

If target-specific authorization or scope is missing, stop at a non-executable assessment plan. Do not probe a live target to infer scope.

Output contract

Deliver:

1. Signed-off or explicitly pending rules of engagement with scope, constraints, stop conditions, contacts, and cleanup duties
2. A system and privilege map plus prioritized threat hypotheses
3. A machine-readable, owner-approved campaign plan with unique case IDs, targets, environment, configuration digest, tester subjects, authorization reference, time window, limits, stop conditions, cleanup duties, protected invariants, and safe oracles
4. Execution records tied to an approved case ID and unique test ID, with matching target/configuration, timestamps, observed limits, structured evidence, and cleanup traceability
5. Deduplicated findings with reproducibility, evidence, impact, likelihood, preconditions, root control failure, and remediation
6. Plan-denominator campaign metrics that distinguish passes, failures, blocked cases, errors, missing records, and tests not run
7. Retest results, residual risk, cleanup confirmation, and any incident or scope deviation

Use assets/campaign-plan-template.md for the human-readable working plan and assets/campaign-plan-template.json for the machine-readable authorization record. Read references/test-taxonomy.md when selecting cases. Validate and summarize JSONL results against the approved JSON plan with score_campaign.py PLAN.json RESULTS.jsonl.

Workflow

1. Establish rules of engagement

Verify owner, authority, authorization reference, exact targets, environment, configuration digest, tester subjects, time window, allowed techniques, prohibited actions, rate/cost ceilings, data-handling requirements, stop conditions, emergency contact, and cleanup owner. Separate production from staging explicitly. Mark each case approved: true only after the owner-approved plan contains it.

Use unique synthetic accounts and inert destinations. Define benign canary values that are recognizable but grant no access. Confirm how to disable tools, revoke test credentials, restore fixtures, and report an unexpected effect before testing begins.

2. Map the attack surface and authority

Trace every path through user input, system instructions, retrieval, memory, tools, code execution, browsers, MCP or plugins, other agents, human approvals, and output sinks. Build a privilege graph showing identities, scopes, tenants, objects, networks, and delegation.

Prioritize hypotheses by credible impact and exposed authority, not novelty. Write each hypothesis as: attacker-controlled source + control weakness + attempted action + observable safe oracle.

3. Build a safe test matrix

Cover relevant categories:

• Direct, indirect, stored, encoded, and multimodal instruction attacks
• Tool misuse, target substitution, excessive agency, unsafe chaining, and external effects
• Authentication, authorization, tenant isolation, approval binding, and confused-deputy paths
• Sensitive-data disclosure, output handling, egress, and canary exposure
• Memory, RAG, training-data, configuration, tool-metadata, and dependency poisoning
• Cross-agent impersonation, delegation escalation, tainted summaries, and cascading failure
• Resource exhaustion and cost amplification within strict budgets
• Monitoring, containment, credential revocation, and recovery

Include benign controls and normal tasks to measure false positives and retained utility. Use one primary variable per case where possible. Avoid weaponized payloads when an inert instruction, fake secret, or mock tool proves the same control failure.

4. Execute incrementally

Begin with offline or mocked components, then staging, then any separately authorized higher-risk environment. Run low-impact cases first. Capture campaign and authorization references, approved case ID, unique test ID, tester subject, target and environment, configuration digest, start/end/record timestamps, input provenance, tool trace, policy decisions, observed rate/cost/time, result, structured evidence objects, and cleanup status.

Respect rate, cost, and time limits. Do not evade monitoring or controls outside the approved hypothesis. Pause after any unexpected cross-tenant access, real secret, external effect, service degradation, or scope ambiguity.

5. Triage findings

Reproduce safely, then distinguish:

• A confirmed invariant violation
• A blocked attack showing the expected control
• A test harness or environment error
• An observation needing more evidence

Rate severity from demonstrated impact, likelihood, prerequisites, affected scope, detectability, and reversibility. Do not rate from prompt wording alone. A pass requires the protected invariant to hold; a failure requires it not to hold and cannot be informational. Blocked, errored, and not-run records have no invariant verdict and no finding severity. Deduplicate findings by root control failure and retain affected variants as evidence.

For every finding, provide the minimum safe reproduction, expected versus observed behavior, evidence, affected configuration, immediate containment, durable remediation, detection opportunity, and a regression case.

6. Remediate and retest

Prefer architectural fixes: reduce privilege, enforce authorization outside the model, constrain tools and egress, isolate untrusted content, validate outputs, bind approvals, and protect memory provenance. Prompts and detectors may add defense in depth but should not be the sole fix for consequential effects.

Retest the original case, close variants, and representative benign tasks. Record whether the finding is fixed, partially mitigated, accepted, transferred, or open, with owner and evidence.