Prompt Injection Defense

Prompt Injection Defense

Design for compromise of model reasoning. Prompt text and classifiers can reduce attack success, but they do not create a reliable security boundary. Keep consequential authority, authorization, validation, and policy enforcement outside the model.

Inputs

Collect or infer, and label assumptions for:

• Agent purpose, system/developer instructions, models, memory, and orchestration
• Every input source, including users, web pages, email, documents, images, audio, tool results, RAG, and other agents
• Tool list, privileges, identities, targets, write effects, and network egress
• Secrets, private data, system prompts, policy data, and other protected assets
• Output sinks such as UI rendering, code execution, messages, databases, and downstream agents
• Authorization model, human gates, monitoring, incident history, and risk tolerance
• Representative benign tasks and a safe evaluation environment

Do not request production secrets or malicious artifacts in chat. Use redacted samples or synthetic fixtures.

Output contract

Deliver:

1. A data-flow and trust-boundary map covering protected assets, all modalities, sinks, memory stores, agent hops, approved destinations, and credential boundaries
2. A threat model listing protected assets, attacker-controlled channels, injection paths, and security invariants
3. A prioritized defense plan that maps each path to preventive, limiting, detective, and recovery controls, each marked missing, planned, implemented, or verified
4. Code or configuration changes only within the user's authorized scope
5. A regression suite with safe direct, indirect, stored, encoded, cross-agent, and multimodal cases as applicable
6. Verification evidence, observed failures, and metrics rather than a blanket claim of prevention
7. Residual risk, operational monitoring, and an incident containment/recovery plan

Describe the architecture in a JSON boundary manifest and lint it with scripts/audit_boundary_manifest.py. Record every control's enforcement point, owner, evidence IDs, test IDs, and expiry when time-limited. A passing structural lint is not evidence that controls work. Read references/defense-patterns.md for attack paths, control placement, and verification patterns.

Workflow

1. Map instructions, data, authority, and sinks

Trace content from origin through parsing, retrieval, model context, memory, tools, renderers, output sinks, and downstream agents. Mark every attacker-controlled or mixed-trust source. Include hidden document text, metadata, code comments, OCR, images, audio, redirects, tool descriptions, and persisted memory. Inventory approved and denied destinations, each credential's holder/audience/storage boundary, and every point where content or authority crosses agents.

List assets and consequences: secret disclosure, private-data access, unauthorized tool calls, external communications, transactions, code execution, policy bypass, corrupted memory, or misleading output.

2. Define enforceable invariants

Express requirements in terms a deterministic component can enforce, for example:

• Retrieved content cannot grant permissions or change the tool allowlist.
• A support agent cannot read records outside the authenticated tenant.
• An email body cannot determine recipients for a send operation.
• Model output cannot execute as code or HTML without validation and safe handling.
• Secrets unavailable to the task never enter model context.

If an invariant exists only as a prompt instruction, record it as weak and move enforcement to code, policy, isolation, or human control.

3. Reduce exposed authority

Remove unused tools, broad tokens, ambient credentials, generic shells, arbitrary URL fetches, raw SQL, and unrestricted file access. Split read from write and preview from commit. Restrict identities by tenant, object, action, fields, time, and destination.

Keep secrets outside model context and tool results. Add network and data egress allowlists. Sandbox code, parsers, browsers, and file processing. Require independent authorization and, where warranted, action-specific approval immediately before consequential effects. A tool classified critical must not have confirmation mode none.

4. Separate control from untrusted content

Treat untrusted content as quoted data with provenance, never as authority. Preserve source boundaries through retrieval and agent handoffs. Use structured typed messages instead of concatenating instructions and data. Limit retrieved content, strip active content when safe, normalize supported formats, and render outputs with context-appropriate escaping.

Instruction hierarchy, delimiters, reminders, content classifiers, and injection detectors can be defense-in-depth signals. Do not depend on any of them as the sole control.

5. Validate every transition

Validate tool arguments against narrow schemas and policy before execution. Derive sensitive target identifiers from trusted application state rather than untrusted text where possible. Reauthorize at execution time. Validate and encode model outputs for their destination; never send them directly to shells, SQL, templates, URLs, or privileged APIs.

For multi-agent systems, authenticate senders, constrain delegation depth and budgets, pass structured claims with provenance, and recalculate permissions at each hop. Never inherit the broadest upstream privilege implicitly.

6. Test with safe adversarial cases

Use an isolated environment, synthetic accounts, benign canary secrets, inert destinations, and non-destructive tools. Test at least:

• Direct attempts to override instructions or elicit protected data
• Indirect instructions embedded in retrieved pages, email, documents, tool results, metadata, and memory
• Obfuscation, encoding, language changes, splitting across turns, and repeated attempts
• Cross-agent delegation and tainted summaries
• Hidden or alternate-modal content when images, audio, PDFs, HTML, or OCR are supported
• Tool-argument manipulation, target substitution, data exfiltration, and unauthorized egress
• False positives on representative benign content and normal task completion

Measure invariant violations, unauthorized tool attempts, canary exposure, successful benign tasks, false-positive rate, and containment behavior. A detector pass rate alone is insufficient. Promote a control to verified only when implementation evidence and named test evidence both exist; use implemented when code exists but the relevant tests have not established behavior.