AI Risk Management

AI Risk Management — Beyond Security, the Whole Model Lifecycle

prompt-injection covers the AI security slice — attackers manipulating LLM inputs. This skill covers everything else risk-related about deploying AI / ML systems: governance, fairness, robustness, transparency, monitoring, incident response specific to AI failures, third-party model risk, and compliance with the emerging AI regulatory landscape.

The framing is NIST AI RMF 1.0 (released 2023) — the most widely-adopted voluntary framework — plus the regulatory layer (EU AI Act, US executive orders, sector-specific guidance). Use this skill when you are deploying AI features beyond a chatbot wrapper, when a regulator asks "how do you govern your AI," or when something has gone wrong with an AI system in production.

Cross-references: prompt-injection for prompt-injection / LLM-specific security attacks; threat-modeling for design-time AI risk modeling; incident-triage and breach-patterns for AI-related incident response patterns; csf-mapping for the broader governance frame that AI RMF sits within.

The NIST AI RMF — four functions

Just like the cybersecurity framework, the AI RMF organizes the work into functions. Same shape, different content.

The framework is voluntary but increasingly cited in contracts, RFPs, executive orders, and emerging regulations. Treat it as the lingua franca of AI risk.

Workflow

Step 1 — Inventory AI systems

Before assessment, build the inventory. Most organizations underestimate how much AI they actually deploy.

For each, record: vendor (if any), training data source, deployment context, who it affects, the decision it informs, how decisions are reviewed.

Step 2 — MAP: assess the context per system

For each AI system in the inventory, answer:

• Purpose — what is this system's stated goal? Does the actual deployment match?
• Stakeholders — who interacts with it, who is affected by its decisions, who is in a position to challenge those decisions?
• Legal / regulatory context — is this in scope for a specific law? (EU AI Act high-risk categories, US HUD fair-housing rules, EEOC for employment AI, FTC for unfair / deceptive practices, sector laws)
• Failure modes — what does "broken" look like? (Wrong answer, biased answer, hallucinated answer, slow answer, expensive answer, refused-to-answer-something-it-should, answered-something-it-should-not)
• Reversibility — when this system makes a wrong call, can the decision be undone? (Mortgage denial: hard to undo. Spam filter: easy)

Step 3 — MEASURE: evaluate the system

The categories of evaluation, with the engineering hooks for each:

Accuracy / performance

• Test set evaluation — held-out data, not the training data
• Performance on slices of data, not just aggregate (the system that's 95% accurate overall may be 60% accurate on the demographic that's most impacted)
• Confusion matrices for classification; quantile-based error analysis for regression
• For LLMs: task-specific evals (HELM, MMLU, custom evals) — and especially custom evals on the application's actual prompts

Fairness / bias

• Demographic parity — does the system produce similar outcomes across protected classes?
• Equalized odds — are false-positive and false-negative rates similar across groups?
• Calibration — when the system says "80% likely," is that actually 80% across all groups?
• Individual fairness — do similar inputs produce similar outputs?

These metrics often conflict — you cannot maximize all of them simultaneously. The MAP step should have decided which is most important for the use case. For hiring AI, equalized odds matters more than demographic parity. For loan approval, the choice depends on whose interests dominate.

Tooling: Fairlearn (Microsoft), AI Fairness 360 (IBM), What-If Tool (Google), Aequitas (University of Chicago), fairlearn.metrics, aif360.metrics.

Robustness

• Adversarial inputs — perturbations that flip predictions (Foolbox, ART for traditional ML)
• Distribution shift — does the model degrade when the input distribution changes (it will, eventually)?
• Stress testing — extreme but plausible inputs

For LLMs:

• Prompt injection (see prompt-injection)
• Jailbreaks (DAN-style, role-play, encoded instructions, multi-turn manipulation)
• Indirect prompt injection (untrusted content the model reads)
• Output stability across paraphrased prompts

Explainability / transparency

• Local explanations — why did the model make this decision? SHAP, LIME, integrated gradients
• Global explanations — what features matter overall to the model?
• Model cards — Google's documentation pattern for ML models. Includes intended use, performance metrics, training data, limitations, ethical considerations
• System cards — for LLM-integrated systems, a longer-form version describing the entire AI pipeline

A model that cannot be explained at all is a model you cannot defend in a regulatory inquiry. For high-impact decisions, explainability is not optional.

Privacy

• Does the model leak training data? (Membership-inference attacks, training-data-extraction attacks for LLMs)
• Are inputs / outputs containing PII appropriately scoped (see privacy-engineering)?
• For LLM fine-tuning: are PII redaction passes applied to training data?

Security

See prompt-injection — prompt injection, indirect injection, agent privilege boundaries, MCP security. Output to the AI RMF assessment is the security posture summary.

Step 4 — MANAGE: treat the risks

For each material risk surfaced in MEASURE: