Detecting Model Extraction Attacks

Detecting Model Extraction Attacks

Authorized Use Only: The extraction, inversion, and membership-inference techniques described here are intended for defenders testing their own models and for red teams operating under written authorization. Querying a third-party model to clone it, reconstruct its training data, or infer membership without permission may violate terms of service, copyright, and privacy law.

Overview

Model extraction is the family of attacks in which an adversary abuses a model's inference API to steal value that the model owner intended to keep private. MITRE ATLAS catalogs these under AML.T0024 — Exfiltration via AI Inference API, in the Exfiltration tactic, with three sub-techniques:

• AML.T0024.000 — Infer Training Data Membership (membership inference): the adversary determines whether a specific record was part of the training set, a privacy violation that can expose, for example, whether a patient's record trained a medical model.
• AML.T0024.001 — Invert AI Model (model inversion): the adversary reconstructs representative training inputs (e.g., faces, text) by exploiting confidence scores returned by the API.
• AML.T0024.002 — Extract ML Model (model stealing): the adversary repeatedly queries the victim model, collects (input, prediction) pairs, and trains a surrogate model offline that mimics the victim's decision boundary — avoiding the per-query cost of a Machine-Learning-as-a-Service offering and stealing the owner's intellectual property.

All three share a common signal: an attacker must send many queries, often crafted to probe the decision boundary (high-entropy, near-boundary, synthetic, or systematically grid-sampled inputs), and frequently requests full confidence vectors / logits rather than just the top label. Detection therefore centers on per-principal query monitoring, input-distribution analysis, and confidence-exposure controls, while defense centers on rate limiting, output perturbation, and reducing the information returned per query. This skill follows the MITRE ATLAS technique definition for AML.T0024 (https://atlas.mitre.org/techniques/AML.T0024) and the NIST AI RMF MEASURE function (MEASURE-2.6, security and resilience of the AI system).

When to Use

• When you operate a model behind a public or partner inference API and need to detect cloning, inversion, or membership inference.
• When performing a pre-deployment AI red-team exercise to measure how many queries are needed to extract your own model.
• When validating that rate limiting, output perturbation, and confidence-suppression controls actually reduce extractability.
• When investigating anomalous billing/usage spikes that may indicate surrogate-model harvesting.
• When responding to a privacy incident where membership inference against a model is suspected.

Prerequisites

• Python 3.9+ environment.
• Access to inference-API access logs (per-API-key/per-principal query counts, timestamps, input features or hashes, returned confidence vectors).
• For self-assessment red-teaming, install the Adversarial Robustness Toolbox (ART), the reference framework for extraction/inference attacks and defenses:

  pip install adversarial-robustness-toolbox scikit-learn numpy

• Optional: access to the target model object (white/grey-box) or only its API (black-box).
• Authorization to test the target model.

Objectives

• Instrument the inference API to record per-principal query volume, input diversity, and confidence-exposure.
• Build a detector that scores principals for extraction-like behavior (volume, near-boundary sampling, full-vector requests).
• Run an ART-based extraction attack against your own model to measure fidelity vs. query budget.
• Run a membership-inference attack to quantify training-data leakage.
• Apply and validate defenses: rate limiting, label-only responses, confidence rounding/perturbation, and prediction poisoning.

MITRE ATT&CK Mapping

Workflow