AI Model Privacy Audit

AI Model Privacy Audit

Overview

AI model privacy auditing is the systematic assessment of whether trained ML models leak information about their training data. Models can memorize individual training records, enabling adversaries to extract personal data, determine dataset membership, reconstruct input features, or infer sensitive attributes. This skill implements a comprehensive model privacy audit methodology using established attack techniques and tools (ML Privacy Meter, ART, Foolbox) to quantify privacy leakage before deployment and periodically during operation. The audit results feed directly into the AI DPIA risk assessment and inform mitigation measure selection.

Privacy Attack Taxonomy

1. Training Data Extraction

Objective: Extract verbatim or near-verbatim records from the model's training data.

Risk Factors Increasing Extraction Likelihood:

• Large model capacity relative to training data size (overfitting)
• Training data containing duplicated or near-duplicated records
• Longer training duration (more epochs)
• Lower regularisation
• Models with high output granularity (logits, probabilities)

Testing Methodology:

1. Insert canary records with unique identifiers into training data
2. Train the model
3. Attempt extraction through various prompting strategies
4. Measure extraction success rate (percentage of canaries recovered)
5. Threshold: extraction rate should be below 0.1% for acceptable risk

2. Membership Inference

Objective: Determine whether a specific record was in the model's training set.

ML Privacy Meter Implementation:

• Population metric-based attack: compares target model's loss on a sample against population loss distribution
• Reference metric-based attack: uses reference models to compute per-sample metrics
• Shadow model attack: trains shadow models and uses the attack model to classify members

Testing Methodology:

1. Partition data: training set (members) and held-out set (non-members)
2. Run membership inference attacks using ML Privacy Meter
3. Measure attack success: true positive rate at low false positive rate (TPR@FPR=0.1%, 1%)
4. Generate ROC curves per sample and aggregate
5. Threshold: TPR@1%FPR should be below 5% for acceptable privacy

3. Model Inversion

Objective: Reconstruct input features from model outputs.

Testing Methodology:

1. Select target classes or individuals
2. Run inversion attacks with various initializations
3. Measure reconstruction quality (SSIM, PSNR for images; cosine similarity for embeddings)
4. Assess re-identification risk: can reconstructed data identify specific individuals?
5. Threshold: reconstruction similarity should be below 0.3 (SSIM) for acceptable risk

4. Attribute Inference

Objective: Infer sensitive attributes not present in the model's output.

Testing Methodology:

1. Identify sensitive attributes (Art. 9 categories) that may be correlated with model features
2. Train attack models to predict sensitive attributes from model outputs
3. Measure inference accuracy for each sensitive attribute
4. Compare against random baseline
5. Threshold: inference accuracy should not exceed random baseline + 10%

Audit Methodology

Phase 1: Audit Scoping (Days 1-3)

1. Define audit scope: which models, what deployment context, what threat model
2. Identify assets: training data, model artefacts, deployment infrastructure
3. Define threat model: who are the adversaries, what access do they have?
   • Black-box: API access only (queries and responses)
   • Grey-box: API access plus model architecture knowledge
   • White-box: Full access to model weights and architecture
4. Select attacks based on threat model and model type
5. Define success criteria (acceptable leakage thresholds)
6. Obtain audit authorisation from model owner and legal