De-identification Leakage Audit

What it does

When your agent uses it

• Verifying a redaction before de-identified clinical text is exported or shared
• Gating a dataset release on zero residual identifiers
• Adding a CI check that fails the build when any identifier survives
• Getting an independent detector's view of another model's redaction output

Example prompts

• “Audit the de-identified notes in ./deid/output for leftover identifiers and block the release if anything shows up.”
• “Check this redacted discharge summary for SSNs, phone numbers and card numbers and give me a severity-scored report.”
• “Add a CI step that runs the leakage audit on our de-identified export and fails on any finding.”

Requirements

• The `openmed` Python package

Workflow steps

5 steps, taken from the first numbered list in SKILL.md.

1. 1Run deterministic format + checksum detectors on the de-identified text
2. 2Add an entropy heuristic for high-randomness tokens (long base36/base64
3. 3Run a model second-pass: re-run openmed.extract_pii on the output and
4. 4Score severity. critical (SSN, card, full DOB+name co-occurrence) > high
5. 5Block on any leak. The gate is binary for release: if findings is

What it can do on your machine

Read from SKILL.md and the folder at commit 34d7b8c. It shows what the files ask for, not the result of running them.

• Tool permissions

  Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

  From allowed-tools in the SKILL.md frontmatter.

• Runs code

  No scripts in the folder and no shell commands in SKILL.md (its code samples are python).

  From the folder's file list and the shell code blocks in SKILL.md.

• Network

  Links to these hosts (documentation or services it may open):

  • hhs.gov
  • iso.org
  • csrc.nist.gov

  From URLs in SKILL.md, links to its own repository left out.

• Credentials

  Names no API keys, tokens, secrets or passwords.

  From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

De-identification Leakage Audit loads about 1.9k tokens when it runs. Until then it costs about 188 tokens; SKILL.md has 645 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it

~188

When it runs · the whole SKILL.md, loaded when a task matches

~1.9k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from maziyarpanahi/openmed at commit 34d7b8c, republished under its Apache-2.0 licence (© maziyarpanahi). 645 words, ~1,852 tokens.

name

auditing-deid-leakage

description

Adversarially scan already-de-identified clinical text for residual identifiers and emit a leakage report that blocks release on any hit. Use after OpenMed de-identification when the user asks to verify a redaction, prove no PHI/PII leaked, gate a dataset before sharing, or run a second-pass detector. Covers format and checksum detectors (SSN, Luhn for card numbers, MRN/account patterns, emails, phones, dates), entropy heuristics for high-randomness tokens, severity scoring, and a hard block-on-leak rule. This is the verification half of OpenMed's leakage-first ethos. Hand-off: re-run openmed.extract_pii on the de-id output and diff against expectations. License-free, local-first. Pairs after deidentifying-clinical-text.

license

Apache-2.0

metadata.project

OpenMed

metadata.category

de-identification

metadata.pairs

after

metadata.version

1.0

import re
import openmed

# Synthetic — the de-identified OUTPUT we are auditing for residual leaks.
deid_text = "Patient [NAME] seen on [DATE]. Backup contact 415-555-0184; acct 4111111111111111."

def luhn_ok(digits: str) -> bool:
    nums = [int(d) for d in digits]
    nums[-2::-2] = [(2 * d - 9 if 2 * d > 9 else 2 * d) for d in nums[-2::-2]]
    return sum(nums) % 10 == 0

DETECTORS = {
    "SSN":   (r"\b\d{3}-\d{2}-\d{4}\b", "critical", None),
    "EMAIL": (r"\b[\w.+-]+@[\w-]+\.[\w.-]+\b", "high", None),
    "PHONE": (r"\b(?:\+?1[-.\s]?)?\(?\d{3}\)?[-.\s]?\d{3}[-.\s]?\d{4}\b", "high", None),
    "DATE":  (r"\b\d{1,2}[/-]\d{1,2}[/-]\d{2,4}\b", "medium", None),
    "MRN":   (r"\bMRN[:#\s]*\d{5,}\b", "high", None),
    "CARD":  (r"\b(?:\d[ -]?){13,19}\b", "critical", luhn_ok),  # checksum-gated
}

findings = []
for label, (pattern, severity, checksum) in DETECTORS.items():
    for m in re.finditer(pattern, deid_text, flags=re.IGNORECASE):
        token = m.group()
        if checksum and not checksum(re.sub(r"\D", "", token)):
            continue  # fails Luhn -> not a real card number, skip
        findings.append({"label": label, "severity": severity,
                         "start": m.start(), "end": m.end()})  # offsets, not text

# Second-pass model detector: re-run PII extraction on the de-id output.
residual = openmed.extract_pii(deid_text)          # PredictionResult
for ent in residual.entities:
    findings.append({"label": ent.label, "severity": "high",
                     "start": ent.start, "end": ent.end})

leaked = bool(findings)
print({"leak": leaked, "count": len(findings)})    # report carries NO plaintext
assert not leaked, "Release BLOCKED: residual identifiers detected."

© maziyarpanahi, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

Just SKILL.md in skills/auditing-deid-leakage of maziyarpanahi/openmed.

Open the folder on GitHubat commit 34d7b8c