Gating Deid Leakage

What it does

When your agent uses it

• The user wants a pytest test
• CLI step that exits nonzero on de-id regression
• Wants to wire OpenMeds leakage-first release gates into GitHub Actions / CI
• Needs a recall floor plus zero-leakage assertion against a synthetic held-out set

Example prompts

• “CI gate”
• “fail the build”
• “regression test”
• “/gating-deid-leakage”

Requirements

• Python 3

Workflow steps

6 steps, taken from the first numbered list in SKILL.md.

1. 1Curate a synthetic held-out PHI set and commit it (offsets + labels, no
2. 2Pick the floor from policy. Don't hardcode a magic number you invented —
3. 3Run the suite → gate the report (pytest fixture above).
4. 4Assert two invariants: critical_leakage_count == 0 and per-label recall
5. 5Make it required. Mark the job a required status check so a red gate
6. 6On failure, the harness CLI can open/refresh a tracking issue

What it can do on your machine

Read from SKILL.md and the folder at commit 34d7b8c. It shows what the files ask for, not the result of running them.

• Tool permissions

  Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

  From allowed-tools in the SKILL.md frontmatter.

• Runs code

  Shell commands in SKILL.md call:

  • python

  From the folder's file list and the shell code blocks in SKILL.md.

• Network

  Links to these hosts (documentation or services it may open):

  • hhs.gov
  • csrc.nist.gov
  • docs.github.com

  From URLs in SKILL.md, links to its own repository left out.

• Credentials

  Names no API keys, tokens, secrets or passwords.

  From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Gating Deid Leakage loads about 1.7k tokens when it runs. Until then it costs about 162 tokens; SKILL.md has 554 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it

~162

When it runs · the whole SKILL.md, loaded when a task matches

~1.7k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from maziyarpanahi/openmed at commit 34d7b8c, republished under its Apache-2.0 licence (© maziyarpanahi). 554 words, ~1,743 tokens.

name

gating-deid-leakage

description

Add a CI gate that fails the build when an OpenMed de-identification model's recall on a held-out PHI set drops below threshold or any critical identifier leaks. Use when the user wants a pytest test or CLI step that exits nonzero on de-id regression, wants to wire OpenMed's leakage-first release gates into GitHub Actions / CI, needs a recall floor plus zero-leakage assertion against a synthetic held-out set, or wants to block merges that weaken de-identification. Trigger on "CI gate", "fail the build", "regression test", "de-id recall threshold", "block the merge", "exit nonzero", or "leakage check in CI" for OpenMed.

license

Apache-2.0

metadata.project

OpenMed

metadata.category

evaluation-quality

metadata.pairs

adjacent

metadata.version

1.0

# tests/eval/test_deid_leakage_gate.py
import pytest
from openmed.eval import run_suite, ReleaseGate, RELEASABLE

RECALL_FLOOR = 0.99          # direct-identifier recall floor
HELD_OUT = "eval/heldout/phi_synthetic.json"   # SYNTHETIC, committed

@pytest.fixture(scope="module")
def gate_report():
    report = run_suite(
        HELD_OUT,
        suite="golden",
        model_name="OpenMed/Privacy-PII-Detection",
        device="cpu",
        metadata={"family": "PII", "tier": "base", "policy": "hipaa_safe_harbor"},
    )
    return ReleaseGate(milestone="v1.6", policy="hipaa_safe_harbor").evaluate(report)

def test_no_critical_leakage(gate_report):
    # Hard zero: one leaked SSN/credit-card is a breach, full stop.
    assert gate_report.critical_leakage_count == 0, "critical PHI leaked"

def test_recall_floor(gate_report):
    low = {
        label: r
        for label, r in gate_report.per_label_recall.items()
        if r < RECALL_FLOOR
    }
    assert not low, f"recall below floor: {low}"

def test_releasable(gate_report):
    # The structural decision: any failed gate -> QUARANTINED -> red build.
    failed = [c.gate for c in gate_report.gate_results if not c.passed]
    assert gate_report.decision == RELEASABLE, f"quarantined; failed gates: {failed}"

# Produce a candidate report, then gate it. Nonzero exit blocks the job.
python -m openmed.eval.release_gates \
  --candidate eval/out/candidate_report.json \
  --baseline-store eval/baselines/last_green.json \
  --milestone v1.6 --policy hipaa_safe_harbor \
  --output release-gate-report.json

# .github/workflows/deid-gate.yml
name: de-id leakage gate
on: [pull_request]
jobs:
  gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v5
      - run: uv pip install --system -e ".[hf]"
      - name: Run de-id leakage gate
        run: uv run pytest tests/eval/test_deid_leakage_gate.py -q
      # The job fails (red) automatically if pytest exits nonzero.

© maziyarpanahi, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

Just SKILL.md in skills/gating-deid-leakage of maziyarpanahi/openmed.

Open the folder on GitHubat commit 34d7b8c