Evaluating With Leakage Gates

What it does

When your agent uses it

• The user wants to run the OpenMed eval harness on a synthetic golden set
• Decide whether a de-id model is RELEASABLE
• Enforce direct-identifier recall floors
• Require zero critical leakage

Example prompts

• “release gate”
• “leakage”
• “is this model safe to ship”
• “/evaluating-with-leakage-gates”

Requirements

• Python 3
• A credential in API_KEY
• A credential in OPENMED_RELEASE_GATE_KEY

Workflow steps

7 steps, taken from the first numbered list in SKILL.md.

1. 1Build a synthetic golden suite. Fixtures are JSON with text and
2. 2Fit calibration thresholds for any policy that masks or replaces
3. 3Run the suite (run_suite / run_benchmark) to get a BenchmarkReport.
4. 4Evaluate with ReleaseGate(...).evaluate(report).
5. 5Read the per-gate results. Each GateCheck carries gate, passed,
6. 6Fail closed. Treat anything other than RELEASABLE as a hard stop.
7. 7Audit subgroups with fairness_report (see auditing-subgroup-fairness)

What it can do on your machine

Read from SKILL.md and the folder at commit 34d7b8c. It shows what the files ask for, not the result of running them.

• Tool permissions

  Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

  From allowed-tools in the SKILL.md frontmatter.

• Runs code

  Shell commands in SKILL.md call:

  • python

  From the folder's file list and the shell code blocks in SKILL.md.

• Network

  Links to these hosts (documentation or services it may open):

  • hhs.gov
  • csrc.nist.gov
  • doi.org

  From URLs in SKILL.md, links to its own repository left out.

• Credentials

  Names these keys or tokens, usually read from environment variables:

  • API_KEY
  • OPENMED_RELEASE_GATE_KEY

  From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Evaluating With Leakage Gates loads about 2k tokens when it runs. Until then it costs about 163 tokens; SKILL.md has 655 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it

~163

When it runs · the whole SKILL.md, loaded when a task matches

~2k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from maziyarpanahi/openmed at commit 34d7b8c, republished under its Apache-2.0 licence (© maziyarpanahi). 655 words, ~2,033 tokens.

name

evaluating-with-leakage-gates

description

Evaluate an OpenMed de-identification or clinical NER model against the leakage-first release gates G1a through G8, which gate releases on residual PHI leakage rather than on F1. Use when the user wants to run the OpenMed eval harness on a synthetic golden set, decide whether a de-id model is RELEASABLE or QUARANTINED, enforce direct-identifier recall floors, require zero critical leakage, fit calibration thresholds, or produce a signed gate report. Trigger on "release gate", "leakage", "is this model safe to ship", "G1a", "G3", "quarantine", "recall floor", or "calibration thresholds" in an OpenMed de-id context.

license

Apache-2.0

metadata.project

OpenMed

metadata.category

evaluation-quality

metadata.pairs

adjacent

metadata.version

1.0

from openmed.eval import run_suite, ReleaseGate, RELEASABLE

# 1) Produce a candidate BenchmarkReport from a SYNTHETIC fixtures file.
#    Each fixture carries gold PHI spans; no real patient text is committed.
report = run_suite(
    "eval/golden/phi_synthetic.json",     # user-supplied synthetic fixtures
    suite="golden",
    model_name="OpenMed/Privacy-PII-Detection",
    device="cpu",
    metadata={
        "family": "PII",
        "tier": "base",
        "policy": "hipaa_safe_harbor",
        # calibration artifacts are required for mask/replace policies (see below)
        "thresholds_path": "eval/artifacts/thresholds.json",
        "calibration_report_path": "eval/artifacts/calibration_report.json",
    },
)

# 2) Gate it. The gate reads the last-green baseline store read-only and
#    returns a signed GateReport.
gate = ReleaseGate(milestone="v1.6", policy="hipaa_safe_harbor")
decision = gate.evaluate(report)

print(decision.decision)                  # "RELEASABLE" or "QUARANTINED"
for check in decision.gate_results:
    if not check.passed:
        print(check.gate, "->", check.reason, check.details)

assert decision.decision == RELEASABLE, "do not ship a quarantined model"

python -m openmed.eval.release_gates \
  --candidate eval/out/candidate_report.json \
  --milestone v1.6 --policy hipaa_safe_harbor \
  --output release-gate-report.json

© maziyarpanahi, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

Just SKILL.md in skills/evaluating-with-leakage-gates of maziyarpanahi/openmed.

Open the folder on GitHubat commit 34d7b8c