Agent skill

Benchmark Pii Recall

by maziyarpanahi in maziyarpanahi/openmed

Benchmark an OpenMed PII model with synthetic gold spans and report label-aware exact-span and grapheme recall without emitting identifier surfaces.

Apache-2.0Auto-check passedAI & LLM Engineering

Install Benchmark Pii Recall

skills CLI

$ npx skills add maziyarpanahi/openmed --skill benchmark-pii-recall -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI

$ gh skill install maziyarpanahi/openmed benchmark-pii-recall --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy

$ git clone --depth 1 https://github.com/maziyarpanahi/openmed.git skills-src && mkdir -p .claude/skills && cp -r skills-src/skills/benchmark-pii-recall .claude/skills/benchmark-pii-recall && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Paste into Claude Code

Install the "benchmark-pii-recall" agent skill from https://github.com/maziyarpanahi/openmed/tree/master/skills/benchmark-pii-recall into .claude/skills/benchmark-pii-recall/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "benchmark-pii-recall", then confirm the skill loads.

Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

$skill-installer

$skill-installer install https://github.com/maziyarpanahi/openmed/tree/master/skills/benchmark-pii-recall

Type this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.

skills CLI

$ npx skills add maziyarpanahi/openmed --skill benchmark-pii-recall -a codex

Project install goes to .agents/skills/; add -g for ~/.codex/skills/.

GitHub CLI

$ gh skill install maziyarpanahi/openmed benchmark-pii-recall --agent codex

Project scope by default (.agents/skills/); add --scope user for a personal install.

Manual copy

$ git clone --depth 1 https://github.com/maziyarpanahi/openmed.git skills-src && mkdir -p .agents/skills && cp -r skills-src/skills/benchmark-pii-recall .agents/skills/benchmark-pii-recall && rm -rf skills-src

Use ~/.agents/skills/ instead of .agents/skills for a personal install.

Codex skills documentation · loads skills from .agents/skills/

Paste into Codex

Install the "benchmark-pii-recall" agent skill from https://github.com/maziyarpanahi/openmed/tree/master/skills/benchmark-pii-recall into .agents/skills/benchmark-pii-recall/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "benchmark-pii-recall", then confirm the skill loads.

Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

skills CLI

$ npx skills add maziyarpanahi/openmed --skill benchmark-pii-recall -a cursor

Project install goes to .agents/skills/; add -g for ~/.cursor/skills/.

GitHub CLI

$ gh skill install maziyarpanahi/openmed benchmark-pii-recall --agent cursor

Project scope by default (.agents/skills/); add --scope user for a personal install.

Manual copy

$ git clone --depth 1 https://github.com/maziyarpanahi/openmed.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/skills/benchmark-pii-recall .cursor/skills/benchmark-pii-recall && rm -rf skills-src

Use ~/.cursor/skills/ instead of .cursor/skills for a personal install.

Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/

Paste into Cursor

Install the "benchmark-pii-recall" agent skill from https://github.com/maziyarpanahi/openmed/tree/master/skills/benchmark-pii-recall into .cursor/skills/benchmark-pii-recall/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "benchmark-pii-recall", then confirm the skill loads.

Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

gemini skills install

$ gemini skills install https://github.com/maziyarpanahi/openmed.git --path skills/benchmark-pii-recall

--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.

skills CLI

$ npx skills add maziyarpanahi/openmed --skill benchmark-pii-recall -a gemini-cli

Project install goes to .agents/skills/; add -g for ~/.gemini/skills/.

GitHub CLI

$ gh skill install maziyarpanahi/openmed benchmark-pii-recall --agent gemini-cli

Project scope by default (.agents/skills/); add --scope user for a personal install.

Manual copy

$ git clone --depth 1 https://github.com/maziyarpanahi/openmed.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/skills/benchmark-pii-recall .gemini/skills/benchmark-pii-recall && rm -rf skills-src

Use ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.

Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/

Paste into Gemini CLI

Install the "benchmark-pii-recall" agent skill from https://github.com/maziyarpanahi/openmed/tree/master/skills/benchmark-pii-recall into .gemini/skills/benchmark-pii-recall/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "benchmark-pii-recall", then confirm the skill loads.

Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

GitHub CLI

$ gh skill install maziyarpanahi/openmed benchmark-pii-recall

Installs for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).

skills CLI

$ npx skills add maziyarpanahi/openmed --skill benchmark-pii-recall -a github-copilot

Project install goes to .agents/skills/; add -g for ~/.copilot/skills/.

Manual copy

$ git clone --depth 1 https://github.com/maziyarpanahi/openmed.git skills-src && mkdir -p .github/skills && cp -r skills-src/skills/benchmark-pii-recall .github/skills/benchmark-pii-recall && rm -rf skills-src

Use ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.

GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/

Paste into GitHub Copilot

Install the "benchmark-pii-recall" agent skill from https://github.com/maziyarpanahi/openmed/tree/master/skills/benchmark-pii-recall into .github/skills/benchmark-pii-recall/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "benchmark-pii-recall", then confirm the skill loads.

GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

skills CLI

$ npx skills add maziyarpanahi/openmed --skill benchmark-pii-recall -a opencode

OpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.

GitHub CLI

$ gh skill install maziyarpanahi/openmed benchmark-pii-recall --agent opencode

Project scope by default (.agents/skills/); add --scope user for a personal install.

Manual copy

$ git clone --depth 1 https://github.com/maziyarpanahi/openmed.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/skills/benchmark-pii-recall .opencode/skills/benchmark-pii-recall && rm -rf skills-src

Use ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.

OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/

Paste into OpenCode

Install the "benchmark-pii-recall" agent skill from https://github.com/maziyarpanahi/openmed/tree/master/skills/benchmark-pii-recall into .opencode/skills/benchmark-pii-recall/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "benchmark-pii-recall", then confirm the skill loads.

OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

Facts

Skill name: benchmark-pii-recall
GitHub stars: 5.5k
Token cost: ~1k tokens
SKILL.md length: 187 words
Files: 1
Skills in repo: 74
Repo updated: Oct 11, 2026
First seen: Oct 7, 2026
Licence: Apache-2.0

At a glance

Benchmark an OpenMed PII model with synthetic gold spans and report label-aware exact-span and grapheme recall without emitting identifier surfaces.

Works in 6 steps: Build synthetic fixtures with exact… → Include direct identifiers, boundary… → Run extract_pii at the candidate… → …
An agent must compare a model
SKILL.md covers Procedure, Runnable synthetic benchmark, Release gates and Repository example
Calls python

What it does

Benchmark Pii Recall is an agent skill from maziyarpanahi/openmed. Benchmark an OpenMed PII model with synthetic gold spans and report label-aware exact-span and grapheme recall without emitting identifier surfaces. Use when an agent must compare a model, threshold, backend, or quantized artifact and enforce a recall floor before release.

Its SKILL.md is about 1k tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts.

It sits in AI & LLM Engineering. The repository describes itself as: Local-first healthcare AI: clinical NER and HIPAA PII de-identification on hardware you control. 2,200+ medical models, 35 model-backed PII languages, and Python, MLX, Android… The licence is Apache-2.0.

When your agent uses it

An agent must compare a model
Quantized artifact and enforce a recall floor before release

Example prompts

“/benchmark-pii-recall”

Requirements

Python 3

Workflow steps

6 steps, taken from the first numbered list in SKILL.md.

Build synthetic fixtures with exact offsets and canonical PII labels.
Include direct identifiers, boundary cases, languages/scripts, and the
Run extract_pii at the candidate threshold.
Normalize prediction labels and score each document separately.
Aggregate counts only; do not persist raw text or identifier surfaces.
Fail the release when the recall floor or zero-critical-leak requirement is

What it can do on your machine

Read from SKILL.md and the folder at commit 34d7b8c. It shows what the files ask for, not the result of running them.

Tool permissions
Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.
From allowed-tools in the SKILL.md frontmatter.
Runs code
Shell commands in SKILL.md call:
- python
From the folder's file list and the shell code blocks in SKILL.md.
Network
No URLs in SKILL.md.
From URLs in SKILL.md, links to its own repository left out.
Credentials
Names no API keys, tokens, secrets or passwords.
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Benchmark Pii Recall loads about 1k tokens when it runs. Until then it costs about 74 tokens; SKILL.md has 187 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it: ~74
When it runs · the whole SKILL.md, loaded when a task matches: ~1k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from maziyarpanahi/openmed at commit 34d7b8c, republished under its Apache-2.0 licence (© maziyarpanahi). 187 words, ~1,030 tokens.

Download SKILL.mdSave it as .claude/skills/benchmark-pii-recall/SKILL.md (or your agent's skills folder).

name: benchmark-pii-recall
description: Benchmark an OpenMed PII model with synthetic gold spans and report label-aware exact-span and grapheme recall without emitting identifier surfaces. Use when an agent must compare a model, threshold, backend, or quantized artifact and enforce a recall floor before release.

Benchmark PII recall

Measure PII recall before optimizing F1, size, or latency. A missed direct identifier is a privacy failure even when aggregate F1 improves.

Procedure

Build synthetic fixtures with exact offsets and canonical PII labels.
Include direct identifiers, boundary cases, languages/scripts, and the target device or quantization.
Run extract_pii at the candidate threshold.
Normalize prediction labels and score each document separately.
Aggregate counts only; do not persist raw text or identifier surfaces.
Fail the release when the recall floor or zero-critical-leak requirement is not met.

Runnable synthetic benchmark

Install the model runtime first with python -m pip install "openmed[hf]".

python

from openmed import extract_pii
from openmed.core.labels import normalize_label
from openmed.eval import compute_character_recall, compute_exact_span_f1

MODEL = "OpenMed/OpenMed-PII-SuperClinical-Small-44M-v1"
RECALL_FLOOR = 0.99
FIXTURES = [
    {
        "text": (
            "Call the synthetic clinic at 212-555-0198 or email "
            "demo.patient@example.test."
        ),
        "spans": [
            ("PHONE", "212-555-0198"),
            ("EMAIL", "demo.patient@example.test"),
        ],
    },
    {
        "text": (
            "The synthetic callback number is 415-555-0136 and the contact "
            "address is sample.user@example.test."
        ),
        "spans": [
            ("PHONE", "415-555-0136"),
            ("EMAIL", "sample.user@example.test"),
        ],
    },
]

true_positives = false_positives = false_negatives = 0
covered_graphemes = total_graphemes = 0

for fixture in FIXTURES:
    text = fixture["text"]
    gold = []
    for label, surface in fixture["spans"]:
        start = text.index(surface)
        gold.append(
            {"start": start, "end": start + len(surface), "label": label}
        )

    result = extract_pii(
        text,
        model_name=MODEL,
        confidence_threshold=0.5,
        lang="en",
    )
    predicted = [
        {
            "start": entity.start,
            "end": entity.end,
            "label": normalize_label(entity.label),
        }
        for entity in result.entities
        if entity.start is not None and entity.end is not None
    ]

    exact = compute_exact_span_f1(gold, predicted, source_text=text)
    recall = compute_character_recall(gold, predicted, source_text=text)
    true_positives += exact.true_positives
    false_positives += exact.false_positives
    false_negatives += exact.false_negatives
    covered_graphemes += int(recall.numerator)
    total_graphemes += int(recall.denominator)

exact_recall = true_positives / max(true_positives + false_negatives, 1)
grapheme_recall = covered_graphemes / max(total_graphemes, 1)
print(
    {
        "documents": len(FIXTURES),
        "exact_span_recall": exact_recall,
        "grapheme_recall": grapheme_recall,
        "false_positives": false_positives,
        "false_negatives": false_negatives,
    }
)
assert grapheme_recall >= RECALL_FLOOR, "PII recall floor not met"

Release gates

Require zero misses for critical direct identifiers even if aggregate recall passes.
Report per-label, language, script, section, and device slices.
Compare quantized and full-precision outputs; reject recall regressions.
Add hard negatives so over-redaction does not hide behind high recall.
Store fixture hashes, model identity, threshold, and aggregate counts only.
Keep DUA-gated corpora outside the repository and load them only from the user's approved location.

Repository example

Read the policy and release-evidence walkthrough for PHI-free leakage metrics and audit evidence.

Files

Just SKILL.md in skills/benchmark-pii-recall of maziyarpanahi/openmed.

Open the folder on GitHubat commit 34d7b8c

Compare with similar skills

Benchmark Pii Recall next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Benchmark Pii Recall compared with similar skills
Skill	Stars	Used in	Tokens	Auto-check	Licence	Repo updated
Benchmark Pii Recall this skillmaziyarpanahi/openmed	5.5k	—	~1k	Automated safety check: Pass	Apache-2.0	today
Agent BuildershareAI-lab/learn-claude-code	78k	4 repos	~1.2k	Automated safety check: Pass	MIT	12 days ago
Add Uint Supportpytorch/pytorch	104k	2 repos	~2.3k	Automated safety check: Pass	Custom licence	today
LLM Benchmarking with lm-evaluation-harnessOrchestra-Research/AI-Research-SKILLs	13k	8 repos	~3k	Automated safety check: Pass	MIT	3 mo ago
Segment Anything Model GuideOrchestra-Research/AI-Research-SKILLs	13k	8 repos	~3.3k	Automated safety check: Pass	MIT	3 mo ago
1passwordtrpc-group/trpc-agent-go	1.9k	14 repos	~656	Automated safety check: Pass	Apache-2.0	yesterday

Similar skills

Agent Builder
shareAI-lab/learn-claude-code
Design and build AI agents for any domain. An agent skill from shareAI-lab/learn-claude-code.
78k GitHub starsUsed in 4 repos~1.2k tokens
AI & LLM EngineeringAuto-check passed
Add Uint Support
pytorch/pytorch
Add unsigned integer (uint) type support to PyTorch operators by updating ATDISPATCH macros.
104k GitHub starsUsed in 2 repos~2.3k tokens
AI & LLM EngineeringAuto-check passed
LLM Benchmarking with lm-evaluation-harness
Orchestra-Research/AI-Research-SKILLs
Runs lm-evaluation-harness to benchmark language models on academic suites such as MMLU, GSM8K and HumanEval, compare models and track training checkpoints.
13k GitHub starsUsed in 8 repos~3k tokens
AI & LLM EngineeringAuto-check passed
Segment Anything Model Guide
Orchestra-Research/AI-Research-SKILLs
Guide to using Meta's Segment Anything Model for zero-shot image segmentation with point, box or mask prompts, or automatic mask generation.
13k GitHub starsUsed in 8 repos~3.3k tokens
AI & LLM EngineeringAuto-check passed
1password
trpc-group/trpc-agent-go
Set up and use 1Password CLI (op). An agent skill from trpc-group/trpc-agent-go.
1.9k GitHub starsUsed in 14 repos~656 tokens
AI & LLM EngineeringAuto-check passed
Planning With Files
jarrodwatts/claude-code-config
Transforms workflow to use Manus-style persistent markdown files for planning, progress tracking, and knowledge storage.
1.1k GitHub starsUsed in 5 repos~967 tokens
AI & LLM EngineeringAuto-check passed

Questions about Benchmark Pii Recall

What does Benchmark Pii Recall do?

Benchmark an OpenMed PII model with synthetic gold spans and report label-aware exact-span and grapheme recall without emitting identifier surfaces. Benchmark Pii Recall is an agent skill from maziyarpanahi/openmed. Benchmark an OpenMed PII model with synthetic gold spans and report label-aware exact-span and grapheme recall without emitting identifier surfaces.

When should I use Benchmark Pii Recall?

Benchmark Pii Recall fits situations like: an agent must compare a model; quantized artifact and enforce a recall floor before release.

How do I install Benchmark Pii Recall in Claude Code?

Run `npx skills add maziyarpanahi/openmed --skill benchmark-pii-recall -a claude-code`. Or copy the skill folder (skills/benchmark-pii-recall in maziyarpanahi/openmed) into .claude/skills/benchmark-pii-recall in your project. Claude Code loads it when a task matches its description.

How do I install Benchmark Pii Recall in Codex?

Run `npx skills add maziyarpanahi/openmed --skill benchmark-pii-recall -a codex`. Or copy the skill folder (skills/benchmark-pii-recall in maziyarpanahi/openmed) into .agents/skills/benchmark-pii-recall in your project. Codex loads it when a task matches its description.

Can I use Benchmark Pii Recall in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add maziyarpanahi/openmed --skill benchmark-pii-recall -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/benchmark-pii-recall, .gemini/skills/benchmark-pii-recall, .github/skills/benchmark-pii-recall and .opencode/skills/benchmark-pii-recall in your project.

What does Benchmark Pii Recall need to run?

Going by SKILL.md and its folder, Benchmark Pii Recall needs the command-line tools its instructions call (python). Our summary lists: Python 3.

Does Benchmark Pii Recall access the network?

SKILL.md contains no URLs. Any network use would come from the scripts or tools the agent runs. This is read from the text; nothing was executed.

Is Benchmark Pii Recall safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.

What licence does Benchmark Pii Recall use?

Benchmark Pii Recall is published under the Apache-2.0 licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does Benchmark Pii Recall use?

About 1k tokens (SKILL.md is roughly 4.1k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.

What are the alternatives to Benchmark Pii Recall?

Skills that share tags, products or a category with Benchmark Pii Recall: Agent Builder (shareAI-lab/learn-claude-code, 78k stars), Add Uint Support (pytorch/pytorch, 104k stars), LLM Benchmarking with lm-evaluation-harness (Orchestra-Research/AI-Research-SKILLs, 13k stars) and Segment Anything Model Guide (Orchestra-Research/AI-Research-SKILLs, 13k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Benchmark Pii Recall?

maziyarpanahi (a GitHub user) maintains it in maziyarpanahi/openmed, which has 5,506 GitHub stars. The repository holds 74 skills in this directory. The repository was last updated on October 11, 2026.

Source: maziyarpanahi/openmed on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.

Benchmark Pii Recall

Install Benchmark Pii Recall

Facts

At a glance

What it does

When your agent uses it

Example prompts

Requirements

Workflow steps

What it can do on your machine

Tool permissions

Runs code

Network

Credentials

Context cost

Safety

SKILL.md

Benchmark PII recall

Procedure

Runnable synthetic benchmark

Release gates

Repository example

Files

Compare with similar skills

Similar skills

Agent Builder

Add Uint Support

LLM Benchmarking with lm-evaluation-harness

Segment Anything Model Guide

1password

Planning With Files

More from maziyarpanahi/openmed

HIPAA Safe Harbor Coverage Audit

OpenMed Model Card Writer

HIPAA Pre-Deployment Compliance Check

ICD-10 Coding Assistant

OpenMed ETL to OMOP CDM

Extracting SDOH and Z-Codes

Questions about Benchmark Pii Recall

Install Benchmark Pii Recall

Facts

At a glance

What it does

When your agent uses it

Example prompts

Requirements

Workflow steps

What it can do on your machine

Tool permissions

Runs code

Network

Credentials

Context cost

Safety

SKILL.md

Benchmark PII recall

Procedure

Runnable synthetic benchmark

Release gates

Repository example

Files

Compare with similar skills

Similar skills

More from maziyarpanahi/openmed

Related

Questions about Benchmark Pii Recall