Benchmarking Clinical Ner

What it does

When your agent uses it

• The user wants a seqeval-style scorecard
• Strict vs partial (relaxed) span matching
• A per-label confusion matrix
• False-negative / false-positive examples

Example prompts

• “evaluate NER”
• “entity-level F1”
• “seqeval”
• “/benchmarking-clinical-ner”

Requirements

• Python 3

Workflow steps

6 steps, taken from the first numbered list in SKILL.md.

1. 1Align the corpus to OpenMed fixtures. Convert CoNLL/BIO or BRAT
2. 2Normalize labels to OpenMed's canonical set so DRUG/MEDICATION variants
3. 3Run run_suite / run_benchmark to get a BenchmarkReport.
4. 4Read both F1s. A large strict↓ / relaxed↑ gap means boundary errors, not
5. 5Run error_report for the per-label confusion matrix and capped
6. 6Triage per label. Fix the worst-recall label first; in clinical NER a few

What it can do on your machine

Read from SKILL.md and the folder at commit 34d7b8c. It shows what the files ask for, not the result of running them.

• Tool permissions

  Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

  From allowed-tools in the SKILL.md frontmatter.

• Runs code

  No scripts in the folder and no shell commands in SKILL.md (its code samples are python).

  From the folder's file list and the shell code blocks in SKILL.md.

• Network

  Links to these hosts (documentation or services it may open):

  • github.com
  • davidsbatista.net
  • aclanthology.org
  • brat.nlplab.org

  From URLs in SKILL.md, links to its own repository left out.

• Credentials

  Names no API keys, tokens, secrets or passwords.

  From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Benchmarking Clinical Ner loads about 1.7k tokens when it runs. Until then it costs about 166 tokens; SKILL.md has 577 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it

~166

When it runs · the whole SKILL.md, loaded when a task matches

~1.7k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from maziyarpanahi/openmed at commit 34d7b8c, republished under its Apache-2.0 licence (© maziyarpanahi). 577 words, ~1,666 tokens.

name

benchmarking-clinical-ner

description

Score an OpenMed clinical or biomedical NER model against a user-supplied gold corpus with entity-level precision, recall, and F1, then break errors down per label. Use when the user wants a seqeval-style scorecard, strict vs partial (relaxed) span matching, a per-label confusion matrix, false-negative / false-positive examples, or to debug why a model misses entities. Trigger on "evaluate NER", "entity-level F1", "seqeval", "precision recall F1", "confusion matrix", "error analysis", "strict vs partial match", or "score against gold" in an OpenMed context. The gold corpus is user-supplied; OpenMed bundles no i2b2/n2c2/MIMIC data.

license

Apache-2.0

metadata.project

OpenMed

metadata.category

evaluation-quality

metadata.pairs

adjacent

metadata.version

1.0

from openmed.eval import run_suite, error_report

# Fixtures: JSON list of {"id", "text", "gold_spans": [{start, end, label}, ...]}
report = run_suite(
    "eval/gold/clinical_ner.json",        # YOUR gold corpus, not bundled
    suite="golden",
    model_name="OpenMed/Disease-Detection",
    device="cpu",
)

m = report.metrics
print("exact F1 :", m["exact_span_f1"]["f1"])      # strict
print("relaxed F1:", m["relaxed_span_f1"]["f1"])    # partial
print("recall by label:", m["recall_slices"]["by_label"])

# Per-label confusion matrix + capped, no-PHI error examples.
errors = error_report(
    "OpenMed/Disease-Detection",
    "eval/gold/clinical_ner.json",
    suite_name="clinical_ner",
    example_cap=5,
)
print(errors.to_markdown())                 # confusion matrix + FN/FP tables
errors.write_json("eval/out/error_analysis.json")

from openmed.eval import compute_exact_span_f1, compute_relaxed_span_f1

strict = compute_exact_span_f1(gold_spans, predicted_spans)
partial = compute_relaxed_span_f1(gold_spans, predicted_spans)

© maziyarpanahi, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

Just SKILL.md in skills/benchmarking-clinical-ner of maziyarpanahi/openmed.

Open the folder on GitHubat commit 34d7b8c