LLM Benchmarking with lm-evaluation-harness
Orchestra-Research/AI-Research-SKILLs
Runs lm-evaluation-harness to benchmark language models on academic suites such as MMLU, GSM8K and HumanEval, compare models and track training checkpoints.
Load OpenMed clinical/biomedical NER models from the Hugging Face Hub or a local path and reuse them efficiently across calls.
$ npx skills add maziyarpanahi/openmed --skill loading-openmed-models -a claude-codeProject install by default; add -g for ~/.claude/skills/.
$ gh skill install maziyarpanahi/openmed loading-openmed-models --agent claude-codeProject scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).
$ git clone --depth 1 https://github.com/maziyarpanahi/openmed.git skills-src && mkdir -p .claude/skills && cp -r skills-src/skills/loading-openmed-models .claude/skills/loading-openmed-models && rm -rf skills-srcUse ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.
Claude Code skills documentation · loads skills from .claude/skills/
Install the "loading-openmed-models" agent skill from https://github.com/maziyarpanahi/openmed/tree/master/skills/loading-openmed-models into .claude/skills/loading-openmed-models/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "loading-openmed-models", then confirm the skill loads.Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$skill-installer install https://github.com/maziyarpanahi/openmed/tree/master/skills/loading-openmed-modelsType this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.
$ npx skills add maziyarpanahi/openmed --skill loading-openmed-models -a codexProject install goes to .agents/skills/; add -g for ~/.codex/skills/.
$ gh skill install maziyarpanahi/openmed loading-openmed-models --agent codexProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/maziyarpanahi/openmed.git skills-src && mkdir -p .agents/skills && cp -r skills-src/skills/loading-openmed-models .agents/skills/loading-openmed-models && rm -rf skills-srcUse ~/.agents/skills/ instead of .agents/skills for a personal install.
Codex skills documentation · loads skills from .agents/skills/
Install the "loading-openmed-models" agent skill from https://github.com/maziyarpanahi/openmed/tree/master/skills/loading-openmed-models into .agents/skills/loading-openmed-models/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "loading-openmed-models", then confirm the skill loads.Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add maziyarpanahi/openmed --skill loading-openmed-models -a cursorProject install goes to .agents/skills/; add -g for ~/.cursor/skills/.
$ gh skill install maziyarpanahi/openmed loading-openmed-models --agent cursorProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/maziyarpanahi/openmed.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/skills/loading-openmed-models .cursor/skills/loading-openmed-models && rm -rf skills-srcUse ~/.cursor/skills/ instead of .cursor/skills for a personal install.
Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/
Install the "loading-openmed-models" agent skill from https://github.com/maziyarpanahi/openmed/tree/master/skills/loading-openmed-models into .cursor/skills/loading-openmed-models/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "loading-openmed-models", then confirm the skill loads.Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gemini skills install https://github.com/maziyarpanahi/openmed.git --path skills/loading-openmed-models--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.
$ npx skills add maziyarpanahi/openmed --skill loading-openmed-models -a gemini-cliProject install goes to .agents/skills/; add -g for ~/.gemini/skills/.
$ gh skill install maziyarpanahi/openmed loading-openmed-models --agent gemini-cliProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/maziyarpanahi/openmed.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/skills/loading-openmed-models .gemini/skills/loading-openmed-models && rm -rf skills-srcUse ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.
Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/
Install the "loading-openmed-models" agent skill from https://github.com/maziyarpanahi/openmed/tree/master/skills/loading-openmed-models into .gemini/skills/loading-openmed-models/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "loading-openmed-models", then confirm the skill loads.Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gh skill install maziyarpanahi/openmed loading-openmed-modelsInstalls for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).
$ npx skills add maziyarpanahi/openmed --skill loading-openmed-models -a github-copilotProject install goes to .agents/skills/; add -g for ~/.copilot/skills/.
$ git clone --depth 1 https://github.com/maziyarpanahi/openmed.git skills-src && mkdir -p .github/skills && cp -r skills-src/skills/loading-openmed-models .github/skills/loading-openmed-models && rm -rf skills-srcUse ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.
GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/
Install the "loading-openmed-models" agent skill from https://github.com/maziyarpanahi/openmed/tree/master/skills/loading-openmed-models into .github/skills/loading-openmed-models/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "loading-openmed-models", then confirm the skill loads.GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add maziyarpanahi/openmed --skill loading-openmed-models -a opencodeOpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.
$ gh skill install maziyarpanahi/openmed loading-openmed-models --agent opencodeProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/maziyarpanahi/openmed.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/skills/loading-openmed-models .opencode/skills/loading-openmed-models && rm -rf skills-srcUse ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.
OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/
Install the "loading-openmed-models" agent skill from https://github.com/maziyarpanahi/openmed/tree/master/skills/loading-openmed-models into .opencode/skills/loading-openmed-models/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "loading-openmed-models", then confirm the skill loads.OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
loading-openmed-modelsLoad OpenMed clinical/biomedical NER models from the Hugging Face Hub or a local path and reuse them efficiently across calls.
Loading Openmed Models is an agent skill from maziyarpanahi/openmed. Load OpenMed clinical/biomedical NER models from the Hugging Face Hub or a local path and reuse them efficiently across calls. Use when the user wants to load an OpenMed model, control the model cache, run fully offline after a one-time download, reuse a ModelLoader to avoid reloading, set a cachedir or device, or pick between a registry key, a full Hugging Face id, and a local directory. Pairs with choosing-openmed-models (pick the model) and extracting-clinical-entities (run it).
Its SKILL.md is about 2.1k tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts.
It sits in AI & LLM Engineering, covering Model hubs and datasets. It works with Hugging Face. The repository describes itself as: Local-first healthcare AI: clinical NER & HIPAA PII de-identification that runs 100% on-device. 2,200+ medical models, 21 languages, Apple MLX + Python, no cloud, no patient data…. The licence is Apache-2.0.
2 steps, taken from the first numbered list in SKILL.md.
Read from SKILL.md and the folder at commit 6b1bb2c. It shows what the files ask for, not the result of running them.
Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.
From allowed-tools in the SKILL.md frontmatter.
Shell commands in SKILL.md call:
pipFrom the folder's file list and the shell code blocks in SKILL.md.
Links to these hosts (documentation or services it may open):
huggingface.coFrom URLs in SKILL.md, links to its own repository left out.
Names these keys or tokens, usually read from environment variables:
HF_TOKENFrom names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.
Loading Openmed Models loads about 2.1k tokens when it runs. Until then it costs about 128 tokens; SKILL.md has 678 words of instructions outside code blocks.
Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.
The automated check found no risky patterns in SKILL.md.
Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.
The full file from maziyarpanahi/openmed at commit 6b1bb2c, republished under its Apache-2.0 licence (© maziyarpanahi). 678 words, ~2,087 tokens.
.claude/skills/loading-openmed-models/SKILL.md (or your agent's skills folder).OpenMed models download once from the Hugging Face Hub into a local cache, then run fully on-device — no network, no telemetry. This skill covers how to load a model, reuse it across many calls without reloading weights, point at a local copy, and run offline.
cache_dir) or force CPU/GPU.For which model to load, see choosing-openmed-models. To actually run it, see
extracting-clinical-entities.
pip install "openmed[hf]" # adds Hugging Face transformers + hub downloadanalyze_text, extract_pii, load_model, and ModelLoader.load_model all
accept the same model_name in three forms:
| Form | Example | Notes |
|---|---|---|
| Registry key | "disease_detection_superclinical" | Short, resolved via the bundled registry. |
| Full HF id | "OpenMed/OpenMed-NER-DiseaseDetect-BigMed-278M" | Anything org/name; downloaded from the Hub. |
| Local path | "/models/my-openmed-ner" | An existing directory; loaded with local_files_only=True. |
A bare name without / is prefixed with the default org (OpenMed). An existing
local path is detected automatically and never hits the network.
The single most important pattern — build one ModelLoader, pass it everywhere.
The loader caches models, tokenizers, and pipelines in memory, so the second call
is instant.
import openmed
from openmed import ModelLoader, OpenMedConfig
# One loader, reused across calls. Weights load on the first call only.
loader = ModelLoader()
notes = [
"Patient prescribed 500 mg metformin for type 2 diabetes.",
"History of myocardial infarction; started on atorvastatin.",
]
for note in notes:
result = openmed.analyze_text(
note,
model_name="disease_detection_superclinical",
loader=loader, # <-- reuse; no reload on subsequent calls
output_format="dict",
)
print(result.entities)Without loader=, each analyze_text call constructs a fresh ModelLoader. The
underlying Hugging Face cache still prevents re-downloads, but you pay to
re-instantiate the pipeline — avoid that in loops and services.
When you want the raw model/tokenizer (e.g. to inspect config or build a custom pipeline):
from openmed import load_model
bundle = load_model("disease_detection_superclinical")
model = bundle["model"]
tokenizer = bundle["tokenizer"]
config = bundle["config"]load_model(model_name, config=None, **kwargs) is a thin convenience wrapper that
builds a ModelLoader and calls loader.load_model(...). For reuse, prefer
constructing the loader yourself:
loader = ModelLoader()
bundle = loader.load_model("disease_detection_superclinical")
# Second call returns the cached bundle (no reload):
bundle2 = loader.load_model("disease_detection_superclinical")
# Force a fresh load if you replaced files on disk:
fresh = loader.load_model("disease_detection_superclinical", force_reload=True)OpenMedConfig is a dataclass. Pass it to ModelLoader(config=...).
from openmed import ModelLoader, OpenMedConfig
config = OpenMedConfig(
cache_dir="/data/openmed-cache", # default: ~/.cache/openmed
device="cpu", # None = auto-detect
default_org="OpenMed", # prepended to bare model names
hf_token=None, # or set env HF_TOKEN for private repos
)
loader = ModelLoader(config)Relevant OpenMedConfig fields: cache_dir, device, default_org, hf_token,
timeout (default 300s), backend (None auto / "hf" / "mlx"), log_level.
hf_token falls back to the HF_TOKEN environment variable.
cache_dir.To guarantee no network access (air-gapped, CI, PHI environments), set the standard Hugging Face offline switch before importing:
export HF_HUB_OFFLINE=1
export TRANSFORMERS_OFFLINE=1Or vendor the model and pass a local path — that path is loaded with
local_files_only=True and never contacts the Hub:
result = openmed.analyze_text(note, model_name="/models/openmed-disease-ner")To pre-warm a cache for offline use, run one inference (or load_model) once with
network access, then disable it.
Useful before chunking long documents:
from openmed import get_model_max_length, ModelLoader
loader = ModelLoader()
max_len = get_model_max_length("disease_detection_superclinical", loader=loader)
print(max_len) # e.g. 512 — None if it can't be inferredget_model_max_length(model_name, *, config=None, loader=None) delegates to
loader.get_max_sequence_length(model_name). Pass the same loader you use for
inference so the tokenizer is loaded only once.
The loader holds models in RAM until released:
loader.unload_model("disease_detection_superclinical") # drop one model
loader.unload_all_models() # drop everything
loader.loaded_models() # inspect what's cachedchoosing-openmed-models: that skill yields a model key or HF id; feed
it straight into ModelLoader.load_model(...) or as model_name=.extracting-clinical-entities: pass your reused loader= into
openmed.analyze_text(...) so a long batch loads weights exactly once.openmed.extract_pii(..., loader=loader) and
openmed.deidentify(..., loader=loader) accept the same loader — share one
loader across NER and PHI steps in a pipeline.loader = ModelLoader(OpenMedConfig(cache_dir="/data/openmed-cache"))
phi = openmed.deidentify(note, method="mask", loader=loader)
ner = openmed.analyze_text(phi.deidentified_text, loader=loader)pip install openmed alone is not enough to download models — add the
[hf] extra (or have transformers + huggingface_hub installed). ModelLoader
raises ImportError with an install hint if transformers is missing.force_reload=True is required after you overwrite files in a local model
directory; otherwise the in-memory cache is served.hf_token (or HF_TOKEN) and HF_HUB_OFFLINE unset for
the first download.cache_dir should not live inside a PHI data directory.© maziyarpanahi, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file
Just SKILL.md in skills/loading-openmed-models of maziyarpanahi/openmed.
Open the folder on GitHubat commit 6b1bb2c
Loading Openmed Models next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.
| Skill | Stars | Used in | Tokens | Auto-check | Licence | Repo updated |
|---|---|---|---|---|---|---|
| Loading Openmed Models this skillmaziyarpanahi/openmed | 5.5k | — | ~2.1k | Automated safety check: Pass | Apache-2.0 | |
| LLM Benchmarking with lm-evaluation-harnessOrchestra-Research/AI-Research-SKILLs | 13k | 8 repos | ~3k | Automated safety check: Pass | MIT | |
| SageMaker Serving Image Selectionhuggingface/skills | 11k | 1 repos | ~4.6k | Automated safety check: Pass | Apache-2.0 | |
| Hugging Face Local Model Evalshuggingface/skills | 11k | 2 repos | ~1.6k | Automated safety check: Pass | Apache-2.0 | |
| Upload Post Imagehuggingface/blog | 3.5k | — | ~1.1k | Automated safety check: Pass | None | |
| Add Archon Modelareal-project/AReaL | 5.8k | — | ~4.9k | Automated safety check: Pass | Apache-2.0 |
Orchestra-Research/AI-Research-SKILLs
Runs lm-evaluation-harness to benchmark language models on academic suites such as MMLU, GSM8K and HumanEval, compare models and track training checkpoints.
huggingface/skills
Chooses the right serving container and current image URI for deploying a Hugging Face model to a SageMaker endpoint, preferring Hugging Face images over generic ones.
huggingface/skills
Runs evaluations of Hugging Face Hub models on local hardware with inspect-ai or lighteval, and helps choose between vLLM, Transformers and accelerate backends.
huggingface/blog
A skill your agent uses when adding or migrating non-thumbnail images for a Hugging Face Blog post.
areal-project/AReaL
Guide for adding a new model to the Archon engine. An agent skill from areal-project/AReaL.
JimLiu/science-skills
Biohub ESMFold2 / ESMFold2-Fast all-atom co-folding (Candido et al.
maziyarpanahi/openmed
Checks OpenMed de-identified clinical text against the 18 HIPAA Safe Harbor identifier categories and reports gaps and residual re-identification risk.
maziyarpanahi/openmed
Fills in a model card for an OpenMed clinical NER or de-identification model from its evaluation reports: intended use, metrics, subgroups and limitations.
maziyarpanahi/openmed
Walks a data pipeline against the HIPAA Privacy and Security Rule checklist and produces a gap report before it processes patient data.
maziyarpanahi/openmed
Suggests candidate ICD-10-CM diagnosis and ICD-10-PCS procedure codes for clinical text extracted by OpenMed, with rationale for a certified coder to review.
maziyarpanahi/openmed
Maps OpenMed-extracted, terminology-coded conditions, drugs and measurements into OMOP CDM v5.4 tables for OHDSI and ATLAS analytics.
maziyarpanahi/openmed
Finds social risks such as housing instability or food insecurity in clinical notes and proposes matching ICD-10-CM Z-codes for a coder to confirm.
Works with
Categories
Load OpenMed clinical/biomedical NER models from the Hugging Face Hub or a local path and reuse them efficiently across calls. Loading Openmed Models is an agent skill from maziyarpanahi/openmed. Load OpenMed clinical/biomedical NER models from the Hugging Face Hub or a local path and reuse them efficiently across calls.
Loading Openmed Models fits situations like: the user wants to load an OpenMed model; control the model cache; run fully offline after a one-time download; reuse a ModelLoader to avoid reloading.
Run `npx skills add maziyarpanahi/openmed --skill loading-openmed-models -a claude-code`. Or copy the skill folder (skills/loading-openmed-models in maziyarpanahi/openmed) into .claude/skills/loading-openmed-models in your project. Claude Code loads it when a task matches its description.
Run `npx skills add maziyarpanahi/openmed --skill loading-openmed-models -a codex`. Or copy the skill folder (skills/loading-openmed-models in maziyarpanahi/openmed) into .agents/skills/loading-openmed-models in your project. Codex loads it when a task matches its description.
Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add maziyarpanahi/openmed --skill loading-openmed-models -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/loading-openmed-models, .gemini/skills/loading-openmed-models, .github/skills/loading-openmed-models and .opencode/skills/loading-openmed-models in your project.
Going by SKILL.md and its folder, Loading Openmed Models needs the command-line tools its instructions call (pip) and credentials named HF_TOKEN. Our summary lists: Python 3.
SKILL.md names 1 domain. As links in the text: huggingface.co. This is read from the text; nothing was executed.
Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.
Loading Openmed Models is published under the Apache-2.0 licence (declared in SKILL.md). It allows redistribution, so the full SKILL.md is shown on this page.
About 2.1k tokens (SKILL.md is roughly 8.3k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.
Skills that share tags, products or a category with Loading Openmed Models: LLM Benchmarking with lm-evaluation-harness (Orchestra-Research/AI-Research-SKILLs, 13k stars), SageMaker Serving Image Selection (huggingface/skills, 11k stars), Hugging Face Local Model Evals (huggingface/skills, 11k stars) and Upload Post Image (huggingface/blog, 3.5k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.
maziyarpanahi (a GitHub user) maintains it in maziyarpanahi/openmed, which has 5,493 GitHub stars. The repository holds 74 skills in this directory. The repository was last updated on October 9, 2026.
Source: maziyarpanahi/openmed on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.