Model Evaluation Metrics
jeremylongshore/tons-of-skills-marketplace
Build model evaluation metrics operations. An agent skill from jeremylongshore/tons-of-skills-marketplace.
A skill your agent uses for H2O LLM Studio model wrappers, losses, metrics, evaluation outputs, inference routing, plots, and AI-judge metric behavior.
Apache-2.0Auto-check passed
Install Modeling And Evaluation
$ npx skills add VectorSpaceLab/AREX-Skill --skill modeling-and-evaluation -a claude-codeProject install by default; add -g for ~/.claude/skills/.
$ gh skill install VectorSpaceLab/AREX-Skill modeling-and-evaluation --agent claude-codeProject scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).
$ git clone --depth 1 https://github.com/VectorSpaceLab/AREX-Skill.git skills-src && mkdir -p .claude/skills && cp -r skills-src/skills/repositories/repo-skills/h2o-llmstudio/sub-skills/modeling-and-evaluation .claude/skills/modeling-and-evaluation && rm -rf skills-srcUse ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.
Claude Code skills documentation · loads skills from .claude/skills/
Install the "modeling-and-evaluation" agent skill from https://github.com/VectorSpaceLab/AREX-Skill/tree/main/skills/repositories/repo-skills/h2o-llmstudio/sub-skills/modeling-and-evaluation into .claude/skills/modeling-and-evaluation/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "modeling-and-evaluation", then confirm the skill loads.Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$skill-installer install https://github.com/VectorSpaceLab/AREX-Skill/tree/main/skills/repositories/repo-skills/h2o-llmstudio/sub-skills/modeling-and-evaluationType this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.
$ npx skills add VectorSpaceLab/AREX-Skill --skill modeling-and-evaluation -a codexProject install goes to .agents/skills/; add -g for ~/.codex/skills/.
$ gh skill install VectorSpaceLab/AREX-Skill modeling-and-evaluation --agent codexProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/VectorSpaceLab/AREX-Skill.git skills-src && mkdir -p .agents/skills && cp -r skills-src/skills/repositories/repo-skills/h2o-llmstudio/sub-skills/modeling-and-evaluation .agents/skills/modeling-and-evaluation && rm -rf skills-srcUse ~/.agents/skills/ instead of .agents/skills for a personal install.
Codex skills documentation · loads skills from .agents/skills/
Install the "modeling-and-evaluation" agent skill from https://github.com/VectorSpaceLab/AREX-Skill/tree/main/skills/repositories/repo-skills/h2o-llmstudio/sub-skills/modeling-and-evaluation into .agents/skills/modeling-and-evaluation/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "modeling-and-evaluation", then confirm the skill loads.Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add VectorSpaceLab/AREX-Skill --skill modeling-and-evaluation -a cursorProject install goes to .agents/skills/; add -g for ~/.cursor/skills/.
$ gh skill install VectorSpaceLab/AREX-Skill modeling-and-evaluation --agent cursorProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/VectorSpaceLab/AREX-Skill.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/skills/repositories/repo-skills/h2o-llmstudio/sub-skills/modeling-and-evaluation .cursor/skills/modeling-and-evaluation && rm -rf skills-srcUse ~/.cursor/skills/ instead of .cursor/skills for a personal install.
Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/
Install the "modeling-and-evaluation" agent skill from https://github.com/VectorSpaceLab/AREX-Skill/tree/main/skills/repositories/repo-skills/h2o-llmstudio/sub-skills/modeling-and-evaluation into .cursor/skills/modeling-and-evaluation/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "modeling-and-evaluation", then confirm the skill loads.Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gemini skills install https://github.com/VectorSpaceLab/AREX-Skill.git --path skills/repositories/repo-skills/h2o-llmstudio/sub-skills/modeling-and-evaluation--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.
$ npx skills add VectorSpaceLab/AREX-Skill --skill modeling-and-evaluation -a gemini-cliProject install goes to .agents/skills/; add -g for ~/.gemini/skills/.
$ gh skill install VectorSpaceLab/AREX-Skill modeling-and-evaluation --agent gemini-cliProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/VectorSpaceLab/AREX-Skill.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/skills/repositories/repo-skills/h2o-llmstudio/sub-skills/modeling-and-evaluation .gemini/skills/modeling-and-evaluation && rm -rf skills-srcUse ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.
Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/
Install the "modeling-and-evaluation" agent skill from https://github.com/VectorSpaceLab/AREX-Skill/tree/main/skills/repositories/repo-skills/h2o-llmstudio/sub-skills/modeling-and-evaluation into .gemini/skills/modeling-and-evaluation/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "modeling-and-evaluation", then confirm the skill loads.Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gh skill install VectorSpaceLab/AREX-Skill modeling-and-evaluationInstalls for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).
$ npx skills add VectorSpaceLab/AREX-Skill --skill modeling-and-evaluation -a github-copilotProject install goes to .agents/skills/; add -g for ~/.copilot/skills/.
$ git clone --depth 1 https://github.com/VectorSpaceLab/AREX-Skill.git skills-src && mkdir -p .github/skills && cp -r skills-src/skills/repositories/repo-skills/h2o-llmstudio/sub-skills/modeling-and-evaluation .github/skills/modeling-and-evaluation && rm -rf skills-srcUse ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.
GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/
Install the "modeling-and-evaluation" agent skill from https://github.com/VectorSpaceLab/AREX-Skill/tree/main/skills/repositories/repo-skills/h2o-llmstudio/sub-skills/modeling-and-evaluation into .github/skills/modeling-and-evaluation/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "modeling-and-evaluation", then confirm the skill loads.GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add VectorSpaceLab/AREX-Skill --skill modeling-and-evaluation -a opencodeOpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.
$ gh skill install VectorSpaceLab/AREX-Skill modeling-and-evaluation --agent opencodeProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/VectorSpaceLab/AREX-Skill.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/skills/repositories/repo-skills/h2o-llmstudio/sub-skills/modeling-and-evaluation .opencode/skills/modeling-and-evaluation && rm -rf skills-srcUse ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.
OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/
Install the "modeling-and-evaluation" agent skill from https://github.com/VectorSpaceLab/AREX-Skill/tree/main/skills/repositories/repo-skills/h2o-llmstudio/sub-skills/modeling-and-evaluation into .opencode/skills/modeling-and-evaluation/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "modeling-and-evaluation", then confirm the skill loads.OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
Facts
- Skill name
modeling-and-evaluation- GitHub stars
- 330
- Token cost
- ~801 tokens
- SKILL.md length
- 294 words
- Files
- 5 (incl. scripts, references)
- Skills in repo
- 159
- Repo updated
- First seen
- Licence
- Apache-2.0
At a glance
A skill your agent uses for H2O LLM Studio model wrappers, losses, metrics, evaluation outputs, inference routing, plots, and AI-judge metric behavior.
- Works in 5 steps: Identify the problem_type first. Valid… → Load… → Load references/evaluation-workflows.md… → …
- H2O LLM Studio model wrappers
- SKILL.md covers Operating workflow, Quick commands and Decision rules
- Runs Python scripts from its folder; calls python
What it does
Modeling And Evaluation is an agent skill from VectorSpaceLab/AREX-Skill. Use for H2O LLM Studio model wrappers, losses, metrics, evaluation outputs, inference routing, plots, and AI-judge metric behavior.
Its SKILL.md is about 800 tokens, which your agent loads only when the skill is triggered. The skill folder holds 6 other files, including scripts and reference files (for example `references/evaluation-workflows.md`, `references/model-and-metric-reference.md` and `references/troubleshooting.md`).
The repository describes itself as: A Skill Library for Automated Machine Learning. The licence is Apache-2.0.
When your agent uses it
- H2O LLM Studio model wrappers
- Evaluation outputs
- Inference routing
- AI-judge metric behavior
Example prompts
- “/modeling-and-evaluation”
Requirements
- Python 3
Workflow steps
5 steps, taken from the first numbered list in SKILL.md.
- Identify the problem_type first. Valid modeling/evaluation problem types are
- Load references/model-and-metric-reference.md for class mappings, forward/generate contracts, losses, metric direction, and expected…
- Load references/evaluation-workflows.md for run_inference, run_eval, prediction files, plot files, generation-vs-forward routing, and…
- For unsupported metrics, NaNs, empty outputs, OpenAI/Azure endpoint issues, missing prompts/, or classification/regression shape errors…
- Use scripts/inspect_problem_type.py for a safe local inspection that lists the configured model/loss/metric/plot classes without…
What it can do on your machine
Read from SKILL.md and the folder at commit ac3fe1a. It shows what the files ask for, not the result of running them.
Tool permissions
Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.
From allowed-tools in the SKILL.md frontmatter.
Runs code
Ships 1 file in scripts/ (Python), which the agent can run.
Shell commands in SKILL.md call:
python
From the folder's file list and the shell code blocks in SKILL.md.
Network
No URLs in SKILL.md.
From URLs in SKILL.md, links to its own repository left out.
Credentials
Names no API keys, tokens, secrets or passwords.
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.
Context cost
Modeling And Evaluation loads about 801 tokens when it runs, and up to ~7.4k if it reads all its reference files. Until then it costs about 39 tokens; SKILL.md has 294 words of instructions outside code blocks.
Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.
Safety
Auto-check passed
The automated check found no risky patterns in SKILL.md.
Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); the scripts in this folder are not scanned.
SKILL.md
The full file from VectorSpaceLab/AREX-Skill at commit ac3fe1a, republished under its Apache-2.0 licence (© VectorSpaceLab). 294 words, ~801 tokens.
Download SKILL.mdSave it as
.claude/skills/modeling-and-evaluation/SKILL.md (or your agent's skills folder). This skill also uses 4 other files; get the full folder from GitHub.- name
- modeling-and-evaluation
- description
- Use for H2O LLM Studio model wrappers, losses, metrics, evaluation outputs, inference routing, plots, and AI-judge metric behavior.
- disable-model-invocation
- true
- metadata.disco-role
- operating
- license
- Apache 2.0
H2O LLM Studio modeling and evaluation
Use this sub-skill when the task is about how H2O LLM Studio maps a problem type to model wrappers, losses, metrics, inference behavior, generated prediction outputs, validation plots, or AI-judge metrics.
Do not use this sub-skill for:
- creating or repairing experiment YAML/data schemas: use
configuration-and-data; - launching, scheduling, or debugging training runs: use
training-and-experiments; - interactive chat, model-card export, or Hugging Face publishing: use
export-and-prompt.
Operating workflow
- Identify the
problem_typefirst. Valid modeling/evaluation problem types are:text_causal_language_modeling,text_sequence_to_sequence_modeling,text_dpo_modeling,text_causal_classification_modeling, andtext_causal_regression_modeling. - Load references/model-and-metric-reference.md for class mappings, forward/generate contracts, losses, metric direction, and expected result keys.
- Load references/evaluation-workflows.md for
run_inference,run_eval, prediction files, plot files, generation-vs-forward routing, and GPT/MT-Bench judge behavior. - For unsupported metrics, NaNs, empty outputs, OpenAI/Azure endpoint issues, missing
prompts/, or classification/regression shape errors, load references/troubleshooting.md. - Use scripts/inspect_problem_type.py for a safe local inspection that lists the configured model/loss/metric/plot classes without instantiating or downloading a model.
Quick commands
python scripts/inspect_problem_type.py --problem-type text_causal_classification_modeling --list-metrics
python scripts/inspect_problem_type.py --problem-type all --json
python scripts/inspect_problem_type.py --problem-type text_dpo_modeling --verify-importsThe inspection script is safe by default: it prints static and import-level metadata only. It does not start training, call an AI judge, load Hugging Face weights, or create model instances.
Decision rules
- For generative validation metrics other than
Perplexity, causal LM, sequence-to-sequence, and DPO use generation;Perplexityuses a forward pass. - Classification and regression are non-generation tasks. They always use a forward pass and postprocess logits or regression head outputs into predictions before metrics are computed.
GPTmetrics call an OpenAI-compatible Chat Completions endpoint and can incur network cost. Require endpoint, credential, and budget confirmation before running; use static reasoning or mocked tests when possible.- Validation artifacts are written by the training/evaluation workflow, not by the model classes themselves. Expect raw prediction pickle, prediction CSV, and parquet plot data when evaluation reaches rank 0 successfully.
© VectorSpaceLab, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file
Files
SKILL.md and 4 other files (scripts, references) in skills/repositories/repo-skills/h2o-llmstudio/sub-skills/modeling-and-evaluation of VectorSpaceLab/AREX-Skill.
- SKILL.md
- references/evaluation-workflows.md
- references/model-and-metric-reference.md
- references/troubleshooting.md
- scripts/inspect_problem_type.py
Open the folder on GitHubat commit ac3fe1a
Compare with similar skills
Modeling And Evaluation next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.
| Skill | Stars | Used in | Tokens | Auto-check | Licence | Repo updated |
|---|---|---|---|---|---|---|
| Modeling And Evaluation this skillVectorSpaceLab/AREX-Skill | 330 | — | ~801 | Automated safety check: Pass | Apache-2.0 | |
| Model Evaluation Metricsjeremylongshore/tons-of-skills-marketplace | 2.8k | — | ~578 | Automated safety check: Pass | MIT | |
| Code Model Evaluation HarnessOrchestra-Research/AI-Research-SKILLs | 13k | 4 repos | ~2.9k | Automated safety check: Pass | MIT | |
| Modeling Activation MetricsPostHog/posthog | 40k | — | ~1.4k | Automated safety check: Pass | Custom licence | |
| Modeling Revenue MetricsPostHog/posthog | 40k | — | ~2.1k | Automated safety check: Pass | Custom licence | |
| Modeling Product Usage MetricsPostHog/posthog | 40k | — | ~1.3k | Automated safety check: Pass | Custom licence |
Similar skills
Code Model Evaluation Harness
Orchestra-Research/AI-Research-SKILLs
Benchmarks code generation models with the BigCode Evaluation Harness across HumanEval, MBPP, MultiPL-E and other suites using pass@k metrics.
- Official
Modeling Activation Metrics
PostHog/posthog
Build reusable activation models — an activation-rate metric and a per-user/per-account activated flag — on either PostHog data-warehouse views (HogQL) or an external dbt project.
- Official
Modeling Revenue Metrics
PostHog/posthog
Build reusable revenue models — MRR, ARR, gross revenue, new/expansion/contraction/churn, ARPU, LTV, and per-customer/per-account revenue — on either PostHog data-warehouse views (HogQL) or an…
- Official
Modeling Product Usage Metrics
PostHog/posthog
Build reusable product-usage and engagement models — retention, stickiness, and lifecycle — on either PostHog data-warehouse views (HogQL) or an external dbt project.
- Official
Modeling Conversion Metrics
PostHog/posthog
Build reusable conversion models — funnel/step conversion rates, drop-off, and time-to-convert — on either PostHog data-warehouse views (HogQL) or an external dbt project.
More from VectorSpaceLab/AREX-Skill
All 159 skills in this repoAgent Lightning
VectorSpaceLab/AREX-Skill
Use this repo skill for Agent Lightning package tasks: authoring trainable agents, tracing rewards and spans, running LightningStore/Trainer loops, using agl CLI services, choosing examples, and…
Agent Tools
VectorSpaceLab/AREX-Skill
A skill your agent uses when configuring LiteLLM for MCP tools, A2A agents, Claude Code/Cursor agent gateway traffic, MCP auth/OAuth, tool permissions, semantic filtering, or agent-specific proxy…
Agents And Awel
VectorSpaceLab/AREX-Skill
Build and debug DB-GPT agents, tools, skills, teams, and AWEL workflows, including deterministic local DAG runs and HTTP-trigger topology without assuming an LLM, credential, or external service.
Agents And Middleware
VectorSpaceLab/AREX-Skill
Work on the actively maintained LangChain v1 agent package: initchatmodel, createagent, structured output, tools, middleware, embeddings initialization, provider routing, and agent runtime…
Agents Workflows
VectorSpaceLab/AREX-Skill
A skill your agent uses for giskard.agents async chat workflows, tools, prompt templates, structured outputs, retries, rate limiting, embeddings, and optional LiteLLM backend.
Alphafold3
VectorSpaceLab/AREX-Skill
A skill your agent uses for AlphaFold 3 input preparation, prediction command planning, output interpretation, and Python API inspection.
Questions about Modeling And Evaluation
What does Modeling And Evaluation do?
A skill your agent uses for H2O LLM Studio model wrappers, losses, metrics, evaluation outputs, inference routing, plots, and AI-judge metric behavior. Modeling And Evaluation is an agent skill from VectorSpaceLab/AREX-Skill. Use for H2O LLM Studio model wrappers, losses, metrics, evaluation outputs, inference routing, plots, and AI-judge metric behavior.
When should I use Modeling And Evaluation?
Modeling And Evaluation fits situations like: H2O LLM Studio model wrappers; evaluation outputs; inference routing; AI-judge metric behavior.
How do I install Modeling And Evaluation in Claude Code?
Run `npx skills add VectorSpaceLab/AREX-Skill --skill modeling-and-evaluation -a claude-code`. Or copy the skill folder (skills/repositories/repo-skills/h2o-llmstudio/sub-skills/modeling-and-evaluation in VectorSpaceLab/AREX-Skill) into .claude/skills/modeling-and-evaluation in your project. Claude Code loads it when a task matches its description.
How do I install Modeling And Evaluation in Codex?
Run `npx skills add VectorSpaceLab/AREX-Skill --skill modeling-and-evaluation -a codex`. Or copy the skill folder (skills/repositories/repo-skills/h2o-llmstudio/sub-skills/modeling-and-evaluation in VectorSpaceLab/AREX-Skill) into .agents/skills/modeling-and-evaluation in your project. Codex loads it when a task matches its description.
Can I use Modeling And Evaluation in Cursor, Gemini CLI or GitHub Copilot?
Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add VectorSpaceLab/AREX-Skill --skill modeling-and-evaluation -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/modeling-and-evaluation, .gemini/skills/modeling-and-evaluation, .github/skills/modeling-and-evaluation and .opencode/skills/modeling-and-evaluation in your project.
What does Modeling And Evaluation need to run?
Going by SKILL.md and its folder, Modeling And Evaluation needs Python for the scripts in its folder and the command-line tools its instructions call (python). Our summary lists: Python 3.
Does Modeling And Evaluation access the network?
SKILL.md contains no URLs. Any network use would come from the scripts or tools the agent runs. This is read from the text; nothing was executed.
Is Modeling And Evaluation safe to install?
Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. The check reads SKILL.md only: the scripts in the folder are not scanned, so read them before running anything.
What licence does Modeling And Evaluation use?
Modeling And Evaluation is published under the Apache-2.0 licence (declared in SKILL.md). It allows redistribution, so the full SKILL.md is shown on this page.
How many tokens does Modeling And Evaluation use?
About 801 tokens (SKILL.md is roughly 3.2k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full. Its references folder adds about 6.6k tokens, read only when the agent opens those files.
What are the alternatives to Modeling And Evaluation?
Skills that share tags, products or a category with Modeling And Evaluation: Model Evaluation Metrics (jeremylongshore/tons-of-skills-marketplace, 2.8k stars), Code Model Evaluation Harness (Orchestra-Research/AI-Research-SKILLs, 13k stars), Modeling Activation Metrics (PostHog/posthog, 40k stars) and Modeling Revenue Metrics (PostHog/posthog, 40k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.
Who maintains Modeling And Evaluation?
VectorSpaceLab (a GitHub organization) maintains it in VectorSpaceLab/AREX-Skill, which has 330 GitHub stars. The repository holds 159 skills in this directory. The repository was last updated on September 3, 2026.
Source: VectorSpaceLab/AREX-Skill on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.