Arize Evaluator
github/awesome-copilot
Handles LLM-as-judge evaluation workflows on Arize including creating/updating evaluators, running evaluations on spans or experiments, managing tasks, trigger-run operations, column mapping, and…
A skill your agent uses when the user wants a quality review, interaction audit, or to test the workflow against realistic scenarios.
MITAuto-check passed
Install Evaluate
$ npx skills add sharpdeveye/maestro --skill evaluate -a claude-codeProject install by default; add -g for ~/.claude/skills/.
$ gh skill install sharpdeveye/maestro evaluate --agent claude-codeProject scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).
$ git clone --depth 1 https://github.com/sharpdeveye/maestro.git skills-src && mkdir -p .claude/skills && cp -r skills-src/source/skills/evaluate .claude/skills/evaluate && rm -rf skills-srcUse ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.
Claude Code skills documentation · loads skills from .claude/skills/
Install the "evaluate" agent skill from https://github.com/sharpdeveye/maestro/tree/main/source/skills/evaluate into .claude/skills/evaluate/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "evaluate", then confirm the skill loads.Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$skill-installer install https://github.com/sharpdeveye/maestro/tree/main/source/skills/evaluateType this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.
$ npx skills add sharpdeveye/maestro --skill evaluate -a codexProject install goes to .agents/skills/; add -g for ~/.codex/skills/.
$ gh skill install sharpdeveye/maestro evaluate --agent codexProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/sharpdeveye/maestro.git skills-src && mkdir -p .agents/skills && cp -r skills-src/source/skills/evaluate .agents/skills/evaluate && rm -rf skills-srcUse ~/.agents/skills/ instead of .agents/skills for a personal install.
Codex skills documentation · loads skills from .agents/skills/
Install the "evaluate" agent skill from https://github.com/sharpdeveye/maestro/tree/main/source/skills/evaluate into .agents/skills/evaluate/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "evaluate", then confirm the skill loads.Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add sharpdeveye/maestro --skill evaluate -a cursorProject install goes to .agents/skills/; add -g for ~/.cursor/skills/.
$ gh skill install sharpdeveye/maestro evaluate --agent cursorProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/sharpdeveye/maestro.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/source/skills/evaluate .cursor/skills/evaluate && rm -rf skills-srcUse ~/.cursor/skills/ instead of .cursor/skills for a personal install.
Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/
Install the "evaluate" agent skill from https://github.com/sharpdeveye/maestro/tree/main/source/skills/evaluate into .cursor/skills/evaluate/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "evaluate", then confirm the skill loads.Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gemini skills install https://github.com/sharpdeveye/maestro.git --path source/skills/evaluate--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.
$ npx skills add sharpdeveye/maestro --skill evaluate -a gemini-cliProject install goes to .agents/skills/; add -g for ~/.gemini/skills/.
$ gh skill install sharpdeveye/maestro evaluate --agent gemini-cliProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/sharpdeveye/maestro.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/source/skills/evaluate .gemini/skills/evaluate && rm -rf skills-srcUse ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.
Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/
Install the "evaluate" agent skill from https://github.com/sharpdeveye/maestro/tree/main/source/skills/evaluate into .gemini/skills/evaluate/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "evaluate", then confirm the skill loads.Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gh skill install sharpdeveye/maestro evaluateInstalls for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).
$ npx skills add sharpdeveye/maestro --skill evaluate -a github-copilotProject install goes to .agents/skills/; add -g for ~/.copilot/skills/.
$ git clone --depth 1 https://github.com/sharpdeveye/maestro.git skills-src && mkdir -p .github/skills && cp -r skills-src/source/skills/evaluate .github/skills/evaluate && rm -rf skills-srcUse ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.
GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/
Install the "evaluate" agent skill from https://github.com/sharpdeveye/maestro/tree/main/source/skills/evaluate into .github/skills/evaluate/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "evaluate", then confirm the skill loads.GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add sharpdeveye/maestro --skill evaluate -a opencodeOpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.
$ gh skill install sharpdeveye/maestro evaluate --agent opencodeProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/sharpdeveye/maestro.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/source/skills/evaluate .opencode/skills/evaluate && rm -rf skills-srcUse ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.
OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/
Install the "evaluate" agent skill from https://github.com/sharpdeveye/maestro/tree/main/source/skills/evaluate into .opencode/skills/evaluate/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "evaluate", then confirm the skill loads.OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
Facts
- Skill name
evaluate- GitHub stars
- 592
- Token cost
- ~776 tokens
- SKILL.md length
- 362 words
- Files
- 1
- Skills in repo
- 25
- Repo updated
- First seen
- Licence
- MIT
At a glance
A skill your agent uses when the user wants a quality review, interaction audit, or to test the workflow against realistic scenarios.
- Works in 4 steps: Overall quality grade (A-F) → Per-dimension scores with evidence → Specific scenario results → …
- The user wants a quality review
- Instructions only: no scripts, shell commands, URLs or credentials in SKILL.md
- Interaction audit
What it does
Evaluate is an agent skill from sharpdeveye/maestro. Use when the user wants a quality review, interaction audit, or to test the workflow against realistic scenarios.
Its SKILL.md is about 780 tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts.
The repository describes itself as: Workflow fluency for AI coding agents. 1 core skill · 25 commands · 7 domain references · memory layer · audit trail — works across Cursor, Claude Code, Gemini CLI, Copilot, and… The licence is MIT.
When your agent uses it
- The user wants a quality review
- Interaction audit
- Test the workflow against realistic scenarios
Example prompts
- “/evaluate”
Workflow steps
4 steps, taken from the first numbered list in SKILL.md.
- Overall quality grade (A-F)
- Per-dimension scores with evidence
- Specific scenario results
- Priority improvements with recommended Maestro commands
What it can do on your machine
Read from SKILL.md and the folder at commit 00f9115. It shows what the files ask for, not the result of running them.
Tool permissions
Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.
From allowed-tools in the SKILL.md frontmatter.
Runs code
No scripts in the folder and no shell commands in SKILL.md.
From the folder's file list and the shell code blocks in SKILL.md.
Network
No URLs in SKILL.md.
From URLs in SKILL.md, links to its own repository left out.
Credentials
Names no API keys, tokens, secrets or passwords.
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.
Context cost
Evaluate loads about 776 tokens when it runs. Until then it costs about 31 tokens; SKILL.md has 362 words of instructions outside code blocks.
Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.
Safety
Auto-check passed
The automated check found no risky patterns in SKILL.md.
Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.
SKILL.md
The full file from sharpdeveye/maestro at commit 00f9115, republished under its MIT licence (© sharpdeveye). 362 words, ~776 tokens.
Download SKILL.mdSave it as
.claude/skills/evaluate/SKILL.md (or your agent's skills folder).- name
- evaluate
- description
- Use when the user wants a quality review, interaction audit, or to test the workflow against realistic scenarios.
- argument-hint
- [workflow or scenario]
- category
- analysis
- version
- 2.0.0
- user-invocable
- true
MANDATORY PREPARATION
Invoke /agent-workflow — it contains workflow principles, anti-patterns, and the Context Gathering Protocol. Follow the protocol before proceeding — if no workflow context exists yet, you MUST run /teach-maestro first. Consult the feedback-loops reference in the agent-workflow skill for evaluation patterns, golden test sets, and regression detection.
Evaluate the workflow's actual interaction quality by testing it against scenarios that represent real usage.
Evaluation Dimensions
1. Task Completion
- Does the workflow actually accomplish what it's supposed to?
- Does it handle the complete task or only the happy path?
- Are edge cases addressed or silently dropped?
2. Output Quality
- Is the output accurate, complete, and well-formatted?
- Does it match the defined output schema (if any)?
- Would a domain expert approve the output?
3. Error Behavior
- What happens when input is malformed?
- What happens when a tool fails?
- What happens when the model is uncertain?
- Is the error message useful or generic?
4. User Experience
- Is the interaction natural and intuitive?
- Are confirmations requested for destructive operations?
- Is the response time acceptable?
- Does the workflow communicate its limitations?
5. Consistency
- Does the same input produce consistent output quality?
- Are there random failures that aren't reproducible?
- Does quality degrade over long conversations?
Show full SKILL.md (164 more words)Show less
Scenario Testing
Create and run test scenarios:
| Scenario | Input | Expected | Actual | Grade |
|---|---|---|---|---|
| Happy path | Normal input | Correct output | ? | A-F |
| Edge case | Unusual input | Graceful handling | ? | A-F |
| Error case | Bad input | Helpful error | ? | A-F |
| Stress case | Large/complex input | Reasonable handling | ? | A-F |
| Adversarial | Tricky/malicious input | Safe response | ? | A-F |
Evaluation Report
Produce a structured report with:
- Overall quality grade (A-F)
- Per-dimension scores with evidence
- Specific scenario results
- Priority improvements with recommended Maestro commands
Evaluation Checklist
- All 5 dimensions tested with concrete scenarios
- At least one edge case and one adversarial case tested
- Results documented in the scenario table
- Overall grade assigned with justification
- Improvement actions reference specific Maestro commands
Recommended Next Step
After evaluation, run /fortify to address error behavior gaps, /refine for output quality improvements, or /iterate to set up continuous quality monitoring.
NEVER:
- Evaluate theoretically — run actual scenarios
- Give an A grade unless the workflow handles all scenario types well
- Skip adversarial testing for user-facing workflows
- Evaluate only the happy path
© sharpdeveye, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file
Files
Just SKILL.md in source/skills/evaluate of sharpdeveye/maestro.
Open the folder on GitHubat commit 00f9115
Compare with similar skills
Evaluate next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.
| Skill | Stars | Used in | Tokens | Auto-check | Licence | Repo updated |
|---|---|---|---|---|---|---|
| Evaluate this skillsharpdeveye/maestro | 592 | — | ~776 | Automated safety check: Pass | MIT | |
| Arize Evaluatorgithub/awesome-copilot | 40k | 2 repos | ~8.1k | Automated safety check: Notes | MIT | |
| LLM Evaluationdavila7/claude-code-templates | 32k | 13 repos | ~3.5k | Automated safety check: Pass | MIT | |
| Agent Evaluationsickn33/agentic-awesome-skills | 47k | 1 repos | ~2k | Automated safety check: Pass | MIT | |
| EvaluatorsArize-ai/phoenix | 12k | — | ~1.7k | Automated safety check: Pass | Custom licence | |
| Agent Evaluation Reportingsickn33/agentic-awesome-skills | 47k | 1 repos | ~2.1k | Automated safety check: Pass | MIT |
Similar skills
- Official
LLM Evaluation
davila7/claude-code-templates
Master comprehensive evaluation strategies for LLM applications, from automated metrics to human evaluation and A/B testing.
Agent Evaluation
sickn33/agentic-awesome-skills
Evaluate agent behavior with versioned cases and explicit verifiers.
Evaluators
Arize-ai/phoenix
Author or refine a Phoenix evaluator — code or LLM-as-a-judge — that scores a run's output.
Agent Evaluation Reporting
sickn33/agentic-awesome-skills
A skill your agent uses when summarizing agent evaluations where autonomous, assisted, failed, timed-out, or invalid outcomes must remain distinct and comparable.
- Official
Creating Online Evaluations
PostHog/posthog
Author continuously-running online evaluations in PostHog AI observability, grounded in real failure modes you've identified.
More from sharpdeveye/maestro
All 25 skills in this repoAccelerate
sharpdeveye/maestro
A skill your agent uses when the workflow is too slow, too expensive, or both and needs latency, cost, or token usage optimization.
Chain
sharpdeveye/maestro
A skill your agent uses when the workflow needs multi-step processing with sequential, parallel, or conditional tool compositions and proper data flow.
Compose
sharpdeveye/maestro
A skill your agent uses when a single agent demonstrably cannot handle the task and multi-agent coordination is justified.
Diagnose
sharpdeveye/maestro
A skill your agent uses when the user wants to find problems, audit workflow quality, or get a comprehensive health check on their AI workflow.
Extract Pattern
sharpdeveye/maestro
A skill your agent uses when the user wants to create templates, extract reusable patterns, document solutions, or build a pattern library from working workflows.
Fortify
sharpdeveye/maestro
A skill your agent uses when the workflow lacks error handling, has been failing in production, or needs retry logic, fallback strategies, and circuit breakers.
Questions about Evaluate
What does Evaluate do?
A skill your agent uses when the user wants a quality review, interaction audit, or to test the workflow against realistic scenarios. Evaluate is an agent skill from sharpdeveye/maestro. Use when the user wants a quality review, interaction audit, or to test the workflow against realistic scenarios.
When should I use Evaluate?
Evaluate fits situations like: the user wants a quality review; interaction audit; test the workflow against realistic scenarios.
How do I install Evaluate in Claude Code?
Run `npx skills add sharpdeveye/maestro --skill evaluate -a claude-code`. Or copy the skill folder (source/skills/evaluate in sharpdeveye/maestro) into .claude/skills/evaluate in your project. Claude Code loads it when a task matches its description.
How do I install Evaluate in Codex?
Run `npx skills add sharpdeveye/maestro --skill evaluate -a codex`. Or copy the skill folder (source/skills/evaluate in sharpdeveye/maestro) into .agents/skills/evaluate in your project. Codex loads it when a task matches its description.
Can I use Evaluate in Cursor, Gemini CLI or GitHub Copilot?
Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add sharpdeveye/maestro --skill evaluate -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/evaluate, .gemini/skills/evaluate, .github/skills/evaluate and .opencode/skills/evaluate in your project.
What does Evaluate need to run?
SKILL.md names no scripts, command-line tools or credentials: Evaluate is instructions for the agent only.
Does Evaluate access the network?
SKILL.md contains no URLs. Any network use would come from the scripts or tools the agent runs. This is read from the text; nothing was executed.
Is Evaluate safe to install?
Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.
What licence does Evaluate use?
Evaluate is published under the MIT licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.
How many tokens does Evaluate use?
About 776 tokens (SKILL.md is roughly 3.1k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.
What are the alternatives to Evaluate?
Skills that share tags, products or a category with Evaluate: Arize Evaluator (github/awesome-copilot, 40k stars), LLM Evaluation (davila7/claude-code-templates, 32k stars), Agent Evaluation (sickn33/agentic-awesome-skills, 47k stars) and Evaluators (Arize-ai/phoenix, 12k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.
Who maintains Evaluate?
sharpdeveye (a GitHub user) maintains it in sharpdeveye/maestro, which has 592 GitHub stars. The repository holds 25 skills in this directory. The repository was last updated on April 29, 2026.
Source: sharpdeveye/maestro on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.