New: 98,000+ skills indexed from GitHub repos with 100+ stars · every skill safety-checkedBrowse top skillsTop skills
Esc to closeBrowse all skills
Submit a skillBrowse skills
Submit a skillBrowse skills

Agent skill

Dspy Evaluation Harness

by intertwine in intertwine/dspy-agent-skills

Build DSPy evaluation harnesses with rich-feedback metrics that are essential for GEPA optimization.

MITAuto-check passedAI & LLM Engineering

Install Dspy Evaluation Harness

skills CLI
$ npx skills add intertwine/dspy-agent-skills --skill dspy-evaluation-harness -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI
$ gh skill install intertwine/dspy-agent-skills dspy-evaluation-harness --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy
$ git clone --depth 1 https://github.com/intertwine/dspy-agent-skills.git skills-src && mkdir -p .claude/skills && cp -r skills-src/skills/dspy-evaluation-harness .claude/skills/dspy-evaluation-harness && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Facts

Skill name
dspy-evaluation-harness
GitHub stars
278
Token cost
~1.5k tokens
SKILL.md length
356 words
Files
3
Skills in repo
5
Repo updated
First seen
Licence
MIT

At a glance

Build DSPy evaluation harnesses with rich-feedback metrics that are essential for GEPA optimization.

  • Works in 2 steps: Return a dspy.Prediction(score=...,… → Separate valset. Never optimize and…
  • Writing a metric function
  • SKILL.md covers Two rules, Canonical rich-feedback metric, Canonical harness and Dataset hygiene, plus 5 more sections
  • Runs Python scripts from its folder

What it does

Dspy Evaluation Harness is an agent skill from intertwine/dspy-agent-skills. Build DSPy evaluation harnesses with rich-feedback metrics that are essential for GEPA optimization. Use when writing a metric function, calling dspy.Evaluate, splitting dev/val sets, debugging "why is my optimizer not improving?", or designing CI-ready DSPy eval suites.

Its SKILL.md is about 1.5k tokens, which your agent loads only when the skill is triggered. The skill folder holds 2 other files (for example `example_metric.py` and `reference.md`).

It sits in AI & LLM Engineering, covering LLM evaluation and Structured output and tool calling. It works with MLflow. The repository describes itself as: Production-grade DSPy 3.2.x agent skills + validated end-to-end examples for Claude Code and Codex CLI — fundamentals, evaluation, GEPA, BetterTogether, and RLM. The licence is MIT.

When your agent uses it

  • Writing a metric function
  • Calling dspy.Evaluate
  • Splitting dev/val sets
  • Debugging why is my optimizer not improving?

Example prompts

  • “why is my optimizer not improving?”
  • “/dspy-evaluation-harness”

Requirements

  • Python 3

Workflow steps

2 steps, taken from the first numbered list in SKILL.md.

  1. Return a dspy.Prediction(score=..., feedback=...), not a dict. dspy.Evaluate's parallel executor aggregates scores via sum, which breaks…
  2. Separate valset. Never optimize and evaluate on the same examples. Optimizers overfit fast.

What it can do on your machine

Read from SKILL.md and the folder at commit 623dca0. It shows what the files ask for, not the result of running them.

  • Tool permissions

    Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

    From allowed-tools in the SKILL.md frontmatter.

  • Runs code

    Ships script files (Python), which the agent can run.

    From the folder's file list and the shell code blocks in SKILL.md.

  • Network

    No URLs in SKILL.md.

    From URLs in SKILL.md, links to its own repository left out.

  • Credentials

    Names no API keys, tokens, secrets or passwords.

    From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Dspy Evaluation Harness loads about 1.5k tokens when it runs. Until then it costs about 74 tokens; SKILL.md has 356 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it
~74
When it runs · the whole SKILL.md, loaded when a task matches
~1.5k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from intertwine/dspy-agent-skills at commit 623dca0, republished under its MIT licence (© intertwine). 356 words, ~1,455 tokens.

Download SKILL.mdSave it as .claude/skills/dspy-evaluation-harness/SKILL.md (or your agent's skills folder). This skill also uses 2 other files; get the full folder from GitHub.
name
dspy-evaluation-harness
description
Build DSPy evaluation harnesses with rich-feedback metrics that are essential for GEPA optimization. Use when writing a metric function, calling dspy.Evaluate, splitting dev/val sets, debugging "why is my optimizer not improving?", or designing CI-ready DSPy eval suites.
when_to_use
User mentions `dspy.Evaluate`, a "metric", a devset/valset/trainset, evaluation, scoring, or asks why their GEPA optimization isn't converging ; consider…

DSPy Evaluation Harness (3.2.x)

The metric is usually more important than the program. For dspy.GEPA especially, the quality of textual feedback in your metric determines whether optimization converges.

Two rules

  1. Return a dspy.Prediction(score=..., feedback=...), not a dict. dspy.Evaluate's parallel executor aggregates scores via sum, which breaks on dict outputs (TypeError: unsupported operand type(s) for +: 'int' and 'dict'). dspy.Prediction supports __float__/__add__ and is what GEPA's adapter natively unwraps. A bare float still works for pure dspy.Evaluate scoring, but GEPA needs the score+feedback pair.
  2. Separate valset. Never optimize and evaluate on the same examples. Optimizers overfit fast.

Canonical rich-feedback metric

python
import dspy

def rich_metric(gold: dspy.Example, pred: dspy.Prediction, trace=None,
                pred_name: str | None = None, pred_trace=None):
    # 1. Compute sub-scores — multi-axis beats scalar
    correctness = 1.0 if _normalize(pred.answer) == _normalize(gold.answer) else 0.0
    cited = _has_citation(pred.answer, gold.sources) if hasattr(gold, "sources") else 1.0
    concise = 1.0 if len(pred.answer.split()) <= 50 else 0.5
    score = 0.6 * correctness + 0.25 * cited + 0.15 * concise

    # 2. Write feedback that teaches the optimizer
    parts = []
    if correctness < 1.0:
        parts.append(
            f"Answer mismatch. Predicted: {pred.answer!r}. Expected: {gold.answer!r}. "
            f"Likely cause: reasoning skipped the units/quantity in the question."
        )
    if cited < 1.0:
        parts.append("Did not ground the claim in the provided sources. Quote a source fragment.")
    if concise < 1.0:
        parts.append("Answer exceeded 50 words — tighten to one sentence.")
    if not parts:
        parts.append("Correct, grounded, and concise.")
    feedback = " ".join(parts)

    return dspy.Prediction(score=score, feedback=feedback)

Canonical harness

python
evaluator = dspy.Evaluate(
    devset=valset,
    metric=rich_metric,
    num_threads=8,
    display_progress=True,
    display_table=10,          # pretty-print first 10 rows
    provide_traceback=True,    # surface exceptions, don't swallow them
    max_errors=5,
    failure_score=0.0,
    save_as_json="eval_runs/baseline.json",
)
result = evaluator(program)
print("Overall:", result.score)
for example_result in result.results[:3]:
    print(example_result)

dspy.Evaluate returns an EvaluationResult with .score (aggregate float) and .results (list of (example, pred, score) tuples).

Dataset hygiene

  • Size: 20–50 examples is enough for GEPA's reflective loop; 100–500 for MIPROv2-style bootstrapping.
  • Split: hand-curate two disjoint sets — trainset (for optimization) and valset (for metric-on-optimized-program). A test set you never look at during development is gold.
  • Representativeness beats size. Include edge cases, ambiguity, adversarial inputs.
  • Build dspy.Example(...).with_inputs("question", "context") — the with_inputs call marks which fields are inputs vs. gold outputs.
python
trainset = [
    dspy.Example(question="…", answer="…").with_inputs("question"),
    ...
]

Combine correctness, faithfulness, format adherence, latency, and cost. Each axis should be a 0–1 float with a written definition. Weight them explicitly; don't hide weights inside magic numbers — make them constants so optimizers can be told to trade off.

Show full SKILL.md (135 more words)Show less

CI-ready eval suite

python
# tests/test_dspy_eval.py
import dspy, pytest
from my_program import program, valset, rich_metric

@pytest.fixture(scope="module")
def evaluator():
    return dspy.Evaluate(devset=valset, metric=rich_metric, num_threads=8,
                         display_progress=False, provide_traceback=True)

def test_program_meets_threshold(evaluator):
    result = evaluator(program)
    assert result.score >= 0.75, f"Regression: {result.score:.3f}"

Run offline in CI with a cached LM (dspy.LM(..., cache=True)) + pre-populated DSPY_CACHEDIR.

Tracing & observability

  • track_usage=True on dspy.configure accumulates token counts on predictions (pred.get_lm_usage()).
  • MLflow: import mlflow; mlflow.dspy.autolog() → traces every prediction.
  • W&B: pass use_wandb=True to dspy.GEPA to log Pareto fronts.
  • Always log eval results to a versioned JSON file (save_as_json=...) so you can diff runs.

Anti-patterns

  • Scalar-only metrics (float but no feedback) when using GEPA — wasted signal.
  • return {"score": s, "feedback": f} (dict) — crashes dspy.Evaluate's parallel aggregator. Use dspy.Prediction(score=s, feedback=f).
  • Exact-match metrics on open-ended generation tasks — use semantic or LM-as-judge scoring.
  • Evaluating on the trainset — optimistic by 10–30 points.
  • Silently swallowing exceptions (provide_traceback=False) — you'll blame the LM for a KeyError.
  • Changing the metric mid-experiment without re-baselining — prior numbers become incomparable.

Next

© intertwine, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 2 other files in skills/dspy-evaluation-harness of intertwine/dspy-agent-skills.

  • SKILL.md
  • example_metric.py
  • reference.md

Open the folder on GitHubat commit 623dca0

Compare with similar skills

Dspy Evaluation Harness next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Dspy Evaluation Harness compared with similar skills
SkillStarsUsed inTokensAuto-checkLicenceRepo updated
Dspy Evaluation Harness this skillintertwine/dspy-agent-skills278—~1.5kAutomated safety check: PassMIT
Vss Benchmark Vlm QANVIDIA-AI-Blueprints/video-search-and-summarization1.9k—~2.5kAutomated safety check: PassApache-2.0
Building Agent Systemstelagod/code-abyss243—~691Automated safety check: PassMIT
MLtelagod/code-abyss243—~566Automated safety check: PassMIT
Prompt EngineerJeffallan/claude-skills12k—~1.5kAutomated safety check: PassMIT
Agent Harness DesignAnastasiyaW/codex-claude-code-config154—~764Automated safety check: PassMIT

Similar skills

  • Vss Benchmark Vlm QA

    NVIDIA-AI-Blueprints/video-search-and-summarization

    Benchmark video Q&A accuracy and latency of a deployed RT-VLM (Cosmos Reason 3) via vss vlm run, using questions and videos from the DSS vss-devx-base dataset.

    1.9k GitHub stars~2.5k tokensUpdated today
    AI & LLM EngineeringAuto-check passed

  • Building Agent Systems

    telagod/code-abyss

    AI agent and LLM system engineering reference covering single-agent dev (ReAct, tool calling, plan-execute), multi-agent coordination (swarm, role decomposition, file locking), LLM security (prompt…

    243 GitHub stars~691 tokensUpdated 2 mo ago
    AI & LLM EngineeringAuto-check passed

  • ML

    telagod/code-abyss

    Machine learning and LLM engineering judgment, distilled from a stronger model - invoke when DECIDING whether/how to use ML or an LLM for a task (prompt vs RAG vs fine-tune vs classical); working…

    243 GitHub stars~566 tokensUpdated 2 mo ago
    AI & LLM EngineeringAuto-check passed

  • Prompt Engineer

    Jeffallan/claude-skills

    Designs, tests and refines LLM prompts: zero-shot, few-shot and chain-of-thought patterns, system prompts, structured output schemas and evaluation test suites.

    12k GitHub stars~1.5k tokensUpdated 6 days ago
    AI & LLM EngineeringAuto-check passed

  • Agent Harness Design

    AnastasiyaW/codex-claude-code-config

    Designing agent harnesses and tool systems — risk taxonomy for tools, permission decisions, draft/commit pattern, structured tool results, agent budgets (10 types), context trust labels against…

    154 GitHub stars~764 tokensUpdated today
    AI & LLM EngineeringAuto-check passed

  • Pydanticai

    magnus919/agent-skills

    Build type-safe AI agents and graph-based workflows with PydanticAI and PydanticGraph.

    116 GitHub stars~4k tokensUpdated yesterday
    AI & LLM EngineeringAuto-check passed

More from intertwine/dspy-agent-skills

  • Dspy Advanced Workflow

    intertwine/dspy-agent-skills

    Build DSPy 3.2.x programs through spec, program, metric and baseline; extend to optimization and export when requested and justified by task budget.

    278 GitHub stars~1.7k tokensUpdated 1 mo ago
    Auto-check passed

  • Dspy Fundamentals

    intertwine/dspy-agent-skills

    Write idiomatic DSPy 3.2.x programs — typed Signatures, dspy.Module subclasses, Predict/ChainOfThought/ReAct/ProgramOfThought, and save/load.

    278 GitHub stars~1.4k tokensUpdated 1 mo ago
    Auto-check passed

  • Dspy Gepa Optimizer

    intertwine/dspy-agent-skills

    Optimize DSPy programs with dspy.GEPA — a reflective/evolutionary optimizer to consider against task-specific baselines within an authorized evaluation budget.

    278 GitHub stars~2.6k tokensUpdated 1 mo ago
    Auto-check passed

  • Dspy Rlm Module

    intertwine/dspy-agent-skills

    Use dspy.RLM (Recursive Language Model) for reasoning over contexts too large to fit in an LLM's working window — entire codebases, long logs, massive documents, or multi-step data exploration that…

    278 GitHub stars~1.3k tokensUpdated 1 mo ago
    Auto-check passed

Works with

Categories

Questions about Dspy Evaluation Harness

What does Dspy Evaluation Harness do?

Build DSPy evaluation harnesses with rich-feedback metrics that are essential for GEPA optimization. Dspy Evaluation Harness is an agent skill from intertwine/dspy-agent-skills. Build DSPy evaluation harnesses with rich-feedback metrics that are essential for GEPA optimization.

When should I use Dspy Evaluation Harness?

Dspy Evaluation Harness fits situations like: writing a metric function; calling dspy.Evaluate; splitting dev/val sets; debugging why is my optimizer not improving?.

How do I install Dspy Evaluation Harness in Claude Code?

Run `npx skills add intertwine/dspy-agent-skills --skill dspy-evaluation-harness -a claude-code`. Or copy the skill folder (skills/dspy-evaluation-harness in intertwine/dspy-agent-skills) into .claude/skills/dspy-evaluation-harness in your project. Claude Code loads it when a task matches its description.

How do I install Dspy Evaluation Harness in Codex?

Run `npx skills add intertwine/dspy-agent-skills --skill dspy-evaluation-harness -a codex`. Or copy the skill folder (skills/dspy-evaluation-harness in intertwine/dspy-agent-skills) into .agents/skills/dspy-evaluation-harness in your project. Codex loads it when a task matches its description.

Can I use Dspy Evaluation Harness in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add intertwine/dspy-agent-skills --skill dspy-evaluation-harness -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/dspy-evaluation-harness, .gemini/skills/dspy-evaluation-harness, .github/skills/dspy-evaluation-harness and .opencode/skills/dspy-evaluation-harness in your project.

What does Dspy Evaluation Harness need to run?

Going by SKILL.md and its folder, Dspy Evaluation Harness needs Python for the scripts in its folder. Our summary lists: Python 3.

Does Dspy Evaluation Harness access the network?

SKILL.md contains no URLs. Any network use would come from the scripts or tools the agent runs. This is read from the text; nothing was executed.

Is Dspy Evaluation Harness safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.

What licence does Dspy Evaluation Harness use?

Dspy Evaluation Harness is published under the MIT licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does Dspy Evaluation Harness use?

About 1.5k tokens (SKILL.md is roughly 5.8k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.

What are the alternatives to Dspy Evaluation Harness?

Skills that share tags, products or a category with Dspy Evaluation Harness: Vss Benchmark Vlm QA (NVIDIA-AI-Blueprints/video-search-and-summarization, 1.9k stars), Building Agent Systems (telagod/code-abyss, 243 stars), ML (telagod/code-abyss, 243 stars) and Prompt Engineer (Jeffallan/claude-skills, 12k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Dspy Evaluation Harness?

intertwine (a GitHub user) maintains it in intertwine/dspy-agent-skills, which has 278 GitHub stars. The repository holds 5 skills in this directory. The repository was last updated on September 6, 2026.

Source: intertwine/dspy-agent-skills on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.