Golden Dataset

What it does

When your agent uses it

• Building evaluation datasets
• Managing dataset versions
• Validating quality scores
• Integrating golden tests into pipelines

Example prompts

• “/golden-dataset”

Requirements

• Python 3
• Compatibility (from SKILL.md): Claude Code 2.1.277+.
• Pre-approved tools (allowed-tools): Read, Glob, Grep, WebFetch, WebSearch

Workflow steps

8 steps, taken from the first numbered list in SKILL.md.

1. 1Using placeholder URLs instead of canonical source URLs
2. 2Skipping embedding regeneration after restore
3. 3Not validating referential integrity between documents and queries
4. 4Over-indexing on articles (neglecting tutorials, research papers)
5. 5Missing difficulty distribution balance in test queries
6. 6Not running verification after backup/restore operations
7. 7Testing restore procedures in production instead of staging
8. 8Committing SQL dumps instead of JSON (not version-control friendly)

What it can do on your machine

Read from SKILL.md and the folder at commit e4ff8d9. It shows what the files ask for, not the result of running them.

• Tool permissions

  Pre-approves these tools, so the agent can use them without asking each time:

  • Read
  • Glob
  • Grep
  • WebFetch
  • WebSearch

  From allowed-tools in the SKILL.md frontmatter.

• Runs code

  Ships 2 files in scripts/ (Python), which the agent can run.

  From the folder's file list and the shell code blocks in SKILL.md.

• Network

  Links to these hosts (documentation or services it may open):

  • json-schema.org
  • zod.dev
  • github.com
  • docs.github.com
  • langfuse.com
  • postgresql.org

  From URLs in SKILL.md, links to its own repository left out.

• Credentials

  Names no API keys, tokens, secrets or passwords.

  From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

• Compatibility

  Claude Code 2.1.277+.

  From compatibility in the SKILL.md frontmatter.

Context cost

Golden Dataset loads about 2k tokens when it runs, and up to ~8.1k if it reads all its reference files. Until then it costs about 63 tokens; SKILL.md has 676 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it

~63

When it runs · the whole SKILL.md, loaded when a task matches

~2k

With references · SKILL.md plus every file in references/, read only if the agent opens them

~8.1k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); the scripts in this folder are not scanned.

SKILL.md

The full file from yonatangross/orchestkit at commit e4ff8d9, republished under its MIT licence (© yonatangross). 676 words, ~2,042 tokens.

name

golden-dataset

description

Golden dataset lifecycle patterns for curation, versioning, quality validation, and CI integration. Use when building evaluation datasets, managing dataset versions, validating quality scores, or integrating golden tests into pipelines.

allowed-tools

Read, Glob, Grep, WebFetch, WebSearch

compatibility

Claude Code 2.1.277+.

license

MIT

user-invocable

false

disable-model-invocation

true

metadata.owner-agent

data-pipeline-engineer

metadata.category

document-asset-creation

metadata.version

2.0.0

metadata.author

OrchestKit

metadata.complexity

medium

metadata.tags

golden-dataset, evaluation, dataset-curation, dataset-validation, quality, llm-testing

async def validate_before_add(document: dict, source_url_map: dict) -> dict:
    """Pre-addition validation for golden dataset entries."""
    errors = []

    # 1. URL contract check
    if "placeholder" in document.get("source_url", ""):
        errors.append("URL must be canonical, not a placeholder")

    # 2. Content quality
    if len(document.get("title", "")) < 10:
        errors.append("Title too short (min 10 chars)")

    # 3. Tag requirements
    if len(document.get("tags", [])) < 2:
        errors.append("At least 2 domain tags required")

    return {"valid": len(errors) == 0, "errors": errors}

import { LangfuseClient } from "@langfuse/client";

const langfuse = new LangfuseClient();
const dataset = await langfuse.dataset.get("my-evaluation-dataset");

const result = await dataset.runExperiment({
  name: "Retrieval quality",
  task: myTask,               // (params) => Promise<any>
  evaluators: [myEvaluator],  // per-item: (params) => Promise<Evaluation | Evaluation[]>
});

© yonatangross, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 16 other files (scripts, references) in src/skills/golden-dataset of yonatangross/orchestkit.

• SKILL.md
• metadata.json
• references/ork-delta.md
• references/quality-metrics.md
• references/storage-patterns.md
• references/versioning.md
• rules/_sections.md
• rules/_template.md
• rules/curation-add-workflow.md
• rules/curation-annotation.md
• rules/curation-collection.md
• rules/management-storage.md
• rules/management-versioning.md
• rules/validation-regression.md
• scripts/backup-golden-dataset.md
• scripts/backup-script.py
• test-cases.json

Open the folder on GitHubat commit e4ff8d9