New: 98,000+ skills indexed from GitHub repos with 100+ stars · every skill safety-checkedBrowse top skillsTop skills
Esc to closeBrowse all skills
Submit a skillBrowse skills
Submit a skillBrowse skills

Agent skill

Synthetic Eval Data Generator

by ai-evals-course in ai-evals-course/evals-skills

Builds diverse synthetic test inputs for LLM pipeline evaluation by defining failure-focused dimensions, drafting tuples with you and turning them into realistic queries.

Apache-2.0Auto-check passedAI & LLM Engineering

Install Synthetic Eval Data Generator

skills CLI
$ npx skills add ai-evals-course/evals-skills --skill generate-synthetic-data -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI
$ gh skill install ai-evals-course/evals-skills generate-synthetic-data --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy
$ git clone --depth 1 https://github.com/ai-evals-course/evals-skills.git skills-src && mkdir -p .claude/skills && cp -r skills-src/skills/generate-synthetic-data .claude/skills/generate-synthetic-data && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Facts

Skill name
generate-synthetic-data
GitHub stars
1.5k
Token cost
~1.4k tokens
SKILL.md length
489 words
Files
2
Skills in repo
9
Repo updated
First seen
Licence
Apache-2.0

At a glance

Builds diverse synthetic test inputs for LLM pipeline evaluation by defining failure-focused dimensions, drafting tuples with you and turning them into realistic queries.

  • Works in 6 steps: Define Dimensions → Draft 20 Tuples with the User → Generate More Tuples with an LLM → …
  • Bootstrapping an eval dataset before any real user data exists
  • SKILL.md covers Prerequisites, Core Process, Sampling Real User Data and Anti-Patterns
  • Instructions only: no scripts, shell commands, URLs or credentials in SKILL.md

What it does

The process begins by finding where the pipeline is likely to fail, through questions to you, existing user feedback or hypotheses from traces. The agent then defines dimensions, the axes of variation specific to the application (for a real estate assistant: task, buyer persona and how clear the request is), starting with three. It drafts 20 tuples with you, each one a combination of dimension values, and iterates until you agree they reflect scenarios that really occur.

More tuples are generated by an LLM, and each is converted into a natural-language query in a separate prompt, since doing both in one step makes the phrasing repetitive. Queries that sound awkward, miss the tuple's intent or resemble each other are discarded, optionally using a 1 to 5 realism rating with a cutoff at 3. All queries then run through the full pipeline with complete traces captured. It is not meant for cases where you already have 100 or more representative real traces, or for collecting production logs.

When your agent uses it

  • Bootstrapping an eval dataset before any real user data exists
  • Stress-testing a specific suspected failure of an LLM application
  • Broadening test coverage when real traces are sparse

Example prompts

  • “Help me build a synthetic eval set for my support chatbot, starting with where it tends to fail.”
  • “Draft 20 dimension tuples for a recipe assistant and let me adjust them.”
  • “Our real traces are sparse; generate test inputs for the booking agent covering ambiguous and out-of-scope requests.”

Requirements

  • An LLM pipeline you can run test queries through

Workflow steps

6 steps, taken from the step headings in SKILL.md.

  1. Define Dimensions
  2. Draft 20 Tuples with the User
  3. Generate More Tuples with an LLM
  4. Convert Each Tuple to a Natural Language Query
  5. Filter for Quality
  6. Run Queries Through the Pipeline

What it can do on your machine

Read from SKILL.md and the folder at commit 80d5f7b. It shows what the files ask for, not the result of running them.

  • Tool permissions

    Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

    From allowed-tools in the SKILL.md frontmatter.

  • Runs code

    No scripts in the folder and no shell commands in SKILL.md.

    From the folder's file list and the shell code blocks in SKILL.md.

  • Network

    No URLs in SKILL.md.

    From URLs in SKILL.md, links to its own repository left out.

  • Credentials

    Names no API keys, tokens, secrets or passwords.

    From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Synthetic Eval Data Generator loads about 1.4k tokens when it runs. Until then it costs about 100 tokens; SKILL.md has 489 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it
~100
When it runs · the whole SKILL.md, loaded when a task matches
~1.4k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from ai-evals-course/evals-skills at commit 80d5f7b, republished under its Apache-2.0 licence (© ai-evals-course). 489 words, ~1,354 tokens.

Download SKILL.mdSave it as .claude/skills/generate-synthetic-data/SKILL.md (or your agent's skills folder). This skill also uses 1 other file; get the full folder from GitHub.
name
generate-synthetic-data
description
Create diverse synthetic test inputs for LLM pipeline evaluation using dimension-based tuple generation. Use when bootstrapping an eval dataset, when real user data is sparse, or when stress-testing specific failure hypotheses. Do NOT use when you already have 100+ representative real traces (use stratified sampling instead), or when the task is collecting production logs.

Generate Synthetic Data

Generate diverse, realistic test inputs that cover the failure space of an LLM pipeline.

Prerequisites

Before generating synthetic data, identify where the pipeline is likely to fail. Ask the user about known failure-prone areas, review existing user feedback, or form hypotheses from available traces. Dimensions (Step 1) must target anticipated failures, not arbitrary variation.

Core Process

Step 1: Define Dimensions

Dimensions are axes of variation specific to your application. Choose dimensions based on where you expect failures.

Dimension 1: [Name] — [What it captures]
  Values: [value_a, value_b, value_c, ...]

Dimension 2: [Name] — [What it captures]
  Values: [value_a, value_b, value_c, ...]

Dimension 3: [Name] — [What it captures]
  Values: [value_a, value_b, value_c, ...]

Example for a real estate assistant:

Feature: what task the user wants
  Values: [property search, scheduling, email drafting]

Client Persona: who the user serves
  Values: [first-time buyer, investor, luxury buyer]

Scenario Type: query clarity
  Values: [well-specified, ambiguous, out-of-scope]

Start with 3 dimensions. Add more only if initial traces reveal failure patterns along new axes.

Step 2: Draft 20 Tuples with the User

A tuple is one combination of dimension values defining a specific test case. Present 20 draft tuples to the user and iterate until they confirm the tuples reflect realistic scenarios. The user's domain knowledge is essential here — they know which combinations actually occur and which are unrealistic.

(Feature: Property Search, Persona: Investor, Scenario: Ambiguous)
(Feature: Scheduling, Persona: First-time Buyer, Scenario: Well-specified)
(Feature: Email Drafting, Persona: Luxury Buyer, Scenario: Out-of-scope)
Step 3: Generate More Tuples with an LLM
Generate 10 random combinations of ({dim1}, {dim2}, {dim3})
for a {your application description}.

The dimensions are:
{dim1}: {description}. Possible values: {values}
{dim2}: {description}. Possible values: {values}
{dim3}: {description}. Possible values: {values}

Output each tuple in the format: ({dim1}, {dim2}, {dim3})
Avoid duplicates. Vary values across dimensions.
Step 4: Convert Each Tuple to a Natural Language Query

Use a separate prompt for this step. Single-step generation (tuples + queries together) produces repetitive phrasing.

We are generating synthetic user queries for a {your application}.
{Brief description of what it does.}

Given:
{dim1}: {value}
{dim2}: {value}
{dim3}: {value}

Write a realistic query that a user might enter. The query should
reflect the specified persona and scenario characteristics.

Example: "{one of your hand-written examples}"

Now generate a new query.
Step 5: Filter for Quality

Review generated queries. Discard and regenerate when:

  • Phrasing is awkward or unrealistic
  • Content doesn't match the tuple's intent
  • Queries are too similar to each other

Optional: use an LLM to rate realism on a 1-5 scale, discard below 3.

Step 6: Run Queries Through the Pipeline

Execute all queries through the full LLM pipeline. Capture complete traces: input, all intermediate steps, tool calls, retrieved docs, final output.

Target: ~100 high-quality, diverse traces. This is a rough heuristic for reaching saturation (where new traces stop revealing new failure categories). The number depends on system complexity.

Show full SKILL.md (200 more words)Show less

Sampling Real User Data

When you have real queries available, don't sample randomly. Use stratified sampling:

  1. Identify high-variance dimensions — read through queries and find ways they differ (length, topic, complexity, presence of constraints).
  2. Assign labels — for small sets, with the user; for large sets, use K-means clustering on query embeddings.
  3. Sample from each group — ensures coverage across query types, not just the most common ones.

When both real and synthetic data are available, use synthetic data to fill gaps in underrepresented query types.

Anti-Patterns

  • Unstructured generation. Prompting "give me test queries" without the dimension/tuple structure produces generic, repetitive, happy-path examples.
  • Single-step generation. Generating tuples and queries in one prompt produces less diverse results than the two-step separation.
  • Arbitrary dimensions. Dimensions that don't target failure-prone regions waste test budget.
  • Skipping user review of tuples. Without the user validating tuples first, you can't judge whether LLM-generated tuples are realistic.
  • Synthetic data when no one can judge realism. If no one can judge whether a synthetic trace is realistic, use real data instead.
  • Synthetic data for complex domain-specific content (legal filings, medical records) where LLMs miss structural nuance.
  • Synthetic data for low-resource languages or dialects where LLM-generated samples are unrealistic.

© ai-evals-course, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 1 other file in skills/generate-synthetic-data of ai-evals-course/evals-skills.

  • SKILL.md
  • agents/openai.yaml

Open the folder on GitHubat commit 80d5f7b

Compare with similar skills

Synthetic Eval Data Generator next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Synthetic Eval Data Generator compared with similar skills
SkillStarsUsed inTokensAuto-checkLicenceRepo updated
Synthetic Eval Data Generator this skillai-evals-course/evals-skills1.5k—~1.4kAutomated safety check: PassApache-2.0
Agent Eval Engineeringlangchain-ai/langchain-skills1.3k—~4kAutomated safety check: PassMIT
Chatbox Session RAG Evalchatboxai/chatbox42k—~758Automated safety check: PassGPL-3.0
Quality FlywheelGoogleCloudPlatform/vertex-ai-samples792—~2kAutomated safety check: PassApache-2.0
Axiom Eval Writeropenclaw/clawhub9.5k—~4.1kAutomated safety check: WarnMIT
Opik Evaluatecomet-ml/opik-mcp220—~2.5kAutomated safety check: NotesApache-2.0

Similar skills

  • Agent Eval Engineering

    langchain-ai/langchain-skills

    Official

    Builds agent evaluations in stages: inspect the repository and traces, agree a Task Spec with you, then build, audit and run a Harbor task with an independent verifier.

    1.3k GitHub stars~4k tokensUpdated 2 days ago
    AI & LLM EngineeringAuto-check passed

  • Chatbox Session RAG Eval

    chatboxai/chatbox

    Runs and debugs evaluations of how Chatbox models answer questions about large attached files, using synthetic and real long-document fixtures.

    42k GitHub stars~758 tokensUpdated 16 days ago
    AI & LLM EngineeringAuto-check passed

  • Quality Flywheel

    GoogleCloudPlatform/vertex-ai-samples

    Evaluate and improve GenAI models and agents using the Google GenAI Evaluation SDK.

    792 GitHub stars~2k tokensUpdated yesterday
    AI & LLM EngineeringAuto-check passed

  • Axiom Eval Writer

    openclaw/clawhub

    Scaffolds evaluation suites for the Axiom AI SDK: eval files, scorers, flag schemas and axiom.config.ts, generated from plain descriptions of an AI capability.

    9.5k GitHub stars~4.1k tokensUpdated yesterday
    AI & LLM EngineeringAuto-check: warnings

  • Opik Evaluate

    comet-ml/opik-mcp

    Build an LLM evaluation and run it against the app, returning an Opik experiment with scores and its link.

    220 GitHub stars~2.5k tokensUpdated 2 days ago
    AI & LLM EngineeringAuto-check: notes

  • Eval Guide

    microsoft/eval-guide

    Official

    Eval enablement accelerator — help customers think through "what does good look like" for their AI agent, then generate a structured eval plan and test cases they can use immediately.

    138 GitHub stars~22k tokensUpdated 3 mo ago
    Testing & QAAuto-check: warnings

More from ai-evals-course/evals-skills

All 9 skills in this repo

  • LLM Trace Review Interface

    ai-evals-course/evals-skills

    Builds a browser-based annotation page for reviewing LLM traces one at a time with pass/fail labels, notes and saved results, tailored to your data.

    1.5k GitHub stars~1.4k tokensUpdated 16 days ago
    Auto-check passed

  • LLM Eval Pipeline Audit

    ai-evals-course/evals-skills

    Inspects an LLM evaluation setup for missing error analysis, unvalidated judges and vanity metrics, and ranks the problems by impact with fixes.

    1.5k GitHub stars~2.5k tokensUpdated 16 days ago
    Auto-check passed

  • Evaluate RAG

    ai-evals-course/evals-skills

    Guides evaluation of a RAG system by diagnosing failures in traces, building a retrieval test set and scoring retrieval and generation separately.

    1.5k GitHub stars~1.9k tokensUpdated 16 days ago
    Auto-check passed

  • LLM Judge Validation

    ai-evals-course/evals-skills

    Checks an LLM judge against human labels using train, dev and test splits, TPR and TNR, and a bias correction applied to production data.

    1.5k GitHub stars~2.2k tokensUpdated 16 days ago
    Auto-check passed

  • LLM-as-Judge Prompt Writer

    ai-evals-course/evals-skills

    Designs a binary Pass/Fail LLM-as-Judge prompt for one subjective failure mode, built from a task statement, clear definitions, labeled examples and a structured output format.

    1.5k GitHub stars~1.9k tokensUpdated 16 days ago
    Auto-check passed

  • Write Code Eval

    ai-evals-course/evals-skills

    Write code evaluators for known failure modes with objective rules.

    1.5k GitHub stars~385 tokensUpdated 16 days ago
    Auto-check passed

Categories

Questions about Synthetic Eval Data Generator

What does Synthetic Eval Data Generator do?

Builds diverse synthetic test inputs for LLM pipeline evaluation by defining failure-focused dimensions, drafting tuples with you and turning them into realistic queries. The process begins by finding where the pipeline is likely to fail, through questions to you, existing user feedback or hypotheses from traces. The agent then defines dimensions, the axes of variation specific to the application (for a real estate assistant: task, buyer persona and how clear the request is), starting with three.

When should I use Synthetic Eval Data Generator?

Synthetic Eval Data Generator fits situations like: bootstrapping an eval dataset before any real user data exists; stress-testing a specific suspected failure of an LLM application; broadening test coverage when real traces are sparse.

How do I install Synthetic Eval Data Generator in Claude Code?

Run `npx skills add ai-evals-course/evals-skills --skill generate-synthetic-data -a claude-code`. Or copy the skill folder (skills/generate-synthetic-data in ai-evals-course/evals-skills) into .claude/skills/generate-synthetic-data in your project. Claude Code loads it when a task matches its description.

How do I install Synthetic Eval Data Generator in Codex?

Run `npx skills add ai-evals-course/evals-skills --skill generate-synthetic-data -a codex`. Or copy the skill folder (skills/generate-synthetic-data in ai-evals-course/evals-skills) into .agents/skills/generate-synthetic-data in your project. Codex loads it when a task matches its description.

Can I use Synthetic Eval Data Generator in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add ai-evals-course/evals-skills --skill generate-synthetic-data -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/generate-synthetic-data, .gemini/skills/generate-synthetic-data, .github/skills/generate-synthetic-data and .opencode/skills/generate-synthetic-data in your project.

What does Synthetic Eval Data Generator need to run?

SKILL.md names no scripts, command-line tools or credentials: Synthetic Eval Data Generator is instructions for the agent only. Our summary lists: An LLM pipeline you can run test queries through.

Does Synthetic Eval Data Generator access the network?

SKILL.md contains no URLs. Any network use would come from the scripts or tools the agent runs. This is read from the text; nothing was executed.

Is Synthetic Eval Data Generator safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.

What licence does Synthetic Eval Data Generator use?

Synthetic Eval Data Generator is published under the Apache-2.0 licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does Synthetic Eval Data Generator use?

About 1.4k tokens (SKILL.md is roughly 5.4k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.

What are the alternatives to Synthetic Eval Data Generator?

Skills that share tags, products or a category with Synthetic Eval Data Generator: Agent Eval Engineering (langchain-ai/langchain-skills, 1.3k stars), Chatbox Session RAG Eval (chatboxai/chatbox, 42k stars), Quality Flywheel (GoogleCloudPlatform/vertex-ai-samples, 792 stars) and Axiom Eval Writer (openclaw/clawhub, 9.5k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Synthetic Eval Data Generator?

ai-evals-course (a GitHub organization) maintains it in ai-evals-course/evals-skills, which has 1,479 GitHub stars. The repository holds 9 skills in this directory. The repository was last updated on September 24, 2026.

Source: ai-evals-course/evals-skills on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.