Quality Flywheel

What it does

When your agent uses it

• Asked to evaluate my agent
• Evaluate my model
• Create eval dataset
• Analyze eval results

Example prompts

• “evaluate my agent”
• “evaluate my model”
• “create eval dataset”
• “/quality-flywheel”

Requirements

• Python 3

Workflow steps

6 steps, taken from the step headings in SKILL.md.

1. 1Setup & Project Initialization
2. 2Dataset Creation & Formatting
3. 3Metric Selection & Customization
4. 4Automated Execution
5. 5Result Analysis & Auto-Optimization
6. 6Iterate (The Flywheel)

What it can do on your machine

Read from SKILL.md and the folder at commit d0aed81. It shows what the files ask for, not the result of running them.

• Tool permissions

  Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

  From allowed-tools in the SKILL.md frontmatter.

• Runs code

  Ships 3 files in scripts/ (Python), which the agent can run.

  From the folder's file list and the shell code blocks in SKILL.md.

• Network

  No URLs in SKILL.md.

  From URLs in SKILL.md, links to its own repository left out.

• Credentials

  Names no API keys, tokens, secrets or passwords.

  From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Quality Flywheel loads about 2k tokens when it runs, and up to ~9.9k if it reads all its reference files. Until then it costs about 155 tokens; SKILL.md has 607 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it

~155

When it runs · the whole SKILL.md, loaded when a task matches

~2k

With references · SKILL.md plus every file in references/, read only if the agent opens them

~9.9k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); the scripts in this folder are not scanned.

SKILL.md

The full file from GoogleCloudPlatform/vertex-ai-samples at commit d0aed81, republished under its Apache-2.0 licence (© GoogleCloudPlatform). 607 words, ~1,979 tokens.

name

quality-flywheel

description

Evaluate and improve GenAI models and agents using the Google GenAI Evaluation SDK. Creates eval datasets (from session traces or synthetic generation), selects and configures metrics (RubricMetric, LLMMetric, CodeExecutionMetric), executes evals via client.evals.evaluate(), and analyzes results to suggest concrete fixes. Supports both single-turn model evaluation and multi-turn agent trajectory evaluation. Use when asked to "evaluate my agent", "evaluate my model", "create eval dataset", "run evals", "analyze eval results", "which metrics should I use", "generate test data", or "improve quality".

import vertexai
from vertexai import Client, types
from google.genai import types as genai_types

# Initialize client
client = vertexai.Client(project="PROJECT_ID", location="LOCATION")

# --- SINGLE-TURN EVAL ---
dataset = types.EvaluationDataset(eval_cases=[
    types.EvalCase(prompt="Query here", response="Model response here"),
])

# --- MULTI-TURN AGENT EVAL ---
agent_data = types.evals.AgentData(
    agents={"my_agent": types.evals.AgentConfig(
        agent_id="my_agent", instruction="You are helpful.")},
    turns=[types.evals.ConversationTurn(turn_index=0, events=[
        types.evals.AgentEvent(author="user",
            content=genai_types.Content(role="user",
                parts=[genai_types.Part(text="Hello")])),
        types.evals.AgentEvent(author="my_agent",
            content=genai_types.Content(role="model",
                parts=[genai_types.Part(text="Hi! How can I help?")])),
    ])],
)
dataset = types.EvaluationDataset(
    eval_cases=[types.EvalCase(agent_data=agent_data)])

# --- METRICS ---
predefined = types.RubricMetric.MULTI_TURN_TRAJECTORY_QUALITY
custom_llm = types.LLMMetric(name="tone",
    prompt_template="Is this polite? Response: {response}")
custom_code = types.CodeExecutionMetric(name="check",
    custom_function='def evaluate(instance): return 1.0')

# --- EVALUATE ---
result = client.evals.evaluate(dataset=dataset, metrics=[predefined])

# --- RESULTS ---
for s in result.summary_metrics:
    print(f"{s.metric_name}: mean={s.mean_score}, pass_rate={s.pass_rate}")
for case in result.eval_case_results:
    for cand in case.response_candidate_results:
        for name, r in cand.metric_results.items():
            print(f"  {name}: score={r.score}, explanation={r.explanation}")

© GoogleCloudPlatform, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 9 other files (scripts, references) in skills/quality-flywheel of GoogleCloudPlatform/vertex-ai-samples.

• SKILL.md
• EVAL.yaml
• TEST.md
• references/dataset_schema.md
• references/failure_patterns.md
• references/metric_registry.md
• references/sdk_patterns.md
• scripts/generate_eval_code.py
• scripts/parse_adk_traces.py
• scripts/validate_dataset.py

Open the folder on GitHubat commit d0aed81