Agento11y Eval Starter

Agent Observability eval starter

Help a developer who has an AI agent but no evaluation set up yet. The hard part isn't running an experiment — it's knowing what to evaluate and having cases to test against before there's any traffic. Answer both, grounded in the agent's actual code.

Always produce:

1. A ranked, justified evaluator recommendation for this agent.
2. A starter suite YAML the developer reviews and extends.

Then, depending on how runnable the agent is (Step 1):

3. For an easily-invoked agent, a runner stub (run_experiment.py, or run-experiment.ts for a TypeScript agent) that wires the suite to the SDK with one hole to fill — and optionally run it (Step 6), only with permission. For an agent that needs a harness or full runtime, point to the existing eval infra instead of a runner that can't actually call it.

This skill is language-agnostic — the reading, recommending, and YAML it produces do not depend on the agent's language. What differs is how runnable the agent is: recommendations + YAML always apply, but the runner (Step 4) and the optional run (Step 6) adapt to whether the agent has a clean function seam or needs a harness / full stack. For deeper run-side patterns (binding existing generations, cross-process verifiers) point to the per-language run skill (Python: agento11y-experiments).

Rules

• Do not create, enable, or modify evaluators, rules, or guards in any Agent Observability tenant. No control-plane writes. (Running an offline experiment only publishes that run's scores — it does not create tenant-level evaluators/rules/guards — but only do it via Step 6.)
• Do not rewrite the agent's prompt, optimize, or redeploy.
• Never run the experiment without asking first (Step 6). Never run against a target the developer did not configure — use their AGENTO11Y_ENDPOINT + credentials (Grafana Cloud); if they are not set, ask for them, do not invent an endpoint.
• Never mint, generate, or store credentials. The developer owns their Grafana Cloud ingestion token; read it from the environment or ask them to paste it — do not create one.
• Never present the generated cases as validated. They are a draft to review and extend.
• Agent Observability is a Grafana Cloud product. Do not hardcode or assume any endpoint (no localhost); the developer supplies their Cloud endpoint and token.
• If a required input is missing (entrypoint, prompt, tools), ask the developer — don't guess.

Step 1 — Read the agent

Find and read these in the target repo, and record the file path and line range of each:

1. The agent entrypoint (where the model is invoked).
2. The system prompt / instructions.
3. The tool / function definitions the agent can call.
4. One or two real user requests and what a correct answer looks like.

Every recommendation must cite one of these locations.

Also assess runnability — how hard it is to invoke this agent for one input and get its output — because it decides what Step 4 produces. Classify it as one of:

• easy — clean function seam. A single call takes an input and returns text, with no live services (a small Python/TS agent, an LLM call behind one function). The generated runner works as-is.
• in-process — needs a harness. No 3-line call, but there is an injectable seam (e.g. a Go engine whose LLM client is an interface you can fake, callable from a test binary). Runnable in-process for a smoke/behavioral eval, but not a 3-line Python run_agent.
• full-stack — needs the whole runtime. The agent needs live backends/auth/queues and is exercised over HTTP against a running stack (tools that hit real datasources, a long multi-step loop). Existing eval infra likely runs it via a dedicated harness and polling, not a function.

Signals that push toward in-process/full-stack: the agent isn't Python; tools hit real APIs/datasources; there is already a dedicated eval harness, Docker stack, or scenario/ground-truth files in the repo. If a repo already has eval scenarios with expected outputs, note them — they are better test cases than anything generated, and later steps should point to or reuse them.

Separately, note the agent's language, because the agento11y experiments SDK exists in Python, Go, and JavaScript/TypeScript (not Java or .NET yet). This is independent of runnability — a Java agent can be trivially runnable yet have no experiments SDK in its language. If the agent is Python, Go, or TypeScript, the runner is native. If it is Java or .NET, say so plainly: the runner must be Python, Go, or TypeScript (calling the agent across a process boundary), or the developer waits for experiments support in their language. Do not imply a native Java or .NET experiments API exists.

Step 2 — Choose evaluators

The SDK models an evaluator as an id plus a kind. There are two kinds:

• llm_judge — a model grades the output against a rubric (relevance, helpfulness, groundedness, tone, task completion, format, safety, and similar judgment calls).
• deterministic — code decides pass/fail (exact/substring match, JSON validity, schema or regex shape, length bounds, "not empty", a required field is present).

Pick 3–6 (not more). Map each to what you read in Step 1, and give a one-line why per pick that cites a file:line. Common mappings:

Evaluator ids are yours to choose — pick clear, stable ids. Be concrete about what "defining" each one means, because it is not an Agent Observability control-plane action here: in the offline SDK flow an evaluator is code the developer writes in the runner (Step 4). An llm_judge is a function that calls a model and returns a score; a deterministic one is a plain code check. experiments.Evaluator(evaluator_id=..., kind=...) is only the label attached to the score, not the logic. (Forking a predefined template is a separate online-eval path, not needed here.)

One exception, for tenants that already have evaluators: if the developer points at an evaluator that exists in Agent Observability already, the Python runner can bind the trial to the agent's conversation and let that evaluator grade it, instead of writing the check in the runner: trial.bind_conversation(conversation_id) then trial.evaluate("<existing-evaluator-id>"). The trial then closes as completed with no local final_score. Do not create the evaluator to make this work, and do not suggest it when the tenant has none; this skill only uses what is already there.

This skill targets the offline phase: run these evaluators as offline experiments against the suite from Step 3, before there is traffic. Do not recommend live/online evaluation to an agent with no traffic.

Once the agent ships and has traffic, the same evaluation criteria can also be applied online — as Agent Observability rules over ingested conversations, or as SDK guard hooks on the request path — but those are separate Agent Observability surfaces, configured elsewhere, and out of scope for this skill. Mention this only as a one-line "next, once you have traffic" note; do not instruct on it here.

Step 3 — Write the suite YAML

Write a suite file in the target repo (suggest evals/<agent>-starter.yaml). It must load with the SDK's TestSuite.from_yaml(...), so match this schema exactly:

• Top level: suite_id (required), plus optional name, version, description, tags, changelog, and cases (a list). version defaults to 1.0.0.
• Each case: id (required), plus optional name, description, tags, category, input, expected, weight, metadata. input and expected are free-form (a string or a mapping).

Derive cases from the agent's real task. Produce at least 6, weighted toward edge and adversarial (generated cases skew easy). Use category values happy, edge, adversarial. Keep the header comment verbatim.

Every case must actually reach the agent — test the agent, not its harness. From Step 1, note where the entrypoint parses/validates input before the model runs (e.g. a JSON parse, a CLI arg check). Do not generate cases that fail in that pre-agent layer (malformed JSON, wrong CLI flags) as if they tested the agent; if such a boundary is worth covering, label it clearly as a harness case in its notes, or leave it out.

# STARTER DRAFT — review before use. Generated from your agent code (<file:line refs>).
# NOT validated. Add your own real cases; the edge/adversarial cases need your judgment on
# expected behavior. Loads with agento11y TestSuite.from_yaml(...).
suite_id: <agent>-starter
name: <Agent> starter suite
version: 1.0.0
cases:
  - id: happy-basic-request
    category: happy
    tags: [smoke]
    input:
      prompt: "<a real request this agent is built to answer>"
    expected: "<what a good answer looks like, or a rubric note>"
  - id: edge-underspecified
    category: edge
    input:
      prompt: "<a vague / multi-part / boundary request>"
    expected: "<how the agent should handle ambiguity>"
  - id: adversarial-prompt-injection
    category: adversarial
    input:
      prompt: "<an injection / out-of-scope / data-extraction attempt>"
    expected: "<agent should refuse / stay in scope / not leak>"