Run Assert Eval

Run an ASSERT evaluation

When to use

The user wants evidence of how their agent or model actually behaves. Not for fixing the agent — this skill finds and reports failures.

This skill has two entry modes:

• Run mode — no usable results exist yet. Establish a risk source (Steps 1-2): Clarity (recommended) — an existing .clarity-protocol/ directory or a fresh discovery run driven through the Clarity MCP server (run_clarity), in-IDE — or risks the user supplies directly. Then turn each selected risk into an atomic config, run the pipeline (Steps 3-5), and report (Step 6).
• Results Q&A mode — judged artifacts already exist under artifacts/results/<suite>/<run>/ and the user asks a question about them ("what are the highlights?", "top 3 examples of the worst failure mode?", "why did case X fail?"). Skip to Step 6 and answer THAT question from the artifacts — do not re-run, and do not fall back to the full canned report unless asked.

Choosing a risk source (Clarity recommended, never required)

Every eval starts from a risk. There are two supported sources, and the user chooses — never decide for them and never block on Clarity.

Path A — Clarity discovery (recommended — present it first, but never alone). Use an existing .clarity-protocol/ or a fresh run via the Clarity MCP run_clarity tool. Clarity's value is finding failure modes the user has not thought of, along with severity and causal chains. Recommend it whenever the user is unsure what to measure, is new to the agent, or wants coverage rather than one known bug.

Path B — user-supplied risks. The user names the risk themselves, as prose or by pointing at a PRD, design doc, threat model, red-team finding, incident report, risk assessment, or test plan. This is the right path when they already know what they want measured.

Both paths answer what to test for. Neither answers how. That is the job of the research procedure in Step 3: once a risk is named, it runs a literature review of how that risk has actually been evaluated and turns the findings into the test-set design. The output is not a restatement of the topic — it is how the topic manifests:

• Timescale. Psychosocial and relational harms are typically observed across turns, not in one answer, so the literature drives scenario over prompt. max_turns itself is fixed at 6 — the timescale finding chooses the test mode, not the turn budget.
• Viewpoint. A hospital helpdesk is exercised by its primary users — patients, nurses, schedulers — not solely by one adversarial persona. Population and role become stratification dimensions when the evidence says they change the harm.
• Conditions. Pressure, severity, context position, and trajectory stage become explicit levels when sources support them.

This is the difference between a config that names a risk and a config that can actually measure it.

Whenever you need a new risk to measure, and the user has not already named one, offer the choice:

I can find a risk two ways. Clarity interviews you and surfaces failure modes you may not have considered — recommended when you know the agent but aren't sure what to measure. Or you name it directly, in your own words or by pointing me at a PRD, design doc, threat model, red-team finding, or risk assessment — best when you already know what you want measured. Either way I then research how that risk has been evaluated and build the test set from that evidence. Which do you prefer?

An existing .clarity-protocol/ changes the default, never the choice. Offer it as the recommended option ("I found an existing Clarity protocol with these risks — measure one of those, or is there a different risk you have in mind?"), then take the user's answer.

Rules that hold on both paths:

• An explicit user-supplied risk always wins. If the user names a risk — in prose, or by pointing at a document — measure that, whether or not a .clarity-protocol/ exists. Never substitute the protocol's risks for one the user just stated. If you think the protocol covers the same ground, say so and let them decide; do not decide for them.
• Never silently pick a path, and never stall the user on Clarity setup. If the Clarity MCP tools are missing and the user wants Path A, offer SETUP-CHECKLIST.md — but if they'd rather not set it up now, take Path B.
• Do not imitate Clarity's interview from your own head. This is the real prohibition: if the user picked Path A, drive the actual run_clarity tool, which returns Clarity's genuine process guide inlined. Path B is not a degraded impression of Clarity — it is a distinct, structured intake (Step 1b).
• Path B meets the same quality bar. One atomic behavior per config, researched and cited stratify dimensions, pinned systematize/judge models, an explicit sample_size. Steps 3-6 are risk-source agnostic; nothing about the config, run, or report changes.
• Offer Clarity again later. Declining once is not a permanent opt-out — after a run, it's a natural next step for finding what they didn't think to measure.
• Clarity write-backs degrade to no-ops. record_failure / record_suggestion apply only when a protocol exists. On Path B, skip them and say so once; never treat their absence as an error.

Copilot vs. the local viewer

Copilot is for answering questions and synthesis — direct answers, failure-mode clustering, cited examples, next actions — with no clicking. The bundled local viewer is for visual exploration — forest plots, baseline compare, facet grouping, and stepping through a transcript with the judge's citations highlighted. Answer in chat when the user asks "what / why / which"; hand off to the viewer (Step 7) when they want to see, read a full transcript, compare runs, or watch a live run.

Preconditions (check, don't assume)

1. ASSERT installed: assert-ai --help succeeds. If not, guide install from PyPI — not an editable install of the user's own repo:

    python -m pip install "assert-ai[phoenix]"

   The target project owns its agent framework dependencies. For repository examples, install the adjacent requirements.txt; for a customer project, use that project's existing dependency manifest. target.endpoint needs aiohttp, which ships directly with ASSERT — no separate extra to install. Use pip install -e ".[phoenix]" only when the working directory is a clone of the ASSERT repo itself; inside a customer repo it installs the wrong package.

2. Clarity MCP server available (needed only for Path A): the clarity-agent MCP tools (run_clarity, write_protocol_document, record_failure, record_suggestion, …) are callable in this session. Clarity is the risk-discovery engine — the skill drives its real MCP tools, it does not reimplement it. If the tools are missing, the server is not wired up yet: offer SETUP-CHECKLIST.md (install clarity-agent with the [mcp] extra, run clarity embed . to generate .vscode/mcp.json, reload MCP servers) and confirm the LLM provider is configured (clarity doctor — Clarity supports GitHub Copilot, Anthropic, OpenAI, Azure AI, and Gemini).

   This is not a blocker. If the tools can't be made available, or the user would rather not set them up now, say so plainly and continue on Path B (Step 1b). Never strand the user on MCP setup when they came to measure something.

3. Provider creds exist in .env. NEVER read or print .env. If a run fails with an auth error, tell the user which variable NAMES are required (AZURE_API_KEY, AZURE_API_BASE, OPENAI_API_KEY, GITHUB_TOKEN, ANTHROPIC_API_KEY, etc.) — never their values.

Steps

1. Establish the risk source

Ask which path the user wants (see "Choosing a risk source" above), then follow 1a or 1b.

• The user already named a risk (prose, PRD, design doc, threat model, incident report, test plan) → that is an explicit Path B choice. Go to 1b, even if a .clarity-protocol/ exists. Do not silently switch to the protocol's risks.
• Intent is ambiguous and a .clarity-protocol/ exists → offer it as the default and say what's in it, but still ask before selecting it: "I found an existing Clarity protocol covering X and Y — want to measure one of those, or is there a different risk you have in mind?"
• Intent is ambiguous and no protocol exists → offer the choice as written above.

1a. Clarity discovery (recommended)

Risks come from Clarity's real engine, driven through the Clarity MCP server — never by imitating Clarity's interview from your own head.

• If a .clarity-protocol/ directory already exists in the workspace, and the user has chosen Path A for this risk, use it directly as the risk source — skip straight to reading its output below. (Selecting Path A is the user's decision, made in Step 1; the protocol's presence alone does not make it.)
• Otherwise run discovery via the Clarity MCP tools:
  1. Call run_clarity. It returns Clarity's real process guide inlined as text.
  2. Follow that guide to ask the user the clarifying questions in chat — this is Clarity's genuine multi-perspective flow, surfaced through you as the host agent (Copilot agent mode supports MCP tools, so drive the loop yourself rather than expecting a separate chat UI).
  3. Persist what you learn with write_protocol_document and record_failure. Continue until the failure-analysis process has written .clarity-protocol/failures/failures.md.

Read Clarity's output to enumerate risks:

• .clarity-protocol/failures/failures.md — the failure modes, causal chains, and management plans. Each distinct failure mode is one candidate ASSERT behavior.
• .clarity-protocol/summary.md, goal/requirements.md, solution/architecture.md — target/context for the eval's context field.

For the full measurement path — parse → triage → one atomic config per selected failure → sequential runs → report → close the loop → curate the example — follow workflows/measure-clarity-failures.md. Use the intake parser (clarity_intake.py) to convert failures.md into candidate behaviors with severity→priority mapping and variant-derived stratify dimensions.

Before a fresh discovery run, check the preservation gate. .clarity-protocol/ is gitignored, single-domain scratch; run_clarity overwrites it, destroying the prior domain's failures/, goal/, and solution/ with no git recovery. If a protocol from another domain is present, STOP and let the user export it to a user-owned location or explicitly discard it. Never commit the raw discovery workspace into examples/.

Clarity records severity/management-plan signal (the parser maps Critical→P1, High→P2, Medium→P3, ranges→max). Order and annotate by what Clarity actually captured; do not fabricate priorities.

1b. User-supplied risks

The user already knows what to measure. Your job is to turn their input into the same candidate-behavior shape clarity_intake.py produces on Path A — {name, description, severity, priority, source_doc, candidate_dimensions, multi_behavior, suggested_splits} — so Steps 2-6 are identical either way.

1. Take the input as given. If they point at a PRD, design doc, threat model, incident report, or test plan, read it and extract candidate risks from it. Quote what you extracted so they can correct you. If they describe it in prose, start from their words — do not paraphrase the risk into something broader.
2. Ask only what the config actually needs, in one batch, and skip anything already answered by their input or the repo:
   • What does the agent do, and who uses it? → context
   • What must it never do? → behavior.name + behavior.description
   • What is it explicitly allowed to do that looks similar? → the permissible boundary. Do not skip this: without it the run has no permissible behaviors to attribute violations to, so the permissibility split cannot be computed and collapses to a single uninterpretable rate.
   • How would a user trigger this — innocently, and deliberately? → a candidate elicitation_variant stratify dimension — a research seed for Step 3, not the final set
   • How bad is it when it happens? → severity → priority
3. Enforce atomicity now. If their description bundles several independently testable behaviors ("it leaks data and hallucinates prices"), say so and propose the split — one config each. This is the multi_behavior / suggested_splits check, applied by hand.
4. Play it back for confirmation as an explicit candidate list before generating anything, exactly as triage does on Path A.

Set source_doc to the file you read, or user-described when it came from chat. Record severity as the user rated it; do not invent a priority they didn't give.

For the full measurement path — triage → one atomic config per selected risk → sequential runs → report → curate the example — follow workflows/measure-clarity-failures.md, the same workflow Path A uses. Skip its Step 1 (Parse): there is no failures.md to parse, so join at Step 2 with the candidate list you just built. Skip its Step 8 (close the loop in Clarity) too, unless a .clarity-protocol/ exists.

Then continue to Step 2. Everything downstream is unchanged.

2. Triage — choose which risks to measure now

Clarity intentionally over-produces (whole-lifecycle threat modeling). Do NOT auto-generate an eval for every failure mode. Surface the enumerated list (ordered by severity signal) and ask the user which to measure now (e.g. "top-severity only?", or named picks). Carry only the selected risks forward.

On Path B the list is usually short and already chosen — still play it back and confirm scope before generating configs, rather than assuming every risk they mentioned should be measured in this pass.

3. Turn each selected risk into an atomic config

ASSERT performs best with one atomic behavior per eval. Never bundle multiple risks into one config — bundling makes the reported impermissible behavior violated rate a fuzzy logical-OR and hides per-behavior signal.

• 1 selected risk → generate one config and run once.
• N selected risks → generate N configs and run them sequentially, one per behavior.

Configs are researched, cited, and user-approved — not scaffolded and hoped for. Follow workflows/research-eval-dimensions.md for each selected risk. That workflow owns the whole of config generation; do not hand-roll a config here and do not skip its gates.

Its purpose is narrow and worth stating plainly: the risk already has a name by the time you arrive here. What the research supplies is how that risk has been evaluated — the timescale it becomes observable on, whose viewpoint exercises it, and which conditions change it — expressed as stratify dimensions, behavior_category_count, judge dimensions, and the scenario vs prompt test mode. It does not re-open what to measure.

Collect one input before entering it, and never silently default it:

What that workflow does, in order:

1. Reuse a repo spec first — assert-ai library list / show <name>; prefer behavior.preset or a copy-in spec from examples/behavior_specs/ over reinventing a description. This is also what settles the harm's stable slug.
2. Isolation preflight (generation-isolation-workflow.md) — once the slug is stable, detects prior generations for it by path only, and asks before using a new dated directory. It never reads a prior generated YAML.
3. Research the dimension model — classify the harm's observability, build a dimension ledger, and gate each dimension on at least two independent authoritative sources (or one plus the repo spec). Behavior categories, stratify dimensions, and judge dimensions are researched as three separate namespaces.
4. Run N passes and deduplicate (iterative-dimension-workflow.md) — N complete passes, then semantic deduplication within each namespace.
5. Review and approve — a compact table per namespace, and an explicit user approval. Silence is not approval, and the pre-write gate enforces this mechanically.
6. Write the cited config to examples/<domain>/<risk>[_YYYY-MM-DD]/eval_config.yaml, with inline # sources: citations and a consolidated # References block.

Outputs land at examples/<domain>/<risk>[_YYYY-MM-DD]/eval_config.yaml — one directory per generation, never overwritten. Prefix the eval suite name with a domain slug (<domain>-<risk>) so artifacts/results/<suite>/ and artifacts/acs/<suite>/ do not collide across domains.

Two things that workflow will ask you to decide, and that matter downstream:

• behavior_category_count is 25 — the standard count, and ASSERT's own default (DEFAULT_BEHAVIOR_CATEGORY_COUNT). Research shapes which categories are generated, not how many.
• sample_size is a question for the user, but it has a hard floor: ≥ behavior_category_count (so ≥25). Below the category count some behavior categories receive zero cases and are silently unmeasured. Each rate is also violations / sample_size, so even at the 25 floor one flipped case moves the number 4 percentage points, and the swing grows as the sample shrinks. The floor protects coverage, not precision — prefer 50+ when the expected delta is small. Write the chosen value into the config with an inline review comment (# min for behavior-category coverage -- user should review; 50+ tightens the signal) so it reads as a floor the user still owns, not a settled default.
• max_turns is fixed at 6 — ASSERT's default (DEFAULT_TESTER_MAX_TURNS) and the config template's value. The research does not move it, and it must be identical in the baseline and governed configs or the "only ACS differs" comparison breaks. A genuinely single-turn harm is expressed by writing prompt test cases, not by lowering the turn budget — max_turns is read only for scenario cases.

Judge dimensions are authored from the research, and are added on top of the built-ins — but never reuse a built-in name. policy_violation and overrefusal are BUILT_IN_DIMENSIONS (assert_ai/core/judge.py) and are always judged unless explicitly disabled. Config dimensions merge over the built-ins by name into the same dict, so a researched dimension called policy_violation silently replaces the built-in rubric — no warning, no error.

What shadowing does and does not affect. It does not move the headline pair. Both not_permissible_policy_violation_rate and permissible_policy_violation_rate are recomputed in assert_ai/results.py from the judge's per-behavior node_judgments — each relevant node's violated flag, bucketed by that behavior category's permissible value — so they never read either dimension's verdict. What shadowing does change is the dimension verdict persisted in the run JSON, and DEFAULT_COMPARE_METRIC (assert_ai/cli.py), which is still policy_violation. That is a comparability and stored-artifact problem, not a corrupted headline.

The same applies to judge presets, which expand into that same merged list:

• Do not set judge.preset: safety-core. Its only two dimensions are named exactly policy_violation and overrefusal, so it replaces both built-in rubrics — and buys nothing, because the engine treats that pair as superseded: _SUPERSEDED_DISPLAY_METRICS (assert_ai/cli.py) hides both from CLI detail surfaces whenever the permissibility split is available, which for this skill's configs is always. You would pay judge tokens per dimension per row for two numbers that are never shown and never reported.
• safety-extended is safe and recommended for nuanced harms — it defines harm_actionability and pii_leakage, which collide with nothing and are displayed.

The post-write gate rejects both the inline and the preset form.

If live source retrieval is unavailable, say so and stop at the ledger. The evidence gate cannot be met without it, and a config with remembered or invented citations is worse than no config. assert-ai init --describe-file … remains available as an explicitly unvalidated scaffold — never for a measurement you intend to report or govern against.

After generation, show the user the resolved behavior.description, context, pipeline.judge settings, the systematize / judge models, and the reference list. Confirm before running.

4. Identify the target shape

Help the user set the right target in the config:

• Framework agent (LangGraph, CrewAI, etc.) with a Python entry function: use target.callable WITH target.trace so the judge can cite tool calls and routing.
• Hosted model with a system prompt and optional tools: use target.model and target.tools.
• Pre-collected traces (no live inference needed): use assert-ai judge-traces --traces <path> --config <path>; do not add a --trace flag to assert-ai run.
• Black-box HTTP endpoint you cannot import as Python: use target.endpoint — the runtime POSTs {"message": ..., "history": [...]} and reads {"response": ...}, so no wrapper code is needed (requires aiohttp). Only write a thin target.callable shim if the service's request/response shape differs. Either way the judge sees only final text, so this is a fallback, not the recommended path.

The callable contract — verify before the first run

target.callable takes a module.path:function reference. The full signature and return-type contract lives in docs/targets/callable.md. Two behaviors that doc does not cover can silently corrupt a run:

• history is detected by parameter name, not position. ASSERT introspects the signature and enables multi-turn only when a parameter is literally named history. Name it messages, conversation, or chat_history and every scenario silently degrades to single-turn — the run completes, the viewer renders, and the numbers are wrong with no warning. Confirm the name before trusting any multi-turn baseline, and therefore any ACS delta measured against it.
• Module resolution has a four-step fallback: sys.path → the config's own directory → the current working directory → direct file load. An agent.py sitting beside the YAML config resolves even when the CLI is invoked from the repo root — but a same-named module earlier on sys.path wins, so prefer a domain-unique module name over a bare agent.

Why target.trace is not optional

Tracing decides how much of the agent the judge can actually see. Per the observability matrix in docs/targets/callable.md ("What the judge sees, by integration path"):

So without traces a tool-misuse or wrong-routing failure is largely invisible to scoring — which is why this skill mandates target.callable with target.trace. You rarely hand-write spans: ASSERT ships OTel auto-instrumentation for 33 frameworks (LangChain / LangGraph, CrewAI, OpenAI Agents SDK, DSPy, LlamaIndex, AutoGen, MAF, Pydantic AI, …) as a single helper call at the top of the callable module — see docs/targets/callable.md ("Recommended: OTel-traced agent (33 frameworks)").