LLM Eval Harness

Dimension 3 — Protocol compliance (Anthropic thinking block: generation AND history-replay)

uv run python scripts/protocol_probe.py \
  --url <…/v1/messages> --model <model> --key-env <ENV> --check all --repeat 10 --output /tmp/proto.json

• Only relevant for endpoints claiming Anthropic /v1/messages compatibility. --check all (default) runs BOTH sub-checks — they test different code paths and a vendor can pass one while hard-failing the other:
  • generation: does thinking: {type: enabled} actually produce thinking_delta / signature_delta SSE events when you request it? (the original check)
  • history-replay: does the endpoint ACCEPT a type: "thinking" block that's already sitting in a prior assistant turn, when that history is replayed back on a later turn — exactly what Claude Code and every other agentic client does on every continuation? (--check history-replay to run just this one)
• Real incident (2026-07-21) that motivated the history-replay check: a Kimi/Moonshot model via a China reseller passed generation fine (emits thinking correctly when asked) but hard-rejected history-replay — 400 "invalid part type: thinking" the instant a prior thinking block came back as input, killing the session on every subsequent turn. Passing generation told us nothing about this; they're orthogonal failure modes (response generation vs. request validation).
• Don't conclude "vendor/reseller X is broken" from one cross-axis comparison — and don't stop at the first single-axis comparison that confirms a vendor-documented parameter either. This took three rounds to get right, kept in disciplines §§17-19 as the canonical cautionary tale: round 1 compared a different model AND a different reseller at once and (wrongly) blamed the reseller. Round 2 fixed that — same reseller, only the model varied — and the result matched a documented, named vendor parameter (Moonshot's own preserve_thinking), which looked like confirmation. Round 2 was STILL wrong: the probe never left that one reseller, so it couldn't see that the vendor's own direct/native endpoint handled the "rejected" model fine — real production traffic proved it. Read §19 before treating any single-reseller-confirmed result as final; check the vendor's own endpoint or real traffic for the actual channel in question before writing an "X doesn't support thinking" conclusion into anything.
• Compliance is often probabilistic, not binary, for BOTH checks. One real vendor honored the thinking block on only ~13% of generation requests (vs 100% for two competitors). That's why --repeat defaults to 10; generation's verdict has three states (fully-implemented, intermittent (k/N), not-implemented), history-replay's has its own three-plus states (accepts-thinking-in-history, rejects-thinking-in-history, inconsistent, or inconclusive when errors look unrelated to thinking at all). Never conclude from a single sample.
• It forces Connection: close per request so a load balancer can't pin all samples to one replica and hide the real distribution (a real probe saw 0/10 with keep-alive vs 17/90 with close on the same endpoint).

Dimension 4 — Quality / use-case regression (blind judge)

This is two halves on purpose: collect, then judge independently.

Step 1 — collect the model's answers to your use-case library:

uv run --with openai python scripts/usecase_runner.py \
  --base-url <…/v1> --model <model> --key-env <ENV> \
  --usecases ~/.llm-eval/usecases.json --output-dir ~/.llm-eval/runs/<model>

Step 2 — judge with independent blind judges (orchestrate inline — do NOT let the model grade itself). For each answer in the run directory, spawn 3 independent Task agents (or fewer for a quick pass). Each judge gets ONLY: the prompt, the answer, and the case's rubric — and is explicitly told it is judging in isolation, with no knowledge of other judges' scores or any prior evaluation (this prevents anchoring). Then aggregate:

• A case passes only on majority agreement among judges.
• Compute precision per category (using each case's tags): a category where judges systematically disagree with the rubric is a real weakness — on one real eval, a whole category scored 12.5% precision and exposed a systematic misclassification that a single grader would have missed.
• Count only explicit judgments. A judge that didn't return a verdict is not a pass — silence ≠ consent. This guards against automation bias.

For the rubric-scoring mechanics (LLM-as-judge thresholds, llm-rubric), you can also compose with the promptfoo-evaluation skill — point its providers at the same endpoint. This harness's blind-judge method and promptfoo's rubric assertions are complementary: use promptfoo for fast per-case pass/fail gating, blind judges for precision on a category you suspect is weak. Full method: references/quality_blind_judge.md.

Use-case library

Keep it OUTSIDE this bundle (e.g. ~/.llm-eval/usecases.json) so it survives skill updates and never lands in a public repo. It's a plain JSON list — version it in a private repo to accumulate a regression suite over time:

[
  {"id": "refund-window", "prompt": "A customer asks for a refund 20 days after purchase. Reply as support.",
   "rubric": "1.0 if it correctly cites the 30-day refund window; 0.0 if it refuses or invents a different window.",
   "tags": ["support", "policy"]},
  {"id": "lru-cache", "prompt": "Implement an LRU cache in Python with O(1) get/put.",
   "rubric": "1.0 if get and put are both O(1) via dict + doubly linked list and the self-test passes.",
   "tags": ["code"]}
]

assets/example_usecases.json is a starter you can copy. Only id and prompt are required; rubric, expected, and tags make judging sharper.

Running a full evaluation

When the user says "evaluate / benchmark this model", the typical flow is:

1. Identify the shape — OpenAI-compatible (/v1/chat/completions) or Anthropic-Messages (/v1/messages)? Hit GET /v1/models or read the vendor docs; don't assume. This decides which probes apply (protocol probe is Anthropic-only). Remember the listing is incomplete evidence either way (disciplines §9).
2. For a new endpoint, availability and fidelity come first — speed numbers for a model whose system prompt never arrives are answering the wrong question.
3. Run the dimensions the user cares about — speed and concurrency for "is it fast/stable", add protocol for an Anthropic vendor, add quality when they have a use-case suite. Write each probe's --output JSON to a run directory.
4. Report honestly, separating measured from inferred. Lead with the headline the user asked about (e.g. "sustained decode ceiling exceeds the vendor's claimed tok/s, while real-task throughput runs lower"). If a number looks impossible (e.g. throughput far above the vendor claim, or a single-sample protocol verdict), treat it as a measurement artifact to investigate, not a result — that skepticism is the whole point of this harness. Rates on routed gateways are additionally time-varying: re-sample another window before freezing any number into a document (disciplines §12).
5. Comparing two models? Run the identical probes against each with the same flags, and put the two JSON outputs side by side. Keep the test conditions identical (same concurrency levels, same use cases) or the comparison is meaningless.
6. Found a vendor bug worth reporting? Don't paste raw probe output at their support channel — build the evidence package per references/vendor_evidence_protocol.md (self-audit first, observation wording, request-id table, pre-registered thresholds, adversarial counter-review).

For tests that will run repeatedly against a live system — deployment gates, resident canaries, fault-injection mocks, and the monitoring statistics that lie — see references/production_testing_patterns.md.

Next step

After a run, offer the natural follow-ups:

Evaluation complete for <model>.

Options:
A) Render an HTML dashboard of the results — compose with a visualization skill (Recommended if sharing)
B) Compare against another model — same probes, side-by-side
C) Add the failing cases to ~/.llm-eval/usecases.json as a permanent regression guard
D) Done — the numbers answer the question