Commerce Evals

Commerce agent evals

The repo ships no eval harness; the suite is yours, because a case only means something against your catalog, orders, and fixtures. Paths below are in the reference repo: commerce_common/ is commerce-common/commerce_common/. Gate behavior that needs no model (provenance, caps, guardrails, approval) is unit-tested with FakeClient in commerce_common/testing.py; evals cover what the model decides.

The case shape

This block is the one home of the case shape and the scorer names; /author-commerce-evals refers to it.

{
  "id": "<flow>-<nnn>-<behavior>",
  "priority": "critical | high | medium | low",   "difficulty": "easy | medium | hard",   "tags": ["..."],
  "skip": "<reason, when a case cannot run yet>",
  "state": {"seen_products": ["..."], "cart": [...], "memory": [...], "staged_changes": [...]},
  "turns": ["<the customer's or operator's message>", "..."],
  "expected": {
    "calls_tool": ["..."],            "calls_one_of": ["..."],          "never_calls": ["..."],
    "first_tool": "...",              "first_tool_not": "...",
    "ui_components": ["..."],         "no_ui": true,
    "cart_contains": ["..."],         "cart_item_count": 0,             "cart_not_contains": ["..."],
    "staged_change_kinds": ["..."],   "no_applied_changes": true,
    "memory_contains": ["..."],       "memory_not_contains": ["..."],
    "skill_loaded": "...",            "skill_not_loaded": "...",        "no_skill_load": true,
    "reply_includes": ["..."],        "reply_omits": ["..."],           "max_tool_calls": 0,
    "rubric": "PASS if <condition>. FAIL if <condition>."
  },
  "notes": "<what the case pins and the fixture fact that decides it>"
}

state is the precondition; turns is one message unless the behavior under test is carrying state across turns; expected holds only the keys the case is about. Ids in state and expected are real ids from your fixtures. priority and difficulty let a report say which failures matter; a case that cannot run yet carries skip with its reason rather than being deleted.

Authoring rules

• Preconditions go in injected state (the products already seen, the cart, the memory facts, the staged queue), which the runner loads into the session state and the memory store before the turn. Earlier turns are for state the behavior itself carries, and for nothing else.
• Every positive has a negative: for each case that asserts a component, a skill load, a gate, a disclosure, or a memory write, a case in the same niche asserts its absence. A refusal case has a should-serve counterpart.
• Memory is three cases: a remark worth keeping is written; an identifier or excluded content is refused (nothing stored, no error to the person); a stored fact changes the next session's pick.
• Grade the final tool arguments and the state they produced: the ids on the last presentation call, the fields on the staged change, the cart and the memory store after the turn. The reply's wording is graded only for strings that must or must not appear. Which route the agent took (skill_loaded, one named component, never_calls on a presentation tool) is asserted only where the route is the behavior (a grounding read first, a write that must never happen); elsewhere calls_one_of names the acceptable set. When a live run takes a route the case did not expect and the answer was right, widen the case to the acceptable set; do not re-pin it to the route observed.
• A rubric is one PASS condition and one FAIL condition that no response satisfies both of; it names the fixture fact that decides it (the updated delivery date, the price today); variants you accept are written into it; it says nothing about tone, length, or the order components appear in.
• A turn that mentions health, a one-off errand, or hostile content asserts the memory end-state (memory_not_contains, or never_calls on save_memory). A case where a stored fact should change the pick uses a query whose results contain both the item the fact favors and the one it rules out; run the search before writing the case.
• max_tool_calls is set from what a well-behaved agent needs; a multi-item request fans out several searches in one round, and the present_suggestions call that ends the turn is not counted.