Agentsop Output Format By Model

Case D — "Reviewer says 'use structured output everywhere for safety'"

Trigger: Eng-org-wide push to add response_format=json_schema to every LM call.

Constraints:

• The org has been bitten by json.loads failures.
• Some calls produce code; some produce prose; some produce typed objects.
• The reviewer is conflating "schema-valid" with "quality."

Decision steps:

1. Surface the empirical finding: JSON validity ≠ content quality. Cite Aider's 20%→61% (unified diff vs JSON) [aider.chat/2024/08/14/code-in-json.html] and "Let Me Speak Freely?" [arxiv.org/abs/2408.02442].
2. Audit each call by content type using §3 Step 1 table.
3. For code-emitting calls: keep markdown; add a defensive parser (regex extract from fenced block). The robustness cost is small; the quality gain is large.
4. For typed-object calls: yes, add json_schema. This is where structured output earns its keep.
5. Document the policy: "Structured output is mandatory for typed-data extraction; forbidden for code edits and multi-step reasoning; optional for everything else."

Outcome: Reject the blanket policy; replace it with a content-type-driven one. This is the inverse of the "JSON everywhere" reflex.

6. 反模式与边界 (Anti-patterns & boundaries)

Anti-patterns

1. "JSON everywhere" reflex. Structured ≠ better. Code, multi-step reasoning, long-form prose all degrade inside JSON envelopes — Aider measured 3× on code; arXiv 2408.02442 measured 10–15% on reasoning. Use JSON where the content itself is structured.
2. "Markdown only when a human reads it" reflex. Markdown is also an excellent LM-to-LM wire format and often the highest-quality format for the producing model. DSPy's default chat adaptor uses field-marked markdown for many signatures even with no human in the loop. [dspy.ai/learn/programming/signatures/]
3. Wrapping code in JSON tool calls without budgeting quality loss. If your harness requires tool calls, fine — but acknowledge the tax and put the code in a single string field. Don't sub-structure it into {language, lines: [...], imports: [...]}. Every level of nesting compounds the escape tax.
4. Forcing reasoning into a reasoning JSON field. Models compress and stiffen when reasoning is inside JSON — they treat it as a label rather than as thinking. Prefer free markdown reasoning followed by a structured tail.
5. Using JSON schema as a substitute for prompt engineering. Schema enforces shape. It does nothing for correctness of contents. Write the prompt as if there were no schema, then add the schema for safety.
6. Asking for typed data as freeform prose, then post-hoc parsing with regex. The inverse mistake. If the consumer is a parser and the fields are scalars, use a schema — the model is happy to oblige.
7. Mixing code and prose in one JSON document. Use two-zone output (prose followed by fenced block) or two passes.
8. Ignoring per-model edit-format defaults. Aider's defaults exist because the same edit format performs differently across models. Don't override without an A/B. [aider.chat/docs/more/edit-formats.html]
9. Treating tool-use schemas as documentation-free zones. Anthropic's tool-writing guide emphasizes high-signal field names (name, file_type) over technical identifiers (uuid, mime_type). The model reads your schema as part of its prompt. [anthropic.com/engineering/writing-tools-for-agents]
10. "The task is fixing JSON, so format doesn't matter." Counter-edge case: when the content itself is JSON (e.g. "fix this malformed JSON"), output it as a fenced ```json block in markdown — not as a JSON tool-call wrapping JSON-as-string. Even here, the rule holds: code-shaped content goes in a plain string surface, not nested escapes. [arxiv.org/html/2510.04717v1 — JSON Whisperer]

Boundaries (when format choice doesn't matter much)

• The model is overwhelmingly capable for the task (GPT-5-class on entity extraction with 4 scalar fields). The format tax is dwarfed by headroom.
• The output is one token / one number / yes-no. Any format works; pick by parser convenience.
• The pipeline is one-shot exploration with no downstream consumer.
• You're forced into a format by an external contract (e.g. publishing to a webhook expecting JSON). Then your job is mitigation (single string fields for code-shaped payloads), not format choice.

7. 跨框架对照 (Ecosystem cross-reference)

How major frameworks implement format selection. Use this to translate the principles into the stack you're already in.

Aider edit formats [aider.chat/docs/more/edit-formats.html]

No JSON-wrapped edit format exists — it was tested and rejected. [aider.chat/2024/08/14/code-in-json.html]

DSPy adaptors [dspy.ai/learn/programming/signatures/]

DSPy's Signature → Module pipeline ships with multiple adaptors that materialize the same logical I/O contract into different wire formats:

• ChatAdapter (default): field-marked markdown with [[ ## field_name ## ]] headers. Used even for typed outputs because models reliably emit it.
• JSONAdapter: schema-enforced JSON. Used when the consumer must parse strictly (e.g., feeding another typed pipeline).
• TwoStepAdapter: free-form generation, then a second cheaper call reformats into JSON. Direct application of the "Let Me Speak Freely?" finding — generate freely, format separately. [arxiv.org/abs/2408.02442]

DSPy's recommendation: stay on ChatAdapter unless a downstream consumer demands JSON. The framework itself encodes the principle of this skill.

Outlines / Guidance / LMQL (grammar libs)

• Token-level grammar enforcement (regex, CFG, JSON schema). Guarantees validity, not quality of content.
• Best used for: forced enums, numeric ranges, valid JSON shells around scalar fields.
• Worst used for: forcing code or long prose through a JSON grammar — you'll get valid JSON containing degraded code.
• Pair with this skill: choose format using §3, then use Outlines to enforce whichever format you chose.

OpenAI tool_use / function calling

• Tool selection: structured by definition. Use it.
• Tool arguments: keep scalar where possible; wrap code-shaped payloads in a single string field.
• response_format=json_schema for typed extraction: yes. For code generation: avoid; if forced, single string field with code as the type.
• "Strict mode" (strict: true) guarantees schema validity but does not fix content degradation — Aider tested this explicitly. [aider.chat/2024/08/14/code-in-json.html]

Anthropic tool_use / text-editor tool [docs.anthropic.com text-editor-tool]

• Built-in text_editor tool: commands view, str_replace, create, insert, undo_edit. str_replace takes old_str and new_str as single string fields — explicitly avoiding the JSON-nests-code anti-pattern. This is the principle of this skill made into a first-party API.
• Tool-writing guidance: high-signal scalar field names; avoid low-level identifiers in tool schemas. [anthropic.com/engineering/writing-tools-for-agents]

Cross-framework summary

Layer                  | Format mechanism
-----------------------|-----------------------------------
Generation grammar     | Outlines / Guidance — enforce
Per-task adaptor       | DSPy ChatAdapter vs JSONAdapter
Per-model edit format  | Aider's edit-format table
Tool envelope          | OpenAI / Anthropic tool_use
Wire serialization     | markdown vs JSON vs YAML

Every layer can independently make a wrong format choice. This skill operates at the per-task layer: decide first what format the content wants, then let each lower layer enforce it.

Quick reference card

┌──────────────────────────────────────────────────────────────────────┐
│                        FORMAT DECISION CARD                          │
├──────────────────────────────────────────────────────────────────────┤
│ Code edits          → text + diff (per-model: udiff/SEARCH/whole)    │
│                       [Aider: 20% → 61% on GPT-4 Turbo]              │
│ Code generation     → markdown fenced block                          │
│ SQL / regex / shell → string field inside tool_use, NOT sub-JSON     │
│ Entity extraction   → JSON / Pydantic / typed OutputField            │
│ Classification      → single token or JSON {label}                   │
│ Action selection    → JSON tool_use                                  │
│ Reasoning + answer  → markdown CoT + fenced final answer             │
│                       [Format-restricted reasoning: -10 to -15%]     │
│ Prose / explanation → markdown                                       │
│ Mixed content       → two passes OR two-zone output                  │
├──────────────────────────────────────────────────────────────────────┤
│ NEVER:                                                               │
│  • Nest code inside JSON sub-structure                               │
│  • Force CoT reasoning into a JSON "reasoning" field                 │
│  • Use schema as substitute for prompt engineering                   │
│  • Assume strict-mode JSON fixes content quality                     │
└──────────────────────────────────────────────────────────────────────┘

引用源 (Citations)

Primary empirical anchors:

• Aider code-in-JSON benchmark: [aider.chat/2024/08/14/code-in-json.html] — "All of the models did worse on the benchmark when asked to return code in a structured JSON response."
• Aider unified diff vs JSON: [aider.chat/2023/12/21/unified-diffs.html] — GPT-4 Turbo refactor 20% → 61% with udiff; "lazy comments" cut to 1/3.
• Aider edit-format docs: [aider.chat/docs/more/edit-formats.html]
• Aider architect mode: [aider.chat/2024/09/26/architect.html] — o1-preview alone 79.7% → o1-preview + Sonnet editor 82.7%.
• Aider Sonnet-not-lazy: [aider.chat/2024/07/01/sonnet-not-lazy.html]

Academic generalization:

• "Let Me Speak Freely?" — arXiv 2408.02442 [arxiv.org/abs/2408.02442] — Format restrictions cause 10–15% drop on reasoning tasks.
• StructEval — arXiv 2505.20139 [arxiv.org/html/2505.20139v1] — Even o1-mini 75.58 avg on structural-output generation.
• JSON Whisperer (JSON-editing) — arXiv 2510.04717 [arxiv.org/html/2510.04717v1]
• To Diff or Not to Diff (2025 edit-format study) — [arxiv.org/html/2604.27296]

Framework / API docs:

• DSPy Signatures + adaptors: [dspy.ai/learn/programming/signatures/]
• DSPy FAQ on structure vs free-form: [dspy.ai/faqs/]
• Anthropic text-editor tool: [docs.anthropic.com/en/docs/agents-and-tools/tool-use/text-editor-tool]
• Anthropic writing-tools-for-agents: [anthropic.com/engineering/writing-tools-for-agents]
• OpenAI structured outputs: [platform.openai.com/docs/guides/structured-outputs]

Companion skills in this collection:

• aider-sop-skill/SKILL.md — full edit-format treatment in coder-agent context.
• dspy-sop-skill/SKILL.md — adaptor selection within compiled programs.