Agentsop Code Execution Decision

Case B — Over-coding a judgment task wastes a sandbox round-trip (over-coding)

Trigger: A support-triage agent step: "Read this customer message and decide whether the tone is hostile, neutral, or warm." An over-eager engineer wires it through ProgramOfThought because "code is more reliable."

Constraints:

• "Tone" has no single verifiable answer — it is a judgment.
• The sandbox round-trip adds latency + cost.
• Forcing it into code produces something like if "!!!" in msg: tone = "hostile" — a brittle rule that encodes a subjective rubric as if it were a formula, and is worse than the model's native judgment.

Decision steps:

1. Gate (§3 Step 1): "Single verifiable answer a program could check?" No — tone is judgment. → reason in prose. Stop.
2. Invoke the over-coding veto (op #7). There is no deterministic core to delegate to an oracle. Code adds no determinism — it just relocates the same judgment into worse, hard-coded heuristics.
3. Cost it out. The sandbox call adds a full round-trip (and a code-gen + parse step) for zero precision gain. This is pure waste — the loud, cheap failure mode.
4. Keep it as ChainOfThought (or Predict). The LM is the right engine for judgment.
5. Counter-check: if a later spec adds "and count how many exclamation marks" — that sub-step is computable; decompose (op #2) and route the count to code, the tone to prose.

Outcome: No sandbox call. The judgment stays where judgment belongs. Over-coding is a real and common anti-pattern: not everything benefits from an interpreter, only things with a verifiable deterministic core.

Extractable operation: No verifiable answer → no code. Veto the sandbox round-trip (op #7).

Case C — Mixed step: summary + total (decompose)

Trigger: "Summarize this quarter's sales narrative and give me the exact total revenue."

Constraints: One sentence contains both a judgment (summary) and a computation (total).

Decision steps:

1. Gate → mixed. Decompose (op #2).
2. Route the total to ProgramOfThought: code sums the figures, interpreter verifies.
3. Route the summary to ChainOfThought: prose synthesis, no oracle exists.
4. Feed the code result back into the prose step so the narrative cites the exact, verified total.

Outcome: Each sub-step uses its correct engine. This is the execution-decision analogue of the mixed-content two-pass pattern in [[agentsop-output-format-by-model]] §5 Case B.

Extractable operation: One task ≠ one strategy. Gate per step, decompose mixed steps.

6. 反模式与边界 (Anti-patterns & boundaries)

Anti-patterns

1. Code for everything ("code is always more reliable"). False. Code is more reliable only where a deterministic, verifiable answer exists. For judgment tasks, code just hard-codes a subjective rubric and adds a wasted round-trip (Case B). The interpreter is an oracle only for computable questions.
2. Reasoning for arithmetic ("the model can just add it"). The headline failure. LMs are unreliable calculators; multi-step prose arithmetic hallucinates plausible wrong numbers, silently (Case A). Any non-trivial computation → code [arXiv 2211.12588].
3. Emitting code but never feeding the result back. PoT is "code computes, LM contextualizes." If the value never re-enters the LM turn, you have a dangling computation the agent can't use or narrate (op #4).
4. Unbounded error-retry loops. On a traceback, regenerate-and-rerun is correct — but cap it (max_iters ≈ 3) and fall back to prose or surface the failure. Infinite code-repair loops burn cost. See [[agentsop-test-fix-loop]].
5. Nesting generated code inside JSON sub-structure. The execution decision says emit code; [[agentsop-output-format-by-model]] says emit it as a fenced block / single string, never as nested JSON — code-in-JSON degrades the code (Aider 61%→20%). Pass both gates.
6. Treating LangChain PythonREPLTool as a safe sandbox. It runs in-process, unsandboxed. Fine for trusted self-authored code; hostile to untrusted input. Use a real sandbox (OpenAI Code Interpreter container, Anthropic code-exec VM, DSPy's interpreter) when inputs are untrusted.
7. Over-decomposing trivial computation. A 3-item count or single-digit add does not need a sandbox round-trip; the tax exceeds the benefit. The gate is for non-trivial computation.

Boundaries (when this gate doesn't fire)

• No sandbox available and determinism is soft. Sometimes you must reason in prose because there is no interpreter; accept the precision risk and flag it.
• The model is overwhelmingly capable for a tiny computation. Frontier models reliably handle small arithmetic; the round-trip isn't worth it below a complexity threshold.
• The task is one-shot exploration with no downstream consumer of the precise value.
• Latency-critical paths where a sandbox round-trip violates a hard SLA and an approximate answer is acceptable — a deliberate, documented tradeoff, not a default.

7. 跨框架对照 (Ecosystem cross-reference)

How major frameworks expose the emit-code-vs-reason mechanism. This skill operates at the decision layer; each framework supplies the mechanism.

DSPy ProgramOfThought [dspy.ai/learn/programming/modules/]

The canonical declarative version. A signature compiled with dspy.ProgramOfThought(Sig) makes the LM emit Python, runs it in an interpreter sandbox, and feeds the result back — with bounded retry on error (max_iters). The sibling dspy skill lists ChainOfThought vs ProgramOfThought as module choices but does not give the decision rubric; this overlay is that rubric. Use ProgramOfThought exactly when §3 Step 1 returns "computable."

OpenAI Code Interpreter / Assistants code tool

A managed container that the model can write Python into and execute, with files and state persisted across turns. Heavier and stateful — good for data-analysis sessions (load CSV, compute, plot). Same gate applies: route computable steps in, keep judgment in chat.

Anthropic code-execution tool

A sandboxed VM exposed as a tool; the model emits code, it runs, results return to the conversation. First-party, sandboxed — safe for untrusted inputs. The execution-decision gate maps directly: offer the tool, but the model/agent should only invoke it for computable steps.

LangChain PythonREPLTool

A tool wrapping a Python REPL. Runs in-process and is unsandboxed — powerful and dangerous. Use only for trusted, self-authored computation; never expose it to untrusted input without an external sandbox. The decision rubric is identical; the safety profile is worst-in-class.

Cross-framework summary

Framework            | Mechanism                | Sandbox     | Result-back
---------------------|--------------------------|-------------|------------
DSPy PoT             | ProgramOfThought module  | Python intp | automatic (max_iters)
OpenAI Code Interp.  | Assistants code tool     | container   | persisted state
Anthropic code-exec  | code-execution tool      | VM (safe)   | into conversation
LangChain            | PythonREPLTool           | NONE (proc) | manual wiring

Every framework can be mis-invoked — pointed at a judgment task (over-code) or skipped for a computation (under-code). This overlay is the gate that decides invocation, regardless of which mechanism is underneath.

Quick reference card

┌──────────────────────────────────────────────────────────────────────┐
│                  EMIT-CODE-VS-REASON DECISION CARD                   │
├──────────────────────────────────────────────────────────────────────┤
│ Single verifiable answer a program could check?                      │
│   YES → EMIT CODE → sandbox → run → feed result back into LM         │
│   NO  → REASON IN PROSE (no oracle exists for judgment)              │
│   MIXED → decompose; route each sub-step independently               │
├──────────────────────────────────────────────────────────────────────┤
│ Arithmetic / parse / sort / count / regex / symbolic → CODE          │
│ Tone / quality / summary / design / synthesis        → PROSE         │
│ "summary AND total"                                  → SPLIT         │
├──────────────────────────────────────────────────────────────────────┤
│ LMs are unreliable calculators but reliable coders.                  │
│ Under-coding fails SILENTLY (wrong plausible number).                │
│ Over-coding fails LOUDLY+CHEAPLY (wasted sandbox round-trip).        │
│ When genuinely uncertain AND a verifiable core exists → lean CODE.   │
├──────────────────────────────────────────────────────────────────────┤
│ NEVER:                                                               │
│  • code a judgment task (over-coding veto)                          │
│  • do multi-step arithmetic in prose (under-coding)                 │
│  • forget to feed the code result back into the LM                  │
│  • nest generated code in JSON (see [[agentsop-output-format-by-model]])     │
│  • loop code-repair unbounded (see [[agentsop-test-fix-loop]])               │
│  • trust LangChain PythonREPLTool on untrusted input (unsandboxed)  │
└──────────────────────────────────────────────────────────────────────┘

引用源 (Citations)

Primary anchors:

• Program of Thoughts (PoT) — Chen et al., arXiv 2211.12588 — decouple computation from reasoning; LM writes code, interpreter computes.
• PAL: Program-aided Language Models — Gao et al., arXiv 2211.10435 — LM as coder, runtime as deterministic solver.
• DSPy ProgramOfThought module — [dspy.ai/learn/programming/modules/] — emit+run+retry mechanism.

Framework / API docs:

• OpenAI Code Interpreter / Assistants code tool — [platform.openai.com/docs/assistants/tools/code-interpreter]
• Anthropic code-execution tool — [docs.anthropic.com/en/docs/agents-and-tools/tool-use/code-execution-tool]
• LangChain PythonREPLTool — [python.langchain.com/docs/integrations/tools/python] (unsandboxed; in-process).

Companion / overlaid skills:

• dspy-sop-skill/SKILL.md — ships ProgramOfThought but not this decision rubric (the gap this overlay fills).
• d-output-format-by-model-skill/SKILL.md — sibling: once you emit code, how to serialize it (PoT for math/parse; code never nested in JSON). Cross-linked as [[agentsop-output-format-by-model]].
• test-fix-loop — the execute → error → retry loop this skill defers to for bounded code repair. Cross-linked as [[agentsop-test-fix-loop]].