Test Opencode Tooling

Test OpenCode Tooling

Automated loop that runs OpenCode tasks against afm, captures tool call errors from both sides, classifies each as an afm bug or model error, and generates a report. Does not fix anything.

When to Use

• After changing tool call parsing code (XML, streaming, type coercion)
• Onboarding a new model to verify tool call reliability
• Investigating user-reported tool call failures with OpenCode
• Comparing tool call error rates across models

First Questions to Ask

1. Prompt/PRD — Ask the user to paste the prompt text or provide a file path. This is the task OpenCode will execute (e.g., a PRD, coding task, or test scenario that exercises tool calls).
2. Model(s) — Which model(s) to test? Show available:

   MACAFM_MLX_MODEL_CACHE=/Volumes/edata/models/vesta-test-cache ./Scripts/list-models.sh

3. afm start parameters — Any extra flags beyond defaults? (e.g., --tool-call-parser afm_adaptive_xml, --enable-prefix-caching, --enable-grammar-constraints, --no-think). Recommended: --tool-call-parser afm_adaptive_xml --enable-grammar-constraints — this combination gives the highest tool call success rate (100% on 35B-A3B vs 60% without grammar constraints on realistic workloads).
4. Iterations — How many times to run the same prompt per model? Default: 1. More runs help distinguish flaky model errors from deterministic afm bugs.
5. Working directory — Temp dir for OpenCode to work in. Default: create a fresh /tmp/opencode-test-TIMESTAMP per run.

OpenCode CLI Gotchas

CRITICAL: opencode run hangs silently without a PTY. It prints one INFO line and freezes — no error, no output. You must use one of these approaches:

1. opencode serve + run --attach (recommended): Start a headless server, then attach run to it via expect for PTY
2. expect wrapper: Provides the pseudo-TTY that opencode run requires

Other gotchas:

• opencode.json model field must be a string, not an object — "model": "ollama/model-id" not "model": {"default": "..."}
• The npm provider format (@ai-sdk/openai-compatible) is required for custom baseURL — the "api": "openai" format does NOT accept baseURL
• OpenCode config is loaded from both ~/.config/opencode/opencode.json (global) AND $WORKDIR/opencode.json (local) — local overrides global
• The workdir should be a git repo (git init) for OpenCode to function properly

OpenCode Log & Error Data

Log Files (limited — no tool call errors)

OpenCode writes logs to ~/.local/share/opencode/log/ in UTC-timestamped files (e.g., 2026-03-09T172212.log). These logs do NOT contain tool call errors or tool input/output. They only log permission checks, bus events, and registry start/complete.

# Find the latest OpenCode log
ls -t ~/.local/share/opencode/log/*.log | head -1

# Monitor the latest log in real-time
tail -f "$(ls -t ~/.local/share/opencode/log/*.log | head -1)"

Gotcha: Log filenames use UTC timestamps but ls -lt shows local time. A file named 2026-03-10T001322.log was created at 8:13 PM EDT. Use lsof -p <PID> | grep log to find the current session's log file if it doesn't appear in directory listings yet (OpenCode buffers writes).

When monitoring both afm and OpenCode simultaneously:

• afm log: /tmp/afm-opencode-test.log (or wherever you tee'd it)
• OpenCode log: ~/.local/share/opencode/log/<latest>.log

ALWAYS start OpenCode with --log-level "DEBUG" --print-logs — both opencode serve and opencode run commands must include these flags.

SQLite Database (structured tool call data with errors)

Tool call inputs, outputs, and errors are stored in OpenCode's SQLite database — not in the log files. This is the only place to get the full JSON of failed tool calls.

Database path: ~/.local/share/opencode/opencode.db

Schema: Tool calls are in the part table as JSON in the data column, keyed by session_id.

# List recent sessions
sqlite3 ~/.local/share/opencode/opencode.db \
  "SELECT id, title, datetime(time_created/1000, 'unixepoch', 'localtime') FROM session ORDER BY time_created DESC LIMIT 5;"

# Get ALL tool call errors for a session
sqlite3 ~/.local/share/opencode/opencode.db \
  "SELECT data FROM part WHERE session_id = '<SESSION_ID>' AND data LIKE '%\"status\":\"error\"%';"

# Get errors for the most recent session
sqlite3 ~/.local/share/opencode/opencode.db \
  "SELECT data FROM part WHERE session_id = (SELECT id FROM session ORDER BY time_created DESC LIMIT 1) AND data LIKE '%\"status\":\"error\"%';"

# Get all edit tool errors across all sessions
sqlite3 ~/.local/share/opencode/opencode.db \
  "SELECT data FROM part WHERE data LIKE '%\"tool\":\"edit\"%' AND data LIKE '%\"status\":\"error\"%' ORDER BY time_created DESC LIMIT 10;"

Error JSON format:

{
  "type": "tool",
  "callID": "call_8B05B790A94F4A0EBF2850C0",
  "tool": "edit",
  "state": {
    "status": "error",
    "input": {
      "filePath": "/path/to/file.py",
      "oldString": "text the model expected to find",
      "newString": "replacement text"
    },
    "error": "Error: Could not find oldString in the file. It must match exactly, including whitespace, indentation, and line endings.",
    "time": {
      "start": 1773102597849,
      "end": 1773102597850
    }
  }
}

Successful tool call JSON format:

{
  "type": "tool",
  "callID": "call_34CB225B0D184310BD64A839",
  "tool": "edit",
  "state": {
    "status": "completed",
    "input": {
      "filePath": "/path/to/file.py",
      "oldString": "...",
      "newString": "..."
    },
    "output": "Edit applied successfully.",
    "title": "path/to/file.py",
    "metadata": {
      "diagnostics": {},
      "diff": "Index: /path/to/file.py\n===...",
      "filediff": { "file": "...", "before": "...", "after": "..." }
    }
  }
}

Useful queries for test analysis:

# Count tool calls by status for a session
sqlite3 ~/.local/share/opencode/opencode.db \
  "SELECT json_extract(data, '$.tool') as tool,
          json_extract(data, '$.state.status') as status,
          COUNT(*) as cnt
   FROM part
   WHERE session_id = '<SESSION_ID>' AND json_extract(data, '$.type') = 'tool'
   GROUP BY tool, status;"

# Get all tool call inputs/outputs (pipe to jq for pretty-printing)
sqlite3 ~/.local/share/opencode/opencode.db \
  "SELECT data FROM part WHERE session_id = '<SESSION_ID>' AND json_extract(data, '$.type') = 'tool';" | python3 -mjson.tool

Execution Workflow

1. Setup

TIMESTAMP=$(date +%Y%m%d_%H%M%S)
TEST_PORT=9877
OC_PORT=4096
REPORT_DIR="test-reports/opencode-tooling-${TIMESTAMP}"
mkdir -p "$REPORT_DIR"

Save the user's prompt to a file:

cat > "$REPORT_DIR/prompt.md" << 'PROMPT_EOF'
<paste user's prompt here>
PROMPT_EOF

2. Start OpenCode Serve

Create a workdir with git init and config pointing at afm:

OC_WORKDIR="/tmp/opencode-serve-${TIMESTAMP}"
mkdir -p "$OC_WORKDIR"
cd "$OC_WORKDIR" && git init -q && cd -

Write the OpenCode config. Must use npm provider with options.baseURL:

cat > "$OC_WORKDIR/opencode.json" << EOF
{
  "\$schema": "https://opencode.ai/config.json",
  "provider": {
    "ollama": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "afm-test",
      "options": {
        "baseURL": "http://localhost:${TEST_PORT}/v1"
      },
      "models": {
        "${MODEL}": {
          "name": "${MODEL}"
        }
      }
    }
  }
}
EOF

Start the headless server:

cd "$OC_WORKDIR"
opencode serve --port $OC_PORT --print-logs --log-level DEBUG \
  > "$REPORT_DIR/opencode-serve.log" 2>&1 &
OC_SERVE_PID=$!
cd -

# Wait for serve to be ready
until curl -sf http://127.0.0.1:${OC_PORT}/ >/dev/null 2>&1; do sleep 1; done

3. For Each Model

3a. Start afm with verbose logging

AFM_DEBUG=1 MACAFM_MLX_MODEL_CACHE=/Volumes/edata/models/vesta-test-cache \
  .build/release/afm mlx -m "$MODEL" --port $TEST_PORT -V \
  $EXTRA_AFM_FLAGS \
  > "$REPORT_DIR/${MODEL_SLUG}-afm.log" 2>&1 &
AFM_PID=$!

# Wait for server ready
until curl -sf http://127.0.0.1:${TEST_PORT}/v1/models >/dev/null 2>&1; do sleep 1; done

Where MODEL_SLUG is the model ID with / replaced by _.

3b. Run OpenCode via expect + attach (per iteration)

expect provides the PTY that opencode run requires. The --attach flag connects to the serve instance which already has the config and workdir.

/usr/bin/expect << EXPECT_EOF > "$REPORT_DIR/${MODEL_SLUG}-run${RUN}-opencode.json" 2>&1
set timeout 600
log_user 1

spawn opencode run --attach http://localhost:${OC_PORT} --log-level "DEBUG" --print-logs --format json "${PROMPT}"
expect {
    timeout { puts "TIMEOUT"; exit 1 }
    eof { puts "EOF"; exit 0 }
}
EXPECT_EOF

The --format json flag outputs structured JSON events:

• {"type":"tool_use",...} — tool call with input/output/error
• {"type":"text",...} — assistant text content
• {"type":"step_start",...} / {"type":"step_finish",...} — generation boundaries

Each run creates a new session on the same serve instance. The timeout (600s = 10 min) should be enough for most PRDs — increase for complex tasks.

IMPORTANT: Clean workdir between iterations. Before each run, remove all generated files from the OpenCode workdir so that results from a previous iteration don't contaminate the next one (e.g., OpenCode's "must read file before overwriting" guard triggers on leftover files). The cleanest approach is to stop opencode serve, recreate the workdir from scratch (rm -rf "$OC_WORKDIR" && mkdir -p "$OC_WORKDIR" && cd "$OC_WORKDIR" && git init -q && cd -), copy the opencode.json config back, and restart opencode serve. This ensures each iteration starts with a pristine empty git repo.

# Between iterations: reset workdir
kill $OC_SERVE_PID 2>/dev/null; wait $OC_SERVE_PID 2>/dev/null
rm -rf "$OC_WORKDIR"
mkdir -p "$OC_WORKDIR"
cd "$OC_WORKDIR" && git init -q && cd -
# Re-copy opencode.json config (same as setup step)
cat > "$OC_WORKDIR/opencode.json" << EOF
{ ... same config as before ... }
EOF
cd "$OC_WORKDIR"
opencode serve --port $OC_PORT --print-logs --log-level DEBUG \
  >> "$REPORT_DIR/opencode-serve.log" 2>&1 &
OC_SERVE_PID=$!
cd -
until curl -sf http://127.0.0.1:${OC_PORT}/ >/dev/null 2>&1; do sleep 1; done

3c. Stop afm (after all iterations for this model)

kill $AFM_PID 2>/dev/null
wait $AFM_PID 2>/dev/null

4. Stop OpenCode Serve

kill $OC_SERVE_PID 2>/dev/null

3. Analyze Logs

For each run, analyze both log files to extract and classify errors.