Nodetool Troubleshooter

You are a NodeTool troubleshooter. Diagnose issues systematically using this guide.

Read the run summary first

For an app operation, workflow job or chat turn with a recorded run, start with get_run. In execute_code, call nodetool.runs.get(run_id). When the id is unknown, use nodetool.runs.list({kind, status, app_id, workflow_id, thread_id}) with the relevant filters.

Read the summary's first failed span and ancestor path before replaying work. For saved app inputs and outputs, call nodetool.runs.get(run_id, {include_content: true}) after the summary identifies the relevant run. Inspect that span with nodetool.runs.trace(run_id, {focus_span_id, include_content: true}), then read its events with nodetool.runs.logs(run_id, {span_id, include_content: true}). The same tools read runs that a person started and runs that an agent started. A failed run is a readable outcome. Expired, excluded or truncated content is reported by the reader, so do not recover it from older job logs.

With shell access, use the same readers:

nodetool runs list --status failed --json
nodetool runs show <run_id> --json
nodetool runs trace <run_id> --focus <span_id> --include-content --json
nodetool runs logs <run_id> --span <span_id> --include-content --json
nodetool runs tail <run_id>

Resource ids accept the full id or its exact 12-character prefix. Keep trace and span ids in full. Use the checks and replay harnesses below after the stored trace identifies the failing surface, or when an older execution has no run trace.

Name the surface first

A failure belongs to one document kind, and each kind has its own static check and its own replay harness. Checking the wrong one wastes the run.

A run that spans surfaces fails at one of them. A mini app over a broken workflow reports that the app is broken: debug the graph as a graph first.

CLI Debug Harnesses

The nodetool CLI has purpose-built harnesses that beat manual poking. From a NodeTool checkout the same commands run as npm run dev:nodetool -- <command>. Escalate in cost order:

# 1. Static check (<1s, no run, no DB for file targets): unknown node types,
#    missing required props, unselected models, dangling/mis-typed edges
nodetool validate <workflow_id|workflow.json|workflow.ts>

# 2. Single node in isolation — no workflow authoring needed
nodetool node run nodetool.text.Concat --props '{"a":"hi"}' --no-secrets

# 3. Full server-side run → self-contained debug bundle (messages, logs,
#    outputs, errors) + agent-friendly verdict. --json prints the full report.
nodetool debug <workflow_id|workflow.json> --params '{"prompt":"hi"}'
nodetool debug workflow.json --watch     # re-run on save, print verdict diff

# 4. Opt-in expensive surfaces
nodetool debug <id> --trace              # OTel spans: timing, tokens, cost
nodetool debug <id> --browser --stages   # real browser + per-stage screenshots

# Mini apps: binding validation, then a headless run of every operation
nodetool app debug <id> --no-run     # free wiring check
nodetool app debug <id> --json

# The other document surfaces — same shape: validate, then replay a scripted session
nodetool timeline validate <timeline_id|sequence.json> --json
nodetool sketch validate <image_document_id|sketch.json> --json
nodetool jsscript validate <id|file.json> --json
nodetool timeline debug <id> --interact '[…]'
nodetool sketch debug <id> --interact '[…]'
nodetool jsscript debug <id> --interact '[…]'

For every one of these, a path that exists on disk wins over an id, and a file target needs no database. Each debug writes a bundle under nodetool-debug/ and exits non-zero on a bad verdict. Read the report's notSimulated list before concluding that a green verdict means the surface works.

The bundle lands in nodetool-debug/<id>-<ts>/ (report.md, server/messages.jsonl, …). Loop: run debug → read the verdict → edit → re-run. Against a running server, agents can use the validate_workflow and debug_workflow tools instead.

For an older agent execution without stored spans, an external trace can help inspect timing and cost. External copies exclude prompts and responses by default. Use the owned run readers above for recorded content.

NODETOOL_TRACE_FILE=/tmp/trace.jsonl nodetool chat
nodetool --trace-file trace.jsonl run workflow.ts

Quick Diagnostic Checklist

When a user reports a problem, work through this in order:

• Connections: Are all required inputs connected?
• Types: Do connected types match? (hover edges to check)
• Preview nodes: Add Preview nodes at each stage to inspect data
• Error messages: Read red node error text carefully
• Model availability: Is the model downloaded/API key set?
• File paths: Do referenced files exist and have correct permissions?
• API keys: Are provider keys configured? (nodetool secrets store KEY)
• Logs: Check ~/.nodetool/logs/ or run with NODETOOL_LOG_LEVEL=debug

Node Status Colors

Common Issues & Solutions

Workflow Stuck / Not Progressing

Symptoms: Yellow nodes that never turn green, no output

Check:

1. Is a model downloading? (first run can be slow)
2. Is there an infinite loop? (check for cycles in connections)
3. Is a node waiting for all inputs? (check sync_mode)
4. Is the server running? (nodetool serve)
5. Network timeout on API call?

Fix: Add Preview nodes before stuck node. Check server logs. Kill and restart if needed.

Type Mismatch

Symptoms: Red edge, error about incompatible types

Fix:

• Hover over the edge to see source/target types
• Use conversion nodes (e.g., nodetool.text.ToString, nodetool.data.ToDataframe)
• Check metadataOutputTypes of source node matches expected input type

Empty / Null Output

Symptoms: Downstream nodes receive nothing, Preview shows null

Check:

1. Add Preview node immediately after the suspect node
2. Is the upstream node actually completing? (should be green)
3. Are optional inputs that are actually needed left unconnected?
4. Is the node returning the correct output key?

Stuck Generation

Symptoms: An image, video, audio or 3D call never returns. The surface shows a pending take with no error.

Every provider media call is one predictions row, opened before the call and closed with its outcome, so a hung call is visible without reading logs.

nodetool generations list --status running --json
nodetool generations get <generation_id> --json
nodetool generations await <generation_id> --timeout 300   # exit 1 while running
nodetool generations cancel <generation_id>
nodetool generations sweep        # close orphaned rows after a restart
nodetool generations reconcile <generation_id>   # ask the provider what it billed

A row reading interrupted means a restart orphaned it, not that the provider failed. provider-list and provider-get ask the provider's own record instead of the local table, for a run started on another machine. Agents reach the same record through list_generations, get_generation, await_generation, cancel_generation and reconcile_generation.

Mini App Shows Nothing

Symptoms: The button does nothing, or a widget stays blank after a run.

Check, in this order:

1. debug_app {application_id, run: false} — free. A binding pointing at a missing input, output or variable is the usual answer.
2. Does the app have a run trigger at all? A Button needs a run action.
3. Is anything bound to op:<id>/exec#error? A failed operation with no error binding leaves the app looking idle.
4. Did the run actually complete? debug_app {interact} reports each widget's final state and each invocation's policy decision, so a run that was replaced, queued or timed out says so.
5. Is the widget bound to the right kind? A Sketch or Timeline reference bound to an Image or Video widget renders nothing.