Execution Pipeline

Cell Execution Pipeline

Use this skill when debugging execution-related issues: cells that don't execute, outputs that don't appear, execution that times out, or MCP tools that return empty results. This traces the full path from tool invocation to resolved outputs.

The Five Stages

Stage 1: Causal Precondition (required_heads)

Before sending an execute request, the client captures the current Automerge heads of the notebook document:

let required_heads = handle.current_heads_hex()?;

These heads are attached to the request envelope. The daemon's wait_for_required_heads() defers processing until all listed change hashes exist in its copy of the notebook document (checked via get_change_by_hash, a ChangeGraph containment check).

Why this matters: Without required_heads, the daemon might execute against stale cell source: the client wrote "x = 1" but the daemon hasn't received that sync frame yet, so it executes the old "x = 0".

Timeout: 10 seconds. If heads don't arrive, the daemon returns NotebookResponse::Error with "Timed out waiting for required notebook heads" without processing the request. See wait_for_required_heads() and its caller in crates/runtimed/src/notebook_sync_server/peer_writer.rs.

Frontend optimization: Before capturing heads, the frontend calls flushSync() to push any pending source edits into the sync stream, minimizing the daemon-side wait.

Stage 2: Request Submission

The client sends an ExecuteCell or batch RunAllCells request:

let request = NotebookRequest::ExecuteCell {
    cell_id: cell_id.to_string(),
    execution_id: None,
};
let response = handle.send_request_after_heads(request, required_heads).await;

send_request_after_heads wraps the request in a NotebookRequestEnvelope with the captured heads and sends it through the sync task.

Stage 3: Daemon Queuing

The daemon receives the request, waits for required heads (Stage 1), reads the cell source from its synced notebook document, and queues execution with the kernel:

1. Daemon reads source from its NotebookDoc (the CRDT, not the .ipynb file)
2. Creates a unique execution_id
3. Writes to RuntimeStateDoc: execution entry with status "queued", then "running" when the kernel starts
4. Responds with CellQueued { cell_id, execution_id } immediately
5. Forwards code to the kernel via ZMQ execute_request

Key invariant: The daemon writes set_execution_done(eid, success) to RuntimeStateDoc ONLY AFTER all output manifests for that execution are committed. This ordering guarantee is what makes RuntimeStateDoc polling reliable.

Control-plane invariant: Kernel lifecycle signals (KernelIdle, ExecutionDone, CellError, KernelDied) are not output transport. They must not share bounded queues with stdout floods, display churn, or Output widget replay. If output work is pending, drain lifecycle/control signals first so interrupts and queue release remain responsive.

Output-widget replay: RuntimeStateDoc is the durable source of truth for captured Output widget outputs. The kernel-facing SendCommUpdate replay is best-effort output work on a bounded queue. IOPub output arms must use non-blocking enqueue/drop semantics for replay and must not .await a bounded work-channel send before they can observe later status messages.

Display updates: update_display_data messages with display_id are transient display churn. Coalesce them by display_id and commit only the latest pending update off the IOPub hot path. Flush pending display updates after KernelIdle and before ExecutionDone so terminal runtime state still means durable output state is available. The display-update committer's Notify is only a wake hint; the pending map is the source of truth, and each wake or priority flush drains all currently pending display IDs.

Stream-output committer: stdout/stderr chunks may be coalesced and periodic flushes may be dropped when pressure is high. The terminal buffer holds the latest rendered state. Ordering-sensitive boundaries, such as display/error output after a stream, use the stream committer's priority path and wait for the stream flush before clearing terminal state. ExecutionDone also uses the priority path so the final stream manifest is durable before the runtime state becomes terminal.

Stage 4: Terminal Wait (RuntimeStateDoc Polling)

The client polls RuntimeStateDoc for execution completion:

await_execution_terminal(handle, &execution_id, timeout, None).await

Phase 1: Terminal status poll:

• Polls every 50ms
• Checks executions[eid].status for "done" or "error"
• Also watches for kernel-level failure (kernel.lifecycle == Error|Shutdown)
• Returns KernelFailed if the kernel dies while execution is pending

Phase 2: Output-sync grace:

• After terminal status is reached, the output list might still be empty on the client's replica (sync lag)
• Polls every 10ms for up to 500ms (the grace period)
• Exits as soon as outputs appear

Why RuntimeStateDoc, not broadcasts: The ExecutionDone broadcast arrives over a separate channel and the client's Automerge replica may not have caught up on the final stream writes. The RuntimeStateDoc is authoritative: once status is "done", outputs are guaranteed to be in the same document.

Stage 5: Output Resolution

Outputs are inline manifest Maps in RuntimeStateDoc, containing ContentRef entries per MIME type:

let outputs = output_resolver::resolve_cell_outputs_for_llm(&output_manifests, ctx).await;

Resolution depends on MIME type:

• Text MIME (text/*, application/json, image/svg+xml): Inline string if ≤1KB, or fetch from blob store as UTF-8
• Binary MIME (image/png, audio/*, etc.): Always blob store. Frontend gets http:// URL. Python gets raw bytes.
• Widget output (application/vnd.jupyter.widget-view+json): References comm topology/output routing in RuntimeStateDoc; mutable widget values live in the paired CommsDoc

MCP execution paths use preview mode: output is truncated for LLM consumption. Agents that need full output call get_cell(full_output=true) separately.