Sandboxed.sh Missions

Sandboxed.sh Missions

Delegate coding/automation tasks to isolated containerised missions via the mcp_sandboxed_assistant_* MCP. Each mission runs in a chosen workspace (a fresh container with a known set of init scripts and pre-installed tools) and executes the prompt autonomously via a configured agent. Intermediate tool calls stay in the mission's own context.

start_mission with project / track is a worker. Confirm pending/active, then end the turn — do not poll, do not invent a cron just to wait, and do not wait for a callback in this chat. Completion is a mission row; a controller tick (Orb cron page) reconciles it. origin_session_id is v1 plumbing when a Hermes plugin injects it.

When dispatching against a project roadmap, also pass the declared track, its acceptance_criteria, and a stable retry-safe idempotency_key. The server reserves the track owner and links the mission as one durable intent; reusing the key returns the original launch instead of duplicating work. Mission completion alone does not satisfy the track. Accepted criterion evidence at the governed artifact version must be recorded separately.

For long workspace commands, use start_workspace_job with a stable idempotency_key. The unified MCP returns an action receipt first. Read get_action(action_id) to obtain the job ID, then inspect that existing job. workspace_bash and the duplicate worker launch aliases are retired.

With sandboxed-mcp, every mutation (including ask, resume, cancel, task and project updates) requires an idempotency_key and returns a durable action receipt. Retry an interrupted submission with the same key and arguments. A completed action means its tool call completed; inspect the returned mission or job to determine whether the underlying work completed. For reconciliation_required, inspect that target before doing anything else; never create a fresh key to bypass uncertainty. Calls use a 60-second transport budget; the accepted work continues on Core after a disconnect.

Every executor may launch a cloud mission using start_mission and its cloud selection. Discover available accounts/models through list_cloud_accounts and list_cloud_models; owner account quotas remain enforced. Credentials are supplied by the trusted launcher. Never request Core's signing secret or copy the operator's login token into a harness. Executor access covers its own mission and direct children; Core supplies the parent identity.

This is not the same as delegating to a CLI coding agent (Claude Code, Codex, OpenCode) via the terminal tool. The MCP runs an entire conversation loop inside the container; the CLI agents are interactive programs you spawn in a single terminal() call. Use this skill for isolated multi-step research/coding, or work that needs a specific pre-baked workspace (e.g. tailscale-ubuntu, minecraft, dgx-spark).

To withdraw work still queued, call cancel_action(action_id, idempotency_key). Read its receipt: cancelled=false means dispatch already started or settled; use the existing mission/job cancellation tool after inspecting the target. Cancelling an MCP request or closing stdio does not cancel accepted work.

When to use

• User says "lance une mission", "sandboxed", "spawn a worker", "delegate to a coder in a clean room".
• Task is non-trivial: a project to scaffold, a refactor, a test suite, a multi-file code review.
• You want isolation: the work must not touch the host's working state, dependencies, or secrets.
• The task benefits from a specific workspace's pre-installed tools (Python/uv, Node/bun, gh CLI, Tailscale, etc.).

When NOT to use:

• Single trivial edit → patch / write_file is faster and cheaper.
• Task needs back-and-forth with the user → subagents can't use clarify.
• You need a long-running daemon → use cronjob or terminal(background=true) instead. Missions terminate when the agent emits its final message.
• Task is purely a tool call with no reasoning → execute_code is more direct.

The MCP surface

Terminology: backend, workspace, host, and remote node are different layers

When a user asks which “servers/backends” sandboxed.sh uses, answer by layer instead of conflating them:

1. Control-plane host — runs sandboxed.sh production and orchestrates mission state.
2. Workspace — container/host execution environment selected by workspace_id; examples include project workspaces and the dedicated dgx-spark workspace.
3. LLM backend — the backend field (codex, claudecode, opencode, grok, antigravity); this selects the agent/model transport, not a physical machine.
4. Remote node/build worker — extra compute reached through sandboxed-node or /api/remote-build; current documented general runners are babylon, nippur, and ashur, while the Lean build fleet also includes dgx-spark.
5. Adjacent runner — e.g. a GitHub Actions self-hosted runner. This is not automatically a sandboxed.sh remote node even if it runs on the same machine.

For inventory questions, report all SSH-reachable machines separately from the subset registered as sandboxed.sh compute. Date any “currently deployed” claim unless live state was checked via GET /api/remote-nodes and GET /api/health/fleet.

Production DGX routing incident rule. If disk-sentinel reports only dgx-spark unreachable, check systemctl is-enabled tailscaled, systemctl is-active tailscaled, and tailscale ping -c 2 100.77.4.93 on agent-core before changing keys or topology. The expected recovery is systemctl enable --now tailscaled, followed by strict-host-key SSH and a silent sentinel run. Do not recreate an old-agent tunnel: old-agent is compute-only.

Picking the agent

The agent field is one of the agent names registered on the platform. The two universal agents are build and plan. Some setups also offer ana (audit) and paloma (coordinator), but availability varies per installation and workspace — do NOT assume all four are available. The right one depends on the work:

• build — general implementation agent. Use for "build this project", "scaffold this repo", "implement feature X". Default for coding tasks.
• plan — research/planning agent. Use for "investigate options", "draft an architecture", "compare libraries", "write a draft", "research and summarize". No code (or minimal code). Default for research/article/analysis tasks.
• ana — analytical/audit agent. May not be available — check first or use build if you just need code review.
• paloma — coordinator/operator agent. May not be available.

Pitfall — agent availability and naming vary by deployment. agent is a registered platform profile and backend selects the execution backend; they are logically separate, but current deployments may register backend-aligned agent IDs such as codex in addition to profiles such as build and plan. Do not hard-code a universal agent list. Use the live tool schema/catalog or a previously verified route. For Codex, always set backend="codex" and the exact OpenAI model ID in model_override; use the deployment's accepted agent profile (codex, build, or plan) and verify the created mission records the requested agent, backend, model_override, and model_effort. A mission merely reaching pending is not enough—re-read it after startup and require execution evidence before calling the route healthy.

The backend parameter (opencode, claudecode, codex, grok, antigravity) selects the LLM backend, not the agent — this is the underlying model the agent uses.

Native Antigravity

Use backend="antigravity" for Google's native agy CLI, independently of the Gemini API backend. Use an exact model ID discovered on the execution machine; Core, local Orb, and each node can have different Google account entitlements. get_backend_auth_status reports native login as indeterminate (ready: null), not as a missing API key. Verify target-account model discovery and an actual turn. Nodes require their configured managed Antigravity profile. Keep prompts within 16 KiB and put larger context in workspace files. Resume the recorded native conversation; only an explicit machine transfer authorizes replacing its source-machine identity. See setup and recovery.

User-mandated model runs: bound, verify, then independently gate

When the user explicitly requests a particular non-default model, honor the request with the exact backend/model identifier and verify the created mission records that identifier plus an active status shortly after dispatch. Treat that worker as a bounded implementation or investigation lane, not as the sole merge or architecture authority:

1. Give it a concrete scope, reproduction/validation criteria, and explicit no-go boundaries (for example: no merge, no legal attestation, no secrets).
2. Require it to distinguish source evidence from assumptions and return verifiable handles for every external change.
3. Independently verify pushed commits, tests, PR state, and any production claim before reporting success.
4. Route material architecture decisions or merge readiness through the normal stronger/latest-head review gate unless the user explicitly waives it.

This keeps a user-requested model useful without converting its self-report into an unverified operational decision.

Codex /goal objective length limit

Codex goal mode has a strict objective-size limit: if the prompt starts with /goal and the objective is too long, the mission can immediately fail with:

codex thread/goal/set failed: goal objective must be at most 4000 characters

Recovery pattern:

1. Keep /goal itself short (under ~3500 characters to leave margin).
2. Put only the durable objective, constraints, and final-output schema in the /goal text.
3. Move bulky context into linked repo docs, a branch issue, project files, or a follow-up non-goal prompt if the platform supports it.
4. Relaunch a fresh mission with the shortened /goal; do not keep retrying the same oversized objective.

This limit is separate from model context size. A model can handle the information, but the Codex goal API rejects overlong goal objectives before execution starts.

Backend tiering for cost optimisation (multi-provider routing)

The opencode backend is the universal router — it can target ANY provider model via model_override, including non-OpenAI/Anthropic providers that the other backends can't reach (Z.AI GLM, Minimax, Kimi/Moonshot, Cerebras, Spark, Virtuals). The other backends (claudecode, codex, grok, antigravity) are locked to their respective providers.

This enables a cost-tiering strategy when the user has "unlimited" quota on alternative providers:

How to dispatch on an alternative provider:

mcp_sandboxed_assistant_start_mission(
    agent="plan",           # research/planning agent
    backend="opencode",     # the universal router
    model_override="glm-5.2",   # or "MiniMax-M3", "moonshotai-kimi-k2-7-code", etc.
    ...
)

Discovering available providers and their model IDs:

# GET /api/ai/providers (JWT-authenticated) returns:
# - id (UUID), name, provider_type, use_for_backends[]
# - Models live in the static catalog: GET /api/providers → {providers: [...]}
# Provider types seen (June 2026): anthropic, openai, google, xai, cerebras,
# zai, minimax, kimi, spark

Fable 5.1 via claudecode only. Virtuals is deprecated and no longer available. For demanding reasoning and long-horizon agentic work, launch via backend="claudecode" with the exact catalog ID model_override="claude-fable-5-1". The older claude-fable-5 remains available only for explicit compatibility. Do NOT use opencode + virtuals/claude-fable-5-1 — Virtuals has been removed from the provider catalog.

Model ID gotchas:

• Z.AI: glm-5.2, glm-5.1, glm-5-turbo (lowercase, hyphenated)
• Minimax: MiniMax-M3, MiniMax-M2.7 (camelCase as in catalog)
• Kimi: moonshotai-kimi-k2-7-code (not bare kimi-k2.7)
• If model_override fails, the error message lists valid IDs — copy from there

User-mandated model that produces no output: do not silently replace it with another model and present the substitute as its opinion. Retry once with a sharply bounded, direct-output prompt (for example: no tools/web, explicit word cap and output schema). If the same model again reaches an idle/stall timeout without content, report that it was successfully addressed but returned no usable answer, then synthesize only from workers that actually produced output. Preserve the distinction between provider failure and research conclusions.

The routing mental model: opencode is a superset of the provider surface. If a provider's use_for_backends includes opencode, you can route to it. If it only lists a specialised backend (e.g. codex for OpenAI), you MUST use that backend and cannot reach it via opencode.

Pitfall — model_override needs the exact catalog ID. To override which LLM a backend uses, pass model_override — but it must match the provider's catalog ID exactly. Anthropic uses claude-<family>-<ver> with hyphens, not the shorthand leaderboard name: claude-opus-4-8 ✅, NOT opus-4.8 ❌ (the leaderboard/website may call it "opus-4.8", but the API catalog does not). If you get Model 'X' not found in <provider> catalog, the error message lists every valid ID — copy the exact string from there. Other providers (OpenAI, etc.) generally accept their public model names directly.

Pitfall — model_effort caps differ per backend and may change by model/backend version. The model_effort field accepts low/medium/high/xhigh/max, but not all backends honor all values. Historically codex rejected "max"; recent codex + gpt-5.5 missions have accepted "xhigh" and moved to active successfully. If a user explicitly requests codex + xhigh, pass it through and verify the mission status immediately with get_mission / list_active_missions. If launch fails with an effort-related error, drop one level ("max" → "xhigh" → "high") until accepted. When no model/effort is specified, "high" remains the safest cross-backend default.

Pitfall — opencode backend can crash on non-standard model_override values with ProviderModelNotFoundError. Seen in the Beal campaign (June 2026): launching opencode with model_override="glm-5.2" or model_override="MiniMax-M3" (passed as if they were model IDs) returned awaiting_user status with ProviderModelNotFoundError. The opencode routing treats the override as a provider identifier for some configurations, not a model name, even though the same model_override is documented to work as a model name in other contexts. Workaround: use an approved reliable route for critical work (for Verity: codex + gpt-5.6-terra), and always verify the dispatch succeeded by checking get_mission status within 30s. If it is terminal before repo work, do not treat it as a model result or burn a cache build on a doomed mission.

Pitfall — a shared OpenCode adapter crash is an infrastructure incident, not three independent bad-model results. A launch can pass creation, run the one-time OpenCode database migration, then terminate before any repository command with fn3 is not a function (observed identically for glm-5.2 and MiniMax-M3). When two or more alternative-provider scopes fail at that same initialization boundary: (1) classify them as failed transport attempts with no proof/review evidence; (2) stop further OpenCode launches rather than serially burning surplus quota; (3) start exactly one bounded sandboxed-sh-dev repair mission on a reliable route, requiring an alternative-override regression test and a PR; and (4) keep critical work moving only on approved reliable routes, respecting Codex OAuth launch staggering. After the fix is merged/deployed, validate with one narrow no-push alternative-provider probe before relaunching deferred workers.

Picking the workspace

list_workspaces returns the full set. The right choice depends on what tools the task needs. Key workspace properties to check:

• init_scripts — e.g. base, uv-python, bun-mcp, github-cli, tailscale, browser-x11. These are the only things pre-installed.
• mcps — additional MCPs the agent gets in-container (e.g. google-calendar, orchestrator).
• env_vars — secrets/tokens injected at start (e.g. GH_TOKEN, GOOGLE_OAUTH_CREDENTIALS).
• tailscale_mode — if you need network reach through the tailnet.
• status — ready is what you want; pending means the container hasn't been started yet and may take minutes to initialise.

Secret-safe workspace discovery. Treat a workspace record as potentially secret-bearing, even for a read-only REST lookup: some control-plane responses can include env_vars values rather than just their names. For selection, reduce the response at the source to only id, name, status, init_scripts, skills, and an array of environment-variable names. Never print or retain the env_vars map. If an inspection nevertheless exposes a credential-like value, stop expanding that output, do not copy it into prompts/reports/other calls, and report only the exposure class and endpoint to the security owner for normal rotation/remediation.

Backend-specific workspace-visibility canary

A mission record carrying workspace_id / workspace_name does not prove that its selected backend sees the intended workspace mount or project checkout. This matters for MCP canaries: one backend can reach the project while another starts in an isolated empty mission directory.

Before declaring a workspace-dependent canary successful, require all of:

1. the expected checkout/project path exists in that mission's execution scope;
2. its Git ref and project pin (lean-toolchain, when relevant) can be read there;
3. an actual registered MCP tool resolves a file from that checkout, not merely that its tool name is callable.

If the route sees only an ephemeral mission directory and MCP calls report project/file-not-found, classify it as a workspace-mount/configuration defect, not a Lean or MCP semantic failure. Keep the canary read-only and retain only concise path/tool-result evidence. When authorized, dispatch at most one bounded no-deploy control-plane repair on a reliable route; require a regression covering backend mount/project-path propagation and prohibit it from modifying the target workspace, checkouts, credentials, or permissions. Re-run the canary only after the repair is independently verified.

Stored GitHub auth may not appear in workspace env_vars

A workspace inventory with no GH_TOKEN / GITHUB_TOKEN does not by itself prove authenticated GitHub access is unavailable: it may have valid stored gh and git credentials. Before recording an auth blocker for an existing-branch repair, run this secret-safe probe in the same workspace and retain only booleans:

# Do not print login, token, credential-bearing remote, or git config.
env -u GH_TOKEN -u GITHUB_TOKEN gh api user --jq .login >/dev/null
gh repo view <owner>/<repo> --json nameWithOwner >/dev/null
git ls-remote https://github.com/<owner>/<repo>.git HEAD >/dev/null
echo 'github_api_capability=ok; git_read_capability=ok'

This proves authenticated API/read transport in that execution scope and prevents a false external-auth blocker. It does not prove normal push transport; require the worker to re-check that immediately before a guarded non-force push. It also does not waive any controller policy that explicitly requires propagation of a named environment secret: record that distinction rather than inventing a secret name or exposing stored credentials.

Known workspace init-script matrix (June 2026)

Critical pitfall — assistant workspace has NO coding-agent CLI installed. Missions in assistant cannot invoke opencode, codex, or claude (the Anthropic CLI), AND the workspace lacks curl, wget, npm, and bun globally. The bun-mcp init script only enables bun for MCP server processes — not for shell-level installs. Errors surface as one of:

• OpenCode CLI 'opencode' not found and neither curl nor wget is available in the workspace.
• Codex CLI 'codex' not found and neither npm nor bun is available in the workspace.
• Claude Code CLI 'claude' not found and neither npm nor bun is available in the workspace.

Any Python script that does import requests and hits an external URL still works (Python ships its own HTTP), but shell-level network calls and CLI invocations fail. If your task needs ANY coding-agent CLI or shell-level network tools, pick a different workspace or run on host.

Workaround when the agent's workspace is too stripped-down. Pivot to host (workspace_id 00000000-0000-0000-0000-000000000000, path /root). The host has the full user shell, curl/wget/gh/uv/python3 all pre-installed, and access to secrets in /var/lib/hermes-assistant/.env. The trade-off: you lose the per-task container isolation and you may have to be more explicit about cleanup.

Pitfall — codex backend has a single-use OAuth refresh token. Launching multiple codex missions simultaneously (parallel batch in one turn) causes an auth race: each mission's container tries to refresh the same OAuth token concurrently, and since the refresh token is single-use, all but one fail with auth_error / refresh token already used. This is fatal for those missions — the token is consumed and the mission cannot recover. Never launch a second mission onto a worktree that already has a live occupant — create_mission returns workspace_occupied. Stagger remaining sequential Codex launches, or use claudecode / kimi / glm. The other backends (claudecode, opencode) do not have this OAuth race and can be batch-dispatched freely. If a Codex mission dies with auth_error / refresh_token_invalidated on launch, create_mission will reject further backend=codex with codex_oauth_invalidated until the token is repaired — re-dispatch on claudecode (or kimi/glm) on the existing worktree writer. Do NOT retry on Codex. A newly accepted Codex mission can show pending/active for a few seconds and then flip to failed/acknowledged with an OAuth error such as refresh_token_invalidated, so for watchdog/anti-stall dispatches do a second verification read (or list_active_missions) shortly after launch before updating trackers or reporting the launched ID. If an automatic/manual fallback mission appears tagged like redispatch-after-codex-auth, track and report the fallback mission ID as the live worker, and keep the failed Codex ID only as a failed attempt. Fallback metadata may not exactly mirror the original (backend=claudecode, intent=implement, github_pr=null are all plausible), so identify it by project/title/tags/workspace as well as intent. Auto-redispatch may lag behind the first failed-mission read: after seeing Codex refresh_token_invalidated, wait briefly and read exact mission + active list again before marking the workstream blocked or patching trackers to blocked-external.

Library skills — the mission agent's knowledge base

Each workspace exposes a set of library skills (visible as skills: [...] in list_workspaces output). These live at /root/.sandboxed-sh/library/skill/<name>/SKILL.md and are git-versioned — the entire library/ directory is a repo (Th0rgal/sandboxed-library, branch main).

The mission agent reads library skills, NOT Hermes skills. Your Hermes skills (oraxen-pr-review, release-oraxen, etc.) are only read by you (Paloma). The agent inside the container reads the library skills attached to its workspace. This has a direct operational consequence:

• When a mission repeatedly fails at the same step (wrong build flag, missing asset, wrong path, wrong API endpoint), the root cause is often an under-specified or outdated library skill, not a bad mission prompt. Patch the library skill at /root/.sandboxed-sh/library/skill/<name>/SKILL.md and git commit for a durable fix that propagates to all future missions in that workspace.
• Patching only your Hermes skill does NOT help the mission agent — it never sees it. Enriching the Hermes skill helps you write better prompts, but the failing agent still starts from the stale library skill.
• Always git add + commit library-skill changes so they survive server rebuilds and container re-provisioning. Uncommitted edits to library/ can be lost on the next deploy.

Concrete example (Oraxen jar builds). Every mission produced a stripped demo jar (no textures) because the library skill oraxen was missing the -Poraxen_compiled=false flag and the core/src/main/pack/ path migration (v1.217+). Fixing the Hermes oraxen-pr-review skill helped Paloma write more explicit prompts, but missions kept failing until the library skill oraxen itself was enriched and committed. Lesson: patch the layer the failing agent actually reads.

The launch pattern

mcp_sandboxed_assistant_start_mission(
    title="<short human title>",
    prompt="<full self-contained prompt>",
    workspace_id="<from list_workspaces>",
    agent="build",                # one of the platform agent names
    backend="opencode",            # or claudecode / codex / antigravity / grok
    model_effort="medium",         # low/medium/high/xhigh/max
)

Conversational launches still call start_mission. End the turn after dispatch. Do not poll and do not create a cronjob just to wait. Completion is the mission row (and the next controller tick), not a [Mission callback] you sit on. Prefer delegate_task(backend="mission") when you want an explicit handle; start_mission from Orb / desktop / API is equivalent.

Prompt structure (template):

You are <doing X> for <person/repo>. Goal: <one-line outcome>.

## Context
- <inputs the agent has no way to know without this section>
- URLs, IDs, file paths, repo owner, etc.

## Tech stack
- <language, deps, what tools to use>

## Architecture / approach
- <high-level shape of the solution>

## Deliverables
1. <concrete artifact 1>
2. <concrete artifact 2>
3. ...

## Constraints / gotchas
- <what NOT to do>
- <known library/tool quirks>

## Workflow
1. <step 1>
2. <step 2>
3. Verify: <how to know it worked>

## Workspace
- <which init scripts the agent can rely on, and which NOT>
- <any pre-existing env vars to use>

Self-containment is the single biggest factor in mission success. The mission agent has no access to your context — every fact, every file path, every constraint the agent needs to make a correct decision must be in the prompt. If you find yourself wanting to say "see above" or "you know what I mean", rewrite the prompt.

Monitoring and recovery

How the result comes back

Conversational / Orb launch: start_mission with project when you have one. Confirm pending/active, then end the turn. Look at the mission row. Do not verify PALOMA_WEBHOOK_FORWARD_URL, do not start fleet-heartbeat, and do not create a cronjob to poll.

Controller launch (cron tick with deliver: project:<slug> plumbing): pass project, track (a key from get_situation; unknown keys are absorbed as unplanned items), intent, and a stable idempotency_key. One writer per track: 409 track_owned names the holder — attach to it or dispatch read-only (writer=false). Do not wait. Report on the next tick. Never stamp a cron_* session as origin.

On completion: inspect get_mission / get_mission_digest plus artifacts through the direct source before reporting. Mission self-report is not success. Notify Thomas for a user-launched completion, failure, blocker, decision, PR opened/merged, or useful research result. Stale duplicate ACKs may stay silent.

Do not block a Hermes turn with sleep until complete. Missions can run for hours; a sleeping parent can be compacted or killed.

• Status values: pending → active → awaiting_user (the agent has a question for you, use send_message_to_mission) → terminal (completed / acknowledged / interrupted / failed). There is no cancelled status in the enum. completed is a real terminal state produced by background-job reconciliation and explicit completion transitions — treat it as done, do not keep monitoring it.
• If a mission is awaiting_user, the agent is blocked OR has finished and is waiting for ack — the two are indistinguishable from status alone; read the last transcript event to tell them apart (see the pitfall in "Surveying a fleet" below).
• To close a finished awaiting_user mission: do NOT use cancel_mission — it returns 404 on non-active missions (the cancel handler needs a running control actor that doesn't exist post-completion). Use POST /api/control/missions/:id/status {"status": "acknowledged"} via the REST API. For bulk cleanup, loop /status calls with a JWT + ThreadPoolExecutor. Only use cancel_mission (or resume→cancel) on active/pending missions you want to kill mid-flight.

Non-interrupting Ask fallback when sandbox isolation is unavailable

When using /ask as a side-channel, prefer sandbox: true for extra isolation. If it returns 400 Sandbox mode requires a git workspace (no isolated worktree could be created), this means Ask could not create an isolated worktree for that mission; it does not mean the side-channel is broken. If the user asked for non-interrupting inspection, retry /ask without sandbox only with a strict read-only prompt: no file modification, no task execution/reruns, no commits/pushes, no secret/env printing.

Circuit-breaker-safe sequence: do not probe sandbox:true on several missions in parallel. A predictable sandbox_unavailable response counts as an MCP failure; three parallel failures can trip the sandboxed-assistant circuit breaker and block the immediate unsandboxed fallback. Probe one representative mission first. If sandbox creation fails, switch that mission to a strict read-only unsandboxed Ask, then inspect additional missions sequentially or in batches of at most two. If the breaker already tripped, use the authenticated REST mission/events endpoints for read-only reconciliation during cooldown rather than retrying MCP into the cooldown. For long histories, page events with since_seq; a mission whose first page looks stale may have thousands of later executable events.

Mission PR dispatch: verify the PR exists, not just that the mission launched

When the user asks for a PR link from a mission, do not answer from launch status. Read get_mission history/status and only provide a PR URL if the mission returned one or if you independently verify it in GitHub. If the mission failed before producing a PR (e.g. terminal_reason=infinite_loop), say no PR exists yet, then relaunch with a shorter/bounded prompt or different backend. Long Claude Code website/update prompts can loop; retrying with a compact gate + final-output schema on a different backend is the durable recovery pattern.

Mission PR completion is not merge-readiness

A mission can finish as acknowledged with a self-report like "PR opened, builds passed" while the PR is still not merge-ready: later CI jobs may still be queued, OCR/advisory jobs may post retryable failures, or a latest-head Codex review may contain unresolved threads. Before telling Thomas the workstream is done, verify the PR live:

1. gh pr view <N> --json state,isDraft,headRefOid,mergeStateStatus,statusCheckRollup,reviews,comments.
2. Check latest-head Codex specifically: identify the current headRefOid, then inspect review threads for unresolved/non-outdated comments authored by Codex on that commit. A top-level "no major issues" comment is useful, but active review threads win.
3. Treat queued/in-progress required CI as pending, not green; if only long Foundry/Lean jobs remain, say that precisely.
4. If a real blocker exists, resume the same mission with a bounded follow-up containing the exact PR, current head SHA, thread URL/path/line, and requested validation loop. Do not start a duplicate mission unless the original is unrecoverable.

Merge authority, independence, and provenance

For repositories/campaigns Thomas asks Hermes to manage, autonomous guarded merge is authorized. Keep the roles explicit:

1. Give implementation writers no merge; this limits that mission and does not revoke the campaign-level merge grant.
2. Give the dedicated integrator the exact repository/PR scope and immutable head, set writer=true so it holds the exclusive PR writer lease, then set request_merge_authority=true. Never supply may_merge or merge_authority_source from model-controlled arguments. The assistant-mcp server derives the grant itself only when the canonical owner/repository#number target matches its operator-configured HERMES_MERGE_AUTHORITY_REPOSITORIES; it signs the receipt with HERMES_MERGE_AUTHORITY_SOURCE. An absent/mismatched grant—or a merge request without writer=true—must fail closed. Require --match-head-commit or an equivalent API precondition.
3. Treat only required SUCCESS checks as green. NEUTRAL, SKIPPED, ACTION_REQUIRED, missing checks, and a bot declining to run are not green. If no CI exists, say NO_CI_CONFIGURED and require clean local gate reproduction plus a separate exact-head review.
4. If a reviewer pushes a fix, it is now a writer for that head. A different read-only reviewer (or a GitHub Codex review received after that push on the exact head) must certify it before merge.
5. Resolve ordinary branch conflicts with a normal merge from the current base, then rerun all exact-head gates. Never rebase or force-push under this standing authorization.
6. Persist a merge receipt: executing mission ID/role, controller origin when available, authority source, PR/head, CI classification, reviewer/head/time, thread count, merge method, and merge commit. GitHub's mergedBy is the shared account identity, not actor provenance.

Terminal push versus the current PR head

A worker can correctly finish a normal push, local full/trust validation, and an @codex review request, while repository automation appends a derived commit before the controller's next read. The worker's reported SHA is then no longer the governed artifact.

Before calling the PR ready, waiting on CI, or dispatching a replacement:

1. Independently compare the worker-reported pushed SHA with the live headRefOid and inspect the current workflow's head_sha.
2. If they differ, record the worker as terminal after push, never as an active owner. Re-evaluate CI, exact-head review, and unresolved technical threads on the live head only.
3. If the live-head workflow is terminal action_required with no child jobs, classify it as an external platform/approval gate. Do not launch a polling or repair worker and do not use the old-head Codex request as current-head review evidence.
4. After the owner/platform starts the exact-head workflow, re-run current-head CI and review/thread reconciliation before considering merge or downstream stack progression.

This avoids two false moves: a duplicate repair for a branch that already advanced, and a claim that an earlier clean review covers an automation-derived head.

This pattern prevents a finished mission's optimistic self-report from masking post-PR review findings.

Safety-audit → existing-branch repair handoff

For PRs that implement retention, garbage collection, storage lifecycle, or any code that could later delete operator data, a read-only audit finding is a dispatch trigger — but it is not enough by itself to authorize a fix:

1. Pin the audit to the immutable PR head and independently inspect the cited current-source lines before acting. Confirm the executable control-flow claim; an audit can correctly spot one defect while over-reading a nearby predicate or documentation comment.
2. Re-read the exact audit mission and global active/pending list. If it ended awaiting_user with turn_complete, ACK it so it cannot remain a phantom evidence owner.
3. If the confirmed finding is current-head actionable and no scoped writer exists, launch exactly one existing-branch repair with no-merge/no-force-push/no-new-PR/no-deploy/no-real-cleanup authority. Pin the before-SHA and require a normal guarded push only after tests pass.
4. Test both dry-run and opt-in execution semantics. Protection for live processes/scopes and active missions must be checked before candidate enumeration and before deletion, not only in a later orphan sweep. Regression fixtures must show caches, worktrees, images, unattributed paths, and nested build caches stay retained unless an explicitly authorized policy says otherwise.
5. After any pushed repair, restart the normal final-head CI/review/thread gates. Never infer that a dry-run-only implementation is safe to merge merely because its ordinary CI is green.

Use it deliberately for long-running missions that have parked themselves with a background waiter or vague "standing by" note. Ask for a bounded, safe status report instead of a broad "what's up?" Example:

Please report current progress in exact numbers: total work items identified at start,
completed now, currently running, queued/not started, failed, current phase, ETA, and
any blockers. Do not print secrets or credential values.

Then read get_mission, not only get_mission_events(view='transcript'): event pagination can return [] while get_mission.history contains the newly appended user message and the agent's latest response. Trust concrete counts from the mission only as a self-report until independently verified from repo artifacts / PRs / logs.

Distinguish a verified blocker from unfinished reconnaissance

For implementation watchdogs, do not promote a repository gap into an “exact reproducible blocker” merely because a stalled worker found empty artifact directories, .gitkeep placeholders, or documentation saying a target is blocked. Those facts prove only that the artifact is not checked in; they do not prove it cannot be generated.

Before reporting a technical blocker, require evidence that the worker attempted the relevant build/generation/execution path and captured:

1. the exact failing command and key output;
2. source revision and relevant tool/interface versions;
3. the smallest prerequisite that would make that command advance;
4. confirmation that no branch, commit, test log, or PR artifact already exists.

If the worker stalled during read-only reconnaissance and never invoked the pipeline, classify the failure as a transient stall, not a technical blocker. After checking for an active successor and live GitHub artifacts, launch exactly one bounded replacement when authorized. Its prompt should explicitly require an immediate pipeline attempt and say that absence of pre-generated artifacts alone is insufficient grounds to stop.

The Ask sidecar can recover worktree facts or a final report missing from get_mission_events, but its summary remains secondary evidence: use it to locate paths, commands, SHAs, and logs, then independently verify external PR state. A claim such as “the required toolchain may be absent” is inference unless a command actually demonstrated it.

Acknowledged while background work is running is not done

A mission can self-report “validation/build/background waiter running” and then become acknowledged. Treat that as an observability/lifecycle smell, not completion. acknowledged is terminal: never call resume_mission for it (the API intentionally accepts only interrupted, blocked, or failed). First reconcile registered processes, durable jobs, GitHub artifacts, and any active successor. If productive background work still has a durable owner, attach a callback/waiter and do not duplicate it. If there is no external artifact yet and no live owner, launch exactly one bounded replacement against the preserved branch/worktree after the normal ownership preflight; carry forward immutable receipts and require it to inspect the actual process/log state before rerunning anything. Do not tell Thomas the issue is fixed from “implementation done, build running”; live GitHub/API state wins.

False TurnComplete around detached builds. A native TurnComplete can still report that a Lean/build process is live after that process has died without an exit-status file or target artifact. Before treating the terminal message as a legitimate external wait, run one strict read-only Ask inspection of the preserved worktree. Reduce it to: branch/HEAD, changed-file names, process-group liveness, PID/status/log paths, terminal exit code if recorded, and expected artifact presence. Never request argv, environment, raw logs, or credentials. If the group is live, preserve it and dispatch nothing. If it is gone with no status/artifact, classify the predecessor as terminal-without-validation, preserve its WIP, and launch exactly one bounded existing-branch continuation after normal duplicate/resource preflight. The continuation must reuse/inspect the preserved diff, confirm no process remains before starting one bounded build, write durable PID/exit-status/log artifacts, and inherit all no-merge/no-force-push/no-new-PR constraints. Patch the tracker immediately: terminal predecessor as history, verified continuation as the sole live owner.

Never hand /tmp state to a successor mission. A mission's /tmp may be a private nspawn tmpfs and is not a durable cross-mission namespace. A path such as /tmp/pr27-univ can disappear or resolve to unrelated state in the successor, even when both missions target the same named workspace. Persist handoff worktrees, PID/status/log receipts, and patches beneath the mounted workspace (for example /workspaces/mission-<id>/... or a repository-local .paloma/attempts/<id>/ directory), and record the remote branch plus immutable head SHA. Before a successor trusts any handed-off path, require it to verify repository identity (git remote get-url origin), PR number/head, branch, and expected receipt manifest. If the predecessor used /tmp, treat that path as unavailable: reconstruct from the exact remote PR branch/commit and durable receipts instead of inspecting or resuming it.

For GitHub-producing missions, check the target fork/repository in parallel with mission status: branch existence, commit SHA, fork PR, and upstream PR are stronger evidence than a transcript saying work is local or tests are running. If none exists, report “no verifiable artifact yet,” not “nearly done.”

Recovery ladder when workspace inspection is needed:

1. Use Ask with sandbox=true and a strict read-only process/log/artifact prompt.
2. If sandbox creation is unavailable, retry Ask without sandbox but keep the prompt explicitly read-only: no reruns, writes, commits, pushes, or secret/env output.
3. If the read-only Ask itself times out, do not repeatedly poll or infer progress. Resume the original mission once with an exact bounded instruction: locate the existing PID/log without restarting blindly, capture exit/output, fix or continue, and do not terminate again until it returns a verifiable branch/commit/PR or an exact blocker.
4. Re-read the exact mission and external GitHub state after resumption. active proves only that the continuation was accepted; it does not prove execution or test success.

Benchmark preflight transport parity

A Python urllib.request preflight can receive HTTP 403 from WAF/Cloudflare while curl with the same valid routing credential receives HTTP 200; invalid proxy auth may instead return 401. Do not classify this divergence as a bad key or consume a provider retry. Preserve the default-client 403 as preflight_client_blocked, then run one bounded client correction with a stable explicit User-Agent and Accept: application/json. The corrected gate must cover both GET /models and a minimal native-tool POST /chat/completions, recording returned model, tool-call shape, and usage without secrets. Probes and the harness should share transport, User-Agent, and relevant headers to prevent false auth diagnoses.

Benchmark/result/planning missions — Ask can verify artifacts when transcripts lag

For long benchmark/result campaigns, a mission may be acknowledged or active with stale short_description and empty or truncated get_mission_events, while the real run artifacts have finished. The same pattern applies to planning/roadmap missions: the platform lifecycle can remain active even after the agent has written a usable output/*.md roadmap and marked its internal todos complete. Before reporting stale progress or blocking downstream dispatch, use the non-interrupting Ask side-channel to inspect artifacts/logs read-only. Use POST /api/control/missions/:id/ask (or /ask/stream) as the non-interrupting sidecar; retry without sandbox if isolated worktree creation fails.

Cron-monitor pitfall — exact mission failed, fallback mission fixed it. When a scheduled monitor is given one mission ID, still inspect the surrounding project/track/recent missions before reporting FAILED from that one ID. Provider/auth/DNS failures often trigger redispatch/fallback missions with tags such as redispatch-after-*, fallback-after-*, or the same track/github_pr. If the exact watched mission failed before doing work, check list_missions(project=..., limit=10) / active missions for successors, read their final transcripts, and verify any PR/merge via GitHub/API before alerting Thomas. Report the useful terminal state of the workstream (fixed/merged/blocked), while naming the original mission's transport failure as context.

Recommended pattern:

1. Read the historical mission history via REST/MCP to capture last known counters, artifact/worktree paths, and the original deliverable shape.
2. Ask side-channel on the relevant mission with a strict read-only prompt. For benchmark/result missions, verify canonical task count, distinct task refs, run.json statuses, pass/fail counts, active locks/process markers, and whether aggregate summaries are stale. For roadmap/planning missions, verify whether the expected artifact exists (for example output/v0.2-roadmap.md), whether it contains the requested sections/backlog/prompts, and whether the mission has real blockers versus a stale lifecycle state.
3. If a usable roadmap artifact is verified even while the mission status remains active, treat it as sufficient to synthesize decisions and dispatch non-duplicate downstream workers; report the lifecycle mismatch explicitly instead of waiting indefinitely.
4. If raw artifacts are complete but committed summaries are stale, launch a separate publication/aggregation mission that is explicitly forbidden from rerunning model/API tasks and only regenerates/validates/packages artifacts.

Sandboxed.sh has a separate Ask/copilot lane used by the web/iOS UI:

• POST /api/control/missions/:id/ask — synchronous copilot answer
• POST /api/control/missions/:id/ask/stream — SSE streaming copilot answer
• GET /api/control/missions/:id/ask/threads — list copilot threads
• GET /api/control/missions/:id/ask/threads/:thread_id — read a copilot thread

The server-side contract is that Ask routes are mission-scoped but run in an independent lane: they do not acquire the harness lock, do not enqueue into the mission message queue, and do not write to mission_events. The frontend calls this the “non-interrupting sidecar co-pilot”. Use it for questions like “what exact progress counters can you infer from logs/files?” while the main agent is building or running a long batch.

As of the MCP surface documented above, there may be no ask_mission wrapper. If the user asks for the side-channel and the MCP lacks it, either:

1. call the REST endpoint directly with a sandboxed.sh JWT/API token if available in the environment/config, or
2. report the MCP gap and fall back only to read-only tools (get_mission, get_mission_events, health/diagnostics) — do not silently use send_message_to_mission.

Ask routes are POST /api/control/missions/:id/ask and /ask/stream; they do not acquire the harness lock.

Pitfall — get_mission is read-only but not copilot. It can safely inspect status/history without waking the agent, but it only returns persisted mission state and self-reports; it cannot run workspace/log inspection like the Ask copilot can.

Pitfall — active only proves dispatch, not execution. A newly launched mission can show status=active while get_mission_events(view='transcript') and even view='all' return []. In that state, do not tell the user “it works” or that the external service/API has been reached; all you know is that the platform accepted the mission and attached the prompt. For “does it work?” verification, require at least one of: a transcript/event showing the agent ran a command, a progress short_description based on actual work, a final report, or an independently verifiable artifact/PR/log. If there are no events after relaunch, report it as “accepted/active but no execution evidence yet” and check queue/capacity/backends before debugging the task itself.

Ghost-active completion and corrective handoff

A mission can remain active after it already completed its actual work, while a later send_message_to_mission only returns queued: true and produces no new transcript/history event or workspace execution. Treat queued as delivery acknowledgement, not as proof of a live corrective owner.

A related long-tool-call failure occurs when app-server thread/resume reconnects but leaves a persisted tool_call without a matching tool_result, even though its child process is gone. Classify this as INFRA/transport reconciliation failure rather than model failure; steer once from the preserved checkpoint, prohibit expensive-command replay, and require detached process groups plus PID/status/log artifacts and sub-60-second polls. If the steering produces no new execution event in a short bounded window, cancel without workspace cleanup and launch exactly one replacement against the preserved artifact paths. Preserve the checkpoint, do not replay expensive commands, and require detached process groups plus PID/status/log artifacts.

For a PR controller that sees this state:

1. Inspect the transcript/final report and, where needed, use a strict read-only Ask sidecar to establish the branch/HEAD, uncommitted state, and completed validation.
2. Reconcile the live PR head, CI, REST inline comments, and GraphQL threads. If later current-head findings are outside the completed scope, the old worker does not cover them.
3. Send one narrow continuation only as a low-risk recovery attempt, then verify that it appears in mission history/transcript or produces execution evidence after a short delay.
4. If it remains ghost-active, cancel the stale owner before launching a replacement. Preserve its verified validation result in the tracker; never leave two active writers on the same branch.
5. Launch exactly one replacement for the exact current-head findings on the mandated model route, with explicit no-merge/force-push guardrails. Verify the exact mission and active list after launch, then replace the stale tracker owner with the actual one.

This prevents a finished validation worker from silently leaving actionable review threads uncovered.