Minion Orchestrator

Minion Orchestrator

Contract

Minions is a Postgres-native job queue for durable, observable background work. This single skill handles two lanes:

• Deterministic shell jobs (gbrain jobs submit shell ...)
• LLM subagent jobs (gbrain agent run ...)

When to route to Minions: durable, observable work that must survive restarts, fan out across many parallel tasks, or persist across sessions. Routing policy is defined in skills/conventions/subagent-routing.md — the project default is pain_triggered (native subagents first, Minions after specific pain signals fire); Mode A (all-through-Minions) is opt-in.

Guarantees:

• Jobs survive gateway restart (Postgres-backed)
• Every job has structured progress, token accounting, and session transcripts
• Running agents can be steered mid-flight via inbox messages
• Jobs can be paused, resumed, or cancelled at any time
• Parent-child DAGs with configurable failure policies

Durable-execution doctrine (routing convention the agent follows — nothing mechanically enforces it; see "Durable execution" below):

• Operations expected to exceed ~2 minutes route through the capability ladder, never a bare background shell that dies with the session.
• A deadman check verifies the result was REPORTED to the user, not merely that the process exited — a swallowed completion event looks identical to a dead job from the user's side.
• The deadman is second-line insurance, not a delivery guarantee: it can itself die before firing (scheduler restart, deleted entry). Backstop: on the next turn, sweep gbrain jobs list --status active and recent completions for overdue work whose result never reached the user.
• Deadman checks are idempotent. A double-fire, or a fire that races the normal completion report, produces silence — it checks reported-state first and tags any recovery report with the job ID so duplicates are detectable.
• A checkpoint or progress file is trusted only with a freshness check. A stale checkpoint reads as "still running" forever; compare its timestamp against the expected progress interval and against gbrain jobs get <id>.

Route the Request: Shell Job vs Subagent

If intent is ambiguous, ask one clarification: "Do you want a deterministic shell command job, or an LLM agent job?"

Shell Jobs (Deterministic Scripts)

Use for reproducible command execution, ETL steps, cron work, and scriptable tasks where no LLM reasoning loop is needed.

Preconditions (read before submitting your first shell job)

• The worker must be started with gbrain jobs work --allow-shell-jobs (equivalently GBRAIN_ALLOW_SHELL_JOBS=1 exported on the worker; a .env in the worker's directory cannot set it). The shell handler is always registered but guarded: an unflagged worker that claims a shell job dead-letters it immediately (UnrecoverableError, straight to dead, no retries) with the flag named in error_text. A job that sits in waiting means NO worker is running at all — check gbrain jobs supervisor status. Gate lives in src/core/minions/handlers/shell.ts.
• Security: flipping GBRAIN_ALLOW_SHELL_JOBS=1 authorizes arbitrary command execution on the worker. On a shared queue, this is a remote code execution surface. Treat as privileged infrastructure authorization.
• Execution mode — pick one:
  • Postgres + daemon: gbrain jobs work runs a persistent worker that claims and executes jobs from the queue.
  • PGLite + --follow: gbrain jobs submit ... --follow runs inline. The daemon mode is not available on PGLite (exclusive file lock). See docs/guides/minions-shell-jobs.md.
• MCP boundary: shell-job submission is CLI-only. submit_job name="shell" over MCP throws an OperationError with code permission_denied; generic remote submission accepts only sync, import, lint, and lint-fix. Agents CAN observe shell jobs via get_job / list_jobs / get_job_progress (not protected), but cannot submit them. Operator or autopilot submits; agent observes.
• Verify setup: after configuration, run gbrain jobs stats (CLI) to confirm the worker is registered and consuming the queue.

Submit (CLI, operator or autopilot)

Shell jobs take their command via --params as a JSON object with cmd (string) or argv (array), plus cwd and optional env.

Command string form:

gbrain jobs submit shell --params '{"cmd":"echo hello","cwd":"/abs/path"}'

Argv form (no shell expansion):

gbrain jobs submit shell --params '{"argv":["bash","-lc","echo hello"],"cwd":"/abs/path"}'

Inline execution on PGLite or any one-shot deployment:

gbrain jobs submit shell --params '{"cmd":"echo hello","cwd":"/tmp"}' --follow

Queue/lifecycle flags exposed by gbrain jobs submit --help: --queue, --priority, --delay, --max-attempts, --max-stalled, --backoff-type, --backoff-delay, --backoff-jitter, --timeout-ms, --idempotency-key, --dry-run.

Monitor (agents or operator)

These operations are MCP-callable and safe for agent use:

list_jobs --name shell --status active
get_job ID
get_job_progress ID

Check structured result fields (exit code, stdout/stderr tails, attempts, timings) from get_job. Use get_job_stats (MCP) or gbrain jobs stats (CLI) for the worker/queue health dashboard incl. the wedged-queue signal.

Control (MCP-callable)

cancel_job id=ID
replay_job id=ID

replay_job is not protected — only shell submission is. Agents can cancel or replay a shell job without CLI access.

Use idempotency keys for recurring shell workloads to avoid duplicate runs.

Subagent Jobs (LLM Orchestration)

Use for open-ended reasoning, tool-using research, and fan-out synthesis.

User-facing entrypoint: gbrain agent run <prompt> is the canonical way to submit subagent work. It handles the elevated-trust plumbing — subagent and subagent_aggregator are protected job names. Remote agents use the dedicated submit_agent operation with their configured source, tools, and slug binding; generic submit_job does not accept these job names.

Phase 1: Submit

gbrain agent run "Research Acme Corp revenue" --tools "search,query"

--tools accepts a comma-separated subset of BRAIN_TOOL_ALLOWLIST (see src/core/minions/tools/brain-allowlist.ts): query, search, get_page, list_pages, get_backlinks, traverse_graph, list_link_sources, resolve_slugs, get_ingest_log, put_page, add_timeline_entry, get_recent_salience, find_anomalies. Attachment tools and local-only operations are unavailable. Update old bindings before resubmitting jobs that reference removed tools. Remote bindings must be nonempty; an explicit empty local tool list grants no tools.

For parallel work with a fan-out manifest:

gbrain agent run --fanout-manifest companies.json

The manifest describes N children + 1 aggregator. Each child runs name="subagent" under the hood; the aggregator runs name="subagent_aggregator" and claims AFTER every child terminates. See src/core/minions/handlers/subagent.ts and src/core/minions/handlers/subagent-aggregator.ts.

Flags (from src/commands/agent.ts):

• --subagent-def <name> — named subagent definition
• --model <id> — override model
• --max-turns <N> — cap the LLM loop
• --tools <csv> — allow-listed brain tools (see above)
• --timeout-ms <N> — hard timeout per job
• --fanout-manifest <file> — N children + 1 aggregator
• --follow / --no-follow — stream logs + wait (default on TTY)
• --detach — submit and return immediately

Queue/priority/retry tuning is not exposed by gbrain agent run; submit the raw subagent handler via gbrain jobs submit (requires CLI trust) if you need those knobs.

Admission control (v0.46.11.0). Identical parentless subagent submits (same owner lane, payload, and execution options) coalesce onto the existing waiting job: gbrain agent run prints coalesced with the matched job id, and the submit_agent MCP response carries coalesced: true. Treat that as success — monitor the matched id, do NOT resubmit. Jobs still waiting after the TTL (48h default for subagent; minions.ttl_waiting_hours.<name>) are cancelled with reason prefix waiting_ttl_expired. If an operator has configured a waiting quota (minions.quota_max_waiting.<name>), a submit past the cap returns a structured, retryable rate_limited error — back off and check gbrain jobs stats for a DIVERGENT QUEUE line before retrying.

Phase 2: Monitor

list_jobs --status active          # MCP — what's running?
get_job ID                         # MCP — full details + logs + tokens
get_job_progress ID                # MCP — structured progress snapshot
gbrain jobs stats                  # CLI — queue health dashboard
gbrain agent logs ID --follow      # CLI — streaming transcript + heartbeat

Progress includes: step count, total steps, message, token usage, last tool called.

Phase 3: Steer

Send a message to redirect a running agent:

send_job_message id=ID payload={"directive":"focus on revenue, skip headcount"}

The agent handler reads inbox messages on each iteration and injects them as context. Messages are acknowledged (read receipts tracked).

Only the parent job or admin can send messages (sender validation).

Phase 4: Lifecycle

pause_job id=ID                    # freeze without losing state
resume_job id=ID                   # pick up where it left off
cancel_job id=ID                   # hard stop
replay_job id=ID                   # re-run with same or modified params
replay_job id=ID data_overrides={"depth":"deep"}  # replay with changes

All lifecycle ops are MCP-callable.

Phase 5: Review Results

get_job ID                         # result, token counts, transcript

Token accounting: every job tracks tokens_input, tokens_output, tokens_cache_read. Child tokens roll up to parent automatically on completion.

Durable execution (operations >2 minutes)

Background shells die silently: session compaction, harness restart, tool timeout. Long gbrain operations (extract all, embed --stale, a full sync --all) routinely run 10-60 minutes — past every one of those ceilings. And even when the work survives, the completion event can be swallowed (worker restart, dropped notification), leaving the user staring at silence while a finished result sits unreported. Durable execution covers both halves: the work survives, and the report provably lands.

Route any operation expected to exceed ~2 minutes through the highest rung of this ladder the deployment supports. This is a harness-routing convention the agent follows, not a mechanical guarantee — nothing stops a bare background shell except this skill saying don't.

Paid work gets consent before it is submitted. A job runs without a terminal, so a paid command (embed --stale, or anything else that calls a model provider) stops with exit 3 and a consent payload unless it already carries the user's approval. Get that approval first, in the conversation: run the command's preview (for example gbrain embed --stale --dry-run), relay the estimate, and only after the user agrees put the approval in the submitted command (gbrain embed --stale --yes --max-usd <cap> with the cap they approved), or rely on a standing approval the user set with gbrain config set consent.preapprove.paid.max_usd_per_run <usd>. Never add --yes without the user's answer. When a job ends with exit 3 or the confirmation_required code, relay its message to the user and stop; don't resubmit it with --yes.

Rung 1 — Minion job + deadman (Postgres + worker)

Requires: Postgres engine, a running gbrain jobs work worker, and — for the shell lane — a worker started with --allow-shell-jobs (or GBRAIN_ALLOW_SHELL_JOBS=1 exported on it). All the Preconditions above still hold: the flag defaults OFF, shell submission is CLI-only across the MCP trust boundary, and PGLite has no worker daemon (see Rung 3). Nothing in this section loosens that contract.

1. Submit as a job so the work survives restarts:

   gbrain jobs submit shell \
     --timeout-ms 3600000 \
     --params '{"cmd":"gbrain extract all","cwd":"/abs/path","inherit":["database_url"]}'

   Work that needs LLM judgment goes through the subagent lane (gbrain agent run, above) instead — a submitted agent job you never check on is the same bug as an unwatched shell job.

2. Arm a deadman in the same action block (pattern below). Submitting without arming is the classic half-fix: the job survives, the silence doesn't.

Rung 2 — cron-checked progress file (no agent-waking scheduler)

When no scheduler can wake the agent but the host has a plain crontab (or the work runs outside the queue entirely): make the operation write a progress/heartbeat file as it advances (or rely on get_job_progress for queue jobs), and register a recurring host-cron check that compares the file's freshness against the expected progress interval. The agent also checks it at the start of the next turn. A checkpoint that stops advancing means stalled, not "still running" — the freshness comparison is the entire value of this rung.

Rung 3 — foreground + output buffering (PGLite / no worker / no scheduler)

Run the operation inline in the foreground — on PGLite that's gbrain jobs submit ... --follow, or just the raw command — and buffer all output to a file, reading bounded slices, per skills/conventions/exec-output.md. An empty tool result after a long command is truncation, not a dead shell. This rung has no silent-death insurance, so keep the operation in the foreground and stay with it; backgrounding here recreates the exact failure the ladder exists to prevent.