Babysit Runs

Babysit Runs

Operate existing runners toward the authorized outcome at the agreed quality. Prefer targeted repairs over a new scheduler or pipeline rewrite. Use the selected provider/runtime skill when invocation or access needs troubleshooting; use live-ai-pipelines only when recovery architecture needs implementation.

Choose the operating mode

Completion mode: finish the authorized workload using the established implementation, repairing defects as needed.

Pilot-and-improve mode: exercise an existing or modified workflow on representative inputs, inspect execution and output quality, and make bounded improvements before broader rollout. State what is being tested, the quality baseline, resource/spending limits and the completion boundary. A pilot is complete when the requested behavior is demonstrated and material findings are resolved or clearly reported; optional optimization ideas do not keep it running indefinitely.

Adopt, detach and schedule

Inspect actual processes, owners, logs and retained results. Adopt a matching live run rather than launch a duplicate. Discover routine facts from existing metadata and task history instead of re-interviewing the user.

Keep a compact continuation brief in the existing run metadata or monitor prompt:

• Goal, operating mode, scope, checkout, command, owner and run/log/artifact locations.
• Requested model/provider, shared concurrency ceiling, independent transfer limits and applicable resource/cost bounds.
• Agreed quality references, authorized repairs, completion evidence and release or handoff boundary.
• Current progress, retry state, next recovery time and monitoring/reporting cadence.

Check changed or uncertain dependencies before submitting work: launcher permissions, authentication/model access, actual tool configuration and capacity. Reuse recent comparable successful evidence. Record the effective rules and versions where acceptance or replay depends on them; stale historical instructions must not override the current authorized contract. Do not silently substitute models.

Detach substantial runs using the existing manager or simplest reliable mechanism. Verify that execution survives the launching shell/tool and logs remain discoverable.

Create or update one temporary monitor. In Codex, use the automation tool and a heartbeat in the main thread, normally every 2–5 minutes during a pilot or unstable run; reuse a matching automation. Use another scheduler when requested or appropriate. Disclose unavailable scheduling rather than promise future check-ins. Make the saved prompt self-contained using the continuation brief, including intervention authority and stopping conditions, and update it when the run moves or resumes. Persist decisions so wakeups do not depend on conversation memory or a stale PID.

Each wakeup inspects actual ownership, progress, recent traces, errors and representative new results. Periodically use a stronger model within the authorized model and spending scope to spot-check substantive outputs and diagnose trace patterns, especially after model substitutions, implementation changes, repeated failures or unexpected slowdowns. Reuse prior assessments when their inputs are unchanged. Process health and valid schemas do not establish output quality.

Reduce monitoring frequency once execution is healthy. Notify the user of meaningful milestones, material regressions, required external action and completion; stay quiet when unchanged.

Saturate useful work

Fill slots with ready work and advance independent items without whole-batch barriers. Parallelize preparation, LLM calls, transfers and downstream stages where dependencies permit. Do not add inference or busywork merely to occupy capacity.

Apply the allowance across all runners sharing the same constrained resource, rather than giving each runner the full ceiling. Keep independently justified transfer/media/provider caps. Retain established healthy concurrency instead of repeating ramp experiments on every resume.

Identify the current bottleneck, commonly LLM calls or data transfer. Adjust parallelism using comparable accepted throughput, latency, retries, throttling and resource pressure. Downshift when useful performance degrades and recover capacity when healthy. Increase capacity at the demonstrated bottleneck; active-call count alone is not success.

Recover without losing successful work

Each check compares accepted/remaining counts, active/queued stages, oldest work, ownership, recent logs, deliveries and retry state with the previous observation. Quiet logs alone do not establish a stall; use representative durations and other progress signals.

Isolate failed items or surfaces while independent work continues. Before replacement, reconcile retained results and cancel or fence a stale owner. Drain affected active work before activating incompatible runner changes. Preserve successful stages and repair only the failed dimension.

• Established intermittent failure: recover automatically with bounded backoff and jitter. A status code alone does not establish quota exhaustion or permanent denial.
• Explicit authentication, entitlement or quota failure: isolate the affected surface and use normal authorized recovery. If external action is needed, report the concrete requirement.
• Unknown delivery: inspect retained output, request IDs and available provider status/idempotency before resubmission. Preserve attempt history; missing local output does not establish that no result was produced.
• Runner or content defect: fix the demonstrated cause and resume unfinished work, rather than regenerate accepted results.

Use bounded immediate retries, followed by persisted cooldowns for periodic outages. Respect existing budgets. Wakeups must not reset retry history. When repeated recovery produces no useful progress, diagnose or pause the affected work and report evidence instead of maintaining a hot retry loop. Routine recoverable failures should not require another approval interruption.