Cao Workflow

CAO Workflows

A CAO workflow is a Python script you write, validate, and — only after asking the user — run through cao workflow run. Each script drives one or more agent steps through CAO's shared substrate, so you can fan work out across agents, collect their results, and resume a run that was interrupted.

Your job as an author ends at a validated script file on disk. Authoring does NOT run the workflow. Never claim a workflow ran, or will run, when all you did was write it. Running is a separate step the user must approve (see Lifecycle step c).

When to use

Reach for this skill when the user asks to build or run a multi-step or parameterized workflow — for example:

• "Analyze every file in reports/ and summarize the findings."
• "Run a review pipeline: implement, then review, then verify."
• "Do the same batch job but with a different input directory each time."

If the work is a single one-off agent call, you don't need a workflow. Workflows earn their keep when there are multiple steps, fan-out, parameterization, or a need to resume.

The script API

Author scripts import from the cao_workflow package. This package runs only in the script subprocess and imports nothing from cli_agent_orchestrator.* — it talks to CAO over HTTP. Its public surface:

• step(provider, agent, prompt, *, recovery, step_id=None, timeout=None, **opts) -> StepHandle — run one agent step and declare what re-running it would mean. recovery is keyword-only with no default, so omitting it is a TypeError at the call. See "Declaring a recovery policy" below before you pick a value.
• run_step(provider, agent, prompt, *, step_id=None, timeout=None, **opts) -> StepHandle — the same call, declaring no policy. That is the only difference between the two. A recovery= passed to run_step lands in **opts; the server validates it, the shim does not — see below.
• StepHandle has five fields: .step_id, .terminal_id, .output, .status, and .replayed. .replayed qualifies .terminal_id. When it is True the server returned a stored result and ran nothing, and .terminal_id is the ORIGINAL id — it names a terminal that no longer exists. That flag is the only thing standing between you and reading, writing to, or waiting on a dead id, so check it before you touch .terminal_id.
• get_inputs() -> dict — the run's resolved inputs (see Parameterized workflows). Returns {} when nothing was declared; never raises on absence.
• emit_output(value) — print the run-level CAO_WORKFLOW_OUTPUT: sentinel (the run's return).
• ShimError (and ShimIdentityError, ShimTransportError, ShimHTTPError) — the failure hierarchy step and run_step raise. Failures surface unchanged — the shim never retries. A structured HTTP error with a non-empty string detail.kind prints as run-step returned HTTP <status> (<kind>): <message> (without the message suffix when absent); an unstructured error keeps the original run-step returned HTTP <status> text.

Declaring a recovery policy

recovery= is the author's claim about the step, and nothing more. CAO has no mechanism to prove what a step does to the outside world, so it cannot and does not verify the claim. A recovery policy DECLARES what re-running this step would mean; it never grants permission.

The three values, all of which are statements you are making, not protections you are getting:

"idempotent" grants nothing and protects nothing. It does not make a step safe to re-run; it tells the resume gate that you believe it already is — and wherever the gate would otherwise stop and ask a human, it re-executes the step on your word instead. Declare it on a step that charges a card, sends mail, or files a ticket and CAO will charge the card again, exactly as instructed. If you cannot show the step is safe to repeat, "manual" is the honest declaration.

Omitting a policy is a fourth, distinct state — it is never silently read as "manual". Use run_step for it deliberately: an undeclared step still replays (replay executes nothing), but where the alternative is re-execution it halts for a human.

recovery= on run_step is checked late, not never. run_step has no recovery parameter, so the value rides **opts to the server, which stores it, lets the resume gate honour it, and rejects an unknown value with a 422 — the route types that field as the closed policy enum. What run_step lacks is step()'s client-side check, which refuses a bad value before any HTTP attempt; on run_step a typo instead fails that step mid-run. Neither surface has its value checked by validate (the linter sees the keyword, not its contents), which is why validate reports the run_step form as unenforced-recovery-policy. Use step() to declare, and run_step only to declare nothing.

Lifecycle

Follow every step in order. No step may be skipped — validate is mandatory, and you must ask before running.

a. AUTHOR

Write a .py file to ~/.aws/cli-agent-orchestrator/workflows/<name>.py. The workflow is run by its stem (<name>), so:

• The name must be a bare stem — no path separators, no directory prefix.
• Do not create a same-stem .yaml sibling — a <name>.yaml next to <name>.py collides on the run surface.

b. VALIDATE (mandatory gate)

cao workflow validate ~/.aws/cli-agent-orchestrator/workflows/<name>.py

Fix every finding before proceeding — the lint findings are load-bearing, not style nits:

• import cli_agent_orchestrator is banned. The script runs in a separate subprocess and must reach CAO only over HTTP (the cao_workflow shim). Importing the server package breaks that boundary.
• random / time / datetime / uuid warnings. Resume re-executes the script top-to-bottom and replays journaled step results. Any nondeterministic value computed at the top level will differ on replay and raise ReplayDivergenceError. Keep the script deterministic: derive IDs from inputs, not from the clock or an RNG.
• missing-recovery-policy is a blocking ERROR. A step() call with no recovery= keyword fails validation — the signature requires one and so does the linter. Two related warnings fire without blocking: unverifiable-recovery-policy (a step() call passing **kwargs, so the linter cannot see whether a policy is in there) and unenforced-recovery-policy (a recovery= on run_step, which is honoured at resume and validated by the server with a 422, but is not checked client-side before it is sent). See "Declaring a recovery policy" above.

c. ASK the user — NEVER auto-run

The script tier executes generated Python. Never run a workflow without the user's explicit approval. Present the validated file and ask before doing anything in step d.

d. RUN with an explicit, pre-announced run-id

Announce the run-id before you start so the user can cancel it: "Starting run kb-1 — cancel with cao workflow cancel kb-1."

Choose the invocation by how the run is triggered, because the two paths have very different client-side ceilings:

• cao workflow run (CLI) uses a client socket timeout of ~8820s (~2.45h) — the CLI itself won't give up early.
• workflow_run MCP tool is bounded by the MCP host's own per-tool-call timeout — a host-dependent, much-shorter limit that can drop a long blocking call and lose its return value even though the server run keeps going.

So:

• Short runs: call the workflow_run MCP tool (blocking) and read the result directly.
• Long runs: background the run and poll, rather than blocking on it —

  cao workflow run <name> --run-id <id> --json &

  Backgrounding keeps the run alive server-side without a short MCP host timeout silently dropping the return.