Live AI Pipelines

Live AI Pipelines

Use this skill when people need to inspect useful work while later work is still running, or reopen a run after interruption. It owns logical stages, artifacts, progress, recovery, and publication—not a particular queue, database, or provider SDK.

Pair with ai-engineering when model request/retry/rate-limit behavior is itself the problem. Pair with a provider/runtime skill only for that platform’s current API behavior.

Useful defaults

• Keep canonical complete artifacts separate from provisional previews.
• Give artifacts and independently resumable items stable IDs.
• Make the UI a projection of stored results and events, not the source of workflow truth.
• Publish a complete new snapshot rather than exposing a half-rendered site.
• Let core output publish with optional enrichment visibly pending unless the user makes that enrichment a completion requirement.

Workflow

1. Choose the level of machinery

Identify deterministic preparation, independent fan-out, fan-in synthesis, projection, and audit/publication. A local runner with atomic files, JSONL events, and polling/SSE is often enough. Add a queue, database, ownership lease, or process supervision only when multiple workers, long detachment, or safe handoff actually needs it.

For each stage, choose its item key, input/output contract, retry behavior, and whether a partial result is safe to show. Feed fan-in synthesis a bounded normalized projection of completed artifacts, not the raw corpus and every intermediate response.

For each await-all, drain, or batch-wide join, identify the shared result its downstream work actually requires. Start item-local projection, rendering, and validation when that item’s dependencies are accepted. Keep snapshot assembly and publication joins where consistency requires them. Batched publication does not imply batched preparation.

2. Define complete versus preview data

Validate a response before writing a canonical artifact. Write it atomically, then emit a completion/failure event and update a replaceable status snapshot. Streamed token fragments and drafts may support a preview, but do not give them canonical links or present them as complete facts.

For a costly broad run, a small representative calibration and per-request-class budget notes can prevent systematic truncation. They are planning aids, not a prerequisite for an exploratory pilot.

3. Make resume proportional to the risk

For a simple local job, derive completion from validated result artifacts and idempotency keys. For a job that can outlive its caller or be resumed by another process, also maintain an item index, owner/lease, heartbeat, and explicit cancellation/detach policy. On cancellation, stop new admission, preserve valid work, and classify unfinished items so a later run can decide whether to retry them.

Record which input fields invalidate which stages. A transcript change need not invalidate a biography; a prose change need not invalidate media. Reuse acceptance and validation evidence only when the relevant source, content, contract and artifact versions match. Preserve accepted content while replacements are held. For entity releases, use one completion ledger with receipts behind it rather than competing status systems. Reconcile ambiguous delivery before another submission.