Ops Pipelines

Ops Pipelines

Monitor Boundless service deployments running through AWS CodePipeline / CodeBuild in the ops account. Designed for the post-merge workflow: track a commit through staging, surface failures with build logs, and prompt the user to approve the production rollout.

Setup

Read network_secrets.toml from the repo root and extract [aws.ops] (access_key_id, secret_access_key). These are read-only and can query CodePipeline / CodeBuild / CloudWatch but cannot approve, start, or retry pipelines. If the file is missing, point the user at the Boundless runbook.

export AWS_ACCESS_KEY_ID="..."
export AWS_SECRET_ACCESS_KEY="..."
export AWS_DEFAULT_REGION="us-west-2"

All Boundless pipelines live in us-west-2 in account 968153779208 (BoundlessOps).

Pipelines

Pipeline definitions live in infra/pipelines/pipelines/ — read that dir to see what's deployed, branch config, stage layout per service. The boundless repo's pipelines are the l-* ones; signal-pipeline, kailua-order-generator-pipeline, zeth-requestor-pipeline come from different repos.

Discover the live list any time:

aws codepipeline list-pipelines --query 'pipelines[].name' --output table

Typical stage layout: Source → DeployStaging (parallel CodeBuild per chain) → DeployProduction (manual approval, then parallel CodeBuild per chain). l-prover-ansible-pipeline adds a DeployNightly stage between staging and production.

CodeBuild project names have a Pulumi resource hash suffix (l-indexer-staging-167000-build-aee3645); always derive the suffix from pipeline state, never hardcode it.

Core workflows

1. Monitor a deployment for a freshly merged commit

Primary use case. Given a commit SHA (or PR), find the pipeline executions for that SHA and poll until staging finishes.

Resolve the SHA if needed:

gh pr view <number> --json mergeCommit --jq '.mergeCommit.oid'

Find the matching execution per pipeline (executions store the SHA in sourceRevisions[0].revisionId):

SHA="..."
for P in $(aws codepipeline list-pipelines --query 'pipelines[?starts_with(name, `l-`)].name' --output text); do
  EXEC=$(aws codepipeline list-pipeline-executions --pipeline-name "$P" --max-items 20 \
    --query "pipelineExecutionSummaries[?sourceRevisions[0].revisionId=='$SHA'] | [0].pipelineExecutionId" \
    --output text)
  echo "$P -> $EXEC"
  sleep 1
done

Default to all l-* pipelines unless the user specifies a service. Skip l-prover-ansible-pipeline unless the change touched ansible/ or infra/cw-monitoring/.

If the SHA's execution is Superseded, a later commit took over and that SHA will not deploy to prod on its own. Call this out — common source of confusion when merging PRs back-to-back.

Use the Monitor tool to poll in the background so the user can keep working while staging runs (CodePipeline deployments take 10–30+ minutes). The Monitor tool runs a script in the background and feeds each output line back, so the agent can interject as soon as a stage transitions.

Run one Monitor per tracked pipeline with a script that prints a status heartbeat every 30s and exits on a terminal event. Use the execution's overall status from get-pipeline-execution, and get-pipeline-state to detect the approval gate (filtered to the inbound exec at DeployProduction so a newer superseding exec doesn't trigger a false approval signal):

PIPELINE="l-indexer-pipeline"
EXEC="<pipelineExecutionId from step above>"
while true; do
  STATUS=$(aws codepipeline get-pipeline-execution \
    --pipeline-name "$PIPELINE" --pipeline-execution-id "$EXEC" \
    --query 'pipelineExecution.status' --output text 2>/dev/null || echo Unknown)
  echo "$(date -u +%H:%M:%SZ) $PIPELINE exec=$EXEC status=$STATUS"
  case "$STATUS" in
    Succeeded)
      echo "DONE pipeline=$PIPELINE exec=$EXEC"; break ;;
    Failed|Stopped|Cancelled|Superseded)
      echo "ALERT pipeline=$PIPELINE status=$STATUS exec=$EXEC"; break ;;
    InProgress)
      APPROVAL=$(aws codepipeline get-pipeline-state --name "$PIPELINE" --output json \
        | jq -r --arg E "$EXEC" '
          .stageStates[]
          | select(.stageName=="DeployProduction"
                   and .inboundExecution.pipelineExecutionId == $E)
          | .actionStates[] | select(.actionName=="ApproveDeployToProduction")
          | .latestExecution.status // empty' | head -1)
      if [ "$APPROVAL" = "InProgress" ]; then
        echo "READY-TO-APPROVE pipeline=$PIPELINE exec=$EXEC"; break
      fi ;;
  esac
  sleep 30
done

Each tracked pipeline gets its own Monitor (run them in parallel — the boundless ops account handles the call rate fine at 30s intervals). React when a line starting with ALERT, READY-TO-APPROVE, or DONE arrives:

• READY-TO-APPROVE → prompt the user to approve production (workflow 2).
• ALERT ... Failed → surface the failure (workflow 3).
• ALERT ... Superseded → a newer commit took over; tell the user this SHA will not deploy to prod on its own.
• DONE ... Succeeded → pipeline fully complete (rare without approval).

If the user cancels the run, moves on to unrelated work, or asks to stop monitoring, cancel the monitors — they cost API calls and clutter context. If the Monitor tool isn't available, fall back to manual polling with get-pipeline-state every 30s.