Pick Integ

Integ Test Picker

Decide which integration tests to run right now, ranked by two signals: how stale the last run is (from the committed ledger) and whether recent changes touch the area a test exercises. It RECOMMENDS only; the orchestrator runs the chosen tests via /run-integ.

The ranking is a running ORDER, not a budget (AGENTS.md → "Cost is not a tiebreaker"): when several tests plausibly cover the touched code, run all of them, and prefer a broad-set fixture whenever the change is cross-cutting. Priorities decide what runs FIRST; a plan too long for one session is handed forward per "Running a large plan across sessions" — never silently truncated, and the wrap-up names exactly which tests were not run.

Inputs

• Optional positional base-ref (default origin/main): the diff base. Use the last release tag when picking post-merge; origin/main for a branch.

Data sources

1. Ledger docs/_generated/integ-last-run.tsv (committed, one row per test; written by /run-integ on every run):

   • Stale: age > the integ-destroy gate TTL (14 days) — past that the marker expires, so a clean result no longer proves today's AWS behavior.
   • Expiring soon: age 12–14d. The ledger accumulates in sweep-shaped cohorts (one sweep stamps many rows with one timestamp), so a strict >14d boolean answers "nothing" or "everything" — always report this tier.
   • Failing: result == FAIL — always a candidate.
   • Never-run: a fixture directory with NO ledger row — highest staleness.

2. Recent changes: git diff <base-ref>...HEAD --name-only, mapped to tests via the table below.

Steps

1. Discover the universe + ledger state:

   LEDGER="docs/_generated/integ-last-run.tsv"
   now=$(date -u +%s)
   # Union-merge can leave duplicate rows — LAST row per test wins. The
   # header is MULTIPLE `#` lines; skip them all (a single NR==1 guard let
   # later header lines through as phantom stale rows).
   DEDUPED="$(mktemp)"
   awk -F'\t' '/^#/{next} {last[$1]=$0} END{for (t in last) print last[t]}' "$LEDGER" > "$DEDUPED"
   # never-run: fixtures with no ledger row
   comm -23 \
     <(ls -d tests/integration/*/ | sed 's#tests/integration/##;s#/##' | sort) \
     <(awk -F'\t' '{print $1}' "$DEDUPED" | sort)
   # resolve each row's age ONCE, then slice for the views below.
   AGES="$(mktemp)"
   awk -F'\t' -v now="$now" '{
     cmd="date -u -j -f %Y-%m-%dT%H:%M:%SZ \""$2"\" +%s 2>/dev/null || date -u -d \""$2"\" +%s 2>/dev/null";
     cmd | getline t; close(cmd);
     printf "%s\t%d\t%s\t%s\n",$1,int((now-t)/86400),$3,$6
   }' "$DEDUPED" > "$AGES"
   
   # age histogram — makes a clustered cohort visible even when NOTHING is >14d
   awk -F'\t' '{print $2}' "$AGES" | sort -n | uniq -c |
     awk '{printf "%6d tests at age %sd%s\n",$1,$2,($2>14?"   <-- STALE":($2>=12?"   <-- crosses the TTL in "(15-$2)"d":""))}'
   
   # stale (>14d) or failing:
   awk -F'\t' '$3=="FAIL" || $2>14 {printf "%s\tage=%sd\tresult=%s\t%s\n",$1,$2,$3,$4}' "$AGES"
   
   # expiring soon (12-14d, not already failing) — the cliff cohort:
   awk -F'\t' '$2>=12 && $2<=14 && $3!="FAIL" {printf "%s\tage=%sd\texpires in %dd\n",$1,$2,15-$2}' "$AGES"
   
   # one-line cliff summary for the step 4 header:
   awk -F'\t' '$2>=12 && $2<=14 {n[15-$2]++} END{for (d in n) printf "%d\t%d tests expire in %dd\n",d,n[d],d}' "$AGES" |
     sort -n | cut -f2-

   (The date line handles both BSD/macOS -j -f and GNU -d.)

2. Map recent changes to tests (a changed path pulls in the listed tests; when in doubt, include the broad set):

   Changed path	Integ tests to run
   src/deployment/deploy-engine.ts (and deploy-engine/*.ts, deploy-value-equality.ts, label-routing.ts, split out of it), src/deployment/intrinsic-function-resolver.ts (and the intrinsic-resolver/*.ts modules split out of it), src/deployment/retry.ts, src/deployment/retryable-errors.ts (and retryable-errors/*.ts), src/deployment/rollback-executor.ts (and rollback-executor/*.ts), src/analyzer/dag-builder.ts, src/analyzer/template-parser.ts, src/cli/commands/{deploy,destroy,destroy-runner}.ts, src/provisioning/{provider-registry,register-providers}.ts	BROAD set (bench-cdk-sample, lambda, microservices, multi-resource, multi-stack-deps, drift-revert, drift-revert-vpc, remove-protection, export) — cross-cutting deploy/destroy. Also cross-stack-references for any cross-stack/exports change.
   src/state/**, src/state/export-index-store.ts	schema-v<N>-to-v<N+1>-migration (latest), cross-stack-references, import-value-strong-ref
   src/provisioning/providers/<Svc>*	the matching service integ (iam-managed-policy, dynamodb-*, s3-*, rds-*, kms-encryption, stepfunctions, eventbridge, sns-sqs-event, cognito, wafv2, route53, …). Custom Resource changes → custom-resource-provider.
   src/provisioning/cloud-control-provider.ts	cc-api-fallback, cc-api-fallback-transitions, recreate-via-cc-api
   src/local/**, src/cli/commands/local-*.ts	the matching local-* test; a cross-cutting src/local/ change → at least local-invoke + local-start-api.
   src/cli/commands/{export,import,retire-cfn-stack}*	export, export-nested-stack, migrate-from-cfn, import-nested-stack, import-value-strong-ref
   src/synthesis/macro-*	macro-expansion
   package.json (cdk-local bump)	local-* cluster (the bump's blast radius)

   Mark every name an agent session cannot run, and offer a substitute. A fixture drives itself from verify.sh, or from run.sh invoked directly; with neither, /run-integ falls back to a flow needing a bare cdkd deploy, which the harness refuses — legal only when a human drives the shell. Such a name is a MAINTAINER-only recommendation, never a lane's. Several BROAD-set entries are in this state, so derive the split rather than recalling it:

   for t in NAME1 NAME2; do   # the names about to be recommended
     d=<LANE_TREE>/tests/integration/$t
     [ -f "$d/verify.sh" ] || [ -f "$d/run.sh" ] || echo "MAINTAINER-ONLY: $t"
   done

3. Rank the union of {changed-area} ∪ {failing} ∪ {stale >14d} ∪ {expiring soon} ∪ {never-run}:

   • P0: changed-area AND (stale OR failing OR never-run).
   • P1: changed-area but recently-green — verify no regression.
   • P2: not changed-area but stale / failing / never-run / expiring soon — coverage hygiene. Prefer the BROAD set plus a spread of providers, and ORDER them rather than capping: a long tail is handed forward, and whatever is not run is named in the wrap-up. An expiring-soon test ranks below an already-stale one and above a recently-green one; within a tier, oldest first. Bias up for AWS-coupled, deletion-sensitive, multi-resource paths; pure docs / test / skill changes often need NO integ.

4. Render the plan:

   Recommendation: run N integ tests (P0: ..., P1: ..., P2: ...)
   Base: <base-ref> (<changed-file-count> files changed)
   Ledger: <count> stale (>14d), <count> failing, <count> never-run,
           <count> expiring soon (12-14d) — <N> tests expire in <M>d
   Age histogram:
     <the uniq -c output from step 1, cliff rows annotated>
   
   P0 (changed + stale/failing/never-run):
     /run-integ <name>    # <why: which changed path + age/result>
   P1 (changed, recently green):
     /run-integ <name>    # verify no regression in <area>
   P2 (coverage hygiene):
     /run-integ <name>    # stale <age>d / expires in <M>d / never-run
   
   Not scheduled this pass (recently green + untouched): <count> tests — list a few.
   (A count, not a cap: these are DEPRIORITIZED, not excluded.)

   Never render "zero stale" on its own — if a cohort sits at 12–14d, the headline is the cliff ("0 stale, but 105 tests expire in 1d").