Release Readiness

<objective>
"I think it's fine" ships the bug that the rollback you never practiced can't undo at 6 PM on a Friday. This skill turns "ready to ship" into something measurable: a go/no-go checklist with evidence for each item, a sub-5-minute smoke suite, a staged rollout with metric-gated promotion, rollback thresholds defined before deploy, and post-deployment verification. Every section gives concrete criteria, not aspirations.
</objective>

---

Discovery Questions

Check .agents/qa-project-context.md first — if it exists, use it and skip anything answered there. Then ask only what's missing.

Release cadence and process:

• How often do you release, and who makes the go/no-go call? (Continuous/daily/weekly vs engineering lead/QA lead/release manager — sets how heavyweight the checklist should be.)
• Is there a release train schedule or is it ad-hoc?
• How many environments exist between dev and production? (staging, pre-prod, canary)

Current state:

• What does the current go/no-go process look like, and is it documented?
• Has a release ever been rolled back? How long did it take? (Reveals whether rollback is real or aspirational.)
• What was the last release incident and its root cause? Any release-blocking bugs right now?

Infrastructure and capabilities:

• Do you have rollback capability, and how long does a rollback take?
• Can you do staged/canary deployments?
• Do you have feature flags, and how are they managed? (Decides flag-based vs infra-level rollout.)
• What monitoring and alerting is in place? Are database migrations reversible?

Team and communication:

• Who is on-call during and after releases?
• How are stakeholders notified, and is there a release communication channel?
• How are release notes generated?

---

Core Principles

1. Release confidence comes from evidence, not feelings

"I think it's fine" is not a go/no-go criterion. Evidence means: all CI pipelines green, smoke tests pass on staging, performance budgets met, no open P0/P1 bugs. If you can't point to data, you're not ready.

2. Smoke tests are the last safety net, not the only safety net

Smoke tests catch catastrophic failures. They are not a substitute for thorough testing throughout the development cycle. If your smoke test suite is the only thing between you and production, you have a process problem upstream.

3. Staged rollouts reduce blast radius

Deploying to 100% of users simultaneously means 100% of users are affected by any bug. Staged rollouts (canary, percentage-based, ring-based) let you catch issues when they affect 1% of users instead of all of them.

4. Rollback criteria must be defined BEFORE release

If you wait until something is on fire to decide whether to roll back, you'll waste critical minutes debating. Define the criteria in advance, relative to baseline: "If error rate exceeds 2x baseline within 15 minutes, we roll back. No discussion needed." Tie the trigger to your DORA targets — a release whose error rate would push you past your change-failure-rate target, or whose recovery would blow your MTTR target, is one the rollback rule exists to stop.

5. Every release is a learning opportunity

Post-deployment verification isn't just about catching bugs. Track what went well, what was slow, what was stressful. Improve the process continuously.

---

Go/No-Go Checklist

Use this as a template. Adapt it to your context. Every item should be verifiable with evidence, not just "I checked." Store the completed checklist as a versioned artifact (e.g. RELEASE-<version>.md or a tracked issue) so sign-off is auditable.

Automated Checks (Must Pass)

• All CI pipelines green — Unit tests, integration tests, E2E tests, type checking, linting
• Smoke test suite passes on staging — Critical user journeys verified in the staging environment
• No open P0/P1 bugs for this release — Check issue tracker, filter by milestone/label
• Performance budgets met — API response times and bundle size within thresholds; Lighthouse CI for frontend releases (skip Lighthouse for API/backend-only releases — it measures page load, not service health)
• Security scan clean — No high/critical vulnerabilities in npm audit / Snyk / Dependabot
• API contract tests pass — No breaking changes to public APIs
• Visual regression tests pass — No unintended visual changes
• Accessibility checks pass — axe-core scan shows no new violations

Manual Checks (Verify Before Go)

• Feature flags reviewed — Document which flags are enabled/disabled in this release; confirm flag states for production
• Monitoring and alerts configured — New features have corresponding alerts (error rate, latency, business metrics)
• Rollback plan documented and tested — Written procedure exists; rollback has been practiced on staging
• Database migrations tested — Tested forward migration; backward migration verified if schema change is reversible
• Third-party dependency changes reviewed — New or upgraded external dependencies checked for breaking changes
• Release notes prepared — Changelog updated, stakeholder-facing summary written
• On-call engineer identified — Named person is available and has context on the release contents
• Communication plan ready — Stakeholders know the release is happening; support team briefed on changes
• No conflicting releases — Other teams aren't deploying simultaneously
• Deploy window confirmed — Not deploying during peak traffic or before a weekend (unless continuous deployment)

Risk Assessment

• Change scope categorized — Small (config change, copy update), Medium (new feature, refactor), Large (architecture change, migration)
• Blast radius estimated — What percentage of users could be affected if something goes wrong?
• Revert complexity assessed — Can this be reverted in <5 minutes? Does reverting require a data migration?

---

Smoke Test Suite Design

What to Include

Smoke tests cover critical user journeys only. If these fail, the application is fundamentally broken.

Typical smoke test suite (5-8 tests):

1. Application health — Homepage loads, returns 200, no JavaScript errors in console
2. Authentication — User can log in with valid credentials, session is established
3. Core workflow — The primary value-delivering action works (e.g., create a document, submit a form, add to cart or complete a purchase flow)
4. Data retrieval — Key data loads correctly (dashboard populates, search returns results, product page loads)
5. Payment/transaction (if applicable) — Payment flow completes with test credentials
6. API health — Primary API endpoints return valid responses with correct schemas
7. Navigation — Critical navigation paths work (deep links, redirects, menu items)
8. Error handling — Application shows a user-friendly error page for invalid routes (404)

What NOT to Include

• Edge cases (those belong in regression tests)
• Visual perfection (that belongs in visual regression tests)
• Performance benchmarks (that belongs in performance tests)
• Exhaustive form validation (that belongs in unit/integration tests)

Keeping It Fast

Target: under 5 minutes for the entire smoke suite.

• Run tests in parallel where possible
• Use API calls instead of UI interactions for setup (create test user via API, not through registration form)
• Skip non-critical assertions (don't check exact copy text, check that elements exist)
• Use a dedicated test account with pre-created data (don't create data from scratch each run)
• Avoid unnecessary waits — use smart waiting (wait for element, not sleep(3000) / waitForTimeout)

Environment-Specific Smoke Tests

Staging smoke tests:

• Full smoke suite (all 5-8 tests)
• Can use test payment providers
• Can test with feature flags in upcoming release configuration
• Can test database migrations

Production smoke tests:

• Subset of staging smoke tests (3-5 tests)
• Use synthetic test accounts (clearly labeled, won't affect analytics)
• Never test with real payment transactions (use sandbox mode or skip)
• Focus on: app loads, auth works, core read operations work, API responds

Post-deployment smoke tests:

• Run immediately after deploy completes (within 60 seconds)
• Same as production smoke tests
• If any fail, trigger alert and begin rollback evaluation

Staging is not production: it has different data volumes, traffic patterns, third-party configurations, and infrastructure scale. That gap is exactly why production and post-deployment smoke tests exist on top of staging verification.

---

Staged Rollout Validation

Rollout Stages

A typical staged rollout. The same ladder expressed for flag-based rollouts adds a 25% step (see below):

What to Monitor Between Stages

Before promoting to the next stage, verify all of these:

Error metrics:

• Error rate (HTTP 5xx) is not higher than baseline
• Exception count is not higher than baseline
• No new error types appearing in logs

Performance metrics:

• P50 and P95 latency are within acceptable range (relative to baseline, not an absolute ceiling)
• No increase in timeout errors
• Database query times are stable

Business metrics:

• Conversion rate is not dropping
• User engagement (page views, actions) is stable
• Revenue/transaction volume is normal (if applicable)

Infrastructure metrics:

• CPU and memory usage are normal
• No increase in queue depth or message backlog
• No disk space issues from new logging

Automated Promotion Criteria

Define rules for automatic promotion between stages. Each gate combines an error-rate ceiling, a latency ceiling expressed relative to baseline, a stability window, and (at higher stages) business-metric guardrails. See references/rollout-automation.md for the full canary→10%→50%→100% promotion ruleset.

Feature Flag Gradual Rollout

An alternative to infrastructure-level canary deploys:

1. Deploy new code to 100% with the feature flag OFF
2. Enable the flag for internal users first (dogfooding)
3. Enable for 1% of users (canary equivalent)
4. Gradually increase: 10%, 25%, 50%, 100%
5. Remove the flag after full rollout is stable for 1 week

Advantages: Faster rollback (just flip the flag), no infrastructure changes, can target specific user segments.

Disadvantages: Code complexity (branching logic), stale flags become tech debt, doesn't catch infrastructure issues.

Tooling

Vendor-native canary analysis (LaunchDarkly Guarded Rollouts, Statsig Auto-tune, Flagger) is now common — if your platform offers it, prefer it over hand-rolled rollout-policy YAML. Given the vendor churn this table documents (Statsig→OpenAI, Split→Harness FME), prefer OpenFeature-compatible SDKs (CNCF spec; Harness FME and others now standardize on it) so flag tooling stays swappable.

Rolling Out AI/LLM Features

AI features need a distinct rollout pattern: prompt versions and model IDs are configurable separately from code, and a kill switch is mandatory.

1. Pin the prompt template version and model ID in your AI Configs platform (LaunchDarkly AI Configs, custom dataset, or feature-flag JSON).
2. Roll out the prompt/model combo behind a flag — internal first, then 1%, 10%, etc.
3. Watch eval metrics (hallucination rate, jailbreak success rate, cost per request) per cohort, not just error rate.
4. Cost guardrail: a budget circuit breaker that fails the feature open (graceful fallback) when a model's per-request cost spikes.
5. Kill switch: a single flag that disables the AI path and routes to a deterministic fallback or a "feature unavailable" state — testable in staging before launch.

See ai-system-testing for prompt-level eval test patterns and testing-in-production for canary metric design.

---

Rollback Criteria and Process

Automated Rollback Triggers

Define these thresholds BEFORE deployment. When any trigger fires, rollback begins automatically. All thresholds are relative to the measured baseline, not absolute ceilings.

Manual Rollback Triggers

These require human judgment but should have clear guidelines:

• Customer-reported critical issue — Multiple users reporting the same problem
• Data integrity concern — Evidence of corrupted or incorrect data
• Security vulnerability discovered — Active exploitation or high-severity CVE
• Monitoring blind spots — You realize you can't monitor a critical metric for the new feature
• On-call engineer judgment — The on-call engineer always has authority to trigger a rollback