Testing In Production

<objective>
Production is the only environment that is production. Every other environment is an approximation — staging never replicates real data volume, traffic, or third-party quirks. This skill covers how to validate quality in production safely: controlled blast radius, automated rollback, guardrail metrics, and smoke tests that catch problems before users do. The recurring failure it prevents: shipping to 100% of users with no flag to flip, no baseline to compare against, and no tested way back.
</objective>

Quick Route

Discovery Questions

Check .agents/qa-project-context.md first. If it exists, use it as context and skip questions already answered there.

Feature flag system:

• Do you have a feature flag platform? (LaunchDarkly, Statsig — now part of OpenAI, GrowthBook, Unleash, Flagsmith, Harness FME — formerly Split, custom, none)
• How are flags managed? (Dashboard, config file, environment variables)
• Can flags target specific users, percentages, or segments?
• How many active flags exist today? Is there a cleanup process?

Rollout capability:

• Can you deploy to a subset of traffic? (Canary infrastructure, weighted routing, feature flags)
• How long does a deployment take? How long does a rollback take?
• Do you have blue-green or rolling deployments?
• Can you route traffic by region, user cohort, or percentage?

Monitoring maturity:

• What observability is in place? (APM, logging, error tracking, metrics)
• Do you have dashboards for error rate, latency, and business metrics?
• Are alerts configured with appropriate thresholds?
• Can you compare metrics between canary and baseline in real time?

Production access and safety:

• Who has production access? Is there an approval process?
• Are there dedicated test accounts in production?
• Can you run operations in production without affecting real user data?
• Is there a production incident response process?

Core Principles

1. Production is the final test environment

Staging approximates production. It does not replicate production's data volume, traffic patterns, third-party integrations, infrastructure quirks, or user behavior. Testing in production is not reckless — it is realistic. The question is not whether to test in production, but how to do it safely.

2. Safety through blast radius control

Every production test must answer: "If this goes wrong, how many users are affected?" The answer must be as small as possible. Feature flags, canary deploys, and traffic splitting exist to shrink the blast radius from 100% to 1% or less.

3. Always have a tested rollback plan

Before any production test begins, the rollback mechanism must be identified, tested, and fast. "Disable the flag" is a good rollback plan. "Redeploy the previous version" is acceptable. "We'll figure it out" is not a plan. A rollback you have never fired is a hypothesis, not a plan — see Verification for how to prove it works.

4. Monitoring is a prerequisite, not a nice-to-have

You cannot test in production without monitoring. If you cannot measure error rates, latency, and business metrics in real time, you cannot detect problems. Fix monitoring gaps before adding production tests.

5. Production tests must be non-destructive

Production tests must never corrupt real user data, send real notifications to real users, charge real payment methods, or create side effects that require manual cleanup. Synthetic accounts, test flags, and isolated resources are mandatory.

Feature Flag Testing

Feature flags are the safest mechanism for production testing. They decouple deployment from release and provide instant rollback.

Test with flags ON and OFF

Every flagged feature needs tests in both states. The flag-off path is the rollback path and must work flawlessly — use setFeatureFlag(name, true|false, { userId: TEST_USER_ID }) to drive both. See references/patterns.md for the full ON/OFF test pair.

Flag lifecycle testing

Flags are not just on or off. They transition through states, and each transition must be validated.

Flag lifecycle:
  Created → Targeting internal users → Canary (1%) → Partial (10-50%) → Full (100%) → Cleanup (removed)

Test at each stage:
  - Internal: Feature works for internal accounts, hidden from external
  - Canary: Metrics are comparable between flag-on and flag-off cohorts
  - Partial: No performance degradation at scale
  - Full: All user segments work correctly
  - Cleanup: Code with flag removed behaves identically to flag-on

Stale flag cleanup

Flags left in code become technical debt. Run a weekly CI job that queries the flag provider for flags that are 100% rolled out and older than 14 days. These are candidates for code cleanup — remove the flag branching logic and retain only the enabled path. Removing the flag without removing the dead code is half a cleanup.

Flag combination testing

When multiple flags interact, test the combinations that matter. Do not test all 2^N combinations — focus on flags that affect the same user flow (e.g. new-checkout, express-pay, discount-engine-v2 all touch checkout). Pick the critical combinations: all-new, a representative mixed state, and all-legacy. See references/patterns.md for the combination-test loop.

Progressive Rollout

Vendor-native canary analysis. Before hand-rolling the rollout-policy YAML below, check whether your platform already does it: LaunchDarkly Guarded Rollouts (auto-monitored progressive rollouts with metric-based auto-rollback; uses a frequentist sequential-testing analysis model since early 2026), Statsig Auto-tune, Argo Rollouts AnalysisRun, Flagger, Harness Continuous Verification. If you have one, prefer it — the integration with your metrics and rollback mechanics is cheaper than maintaining a custom analysis loop.

AI feature rollout is its own pattern: model variant + prompt as a flag value, with cost guardrails and a kill switch. LaunchDarkly AI Configs / AgentControl (AI Configs rebranded under the AgentControl umbrella, announced May 2026) is the documented path for shipping LLM features behind progressive rollout. See release-readiness for the full pattern.

Canary stages: 1% to 100%

A structured rollout with explicit promotion criteria at each stage.

Automated promotion and rollback

Define machine-checkable conditions for advancing between stages — hold_duration plus metric conditions (error_rate_5xx < 0.5%, latency_p95 < 500ms, crash_rate == 0). Automatic rollback fires when guardrails are breached, with no human approval needed: error_rate_5xx > 2x_baseline for 5m, latency_p99 > 3x_baseline for 5m, crash_rate > 0.1% for 2m, each notifying on-call. Gate on error-budget burn rate, not only raw multipliers, so slow burns that still blow the SLO are caught. See references/rollout-policy.md for the full promotion YAML, rollback triggers, and SLO-gate config.

Verify the rollback fired correctly

Auto-rollback firing is not the same as the incident being resolved. After a rollback triggers, do not declare "recovered" until you have confirmed the system is actually back:

1. Re-run the health-check smoke test against production (GET /api/health returns healthy and the previous version).
2. Confirm the flag/deploy state reverted — query the flag platform that the flag is off (or the deploy that the previous build is serving), don't assume.
3. Confirm guardrail metrics returned to baseline — error rate, P99 latency, and crash rate back within their pre-deploy windows.
4. Confirm the rollback notification reached on-call (Slack/PagerDuty) so the incident is owned.

Only when all four hold is the rollback verified. See Verification for the staging dry-run that proves this chain before first production use.

Production Smoke Tests

Post-deploy critical path tests

Run immediately after every deployment as a pipeline stage (not only in pre-deploy CI). These verify core functionality works with production configuration, data, and infrastructure: a /api/health check, the authentication flow with synthetic credentials, core data loading, and search. Configure retries: 1 and a timeout so a flaky post-deploy run doesn't block the pipeline on the first blip. See references/patterns.md for the full production-smoke.spec.ts.

Synthetic user accounts

Production test accounts must be clearly distinguishable from real users: a reserved email pattern (smoke-test+{env}@yourcompany.com), an is_synthetic = true flag, and exclusion from analytics, billing, and email campaigns. Create them via admin API, and prefer short-lived OIDC / workload-identity tokens over long-lived passwords. See references/patterns.md for the full conventions.

Non-destructive assertions

Production smoke tests must read, not write. When writes are unavoidable, clean up in fixture teardown so cleanup runs whether the test passes or fails.

Do not call test.afterEach() inside a test() body — Playwright registers hooks at describe/file scope, so a hook registered mid-test never schedules teardown and throws test.afterEach() can only be called in a describe block. The data leaks. Use an auto-cleanup fixture that records created resource IDs and deletes them on teardown (or try/finally for a one-off script). See references/patterns.md for the fixture-based create-verify-cleanup pattern.

Guardrail Metrics

What to monitor during rollout

Baseline comparison

Compare canary metrics against a control group running the previous version, not against historical data alone.

Comparison approaches (best to worst):
  1. Canary vs. control: split traffic, compare groups in real time (best)
  2. Before/after: compare post-deploy metrics to pre-deploy window (good)
  3. Historical: compare to same time last week (acceptable for trends)
  4. Absolute thresholds: fixed thresholds regardless of baseline (fragile)

Statistical significance

For business metrics (conversion, revenue), small sample sizes produce noisy results. Wait for statistical significance before drawing conclusions.

Minimum sample sizes for rollout decisions:
  - Error rate: 1,000 requests (errors are rare events, need volume)
  - Latency: 500 requests (more stable, converges faster)
  - Conversion rate: 5,000 sessions (business metrics have high variance)
  - Crash rate: 10,000 app launches (crashes are rare events)

Rule of thumb: if you don't have enough traffic at 1% to reach
significance in 30 minutes, increase to 5% or extend the hold window.