Synthetic Monitoring

<objective>
Synthetic monitoring runs scripted tests against production on a schedule, 24/7. It catches outages, performance degradation, and broken flows before real users report them — at 3 AM when traffic is zero, probes are the only thing checking your app works. A login page that always returns 200 but never authenticates passes a naive uptime check; a synthetic probe that asserts the dashboard loads catches it. This skill covers probe design, alerting integration, SLA validation, multi-region execution, and the runbook discipline that keeps a 3 AM page actionable.
</objective>

Quick Route

Discovery Questions

Check .agents/qa-project-context.md first. If it exists, use it as context and skip questions already answered there. Each answer changes the probe set, the alerting config, or the SLA math.

Critical flows (decides which probes you write):

• What are the 5-10 most important user journeys (login, search, checkout, signup, core workflow)? These become your probe list.
• Which flows, if broken, cause immediate revenue loss or churn? These get the shortest frequency and page on-call.
• Are there flows that break silently (data sync, background jobs, webhooks)? Silent failures need probes most — no user reports them.
• Do you have documented SLAs/SLOs for availability and response time? They set the downtime budget and the alert thresholds.

Current monitoring (decides where the gaps are):

• What exists today (uptime checks, APM, error tracking, dashboards)? Avoid duplicating; find the gap.
• Are there gaps between what monitoring covers and what users experience? That gap is where probes earn their keep.
• How do you learn about production issues today (alerts, user reports, social media)? If it's user reports, detection time is your first metric to fix.
• What was the last outage, and how long before it was detected? Sets the detection-time target probes must beat.

Infrastructure (decides regions and CDN assertions):

• Is the app served from multiple regions or one? Multi-region serving needs multi-region probing.
• Are there CDN, caching, or edge layers that could mask origin failures? If yes, probes must assert on x-cache/origin headers.
• Do third-party dependencies (payment, auth, email) have their own monitoring? Their status pages feed your runbooks.
• What alerting systems exist (PagerDuty, OpsGenie, Slack, email)? Decides the routing config.

Test accounts (decides data safety):

• Do dedicated synthetic test accounts exist in production? Without them, probes pollute real data and analytics.
• Can test accounts be excluded from analytics, billing, and email campaigns? If not, probe traffic skews every downstream number.
• Is there API access for programmatic account setup and data cleanup? Decides whether create-then-delete probes are viable.

Core Principles

1. Synthetic tests validate the user experience continuously

RUM tells you what happened; synthetic tells you what is happening right now, whether or not real users are active. At 3 AM when traffic is zero, synthetic probes are the only thing checking the app works. Complement RUM, never replace it: synthetic covers known paths with predictable inputs, RUM discovers the creative ways real users break things.

2. Keep probes simple and fast

A probe that takes 2 minutes and touches 15 pages is not a probe — it is an E2E suite running in production. Probes are short (per-probe wall-clock budget under 30 seconds), focused (one critical path each), and stable (retries: 0, zero flakiness tolerance). The 30-second ceiling is a hard budget: a slow probe that "still passes" is masking a degradation users feel.

3. Alert on trends, not single failures

A single probe failure is noise — network blips and DNS hiccups cause them constantly. Two consecutive failures are a signal; failures from 2+ regions confirm it is not local. Configure consecutive-failure and multi-region thresholds, or alert fatigue teaches the team to ignore the pager.

4. Probes must not affect production data

Probes run every few minutes, 24/7. Even small side effects (a created record, an incremented counter) compound. Probes must be non-destructive, created data cleaned up immediately, and synthetic traffic excluded from analytics, billing, and — critically — from SLO/error-budget math itself, or probes inflate your own reliability numbers.

5. Assert on the goal, not the status code

A login page that returns 200 but never authenticates is broken. A search page that returns 200 with zero results is broken. Probes assert that the user can accomplish their goal — data loads, auth succeeds, results appear — not merely that the page returns a 2xx.

Probe Design

Design probes around critical user journeys, not infrastructure components. Users do not care if your load balancer is healthy — they care if they can log in and use the product.

Probe implementations

Three probe shapes cover most needs: a login flow (Playwright browser), an API health check (status + auth + latency budget), and a search probe (fill query → submit → assert on results, not just status). Keep each to one critical path, a tight timeout, and retries: 0. See references/probe-implementations.md for runnable login-flow, API-health, and search probes plus the environment-aware config.

Non-destructive probe patterns

Safe patterns:
  - Read-only operations: GET requests, page loads, searches
  - Sandbox transactions: payment in test mode, email to internal addresses
  - Create-then-delete: create a draft, verify, delete immediately
  - Synthetic flag: operations tagged as synthetic, excluded from processing

Dangerous patterns (avoid):
  - Creating real orders, tickets, or user-facing records
  - Triggering real notifications (email, SMS, push)
  - Modifying shared resources (config, permissions, settings)
  - Operations that cannot be automatically cleaned up

Dedicated test accounts

Account requirements:
  - Clearly identifiable: email contains "synthetic" or "monitor"
  - Flagged in database with an is_synthetic flag (is_synthetic = true)
  - Excluded from analytics, billing, email campaigns, support queues
  - Excluded from RUM and SLO/error-budget pipelines (synthetic traffic is not real traffic)
  - Pre-populated with stable test data that probes can rely on
  - Credentials in a secrets manager (process.env), rotated quarterly
  - Separate accounts per concurrent probe (avoid state conflicts)

Platform Options

Probes can be authored in Playwright (TS/JS), Puppeteer, k6 (JS — k6 2.0 shipped May 2026 with AI-assisted test authoring and a clearer Assertions API; first-class for synthetic), or Python (CloudWatch Synthetics, Checkly). Pick what your team already maintains. Avoid: Pingdom for new setups — it is a legacy uptime tool; prefer code-first alternatives (Checkly, Grafana, custom Playwright) that version-control probes alongside your app.

Custom and managed implementations

Self-managed: schedule Playwright probes with a GitHub Actions schedule cron (every 5 minutes), inject prod credentials via secrets, and report results to a monitoring webhook. Managed: Checkly is Playwright-native and runs from multiple locations on a fixed frequency. Either way, make probes environment-aware so the same code runs against staging and production with different base URLs and thresholds. See references/platforms-and-ci.md for the GitHub Actions workflow, the Checkly config, the alert-routing rules, and the runbook template.

Alerting Integration

Not every probe failure is an incident. Configure rules that cut noise while catching real problems.

Alerting rules:
  - Single failure: log, do not alert (transient network issue)
  - 2 consecutive failures from same region: warning (possible issue)
  - 2 consecutive failures from 2+ regions: alert on-call (confirmed outage)
  - Latency >2x baseline for 10 minutes: warning (performance degradation)
  - Latency >3x baseline for 5 minutes: alert on-call (severe degradation)
  - Any probe timeout: alert if 3 consecutive (service unresponsive)

Routing. Critical failures on revenue paths (login, checkout, api-health) page on-call (PagerDuty + Slack incidents) with a short repeat interval; warnings on secondary probes go to a Slack monitoring channel; info-level events use a long repeat interval. The probe must tag each result with a severity and probe label for these routes to match — see the routing note in references/platforms-and-ci.md for the full alerting-rules.yaml and the tagging step.

Suppress synthetic alerts during planned maintenance. A maintenance window should silence synthetic paging (the probes will fail by design) and exclude that window from error-budget math, or scheduled work burns budget and pages on-call for nothing.

Alert message format — include enough context to start investigating immediately:

Alert template:
  Title: [SYNTHETIC] {probe_name} failing from {region}
  Severity: {critical|warning|info}
  Consecutive failures: {count}
  Last success: {timestamp}
  Error: {error_message}
  Duration: {last_response_time_ms}ms (threshold: {threshold}ms)
  Dashboard: {link_to_dashboard}
  Runbook: {link_to_runbook}
  Regions affected: {list_of_failing_regions}

The {link_to_runbook} points at a per-probe runbook (six lines: what it tests, first checks, manual repro, escalation, dashboard, owner). See the template in references/platforms-and-ci.md.

SLA Validation

Availability calculation

Availability = (total_minutes - downtime_minutes) / total_minutes × 100

Where:
  - total_minutes = calendar month in minutes (43,200 for a 30-day month)
  - downtime_minutes = minutes where synthetic probes detected failure

SLA tiers (downtime/month, monthly basis on 43,200 min):
  99.0%  = 432 min   = 7h 12min     (basic web app)
  99.9%  = 43.2 min  = 43min 12s    (business application)
  99.95% = 21.6 min  = 21min 36s    (critical SaaS)
  99.99% = 4.32 min  = 4min 19s     (infrastructure/platform)

These are common availability targets, not prescriptive tiers. Pick targets from a user-impact analysis, not by tier name. Modern practice (Google SRE Workbook, OpenSLO) favors explicit SLO + error-budget policies over labelled tiers — define what user-visible failure looks like, set the budget user impact tolerates, and let the SLO follow. References: https://sre.google/workbook/ ; https://openslo.com/

Response time percentiles

Track percentiles, not averages — averages hide the worst experiences.

SLA response time targets (example):
  Homepage load:  P50 < 1s,    P95 < 3s,   P99 < 5s
  API response:   P50 < 200ms, P95 < 500ms, P99 < 1s
  Search results: P50 < 500ms, P95 < 2s,   P99 < 4s
  Login flow:     P50 < 2s,    P95 < 5s,   P99 < 8s

Error budget tracking

Error budget connects SLA targets to engineering decisions.

Error budget calculation:
  SLO: 99.9% availability on a 43,200-min month
  Budget: 0.1% of total time = 43.2 minutes/month

  Budget consumed this month: 12 minutes (28%)
  Budget remaining: 31.2 minutes (72%)

Actions by budget status:
  >50% remaining: normal operations, ship features
  25-50% remaining: caution, review recent changes
  <25% remaining: freeze non-critical deploys, focus on reliability
  Budget exhausted: incident mode, every deploy needs extra scrutiny

Exclude synthetic-probe downtime caused by your own maintenance windows from this calculation, or planned work shows as budget burn.