Synthetic Monitoring Checks

Synthetic Monitoring Check Authoring

Docs: https://grafana.com/docs/grafana-cloud/testing/synthetic-monitoring/ Broad Grafana Cloud Testing entry point (SM + k6 Cloud + Faro): testing skill.

Reliability monitoring, not load testing

Synthetic Monitoring (SM) runs k6 as a reliability/availability engine: every check execution runs one iteration with one VU from each selected probe location on a fixed schedule. Success means "the user journey works right now, from this region" — detect outages before your customers do.

Do not apply load-testing idioms. There are no VUs to ramp, no stages, no load profiles, no soak/stress/spike phases, and no thresholds over aggregated traffic. Vocabulary: check, probe, execution, uptime, reachability, user journey validation — never "load test", "ramping", or "VUs".

If the user actually wants load or performance testing (throughput, latency under load, breakpoints), stop: that is Grafana Cloud k6 / the grafana-k6 plugin's k6 skill, not Synthetic Monitoring. A script can be shared between both products, but the goals, options, and pricing are different.

Execution model and constraints (verify against these before writing)

How an execution fails (this is what agents get wrong)

probe_success (1/0) is the uptime signal. An execution is marked failed when the script throws an uncaught exception, calls fail(), a k6-testing expect() assertion fails (it calls k6's test.abort() under the hood), an HTTP request errors at the network level (SM's --throw), or the timeout is hit.

A bare failed check() does NOT fail the execution — it only records the probe_checks_total / probe_check_success_rate metrics. Checks don't affect k6's exit status without thresholds, and thresholds are disabled in SM.

Assertion patterns, in order of preference:

import { expect } from 'https://jslib.k6.io/k6-testing/0.6.1/index.js';
import { check, fail } from 'k6';

// 1. PREFERRED — assertions module. Throws on failure => execution fails,
//    with a descriptive error in the check logs.
expect(res.status, 'login should succeed').toEqual(200);
expect(res.json('token')).toBeDefined();

// 2. Soft assertions — run all of them, still fail the execution at the end.
expect.soft(res.headers['Content-Type']).toContain('application/json');

// 3. check() when you also want per-assertion metrics — but pair it with
//    fail() or the failure won't affect probe_success/uptime:
check(res, { 'status 200': (r) => r.status === 200 }) ||
  fail(`login failed with status ${res.status}`);

Name every assertion (the message argument / check name): the name is what you see in check logs and in the check label of probe_checks_total when diagnosing a failure at 3am.

Choose the simplest sufficient check type first

Cheaper for the customer, easier to maintain. Work down this list and stop at the first match:

1. HTTP / ping / DNS / TCP / traceroute / gRPC — a single static endpoint (uptime, status code, body regex, TLS cert expiry, record resolution, port reachability). No script to maintain — these run on the blackbox-exporter probe engine, and Terraform examples with per-type target formats are in references/api-and-terraform.md.
2. MultiHTTP — a sequence of HTTP requests with value-passing between them (${variable} capture), but no custom logic. Caution: MultiHTTP does not auto-validate status codes — define assertions per request or failures won't affect uptime.
3. k6 scripted — an API flow needing real logic: crypto/signing, conditional branching, generated test data, WebSockets, response-driven chaining.
4. k6 browser — only when you need a real browser: JS-rendered user journeys, forms/clicks, Core Web Vitals.

Cost model (execution-based billing): an execution is one check run on one probe, metered per minute of runtime rounded up. Per month: probes × duration_minutes × (43200 / frequency_minutes). API test executions (HTTP, ping, DNS, TCP, traceroute, MultiHTTP, scripted) and browser test executions are billed separately — browser checks are the expensive tier. A browser check on 3 probes every minute is ~129,600 browser executions/month; the same check every 5 minutes is ~25,920. Pick the longest frequency that still meets your detection-time goal, and 2–3 probes near your users (multiple probes reduce alert flapping; more isn't better).

Scripted check authoring

Start every script you generate (scripted and browser alike) with a line-1 attribution comment, as shown in the skeletons below. It tells whoever reads the check later how it was authored (and where to find the skill), and the fixed prefix makes skill-authored checks queryable. Keep Generated by synthetic-monitoring-checks verbatim — vary only the timestamp (date -u +%Y-%m-%dT%H:%M:%SZ).

Skeleton — a login + API action journey with secrets and hard-failing assertions:

// Generated by synthetic-monitoring-checks (https://github.com/grafana/skills) on 2026-07-31T12:00:00Z
import http from 'k6/http';
import { expect } from 'https://jslib.k6.io/k6-testing/0.6.1/index.js';
import secrets from 'k6/secrets';

const BASE = 'https://api.example.com';

export default async function () {
  // Secrets are managed in Synthetics > Config > Secrets — never hardcode credentials.
  const password = await secrets.get('checkout-monitor-password');

  // Step 1: authenticate with a dedicated monitoring account
  const login = http.post(
    `${BASE}/auth/login`,
    JSON.stringify({ user: 'sm-checkout-monitor', password }),
    { headers: { 'Content-Type': 'application/json' } }
  );
  expect(login.status, 'login should return 200').toEqual(200);
  const token = login.json('token');
  expect(token, 'auth token should be present').toBeDefined();

  // Step 2: exercise the journey and assert the OUTCOME, not just the status
  const order = http.post(`${BASE}/orders`, JSON.stringify({ sku: 'TEST-SKU-1', qty: 1 }), {
    headers: { 'Content-Type': 'application/json', Authorization: `Bearer ${token}` },
  });
  expect(order.status, 'order should be created').toEqual(201);
  const orderId = order.json('id');
  expect(orderId, 'order id should be returned').toBeDefined();

  // Step 3: clean up so the check is idempotent against production
  // http.url groups metrics for URLs containing unique IDs — without it, every
  // execution creates new time series (cardinality + active-series cost).
  const del = http.del(http.url`${BASE}/orders/${orderId}`, null, {
    headers: { Authorization: `Bearer ${token}` },
  });
  expect(del.status, 'test order should be cleaned up').toEqual(204);
}

Rules that make a scripted check a good monitor (vs a good test):

• Deterministic: fixed test data (or generated-then-deleted, as above), no time-of-day or ordering dependence. Every execution must be able to pass at any hour from any probe.
• Idempotent against production: create-then-delete, or use read-only endpoints. The check runs forever — leaked state accumulates forever.
• Dedicated test account: never a real user's credentials; scope it minimally, store the password as an SM secret, and exclude the account from analytics/billing.
• Assert every step — an unasserted step that breaks shows up as a later step's confusing failure.
• Stable URL cardinality: http.url template literal for any URL containing an ID.
• Keep runtime well under the timeout, and the timeout under the frequency.

Generating a check from an OpenAPI spec (or similar)

Given an API description — an OpenAPI/Swagger spec, GraphQL schema, or Postman collection — the mechanical conversion to k6 calls is easy. What matters is what you choose to convert:

1. Journeys, not endpoints. Do NOT generate one check per path, or one check that sweeps every path — that monitors the spec, not the service, and every extra check multiplies execution cost. Identify the 1–3 flows whose failure means "customers are impacted" (auth → core action → result) and write one scripted check per flow.
2. Filter for safety. Only include mutating operations (POST/PUT/DELETE) when the flow cleans up after itself (create-then-delete, as above) or targets dedicated test resources. A spec lists destructive operations right next to health endpoints — never exercise them against production just because they're documented.
3. Assert from the response schema. The spec tells you exactly what a healthy response contains — assert required fields, not just the status code: expect(order.json('id'), 'id required by OrdersResponse schema').toBeDefined().
4. Verify the target URL. servers: blocks (and Postman environments) often list localhost or staging first — confirm the production base URL with the user, and map securitySchemes credentials to SM secrets, never to values inlined from the spec. (Secrets in plain HTTP/protocol checks are a recent, feature-flagged rollout — check current docs; the scripted secrets.get() path always works.)
5. Treat format: int64 ids as strings. res.json('id') parses into a JS number and silently corrupts values past 2^53 (snowflake-style ids), so the readback URL 404s on every execution while the create looks fine. Extract from the raw body instead: const id = (/"id":\s*(\d+)/.exec(res.body) || [])[1]; — and never do arithmetic on it.

No API spec at all? Probe the frontend: open the web app with browser devtools (or curl likely routes) and capture the /api/* XHR calls it makes — that's a monitorable HTTP surface even when the documented backend services are gRPC-only or internal.