K6 Trend Analysis

k6 Trend Analysis

Analyze metric trends across multiple runs of a Grafana Cloud k6 test to catch degradation early -- before thresholds breach and alerts fire. A P95 at 380ms against a 500ms threshold is "green" today, but if it was 250ms a month ago, something is quietly degrading; this skill surfaces that drift and recommends action.

What this skill does NOT do

• Deep-dive into a single run's failure: load k6-cloud-investigate-test
• Edit scripts or apply threshold changes: load k6-test-maintenance
• Create new test scripts: use the appropriate test creation workflow
• Query service-side metrics directly: this skill hands off to debug-with-grafana when observability correlation is needed

Dependencies

This skill delegates all GCk6 API mechanics to k6-manage. Read it before executing any API call -- it covers auth, path construction (the doubled cloud/cloud/ prefix), pagination with @nextLink, the spill envelope, and metric query syntax. Do not duplicate that knowledge here.

Tools used: gcx (via k6-manage patterns).

Workflow

Follow these steps in order. Present findings at the end -- do not apply changes.

Step 1: Identify the test

The user provides one of:

• A GCk6 URL (extract the load test ID from the path)
• A test ID directly
• A test name (search via gcx k6 tests list or the v6 API)

Confirm the test exists by fetching its metadata. Record the id, name, project_id, and created timestamp. You will need the load test ID (not a run ID) for the multi-run metric endpoints.

Step 2: Determine the analysis window

Default to last 30 days. Adjust if:

• The user requests a specific window
• The test has very few runs (<5 in 30 days) -- widen the window and note this
• The test runs very frequently (hundreds of runs in 30 days) -- consider sampling or narrowing. Ask the user if the volume is extreme (>200 runs)

State the window explicitly: "Analyzing runs from {start_date} to {end_date}."

Step 3: Fetch runs in the window

List all runs for the test using the v6 API with $orderby=created desc pagination (see k6-manage Section 3 for the @nextLink loop pattern). Filter to runs within the analysis window.

For each run, record:

• id, created, ended
• result (passed/failed/timed_out)
• status (created/queued/initializing/running/finished/aborted)
• note (if present -- users sometimes annotate runs)

Discard runs that did not reach finished status (aborted, timed_out, etc.) unless the user specifically asks about them -- incomplete runs produce unreliable metric aggregates.

Count the runs. If fewer than 3 usable runs exist, inform the user that trend analysis is not meaningful with this sample size and suggest widening the window or waiting for more runs.

Step 4: Fetch all metrics and their types

Before querying values, discover what metrics the test emits. Use the multi-run metric listing endpoint (k6-manage references/metrics.md Section 2):

GET /cloud/v5/load_tests/{loadTestId}/metrics(test_run_ids=[{id1},{id2},...])

Pass a representative subset of run IDs (the first and last few) to catch metrics that may have been added or removed over the window. Record each metric's name and type (counter, gauge, trend, rate).

Group metrics by type -- the query method must match the metric type (see metrics.md "Query methods"). Using the wrong method returns empty results.

Step 5: Fetch per-run aggregate values

For each metric, query its aggregate value across all runs in the window using the multi-run aggregate endpoint (metrics.md Section 8):

GET /cloud/v5/load_tests/{loadTestId}/query_aggregate_k6(
  query='<method>',
  metric='<metric_name>',
  test_run_ids=[{id1},{id2},...]
)

Concrete gcx form (proxy prefix per k6-manage §2; -o json avoids the spill envelope):

LT=<load_test_id>; IDS="123,124,125"
gcx --context <ctx> api "/api/plugins/k6-app/resources/cloud/cloud/v5/load_tests/$LT/query_aggregate_k6(query='histogram_quantile(0.95)',metric='http_req_duration',test_run_ids=[$IDS])" -o json

Choose the aggregate method based on metric type:

For trend-type metrics (latencies), query multiple quantiles (P50, P90, P95, P99) to see if degradation is uniform or concentrated in the tail.

Always break down by request grouping for multi-target tests. A single aggregate p95 across a whole test obscures regressions confined to one endpoint or one page -- a 3x slowdown on one URL can be invisible at the test-level p95 if the test hits many URLs. Default to grouped queries:

Discover available labels first via the labels endpoint (metrics.md §4) and label-values endpoint (metrics.md §5) before constructing the grouped query. Don't assume labels from reading the script -- a tag rename or refactor can silently shift the label space. Example:

# List labels for a specific metric on a representative run
GET /cloud/v5/test_runs/{id}/labels?match[]=browser_web_vital_lcp

# Then enumerate values for a useful label
GET /cloud/v5/test_runs/{id}/label/url/values

Only fall back to the bare aggregate (no by) for metrics where grouping adds no information -- e.g., vus, load_generator_cpu_percent, single-target tests where every request hits the same URL.

The response includes test_run_id as a label -- use this to map each value back to its run timestamp from Step 3. Grouped queries return one series per (group, run) combination; flatten into a tidy "rid x group" table for the trend computation in Step 7.

If there are too many run IDs to fit in a single URL (hundreds), batch the queries into groups of 50 run IDs and merge the results.

Step 6: Extract thresholds

Thresholds define what "passing" means. Fetch the test's current script (via k6-manage Section 5) and parse the export const options = { thresholds: {...} } block. Record each threshold's:

• Metric name and selector (e.g., http_req_duration{name:homepage})
• Condition (e.g., p(95)<500)
• Whether abortOnFail is set

Also check the most recent run's threshold results from the run data to see which thresholds are currently passing vs. failing.

Not all metrics will have thresholds -- that's fine. Metrics without thresholds still get trend analysis; they just won't have headroom calculations.

Step 7: Compute trends

Detect inflection points and rule out script changes deterministically. Before drawing conclusions about a regression, look for discontinuities in the metric values -- sudden jumps or drops that align across multiple metrics on the same date. When you spot one, do not guess whether the script changed. The run-bundled script endpoint gives a deterministic answer in seconds:

1. sha256-diff the bundled scripts at the boundary. Fetch the snapshot from the last "before" run and the first "after" run via GET /cloud/v6/test_runs/{id}/script (k6-manage §5, "Two distinct script endpoints" -- use the run-scoped endpoint, not the load-test one, since the latter only shows the current version). Compare with shasum -a 256.

   gcx --context <ctx> api /api/plugins/k6-app/resources/cloud/cloud/v6/test_runs/<before_id>/script > /tmp/before.bin
   gcx --context <ctx> api /api/plugins/k6-app/resources/cloud/cloud/v6/test_runs/<after_id>/script > /tmp/after.bin
   shasum -a 256 /tmp/before.bin /tmp/after.bin

   For thoroughness on multi-run inflections, hash every run across the transition -- if N consecutive runs share one hash and N more share another, you have a clean before/after boundary. If hashes change mid-stream, the test was edited multiple times.

2. If the sha256 differs, the test script changed -- the inflection may be a test-side artifact, not a service regression. Split the analysis into distinct eras at the boundary and compute trends within each era separately. Comparing metrics across script changes produces misleading trends -- a P95 drop from 3,000ms to 150ms is not an "improvement" if the script simply stopped hitting a slow endpoint. State the eras explicitly in the report and focus recommendations on the most recent era.

3. If the sha256 matches, the script is byte-identical and the regression is external to the test (service-side, infrastructure, or load-zone). This is high-confidence information -- carry it into Step 9 (service-side correlation) instead of leaving "did the test change?" as an open question.

This sha256 diff is a 5-second deterministic check that rules out a huge class of causes. Run it at every detected inflection, not just when the user asks.

For each metric, build a time-ordered series of (run_timestamp, value) pairs. Then compute:

Basic statistics:

• Mean, standard deviation, min, max across all runs
• Current value (most recent run)
• Baseline value (oldest run in window, or mean of first 3 runs for stability)

Trend direction: Split the runs into two halves (first half and second half of the time window). Compare the mean of each half:

• Degrading: second-half mean is worse by >10% (higher for latency/errors, lower for success rates)
• Improving: second-half mean is better by >10%
• Stable: change is within 10%
• Volatile: standard deviation exceeds 25% of the mean, regardless of direction

The 10% and 25% thresholds are starting points. If the user's test has very tight tolerances or very noisy metrics, adjust and explain the reasoning.

Rate of change: Express as percentage change per week: ((recent_mean - baseline_mean) / baseline_mean) * 100 / weeks_in_window. This normalizes across different window sizes.

Anomaly detection: Flag any run where the metric value is more than 2 standard deviations from the overall mean. These are potential inflection points worth investigating individually.