Review Experiment Results

Review Experiment Results

Explain an FME experiment's outcome in plain language: whether there's a winner, why or why not, how guardrail metrics were affected, and any data-quality caveats - with underlying statistics available on request.

Related: manage-experiments (experiment changes), choose-metric (metric selection).

Tools

Works through the Harness MCP server or the Harness CLI; names are from tool-map.md.

Instructions

Phase 1: Establish scope

Follow scope-establishment.md.

Step 1: Resolve the experiment

If the user gives an exact experiment ID, Get experiment directly - skip list. Otherwise List experiments for parent type FEATURE_FLAG (or AI_CONFIG if user specified), with name substring match if provided (MCP: match_type: "contains"). Default parent type: FEATURE_FLAG.

If user didn't specify status, list ACTIVE, PAUSED, COMPLETED, and ARCHIVED - a finished or archived experiment is exactly the kind a results question is usually about, so never default to active-only here. Fully paginate all selected statuses per pagination before deciding uniqueness or absence; CLI needs a separate paginated query per status. More than one match or no name given → ask which experiment. If a scan is incomplete, report that instead of silently selecting a first-page match.

Step 2: Fetch experiment definition

Get experiment. Key fields: status, environment, startAt/endAt, baselineTreatment, comparisonTreatments[], keyMetrics[], supportingMetrics[].

keyMetrics/supportingMetrics are the only metric lists on the experiment. GUARDRAIL and ALERT metrics are workspace-wide categories applying to every experiment, so they never appear here - read role from each result's category (Step 4). Empty keyMetrics → no winner criterion; ask which metric(s) should drive verdict.

More than one comparisonTreatments and user didn't name one → report every treatment (one verdict each, see Output Format). If question presumes a single treatment ("did it win?") → ask which treatment or confirm per-treatment readout.

Step 3: Fetch experiment settings

Get settings. Always returns applied settings (404 only if experiment doesn't exist). source: "EXPERIMENT_OVERRIDE" means experiment has its own override; other values mean it inherits org defaults.

Fields to carry forward: significanceThreshold (not always 0.05), multipleComparisonCorrection (NONE or GROUPWISE_HOCHBERG; applies only to KEY/SUPPORTING, never GUARDRAIL/ALERT), minimumSampleSize (per treatment), statisticalTestType (FIXED_HORIZON/SEQUENTIAL), reviewPeriod (ISO-8601 duration), varianceReduction.method (NONE/CUPED).

Step 4: Fetch evaluated results

List results for the experiment, optionally narrowed to specific comparison treatments. List-only (no single-result get, no per-environment filter). Returns latest calculation run only, as one row per (metric, comparison treatment) pair, plus top-level calculatedAt (null if never calculated). If narrowing to specific treatments, filter the returned rows on each item's comparison (see tool-map.md).

Per-row fields: category (KEY, SUPPORTING, GUARDRAIL, or ALERT), comparison (comparison treatment this row is for), metricId.id / metricId.name (name can be null; resolve in Step 5), metricResultState (stats engine verdict; null if none yet), pvalue, impactLower/impactUpper, value, errorMargin, baselineMean/comparisonMean (each with Lower/Upper), baselineSampleSize/comparisonSampleSize, varianceReduction, positive (metric's configured isPositive, not observed outcome; non-null even when numeric fields are null).

No SRM field in response, so can't confirm observed treatment allocation matched configured treatment allocation - Step 8 says so whenever it reports a winner. Report metricResultState/pvalue as given; never recompute significance or claim to correct for peeking.

Reconcile before scoring. The expected row set is the Cartesian product of every configured metric (keyMetrics ∪ supportingMetrics from Step 2) × every comparison treatment in scope (all of them, or the one(s) picked in Step 2); it must be non-empty, since Step 2 already stops on empty keyMetrics. Match returned rows against it:

• Two or more rows for the same (metricId.id, comparison) pair → duplicate/conflicting data; don't pick one arbitrarily - treat that combination as needs_more_data and note the conflict in Step 8.
• An expected (metric, comparison) combination has no row at all → treat as needs_more_data, same as a row with metricResultState: null; never treat a missing combination as "nothing to report."
• A row's comparison isn't in the experiment's current comparisonTreatments → drop it (stale data from a removed treatment).
• calculatedAt is null, or the whole result list is empty → no calculation has run; every KEY result is needs_more_data, so Step 7's verdict is NEED_MORE_DATA - never report WINNER from an empty or uncalculated result set.

GUARDRAIL/ALERT rows aren't part of the expected set (they're workspace-wide, not listed on the experiment) but must still flow through to Step 7's DATA_QUALITY_CONCERN modifier when present with no data - don't drop them while reconciling KEY/SUPPORTING coverage.

Step 5: Resolve metric names and descriptions

Resolve every distinct metric ID from returned rows and configured metrics missing from those rows, so gaps can be named. MCP may List metrics by IDs, fully paginating until every requested ID is resolved or the complete inventory establishes a missing ID; alternatively get each ID directly. CLI must Get metric separately for each ID (or one fully paginated single-ID list per call). An unresolved ID stays unverified, not absent from a first-page miss. Include returned GUARDRAIL/ALERT IDs too. Descriptions help judge severity: a 2% dip on "leading indicator, noisy" reads differently from same dip on "primary revenue guardrail".

Step 6: Classify each result

Map each metricResultState to desired, undesired, inconclusive, or needs_more_data using state-classification.md. For needs_more_data results, compare baselineSampleSize/comparisonSampleSize to minimumSampleSize so readout can say how far along collection is.