Debugging Experiments

Debugging experiments

PostHog Experiments are A/B tests: a feature flag randomizes users into variants, the SDK records an exposure when the flag is read, and PostHog computes per-variant metrics and significance. A customer looks at that results page and asks why it looks wrong.

Most experiment-results tickets are config or exposure-collection problems, not statistics bugs. The randomization is fine; something upstream is skewing which users get exposed, or stopping exposures from being recorded. The job is to find which, prove it with the customer's own data, and hand back a plain-language explanation plus the fix.

This skill is the customer-support front door. It carries the two most common complaints inline (uneven exposures, missing exposures) and loads diagnosing-experiment-health as a diagnostic library for the deeper long tail (interpretation traps, numbers-vs-SQL, mid-run surprises).

Debugging workflow

1. Parse the ticket. Extract project ID, instance (US vs EU — the URLs and data live in different places), experiment ID or name, the lib/platform if relevant, the exact complaint in the customer's words, and what they already tried. Aged or multi-reply tickets are dirty: the config may have been edited mid-thread, so re-pull current state and treat earlier claims as stale.
2. Resolve the experiment. If the ticket names it rather than giving an ID, load finding-experiments to resolve it, then call posthog:experiment-get.
3. Pull the data read-only. Run the fixed data-pull sequence in references/pulling-the-data.md. This produces the "pertinent numbers" you will show the customer: per-variant exposed-person counts, $multiple share, the distinct_id/person fragmentation ratio, the SRM chi-squared result, the exposure trajectory, and the flag/experiment activity log. Verify from data before asking the customer anything.
4. Match the complaint to the known-cause catalog below. Confirm the single leading cause with one targeted number from step 3 before writing. Treat the customer's own conclusion ("it's just noise", "a measurement bug") as a hypothesis to disconfirm, not confirm — pull the data independently rather than re-deriving their answer. Quantify a suspected cause before asserting its impact (count the contaminating cohort, don't eyeball it). One trap in particular: never run the SRM chi-square against an assumed even split — read the configured rollout_percentage first, since an intended 34/33/33 reads as a ~2% SRM under an equal-split assumption.
5. Scope the fix to the experiment's state before recommending it. On a draft, config changes are free — recommend freely. On a running experiment every change has a mid-run tradeoff (changing the split is an anti-pattern — prefer reset or end+restart; see configuring-experiment-rollout and managing-experiment-lifecycle). On a stopped/shipped experiment the flag and results are the documented outcome, so recommend interpretation or a next experiment, not a mid-run edit. Don't propose reversing a state change unless the customer asks how to undo it.
6. Write the reply using references/customer-reply.md: cause → fix → the numbers that prove it, in the customer's UI language.

Known-cause catalog — "exposures aren't even" / "one variant has no traffic"

Ordered by how often they're the answer. Full mechanism detail lives in diagnosing-experiment-health/references/bias-and-skew.md (group A) — load it when a case needs more depth than the summary here.

First, split a real SRM into its two possible homes. Assignment is a deterministic hash of a stable identifier (the distinct_id by default; the device ID or group key for those flag types — see references/pulling-the-data.md), so with an unchanged split every user has a fixed variant and any set of users must fall close to the configured percentages. A confirmed SRM (chi-squared p < 0.001 at healthy volume — not eyeballed) therefore lives in exactly one of two places:

• Assignment-side — the recorded variant disagrees with what the hash would assign. Something overrode assignment at serve time: a stale local-evaluation definition, an inherited bootstrap value, a forced release-condition variant, or a mid-run rehash.
• Capture-side — the recorded variant agrees with the hash (assignment is fine), but which users get an exposure recorded is selected: one arm reaches a surface the other never does, or one arm's users read the flag before it loaded and are silently dropped.

The decisive test that tells you which half you're in — recompute the assignment hash offline, then split the observed gap into the part explained by which users got recorded (selection ⇒ capture-side) and the part explained by users recorded onto the wrong arm (reassignment ⇒ assignment-side) — is the decisive test in references/pulling-the-data.md, with a runnable srm_check.py. Run it before guessing. It names a side only when one component both dominates the gap and is statistically distinguishable from zero; otherwise it reports the split as mixed, or the test as inapplicable, and says why. Don't route on the raw agreement percentage — scattered disagreements can't produce a directional SRM, so a large capture-side skew under a little override noise still reads as high agreement. The causes below are tagged with the half they sit in.

• Uneven split + "Exclude from analysis" (the bias banner). This is the most common real cause. When the variant split is uneven and multiple-variant handling is set to Exclude from analysis (the default) and some users were exposed to more than one variant, the excluded $multiple users are dropped asymmetrically — the smaller variant loses a larger fraction of its users, so it looks artificially worse. PostHog raises the "Setup likely introduced bias" banner once the $multiple share crosses 0.1%. Detect it purely from posthog:experiment-get (split + exposure_criteria.multiple_variant_handling) and the $multiple total from the exposure query. Fix: switch handling to Use first seen variant, and/or move to an even split.
• Sample ratio mismatch (SRM). The observed split is statistically far from the configured split. Confirm with the chi-squared test (p < 0.001) from references/pulling-the-data.md — don't eyeball ratios; a 2:1 skew at a few dozen exposures is normal noise. Count people, not events — run the test on the per-person total_exposures from posthog:experiment-results-get, since raw $feature_flag_called counts vary by how often each arm re-reads the flag and will manufacture an SRM that isn't there. Once confirmed, use the decisive test above to pick the half, then work the tagged causes below. Bot traffic and identity fragmentation are weak directional causes — a crawler counts once per person, and fragmentation only inflates the excluded $multiple bucket — so suspect either only when it correlates with one arm.
• Capture-by-surface (capture-side). One arm reaches a page or screen the other never does, so it collects exposures the other structurally can't. Confirm: split the first-exposure variant by $pathname / $screen_name (query in references/pulling-the-data.md). Some paths near 50% and others near 100% one variant ⇒ this is it; every path showing the same skew ⇒ capture-by-surface is out and the bias is upstream.
• Flag read before it loaded (capture-side). A user who evaluates the flag before flags have loaded (or who doesn't match a release condition) gets false/undefined, which the variant allow-list silently drops — so those users vanish from their arm instead of showing up wrong. If one arm is short by ~N persons, check whether the false/null person count (broken down by $lib/surface) is near N and concentrated on the short arm. If so, flag-read timing is the lead and the fix is in the customer's code.
• Identity fragmentation. The same person is split across multiple distinct_ids (usually identify() called after the flag is read, or anonymous→identified transitions), so they appear in both arms and inflate the $multiple bucket (and, with an uneven split + Exclude, feed the bias banner above). Signal: distinct_id/person ratio noticeably above 1 (use 1.2 as a soft cue), or persons seen under more than one variant. On its own this does not create a directional SRM — the chi-squared test excludes $multiple symmetrically — so don't pin a large directional skew on fragmentation unless the fragmentation rate itself differs by arm. Fix: call identify() before evaluating the flag, or enable experience continuity.
• No randomization / a forced variant. One arm starves because a release condition pins a variant instead of randomizing. Read posthog:experiment-get → feature_flag.filters.groups[]: a group with a non-null variant and broad/empty properties at high rollout, or no group left with variant: null, means users are assigned by rule, not by hash. Fix: remove the pinned-variant release condition so assignment is randomized.
• Mid-run rebucketing. The split, bucketing identifier, or release conditions were edited after start_date, rehashing already-exposed users and stamping them $multiple. Signal: residual exposures for a variant now configured at 0%. Detect via posthog:feature-flags-activity-retrieve diffs. Fix: avoid changing the split mid-run; explain the contamination window.
• Flag dependency failing closed. The experiment's flag can gate on another flag (a release condition of type flag). Dependencies fail closed: a user who doesn't match the parent gets false/no variant instead of being randomized — shrinking the population, and skewing it if the parent's own rollout correlates with anything. Detect via posthog:feature-flags-dependent-flags-retrieve, or a type-flag property in feature_flag.filters.groups[].properties. Fix: widen/align the parent flag, or remove the dependency.

Known-cause catalog — "missing exposures" / "too few exposures" / "0 exposures"

Full detail in diagnosing-experiment-health/references/empty-experiment.md (group B).

• Wrong SDK method. Only single-flag accessors (getFeatureFlag(), isFeatureEnabled()) fire the $feature_flag_called exposure event. Payload/bulk accessors (getFeatureFlagPayload(), getFlags() in posthog-js / getAllFlags() in posthog-node) don't — the flag works but no exposure is recorded. Fix: read the flag with a single-flag accessor, or wire a custom exposure event.
• Capture disabled (send_feature_flag_events: false). The right accessor can still emit no exposure if the SDK is told not to — the send_feature_flag_events init/per-call option (or local/bulk evaluation with events off). The flag works; $feature_flag_called never fires, so it looks identical to the wrong-method case but the cause is config, not the accessor. Fix: enable feature-flag events, or wire a custom exposure event.
• Holdout siphoning the population. If the experiment has a global holdout, a deterministic slice of users is held out and recorded as holdout-<id> rather than a variant — correctly excluded from control/test, but it lowers the analyzable N, which reads as "fewer users than expected." Detect via posthog:experiment-get (holdout field) / posthog:experiment-holdouts-list and a holdout-<id> bucket in the exposure breakdown. It removes users evenly from both arms, so it never creates a directional SRM. Usually nothing to fix — explain it; revisit only if the holdout % is larger than intended.
• identify() timing / dedup. The web SDK deduplicates $feature_flag_called per identity, so users who saw the flag before launch (or before identify()) never re-fire an exposure. Signal: healthy traffic but flat/low exposures for known-active users. Fix: per-session dedup, or trigger on a later event.
• Custom exposure event missing the variant property. A custom exposure event must carry $feature/<flag-key> = the variant value; unlike $feature_flag_called this isn't automatic. Signal: exposures exist but variant is blank. Fix: stamp the property when capturing the event.
• Test-account filter excluding real traffic. exposure_criteria.filterTestAccounts defaults to true; if the customer's own email/domain/IP matches the project's test-account filter, their exposures are silently dropped. Confirm by translating the project's test-account filters to HogQL and counting would-be-excluded exposures.
• Flag-reading code removed / page deprecated. The experiment reads running, but the app stopped calling the flag (a refactor removed the code path, or the page was rerouted). Signal: exposure timeseries flat for weeks with no post-launch flag edits in posthog:feature-flags-activity-retrieve — so config can't explain it; it's application-side.
• Eligibility checked after the flag. If ineligible users hit the flag before the eligibility gate, they get bucketed and inflate the denominator, diluting conversion. Signal: exposures higher than expected, conversion lower. Needs a code read to confirm.