Ab Test Setup

A/B Test Setup

Overview

An A/B test answers one question: did this change move this metric, or was it noise? The answer is only worth something when the sample size, the metric and the stopping rule were fixed before the first visitor arrived. This skill produces three things: a written experiment plan committed before launch, an assignment and logging spec a developer can implement, and a readout that states the effect as an interval instead of a "winner" badge.

Most failed tests fail on arithmetic that could have been done on day zero: the site does not have enough traffic for the effect the team hopes for. Do that arithmetic first.

Instructions

1. Gather the inputs

Ask for what is missing, and look in the project for the rest (feature-flag SDK, analytics events, an experiments/ folder with earlier plans):

• The change and the evidence behind it. Session recordings, support tickets, a funnel drop, a survey. "We want to try it" is allowed, but record that the prior is weak.
• Unit of assignment. Logged-in user, account, or anonymous visitor ID. Accounts when users inside one company would otherwise see different prices or features.
• Primary metric and its baseline, measured over the last 4 full weeks on the same population that will enter the test (same pages, same devices, same traffic sources).
• Eligible units per week — people who actually reach the changed element, not total site traffic.
• Smallest lift worth shipping. This is a business question: below what improvement would nobody bother to keep the variant?
• What else is running on the same pages, and any sale, launch or campaign planned during the run.

2. Do the arithmetic before anything else

For a conversion rate, with control rate p1, variant rate p2 = p1 × (1 + lift) and p̄ their mean, the visitors needed in each arm are:

n = ( z_a × sqrt(2 × p̄ × (1 − p̄)) + z_b × sqrt(p1 × (1 − p1) + p2 × (1 − p2)) )² ÷ (p2 − p1)²

z_a = 1.960   two-sided significance level 0.05
z_b = 0.8416  power 0.80 (an effect of this size is detected 4 times out of 5)

Save the calculator as abtest.py. It also covers the two checks used later.

"""Sample size, sample-ratio check and readout for a two-arm conversion test. Standard library only."""
import math, sys
from statistics import NormalDist

Z = NormalDist()

def size(base, rel_lift, alpha=0.05, power=0.80, comparisons=1):
    """Visitors per arm to detect base -> base*(1+rel_lift), two-sided."""
    p1, p2 = base, base * (1 + rel_lift)
    za = Z.inv_cdf(1 - alpha / comparisons / 2)
    zb = Z.inv_cdf(power)
    pbar = (p1 + p2) / 2
    root = za * math.sqrt(2 * pbar * (1 - pbar)) + zb * math.sqrt(p1 * (1 - p1) + p2 * (1 - p2))
    return math.ceil(root ** 2 / (p2 - p1) ** 2)

def srm(n_a, n_b, share_a=0.5):
    """Chi-square p-value that the observed split matches the planned one."""
    total = n_a + n_b
    exp_a, exp_b = total * share_a, total * (1 - share_a)
    chi2 = (n_a - exp_a) ** 2 / exp_a + (n_b - exp_b) ** 2 / exp_b
    return math.erfc(math.sqrt(chi2 / 2))

def result(n_a, conv_a, n_b, conv_b, alpha=0.05):
    """Two-proportion z-test plus a confidence interval for the absolute difference."""
    pa, pb = conv_a / n_a, conv_b / n_b
    pooled = (conv_a + conv_b) / (n_a + n_b)
    z = (pb - pa) / math.sqrt(pooled * (1 - pooled) * (1 / n_a + 1 / n_b))
    p_value = 2 * (1 - Z.cdf(abs(z)))
    half = Z.inv_cdf(1 - alpha / 2) * math.sqrt(pa * (1 - pa) / n_a + pb * (1 - pb) / n_b)
    return pa, pb, pb - pa, (pb - pa - half, pb - pa + half), p_value

if __name__ == "__main__":
    cmd, *a = sys.argv[1:]
    if cmd == "size":
        print(size(float(a[0]), float(a[1]), comparisons=int(a[2]) if len(a) > 2 else 1), "visitors per arm")
    elif cmd == "srm":
        p = srm(int(a[0]), int(a[1]))
        print(f"SRM p = {p:.4g} ->", "STOP: assignment is broken" if p < 0.001 else "split is fine")
    elif cmd == "result":
        pa, pb, diff, ci, p = result(*map(int, a))
        print(f"control {pa:.2%}  variant {pb:.2%}  diff {diff:+.2%} points "
              f"(95% CI {ci[0]:+.2%} to {ci[1]:+.2%})  relative {diff / pa:+.1%}  p = {p:.4f}")

Then: weeks = ceil(arms × n ÷ eligible units per week), rounded up to whole weeks, never fewer than 2. Whole weeks matter because weekday and weekend visitors behave differently.

Other cases:

• Several variants. Each comparison against control gets 0.05 ÷ number of variants (Bonferroni). Pass the count as the third argument: size 0.04 0.15 2.
• Unequal split. Total sample grows by 1 ÷ (4 × w × (1 − w)) where w is the control share: 70/30 costs 1.19×, 80/20 costs 1.56×, 90/10 costs 2.78×.
• Averages (revenue per visitor, order value): n = 2 × σ² × (z_a + z_b)² ÷ δ², about 16 × σ² ÷ δ², with σ the standard deviation and δ the absolute difference to detect. Revenue is heavy-tailed, so state in the plan how extreme orders are capped.

3. Write the plan and commit it

Create experiments/YYYY-MM-DD-short-name.md before launch. Example 1 shows a filled one. It must contain:

1. Hypothesis in one sentence: the change, who sees it, the metric, the direction, and the evidence.
2. Primary metric — exactly one, with numerator, denominator and the window in which a conversion counts (for example "paid within 14 days of first exposure").
3. Guardrails — two to four metrics that must not get worse (refund rate, page errors, support contacts, revenue per visitor), each with the drop that would stop the test.
4. Sample size, split, start and end dates.
5. Stopping rule (section 5) and the decision rule for each outcome.
6. Segments that will be reported, named in advance. Anything else found later is a new hypothesis, not a finding.

4. Specify assignment and exposure logging

import { createHash } from "node:crypto";

// Same unit + same experiment key -> same arm, on every server and every visit.
export function assign(experimentKey, unitId, arms = ["control", "variant"]) {
  const digest = createHash("sha256").update(`${experimentKey}:${unitId}`).digest();
  const bucket = digest.readUInt32BE(0) / 2 ** 32; // uniform in [0, 1)
  return arms[Math.floor(bucket * arms.length)];
}

• Hash a stable identifier. A new random draw per page view puts one person in both arms.
• Put the experiment key in the hash so two experiments split independently of each other.
• Log an exposure event (experiment_key, arm, unit_id, timestamp) at the moment the person reaches the changed element, not at assignment. Analyse only exposed units, and analyse by the same unit that was randomised.
• Render the variant on the server or before first paint. A page that flashes the control and then swaps is a different treatment from the one in the plan.
• For tests that split by URL, follow Google Search's guidance for site tests: no different content for the crawler than for people, rel="canonical" from each variant URL to the original, a 302 redirect rather than a 301, and remove the test when it ends.

5. Launch checks and the stopping rule

Before launch, force each arm with an override and confirm on desktop and mobile that the page renders, the exposure event fires once, and the conversion event carries the arm.

Daily during the run, look only at: python3 abtest.py srm on exposed counts, guardrails, and error rates. A sample-ratio mismatch (p below 0.001) means units are being lost in one arm (redirects, bot filtering, a crash, late logging). The result cannot be read: stop, fix, restart with a new experiment key.

Do not act on the primary metric early. Checking it repeatedly and stopping at the first p < 0.05 produces "winners" from identical pages:

Pick one rule and write it in the plan:

• Fixed horizon (default): read the primary metric once, at the planned sample size.
• Planned interim looks: stop early only if an interim p-value is below 0.001; the final look still uses 0.05. This keeps the overall false-positive rate at about 5%.
• Sequential or Bayesian engine in the testing tool: follow that tool's own stopping rule and do not mix it with the fixed-horizon numbers above. Sequential methods allow continuous monitoring at the price of wider intervals.