Statistical Analyst

Statistical Analyst

Most bad statistics in business are not arithmetic errors. They are the wrong test on the right data, a null result reported as "no difference," a p-value mistaken for an effect size, or twenty comparisons run and the one that cleared 0.05 written up. This skill covers the applied path: pick the test the data shape actually calls for, check the assumptions that carry weight, size the study before running it, report effects with intervals rather than bare p-values, and say what you found to people who do not want a statistics lecture.

Everything here runs on the Python standard library — the t, chi-square, and normal distributions are implemented directly, so there is no scipy dependency between a question and an answer. Because those implementations are hand-rolled, stats_core.py --selftest verifies all of them against published reference values; run it once before trusting any result.

When to use this skill

• An A/B test finished and someone needs to know whether to ship
• A study is being designed and nobody has computed how many observations it needs
• A stakeholder is quoting a p-value as if it were an effect size
• A "no difference" result is about to be reported from a study that was underpowered
• Several variants, metrics, or segments were compared and no correction was applied
• The data are skewed or outlier-heavy and the default t test is about to be run anyway

Inputs the skill expects

• The business question and what decision it will change
• The outcome variable and its type (binary, continuous, count, ordinal, categorical)
• Group structure — how many groups, independent or paired, randomized or observational
• Observation counts per group, and the unit of measurement (user, session, order)
• The baseline value and the smallest effect worth acting on
• How many comparisons are in the family, and which metric was named primary before the data arrived

Clarify First

Before analyzing, confirm these inputs. If any is unknown or vague, ASK — do not assume:

• The unit of measurement, and whether each unit appears once — clustering (many sessions per user counted as independent rows) deflates standard errors and manufactures significance; it invalidates every test below
• The smallest effect worth acting on — without it there is no way to size the study or to say whether a significant result matters
• How many comparisons are in the family, and which metric was primary — decides the correction and whether the result is confirmatory or exploratory
• Whether anyone has already looked at the data while it accumulated — peeking invalidates a fixed-horizon test and changes the whole approach

Stop rule: ask only the 2-3 that most change the output. If the user says "just draft it," proceed and list your assumptions at the top of the artifact.

Workflows

Workflow 1 — Choose the test and size the study before collecting data

1. Write down the business question and the decision it changes. If the answer changes nothing, stop — do not run the study.
2. Fill in the outcome type, group structure, baseline, and the minimum effect worth shipping into a question spec.
3. Run the selector. It returns one test, its load-bearing assumptions, the fallback when they fail, and the sample size the stated effect requires.
4. If the achievable sample is below the required sample, say so before running. An underpowered study should be a conscious decision, not a discovery at write-up time.
5. Record everything in assets/experiment_design_template.md.

python3 data-analytics/statistical-analyst/scripts/test_selector.py \
  --input data-analytics/statistical-analyst/assets/sample_question.json \
  --power 0.9

Workflow 2 — Run the test and report the effect, not the p-value

1. Check independence first: count units versus count rows. More rows than units means clustering, and no test below is valid until that is handled.
2. Inspect skew and outliers (mean vs median, top and bottom five values). If the data are heavy-tailed at small n, switch to Mann-Whitney.
3. Run the test with the number of comparisons in the family declared, so the threshold is corrected.
4. Read the effect size and the interval first. Report the effect in domain units, the interval in the same units, and the decision implication.

python3 data-analytics/statistical-analyst/scripts/stats_core.py --selftest

python3 data-analytics/statistical-analyst/scripts/run_test.py \
  --input data-analytics/statistical-analyst/assets/sample_experiment.json \
  --comparisons 3

Workflow 3 — Audit someone else's statistical claim

1. Ask what the unit of measurement was and whether units repeat. This finds more real errors than every distributional check combined.
2. Ask how many comparisons were run in total, including the ones not reported, and whether the primary metric was named before the data arrived.
3. Re-run the test from the raw counts, with the true comparison count.
4. If the claim is a null result, compute the interval and state the largest effect the study could have missed — "no difference" and "we could not tell" look identical in a significance test and are opposite conclusions.

python3 data-analytics/statistical-analyst/scripts/run_test.py \
  --input data-analytics/statistical-analyst/assets/sample_revenue.json \
  --test welch_t --comparisons 4 --format json

Decision frameworks

Test selection

Use Welch's t, never Student's t, as the two-group default. It does not assume equal variances and costs a fraction of a degree of freedom when they are equal. Testing for equal variance first and then choosing is worse than always using Welch — the pre-test inflates the error rate of the whole procedure.

Reading an interval against your decision threshold

The last two are identical in a significance test and are opposite conclusions. That is the strongest single argument for reporting intervals.

Sample size reality check

Per group, α = 0.05, power = 0.80:

Most product experiments are sized by "how long can we wait," which is how underpowered studies get written up as "no difference."