Product Experiments

Product Experimentation Excellence

Drive measurable growth and mitigate risk through rigorous A/B testing and data-driven learning.

Help the user with product experimentation excellence using insights from 9 guests and posts across Lenny's Podcast and Newsletter.

How to Help

1. Hypothesis Definition - Guide the user in drafting clear, falsifiable hypotheses based on user behavior theories.
2. Experimental Design - Help determine the right metrics, sample sizes, and guardrail metrics for a clean test.
3. Statistical Analysis - Support the interpretation of p-values, confidence intervals, and potential sample ratio mismatches.
4. Strategic Evaluation - Assist in deciding whether to ship, iterate, or kill a feature based on experiment results and long-term business impact.

Core Principles

Use long-term holdouts for true incrementality

Archie Abrams: "So we constantly will relook at an experiment a year later, see that the way the GMV curve for the distribution was different than we might've originally thought. And that'll actually change what we do from that previous experiment. And so there's a lot of longterm monitoring of experiments over these very long time horizons to both inform what those input metrics are and more importantly hold ourselves accountable to, did we actually move what we cared about, which is that longterm GMV, in the right way?"

Implement holdout groups for one or more years to distinguish between immediate growth and short-term pull-forward effects. This ensures you are measuring the genuine downstream business impact of changes.

Focus experiments on risk mitigation

Lauryn Isford: "So, with all that said, generally my advice is to experiment when you need to and to primarily see it as a risk mitigation tactic when you're making dramatic changes and to let the product development process do more work. So, spend more time with customers, be more rigorous in understanding precisely what problem you're solving, get mocks in front of people and see how they react, and hopefully have more conviction than you otherwise would when you ship something that it's okay if every customer sees it tomorrow and that the experiment doesn't actually matter as much."

Prioritize A/B testing for high-stakes, dramatic product changes rather than using it solely for precise metric attribution. Invest in qualitative research first to build conviction before launching high-risk tests.

Balance success metrics with guardrails

Ronny Kohavi: "It's very easy to increase revenue by doing theatrics. Displaying more ads is a trivial way to raise revenue, but it hurts the user experience. And we've done the experiments to show that. In this case, this was just a home run that improved revenue, didn't significantly hurt the guardrail metrics."

Develop a robust Overall Evaluation Criterion (OEC) that includes guardrail metrics. This prevents short-term wins from inadvertently degrading the long-term user experience or retention.

Normalize a high failure rate

Ronny Kohavi: "At Bing, which is a much more optimized domain after we've been optimizing it for a while, the failure rate was around 85%. So it's harder to improve something that you've been optimizing for a while. And then at Airbnb, this 92% number is the highest failure rate that I've observed."

Expect that 80 percent to 92 percent of experiments in optimized domains will fail. Calibrating team expectations around these industry standards prevents discouragement and maintains high testing volume.

Skip testing for low-risk best practices

From "When NOT to run an experiment – Issue 54": "If you can run experiments quickly and easily (e.g. a few hours), this decision is generally easy: run the experiment. If running experiments is a pain in the butt, and the changes are relatively benign, you can probably skip the experiment."

Shipping directly is often superior to experimenting when the time required for statistical significance outweighs the data value. Avoid formal tests for standard industry practices where downside risk is minimal.

Apply Twyman's Law to surprising wins

Ronny Kohavi: "We can talk later about Wyman's law, but that was the first reaction, which is, 'This is too good to be true. Let's find a bug.' And we did. And we looked for several times, and we replicated the experiment several times, and there was nothing wrong with it."

Treat any result that looks too good to be true with immediate skepticism. Conduct rigorous bug-hunting and replicate surprising results multiple times to ensure they are not technical flukes.