A/B testing is the most widely used and most widely misused tool in product management. It's easy to run a test. It's hard to run a test that actually tells you something real. Here's the honest guide to doing it right.
The most dangerous A/B test I've seen wasn't one that had bad results — it was one that had good results that weren't actually valid. The team had run a test on their checkout flow. Variant B showed a 12% improvement in conversion rate with p < 0.05. They shipped it. Conversion rate improved for two weeks. Then it returned to baseline. The test had detected a novelty effect, not a genuine improvement. They'd run it during a holiday promotional period when buyer intent was elevated for external reasons. The A/B test design hadn't accounted for any of this.
This is what good A/B testing prevents. And it's what most teams aren't doing.
The Prerequisites Most Teams Skip
Statistical significance is necessary but not sufficient
Most teams treat p < 0.05 as the signal to ship. It means your result is unlikely to have occurred by chance — but "unlikely by chance" is not the same as "definitely real." You also need adequate sample size (pre-calculated, not post-hoc), a consistent testing period that spans at least one full user behavior cycle, and guardrail metrics to ensure your improvement on one metric isn't degrading another.
Test one variable at a time
If you change the headline AND the button color AND the image AND the layout in your variant, you can't know which change drove the result. This is basic experimental design and it's violated constantly in product A/B testing because teams want to test their big redesign instead of individual elements. Multivariate testing exists for this purpose — but it requires much larger sample sizes.
Define success metrics before running the test
Defining what you'll measure after you see the results is how p-hacking happens. Decide before launch: what is the primary success metric? What are the guardrail metrics that must not degrade? What threshold of improvement is meaningful enough to ship? Document this. Then run the test. Then look only at those metrics in evaluating the result.
What A/B Testing Can and Can't Tell You
A/B testing can tell you: whether variant B performs better than variant A on your defined metrics, within the context of the users who were exposed to the test, during the period of the test.
A/B testing cannot tell you: why the result happened, whether the result will hold for users not in the test, whether the result will persist over time, or whether you were testing the right thing in the first place.
The PMs who use A/B testing most effectively treat test results as confirmation of hypotheses, not as answers. When variant B wins, they ask: why did it win? What does this tell us about user behavior? What follow-on hypotheses does this generate? The test is a step in a learning cycle, not a decision-making machine.
Building your experimentation practice?
Good experimentation is one of the highest-leverage capabilities a product team can build. I've helped teams move from ad-hoc testing to systematic experimentation. Let's talk about what that looks like for your product.
Work with Yogesh →