Top 5 A/B Testing Articles
My Thoughts on A/B Testing in One Place
Summary: This is a one-stop shop for my articles on A/B testing and related statistical comparisons. I pulled these links together so they are easier to find, read together, and share among teams. You won’t find this info, organized in this way, anywhere else on the web. Check it out!
I recently wrote a couple of articles about A/B testing, and after publishing them, subscribers started reaching out with a bunch of follow-up questions. I responded to a handful of them until I realized I had already talked about this stuff in older posts!
So rather than keep sending people five different links in separate replies, I wanted to put everything in one place. This post is basically a reference list for my thoughts on A/B testing, sample sizing, multivariate testing, chi-squared testing, and the ways experiment results can mislead teams when they are read too narrowly. The shared theme across these articles is that A/B testing is useful when it is framed well, sized for the real world, and the data isn’t weaponized or misinterpreted. Enjoy!
Top 5 A/B Testing Articles
A/B Test Strategies That Work
This article is about making A/B tests more useful before the test ever goes live. It focuses on the strategy behind the comparison, including how to define a better hypothesis, choose a meaningful outcome, and avoid running tests that look rigorous but do not actually help the team make a better product decision.
A/B Test Sample Sizing
This post is a companion piece to the A/B Test Strategies That Work article above. It explains why sample size matters, what inputs shape it, and how concepts like baseline rate, minimum detectable effect, confidence, and power connect back to the decision the team is trying to make.
Another Way A/B Tests Mislead
This article looks at how A/B tests can still mislead teams even when the test itself seems reasonable. It is mostly about interpretation, especially the risk of treating a winning variation as automatically better without asking what changed, what the metric actually captured, and what the result does not explain.
How to Conduct a Multivariate Test Using Excel or Google Sheets
This post is a practical walkthrough for comparing multiple variables instead of forcing every product question into a simple A/B structure. It shows how a multivariate test can help separate the effects of different design elements, and it keeps the method approachable by using tools many teams already have access to.
More Meaningful Comparisons With Chi-squared Testing
This article is about using chi-squared testing to make better comparisons when the data is categorical. That matters for A/B testing because a lot of product and UX outcomes show up as counts, choices, completions, failures, clicks, selections, or other patterns that are usually compared too casually with percentages alone.
Conclusion
I think these articles work well as a set because they cover different parts of the same broader problems I see in my day-to-day work. Knowing this stuff better is where UX researchers can really add a lot of value.
UXRs can help teams think through what is actually being tested and make sure our “move fast and break things” PM partners are not using A/B testing in ways that cause more harm than good.
If your team uses A/B testing, is trying to use it more often, or is trying to get better at interpreting experiment results, these 5 posts are a good place to start. They are not meant to make experimentation feel more complicated for the sake of it. They are meant to make the method more useful in the real-world conditions where product decisions actually happen.
Thanks for reading, and I hope having these articles in one place makes them easier to use. If any of them help you have a better conversation about A/B testing, sample sizing, or statistical comparisons in your org, I would love to hear about it.