Sample Sizing Cheat Sheet for Startups
...And For Most People Doing UX Research In the Real World
Summary: This article shares sample size recommendations for the five UX research methods I see most commonly used in the startup world.
For the first 15 years of my career, I worked on large UX teams in large organizations. Recently, I've transitioned to the startup sector, which led me to develop this cheat sheet. Many of these recommendations are controversial and counterintuitive but have been proven valuable in the real world. This cheat sheet is currently in use at one large fintech org, three medium-sized software companies, two pre-profit startups, all of the tech startups coming out of Penn State's startup incubator, and countless clients in my consulting agency days. It aims to streamline decision-making, focus on strategic, fast-paced development, and optimize for short-term gains. I hope you find this cheat helpful sheet as well!
📔 NOTE: Want more information on sample sizing strategies? In that case, I recommend reading the content found at the resources links in the "Reasoning and Resources" section at the end of this article.
TL;DR
I made an explainer video to accompany the cheat sheet for one of the startups mentioned above. Watch this snippet for additional context on the cheat sheet below.
Sample Sizing Cheat Sheet
For the purpose of this cheat sheet, I’ve broken these recommendations into 5 categories:
Discovering Problems or Insights
Benchmarking
Comparing Designs
A/B tests
Surveys
📔 NOTE: Curious how I came up with these recommendations? Go to the “Reasoning and Resources” section at the end of this article to learn why sample sizing is both an art and a science, and how most online sample size calculators can be incredibly misleading.
1. Discovering Problems or Insights
For methods like moderated and unmoderated usability test. (finding UX issues, not benchmarking)
For methods like:
User interviews
Diary studies
Field studies
Concept tests
Contextual inquiry
Qualitative preference tests
📔 NOTE: If you can't get 5 participants, I encourage you to conduct the test anyway with fewer participants. Just understand that your confidence in the findings will be lower when making decisions.
2. Benchmarking
For methods like:
Benchmarking usability tests (tests that must include metrics for completion rate, time-on-task, and satisfaction.)
Tree tests
Card sorts
Forced ranking
📔 NOTE: If you can't get 10 participants, I encourage you to conduct the test anyway with fewer participants. Just understand that you won't be able to report your findings as percentages, but anecdotal evidence can still be useful.
3. Comparing Designs
For methods like:
Usability test series for comparison
Survey series for comparison.
📔 NOTE 1: Between-subject studies are good for benchmarking data intended for comparison. Within-subject studies are ideal for finding data about a design’s learnability and memorability.
📔 NOTE 2: If you can't get 11 participants for between-subjects or 9 for within-subjects, I encourage you to conduct the test anyway with fewer participants. Just understand that the findings won't be reported in percentages, but you'll still gain valuable insights.
4. A/B Tests
📔 NOTE: If you can't get 11 participants in each group, I encourage you to conduct the test anyway with fewer participants. Just understand that the results won't be statistically significant in terms of percentages, but anecdotal evidence can still inform decisions.
5. Surveys
Standalone surveys (not involving comparison)
Quantitative preference tests (binary)
📔 NOTE: If you can't get 18 participants for ratings-based surveys or 21 for binary questions, I encourage you to conduct the test anyway with fewer participants. Just understand that you won't be able to report findings as percentages, but the qualitative data you collect will still be valuable.
Reasoning and Resources
Sample Sizing is an Art and a Science
Determining the right sample size for a project is an art and a science. These recommendations are calculated taking three main strategic considerations into account along with statistical rigor.
📔 NOTE: strategic = art, statistics = science
The strategic considerations for this cheat sheet are based on the current types of research methods I see used most in startup organizations. They are predicated on the following principles:
Be scrappy. Do More with Less.
Err on the Side of Decision. Test, Iterate.
I need more data addressing broader UX concerns before gathering more data around narrower concerns.
📔 NOTE: The good news is that broad UX data collection is best done with smaller sample sizes.
Online Calculators Can Be Misleading
Most one-size-fits-all sample size calculators online are often misleading and typically give very large sample size recommendations. This happens because they assume:
Your study is designed to gather binary data, which requires the largest sample size of all the research methods I use.
Your sample size will need to be in the hundreds, which is rarely the case in usability studies. This creates margins of error that are far too wide, rendering the data less useful in the real world.
The all too commonly misunderstood statement of, “I need confidence levels the same as standard polling, which is +/- 4 margin of error.” (This standard is wayyyy too restrictive for most business decisions in the real world.)
📔 NOTE: I use the interactive sample size calculation system designed by Jeff Sauro of MeasuringU. This system is specifically designed for usability-related research done in real-world scenarios.