How to Conduct a Multivariate Test Using Excel or Google Sheets
A Step-by-Step Tutorial
Summary: A/B testing is a commonly misunderstood and misused method in UX research. In many cases where UX professionals think an A/B test is appropriate, multivariate testing is actually the better choice. This article contains a step-by-step template for running your multivariate test with Microsoft Excel or Google Sheets.
A/B testing is often the first choice for UX professionals when comparing designs. However, it's not always the best for demonstrating how these designs truly impact user experience, and it's certainly not the most efficient method under a time crunch. A common misconception is that A/B tests quickly determine a 'winner' between two competing designs and can tell you why one was better. While this can be true, the method is frequently misunderstood and misused. The only way to fully understand why a design was the 'winner' of an A/B test is if you only change one variable between the two designs at a time.
I've seen too many UX researchers lose credibility by being unable to answer the question, 'Why did Design A perform better for users than Design B?' This is a simple question we should all be able to respond to confidently when testing design variations.
If you're anything like me, you won't have time to change just one element in a UI, test the variations, make that change, identify a second variable to change, test that, and so on. (Not to mention how the variables interact with one another as you go!) Instead, you'll most likely have two designs with multiple differences, all of which you believe will impact usability. That's where multivariate testing steps in, filling the gap and doing what many assume A/B testing is capable of.
Before going into how to run a proper multivariate test, let's define some terms:
Variable: These are elements that affect the user experience. It could be an aspect of user interaction, like choosing between a touch gesture or a click, or it could be a specific UI control, such as a button. (For some reason, many UX professionals forget the interaction variables and think only about aesthetics.)
Variant: These are the various implementations or versions of a variable. For instance, in the case of user input methods, variants might include options like a swipe gesture, a double tap, or a long press. When considering a button as a variable, its variants could be differences in size or style, such as a large, prominently styled button versus a small, text-only button.
Variation: This is the combination of variants across your variables, bringing a more dynamic dimension to the testing process. For example, a variation in the context of user interaction might be a swipe gesture paired with haptic feedback or a long press followed by an audio response. A variation for UI controls like a button grouping, could be a large, bold button combined with a minimalist, text-only small button. Each variation represents a unique mix of these elements, providing diverse experiences to test in your design.
By understanding these concepts, it becomes clear how multivariate testing allows for a comprehensive evaluation of multiple design elements simultaneously, revealing how different combinations can influence user behavior and overall experience. This method provides a depth of insight that can lead to more refined and effective design decisions, and it's almost as easy as running an A/B test.
Now, let's look at the step-by-step process of conducting a Multivariate Test, focusing on the crucial phase of data synthesis and visualization using tools like Google Sheets and Microsoft Excel.
General Process for Conducting Multivariate Testing
Identify Variables and Variants: First, choose the UI elements you want to test along with their different versions. Let's consider two simple variables as an example to help clarify. Let's say you want to test the layout of a product page (grid or list view) and the page's color scheme (light or dark theme).
Create Design Variations: Develop all possible combinations of these variants. In this product page scenario, there would be four combinations: grid layout with a light theme, grid layout with a dark theme, list layout with a light theme, and list layout with a dark theme.
Set Up the Test: Implement these variations on your live product-detail page. Each visitor to your site should randomly encounter one of these four variations. This ensures that your test results are not biased by any pre-existing user preferences or behaviors.
Collect Data: Monitor and collect data on how users interact with each variation. Key metrics could include time spent on the page, click-through rates, and conversion rates – such as adding a product to the cart or completing a purchase.
Analyze Results: Use statistical methods to analyze the data collected. This analysis will help you determine which combination of layout and color scheme leads to better user engagement and higher conversion rates.
Data Synthesis and Visualization in Google Sheets and Microsoft Excel
When synthesizing and visualizing data from a multivariate test in Google Sheets or Microsoft Excel, the process involves both analytical and communicative components. Here's a guide that encompasses data synthesis along with recommendations for effective data visualization:
Spreadsheet Setup:
Open Google Sheets or Microsoft Excel and create a new document.
Organize the spreadsheet with columns for each variation and rows for metrics like views, interactions, and conversions.
Data Entry:
Input the data collected from each variation of your multivariate test, including key metrics relevant to your test objectives.
Calculate Key Metrics:
Use a formula (typically =Conversions/Views) to calculate conversion rates or other relevant metrics for each variation.
Statistical Analysis:
Employ statistical functions available in both platforms, such as AVERAGE, STDEV.P, and T.TEST, to analyze the effectiveness of each variation.
This analysis helps in comparing and understanding the performance of each design variation.
Data Visualization:
Bar or Column Charts: Ideal for comparing the performance of different variations in a straightforward, easy-to-understand manner. Use these for showcasing conversion rates or other discrete metrics across variations.
Line Graphs: Useful if you're tracking data over time, such as the change in conversion rates throughout the duration of the test.
Pie Charts: Can be used for showing the proportion of total conversions or interactions each variation contributed to.
Scatter Plots: If you're analyzing correlations between different types of metrics (e.g., views vs. conversions), scatter plots can be insightful.
📓 NOTE: Create these visualizations by selecting the relevant data and choosing Insert > Chart in both Google Sheets and Excel.
By following these steps, you not only synthesize the data effectively but also create compelling visual narratives that can inform decision-making and guide future design strategies. The right choice of visualizations will significantly enhance the clarity and impact of your findings when presenting to stakeholders.
Communicating Your Findings
Interpreting:
Analyze the visualizations to draw conclusions about which variations performed best and identify any significant patterns or correlations.
Look for data that helps you understand the interaction between different variables and how they affect desired user outcomes.
Use these insights to make informed recommendations for design changes.
Reporting:
When preparing to communicate your findings to stakeholders, select the visualization(s) that most clearly demonstrates the impact of different variations on your key metrics.
Include brief descriptions or annotations in your charts to explain what the data shows, especially noting any significant findings or unexpected results.
Consider creating a dashboard or a summary page within your report that brings together all key visualizations and insights for a cohesive presentation.
Remember, the goal of multivariate testing and this data synthesis process is to identify not just the most effective individual elements, but also to understand how different elements interact with each other and influence user behavior.
Next Steps
Next Steps:
Implement the most effective design variation based on your findings.
Continue Testing:
Consider further research to refine other elements of the design.
By following these steps, you not only synthesize the data effectively but also create compelling visual narratives that can inform decision-making and guide future design strategies.
The right choice of visualizations will significantly enhance the clarity and impact of your findings when presenting to stakeholders.
Conclusion
It's crucial to recognize that while A/B testing has its place, it's not the be-all and end-all for understanding the full impact of design choices on user experience, especially under tight deadlines. The method, popularized by Silicon Valley charlatans for its ability to pinpoint a 'winner' between two designs, actually falls short of revealing the deeper 'why' behind one design outperforming another – unless, of course, you're only tweaking one variable at a time. But let's face it, who has the time for that?
📓 NOTE: When I say finding the 'why' behind things, remember that a multivariate test alone will not give you this; you will need some form of qualitative research as well.
As we move forward, remember that the goal is not only to identify the most effective design elements but also to understand their interplay and impact on user behavior. This continuous cycle of testing, analysis, and refinement is the key to crafting optimized, user-centric UX designs. So, as you tackle your next project, consider giving multivariate testing a go – it might just be the game-changer you need.