GOMS-KLM: An Old Research Method That's Perfect For AI
Calculations Template & Tutorial Video
Summary: This article explains the GOMS-KLM method for estimating time-on-task. It also provides a Google Sheets calculation template and a video tutorial, found at the end, to help you conduct GOMS-KLM audits on your own user interfaces.
I currently work for a small pre-profit startup in the commercial real estate space called Bowery Valuation, Bowery for short. Bowery's goal is to use technology to disrupt the old antiquated processes around real estate appraisals. Like most modern tech companies, Bowery has recently pivoted most of our efforts towards integrating AI functionality into our tech products. During this transition, our CEO approached me with a critical and time-sensitive challenge. He asked if there were any UX research methods that could rank features within our flagship product quickly and reliably based on the time users would save if their workflows were automated with AI.
That's when I introduced him and the rest of the organization to GOMS-KLM, a method I remembered from my early days in UX research.
To my surprise, none of my colleagues had even heard the term GOMS-KLM before, let alone used it in the real world.
I then started Googling. Surely there must be material out there teaching these things that I could share with my team. I learned that despite its utility, GOMS-KLM has gradually receded into the background. I tried looking for someone teaching GOMS-KLM calculation methods in boot camps, undergraduate programs, or even graduate programs but came up empty-handed. (With that said, NN/g has some GOMS-related training and materials, but it could be more in-depth.) This gap in modern UX education inspired me to write this article. Especially now, with the surge in AI integration, I felt a renewed affinity for this "old-school" method. Why? Because GOMS-KLM stands out for its unparalleled precision and efficacy in predicting time-on-task.
I'm convinced the rise of AI presents a golden opportunity to reintroduce GOMS-KLM into the forefront of UX research. In this article, I'll define and explain GOMS-KLM, share my experience using GOMS-KLM to revolutionize an organization's prioritization methods, and I'll share my custom GOMS-KLM Google Sheets template so you can conduct this method with your own projects.
What is GOMS-KLM Anyway
Let's start with the acronym GOMS. GOMS stands for Goals, Operators, Methods, and Selection Rules. It is a framework for calculating estimated time-on-task. GOMS provides an efficiency measurement that organizations can use to compare new and old designs or to compare against competitors' products. GOMS calculations can be conducted on any level of design fidelity. (paper prototyping to live code)
GOMS Breakdown
Goals: These represent what a user aims to achieve. Goals can be either high-level or low-level. For instance, a high-level goal like writing a book is achieved through numerous low-level goals, such as typing or deleting characters.
Operators: These are the perceptual, cognitive, or motor actions employed to achieve goals. It could be as simple as pressing a key, tapping an icon on the screen, or using a trackpad.
Methods: These are the procedures or algorithms of operators designed to accomplish a goal. It's the step-by-step process a user might take to achieve a specific outcome.
Selection Rules: These are the decisions an individual makes to select a particular method in a given context. It's the rationale behind choosing one method over another, depending on the situation.
Defining KLM
Now let's discuss the second acronym: KLM. KLM stands for Keystroke Level Model. It's a method used to quantify and assign specific time metrics to any user interaction. Essentially, KLM enables the measurement of the time taken for each operator within a GOMS model.
GOMS and KLM Combine
In summary, GOMS gives us the framework for calculating extremely accurate time-on-task estimates, and KLM gives us the specific GOMS operators needed for evaluating contemporary digital interfaces.
GOMS-KLM Facts
Origins: GOMS-KLM was pioneered over 30 years ago by esteemed researchers at XEROX PARC and Carnegie Mellon.
Replication: Its results have been replicated tens of thousands of times, amassing over three decades of invaluable usability research findings.
Accuracy: GOMS-KLM boasts the ability to predict a skilled user's task time with an impressive 10%–20% accuracy. For context, achieving this margin of error through traditional means would necessitate testing at least 80 users.
Cognitive & Motor Processes: GOMS delves deep, calculating both the cognitive and motor processes of users.
System Response: It doesn't just stop at user actions; GOMS-KLM also accounts for system response and render times, ensuring a holistic analysis.
Motor Processing Research: The model leans heavily on established motor processing research, incorporating principles such as Fitts's Law.
Cognitive Model: At its core, GOMS is based on the Human Information Processing Theory, a foundational model of cognition.
GOMS-KLM Limitations
While GOMS-KLM is a powerful tool, it's essential to know what it can't tell us:
Learning Curve: GOMS-KLM doesn't provide insights into the time users might need to learn the system.
Discoverability: It doesn't shed light on content discoverability within a system.
Errors & Difficulties: GOMS-KLM won't tell you about the errors users might encounter or difficulties they might face. (based on the user's “Happy Path.”)
User Satisfaction: While it's great for task times and processes, GOMS-KLM doesn't gauge user satisfaction.
3 Reasons Why GOMS-KLM is More Relevant Now Than Ever
Predictive Precision: GOMS-KLM's precision in predicting time-on-task is unparalleled. At Bowery, we rigorously tested its capabilities. We executed 42 distinct GOMS-KLM estimates on diverse UX designs. Post-launch, we juxtaposed our GOMS-KLM predictions with actual user analytics. The findings were compelling. Our GOMS-KLM predictions consistently hovered around a deviation of just 15% from actual times. This consistency underscores the reliability of GOMS-KLM in predicting user interactions.
Most of the actual times are within plus or minus 15% of the estimated times, indicating a close match between predictions and real-world performance. The table provides insights into the accuracy of the GOMS-KLM time estimates. Fast Calculations: GOMS-KLM operators and estimated times are easy to templatize and reuse. At Minitab, a leader in statistical software, we adopted a 'calculation template' methodology. Leveraging an extensive operator times database, we were able to produce GOMS predictions for various design options in a fraction of the traditional time. The Google Sheets template towards the bottom of this article is an advanced version of what we developed at Minitab.
Tailored for Agile: Agile methodology emphasizes adaptability and swift feedback. At Siemens Software, GOMS-KLM seamlessly integrated into our project sprints. When we were tasked with transforming a desktop application—designed for creating digital twins of manufacturing plans—into a virtual reality version, GOMS-KLM's time-on-task estimates streamlined our decision-making process. The outcome? Objective, data-driven decisions replaced lengthy, subjective team debates that often frustrated and demoralized the team.
How To Conduct a GOMS-KLM Audit
Here's a step-by-step guide to conducting a GOMS-KLM estimated time-on-task calculation:
Define the Real-World Workflow(s): Clearly define the task you want to analyze. It should be a specific, user-centered task that a user would perform with the interface. For example, "Create a new document in a word processor."
Identify the User's Goals: Break down the task into a series of goals that the user needs to achieve. These goals should be high-level and represent what the user is trying to accomplish. For example, "Open the word processor," "Navigate to the 'New Document' option," and "Create the new document."
Break Down the Goals into Methods: For each goal, identify the methods that a user could use to achieve it. A method is a sequence of actions or steps that a user takes to achieve a goal. For example, to "Open the word processor," a user might "Double-click the word processor icon on the desktop."
Identify the Operators: Break down each method into a series of operators. Operators are the basic actions that a user performs, such as keystrokes, mouse clicks, and mental operations. For example, the method "Double-click the word processor icon on the desktop" might be broken down into the operators "Move the mouse to the icon" (Pointing), "Select a button" (Select), and "Wait for the application to render" (Render). The template below is prepopulated with the most common operators, so you don't have to spend time researching all of these.
Assign KLM Times to Each Operator: Assign a time value to each operator based on the KLM operator times. These times are based on research and represent the average time it takes for a user to perform each operator. For example, the time for a "Pointing" operator might be 950 milliseconds, while the time for a "Select" operator might be 320 milliseconds. The template below is prepopulated with times vetted by at least 30 years of usability research, so you don't have to spend time assigning these operator durations.
Calculate the Total Time: Add up the times for each operator to get the total estimated time-on-task. Be sure to account for any overlaps or interactions between operators, as well as any potential delays or interruptions. The template below does these calculations for you.
Validate the Model: Test the model by having real users perform the task and compare the actual time-on-task to your estimated times, as I did at Bowery. Adjust the model as needed based on the results.
Analyze and Interpret the Results: Use the estimated time-on-task to identify potential bottlenecks or inefficiencies in the user interface. Consider how the interface could be redesigned to reduce the time-on-task and improve the user experience.
Iterate: As you make changes to the interface, repeat the GOMS-KLM analysis to assess the impact of the changes on the estimated time-on-task. The template below also provides comparison calculations for you to compare designs as they evolve.
Template & Video Tutorial
Download The Template
Subscribe, and I'll grant your email address access to the template. Once you've subscribed, you can access it using this link. GOMS-KLM Template Google Sheets Template
Conclusion
GOMS-KLM isn't merely a tool from the past; it's the future of UX research's impact on organizations. Let's champion GOMS-KLM for its ability to help guide organizations through this new AI paradigm of user task automation in a user-centric way.
For a deeper understanding of GOMS-KLM, "The Psychology of Human-Computer Interaction" by Card, Moran, and Newell is a recommended read. It offers an in-depth exploration of this timeless method.