ROI from UX: Fake Facts
“User experience design? Why should we pay you to make the work of our top-tier engineers more ‘user friendly?’ How do you even know what that means?”
Quite often, we have to make the case for user experience design. Usually, a business case. Usually briefly. What return can a company expect on investment in a better designed user experience? Too often we turn to very poor sources of information and present them with an overinflated authority based, in practice, on no one checking their facts.
Tom Gilb and the 1:100 ratio
Recently, a very popular factoid is that every $1 put into UX yields $100 in value. And that’s true because someone named Tom Gilb said so in a book called Principles of Software Engineering Management (1988).
But does anyone take a second to think about how dubious this statistic really sounds? The ROI from something that someone called UX in 1988? Is that the ROI in one case when the field was new, or across some range of cases, where and with what type of (pre-web and pre-smartphone) product? Is that the ROI from UX in particular, as distinct from marketing and engineering and any other overlapping department? Is that really the ROI in a media industry, where most projects bomb?*
This glib 1:100 “fact” is winsome, unless you think about it.
Now before telling you that this fact does not appear in that book at all, I’d like to point out the more important point that this ignorance evidences: popular explanations of UX are often rather weak on fact checking and have very poor citational hygiene. Where are you getting your ideas from? Is that a good source? Have you checked it at all? This is certainly the more important point and I’d love to hear what you think it means. Is it proof of success that no one checks the references? Proof of charlatanry? Something else? You tell me.
The Gilb book does not say that the ROI from UX is 100:1. I read it and pored over it for a week. No, this book is him inventing parts of agile, advocating Fagan inspections (QA for software developed at IBM in the first decade of the software industry), and providing templates for hypothetical cost/benefit ratios for project management. He presents almost no original empirical evidence and does not aggregate data from other studies. It’s not that kind of book. What does he say about predictive ratios like “UX pays 100:1?”
There is a great deal of interest in software cost estimation models […] I want to give an alternative point of view. The important question is, do you want prediction or do you want control? (p 312)
The problem lies in the fact that such general models will be inadequate for giving us the answers we need for project management, though they may give us a much-needed ‘jolt’, [stet] and some insights, before we are ready to get into the detailed work of our projects. (p 313)
Even when we have some kind of a specification of all critical and ambitious attributes, we have no formula for predicting the exact result in terms of cost. All experience clearly shows that we must be prepared to specify priorities, and allow attribute trade-offs as we get a more realistic view of the system through a design or development process. (p 313)
In fact, Gilb argues that any part of a project will yield diminishing returns when it crosses the “state of the art” borderline.
There can never, therefore, be a guide to the costs of ambitious levels of quality or required resource usage to obtain those levels. In fact, these models may give us the illusion that we have some idea of the cost, when a single over-ambitious, high-priority quality requirement may give us more than an order-of-magnitude of cost estimation error. (p 314)
In other words, you can’t know the ROI from UX as a general rule of thumb, because it depends on the project, but it is likely that if you haven’t done any you should and if you’ve done as much as everyone else it will be marginally less valuable to do more. Very general advice.
Really, we should not be relying at all on a software engineering management book about a consultant’s experience in the 1980s. If there is a good ROI from UX, there must be better evidence for it.
Pressman’s 1:10:100
Roger Pressman’s 1992 Software Engineering: A Practitioner’s Approach is an oft-cited follow-up to Gilb, but is no different. Pressman points out that many bugs in software are baked in from early on, and are cheaper to fix upstream than down. In particular, he cites his notes from a 1981 IBM course called “Implementing Software Inspections,” where errors that could be solved for $1 in design become $6 costs at coding, $15 costs at testing and $60–100 costs after product release. This is a nice series of numbers for showing orders of magnitude of cost in software architecture and QA, but is not very specific to UX and, again describes business realities that largely pre-date what we would now call UX. Software updates and patches were a huge hassle before the internet, but slipstreaming and early releases are now the norm.
In later editions of his book, Pressman also cites Secrets of Software Quality: 40 Innovations from IBM (1995); it reviews the same IBM findings. Pressman and Gilb both discuss another source that shows the same relationship, possibly from different data: Barry Boehm’s 1981 Software Engineering Economics. That book, never cited when it comes to UX, reflects his experience at TRW in the 1970s (focused in aerospace at the time) where Bill Gates got his first big break debugging software in 1970. This is before personal computing, and points out important facts that are not specific to UX.
This 1:10:100 ratio, popularized by Pressman but drawn from others, justifies investment in design and coding, but is based entirely on a world where software inspections, development methods, and management were still just being invented. Poorly structured data, poor module design, and antipatterns are more the costs that Pressman is pointing out. Not the UX value of easy to complete forms or easier to navigate menus.
The real ROI of UX
There is no consistent ROI from UX. As with all things, if it’s a problem with your product, you should try to fix it. If it’s already very good, maybe you’re ok.
Would you like to know more? Check out these sources, and read skeptically! The bigger problem is not that UX has no value, that’s false. The bigger problem is that those talking about UX make baseless claims freely and their readers ought to be more skeptical.
https://www.amazon.com/Cost-Justifying-Usability-Second-Interactive-Technologies/dp/0120958112
http://www.userzoom.com/roi-of-ux/13-questions-answered-about-measuring-the-roi-of-ux/
http://www.usabilitynet.org/papers/Cost_benefits_evidence.pdf
http://www.usabilityfirst.com/documents/U1st_BCO_CaseStudy.pdf
http://www.usabilitynet.org/trump/documents/Business_case.ppt.pdf
http://www.fastcodesign.com/1669283/dollars-and-sense-the-business-case-for-investing-in-ui-design
https://www.experiencedynamics.com/blog/2014/07/making-strong-business-case-roi-ux-infographic
*Most projects bomb across media industries. For movies, television, music, publishing, software, and video games, the rare hits cover the common flops. If you think about the margins on publishing any kind of media, and the returns of a hit, you can see why so many competitors produce the best content they can just for a shot at success.
