The Reasons for User Testing

"We are what we repeatedly do. Excellence, then, is not an act, but a habit." -- Aristotle

User Tests Are Like Fitness Exercises

Nobody on Earth would seriously claim that doing fitness exercises is a bad idea. Well, the same goes for user testing -- at least we haven't met anybody yet who said that it would be a bad idea to test their products and services with real people.

While everybody seems to be convinced that doing more testing would make sense, very few people put theory into practice. Thus, we must address this fundamental problem before describing how exactly user tests work. The question to be answered here is: Why does everybody believe that user testing is important, but actually they don't conduct any (or too few) user tests?

More than ten years of practical experience have shown us that most companies just start to think about user testing when they are already experiencing serious problems. You might see these companies as patients who have been told by their doctor that they have health problems (customer complaints) and therefore need to pay attention to their health (product usability). They are forced to change their behavior, eat healthier, quit smoking, and start exercising (conduct user testing). However, this profound change in behavior is far from easy for most people, leading them to prefer quick fixes or pseudo-solutions, or alternatively, make excuses for not having to change their behavior in spite of everything.

An example of such a quick fix would be to hire an expert to conduct a usability study to identify all the problems of your product. While there's nothing to stop you from doing so, at this point we need to make it clear that a single usability study -- no matter how many people you conduct the test with -- will not actually help you to improve your product. It would be the same as thinking that it would suffice to go to the gym only one day, train like a madman using all the equipment available and suddenly be absolutely fit. The truth is that being really fit, i.e. developing a really simple, clear and understandable product, is an ongoing process.

Once you realize that user testing is not a "once-only and that's it" activity, and that the idea of solving your problems by means of a single usability study is just as unrealistic as the idea of "getting a perfect body in 14 days", you have two options:

1. You change your behavior fundamentally and integrate ongoing user testing into your way of working; or

2. You find excuses for not changing your behavior.

In this book, we describe a method to fundamentally change your behavior and show you how to integrate user testing into your way of working as a fixed component. But before we start, here's a list of typical excuses we keep hearing about why right now isn't a good time to start user testing:

• Unfortunately, it's too late to test for our current project.
• It's still too early to start testing now. (A classic one!!)
• Our usability is currently so bad that it makes no sense to test now.
• Besides, we have no time, too much stress and a too small budget anyway.
• If we actually start with this, we want to do everything the right way, of course.

This list of excuses can continue indefinitely, but we'll stay with the last-mentioned excuse, because it provides us with a perfect transition to the second chapter: About Usability Experts and Bodybuilders.

About Usability Experts and Bodybuilders

The biggest mistake (actually the only "real" mistake) that can be made when carrying out user testing, is not to start testing. And since one of the main reasons for not testing is that you don't know enough about user testing to get it right, let's address this excuse right away and get it out of the way, once and for all.

The truth is, you don't need to know anything about user testing to get started. So, now it's out! All you really need to do is show your product to somebody and ask them to use it while thinking out aloud. You'll find out how exactly this works in the second part of this book, with examples in the third part, but as said previously, you shouldn't worry about whether you do everything right or not: just do it.

And of course, we can imagine how a usability expert who's just read the previous paragraph has a fit and throws this book into a corner in rage, turns red and his eye starts to twitch, while thinking that we are completely crazy. How can we dare to say that you needn't know anything about user testing? After all, there are experts who do nothing but engage in user testing throughout the entire day. Are we saying that these experts are wasting their time? Well, no, we actually don't want to do that. We just want to show you how user testing really works in practice, and we believe that you don't have to be a usability expert to conduct user tests, just like you don't have to be a bodybuilder to do fitness exercises.

However, one thing is clear: Usability experts have extensive knowledge of user testing and can use this method to do things that are virtually impossible for lay people. For example, through skillful moderation of a user test, experts can gain insights that somebody without experience would stumble on by accident at most. Experts also know the difference between "formative" and "summative" tests, and they can set up scientific experiments to verify in a statistically relevant way whether variant A is better than variant B, or vice versa. They record so-called "task completion rates" and measure the time that different test participants need to complete individual test tasks. Of course, experts also know how to present and summarize their results with the help of beautiful graphs and diagrams and submit them to their customers and clients in so-called "usability reports"...

Oh yes, usability experts know so much, you could fill entire books with their knowledge. In fact, many of them have done it, and there is more than enough to read about these topics, which is why we can refrain from repeating this information here. Instead, we'll focus solely on how you can continually improve your solutions yourself by means of user testing. And you'll see that you need to know relatively little to achieve just this goal -- something you've probably already realized with the size of this book.

But before we move on, just a clarification: Of course you can conduct user tests to measure whether variant A is better than variant B, but we advise against it. Firstly, because you have to know very well what you are doing and work with the care of a scientist, as otherwise you won't be able to trust your data and can easily get to wrong conclusions and make bad decisions (which you falsely believe are "proven"). Secondly, this kind of scientific experiments is so expensive, that in practice almost nobody has the resources to carry them out on a regular basis, making this method completely unsuitable for actually improving products.

Making Mistakes = Learning New Things

Making "mistakes" is something inevitable when in search of knowledge. In fact, the willingness to make mistakes is an absolute prerequisite for learning something new. And actually that's what user testing is all about: discovering the mistakes of the current solution and learning something new from these mistakes to improve the solution in the future.

However, some people take user testing too literally and consider it a kind of test or exam. They believe that it is a question of getting a result as good as possible in these tests and, ideally, pass them with distinction. Many are actually scared of failing these tests, and this subconscious fear of failure is the main reason why most product teams don't conduct user tests regularly.

Thus, in this chapter we need to briefly address this subconscious fear of failure and identify it as the main reason why many product teams conduct no user tests or not enough of them. It's important to become aware of this fear and to accept that you -- like almost every other person -- are afraid of failing. The reasons for this don't actually matter and it's sufficient to admit that you're afraid of failing or of not being good enough. And if you simply don't see how this is possible, just think about how nervous you are before an exam or before giving a public presentation, or when telling somebody that you love them for the first time. The nervousness in these situations actually arises from the same subconscious fear of failing or not being good enough, which also keeps you from conducting regular user tests.

So, how can we overcome this fear and make sure that user testing really works in practice? The easiest way we know is to start by being aware of this subconscious fear (done!) and then keep reminding yourself that making mistakes is the only way to learn something new.

Just imagine that you won't make any mistakes for the rest of your life: wouldn't that mean that you'll never learn anything new for the rest of your life? Sure, nobody likes to make mistakes and what we'd like most is to avoid them altogether and get everything done right the first time, but this is not only unrealistic but also undesirable, because it would simply mean that you're arresting your own development. And of course, our goal isn't making mistakes: Mistakes are just the inevitable result of trying something new. And trying something new is, in our opinion, a pretty good description of what developing a product means.

Just Take the Stairs, Step by Step

You know something funny? People who take the elevator to get to the gym on the 2nd floor. It's even funnier when they get out of the elevator and get onto the step machine to start their workout...

As (at least we hope so!) has been clarified in the previous chapter, you don't have to be an expert to conduct user tests. Well, in this chapter you'll learn that user tests are actually much easier than you would ever have imagined (even if you already have experience with user testing). Our secret tip: Just take the stairs, step by step!

What we'll show you now is the most basic user test we know. This kind of user test is so obvious that you can actually compare it to simply taking the steps rather than taking the elevator. But just like taking the stairs, most people don't even dream of using this kind of user test. We explained earlier why, when we talked about people being afraid of making mistakes. But since we've also said that making mistakes is the only way to learn something new, we're now going to show you probably the simplest user test in the world with an example.

Imagine that you or your team are currently working on the new marketing website for your product. In fact, someone from your team has already made a first draft of the new page -- let's just assume that it was you. What can you do now to turn this design into a finished page?

There are two possibilities:

1. You take the elevator; or
2. You take the stairs.

Taking the elevator is like trying to convince somebody that your design is so good that it should be implemented. Similarly to an elevator pitch, you'll try to sell the benefits of your design to your listeners. While there's absolutely nothing wrong with showing people the benefits of your design, you're missing the opportunity of improving your design already early in the process with the help of user tests.

What does the alternative look like? Well, instead of trying to convince somebody about the goodness of your design, we suggest that you simply ask one of your friends if they have a few minutes to help you out by saying something like this:

"This is a current draft of something we're doing at work. Would you
mind taking a quick look at it, if you have the time, and just speak
out while you're looking through it and tell me everything that comes
to your mind?"

Once you've said this, you'll show the current draft to your friend and remind them to just think out loud or say anything that comes to their mind when looking at your design. Also tell them to be critical, because you want to find out what you can improve. As soon your friend starts to speak, you'll feel the urge to respond to their comments and defend your design (especially if those comments are negative), so it's important to remember not to do this at this time. Instead, you should simply shut up and say nothing, maybe nod and utter something approving like "mmm", "I see" or "okay". And if your friend asks you how this or that works, you shouldn't even try to explain your solution -- just ask how they imagine how this or that should work.

Done! You've just conducted the simplest user test in the world. And all you had to do was to show your work to somebody, and instead of explaining or defending your solution, simply ask that person to tell you how they think your solution works. As you'll probably have realized, it's harder to test your solution and possibly hear negative comments than simply present your solution, but that's why we're talking about taking the stairs instead of the elevator.

As taking the stairs is only the beginning anyway. In the second part of this book we'll show you all the steps needed to perform user testing on a regular basis.