Practical Examples for Test Scenarios

"Tell me and I will forget. Show me and I may remember. Involve me and I will understand." -- Confucius

Tips & Tricks for Test Scenarios

As we've made clear a few times in this book by now, you don't have to be a usability expert to improve your product through user testing. How many usability problems you'll find in your tests depends primarily on your test scenario. In the following chapter, you'll find helpful tips and tricks for useful test scenarios that we've collected over the last decade by creating and assessing thousands of user tests. Additionally, you'll find pre-formulated test scenarios for different types of webpages that you can easily copy and use as a basis for your own tests.

Always Start with the Context

This is one of the most important tips for creating your test scenarios: Your test participants should be able to empathize with the situation as well as possible and be influenced as little as possible. They should behave as they would do outside of your test situation. The best way to do this is to describe the most realistic situation (the context) in which your product is to be used.

Thus, before starting with your first, specific test tasks, you should share with the testers the situation they are in when dealing with your product. The following structure is well suited for this:

• Introduce yourself...
• You would like to...
• That's why you have to...
• This leads to...

An example:

Imagine that you're planning your summer vacation for your wife
and six-year-old son. You'd like to spend your 14 vacation days
in Greece this year, because your neighbor told you about the
beautiful sandy beaches there. That's why you're looking for a
suitable hotel for you and your family. You're considering the
last two weeks of August for your vacation, because your wife
has also got leave during this time.

After the context has been explained, we can finally proceed with the first specific task, such as booking a hotel with a good price-performance ratio in the specified period. The big advantage, if we explain the context well, is that these three sentences are often sufficient for our test participants to derive details for our test tasks on their own, and that we can formulate them in a very open manner, which in turn leads to our not influencing our participants too much with our test tasks. Thus, for example, in our task we no longer need to include finding a "family-friendly hotel". We can leave our testers to get to this conclusion in their own words in view of the context.

Don't Ask About Usability

It may sound contradictory, but the worst way to test the usability of your product is to ask your user. Maybe that's why it's one of the most common mistakes beginners make with their first tests.

Wrong:

Please rate the shop for usability: Is the shop user-friendly?

Wrong:

How easy to use is our website?

Wrong:

Is the search form intuitive for you?

Having users rate the usability of a website doesn't work. There are three reasons for this:

• People would rather tell you what you want to hear or what they consider you expect them to tell you in this situation.
• Human memory is faulty and highly selective; for example, we like to remember positive rather than negative experiences.
• People rationalize their behavior. "I was distracted at the beginning, but now I fully understand it."

To learn something about the usability of your product, you have to give people a representative task and watch them while using the product. Thus, in the end, you're the one who assesses the usability of your product based on data.

Right:

Imagine your girlfriend's birthday is coming up next week
and you're looking for a gift. You plan to spend a maximum of 50 €.

Don't Ask About the Future

What's more obvious in a user test than to ask if people would use your offer also outside of the test situation, i.e. "in the real world"? In the last months, we've seen a lot of tasks that were very similar to this one:

Wrong:

Would you buy something on this page in the future?

Don't ask your subjects about what they will do in the future, because they simply don't know. On the one hand, people overestimate how much time they actually have in the future, and they also underestimate the effort actually required to use your product. Thus, the question about a hypothetical future won't give you any usable results. Instead of asking about the future, ask about the past. This way you'll learn about actual behavior of your test participants in the past. And past behavior is the best indicator of a similar behavior in the future.

Right:

Have you purchased a similar product in the last 3 months?
If yes, please briefly describe your experience. If not, please
explain briefly why not.

You'll learn a lot more if you understand what has influenced the behavior of your test participants in the past than if you make them explore the crystal globe for a nebulous future.

Don't Ask for Opinions

User testing is not about asking people's opinions, but about discovering usability problems through their behavior. Thus, you should absolutely avoid questions like the one below:

Wrong:

How do you like this page?

Even if five test participants answer this question, we won't get any usable results. The fact that all five of them find our site visually appealing isn't representative with such a small number of individual opinions. If you really want to know how good your page looks, send a survey (a quantitative method) to a statistically representative number (preferably hundreds) of people. User tests are not the right method to conduct market research -- even if many people believe it is, they're wrong.

Avoid Leading Questions

Leading questions influence test participants by providing the solution to the question. Since we want to influence our participants in our tests as little as possible so as to observe the most realistic possible behavior, we should take care to avoid such leading questions:

Wrong:

Did you understand that you can filter for specific
products on our website?

This task is leading in two ways. On the one hand, it will be difficult for your testers to tell you that they "didn't understand" something -- even if they actually didn't -- because nobody likes to admit one's own incomprehension. On the other hand, by expressing this question, you are implicitly saying that it's possible to filter for products on your site. Thus, with your question, you're anticipating the answer. A better way would be like this:

Right:

Try to get displayed only those products that you're
actually interested in. Please don't use the search feature
for this purpose.

So, you see, we don't need to talk about "filtering" at all. This way, we don't give our test participants any indication about the solution to the problem, but simply observe whether they use the filter function on our site (if they have the task of displaying only those products that "actually interest them").

Avoiding leading questions is often not so easy and it takes a little finesse to set tasks so that they anticipate as little as possible.

To get as much out of your tests as possible, in the following chapters you'll find formulated test scenarios that you can use for different types of websites.

E-Commerce

Online shops are among the most rewarding things to test, as small usability weaknesses can already have a huge impact on the conversion rate of the entire website. Generally, you should start by testing your checkout process, from the shopping cart to registration, entering address information and selecting payment methods. Don't worry, your test participants needn't actually make a purchase on your site to get usable results. We simply tell them to stop their test on the last page, where your customers normally pay.

Below you'll find more than 20 test tasks suitable for user testing of online shops:

Product Range

Think of a product that you can purchase on this site. Please
say aloud which product it is. Now try to find the mentioned
product on this page. Stop as soon as you've found a product
that you'd purchase.

Find two similar products that you'd like and compare them to
each other. Explain the differences between the products in
your own words.

Compare the cheaper product to the more expensive alternative.
Explain the differences in your own words.

Filter

Find the cheapest product in the category [TREKKING SHOES]
that you like.

Please limit the list of products so that you only see those
products that you are actually interested in. Please don't use
the search feature for this. Explain why you restricted your
selection to the selected criteria.

Search

Think of a product that you would like to purchase on this
website. Now use the search feature (DO NOT use the menu) to
find the product you're looking for.

Please use the search feature to find a suitable product in
[CATEGORY X] for you. Please don't click on a product, but
explain what the search results you receive are and if the
displayed products are in line with your idea.

Shopping Cart

Find out how many items you currently have in your shopping cart.

How much would ordering all the items in your shopping cart cost?

Remove the item that you like the least from your shopping cart.
You've received the coupon code [YSASK]. Add the coupon code
to your order. How much do you save?

Checkout

Purchase the product you just found. You can use sample data.
Go as far as possible without entering any payment details.
Continue with the next task when you are done.

Returns

After trying on [THE SHOES], you notice that they are a bit too
small. Please look around on the website to try to find out what
exactly you have to do to return the pair that is too small and
get a pair of the right size.

Support

Let's suppose you have problems with your order. How do you
contact somebody from this website?

What would your preferred contact method be in case of problems
on this website? Please explain briefly why.

What are the costs associated to the different contact options?

Shipping and Fees

What shipping fees apply to a delivery at your home?

Are there any shipping fees when placing an order on this page?
If so, what are the shipping fees you'll have to pay?

Are there different shipping fees on this page?

You need your order very urgently. Are there any possibilities
to speed up the shipping? What additional costs do you incur?

Payment Methods

How can you pay on this page?

Which payment method would you choose? Please explain briefly
your decision.

Does the shop seem trustworthy? Please explain briefly your answer.

Navigation

Read out aloud the navigation of the page. Please don't click on
anything yet and don't hover the mouse over the menu items. What
do you think is behind every single menu item? Please describe
briefly your expectations about the content of each individual
menu item.

Now look at the individual menu items -- do they actually contain
the things you suspected?

FAQs

What questions would you pose when using this page for the first
time? Now try to find the answers to three of your questions.

Competition

Please look also at the products on this page. Now go to Google
and find another company offering similar products. Compare the
two websites. Please explain at least three differences. Which
page do you prefer? Please explain briefly why.

SaaS (Software as a Service)

Optimizing SaaS products with the help of user testing is almost as rewarding as testing online stores. The subscription-based sales model constantly delivers updates, bug fixes and new features to paying users. Thus, the product changes frequently and SaaS is perfectly suited for ongoing user testing due to this continuous development process.

As with any other kind of page, we advise you to begin the first rounds of testing with your most important use cases. Usually these are account registration, subsequent subscription and initial setup of your software. If you decide to use remote user testing tools for your tests, we also recommend that you set up a test that will automatically send you new videos about your registration and onboarding process from a handful of testers at least every 14 days (with a tool like Userbrain this is quite easy). This proactive test plan ensures that you quickly recognize usability problems in your page's key processes, rather than accidentally stumbling upon problems that may have already cost you a lot of money.

Below are some examples of test scenarios for SaaS products to be combined as you wish:

First Impression

Without leaving the homepage -- what's your first impression
of this page? Please explain your answer.

Without leaving the homepage -- explain what is being offered
on this page. Please be as specific as possible.

For whom is the offer on this page intended? Which is the main
target group of this page?

Which three words would you use to describe this page? Please
explain briefly your answer.

Which would be the first thing you'd click on this page? Why?

Explanatory Video

Please watch the explanatory video on this page. Please
summarize in your own words what is offered on this page after
watching the video. What's your impression of this video?

Registration

Register on this site (you can use fake data). Stop before
entering payment information. Then proceed with the next task.

Navigation

Read out aloud the navigation of the page. Please don't click
on anything yet and don't hover the mouse over the menu items.
What do you think is behind every single menu item? Please
describe briefly your expectations about the content of each
individual menu item.

Now look at the individual menu items -- do they actually
contain the things you expected?

FAQs

What questions would you pose when using this page for the
first time? Now try to get the answers to three of your questions.

Find instructions for [FUNCTION] on the page. Read at least
the first paragraph of these.

Prices and Payment

Please explain the prices of the service packages on this page
in your own words. Are there any differences between the
individual service packages? If so, please describe the
differences in your own words.

How can you pay on this page?

Please find the cheapest package on this page and explain
exactly what you get for this price.

Contact

You have a question about pricing on this page. What's the
quickest way to get somebody to answer your question?

Refunds

Suppose you aren't satisfied with the service offered on this
page. Find out what you can do on this page to get your money back.

Please summarize three things that you've learned about the
conditions for a refund of this page.

Login and Password

Please use this login information to log in to the page:
[SAMPLE DATA]

You've forgotten your password. Find how to solve this problem
on the page.

Competition

Do you know any websites or companies that offer a similar
solution to the one offered by this page? If so, please explain
briefly how they compare to the page you are looking at.

Please take a look at the products offered on this page. Now go
to Google and find another company offering similar products.
Now compare the two websites. Please explain at least three
differences. Which page do you prefer? Please explain briefly why.

Marketing Websites

Under Marketing Website we summarize all types of sites promoting a company's services. This category includes websites belonging to local stores, lawyers or other service providers who want to make their services available on the Internet. The economic purpose of these sites is either to attract customers to the premises (in the case of local stores) or to establish a first contact with the future customer/client (in the case of service providers). Thus, you should especially check these processes in your first rounds of testing.

Below are some tasks that are well suited for the testing of marketing websites:

What service is offered on this page?

Please explain in your own words the differences between the
offered service variants.

Does this page look trustworthy to you? Please explain briefly
your answer.

You have a question about the product range and would like to
get in contact with this company. Please do it now (you can use
fake data).

You wish to reach this company by phone. Which number would you
choose?

Do you have the impression that this website has given you
enough information to purchase this service? Please justify
briefly your answer.

You are urgently looking for [PRODUCT X] -- find out until when
this store is open today.

How can you pay in this store?

You would like to visit this store next Saturday. Find out where
it is and the quickest way to get there.

Do you know any companies or websites offering a similar
service? If so, please explain how they can be compared to the
offer on this page.

Find on Google the website of a company offering a similar
service. Now compare the websites of the two companies. Name
at least 3 differences. Which company would you choose? Please
justify briefly your answer.

Prototypes

We advise you not to wait until your product is fully operational to conduct user tests, but to already test very early prototypes. The reason for this is that changes to a prototype are done much faster and cost significantly less than having to correct a finished product. This is why user testing in the prototyping phase is one of the most efficient and cost-effective ways to improve your product.

You can test almost everything -- clickable PDFs, single images of a page, wireframes or interactive click dummies. Below are some tips on getting the most out of your prototype tests:

• Not everything has to work in your prototype. Focus on the most important use case of your product and start with just this one.
• Even if not everything works in your prototype, you can ask your test participants what they expect from things that aren't ready yet. Thus, you can find out if your solution would match their expectations.
• Explain to your test participants right from the start that they're using an unfinished prototype where some things aren't working yet.
• Make sure that your pages are linked correctly to each other and that these links are present on every page in your prototype, because otherwise your test participants might "get stuck" on a page when testing.
• Don't include links outside the prototype, because otherwise your test participants will get lost on other pages and may not find their way back to your prototype.
• Use real data and real names and no wildcards like Lorem Ipsum -- your test participants will treat any text in your prototype as if it were final.
• Make sure your content is consistent on the different pages -- for example, if the logged-in user on one page is called "John", it should be the same on all other pages.
• Provide login information, should your prototype be password protected.
• Some prototyping tools, such as InVision or Axure, provide controls such as a commenting function. Hide all distracting elements, if possible, because your test participants might think they are part of your solution.
• In addition, prototyping tools such as InVision highlight clickable areas with a brief flash when first invoked. You should also switch off this function in the settings so as not to influence your test participants.

Below is an example scenario for a prototype that we often use ourselves to test early versions:

You are now looking at a very early prototype of a website.
There are still many things that are missing, not working, or
wrong. Not all sections of the website are included in this
prototype.

Please try to perform the tasks indicated using this prototype.
If you land in a section that isn't ready yet, please explain
briefly what you would have expected to find there.

Should you get "stuck" on a page, you can use this link to
find your way back to the homepage [Link to the first page of
the prototype]

After this introduction, you can use sample tasks for the appropriate category of your page from the previous chapters. However, the less interactivity your prototype offers, the more care you should take when asking questions to your test participants. After all, it will be possible to do only relatively little with your prototype at first -- at least if you follow our advice and test your solution as early as possible.

Good questions to ask when testing your prototypes are, for example:

Please describe the individual items on this page. What can you
do with them?

What different pieces of information do you see here?

What would you expect to find in the "Gifts" category?

Please describe the individual steps you need to follow to ...

What do you think happens when you click on the link "Shipping
Options"?

What questions are you asking yourself at this moment?

What do you think could be done with this function?

What is the white icon in the right corner of the screen for?

Can this solution be compared to a product you've already used?

Is there something missing on this page?