One of the first questions to ask during your research phase is whether something similar to what you’re developing has already been done, albeit in a different way. For example, perhaps you are creating a mobile app for a service that already exists as a website. It’s important to examine what already exists to learn what works (or doesn’t). For VUI systems in particular, an interactive voice response (IVR) system version often already exists (sometimes DTMF [touch tone] only, and sometimes voice-enabled). This is a great place to discover what similar features are already out there and how they are handled.

Mobile VUIs should never be an exact replica of their IVR counterparts, but it is still possible to learn a lot from them, such as how concepts are grouped and which ones are used most frequently.

If there is a similar IVR system out there, it often means there is a human agent supporting it, and therefore a call center. Sitting in a call center for an afternoon listening in on calls can provide a wealth of information. Listening to real users call in can reveal things that a study of just the IVR system itself might not identify.

In addition, interviewing call center agents can also provide a huge amount of valuable information. They are the ones who know what users are really calling in about, what the biggest complaints are, and what information can be difficult for callers to find.

But perhaps you’re designing a mobile app that has no IVR component. What problem does your VUI solve? What real-world things exist that might help you gain insights?

Cortana, for example, is a virtual assistant. For inspiration, Microsoft interviewed real-life personal assistants. Designers found that human assistants often needed to ask for help when they did not understand an assigned task. Therefore, Microsoft made sure to have Cortana ask for help if she needs it, just like her human counterparts.^[34]

In addition, real personal assistants often carry around a notebook about the person they’re assisting; Cortana maintains a similar notebook.

Always interview people who are in the space you’re trying to serve. As an example, a company I was consulting for had decided to create a VUI app about Parkinson’s disease. The original idea was an app to help people with Parkinson’s, providing more information about the disease, helping with medication management, and other self-care tasks. When conducting user research, I also wanted to speak to those closest to Parkinson’s patients: their caregivers. When I did so, I realized that an equally important problem was helping the caregivers themselves! It modified the direction and feature set we had planned for building the app. I also reached out to support groups for both Parkinson’s patients and their caregivers. This goes without saying, but be respectful and transparent when conducting your user research—people are often willing to talk, and want to help, but be open about what you’re trying to accomplish. Don’t dismiss people’s ideas—you’re not making a promise to implement them, but you need to listen closely to the issues people are dealing with.

Whether doing early-stage testing, trying out a prototype, or conducting a full-blown usability test on a working system, it’s important to carefully define the tasks you’ll request your users to perform. As introduced in the book Voice User Interface Design:

Be careful when writing your task: you want to avoid giving away too much and instead provide just the essential information. As Harris says, “the scenario needs only a hint of plot.”^[36] Describe tasks in the way users would talk about them—don’t use technical terms, or things that give away key commands.

Harris suggests making the early tasks “relatively simple, even trivial, to put the participants at ease.”^[37]

Jennifer Balogh, coauthor of Voice User Interface Design, stresses the importance of task order, and that to avoid ordering effects, it’s best to randomize them:

A Latin Square design allows you to present each task in every position and have each task follow every other task without having to go through every permutation. For example, 5 tasks would result in 120 permutations. In a Latin Square design, there would only be five conditions, as presented in Table 6-1.

You can find more on Latin Square design by searching online.

As with all user testing, it’s best to sample subjects who are in the demographics of the people for whom you’re actually designing the system. If you are creating a health app for people with chronic heart failure, your test subjects should not be healthy college students. The further away from the true demographic your subjects are, the less reliable the results will be.

How many people should you test? If you get the right demographics, you don’t need that many. Usability expert Jakob Nielsen recommends five users for most types of testing. The reason five is enough? Here’s how he explains it:

Recruiting users can be tough. There are companies that can help you find subjects, but their costs are often prohibitive for smaller companies. If your company is large enough to afford a recruiting firm, it’s a very effective way to get users in the right demographic. Recruiting companies will require a “screener,” a document outlining the qualifications you need for your users. Questions might include age range, what type of smartphone they own, their level of expertise with certain apps, their geographic location, or anything else to ensure the right demographics.

For some testing, it’s also appropriate to recruit friends and family. That can be a great way to do some quick and dirty testing and get feedback on crucial aspects quickly.

Remote testing widens the net because you can recruit people who aren’t local. For quick and affordable testing, services such as Task Rabbit can identify people to do your study, although you can’t choose the demographics. There are also online services such as UserTesting (https://usertesting.com/) and UserBob (https://userbob.com/), both of which have a pool of users available for testing. There are some technical hurdles, however, as the screen capture software used is not always able to capture both the app’s audio (e.g., the virtual assistant speaking) as well as the user’s speech, and the user’s screen. You can overcome this by having the user record the session using a webcam to record the session, but this will limit the pool of users and can add to the cost of user payment, as well.

Be sure to compensate your subjects. Local subjects who come to you for in-person testing should generally be paid more. Subjects who represent very specific demographic groups will also have a higher rate.

The questions you ask are crucial to gaining insights from your user testing. Observational data is important, but the right questions can tease out important pieces of the user’s experience. In addition, subjects tend to skew on the positive side, especially during face-to-face sessions—most people want to be nice and will sometimes hesitate to provide negative feedback, or exaggerate the positive. It’s often unconscious, but a good interviewer can get past these issues.

Beware of priming your subjects when giving instructions. As Harris notes, when it comes to what to say, users take substantial cues from the person conducting the testing.^[39]

If possible, ask subjects a few questions after each task, and then a set of questions at the very end. The reason for this is because subjects might forget things as they move from task to ask. The best time to ask about first impressions is right after the very first task is completed.

Avoid leading the subject—let them explain in their own words. If a subject asks, “Could I have spoken to the app during that task?” you should first ask, “Would you want to do that?” If the subject says, “I didn’t like it,” don’t say, “Oh, because it didn’t understand when you said ‘show me my shopping list’”?” Instead, say, “Tell me more about that.” Do more listening and less talking. Pauses are OK; sometimes you’ll be typing or writing notes, and the user will add more information on her own.

When you begin testing, remind participants that they’re not being tested, but that they are there to help improve the system, and that you won’t be offended by their feedback. Don’t interrupt when you see them struggling, unless they become too frustrated to continue.

For quantitative questions, the Likert scale is commonly used. Here are some good guidelines about using Likert, L., and Algina, J.:^[40]

Table 6-2 presents a sample questionnaire, created by Jennifer Balogh at Intelliphonics, that you should ask at the very end of an in-person user testing session.

The questionnaire consisted of questions that probed seven different dimensions of the application (labeled categories): accuracy, concept, advice offered (content), ease of use, authenticity of the conversation, likability, and video flow. For each category, the questionnaire included both a positively phrased statement such as, “I like the flow of the video,” and a negatively phrased statement such as, “The video is too choppy.” Because likability was of interest in the study, three additional positively phrased statements covering this category were included in the questionnaire.

While observing your subjects (or looking at videos of remote testing), it’s important to take note of not just what the users did or did not do in the app, but facial expressions and body language. Did they laugh in a place that was unexpected? Did they frown when asked a particular question by the app?

Here are some other things to look for when conducting usability testing of a VUI:

If your user asks “What should I do now?” or “Should I press this button?”, don’t answer directly. Gently encourage the user to do whatever they might try when alone. If the app crashes or the user is becoming too frustrated, step in.

One of the first steps for early-stage testing of VUIs, after the initial concepts have been determined, is to create sample dialogs. As you might remember from Chapter 2, a sample dialog is a conversation between the VUI and the user. It’s not meant to be exhaustive, but it should showcase the most common paths as well as rarer but still important paths such as error recovery.

A sample dialog looks like a movie script, with the system and the user taking turns having a dialog.

Here’s a sample dialog for an app that allows the user (a child) to have a conversation with the big man himself, Santa Claus. This was written by Robbie Pickard while at Volio.

When you have a few sample dialogs written up, do some “table reads” with other people. One person reads for the VUI, and one for the user. How does it sound? Is it repetitive? Stilted?

Do a table read with your developers, as well. You might have some items in your design (such as handling pronouns or referring to a user’s previous behavior) that will take more complex development, and it’s important to get buy-in from the outset and not surprise them late in the game.

One common finding from table reads is having too many transitions with the same wording; for example, saying “thank you” or “got it” three times in a row.

As with any type of mobile design, mock-ups are a great way to test the look-and-feel of your app in the early stages. If you have an avatar, a mock-up is a good first start to get user’s reactions to how the avatar looks, even if it’s not yet animated, and even if speech recognition is not yet implemented.

Wizard of Oz (WOz) testing occurs when the thing being tested does not yet actually exist, and a human is “behind the curtain” to give the illusion of a fully working system.

WOz testing comes early. According to Harris, WOz testing is a “part of the creative process of speech-system design, not a calibration instrument for an almost-finished model developed at arm’s length from the users.”^[41]

To conduct WOz testing, you will need two researchers: the Wizard, and an assistant. The Wizard will generally be focused on listening to the user and enabling the next action, so this same person cannot be the one responsible for interviewing and note taking.

When I was at Nuance Communications, WOz testing was a common and low-cost way to test IVR systems before they were built. Because users were interacting over the phone, it was easy to simulate a real IVR system. We created a web-based tool that had a list of currently available prompts (including error prompts) in each step of the conversation, and the “wizard” would click on the appropriate prompt, which would then play out over the phone line.

We still had to design the flow and record the prompts, but no system code had to be written—just some simple HTML files (see Figure 6-1 for an example).

Figure 6-1. WOz screen for IVR testing

Conducting WOz testing on a mobile app is trickier, because it’s much more difficult to control remotely. Nonetheless, gathering user data before you have a fully working prototype can be valuable. Here are a few ways to do this type of testing before you have a working app:

Although one advantage of WOz is the ability to test an early prototype, the other advantage is that it’s much cheaper to make design changes before you’ve written most of the code than later on in the development cycle.

There is one major difference between WOz and usability testing when working with VUIs, and that is recognition accuracy.

When conducting WOz on a GUI, it’s usually quite clear where the user has clicked the screen, or what part they’ve swiped or tapped. There’s no ambiguity; you know that as soon as that button or list is hooked up to real code, it will do what is expected.

With VUI and natural-language interfaces, the Wizard has to do some on-the-fly interpretation and determine whether that’s something we could realistically recognize. Sometimes it’s easy and what the user said is quite obviously in-grammar. But other times it’s more complex and the Wizard must make a snap judgment call to decide what the user really meant and whether it could be handled.

It’s better to be on the conservative side as a Wizard, but don’t worry too much—the testing will still do a good job capturing early issues.

Some researchers still frown on remote usability testing, but it can be extremely effective if you do it correctly. Advantages to remote testing include the following:

Video recording

Because you obviously will not be there in person during a remote usability test, you need to come up with a way to record what’s going on with the user. When testing mobile apps with no audio component, using a screen recording app on a phone can do the trick, but it’s more difficult when the app is speaking and the user is speaking to the app.

Some companies offer a service wherein some of their users have their own webcams and are comfortable recording testing sessions. This can become expensive, however, and limit your demographics to people who own and are able to use a webcam or other recording device.

When I was at Volio, we were already recording users’ audio and video with our picture-in-picture feature on the iPad (users could see themselves in the corner, as they do in FaceTime), as shown in Figure 6-2. For remote testing, we simply left the recording on even when it was not the user’s turn to speak, to capture any facial expressions and other audio. (We of course obtained their permission to do so, and kept the data internal.)

Figure 6-2. When using the Volio app, users’ video can be recorded, which is useful for user testing

This methodology proved to be remarkably effective. In addition to capturing what users were saying to the app during their turn, we were also observing their reactions to the content being presented to them. For example, when testing our stand-up comedian interactive app, while the comedian was telling a joke, we could observe the users’ expressions—were they laughing? Bored? Offended?

After having users do several tasks, they completed a survey, which provided us with information about their subjective perceptions of the system. One downside to this type of remote testing is that you cannot follow-up on participants’ responses in real time (unless of course you’re testing an IVR system), but the benefits can often outweigh the disadvantages.

Services for remote testing

When conducting usability tests, sometimes you need to get creative, especially if you’re at a small company with no real budget for user testing.

Another method for conducting remote user testing is to find a group of testers by using services such as Amazon’s Mechanical Turk. “Workers” sign up for Mechanical Turk, and “Requesters” create tasks that can be done online via a web page. Workers can choose which tasks they want to do, and requesters pay them. Figure 6-3 presents an example of what a Mechanical Turk worker would see when performing a task.^[42]

Figure 6-3. Sample task for Amazon Mechanical Turk

Ann Thyme-Gobbel, currently VUI design lead at D+M Holdings, has successfully used Mechanical Turk to test comparisons of text-to-speech (TTS) versus recorded prompts. She asked Turkers to listen to various prompts that were happy, discouraging, disappointing, and other emotions, and then asked follow-up questions about trust and likability.

She feels remote testing has a variety of advantages over lab testing, and not just the fact that people are not required to come in to a lab. Remote testing can make it easier to simulate scenarios “in the wild.” An example she gives is asking a user to refill a prescription over the phone. You can ask the remote user to find a real pill bottle in their house and read off the prescription number. Having them walk around the house while on the phone and looking for it will provide a much more realistic example of what it’s like to do this task than to bring them into a lab and hand them a bottle.

When conducting user studies for their voice-enabled cooking app “Yes, Chef,” Conversant Labs made sure to test in people’s kitchens while they made real recipes, to ensure that the app would understand the way people really speak while cooking.

In addition, Thyme-Gobbel says that during remote testing:

Participants share more. If you are sitting in the room with the participant, they’re more likely to say, “You saw what happened,” than to describe what they thought happened in some situation.

Remember to pay your subjects well, even with outsourcing. Mechanical Turk can be a much more affordable method than recruiting people yourself, but that doesn’t mean you should not pay fairly. These methods also mean that your budget can be stretched further, so you can recruit more subjects. The rule of thumb of 5 to 10 usability subjects applies when the subjects are all in the appropriate demographic; if you have less control, you’ll need more subjects.

Traditional usability testing is often done in a lab, with one-way mirrors and recording equipment. Recruiting subjects to do in-person testing can be very effective. Having recording equipment already set up is a great way to reliably record what’s needed. The pros of lab testing include having a dedicated space to perform user testing (particularly important when you need a quiet place to do VUI testing), and the ability to have permanent recording equipment set up, allowing the user to do a task without the tester sitting right next to them and possibly making them uncomfortable (assuming a one-way mirror).

Some cons include the price, both of having and maintaining the lab as well as the cost of paying subjects to keep appointments and show up in person. It can also limit the demographic if the people you need are not available in your area.

There are of course variations in “lab” testing: you do not necessarily need a dedicated space, a particularly difficult thing to come by in a small company. A private quiet room can do just fine. Ideally, you will have a camera showing you what the user is seeing, so you do not need to sit too closely to observe.

When you don’t have a budget for user testing, sometimes you need to simply go out into the world and ask people to try your app. Coffee shops, shopping malls, parks—any of these can be effective places to recruit subjects for on-the-fly testing. Be well armed with your mobile device, task definition, questions, and a reward. Homemade cookies, Starbucks gift cards, even stickers can be enough motivation for people to spend 5 to 15 minutes trying out your app. Remind them they’re helping design a better app and that their feedback will help others.

Testing in the car is challenging. Large car companies and some universities have driving simulators, but for smaller companies it can be difficult to fund such equipment. Lower-cost options can include a mock-up of a car, with a monitor displaying a driving simulator, along with a steering wheel and holder for a phone or tablet for the driving app itself.

Lisa Falkson, lead voice UX designer at electric car company NEXTEV, often runs car usability testing in a real car, but while the car is stopped in a parking lot. This allows her to gain valuable information about how the user interacts with the system, their level of distraction, and what they can and cannot accomplish—in a much safer environment than if the user were actually driving.

Karen Kaushansky, director of experience at self-driving car company Zoox, reminds us that you don’t need to have a fully working prototype to do initial experience testing. While at Microsoft, working on the Ford SYNC system (which allowed users to push a button and ask for their commute time, among other features), she used a low-fidelity method to do WOz testing.

The car did not yet have its push-to-talk button, so they attached a foam one to the steering wheel (Figure 6-4). While sitting in the parking lot, the Wizard sat in the backseat with a laptop, playing prompts that would, in the future, come from the car! Subjects were quickly caught up in the experience, allowing the team to do useful initial testing to discover things such as how people would ask for information about traffic.

Figure 6-4. Ford SYNC prototype with foam “push-to-talk” button (with permission from K. Kaushansky)

Kaushansky talks about ways to test other devices, as well, not just cars, in the early stages, which she calls “experience prototyping.” Suppose that you’re building a smartwatch, but it will be a while before the first model exists. Even before that happens, you can test what it would be like to have such a device—strap one on your wrist and walk around the mall, talking to it. As Kaushansky says, “be Dick Tracy and put your mouth near your watch and say, ‘What’s the weather tomorrow?’”

As Tom Chi said in his TED talk in 2012 about rapid prototyping Google Glass, “Doing is the best kind of thinking.” In other words, you can think long and hard about how people might interact with your device, but nothing beats having people actually interact with it (even in a crude prototype form).

If you’re testing a device such as an Amazon Echo–type system, you can rig something up to do WOz testing, because all you need to do is play back prompts through the device. Imagine a scenario in which the user is in a room with the device, and the Wizard sits in the other room at the computer, ready to play prompts in response to the user’s queries.

You can even use WOz methodology for testing robots. Ellen Francik, lead UX designer at Mayfield Robotics, needed to test how people reacted to the robot’s responses to user input. Before the robot even had a microphone, she was able to test this scenario with a WOz setup.

The Wizard would use an iPad app, which was connected to the robot via Bluetooth, to control the robot’s movements, sounds, and animations. When the user said a command to the robot or asked a question, the Wizard would play the appropriate sound and move the robot. In this way, Francik could test whether users understood the robot’s (nonverbal) responses. Did the user know when the robot “said” yes? Could the user tell when it indicated a “got it!” response?

Because they were testing in the environment in which the robot would actually be used—in people’s homes—they did not try to hide the Wizard. Francik said this was not an impediment; people were still game to interact with the robot, and the sessions proved highly valuable.