In the early 2000s, IVR systems were becoming more common. Initially primitive touch-tone/voice hybrids (“Please press or say 1”), they became an expected way to communicate with many companies. IVRs could help callers get stock quotes, book flights, transfer money, and provide traffic information. Many of them were designed poorly, and websites popped up with backdoors on how to get transferred immediately to an operator (something many companies actively tried to hide). IVRs acquired a bad reputation, ending up the subject of satire on Saturday Night Live.

IVR systems were created to automate tasks so that customers would not always need to speak to a live person to get things done. They were created before the Internet became commonly used and before smartphones were invented.

Today, many IVR systems are used as the “first response” part of a phone call, so that even if the caller ends up speaking with an agent, basic information has been collected (such as a credit card number). For many tasks, even complex ones such as booking a flight, an IVR system can do the job. In addition, IVR systems are great at routing customers to a variety of different agent pools so that one phone number can serve many needs. Finally, some users actually prefer using an IVR system versus speaking with an agent because they can take their time and ask for information over and over (such as the 1990s-era Charles Schwab stock quote system) without feeling like they’re “bothering” a human agent.

Although some of the design strategies from the IVR world also apply to mobile VUI design (as well as VUI system for devices), mobile VUIs also present a unique set of challenges (and opportunities). This chapter outlines design principles for the more varied and complex world of designing modern VUI systems.

One of the challenges of mobile VUIs is determining whether it will have a visual representation such as an avatar. Another challenge is establishing when your VUI will allow the user to speak. Will users be able to interrupt? Will it use push-to-talk? These challenges are discussed later in the book.

However, unlike IVR systems, with mobile devices there is an opportunity to have a visual component. This can be a big advantage in many ways, from communicating information to the user, to confirming it, even to helping the user know when it’s their turn to speak. Allowing users to interact both via voice and by using a screen is an example of a multimodal interface. Many of the examples in this book are for multimodal designs. In some cases, the modes live together in one place, such as a virtual assistant on a mobile phone. In others, the main interaction is voice-only, but there is also a companion app available on the user’s smartphone.

For example, suppose that you ask Google, “Who are the 10 richest people in the world?” Google could certainly read off a list of people (and their current worth), but that is a heavy cognitive load. It’s much better to display them, as shown in Figure 2-1.

Figure 2-1. Google visually showing results for voice request, “Who are the 10 richest people in the world?”

Taking advantage of the visual capabilities of mobile is essential to creating a rich VUI experience. In addition, this visual component can allow the user to continue at a more leisurely pace. In an IVR system, it is rare to be able to pause the system—instead, the user must continually interact.

If your VUI will have a visual component, such as a mobile app, video game, or smartwatch, it’s important to design the visual and the voice in tandem. If the visual designer and the VUI designer don’t work together until the end, the joining of the two mediums can be awkward and haphazard. VUI and visual are two components of the same conversation that the user is having with the system. It’s essential to design together from the beginning.

Another current common difference between IVR systems and VUIs on mobile apps or devices is that they are often used for one-turn tasks. For example, I’ll ask Cortana to set an alarm (Figure 2-2) or Google what the fastest land animal is, or instruct Amazon Echo’s Alexa to start playing my favorite radio station. These types of interactions are quite contained and do not require the system to maintain a lot of information.

Figure 2-2. Cortana confirming a voice-requested alarm

Although this is quite common now, do not confine your VUI experience to this model. To begin thinking more specifically about how to best design VUI for mobile, let’s dive first into the topic of conversational design.

Imagine you’re having a conversation with a friend. You’re sitting in a coffee shop, catching up after a long absence. Your friend says, “Did you see the new Star Wars movie?” “Yes,” you reply. “Did you like it?” she asks next. You say, “I’m sorry, I don’t understand.” No matter how many times she repeats herself, you never answer her question.

That level of frustration is about where we’re at with many VUI systems today. Despite the many recent advancements of speech recognition technology, we’re still a long way from simulating human conversation. Here’s a real-world example from Ok Google, illustrating two conversational turns (a turn is one interaction between the user and the system):

Figure 2-3. Google not being conversational

Google has let down its end of the conversation. It’s like the first part never happened. Conversational design is becoming a common term, but it is often misused. Many people use it to mean any time you have an interaction with a system in which you speak, or text. But many of these “conversations” have only one turn; for example, asking Hound where the nearest coffee shop is located.

In this book, I define conversational design to mean thinking about an interaction with a VUI system beyond one turn. Humans rarely have conversations that only last one turn. Design beyond that one turn; imagine what users might want to do next. Don’t force them to take another turn, but anticipate and allow it. In addition, it is vital to keep a recent history of what the user has just told you. Having a conversation with a system that can’t remember anything beyond the last interaction makes for a dumb and not very useful experience.

When designing a VUI, many people only consider one-off tasks, such as answering a search query, setting up a calendar appointment, placing a phone call, playing a song, and so on. Sometimes these tasks can be accomplished in one fell swoop. But the best VUI designs also consider what happens next.

Here’s an example in which Google does a good job of remembering what occurred in previous conversational turns:

Figure 2-4. Google showing the best restaurants in the town in which Abraham Lincoln was born

It’s not quite the same as talking to a human, but Google successfully carried on the conversation for four turns, knowing the references for “he” and “there.” In addition, Google switched to a more visual mode at the appropriate time: to show the map and reviews for the restaurant.

A good rule of thumb is to let the user decide how long the conversation will be.

Good conversational design is not just about crafting nice prompts. As Margaret Urban, Interaction Designer at Google, suggests: don’t ask a question if you won’t be able to understand the answer. She gives the example of a prompt that occurs after the user has finished writing an email: “Do you want to send it or change it?” One response, which you might not initially have planned for, will be “yes”—so build a response in your system to handle it. Although coaching the voice talent to use the appropriate stress and intonation can help with this issue, it is often not enough. For a case in which you’re seeing a lot of “yes” responses, you might want to consider rewording the prompt to something more clear, such as “What would you like to do: send it or change it?”

Urban emphasizes it’s important to set user expectations early on. How does your app introduce voice? You can offer a “tour” to first-time users, and provide educational points along the way. As Urban says:

Be careful about telling users that tasks were successful. Urban says, “‘Setting the alarm,’ for example, implies to the user that the alarm has been set, whereas the engineer may argue that the task hasn’t necessarily been completed yet and should have an additional prompt that says ‘Alarm set successfully’.”

The Amazon Echo has the following dialog when setting a timer:

Imagine the conversation with an additional confirmation:

It’s unnecessary verbiage. If in fact the time did at some point fail to be set, it would be good to alert the user—but that’s the exception.

Urban offers a good analogy about designing with breadth. Perhaps you’ve designed a system that allows people to set an alarm, but you didn’t give them a way to cancel it. She likens this to giving someone a towel for a shower, but no soap. If you set an expectation that you can accomplish a task, think about the corresponding (symmetrical) task that goes with it.

Discoverability is another important element of design. How does your user know when they can speak and what they can say? I discovered that my Android camera app was voice-enabled purely by accident—while taking a picture one day, I naturally said “smile!” and the photo clicked. I quickly discovered I could also say “1...2...3!” and “say cheese!” and it would also take a photo. This is a great example of piggybacking off of a user’s natural speech.

Another example of a command I discovered by accident occurred after I had to reboot my Amazon Echo. When it came back to life, without thinking I said “Alexa, are you working?” and she replied that everything was in working order. I never stopped and thought, “What can I ask Alexa to see if everything’s working again?” but my spontaneous request was handled. That’s a much better way to check Internet connectivity rather than going to Network Settings on a computer!

When asking the user for information, it’s often better to give examples than instructions. If you’re asking for date of birth, for example, rather than say “Please tell me your date of birth, with the month, day, and year,” use, for example, “Please tell me your date of birth, such as July 22, 1972.” It’s much easier for users to copy an example with their own information than translate the more generic instruction.^[8]

To assist you in creating great conversational designs, let’s talk about tools.

Tools to create VUIs are becoming more common, but some of them are not specific to one piece of software; they are instead a methodology.

Flow

After you have written and reviewed a variety of sample dialogs, the next step is to sketch the VUI’s flow. Flows (referred to as callflows in the IVR world) are diagrams that illustrate all the paths that can be taken through your VUI system. The level of detail for this flow depends on the type of system you are designing. For an IVR system, or a closed conversation, the flow should include all possible branches the user can go down (Figure 2-5). This means that for each turn in the conversation, the flow will list out all the different ways the user can branch to the next state. This could be for simple states, that allow only “yes” and “no” type responses as well as more complex ones that might have 1,000 possible song titles. The diagram does not need to list every phrase someone can say, but it should group them appropriately.

In the case of something more open-ended, such as a virtual assistant, the flow can be grouped into types of interactions (e.g., calendar functions, search, calling/texting, etc.). In these cases, not all possible interactions can be spelled out, but it helps to group the various intents, as illustrated in Figure 2-6.

You can use any flow tools for this—yEd, Omnigraffle, Google Draw, and Visio are all good options. In addition, storybuilding tools such as Twine can be helpful in this phase.

Figure 2-5. Sample flow with complete branching

After you’ve designed the basic flow and have a set of sample dialogs completed, you can focus on some of the other important details, such as confirming input.

Making sure that users feel understood is an important part of any good VUI design. This also serves the purpose of letting a user know when they were not understood.

Figure 2-6. Flow for system with more complex branching

In the early IVR days, confirmations were sometimes used to an excessive degree. Here’s an example:

A funny example of why over-confirmation can sound so unnatural was highlighted by a Saturday Night Live skit from 2006, called “Julie, the Operator Lady.” It was a parody between Rachel Dratch (playing the part of Julie, the Amtrak IVR persona) and Antonio Banderas, talking at a party. Ten years later, Amtrak now has an award-winning IVR system—but you can see how the same style of dialog doesn’t quite work in the real world:

Over-confirming information might ensure accuracy, but it will also drive people (like Antonio Banderas) crazy.

When determining the right confirmation strategy for a VUI experience, consider the following:

If someone is transferring money from one place to another, confirmation is extremely important. On the other hand, if the app is purely for entertainment, getting one thing wrong might not be a big deal, and in fact, might break the user out of the experience of the conversation.

Always take advantage of the channels you have available. When you want to talk to the Amazon Echo (by saying “Alexa”), the top rim of the device lights up in blue, letting you know that she’s now listening to a command. Note that the Echo is in fact always listening, but it does not light up until it knows that you want to initiate a conversation with it. I talk more about how devices show you they’re listening in Chapter 5.

When you ask the Amazon Echo a question, its primary channel for output is audio. It does have the aforementioned illuminating ring at the top, but this is generally just to indicate it’s listening (or processing), not to give you actual information. In addition, you can access information you give to the Echo on another device—such as your phone—after the fact (e.g., if you asked Alexa to add an item to your shopping list).

For VUI experiences for which audio feedback is the primary method to provide feedback, it is important to craft confirmations carefully. One way to do this is by using confidence thresholds.

A confidence threshold is how the speech recognition engine lets you know how well it thinks it’s performed. For example, it might recognize that you said “YES, PLEASE”—but the confidence could vary. Perhaps it’s only 45 percent confident that this is what it heard, or perhaps it’s 85 percent. Your app can access this information, and handle them in different ways:

There are many different ways to confirm information. The next section describes them in more detail.

Method 5: Visual Confirmation

With mobile devices, visual confirmation is a commonly used method. For example, when asking Google a question, it often provides an audio confirmation as well as a visual one (Figure 2-7):

• USER

  Ok Google. When’s my next meeting?

• GOOGLE

  You have a calendar entry tomorrow. The title is “Chinatown field trip.”

Figure 2-7. Google’s visual confirmation of a meeting

Take advantage of the screen! Communicating a list of items is much more effective on a screen. Humans have a limited capacity to remember a string of information; typically, a user can’t remember more than about seven auditory items at a time.^[9] “The ephemeral nature of output in auditory interfaces places significant cognitive demands on the user.”^[10]

However, by putting those same items into a visual list on a screen, the cognitive load is lessened. Now the user can examine the list at their leisure, without remembering each little detail. This also gives the user more time to make a decision.

Another great way to use the screen is for confirming the user’s choice. Imagine a system that allows the user to select a response by either speaking or pressing a button. The system asks, “Did you take your antibiotics last night?” and provides a Yes and a No button. The user could say, “Yes,” and rather than have the system say, “I think you said ‘yes,’” simply highlight the button, as if the user had pressed it. The user will know whether she has been correctly (or incorrectly) understood (Figure 2-8).

In addition to thinking about confirmations, an important design decision is when to allow your user to speak.

Most of the VUI systems out there today are command and control, meaning the user must explicitly indicate when they want to speak. Another type of design, which is becoming more common as systems become more conversational, uses a more natural turn-taking approach.

To know which type is right for your VUI design, answer the following questions:

Command-and-Control

Many systems use this approach whereby users must do something explicit to indicate to the system that they are going to speak. For example, Siri requires the user to press the home button before speaking (or, from the Siri screen, press the microphone soft button (Figure 2-9). Ok Google requires either pressing the microphone icon, or saying “Ok Google” (Figure 2-10). The Amazon Echo has a physical button, but users can also indicate that they are about to speak by saying the wake word, “Alexa.” Even the starship Enterprise required the crew to say “Computer” before making a request.

Figure 2-8. Users can speak or press buttons to reply to Sensely’s avatar

Figure 2-9. After holding down the home button to activate Siri and then failing, the microphone button appears

Figure 2-10. To speak to Google, say “Ok Google” or tap microphone button

In many cars, the user must use a “push-to-talk” approach; that is, the driver must press a specific physical button on the steering wheel to indicate that they will speak (Figure 2-11).

Figure 2-11. The push-to-talk button in the Toyota Matrix (photo by author)

When this happens, the system typically responds with either nonverbal audio (the “bonk” noise) and/or with visual feedback (a wavy line, animated dots, a glowing light on the device). The user then knows that the system is listening and can speak (Figure 2-12).

Figure 2-12. Listening indicators: (from upper left to bottom) Hound, Cortana, Google, and Siri

When the system has decided that the user has finished speaking, it indicates in some way, often with more nonverbal audio (the “up” noise from Siri), and then responds. Identifying when the user has stopped speaking is called endpoint detection; I discuss this in more detail in Chapter 4.

At this point, the conversation might well be over. The system is not expecting the user to speak again, although sometimes it can handle a related follow-up request. But the user must again explicitly indicate that they are going to speak.

This works well for situations in which the device has no idea when the user might speak. Imagine you’re in the family room and your spouse is in the kitchen. You haven’t spoken for a while. You will probably use some kind of explicit notification to your spouse to let them know you’re going to speak, such as, “Hey, honey, can I ask you a question?” or, “Hey, Chris.” This prepares your spouse to listen. If you went for 30 minutes without speaking and suddenly you said “Where is it?” without any context of what you were looking for, your spouse will likely be confused.

The time window in which your system should still be listening after the wake word or button push should be carefully chosen. Too short, and you’ll miss users who hesitate briefly before speaking; too long and the system might be listening to a conversation that’s not intended for it. As a rule of thumb, 10 seconds is a good starting point.

Another way to make your VUI more human-like (as well as more engaging) is to use conversational markers. See if you can spot the difference between these conversations:

Versus:

Both of these examples collected the same three pieces of information. But which virtual assistant do you think the user will prefer speaking with, day after day?

Conversational markers are an important way to let the user know where they’re at in the conversation, and that they are understood. Users will also be more engaged when the system is using basic manners and will respond in kind. They are often the “glue” that keeps the various required pieces of the interaction together.

Conversational markers include:

One practical way to see where conversational markers should be added to your dialog is to do a “table read” with someone. Write down a sample dialog (the back and forth between the system and the user) and each of you read one part. You’ll quickly see where things might be stilted or unnatural, or add to a user’s frustration because they have no idea how long the conversation is going to last.

Here are a couple of common concerns I hear from clients: “Computers don’t speak that way,” and “People will be put off because it’s a computer, not a person.” Some clients worry that the system will sound too informal or annoy people by pretending to be human.

It is important to use conversational markers that are appropriate to your system’s persona, but even the most formal systems will benefit. Users know they’re speaking to a machine, but humans appreciate these conversational basics nonetheless.

Next, we’ll discuss a crucial piece of design that addresses something that you hope never happens but is essential nonetheless.

In the traditional IVR world, if the user was not heard or understood, the system prompts the user to speak again. This is important because otherwise the user might think the phone call was cut off or the system isn’t functioning. Another reason is that there is a fixed dialog that is expected; at any given point, the user is required to provide input to move the conversation forward. If the system doesn’t hear anything, it times out and prompts the user to speak again. If these timeouts are not planned carefully, the user and the system often end up cutting each other off, resulting in an awkward back and forth.

In the mobile VUI and device world, however, it’s not always necessary to reprompt the user when there is a failure. The Amazon Echo, for example, does nothing if it doesn’t hear you after saying the wake word. (If it heard but did not understand, it plays a short sound.)

When speaking to a device (especially a device with a name), users are more likely to respond to silence the same way they would with a human—by repeating themselves. In addition, the system is not waiting for the next piece of the conversation, because this is often a one-off command. Therefore, if the system fails to respond, it’s not that the entire transaction fails—just that one particular command. It’s not the same as suddenly ending a conversation in the middle of an important transaction; it has just failed once. Because of the higher fault tolerance, it’s more forgiving for Alexa to ignore you when she did not understand.

Imagine, however, that your app said, “I’m sorry, I didn’t understand,” every time it failed to recognize your latest command. It would get old very quickly. “I didn’t understand, I didn’t understand, I didn’t understand.” Alright, already! Users can quickly grow accustomed to a mode in which they simply need to repeat themselves if the device doesn’t understand on the first try.

We’ve talked a lot about “best path” behavior for a VUI design. But, as any good designer knows, you can’t just design for when things work—you need to design for when things go wrong, as well. This is especially true with VUIs, because something will always go wrong.

Although speech recognition has improved drastically in the past 10 years (greater than 90 percent accuracy given the right conditions), this will in no way ensure that your users will have a good experience when you add voice to your design. Think for a moment about human-to-human conversation. We often miss a word (or multiple words) when someone else is talking.

All of these factors play a role in VUI, as well. But humans are much better than computers about getting back on track, thanks to a rich understanding of context, and being able to recover from conversational errors. If I say something to you, and you stare at me quizzically, I’m likely to repeat myself, knowing you did not understand. I can ask you to repeat yourself. I can ask you to clarify. We have many ways to course-correct, because we’re both steeped in years of practice of human conversation.

When the VUI does not understand us, things often break down. How you decide to handle these error conditions in your VUI is extremely important. As Pilar Manchon, GM of voice and digital assistants at Intel said, “Every time that you score down because you make a mistake, or you don’t know something, it actually counts a hundred times for every time that you had it right.”^[12]

If you do a good job, error conditions won’t derail users, and you can get them back on track and have them successfully complete a task. Handle it poorly, and not only will they fail to complete their tasks, they’re not likely to come back.

There are a variety of ways VUIs can make mistakes:

When at all possible, do not blame the user. Blame it on something else, or at the very least blame the system.

In a study performed by Clifford Nass and Scott Brave, users performed tasks with a driving simulator, and throughout the tasks, the system voice made comments about their driving performance. Half of the participants heard messages blaming the driver (“You’re driving too fast.”) and half heard messages blaming external factors, such as “Steering is difficult on this road.”^[13]

The participants who were blamed by the system rated their driving performance lower, liked the voice less, and most importantly, showed less attention while driving. Error messages might seem like a small thing, but they can affect the user’s perception of the system and even their performance.

If your users will be using your system on a regular basis, it’s important to include different strategies in the design.

For example, a healthcare app might require users to log in every day and take their blood pressure. In the beginning, having the prompts include more instructional details is useful and necessary. However, after the user has become familiar with the app, there is no need to continue lengthy instructions (as well as other prompts). Let’s take a look at an example of each scenario.

Novice user:

User has interacted with the app every day for a week:

Don’t just count the number of times the app has been used, however. It might be that an individual has used it many times but only once every month or two. In this case, you would continue to use the novice prompts.

You can also shorten explanatory prompts. For example, the system might say, “I’ll be asking you a series of questions about how you’re feeling. It will take a few minutes, but the questions will help the doctor determine the best course of action. First, did you take your medication yesterday?”

After doing this every other day for a week, you can shorten it to, “Did you take your medication yesterday?” (Be sure to use “conversational markers” even though the prompts are shorter, to keep the user informed that their answers have been understood, and where they are in the set of questions.)

Margaret Urban, interaction designer at Google, says it’s important to ensure that your goal is not simply to “train” your user. She says rather than beat them over the head with the available commands, adapt to their behavior.

In addition, take advantage of the concept of priming. Priming refers to the fact that exposing someone to a particular stimulus (such as a word or image) will influence their response to a later stimulus. For example, if you’ve been watching a nature program about llamas in Peru, and afterward someone asks you to name an animal that begins with the letter “L,” you’re much more likely to say llama than lion.

Letting someone know up front that you’ll be asking them a certain number of questions is also a form of priming; it gives the person an indication of what to expect, so they know how to prepare themselves.

Priming can be subtler. If your VUI confirms commands in a certain way, it primes the user to phrase it similarly in the future. For example, if I say, “I would like to hear the song by the group the Barenaked Ladies that’s called ‘Call and Answer,’” and my VUI responds with, “Playing ‘Call and Answer’ by the Barenaked Ladies,” next time I might just say, “play ‘Call and Answer’ by the Barenaked Ladies,” instead.

Remember the earlier example in which Google was able to continue the conversation about Abraham Lincoln? Let’s take a look at it again:

What’s good about this example? Google continued the conversation, and remembered the context. Specifically, the use of pronouns. It knew “he” referred to Abraham Lincoln. It also knew that “there” meant Hodgenville, Kentucky. Keeping track of this information is not always easy, but without it, your app won’t be capable of anything but one-turn actions. The use of two different terms to refer to the same thing is called coreference, and is an essential part of communication. Without it, conversations will quickly break down.

Here’s another example, though a fictitious one, about an app that can help find movies to watch on TV:

In this case, the app needs to know what “ones” means as well as “him.”

It can be difficult to anticipate all the things your system might need to track, but you can begin with the basics and build up based on what your users are actually saying (see more in Chapter 5).

If the user asks a question about a person, store that information. If the person is famous, you can even look up the gender. However, you can also use a simpler method, which is to always store the last person mentioned, and refer to that person whether the user says ‘he’ or ‘she.’ (Of course, this will not work if the user has mentioned more than one person, but in many instances it will work just fine.)” If you can’t determine the gender, you can also just use the person’s name, although it can sound robotic to keep using the full name rather than a pronoun. “They,” the gender-neutral form, is also gaining acceptance.

You can apply similar processes to remembering the last city or place the user (or the system) mentioned.

Systems often struggle with interpreting the user’s references. Here’s a Cortana example that isn’t able to handle the second query in the conversation:

As you can see, Cortana did not assign a meaning to “one” and treated the query as brand new. To contrast this, look at what Google does:

Here, Google successfully understood the word “one” in the second query, and gave some relevant search results.

When I worked on IVR systems, we made sure to include a set of universals at every state: repeat, main menu, help, operator, and goodbye.

For mobile apps, this is not a hard-and-fast rule. Many mobile apps (or connected devices) do not have the concept of a main menu. However, it is still important to ensure that users can get help when needed.

Supporting the “help” command is useful in many cases, but is traditionally used for context-specific help. For example, imagine an insurance IVR system in which a user has been asked to enter their medical record number (MRN). If the user says, “help,” at this point, it would be useful to remind the user that the MRN is on their insurance card.

But what if the user asks for help in an open-ended situation such as a general assistant, like Google, Cortana, or Siri? If the user has just initiated the conversation (by pressing the home button or saying, “Ok Google,” and then says “help”), you have no context around which to know what type of help the user needs.

In addition, users are not always accustomed to using these types of universal phrases, despite us IVR designers trying to train them for many years! Therefore, it’s important to think not only about what type of help to provide, but how users might say it. For example, “Alexa, what can you do?” or asking Cortana, “What can you do for me?” Or with Google, asking, “Ok Google, what can I say to you?” In fact, Ok Google will display examples if you simply tap the mic and say nothing.

Alexa doesn’t even try to handle that question on her own; instead, she says, “For help with what you can say, take a look at the ‘things to try section’ in the Alexa app.”

Cortana says “Here are some things I can help you do” and provides a visual set of examples, depicted in Figure 2-13.

Figure 2-13. Cortana lists visual examples of possible voice commands

Google says nothing, but provides a screen of examples, as shown in Figure 2-14.

Figure 2-14. Google provides a set of functions and how to request them by voice

In addition to allowing users to ask for help, take advantage of visual space if you have it, such as on a mobile app. Imagine designing an adventure game in which the player can talk to various characters and explore different worlds. Having a “help” or “info” button always available in the GUI commands is an obvious way to let the user know assistance is available.

In the current world of VUIs, “help” (and other ways people can ask “What can I do here?”) is especially important because there is no all-powerful VUI out there. Despite many VUIs saying something very open, such as “How may I help you?” they are still very limited in what they can understand and act on. Users need a way to find out just what it is they can really do.

Another important piece of good IVR design was to include a “goodbye” universal. In the early VUI designs, goodbye was not considered important because people could simply hang up to end the conversation. But years of collecting data made designers realize that, because people were used to saying goodbye to people when ending a phone conversation, they often did it with IVR systems, as well, and sometimes felt uncomfortable ending conversations by simply hanging up. Therefore, we added dialog to handle this case. Another important lesson was to make sure the system was very confident before actually hanging up! If your system will be using a “goodbye” option, be sure to use three-tiered confidence and confirm that the user really wants to exit the system if confidence is not high.

Here’s an example from the 511 IVR system, when confidence was medium: “I thought I heard you say goodbye. Do you really want to exit the system?”

The other night on my way to bed, I passed by my Amazon Echo and said, “Alexa, good night.” She responded with “Good night.” It might sound like a silly thing in which to invest development time, but it was very satisfying.

Allowing users to repeat and to go back a step are important for conversational systems as well. “Go back” is not necessary for things like Google Now, in which interactions are short, but for a task-oriented conversation, go back is very useful. During tasks that are not within a set of conversational turns, “go back” can take on different meanings. For example, if your user is listening to music, it might indicate they want to hear a previous song. These are commands that are often good candidates for GUI controls, as well.

Another component that is sometimes missed by designers is latency. It is important to determine as early as possible whether or not your system will have any latency, or delays. Latencies are generally caused by:

Perhaps your VUI system needs to access patient records, and you know that this will require a database lookup. Find out as early as possible how long this is likely to be, and plan accordingly.

If there is a known latency, ensure that the system has a way to handle this. You can do this both by the system telling the user about the delay (“One moment while I look up your record...”) as well as nonverbal and visual cues, such as a latency noise (often used on IVR systems) and visual (animated loading icon), as demonstrated in Figure 2-15.

Figure 2-15. Sensely avatar says “One moment please,” displays the message, and shows the loading GIF

There might be cases in which the expected latency time could range from none to 10 seconds. If so, in the case of no latency, pad the latency by a few seconds, because it sounds broken to the user if you say “One moment please” and immediately continue the conversation. If you set up the expectation of a delay, do not violate that expectation.

Sometimes latency can occur when processing speech, as well. Many devices use local recognition for their wake word, which is fast, and then send the rest of the audio for processing on the cloud.

There can be times when the user provides some but not all of the details to take an action. For example, the user might ask for the weather for a location that exists in more than one place: “What’s the weather in Springfield?”

If possible, rely on any known information to determine the answer without having to ask the user. For example, the Amazon Echo requires the user to specify the home location as a part of the set up; thus, when you ask, “What’s the weather?” Alexa produces local conditions automatically. Knowing the home location can be used when the user asks for the weather somewhere else, as well—for instance, choose the “Springfield” close to home rather than across the country.

Other contextual clues can also be used. If the user just looked up a restaurant in Springfield, Illinois, and then asks, “What’s the weather in Springfield?” you can pretty safely bet that they mean the one in the location they just referenced.

If no contextual information is available, the system will need to ask the user to clarify:

If the system has high confidence for the word “Springfield,” it can use the reference word “one” rather than explicitly stating the name again. Also, be sure to allow the user flexibility in their response: the user should be able to say “Springfield, Illinois” or just “Illinois” or even “the first one” (imagine a list).

Another example where disambiguation is required is when an action is not clear:

Note the confirmation of the name came at the end, as an implicit conversation. This would be done if the system had (a) high confidence about the name and (b) there was only one Cindy in the caller’s Contacts list.

I noticed Google recently improved its dialing design. It used to be that when I said, “Text Chris Leggetter,” Google responded with, “Home or mobile?” and I had to choose. Now it’s smart enough to know I meant mobile, because I can’t text a home phone number.

Disambiguation might also be required when the user answers with more information than your VUI can handle.

It would be ideal if your system could handle both symptoms at the same time, but given that these systems do have underlying constraints, sometimes it is necessary to ask the user to narrow it down.

In addition to the previously mentioned sample dialogs and dialog flow document, there are other tangible things that you might need to think about.

To discuss accessibility, I’ve brought in an expert: Chris Maury. Maury is the founder of Conversant Labs, a company focused on improving the lives of the blind through providing improved access to technology. In 2011, Maury was told that he was going blind. Maury turned to technology to begin preparing for this future, but was dismayed at what was (or was not) available. Maury quickly realized that standard accessibility technology such as screen readers were not always pleasant to use. As Maury wrote:

He set out to design audio experiences from scratch. In the sections that follow, he offers some tips on how to do just that.

Although designing experiences for everyone, no matter their abilities, should be a core requirement in every project, it becomes especially meaningful as we begin to explore interactions that go beyond a touchscreen and keyboard. What is a VUI but the ideal, nonvisual experience for the blind and visually impaired? The constraints of designing for people with different disabilities can help to inform how we solve similar challenges in not just VUI design but other emerging arenas, as well: conversational applications (chatbots) and immersive computing (virtual and augmented reality).

Here are a few best practices for VUI design informed by accessibility:

Keep It Short

In a visual experience, users can quickly focus their attention to the information that is most relevant to them, skipping over sections of the interface that they don’t care about. Audio interfaces, on the other hand, are linear. There is no skipping around. You are forced to listen to everything the app decides to tell you. This being the case, keep it short. Limit text to the most important information, and within that, present the most important information first.

In the example of a shopping app, imagine the user is listening to search results for a given product. Present the title of the product, the price, and the rating. Everything else can be deferred to a single product details dialog, as illustrated in Figure 2-16:

“Harry Potter and the Cursed Child by J.K. Rowling” — $17.99 — 3.5 out of 5 stars

Figure 2-16. Amazon book sample page for Harry Potter

Compare this to the visual experience, which can cram in far more details into a single search result, even on mobile.

For information that doesn’t make the cut, allow the user to ask for it. Continuing with the shopping example, here are some possible other queries the user might have:

• “What are the product specifications?”

• “Read me the reviews”

• “Is it hardcover or paperback?”

This chapter introduced many of the key concepts when designing VUIs. Many of them are drawn from the IVR world, with key differences. Basic strategies, such as well-designed error behavior, implicit versus explicit confirmation, and design artifacts such as sample dialogs and flows, apply in both cases.

Typical VUI project deliverables include:

If working with an external customer, these design deliverables will communicate what the finished product will look like and allow the customer to review and provide feedback. It also provides a way for everyone to agree on the design before it’s been implemented.

Some of the key design concepts covered in this chapter are:

Designing for mobile can be both a richer experience as well as more complex. You’ll need to make decisions about how to let your users know when and where they can speak, and when and where to use visual feedback. In many cases, there will be no human to back up your experience.

Enabling users to speak to their phones and devices opens up an entire world of experiences; whether it’s looking up a piece of trivia during a dinner argument, asking a device to dim the lights, or managing the everyday tasks of your life, VUI can enhance them all.