How do we hear sound?

Sound is created through the vibration of an object. This causes particles to constantly bump into one another, sending vibrations as sound waves to our ears and, more specifically, to our eardrums. When a sound wave reaches the eardrum, it too will vibrate at the same rate. Then the cochlea, inside the ear, processes the sound into a format that can be read by the brain. To do this, the sound has to travel from the ear to the brain along the auditory nerve. Sound requires an element or medium to travel through, such as air, water, or even metal. You may already understand this process, but as we look to design for sound, there are some key properties that are essential to understand, including loudness and pitch.

Because we can detect a wide range of sound, we need a way to measure the intensity of the sound. This is called loudness, which uses the unit of decibels (dB) to measure how loud or soft a sound is.

To help you add some perspective to dB measurements:

• A whisper is between 20 and 30 dB.

• Normal speech is around 50 dB.

• A vacuum cleaner is about 70 dB.

• A lawn mower is about 90 dB.

• A car horn is about 110 dB.

Sound changes depending on how fast the object is vibrating. The faster the vibration, the higher the sound. This pitch is measured using frequency, or how many times the object vibrates per second. Humans can hear a wide range of pitches, but cannot detect very low and very high sounds. Frequency measures the cycle rate of the physical waveform; pitch is how high or low it sounds when you hear it.

Frequency is measured in hertz (Hz). The human hearing ranges from 20 to 20,000 Hz. However, our hearing is most sensitive to sounds ranging in frequency between 2000 and 5000 Hz. Those who experience hearing loss will often start to lose or have the upper pitches affected first.

To re-create sound, we need to record a digital representation of the sound. This digital audio encodes the sound waves of the audio signal in sequence. You can record these at different rates. TABLE 12.1 breaks down the formats.

How do you choose which format to use? The answer depends on what you’re working with.

• Raw audio to be captured and edited: Uncompressed formats allow you to work with the highest quality file, and then you can compress the files to be smaller afterward.

• Music and voice audio: Lossless audio compression files maintain the audio quality but also the larger file sizes.

• Transferable and sharable audio: Lossy audio compression formats produce smaller files sizes, which facilitates sharing.

How is sound useful?

Sound is much like a ripple in water; it starts in a central spot, and then it slowly extends out gradually getting smaller and smaller (or quieter and quieter) as it moves away from the center. Even if you hear a sound from far away, you can still detect where the sound is coming from or at least an approximate direction. You can tell the difference between footsteps walking behind you or down another hallway. You can tell the difference between a crowded restaurant and an empty one from the lobby, all because of the sound cues of chatter. The more chatter you hear, the more people must be inside.

These sound cues help the user process information about a space both before and as they enter it. Sound adds an additional layer of information that will help the user further grow their understanding of what is going on around them.

We have discussed how light can be used to understand space and depth, and sound can also be used to calculate distance and depth. Through the use of SONAR (sound navigation and ranging), you can measure the time it takes for a sound to reflect back its echo. This idea is used by boats and submarines to navigate at sea and to learn about the depth of the ocean as well.

How do we use sound in XR?

There are many ways that sounds play a role in our understanding of space. Within XR there are three main ways sound is used.

• Ambient sound

• Feedback sound

• Spatial sound

Single-point audio capture

To help your stereo recordings sound even more natural, one option is a binaural audio recording format. To record binaurally, you record from two opposite sides and place each microphone inside a cavity to replicate the position and chamber of an ear. This concept is used to re-create sound as closely as possible to the way we hear it ourselves. Headphones are needed to accurately listen to binaural sound. Because the sound was captured in-ear, the idea is to then listen to it in the same format. People experiencing sound via VR are almost guaranteed to be wearing headphones or listening from the speakers built into the headset, so binaural sound recording is a good fit: It records the sound exactly how people will experience it.

Another prominent type of immersive sound uses ambisonics. This format is not reliant on the user wearing headphones. Ambisonic audio uses four channels (W, X, Y, and Z) of sound versus the standard two channels. An ambisonic microphone is almost like four microphones in one. You can think of this as 2D (stereo) versus 4D (ambisonic) sound. Ambisonic microphones have four pickups, each pointed and oriented in a different direction making a tetrahedral arrangement. Sound from each direction is recorded to its own channel to create a sphere of sound (FIGURE 12.2). This really does make a huge difference in the ability to capture an immersive soundscape.

The ambisonics format is not new either. It was developed by the British National Research Development Council in the 1970s, more specifically, by engineer Michael Gerzon. Even though this technology has been around for decades, it has struggled to find a commercial use, which limited its success—until now. With its success in the VR industry, this audio format is seeing rejuvenation.

The amazing part of these spatial audio techniques is you can record the sound in the exact location of the original situation that you want to re-create in your 3D experience. You can hear an orchestra how the conductor does, by recording in the exact location of where they stand. This idea of presence, or the sense of feeling like you are there, can be reinforced with the use of this kind of audio. While the visuals are an important component to creating this illusion, they are only a part of it. Because human beings engage all of our senses in any place we are, having audio that is as immersive as the visuals can provide a realistic experience. The more real it is, the more we are able to connect with an experience emotionally, mentally, and physically.

The one downside of audio capture from a single location listening in all directions around it—such as using binaural or ambisonic sound capture—is that it is reliant on a sweet spot. When the audio is captured this way, the microphones remain still. However, humans are often in a state of motion. Even if we are sitting, we still will move our heads either toward a sound or an image or simply to remain comfortable throughout the experience. When the user moves from that specific location, they move out of that sweet spot. Some adjusting can be made during the audio edit, but to have the best immersion, different headphones and HMDs have built-in head trackers to adjust the audio with the user’s motions. Positioning of the sound in relation to the user’s ears really changes the sense of direction just as it would if they were physically in the space. This will have a direct effect on the user’s perception of reverb, frequency, pitch, and intensity of the sound.

Paradise case study

Many XR experiences rely on an individual user experience, where each person will have their own set of headphones on or be inside their own virtual space. However, there is now technology that allows these experiences to become social and collaborative. This is a feature for which there is more and more of a demand. In a social situation, the sound may not track with the user. However, it could be designed to be static within a space, allowing the sound to change as the user moves through space (as in the real life).

Paradise is an interactive sound installation and gestural instrument for 16 to more than 24 loudspeakers (FIGURE 12.3). For this collaborative project, Douglas Quin and Lorne Covington joined their backgrounds in interaction design and sound design to create a fully immersive sound experience that they optimized for four to eight people. The installation allows users to “compose a collage of virtual acoustic spaces drawn from the ‘natural’ world.”¹ As users move through the space and change their arm positioning, sensors activate different soundscapes from wilderness and nature to create a musical improvisation. This composition is unique each time as it relies on how each user moves and interacts within the space. Motions can change the density of sounds, the volume of them, the motion or placement of the sound in the space, and the overall mix of the sounds together.

1 www.veniceperformanceart.org/the-art-week/fragile-body-material-body-2016/artists/douglas-quin-lorne-covington

This experience was reimagined for both interior and exterior spaces. Changing the location “creates a different spatial image,” Quin explained when I spoke with him and Covington about the challenges of the project. As they re-created the experience, they had to adjust for the location. The exterior exhibit required fewer ambient sounds, as they were provided naturally. The interior exhibit required more planning based on how sound would be reflected and reverbed by the architecture of the space (FIGURES 12.4 and 12.5).

Covington compared designing spatial interactive audio to having a good dance partner who responds to you. In the real world you don’t know what will happen next, so you have to be prepared for whatever they will do next, and respond accordingly.

Both Covington and Quin described their interactive experience as a complex reactive system. As Quin said, “The sound has to react along with the world. Instead of thinking of the audio as a soundtrack, consider the audio more as the action and reaction.” If someone does this, then do A; if someone does that, then do B; and so on.

This allows for a “shared delight in surprises.” Delight was shared by both the participants and Quin and Covington, who enjoy watching all the ways people interact with the experience. It’s the participant’s agency that creates a unique composition of sounds in each space.

How does it work?

Using what is called open-ear technology, a speaker is built into each arm of the audio glasses. What helps make them augmented, while also staying private, is the position and direction of the speakers. One speaker is placed on each arm of the glasses near the temple so that the sound is close, but still allows other sounds to enter the ear cavity (FIGURE 12.7). The speakers point backward from the face, so they are angled right toward the ears. This angle reduces how much of the sound can be heard by others around the wearer. Even in a 3D printed prototype there was not much sound escaping from the glasses, and very little could be heard even by those standing on either side.

In addition to the speakers themselves, there is also a head-motion sensor built in that can send information from the multi-axis points to your smartphone. This allows the app to know both the wearer’s location as well as what direction they are looking. This information can help customize directions—knowing the wearer’s right from left for example—as well as making sure they see key parts of an experience along the way.

The exact functionality changes among the brands of glasses, but the concept remains the same. With micro-directional speakers, the goal is to allow the wearer to hear, while minimizing that sound for everyone else around them. Many frames also have the option to automatically adjust the volume based on the environment. This is a feature that is built into car audio as well, and often is correlated with the speed of the car. The concept is to automatically adjust your volume relative to the space you are in. This further enhances the hands-free concept, minimizing the need to fiddle and adjust as you are out on an adventure. This feature could be greatly appreciated by those wearing these glasses on a bike.

More than a speaker

Listening is only half of the conversation. To allow for user feedback, these glasses also include a microphone. This allows the glasses to connect with the user’s voice assistant (more on this in the next section). Once again, this function helps maintain the hands-free functionality. It also allows the glasses to be used for phone calls and voice memos for those who want to communicate on the go. Many models have the option to turn the microphone feature on and off for privacy when needed. This is an important consideration. If you do purchase a pair, make sure that you can control when the device is listening and when it is not.

To further customize the experience, one arm of the glasses is equipped with a multi-function button that you can tap, touch, or swipe. Other than microphone control, this is the only other button you will find on the glasses. This allows you change your volume, change tracks, make a selection, and navigate within an experience—without having to access your phone directly.

How can you get your own?

Although Bose has recently announced they would stop manufacturing their audio sunglasses line, they are still currently available for purchase as of this writing. They were the first to the market, but decided to not continue manufacturing the glasses as they didn’t make as much profit as the company had hoped. When interviewed about, this a Bose spokesperson said,

Bose AR didn’t become what we envisioned. It’s not the first time our technology couldn’t be commercialized the way we planned, but components of it will be used to help Bose owners in a different way. We’re good with that. Because our research is for them, not us.²

2 Roettgers, J. (2020, June 16). Another company is giving up on AR. This time, it’s Bose. Protocol. www.protocol.com/bose-gives-up-on-augmented-reality

Just as with all technology, each advancement is a step up toward another great advancement, so new products are sure to be on the horizon that would never have been created if it weren’t for these audio glasses.

Since Bose’s first launch, others have stepped up production of their own version of Bluetooth shades. Leading the way is Amazon with their Echo Frames, which bring their well-known Alexa assistant into a pair of sunglasses. Everything many have learned to love about having a voice-powered home assistant is now available on the go. Other options to check out include audio glasses from GELETE, Lucyd, Scishion, AOHOGOD, Inventiv, and OhO.

If you are looking to use your glasses for more than just audio communication, some of shades on the market also include cameras allowing for some action-packed capture. Leading the way in this market is Snapchat with their Spectacles Bluetooth Video Sunglasses.

Audio glasses might remain as stand-alone audio devices. Augmented audio may also be incorporated into full visual and auditory glasses. But in either case, the focus on exceptional sound quality will pave the way.

Not a replacement

It is important not to get into a mindset that a voice interaction can serve as a replacement for a visual interaction. For example, adding a voice component is not an exact replacement for providing a keyboard. You also need to be aware that the design approach must be different. If you show a keyboard to a user, they will likely understand what action to complete thanks to their past experiences with keyboards. A keyboard, in and of itself, will communicate to the user that they need to enter each letter, number, or symbol. If they are able to do this with both hands, like on a computer, it may be an easy enough task. But if they have to enter a long search term or password using an interface where they have to move a cursor to each letter individually, this task may be greeted with intense resentment. One way to overcome this daunting task is to provide a voice input option instead. It is often much easier to say a word than type it all out. However, the process of inputting data this way is much different than with a traditional QWERTY keyboard or even an alphabet button selection, and it not as familiar.

When a user sees the letters of the alphabet or a standard QWERTY keyboard, they connect their past experiences with it, so they can easily start to navigate to the letter they want to choose (FIGURE 12.8). But when you are relying on voice, the user will connect to their past communication with other people as how to interact.

There needs to be something in the UI that communicates to the user that they can use their voice to interact. This is often shown through the use of a microphone icon. However, it can also come in the form of speech. One way to let someone know that they can speak is by starting the conversation with a question such as “How can I help you?” Depending on the device, and where it will be used, design this experience to match what will work best to communicate to the user that they can use their voice and to start the conversation.

What do you do first in a conversation? Before you speak, you might make sure the person you are speaking to is listening. But if you are speaking to a computer, you don’t have body cues or eye contact to rely on. So, this active listening state needs to be designed.

Though most of the conversation experience will not use visuals, this is one area where a visual provides great benefit. Using a visual to let the user know that the device is listening can help substitute for the eye contact they are used to when talking with another person. This can be a visual change, such as a light turning on or a colorful animation, so they know that what they are saying is being heard.

Companies like Apple have created their own custom circular animations that they use across all their devices; when a user sees Apple’s colorful circle of purples, blues, and white, they connect it with a voice interaction. Seeing this animation communicates that the device is ready and listening for a voice command. All of this happens instead of a keyboard appearing. So, it isn’t a replacement, but rather a totally different way of communicating, and therefore in need of a totally different interface and design.

Context

When people communicate, we use our knowledge of the context to create a shared understanding. Once the user knows the device is listening, they may know to start talking, but how does the user know what to say or even what is okay to say without any prompts? With a voice interface there is no visual to show what the options are. It is best practice to have some options voiced, such as “you can ask...” followed by a few options. You may be familiar with the options provided by a teleprompt: “Press 1 to talk to HR, press 2 to talk to the front desk.” Those are often very frustrating, because they are very one sided and not conversational. The goal here is to start by understanding the user’s goal.

This is where everything we have been talking about is coming together. Once you have identified the why in your project, planned out roughly how it might look, done user research, and created a user flow, you can start to predict some options that a user may be looking for. You can start off by letting the user know what their options are based on this research. This can be laid out in a set of questions or by asking an open-ended question.

When you record an interview with someone, it is the best practice to ask open-ended questions or compound questions. The reason is that you want the person to answer with context. If you ask two questions in one, a compound question, it is a natural tendency for them to clarify the answer as they respond. Perhaps you ask “What is your favorite way to brew coffee, and what do you put in it?” Instead of answering “French press and with cream,” it is likely that they will specify which of the questions they are answering within the answer itself. So, a likely response would be, “My favorite way to brew coffee is using a French press, and I like just a little cream in it.” Traditional media interviews don’t include the questions—so it’s important to get context in the answer. We’re discussing question and answer methods here because such exchanges point out an important way that humans communicate. We like to make sure that our answers have the correct context to them. This is especially true when there is more than one question asked.

The need for context is important to understand as it relates to voice interfaces: Humans may not provide needed context in their voice commands. Having the computer ask questions that are open-ended or have multiple options will trigger the user to provide more context in their answer, which will help the device more successfully understand what the user is asking. Using the power of machine learning and natural language processing, the device creates a system that recognizes specific voice commands. These commands must be written out as scripts.

Scripts

Think about all the different ways within just the English language someone can say no:

nope, nah, no way, not now, no thanks, not this time...

And this is just to name a few. You also need to consider what questions the user may ask and the answers they may give. This requires anticipating what the user will say and then linking that response to activate the next step. With a traditional computer screen, the user has a limited number of options based on the buttons and links you provide. With the input of a click or a tap, the computer knows to load the connected screen based on that action. With voice, the interaction is reliant only on spoken language.

As part of the voice interface design, an important step of the process is to create a script. This script should embrace the dynamic qualities of conversation (FIGURE 12.9). A successful script should go through multiple levels of user testing to identify questions that users answer—and also all the different ways they answer. When the user isn’t given a set number of options to choose from, the script helps translate the human response into something actionable by the computer.

While it is easy to tell a computer what to do if someone says “yes” or “no,” it is less likely that the user will stick to these words. Computers may easily be able to understand yes and no commands, but what happens when a user, who is speaking as they always do to other people, says “I’m going to call it a day.” Or “I’m going to hit the sack.” These idioms are not going to be understood by a computer, unless they are taught them. Without understanding the cultural context, it could be understood that you are going to physically hit a bag, instead of go to sleep. How do you anticipate responses of users to build into a script? You ask them, and you listen to them.

Use a script and thorough user testing to collect anticipated responses to help keep the experience complete. Multiple rounds of quality assurance testing must be completed throughout the whole process. Think of how color, image choice, and copy set the tone for a design in print or on the web. In the same way, sound quality, mood, and content of the responses of the voice will set the tone for the voice experience. As you can imagine, this is no small task. To do this well requires a team of people dedicated to designing these VUI skills. However, scripts that are created have components that can be used across the different skills.

With voice and choice of words comes personality. Many people have such a strong connection with their voice assistants that they refer to them by name, and they speak very highly of them, almost as if they are a member of the family. The conversational experiences we have will start to build a relationship with the device and will also help establish a level of trust. These experiences have a way of making an experience personal. Think about how nice it is when someone knows and uses your name. What if they could also learn and know your favorite settings and words? What if they could mimic these preferences as they guide you, provide clear instructions, and help reduce your anxiety by saying the right words? This is what voice experiences can do. They are rooted in our experiences of conversations with friends and colleagues, so it is no surprise that we start to trust them like one too.