The greeting

Consider our greetings to one another. Although it varies in different cultures, in the United States, you extend your hand away from your body offering a handshake to someone you’ve just met. This allows for connection, but at a comfortable distance. As you get to know someone and become more comfortable with them, you might open your arms in greeting and allow them in for a hug. This signifies that it is okay for them to come closer; you have let them in. A hug is very personal, as it is closer and allows for more of a direct connection. Have you ever hugged someone the first time you met? I have. Usually, it is someone who is strongly connected to someone you already know and trust. You feel like you already know them from all you have heard about them, and your hug signifies your acceptance of them as a new friend or member of the family. The most personal greetings are when you allow contact with your face, perhaps for a kiss on the cheek or two—or for those to whom you are most connected, the lips. The closer you are invited into a person’s personal space during a greeting, the more genuine and personal it should feel.

The face is an extraordinarily personal space, one we protect and safeguard. Understanding this aspect of social interaction is important as you set out to create an experience that is in direct contact with the user’s face. Understand that it will need to be introduced to a new user carefully and, for some, incrementally. The greeting needs to be personal and establish trust, as they are going to be allowing the experience into their spaces, trusting you with what their eyes see, what they can hear, how they are feeling, all while coming in close proximity with them (virtually and physically). If at any time during that first introduction they feel too uncomfortable, they will retreat and may hesitate to try to connect with the experience again. Just as in a normal greeting, there are things you can do to help ease any uncertainty by providing them knowledge about the experience and what to expect so that they feel comfortable with that initial hug or kiss.

Our insight into initial greetings and interactions reached a pivotal point during the COVID-19 pandemic, when hugs turned into elbow taps and grandparents used plastic separators to hug their grandchildren. A generation of children have been instructed to not greet others with a physical interaction, and it will be interesting to see how these influences will change greetings as we know them. If anything, it may open people up to be more accepting of virtual greetings and interactions in digital spaces as a result.

When Tim Cook introduced the first Apple Watch in his 2014 keynote, he explained how revolutionary it was by calling it the most personal device the company had ever created. It was the first time Apple released a computer that was so personal that it would be worn on the body. The audience was excited, but would people actually purchase it? A couple of factors helped encourage buyers to take the leap. First, it was an expansion and a companion to familiar and possibly trusted technology: the iPhone. Accessories, such as arm bands to hold an iPhone or iPod while exercising, were already common. This reduced the jump required for potential users to trust something that they wrap around their arm. Wearers of traditional watches were accustomed to consulting them, so trying an Apple Watch may not have felt as monumental. All of these steps built trust, so when users first encountered an Apple Watch, they did not hesitate to wrap one around their wrist and start discovering.

As a designer, what you need to understand is that all the factors just listed are important for the user, whether they’re aware of them or not. It isn’t just the first time someone puts a device on and starts using it that counts; it is the steps and actions that have prepared them for that moment and initial introduction.

Just as none of us had worn an Apple Watch prior to 2014, today many of us may have never worn a head-mounted display, but many of us have worn glasses, either prescription or with UV protection. So, the concept of having something so close to our eyes is familiar. Those glasses have lenses but do not have screens built in. Still, there is some sense of comfort if the head-mounted display looks like the glasses we have worn before. If there is a band that wraps around the whole head and has wires attached to it, that likely reduces the familiarity of the experience. It is harder to relate to.

Observing first-time users of any type of HMD shows clear evidence that there is a sense of discomfort in putting the device on their head. All the interactions I have observed at trade shows, at conferences, and in classroom demos have involved one person explaining and walking though the technology before handing over the device. I have seen those who are the guides try on the headset first to make sure that the menu is correct and everything is reset so that the user can enter the experience at an optimal point. This also allows for the user to see how the device is worn and how to interact with the different parts of it. Even so, a rare few have just put the device on immediately after it was given to them. Many ask one or two questions before they attempt to try it for the first time. It is obvious, even from a distance, who has used some type of XR display and who has not. There is an initial orientation period that you can see in a first-time user who is trying to understand their positioning in the new 3D world they have entered. Repeat users approach this initial step with more confidence and more trust.

Orientation into this new form of reality is essential to establishing trust. The first step is setting up an onboarding tutorial for the user (FIGURE 6.1). Although it is common to have someone with you for your first VR or MR experience, it is much less likely that you will have someone to walk you through a mobile AR experience. Knowing that new users will be navigating their first experiences alone, offering these introductory steps will make a better overall experience, especially for those without a human guide.

Think of this experience as the equivalent of the guided tutorial a person standing next to you would provide. It should help instruct and guide the user through any necessary setup first, such as creating an account, scanning a room, or finding a plane. Clear, focused instructions with visuals are essential here. We will discuss how to design this further in the next chapter. It is also important to explain and show the main functions of the experience that will help the user achieve their goal. Because there are not yet universal standards for XR, it is safe to assume that you will need to provide these instructions and have them available for your users to access at any time. You don’t need to show every function here; in fact, that would likely be overwhelming, so just focus on essential actions that will help guide the user to get started. This helps the initial interactions feel more approachable as they have some familiarity versus leaving the user feeling completely lost. Lost and insecure are not great first impressions and will likely lead you to lose the user.

Positive user experiences

First impressions matter, and in XR they are essential to creating a positive user experience (UX) for the user. The UX is the design that happens behind all the visuals. The role of the designer is to create an environment where the user understands the space and is able to distinguish the physical and digital worlds. This goal can be achieved using both diegetic and non-diegetic elements. The UX includes many components that work together to help the user achieve their goal. This includes three main components.

INTERACTION More than just a user and an object, but rather the result and impressions created by the user from interacting with that object, as a result of how it is designed. These impressions are emotional.

CONTEXT The situation where the interaction occurs in collaboration with the user and the object (or service) provides the context.

FUSION The user experience is multifaceted. It is constructed by a combination of all experiences, both the single experiences and also the combination of all those experiences together.

Having a thorough understanding of how to make a good user experience for an interaction outside of XR can help you master the challenge in this space. Many of the same rules and theories apply. Anytime you have a computer and human interaction, there will be a number of considerations that need to be addressed to help the user and make the interaction successful. The foundation for this is the consideration of how interaction, context, and the fusion of experiences all work together to aid the user toward their goal. Working in an immersive, 3D environment adds complexity to creating a good user experience. Some additional elements are unique to this industry and should be considered for an optimal user experience in XR, for example:

• Personal space

• Agency

• Social signifiers

• Feedback

• Affordances

• Interactions

• Safety

• Guides

• Cohesion

It’s worth delving into each of these elements, in addition to the well-known elements of any digital experience.

Social signifiers

If someone has a red flag up on their mailbox, what does that mean? What about lines painted on a street? Knowing that a dashed yellow line in the middle of the road means it is safe to pass someone in that area is a social understanding (FIGURE 6.2). The context of these signifiers is also essential to understanding their meaning. Social signifiers can communicate critical information for the user to understand an experience and can also influence their behavior and actions. Signifiers can be used to help users understand what action should be taken without the need for instruction. If someone raises their hand in the context of a classroom, it is understood that they have a something to contribute. This idea has been converted into the digital environment and through the video conference platform Zoom. A student studying remotely can click the hand icon, and the instructor will see the raised hand graphic on the student’s video thumbnail, mimicking the real-world action of raising of the hand. Just as this action would grab your attention in a classroom, Zoom replicates that action by bringing the student’s video thumbnail to the top of the stack or grid. Without needing an explanation, the first time you see the “raised hand” interaction, it is clear that the student has a question. Utilizing strong signifiers that exist in the physical world in the digital world will make an XR experience feel more approachable and natural to a user.

Social signifiers A visual cue that carries meaning that is based on a social or cultural agreement.

Interactions

Once you understand that you can interact with an object, how do you actually interact with it? There are three types of interactions:

• Direct interactions in which you physically touch the object or screen you are using, such as using a touch screen

• Indirect interactions in which you use a physical object to control a digital one, such as typing on a keyboard or scrolling with a mouse or trackpad

• Semi-direct actions, which are a mix between direct and indirect interactions such as scaling an object by pinching directly on the object and then tapping a button to save the size property

Two of the benefits of interactions within XR are the high proportion of direct interactions and how active and important your hands are to interact with the interface. Gestures, clicks, and drags can all be done with your hands or hand controllers (FIGURE 6.3).

1 Feix, T., Romero, J., Schmiedmayer, H., Dollar, A. M., & Kragic, D. (February, 2016). The GRASP taxonomy of human grasp types. IEEE Transactions on Human-Machine Systems, 46(1), 66–77.

Understanding how we can interact with an object in XR should be easier than using other screens that require the use of a flattened version of an object. Use this to your advantage. We have an established understanding of how objects behave, so use this knowledge to influence the actual interaction. Consider the use of 3D or volumetric representation of objects when it seems appropriate. This way you can really leverage the benefits of this immersive technology.

Planning out interactions to minimize the number of different types of interactions will go a long way to improving the user experience. The fewer kinds of interactions, the easier they will be to remember. The more natural the gestures used by your hands, the easier the learning curve and retention of how to interact throughout the experience.

Guides

When you are visiting a new place for the first time, a tour guide can be helpful to help you find all the must-see places. The same is true when you first explore an XR experience. Use visual cues to help direct the user to discover the space and to show them where they can explore all the must-see places and experiences. Depending on the field of view, it may be hard to see an experience in full, so directors that help guide the user to needed places are an essential visual tool. These can be anything that captures attention, such as arrows, lights, colors, motion, sound, path, lines, or anything else you come up with. There are many useful AR applications that use these guides as the main function of the experience. Navigational AR, for example, is all about creating these visual directors to help find something as you are driving, walking, or even in a grocery store. If you want to make sure that the user finds something, guide them to it (FIGURE 6.4).

While exploring deeper into a 3D space, users can easily lose their way or be unsure what they are looking at. Guides can play an important role in keeping the user oriented in these situations, too. In addition, guides can also help the user navigate the digital and physical worlds in more effective ways.

Develop user personas

With so much of the design process focused on a specific target audience, it is important to include that audience as part of the design process as well. They can provide feedback and insights that only a user may consider. To start, identify three to five diverse user personas that you can use to test your user flow. To create these, it is best if you can use real data about actual people that can be collected from interviews, surveys, or analytics. A combination of qualitative and quantitative research should be completed and analyzed. Although we as designers may not be responsible for this process, it is important for us to be a part of the research that is being completed to ensure more well-rounded and encompassing results. From this data, you can start to formulate a few specific personas, each representing a realistic reference as a user. For each persona created, it is helpful to identify their goal as a user, their demographic information, and any additional information that is relevant within the context of the experience (FIGURE 6.7).

According to usability.gov, “Your personas are only as good as the research behind them.” To ensure that your personas are effective, they should:

• Represent the main user groups for the experience

• Express the needs and expectations of that group

• Share their expectations and how they anticipate they would use the product

• Help identify any needed features and functionalities

• Represent real people with diverse backgrounds, goals, values, and a range of experience using XR

• Represent a diverse group of physical and mental abilities

Walk through with personas

Once you create some user personas, you can take a more specific user journey, walking through your user flow with the goals and needs of one of those personas in mind. Imagine each user persona as they walk through each step of the process, and create user stories for them (FIGURE 6.8). At the outset these stories must:

• Identify the type of user

• Identify what their goal is

• Identify the motivations driving their journey

To do this well, empathize with their needs and values. Consider their feelings and as a result what emotional decisions they might make. A mother shopping for her child will have a different emotional connection to a product than when she is shopping for herself, for example.

Let your research inform the decisions as you walk through the user flow. This promotes human-centered design, where you place the people you are designing for as the focus for the experience. Then, conduct testing to confirm the success of these stories with real users.

In XR especially, adding any sense of humanity into a very computer- and tech-heavy field will make the experience more approachable. Using user research, such as observations, interviews, and evaluations and analytics, to guide your design decisions keeps the experiences human centered. Using the mindset and data of a variety of user personas, journey through the user flow chart and try to achieve their goals. Take notes. Pay special attention to areas that are possible pain points, where you are unable to achieve the goal, or any other challenges that arise.

Afterward, you want to go back through your user flow. Adjust it to help resolve any issues you found on your walk-through or make notes on customizations needed to accommodate the variety of needs of your target audience. This is an important step. As helpful and great as user personas can be, they also can be misleading if you begin to categorize users into a small number of kinds of users. The goal of using these user personas is to walk through the process through the eyes of people different from you. That doesn’t mean that a persona represents all users. It is important to remember that people are unique and each person may have unique challenges that should be part of the design process. We will explore this even further in the next section, so stay tuned, but it is important to understand that as you explore a persona’s user journey think of it as the mind of one user, not all users.

Designers need to have a solid understanding of why they are including an object in a design and why they are placing it in a particular place. Users don’t need to know your reasoning, but they will appreciate how it helps them. In fact, if they don’t even notice some design elements, but instead notice only the feeling of accomplishment from interacting with them, then this is a success. Making research-informed design decisions can help you work toward this success. Listening and watching are incredible design tools, and they are essential to gaining a true understanding of what your users need and want.

Focus

Could you imagine what it would be like to learn and create or perform in the same space? Many areas of education, training, and medicine are early adaptors of this concept where they can clearly see its benefits. With so many screens grabbing our attention it is hard to stay focused on one area. This can be even harder for those with cognitive processing disorders and learning disabilities. However, learning disabilities don’t end when we leave a classroom, as they extend far into our everyday lives and careers. Having the ability to combine our focus into one area, to both learn and then perform, reduces the cognitive load that is currently needed with multiple windows, devices, and screens. For anyone who faces challenges with ADD and ADHD the ability for focused attention is essential. Combining content into one space can assist in keeping a visual flow in one central location.

Memory is also connected to our ability to focus. It is important to be respectful of the fact that users will have varying lengths of memory, and some will have short memory spans. A common practice in form design for web and mobile is to keep the forms clean to be more efficient with space. However, if the tool tip is in the form field, then it disappears when the user tries to type. This can be even harder when there is a specific format that is required, such as a date. Was it 8/14/20 or 08/04/20 or 08/14/2020? It can be frustrating to the user to not have this example to view at the same time they are entering the information. This may save space, but it is not as accessible to the users.

Focus also relates to our vision and our eye sight. One of the challenges with HMDs is that they can be hard to use for those with corrective lenses, such as glasses. Depending on the device, you may have to remove your prescription glasses to wear an HMD, which would mean leaving your clarity and focus behind. However, several manufactures have been working to overcome this. Magic Leap, as an example, announced a partnership with Frames Direct to produce compatible prescription inserts for HMDs that are anti-reflective, are light managed, and even have eye-tracking features. However, there are other eyesight challenges that can add additional challenges for users. With so many varieties of vision needs—from lazy eye, astigmatism, cataracts, or increased sensitivity to light to eye surgery complications to the low vision that is increasingly common with age—there is not a one-size-fits-all design. Instead, look for ways in which users can customize and personalize their experience to optimize their view. Learn about some of the challenges that your users face so you can use that information as a launch pad to determine settings and options that they user can explore.

Multimodal

There is a difference between requiring two hands to engage with an experience, which is likely to exclude, versus designing it to work in alternative ways such as with two hands, one hand, or with speech. Offering alternatives allows opportunity. As we discussed in Chapter 3, “The Immersive Experience,” providing multiple kinds of information, such as an icon and type or audio and a visual, expands the reach of who can access that information. If a user doesn’t understand the language being used, an icon can help them derive meaning in spite of the language barrier. Providing a combination of audio and visuals in an experience expands the accessibility for anyone with difficulty hearing, deafness, low vision, or blindness.

These efforts often assist the larger audience as well. Those who have a specific learning disability, such as dyslexia, can benefit from the option of more than one delivery mode. Although these are often permanent challenges, it is important to account for temporary and situational elements as Kat Holmes reminds us. Finally, engaging more of the senses makes more of an emotional connection to the information, which will result in more engaged users. The ultimate goal is to make sure that you are not relying on just one sense to deliver the experience. This will create a better user experience overall and make it more accessible too.

Personalization

You can’t design a different experience for the needs of every person. However, you can allow opportunities for the user to have control of their experience. To do this effectively, as always, it is important to learn more about the needs of the user. Then you can offer what is most helpful versus allowing the user full agency to control the entire experience; we have discussed how too much control can create confusion. Settings such as volume, brightness, type size, and speech control allow personalization without changing the overall experience. It is important that a user be able to access the options without having to quit or leave an experience. Just as you always have access to a volume and brightness control on a personal computer, it is the expectation that these options will also be available in an XR experience.

With immersive experiences there are additional factors that can affect the comfort level of the user. This includes sitting, standing, flexibility, range of motion, height, and physical environment factors to just name a few. Because the experiences are fully immersive, it means that more of the body is used and can require the need for more customization. The need for these customizations is not new, however. Think of gyms and similar places; people of all different heights and capabilities are able to use the same equipment. The setup allows for each user to select their own weight and adjust the positioning of the equipment. For most equipment there is a primary interaction method, but alternative solutions are possible and often shown in both written and visual format. These solutions can hold valuable information to how product and industrial designers have formulated viable solutions that users are already used to interacting with. That way they don’t feel as new and unknown to a user during their first interactions (FIGURE 6.10).

It is important to design with more than one user in mind, and as you walk through your user journeys, consider what customization options would increase the inclusion of users. Just as important, go through the user flow and consider who is being excluded from the experience and what alternative options can be provided to improve their experience.

Context

Understanding the motivation as to why someone wants to interact with a solution or product is invaluable information that can help provide context to the designer. The why is an essential part of any design process, so that part shouldn’t be a surprise. However, providing information in context is a game changer, and you’ll want to harness that potential. In AR, the context is the physical world, and as the physical environment changes so does the experience. So, AR and MR do have more built-in accessibility because users can utilize some of their own adaptations from that space into the experience you have designed.

Consider someone with Autism Spectrum Disorder. Though it varies for each person, one challenge can be social interaction. AR/MR using object recognition, and voice technology all have the ability to determine emotion. In fact, Amazon’s Lab126 and the Alexa voice team have been working on a wearable device to “discern the wearer’s emotional state from the sound of his or her voice,”³ as stated by a Bloomberg report in 2019. This concept allows users assistance in identifying emotions, in context.

3 www.bloomberg.com/news/articles/2019-05-23/amazon-is-working-on-a-wearable-device-that-reads-human-emotions

Another example is in education. There are many different learning styles and learning challenges. However, research shows that hands-on experiences allow for a greater retention of information. Again, more senses are engaged, and the connection becomes more emotional. Think of being able to engage with an object (even a digital one), in the same context that you will one day have to engage similarly in a fully physical experience. This allows you to practice and learn in the same space. An example of this is in surgical residencies where doctors are learning to perform surgeries (FIGURE 6.11).

This can apply to design as well, as you consider how you learn computer software. If you watch an in-person tutorial, you often have to look up at a projection screen to follow along step-by-step and try to apply the same thing in your workspace. If you are watching a video tutorial, then you have to continuously switch between windows and views. Imagine if you could get that tutorial overlaid into your workspace so you could see where the tools are and select them to start creating all in the same view. This could help increase the efficiency of how we learn, and for those with specific learning disabilities, that learning in context has the power to be life changing.