Solve a problem

If you started this journey thinking, “I want to create an XR experience,” and are now seeking an experience to design, then you need to change your approach. For this experience to be practical and helpful to your users, you need to first identify the reason that this experience will exist. There are plenty of examples of XR that merely show the technology, but these examples do not tap into the full potential of what this immersive platform can provide. Sure, it is fun to see dinosaurs made of paper walking down the street in front of you, but what problem does that solve? What do we gain from this experience? Consider why people would want to experience this. Perhaps they want to learn more about dinosaurs. Perhaps it is to show the size proportions of a dinosaur to our modern spaces. There are likely many other reasons you could brainstorm to determine why a dinosaur’s AR experience could be beneficial, but the one you choose will influence all the other design decisions that come next in the process. Why are there dinosaurs? Why are they made of paper? Why are they walking down the street in front of you? Without this foundation, people don’t have a reason to come back to the experience over and over again. Once your user has experienced it, they move on to the next thing—that is, unless you give them a reason to return.

If you are in search of a purpose for your project, two approaches can be helpful.

STEP BACK AND OBSERVE. First, you need to step back—literally, away from your computer, your screens, the rectangles that confine our views—and be inspired by actual reality. Bring a pen and something to sketch on with you wherever you go, and just observe.

IDENTIFY WHERE PEOPLE FACE CHALLENGES IN THEIR EVERYDAY LIFE. It could be reading a menu from far away, finding something in a grocery store, or trying to watch the road and traffic while simultaneously watching the turn-by-turn directions on a GPS. Think about a common frustration in your life or that you have observed others around you experience to see if there is an opportunity for an augmented experience to improve it or reduce the dissatisfaction.

Remember when you are interacting with an immersive application, you are experiencing the final product that came to fruition only because of the many rounds of evolutions, exploration, and brainstorming. Every product started off at the same place you are now. If you focus on the end product, the ideation task may feel overwhelming, but breaking it down to really look at what you are trying to improve or the problem you are trying to solve will make the creation of the product a more approachable task.

Form follows function

In consulting with a student on which technology they should use for their project, the first question I ask always is, why? Why are you creating this experience? What do you want people to learn, feel, or experience while interacting with your project? The answer guides the choice of specific medium. In one case, the student project was an art installation that was created to make people more aware of the light in their environments. With light as a primary inspiration for the project, it was clear that the medium selected should celebrate light and make people even more aware of it. Projection mapping was the direction they selected as a result, which re-created moments of light into the different spaces they installed the project (FIGURE 4.1).

Once you make your intent clear, the path will also seem clear. But if you skip this step, often by selecting the technology or medium before you have spent the time to decide, then you will miss an opportunity to make your experience as effective as it can be. You have to be honest with yourself that you may have jumped too far ahead in the process before exploring your options and doing the necessary research before choosing which way to go. If you take your time in this step, then you can move forward confidently knowing that this is the best way to design the idea you are working on and that you are designing with the right media. Your content controls the context—not the other way around.

When approaching a design project, you have to always consider the best medium for the product to be delivered. There are cases when print will be more effective for delivery of the information just as there are times when digital would be a more efficient delivery than print. The same is true in XR. You have to make sure that the problem you are looking to resolve through design is best approached using this technology. It is easy to be intrigued by the shiny-and-new mentality, but in order to really give the project the consideration it deserves, you need to feel confident that AR, VR, or MR is the best platform to create the experience and deliver the information. In fact, this is important in determining if you should use mixed reality versus augmented or even virtual. With more and more technologies emerging every week it seems, knowing why you are creating the experience will help you with the many choices that you will be faced with along the way. Consider all the possibilities.

Never forget a name again

As a professor, I meet 100 or more new students every semester, and I am always struggling to remember their names. When coming up with the concept for tagAR, I was looking to create a tool that helped people helped solve this problem. Think of how impactful it is when someone you have met only once comes up and addresses you by name. That was the scenario I wanted to help facilitate.

IDENTIFIED PROBLEM. Forgetting people’s names, as well as the need for multi-modal introductions, offering auditory and visual cues to assist those with dyslexia and other specific learning disabilities.

MEDIUM. Augmented reality mobile application for initial launch, but ultimately created for a hands-free experience using AR glasses with built-in speakers.

ADVANTAGES. AR allows the user to see name tags right where needed, above a person’s head. Using AR glasses, the user can look the person in the eye and address them by name. With spatial speakers that are part of AR glasses, users can tap on a name tag to hear the correct name pronunciation in-ear before having to say it out loud.

I know many people have trouble remembering names for various reasons. The specific inspiration for this AR app, however, was to help those who are dyslexic. I am a big advocate of looking at ways to help overcome dyslexia, as well as other learning disabilities, without those who use the assistance having to self-identify with their challenges in order to receive assistance. I have been coming up with ways to help overcome my own dyslexia throughout my life. I have found that AR offers powerful opportunities to provide assistance to overcome some of the challenges faced in educational, social, and communication environments. While the target audience is intended to be much wider than just those with specific learning disabilities, that was the initial motivation. Features of the application were created to offer both visual and auditory versions of each person’s name. The visual is an augmented name tag that appears just above each person’s head, which is intended to replace the traditional sticker or lanyard name tag. In addition, when the user taps the play button on a name tag, the user can hear the correct pronunciation of that person’s name (FIGURE 4.2).

Using AR for this application makes the searching, filtering, and identifying of people much more efficient. It was pretty clear at the initial stage that AR was the only way to develop this idea, because this concept works only if you can see the people around you while at the same time adding the additional layer of information, their names. Because print versions of name tags already existed, I could more easily research what worked well and what could be improved with this application. This solution also solves the problem of name tags that flip to show their back, hiding the name as a result, as well as the problem of a person’s long hair sticking to the adhesive. Most of all it avoids the awkward action of looking down subtly, to get a glimpse at the name while you are speaking to a person, without making it too obvious—which would make them realize that you don’t know their name.

The additional gains realized by offering the tags in AR are the digital quality (removing the need to try to decipher someone’s handwriting), the ability to search for someone around you, accessibility benefits (offering a multi-modal experience with an audio component beyond the first introduction), and the ability to connect easily with people around you without the exchange of a business card or trying to remember the spelling of their names later. These all led to concepts and functions being created and tested in the app. All of this started from the singular idea that answered why this experience was being created in the first place.

Merging realities

To further expand your ideas for the potential of what AR can provide in this ideation stage, consider how you can merge realities. How can the experience be improved by combining things that will already be in the environment with the digital objects that your XR solution will provide? Look for consistent elements. If you know that people will always be using the application in a specific location, for example, you can use that to inspire the experience.

One experience that I was designing for a children’s edutainment experience was heavily reliant on the idea that kids would be using this app in their homes. With this constant, you can search for a why: Why would kids want to use this experience in their homes? With so many other toys, devices, shows, games, you name it, why would they want to engage with this in their home? The answer came by exploring what all homes may have in common to enhance the experience. The solution led to the incorporation of doors within the home to serve as portals to other realities and locations. This allows for kids to experience their homes like they never have before. That is the why. You may be used to your home, but this experience offers a chance for children to experience their homes in a new and even possibly magical way—in a way that taps into their imagination, using the built-in structure of their home. The door that used to allow them to walk into their bedroom could now take them to a whole different world that they could only have experienced in their imagination previously. A huge benefit, and to be honest also a huge challenge, to augmented reality over virtual reality is that you can leverage your concepts into the existing environment which can be forever changing. It is reliant on the location of the user and not on what was only produced for the experience.

As you brainstorm and sketch, be inspired by the possibility of the ever-changing locations as well as any consistencies that can enhance your idea. Perhaps your idea is reliant on someone being able to get a break from their current environment, then a virtual reality experience might be easier. The Stanford CHARIOT Program allows children to take an adventure using their imaginations and XR technology to get away from the medical facility they are currently in to a place that can distract from their stressful real-world situation. Identifying the challenge of giving children their needed shots in the doctor’s office, paired with their love of stories, characters, adventures, and games, you can see how this idea was born. But it all started by identifying the problem (FIGURE 4.3).

Starting a new project in XR, especially your first, could easily overwhelm you, but if you change the perspective of your approach, this could lead you to a solution you may not have considered otherwise. If you can easily explain why you are creating your experience as well as why you are using the tech you are, then you are ready to explore your idea in the next phase.

Now you should have the understanding that the why is important, but to stand with complete confidence behind your idea, you should also be able to easily answer:

• Who your project is for?

• When will it be used?

• What length of time if is used?

• Where it will be used?

• How best is it used?

• How will it influence the user?

In this chapter we will explore these questions one by one throughout our discussion of the ideation process.

You don’t have to be innovative

The risk of this rush to be innovative has its downfalls. Many businesses and companies release products before they are ready, quite possibly so they can claim to be the first to do so. They also may not spend as much time, energy, or focus on the design of the experience as needed in order to get the working product out to market to claim the spot as first. This never ends well, however, as customers who don’t have a positive experience with a product are less likely to return as a repeat customer. The push to become tagged as an innovator often leads to poorly designed experiences, as has been the case in the XR world. There is a flood of work created from programmers and developers who are often first to get their hands on the new technology. This leads to the creation of products that become examples of how the tech can work, but little more. Without the attention and use of design that these experiences need, users don’t really have a strong connection to the product and consider it a weak experience. A great idea with a poor execution becomes a failure.

This is why design is such an important part of the process and why it needs to be considered from the very beginning. Early adopters of technology can often see the potential of what their ideas can become with time and a focused vision. To me, this is much more innovative. Don’t create an idea to simply use the technology and figure it out from there. Focus your creativity toward an end goal that can be achieved at a higher standard through the use of emerging technology.

Remembering your goal, your purpose—understanding why you are creating the experience—will keep you focused during the process. As Steve Jobs said, “I’m as proud of many of the things we haven’t done as the things we have done. Innovation is saying no to a thousand things.” You can see how this can be true in other scenarios, such as if you have ever gone grocery shopping without a list. With so many options and things you could buy, you may come home with bags full of food, but nothing to eat for dinner or a complete meal. Focus. This may feel intuitive, but when searching for innovative solutions, there is more distraction caused by seeing and trying all that is “shiny and new.” Technology is evolving and improving at such a rapid rate, it is phenomenal and also overwhelming. Trying to create content for multiple platforms may not be possible with these changes happening so rapidly; instead, try focusing on one, for the moment, getting that experience to where it becomes successful and impactful to the user, and then looking to expand from there. Expanding too quickly can be costly, both from a developmental as well as a personnel perspective. Do one thing well on one device or in one reality even, before you try another.

Make thoughtful choices

If you were about to go for a run, you would have to select a pair of shoes to wear. It is more practical to wear a pair that is comfortable and that will help you as you run. In fact, there are shoes made just for the sole purpose of being worn for running. You can even be so selective as picking a pair based on your terrain preference. This decision improves your overall running experience. The same should be true for an immersive experience. You want it to be practical for the users.

It’s vital to have answers to these essential questions:

• Where will users be when they interact with your experience? Indoors? Outdoors? In public? In private?

• For how long will they typically be interacting?

• Will they be sitting, standing, walking around?

You need to have answers to these questions before choosing your technology. If you know that people will need to be walking around while engaging with your experience, then it would not make sense to choose a VR head-mounted display where users can’t see the world around them; a better choice would be an untethered device that isn’t reliant on being connected to a computer. You need to make associations between the needs of the user and the goal of your concept.

The power of deadlines

Stress and time constraints confine innovation. As does lack of sleep, taking the same route each time you go somewhere, and working toward perfectionism. Knowing these variables will alter your ideation process, you want to try to find optimal times to work and focus on your ideas. If you are like me, having a deadline is helpful to completion of a project or task, but at the same time, it can add stress if you haven’t planned ahead of time and managed your time. Having a deadline creates a need for definitiveness which magically allows you to make final decisions and to get things done. However, use the time before a deadline for this purpose—to make final decisions on anything you have been deciding about—versus to create new ideas, as they will be strained with the pressure.

With new technology, or really any technology, you need to be realistic about the possibility of change. Ideas on their own don’t make a difference; things do. You have to continue to work ideas into a real product to start to measure their impact. Working within this field there is great potential to make a difference and a change in people’s lives. Do this, and you will create experiences with purpose.

Promote divergent thinking

If you hand a child a basket full of blocks, they can start building towers, cities, roads, houses, towns, anything they can imagine. Each time they sit down with the same basket of blocks they likely will come up with a new design. This idea of divergent thinking is an essential building block (pun intended) to the ideation process. To find the best solution, or even gain confidence to choose a specific direction, you first need to explore multiple solutions.

What are you thinking of right now? As you read, perhaps you have an example or use case in your mind. As you digest each new sentence, you think how it could apply to your project. Now, whatever you do, don’t think about a red apple. In fact, stop reading for a moment, and focus—close your eyes if it helps—but think of anything but a red apple (FIGURE 4.4).

How did it go? Chances are, no matter what you did, you thought of a red apple. Maybe you tried to consider a green apple instead just to follow instructions surrendering to the fact that apple was now in your head. This illustrates how our thoughts are highly influenced by stimuli around us. In this case, it was the object I placed into your mind, an object that was likely not there until I introduced it. Knowing this, think about how going for a walk or taking a different route home can suddenly inspire new thoughts and ideas. When you surround yourself with a different environment, filled with different stimuli than you are used to, the energy changes, and your perspective changes with the unfamiliarity of your route.

Random influence

If you embrace the idea that our thoughts can be influenced by an external source to introduce new, different, and additional thoughts, then whenever you are stuck or feeling uninspired in your brainstorming sessions, introduce this notion into your ideation flow. This is in essence the fluid process of coming up with an idea or concept. The expression train of thought provides a visual of what this connected thought process looks like. It starts the moment you are faced with a problem to solve. To start this flow of ideas you can try a few different techniques. One technique is to add a random object or word into your thought. This simple step can help diverge your overall thought flow. The idea is to change how you were thinking about the original idea.

If we return to our drive-thru example, suppose you were thinking about a concept to create a new, immersive way to display a menu in a restaurant. Paradoxically, what you already know about the ordering process can become a limiting factor that is challenging to overcome. If you use a random word to filter or pivot your thoughts, however, you can start to ideate concepts that you might not have previously considered as a result. For example, an online random word generator produced the word traffic for me. Use that as a prompt to influence how you think and specifically what could be improved in the drive-thru as it relates to handling traffic and traffic flow considering:

• Challenges of high-traffic times

• Differences at low-traffic times

• Optimal placement of the menu

• Location of pick-up windows

• Distance of the drive-thru in relationship to the main street to accommodate traffic

How does this change with the influence of the word frame, and how about health? Each of these words were provided from an online random word generator, but you can see how the juxtaposition of them with the project focus can generate additional thoughts and considerations that you might not have considered without it.

Narrow with convergent thinking

This is the beauty of divergent thinking and outside thinking. You can open up the possibilities to explore for more than one solution. Every idea that you think of will not be viable. So, once you have thought wide and far, you will then begin the process of convergent thinking, where you narrow your ideas from the large pool of options to one or just a few of the top solutions. These can then be refined in preparation of the creation and testing stages that will follow. It is much easier to narrow in your options in this way of thinking after you have spent the time exploring a wide and vast array of options. If you choose one direction from a list of three possible ones, it could be okay, but with so few options it is hard to really understand rank. However, if you choose one direction from 150 options, then you know that that idea ranked higher than 149 others, giving you more confidence in your solution.

If we go back to our drive-thru example, after you have explored a number of different ideas, you would then narrow to just one approach to explore as a solution to just one problem. If you determined from your brainstorming session that your ideas about health in the drive-thru experience was the most important direction, then you would deep dive into that one topic. How can health and nutrition be communicated to consumers as a part of the drive-thru experience?

A typical drive-thru menu is limited in size and already is full of information, as shown in FIGURE 4.5. If this menu could be viewed in AR, then the information that was provided could be customizable to the user. They could choose what food options they would like to see, filter by any dietary restrictions, or have all the nutrition facts at a size that is visible for each menu item, just to name a few opportunities. These ideas will come only when you spend time exploring one single idea in depth using convergent thinking.

As you explore this idea be sure to spend time just looking. In fact, think of it more as listening with your eyes. Instead of just coming up with ideas about how to make the drive-thru process more immersive, go to the drive-thru. Watch others order at a drive-thru. Now, as you start to explore your idea with a more narrow focus, you need to consider the where.

• Where do you anticipate people will be using the product or experience you are creating?

• What is there at that location?

• What do people do there?

• Where are the problems that exist currently?

Look for themes and patterns

If you consider designing an augmented drive-thru experience, you have a very clear and focused location to answer the question of where you are designing for: a vehicle. However, it is even more important to identify priorities for the design. In this case, it is the driver who needs to be able to access the menu information easily as they will have to place the order while also keeping their hands on the wheel and keeping their eye on the cars and other barriers around them. By exploring what you know about the driver’s needs, you can start to come up with ideas for how this experience could be improved through AR. Even if you have driven through a drive-thru before, you want to make sure that you do a test-drive using and trying out the different designs to identify your own pain points through the lens of considering how AR could improve the experience.

Having this hands-on experience will offer one point of view. Then, you need to start the process of looking and really listening to what you see by watching others as they go through the full process. To actively be looking, you need to just make observations. Try to avoid assumptions. The goal is to identify themes and patterns. Make sure to observe a diverse group of people. Mothers with a car full of kids might have different needs than a businessperson on their lunch break. Through observation of a wide variety of people, the patterns that arise will become clearer when you see them as a constant need among quantities of people all with different goals and needs.

Reimagine how we see

Designing for XR is about reimagining the way we see. But to change the way we see, we first have to look. To come up with unique ideas it is helpful to know how we create these images or visual ideas in our minds. We have discussed how to listen with our eyes in order to really look, and the next step is how our minds perceive what we are observing (FIGURE 4.6).

Activity in the primary visual cortex, commonly referred to as V1, is where the conscious processing of visual stimuli occurs in the brain. The primary visual cortex works as a system that starts with your eyes, specifically the retina, and is transmitted using the optic nerve to both sides of the brain, or cerebral hemispheres.

Activity in all these different areas come together to coherently comprehend the visual scene, including depth perception. We will discuss this further in Chapter 8, “Human Factors,” but in essence it is important to understand that as we walk into a space we rely on our vision pathways to transfer all the visual signals we see from the retina to the visual cortex so we can orient ourselves in new spaces. We, without even consciously knowing, create a 3D model of the space including trying to identify which direction is up. Once we can create that in our minds, we must then understand our position within it. This is an essential part of the creation process to understand so that we can then mimic the way the mind orients itself in space into the virtual and digital spaces we are creating.

Have you ever walked into a space and had to stop to take a minute to look around to create a better understanding of where you are in context of the space? If you open a door expecting to walk into a hallway and you see desks and people sitting around, you will be taken back and likely will retreat for fear of embarrassment. This is part of understanding how humans comprehend space and their environment. Start to pay attention to the thoughts you have as you enter an entirely new space; these steps are essential to your orientation. This orientation process is the foundation of the launch of every XR experience.

Google Glass example

In 2014, Google released the Google Glass, which was Google’s first wearable product. This is a classic of example of why empathy and understanding your user are necessary parts of the design process, both in traditional design and even more so when exploring new territories such as the XR space. Google made assumptions about the users and did not clearly define what problem the product was solving for the users. It was as if they expected people to buy the product based on the hype; Google put being first before being the best. They lacked a clear understanding and communication about the best use cases for the product and how the product would fit in to culture and daily activities (FIGURE 4.7).

I once sat in a meeting that was led by someone wearing the Google Glass, and it was the most awkward encounter. As he was talking, he was also navigating within his glasses. As the viewer, I was able to see that he was looking elsewhere, instead of at those who he was supposed to be talking and interacting with. He was so caught up into the information within his glasses, he couldn’t concentrate on what he was telling us. My perspective was that he was too busy to focus on the meeting, and that made me question why I needed to be in the meeting if there were other things to be done. For all I know, he was using notes or reference in the Google Glass to help direct his thoughts, but the technology certainly got in the way. There was a social awkwardness to the whole environment not knowing how to react to this new product that had not been socially or culturally accepted.

It would have been helpful if the Google Glass wearer had acknowledged the disconnect by discussing the elephant in the room—this wearable technology—which would help to humanize the experience. Pointing out what he was doing and why he was wearing the glasses would have made the whole experience a bit more comfortable. Lacking that, I am not convinced that he even knew why he was wearing them. From what I could tell, his wearing of the product didn’t help or assist the meeting at all; in fact, it made it worse. Was the Google Glass intended to help in our daily tasks, such as running a meeting? That is unknown. Without knowing who the product is for, and how it will enhance their lives, the Google Glass sales went poorly, and Google halted production as a result.

In 2019, Google launched a new version of the Google Glass, geared toward businesses and the workplace. This time they released it as an official product, unlike the first release, which was classified as an experiment. They positioned this newly released product as intended for use by surgeons and factory workers specifically. With this more narrow focus, they identified the needs of these users more clearly and then enhanced the experience accordingly.

Perfection is not human

It is safe to say that humans are not perfect. That is good news because perfectionism is boring and mundane, so striving for perfection should not be the goal. Knowing this, and that we will make mistakes along the way in the design process, allows us to own up to them, learn from them, and then continue creating. You may have to continually adjust your focus and make changes as you go through the design process. As you make adjustments, you also have to accommodate any possible technology restrictions. It is good to embrace some sense of imperfection in your work to accommodate this. As we start to explore how to merge physical and digital worlds, we must bring some humanity into the digital environments and spaces that we create; this will make them more approachable and inviting.

Once I and my two young children were invited to an open house, and the host invited us to sit down on their living room couch to eat. They insisted it was okay, but as I looked at the red strawberries on my children’s plates in comparison to the brand-new white couch, white carpet, and pristine environment they had just welcomed my messy children into, I was instantly put on edge making sure that a dirty hand did not go unwiped. I was anxious the entire time I was there as a result. This is the same for the environments we are creating: If they become too perfect, they become less approachable to the user and can cause anxiety. Because the act of trying a new or unfamiliar technology can already cause a certain amount of uncertainty and anxiety, we want to seek design solutions that can soothe.

There are many ways to help simulate calming effects on the user based of what we know and understand about how meditation works to find a sense of peace. By being so immersed into an activity you can simulate the same effects as if you were mediating, because you achieve a focus or a flow to what you are doing. When the focus becomes so deep, you can actually achieve a temporary release of dopamine, which is your body’s natural anti-depressant. This response comes from finding a rhythm or a cadence to your thoughts that becomes almost musical. It is the same effect you can achieve from walking or running with a repetitive pace and is often called the runner’s high. Being able to simulate these moments of rhythm into areas that could cause an anxious reaction will create a calming effect instead.