Chapter 5
Creating the Prototype
Before you get into the full digital design process, it is important to test out your ideas to see how they will work. This can be for internal reasons, such as to quickly visualize or try different ideas, but it can be important for external reasons, as well. Perhaps you need to get approval on the concept before proceeding further into the design. To navigate the various ways to create your prototype, we will be discussing:
FAKE IT ’TIL YOU MAKE IT Overcome imposter syndrome—through creation and experimentation—to foster a more optimistic mindset that enables you to try something new or unknown.
SKETCH TRANSPARENTLY When brainstorming for augmented and mixed reality, it is important to test your ideas in changing environments and backgrounds without the mechanism interfering.
THE POWER OF PROTOTYPES When you are trying to figure out the experience and flow of your experiences, you need to create rapid tests using basic materials.
USE WHAT YOU KNOW When creating with new technology, lean on the familiar to take away some of the intimidation.
Fake it ’til you make it
It is said that if you change the way you feel or think about something it will provide the confidence you need to continue to feel and think in that new way. This approach can be useful in overcoming the lack of confidence you may have when you are learning something new. If you tell yourself over and over that you can do something, then you start to believe it and you will eventually be able do that thing. This optimistic mindset is especially important when working with something new and unfamiliar to you.
The truth is that for many who start to work in the XR field, much of it will be in this unknown territory. Here’s where you will have to fake it at first, because so much of it is new—not mention constantly changing and evolving. The only way to keep up is to dive in and create something. Experience the full process.
This positive thinking approach can also help overcome imposter syndrome, a psychological pattern of self-doubt caused by feeling that you aren’t actually qualified for the work you are doing. If that doubt takes over, it can stop a great idea before it ever has a chance to get off the ground, or worse, it could alter an entire career path.
Imposter syndrome Doubting yourself, specifically your skills and accomplishments, in fear that it will someday be discovered that you are an imposter or fraud and that you aren’t as talented as once thought.
Well, let’s think about this. If you had only ever worked in a field where you felt like you had enough experience, how would you ever have gained that experience? In fact, in XR, there are so few people who have the experience that everyone is learning together. So, if you feel like you have a lot to learn, then you are far from alone.
People who create something that is considered innovative are perceived as innovators because they created something different, in a way that was never thought of or used before. So, in essence, all great creatives could be considered imposters. However, without faking it at first, they would never have pushed through their own self-doubt to give their idea a chance. Imposter syndrome is completely normal; in fact, psychological research shows why this behavior forms this pattern of self-doubt. By changing the message, to adopt the glass half-full perspective, you can:
Tell yourself you can do it.
Go do it.
But if you are trying something new, something that you have never done before, where do you even begin? Throughout this chapter we will look at different ways to start the process. All of them involve using what you know and what you are comfortable with to help pack your process with confidence. Think back to your childhood, or even observe a child to inspire this mindset. Each and every day, they are faced with activities that are new to them. These start the day they are born when they see a smile and try to imitate it. This continues throughout childhood where they attempt to hop on one foot, ride a bike, or even make a paper airplane.
If you handed me a piece of paper and asked me to make a paper airplane, I would instantly begin to experience self-doubt. I would think about my lack of knowledge in aerodynamics and perhaps try to think back to a time when I previously made a plane to recall the first step or even an idea of what the plane was shaped like.
If you ask a child to make a paper airplane, however, they just start folding. Even if the plane doesn’t fly across the room, they are inspired to try it again. If they are able to get any flight, they see it as progress and possibly success. In truth, this should be viewed as a success; they took a step closer to completing their goal. Having a clear goal of making the paper fly and with the use of easy, familiar materials, there’s no reason why you, too, can’t at least give it a try. The best part is, you don’t have to invest a lot of time making it before you can see if works. And you can quickly and easily learn a few things that work well and are essential for the paper to actually move through the air. It is likely that the first try won’t be amazing. However, if you sat there for an hour and observed, experimented, and continued to tell yourself that you could get it to fly to a specific spot in the room—you would gain so much more than if you had just “accepted the fact” that you didn’t know how to make an airplane out of paper.
Children have an easier time jumping into these unknown territories, because they are faced with them every day (FIGURE 5.1). This is a great reminder that we all had this optimistic mindset at one point in our lives, and really everything we have ever done throughout our developmental life was a learned experience. At one point we didn’t know how to walk, but we held ourselves up and took the first step, and without that we wouldn’t be walking today. Knowing and appreciating that everyone has different level of physical abilities, it becomes even more inspiring to see how someone who isn’t physically able to walk independently finds a way to move with independence. This can be done only by trying the first time and not giving up the second time around either.
FIGURE 5.1 Dimensional Sketch. Aerial sketch of a city with an airplane flying overhead. Drawn with creative perspective by a thirteen-year-old.
Illustrator: Andrew Onsager
If the goal is to overcome imposter syndrome, we need to start creating something—and quickly. After you have created your first paper airplane, you can no longer say that you don’t know how. This is the beauty of making a prototype. It is essentially the warm-up before the actual race. There are many ways to create a prototype, but the best is to find a way that allows you to lean on something that is familiar so that you don’t become overwhelmed with the technology and can focus on your ideas.
It is important to understand that the main reason you create a prototype is to answer questions. When you are creating something new, there are a lot of things that might be unknown, and the way to learn more about them is to try. Before you set off to create a prototype of any kind, you need to have a specific question in mind that you hope to answer. In the case of the paper airplane, you can start to think of multiple questions: What is the best wing shape? What is the best length? What paper weight is the most effective? I could keep adding to the list, but having too many questions within one experiment can make it hard to identify and see the change that has the most effect. While all these questions are interesting and deserve experimentation to find the best answer, they each deserve their own focus.
Tip
Keep it simple, and pick one question you want answered at a time. Once you find an answer to that question, then you can move on to address another question.
If you change multiple variables and have no constant in your experimentation or research, then you cannot identify as clearly what caused the change. To test paper weight, you need a constant, such as the overall design and form of each plane, so the only variable is the paper weight. With a flying test, you can observe which plane flies the farthest. Because no other variables have changed, you can then identify that paper weight as an optimal weight to use as you continue testing.
Sketch transparently
Think about your experience reading this book: How much are you noticing the type itself compared to the ideas conveyed in the type? If designed well, the answer should be “not much at all.” The design of the typography should not interfere with the delivery of the message. As Beatrice Warde, a twentieth-century scholar of typography, wrote in her essay “The Crystal Goblet,”
Type well used is invisible as type, just as the perfect talking voice is the unnoticed vehicle for the transmission of words, ideas.1
1 “The Crystal Goblet or Printing Should be Invisible” from Warde, B. (1956), The crystal goblet, sixteen essays on typography. Sylvan Press.
The vessel that transfers information should be as transparent as a clear glass; the less you notice about the typography of a book, the more it aids in the process of communicating the book’s ideas. Warde’s thoughts on typography extend to other areas of graphic design, as well. You don’t want the vehicle that you are using to transmit a message get in the way of the message itself.
This concept is intriguing to explore as it relates to augmented and mixed reality. The intent is to design for a transparent interface that will allow for information to be experienced within the context of the environment that you are in (FIGURE 5.2). Taking this concept literally, what then could happen if you did, in fact, design for the transparent page on a transparent page?
FIGURE 5.2 Transparent Sketch. A prototype of an AR navigational experience sketched on a transparency sheet and tested by holding it up in the physical space.
Photographers and designers: Jiaqi Liu and Xiaoqian Dong
Rather than sketch your design on a piece of paper, try using a transparent acetate sheet to supplement the traditional sketchbook. The intent in sketching transparently is to identify the overall interface elements that will be static while simultaneously viewing the environment around you. Using a Sharpie, you can draw out the concept wireframe for the experience on the transparency and then hold it up to see how the environment and elements interact. This will be closer to the experience you are designing for than working on traditional paper. This works flat, but it’s even better if you find a glass window or door to hang the transparency on while you sketch, so you can see as if you are looking at the camera view from your phone or head-mounted display.
Alternatively, you can create your wireframes digitally and print them on transparencies to test them, if your printer can accommodate this workflow. You can go one step further and make the transparencies the size of the screen you are designing for so you can sketch using a more realistic display area. This will help determine size and hierarchy relationships at this stage in the process. (For those who are XR design novices, this allows you to lean on the familiar sketching process to make it easier to step into something new.)
Transparent sketching can be expanded, to accommodate different technologies such as a head-mounted display concept, for which you would need to consider sketching views for each eye. You can quickly and easily test how well placement works by holding up the transparency and looking through it. This should give you a good sense of what might work and quickly reveals issues to address. It is valuable to check this in such a low-fidelity platform, instead of getting five steps further into the process of designing, coding, and launching a device only to learn that the placement and the experience is not going to work. This serves as a transitional way to get your idea onto paper without the intimidation or distraction of technology.
Tip
Trim your transparent material to be the size of the media, such as the screen size for a mobile AR experience.
We think differently while holding a pen and paper than we do when working with a trackpad or mouse. Likewise, we think differently when we are sketching out an idea on a transparency versus an opaque sheet of paper. One thing that intrigues me about this approach is how backwards it is: utilizing a component of an older piece of technology to help you embrace a new one. Working this way removes the pressure of creating something for high-tech and allows you to just focus on the idea and identifying possible design solutions in a very low-tech, low-pressure format. If you are designing for a mobile device, this helps you see the interaction of your design elements in many scenes quickly, because you can hold it up and test it easily in many test locations without writing any code. The goal of this part of the process is to start thinking about and making considerations for the challenges and potential of immersive design.
Transparent sketching can be done on anything that you can see through. You could use dry erase markers or chalk markers on a window, plexiglass, unused screen protectors for a mobile device, a roll of clear vinyl, gallon-size plastic bags, or really anything that is clear and available to you (FIGURE 5.3).
FIGURE 5.3 Window Sketches. Sketch exploration for a cloud identifier AR experience drawn on a window as a rain storm is rolling in.
The power of prototypes
Grab some supplies: paper, scissors, glue, tape, sticky notes, foam core, Play-Doh, and an open mind.
You may wonder how I could be suggesting using such basic materials to start creating for such a highly technical medium, but that is what makes quick prototyping so great. Sketching is such an important part of this process. You can really focus on exploring and pushing your ideas when just working with your hands. Once you bring in a computer into the process, you start focusing on the technology of making it work instead of the idea itself. Designing for 3D on a 2D device can be difficult to wrap your head around. Really, what better way is there to work through a three-dimensional idea than in 3D? Another prototyping challenge is being able to view the environment or model in its entirety. When you brainstorm physically with whatever materials you have and work without the restraint of any frames or screens, you will find that you are able to focus on form and relationships.
Make it fast
With paper and a few other supplies, you can create something fast; this is called rapid prototyping. To get started in this process, you should first select materials based on what you are creating. By this point you should have answered the why, who, what, and how as discussed in Chapter 4, “Ideation.” If you are exploring an environmental space, then you could create a diorama to create a small model of the space you are trying to design for. If you would prefer to sketch more at scale, which can be helpful for spatial awareness and depth, then you might want to think bigger. My vision of a perfect way to sketch for XR involves wrapping a small room with paper and drawing out the full 360-degree design viewpoints.
Rapid prototyping A technique that involves creating a 3D model using any number of materials to test out an overall idea quickly.
You can even surround yourself with large sheets of paper on stands that are as wide as your intended field of view (FIGURE 5.4). This will guide you to think about the user and their direction of gaze. Where will they be looking throughout the experience? Their direction of gaze may change throughout the experience, but for the initial drawing, just select a key moment of the experience to illustrate. This could be the first interaction of the experience or the key interaction of the concept, depending on what makes the most sense for the project.
FIGURE 5.4 Field of View Sketches. Sketches exploring the visible space around the user for an augmented experience. Templates for AR/VR Sketches. www.vovakurbatov.com/articles/templates-for-ar-vr-sketches
Designer: Volodymyr Kurbatov
Field of view (FOV) The size, big or small, of the viewing space for an augmented experience.
If you will be working with a 3D model, then you can use some Play-Doh or clay to quickly explore shape and form. These can be used as separate explorations, or even better, you can combine these into your spatial sketch so you can see how the 3D form will relate to the environment. The best place to be inspired about how to create for any virtual or augmented reality is in reality itself. Be sure to immerse yourself in the type of environment you are trying to create digitally. If you are using AR, then test your sketches in the space for which you’re creating the experience. If you are creating an experience to be used within a museum, for example, then you should spend a lot of time in museums to develop ideas and test your sketches in those spaces.
The same holds true for the 3D objects you will be designing. Physically interact with the object you are trying to re-create or that you are using for reference. You should know it so well that it feels familiar to be creating it in a digital space.
Because you haven’t invested as much time into these rapid prototypes, you can really let your mind wander. You can be open to the possibility of ideas that come to you as you are working out the concept. Just as with the creation of the paper airplane, you want to focus on answering a specific question as you are working through these iterations. If you keep this in mind as you are rapid prototyping, then you can work through them and find the answer that works best. You’ll find that you can learn a lot about the experience pretty quickly, making adjustments before you invest a lot more time and resources toward a chosen direction.
The walk-through
A creative way for you to get the most out of these immersive and 3D sketches is to physically act out the experience. Role play a user going through the experience, and be sure to move your body as the user will.
If you intend for the user to be sitting the whole time, then you should test the idea while sitting.
If you want them to be walking around, then physically walk around as they would.
As you do this, pay attention to your physical experience. Do you get tired from looking up or down, for example, or from holding one or both hands out for an extended period? If you want the user to hold up their mobile device for a long period of time, gravity will start to tire their arm and hand; holding hand controllers can have a similar effect.
A role-playing walk-through is a powerful method to test all of these physical factors as you test out your idea. You can even go one step further by recording yourself going through the interactions, so you can observe the experience multiple times and from different points of view.
Going through the experience yourself will help you empathize with your users, by learning firsthand what the experience can feel like to go through it. You may not think it is hard to hold up your hand for the full experience, but the only way to really know that is to test it yourself. Later in the process you will have numbers of people test your high-fidelity prototypes, but it is essential to know how the idea holds up before you get that far into the process.
Although Chapter 6, “The UX of XR,” will delve deeper into the user experience, it is important at this early sketch-and-prototype stage to consider how accessible you are making your experience. Ask yourself, who are you excluding with your design? If the experience is intended to occur at eye level when standing, how does that change the experience for someone who will remain seated? Identify the key points that are essential to the user understanding the experience and make sure the information is delivered in a range of modes and media. For example, if you have written words for the user to read, then also offer an option for the words to be read to them.
These considerations, starting at the beginning of your ideation process, will allow for a more wholistic and complete experience for a range of needs of users. Also realize that some accessibility needs might be permanent or they might be temporary. If someone has to take off their glasses to put on a head-mounted display to demo something, for example, they may not be able to see as well without the help of their corrective lenses. In addition, if someone starts an experience standing but begins to feel dizzy, they may opt to sit down for the remainder of the experience. Ask yourself who you may be excluding with some of the design choices you make for your immersive experience. Consider how needs could vary from a permanent need to a temporary one, and even how these could change in response to the environment.
To fully empathize with some of these possible situations, do your test walk-throughs in multiple locations, standing and sitting, in bright light and low light, and with any other variations you can think of that would be important to consider for your intended audience.
Picture this
In addition to the time benefits, rapid prototyping affords the opportunity to effectively share your vision with team members. A quickly created panoramic sketch can allow others to see the vision of the experience, offer feedback, or ask questions as it relates to their role in the project (FIGURE 5.5). This works best for those who have had experience with the specific technology you are sketching for, as they can make the leap from the flat sketch to the immersive experience.
FIGURE 5.5 Yellow Boat of Hope Panoramic. Layout concept for open-ended house overlooking the water that maximizes use of 360-degree space. This design allows the audience to see devices being created in the workshop and then tested in the water, without the need for cutting from one scene to the next.
Credits for Figures 5.5 and 5.9: Director: Samantha Quick; Producer: Michaela Holland, Lauren Burmaster, Paula Cuneo, Amy Seidenwurm, Averie Timm; Animation: Chibot VR, Juan Gabriel Fernandez, DJ Baylon; Sound: Matt Hauser, Dan Teicher, Teresa Barrozo; Partners: Oculus VR for Good, The Yellow Boat of Hope; Writers: Samantha Quick, Michaela Holland, Irene Jiang; Co-creator: Michaela Holland; Collaborator: Jay Jaboneta, founder of The Yellow Boat of Hope
On the other hand, if you are pitching an idea to a client, be aware that they may not have the ability to make such a big leap visually. It might be better to share an augmented photo or video mockup (FIGURE 5.6). This is a bit less rapid. It will require more time designing and creating the experience, but it will make the difference in delivering a successful pitch to your client. You could work out the ideas in a sketch, work through different ideas, and then when you choose a direction to pitch, you could just scan that in to the computer and add it into the image to show the concept with proper context.
FIGURE 5.6 Augmented Photo. New home office prototype, showing an interior designer using augmented reality software on a digital tablet computer to select furniture for his home office studio.
Designer: Gorodenkoff
As you begin to refine this context and add the sketch into a physical space, you will want to start focusing on aligning the perspective of the models you are creating. Because this is the first time in the process you may be combing the physical and digital worlds together, even in a lo-fi way, this is a great opportunity to start paying attention to relationships in space. The best way to get started thinking in this way is to use a grid that will help set you up for success. By starting with paper that has the x-, y-, and z-axes drawn out, you can sketch freely without worrying as much about the keeping a consistent perspective. This is called axonometric drawing and is a common practice for architects as they are sketching out initial ideas (FIGURE 5.7).
FIGURE 5.7 Axonometric Drawing. Sketches exploring 3D space using axes in perspective. Templates for AR/VR Sketches. www.vovakurbatov.com/articles/templates-for-ar-vr-sketches
Designer: Volodymyr Kurbatov
Axonometric drawing Sketching using paper with axes drawn to ensure consistent uses of each axis.
There are some interesting parallels between architecture and designing for XR, as they both are designing for 3D:
Architects go through a number of sketching stages in their schematic design concept stage to avoid major revisions during the construction (FIGURE 5.8).
FIGURE 5.8 Architectural Drawing Process. Architectural design is enhanced by using computer assistance to understand different three-dimensional aspects of a building. First, a simple line drawing can identify a basic form. The computer can then create versions that show shade and shadow, a virtual clay model, an illustrated version, or a fully rendered image of the design as it is being developed. Different three-dimensional designs can then be quickly and easily compared and assessed.
Architect: Peter J. Arsenault
They seek a well-defined concept that connects the overall system of the structure with the material and relationships of space.
The structured process helps ensure a smooth project and avoid expensive false directions.
Observing and learning about the architectural design process can inform the design process you adopt in XR. Just as in building construction, in XR design it is much harder to make large changes once you have built spaces and models. Working out all these ideas and challenges at the start of the process will save time, frustration, and money. Considering these similarities, it should come as no surprise that architecture is a field that is highly acceptive of XR technology.
We already talked about a few ways that you can prototype your XR experience, but they are only a fraction of your options. To make these lo-fi prototypes, you could use:
Transparent sketches
Panoramic sketches
Augmented photos (a sketch on top of an image)
Augmented video experiences (sketches on top of a video)
3D models in paper or clay
360-degree paper sketches
Dioramas
Interactive sketches (scan and import into Adobe XD, Figma, or InVision)
Storyboards
Illustrated diagrams or models
Axonometric drawing
User flow
Up until this point we have been focusing on sketching out only single moments in time. It is smart to start the prototyping process focusing on a key interaction or part of the experience, and just sketch that. However, you will need to expand this idea further when you acknowledge that these interactions and experiences will happen over time and throughout space. The next step of the process is to explore flow and sequence. This is where the narrative of the piece really comes together.
Because storytelling is such a prominent part of the human experience, thinking of your full user experience as a narrative can help structure the overall flow. It can vary in complexity, but at the core you want to ensure that you have a beginning, a middle, some type of conflict or struggle, and an end. Often you can expect to see a much more complex story structure in a VR experience versus an AR or MR experience. But regardless of the number of twists and turns throughout the experience, the foundation remains similar. To help map the story out, you can take your initial sketches, created from the various ways we discussed already, and make them more robust by adding in a sequence (FIGURE 5.9).
FIGURE 5.9 Yellow Boat of Hope Storyboard and Styleboard. Layout concept for open-ended house overlooking the water that maximizes use of 360-degree space. Character’s movements throughout the space were first visualized as a top-down view, and then translated into storyboards.
(See Figure 5.4 for credits)
As you begin to storyboard your experience, think of the user’s point of view (POV): What makes sense for them to be able to see and interact with during the experience? In traditional film storytelling, this POV is controlled by the director, but in XR, the user gains control to determine how they observe or experience each part of the experience. Some experiences might allow the user to be able to have the ability to change their perspective, which becomes more complicated. This ability for the user to interact with the experience and change the overall story is called agency. Start to think about what freedom and control you would like the users to have to control their own story. Because this is not an option in passive storytelling mediums such as a movie or film, these decisions are opportunities to make storytelling in XR more unique.
Point of view (POV) How a scene or event is observed or experienced.
Agency The ability for a user to control or change something in an experience versus just observing. This ability to choose their own story will have an effect on the full experience.
Each time you decide to allow users an interaction point in the experience—where they can choose what to do next—you need to draw it to show all the possible journeys that a user can take. As soon as you add these variations, the story goes from linear to nonlinear, and then it becomes more complicated to keep track of the overall flow. To assist with this, you can draw this experience that is growing in complexity into one single diagram to show the flow of the interaction. This diagram is called the user flow (FIGURE 5.10). Within this chart you want to identify the start and end, which should be the goal of the experience, and draw out the scenes, user interactions, and any audio cues. To show these, you want to draw each display, decision, and action that a user will experience. This diagram can help you picture the whole story in one space so you can further investigate the story. With this visual, you can now go back to your storyboards to sequence out each possible journey that the user might take. Agency creates a more robust and active experience for the user, and it also means that you have a lot more to figure out in the paper prototyping stage, because having a clear and direct plan will make your transition onto the computer so much easier.
Interaction point The part of an experience where a user is able to directly or indirectly interact with the interface or select the next action of the experience.
FIGURE 5.10 User Flow. Initial mapping of the flow of information for the concept for an AR learning experience for children to learn about topics of science—immersively.
Designer: Paula Nelson
User flow A diagram that shows the path taken by a user from the entry point of an interaction all the way through achieving their goal. Also called a UX flow or flowchart.
As shown in Figure 5.10, each flow starts with one or more entry points, which is where the experience begins or the different places where the experience can start. Identifying the entry point is a critical first part in your planning of the sequence for two main reasons.
To consider what options the user has right from the moment that the experience starts.
To help create a three-dimensional diagram of the space in your head to visualize the spatial relationships.
Keeping the first reason in mind and knowing the user’s end goal, it might be helpful to work backwards a bit to make sure they have a direct path from that entry point that will allow for their goal to be met. Then you can go back to the beginning (the entry point) and move through it again, pinpointing any decisions or action points that come up along the way. These are the choices that the user will be paying attention to. They will be looking at what actions can be taken. To guide the user through this journey, it is important to use hierarchy and cueing to draw their attention to the next step. Successful use of cueing occurs when there is a variation created, such as a flash of light that draws the user’s attention to a button or an interactive element that signals a specific action to the user.
Cueing Using stimuli to draw user attention to a specific location where the user can perform a specific task.
As you are storyboarding, consider how you can plan out where you want the user to be looking, also called gaze. Use of anticipation, which is very effective in motion design, can signal what will come next and where the user should be looking so as not to miss it. This basic idea is that things don’t just start moving; for instance, as someone is about to walk, they have to shift all their weight to their back foot in order to lift the other foot to take a step. This subtle motion, which is seen as a movement backwards from the weight shift, creates a sense of anticipation that we observe and can then deduce that the person is about to move. This same idea of an anticipatory clue holds true inside an interface design as well as an immersive experience.
Two more important considerations to make are what you want people to pay attention to closely and what other secondary actions or elements you want them to notice. In other words, you need to decide whether you want users to pay overt or covert attention to an element of the experience. When you use a cueing technique to highlight something important, for example, the reaction you want from the user is that they turn their head in the direction of that stimuli (overt attention). Secondary actions can be taken in by the user in a less obvious way; however, they don’t have to turn their head and look right at it to be able to notice it (covert attention).
Overt attention An obvious reaction from a user in which they physically move by turning their head or eyes to look at a stimuli.
Covert attention A less obvious way of perceiving an object within a space, such as seeing it in the peripheral vision. The user doesn’t have to physically move to look at it in order to see it.
Determining this level of user gaze is something that you want to start mapping out in the storyboard level, as it will help aide in your overall flow of the experience. These techniques can also help you guide the user. If a digital object in the environment catches the user’s attention, for example, the user will likely walk toward it as it could be perceived as a clue. If you would like them to stay still and have the object come to them, then you wouldn’t want it to grab as much attention, so they didn’t have the desire to move toward it. These ideas are important to consider at the beginning, but more will be discussed as we dive deeper into designing the user experience in XR in Chapter 6, “The UX of XR.”
The second reason the first user flow entry point is so critical is that the point will be the user’s first experience with the digital world that they are experiencing. So, there are a number of things the brain will be perceiving in the background that you as the designer should be aware of—even if the user isn’t conscious of them. As soon as we enter a new environment our brains automatically create a three-dimensional diagram of the space in our heads. Without us really knowing, our brains help each of us determine the orientation of the environment and our position within it. One of the first ways this happens is that we quicky determine which way is up. Then, from there as we look around the space, each time we see a wall, or the floor, or the ceiling, we can form a more complete model of the space we are in. You may not realize this is happening, as it happens so fluidly in the relationship between your eyes and your visual cortex.
It may actually be easier to recall a time when you walked into a room and felt disoriented. Perhaps you thought you were walking into the back of the auditorium, and as you opened the door and walked in, you found yourself in the front right by the stage—or, even worse, directly on the stage. This comes as a shock because it doesn’t follow the prediction that you had made about your position and orientation in the space. Funhouses, often found in amusements parks and fairs, are designed to delight and confuse based on the whole idea of working against your brain’s expected spatial orientation (FIGURE 5.11). You often are still trying to determine which way is up, and before you can confidently determine that, it changes.
FIGURE 5.11 Hall of Mirrors. A woman stands reflected in a confusing funhouse hall of mirrors.
Photographer: JudeAnd for Shutterstock
The next time you walk into a new space, try to pay more attention to your perceptions:
Do you find yourself looking up and down, left and right?
Are you walking deeper into the space until you are able to have a better picture?
Do you start to notice who else is in that space, and even if you know or recognize any of them?
Is it after all that happens (which can be very quick) that you start to look for what actions you can take?
To aid the user in this process, you can create spatial or environmental cues in your design. Use cues to present the overall orientation of the space including lighting and sound. These use multiple senses in order to help the user process their orientation faster. In virtual reality, these spatial cues are even more important as the user will more likely feel disoriented when they perceive a different space than the one they are physically in. Some VR experiences rely on this optical illusion, but depending on your end goal, you may want to consider ways you can show the full space at the beginning of the scene to help them understand their orientation and position into this new reality. This process designs the user’s spatial cognition to help them feel more comfortable, which will empower them to continue the experience. If they feel too out of place, they will likely stop the experience as a way to regain control.
With the Oculus Quest headsets, once the user steps out of the blocked-off space for the experience, the view switches from a fully virtual view to a mixed-reality view as a way to orient the user and make sure they don’t go outside the safe bounds that the user had set up before putting the headset on. This feature also works as a safeguard for the user, reassuring them that they will have a visual cue if they move more than they are supposed to during the experience.
There are a lot of different considerations during the sketch stage of an immersive experience. No matter what technique you start off trying, just remember that the point of this stage is get answers to the questions you have. This should be an exploratory stage, and no idea should be passed up to try, especially when you can create a quick sketch or prototype to test it out. Because some of this may feel out of your comfort zone, let’s look at ways you can make the process more approachable by relying on some familiarity of your current design process.
Use what you know
Before becoming an AR designer, I spent a lot of time working in motion design. This dated back to the early Flash days, but my interest really grew when I opened Adobe After Effects for the first time and saw the potential of how this could change the way we experienced design. The exploration of how time-based design and 3D space could impact the way information was communicated ignited a new part of my creative process and storytelling potential. In addition to motion, I also have a background as a UI designer and UX researcher. This journey ultimately led me to XR design, which is really an advanced version of motion design and UI/UX, but without the confines of the frame. Realizing this allowed me to rely on many aspects of the design process that I was already using in those fields for my XR design process. There are many additional uniquenesses of this immersive experience that require some different steps to be taken along the way, as we have just been exploring, but having a launchpad that feel familiar while leaping into an unknown place is really comforting (FIGURE 5.12).
FIGURE 5.12 Hand UI Exploration. Sketches exploring UI elements using hand gestures applying the familiar actions with the less familiar immersive interface. Templates for AR/VR Sketches. www.vovakurbatov.com/articles/templates-for-ar-vr-sketches
Designer: Volodymyr Kurbatov
You can start to pull out and expand upon some of the processes that are familiar and that you already know, such as:
FROM THE UX PROCESS you can rely on skills such as user research, observation, and wireframing. You can then just adjust some of these ideas into this immersive framework, such as using the transparency sketching method mentioned earlier.
FROM THE UI PROCESS you can use a design systems and pattern approach. A goal of a UI designer is to make the design scalable, and so in this case it just extends the UI into 3D space. The interactions you will sketch need to account for depth and that the environment is the new interface; however, the basis of interface design has not changed.
FROM THE MOTION DESIGN PROCESS you can use storyboard drawings to show the sequence. Instead of just your typical storyboards, extend these to be a field-of-view storyboard taking into consideration all that would be in the user’s peripheral viewpoints as well as the central focus. This takes into consideration the fact that the user is in control of where they look and what they see within the XR environment. Because you cannot predict that in most cases, designing for all possible views allows for a seamless experience. In addition, you can also create an animatic that will map out your sound design, sequence, and flow of the experience.
Really, you will be combining all three of these areas together, along with design foundations and theories, to produce memorable and impactful immersive experiences. Instead of getting overwhelmed with what you may not know, focus instead on what you already do know. Start there.
You already have a lot of answers, which were provided while creating your rapid prototype, so you can now start making digital wireframes. Start designing them in your preferred program—Adobe Photoshop, Adobe Illustrator, Adobe XD, Figma, or Sketch, whatever is familiar to you. If that is too much too quick, then you can stick to paper and pen and start to block out where elements should go within the visual field. As these interfaces will change over time and space you can start to sequence these.
Just as in a motion design project, I take my storyboards, or in this case wireframes too, and clean them up in Photoshop (by cleaning up any smears from the background, making sure the crop is correct, increasing the contrast of the drawings for visibility, and so forth). Next, I place them into Adobe Premiere to create an animatic determining my pace and timing. In this hybrid method, you would combine all the storyboards and wireframes and bring them into a 360-degree video editor. My preference is to use the Mettle SkyBox plug-ins inside of After Effects or Premiere Pro. Another alternative would be the GoPro VR Player, which is free. The goal of this is to stitch together your overall storyboards and wireframes to show a UI simulation through a video prototype of the full immersive experience and to make the final spatial decisions before creating your design interface (FIGURE 5.13).
FIGURE 5.13 Building Simulation. RealAR helps people see property before it’s built.
Designer: Dr Daniel Swan, RealAR
If you are working in AR or MR, it can be helpful to make an augmented video to add your wireframes onto. You can capture a video of the environment where people would be interacting with the experience. Then you can bring this into After Effects (or any nonlinear video editor; remember to use what you already know), and you can export a quick prototype that will help test out your wireframes in context. This is also a great way to pitch an idea or concept, as it is an easier way to show how the physical world (shown in the video) will become the interface for the digital world shown in an experience wireframe.
Experience wireframe This is an immersive representation of the overall flow and structure of an immersive experience design. Includes a combination of storyboards or video and wireframes to help visualize both the experience and also the interface of the experience.
Depending on the XR device you are using, as mentioned in Chapter 1, “Pick Your Reality,” there may be a need for some hand-interaction or gesturing sketching that you should consider at this point. Hand interactions can vary from touch gestures on a mobile device to interacting with a controller to navigate the interface. Knowing how users will navigate through your experience will influence a lot of your design decisions moving forward. Think about the process of drawing on the computer with a mouse versus picking up a pen and drawing on a piece of paper. Those experiences are extremely different, and as a result they require different considerations to be made to the UI. We will discuss the specifics of this in Chapter 7, “The UI of XR.” For now, identify any areas in the process that will rely on the need for user input.
Process work
At the end of the process, you should have:
Your focused concept
Your ideas on paper (transparent or otherwise)
A user flow diagram
Field of view storyboards
All these assets brought into the computer and combined in a 360-degree video or as an augmented video in order to test the full experience
Truthfully, it isn’t important how you create these (as there will constantly be new programs that will be released to assist you). What is more important is the thinking that happens to get through each step of this process. If you find it easier to create this whole thing on a tablet or mobile device, then go for it.
The point is to create a low-fidelity representation of an idea and concept so that it can be more clearly explained. Your goal is to embrace the design thinking process and work through a lot of the initial questions needed to refine your ideas before you start working on a high-fidelity experience. This final output is the XR experience wireframe. Chances are when you bring it into this view, you will need to make adjustments to the design, which means you are doing it right. This is considered a living prototype that may be adjusted at least a few more times throughout the next steps as you refine your initial layout and design.