Chapter 10. Strategies in Designing and Developing 3D User Interfaces

This chapter discusses some general strategies and principles for designing 3D UIs. The design strategies we discuss in this chapter are high level and can be used in a wide variety of 3D tasks and applications. Some of these strategies are designed to match 3D UIs to the basic properties of human psychology and physiology; others are based on common sense, rules of thumb, or cultural metaphors.

At a micro-level, the devil is in the details, meaning that the effectiveness and usability of a 3D UI depends on the minute implementation details of interaction techniques, such as the careful choice of parameters and the match between the properties of interaction techniques and input/output devices.

At a macro-level, there are many high-level, general design strategies driven not by the requirements of an interaction task but derived from more general principles, such as the strengths and limitations of human psychology and physiology, common sense, rules of thumb, cultural metaphors, and so on. For example, the basic principles for designing two-handed interaction techniques were developed independently of any interaction task. Rather, they were motivated by the simple observation that people naturally use two hands in real-world activities, so using two hands in a 3D UI might improve usability or performance. Similarly, many of the principles that we discuss in this section are general enough to be applicable to a wide range of interaction tasks.

The strategies and principles discussed in this chapter can be roughly divided into two large groups: designing for humans (i.e., strategies to match the design of interaction techniques and applications to human strengths, limitations, and individual differences) and inventing 3D user interfaces (i.e., designing techniques based on common sense approaches, creative exploration of 3D UI design, rules of thumb, etc.). Both of these approaches can affect each other.

However, looking at human factors issues only as potential bottlenecks may also be limiting. For some time, researchers have been looking at the potential of the human body, rather than its limitations, as an approach to design novel interaction techniques. This design approach has often led to so-called unconventional user interfaces. Many of the techniques developed in this way did not match traditional UI directions but rather provided alternative ways of input and output to computers.

Interestingly, many of these techniques have actually found their way into mainstream interfaces, including 3D UIs. Two examples are advanced speech systems and vibrotactile cues. Such techniques are often found first in gaming systems, as their unconventional nature offers a compelling way to interact with virtual worlds (Beckhaus and Kruijff 2004; Kruijff 2007). However, they can be found in many other kinds of systems too. For example, it is hard to imagine mobile phones without speech recognition anymore.

The design approach is based on a simple principle: look at the true potential of any of the human sensory and control systems and try to match them with a new technique or a new kind of device. For a closer look at how the approach fits into a complete design and evaluation process, please refer to Kruijff (2013).

It has long been understood that our ability to self-regulate body movements, such as manipulating objects or walking through the environment, is mediated by feedback-control mechanisms (Wiener 1948). In human–machine interaction, the user controls his movements by integrating various kinds of feedback provided by external sources (e.g., the UI) and self-produced by the human body (e.g., kinesthetic feedback). When interacting with a 3D UI, the user’s physical input, such as hand movements, is captured by devices and translated into visual, auditory, and haptic feedback displayed to the user via display devices. At the same time, feedback is generated by the user’s own body; this includes kinesthetic and proprioceptive feedback, which allow the user to feel the position and motion of his limbs and body (see section 3.3.3 for more details).

Therefore, one goal of a 3D UI designer is to create an interactive system that provides sufficient levels of feedback to the user and ensures compliance (agreement) between different levels and types of feedback.

The sensory dimensions of feedback include visual, auditory, tactile, and chemical (olfactory and taste) feedback from sources external to the user’s body, along with proprioceptive and kinesthetic feedback generated by the user’s body in response to limb and body movements. The 3D UI designer has direct control of the external feedback provided to the user. Given the appropriate devices, the 3D UI can provide feedback to most sensory feedback channels, except for self-produced kinesthetic and proprioceptive cues. Providing compliant feedback to multiple sensory channels, such as combining haptic and visual feedback, improves user performance and satisfaction in 3D UIs. Proprioceptive or kinesthetic feedback can provide additional cues to the user, for example, while performing repetitive motions that may eventually lead to so-called muscle memory (Salvendy 2012). Olfactory cues may help improve recall of experiences in 3D applications (Howell et al. 2015)

From a systems point of view, feedback can be split into three categories: reactive, instrumental, and operational feedback (Smith and Smith 1987). Reactive feedback combines all the self-generated visual, auditory, haptic, and proprioceptive information that results from operating the UI. Instrumental feedback is generated by the interface controls and tools operated by the human user, such as the vibration of a pen when writing. Finally, operational feedback is feedback that the user receives from the system as the results of his or her actions. For example, when a user manipulates a 3D virtual object with a 6-DOF device, the user gets reactive feedback from moving the device with her arm (in the form of kinesthetic feedback), instrumental feedback from observing the movement of the 6-DOF device and feeling its shape, and operational feedback from observing the motion of the virtual object.

The boundaries of the categories in this classification are a bit fuzzy, but they are still important in analyzing the different techniques for providing feedback to the user and particularly in discussing the main principle in designing feedback—compliance.

Because 3D UIs may tend to involve and engage users more than other UIs, a lack of compliance between the sensory dimensions may result not only in decreased performance, but also in much more dramatic effects such as headaches, blurred vision, dizziness, disorientation, or even severe nausea (cybersickness). The most basic explanation for cybersickness is a conflict between the feedback from the visual sense and other intrinsic feedback systems, such as the vestibular and proprioceptive systems (LaViola 2000a).

A number of specific types of compliance have been discussed in the human factors and 3D UI literature, and we discuss them in this section.

Another category of spatial compliances is nulling compliance. Nulling compliance means that when the user returns the device to the initial position or orientation, the virtual objects also return to the corresponding initial position or orientation. The importance of preserving nulling compliance depends on the application. It has been shown, for example, that if the device is attached to the user’s body, the nulling compliance might be important, as it allows the user to use muscle memory to remember the initial, neutral position of the device and object (Buxton 1986)

On the system level it is also desirable that instrumental and operational feedback are spatially compliant: for example, a virtual hand should be aligned as closely as possible with the user’s real hand.

Not only absolute latency but also its variability may affect user performance. Indeed, in most complex computer graphics applications, latency is not constant. Experimental studies have found that at a 20 frames per second (fps) update rate, latency fluctuations as large as 40% do not significantly affect user performance on a manipulation task. At 17 fps, their effect becomes noticeable, and fluctuations become a significant problem at 10 fps (Watson et al. 1997).

With the rapid improvements in computer graphics hardware and software, the problem of latency is lessening. For example, an experimental AR system has an end-to-end latency of 80 microseconds (Lincoln et al. 2016). On the other hand, as rendering performance increases, the requirements for visual realism and environment complexity also increase, so latency should still be considered. Even if users can complete tasks in the presence of latency, relatively low levels of latency can still affect the overall user experience.

The simplest technique for dealing with latency is to increase the update rate by reducing the environment’s complexity, using more sophisticated culling algorithms, or rendering the scene progressively (simple rendering during interaction and high-quality rendering during breaks in interaction (Airey et al. 1990). However, culling might not be trivial to perform, especially in AR applications in which users are looking through physical objects. For a discussion of methods, refer to Kalkofen et al. (2011).

Also, note that simply increasing the update rate does not eliminate all the sources of latency. For example, tracking devices have an inherent latency that is not tied to the update rate. Predictive filtering of the user input stream (e.g., tracking data) has also been investigated and has resulted in some success in reducing latency (Liang et al. 1991; Azuma and Bishop 1994). For a more detailed discussion of methods and effects in VR systems, please refer to Allen and Welch (2005), Welch et al. (2007), and Teather et al. (2009). For specific issues in AR tracking, a good starting point is Wagner et al. (2010).

While latency affects user experience in VR systems, in AR systems it can cause even more problems. In particular, offsets (misregistrations) between augmentations and real-world objects due to latency will result in difficulties processing the spatial information. A simple example can be seen in Figure 10.1: with increasing tracking problems, overlaid information cannot be correctly registered.

There are many examples of sensory feedback substitution in VR, some of which are surprisingly effective. For example, Kruijff et al. (2016) found that self-motion perception could be increased in the absence of normal vestibular and haptic feedback by playing walking sounds, showing visual head bobbing, and providing vibrotactile cues to the user’s feet (using the setup shown in Figure 10.3).

One of the first uses of props in a 3D UI was a two-handed interface for interactive visualization of 3D neurological data (Hinckley et al. 1994). A 6-DOF magnetic tracker was embedded into in a doll’s head (Figure 10.4), and by manipulating the toy head, the user was able to quickly and reliably relate the orientation of the input device to the orientation of the volumetric brain data on the screen. This resulted in efficient and enjoyable interaction, because from the user perspective, the interaction was analogous to holding a miniature “real” head in one hand. The tactile properties of the prop allowed the user to know its orientation without looking at the device, so the focus of attention could be kept entirely on the task. Although this interface was nonimmersive, in immersive VR, passive props can be even more effective. By spatially registering tracked physical objects with virtual objects of the same shape, the designer can provide the user with inexpensive yet very realistic haptic feedback. Hoffman refers to this technique as tactile augmentation (Hoffman et al. 1998).

Using passive physical props is an extremely useful design technique for 3D UIs. Props provide inexpensive physical and tactile feedback, significantly increasing the sense of presence and ease of interaction (Hoffman et al. 1998). They establish a common perceptual frame of reference between the device and the virtual objects, which in addition to ease of use may make it easier to learn the 3D UI (Hinckley et al. 1994). The introduction of tactile augmentation allows designers to explicitly control the realism of VEs, which can be useful in such applications as the treatment of phobias (Carlin et al. 1997). Passive haptics can also be used to enhance the illusion of redirected walking (Matsumoto et al. 2016), and a single passive haptic prop can be reused multiple times to produce the perception of a much more extensive haptic world (Azmandian et al. 2016).

There are also some disadvantages of using passive haptic feedback. First is the issue of scalability: using multiple movable physical props requires multiple trackers, which might be expensive and difficult to implement. Second, few experimental studies have shown any quantitative improvement in user performance when using props (Hinckley et al. 1997; Ware and Rose 1999). However, the studies did show that users preferred physical props. Props have been pushed forward in particular by the advent of many different kinds of tangible user interfaces (Ishii 2008).

For interactive computer graphics, constraints can be defined as relations that define some sort of geometrical coherence of the virtual scene during the user’s interaction with it. Although the implementation of constraints in interactive 3D computer graphics can use the theory and algorithms developed in constraint theory, from the user-interaction point of view, the way constraints are applied is more important than the details of their implementation. The main reason that constraints are used in 3D UIs is that they can significantly simplify interaction while improving accuracy and user efficiency. Several types of constraints can be used in 3D UIs.

Physically realistic constraints are an often-used category. One of the best examples of such constraints is collision detection and avoidance. When collision detection is enabled, the user’s freedom of movement is constrained by the boundaries of the virtual objects—the user’s virtual viewpoint or virtual hand is not allowed to pass through them. Another typical constraint is gravity—objects fall to the virtual ground when the user releases them. Physical constraints should be used with caution: in some applications, such as training or games, satisfying physical constraints is important because it is a critical part of the experience. However, in other applications, introducing such constraints may make interaction more difficult and frustrating. For example, in modeling applications, it might be convenient to leave objects “hanging in the air” when the user releases them (Smith 1987). Not allowing the user to pass through virtual walls could also lead to feedback incompliance. The flexibility of 3D UIs allows us to selectively choose which physical properties of the physical environment are implemented in the virtual world, and the choice should be based on the requirements of the application.

Constraints are also used to reduce the number of DOF of the user input to make interaction simpler. For example, a virtual object can be constrained to move only on the surface of a virtual plane, which makes positioning it easier because the user has to control only 2-DOF instead of 3 DOF. Such constrained manipulation has often been used in desktop 3D UIs to allow effective manipulation of 3D models using a mouse (Bier 1990). Another example is constraining travel to the ground or terrain of the virtual scene. This allows a user to effectively manipulate the viewpoint using only 2D input devices (Igarashi et al. 1998), and it may help users with 3D input devices to maintain spatial orientation.

Dynamic alignment tools, such as snap grids, guiding lines, and guiding surfaces that are well-known in 3D modeling applications, are a more complex way to reduce the required DOF. In this approach, the position and orientation of objects are automatically modified to align them with a particular guiding object, which can be as simple as a grid in space or as complex as a 3D surface (Bier 1986, 1990). With this type of constraint, objects snap to alignment with these guides. For example, objects can snap to an equally spaced 3D grid in space in order to make the alignment of several objects easier, or an object can automatically rotate so that it lies exactly on the guiding surface when the user brings it near to that surface.

Intelligent constraints take into account the semantics of objects and attempt to constrain their interaction in order to enforce meaningful relations. For example, a virtual lamp can be constrained to stand only on horizontal surfaces such as tables, while a picture frame only hangs on vertical walls (Bukowski and Séquin 1995).

The disadvantage of all constraints is that they reduce user control over the interaction, which might not be appropriate for all applications. In many applications, therefore, it is important to allow the user to easily turn constraints on and off. However, when the user requirements are clearly understood and interaction flow carefully designed, constraints can be a very effective design tool for 3D UIs.

Therefore, two-handed or bimanual input has been an active topic of investigation in UIs since the early 1980s (Buxton and Myers 1986). The benefits of two-handed input have been demonstrated in various tasks and applications, including both 2D interfaces (Bier et al. 1993) and 3D interfaces (Sachs et al. 1991; Hinckley et al. 1994; Mapes and Moshell 1995; Zeleznik et al. 1997). In this section, we briefly describe some of the important principles that have been proposed for designing bimanual 3D UIs.

He observed that some tasks are inherently unimanual, such as throwing darts. Other tasks are bimanual symmetric, where each hand performs identical actions either synchronously, such as pulling a rope or weightlifting, or asynchronously, such as milking a cow or typing on the keyboard. A third class of bimanual actions is bimanual asymmetric tasks (sometimes called cooperative manipulation), in which the actions of both hands are different but closely coordinated to accomplish the same task (Figure 10.5). A familiar example of such an asymmetric bimanual task is writing: the nondominant hand controls the orientation of the page for more convenient and efficient writing by the dominant hand.

Guiard proposed three principles that characterize the roles of the hands in tasks involving an asymmetric division of labor (Guiard 1987):

The nondominant hand dynamically adjusts the spatial frame of reference for the actions of the dominant hand.

The dominant hand produces fine-grained precision movements, while the nondominant hand performs gross manipulation.

The manipulation is initiated by the nondominant hand.

We should note, however, that the separation of labor between hands is a fluid, dynamic process, and in complex tasks, hands can rapidly switch between symmetric and asymmetric manipulation modes.

Guiard’s principles provide an important theoretical framework for investigating and designing two-handed 3D UIs. In the rest of this section, we discuss some examples of such interfaces, classifying them according to the asymmetric and symmetric division of labor.

The basic method presented above has been explored in a number of 3D interaction systems and for a variety of tasks. A two-handed interface for an immersive workbench display (Cutler et al. 1997), for example, provides tools for two-handed object manipulation and rotation using 6-DOF input devices. The nondominant hand controls the position of a virtual object and the orientation of its rotation axis, while the dominant hand controls rotation around the axis. Zeleznik et al. (1997) propose a similar 3D position and rotation technique using two mice as input devices (Figure 10.6). With the nondominant hand, the user positions a point on a 2D ground plane, creating a vertical axis (left cursor in Figure 10.6), while the dominant hand allows the user to rotate an object around that axis (right cursor in Figure 10.6). Scaling and zooming were implemented in a similar way.

Virtual menus (see section 9.5) have often been implemented using bimanual manipulation in which the virtual menu is held in the nondominant hand while the dominant hand is used to select an item in the menu (Mapes and Moshell 1995; Cutler et al. 1997). Virtual writing techniques (Poupyrev et al. 1998) use the nondominant hand to hold a tracked tablet input device, while the user writes on the tablet with the dominant hand. Similar two-handed prop-based interfaces are quite common (Coquillart and Wesche 1999; Schmalstieg et al. 1999). The Voodoo Dolls interaction technique that we discussed in Chapter 7, "Selection and Manipulation," (section 7.7.2) is yet another example of using asymmetric bimanual control for object manipulation (Pierce et al. 1999).

A somewhat different two-handed interface has been implemented for desktop 3D viewpoint control (Balakrishnan and Kurtenbach 1999). The nondominant hand controls the 3D position of the virtual viewpoint, while the dominant hand performs application-specific tasks, such as selection, docking, and 3D painting.

A number of user studies have shown that carefully designed asymmetric bimanual interfaces have a significant performance advantage and are strongly preferred by users (Hinckley, Pausch et al. 1997). A study by Balakrishnan and Kurtenbach (1999), for example, found that in a selection task, user performance was 20% faster than a one-handed interface when the user controlled the viewpoint using the nondominant hand and used the dominant hand to perform another task. Ulinski et al. (2007) found that asymmetric bimanual selection techniques were less fatiguing and induced less mental workload that symmetric bimanual techniques.

Asynchronous symmetric two-handed manipulation can also be implemented for interaction with 3D environments. For example, the Polyshop system implemented a rope-pulling gesture for 3D travel (see Chapter 8, “Travel,” section 8.7.2): the user pulled himself through the environment by pulling on an invisible rope with both hands (Mapes and Moshell 1995).

The user’s handedness (i.e., his dominant hand) is another example: some input devices are designed only for right-handers. Asymmetric bimanual interfaces must be adaptable for both left- and right-handed users.

The advantage of using this approach is that the user already knows how to use the interface from everyday experience, so the time spent learning how to use the system can be kept to a minimum. Furthermore, the interface often can be implemented based either on the designer’s intuition and common sense or on the clearly specified technical design requirements of the application.

Advanced and special-purpose 3D interaction techniques might not be necessary in such applications. Interaction with virtual space in this type of interface might be frustrating and difficult, but if the real-world interaction is also frustrating and difficult, then this is actually a feature!

Designers may, however, need to compromise on how realistic the simulation needs to be. Because of the limitations of current technology, the simulations that we can create may be far from real-life experiences or prohibitively expensive (e.g., professional flight simulators). In addition, there is some evidence that semirealistic interaction may in some cases provide a worse user experience than interaction not based on the real world at all (Nabiyouni and Bowman 2015). The realism of simulation that the developer should aim for thus depends heavily on the requirements of the application and the capabilities of the technology.

For example, in a VE for entertainment, the goal might be to provide visitors with a first-person immersive experience in an environment that cannot be normally experienced in the real world. Exact visual realism might be less important than responsiveness and ease of learning for this application, and thus interaction can be limited to a very simple flying technique that allows the user to fully experience the environment. As another example, in a medical training simulator designed to teach medical students palpation skills in diagnosing prostate cancer, a realistic visual simulation is not required at all, so only a primitive 3D visual model is needed. A realistic and precise simulation of haptic sensation, however, is a key issue, because learning the feel of human tissue is the main goal of the system (Burdea et al. 1998). On the other hand, in a system developed for treatment of spider phobia, a simple toy spider was used to provide the patients with a passive haptic sensation of the virtual spider that they observed in the VE (Carlin et al. 1997). It was found that this haptic feedback combined with the graphical representation of the spider was realistic enough to trigger a strong reaction from the patient.

These examples demonstrate that the importance of realism is dependent on the application; 3D UIs should deliver only as much realism as is needed for the application. The effect of realism, or level of detail, on user performance, learning, engagement, and training transfer is an important topic of research but is outside the scope of this book. For further reading on this subject, see Luebke et al. (2002), McMahan et al. (2012), Bowman et al. (2012), and Kruijff et al. (2016).

The use of real-world metaphors—adapting everyday or special-purpose tools as metaphors for 3D interaction—has been a very effective technique for the design of 3D widgets and interaction techniques. For example, a virtual vehicle metaphor has been one of the most often used metaphors for 3D navigation. A virtual flashlight has been used to set viewpoint or lighting directions, and shadows have been used not only to add realism to the rendering but also to provide simple interaction techniques—users can manipulate objects in a 3D environment by dragging their shadows (Figure 10.7; Herndon et al. 1992).

As these examples show, the metaphors are only a starting point, and interaction techniques based on them should be carefully designed to match the requirements of the applications and limitations of the technology used. Not just individual tools, objects, and their features but also entire domains of human activity can inspire and guide the design of 3D UIs. Architecture and movies have been an important source of inspiration—movies such as Minority Report and Ironman or books by William Gibson (Neuromancer) or Neal Stephenson (Snow Crash) are just a few of the many examples. The objective is not simply to replicate but instead to creatively adapt the basic principles and ideas in designing 3D UIs and virtual spaces. For example, games have been heavily influenced by the movie culture, exploring transition and storytelling techniques fundamental to film. It has also been observed that both architecture and virtual worlds are based on ordering and organizing shapes and forms in space (Alexander et al. 1977); the design principles from architecture therefore can be transferred to the design of 3D UIs.

Campbell (1996), for example, attempted to design VEs using basic architectural principles (Figure 10.8), suggesting that this would allow users to rely on their familiarity with real-world architectural spaces to quickly understand and navigate through 3D virtual worlds. At the same time, designers of VEs are not limited by the fundamental physical restrictions that are encountered in traditional architecture, giving them new creative freedom in designing 3D interactive spaces. See the discussion of architectural wayfinding cues in Chapter 8, “Travel,” section 8.9.2, for more on this topic.

Another common technique for adapting elements from the real world to the design of a 3D UI is to borrow natural physical actions that we use in real life. For example, in the Osmose interactive VE, the user navigated by using breathing and balance control, a technique inspired by the scuba diving technique of buoyancy control (Davies and Harrison 1996). The user was able to float upward by breathing in, to fall by breathing out, and to change direction by altering the body’s center of balance. The intention was to create an illusion of floating rather than flying or driving in the environment. In a more recent example, the Virtual Mitten technique (Achibet et al. 2014) used a mitten metaphor to help users understand the grasping action and haptic feedback provided by a simple elastic feedback device. Since the device only provided one-DOF of feedback, a mitten (which allows grasping only between the thumb and the four other fingers together) was the perfect analogy for this action.

Numerous real-world artifacts and concepts can be used in designing 3D UIs, providing an unlimited source of ideas for the creativity of designers and developers. Because users are already familiar with real-world artifacts, it is easy for them to understand the purpose and method of using 3D interaction techniques based on them. Metaphors, however, are never complete, and an important goal is to creatively adapt and change the real-world artifacts and interactions. It is also difficult to find real-world analogies and metaphors for abstract operations. In such cases, a symbolic representation might be more appropriate.

The shortcoming with this approach is that the GUI interaction elements are introduced as a separate layer on top of the 3D world, so it introduces an additional mode of interaction: the user has to switch into the menu mode and then switch back to 3D UI mode, which also includes the switching between different disparity planes (Kruijff et al. 2010, 2003). The approach also does not scale well to other 2D tasks.

The difficulty with this approach is that there is no haptic feedback, so interacting with the 2D interface might be difficult and frustrating. To overcome this problem, a physical prop—such as a clipboard—can be tracked and registered with the 2D interface so that it appears on top of the clipboard. The user, holding the physical clipboard in one hand, can touch and interact with 2D interface elements using a finger or a pen held in the other hand, which is also tracked. This design approach is sometimes called the pen-and-tablet technique.

One of the first systems that used this approach was implemented by Angus and Sowizral (1996); it provided the immersed user (who was inspecting a virtual model of an aircraft) with a hyperlinked document that included 2D plans, drawings, and other information (Figure 10.9). A similar technique was developed for the semi-immersive workbench environment by Coquillart and Wesche (1999), where instead of using an everyday clipboard, 2D data was spatially registered with a tracked transparent plastic pad. When the user looked through the pad, he perceived the illusion that the 2D information appeared on the pad (more on transparent props below).

The pen-and-tablet technique can be extended by using a touch-sensitive tablet instead of a passive prop. With this active prop, the user can perform significantly more advanced interaction than simply pressing 2D buttons. The Virtual Notepad, for example, allows users not only to view 2D information while immersed in a VE but also to annotate it with 2D drawings (Poupyrev et al. 1998). Another example is the hybrid system developed by Bornik et al. (2006), which uses a special-purpose input device that can be used to interact on a (synced) tablet or directly with the 3D medical content on a stereo screen. Such hybrid systems may offer the best of both worlds, with the precision of a 2D interface and the intuitiveness of a 3D UI (see Figure 10.10)

You have probably noticed that we have already discussed the pen-and-tablet idea several times in the context of different tasks. That’s because it’s a rather generic approach in designing 3D UIs that also incorporates many of the strategies we have discussed before. To summarize, the pen and tablet makes use of

two-handed, asymmetric interaction

physical props (passive haptic feedback)

2D interaction, reducing the DOF of input

a surface constraint to aid input

body-referenced interaction

Schmalstieg et al. (1999), for example, developed an effective 2D gestural interface for 3D interaction by using tracked transparent passive props with a workbench display (see Chapter 9, Figure 9.20). The user looks at the 3D environment through the transparent prop (e.g., a simple transparent Plexiglas plate) and interacts with objects by drawing 2D gestures on the prop. The transparent plate acts as a physical constraint for the user input, making drawing relatively easy.

The system determines the objects that the user interacts with by casting a ray from the user’s viewpoint through the pen and transparent pad. For example, Schmalstieg et al. (1999) demonstrated how a group of 3D objects can be selected by drawing a lasso around them on the prop.

Transparent props allow the development of very generic techniques that can be used to interact with any 3D objects in the scene. Furthermore, when the position of objects can be constrained to lie on a virtual ground plane, the user can draw 2D gestures directly on the surface of the display, as long as the pen-tracking technology is provided. For example, in Ergodesk, the user interacts in 2D by drawing directly on the display surface (Figure 10.11), sketching 3D models using a three-button stylus (Forsberg et al. 1997). The system is based on the SKETCH modeling system (Zeleznik et al. 1996) that interprets lines as operations and parameters for 3D modeling commands. Creating a cube, for example, requires the user to draw three gesture lines, one for each of the principal axes, meeting at a single point.

The sketching interface is simple to learn and allows the user to easily create 3D objects by sketching them on a 2D physical surface. This is significantly easier then manipulating or sculpting 3D objects directly using all 6-DOF of input. Furthermore, the user can use another hand for performing traditional 3D input tasks, such as navigating or manipulating 3D objects.

There are many approaches that can help to develop new magical techniques. Considering human limitations and looking for solutions that help to overcome them is one possible approach. Cultural clichés and metaphors, such as a flying carpet, can also suggest interesting possibilities for designing magical 3D UIs. For example, the Voodoo Dolls technique (Chapter 7, section 7.7.2) is based on a magical metaphor that suggests that the user can affect remote objects by interacting with their miniature, toy like representations. Because such metaphors are rooted in the popular culture, users may be able to grasp interaction concepts almost immediately.

The relationships between techniques and cultural metaphors, however, can also be a source of difficulties. For example, in order to understand the metaphor of flying carpet techniques—a very popular metaphor in VEs (Butterworth et al. 1992; Pausch et al. 1996)—the user needs to know what a magic carpet is and what it can do. Although some metaphors are quite universal, others might be significantly more obscure. For example, the Virtual Tricorder metaphor (Wloka and Greenfield 1995) is based on imaginary devices from the Star Trek television series, which may not be well known by users outside of countries where this show was popular. It is not easy to find effective and compelling metaphors for magical interaction techniques; however, the right metaphor can lead to a very enjoyable and effective 3D UI.

The discussion of realism and magic in designing 3D UIs also directly relates to the aesthetics of the 3D environment. The traditional focus of interactive 3D computer graphics, strongly influenced by the film and simulation industries, was to strive for photorealistic rendering—attempting to explicitly reproduce physical reality. Although this approach is important in specific applications, such as simulation and training, photorealism may be neither necessary nor effective in many other 3D applications. In fact, modern computer graphics are still not powerful enough to reproduce reality so that it is indistinguishable from the real world, even though realism (especially in games) can already be incredibly high. Furthermore, humans are amazingly skilled at distinguishing real from fake, particularly when it comes to humans; this is why faces rendered with computer graphics often look artificial and therefore unpleasant (the “uncanny valley” effect).

In many applications, however, photorealism may not be entirely necessary: sketchy, cartoonlike rendering can be effective and compelling in communicating the state of the interaction, while at the same time being enjoyable and effective. One example of such an application is a 3D learning environment for children (Johnson et al. 1998).

Nonphotorealistic (cartoonlike) rendering can be particularly effective in drawing humans, because it can suggest similarities using a few strong visual cues while omitting many less relevant visual features. We can observe this effect in political cartoons: even when drawings are grossly distorted, they usually have striking similarities with the subjects of the cartoons, making them immediately recognizable. Simple, cartoonlike rendering of virtual humans or the avatars of other users is also effective from a system point of view, because simpler 3D models result in faster rendering. Thus, more computational power can be dedicated to other aspects of human simulation, such as realistic motion. This allows designers to create effective and enjoyable interfaces for such applications as online 3D communities, chat environments, and games. Research on the effectiveness of virtual humans has also shown that nonphotorealistic virtual characters are more acceptable and more effective in a variety of real-world settings (Bickmore et al. 2009).

Another advantage of non-photorealistic aesthetics in 3D UIs is the ability to create a mood or atmosphere in the 3D environment, as well as to suggest properties of the environment rather than render them explicitly. For example, a rough pencil-style rendering of an architectural model can inform the client that the project is still unfinished (Klein et al. 2000). The possibility to create mood and atmosphere is key in entertainment applications as well as in media art installations. One of the systems that pioneered this broader sense of aesthetics for 3D UIs was Osmose, a VE designed by Char Davies (Davies and Harrison 1996). The aesthetics of Osmose were neither abstract nor photorealistic, but somewhere in between. Implemented using multilayer transparent imagery and heavy use of particle animation, Osmose (see Figure 10.12) provided the user with the impression of being surrounded by the VE, creating a strong sense of being there (i.e., a sense of presence).

The aesthetic aspect of 3D UI design is an interesting and very important part of compelling, immersive, and easy-to-use interfaces. As the rendering power of computer graphics increases, and therefore the range of tools available to artists and designers broadens, the importance of aesthetics in 3D interaction will continue to grow in the future.

In the design of 2D UIs, a great deal of effort is spent in designing a system that is understandable, intuitive, and well organized, but very rarely do designers have to consider the physical actions users will be performing (although issues such as repetitive stress injuries have made these actions more prominent). In 3D UIs, however, users are often interacting while standing, while wearing or holding heavy or bulky devices, and while moving the whole body. Thus, issues of user comfort and safety are extremely relevant. We offer several practical guidelines for designing comfortable 3D UIs, with a particular emphasis on public installations as an example of 3D UIs for real-world use.

HWDs, trackers, and other input/output devices are typically tethered to the host computer or to an interface box (although wireless devices are becoming more common). One of the most common complaints of VE users is that wires and cables get in the way during system use. If the wires are near the floor, users can trip over them or get them wrapped around their legs when they turn. Hanging wires can get in the way during arm movements. Such situations are annoying, can interrupt the user’s interaction, and can reduce the sense of presence. Especially for public systems, it is important to find a cable-management solution, such as hanging cables from the ceiling, to minimize these problems. Care should be taken to leave enough length in the cables to allow users free physical movement.

Hanging wires from the ceiling can also reduce the weight of the equipment worn by the user. Weight is an important problem, especially in immersive VR systems; some 3D input and display devices are heavy. The weight of some HWDs is especially troublesome—even short periods of use can result in user fatigue. In mobile AR systems, the user must also carry the computing and power, so total weight may be even more important. Every effort should be made to reduce the overall weight the user must bear, and supporting comfortable poses for operation also helps considerably (Veas and Kruijff, 2010).

With HWD-based VR systems, the user cannot see the physical world and thus is prone to walk into walls, chairs, tables, or other physical objects. In surround-screen systems, the screens seem to disappear when the virtual world is projected on them, so users tend to walk into the screens, which may damage them. The most common approach to addressing these issues is to establish a safe zone around the user by making a physical barrier, such as a railing, that the user can hold on to. The safe zone should be large enough to allow sufficient physical movement but small enough to keep the user away from any potentially dangerous areas.

Virtual barriers can also be effective. These typically take the form of visuals that appear when a user is approaching the boundary of the safe area. Some systems that include a front-facing camera, allowing the system to fade in a view of the real world when the user is moving close to a dangerous area.

The interface design should limit free-space interaction in which the user’s hand is not physically supported. For example, the image-plane interaction technique (Chapter 7, section 7.5.1) requires the user to hold her hand in free space to select an object, which over a long period of time is very tiresome. Thus, interaction sequences should be designed so that free-space interaction occurs in short chunks of time with resting periods in between. The user should be allowed to rest her hands or arms without breaking the flow of interaction. Also, consider providing a resting place for a device if it is not attached to the user—this could be a stand, a hook, a tool belt, or some other method for putting a device away when it is not needed.

When a large number of users wear, hold, or touch the same devices, hygiene and health issues are important (Stanney et al. 1998). The designer might consider a routine schedule for cleaning the devices. Another approach is to use removable parts that can be disposed of after a single use, such as thin glove liners for glove-based input devices or HWD liners—light plastic disposable caps that users put on before using the HWD. Also, some HWD providers sell face cushion replacements.

Cybersickness and fatigue can be a significant problem when using immersive VE systems. Even though graphics, optics, and tracking have greatly improved, many users may still feel some symptoms of sickness or fatigue after 30 minutes to 1 hour of use. In public VR installations, the time that the user spends in the system can be explicitly limited (Davies and Harrison 1996). In other applications, the interface designer can mitigate these problems by allowing the work to be done in short blocks of time.

As we mentioned in our first tip, supporting a comfortable pose can considerably improve user comfort over time. In particular with gestural interfaces and handheld AR interfaces, holding poses or performing actions at a specific height can be very uncomfortable. Consider poses that place the hands at a relatively low, possibly angled position and close to the body (see Figure 10.13; Veas and Kruijff 2008, 2010). Some commercial gesture systems suggest that users rest during gameplay and limit overall duration to avoid any discomfort or repetitive motion injuries.

With respect to feedback and constraints, we can provide the following tips.

It is crucial to ensure the spatial and temporal correspondence between reactive, instrumental, and operational feedback dimensions. In particular, interfaces must be designed to ensure directional compliance between a user’s input and the feedback she receives from the system. Reduce latency and use multisensory feedback when appropriate, such as auditory and haptic feedback in combination with visuals.

Using constraints can greatly increase the speed and accuracy of interaction in many tasks that rely on continuous motor control, such as object manipulation and navigation. However, also consider including functionality that allows the user to switch constraints off when more freedom of input is needed.

Using props and passive haptic feedback is an easy and inexpensive way to design a more enjoyable and effective interaction. The drawback of props is that they are highly specialized—the physical shape of props cannot change—therefore, they are particularly effective in very specialized applications.

Guiard’s framework has proven to be a very useful tool for designing effective two-handed interfaces. Use these principles to determine the functions that should be assigned to each hand.

Adapting everyday or special-purpose tools as metaphors has been a very effective method for designing 3D interaction techniques. Because users are often already familiar with real-world artifacts, it is easy for them to understand the purpose and method of using 3D interaction techniques based on them.

Two-dimensional interaction techniques have many attractive properties: they have been thoroughly studied and are well established, most users are already fluent with 2D interaction, and interaction in two dimensions is often significantly easier than interaction in a 3D environment.

The real power of 3D UIs is in creating a “better” reality through the use of magical interaction techniques. Magical interaction allows us to overcome the limitations of our cognitive, perceptual, physical, and motor capabilities. It is not easy to find effective and compelling metaphors for magical interaction techniques, but if one is found, it can lead to a very enjoyable and effective 3D UI.

In many applications, photorealism may not be entirely necessary; sketchy, cartoonlike rendering can be effective and compelling in communicating the state of the interaction while at the same time being enjoyable. Nonphotorealistic cartoon rendering suggests similarities using a few strong visual cues, which results in simple and more effective rendering. Another advantage of nonphotorealistic aesthetics in 3D interfaces is the ability to create a mood or atmosphere as well as to suggest properties of the environment rather than render them explicitly.

The primary design issue we faced, however, has to do with the naturalism (or interaction fidelity) of the 3D UI in the game. And since this is one of the most prominent design tensions faced by many application developers, it’s useful to review how that tension was resolved in this particular game design.

Since VR games take place in purely imaginary environments and since there is no expectation on the part of the user that the game world should work exactly like the real world, we as designers had complete freedom to choose the level of naturalism for the various game interactions. Furthermore, the level of naturalism could also vary from one interaction to another (e.g., we chose natural body movements for the primary navigation technique but a fantastical frog tongue shooter for the primary selection technique).

We tried not to abuse this freedom, however. The game does take place in a semirealistic environment, one that players will find somewhat familiar. So if all the rules of real-world interaction and physics were changed, the game would be quite confusing. Like many games, then, we start with a real-world base (you’re in a hotel; you see familiar sights; gravity works in the regular way; you can look around and walk around just as you do in real life; you have a flashlight in your hand) and then enhance it (your other hand has a tool that you’ve never seen before, but if you accept the premise of a frog tongue shooter, it kind of works the way you might imagine; you’re carrying items in a bag, but you can also get inside the bag and see all the objects it’s holding).

This is an approach we call hypernaturalism, which has the nice property that it allows users to make use of their existing knowledge and understanding of the world but is not limited in the same ways that the real world is. Contrast this to the purely natural approach, which attempts to replicate real-world interaction precisely and usually falls short, leading to an inferior user experience. Although it’s not always possible to use the hypernatural approach (e.g., a surgical training application should try to replicate real-world interaction as perfectly as possible), it can be a powerful and freeing design strategy for many 3D UI applications.

Key Concepts

Consider carefully the level of interaction fidelity for your 3D UI. Naturalism is not always the most desirable or most effective approach.

Especially in games, fun and playful hypernatural interactions can significantly enhance the user experience.

While human factors problems may not be noticeable by end users of systems that are only occasionally used, our system was used for longer durations and exhibited a higher level of complexity. Complexity was caused by, among other things, the difficulty of processing the large amount of spatial data and the number of functions accessible to match the users’ needs. Longer durations imply a need for guaranteed comfortable use.

To address the various dimensions of human factors, we adopted an iterative design approach. Not solving the human factors issues would diminish the chance of acceptance of the system, as normal mobile devices were already used at the site to access static data in numerical format. We had to show the end users the added advantages of graphical analysis and AR.

We used two specific approaches from this chapter: two-handed interaction and designing for different user groups.

Performing interaction while holding the setup with both hands was possible but difficult, thus limiting performance. We had to reflect on what was more important: user comfort or interaction performance. Our solution was to provide the user with a comfortable option to hold the setup one-handed using a single grip supported by the shoulder strap, while the other hand could be used for interaction. Designers should perform careful evaluation in such situations, especially of user comfort, comparing different physical interaction modes. In the end, data from the user decides the issue, not the designer’s intuition. The user may be able to cope with some limitations.

The second issue was user group variance: different users with very different capabilities would interact with the system, posing different requirements. Users varied from environmental scientists to insurance firm analysts and the general public. While most environmental scientists were physically well-trained users with good spatial knowledge of the observed environment, other users were more challenged in these categories. This placed a higher requirement on ergonomic and cognitive issues in comparison to a system used solely by environmental scientists, the users that mainly used the system during the development phases. In many research projects, user variety and involvement during design is limited—often direct colleagues or students are used for feedback. This can lead to systems that do not work for the full range of actual users. In our case, at frequent intervals we interviewed end-users with different backgrounds or invited them to take part in user studies.

Key Concepts

Human-factors-driven design: designing with perceptual, cognitive, and physical ergonomics in mind has great potential when designing AR systems. While in simple systems these issues may not be evident, when complexity increases, addressing these issues can be a make-or-break item for user performance and acceptance.

User groups: always consider the full range of potential end-users and involve them frequently in all cycles of iterative system design.

Recommended Reading

For an excellent review of the basics in feedback-control mechanisms and a discussion of early psychological and human factors experiments in this area, consider the following books:

Smith, T., and K. Smith (1987). “Feedback-Control Mechanisms of Human Behavior.” In G. Salvendy (ed.), Handbook of Human Factors. G. Salvendy, 251–293. New York: John Wiley & Sons.

Norman, Donald (1988). Design of Everyday Things. New York: Basic Books.

For an in-depth discussion of some of the aspects of realism and level of detail and their effect on user performance, see the following book:

Luebke, D., M. Reddy, J. Cohen, A. Varshney, B. Watson, and R. Huebner (2002). Level of Detail for 3D Graphics. Boston, MA: Morgan Kaufmann.

Two classic texts on architectural design and philosophy can inspire you to develop new interaction techniques and approaches for planning and developing VEs:

Alexander, C., S. Ishikawa, and M. Silverstein (1977). A Pattern Language: Towns, Buildings, Construction. New York: Oxford University Press.

LYNCH, K. (1960). The Image of the City, Cambridge, MA: MIT Press.