"It is the tactile sense that demands the greatest interplay of all the senses."
This chapter concerns the patterns that have emerged—some over the past 30 years—for using touchscreens and other touch-sensitive devices. Patterns are a combination of a gesture plus a system response that can be repeated in a variety of situations across many devices (although most, if not all, of these patterns require certain sensors to be in place). Patterns capture a structure of call and response. Where possible, context has been left out or relegated to the EXAMPLES section of each pattern.
Note
See GESTURES FOR TOUCHSCREENS in this book's appendix for a description of the gestures used in these patterns.
Patterns for touchscreens can also, in many cases, be applied to free-form interactive gestures, just slightly modified. For instance, the Spin to Scroll pattern can be performed with a finger on a device, but it could also be performed in the air with a finger.
There are several ways to use patterns. The first is the simplest: just read through them and be aware that they exist. Internalize them so that they will appear organically as you design. The second way to use them is as a reference while designing: if you get stuck, a pattern may suggest a solution. The third way is to use them as Walt Disney did, to make a good idea even better, a concept he called plussing. Incorporating these patterns into a product can make the product richer and more intuitive (i.e., familiar). Patterns are, by their nature, tropes that appear throughout many products, so users will hopefully be accustomed to them by the time they reach your product.
New patterns, such as swirling a finger around an object to select it, are being created and tested by designers every day, but the patterns in this chapter and in the next exist as of this writing (summer 2008). A pattern has to show up in multiple products over a period of time before it becomes established. However, some of these patterns, such as the Ghost Fingers pattern, are emergent—many users will never have encountered them before, so communication about their use will be necessary (see Chapter 7).
With this pattern, a tap in a specific area or on a specific object (e.g., a button) triggers a feature.
Use Tap to Open/Activate whenever an object needs to be activated or a button needs to be pushed. Tapping could be used for selecting multiple items simultaneously as well. This pattern can be employed safely anywhere on any device.
Tap is the simplest manual touch gesture (aside from proximity) and is a natural replacement for clicking with a mouse.
Equivalent to (in code) OnMouseDown and OnMouseUp, this action is performed in less than one second and is comparable to a left click on a mouse in desktop operating systems. There are two types of Tap to Open/Activate: activate-on-press and activate-on-release. Most mechanical buttons are activate-on-press, but it is a better practice for touchscreens to be activate-on-release because it gives the user time to move off the button if he changes his mind, especially if tapping will perform an action that cannot be undone.
A tap, when held, can also be a press-and-hold, which can execute multiple actions in a row. An example is pressing a button that turns a page on an e-reader. Holding the button down could rapidly flip pages.
One of the most basic gestures, this pattern can be used in almost any system in any location for selecting objects on-screen.
In touchscreens, Tap to Select is simply a matter of matching the position of a touch event with the position of an object on-screen. As with Tap to Open/Activate, there are two types of Tap to Select: activate-on-press and activate-on-release. Most mechanical buttons are activate-on-press, but it is a better practice for touchscreens to be activate-on-release because it gives the user time to move off the button if he changes his mind, especially if tapping will perform an action that cannot be undone.
Tap to Select is usually the alternative to Tap to Open/Activate. Either one tap selects an item (with perhaps a double tap to open), or one tap opens/activates that item.
Using a single finger on top of an object on-screen moves that object, along with the finger, to a new location.
Use this pattern for drag-and-drop, sliders, and other controls to enable the user to move an object to another part of the screen.
Without being able to grasp and pick up an object, users still need a way to move objects, so pushing and pulling them with their finger is a natural action to accomplish this. Some controls, such as sliders, require this pattern.
Either the object has to be selected first, or this gesture has to be combined with Tap to Select so that an object that is beneath a touch event can be automatically selected and moved with the finger as it moves across the screen. Objects on-screen can be constrained to slide in only certain directions (e.g., left and right on a slider).
Figure 3-7. A concept for Comfort, an environmental system controlled by a touchscreen remote control. Sliding a thumb down the slider changes the temperature. Courtesy Valliant.
This is similar to Drag to Move Object, but in this pattern, sliding a finger on the screen in one direction scrolls the screen or a list of items in that direction. You can use this along with Slide and Hold for Continuous Scroll.
Use Slide to Scroll when content that users want to access resides outside the visible viewing area, such as text drifting off the screen, a large map, or a large data set such as search results. You can also use this for moving through simple menus.
Many screens, especially on mobile devices, have a limited area in which to view content, so scrolling is necessary. Slide to Scroll, along with Fling to Scroll, is a simple way to accomplish this scrolling. It is also used extensively on trackpads.
The system should check to make sure the user isn't performing Drag to Move Object by seeing whether the finger and an object align, and if they don't, by then moving the screen in the direction of the slide if possible. (If that is not possible, some sort of feedback, such as a bounce or sound, should trigger.) For clarity, you should not use this pattern with Two Fingers to Scroll.
Scroll bars indicating position can also be helpful, either always visible or appearing as necessary.
Utilizing some kind of feedback, such as a visual bounce, haptic buzz, or sound when the user reaches the end of a scroll, is a good practice.
The user moves a single digit (typically a thumb or index finger) in a circular motion to scroll. Usually, a clockwise gesture scrolls down/forward/right and a counterclockwise gesture scrolls up/backward/left.
Use Spin to Scroll when there is a list of items (such as menu or search results) or a set of screens (more than three) through which to rapidly scroll.
Spin to Scroll generally appears on mobile devices, where there is a lot of information to scroll through but limited screen real estate in which to display the information. It can also be done using trackpads.
Spin to Scroll allows for rapid scrolling with a motion that is more continuous and smoother than up and down or side to side (see Slide to Scroll and Two Fingers to Scroll).
A slight movement in one direction in a circle starts to scroll left/right or up/down. A full circle can scroll through a full screen of items. A speed or time-scrolling threshold can also be set so that after it is reached, the scrolling speeds up. Care should be taken, as rapid scrolling can be difficult for users to follow. For this reason, it is best if the user understands the structure of the list (e.g., that it is in alphabetical order).
It can also be difficult for precisely stopping on an item, often requiring users to go back and forth to pinpoint the item they want. Visual and even auditory feedback can be of assistance here.
Scroll bars indicating position can also be helpful, either always visible or appearing as necessary.
Spin to Scroll can be done with a trackpad or other touch mechanism, or it can be done as a free-form gesture (by twirling a finger in the air).
This pattern is similar to Drag to Move Object, except this one moves the whole screen or an object with scrolling possibilities (e.g., a menu or any list of items) and continues to scroll until the user lifts her finger, upon which the scrolling stops.
Use Slide and Hold for Continuous Scroll to scroll through many screens or large amounts of text.
Instead of constantly scrolling the screen with multiple motions (see Slide to Scroll and Two Fingers to Scroll), with this pattern, one motion suffices. (You can use this pattern alongside those other patterns, however.)
A visual cue of a place to hold can be necessary, such as a "More..." indicator at the bottom of a list.
Scroll bars indicating position can also be helpful, either always visible or appearing as necessary.
Utilizing some kind of feedback, such as a visual bounce, haptic buzz, or sound when the user reaches the end of a scroll, is a good practice.
A slight flick of the index finger in any direction moves either the screen itself or a selected object in that direction.
Use Flick to Nudge for simple drag-and-drop, to push objects away (or off-screen), or for slider movements.
If the "object" is the entire screen or a list, the related pattern is Fling to Scroll.
Since grasping and moving aren't possible with a 2D screen, flicking is a natural gesture for moving items around the screen.
The index finger lightly touches the screen and moves in a straight line in one direction. If the tip of the finger is used, it can indicate a light flick, whereas the use of a finger pad (combined with a longer duration) can indicate more of a push or pull. (Objects on-screen can react differently to both.)
One addition to this is physics to mimic slowing to a stop. Based on the speed of the flick, the movement of the object will continue after the gesture is complete, slowing to a gentle stop. The rate of the flick is translated into momentum, which is slowed and eventually stopped by simulated friction.
Similar to Flick to Nudge, a strong fling of the index finger in any direction moves the screen in that direction, if possible. This gesture is also known as kinetic or momentum scrolling.
Since grasping and moving aren't possible with a 2D screen, flinging is a natural gesture for moving items around the screen or for scrolling long lists to be traversed quickly in a very natural interaction that mimics the physical world.
The index finger lightly touches the screen and moves in a straight line in one direction. If the tip of the finger is used, it can indicate a light fling, whereas the use of a finger pad (combined with a longer duration) can indicate more of a push or pull. (Objects on-screen can react differently to both.)
Based on the speed of the fling, the movement of the object or scrolling of the screen will continue after the gesture is complete, slowing to a gentle stop. The rate of the fling is translated into momentum, which is slowed and eventually stopped by simulated friction. One variation on this is to use two finger flings, which could trigger a similar system response, such as rapid scrolling or a multipage scroll.
Scroll bars indicating position can also be helpful when this pattern is deployed, either always visible or appearing as necessary.
Utilizing some kind of feedback, such as a visual bounce, haptic buzz, or sound when the user reaches the end of a scroll, is a good practice.
Use Tap to Stop when there is an ongoing action that takes an extended time to perform (more than several seconds) and that a user might want to interrupt—especially if the ongoing action is displaying a set of screens or objects that the user might want to act on (e.g., a menu item or a list of content).
This pattern allows users to manually stop a moving object, something humans expect to be able to do with their hands.
Tap to Stop can cancel/stop an ongoing action or simply pause the action, depending on need. If it is pausing an ongoing action, a second, distinct tap should restart the action.
Figure 3-25. Pocket Player for Windows Mobile allows taps to stop scrolls. Courtesy Conduits Technologies, Inc.
Two fingers—either the thumb and index finger on a single hand, or both left and right index fingers if both hands are used—are brought closer together (Pinch to Shrink) or farther apart (Spread to Enlarge) while on top of an object, causing that object to change in size, scaling smaller if the fingers are moving closer together and larger if the fingers move farther apart. These two patterns are almost always found together.
Use these patterns to increase or decrease the size of objects (e.g., maps, web pages, documents, images) or the entire screen, effectively zooming in and out.
For small devices in particular, users may need to zoom in and zoom out to see an object at the correct level of fidelity and detail. This also effectively eliminates a need to increase font size for some objects with text.
These two patterns require a system to be able to recognize two touch events (multitouch). The patterns typically scale an object proportionally, although it doesn't necessarily have to be so. They also typically utilize a smooth motion on a scale, not set levels of shrinking. That is, the starting points of the two fingers are usually equal to 100% of the object's starting size. As the fingers move closer together or farther apart, the object scales in real time to match the new distance between the fingers at a ratio that designers will need to determine based on the sizes of the original objects and the size of the screen. For instance, if the fingers are 50% closer, the object could be 50% of its original size, or it could be another percentage altogether. There can be, however, a point beyond which a user cannot (or should not) shrink or enlarge an object (thus making it impossible to see or interact with), and that point should be predetermined.
With two fingers (the index and middle fingers, typically) on the screen together (usually but not necessarily side by side), the user is able to scroll a list of items or the entire screen up/down or left/right.
Used mainly on trackpads, this gesture allows users to manipulate the overall screen (and its accompanying scroll bar) with both fingers while still being able to control the cursor on the visible screen with a single finger (see Drag to Move Object). If there is no cursor to move, Slide to Scroll could be used instead.
This pattern allows for control of metalevel objects such as scroll bars while retaining control of individual objects via a cursor.
The touchscreen needs to be sensitive enough to detect multitouch; otherwise, the system has to compensate by looking for a wider touch area than a single finger would typically create.
Ghost fingers are a visualization of a user's fingers when the fingers are out of the user's line of sight, such as on the other side of a mobile device, inside an object, or on the other side of a wall.
Use the Ghost Fingers pattern when the user has to see her fingers to manipulate controls but is prevented from seeing them because of the hardware or environment. An example is being able to see your fingers typing on a keyboard on the back of a device.
With Ghost Fingers, you may be able to overcome the limitations of opaque surfaces and screen coverage (see Chapter 2), and place controls in more ergonomic places (such as on the back of a mobile device) or in places that make functional sense. Being able to use the back of a device for gestures allows for multitouch with all 10 fingers at once.
Ghost fingers also have the added bonus, when transparent, of not covering up the screen as physical hands do.
Ghost Fingers requires a sensor to measure where the user's fingers are, which may be an additional sensor or the main sensor. A second touchpad or additional cameras may be necessary to employ Ghost Fingers.
The visualization of the ghost fingers can take many forms, from shadow-like fingers to transparent outlines to photorealistic fingers.
Figure 3-33. LucidTouch allows users to "see through" a mobile device. Courtesy Microsoft and Mitsubishi Electronic Research Lab.
Figure 3-34. The Under the Table Interaction project explored a two-sided touch table. Users were able to use both hands to manipulate maps, play games, and do 3D modeling. Courtesy Daniel Wigdor, Darren Leigh, Clifton Forlines, Samuel Shipman, John Barnwell, Ravin Balakrishnan, and Chia Shen.
In the next chapter, we'll look at the patterns for free-form gestures—those that do not typically use an interactive surface for execution.