1. AR as a Visualization Experience

As a visualization experience, AR is applied to make a transformation possible, evidencing a “before and after” state that temporarily departs from the present moment. Experiences in this category are typically visualized in situ to give greater meaning to the transformation by placing it in context.

The future is often depicted in this experience category; however, the past can also be shown, as with AR historical recreations and cultural heritage projects like Archeoguide at Greece’s Olympia archaeological site. With Archeoguide, historical monuments are visualized to show the way a place once looked. It is also possible to depict a future state, such as with proposed or planned architectural and real-estate construction visualizations, showing what a building could look like in a particular location.

Visualization with AR is applicable to retail and shopping experiences, as well. Preview future furniture in your space when decorating your home by using the Pottery Barn AR app 3-D Room Designer on a Tango-enabled phone. Try on different makeup products and looks with Sephora’s Virtual Artist by Modiface using an in-store kiosk or your smartphone. And even preview a tattoo design on your body by using the AR app InkHunter on your smartphone. These AR visualization experiences are meaningful in that they can potentially help limit buyer’s remorse by applying a “try before you buy” scenario.

AR as a visualization experience also can be applied in conditions that might be difficult, uncomfortable, or not possible otherwise. My AR pop-up book “Who’s Afraid of Bugs?” (2011) explored the use of AR to assist with phobia exposure therapy through storytelling. Throughout the storybook, various creepy crawlies appeared over the user’s hands including an AR tarantula. A study¹ conducted by researchers at the Spanish Universitat Jaume in 2010 on treating cockroach phobia with AR demonstrated the effectiveness of using AR in exposure therapy, with a significant improvement in all participants. Study subjects went from a phobia that interfered with their daily lives to passing a test that involved walking into a room, trapping a cockroach in a container, removing its lid, and placing their hand in the container for at least a few seconds.

AR can be used to inspire and motivate change in health practices, looking to the effects our choices might have in the future, and help visualize the impact of medical procedures. Modiface has applied the same facial tracking and simulation technology used in Sephora’s Virtual Artist to help preview the potential results of dental and cosmetic surgery procedures, and in a partnership with Dai-Ichi Life, Japan’s largest insurance company, to create a “health promotion” app enabling photorealistic aging, anti-aging, and weight simulation effects on a user.

2. AR as an Annotated Experience

Additional information, namely text, or even instructional graphics like arrows and indicators, are used to annotate physical objects and spaces with AR in applications across repair, collaboration, navigation, and travel and tourism. AR as an annotated experience is about narration and walking you through an event or place, whether it’s with visual cues, or vocal guidance.

AR is being used in repair and maintenance experiences, precluding the need to consult a physical manual. Examples include replacing the toner cartridge of your printer, or repairing a car engine with the assistance of step-by-step three-dimensional (3-D) visual instructions in AR layered atop what you’re looking at in reality. WorkLink, by Scope AR, is a content creation platform for companies to transform traditional paper-based work instructions into AR annotated instructions. Coding knowledge is not required to write and publish these AR manuals with WorkLink.

AR as an annotated experience is also being used in a collaborative way. Scope AR’s Remote AR software (for use in the enterprise on smartphones, tablets, and HoloLens) connects remote experts and line workers, manufacturing engineers, or field service technicians in real time to collaborate through a task. Both users have the ability to manipulate augmented content and add annotations that “lock” onto real world objects in the technician’s field of view.

Microsoft’s Skype for HoloLens offers a hands-free remote annotation collaboration experience that is not limited to the enterprise and can be applied to home repair and maintenance, like installing a light switch, and repairing a bathroom sink. Such a service could also be beneficial in customer service and tech support experiences for consumers to assist with product maintenance and debugging.

Annotation in AR can be used for navigation, like with home improvement store Lowe’s indoor store navigation app Lowe’s Vision, powered by Tango. The app helps shoppers search for products and locate them in the store using AR. Directional prompts are overlaid onto the real world, guiding the customer to each item using the most efficient route around the store. Here AR serves as a dynamic form of way-finding (how people orient themselves in physical space and navigate from place to place).

Further, this experience category applies to travel and tourism—sites of interest and monuments are identified and labelled, such as with Google Lens, as well as museums and art galleries where additional information about artwork and artefacts is shared. Annotation in AR isn’t limited to the visual, as we saw with the Detour app (in Chapter 4), which uses audio to annotate locations. It’s another way to share stories and information with AR, beyond what is available to the naked eye.

3. AR as a Real Time Translation Experience

This AR experience category helps you to connect with and better access your surroundings and environment where there might otherwise be a potential communication barrier. This includes written and auditory language translation, as well as sign language.

Google Translate uses your smartphone to translate printed text, such as a street sign or menu, to 37 languages, which can be especially helpful when you’re traveling to a country where you aren’t fluent in the local language. Assistive devices like OrCam use AR to empower people who are visually impaired to interact with their surroundings by reading back printed text to the wearer, identifying objects, and even recognizing known faces. Another application, UNI, is a tablet case designed by California-based startup MotionSavvy for people with hearing disabilities. It uses a motion sensor and gesture detection to translate sign language into audio and then text. There’s an incredible capacity and opportunity for AR and assistive technology to make people’s lives better on a daily basis, enabling everyone.

4. AR as a Magical Experience

All AR experiences should have a touch of magic that evokes a sense of wonder. When AR feels magical it triggers our curiosity and initiates a sense of play and exploration. Pokémon Go, the location-based AR game by Niantic in which you physically travel to explore the world around you catching and training magical creatures, became an international phenomenon. AR as a magical experience isn’t limited to gaming. Educational examples in this category include Daqri’s Elements wooden chemistry blocks to inspire learning. Each block face depicts a different chemical symbol representing the elements of the periodic table; when you place two Elements blocks together, a chemical reaction is magically depicted in AR.

This category also has an element of surrealism as seen in AR books and art like Camille Scherrer’s “Souvenirs du monde des montagnes” (2009), and “Mirages and Miracles,” an art installation at France’s Scène Nationale Albi (2017), all bringing forth imaginative dream-like visions that do not otherwise exist in reality. AR as a magical experience reminds us that we need not strive to perfectly recreate reality with AR. It allows our imagination to roam and experiment creatively without the rules of the real world always applying.

5. AR as a Multisensorial Experience

Reality isn’t just visual. With products and prototypes possible like Adrian Cheok’s Digital Taste Interface, Ultrahaptics touch technology, Doppler Lab’s Here One augmented audio earbuds, and oNotes’s digital smell device, it’s important that we don’t abandon the other senses in AR. AR as a multisensorial experience can provide a deeper sense of immersion by activating the other senses.

Not everyone in the world is sighted and there is great benefit to expanding AR to the other senses for all. The notion of accessible design having a potential impact on everyone, not just a marginalized group, was touched on in Chapter 4 with Bill Buxton’s insights on how you’re actually designing for many: “If you understand and design for the needs of highly specialized users, you’ll often end up making something good for everyone.”

6. AR as a Directed or Guided Experience Using Calm Technology

This differs from AR  experience category 2, AR as an annotation experience, in that it incorporates haptics, or other subtle cues on the body to subtly note and guide, beyond textual notifications that are in your direct field of view. Experiences range from location-based (getting you to your desired destination with No Place Like Home GPS shoes) to fitness and wellbeing coaching (like Nadi X fitness tights), as we see in Chapter 8.

The intention of calm technology is to help sustain focus, appearing only when needed, minimizing distractions, so that you can remain in the present moment in whatever activity you are currently engaged in. This is an area we will see grow as technology becomes embedded in our clothing and worn on the body more and more. As we move forward with designing experiences in AR, it will be important to not overload the user cognitively, keeping the user’s attention rooted in reality.

7. AR as a Communication Experience

For AR to truly grow into the mass medium it is becoming, communication must be multi-user and two-way. This entails being co-present with the use of telepresence, collaboration, and communication across distances (as discussed in Chapter 6). Unlike AR experience category 3, this experience is not translation focused.

AR as a communication experience includes the ability to collaborate remotely on design. Canadian entertainment company, Cirque du Soleil, the largest theatrical producer in the world, has partnered with HoloLens to design stage sets and plan choreography for shows. Prior to using HoloLens, much of the time was spent building the show’s sets at Cirque’s studios in Montreal. Chantal Tremblay, the director of creation for Cirque du Soleil, says,² “Usually we have to wait until we finalize our casting and the artists come to Montreal, but now by looking at it we could even make changes.”

Using AR to communicate and collaborate across international borders has the potential to be life-changing and even life-saving, particularly in health. Proximie’s AR platform is being used in surgical applications to collaborate and share expertise with parts of the world where medical care is underresourced and surgical knowledge is underdeveloped, particularly in war and disaster zones. Proximie allows remote specialists to have an immediate and interactive experience of an operation without physically being there. Using the Proximie platform, whether on a tablet, a computer, or a mobile device, you can log on and connect with your surgical team, attending surgeon, and remote surgeon in AR. Proximie has worked with Global Smile Foundation, Facing the World, EsSalud Hospital Trujillo, Peru Cleft Program, and Al Awda Hospital, Gaza, to help surgeons change the lives of people who would not otherwise have access to surgical expertise.

AR is also a way to bring families together across distances to enjoy shared experiences, bridging the divide between here and there, to create a common “here.” Near future examples like Holoportation, 3-D capture technology that allows high-quality 3-D models of people to be transmitted anywhere, demonstrate a play experience between a father and his daughter using HoloLens. Systems like this change the way we create, store, share, and relive memories with the ability to record and play back the entire shared augmented experience. As discussed in Chapter 7, personalized photo-realistic avatars may even one day become our surrogates, remembering who we were and continuing our legacy.

8. AR as a Superhuman Experience

AR is  used to tap into an entirely new sensorial spectrum that couldn’t previously be accessed, one that is beyond our natural human capabilities. This includes X-ray vision (HoloAnatomy for HoloLens in collaboration with Case Western Reserve University), thermal vision (Thermographer by Daqri), electromagnetic fields (accessed by grinders with embeddables implanted under the skin³), and even creating new senses, as discussed in Chapter 3 with neuroscientist David Eagleman’s work on sensory substitution to feed information into the brain via unusual sensory channels.

9. AR as a Real Time Measurement Experience

AR is applied to measure the dimensions of a physical object as well as analyzing physical body responses with biometric data. Examples include the Lowe’s Vision app, powered by Tango, which uses AR to measure objects and spaces in your environment to aid with home improvement and decor. This experience category can be extended to the human body for which AR is triggered based on real time biometrics such as stress levels, using the wearer’s heart rate, sweat production, brainwave activity, and other body signals.

Microsoft’s stress sensors patent for “AR help” could  one day allow HoloLens to come to your aid without having to be asked, presenting helpful content via the head-mounted display. For example, you might become stressed by the fact you are late for a meeting, which the software could confirm by cross-checking your calendar. The patent suggests the headset would automatically display a map showing the fastest route to the location of the meeting.

Other devices like the Muse headset monitor and measure your brain activity to help promote relaxation through meditation. As devices tracking and monitoring our body’s performance begin to be linked with AR hardware and software, our AR experience can become more personalized and relevant to our current state and situation, providing benefits in both the workplace and the home.

10. AR as a Highly Personalized Experience That You Customize

Your needs and your context create your personalized reality in AR; the experience is created by you and is unique to you. In this experience category, you are the director of your augmented environment, you define it. The biometrics example in AR experience category 9, AR as a real time measurement experience is one way to do this. Other examples include Doppler Labs Here One augmented audio earbuds, with which you tune your personal audio experience of a space, such as on a plane, in a restaurant, or at a concert; and applications like Valentin Heun’s Reality Editor that empower you to connect and manipulate the functionality of physical objects just the way you want them to work. This is your reality to create.

Artists and Wonderment

One way to continue to grow AR as a new experience medium is to not limit this responsibility (and joy) to computer scientists and engineers. Artist and engineer Golan Levin points out that artists have early on prototyped many of today’s technologies. To get a jump start on the future, Levin urges looking to artists working with new technologies.

He writes,⁴ “As an occasional emissary for new-media arts, I increasingly find myself pointing out how some of today’s most commonplace and widely-appreciated technologies were initially conceived and prototyped, years ago, by new-media artists.” Levin cites the examples of Google Street View and Google Earth.

The core ideas of artist Michael Naimark’s work on Aspen Movie Map (1978–1980), which enabled users to interactively explore and navigate panoramic streets in Aspen, Colorado, live on, four decades later around the world, in Google’s widely used Street View service (launched in 2007). In 1996, Art+Com, a collective of German new-media artists and technologists, developed Terravision, a networked virtual representation of the Earth based on satellite images, aerial shots, altitude data, and architectural data where users can navigate seamlessly from overviews of the Earth to extremely detailed objects and buildings. Google Earth, which “lets you fly anywhere on Earth to view satellite imagery, maps, terrain, 3-D buildings, from galaxies in outer space to the canyons of the ocean,” was originally called Earth Viewer and was created in 2001 by Keyhole Inc., a company acquired by Google in 2004. One significant difference between Terravision and Google Earth is that Google Earth integrates user-generated map annotations, allowing users to save and share their favorite places. Writes Levin:

In some instances, we can pick out the unmistakable signature of a single person’s original artistic idea, released into the world decades ahead of its time—perhaps even dismissed, in its day, as useless or impractical—which after complex chains of influence and reinterpretation has become absorbed, generations of computers later, into the culture as an everyday product.

Levin stresses the importance of including artists as an intrinsic part of researching new technologies, as was practiced at Xerox PARC, the MIT Media Laboratory, and the Atari Research Lab, to name just a few examples. He points to how artists posed novel questions that wouldn’t have arisen otherwise. To catapult into the future, Levin believes it’s necessary to include artists to also help explore the social implications and experiential possibilities of technology. He comments on how what begins as an artistic and speculative experiment comes to materialize as an inevitable tool.

The role of the artist in the early stages of an emerging technology like AR is more important than one might think, with such artistic explorations not only leading to commonplace technologies as Levin observes, but also of great value to understanding the social and cultural advancement and impact of such emerging technologies.

I believe artists act as wonderment operators. They are magical weavers stitching together pieces of daily life into an extraordinary reality and future. As channels, conduits, and translators, they help us to see the world with new eyes. The role of an artist is to observe, empathize, and to reflect those things back to the world, presenting another way of being, feeling, hearing, and seeing. To me, this is the definition of innovation, and another reason to seek out artistic exploration in AR.

My wish for AR’s legacy is that it elevates the experience of wonder and extends our imagination in new ways to inspire positive change in the world and humanity at large. One way AR can do this is as a powerful visualization medium. Seeing realities that are not yet actualized can stir our willingness to welcome and celebrate new possibilities, in turn expanding our consciousness to better humanity and activate change to benefit many. Let’s make it our collective goal and commitment to design for the best of technology and the best of humanity.