Chapter 6. Storytelling and the Human Imagination
In my 2013 talk at the annual Augmented World Expo (AWE) in Silicon Valley, I gave a presentation using only metaphors to describe Augmented Reality (AR), without showing any actual AR. I used images from outside of the field with the intent to refresh and reframe the community’s and industry’s perspective on where we currently were at the time and where we needed (and still need) to go. One of my slides included the design of a clear kayak. A transparent kayak allows for a more immersive experience for the kayaker and a direct relationship with the kayaker’s environment. The clear kayak provides an invisible barrier that permits an entryway: the kayaker is able to access a view and experience that was not accessible before with an ordinary opaque kayak.
The clear kayak functions as a large open window, with the user and content sharing the same physical environment. The clear kayak gives the illusion of “disappearing” with the kayaker left in the direct presence of the environment. This clear kayak is significant and symbolic of the new AR that is emerging in which the user is more deeply immersed and directly engaged. AR experiences began on the computer screen, tethered to the desktop. The technology migrated to smartphones and tablets, and today is moving to eyewear and beyond. The AR screen will ultimately disappear and we will be immersed directly in the story of this hybrid reality. This sense of immersion will be further enriched by personalized experiences driven by context where the user is at the center.
We learn about the world through stories. The best stories and storytellers make you feel a sense of immersion, like you’re really there living the story. Stories also can make us an insider to see through another’s eyes. A great story is visceral; it ignites, evokes, and stirs. It sparks a reaction and a response. It transforms us. Stories take us places: to events, geographical locations, and other times.
I see AR as a form of make-believe, creating a virtual story that can be visual, audible, tangible, olfactory, and even one that you can taste. The human ability to make-believe is an extraordinary power. It can give a visual or voice to something that does not exist in reality, transforming a person, object, or place, and transporting you to a different time and space. When we make-believe, we create stories with our imagination. Make-believe is an important part of learning, play, creativity, and invention. It breeds the impossible and ignites novel ideas. Make-believe is not only for children: it is an effective approach in problem-solving and gaining a new perspective.
AR is a new communication medium that has great possibilities for extending the human condition, reimagining the way stories are told and experienced. This chapter looks at how AR is moving past novelty to create compelling storytelling experiences at a time when this medium is still new and malleable. We take a look at where we’ve been with recurring storytelling themes and conventions and where we’re headed with emerging styles and mechanisms that point to the future of AR storytelling.
Imagination and Creativity
A sentiment that has stayed with me since one of my first presentations on AR and storytelling in 2005 is this: “But what if I want to imagine it for myself, without AR?” An audience member posed this question when I shared a project called LIFEPLUS, an example of an AR historical recreation at the site of Pompeii, Italy. LIFEPLUS visually reconstructed the ruins of Pompeii, and simulated what life was like during ancient times with fictional dramatic reenactments. Visitors to the historic site were fitted with a head-mounted display and backpack (carrying computer components and a battery to power the experience) that enabled them to see and look inside augmented structures to watch fictional historic characters interact among themselves.
If you want to imagine how something looks, or once looked, using your imagination rather than seeing visuals that re-create it using technology, that should be your choice, just as you can choose between reading a book or watching a film. When you do watch a film or experience something that is technologically represented or enhanced, it doesn’t mean that you are no longer using your imagination; you continue to apply it, further extending what you are experiencing. This is what I believe AR does as well. AR will not supplant or be a substitute for the human imagination. Rather, it will extend it in new ways to further things like learning, design, and empathy, and will even place a new value on creativity. Our technology is only as good as our imagination and AR needs the human power of make-believe to dream up and build new experiences that were not possible before.
In fact, I believe a new value and importance will be placed on imagination and creativity with AR. For the most part, we have the technology available, so what are we going to do with it? We need to imagine the creative possibilities in parallel and in relation to building the technology. The best AR experiences will be the most creative. It’s important to note that we are not limited to mimicking reality, nor are we bound to the laws of the real. This medium is prime for new modes of expression. It will be these new unique expressions of creativity that continue to drive wonder and awe when the technology becomes habitual.
We’re at the very beginning of AR’s creative evolution. We can see hints of a stylistic language beginning to form, but the medium is still super malleable in these early days. Let’s call it the “wet clay” period: it’s a time to experiment and play with an incredibly wide-open scope in which there are no rules yet. Over time, genres and stylistic techniques develop when it can be more difficult, although certainly not impossible, to break free from established conventions. It’s not too often that an entire new medium comes around; this is a pioneering period of exploration and discovery that will not only help define AR, but all future mediums.
Presence
Presence is a term used in Virtual Reality (VR) to describe the perception and sensation of really “being there” in the computer-generated virtual environment, as though it were a real place. Presence is a measure of how well a virtual environment succeeds in immersing a user. Media theorists Matthew Lombard and Theresa Ditton define presence as “the illusion of nonmediation” where “the medium can appear to be invisible or transparent and function as would a large open window, with the medium user and the medium content (objects and entities) sharing the same physical environment.”1 Factors that can contribute to presence include the precision of visual alignment, synchronization with the environment, and the speed with which the environment responds to the user’s inputs and movements.
If presence in VR is the feeling of you “really being there,” presence in AR is the perception of the virtual content melding with the physical environment as though the content is “really here” in your physical space, fusing with your surroundings. Another way to think about this is in terms of “immediacy.” Media theorists Jay David Bolter and Richard Gruisin write, “The logic of immediacy dictates that the medium itself should disappear and leave us in the presence of the thing represented: sitting in the race car or standing on a mountain top.”2 Bolter and Gruisin further describe this as “the user is no longer aware of confronting a medium,” now standing instead in an “immediate relationship to the contents of that medium.”3
Standing now transforms to jumping, or even running, in believing that the augmented content is actually there rushing at you, as with the augmented bus shelter by Pepsi Max in London, England (2014) where unsuspecting commuters were startled by virtual tigers, robots, and UFOs.
A sense of presence evoking both emotional and physical reactions in human perception is not uncommon with a new technology; this was something experienced in early cinema as well. L’Arrivée d’un train en gare de La Ciotat (The Arrival of a Train at The Ciotat Station), a 50-second film directed and produced by Auguste and Louis Lumière in 1895, was said to have people jumping out of their seats and fleeing the theatre in fear of being hit by the train that appeared on screen, as though it were charging directly at the audience. Film pioneer Georges Méliès stated, “The train dashed toward us, as if about to leave the screen and land in the hall”4 and the French newspaper Le courrier du centre (July 14, 1896) reported that “spectators draw back instinctively, fearing they’ll be run over by the steel monster.”5 Cinema at that time, and like AR now, was a completely new format. With AR today, like the Lumière’s train, we too believe, if only temporarily, that virtual zombie or dinosaur is coming straight at us, except we’re not in a theatre—we are in our home, at work, or on the street.
Moving Past Novelty
Film theorist Tom Gunning discusses early cinema as a “cinema of attractions” in which “the machine which made the pictures move was the source of fascination rather than themes and stories represented.”6 In this second wave of AR, we need to move beyond a fascination with the trick of the technology and toward compelling content and meaningful experiences. When I experienced AR for the first time in 2005, I was so taken by the fact that something appeared in my physical space that wasn’t really there. It was pure magic for me and unlike anything I had seen before. The content of the demonstration itself wasn’t dazzling: it was simply a static blue three-dimensional (3-D) cube, but it was the possibilities that this virtual object and the power of the technology represented for me that had me awe-struck. Logically, I knew that the cube wasn’t actually in my physical space; yet, somehow, there it appeared. I questioned what else we could do with that cube, beyond a stage trick, and how we could use the technology to tell stories.
Presence is usually very strong the first time someone experiences AR or VR, but as the novelty wears off, the illusion presented can become habitual and presence can decrease. In Virtual Art: Illusion to Immersion (MIT Press, 2004), media theorist and art-historian Oliver Grau discusses how audiences are first overwhelmed by new and unaccustomed visual experiences, but later, after “habituation chips away at the illusion,” a new medium no longer possesses “the power to captivate.”7 Grau writes that at this stage a medium becomes “stale and the audience is hardened to its attempts at illusion;” however, he notes, that it is at this stage that “the observers are receptive to content.”8 Like cinema before it, with AR we will need to shift from a dependence on the technology alone to wow audiences, and toward cultivating captivating experiences and stories to evoke wonderment. This is a critical time to be exploring the capacity for storytelling in AR, as Grau writes, before the jeopardy of any staleness arises.
Aura and Contextualized Presence
One way to move beyond novelty and create meaningful experiences is to connect with the user contextually. “Aura” is another term that is linked to presence. Researchers Blair Macintyre, Maribeth Gandy, and Jay David Bolter define aura as the combination of an object’s or place’s “cultural and personal significance for a user or group of users.”9 They note that all aura is personal because it describes an individual’s psychological response to an object or place and that the personal nature of aura is essential because “aura can only exist if the individual can connect the object or place to his or her own understanding of the world.”10 Part of the power of context in Entryway is the combination of the real world (such as an object or place) and the personalized context that you bring to it: your memories, your stories, and your experiences.
It is my opinion that aura will come to affect presence more so in this second wave of AR because it will be more tailored to personalized experiences for the user based on her preferences and unique context. Macintyre et al. introduce the work of researchers in the BENOGO (Being There Without Going) project who are studying the uniqueness of physical locations in order to create “more compelling” VR experiences. In respect to presence in VR, they suggest that research in this area has been hindered by typical “generic” visual worlds in VR applications and that the best method of achieving presence is to place the user in a meaningful context, an approach that they call “contextualized presence.”
In a talk at The Stanford Center for Image System Engineering in May 2017, AR pioneer Ronald Azuma shared his thoughts on AR storytelling expressing his belief (one with which I fully agree) that the new compelling experiences in AR will be where you have a meaningful combination of and connection between the real and the virtual. “Because if all the power just comes from reality, then why bother with augmentation? And, if all of the power comes from just the virtual stuff, why do AR? Why not just VR?”
Azuma shared the example of Reality Fighters, a PlayStation Vita AR video game by Sony Computer Entertainment (2012). He explained how the game used reality as a backdrop without any real connection to the uniqueness of the physical surroundings. To connect reality to the game, and create a more immersive experience, he suggested enabling the fighter to use a real chair from the environment to attack the other character, or integrating a brick wall from the real world into the game to help defend from an attack.
Azuma discussed AR storytelling strategies in his talk, two of which, Reinforcing and Remembering, are particularly relevant here to aura and contextualized presence.
The strategy of Reinforcing acknowledges a real location that is inherently strong whether you augment it or not. You can augment it in an appropriate manner that attaches to the realities of that place and builds an experience that together is more impactful. Azuma gave the example of the Battle of Gettysburg, saying that if you visit the physical site of Gettysburg, and you know why that place is so crucial in American History, it’s an emotionally powerful experience just being there where those events actually occurred.
He also referenced 110 Stories, an AR experience by Brian August on the site of the World Trade Center in New York City (2011). 110 Stories is an AR smartphone app that enables you to see, in reality, where the twin towers once stood. Azuma pointed out two things he felt made 110 Stories an impactful experience. The first is that instead of a photo-realistic depiction of the World Trade Center, August rendered it as though the towers were sketched against the sky with just an outline, like with a grease pencil. “This is technically easier, but to me it is more compelling because it matches the story the experience is trying to convey, which is that the towers aren’t there anymore and they are supposed to be there,” said Azuma. The second aspect is that you could take a photo of the augmentation with the app and were invited to write a few lines about why you took the photo and what it meant to you, which then was shared on the project website. Azuma commented on how these shared stories were incredibly compelling emotionally.
I see the Reinforcing strategy as a way of honoring the reality and the “aura” (to use Macintyre, Gandy, and Bolter’s term) of a specific place in a contextualized way, and with the 110 Stories example as a way of bringing your personal story to the augmentation to further enrich the experience. The second strategy Azuma presented was Remembering, which builds on this idea.
Azuma explained Remembering as being similar to the Reinforcing strategy, except it is much more personal. “Gettysburg has a meaning that overwhelms. Everyone knows and shares the meaning of that place. But I may have a very different experience at Stanford than the rest of you, and I would have different memories of a particular location,” he said. The Remembering strategy combines memories and stories with the real location where they actually happened.
Azuma used the example of his wedding, from which he has photos and videos, but what if he could see those media embedded, at full scale, using AR at the location where the event happened? “That would be a powerful experience!” he exclaimed. It would be powerful because it was a deeply meaningful event that personally mattered to him, and he would be able to relive it at the place it originally occurred, creating a strong sense of contextualized presence.
Azuma stressed, “Not all stories have to be written by professional storytellers aimed at a mass market audience. Sometimes the most important stories to us are the personal ones, shared only with family or a close circle of friends.” I think it’s valuable for us to remember this point as we design the future of AR storytelling experiences and embrace the powerful interplay of reality and the personal stories that we each embody.
Remediation and Transcending the Old
Each medium borrows from the medium before it. Bolter and Gruisin call this remediation. AR remediates film and special effects, and the medium after AR will also remediate AR. (We don’t know what that medium is yet, and let’s not worry about it for now, but my guess, if you are curious, is an extension of AR that melds the human brain and computing to even more directly experience and feel the world with all of our senses more wholly integrated and synthesized.) The challenge of remediation is that when a medium is new it typically replicates the qualities of the prior medium. The danger of this is that the new medium focuses on the characteristics of the older medium rather than building on what is truly new.
So, how do we drive the medium forward without replicating what has come before? “Wet clay.” Lots of it. This is not only an excellent time for artists of all types to be involved, it’s a critical time and we need to begin now. A new medium’s expressive language doesn’t develop overnight; film’s long history of stylistic development is an excellent example of this. “When Edison was making his early films in the 1890s, he had most of the technology needed to make a Hollywood feature film,” said Ken Perlin, of the NYU Media Research Lab.11 “It took decades to work out over the shoulder shots, two-shots, and editing. It’s not about technology.”
Here’s a question I often ask about AR: “Does the technology drive the storytelling, or does storytelling drive the technology?” I believe AR requires both. In the first wave of AR, technology led heavily, but now in this second wave we are seeing a shift toward the importance of experience design driven by compelling storytelling. This requires building off of the unique characteristics of AR as a medium, where the technology can influence the storytelling. In this wet clay period, we also have the ability to influence how the technology develops by defining the types of stories that we want to tell and building those capabilities into the technology.
Media theorist Steven Holtzman argues that approaches of repurposing “don’t exploit the special qualities” of a medium, and that “it’s those unique qualities that will ultimately define entirely new languages of expression.”12 He describes repurposing as a “transitional step” that permits “a secure footing on unfamiliar terrain.” However, Holtzman urges that we must “transcend the old” to discover the new, for, “like a road sign, repurposing is a marker indicating that profound change is around the bend.”13 And profound change is in fact just around the bend for AR, particularly as we bear witness to the emergence of this second wave. We can describe the first wave of AR as a “transitional step” in which we saw techniques like two-dimensional (2-D) video being layered atop paper in AR, such as in augmented books and newspapers (an example of a previous media form seen within a new format). The second wave of AR begins to truly “exploit the special qualities” of the medium of AR, which is context. I see the first wave of AR as repurposing and one of novelty, very common with new mediums, and as being a necessary step to arrive at the second wave to build toward a new language of expression.
Media Specificity
In The Cinema as a Model for the Genealogy of Media, media theorists André Gaudreault and Phillipe Marion comment upon the innate and “essential” qualities of cinema: “The act of capturing and reconstituting lies at the heart of the medium’s technology. What is most essential is the innate storytelling quality the technology derives from its ability to show duration to project moving images upon a screen.”14 In my early work in AR, I believed that like cinema, AR also possessed an innate storytelling quality in its ability to represent time in a new way by uniquely superimposing or projecting images, moving or still, upon reality and the physical environment. My first prototypes and artworks in AR focused on cinematic storytelling experiences using video, an approach that was unique at a time when others focused on using 3-D models.
I still believe AR possesses an innate storytelling quality; however, I think it goes beyond the superimposition qualities of the first wave of AR. AR’s possibilities stretch far beyond projecting images, moving toward an emphasis on a real time awareness, translation, immersion, sensorial integration, and overall new understanding of our world. As we’ve seen in previous chapters, the second wave of AR is heading toward a newfound awareness and immersion utilizing multiple sensors and detection capacities. The user’s environment will be continually analyzed to provide the best contextually relevant experience (as opposed to prescribed content that is the exact same for each individual).
In the article, “Video: From Technology to Medium,” Yvonne Spielmann discusses how a medium develops “from the emergence of a novel technology and through the articulation of a specific media language” and “aesthetic vocabulary” that is specific to the medium’s capacities.15 Writing specifically about video she notes, “Once such a media-specific set of means of expression is achieved, video becomes a medium that can be distinguished from other, already existing media.” Spielmann identifies what technical characteristics video shares with other media forms like television, how it builds upon these qualities, and how video-specific images differ and deliberately depart from other media forms. These are important questions to ask about AR in a process of remediation and how AR can be distinguished from existing media forms through media specificity and toward new modes of expression.
Let’s look specifically at how AR builds on film as a medium. The first wave of AR shares with film a means of applying multimedia components such as motion graphics, audio, and recordings as a mode of representation through editing and artistic decisions to tell stories that are fact or fiction, documentary or fantastical. It builds upon film by extending these possibilities dynamically in real time to any physical surface or environment now possible to be used as a screen or venue. AR experiences differ and deliberately depart from film by presenting these stories within a new time and space model: a real time integration in situ (locally in place) and with contextual relevance atop and within reality.
This doesn’t mean that AR needs to or will exist in isolation from other mediums: it can play with other mediums, and other mediums will be and are curious about it. Take for instance the Lucasfilm Industrial Light & Magic Experience Lab (ILMxLAB) announcement in June 2016 to partner with Magic Leap. ILMxLAB was formed in 2015 to create experiences for immersive platforms like VR and AR. Together, they’ll be developing Star Wars–related content for Magic Leap’s secretive technology. Rony Abovitz, President and CEO of Magic Leap, says,16 “We’ve been testing these experiential storytelling moments and trying to make mixed reality not a novelty, but a way filmmakers and others can actually create real experiences and things that elevate and add to the universe of something like Star Wars’s properties.” John Gaeta, Executive Creative Director ILMxLAB, furthers:
At its center, it’s about the storytelling experience and where that goes. We understand that these emerging platforms are about experience. It’s about your experience, it’s about placing you in a universe and having storytelling happen around you and to you. It’s a different form that we’re going to be exploring in this lab.
Another way to think about AR is as a combination of prior media formats. The internet is a communication medium that is a collection of other media (audio, video, printed communication), but it still has its own stylistic language and formal qualities. Writing for the web, for instance, is typically shorter than traditional print, although a newspaper article can be remediated online and still is. AR is like the internet in the sense that it is merging prior media, bringing them together in a new stylistic way. HoloLens and teaser demonstration videos from Magic Leap have used flat planes in AR to emulate screens for email and even movies. This is an example of AR borrowing from previous mediums. It’s a way of bringing in familiar and known ways of using prior media, but what we will likely learn is that these old media formats need to be rethought within the container of the new medium of AR, particularly from a user experience perspective. The old rules not only no longer apply in the new format, they don’t really work. And because this is all brand new, we get to define these new experiences.
Storytelling Conventions: Where We’ve Been
So, what conventions and styles are beginning to take shape in AR storytelling and where is it headed? Over the past 12 years I’ve observed the approaches described in the subsections that follow repeated in AR. It is worth noting first that a new visual language, or “what it looks like,” is only one part of AR. As discussed in previous chapters, AR is not limited to the visual, and we will also see styles and sensorial treatments develop that are specific to smell, taste, touch, and audio, and a combination of multiple senses. For our purposes here, as the majority of AR experiences to date have been visual, this is where we will focus.
1. Virtual Try-On
Virtual Try-On makes it possible for you to become a part of the story. Whether it’s donning a virtual mask, face paint, an item of clothing, or a prop, this convention recalls make-believe and dress up play, enabling you to be transformed into someone or something else. In 2013, Disney teamed up with Apache in the United Kingdom to create an AR experience in which you became Iron Man by virtually trying on Tony Stark’s Iron Man 3 suit. Kinect was used to measure your body proportions for a perfectly fitted Mk XLII suit. You then watched yourself transformed into Iron Man on the screen in front of you and were able to test out the unique features of the virtual suit.
Virtual Try-On isn’t limited to character costumes: it’s being used in retail experiences to try on products like clothing, eyeglasses, jewelry, and makeup. It points to a contextual storytelling mechanism that is driven by and focused on you: you become the subject and the star. Presence is typically strong with Virtual Try-On because it is personal—your body becomes part of the experience with the augmented content mapped to you and moving along with you.
Snapchat’s “Lenses” is a playful experience using the front-facing camera on your smartphone to map augmented imagery to your face in real time with short animations. The lenses, which are continually updated, range from transforming you into a mouse nibbling on a block of cheese, wearing a crown of fluttering butterflies, and even swapping faces with a friend. But the potential for AR in Virtual Try-On goes beyond becoming a preexisting character: We can apply AR as another way to personally express ourselves, the way we do with fashion. We can use this storytelling convention to highlight our personal creativity and imagination by the way we choose to style and express ourselves to others, blurring the lines between our virtual avatars and our physical being. In 2017, Facebook announced its AR platform, AR Studio, enabling artists and developers to create AR masks, which could be a step toward exploring such self-expression.
2. Hole in the Wall, Floor, or Table
This storytelling style helps to transition the virtual content entering your physical space by using a visual illusion. Common in AR gaming, it’s a visual technique that merges the virtual story elements with your physical surroundings to enhance presence and further immerse you in the story unfolding around you. This convention remediates traditional trompe l’oeil (to “trick the eye”) paintings like Pere Borrel Del Caso’s “Escaping Criticism,” 1874, in which a young boy appears to be stepping out of a framed painting, breaking the perceived boundaries of reality, and now entering your space.
We’ve seen this visual trompe l’oeil effect in AR with such examples as Kweekies (2009) by Int13 and RoboRaid for the HoloLens (2016). Kweekies was a mobile AR table top experience that used a printed AR marker to trigger a launch pad from which virtual characters emerged and into which they returned. Wearing AR eyewear, RoboRaid is a shooting game on the HoloLens in which you defend your home from an invasion of robots breaking through the walls around you with cracking and crumbling sounds using spatial audio to help further the effect and illusion.
We can think of most, if not all, of these AR storytelling styles as a form of “special effects,” this one particularly so. Hole in the Wall serves as a bridge to greater believability and heightened presence, but this trope can become old because it’s somewhat of a gimmick. There’s potential here to extend the concept of breaking through in other ways with examples like inFORM, a Dynamic Shape Display by MIT Media Lab researchers Daniel Leithinger, Sean Follmer, Alex Olwal, Akimitsu Hogge, and Hiroshi Ishii (2013).
inFORM renders 3-D content physically so users can interact with digital information in a tangible way. In one of the demonstrations, a live human appears on a screen with his hands appearing to extend out of the screen (represented in real time by moving tangible blocks) and moving a physical ball around on the table’s surface. The researchers state, “Remote participants in a video conference can be displayed physically, allowing for a strong sense of presence and the ability to interact physically at a distance.” So not only can virtual characters break through physical boundaries to appear in your space, in this example we see how they also can begin interacting with and even altering your physical space in real time beyond merely a visual illusion.
3. Ghosts
Unseen to the human eye, disappearing and reappearing, and defying the rules of physics (floating and walking through objects), Ghosts are a storytelling element in AR that work thematically with the technology. In the early, glitchy days of the technology, Ghosts (like in the game Ghostbusters Paranormal Blast by XMG Studio in Toronto, Canada) worked particularly well because the tracking and computer vision didn’t need to be precise or fully accurate: this didn’t necessarily break the illusion of the AR experience, as we expect ghosts to be unpredictable with the typical rules of reality not being applicable. The aspect of using AR technology to “see” and reveal ghosts also works well, with AR providing a window into another dimension, a world otherwise inaccessible to the naked eye.
Ghosts point to the opportunities for the fantastical and surreal in AR. We’re not limited to re-creating reality; why not use AR to present something that is physically impossible to experience in reality?
4. Living Pictures
Living Pictures presents a magical reality with AR, like the portraits in the Harry Potter story series in which the people portrayed in the paintings move and interact within the painted scene and the outside world. With the Living Pictures storytelling convention, the inanimate now becomes animated, springing to life. It breaks the barrier of something that is supposed to be static or frozen in time and creates a window into another world with content inside the frame coming alive.
We see this style presented in AR-enhanced magazines. For instance, a still photo on a magazine cover transforms into an AR video of a behind-the-scenes shoot or an interview with the person in the photograph. One of the early AR-enabled magazines was by Esquire in 2009 featuring actor Robert Downey, Jr. on the cover. The experience required downloading an AR plug-in to your computer and pointing your webcam at the magazine to view the AR content on your computer screen. In AR mode, Downey, Jr. and advertisements throughout the magazine came to life with videos and animations.
A digital trend has emerged in the past few years toward depicting motion in otherwise still photos, not just using AR. This shift in digital photo manipulation blurs the lines between animation, video, and photography. Examples include cinemagraphs, created with software like Flixel, for which isolated elements of a still photo come to life, like a model’s hair blowing in the wind, and Apple’s Live Photos, introduced on the iPhone 6S in 2015, in which a few seconds are recorded before and after a still photo, allowing you to replay a short moment in time.
Each of these examples points to an expansion of time to tell a story. They become short vignettes, somewhere in between a video clip and a flip-book animation. The Living Pictures convention represents an opportunity to share more details and information behind a story if the user desires to learn more, whether it’s in advertising, art, or education.
5. X-Ray Vision
Part superhero power and part gag, this storytelling convention is reminiscent of the X-Ray Spectacles advertised in the back of comic books in the 1960s and 70s: “See the bones in your hand, see through clothes!” X-Ray Vision has been used in AR from disrobing models in advertising, like the Moosejaw X-Ray app (2012) that allowed catalog browsers to see outerwear models in their underwear, to examples in education teaching human anatomy, such as Daqri’s Anatomy 4D (2015) and the HoloLens collaboration with Case Western Reserve University (2016) to expand medical education.
This storytelling convention also can be applied to museums and delicate artifacts. The exhibit “Ancient Lives, New Discoveries” at the British Museum in London (2014) featured interactive video monitors (not AR technology) with which you could peel away the layers of mummified Egyptians. Using a CT scanner, the museum developed 3-D imagery that provided a detailed look at what was hidden inside the ornately decorated sarcophagi. This experience could be enhanced by using AR technology to look directly at the sarcophagi using AR glasses, a smartphone, or tablet (rather than at a screen above the artifact) and strip away the various layers to explore the 3-D models.
Whether it’s seeing through the layers of a fragile artifact, or through someone’s skin and clothes, X-Ray Vision in AR is symbolic of the desire to surpass our natural human capacities. It is an example of AR extending the imagination to make the unseen now seen. Beyond a marketing gimmick, this convention is a powerful storytelling mechanism for education that enables a user to unveil and safely learn about a subject by going beyond the surface.
6. 3-D Drawing
This storytelling style is about play and authoring your own reality in a familiar way: drawing and coloring with your hands. Scrawl is an experimental AR drawing app for iPhone created by String (2010). The app allows anyone with an iPhone to create something quickly in AR without any coding experience—you simply draw with your finger. Choosing from a palette of different colors and brushes in the app, you use your finger to draw in 3-D on the iPhone screen. You then can move around your 3-D AR drawing to view it from different angles. Scrawl provides an accessible creative play experience in AR using one of the best tools we possibly have: our fingers. (In Chapter 4, we looked at Konstrukt, a sound-reactive AR experience for the iPhone by James Alliban, powered by String’s AR technology. Instead of drawing with your finger, you can create a virtual sculpture by speaking, whistling, or blowing into the device’s microphone.)
Quiver (formerly ColAR) is a 3-D AR coloring app developed at the Human Interface Technology Laboratory New Zealand (HIT Lab NZ) and commercialized by Puteko Limited in 2011. Quiver works by first printing off a coloring page from the app or website. You then use any physical tool of your choice: markers, paint, pencils, or crayons. When your coloring is complete, you use your phone or tablet to view the 2-D page come to life in 3-D using AR. The character or object is animated just the way you colored it, popping off the page, and is viewable from different angles.
Scrawl and Quiver demonstrate ways to make and be expressive in AR. This is an area that will continue to develop and is an important one enabling makers of all types, not just computer programmers, to create in AR. Both apps use familiar ways of interacting with the world: drawing and coloring using our hands, with AR used to enhance the experience and further creativity and imagination.
Storytelling Conventions: What’s Emerging
Now let’s take a peek at a few of the other storytelling approaches that are emerging in AR.
1. Abstract and Artistic AR Filters
Visual filters, and the ease of applying them with such photo-editing tools as Instagram, which artistically change our reality, have become commonplace. This is something we see applied in AR, as well. AR stories aren’t limited to accurately representing reality visually: Abstract and Artistic AR Filters provide a way to creatively express how we see the world in real time.
I first encountered the use of abstract AR Filters in 2005 at SIGGRAPH in Boston (an annual computer graphics conference bringing together scientists, engineers, and artists), with a demo from the HIT Lab NZ. The demonstration used a variety of painterly filters atop reality with real time rendering styles ranging from a tile effect, to color temperature, to Gaussian blur applied to the entire scene of both the virtual objects and the physical environment. The filters made it difficult to distinguish virtual elements from physical ones, all meshing into one world.
The goal of the demo was to increase immersion and present new expressive modes in AR. It is important to note that this 2005 demonstration used fiducial markers (black and white glyphs printed on paper) to trigger the AR objects. Using this form of tracking, you could typically see the white edges of the fiducial marker throughout the augmented experience, even with a virtual object presented atop it. By applying these real time painterly filters to the entire scene, the demo helped to obscure the fiducials and create continuity between the virtual and physical, further immersing the user. Fiducials are no longer necessary today (ordinary physical objects, images, and locations can be tracked, instead), freeing up this approach of Abstract and Artistic AR Filters to focus on creative modes of expression.
Spectacle is a contemporary app (2016) created by Cubicle Ninjas that offers 50 different filters to augment a user’s physical surroundings using the Samsung Gear VR headset’s passthrough camera. It also allows you to capture photos. Unlike the HIT Lab NZ demonstration, there is no virtual content added to the scene, just a real time filter effect atop reality, but it’s still a hint of what’s to come with future AR experiences.
Two other non-AR apps that apply abstract filters, or “style transfer,” are Prisma AI and Artisto. Prisma is a photo-editing app with which you can select a photo from your smartphone or tablet’s image library, or take an image with the camera, and then apply one of 33 different filters inspired by art masters like Picasso, Monet, Van Gogh, and Kandinsky. According to Prisma’s CEO and cofounder Alexey Moiseenkov, Prisma uses three neural networks that each perform a different task, from analyzing your image to extracting an art style to applying it to the image. Moiseenkov states,17 “We’re not just overlaying like an Instagram filter. We create the photo from scratch. So, there is no photo: we took your photo, and then perform some operations and give a new photo to you.” Artisto is a similar app with artistic filters applied to short video clips instead of still photos. We also see style transfer applied in Apple’s Clips app and as a feature in Facebook’s Camera app.
This type of machine learning could be combined with AR to transform the real world into artworks, coloring our reality to show us our world in new ways in real time. In the 1971 musical film Willy Wonka and the Chocolate Factory, when the families first walked into the fantastical reality of the chocolate factory, they couldn’t believe their eyes. “Come with me and you’ll be in a world of pure imagination,” sang Willy Wonka. The Abstract and Artistic AR Filters convention offers a doorway into a fantastical dream world in which stories can be freed from the limitations of reality.
3. Objects Telling Stories
What if an object could tell you its story? Blippar’s AR visual discovery browser app is a combination of computer vision, machine learning, and artificial intelligence to help you discover more about the world around you. By pointing the camera on your smartphone at everyday objects like products, food, flowers, and even pets, the app identifies what you are looking at and presents ways to further engage with the subject. This includes related articles, videos, and nearby places of interest, all appearing on your smartphone screen. For instance, while pointing your phone at a vegetable you might not be familiar with, the app identifies the name of the vegetable, offers up recipes, and even lists locations near you where you can buy that vegetable. Blippar presents a novel way to seek out information about our physical reality by using images and objects instead of words.
Ambarish Mitra, Blippar cofounder and CEO, refers to Blippar as “the next evolution beyond search.” He goes on to say:19
This is a fun way to learn about your surroundings and even find excuses to Blipp ordinary things and discover extraordinary facts and stories.
Blippar’s visual browser creates an immersive experience by being able to learn more about what you are looking at without having to open up a web browser and type in a description of the subject you’re seeking.
As this technology improves and becomes available on AR glasses, the experience will become even more immersive and immediate: we won’t even need to point our phone at an object, we will just look and ask “What is this?” Objects might even come to tell you their stories based on your preferences and context, asking you, “Would you like to learn more?”
Beyond retail and products, Blippar’s visual discovery browser has great potential for education and the way we search and retrieve information while engaging the world around us. The Blippar blog reads,20 “We think of this like putting the world’s most intelligent eye into the pockets of kids anywhere in the world, and connecting it to the world’s brainiest personal tutor.” Colum Elliott-Kelly, head of Blippar Education, explains how as a “tutor” Blippar knows everything “in the sense of the internet” to help explain the real world. Elliott-Kelly is quick to point out, however, that a teacher is central to successful education. “We believe in complementing that individual, where they are present, and enabling them to do what educators do best, which is to enable learners to learn best. A teacher armed with Blippar can focus on adding value in areas where only a human teacher can,” he says.
Elliott-Kelly identifies three categories for uses of Blippar’s visual discovery browser within education: unlocking the discovery phase of learning in the classroom, or in school-led activities outside the classroom; bringing the real world to life as a learning portal; and combating illiteracy and a lack of access to educational institutions.
The first category—the discovery phase of learning—incorporates use cases in which students “Blipp” (use the Blippar app to interact with the world) objects around the classroom or are asked to Blipp things outside school as part of their learning. “The ability to recognize any object combined with educator systems and platforms to control the experience means that learning can be fully ‘flipped,’” says Elliott-Kelly. “Students Blipp, for example, a plug socket to learn about electronics or a local bridge to visualize engineering sciences, with content selected and sometimes created by educators, and behavior and performance monitored.”
In the second category, for which the real world is a learning portal, learners not under the formal guidance of an educator have the same ability to recognize anything through Blippar’s visual discovery, initiated by the user’s curiosity. “Museums and works of art are great examples of this, but there is a vast amount of informal learning to be had about pretty much anything in the world. We are thinking a lot about animate things, food, art, workplaces, and landmarks in this regard,” says Elliott-Kelly. He also notes an interest at Blippar in making learning an experience you contribute to and not only consume:
Your perspective on a work of art is as interesting and valuable to me, potentially, as that of a written text or other traditional source. So, when you Blipp a work of art and yourself consume content, we would love for you to be able to contribute content that I can then access when “I” Blipp.
The third category intends to combat illiteracy and a lack of access where learners might not have great guides in their classrooms or homes, or are unable to read. Elliott-Kelly says:
Consuming credible content about biology, just to use one example, requires literacy and an experienced educator of some kind. We want to deliver digital content which requires neither of these things right off the object itself. It’s the same idea, recognition by visual discovery and content curation by smart AI driven engines, but applied to jump the hurdles faced by learners without basic skills and basic access.
The notion of contributing in addition to consuming content is an important factor as conventions and tools for AR storytelling and learning continue to develop. This supports a read–write culture where we are active participants in sharing our stories of the world and with the world. It helps to make knowledge and learning human-powered. TechCrunch referred21 to Blippar as “building a Wikipedia of the physical world” with its visual browser, and if this “contribute and consume” model is applied, it really could become an augmented Wikipedia for humanity.
4. Action Galleries and 3-D AR Stickers
Choosing from a gallery of 3-D animated AR content, Actiongram for HoloLens allows you to manipulate and play with virtual items in your surroundings, which you can create a video of and share. Gallery items include people (like famous actor George Takei), animals (like a unicorn), other objects (like a UFO), and customizable text. “We give you a huge collection of holographic characters and props and tools and you can use that to put holograms into the story that you want to tell,” says22 Dana Zimmerman, Microsoft Studios executive producer.
You become the director with Actiongram, defining the story based on the characters and objects you choose and how you place them in your unique setting. Each story is different and the experience is playful. Your imagination is what brings the virtual and physical elements together into a story that you can record and share.
The story you create with Actiongram becomes not as much about the characters or objects, but the context in which you choose to place them. Like playing with dolls or action figures, you build the stories around the characters in an act of play. Even though there is a predefined library of characters and actions to choose from in the gallery, it’s about the make-believe you create blending virtual elements with your reality.
One of the nice things about Actiongram is that you don’t need to be a programmer or animator to create and share a story in AR. “Actiongram allows people without 3-D skills and visual effects experience to be amazing holographic storytellers,” says23 Kudo Tsunoda, CVP Next Gen Experiences, Windows and Devices Group. Actiongram also makes it possible to place AR content in the real world to record scenes that might otherwise be impractical or even improbable. Tsunoda explains, “Actiongram allows people to create videos with holograms and advanced visual effects that would normally require expensive software and years of experience to do.” The American band Miniature Tigers used Actiongram to create a music video for the song, “Crying in the Sunshine” (2016). “With this technology, we built a story inspired by ‘Crying in the Sunshine’ around our hero, the astronaut. It was really fun and mind blowing to be able to ‘place’ this astronaut in the space with us, and shoot around him, just like using a live actor,” says24 Meghan Doherty, one of the video’s directors. “It feels exciting and fresh to use a new tool to tell a story.”
I believe Actiongram will inspire more action-based AR libraries from developers, similar to digital sticker packs we use in messaging apps today. Digital stickers (and I predict 3-D AR stickers soon) are an extension of emojis and are a way to tell short expressive stories with pictures, beyond traditional language. I can see galleries of animated objects being used as a way to playfully augment communication in 3-D, extending how we use stickers today in messaging applications like Facebook Messenger and on the iPhone with iOS 10. Using iOS 10 allows you to place stickers directly on messages rather than just replying with a sticker. Why not place a 3-D AR sticker on the real world or in someone else’s environment to let them know that you’re thinking of them? Respond to a friend’s message on your AR glasses with a nodding 3-D virtual unicorn sticker, or build a shared augmented story between friends. Actiongram and 3-D AR stickers become present-day virtual telegrams.
5. You Are the Star: 3-D Photo-Realistic Personalized AR Avatars
Tools like Actiongram and 3-D AR stickers could become even more engaging by featuring personalized AR avatars. Consider the popularity of Bitmoji, a smartphone app by Bitstrips (acquired by Snapchat in 2016 for more than $100 million) that lets you personalize an expressive 2-D cartoon avatar for use in messaging applications. Creating a digital you in Bitmoji is easy: choose from several options to help mirror your appearance like face shape, hair color, eye color, and even face lines. After you’re done, your avatar appears in a variety of scenes within a library of digital stickers. You then can send your customized Bitmoji using applications like iMessage, Gmail, and Snapchat. “So much information is conveyed in your face and not just your words,” says25 Bitmoji cofounder Ba Blackstock. With Bitmoji, “you’re not just seeing text, you’re seeing your friend—it makes your text feel more human.”
We might soon trade in 2-D cartoony avatars of ourselves in messaging apps for photo-realistic 3-D avatars in AR. Loïc Ledoux, cofounder and CEO of Uraniom, wants to help you become the star of your augmented story. Uraniom is a web platform, with a mobile app in the works that helps you build a realistic-looking 3-D avatar, which you can use in any AR or VR app or video game. The venture began as a way to fix a big frustration for gamers—avatars looking horrible—with Ledoux realizing there were more applications beyond gaming. “Pairing our avatars with a device like HoloLens makes it possible to re-create almost authentic human interactions,” says Ledoux. “In AR and VR experiences, we will be able to get together with colleagues, family members, friends, to share many experiences. I am deeply convinced that in order to re-create authentic social interactions, using realistic, look-alike avatars is fundamental.”
Creating your photorealistic 3-D avatar with Uraniom presently works in three steps. First, use a 3-D scanning device like Structure Sensor or a tablet equipped with Intel RealSense to get a scan of yourself. Next, create an account on Uraniom’s web platform and upload the scan. Lastly, choose the game or app for which you want to create an avatar and follow the configuration process (adjusting the size of your head, skin tone, etc.). “We want to redefine what is your virtual identity,” says Ledoux. He continues:
Of course, you will have a perfect look-alike avatar for some situations (for example, business collaboration). However, what if you chose to have an avatar with a different look for meeting with your families, or something specific to strangers? We want you to have complete control of your digital self, in all digital environments, whatever the device or the platform.
Uraniom differs from Holoportation technology in that Holoportation is a live capture and real time transfer, whereas with Uraniom your customized 3-D avatar is pre-made and is ready to be inserted into AR experiences. Ledoux thinks Holoportation is a great project, but sees reasons why it could be difficult to scale. “You need a full 360 real time capture. This requires a hardware setup that, besides its cost, might not be adequate in some spaces,” says Ledoux. “Even if it is compressed, the amount of 3-D data you need to transfer in real time is pretty huge. Using an avatar, you just have to move the animation points. Holoportation is interesting for real time interaction, but might not be the best solution for some use cases.”
Ledoux sees Uraniom as a way to also enrich and further AR gaming. He explains:
Instead of interacting with a NPC [nonplaying character], what if your partner for a game were a close friend? What if the villain of the story were a family member? When you see these life-like characters, people you know in real life, will you have the same behavior as with random computer-generated characters? Likely not!
Ledoux’s sentiment echoes Blackstock’s perspective on how Bitmoji makes your text “feel more human” by “seeing your friend.” Similarly, using photorealistic avatars is a way to humanize an AR experience.
I also can see Uraniom being used in humorous shareable short AR films, similar to the wildly popular Elf Yourself, a viral website (2006) by director Jason Zada. By uploading a photo of yourself, or a friend, to the Elf Yourself website you see yourself dancing as an elf in a sharable video. The website JibJab offers a similar experience with personalized e-cards for a variety of occasions where you, or your friend, are the star of a short video. Now, instead of a 2-D dancing elf or other character with your face, imagine an AR experience you can share that features a 3-D avatar of yourself or someone else. Tools like Uraniom could make that possible.
As we become augmented humans, will our avatars be true to life? Will we be taller, more handsome or beautiful, and have a different eye color, augmenting our physical appearance the way we transform ourselves with high heels, makeup, colored contact lenses, and plastic surgery in reality? Will our avatars communicate stories of our true self, or will we choose to become someone, or even something else? I believe we will have a library of personalized AR avatars to send and share with multiple versions of ourselves, even out in the real world simultaneously. After all, an AR avatar that contextually changes and adapts is the next step in this second wave of AR.
1 Matthew Lombard and Theresa Ditton, “At the Heart of It All: The Concept of Presence.” Journal of Computer Mediated Communication, 3, no. 2 (1997).
2 Jay David Bolter and Richard Gruisin, Remediation: Understanding New Media (Cambridge: MIT Press, 1999), 8.
3 Ibid, 24.
4 Stephen Bottomore, “The Panicking Audience?: Early cinema and the train effect.” Historical Journal of Film, Radio, and Television 19 no. 2 (1999): 194.
5 Ibid, 213.
6 Dan North, “Magic and Illusion in Early Cinema.” Studies in French Cinema 1 no. 2 (2001): 70.
7 Oliver Grau, Virtual Art: From Illusion to Immersion (Cambridge: MIT Press, 2003), 152.
8 Ibid.
9 Jay David Bolter et al., “Presence and the Aura of Meaningful Places.” 7th Annual International Workshop on Presence Polytechnic University of Valencia, Spain: 13–15 October (2004): 37.
10 Ibid.
11 “Exploring Future Reality,” NYC Media Lab.
12 Bolter, Remediation, 49.
13 Ibid.
14 André Gaudreault and Phillipe Marion, “The Cinema as a Model for the Genealogy of Media.” Convergence 8 no. 12 (2002): 17.
15 Yvonne Spielmann, “Video: From Technology to Medium.” Art Journal 65 no. 3 (2006): 55.
16 “Magic Leap Partners With Lucasfilm’s ILMxLAB,” The Scene.
17 Natasha Lomas, “Prisma uses AI to turn your photos into graphic novel fodder double quick,” TechCrunch, June 24, 2016.
18 “holoportation: virtual 3D teleportation in real-time (Microsoft Research).”
19 “Introducing the New Blippar App: The Power of Visual Discovery.”
20 “Augmented Reality in Education: How To Turn The World Into An Interactive Learning Environment.”
21 “Blippar Is Building a Wikipedia of the Physical World,” TechCrunch, December 8, 2015.
22 “Microsoft HoloLens: Actiongram.”
23 Kudo Tsunoda, “Introducing first ever experiences for the Microsoft HoloLens Development Edition,” Windows Blogs, February 29, 2016.
24 Kim Taylor Bennett, “Miniature Tigers Go Astronautical with Their Video for ‘Crying in the Sunshine’,” Vice, September 26, 2016.
25 Joanna Stern, “Bitmoji? Kimoji? Digital Stickers Trump Plain Old Emojis,” The Wall Street Journal, September 28, 2016.