Perception and immersivity

VR relies upon a seamless interaction between the subject and the technology. When VR technologies project stimuli that surrounds and matches the users’ expectations, an immersive experience can take place and the subject experiences ‘presence’ – the internal psychological and physiological state whereby the subject has a sense of existing in a physical space even when physically located elsewhere. Generally, the more effective the VR system is at stimulating the subject’s senses in a realistic and expected way the more immersive the experience can be and the greater the potential for the subject to feel present in the virtualised world. However, when the mechanics of this stimuli are visible the illusion of a virtual reality is disrupted, immersion is limited or lost, and any feeling of presence is lost (Jerald 2016).

This relationship broadly holds true but is substantially different in the case of CVR. While CVR similarly relies on a hidden technology to convince us of the reality of the space we are seeing, the lack of mobility means that there is generally less reliance on the direct involvement of the viewer in the scene and more of a focus on allowing a scene to play out in the surrounding space. One of the main component illusions for creating a sense of reality in VR is that of a stable spatial place. That is, the stimuli presented to the viewer needs to feel and behave as though originating from real world objects in a three-dimensional space (Jerald 2016). Cutting and Vishton propose that the space around the subject can be segmented into three circular egocentric regions that gradually transition into one another:

It is interesting then to note that the experience of personal space ends at roughly the two-metre mark, which just so happens to be a key point for CVR. In most 360° camera rigs, moving closer than two metres results in a distortion of the image. Moving an object or person closer to the rig requires more cameras spaced closer together to ensure an accurate representation of the image (Anderson 2016: 24). While there are compositing tricks that can be done to work around this, most 360° camera rigs available to the amateur and semi-professional film maker do not allow for variations in their rig, often utilising simple rigs of two cameras. While this makes the technology cheaper and more available, it has resulted in a large amount of output that eschews action taking place within the personal space of the viewer, resulting in a feeling of distance from the subject or environment. While there are cases where interaction within the viewer’s personal space benefits the material, generally speaking some distance between the action and the viewer benefits a CVR experience. Unlike a VR environment where the viewer has a degree of agency and interaction, and can respond to actions within their personal space (they can interact or move away from the intrusion, for example), CVR plants the viewer in a fixed space. Intrusion into personal space directly affects the viewer, however unlike VR they are unable to undertake any response to this other than turning their backs on the intrusion. The result is somewhat akin to locked-in syndrome, where the viewer is aware but paralysed, a potentially distressing experience, creating a highly unnatural situation and ultimately reinforcing the awareness of the technology used to mediate the experience and compromising the viewer’s ability to feel present within the space.

This role of the viewer as an observer of the scene is, in many ways, the cinematic component of CVR, and is in keeping with the cinematic experience of watching a moving image projected onto a flat screen. It is this combination of observation, immersivity, and presence that defines CVR as a unique media, foregoing VR’s focus on embodying the subject in a space but immersing the viewer in a space or place just the same. Führerstandsmitfahrt U44 in 360° // DSW21 (einundzwanzig 2017) demonstrates this.¹ The video is a 25-minute real time recording of the Westfalenhütte to Marten tram route, in Dortmund, Germany, from the perspective of the driver’s cab. Across this journey we see the sights and sounds of the city from the familiar perspective of public transport, looking out the window as the city passes by. While the experience is passive, it is ultimately the same kind of passivity as the lived experience of being driven by public transport, arguably helping create immersivity by placing the subject in a situation that we know through lived experience is naturally lacking in agency. Meanwhile the technological illusion is unbroken, allowing the viewer to experience the sights and sounds of a foreign city without a clear disjoint between perspective and technology.

While intrusion into personal space can often be undesirable, it can nonetheless be effective if the mechanisms of the technology are readily addressed as a part of the immersive experience. That is, if the medium of CVR is not trying to convince the viewer they are in a space but rather convince them of the truthful documentation of a space. Scott Base 360 VR Walkthrough by Anthony Powell (2017) is a 45-minute real time walkthrough of the Scott Antarctic base, designed to help prepare future visitors for their time there. The video consists of Powell walking around the site with the 360° camera attached to a selfie-stick in a single unedited take. The technology of the video is clearly visible throughout, with the central figure of Powell moving in and out of what could be considered personal and action spaces of perception. The open Antarctic landscape is quickly replaced with familiar clean but utilitarian, cramped work spaces and corridors, largely non-descript but immediately familiar. What we have here is not an immersive site per se, but rather immersive documentation brought about through CVR. Viewer agency and intrusion into personal space is less problematic here because it is not an intrusion into a virtualised reality, rather we are clearly watching the documentation of Powell’s intrusion into the camera’s personal space as a product of his navigation of the surrounding environment. This reinforces the truthful expression of physical space, added greater weight through the inside-look nature of the documentation itself, creating an equally immersive experience.

Space and reality

Unlike fixed frame media which necessarily presents a perspective of any given location, CVR concerns itself with placing the viewer within a representation of a physical space. This space may seek to document the unique characteristics of the space itself, or to stage some sort of event or happening within and in relation to it. The use of a recognisably real space points to a kind of truth in the medium – regardless of what happens in the space itself, the space in which it is set retains a connection to a reality. Send Me Home (LONLEYLEAP 2018) documents the story of Rickey Jackson who was wrongfully imprisoned for murder at the age of 18 and spent almost four decades in prison before being exonerated and released in 2014. Throughout the documentary the viewer is situated in contrasting spaces, from the close confines of Jackson’s death row cell and prison hallways to the wide open spaces of the home and property where Jackson now raises a family. The emotionally affective visual language of this documentary is a product of CVR’s ability to represent these physical spaces.

While a representation of the reality of space in CVR perhaps best lends the medium to documentary and field recording projects, creatives have attempted to utilise the media within narrative fiction as well. However, the construction of narrative stories within CVR does not negate the role that space and the representation of reality have to the medium itself. To explore this relationship, we can look at two examples of narrative film that uses CVR – Hard World for Small Things and Home Invasion.

Hard World for Small Things (Transport by Wevr 2017) goes to great effort to create a realistic experience. Set in South Central Los Angeles, the majority of the action takes place on the street outside a corner store, allowing the viewer to watch the different interactions unfold between characters in a naturalistic way. While utilising actors, the exchanges feel rooted in the communities in which they are set, and the absence of visible film making accoutrement or production techniques reduces any barrier to immersion. Although staged, the film presents its characters within a real and common enough location. The action that takes place is almost banal, as befits the setting. When the film does finally finish on an act of violence, it is handled naturalistically, underplayed, maintaining the illusion of reality established by the space itself.

We can compare this to Alex T. Hwang’s Home Invasion (Tiberiusfilm LLC 2017), a short narrative film about a family subjected to a violent assault by gangster types. The ‘home’ appears to be a real home – in the grand tradition of low-budget film making it would not be surprising to find out that it was owned by a relation of someone in the film crew. As such, the setting is effective in creating a real home in which to set the ‘invasion’. However, the action in the film is never able to create a sense of reality – the unnatural dialogue, overdubbed film soundtrack, awkward blocking, and poorly established character motivations all work to undo any sense of immersion. Instead the viewer is constantly reminded that they are watching people making a film and acting within a real space. The film has a large number of issues quite aside from its inability to communicate effectively within CVR, but through this failure we can see some core facets of the media. CVR accurately captures action within the context of the space in which it is set, and if the creative plans to introduce artificial elements to this space, they will need to do so in a way that effectively works within the reality of the space itself.

However, it is possible to create a CVR experience that operates outside of a conventional experience of reality, instead creating an experience that constructs its own reality. The 360° music video accompanying Gorillaz’s Saturnz Barz (Spirit House) (Gorillaz 2017) utilises the potential for animation to blend real and fantastical elements, bending time and space so that, whilst unrealistic, feels totally in keeping with the world that has been constructed. Similarly, Mike Celona’s 360° video experiences utilise a mobile frame to experience audiovisual works that resist any sort of reality (Celona 2017). Mapping his fixed frame AV productions to a sphere so as to surround the viewer, Celona fuses textures and shapes with stock film footage, creating reworkings of recognisable images and forms, remapped to a new virtual space. This new virtual space creates its own sense of depth, of shapes and imagery that is always at the fringe of being perceptible. The mobile frame in turn restricts the viewer’s perspective, ensuring that the entire scope of the visual environment is never able to be completely understood. The result is the creation of a virtual space that operates outside of any sort of realistic spatial understanding, and instead creates its own unreal but navigable space.

Audible spaces

In Audio-Vision: Sound on Screen Michel Chion poses this question with response to sound in film:

The relationship between sound and the mobile frame is somewhat different due to the latter not acting as an image in and of itself, rather it is a perspective of a larger, freely navigable panorama. As such, we could propose a rewording of this excerpt to fit this new context: “what do sounds do when put together with a mobile frame? They dispose themselves in relation to the space in which they are set.” CVR has an inherent spatial quality as a visual medium, built on a literal positioning of perspective within a space that can only ever be suggested in fixed frame media. As such, the creation of an effectively immersive VR experience can be thought to rely on the creation of a field of sound that:

1. Is reflective of the space in which the perspective is set, and
2. Dynamically responds to changes in the viewer’s perspective of the space.

This first point reflects the creative decision making and selection of audio materials for a scene. These may be obtained through field recordings or created abstractly, separate to the space in which the video is recorded, and later paired with the video. The second point is more technical, and more of a question of implementation than creative decision making. Ambisonic audio has established itself to be the dominant audio format for CVR, allowing for 3D sound relationships to be recorded or encoded to a single multichannel audio file, and decoded dynamically in response to changes in viewer perspective. Outside of bespoke experiences in concert or exhibition settings, YouTube has supported positional audio for first order ambisonics since April 2016 (Wiggins 2016) while the feature is promised, but as of publication yet to be implemented, by alternative video sharing platform Vimeo. As such, while dynamically changing audio feedback is an essential element of CVR, its generation and implementation is not insignificantly complicated. For this reason, a large number of CVR experiences distributed online have opted to work only with stereo audio. As such, this chapter is primarily concerned with audiovisual relationships established through sound materials, and not wider spatial relationships.

Sights & Sounds of a Coffee Plantation by Shivakumar Lakshminarayana (2017) is a one-minute field recording of a space in a coffee plantation accompanied entirely by audio captured from the site itself. It is a clear example of a site-specific audio-visual field recording, situating the viewer within a VR representation of the space as accurately as possible. What is interesting to note here is the disjoint between the sonically rich field and its accompanying, relatively static scene, highlighting that the implementation of field recording in a CVR context does not necessarily have a correspondingly dynamic interplay of sound and visual activity. Similarly, Powell’s tour of the Scott Antarctic base paints a similar audiovisual picture. The environmental sound changes dynamically as Powell moves throughout the site, with sounds from the harsh industrial sounds of the generator room to the quieter monitoring stations and administration spaces not necessarily reflecting a dramatic shift in the visual appearance of the corresponding spaces. In this way, the auditory experience is directly related to the experience of the space itself and helps to build a convincing virtual reality.

Some examples of CVR have opted to use sound that is not reflective of the space in which the experience is set, rather opting for a music overdub. Legal Nomads’s Uganda w 360 (2017) is a good example of this. While there are sounds from the field present, there is a non-diagetic music track that dominates the video throughout. The result of this is a constant reminder of the artificial construction of the media experience, resulting in a less immersive experience of the various locations in which the viewer is placed. The importance of sound design in video work has long been acknowledged, however in the advent of a mobile frame the experience of the sound field is now tied directly to the experience of space and place.

My original work Inland (Gillies 2018) explores the dynamic between real and unnatural sound by blending field recordings and processed sound samples to create an evolving soundscape that weaves between the audiovisual interactions discussed earlier. The piece establishes early on that it operates within an unreal experience of space akin to the work of Mike Celona, pivoting between recognisable physical spaces and visual texture. A unique audiovisual interaction is constructed, so that abstract ‘musical’ soundscapes are paired with visual spaces that retain their identifiable physical properties, whilst field recordings are paired with spaces constructed of distorted, unnatural, or unidentifiable environments. By establishing from the outset that the piece intends not to operate within a construction of the real but instead contrasting audiovisual elements that reflect various aspects of real spaces, Inland utilises elements of site-specific field recordings to build a satisfying CVR experience without necessarily devoting the work to a literal depiction of a space.

Viewer attention

While the function of space and sound to create an effectively immersive experience can be addressed separately, they often act as mutually affective forces to guide the viewer’s interaction with CVR. One important part of CVR where both elements combine is in the effective direction of viewer attention. Kath Dooley has discussed the implications of CVR’s viewer attention mechanic, explaining that:

This freedom opens up potential problematic elements however, as the audience can choose to ignore or simply miss the action you are attempting to direct their attention towards (Dooley 2017). As a way of counteracting this effect, Anderson suggests that the action of a shot takes place within a 150° space in front of the viewer. This is based on a field of view of 90°, with an extra 30° of space made visible through head turns (Anderson 2016: 39). Action that takes place outside of this 150° field of view requires some degree of contortion by the viewer which at best makes for an uncomfortable experience and at worst can result in the viewer either losing attention or becoming unsure as to which direction they are supposed to be looking.

Through observation it appears that in watching a work of CVR, there are four possible outcomes with respect to directing the viewer’s attention during a CVR experience:

1. The viewer’s attention is effectively directed to the desired focus point.
2. The viewer’s attention is lost or misplaced, leading to them missing the subject or points of action.
3. The viewer’s attention is split between two equal points of focus in exclusionary positions in the field, leading to them being unsure in which direction to look.
4. The direction of the viewer’s attention is not an essential element to the audio-visual work.

To explore these states we can examine three different 360° music videos – Gorillaz’s Saturnz Barz (Spirit House) (2017), Muse’s Revolt (2016), and Björk’s stonemilker (2015). Gorrilaz’s Saturnz Barz (Spirit House) (henceforth Spirit House) is filled with well-executed and intuitive viewer direction. The pacing of action within, and cuts between, scenes is established early on and remains consistent throughout, such that the viewer quickly develops an understanding of the expected rate of change they should be experiencing throughout the video. When a perspective lacks action for a period the viewer is given a subtle hint to move their head in another direction through lighting or some sort of minor action or activity. The decision to do so is immediately rewarded by the presence of a new subject interaction. As such, at points of ambiguity, the viewer is clearly empowered, and subtly directed, to find the desired perspective in a manner that feels free and intuitive. Most importantly, all of the action guiding viewer attention takes place within the action space of the viewer, maximising viewer engagement.

We can compare this experience of viewer direction to the behaviour exhibited in Muse’s 360° music video Revolt. The viewer witnesses the action ostensibly from the perspective of a drone flying around a clearly staged and stylised clash between a civilian rebellion and authoritarian stormtroopers. Issues arises where there are potentially a large number of elements on screen to look at, and no clear visual line to follow. For example, the video begins with a military convoy converging on an empty lot, with government vehicles approaching and passing by the viewer-drone as it flies through the convoy in the opposite direction. As the viewer flies over the scene, the impulse is the turn around and watch the convoy pass by as this is clearly a point of interest and there is little of note on the horizon in the direction in which the drone is heading. However, doing so positions the viewer to be facing away from the subject of the next shot as, when the scene changes, the viewer now has their back to the band performing live, requiring a further 180° turn once they realise the point of interest is behind them. In this way, there’s a lack of continuity between shots.

At some points this is a mere annoyance, but at others it results in the viewer potentially missing important plot points. In one case about halfway through, the viewer’s perspective cuts from being surrounded by figures to a distant shot of the conflict, seemingly giving the viewer a respite from the intensity of being within a riotous conflict. However, immediately after this cut an important plot point of revolutionaries being arrested takes place almost directly below the viewer at a distance (in the vista space or at least beyond the viewer’s action space). Some attempt is made to orient the viewer through the use of sound design, adding audible glitch sounds to accompany changes in the viewer/drones HUD, drawing the viewer’s attention to the action. However, these cues are reliant on the viewer facing in the general direction of these events for them to be effective. If the viewer was facing away from the action, as they would be were they to focus on the aggressors in the previous scene, they might start hunting for the source of the sound and visual markers rather than engaging with the scene unfolding in front of them.

The end result of this poor scene construction is that the viewer is disoriented and does not feel meaningfully engaged with the action in the space around them. Instead of intuitively navigating the virtual world to follow the action in a meaningful way they are instead constantly forced to engage with the technology to reorient their perspective to try to find the more desirable perspective. Much of the action appears to have been created with an eye towards interesting spaces in which to inhabit – the 360° views of riots and drones both near and far are all engaging spaces in their own right. The problem is that these scenes don’t communicate between one another as a coherent entity. The video appears to want to let the viewer freely look around in some scenes, but then constructs other scenes such that they can only convey meaningful information when the viewer happens to be looking in a particular direction and at a particular angle. This dissonance ultimately reminds the viewer of the virtual world they are inhabiting.

One possible solution to this concern with viewer attention is to create an experience that does not rely on guiding the viewer’s perspective. Björk’s video for stone-milker (Björk 2015) effectively creates such a space. Set on the Icelandic beach where Björk wrote the lyrics for the song, the video takes place across two key scenes on this beach, the first on an empty stretch of desolate beach, the second amongst the rocks in a lightly more detailed environment. Both scenes focus on Björk slowly working her way around the camera, singing to the viewer, and always occupying the viewer’s action space. As scenes progress, multiple instances of Björk appear and occupy different points of the field. In many cases there may not be an easy way to take in all of Björk’s action, forcing the viewer to focus on one particular instance of the singer at any given point in time. However, the actions themselves are fairly non-descript, consisting of Björk singing, dancing, or moving in ways that feel unconnected to her multiple instances.

In stonemilker, Björk becomes a part of the field, and the significance of the space becomes less about the action taking place within it than of the viewer being situated in a space significant to the song’s creation. Indeed, the original VR mix of the track takes the string arrangement and situates each of the 30 instruments in a tight circle around the viewer, steadfastly placing the viewer in the middle of a spatial experience (Björk.fr 2015) – but a spatial experience that has no directional queue, rather orienting the musical components as objects in the field, crafting a sonic character for the space. As such, while there are elements for the viewer to focus on, viewer attention is not a key element of narrative comprehension. This approach is in many ways contrary to the conventions of fixed media, but highlights the unique characteristic of CVR – its ability to accurately convey and communicate space.

Conclusion

CVR has several unique characteristics which creates a multimedia experience unique to that of conventional fixed frame media. All CVR experiences are based on convincing the viewer they are present in a given space and seeks to minimise the presence of conscious technological mediation of that space. This is the experiential foundation from which CVR’s screen grammar emerges. As such, the audiovisual relationship between elements of a CVR experience differs from that of fixed frame media, focusing not on forming a relationship between image and sound, but rather the relationship between sound, action, and space. What has been found through observation and experimentation is that works that base their materials – sound, action, object, setting – in the space in which they’re set, or in which they’re suited, more effectively generate a feeling of being present within the experience.

But inasmuch as the materials must match the experience of space, so too must the presentation of that space not draw attention to its artificiality, resulting in a number of conventions that minimise conscious technological mediation. The viewer’s inactivity within the space must be normalised as natural for the experience while also minimising activity within the viewer’s personal space, ensuring the viewer is appropriately positioned so as to view all important action within their field of view. Traditional fixed media techniques such as hard edits and montage draw attention to the artificiality of the visual space and disorient the viewer. This reinforces the idea that the viewer derives meaning not through the relationship of different perspectives of space, as is commonly articulated in fixed frame media, but rather through the relationship of the viewer to the space itself. CVR’s more effective implementations trend towards long periods spent in a single location, with perspective changes corresponding to scene changes that allow for plenty of time for the viewer to orient themselves to changes in their surroundings. Finally, the use of these spatial and technical elements must act harmoniously together to effectively guide the viewer’s mobile perspective, either directing their perspective to desired points of focus or encouraging them to freely experience the virtual world.

CVR represents a significant development in visual media, eschewing a composed presentation of perspective through a fixed frame in favour of situating the viewer within a space with a freely navigable perspective. The uniqueness of this media renders many of the conventions of fixed frame media ineffective, and by understanding CVR’s relationship with presence and immersion, this chapter provides a framework for the development of meaningful screen grammar and the effective use of CVR not as a novelty but as a way to create audiovisual experiences not possible any other way.

References

• Anderson, G. (2016) The Cinematic VR Field Guide: A Guide to Best Practices for Shooting 360° (1st ed.). [online]. Available online: www.jauntvr.com/cdn/uploads/jaunt-vr-field-guide.pdf [Accessed 28/03/17].
• Björk. (2015) björk: stonemilker (360 degree virtual reality). Available online: www.youtube.com/watch?v=gQEyezu7G20 [Accessed 15/05/19].
• Björk.fr. (2015) Stonemilker VR | Björk Vulnicura VR. Available online: www.bjork.fr/Stonemilker-VR [Accessed 15/05/09].
• Celona, M. (2017) Remorse Code (360). Available online: https://vimeo.com/217303220 [Accessed 28/05/19].
• Chion, M. (1994) Audio-Vision: Sound on Screen. New York: Columbia University Press.
• Cutting, J.; Vishton, P. (1995) Perceiving Layout and Knowing Distances: The Integration, Relative Potency, and Contextual Use of Different Information about Depth. In: Handbook of Perception and Cognition (Vol. 5). San Diego, CA: Academic Press. pp. 19–20.
• Dooley, K. (2017) Storytelling with Virtual Reality in 360-Degrees: A New Screen Grammar. Studies in Australasian Cinema. 11(3). Routledge. pp. 161–171.
• einundzwanzig. (2017) Führerstandsmitfahrt U44 in 360° // DSW21. Available online: https://vimeo.com/208103595 [Accessed 15/05/19].
• Gillies, S. (2018) Inland (2018) [1OA, 360]. Available online: www.youtube.com/watch?v=wuBTy2-fUDc&t=585s [Accessed 02/06/19].
• Gorillaz. (2017) Gorillaz-Saturnz Barz (Spirit House) 360. Available online: www.youtube.com/watch?v=lVaBvyzuypw [Accessed 15/05/19].
• Jerald, J. (2016) The VR Book. IL: Association for Computing Machinery and Morgan & Claypool Publishers. pp. 46–49.
• Lakshminarayana, S. (2017) Sights & Sounds of a Coffee Plantation. Available online: https://vimeo.com/241473904 [Accessed 31/05/19].
• Legal Nomads. (2017) Uganda w 360. Available online: https://vimeo.com/218451644 [Accessed 01/06/19].
• LONLEYLEAP. (2018) Send Me Home | 360° Experience. Available online: https://vimeo.com/295684342 [Accessed 27/05/19].
• Merriam-Webster Dictionary. (n.d.) Definition of Virtual Reality. Available online: www.merriam-webster.com/dictionary/virtual%20reality [Accessed 15/05/19].
• Muse. (2016) Muse-Revolt [360° Music Video]. Available online: www.youtube.com/watch?v=91fQTXrSRZE [Accessed 15/05/19].
• Powell, A. (2017) Scott Base 360 VR Walkthrough. Available online: https://vimeo.com/207563097 [Accessed 15/05/19].
• Sterling, B. (2009) Augmented Reality: ‘The Ultimate Display’ by Ivan Sutherland, 1965. Available online: www.wired.com/2009/09/augmented-reality-the-ultimate-display-by-ivan-sutherland-1965/ [Accessed 15/05/19].
• Tiberiusfilm LLC. (2017) Home Invasion VR 360. Available online: https://vimeo.com/207889899 [Accessed 28/05/19].
• Transport by Wevr. (2017) Hard World for Small Things | 360 VR. Available online: www.youtube.com/watch?v=VPTPQ_CYRsQ [Accessed 27/05/19].
• Wiggins, B. (2016) YouTube, Ambisonics and VR. Available online: www.brucewiggins.co.uk/?p=666 [Accessed 01/04/19].