1. Inform: How, What

In a 3D or VR environment sound can and must play an important role when it comes to conveying information about the immediate surroundings of the user. Keeping in mind that the visual window available to the player usually covers between 90–120 degrees out of 360 at any given time, sound quickly becomes indispensable when it comes to conveying information about the remaining portion of the environment. It should also be noted that, while the visual field of humans is about 120 degrees, most of that is actually peripheral vision; our actual field of focus is much narrower. The various cues that our brain uses to interpret these stimuli into a distance, direction and dimension, will be examined in more detail in a future chapter, but already we can take a preliminary look at some of the most important elements we can extract from these aural stimuli and what they mean to the interactive and immersive content developer.

a. Geometry/Environment: Spatial Awareness

In a game engine, the term geometry refers to the main architectural elements of the level, such as the walls, stairs, large structures and so on. It shouldn’t be surprising that sound is a great way to convey information about a number of these elements. Often, in gaming environments, the role of the sound designer extends beyond that of creating, selecting and implementing sounds. Creating a convincing environment for sound to propagate in is often another side of the audio creation process, known as environmental modeling. A well-designed environment will not only reinforce the power of the visuals but is also a great way to inform the user about the game and provide a good backdrop for our sounds to live in.

Figure 2.1

Some of the more obvious aspects of how sound can translate into information are:

• Is the environment indoors or outdoors?
  • If indoors, what is the order of the size of the room we find ourselves in?
  • If outdoors, are there any large structures, natural or man-made, around?

• Do we have a clear line of sight with the sound we are hearing, or are we partially or fully cut off from the source of the sound? We can isolate three separate scenarios:
  1. We are fully cut off from the audio source. The sound is happening in an adjacent room or outside. This is known as occlusion. There is no path for the direct or reflected sound to get to the listener.
  2. The path between the audio source and the player is partially obstructed, as in a small wall or architectural feature (such as a column for instance) blocking our line of sight. In this case the direct audio path is blocked, but the reflected audio path is clear: that is known as obstruction.
  3. The direct path is clear, but the reflected sound path isn’t, blocking the reverberated sound: this is known as exclusion.

Each of these situations can be addressed and simulated in a soundtrack and provide the user with not just an extremely immersive experience but also valuable information to help them navigate their environment and the game itself.

2. Entertain

The focus of this book being on sound design and not composition, we will think of music in relation to the sound design and overall narrative and emotional functions it supports.

3. Defining Immersion

Entire books have been dedicated to the topic of immersion – or presence – as psychologists have referred to it for several decades. Our goal here is not an exhaustive study of the phenomenon but rather to gain an understanding of it in the context of game audio and virtual reality.

We can classify virtual reality and augmented reality systems into three categories:

• Non-immersive systems: typically, simple Augmented Reality systems that affect one sensory input. Playing a 3D game on a laptop is a common example. This is the type of system most people are familiar with.
• Semi-immersive systems: typically allows the users to experience a 3D world while remaining connected to the real world. A flight simulator game played on a multiscreen system with realistic hardware, such as a flight yoke, would be a good example of such a system.
• Fully immersive systems: affect all or most sensory inputs and attempt to completely cut off the user from their surroundings through the use of head mounted displays, headphones, and additional systems such as gaming treadmills, which allow the user to walk or even run though a virtual environment.

An early definition of presence based on the work of Minski, 1980 would be:

The sense an individual experiences of being physically located in an environment different from their actual environment, while also not realizing the role technology is playing in making this happen

We in the gaming world tend to think of presence or immersion as a rather novel topic, one that came about with games and virtual reality. Truly, however, the concept has been part of conventional medias such as literature for hundreds of years. Narrative immersion happens when a player or reader is so invested in the plot that they momentarily forget about their surroundings.

There is no doubt, however, that games and virtual reality have given us a new perceived dimension in the immersive experience, that is, the possibility to act in an environment, not simply having the sensation of being there. So, what are the elements that scientists have been able to identify as most likely to create immersion?

The research of psychologist Werner Wirth suggests that successful immersion requires three steps:

1. Players begin to create a representation in their minds of the space or world the game is offering.
2. Players begin to think of the media space or game world as their main reference (aka primary ego reference).
3. Players are able to obtain useful information from the environment.

Characteristics that create immersion tend to fall in two categories:

1. Characteristics that create a rich mental model of the game environment.
2. Characteristics that create consistency amongst the various elements of the environment.

Clearly, sound can play a significant role in all these areas. We can establish a rich mental model of an environment through sound by not only ‘scoring’ the visuals with sound but by also by adding non-diegetic elements to our soundtrack. For instance, a pastoral outdoor scene can be made more immersive by adding the sounds of birds in various appropriate locations, preferably randomized around the player, such as trees, bushes etc. Some elements can be a lot more subtle, such as the sound of wood creaking, layered every once in a while, with footsteps over a wooden surface, for instance. While the player may not be consciously cognizant of such an event, there is no doubt that these details will greatly enhance the mental model of the environment and therefore contribute to creating immersion.

Consistency, this seemingly obvious concept, can be trickier to implement when it comes to creature sounds or interactive objects such as vehicles. The sound an enemy makes while it is being hurt in battle should be different than the sound that same creature might make when trying to intimidate its enemies, but it should still be consistent overall with the expectations of the player based on the visuals and, in this case, the anatomy of the creature and the animation or action. Consistency is also important when it comes to sound propagation in the virtual environment, and, as was seen earlier in this chapter, gaming extends the role of the sound designer to modeling sound propagation and the audio environment in which the sounds will live.

Inconsistencies in sound propagation will only contribute to confusing the player and cause them to eventually discard any audio cue and rely entirely on visual cues.

Indeed, when the human brain receives conflicting information between audio and visual channels, the brain will inevitably default to the visual channel. This is a phenomenon known as the Colavita visual dominance effect.

As sound designers, it is therefore crucial that we be consistent in our work. This is not only because we can as easily contribute and even enhance immersion as we can destroy it, but beyond immersion, if our work is confusing to the player, we take the risk of having the user discard audio cues altogether.

It is clear that sensory rich environments are much better at achieving immersion. The richness of a given environment maybe given as:

• Multiple channels of sensory information.
• Exhaustiveness of sensory information.
• Cognitively challenging environments.
• Possessing a strong narrative element.

Additionally, while immersion can be a rather tricky thing to achieve, it is rather easy to break. In order to maintain immersion, research suggests that these elements are crucial:

• Lack of incongruous audio/visual cues.
• Consistent behavior from objects in the game world.
• Continuous presentation of the game world – avoid commercials, level reset after a loss.
• The ability to interact with objects in the game world.

While some of these points may be relatively obvious, such as the lack of presence of incongruous elements (such as in-game ads, bugs in the game, the wrong sound being triggered), some may be less so. The third point presented in this list, ‘continuous presentation of the game world’, is well illustrated by the game Inside by Playdead studios. Inside is the follow-up to the acclaimed game Limbo, and Inside’s developers took a very unique approach to the music mechanics in the game. The Playdead team was trying to prevent the music from restarting every time the player respawned after being killed in the game. Something as seemingly unimportant as this turns out to have a major effect on the player. By not having the music restarted with every spawn, the action in the game feels a lot smoother, and the developers have removed yet one more element that may be guilty of reminding the player they are in a game, therefore making the experience more immersive. Indeed, the game is extremely successful at creating a sense of immersion.

It is important to note than the willingness to be emotionally involved is also an important, perhaps crucial, factor to achieving immersion. This is something that developers have no control over and that pre-supposes the desire of the user to be immersed. This is sometimes referred to as the ‘Fan Gene’. As a result, two users may have wildly differing experiences when it comes to the same experience, based, partially, on their willingness to ‘be immersed’.

1. Implementation

It is impossible to understate the importance and impact of implementation on the final outcome, although what implementation actually consists of, the process and its purpose often remain a somewhat nebulous affair. In simplistic terms, implementation consists of making sure that the proper sounds are played at the right time, at the right sound level and distance and that they are processed in the way the sound designer intended. Implementation can make or break a soundtrack and, if poorly realized, can ruin the efforts of even the best sound designers. On the other hand, clever use of resources and smart coding can work their magic and enhance the efforts of the sound designers and contribute to creating a greater sense of immersion.

Implementation can be a somewhat technical process, and although some tools are available that can partially take out the need for scripting, some programming knowledge is definitely a plus in any circumstance and required in most. One of the most successful third-party implementation tools is Audio-kinetic’s Wwise, out of Montreal, Canada, which integrates seamlessly with most of the leading game engines such as Unity, Unreal and Lumberyard. The Unreal engine has a number of tools useful for audio implementation. The visual scripting language Blue Print developed by Epic is a very powerful tool for all-purpose implementation with very powerful audio features. As a sound designer or audio developer, learning early on what the technical limitations of a game, system or environment are is a crucial part of the process.

Because the focus of this book is to work with Unity and with as little reliance on other software as possible, we will look at these concepts and implementation using C# only, although they should be easy to translate into other environments.

2. Repetition and Fatigue Avoidance

We have already seen in Chapter one that the first generations of gaming hardware did not rely on stored PCM data for audio playback as is mostly the case today but instead used on-board audio chips to synthesize sounds in real time. Their concerns when it came to sound therefore had more to do with number of available voices than trying to squeeze as many samples as possible on a disc or download. Remember that the Atari 2600 had a polyphony of two voices.

The 1980s saw the rise and then dominance of PCM audio as the main building blocks of game soundtracks. Audio samples afforded a level of realism that was unheard of until then, even at the low resolutions early hardware could (barely) handle. Along with increased realism, however, came another host of issues, some of which we are still confronted with today.

Early video game systems had very limited available RAM, as a result of which games could ship only with a small amount of samples. Often these samples were heavily compressed (both in terms of dynamic range and data reduction), which severely reduced the fidelity of the recording or sound, making them hard to listen to, especially over time. In addition, since so few samples could be included, they were played frequently and had to be used for more than one purpose. In order to deal with listener fatigue, game developers early on developed techniques that are still relevant and in use today, the most common being randomization.

The use of random and semi-random techniques in sound and music, also known as Stochastic techniques, had been pioneered by avant-garde composers such as John Cage and Iannis Xenakis in the 1950s and 1960s. These techniques, directly or indirectly, have proved to be extremely helpful for game developers.

The use of random behaviors is a widespread practice in the gaming industry, which can be applied to many aspects of sound.

Randomization can be applied to but is not limited to:

1. Pitch
2. Amplitude
3. Sample Selection
4. Sample concatenation – the playback of samples sequentially
5. Interval between sample playback
6. Location of sound source
7. Synthesis parameters of procedurally generated assets

(Working examples of each of the techniques listed in the following and more are provided in the scripting portion of the book.)

The most common of these techniques is the randomization of pitch and amplitude, often built into game engines, such as Unreal, in which it’s been implemented as a built-in feature for iterations. Pitch and amplitude randomization might be a good start, but it’s often no longer enough to combat listener fatigue. Nowadays developers rely on more sophisticated techniques, often combining the randomization of several parameters. These more advanced, combinatorial techniques are sometimes referred to as procedural, a term in this case used rather loosely. In this book, we will tend to favor the stricter definition of the term procedural, that is, real time creation of audio assets, as opposed the real time manipulation of existing audio assets. The difference between procedural asset creation and advanced stochastic techniques are sometimes blurry, however. These more advanced random or stochastic techniques are certainly very important, and their usefulness should not be underestimated.

3. Interactive Elements and Prototyping

One of the challenges that even very accomplished sound designers coming from linear media tend to struggle with the most initially when working in gaming is the interactive elements, such as vehicles, machines, weapons and other devices the user may interact with. Interactivity makes it difficult to predict the behavior of a game object and therefore cannot be approached in a traditional linear fashion. How can one design sounds for a vehicle not knowing in advance how the user will interact with it? Simple things such as acceleration, braking sounds and the sound of tires skidding when the vehicle moves at high speed are all of a sudden part of a new equation.

The answer when addressing these issues is often prototyping. Prototyping consists of building an interactive audio model of the object, often in a visual environment such as Cycling74’s MAXMSP, Native Instrument’s Reaktor or Pure Data by Miller Puckette, to recreate the intended behavior of the object and test in advance all possible scenarios to make sure that our sound design is on point and, just as importantly, that the sounds behave appropriately. For instance, in order to recreate the sense of a vehicle accelerating, the engine loop currently playing back might get pitched up; inversely, when the user is slamming on the breaks the sample will get pitched down, and eventually, in more complex simulation, another sample at lower RPM might get triggered if the speed drops below a certain point and vice versa.

Working with interactive elements does imply that sounds must be ‘animated’ by being pitched up, down, looped and processed in accordance with the circumstances. This adds another layer of complexity to the work of the sound designer: they are not only responsible for the sound design but also for the proper processing and triggering of these sounds. The role of the sound designer therefore extends to determining the range of the proper parameters for these actions, as well as the circumstances or threshold for which certain sounds must be triggered. The sound of tires skidding would certainly sound awkward if triggered at very low speeds, for instance. Often, these more technical aspects are finely tuned in the final stages of the game, ideally with the programming or implementation team, to make sure their implementation is faithful to your prototype. In some cases, you might be expected to be fluent both as a sound designer and audio programmer, which is why having some scripting knowledge is a major advantage. Even in situations where you are not directly involved in the implementation, being able to interact with a programmer in a way they can clearly comprehend, with some knowledge of programming, is in itself a very valuable skill.

4. Physics

The introduction and development of increasingly more complex physics engines in games introduced a level of realism and immersion that was a small revolution for gamers. The ability to interact and have game objects behave like ‘real-world’ objects was a thrilling prospect. Trespasser: Jurassic Park, released in 1998 by Electronic Arts, is widely acknowledged as the first game to introduce ragdoll physics, crossing another threshold toward full immersion. The case could be made that subsequent games such as Half Life 2, published in 2004 by Valve Corporation, by introducing the gravity gun and allowing players to pick up and move objects in the game, truly heralded the era of realistic physics in video games.

Of course, physics engines introduced a new set of challenges for sound designers and audio programmers. Objects could now behave in ways that were totally unpredictable. A simple barrel with physics turned on could now be tipped over, dragged, bounce or roll at ranges of velocities, each requiring their own sound, against any number of potential materials, such as concrete, metal, wood etc.

The introduction of physics in game engines perhaps demonstrated the limitations of the sample-based paradigm in video game soundtracks. It would be impossible to create, select and store enough samples to perfectly cover each possible situation in the barrel example. Some recent work we shall discuss in the procedural audio chapter shows some real promise for real-time generation of audio assets. Using physical modeling techniques we can model the behavior of the barrel and generate the appropriate sound, in real time, based on parameters passed onto us by the game engine.

For the time being, however, that is, until more of these technologies are implemented in production environments and game engines, we rely on a combination of parameter randomization and sample selection based on data gathered from the game engine at the time of the event. Such data often include the velocity of the collision and the material against which the collision occurred. This permits satisfactory, even realistic simulation of most scenarios with a limited number of samples.

5. Environmental Sound Design and Modeling

In creating the soundtrack for a large 3D game or environment, one should consider the resulting output as a cohesive whole instead of a collection of sounds playing somewhat randomly on top of each other. This kind of foresight and holistic approach to sound design allows for much more engaging and believable environments and a much easier mix overall. The soundtrack of a game is a complex environment, composed of many layers playing on top of each and changing based on complex parameters determined by the gameplay. In a classic first-person shooter game, the following groups or layers of sounds could be playing at any single time over each other:

• Room tones: drones, hums.
• Environmental sounds: street sounds, weather.
• Dialog and chatter.
• Foley: footsteps, movement sounds.
• Non-player characters: AI, creatures, enemies.
• Weapons: small arms fire, explosions.
• Machinery: vehicles, complex interactive elements.
• Music.

This list does gives us a sense of the challenge that organizing, designing, prioritizing and playing back all these sounds together and keeping the mix from getting cluttered represents.

In essence, we are creating a soundscape. We shall define soundscape as a sound collage that is intended to recreate a place and an environment and provide the player with an overall sonic context.

In addition to having the task of creating a cohesive, complex and responsive sonic environment, it is just as important that the environment itself, within which these sounds are going to be heard, be just as believable. This discipline is known as environmental modeling and relies on tools such as reverberation and filtering to model sound propagation. Environmental modeling is a discipline pioneered by sound designers and film editors such as Walter Murch that aims at recreating the sonic properties of an acoustical space – be it indoors or outdoors – and provides our sounds a believable space to live in. The human ear is keenly very sensitive to the reverberant properties of most spaces, even more so to the lack of reverberation. Often the addition of a subtle reverberation to simulate the acoustic properties of a place will go a long way in creating a satisfying experience but in itself may not be enough. Environmental modeling is discussed in further detail in this book.

6. Mixing

The mix often remains the Achilles’ heels of many games. Mixing for linear media is a complex and difficult process usually acquired with experience. Mixing for games and interactive media does introduce the added complexity of unpredictability, as it isn’t always possible to anticipate what to expect sonically in an interactive environment where events may unfold in many potential ways. We must teach the engine to deal with all potential situations using a carefully thought-out routing and rules of architecture for the game to follow. In most situations the game has no or little awareness of its own audio output.

Our challenge is, as it is so often in game audio, twofold: ensure a clean, crisp and dynamic mix while making sure that, no matter what, critical audio such as dialog is heard clearly under any circumstances and is given priority. Discussing the various components of a good mix is beyond the scope of this chapter and shall be addressed in detail in Chapter twelve.

7. Asset Management and Organization

A modern game or VR simulation requires a massive number of audio assets. These can easily range in the thousands, possibly tens of thousands for a AAA game. Managing these quickly becomes a challenge in itself. Game engines, even third-party software such as Wwise, should be thought of as integration and creative tools rather than asset creation tools. The line between the two is not always an absolute one, but as a rule you should only import in the game engine and work with polished assets ready to be plugged in as quickly and painlessly as possible. While you can fix some issues during the implementation process, such as amplitude or pitch adjustments, you should avoid consistently relying on adjusting assets in the game engine for matters that could have been taken care of sooner. This tends to cost time and create unnecessarily complex projects. It is much more time-efficient to make sure all assets are exported and processed correctly prior to importing them.

An asset delivery checklist, usually in the form of a spreadsheet, is a must. It should contain information about the following, but this list is not exhaustive:

• Version control: you will often be dealing with multiple versions of a sound, level, game build etc. due to fixes or changes. Making sure you are working with and delivering the latest or correct file is obviously imperative.
• Deadlines: often the work of the sound design team is split up into multiple deadlines for various assets types in order to layer and optimize the audio integration and implementation process. Keeping track of and managing multiple deadlines is a highly prized and useful organizational skill.
• Consistency and format: making sure that all the files you will be delivering are at the proper format, sample rate, number of channels and at consistent sound levels across variations, especially for sounds that are related (such as footsteps for instance), quickly becomes challenging and an area where it is easy to make mistakes.
• Naming convention: dealing with a massive number of assets requires a naming convention that can easily be followed and understood by all the team members. The naming convention should be both descriptive and as short as possible:
  • Hero_Fstps_Walk_Wood_01.wav
  • Hero_Fstps_Walk_Metal_02.wav
  • Hero_Fstps_Run_Stone_09.wav

Deciding on a naming convention is something that should be carefully considered in the preproduction stages of the game, as it will be very inconvenient to change it halfway through and could cause disruptions in the production process. Keep in mind that audio files are usually linked to the engine by name.