1. Mix Considerations

1. Clarity: as with any mix, linear or not, real time or not, achieving clarity is an essential aspect of our work. Many sounds sharing similar characteristics and spectral information will likely play on top of each other; our job is to make sure that all sounds are heard clearly and that, no matter what, the critical sounds for the gameplay are heard clearly above all else.
2. Dynamic range: a good mix should have a decent dynamic range, giving the player’s ears time to rest during low intensity moments and highlighting and enhancing the gameplay during action-packed sequences. A good dynamic range management will make it easier to hear the details of a well-crafted soundtrack, immersing the player further.
3. Prioritization: at any given moment, especially during the more intense portions of the game, the engine might attempt to trigger a large number of audio sources. The question for us is which of these sounds are the most relevant to the player and can provide them with information to play the game better, giving them a better gaming experience. For instance, a bit of critical dialog may be triggered at the same time as an explosion. While both need to be heard, the dialog, although much softer than the explosion, still needs to be heard clearly, and it is the responsibility of the developer to see to it.
4. Consistency: a good mix should be consistent across the entire game. The expectations developed during the earlier portions of the game in terms of quality and levels should be met throughout. Audio levels between scenes should be consistent and of course so should sounds by categories such as dialog, footsteps, guns etc.
5. Narrative function: the mix needs to support and enhance the story-line and gameplay. It needs to be both flexible and dynamic, reflecting both the environment and plot developments. This can mean something as obvious as the reverb changing when switching to a different environment, but it is often much more subtle. Simple moves like making the beginning of a sound slightly louder when it is introduced for the first time can tell the player to pay attention to something on the screen or the environment without being too obvious about it.
6. Aesthetics: this is harder to quantify, but there are certain things to look out for when thinking about the overall aesthetics of the mix. Does it sound harsh when played at high levels; is the choice of effects such as reverbs, delays and other processes optimized to serve the soundtrack as well as possible? Is it pleasant to listen to over long periods and at all levels, or is the bottom end clear and powerful yet not overpowering? These and many more questions are the ones that relate specifically to the aesthetics of a mix.
7. Spatial imaging: 3D and virtual/mixed reality environments require special attention to the spatial placement of sounds. Our mix needs to accurately represent the location of sounds in 3D space using the technologies at our disposal to the best of our abilities.
8. Inform: How do we create a mix that informs the player, providing them with important cues and establishing a dialog between the user and game itself? If all the points mentioned so far have been carefully weighted into your mix, very likely you’ve already succeeded in doing so.
   • Are the important sounds prioritized in the mix?
   • Does the mix reflect the environment accurately or appropriately? In this way the player is able to gain information on the space the scene takes place in.
   • In a 360-degree environment, sounds can be used to focus the attention of the player. Do make sure that sounds used in such a way are clearly heard, designed to be able to be easily localized; remember the chime vs. buzzer principle. Sounds with brighter spectrums and a sharp attack are easier to localize than low-frequency hums.

With so many variables involved, it isn’t very surprising that mixing is a skill that is acquired over time, likely by working on both linear and non-linear material. It is important to understand that a good mix is a dynamic one and that we should always be in control of it. Let’s begin by breaking down the mix into three main categories – music, dialog and sound effects – and understand each’s function in the context of a mix.

2. Music, Dialogue and Sound Effects

The soundtrack of games and movies can be broken down in terms of its three most important structural components: music, dialog and sound effects. Each serves a different purpose, and the mix is – or should – ultimately be dominated by one of these three elements at any given point based on the desired perspective and emotional impact.

Music serves to underscore or manipulate the emotional perspective of the game. It tells us how to feel and is usually the most emotionally impactful aspect of the soundtrack. The music throughout the development of a game or movie is often quite dynamic, from very soft to very loud, and we might need to make sure that, while preserving the original intentions of the composer and the needs of the scene, we keep the music within a comfortable range, only breaking out of it when there is a need to make a certain point.

The dialog is the main narrative element of the soundtrack, and as such it is usually treated as the most important aspect of the soundtrack when present. Nothing should get in the way of dialog unless intentional, taking precedence over music and sound effects. In games, dialog falls into two broad categories, critical, which contains the important narrative elements – in other words it moves the plot forward – and non-critical, which does not contain important information and can therefore be treated as chatter.

The sound effects serve a number of purposes. They greatly contribute to the feeling of overall immersion by giving us a detailed and rich aural picture of our environment; they take the place of sense that we cannot experience over a screen and speakers, and, crucial to gaming, provide us with information on the objects and the environment that we evolve in, such as location, texture, movement etc. Sound effects can also become part of the emotional or narrative aspect of a game or a scene. Indeed, none of these categories are absolute. A good sound designer will sometimes blur the lines between the music and sound effects by using sounds that blend with and perhaps even augment the musical score.

Note: when present, narration can sometimes be considered a fourth component of the soundtrack, to be treated independently of the dialog.

At any given moment, the mix should be driven or dominated by one of these categories – and usually only one. The same principle applies to movies. If there is dialog, the music and the sound effects should not get in its way, and we should consider taking them down in the mix. The choice of which category should dominate and when usually depends on the gameplay itself.

In video games you will hear the terms states or game states used quite often. Game states can be used to mean any number of things, as they are a technique for implementing artificial intelligence in games, sometimes described as finite state machine. Game states, as they relate to mixing, are usually derived from the significant changes in the gameplay such as switching from an exploration mode to battle mode. These changes in game states can be useful references for our mix to follow and adapt, and they ideally organically stem from the game itself.

3. Planning and Pre-Production

Planning is an essential part of the mixing process.

A mix can be approached like a complex problem that we need to solve, the problem being: how do we get a large number of audio files playing all at once, in an impossible-to-predict sequence, to sound like an ordered, easy to understand and pleasant mix, rather than a cacophony of overlapping noises?

b. Routing

Careful routing is essential in order to get the most flexible mix. Establishing a good routing structure is critical. It usually starts from the basic categories that constitute a traditional soundtrack, music, dialog and sound effects and gets further subdivided based on the sounds present in the soundtrack. At this stage you can effectively architect the mix and plan the various places where you will place dynamic compressors and set up side chain inputs. Every mix, every game is slightly different, but the following diagram should make for a good starting point from which to work.

Figure 11.1

As you can see, music, dialog and sound effects get their own subgroup, all routed to the main output, at the top of which sits a limiter, to prevent any signal to exceed 0dBFS and cause distortion. The limiter should probably have its output ceiling or maximum level output set slightly below 0dBFS – such as −0.3dBFS – and a quick attack time to catch fast transients and prevent them from getting past the limiter.

c. Dynamic Range

It is impossible to study mixing without at some point discussing the concept of dynamic range. The term can be confusing because dynamic range can be employed in either one of two contexts:

1. The difference between the loudest and softest points of a recording, expressed in dB.
2. The ratio between the softest and loudest signal intensity a particular piece of equipment may accurately capture or playback.

Concept 1 is relevant to us because a good mix should have a decent dynamic range, in order to create contrast, surprise and give the player a pleasant experience by giving their ears a break once appropriate. Without adequate dynamic range the mix becomes tough to listen to very quickly for a number of reasons. Without giving the player ear’s the opportunity to rest, it will inevitably become tiring, leading them to possibly turn off the soundtrack altogether. A mix without dynamic range or an insufficient one will also tend to deteriorate the transients of the sounds it plays back, shaving them off. This will blur the attack portion of percussive sounds and make it harder for our ears and brain to process. Over-compression and poor gain staging leading to levels on individual groups and/or the master level clipping are amongst the main culprits.

If the dynamic range is too large, however, and the difference between the softest sounds and loudest sounds is too great, likely the player will adjust their monitoring levels based on the loudest sounds played and will simply not hear the softest sounds in the mix, usually ambiences but possibly more, including dialog.

A good balance between the two is therefore required and needs to be achieved in order to create the best possible experience.

Measuring Dynamic Range in the Digital Audio Domain:

Figure 11.2

A compressor typically sits on the output of each of the three main subgroups as well. These tend to serve either one of two purposes: they can be used as a regular bus compressor, taking care of lowering loudness peaks in the signal routed through them, as well as blending all the sounds together via mild compression. They can also work as side chain compressors or ‘ducking’ compressors, usually taking their key input from the dialog and applying compression on the music and sound effects busses when dialog is present. For that reason and other potential ones, the dialog bus is usually split in two submixes: critical and non-critical dialog. Only the critical dialog would trigger the sidechain inputs on the compressors located on the music and sound effects busses. Typically, the compressor on the dialog will not have a key input and will work as a regular bus compressor.

The music bus will usually be a simpler setup, as while music soundtrack can get quite complex in terms of branching and adaptivity, the music or stems that comprise the soundtrack are usually already mixed. In some instances, if one is available a multiband compressor will sometimes help mix complex stems together. Since dialog may be triggered on top of the music, a compressor with a side chain input listening to the dialog will usually sit atop the music bus.

The sound effect bus is usually the busiest and most complex due to the number and variety of sound effects that make up a soundtrack. Just like the music bus, the sound effect bus will usually have a compressor keyed to the dialog group, sitting atop the bus, but the subgroup structure is usually much more complex. It is impossible to come up with a one-fits-all template, and each game has to be considered individually, but if we were to set up a mix for a first-person shooter, we might consider the following subgroups:

• Ambiences: room tones, outdoors sounds, city sounds.
• Player sounds: footsteps, player Foley, player vocalizations.
• Vehicles.
• Weapons: usually further divided into two main subgroups: friendly and enemies. Hearing your enemies’ positions, activity and fire is arguably more important than being able to hear the weapons from your own team, but further subgroups are possible.
• Explosions: explosions are dangerous, to our mixes anyhow. They tend to be the loudest elements of a game, and great care must be applied to avoid overwhelming the sound effect bus and possibly even the master bus. In order to do so, a limiter usually sits on top of the explosion bus.
• Enemies: footstep sounds, Foley, vocalizations.

Routing in mixing is usually done via busses, which are circuits or pathways that allow the mix engineer to route several audio tracks to a single destination. Unity uses a system of groups, which acts as a destination for multiple audio sources and send and receive modules to send signals from one group to another.

1. Adding Groups to the Unity Mixer

Groups can be added by clicking on the + button to the right of the groups label on the left panel. All subsequent subgroups will eventually run into the master group, but it is also possible to route the output of a group into other, cascading multiple subgroups. The Unity mixer defines two types of groups: child groups – which are routed into another – and sibling groups, that run in parallel to an existing group. You can change the routing hierarchy by dragging a group in the groups panel of the mixer window on top of the desired destination group or, when creating a new group, right clicking on an existing group and selecting either add child group or add sibling group. You can use the same contextual menu to rename, duplicate as well as delete groups.

Figure 11.4

The letters at the bottom of each group allow the developer to mute the group by clicking the M button, solo the group with the S button and bypass effects using the B button.

Figure 11.5

In the earlier snapshot, the ambiences group is a child of the SFX group. The audio output of the ambiences group will be routed to the SFX group, itself routed to the master group.

2. The Audio Group Inspector

When a group is selected in the hierarchy or directly by clicking on it in the mixer, the audio group inspector for that group will become active in the inspector window. Here you will find the different components that make up a mixer group. Whenever a group is created the following units are added automatically:

• Inspector Header: here you will find the name of the group. By right-clicking anywhere in this window a contextual menu will appear with two options.
• Copy all effects settings to all snapshots: this will copy all of the group’s settings on this group top all other snapshots in the mixer, allowing you to pass on to the group’s settings to all other snapshots.
• Toggle CPU usage display: will turn on CPU performance metering for all effects present in the group.
• Pitch Slider: this slider controls the pitch of all the audio routed through this group.
• Attenuation Unit: every group can only have one attenuation unit, which acts as a gain stage control, ranging from −80dB, which is silence, to +20dB. Each attenuation unit has a VU meter, which displays both the RMS value of the signal as well as its peak hold value. The RMS value is displayed by the colored bar itself while the peak value is displayed by a gray line at the top of the range.

3. Working With Views and Colors in the Unity Mixer

Managing large mixing sessions is always a bit of a challenge once your session has reached a certain size and you are dealing with a large number of groups. Visually, Unity provides us with two tools that can help us manage how we view and organize the information from the mixer window.

The first is the ability to color code our groups, which we can access by right clicking on the top of any group and selecting one of the available colors at our disposal.

Figure 11.6

This will add a colored strip at the top of each group right below the name and help visually break the monotony of the mixer window.

The other visual tool that Unity puts at our disposal is the ability to display only the relevant groups at any given time in our mix, hiding the ones we are not focused on, in order to minimize visual clutter. This is done with the views feature, located in the bottom left panel of the mixer window.

4. Adding Effects to Groups in Unity

Working with effects in the mixer opens a world of possibilities. With effects we can better control our mix and use in real time some of the same types of plugins that we are used to working with in DAWs. Keep in mind that any effects added to the mixer will increase the load on the CPU and need to be monitored and carefully thought through. Any effect that could be rendered as an audio file prior to being imported in Unity should be. That being said, real time effects are a very powerful way to make our mix more dynamic, engaging and fun.

You may add effects to each group by clicking at the bottom of the group itself on the Add … button. The available effects are: (Note: some of these effects were described in more general terms in the sound design chapter)

• Duck volume: this really is a compressor optimized for side chain input; in order to control the audio levels of the group it is added to using the level of another one, such as a compressor on the sound effect group, listening to input from the dialog group for instance.
• Low pass: a low-pass filter with resonance control.
• High pass: a high-pass filter with resonance control.
• Echo: a simple delay line.
• Flange: time-based modulation effect.
• Distortion: a simple distortion effect.
• Normalize: normalization is a process that adds gain to a signal in order to raise its peak amplitude. Unity developers intended for the normalize effect to preprocess a signal prior to sending it to a compressor.
• ParamEQ: an equalizer with independent frequency and bandwidth control and a visual representation of the curve applied to the sound.
• Pitch shifter: Unlike changing the pitch value of an audio source, which changes the pitch and the duration of the audio accordingly, the pitch shifter plugin provides independent time and pitch control. For instance, raising the pitch by an octave will not make the sound twice as short as it otherwise would be. This will prevent the ‘chipmunk’ effect but also does require more processing power than a simple pitch shift. Use sparingly.
• Chorus: another time-based modulation effect, often used for thickening sounds.
• Compressor: a full-featured dynamic range processor.
• SFX reverb: a full-featured procedural reverb, which we will look at in more detail shortly.
• Low pass simple: a low-pass filter without resonance, cheaper computationally than the low pass.
• High pass simple: a high-pass filter without resonance, cheaper computationally than the high pass.

5. Inserts vs. Effect Loops

Under the same menu you will also find send and receive, although these are not effects per se but rather a way to send a signal to another group. Signals tend to travel from top to bottom in a mixer, going through the various components of the group it is routed to. There will be times, however, where you will wish to send the signal to another place in the mixer or maybe send a copy of the signal to another group. In traditional mixing, this is done using sends and busses. A send is a circuit that lets the signal flow out of the current group it is routed to, and it uses busses to travel to various destinations in the mixer. Unity does not rely on the same terminology, shying away from using the word bus. Rather, Unity uses send and receive.

Most audio effects in a mixer can be added in one of two ways, as an insert or as an effect loop. While there are instances where either solution could be appropriate, there usually are good reasons to go with one or another method.

Adding an effect as an insert simply means inserting the effect on the group that the audio is routed to by simply pressing the Add … button at the bottom of the mixer and selecting an effect. The effect is now inserted into the signal path. This method is appropriate for effects we only wish to apply to one particular group. This is usually the case for equalization and compression, although there are instances where you might wish to use the effect loop method for compression, something known as parallel compression or ‘New York’ compression. Working with inserts is fine for these types of situations but starts to become more difficult to manage when the same effect needs to be applied to more than one group, such as reverberation. If we wish to apply reverberation to multiple groups, then we need to insert multiple copies of the plugin on each group, which is costly computationally and inefficient in terms of workflow. Inefficient computationally because inserting multiple versions of the same plugin, especially reverberation, is going to increase the computational load on the CPU but also inefficient in terms of workflow because any change made to one reverb will also need to be applied to all other instances if we wish for the effect to be consistent across all channels. When working with a complex mix, a simple task may turn into a much more difficult and therefore time-consuming one. A much better solution in this case is to set up a separate group just for reverberation, insert one instance of a reverb plugin on it, then route all the audio requiring reverberation to be applied to be routed directly to that group, creating an effect loop.

Figure 11.7

6. Setting Up an Effect Loop for Reverberation in Unity Using Send and Receive

Follow these simple steps to set up an effect loop you can use to route any group to a reverb plugin:

1. Create a new, dedicated group for reverberation and name it appropriately.
2. By clicking on Add … at the bottom of your new group, select Receive.
3. By clicking on Add … at the bottom of your new group, select SFX Reverb. Note: the signal in a group flows from top to bottom; it is important that the receive be added prior to the SFX Reverb plug in or if added after the fact, be moved up before it.
4. Select another group on the mixer that you wish to add reverb to, and using the Add … button select Send. In the inspector for that group you should now see the send component. Click the popup menu next to the word Receive and select the group you added the reverb to. The send level slider allows you to adjust how much signal you are sending to the group, therefore how much reverb you will hear on the sounds from that group. Feel free to experiment!

Some effects, such as reverberation, will allow the user to have independent control over the dry, unprocessed signal and wet signals. This raises the CPU usage a bit but does allow us to have much more control over our mix. To turn on that feature right-click on the SFX Reverb label in the reverb’s unit in the inspector and select Allow Wet Mixing.

Figure 11.8

Note: you may not use the Send/Receive technique on a group that is a child of another one, as that may result in a feedback loop. In other words, the output the of group on which reverb was applied cannot be routed to the group you are sending from. The receive group needs to be routed to the master group or another group that runs in parallel to the group we are sending from.

This technique is highly recommended for reverberation, echoes/delays and any processor that you wish to apply in the same way to multiple groups. Remember, by creating a send, you are sending a copy of the channel to another group; the original signal still flows through its original group. This gives the developer individual control over the dry signal (the original signal) and the wet signal (the copy going to the group reverb).

Note on Adjusting Levels During Gameplay

Figure 11.9

When Unity is in play mode, any change made to the game or any of its components will be lost as soon as you hit stop, and, as was pointed out earlier, you will need to make a note, mental or otherwise, if you wish to implement these changes after the fact. The Unity mixer is the exception. When in play mode, if you bring the mixer window in focus you will notice a button appearing labeled Edit in Play Mode. When pressed, changes you make to the mixer while playing will be remembered, allowing you to make adjustments to the mix as you play the game in real time.

7. Ducking in Unity

Ducking is especially useful when it comes to automating certain aspects of the mix. Ducking occurs when a compressor placed on a group, say group A, listens for a signal from another group, group B. When group B is active, the compressor will duck a volume on group A, making the signal from group A easier to hear. A common example of this is in radio, where the DJ’s voice will turn the music down when it comes on. The most common application of ducking in games is for dialog, which will often duck the volume on the music and sound effect groups. The control signal in the case of the DJ’s voice, also known as the key. Setting up a ducking compressor is very much like setting up an effect loop.

Usually this effect is achieved with a compressor equipped with a key signal input; Unity provides us with a dedicated tool for this, the duck volume plugin, which is in fact a regular compressor with a key input built in.

1. Working With Snapshots

Snapshots is a term borrowed from music production, where snapshot automation was developed in order to automate the changes needed during a mix on large format mixing consoles, which became too complex to work in real time. A snapshot of the mixer and all of its setting could be stored at various points in the song and recalled in real time during mixdown. This technique, borrowed from a very linear world (most music was still recorded to tape when this technology came of age), is turning out to be quite useful in video games. By using snapshots, we can adjust our mix to match the developments of the game.

Working with snapshots in Unity is a simple process. To create a snapshot, follow these steps:

1. Adjust the mixer to the desired setting.
2. Once dialed in, in the left panel of the mixer window, press the + sign to the right of the word Snapshots.
3. Name the snapshot, then press enter; you’re done!

Recalling a snapshot can be achieved either by clicking on the snapshot’s name in the mixer window – which is really only a valid method when mixing – or via script, as we shall see shortly.

2. Recalling Snapshots via Scripting

Snapshots can be easily recalled with scripting, using the TransitionTo() method, which will interpolate a transition to the new snapshot over the time specified by the user. The following example will demonstrate. This simple script will interpolate a transition between three snapshots labelled ambient, combat and victory in response to the user pressing the keys 1, 2 and 3 respectively.

First, create a new mixer or use an existing one, and create three snapshots, one for each of the three we outlined earlier: ambient, combat and victory.

You’ll notice right away that we added a new namespace using UnityEngine. Audio; which we need in order to use AudioMixerSnapshot. Next, after the class declaration we declare three new variables of type AudioMixerSnapshot, and by making them public they will show up in the inspector in the slot for the script. Prior to running this script, we need to assign an actual audio snapshot to each of the variables we just declared by clicking on the slot next to them in the inspector and selecting one of the three snapshots we created earlier in this example as demonstrated in the following illustration.

The transition time has been set to one second by default but may be changed by the user, in this case, simply by changing the value in the slot labelled transTime.

To see the example at work, make sure the mixer is showing upon entering play mode, and press the 1, 2 and 3 keys; you should see the sliders for the three subgroups move over the course of a second. Of course, in most cases the changes in the mix would not come from keystrokes by the user (although they might in some cases) but rather would be pushed by the game engine. It would be very easy to change this script to respond to another input, such as entering a trigger, an object or player getting spawned etc.

Figure 11.10

Note: transitions between snapshots are done using linear transitions by default. That can be changed by right clicking on any unit in the audio group inspector and selecting one of the other options.

Note on Edit in Playmode: this option will only appear while the editor is in play mode. When the game is running, the mixer is not editable and is controlled by the current snapshot or default one if none have been created. By enabling Edit in Playmode, the current snapshot is overridden and the game developer can now make changes and adjustments to the current snapshot.

Figure 11.11

3. Editing Mixer and Plugin Parameters via Scripting

Snapshots are a great way to manage a mix, but there might be time when you will need to control a single parameter and adjust it individually. In this case snapshots may not be a very good option. Rather, Unity allows you to control a single parameter from the mixer via the SetFloat() method, which takes only two arguments: a string, the name of the parameter to change and a float, which is the value for the parameter. However, before using SetFloat(), the parameter you wish to control individually has to be exposed. An exposed parameter will respond to values passed onto it by SetFloat() but will be removed from the snapshot, although as we shall see shortly, it can be returned to the snapshot if needed.

4. Exposing a Parameter: Controlling a Volume Slider

1. In order to expose a parameter, open the mixer you wish to expose a parameter of.
2. Select the group you wish to control the volume slider of.
3. In the inspector, locate the attenuation component and right click on the word Volume.
4. Select Expose ‘Volume of (name of group)’ to script. You will get confirmation that the parameter is now exposed by the arrow pointing right that will appear next to the name of the exposed parameter.
5. At the top right of the mixer window, you will notice a textbox that should now say Exposed Parameters (1). Clicking once on it will reveal the newly exposed parameter. Double click on the parameter to rename it.

Once the parameter has been exposed, we can now control it with a script, using SetFloat(). This simple script will change the value of a slider when the user presses the 8 or 9 keys on the keyboard.

A very simple script indeed. Note that the mixer, which contains the exposed parameter you wish to change has to be explicitly named, which is why we are including a reference to it at the top of the script by creating a public AudioMixer variable. Since it is public, this variable will show up as a slot in the script in the inspector and has to be assigned by the developer by either dragging the proper mixer onto the slot itself, or by clicking the little disc next to the words Main Mixer in the inspector.