Art is a very subjective area, dominated by personal opinion and preference. It can be difficult to say whether one piece of art is "better" than the other, and it's likely that we won't be able to find complete consensus on our opinions. The technical aspects behind art assets that support a game's artistry can also be very subjective. There are multiple workarounds that can be implemented to improve performance, but these tend to result in a loss of quality for the sake of speed. If we're trying to reach peak performance, then it's important that we consult with our team members whenever we decide to make any changes to our art assets as it is primarily a balancing act, which can be an art form in itself.
Whether we're trying to minimize our runtime memory footprint, keeping the smallest possible executable size, maximizing loading speed, or maintaining consistency in frame rate, there are plenty of options to explore. There are some methods that are clearly always ideal, and others which may require a little more care and forethought before being adopted, as they might result in reduced quality or increase the chances of developing bottlenecks in other components.
We will begin examining audio files, followed by Texture files, and finish up with meshes and animations. In each case, we will investigate how Unity loads, stores, and manipulates these assets during runtime, what our options are, and what we can do to avoid behavior that might generate performance bottlenecks.
Depending on the scope of the project, audio files can range anywhere from the largest disk space consumer, to the smallest. Unity, as a framework, can be used to build small applications that require only a handful of sound effects and a single background track, or to build large expansive role playing games that need millions of lines of spoken dialog, music tracks, and ambient sounds. It will be useful to make sense of how audio files are managed in Unity to better understand what we should do for the sake of optimization in both cases.
Many developers are surprised to find that runtime audio processing can turn into a significant source of CPU and memory consumption. Audio is often neglected on both sides of the gaming industry; developers tend not to commit many resources to it until the last minute, while users will rarely draw attention to it. Nobody notices when the audio is good or passable, but we all know what bad audio sounds like; it's instantly recognizable, jarring, and guaranteed to catch unwanted attention. This makes it crucial not to sacrifice too much audio clarity in the name of performance.
Audio bottlenecks can come from a variety of sources. Excessive compression, too much audio manipulation, too many active audio clips, inefficient memory storage methods, and access speeds are all ways to invite poor memory and CPU performance. But, with a little effort and understanding, all it takes is a few tweaks here and there to save us from a user-experience disaster.
When we select an imported Audio Clip in the Unity Editor, we see that the first option is the file's Load Type. There are three possible settings available:
- Decompress On Load
- Compressed In Memory
- Streaming
The actual moment when the file is first loaded will depend on other Audio Clip settings and how they are used during runtime, which we will explore later. This option merely defines the method of loading when it occurs.
Decompress On Load compresses the file on disk to save space, and decompresses it into memory when it is first loaded. This is the standard method of loading an audio file, and should be used in most cases.
Compressed In Memory loads the compressed file straight into memory when it is loaded, and decompresses it during runtime when it is being played. This will sacrifice runtime CPU when the clip is played, but improves loading speed and reduces runtime memory consumption. Hence, this option is best used for very large audio files that are used fairly frequently, or if we're incredibly bottlenecked on memory consumption and are willing to sacrifice some CPU cycles to play the Audio Clip.
Finally, the Streaming option will load, decode, and play files on-the-fly at runtime by gradually pushing the file through a small buffer. This method uses the least amount of memory for a particular Audio Clip and the largest amount of runtime CPU. This comes with the unfortunate drawback that they cannot be referenced more than once. Attempting to stream multiple copies of the same Audio Clip will generate a new buffer for each instance, resulting in a significant amount of RAM and runtime CPU cost if used recklessly. Consequently, this option is best reserved for single-instance Audio Clips that play regularly and never need to overlap with other instances. Ergo, this setting is best used with background music and ambient sound effects that need to play during the majority of a Scene's lifetime.
We can confirm much of this behavior by playing multiple instances of any given Audio Clip via multiple Audio Sources in a Scene, and benchmarking using the Audio View of the Profiler to see how much memory and CPU is consumed using the various Load Type options. However, be aware that the Profiler output for audio memory and CPU consumption in Editor Mode can be very misleading, since it performs audio loading differently to how it would occur in a runtime application.
The first time we load the Editor and enter Play Mode, the Editor will decompress audio files, costing a certain amount of memory and CPU cycles during initialization in order to accomplish this. We can observe the memory costs of this process in the Profiler under the Audio Area. However, if we restart the Scene, then we may notice that the memory spent on decompressing audio files suddenly drops to nearly 0 KB, since files have already been decompressed and the Editor has flushed away data it no longer needs. This is not representative of a real situation, as applications would need to perform this decompression.
So, if we want accurate audio profiling in Unity, we should run the Profiler against a standalone, or remote, version of our application on the intended platform/device.
There is an additional pair of options that affect audio file loading behavior:
- Load In Background
- Preload Audio Data
In the typical use case, where an Audio Clip is assigned to an Audio Source Component in a Scene, the audio file will be loaded into memory during Scene initialization. But, if the Load In Background option is enabled, then loading the Audio Clip will be deferred to a background task that begins once the Scene has finished initializing and gameplay has essentially begun. So, it stands to reason that we can improve the Scene loading speed by enabling this option for Audio Clips that we won't need until later in the Scene's lifetime, such as sound effects used in mid-level cut Scenes.
We can use an AudioClip object's loadState property to verify if the asset is loaded before attempting to use it. But, using the Load In Background option risks introducing jarring behavior if we try to access a sound file before it is fully loaded. In these cases, the sound file won't play until the background loading task has completed, in which case the sound will probably play at an inappropriate moment.
The second option, Preload Audio Data, is enabled by default, which tells the Unity Engine to automatically begin loading the file during Scene initialization. Disabling this option defers loading to the first instant that the AudioSource object's Play() or PlayOneShot() methods are invoked at runtime. This will cause a spike in the CPU, as the audio data is loaded from the disk, decompressed (or not, depending on the Load Type setting), pushed into memory, and eventually played once loaded.
Due to the amount of playback delay, and the performance cost, it is not recommended to rely on loading occurring at the moment that audio playback is needed. We should instead control loading ourselves by calling the AudioClip object's LoadAudioData() at some convenient moment before the file is needed. Modern games typically implement convenient stopping points in levels to perform tasks such as this, such as an elevator between floors or long corridors, where very little action is taking place. Note that we can also manually control the freeing of audio file memory using the AudioClip object's UnloadAudioData() method.
Solutions involving custom loading and unloading of audio data via these methods would need to be tailor-made to the particular game, depending on when Audio Clips are needed, how long they're needed for, how Scenes are put together, and how player(s) traverse them. This can require a significant number of special-case changes, testing, and asset management tweaks to achieve. So, it is recommended to save this approach as a "Nuclear Option" to be used late in production, in the event that all other techniques have not succeeded as well as we hoped.
Unity supports three general-case encoding formats for audio files, with a few specific cases that are platform-dependent (such as HEVAG for the PS Vita and XMA for XBox One). We will focus on the three optional encoding formats:
- Compressed
- PCM
- ADPCM
The compression algorithm used with the Compressed format will depend on the platform being targeted. Standalone applications, WebGL, and other non-mobile platforms use Ogg-Vorbis for compression, while mobile platforms use MPEG-3 (MP3).
The audio files we import into the Unity Engine can be one of many popular audio file formats, but the actual encoding that is bundled into the executable will end up in one of the previously mentioned formats. Statistics are provided for the currently selected format in the area following the Compression Format option, providing an idea of how much disk space is being saved by the compression, and how much memory the Audio Clip will consume at runtime (note that this also depends on the currently selected Load Type).
The encoding/compression format used can have a dramatic effect on the quality, file size, and memory consumption of the audio file during runtime, and only the Compressed option gives us the ability to alter the quality without affecting the sampling rate of the file. The PCM and ADPCM formats do not provide this luxury, and we're stuck with whatever file size those compression formats decide to give us, unless we're willing to reduce audio quality for the sake of file size by reducing the sampling rate.
Each encoding and compression format provides different benefits and pitfalls, and each audio file may be able to sacrifice one metric for another based on its purpose and contents. If we wish to properly optimize our application's audio files, then we need to be willing to use all of the available formats in a single application to make the most of our audio files.
The PCM format is a lossless and uncompressed audio format, providing a close approximation of analog audio. It trades large file sizes for higher audio quality, and is best used for very short sound effects that require a lot of clarity where any compression would otherwise distort the experience.
Meanwhile, the ADPCM format is far more efficient in both size and CPU consumption than PCM, but compression results in a fair amount of noise. This noise can be hidden if it is reserved for short sound effects with a lot of chaos, such as explosions, collisions, and impact sounds where we might not be aware of any generated artefacts.
Finally, the Compressed format will result in small files that have lower quality than PCM, but significantly better quality than ADPCM, at the expense of additional runtime CPU usage. This format should be used in most cases. This option allows us customize the resultant quality level of the compression algorithm, to tweak quality against file size. Best practices with the Quality slider are to search for a quality level that is as small as possible, but unnoticeable to users. Some user testing may be required to find the "sweet spot" for each file.
Now that we have a better understanding of audio file formats, loading methods, and compression modes, let's explore some approaches that we can make to improve performance through tweaking audio behavior.
Since each actively playing Audio Source consumes a particular amount of CPU, then it stands to reason that we can save CPU cycles for each redundant Audio Source that is playing in our Scene. One approach is to control our Audio Sources in such a way that we put a hard cap on how many instances of an Audio Clip can play simultaneously so that we don't attempt to play too many of the same sound effects simultaneously. We can do this by having a management object take control of our Audio Sources. It should accept sound playback requests and plays the file through any Audio Sources that aren't already busy playing an existing sound.
The management object could also throttle the requests such that there is a limit to how many versions of the same sound effect can play, and how many total sound effects are playing at any given time. This would be best used on 2D sound effects or single instances of 3D sound effects (which would require moving the Audio Source to the correct location at the moment of playback).
Almost every Audio Management Asset available in the Unity Asset Store implements an audio-throttling feature of some kind (often known as "audio pooling"), and for good reason; it's the best trade-off in minimizing excessive audio playback with the smallest amount of effort in preparing audio assets to make use of it. For this reason, and because these assets often provide many more subtle performance-enhancing features, it is recommended to use a pre-existing solution, rather than rolling out our own, as the scripting process can be very involved.
Note that ambient 3D sounds still need to be placed at specific locations in the Scene to make use of the logarithmic volume effect, which gives it the pseudo-3D effect, so the audio management system would probably not be an ideal solution. Limiting playback on ambient sound effects is best achieved by reducing the total number of sources. The best approach is to either remove some of them or reduce them down to one larger, louder Audio Source. Naturally, this approach affects the quality of the user experience, since it would appear that the sound is coming from a single source and not multiple sources, and so it should be used with care.
Each Audio Source in the Scene with a reference to an Audio Clip with Preload Audio Data enabled will consume a certain amount of memory for the Audio Clip (compressed, decompressed, or buffered, depending on Load Type) throughout the entire length of the given Scene. The exception to this rule is if two or more Audio Sources reference the same Audio Clip, in which case no additional memory is consumed as all of the Audio Sources are referencing the same location in memory and reading from it on an as-needed basis.
Audio Clips are unmanaged resources within Unity, which means they will not be released from memory merely by setting all references to null. Unity expects us to load and release these resources ourselves, creating and releasing them from memory as-needed. Keeping files in memory for long periods is reasonable for frequently used sound effects, since loading a file into memory every time it is requested would cost us some measure of CPU use.
However, if we find that we are using too much memory from these frequently used sound effects, then we must make the difficult choice of either reducing their quality or removing them entirely, to find memory savings. Removing sound effects completely is not necessarily a bad option. We might be able to achieve a unique sound effect by reusing existing sound effects combined with special effects and filters.
On the other hand, keeping infrequently used sound effects in memory for the length of a Scene can pose a significant problem. We might have many one-time-only sound effects, such as dialog clips, where there is little reason to keep them in memory just for the one unique moment they are needed. Creating Audio Sources with assigned Audio Clips (in their AudioClip property) will generate such references, and cause excess memory consumption even if we only use them for a single moment during gameplay.
The solution is to make use of Resources.Load() and Resources.UnloadAsset() to keep the audio data in memory only for as long as it is being played, and then immediately release it once it is no longer needed. The following code sample will achieve this effect in the simplest manner possible (tracking only a single Audio Source and Audio Clip) using our SingletonAsComponent class, just to give us an idea of what such a system might look like:
using UnityEngine;
using System.Collections;
using System.Collections.Generic;
public class AudioSystem : SingletonAsComponent<AudioSystem> {
[SerializeField] AudioSource _source;
AudioClip _loadedResource;
public static AudioSystem Instance {
get { return ((AudioSystem)_Instance); }
set { _Instance = value; }
}
public void PlaySound(string resourceName) {
_loadedResource = Resources.Load (resourceName) as AudioClip;
_source.PlayOneShot (_loadedResource);
}
void Update() {
if (!_source.isPlaying && _loadedResource != null) {
Resources.UnloadAsset(_loadedResource);
_loadedResource = null;
}
}
}This class can be tested with the following Component, which loads a sound file when the A key is pressed:
public class AudioSystemTest : MonoBehaviour {
void Update() {
if (Input.GetKeyDown(KeyCode.A)) {
AudioSystem.Instance.PlaySound("TestSound");
}
}
}Using this approach to play sound effects, we should note that memory for the sound is only allocated when it is played, and then immediately freed when the sound effect ends. Once again, multiple audio tools in the Unity Asset Store provide this feature, but the AudioSystem class can be expanded to handle multiple Audio Sources and Audio Clips fairly easily if we wish to customize the solution.
Enabling the Force to Mono setting on a stereo Audio File will mix it together with the channels into a single channel, saving 50 percent of the file's total disk and memory space usage. Enabling this option is generally not a good idea for some 2D sound effects where the stereo effect is often used to create a certain audio experience. But we can enable this option for some good space savings on 3D sound effects, where being a stereo source is generally meaningless, and 2D sounds where the stereo effect isn't important.
Resampling imported audio files to lower frequencies will reduce the file size and runtime memory footprint. This can be achieved through an Audio Clip's Sample Rate Setting and Sample Rate properties. Some files require high sample rates to sound reasonable, such as files with high pitches and modern music files. However, lower settings can reduce the file's size without much noticeable quality degradation in most cases. 22050 Hz is a common value for sources that involve human speech and classical music. Some sound effects may be able to get away with even lower frequency values. But, each sound effect will be affected by this setting in a unique way, so it would be wise to spend some time doing some testing before we finalize our decision on sampling rate.
If memory and hard disk consumption are not a burden on our application, then the WAV format can be used to reduce CPU costs at runtime, due to having a smaller overhead required to decode the data during each playback. Meanwhile, if we have CPU cycles to spare, we can save space with compressed encoding.
Streaming files from disk should be restricted to large, single-instance files only, as it requires runtime hard disk access; one of the slowest forms of data access available to us. Layered or transitioning music clips may run into major hiccups using this method, at which point it would be wise to consider the Resources.Load() approach. We should also avoid streaming more than one file at a time as it likely to inflict a lot of cache misses on the disk that interrupt gameplay.
Filter effects are additional Components that we can attach to an Audio Source to modify sound playback. Each individual Filter effect will cost some amount of both memory and CPU, and attaching duplicates to many Audio Sources in our Scene can result in dire consequences if they're used too frequently. A better approach is to make use of Unity's Audio Mixer utility to generate common templates that multiple Audio Sources can reference to minimize the amount of memory overhead.
The official tutorial on Audio Mixers covers the topic in excellent detail:
Unity's WWW class can be used to stream game content in via the Web. But, accessing a WWW object's audioClip property will allocate a whole new Audio Clip resource each time it is invoked, and similarly with other WWW resource-acquiring methods. This resource must be freed with the Resources.UnloadAsset() method once it is no longer required.
Discarding the reference (setting it to null) will not automatically free the resource, so it will continue to consume memory. Ergo, we should only obtain the Audio Clip through the audioClip property once to obtain the resource reference, use only that reference from that point forward, and release it when it is no longer required.
Audio Module files, also known as Tracker Modules, are an excellent means of saving a significant amount of space without any noticeable quality loss. Supported file extensions in Unity are .it, .s3m, .xm, and .mod. Unlike the common PCM audio formats, which are read as bit streams of data that must be decoded at runtime to generate a specific sound, Tracker Modules contain lots of small, high-quality PCM samples and organize the entire track similar to a music sheet; defining when, where, how loud, with what pitch, and with what special effects each sample should be played with. This can provide significant size savings, while maintaining high-quality sampling. So, if the opportunity is available to us to make use of Tracker Module versions of our music files, then it is worth exploring.