1. The Genesis of Audio in Games

Video games are a relatively new art form, one that was borne out of the boredom and curiosity of computer scientists, advances in technology and the human need for new entertainment. It is generally agreed upon that the first commercially released, mass produced video game was an arcade game called Computer Space, in 1971, by Nutting Associates. The origins of video games, however, can be traced to the Massachusetts’ Institute of Technology in the United States, where, in 1962, Steve Russel developed Spacewar! on a DEC PDP-1 computer. But it was in 1972 that the iconic game Pong was released. Pong was perhaps the tipping point, the game that took video games out of the realm of computer scientists, science fiction fans and software engineers and brought it out to the general public, introducing the art form to our culture at large. The game was not about computers or spaceships and as such didn’t necessarily and specifically appeal to the science and computer nerds amongst us, at least not specifically. It was, of all things, about sports. Table tennis, a game most people could relate to and probably enjoy or have enjoyed playing at some point. There perhaps was the genius behind it all, when Nolan Bushnell, who co-founded the mythical gaming company Atari, asked programmer and game developer Allan Acorn to create a table tennis game as an exercise. Although extremely primitive by today’s standards – the game was black and white, the graphics were entirely made up of squares and rectangles and the gameplay was extremely simple – still, it was fun to watch for onlookers and the game demanded attention wherever it was found. Pong’s contribution to the video game industry and our culture in general cannot be understated.

In many ways, Pong hit all the marks a successful game ought to. It was easy to learn but hard to master, could be played alone or with a friend and was just the right amount of difficult (the ball speed would slowly increase as the play continued on then reset at the next play). In some ways, the soundtrack was perhaps the crudest aspect of the game. There was no music, a simple, musical ping to let you know you had hit the ball, a similar sound but slightly lower in pitch when the ball hit the walls and a slightly noisier sound, more akin to a buzzer, when you failed to hit the ball at all. Yet, this simple audio implementation, realized by someone with no audio training, still resonates with us to this day and was the opening shot heard around the world for game audio. Indeed, Allan Acorn may not have studied modern sound design, but his instincts for game development extended to audio as well. The soundtrack was definitely primitive, but it reinforced and possibly even enhanced the very basic narrative of the game and is still with us today.

To say that technology and games have come a long way since then would be both an understatement and commonplace. Today’s games bear little resemblance to Pong. The level of sophistication of technology used by modern game developers could not have been foreseen by most Pong gamers as they eagerly dropped their quarters in the arcade machine.

1972 also marked what’s commonly referred to as the first generation of home gaming consoles, with the release of a few devices meant for the general public. One of the most successful of these was the Magnavox Odyssey. It had no audio capabilities whatsoever, and although it enjoyed some success, its technology was a bit crude, even for its time. The games came with overlays that the gamer had to place on their TV screen to make up for the lack of graphic processing power, and with hindsight, the Odyssey felt a bit more like a transition into interactive electronic home entertainment systems than the first genuine video gaming console. It wasn’t until the next generation of home gaming hardware and the advent of consoles such as the Atari 2600, introduced in 1977, that the technology behind home entertainment systems became mature enough for mass consumption and started to go mainstream and, finally, included sound.

The Atari 2600 was a huge commercial success. It made Atari an extremely successful company and changed the way we as a culture thought of video games. Still, it suffered from some serious technical limitations, which made it difficult to translate the hit coin-operated games of the times such as Pac Man or even Space Invaders into compelling console games. Still, these did not stop Atari from becoming one of the fastest growing companies in the history of the US. When it came to sound, the Atari 2600 had a polyphony of two voices, which was usually not quite enough for all the sounds required by the games, especially if the soundtrack also included music.

Besides the limited polyphony, the sound synthesis capabilities of the 2600 were also quite primitive. The two voice polyphony was created by two onboard audio chips that could only produce a very narrow array of tones, pitches and amplitude levels. No audio playback capabilities and limited synthesis technology meant that the expectation of realistic sound was off the table for developers back then.

It’s also sometimes easy to forget that nowadays, when major game studios employ thousands of designers, coders and sound designers, game development in the early days of the industry was a very personal matter, often just one person handling every aspect of the game design, from game logic to graphics and, of course, music and sound design. Sounds in early video games were not designed by sound designers, nor was the music written by trained composers. Perhaps it is the combination of all these factors, technical limitations and limited expertise in sound and music, combined with a new and untested artform pioneered by visionaries and trailblazers, that created the aesthetics that we enjoy today when playing the latest blockbusters.

Figure 1.1

2. From Sample Playback to Procedural Audio

Technology evolved quickly after the Atari 2600. As the graphics and game-play improved with each generation of new hardware, audio sample playback technology started to find its way into video games in arcades at first and in-home entertainment systems later on. Although the first attempts to implement sample playback in games were not always very satisfying or even convincing, due to the initial limitations of the technology such as low sample rates (as low as 11Khz), bit depth (as low as 8 bits) and heavily compressed audio formats at low bit rates, eventually, as the technology improved so did the fidelity of the audio samples we could include and package in our games. And so, eventually, along with audio playback technology and the ability to use recorded sound effects in games, games soundtracks started to improve in terms of fidelity, impact and realism. It also started to attract a new generation of sound designers, often coming from linear media and curious or downright passionate about gaming. Their expertise in terms of audio production also helped bring game soundtracks out of the hands of programmers and into those of dedicated professionals. Although game audio still suffered from the stigma of the early days of low fidelity and overly simplistic soundtracks, over time these faded, and video game studios started to call upon the talents of established composers and sound designers to improve the production values of their work further still. With better technology came more sophisticated games, and the gaming industry started to move away from arcade games toward games with complex story lines and narratives. These, in turn, provided sound designers and composers with more challenging canvases upon which to create and, of course, also provided more challenges for them to overcome. More complex games required more sounds and more music, but they also demanded better sounds and music, and the expectations of the consumers in terms of production values started to rival those of Hollywood blockbusters. This, however, meant much more than to simply create more and better sounds. Issues in gaming, which had been overlooked so far, became much more obvious and created new problems altogether. It was not quite enough to create great sounds, but the mix and music had to be great while at the same time adapt and reflect the gameplay. This demanded the creation of new tools and techniques.

Over the years, however – with increasing levels of interactivity and complexity in gameplay, sample playback’s dominance in the world of game audio and the inherent relative rigidity that comes with audio recordings – signs that other solutions were needed in order for our soundtracks to respond to and keep up with the increasingly complex levels of interaction available in games started to appear. This became more obvious when real-world physics were introduced in gaming. With the introduction of physics in games, objects could now respond to gravity, get picked up and thrown around, bounce, scrape and behave in any number of unpredictable manners. The first major release to introduce ragdoll physics is generally agreed to be Trespassers: Jurassic Park, a game published in 1998 by Electronic Arts. Although game developers usually found ways to stretch the current technologies to provide acceptable solutions, it was impossible to truly predict every potential situation, let alone create and store audio files that would cover them. Another crack in the façade of the audio playback paradigm appeared more recently, with the advent of virtual and augmented reality technologies. The heightened level of expectations of interaction and realism brought on these new technologies means that new tools had to be developed still, especially in the areas of environmental modeling and procedural audio.

Procedural audio is the art and science of generating sound effects based on mathematical models rather than audio samples. In some ways it is a return to the days of onboard sound chips that generated sound effects from primitive synthesis chips in real time. Generating sounds procedurally holds the promise of sound effects that can adapt to any situation in the game, no matter what.

Procedural audio is still a relatively nascent technology, but there is little doubt that the level of expertise and fluency in audio technologies significantly increases with each new technical advance and will keep doing so. As a result, we can expect to see a fragmentation in the audio departments of larger game development studios, labor being divided in terms of expertise, perhaps along a similar path to the one seen in graphic departments. Sound design and the ability to create compelling sounds using samples are going to remain a crucial aspect of how we generate sounds, but in addition we can expect to see increased specialization in several other areas, such as:

• Spatial audio: the ability to create and implement sound in 360 degrees around the listener.
• Procedural sound synthesis: designing audio models via scripting or programming that can accurately recreate a specific sound.
• Virtual reality and augmented reality audio specialists: working with these technologies increasingly requires a specific set of skills specific to these mediums.
• Audio programming and implementation: how to make sure the sound designed by the audio team is triggered and used properly by the game engine.
• Technical sound design: the ability to connect the sound design team to the programming team by designing specialized tools and optimizing the overall workflow of the audio pipeline.

Each of these topics could easily justify a few books in their own rights, and indeed there are lots of great tomes out there on each specific topic. As we progress through this book, we will attempt to demystify each of these areas and give the reader not only an overview of the challenges they pose but also solutions and starting points to tackle these issues.

3. How to Use This Book