• 14.1 Create Sound and Music Files
• 14.2 Load Music and Sound Effects
• 14.3 Specify Sound and Music Controls
Music and sound effects set mood and heighten realism, and thus are essential elements for video games. In this chapter we discuss how to play music and multiple sound effects simultaneously, as illustrated in Figure 14.1, which shows Snail Bait just before the runner is about to collide with multiple sprites, resulting in multiple sound effects.
Games typically provide user interface controls to let players turn sound and music on and off. Additionally, games package their sound effects in a single audio file, known as an audio sprite sheet to reduce startup time.
In this chapter you will learn how to do the following:
• Load audio files (Section 14.2 on p. 349)
• Implement controls that let players turn music and sound on and off (Section 14.3 on p. 350)
• Play music in a loop (Section 14.4 on p. 351)
• Pause music when the game pauses (Section 14.4 on p. 351)
• Create an audio sprite sheet (Section 14.6.1 on p. 358)
• Seek to locations in an audio sprite sheet (Section 14.6 on p. 355)
• Play sounds from an audio sprite sheet (Section 14.6 on p. 355)
• Implement multichannel sound (Section 14.6.3 on p. 361)
The online examples for this chapter are listed in Table 14.1.
Audio has been one of the most difficult HTML5 APIs for browser vendors to implement reliably. An interesting case study in HTML5 audio is Microsoft’s Cut the Rope HTML5 game. It uses HTML5 audio, but falls back to Flash audio for some browsers that don’t properly implement audio. You can find Cut the Rope on the web at www.cuttherope.ie.
In this chapter, we use the rather simplistic HTML5 audio
element for music and sound effects. Although it’s limited in scope, the audio
element is robust enough to play multiple sound effects simultaneously overlaid on a soundtrack on most platforms; most notably, iOS can play only one sound at a time.
Some games, however, need more sophisticated audio support. HTML5’s Web Audio API provides that support by letting you define a graph of audio nodes you connect to render audio.
At the time this book was written, the Web Audio API was not well supported among browsers, with only 57% support vs. 83% for the audio
element. See caniuse.com for more current figures.
This chapter discusses how to implement Snail Bait’s music and sound effects on the desktop. HTML5 audio on mobile devices has a few peculiarities that are discussed in Chapter 15.
Snail Bait uses two audio files, one for the game’s music and another for its sound effects, as you can see in Figure 14.2. For each of those files, Snail Bait has an MP3 version and an Ogg version, which suffices for all modern browsers that support HTML5.
All Snail Bait’s music and sound effects could reside in a single file, which would save an additional HTTP request at startup. As we discussed in [Missing XREF!], it’s important to keep HTTP requests at a minimum so that games load quickly.
In this case, however, placing everything in a single file puts the sound effects into a rather large file: 2.5 MB for the MP3 version, instead of the much smaller 108 KB file containing only audio sprites. As this book was being written, some browsers exhibited a considerable performance penalty when seeking for audio in a large file, so Snail Bait opts for separate files for sounds and music instead of one.
Now that you’ve seen Snail Bait’s sound and music files, let’s see how the game loads them.
The HTML5 specification originally required the Ogg Theora format for video because it was freely available and open source and because the specification’s authors believed it was better to specify a single format rather than many. Mozilla and Opera are big supporters of Ogg Theora.
Some companies, however, such as Apple and Nokia, were concerned about patent issues, and Apple didn’t think it was a good idea to directly specify a video format in the specification. As a result, the specification was rewritten and the requirement for Ogg Theora was removed.
Subsequently, in 2010, Google acquired On2’s VP8 format and released the software under an irrevocable free patent and BSD-like license. In January 2011, Google announced that it would end native support for MPEG-4 in Chrome.
Snail Bait loads audio files with the HTML shown in Example 14.1.
<html>
...
<body>
<!-- The music soundtrack -->
<audio id='snailbait-music' preload='auto'>
<source src='music/music.ogg' type='audio/ogg'>
<source src='music/music.mp3' type='audio/mp3'>
</audio>
<!-- Sound effects -->
<audio id='snailbait-audio-sprites' preload='auto'>
<source src='sound-effects/audio-sprites.ogg'
type='audio/ogg'>
<source src='sound-effects/audio-sprites.mp3'
type='audio/mp3'>
</audio>
...
</body>
</html>
Like most games, Snail Bait prefers to load its audio when the game starts. Snail Bait signifies that preference with the preload
attribute of the HTML5 audio
tag, meaning the browser should load the files when the game’s page loads, as you can see in Table 14.2.
Unlike most HTML tag attributes, which are essentially directives, the audio tag’s preload
attribute is a mere suggestion for the browser. The browser can arbitrarily ignore the attribute.
iOS ignores the preload
attribute for the HTML5 audio
tag because it only loads sounds as a result of direct user manipulation, such as clicking on a button. See Chapter 15 for more details on the vagaries of sound on mobile devices.
Snail Bait specifies the sound and music checkboxes shown in Figure 14.1 with the HTML listed in Example 14.2.
<html>
...
<body>
...
<!-- Sound and music.....................................-->
<div id='snailbait-arena'>
...
<div id='snailbait-sound-and-music'>
<div id='snailbait-sound-checkbox-div'
class='snailbait-checkbox-div'>
Sound <input id='snailbait-sound-checkbox'
type='checkbox' checked/>
</div>
<div class='snailbait-checkbox-div'>
Music <input id='snailbait-music-checkbox'
type='checkbox' checked/>
</div>
</div>
...
</div>
...
</body>
</html>
In the sections that follow, we discuss how to access the sound and music checkboxes in JavaScript and attach event handlers to them. See Chapter 1 for a discussion of the CSS for the HTML shown in Example 14.2. Next, let’s see how to play a game’s soundtrack.
To play the game’s soundtrack, Snail Bait starts by accessing the soundtrack’s HTML element in the game’s constructor, as shown in Example 14.3. Snail Bait also obtains a reference to the music checkbox.
var SnailBait = function () {
...
// Music.............................................................
this.musicElement = document.getElementById('snailbait-music');
this.musicCheckboxElement =
document.getElementById('snailbait-music-checkbox');
this.musicElement.volume = 0.1;
this.musicOn = this.musicCheckboxElement.checked;
...
};
The preceding code sets the music element’s volume to 0.1
(it’s a loud soundtrack), and creates a Boolean variable named musicOn
. That variable’s initial value coincides with the music checkbox element: If the music checkbox is checked, music is on, and the musicOn
variable’s value is true
.
To keep the musicOn
variable in sync with the checkbox, Snail Bait implements a change
event handler, listed in Example 14.4. That event handler sets the musicOn
variable and plays the soundtrack if the musicOn
variable is true
; otherwise, the event handler pauses the soundtrack.
snailBait.musicCheckboxElement.addEventListener(
'change',
function (e) {
snailBait.musicOn = snailBait.musicCheckboxElement.checked;
if (snailBait.musicOn) {
snailBait.musicElement.play();
}
else {
snailBait.musicElement.pause();
}
}
);
Snail Bait’s togglePaused()
method also pauses or plays the soundtrack according to what the value of the musicOn
variable is and whether the game is paused, as you can see in Example 14.5.
SnailBait.prototype = {
...
togglePaused: function () {
...
if (this.musicOn) {
if (this.paused) {
this.musicElement.pause();
}
else {
this.musicElement.play();
}
}
},
...
};
Now that you’ve seen how to access the soundtrack’s HTML element and subsequently play and pause the element’s associated audio, let’s play music in a loop.
The HTML5 audio
element has a loop
attribute that causes the browser to endlessly loop through the element’s audio, which is typically what you want for a game’s soundtrack. Unfortunately, that attribute doesn’t work in all browsers that support HTML5, so Snail Bait implements looping by hand. Fortunately, looping in JavaScript is a relatively simple matter.
When the game begins, Snail Bait starts the game’s music with the startMusic()
method, listed in Example 14.6.
SnailBait.prototype = {
...
startMusic: function () {
var MUSIC_DELAY = 1000;
setTimeout( function () {
if (snailBait.musicCheckboxElement.checked) {
snailBait.musicElement.play();
}
snailBait.pollMusic(); // Restarts the soundtrack when it stops
// by invoking this method (startMusic()).
}, MUSIC_DELAY);
},
startGame: function () {
...
this.startMusic();
...
},
...
};
The startMusic()
method waits one second and then, if the music checkbox is checked, starts playing the music. The one-second delay serves two purposes. First, when Snail Bait calls startMusic()
at the beginning of the game, the delay lets the game fade partially into view before the music starts. Second, when the soundtrack ends, Snail Bait calls restartMusic(),
which is discussed below. That method restarts the music by once again calling startMusic().
In that case, the one second delay implemented by startMusic()
provides a brief moment of silence before the soundtrack starts again.
To continuously loop over the game’s soundtrack, Snail Bait’s startMusic()
method invokes a method named pollMusic().
That method uses the browser’s built-in setInterval()
method to poll the music element’s currentTime
property, as you can see in Example 14.7.
SnailBait.prototype = {
...
pollMusic: function () {
var POLL_INTERVAL = 500, // Poll every 1/2 second
SOUNDTRACK_LENGTH = 132, // seconds
timerID;
timerID = setInterval( function () {
if (snailBait.musicElement.currentTime > SOUNDTRACK_LENGTH) {
clearInterval(timerID); // Stop polling
snailBait.restartMusic(); // Restarts music and polling
}
}, POLL_INTERVAL);
},
...
};
When the value of the soundtrack’s currentTime
property exceeds the length of the soundtrack, pollMusic()
stops polling and invokes restartMusic(),
which is listed in Example 14.8.
SnailBait.prototype = {
...
restartMusic: function () {
snailBait.musicElement.pause();
snailBait.musicElement.currentTime = 0;
snailBait.startMusic();
},
...
};
The restartMusic()
method pauses the soundtrack and sets the music element’s currentTime
property to 0
, which resets the soundtrack. Subsequently, resartMusic()
invokes startMusic()
to start playing the soundtrack once again.
To endlessly loop through its soundtrack, Snail Bait needs to know the length of the soundtrack. One way to ascertain the length of an audio file is to open it in Audacity and select the entire soundtrack, as shown in Figure 14.3.
As you can see from Figure 14.3, Snail Bait’s soundtrack runs for two minutes and 12 seconds, which equates to the 132 seconds in Example 14.7.
Other than the minor nuisance of implementing looping by hand, starting, pausing, and playing music is straightforward. Playing sound effects, however, is a little more complicated, as you’ll see in the next section.
Snail Bait has six sound effects, listed in Table 14.3.
Snail Bait plays sounds with a playSound()
method. Example 14.9 shows the various places in the game where Snail Bait plays sounds.
var SnailBait = function () {
...
this.fallBehavior = {
...
fallOnPlatform: function (sprite) {
...
snailBait.playSound(snailBait.thudSound);
},
execute: function (sprite, now, fps, context,
lastAnimationFrameTime) {
if (sprite.falling) {
if (...) {
...
}
else { // Out of play or exploding
...
if (this.isOutOfPlay(sprite)) {
...
snailBait.playSound(
snailBait.electricityFlowingSound);
}
}
}
else { // Not falling
...
}
}
...
};
this.collideBehavior = {
...
processPlatformCollisionDuringJump: function (sprite, platform) {
var isDescending = sprite.descendTimer.isRunning();
...
if (isDescending) {
...
}
else { // Collided with platform while ascending
...
snailBait.playSound(snailBait.thudSound);
}
},
processAssetCollision: function (sprite) {
sprite.visible = false; // sprite is the asset
if (sprite.type === 'coin')
snailBait.playSound(snailBait.coinSound);
else
snailBait.playSound(snailBait.pianoSound);
},
...
};
this.snailShootBehavior = { // sprite is the snail
execute: function (sprite, now, fps, context,
lastAnimationFrameTime) {
var bomb = sprite.bomb,
MOUTH_OPEN_CELL = 2;
...
if ( ! bomb.visible &&
sprite.artist.cellIndex === MOUTH_OPEN_CELL) {
...
snailBait.playSound(snailBait.cannonSound);
}
}
};
...
};
SnailBait.prototype = {
...
explode: function (sprite, silent) {
if ( ! sprite.exploding) {
...
if ( ! silent) {
snailBait.playSound(this.explosionSound);
}
...
}
},
...
};
As the preceding code illustrates, to play sounds in Snail Bait you simply invoke the playSound()
method, passing it an object representing a sound. The sections that follow show you how to implement the playSound()
method and how to define sound objects. First however, we need to discuss audio sprites.
Recall that Snail Bait puts all 63 of its images in a single sprite sheet. If each image resided in a separate file, the game would incur 62 more HTTP requests at startup, which would significantly degrade startup time. To draw individual images from the sprite sheet, Snail Bait uses simple objects to store the locations of each image in the sprite sheet. See Chapter 6 for more details.
Snail Bait does the same thing for sound effects by putting all the game’s sounds in a single file and accessing them with objects containing the location and duration of each sound in the audio sprite sheet. Figure 14.4 shows Snail Bait’s audio sprite sheet in Audacity.
Creating audio sprite sheets is a simple matter. Next, let’s define JavaScript objects that represent the individual sounds in an audio sprite sheet.
Audacity’s File → Import → Audio... menu lets you import multiple files into one. You can export the resulting file to multiple formats with the File → Export... menu.
Snail Bait’s sound objects have three properties:
• position
: Where the sound begins in the audio file (seconds)
• duration
: How long the sound lasts (milliseconds)
• volume
: A number from 0.0 (silent) to 1.0 (full volume)
You can determine the starting position and duration for a sound by selecting it in Audacity, as shown in Figure 14.5.
Example 14.10 shows the sound object definitions in Snail Bait’s constructor.
You can determine each sound’s position and duration with a sound editor such as Audacity, but the volume level for each sound—a value between 0.0 and 1.0—is best determined empirically. Sounds are recorded at different volume levels, so volume settings can vary quite a bit, as Example 14.10 illustrates.
var SnailBait = function () {
...
// Sounds............................................................
this.cannonSound = {
position: 7.7, // seconds
duration: 1031, // milliseconds
volume: 0.5
};
this.coinSound = {
position: 7.1, // seconds
duration: 588, // milliseconds
volume: 0.5
};
this.electricityFlowingSound = {
position: 1.03, // seconds
duration: 1753, // milliseconds
volume: 0.5
};
this.explosionSound = {
position: 4.3, // seconds
duration: 760, // milliseconds
volume: 1.0
};
this.pianoSound = {
position: 5.6, // seconds
duration: 395, // milliseconds
volume: 0.5
};
this.thudSound = {
position: 3.1, // seconds
duration: 809, // milliseconds
volume: 1.0
};
...
};
You’ve seen how to create audio sprites and how to define JavaScript objects that represent the audio sprite’s individual sounds, so the only remaining aspect of playing sound effects is implementing Snail Bait’s playSound()
method. That method plays sounds on multiple sound channels, as discussed in the next section.
Snail Bait frequently plays multiple sounds simultaneously, as illustrated in Figure 14.6.
In the screenshot in Figure 14.6, the runner is about to land on the button, colliding with the coin and sapphire on her way down. From top to bottom, the illustrations in Figure 14.6 show the sequence of events as Snail Bait plays four sounds that overlap. First the runner collides with the coin and plays the coins sound. Subsequently she collides with the sapphire, playing the piano sound. Finally, the runner falls on the button, which explodes two of the game’s bees in succession. With those final two explosions, all four of Snail Bait’s audio channels are momentarily busy, as shown in the bottom illustration in Figure 14.6.
To play sound effects on multiple channels, Snail Bait implements the methods listed in Table 14.4.
Here’s how Snail Bait uses those methods:
• Initialization:
1. Create audio elements (createAudioChannels())
2. Coordinate with sprite sheet loading to start the game (soundLoaded()
callback)
• Play a sound:
1. Get the first available audio channel (getFirstAvailableAudioChannel())
2. Seek to a sound’s location in the audio sprite sheet (seekAudio())
3. Play the sound at the current location in the audio sprite sheet (playSound())
In the sections that follow we look at each of the preceding methods in turn.
Snail Bait’s constructor accesses the snailbait-audio-sprites
HTML element and declares an array of audio channels. Audio channels are JavaScript objects that keep track of an audio element and the status of that element: currently playing or not. Example 14.11 lists the channels.
var SnailBait = function () {
...
this.audioSprites =
document.getElementById('snailbait-audio-sprites');
this.audioChannels = [ // 4 channels
{ playing: false, audio: this.audioSprites, },
{ playing: false, audio: null, },
{ playing: false, audio: null, },
{ playing: false, audio: null }
];
...
};
The HTML audio element for Snail Bait’s first audio channel is the snailbait-audio-sprites
element that Snail Bait declares in its HTML. The elements for the remaining channels are created programmatically by Snail Bait’s createAudioChannels()
method, which is listed in Example 14.12.
The createAudioChannels()
method uses the browser’s built-in createElement()
method to create three copies of the snailbait-audio-sprites
element that Snail Bait declares in its HTML. For each of those copies and the snailbait-audio-sprites
element itself, createAudioChannels()
sets auto-buffering to true
and adds an event handler to the element. That event handler coordinates with sprite sheet loading to start the game.
SnailBait.prototype = {
...
createAudioChannels: function () {
var channel;
for (var i=0; i < this.audioChannels.length; ++i) {
channel = this.audioChannels[i];
if (i !== 0) {
channel.audio = document.createElement('audio');
channel.audio.src = this.audioSprites.currentSrc;
}
channel.audio.autobuffer = true;
channel.audio.addEventListener('loadeddata', // event
this.soundLoaded, // callback
false); // use capture
}
},
...
};
The assignment to the src
attribute of Snail Bait’s audio
elements in the preceding code listing causes the browser to download another copy of Snail Bait’s sound effects.
Snail Bait doesn’t start until all its graphics and sound effects are loaded. To monitor the loading of sound effects and graphics, Snail Bait maintains two variables, as shown in Example 14.13.
var SnailBait = function () {
...
this.audioSpriteCountdown = this.audioChannels.length;
this.gameStarted = false;
...
};
The audioSpriteCountdown
represents the number of remaining audio files that Snail Bait must download. When the game starts, Snail Bait’s startGame()
method sets the gameStarted
boolean variable to true
.
Recall that Snail Bait attached a listener to each audio element in Example 14.12. That listener is listed in Example 14.14.
The listener decrements Snail Bait’s audioSpriteCountdown
variable. If the variable’s value is zero, and the game has not started and the sprite sheet has been loaded, then the event handler starts the game by invoking Snail Bait’s startGame()
method.
SnailBait.prototype = {
...
soundLoaded: function () {
snailBait.audioSpriteCountdown--;
if (snailBait.audioSpriteCountdown === 0) {
if (!snailBait.gameStarted && snailBait.spritesheetLoaded) {
snailBait.startGame();
}
}
},
...
};
Example 14.15 lists the callback function that the browser invokes when it finishes loading Snail Bait’s sprite sheet. That spritesheetLoaded()
function starts the game if the game hasn’t started and audioSpriteCountdown
is zero.
SnailBait.prototype = {
...
spritesheetLoaded: function () {
var LOADING_SCREEN_TRANSITION_DURATION = 2000;
this.fadeOutElements(this.loadingElement,
LOADING_SCREEN_TRANSITION_DURATION);
setTimeout ( function () {
if (! snailBait.gameStarted &&
snailBait.audioSpriteCountdown === 0) {
snailBait.startGame();
}
}, LOADING_SCREEN_TRANSITION_DURATION);
},
...
};
We’ve taken care of initializing the sounds, so now let’s play them.
Example 14.16 shows the implementation of Snail Bait’s playSound()
method.
SnailBait.prototype = {
...
playSound: function (sound) {
var channel,
audio;
if (this.soundOn) {
channel = this.getFirstAvailableAudioChannel();
if (!channel) {
if (console) {
console.warn('All audio channels are busy. ' +
'Cannot play sound');
}
}
else {
audio = channel.audio;
audio.volume = sound.volume;
this.seekAudio(sound, audio);
this.playAudio(audio, channel);
setTimeout(function () {
channel.playing = false;
snailBait.seekAudio(sound, audio);
}, sound.duration);
}
}
},
...
};
If the sound is on, playSound()
invokes getFirstAvailableAudioChannel()
to obtain a reference to the first audio channel in Snail Bait’s audioChannel
array that’s not currently playing a sound. Snail Bait has four audio channels, all of which could be busy when Snail Bait invokes the playSound()
method, so in that case, the playSound()
method prints a warning to the console.
If an audio channel is available, playSound()
invokes seekAudio(),
which pauses the audio and seeks to the sound’s position in the audio file. Subsequently, playSound()
plays the sound with its playAudio()
method. When the sound is done playing, playSound()
pauses the sound and seeks back to its starting position in the audio file in preparation for the next time the game plays the sound.
Snail Bait’s getFirstAvailableAudioChannel()
is listed in Example 14.17.
SnailBait.prototype = {
...
getFirstAvailableAudioChannel: function () {
for (var i=0; i < this.audioChannels.length; ++i) {
if (!this.audioChannels[i].playing) {
return this.audioChannels[i];
}
}
return null;
},
...
};
Recall that audio channel objects have a playing
property whose value is true
when the channel’s audio element is playing a sound. The getFirstAvailableAudioChannel()
method iterates over the audio channels and returns a reference to the first channel whose playing
property’s value is false
. If all the channels are busy, getFirstAvailableAudioChannel()
returns null
.
Snail Bait’s seekAudio()
method is listed in Example 14.18.
SnailBait.prototype = {
...
seekAudio: function (sound, audio) {
try {
audio.pause();
audio.currentTime = sound.position;
}
catch (e) {
if (console) {
console.error('Cannot seek audio');
}
}
},
...
};
The seekAudio()
method pauses the audio element and sets its currentTime
property to the sound’s position in the audio file. If those actions cause an exception to be thrown—which can happen if, for example, the audio file failed to load—seekAudio()
prints an error to the console.
Snail Bait’s playAudio()
method is listed in Example 14.19.
SnailBait.prototype = {
...
playAudio: function (audio, channel) {
try {
audio.play();
channel.playing = true;
}
catch (e) {
if (console) {
console.error('Cannot play audio');
}
}
},
...
};
The playAudio()
method plays the audio element and sets the channel’s playing property to false
.
Snail Bait lets players turn sound effects on and off with the Sound checkbox underneath the game. The game’s JavaScript obtains a reference to that HTML element, as shown in Example 14.20.
var SnailBait = function () {
...
this.soundCheckboxElement =
document.getElementById('snailbait-sound-checkbox');
this.soundOn = this.soundCheckboxElement.checked;
...
};
Snail Bait keeps the soundOn
property in sync with the Sound checkbox with the event handler listed in Example 14.21.
snailBait.soundCheckboxElement.addEventListener(
'change',
function (e) {
snailBait.soundOn = snailBait.soundCheckboxElement.checked;
}
);
In this chapter, you saw how to implement multichannel sound to play multiple sounds simultaneously with a soundtrack running continuously in the background. We accomplished all that with HTML5’s audio
element, which is less capable but more widely supported than is the more sophisticated Web Audio API.
To reduce the number of HTTP requests your game incurs at start up, you should put all your sound effects in a single file, referred to as an audio sprite sheet because it resembles graphic sprite sheets that contain images.
The code that we discussed in this chapter pertains to Snail Bait running on the desktop; in the next chapter, we discuss how to deal with audio on mobile devices.
1. Experiment with sound levels for Snail Bait’s sound effects.
2. Add a sound to the game and play that sound when the runner jumps.
3. Open the browser’s console and watch for output as you play Snail Bait. You should occasionally see warnings that Snail Bait could not play a sound. Add more channels to the audioChannels
array until the warnings disappear.
3.145.17.140