Both <audio> and <video> are welcome additions to the HTML spec. We don’t use proprietary tools (such as Flash) to display images, so why should we need them to play audio or video?

Don’t get me wrong: I’m not here to bash Flash. Without it we wouldn’t have YouTube, Vimeo, and the video revolution that has taken place over the past few years. And Flash still provides advanced video features (such as live streaming, full-screen playback, and DRM, as icky as DRM is) that have either no or very preliminary implementations as open standards.

Nevertheless, HTML5 <video> and <audio> have become almost mandatory for media delivery for one reason: iOS. Given Apple’s decision not to allow Flash on its mobile devices, the only way to embed video and audio so it’s available for iPhone, iPad, and iPod touch users is to use these new HTML5 elements.

But it doesn’t stop at iOS. As we saw in the previous chapter, in late 2011 Adobe announced it was abandoning the Flash plug-in on mobile altogether and shifting its focus to native apps and HTML5.(For more on the demise of the Flash plug-in, check out the discussion in the previous chapter.)

We’re racing toward a post-Flash future. Unfortunately, our open technologies are just not ready to replace all that Flash offers. Web standards won, and we’re going to get caught with our pants down if we’re not careful. That leaves apps to fill the gaps, taking us back to the ’90s with platform-specific software.

We’ll return to the issue of our post-Flash future shortly. For now, let’s look at the new HTML5 <audio> and <video> elements as they currently stand. While these elements are specified quite simply, issues outside the spec make their implementation interesting, to say the least. First let’s look at the basics.

Native <video> and <audio> in Action

So, how do we use these new elements? In theory, it’s pretty simple. Let’s start with <audio>.

The <audio> Element

For audio, we can use this:

<audio controls autoplay loop muted preload="auto">
   <source src="myaudio.ogg" type="audio/ogg">
   <source src="myaudio.mp3" type="audio/mpeg">
   <!-- Fallback content, such as a Flash player, here -->
</audio>

This will give us something like the Chrome example shown in Figure 10-1 (note that browsers render the <audio> element however they want, and there can be considerable difference in player size).

Let’s run through this. First, you can stick the src attribute in the opening <audio> tag if you like, instead of using the <source> element. But because of mind-bogglingly annoying issues with codec support (which we’ll delve into shortly), we often need to specify two source files for maximum HTML5 compatibility.

And that’s what the <source> element is for. Browsers work through the list of <source> elements until either they find a file format they support or (if we’ve included one) they get a fallback option—perhaps a link to the file or (more commonly) a Flash media player. (See the following resources for a tutorial on implementing a Flash fallback.)

Both <audio> and <video> are implemented in a backward-compatible way, insofar as older browsers (such as IE8) will ignore the <audio> and <source> elements altogether (IE8 just sees those elements as generic tags, like <mymadeuptag>). This means whatever fallback content we include will be visible to the older browsers but ignored by the modern ones.

We can also use scripting to give a Flash fallback to browsers that support the <audio> (or <video>) elements but not the codec we’ve used. We’ll look at media players that do the heavy lifting for us at the end of this chapter.

<source src="myaudio.mp3" type="audio/mpeg">

The <video> Element

For video, we can use this:

<video controls autoplay loop muted preload="auto"
poster="myvideobackground.jpg" height="250" width="300">
   <source src="myvideo.webm" type="video/webm">
   <source src="myvideo.mp4" type="video/mp4">
   <!-- Fallback content, such as a Flash player, or a link to the file here -->
</video>

(If you want to see what’s really involved with <video> implementation, including browser issues, setting MIME types, and more, see Kroc Camen’s wonderfully thorough “Video for Everybody” article: http://camendesign.com/code/video_for_everybody.)

As you can see, it’s a similar setup to the <audio> example. In fact, the controls, autoplay, loop, muted, and preload attributes all behave the same way. But they’re not quite the devil’s tools as they are for audio. Here we could have an autoplay, looping video advertisement that’s muted until the user decides otherwise.

And just like <audio>, we have to deal with codec support issues by specifying multiple video files for maximum HTML5 compatibility. We use the <source> tag to give browsers a list of video files, and they use either the first one they support or the fallback content (such as Flash Player). The type attribute gives browsers a hint as to which file they should try to play. We’ll discuss this after looking at the codec situation.

The <video> element has several of its own unique attributes. The main one is poster, which is the static image that’s displayed until the first frame of the video is available. This may be only for a second or two in some cases, but on mobile devices (such as iOS) the poster is shown until the user initiates playback.

(At least that’s the theory. IE’s handling of the poster image is quirky for versions 9 and 10, as Ian Devlin found: www.iandevlin.com/blog/2011/12/html5/the-problem-with-the-poster-attribute. The bug appears to be fixed in IE11. Older versions of iOS and Android had problems as well, but they’re largely unused at this point, while IE9 and 10 continue to have significant percentages of the market. Speaking of Ian Devlin, he has a great post on how to implement poster in newer browsers here: www.iandevlin.com/blog/2013/03/html5/html5-video-and-background-images.)

The height and width attributes are also specific to the <video> element. But again, because the <video> element is just another bit of HTML, it can be styled and manipulated with CSS, including advanced CSS3. You can transform and animate the video itself, add shadows, and so on. This is one of the coolest things about <video> being just another element in the DOM. You can even use the <canvas> element to manipulate your video source, as discussed here: http://html5doctor.com/video-canvas-magic/.

<track kind="subtitles" src="moviecaptions.en.vtt" srclang="en" label="English">

Codecs: What to Do?

Phew! So, where does that leave us?

If you want maximum native HTML5 support—including Opera—you need to store two copies of your audio (MP3 and Ogg Vorbis) and potentially three copies of your video files (H.264, WebM, and Ogg Theora for legacy support in Firefox 3.x). Having to encode and store so many different versions is a royal pain in the ass, but there you go.

Alternatively, you can do one of the following:

• Use MP3 for audio, which works natively in all of the modern incarnations of major browsers on desktop and mobile (except Firefox for Macintosh, but this is coming soon, and Opera).
• Use H.264 for video, which works natively in iOS, Safari, Firefox, and IE9.
• Use a media player (or script your own) that uses Flash Player for older and non-codec-supporting devices, given Flash can play MP3 and H.264 in any Flash-enabled browser or device.

Given we need H.264 for iOS (and mobile in general), this scenario—which I would wager will be the most common—means Opera (and potentially Chrome someday if it really drops H.264) will end up getting Flash. That’s right—those who stood up for free, open software will end up with proprietary, closed Flash. That is more ironic than a hipster’s mustache.

(Alternatively, you can, as mentioned, double encode your video and provide a separate WebM file in a secondary <source> element for broad native HTML5 support in modern browsers.)

Reality Bites

The reality may be a little different, though. For one, Google has yet to drop H.264 support, and Firefox finally added it after much debate. Can Opera’s support really be that far off? It’s difficult to imagine Opera holding out indefinitely. The practical implication for this is encoding media in H.264 (with a Flash fallback for older browsers) and MP3 (ditto) will be sufficient in the long term.

Video Types…Oh My

Now that you (ideally) understand the complexity of the codec situation and the way browser support for each codec differs, let’s look at how we can tell browsers which codecs we’re using so they can make smart decisions about which video they load.

That brings us to the one video attribute we haven’t discussed—the type attribute on the <source> element. Here’s an example:

<source src="myvideo.mp4" type="video/mp4">

This attribute tells the browser what container and codec is used for the video specified in the src attribute. In the previous example, we’ve specified only what container format is used by listing its MIME type (that is, media format type). The MIME type shown earlier tells the browser “This file is a video using the mp4 container format.” Container formats like mp4 are a little like zip files, in that they are simply a container for the actual video and audio files, which are encoded with specific codecs and wrapped up to make the final video file. (The type attribute can also be used on the <video> element itself, not just a nested <source> element, if you’re using only one file. The same goes for <audio>.)

The information we put in the type attribute is just a hint to the browser, but it’s not necessary for browsers to play the video. What is necessary is to edit your .htaccess file on Linux or similarly configure IIS to make sure your server sends this file with the right MIME type, as the instructions describe at http://mediaelementjs.com/ or at http://technet.microsoft.com/en-us/library/cc725608(v=ws.10).aspx and are covered elsewhere (for example, the “Video for Everybody” article mentioned earlier).

We can also specify both the container format and the codec used. Here’s an example:

<source src="myvideo.mp4" type='video/mp4; codecs="avc1.42E01E, mp4a.40.2"'>

Here we’ve specified the container format and the codecs for both the video and the audio in the source file. (The video codec is a flavor of H.264, along with the AAC audio codec. Note we also have to use single quotes for the type attribute here because the codecs parameter uses double quotes.)

What’s the point of all this? Well, specifying the type attribute means the browser doesn’t have to start downloading every listed file just to check whether it can play it. It can scan the markup and potentially start preloading the video file it supports. Leaving it out, however, isn’t the end of the world. The following is from “The State Of HTML5 Video” by LongTail Video, written in 2012 (www.longtailvideo.com/html5/):

Note the Android 2.2 compatibility issue. The comment about codecs suggests that from LongTail’s testing we (by and large) need to specify only the container and the browser will have a go at loading it.

Querying Supported Video Types with JavaScript

We can also query the browser using the <video> JavaScript API and its canPlayType() method to see which formats the browser supports. For example, with the codecs we specified earlier (avc1.42E01E, mp4a.40.2), browsers that support these formats (Safari and IE9+) will respond with probably, which is as close to “Yes, we support this file” as we get in HTML5.

If we specify only the container format (video/mp4), Safari and IE9+ (for example) respond with maybe because they know they can read that container format but don’t know what codecs lie inside. Browsers that don’t support a given container or codec format just return a null string.

Things get awfully complicated, though. Here are the three variables we have to deal with:

• Browser response: Because of the complexities of encoding media (and especially video), browsers only know for sure that they can’t play formats they don’t understand. In those cases, they return a null string (when queried through the canPlayType() method). Beyond that, the HTML5 spec says they must return either maybe or probably depending on the browser’s confidence it can play a certain file based on the information we’ve provided.
• Containers and codecs: There are container formats such as mp4, and there are the actual codecs or, more accurately, flavors of the actual codecs (such as avc1.42E01E) that can be queried.
• Browser support: Finally, as we’ve seen, the codec support from the major browsers is a pretty complex situation.

Therefore, we have multiple browsers supporting multiple container/codec varieties (for both audio and video) and giving one of three responses. Thankfully, the WHATWG maintains a table of browser responses so we can see which response for a given container format, or container and codec combination, we should get from a given browser: http://wiki.whatwg.org/wiki/Video_type_parameters#Browser_Support. Microsoft has a small script demonstrating how this works here: http://msdn.microsoft.com/de-de/library/hh325437(v=vs.85).aspx.

Audio and Video Media Players to the Rescue

What a mess.

Fortunately, people have written tools to take the pain out of serving the right video (or audio) to the right browser. These media players hold your hand through the whole codecs mess and legacy support issues, provide plenty of customization options, and generally smooth out the whole implementation process.

Here are a few examples.

http://mediaelementjs.com

http://videojs.com

http://flowplayer.org

Other Flies in the HTML5 Video Ointment: DRM, Streaming, and Full-Screen Video

We’ve got basic <video> and <audio> embedding down, but even with our handy, ready-to-go media players and their Flash fallbacks, there are still some significant missing (or very immature) features that HTML5 video lacks but Flash supports. As codecs continue to mature, there will be pressure on browser makers to support all of the important features of HTML5 video and audio sooner rather than later. When it comes to DRM, that may not be a good thing.

DRM

In February 2012, representatives from Google, Microsoft, and Netflix submitted the Encrypted Media Extensions v0.1 draft proposal (http://dvcs.w3.org/hg/html-media/raw-file/tip/encrypted-media/encrypted-media.html) to the W3C’s HTML Working Group. The abstract for the proposal said this:

That is, this would provide a mechanism for DRM to happen on top of HTML5, for both <audio> and <video>, by extending the JavaScript API (the HTMLMediaElement interface). The HTML5 editor, Ian Hickson, responded with the following (http://lists.w3.org/Archives/Public/public-html/2012Feb/0274.html):

Hickson also stated “DRM is evil” and then further elaborated on his reasons for rejecting the proposal outright here: https://www.w3.org/Bugs/Public/show_bug.cgi?id=10902#c24. Mozilla also expressed concerns, given that DRM and an open source browser are generally mutually exclusive (see this Ars Technica article for more: http://arstechnica.com/business/news/2012/02/unethical-html-video-copy-protection-proposal-criticized-by-standards-stakeholders.ars).

There is one important difference between DRM for video and DRM for audio, though, and that’s streaming. We have a culture of streaming (and renting) video, in a way that we didn’t with audio when the music DRM wars were raging. DRM sucked when it was applied to music you bought because if the DRM platform died (and they did), so did your music collection. The temporal nature of streaming media alleviates these concerns, but only to a point. (And if it can be implemented for streaming media, it’s not hard to imagine the media majors insisting on DRM for purchased content, too.)

These issues around streaming “protected” content explains Netflix’s interest in seeing some sort of DRM available when using web standards. Netflix was keen to use HTML5 (in the broad “web platform” sense) where it can (as it discusses here: http://techblog.netflix.com/2010/12/why-we-choose-html5-for-user.html), but Netflix obviously felt it needed some kind of DRM system in place to stream the content it licenses with HTML5. (They also need an actual streaming protocol, as we’ll see next.) Some at Google and Microsoft evidently feel this is a necessity too. (Note, though, that Hickson also works for Google, so it’s not a company-wide position.)

This may be another case where the post-Flash Web results not in an open standards utopia but in a return to platform-specific apps. That said, given the cost of implementing DRM, that may be a price the standards movement is willing to bear.

Debate raged on the W3C mailing lists, with a summary here: http://lists.w3.org/Archives/Public/public-html/2012Mar/0087.html.

But that debate may have always been moot. In March 2012, Philippe Le Hegaret from the W3C wrote the following (http://lists.w3.org/Archives/Public/public-html/2012Mar/0097.html):

That is, we’re going to implement DRM with or without you because our paid-up members want it. Ominous stuff.

By late 2013 it appears the fight against DRM is over, and DRM won.

First, the Encrypted Media Extensions proposal is in full working draft. It was first publically released in May 2013 with much controversy (www.w3.org/blog/2013/05/perspectives-on-encrypted-medi/), particularly from the Free Software Foundation (you can read the recommendation against EME here: www.fsf.org/news/coalition-against-drm-in-html). The editor’s draft can be found here: https://dvcs.w3.org/hg/html-media/raw-file/tip/encrypted-media/encrypted-media.html.

Second and more importantly, Chrome and IE11 already support EME. As everyone should remember by now, the browser makers make the rules. Without them, the spec is a work of fiction. Despite the controversy, it appears like this fight is over, and DRM won this round. It’s unlikely to make it into the final W3C spec by 2014 since it’s not in revision status yet, but once the browsers are on board, the spec is just a formality.

Third, Netflix is streaming over HTML5 in some cases, using EME. As of late 2013 it’s not its mainstream distribution channel, but it’s available. Netflix has been a big supporter of EME. When you have browser support and major corporations already using DRM in HTML5, it’s hard to imagine a universe in which the spec and other browsers don’t get on board very quickly.

Streaming

For a long time, streaming video over HTML5 was one of the largest challenges to moving off of Flash. Flash provided a very stable streaming media solution for decades and was the entrenched solution of choice, and the HTML5 spec wasn’t quick in coming up with a better alternative. Ars Technica summed it up well in 2011 through an excellent article titled “The trials and tribulations of HTML video in the post-Flash era” (http://arstechnica.com/business/news/2011/11/the-trials-and-tribulations-of-html-video-in-the-post-flash-era.ars):

A streaming standard has been in the works for some time, called Dynamic Adaptive Streaming over HTTP (DASH), which has the support of Microsoft , the BBC, and others, but there’s still a way to go there before support materializes. DASH is also codec agnostic—it doesn’t resolve the codec impasse we discussed earlier. (For more, see “What is MPEG DASH?”: www.streamingmedia.com/Articles/ReadArticle.aspx?ArticleID=79041.)

Apple currently has its own streaming protocol, HTTP Live Streaming (HLS), which is used to deliver content to its iOS devices. Google added support in Android 3.0+, and it allows encrypted data and works with third-party DRM solutions. See “What is HLS (HTTP Live Streaming)?” for more: www.streamingmedia.com/Articles/Editorial/What-Is-…/What-is-HLS-%28HTTPLive-Streaming%29-78221.aspx.

Regardless of what protocol one uses to stream, by late 2013 the W3C is in working draft for the Media Source Extensions (MSE) API, which allows a page to define support for media streams through JavaScript (www.w3.org/TR/media-source/). The spec currently supports several different types of streams and the ability to define additional through JavaScript libraries (like DASH through dash.js). The major players are all planning to support it. Once MSE is in place and fully supported, then streaming is solved, and the provider must simply provide any necessary JavaScript libraries to translate the stream and send the data in a codec supported by the browser to provide real streaming video. IE11 and Chrome already support it, which is how Netflix can stream in HTML5 through extensions on those browsers.

Fullscreen API

Finally, one of the most common things we do with Flash video is make it full-screen. This wasn’t possible in HTML5 until recently, when the Fullscreen API spec moved ahead in the W3C (http://dvcs.w3.org/hg/fullscreen/raw-file/tip/Overview.html). All of the major desktop browsers are moving ahead in anticipation of the spec. Mobile browsers aren’t on board yet, but most of them go full-screen for all video elements by default anyway. Interestingly, the Fullscreen API can make any element full-screen, including (for example) the <canvas> element, and it could be used for a full-screen reading mode too, for example.

For more on the Fullscreen API, see the following:

• Mozilla’s guide to using the Fullscreen API is here: https://developer.mozilla.org/en-US/docs/Web/Guide/API/DOM/Using_full_screen_mode.
• The Mozilla Hacks blog has a tutorial (including styling information) at http://hacks.mozilla.org/2012/01/using-the-fullscreen-api-in-web-browsers/ and a demo at http://robnyman.github.com/fullscreen/.
• Keep an eye on http://caniuse.com/#feat=fullscreen for browser support stats and more resources.

Is HTML5 <audio> Ready for Games?

One final note on <audio> and its potential for gaming.

Dominic Szablewski posted an epic (and profanity-laced!) rant on the state of HTML5 audio, in relation to developing HTML5 games. It’s well worth a read for entertainment value and gives some indication of the (im)maturity of HTML5 audio support, particularly for interactive purposes. Szablewski says this:

(See the article for more: www.phoboslab.org/log/2011/03/the-state-of-html5-audio. I love the Scumbag Steve hat.)

In 2013 the story is mostly still the same. WebAudio is almost ready, the codecs seem to be coalescing, but game developers still run into problems. In late 2013 the game development blog Indiegamr summed it up this way in a post titled “The State of Audio in HTML5 Games” (http://indiegamr.com/the-state-of-audio-in-html5-games/):

The short answer? No, it’s not really ready for games.

In the meantime, cross-platform libraries, perhaps with Flash fallbacks built in, are required to get the job done. Here’s a short list of libraries to look into:

• SoundJS, part of the CreateJS suite championed by Adobe, Microsoft, and AOL, provides a Flash fallback and handles cross-browser issues gracefully: www.createjs.com/#!/SoundJS.
• The HowlerJS framework is open source and freely available under the MIT license. It has no Flash fallback, so it won’t support older browsers easily: https://github.com/goldfire/howler.js/.
• SoundManager has long been the standard of JavaScript audio. It provides a Flash fallback if required: www.schillmania.com/projects/soundmanager2/.

Until IE and others get on board with the WebAudio API, we’re stuck with hiding Flash in the background to serve audio. Welcome to the modern Web.

Wrapping Up

Native support for <audio> and <video> is welcome and necessary for mobile devices in particular. But keep in mind that the technology is still somewhat immature (especially on Android).

The codec issues will not be resolved any time soon, and iOS will still need H.264 for video. So, we need to tread carefully. Don’t assume it will work flawlessly just because it’s in the spec.

At the time of this writing, we’ve come a long way. Just a few years ago the codecs were completely fractured and the spec still totally up in the air on basic things such as full-screen video or whether an audio API would ever be included, let alone controversial topics such as DRM. Yet today we have all three included in the working spec, and a majority of the browsers are on board. The codec wars feel like they’re winding down, despite Mozilla’s push for Opus in audio, and it seems like stability is just around the corner.

But we’re not there yet. Until every major browser supports a small set of codecs (let alone a single one) for audio and video, a single API for audio, and consistent streaming and DRM capabilities, audio and video will remain difficult areas for developers. Commitment to the open Web means supporting multiple ways of doing everything. JavaScript frameworks provide some help, and we’re still stuck with legacy support through Flash for a good while regardless.

My advice? Find a few good components that do the heavy lifting. Then, keep an eye on what your favorite media player supports, and let it be your guide for which technologies to bet on in the future.