Chapter 10. Audio and Video
IN THIS CHAPTER
- Playing Audio and Video with
MediaElement - Controlling Audio and Video with JavaScript
- Using Expression Encoder
Silverlight’s support for audio and video is one of its key features in version 1.0. You can get outstanding video quality with Silverlight and make it go full screen—up to 720p high-definition quality on a reasonably-equipped computer—although its performance greatly depends on the computer viewing it. Video can be seamlessly intermixed with any other Silverlight content and transformed just like any other element.
Fortunately, due to the hard work that the Silverlight team had to do behind-the-scenes to enable this rich media functionality on all supported operating systems, taking advantage of this support is actually very straightforward. This chapter is all about what you can do with MediaElement, the versatile element that hosts all audio and video in Silverlight. After looking at what you can do with MediaElement in XAML and in JavaScript, this chapter explains how Expression Encoder can save you enormous amounts of time and effort and help you produce a truly professional result.
What audio and video formats are supported by Silverlight?
Silverlight 1.0 supports five video formats and four audio formats. The following table lists the video formats:
VC-1 is a widely supported video format standardized by the Society of Motion Picture and Television Engineers. It has three profiles—Simple, Main, and Advanced. Windows Media Video 9 Advanced Profile is Microsoft’s implementation of the VC-1 Advanced Profile specification. The older Windows Media Video 9 (WMV3) format has three profiles—Simple, Main, and Complex—and all but the latter are compliant with the corresponding VC-1 profiles.
The four supported audio formats are Windows Media Audio (WMA) 7, 8, and 9, as well as MP3. Not every MP3 file can be played, however. Like most MP3 players, only bit rates up to 320kbps are supported.
Note that several Windows Media formats are not supported by Silverlight, such as WMA Voice (a format optimized for speech) or WMA Professional (which can support 7.1 channel surround sound instead of the mono/stereo support of standard WMA). In addition, Windows Media Video that uses MP3 audio is not supported, despite the fact that these two formats are supported independently!
If you have existing audio or video in an unsupported format, it shouldn’t be hard to transcode it into a supported one using any number of existing audio/video tools.
What’s the difference between audio and video in Silverlight versus audio and video in WPF?
Both WPF and Silverlight have a MediaElement element for embedding media in a user interface, and WPF supports all the formats that Silverlight does. Because audio and video is such a core scenario for Silverlight 1.0, the Silverlight MediaElement has been given extra capabilities that the WPF MediaElement does not have, whether it’s something sophisticated, such as the capability to retrieve metadata or timeline markers, or something simple, such as an AutoPlay property. The VideoBrush described in Chapter 5, “Brushes and Images,” is also unique to Silverlight, although you can accomplish the same effect in WPF with the more general-purpose VisualBrush.
Although the exposed functionality is very similar between the two technologies, the underlying implementation is much different. WPF’s audio and video support is built on top of Windows Media Player, but Silverlight’s is not. Depending on Windows Media Player was not an option for Silverlight due to its cross-platform support and requirement for being a small download.
Playing Audio and Video with MediaElement
MediaElement, seen briefly in previous chapters, is designed for playing audio and video, whether it’s a normal media file that needs to be downloaded or streaming media. This section examines the various ways to retrieve and display content with MediaElement.
Does Silverlight support Digital Rights Management (DRM)?
Silverlight 1.0 does not, but the next version of Silverlight should support Microsoft’s PlayReady technology for providing DRM capabilities.
The Source Property
You can set MediaElement’s Source property to the URL of an audio or video file, and it plays automatically as soon as the content is loaded. Therefore, annoying users of your web page with automatic background music (similar to Internet Explorer’s BGSOUND element) is as simple as loading XAML content such as the following:
Or, playing a video automatically in a rectangular region is as simple as
As with the Source on Image or ImageBrush, MediaElement’s Source can be an absolute URL pointing to any domain or a relative URL (relative to the host HTML file). And if you have the media file in the same directory as an HTML file on the local file system, you can point the Source to that file using just the filename. The content renders at its natural size by default.
MediaElement supports three protocols for the URL: HTTP, HTTPS, or MMS (Microsoft Media Services). HTTPS can only be used if the hosting page is served via HTTPS, and HTTP or MMS can only be used if the hosting page is served via HTTP. The Source can also be pointed to an Advanced Stream Redirector (ASX) playlist, which is a simple XML file that combines multiple media files into a single entity. For example, the following ASX file combines two videos into a single source:
If you point MediaElement’s Source to this file, first.wmv starts playing and then second.wmv plays as soon as first.wmv finishes. (You can also open ASX files with Windows Media Player and watch the entire sequence, jump to later movies in the list, and so on.) Examining the format of ASX files is outside the scope of this book, but there are many online resources describing all the options.
Visual Effects
To use video in a nonrectangular region, you could either paint a shape with a VideoBrush that points to a named MediaElement, as shown in Chapter 5, or you could just use a single MediaElement but set its Clip property to an arbitrary geometry, using the techniques from Chapter 3, “Shapes, Lines, and Curves.” Because MediaElement is a UI element, you can not only set its Width, Height, and Clip, but also set its Opacity or OpacityMask, give it any number of RenderTransforms, use Canvas.ZIndex on it, and so on.
The flexibility of video in Silverlight can seem unusual to people used to dealing with video as a “black box,” but it really can be treated just as any other shape, image, or text. The following XAML, rendered in Figure 10.1, places two instances of a video on top of each other, both half-transparent, both clipped with a circle, and one rotated 180°:
Figure 10.1 Clipped, rotated, and half-transparent video inside two MediaElements.
Audio-Specific Features
MediaElement has a few features for controlling audio, whether it’s audio-only content or the audio that’s part of a video:
- You can set the
Volumeproperty to any value from0to1. The default value is0.5. - You can mute the audio by setting its
IsMutedproperty totrue. - You can shift the balance toward the left or right speaker by setting its
Balanceproperty to a value between-1and1.-1means that all the audio is sent to the left speaker;0(the default) means that all the audio is sent to both speakers; and1means that all the audio is sent to the right speaker. - If the media has multiple audio tracks, you can select a different one by setting the
AudioStreamIndexproperty (whose default value is0). TheAudioStreamCountproperty tells you how many tracks exist.
Does Silverlight video support Chroma Key (also known as bluescreen or greenscreen) effects?
No, Silverlight has no automatic support for making certain colors in video content transparent or swappable with something else. However, depending on your needs, you could always use a separate video production tool to apply the desired effects to the video ahead of time. Additionally, you can achieve certain types of special effects by overlaying non-video content on top of the playing video. The timeline marker support described later in the chapter can help you keep non-video effects in-sync with video to provide a single seamless experience.
Video-Specific Features
MediaElement has three properties specific to video: NaturalVideoWidth, NaturalVideoHeight, and Stretch. NaturalVideoWidth and NaturalVideoHeight, which give the original dimensions of the video (or 0 for audio), can be helpful for making decisions on how to display the video. If you don’t give MediaElement an explicit Width and Height, it is given these dimensions.
If you give MediaElement an explicit Width and Height that don’t match the natural values, Stretch controls how the video fills the space. Stretch works the same way as on shapes, Image, ImageBrush, and VideoBrush. Figure 10.2 demonstrates the four Stretch values with the following XAML:
Figure 10.2 MediaElement with four different Stretch settings.
The Rectangle is there just to help demonstrate the Stretch of Uniform, as it’s the only setting in which the entire 150x150 region is not filled with video.
Progressive Download Versus Streaming
Silverlight supports both HTTP-based progressive download and streaming of audio and video files. This happens automatically when you set the Source of a MediaElement, and the choice of one versus the other is determined by the URL and the media it points to. If the Source uses the HTTP or HTTPS protocol, Silverlight attempts to perform a progressive download but falls back to streaming if that fails. However, if the Source uses the MMS protocol, Silverlight attempts to stream the file but falls back to progressive download on failure.
Tip
For maximum performance, you should avoid setting an explicit Width or Height on MediaElement. If its natural size is not desirable, you should explore the possibility of re-encoding the video file(s) with the ideal size.
Progressive download simply means that although the media file is downloaded to the client via a standard HTTP GET request, playback can begin fairly instantly. Users can rewind and fast forward the media at any time, but only within the subset of content that has already been downloaded.
Streaming, on the other hand, refers to an efficient protocol that keeps an active connection between the client and server. This enables users to rewind and fast forward to any part of the content at any time. Most importantly (for the content providers, anyway), streaming is often much less expensive than a progressive download because only the portions of media actually viewed or heard (plus a little bit of buffer, controlled by MediaElement’s BufferingTime property) are delivered by the server. Compare this to progressive download, for which the file is retrieved as quickly as the web server can send the bytes. Note that Silverlight’s streaming support requires Windows Server.
How does the progressive download feature interact with Silverlight’s downloader object?
The explicit downloading enabled by Silverlight’s downloader object is separate from the implicit downloading done by MediaElement. MediaElement does contain a SetSource function (just like Image and ImageBrush) that enables it to work with explicitly downloaded audio or video files. However, unless you want to download the media simultaneously with other files in a .ZIP file, or unless you want to guarantee that the entire media file has been downloaded before playback begins, there’s no reason to choose explicit downloading over the implicit progressive downloading.
Much like Silverlight’s generic downloader object, MediaElement contains a DownloadProgressChanged event and corresponding DownloadProgress property that enables you to customize your interaction with the progressive download process. You could implement a user interface similar to sites such as YouTube, for example, which show a background progress bar that makes it easy to compare the amount of video downloaded to the current position in the video. DownloadProgressChanged is raised whenever download progress changes by at least .05 percent from the time the download begins to the time it is complete. The DownloadProgress property starts at 0 and ends at 1.
Selecting a Stream in a Multiple Bit Rate File
Progressive download and streaming differ in how they handle a multiple bit rate (MBR) file. An MBR file contains multiple encodings of the same content, each with a different bit rate. The idea is to provide a tailored experience depending on the bandwidth available. Streaming honors this and selects the appropriate stream based on current conditions. Progressive download, however, always uses the stream with the highest bit rate.
For streaming media, the DownloadProgressChanged event is raised only once (with DownloadProgress set to 1) when the media is opened and buffered. Rather than using the DownloadProgressChanged event and the DownloadProgress property, the relevant event and property for streaming media is the BufferingProgressChanged event and the BufferingProgress property. They work the same way as the download event and property.
Controlling Audio and Video with JavaScript
MediaElement has a number of functions, properties, and events that make it possible for JavaScript to interact with audio or video in very rich ways. The functionality provided by these members range from the obvious (such as Play and Stop functions) to the not-so-obvious (such as the Attributes or Markers properties).
Changing the Media’s State
JavaScript can call MediaElement’s Play and Stop functions, as well as Pause if the read-only CanPause property is true. (Silverlight sets CanPause to false when the MediaElement contains streaming media, which doesn’t support pausing. When CanPause is false, calls to Pause fail silently.) Calls to these functions are always asynchronous, so if you called Stop immediately after Play, none of the content would be played.
A single instance of MediaElement can play multiple audio or video files, but only one at a time (either by swapping the Source or pointing the Source to an ASX playlist). You can play multiple audio or video files simultaneously, but to do that, you need a separate instance of MediaElement for each file.
A CurrentState property tracks the state of the media, and a corresponding CurrentStateChanged event is raised whenever its value changes. The values for CurrentState are Opening, Buffering, Playing, Paused, Stopped, Closed, and Error.
As mentioned at the beginning of the chapter, content in a MediaElement plays automatically as soon as it is loaded. This might not always be desirable, of course. You might want instances of MediaElement to contain sound effects to be used in response to specific user actions, such as clicking or hovering over an element. There are a few options for preventing the automatic playing behavior. For example, you could call Stop in a Silverlight onLoad event handler, or you could wait to set the Source of the MediaElement until you’re ready for the media to be played. But the best option is to take advantage of MediaElement’s AutoPlay property, which is set to true by default. Simply set it to false, and the media won’t be played until you call the Play function.
Basic Media Events
MediaElement has all the standard UI element events (such as MouseEnter, MouseLeftButtonDown, and so on), and it has a number of specialized events mentioned throughout this chapter. Three core media-related events that are good to be aware of are MediaOpened, MediaEnded, and MediaFailed.
The MediaOpened event is especially useful because some of MediaElement’s properties are not valid until this event is raised. These properties are AudioStreamIndex and AudioStreamCount, NaturalVideoWidth and NaturalVideoHeight, NaturalDuration, CanPause, and Markers. If you need to interact with any of these properties, you should handle the MediaOpened event and use them inside this event handler.
The MediaFailed event provides a way to provide special error-handling logic specific to MediaElement. By default, any failures are bubbled up to the Silverlight control’s onError handler.
Tip
To create continuously looping background audio or video, you can handle the MediaEnded event and call Stop (to reset the position to the beginning) then Play inside of it.
Positioning the Audio or Video
MediaElement defines two properties that make it easy to jump to arbitrary parts of audio or video content or to implement a user interface that allows users to do this. Its Position property always returns the current spot in the media, whether it’s playing, paused, or stopped. Position can also be set to a new value to change the current position (if the media format supports seeking). Position accepts the same syntax as an animation’s Duration for representing time. Therefore, to set the Position to 10 seconds from the beginning, you would set it to the string "0:0:10". Attempting to set the Position to a negative time or a time greater than the duration of the media is treated the same as setting the Position to zero.
The string syntax for setting Position is enabled by a type converter, so when you retrieve the value of Position, you don’t get the same string back. Instead, you get an object with a numeric Seconds field. This makes it easier to do math, but if you want to display this value, you’ll probably want to do a bit of work to format it nicely. For example:
If you need to know what the duration of the media is, MediaElement has a NaturalDuration property that returns this length. Like Position, the information is returned as an object with a Seconds field, so doing math with the two properties is straightforward. Note that for streaming media, NaturalDuration is always null because the exact length is unknown (perhaps even to the server, as with live broadcasts). Also, if the Source is an ASX playlist, NaturalDuration and Position are associated with the current media only and get reset as each new item in the playlist begins.
Using Timeline Markers
Sometimes audio or video files have built-in timeline markers that contain custom information associated with specific points in time. An example of this would be markers that designate when a new chapter begins in a movie or even closed captioning text. (You can easily embed timeline markers in your own content, too, thanks to tools such as Expression Encoder.) Silverlight’s MediaElement provides an easy way to retrieve these Windows Media–based timeline markers so that you can provide a custom, perhaps interactive, experience.
Windows Media defines two main types of timeline markers—one is just called a marker, and one is called a script command. They are essentially the same thing—just a way to associate custom text with time-based positions—but their intent is different. Script commands are suggestions for the host to run custom code at that point, whereas markers tend to convey static data. Technically, the only difference is that a marker has one text value, whereas a script command has two text values (a type and a command, sometimes called a param).
MediaElement hides the difference between these types of timeline markers and exposes them via a single Markers property. This property is a collection of marker objects with three properties:
Time—The position (relative to the beginning of the media) of the marker.Text—The main text value from the marker.Type—The type of the marker, if it’s a script command. (If it’s not a script command,Typeis set to the same value asText.)
You can retrieve and process all markers as soon as the media is opened, which can be useful for showing a display of all chapters. For other types of markers that might be easier to process on-demand, you can leverage MediaElement’s MarkerReached event. MarkerReached is raised whenever MediaElement’s Position matches the Time of a marker during playback. The sender passed to a MarkerReached event handler is the MediaElement, and the eventArgs object is the marker (with the Time, Text, and Type properties).
Warning
Some markers are only accessible via the MarkerReached event!
Windows Media supports timeline markers encoded as a separate stream, rather than embedded in the media file’s header. This low-level detail normally wouldn’t matter, but MediaElement treats these separate-stream markers differently. MediaElement raises the MarkerReached event for separate-stream markers as expected, but they are never placed in its Markers collection.
How can I get metadata associated with audio or video, such as Album Artist or Genre?
MediaElement has a read-only Attributes property that gives a collection of metadata objects, each with a Name property and a Value property. This only exposes ASX metadata, however. Retrieving embedded metadata is not supported.
To get all the ASX metadata associated with audio or video content, you could loop through the collection as follows, assuming that element is set to an instance of a MediaElement:
Because Attributes is a read-only collection, it has Count and GetItem functions but no Add or Remove functions. However, it has a special GetItemByName function that enables direct retrieval of a specific attribute. For example:var attribute = element.Attributes.GetItemByName("Genre");
If the attribute does not exist, GetItemByName returns null.
Building a Media Player User Interface
With all the functionality that MediaElement provides, you could build a fairly sophisticated Silverlight-based media player that rivals the core functionality of a program such as Windows Media Player. If you leverage Expression Encoder, discussed in the next section, you won’t need to implement a media player from scratch. But to give you an idea of how it can be done, Listings 10.1 and 10.2 show all the XAML and JavaScript necessary to construct the very simple media player pictured in Figure 10.3.
Listing 10.1 XAML for a Simple Media Player
Figure 10.3 A basic hand-crafted Silverlight media player.
The Canvas contains five children: the MediaElement, a TextBlock for displaying the current position and total duration, and three “buttons” for playing, stopping, and pausing. Each button is just a Canvas with a TextBlock and a background Rectangle.
Listing 10.2 JavaScript for the Simple Media Player
The handler for MediaElement’s MediaOpened event kicks off a timer that updates the positionText TextBlock every second. (The “empty Storyboard” technique from the preceding chapter would be more precise and reliable, but setInterval and clearInterval are used here for simplicity.) It also stores a few items in global variables to be used by other functions. The MediaEnded event handler simply clears the timer. The updatePosition function that’s called every second displays the values of Position and NaturalDuration with the help of the formatSeconds function that turns the raw number of seconds into a readable display. Finally, the three button event handlers do nothing more than call the corresponding Play, Stop, and Pause functions on MediaElement. That’s all there is to it.
Using Expression Encoder
Microsoft’s Expression Encoder is a new tool that enables audio and video professionals to create stunning media content to use with Silverlight. The core job of Expression Encoder is to import an audio or video file, and then export it in a different format. It has a number of features to help you choose an encoding and make the right trade-offs between a large number of settings, such as an amazing A/B compare feature that compares your chosen settings in real-time. It also has a command-line interface for doing batch (or even server-side) processing.
But it allows you to do much more than the core task of encoding. You can do basic stitching and trimming; you can add timeline markers and take advantage of special support for chapter thumbnails; you can add metadata; you can include live sources (such as a web cam or DV cam); and you can even switch between multiple sources (live or prerecorded) during a live broadcast, as if you’re running your own television studio.
Another extremely handy feature is the ability to produce not just audio or video content, but a fully functioning Silverlight-based media player complete with buttons for playing, stopping, pausing, rewinding, fast forwarding, muting, and so on. It even provides a thumbnail-based chapter selection user interface, and full-screen video playback simply by double-clicking the video. And, it comes with a long list of professionally designed skins, three of which are shown in Figure 10.4. The output of the tool is a functioning HTML page with all the corresponding JavaScript and XAML files (plus the media file, of course). The skins are entirely vector based, and are set up to resize to fill the page while maintaining the aspect ratio. Several of them even have slick animations for showing and hiding pieces of the user interface.
Figure 10.4 Three built-in skins for the Silverlight media player produced by Expression Encoder.
Therefore, Expression Encoder can save an enormous amount of time, especially for straightforward user-controlled media playback. If that is your goal, you might not even need to know anything else covered in this chapter! But even if you want to do something more advanced that requires custom XAML and/or JavaScript, having Expression Encoder spit out its XAML and JavaScript and then using that as a starting point is an excellent strategy.
What’s the difference between Expression Encoder, Windows Media Encoder, and Windows Movie Maker?
All these tools can be used to produce video supported by Silverlight. Windows Movie Maker is for casual movie production and focuses more on glitzy transitions and editing effects rather than supporting a rich set of formats. The two “Encoder” tools are aimed for professionals and are more powerful, except that they omit the visual effects and transitions. (The assumption is that the Hollywood studios or other professionals using these encoding tools already have better editing and special effects tools than what Microsoft provides.)
As for Windows Media Encoder versus Expression Encoder, the quality of their encoding is the same because they use the exact same codecs. Each tool has features that the other lacks, but Expression Encoder is a much richer tool in terms of both feature set and ease of use. And unlike Windows Media Encoder, Expression Encoder has many Silverlight-specific features built in, such as encoder profiles compatible with Silverlight Streaming by Windows Live, direct publishing to the Silverlight Streaming site, and the ready-to-use media player templates built with XAML and JavaScript. Note that Windows Media Encoder is a free utility, but Expression Encoder is not.
Conclusion
Silverlight’s audio and video support, together with a powerful tool such as Expression Encoder, makes it very easy to create compelling multimedia content on the Web. If you combine that with the highly scalable distribution provided by Silverlight Streaming by Windows Live, you’ve got a complete solution for delivering multimedia (intermixed with other Silverlight content) that is truly first class. It almost couldn’t get any easier!
Even if you don’t take advantage of additional tools and services, the core functionality of MediaElement is compelling by itself. With a number of ways to control the audio or video, and the ability to break outside of a rigid rectangle (as with using the Windows Media Player ActiveX control), the things you can accomplish with multimedia are pretty much limited by your imagination!