CHAPTER 26
Silverlight

There are a few different ways people think of Silverlight. (Disclosure: I’m the Video Strategist on the Silverlight team at Microsoft.)

Folks coming from the Windows Media world may think of it as a replacement for the Windows Media Player OCX used for embedding WMP in web pages. For that audience, Silverlight provides several key improvements:

•  Good cross-platform and cross-browser support, a notable weakness in the WMP OCX

•  Windows Media DRM on the Mac for the first time in years

•  RIA functionality for much richer interactivity and design

Those coming from the RIA industry may think of Silverlight as another competitor to Flash and JavaFX. To that market, Silverlight offers several unique offerings:

•  Support for the huge .NET and Visual Studio ecosystems

•  Windows Media support

•  Adaptive Streaming via Smooth Streaming

And existing .NET developers consider Silverlight the portable version of their platform, bringing them:

•  Cross-platform/browser support

•  A much smaller installer than .NET

Silverlight is available for Windows and Intel Mac computers today. Microsoft is collaborating with Novell on its open-source implementation for Linux called Moonlight. Mobile versions have been announced, and Silverlight on Xbox has been demonstrated.

History of Silverlight

NET

While Silverlight itself is only a few years old, it’s fundamentally an implementation of the Microsoft .NET Framework, first released in beta in 2000.

.NET was designed to provide a more flexible, secure, and easy-to-develop-for base for Windows applications. Core to this was the adoption of managed code. Like Java bytecode, .NET managed code is an intermediate language that doesn’t get executed directly on the processor, and so is much less capable of accidentally or maliciously harming the underlying system. Unlike the Java or Flash runtime interpreters, .NET’s Common Language Infrastructure (CLI) supports a wide variety of languages. The most common language used in .NET is C#, a variant of C, but many other languages can be compiled to CLI-compatible code.

.NET also contains a large set of built-in functions, the “Class Library,” that a programmer knows will be available for use in particular versions of .NET.

While .NET started as a Microsoft technology, substantial portions of it have been released as standards, open source, or under an Open Specification Promise. The C# language and CLI are a license-free ECMA spec, like JavaScript.

.NET 3.0, released in 2006, introduced a new presentation layer called the Windows Presentation Foundation (WPF), that defines user interfaces via a XML language called XAML (eXtensible Application Markup Language), making it dramatically easier to design and maintain rich interfaces. This came along with the Expression Studio products for authoring XAML for .NET and Silverlight (and, of course, compression via Expression Encoder). The Expression apps are themselves WPF apps; they were used to develop themselves.

Looking at Silverlight from a .NET perspective, it’s essentially the CLI + XAML + Media Foundation (described in the Media on Windows chapter). Portable WPF support was seen as Silverlight’s defining feature; the codename for Silverlight was WPF/E, for WPF/Everywhere.

Silverlight 1.0

Silverlight 1.0 was released in 2007. It had XAML and media, but lacked the CLI; all programming had to be done via the browser’s own JavaScript engine. Thus it was mainly used for media playback.

Silverlight 1.0 was the last version to support PowerPC Macs (Apple shipped the last PowerPC Mac in 2006).

Silverlight 2

Silverlight 2 was released in the fall of 2008, after Beta 2 had been used for NBC’s successful Beijing Olympics streaming coverage. Some call Silverlight 2 the “real” version of Silverlight as it added the CLI, programmable with .NET. It included some notable additions for media:

•  The MediaStreamSource API, enabling new protocol and file format support to be written in managed code

•  Smooth Streaming was launched later in 2008, based on MediaStreamSource

•  WMA 10 Pro audio decoding

•  Windows Media and PlayReady DRM support

Silverlight 3

Silverlight 3 was released summer 2009. It didn’t include anything as disruptive as the CLI, but added many new media features:

•  MPEG-4 file format with H.264 High Profile and AAC-LC codec support

•  video and audio decoder decoders via MediaStreamSource

•  GPU compositing and scaling for better media performance

•  General media playback improvements—Core 2 Duo 2.5 GHz can play 1080p24 6 Mbps VC-1

•  Out-of-browser support, allowing Silverlight applications to install and run as local apps on Mac and Windows

The Future

A few Silverlight 4 features have been announced:

•  Native Windows Media multicast support

•  Offline DRM (no net connection required to install or play DRM’ed content)

There are some other great media features in Silverlight’s future, but I can’t include more details here. Check out Silverlight.net and the <URL for book updates> for announcements.

Microsoft has long been in the device world, and has versions of .NET for devices in the .NET Compact and Micro frameworks. Silverlight Mobile has been announced for Windows Mobile and Symbian handsets (specifically Nokia’s Series 60 and 80). Intel has also announced and demonstrated Silverlight for Moblin, their Linux-based device platform Silverlight for the Xbox has also been demonstrated, as has Silverlight on Intel’s Moblin platform.

Why Silverlight?

Uniform Cross-Platform/Browser Experience

For those with lots of Windows Media content or services, Silverlight is the easiest way to get those delivered in the browser for Macs and non-IE Windows browsers.

Moonlight on Linux is rapidly catching up with the released versions of Silverlight for Linux support, and had Smooth Streaming support working in early 2009.

Broad and Extensible Media Format Support

Silverlight 3 supports a wide array of media files directly:

•  Windows Media

•  MPEG-4 with H.264 and AAC

•  MP3

And it’s also very extensible, using a feature called MediaStreamSource (MSS). MSS can be used to implement new protocols or file formats in native code, added seamlessly to a Silverlight player. And with the new Raw AV feature in MSS, video and audio decoders can also be implemented in managed code. Some early demonstrations have included Ogg Vorbis playback and a Commodore 64 emulator (which I’d never thought of as a codec, but it works).

Smooth Streaming

IIS Smooth Streaming is an adaptive streaming technology, using IIS as the server, with Silverlight as the first platform supporting it. Smooth Streaming offers seamless bitrate switching and proxy cachability for live and on-demand content. It’s the first adaptive streaming technology for a RIA architecture.

.NET Tooling

There were about four million registered .NET developers as of 2009; that’s a big pool of talent familiar with development with C#, XAML, and their associated tools. Finding skilled .NET developers and teams is a lot easier than for Flash and JavaFX.

.NET is particularly popular in “Line of Business” applications that corporations build and use internally.

Silverlight Enhanced Movies

Silverlight 3 provides a great platform for interactive video experiences like those provided for DVD and Blu-ray. This is a particular passion of mine, and a project I’ve been working on for a while. Microsoft is providing a reference player to movie studios that supports menus, extras, and multi-langauge audio tracks and subtitles, enabling the DVD and Blu-ray experience and beyond via disc, download, or Smooth Streaming.

Leading UK retailer Tesco is the launch partner, and will include Silverlight enhanced versions of specific titles starting fall 2009.

Why Not Silverlight?

Ubiquity

Silverlight’s had over 400 million installs in its first year and a half, but it’ll take a little while to catch up with Flash and Java’s ubiquity. For the time being, a greater portion of users will need to install or update Silverlight than Flash.

Silverlight 2 and 3 are also not available for PowerPC Macs. They’ve dropped below 1 percent of total web use, but can be a vocal minority. Apple stopped selling PowerPC computers in 2006, and the newest of them are now outside of three-year extended warranties; that minority will continue to shrink.

Performance

Like its RIA cousin Flash, the overhead of the rich rendering model increases system requirements for playback compared to simpler players like Windows Media Player.

For systems with compatible GPUs, Silverlight 3 makes this much better by offloading scaling and compositing operations, leaving the CPU largely free for the video decoder. However, there is a significant minority of PCs without sufficiently capable GPUs to take advantage of this. And due to browser limitations on the Mac, Silverlight 3 only uses the GPU on Mac OS X in full-screen mode.

Playback overhead can be substantially reduced by not scaling the video rectangle, called a “Fast Path” player. This is the recommended behavior for non-GPU-enabled systems.

Silverlight’s .NET runtime is also much faster than ActionScript in Flash. CPU-heavy applications can be many times faster in Silverlight. This is a big part of what makes Raw AV codecs possible.

Silverlight for Progressive Download

Silverlight supports progressive download for WMV, MP4, and MP3 files. If the files are on a web server that enables byte-range access, Silverlight will use that for random access.

Silverlight for Real-Time Streaming

Silverlight works as a player with Windows Media Services. Silverlight 3 always uses the HTTP fallback mode of WMS, which needs to be turned on server-side.

Silverlight can be more responsive than WMP, in fact. By always using HTTP fallback, it can skip several seconds of protocol negotiation that WMP sometimes uses. Silverlight also has a programmable buffer size, as described in Chapter 16, enabling much lower latency playback than out-of-the-box WMP.

Silverlight doesn’t have native support for MBR stream switching, but it does support the URL nomenclature for selecting a particular bitrate. The following code would request the 100 Kbps band from content.wmv: mms://server/content.wmv? WMContentBitrate=100000

Thus, stream-switching logic can be implemented in managed code for cases where the same stream needs to work with both WMP and Silverlight; we used this with the 2009 NCAA March Madness. For Silverlight-only projects, it’s easier to just use Smooth Streaming.

While Silverlight 3 doesn’t have UDP support, Silverlight can receive Windows Media multicast streams when the Project Starlight plug-in is also installed. It is available free as compiled binaries and source code at Microsoft’s Codeplex web site. Silverlight 4 will add native multicast support.

Silverlight doesn’t have native RTSP support for MPEG-4 files. However, it could be implemented using MediaStreamSource, similar to Starlight.

IIS Smooth Streaming

While Silverlight has good compatibility with existing content libraries, most media content created specifically for Silverlight is now using the IIS Smooth Streaming adaptive streaming technology.

Smooth Streaming isn’t so much a feature of Silverlight as a feature that runs on Silverlight. The hard work in the client is all managed code using MediaStreamSource, like Project Starlight. Beyond that, it uses specially structured MPEG-4 files and a module for the IIS web server. Smooth Streaming support is coming to other platforms as well, including Xbox. Full specs for the protocol and file format have been published under Microsoft’s Open Specification Promise, allowing third parties to build interoperable implementations without charge or restriction from Microsoft.

The Smooth Streaming File Format

Smooth Streaming uses the MPEG-4 format using the fragmented subtype (fMP4). It’s described in more detail in Chapter 11, but the core reason fMP4 was used is that it allows a header per fragment instead of one big header for an entire file (see Figures 26.1a and 26.1b). Since most MPEG-4 players don’t support fMP4 yet, Smooth Streaming uses a .ismv extension.

Figure 26.1 (A) The structure of a fragmented MPEG-4 file (courtesy of Alex Zambelli). (B) The structure of a fragment of fragmented MPEG-4; there is one fragment per chunk delivered with Smooth Streaming.

image

Each set of media files also has two XML files that describe the content, and tell the Silverlight client how to request the right chunk and IIS where to find it in the ISMV files.

The smaller file is the .ism, which is a SMIL 2.0 file just listing the available bitrates, like this: <?xml version=”1.0” encoding=”utf-16”?></--Created with Expression Encoder version 3.0.1308.0--><smil xmlns=”http://www.w3.org/2001/SMIL20/Language”> <head>  <meta  name=”clientManifestRelativePath”   content=”Foo.ismc”/> </head> <body> <switch>  <video  src=”Foo4000.ismv”  systemBitrate=”4000000”>  <param  name=”trackID”  value=”2”  valuetype=”data”/>  </video>  <video  src=”Foo_2800.ismv”  systemBitrate=”2800000”>  <param  name=”trackID”  value=”2”  valuetype=”data”/>  </video>

The larger is the .ismc, which stores the size and location of each chunk for each bitrate, as well as the display properties for the whole experience. The overall data and the first couple of streams look like this: <SmoothStreamingMedia MajorVersion=”2” MinorVersion=”0” Duration=”1448948980”> <StreamIndex  Type=”video”  Chunks=”109”  QualityLevels=”10”  MaxWidth=”1280”  MaxHeight=”692”  DisplayWidth=”1280”  DisplayHeight=”692”  Url=”QualityLevels({bitrate})/Fragments(video={start time})”>  <QualityLevel  Index=”0”   Bitrate=”4000000”   FourCC=”WVC1”   MaxWidth=”1280”   MaxHeight=”692”   CodecPrivateData=”250000010FE37E27F1598A27F856718048800000010E5A2040”/> <QualityLevel   Index=”1”   Bitrate=”2800000”   FourCC=”WVC1”   MaxWidth=”1280”   MaxHeight=”692”   CodecPrivateData=”250000010FE37E27F1598A27F856718048800000010E5A2040”/> <QualityLevel   Index=”2”   Bitrate=”2000000”   FourCC=”WVC1”   MaxWidth=”1280”   MaxHeight=”692”

And a few chunks for a given bitrate look like this: <c  n=”0”  d=”22522482”>  <f  i=”0”  s=”67”  q=”4170”/>  <f  i=”1”  s=”72”  q=”4486”/>  <f  i=”2”  s=”67”q=”4170”/>

An actual playable directory of Smooth Streaming content as exported from EEv3 looks like Figure 26.2.

Figure 26.2 The standard directory structure of a Smooth Streaming encode plus player from Expression Encoder 3.

image

Beyond the .ism, ismc, and .ismv files, there are also the following files:

•  Default.html is a web page that embeds and launches the player. Lots of player parameters like display size, default caption state, GPU acceleration, and offline playback support are controlled from HTML to make it easy to change player behavior after deployment.

•  The Thumb.jpg is a thumbnail to represent the file in a media browser.

•  MediaPlayerTemplate.xap is the Silverlight media player itself, including all the assets and managed code. A default template can be used as is, modified in Expression Blend, or a new one created from scratch. Source code is provided with Expression Encoder.

•  Preview.png is a preview image of the media player’s GUI.

•  Settings.dat is a log file recording the compression settings used making the media. This can be useful to troubleshoot or recreate past jobs.

•  SmoothStreaming.xap is the Smooth Streaming heuristics. They’re in a separate file, making it possible to update without changing the media player and vice versa.

Tip: Expression Encoder 3 Installs ISMV Playback

Installing EEv3 (including the free version) also installs a DirectShow filter for ISMV files so that WMP and other DirectShow-based players can be used for testing and local playback. If you’re not on Windows 7, you’ll need to have a third-party H.264 and AAC decoder installed to play .ismv files using those codecs.

CBR Smooth Streaming: v1

The initial implementation of Smooth Streaming encoder shipped with Expression Encoder 2 SP1, using the VC-1 Encoder SDK. The only encoder mode that assured all streams would start new GOPs at the same time was 1-pass CBR, which had been tuned for broadcast applications with that requirement.

Encoding in this mode uses 2-second GOPs, so with 24p content every 48th frame starts a new GOP. Additional I-frame may also be inserted mid-GOP to helps quality, but the cadence will be preserved.

Since Smooth Streaming tries to maintain a buffer of several chunks, some bitrate variability per chunk is allowed, so a 5-second buffer is used to improve quality.

Since 1-pass CBR is used for live, this model was easy to adapt to live encoding, as used in the live encoders from Inlet, Envivio, Digital Rapids, and announced by ViewCast, Anystream, and Telestream.

1-pass CBR is also used in the initial H.264 Smooth Streaming implementation in Expression Encoder 3.

VBR Smooth Streaming: v2

The next generation of on-demand Smooth Streaming uses 2-pass VBR encoding. And it’s not just variable bitrate, but also can vary GOP length and frame size to deliver a more efficiently encoded and consistent experience.

Microsoft’s Expression Encoder 3 was the first product with the new Smooth Streaming Encoder SDK (SSE SDK). This initial version is VC-1 only, but the techniques are quite applicable to H.264. The SSE SDK introduces some core innovations for Smooth Streaming encoding that improve encoding time and quality:

•  A single analysis first pass suffices for any number of bitrates, speeding encoding time.

•  Variable chunk duration, adjusting to align with scene changes, reducing keyframe flashing. Typical GOP length remains 2 seconds, but can go up to 4.

•  Variable frame size; the coded resolution of a given frame is reduced to the largest frame that wouldn’t show bad artifacts. As hard-to-encode scenes typically have a lot of motion, and hence motion blur, this reduced resolution is generally not detectable. The resolution reduction can be horizontal, vertical, or both depending on the axis of motion. Reducing the pixels processed on the most challenging scenes speeds encoding and decoding.

•  Variable bitrate encoding, so each bitrate bad has a peak and average. The heuristics client can the best chunk based on individual file size, giving hardest content more bits while keeping the maximum bandwidth utilization predictable.

•  An improved core VC-1 implementation the Windows 7 with 8-way threading, faster per-thread performance, and improved quality with noisy backgrounds.

Figure 26.3 (A) A frame encoded at 1.2 Mbps 848 × 480 using the CBR codec. (B) The same encoded at 848 × 480 max with the SSE SDK VBR, which encoded this chunk at 640 × 366.

image

See Figure 26.4 for how this works in practice. VBR audio can also be used with VBR video, taking advantage of WMA 2-pass VBR modes.

Figure 26.4 (A) Comparing the QP of a CBR and a VBR encode with a particularly challenging piece of content, we see that the VBR encode reduces frame size to keep QP from getting too high. For easier sections, CBR and VBR have similar QP and VBR has full frame size. Harder sections drive down frame size dramatically; places it drops to below 5 percent of the CBR. (B) A table showing average size and minimum and max QP per chunk of the same challenging encode. We see the worst-case QP 31 in a number of chunks. (C) The same source encoded with VBR. Due to very fast editing, the average GOP and chunk duration is much shorter. We can see how challenging chunks get scaled down, with 16 frames down to 368 × 224. The worst case is QP 15. Note that the aspect ratio of the encoded rectangle varies depending on direction of motion, T from 1120 × 272 (4.12:1) to 832 × 528 (1.58:1).

image

Compared to the VC-1 Encoder SDK, the SSE SDK offers far fewer parameters to tweak. It baked in the settings that testing showed provided optimum quality in the lowest encoding time.

Parallelism is handled by encoding multiple simultaneous chunks. So the SSE SDK can easily saturate a 16-core machine, and probably beyond that.

Watch out for memory use, however; a 1080p encode can run the risk of exceeding the default 2 GiB process limit in 32-bit Windows. Running under 64-bit Windows lets 32-bit apps like EEv3 access 4 GiB of memory, which should be ample even for 2 K encoding.

Authoring Smooth Streaming

The initial round of Smooth Streaming encoding tools use the SSE SDK. But now the Smooth Streaming file format and manifests have been published, other vendors are building their own compatible implementations.

While the stock Smooth Streaming defaults are several bands of video and one of audio, there are a lot of other configurations and features that can be used.

Bitrates

Smooth VBR is quite a bit different than a traditional MBR approach. Since there’s no cost to switching between bitrates, there’s no need or even attempt to stick to a single bitrate. If watching a clip with 3000 and 2000 Kbps bands with a 2500 Kbps connection, the client can switch between 2000 and 3000 chunks in whatever pattern provides the optimal viewing experience.

,It’s a good idea to keep a relatively consistent ratio between average and peak bitrates when targeting Silverlight. The EEv3 default is 1.5x, which allows some variability but still reduces decode complexity of the hardest chunks by reducing frame size.

The places where you’d diverge from the standard ratio are at the lowest, and potentially highest bitrates.

Once the user is getting the lowest bitrate, there’s no way to go lower, so if you have a 300 Kbps minimum, you’ll want peak and average to be equal and with video total 300 Kbps. Otherwise there may be a sustained 450 Kbps period, and the user will see buffering. The SSE SDK is happy to have average and peak be the same value. I then normally use a 1.25x ratio for the second to lowest band to taper up to the full 1.5x.

If the highest bitrate is HD, decode complexity can be the limiting factor. If going up to, say 1080p24 6 Mbps, I’d want have peak = average there as well; a 9 Mbps peak can be asking for trouble. Also, with a flatter bitrate, hard chunks will get more aggressively downscaled, balancing out all the extra vectors in complex scenes that are harder to encode. If doing a flat top rate, I’ll use a 1.25x for the second bitrate to taper to the full VBR.

For 720p24 or below, bitrates and frame sizes are enough lower that it’s generally fine to stick with the 1.5x ratio; 720p has only 40 percent of the pixels as 1080p.

Frame size

Because of automatic frame resizing, we can be a lot more aggressive about frame rate; if the frame’s too big for a given chunk, it’ll be reduced as needed. So we can avoid the lowestcommon-denominator frame sizes where it’s almost too big a frame for the hardest bits, but quite a bit smaller than would work for the easy majority of typical content.

If we weren’t worried about decode complexity or encoding time, we could just leave all bitrates at the max frame size and see what comes out. And it’s often good practice to have more than one top bitrate share the same frame size. However, we do care about decode complexity, and as a frame’s maximum size sets a maximum cap on decode complexity, it’s a good idea to specify a lower frame size for bitrates that are going to get scaled down a bunch anyway. My rule of thumb is to reduce frame size at least to the point where 75 percent of chunks are left at full size.

Chunk/GOP duration

The default chunk length of two seconds is going to be fine for most content. By default, the SSE SDK will extend up to twice the default length to align with a natural keyframe, and use shorter chunks with rapid cuts.

Captions

Smooth Streaming supports synchronized captions as part of the manifest. These can be in a variety of languages, including Unicode languages like Chinese. It’s up to the Silverlight player to determine when and how they’re displayed. Since captions are delivered in advance as part of the index, a Silverlight player can make the entire script’s captions available for navigation or search.

Multiple language audio

Smooth Streaming can support multiple audio tracks, normally used for alternative languages. The same functionality could be used for audio commentary, with commentary and main audio in different tracks and mixed together at the player.

Audio-only

Smooth Streaming also works for audio-only applications, live and on-demand; it’s just an .ismv file with no video.

Live smooth streaming

Smooth Streaming was launched as an on-demand technology, with a live beta announced soon after. The live beta was used in a number of successful live events through the summer of 2009, including Wimbledon and the Tour de France, and shipped summer 2009.

Functionally, the live encode appends chunks as created to an .ismv and manifest. Thus a live event can be a “PVR in the cloud” with users able to play, pause, and scrub anywhere between the start of the event and the current time.

Silverlight for Interactive Media

Silverlight can deliver much more complex media presentations than WMP ever could. This can include multiple camera views, with thumbnails using the lowest Smooth Streaming bitrate. Silverlight can also use synchronized triggers to control client-side behavior.

Tip: Simulated Live Encoder

You don’t need a real live encoder to experiment with hosting and playback of Live Smooth Streaming. The Live IIS module comes with an emulator that loops preencoded ISMV files pushing them server as if it were a live encoder.

Given the depth that XAML and .NET are capable of, a Silverlight media player doesn’t have to be just a media player, and can be a real application that happens to involve media. There are video editors and multicamera video monitoring products using Silverlight already.

Silverlight for Devices

Silverlight has been announced for Windows Mobile, Symbian, Moblin, and Xbox 360, but few details have been released yet.

The Xbox 360 demo was specifically of the use of Silverlight for advertising as part of Xbox Live.

Formats and Codecs for Silverlight

In general, format choice in Silverlight is driven by what compatibility is required, if any. Windows Media offers easy interoperability with WMP clients, and MPEG-4 with QuickTime, Flash, and other MPEG-4 players. Smooth Streaming is Silverlight-only, for the moment and can offer a compelling experience for Silverlight-only content.

Windows Media

Silverlight’s Windows Media support includes the mainstream video and audio codecs:

•  Windows Media Video 9 Advanced Profile, no interlaced (VC-1 AP)

•  Windows Media Video 9 (VC-1 SP, MP)

•  Windows Media Video 8

•  Windows Media Video 7

The most widely used codec not listed is WMV 9 Screen. However, VC-1 does better than WMV 9 Screen with modern Aero Glass and Mac OS X user interfaces.

WMV9 would be used for any new content, as it’s much more efficient than the very old WMV 7/8 codecs. For most Silverlight tasks, Main and Advanced profiles offer equivalent quality, and Main is more compatible with older versions of WMP.

On the audio side, Silverlight supports the following codecs:

•  Windows Media Audio 7-9.2

•  Windows Media Audio 9-10 Professional (including the 10 Pro LBR modes)

Silverlight 3’s audio pipeline is stereo-only. It can decode 5.1 and 7.1 WMA Pro audio, but converts it to stereo for playback.

For content targeting Silverlight 2+ and/or WMP 11+, WMA 10 Pro’s 32–96 Kbps range is very high quality; 64 Kbps is the default for Smooth Streaming, and sound great for most content. Those modes provide reduced fidelity when played on older versions of WMP. If broader compatibility is needed, WMA Pro at 128 Kbps and higher is excellent and highly compatible. Below 128 Kbps, WMA 9.2 is the compatible choice, but audio fidelity is worse than WMA 10 Pro at lower bitrates. If WMA 9 must be used, remember that 2-pass VBR can offer a big quality boost over CBR at the same bitrate.

Silverlight 2+ can decode WMV files using Windows Media DRM. However, the Silverlight client needs to contact a PlayReady license server for authentication. Don’t sweat it if you’re not using DRM. If you are, a DRM service provider should already have implemented the required infrastructure to issue proper licenses to both Windows Media Player and Silverlight. Silverlight 4 will include a secure license store enabling Offline DRM.

MPEG-4 and H.264

MPEG-4 support was added in Silverlight 3. The initial implementation supports H.264 (High, Main, and Baseline profiles, progressive scan only) and AAC-LC (mono or stereo only).

H.264 High Profile can outperform VC-1 in efficiency, particularly as bits per pixel goes down. Decoder requirements are higher with H.264, particularly when expensive features like CABAC are used.

Silverlight 3 doesn’t support DRM or RTSP streaming of MPEG-4 content; both require WMV today. The built-in .mp4 playback is progressive download (with byte-range request support) only. Extensions to other protocols are possible using MediaStreamSource.

Smooth Streaming

Smooth Streaming currently requires codecs go in pairs; H.264 with AAC, or VC-1 with a WMA codec.

The differences in encoder implementations are a lot bigger today than in the bitstreams themselves, but that area is evolving quickly.

VC-1 and WMA

Silverlight 2 introduced Smooth Streaming with VC-1 Advanced Profile and WMA Standard or Pro. VC-1 Simple and Main profiles are not supported, as they aren’t MPEG-4 file-format-compatible.

At the moment, the SSE SDK for VC-1 offers dramatic quality improvements over both the old CBR VC-1 mode and EEv3’s H.264 mode, which is CBR Main Profile only.

VC-1’s lower decoder complexity means that CPU-bound computers will be able to decoder higher resolutions than H.264. This is further helped by the SSE SDK’s variable frame sizing, since the most complex, high-motion scenes get encoded with fewer pixels making decoder complexity “flatter” than with traditional encodes.

H.264 and AAC

H.264 High Profile is a more efficient codec than VC-1, and the EEv3 Main Profile CBR Smooth encodes look better than its VC-1 CBR encodes at the same bitrate.

But the combination of adaptive GOP placement, VBR, dynamic frame size, and lower decode complexity makes the SSE VC-1 implementation a better choice in most cases when targeting Silverlight on a PC. This will change as H.264 implementations become available that integrate SSE SDK techniques.

While H.264 decode complexity will be somewhat higher than VC-1, this can be mitigated by adjusting encoding settings per band. For example, using CABAC at low bitrates (where it pays off most in quality) and CAVLC at higher bitrates (where CABAC has the biggest impact on decode complexity) can help 1080p playback.

H.264 Smooth will be required for Silverlight on devices with only H.264 decode, of course. H.264 Baseline works perfectly in Smooth, unlike VC-1 Simple/Main.

Silverlight 3’s AAC decoder only handles AAC-LC, so audio bitrates need to be a little higher to match WMA 10 Pro quality. I recommend using 96 Kbps minimum, and ideally 128 Kbps.

MP3

Silverlight can also do progressive download of MP3 files. WMA 10 Pro can match that quality at less than half the bitrate, but there’s no reason not to use existing MP3 files if they’re of adequate quality and bitrate.

Raw AV

Silverlight 3’s Raw AV pipeline makes it possible to add new codecs, formats, and protocols to Silverlight using managed code. They are compiled into a .dll like any other Silverlight code, and built into a player. There’s no install or user action required; the user experience is identical to playback with the built-in formats.

The big drawbacks to Raw AV decoders are that they’re roughly half the speed of a native decoder, and someone has to make them. There’s been initial work by third parties on MPEG-2 and Ogg Theora/Vorbis players using Raw AV, and some companies are offering Raw AV decoders for license.

Silverlight 3 doesn’t support UDP, so any protocol would need to support HTTP or TCP. One example is Edgeware, which has made a MPEG-2 transport stream Silverlight client that, with their appliance converts an IPTV stream to a Silverlight-compatible HTTP stream.

Encoding Tools for Silverlight

In general, any compression tool that makes MPEG-4 and/or WMV content is already Silverlight-compatible. What’s new are tools that take advantage of specific Silverlight features, and which make Smooth Streaming-compatible content (sets of .ismv, ism, and ismc).

Expression Encoder

A core goal of Expression Encoder is making great Silverlight content. It’s also very focused on usability, particularly for the designer audience that Expression Studio targets.

Expression Encoder exists more to foster the Silverlight and broader Microsoft ecosystems than to be a big cash cow itself. Hence EEv3 with just WMV encoding is free, and adding Smooth Streaming and H.264 encoding, plus MPEG-2 and MPEG-4 decode, is only $49.

The underlying technology used in Expression Encoder is shared with other compression tool vendors, so similar quality and features should soon be available in other products as well.

Inlet

Inlet is incorporating Smooth Streaming authoring across their line of products. They had Live Smooth in Spinnaker and SSE SDK in Armada before Expression Encoder 3 shipped.

Envivio

Envivio’s 4Caster was the first Live Smooth Streaming encoder incorporating PlayReady DRM.

Carbon

Rhozet’s Carbon will include SSE SDK by the time this book hits the market.

Digital Rapids

Digital Rapids has added live and on-demand Smooth Streaming to StreamZ 3.1.

ViewCast

ViewCast, creators of the Osprey card and Niagra streaming appliances, will have Live Smooth Streaming soon.

Grab Networks

Agility, from the former Anystream, will have integrated Smooth Streaming by now.

Silverlight Tutorial: Smooth Streaming

Smooth Streaming is the obvious tutorial here. Since the scenarios are simple, I’ll be using the same multiple bitrate scenario as we used last chapter for Flash, tuned for Smooth Streaming. I’ll includes captioning and chapter marks to keep things from being too repetitive.

Scenario

As previous, we’re a web media company looking to publish 1080p24 sources to HD multiple bitrates. Due to new accessibility requirements, we’re adding captions to our video, and chapter-based navigation.

Three Questions

What Is My Content?

Still 1080p24 film/video content. This time we have caption files, too.

Who Is My Audience?

We’re still targeting a broad swath of consumers on a variety of hardware and software. We’re adding captions to increase accessibility, further growing our potential audience.

What Are My Communication Goals?

Again, we want everyone to have a good, high-quality playback experience. With Smooth Streaming, we also increase the reliability and scalability of delivery, and reduce our costs, with proxy caching.

Tech Specs

We’ll stick with the same bitrate ranges we used for the Flash encoding. However, we’ll encode with Smooth Streaming VBR, so those will only be the averages. As the 3 Mbps cap was to constrain bandwidth costs, we can use higher peaks as appropriate. We’ll use the standard 1.5x for the higher rates, tapering down to 1.0x to make sure that 300 Kbps is sustainable with 300 Kbps of bandwidth.

And we can raise our target frame size a bit since we’ll get automatic downconversion where needed, and generally lighter-weight decoding with VC-1 (see Table 26.1). That lets us use 720p for the top two bitrates; it’ll reduce frame size more often at 720p, but we’ll have more detail most of the time.

Table 26.1 Bitrates and Frame Sizes for Our Silverlight Tutorial.

Total bitrateAudio averageAudio peakVideo averageVideo peakMax widthMax height
3006496236236432240
5006496436545512288
80064967361104848480
1200649611361704960544
20006496193629041280720
30006496293644041280720

Audio is can be the straightforward default of WMA 10 Pro 64 Kbps average 96 Kbps peak at 44.1 stereo. 64 Kbps CBR WMA 10 Pro is normally quite transparent, but VBR will let the most challenging parts get a boost for optimum fidelity.

Expression Encoder 3 Settings

That’s all pretty straightforward to implement in EEv3. We can start with the “VC-1 IIS Smooth Streaming–HD 720p VBR” preset, delete a couple surplus bitrates, and make the changes described previously (see Figure 26.5).

Figure 26.5 Our settings in Expression Encoder. It’s nice to have all the settings that can vary as streams in tabs, with the global settings global, so you don’t have to keep double checking to make sure audio settings and frame rates are set correctly in six different places.

image

Which was boring enough that it’s good we have metadata to keep us on our toes.

EEv3 can import DXFP, SAMI, and XML formatted captions. Smooth Streaming captions follow the Windows Media/SAMI style captions where there’s a script event for text to display, and a second blank one that erases it.

Most captions will be available in one of these formats, or in a format that’s easily converted to one of these. But if a customer just dumps a text file on you, it’s actually pretty easy to convert that into XML with Excel as long as you can get the right fields into the right columns. Here’s a simple example: <ScriptCommands><ScriptCommand Type=”caption” Time=”00:00:15.000” Command=”at the. at the left we can see”/><ScriptCommand Type=”caption” Time=”00:00:17.500” Command=” “/><ScriptCommand Type=”caption” Time=”00:00:18.000” Command=”at the right we can see the”/></ScriptCommands>

It’s also possible to manually type in the captions directly into EE, but it’s not recommended unless you’ve got a very short project. Dedicated captioning tools are a much better way to go. Public broadcaster WGBH’s MAGpie has long been a popular free choice.

Chapter marks in EEv3 can be imported and exported with a similar XML format, which is handy for automating placement. They’re also easy to place manually. For a TV episode like this, we’d definitely want a chapter marker after each commercial break. See Figure 26.6.

Figure 26.6 Metadata for the file, chapters, and captions set up. I didn’t bother to set the first frame as a keyframe, since the first frame automatically is one.

image

Here’s a simple example of chapter markers: <Markers><MarkerTime=”00:00:25.8333334”Value=”Flying Cables”GenerateKeyFrame=”True”GenerateThumbnail=”True” /><MarkerTime=”00:01:09.7916667”Value=”Cable Run”GenerateKeyFrame=”True”GenerateThumbnail=”True” /></Markers>And that’s that!

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
18.226.187.101