CHAPTER 25
Flash

Introduction

Flash is often called an RIA (Rich Internet/Interactive Application) platform. There’s plenty of argument about what exactly that means, but I think of it as enabling rich applications for the web capable of doing much more than HTML. As a platform, a vast industry writes applications for Flash that run in a secure “sandbox” without access to the underlying computer or its data. Flash apps can’t do anything damaging to the underlying computers (or at least they’re not supposed to; there’ve been a number of exploited security faults in Flash over the years) so users don’t need to install anything or type in an admin password. Flash apps should “just work” like anything else in a web page.

In this Flash competes with the other mainstream RIA platforms of JavaFX and Silverlight, and potentially with HTML5 as it is standardized and supported in browsers.

Early Years: Flash 1–5

It’s sometimes hard to remember that Adobe’s Flash has only been a video platform for half of its existence. Flash started out as FutureSplash Animator, a simple vector graphics animation layer. I first saw it at the RealNetworks 1996 conference, pitched as a way to deliver animated content with compressed audio over the slow modem connections of the era.

Macromedia acquired FutureSplash and launched Flash 1.0 later in 1996. Following a similar course to Macromedia’s Director, Flash slowly gained broader functionality, including programmability via the ActionScript scripting language.

Good bundling deals, a consistent cross-platform/browser experience, and a small runtime (quick to download even over modems) led Flash into becoming a ubiquitous plug-in, and the SWF format became a standard for internet advertising chafing at the limitations of Animated GIF.

Unsurprisingly, many got excited about the possibility of getting video playback to work in Flash (similar efforts existed for Java). QuickTime even supported Flash as a track type to add richer navigation on top of QuickTime media playback, and Windows Media integration was also possible. But those required both Flash and the other media player be installed.

Video Is Introduced: Flash 6–7

Macromedia listened to its audience, and launched basic media support in 2002’s Flash MX (6). This introduced FLV as an ingest format for the Flash application; video assets needed to be bundled into the SWF. The initial codec was the simple but fast and small Sorenson Spark H.263 implementation, with MP3 audio. The first version of Flash Communication Server MX, the predecessor of today’s Flash Media Server, was released later that year.

Initially, Flash Video was more interesting as a way to get video inside Flash-centric experiences than it was interesting for video-centric experiences. Things got a lot better with Flash MX 2004 (7), which enabled progressive download of FLV files and performance improvements.

VP6 and the Video Breakout: Flash 8–9

Video in Flash came of age in 2005 with Flash 8 (thankfully reverting to version numbers), which introduced On2’s much more efficient VP6 codec, and was coupled with the rebranded Flash Media Server 2.0. However, the two most important events for Flash in 2005 were arguably in the corporate, rather than technological, world: Adobe acquired Macromedia, and YouTube was founded.

Flash utilization for video grew rapidly in this period, eating into the market share of Windows Media only a few years after it had largely replaced RealMedia.

The RealMedia/QuickTime/Windows Media competition had focused on lower-level technologies like codecs, playback performance, and hosting costs. While Flash was weaker on those metrics, Flash won on critical new ones. Flash changed the game with a focus on seamless cross-platform, cross-browser support and rich interactivity and design in the player.

The adoption of VP6 did have some downsides, however; as a proprietary codec only On2 provided professional-grade encoder implementations. And On2 wasn’t beholden to Adobe or Flash; it was there to make money for its shareholders. As the popularity of Flash grew, so did the cost of VP6, with many companies chafing at the cost of using it, and some choosing to deliver lower-quality video with the consumer-grade implementation provided by Adobe. VP6’s encoder was slow as well, and remained single-threaded long after competing codecs could utilize all of a 4-core CPU. YouTube skipped VP6 entirely.

The H.264 Era: Flash 9–10

Adobe’s corporate culture had been more standards-focused than Macromedia’s, and so as it looked at next generation media technologies, MPEG-4, H.264, and AAC were obvious choices. There was a big, existing ecosystem of companies making interoperable encoders, so Adobe wouldn’t be caught in a single-vendor bind like it had with VP6. While Adobe had a big business in video authoring tools, they didn’t have any compression-specific products to undercut. And clearly these were great codecs. Adobe had the mobile world in its sights as well, and hardware support for H.264 and AAC was rapidly being added to those devices.

MPEG-4 and H.264 support was added in the big Flash 9 Update 3 (version 9.0r115) late in 2007.

One part of MPEG-4 that Adobe didn’t add support for was RTSP-based streaming. The company decided to leave its own RTMP as the only supported streaming protocol in Flash.

The Future: Mobile and CE Devices

Adobe’s “Open Screen Project” is an ambitious effort to make Flash as ubiquitous in phones and other consumer electronics devices as it is on the PC. This builds on the significant traction of Flash Lite in various devices. Adobe has also announced GPU-accelerated hardware decode is coming in Flash, promising it will address long-stranding complaints about poor video decode performance.

Why Flash?

Ubiquitous Player

The biggest single reason Flash has been adopted is its ubiquity on Mac and Windows PCs; most users have a reasonably current version installed, and it’s an easy upgrade for most who don’t. Adobe’s surveys suggest that 98 percent of consumers have some version Flash installed on at least one computer they use although many may be a version or two behind. Enterprise penetration is similar, although tends to update more slowly.

Adobe’s working hard to get Flash broadly deployed on mobile devices as well, although that’s at a much earlier stage, and with much bigger differences in media support per device.

Uniform Rich Cross-Platform/Browser Experience

Beyond being everywhere, Flash can do a lot with the nonvideo features of Flash. This includes animations, overlays, synchronized data displays, and branded controls.

And perhaps most importantly, advertising. SWF has long been popular in web advertising, so being able to integrate Flash advertising with video playback was an important factor for ad-funded video sites.

Excellent Codec Support

Flash has great codec support with H.264 High Profile and AAC (up to HE v2). This also makes it compatible with lots of existing content in the QuickTime and MPEG-4 worlds.

Why Not Flash?

Higher Total Cost of Ownership for Streaming

Flash Media Server is certainly capable, but it’s also expensive to scale up for large audiences due to its unicast-only approach to content delivery. Adaptive Streaming technologies can deliver significantly lower cost per user for large audiences. FMS also has a higher per-server license fee than competing technologies; both Mac OS X Server and Windows Server 2008 include full streaming support as part of the OS.

Playback Performance

Flash has long had a reputation for sluggish and choppy playback, and even after GPU acceleration for compositing was added in Flash 10, things still get pretty dodgy as frame sizes go up.

Scaling to full-screen is particularly problematic, especially for higher-resolution screens (my main ones are 1920 × 1200 and 2560 × 1600, so I really feel this pain). It gets really slow, with lots of dropped frames, and typically with nearest-neighbor scaling artifacts.

We can hope further refinements of GPU acceleration will address these issues; the equivalent feature in Silverlight works fine on the same systems. Adobe has announced they’ll be adding GPU video decoding in a future version, which should further help.

Flash 10’s Mac and Linux versions perform somewhat poorly compared to the Windows version on the same hardware, and particularly with video. It’s not uncommon to see 3x higher CPU utilization booted into Mac OS X versus Windows in Boot Camp.

Flash for Progressive Download

Flash handles progressive download quite nicely with both FLV and MP4 content. ActionScript players can enable byte-rate request based random access, providing a more streaming-like experience à la YouTube.

Flash for Real-Time Streaming

Flash streaming uses Adobe’s RTMP protocol, which the company has now published. The primary RTMP server is Adobe’s own Flash Media Server, but Wowza Media Server Pro and the open-source Red5 also support Flash-compatible RTMP delivery.

RTMP uses TCP with HTTP fallback, but unlike other streaming protocols doesn’t support UDP or multicast. Designed and introduced later than RTSP and its many implementations implementations, it’s tuned for the much more reliable and lower-latency Internet of today. FMS is lauded for providing very low broadcast delays, generally less than two seconds and sometimes even subsecond. The relatively small buffers that implies suggests that RMTP might not perform as reliably on slow or lossy connections, but those are rare and becoming rarer.

Flash takes a somewhat different model for content protection compared to other platforms. Instead of encrypting the content before serving, FMS’s RTMPe protocol applies unique encryption in real-time for each user’s unicast-delivered stream. This increases server-side CPU load per user by perhaps 25 percent. Also, as the files or streams are unencrypted before they arrive at FMS, there’s some additional vulnerability compared to systems where encryption is applied with encoding.

RTMPe requires FMS, and remains proprietary to Adobe.

Note that neither the Flash client nor FMS supports standard MPEG-4 RTSP streaming; RTMP and RTMPe are the only supported real-time protocols.

Dynamic Streaming

FMS 3.5 introduced a new MBR switching technology called Dynamic Streaming. It’s a multiple bitrate stream switching technology implemented on RTMP.

Compared to older MBR implementations like Intelligent Streaming (Windows Media) and SureStream (RealMedia), Dynamic Streaming offers much smoother bitrate transitions and better measurement of available bandwidth (the Achilles’ heel of past MBR techniques). The maximum bitrate is determined by “bursting” data to the client periodically instead of keeping a continuous stream of bytes. Thus 5 seconds of content could be sent to the Flash client as quickly as possible, and if it only took one second to arrive, then available bandwidth may be five times higher than the media bitrate. Bitrate switches are done automatically to match network conditions and CPU decoding performance (if at least 20 percent of frames are dropped at the current stream). They can triggered programmatically as well—for example, to switch between camera angles.

In terms of the customer experience, Dynamic Streaming seems to be in the ballpark of Adaptive Streaming solutions like Move Networks and Smooth Streaming. The bigger differences are going to be in the proxy caching inherent in adaptive streaming, which the unicast-only RTMP can’t provide.

However, as I write this about nine months after FMS 3.5’s release, there simply aren’t any high-profile Dynamic Streaming sites to judge the experience on. Everything seems promising in theory, and I’m very curious to see what it’s like in practice. The Adobe/Akamai demo player yields a lot of dropped frames and has trouble making it to HD bitrates even on my 8-core monster workstation, but that could be due to player tuning or other issues.

Authoring Dynamic Streaming

Dynamic Streaming is simpler than adaptive streaming solutions on the authoring side, since it has no requirement for aligned GOPs or fragmented files; it works fine with existing FLV and MPEG-4 content. This is achieved with a couple of mechanisms:

So no particular care should be required to enable video stream switching. Audio stream switching is another matter; there’s typically a “pop” if there’s any audio ongoing as audio bitrates are switched. The only way to avoid this is to use the exact audio input and settings for all streams. The efficiency of HE AAC comes in handy here; even 48–64 Kbps can sound quite good, and can be paired with both low-and high-bitrate video.

Flash for Interactive Media

Flash is an interactive and rich media format that includes video. Flash itself is a topic for many, many books, and most people have seen plenty of Flash movies, so I won’t belabor the topic here. Simply stated, Flash can do pretty much anything you can image on the interactive side, with sufficient code and design effort.

Flash for Conferencing

While I’m generally not talking about videoconferencing in this book, Adobe’s Real Time Media Flow Protocol provides some interesting functionality for Flash. RTMFP, unlike RTMP is UDP based and can directly connect two users to improve latency. It applies P2P techniques to enable two clients to communicate directly. However, it doesn’t support multicast.

I’ve imagined that Adobe would merge some of this functionality with RTMP, but so far it’s strictly been for videoconferencing between small numbers of users.

Flash for Phones

Macromedia was working on Flash Lite for phones and other mobile devices well before it was acquired by Adobe.

Even though Flash is a relatively small piece of code by PC standards, it’s a lot for a phone, so Flash Lite has always supported only a subset of the full Flash. And the different screen and control types of a phone are generally a poor fit for a Flash app that assumes keyboard, mouse, and 1024 × 768 display.

Thus, Flash Lite is more about reusing the skills and tools of Flash authoring than having current Flash applications “just work” on devices.

That said, Adobe’s Open Media Project (OMP) is a big effort to bring a much fuller implementation of Flash to forthcoming generations of powerful phones and other CE devices. It’ll be a while before we see what exactly OMP is going to provide for content authors.

The current version of Flash Lite is 3.1 (based on Flash 8), which offers a good base of media features. It’s available for both phones and “digital home” devices like set-top boxes, and it offers several key features:

Note that, unlike the desktop, phones don’t get automatic Flash updates. Thus older devices have older versions of Flash Lite; there are many more phones running Flash Lite 2.0 than 3.1 as I write this.

While Flash Lite supports all of the listed codecs, it’s up to the handset vendors to integrate playback, and they may not include them all (particularly VP6). And phones vary widely in screen size and decoding horsepower; it’s nearly impossible to make video files that “just work” on all Flash Lite devices.

Adobe has done a good job of integrating mobile authoring and design of Flash Lite with Flash. Their Device Central app (bundled with Flash and Creative Suite) emulates a wide range of devices running Flash Lite, including media playback.

Going forward, Adobe promises great things for Flash on phones and CE devices as part of the Open Screen Project, with much more of Flash 10 included. It has yet to yield shipping products, so it’s too early to say what it’ll mean for content delivery.

Formats and Codecs for Flash

In general, H.264 and AAC are the best codecs available in Flash, so you should use F4V unless you have a compelling reason to use FLV. It’s a three-way choice based on video codec.

FLV with H.263

FLV with VP6

F4V

FLV

FLV and its encoding options are fully covered in Chapter 15.

MP3

Flash can play a standalone .mp3 file quite nicely, all the way back to Flash 4.

AAC offers better efficiency, particularly at lower bitrates, but there’s no reason not to continue using existing MP3 assets if they provide adequate quality and size. New audio-only content should be .m4a.

F4V

MPEG-4, H.264, and AAC are covered in detail in their respective chapters. This section just documents the specifics of their Flash implementations.

A .f4v file is just a .mp4 that contains Flash-compatible video and audio. There’s no requirement to use the .f4v extension; .mp4 can be used interchangeably. This is generally simpler for content meant to play in other media players as well as Flash.

Flash can play AAC-only MPEG-4 (.m4a) files just fine. Flash calls them “.f4a” by default, but .m4a works fine as well.

H.264 in Flash

Flash supports a broader range of H.264 profiles and modes than many PC players, although some aren’t of much practical use. Flash’s 8-bit rendering pipeline keeps 10-bit High Profile from offering any image quality improvement, although there’s no real downside to it either. And although Flash can decode interlaced H.264, it lacks a deinterlacer, leaving those ugly horizontal lines painfully obvious.

In general, your choice is going to be Baseline when targeting devices and High 4:2:0 otherwise. Flash video playback can be pretty hardware demanding, so make sure to test playback on your minimum spec hardware. You may want to turn off CABAC at higher bitrates, and definitely want to keep peaks as low as you can with sufficient quality.

AAC in Flash

Flash’s MPEG-4 support came along with extensive AAC support. I recommend that the following bitrate ranges use the particular profiles. AAC-LC can be the most transparent at high bitrates and HE v2 the least, but the lower the bitrate, the better the more complex codecs perform.

Note that Adobe Media Encoder labels HE AAC as AAC +.

Flash’s that audio pipeline is limited to stereo playback only. While it can decode 5.1 AAC audio just fine, it’ll be mixed down to two channels of output.

ActionScript Audio Codecs

Flash 10 introduced new audio APIs with much finer control over audio generation and playback. This can support audio decoders in ActionScript 3, like an audio-only version of Silverlight’s Raw AV MediaStreamSource.

So far, there have been just some demonstration releases of Ogg Vorbis decoding using this API.

Encoding Tools for Flash

This section covers tools for Flash-compatible MPEG-4 creation with Flash-specific functionality. FLV tools are covered in Chapter 25, and general H.264 compression tools are covered there.

Adobe Media Encoder

Adobe Media Encoder is bundled with the Creative Suite products. It’s a simple tool designed for nonprofessional encoders, and does a good job of helping those new to compression make reasonable choices.

AME uses the quite capable Main Concept H.264 implementation, and includes all three AAC flavors. However, it offers little configuration beyond bitrate, Profile, and Level. It doesn’t even expose 2-pass CBR.

Sorenson Squeeze

Squeeze was the first professional-grade compression tool with Flash support, from when Flash 6 added Sorenson Media’s Spark codec.

Squeeze continues to have good Flash support for FLV (including perhaps the best H.263 implementation) and F4V. Squeeze’s H.264 also uses Main Concept underneath, but exposes more parameters than AME for quality tuning.

Squeeze has a very handy ability to publish a SWF player along with the encode, using its own or custom templates.

Rhozet Carbon/Adobe Flash Media Encoding Server

Carbon is one of the few tools to actually offer a .f4v extension, albeit as part of its standard H.264 target. F4V otherwise is the same as Carbon’s MPEG-4 Program Stream.

Like AME, Carbon uses Main Concept’s H.264 implementation, but exposes many more parameters for finer-quality tuning. Its simultaneous rendering makes for faster throughput of Dynamic Streaming files, particularly on highly multicore systems.

Adobe’s Flash Media Encoding Server is a version of Carbon with all the non-Flash-compatible formats ripped out. It’s otherwise identical in functionality.

Adobe Flash Media Live Encoder

Adobe’s Flash Media Live Encoder (AFMLE) is a free (beyond needing FMS for it to be useful) live encoder for Flash, as the name suggests. It’s available for Windows and Linux, and supports standard capture cards and devices for each.

AFMLE, not just a mouthful of an acronym, also supports FLV (H.263 and VP6) and H.264, with multiple bitrate encoding. However, it’s a pretty basic webcasting tool. It’s easy to learn and use, but isn’t meant for carrier grade use.

For some reason, its H.264 is limited to Main Profile, and its AAC is LC and HE v1 only, no v2.

Most high-end live encoding products—like those from Inlet, ViewCast, and Digital Rapids—also support live Flash encoding. Since FMS doesn’t support RTSP, FMS support needs to be added to each encoder.