CHAPTER 14
H.264

H.264 has entered into the public consciousness only in the last few years, but the effort actually started back in 1997, as H.263+ was finishing up. At that point, it was becoming clear to some that adding new layers on top of H.263 was running out of steam, and yielding bigger increases in complexity than efficiency. What was then called “H.26L” was formally launched in the ITU’s Video Quality Expert’s Group (VCEG) in 1999, chaired by my now-colleague Gary Sullivan.

By eschewing the “let’s enhance H.263 again!” trend and building a new codec from the ground up, all kinds of proposed tools to improve compression efficiency were considered, even if they broke the traditional models of codec design. Their goal was to at least double the efficiency of other standard codecs.

As it became clear that MPEG-4 Part 2 wasn’t scaling as well as hoped, MPEG put out a call for proposals for new codecs. H.26L was proposed, and the ITU and MPEG agreed to collaborate on its development, forming the Joint Video Team (JVT).

The H.264 spec was delivered in 2003, with the Fidelity Range Extensions (FRExt), including the broadly used High Profile, coming in 2004.

The effort paid off, and H.264 is clearly the most efficient codec on the market, particularly as the bits per pixel drops down to very low levels. However, there’s a price to pay: Decoding the more complex profiles of H.264 requires more MIPS in software or more silicon in hardware to decode than past codecs. However, even the Baseline profile offered notable efficiency improvements over older complexity-constrained codecs, and has quickly taken over the market for lower-power media players.

Why H.264?

Compression Efficiency

H.264’s biggest draw has been its industry-leading compression efficiency. Its novel design enabled the bitrate requirements for common delivery scenarios to drop substantially compared to past codecs as H.264 encoders matured.

While we’ll certainly have even better codecs in the next few years, right now H.264

High Profile is the undisputed king of low-bitrate quality, with plenty of room for further optimization in current profiles.

Ubiquity

Nothing begets success like success. By combining a big technical advantage over MPEG-2 and MPEG-4 part 2, H.264 has quickly been added to new products that used the older technologies. It’s an ongoing process, but H.264 is certainly becoming a given in any consumer electronics device that does any form of video playback.

And since it’s an international standard, everyone has equal access to the specification and licensing, and can compete on implementation while participating in a big interoperable ecosystem.

Why Not H.264?

Decoder Performance

The biggest downside to H.264, particularly when using the more advanced settings with more advanced profiles, is decoder performance. It’s no big deal when GPU acceleration or an ASIC decoder is available, but in pure software, H.264 can require 2–4x the MIPS/pixel as simpler codecs.

Of course, there are optional features like CABAC (defined later in this chapter) that can be turned off to improve decoder performance, albeit with a reduction in compression efficiency.

Older Windows Out of the Box

Windows only started shipping with H.264 in-box with Windows 7, and even then not in the very low end Starter and Home Basic editions. For content targeting Windows machines, WMV is the only format that absolutely will play on every one, particularly on slow-to-update corporate desktops. Flash or Silverlight are likely to be installed as well, but they can’t play a double-clicked media file.

Profile Support

While H.264 High Profile is a clear efficiency winner, that’s not the profile supported everywhere. In particular many devices support both H.264 Baseline and VC-1 Main, and VC-1 can offer superior detail retention at moderate bitrates in that case.

Licensing Costs

Lastly, there has been some concern about MPEG-LA licensing fees. I am not a lawyer, but my reading of the published fees is that anyone with a business big enough to be impacted would have a business big enough to pay the fees. And these are nothing new; MPEG-2 and MPEG-4 part 2 had licensing fees, and VC-1 has similar ones today. However, this is the first intersection of the web world with standards-based codecs, so it’s new to a lot of folks. Concerns about H.264 patent licensing have been the major source of interest in the Theora and Dirac codecs.

What follows is my personal summary of a PDF from MPEG-LA’s web site; if you have legal questions about this, seek real legal advice.

The agreements renew every five years, with prices rising no more than 10 percent each.

Branded encoders and decoders (Built-in decoders are covered by Apple and Microsoft)

Video content and service providers

End user pays directly on title-by title basis

End user pays on subscription basis (yearly fee)

Remuneration from other sources, like free TV not paid by end user (yearly fee); may apply to internet broadcasting in 2010+

Annual cap for service provider enterprises

What’s Unique About H.264?

If VC-1 can be thought of as the culmination of the H.261/MPEG-1/MPEG-2/H.263 lineage, keeping what worked (IBP structure, 8 × 8 as the “default” block size, field/frame adaptive interlace coding) and addressing what didn’t (using an integer transform, adding in-loop deblocking, offering more flexibility and simplicity in block types per frame type), H.264 changed many of the fundamentals of how codecs work, from DCT and blocks all the way up to I-, B-, and P-frames.

4 × 4 blocks

Perhaps the most startling thing about H.264 was replacing the long-standard 8 × 8 block size for the much smaller 4 × 4 (although 8 × 8 was added later as an option in High Profile). With smaller block sizes, ringing and blocking artifacts from high QP blocks are smaller and hence less noticible. Plus, edges go through fewer blocks, and so fewer pixels need higher bitrates to encode well.

4 × 4 blocks were designed along with the in-loop deblocking filter, so that the loss of efficiency from having only 16 samples instead of 64 samples per block had less of an impact, reducing the visual and predictive impact of higher quantization.

The return of 8 × 8 blocks

While 4 × 4 worked well for the lower resolutions that H.264 was originally tested at, they didn’t scale as well to larger frame sizes, particularly with film content where there can be large areas that are relatively flat except for grain texture. In the initial DVD Forum testing to select a HD codec for HD DVD, several MPEG-2 and early H.264 encoders were tested along with VC-1, and to broad surprise, VC-1 won the quality shootout, followed by MPEG-2, and with the H.264 encoders in last place. That result likely had more to do with the relative refinement of the implementations of each codec. Before long, the High Profiles were launched, including the optional 8 × 8 block type, and H.264 was a more competitive HD codec.

Strong In-Loop Deblocking

Probably the single most valuable feature added in H.264 is its strong, adaptive in-loop deblocking filter. The goal of in-loop deblocking is obvious; soften the edges between highly quantized blocks in order to remove the blocking artifacts. And not just in postprocessing, but on reference frames, keeping block artifacts from propagating forward.

But H.264 was the first to allow the filter to be so strong (in terms of how much it softened) and broad (in terms of how many pixels it touches) as to really be able to eliminate blocking artifacts outright. As you crank up the compression with H.264, the image gets softer and softer, but doesn’t get the classic DCT sharp-edged blocks. Some H.264 encoders rely on this too much, in my opinion; it allows some sloppiness in encodes, throwing away more detail than necessary. Even if viewers don’t realize it’s due to compression, not a soft source, it’s still throwing away important visual information. But done correctly, this makes H.264 less vulnerable to QP spikes and distractingly poor quality, particularly with 1-pass CBR encoding.

The downside of in-loop deblocking is that it can also soften the image somewhat at even moderate QPs. Fortunately, it can be tuned down or even turned off in many encoders. Typically Blu-ray H.264 encodes won’t use in-loop deblocking for most scenes, reserving it for those for those high-motion/complexity scenes that would otherwise have artifacts (see Figures 14.1 and 14.2).

Figure 14.1 A H.264 encode with in – loop deblocking off.

Figure 14.2 Same source with in – loop deblocking on. Deblocking not only reduces blocking, but improves image data by improving reference frames.

Variable Block-Size Motion Compensation

H.264 supports a wide variety of different block sizes for motion compensation compared to the simple 16 ×16 and 16 × 8 options in MPEG-1/2. Each macroblock can use 16 ×16, 16 × 8, 8 × 16, 8 × 8, 8 4, 4 × 8, or 4 × 4 partitions.

Quarter-Pixel Motion Precision

H.264 has a more efficient implementation of quarter-pixel motion estimation than part 2’s, delivering a good efficiency improvement.

Multiple Reference Frames

In the classic codecs, there would be at most two reference frames at a time: A B-frame would reference the previous and next I-or P-frame. H.264 radically expanded the concept of a reference frame by allowing any frame to have up to sixteen reference frames it could access. And that goes for P-frames, not just B-frames. Essentially, the decoder can be required to cache up to 16 past frames, any or all of which can referenced in a motion vector to construct a future frame.

H.264 doesn’t require that decoded and displayed order match either. It’s perfectly legal for a P-frame to reference a later P-frame in display order, although it’d have to come first in encode order.

And frames don’t even need to be all P or B. Each frame can be made up of I-, P-, and B-slices (see Figure 14.3). What we think of as an I-frame in most codecs is called an IDR in H.264—Instantaneous Decoder Refresh. All frames that reference each other must all reference the same single IDR frame. Non-IDR I-frames are also allowed in H.264 (like a Open GOP to an IDR’s Closed GOP).

Figure 14.3 One possible structure of slices in a four–slice encode.

While awesome-sounding, and awesomely complex-sounding, it’s not as dramatically hard to implement or as dramatically useful as it appears on first blush. The main value of multiple reference frames is when a part of the image is occluded (covered) in the previous reference frame, but visible in a frame before that. Imagine the classic shot through a spinning fan. At any given point, the fan’s blades are covering a third of the screen. So frame 100 would include a part of the image obscured in frame 99 and frame 98, but visible in frame 97. If the encoder was using three reference frames, frame 100 could pull a motion vector from frame 97 to predict frame 100, instead of having to use intrablocks to encode that as a new part of the image.

The number of allowed reference frames is constrained by a maximum number of macroblocks for the given profile@level. Thus smaller frame sizes can support more reference frames than ones near the max capacity of the level. That said, it’s rare for the full 16 to be materially better than 3–4 for most film/video content. Cel animation or motion graphics can take better advantage of lots of reference frames.

All of this information can seem conceptually confusing, but the details aren’t really exposed to a compressionist. Normally, the choices are about how many reference frames to use. The rest follows from that.

While software decoders typically don’t enforce limits on maximum reference frames, hardware decoders very often will. This includes GPU decoder technologies like DXVA in Windows. So stick to the constraints of your target profile@level.

Pyramid B-Frames

Pyramid B-frames are another concept that can blow the mind of a classical compressionist.

It’s long been a matter of definition that a B-frame is never a reference frame, and is discarded as soon as displayed. Thus a B-frame only needs as many bits as it takes to look decent, without any concern for it being a reference frames.

But with pyramid B-frames, a B-frame can be a reference frame for another layer of B-frames. And that layer itself could be used as a reference for yet another layer of B-frames.

Typically the first tier of B-frames is written as “B” while a second tier that could reference those is written as b. So instead of the classic IBBPBBPBBP pattern, you could get IbBbPbBbPbBbP, with the “B” frames based on I-and P-frames, and the “b” frames referencing the I-, P-, and B-frames (see Figure 14.4).

Figure 14.4 Pyramid B–frames strucuture.

These complex structures, when coupled with an encoder that can dynamically adjust its pattern to the content, can further improve efficiency of encoding with motion graphics, cel animation, rapid or complex editing patterns, and lots of flash frames. It doesn’t make that much of a difference with regular content, but is still worthwhile.

Weighted Prediction

Weighted prediction allows for whole-frame adjustments in prediction. This is mainly useful for efficient encoding of fades, both to/from a flat color and cross-fades. Coupling weighted prediction with multiple reference frames and B-frames can finally make that kind of challenging content encode well.

Logarithmic Quantization Scale

The scaling factor in H.264’s quantizer isn’t linear as in other codecs, but logarithmic. So going up one QP has less visual impact on the lower end of the scale. So, while QP 10 is typically pretty ugly in MPEG-2 or VC-1, QP 10 in H.264 is around the threshold of visually lossless.

Flexible Interlaced Coding

I have mixed feelings about H.264’s interlaced support. I’m impressed it was done so well. But I worry that gives the industry one more excuse to keep the interlaced millstone around our necks another decade.

There are two basic levels of interlaced in H.264.

Macroblock adaptive field-frame

Macroblock Adaptive Field-Frame (MBAFF) is conceptually similar to classic MPEG-2 and VC-1 interlaced coding, where individual macroblocks can be either interlaced or progressive, with some clever tweaks around the edges for in-loop deblocking and the like.

Picture adaptive field-frame

Picture Adaptive Field-Frame (PAFF) allows individual frames to be MBAFF (good for interlaced frames with mixed areas of motion and static elements), or fields separately encoded (for when everything is moving).

So far, interlaced H.264 has been mainly the domain of broadcast, and hasn’t been supported in QuickTime, Silverlight, Flash, or many devices. (Well, Flash can display interlaced H.264 bitstreams, but it leaves them as interlaced, so you should deinterlace on encode anyway.)

CABAC Entropy Coding

Entropy coding has been pretty similar for most codecs, with some flavor of Huffman coding with variable length tables. And H.264 has a variant of that with its CAVLC mode—Context Adaptive Variable Length Coding. It’s a fine, fast implementation of entropy coding.

What’s new in H.264 is CABAC—Context Adaptive Binary Arithmetic Coding. As you may remember from the section on data compression, Huffman compression can’t be more efficient than one-bit-per-symbol, but arithmetic compression can use fractional bits and thus can get below one. Thus, CABAC can be quite a bit more efficient than CAVLC; up to a 10–20 percent improvement in efficiency with high compression, decreasing with less compression. With visually lossless coding it is more like 3 percent more efficient. CABAC is probably second only to in-loop deblocking in importance to H.264’s efficiency advantage. But it comes with some limitations as well:

Differential Quantization

H.264 can specify the quantization parameter per macroblock, enabling lots of perceptual optimization. Improved use of DQuant has been a big area of H.264 advancement in recent encoders.

Quantization Weighting Matricies

This is a High Profile–only feature brought over from MPEG-2. Custom quantization matricies allow tuning of what frequencies are retained and discarded.

The popular x264 recommends a flat matrix and then aggressively uses Differential Quantization.

Modes Beyond 8-bit 4:2:0

Relatively unique for an interframe codec, H.264 has profiles that support greater-than-8-bit-per-channel luma, and both 4:2:2 and 4:4:4 sampling. H.264 also supports using a full 0-255 luma range instead of the classic 16-235, making round-trip conversion to/from RGB much easier.

None of those modes are widely supported in decoders yet, though. H.264 is still mainly used at good old 16-235 8-bit 4:2:0.

H.264 Profiles

Baseline

The Baseline profile is a highly simplified mode that discards many of the computationally expensive features of H.264 for easy implementation on low-power devices. Generally, if it runs on a battery or fits in your hand (like an iPod, Zune, or phone), it likely will only have Baseline support. Baseline leaves out several prominent features:

So, Baseline requires quite a few more bits to deliver the same quality as the more advanced profiles, particularly High. But it works on cheap, low-power decoder ASICs.

While Baseline is certainly easier on software decoders than a full-bore High encode, that absolutely doesn’t mean that Baseline should be used for software decodes even when decode complexity is a constraint. In particular, B-frames and 8 × 8 blocks offer big improvements in efficiency, and minor increases in decoder complexity. In software, they’ll outperform Baseline when comparing the same quality.

Constrained baseline

There’s an effort ongoing to add a new “Constrained Baseline” profile to document what’s emerged as the de facto Baseline standard. It turns out that there are some required features in Baseline not used by Baseline encoders, and hence not tested and not working in existing “Baseline” decoders. Constrained Baseline would just codify what is currently implemented as Baseline.

Extended

The Extended profile targets streaming applications, with error resiliency and other features useful in that market. However, it hasn’t been implemented widely in encoders or decoders. Its most prominent appearance has been as a mysteriously grayed-out checkbox in QuickTime since H.264’s addition in 2005.

Main

Main Profile was intended to be the primary H.264 codec for general use, and includes all the useful features for 8-bit 4:2:0 except for 8 × 8 block sizes. However, adding 8 × 8 turns out to help compression a lot without adding significant complexity, so any new decoder that does Main also does the superior High.

The primary reason to use Main today is if you’re targeting legacy players that do it but not High. Silverlight, Flash, and WMP all included High in their first H.264 implementations.

QuickTime was Baseline/Main only in QuickTime 7.0 and 7.1, with High decode added in 7.2.

Many Main decoders can’t decode interlaced content, although that’s a required feature of the Main Profile spec.

High

High Profile adds three primary features to Main:

8 × 8 blocks are quite useful in terms of improving efficiency, particularly with film content and higher resolutions.

Monochrome is used much less in practice (and it’s not like coding big matrices where Cb and Cr = 128 took a lot of bits), and decoder support is hit-and-miss.

Adaptive Quantization Tables have been used to good success in some recent encoders, and are becoming another important High advantage.

High 10

After the initial launch of H.264, attention quickly turned to “Fidelity Range Extensions”—FRExt—to get beyond 8-bit luma coding. However, it was the addition of 8 × 8 blocks to the baseline High 8-bit codec that was broadly adopted. It’s great, but so far not used for content distribution. Having 10-bit would require less dithering on authoring. However, as the end-to-end ecosystem is all based around 8-bit sampling, it would be a slow transition as best.

The only widely supported player that decodes High 10 is Flash. Adobe doesn’t support it in their compression tools, however, and Flash itself has an 8-bit rendering model that doesn’t take advantage of the additional precision.

High 4:2:2

High 4:2:2 simply adds 4:2:2 chroma sampling with 10-bit luma. It would be most useful for interlaced content.

High 4:4:4 Predictive

High 4:4:4 Predictive (normally just called high 4:4:4) goes further to 14-bit 4:4:4. Again, points for awesome, but it is much more likely to be used as an intermediate codec than for content delivery for many years to come.

Intra Profiles

H.264 supports intra-only implementations for production use (like an updated MPEG-2 High Profile I-frame only). These are subsets of the following:

Really, intra codecs should use CAVLC anyway. CABAC is worth around only 3 percent extra efficiency at these high bitrates, while these same high bitrates make the decode complexity of CABAC much more painful.

Scalable Video Coding profiles

I can’t decide if Scalable Video Coding (SVC) is going to be the Next Big Thing in IP-based video delivery, or just something neat used in video conferencing.

The idea of scalable coding has been around for quite a while. The concept is to use prediction not just for future frames, but to enhance a base stream. B-frames are actually scalable video already, and could be thought of as a temporal enhancement layer. Since no frames are based on B-frames (putting pyramid B aside for a moment), a file could have all its B-frames stripped out and still be playable, or have new B-frames inserted to increase frame rate without changing the existing frames.

Temporal scalability

For example, take a typical 30-fps encode at IBPBPBP. A lower-bitrate, lower-decode complexity version could be made by taking out B-frames and leaving a 15 fps IPPP. Or extra B-frames (pyramid b, typically) could be added making it IbBbPbBbPbBbP and 60 fps (see Table 14.1).

Table 14.1 Temporal Scalabilty à la Pyramid B-Frames. The reference frames are at 15 fps, the first set of B-frames is at 30p, and then the second layer of B-pyramids.

Sec.

1/60

2/60

3/60

4/60

5/60

6/60

7/60

8/60

9/60

10/60

11/60

12/60

13/60

15p

I

P

P

P

30p

I

B

P

B

P

B

P

60p

I

b

B

b

P

b

B

b

P

b

B

b

P

It’s conceptually simple. However, only the original 30 fps encode would have the best coding efficiency. If you knew you wanted 15 fps or 60 fps all along, you could have used the optimal bitrate and frame structure to encode that band. But where this really shines is when you don’t know how much bitrate is going to be available; instead the video can be encoded once (as IbBbP in the example) and depending on bitrate and/or processing power, just the IP, IP + B, or IP + B + b frames could be delivered on the fly. Each layer of extra data that can be added is called an enhancement layer. For our example here, the I-and P-frames are the base layer, the B-frames are the first enhancement layer, and the b-frames are the second enhancement layer.

And thus the basic concept of scalable video: trading some compression efficiency for increased flexibility.

Spatial scalability

Another place where scalable coding has long been used is “progressive” JPEG and GIF files, where a lower resolution of the image is transmitted and displayed first, with a higher resolution version then predicted from that. This is of course the basic mode of operation of wavelet codecs, and wavelet codecs can be thought of as being natively scalable, with each band serving as an enhancement layer.

H.264 SVC goes well beyond that simplified example in spatial scalability, but the concept is the same—predicting a higher-resolution frame from a lower-resolution frame. The key innovation in H.264 prediction is adaptive mixing of both spatial prediction (of the same frame in a lower band) and a temporal prediction (using another frame of the same resolution).

Quality scalability

The last form of scalability in SVC is quality scalability, which adds extra data to improve the quality of a frame. Essentially, the enhancement layers add additional coefficients back to the matrix. Imagine them as the difference between the coefficients of the base stream’s matrix QP and a less coarse QP.

SVC profiles

There are three profiles for SVC today, mapping to the existing profiles in use:

Note the lack of Main; there’s no point in using it instead of High, since SVC requires a new decoder. High is getting by far the most attention, as it as the best compression efficiency. High Intra could be of use for workflow project to enable remote proxy editing of HD sources.

Why SVC?

Looking at SVC, we can see that it has the potential to offer better scalability than classic stream switching or adaptive streaming. This is because stream switching needs to make sure that there’s some video to play, and so it can’t be very aggressive at grabbing higher bitrates with low latencies. With stream switching, if a 600 Kbps stream only gets 500 Kbps of bandwidth, there’s a risk of running out of buffered frames to decode and the video pausing. But with SVC, each enhancement layer can be pulled down one-by-one—a 100 Kbps base layer can be delivered, and then additional 100 Kbps layers could be downloaded. If a player tried to pull down the 600 Kbps layer but didn’t get it all, the layers at 500 Kbps and below are still there, and there’s no risk of a loss of video playback.

Buffering could be prioritized per stream, with the base stream always buffering ahead several minutes, with relatively less buffer the higher the enhancement layer. And with this buffering comes greater responsiveness. With stream switching, it takes a while to request a new stream; if background CPU activity causes dropped frames, it may be a few seconds before a new stream can be selected. But SVC can turn layers on and off on the fly without having to request anything new from the server.

Lastly, SVC can save a lot of server storage space. 4 Mbps across 10 layers is a lot smaller than a 4 Mbps stream + 3 Mbps stream + 2 Mbps + 1.5 Mbps…

Why not SVC?

There are a few drawbacks for SVC, of course. First, decoders aren’t widely deployed, nor are encoders widely available. It’s essentially a new codec that hasn’t solved the chicken-and-egg problem of being used enough to drive decoders, or being in enough decoders to drive use. This is particularly an issue with hardware devices.

SVC also has more decoder overhead; 720p24 4 Mbps with eight bands requires more power to play back than 720p24 4 Mbps encoded as a single layer.

There’s also an efficiency hit, particulary when spatial scalability is used. It might take 10–15 percent higher bitrate by the highest layer of a low bitrate to HD layer set than to encode a single-layer HD stream. So, to get the quality of a 3000 Kbps single stream, SVC with eight layers may need 3300–3450 Kbps.

When SVC?

SVC is more interesting the less predictable the delivery environment is. It makes little sense for file-and physical-based playback, since the available bandwidth is already known. And for IPTV-like applications with fixed provisioned bandwidth, it’s more efficient to just encode at the desired bitrate.

But when bandwidth is highly variable, like with consumer Internet use, SVC could shine. And the lower the broadcast delay required, the harder it is to use stream switching.

Thus, the first place we’re seeing SVC catch on is in videoconferencing, where the broadcast delay is a fraction of a second.

Where H.264 Is Used

QuickTime

QuickTime was the first major platform to support H.264, introduced with QuickTime 7 in 2005. Initially only Baseline and Main were supported for decode. QuickTime 7.2 added High Profile decoding in 2007; it’s safe to assume nearly all Mac and iTunes users would have upgraded by now. QuickTime doesn’t support interlaced decode at all.

QuickTime’s decoder is decent, albeit software-only and not very fast.

Flash

Flash added H.264 decode in version 9.115. They have one of the deeper implementations (based on the Main Concept decoder), and can decode three profiles:

Table 14.2 H.264 Levels.

Level number	Max macro-blocks/second	Max macro-blocks/frame	Max bitrate Baseline/Main	Max bitrate High	Max 4:3	Max 16:9	Max 4:3 24p	Max 16:9 24p	Max 4:3 30p	Max 16:9 30p
1	1485	99	64 Kbps	80 Kbps	176 × 144	208 × 128	144 × 112	160 × 96	128 × 96	144 × 96
1b	1485	99	128 Kbps	160 Kbps	176 × 144	208 × 128	144 × 112	160 × 96	128 × 96	144 × 96
1.1	3000	396	192 Kbps	240 Kbps	368 × 272	432 × 240	208 × 160	240 × 128	192 × 128	208 × 128
1.2	6000	396	384 Kbps	480 Kbps	368 × 272	432 × 240	288 × 224	336 × 192	256 × 208	304 × 176
1.3	11880	396	768 Kbps	960 Kbps	368 × 272	432 × 240	416 × 304	480 × 272	368 × 272	432 × 240
2	11880	396	2 Mbps	2.5 Mbps	368 × 272	432 × 240	416 × 304	480 × 272	368 × 272	432 × 240
2.1	19800	792	4 Mbps	5 Mbps	512 × 400	608 × 336	528 × 400	608 × 352	480 × 352	544 × 304
2.2	20250	1620	4 Mbps	5 Mbps	736 × 560	864 × 480	528 × 416	624 × 352	480 × 368	560 × 304
3	40500	1620	10 Mbps	12.5 Mbps	736 × 560	864 × 480	752 × 576	880 × 496	672 × 512	784 × 448
3.1	108000	3600	14 Mbps	14 Mbps	1104 × 832	1280 × 720	1232 × 928	1424 × 816	1104 × 832	1280 × 720
3.2	216000	5120	20 Mbps	25 Mbps	1328 × 992	1520 × 864	1744 × 1328	2016 × 1136	1568 × 1168	1808 × 1024
4	245760	8192	20 Mbps	25 Mbps	1664 × 1264	1936 × 1088	1872 × 1408	2160 × 1216	1664 × 1264	1936 × 1088
4.1	245760	8192	50 Mbps	62.5 Mbps	1664 × 1264	1936 × 1088	1872 × 1408	2160 × 1216	1664 × 1264	1936 × 1088
4.2	522240	8704	50 Mbps	62.5 Mbps	1728 × 1296	1984 × 1120	2720 × 2048	3152 × 1760	2432 × 1840	2816 × 1584
5	589824	22080	135 Mbps	168.75 Mbps	2736 × 2064	3168 × 1792	2896 × 2176	3344 × 1888	2592 × 1936	2992 × 1680
5.1	983040	36864	240 Mbps	300 Mbps	3536 × 2672	4096 × 2304	3728 × 2816	4096 × 2304	3344 × 2512	3856 × 2176

Flash can decode interlaced H.264, unlike QuickTime and Silverlight. However, it doesn’t have any deinterlacing support, so both fields will be painfully visible on playback.

Silverlight

Silverlight 3 introduced MPEG-4 and H.264 support, and it supports the same H.264 profiles as QuickTime:

Smooth streaming with H.264

With Silverlight 3, H.264 is now supported in Smooth Streaming. While tool support for that combination was just emerging as this book was being written, support was forthcoming in Expression Encoder 3, plus products from Inlet, Telestream, Envivo, and others.

The Smooth Streaming Encoder SDK used for VC-1 Smooth Streaming encoding provides resolution switching to maintain minimum quality. There’s less need for resolution switching with H.264 due to its strong in-loop deblocking.

One nice feature of Smooth Streaming is that different versions of the content can be encoded at the same bitrate. So codecs can be mixed and matched; for example, a single player could link to different 300 Kbps streams containing:

Windows 7

While there have been third-party DirectShow H.264 decoders for years, and hardware acceleration support for H.264 since Vista, Windows 7 is the first version with out of the box support for the MPEG-4 formats, including H.264.

Windows 7 has a deep implementation, including interlaced support and hardware acceleration via the Windows 7 Media Foundation API, including HD H.264 playback on the next generation of netbooks.

Portable Media Players

H.264 Baseline has become the dominant codec for portable media players and mobile phones. We’ll give specific examples in Chapter 25.

Consoles

Both the Xbox 360 and PlayStation 3 can play back H.264 files and streams from media services. There is not an equivalent for the Wii (which has much more limited hardware capabilities).

Settings for H.264 Encoding

Profile

Profile is your first question, and largely dependent on the playback platform. This will almost always be Baseline or High. Even if you’re concerned about decode complexity with a software player, use High if you can, while turning off the expensive-to-decode features. Features like 8 × 8 blocks improve compression without hurting decode complexity.

Level

Level defines the constraints of the encode to handle particular decoders. The level is stored in the bitstream, and decoders that can’t play back that level may not even try. It’s important to always have the level set and specified, even if implicitly (some encoders figure out the lowest Level compatible with your encoding settings). It’s bad practice to do 640× 360 encoding with Level left at 5.1; use the lowest Level your content needs in order to deliver broader compatibility. QVGA 30p encodes never need more than 2.1, SD encodes never need more than 3.1, and 1080 p24 encodes never need more than 4.1.

Bitrate

Compared to older codecs, a well-tuned H.264 encode can deliver great quality at lower bitrates. However, that comes at the cost of extra decoder complexity; encoding H.264 at MPEG-2 bitrates is a waste of bits and a waste of MIPS. Make sure that your average (and, if VBR, peak) bitrates aren’t higher than needed for high quality. There’s no need to spend bits beyond the point where the video stops looking better.

Conversely, H.264 isn’t magic, and if you’re used to just watching out for blocking artifacts when compressing, you’ll need to take a more detailed look with H.264 since in-loop deblocking yields softness instead of blockiness when overcompressed. Use enough bits to get an image that looks good, not just one that’s not obviously bad.

Entropy Coding

The two entropy coding modes in H.264 Main and High are CAVLC and CABAC. CAVLC is faster and less efficient, CABAC is slower to encode and decode, but more efficient.

Most of the time, you’ll want to use CABAC; CAVLC only makes sense when you’re worried about decode complexity.

Since CABAC’s decode hit is proportional to bitrate, watch out for performance with VBR and high peak bitrates. Make sure the hardest part of the video plays well enough on the target platform. A peak of 1.5x the average bitrate is a good starting point; don’t use a higher peak than one that significantly contributes to quality.

Slices

One way to allow entropy decoding to parallelize is to encode with slices, breaking the frame into (typically) horizontal stripes. Each slice is treated somewhat like an independent frame, with separate entropy coding.

This can improve playback on some multicore software decoders, and is required by the Bluray spec for encodes using Level 4.1. There’s a slight loss in efficiency with slices (perhaps 1 percent per slice as a ballpark). The hit is bigger the smaller the frame; a good rule of thumb is for each slice to be at least 64 pixels tall.

It’s generally a lot more bitrate efficient to encode CABAC with slices than CAVLC without slices.

The Main Concept and Microsoft encoders, among others, also use slices for threaded encoding; the default is one slice per physical core in the encoding machine. However the trend in H.264 is definitely single-slice encoding for maximum efficiency.

Number of B-frames

Two B-frames is a good default for most non-pyramid H.264 encodes. Most H.264 encoders treat B-frame number as a maximum value when B-frame adaption is turned on, with Microsoft’s an exception.

Pyramid B-frames

Pyramid B-frames can be a welcome, if not huge, boost to encoding efficiency. You’ll need to be using at least 2 B-frames to use pyramid, obviously. Pyramid should be used with an adaptive B-frame placement mode for best results.

Number of Reference Frames

As mentioned above, large numbers of reference frames rarely pay off for film/video content, although they can be quite useful with cel animation and motion graphics. Four is generally sufficient. However, using the full 16 often has little impact on decode performance for playback, but can slow down random access and delay the start of playback.

Strength of In-Loop Deblocking

The default in-loop deblocking filter in H.264 is arguably tuned for video and lower bitrates more than for film grain and higher bitrates, and may soften detail more than is required. For tools that offer finer-grained control over it, turning it down can increase detail. Beyond on/off, there are two parameters that control behavior of in-loop deblocking:

It’d be rare to increase the strength of the filters. The range for each is –6 to +6, with 0 being the default. If adjusted, the values almost always are negative, typically –1 or –2 is sufficient.

Some Blu-ray encoders default to having in-loop deblocking off entirely in order to preserve maximum detail. It seems unlikely to me that the job couldn’t be accomplished by just using lower settings.

H.264 Encoders

Main Concept

There’s a huge number of H.264 encoders out there, but just a few that are used across a variety of products you’ll see again and again. Main Concept is the most broadly licensed H.264 encoder SDK, and provides support in video products from companies including the following:

The Main Concept SDK has seen progressive improvement in speed, quality, and configurability over the years. At any given time, most products will be using the same implementation; differences are mainly in what controls are exposed and what defaults are used.

In general, Main Concept provides the same settings as other encoders (Figure 14.5). One difference is that it doesn’t have an explicit Complexity control, but rather a series of options with a Fast/Complex switch. Generally the Fast modes are faster than they are bad, but the Complex modes will eke out a little extra quality if needed. Unfortunately Main Concept’s documentation has infamously lacked useful settings guidelines to users or licensees, so we’re often in the dark as to the impact of any given option. However, a few comments:

Figure 14.5 The Main Concept encoding controls as presented in their own Reference product. I just wish they had more documentation on the impact of these settings.

All that said, the general defaults are pretty good in most products these days. Perhaps the biggest criticism of Main Concept is that it’s less tuned for detail preservation than other implementations, particularly x264.

x264

x264 (Figure 14.6) is an open source H.264 implementation following the successful model of LAME for MP3 and Xvid for MPEG-4 Part 2. It’s absolutely competitive with commercial implementations, has an active and innovative developer community, and is used in a wide variety of non-commercial products. Some of its cool features:

Figure 14.6 MeGUI settings. A good front-end to the great x264, but I can’t say it does a very good job of guiding users to setting that are tweakable and the ones that should really be left alone. But I can always find out exactly what each option does.

Great performance

Using defaults, it can easily encode 720p in real-time on a decent machine, and is still quite fast even with every reasonable quality-over-speed option selected.

Great quality

x264 has done great work with classic rate distortion optimizations, and is able to perform rate distortion down to the macroblock level. But perhaps even more impressive is its perceptual optimizations. Its Psy-RDO mode does a pretty incredible job of delivering more psychovisually consistent encodes, reducing blurring, retaining detail, and maintaining perceptual sharpness and texture. It’s my go-to encoder these days when I need to get a good looking image at crazy-low bitrates.

MB-tree

The new Macroblock rate control tree mode optimizes quality based on how future frames reference different macroblocks of the current frame. That lets it shift bits to parts of the image that have a long-term impact on quality away from transient details that may last for just a frame. This yields a massive efficiency improvement in CGI and cel animation, and a useful improvement for film/video content.

Single-slice multithreading

It can use 16 cores with a single-slice encode for maximum efficiency. This is a significant advantage over the Main Concept and Microsoft implementations.

CRF mode

This is a perceptually tuned variant of constant quantizer encoding, allowing a target quality and maximum bitrate to be set, and the encode will be as small as it can be while hitting the visual quality target and maintaining VBV. It’s a great feature for file-based delivery, where we care more about “good enough and small enough” than any particular bitrate.

Rapid development

There can be new builds with new features or tweaks quite often. On the other hand, these rapid changes can be difficult to stay on top of.

Telestream

Telestream’s H.264 codec (Figure 14.7) is derived from PopWire’s Compression Master, and so has many advanced features of use in the mobile and broadcast industries, a la Main Concept. It includes some unique features:

Figure 14.7 (a) Episode has all the right basic controls, including a choice of specifying buffer as VBV or bitrate. Oddly, its multiple reference frames maxes out at 10. (b) Episode uses its usual Lookahead–based 2–pass mode for H.264 as well. It can also do High 4:2:2, unlike many other encoders. (c) Episode also has some more unusual settings that allow streams to meet some unusual requirements, like easy switching between bitrates. It uses multithreaded slicing.

It had some quality limitations in older versions, but as of Episode 5.1 is competitive.

QuickTime

Apple’s encoder is relatively weak, particularly beyond Baseline. It is Baseline/Main only and its implementation lacks three key features:

In fact, Apple’s Main is pretty much Baseline + B-frames.

There’s very little control in QuickTime’s export dialog (Figure 14.8), but here are a few tips:

Figure 14.8 QuickTime’s export settings. It’s the same underlying codec, but you get 14.6a and 14.6b when exporting to a MPEG – 4 file (I hate modal dialogs), but a nice unified one in 14.6c when exporting to .mov. Note the CD – ROM/Streaming/Download control is only available in multi–pass mode. We lack good documentation on the difference between CD – ROM and Download; I presume Streaming is CBR.

In general, QuickTime’s exporter is a fine consumer-grade solution, but I can’t imagine using it for quality-critical content.

Microsoft

Microsoft first shipped an H.264 encoder with Expression Encoder 2 SP1, which was a quite limited implementation just for devices. Windows 7 adds much improved H.264 support, although the implementation in Windows itself is focused on device support.

Windows 7 also supports hardware encoding, which can make transcoding for devices as fast as a file copy.

A fuller software implementation with a bevy of controls is included in Expression Encoder 3, for both .mp4 and Smooth Streaming .ismv files.

EEv3’s H.264 (Figure 14.9) is quite a bit better than QuickTime’s, although it’s also missing a few quite useful features:

Figure 14.9 Expression Encoder 3’s H.264 controls. I like having a single–pane nonmodal control like this.

However, unlike QuickTime, it does support multiple reference frames, peak buffer size, and CABAC. It also has higher complexity modes that can deliver quite good quality within those limitations. I look forward to a future version with High Profile, of course.

H.265 and Next – Generation Video Codec

Of course, H.264 isn’t the end of codec development by any means. Its replacement is already in early stages. Recognizing that many of H.264’s most computationally expensive features weren’t the source of the biggest coding efficiency gains, a new codec could provide a mix of better compression and easier implementation. The VQEG has ambitious targets:

It’s too early to say if either or both will be met, but getting close to either should be as disruptive to H.264 as H.264 was to MPEG-2. These improvements may be addressed via an extension to H.264 or an all-new H.265. Both are very much research projects at this point. A H.264 extension could potentially emerge in 2010, but a full H.265 is unlikely before 2012 at the earliest.