The H.261 codec was developed by the ITU (International Telecommunications Union). H.261 is a legacy codec used for only two purposes:

Table A-1 shows the video frame parameters for H.261.

Technically, H.261 defines frames that may occur only at intervals corresponding to 29.97 Hz. However, it allows the encoder to send a lower frame rate, achieved by “dropping” a pre-fixed number of consecutive frames between each sent frame. If the encoder plans to send video at a fixed lower frame rate, that lower frame rate is not signaled in the bitstream, but it can be transmitted out of band.

Table A-2 shows the characteristics of motion vectors (MV) for H.261 coding. The allowable options are the simplest for any of the standard codecs and allow a single MV per macroblock (MB).

H.261 does not perform prediction of pixel values in the spatial domain; the algorithm applies a discrete cosine transform (DCT) directly to either original pixel values or residual pixel values. Table A-3 shows the characteristics of DCT coding for H.261.

Table A-4 shows the characteristics of the quantization process used in the H.261 standard, after the DCT.

Table A-5 shows the methods by which the H.261 codec applies entropy coding to each bitstream element. H.261 uses a fixed-length code for the intra DC coefficients and variable-length coding (VLC) for other elements of the bitstream. H.261 does not use an arithmetic coder.

Table A-6 shows that two of the more significant advanced features commonly available in other codecs are not options for H.261.

Table A-7 shows that H.261 offers forward error correction as the only kind of built-in data resiliency.

The H.261 codec is the only codec in this appendix that applies a loop filter to the predicted frame. This loop filter simply blurs the reconstructed frame before the frame is used as a predicted frame. The loop filter might be necessary when the video sequence has objects with sharp edges. Because H.261 is limited to MVs with single-pel accuracy, sharp edges in the original frame and predicted frame might not line up. The result is a residual image containing high-frequency edge-difference information. These high frequency-edges result in larger values for AC DCT coefficients and in turn increase the bit rate of the output stream. The goal of the loop filter is to blur the predicted frame slightly, to soften the sharp edges of the residual image. Other codecs use MVs with subpixel accuracy and do not have this alignment problem.

The H.263 codec was developed by the ITU. H.263 and went through three iterations. The first version of the standard was finalized in 1995 and added many enhancements relative to H.261. In the following tables, this version is referred to as Base H.263. The next two iterations of H.263 were issued in 1998 and 2000, with the following further enhancements:

H.263v3 encompasses Base H.263 and H.263v2. This discussion covers H.263v3 and includes the common annexes supported by many codecs. The next section, “Additional H.263 Annexes,” covers the remaining annexes.

The H.263 codec defines nine profiles and multiple levels in annex X. The profiles define the allowed features, such as the use of B-frames. Levels define allowable frame sizes, bit rates, and frame rates.

Table A-8 shows the source video formats possible with H.263.

The segmentation of each frame into GOBs and MBs is similar to H.261: Each frame is segmented into GOBs or slices. GOBs span the entire width of an image and consist of rows of MBs.

Table A-9 shows the features and limitations of MVs in H.263.

Table A-10 shows the attributes and characteristics of the H.263 DCT.

For intrablocks in either intra- or interframes, annex I allows three possible modes for intra coefficient prediction, signaled explicitly. This prediction uses information from the block to the left and the block above the current block. Figure A-1 shows the blocks used in the calculations.

Figure A-1. Coefficient Prediction in H.263

Using these surrounding blocks, the calculations for the three modes are as follows:

Table A-11 shows the basic attributes and characteristics of the quantization process in H.263.

Table A-12 shows the variations in quantizer capabilities for intra AC coefficients and inter AC + inter DC coefficients for H.263.

In H.263 annex I (advanced intra coding mode), all modes use a special VLC table for both DC and AC coefficients. In addition, the DC coefficient is handled the same as the AC coefficients.

Table A-13 shows the attributes and characteristics of entropy coding in H.263.

Base H.263 does not support B-frames; however, B-frames are supported in annex O. B-frames in direct mode do not include a delta MV to compensate for a deviation from linear prediction. The possible prediction modes for B-frames include the following:

In addition to B-frames in annex O, H.263 offers PB-frames, defined in annexes G and M. A single PB-frame consolidates two consecutive frames: a frame represented by the B-part of a PB-frame, followed by a frame represented by the P-part of a PB-frame. To prevent confusion, use the term B-frame to refer to only the standalone B-frames of annex O, and use the terms B-part and P-part to refer to the respective parts of a PB-frame, defined in annexes G and M. For PB-frames, H.263 consolidates the B-part and the P-part for several reasons:

For the definition of PB-frames, annex M supersedes annex G: Use Annex G only if you need to interoperate with legacy equipment that is limited to annex G.

Table A-14 shows the PB-frame capabilities of the Base H.263 codec and the enhanced capabilities of annexes G and M.

The forward prediction mode of annex M applies only a single forward MV to the B-part of the MB. This mode is useful if there is a scene cut immediately after the B-part and before the next P-part.

The backward prediction mode of annex M is used if the B-part of the PB-frame immediately follows a scene cut. In this case, the prediction just uses the corresponding block in the following P-part, without using MVs.

The Base H.263 codec cannot implement scalability. However, annex O supports three scalability options:

In H.263, spatial and SNR scalability is achieved with layers of enhancement. Each layer of enhancement provides a residual layer, which is added to the underlying reconstructed layer to produce a new reconstructed layer. Each residual layer is created by taking the difference between the original image sequence and the underlying reconstructed layer. This process is essentially a prediction loop, which uses the underlying reconstructed layer as the prediction. The prediction loop then codes the residual difference between the prediction layer and the original layer. However, in this case, no MVs are used to create the prediction.

The residual layer can be coded using EI pictures and EP pictures:

The Base H.263 codec supports GOBs, but not slices. Annex K supports slices and provides two features related to slices:

Annex K imposes several restrictions on slices:

Data independence is supported in annex R, which ensures that no slices have dependencies with each other. When the bitstream uses GOBs rather than slices, a GOB may omit the GOB header, which indicates that the GOB uses the same state information from the previous GOB. However, when a GOB header is present, it means that the GOB changes the state information. Annex R prevents inter GOB dependency if a GOB header is present. Annex R ensures data independence for the following:

Annex R imposes several restrictions:

Annex V provides data partitioning. It reorganizes the bitstream as follows:

Markers separate each of the sections. Annex V also specifies the use of a reversible VLC table for the MVs. In addition, annex V must be used with annex K. Annex V can be used with annex O, cannot use annex E (arithmetic coding) or U, and should not be used with annex H.

Finally, H.263 allows the use of FEC, detailed in annex H.

This section delves into less frequently used annexes. These annexes might be used in video conferencing endpoints in the future to implement specialized functionality.

The H.264 codec was jointly developed by two standards bodies: the ITU and the ISO/IEC (International Organization for Standardization / International Electrotechnical Commission). As a result, H.264 can be found in two different documents: the ITU document H.264, and the ISO document MPEG-4, Part 10. H.264 is also known by its more generic name AVC, for Advanced Video Codec.

H.264 has superior performance compared to previous standards such as H.263 or MPEG-4, Part 2. For the same perceptual quality or peak signal-to-noise ratio (PSNR), H.264 generates bit rates that are 30 percent to 50 percent less than H.263 or a MPEG-4 simple profile. However, this improvement in performance comes at a cost of CPU cycles. H.264 encoders may have a CPU load that is about four times that of other codecs, and H.264 decoders may consume up to three times the CPU load of other decoders.

All profiles of H.264 have several distinguishing features:

Each profile also defines a set of features optimized for a particular application. The H.264 Baseline profile is intended for video conferencing and wireless, with the following attributes:

H.264 Extended profile is designed for streaming over the Internet and contains all baseline features in addition to these attributes:

H.264 Main profile is intended for broadcast and entertainment, with these characteristics:

A later addition to H.264 included the Fidelity Range Extensions (FRExt), which added the following capabilities:

Profiles added by FRExt include the following:

H.264 also defines multiple levels that place upper limits on the bit rate, frame size, total buffer size, and so on.

Motion Vectors

For the purpose of assigning MVs, each 16×16 MB may be segmented in several ways: as a 16×16 block, as two 8×16 blocks, as two 16×8 blocks, or as four 8×8 blocks. The four 8×8 segmentation mode allows any of the 8×8 blocks to be further subdivided as two 4×8 blocks, two 8×4 blocks, or four 4×4 blocks, as shown in Figure A-2.

Figure A-2. Segmentation of a Macroblock in H.264

As a result, an H.264 MB may contain 16 4×4 blocks, and in a B-frame, each block may have up to two MVs, for a total of 32 possible MVs per MB. Further, each MV may refer to a different reference frame. Each MV may also have an associated weighting, w, and offset, o, and these parameters are applied to the predicted pixels referenced by the MV. The weighting and offset are useful to generate predictions for scene cuts and cross-fades.

Luma MVs with an accuracy of 1/4 pel may be specified. Given a 1/4 pel accurate MV, the predicted pixels are interpolated in a two-step process:

• MVs at 1/2 pel locations are calculated with high precision using a six-tap one-dimensional filter.

• If necessary, 1/4 pel MVs are interpolated using bilinear interpolation on the 1/2 pel accurate values.

The codec achieves 1/8 pel resolution for chroma predicted blocks by using linear interpolation.

Table A-16 shows the features and limitations of MVs in H.264.

Table A-16. Motion Vectors for H.264

Motion Vector Attribute	Attribute Options
MV block sizes	Motion vectors can apply to block sizes of 16×16, 16×8, 8×16, 8×8, 8×4, 4×8, or 4×4
MVs per MB	Up to 32
MV prediction	Uses a median filter, applied to the MV of three nearby blocks
MV resolution	1/4 pel for luma, 1/8 pel for chroma
Luma 1/2 pel	Attained with a one-dimensional six-tap filter
Luma 1/4 pel	Attained with a bilinear filter applied to the two nearest 1/2 pel interpolated values
Chroma 1/8 pel interpolation	Linear interpolation
MV H / V range	Set by the level
Unrestricted MVs	Yes
MV restricted to the same slice	Yes

Intra Prediction

H.264 has an intra prediction mode that predicts pixels in the spatial domain before the intra transform process. For luminance, the encoder can use two different modes: a 16×16 prediction mode or a 4×4 prediction mode. For chrominance, the encoder can use an 8×8 prediction mode. In both cases, the pixels inside the block are predicted from previously decoded pixels adjacent to the block.

The 16×16 prediction mode has four methods of prediction. Figure A-3 shows two modes.

Figure A-3. Two of the Four Modes for 16×16 Intra Prediction in H.264, Showing Vertical Prediction and Horizontal Prediction

In the vertical prediction mode, the horizontal sequence of pixels just above the current block is used to predict the pixels inside the block by sweeping these pixels downward. The horizontal prediction mode uses the vertical line of pixels to the left and sweeps them horizontally. The 16×16 prediction mode also has two other variations:

• A DC prediction mode, where all interior pixels are estimated with a single value by using the average value of surrounding edge pixels.

• A planar prediction mode, which uses surrounding pixels at the top and left of the block to interpolate each pixel in the interior of the 16×16 block. The interpolation process operates by modeling the interior pixels as a plane in three dimensions, with the pixel values defining the height of the plane above the 16×16 grid.

Figure A-4 shows the edge pixels used for the 4×4 intra prediction modes.

Figure A-4. Intra Prediction for 4×4 Blocks for H.264

In addition to the nine neighboring pixels to the left and above the 4×4 block, this prediction mode uses four previously decoded pixels to the right. Eight of the nine variations of the 4×4 intra prediction mode use simulated motion to interpolate the interior pixels. Each of these modes corresponds to a direction, shown by the eight direction arrows. The edge pixels are “swept” over the 4×4 pixel block to interpolate the interior pixels. The bottom diagram in Figure A-4 shows the interpolation dependencies of two interior pixels for mode 3 as a function of edge pixels. In addition to the eight simulated motion directions, the 4×4 intra prediction mode has a DC prediction, where all interior pixels are estimated with a single value by using the average value of surrounding edge pixels.

Figure A-5 shows two of the four prediction modes for 8×8 chrominance blocks.

Figure A-5. Two of the Four Modes for 8×8 Intra Prediction in H.264 for Chrominance Blocks, Showing Vertical Prediction and Horizontal Prediction

Like the 16×16 prediction mode, the other two modes for chrominance consist of a DC prediction mode and a planar prediction mode.

Integer Transform

Unlike the 8×8 transform of most other codecs, H.264 initially defined a 4×4 integer-based transform. The transform provides almost as much frequency separation as the 8×8 DCT but has a simpler integer implementation. The FRExt subsequently added the option of an 8×8 integer-based transform.

H.264 takes a two-stage approach when applying the 4×4 transform:

• As shown in Figure A-6, when the MB is segmented into 16 4×4 blocks, and when the MB is entirely intracoded, the DC coefficients from each 4×4 block are arranged in a new 4×4 block and then are transformed using a simplified 4×4 transform.

• As shown in Figure A-7, for all modes (intra and inter), a similar process is applied to the DC coefficients of the chroma blocks: The DC values are arranged into a 2×2 array and then are processed with a 2×2 transform.

Figure A-6. H.264 Transform Processing for Luminance (Intra Mode Only)

Figure A-7. Transform Processing for Chrominance (Intra or Inter Modes)

Quantization

Table A-17 shows the characteristics of the quantizer for H.264.

Table A-17. Quantization for H.264

Quantization Attribute	Attribute Options
Quantizer dead zone	No.
Quantizer clipping	No upper limit on the input quantizer value.
Matrix quantization	Except for secondary DC 4×4 luma and DC 2×2 chroma mode, H.264 always uses matrix quantization, based on fixed quantization tables.
Step size	Q has a range of [0, 51] and is used to generate the matrix of quantization values.
	Q can change by a large amount per MB [–26, +25].
Intra DC coefficient	The quantizer is less coarse than for other coefficients.
Chroma Q values	For higher Q values, step sizes for chroma are lower.

An innovation in the H.264 codec is backward zigzag scanning in the entropy coder. Instead of scanning from large-coefficient values (upper left of the transform) to small-coefficient values (lower right of the transform), the entropy coder scans in the reverse direction. This approach allows the adaptive VLC process to better predict future coefficient values.

In addition, the codec takes advantage of a characteristic of most transform coefficients: Nonzero coefficients near the lower-right corner of the transform often have a value of either +1 or –1. The entropy coder has a “trailing 1s” special case to efficiently code these values.

The only variation on the zigzag scan is used for field coding, as shown in Figure A-8.

Figure A-8. Two Zigzag Scanning Patterns of H.264

When field coding, each MB contains data from only a single field. As a result, the content in a field-coded MB contains every other line of video from the frame, which means there is less vertical pixel-to-pixel correlation. As a result, the MBs tend to have high vertical frequencies, corresponding to larger coefficients on the left half of the 4×4 block. As a result, the zigzag pattern for field coding scans the block from left to right.

The MPEG-4, Part 2 codec was developed by the ISO/IEC. The formal codec designation is ISO/IEC 14496-2. It is used mostly in 3G mobile phones, still cameras, and IP video cameras.

MPEG-4, Part 2 defines several profiles. One of the profiles is called the Short Header profile, which is simply the H.263 video stream encapsulated with MPEG-4 video stream headers.

For video conferencing, the common profiles for MPEG-4, Part 2 consist of the Simple profile and the Advanced Simple profile (ASP).

The following list describes the Simple profile:

ASP includes all features in the Simple profile, plus the following:

Table A-23 identifies the annexes and codecs supported by different enterprise endpoints.

Table A-24 summarizes the major features of each codec. This table reflects the capabilities of all optional annexes of each codec.

In general, the codec descriptions reveal that advanced codecs offer more flexibility for the encoder at each stage of the pipeline. The H.264 codec can assign up to 32 MVs per MB, with each MV pointing to a different reference image and carrying a different weighting value. In contrast, the simpler H.261 codec applies a single MV to an MB. However, to take full advantage of this flexibility, the H.264 encoder requires significantly more CPU power.

ITU-T Recommendation H.264 / ISO/IEC 14496-10, March 2005. Advanced video coding for generic audiovisual services.

ITU-T Recommendation H.261, March 1993. Video Codec for Audiovisual Services at p x 64 kbits.

ITU-T Recommendation H.263, January 2005. Video coding for low bit rate communications.

MPEG-4, Part 2: ISO/IEC 14496-2: Coding of audio-visual objects—Part 2, Visual, Third Edition. May 2004.

Wiegand, T., H. Schwarz, A. Joch, F. Kossentini, and G. J. Sullivan. Rate-Constrained Coder Control and Comparison of Video Coding Standards. IEEE Transactions on Circuits and Systems for Video Technology, Vol. 13, Issue 7. July 2003.