The fields in this first octet of the RTP header are described as follows:

RFC 3550 defines payload type as a 7-bit field that identifies the codec type and sample rate of media carried in the packet. When the endpoint or conference server receives an RTP packet, it uses the payload type to determine how to interpret the payload. The numeric value of the payload type may be predefined (called static payload types in the range of 0 to 96) or can be dynamically assigned during the capability negotiation between the conference server and the endpoint. There is one important distinction: For the static payload types, the clock rate is specified in the payload format. When using SIP signaling with dynamic payload types, the clock rate should be defined in the appropriate attribute line of the Session Description Protocol (SDP) offer. For example, G.711µ-Law uses a static payload type of 0, and the clock rate is defined in RFC 3551. H.264 is a dynamic payload type, and the clock rate is 90 kHz, which is specified in the SDP as follows:

m = video rtp port number RTP/AVP 97
a = rtpmap:97 H.264/90000

The sequence number is a two-octet field that identifies the order in which RTP packets were transmitted. The sequence number allows the receiver to detect packets that were dropped on the network and allows the receiver to handle out-of-order packets. The sender increments the sequence number by 1 for each RTP packet it sends. As defined in RFC 3550, the endpoint or conference server should choose the initial value of the sequence number at random, rather than starting from 0, to prevent known-value encryption attacks.

The time stamp is a 32-bit integer that increments at the media-dependent rate. As stated in RFC 3550, the time stamp reflects the sampling instance of the first octet of the media data in the RTP packet. As with sequence numbers, senders should choose a random value for the time stamp of the first packet, rather than starting at 0. The time stamp will also wrap around to 0 if it exceeds its maximum 32-bit value. The sender must transmit packets according to the real-time rate of the media, which means that if the sender issues packets with a fixed number of media samples, the delay between RTP packet transmissions should also be fixed. Table 4-1 shows audio sampling rates and their packet sizes.

In the case of MPEG bitstreams, which transmit frames out of order, the sender may transmit the RTP packets with out-of-order time stamps, but the sequence numbers will still increase. The receiver must reconstruct the data and play out the media accordingly based on the RTP time stamps. Also, note that a frame of video bitstream may be fragmented across multiple packets, which means that each packet will have the same RTP time stamp, but the sequence numbers will increase.

RTP packetization for audio codecs uses an RTP time-stamp clock that is the same as the sample clock, which means that the sampling clock increases by 1 for each sample. As a result, RTP time stamps for audio are essentially sample indexes. For example, an endpoint uses an audio codec with an 8000-Hz sample rate and an H.261 video codec. Because the sample rate of the audio stream is 8000 Hz, the RTP time stamp uses a sample clock of 8000 samples/second. If an audio stream packet has a size of 20 ms, the number of samples in that packet is 160, and therefore, the size of the packet is 160 RTP time-stamp units. H.261 uses an RTP sample clock of 90 kHz, which means a 29.97 FPS. An H.261 video stream will have a duration between frames of 33.37 ms, and the RTP time-stamp duration will equal 33.37 ms * 90,000 samples/second = 3003 RTP time-stamp units. The sender must assign RTP time stamps based on the absolute position in the source stream, which means that the RTP time-stamp sequence must account for packets not sent because of silence suppression at the sender.

The SSRC is a 32-bit field that serves as a unique identifier for an instance of an RTP stream. The originator of the RTP connection should choose this value at random. No two RTP streams within the same RTP session can have the same SSRC value. If the endpoint or the conference server changes the source IP address, the RTP packet stream must change to use a new SSRC value.

In a conference session, each endpoint transmits audio and video RTP packets to the audio mixer. The audio mixer then picks the top three or four speakers, mixes them, and sends the resulting output stream back to the endpoints. The output RTP packets should include the CSRC field, which is a list of the SSRC values of all participants selected for the mix. The audio mixer sets the CC bit in the first octet of the header to indicate the presence of a CSRC list. Many conferencing systems do not include this CSRC list because the endpoints are not conference-aware.

The RTP packetization method for the media is defined by a payload format definition, which is unique to each codec. A payload format might define a payload header, which resides in each RTP payload. The primary purpose of this header is to convey the state of the encoder to the destination. If the network drops packets, the receiver may use this state information to continue decoding the bitstream after the dropped packets. For instance, Figure 4-3 shows the format of the H.263 RTP packet.

Figure 4-3. H.263 RTP Packet

The payload header identifies (among other things) the group of blocks (GOB), slice, or macroblock (MB) index for data at the start of the packet. It also indicates whether this packet is part of an I-frame.

The payload is the actual media data sent and received between endpoints and the conference server. The payload may contain multiple audio frames, which means that the decoder may need to parse the bitstream to determine whether the packet contains more than one frame. RTP packets generally do not contain more than one frame of video; instead, single frames of video typically fragment across multiple RTP packets.

The following shows an example of the RTP header data structure with one CSRC identifier:

typedef struct _RTP_HEADER_ {
    u8          v_p_x_cc_m;
    u8          payload_type;
    u16         seq_number;
    u32         time_stamp;
    u32         ssrc;
    u32         csrc[1];
} rtp_header_t;

Cisco CallManager MTPs are RTP modules (also called RTP proxies) that serve one function: They terminate and re-originate RTP/RTCP streams without processing the RTP data. Only centralized call agents, like a Cisco CallManager, use these modules. Endpoints always connect directly to CallManager at the signaling level (SIP, H.323, SCCP). However, at the media level, CallManager may either connect two endpoints directly or may insert an MTP in the media path of the connection. As far as the endpoints are concerned, the MTP appears to be the other endpoint. MTPs provide several features:

Conference servers do not need MTPs to handle signaling protocol conversions, such as H.323 to SIP translation, because even though the signaling protocols change, the RTP and RTCP packets remain identical between the two signaling protocols.

MTP devices do not change any of the RTP header parameters or the payload. However, the process of terminating and then reoriginating RTP media packets does consume CPU resources on a call control server. Figure 4-12 shows Cisco CallManager inserting an MTP device into the media path between an endpoint and a conference server.

Figure 4-12. Media Termination Points

Transcoders convert RTP media from one codec to another, and transraters convert RTP media from one bit rate to a lower bit rate. For instance, if an endpoint supports only the Internet Low Bandwidth Codec (ILBC), and the conference server does not support ILBC, an audio transcoder between the endpoint and the conference server can perform the conversion.

In most implementations, audio mixers contain built-in transcoders, because the audio data must be decoded before the summation process. A transcoder is similar to a mixer with one input and one output: Each RTP header is rewritten with a new sequence number, time stamp, and other parameters. However, the SSRC stays the same. In addition, transcoders that change the code type must create new RTCP packets for the output stream.

Video conferencing systems use transraters to lower the bit rate of video or audio RTP streams. For instance, a transrater may convert high-bandwidth 704-kbps H.264 30 frames per second to low-bandwidth 320-kbps H.264 30 frames per second. Transraters handle RTP headers in the same way as RTP transcoders.

Figure 4-13 shows CallManager inserting a transcoder between the endpoint and the conference server to transcode between G.729 and G.711 audio codecs. Figure 4-13 also shows a transrater in the media path between the endpoint and the conference server. Transraters are usually built in to the conference server (or video multipoint control unit [MCU]) and not as a separate device.

Figure 4-13. Transcoder

RFC 2190 defines the payload format for encapsulating H.263-1996 (H.263 or H.263v1) bitstreams in RTP packets. For this version, the payload format defines three modes for the H.263 payload header: mode A, mode B, and mode C. An RTP packet can use one of these three modes, depending on the desired packet size and the encoding options used in the H.263 bitstream. The F and P fields in the payload header determine the mode. The endpoints and the conference servers must be prepared to receive packets in any mode. These modes are not negotiated in the SDP offer/answer.

The next sections describe the three H.263 modes.

Mode A

In mode A, an H.263 payload header of 4 bytes is present before the actual payload, as shown in Figure 4-19. Note that mode A packets always start with a picture code or a GOB start code.

Figure 4-19. H.263 Mode A Payload Header

Table 4-2 explains the different bit fields.

Table 4-2. H.263-1996 Mode A Bit Fields

Bit Field	Size (in Bits)	Description
F	1	Indicates the mode of the payload header. A value of 0 indicates mode A; a value of 1 indicates mode B or mode C.
P	1	When F=1, a P bit value of 0 indicates mode B, and a value of 1 indicates mode C.
SBIT	3	Start bit position. Specifies the number of most significant bits ignored in the first byte.
EBIT	3	End bit position. Specifies the number of least significant bits ignored in the last data byte.
SRC	3	Specifies the resolution of the current picture.
I	1	Set to 0 for an intracoded frame and to 1 for intercoded.
U	1	Set to 1 if the Unrestricted Motion Vector option, bit 10 in PTYPE defined by H.263, was set to 1 in the current picture header otherwise, set to 0.
S	1	Set to 1 if the Syntax-based Arithmetic Coding option, bit 11 in PTYPE defined by H.263, was set to 1 in the current picture header; otherwise, set to 0.
A	1	Set to 1 if the Advanced Prediction option, bit 12 in PTYPE defined by H.263, was set to 1 in the current picture header; otherwise, set to 0.
R	4	Reserved: These bits must be set to 0.
DBQ	2	Differential quantization (DBQ)—the value should be the same as DBQUANT defined by H.263. Set to 0 if the PB-frames option is not used.
TRB	3	Temporal Reference for the B frame as defined by H.263. It is set to 0 if the PB-frames option is not used.
TR	8	Temporal Reference for the P-frame as defined by H.263. It is set to 0 if the PB-frames option is not used.

Example 4-1 shows the Ethereal capture of an H.263v1 mode A frame. Note that the F bit is set to 0.

Example 4-1. H.263 Ethereal Packet Trace

Frame 4733 (1327 bytes on wire, 1327 bytes captured)
Ethernet II, Src: 00:14:38:be:ec:57, Dst: 00:13:20:12:b5:5d
Internet Protocol, Src Addr: 172.27.75.146 (172.27.75.146), Dst Addr:
  172.27.75.187 (172.27.75.187)
User Datagram Protocol, Src Port: 21468 (21468), Dst Port: 5445 (5445)
Real-Time Transport Protocol
   10.. ....=Version: RFC 1889 Version (2)
   ..0. ....=Padding: False
   ...0 ....=Extension: False
   .... 0000=Contributing source identifiers count: 0
   0... ....=Marker: False
   .010 0010=Payload type: ITU-T H.263 (34)
Sequence number: 59
Timestamp: 117123
Synchronization Source identifier: 2887470011
ITU-T Recommendation H.263 RTP Payload header (RFC2190)
   F: False
p/b frame: False
Start bit position: 0
End bit position: 0
SRC format: CIF 352x288 (3)
Inter-coded frame: True
Motion vector: False
Syntax-based arithmetic coding: False
Advanced prediction option: False
Reserved field: 0
Differential quantization parameter: 0
Temporal Reference for B frames: 0
Temporal Reference for P frames: 0
H.263 stream: 000081DA0E043FFFFC03958989935EC9AF76C8B3A07FFFFF...

Mode B

In mode B, an H.263 bitstream can be fragmented at MB boundaries. Whenever the packet starts at an MB boundary, this mode is used without the PB-frames option. Figure 4-20 shows the mode B payload header.

Figure 4-20. H.263-1996 Mode B Payload Header

The fields F, P, SBIT, EBIT, SRC, R, I, U, S, and A are defined as in mode A. The F bit should be set to a value of 1, and the P bit should be set to a value of 0. Table 4-3 explains the remaining fields.

Table 4-3. H.263-1996 Mode B Fields

Bit Field	Size	Description
QUANT	5	Quantization value for the first MB coded at the beginning of the packet. Set to 0 if the packet begins with a GOB header.
GOBN	5	GOB number in effect at the start of the packet. GOB numbers are specified differently for different resolutions. Refer to RFC 2190 for details.
MBA	9	Macroblock address (MBA)—the address within the GOB of the first MB in the packet, counting from 0 in scan order. For example, the third MB in any GOB is given MBA = 2.
HMV1	7	Horizontal motion vector predictors for the first MB in this packet.
VMV1	7	Vertical motion vector predictors for the first MB in this packet.
HMV2	7	Horizontal motion vector predictors for block number 3 in the first MB in this packet when four motion vectors are used with the advanced prediction option. This information is needed because block number 3 in the MB needs different motion vector predictors than other blocks in the MB. These two fields are not used when the MB has only one motion vector. Refer to RFC 2190 for block organization in an MB.
VMV2	7	Same as HMV2, except that this entry is for vertical motion vector predictors.

Mode C

In mode C, an H.263 bitstream is fragmented at MB boundaries of P-frames if those P-frames have the PB-frames option set. This mode is intended for GOBs whose sizes are larger than the maximum packet size allowed in the underlying protocol when the PB-frames option is used. The F bit is set to 1, and the P bit is set to 1 to indicate mode C. Figure 4-21 shows the mode C payload header.

Figure 4-21. H.263-1996 Mode C Payload Header

The bit fields are defined the same as in mode A and mode B. The only exception is the 19-bit RR field—these bits are reserved and are set to 0.

Most H.263 endpoint implementations use mode A because of its simplicity and small payload header.

RFC 2429 defines the payload header format for H.263+ (also known as H.263-1998) and H.263++ (also known as H.263-2000) codecs. These codecs are also referred to as H.263v2. The payload header consists of a mandatory fixed part of two octets followed by a variable-length optional header. Figure 4-22 shows the packet structure.

Figure 4-22. H.263v2 Packet Structure

Table 4-4 explains the different bit fields used in the payload header.

VRC is an optional mechanism intended for error resiliency in packet networks. If V is set to 1, a 1-byte header is attached immediately after the 2-byte fixed payload header.

If P is set to 0, this packet indicates a follow-on stream, meaning that it does not include the start of a picture or a slice. If P is set to 1, this packet contains the start of a picture or a slice. (2 bytes of 0 bits then have to be prefixed to the payload of such a packet to compose a complete picture, GOB, slice, end of sequence [EOS], or end of sub-bit stream [EOSBS] start code.)

The packet may contain an optional extra picture header, which is indicated by a nonzero PLEN value. Encoders add extra picture headers to provide greater error resilience. The value of PLEN indicates the size of the extra picture header.

The actual payload data for each picture consists of an optional picture header, followed by data for a GOB or slices, eventually followed by an optional end-of-sequence code, followed by stuffing bits. As shown in Figure 4-22, the picture header starts with a picture start code (PSC).

Table 4-5 explains the fields of the PSC for the H.263v2 codec

PTYPE has 13 bits and carries a variety of information. Bit 9 indicates the picture coding type and determines whether the packet carries a key frame (value 0) or P-frame (value 1). If bits 6–8 of the PTYPE field are set to 111, an extended header called a PlusType header is added to the payload header. If the PlusType header is added, bit 9 no longer indicates whether the packet carries a key frame, and this I-frame information is instead added to the PlusType header.

The PlusType header consists of the following three fields:

Video MCUs have a tricky requirement: They must be able to create a seamless output stream for each endpoint, while at the same time switching between multiple input streams based on the loudest speaker floor control policy. The problem is that if the MCU were to switch immediately to a new input stream, endpoints receiving that output stream would detect a discontinuity in the bitstream and would have to wait for a key frame from the new selected stream to resume decoding the stream. To make the output stream appear seamless, the MCU should delay the switch to a new input stream until a key frame arrives from that new input stream. The receiving endpoint detects the discontinuity, but it does not have to wait for a key frame, because the MCU provides a key frame immediately after the switch. The video encoders, depending on the codec and packetization, set bits in the RTP payload header to indicate the presence of an I-frame.

In H.263v1, an I bit in the RTP payload header indicates that the packet belongs to an I-frame. For H.263 mode A, a value of 0 in the twelfth bit from the start of the payload header indicates that the packet belongs to an I-frame. For mode B and mode C, a value of 1 in the thirty-third bit from the start of the payload header indicates that the packet belongs to an I-frame.

With H.263v2, the MCU must resort to deep packet inspection of the codec bitstream to identify the start of an I-frame. The H.263 PSC indicates the I-frame in one of two places, depending on the contents of the PTYPE field:

Figure 4-23 shows the format of a NAL header, which is the basic structure of an H.264 RTP packet. Per RFC 3984, all H.264 RTP packets contain the 1-byte NAL header field after the RTP header.

Figure 4-23. NAL Unit Packet Format

Table 4-6 explains the bit fields.

The Type field defines the packetization mode, as shown in Table 4-7.

The NAL unit type field indicates the type of the packet (and thus the structure of the RTP packet). There are three possible types:

Figure 4-24 shows the three possible H.264 packet types.

Figure 4-24. H.264 Packet Type Formats

The following sections describe the three NAL unit packet types in more detail.

The SNALU payload type contains only a single NAL unit in the payload. Figure 4-25 shows the format of the SNALU. It contains a 1-byte header (the fields are explained in Table 4-6 in the preceding section). The value of the Type field is in the range of 1 to 23.

Figure 4-25. Format of the SNALU RTP Packet

NAL units must be transmitted in the same order as their NAL unit decoding order, and the RTP sequence number should reflect this transmission order.

RFC 3984 defines two basic types of aggregation packets:

The STAP and MTAP packets must not be fragmented and should be contained within a single RTP packet. MTAP is not commonly used for video conferencing.

STAP

STAP aggregates NAL units with identical NALU-time. NALU-time is the value that the RTP time stamp would have if that NAL were transported in its own RTP packet. RFC 3984 defines two types of STAP packets:

• STAP-A—. NAL units in the aggregation packet share the same time stamp and appear in valid decoding order.

• STAP-B—. NAL units in the aggregation packet share the same time stamp and may not be in the correct decoding order.

Figure 4-26 shows the packet format of the STAP-A packet. The value of the Type field in the NAL header is set to 24. The Size field (two octets) indicates the size of the NAL unit in bytes, which includes the NAL unit header plus data.

Figure 4-26. H.264 STAP-A Packet Format

Figure 4-27 shows the format of the STAP-B packet. The type field is set to a value of 25. The STAP-B packet consists of a two-octet decoding order number (DON) that indicates the NAL unit decoding order. The DON is required because the transmission order and the decoding order might differ, and the DON indicates the decoding order.

Figure 4-27. H.264 STAP-B Packet Format

MTAP

MTAP aggregates NAL units with potentially different NALU times. RFC 3984 defines two types of MTAP packets:

• MTAP-16 (16-bit time-stamp offset)

• MTAP-24 (24-bit time-stamp offset)

Figure 4-28 shows the packet format of MTAP-16.

Figure 4-28. H.264 MTAP-16 Packet Format

• The Type field is set to a value of 27.

• The payload header contains a two-octet decoding order number base (DONB). The MTAP packet contains multiple NAL units. The DONB contains the value of DON for the first NAL unit in the MTAP packet.

Figure 4-29 shows the packet format of MTAP-16. The Type field is set to a value of 27.

Figure 4-29. H.264 MTAP-24 Packet Format

The choice between MTAP-16 and MTAP-24 is application-dependent. The only difference between the two packet formats is the length of the time-stamp offset field.

The fragmentation unit (FU) allows a sender to fragment a single NAL unit into several RTP packets. The sender of the FU packet must send the fragments in consecutive order with ascending RTP sequence numbers. The receiver should reassemble the NAL unit according to the same RTP sequence number. The RTP time stamp of an RTP packet carrying an FU is set to the NALU time of the fragmented unit.

RFC 3984 defines two types of fragmentation unit packets. Figure 4-30 shows the FU-A packet format, which consists of a one-octet NAL header, followed by a one-octet FU header, followed by FU payload.

Figure 4-30. H.264 FU-A Packet Format

Table 4-8 summarizes the Fragmentation Unit header fields.

Figure 4-31 shows the packet format of the FU-B packet. The packet structure of an FU-B packet is similar to that of an FU-A packet, except for the presence of a DON field.

Figure 4-31. H.264 FU-B Packet Format

If a NAL unit can fit into a single FU, the NAL unit should be fragmented. In other words, the fragmented NAL unit must not be transmitted in one FU, which means that the Start and End bit of the FU header cannot be set to 1 in the same FU packet. If a fragmentation unit is lost, the receiver should discard all remaining FUs of that NAL unit.

Key frame detection with H.264 packets is straightforward. Table 4-9 summarizes the steps to detect a key frame for each packet type.