After getting the latest source code revision from the FFmpeg SVN repository, I generated a list of the demuxers that are available in the libavformat library, which is included with FFmpeg (see Figure 4-1). I noticed that FFmpeg separates most demuxers in different C files under the directory libavformat/.

Figure 4-1. FFmpeg libavformat demuxers

Next, I tried to identify the input data processed by the demuxers. While reading the source code, I discovered that most demuxers declare a function called demuxername_read_header(), which usually takes a parameter of the type AVFormatContext. This function declares and initializes a pointer that looks like this:

[..]
ByteIOContext *pb = s->pb;
[..]

Many different get_something functions (e.g., get_le32(), get_buffer()) and special macros (e.g., AV_RL32, AV_RL16) are then used to extract portions of the data pointed to by pb. At this point, I was pretty sure that pb had to be a pointer to the input data of the media files being processed.

I decided to search for bugs by tracing the input data of each demuxer at the source code level. I started with the first demuxer file from the list, called 4xm.c. While auditing the demuxer of the 4X movie file format,^[40] I found the vulnerability shown in the listing below.

[..]
 93    static int fourxm_read_header(AVFormatContext *s,
 94                                  AVFormatParameters *ap)
 95    {
 96      ByteIOContext *pb = s->pb;
 ..
101      unsigned char *header;
 ..
103      int current_track = −1;
 ..
106      fourxm->track_count = 0;
107      fourxm->tracks = NULL;
 ..
120       /* allocate space for the header and load the whole thing */
121       header = av_malloc(header_size);
122       if (!header)
123           return AVERROR(ENOMEM);
124       if (get_buffer(pb, header, header_size) != header_size)
125           return AVERROR(EIO);
 ..
160      } else if (fourcc_tag == strk_TAG) {
161          /* check that there is enough data */
162          if (size != strk_SIZE) {
163              av_free(header);
164              return AVERROR_INVALIDDATA;
165          }
166          current_track = AV_RL32(&header[i + 8]);
167          if (current_track + 1 > fourxm->track_count) {
168             fourxm->track_count = current_track + 1;
169             if((unsigned)fourxm->track_count >= UINT_MAX / sizeof(AudioTrack))
170               return −1;
171             fourxm->tracks = av_realloc(fourxm->tracks,
172                 fourxm->track_count * sizeof(AudioTrack));
173             if (!fourxm->tracks) {
174               av_free(header);
175               return AVERROR(ENOMEM);
176             }
177          }
178          fourxm->tracks[current_track].adpcm = AV_RL32(&header[i + 12]);
179          fourxm->tracks[current_track].channels = AV_RL32(&header[i + 36]);
180          fourxm->tracks[current_track].sample_rate = AV_RL32(&header[i + 40]);
181          fourxm->tracks[current_track].bits = AV_RL32(&header[i + 44]);
[..]

The get_buffer() function in line 124 copies input data from the processed media file into the heap buffer pointed to by header (see lines 101 and 121). If the media file contains a so-called strk chunk (see line 160) the AV_RL32() macro in line 166 reads an unsigned int from the header data and stores the value in the signed int variable current_track (see line 103). The conversion of a user-controlled unsigned int value from the media file to a signed int could cause a conversion bug! My interest piqued, I continued to search through the code, excited that I might be on to something.

The if statement in line 167 checks whether the user-controlled value of current_track + 1 is greater than fourxm->track_count. The signed int variable fourxm->track_count is initialized with 0 (see line 106). Supplying a value >= 0x80000000 for current_track causes a change in sign that results in current_track being interpreted as negative (to find out why, see Section A.3). If current_track is interpreted as negative, the if statement in line 167 will always return false (as the signed int variable fourxm->track_count has a value of zero), and the buffer allocation in line 171 will never be reached. Clearly, it was a bad idea to convert that user-controlled unsigned int to a signed int.

Since fourxm->tracks is initialized with NULL (see line 107) and line 171 is never reached, the write operations in lines 178–181 lead to four NULL pointer dereferences. Because NULL is dereferenced by the user-controlled value of current_track, it’s possible to write user-controlled data at a wide range of memory locations.

The expected behavior of FFmpeg is shown in Figure 4-2 as follows:

Figure 4-2. Expected behavior when FFmpeg operates normally

Figure 4-3 shows what happens when this bug affects FFmpeg:

Figure 4-3. Unexpected behavior of FFmpeg causing memory corruption

What a beautiful bug!