Now that I’ve discovered a security vulnerability, I could disclose it in several ways. I could contact the software developer and “responsibly” tell him what I’ve found and help him to create a patch. This process is referred to as responsible disclosure. Since this term implies that other means of disclosure are irresponsible, which isn’t necessarily true, it is slowly being replaced by coordinated disclosure.

On the other hand, I could sell my findings to a vulnerability broker and let him tell the software developer. Today, the two primary players in the commercial vulnerability market are Verisign’s iDefense Labs, with its Vulnerability Contribution Program (VCP), and Tipping Point’s Zero Day Initiative (ZDI). Both VCP and ZDI follow coordinated-disclosure practices and work with the affected vendor.

Another option is full disclosure. If I chose full disclosure, I would release the vulnerability information to the public without notifying the vendor. There are other disclosure options, but the motivation behind them usually doesn’t involve fixing the bug (for example, selling the findings in underground markets).^[11]

In the case of the VLC vulnerability described in this chapter, I chose coordinated disclosure. In other words, I notified the VLC maintainers, provided them with the necessary information, and coordinated with them on the timing of public disclosure.

After I informed the VLC maintainers about the bug, they developed the following patch to address the vulnerability:^[12]

--- a/modules/demux/ty.c
+++ b/modules/demux/ty.c
@@ −1639,12 +1639,14 @@ static void parse_master(demux_t *p_demux)
     /* parse all the entries */
     p_sys->seq_table = malloc(p_sys->i_seq_table_size * sizeof(ty_seq_table_t));
     for (i=0; i<p_sys->i_seq_table_size; i++) {
-        stream_Read(p_demux->s, mst_buf, 8 + i_map_size);
+        stream_Read(p_demux->s, mst_buf, 8);
         p_sys->seq_table[i].l_timestamp = U64_AT(&mst_buf[0]);
         if (i_map_size > 8) {
             msg_Err(p_demux, "Unsupported SEQ bitmap size in master chunk");
+            stream_Read(p_demux->s, NULL, i_map_size);
             memset(p_sys->seq_table[i].chunk_bitmask, i_map_size, 0);
         } else {
+            stream_Read(p_demux->s, mst_buf + 8, i_map_size);
             memcpy(p_sys->seq_table[i].chunk_bitmask, &mst_buf[8], i_map_size);
         }
     }

The changes are quite straightforward. The formerly vulnerable call to stream_Read() now uses a fixed size value, and the user-controlled value of i_map_size is used only as a size value for stream_Read() if it is less than or equal to 8. An easy fix for an obvious bug.

But wait—is the vulnerability really gone? The variable i_map_size is still of the type signed int. If a value greater than or equal to 0x80000000 is supplied for i_map_size, it’s interpreted as negative, and the overflow will still occur in the stream_Read() and memcpy() functions of the else branch of the patch (see Section A.3 for a description of unsigned int and signed int ranges). I also reported this problem to the VLC maintainers, resulting in another patch:^[13]

[..]
@@ −1616,7 +1618,7 @@ static void parse_master(demux_t *p_demux)

 {
     demux_sys_t *p_sys = p_demux->p_sys;
     uint8_t mst_buf[32];
-    int i, i_map_size;
+    uint32_t i, i_map_size;
     int64_t i_save_pos = stream_Tell(p_demux->s);
     int64_t i_pts_secs;
[..]

Now that i_map_size is of the type unsigned int, this bug is fixed. Perhaps you’ve already noticed that the parse_master() function contains another buffer overflow vulnerability. I also reported that bug to the VLC maintainers. If you can’t spot it, then take a closer look at the second patch provided by the VLC maintainers, which fixed this bug as well.

One thing that surprised me was the fact that none of the lauded exploit mitigation techniques of Windows Vista were able to stop me from taking control of EIP and executing arbitrary code from the stack using the jmp reg technique. The security cookie or /GS feature should have prevented the manipulation of the return address. Furthermore, ASLR or NX/DEP should have prevented arbitrary code execution. (See Section C.1 for a detailed description of all of these mitigation techniques.)

To solve this mystery, I downloaded Process Explorer^[14] and configured it to show the processes’ DEP and ASLR status.

The output of Process Explorer, illustrated in Figure 2-8, shows that VLC and its modules use neither DEP nor ASLR (this is denoted by an empty value in the DEP and ASLR columns). I investigated further to determine why the VLC process does not use these mitigation techniques.

Figure 2-8. VLC in Process Explorer

DEP can be controlled by system policy through special APIs and compile-time options (see Microsoft’s Security Research and Defense blog^[15] for more information on DEP). The default system-wide DEP policy for client operating systems such as Windows Vista is called OptIn. In this mode of operation, DEP is enabled only for processes that explicitly opt in to DEP. Because I used a default installation of Windows Vista 32-bit, the system-wide DEP policy should be set to OptIn. To verify this, I used the bcdedit.exe console application from an elevated command prompt:

C:Windowssystem32>bcdedit /enum | findstr nx
nx                      OptIn

The output of the command shows that the system was indeed configured to use the OptIn operation mode of DEP, which explains why VLC doesn’t use this mitigation technique: The process simply doesn’t opt in to DEP.

There are different ways to opt a process in to DEP. For example, you could use the appropriate linker switch (/NXCOMPAT) at compile time, or you could use the SetProcessDEPPolicy API to allow an application to opt in to DEP programmatically.

To get an overview of the security-relevant compile-time options used by VLC, I scanned the executable files of the media player with LookingGlass (see Figure 2-9).^[16]

LookingGlass showed that the linker switch for neither ASLR nor DEP was used to compile VLC.^[18] The Windows releases of VLC media player are built using the Cygwin^[19] environment, a set of utilities designed to provide the look and feel of Linux within the Windows operating system. Since the linker switches that I mentioned are supported only by Microsoft’s Visual C++ 2005 SP1 and later (and thus are not supported by Cygwin), it isn’t a big surprise that they aren’t supported by VLC.

Exploit mitigation techniques of Microsoft’s Visual C++ 2005 SP1 and later:

Figure 2-9. LookingGlass scan result of VLC

See the following excerpt from the VLC build instructions:

[..]
Building VLC from the source code
=================================
[..]
- natively on Windows, using cygwin (www.cygwin.com) with or
 without the POSIX emulation layer. This is the preferred way to compile
 vlc if you want to do it on Windows.
[..]
UNSUPPORTED METHODS
-------------------
[..]
- natively on Windows, using Microsoft Visual Studio. This will not work.
[..]

At the time of this writing, VLC didn’t make use of any of the exploit mitigation techniques provided by Windows Vista or later releases. As a result, every bug in VLC under Windows is as easily exploited today as 20 years ago, when none of these security features were widely deployed or supported.