To generate a list of the IOCTLs of the kernel, I simply searched the kernel source code for the usual IOCTL macros. Every IOCTL is assigned its own number, which is usually created by a macro. Depending on the IOCTL type, the XNU kernel of OS X defines the following macros: _IOR, _IOW, and _IOWR.

osx$ pwd
/Users/tk/xnu-792.13.8

osx$ grep -rnw -e _IOR -e _IOW -e _IOWR *
[..]
xnu-792.13.8/bsd/net/bpf.h:161:#define BIOCGRSIG        _IOR('B',114, u_int)
xnu-792.13.8/bsd/net/bpf.h:162:#define BIOCSRSIG        _IOW('B',115, u_int)
xnu-792.13.8/bsd/net/bpf.h:163:#define BIOCGHDRCMPLT    _IOR('B',116, u_int)
xnu-792.13.8/bsd/net/bpf.h:164:#define BIOCSHDRCMPLT    _IOW('B',117, u_int)
xnu-792.13.8/bsd/net/bpf.h:165:#define BIOCGSEESENT     _IOR('B',118, u_int)
xnu-792.13.8/bsd/net/bpf.h:166:#define BIOCSSEESENT     _IOW('B',119, u_int)
[..]

I now had a list of IOCTLs supported by the XNU kernel. To find the source files that implement the IOCTLs, I searched the whole kernel source for each IOCTL name from the list. Here’s an example of the BIOCGRSIG IOCTL:

osx$ grep --include=*.c -rn BIOCGRSIG *
xnu-792.13.8/bsd/net/bpf.c:1143:        case BIOCGRSIG:

To identify the user-supplied input data of an IOCTL request, I took a look at some of the kernel functions that process the requests. I discovered that such functions typically expect an argument called cmd of type u_long and a second argument called data of type caddr_t.

Here are some examples:

[..]
135 int
136 at_control(so, cmd, data, ifp)
137      struct socket *so;
138      u_long cmd;
139      caddr_t data;
140      struct ifnet *ifp;
141 {
[..]

[..]
1025 int
1026 ifioctl(so, cmd, data, p)
1027     struct socket *so;
1028     u_long cmd;
1029     caddr_t data;
1030     struct proc *p;
1031 {
[..]

[..]
877 static int
878 vnioctl(dev_t dev, u_long cmd, caddr_t data,
879     __unused int flag, struct proc *p,
880     int is_char)
881 {
[..]

The names of these function arguments are quite descriptive: The cmd argument holds the requested IOCTL code, and the data argument holds the user-supplied IOCTL data.

On Mac OS X, an IOCTL request is typically sent to the kernel using the ioctl() system call. This system call has the following prototype:

osx$ man ioctl
[..]
SYNOPSIS
   #include <sys/ioctl.h>

   int
   ioctl(int d, unsigned long request, char *argp);

DESCRIPTION
   The ioctl() function manipulates the underlying device parameters of spe-
   cial files. In particular, many operating characteristics of character
   special files (e.g. terminals) may be controlled with ioctl() requests.
   The argument d must be an open file descriptor.

   An ioctl request has encoded in it whether the argument is an "in"
   parameter or "out" parameter, and the size of the argument argp in
   bytes. Macros and defines used in specifying an ioctl request are
   located in the file <sys/ioctl.h>.
[..]

If an IOCTL request is sent to the kernel, the argument request has to be filled with the appropriate IOCTL code, and argp has to be filled with the user-supplied IOCTL input data. The request and argp arguments of ioctl() correspond to the kernel function arguments cmd and data.

I had found what I was looking for: Most kernel functions that process incoming IOCTL requests take an argument called data that holds, or points to, the user-supplied IOCTL input data.

After I found the locations in the kernel where IOCTL requests are handled, I traced the input data through the kernel functions while looking for potentially vulnerable locations. While reading the code, I stumbled upon some locations that looked intriguing. The most interesting potential bug I found happens if the kernel tries to handle a special TTY IOCTL request. The following listing shows the relevant lines from the source code of the XNU kernel.

[..]
 816    /*
 817     * Ioctls for all tty devices.  Called after line-discipline specific ioctl
 818     * has been called to do discipline-specific functions and/or reject any
 819     * of these ioctl commands.
 820     */
 821    /* ARGSUSED */
 822    int
 823    ttioctl(register struct tty *tp,
 824       u_long cmd, caddr_t data, int flag,
 825       struct proc *p)
 826    {
[..]
 872       switch (cmd) {            /* Process the ioctl. */
[..]
1089       case TIOCSETD: {        /* set line discipline */
1090           register int t = *(int *)data;
1091           dev_t device = tp->t_dev;
1092
1093           if (t >= nlinesw)
1094               return (ENXIO);
1095           if (t != tp->t_line) {
1096               s = spltty();
1097               (*linesw[tp->t_line].l_close)(tp, flag);
1098               error = (*linesw[t].l_open)(device, tp);
1099               if (error) {
1100                   (void)(*linesw[tp->t_line].l_open)(device, tp);
1101                   splx(s);
1102                   return (error);
1103               }
1104               tp->t_line = t;
1105               splx(s);
1106           }
1107           break;
1108       }
[..]

If a TIOCSETD IOCTL request is sent to the kernel, the switch case in line 1089 is chosen. In line 1090, the user-supplied data of type caddr_t, which is simply a typedef for char *, is stored in the signed int variable t. Then in line 1093, the value of t is compared with nlinesw. Since data is supplied by the user, it’s possible to provide a string value that corresponds to the unsigned integer value of 0x80000000 or greater. If this is done, t will have a negative value due to the type conversion in line 1090. Example 7-1 illustrates how t can become negative:

01    typedef char *  caddr_t;
02
03    // output the bit pattern
04    void
05    bitpattern (int a)
06    {
07           int             m       = 0;
08           int             b       = 0;
09           int             cnt     = 0;
10           int             nbits   = 0;
11           unsigned int    mask    = 0;
12
13           nbits = 8 * sizeof (int);
14           m = 0x1 << (nbits - 1);
15
16           mask = m;
17           for (cnt = 1; cnt <= nbits; cnt++) {
18                   b = (a & mask) ? 1 : 0;
19                   printf ("%x", b);
20                   if (cnt % 4 == 0)
21                           printf (" ");
22                   mask >>= 1;
23           }
24           printf ("
");
25    }
26
27    int
28    main ()
29    {
30           caddr_t data    = "xffxffxffxff";
31           int     t       = 0;
32
33           t = *(int *)data;
34
35           printf ("Bit pattern of t: ");
36           bitpattern (t);
37
38           printf ("t = %d (0x%08x)
", t, t);
39
40           return 0;
41    }

Lines 30, 31, and 33 are nearly identical to lines in the OS X kernel source code. In this example, I chose the hardcoded value 0xffffffff as IOCTL input data (see line 30). After the type conversion in line 33, the bit patterns, as well as the decimal value of t, are printed to the console. The example program results in the following output when it’s executed:

osx$ gcc -o conversion_bug_example conversion_bug_example.c

osx$ ./conversion_bug_example
Bit pattern of t: 1111 1111 1111 1111 1111 1111 1111 1111
t = −1 (0xffffffff)

The output shows that t gets the value −1 if a character string consisting of 4 0xff byte values is converted into a signed int. See Section A.3 for more information on type conversions and the associated security problems.

If t is negative, the check in line 1093 of the kernel code will return FALSE because the signed int variable nlinesw has a value greater than zero. If that happens, the user-supplied value of t gets further processing. In line 1098, the value of t is used as an index into an array of function pointers. Since I could control the index into that array, I could specify an arbitrary memory location that would be executed by the kernel. This leads to full control of the kernel execution flow. Thank you, Apple, for the terrific bug.

Here is the anatomy of the bug, as diagrammed in Figure 7-1:

Figure 7-1. Description of the vulnerability that I discovered in the XNU kernel of OS X

Because the value of t is supplied by the user (see (2)), it is possible to control the address of the value that gets copied into EIP.