Debugging Mechanisms

There are many debugging techniques; which one to use typically depends on the nature of the problem at hand. Here, we will look at some of the techniques that can help aid in debugging kernel problems. Table 16-1 provides a brief overview of some of the mechanisms we will discuss in this chapter.

Most of the preceding debugging technologies are included by default, but are not always enabled in the kernel by default because they may cause interference with the running kernel or hardware devices, or may pose a security risk, as it may be possible to obtain sensitive contents of the target’s memory.

Enabling debugging mechanisms or controlling the kernel’s debug behavior can be done by setting kernel boot arguments. Boot arguments can be set in two ways, either by using the nvram command or by adding it to the boot arguments key in /Library/Preferences/SystemConfiguration/com.apple.Boot.plist. There are a heap of available boot arguments, but we are most interested in the debug argument, which controls debugger and system debug behavior. The argument is an integer value and can consist of the flags shown in Table 16-2.

The values form a bitmask and you can combine multiple values together by ORing them. For example, to enable NMI, disable the graphical panic screen and enter the debugger upon panic. We can combine the values so that:

The value can be set in the com.apple.Boot.plist or using the nvram command:

To disable or remove debugging options, simply do the following:

If you have existing boot arguments set, this command will overwrite them, so be sure to query the boot-args argument first to prevent overwriting them.

Recovering from Crashes During Boot

Your extension may be installed in the /System/Library/Extensions directory and get loaded automatically during system boot. If there is a problem that causes the extension to crash repeatedly during system boot, the system can be recovered in the following different ways.

• Boot in safe mode by holding the shift key down after you hear the startup tone and release the key when the Apple logo appears. This should ensure only essential kernel extensions (KEXTs) are loaded and you will be able to remove your KEXT so the system can boot manually.
• Attach the system to another computer and boot it in target disk mode using a FireWire or Thunderbolt cable by holding down the T key during boot. You should then be able to remove the offending extension form the system’s disk.
• Boot into a different partition if one is available.
• If the offending KEXT is a driver for a piece of hardware, removing it from the system will likely prevent the driver from loading.
• Perform an NVRAM reset if you need to reset boot arguments.

If you are unsure what causes the crash, you can boot the system in verbose mode by pressing Command-V.

Tracing with IOLog()

We have already seen tracing in action throughout this book. Tracing involves strategically placing IOLog(...) statements in your code to print variables and to test if conditional blocks are triggered. IOLog() output eventually ends up in the kernel.log file. The kernel has quite a limited buffer for storing messages from IOLog(), so if you write a large amount of long messages too quickly, the buffer will wrap and you may overwrite data before the syslog daemon has the chance to store it to the log. This may cause confusion and lead to incorrect assumptions when expected output is not seen.

Many functions of your driver may end up being called hundreds or even thousands of times per second, so printing a message on each invocation may be counterproductive. Although the logging daemon has features to coalesce identical messages, this falls apart if there are slight variations between them. It also does nothing to prevent the buffer wrapping around and potentially overwriting some messages.

You can change the size of the system log buffer that IOLog() writes into by re-compiling the kernel or using the much simpler way of adding the kernel flag msgbuf=n, where n is the size of the buffer in bytes, to the kernel’s boot configuration property list com.apple.Boot.plist. This file is found in the /Library/Preferences/SystemConfiguration/ directory.

Even though a larger buffer may decrease the likelihood of data going missing, it doesn’t prevent the log from being flooded with messages when printing from functions in the “hot path” of the driver. To avoid this, you can use variations of the technique shown in Listing 16-1.

This will log the amount of times a condition occurs, but prints a message to the log only every thousandth time. If you wish to debug a primary interrupt filter, you can use a variant of this approach. However, the IOLog() call needs to be moved out of the primary interrupt filter routine to a place it can be safely called, for example, the secondary interrupt handler or a custom IOUserClient method.

If your driver has multiple instances, you may wish to print additional information so you can tell which instance is doing what. If you are in the context of an IOService, you can print the “this” pointer address to help uniquely identify each instance.

Should you always leave IOLog() statements in your driver code? Yes and no. Opinions differ on this, but it is definitely frowned upon for a driver to spam the system logs with unnecessary output that may hide other potentially important messages from other parts of the system. However, it may be acceptable to print a few messages about exceptional conditions. If you prefer to leave IOLog() statements in your code, you can prevent them from being outputted, using a conditional variable, which can be toggled on or off by a user space client. The problem with this is a slight increase in the executable size, as well as extra work to be done by the CPU in executing the conditional debug statements. The other approach is to use pre-processor directives, such as #ifdef DEBUG … #endif, so the statements will be compiled away from the resulting executable. Of course, there is also an option to leave most debug statements out entirely, which may make the code more readable. If a user reports a problem, the downside to the two last approaches is that there is no way to enable debugging once the driver is in the field, short of asking the user to install a debug version of the driver. A combination of all three is certainly also possible, but the general advice here is not to litter your code with debug logging, but to place them at strategic places where they are likely to be of use to you even for problems you didn’t anticipate.

While tracing using IOLog() seems like a primitive approach, it is often very effective in finding bugs. Of course, this approach works best when the system doesn’t actually crash so you can observe the behavior through the system log. However, if the system crashes, the syslog daemon will not be able to write the latest contents of the log buffer to the log file. Consequently, the output may be lost at the next reboot. There are ways to get around this, including remote tracing over FireWire, which is discussed later in this chapter.

Printing Stack Traces

The IOLog() function is good enough for many purposes; but in some cases, it is not enough to know a function is called. You also need to know the call stack that led to a call to your function as it may be called from multiple code paths. Printing the call stack can be achieved using the OSReportWithBackTrace() function, as demonstrated in Listing 16-2.

The code in Listing 16-2 should give the following results:

---

---

You may notice the printed addresses of the test functions are similar, but not the same as the ones printed in the back trace. The addresses printed by IOLog() are relative to the start of each function; however, in the back trace, you instead see the address of where each test function is calling the next test function. The OSReportWithBacktrace() function also prints the start and end addresses of any KEXTs involved in the back trace. We see our KEXT is loaded at the memory address 0x9ab000 and you may notice that testFunc1(), testFunc2(), and testFunc3() are all within the address range of the MyDebugDriver. Using this information, we can also work out the offset where a function is located within its executable image by subtracting the address of a function from the base address, for example, for testFunc1(): 0x9ac280 - 0x9ab000 = 4736 bytes. You can then use the GDB debugger with the disassemble <offset> command.

Remote Tracing over FireWire

It is possible to redirect output from the krprintf() function over a FireWire connection to another system. This method is more robust than using IOLog(), as log output will be preserved on the remote system in the event of a crash. Another advantage of this approach is that it is available in the very early boot process, which is useful for debugging a driver that’s involved with the system boot process, such as storage and display drivers. It also allows debugging of shutdown and sleep events. Your driver does not need any special support or modifications to support outputting log information over FireWire.

Everything needed to configure remote logging over FireWire is already included in Mac OS X from versions 10.5 and above and there is no need to install additional KEXTs. You need two Macs to set this up. It is not necessary for both systems to run the same version of Mac OS X.

On the target machine, the machine you wish to send log output from, do the following.

This boot option enables redirection of the kprintf() function so output will be mirrored to the FireWire interface as well as the system log. The system should be rebooted for this option to take effect. The next step is to connect a FireWire cable between the two systems. Unlike IOLog(), the kprintf() function is synchronous, which means by the time it returns, it will have transmitted the message over FireWire. The kprintf() disables interrupts until it completes, which can affect timing-related issues when used excessively. Because interrupts are disabled, the function may cause a crash if memory referenced by the functions arguments happens to be paged out.

On the machine that will receive the debug output, run the fwkpfv command, which is the FireWire log viewer utility. If the target machine was connected correctly and the cable properly attached, you will receive debug output after a few seconds. The following example shows an extract from a session captured while the target machine boots:

---

---

If you wish to log over FireWire from your own KEXT, you will have to use the kprintf() function to log with rather than IOLog(), which doesn’t use kprintf internally but rather calls printf(), which only goes to the kernel log. A strategy to deal with this is to create your own wrapper function for IOLog() and kprintf() that calls the former for a release build and the latter for a debug build.

Listing 16-3 shows an example of how to log using kprintf().

Running this yields the following on the remote system:

---

---

If the kernel crashes, you can also get the panic log through the FireWire log viewer. It is also possible to use FireWire to attach the GNU debugger remotely to the kernel, as we will see later in this chapter.

Remote Kernel Core Dumps

Mac OS X provides a mechanism for transmitting core dumps from a crashed (or hung) system to a remote machine over the network. A core dump is a binary image of the wired contents of the system’s memory. By capturing a core dump, we can retain evidence of the exact state the system was in at the time of the crash and we can use this image with the GDB to get stack traces for all threads in the system and examine memory contents as well as kernel data structures.

Conveniently, everything you need to enable core dumps is already present in Mac OS X. Only minor configuration is required. On the dump server, the machine that receives the core dumps from crashed machines, you need to activate the kdumpd daemon, as follows.:

Change the Disabled key from true to false. If you wish, you can also configure the directory where dumps will be located. The default is /PanicDumps. The kdumpd daemon is started as follows:

The server uses UDP on port 1069, so you should ensure there is no firewall between the target machines and the server. The target and server do not need to be running the same version of Mac OS X.

The kdumpd daemon is able to receive core dumps from multiple machines and will archive each dump with the machine’s IP address. If you work for a company that develops software that runs in the kernel, you can configure all your Macs to automatically send dumps to a central server when a crash occurs. This saves a lot of time when quality assurance testers encounter problems during testing, as engineering can simply start debugging the dumped image immediately.

Configuring the target machine is equally simple and is done by setting kernel boot arguments either in /Library/Preferences/SystemConfiguration/com.apple.Boot.plist or using the nvram command, as follows:

The preceding instructs the kernel to dump core when it panics or if an NMI (non-maskable interrupt) event was triggered. The latter is highly useful in the cases where the system hangs completely and appears unresponsive but does not actually panic. In this case, you can use the power button on the computer to trigger the NMI event, which will start the core dump. During this time, the machine will be frozen and no processes will run. If the machine is responsive and you do this, the machine will simply resume as if nothing had happened after the core dump is transferred.

The _panicd_ip parameter specifies the IP address of the machine running kdumpd. If you plan on having a permanently running panic server, it is recommended this IP be static. It is not possible to use a hostname or DNS name for the server, as name resolution is not possible.

The following output will appear on the screen of the target machine if you press the power button:

---

---

The target machine will write dots (.) to the screen until the dump is complete. If it was an NMI event that triggered the dump, the system will resume; if it was a panic, the system will wait for a remote debugger to be attached.

KDB

The kernel supports an in-kernel debugger called KDB. KDB only supports debugging over the serial port. It is not included in the kernel build by default; therefore, you need to compile and install a custom kernel in order to use it. KDB has some applications for very low-level debugging where neither FireWire nor Ethernet is available. KDB requires a native serial port on the machine being debugged, which is only found on the now discontinued Xserve (although a serial port is available if Mac OS X is used under a virtual machine). For all intents and purposes, the GNU Debugger (GDB) is recommended; we only mention KDB here to avoid confusion with GDB.

Remote Debugging with GDB over Ethernet or FireWire

The kernel has support for using the GDB over Ethernet (IP/UDP) or FireWire connections. Again, this requires two computers running Mac OS X. While GDB is supported through Xcode, you cannot debug the kernel using Xcode; you will have to use the command-line interface. GDB is, however, part of Xcode. It is not necessary to install Xcode on the machine being debugged; it is only needed on the remote system (the client).

The debugging support is built in to the kernel by default, unlike KDB. However, the debugging capabilities are disabled by default, but can easily be enabled by adding the appropriate boot arguments on the target machine, for example:

The “-v” (verbose) flag isn’t strictly necessary. It has the effect of disabling the grey screen with the Apple logo during boot and instead showing a text console, commonly found on UNIX and Linux systems, which shows log messages as the system boots. Once the boot arguments have been set, the system needs to be rebooted for the changes to take effect.

Configuring the Host Machine

Setting up the target is straightforward. However, the host machine requires a little more preparation. Before you can start debugging, you need to download the correct Kernel Debug Kit for the kernel version used by the target system. Apple doesn’t appear to publish a version for every build in a timely manner, so the target system would have to be downgraded or upgraded to match a published version of the Kernel Debug Kit. If you use the wrong version, GDB may fail to resolve the correct symbols and data structures and the results may be wrong and cause great confusion, for example, functions getting called that should not have.

The Kernel Debug Kit contains the following:

• Debug version of the kernel (mach_kernel)
• Debug version of I/O Kit families and selected KEXTs
• Symbol files
• Various scripts
• Macros for GDB

You can replace the default kernel (mach_kernel) of a Mac OS X system with the debug version found in the Kernel Debug Kit. This will help you get more accurate results and stack traces, as optimization has been disabled.

If you have the sources for the XNU kernel installed on the host system, it is possible to link this with the debugger, which allows you to see source code instead of assembly code in the debugger, though this is often not needed if you are only debugging your own extension (in which case you can link the source for your own extension only).

Attaching to the Remote Target

If you have set the appropriate boot arguments with the nvram command on the target system and have the Kernel Debug Kit ready on the host machine, you can now trigger an NMI event on the target system by pressing the system’s power button. This should cause the following text to appear on the top left corner of the screen:

---

---

Starting GDB and attaching to the remote target can be done using the following steps:

The arch argument can specified if the debug host is running on a different architecture from the target system. For example, in this case, we are debugging a target running a 32-bit kernel on a system with a 64-bit kernel. You can also do the reverse by specifying x86_64.

The preceding line will load specialized GDB macros, which will help you examine the state of the target system’s kernel. You can, for example, dump a list of running tasks or threads. Type help kgm to get a full list of available macros.

To attach to the target, use the following:

The preceding will attach to the remote target so we can begin our debugging session. We can then start issuing commands, for example, bt to get a stack trace, which will look something like the following:

---

---

The command will show the kernel stack, which is the sequence of function calls the CPU was executing at the time of the NMI event. We can see in this case that the last thing the system did before we halted it was to respond to the NMI interrupt. If you are curious what the other CPUs (cores) were doing at the time, you can issue the command showcurrentstacks, which will print a stack trace for each CPU (core) in the system.

You can now set breakpoints or examine the state of the kernel. Issuing the continue command will resume the kernel. We will look at how GDB can be used in more detail later in this chapter.

Debugging Using FireWire

In addition to Ethernet, the Kernel Debugging Protocol can also be used over FireWire, using the FireWireKDP mechanism. The fwkdp tool can be used to help set the appropriate debug parameters, but you can also set them manually. FireWireKDP is also compatible with logging over FireWire and can be used to transmit core dumps to a remote system.

To configure FireWireKDP on the target system, you can do as follows:

On the host system, where you will run the debugger, you will need to run fwkdp as well. Be sure to start it in proxy mode. Once that is done, you can use GDB to debug in the same way as with Ethernet. The process of attaching to the remote target is nearly identical; the only major difference is that you attach to localhost, not the target’s IP address. Once the target system is rebooted, you can enter the debugger by pressing the power button to generate an NMI event as before, which should give the following results on the target’s screen:

---

---

If all goes well, the fwkdp proxy running on the host will download the core file to its working directory. The core dump in the preceding example was named core-xnu-1504.15.3-0.0.0.0-ff28cfb5. After the core is downloaded, the remote system will wait for a debugger to be attached.

Live Debugging of a Running Kernel

A less known but powerful feature available in Mac OS X (since version 10.5) is the ability to attach the debugger to a running system. Live debugging requires you to enable support for the /dev/kmem character device file, which allows a user space process to read and write to the kernel’s memory address space. You can enable support for /dev/kmem with the following command:

You can test if it worked by checking that the /dev/kmem file is present after a reboot. The process of attaching to the live kernel is similar to that of attaching to a remote target. The steps are as follows:

At this point, you can examine the state of the kernel, for example, using the showcurrentthreads command.

Live debugging is useful in a number of cases, for example, if an application using your driver hangs in the kernel while executing a user client method. You can attach to the kernel and find out where the issue is. You can also examine the memory of your driver and its data structures. Live debugging can only be used when the system is operational and cannot be used to debug a crashed or deadlocked system.

Debugging Using a Virtual Machine

If you do not have a second machine available, it is possible to perform kernel debugging using a virtual machine. Software such as VMWare Fusion or Parallels desktop allows another copy of Mac OS X to run virtualized. Enabling the kernel debug features on the virtual machine can be done by putting the desired boot arguments in the/Library/Preferences/SystemConfiguration/com.apple.Boot.plist. Debugging hardware drivers may not be possible under a virtual machine, as you cannot use PCI or Thunderbolt devices directly under a virtual machine. However, it is possible to assign USB devices to a virtual machine instance. Mac OS X Lion is able to run as a virtualized instance, however, prior to that, only the server version of Mac OS X could be virtualized.

Debugging in the Kernel Using GDB

There are several ways GDB can be used to debug the kernel or KEXTs loaded into it, such as:

• Remotely over Ethernet
• Remotely over FireWire
• On a captured core dump file
• On a Live live system
• Using a virtual machine

Additionally, a kernel can drop into the debugger in the following ways:

• Because of a kernel panic
• Manually triggered NMI event
• Because the DB_HALT option was set to halt the system at boot and enter the debugger
• Programmatically in code by calling the PE_enter_debugger() function

Programmatically entering the debugger is only possible with a remote debugging setup; live debugging cannot occur during boot, when the system is crashed, or halted with an NMI event, nor programmatically, as this stops the system until a remote debugger is attached and thereby the gdb instance running the debugging session as well.

Kernel GDB Macros

The Kernel Debug Kit comes with a wealth of helpful GDB macro functions, which greatly simplify the task of examining the kernel. The macros can help interpret common kernel data structures, such as task and thread descriptors, and memory-related data structures, such as VM maps. Furthermore, it provides functions for accessing PCI configuration space, I/O space, and un-translated access to physical memory. A small subset of available macros is shown in Table 16-3.

Creating Symbol Information for KEXTs

Before you can debug your own KEXTs in GDB, you need to generate symbol information for the KEXT. Because a KEXT is dynamically loadable, we have no way of knowing where in memory it will be located. Although our KEXT includes symbol information, the addresses of functions and data are relative to the KEXT binary. The absolute address of a function in a KEXT will be the kernel_load_address + relative_address once the KEXT is loaded.

Fortunately, the Kernel Debug Kit gives us a helping hand by providing a small script that helps generate the final symbol table containing the absolute addresses within the kernel address space. The script is called createsymbolfiles and can be used as follows:

The load address can be found in several places, for example, with kextstat, as follows:

In the preceding case, the KEXT had no additional dependencies; however, in a real world situation, a KEXT may have dependencies on one or more other KEXTs, such as an I/O Kit family, in which case you will be prompted to enter their addresses as well. If you are debugging a crash dump from a remote system or directly debugging a crashed system, the load address of your extension and any dependencies will be found in the panic log if your extension was involved in the crash. Note that your KEXT may be given a different address each time it loads, so you will need to regenerate the symbol information each time.

The above is enough to get us basic symbol information, but it is restricted to symbolic names of functions only and is not able to give us source code line information. You can get this by configuring your extension’s debug information format, as shown in Figure 16-2.

Figure 16-2. Setting the debug information format with Xcode

DWARF with dSYM creates a separate file for you containing a full set of symbols and information needed to map a code location back to a source code line. This information is normally embedded into an executable when doing a debug build; however, you can generate that information in an external file for release builds of the driver. It is good practice to archive this debug information for later use. This will make debugging easier if crashes should be reported by a user. The dSYM file (actually bundle) will be named MyDebugDriver.kext.dSYM in our case. The dSYM bundle should be placed together with the KEXT before the createsymbolfiles script is run.

Debugging KEXTs with GDB

In the following sections, we will analyze a crash caused by a hypothetical driver named MyDebugDriver. The header file specification for MyDebugDriver is shown in Listing 16-4.

The driver starts a timer, from which testFunc1() is called, which in turn calls testFunc2(), which calls testFunc3(), which causes a kernel panic due to a null pointer dereference. Each function accepts four integers that have no significance (picked randomly) and are passed unchanged from the first function to the last. The values passed as arguments are 65261, 48879, 0, and 5380.

We have successfully generated a core dump from a crash system using the FireWire core dump mechanism and previously, we built symbol information for our driver using the correct load address. We are now ready to load it up in the GDB and get our hands dirty. The steps we need to perform to find the bug in our driver are as follows:

• Create a symbol file with the correct load address (see the last section)
• Start GDB and load the core dump
• Add symbol information from the kernel
• Load Kernel Debug Kit GDB macros
• Load our KEXT binary of MyDebugDriver into GDB
• Load symbol information for MyDebugDriver (if GDB cannot find them in the same location as the KEXT)
• Tell GDB the location of the source code for MyDebugDriver (optional)

Following is a complete debug session for MyDebugDriver:

The preceding sequence of commands loads the core dump file, adds symbol information for the kernel from the Kernel Debug Kit, and finally loads the GDB macros, which must be loaded after the kernel symbol information to initialize properly.

We can now issue the backtrace command to see where the system crashed, as follows:

Because we have not yet loaded our KEXT, symbols #5–#8 are showing up as bogus, as the debugger is unable to resolve the addresses of the functions to their symbolic names. To fix this, we will load the MyDebugDriver KEXT into GDB along with its symbol information, as follows:

We have now loaded the driver along with its symbol information and informed GDB of the location where it should look for the source code of MyDebugDriver. The location of the kernel’s own source code is hard wired into the debug kernel image. If you wish to show the source code in GDB, you need to create a symlink for the location where you downloaded the XNU source to the directory: /SourceCache/.

Let’s try the backtrace command again with the KEXT and symbols loaded, as follows:

That’s much more readable. We have now identified the exact call stack and we can see which methods in our in our driver that was involved, down to the file and line number. We can also see the arguments that were passed to the methods and that they correspond to the values we picked earlier. Let’s examine the crash further by jumping to the fifth stack frame, the location where the crash occurred, as follows:

I think we found the problem! We are trying to assign a value to the member variable fVariable1 but the object is not initialized. We can also list the source code of testFunc3(), as follows:

Well, that would never work! We have found our bug, which appears to be triggered only when the third argument passed is set to zero.

If you only have symbol information and lack the debug information required to map addresses to a specific source code location, you can use the disassemble command in GDB to show a dump disassembly of the method from its address. Let’s look at the disassembly of testFunc3(), as follows:

While this looks very uninviting if you are not familiar with assembly, you will be able to infer a number of things by comparing the disassembly to the original source code. For example, the cmp instruction compares the value of the eax register against the constant value $0x0, which we can correctly guess corresponds to the if statement on line 11.

Although we have already found the source of the problem, let’s pretend for a moment we are curious as to why a zero value was passed for the third argument. Perhaps our driver used an internal state to calculate the value passed to testFunc3(). In this case, we could continue our examination by looking at the state of the driver was in at the time of the crash. Because testFunc3() is a member method of the com_osxkernel_MyDebugDriver, we know that a pointer to the class instance is always passed automatically to the member function as the this pointer. We can dereference the this pointer address from the previous stack trace as follows:

We can now examine the internal state of our driver instance and we can see the values of its member variables fVariable1 and fVariable2. We can also see how many instances of our class exist from the meta class information and determine the retain count of the driver.

Understanding Kernel Panic Logs

A panic log can be found in the /Library/Logs/DiagnosticReports/ directory after a crash or can be obtained by extracting it from a core dump or a remote GDB session to a crashed target. As a kernel programmer, you might be expected to analyze kernel panic logs sent from customers’ computers, which you will rarely have, physical access to. Furthermore, the customer may be reluctant or unable to assist you in getting a core dump. It is therefore vital to be able to understand and extract as much information as possible from the logs. Let’s start to look at the panic log and what information we can extract from it. A panic log for the MyDebugDriver crash discussed in the previous sections is shown in Listing 16-5.

The panic log in Listing 16-5 was generated on a different system than before. This system is running a newer version of Mac OS X Lion, which only runs the 64-bit version of the kernel. The panic log consists of the following elements:

• The type of panic/problem that occurred and the CPU (core) number it occurred on
• A dump of the CPU state (register values)
• Back trace of what the CPU was doing at the time of the crash
• Kernel extensions involved in the crash and their dependencies (none above)
• The name of the process (task) that caused the crash
• Kernel build and version numbers
• System model
• Information about recently loaded/unloaded KEXTs
• A complete list of KEXTs loaded

The first thing you may notice is that the panic was caused by a page fault, which gives us a clue about what to look for later in our code. It is often useful to look at the task that caused the problem as well. In this case, our driver was executing in kernel context (kernel_task) when the crash occurred and not on the behalf of a user space thread.

First let’s look at the back race and try to prove that our driver was indeed involved. There is a very good chance that it was, as our driver is listed as being part of the back trace. You will also notice two addresses after our driver: 0xffffff7f81345000->0xffffff7f81348fff. This is the load address where the instructions and data for our KEXT were loaded into the kernel’s address space. To determine which functions on the stack belong to our driver, we can simply look for addresses on the stack that are within that range. Four addresses can be identified—0xffffff7f81345570, 0xffffff7f813458b6, 0xffffff7f81345914, and 0xffffff7f813459f4.

We already know from earlier that they will most likely correspond to testFunc3(), testFunc2(), testFunc1(), and timerFired().

Assuming we had no clue what functions the addresses corresponded to within our driver, we can employ a simple trick. By simply subtracting the address of one of the functions from the load address, we can determine the offset of the function in the executable image of the driver, as follows:

---

---

We now know that the function is 1392 bytes from the start of the KEXT and assuming we have the executable image (the exact version and build that was involved in the crash) of the driver available, we can do the following:

And we have found the location of the crash! A full description of CPU registers is outside the scope of this book, but suffice it to say they contain a wealth of useful information. We will discuss how we can use register information to retrieve function arguments in the next section. The processor in the panic log was running in 64-bit mode. The x86_64 has a larger amount of registers available than i386 systems, and local variables are usually passed in general purpose registers instead of the stack.

x86-64 Calling Conventions

A calling convention is a scheme for how functions are passed their arguments. The calling convention depends on the programming language, operating system, architecture, and compiler. Understanding the calling convention used can help us decode the register state when a crash occurs. For example, on Mac OS X running a 64-bit executable or kernel, the System V AMD64 ABI convention is used (note that Windows uses a different calling convention, so the register usage will be different). On Mac OS X for a 64-bit task, the register assignments for function call arguments are shown in Table 16-4.

If a function takes more than six arguments, the remaining arguments will be passed on the stack. One thing to consider when examining C++ code is that a non-static C++ member method always passes the this pointer as the first argument, so the actual first argument to the method is passed in RDI whereas the this pointer will be put in the register RSI.

Let’s look at the registers in the panic log shown in Listing 16-5 and see if we can work out which arguments were passed to our function by examining the register state, as follows:

• RDI: 0xffffff801ab61100 (this pointer)
• RSI: 0x000000000000feed (decimal = 65261)
• RDX: 0x000000000000beef (decimal = 48879)
• RCX: 0x0000000000000000 (decimal = 0)
• R8: 0x0000000000001504 (decimal = 5380)

As you can see, the register contents from Listing 16-5 match exactly the four arguments passed to testFunc3(): 65261, 48879, 0, and 5380. We can also see that the first argument looks like a pointer and is likely to be the this pointer representing the current instance of MyDebugDriver.

Assuming testFunc3() was a longer and more complicated method and that the crash happened further down in the function, it is possible that the registers may have been reused and overwritten at the point of the crash. In that case, you may not be able to recover the original values of the arguments.

Diagnosing Hung Processes with Activity Monitor

The Mac OS X activity monitor shown in Figure 16-3 can be helpful in diagnosing kernel problems.

Figure 16-3. Sample process output from Acitivty Monitor

Any task shown in the Activity Monitor can be sampled (except the kernel_task), which will generate call graphs for all the threads of that task during the sample period. This is useful from a kernel debugging point of view in that you are able to see if a process has threads that are calling in to your driver through system calls like IOConnectCallMethod(). If a process is hung, force quitting it will start the Crash Reporter, which will give you a more detailed log, including the kernel stack of a thread if it is currently running in the kernel. The sample process function can also help you determine performance issues and find where an application is spending the most time.

Finding Memory and Resource Leaks

Preventing memory and resource leaks is particularly important for extensions that can be dynamically loaded and unloaded during runtime. A handy tool to detect leaks is the ioclasscount utility, which shows the instance count of each known class (loaded into the kernel). Typical output will look like the following:

---

---

It shows that our driver has been retained (retain()) five times. The driver’s free() function will not be called until the retain count reaches zero, even if the hardware device it controls is removed. This will prevent the kernel extension from being completely unloaded. The retain count typically increases for each user space application that opens the driver, or it can increase because another driver or ancillary support class used by the driver calls retain() on it. Failure to balance a call to retain() with a call release() will result in a leak. The ioclasscount utility can be used without changes to kernel boot parameters; it is not installed on a system by default, but is installed as part of the Xcode distribution. It can be copied onto a system without Xcode for debugging purposes. A driver that has been unloaded and has all references to it released (retain count = 0) will be unloaded by the kextd daemon. Although the reference count has dropped to zero, it may take up to a minute for the KEXT to be fully unloaded.

If you are able to live debug the kernel using GDB, you can use macros such as showallclasses or showregistry, as follows:

To further help debug reference counting bugs, it is possible to override OSObject::taggedRetain(const void *tag) and OSObject::taggedRelease(const void *). For example, print a message or print a back trace of the caller’s stack to help identify where the leak comes from.