PCI in I/O Kit

PCI in the I/O Kit is handled by the IOPCIFamily, which, just like the IOUSBFamily, is implemented in its own KEXT. The IOPCIFamily is simpler than the IOUSBFamily in terms of the number of provided classes. This means there are fewer building blocks to help us out when implementing drivers. PCI is more low-level than USB from a driver point-of-view, and, as such, the writing of drivers for PCI devices is often more complex. Figure 9-2 shows the class hierarchy of the IOPCIFamily.

Figure 9-2. IOPCIFamily class hierarchy

The IOPCIDevice object acts a nub or provider for all PCI-based devices, including PCIe, Thunderbolt, and ExpressCard. An IOPCIDevice subclass called IOAGPDevice handles older AGP (Advanced Graphics Port)-based graphics cards; however, no Intel-based Macs feature AGP. In many cases, you only need to interact with the IOPCIDevice class from the IOPCIFamily. An instance of this object is provided for each PCI device in the system; similarly, an IOPCIBridge instance exists for each PCI bridge in the system. There are cases where a driver may need to interact with its bridge to read or write the bridge's configuration space. Although uncommon, it is also possible to create your own PCI bridge driver. We will discuss device and bridge access later in this chapter. The root PCI bridge (also known as the host bridge and root complex) is implemented by a subclass of IOPCIBridge called AppleACPIPCI class. This class is part of the Platform Expert implemented by the AppleACPIPlatform KEXT and controls access to all devices and bridges in the system. There is only one instance of this class. The IOPCI2PCIBridge class is the driver for PCI-to-PCI bridges.

Matching and Loading Drivers

PCI drivers are commonly matched against properties found in their configuration space registers, such as the vendor ID, device ID, class, subsystem vendor ID, and subsystem device ID. Often, the latter two are needed if a PCI device is based on a generic chip.

Though the configuration spaces contain more fields, they cannot be matched against using a matching dictionary (property-based matching). If you need more advanced matching, your driver will have to override the IOService::probe(IOProvider* service) method, and you will have to examine the IOPCIDevice yourself to determine if your driver matches the device. The keys listed in Table 9-1 can be used for matching against PCI-based devices.

Listing 9-1 shows the matching dictionary for a typical PCI device with a vendor ID or subsystem vendor ID of 0xabcd and a device ID or subsystem device ID of 0x1234.

The value is specified as a 32-bit hexadecimal string in little-endian format. The first four characters will represent the device ID and the last four characters will represent the vendor ID. It is worth noting that the value is of the string type and not an integer. The key IOProviderClass must have the value IOPCIDevice in order for the I/O Kit to pass your driver an IOPCIDevice instance. If you need to match against the vendor ID and device ID, you can substitute IOPCIMatch with IOPCIPrimaryMatch or, if you only wish to match the subsystem IDs, you can use IOPCISecondaryMatch.

If your driver handles multiple devices, this can be done as a space-separated list as follows:

This will match device IDs 0x1234, 0x1235, and 0x1236 of vendor 0xabcd. If your driver supports a large family of devices, you can use masks to achieve the same effect, rather than enumerating each device separately.

This will match every device ID beginning with 0x123X; for example, the range from 0x1230 to 0x123F, and a vendor ID of 0xabcd. Bits that should be ignored by the mask must be set to zero.

You can also match against the class register. To do this, you must specify the IOPCIClassMatch key. The class register is 3 bytes wide. However, to match against it, the I/O Kit requires you to specify a 4 byte value. The last byte is ignored. The following example matches display controllers (base class code 0x03):

As all PCI devices are assigned names according to their devices and vendor IDs, it is also possible to use IONameMatch to match PCI devices as shown in Listing 9-2.

The previous approach is perhaps more readable, but the downside is that it is not possible to match against the subsystem vendor and device ID.

During system boot, drivers for PCI devices installed in a physical slot or embedded on the motherboard are loaded. Thunderbolt and ExpressCard drivers are loaded at boot-time or on demand as they are plugged in.

While Thunderbolt devices follow PCI devices' rules for identification, they need an additional change in order for the driver to load. In the driver's Info.plist file, under each personality specified, the following key needs to be set:

This tells the system that the driver is Thunderbolt-ready, and therefore, is safe to unload. It is possible for a Thunderbolt and a PCIe device to share the same device driver; however, PCI drivers may in many cases be written under the assumption that the driver/device will never be removed during operation. A driver will not be loaded against a Thunderbolt device unless this key is set.

THUNDERBOLT UNIQUE INDENTIFIER

Driver Example: A Simple PCI Driver

It's time to get our hands dirty. In order to demonstrate a PCI driver in action, we will take advantage of the IOMatchCategory key to allow the loading of a secondary driver for a device. We will load our driver against the display controller (graphics card/GPU) in this case, as it is a device guaranteed to be present on all Macs—even laptops—as they all use PCIe internally. We will use the following to match against the display controller:

Recall that 0x03 is a base class for display controllers. If you have more than one GPU, this will cause multiple instances of the driver to be instantiated—one per device.

Let's start with the class declaration for our driver, as shown in Listing 9-3.

There should be few surprises here if you've followed earlier examples. We simply declare a sub-class of IOService and override the start() and stop() methods. Note that we include the file IOKit/pci/IOPCIDevice.h that contains the definition of the IOPCIDevice class.

The implementation of MyFirstPCIDriver is shown in Listing 9-4.

When a driver is matched successfully, either from the Info.plist dictionary or by invocation of the driver's probe() method, your driver will have its start() method called. As with the USB driver in chapter 8, we check to ensure that the provider that is passed to us is in fact of the right type (IOPCIDevice), which is good practice although it shouldn't happen if your Info.plist correctly specifies the IOProviderClass key.

If we have a valid IOPCIDevice, the next step is to enable the I/O resources of the device by calling the IOPCIDevice::setMemoryEnable(bool enable) method. This will set a toggle bit in the device's command register, letting it know that we want to access its resources. Finally, our driver calls registerService(), which will notify potential clients (possibly a higher-level driver) of our driver's arrival. We return true to indicate to the I/O Kit that the driver was loaded successfully.

We can now attempt to load MyFirstPCIDriver using the kextload utility. You can verify that it gets loaded correctly by checking kernel.log in Console.app or by searching for it using IORegistryExplorer as shown in Figure 9-3.

Figure 9-3. IORegistryExplorer showing MyFirstPCIDriver loaded

Accessing Configuration Space Registers

The IOPCIDevice class contains a number of helper methods that make it easy to access a device's configuration space registers. The following methods allow you to read and write configuration space registers.

There are three variants of read methods and three variants of write methods, which allow you to read or write an 8-bit value, a 16-bit value, or a 32-bit value from the offset specified. The offset parameter is a byte-offset into the configuration space and must be between 0-255. To read a device's device ID and vendor ID, we can do the following:

The previous request could also be achieved by a single call:

The preceding call uses integer byte offsets, but the IOPCIDevice.h file specifies constants that can be used to address common register locations. So to make the code more readable, you can use kIOPCIConfigVendorID and kIOPCIConfigDeviceID instead of the hard coded values. The full list of available constants is shown in Listing 9-5.

IOPCIDevice also provides a convenient method for setting individual bits of a register called setConfigBits().

A read request to a missing or malfunctioning device will return a value of 0xFFFF (0xFF or 0xFFFFFFFF for the 8 and 32-bit variants), which is an invalid device/vendor ID. So if this value is returned while reading either register it can be used to determine if a problem has occurred or if a Thunderbolt device has been unplugged.

Writing values to the configuration space is simple but there are a few things to note. Many areas of the configuration space are read-only. For example, the device ID and vendor ID are programmed into the device's PCI controller firmware. Also note that it is not possible to determine if a write to a register location succeeded; you would have to read back the register or another that was affected by the write transaction in order to determine its success.

A number of methods of IOPCIDevice, such as setMemoryEnable(), are simply convenient abstractions that perform the appropriate configuration space reads or writes on your behalf. I/O to configuration space is forwarded by an IOPCIDevice to its parent (an IOPCIBridge in most cases) until it reaches the root bridge, which is implemented by the Platform Expert, as the exact details are system dependent.

Accessing the Extended Configuration Space

You may have noticed an inconsistency with the I/O functions in the previous section. We saw earlier that the extended configuration space is 4096 bytes. How do you address offsets greater than 255 when the config*() functions take a UInt8 type for the offset argument? The answer is the following family of methods.

The methods have the same interface as is shown above. However, they use the wider data-type IOByteCount for the offset parameter to allow access to offsets greater than 255.

Searching for Capabilities Registers

Because capability registers are not located at a fixed offset, the process of finding a capability register involves searching for a capability ID and then reading the next byte to determine the length of the capability, which also tells you the offset of the next capability. This process is followed moving down the list until the right capability is located. Fortunately, we do not need to write this code manually as the IOPCIDevice class provides two helper methods to locate capabilities:

The following demonstrates how to fetch the PCIe link status register, which contains the number of active lanes (bits 4-9) and the link speed (bits 0-3) for the device.

The method will return the capability ID (kIOPCIExpressCapability in this case) or zero if the capability with the specified ID could not be found. The output argument offset is used to store the offset of the found capability. Once the capability is found we can read the link status register by adding 0x12 (18) to the offset.

PCI I/O Memory Regions

PCI devices may have up to six I/O regions. Each region contains either I/O memory or I/O space (ports). The latter is seldom used in new devices as I/O ports are generally a very slow way of performing I/O and can only be accessed using special in/out CPU instructions. Some legacy devices, such as IDE controllers, may have both I/O ports and memory and can be controlled by either. On the other hand, I/O memory is more efficient and also easier to access, as it can simply be mapped into the system's memory space and accessed like regular memory. I/O memory is commonly referred to as Memory Mapped I/O (MMIO). This concept is not to be confused with mapping of memory between virtual address spaces or the mapping of files in memory (mmap).

Access to and from mapped device memory can be cached by the CPU if the region has the memory prefetchable bit set.

How is a device controlled through a memory region? That is entirely up to the device. For example, one region could be used for control and status registers, while a second region could be used to read or write data, for example input video from a camera. If you are reading this in electronic form, then this very text may be continuously written to the memory region representing the frame buffer of your graphics card. Just like USB, there are a number of standardized interfaces for PCI-based devices as well. An example of this is VGA compatible graphics cards, which allow for the basic operation of a graphics card using a known interface including memory regions and/or ports. Standardized interfaces for IDE, SATA, and PCI-based USB controllers also exist, enabling an operating system to use its default driver for any device that complies with such an interface.

Because PCI is “plug and play,” I/O resource for a PCI device is configured automatically by the kernel/EFI (or BIOS in traditional PCs), in contrast to the obsolete ISA bus, where jumpers had to be physically placed to select the base I/O addresses and interrupt line for each device separately in an attempt to avoid resource conflicts.

When a device is configured, each region present in the configuration space will be configured with its own address range. The size of the range depends on the device.

When a device is configured, it will be assigned a physical memory address range by the system. As you can see from Figure 9-1 there is no register for storing the size of each memory region. So how does the system know how big each region is? The size of a memory region is determined by the system by setting all bits in one of the base address slots in the configuration space and then reading back the value. A region must be of a size that is a power of two. Devices, if they support it, can combine two BARs to form a 64-bit address.

Before the system or a driver can access any of the I/O regions, they need to be enabled by toggling a bit in the device's command register. We already saw how this was done in MyFirstPCIDriver by calling fPCIDevice->setMemoryEnable(true) in the driver's start() method.

Enumerating I/O Regions

To discover available memory regions of a PCI device (there may be up to six), let's modify MyFirstPCIDriver to dump some additional information about the device in its start() method, by adding the code in Listing 9-6 after the call to setMemoryEnable().

If you compile and load the driver, you should see something like the following printed in the kernel.log:

---

---

Your output may differ depending on your system model and graphics card (you may even have multiple). In this case, the largest region (256 MB) is region 1, which is the graphics card's frame buffer.

Mapping and Accessing Device Memory Regions

The previous section showed us how we can obtain information about available I/O memory regions. We need to do some more work before we can actually access data from the regions. Furthermore, in most real-world drivers, it is unnecessary to explicitly enumerate the regions, as a driver usually knows exactly which regions, if not all that it needs to map. The following IOPCIDevice method can be used to map a BAR region directly:

The following is an example of its use.

If you have already obtained an IODeviceMemory (subclass of IOMemoryDescriptor) object by calling getDeviceMemoryWithIndex() as in Listing 9-6, you can simply call the map() method which does the same thing. In fact, that is exactly what mapDeviceMemoryWithRegister() does under the hood. Once an IOMemoryMap object is obtained, you can call the getVirtualAddress() method to obtain a kernel virtual address which can be used to access the mapping. The returned pointer can be read and written to in the same way as regular memory assuming it points to I/O memory and not I/O space.

When a driver is done accessing the memory it should call the unmap() method.

Accessing I/O Space

I/O Space consists of a 16-bit address space and is an older way of communicating with devices. I/O ports were also used for communication with serial and parallel ports in older computers, so it is not specific to PCI, but rather a way for an external device (to the CPU) to interface with the processor. I/O Space ranges assigned to a device can be accessed and mapped just like memory regions by using mapDeviceMemoryWithRegister() and getDeviceMemoryWithIndex(). The difference however, is that you cannot simply access the pointer returned by getVirtualAddress() as above directly. You have to use one of the following methods.

Using I/O space in new devices is frowned upon, due to poor performance and the limited address space it provides. Accessing mapped memory can take as little as 1 CPU cycle, while accessing a port can take as many as 100 cycles on certain architectures.

Before I/O space can be accessed, it needs to be enabled in the device's command register. IOPCIDevice provides the setIOEnable() method for this purpose.

Handling Device Removal

Thunderbolt and ExpressCard devices may be unplugged during operation. Therefore, drivers that handle these devices need some additional modifications over traditional PCI drivers, which are usually not written with removal of the device in mind. Improper handling of device removal may lead to hanging applications, system crashes, or disruptions to system stability or performance. For Thunderbolt devices, removal is not an exceptional condition so a driver must be able to cope with the removal of the device.

While it may seem complicated to handle device removals, the I/O Kit was designed specifically to allow removal of devices. The IOService class handles a lot of the heavy lifting for us automatically.

Your driver may detect the first sign that a device has been removed if it receives the value 0xffffffff (assuming a 32-bit read) while reading a value from memory mapped I/O (MMIO) or PCI configuration space registers. Of course the value might actually be valid for some registers, however you can read an alternate register or memory location that you know is guaranteed never to contain that value to confirm if the device is unresponsive. The driver may detect this condition before the I/O Kit messages the driver informing it that the device has been removed. If a driver determines that a device is removed, it should cease all access to mapped memory and the configuration space as further requests will result in timing out requests, which can take up to several milliseconds and may affect overall system performance. Apple recommends funneling all accesses to MMIO through a single method, as follows:

The method will prevent further accesses to registers once the device has been removed. We can now use the member variable fDeviceRemoved in other parts of the driver to prevent operations that will communicate with the hardware.

The I/O Kit handles device removal in three phases:

1. The bus controller (PCI root) will call the terminate() method on its client nub, which will message its clients again and so forth until it reaches the bottom of the stack. An IOService object that overrides the message() method will also receive a kIOServicesIsTerminated message. The driver is now considered inactive and cannot be enumerated or attached to by new clients. Existing clients holding the driver open will still remain active.
2. Drivers in the stack will have their willTerminate() method called, and thereafter didTerminate(). This process happens in reverse order, so clients will call their providers instead of the other way around, until it reaches the original provider that initiated the call to terminate() in the first place. Remember that these methods are optional, and you can choose to implement these based on your driver's needs. In response to having its willTerminate() method called, a driver should clear all queued requests and cancel in-flight I/O such as unfinished DMA transfers.
3. The last phase of the removal will call the drivers stop() method, then detach() which will remove it from the I/O Registry. If the driver's retain count reaches zero, the driver will be deallocated and its free() method will be called.

If a user plugs the same device back again, a new instance of the driver will be allocated. Any applications accessing the driver at the time will still be attached to the old instance of the driver. To handle this situation, the application must install a notification to detect when the driver/device is removed or added to the system. Because a driver instance is not reused when a device is reinserted, it doesn't need to return to its default state once it has handled a device removal. However, it is important it properly release and free any used resources, as the new instance will reallocate or reclaim those which could result in a memory leak or the new driver instance not coming up properly.