Each peripheral is identified by a bus number, a device number, and a function number. While the PCI specification permits a system to host up to 256 buses, PCs have only one. Each bus hosts up to 32 devices, and each device can be a multifunction board (such as an audio device with an accompanying CD-ROM drive) with a maximum of 8 functions. Each function can be identified by one 16-bit key or two 8-bit keys. The Linux kernel uses the latter approach.

The hardware circuitry of each peripheral board answers queries pertaining to three address spaces: memory locations, I/O ports, and configuration registers. The first two address spaces are shared by all the devices on a PCI bus (i.e., when you access a memory location, all the devices see the bus cycle at the same time). The configuration space, on the other hand, exploits ``geographical addressing;'' each slot has a private enable wire for configuration transactions, and the PCI controller accesses one board at a time with no address collision. As far as the driver is concerned, memory and I/O are accessed in the usual ways via inb, memcpy, etc. Configuration transactions, on the other hand, are performed by calling specific kernel functions to access configuration registers. As far as interrupts are concerned, every PCI device has 4 interrupt pins, whose routing to the processor IRQ lines is the responsibility of the motherboard; PCI interrupts can be shared by design, so that even a processor with a limited number of IRQ lines can host many PCI interface boards.

The I/O space in a PCI bus uses a 32-bit address bus (leading to 4GB of I/O ports), while the memory space can be accessed with either 32-bit or 64-bit addresses. Addresses are supposed to be unique to one device, but it’s possible for two devices to map erroneously to the same address, making it impossible to access either one. The good news is that every memory and I/O address region offered by the interface board can be remapped by means of configuration transactions. This is the mechanism by which the devices can be initialized at boot time to avoid address collisions. The addresses to which these regions are currently mapped can be read from the configuration space, so the Linux driver can access its devices without probing. Once the configuration registers have been read, the driver can safely access its hardware.

The PCI configuration space consists of 256 bytes for each device function, and the layout of the configuration registers is standardized. Four bytes of the configuration space hold a unique function ID, so the driver can identify its device by looking for the specific ID for that peripheral.^[38] In summary, each device board is geographically addressed to retrieve its configuration registers; this information can be used to identify the board and take further action.

It should be clear from this description that the main innovation of the PCI interface standard over ISA is the configuration address space. Therefore, in addition to the usual driver code, a PCI driver needs the ability to access configuration space.

For the remainder of this chapter, I’ll use the word ``device'' to refer to a device function, because each function in a multi-function board acts as an independent entity. When I refer to a device I mean the tuple ``bus number, device number, function number.'' As I mentioned earlier, each tuple is represented in Linux by two 8-bit numbers.

Let’s look at how PCI works, starting from system boot, since that’s when the devices are configured.

When power is applied to a PCI device, the hardware shuts down. In other words, the device will only respond to configuration transactions. At power on, the device has no memory and no I/O ports mapped in the computer’s address space; every other device-specific feature, like the interrupt lines, is disabled as well.

Fortunately, every PCI motherboard is equipped with PCI-aware firmware, called the BIOS, NVRAM, or PROM depending on the platform. The firmware offers access to the device configuration address space, even if the processor’s instruction set doesn’t offer such a capability.

At system boot, the firmware performs configuration transactions with every PCI peripheral in order to allocate a safe place for any address region it offers. By the time a device driver accesses the device, its memory and I/O regions have already been mapped into the processor’s address space. The driver can change this default assignment, but it usually doesn’t unless there are device-dependent reasons to do so.

In Linux, the user can look at the PCI device list by reading /proc/pci, which is a text file that has an entry for each PCI board in the system. Here is an example of a /proc/pci entry:

Bus  0, device  13, function  0:
  Multimedia video controller: Intel SAA7116 (rev 0).
    Medium devsel.  IRQ 10.  Master Capable.  Latency=32.
    Non-prefetchable 32 bit memory at 0xf1000000.

Each entry in /proc/pci is a summary of the device-independent features of one device as described by its configuration registers. The entry above, for example, tells us that the device has on-board memory that has been mapped to address 0xf1000000. The meaning of some of the more exotic details will become clear later, after I introduce the configuration registers.

As mentioned earlier, the layout of the configuration space is device-independent. In this section, we’ll look at the configuration registers that are used to identify the peripherals.

PCI devices feature a 256-byte address space. The first 64 bytes are standardized, while the rest is device-dependent. Figure 15.1 shows the layout of the device-independent configuration space.

Figure 15-1. The standardized PCI configuration registers

As the figure shows, some of the PCI configuration registers are required and some are optional. Every PCI device must contain meaningful values in the compulsory registers, while the contents of the optional registers depend on the actual capabilities of the peripheral. The optional fields are not used unless the contents of the compulsory fields indicate that they are valid. Thus, the compulsory fields assert the board’s capabilities, including whether the other fields are usable or not.

It’s interesting to note that the PCI registers are always little-endian. Although the standard is designed to be architecture-independent, the PCI designers sometimes show a slight bias toward the PC environment. The driver writer should be careful about byte ordering when accessing multi-byte configuration registers; code that works on the PC might not work on other platforms. The Linux developers have taken care of the byte-ordering problem (see the next section, Section 15.1.4), but the issue must be kept in mind. Unfortunately, the standard functions ntohs and ntohl can’t be used because the network byte order is the opposite of the PCI order; no standardized functions exist in Linux 2.0 to convert from PCI byte-order to host byte-order, and every driver that builds up multi-byte values from single bytes needs to be careful to deal correctly with endianness. Version 2.1.10 of the kernel introduced a few functions to deal with these byte-order issues; they are introduced in Section 17.7, in Chapter 17.

Describing all the configuration items is beyond the scope of this book. Usually, the technical documentation released with each device describes the supported registers. What we’re interested in is how a driver can look for its device and how it can access the device’s configuration space.

Three PCI registers identify a device: vendor, deviceID, and class. Every PCI peripheral puts its own values in these read-only registers, and the driver can use them to look for the device. Let’s look at these registers in more detail:

The following headers, macros, and functions should be used by a PCI driver to look for its hardware device:

The code below is typical of the code used by a driver at load time to detect its devices. As outlined above, the lookup can be based on either the signature or the device class. In either case, the driver must store the bus and function values, which are used later to identify the device. The first five bits if the function value identify the device and the next three bits identify the function.

In the code below, every device-specific symbol is prefixed with jail_ (just another instruction list), lowercase or uppercase depending on the kind of symbol.

If the driver can rely on a unique vendor/id pair, the following loop can be used to initialize the driver:

#ifdef CONFIG_PCI
    if (pcibios_present()) {
        unsigned char bus, function;
        int index, result;

        for (index=0; index < JAIL_MAX_DEV; index++) {
            result = pcibios_find_device(JAIL_VENDOR, JAIL_ID, index,
                                         &bus, &function);
            if (result != PCIBIOS_SUCCESSFUL)
                break;
            jail_init_dev(bus, function);
        }
        if (result != PCI_BIOS_DEVICE_NOT_FOUND) {
            printk(KERN_WARNING "jail: pci error: %s
",
                   pcibios_strerror(result));
        }
    }
    if (index == 0)
        return -ENODEV;
#else
   return -ENODEV; /* no PCI bios: no devices */
#endif

This code excerpt is correct if the driver deals only with one kind of PCI device, identified by JAIL_VENDOR and JAIL_ID.

However, many drivers are more flexible and can handle both PCI and ISA boards. In that case, the driver probes for the ISA device only if no PCI board is detected or if CONFIG_PCIBIOS is undefined.

It is also common for different vendors to build compatible hardware. The driver should work with all compatible devices even if their signatures are different. To do that, init_module should invoke pcibios_find_class instead of pcibios_find_device.

Using pcibios_find_class requires that jail_init_dev perform a little more work than in the example. The function returns successfully any time it finds a device belonging to the right class, but the driver still has to verify that the signature is one of the supported ones. This task is performed by a series of conditionals that end up discarding any unexpected device.

Some PCI peripherals contain a general-purpose PCI interface chip and device-specific circuitry. Every peripheral board that uses the same interface chip has the same signature and the driver must perform additional probing to be sure it is dealing with the correct peripheral device. Therefore, sometimes a function like jail_init_dev has to be ready to do the device-specific extra checking and to discard a device even though it might have the correct signature.

After the driver has detected the device, it usually needs to read from or write to the three address spaces: memory, port, and configuration. In particular, accessing the configuration space is vital to the driver because it is the only way it can find out where the device is mapped in memory and in the I/O space.

Since the microprocessor has no way to access the configuration space directly, the computer vendor has to provide a way to do it. The exact implementation is therefore vendor-dependent and not relevant to this discussion. Fortunately, the software interface to the transactions described below is standardized, and neither the driver nor the Linux kernel need to be aware of the details.

As far as the driver is concerned, the configuration space can be accessed through 8-bit, 16-bit, or 32-bit data transfers. The relevant functions are prototyped in <linux/bios32.h>:

The best way to access the configuration variables is to use the symbolic names defined in <linux/pci.h>. For example, the following two-liner retrieves the revision ID of a device by passing the symbolic name for where to pcibios_read_config_byte:

unsigned char jail_get_revision(unsigned char bus, unsigned char fn)
{
    unsigned char *revision;

    pcibios_read_config_byte(bus, fn, PCI_REVISION_ID, &revision);
    return revision;
}

When accessing multi-byte values, the programmer must remember to watch out for byte-order problems.

morgana% dd bs=256 skip=1 count=1 if=/proc/pcidata | od -Ax -t x1
1+0 records in
1+0 records out
000000 86 80 23 12 06 00 00 02 00 00 00 04 00 20 00 00
000010 00 00 00 f1 00 00 00 00 00 00 00 00 00 00 00 00
000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000030 00 00 00 00 00 00 00 00 00 00 00 00 0a 01 00 00
000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
*
000100

morgana% dd bs=256 skip=1 count=1 if=/proc/pcidata | ./pcidump
1+0 records in
1+0 records out
        Compulsory registers:
Vendor id: 8086
Device id: 1223
I/O space enabled: n
Memory enabled: y
Master enabled: y
Revision id (decimal): 0
Programmer Interface: 00
Class of device: 0400
Header type: 00
Multi function device: n
        Optional registers:
Base Address 0: f1000000
Base Address 0 Is I/O: n
Base Address 0 is 64-bits: n
Base Address 0 is below-1M: n
Base Address 0 is prefetchable: n
Does generate interrupts: y
Interrupt line (decimal): 10
Interrupt pin (decimal): 1

A PCI peripheral implements six address regions. Each region consists of either memory or I/O locations, or it doesn’t exist. Most devices substitute a memory region for their I/O ports because some processors (like the Alpha) have no native I/O space and because the I/O space on the PC is quite congested. The structural difference between memory and I/O space has been addressed by implementing a ``memory-is-prefetchable'' bit;<sup>[39]</sup> peripherals that map their control registers to a memory address range declare that range as non-prefetchable, whereas something like video memory on PCI boards is prefetchable. In this section, I’m using the word ``region'' to refer to a PCI address range whenever the discussion applies to either memory or I/O.

An interface board reports the size and current location of its regions using configuration registers--the six 32-bit registers shown in Figure 15.1 whose symbolic names are PCI_BASE_ADDRESS_0 through PCI_BASE_ADDRESS_5. Since the I/O space defined by PCI is a 32-bit address space, it does make sense to use the same configuration interface for memory and I/O. If the device uses a 64-bit address bus, it can declare regions in the 64-bit memory space by using two consecutive PCI_BASE_ADDRESS registers for each region. It is possible for one device to offer both 32-bit regions and 64-bit regions.

I won’t go into detail here, because if you’re going to write a PCI driver, you will need the hardware manual for the device anyway. In particular, I am not going to use either the prefetchable bit here or the two ``type'' bits of the registers, and I’ll limit the discussion to 32-bit peripherals. It’s nonetheless interesting to see how things are implemented in the general case and how Linux drivers deal with PCI memory.

The PCI specs state that each implemented region must be mapped to a configurable address. This means that the device must be equipped with a programmable 32-bit address decoder for each region it implements, and a 64-bit programmable decoder must be present in any board that exploits the 64-bit PCI extension. While 64-bit PCI buses don’t exist on PCs yet, some Alpha workstations have them.

The actual implementation and use of a programmable decoder is simplified by the fact that usually the number of bytes in a region is a power of two; for example 32, 64, 4KB, or 2MB. Moreover, it wouldn’t make much sense to map a region to an unaligned address; 1MB regions naturally align at an address that is a multiple of 1M, and 32-byte regions at a multiple of 32. The PCI specification exploits this alignment; it states that the address decoder must look only at the high bits of the address bus and that only the high bits are programmable. This convention also means that the size of any region must be a power of two.

Remapping a PCI region is thus performed by setting a suitable value in the high bits of a configuration register. For example, a 1M region, which has 20 bits of address space, is remapped by setting the high 12 bits of the register; writing 0x008 xxxxx to the register tells the board to respond to the 8MB-9MB address range. In practice, only very high addresses are used to map PCI regions.

This ``partial decoding'' technique has the additional advantage that the software can determine the size of a PCI region by checking the number of non-programmable bits in the configuration register. To this end, the PCI standard states that unused bits must always read as 0. By imposing a minimum size of 8 bytes for I/O regions and 16 bytes for memory regions, the standard can fit some extra information into the same PCI register: the ``space'' bit, which says whether the region is memory or I/O; the two ``type'' bits; and the prefetchable bit, which is defined only for memory. The type bits select between a 32-bit region, a 64-bit region, and a ``32-bit region that must be mapped below one meg.'' That last value is used for obsolete software that still runs on some PCs.

Detecting the size of a PCI region is simplified by using several bitmasks defined in <linux/pci.h>: PCI_BASE_ADDRESS_SPACE is set if this is a memory region, PCI_BASE_ADDRESS_MEM_MASK masks out the configuration bits for memory regions, and PCI_BASE_ADDRESS_IO_MASK masks out the bits for I/O regions.

Typical code for reporting the current location and size of the PCI regions looks like this:

static u32 addresses[] = {
    PCI_BASE_ADDRESS_0,
    PCI_BASE_ADDRESS_1,
    PCI_BASE_ADDRESS_2,
    PCI_BASE_ADDRESS_3,
    PCI_BASE_ADDRESS_4,
    PCI_BASE_ADDRESS_5,
    0
};

int pciregions_read_proc(char *buf, char **start, off_t offset,
                   int len, int unused)
{
    #define PRINTF(fmt, args...) sprintf(buf+len, fmt, ## args)
    len=0;

    /* Loop through the devices (code not printed in the book) */

        /* A device was found: print its regions */
        for (i=0; addresses[i]; i++) {
            u32 curr, mask;
            char *type;

            pcibios_read_config_dword(bus,fun,addresses[i],&curr);
            cli();
            pcibios_write_config_dword(bus,fun,addresses[i],~0);
            pcibios_read_config_dword(bus,fun,addresses[i],&mask);

            
            pcibios_write_config_dword(bus,fun,addresses[i],curr);
            sti();
            
            len += PRINTF("	region %i: mask 0x%08lx, now at 0x%08lx
",
                          i, (unsigned long)mask, (unsigned long)curr);
            if (!mask) {
                len += PRINTF("	region %i not existent
", i);
                break;
            }
            /* extract the type, and the programmable bits */
            if (mask & PCI_BASE_ADDRESS_SPACE) {
                type = "I/O"; mask &= PCI_BASE_ADDRESS_IO_MASK;
            } else {
                type = "mem"; mask &= PCI_BASE_ADDRESS_MEM_MASK;
            }
            len += PRINTF("	region %i: type %s, size %i
", i,
                          type, ~mask+1);
        }
    return len;
}

This code is part of the pciregions module, distributed in the same directory as pcidata; the module creates a /proc/pciregions file, using the code shown above to generate data. Interrupt reporting is disabled while the configuration register is being modified, to prevent a driver from accessing the region while it is mapped to the wrong place. cli is used instead of save_flags because the function is executed only during the read system calls, and interrupts are known to be enabled during system calls.

Here, for example, is what /proc/pciregions reports for my frame grabber:

Bus 0, device 13, fun  0 (id 8086-1223)
        region 0: mask 0xfffff000, now at 0xf1000000
        region 0: type mem, size 4096
        region 1: mask 0x00000000, now at 0x00000000
        region 1 not existent

The computer’s firmware uses a loop like the one shown earlier to correctly map the regions at boot time. Since the firmware prevents any collision in address assignment, Linux drivers don’t usually change the mappings of the PCI ranges.

It’s interesting to note that the memory size reported by the program above can be overstated. For instance, /proc/pciregions reports that my video board is a 16MB device. This isn’t currently true (although I could expand my video RAM). However, since the size information is used only by the firmware to allocate address ranges, region oversizing is not a problem for the driver writer who knows the internals of the device and can correctly deal with the address range assigned by the firmware.

As far as interrupts are concerned, PCI is easy to handle. The computer’s firmware has already assigned a unique interrupt number to the device, and the driver just needs to use it. The interrupt number is stored in configuration register 60 (PCI_INTERRUPT_LINE), which is one byte wide. This allows for as many as 256 interrupt lines, but the actual limit depends on the CPU being used. The driver doesn’t need to bother checking the interrupt number, as the value found in PCI_INTERRUPT_LINE is guaranteed to be the right one.

If the device doesn’t support interrupts, register 61 (PCI_INTERRUPT_PIN) is 0; otherwise, it’s non-zero. However, since the driver knows if its device is interrupt-driven or not, it doesn’t usually need to read PCI_INTERRUPT_PIN.

Thus, PCI-specific code for dealing with interrupts just needs to read the configuration byte to obtain the interrupt number, as shown in the code below. Otherwise, the information in Chapter 9 applies.

result = pcibios_read_config_byte(bus, fnct, PCI_INTERRUPT_LINE,
                                  &my_irq);
if (result) { /* deal with error */ }

The rest of this section provides additional information for the curious reader, but isn’t needed for writing drivers.

A PCI connector has four interrupt pins, and peripheral boards can use any or all of them. Each pin is individually routed to the motherboard’s interrupt controller, so interrupts can be shared without any electrical problem. The interrupt controller is then responsible for mapping the interrupt wires (pins) to the processor’s hardware; this platform-dependent operation is left to the controller in order to achieve platform independence in the bus itself.

The read-only configuration register located at PCI_INTERRUPT_PIN is used to tell the computer which single pin is actually used. It’s worth remembering that each device board can host up to 8 devices; each device uses a single interrupt pin and reports it in its own configuration register. Different devices on the same device board can use different interrupt pins or share the same one.

The PCI_INTERRUPT_LINE register, on the other hand, is read/write. When the computer is booted, the firmware scans its PCI devices and sets the register for each device according to how the interrupt pin is routed for its PCI slot. The value is assigned by the firmware because only the firmware knows how the motherboard routes the different interrupt pins to the processor. For the device driver, however, the PCI_INTERRUPT_LINE register is read-only.