4.3. Kernel Module Loading and Linking
Much of the kernel architecture is layered architecture, and many of the kernel's major subsystems call into a common set of lower-level services. In some respects, the kernel architecture is similar to that of a client/server application, except that the interfaces in this case are, for the most part, private to the kernel. That is, many of the callable routines are not public interfaces available for use for general application software development. The other obvious distinction is that the users of the services, or clients, in this context, are other kernel subsystems, as opposed to application code. Exceptions to the private interface generalization are those interfaces defined for device drivers and kernel STREAMS modules, which are documented as the Solaris Device Driver Interface (DDI) specifications, found in section 9 of the manual pages.
The kernel module loading facility serves as a good example of one such service. Several phases are involved in dynamic module loading:
Load the module (a binary object file) into memory.
Establish kernel address space mappings for module segments.
Link the module's segments into the kernel.
Perform the module-type-specific install function.
The loadable module types are defined in /usr/include/sys/modctl.h; they are device drivers, systems calls, file systems, miscellaneous modules, streams modules, scheduling classes, and exec modules (exec support for executable objects). For each of these module types, some specific kernel installation steps are required to complete the loading process. We will take a look at these shortly.
The steps involved in loading a module into the kernel are similar conceptually to what happens when a dynamically linked program is started under Solaris. That is, shared objects to which a binary is linked are dynamically loaded into memory and linked into the process's address space during execution. The kernel exists in memory as a large, dynamically linked executable, with essentially the same address space segments that exist in any process running on the system: memory mappings for text (instructions), data (initialized and uninitialized), and a stack. Thus, loading a kernel module involves reading the target module into memory and creating the address space mappings in the kernel's address space for the module's text and data. This process is illustrated in Figure 4.4.
Figure 4.4. Loading a Kernel Module
The illustration in Figure 4.4 is not a precise depiction of the kernel address space or binary object segments but provides a conceptual view. Subsequent chapters describe executable objects and address space mappings in detail.
The system loads the modules required for a functional system at boot time. A path variable that defines the module search path (which directories to search for loadable modules) guides the boot code in locating the appropriate kernel objects. The kernel modload() function is the entry point in the code that begins the module load process. The primary data structures used in support of module loading are the modctl (module control) structure and module structure. You can find the structure definitions for modctl and module in /usr/include/sys/modctl.h and /usr/include/sys/kobj.h, respectively.
The kernel maintains a linked list of modctl structures for all modules in the kernel, as shown in Figure 4.5.
Figure 4.5. Module Control Structures
The kernel modules pointer marks the beginning of the doubly linked list of module control structures. Not every structure member is shown in the figure. In addition to the structure links (next and previous pointers), the module identification (mod_id) is maintained, along with links to a mod_modinfo structure and mod_linkage structure. mod_mp links to a module structure, defined in the kernel object linker code (/usr/include/sys/kobj.h) and used by the kernel runtime linker and the module load facility. Other interesting bits include character string pointers to the directory name, mod_filename, that holds the module and the module filename, mod_modname.
A given kernel module may depend on the existence of other kernel modules in order to function or may have other kernel modules depend on it. For example, the System V Interprocess Communication (IPC) modules for shared memory, semaphores, and message queues all depend on the kernel ipc module to function. Such dependencies are defined in the module code and maintained in the modctl structure by pointers in mod_requisites (other modules this module depends on) and mod_dependents (modules that depend on this one).
The kernel module loading facility is threaded. That is, when the kernel modload() function is called (during bootstrap or by the modload(1M) command), it creates a kernel thread to perform the module load function. The benefit here is that concurrent module loads are possible on multiprocessor systems and so provide faster system boot and initialization. Once the kernel thread is created, the following series of events is executed to complete the loading and linking of the module:
Create and allocate a modctl structure. First, search the linked list of modctl structures, looking for a match to the module name (mod_modname). If a match is found, return the address of an existing structure; otherwise, create a new one. Add a new modctl structure to the linked list.
Enter the kernel runtime linker, krtld, to create address space segments and bindings, and load the object into memory.
Allocate module structure().
Allocate space for the module's symbols in the kernel's kobj_map resource map.
Loop through the segments of the module being loaded, and allocate and map space for text and data.
Load the kernel object into memory, linking the object's segments into the appropriate kernel address space segments.
Set the mod_loaded bit in the module's modctl structure, and increment the mod_loadcnt.
Create a link to the module's mod_linkage structure.
As the preceding steps indicate, the major kernel subsystems involved in module loading are the module facility and the kernel's runtime linker, krtld, which is loaded very early in the bootstrap procedure. The module subsystem does not free a modctl structure when a module in unloaded. The structure remains on the linked list, and the mod_loaded bit is cleared. This is why step 1 searched the list first; in case the module was loaded and subsequently unloaded, the modctl structure would already exist. This is also why a mod_loaded status bit is maintained—the existence of a modctl structure does not necessarily mean that the module is loaded.
The facts that the Solaris kernel is dynamic in nature and that kernel objects can be loaded and unloaded during the life of a running system require that the kernel's symbol table (step 2) exist as a dynamic entity. All executable object files have a symbol table that holds information required to resolve an object's symbolic references. A symbolic reference is the correlation of the virtual address of a function or variable, and its name. The Solaris kernel's symbol table is maintained through a pseudodevice, /dev/ksyms, and corresponding device driver, /usr/kernel/drv/ksyms. In Solaris 7, the kernel symbol table is updated by a kernel thread created specifically for that purpose. The kernel runtime linker issues a wakeup to the ksyms_update_thread() when a module is loaded (or unloaded), and the kernel symbol table is updated to reflect the current state of loaded kernel objects.
In Solaris 2.5.1 and 2.6, a different update mechanism is used. A kernel variable is updated when a module is loaded or unloaded. Inside the ksyms driver ksyms_open() code, the variable is tested to determine whether a new symbol table image needs to be created. The implication here is that if an open has been issued in the ksyms driver, meaning that a user (or program) is examining the kernel symbol table, and a kernel module is loaded or unloaded, then the currently opened version will not reflect the change. A close and subsequent open must be issued for an updated view. You can use the nm(1) command to examine an object's symbol table; use /dev/ksyms to examine the kernel's table.
# nm -x /dev/ksyms | grep modload [1953] |0xf011086c|0x000000ac|FUNC |LOCL |0 |ABS |modctl_modload [10072] |0xf011113c|0x000000a0|FUNC |GLOB |0 |ABS |modload [1973] |0xf0111398|0x000000b8|FUNC |LOCL |0 |ABS |modload_now [1972] |0xf01111dc|0x000000c8|FUNC |LOCL |0 |ABS |modload_thread [9926] |0xf0111450|0x000000a4|FUNC |GLOB |0 |ABS |modloadonly
The preceding example searches the symbol table of the running kernel for modload, a kernel function we discussed earlier. The command returned several matches that contain the modload string, including the desired modload function symbol. (For more information on symbol tables and specific information on the columns listed, see the nm(1), a.out(4), and elf(3E) manual pages. Also, refer to any number of texts that describe the Executable and Linking Format (ELF) file, which is discussed in more detail earlier in this chapter.)
In step 5, we indicate that the module install code is invoked indirectly through the module's _init() function. Several functions must be included in any loadable kernel module to facilitate dynamic loading. Device drivers and STREAMS modules must be coded for dynamic loading. As such, a loadable driver interface is defined. In general, the required routines and data structures that are documented apply to all loadable kernel modules—not just to drivers and STREAMS modules (although there are components that are specific to drivers)—and do not apply to objects such as loadable system calls, file systems, or scheduling classes.
Within a loadable kernel object, an initialization, information, and finish routine must be coded, as per the definitions in the _init(9E), _info(9E), and _fini(9E) manual pages. A module's _init() routine is called to complete the process of making the module usable after it has been loaded. The module's _info() and _fini() routines also invoke corresponding kernel module management interfaces, as shown in Table 4-2.
| Kernel Module Routine | Module Facility Interface | Description |
|---|---|---|
| _init() | mod_install() | Loads a kernel module. |
| _info() | mod_info() | Retrieves module information. |
| _fini() | mod_remove() | Unloads a kernel module. |
Module installation is abstracted to define a generic set of structures and interfaces within the kernel. Module operation function pointers for installing, removing, and information gathering (the generic interfaces shown in Table 4-2) are maintained in a mod_ops structure, which is extended to provide a definition for each type of loadable module. For example, there is a mod_installsys() function specific to loading system calls, a mod_installdrv() function specific to loading device drivers, and so forth.
For each of these module types, a module linkage structure is defined; it contains a pointer to the operations structure, a pointer to a character string describing the module, and a pointer to a module-type-specific structure. For example, the linkage structure for loadable system calls, modlsys, contains a pointer to the system entry table, which is the entry point for all system calls. Each loadable kernel module is required to declare and initialize the appropriate type-specific linkage structure, as well as a generic modlinkage structure that provides the generic abstraction for all modules.
Within the module facility is a module type-specific routine for installing modules, entered through the MODL_INSTALL macro called from the generic mod_install() code. More precisely, a loadable module's _init() routine calls mod_install(), which vectors to the appropriate module-specific routine through the MODL_INSTALL macro. This procedure is shown in Figure 4.6.
Figure 4.6. Module Operations Function Vectoring
Figure 4.6 shows the data structures defined in a loadable kernel module: the generic modlinkage, through which is referenced a type-specific linkage structure (modlxxx), which in turn links to a type-specific operations structure that contains pointers to the type-specific functions for installing, removing, and gathering information about a kernel module. The MODL_INSTALL macro is passed the address of the module's generic linkage structure and from there vectors in to the appropriate function. The module-specific installation steps are summarized in Table 4-3.
| Module Type | Install Function | Summary |
|---|---|---|
| Device driver | mod_installdrv | Wrapper for ddi_installdrv(). Installs the driver entry in the kernel devops table. |
| System call | mod_installsys | Installs the system call's sysent table entry in the kernel sysent table. |
| File system | mod_installfs | Installs the file system Virtual File System (VFS) switch table entry. |
| STREAMS modules | mod_installstrmod | Installs the STREAMS entry in the kernel fmodsw switch table. |
| Scheduling class | mod_installsched | Installs the scheduling class in the kernel sclass array. |
| Exec module | mod_installexec | Installs the exec entry in the kernel execsw switch table. |
The summary column in Table 4-3 shows a definite pattern to the module installation functions. In many subsystems, the kernel implements a switch table mechanism to vector to the correct kernel functions for a specific file system, scheduling class, exec function, etc. The details of each implementation are covered in subsequent areas of the book, as applicable to a particular chapter or heading.
As we've seen, the dynamic loading of a kernel module is facilitated through two major kernel subsystems: the module management code and the kernel runtime linker. These kernel components make use of other kernel services, such as the kernel memory allocator, kernel locking primitives, and the kernel ksyms driver, taking advantage of the modular design of the system and providing a good example of the layered model discussed earlier.