A boot loader isn't unique to Linux or embedded systems. It's a program first run by a computer so that a more sophisticated program can be loaded next. In a Linux system, two boot loaders usually run before the Linux kernel starts running.

The program that the boot loader runs can be anything, but it's usually an operating system that then starts additional programs so the system can be used by a user. The interesting question is, how does the processor know to run the boot loader? At power-up, the processor goes to a certain memory address (put there by the processor designer), reads the content of that address, and performs a jump to the address stored at that location. This address is hard-coded into the processor so the chip maker doesn't have to change the silicon design for each customer or when the boot loader changes. This little bit of code is referred to as the first-stage boot loader.

The code that runs next is what is commonly viewed as the boot loader in a Linux system. This may be a program like U-Boot, RedBoot, or something else (don't worry, they get detailed treatment later in the chapter) smart enough to then load the operating system. After Linux has been loaded into memory, the boot loader is no longer needed and is discarded; any evidence of the code in RAM is overwritten by the operating system. Any device configuration done by the boot loader—for example, setting the speed of the serial port or an IP address assigned to the network adapter—is also lost. Linux reinitializes these devices during its startup process.

Boot loaders can be configured either to automatically run a sequence of commands or to wait for user input via a serial connection or console. Because embedded systems traditionally lack monitors and keyboards, you interact with the boot loader over a serial connection using a terminal emulator like minicom. The serial port is the favored way of presenting a user interface because the programming necessary to interact with a universal asynchronous receiver/transmitter (UART) that manages a serial port is an order of magnitude simpler than code to use a USB device or start a remote console over the network. The boot loader's user interface looks a little like a terminal shell in Linux, but it lacks the bells and whistles you normally find, like auto-completion and line editing.^[18]

Boot loaders also act as an interface to the flash devices on the board. Flash memory (named so because the reprogramming process was reminiscent of a flash camera to the designer) is a type of Electrically Erasable Programmable Read-Only Memory (EEPROM) where individual areas (called blocks or erase blocks) can be erased and written; before the invention of flash memory, EEPROMs could only be erased and rewritten in their entirety. With no moving parts and low power consumption, flash memory is an excellent storage medium for embedded devices. Flash support in the boot loader gives you the ability to manage the flash memory by creating segments, which are named areas of the flash memory, and to write data, such as a root file system or kernel image, into those areas.

Boot loaders have been getting increasingly sophisticated over the years. Early boot loaders only accepted one-letter commands and positional parameters and were very unforgiving. Incorrect input resulted in an unrecoverable error, and the board had to be power-cycled (or the reset button pressed, if it had one) to return to the boot loader. Help was non-existent or so minimal that it was equivalent to non-existent. The board usually included a manual explaining how to use the boot loader, or a live person showed up and helped with the boot process, depending on the value of the account.

Boot loaders on a desktop system run in two steps: a first- and a second-stage boot loader. The firststage boot loader does just enough to get the second-stage boot loader running. On a desktop system, the boot sequence reads one sector of data into memory and begins executing that code. The first-stage boot loader contains a driver so that it can access a file system on a fixed drive or possibly download the kernel from a remote source, like a TFTP server or an NFS share. On a PowerPC or ARM system, the first-stage boot loader is the code the chip runs after power-up instead of code that is loaded from the hardware.

After the boot loader does its job of getting Linux into memory, the Linux boot-up process starts. Sometimes the kernel is compressed, and the first code decompresses the kernel and jumps to an address that's the kernel's entry point. This code runs and performs processor-level configuration, such as configuring the memory management unit (MMU—the part of the processor that handles virtual memory addressing) and enabling the processor's cache. The code also populates a data structure that you can view by doing the following after the system is up and running:

$ cat /proc/cpuinfo

Next, the kernel runs its board-level configuration. Some boards have peripherals like PCI controllers or flash-management hardware that must be initialized so they can be accessed later during the kernel startup process. When it's ready, the code jumps into the processor-independent startup code. Linux is now in control of the system; the software starts the kernel's threads and process management, parses the command line, and runs the main kernel process. The kernel first runs what you indicated on the command line (via the init parameter) and then attempts to execute /sbin/init, /etc/init, /bin/init, and, finally, /bin/sh.

When the kernel starts the main user process, that process must continue running, because when the process stops the kernel panics and stops as well. A kernel panic is the worst sort of kernel error, because this sort of problem results in the system coming to halt. The following illustration describes the booting process from power-on until the first program runs.

Figure 7-1 shows the boot process in detail. The following reviews the previous discussion and explains what happens each step of the way:

Figure 7.1. Linux boot process

In Linux, a bright line exists between the kernel's execution context and that of the programs being run. The kernel's context is called kernel-land or kernel space and encompasses all the device interaction, threads, interrupt handlers, data structures, and memory. When a program is executed, the kernel creates a virtual memory address, configures resource descriptors (like those for files, timers, and semaphores), loads the program into memory, and configures the processor's registers to start execution. All of this activity occurs in userland. These two execution contexts can communicate under controlled circumstances but are otherwise separate and distinct.

Although the kernel and userland are separate and distinct, they're very much related. When the kernel boots, it initializes (and registers) devices that can be used later in the startup process or when the system is up and running. For example, when a board boots that has a USB host controller, the driver for that device initializes in the kernel; in userland, programs can run that interrogate the devices attached to the USB bus and load corresponding drivers.

Understanding the hand-off from the kernel to the first userland process helps when you're debugging startup problems and trying to reduce startup time. From a diagnostic perspective, knowing that the kernel started successfully means booting problems have been automatically narrowed to something in the root file system or initial RAM disk. From a performance perspective, knowing that a configuration step can be done in the kernel or userland (such as assigning an IP address to a network adapter) means certain activities can be deferred until later and scheduled such that system appears to boot faster, or when you can more readily control the configuration process.

Flash memory resembles other types of memory in that it's one large array of bytes. The starting address is at 0 and the ending address is one less than the total number of bytes on the device. In order to make this pool of memory more manageable, the concept of flash partitions was added a few years back; these are analogous to partitions on disk drives. With flash partitions, a small amount of flash is reserved and populated with a start address, length, and name. You can refer to the starting address using a name rather than a hex number; and when partitions change in size or location, code that refers to the names of partitions doesn't need to change.

The kernel also uses the partition table to manage the flash memory. In Linux, the device used to access flash is /dev/mtdN, where N is the ordinal of the flash partition, starting at 0. The first flash partition is 0, and the number increases for each entry subsequent partition. Memory technology device (MTD) is the generic term for EEPROM storage devices in Linux.

Flash memory comes in two flavors: NOR and NAND. Although the MTD software interface presents a uniform interface to these flavors, they're very different. The differences between NAND and NOR from a functional perspective are that because NAND memory can't be read and written in a random manner, each block within a NAND device (where a block is a subdivision of the device's memory) must be accessed sequentially. On a NAND device, the driver can't write to byte 5 on block 100 without first writing to bytes 0 through 4. NOR memory also has a different tolerance for errors: the device can have some number of bad blocks that the driver software must be able to work around, as opposed to NOR memory, which is designed to have no bad areas.

When booting, the kernel needs to know what device contains the root file system. If the kernel uses a flash-based root file system, one of these partitions contains the root file system; looking at the flash partition table tells you the location of the root file system so the proper device name can be sent to the kernel.

The kernel's entry point is located in init/main.c in the routine start_kernel. The architecturespecific code jumps to this address after the hardware setup is complete.

The sure way you know you've arrived at this step is when the following appears:

Linux version 2.6.17 ([email protected]) (gcc version 4.2.1)

The code here starts the interrupt handlers, process scheduler, virtual memory management system, and power management; it also scans the PCI bus for peripherals, enables networking, and performs other housekeeping tasks. While this activity is occurring, the device drivers print out information that documents the current state of the system:

CPU: ARM926EJ-Sid(wb) [41069265] revision 5 (ARMv5TEJ)
Machine: ARM-IntegratorCP
Memory policy: ECC disabled, Data cache writeback
CPU0: D VIVT write-through cache
CPU0: I cache: 4096 bytes, associativity 4, 32 byte lines, 32 sets
CPU0: D cache: 65536 bytes, associativity 4, 32 byte lines, 512 sets
Built 1 zonelists
Kernel command line: console=ttyAMA0 mem=128M ip=192.168.20.100:::::eth0:off

The kernel command-line printing is an important milestone: if the kernel has made it this far, then the scheduler (the part of the kernel that decides what software should be running) has started and is ready to handle threads started by the kernel itself as well as the initial user process. The output from the kernel startup process is available from the command line by using the dmesg command immediately after the kernel boots. dmesg prints out the contents of the kernel's temporary buffer, which is overwritten with newer data as the system runs. For example, on a desktop machine, try

$ dmesg | less

That command sends the results of dmesg to the less program, which lets you control the file's scrolling.

Hardware initialization continues, ending with the configuration of the network adapter:

eth0: link up
Sending DHCP requests ...... timed out!
IP-Config: Reopening network devices...
eth0: link up
Sending DHCP requests ., OK
IP-Config: Got DHCP answer from 10.0.2.2, my address is 10.0.2.15
IP-Config: Complete:
      device=eth0, addr=10.0.2.15, mask=255.255.255.0, gw=10.0.2.2,
     host=10.0.2.15, domain=, nis-domain=(none),
     bootserver=10.0.2.2, rootserver=10.0.2.2, rootpath=

These messages show an adapter that attempts to get an address via Dynamic Host Configuration Protocol (DHCP) and succeeds on the second attempt. DHCP is a protocol in which the board sends out a packet to the entire network requesting an address and waits for a reply. Because the board doesn't know if it will get a reply, it makes several attempts before giving up.

The kernel initialization of the hardware interface is necessary for kernels that will attempt to mount a root file system over the network using NFS or PXE, both of which were covered earlier in the book. If the kernel doesn't need to mount a remote file system as root, it doesn't need to configure the network adapter at this point, because this may happen later in the boot process; doing so results in the system getting to userland slightly faster, and you have more control over the assignment of the IP address if the operation is deferred until a userland scripting language can be used.

# mkdir /sys
# mount -t sysfs none /sys
$ cd /sys

root=/dev/mtd0 rootfstype=jffs2

/sbin/init
/etc/init
/bin/init
/bin/sh

No init found. Try passing init= option to kernel.

ip=host-ip:server-ip:gw-ip:netmask:hostname:device:auto-configure

ip=10.0.0.1:10.0.0.2:10.0.0100:255.0.0.0:theboard:eth0:off

ip=:::::eth0:dhcp

nfsroot=nfs-server-ip-address:export-name,options

nfsroot=10.0.0.2:/opt/boardrfs

At this point, the system kernel is up and running, the root file system is mounted, and the system has found something to run. On most desktop systems, that something to run is usually init, but it could be any program. To find out, do the following:

$ ps aux | head −5
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0   3056   128 ?        Ss   Apr19    0:01 /sbin/init
root         2  0.0  0.0      0     0 ?        S<   Apr19    0:00 [kthreadd]
root         3  0.0  0.0      0     0 ?        S<   Apr19    0:00 [migration/0]
root         4  0.0  0.0      0     0 ?        S<   Apr19    0:00 [ksoftirqd/0]

The process with the PID of 1 is the init process, in this case /sbin/init. On a desktop system, this file is either SysV init or Upstart (from Ubuntu, the binary name is called init, but it's nothing like the SysV init). For an embedded system, both of these solutions are overkill.

SysV init, the most common init, works by running certain programs based on a runlevel. A runlevel is an integer that represents what tasks the system should be running. The level part of runlevel may cause you to think that runlevels are scalar; however, this number is just an identifier. Thus runlevel 1 isn't "less than" runlevel 4. Systems with a graphical environment typically start in runlevel 5. Table 7-4 lists the conventions for assigning programs that should be run for a given runlevel

The parameters controlling how init works are stored in /sbin/init. The format for this file is cryptic; it's composed of lines in this format (see Table 7-5).

id:runlevel:action:process

There are many potential actions for each line. Most are unused, but it's helpful to know what's available. The order of Table 7-6 is the order in which init executes the actions.

The sysinit file can be very difficult to parse. Table 7-7 breaks down a sample inittab file and explains each part in detail.

id:2:initdefault:

si::sysinit:/etc/init.d/rcS

l0:0:wait:/etc/init.d/rc 0
l1:1:wait:/etc/init.d/rc 1
l2:2:wait:/etc/init.d/rc 2
l3:3:wait:/etc/init.d/rc 3
l4:4:wait:/etc/init.d/rc 4
l5:5:wait:/etc/init.d/rc 5
l6:6:wait:/etc/init.d/rc 6

ca::ctrlaltdel:/sbin/shutdown -t1 -h
now

1:23:respawn:/sbin/getty tty1 VC
linux
2:23:respawn:/sbin/getty tty2 VC
linux
3:23:resawn:/sbin/getty tty3 VC linux

This /etc/sysinit results in the following commands being run in the following order:

/etc/init.d/rcS
/etc/init.d/rc 2
/sbin/getty tty3 VC linux

Init continues to execute the last command any time it stops. The init command doesn't indiscriminately reexecute this command. If the command fails quickly a few times, init waits a few seconds before attempting to run it again.

K|S##command

BusyBox's init implementation is a common replacement for the SysV init. The BusyBox init program works much like the original, with the following exceptions

The default init file is usually good enough to quickly get a system running. A production system needs to define something different.

The program run for init can be anything that the system can execute: a binary, shell script, Perl script—the only requirement is that it can be run. Like any other init process, when this program stops, the kernel panics and halts. That means the program must be coded so that it works something like the respawn command in init. The typical strategy for implementing this sort of protection is to spawn the program such that the system runs in a separate process and restarts when it fails.

The code for this sort of program follows this pattern:

#include <sys/wait.h>
#include <stdlib.h>
#include <unistd.h>
#include <stdio.h>

#define THE_APPLICATION "somebinary"

int main(int argc, char **argv)
{
    pid_t app, appstatus;
    int status;

    while (1) {
      app = fork();
      if (app == −1) {
        perror("could not fork, application not run");
        exit(EXIT_FAILURE); /* this is a panic */
      }

      if (app == 0) {
        execl(THE_APPLICATION, "");
        sleep(3);
      }
      else {
        appstatus = waitpid(app, &status, 0);
        printf("process exited with %d status, restarting
", status);
      }
   }
}

The code is very simple in that when the program fails, the controlling program prints out a message, and the loop restarts. This code is basic enough that the opportunity for bugs is low; the simplicity means the resulting binary is small as well. The execution time for this approach is very low, because this init program doesn't need to open and parse a file in order to run the application.