As I said earlier, the glibc package contains a number of libraries. You can see the entire list of libraries installed during the package’s build process by looking at your ${TARGET_PREFIX}/lib directory. This directory contains mainly four types of files:

You will also find some other types of files in ${TARGET_PREFIX}/lib, such as crti.o and crt1.o, but you will not need to copy these to your target’s root filesystem.

Out of the four types of files described above, we will need only two for each library: the actual shared libraries and the major revision version symbolic links. The two other file types are needed only when linking executables and are not required for the runtime operation of our applications.

In addition to the library files, we will need to copy the dynamic linker and its symbolic link. The dynamic linker itself follows the naming convention of the various glibc libraries, and is usually called ld- GLIBC_VERSION .so. In what is probably one of the most bizarre aspects of the GNU toolchain, however, the name of the symbolic link to the dynamic linker depends on the architecture for which the toolchain has been built. If the toolchain is built for the i386, the ARM, the SuperH, or the m68k, the symbolic link to the dynamic linker is usually called ld-linux.so. MAJOR_REVISION_VERSION. If the toolchain is built for the MIPS or the PowerPC, the symbolic link to the dynamic linker is usually called ld.so. MAJOR_REVISION_VERSION.

Before we actually copy any glibc component to the target’s root filesystem, however, we need to select the glibc components required for our applications. Table 6-2 provides the description of all the components in glibc^[1] and provides inclusion guidelines for each component. In addition to my guidelines, you will need to evaluate which components your programs need, depending on their linking.

Apart from keeping track of which libraries you link your applications with, you can usually use the ldd command to find out the list of dynamic libraries that an application depends on. In a cross-platform development environment, however, your host’s ldd command will fail when provided with target binaries. You could still use the cross-platform readelf command we installed in Chapter 4 to identify the dynamic libraries that your application depends on. Here is an example showing how the BusyBox utility’s dependencies can be retrieved using readelf:

$ powerpc-linux-readelf -a ${PRJROOT}/rootfs/bin/busybox | 
               > grep "Shared library"
 0x00000001 (NEEDED)                     Shared library: [libc.so.0]

Ideally, however, if you installed uClibc, you should use the cross-platform capable ldd-like command installed by uClibc. For our control module target, which is based on a PowerPC board, the command’s name is powerpc-uclibc-ldd. This way, you can build the list of libraries your target binaries depend on. Here are the dependencies of the BusyBox utility, for example (one line has been wrapped to fit the page):

$ powerpc-uclibc-ldd ${PRJROOT}/rootfs/bin/busybox
        libc.so.0 => /home/karim/control-project/control-module/tools/uclibc/lib/
            libc.so.0
/lib/ld-uClibc.so.0 => /lib/ld-uClibc.so.0

Having determined the library components we need, we can copy them and the relevant symbolic links to the /lib directory of the target’s root filesystem. Here is a set of commands that copy the essential glibc components:

$ cd ${TARGET_PREFIX}/lib
$ for file in libc libcrypt libdl libm 
               > libpthread libresolv libutil
               > do
               > cp $file-*.so ${PRJROOT}/rootfs/lib
               > cp -d $file.so.[*0-9] ${PRJROOT}/rootfs/lib
               > done
$ cp -d ld*.so* ${PRJROOT}/rootfs/lib

The first cp command copies the actual shared libraries, the second one copies the major revision version symbolic links, and the third one copies the dynamic linker and its symbolic link. All three commands are based on the rules outlined earlier in this section regarding the naming conventions of the different files in ${TARGET_PREFIX}/lib. The -d option is used with the second and third cp commands to preserve the symbolic links as-is. Otherwise, the files pointed to by the symbolic links are copied in their entirety.

Of course, you can remove the libraries that are not used by your applications from the list in the set of commands above. If you would rather have the complete set of libraries included in glibc on your root filesystem, use the following commands:

$ cd ${TARGET_PREFIX}/lib
$ cp *-*.so ${PRJROOT}/rootfs/lib
$ cp -d *.so.[*0-9] ${PRJROOT}/rootfs/lib
$ cp libSegFault.so libmemusage.so libpcprofile.so 
               > ${PRJROOT}/rootfs/lib

If you have applications that use the glibc NSS, don’t forget to copy the libnss_ SERVICE libraries you need to your target’s root filesystem. libnss_files and libnss_dns are the ones most often used. You will also need to copy the sample nsswitch.conf provided with glibc to your target’s /etc directory and customize it to your setup:^[4]

$ cp ${PRJROOT}/build-tools/glibc-2.2.1/nss/nsswitch.conf 
               > ${PRJROOT}/rootfs/etc

Whether you copy all or part of the glibc libraries, you will notice that some of these libraries are large. To reduce the size of the libraries installed, we can use the cross-platform strip utility we built earlier. Be careful not to strip the original libraries, since you would have to install them all over again. Strip the libraries only after you copy them to the root filesystem:

$ powerpc-linux-strip ${PRJROOT}/rootfs/lib/*.so

On my control module, the ${PRJROOT}/rootfs/lib directory with all the glibc libraries weighs around 10 MB before stripping. By stripping all the libraries, the directory is reduced to 2.5 MB.

The glibc components have now been installed on the target’s root filesystem and are ready to be used at runtime by our applications.

As with glibc, uClibc contains a number of libraries. You can see the entire list by looking at your ${PREFIX}/uclibc/lib directory. This directory contains the same four different types of files as the glibc directory.

Because uClibc is meant to be a glibc replacement, the names of the uClibc components and their use is identical to the glibc components. Hence, you can use Table 6-2 for uClibc components. Note, however, that not all glibc components are implemented by uClibc. uClibc implements only ld, libc, libcrypt, libdl, libm, libpthread, libresolv, and libutil. Use the same method as described for glibc to identify the uClibc components you will need on your target.

Having determined the list of components we need, we can now copy them and their relevant symbolic links to the /lib directory of our target’s root filesystem. The following set of commands copies the essential uClibc components:

$ cd ${PREFIX}/uclibc/lib
$ for file in libuClibc ld-uClibc libc libdl 
               > libcrypt libm libresolv libutil
               > do
               > cp $file-*.so ${PRJROOT}/rootfs/lib
               > cp -d $file.so.[*0-9] ${PRJROOT}/rootfs/lib
               > done 

The commands are likely to report that two files haven’t been found:

cp: libuClibc.so.[*0-9]: No such file or directory
cp: libc-*.so: No such file or directory

This is not a problem, since these files don’t exist. The set of commands above is meant to be easy to type in, but you could add conditional statements around the cp commands if you prefer not to see any errors.

As with glibc, you can modify the list of libraries you copy according to your requirements. Note that, in contrast to glibc, you will not save much space by copying only a select few uClibc components. For my control module, for instance, the root filesystem’s /lib directory weighs around 300 KB when all the uClibc components are copied. The following commands copy all uClibc’s components to your target’s root filesystem:

$ cd ${PREFIX}/uclibc/lib
$ cp *-*.so ${PRJROOT}/rootfs/lib
$ cp -d *.so.[*0-9] ${PRJROOT}/rootfs/lib

There is no need to strip uClibc components, since they were already stripped by uClibc’s own build scripts. You can verify this using the file command.

If you would like to selectively include some of the standard applications found in mainstream distributions, your best bet is to start with the Linux From Scratch project located at http://www.linuxfromscratch.org/. This project aims at providing explanations and links to packages to help you build your own custom distributions. The Linux From Scratch book available through the project’s web site is its main documentation. It includes instructions and links to build each application one by one. For each package, the instructions provide build-time and disk-space estimates.

Alternatively, you can download applications off the Net one by one and follow the instructions of each package for compiling and cross-compiling. Because few packages include full cross-compilation instructions, you may need to look in the packages’ Makefiles to determine the appropriate build flags or make the proper modifications for the packages to cross-compile adequately.

The BusyBox project was initiated by Bruce Perens in 1996 to help build install disks for the Debian distribution. Since 1999, the project has been maintained by Erik Andersen, the maintainer of uClibc, first as part of Lineo’s open source efforts and currently as a vendor-independent project. During this time, the BusyBox project has grown extensively and is now one of the corner stones of many embedded Linux systems. It is included in most embedded Linux distributions and has a very active user community. The project’s current location is http://www.busybox.net/. The project’s web site includes documentation, links, and a mailing list archive. The BusyBox package is available under the terms of the GNU GPL from this same web site.

Enthusiasm for BusyBox stems from the functionality it provides while still remaining a very small-sized application. BusyBox implements many commands. Here are a few: ar, cat, chgrp, chmod, chown, chroot, cp, cpio, date, dd, df, dmesg, dos2unix, du, echo, env, expr, find, grep, gunzip, gzip, halt, id, ifconfig, init, insmod, kill, killall, ln, ls, lsmod, md5sum, mkdir, mknod, modprobe, more, mount, mv, ping, ps, pwd, reboot, renice, rm, rmdir, rmmod, route, rpm2cpio, sed, stty, swapon, sync, syslogd, tail, tar, telnet, tftp, touch, traceroute, umount, uname, uuencode, vi, wc, which, and whoami.

Although BusyBox does not support all the options provided by the commands it replaces, the subset it provides is sufficient for most typical uses. See the docs directory of the BusyBox distribution for the documentation in a number of different formats.

BusyBox supports all the architectures covered in Chapter 3. It can be linked both statically and dynamically to either glibc or uClibc. You can also modify the BusyBox default build configuration to remove support for the commands you are unlikely to use.

$ cd ${PRJROOT}/sysapps/busybox-0.60.5

$ make TARGET_ARCH=ppc CROSS=powerpc-linux- 
                  > PREFIX=${PRJROOT}/rootfs all install

$ make TARGET_ARCH=ppc CROSS=powerpc-uclibc- 
                  > PREFIX=${PRJROOT}/rootfs all install

# Set path
PATH=/bin:/sbin:/usr/bin:/usr/sbin
export PATH

Much like BusyBox, TinyLogin is a collection of many login utilities into a single binary. TinyLogin is often used in conjunction with BusyBox, although it can be used alone. Both packages are maintained by the same developers and are therefore easy to use together. Because of their common use together, the project developers have integrated all of TinyLogin’s functionality into the BusyBox CVS, and once the CVS development version is released as a stable version, it will be possible to rely on a single package instead of two. There are, however, advantages to continue using the TinyLogin functionality separately from BusyBox. Mainly, many of the commands implemented in TinyLogin must run with root privileges, which in turn requires that the TinyLogin binary file belong to the root user and have its “set user” permission bit enabled—a configuration commonly known as “setuid root.” Since TinyLogin uses symbolic links in the same way BusyBox does, a single binary containing the functionality of both packages would also result in having commands such as ls and cat run as root, which increases the likeliness that a programming error in one command could be exploited to gain root privileges. Though BusyBox drops its root privileges when unnecessary, and though it can be configured to check a configuration file for those commands requiring root privileges, it remains that using separate packages is the safest setup.

The TinyLogin project’s web site is located at http://tinylogin.busybox.net/. It contains documentation, a mailing list archive, links to other sites, and pointers to download the TinyLogin package both using FTP or CVS. For my control module, I will be using TinyLogin 1.2.

As with BusyBox, TinyLogin supports all the architectures we discussed in depth in Chapter 3 and can be linked either statically or dynamically with glibc or uClibc. TinyLogin can effectively replace the following commands: addgroup, adduser, delgroup, deluser, getty, login, passwd, su, sulogin, and vlock.

$ cd ${PRJROOT}/sysapps/tinylogin-1.2

$ make CROSS=powerpc-linux- 
                  > PREFIX=${PRJROOT}/rootfs all

$ make CROSS=powerpc-uclibc- 
                  > PREFIX=${PRJROOT}/rootfs all

$ su -m
Password:
# make PREFIX=${PRJROOT}/rootfs install
# exit

tmp:$1$3cdOSELf$XWRLoKIL7vMSfLYbRCWaf/:11880:0:99999:7:-1:-1:0

root:$1$3cdOSELf$XWRLoKIL7vMSfLYbRCWaf/:11880:0:99999:7:-1:-1:0

root:x:0:0:root:/root:/bin/sh

PATH=/bin:/usr/bin

PATH=/bin:/sbin:/usr/bin:/usr/sbin
export PATH

embutils is a set of miniaturized and optimized replacements for mainstream Unix commands. embutils was written and is maintained by Felix von Leitner, the author of diet libc, with goals very similar to those of diet libc. Currently, embutils supports four of the architectures discussed in Chapter 3, the ARM, the i386, the PPC, and the MIPS. embutils is available from http://www.fefe.de/embutils/.^[7]

Although embutils groups some of the commands in a single binary, its main approach is to provide one small binary for each command. embutils provides the following commands: arch, basename, cat, chmgrp, chmod, chown, chroot, chvt, clear, cp, dd, df, dirname, dmesg, domainname, du, echo, env, false, head, hostname, id, install, kill, ln, ls, md5sum, mesg, mkdir, mkfifo, mknod, mv, pwd, rm, rmdir, sleep, sleep2, soscp, sosln, soslns, sosmv, sosrm, sync, tail, tar, tee, touch, tr, true, tty, uname, uniq, wc, which, whoami, write, and yes.

As with BusyBox, not all the options provided by the full commands are supported, but the subset provided is sufficient for most system operations. In contrast to BusyBox, however, embutils can only be statically linked with diet libc. It can’t be linked to any other library. Because diet libc is already very small, the resulting command binaries are reasonably small. In terms of overall size, nevertheless, BusyBox and embutils are fairly similar.

$ cd ${PRJROOT}/sysapps/embutils-0.15

$ make ARCH=ppc CROSS=powerpc-linux- all

$ make ARCH=ppc DESTDIR=${PRJROOT}/rootfs prefix="" install

The standard init package found in most Linux distributions is written by Miquel van Soorenburg and is available at ftp://ftp.cistron.nl/pub/people/miquels/sysvinit/. By using this package, you get the same flexibility to configure your target’s startup as you would in configuring the startup of a workstation or a server. However, the extra functionality and flexibility requires additional space. Also, it requires that you keep track of the development of yet another software package. The package includes the following commands: halt, init, killall5, last, mesg, runlevel, shutdown, sulogin, utmpdump, and wall.

The package can be cross-compiled easily. First, download the package and uncompress it into your ${PRJROOT}/sysapps directory. For my control module, I used sysvinit Version 2.84. Then, move into the package’s source directory and build it:

$ cd ${PRJROOT}/sysapps/sysvinit-2.84/src
$ make CC=powerpc-linux-gcc

Replace the value of CC to match the cross-compiler for your target. With the package now built, we can install it on the target’s root filesystem:

$ make BIN_OWNER="$(id -un)" BIN_GROUP="$(id -gn)" 
               > ROOT=${PRJROOT}/rootfs install

This command will install all the binaries in the target’s root filesystem but will fail afterward, since the Makefile tries to install the manpages on the root filesystem as well. You can modify the Makefile to avoid this, but you can also ignore the failure.

The command just shown set the BIN_OWNER and BIN_GROUP variables to be that of your own current user. By default, the Makefile attempts to install the various components and set their ownership to the root user. Since you aren’t logged in as root, the Makefile would fail. The ownership of the binaries matters little on the target, since it isn’t a multiuser system. If it were, however, you need to log in as root and then run the install command. Be very careful, in any case, to appropriately set the value of ROOT to point to your target’s root filesystem. Otherwise, you may end up overwriting your workstation’s init with a target binary. Alternatively, to avoid having to log in as root, you could still run the install command using your normal user privileges and then use the chown command as root to change the privileges on each file installed. This, however, involves going through the Makefile to find each file installed and its destination.

With init installed on your target’s root filesystem, you will need to add the appropriate /etc/inittab file and fill the /etc/rc.d directory with the appropriate files. In essence, /etc/inittab will define the runlevels for your system, and the files in /etc/rc.d will define which services run on each runlevel. Table 6-5 lists init’s seven runlevels and their typical use in a workstation and server distribution.

Each runlevel corresponds to a certain set of applications. When entering runlevel 5 on a workstation, for example, init starts X11 and the user is prompted to enter his username and password using a graphical login. When switching between runlevels, the services started in the previous runlevel are shut down and the services of the new runlevel are started. In this scheme, runlevels 0 and 6 have a special meaning. Particularly, they are used for stopping the system safely. This may involve, for example, unmounting all the filesystems except the root filesystem and remounting the root filesystem read-only so that no filesystem corruption occurs.

On most workstations, the default runlevel at system startup is 5. For an embedded system, it can be set to 1, if no access control is necessary. The system’s runlevel can be changed after system startup either using init or telinit, which is a symbolic link to init. In both cases, the newly issued init command communicates with the original init through the /dev/initctl fifo. To this end, we need to create a corresponding entry in our target’s root filesystem:

$ mknod -m 600 ${PRJROOT}/rootfs/dev/initctl p

For more information on the format of /etc/inittab and the files found in /etc/rc.d, refer to the resources provided above.

Among the commands it supports by default, BusyBox provides init-like capabilities. As with the original mainstream init, BusyBox can handle the system’s startup. BusyBox init is particularly well adapted to embedded systems, because it provides most of the init functionality an embedded system typically needs without dragging the weight of the extra features found in System V init. Also, because BusyBox is a single package, there is no need to keep track of an additional software package when developing or maintaining your system. There are cases, however, where BusyBox init may not be sufficient for your system. BusyBox init, for example, does not provide runlevel support.

Since I already described how to obtain, configure, and build BusyBox, I will limit this discussion to the setup of the init configuration files.

Because /sbin/init is a symbolic link to /bin/busybox, BusyBox is the first application to run on the target system. BusyBox identifies that the command being invoked is init and immediately jumps to the init routine.

The init routine of BusyBox carries out the following main tasks in order:

Once it has done this, the init routine loops forever carrying out the following tasks:

During console initialization, BusyBox determines whether the system was configured to run the console on a serial port (by passing console=ttyS0 as a kernel boot parameter, for instance). If so, BusyBox versions prior to 0.60.4 used to disable all virtual terminals. Since 0.60.4, however, BusyBox continues through its initialization without disabling virtual terminals. If in fact there are no virtual terminals, its attempts to start shells on some virtual terminals later will fail anyway, so there is no need to disable virtual terminals outright.

After having initialized the console, BusyBox checks for the existence of an /etc/inittab file. If no such file exists, BusyBox uses a default inittab configuration. Mainly, it sets up default actions for system reboot, system halt, and init restart. Also, it sets up actions to start shells on the first four virtual consoles, /dev/tty1 through /dev/tty4. BusyBox will complain if you haven’t created these device entries.

If an /etc/inittab file is found, it is parsed and the commands it contains are recorded inside internal structures to be carried out at the appropriate time. The format of the inittab file as recognized by BusyBox is well explained in the documentation included in the BusyBox package. The documentation provided in the BusyBox package includes an elaborate example inittab file.

Each line in the inittab file follows this format:

               id:runlevel:action:process

Although this format resembles that of traditional System V init, take note that the meaning of id is different in BusyBox init. Mainly, the id is used to specify the controlling tty for the process to be started. You can safely leave this entry empty if the process to be started isn’t an interactive shell. Interactive shells, such as BusyBox’s sh, should always have a controlling tty. BusyBox’s sh will actually complain if it has no controlling tty. BusyBox completely ignores the runlevel field, so you can leave it blank. The process field specifies the path of the program to run, along with its command-line options. The action field is one of eight recognized actions to be applied to process as described in Table 6-6.

The following is a simple inittab file for my control module:

::sysinit:/etc/init.d/rcS
::respawn:/sbin/getty 115200 ttyS0
::respawn:/control-module/bin/init
::restart:/sbin/init
::shutdown:/bin/umount -a -r

This inittab file does the following:

The id is left blank in this case, because it doesn’t matter to the normal operation of the commands. runlevel is also left blank, since it’s completely ignored by BusyBox.

As shown earlier, however, none of these actions will take place until init runs the system initialization script. This script can be quite elaborate and can actually call other scripts. Use this script to set all the basic settings and initialize the various components of the system that need special handling. Particularly, this is a good place to:

Here is the initialization script for my control module:

#!/bin/sh
 
# Remount the root filesystem in read-write (requires /etc/fstab)
mount -n -o remount,rw /

# Mount /proc filesystem
mount /proc
 
# Start the network interface
/sbin/ifconfig eth0 192.168.172.10

The above initialization script depends on the existence of an /etc/fstab file in the target’s root filesystem. I will not discuss the content and use of this file as it is already discussed in depth in Running Linux. Nevertheless, here’s the /etc/fstab file I use for my control module during development:

# /etc/fstab
# device      directory   type    options
#
/dev/nfs      /           nfs     defaults
none          /proc       proc    defaults

In this case, I mount the target’s root filesystem on NFS to simplify development. We will discuss filesystem types in Chapter 8 and NFS mounting in Chapter 9.

Minit is part of the miniaturized tools developed by Felix von Leitner, such as diet libc and embutils. Minit is available from http://www.fefe.de/minit/.^[8] As with the other tools distributed by Felix, Minit requires a properly configured diet libc.

Minit’s initialization procedure is a complete departure from the traditional System V init. Instead of using a /etc/inittab, for instance, Minit relies on the existence of a properly built /etc/minit directory. Firdtjof Busse wrote a description of how Minit operates at http://www.fbunet.de/minit.shtml. Firdtjof also provides pointers to example /etc/minit directories.

Unfortunately, as of Version 0.8, Minit is not yet as mature as the other tools provided by Felix. Its Makefile, for instance, is unable to deal with installing the various components in a root filesystem other than the host’s own. For the time being, Minit is not appropriate for an embedded system, but it may be worth considering sometime in the near future.