Change is as ever-present as death and taxes. Companies sometimes ship embedded devices with a set of software that fulfills requirements, only to find out that what the software does and what the customer wants don't quite match. Software passes through code reviews and test suites with flying colors but fails in the field. The new industrial designer likes rounded corners and blue, not square corners and earth tones. A competitor has released a similar device, getting rave reviews for features your product managers said nobody would ever want. No matter what the reason, the software that goes on the device as it leaves the factory will need some sort of update. The question isn't about the need but rather how to fulfill that need.
One way to update devices is to have them returned to your customer service department, where a technician opens the device, connects a serial cable, and follows instructions to load a new kernel and root file system on the board. This is the least risky approach, but it's easily the most expensive and, as I'm sure you've heard from your management, not "scalable."
This chapter looks at the process of updating devices after they've been shipped to customers or creating highly-automated update systems that your in-house customer service group or partners can use. Field updates need not be complex; you can create a reasonable solution with a few straightforward shell scripts;—no major surgery is necessary. However, if the need arises for a more complex solution, the same tools used to update the packages on your Linux desktop can be used on devices with modest resources, granting you nearly endless flexibility.
Updating the root file system is the first topic, because you can use many of the concepts and practices to update the kernel as well. In many cases, the ability to update the root file system fulfills the requirements for field updates, and the kernel itself remains unchanged over the life of the product.
The root file system on an embedded device is frequently stored on a flash device. Even though that device can be written to, the actual file system is read-only, meaning the entire root file system must be replaced. The "Basic Strategies" section describes the different ways you can update a root file system; which strategy fits best for your application depends on the device's resources, the frequency of updates, and the complexity of the software deployed on the device. Some devices contain a root file system that's identical for each device shipped, whereas others have a mix of software installed depending on the configuration of the device. There is no "right" solution, just the one that best fits how the device is used by your customers and their expectations.
You can follow a few basic strategies to update the programs on a root file system. No matter what approach you take, because you're updating an embedded system that you control, the update strategy isn't as difficult as a desktop system; but it does present challenges in that the device is likely resource constrained and may not have the most complex user interface. For example, the user can't be prompted for information if the update process can't figure out what to do. Each of the strategies is listed here and then each described in detail later in the chapter.
All of the strategies require a good communication mechanism to fetch the new binaries and enough data to for temporary storage, if just to have a temporary root file system that can start in case the root file system update system fails:
Forklift upgrade: This means the complete replacement of the old root file system with something new. On an embedded system, you replace the entire root file system with a new flash image. This is a hold-over from the days when the software in telco equipment could only be upgraded by replacing the existing equipment with new.
Parallel systems: This approach works by having two kernels and two root file systems. One root file system and kernel pair is the production system, and the other is the staging system for upgrades. You first upgrade the staging system; when the staging system is considered ready, the boot loader is changed to point at the staging system, and the board is rebooted. Obviously, this works best where the device has sufficient resources to store a second root file system and kernel.
Build your own: In this option, you distribute the software changes as a tar or cpio archive with a script or other software that then updates the system. For an embedded system with little to no variation of the software on the target, this is a very practical choice.
Package manager: No doubt you're familiar with the concept of a package manager and have used one, like RPM and Debian packages. Both of these systems are geared toward desktop systems and have facilities not only to distribute software but also to build the software from source. Embedded systems also use a lighter-weight packing system, IPKG; like many embedded tools, it favors simplicity over features.
In all cases, you need a way to handle failures. If the update process fails, the device needs to know how to go back to some state where it can attempt the upgrade process again, or revert to the original state. When the device reverts, the user needs to know that a failure happened so that they can contact technical support or attempt the upgrade process again. Of course, knowing about a failure means that upgrade process needs some software that decides whether the update process worked in the first place, not just that it failed. The sections for each of the update types cover these areas with solutions appropriate to the approach.
Going this route means making a new image of the root file system that is burned on the flash storage. For a read-only file system, a forklift upgrade is a necessity because the file system drivers can't write data: that is one way they manage to economize on space. Most of the work involves figuring out how to get the new root file system to the device and where to store it before copying to the target. In order to forklift-upgrade a root file system, you need at least the following:
MTD utilities: The flash device on the board can't be written to without first performing an erase process. In addition, boards with NAND memory need a special tool when writing, because the standard /
dev/mtdblockXdevice driver doesn't handle bad blocks properly.Extra storage space and communications link: In order to get your shiny new root file system on the board, there must be some way of accessing an external storage device, or additional storage must be on the board to hold the root file system image before it's copied to the target. It isn't recommended that you copy the new root file system image from the network or by some other means and then directly overwrite the original root file system. There are too many opportunities for error, and recovering can become difficult if not impossible. However, this is an acceptable route if the environment is controlled enough; for example, if the upgrade process is performed by a technician rather than by a consumer, it works in the majority of cases. In the example, the device has enough memory to store a copy of the new RFS in RAM along with the existing root file system, which is copied into RAM as a backup in case the upgrade goes horribly wrong.
Tools on the OS: You need some additional tools to perform the copy process and ensure that it worked. These tools are nothing special; in fact, they're what you find on a desktop system. You just need to make sure these tools are on the target system.
If this is the route you want to use to upgrade your device, you can make it easier by adjusting the software configuration of the device in order to make it simpler to upgrade in the first place. The most practical change is to store your application in a segment of flash that is separate from the core operating system so that you can upgrade the application independently from the OS. This process involves a little more work up front, but it makes the remaining process much easier and failsafe because a possible failure to upgrade the file system containing the application won't disturb the rest of the operating system. This approach also gives you the greatest ability to accommodate failures.
Note
If you're planning on the forklift upgrade, the best practice is to create a separate flash partition for your application programs. This is the only partition that is modified during the upgrade process.
The tradeoff is that the rest of the operating system is off limits when you're performing an upgrade. You can mitigate this issue by having first in the path some directories in the application directory that can be used to store upgraded system utilities and also by using the LD_LIBRARY_PATH environment variable to control what shared libraries are used. There are also programs and tools that normally appear in the root file system under /bin or /sbin that are subject to frequent change. They're placed in the application file system and appear in the path first. There are no generic criteria for selecting applications for this process; you'll know which application to choose by understanding your application in detail.
This process requires some special utilities that probably aren't installed on the target system: MTD utilities. This set of programs is maintained by the developers who brought you the JFFS2 file system and can be used to work with flash memory, even if it isn't using JFFS2. In order to use these utilities, you need to fetch them using Git and cross-compile them:
$ git clone git://git.infradead.org/mtd-utils.git
$ make CC=arm-linux-gcc
LDFLAGS="-static"
WITHOUT_XATTR=1After these are compiled, they need to be available on the board, because they're used during the upgrade process. These utilities were built with static linking because they may be placed in a file system that doesn't have a C library available to use for dynamic linking. WITHOUT_XATTR disables extended attribute handling for supporting SELinux, but this has an effect only when you're building the mkfs.jffs utility.
This is an example process for performing an upgrade of a read-only root file system. It has some components cooperating to ensure that the upgrade process worked and that the embedded system has a reasonable chance of booting. It works by mounting a small file system with the necessary tools as root, and then upgrading the flash file system with the new image. This example assumes the device has enough RAM memory to store the file system before it's copied into the flash partition:
#!/bin/sh # create the files you need FLASH_DEVICE=/dev/mtd2 FILELIST="image md5" APP_NAME="your_app" APP_MOUNT_POINT=/app FIFO=/tmp/upgrade mkdir /tmp/newrfs mount -f tmpfs -o size=xxK nodev /tmp/newrfs
This is a simple example that gets a fixed list of files from an FTP server. Because you're replacing the entire file system, which contains a group of files, this is a reasonable approach. If you want a more flexible system, change this script to download a file that then contains a list of files to download. At that rate, why stop at a list of files to download? You can download a script and execute it on the machine, although that is advisable only for systems in a controlled setting, because that leaves the device open for all sorts of exploits. Coding either of these is simple and is left as an exercise for you:
# fetch from the target the necessary files
for f in $FILELIST; do
if ! ftpget -u username -p secret
hostname /tmp/newrfs/$f /remote/path/$f ; then
echo "failed to download file $f"exit 1 done
Here, you check that the downloaded files have the expected content:
# check that the file is what you wanted
if ! md5sum -c md5 > /dev/null ; then
echo "md5 sum did not match"
exit 2
fi;Note
Recall that a FIFO (First In, First Out) is a special file that allows you to pass data between processes. In this case, the data is a side effect, because you're using the FIFO to control the execution flow of the processes.
The little trick here is that the system knows it's starting to upgrade. The target application is killed, and the init process should restart the application when it dies; however, you don't want to restart while upgrading, so you can use something simple like a FIFO to have the script or program that launches the application wait until the upgrade is over. With a FIFO, the program that does the read blocks until data appears in the input, so this is acting like a semaphore for your upgrade process:
mkfifo $FIFO killall $APP_NAME umount $APP_MOUNT_POINT
Now you have all the files necessary for the upgrade. You just need to switch out the root file system and restart the application by writing something into the FIFO:
mtderase $FLASH_DEVICE
if ! mtdcopy /tmp/newrfs $FLASH_DEVICE ; then
echo "problem copying new rfs"
## notice, that you did not write to the fifo
exit 3
fi
mount -t cramfs $FLASH_DEVICE /app
umount /tmp/newrfs
echo go > $FIFO
rm $FIFOThe script that runs the application needs to wait on the end of the FIFO, which can be done by attempting to read a line from the file:
if [ -f $FIFO ] ; then
read l < $FIFO
# the environment variable l will have "go" from the prior codefi
The key to this script is that is only updates a small part of the system in such a way that the core operating system bits keep running. Because the application code is segmented from the rest of the operating system, the code used to start the application can perform additional checks to ensure that the flash partition containing the application is in a proper state and goes into diagnostic mode or attempt to restart the upgrade process in case it hasn't worked correctly on the first attempt.
Parallel systems work somewhat like forklift upgrades, the difference being that the changes happen to a file system other than the root file system currently in use. This has several advantages over the forklift upgrade:
Safer: No matter how badly the upgrade process goes, all the changes are made to the staging system, which isn't the one being used by the system. If the reboot into the new system fails, you have a fallback, working root file system and kernel. The worst possible case is that the machine needs to download a complete root file system, which means the update will take longer.
Better diagnostics: Because the system being upgraded is a matter of updating a second partition, the software performing the upgrade can easily keep track of the process because it still has a complete, working file system.
Less disruptive: This is an oft-overlooked benefit: because the staging root file system isn't in use, the update process can be done a little at a time without causing disruptions. For the forklift upgrade, the system needs to be stopped while performing the upgrade; but in this case, the system can keep running while the update occurs.
However, this approach requires enough space to store three copies of the changes: the production root file system, the staging root file system, and the updates stored in the production file system before they're copied to the staging root file system. The production copy of the root file system doesn't need all the data for the update, because the changes to the staging root file system can happen in smaller chunks to conserve space.
In this example, you perform an update of the system by downloading a root file system and putting it into a flash partition. After the root file system has been written, the boot loader's environment variables are changed so that the next time the system boots, it uses the new root file system. The boot loader in the example is U-Boot, because it has tools that let you easily change the variables in flash from Linux. To do this, you configure the environment for the Atmel 9263 development board and build the environment update tools:
$ cd <u-boot sources>
$ make at91sam9263ek_config
$ make envThe last step built the tool ./tools/env/fw_printenv. You can use this tool to print the state of the environment and write to the environment when the filename is fw_setenv; to accomplish this, create a symlink to the file with the following command:
$ cd tools/env $ ln -s ./fw_printenv fw_setenv
When fw_printenv is copied to the target machine, you need to also copy or re-create the symlink This program uses the flash layout stored with the configuration for the board to know what memory to change in flash. If you change the flash layout of the board, you need to recompile this program as well; otherwise, it will write to the wrong areas of flash.
Now you do the work on the target system. You fetch the root file system from the update server and store it in the current root file system:
$ wget http://yourdomain.com/path/to/new/rootfilesystem/newrfs.
The U-Boot boot loader allows environment variables to contain references to other environment variables that are expanded when the board boots. Because you may change the root file system used to boot the device, you create a kernel command line that looks like the following in your U-Boot environment:
UBOOT> setenv bootargs 'console=ttyS1 rootfstype=jffs2 root=${rootdev}'The environment variable ${rootdev} is set to the production root file system in the boot loader's environment. To change the root file system after the new root file system has been written to the flash device, you run the fw_setenv command. For example, a script to accomplish this looks like the following:
FLASH_DEVICE=/dev/mtd4
mtderase $FLASH_DEVICE
if mtdcopy /tmp/newrfs $FLASH_DEVICE ; then
fw_setenv rootdev $FLASH_DEVICE
fiThe next time the system reboots, it will use the new root file system.
There still is one more option on the table for handling updates: doing it yourself. The primary benefit of this approach is also its greatest source of reliability, because you have control over everything. However, by using the high-level tools on the system for bundling and compressing files as well as fetching data from a remote host, you're relieved of most low-level coding problems. This relief comes at the cost of a solution that has a larger footprint.
All these package-management tools are wrappers around lower-level archive tools. In the case of dpkg and ipgk, the underlying tools are tar and gzip, whereas RPM uses cpio and gzip. The point isn't to lessen the value of the tools, but to point out that if you really need just the bare minimum in package management, you can use tar and zip to create a reasonable solution that doesn't have all the bells and whistles but does enough that updates get to the target computer.
The most reasonable choice would be to use tar or cpio because BusyBox has tools for working with these file formats. Depending on the size of your application and the amount of memory on the board, compression is optional.
To make this an easy and simple example, create a directory on your development host to hold the files that is sent to the target device; call this the <build dir>:
<build dir>/
lib/
libapp.so.0
libapp.so -> libapp.so.0
bin/
appfileTo create a tar file for this directory, do the following in a script file:
BUILDDIR=<build-dir> OUTPUTFILE=<output-file> tar -C $BUILDDIR tf $OUTPUTFILE *
The output file can now be sent to the target machine and uncompressed from the root directory in order to install its contents. The command looks much like the command to build the system. Using ftpget to fetch the tar file, the script looks like the following:
if ftpget -u username -p secret
hostname /tmp/newfile.tar ; then
tar -C / -xf /tmp/newfile.tar
doneAfter putting the new code on the system, you need to restart the application so the changes take effect.
As a user of a desktop system, you're familiar with package managers, because nearly every Linux distribution has its origins in some base set of packages that were compiled and turned into ISO images used to boot Linux on your desktop machine. Package-management systems have the ability to track what has been installed previously on a system. They also track information about individual packages such as the files in the package, some grouping information, and a list of other packages that must be installed in order for the software inside the package to function correctly. The information about package dependencies is paired with an online database where the package-management software can download not only a package but all the dependencies the package needs.
Using a package-management system means the device has a root file mounted as read/write, so devices using flash use JFFS2 or YAFFS2. If the device has a read-only file system, there is no way the package manager can perform the updates on the media. The entire root file system doesn't need to be read/write, only the parts where the package-management software is writing files and keeping the database.
There are several package-management systems: this section looks at two systems frequently used on desktop systems, RPM and dpkg, and another that was created just for embedded systems, ipkg. A nice thing about using a package-management system is that it generally includes the ability to create a store of packages on a remote site and have the software download new packages and any related packages in one transaction, as described earlier. For instance, if your software says it needs packages one, two, and three, and package three needs A and B, the package-management software can use the package-dependency information to download packages one, two, three, A, and B and install all of them before attempting to install your application.
RPM was created by Red Hat as a way to maintain the build and installation of the company's distribution. On an embedded system, RPM can be used to install packages on the system in the same way it does on your desktop system. Downloading packages from a remote system is done through a separate program called YUM that works with RPM packages. You can use RPM without YUM if desired.
Creating a binary RPM is a matter of making a spec file that tells RPM where to look for the files to add to the archive, the file types, and their permissions settings when installed. Many people think an RPM can only be created by compiling a source RPM, but this isn't the case. You can create an RPM by creating a simple specification file that looks for a list of files to copy to the target, regardless of how they're built. Having the ability to create a RPM binary without using RPM as your build system means you can still use RPM for package management, but you don't have to give up your current build system. Here, you see the contents of an example file that builds a binary RPM from packages that have been built elsewhere. Although RPM likes the idea of you building your application from source using RPM, that isn't always practical. This procedure creates an RPM file from a file system containing binaries built using outside of RPM:
Summary: Embedded Application Name: embedded-app Version: 1.1 Release: 001 Group: Applications/Misc URL: http://your.url Packager: Resource 1, BuildRoot: %_buildsystem License: GNU %description The application for the target system %files %defattr(-,root,root) /usr/bin/appfile /usr/lib/libapp.so.0 /usr/lib/libapp.so
In order to have this build correctly, you need to have the following ˜/.rpmmacros file:
%_buildsystem %(echo $HOME)/<build dir> %_targetcpu arm %_topdir %(echo $HOME)/<output dir>
The RPM build process reads the files from <build dir>, which mirrors the root file system of the target device, and then creates an RPM that is placed in the <output dir> directory. These directories can be anyplace on the machine; the example uses the home directory for simplicity. After creating these files, you can use RPM build to read the spec file:
$ rpmbuild -v -bb --target arm app.specfile
Building target platforms: arm
Building for target arm
Processing files: embedded-app-1.1-001
Requires(rpmlib): rpmlib(CompressedFileNames)
<= 3.0.4-1 rpmlib(PartialHardlinkSets)
<= 4.0.4-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1
Checking for unpackaged file(s):
/usr/lib/rpm/check-files /home/gene/book-code/build
Wrote: /home/gene/book-code/output/RPMS/arm/embedded-app-1.1-001.arm.rpmThis produces an RPM file ready for installation on a target machine. You can take two paths to install this RPM file: copy it to the target and use RPM to install the file, or create a store and have YUM manage copying the file and its dependencies. Let's take a look at each.
By using RPM, you can either build the full RPM binaries for your system to use the RPM utilities included with BusyBox. The BusyBox rpm utility, fitting the BusyBox project's philosophy, is a minimal implementation and doesn't include dependency checking or a database to record what has been installed on the system: it copies the files out of the RPM and deposits them in the right spots in the file system. For most embedded systems, this is perfectly acceptable.
To install the file created in the example, you copy the RPM you intend to install to the target machine, using whatever utilities are the most convenient. For this example, the files are copied to the /tmp directory using ftpget (a BusyBox utility) and installed from there.
This is the list of files to copy from the remote machine and the name of the target application. You'll use these later in the script; the macros make the code easier to read:
FILELIST="file.list md5" APP_NAME="your_app"
Here you fetch the files and put them in the /tmp file system. You copy all the files to the target machine so that if you have multiple files, you can ensure that all files are present on the host before attempting an upgrade. This script downloads a list of files and then attempts to download all the files in the list:
for f in $FILELIST; do
if ! ftpget -u username -p secret
hostname /tmp/$f /remote/path/$f ; then
echo "failed to download file $f"
exit 1
done
if ! md4sum -c /tmp/md5 > /dev/null ; then
echo "list of files not downloaded properlyfi;
while read f ; do
if ! rpm -i $f ; then
echo error installing $f
exit 4
fi
done < /tmp/file.listNow that all the files have been copied, you can delete them to free up the resources they were consuming. You do so in a separate step to give the most freedom for handling errors:
for f in $FILELIST; do
rm /tmp.$f
doneAfter you upgrade the system, you can restart the application. If you have the system configured to restart the main application if and when the program ends, the following line causes the application to restart:
killall $APP_NAME
The YUM system works with RPM files to create a way to download an RPM, along with any dependencies. After YUM has downloaded all dependencies, it then installs them on the target system.. YUM isn't a fit for all systems because it uses Python and some systems may not have the additional storage space for the files required by YUM. Using YUM means creating a repository that is placed on a machine reachable by the target board. To create the repository, you need the createrepo utility. Even if you have a Red Hat system and use YUM, this tool isn't always present. To download, build, and install, follow these steps:
$ wget http://createrepo.baseurl.org/download/createrepo-0.4.11.tar.gz $ cd createrepo-0.4.11/ $ make $ sudo make install
Creating a repository is straightforward. Consider the build of the PRM file earlier in the chapter; you can use that as your repository, located at $REPO_DIR, which is any directory where you have write access. To create the YUM repository for that directory, do the following:
$ createrepo $REPO_DIR Saving Primary metadata Saving file lists metadata Saving other metadata
Now that the repository is ready for use, you need to put the contents of REPO_DIR in a location that is accessible by your HTTP server so remote devices can locate the data with an HTP URL. This example refers to this location as REPO_URL.
The next thing to do is to build and configure YUM so it can read this repository. When you're experimenting with YUM, you can use your development machine, so long as you don't try to install, because the mismatch between the RPM's architecture and your development host will likely result in the RPM refusing to install due to the differences in architecture. In order to use YUM on the target, you first need to download and build the tool, if it isn't installed on your system already:
Note
YUM is Python-based. If you want to use YUM on your target board, Python must be present.
$ wget http://yum.baseurl.org/download/3.2/yum-3.2.24.tar.gz $ tar xzf yum-3.2.24.tar.gz $ cd yum-3.2.24.tar.gz $ make $ sudo make install DESTDIR=/path/to/board/rfs
This step ensures that the components necessary for YUM are available on the board. On the board, you need to create a configuration file so that YUM knows to use that URL when updating the system. Create this file in the /etc/yum/repos.d directory; you can give it any filename as long as it ends in .repo:
[embedded] name = EmbeddedApp baseurl = file:///tmp/output/repodata
Before you can use the YUM repository, you need to create a cache. This process must be done every time new packages are added to the cache. In this example, the repository is small, so the time to create the cache is very small:
$ sudo yum makecache embedded 100% |================| 951 B 00:00 Metadata Cache Created
The next step is to ensure that the cache contains what you expect. You can do this check on any machine that can reach the URL where the repository is stored. In this example, you use a file URL, so this test is run from the machine where the YUM repository was created:
$ yum repolist repo id repo name status embedded EmbeddedApp enabled: 1 repolist: 1
You're now ready to use this repository. You can adjust the base URL to be something under your HTTP document root or served by FTP for remote hosts. The board accessing the repository needs to know this in order to reach the YUM update service, which means the devices need an appropriately configured /etc/yum directory when building the root file system.
The Debian package system is the underlying package-management system for the Debian distribution and is used by Ubuntu, one of the more popular Debian-based distributions. The Debian package system works much like the RPM system: you create a file with the package metadata and then combine the metadata into a package that another program uses to install the software on another system. Debian has the notion of source and binary packages, and ideally you should build your binary package sources. However, not all projects can accommodate this, so packages can be built from binaries as well.
This packaging system also has the ability to build a repository of packages that can be accessed by remote clients who can request the latest version of a package along with its dependencies. The following sections look at how to create a simple package and repository.
You build a dpkg using a control file, similar to the way RPM files are built using a spec file.. Referencing the example project from the RPM, here's a Debian package control file for the same binary project:
Package: embedded-app Version: 1.1 Section: application Priority: required Architecture: arm Essential: yes Installed-Size: 1024 Maintainer: Resource 1 <[email protected]> Description: The application for the target system, now with positive electrons, just like marketing asked!
This file contains the metadata to describe the package so that the device installing it can decide whether it's the right file, say by comparing the architecture or version of the package with that of the host. The example shown is the minimal set of information to create a package; thus, you can keep a focus on the higher-level process of creating a package and using it in a repository.
In the example package, the files are built into a directory, build/, that looks like the root file system on a device. For example:
build/
lib/
libapp.so.0
libapp.so -> libapp.so.0
bin/
appfileIn order to have dpkg build this directory (using default file locations, which is the easiest route by far), you name this control file DEBIAN/control, resulting in a directory structure that looks like the following:
build/ lib/
libapp.so.0
libapp.so -> libapp.so.0
bin/
appfile
DEBIAN/
controlAfter the control file is in the correct location,[40] use the dpkg build command to build the package:
$ dpgk -b build embedded-app.deb dpkg-deb: building package `embedded-app' in `embedded-app.deb'
The file is now built and ready for use. If you want to inspect the contents, use the less command to get a quick overview:
$ less embedded-app.deb
Note
BusyBox is the Swiss army knife of embedded systems, and less is the Swiss army knife of your desktop system. Use 17.300 less to display nearly any type of file, and it does the right thing.
After embedded-app.deb is on the target system, BusyBox contains a set of applets for installing Debian packages. Getting these packages on your system and installing them works much the same way as the RPM example: instead of using RPM, you use dpkg, and the modified part of the earlier script looks like this:
while read f ; do
if ! dpkg $f ; then
echo error installing $f
exit 4
fi
done < /tmp/file.listThe rest of the script is exactly the same. In fact, you can make the command that performs the installation a macro so you can easily test both systems to see which one you prefer, although a resource-sensitive embedded engineer will never waste a few bytes that way.
You may not be keen on creating and maintaining a script that figures out what to download in order to update a system. It may be that the systems in the field have enough variations in their configuration that a simple script that downloads a list of packages and installs them isn't sufficient to update the system. The Debian analogue to YUM is apt-get (part of the Debian Advanced Packaging Tool [APT] collection of utilities), and you can use it to create a pool of packages with dependency information so the user can fetch a package and all of its dependents in one operation.
Creating an APT repository involves creating a set of directories to hold the packages and using a tool to create the supporting files for the APT tools to use to access the repositories. To get the necessary tools, fetch the package reprepro:
$ sudo apt-get install reprepro
Next, create the directories for the repository. In this example, you create them in the /tmp directory, but any directory where you have write privileges can be used:
mkdir -p /tmp/apt/conf mkdir -p /tmp/apt/incomming
Next, the repository needs a configuration file so that when clients access the repository, they know what types of packages they can expect to find there. This is an example configuration file appropriate for the files you adding for the embedded device. This file needs to be stored as /tmp/apt/conf/distributions:
Origin: Embedded Company Label: Embedded Device Software Suite: stable Codename: edasich Version: 1.0 Architectures: arm Components: main Description: This is an embedded code repository for an embedded device
After this file has been created, you can add the embedded-app file to the repository with this command:
$ reprepro -b /tmp/apt includedeb edasich
<path to package>/embedded-app.deb
Exporting indices...You can verify that the package is located inside the repository by doing the following:
$ reprepro -b /tmp/apt list edasich edasich|main|arm: embedded-app 1.1
Now that the repository is ready, it's a matter of getting the tools ready to run on the target board. The apt tools are written in C++, so they don't need additional tools like Python on the target to work (that is, if you're not already using Python). In order to get the sources, use the apt-get source command:
$ apt-get source apt dpkg-source: extracting apt in apt-0.7.20.2ubuntu6 dpkg-source: info: unpacking apt_0.7.20.2ubuntu6.tar.gz
The apt project uses the autoconf tools to build, so creating them for your target board should be a familiar process. For the compilation process to work correctly, you also need the curl library for your toolchain; if it isn't present, download it and build it first:
$ wget http://curl.netmirror.org/download/curl-7.17.0.tar.gz
$ CC=arm-linux-gcc ./configure
--host=arm-linux-gnu
--prefix=<toolchain sysroot>
--disable-ldap --disable-ldaps
$ make
$ make installCurl is a tool that allows high-level processing of URLs. This configuration command drops support for the LDAP and LDAPS authentication schemes, because these aren't present in most toolchains and require additional steps to build. Unless you plan to use LDAP authentication, you can leave these out and save some space as well. After they're installed, you can build the apt library like so:
$ cd apt-0.7.20.2ubuntu6
$ CC=arm-linux-gcc CXX=arm-linux-g++
CPPFLAGS="-I <path to your toolchain sysroot>"
./configure --host=arm-linux-gnu
-prefix=<path to your rfs>
$ make
$ make installWhen it's installed on your target machine, you need to tell apt-get to use your repository when doing an update. You do so by putting an entry in the /etc/apt/sources.list file on the target machine. Using what you've created so far, the entry is
deb file:///tmp/apt edasich
The board can now be read from the repository and download packages when requested. This line says to check the repository at file:///tmp/apt, which contains packages for the edasich distribution, which is the label you've assigned to the set of packages. This file can have several lines and can use HTTP and FTP URLs. The example uses a file URL to make it as simple as possible.
The Itsy Package Manager (ipkg) is part of the Open Handhelds project and is designed for low-resource machines. Originally, this project was a collection of shell scripts that unpacked a tar file. Not that this was a bad thing; rather, it was a great, low-resource way to manage updates. The code has since been rewritten in C with a mind toward doing the job as simply as possible while adding some of the features found in the other, bigger package-management tools. If you're short on resources (both RAM and flash), but you still require a more structured way of interacting with packages on the system, ipkg is a great solution.
The tools for ipkg aren't available in a standard Linux desktop distribution, so you need to download some sources and compile them for use on the development machine. These sources are compiled later for the target, but having them handy on the host machine is very helpful. The Open Handhelds site has an area where you can download snapshots or get the sources from CVS; this example gets the sources from a snapshot file:
$ wget http://www.handhelds.org/download/packages/ipkg/ipkg-0.99.163.tar.gz $ tar zxf ipkg-0.99.163.tar.gz $ ./configure $ make $ sudo make install
The snapshot I used didn't copy the library file used by ipkg to the /usr/lib directory, so the program wouldn't work after installation. To get around this problem, copy it yourself using these steps:
$ sudo cp ./.libs/libipkg.so /usr/lib $ sudo ln /usr/lib/libipkg.so /usr/lib/libipkg.so.0
You also need the utilities for creating ipkg packages. These are stored on www.handhelds.org, but in this example they're downloaded from a mirror because the link on www.handhelds.org sometimes doesn't work.
$ wget ftp://ftp.cerias.purdue.edu/pub/os/gentoo/distfiles/ipkg-utils-1.7.tar.gz
These files are a combination of shell and Python scripts, so no compilation is necessary. To make accessing these files easier, you can copy them into /usr/bin or append to your search path the directory where there were unpacked. Because these files won't be on the target, you don't take the same level of care to make them small.
This process works much like the Debian process for creating a package: you create a special file in a well-known directory, in this case CONTROL/control. The CONTROL directory is a peer of other directories that is included in the build of the ipkg, similar to the way Debian creates packages. Going back to the example, an example CONTROL/control file looks like the following:
Package: embedded-app Priority: required Version: 1.1 Section: base Architecture: arm Source: none
Maintainer: Valued Resource 1<[email protected]> Description: Primary application. Comment remarking about the new electrons added at the behest of marketing.
After the file is created, use the ipkg-build command to create a package file:
$ ipkg-build build-ipkg
*** Removing the following files: ./CONTROL/control˜
Packaged contents of build-ipkg into
<parent dir>/embedded-app_1.1_arm.ipkThe ipkg system also supports fetching packages from a remote location, called a feed. Feeds are very easy to create: make a directory in a place accessible via FTP or HTTP, copy the ipkgs you want to make available, and use a tool to prepare the feed. This section goes through the process one step at a time:
You need to create directories in some place that can be reached by a web server (it can be any web server type; there's no requirement for Apache). In this example, the web server's document root is
DOCROOTto make the directory's location clear. This directory can be any directory; the one chosen was named to make the example clear:mkdir $DOCROOT/ipkg-feed
Copy in package files. Any ipkg files that need to be part of this feed must be copied into this directory. Using the example package:
cp <
package dir>/ipkg-feed/embedded-app_1.1_arm.ipk $DOCROOT/ipkg-feedRun the indexer to create the index file that the ipkg file uses to see what's available on the target machine. This tool is designed to use Python 2.2. If it isn't present on your system, the script produces a "no such file or directory" error message. Because most systems use Python 2.4 or newer, you need to modify the first line of the script to use the Python on your system. The
ipkg-make-indexscript contains this as the first line:#!/usr/bin/python
However, on newer systems, you should use the Python interpreter on the path, so the first line of the file read
#!/usr/bin/env python
After the script has been changed to run on your system, invoke it by doing the following:
$ cd $DOCROOT/ipkg-feed $ ipkg-make-index -v . > Packages Reading in all the package info from .
Reading info for package ./embedded-app_1.1_arm.ipk Generating Packages file Writing info for package None MD5Sum: 19c92f2ebc4a942b4e5c47d3c528c44f Size: 824 Filename: embedded-app_1.1_arm.ipk
The system is now ready for a client to connect and download files. To test your work, you can also use the ipkg tools created for the host to query the repository. The next step is getting ipkg running on the target computer; to do this, you need to cross-compile the ipkg tools and put them on the root file system. Using the sources unpacked earlier in this section, you can do the following from the directory where the ipkg files were unpacked:
$ make clean
$ CC=arm-linux-gcc ./configure
--host=arm-linux-gnu
--prefix=<target board rfs>
$ make ; make installThe board needs a configuration file so it knows where to look for ipkg files. This file is called /etc/ipkg.conf, and for this example it contains the following:
src edasich http://<some host>/ipkg-feed dest root /
This configuration file says that files belong to the edasich group can be found in the http://<some host>/ipkg feed directory and that these files should be installed in the root directory. A file can have multiple feeds; the field after the src command is the label used by ipkg to keep track of what feeds belong to what section.
Some users opt for a root file system that uses the initramfs file system. For review, this file system is stored alongside the kernel as a cpio archive that is uncompressed into the file system cache so it works like a RAM disk. This file system is read/write, but all changes are lost when the system reboots. If you have initramfs as your root file system, you need to replace the entire kernel as a way of making updates.
The kernel is the other part of the system that you can update in the field. Updating the kernel is often approached with trepidation, because the problems are often viewed as catastrophic. In fact, a failure updating the kernel leads to nearly the same level of nonfunctionality as a botched update to the root file system and results in a kernel panic because the main application won't run.
Even though kernel updates are no more risky or problematic than updating the root file system, the need to update the kernel is much less frequent. On a desktop system, a user may update the kernel to take advantage of new peripherals or get performance updates or patches that affect security. For an embedded system, with fixed hardware and very controlled communication, these needs typically don't arise due to the nature of the device. However, with more consumer electronic devices that are more open, the need for security updates is more prevalent.
The strategies for updating the kernel are much like those for the root file system. The strategy you eventually take depends on factors around the application, which are discussed with each of the strategies. These choices aren't mutually exclusive; after reading them, you may pick both options. Whatever option you select, you need to be careful with your implementation because if the kernel doesn't update properly, you can cripple your system:
Using kernel modules lends a great amount of flexibility to the kernel. Kernel modules are essentially files containing relocatable object code linked into the kernel at runtime, much as a program uses a shared object. During development of the kernel, if it becomes apparent that some modules are subject to changes after the product is shipped, these should be put on the root file system as modules.
How does a module make itself apparent? Here are some factors to weigh when deciding if you should create a kernel module:
Problematic during development: Modules that caused grief when developed will likely continue being troublesome when released.
Works with new hardware: Code that enables new hardware is another good candidate, because the documentation for the hardware isn't always clear or doesn't fully describe how the deice works. Another indicator is hardware that is changed close to the shipping date, because you don't have as much time to test it as desired.
Frequently changing hardware: In this case, the supported hardware on the device is sourced from several different and subject to change over the production run. LCD displays are a good example, because this component is subject to price fluctuations and the buyer is constantly looking for a good deal.
Expected upgrades: Any kernel functionality that is likely to be upgraded while the kernel remains running is a candidate, because these modules can be unloaded and updated and then new versions can be loaded.
If you can afford the time and space necessary to load kernel modules during the boot process, then use kernel modules, because they make updating the kernel a much easier process.
This is like the update for the root file system where the contents of the flash partition containing the kernel are erased and updated with a new kernel image. The methodology for this approach is nearly the same as replacing the root file system. For resource-constrained devices, this is the most practical approach. Even if the system uses kernel modules, you still may need to replace the core kernel, necessitating a forklift update approach.
The following sections look at examples for implementing a field update for the kernel.
The easiest way to conceptualize this approach is to think of kernel modules as being no different than updating any other sort of file that can be updated on a root file system. This means a package manager can be used to replace the kernel modules on the root file system. The system reboots to take advantage of the change; or, if the device needs to stay operational, the module can be unloaded and then the new version loaded.
There is no requirement that kernels use the modprobe utility to load and manage modules, but it certainly makes life easier; and for systems that use modules, this utility pays for its overhead through added flexibility. However, not using modprobe gives you additional control over how modules are loaded. This section looks at both approaches, because each has its advantages and liabilities.
No matter what approach you take, it's worth mentioning that a writable file system is required in order to store the updated modules and possibly update some configuration files in the root file system. The entire root file system doesn't need to be writable, just the mount point containing the kernel modules subject to update.
Modprobe exists to make managing modules easier, by using a configuration file to contain information about the module and formalizing where the modules are stored on the system. For the purposes of this section, you stick to the subset of capabilities offered by the BusyBox implementation of modprobe.
The modprobe utility use a file /etc/modprobe.conf to hold the modules that can be loaded on the system. A typical modprobe.conf file looks like the following:
alias network eepro1000 alias gpio-input gpio-keys-2 options eepro1000 irq=4
Each line that starts with alias allows a module stored on the disk to be loaded through its alias's name, as in the example:
modprobe network
This results in modprobe attempting to load the module eepro1000.
The options line lets you define default options that are passed into the kernel module when it loads. Modprobe expects modules to be located in the directory /lib/modules/`uname -r`, where uname -r is the kernel release string; when surrounded by backtick ` characters, it tells the shell to execute the command between the backticks and substitutes the result. The backtick syntax only works from a command shell prompt. On a typical system, this expression evaluates to /lib/modules/2.6.28-11-generic, but with the kernel version matching what is running on the system.
The indirection that modprobe allows through the configuration file lets you download kernel modules with a revision number associated with the name, but still have consistent module names in other scripts in the system. For example, on the file system, before an update, these files exist in the root file system:
/lib/modules/2.6.11-yourkernel/
somemodule-rev1.ko=After an update, the following are in the file system:
/lib/modules/2.6.11-yourkernel/
somemodule-rev1.ko
somemodule-rev2.koBefore performing an update, the modprobe conf file contains the following:
alias somemodule somemodule-rev1.ko
Post update, the modprobe.conf file contains the following:
alias somemodule somemodule-rev2.ko
The script performing the update is responsible for ensuring that the module somemodule-rev2.ko is the correct file before making the change. By keeping the old module file in the system, if the somemodule-rev2.ko file doesn't load properly, the prior revision is still available. A simple way to quickly test if a module is present and conforms to the format of a module, and is thus likely to load, is to use the modinfo utility. For example, if you have a list of modules to check, the could looks something like this:
MODULE_LIST="module1 module2"
ERRORS=0
for m in $MODULE_LIST ; do
if ! modinfo $m > /dev/null ; then
let "ERRORS = $ERRORS + 1"
fi
done
if $ERRORS == 0 ; then
update_modules_conf
fi;Notice that this section doesn't include information regarding how to get the modules on the target system. Because the modules are no different from any other file on the file system, they can be put into a Debian, ipkg, or RPM for delivery to the target machine. Using a package manager also allows you to package the script for maintaining the modprobe configuration file as part of the package itself; encapsulating that information with the modules removes a step from the deployment process and makes the update process easier to test and validate.
If you have complex module-loading requirements, such as loading some modules conditionally or from nonstandard locations, the modprobe utility may not be sufficient for your application. Module loading without modprobe means using the insmod utility; insmod is much lower level than modprobe—it just loads the requested module into the kernel, passing along any parameters.
Systems that use insmod also need to keep track of module load order. That can be done with a simple script that loads the modules in the right order instead of using the module dependency information modprobe uses to do the same job. For example, a common approach is to put the modules to be loaded in a directory with a small shell script that loads then with the right parameters in the right order. For example:
/var/modules-rev1
somemodule.ko
anothermodule.ko
loadmodules
/var/modules-rev2
somemodule.ko
anothermodule.ko
athirdmodule.ko
loadmodules
/var/modules-current -> /var/modules-rev2In this example, the loadmodules script (for the first directory) contains a few lines of code to load the modules in the directory:[41]
insmod somemodule insmod anothermodule
In this example, when a new set of modules is added to the system, they go into a separate directory so they don't overwrite the existing set of modules, in case the modules were in a package that was corrupted.
Notice in the example that the directory tree contains a symlink to the current set of modules to load. This symlink makes it so the script that loads the modules can be the same no matter what set is being loaded. When the new modules have been checked and are considered in a good state, the script changes the symlink to point to the correct directory.
The forklift upgrade for the kernel is also very similar to the forklift upgrade of the root file system, but with one important difference: once the kernel is loaded into memory, it doesn't have a dependency on the partition like a file system. No dependencies mean that the upgrade routine doesn't need to do anything to make sure that the data in the partition will remain untouched during the update process. Much of the code for updating the root file system forklift-style was ensuring that the update had exclusive accesses to the partition.
Here's an example script that performs an update for the kernel:
KERNEL_FILE=uimage.new FLASH_DEVICE=/dev/mtd1
mtderase $FLASH_DEVICE
if [ -f $KERNEL_FILE && -f $FLASH_DEVICE ] ; then
mtdcopy $KERNEL_FILE $FLASH_DEVICE
fiThe next logical step in this system is to use a package-management tool to contain a package with this new kernel and the script that was run after the files were installed on the target. The mechanics for this step are slightly different for each package manager, but all support the concept of a post-installation script. The advantage of using a package manager is that it ensures that the package downloads and unpacks successfully before attempting to run any installation scripts.
After the copy process finishes, the system can be reset, or you can wait for the user to reset the system so the new kernel loads.
Just as change is ever-present as death and taxes, there is another omnipresent, universal constant: screw-ups. The sections in this chapter include a fair amount of code and practice to make sure that when the update occurs, the data written to the device is good in a formal sense; that is, it may not perform to specifications, but it won't crash the device. Thus, the chances of a failure are greatly reduced; but no matter what happens, there's the possibility that the upgrade process will fail.
What do you do in such a case? You can take several approaches.
This seems like a non-solution, but telling the user that the device is in a compromised mode and requires service is much better than nothing at all. For a very minimal machine, a LED that can be turned on by the boot loader and then off by the kernel during the boot process gives the user some idea that the device isn't functioning normally. A device with a screen offers much more bandwidth for communicating failure and how the user should get help.
This is a great use for the initramfs file system. The initial file system has the primary job of ensuring that the last boot or system update did not end in failure. Failure can be detected by the file system not mounting or a file being present on the device indicating that the boot process didn't complete. The job of the failsafe root file system is to determine what steps are necessary to get the device back in working order, typically by downloading and installing on the board a failsafe root file system that can then run through the process again.
The nice feature about a failsafe root file system is that you have a great degree of control over how the failure should be handled. In some cases, the device can retry the update process; in others, the device can present a message telling the user the nature of the problem and direct them to technical support.
In this case, the board contains a kernel (with a small root file system) that is never changed. This failsafe kernel is the first thing to boot and is responsible for checking that the system booted properly the last power cycle. This communication is accomplished by writing data to an agreed-upon register or memory location, after the fail-safe kernel has determined that the system has booted improperly.
The GRUB boot loader has fail-over support as a feature. Consider the following GRUB configuration file:
fallback 1
title "Primary Kernel"
root (hd0,0)
kernel /primary.kernel
title "Fallback Kernel"
root (hd1,0)
kernel /fallback.kernelThe fallback kernel is Fallback Kernel, as indicated by the fallback 1 statement; that kernel is booted if the first kernel fails to boot or panics. Recall that GRUB numbers boot entries starting at zero, from the top of the file to that bottom.
In order to have a system that's as survivable as possible, the fallback kernel has its own root file system located at (hd1,0), The root file system used by the fallback kernel should never be updated in the field so that it can't be corrupted and therefore is unavailable to boot the system. As an extra measure of safety, the fallback kernel should also include a small initramfs so it can perform minimal tasks in case the fallback root file system can't be mounted.
After the primary kernel boots, another command needs to be issued. Thus, the boot loader knows to use the primary kernel for the next boot cycle. This command must be run each time the primary kernel boots successfully:
$ grub-set-default
If you don't issue this command, GRUB doesn't know if the boot was successful; and the next time the system restarts, GRUB will try to use the fallback kernel.
Keeping the software on the device doesn't get the attention it deserves because for many workers in the embedded field, the devices lacked the ability to update themselves—the software that was put on the device as part of the production software load was the software that ran on the device. Linux, with its excellent communication capabilities paired with the tools to perform an update, gives you the ability to update nearly any device. Linux offers choices for updating systems that span the complete resource spectrum, so you have a great deal of freedom to pick the update system that works best for your device and users. Some of the update approaches work best on a system with a high-bandwidth connection whereas others are perfectly suited for low-resource devices that communicate over a serial line.
All the update choices mean you also need to think about software updates sooner in the development process. That way, you can put the right tools on the board and test the update features before they leave the factory, ensuring that your users will have trouble-free updates.
[40] Unlike the spec file, which resides somewhere outside of the directory structure of the files going into the package, the dpkg file resides along with the files. As somebody who learned RPM first, I found this a little disquieting.
[41] This script could be much more complex: using a numerical prefix to the filename to control load order, for example. However, because this script becomes more complex, it really reimplements modprobe; chances are, if you're an engineer taking this approach, the desire is to make things as simple as possible.