Cross-Compiling
Embedded computers often lack the necessary resources for developing and compiling software. The Raspberry Pi is rather special in this regard since it already includes the gcc compiler and the needed linking tools (under Raspbian Linux). But while the code can be developed and built on the Raspberry Pi, it may not always be the most suitable place for software development. One reason is the lower performance of the SD card.
To compile native code for the Raspberry Pi, you need a compiler and linker that knows how to generate ARM binary executables. Yet it must run on a host with a different architecture (for example, Mac OS X). Hence the reason it is called a cross-compiler. The cross-compiler will take your source code on the local (build) platform and generate ARM binary executables, to be installed on your target Pi.
There are prebuilt cross-compile tools available, including the Raspberry Pi Foundation’s own tools (git://github.com/raspberrypi/tools.git), but these can be problematic for some versions of Linux. Running the cross-compiler on a different Linux release than it was built for may cause the software to complain about missing or incompatible shared libraries. But if you find that you can use a prebuilt release, it will save considerable time.
In this chapter, you’ll walk through how to build your own cross-compiler. This may permit you to get the job done using your existing Linux release.
Terminology
Let’s first cover some terminology used in this chapter:
- build: Also called the local platform, this is the platform that you perform the compiling on (for example, Mac OS X).
- target: The destination platform, which is the Raspberry Pi (ARM) in this chapter.
Let’s now consider some of the cross-compiling issues before you take the plunge. There are two main problem areas in cross-compiling:
- All C/C++ include files and libraries for the Raspberry Pi (ARM) must be available on your build platform.
- The cross-compiler and related tools must generate code suitable for the target platform.
So before you decide that you want to build a cross-compiler environment, are you prepared to
- Provide all matching C/C++ header files from the ARM platform?
- Provide all ARM libraries needed, including libraries for third-party products like sqlite3 that you intend to link with?
- Provide sufficient disk space for the cross-compiler and tools?
The crosstool-NG software will mitigate some of these issues. For example, the correct Linux headers are chosen by the configuration step shown later in this chapter.
Disk space solves many issues by holding a copy of your Raspberry Pi’s root file system on your build platform. Simple programs won’t require this (for example, a Hello World). But software linking to libraries may require this. Even if you’re strapped for disk space, you may be able to mount the Raspbian SD card on the build platform, thus gaining access to the Raspberry Pi’s root file system.
The procedure used for building a cross-compiler environment is somewhat complex and fragile. Using the crosstool-NG software simplifies things considerably. Despite this advantage, it is best to stick with proven cross-compiler platforms and configurations.
You might be tempted to say, “The source code is open, and so it should work on just about any operating system.” (You might even say, “I’ll fix the problems myself.”) The reality is not quite so simple. I use the Mac Ports collection (www.macports.org) for a number of things and quickly discovered the limitations of building a crosstool-NG on Mac OS X. For example, I found that objcopy was not supported when ./configure was run for the cross-compiler. Unless you are willing to spend time on Internet forums and wait for answers, I suggest you take a more pragmatic approach—build your cross-compiling environment on a recent and stable Ubuntu or Debian environment.
This chapter uses Ubuntu 14.04 LTS hosted in VirtualBox 4.3.12 (www.virtualbox.org) on a Mac OS X Mavericks MacBook Pro, running an Intel i7 processor. Current versions of Ubuntu are recommended. Ubuntu 12.10 was the version tested and used by the process documented at this link:
www.kitware.com/blog/home/post/426
Host, Guest, Build, and Target
At this point, a short note is in order because these terms can get confusing, especially for those performing this for the first time. Let’s list the environment terms, which will be referred to throughout the remainder of this chapter:
So many environments! The terms host and guest environments enter the picture when you are using a virtual machine like VirtualBox. VirtualBox is used to host” another operating system on top of the one you are using. For example, you might be running Mac OS X on your laptop. In this example, the OS X environment hosts Ubuntu Linux within VirtualBox. The Ubuntu Linux operating system is thus referred to as the guest operating system.
The term build (or local) environment refers to the Linux environment that is executing the cross-compiler and tools. These Linux tools produce or manipulate code for the target environment (your Raspberry Pi’s ARM CPU).
Platform Limitations
Many people today are using 64-bit platforms similar to my own MacBook Pro, with an Intel i7 processor. This may present a problem if you want to build a cross-compiler for the Raspberry Pi, which is a 32-bit platform. Many people building a cross-compiler on a 64-bit platform have encountered software problems building for a 32-bit platform.
For this reason, if you are using a 64-bit platform, you’ll probably want to choose a VirtualBox solution. This will allow you to run a 32-bit operating system to host the cross-compiler. On the other hand, if you are already running a 32-bit operating system, creating a native cross-compiler for the Pi is a real possibility.
Without VirtualBox (Native)
If you are already using a Linux development environment like Debian or Ubuntu, the term host is equivalent to the build (or local) environment. The host and guest environments are likewise equivalent, though it is probably more correct to say there is no guest operating system in this scenario. This simpler scenario leaves us with just two environments:
- Host/guest/build: Native environment running the cross-compiler tools
- Target: The destination execution environment (Raspberry Pi)
Using VirtualBox (Ubuntu/Linux)
If you do not have a suitable Linux environment, one can be hosted on the platform you have. You can host Linux from Windows, Mac OS X, Solaris, or another distribution of Linux. VirtualBox can be downloaded from the following:
www.virtualbox.org
When VirtualBox is used, the host environment is the environment that is running VirtualBox (for example, Mac OS X). The guest operating system will be Ubuntu (recommended) or Debian. This leaves us with three environments in total:
- Host: Or native, running VirtualBox (for example, Windows)
- Guest/build: Ubuntu/Debian development environment within VirtualBox
- Target: The destination execution environment (your Raspberry Pi)
Planning Your Cross-Development Environment
The main consideration at this point is normally disk space. If you are using VirtualBox, limited memory can be another factor. If you are using Linux or Mac OS X, check your mounted disks for available space (or Windows tools as appropriate):
$ df −k
Filesystem 1024−blocks Used Available Capacity Mounted on
/dev/disk0s2 731734976 154168416 577310560 22% /
devfs 186 186 0 100% /dev
map –hosts 0 0 0 100% /net
map auto_home 0 0 0 100% /home
map –static 0 0 0 100% /Volumes/oth
$
In the preceding example output, we see that the root file system has plenty of space. But your file system may be laid out differently. Symlinks can be used when necessary to graft a larger disk area onto your home directory.
If you’re using VirtualBox, create virtual disks with enough space for the Linux operating system and your cross-compiler environment. You may want to put your Linux software on one virtual disk with a minimum size of about 8–10 GB (allow it to grow larger).
Allow a minimum of 10 GB for your cross-compiler environment (and allow it to grow). 9 GB is just barely large enough to host the cross-compiler tools and allow you to compile a Hello World type of program. But you must also factor in additional space for the Raspberry Linux kernel, its include files, and all other third-party libraries that you might need to build with (better still, a copy of the Raspberry Pi’s root file system).
Within your development Ubuntu/Debian build environment, make sure your cross-compiler and build area are using the disk area that has available space. It is easy to glibly create a directory someplace convenient and find out later that the space that you thought you were going to use wasn’t available.
Building the Cross-Compiler
At this point, I’ll assume that you’ve set up and installed Ubuntu in VirtualBox, if necessary. Otherwise, you are building your cross-compiler tools on an existing Ubuntu/Debian system, with disk space sufficient for the job.
We will be using the basic recipe outlined from this web resource:
“Cross-Compiling for Raspberry Pi,” www.kitware.com/blog/home/post/426
The released crosstool-NG downloads are found at this site:
http://crosstool-ng.org/download/crosstool-ng/
current release: crosstool-ng-1.19.0.tar.bz2
It is normal practice to download the newest stable version of software. I am using 1.19.0 in this text because it was current at the time of writing.
I’ll assume that you’ve symlinked to your disk area with sufficient available disk space. In other words, the symlink named ~/devel points to the development area to be used. This can be just a subdirectory if you have sufficient disk space there.
In directory ~/devel, create a subdirectory named staging (~/devel/staging) and change it to the following:
$ cd ~/devel # Dir is ~/devel
$ mkdir staging
$ cd ./staging # Dir is ~/devel/staging
$ pwd
/home/myuserid/devel/staging
$
Unpack the Tarball
Assuming the tarball crosstool-ng-1.19.0.tar.bz2 was downloaded to your home directory, you would perform the following (change the option j if the suffix is not .bz2):
$ tar xjvf ~/crosstool−ng−1.19.0.tar.bz2
. . .
$
After the unpacking completes, you should have a subdirectory named crosstoolng-1.19.0 in your staging directory.
You can choose a different location if you like, but I’m going to assume that the crosstool-NG software is going to install into /opt/x-tools. We’ll also assume your user ID is fred (substitute your own).
$ sudo mkdir -p /opt/x−tools
$ sudo chown fred /opt/x−tools
Once your installation is complete later, you can change the ownership back to root for protection.
The crosstool-NG build depends on several packages provided by Ubuntu/Debian as optionally installed software:
- bison: GNU yacc
- flex: GNU lex
- subversion: Subversion source control package
- libtool: Library tool
- texinfo: Install the texinfo package
- gawk: GNU awk (gawk)
- gperf: Perfect hash function generator
- automake: Tool for generating GNU standards-compliant Makefiles
Save time by making sure these packages are installed before proceeding to the next step. Package dependencies often change over time. Depending on your host system, you may find that additional packages (such as libncurses5-dev, for example) are also needed. If more packages are needed, the configure step usually identifies them.
With the package dependencies installed, you are now in a position to make the crosstool-NG software:
$ cd ~/devel/staging/crosstool−ng−1.19.0
$ ./configure −−prefix=/opt/x−tools
If this completes without errors, you are ready to build and install the crosstool-NG software. If it reports that you are missing package dependencies, install them now and repeat.
At this point, you should have no trouble building crosstool-NG. Perform the following make command:
$ cd ~/devel/staging/crosstool−ng−1.19.0
$ make
This takes very little time and seems trouble free.
make install
Once the crosstool-NG package has been compiled, it is ready to be installed into /opt/x-tools. From the same directory:
$ sudo make install
If you still own the directory /opt/x-tools from earlier (recall sudo chown fred /opt/x−tools), you won’t need to use sudo in the preceding step. After make install is performed, you will have the crosstool-NG command ct-ng installed in the directory /opt/x-tools/bin.
PATH
To use the newly installed ct-ng command, you will need to adjust your PATH environment variable:
$ PATH="/opt/x−tools/bin:$PATH"
The website also indicates that you might have to unset environment variable LD_LIBRARY_PATH, if your platform has it defined. If so, then unset it as follows:
$ unset LD_LIBRARY_PATH
Now you should be able to run ct-ng to get version info (note that there are no hyphens in front of version in the following command). Seeing the version output confirms that your ct-ng command has been installed and is functional:
$ ct–ng version
This is crosstool-NG version 1.19.0
Copyright (C) 2008 Yann E. MORIN <[email protected]>
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
The command ct-ng simplifies the work necessary to configure and build the cross-compiler tool chain. From here, we are concerned with building the cross-compiler tools themselves. When that process is completed, you will have populated the cross-compiler tools into the directory /opt/x-tools/arm-unknown-linux-gnueabi.
Before ct-ng can build your cross-compiler, it must first be configured:
$ cd ~/devel/staging
$ ct−ng menuconfig
If you get a “command not found” error message, check that the PATH variable is set properly.
When the command starts up, the menu configuration screen is presented.
Press Enter, to open the Paths and Misc Options submenu.
Once in the Paths and Misc Options menu, as shown next, use the cursor key to move down to Try Features Marked as Experimental. Once that line is highlighted, press the spacebar to put an asterisk inside the square brackets, to select the option (pressing space again toggles the setting).
After doing that, while in the same menu, move the cursor down to the middle entry labelled Prefix Directory and press Enter to select it.
For the procedure used in this book, modify the path to the following:
/opt/x-tools/${CT_TARGET}
as illustrated here:
Once the pathname is established, press Enter on the OK button shown. This returns you to the Paths and Misc Options menu.
Then select the Exit button shown at the bottom, and press Enter again.
From the main menu, select Target Options with the cursor and press Enter to open that menu. Then choose Target Architecture and press Enter. In that menu, choose Arm and use the Select button at the bottom. This returns you to the Target Options menu.
While in the Target Options menu (shown next), verify the Endianness setting by reviewing the status in brackets. It should read Little Endian. If not, enter that menu and change it to Little endian. Below the Endianness menu item is the Bitness option. It should already indicate 32-bit. If not, select it and change the setting to 32-bit.
Finally, exit this submenu with the Exit button.
Operating System
At the main menu again, choose Operating System and then choose Linux Kernel Version.
It is best to choose the release that most closely matches the kernel that you are using (perhaps, for example, 3.10.2). Once you have chosen, exit back to the main menu.
Binary Utilities
At the main menu, open the Binary Utilities menu. Cursor down to Binutils Version and open that submenu:
In this menu, you are presented various versions of binutils that can be used. Choose the most current stable (nonexperimental) version. Version 2.22 was chosen here. Select the version and exit back to the main menu.
C Compiler
At the main menu, open the C Compiler submenu. Here it is recommended that you enable the Show Linaro Versions option.
Once that is enabled, you can select the submenu Gcc Version:
The preceding figure shows linaro-4.8.2013.06-1 being chosen (which I had good results with). Newer versions are always becoming available. Choose the compiler and then choose the Select button at the bottom.
Then choose Exit once again to return to the main menu.
Save Configuration
Unless you have a reason to change anything else, exit the menu again to cause the Save prompt to appear:
Upon selecting Yes, the command exits with the following session output showing in the terminal window:
$ ct−ng menuconfig
IN config.gen/arch.in
IN config.gen/kernel.in
IN config.gen/cc.in
IN config.gen/libc.in
IN config/config.in
#
# configuration saved
#
At this point, it is worth mentioning that you may want to save your configuration somewhere outside the current directory. The configuration is saved in a file named .config and can be copied elsewhere. The following is a suggestion:
$ cp .config ~/ct-ng.config.bak
Saving the file outside the current directory will prevent accidental loss if ct-ng distclean is invoked.
Build Cross-Compiler
Check the ownership of /opt/x-tools. If you don’t own this directory, change the ownership now:
$ sudo chown −R fred /opt/x−tools
This will save you from having to execute the build process with root privileges. Now at long last, you can initiate the building of the cross-compiler:
$ cd ~/devel/staging
$ ct−ng build
Allow a good block of time for this job. This is not something that can be pushed through in a hurry. Ideally, you can just leave the command to run and check for successful completion in an hour or so. However, it is not uncommon for different software problems to arise at this stage. I once spent an entire Saturday troubleshooting this step. If you do encounter problems, read the next section for some troubleshooting tips.
If all goes well, ct-ng compiles and installs tools into /opt/x-tools without any further interaction. In the following session, Retrieving needed toolchain components is rather brief, because this was a session rerun with the components cached somewhere. Your download times will be longer when doing this for the first time.
[INFO] Performing some trivial sanity checks
[INFO] Build started 20140103.102402
[INFO] Building environment variables
[INFO] ============================================================
[INFO] Retrieving needed toolchain components' tarballs
[INFO] Retrieving needed toolchain components' tarballs: done in 0.13s (at 00:04)
[INFO] ============================================================
[INFO] Extracting and patching toolchain components
[INFO] Extracting and patching toolchain components: done in 3.96s (at 00:08)
[INFO] ============================================================
[INFO] Installing GMP for host
[INFO] Installing GMP for host: done in 37.57s (at 00:46)
[INFO] ============================================================
[INFO] Installing MPFR for host
[INFO] Installing MPFR for host: done in 18.16s (at 01:04)
[INFO] ============================================================
[INFO] Installing PPL for host
[INFO] Installing PPL for host: done in 268.27s (at 05:32)
[INFO] ============================================================
[INFO] Installing CLooG/PPL for host
[INFO] Installing CLooG/PPL for host: done in 6.45s (at 05:39)
[INFO] ============================================================
[INFO] Installing MPC for host
[INFO] Installing MPC for host: done in 7.97s (at 05:47)
[INFO] ============================================================
[INFO] Installing binutils for host
[INFO] Installing binutils for host: done in 53.52s (at 06:40)
[INFO] ============================================================
[INFO] Installing pass -1 core C compiler
[INFO] Installing pass -1 core C compiler: done in 222.36s (at 10:23)
[INFO] ============================================================
[INFO] Installing kernel headers
[INFO] Installing kernel headers: done in 4.54s (at 10:27)
[INFO] ============================================================
[INFO] Installing C library headers & start files
[INFO] Installing C library headers & start files: done in 31.26s (at 10:58)
[INFO] ============================================================
[INFO] Installing pass -2 core C compiler
[INFO] Installing pass -2 core C compiler: done in 512.54s (at 19:31)
[INFO] ============================================================
[INFO] Installing C library
[INFO] Installing C library: done in 805.58s (at 32:57)
[INFO] ============================================================
[INFO] Installing final compiler
[INFO] Installing final compiler: done in 484.58s (at 41:01)
[INFO] ============================================================
[INFO] Cleaning -up the toolchain's directory
[INFO] Stripping all toolchain executables
[INFO] Cleaning -up the toolchain's directory: done in 3.86s (at 41:05)
[INFO] Build completed at 20130103.110507
[INFO] (elapsed: 41:04.93)
[INFO] Finishing installation (may take a few seconds)...
[41:05] /
The overall time for my build was 41 minutes (reported to be 83 minutes on a Windows 8 Intel i5 using VirtualBox). My build was performed in VirtualBox running on Mac OS X Mavericks, using the Intel i7 processor (2.8 GHz). On the same Mac, I found that the times approximately doubled when the VirtualBox disk images were located on a USB 2.0 disk drive. From these figures, you can estimate your build time.
The session output that you get from this build process is very terse. As such, you don’t always get a clear idea of what the real failure was. For this reason, you’ll often need to check the build.log file:
$ less build.log
Using less, you can navigate to the end of the build.log file by typing a capital G.
One failure that frequently occurs in the beginning is a failed download. While the build process does retry downloads and tries different download methods, it can still fail. All that you need to do is to start the build again. It will download only the remaining files needed. Sometimes it will succeed on the second or third retry attempt.
Sometimes a component will fail in its configuration phase. Check the build.log file first to determine precisely which component is involved. Next you will want to examine the config.log file for that particular component. For example, let’s say the isl component failed. Dive down into the .build subdirectory until you find its config.log file:
$ cd .build/arm-unknown-linux-gnueabi/build/build-isl-host-i686-build_pc-linux-gnu
$ less config.log
Navigate to the end of config.log and work backward a few pages. Eventually, you will see text describing the command that was tried and the error message produced. In one instance, I was able to determine that the custom compiler option that I added (-fpermissive) was causing the failure. The solution then was to remove that option and try again.
Some errors will occur only with certain version choices. At one time, I was receiving errors related to PPL and needed a patch to correct it. Google is your friend (the following patch is an example):
http://patchwork.ozlabs.org/patch/330733/
I found that saving that patch to a file and applying it to the sources corrected the issue. Later, when I decided to start over with a different choice of compiler, this patch became unnecessary (the software was downloaded fresh again).
In getting through these issues, you can simply make corrections and then rerun the ct-ng build command. It is recommended that you plan for a later rebuild of everything again (after a clean), once the problems are solved. This will ensure that you have a good build without dependency issues.
If, after a correction, you run into the same problem, you may need to do a clean step first and start over. Depending on how deep you think the problem may be, choose one of the following:
- ct-ng clean
- ct-ng distclean (Be careful; see the following text.)
The ct-ng clean command will usually be enough, forcing the next build to start fresh. Any downloaded files and configuration will remain and are reused.
The ct-ng distclean command is much more drastic, since it removes all of the downloaded content and your configuration files. I copied the .config file to .config.bak and discovered to my horror that .config.bak had been removed! So if you back up the .config file, copy it outside the current directory for safety.
Above all, keep your head. It’s difficult to troubleshoot these issues if you feel time pressure or get angry over the time invested. When under time pressure, leave it for another day when you can deal with it leisurely and thoughtfully. Each redo takes considerable time. Wherever possible, eliminate the guesswork.
With each problem, take a deep breath, patiently look for clues, and pay attention to the details in the error messages. Remember that line in the movie Apollo 13 : “Work the problem, people!”