Cover image for Mastering Embedded Linux Programming - Third Edition

Mastering Embedded Linux Programming - Third Edition

Release Date: 2021/05/01
ISBN: 9781789530384
Topic:
0%
30
Chapters
0-1
Hours read
0k
Total Words
    Start Reading Now    
       Add to Wishlist       
View table of contents

Harness the power of Linux to create versatile and robust embedded solutions

Key Features

  • Learn how to develop and configure robust embedded Linux devices
  • Explore the new features of Linux 5.4 and the Yocto Project 3.1 (Dunfell)
  • Discover different ways to debug and profile your code in both user space and the Linux kernel

Book Description

Embedded Linux runs many of the devices we use every day. From smart TVs and Wi-Fi routers to test equipment and industrial controllers, all of them have Linux at their heart. The Linux OS is one of the foundational technologies comprising the core of the Internet of Things (IoT).

This book starts by breaking down the fundamental elements that underpin all embedded Linux projects: the toolchain, the bootloader, the kernel, and the root filesystem. After that, you will learn how to create each of these elements from scratch and automate the process using Buildroot and the Yocto Project. As you progress, the book explains how to implement an effective storage strategy for flash memory chips and install updates to a device remotely once it's deployed. You'll also learn about the key aspects of writing code for embedded Linux, such as how to access hardware from apps, the implications of writing multi-threaded code, and techniques to manage memory in an efficient way. The final chapters demonstrate how to debug your code, whether it resides in apps or in the Linux kernel itself. You'll also cover the different tracers and profilers that are available for Linux so that you can quickly pinpoint any performance bottlenecks in your system.

By the end of this Linux book, you'll be able to create efficient and secure embedded devices using Linux.

What you will learn

  • Use Buildroot and the Yocto Project to create embedded Linux systems
  • Troubleshoot BitBake build failures and streamline your Yocto development workflow
  • Update IoT devices securely in the field using Mender or balena
  • Prototype peripheral additions by reading schematics, modifying device trees, soldering breakout boards, and probing pins with a logic analyzer
  • Interact with hardware without having to write kernel device drivers
  • Divide your system up into services supervised by BusyBox runit
  • Debug devices remotely using GDB and measure the performance of systems using tools such as perf, ftrace, eBPF, and Callgrind

Who this book is for

If you're a systems software engineer or system administrator who wants to learn Linux implementation on embedded devices, then this book is for you. Embedded systems engineers accustomed to programming for low-power microcontrollers can use this book to help make the leap to high-speed systems on chips that can run Linux. Anyone responsible for developing new hardware that needs to run Linux will also find this book useful. Basic working knowledge of the POSIX standard, C programming, and shell scripting is assumed.

Table of Contents

  1. Mastering Embedded Linux Programming Third Edition
  2. Contributors
  3. About the authors
  4. About the reviewers
  5. Preface
    1. Who this book is for
    2. What this book covers
    3. To get the most out of this book
    4. Download the example code files
    5. Download the color images
    6. Conventions used
    7. Get in touch
    8. Reviews
  6. Section 1: Elements of Embedded Linux
  7. Chapter 1: Starting Out
    1. Choosing Linux
    2. When not to choose Linux
    3. Meeting the players
    4. Moving through the project life cycle
    5. The four elements of embedded Linux
    6. Navigating open source
    7. Licenses
    8. Selecting hardware for embedded Linux
    9. Obtaining the hardware for this book
    10. The Raspberry Pi 4
    11. The BeagleBone Black
    12. QEMU
    13. Provisioning your development environment
    14. Summary
  8. Chapter 2: Learning about Toolchains
    1. Technical requirements
    2. Introducing toolchains
    3. Types of toolchains
    4. CPU architectures
    5. Choosing the C library
    6. Finding a toolchain
    7. Building a toolchain using crosstool-NG
    8. Installing crosstool-NG
    9. Building a toolchain for BeagleBone Black
    10. Building a toolchain for QEMU
    11. Anatomy of a toolchain
    12. Finding out about your cross compiler
    13. The sysroot, library, and header files
    14. Other tools in the toolchain
    15. Looking at the components of the C library
    16. Linking with libraries – static and dynamic linking
    17. Static libraries
    18. Shared libraries
    19. The art of cross-compiling
    20. Simple makefiles
    21. Autotools
    22. Package configuration
    23. Problems with cross-compiling
    24. CMake
    25. Summary
    26. Further reading
  9. Chapter 3: All about Bootloaders
    1. Technical requirements
    2. What does a bootloader do?
    3. The boot sequence
    4. Phase 1 – ROM code
    5. Phase 2 – secondary program loader
    6. Phase 3 – TPL
    7. Moving from the bootloader to a kernel
    8. Introducing device trees
    9. Device tree basics
    10. The reg property
    11. Labels and interrupts
    12. Device tree include files
    13. Compiling a device tree
    14. U-Boot
    15. Building U-Boot
    16. Installing U-Boot
    17. Using U-Boot
    18. Booting Linux
    19. Porting U-Boot to a new board
    20. Building and testing
    21. Falcon mode
    22. Summary
  10. Chapter 4: Configuring and Building the Kernel
    1. Technical requirements
    2. What does the kernel do?
    3. Choosing a kernel
    4. Kernel development cycle
    5. Stable and long-term support releases
    6. Building the kernel
    7. Getting the source
    8. Understanding kernel configuration – Kconfig
    9. Using LOCALVERSION to identify your kernel
    10. When to use kernel modules
    11. Compiling – Kbuild
    12. Finding out which kernel target to build
    13. Build artifacts
    14. Compiling device trees
    15. Compiling modules
    16. Cleaning kernel sources
    17. Building a 64-bit kernel for the Raspberry Pi 4
    18. Building a kernel for the BeagleBone Black
    19. Building a kernel for QEMU
    20. Booting the kernel
    21. Booting the Raspberry Pi 4
    22. Booting the BeagleBone Black
    23. Booting QEMU
    24. Kernel panic
    25. Early user space
    26. Kernel messages
    27. The kernel command line
    28. Porting Linux to a new board
    29. A new device tree
    30. Setting the board's compatible property
    31. Summary
    32. Additional reading
  11. Chapter 5: Building a Root Filesystem
    1. Technical requirements
    2. What should be in the root filesystem?
    3. The directory layout
    4. The staging directory
    5. POSIX file access permissions
    6. File ownership permissions in the staging directory
    7. Programs for the root filesystem
    8. Libraries for the root filesystem
    9. Device nodes
    10. The proc and sysfs filesystems
    11. Kernel modules
    12. Transferring the root filesystem to the target
    13. Creating a boot initramfs
    14. Standalone initramfs
    15. Booting the initramfs
    16. Booting with QEMU
    17. Booting the BeagleBone Black
    18. Building an initramfs into the kernel image
    19. Building an initramfs using a device table
    20. The old initrd format
    21. The init program
    22. Starting a daemon process
    23. Configuring user accounts
    24. Adding user accounts to the root filesystem
    25. A better way of managing device nodes
    26. An example using devtmpfs
    27. An example using mdev
    28. Are static device nodes so bad after all?
    29. Configuring the network
    30. Network components for glibc
    31. Creating filesystem images with device tables
    32. Booting the BeagleBone Black
    33. Mounting the root filesystem using NFS
    34. Testing with QEMU
    35. Testing with the BeagleBone Black
    36. Problems with file permissions
    37. Using TFTP to load the kernel
    38. Summary
    39. Further reading
  12. Chapter 6: Selecting a Build System
    1. Technical requirements
    2. Comparing build systems
    3. Distributing binaries
    4. Introducing Buildroot
    5. Background
    6. Stable releases and long-term support
    7. Installing
    8. Configuring
    9. Running
    10. Targeting real hardware
    11. Creating a custom BSP
    12. Adding your own code
    13. License compliance
    14. Introducing the Yocto Project
    15. Background
    16. Stable releases and supports
    17. Installing the Yocto Project
    18. Configuring
    19. Building
    20. Running the QEMU target
    21. Layers
    22. Customizing images via local.conf
    23. Writing an image recipe
    24. Creating an SDK
    25. The license audit
    26. Summary
    27. Further reading
  13. Chapter 7: Developing with Yocto
    1. Technical requirements
    2. Building on top of an existing BSP
    3. Building an existing BSP
    4. Controlling Wi-Fi
    5. Controlling Bluetooth
    6. Adding a custom layer
    7. Capturing changes with devtool
    8. Development workflows
    9. Creating a new recipe
    10. Modifying the source built by a recipe
    11. Upgrading a recipe to a newer version
    12. Building your own distro
    13. When and when not to
    14. Creating a new distro layer
    15. Configuring your distro
    16. Adding more recipes to your distro
    17. Runtime package management
    18. Provisioning a remote package server
    19. Summary
    20. Further reading
  14. Chapter 8: Yocto Under the Hood
    1. Technical requirements
    2. Decomposing Yocto's architecture and workflow
    3. Metadata
    4. Build tasks
    5. Image generation
    6. Separating metadata into layers
    7. Troubleshooting build failures
    8. Isolating errors
    9. Dumping the environment
    10. Reading the task log
    11. Adding more logging
    12. Running commands from devshell
    13. Graphing dependencies
    14. Understanding BitBake syntax and semantics
    15. Tasks
    16. Dependencies
    17. Variables
    18. Functions
    19. RDEPENDS revisited
    20. Summary
    21. Further reading
  15. Section 2: System Architecture and Design Decisions
  16. Chapter 9: Creating a Storage Strategy
    1. Technical requirements
    2. Storage options
    3. NOR flash
    4. NAND flash
    5. Accessing flash memory from the bootloader
    6. U-Boot and NOR flash
    7. U-Boot and NAND flash
    8. U-Boot and MMC, SD, and eMMC
    9. Accessing flash memory from Linux
    10. Memory technology devices
    11. The MMC block driver
    12. Filesystems for flash memory
    13. Flash translation layers
    14. Filesystems for NOR and NAND flash memory
    15. JFFS2
    16. YAFFS2
    17. UBI and UBIFS
    18. Filesystems for managed flash
    19. Flashbench
    20. Discard and TRIM
    21. Ext4
    22. F2FS
    23. FAT16/32
    24. Read-only compressed filesystems
    25. SquashFS
    26. Temporary filesystems
    27. Making the root filesystem read-only
    28. Filesystem choices
    29. Summary
    30. Further reading
  17. Chapter 10: Updating Software in the Field
    1. Technical requirements
    2. From where do updates originate?
    3. What to update
    4. Bootloader
    5. Kernel
    6. Root filesystem
    7. System applications
    8. Device-specific data
    9. Components that need to be updated
    10. The basics of software updates
    11. Making updates robust
    12. Making updates fail-safe
    13. Making updates secure
    14. Types of update mechanism
    15. Symmetric image update
    16. Asymmetric image update
    17. Atomic file updates
    18. OTA updates
    19. Using Mender for local updates
    20. Building the Mender client
    21. Installing an update
    22. Using Mender for OTA updates
    23. Using balena for local updates
    24. Creating an account
    25. Creating an application
    26. Adding a device
    27. Installing the CLI
    28. Pushing a project
    29. Summary
  18. Chapter 11: Interfacing with Device Drivers
    1. Technical requirements
    2. The role of device drivers
    3. Character devices
    4. Block devices
    5. Network devices
    6. Finding out about drivers at runtime
    7. Getting information from sysfs
    8. Finding the right device driver
    9. Device drivers in user space
    10. GPIO
    11. LEDs
    12. I2C
    13. SPI
    14. Writing a kernel device driver
    15. Designing a character driver interface
    16. The anatomy of a device driver
    17. Compiling kernel modules
    18. Loading kernel modules
    19. Discovering the hardware configuration
    20. Device trees
    21. The platform data
    22. Linking hardware with device drivers
    23. Summary
    24. Further reading
  19. Chapter 12: Prototyping with Breakout Boards
    1. Technical requirements
    2. Mapping schematics to the device tree's source
    3. Reading schematics and data sheets
    4. Installing Debian on the BeagleBone Black
    5. Enabling spidev
    6. Customizing the device tree
    7. Prototyping with breakout boards
    8. Closing the SPI jumper
    9. Attaching the GNSS antenna
    10. Attaching the SPI header
    11. Connecting the SPI jumper wires
    12. Probing SPI signals with a logic analyzer
    13. Receiving NMEA messages over SPI
    14. Summary
    15. Further reading
  20. Chapter 13: Starting Up – The init Program
    1. Technical requirements
    2. After the kernel has booted
    3. Introducing the init programs
    4. BusyBox init
    5. Buildroot init scripts
    6. System V init
    7. inittab
    8. The init.d scripts
    9. Adding a new daemon
    10. Starting and stopping services
    11. systemd
    12. Building systemd with the Yocto Project and Buildroot
    13. Introducing targets, services, and units
    14. How systemd boots the system
    15. Adding your own service
    16. Adding a watchdog
    17. Implications for embedded Linux
    18. Summary
    19. Further reading
  21. Chapter 14: Starting with BusyBox runit
    1. Technical requirements
    2. Getting BusyBox runit
    3. Creating service directories and files
    4. Service directory layout
    5. Service configuration
    6. Service supervision
    7. Controlling services
    8. Depending on other services
    9. Start dependencies
    10. Custom start dependencies
    11. Putting it all together
    12. Dedicated service logging
    13. How does it work?
    14. Adding dedicated logging to a service
    15. Log rotation
    16. Signaling a service
    17. Summary
    18. Further reading
  22. Chapter 15: Managing Power
    1. Technical requirements
    2. Measuring power usage
    3. Scaling the clock frequency
    4. The CPUFreq driver
    5. Using CPUFreq
    6. Selecting the best idle state
    7. The CPUIdle driver
    8. Tickless operation
    9. Powering down peripherals
    10. Putting the system to sleep
    11. Power states
    12. Wakeup events
    13. Timed wakeups from the real-time clock
    14. Summary
    15. Further reading
  23. Section 3: Writing Embedded Applications
  24. Chapter 16: Packaging Python
    1. Technical requirements
    2. Getting Docker
    3. Retracing the origins of Python packaging
    4. distutils
    5. setuptools
    6. setup.py
    7. Installing Python packages with pip
    8. requirements.txt
    9. Managing Python virtual environments with venv
    10. Installing precompiled binaries with conda
    11. Environment management
    12. Package management
    13. Deploying Python applications with Docker
    14. The anatomy of a Dockerfile
    15. Building a Docker image
    16. Running a Docker image
    17. Fetching a Docker image
    18. Publishing a Docker image
    19. Cleaning up
    20. Summary
    21. Further reading
  25. Chapter 17: Learning about Processes and Threads
    1. Technical requirements
    2. Process or thread?
    3. Processes
    4. Creating a new process
    5. Terminating a process
    6. Running a different program
    7. Daemons
    8. Inter-process communication
    9. Threads
    10. Creating a new thread
    11. Terminating a thread
    12. Compiling a program with threads
    13. Inter-thread communication
    14. Mutual exclusion
    15. Changing conditions
    16. Partitioning the problem
    17. ZeroMQ
    18. Getting pyzmq
    19. Messaging between processes
    20. Messaging within processes
    21. Scheduling
    22. Fairness versus determinism
    23. Time-shared policies
    24. Real-time policies
    25. Choosing a policy
    26. Choosing a real-time priority
    27. Summary
    28. Further reading
  26. Chapter 18: Managing Memory
    1. Technical requirements
    2. Virtual memory basics
    3. Kernel space memory layout
    4. How much memory does the kernel use?
    5. User space memory layout
    6. The process memory map
    7. Swapping
    8. Swapping to compressed memory (zram)
    9. Mapping memory with mmap
    10. Using mmap to allocate private memory
    11. Using mmap to share memory
    12. Using mmap to access device memory
    13. How much memory does my application use?
    14. Per-process memory usage
    15. Using top and ps
    16. Using smem
    17. Other tools to consider
    18. Identifying memory leaks
    19. mtrace
    20. Valgrind
    21. Running out of memory
    22. Summary
    23. Further reading
  27. Section 4: Debugging and Optimizing Performance
  28. Chapter 19: Debugging with GDB
    1. Technical requirements
    2. The GNU debugger
    3. Preparing to debug
    4. Debugging applications
    5. Remote debugging using gdbserver
    6. Setting up the Yocto Project for remote debugging
    7. Setting up Buildroot for remote debugging
    8. Starting to debug
    9. Native debugging
    10. Just-in-time debugging
    11. Debugging forks and threads
    12. Core files
    13. Using GDB to look at core files
    14. GDB user interfaces
    15. Terminal User Interface
    16. Data Display Debugger
    17. Visual Studio Code
    18. Debugging kernel code
    19. Debugging kernel code with kgdb
    20. A sample debug session
    21. Debugging early code
    22. Debugging modules
    23. Debugging kernel code with kdb
    24. Looking at an Oops
    25. Preserving the Oops
    26. Summary
    27. Further reading
  29. Chapter 20: Profiling and Tracing
    1. Technical requirements
    2. The observer effect
    3. Symbol tables and compile flags
    4. Beginning to profile
    5. Profiling with top
    6. The poor man's profiler
    7. Introducing perf
    8. Configuring the kernel for perf
    9. Building perf with the Yocto Project
    10. Building perf with Buildroot
    11. Profiling with perf
    12. Call graphs
    13. perf annotate
    14. Tracing events
    15. Introducing Ftrace
    16. Preparing to use Ftrace
    17. Using Ftrace
    18. Dynamic Ftrace and trace filters
    19. Trace events
    20. Using LTTng
    21. LTTng and the Yocto Project
    22. LTTng and Buildroot
    23. Using LTTng for kernel tracing
    24. Using BPF
    25. Configuring the kernel for BPF
    26. Building a BCC toolkit with Buildroot
    27. Using BPF tracing tools
    28. Using Valgrind
    29. Callgrind
    30. Helgrind
    31. Using strace
    32. Summary
    33. Further reading
  30. Chapter 21: Real-Time Programming
    1. Technical requirements
    2. What is real time?
    3. Identifying sources of non-determinism
    4. Understanding scheduling latency
    5. Kernel preemption
    6. The real-time Linux kernel (PREEMPT_RT)
    7. Threaded interrupt handlers
    8. Preemptible kernel locks
    9. Getting the PREEMPT_RT patches
    10. The Yocto Project and PREEMPT_RT
    11. High-resolution timers
    12. Avoiding page faults
    13. Interrupt shielding
    14. Measuring scheduling latencies
    15. cyclictest
    16. Using Ftrace
    17. Combining cyclictest and Ftrace
    18. Summary
    19. Further reading
    20. Why subscribe?
  31. Other Books You May Enjoy
    1. Packt is searching for authors like you
    2. Leave a review - let other readers know what you think
3.138.200.66