In Chapter 6, Working with Files and Directories in Rust, we covered the details of how to perform file I/O operations (such as reading and writing to files) using the Rust Standard Library. In Unix-like operating systems, a file is an abstraction that is used to work not only with regular disk files (which are used to store data) but also with several types of devices that are connected to a machine. In this chapter, we will look at the features of the Rust Standard Library that enable us to perform reads and writes to any type of device (also called device I/O) in Rust. Device I/O is an essential aspect of system programming to monitor and control various types of devices attached to a computer, such as keyboards, USB cameras, printers, and sound cards. You may be curious to know what support Rust provides to a system programmer to handle all these different types of devices. We'll answer this question as we go through the chapter.
In this chapter, we will review the basics of I/O management in Unix/Linux using the Rust Standard Library, including handling errors, and then write a program to detect and print details of connected USB devices.
We will cover these topics in the following order:
By the end of this chapter, you will have learned how to work with standard readers and writers, which constitute the foundation of any I/O operation. You'll also learn how to optimize system calls through the use of buffered reads and writes. We'll cover reading and writing to standard I/O streams of a process and handling errors from I/O operations, as well as learning ways to iterate over I/O. These concepts will be reinforced through an example project.
Verify that rustup, rustc, and cargo have been installed correctly with the following command:
rustup --version
rustc --version
cargo --version
The Git repo for the code in this chapter can be found at https://github.com/PacktPublishing/Practical-System-Programming-for-Rust-Developers/tree/master/Chapter10/usb.
For running and testing the project in this book, you must have the native libusb library installed where it can be found by pkg-config.
The project in this book has been tested on macOS Catalina 10.15.6.
For instructions on building and testing on Windows, refer: https://github.com/dcuddeback/libusb-rs/issues/20
For general instructions on environmental setup of libusb crate, refer to: https://github.com/dcuddeback/libusb-rs
In previous chapters, we saw how to schedule work on CPUs using processes and threads, and how to manage memory by controlling the memory layout of a program. In addition to the CPU and memory, the operating system also manages the system's hardware devices. Examples of hardware devices include keyboards, mice, hard disks, video adapters, audio cards, network adapters, scanners, cameras, and other USB devices. But the peculiarities of these physical hardware devices are hidden from the user programs by the operating system, using software modules called device drivers. Device drivers are indispensable software components for doing device I/O. Let's take a closer look at them.
Device drivers are shared libraries loaded into the kernel that contain functions to perform low-level hardware control. They communicate with the devices through the computer bus or communication subsystem to which the device is connected. They are specific to each device type (for example, a mouse or network adaptor) or class of devices (for example, IDE or SCSI disk controllers). They are also specific to an operating system (for example, a device driver for Windows doesn't work on Linux even for the same device type).
Device drivers handle the peculiarities of the devices (or device classes) for which they are written. For example, a device driver to control a hard disk receives requests to read or write some file data identified by a block number. The device driver translates the block number into track, sector, and cylinder numbers on the disk. It also initializes the device, checks whether the device is in use, validates input parameters to its function calls, determines the commands to be issued, and issues them to the device. It handles the interrupts from the device and communicates them back to the calling program. The device driver further implements the specific hardware protocols that the device supports, such as SCSI/ATA/SATA for disk access or UART for serial port communications. Device drivers thus abstract away a lot of the hardware-specific details of controlling devices.
The operating system (specifically the kernel) accepts system calls from the user programs for device access and control, and then uses the respective device driver to physically access and control the device. Figure 10.1 illustrates how user space programs (for example, Rust programs that use the standard library to talk to the operating system kernel) use system calls to manage and control various types of devices:
In Chapter 6, Working with Files and Directories in Rust, we saw that Linux/Unix has the philosophy that everything is a file, characterized by the universality of I/O. The same system calls, such as open(), close(), read(), and write(), can be applied to all types of I/O whether it's a regular file (used to store text or binary data), a directory, device files, or network connections. What this means is that programmers of user space programs can write code to communicate with and control devices without worrying about the protocol and hardware specifics of the devices, thanks to the abstraction layers provided by the kernel (system calls) and device drivers. Furthermore, the Rust Standard Library adds another layer of abstraction to provide a device-independent software layer, which Rust programs can use for device I/O. This is the primary focus of this chapter.
In Unix/Linux, devices are broadly classified into three types:
A hardware device is identified by its type (block or character) and a device number. The device number in turn is split into a major and minor device number.
When a new hardware is connected, the kernel needs a device driver that is compatible with the device and can operate the device controller hardware. A device driver, as discussed earlier, is essentially a shared library of low-level, hardware-handling functions that can operate in a privileged manner as part of the kernel. Without device drivers, the kernel does not know how to operate the device. When a program attempts to connect to a device, the kernel looks up associated information in its tables and transfers control to the device driver. There are separate tables for block and character devices. The device driver performs the required task on the device and returns control back to the operating system kernel.
As an example, let's look at a web server sending a page to a web browser. The data is structured as an HTTP response message with the web page (HTML) sent as part of its data payload. The data itself is stored in the kernel in a buffer (data structure), which is then passed to the TCP layer, then to the IP layer, on to the Ethernet device driver, then to the Ethernet adaptor, and onward to the network. The Ethernet device driver does not know anything about connections and only handles data packets. Similarly, when data needs to be stored to a file on the disk, the data is stored in a buffer, which is passed on to the filesystem device driver and then onward to the disk controller, which then saves it to the disk (for example, hard disk, SSD, and so on). Essentially, the kernel relies on a device driver to interface with the device.
Device drivers are usually part of the kernel (kernel device driver), but there are also user space device drivers, which abstract out the details of kernel access. Later in this chapter, we will be using one such user space device driver to detect USB devices.
We've discussed the basics of device I/O, including device drivers and types of devices in Unix-like systems, in this section. Starting from the next section, we'll focus on how to do device-independent I/O using features from the Rust Standard Library.
Reads and writes are the fundamental operations performed on I/O types such as files and streams and are very crucial for working with many types of system resources. In this section, we'll discuss different ways to do reads and writes to I/O in Rust. We'll first cover the core traits – Read and Write – which allow Rust programs to perform read and write operations on objects that implement these traits (which are also called readers and writers). Then, we'll see how to do buffered reads and buffered writes, which are more efficient for certain types of read and write operations.
Let's start with the basic Read and Write traits.
In line with the everything-is-a-file philosophy, the Rust Standard Library provides two traits – Read and Write – which provide a general interface for reading and writing inputs and outputs. This trait is implemented for different types of I/O, such as files, TcpStream, standard input, and standard output streams of processes.
An example of using the Read trait is shown in the following code. Here, we are opening a records.txt file with the open() function in the std::fs::File module (which we learned earlier). We're then bringing the Read trait from the std::io module into scope, and using the read() method of this trait to read bytes from a file. The same read() method can also be used to read from any other entity implementing the Read trait, such as a network socket or a standard input stream:
use std::fs::File;
use std::io::Read;
fn main() {
// Open a file
let mut f = File::open("records.txt").unwrap();
//Create a memory buffer to read from file
let mut buffer = [0; 1024];
// read from file into buffer
let _ = f.read(&mut buffer[..]).unwrap();
}
Create a file called records.txt in the project root and run the program with cargo run. You can optionally print out the value of the buffer, which will display the raw bytes.
Read and Write are byte-based interfaces, which can get inefficient as they involve continual system calls to the operating system. To overcome this, Rust also provides two structs to enable doing buffered reads and writes – BufReader and BufWriter, which have a built-in buffer and reduce the number of calls to the operating system.
The previous example can be rewritten as shown here, to use BufReader:
use std::fs::File;
use std::io::{BufRead, BufReader};
fn main() {
// Open a file
let f = File::open("records.txt").unwrap();
// Create a BufReader, passing in the file handle
let mut buf_reader = BufReader::new(f);
//Create a memory buffer to read from file
let mut buffer = String::new();
// read a line into the buffer
buf_reader.read_line(&mut buffer).unwrap();
println!("Read the following: {}", buffer);
}
The code changes (from the previous version) have been highlighted. BufReader uses the BufRead trait, which is brought into scope. Instead of reading directly from the file handle, we create a BufReader instance and read a line into this struct. The BufReader methods internally optimize calls to the operating system. Run the program and verify that the value from the file is printed correctly.
BufWriter similarly buffers writes to the disk, thus minimizing system calls. It can be used in a similar manner as shown in the following code:
use std::fs::File;
use std::io::{BufWriter, Write};
fn main() {
// Create a file
let f = File::create("file.txt").unwrap();
// Create a BufWriter, passing in the file handle
let mut buf_writer = BufWriter::new(f);
//Create a memory buffer
let buffer = String::from("Hello, testing");
// write into the buffer
buf_writer.write(buffer.as_bytes()).unwrap();
println!("wrote the following: {}", buffer);
}
In the code shown, we're creating a new file to write into, and are also creating a new BufWriter instance. We then write a value from the buffer into the BufWriter instance. Run the program and verify that the specified string value has been written to a file with the name file.txt in the project root directory. Note that here, in addition to BufWriter, we also have to bring the Write trait into scope as this contains the write() method.
Note when to use and when not to use BufReader and BufWriter:
In this section, we saw how to do both unbuffered and buffered reads and writes. In the next section, we'll learn how to work with standard inputs and outputs of a process.
In Linux/Unix, streams are communication channels between a process and its environment. By default, three standard streams are created for every running process: standard input, standard output, and standard error. A stream is a communication channel that has two ends. One end is connected to the process and the other end to another system resource. For example, a standard input can be used by a process to read characters or text from a keyboard or another process. Similarly, a standard output stream can be used by a process to send some characters to the terminal or to a file. In many modern programs, the standard error of a process is connected to a log file, which makes analyzing and debugging errors easier.
The Rust Standard Library provides methods to interact with standard input and output streams. The Stdin struct in the std::io module represents the handle to the input stream of a process. This handle implements the Read trait, which we covered in the previous section.
The code example here shows how to interact with the standard input and standard output streams of a process. In the code shown, we are reading a line from the standard input into a buffer. We're then writing back the contents of the buffer to the standard output of the process. Note that here, the word process refers to the running program that you have written. You are essentially reading from and writing to the standard input and standard output, respectively, of the running program:
use std::io::{self, Write};
fn main() {
//Create a memory buffer to read from file
let mut buffer = String::new();
// read a line into the buffer
let _ = io::stdin().read_line(&mut buffer).unwrap();
// Write the buffer to standard output
io::stdout().write(&mut buffer.as_bytes()).unwrap();
}
Run the program with cargo run, enter some text, and hit the Enter key. You'll see the text echoed back on the terminal.
Stdin, which is a handle to the input stream of a process, is a shared reference to a global buffer of input data. Likewise, Stdout, which is the output stream of a process, is a shared reference to a global data buffer. Since Stdin and Stdout are references to shared data, to ensure exclusive use of these data buffers, the handles can be locked. For example, the StdinLock struct in the std::io module represents a locked reference to the Stdin handle. Likewise, the StdoutLock struct in the std::io module represents a locked reference to the Stdout handle. Examples of how to use the locked reference are shown in the code example here:
use std::io::{Read, Write};
fn main() {
//Create a memory buffer
let mut buffer = [8; 1024];
// Get handle to input stream
let stdin_handle = std::io::stdin();
// Lock the handle to input stream
let mut locked_stdin_handle = stdin_handle.lock();
// read a line into the buffer
locked_stdin_handle.read(&mut buffer).unwrap();
// Get handle to output stream
let stdout_handle = std::io::stdout();
// Lock the handle to output stream
let mut locked_stdout_handle = stdout_handle.lock();
// Write the buffer to standard output
locked_stdout_handle.write(&mut buffer).unwrap();
}
In the code shown, the standard input and output stream handles are locked before reading and writing to them.
We can similarly write to the standard error stream. A code example is shown here:
use std::io::Write;
fn main() {
//Create a memory buffer
let buffer = b"Hello, this is error message from
standard
error stream ";
// Get handle to output error stream
let stderr_handle = std::io::stderr();
// Lock the handle to output error stream
let mut locked_stderr_handle = stderr_handle.lock();
// write into error stream from buffer
locked_stderr_handle.write(buffer).unwrap();
}
In the code shown, we're constructing a handle to the standard error stream using the stderr() function. Then, we're locking this handle and then writing some text to it.
In this section, we've seen how to interact with the standard input, standard output, and standard error streams of a process using the Rust Standard Library. Recall that in the previous chapter on managing concurrency, we saw how, from a parent process, we can read from and write to the standard input and output streams of the child process.
In the next section, let's look at a couple of functional programming constructs that can be used for I/O in Rust.
In this section, we'll look at how to use iterators and chaining with the std::io module.
Many of the data structures provided by the std::io module have built-in iterators. Iterators let you process a series of items, such as lines in a file or incoming network connections on a port. They provide a nicer mechanism compared to while and for loops. Here is an example of using the lines() iterator with the BufReader struct, which is a part of the std::io module. This program reads lines from the standard input stream in a loop:
use std::io::{BufRead, BufReader};
fn main() {
// Create handle to standard input
let s = std::io::stdin();
//Create a BufReader instance to optimize sys calls
let file_reader = BufReader::new(s);
// Read from standard input line-by-line
for single_line in file_reader.lines() {
println!("You typed:{}", single_line.unwrap());
}
}
In the code shown, we have created a handle to the standard input stream and passed it to a BufReader struct. This struct implements the BufRead trait, which has a lines() method that returns an iterator over the lines of the reader. This helps us to type inputs on the terminal line by line and have it read by our running program. The text entered on the terminal is echoed back to the terminal. Execute cargo run, and type some text, and then hit the Enter key. Repeat this step as many times as you'd like. Exit from the program with Ctrl + C.
Likewise, the iterator can be used to read line by line from a file (instead of from standard input, which we saw in the previous example). A code snippet is shown here:
use std::fs::File;
use std::io::{BufRead, BufReader};
fn main() {
// Open a file for reading
let f = File::open("file.txt").unwrap();
//Create a BufReader instance to optimize sys calls
let file_reader = BufReader::new(f);
// Read from standard input line-by-line
for single_line in file_reader.lines() {
println!("Line read from file :{}",
single_line.unwrap());
}
}
Create a file called file.txt in the project root directory. Enter a few lines of text in this file. Then, run the program using cargo run. You'll see the file contents printed out to the terminal.
We've so far seen how to use iterators from the std::io module. Let's now look at another concept: chaining.
The Read trait in the std::io module has a chain() method, which allows us to chain multiple BufReader together into one handle. Here is an example of how to create a single chained handle combining two files, and how to read from this handle:
use std::fs::File;
use std::io::Read;
fn main() {
// Open two file handles for reading
let f1 = File::open("file1.txt").unwrap();
let f2 = File::open("file2.txt").unwrap();
//Chain the two file handles
let mut chained_handle = f1.chain(f2);
// Create a buffer to read into
let mut buffer = String::new();
// Read from chained handle into buffer
chained_handle.read_to_string(&mut buffer).unwrap();
// Print out the value read into the buffer
println!("Read from chained handle: {}", buffer);
}
The statement using the chain() method has been highlighted in the code. The rest of the code is fairly self-explanatory, as it is similar to what we've seen in previous examples. Ensure to create two files, file1.txt and file2.txt, under the project root folder and enter a few lines of text in each. Run the program with cargo run. You'll see the data from both files printed out line by line.
In this section, we've seen how to use iterators and how to chain readers together. In the next section, let's take a look at error handling for I/O operations.
In this section, we'll learn about the built-in error handling support in the std::io module. Handling recoverable errors in an appropriate manner makes Rust programs more robust.
In the code examples we've seen so far, we've used the unwrap() function to extract the return value from the std::io module methods and associated functions, such as Read, Write, BufReader, and BufWriter. However, this is not the correct way to handle errors. The std::io module has a specialized Result type that is returned from any function or method in this module that may produce an error.
Let's rewrite the previous example (of chaining readers) using the io::Result type as the return value from the function. This allows us to use the ? operator to directly pass errors back from the main() function, instead of using the unwrap() function:
use std::fs::File;
use std::io::Read;
fn main() -> std::io::Result<()> {
// Open two file handles for reading
let f1 = File::open("file1.txt")?;
let f2 = File::open("file3.txt")?;
//Chain the two file handles
let mut chained_handle = f1.chain(f2);
// Create a buffer to read into
let mut buffer = String::new();
// Read from chained handle into buffer
chained_handle.read_to_string(&mut buffer)?;
println!("Read from chained handle: {}", buffer);
Ok(())
}
Code related to error handling has been highlighted. Run the program with cargo run, this time making sure that neither file1.txt nor file3.txt exists in the project root folder.
You'll see the error message printed to the terminal.
In the code we've just seen, we're just propagating the error received from the operating system while making the calls. Let's now try to handle the errors in a more active manner. The code example here shows custom error handling for the same code:
use std::fs::File;
use std::io::Read;
fn read_files(handle: &mut impl Read) ->
std::io::Result<String> {
// Create a buffer to read into
let mut buffer = String::new();
// Read from chained handle into buffer
handle.read_to_string(&mut buffer)?;
Ok(buffer)
}
fn main() {
let mut chained_handle;
// Open two file handles for reading
let file1 = "file1.txt";
let file2 = "file3.txt";
if let Ok(f1) = File::open(file1) {
if let Ok(f2) = File::open(file2) {
//Chain the two file handles
chained_handle = f1.chain(f2);
let content = read_files(&mut chained_handle);
match content {
Ok(text) => println!("Read from chained
handle: {}", text),
Err(e) => println!("Error occurred in
reading files: {}", e),
}
} else {
println!("Unable to read {}", file2);
}
} else {
println!("Unable to read {}", file1);
}
}
You'll notice that we've created a new function that returns std::io::Result to the main() function. We're handling errors in various operations, such as reading from a file and reading from the chained readers.
First, run the program with cargo run, ensuring that both file1.txt and file2.txt exist. You'll see the contents from both files printed to the terminal. Rerun the program by removing one of these files. You should see the custom error message from our code.
With this, we conclude the section on handling errors. Let's now move on to the last section of the chapter, where we will go through a project to detect and display details of USB devices connected to a computer.
In this section, we will demonstrate an example of working with devices in Rust. The example chosen is to display details of all connected USB devices of a computer. We'll be using libusb, a C library that helps to interact with USB devices. The libusb crate in Rust is a safe wrapper around the C libusb library. Let's first look at the design.
We're using the libusb library because writing a USB device driver from scratch requires implementing the USB protocol specifications, and writing device drivers is the subject of a separate book in itself. Let's look at the design of our program:
Figure 10.2 shows the structs and functions in the program. Here is a description of the data structures:
These are the functions that we will write:
We can now begin to write the code.
In this section, we'll write the data structures for storing the USB device list and USB details and for custom error handling. We'll also write a few utility functions:
cargo new usb && cd usb
[dependencies]
libusb = "0.3.0"
Here are the module imports:
use libusb::{Context, Device, DeviceHandle};
use std::fs::File;
use std::io::Write;
use std::time::Duration;
use std::fmt;
We're importing the libusb modules and a few modules from the Rust Standard Library. fs::File and io::Write are for writing to an output file, result::Result is the return value from the functions, and time::Duration is for working with the libusb library.
#[derive(Debug)]
struct USBError {
err: String,
}
struct USBList {
list: Vec<USBDetails>,
}
#[derive(Debug)]
struct USBDetails {
manufacturer: String,
product: String,
serial_number: String,
bus_number: u8,
device_address: u8,
vendor_id: u16,
product_id: u16,
maj_device_version: u8,
min_device_version: u8,
}
USBError is for custom error handling, USBList is to store a list of the USB devices detected, and USBDetails is to capture the list of details for each USB device.
impl fmt::Display for USBList {
fn fmt(&self, f: &mut fmt::Formatter<'_>) ->
fmt::Result {
Ok(for usb in &self.list {
writeln!(f, " USB Device details")?;
writeln!(f, "Manufacturer: {}",
usb.manufacturer)?;
writeln!(f, "Product: {}", usb.product)?;
writeln!(f, "Serial number: {}",
usb.serial_number)?;
writeln!(f, "Bus number: {}",
usb.bus_number)?;
writeln!(f, "Device address: {}",
usb.device_address)?;
writeln!(f, "Vendor Id: {}",
usb.vendor_id)?;
writeln!(f, "Product Id: {}",
usb.product_id)?;
writeln!(f, "Major device version: {}",
usb.maj_device_version)?;
writeln!(f, "Minor device version: {}",
usb.min_device_version)?;
})
}
}
impl From<libusb::Error> for USBError {
fn from(_e: libusb::Error) -> Self {
USBError {
err: "Error in accessing USB
device".to_string(),
}
}
}
impl From<std::io::Error> for USBError {
fn from(e: std::io::Error) -> Self {
USBError { err: e.to_string() }
}
}
//Function to write details to output file
fn write_to_file(usb: USBList) -> Result<(), USBError> {
let mut file_handle = File::create
("usb_details.txt")?;
write!(file_handle, "{} ", usb)?;
Ok(())
}
We can now move on to the main() function.
In this section, we'll write the main() function, which sets up the device context, gets a list of connected USB devices, and then iterates through the list of devices to retrieve the details of each device. We'll also write a function to print out the device details:
fn main() -> Result<(), USBError> {
// Get libusb context
let context = Context::new()?;
//Get list of devices
let mut device_list = USBList { list: vec![] };
for device in context.devices()?.iter() {
let device_desc = device.device_descriptor()?;
let device_handle = context
.open_device_with_vid_pid(
device_desc.vendor_id(),
device_desc.product_id())
.unwrap();
// For each USB device, get the information
let usb_details = get_device_information(
device, &device_handle)?;
device_list.list.push(usb_details);
}
println!(" {}", device_list);
write_to_file(device_list)?;
Ok(())
}
In the main() function, we're first creating a new libusb Context that can return the list of connected devices. We are then iterating through the device list obtained from the Context struct, and calling the get_device_information() function for each USB device. The details are finally also printed out to an output file by calling the write_to_file() function that we saw earlier.
// Function to print device information
fn get_device_information(device: Device, handle:
&DeviceHandle) -> Result<USBDetails, USBError> {
let device_descriptor =
device.device_descriptor()?;
let timeout = Duration::from_secs(1);
let languages = handle.read_languages(timeout)?;
let language = languages[0];
// Get device manufacturer name
let manufacturer =
handle.read_manufacturer_string(
language, &device_descriptor, timeout)?;
// Get device USB product name
let product = handle.read_product_string(
language, &device_descriptor, timeout)?;
//Get product serial number
let product_serial_number =
match handle.read_serial_number_string(
language, &device_descriptor, timeout) {
Ok(s) => s,
Err(_) => "Not available".into(),
};
// Populate the USBDetails struct
Ok(USBDetails {
manufacturer,
product,
serial_number: product_serial_number,
bus_number: device.bus_number(),
device_address: device.address(),
vendor_id: device_descriptor.vendor_id(),
product_id: device_descriptor.product_id(),
maj_device_version:
device_descriptor.device_version().0,
min_device_version:
device_descriptor.device_version().1,
})
}
This concludes the code. Make sure to plug in a USB device (such as a thumb drive) to the computer. Run the code with cargo run. You should see the list of attached USB devices printed to the terminal, and also written to the output usb_details.txt file.
Note that in this example, we have demonstrated how to do file I/O using both an external crate (for retrieving USB device details) and the standard library (for writing to an output file). We've unified error handling using a common error handling struct, and automated conversions of error types to this custom error type.
The Rust crates ecosystem (crates.io) has similar crates to interact with other types of devices and filesystems. You can experiment with them.
This concludes the section on writing a program to retrieve USB details.
In this chapter, we reviewed the foundational concepts of device management in Unix/Linux. We looked at how to do buffered reads and writes using the std::io module. We then learned how to interact with the standard input, standard output, and standard error streams of a process. We also saw how to chain readers together and use iterators for reading from devices. We then looked at the error handling features with the std::io module. We concluded with a project to detect the list of connected USB devices and printed out the details of each USB device both to the terminal and to an output file.
The Rust Standard Library provides a clean layer of abstraction for doing I/O operations on any type of device. This encourages the Rust ecosystem to implement these standard interfaces for any type of device, enabling Rust system programmers to interact with different devices in a uniform manner. Continuing on the topic of I/O, in the next chapter, we will learn how to do network I/O operations using the Rust Standard Library.
52.14.121.242