Iterating the Files

Now to work on the counting. You might start by processing each file, counting the various bits, and printing the desired columns. I suggest you once again use the open function from Chapter 2 for opening the files:

fn open(filename: &str) -> MyResult<Box<dyn BufRead>> {
    match filename {
        "-" => Ok(Box::new(BufReader::new(io::stdin()))),
        _ => Ok(Box::new(BufReader::new(File::open(filename)?))),
    }
}

Be sure to expand your imports to the following:

use clap::{App, Arg};
use std::error::Error;
use std::fs::File;
use std::io::{self, BufRead, BufReader};

Here is a run function to get you going:

pub fn run(config: Config) -> MyResult<()> {
    for filename in &config.files {
        match open(filename) {
            Err(err) => eprintln!("{}: {}", filename, err),
            Ok(_file) => println!("Opened {}", filename),
        }
    }

    Ok(())
}

Using your open filehandle, start as simply as possible using the empty file and make sure your program prints zeros for the three columns of lines, words, and bytes:

$ cargo run -- tests/inputs/empty.txt
       0       0       0 tests/inputs/empty.txt

Next use tests/inputs/fox.txt and make sure you get the following counts. I specifically added various kinds and numbers of whitespace to challenge you on how to split the text into words:

$ cargo run -- tests/inputs/fox.txt
       1       9      48 tests/inputs/fox.txt

Be sure your program can handle the Unicode in tests/inputs/atlamal.txt correctly:

$ cargo run -- tests/inputs/atlamal.txt
       4      29     177 tests/inputs/atlamal.txt

And that you correctly count the characters:

$ cargo run -- tests/inputs/atlamal.txt -wml
       4      29     159 tests/inputs/atlamal.txt

When you can handle any one file, use more than one to check that you print the correct total column:

$ cargo run -- tests/inputs/*.txt
       4      29     177 tests/inputs/atlamal.txt
       0       0       0 tests/inputs/empty.txt
       1       9      48 tests/inputs/fox.txt
       5      38     225 total

When all that works correctly, try reading from STDIN:

$ cat tests/inputs/atlamal.txt | cargo run
       4      29     177

Run cargo test often to see how you’re progressing. Don’t read ahead until you pass all the tests.

Counting the Elements of a File or STDIN

I suggested you start by iterating the filenames and printing the counts for the file. To that end, I decided to create a function called count that would take a filehandle and possibly return a struct called FileInfo containing the number of lines, words, bytes, and characters each represented as a usize. I say that the function will possibly return this struct because the function will involve IO, which could go sideways. I place the following definition just after the Config struct. For reasons I will explain shortly, this must derive the PartialEq trait in addition to Debug:

#[derive(Debug, PartialEq)]
pub struct FileInfo {
    num_lines: usize,
    num_words: usize,
    num_bytes: usize,
    num_chars: usize,
}

To represent this the function might succeed or fail, it will return a MyResult<FileInfo> meaning that on success it will have Ok<FileInfo> and on failure it will have an Err. To start this function, I will initialize some mutable variables to count all the elements and will return a FileInfo struct:

pub fn count(mut file: impl BufRead) -> MyResult<FileInfo> { 
    let mut num_lines = 0; 
    let mut num_words = 0;
    let mut num_bytes = 0;
    let mut num_chars = 0;

    Ok(FileInfo {
        num_lines, 
        num_words,
        num_bytes,
        num_chars,
    })
}

The count function will accept a mutable file value, and it might return a FileInfo struct.

Initialize mutable variables to count the lines, words, bytes, and characters.

For now, return a FileInfo with all zeros.

In Chapter 3, I showed how to write a unit test, placing it just after the function it was testing. I’m going to create a unit test for the count function, but this time I’m going to place it inside a module called tests. This is a tidy way to group unit tests, and I can use a configuration option that tells Rust to only compile the module during testing. This is especially useful because I want to use std::io::Cursor in my test to make a fake filehandle for the count function. The module will not be included when I build and run the program when not testing.

A Cursor is “used with in-memory buffers, anything implementing AsRef<[u8]>, to allow them to implement Read and/or Write, allowing these buffers to be used anywhere you might use a reader or writer that does actual I/O.” Following is how I create the tests module and then import and test the count function:

#[cfg(test)] 
mod tests { 
    use super::{count, FileInfo}; 
    use std::io::Cursor; 

    #[test]
    fn test_count() {
        let text = "I don't want the world. I just want your half.
";
        let info = count(Cursor::new(text)); 
        assert!(info.is_ok()); 
        let expected = FileInfo {
            num_lines: 1,
            num_words: 10,
            num_chars: 48,
            num_bytes: 48,
        };
        assert_eq!(info.unwrap(), expected); 
    }
}

The cfg enables conditional compilation, so this module will only be compiled when testing.

Define a new module (mod) called tests to contain test code.

Import the count function and FileInfo struct from the parent module super, meaning next above or higher and refers to the module above tests that contains it.

Import std::io::Cursor.

Run count with the Cursor, which implements BufRead.

Ensure the result is Ok.

Compare the result to the expected value. This comparison requires FileInfo to derive PartialEq.

Run this test using cargo test test_count. You will see lots of warnings from the Rust compiler about unused variables or variables that do not need to be mutable. The most important result is that the test fails:

failures:

---- tests::test_count stdout ----
thread 'tests::test_count' panicked at 'assertion failed: `(left == right)`
  left: `FileInfo { num_lines: 0, num_words: 0, num_bytes: 0, num_chars: 0 }`,
 right: `FileInfo { num_lines: 1, num_words: 10, num_bytes: 48,
 num_chars: 48 }`', src/lib.rs:146:9

Take some time to write the rest of the count function that will pass this test. I will take my squeaky clean dog for a walk and maybe have some tea.

OK, we’re back. Now I’ll show you how I wrote my count function. I know from Chapter 3 that BufRead::lines will remove the line endings, and I don’t want that because newlines in Windows files are two bytes ( ) but Unix newlines are just one byte ( ). I can copy some code from Chapter 3 that uses BufRead::read_line instead to read each line into a buffer. Conveniently, this function tells me how many bytes have been read from the file:

pub fn count(mut file: impl BufRead) -> MyResult<FileInfo> {
    let mut num_lines = 0;
    let mut num_words = 0;
    let mut num_bytes = 0;
    let mut num_chars = 0;
    let mut line = String::new(); 

    loop { 
        let line_bytes = file.read_line(&mut line)?; 
        if line_bytes == 0 { 
            break;
        }
        num_bytes += line_bytes; 
        num_lines += 1; 
        num_words += line.split_whitespace().count(); 
        num_chars += line.chars().count(); 
        line.clear(); 
    }

    Ok(FileInfo {
        num_lines,
        num_words,
        num_bytes,
        num_chars,
    })
}

Create a mutable buffer to hold each line of text.

Create an infinite loop for reading the filehandle.

Try to read a line from the filehandle.

End of file (EOF) has been reached when zero bytes are read, so break out of the loop.

Add the number of bytes from this line to the num_bytes variable.

Each time through the loop is a line, so increment num_lines.

Use the str::split_whitespace method to break the string on whitespace and use Iterator::count to find the number of words.

Use the str::chars method to break the string into Unicode characters and use Iterator::count to count the characters.

Clear the line buffer for the next line of text.

With these changes, test_count passes. I’d like to see how this looks, so I can change run to print the results of successfully counting the elements of the file or print a warning to STDERR when the file can’t be opened:

pub fn run(config: Config) -> MyResult<()> {
    for filename in &config.files {
        match open(filename) {
            Err(err) => eprintln!("{}: {}", filename, err),
            Ok(file) => {
                if let Ok(info) = count(file) { 
                    println!("{:?}", info); 
                }
            }
        }
    }

    Ok(())
}

Attempt to get the counts from a file.

Print the counts.

When I run it on one of the test inputs, it appears to work for a valid file:

$ cargo run -- tests/inputs/fox.txt
FileInfo { num_lines: 1, num_words: 9, num_bytes: 48, num_chars: 48 }

It even handles reading from STDIN:

$ cat tests/inputs/fox.txt | cargo run
FileInfo { num_lines: 1, num_words: 9, num_bytes: 48, num_chars: 48 }

Now to make the output meet the specifications.

Formatting the Output

To create the expected output, I can start by changing run to always print the lines, words, and bytes followed by the filename:

pub fn run(config: Config) -> MyResult<()> {
    for filename in &config.files {
        match open(filename) {
            Err(err) => eprintln!("{}: {}", filename, err),
            Ok(file) => {
                if let Ok(info) = count(file) {
                    println!(
                        "{:>8}{:>8}{:>8} {}", 
                        info.num_lines,
                        info.num_words,
                        info.num_bytes,
                        filename
                    );
                }
            }
        }
    }

    Ok(())
}

Format the number of lines, words, and bytes into a right-justified field 8 characters wide.

If I run it with one input file, it’s already looking pretty sweet:

$ cargo run -- tests/inputs/fox.txt
       1       9      48 tests/inputs/fox.txt

If I run cargo test fox, I pass one out of eight tests. Huzzah!

running 8 tests
test fox ... ok
test fox_bytes ... FAILED
test fox_chars ... FAILED
test fox_bytes_lines ... FAILED
test fox_words_bytes ... FAILED
test fox_words ... FAILED
test fox_words_lines ... FAILED
test fox_lines ... FAILED

Inspect tests/cli.rs to see what the passing test looks like. Note that the tests reference constant values declared at the top of the module:

const PRG: &str = "wcr";
const EMPTY: &str = "tests/inputs/empty.txt";
const FOX: &str = "tests/inputs/fox.txt";
const ATLAMAL: &str = "tests/inputs/atlamal.txt";

Again I have a run helper function to run my tests:

fn run(args: &[&str], expected_file: &str) -> TestResult {
    let expected = fs::read_to_string(expected_file)?; 
    Command::cargo_bin(PRG)? 
        .args(args)
        .assert()
        .success()
        .stdout(expected);
    Ok(())
}

Try to read the expected output for this command.

Run the wcr program with the given arguments. Assert that the program succeeds and that STDOUT matches the expected value.

The fox test is running wcr with the FOX input file and no options, comparing it to the contents of the expected output file which was generated using 05_wcr/mk-outs.sh:

#[test]
fn fox() -> TestResult {
    run(&[FOX], "tests/expected/fox.txt.out")
}

Look at the next function in the file to see a failing test:

#[test]
fn fox_bytes() -> TestResult {
    run(&["--bytes", FOX], "tests/expected/fox.txt.c.out") 
}

Run the wcr program with the same input file and the --bytes option.

When run with --bytes, my program should only print that column of output, but it always prints lines, words, and bytes. I decided to write a function called format_field in src/lib.rs that would conditionally return a formatted string or the empty string depending on a Boolean value:

fn format_field(value: usize, show: bool) -> String { 
    if show { 
        format!("{:>8}", value) 
    } else {
        "".to_string() 
    }
}

The function accepts a usize value and a Boolean and returns a String.

Check if the show value is true.

Return a new string by formatting the number into a string 8 characters wide.

Otherwise, return the empty string.

I can expand my tests module to add a unit test for this:

#[cfg(test)]
mod tests {
    use super::{count, format_field, FileInfo}; 
    use std::io::Cursor;

    #[test]
    fn test_count() {} // Same as before

    #[test]
    fn test_format_field() {
        assert_eq!(format_field(1, false), ""); 
        assert_eq!(format_field(3, true), "       3"); 
        assert_eq!(format_field(10, true), "      10"); 
    }
}

Add format_field to the imports.

The function should return the empty string when show is false.

Check width for a single-digit number.

Check width for a double-digit number.

Here is how I use it in context where I also handle printing the empty string when reading from STDIN:

pub fn run(config: Config) -> MyResult<()> {
    for filename in &config.files {
        match open(filename) {
            Err(err) => eprintln!("{}: {}", filename, err),
            Ok(file) => {
                if let Ok(info) = count(file) {
                    println!(
                        "{}{}{}{}{}", 
                        format_field(info.num_lines, config.lines),
                        format_field(info.num_words, config.words),
                        format_field(info.num_bytes, config.bytes),
                        format_field(info.num_chars, config.chars),
                        if filename.as_str() == "-" { 
                            "".to_string()
                        } else {
                            format!(" {}", filename)
                        }
                    );
                }
            }
        }
    }

    Ok(())
}

Format the output for each of the columns using the format_field function.

When the filename is “-”, print the empty string; otherwise, print a space and the filename.

With these changes, all the tests for cargo test fox pass. If I run the entire test suite, I’m still failing the all tests:

failures:
    test_all
    test_all_bytes
    test_all_bytes_lines
    test_all_lines
    test_all_words
    test_all_words_bytes
    test_all_words_lines

Look at the all function in tests/cli.rs to see that the test is using all the input files as arguments:

#[test]
fn all() -> TestResult {
    run(&[EMPTY, FOX, ATLAMAL], "tests/expected/all.out")
}

If I run my current program with all the input files, I can see that I’m missing the total line:

$ cargo run -- tests/inputs/*.txt
       4      29     177 tests/inputs/atlamal.txt
       0       0       0 tests/inputs/empty.txt
       1       9      48 tests/inputs/fox.txt

Here is my final run function that keeps a running total and prints those values when there is more than one input:

pub fn run(config: Config) -> MyResult<()> {
    let mut total_lines = 0; 
    let mut total_words = 0;
    let mut total_bytes = 0;
    let mut total_chars = 0;

    for filename in &config.files {
        match open(filename) {
            Err(err) => eprintln!("{}: {}", filename, err),
            Ok(file) => {
                if let Ok(info) = count(file) {
                    println!(
                        "{}{}{}{}{}",
                        format_field(info.num_lines, config.lines),
                        format_field(info.num_words, config.words),
                        format_field(info.num_bytes, config.bytes),
                        format_field(info.num_chars, config.chars),
                        if filename.as_str() == "-" {
                            "".to_string()
                        } else {
                            format!(" {}", filename)
                        }
                    );

                    total_lines += info.num_lines; 
                    total_words += info.num_words;
                    total_bytes += info.num_bytes;
                    total_chars += info.num_chars;
                }
            }
        }
    }

    if config.files.len() > 1 { 
        println!(
            "{}{}{}{} total",
            format_field(total_lines, config.lines),
            format_field(total_words, config.words),
            format_field(total_bytes, config.bytes),
            format_field(total_chars, config.chars)
        );
    }

    Ok(())
}

Create mutable variables to track the total number of lines, words, bytes, and characters.

Update the totals using the values from this file.

Print the totals if there is more than one input.

This appears to work well:

$ cargo run -- tests/inputs/*.txt
       4      29     177 tests/inputs/atlamal.txt
       0       0       0 tests/inputs/empty.txt
       1       9      48 tests/inputs/fox.txt
       5      38     225 totalcargo run -- -m tests/inputs/atlamal.txt

I can count characters instead of bytes:

$ cargo run -- -m tests/inputs/atlamal.txt
     159 tests/inputs/atlamal.txt

I can show and hide any columns I want:

$ cargo run -- -wc tests/inputs/atlamal.txt
      29     177 tests/inputs/atlamal.txt

Most importantly, cargo test shows all passing tests.