3.1 Downloading a File

To give us a place to start, rather than creating a long file by hand (which is cumbersome) we’ll download a file from the Internet using the powerful curl utility. Sometimes written as “cURL”, the curl program allows us to interact with a URL¹at the command line. Although it’s not part of the core Unix command set, the curl command is widely available for installation on Unix systems. To make sure it’s available on your system, we can use the which command, which looks to see if the given program is available at the command line.² The way to use it is to type which followed by the name of the program—in this case, curl:

1. URL is short for Uniform Resource Locator, and in practice usually just means “web address”.

2. Technically, which locates a file on the user’s path, which is a list of directories where executable programs are located.

$ which curl
/usr/bin/curl

I’ve shown the output on my system (/usr/bin/curl, usually read as “user bin curl”), but the result on your system may differ. In particular, if the result is just a blank line, you will have to install curl, which you can do by Googling for “install curl” followed by the name of your operating system (e.g., “install curl macos”). (This sort of “Google for it” installation step is classic technical sophistication (Box 1.4).)

Once curl is installed, we can use the command in Listing 3.1 to download a file called sonnets.txt, which contains a large corpus of text.³

3. If for any reason using curl fails, you can always visit the URL in a browser and then use the File > Save As feature to save it to your local disk.

Listing 3.1: Using curl to download a longer file.

Click here to view code image

$ curl -OL https://cdn.learnenough.com/sonnets.txt
$ ls -rtl

Be sure to copy the command exactly; in particular, note that the option -OL contains a capital letter “O” (O) and not a zero (0). (Figuring out what these options do is left as an exercise (Section 3.5.1).) Also, on some systems (for mysterious reasons) you might have to run the command twice to get it to work; by inspecting the results of ls -rtl, you should be able to tell if the initial call to curl created the file sonnets.txt as expected. (If you do have to repeat the curl command, you could press up arrow twice to retrieve it, but see Box 3.1 for alternatives.)

The result of running Listing 3.1 is sonnets.txt, a file containing all 154 of Shakespeare’s sonnets. This file contains 2620 lines, far too many to fit on one screen. Learning how to inspect its contents is the goal of the rest of this chapter. (Among other things, we’ll learn how to determine that it has 2620 lines without counting them all by hand.)

3.2 Making Heads and Tails of It

Two complementary commands for inspecting files are head and tail, which respectively allow us to view the beginning (head) and end (tail) of the file. The head command shows the first 10 lines of the file (Listing 3.2).

Listing 3.2: Looking at the head of the sample text file.

Click here to view code image

$ head sonnets.txt
Shake-speare's Sonnets

I

From fairest creatures we desire increase,
That thereby beauty's Rose might never die,
But as the riper should by time decease,
His tender heir might bear his memory:
But thou contracted to thine own bright eyes,
Feed'st thy light's flame with self-substantial fuel

Similarly, tail shows the last 10 lines of the file (Listing 3.3).

Listing 3.3: Looking at the tail of the sample text file.

Click here to view code image

$ tail sonnets.txt
The fairest votary took up that fire
Which many legions of true hearts had warm'd;
And so the general of hot desire
Was, sleeping, by a virgin hand disarm'd.
This brand she quenched in a cool well by,
Which from Love's fire took heat perpetual,
Growing a bath and healthful remedy,
For men diseas'd; but I, my mistress' thrall,
 Came there for cure and this by that I prove,
 Love's fire heats water, water cools not love.

These two commands are useful when (as is often the case) you know for sure you only need to inspect the beginning or end of a file.

$ wc sonnets.txt
  2620  17670  95635 sonnets.txt

$ head sonnets.txt > sonnets_head.txt
$ wc sonnets_head.txt
   10   46   294 sonnets_head.txt

$ head sonnets.txt | wc
   10   46   294

3.3 Less Is More

Unix provides two utilities for the common task of wanting to look at more than just the head or tail of a file. The older of these programs is called more, but (I’d guess initially as a tongue-in-cheek joke) there’s a more powerful variant called less.⁵ The less program is interactive, so it’s hard to capture in print, but here’s roughly what it looks like:

5. On some systems, apparently they’re exactly the same program, so less really is more (or, more accurately, more is less).

Click here to view code image

$ less sonnets.txt
Shake-speare's Sonnets

I

From fairest creatures we desire increase,
That thereby beauty's Rose might never die,
But as the riper should by time decease,
His tender heir might bear his memory:
But thou contracted to thine own bright eyes,
Feed'st thy light's flame with self-substantial fuel,
Making a famine where abundance lies,
Thy self thy foe, to thy sweet self too cruel:
Thou that art now the world's fresh ornament,
And only herald to the gaudy spring,
Within thine own bud buriest thy content,
And tender churl mak'st waste in niggarding:
 Pity the world, or else this glutton be,
 To eat the world's due, by the grave and thee.

II

When forty winters shall besiege thy brow,
And dig deep trenches in thy beauty's field,
sonnets.txt

The point of less is that it lets you navigate through the file in several useful ways, such as moving one line up or down with the arrow keys, pressing the spacebar to move a page down, and pressing ^F to move forward a page (i.e., the same as spacebar) or ^B to move back a page. To quit less, type q (for “quit”).

Perhaps the most powerful aspect of less is the forward slash key /, which lets you search through the file from beginning to end. For example, suppose we wanted to search through sonnets.txt for “rose” (Figure 3.1),⁶ one of the most frequently used images in the Sonnets.⁷ The way to do this in less is to type /rose (read “slash rose”), as shown in Listing 3.6.

6. Image courtesy of Shuang Li/Shutterstock.

7. Although Shakespeare’s sonnets are undated, most of them were probably composed during the reign of Queen Elizabeth, whose royal house adopted a rose (Figure 3.1) as its heraldic emblem. Given this context, Shakespeare’s choice of floral imagery isn’t surprising, but in fact only a few commentators on the Sonnets have noticed the seemingly obvious reference.

Listing 3.6: Searching for the string “rose” using less.

Click here to view code image

Shake-speare's Sonnets

I

From fairest creatures we desire increase,
That thereby beauty's Rose might never die,
But as the riper should by time decease,
His tender heir might bear his memory:
But thou contracted to thine own bright eyes,
Feed'st thy light's flame with self-substantial fuel,
Making a famine where abundance lies,
Thy self thy foe, to thy sweet self too cruel:
Thou that art now the world's fresh ornament,
And only herald to the gaudy spring,
Within thine own bud buriest thy content,
And tender churl mak'st waste in niggarding:
 Pity the world, or else this glutton be,
 To eat the world's due, by the grave and thee.

II

When forty winters shall besiege thy brow,
And dig deep trenches in thy beauty's field,
/rose

The result of pressing return after typing /rose in Listing 3.6 is to highlight the first occurrence of “rose” in the file. You can then press n to navigate to the next match, or N to navigate to the previous match.

The last two essential less commands are G to move to the end of the file and 1G (that’s 1 followed by G) to move back to the beginning. Table 3.1 summarizes what are in my view the most important key combinations (i.e., the ones I think you need to be dangerous), but if you’re curious you can find a longer list of commands at the Wikipedia page on less.

I encourage you to get in the habit of using less as your go-to utility for looking at the contents of a file. The skills you develop have other applications as well; for example, the man pages (Section 1.4) use the same interface as less, so by learning about less you’ll get better at navigating the man pages as well.

3.4 Grepping

One of the most powerful tools for inspecting file contents is grep, which probably stands for something, but it’s not important what. (We’ll actually mention it in a moment.) Indeed, grep is frequently used as a verb, as in “You should totally grep that file.”

The most common use of grep is just to search for a substring in a file. For example, we saw in Section 3.3 how to use less to search for the string “rose” in Shakespeare’s sonnets. Using grep, we can find the references directly, as shown in Listing 3.7.

Listing 3.7: Finding the occurrences of “rose” in Shakespeare’s sonnets.

Click here to view code image

$ grep rose sonnets.txt
The rose looks fair, but fairer we it deem
As the perfumed tincture of the roses.
Die to themselves. Sweet roses do not so;
Roses of shadow, since his rose is true?
Which, like a canker in the fragrant rose,
Nor praise the deep vermilion in the rose;
The roses fearfully on thorns did stand,
 Save thou, my rose, in it thou art my all.
I have seen roses damask'd, red and white,
But no such roses see I in her cheeks;

With the command in Listing 3.7, it appears that we are in a position to count the number of lines containing references to the word “rose” by piping to wc (as in Section 3.3), as shown in Listing 3.8.

Listing 3.8: Piping the results of grep to wc.

$ grep rose sonnets.txt | wc
   10   82   419

Listing 3.8 tells us that 10 lines contain “rose” (or “roses”, since “rose” is a substring of “roses”). But you may recall from Figure 1.11 that Shakespeare’s first sonnet contains “Rose” with a capital “R”. Referring to Listing 3.7, we see that this line has in fact been missed. This is because grep is case-sensitive by default, and “rose” doesn’t match “Rose”.

As you might suspect, grep has an option to perform case-insensitive matching as well. One way to figure it out is to search through the man page for grep:

• Type man grep.

• Type /case and then return.

• Read off the result (Figure 3.2).

  

(As noted briefly in Section 1.4, the man pages use the same interface as the less command we met in Section 3.3, so we can search through them using /.)

Applying the result of the above procedure yields Listing 3.9. Comparing the results of Listing 3.9 with Listing 3.8, we see that we now have 12 matching lines instead of only 10, so there must be a total of 12 – 10 = 2 lines containing “Rose” (but not “rose”) in the Sonnets.⁸

8. Actually, “ROSE”, “RoSE”, “rOSE”, etc., all match as well, but “Rose” is the likeliest candidate. Confirming this hunch is left as an exercise (Section 3.4.1).

Listing 3.9: Doing a case-insensitive grep.

Click here to view code image

$ grep -i rose sonnets.txt | wc
   12   96   508

The grep utility gets its name from a pattern-matching system called regular expressions (also called regexes for short): grep stands for “globally search a regular expression and print.” A full treatment of regular expressions is well beyond the scope of this tutorial, but before moving on we’ll sample just a small taste.

As one simple example, let’s match every line in sonnets.txt that has a word beginning with the letters “ro”, followed by any number of (lowercase) letters, and ending in “s”. The way to represent “any letter” with a regular expression is [a-z], and following a pattern with an asterisk * matches “zero or more” of that thing. Thus, ro[a-z]*s matches “ro” and “s” with zero or more letters in between. We can add spaces to the beginning and end to ensure that the match consists of entire words, like this:

Click here to view code image

$ grep ' ro[a-z]*s ' sonnets.txt
  To that sweet thief which sourly robs from me.
Die to themselves. Sweet roses do not so;
When rocks impregnable are not so stout,
He robs thee of, and pays it thee again.
The roses fearfully on thorns did stand,
I have seen roses damask'd, red and white,
But no such roses see I in her cheeks;

We can see that the regular expression matches strings like “robs” and “rocks” in addition to “roses”.

In general, one of the best tools for learning how to use regexes is an online regex builder, such as regex101, which lets you build up regexes interactively (Figure 3.3). Unfortunately, grep often doesn’t support the precise format used by regex builders (including hard-to-guess requirements for “escaping out” special characters), and precision in regular expressions is everything. As a result, despite its name origins, the truth is I rarely use the regular expression capabilities of grep. By the time the situation calls for regexes, I’m far likelier to reach for a text editor (Part II) or a full-strength programming language (Learn Enough JavaScript to Be Dangerous (https://www.learnenough.com/javascript), Learn Enough Ruby to Be Dangerous (https://www.learnenough.com/ruby)).

Nevertheless, the aspects of grep discussed in this section are nearly enough to be dangerous, covering a huge number of common cases (including the important application of grepping processes (Box 3.2)). We’ll see one final grep variant in Chapter 4 as part of our discussion of Unix directories.

3.5 Summary

Important commands from this chapter are summarized in Table 3.2.