Instead of using a mouse-driven interface to press buttons and fill out text fields, Unix-like server operating systems are driven by command interpreters or shells, text prompts that accept commands and execute them one at a time. As a programmer, you’re likely familiar with programming language expressions: print a string, call a function with some parameters, and so on. The command line is much the same.

Following shortly is a simple example wherein we list the contents of our current directory (Windows calls these “folders”), list the contents of a subdirectory, and remove a file in that subdirectory. Note the $ character is the prompt, and featured in the following examples is a simple prompt denoting which lines are commands being entered and which lines are output from those commands.

Other shells can use different characters—besides $, you can also see > or %. (The Python interpreter shell has a prompt of >>>.) In addition to the character(s) appearing immediately before the user input, many prompts have extra info such as your current username, host, or directory (as do some examples in this book, which utilize user@example $).

Here’s a quick example where we list the contents of the current directory, and then remove a file in a subdirectory.

$ ls
documents  code  temp
$ ls documents
test.py
$ rm documents/test.py
$

Both commands used previously are programs, or binaries, located somewhere in the current executable path. (See the next section for details on paths.) Although programs can theoretically exhibit any behavior, there are established standards for how to specify arguments and options. Traditionally, a Unix command consists of up to three parts: the command’s name, options controlling how the command behaves, and arguments, which specify the subcommands to run, files to operate on, and so forth.

Taking the last command from the previous example, rm is the program name (rm standing for “remove”), and documents/test.py is an argument, specifically the file to remove. If we want to remove an entire directory, we can pass options to rm to control its behavior, like we do here:

$ ls temp
tempfile1  tempfile2
$ rm temp
rm: cannot remove 'temp': Is a directory
$ rm --help
Usage: /bin/rm [OPTION]... FILE...
Remove (unlink) the FILE(s).

  -d, -directory       unlink FILE, even if it is a non-empty directory
  -f, -force           ignore nonexistent files, never prompt
  -i, -interactive     prompt before any removal
      -no-preserve-root do not treat '/' specially (the default)
      -preserve-root   fail to operate recursively on '/'
  -r, -R, --recursive   remove the contents of directories recursively
  -v, --verbose         explain what is being done
      --help     display this help and exit
      --version  output version information and exit
$ rm -rf temp
$

As a general rule, rm cannot remove directories, so that’s why we weren’t able to use it in this situation. However, passing the -r and -f options—combined into a single option string for convenience (see the following “Options and Arguments” section for more)—enables rm to recursively remove directories, no questions asked, and so it removes the temp directory without further trouble.

As you can see from the previous example, programs usually contain built-in help information about what their options and arguments are. Almost every program on a Unix system accepts the --help or -h options, which results in some sort of help message. These generally contain enough info for anyone, novice or expert, to get some use out of the program.

If the built-in help isn’t good enough, or you need more details on a given option, there’s the man system (short for “[user] manual”), which provides a full set of information about each command, often explaining its arguments in greater detail and/or giving usage examples. man itself is, of course, a program like all the others, and it generally takes a single argument—the name of the program whose “man page” (short for “manual page”) you want to read. One command often thrown at novice Unix users is man man, which of course is the man page for the man command itself.

For better or worse, Unix-based systems are often geared toward the self-learner; because you’re obviously not averse to reading, however, the authors are sure you’ll do fine! And in all seriousness, you’ll do yourself a favor if you form such a habit: Those with experience are known to be rather rude to newer folks who ask questions without looking for the answer first.

We presented previous program options and arguments as being two distinct aspects of the program’s specification, but this is not entirely true. At their core, options and arguments are simply a long string presented to the program, which it can interpret any way it pleases. Because of this, the standards presented here are simply that—standards—and variations on them are commonly found, depending on how strictly the program’s author wants to adhere to the norm.

In general, all arguments or options to a program are delimited by spaces with options usually being prefixed by a hyphen or dash character, -, and appearing before arguments, which have no prefix. Some programs also accept a so-called “long option” format, which typically uses two dashes and more than one character for the option name—such as the --help option we mentioned previously.

$ rm --help
Usage: rm [OPTION]... FILE...

This is the standard: utility/program name, followed by zero or more options of either type, followed by zero or more arguments (as some commands can sometimes take no arguments at all). Options can be specified one at a time separated by spaces:

$ rm -r -f temp

but they can also be combined into a single option string to save typing, as we did in the earlier example:

$ rm -rf temp

Note you cannot combine long and short options in this way, as that wouldn’t make much sense, but they can be otherwise interleaved:

$ rm -rf --verbose temp

Options can themselves be parameterized with arguments, depending on the option involved. For example, the head program is intended to return only the top few lines of a given file or chunk of text; how many lines it returns is controlled with the -n argument, such as the following example, which returns the top five lines of the file myfile.txt:

$ head -n 5 myfile.txt

By default (with any -n option), head displays the first ten lines. Note that, as with the combination of multiple options into one string, option arguments do not have to be separated from their option with a space, but can be combined as one string.

$ head -n5 myfile.txt

Finally, although traditional Unix programs tend to accept a strict ordering of <program> <options> <arguments>, many Linux applications are more lenient and enable options to be specified at any point, such as:

$ head myfile.txt -n5

Or:

$ rm -rf temp --verbose

Although this tendency of Linux applications is very convenient (such as when one types a command and only at the end remembers an option he forgot), the authors recommend you try to become used to the more strict form found on true Unix systems. Otherwise you find yourself on a FreeBSD or Mac OS X machine, constantly tripped up by the programs’ complaints about your argument order. Trust us, we’ve been there!

The examples here have been kept simple for purposes of illustration, but if you look at examples of command-line program usage found all over the Web (or check out various man pages and --help outputs), you see that command-line programs provide an astounding amount of power and flexibility in terms of altering their behavior. And we’re not done yet—the next section introduces a whole new dimension of how the Unix command line works.

By their nature, command lines deal almost exclusively with text, both for input and output. However, in addition to the input and output from and to the user, Unix programs also communicate between themselves and files on disk via an input/output abstraction known as pipes. As implied by their name, pipes are a mechanism for directing the flow of text between various combinations of the terminal a user is interacting with, programs, and files.

Every Unix program deals with three potential types of output and input: input, regular output, and error-related output. When nothing special is going on, programs interface with the so-called “standard” pipes, which end up pointing to the text terminal the user is viewing. For example, when you use the cat command (short for “concatenate”) to spit out the contents of a file, what happens is that cat opens the file or files in its arguments and puts their contents into the stdout stream, such as in this example where we cat the contents of a grocery list.

$ cat groceries.txt
Milk
Canned corn
Peanut butter
Can of soup
Powdered milk

Here, because we are just running cat by itself, stdout dumps the text of the grocery list to our terminal. If cat ran into an error condition, such as being given the name of a file that didn’t exist, it would print out an error message to the stderr stream, which also goes directly to the user by default.

$ cat foo.txt
cat: foo.txt: No such file or directory

The neat thing about the abstraction of pipes is when we depart from the normal scheme of things and use what’s known as the pipe operator, |, to tell the command shell to redirect the stdout of one program into the stdin stream of another. stdin, the third type of program I/O, is of course short for “standard input.” Many programs accept text from stdin in addition to, or instead of, expecting the user to give them the names of files to read.

By way of example, let’s revisit the use of head with relation to our grocery list and ascertain what the first item on our list is.

$ cat groceries.txt | head -n1
Milk

Notice how we’re not telling head what file to deal with and instead use the pipe character to redirect the output of the cat command into the head command. The result is the same as if we had passed the filename to head directly.

$ head -n1 groceries.txt
Milk

A more realistic example can use the grep utility, which among other things can be used to return only lines matching a given regular expression (see Chapter 1, “Practical Python for Django,” for information on those). Let’s use pipes to take the output of a grep command, which filters (in a case-insensitive fashion) our grocery list for items with the word “can” in their name:

$ grep -i "can" groceries.txt
Canned corn
Can of soup

and then use head to pare the results down to just the first item.

$ grep -i "can" groceries.txt | head -n1
Canned corn

As mentioned previously, we can use more than one pipe in a single command. Let’s use grep’s sister command, sed, to do a replacement of the word “corn” with the more generic “veggies.”

$ grep -i "can" groceries.txt | head -n1 | sed -e "s/corn/veggies/"
Canned veggies

Finally, as mentioned, you can redirect these text streams to and from files using the > and < characters. Keeping with the pipe notation of operating from left to right, > is used for directing stdout to a file, and < for redirecting a file to stdin. For example, in the previous example we’ve done a search and replace, albeit a not terribly useful one, but once displayed on the terminal, our hard work is gone (short of copying and pasting, of course). Let’s redirect it to a new file.

$ grep -i "can" groceries.txt | head -n1 | sed -e "s/corn/veggies/" > filtered.txt

This command creates a new file (or overwrites an existing file—so be careful!) named filtered.txt, and it contains the line “Canned veggies.” Note the command produces no output to the terminal—that’s because we redirected stdout to the file, and because it’s been redirected, we don’t get to see it ourselves.

Finally, note you can double up the redirect-to-file character (>>) to append to an existing file instead of overwriting it; this, like >, creates a new file if none previously existed.

The command-line shell has what’s known as an environment, or namespace, just like a program’s namespace in Python or any other language, wherein various strings can be bound to variable names and referred to by the user when executing commands, or even by the commands themselves (which have access to the calling user’s environment). The env command prints out the current state of the environment, such as:

$ env
TERM=linux
SHELL=/bin/bash
USER=user
PATH=/usr/local/bin:/usr/bin:/bin:/usr/games:~/bin
PWD=/home/user
EDITOR=vim
HOME=/home/user

Many environment variables are used by common Unix shell utilities or by the shell itself—for example, the EDITOR variable is used by programs such as Subversion, which attempts to call the named program when they need to dump the user into a text editor temporarily. The TERM variable determines the terminal type, which many programs reference when deciding whether to use color output or how to interpret keystrokes; PWD is the current directory; and so forth.

As in Python and other languages, you can alter the environment by assigning values to these variables with an assignment character. Here, we shorten the output of env for brevity.

$ env
EDITOR=vim
$ EDITOR=pico
$ env
EDITOR=pico

In this example, the default value for EDITOR is the well known vim editor, and we’ve changed it to point to a less powerful editor called pico. However, environment variables—as you might expect—are initialized at the beginning of each shell session. Our change to EDITOR is temporary at best, unless we alter our shell configuration files, which are read when the shell starts, to make the change permanent. (See your shell’s man page for details on how to do this.)

Up till now we’ve been using env to print the values of environment variables, but that’s analogous to Python’s globals function—it’s not actually very useful unless you’re troubleshooting something. The shell can automatically expand any environment variables it sees and can substitute them with their values, but only when using the $ character as a prefix. We use the echo program, which simply echoes its arguments back to the user, to demonstrate this:

$ env
EDITOR=vim
$ echo EDITOR
EDITOR
$ echo $EDITOR
vim

As you can see, echoing EDITOR by itself does nothing special—it’s just a string—but echoing $EDITOR results in printing the value of the EDITOR variable. To put things another way, the shell takes any name or expression following a $ and attempts to expand it into a variable value. Those of us used to languages that use $ to denote variables for both assignment and expansion, generally spend a while making mistakes such as this:

$ $EDITOR=pico
-bash: vim=pico: command not found
$ EDITOR=pico
$ echo $EDITOR
pico

A final note on this topic: Environment variables aren’t limited to simple one-word strings, but can hold any string whatsoever. As you saw in the previous example, the shell expanded $EDITOR and combined it with the rest of the line, and then tried to execute the whole as a command. That obviously didn’t work—there’s no binary called vim=pico—but it’s entirely possible to utilize this to save a bit of typing, such as in the following example, where we bind most of a command string to a variable and use it multiple times, appending the argument to the end.

$ FINDMILK="grep -ni milk"
$ $FINDMILK groceries.txt
1:Milk
5:Powdered milk
$ $FINDMILK todo.txt
1:Search grocery list for milk
$ $FINDMILK email_from_reader.txt
3:Also, what's up with all the groceries and milk examples?

Our FINDMILK variable is expanded each time, resulting in commands looking like grep -ni milk groceries.txt. However, this example is slightly contrived—in many cases such as this, what you really want to do is paramaterize an otherwise static call to a program, and in that case you’re better off writing a small shell script. For example, you could write a script that takes two arguments, not just one, and enables you to specify both the term being searched for as well as the location to search.

Details on shell scripting are outside the scope of this chapter, but your shell’s man page has plenty of information, and as with most subjects, there’s a lot of excellent material online.

One environment variable, arguably the most important, is the path, normally stored as PATH, which is a list of directories the shell looks in to find the commands you type.

$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/games:~/bin

When you type a command into your shell, the shell looks at each of the listed directories in turn until it finds an executable file with the name you asked for; at that point, it executes that file with the arguments and options you provided. So, when we type echo, for example, the shell is really executing /bin/echo, and when we type man, it finds and executes /usr/bin/man. (See the following sidebar for an explanation of these different bin directories.)

It’s possible to add things to your path to save time typing. In the previous example, the user in question has added ~/bin to their path; because ~ is a shortcut for one’s home directory, this means the user can now type the names of scripts in their personal binary directory, and they can be easily found by the shell.

Note you can easily execute programs not in the path by specifying the full or relative path when you want to execute it. The shell’s path capability is simply a handy shortcut to doing this, in fact.

$ /tmp/packagename-2.0.1/bin/program

Because /tmp/packagename-2.0.1/bin/ (a potential place for binaries if you’re trying out a program not fully installed yet) is not likely to be in your PATH, you have to tell the shell exactly where to find it. In fact, the entire concept of paths is simply a shortcut, albeit a very pervasive one.

Finally, keep in mind you want the containing directory of your binaries in your PATH and not the binaries themselves—the same is true for other paths such as Python’s own module path (see Chapter 1). Think of a path as a list of containers and not a list of things to reference.

You’ve learned a lot already by this point: how to execute programs with various kinds of options and arguments, making those programs work together and with files via pipes and redirection, and how environment variables and pathing can save you a lot of time.

However, there’s a lot more to the average Unix shell program than what we’ve covered here—most of them are full-fledged programming environments in their own right, including conditional statements, loops, and so forth. Once you’re comfortable navigating around a filesystem and running commands, you can find it worthwhile to explore the depths of the shell your system uses—it can save you a lot of time, just like any other programming tool.