Think of this chapter as an extended Shell Out section. That means it is time to transform you into a master or mistress of the shell. In deciding to join me here, you have identified yourself as one of the bold and curious explorers who really wants to know his Linux systems. Sure, it is possible to work day in and day out with your Linux system and rarely use the command line, but the command line is power. Your reward for continuing to this next level is a deeper understanding of your system and the power to make it do whatever you want.

The things I want to talk about here are basic commands that will serve you well throughout your time with Linux. One of the things I hope to show you is how flexible some of these commands are. With most, you can modify the basic function with command-line switches, flags, or options, and thereby have them yield far more information than a simple execution of the command itself. A little thirst for exploration will open you up to the real potential of everyday commands.

Before I wrap up this chapter, I will also cover editors. Editors aren’t a luxury or an option but an eternal necessity in the life of the system administrator. Every once in a while, you will have to edit some configuration file or other in order to get your work done. I’ll give you some pointers for making this as simple and painless as possible.

Metacharacters are special characters that have particular meaning to your shell—the dollar sign or hash mark prompt where you do your work. The two metacharacters I want to look at are the asterisk and the question mark. The following is what they mean to the shell:

*        Match any number of characters

?        Match a single character

Extending our talk of listing files, you could list all files containing ackle by using this command:

   Similarly, you could find all the words that start with an h like this:

   Now, if you want to see all the seven-letter words in your directory, use this command:

   Each question mark represents a single letter position.

In Chapter 4, I showed you how to look at file permissions with Nautilus. When you use the ls -l command, you are doing the same thing; look at basic Linux security at the file (or directory) level. Here is an example of a long ls listing:

The first entry under the total column shows a directory (I’ll talk about the next nine characters in a moment). The first character is a d, which indicates a directory. At the end of each line, you find the directory or filename—in my example, they are presents, wishlist, and griflong. Because the first character in the permissions field is d, presents is a directory.

The other nine characters (characters 2 through 10) indicate permissions for the user or owner of the file (first three), the group (second group of three), and others or everyone else (last three). In the first line, user root has read (r), write (w), and execute (x) permissions, whereas the system group has only read and execute. The three dashes at the end imply that no one else has any permissions. The next two files are owned by the user called zonthar.

When you delete a file with Linux, it is gone. If you didn’t really mean to delete (or rm) a file, you quickly find out if you have been keeping good backups. The other option is to check with the rm command before you delete a file. Rather than simply typing rm followed by the filename, try this instead:

The -i option tells rm to work in interactive mode. For each of the three files in the example, rm pauses and asks if you really mean it.

If you like to be more wordy than that, you can also try rm --interactive file_name, but that goes against the system administrator’s first principle.

Of course, in following the system administrator’s first principle, you could remove all the files starting with the word file by using the asterisk:

You might find that you want to use the -i option every time you delete something, just in case. It’s a lot easier to type Y in confirmation than it is to go looking through your backups. The problem is that you are adding keystrokes, and everyone knows that system administrators are notoriously lazy people. Then, there’s that whole issue of the first principle—that’s why we shortened list to ls, after all. Don’t despair, though—Linux has a way. It is the alias command.

Now, every time you execute the rm command, it checks with you beforehand. This behavior is only in effect until you log out. If you want this to be the default behavior for rm, you should add the alias command to your local .bashrc file. If you want this to be the behavior for every user on your system, you should add your alias definitions to the system-wide version of this file, /etc/bashrc, and save yourself even more time. There may already be alias definitions set up for you, even if only one or two. The first way to find out what has been set up for you is to type the alias command on a blank line.

Using the cat command, you can look in your local .bashrc file and discover the same information.

As you can see, there are other suggested aliases that you can uncomment by removing that leading hash mark. Incidentally, here are a few I like to add to everyone’s .bashrc file (or to the global /etc/bash.bashrc):

Isn’t this interesting? Notice the two other commands here, the cp (copy files) and mv (rename files) commands, both of which have the -i flag as well. They too can be set to work interactively, requiring your verification before you overwrite something important. Let’s say that I want to make a backup copy of a file called important_info using the cp command.

Perhaps I am actually trying to rename the file (rather than copy it). For this, I would use the mv command.

The only time you would be bothered by an “Are you sure?” type of message is if the file already existed. In that case, you would get a message like the following:

It may sound complicated, but it isn’t. Standard in (STDIN) is where the system expects to find its input. This is usually the keyboard, although it can be a program or shell script. When you change that default, you call it redirecting from STDIN.

Standard out (STDOUT) is where the system expects to direct its output, usually the terminal screen. Again, redirection of STDOUT is at the discretion of whatever command or script is executing at the time. The chain of events from STDIN to STDOUT looks something like this:

STDIN is often referred to as fd0, or file descriptor 0, whereas STDOUT is usually thought of as fd1. There is also standard error (STDERR), where the system reports any errors in program execution. By default, this is also the terminal. To redirect STDOUT, use the greater-than sign (>). As you might have guessed, to redirect from STDIN, you use the less-than sign (<). But what exactly does that mean? Let’s try an experiment. Randomly search your brain and pick a handful of names. Got them? Good. Now type the cat command and redirect its STDOUT to a file called random_names.

Your cursor just sits there and waits for you to do something, so type those names, pressing <Enter> after each one. What’s happening here is that cat is taking its input from STDIN and writing it out to your new file. You can also write the command like this:

The hyphen literally means standard in to the command. The 1 stands for file descriptor 1. This is good information, and you will use it later. Finished with your random names list? When you are done, press <Ctrl+D> to finish. <Ctrl+D>, by the way, stands for EOF, or end of file.

If you cat this file, the names are written to STDOUT—in this case, your terminal window. You can also give cat several files at the same time. For instance, you could do something like this:

Each file would be listed one right after the other. That output could then be redirected into another file. You could also have it print out the same file over and over (cat random_names random_names random_names). cat isn’t fussy about these things and deals with binary files (programs) just as quickly. Beware of using cat to print out the contents of a program to your terminal screen. At worst, your terminal session locks up or rewards you with a lot of beeping and weird characters.

Redirecting STDIN works pretty much the same way, except that you use the less-than sign instead. Using the sort command, let’s take that file of random names and work with it. Many commands that work with files can take their input directly from that file. Unless told otherwise, cat and sort will think that the word following the command is a filename. That’s why you did the STDIN redirection thing. Yes, that’s right: STDIN is just another file. Sort of.

Of course, the result is that you get all your names printed out in alphabetical order. You could have also specified that sort take its input from a redirected STDIN. It looks a bit strange, but this is perfectly valid.

One more variation involves defining your STDIN (as you did previously) and specifying a different STDOUT all on the same line. In the following example, I redirect from my file and redirect that output to a new file:

Sometimes, the thing that makes the most sense is to feed the output from one command directly into another command without having to resort to files in between at every step of the way. This is called piping. The symbolism is not that subtle: Imagine pieces of pipe connecting one command with another. Not until you run out of pipe does the command’s output emerge. The pipe symbol is the broken vertical bar on your keyboard usually located just below or (depending on the keyboard) just above the Enter key and sharing space with the backslash key. Here’s how it works:

In the preceding example, the output from the cat command is piped into sort, whose output is then piped into the wc command (that’s word count). The -w flag tells wc to count the number of words in random_names. So far, so good.

That cat at the beginning is actually redundant, but I want to stack up a few commands for you to give you an idea of the power of piping. Ordinarily, I would write that command as follows:

The cat is extraneous because sort incorporates its function. Using pipes is a great timesaver because you don’t always need to have output at every step of the way.

There is another batch of commands suited to working with directory files (directories being just another type of file).

One way to create a complicated directory structure is to use the mkdir command to create each and every directory.

You could save yourself a few keystrokes and use the -p flag. This tells mkdir to create any parent directories that might not already exist. If you happen to like a lot of verbiage from your system, you could also add the --verbose flag for good measure.

To rename or move a directory, the format is the same as you used with a file or group of files. Use the mv command.

Removing a directory can be a bit more challenging. The command rmdir seems simple enough. In fact, removing this directory was no problem:

Removing this one, however, gave me an error:

You can only use rmdir to remove an empty directory. There is a -p option (as in parents) that enables you to remove a directory structure. For instance, you could remove a couple of levels like this:

All the directories from junk_dir on down are removed, but only if they are empty of files. A better approach is to use the rm command with the -r, or recursive, option. Unless you are deleting only a couple of files or directories, you want to use the -f option as well.

Yeah, I know. It’s a pretty cheesy pun, but I like it.

Because you’ve had a chance to play with a few directory commands, I’d like to take a moment and talk about a very special directory. Every user on your system has a home directory. That directory can be referenced with the $HOME environment variable. To get back to your home directory at any time, simply type cd $HOME and no matter where you were, there you are. Actually, you only need to type cd, press <Enter>, and you are home. The $HOME is implied.

The $HOME shortcut is great for shell scripts or anytime you want to save yourself some keystrokes. For instance, say you want to copy the file remote.file to your home directory and you are sitting in /usr/some_remote/dir. You could use either of the next two commands:

The second command saves you keystrokes, and the more time you spend doing system administration, the more you will love shortcuts like this. To save the maximum keystrokes, you can also use the tilde (~), a special character synonym for $HOME.

What you can and can’t do with a file, as defined by your username or group name, is pretty much wrapped up in four little letters. Look at the following listing (using ls -l) for an example. The permissions are at the beginning of each line.

Each of these letters can be referenced by a number. They are r, w, x, and s. Their numerical representations are 4, 2, 1, and “it depends.” To understand all that, you need to do a little binary math.

Reading from right to left, think of the x as being in position 0. The w, then, is in position 1 and the r is in position 2. Here’s the way it works:

To specify multiple permissions, you can add the numbers together. If you want to specify both read and execute permissions, simply add 4 and 1 and you get 5. For all permissions (rwx), use 7.

File permissions are referenced in groups of three rwx sections. The r stands for read, the w means write, and the x denotes that the file is executable.

Although these permissions are arranged in three groups of three rwx combinations, their meaning is the same in all cases. The difference has to do with who they represent rather than the permissions themselves. The first of these three represents the user, the second trio stands for the group permissions, and the third represents everybody that doesn’t fit into either of the first two categories.

The commands you use for changing these basic permissions are chmod, chown, and chgrp.

Let’s pretend you have a file called mail_test and you want to change its ownership from the root user to natika. Because root is the only user that can make that change, you need to prepend your command with sudo. This is very simple.

You can also use the -R option to change ownership recursively. Let’s use a directory called test_directory as an example. Once again, it belongs to root and you want to make every file in that directory (and below) owned by natika.

The format for changing group ownership is just as easy. Let’s change the group ownership of test_directory (previously owned by root) so that it and all its files and subdirectories belong to group accounts.

You can even combine the two formats. In the following example, the ownership of the entire finance_data directory changes to natika as the owner and accounts as the group. To do so, you use this form of the chown command:

Now, you are aware that files (and directories) are owned by some user and some group. This brings us to the next question.

Every time you create a file, you are submitted to a default set of permissions. Go ahead. Create a blank file using the touch command. I am going to call my blank file fish.

Now have a look at its permissions by doing an ls –l.

Without doing anything whatsoever, your file has read and write permissions for both the user and group, and read permission for everybody else. This happens because you have a default file-creation mask of 002. You can discover this using the umask command.

The 2 is subtracted from the possible set of permissions, rwx (or 7). 7 – 0 remains 7, while 7 – 2 is 5. But wait—5 stands for r-x, or read and execute. How is it that the file only shows a read bit set? That’s because newly created files are not set executable. At best, they provide read and write permissions for everyone. Another way to display this information is by using the -S flag. Instead of the numeric output, you get a symbolic mask displayed.

If you have an application that requires you to provide a default set of permissions for all the files you create, change umask to reflect that inside your scripts. As an example, let’s pretend that your program or script creates text files that you want everyone to be able to read (444). Because the execute bit isn’t a factor anyway, if you mask out the write bit using a 2 all around, everybody will have read permission. Set your umask to 222, create another file (called duck this time), and then do an ls –l to check things out.

One of the most useful commands in your arsenal is the find command. This powerhouse doesn’t get the credit it deserves. Generally speaking, find is used to list files and redirect (or pipe) that output to do some simple reporting or backups. There it ends. If anything, this should only be the beginning. As versatile as find is, you should take some time to get to know it. Let me give you a whirlwind tour of this awesome command. Let’s start with the basics:

One of those options is -print, which only makes sense if you want to see any kind of output from this command. You could easily get a listing of every file on the system by starting at the top and recursively listing the disk.

Although that might be interesting and you might want to redirect that to a file for future reference, it is only so useful. It makes more sense to search for something. For instance, look for all the JPEG-type image files sitting on your disk. Because you know that these images end in a .jpg extension, you can use that to search.

Depending on the power of your system, this can take a while and you are likely to get a lot of Permission Denied messages (particularly as you traverse a directory called /proc). If you are running this as a user other than root, you likely get a substantial number of Permission Denied messages. At this point, the usefulness of find should start to become apparent because a lot of images stashed away in various parts of the disk can certainly add up as far as disk space is concerned. Try it with an .avi or .mpg extension to look for video clips (which can be very large).

If you are trying to locate old files or particularly large files, try the following example. Look for anything that has not been modified (this is the -mtime parameter) or accessed (the -atime parameter) in the last 12 months. The -o flag is the OR in this equation.

A few techniques introduced here are worth noting. The backslashes in front of the round brackets are escape characters; they are there to make sure the shell does not interpret them in ways you do not want it to—in this case, the open and close parentheses on the second line. The first line also has a backslash at the end. This is to indicate a line break, as the whole command does not fit neatly on one line of this page. If you typed it exactly as shown without any backslashes, it would not work; however, the backslashes in the second line are essential. The preceding command also searches for files that are greater than 500KB in size. That is what the -size +1024 means because 1024 refers to 512-byte blocks. The -ls at the end of the command tells the system to do a long listing of any files it finds that fit my search criteria.

Earlier in this chapter, you learned about setuid and setgid files. Keeping an eye on where these files are and determining if they belong there are important aspects of maintaining security on your system. Here’s a command that examines the permissions on your files (the –perm option) and reports back what it finds.

You may want to redirect this output to a file that you can later peruse and decide on what course of action to take. Now let’s look at another find example to help you uncover what types of files you are looking at. Your Linux system has another command called file that can deliver useful information on files and what they are, whether they are executables, text files, or movie clips. Here’s a sample of some of the files in my home directory as reported by file:

The next step is to modify the find command by adding a -exec clause so that I can get the file command’s output on what find locates.

The open and close braces that follow -exec file mean that the list of files generated should be passed to whatever command follows the -exec option (in other words, the command you will be executing). The backslash followed by a semicolon at the end is required for the command to be valid. As you can see, find is extremely powerful. Learning to harness that power can make your administrative life much easier. You’ll encounter find again at various times in this book.

With all the information displayed through ps, you are forgiven if your head is starting to hurt. It is a little like trying to see the forest but being overwhelmed by the sheer number of trees. And yet, all these processes are linked in some way. Luckily, your stock Linux distribution contains tools to make this easier. One of them is called pstree. Here’s a sample of what you get by simply typing the command and pressing <Enter>:

This is only a partial listing, but notice that everything on the system stems from one super, ancestral process called init. Somewhere under there, I have a login that spawns a shell. From that shell, I start an X window session, from which spawns my GNOME display manager, then my login, and so on.

If you want a similar output, but in more detail, you can go back to your old friend, the ps command. Try the f flag, which in this case stands for forest, as in forest view. The following output is the result of my running ps axf. Again, this is a partial listing, but unlike the pstree listing, you also get process IDs, running states, and so on.

In the Linux world, you can find a number of programs devoted to deciphering those numbers, thereby making it possible to find out what processes are doing and how much time and resources they are using to do it and making it possible to manage the resultant information.

Once in a while, I start a process that I think is going to take a few seconds—like parsing a large log file, scanning for some text, extracting something else, sorting the output, and finally sending the whole thing to a file. All of these are very ad hoc in terms of reporting. The trouble is this: Two and a half minutes go by and I start to get a little impatient. Had I thought that the process would take a while, I might have started it in the background.

When you start a process (by typing a command name and pressing <Enter>), you normally start that process in the foreground. In other words, your terminal is still controlling the process and the cursor sits there at the end of the line until the process completes. At that point, it returns to the command or shell prompt. For most (not all) processes, you can run things in the background, thus immediately freeing up your command line for the next task. You do this by adding an ampersand (&) to the end of the command before you press <Enter>.

However, I’ve already confessed that I wasn’t thinking that far ahead and as a result, I am sitting looking at a flashing cursor wondering if I did something wrong and just how long this process will take. Now, I don’t want to end the process, but I would like to temporarily pause it so I can look at its output and decide whether I want to continue. As it turns out, I can do precisely that with a running process by pressing <Ctrl+Z>.

The process is now suspended. In fact, if you do a ps ax and you look for long_process, you see this:

Now on to the wonderful world of editors. I’m talking here about computerized line editors, as opposed to those who want to know whether your book will be delivered in time for the original deadline.

There are times when you need to work with an editor, whether to edit a configuration file, write a script, or do some quick edits on the company Web site. In the Linux world, there are many alternatives, and your Ubuntu system comes with a few. Let me introduce you to a friendly new member of the family.

Meet gedit

Your GNOME desktop comes with a very nice and powerful editor called gedit (command name gedit). With colorful language-sensitive syntax highlighting (sh, C, C++, Perl, HTML, and others), tabs, and a spell checker, gedit is easy and fun to work with. To see gedit in action, look at Figure 21-2. You can start gedit on its own or specify a filename to start with. Notice that I started my session using gksudo because I was editing a system script.

    gksudo gedit /etc/profile

Figure 21-2 gedit is an easy-to-use editor that supports tabs so you can work on multiple files simultaneously.

Tip   In the preceding example, I use gksudo instead of sudo. This is a graphical frontend to sudo. It’s also the best (and correct) way to run a graphical command that requires administrative privileges via the <Alt+F2> quick-start.

As you can see by looking at Figure 21-2, gedit supports tabs, thereby making it simple to work on multiple files at the same time. Just click the tab to select the file you want to edit. Basic editing functions like cut, paste, find, and replace, are available directly below the menu bar. To edit a file without starting from a command line, click the Open icon, which opens up the standard GNOME file selector.

For years, I’ve been a fan of vim, the next editor I will discuss, but I appreciate the beauty of a simple, friendly editor like gedit. It’s easy-to-use with a learning curve close to zero. Still, there may be times when you don’t have access to gedit and times when a slow, remote connection makes a graphical editor too painful to consider. It’s at those times that you’ll be thankful for a simple shell window and vim to get your work done.

:q, :w, :wq, and ZZ

Finished editing? When it comes time to save your work, press <Esc> (to get out of whatever mode you are in) and type ZZ. Another way to exit is to type :wq (write and quit). At any time during an editing session, you can type :w to save your current work. Finally, if you really don’t want to save anything you have done, type :q! The exclamation point essentially says that you won’t take no for an answer. Had you modified the file and simply typed :q, vi would warn you that you were trying to exit from a modified file without having saved your changes.

Keystrokes for Functions

Quick Tip   Need help while in vi? Make sure you aren’t in insert or replace mode, and then type :help.

I urge you not to let vi frighten you. Get to know it. The likelihood that you will ever log on to a modern Linux (or UNIX) system that doesn’t have some form of vi installed is virtually nonexistent. That said, if you need more information than I’ve given you here, consider the vi tutor. This little tool is distributed as part of the vim documentation. It is essentially a text file that tells you what to do and how to do it as you read it. To start the tutor, type the following command:

    vimtutor

When the tutorial starts, you should see a picture like the one in Figure 21-3. The entire tutorial takes between 25 and 30 minutes for most people to complete.

Figure 21-3 Learning vi or vim from the vimtutor.

Recovering a vi Session

From time to time, you may find yourself trying to edit a file, but someone else, maybe you, is already editing it. That session may be open or something may have happened to terminate it accidentally. As a result, you get a nice, long-winded message along the lines of “swap file found” and a whole lot of information on what you can do about it. Here’s a shortened version of that message:

    E325: ATTENTION
    Found a swap file by the name "textfiles/.listing1.swp"
              owned by: mgagne   dated: Sun Dec 28 14:43:20 2003
             file name: ~mgagne/textfiles/listing1

To locate these files, you can use the famous find command and look for anything with a .swp extension. Another and better way is to have vi report on these for you. You can check for the existence of swap files by using the -r option, which provides you with a little more information than the simple find.

    Swap files found:
       In current directory:
    1.    .linux_companies.swp
              owned by: mgagne    dated: Mon Sep 15 14:38:55 2003
             file name: ~mgagne/textfiles/linux_companies
              modified: YES
             user name: mgagne    host name: francois
            process ID: 2266
    2.    .sas.swp
              owned by: mgagne    dated: Mon Sep  8 09:49:02 2003
             file name: ~mgagne/textfiles/sas
              modified: no
             user name: mgagne    host name: ultraman
            process ID: 6340
       In directory ~/tmp:
           -- none --
       In directory /var/tmp:
           -- none --
       In directory /tmp:
           -- none --

Power vi: Start-Up Options

The next time you need a reason to use vi over one of the other editors, consider some of these tricks for getting to what you want as quickly as possible.

    vi +100 ftl_program.c

This takes you right to line 100 in the file called (in this case) ftl_program.c. This can be a great timesaver when you are compiling programs and something goes wrong, as in the following example:

    
    gcc  -O2 -Wall  -c -o ftl_program.o ftl_program.c
    ftl_program.c:100: parse error before `<'
    make: *** [ftl_program.o] Error 1

Another useful start flag is the same one you use to search for text inside a file: the forward slash (/). In the next example, I want to get back to the same place I was working on in my file. To mark my place, I had written my name on a blank line. This vi starter gets me right to where I was working:

    vi +/Marcel ftl_program.c

Note the plus sign before the slash.

Other Editor Options

gedit and vi (or vim) represent only two of many, many different editors available to the Linux user. Your Ubuntu Linux system comes with a few others. One of my old favorites is another nongraphical editor that makes a nice alternative for people who, having gotten up close and personal with vim, would prefer something else. It’s called Nano (command name nano), an enhanced editor based on Pico (in case you are curious). Nano, by the way, stands for Nano’s ANOther editor.

Yeah, I know.