In day-to-day work, we come across different kinds of files such as text files, source code files from different programming languages (for example, file.sh, file.c, and file.cpp), and so on. While working, we often perform various operations on files or directories such as searching for a given string or pattern, replacing strings, printing few lines of a file, and so on. Performing these operations is not easy if we have to do it manually. Manual searching for a string or pattern in a directory having thousands of files can take months, and has high chances of making errors.
Shell provides many powerful commands to make our work easier, faster, and error-free. Shell commands have the ability to manipulate and filter text from different streams such as standard input, file, and so on. Some of these commands are grep, sed, head, tr, sort, and so on. Shell also comes with a feature of redirecting output from one command to another with the pipe ('|'). Using pipe helps to avoids creation of unnecessary temporary files.
One of the best qualities of these commands is that they come along with the man pages. We can directly go to the man page and see what all features they provide by running the man command. Most of the commands have options such as --help to find the help usage and --version to know the version number of the command.
This chapter will cover the following topics in detail:
- Standard I/O and error streams
- Redirecting the standard I/O and error streams
- Pipe and pipelines—connecting commands
- Regular expressions
- Filtering output using
grep - Editing output using
sed - Duplicating a stream using
tee - Sorting and finding unique text
- Character-based translation using
tr - Filtering based on lines—
headandtail - Cut-based selection
In shell programming, there are different ways to provide an input (for example, via a keyboard and terminal) and display an output (for example, terminal and file) and error (for example, terminal), if any, during the execution of a command or program.
The following examples show the input, output, and error while running the commands:
- The input from a user by a keyboard and the input obtained by a program via a standard input stream, that is terminal, is taken as follows:
$ read -p "Enter your name:" Enter your name:Foo
- The output printed on the standard output stream, that is terminal, is as follows:
$ echo "Linux Shell Scripting" Linux Shell Scripting
- The error message printed on the standard error stream, that is terminal, is as follows:
$ cat hello.txt cat: hello.txt: No such file or directory
When a program executes, by default, three files get opened with it which are stdin, stdout, and stderr. The following table provides a short description of each of these:
|
File descriptor number |
File name |
Description |
|---|---|---|
|
|
| |
|
|
| |
|
|
|
File descriptors are integer numbers representing opened files in an operating system. The unique file descriptor numbers are provided to each opened files. File descriptors' numbers go up from 0.
Whenever a new process in Linux is created, then standard input, output, and error files are provided to it along with other needed opened files to process.
To know what all open file descriptors are associated with a process, we will consider the following example:
Run an application and get its process ID first. Consider running bash as an example to get PID of bash:
$ pidof bash 2508 2480 2464 2431 1281
We see that multiple bash processes are running. Take one of the bash PID example, 2508, and run the following command:
$ ls -l /proc/2508/fd
total 0 lrwx------. 1 sinny sinny 64 May 20 00:03 0 -> /dev/pts/5 lrwx------. 1 sinny sinny 64 May 20 00:03 1 -> /dev/pts/5 lrwx------. 1 sinny sinny 64 May 19 23:22 2 -> /dev/pts/5 lrwx------. 1 sinny sinny 64 May 20 00:03 255 -> /dev/pts/5
We see that 0, 1, and 2 opened file descriptors are associated with process bash. Currently, all of them are pointing to /dev/pts/5. pts, which is pseudo terminal slave.
So, whatever we will do in this bash, input, output, and error related to this PID, output will be written to the /dev/pts/5 file. However, the pts files are pseudo files and contents are in memory, so you won't see anything when you open the file.