Gawk is commonly considered to be the most popular version of awk available today. Gawk comes from the GNU Foundation, and in true GNU fashion, it has many enhancements that other versions lack.

The enhancements that gawk has over the traditional awks are too numerous to cover here; however, a few of the most notable follow:

If these features are not enough, the gawk project is very active, with a number of people contributing, whereas mawk has not had a release in several years. Gawk has been ported to a dizzying array of architectures, from Atari, Amiga, and BeOS to Vax/VMS. Gawk is the standard awk that is installed on GNU/Linux and BSD machines.

The additional features, the respect that the GNU Foundation has in making quality free (as in freedom) software, the wide deployment on GNU/Linux systems, and the active development in the project are all probable reasons why gawk has become the favorite over time.

There is no single test to find out what version or implementation of awk you have installed. You can do a few things to deduce it, or you can install it yourself so you know exactly what is installed. Check your documentation, man pages, and info files to see if you can find a mention of which implementation is referenced, looking out for any mention of oawk, nawk, gawk, or mawk. Also, poke around on your system to find where the awk binary is, and see if there are others installed. It is highly unlikely that you have no version installed, but the hard part is figuring out which version you do have.

Gawk takes the standard GNU version flags to determine what version you are running. If you run awk with these flags as shown in the following Try It Out, and it succeeds, you know that you have GNU awk available.

$ awk --version
GNU Awk 3.1.4
Copyright (C) 1989, 1991-2003 Free Software Foundation.
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.

$ awk --version
awk: not an option: --version

$ awk -W versions
mawk 1.3.3 Nov 1996, Copyright (C) Michael D. Brennan

compiled limits:
max NF             32767
sprintf buffer      1020

By far the most popular awk is the GNU Foundation's implementation, gawk. If you find that your system does not have gawk installed, and you wish to install it, follow these steps. If you have a system that gawk has not been ported to, you may need to install a different awk. The known alternatives and where they can be found are listed in the awk FAQ at www.faqs.org/faqs/computer-lang/awk/faq/.

By far the easiest way to install gawk is to install a prepackaged version, if your operating system provides it. Installation this way is much simpler and easier to maintain. For example, to install gawk on the Debian GNU/Linux OS, type this command:

apt-get install gawk

Mac OS X has gawk available through fink. Fink is a command-line program that you can use to fetch and easily install some useful software that has been ported to OS X. If you don't have fink installed on your system, you can get it at http://fink.sourceforge.net/download/index.php.

If your system does not have packages, or if you want to install gawk on your own, follow these steps:

The default field separator in awk is a blank space. When you insert a blank space by pressing the spacebar or Tab key, awk delineates each word in a line as a different field. However, if the data that you are working with includes spaces within the text itself, you may encounter difficulties.

For example, if you add more countries to your countries.txt file to include some that have spaces in them (such as Dominican Republic), you end up with problems. The following command prints the area of each country in the file:

$ awk '{ print $3 }' countries.txt
647500
28748
2381740
.do
468
1246700

Why is the .do included in the output? Because one of the lines of this file contains this text:

Dominican Republic  .do  8833634     48730  901800  2120400

The country Dominican Republic counts as two fields because it has a space within its name. You need to be very careful that your fields are uniform, or you will end up with ambiguous data like this. There are a number of ways to get around this problem; one of the easiest methods is to specify a unique field separator and format your data accordingly. In this case, you need to format your countries.txt file so that any country that has spaces in its name instead had underscores, so Dominican Republic becomes Dominican_Republic.

Unfortunately, it isn't always practical or possible to change your input data file. In this case, you can invoke awk with the -F flag to specify an alternative field separator character instead of the space. A very common field separator is the comma, so to instruct awk to use a comma as the character that separates fields, invoke awk using -F, to indicate the comma should be used instead. Most databases are able to export their data into CSV (Comma Separated Values) files. If your data is formatted using commas to separate each field, you can specify that field separator to awk on the command line, as the following Try It Out section demonstrates.

Afghanistan,.af,647500,28513677,33100,12000
Albania,.al,28748,3544808,255000,1100000
Algeria,.dz,2381740,32129324,2199600,1447310
Andorra,.ad,468,69865,35000,23500
Angola,.ao,1246700,10978552,96300,130000
Dominican Republic,.do,8833634,48730,901800,2120400

$ awk -F, '{ print $3 }' countries.csv
647500
28748
2381740
8833634
468
1246700

The printf (formatted print) command is a more flexible version of print. If you are familiar with C, you will find the printf command very familiar; it was borrowed from that language. Printf is used to specify the width of each item printed. It also can be used to change the output base to use for numbers, to determine how many digits to print after the decimal point, and more. Printf is different from print only because of the format string, which controls how to output the other arguments. One main difference between print and printf is that printf does not include a newline at the end. Another difference is that with printf you specify how you want to format your string. The printf command works in this format:

printf(<string>,<format string>)

The parentheses are optional, but otherwise, the basic print command that you have been using so far is almost identical:

printf("Hi Mom!
")

The string is the same with the exception of the added character, which adds a newline to the end of the string. This doesn't seem very useful, because now you have to add a newline when you didn't before. However, printf has more flexibility because you can specify format codes to control the results of the expressions, as shown in the following Try It Out examples.

$ awk '{ printf "Number of cell phones in use in %s: %d
", $1, $6 }' countries.txt
Afghanistan    .af    647500         28513677       33100          12000
Albania        .al    28748          3544808        255000         1100000
Algeria        .dz    2381740        32129324       2199600        1447310
Andorra        .ad    468            69865          35000          23500
Angola         .ao    1246700        10978552       96300          130000

$ awk '{ printf "Decimal: %d, Hex: %x, Octal: %o
", $6, $6, $6 }' countries.txt
Decimal: 12000, Hex: 2ee0, Octal: 27340
Decimal: 1100000, Hex: 10c8e0, Octal: 4144340
Decimal: 1447310, Hex: 16158e, Octal: 5412616
Decimal: 23500, Hex: 5bcc, Octal: 55714
Decimal: 130000, Hex: 1fbd0, Octal: 375720

These printf format characters are useful for representing your strings and numbers in the way that you expect. You can also add a modifier to your printf format characters to specify how much of the value to print or to format the value with a specified number of spaces.

You can provide an integer before the format character to specify a width that the output would use, as in the following example:

$ awk '{ printf "|%16s|
", $6 }' countries.txt
|     Afghanistan|
|         Albania|
|         Algeria|

Here, the width 16 was passed to the format modifier %s to make the string the same length in each line of the output. You can left-justify this text by placing a minus sign in front of the number, as follows:

$ awk '{ printf "|%-16s|
", $1 }' countries.txt
|Afghanistan     |
|Albania         |
|Algeria         |

Use a fractional number to specify the maximum number of characters to print in a string or the number of digits to print to the right of the decimal point for a floating-point number:

$ awk '{ printf "|%-.4s|
", $1 }' countries.txt
|Afgh|
|Alba|
|Alge|

BEGIN { printf "%-15s %20s

", "Country", "Cell phones" }
      { printf "%-15s %20d
", $1, $6 }

$ awk -f format.awk countries.txt
Country                  Cell phones

Afghanistan                    12000
Albania                      1100000
Algeria                      1447310
Andorra                         23500

The sprintf function operates exactly like printf, with the same syntax. The only difference is that it assigns its output to a variable (variables are discussed in the next section), rather than printing it to standard out. The following example shows how this works:

$ awk '{ variable = sprintf("[%-.4s]", $1); print variable}' countries.txt
|Afgh|
|Alba|
|Alge|

This assigns the output from the sprintf function to the variable variable and then prints that variable, which results in the same output as if you had used printf.

User-defined variables have a few rules associated with them. They must not start with a digit and are case sensitive. Besides these rules, your variables can consist of alphanumeric characters and underscores. A user-defined variable must not conflict with awk's reserved built-in variables or commands. For example, you may not create a user-defined variable called print, because this is an awk command. Unlike some programming languages, variables in awk do not need to be initialized or declared. The first time you use a variable, it is set to an empty string ("") and assigned 0 as its numerical value. However, relying on default values is a bad programming practice and should be avoided. If your awk script is long, define the variables you will be using in the BEGIN block, with the values that you want set as defaults.

Variables are assigned values simply by writing the variable, followed by an equal sign and then the value. Because awk is a "weak-typed" language, you can assign numbers or strings to variables:

myvariable = 3.141592654
myvariable = "some string"

When you perform a numeric operation on a variable, awk gives you a numerical result; if a string operation is performed, a string will be the result.

In the earlier section on printf, the Try It Out example used format string modifiers to specify columnar widths so that the column header lined up with the data. This format string modifier could be set in a variable instead of having to type it each time, as in the following code:

BEGIN { colfmt="%-15s %20s
"; printf colfmt, "Country", "Cell phones
" }
      { printf colfmt, $1, $6 }

In this example, a user-defined variable called colfmt is set, containing the format string specifiers that you want to use in the rest of the script. Once it is defined, you can reference it simply by using the variable; in this case it is referenced twice in the two printf statements.

Built-in variables are very useful if you know what they are used for. The following subsections introduce you to some of the most commonly used built-in variables.

# Awk script to print the number of cell phones in use in each country

BEGIN { FS = "," } # Our data is separated by commas

{ print "Number of cell phones in use in",$1":",$6 }

$ awk -f countries.awk countries.csv

$ echo "a,,,   b,,,,, c,,   d, e,, f,   g" | awk 'BEGIN {FS="[,]+[ ]+"} {print $2}'
b

$ awk 'END { print "Number of countries:", NR }' countries.txt
Number of countries: 5

BEGIN { colfmt="%-15s %20s
"; printf colfmt, "Country", "Cell phones
" }
      { printf "%d. " colfmt, NR, $1, $6 }

$ awk -f format.awk countries.txt
Country                      Cell phones

1. Afghanistan                    12000
2. Albania                      1100000
3. Algeria                      1447310
4. Andorra                         23500
5. Angola                        130000

One of the most important awk decision making statements is the if statement. It follows a standard if (condition) then-action [else else-action] format, as in the following Try It Out.

$ awk '{ if ($3 < 1000) print }'
Andorra .ad 468 69865 35000 23500

{ if ($3 < 1000)
        printf "%s has only %d people!
", $1, $3
  else
        printf "%s has a population larger than 1000
", $1 }

$ awf -f ifelse.awk countries.txt
Afghanistan has a population larger than 1000
Albania has a population larger than 1000
Algeria has a population larger than 1000
Andorra has only 468 people!
Angola has a population larger than 1000

{ if ( $1 == "cat" )
        print "meow";
    else if ( $1 == "dog" )
        print "woof";
    else if ( $1 == "bird" )
        print "caw";
    else
        print "I do not know what kind of noise " $1 " makes!" }

These examples use the less than operation, but there are many other operators available for making conditional statements powerful. Another example is the equal comparison operator, which checks to see if something is equal to another thing. For example, the following command looks in the first field on each line for the string Andorra and, if it finds it, prints it:

awk '{ if ($1 == "Andorra") print }'

The following table lists the comparison operators available in awk.

It is also possible to combine as many comparison operators in one statement as you require by using AND (&&) as well as OR (||) operators. This allows you to test for more than one thing before your control statement is evaluated to be true.

For example:

$ awk '{ if ((($1 == "Andorra") && ($3 <= 500)) || ($1 == "Angola")) print }'
Andorra .ad 468 69865 35000 23500
Angola .ao 1246700 10978552 96300 130000

This prints any line whose first field contains the string Andorra and whose third field contains a number that is less than or equal to 500, or any line whose first field contains the string Angola.

As this example illustrates, each condition that you are testing must be surrounded by parentheses. Because the first and second condition are together (the first field has to match Andorra and the third field must be less than or equal to 500), the two are enclosed together in additional parentheses. There are also opening and closing parentheses that surround the entire conditional.

The comparison operators are useful for making comparisons, but you often will want to make changes to variables. Awk is able to perform all the standard arithmetic functions on numbers (addition, subtraction, multiplication, and division), as well as modulo (remainder) division, and does so in floating point. The following Try It Out demonstrates some arithmetic functions.

$ awk 'BEGIN {myvar=10; print myvar+myvar}'
20
$ awk 'BEGIN {myvar=10; myvar=myvar+1; print myvar}'
11

$ awk 'BEGIN {myvar=10; print ++myvar; print myvar}'
11
11
$ awk 'BEGIN {myvar=10; print myvar++; print myvar}'
10
11

$ awk 'BEGIN {celltotal = 0}
>    {celltotal += $6}
>    END { print celltotal }' /tmp/countries.txt
2712810

Be careful when using comparison operators, because some of them double as shell output variables in different contexts. For example the > character can be used in an awk statement to send the output from a command or a function into the file specified. For example, if you do the following:

$ awk 'BEGIN { print 4+5 > "result" }'

you create a file called result in your current working directory and then print the result of the sum of 4 + 5 into the file. If the file result already exists, it will be overwritten, unless you use the shell append operator, as follows:

$ awk 'BEGIN { print 5+5 >> "result" }'

This appends the summation of 5 + 5 to the end of the result file. If that file doesn't exist, it will be created.

Output from commands can also be piped into other system commands in the same way that this can be done on the shell command line.

While statements in awk implement basic looping logic, using the same concepts introduced in Chapter 3. Loops continually execute statements until a condition is met. A while loop executes the statements that you specify while the condition specified evaluates to true.

While statements have a condition and an action. The condition is the same as the conditions used in if statements. The action is performed as long as the condition tests to be true. The condition is tested; if it is true, the action happens, and then awk loops back and tests the condition again. At some point, unless you have an infinite loop, the condition evaluates to be false, and then the action is not performed and the next statement in your awk program is executed.

$ awk 'BEGIN { while (++myvar <= 10 ) print myvar }'

For loops are more flexible and provide a syntax that is easier to use, although they may seem more complex. They achieve the same results as a while loop but are often a better way of expressing it. Check out this example.

$ awk 'BEGIN { for ( myvar = 1; myvar <= 10; myvar++ ) print myvar }'