Exploring Environment and Shell Variables
The two types of variables are environment variables and shell variables. Environment variables are variables defined in the current shell session and are available to any programs executed within that session. Environment variables often control how programs work. For example, you might set a LANG environment variable that other programs can use to determine the language a program should use to communicate with you. Shell variables are similar, except they’re not available to programs and subprocesses. You can think of environment variables as global variables, and shell variables as local ones.
The env command shows you all the environment variables that are set. A lot’s there, so pipe the results to less:
| | $ env | less |
In the output, you’ll see many environment variables, including the following:
| | ... |
| | USER=brian |
| | PWD=/home/brian |
| | HOME=/home/brian |
| | TERM=xterm-256color |
| | SHELL=/bin/bash |
| | ... |
| | LOGNAME=brian |
| | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin: |
| | /usr/games:/usr/local/games:/snap/bin |
| | ... |
Here’s what a few of these do:
-
The USER and LOGNAME variables hold the username of the current user. The PWD variable holds the full path to the current working directory.
-
The TERM variable defines the terminal output mode, which terminal software uses to determine what fonts, colors, and other features are available.
-
The SHELL variable defines what shell you’re using. In this case, it’s the Bash shell, located at /bin/bash.
-
The HOME variable holds the value of the current user’s home directory. When you issue the cd command with no arguments, this value is used.
-
The PATH variable holds a list of directories the OS uses to locate executable programs.
The env command shows only environment variables. To see shell variables in addition to environment variables, use the set command. In Bash, this command prints environment variables, shell variables, and functions, which may be a huge amount of output you might not want to see.
| | $ set -o posix; set; set +o posix |
This configures the set command to run in POSIX mode, which doesn’t display functions. It then runs the set command, and restores the original behavior. You can shorten this to:
| | $ (set -o posix; set) |
The parentheses around the statement tells Bash to execute the command in a subshell, leaving the current shell’s settings unchanged. This eliminates the need to restore the default behavior. You’ll learn about subshells in Running Commands in Subshells.
This command shows a lot more detail:
| | ... |
| | BASHOPTS=checkwinsize:cmdhist:complete_fullquote:expand_aliases:extglob: |
| | extquote:force_fignore:histappend:interactive_comments:login_shell:progcomp: |
| | promptvars:sourcepath |
| | ... |
| | BASH_VERSION='4.4.19(1)-release' |
| | COLUMNS=211 |
| | ... |
| | DIRSTACK=() |
| | ... |
| | HISTCONTROL=ignoreboth |
| | HISTFILE=/home/brian/.bash_history |
| | HISTFILESIZE=2000 |
| | HISTSIZE=1000 |
| | ... |
| | HOSTNAME=puzzles |
| | HOSTTYPE=x86_64 |
| | ... |
| | LINES=93 |
| | MACHTYPE=x86_64-pc-linux-gnu |
| | ... |
| | OSTYPE=linux-gnu |
| | PPID=3462 |
| | PS1='[e]0;u@h: wa]${debian_chroot:+($debian_chroot)}[ 33[01;32m] |
| | u@h[ 33[00m]:[ 33[01;34m]w[ 33[00m]$ ' |
| | PS2='> ' |
| | PS4='+ ' |
| | SHELLOPTS=braceexpand:emacs:hashall:histexpand:history:interactive-comments: |
| | monitor:posix |
| | UID=1000 |
Unfortunately, there’s no easy way to show only the shell variables due to the way the env and set commands display output. You’d have to take the output of set and then subtract everything returned by env.
Using Variables
To print one of these environment variables to the screen, use the printenv command, followed by the variable name:
| | $ printenv HOME |
| | /home/brian |
This is good for checking the value, but you can also use these variables as values in your shell commands. To do so, prefix the variable with the dollar sign. For example, print out the HOME variable’s value:
| | $ echo $HOME |
| | /home/brian |
If you omit the dollar sign, you’d only see the literal value HOME instead. The dollar sign tells the shell to expand the variable and use its value. This concept is called parameter expansion, or variable expansion.
You can use variable expansion in combination with other strings as well. Run this command to print out the current username in a longer string:
| | $ echo "The current user is ${USER}" |
| | The current user is brian |
Notice the curly braces around the variable? It’s good practice to use these when mixing your variables with other strings. You can leave them off, but in some situations you’ll get the wrong results. Here’s an example of one of those situations.
You can create a text file that will hold your history, but use the value of the USER variable as part of the filename. Before you create the file, use echo to add _history.txt as a suffix to the value of USER to see what the filename would look like:
| | $ echo "$USER_history.txt" |
| | .txt |
All you see is .txt, because the shell thought the variable was USER_history rather than USER. It doesn’t always know what the word boundaries are. Using braces makes it unambiguous. Try this:
| | $ echo "${USER}_history.txt" |
| | brian_history.txt |
This results in the filename you’re looking for, so create the actual history file now that you know the target filename is correct:
| | $ history > "${USER}_history.txt" |
| | $ ls "${USER}_history.txt" |
| | brian_history.txt |
Use braces like this when performing variable expansion when building strings. You won’t be surprised by the results if you do this consistently. In addition, it’s a really good idea to use double quotes around the strings you create when using variables. It’s not always required, but like the braces, you can run into unexpected results. For example, variables may contain spaces or characters that the shell might interpret differently. The Bash manual explains the rules of how quotes work in more detail.[13]
One of the more important environment variables is PATH, which determines where the shell should look for executable programs. Let’s explore that one in detail.
Locating Executables with PATH
When you type commands like cd or ls, you’re executing a program. The PATH environment variable contains a colon-delimited list of directories the OS uses to look up executable programs.
Print out the value of the PATH variable with echo:
| | $ echo $PATH |
| | /home/brian/bin:/home/brian/.local/bin:/usr/local/sbin:/usr/local/bin: |
| | /usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin |
The which command will tell you where an executable is located on your filesystem by looking through all the directories in the PATH and displaying the first instance it finds. Try it out with the ls command:
| | $ which ls |
| | /bin/ls |
According to this output, the ls command is located at bin/ls. That directory is one of the entries on your PATH of executable files, which is why you can execute it from anywhere on your filesystem. Many of the commands you’ve used so far are located somewhere on your PATH, either because they were installed there during the initial installation of the operating system or by an installation tool such as your package manager.
Sometimes you’ll have to download programs or scripts yourself and place them on your PATH so you can run them system-wide. Let’s create a little executable program so you can get more comfortable with the PATH.
First, create a new directory to hold the script, then navigate into that folder:
| | $ mkdir greeting_script |
| | $ cd greeting_script |
Instead of using a programming language like C, Ruby, Python, or Perl, you’ll create a shell script and make it executable. You’ll learn a lot more about how these scripts work in Chapter 9, Automation.
Use cat to create the following shell script. This prints the text “Hello” to the screen:
| | $ cat << 'EOF' > greetings |
| | > #!/usr/bin/env bash |
| | > echo Hello |
| | > EOF |
The line #!/usr/bin/env bash tells the shell which program should run the script. In this case, we’re stating we want Bash to run the script. The second line of the script is the echo statement you’ve seen many times before.
Use the chmod +x command to make this script executable:
| | $ chmod +x greetings |
Now execute the greetings command:
| | $ greetings |
| | greetings: command not found |
The greetings command fails to run because the shell can’t find it on the PATH. By default, the shell prevents you from running executable files in the current working directory without explicitly specifying the path to the executable. This prevents you from accidentally running a malicious script with the same name as a built-in command. Imagine if someone wrote a script called ls that deleted all the files in your home directory. That could be disastrous.
To run an executable file in the current directory, prefix it with ./:
| | $ ./greetings |
| | Hello |
You can now see the output. Switch your working directory to the /var folder and then run the greetings command. This time, since you are not in the same directory as the command, you will have to provide the path to the command:
| | $ cd /var |
| | $ ~/greeting_script/greetings |
| | Hello |
To make the command available system-wide without having to specify the path, copy it to a location contained in your PATH variable, or modify the value of PATH to include a new directory. You’ll do the former for now; you’ll look at modifying the PATH later.
The /usr/local/bin path is a common location for user-added scripts that should be available system-wide. So move the greetings script from the ~/greeting_script directory to that folder:
| | $ sudo mv ~/greeting_script/greetings /usr/local/bin/greetings |
Now that the file is in place, you can execute the greetings command from any location on your filesystem without specifying the full path:
| | $ greetings |
| | Hello |
| | $ cd |
| | $ greetings |
| | Hello |
| | $ cd /tmp |
| | $ greetings |
| | Hello |
Many programs and processes rely on the PATH variable. You can modify this variable or create your own variables to hold information you want to reference later.
You’re not limited to the variables the shell provides for you. You can make your own.
Setting Your Own Variables
You can set your own environment or shell variables in addition to the ones that are set up for you. This is helpful when you have to store long strings like API keys, access tokens, or credentials. Many web frameworks and applications use environment variables to store database credentials and other sensitive information to keep it out of the code.
Try it out. Create a new shell variable named SHIELD_PASSWORD and assign it the value of 12345:
| | $ SHIELD_PASSWORD=12345 |
Now print out the value:
| | $ echo $SHIELD_PASSWORD |
| | 12345 |
This variable is available at the shell level, so it’s not available to any subshells or any other programs. Use the env command to verify this, and use grep to filter for SHIELD_PASSWORD:
| | $ env | grep SHIELD_PASSWORD |
You won’t see any results.
To make SHIELD_PASSWORD an environment variable, use the export keyword:
| | $ export SHIELD_PASSWORD |
Now use env and grep to check if it’s part of the environment. This time you see it returned:
| | $ env | grep SHIELD_PASSWORD |
| | SHIELD_PASSWORD=12345 |
The Bash shell offers a shortcut for creating an environment variable in a single step. Make an environment variable named SECRET_KEY using this method:
| | $ export SECRET_KEY=12345 |
Verify that it’s set:
| | $ env | grep SECRET_KEY |
| | SECRET_KEY=12345 |
Finally, use the unset command to remove it:
| | $ unset SECRET_KEY |
| | $ env | grep SECRET_KEY |
| | $ |
Beware of Secrets in History | |
|---|---|
|
It’s common to use environment variables to store sensitive information. However, remember that every command you enter gets saved to your shell history. Many shells are preconfigured to not record commands starting with a leading space. Later in this chapter you’ll learn how to configure your shell history’s options to enable that behavior. |
Shell and environment variables give you control over how many pieces of your environment work. Many commands have environment variables you can set to change their default behavior. For example, if you plan to use grep a lot, and you want it to always show two lines around every result, you can export the GREP_OPTIONS variable:
| | $ export GREP_OPTIONS='-A 2 -B 2' |
Now when you use grep, those options are applied.
Supplying Environment Variables to Specific Programs
Sometimes you’ll have a script or program that needs some variables from the environment, but you don’t need or want to set these values forever, or you need to override values in your environment temporarily. You can do this by prefixing the command with the variables you need.
To demonstrate this, you’ll create a quick Perl script which grabs values from the environment and prints them out. Perl is great for this because it’s already installed on macOS and Ubuntu, and it makes it easy to write a small program to illustrate this concept.
Create a file named variables that reads the variables HOME and API_KEY from the environment. You’ll use the cat command to create this file quickly, using the heredoc method you’ve used throughout the book.
Variables in Perl start with a dollar sign, and that’s how you get the values of shell and environment variables. That’s why you’ve been placing single quotes around EOF when you’ve used cat to create files. Doing this instructs Bash to treat the contents of the heredoc literally rather than expanding the variables into their values.
First, switch to your home directory:
| | $ cd |
Then execute this command to create the script:
| | $ cat << 'EOF' > variables |
| | > #!/usr/bin/env perl |
| | > $home = $ENV{'HOME'}; |
| | > print "Home directory: $home "; |
| | > |
| | > $apikey = $ENV{'API_KEY'}; |
| | > print "API key: $apikey "; |
| | > EOF |
Then use chmod +x to make the script executable:
| | $ chmod +x variables |
Now run the script. Remember that you have to prefix the script’s name with ./ since it’s in the current working directory:
| | $ ./variables |
| | Home directory: /home/brian |
| | API key: |
The output shows a value for the home directory, but the API key is empty. The program is looking for an environment variable, so define the API_KEY environment variable and run the script again:
| | $ export API_KEY=12345 |
| | $ ./variables |
| | Home directory: /home/brian |
| | API key: 12345 |
If you’re going to be using the API_KEY variable over and over, this is a fine way to do it. But if you only need it for a single run, it can be overkill; you’d have to set the variable, run the program, and, if you didn’t want the variable lying around, you’d have to unset it:
| | $ export API_KEY=abcde |
| | $ ./variables |
| | Home directory: /home/brian |
| | API key: abcde |
| | $ unset API_KEY |
A shorter way to accomplish this is to use the env command, which creates a new environment using the values you define.
| | $ env API_KEY=abcde ./variables |
| | Home directory: /home/brian |
| | API key: abcde |
The value is set, the value is displayed, but the value of abcde doesn’t persist. You can verify that with echo $API_KEY if you like.
The env command isn’t necessary in Bash in most cases. Run the script again, but this time define the API_KEY variable by prefixing its definition to the command without using env:
| | $ API_KEY=12345 ./variables |
| | Home directory: /home/brian |
| | API key: 12345 |
You’ll see people use both methods. The env method is more universal, while the method that doesn’t use it is a Bash feature.
You can override the value of the HOME environment variable this way too. Give it a try:
| | $ API_KEY=abcde HOME=/var/www ./variables |
| | Home directory: /var/www |
| | API key: abcde |
The HOME variable is overridden, but only for that command’s run.
Shell and environment variables won’t persist when you close your shell. To have things persist beyond your existing session, you need to add it to your shell initialization files. You’ll get to that shortly. Before you do, let’s look at a better way to create and edit files while you’re working on the command line. That’ll make it easier to create the shell initialization files you’ll need.