10. Execution Environment
This chapter describes the environment in which Python programs are executed. The goal is to describe the runtime behavior of the interpreter, including program startup, configuration, and program termination.
Interpreter Options and Environment
The interpreter has a number of options that control its runtime behavior and environment. Options are given to the interpreter on the command line as follows:
python [options] [-c cmd | filename | - ] [args]
Here’s a list of the most common command-line options:
Table 10.1 Interpreter Command-Line Arguments
The -i option starts an interactive session immediately after a program has finished execution and is useful for debugging. The -m option runs a library module as a script which executes inside the _ _main_ _ module prior to the execution of the main script. The -O and -OO options apply some optimization to byte-compiled files and are described in Chapter 8, “Modules, Packages, and Distribution.” The -S option omits the site initialization module described in the later section “Site Configuration Files.” The -t, -tt, and -v options report additional warnings and debugging information. -x ignores the first line of a program in the event that it’s not a valid Python statement (for example, when the first line starts the Python interpreter in a script).
The program name appears after all the interpreter options. If no name is given, or the hyphen (-) character is used as a filename, the interpreter reads the program from standard input. If standard input is an interactive terminal, a banner and prompt are presented. Otherwise, the interpreter opens the specified file and executes its statements until an end-of-file marker is reached. The -c cmd option can be used to execute short programs in the form of a command-line option—for example, python -c "print('hello world')".
Command-line options appearing after the program name or hyphen (-) are passed to the program in sys.argv, as described in the section “Reading Options and Environment Variables” in Chapter 9, “Input and Output.”
Additionally, the interpreter reads the following environment variables:
Table 10.2 Interpreter Environment Variables
PYTHONPATH specifies a module search path that is inserted into the beginning of sys.path, which is described in Chapter 9. PYTHONSTARTUP specifies a file to execute when the interpreter runs in interactive mode. The PYTHONHOME variable is used to set the location of the Python installation but is rarely needed because Python knows how to find its own libraries and the site-packages directory where extensions are normally installed. If a single directory such as /usr/local is given, the interpreter expects to find all files in that location. If two directories are given, such as /usr/local:/usr/local/sparc-solaris-2.6, the interpreter searches for platform-independent files in the first directory and platform-dependent files in the second. PYTHONHOME has no effect if no valid Python installation exists at the specified location.
The PYTHONIOENCODING environment setting might be of interest to users of Python 3 because it sets both the encoding and error handling of the standard I/O streams. This might be important because Python 3 directly outputs Unicode while running the interactive interpreter prompt. This, in turn, can cause unexpected exceptions merely while inspecting data. For example:
To fix this, you can set the environment variable PYTHONIOENCODING to something such as 'ascii:backslashreplace' or 'utf-8'. Now, you will get this:
On Windows, some of the environment variables such as PYTHONPATH are additionally read from registry entries found in HKEY_LOCAL_MACHINE/Software/Python.
Interactive Sessions
If no program name is given and the standard input to the interpreter is an interactive terminal, Python starts in interactive mode. In this mode, a banner message is printed and the user is presented with a prompt. In addition, the interpreter evaluates the script contained in the PYTHONSTARTUP environment variable (if set). This script is evaluated as if it’s part of the input program (that is, it isn’t loaded using an import statement). One application of this script might be to read a user configuration file such as .pythonrc.
When interactive input is being accepted, two user prompts appear. The >>> prompt appears at the beginning of a new statement; the ... prompt indicates a statement continuation. Here’s an example:
In customized applications, you can change the prompts by modifying the values of sys.ps1 and sys.ps2.
On some systems, Python may be compiled to use the GNU readline library. If enabled, this library provides command histories, completion, and other additions to Python’s interactive mode.
By default, the output of commands issued in interactive mode is generated by printing the output of the built-in repr() function on the result. This can be changed by setting the variable sys.displayhook to a function responsible for displaying results. Here’s an example that truncates long results:
Finally, in interactive mode, it is useful to know that the result of the last operation is stored in a special variable (_). This variable can be used to retrieve the result should you need to use it in subsequent operations. Here’s an example:
The setting of the _ variable occurs in the displayhook() function shown previously. If you redefine displayhook(), your replacement function should also set _ if you want to retain that functionality.
Launching Python Applications
In most cases, you’ll want programs to start the interpreter automatically, rather than first having to start the interpreter manually. On UNIX, this is done by giving the program execute permission and setting the first line of a program to something like this:
On Windows, double-clicking a .py, .pyw, .wpy, .pyc, or .pyo file automatically launches the interpreter. Normally, programs run in a console window unless they’re renamed with a .pyw suffix (in which case the program runs silently). If it’s necessary to supply options to the interpreter, Python can also be started from a .bat file. For example, this .bat file simply runs Python on a script and passes any options supplied on the command prompt along to the interpreter:
Site Configuration Files
A typical Python installation may include a number of third-party modules and packages. To configure these packages, the interpreter first imports the module site. The role of site is to search for package files and to add additional directories to the module search path sys.path. In addition, the site module sets the default encoding for Unicode string conversions.
The site module works by first creating a list of directory names constructed from the values of sys.prefix and sys.exec_prefix as follows:
In addition, if enabled, a user-specific site packages directory may be added to this list (described in the next section).
For each directory in the list, a check is made to see whether the directory exists. If so, it’s added to the sys.path variable. Next, a check is made to see whether it contains any path configuration files (files with a .pth suffix). A path configuration file contains a list of directories, zip files, or .egg files relative to the location of the path file that should be added to sys.path. For example:
Each directory in the path configuration file must be listed on a separate line. Comments and blank lines are ignored. When the site module loads the file, it checks to see whether each directory exists. If so, the directory is added to sys.path. Duplicated items are added to the path only once.
After all paths have been added to sys.path, an attempt is made to import a module named sitecustomize. The purpose of this module is to perform any additional (and arbitrary) site customization. If the import of sitecustomize fails with an ImportError, the error is silently ignored. The import of sitecustomize occurs prior to adding any user directories to sys.path. Thus, placing this file in your own directory has no effect.
The site module is also responsible for setting the default Unicode encoding. By default, the encoding is set to 'ascii'. However, the encoding can be changed by placing code in sitecustomize.py that calls sys.setdefaultencoding() with a new encoding such as 'utf-8'. If you’re willing to experiment, the source code of site can also be modified to automatically set the encoding based on the machine’s locale settings.
Per-user Site Packages
Normally, third-party modules are installed in a way that makes them accessible to all users. However, individual users can install modules and packages in a per-user site directory. On UNIX and Macintosh systems, this directory is found under ~/.local and is named something such as ~/.local/lib/python2.6/site-packages. On Windows systems, this directory is determined by the %APPDATA% environment variable, which is usually something similar to C:Documents and SettingsDavid BeazleyApplication Data. Within that folder, you will find a "PythonPython26site-packages" directory.
If you are writing your own Python modules and packages that you want to use in a library, they can be placed in the per-user site directory. If you are installing third-party modules, you can manually install them in this directory by supplying the --user option to setup.py. For example: python setup.py install --user.
Enabling Future Features
New language features that affect compatibility with older versions of Python are often disabled when they first appear in a release. To enable these features, the statement from_ _future_ _ import feature can be used. Here’s an example:
When used, this statement should appear as the first statement of a module or program. Moreover, the scope of a _ _future_ _ import is restricted only to the module in which it is used. Thus, importing a future feature does not affect the behavior of Python’s library modules or older code that requires the previous behavior of the interpreter to operate correctly.
Currently, the following features have been defined:
Table 10.3 Feature Names in the _ _future_ _ Module
It should be noted that no feature name is ever deleted from _ _future_ _. Thus, even if a feature is turned on by default in a later Python version, no existing code that uses that feature name will break.
Program Termination
A program terminates when no more statements exist to execute in the input program, when an uncaught SystemExit exception is raised (as generated by sys.exit()), or when the interpreter receives a SIGTERM or SIGHUP signal (on UNIX). On exit, the interpreter decrements the reference count of all objects in all the currently known namespaces (and destroys each namespace as well). If the reference count of an object reaches zero, the object is destroyed and its _ _del_ _() method is invoked.
It’s important to note that in some cases the _ _del_ _() method might not be invoked at program termination. This can occur if circular references exist between objects (in which case objects may be allocated but accessible from no known namespace). Although Python’s garbage collector can reclaim unused circular references during execution, it isn’t normally invoked on program termination.
Because there’s no guarantee that _ _del_ _() will be invoked at termination, it may be a good idea to explicitly clean up certain objects, such as open files and network connections. To accomplish this, add specialized cleanup methods (for example, close()) to user-defined objects. Another possibility is to write a termination function and register it with the atexit module, as follows:
The garbage collector can also be invoked in this manner:
One final peculiarity about program termination is that the _ _del_ _ method for some objects may try to access global data or methods defined in other modules. Because these objects may already have been destroyed, a NameError exception occurs in _ _del_ _, and you may get an error such as the following:
If this occurs, it means that _ _del_ _ has aborted prematurely. It also implies that it may have failed in an attempt to perform an important operation (such as cleanly shutting down a server connection). If this is a concern, it’s probably a good idea to perform an explicit shutdown step in your code, rather than rely on the interpreter to destroy objects cleanly at program termination. The peculiar NameError exception can also be eliminated by declaring default arguments in the declaration of the _ _del_ _() method:
In some cases, it may be useful to terminate program execution without performing any cleanup actions. This can be accomplished by calling os._exit(. This function provides an interface to the low-level status)exit() system call responsible for killing the Python interpreter process. When it’s invoked, the program immediately terminates without any further processing or cleanup.