On UNIX systems, the most common way to specify the behavior of a program when it runs is to add parameters to the command line of a program. Even when a program is not run from the command line but is instead run using fork and exec (more on this later in this chapter), a command line is constructed when it is invoked. This makes it a universal way of controlling the behavior of your programs.

You may have seen, for instance, that many programs can be run so that they provide you with some basic information about how they should be run. Python enables you to do this with -h:

$ python -h
usage: python30 [option] ... [-c cmd | -m mod | file | -] [arg] ...
Options and arguments (and corresponding environment variables):
-c cmd : program passed in as string (terminates option list)
-d     : debug output from parser (also PYTHONDEBUG=x)
-E     : ignore environment variables (such as PYTHONPATH)
[ etc. ]

In the past, different conventions were available on different UNIX platforms to specify these options, but this has largely resulted in two forms of options being used by most projects: the short form, such as the help-message producing option to Python, and a long form, such as - -help for help.

To accept these sorts of options makes sense. Ideally, you'd like to offer a short and a long form of commands that are common, and allow each one to optionally take a specification. So if you wanted to write a program that had a configuration file that the user could specify, you may want one option like -c short for experienced users, but provide a longer option too, like - -config-file. In either case, you'd want them to be the same function in your program to save you time, but you'd like to give users the freedom to use these options however they want to use them.

The getopt module provides two functions to make this standard convention easy to use: getopt.getopt and getopt.gnu_getopt. They are both basically the same. The basic getopt only works until the first non-option is encountered — nothing else is checked.

For getopt to be useful, you have to know what options you want to be useful. Normally, it's considered the least you can do for your users to write programs that provide them with information about how to run the program, such as how Python prints information with the -h option.

In addition, it's often very useful to have a configuration file. Using these ideas as a starting point, you could start your new programs so that -h and - -help both produce a minimal message about how your program is used, and using -c or - -config-file=file would enable you to specify a configuration file that is different from the default configuration:

import sys
import getopt
# Remember, the first thing in the sys.argv list is the name of the command
# You don't need that.
cmdline_params = sys.argv[1:]

opts, args = getopt.getopt(cmdline_params, 'hc:', ['help', 'config='])

for option, parameter in opts:

    if option == '-h' or option == '- -help':
        print("This program can be run with either -h or - -help for this message,")
        print("or with -c or - -config=<file> to specify a different
configuration file")

if option in ('-c', '- -config'): # this means the same as the above
        print("Using configuration file %s" % parameter)

When long options are used and require a parameter (like - -config in the preceding example), the equal sign must connect the option and the value of the parameter. However, when short options are used, one or more space or tab characters can separate the option from its corresponding value. This distinction is to duplicate the behavior of the options on older UNIX machines that persist to the modern day. They persist because so many people expect that behavior. What can you do?

The preceding code snippet, if run in a program with the parameters -c test -h - -config=secondtest, produces the following output:

[('-c', 'test'), ('-h', ''), ('- -config', 'secondtest')] []
Using configuration file test
This program can be run with either -h or - -help for this message,
or with -c or - -config=<file> to specify a different configuration file

Using configuration file secondtest

Note how the second instance of the configuration file is accepted silently; and when it is reached, the same code that sets the config file is revisited so that the second instance is used.

The second list, the args data, is an empty list because all of the options provided to the program on the command line were valid options, or valid parameters to options. If you inserted other strings in the middle of your options, the normal getopt would behave differently. If the parameters used were instead -c test useless_information_here -h - -config=secondtest, the output would say a lot less, and the args array would have a lot more in it.

[('-c', 'test')] ['useless_information_here', '-h', '- -config=secondtest']
Using configuration file test

The gnu_getopt lets you mix and match on the command line so that non-options can appear anywhere in the midst of the options, with more options parsed afterward instead of stopping there:

opts, args = getopt.gnu_getopt(cmdline_params, 'hc:', ['help', 'config='])

for option, parameter in opts:

    if option == '-h' or option == '- -help':
        print("This program can be run with either -h or - -help for this message,")
        print("or with -c or - -config=<file> to specify a different
configuration file")

    if option in ('-c', '- -config'): # this means the same as the above
        print("Using configuration file %s" % parameter)

The important point to note is that if you use something that doesn't meet the criteria for an option (by beginning with a − or a +, or following an option that takes a parameter), the two behave differently. Using the options -c test useless_information_here -h - -config=secondtest, the gnu_getopt function provides the following output, with the odd duck being the only part of the command line left in the args array:

[('-c', 'test'), ('-h', ''), ('- -config', 'secondtest')]
['useless_information_here']
Using configuration file test
This program can be run with either -h or - -help for this message,
or with -c or - -config=<file> to specify a different configuration file

Using configuration file secondtest

In UNIX and UNIX-like operating systems, the main way of performing certain kinds of subtasks is to create a new process running a new program. On UNIX systems, this is done using a system call that is available in Python by using os.fork. This actually tells the computer to copy everything about the currently running program into a newly created program that is separate, but almost entirely identical. The only difference is that the return value for os.fork is zero in the newly created process (the child), and is the process ID (PID) of the newly created process in the original process (the parent). This can be difficult to understand, and the only way to really get it is to use it a few times and to read some other material on fork and exec that's available online. (Or talk to your nearest UNIX guru.)

Based on the one critical difference, a parent and child can perform different functions. The parent can wait for an event while the child processes, or vice versa. The code to do this is simple and common, but it works only on UNIX and UNIX-like systems:

import os
pid = os.fork()
if pid == 0: # This is the child
    print("this is the child")
else:
    print("the child is pid %d" % pid)

One of the most common things to do after an os.fork call is to call os.execl immediately afterward to run another program. os.execl is an instruction to replace the running program with a new program, so the calling program goes away, and a new program appears in its place (in case you didn't already know this, UNIX systems use the fork and exec method to run all programs):

import os
pid = os.fork()
# fork and exec together
print("second test")
if pid == 0: # This is the child
    print("this is the child")
    print("I'm going to exec another program now")
    os.execl('/bin/cat', 'cat', '/etc/motd')
else:
    print("the child is pid %d" % pid)
    os.wait()

The os.wait function instructs Python that you want the parent to not do anything until the child process returns. It is very useful to know how this works because it works well only under UNIX and UNIX-like platforms such as Linux. Windows also has a mechanism for starting up new processes.

To make the common task of starting a new program easier, Python offers a single family of functions that combines os.fork and os.exec on UNIX-like systems, and enables you to do something similar on Windows platforms. When you want to just start up a new program, you can use the os.spawn family of functions. They are a family because they are named similarly, but each one has slightly different behaviors.

On UNIX-like systems, the os.spawn family contains spawnl, spawnle, spawnlp, spawnlpe, spawnv, spawnve, spawnvp, and spawnvpe. On Windows systems, the spawn family contains only spawnl, spawnle, spawnv, and spawnve.

In each case, the letters after the word spawn mean something specific. The v means that a list (a vector is what the v actually stands for) will be passed in as the parameters. This allows a command to be run with very different commands from one instance to the next without needing to alter the program at all. The l variation just requires a simple list of parameters.

The e occurrences require that a dictionary containing names and values that will be used as the environment for the newly created program will be passed in instead of using the current environment.

The p occurrence uses the value of the PATH key in the environment dictionary to find the program. The p variants are available only on UNIX-like platforms. The least of what this means is that on Windows your programs must have a completely qualified path to be usable by the os.spawn calls, or you have to search the path yourself:

import os, sys
if sys.platform == 'win32':
    print("Running on a windows platform")
    command = "C:\winnt\system32\cmd.exe"
    params = []

if sys.platform == 'linux2':
    print("Running on a Linux system, identified by %s" % sys.platform)
    command = '/bin/uname'
    params = ['uname', '-a']

print("Running %s" % command)
os.spawnv(os.P_WAIT, command, params)

Of course, this example will only work on a limited range of systems. You can use the contents of sys.platform on your own computer and for something besides linux2 in case you are on another UNIX system such as Solaris, Mac OS X, AIX, or others.

When you do this, you can either wait for the process to return (that is, until it finishes and exits) or you can tell Python that you'd prefer to allow the program to run on its own, and that you will confirm that it completed successfully later. This is done with the os.P_ family of values. Depending on which one you set, you will be given a different behavior when an os.spawn function returns.

If you need only the most basic invocation of a new command, sometimes the easiest way to do this is to use the os.system function. If you are running a program and just want to wait for it to finish, you can use this function very simply:

# Now system
if sys.platform == 'win32':
    print("Running on a windows platform")
    command = "cmd.exe"

if sys.platform == 'linux2':
    print("Running Linux")
    command = "uname -a"

os.system(command)

This can be much simpler because it uses the facilities that the operating system provides, and that users expect normally, to search for the program you want to run, and it defaults to waiting for the child process to finish.

Creating a new process using fork or spawn can sometimes be too much effort and not provide enough benefit. Specifically, regarding the too much effort, when a program grows to be large, fork has to copy everything in the program to the new program and the system must have enough resources to handle that. Another downside for fork is that sometimes when you need your program to do many things at the same time, some things may need to wait while others need to proceed. When this happens, you want to have all of the different components communicating their needs to other parts of the program.

Using multiple processes, this becomes very difficult. These processes share many things because the child was originally created using the data in the parent. However, they are separate entities — completely separate. Because of this, it can be very tricky to make two processes work together cooperatively.

So, to make some complex situations where subprocesses are not appropriate workable, the concept of threads is available.

Many cooperative threads of program execution are able to exist at the same time in the same program. Each one has potentially different objects, with different state, but they can all communicate, while also being able to run semi-independently of one another.

This means that in many situations, using threads is much more convenient than using a separate process. Note that the following example uses subclassing, which is covered in Chapter 10. To see how this works, try running it with a fairly large parameter, say two million (2000000):

import math
from threading import Thread
import time

class SquareRootCalculator:

"""This class spawns a separate thread to calculate a bunch of square
    roots, and checks in it once a second until it finishes."""

    def __init__(self, target):
        """Turn on the calculator thread and, while waiting for it to
        finish, periodically monitor its progress."""
        self.results = []
        counter = self.CalculatorThread(self, target)
        print("Turning on the calculator thread...")
        counter.start()
        while len(self.results) < target:
            print("%d square roots calculated so far." % len(self.results))
            time.sleep(1)
        print("Calculated %s square root(s); the last one is sqrt(%d)=%f" %
              (target, len(self.results), self.results[−1]))

    class CalculatorThread(Thread):
        """A separate thread which actually does the calculations."""

        def __init__(self, controller, target):
            """Set up this thread, including making it a daemon thread
            so that the script can end without waiting for this thread to
            finish."""
            Thread.__init__(self)
            self.controller = controller
            self.target = target
            self.setDaemon(True)

        def run(self):
            """Calculate square roots for all numbers between 1 and the target,
            inclusive."""
            for i in range(1, self.target+1):
               self.controller.results.append(math.sqrt(i))

if __name__ == '__main__':
    import sys
    limit = None
    if len(sys.argv) > 1:
        limit = sys.argv[1]
        try:
            limit = int(limit)
        except ValueError:
            print("Usage: %s [number of square roots to calculate]"
                  % sys.argv[0])
    SquareRootCalculator(limit)

For many situations, such as network servers (see Chapter 16) or graphical user interfaces (see Chapter 13), threads make much more sense because they require less work from you as the programmer, and fewer resources from the system.

Note how separate threads can access each other's names and data easily. This makes it very easy to keep track of what different threads are doing, which is an important convenience.