Before we started writing functions, all the code we wrote was at the top level of a module (i.e., not nested in a def), so the names we used either lived in the module itself, or were built-ins predefined by Python (e.g., open).^[35] Functions provide nested namespaces (scopes) that localize the names they use, such that names inside a function won’t clash with those outside it (in a module or another function). Again, functions define a local scope, and modules define a global scope. The two scopes are related as follows:

Note that any type of assignment within a function classifies a name as local: = statements, imports, defs, argument passing, and so on. Also, notice that in-place changes to objects do not classify names as locals; only actual name assignments do. For instance, if the name L is assigned to a list at the top level of a module, a statement like L.append(X) within a function will not classify L as a local, whereas L = X will. In the former case, L will be found in the global scope as usual, and the statement will change the global list.

If the prior section sounds confusing, it really boils down to three simple rules. With a def statement:

In other words, all names assigned inside a function def statement (or a lambda, an expression we’ll meet later) are locals by default; functions can use names in lexically (i.e., physically) enclosing functions and in the global scope, but they must declare globals to change them. Python’s name resolution scheme is sometimes called the LEGB rule, after the scope names:

Figure 16-1 illustrates Python’s four scopes. Note that the second E scope lookup layer—the scopes of enclosing defs or lambdas—can technically correspond to more than one lookup layer. It only comes into play when you nest functions within functions.^[36]

Figure 16-1. The LEGB scope lookup rule. When a variable is referenced, Python searches for it in this order: in the local scope, in any enclosing functions’ local scopes, in the global scope, and, finally, in the built-in scope. The first occurrence wins. The place in your code where a variable is assigned usually determines its scope.

Also, keep in mind that these rules only apply to simple variable names (such as spam). In Parts V and VI, we’ll see that qualified attribute names (such as object.spam) live in particular objects and follow a completely different set of lookup rules than the scope ideas covered here. Attribute references (names following periods) search one or more objects, not scopes, and may invoke something called “inheritance” (discussed in Part VI).

Let’s look at a larger example that demonstrates scope ideas. Suppose we write the following code in a module file:

# Global scope
X = 99                # X and func assigned in module: global

def func(Y):          # Y and Z assigned in function: locals
    # Local scope
    Z = X + Y         # X is a global
    return Z

func(1)               # func in module: result=100

This module and the function it contains use a number of names to do their business. Using Python’s scope rules, we can classify the names as follows:

The whole point behind this name segregation scheme is that local variables serve as temporary names that you need only while a function is running. For instance, in the preceding example, the argument Y and the addition result Z exist only inside the function; these names don’t interfere with the enclosing module’s namespace (or any other function, for that matter).

The local/global distinction also makes functions easier to understand, as most of the names a function uses appear in the function itself, not at some arbitrary place in a module. Also, because you can be sure that local names will not be changed by some remote function in your program, they tend to make programs easier to debug.

We’ve been talking about the built-in scope in the abstract, but it’s a bit simpler than you may think. Really, the built-in scope is just a built-in module called _ _builtin_ _, but you have to import _ _builtin_ _ to use built-in because the name builtin is not itself built-in.

No, I’m serious! The built-in scope is implemented as a standard library module named _ _builtin_ _, but that name itself is not placed in the built-in scope, so you have to import it in order to inspect it. Once you do, you can run a dir call to see which names are predefined:

>>> import _  _builtin_  _
>>> dir(_  _builtin_  _)
['ArithmeticError', 'AssertionError', 'AttributeError',
'DeprecationWarning', 'EOFError', 'Ellipsis',
    ...many more names omitted...
'str', 'super', 'tuple', 'type', 'unichr', 'unicode',
'vars', 'xrange', 'zip']

The names in this list constitute the built-in scope in Python; roughly the first half are built-in exceptions, and the second half are built-in functions. Because Python automatically searches this module last in its LEGB lookup rule, you get all the names in this list for free; that is, you can use them without importing any modules. Thus, there are two ways to refer to a built-in function—by taking advantage of the LEGB rule, or by manually importing the _ _builtin_ _ module:

>>> zip# The normal way
<built-in function zip>

>>> import _  _builtin_  _# The hard way
>>> _  _builtin_  _.zip
<built-in function zip>

The second of these approaches is sometimes useful in advanced work. The careful reader might also notice that because the LEGB lookup procedure takes the first occurrence of a name that it finds, names in the local scope may override variables of the same name in both the global and built-in scopes, and global names may override built-ins. A function can, for instance, create a local variable called open by assigning to it:

def hider(  ):
    open = 'spam'              # Local variable, hides built-in
    ...
    open('data.txt')           # This won't open a file now in this scope!

However, this will hide the built-in function called open that lives in the built-in (outer) scope. It’s also usually a bug, and a nasty one at that, because Python will not issue a warning message about it (there are times in advanced programming where you may really want to replace a built-in name by redefining it in your code).^[37]

Functions can similarly hide global variables of the same name with locals:

X = 88                         # Global X

def func(  ):
    X = 99                     # Local X: hides global

func(  )
print X                        # Prints 88: unchanged

Here, the assignment within the function creates a local X that is a completely different variable from the global X in the module outside the function. Because of this, there is no way to change a name outside a function without adding a global declaration to the def (as described in the next section).

By default, names assigned in functions are locals, so if you want to change names outside functions, you have to write extra code (global statements). This is by design—as is common in Python, you have to say more to do the “wrong” thing. Although there are times when globals are useful, variables assigned in a def are local by default because that is normally the best policy. Changing globals can lead to well-known software engineering problems: because the variables’ values are dependent on the order of calls to arbitrarily distant functions, programs can become difficult to debug.

Consider this module file, for example:

X = 99
def func1(  ):
    global X
    X = 88

def func2(  ):
    global X
    X = 77

Now, imagine that it is your job to modify or reuse this module file. What will the value of X be here? Really, that question has no meaning unless qualified with a point of reference in time—the value of X is timing-dependent, as it depends on which function was called last (something we can’t tell from this file alone).

The net effect is that to understand this code, you have to trace the flow of control through the entire program. And, if you need to reuse or modify the code, you have to keep the entire program in your head all at once. In this case, you can’t really use one of these functions without bringing along the other. They are dependent (that is, coupled) on the global variable. This is the problem with globals—they generally make code more difficult to understand and use than code consisting of self-contained functions that rely on locals.

On the other hand, short of using object-oriented programming and classes, global variables are probably the most straightforward way to retain state information (information that a function needs to remember for use the next time it is called) in Python—local variables disappear when the function returns, but globals do not. Other techniques, such as default mutable arguments and enclosing function scopes, can achieve this, too, but they are more complex than pushing values out to the global scope for retention.

Some programs designate a single module to collect globals; as long as this is expected, it is not as harmful. Also, programs that use multithreading to do parallel processing in Python essentially depend on global variables—they become shared memory between functions running in parallel threads, and so act as a communication device (threading is beyond this book’s scope; see the follow-up texts mentioned in the Preface for more details).

For now, though, especially if you are relatively new to programming, avoid the temptation to use globals whenever you can (try to communicate with passed-in arguments and return values instead). Six months from now, both you and your coworkers will be happy you did.

Here’s another scope-related issue: although we can change variables in another file directly, we usually shouldn’t. Consider these two module files:

# first.py
X = 99

# second.py
import first
first.X = 88

The first defines a variable X, which the second changes by assignment. Notice that we must import the first module into the second file to get to its variable—as we’ve learned, each module is a self-contained namespace (package of variables), and we must import one module to see inside it from another. Really, in terms of this chapter’s topic, the global scope of a module file becomes the attribute namespace of the module object once it is imported—importers automatically have access to all of the file’s global variables, so a file’s global scope essentially morphs into an object’s attribute namespace when it is imported.

After importing the first module, the second module assigns its variable a new value. The problem with the assignment, however, is that it is too implicit: whoever’s charged with maintaining or reusing the first module probably has no clue that some arbitrarily far-removed module on the import chain can change X out from under him. In fact, the second module may be in a completely different directory, and so difficult to find. Again, this sets up too strong a coupling between the two files—because they are both dependent on the value of the variable X, it’s difficult to understand or reuse one file without the other.

Here again, the best prescription is generally not to do this—the best way to communicate across file boundaries is to call functions, passing in arguments, and getting back return values. In this specific case, we would probably be better off coding an accessor function to manage the change:

# first.py
X = 99

def setX(new):
    global X
    X = new

# second.py
import first
first.setX(88)

This requires more code, but it makes a huge difference in terms of readability and maintainability—when a person reading the first module by itself sees a function, he will know that it is a point of interface, and will expect the change to the variable X. Although we cannot prevent cross-file changes from happening, common sense dictates that they should be minimized unless widely accepted across the program.

Interestingly, because global-scope variables morph into the attributes of a loaded module object, we can emulate the global statement by importing the enclosing module and assigning to its attributes, as in the following example module file. Code in this file imports the enclosing module by name, and then by indexing sys.modules, the loaded modules table (more on this table in Chapter 21):

# thismod.py

var = 99                              # Global variable == module attribute

def local(  ):
    var = 0                           # Change local var

def glob1(  ):
    global var                        # Declare global (normal)
    var += 1                          # Change global var

def glob2(  ):
    var = 0                           # Change local var
    import thismod                    # Import myself
    thismod.var += 1                  # Change global var

def glob3(  ):
    var = 0                           # Change local var
    import sys                        # Import system table
    glob = sys.modules['thismod']     # Get module object (or use _  _name_  _)
    glob.var += 1                     # Change global var

def test(  ):
    print var
    local(); glob1(); glob2(  ); glob3(  )
    print var

When run, this adds 3 to the global variable (only the first function does not impact it):

>>> import thismod
>>> thismod.test(  )
99
102
>>> thismod.var
102

This works, and it illustrates the equivalence of globals to module attributes, but it’s much more work than using the global statement to make your intentions explicit.

With the addition of nested function scopes, variable lookup rules become slightly more complex. Within a function:

Notice that the global declaration still maps variables to the enclosing module. When nested functions are present, variables in enclosing functions may only be referenced, not changed. To clarify these points, let’s illustrate with some real code.

Here is an example of a nested scope:

def f1(  ):
    x = 88
    def f2(  ):
        print x
    f2(  )

f1(  )                                     # Prints 88

First off, this is legal Python code: the def is simply an executable statement that can appear anywhere any other statement can—including nested in another def. Here, the nested def runs while a call to the function f1 is running; it generates a function, and assigns it to the name f2, a local variable within f1’s local scope. In a sense, f2 is a temporary function that only lives during the execution of (and is only visible to code in) the enclosing f1.

But, notice what happens inside f2: when it prints the variable x, it refers to the x that lives in the enclosing f1 function’s local scope. Because functions can access names in all physically enclosing def statements, the x in f2 is automatically mapped to the x in f1, by the LEGB lookup rule.

This enclosing scope lookup works even if the enclosing function has already returned. For example, the following code defines a function that makes and returns another function:

def f1(  ):
    x = 88
    def f2(  ):
        print x
    return f2

action = f1(  )                            # Make, return function
action(  )                                 # Call it now: prints 88

In this code, the call to action is really running the function we named f2 when f1 ran. f2 remembers the enclosing scope’s x in f1, even though f1 is no longer active.

>>> def maker(N):
...     def action(X):
...         return X ** N
...     return action
...

>>> f = maker(2)# Pass 2 to N
>>> f
<function action at 0x014720B0>

>>> f(3)# Pass 3 to X, N remembers 2
9
>>> f(4)# 4 ** 2
16

>>> g = maker(3)
>>> g(3)# 3 ** 3
27
>>> f(3)# 3 ** 2
9

def f1(  ):
    x = 88
    def f2(x=x):
        print x
    f2(  )

f1(  )                                     # Prints 88

>>> def f1(  ):
...     x = 88
...     f2(x)
...
>>> def f2(x):
...     print x
...
>>> f1(  )
88

def func(  ):
    x = 4
    action = (lambda n: x ** n)          # x remembered from enclosing def
    return action

x = func(  )
print x(2)                               # Prints 16, 4 ** 2

def func(  ):
    x = 4
    action = (lambda n, x=x: x ** n)     # Pass x in manually

>>> def makeActions(  ):
...     acts = []
...     for i in range(5):# Tries to remember each i
...         acts.append(lambda x: i ** x)    # All remember same last i!
...     return acts
...
>>> acts = makeActions(  )
>>> acts[0]
<function <lambda> at 0x012B16B0>

>>> acts[0](2)# All are 4 ** 2, value of last i
16
>>> acts[2](2)# This should be 2 ** 2
16
>>> acts[4](2)# This should be 4 ** 2
16

>>> def makeActions(  ):
...     acts = []
...     for i in range(5):# Use defaults instead
...         acts.append(lambda x, i=i: i ** x)# Remember current i
...     return acts
...
>>> acts = makeActions(  )
>>> acts[0](2)# 0 ** 2
0
>>> acts[2](2)# 2 ** 2
4
>>> acts[4](2)# 4 ** 2
16

>>> def f1(  ):
...     x = 99
...     def f2(  ):
...         def f3(  ):
...             print x# Found in f1's local scope!
...         f3(  )
...     f2(  )
...
>>> f1(  )
99

Here’s an example that illustrates some of these properties at work:

>>> def changer(a, b):# Function
...    a = 2# Changes local name's value only
...    b[0] = 'spam'# Changes shared object in-place
...
>>> X = 1
>>> L = [1, 2]# Caller
>>> changer(X, L)# Pass immutable and mutable objects
>>> X, L# X is unchanged, L is different
(1, ['spam', 2])

In this code, the changer function assigns values to argument a, and to a component in the object referenced by argument b. The two assignments within the function are only slightly different in syntax, but have radically different results:

Figure 16-2 illustrates the name/object bindings that exist immediately after the function has been called, and before its code has run.

Figure 16-2. References: arguments. Because arguments are passed by assignment, argument names may share objects with variables at the call. Hence, in-place changes to mutable arguments in a function can impact the caller. Here, a and b in the function initially reference the objects referenced by variables X and L when the function is first called. Changing the list through variable b makes L appear different after the call returns.

If this example is still confusing, it may help to notice that the effect of the automatic assignments of the passed-in arguments is the same as running a series of simple assignment statements. In terms of the first argument, the assignment has no effect on the caller:

>>> X = 1
>>> a = X# They share the same object
>>> a = 2# Resets 'a' only, 'X' is still 1
>>> print X
1

But, the assignment through the second argument does affect a variable at the call because it is an in-place object change:

>>> L = [1, 2]
>>> b = L# They share the same object
>>> b[0] = 'spam'# In-place change: 'L' sees the change too
>>> print L
['spam', 2]

If you recall our discussions about shared mutable objects in Chapter 6 and Chapter 9, you’ll recognize the phenomenon at work: changing a mutable object in-place can impact other references to that object. Here, the effect is to make one of the arguments work like an output of the function.

Arguments are passed to functions by reference (a.k.a. pointer) by default in Python because that is what we normally want—it means we can pass large objects around our programs without making multiple copies along the way, and we can easily update these objects as we go. If we don’t want in-place changes within functions to impact objects we pass to them, though, we can simply make explicit copies of mutable objects, as we learned in Chapter 6. For function arguments, we can always copy the list at the point of call:

L = [1, 2]
changer(X, L[:])           # Pass a copy, so our 'L' does not change

We can also copy within the function itself, if we never want to change passed-in objects, regardless of how the function is called:

def changer(a, b):
   b = b[:]# Copy input list so we don't impact caller
   a = 2
   b[0] = 'spam'           # Changes our list copy only

Both of these copying schemes don’t stop the function from changing the object—they just prevent those changes from impacting the caller. To really prevent changes, we can always convert to immutable objects to force the issue. Tuples, for example, throw an exception when changes are attempted:

L = [1, 2]
changer(X, tuple(L))       # Pass a tuple, so changes are errors

This scheme uses the built-in tuple function, which builds a new tuple out of all the items in a sequence (really, any iterable). It’s also something of an extreme—because it forces the function to be written to never change passed-in arguments, this solution might impose more limitations on the function than it should, and so should generally be avoided. You never know when changing arguments might come in handy for other calls in the future. Using this technique will also make the function lose the ability to call any list-specific methods on the argument, including methods that do not change the object in-place.

The main point to remember here is that functions might update mutable objects passed into them (e.g., lists and dictionaries). This isn’t necessarily a problem, and it often serves useful purposes. But, you do have to be aware of this property—if objects change out from under you unexpectedly, check whether a called function might be responsible, and make copies when objects are passed if needed.

We’ve already discussed the return statement and used it in a few examples. Here’s a neat trick: because return can send back any sort of object, it can return multiple values by packaging them in a tuple or other collection type. In fact, although Python doesn’t support what some languages label “call-by-reference” argument passing, we can usually simulate it by returning tuples and assigning the results back to the original argument names in the caller:

>>> def multiple(x, y):
...     x = 2# Changes local names only
...     y = [3, 4]
...     return x, y# Return new values in a tuple
...
>>> X = 1
>>> L = [1, 2]
>>> X, L = multiple(X, L)# Assign results to caller's names
>>> X, L
(2, [3, 4])

It looks like the code is returning two values here, but it’s really just one—a two-item tuple with the optional surrounding parentheses omitted. After the call returns, we can use tuple assignment to unpack the parts of the returned tuple. (If you’ve forgotten why this works, flip back to "Tuples" in Chapter 4, and "Assignment Statements" in Chapter 11.) The net effect of this coding pattern is to simulate the output parameters of other languages by explicit assignments. X and L change after the call, but only because the code said so.

This is all simpler in code than the preceding descriptions may imply. Python matches names by position by default like most other languages. For instance, if you define a function that requires three arguments, you must call it with three arguments:

>>> def f(a, b, c): print a, b, c
...

Here, we pass them by position—a is matched to 1, b is matched to 2, and so on:

>>> f(1, 2, 3)
1 2 3

>>> f(c=3, b=2, a=1)
1 2 3

>>> f(1, c=3, b=2)
1 2 3

func(name='Bob', age=40, job='dev')

>>> def f(a, b=2, c=3): print a, b, c
...

>>> f(1)
1 2 3
>>> f(a=1)
1 2 3

>>> f(1, 4)
1 4 3
>>> f(1, 4, 5)
1 4 5

>>> f(1, c=6)
1 2 6

The last two matching extensions, * and **, are designed to support functions that take any number of arguments. Both can appear in either the function definition, or a function call, and they have related purposes in the two locations.

>>> def f(*args): print args
...

>>> f(  )
(  )
>>> f(1)
(1,)
>>> f(1,2,3,4)
(1, 2, 3, 4)

>>> def f(**args): print args
...
>>> f(  )
{  }
>>> f(a=1, b=2)
{'a': 1, 'b': 2}

>>> def f(a, *pargs, **kargs): print a, pargs, kargs
...
>>> f(1, 2, 3, x=1, y=2)
1 (2, 3) {'y': 2, 'x': 1}

>>> def func(a, b, c, d): print a, b, c, d
...
>>> args = (1, 2)
>>> args += (3, 4)
>>> func(*args)
1 2 3 4

>>> args = {'a': 1, 'b': 2, 'c': 3}
>>> args['d'] = 4
>>> func(**args)
1 2 3 4

>>> func(*(1, 2), **{'d': 4, 'c': 4})
1 2 4 4

>>> func(1, *(2, 3), **{'d': 4})
1 2 3 4

>>> func(1, c=3, *(2,), **{'d': 4})
1 2 3 4

Here is a slightly larger example that demonstrates keywords and defaults in action. In the following, the caller must always pass at least two arguments (to match spam and eggs), but the other two are optional. If they are omitted, Python assigns toast and ham to the defaults specified in the header:

def func(spam, eggs, toast=0, ham=0):   # First 2 required
    print (spam, eggs, toast, ham)

func(1, 2)                              # Output: (1, 2, 0, 0)
func(1, ham=1, eggs=0)                  # Output: (1, 0, 0, 1)
func(spam=1, eggs=0)                    # Output: (1, 0, 0, 0)
func(toast=1, eggs=2, spam=3)           # Output: (3, 2, 1, 0)
func(1, 2, 3, 4)                        # Output: (1, 2, 3, 4)

Notice again that when keyword arguments are used in the call, the order in which the arguments are listed doesn’t matter; Python matches by name, not by position. The caller must supply values for spam and eggs, but they can be matched by position or by name. Also, notice that the form name=value means different things in the call and the def (a keyword in the call and a default in the header).

To make this more concrete, let’s work through an exercise that demonstrates a practical application of argument-matching tools. Suppose you want to code a function that is able to compute the minimum value from an arbitrary set of arguments and an arbitrary set of object data types. That is, the function should accept zero or more arguments—as many as you wish to pass. Moreover, the function should work for all kinds of Python object types: numbers, strings, lists, lists of dictionaries, files, and even None.

The first requirement provides a natural example of how the * feature can be put to good use—we can collect arguments into a tuple, and step over each in turn with a simple for loop. The second part of the problem definition is easy: because every object type supports comparisons, we don’t have to specialize the function per type (an application of polymorphism); we can simply compare objects blindly, and let Python perform the correct sort of comparison.

def min1(*args):
    res = args[0]
    for arg in args[1:]:
        if arg < res:
            res = arg
    return res

def min2(first, *rest):
    for arg in rest:
        if arg < first:
            first = arg
    return first

def min3(*args):
    tmp = list(args)            # Or, in Python 2.4+: return sorted(args)[0]
    tmp.sort(  )
    return tmp[0]

print min1(3,4,1,2)
print min2("bb", "aa")
print min3([2,2], [1,1], [3,3])

% python mins.py
1
aa
[1, 1]

def minmax(test, *args):
    res = args[0]
    for arg in args[1:]:
        if test(arg, res):
            res = arg
    return res

def lessthan(x, y): return x < y                # See also: lambda
def grtrthan(x, y): return x > y

print minmax(lessthan, 4, 2, 1, 5, 6, 3)        # Self-test code
print minmax(grtrthan, 4, 2, 1, 5, 6, 3)

% python minmax.py
1
6

Now, let’s look at a more useful example of special argument-matching modes at work. At the end of the prior chapter, we wrote a function that returned the intersection of two sequences (it picked out items that appeared in both). Here is a version that intersects an arbitrary number of sequences (one or more), by using the varargs matching form *args to collect all the passed-in arguments. Because the arguments come in as a tuple, we can process them in a simple for loop. Just for fun, we’ll code a union function that also accepts an arbitrary number of arguments to collect items that appear in any of the operands:

def intersect(*args):
    res = []
    for x in args[0]:                  # Scan first sequence
        for other in args[1:]:         # For all other args
            if x not in other: break   # Item in each one?
        else:                          # No: break out of loop
            res.append(x)              # Yes: add items to end
    return res

def union(*args):
    res = []
    for seq in args:                   # For all args
        for x in seq:                  # For all nodes
            if not x in res:
                res.append(x)          # Add new items to result
    return res

Because these are tools worth reusing (and they’re too big to retype interactively), we’ll store the functions in a module file called inter2.py (more on modules in Part V). In both functions, the arguments passed in at the call come in as the args tuple. As in the original intersect, both work on any kind of sequence. Here, they are processing strings, mixed types, and more than two sequences:

% python
>>> from inter2 import intersect, union
>>> s1, s2, s3 = "SPAM", "SCAM", "SLAM"

>>> intersect(s1, s2), union(s1, s2)# Two operands
(['S', 'A', 'M'], ['S', 'P', 'A', 'M', 'C'])

>>> intersect([1,2,3], (1,4))# Mixed types
[1]

>>> intersect(s1, s2, s3)# Three operands
['S', 'A', 'M']

>>> union(s1, s2, s3)
['S', 'P', 'A', 'M', 'C', 'L']

If you choose to use and combine the special argument-matching modes, Python will ask you to follow these ordering rules:

If you mix arguments in any other order, you will get a syntax error because the combinations can be ambiguous. Python internally carries out the following steps to match arguments before assignment:

After this, Python checks to make sure each argument is passed just one value; if not, an error is raised. This is as complicated as it looks, but tracing Python’s matching algorithm will help you to understand some convoluted cases, especially when modes are mixed. We’ll postpone looking at additional examples of these special matching modes until the exercises at the end of Part IV.

As you can see, advanced argument-matching modes can be complex. They are also entirely optional; you can get by with just simple positional matching, and it’s probably a good idea to do so when you’re starting out. However, because some Python tools make use of them, some general knowledge of these modes is important.

from Tkinter import *
widget = Button(text="Press me", command=someFunction)