Before we started writing functions, all the code we wrote was at the top level of a module (i.e., not nested in a def), so the names we used either lived in the module itself or were built-ins predefined by Python (e.g., open). Functions provide nested namespaces (scopes) that localize the names they use, such that names inside a function won’t clash with those outside it (in a module or another function). Again, functions define a local scope, and modules define a global scope. The two scopes are related as follows:

There are a few subtleties to note here. First, keep in mind that code typed at the interactive command prompt follows these same rules. You may not know it yet, but code run interactively is really entered into a built-in module called __main__; this module works just like a module file, but results are echoed as you go. Because of this, interactively created names live in a module, too, and thus follow the normal scope rules: they are global to the interactive session. You’ll learn more about modules in the next part of this book.

Also note that any type of assignment within a function classifies a name as local. This includes = statements, module names in import, function names in def, function argument names, and so on. If you assign a name in any way within a def, it will become a local to that function.

Conversely, in-place changes to objects do not classify names as locals; only actual name assignments do. For instance, if the name L is assigned to a list at the top level of a module, a statement L = X within a function will classify L as a local, but L.append(X) will not. In the latter case, we are changing the list object that L references, not L itself—L is found in the global scope as usual, and Python happily modifies it without requiring a global (or nonlocal) declaration. As usual, it helps to keep the distinction between names and objects clear: changing an object is not an assignment to a name.

If the prior section sounds confusing, it really boils down to three simple rules. With a def statement:

In other words, all names assigned inside a function def statement (or a lambda, an expression we’ll meet later) are locals by default. Functions can freely use names assigned in syntactically enclosing functions and the global scope, but they must declare such nonlocals and globals in order to change them.

Python’s name-resolution scheme is sometimes called the LEGB rule, after the scope names:

Figure 17-1 illustrates Python’s four scopes. Note that the second scope lookup layer, E—the scopes of enclosing defs or lambdas—can technically correspond to more than one lookup layer. This case only comes into play when you nest functions within functions, and it is addressed by the nonlocal statement.^[36]

Figure 17-1. The LEGB scope lookup rule. When a variable is referenced, Python searches for it in this order: in the local scope, in any enclosing functions’ local scopes, in the global scope, and finally in the built-in scope. The first occurrence wins. The place in your code where a variable is assigned usually determines its scope. In Python 3, nonlocal declarations can also force names to be mapped to enclosing function scopes, whether assigned or not.

Also keep in mind that these rules apply only to simple variable names (e.g., spam). In Parts V and VI, we’ll see that qualified attribute names (e.g., object.spam) live in particular objects and follow a completely different set of lookup rules than those covered here. References to attribute names following periods (.) search one or more objects, not scopes, and may invoke something called “inheritance”; more on this in Part VI of this book.

Let’s look at a larger example that demonstrates scope ideas. Suppose we wrote the following code in a module file:

# Global scope
X = 99                # X and func assigned in module: global

def func(Y):          # Y and Z assigned in function: locals
    # Local scope
    Z = X + Y         # X is a global
    return Z

func(1)               # func in module: result=100

This module and the function it contains use a number of names to do their business. Using Python’s scope rules, we can classify the names as follows:

The whole point behind this name-segregation scheme is that local variables serve as temporary names that you need only while a function is running. For instance, in the preceding example, the argument Y and the addition result Z exist only inside the function; these names don’t interfere with the enclosing module’s namespace (or any other function, for that matter).

The local/global distinction also makes functions easier to understand, as most of the names a function uses appear in the function itself, not at some arbitrary place in a module. Also, because you can be sure that local names will not be changed by some remote function in your program, they tend to make programs easier to debug and modify.

We’ve been talking about the built-in scope in the abstract, but it’s a bit simpler than you may think. Really, the built-in scope is just a built-in module called builtins, but you have to import builtins to query built-ins because the name builtins is not itself built-in....

No, I’m serious! The built-in scope is implemented as a standard library module named builtins, but that name itself is not placed in the built-in scope, so you have to import it in order to inspect it. Once you do, you can run a dir call to see which names are predefined. In Python 3.0:

>>> import builtins
>>> dir(builtins)
['ArithmeticError', 'AssertionError', 'AttributeError', 'BaseException',
'BufferError', 'BytesWarning', 'DeprecationWarning', 'EOFError', 'Ellipsis',
    ...many more names omitted...
'print', 'property', 'quit', 'range', 'repr', 'reversed', 'round', 'set',
'setattr', 'slice', 'sorted', 'staticmethod', 'str', 'sum', 'super', 'tuple',
'type', 'vars', 'zip']

The names in this list constitute the built-in scope in Python; roughly the first half are built-in exceptions, and the second half are built-in functions. Also in this list are the special names None, True, and False, though they are treated as reserved words. Because Python automatically searches this module last in its LEGB lookup, you get all the names in this list “for free;” that is, you can use them without importing any modules. Thus, there are really two ways to refer to a built-in function—by taking advantage of the LEGB rule, or by manually importing the builtins module:

>>> zip                         # The normal way
<class 'zip'>

>>> import builtins             # The hard way
>>> builtins.zip
<class 'zip'>

The second of these approaches is sometimes useful in advanced work. The careful reader might also notice that because the LEGB lookup procedure takes the first occurrence of a name that it finds, names in the local scope may override variables of the same name in both the global and built-in scopes, and global names may override built-ins. A function can, for instance, create a local variable called open by assigning to it:

def hider():
    open = 'spam'              # Local variable, hides built-in
    ...
    open('data.txt')           # This won't open a file now in this scope!

However, this will hide the built-in function called open that lives in the built-in (outer) scope. It’s also usually a bug, and a nasty one at that, because Python will not issue a warning message about it (there are times in advanced programming where you may really want to replace a built-in name by redefining it in your code). Functions can similarly hide global variables of the same name with locals:

X = 88                         # Global X

def func():
    X = 99                     # Local X: hides global

func()
print(X)                       # Prints 88: unchanged

Here, the assignment within the function creates a local X that is a completely different variable from the global X in the module outside the function. Because of this, there is no way to change a name outside a function without adding a global (or nonlocal) declaration to the def, as described in the next section.^[37]

By default, names assigned in functions are locals, so if you want to change names outside functions you have to write extra code (e.g., global statements). This is by design—as is common in Python, you have to say more to do the potentially “wrong” thing. Although there are times when globals are useful, variables assigned in a def are local by default because that is normally the best policy. Changing globals can lead to well-known software engineering problems: because the variables’ values are dependent on the order of calls to arbitrarily distant functions, programs can become difficult to debug.

Consider this module file, for example:

X = 99
def func1():
    global X
    X = 88

def func2():
    global X
    X = 77

Now, imagine that it is your job to modify or reuse this module file. What will the value of X be here? Really, that question has no meaning unless it’s qualified with a point of reference in time—the value of X is timing-dependent, as it depends on which function was called last (something we can’t tell from this file alone).

The net effect is that to understand this code, you have to trace the flow of control through the entire program. And, if you need to reuse or modify the code, you have to keep the entire program in your head all at once. In this case, you can’t really use one of these functions without bringing along the other. They are dependent on (that is, coupled with) the global variable. This is the problem with globals—they generally make code more difficult to understand and use than code consisting of self-contained functions that rely on locals.

On the other hand, short of using object-oriented programming and classes, global variables are probably the most straightforward way to retain shared state information (information that a function needs to remember for use the next time it is called) in Python—local variables disappear when the function returns, but globals do not. Other techniques, such as default mutable arguments and enclosing function scopes, can achieve this, too, but they are more complex than pushing values out to the global scope for retention.

Some programs designate a single module to collect globals; as long as this is expected, it is not as harmful. In addition, programs that use multithreading to do parallel processing in Python commonly depend on global variables—they become shared memory between functions running in parallel threads, and so act as a communication device.^[38]

For now, though, especially if you are relatively new to programming, avoid the temptation to use globals whenever you can—try to communicate with passed-in arguments and return values instead. Six months from now, both you and your coworkers will be happy you did.

Here’s another scope-related issue: although we can change variables in another file directly, we usually shouldn’t. Module files were introduced in Chapter 3 and are covered in more depth in the next part of this book. To illustrate their relationship to scopes, consider these two module files:

# first.py
X = 99                    # This code doesn't know about second.py

# second.py
import first
print(first.X)            # Okay: references a name in another file
first.X = 88              # But changing it can be too subtle and implicit

The first defines a variable X, which the second prints and then changes by assignment. Notice that we must import the first module into the second file to get to its variable at all—as we’ve learned, each module is a self-contained namespace (package of variables), and we must import one module to see inside it from another. That’s the main point about modules: by segregating variables on a per-file basis, they avoid name collisions across files.

Really, though, in terms of this chapter’s topic, the global scope of a module file becomes the attribute namespace of the module object once it is imported—importers automatically have access to all of the file’s global variables, because a file’s global scope morphs into an object’s attribute namespace when it is imported.

After importing the first module, the second module prints its variable and then assigns it a new value. Referencing the module’s variable to print it is fine—this is how modules are linked together into a larger system normally. The problem with the assignment, however, is that it is far too implicit: whoever’s charged with maintaining or reusing the first module probably has no clue that some arbitrarily far-removed module on the import chain can change X out from under him at runtime. In fact, the second module may be in a completely different directory, and so difficult to notice at all.

Although such cross-file variable changes are always possible in Python, they are usually much more subtle than you will want. Again, this sets up too strong a coupling between the two files—because they are both dependent on the value of the variable X, it’s difficult to understand or reuse one file without the other. Such implicit cross-file dependencies can lead to inflexible code at best, and outright bugs at worst.

Here again, the best prescription is generally to not do this—the best way to communicate across file boundaries is to call functions, passing in arguments and getting back return values. In this specific case, we would probably be better off coding an accessor function to manage the change:

# first.py
X = 99

def setX(new):
    global X
    X = new

# second.py
import first
first.setX(88)

This requires more code and may seem like a trivial change, but it makes a huge difference in terms of readability and maintainability—when a person reading the first module by itself sees a function, that person will know that it is a point of interface and will expect the change to the X. In other words, it removes the element of surprise that is rarely a good thing in software projects. Although we cannot prevent cross-file changes from happening, common sense dictates that they should be minimized unless widely accepted across the program.

Interestingly, because global-scope variables morph into the attributes of a loaded module object, we can emulate the global statement by importing the enclosing module and assigning to its attributes, as in the following example module file. Code in this file imports the enclosing module, first by name, and then by indexing the sys.modules loaded modules table (more on this table in Chapter 21):

# thismod.py

var = 99                              # Global variable == module attribute

def local():
    var = 0                           # Change local var

def glob1():
    global var                        # Declare global (normal)
    var += 1                          # Change global var

def glob2():
    var = 0                           # Change local var
    import thismod                    # Import myself
    thismod.var += 1                  # Change global var

def glob3():
    var = 0                           # Change local var
    import sys                        # Import system table
    glob = sys.modules['thismod']     # Get module object (or use __name__)
    glob.var += 1                     # Change global var

def test():
    print(var)
    local(); glob1(); glob2(); glob3()
    print(var)

When run, this adds 3 to the global variable (only the first function does not impact it):

>>> import thismod
>>> thismod.test()
99
102
>>> thismod.var
102

This works, and it illustrates the equivalence of globals to module attributes, but it’s much more work than using the global statement to make your intentions explicit.

As we’ve seen, global allows us to change names in a module outside a function. It has a cousin named nonlocal that can be used to change names in enclosing functions, too, but to understand how that can be useful, we first need to explore enclosing functions in general.

With the addition of nested function scopes, variable lookup rules become slightly more complex. Within a function:

Notice that the global declaration still maps variables to the enclosing module. When nested functions are present, variables in enclosing functions may be referenced, but they require nonlocal declarations to be changed.

To clarify the prior section’s points, let’s illustrate with some real code. Here is what an enclosing function scope looks like:

X = 99                   # Global scope name: not used

def f1():
    X = 88               # Enclosing def local
    def f2():
        print(X)         # Reference made in nested def
    f2()

f1()                     # Prints 88: enclosing def local

First off, this is legal Python code: the def is simply an executable statement, which can appear anywhere any other statement can—including nested in another def. Here, the nested def runs while a call to the function f1 is running; it generates a function and assigns it to the name f2, a local variable within f1’s local scope. In a sense, f2 is a temporary function that lives only during the execution of (and is visible only to code in) the enclosing f1.

But notice what happens inside f2: when it prints the variable X, it refers to the X that lives in the enclosing f1 function’s local scope. Because functions can access names in all physically enclosing def statements, the X in f2 is automatically mapped to the X in f1, by the LEGB lookup rule.

This enclosing scope lookup works even if the enclosing function has already returned. For example, the following code defines a function that makes and returns another function:

def f1():
    X = 88
    def f2():
        print(X)         # Remembers X in enclosing def scope
    return f2            # Return f2 but don't call it

action = f1()            # Make, return function
action()                 # Call it now: prints 88

In this code, the call to action is really running the function we named f2 when f1 ran. f2 remembers the enclosing scope’s X in f1, even though f1 is no longer active.

>>> def maker(N):
...     def action(X):                    # Make and return action
...         return X ** N                 # action retains N from enclosing scope
...     return action
...

>>> f = maker(2)                          # Pass 2 to N
>>> f
<function action at 0x014720B0>

>>> f(3)                                  # Pass 3 to X, N remembers 2: 3 ** 2
9
>>> f(4)                                  # 4 ** 2
16

>>> g = maker(3)                          # g remembers 3, f remembers 2
>>> g(3)                                  # 3 ** 3
27
>>> f(3)                                  # 3 ** 2
9

def f1():
    x = 88
    def f2(x=x):                # Remember enclosing scope X with defaults
        print(x)
    f2()

f1()                            # Prints 88

>>> def f1():
...     x = 88                  # Pass x along instead of nesting
...     f2(x)                   # Forward reference okay
...
>>> def f2(x):
...     print(x)
...
>>> f1()
88

def func():
    x = 4
    action = (lambda n: x ** n)          # x remembered from enclosing def
    return action

x = func()
print(x(2))                              # Prints 16, 4 ** 2

def func():
    x = 4
    action = (lambda n, x=x: x ** n)     # Pass x in manually
    return action

>>> def makeActions():
...     acts = []
...     for i in range(5):                       # Tries to remember each i
...         acts.append(lambda x: i ** x)        # All remember same last i!
...     return acts
...
>>> acts = makeActions()
>>> acts[0]
<function <lambda> at 0x012B16B0>

>>> acts[0](2)                                   # All are 4 ** 2, value of last i
16
>>> acts[2](2)                                   # This should be 2 ** 2
16
>>> acts[4](2)                                   # This should be 4 ** 2
16

>>> def makeActions():
...     acts = []
...     for i in range(5):                       # Use defaults instead
...         acts.append(lambda x, i=i: i ** x)   # Remember current i
...     return acts
...
>>> acts = makeActions()
>>> acts[0](2)                                   # 0 ** 2
0
>>> acts[2](2)                                   # 2 ** 2
4
>>> acts[4](2)                                   # 4 ** 2
16

>>> def f1():
...     x = 99
...     def f2():
...         def f3():
...             print(x)        # Found in f1's local scope!
...         f3()
...     f2()
...
>>> f1()
99

Python 3.0 introduces a new nonlocal statement, which has meaning only inside a function:

def func():
    nonlocal name1, name2, ...

This statement allows a nested function to change one or more names defined in a syntactically enclosing function’s scope. In Python 2.X (including 2.6), when one function def is nested in another, the nested function can reference any of the names defined by assignment in the enclosing def’s scope, but it cannot change them. In 3.0, declaring the enclosing scopes’ names in a nonlocal statement enables nested functions to assign and thus change such names as well.

This provides a way for enclosing functions to provide writeable state information, remembered when the nested function is later called. Allowing the state to change makes it more useful to the nested function (imagine a counter in the enclosing scope, for instance). In 2.X, programmers usually achieve similar goals by using classes or other schemes. Because nested functions have become a more common coding pattern for state retention, though, nonlocal makes it more generally applicable.

Besides allowing names in enclosing defs to be changed, the nonlocal statement also forces the issue for references—just like the global statement, nonlocal causes searches for the names listed in the statement to begin in the enclosing defs’ scopes, not in the local scope of the declaring function. That is, nonlocal also means “skip my local scope entirely.”

In fact, the names listed in a nonlocal must have been previously defined in an enclosing def when the nonlocal is reached, or an error is raised. The net effect is much like global: global means the names reside in the enclosing module, and nonlocal means they reside in an enclosing def. nonlocal is even more strict, though—scope search is restricted to only enclosing defs. That is, nonlocal names can appear only in enclosing defs, not in the module’s global scope or built-in scopes outside the defs.

The addition of nonlocal does not alter name reference scope rules in general; they still work as before, per the “LEGB” rule described earlier. The nonlocal statement mostly serves to allow names in enclosing scopes to be changed rather than just referenced. However, global and nonlocal statements do both restrict the lookup rules somewhat, when coded in a function:

In Python 2.6, references to enclosing def scope names are allowed, but not assignment. However, you can still use classes with explicit attributes to achieve the same changeable state information effect as nonlocals (and you may be better off doing so in some contexts); globals and function attributes can sometimes accomplish similar goals as well. More on this in a moment; first, let’s turn to some working code to make this more concrete.

On to some examples, all run in 3.0. References to enclosing def scopes work as they do in 2.6. In the following, tester builds and returns the function nested, to be called later, and the state reference in nested maps the local scope of tester using the normal scope lookup rules:

C:\misc>c:python30python

>>> def tester(start):
...     state = start             # Referencing nonlocals works normally
...     def nested(label):
...         print(label, state)   # Remembers state in enclosing scope
...     return nested
...
>>> F = tester(0)
>>> F('spam')
spam 0
>>> F('ham')
ham 0

Changing a name in an enclosing def’s scope is not allowed by default, though; this is the normal case in 2.6 as well:

>>> def tester(start):
...     state = start
...     def nested(label):
...         print(label, state)
...         state += 1            # Cannot change by default (or in 2.6)
...     return nested
...
>>> F = tester(0)
>>> F('spam')
UnboundLocalError: local variable 'state' referenced before assignment

>>> def tester(start):
...     state = start             # Each call gets its own state
...     def nested(label):
...         nonlocal state        # Remembers state in enclosing scope
...         print(label, state)
...         state += 1            # Allowed to change it if nonlocal
...     return nested
...
>>> F = tester(0)
>>> F('spam')                     # Increments state on each call
spam 0
>>> F('ham')
ham 1
>>> F('eggs')
eggs 2

>>> G = tester(42)                # Make a new tester that starts at 42
>>> G('spam')
spam 42

>>> G('eggs')                     # My state information updated to 43
eggs 43

>>> F('bacon')                    # But F's is where it left off: at 3
bacon 3                           # Each call has different state information

>>> def tester(start):
...     def nested(label):
...         nonlocal state        # Nonlocals must already exist in enclosing def!
...         state = 0
...         print(label, state)
...     return nested
...
SyntaxError: no binding for nonlocal 'state' found

>>> def tester(start):
...     def nested(label):
...         global state          # Globals don't have to exist yet when declared
...         state = 0             # This creates the name in the module now
...         print(label, state)
...     return nested
...
>>> F = tester(0)
>>> F('abc')
abc 0
>>> state
0

>>> spam = 99
>>> def tester():
...     def nested():
...         nonlocal spam         # Must be in a def, not the module!
...         print('Current=', spam)
...         spam += 1
...     return nested
...
SyntaxError: no binding for nonlocal 'spam' found

Given the extra complexity of nested functions, you might wonder what the fuss is about. Although it’s difficult to see in our small examples, state information becomes crucial in many programs. There are a variety of ways to “remember” information across function and method calls in Python. While there are tradeoffs for all, nonlocal does improve this story for enclosing scope references—the nonlocal statement allows multiple copies of changeable state to be retained in memory and addresses simple state-retention needs where classes may not be warranted.

As we saw in the prior section, the following code allows state to be retained and modified in an enclosing scope. Each call to tester creates a little self-contained package of changeable information, whose names do not clash with any other part of the program:

def tester(start):
    state = start                      # Each call gets its own state
    def nested(label):
        nonlocal state                 # Remembers state in enclosing scope
        print(label, state)
        state += 1                     # Allowed to change it if nonlocal
    return nested

F = tester(0)
F('spam')

Unfortunately, this code only works in Python 3.0. If you are using Python 2.6, other options are available, depending on your goals. The next three sections present some alternatives.

>>> def tester(start):
...     global state                   # Move it out to the module to change it
...     state = start                  # global allows changes in module scope
...     def nested(label):
...         global state
...         print(label, state)
...         state += 1
...     return nested
...
>>> F = tester(0)
>>> F('spam')                          # Each call increments shared global state
spam 0
>>> F('eggs')
eggs 1

>>> G = tester(42)                     # Resets state's single copy in global scope
>>> G('toast')
toast 42

>>> G('bacon')
bacon 43

>>> F('ham')                           # Oops -- my counter has been overwritten!
ham 44

>>> class tester:                          # Class-based alternative (see Part VI)
...     def __init__(self, start):         # On object construction,
...         self.state = start             # save state explicitly in new object
...     def nested(self, label):
...         print(label, self.state)       # Reference state explicitly
...         self.state += 1                # Changes are always allowed
...
>>> F = tester(0)                          # Create instance, invoke __init__
>>> F.nested('spam')                       # F is passed to self
spam 0
>>> F.nested('ham')
ham 1

>>> G = tester(42)                         # Each instance gets new copy of state
>>> G.nested('toast')                      # Changing one does not impact others
toast 42
>>> G.nested('bacon')
bacon 43

>>> F.nested('eggs')                       # F's state is where it left off
eggs 2
>>> F.state                                # State may be accessed outside class
3

>>> class tester:
...     def __init__(self, start):
...         self.state = start
...     def __call__(self, label):         # Intercept direct instance calls
...         print(label, self.state)       # So .nested() not required
...         self.state += 1
...
>>> H = tester(99)
>>> H('juice')                             # Invokes __call__
juice 99
>>> H('pancakes')
pancakes 100

>>> def tester(start):
...     def nested(label):
...         print(label, nested.state)     # nested is in enclosing scope
...         nested.state += 1              # Change attr, not nested itself
...     nested.state = start               # Initial state after func defined
...     return nested
...
>>> F = tester(0)
>>> F('spam')                              # F is a 'nested' with state attached
spam 0
>>> F('ham')
ham 1
>>> F.state                                # Can access state outside functions too
2
>>>
>>> G = tester(42)                         # G has own state, doesn't overwrite F's
>>> G('eggs')
eggs 42
>>> F('ham')
ham 2