What’s new in this chapter

Coverage of the functools.lru_cache decorator was updated to include “Simplified usage of lru_cache in Python 3.8”.

Section “Single Dispatch Generic Functions” was expanded and now uses type hints, the preferred way to use functools.singledispatch since Python 3.7.

“Parameterized Decorators” now includes a class-based example, Example 9-27.

I moved Chapter 10—Design Patterns with First-Class Functions—to the end of part III to improve the flow of the book. Section “Decorator-Enhanced Strategy Pattern” is now in that chapter, along with other variations of the Strategy design pattern using callables.

We start with a very gentle introduction to decorators, and then proceed with the rest of the items listed in the chapter opening.

Decorators 101

A decorator is a callable that takes another function as argument (the decorated function). Python also supports class decorators—they are covered in [Link to Come].

A decorator may perform some processing with the decorated function, and returns it or replaces it with another function or callable object.

In other words, assuming an existing decorator named decorate, this code:

@decorate
def target():
    print('running target()')

Has the same effect as writing this:

def target():
    print('running target()')

target = decorate(target)

The end result is the same: at the end of either of these snippets, the target name is bound to whatever function is returned by decorate(target)—which may be the function initially named target, or may be a different function.

To confirm that the decorated function is replaced, see the console session in Example 9-1.

>>> def deco(func):
...     def inner():
...         print('running inner()')
...     return inner  
...
>>> @deco
... def target():  
...     print('running target()')
...
>>> target()  
running inner()
>>> target  
<function deco.<locals>.inner at 0x10063b598>

deco returns its inner function object.

target is decorated by deco.

Invoking the decorated target actually runs inner.

Inspection reveals that target is a now a reference to inner.

Strictly speaking, decorators are just syntactic sugar. As we just saw, you can always simply call a decorator like any regular callable, passing another function. Sometimes that is actually convenient, especially when doing metaprogramming—changing program behavior at runtime.

To summarize: the first crucial fact about decorators is that they have the power to replace the decorated function with a different one. The second crucial fact is that they are executed immediately when a module is loaded. This is explained next.

When Python Executes Decorators

A key feature of decorators is that they run right after the decorated function is defined. That is usually at import time (i.e., when a module is loaded by Python). Consider registration.py in Example 9-2.

registry = []  

def register(func):  
    print(f'running register({func})')  
    registry.append(func)  
    return func  

@register  
def f1():
    print('running f1()')

@register
def f2():
    print('running f2()')

def f3():  
    print('running f3()')

def main():  
    print('running main()')
    print('registry ->', registry)
    f1()
    f2()
    f3()

if __name__=='__main__':
    main()  

registry will hold references to functions decorated by @register.

register takes a function as argument.

Display what function is being decorated, for demonstration.

Include func in registry.

Return func: we must return a function; here we return the same received as argument.

f1 and f2 are decorated by @register.

f3 is not decorated.

main displays the registry, then calls f1(), f2(), and f3().

main() is only invoked if registration.py runs as a script.

The output of running registration.py as a script looks like this:

$ python3 registration.py
running register(<function f1 at 0x100631bf8>)
running register(<function f2 at 0x100631c80>)
running main()
registry -> [<function f1 at 0x100631bf8>, <function f2 at 0x100631c80>]
running f1()
running f2()
running f3()

Note that register runs (twice) before any other function in the module. When register is called, it receives as an argument the function object being decorated—for example, <function f1 at 0x100631bf8>.

After the module is loaded, the registry list holds references to the two decorated functions: f1 and f2. These functions, as well as f3, are only executed when explicitly called by main.

If registration.py is imported (and not run as a script), the output is this:

>>> import registration
running register(<function f1 at 0x10063b1e0>)
running register(<function f2 at 0x10063b268>)

At this time, if you inspect registry, this is what you see:

>>> registration.registry
[<function f1 at 0x10063b1e0>, <function f2 at 0x10063b268>]

The main point of Example 9-2 is to emphasize that function decorators are executed as soon as the module is imported, but the decorated functions only run when they are explicitly invoked. This highlights the difference between what Pythonistas call import time and runtime.

Considering how decorators are commonly employed in real code, Example 9-2 is unusual in two ways:

• The decorator function is defined in the same module as the decorated functions. A real decorator is usually defined in one module and applied to functions in other modules.

• The register decorator returns the same function passed as argument. In practice, most decorators define an inner function and return it.

Even though the register decorator in Example 9-2 returns the decorated function unchanged, that technique is not useless. Similar decorators are used in many Python frameworks to add functions to some central registry—for example, a registry mapping URL patterns to functions that generate HTTP responses. Such registration decorators may or may not change the decorated function.

We will see a registration decorator applied in “Decorator-Enhanced Strategy Pattern” (Chapter 10).

Most decorators do change the decorated function. They usually do it by defining an inner function and returning it to replace the decorated function. Code that uses inner functions almost always depends on closures to operate correctly. To understand closures, we need to take a step back and have a close look at how variable scopes work in Python.

Variable Scope Rules

In Example 9-3, we define and test a function that reads two variables: a local variable a—defined as function parameter—and variable b that is not defined anywhere in the function.

>>> def f1(a):
...     print(a)
...     print(b)
...
>>> f1(3)
3
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in f1
NameError: global name 'b' is not defined

The error we got is not surprising. Continuing from Example 9-3, if we assign a value to a global b and then call f1, it works:

>>> b = 6
>>> f1(3)
3
6

Now, let’s see an example that may surprise you.

Take a look at the f2 function in Example 9-4. Its first two lines are the same as f1 in Example 9-3, then it makes an assignment to b, and prints its value. But it fails at the second print, before the assignment is made.

>>> b = 6
>>> def f2(a):
...     print(a)
...     print(b)
...     b = 9
...
>>> f2(3)
3
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in f2
UnboundLocalError: local variable 'b' referenced before assignment

Note that the output starts with 3, which proves that the print(a) statement was executed. But the second one, print(b), never runs. When I first saw this I was surprised, thinking that 6 should be printed, because there is a global variable b and the assignment to the local b is made after print(b).

But the fact is, when Python compiles the body of the function, it decides that b is a local variable because it is assigned within the function. The generated bytecode reflects this decision and will try to fetch b from the local environment. Later, when the call f2(3) is made, the body of f2 fetches and prints the value of the local variable a, but when trying to fetch the value of local variable b it discovers that b is unbound.

This is not a bug, but a design choice: Python does not require you to declare variables, but assumes that a variable assigned in the body of a function is local. This is much better than the behavior of JavaScript, which does not require variable declarations either, but if you do forget to declare that a variable is local (with var), you may clobber a global variable without knowing.

If we want the interpreter to treat b as a global variable and still assign a new value to it within the function, we use the global declaration:

>>> b = 6
>>> def f3(a):
...     global b
...     print(a)
...     print(b)
...     b = 9
...
>>> f3(3)
3
6
>>> b
9

After this closer look at how variable scopes work in Python, we can tackle closures in the next section, “Closures”. If you are curious about the bytecode differences between the functions in Examples 9-3 and 9-4, see the following sidebar.

Closures

In the blogosphere, closures are sometimes confused with anonymous functions. The reason why many confuse them is historic: defining functions inside functions is not so common, until you start using anonymous functions. And closures only matter when you have nested functions. So a lot of people learn both concepts at the same time.

Actually, a closure is a function with an extended scope that encompasses nonglobal variables referenced in the body of the function but not defined there. It does not matter whether the function is anonymous or not; what matters is that it can access nonglobal variables that are defined outside of its body.

This is a challenging concept to grasp, and is better approached through an example.

Consider an avg function to compute the mean of an ever-growing series of values; for example, the average closing price of a commodity over its entire history. Every day a new price is added, and the average is computed taking into account all prices so far.

Starting with a clean slate, this is how avg could be used:

>>> avg(10)
10.0
>>> avg(11)
10.5
>>> avg(12)
11.0

Where does avg come from, and where does it keep the history of previous values?

For starters, Example 9-7 is a class-based implementation.

class Averager():

    def __init__(self):
        self.series = []

    def __call__(self, new_value):
        self.series.append(new_value)
        total = sum(self.series)
        return total/len(self.series)

The Averager class creates instances that are callable:

>>> avg = Averager()
>>> avg(10)
10.0
>>> avg(11)
10.5
>>> avg(12)
11.0

Now, Example 9-8 is a functional implementation, using the higher-order function make_averager.

def make_averager():
    series = []

    def averager(new_value):
        series.append(new_value)
        total = sum(series)
        return total/len(series)

    return averager

When invoked, make_averager returns an averager function object. Each time an averager is called, it appends the passed argument to the series, and computes the current average, as shown in Example 9-9.

>>> avg = make_averager()
>>> avg(10)
10.0
>>> avg(11)
10.5
>>> avg(12)
11.0

Note the similarities of the examples: we call Averager() or make_averager() to get a callable object avg that will update the historical series and calculate the current mean. In Example 9-7, avg is an instance of Averager, and in Example 9-8 it is the inner function, averager. Either way, we just call avg(n) to include n in the series and get the updated mean.

It’s obvious where the avg of the Averager class keeps the history: the self.series instance attribute. But where does the avg function in the second example find the series?

Note that series is a local variable of make_averager because the assignment series = [] happens in the body of that function. But when avg(10) is called, make_averager has already returned, and its local scope is long gone.

Within averager, series is a free variable. This is a technical term meaning a variable that is not bound in the local scope. See Figure 9-1.

Figure 9-1. The closure for averager extends the scope of that function to include the binding for the free variable series.

Inspecting the returned averager object shows how Python keeps the names of local and free variables in the __code__ attribute that represents the compiled body of the function. Example 9-10 demonstrates.

>>> avg.__code__.co_varnames
('new_value', 'total')
>>> avg.__code__.co_freevars
('series',)

The value for series is kept in the __closure__ attribute of the returned function avg. Each item in avg.__closure__ corresponds to a name in avg.__code__.co_freevars. These items are cells, and they have an attribute called cell_contents where the actual value can be found. Example 9-11 shows these attributes.

>>> avg.__code__.co_freevars
('series',)
>>> avg.__closure__
(<cell at 0x107a44f78: list object at 0x107a91a48>,)
>>> avg.__closure__[0].cell_contents
[10, 11, 12]

To summarize: a closure is a function that retains the bindings of the free variables that exist when the function is defined, so that they can be used later when the function is invoked and the defining scope is no longer available.

Note that the only situation in which a function may need to deal with external variables that are nonglobal is when it is nested in another function.

The nonlocal Declaration

Our previous implementation of make_averager was not efficient. In Example 9-8, we stored all the values in the historical series and computed their sum every time averager was called. A better implementation would only store the total and the number of items so far, and compute the mean from these two numbers.

Example 9-12 is a broken implementation, just to make a point. Can you see where it breaks?

def make_averager():
    count = 0
    total = 0

    def averager(new_value):
        count += 1
        total += new_value
        return total / count

    return averager

If you try Example 9-12, here is what you get:

>>> avg = make_averager()
>>> avg(10)
Traceback (most recent call last):
  ...
UnboundLocalError: local variable 'count' referenced before assignment
>>>

The problem is that the statement count += 1 actually means the same as count = count + 1, when count is a number or any immutable type. So we are actually assigning to count in the body of averager, and that makes it a local variable. The same problem affects the total variable.

We did not have this problem in Example 9-8 because we never assigned to the series name; we only called series.append and invoked sum and len on it. So we took advantage of the fact that lists are mutable.

But with immutable types like numbers, strings, tuples, etc., all you can do is read, never update. If you try to rebind them, as in count = count + 1, then you are implicitly creating a local variable count. It is no longer a free variable, and therefore it is not saved in the closure.

To work around this, the nonlocal keyword was introduced in Python 3. It lets you declare a variable as a free variable even when it is assigned within the function. If a new value is assigned to a nonlocal variable, the binding stored in the closure is changed. A correct implementation of our newest make_averager looks like Example 9-13.

def make_averager():
    count = 0
    total = 0

    def averager(new_value):
        nonlocal count, total
        count += 1
        total += new_value
        return total / count

    return averager

Now that we have Python closures covered, we can effectively implement decorators with nested functions.

Implementing a Simple Decorator

Example 9-14 is a decorator that clocks every invocation of the decorated function and displays the elapsed time, the arguments passed, and the result of the call.

import time


def clock(func):
    def clocked(*args):  
        t0 = time.perf_counter()
        result = func(*args)  
        elapsed = time.perf_counter() - t0
        name = func.__name__
        arg_str = ', '.join(repr(arg) for arg in args)
        print(f'[{elapsed:0.8f}s] {name}({arg_str}) -> {result!r}')
        return result
    return clocked  

Define inner function clocked to accept any number of positional arguments.

This line only works because the closure for clocked encompasses the func free variable.

Return the inner function to replace the decorated function.

Example 9-15 demonstrates the use of the clock decorator.

import time
from clockdeco import clock


@clock
def snooze(seconds):
    time.sleep(seconds)


@clock
def factorial(n):
    return 1 if n < 2 else n*factorial(n-1)


if __name__ == '__main__':
    print('*' * 40, 'Calling snooze(.123)')
    snooze(.123)
    print('*' * 40, 'Calling factorial(6)')
    print('6! =', factorial(6))

The output of running Example 9-15 looks like this:

$ python3 clockdeco_demo.py
**************************************** Calling snooze(.123)
[0.12363791s] snooze(0.123) -> None
**************************************** Calling factorial(6)
[0.00000095s] factorial(1) -> 1
[0.00002408s] factorial(2) -> 2
[0.00003934s] factorial(3) -> 6
[0.00005221s] factorial(4) -> 24
[0.00006390s] factorial(5) -> 120
[0.00008297s] factorial(6) -> 720
6! = 720

@clock
def factorial(n):
    return 1 if n < 2 else n*factorial(n-1)

def factorial(n):
    return 1 if n < 2 else n*factorial(n-1)

factorial = clock(factorial)

>>> import clockdeco_demo
>>> clockdeco_demo.factorial.__name__
'clocked'
>>>

import time
import functools


def clock(func):
    @functools.wraps(func)
    def clocked(*args, **kwargs):
        t0 = time.perf_counter()
        result = func(*args, **kwargs)
        elapsed = time.perf_counter() - t0
        name = func.__name__
        arg_lst = []
        if args:
            arg_lst.append(', '.join(repr(arg) for arg in args))
        if kwargs:
            pairs = [f'{k}={v!r}' for k, v in kwargs.items()]
            arg_lst.append(', '.join(pairs))
        arg_str = ', '.join(arg_lst)
        print(f'[{elapsed:0.8f}s] {name}({arg_str}) -> {result!r}')
        return result
    return clocked

Decorators in the Standard Library

Python has three built-in functions that are designed to decorate methods: property, classmethod, and staticmethod. We will discuss property in [Link to Come] and the others in “classmethod Versus staticmethod”.

In Example 9-16 we saw another important decorator: functools.wraps, a helper for building well-behaved decorators. Two of the most interesting decorators in the standard library are lru_cache and singledispatch. Both are defined in the functools module. We’ll cover them next.

from clockdeco import clock


@clock
def fibonacci(n):
    if n < 2:
        return n
    return fibonacci(n-2) + fibonacci(n-1)


if __name__ == '__main__':
    print(fibonacci(6))

$ python3 fibo_demo.py
[0.00000042s] fibonacci(0) -> 0
[0.00000049s] fibonacci(1) -> 1
[0.00006115s] fibonacci(2) -> 1
[0.00000031s] fibonacci(1) -> 1
[0.00000035s] fibonacci(0) -> 0
[0.00000030s] fibonacci(1) -> 1
[0.00001084s] fibonacci(2) -> 1
[0.00002074s] fibonacci(3) -> 2
[0.00009189s] fibonacci(4) -> 3
[0.00000029s] fibonacci(1) -> 1
[0.00000027s] fibonacci(0) -> 0
[0.00000029s] fibonacci(1) -> 1
[0.00000959s] fibonacci(2) -> 1
[0.00001905s] fibonacci(3) -> 2
[0.00000026s] fibonacci(0) -> 0
[0.00000029s] fibonacci(1) -> 1
[0.00000997s] fibonacci(2) -> 1
[0.00000028s] fibonacci(1) -> 1
[0.00000030s] fibonacci(0) -> 0
[0.00000031s] fibonacci(1) -> 1
[0.00001019s] fibonacci(2) -> 1
[0.00001967s] fibonacci(3) -> 2
[0.00003876s] fibonacci(4) -> 3
[0.00006670s] fibonacci(5) -> 5
[0.00016852s] fibonacci(6) -> 8
8

import functools

from clockdeco import clock


@functools.lru_cache  
@clock  
def fibonacci(n):
    if n < 2:
        return n
    return fibonacci(n-2) + fibonacci(n-1)


if __name__ == '__main__':
    print(fibonacci(6))

@alpha
@beta
def my_fn():
    ...

my_fn = alpha(beta(my_fn))

$ python3 fibo_demo_lru.py
[0.00000043s] fibonacci(0) -> 0
[0.00000054s] fibonacci(1) -> 1
[0.00006179s] fibonacci(2) -> 1
[0.00000070s] fibonacci(3) -> 2
[0.00007366s] fibonacci(4) -> 3
[0.00000057s] fibonacci(5) -> 5
[0.00008479s] fibonacci(6) -> 8
8

@lru_cache
def costly_function(a, b):
    ...

@lru_cache()
def costly_function(a, b):
    ...

@lru_cache(maxsize=128, typed=False)
def costly_function(a, b):
    ...

import html

def htmlize(obj):
    content = html.escape(repr(obj))
    return f'<pre>{content}</pre>'

>>> htmlize({1, 2, 3})  
'<pre>{1, 2, 3}</pre>'
>>> htmlize(abs)
'<pre>&lt;built-in function abs&gt;</pre>'
>>> htmlize('Heimlich & Co.
- a game')  
'<p>Heimlich &amp; Co.<br>
- a game</p>'
>>> htmlize(42)  
'<pre>42 (0x2a)</pre>'
>>> print(htmlize(['alpha', 66, {3, 2, 1}]))  
<ul>
<li><p>alpha</p></li>
<li><pre>66 (0x42)</pre></li>
<li><pre>{1, 2, 3}</pre></li>
</ul>
>>> htmlize(True)  
'<pre>True</pre>'
>>> htmlize(fractions.Fraction(2, 3))  
'<pre>2/3</pre>'
>>> htmlize(2/3)   
'<pre>0.6666666666666666 (2/3)</pre>'
>>> htmlize(decimal.Decimal('0.02380952'))
'<pre>0.02380952 (1/42)</pre>'

from functools import singledispatch
from collections import abc
import fractions
import decimal
import html
import numbers

@singledispatch  
def htmlize(obj: object) -> str:
    content = html.escape(repr(obj))
    return f'<pre>{content}</pre>'

@htmlize.register  
def _(text: str) -> str:  
    content = html.escape(text).replace('
', '<br>
')
    return f'<p>{content}</p>'

@htmlize.register  
def _(seq: abc.Sequence) -> str:
    inner = '</li>
<li>'.join(htmlize(item) for item in seq)
    return '<ul>
<li>' + inner + '</li>
</ul>'

@htmlize.register  
def _(n: numbers.Integral) -> str:
    return f'<pre>{n} (0x{n:x})</pre>'

@htmlize.register  
def _(n: bool) -> str:
    return f'<pre>{n}</pre>'

@htmlize.register(fractions.Fraction)  
def _(x) -> str:
    frac = fractions.Fraction(x)
    return f'<pre>{frac.numerator}/{frac.denominator}</pre>'

@htmlize.register(decimal.Decimal)  
@htmlize.register(float)
def _(x) -> str:
    frac = fractions.Fraction(x).limit_denominator()
    return f'<pre>{x} ({frac.numerator}/{frac.denominator})</pre>'

Parameterized Decorators

When parsing a decorator in source code, Python takes the decorated function and passes it as the first argument to the decorator function. So how do you make a decorator accept other arguments? The answer is: make a decorator factory that takes those arguments and returns a decorator, which is then applied to the function to be decorated. Confusing? Sure. Let’s start with an example based on the simplest decorator we’ve seen: register in Example 9-21.

registry = []

def register(func):
    print(f'running register({func})')
    registry.append(func)
    return func

@register
def f1():
    print('running f1()')

print('running main()')
print('registry ->', registry)
f1()

registry = set()  

def register(active=True):  
    def decorate(func):  
        print('running register'
              f'(active={active})->decorate({func})')
        if active:   
            registry.add(func)
        else:
            registry.discard(func)  

        return func  
    return decorate  

@register(active=False)  
def f1():
    print('running f1()')

@register()  
def f2():
    print('running f2()')

def f3():
    print('running f3()')

>>> import registration_param
running register(active=False)->decorate(<function f1 at 0x10063c1e0>)
running register(active=True)->decorate(<function f2 at 0x10063c268>)
>>> registration_param.registry
[<function f2 at 0x10063c268>]

>>> from registration_param import *
running register(active=False)->decorate(<function f1 at 0x10073c1e0>)
running register(active=True)->decorate(<function f2 at 0x10073c268>)
>>> registry  
{<function f2 at 0x10073c268>}
>>> register()(f3)  
running register(active=True)->decorate(<function f3 at 0x10073c158>)
<function f3 at 0x10073c158>
>>> registry  
{<function f3 at 0x10073c158>, <function f2 at 0x10073c268>}
>>> register(active=False)(f2)  
running register(active=False)->decorate(<function f2 at 0x10073c268>)
<function f2 at 0x10073c268>
>>> registry  
{<function f3 at 0x10073c158>}

import time

DEFAULT_FMT = '[{elapsed:0.8f}s] {name}({args}) -> {result}'

def clock(fmt=DEFAULT_FMT):  
    def decorate(func):      
        def clocked(*_args): 
            t0 = time.perf_counter()
            _result = func(*_args)  
            elapsed = time.perf_counter() - t0
            name = func.__name__
            args = ', '.join(repr(arg) for arg in _args)  
            result = repr(_result)  
            print(fmt.format(**locals()))  
            return _result  
        return clocked  
    return decorate  

if __name__ == '__main__':

    @clock()  
    def snooze(seconds):
        time.sleep(seconds)

    for i in range(3):
        snooze(.123)

$ python3 clockdeco_param.py
[0.12412500s] snooze(0.123) -> None
[0.12411904s] snooze(0.123) -> None
[0.12410498s] snooze(0.123) -> None

import time
from clockdeco_param import clock

@clock('{name}: {elapsed}s')
def snooze(seconds):
    time.sleep(seconds)

for i in range(3):
    snooze(.123)

$ python3 clockdeco_param_demo1.py
snooze: 0.12414693832397461s
snooze: 0.1241159439086914s
snooze: 0.12412118911743164s

import time
from clockdeco_param import clock

@clock('{name}({args}) dt={elapsed:0.3f}s')
def snooze(seconds):
    time.sleep(seconds)

for i in range(3):
    snooze(.123)

$ python3 clockdeco_param_demo2.py
snooze(0.123) dt=0.124s
snooze(0.123) dt=0.124s
snooze(0.123) dt=0.124s

import time

DEFAULT_FMT = '[{elapsed:0.8f}s] {name}({args}) -> {result}'

class clock:  

    def __init__(self, fmt=DEFAULT_FMT):  
        self.fmt = fmt

    def __call__(self, func):  
        def clocked(*_args):
            t0 = time.perf_counter()
            _result = func(*_args)  
            elapsed = time.perf_counter() - t0
            name = func.__name__
            args = ', '.join(repr(arg) for arg in _args)
            result = repr(_result)
            print(self.fmt.format(**locals()))
            return _result
        return clocked

Chapter Summary

We covered some difficult terrain in this chapter. I tried to make the journey as smooth as possible, but we definitely entered the realm of metaprogramming.

We started with a simple @register decorator without an inner function, and finished with a parameterized @clock() involving two levels of nested functions.

Registration decorators, though simple in essence, have real applications in Python frameworks. We will apply the registration idea in one implementation of the Strategy design pattern in Chapter 10.

Understanding how decorators actually work required covering the difference between import time and runtime, then diving into variable scoping, closures, and the new nonlocal declaration. Mastering closures and nonlocal is valuable not only to build decorators, but also to code event-oriented programs for GUIs or asynchronous I/O with callbacks, and to adopt a functional style when it makes sense.

Parameterized decorators almost aways involve at least two nested functions, maybe more if you want to use @functools.wraps to produce a decorator that provides better support for more advanced techniques. One such technique is stacked decorators, which we saw in Example 9-18. For more sophisticated decorators, a class-based implementation may be easier to read and maintain.

As examples of parametrized decorators in the standard library, we visited the powerful @lru_cache() an @singledispatch from the functools module.

Further Reading

Item #26 of Brett Slatkin’s Effective Python, Second Edition (Addison-Wesley, 2019) covers best practices for function decorators and recommends always using functools.wraps—which we saw in Example 9-16.⁷

Graham Dumpleton has a series of in-depth blog posts about techniques for implementing well-behaved decorators, starting with “How You Implemented Your Python Decorator is Wrong”. His deep expertise in this matter is also nicely packaged in the wrapt module he wrote to simplify the implementation of decorators and dynamic function wrappers, which support introspection and behave correctly when further decorated, when applied to methods and when used as attribute descriptors. [Link to Come] in Part VI is about descriptors.

Chapter 9, “Metaprogramming,” of the Python Cookbook, Third Edition by David Beazley and Brian K. Jones (O’Reilly), has several recipes from elementary decorators to very sophisticated ones, including one that can be called as a regular decorator or as a decorator factory, e.g., @clock or @clock(). That’s “Recipe 9.6. Defining a Decorator That Takes an Optional Argument” in that cookbook.

Michele Simionato authored a package aiming to “simplify the usage of decorators for the average programmer, and to popularize decorators by showing various non-trivial examples,” according to the docs. It’s available on PyPI as the decorator package.

Created when decorators were still a new feature in Python, the Python Decorator Library wiki page has dozens of examples. Because that page started years ago, some of the techniques shown have been superseded, but the page is still an excellent source of inspiration.

“Closures in Python” is a short blog post by Fredrik Lundh that explains the terminology of closures.

PEP 3104 — Access to Names in Outer Scopes describes the introduction of the nonlocal declaration to allow rebinding of names that are neither local nor global. It also includes an excellent overview of how this issue is resolved in other dynamic languages (Perl, Ruby, JavaScript, etc.) and the pros and cons of the design options available to Python.

On a more theoretical level, PEP 227 — Statically Nested Scopes documents the introduction of lexical scoping as an option in Python 2.1 and as a standard in Python 2.2, explaining the rationale and design choices for the implementation of closures in Python.

PEP 443 provides the rationale and a detailed description of the single-dispatch generic functions’ facility. An old (March 2005) blog post by Guido van Rossum, “Five-Minute Multimethods in Python”, walks through an implementation of generic functions (a.k.a. multimethods) using decorators. His code supports multiple-dispatch (i.e., dispatch based on more than one positional argument). Guido’s multimethods code is interesting, but it’s a didactic example. For a modern, production-ready implementation of multiple-dispatch generic functions, check out Reg by Martijn Faassen—author of the model-driven and REST-savvy Morepath web framework.