There’s been a number of complaints about the choice of the name “decorator” for this feature. The major one is that the name is not consistent with its use in the GoF book.1 The name decorator probably owes more to its use in the compiler area—a syntax tree is walked and annotated.
PEP 318 — Decorators for Functions and Methods
Function decorators let us “mark” functions in the source code to enhance their behavior in some way. This is powerful stuff, but mastering it requires understanding closures.
One of the newest reserved keywords in Python is nonlocal
, introduced in Python 3.0. You can have a profitable life as a Python programmer without ever using it if you adhere to a strict regimen of class-centered object orientation. However, if you want to implement your own function decorators, you must know closures inside out, and then the need for nonlocal
becomes obvious.
Aside from their application in decorators, closures are also essential for effective asynchronous programming with callbacks, and for coding in a functional style when it makes sense.
The end goal of this chapter is to explain exactly how function decorators work, from the simplest registration decorators to the rather more complicated parameterized ones. However, before we reach that goal we need to cover:
How Python evaluates decorator syntax
How Python decides whether a variable is local
Why closures exist and how they work
What problem is solved by nonlocal
With this grounding, we can tackle further decorator topics:
Implementing a well-behaved decorator
Powerful decorators in the standard library: @lru_cache
and @singledispatch
Implementing a parameterized decorator
Coverage of the functools.lru_cache
decorator was updated to include “Simplified usage of lru_cache in Python 3.8”.
Section “Single Dispatch Generic Functions” was expanded and now uses type hints, the preferred way to use functools.singledispatch
since Python 3.7.
“Parameterized Decorators” now includes a class-based example, Example 9-27.
I moved Chapter 10—Design Patterns with First-Class Functions—to the end of part III to improve the flow of the book. Section “Decorator-Enhanced Strategy Pattern” is now in that chapter, along with other variations of the Strategy design pattern using callables.
We start with a very gentle introduction to decorators, and then proceed with the rest of the items listed in the chapter opening.
A decorator is a callable that takes another function as argument (the decorated function). Python also supports class decorators—they are covered in [Link to Come].
A decorator may perform some processing with the decorated function, and returns it or replaces it with another function or callable object.
In other words, assuming an existing decorator named decorate
, this code:
@decorate
def
target
():
(
'running target()'
)
Has the same effect as writing this:
def
target
():
(
'running target()'
)
target
=
decorate
(
target
)
The end result is the same: at the end of either of these snippets,
the target
name is bound to whatever function is returned by
decorate(target)
—which may be the function initially named target
,
or may be a different function.
To confirm that the decorated function is replaced, see the console session in Example 9-1.
>>>
def
deco
(
func
)
:
...
def
inner
(
)
:
...
(
'
running inner()
'
)
...
return
inner
...
>>>
@deco
...
def
target
(
)
:
...
(
'
running target()
'
)
...
>>>
target
(
)
running inner()
>>>
target
<function deco.<locals>.inner at 0x10063b598>
deco
returns its inner
function object.
target
is decorated by deco
.
Invoking the decorated target
actually runs inner
.
Inspection reveals that target
is a now a reference to inner
.
Strictly speaking, decorators are just syntactic sugar. As we just saw, you can always simply call a decorator like any regular callable, passing another function. Sometimes that is actually convenient, especially when doing metaprogramming—changing program behavior at runtime.
To summarize: the first crucial fact about decorators is that they have the power to replace the decorated function with a different one. The second crucial fact is that they are executed immediately when a module is loaded. This is explained next.
A key feature of decorators is that they run right after the decorated function is defined. That is usually at import time (i.e., when a module is loaded by Python). Consider registration.py in Example 9-2.
registry
=
[
]
def
register
(
func
)
:
(
f
'
running register({func})
'
)
registry
.
append
(
func
)
return
func
@register
def
f1
(
)
:
(
'
running f1()
'
)
@register
def
f2
(
)
:
(
'
running f2()
'
)
def
f3
(
)
:
(
'
running f3()
'
)
def
main
(
)
:
(
'
running main()
'
)
(
'
registry ->
'
,
registry
)
f1
(
)
f2
(
)
f3
(
)
if
__name__
==
'
__main__
'
:
main
(
)
registry
will hold references to functions decorated by @register
.
register
takes a function as argument.
Display what function is being decorated, for demonstration.
Include func
in registry
.
Return func
: we must return a function; here we return the same received as argument.
f1
and f2
are decorated by @register
.
f3
is not decorated.
main
displays the registry
, then calls f1()
, f2()
, and f3()
.
main()
is only invoked if registration.py runs as a script.
The output of running registration.py as a script looks like this:
$
python3 registration.py running register(
<function
f1 at 0x100631bf8>)
running register(
<function
f2 at 0x100631c80>)
running main()
registry ->[
<function
f1 at 0x100631bf8>, <function
f2 at 0x100631c80>]
running f1()
running f2()
running f3()
Note that register
runs (twice) before any other function in the module. When register
is called, it receives as an argument the function object being decorated—for example, <function f1 at 0x100631bf8>
.
After the module is loaded, the registry
list holds references to the two decorated functions: f1
and f2
. These functions, as well as f3
, are only executed when explicitly called by main
.
If registration.py is imported (and not run as a script), the output is this:
>>>
import
registration
running register(<function f1 at 0x10063b1e0>)
running register(<function f2 at 0x10063b268>)
At this time, if you inspect registry
, this is what you see:
>>>
registration
.
registry
[<function f1 at 0x10063b1e0>, <function f2 at 0x10063b268>]
The main point of Example 9-2 is to emphasize that function decorators are executed as soon as the module is imported, but the decorated functions only run when they are explicitly invoked. This highlights the difference between what Pythonistas call import time and runtime.
Considering how decorators are commonly employed in real code, Example 9-2 is unusual in two ways:
The decorator function is defined in the same module as the decorated functions. A real decorator is usually defined in one module and applied to functions in other modules.
The register
decorator returns the same function passed as argument. In practice, most decorators define an inner function and return it.
Even though the register
decorator in Example 9-2 returns the decorated function unchanged, that technique is not useless. Similar decorators are used in many Python frameworks to add functions to some central registry—for example, a registry mapping URL patterns to functions that generate HTTP responses. Such registration decorators may or may not change the decorated function.
We will see a registration decorator applied in “Decorator-Enhanced Strategy Pattern” (Chapter 10).
Most decorators do change the decorated function. They usually do it by defining an inner function and returning it to replace the decorated function. Code that uses inner functions almost always depends on closures to operate correctly. To understand closures, we need to take a step back and have a close look at how variable scopes work in Python.
In Example 9-3, we define and test a function that reads two variables:
a local variable a
—defined as function parameter—and variable b
that is not defined anywhere in the function.
>>>
def
f1
(
a
):
...
(
a
)
...
(
b
)
...
>>>
f1
(
3
)
3
Traceback (most recent call last):
File"<stdin>"
, line1
, in<module>
File"<stdin>"
, line3
, inf1
NameError
:global name 'b' is not defined
The error we got is not surprising. Continuing from Example 9-3, if we assign a value to a global b
and then call f1
, it works:
>>>
b
=
6
>>>
f1
(
3
)
3
6
Now, let’s see an example that may surprise you.
Take a look at the f2
function in Example 9-4. Its first two lines are the same as f1
in Example 9-3, then it makes an assignment to b
, and prints its value. But it fails at the second print
, before the assignment is made.
>>>
b
=
6
>>>
def
f2
(
a
):
...
(
a
)
...
(
b
)
...
b
=
9
...
>>>
f2
(
3
)
3
Traceback (most recent call last):
File"<stdin>"
, line1
, in<module>
File"<stdin>"
, line3
, inf2
UnboundLocalError
:local variable 'b' referenced before assignment
Note that the output starts with 3
, which proves that the print(a)
statement was executed. But the second one, print(b)
, never runs. When I first saw this I was surprised, thinking that 6
should be printed, because there is a global variable b
and the assignment to the local b
is made after print(b)
.
But the fact is, when Python compiles the body of the function, it decides that b
is a local variable because it is assigned within the function. The generated bytecode reflects this decision and will try to fetch b
from the local environment. Later, when the call f2(3)
is made, the body of f2
fetches and prints the value of the local variable a
, but when trying to fetch the value of local variable b
it discovers that b
is unbound.
This is not a bug, but a design choice: Python does not require you to declare variables, but assumes that a variable assigned in the body of a function is local. This is much better than the behavior of JavaScript, which does not require variable declarations either, but if you do forget to declare that a variable is local (with var
), you may clobber a global variable without knowing.
If we want the interpreter to treat b
as a global variable and still
assign a new value to it within the function, we use the global
declaration:
>>>
b
=
6
>>>
def
f3
(
a
):
...
global
b
...
(
a
)
...
(
b
)
...
b
=
9
...
>>>
f3
(
3
)
3
6
>>>
b
9
After this closer look at how variable scopes work in Python, we can tackle closures in the next section, “Closures”. If you are curious about the bytecode differences between the functions in Examples 9-3 and 9-4, see the following sidebar.
In the blogosphere, closures are sometimes confused with anonymous functions. The reason why many confuse them is historic: defining functions inside functions is not so common, until you start using anonymous functions. And closures only matter when you have nested functions. So a lot of people learn both concepts at the same time.
Actually, a closure is a function with an extended scope that encompasses nonglobal variables referenced in the body of the function but not defined there. It does not matter whether the function is anonymous or not; what matters is that it can access nonglobal variables that are defined outside of its body.
This is a challenging concept to grasp, and is better approached through an example.
Consider an avg
function to compute the mean of an ever-growing series of values; for example, the average closing price of a commodity over its entire history. Every day a new price is added, and the average is computed taking into account all prices so far.
Starting with a clean slate, this is how avg
could be used:
>>>
avg
(
10
)
10.0
>>>
avg
(
11
)
10.5
>>>
avg
(
12
)
11.0
Where does avg
come from, and where does it keep the history of previous values?
For starters, Example 9-7 is a class-based implementation.
class
Averager
():
def
__init__
(
self
):
self
.
series
=
[]
def
__call__
(
self
,
new_value
):
self
.
series
.
append
(
new_value
)
total
=
sum
(
self
.
series
)
return
total
/
len
(
self
.
series
)
The Averager
class creates instances that are callable:
>>>
avg
=
Averager
()
>>>
avg
(
10
)
10.0
>>>
avg
(
11
)
10.5
>>>
avg
(
12
)
11.0
Now, Example 9-8 is a functional implementation, using the higher-order function make_averager
.
def
make_averager
():
series
=
[]
def
averager
(
new_value
):
series
.
append
(
new_value
)
total
=
sum
(
series
)
return
total
/
len
(
series
)
return
averager
When invoked, make_averager
returns an averager
function object. Each time an averager
is called, it appends the passed argument to the series, and computes the current average, as shown in Example 9-9.
>>>
avg
=
make_averager
()
>>>
avg
(
10
)
10.0
>>>
avg
(
11
)
10.5
>>>
avg
(
12
)
11.0
Note the similarities of the examples: we call Averager()
or make_averager()
to get a callable object avg
that will update the historical series and calculate the current mean. In Example 9-7, avg
is an instance of Averager
, and in Example 9-8 it is the inner function, averager
. Either way, we just call avg(n)
to include n
in the series and get the updated mean.
It’s obvious where the avg
of the Averager
class keeps the history: the self.series
instance attribute. But where does the avg
function in the second example find the series
?
Note that series
is a local variable of make_averager
because the assignment series = []
happens in the body of that function. But when avg(10)
is called, make_averager
has already returned, and its local scope is long gone.
Within averager
, series
is a free variable. This is a technical term meaning a variable that is not bound in the local scope. See Figure 9-1.
Inspecting the returned averager
object shows how Python keeps the names of local and free variables in the __code__
attribute that represents the compiled body of the function. Example 9-10 demonstrates.
>>>
avg
.
__code__
.
co_varnames
('new_value', 'total')
>>>
avg
.
__code__
.
co_freevars
('series',)
The value for series
is kept in the __closure__
attribute of the returned function avg
. Each item in avg.__closure__
corresponds to a name in avg.__code__.co_freevars
. These items are cells
, and they have an attribute called cell_contents
where the actual value can be found. Example 9-11 shows these attributes.
>>>
avg
.
__code__
.
co_freevars
('series',)
>>>
avg
.
__closure__
(<cell at 0x107a44f78: list object at 0x107a91a48>,)
>>>
avg
.
__closure__
[
0
]
.
cell_contents
[10, 11, 12]
To summarize: a closure is a function that retains the bindings of the free variables that exist when the function is defined, so that they can be used later when the function is invoked and the defining scope is no longer available.
Note that the only situation in which a function may need to deal with external variables that are nonglobal is when it is nested in another function.
Our previous implementation of make_averager
was not efficient. In Example 9-8, we stored all the values in the historical series and computed their sum
every time averager
was called. A better implementation would only store the total and the number of items so far, and compute the mean from these two numbers.
Example 9-12 is a broken implementation, just to make a point. Can you see where it breaks?
def
make_averager
():
count
=
0
total
=
0
def
averager
(
new_value
):
count
+=
1
total
+=
new_value
return
total
/
count
return
averager
If you try Example 9-12, here is what you get:
>>>
avg
=
make_averager
()
>>>
avg
(
10
)
Traceback (most recent call last):
...
UnboundLocalError
:local variable 'count' referenced before assignment
>>>
The problem is that the statement count += 1
actually means the same as count = count + 1
, when count
is a number or any immutable type. So we are actually assigning to count
in the body of averager
, and that makes it a local variable. The same problem affects the total
variable.
We did not have this problem in Example 9-8 because we never assigned to the series
name; we only called series.append
and invoked sum
and len
on it. So we took advantage of the fact that lists are mutable.
But with immutable types like numbers, strings, tuples, etc., all you can do is read, never update.
If you try to rebind them, as in count = count + 1
, then you are implicitly creating a local variable count
. It is no longer a free variable, and therefore it is not saved in the closure.
To work around this, the nonlocal
keyword was introduced in Python 3. It lets you declare a variable as a free variable even when it is assigned within the function. If a new value is assigned to a nonlocal
variable, the binding stored in the closure is changed. A correct implementation of our newest make_averager
looks like Example 9-13.
def
make_averager
():
count
=
0
total
=
0
def
averager
(
new_value
):
nonlocal
count
,
total
count
+=
1
total
+=
new_value
return
total
/
count
return
averager
The lack of nonlocal
in Python 2 requires workarounds, one of which is described in the third code snippet of PEP 3104 — Access to Names in Outer Scopes, which introduced nonlocal
.
Now that we have Python closures covered, we can effectively implement decorators with nested functions.
Example 9-14 is a decorator that clocks every invocation of the decorated function and displays the elapsed time, the arguments passed, and the result of the call.
clockdeco0.py
: simpple decorator to show the running time of functionsimport
time
def
clock
(
func
)
:
def
clocked
(
*
args
)
:
t0
=
time
.
perf_counter
(
)
result
=
func
(
*
args
)
elapsed
=
time
.
perf_counter
(
)
-
t0
name
=
func
.
__name__
arg_str
=
'
,
'
.
join
(
repr
(
arg
)
for
arg
in
args
)
(
f
'
[
{elapsed:0.8f}
s]
{name}
(
{arg_str}
) ->
{result!r}
'
)
return
result
return
clocked
Define inner function clocked
to accept any number of positional arguments.
This line only works because the closure for clocked
encompasses the func
free variable.
Return the inner function to replace the decorated function.
Example 9-15 demonstrates the use of the clock
decorator.
import
time
from
clockdeco
import
clock
@clock
def
snooze
(
seconds
):
time
.
sleep
(
seconds
)
@clock
def
factorial
(
n
):
return
1
if
n
<
2
else
n
*
factorial
(
n
-
1
)
if
__name__
==
'__main__'
:
(
'*'
*
40
,
'Calling snooze(.123)'
)
snooze
(
.
123
)
(
'*'
*
40
,
'Calling factorial(6)'
)
(
'6! ='
,
factorial
(
6
))
The output of running Example 9-15 looks like this:
$
python3 clockdeco_demo.py **************************************** Calling snooze(
.123)
[
0.12363791s]
snooze(
0.123)
-> None **************************************** Calling factorial(
6)
[
0.00000095s]
factorial(
1)
-> 1[
0.00002408s]
factorial(
2)
-> 2[
0.00003934s]
factorial(
3)
-> 6[
0.00005221s]
factorial(
4)
-> 24[
0.00006390s]
factorial(
5)
-> 120[
0.00008297s]
factorial(
6)
-> 720 6!=
720
Remember that this code:
@clock
def
factorial
(
n
):
return
1
if
n
<
2
else
n
*
factorial
(
n
-
1
)
Actually does this:
def
factorial
(
n
):
return
1
if
n
<
2
else
n
*
factorial
(
n
-
1
)
factorial
=
clock
(
factorial
)
So, in both examples, clock
gets the factorial
function as its func
argument (see Example 9-14). It then creates and returns the clocked
function, which the Python interpreter assigns to factorial
behind the scenes. In fact, if you import the clockdeco_demo
module and check the __name__
of factorial
, this is what you get:
>>>
import
clockdeco_demo
>>>
clockdeco_demo
.
factorial
.
__name__
'clocked'
>>>
So factorial
now actually holds a reference to the clocked
function. From now on, each time factorial(n)
is called, clocked(n)
gets executed. In essence, clocked
does the following:
Records the initial time t0
.
Calls the original factorial
function, saving the result.
Computes the elapsed time.
Formats and displays the collected data.
Returns the result saved in step 2.
This is the typical behavior of a decorator: it replaces the decorated function with a new function that accepts the same arguments and (usually) returns whatever the decorated function was supposed to return, while also doing some extra processing.
In Design Patterns by Gamma et al., the short description of the Decorator pattern starts with: “Attach additional responsibilities to an object dynamically.” Function decorators fit that description. But at the implementation level, Python decorators bear little resemblance to the classic Decorator described in the original Design Patterns work. “Soapbox” has more on this subject.
The clock
decorator implemented in Example 9-14 has a few shortcomings: it does not support keyword arguments, and it masks the __name__
and __doc__
of the decorated function.
Example 9-16 uses the functools.wraps
decorator to copy the relevant attributes from func
to clocked
.
Also, in this new version, keyword arguments are correctly handled.
clockdeco.py
: an improved clock decoratorimport
time
import
functools
def
clock
(
func
):
@functools
.
wraps
(
func
)
def
clocked
(
*
args
,
**
kwargs
):
t0
=
time
.
perf_counter
()
result
=
func
(
*
args
,
**
kwargs
)
elapsed
=
time
.
perf_counter
()
-
t0
name
=
func
.
__name__
arg_lst
=
[]
if
args
:
arg_lst
.
append
(
', '
.
join
(
repr
(
arg
)
for
arg
in
args
))
if
kwargs
:
pairs
=
[
f
'
{k}
=
{v!r}
'
for
k
,
v
in
kwargs
.
items
()]
arg_lst
.
append
(
', '
.
join
(
pairs
))
arg_str
=
', '
.
join
(
arg_lst
)
(
f
'[
{elapsed:0.8f}
s]
{name}
(
{arg_str}
) ->
{result!r}
'
)
return
result
return
clocked
functools.wraps
is just one of the ready-to-use decorators in the standard library. In the next section,
we’ll meet the most impressive decorator that functools
provides: lru_cache
.
Python has three built-in functions that are designed to decorate methods: property
, classmethod
, and staticmethod
. We will discuss property
in [Link to Come] and the others in “classmethod
Versus staticmethod
”.
In Example 9-16 we saw another important decorator: functools.wraps
, a helper for building well-behaved decorators.
Two of the most interesting decorators in the standard library are lru_cache
and singledispatch
.
Both are defined in the functools
module. We’ll cover them next.
The functools.lru_cache
decorator implements memoization:
an optimization technique that works by saving the results of previous invocations of an expensive function,
avoiding repeat computations on previously used arguments.
The acronym LRU stands for Least Recently Used,
meaning that the growth of the cache is limited by discarding the entries that have not been read for a while.
A good demonstration is to apply lru_cache
to the painfully slow recursive function to generate the nth number in the Fibonacci sequence, as shown in Example 9-17.
from
clockdeco
import
clock
@clock
def
fibonacci
(
n
):
if
n
<
2
:
return
n
return
fibonacci
(
n
-
2
)
+
fibonacci
(
n
-
1
)
if
__name__
==
'__main__'
:
(
fibonacci
(
6
))
Here is the result of running fibo_demo.py. Except for the last line, all output is generated by the clock
decorator:
$ python3 fibo_demo.py [0.00000042s] fibonacci(0) -> 0 [0.00000049s] fibonacci(1) -> 1 [0.00006115s] fibonacci(2) -> 1 [0.00000031s] fibonacci(1) -> 1 [0.00000035s] fibonacci(0) -> 0 [0.00000030s] fibonacci(1) -> 1 [0.00001084s] fibonacci(2) -> 1 [0.00002074s] fibonacci(3) -> 2 [0.00009189s] fibonacci(4) -> 3 [0.00000029s] fibonacci(1) -> 1 [0.00000027s] fibonacci(0) -> 0 [0.00000029s] fibonacci(1) -> 1 [0.00000959s] fibonacci(2) -> 1 [0.00001905s] fibonacci(3) -> 2 [0.00000026s] fibonacci(0) -> 0 [0.00000029s] fibonacci(1) -> 1 [0.00000997s] fibonacci(2) -> 1 [0.00000028s] fibonacci(1) -> 1 [0.00000030s] fibonacci(0) -> 0 [0.00000031s] fibonacci(1) -> 1 [0.00001019s] fibonacci(2) -> 1 [0.00001967s] fibonacci(3) -> 2 [0.00003876s] fibonacci(4) -> 3 [0.00006670s] fibonacci(5) -> 5 [0.00016852s] fibonacci(6) -> 8 8
The waste is obvious: fibonacci(1)
is called eight times, fibonacci(2)
five times, etc.
But adding just two lines to use lru_cache
, performance is much improved. See Example 9-18.
import
functools
from
clockdeco
import
clock
@functools
.
lru_cache
@clock
def
fibonacci
(
n
)
:
if
n
<
2
:
return
n
return
fibonacci
(
n
-
2
)
+
fibonacci
(
n
-
1
)
if
__name__
==
'
__main__
'
:
(
fibonacci
(
6
)
)
This line works with Python 3.8 or later. If you’re using Python 3.7 or earlier, lru_cache
must be invoked by adding ()
, so this line would be @functools.lru_cache()
. See “Simplified usage of lru_cache in Python 3.8”.
This is an example of stacked decorators: @lru_cache
is applied on the function returned by @clock
.
To make sense of stacked decorators,
recall that the @
is syntax sugar for applying the decorator function to the function below it.
If there’s more than one decorator, they behave like nested function calls. This:
@alpha
@beta
def
my_fn
():
...
Is the same as this:
my_fn
=
alpha
(
beta
(
my_fn
))
In other words, the beta
decorator is applied first, and the function it returns is then passed to alpha
.
Using lru_cache
in Example 9-18, the fibonacci
function is called only once for each value of n
:
$ python3 fibo_demo_lru.py [0.00000043s] fibonacci(0) -> 0 [0.00000054s] fibonacci(1) -> 1 [0.00006179s] fibonacci(2) -> 1 [0.00000070s] fibonacci(3) -> 2 [0.00007366s] fibonacci(4) -> 3 [0.00000057s] fibonacci(5) -> 5 [0.00008479s] fibonacci(6) -> 8 8
In another test, to compute fibonacci(30)
, Example 9-18 made the 31 calls needed in 0.00017s—total time–while the uncached Example 9-17 took 12.09s in an Intel Core i7 notebook, because it called fibonacci(1)
832,040 times, in a total of 2,692,537 calls.
Besides making silly recursive algorithms viable, lru_cache
really shines in applications that need to fetch information from remote APIs.
All the arguments taken by the decorated function must be hashable, because
lru_cache
uses a dict
to store the results,
and the keys are made from the positional and keyword arguments used in the calls.
Since Python 3.8 lru_cache
can be applied in two ways.
In Example 9-18 we used the newest and simplest way:
@lru_cache
def
costly_function
(
a
,
b
):
...
The other way—the only available before Python 3.8—is invoking it as a function—with ()
:
@lru_cache
()
def
costly_function
(
a
,
b
):
...
Invoking lru_cache()
allows passing optional arguments to configure the cache.
Their default values are:
@lru_cache
(
maxsize
=
128
,
typed
=
False
)
def
costly_function
(
a
,
b
):
...
The maxsize
argument determines how many call results are stored.
After the cache is full, the least recently used results are discarded to make room for new results.
For optimal performance, maxsize
should be a power of 2.
The typed
argument, if set to True
, stores results of different argument types separately,
i.e., distinguishing between float and integer arguments that are normally considered equal,
like 1
and 1.0
.
Now let’s study another powerful decorator: functools.singledispatch
.
Imagine we are creating a tool to debug web applications. We want to be able to generate HTML displays for different types of Python objects.
We could start with a function like this:
import
html
def
htmlize
(
obj
):
content
=
html
.
escape
(
repr
(
obj
))
return
f
'<pre>
{content}
</pre>'
That will work for any Python type, but now we want to extend it to generate custom displays for some types:
str
: replace embedded newline characters with '<br>
'
and use <p>
tags instead of <pre>
.
int
: show the number in decimal and hexadecimal (with a special case for bool
).
list
: output an HTML list, formatting each item according to its type.
float
and Decimal
: output the value as usual, but also in the form of a fraction (why not?).
The behavior we want is shown in Example 9-19.
htmlize()
generates HTML tailored to different object types>>
>
htmlize
(
{
1
,
2
,
3
}
)
'
<pre>{1, 2, 3}</pre>
'
>>
>
htmlize
(
abs
)
'
<pre><built-in function abs></pre>
'
>>
>
htmlize
(
'
Heimlich & Co.
- a game
'
)
'
<p>Heimlich & Co.<br>
- a game</p>
'
>>
>
htmlize
(
42
)
'
<pre>42 (0x2a)</pre>
'
>>
>
(
htmlize
(
[
'
alpha
'
,
66
,
{
3
,
2
,
1
}
]
)
)
<
ul
>
<
li
>
<
p
>
alpha
<
/
p
>
<
/
li
>
<
li
>
<
pre
>
66
(
0x42
)
<
/
pre
>
<
/
li
>
<
li
>
<
pre
>
{
1
,
2
,
3
}
<
/
pre
>
<
/
li
>
<
/
ul
>
>>
>
htmlize
(
True
)
'
<pre>True</pre>
'
>>
>
htmlize
(
fractions
.
Fraction
(
2
,
3
)
)
'
<pre>2/3</pre>
'
>>
>
htmlize
(
2
/
3
)
'
<pre>0.6666666666666666 (2/3)</pre>
'
>>
>
htmlize
(
decimal
.
Decimal
(
'
0.02380952
'
)
)
'
<pre>0.02380952 (1/42)</pre>
'
The original function is registered for object
, so it serves as a catch-all to handle argument types that don’t match the other implementations.
str
objects are also HTML-escaped but wrapped in <p></p>
with <br>
line breaks inserted before each '
'
.
An int
is shown in decimal and hexadecimal, inside <pre></pre>
.
Each list item is formatted according to its type, and the whole sequence rendered as an HTML list.
Although bool
is an int
subtype, it gets special treatment.
Show Fraction
as a fraction.
Show float
and Decimal
with an approximate fractional equivalent.
singledispatch
In “Overloaded signatures” (Chapter 8) we saw the @overload
decorator.
However, that is intended only for type hinting, therefore has no effect at runtime.
The @singledispatch
decorator is very different. It’s a runtime feature.
Because we don’t have Java-style method overloading in Python, we can’t simply create variations of htmlize
with different signatures for each data type we want to handle differently.
A possible solution in Python would be to turn htmlize
into a dispatch function,
with a chain of if/elif/elif
calling specialized functions like htmlize_str
, htmlize_int
, etc.
This is not extensible by users of our module, and is unwieldy:
over time, the htmlize
dispatcher would become too big,
and the coupling between it and the specialized functions would be very tight.
The functools.singledispatch
decorator allows each module to contribute to the overall solution,
and lets you easily provide a specialized function even for classes that you can’t edit.
If you decorate a plain function with @singledispatch
, it becomes the entry point for a generic function:
a group of functions to perform the same operation in different ways,
depending on the type of the first argument.2 Example 9-20 shows how.
functools.singledispatch
exists since Python 3.4,
but it only supports type hints since Python 3.7.
The last two functions in Example 9-20 illustrate the syntax
that works in all versions of Python since 3.4.
@singledispatch
creates a custom @htmlize.register
to bundle several functions into a generic functionfrom
functools
import
singledispatch
from
collections
import
abc
import
fractions
import
decimal
import
html
import
numbers
@singledispatch
def
htmlize
(
obj
:
object
)
-
>
str
:
content
=
html
.
escape
(
repr
(
obj
)
)
return
f
'
<pre>{content}</pre>
'
@htmlize.register
def
_
(
text
:
str
)
-
>
str
:
content
=
html
.
escape
(
text
)
.
replace
(
'
'
,
'
<br>
'
)
return
f
'
<p>{content}</p>
'
@htmlize.register
def
_
(
seq
:
abc
.
Sequence
)
-
>
str
:
inner
=
'
</li>
<li>
'
.
join
(
htmlize
(
item
)
for
item
in
seq
)
return
'
<ul>
<li>
'
+
inner
+
'
</li>
</ul>
'
@htmlize.register
def
_
(
n
:
numbers
.
Integral
)
-
>
str
:
return
f
'
<pre>{n} (0x{n:x})</pre>
'
@htmlize.register
def
_
(
n
:
bool
)
-
>
str
:
return
f
'
<pre>{n}</pre>
'
@htmlize.register
(
fractions
.
Fraction
)
def
_
(
x
)
-
>
str
:
frac
=
fractions
.
Fraction
(
x
)
return
f
'
<pre>{frac.numerator}/{frac.denominator}</pre>
'
@htmlize.register
(
decimal
.
Decimal
)
@htmlize.register
(
float
)
def
_
(
x
)
-
>
str
:
frac
=
fractions
.
Fraction
(
x
)
.
limit_denominator
(
)
return
f
'
<pre>{x} ({frac.numerator}/{frac.denominator})</pre>
'
@singledispatch
marks the base function that handles the object
type.
Each specialized function is decorated with @«base».register
The type of the first argument given at runtime determines when this particular function definition will be used. The name of the specialized functions is irrelevant; _
is a good choice to make this
clear.3
For each additional type to get special treatment, register a new function with a matching type hint in the first paramenter.
The numbers
ABCs are useful for use with singledispatch
.4
bool
is a subtype of numbers.Integral
, but the singledispatch
logic seeks the implementation with the most specific matching type, regardless of the order they appear in the code.
If you don’t want or can’t add type hints to the decorated function, you can pass a type to the @«base».register
decorator. This syntax works in Python 3.4 or later.
The @«base».register
decorator returns the undecorated function, so it’s possible to stack them to register two or more types on the same implementation.5
When possible, register the specialized functions to handle ABCs (abstract classes)
such as numbers.Integral
and abc.MutableSequence
instead of concrete implementations like int
and list
.
This allows your code to support a greater variety of compatible types.
For example, a Python extension can provide alternatives to the int
type with fixed bit lengths as subclasses of numbers.Integral
.6
Using ABCs or typing.Protocol
with @singledispatch
allows your code to support existing or future classes that are actual or virtual subclasses of those ABCs, or that implement those protocols. The use of ABCs and the concept of a virtual subclass are subjects of [Link to Come].
A notable quality of the singledispatch
mechanism is that you can register specialized functions anywhere in the system, in any module. If you later add a module with a new user-defined type, you can easily provide a new custom function to handle that type. And you can write custom functions for classes that you did not write and can’t change.
singledispatch
is a well-thought-out addition to the standard library,
and it offers more features than I can describe here.
PEP 443 — Single-dispatch generic functions is a good
reference—but it doesn’t mention the use of type hints, which were added later.
The functools
module documentation has improved and more up-to-date coverage
with several examples in its singledispatch
entry.
@singledispatch
is not designed to bring Java-style method overloading to Python. A single class with many overloaded variations of a method is better than a single function with a lengthy stretch of if/elif/elif/elif
blocks. But both solutions are flawed because they concentrate too much responsibility in a single code unit—the class or the function. The advantage of @singledispath
is supporting modular extension: each module can register a specialized function for each type it supports. In a realistic use case, you would not have all the implementations of generic function in the same module as in Example 9-20.
We’ve seen some decorators that take arguments, for example, @lru_cache()
and
htmlize.register(float)
created by @singledispatch
in Example 9-20.
The next section shows how to build decorators that accept parameters.
When parsing a decorator in source code, Python takes the decorated function and passes it as the first argument to the decorator function. So how do you make a decorator accept other arguments? The answer is: make a decorator factory that takes those arguments and returns a decorator, which is then applied to the function to be decorated. Confusing? Sure. Let’s start with an example based on the simplest decorator we’ve seen: register
in Example 9-21.
registry
=
[]
def
register
(
func
):
(
f
'running register({func})'
)
registry
.
append
(
func
)
return
func
@register
def
f1
():
(
'running f1()'
)
(
'running main()'
)
(
'registry ->'
,
registry
)
f1
()
In order to make it easy to enable or disable the function registration performed by register
, we’ll make it accept an optional active
parameter which, if False
, skips registering the decorated function. Example 9-22 shows how. Conceptually, the new register
function is not a decorator but a decorator factory. When called, it returns the actual decorator that will be applied to the target function.
registry
=
set
(
)
def
register
(
active
=
True
)
:
def
decorate
(
func
)
:
(
'
running register
'
f
'
(active={active})->decorate({func})
'
)
if
active
:
registry
.
add
(
func
)
else
:
registry
.
discard
(
func
)
return
func
return
decorate
@register
(
active
=
False
)
def
f1
(
)
:
(
'
running f1()
'
)
@register
(
)
def
f2
(
)
:
(
'
running f2()
'
)
def
f3
(
)
:
(
'
running f3()
'
)
registry
is now a set
, so adding and removing functions is faster.
register
takes an optional keyword argument.
The decorate
inner function is the actual decorator; note how it takes a function as argument.
Register func
only if the active
argument (retrieved from the closure) is True
.
If not active
and func in registry
, remove it.
Because decorate
is a decorator, it must return a function.
register
is our decorator factory, so it returns decorate
.
The @register
factory must be invoked as a function, with the desired parameters.
If no parameters are passed, register
must still be called as a function—@register()
—i.e., to return the actual decorator, decorate
.
The main point is that register()
returns decorate
, which is then applied to the decorated function.
The code in Example 9-22 is in a registration_param.py module. If we import it, this is what we get:
>>>
import
registration_param
running register(active=False)->decorate(<function f1 at 0x10063c1e0>)
running register(active=True)->decorate(<function f2 at 0x10063c268>)
>>>
registration_param
.
registry
[<function f2 at 0x10063c268>]
Note how only the f2
function appears in the registry
; f1
does not appear because active=False
was passed to the register
decorator factory, so the decorate
that was applied to f1
did not add it to the registry
.
If, instead of using the @
syntax, we used register
as a regular function, the syntax needed to decorate a function f
would be register()(f)
to add f
to the registry
, or register(active=False)(f)
to not add it (or remove it). See Example 9-23 for a demo of adding and removing functions to the registry
.
>>>
from
registration_param
import
*
running register(active=False)->decorate(<function f1 at 0x10073c1e0>)
running register(active=True)->decorate(<function f2 at 0x10073c268>)
>>>
registry
{<function f2 at 0x10073c268>}
>>>
register
(
)
(
f3
)
running register(active=True)->decorate(<function f3 at 0x10073c158>)
<function f3 at 0x10073c158>
>>>
registry
{<function f3 at 0x10073c158>, <function f2 at 0x10073c268>}
>>>
register
(
active
=
False
)
(
f2
)
running register(active=False)->decorate(<function f2 at 0x10073c268>)
<function f2 at 0x10073c268>
>>>
registry
{<function f3 at 0x10073c158>}
When the module is imported, f2
is in the registry
.
The register()
expression returns decorate
, which is then applied to f3
.
The previous line added f3
to the registry
.
This call removes f2
from the registry
.
Confirm that only f3
remains in the registry
.
The workings of parameterized decorators are fairly involved, and the one we’ve just discussed is simpler than most. Parameterized decorators usually replace the decorated function, and their construction requires yet another level of nesting. Now we will explore the architecture of one such function pyramid.
In this section, we’ll revisit the clock
decorator, adding a feature: users may pass a format string to control the output of the decorated function. See Example 9-24.
For simplicity, Example 9-24 is based on the initial clock
implementation from Example 9-14, and not the improved one from Example 9-16 that uses @functools.wraps
, adding yet another function layer.
import
time
DEFAULT_FMT
=
'
[{elapsed:0.8f}s] {name}({args}) -> {result}
'
def
clock
(
fmt
=
DEFAULT_FMT
)
:
def
decorate
(
func
)
:
def
clocked
(
*
_args
)
:
t0
=
time
.
perf_counter
(
)
_result
=
func
(
*
_args
)
elapsed
=
time
.
perf_counter
(
)
-
t0
name
=
func
.
__name__
args
=
'
,
'
.
join
(
repr
(
arg
)
for
arg
in
_args
)
result
=
repr
(
_result
)
(
fmt
.
format
(
*
*
locals
(
)
)
)
return
_result
return
clocked
return
decorate
if
__name__
==
'
__main__
'
:
@clock
(
)
def
snooze
(
seconds
)
:
time
.
sleep
(
seconds
)
for
i
in
range
(
3
)
:
snooze
(
.
123
)
clock
is our parameterized decorator factory.
decorate
is the actual decorator.
clocked
wraps the decorated function.
_result
is the actual result of the decorated function.
_args
holds the actual arguments of clocked
, while args
is str
used for display.
result
is the str
representation of _result
, for display.
Using **locals()
here allows any local variable of clocked
to be referenced in the fmt
.
clocked
will replace the decorated function, so it should return whatever that function returns.
decorate
returns clocked
.
clock
returns decorate
.
In this self test, clock()
is called without arguments, so the decorator applied will use the default format str
.
If you run Example 9-24 from the shell, this is what you get:
$
python3 clockdeco_param.py[
0.12412500s]
snooze(
0.123)
-> None[
0.12411904s]
snooze(
0.123)
-> None[
0.12410498s]
snooze(
0.123)
-> None
To exercise the new functionality, Examples 9-25 and 9-26 are two other modules using clockdeco_param
, and the outputs they generate.
import
time
from
clockdeco_param
import
clock
@clock
(
'
{name}
:
{elapsed}
s'
)
def
snooze
(
seconds
):
time
.
sleep
(
seconds
)
for
i
in
range
(
3
):
snooze
(
.
123
)
Output of Example 9-25:
$
python3 clockdeco_param_demo1.py
snooze: 0.12414693832397461s
snooze: 0.1241159439086914s
snooze: 0.12412118911743164s
import
time
from
clockdeco_param
import
clock
@clock
(
'
{name}
(
{args}
) dt=
{elapsed:0.3f}
s'
)
def
snooze
(
seconds
):
time
.
sleep
(
seconds
)
for
i
in
range
(
3
):
snooze
(
.
123
)
Output of Example 9-26:
$
python3 clockdeco_param_demo2.py snooze(
0.123)
dt
=
0.124s snooze(
0.123)
dt
=
0.124s snooze(
0.123)
dt
=
0.124s
Graham Dumpleton and Lennart Regebro—one of this book’s technical reviewers—argue that decorators are best coded as classes implementing __call__
, and not as functions like the examples in this chapter. I agree that approach is better for non-trivial decorators, but to explain the basic idea of this language feature, functions are easier to understand. See “Further Reading”, in particular Graham Dumpleton’s blog and wrapt
module for industrial-strength techniques when building decorators.
clock
decoratorAs a final example, Example 9-27 lists the implementation of a parameterized clock
decorator implmented as a class with __call__
. Contrast Example 9-24 with Example 9-27.
Which one do you prefer?
import
time
DEFAULT_FMT
=
'
[{elapsed:0.8f}s] {name}({args}) -> {result}
'
class
clock
:
def
__init__
(
self
,
fmt
=
DEFAULT_FMT
)
:
self
.
fmt
=
fmt
def
__call__
(
self
,
func
)
:
def
clocked
(
*
_args
)
:
t0
=
time
.
perf_counter
(
)
_result
=
func
(
*
_args
)
elapsed
=
time
.
perf_counter
(
)
-
t0
name
=
func
.
__name__
args
=
'
,
'
.
join
(
repr
(
arg
)
for
arg
in
_args
)
result
=
repr
(
_result
)
(
self
.
fmt
.
format
(
*
*
locals
(
)
)
)
return
_result
return
clocked
Instead of a clock
outer function, the clock
class is our parameterized decorator factory. I named it with a lowercase c
to make clear that this implementation is a drop-in replacement for the one in <<Example 9-27.
The argument passed in the clock(my_format)
is assigned to the fmt
parameter here. The class constructor returns an instance of clock
, with my_format
stored in self.fmt
.
__call__
makes the clock
instance callable. When invoked, the instance replaces the decorated function with clocked
clocked
wraps the decorated function.
This ends our exploration of function decorators. We’ll see class decorators in [Link to Come].
We covered some difficult terrain in this chapter. I tried to make the journey as smooth as possible, but we definitely entered the realm of metaprogramming.
We started with a simple @register
decorator without an inner function, and finished with a parameterized @clock()
involving two levels of nested functions.
Registration decorators, though simple in essence, have real applications in Python frameworks. We will apply the registration idea in one implementation of the Strategy design pattern in Chapter 10.
Understanding how decorators actually work required covering the difference between import time and runtime, then diving into variable scoping, closures, and the new nonlocal
declaration. Mastering closures and nonlocal
is valuable not only to build decorators, but also to code event-oriented programs for GUIs or asynchronous I/O with callbacks, and to adopt a functional style when it makes sense.
Parameterized decorators almost aways involve at least two nested functions, maybe more if you want to use @functools.wraps
to produce a decorator that provides better support for more advanced techniques. One such technique is stacked decorators, which we saw in Example 9-18. For more sophisticated decorators, a class-based implementation may be easier to read and maintain.
As examples of parametrized decorators in the standard library, we visited the powerful @lru_cache()
an @singledispatch
from the functools
module.
Item #26 of Brett Slatkin’s Effective Python, Second Edition (Addison-Wesley, 2019) covers best practices for function decorators and recommends always using functools.wraps
—which we saw in Example 9-16.7
Graham Dumpleton has a series of in-depth blog posts about techniques for implementing well-behaved decorators, starting with “How You Implemented Your Python Decorator is Wrong”. His deep expertise in this matter is also nicely packaged in the wrapt module he wrote to simplify the implementation of decorators and dynamic function wrappers, which support introspection and behave correctly when further decorated, when applied to methods and when used as attribute descriptors. [Link to Come] in Part VI is about descriptors.
Chapter 9, “Metaprogramming,” of the Python Cookbook, Third Edition by David Beazley and Brian K. Jones (O’Reilly), has several recipes from elementary decorators to very sophisticated ones, including one that can be called as a regular decorator or as a decorator factory, e.g., @clock
or @clock()
. That’s “Recipe 9.6. Defining a Decorator That Takes an Optional Argument” in that cookbook.
Michele Simionato authored a package aiming to “simplify the usage of decorators for the average programmer, and to popularize decorators by showing various non-trivial examples,” according to the docs. It’s available on PyPI as the decorator package.
Created when decorators were still a new feature in Python, the Python Decorator Library wiki page has dozens of examples. Because that page started years ago, some of the techniques shown have been superseded, but the page is still an excellent source of inspiration.
“Closures in Python” is a short blog post by Fredrik Lundh that explains the terminology of closures.
PEP 3104 — Access to Names in Outer Scopes describes the introduction of the nonlocal
declaration to allow rebinding of names that are neither local nor global. It also includes an excellent overview of how this issue is resolved in other dynamic languages (Perl, Ruby, JavaScript, etc.) and the pros and cons of the design options available to Python.
On a more theoretical level, PEP 227 — Statically Nested Scopes documents the introduction of lexical scoping as an option in Python 2.1 and as a standard in Python 2.2, explaining the rationale and design choices for the implementation of closures in Python.
PEP 443 provides the rationale and a detailed description of the single-dispatch generic functions’ facility. An old (March 2005) blog post by Guido van Rossum, “Five-Minute Multimethods in Python”, walks through an implementation of generic functions (a.k.a. multimethods) using decorators. His code supports multiple-dispatch (i.e., dispatch based on more than one positional argument). Guido’s multimethods code is interesting, but it’s a didactic example. For a modern, production-ready implementation of multiple-dispatch generic functions, check out Reg by Martijn Faassen—author of the model-driven and REST-savvy Morepath web framework.
1 That’s the 1995 Design Patterns book by the so-called Gang of Four.
2 This is what is meant by the term single-dispatch. If more arguments were used to select the specific functions, we’d have multiple-dispatch.
3 Unfortunately, Mypy 0.770 complains when it sees multiple functions with the same name…
4 Despite the warning in Stay away from the Numeric Tower, the number
ABC are not deprecated and you may find them in Python 3 code.
5 Maybe one day you’ll also be able to express this with single unparameterized @htmlize.register
and type hint using Union
, but when I tried, Python raised a TypeError
with a message saying that Union
is not a class. So, although PEP 484 syntax is supported by @singledispatch
, the semantics are not there yet.
6 NumPy, for example, implements several machine-oriented integer and floating-point types.
7 I wanted to make the code as simple as possible, so I did not follow Slatkin’s excellent advice in all examples.
3.142.171.253