I have never considered Python to be heavily influenced by functional languages, no matter what people say or think. I was much more familiar with imperative languages such as C and Algol 68 and although I had made functions first-class objects, I didn’t view Python as a functional programming language.1
Guido van Rossum, Python BDFL
Functions in Python are first-class objects. Programming language theorists define a “first-class object” as a program entity that can be:
created at runtime;
assigned to a variable or element in a data structure;
passed as an argument to a function;
returned as the result of a function.
Integers, strings, and dictionaries are other examples of first-class objects in Python—nothing fancy here. But if you came to Python from a language where functions are not first-class citizens, this chapter and the rest of Part III of the book focuses on the implications and practical applications of treating functions as objects.
The term “first-class functions” is widely used as shorthand for “functions as first-class objects.” It’s not perfect because it seems to imply an “elite” among functions. In Python, all functions are first-class.
“Positional-only parameters” is a new section, covering a feature added in Python 3.8. Section “The Nine Flavors of Callable Objects” was titled “The Seven Flavors of Callable Objects” in the first edition. It now mentions native coroutines and asynchronous generators, introduced in Python 3.5 and 3.6, respectively. Both are covered in later chapters, but here they are listed along with the other callables for completeness.
I moved section “Reading annotations at runtime” to the new Chapter 8, which is about type hints in function definitions. When I wrote the first edition, PEP 484 — Type Hints was still being discussed, and people used annotations in different ways. Since Python 3.5, annotations should conform to PEP 484. Therefore, the best place to cover them is Chapter 8.
Now let’s see why Python functions are full-fledged objects.
The console session in Example 7-1 shows that Python functions are objects. Here we create a function, call it, read its __doc__
attribute, and check that the function object itself is an instance of the function
class.
>>>
def
factorial
(
n
)
:
...
'''returns n!'''
...
return
1
if
n
<
2
else
n
*
factorial
(
n
-
1
)
...
>>>
factorial
(
42
)
1405006117752879898543142606244511569936384000000000
>>>
factorial
.
__doc__
'returns n!'
>>>
type
(
factorial
)
<class 'function'>
This is a console session, so we’re creating a function at “runtime.”
__doc__
is one of several attributes of function objects.
factorial
is an instance of the function
class.
The __doc__
attribute is used to generate the help text of an object. In the Python console, the command help(factorial)
will display a screen like Figure 7-1.
Example 7-2 shows the “first class” nature of a function object. We can assign it a variable fact
and call it through that name. We can also pass factorial
as an argument to map
. The map
function returns an iterable where each item is the result of the application of the first argument (a function) to succesive elements of the second argument (an iterable), range(10)
in this example.
>>>
fact
=
factorial
>>>
fact
<function factorial at 0x...>
>>>
fact
(
5
)
120
>>>
map
(
factorial
,
range
(
11
))
<map object at 0x...>
>>>
list
(
map
(
fact
,
range
(
11
)))
[1, 1, 2, 6, 24, 120, 720, 5040, 40320, 362880, 3628800]
Having first-class functions enables programming in a functional style. One of the hallmarks of functional programming is the use of higher-order functions, our next topic.
A function that takes a function as argument or returns a function as the result is a higher-order function. One example is map
, shown in Example 7-2. Another is the built-in function sorted
: the optional key
argument lets you provide a function to be applied to each item for sorting, as seen in “list.sort and the sorted Built-In Function”.
For example, to sort a list of words by length, simply pass the len
function as the key
, as in Example 7-3.
>>>
fruits
=
[
'strawberry'
,
'fig'
,
'apple'
,
'cherry'
,
'raspberry'
,
'banana'
]
>>>
sorted
(
fruits
,
key
=
len
)
['fig', 'apple', 'cherry', 'banana', 'raspberry', 'strawberry']
>>>
Any one-argument function can be used as the key. For example, to create a rhyme dictionary it might be useful to sort each word spelled backward. In Example 7-4, note that the words in the list are not changed at all; only their reversed spelling is used as the sort criterion, so that the berries appear together.
>>>
def
reverse
(
word
):
...
return
word
[::
-
1
]
>>>
reverse
(
'testing'
)
'gnitset'
>>>
sorted
(
fruits
,
key
=
reverse
)
['banana', 'apple', 'fig', 'raspberry', 'strawberry', 'cherry']
>>>
In the functional programming paradigm, some of the best known higher-order functions are map
, filter
, reduce
, and apply
.
The apply
function was deprecated in Python 2.3 and removed in Python 3 because it’s no longer necessary.
If you need to call a function with a dynamic set of arguments, you can write fn(*args, **kwargs)
instead of apply(fn, args, kwargs)
.
The map
, filter
, and reduce
higher-order functions are still around, but better alternatives are available for most of their use cases, as the next section shows.
Functional languages commonly offer the map
, filter
, and reduce
higher-order functions (sometimes with different names). The map
and filter
functions are still built-ins in Python 3, but since the introduction of list comprehensions and generator expressions, they are not as important. A listcomp or a genexp does the job of map
and filter
combined, but is more readable. Consider Example 7-5.
>>>
list
(
map
(
fact
,
range
(
6
)
)
)
[1, 1, 2, 6, 24, 120]
>>>
[
fact
(
n
)
for
n
in
range
(
6
)
]
[1, 1, 2, 6, 24, 120]
>>>
list
(
map
(
factorial
,
filter
(
lambda
n
:
n
%
2
,
range
(
6
)
)
)
)
[1, 6, 120]
>>>
[
factorial
(
n
)
for
n
in
range
(
6
)
if
n
%
2
]
[1, 6, 120]
>>>
Build a list of factorials from 0! to 5!.
Same operation, with a list comprehension.
List of factorials of odd numbers up to 5!, using both map
and filter
.
List comprehension does the same job, replacing map
and filter
, and making lambda
unnecessary.
In Python 3, map
and filter
return generators—a form of iterator—so their direct substitute is now a generator expression (in Python 2, these functions returned lists, therefore their closest alternative is a listcomp).
The reduce
function was demoted from a built-in in Python 2 to the functools
module in Python 3. Its most common use case, summation, is better served by the sum
built-in available since Python 2.3 was released in 2003. This is a big win in terms of readability and performance (see Example 7-6).
>>>
from
functools
import
reduce
>>>
from
operator
import
add
>>>
reduce
(
add
,
range
(
100
)
)
4950
>>>
sum
(
range
(
100
)
)
4950
>>>
Starting with Python 3.0, reduce
is not a built-in.
Import add
to avoid creating a function just to add two numbers.
Sum integers up to 99.
Same task using sum
; reduce
and adding function are not needed.
The common idea of sum
and reduce
is to apply some operation to successive items in a sequence,
accumulating previous results, thus reducing a sequence of values to a single value.
Other reducing built-ins are all
and any
:
all(iterable)
Returns True
if every element of the iterable
is truthy; all([])
returns True
.
any(iterable)
Returns True
if any element of the iterable
is truthy; any([])
returns False
.
I give a fuller explanation of reduce
in “Vector Take #4: Hashing and a Faster ==” where an ongoing example provides a meaningful context for the use of this function. The reducing functions are summarized later in the book when iterables are in focus, in [Link to Come].
To use a higher-order function, sometimes it is convenient to create a small, one-off function. That is why anonymous functions exist. We’ll cover them next.
The lambda
keyword creates an anonymous function within a Python expression.
However, the simple syntax of Python limits the body of lambda functions to be pure expressions.
In other words, the body cannot contain other Python statements such as while
, try
, etc.
Assignment with =
is also a statement, so it cannot occur in a lambda.
The new assignment expression syntax using :=
can be used—but if you need it,
your lambda is probably too complicated and hard to read,
and it should be refactored in to a regular function using def
.
The best use of anonymous functions is in the context of an argument list for a higher-order function.
For example, Example 7-7 is the rhyme index example from
Example 7-4 rewritten with lambda
, without defining a reverse
function.
>>>
fruits
=
[
'strawberry'
,
'fig'
,
'apple'
,
'cherry'
,
'raspberry'
,
'banana'
]
>>>
sorted
(
fruits
,
key
=
lambda
word
:
word
[::
-
1
])
['banana', 'apple', 'fig', 'raspberry', 'strawberry', 'cherry']
>>>
Outside the limited context of arguments to higher-order functions, anonymous functions are rarely useful in Python. The syntactic restrictions tend to make nontrivial lambdas either unreadable or unworkable. If a lambda is hard to read, I strongly advise you follow Fredrik Lund’s refactoring advice.
The lambda
syntax is just syntactic sugar: a lambda
expression creates a function object just like the def
statement.
That is just one of several kinds of callable objects in Python.
The following section reviews all of them.
The call operator ()
may be applied to other objects beyond user-defined functions and lambdas.
To determine whether an object is callable, use the callable()
built-in function.
As of Python 3.8, the Data Model documentation lists nine callable types:
Created with def
statements or lambda
expressions.
A function implemented in C (for CPython), like len
or time.strftime
.
Methods implemented in C, like dict.get
.
Functions defined in the body of a class.
When invoked, a class runs its __new__
method to create an instance, then __init__
to initialize it, and finally the instance is returned to the caller. Because there is no new
operator in Python, calling a class is like calling a function.2
If a class defines a __call__
method, then its instances may be invoked as functions—that’s the subject of the next section.
Functions or methods that use the yield
keyword in their body. When called, they return a generator object.
Functions or methods defined with async def
. When called, they return a coroutine object. Added in Python 3.5.
Functions or methods defined with async def
that have yield
in their body.
When called, they return an asynchronous generator for use with async for
. Added in Python 3.6.
Generators, coroutines, and asynchronous generators are unlike other callables in many respects. Generators are covered in [Link to Come]. They can also be used as classic coroutines, which are covered in [Link to Come]. [Link to Come] covers native coroutines and asynchronous generator functions.
Given the variety of existing callable types in Python, the safest way to determine whether an object is callable is to use the callable()
built-in:
>>> abs, str, 13 (<built-in function abs>, <class 'str'>, 13) >>> [callable(obj) for obj in (abs, str, 13)] [True, True, False]
We now move on to building class instances that work as callable objects.
Not only are Python functions real objects, but arbitrary Python objects may also be made to behave like functions. Implementing a __call__
instance method is all it takes.
Example 7-8 implements a BingoCage
class. An instance is built from any iterable, and stores an internal list
of items, in random order. Calling the instance pops an item.
import
random
class
BingoCage
:
def
__init__
(
self
,
items
)
:
self
.
_items
=
list
(
items
)
random
.
shuffle
(
self
.
_items
)
def
pick
(
self
)
:
try
:
return
self
.
_items
.
pop
(
)
except
IndexError
:
raise
LookupError
(
'
pick from empty BingoCage
'
)
def
__call__
(
self
)
:
return
self
.
pick
(
)
__init__
accepts any iterable; building a local copy prevents unexpected side effects on any list
passed as an argument.
shuffle
is guaranteed to work because self._items
is a list
.
The main method.
Raise exception with custom message if self._items
is empty.
Shortcut to bingo.pick()
: bingo()
.
Here is a simple demo of Example 7-8. Note how a bingo
instance can be invoked as a function, and the callable(…)
built-in recognizes it as a callable object:
>>>
bingo
=
BingoCage
(
range
(
3
))
>>>
bingo
.
pick
()
1
>>>
bingo
()
0
>>>
callable
(
bingo
)
True
A class implementing __call__
is an easy way to create function-like objects that have some
internal state that must be kept across invocations, like the remaining items in the BingoCage
.
Another good use case for __call__
is implementing decorators. Decorators must be callable,
and it is sometimes convenient to be able to “remember” something between calls of the decorator
(e.g., for memoization—caching the results of expensive computations for later use)
or to split complexity into separate methods.
The functional approach to creating functions with internal state is to use closures. Closures, as well as decorators, are the subject of Chapter 9.
We now move on to another aspect of handling functions as objects: runtime introspection.
Function objects have many attributes beyond __doc__
. See what the dir
function reveals about our factorial
:
>>>
dir
(
factorial
)
['__annotations__', '__call__', '__class__', '__closure__', '__code__',
'__defaults__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__',
'__format__', '__ge__', '__get__', '__getattribute__', '__globals__',
'__gt__', '__hash__', '__init__', '__kwdefaults__', '__le__', '__lt__',
'__module__', '__name__', '__ne__', '__new__', '__qualname__', '__reduce__',
'__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__',
'__subclasshook__']
>>>
Most of these attributes are common to Python objects in general. In this section, we cover those that are especially relevant to treating functions as objects, starting with __dict__
.
Like the instances of a plain user-defined class, a function uses the __dict__
attribute to store user attributes assigned to it. This is useful as a primitive form of annotation. Assigning arbitrary attributes to functions is not a very common practice in general, but Django is one framework that uses it. See, for example, the short_description
, boolean
, and allow_tags
attributes described in The Django admin site documentation. In the Django docs, this example shows attaching a short_description
to a method, to determine the description that will appear in record listings in the Django admin when that method is used:
def
upper_case_name
(
obj
):
return
(
"
%s
%s
"
%
(
obj
.
first_name
,
obj
.
last_name
))
.
upper
()
upper_case_name
.
short_description
=
'Customer name'
Now let us focus on the attributes that are specific to functions and are not found in a generic Python user-defined object. Computing the difference of two sets quickly gives us a list of the function-specific attributes (see Example 7-9).
>>>
class
C
:
pass
>>>
obj
=
C
(
)
>>>
def
func
(
)
:
pass
>>>
sorted
(
set
(
dir
(
func
)
)
-
set
(
dir
(
obj
)
)
)
['__annotations__', '__call__', '__closure__', '__code__', '__defaults__',
'__get__', '__globals__', '__kwdefaults__', '__name__', '__qualname__']
>>>
Create bare user-defined class.
Make an instance of it.
Create a bare function.
Using set difference, generate a sorted list of the attributes that exist in a function but not in an instance of a bare class.
Table 7-1 shows a summary of the attributes listed by Example 7-9.
Name | Type | Description |
---|---|---|
|
|
Parameter and return type hints |
|
|
Implementation of the |
|
|
The function closure, i.e., bindings for free variables (often is |
|
|
Function metadata and function body compiled into bytecode |
|
|
Default values for the formal parameters |
|
|
Implementation of the read-only descriptor protocol (see [Link to Come]) |
|
|
Reference to global variables of the module where the function is defined |
|
|
Default values for the keyword-only formal parameters |
|
|
The function name |
|
|
The qualified function name, e.g., |
In later sections and chapters we will discuss the __defaults__
, __code__
, and __annotations__
functions,
used by IDEs and frameworks to extract information about function signatures.
But to fully appreciate these attributes, we will make a detour to explore
the powerful syntax Python offers to declare function parameters and to pass arguments into them.
One of the best features of Python functions is the extremely flexible parameter handling mechanism, enhanced with keyword-only arguments in Python 3. Closely related are the use of *
and **
to “explode” iterables and mappings into separate arguments when we call a function. To see these features in action, see the code for Example 7-10 and tests showing its use in Example 7-11.
class_
is used to pass “class” attributes as a workaround because class
is a keyword in Pythondef
tag
(
name
,
*
content
,
class_
=
None
,
**
attrs
):
"""Generate one or more HTML tags"""
if
class_
is
not
None
:
attrs
[
'class'
]
=
class_
if
attrs
:
attr_pairs
=
(
f
' {attr}="{value}"'
for
attr
,
value
in
sorted
(
attrs
.
items
()))
attr_str
=
''
.
join
(
attr_pairs
)
else
:
attr_str
=
''
if
content
:
elements
=
(
f
'<{name}{attr_str}>{c}</{name}>'
for
c
in
content
)
return
'
'
.
join
(
elements
)
else
:
return
f
'<{name}{attr_str} />'
The tag
function can be invoked in many ways, as Example 7-11 shows.
>>
>
tag
(
'
br
'
)
'
<br />
'
>>
>
tag
(
'
p
'
,
'
hello
'
)
'
<p>hello</p>
'
>>
>
(
tag
(
'
p
'
,
'
hello
'
,
'
world
'
)
)
<
p
>
hello
<
/
p
>
<
p
>
world
<
/
p
>
>>
>
tag
(
'
p
'
,
'
hello
'
,
id
=
33
)
'
<p id=
"
33
"
>hello</p>
'
>>
>
(
tag
(
'
p
'
,
'
hello
'
,
'
world
'
,
class_
=
'
sidebar
'
)
)
<
p
class
=
"
sidebar
"
>
hello
<
/
p
>
<
p
class
=
"
sidebar
"
>
world
<
/
p
>
>>
>
tag
(
content
=
'
testing
'
,
name
=
"
img
"
)
'
<img content=
"
testing
"
/>
'
>>
>
my_tag
=
{
'
name
'
:
'
img
'
,
'
title
'
:
'
Sunset Boulevard
'
,
.
.
.
'
src
'
:
'
sunset.jpg
'
,
'
class
'
:
'
framed
'
}
>>
>
tag
(
*
*
my_tag
)
'
<img class=
"
framed
"
src=
"
sunset.jpg
"
title=
"
Sunset Boulevard
"
/>
'
A single positional argument produces an empty tag with that name.
Any number of arguments after the first are captured by *content
as a tuple
.
Keyword arguments not explicitly named in the tag
signature are captured by **attrs
as a dict
.
The class_
parameter can only be passed as a keyword argument.
Even the first positional argument can be passed as a keyword when tag
is called.
Prefixing the my_tag
dict
with **
passes all its items as separate arguments, which are then bound to the named parameters, with the remaining caught by **attrs
. In this case we can have a 'class'
key in the arguments dict
, because it is a string, and does not clash with the class
reserved word.
Keyword-only arguments are a new feature in Python 3. In Example 7-10, the class_
parameter can only be given as a keyword argument—it will never capture unnamed positional arguments. To specify keyword-only arguments when defining a function, name them after the argument prefixed with *
. If you don’t want to support variable positional arguments but still want keyword-only arguments, put a *
by itself in the signature, like this:
>>>
def
f
(
a
,
*
,
b
):
...
return
a
,
b
...
>>>
f
(
1
,
b
=
2
)
(1, 2)
Note that keyword-only arguments do not need to have a default value: they can be mandatory, like b
in the preceding example.
Since Python 3.8, user-defined function signatures may specify potitional-only parameters.
This feature always existed for built-in functions, such as divmod(a, b)
,
which can only be called with positional parameters, and not as divmod(a=10, b=4)
.
To define a function requiring positional-only parameters, use /
in the parameter list.
This is example from What’s New In Python 3.8
shows how to emulate the divmod
built-in function:
def
divmod
(
a
,
b
,
/
):
return
(
a
//
b
,
a
%
b
)
All arguments to the left of the /
are positional-only. After the /
, you may specify other arguments, which work as usual.
The /
in the parameter list is a syntax error in Python 3.7 or earlier.
For example, consider the tag
function from Example 7-10.
If we want the name
parameter to be positional only, we can add a /
after it in the function signature, like this:
def
tag
(
name
,
/
,
*
content
,
class_
=
None
,
**
attrs
):
...
You can find other examples of positional-only parameters in What’s New In Python 3.8 and in PEP 570.
After diving into Python’s flexible argument declaration features, we go back to the introspection of function arguments, starting with a motivating example from a web framework, and on through introspection techniques.
An interesting application of function introspection can be found in the Bobo HTTP micro-framework. To see that in action, consider a variation of the Bobo tutorial “Hello world” application in Example 7-12.
I mention Bobo because it pionered the use of parameter introspection to reduce boilerplate code in Python Web frameworks—since 1997! The practice is now common. FastAPI is an example of a modern framework that uses the same idea.
hello
requires a person argument
, and retrieves it from the HTTP requestimport
bobo
@bobo
.
query
(
'/'
)
def
hello
(
person
):
return
'Hello
%s
!'
%
person
The bobo.query
decorator integrates a plain function such as hello
with the request handling machinery of the framework. We’ll cover decorators in Chapter 9—that’s not the point of this example here. The point is that Bobo introspects the hello
function and finds out it needs one parameter named person
to work, and it will retrieve a parameter with that name from the request and pass it to hello
, so the programmer doesn’t need deal with request object directly. This also makes unit testing easier: there is no need to mock the request object to test the hello
function.
If you install Bobo and point its development server to the script in Example 7-12 (e.g., bobo -f hello.py
), a hit on the URL http://localhost:8080/
will produce the message “Missing form variable person” with a 403 HTTP code. This happens because Bobo understands that the person
argument is required to call hello
, but no such name was found in the request. Example 7-13 is a shell session using curl
to show this behavior.
$
curl -i http://localhost:8080/ HTTP/1.0403
Forbidden Date: Thu,21
Aug2014
21:39:44 GMT Server: WSGIServer/0.2 CPython/3.4.1 Content-Type: text/html;
charset
=
UTF-8 Content-Length: 103 <html> <head><title>Missing parameter</title></head> <body>Missing form variable person</body> </html>
However, if you get http://localhost:8080/?person=Jim
, the response will be the string 'Hello Jim!'
. See Example 7-14.
$
curl -i http://localhost:8080/?person=
Jim HTTP/1.0200
OK Date: Thu,21
Aug2014
21:42:32 GMT Server: WSGIServer/0.2 CPython/3.4.1 Content-Type: text/html;
charset
=
UTF-8 Content-Length: 10 Hello Jim!
How does Bobo know which parameter names are required by the function, and whether they have default values or not?
Within a function object, the __defaults__
attribute holds a tuple with the default values of positional and keyword arguments. The defaults for keyword-only arguments appear in __kwdefaults__
. The names of the arguments, however, are found within the __code__
attribute, which is a reference to a code
object with many attributes of its own.
To demonstrate the use of these attributes, we will inspect the function clip
in a module clip.py, listed in Example 7-15.
def
clip
(
text
,
max_len
=
80
):
"""Return text clipped at the last space before or after max_len
"""
end
=
None
if
len
(
text
)
>
max_len
:
space_before
=
text
.
rfind
(
' '
,
0
,
max_len
)
if
space_before
>=
0
:
end
=
space_before
else
:
space_after
=
text
.
rfind
(
' '
,
max_len
)
if
space_after
>=
0
:
end
=
space_after
if
end
is
None
:
# no spaces were found
return
text
.
rstrip
()
return
text
[:
end
]
.
rstrip
()
Example 7-16 shows the values of __defaults__
, __code__.co_varnames
, and __code__.co_argcount
for the clip
function listed in Example 7-15.
>>>
from
clip
import
clip
>>>
clip
.
__defaults__
(80,)
>>>
clip
.
__code__
# doctest: +ELLIPSIS
<code object clip at 0x...>
>>>
clip
.
__code__
.
co_varnames
('text', 'max_len', 'end', 'space_before', 'space_after')
>>>
clip
.
__code__
.
co_argcount
2
As you can see, this is not the most convenient arrangement of information. The argument names appear in __code__.co_varnames
, but that also includes the names of the local variables created in the body of the function. Therefore, the argument names are the first N strings, where N is given by __code__.co_argcount
which—by the way—does not include any variable arguments prefixed with *
or **
. The default values are identified only by their position in the __defaults__
tuple, so to link each with the respective argument, you have to scan from last to first. In the example, we have two arguments, text
and max_len
, and one default, 80
, so it must belong to the last argument, max_len
. This is awkward.
Fortunately, there is a better way: the inspect
module.
Take a look at Example 7-17.
>>>
from
clip
import
clip
>>>
from
inspect
import
signature
>>>
sig
=
signature
(
clip
)
>>>
sig
# doctest: +ELLIPSIS
<inspect.Signature object at 0x...>
>>>
str
(
sig
)
'(text, max_len=80)'
>>>
for
name
,
param
in
sig
.
parameters
.
items
():
...
(
param
.
kind
,
':'
,
name
,
'='
,
param
.
default
)
...
POSITIONAL_OR_KEYWORD : text = <class 'inspect._empty'>
POSITIONAL_OR_KEYWORD : max_len = 80
This is much better. inspect.signature
returns an inspect.Signature
object, which has a parameters
attribute that lets you read an ordered mapping of names to inspect.Parameter
objects. Each Parameter
instance has attributes such as name
, default
, and kind
. The special value inspect._empty
denotes parameters with no default, which makes sense considering that None
is a valid—and popular—default value.
The kind
attribute holds one of five possible values from the _ParameterKind
class:
POSITIONAL_OR_KEYWORD
A parameter that may be passed as a positional or as a keyword argument (most Python function parameters are of this kind).
VAR_POSITIONAL
A tuple
of positional parameters.
VAR_KEYWORD
A dict
of keyword parameters.
KEYWORD_ONLY
A keyword-only parameter (new in Python 3).
POSITIONAL_ONLY
A positional-only parameter; unsupported by function declaration syntax before Python 3.8, but exemplified by existing functions implemented in C—like divmod
—that do not accept parameters passed by keyword.
Besides name
, default
, and kind
, inspect.Parameter
objects have an annotation
attribute that is usually inspect._empty
but may contain function signature metadata provided via the new annotations syntax in Python 3—covered in Chapter 8.
An inspect.Signature
object has a bind
method that takes any number of arguments and binds them to the parameters in the signature, applying the usual rules for matching actual arguments to formal parameters. This can be used by a framework to validate arguments prior to the actual function invocation. Example 7-18 shows how.
>>>
import
inspect
>>>
sig
=
inspect
.
signature
(
tag
)
>>>
my_tag
=
{
'
name
'
:
'
img
'
,
'
title
'
:
'
Sunset Boulevard
'
,
...
'
src
'
:
'
sunset.jpg
'
,
'
cls
'
:
'
framed
'
}
>>>
bound_args
=
sig
.
bind
(
*
*
my_tag
)
>>>
bound_args
<inspect.BoundArguments object at 0x...>
>>>
for
name
,
value
in
bound_args
.
arguments
.
items
(
)
:
...
(
name
,
'
=
'
,
value
)
...
name = img
cls = framed
attrs = {'title': 'Sunset Boulevard', 'src': 'sunset.jpg'}
>>>
del
my_tag
[
'
name
'
]
>>>
bound_args
=
sig
.
bind
(
*
*
my_tag
)
Traceback (most recent call last):
...
TypeError
:
'name' parameter lacking default value
Get the signature from tag
function in Example 7-10.
Pass a dict
of arguments to .bind()
.
An inspect.BoundArguments
object is produced.
Iterate over the items in bound_args.arguments
, which is an OrderedDict
, to display the names and values of the arguments.
Remove the mandatory argument name
from my_tag
.
Calling sig.bind(**my_tag)
raises a TypeError
complaining of the missing name
parameter.
This example shows how the Python Data Model, with the help of inspect
, exposes the same machinery the interpreter uses to bind arguments to formal parameters in function calls.
Frameworks and tools like IDEs can use this information to validate code. Another feature of Python 3, function annotations, enhances the possible uses of this, as we’ll see in Chapter 8.
After examining the anatomy of function objects, the remainder of this chapter covers the most useful packages in the standard library for programming in a functional style.
Although Guido makes it clear that he did not design Python to be a functional programming language,
a functional coding style can be used to good extent, thanks to first-class functions and the support of packages like operator
and functools
, which we cover in the next two sections.
Often in functional programming it is convenient to use an arithmetic operator as a function. For example, suppose you want to multiply a sequence of numbers to calculate factorials without using recursion. To perform summation, you can use sum
, but there is no equivalent function for multiplication. You could use reduce
—as we saw in “Modern Replacements for map, filter, and reduce”—but this requires a function to multiply two items of the sequence. Example 7-19 shows how to solve this using lambda
.
from
functools
import
reduce
def
fact
(
n
):
return
reduce
(
lambda
a
,
b
:
a
*
b
,
range
(
1
,
n
+
1
))
The operator
module provides function equivalents for dozens of operators so you don’t have to code trivial functions like lambda a, b: a*b
.
With it, we can rewrite Example 7-19 as Example 7-20.
from
functools
import
reduce
from
operator
import
mul
def
fact
(
n
):
return
reduce
(
mul
,
range
(
1
,
n
+
1
))
Another group of one-trick lambdas that operator
replaces are functions to pick items from sequences or read attributes from objects:
itemgetter
and attrgetter
are factories that build custom functions to do that.
Example 7-21 shows a common use of itemgetter
: sorting a list of tuples by the value of one field.
In the example, the cities are printed sorted by country code (field 1). Essentially, itemgetter(1)
create a function that, given a collection, returns the item at index 1.
That’s easier to write and read than lambda fields: fields[1]
, which does the same:
>>>
metro_data
=
[
...
(
'Tokyo'
,
'JP'
,
36.933
,
(
35.689722
,
139.691667
)),
...
(
'Delhi NCR'
,
'IN'
,
21.935
,
(
28.613889
,
77.208889
)),
...
(
'Mexico City'
,
'MX'
,
20.142
,
(
19.433333
,
-
99.133333
)),
...
(
'New York-Newark'
,
'US'
,
20.104
,
(
40.808611
,
-
74.020386
)),
...
(
'Sao Paulo'
,
'BR'
,
19.649
,
(
-
23.547778
,
-
46.635833
)),
...
]
>>>
>>>
from
operator
import
itemgetter
>>>
for
city
in
sorted
(
metro_data
,
key
=
itemgetter
(
1
)):
...
(
city
)
...
('Sao Paulo', 'BR', 19.649, (-23.547778, -46.635833))
('Delhi NCR', 'IN', 21.935, (28.613889, 77.208889))
('Tokyo', 'JP', 36.933, (35.689722, 139.691667))
('Mexico City', 'MX', 20.142, (19.433333, -99.133333))
('New York-Newark', 'US', 20.104, (40.808611, -74.020386))
If you pass multiple index arguments to itemgetter
, the function it builds will return tuples with the extracted values, which is useful for sorting on multiple keys:
>>>
cc_name
=
itemgetter
(
1
,
0
)
>>>
for
city
in
metro_data
:
...
(
cc_name
(
city
))
...
('JP', 'Tokyo')
('IN', 'Delhi NCR')
('MX', 'Mexico City')
('US', 'New York-Newark')
('BR', 'Sao Paulo')
>>>
Because itemgetter
uses the []
operator, it supports not only sequences but also mappings and any class that implements __getitem__
.
A sibling of itemgetter
is attrgetter
, which creates functions to extract object attributes by name. If you pass attrgetter
several attribute names as arguments, it also returns a tuple of values. In addition, if any argument name contains a .
(dot), attrgetter
navigates through nested objects to retrieve the attribute. These behaviors are shown in Example 7-22. This is not the shortest console session because we need to build a nested structure to showcase the handling of dotted attributes by attrgetter
.
>>>
from
collections
import
namedtuple
>>>
LatLong
=
namedtuple
(
'
LatLong
'
,
'
lat long
'
)
>>>
Metropolis
=
namedtuple
(
'
Metropolis
'
,
'
name cc pop coord
'
)
>>>
metro_areas
=
[
Metropolis
(
name
,
cc
,
pop
,
LatLong
(
lat
,
long
)
)
...
for
name
,
cc
,
pop
,
(
lat
,
long
)
in
metro_data
]
>>>
metro_areas
[
0
]
Metropolis(name='Tokyo', cc='JP', pop=36.933, coord=LatLong(lat=35.689722,
long=139.691667))
>>>
metro_areas
[
0
]
.
coord
.
lat
35.689722
>>>
from
operator
import
attrgetter
>>>
name_lat
=
attrgetter
(
'
name
'
,
'
coord.lat
'
)
>>>
>>>
for
city
in
sorted
(
metro_areas
,
key
=
attrgetter
(
'
coord.lat
'
)
)
:
...
(
name_lat
(
city
)
)
...
('Sao Paulo', -23.547778)
('Mexico City', 19.433333)
('Delhi NCR', 28.613889)
('Tokyo', 35.689722)
('New York-Newark', 40.808611)
Use namedtuple
to define LatLong
.
Also define Metropolis
.
Build metro_areas
list with Metropolis
instances; note the nested tuple unpacking to extract (lat, long)
and use them to build the LatLong
for the coord
attribute of Metropolis
.
Reach into element metro_areas[0]
to get its latitude.
Define an attrgetter
to retrieve the name
and the coord.lat
nested attribute.
Use attrgetter
again to sort list of cities by latitude.
Use the attrgetter
defined in to show only city name and latitude.
Here is a partial list of functions defined in operator
(names starting with _
are omitted, because they are mostly implementation details):
>>>
[
name
for
name
in
dir
(
operator
)
if
not
name
.
startswith
(
'_'
)]
['abs', 'add', 'and_', 'attrgetter', 'concat', 'contains',
'countOf', 'delitem', 'eq', 'floordiv', 'ge', 'getitem', 'gt',
'iadd', 'iand', 'iconcat', 'ifloordiv', 'ilshift', 'imod', 'imul',
'index', 'indexOf', 'inv', 'invert', 'ior', 'ipow', 'irshift',
'is_', 'is_not', 'isub', 'itemgetter', 'itruediv', 'ixor', 'le',
'length_hint', 'lshift', 'lt', 'methodcaller', 'mod', 'mul', 'ne',
'neg', 'not_', 'or_', 'pos', 'pow', 'rshift', 'setitem', 'sub',
'truediv', 'truth', 'xor']
Most of the 52 names listed are self-evident. The group of names prefixed with i
and the name of another operator—e.g., iadd
, iand
, etc.—correspond to the augmented assignment operators—e.g., +=
, &=
, etc. These change their first argument in place, if it is mutable; if not, the function works like the one without the i
prefix: it simply returns the result of the operation.
Of the remaining operator
functions, methodcaller
is the last we will cover. It is somewhat similar to attrgetter
and itemgetter
in that it creates a function on the fly. The function it creates calls a method by name on the object given as argument, as shown in Example 7-23.
>>>
from
operator
import
methodcaller
>>>
s
=
'The time has come'
>>>
upcase
=
methodcaller
(
'upper'
)
>>>
upcase
(
s
)
'THE TIME HAS COME'
>>>
hiphenate
=
methodcaller
(
'replace'
,
' '
,
'-'
)
>>>
hiphenate
(
s
)
'The-time-has-come'
The first test in Example 7-23 is there just to show methodcaller
at work, but if you need to use the str.upper
as a function, you can just call it on the str
class and pass a string as argument, like this:
>>>
str
.
upper
(
s
)
'THE TIME HAS COME'
The second test in Example 7-23 shows that methodcaller
can also do a partial application to freeze some arguments, like the functools.partial
function does. That is our next subject.
The functools
module provides several higher-order functions. The best known of them is probably reduce
, which was covered in “Modern Replacements for map, filter, and reduce”. Of the remaining functions in functools
, the most useful is partial
and its variation, partialmethod
.
functools.partial
is a higher-order function that allows partial application of a function. Given a function, a partial application produces a new callable with some of the arguments of the original function fixed. This is useful to adapt a function that takes one or more arguments to an API that requires a callback with fewer arguments. Example 7-24 is a trivial demonstration.
>>>
from
operator
import
mul
>>>
from
functools
import
partial
>>>
triple
=
partial
(
mul
,
3
)
>>>
triple
(
7
)
21
>>>
list
(
map
(
triple
,
range
(
1
,
10
)
)
)
[3, 6, 9, 12, 15, 18, 21, 24, 27]
Create new triple
function from mul
, binding first positional argument to 3
.
Test it.
Use triple
with map
; mul
would not work with map
in this example.
A more useful example involves the unicode.normalize
function that we saw in “Normalizing Unicode for Reliable Comparisons”. If you work with text from many languages, you may want to apply unicode.normalize('NFC', s)
to any string s
before comparing or storing it. If you do that often, it’s handy to have an nfc
function to do so, as in Example 7-25.
>>>
import
unicodedata
,
functools
>>>
nfc
=
functools
.
partial
(
unicodedata
.
normalize
,
'NFC'
)
>>>
s1
=
'café'
>>>
s2
=
'cafe
u0301
'
>>>
s1
,
s2
('café', 'café')
>>>
s1
==
s2
False
>>>
nfc
(
s1
)
==
nfc
(
s2
)
True
partial
takes a callable as first argument, followed by an arbitrary number of positional and keyword arguments to bind.
Example 7-26 shows the use of partial
with the tag
function from Example 7-10, to freeze one positional argument and one keyword argument.
>>>
from
tagger
import
tag
>>>
tag
<function tag at 0x10206d1e0>
>>>
from
functools
import
partial
>>>
picture
=
partial
(
tag
,
'
img
'
,
cls
=
'
pic-frame
'
)
>>>
picture
(
src
=
'
wumpus.jpeg
'
)
'<img class="pic-frame" src="wumpus.jpeg" />'
>>>
picture
functools.partial(<function tag at 0x10206d1e0>, 'img', cls='pic-frame')
>>>
picture
.
func
<function tag at 0x10206d1e0>
>>>
picture
.
args
('img',)
>>>
picture
.
keywords
{'cls': 'pic-frame'}
Import tag
from Example 7-10 and show its ID.
Create picture
function from tag
by fixing the first positional argument with 'img'
and the cls
keyword argument with 'pic-frame'
.
picture
works as expected.
partial()
returns a functools.partial
object.3
A functools.partial
object has attributes providing access to the original function and the fixed arguments.
The functools.partialmethod
function (since Python 3.4) does the same job as partial
, but is designed to work with methods.
An impressive functools
function is lru_cache
, which does memoization—an optimization that works by storing the results of function calls to avoid expensive recalculations. We will cover it in Chapter 9, where decorators are explained, along with other higher-order functions designed to be used as decorators: singledispatch
and wraps
.
The goal of this chapter was to explore the first-class nature of functions in Python. The main ideas are that you can assign functions to variables, pass them to other functions, store them in data structures, and access function attributes, allowing frameworks and tools to act on that information.
Higher-order functions, a staple of functional programming, are common in Python.
Using map
, filter
, and reduce
is not as common as it used to be—thanks to list comprehensions (and similar constructs like generator expressions) and the appearance of reducing built-ins like sum
, all
, and any
. The sorted
, min
, max
built-ins, and functools.partial
are examples of commonly used higher-order functions in the language.
Callables come in nine different flavors since Python 3.6, from the simple functions created with lambda
to instances of classes implementing __call__
.
Generators and coroutines are also callable, although their behavior is very different from other callables.
All callables can be detected by the callable()
built-in. Every callable supports the same rich syntax for declaring formal parameters, including keyword-only parameters and annotations—both new features introduced with Python 3.
Lastly, we covered some functions from the operator
module and functools.partial
, which facilitate functional programming by minimizing the need for the functionally challenged lambda
syntax.
Don’t Make It Callable is a post from 2019 where Moshe Zadka—core developer of Python and Twisted—argues against implementing __call__
except in the very specific case of writing a DSL (Domain-Specific Language), which are specialized APIs where it is OK to break with convention to provide users with a “succint syntax dedicated to the task at hand.”
The next chapters continue our exploration of programming with function objects. Chapter 9 dives into function decorators—a special kind of higher-order function—and the closure mechanism that makes them work. Chapter 8 is devoted to type hints in function parameters and return values. Chapter 10 shows how first-class functions can simplify some classic object-oriented design patterns. The code in that chapter also show several uses of type hints.
In The Python Language Reference, “3.2. The standard type hierarchy” presents the nine callable types, along with all the other built-in types.
Chapter 7 of the Python Cookbook, Third Edition (O’Reilly), by David Beazley and Brian K. Jones, is an excellent complement to the current chapter as well as Chapter 9 of this book, covering mostly the same concepts with a different approach.
See PEP 3102 — Keyword-Only Arguments if you are interested in the rationale and use cases for that feature.
A great introduction to functional programming in Python is A. M. Kuchling’s Python Functional Programming HOWTO. The main focus of that text, however, is the use of iterators and generators, which are the subject of [Link to Come].
fn.py
is a package to support functional programming in Python 2 and 3. According to its author, Alexey Kachayev, fn.py
provides “implementation of missing features to enjoy FP” in Python. It includes a @recur.tco
decorator that implements tail-call optimization for unlimited recursion in Python, among many other functions, data structures, and recipes. However, be aware that fn.py
had its last updates in 2014, so it may be “done” or just abandoned.
The StackOverflow question “Python: Why is functools.partial necessary?” has a highly informative (and funny) reply by Alex Martelli, co-author of the classic Python in a Nutshell.
Jim Fulton’s Bobo was probably the first Web framework that deserved to be called object-oriented. If you were intrigued by it and want to learn more about its modern rewrite, start at its Introduction. A little of the early history of Bobo appears in a comment by Phillip J. Eby in a discussion at Joel Spolsky’s blog.
Reflecting on the question “Is Python a functional language?”, I created Beyond Paradigms, a talk presented at PyCaribbean, PyBay and PyConDE (slides, video).
1 “Origins of Python’s Functional Features”, from Guido’s The History of Python blog.
2 Calling a class usually creates an instance of the same class, but other behaviors are possible by overriding __new__
. We’ll see an example of this in [Link to Come].
3 The source code for functools.py
reveals that the functools.partial
class is implemented in C and is used by default. If that is not available, a pure-Python implementation of partial
is available since Python 3.4.in the functools
module.
4 There is also the problem of lost indentation when pasting code to Web forums, but I digress.
18.219.217.55