What’s new in this chapter

“Positional-only parameters” is a new section, covering a feature added in Python 3.8. Section “The Nine Flavors of Callable Objects” was titled “The Seven Flavors of Callable Objects” in the first edition. It now mentions native coroutines and asynchronous generators, introduced in Python 3.5 and 3.6, respectively. Both are covered in later chapters, but here they are listed along with the other callables for completeness.

I moved section “Reading annotations at runtime” to the new Chapter 8, which is about type hints in function definitions. When I wrote the first edition, PEP 484 — Type Hints was still being discussed, and people used annotations in different ways. Since Python 3.5, annotations should conform to PEP 484. Therefore, the best place to cover them is Chapter 8.

Now let’s see why Python functions are full-fledged objects.

Treating a Function Like an Object

The console session in Example 7-1 shows that Python functions are objects. Here we create a function, call it, read its __doc__ attribute, and check that the function object itself is an instance of the function class.

>>> def factorial(n):  
...     '''returns n!'''
...     return 1 if n < 2 else n * factorial(n-1)
...
>>> factorial(42)
1405006117752879898543142606244511569936384000000000
>>> factorial.__doc__  
'returns n!'
>>> type(factorial)  
<class 'function'>

This is a console session, so we’re creating a function at “runtime.”

__doc__ is one of several attributes of function objects.

factorial is an instance of the function class.

The __doc__ attribute is used to generate the help text of an object. In the Python console, the command help(factorial) will display a screen like Figure 7-1.

Figure 7-1. Help screen for factorial; the text is built from the __doc__ attribute of the function.

Example 7-2 shows the “first class” nature of a function object. We can assign it a variable fact and call it through that name. We can also pass factorial as an argument to map. The map function returns an iterable where each item is the result of the application of the first argument (a function) to succesive elements of the second argument (an iterable), range(10) in this example.

>>> fact = factorial
>>> fact
<function factorial at 0x...>
>>> fact(5)
120
>>> map(factorial, range(11))
<map object at 0x...>
>>> list(map(fact, range(11)))
[1, 1, 2, 6, 24, 120, 720, 5040, 40320, 362880, 3628800]

Having first-class functions enables programming in a functional style. One of the hallmarks of functional programming is the use of higher-order functions, our next topic.

Higher-Order Functions

A function that takes a function as argument or returns a function as the result is a higher-order function. One example is map, shown in Example 7-2. Another is the built-in function sorted: the optional key argument lets you provide a function to be applied to each item for sorting, as seen in “list.sort and the sorted Built-In Function”.

For example, to sort a list of words by length, simply pass the len function as the key, as in Example 7-3.

>>> fruits = ['strawberry', 'fig', 'apple', 'cherry', 'raspberry', 'banana']
>>> sorted(fruits, key=len)
['fig', 'apple', 'cherry', 'banana', 'raspberry', 'strawberry']
>>>

Any one-argument function can be used as the key. For example, to create a rhyme dictionary it might be useful to sort each word spelled backward. In Example 7-4, note that the words in the list are not changed at all; only their reversed spelling is used as the sort criterion, so that the berries appear together.

>>> def reverse(word):
...     return word[::-1]
>>> reverse('testing')
'gnitset'
>>> sorted(fruits, key=reverse)
['banana', 'apple', 'fig', 'raspberry', 'strawberry', 'cherry']
>>>

In the functional programming paradigm, some of the best known higher-order functions are map, filter, reduce, and apply. The apply function was deprecated in Python 2.3 and removed in Python 3 because it’s no longer necessary. If you need to call a function with a dynamic set of arguments, you can write fn(*args, **kwargs) instead of apply(fn, args, kwargs).

The map, filter, and reduce higher-order functions are still around, but better alternatives are available for most of their use cases, as the next section shows.

>>> list(map(fact, range(6)))  
[1, 1, 2, 6, 24, 120]
>>> [fact(n) for n in range(6)]  
[1, 1, 2, 6, 24, 120]
>>> list(map(factorial, filter(lambda n: n % 2, range(6))))  
[1, 6, 120]
>>> [factorial(n) for n in range(6) if n % 2]  
[1, 6, 120]
>>>

>>> from functools import reduce  
>>> from operator import add  
>>> reduce(add, range(100))  
4950
>>> sum(range(100))  
4950
>>>

Anonymous Functions

The lambda keyword creates an anonymous function within a Python expression.

However, the simple syntax of Python limits the body of lambda functions to be pure expressions. In other words, the body cannot contain other Python statements such as while, try, etc. Assignment with = is also a statement, so it cannot occur in a lambda. The new assignment expression syntax using := can be used—but if you need it, your lambda is probably too complicated and hard to read, and it should be refactored in to a regular function using def.

The best use of anonymous functions is in the context of an argument list for a higher-order function. For example, Example 7-7 is the rhyme index example from Example 7-4 rewritten with lambda, without defining a reverse function.

>>> fruits = ['strawberry', 'fig', 'apple', 'cherry', 'raspberry', 'banana']
>>> sorted(fruits, key=lambda word: word[::-1])
['banana', 'apple', 'fig', 'raspberry', 'strawberry', 'cherry']
>>>

Outside the limited context of arguments to higher-order functions, anonymous functions are rarely useful in Python. The syntactic restrictions tend to make nontrivial lambdas either unreadable or unworkable. If a lambda is hard to read, I strongly advise you follow Fredrik Lund’s refactoring advice.

The lambda syntax is just syntactic sugar: a lambda expression creates a function object just like the def statement. That is just one of several kinds of callable objects in Python. The following section reviews all of them.

The Nine Flavors of Callable Objects

The call operator () may be applied to other objects beyond user-defined functions and lambdas. To determine whether an object is callable, use the callable() built-in function. As of Python 3.8, the Data Model documentation lists nine callable types:

User-defined functions

Created with def statements or lambda expressions.

Built-in functions

A function implemented in C (for CPython), like len or time.strftime.

Built-in methods

Methods implemented in C, like dict.get.

Methods

Functions defined in the body of a class.

Classes

When invoked, a class runs its __new__ method to create an instance, then __init__ to initialize it, and finally the instance is returned to the caller. Because there is no new operator in Python, calling a class is like calling a function.²

Class instances

If a class defines a __call__ method, then its instances may be invoked as functions—that’s the subject of the next section.

Generator functions

Functions or methods that use the yield keyword in their body. When called, they return a generator object.

Native coroutine functions

Functions or methods defined with async def. When called, they return a coroutine object. Added in Python 3.5.

Asynchronous generator functions

Functions or methods defined with async def that have yield in their body. When called, they return an asynchronous generator for use with async for. Added in Python 3.6.

Generators, coroutines, and asynchronous generators are unlike other callables in many respects. Generators are covered in [Link to Come]. They can also be used as classic coroutines, which are covered in [Link to Come]. [Link to Come] covers native coroutines and asynchronous generator functions.

>>> abs, str, 13
(<built-in function abs>, <class 'str'>, 13)
>>> [callable(obj) for obj in (abs, str, 13)]
[True, True, False]

We now move on to building class instances that work as callable objects.

User-Defined Callable Types

Not only are Python functions real objects, but arbitrary Python objects may also be made to behave like functions. Implementing a __call__ instance method is all it takes.

Example 7-8 implements a BingoCage class. An instance is built from any iterable, and stores an internal list of items, in random order. Calling the instance pops an item.

import random

class BingoCage:

    def __init__(self, items):
        self._items = list(items)  
        random.shuffle(self._items)  

    def pick(self):  
        try:
            return self._items.pop()
        except IndexError:
            raise LookupError('pick from empty BingoCage')  

    def __call__(self):  
        return self.pick()

__init__ accepts any iterable; building a local copy prevents unexpected side effects on any list passed as an argument.

shuffle is guaranteed to work because self._items is a list.

The main method.

Raise exception with custom message if self._items is empty.

Shortcut to bingo.pick(): bingo().

Here is a simple demo of Example 7-8. Note how a bingo instance can be invoked as a function, and the callable(…) built-in recognizes it as a callable object:

>>> bingo = BingoCage(range(3))
>>> bingo.pick()
1
>>> bingo()
0
>>> callable(bingo)
True

A class implementing __call__ is an easy way to create function-like objects that have some internal state that must be kept across invocations, like the remaining items in the BingoCage. Another good use case for __call__ is implementing decorators. Decorators must be callable, and it is sometimes convenient to be able to “remember” something between calls of the decorator (e.g., for memoization—caching the results of expensive computations for later use) or to split complexity into separate methods.

The functional approach to creating functions with internal state is to use closures. Closures, as well as decorators, are the subject of Chapter 9.

We now move on to another aspect of handling functions as objects: runtime introspection.

Function Introspection

Function objects have many attributes beyond __doc__. See what the dir function reveals about our factorial:

>>> dir(factorial)
['__annotations__', '__call__', '__class__', '__closure__', '__code__',
'__defaults__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__',
'__format__', '__ge__', '__get__', '__getattribute__', '__globals__',
'__gt__', '__hash__', '__init__', '__kwdefaults__', '__le__', '__lt__',
'__module__', '__name__', '__ne__', '__new__', '__qualname__', '__reduce__',
'__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__',
'__subclasshook__']
>>>

Most of these attributes are common to Python objects in general. In this section, we cover those that are especially relevant to treating functions as objects, starting with __dict__.

Like the instances of a plain user-defined class, a function uses the __dict__ attribute to store user attributes assigned to it. This is useful as a primitive form of annotation. Assigning arbitrary attributes to functions is not a very common practice in general, but Django is one framework that uses it. See, for example, the short_description, boolean, and allow_tags attributes described in The Django admin site documentation. In the Django docs, this example shows attaching a short_description to a method, to determine the description that will appear in record listings in the Django admin when that method is used:

    def upper_case_name(obj):
        return ("%s %s" % (obj.first_name, obj.last_name)).upper()
    upper_case_name.short_description = 'Customer name'

Now let us focus on the attributes that are specific to functions and are not found in a generic Python user-defined object. Computing the difference of two sets quickly gives us a list of the function-specific attributes (see Example 7-9).

>>> class C: pass  
>>> obj = C()  
>>> def func(): pass  
>>> sorted(set(dir(func)) - set(dir(obj))) 
['__annotations__', '__call__', '__closure__', '__code__', '__defaults__',
'__get__', '__globals__', '__kwdefaults__', '__name__', '__qualname__']
>>>

Create bare user-defined class.

Make an instance of it.

Create a bare function.

Using set difference, generate a sorted list of the attributes that exist in a function but not in an instance of a bare class.

Table 7-1 shows a summary of the attributes listed by Example 7-9.

In later sections and chapters we will discuss the __defaults__, __code__, and __annotations__ functions, used by IDEs and frameworks to extract information about function signatures. But to fully appreciate these attributes, we will make a detour to explore the powerful syntax Python offers to declare function parameters and to pass arguments into them.

From Positional to Keyword-Only Parameters

One of the best features of Python functions is the extremely flexible parameter handling mechanism, enhanced with keyword-only arguments in Python 3. Closely related are the use of * and ** to “explode” iterables and mappings into separate arguments when we call a function. To see these features in action, see the code for Example 7-10 and tests showing its use in Example 7-11.

def tag(name, *content, class_=None, **attrs):
    """Generate one or more HTML tags"""
    if class_ is not None:
        attrs['class'] = class_
    if attrs:
        attr_pairs = (f' {attr}="{value}"' for attr, value
                      in sorted(attrs.items()))
        attr_str = ''.join(attr_pairs)
    else:
        attr_str = ''
    if content:
        elements = (f'<{name}{attr_str}>{c}</{name}>'
                    for c in content)
        return '
'.join(elements)
    else:
        return f'<{name}{attr_str} />'

The tag function can be invoked in many ways, as Example 7-11 shows.

>>> tag('br')  
'<br />'
>>> tag('p', 'hello')  
'<p>hello</p>'
>>> print(tag('p', 'hello', 'world'))
<p>hello</p>
<p>world</p>
>>> tag('p', 'hello', id=33)  
'<p id="33">hello</p>'
>>> print(tag('p', 'hello', 'world', class_='sidebar'))  
<p class="sidebar">hello</p>
<p class="sidebar">world</p>
>>> tag(content='testing', name="img")  
'<img content="testing" />'
>>> my_tag = {'name': 'img', 'title': 'Sunset Boulevard',
...           'src': 'sunset.jpg', 'class': 'framed'}
>>> tag(**my_tag)  
'<img class="framed" src="sunset.jpg" title="Sunset Boulevard" />'

A single positional argument produces an empty tag with that name.

Any number of arguments after the first are captured by *content as a tuple.

Keyword arguments not explicitly named in the tag signature are captured by **attrs as a dict.

The class_ parameter can only be passed as a keyword argument.

Even the first positional argument can be passed as a keyword when tag is called.

Prefixing the my_tag dict with ** passes all its items as separate arguments, which are then bound to the named parameters, with the remaining caught by **attrs. In this case we can have a 'class' key in the arguments dict, because it is a string, and does not clash with the class reserved word.

Keyword-only arguments are a new feature in Python 3. In Example 7-10, the class_ parameter can only be given as a keyword argument—it will never capture unnamed positional arguments. To specify keyword-only arguments when defining a function, name them after the argument prefixed with *. If you don’t want to support variable positional arguments but still want keyword-only arguments, put a * by itself in the signature, like this:

>>> def f(a, *, b):
...     return a, b
...
>>> f(1, b=2)
(1, 2)

Note that keyword-only arguments do not need to have a default value: they can be mandatory, like b in the preceding example.

def divmod(a, b, /):
    return (a // b, a % b)

def tag(name, /, *content, class_=None, **attrs):
    ...

Retrieving Information About Parameters

An interesting application of function introspection can be found in the Bobo HTTP micro-framework. To see that in action, consider a variation of the Bobo tutorial “Hello world” application in Example 7-12.

import bobo

@bobo.query('/')
def hello(person):
    return 'Hello %s!' % person

The bobo.query decorator integrates a plain function such as hello with the request handling machinery of the framework. We’ll cover decorators in Chapter 9—that’s not the point of this example here. The point is that Bobo introspects the hello function and finds out it needs one parameter named person to work, and it will retrieve a parameter with that name from the request and pass it to hello, so the programmer doesn’t need deal with request object directly. This also makes unit testing easier: there is no need to mock the request object to test the hello function.

If you install Bobo and point its development server to the script in Example 7-12 (e.g., bobo -f hello.py), a hit on the URL http://localhost:8080/ will produce the message “Missing form variable person” with a 403 HTTP code. This happens because Bobo understands that the person argument is required to call hello, but no such name was found in the request. Example 7-13 is a shell session using curl to show this behavior.

$ curl -i http://localhost:8080/
HTTP/1.0 403 Forbidden
Date: Thu, 21 Aug 2014 21:39:44 GMT
Server: WSGIServer/0.2 CPython/3.4.1
Content-Type: text/html; charset=UTF-8
Content-Length: 103

<html>
<head><title>Missing parameter</title></head>
<body>Missing form variable person</body>
</html>

However, if you get http://localhost:8080/?person=Jim, the response will be the string 'Hello Jim!'. See Example 7-14.

$ curl -i http://localhost:8080/?person=Jim
HTTP/1.0 200 OK
Date: Thu, 21 Aug 2014 21:42:32 GMT
Server: WSGIServer/0.2 CPython/3.4.1
Content-Type: text/html; charset=UTF-8
Content-Length: 10

Hello Jim!

How does Bobo know which parameter names are required by the function, and whether they have default values or not?

Within a function object, the __defaults__ attribute holds a tuple with the default values of positional and keyword arguments. The defaults for keyword-only arguments appear in __kwdefaults__. The names of the arguments, however, are found within the __code__ attribute, which is a reference to a code object with many attributes of its own.

To demonstrate the use of these attributes, we will inspect the function clip in a module clip.py, listed in Example 7-15.

def clip(text, max_len=80):
    """Return text clipped at the last space before or after max_len
    """
    end = None
    if len(text) > max_len:
        space_before = text.rfind(' ', 0, max_len)
        if space_before >= 0:
            end = space_before
        else:
            space_after = text.rfind(' ', max_len)
            if space_after >= 0:
                end = space_after
    if end is None:  # no spaces were found
        return text.rstrip()
    return text[:end].rstrip()

Example 7-16 shows the values of __defaults__, __code__.co_varnames, and __code__.co_argcount for the clip function listed in Example 7-15.

>>> from clip import clip
>>> clip.__defaults__
(80,)
>>> clip.__code__  # doctest: +ELLIPSIS
<code object clip at 0x...>
>>> clip.__code__.co_varnames
('text', 'max_len', 'end', 'space_before', 'space_after')
>>> clip.__code__.co_argcount
2

As you can see, this is not the most convenient arrangement of information. The argument names appear in __code__.co_varnames, but that also includes the names of the local variables created in the body of the function. Therefore, the argument names are the first N strings, where N is given by __code__.co_argcount which—by the way—does not include any variable arguments prefixed with * or **. The default values are identified only by their position in the __defaults__ tuple, so to link each with the respective argument, you have to scan from last to first. In the example, we have two arguments, text and max_len, and one default, 80, so it must belong to the last argument, max_len. This is awkward.

Fortunately, there is a better way: the inspect module.

Take a look at Example 7-17.

>>> from clip import clip
>>> from inspect import signature
>>> sig = signature(clip)
>>> sig  # doctest: +ELLIPSIS
<inspect.Signature object at 0x...>
>>> str(sig)
'(text, max_len=80)'
>>> for name, param in sig.parameters.items():
...     print(param.kind, ':', name, '=', param.default)
...
POSITIONAL_OR_KEYWORD : text = <class 'inspect._empty'>
POSITIONAL_OR_KEYWORD : max_len = 80

This is much better. inspect.signature returns an inspect.Signature object, which has a parameters attribute that lets you read an ordered mapping of names to inspect.Parameter objects. Each Parameter instance has attributes such as name, default, and kind. The special value inspect._empty denotes parameters with no default, which makes sense considering that None is a valid—and popular—default value.

The kind attribute holds one of five possible values from the _ParameterKind class:

POSITIONAL_OR_KEYWORD

A parameter that may be passed as a positional or as a keyword argument (most Python function parameters are of this kind).

VAR_POSITIONAL

A tuple of positional parameters.

VAR_KEYWORD

A dict of keyword parameters.

KEYWORD_ONLY

A keyword-only parameter (new in Python 3).

POSITIONAL_ONLY

A positional-only parameter; unsupported by function declaration syntax before Python 3.8, but exemplified by existing functions implemented in C—like divmod—that do not accept parameters passed by keyword.

Besides name, default, and kind, inspect.Parameter objects have an annotation attribute that is usually inspect._empty but may contain function signature metadata provided via the new annotations syntax in Python 3—covered in Chapter 8.

An inspect.Signature object has a bind method that takes any number of arguments and binds them to the parameters in the signature, applying the usual rules for matching actual arguments to formal parameters. This can be used by a framework to validate arguments prior to the actual function invocation. Example 7-18 shows how.

>>> import inspect
>>> sig = inspect.signature(tag)  
>>> my_tag = {'name': 'img', 'title': 'Sunset Boulevard',
...           'src': 'sunset.jpg', 'cls': 'framed'}
>>> bound_args = sig.bind(**my_tag)  
>>> bound_args
<inspect.BoundArguments object at 0x...>  
>>> for name, value in bound_args.arguments.items():  
...     print(name, '=', value)
...
name = img
cls = framed
attrs = {'title': 'Sunset Boulevard', 'src': 'sunset.jpg'}
>>> del my_tag['name']  
>>> bound_args = sig.bind(**my_tag)  
Traceback (most recent call last):
  ...
TypeError: 'name' parameter lacking default value

Get the signature from tag function in Example 7-10.

Pass a dict of arguments to .bind().

An inspect.BoundArguments object is produced.

Iterate over the items in bound_args.arguments, which is an OrderedDict, to display the names and values of the arguments.

Remove the mandatory argument name from my_tag.

Calling sig.bind(**my_tag) raises a TypeError complaining of the missing name parameter.

This example shows how the Python Data Model, with the help of inspect, exposes the same machinery the interpreter uses to bind arguments to formal parameters in function calls.

Frameworks and tools like IDEs can use this information to validate code. Another feature of Python 3, function annotations, enhances the possible uses of this, as we’ll see in Chapter 8.

After examining the anatomy of function objects, the remainder of this chapter covers the most useful packages in the standard library for programming in a functional style.

Packages for Functional Programming

Although Guido makes it clear that he did not design Python to be a functional programming language, a functional coding style can be used to good extent, thanks to first-class functions and the support of packages like operator and functools, which we cover in the next two sections.

from functools import reduce

def fact(n):
    return reduce(lambda a, b: a*b, range(1, n+1))

from functools import reduce
from operator import mul

def fact(n):
    return reduce(mul, range(1, n+1))

>>> metro_data = [
...     ('Tokyo', 'JP', 36.933, (35.689722, 139.691667)),
...     ('Delhi NCR', 'IN', 21.935, (28.613889, 77.208889)),
...     ('Mexico City', 'MX', 20.142, (19.433333, -99.133333)),
...     ('New York-Newark', 'US', 20.104, (40.808611, -74.020386)),
...     ('Sao Paulo', 'BR', 19.649, (-23.547778, -46.635833)),
... ]
>>>
>>> from operator import itemgetter
>>> for city in sorted(metro_data, key=itemgetter(1)):
...     print(city)
...
('Sao Paulo', 'BR', 19.649, (-23.547778, -46.635833))
('Delhi NCR', 'IN', 21.935, (28.613889, 77.208889))
('Tokyo', 'JP', 36.933, (35.689722, 139.691667))
('Mexico City', 'MX', 20.142, (19.433333, -99.133333))
('New York-Newark', 'US', 20.104, (40.808611, -74.020386))

>>> cc_name = itemgetter(1, 0)
>>> for city in metro_data:
...     print(cc_name(city))
...
('JP', 'Tokyo')
('IN', 'Delhi NCR')
('MX', 'Mexico City')
('US', 'New York-Newark')
('BR', 'Sao Paulo')
>>>

>>> from collections import namedtuple
>>> LatLong = namedtuple('LatLong', 'lat long')  
>>> Metropolis = namedtuple('Metropolis', 'name cc pop coord')  
>>> metro_areas = [Metropolis(name, cc, pop, LatLong(lat, long))  
...     for name, cc, pop, (lat, long) in metro_data]
>>> metro_areas[0]
Metropolis(name='Tokyo', cc='JP', pop=36.933, coord=LatLong(lat=35.689722,
long=139.691667))
>>> metro_areas[0].coord.lat  
35.689722
>>> from operator import attrgetter
>>> name_lat = attrgetter('name', 'coord.lat')  
>>>
>>> for city in sorted(metro_areas, key=attrgetter('coord.lat')):  
...     print(name_lat(city))  
...
('Sao Paulo', -23.547778)
('Mexico City', 19.433333)
('Delhi NCR', 28.613889)
('Tokyo', 35.689722)
('New York-Newark', 40.808611)

>>> [name for name in dir(operator) if not name.startswith('_')]
['abs', 'add', 'and_', 'attrgetter', 'concat', 'contains',
'countOf', 'delitem', 'eq', 'floordiv', 'ge', 'getitem', 'gt',
'iadd', 'iand', 'iconcat', 'ifloordiv', 'ilshift', 'imod', 'imul',
'index', 'indexOf', 'inv', 'invert', 'ior', 'ipow', 'irshift',
'is_', 'is_not', 'isub', 'itemgetter', 'itruediv', 'ixor', 'le',
'length_hint', 'lshift', 'lt', 'methodcaller', 'mod', 'mul', 'ne',
'neg', 'not_', 'or_', 'pos', 'pow', 'rshift', 'setitem', 'sub',
'truediv', 'truth', 'xor']

>>> from operator import methodcaller
>>> s = 'The time has come'
>>> upcase = methodcaller('upper')
>>> upcase(s)
'THE TIME HAS COME'
>>> hiphenate = methodcaller('replace', ' ', '-')
>>> hiphenate(s)
'The-time-has-come'

>>> str.upper(s)
'THE TIME HAS COME'

>>> from operator import mul
>>> from functools import partial
>>> triple = partial(mul, 3)  
>>> triple(7)  
21
>>> list(map(triple, range(1, 10)))  
[3, 6, 9, 12, 15, 18, 21, 24, 27]

>>> import unicodedata, functools
>>> nfc = functools.partial(unicodedata.normalize, 'NFC')
>>> s1 = 'café'
>>> s2 = 'cafeu0301'
>>> s1, s2
('café', 'café')
>>> s1 == s2
False
>>> nfc(s1) == nfc(s2)
True

>>> from tagger import tag
>>> tag
<function tag at 0x10206d1e0>  
>>> from functools import partial
>>> picture = partial(tag, 'img', cls='pic-frame')  
>>> picture(src='wumpus.jpeg')
'<img class="pic-frame" src="wumpus.jpeg" />'  
>>> picture
functools.partial(<function tag at 0x10206d1e0>, 'img', cls='pic-frame')  
>>> picture.func  
<function tag at 0x10206d1e0>
>>> picture.args
('img',)
>>> picture.keywords
{'cls': 'pic-frame'}

Chapter Summary

The goal of this chapter was to explore the first-class nature of functions in Python. The main ideas are that you can assign functions to variables, pass them to other functions, store them in data structures, and access function attributes, allowing frameworks and tools to act on that information.

Higher-order functions, a staple of functional programming, are common in Python. Using map, filter, and reduce is not as common as it used to be—thanks to list comprehensions (and similar constructs like generator expressions) and the appearance of reducing built-ins like sum, all, and any. The sorted, min, max built-ins, and functools.partial are examples of commonly used higher-order functions in the language.

Callables come in nine different flavors since Python 3.6, from the simple functions created with lambda to instances of classes implementing __call__. Generators and coroutines are also callable, although their behavior is very different from other callables. All callables can be detected by the callable() built-in. Every callable supports the same rich syntax for declaring formal parameters, including keyword-only parameters and annotations—both new features introduced with Python 3.

Lastly, we covered some functions from the operator module and functools.partial, which facilitate functional programming by minimizing the need for the functionally challenged lambda syntax.

Further Reading

Don’t Make It Callable is a post from 2019 where Moshe Zadka—core developer of Python and Twisted—argues against implementing __call__ except in the very specific case of writing a DSL (Domain-Specific Language), which are specialized APIs where it is OK to break with convention to provide users with a “succint syntax dedicated to the task at hand.”

The next chapters continue our exploration of programming with function objects. Chapter 9 dives into function decorators—a special kind of higher-order function—and the closure mechanism that makes them work. Chapter 8 is devoted to type hints in function parameters and return values. Chapter 10 shows how first-class functions can simplify some classic object-oriented design patterns. The code in that chapter also show several uses of type hints.

In The Python Language Reference, “3.2. The standard type hierarchy” presents the nine callable types, along with all the other built-in types.

Chapter 7 of the Python Cookbook, Third Edition (O’Reilly), by David Beazley and Brian K. Jones, is an excellent complement to the current chapter as well as Chapter 9 of this book, covering mostly the same concepts with a different approach.

See PEP 3102 — Keyword-Only Arguments if you are interested in the rationale and use cases for that feature.

A great introduction to functional programming in Python is A. M. Kuchling’s Python Functional Programming HOWTO. The main focus of that text, however, is the use of iterators and generators, which are the subject of [Link to Come].

fn.py is a package to support functional programming in Python 2 and 3. According to its author, Alexey Kachayev, fn.py provides “implementation of missing features to enjoy FP” in Python. It includes a @recur.tco decorator that implements tail-call optimization for unlimited recursion in Python, among many other functions, data structures, and recipes. However, be aware that fn.py had its last updates in 2014, so it may be “done” or just abandoned.

The StackOverflow question “Python: Why is functools.partial necessary?” has a highly informative (and funny) reply by Alex Martelli, co-author of the classic Python in a Nutshell.

Jim Fulton’s Bobo was probably the first Web framework that deserved to be called object-oriented. If you were intrigued by it and want to learn more about its modern rewrite, start at its Introduction. A little of the early history of Bobo appears in a comment by Phillip J. Eby in a discussion at Joel Spolsky’s blog.

Reflecting on the question “Is Python a functional language?”, I created Beyond Paradigms, a talk presented at PyCaribbean, PyBay and PyConDE (slides, video).