Chapter 7. First-Class Functions

I have never considered Python to be heavily influenced by functional languages, no matter what people say or think. I was much more familiar with imperative languages such as C and Algol 68 and although I had made functions first-class objects, I didn’t view Python as a functional programming language.1

Guido van Rossum, Python BDFL

Functions in Python are first-class objects. Programming language theorists define a “first-class object” as a program entity that can be:

  • created at runtime;

  • assigned to a variable or element in a data structure;

  • passed as an argument to a function;

  • returned as the result of a function.

Integers, strings, and dictionaries are other examples of first-class objects in Python—nothing fancy here. But if you came to Python from a language where functions are not first-class citizens, this chapter and the rest of Part III of the book focuses on the implications and practical applications of treating functions as objects.

Tip

The term “first-class functions” is widely used as shorthand for “functions as first-class objects.” It’s not perfect because it seems to imply an “elite” among functions. In Python, all functions are first-class.

What’s new in this chapter

“Positional-only parameters” is a new section, covering a feature added in Python 3.8. Section “The Nine Flavors of Callable Objects” was titled “The Seven Flavors of Callable Objects” in the first edition. It now mentions native coroutines and asynchronous generators, introduced in Python 3.5 and 3.6, respectively. Both are covered in later chapters, but here they are listed along with the other callables for completeness.

I moved section “Reading annotations at runtime” to the new Chapter 8, which is about type hints in function definitions. When I wrote the first edition, PEP 484 — Type Hints was still being discussed, and people used annotations in different ways. Since Python 3.5, annotations should conform to PEP 484. Therefore, the best place to cover them is Chapter 8.

Now let’s see why Python functions are full-fledged objects.

Treating a Function Like an Object

The console session in Example 7-1 shows that Python functions are objects. Here we create a function, call it, read its __doc__ attribute, and check that the function object itself is an instance of the function class.

Example 7-1. Create and test a function, then read its __doc__ and check its type
>>> def factorial(n):  1
...     '''returns n!'''
...     return 1 if n < 2 else n * factorial(n-1)
...
>>> factorial(42)
1405006117752879898543142606244511569936384000000000
>>> factorial.__doc__  2
'returns n!'
>>> type(factorial)  3
<class 'function'>
1

This is a console session, so we’re creating a function at “runtime.”

2

__doc__ is one of several attributes of function objects.

3

factorial is an instance of the function class.

The __doc__ attribute is used to generate the help text of an object. In the Python console, the command help(factorial) will display a screen like Figure 7-1.

Help screen for the factorial function
Figure 7-1. Help screen for factorial; the text is built from the __doc__ attribute of the function.

Example 7-2 shows the “first class” nature of a function object. We can assign it a variable fact and call it through that name. We can also pass factorial as an argument to map. The map function returns an iterable where each item is the result of the application of the first argument (a function) to succesive elements of the second argument (an iterable), range(10) in this example.

Example 7-2. Use function through a different name, and pass function as argument
>>> fact = factorial
>>> fact
<function factorial at 0x...>
>>> fact(5)
120
>>> map(factorial, range(11))
<map object at 0x...>
>>> list(map(fact, range(11)))
[1, 1, 2, 6, 24, 120, 720, 5040, 40320, 362880, 3628800]

Having first-class functions enables programming in a functional style. One of the hallmarks of functional programming is the use of higher-order functions, our next topic.

Higher-Order Functions

A function that takes a function as argument or returns a function as the result is a higher-order function. One example is map, shown in Example 7-2. Another is the built-in function sorted: the optional key argument lets you provide a function to be applied to each item for sorting, as seen in “list.sort and the sorted Built-In Function”.

For example, to sort a list of words by length, simply pass the len function as the key, as in Example 7-3.

Example 7-3. Sorting a list of words by length
>>> fruits = ['strawberry', 'fig', 'apple', 'cherry', 'raspberry', 'banana']
>>> sorted(fruits, key=len)
['fig', 'apple', 'cherry', 'banana', 'raspberry', 'strawberry']
>>>

Any one-argument function can be used as the key. For example, to create a rhyme dictionary it might be useful to sort each word spelled backward. In Example 7-4, note that the words in the list are not changed at all; only their reversed spelling is used as the sort criterion, so that the berries appear together.

Example 7-4. Sorting a list of words by their reversed spelling
>>> def reverse(word):
...     return word[::-1]
>>> reverse('testing')
'gnitset'
>>> sorted(fruits, key=reverse)
['banana', 'apple', 'fig', 'raspberry', 'strawberry', 'cherry']
>>>

In the functional programming paradigm, some of the best known higher-order functions are map, filter, reduce, and apply. The apply function was deprecated in Python 2.3 and removed in Python 3 because it’s no longer necessary. If you need to call a function with a dynamic set of arguments, you can write fn(*args, **kwargs) instead of apply(fn, args, kwargs).

The map, filter, and reduce higher-order functions are still around, but better alternatives are available for most of their use cases, as the next section shows.

Modern Replacements for map, filter, and reduce

Functional languages commonly offer the map, filter, and reduce higher-order functions (sometimes with different names). The map and filter functions are still built-ins in Python 3, but since the introduction of list comprehensions and generator expressions, they are not as important. A listcomp or a genexp does the job of map and filter combined, but is more readable. Consider Example 7-5.

Example 7-5. Lists of factorials produced with map and filter compared to alternatives coded as list comprehensions
>>> list(map(fact, range(6)))  1
[1, 1, 2, 6, 24, 120]
>>> [fact(n) for n in range(6)]  2
[1, 1, 2, 6, 24, 120]
>>> list(map(factorial, filter(lambda n: n % 2, range(6))))  3
[1, 6, 120]
>>> [factorial(n) for n in range(6) if n % 2]  4
[1, 6, 120]
>>>
1

Build a list of factorials from 0! to 5!.

2

Same operation, with a list comprehension.

3

List of factorials of odd numbers up to 5!, using both map and filter.

4

List comprehension does the same job, replacing map and filter, and making lambda unnecessary.

In Python 3, map and filter return generators—a form of iterator—so their direct substitute is now a generator expression (in Python 2, these functions returned lists, therefore their closest alternative is a listcomp).

The reduce function was demoted from a built-in in Python 2 to the functools module in Python 3. Its most common use case, summation, is better served by the sum built-in available since Python 2.3 was released in 2003. This is a big win in terms of readability and performance (see Example 7-6).

Example 7-6. Sum of integers up to 99 performed with reduce and sum
>>> from functools import reduce  1
>>> from operator import add  2
>>> reduce(add, range(100))  3
4950
>>> sum(range(100))  4
4950
>>>
1

Starting with Python 3.0, reduce is not a built-in.

2

Import add to avoid creating a function just to add two numbers.

3

Sum integers up to 99.

4

Same task using sum; reduce and adding function are not needed.

Note

The common idea of sum and reduce is to apply some operation to successive items in a sequence, accumulating previous results, thus reducing a sequence of values to a single value.

Other reducing built-ins are all and any:

all(iterable)

Returns True if every element of the iterable is truthy; all([]) returns True.

any(iterable)

Returns True if any element of the iterable is truthy; any([]) returns False.

I give a fuller explanation of reduce in “Vector Take #4: Hashing and a Faster ==” where an ongoing example provides a meaningful context for the use of this function. The reducing functions are summarized later in the book when iterables are in focus, in [Link to Come].

To use a higher-order function, sometimes it is convenient to create a small, one-off function. That is why anonymous functions exist. We’ll cover them next.

Anonymous Functions

The lambda keyword creates an anonymous function within a Python expression.

However, the simple syntax of Python limits the body of lambda functions to be pure expressions. In other words, the body cannot contain other Python statements such as while, try, etc. Assignment with = is also a statement, so it cannot occur in a lambda. The new assignment expression syntax using := can be used—but if you need it, your lambda is probably too complicated and hard to read, and it should be refactored in to a regular function using def.

The best use of anonymous functions is in the context of an argument list for a higher-order function. For example, Example 7-7 is the rhyme index example from Example 7-4 rewritten with lambda, without defining a reverse function.

Example 7-7. Sorting a list of words by their reversed spelling using lambda
>>> fruits = ['strawberry', 'fig', 'apple', 'cherry', 'raspberry', 'banana']
>>> sorted(fruits, key=lambda word: word[::-1])
['banana', 'apple', 'fig', 'raspberry', 'strawberry', 'cherry']
>>>

Outside the limited context of arguments to higher-order functions, anonymous functions are rarely useful in Python. The syntactic restrictions tend to make nontrivial lambdas either unreadable or unworkable. If a lambda is hard to read, I strongly advise you follow Fredrik Lund’s refactoring advice.

The lambda syntax is just syntactic sugar: a lambda expression creates a function object just like the def statement. That is just one of several kinds of callable objects in Python. The following section reviews all of them.

The Nine Flavors of Callable Objects

The call operator () may be applied to other objects beyond user-defined functions and lambdas. To determine whether an object is callable, use the callable() built-in function. As of Python 3.8, the Data Model documentation lists nine callable types:

User-defined functions

Created with def statements or lambda expressions.

Built-in functions

A function implemented in C (for CPython), like len or time.strftime.

Built-in methods

Methods implemented in C, like dict.get.

Methods

Functions defined in the body of a class.

Classes

When invoked, a class runs its __new__ method to create an instance, then __init__ to initialize it, and finally the instance is returned to the caller. Because there is no new operator in Python, calling a class is like calling a function.2

Class instances

If a class defines a __call__ method, then its instances may be invoked as functions—that’s the subject of the next section.

Generator functions

Functions or methods that use the yield keyword in their body. When called, they return a generator object.

Native coroutine functions

Functions or methods defined with async def. When called, they return a coroutine object. Added in Python 3.5.

Asynchronous generator functions

Functions or methods defined with async def that have yield in their body. When called, they return an asynchronous generator for use with async for. Added in Python 3.6.

Generators, coroutines, and asynchronous generators are unlike other callables in many respects. Generators are covered in [Link to Come]. They can also be used as classic coroutines, which are covered in [Link to Come]. [Link to Come] covers native coroutines and asynchronous generator functions.

Tip

Given the variety of existing callable types in Python, the safest way to determine whether an object is callable is to use the callable() built-in:

>>> abs, str, 13
(<built-in function abs>, <class 'str'>, 13)
>>> [callable(obj) for obj in (abs, str, 13)]
[True, True, False]

We now move on to building class instances that work as callable objects.

User-Defined Callable Types

Not only are Python functions real objects, but arbitrary Python objects may also be made to behave like functions. Implementing a __call__ instance method is all it takes.

Example 7-8 implements a BingoCage class. An instance is built from any iterable, and stores an internal list of items, in random order. Calling the instance pops an item.

Example 7-8. bingocall.py: A BingoCage does one thing: picks items from a shuffled list
import random

class BingoCage:

    def __init__(self, items):
        self._items = list(items)  1
        random.shuffle(self._items)  2

    def pick(self):  3
        try:
            return self._items.pop()
        except IndexError:
            raise LookupError('pick from empty BingoCage')  4

    def __call__(self):  5
        return self.pick()
1

__init__ accepts any iterable; building a local copy prevents unexpected side effects on any list passed as an argument.

2

shuffle is guaranteed to work because self._items is a list.

3

The main method.

4

Raise exception with custom message if self._items is empty.

5

Shortcut to bingo.pick(): bingo().

Here is a simple demo of Example 7-8. Note how a bingo instance can be invoked as a function, and the callable(…) built-in recognizes it as a callable object:

>>> bingo = BingoCage(range(3))
>>> bingo.pick()
1
>>> bingo()
0
>>> callable(bingo)
True

A class implementing __call__ is an easy way to create function-like objects that have some internal state that must be kept across invocations, like the remaining items in the BingoCage. Another good use case for __call__ is implementing decorators. Decorators must be callable, and it is sometimes convenient to be able to “remember” something between calls of the decorator (e.g., for memoization—caching the results of expensive computations for later use) or to split complexity into separate methods.

The functional approach to creating functions with internal state is to use closures. Closures, as well as decorators, are the subject of Chapter 9.

We now move on to another aspect of handling functions as objects: runtime introspection.

Function Introspection

Function objects have many attributes beyond __doc__. See what the dir function reveals about our factorial:

>>> dir(factorial)
['__annotations__', '__call__', '__class__', '__closure__', '__code__',
'__defaults__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__',
'__format__', '__ge__', '__get__', '__getattribute__', '__globals__',
'__gt__', '__hash__', '__init__', '__kwdefaults__', '__le__', '__lt__',
'__module__', '__name__', '__ne__', '__new__', '__qualname__', '__reduce__',
'__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__',
'__subclasshook__']
>>>

Most of these attributes are common to Python objects in general. In this section, we cover those that are especially relevant to treating functions as objects, starting with __dict__.

Like the instances of a plain user-defined class, a function uses the __dict__ attribute to store user attributes assigned to it. This is useful as a primitive form of annotation. Assigning arbitrary attributes to functions is not a very common practice in general, but Django is one framework that uses it. See, for example, the short_description, boolean, and allow_tags attributes described in The Django admin site documentation. In the Django docs, this example shows attaching a short_description to a method, to determine the description that will appear in record listings in the Django admin when that method is used:

    def upper_case_name(obj):
        return ("%s %s" % (obj.first_name, obj.last_name)).upper()
    upper_case_name.short_description = 'Customer name'

Now let us focus on the attributes that are specific to functions and are not found in a generic Python user-defined object. Computing the difference of two sets quickly gives us a list of the function-specific attributes (see Example 7-9).

Example 7-9. Listing attributes of functions that don’t exist in plain instances
>>> class C: pass  1
>>> obj = C()  2
>>> def func(): pass  3
>>> sorted(set(dir(func)) - set(dir(obj))) 4
['__annotations__', '__call__', '__closure__', '__code__', '__defaults__',
'__get__', '__globals__', '__kwdefaults__', '__name__', '__qualname__']
>>>
1

Create bare user-defined class.

2

Make an instance of it.

3

Create a bare function.

4

Using set difference, generate a sorted list of the attributes that exist in a function but not in an instance of a bare class.

Table 7-1 shows a summary of the attributes listed by Example 7-9.

Table 7-1. Attributes of user-defined functions
Name Type Description

__annotations__

dict

Parameter and return type hints

__call__

method-wrapper

Implementation of the () operator; a.k.a. the callable object protocol

__closure__

tuple

The function closure, i.e., bindings for free variables (often is None)

__code__

code

Function metadata and function body compiled into bytecode

__defaults__

tuple

Default values for the formal parameters

__get__

method-wrapper

Implementation of the read-only descriptor protocol (see [Link to Come])

__globals__

dict

Reference to global variables of the module where the function is defined

__kwdefaults__

dict

Default values for the keyword-only formal parameters

__name__

str

The function name

__qualname__

str

The qualified function name, e.g., Random.choice (see PEP-3155)

In later sections and chapters we will discuss the __defaults__, __code__, and __annotations__ functions, used by IDEs and frameworks to extract information about function signatures. But to fully appreciate these attributes, we will make a detour to explore the powerful syntax Python offers to declare function parameters and to pass arguments into them.

From Positional to Keyword-Only Parameters

One of the best features of Python functions is the extremely flexible parameter handling mechanism, enhanced with keyword-only arguments in Python 3. Closely related are the use of * and ** to “explode” iterables and mappings into separate arguments when we call a function. To see these features in action, see the code for Example 7-10 and tests showing its use in Example 7-11.

Example 7-10. tag generates HTML elements; a keyword-only argument class_ is used to pass “class” attributes as a workaround because class is a keyword in Python
def tag(name, *content, class_=None, **attrs):
    """Generate one or more HTML tags"""
    if class_ is not None:
        attrs['class'] = class_
    if attrs:
        attr_pairs = (f' {attr}="{value}"' for attr, value
                      in sorted(attrs.items()))
        attr_str = ''.join(attr_pairs)
    else:
        attr_str = ''
    if content:
        elements = (f'<{name}{attr_str}>{c}</{name}>'
                    for c in content)
        return '
'.join(elements)
    else:
        return f'<{name}{attr_str} />'

The tag function can be invoked in many ways, as Example 7-11 shows.

Example 7-11. Some of the many ways of calling the tag function from Example 7-10
>>> tag('br')  1
'<br />'
>>> tag('p', 'hello')  2
'<p>hello</p>'
>>> print(tag('p', 'hello', 'world'))
<p>hello</p>
<p>world</p>
>>> tag('p', 'hello', id=33)  3
'<p id="33">hello</p>'
>>> print(tag('p', 'hello', 'world', class_='sidebar'))  4
<p class="sidebar">hello</p>
<p class="sidebar">world</p>
>>> tag(content='testing', name="img")  5
'<img content="testing" />'
>>> my_tag = {'name': 'img', 'title': 'Sunset Boulevard',
...           'src': 'sunset.jpg', 'class': 'framed'}
>>> tag(**my_tag)  6
'<img class="framed" src="sunset.jpg" title="Sunset Boulevard" />'
1

A single positional argument produces an empty tag with that name.

2

Any number of arguments after the first are captured by *content as a tuple.

3

Keyword arguments not explicitly named in the tag signature are captured by **attrs as a dict.

4

The class_ parameter can only be passed as a keyword argument.

5

Even the first positional argument can be passed as a keyword when tag is called.

6

Prefixing the my_tag dict with ** passes all its items as separate arguments, which are then bound to the named parameters, with the remaining caught by **attrs. In this case we can have a 'class' key in the arguments dict, because it is a string, and does not clash with the class reserved word.

Keyword-only arguments are a new feature in Python 3. In Example 7-10, the class_ parameter can only be given as a keyword argument—it will never capture unnamed positional arguments. To specify keyword-only arguments when defining a function, name them after the argument prefixed with *. If you don’t want to support variable positional arguments but still want keyword-only arguments, put a * by itself in the signature, like this:

>>> def f(a, *, b):
...     return a, b
...
>>> f(1, b=2)
(1, 2)

Note that keyword-only arguments do not need to have a default value: they can be mandatory, like b in the preceding example.

Positional-only parameters

Since Python 3.8, user-defined function signatures may specify potitional-only parameters. This feature always existed for built-in functions, such as divmod(a, b), which can only be called with positional parameters, and not as divmod(a=10, b=4).

To define a function requiring positional-only parameters, use / in the parameter list.

This is example from What’s New In Python 3.8 shows how to emulate the divmod built-in function:

def divmod(a, b, /):
    return (a // b, a % b)

All arguments to the left of the / are positional-only. After the /, you may specify other arguments, which work as usual.

Warning

The / in the parameter list is a syntax error in Python 3.7 or earlier.

For example, consider the tag function from Example 7-10. If we want the name parameter to be positional only, we can add a / after it in the function signature, like this:

def tag(name, /, *content, class_=None, **attrs):
    ...

You can find other examples of positional-only parameters in What’s New In Python 3.8 and in PEP 570.

After diving into Python’s flexible argument declaration features, we go back to the introspection of function arguments, starting with a motivating example from a web framework, and on through introspection techniques.

Retrieving Information About Parameters

An interesting application of function introspection can be found in the Bobo HTTP micro-framework. To see that in action, consider a variation of the Bobo tutorial “Hello world” application in Example 7-12.

Note

I mention Bobo because it pionered the use of parameter introspection to reduce boilerplate code in Python Web frameworks—since 1997! The practice is now common. FastAPI is an example of a modern framework that uses the same idea.

Example 7-12. Bobo knows that hello requires a person argument, and retrieves it from the HTTP request
import bobo

@bobo.query('/')
def hello(person):
    return 'Hello %s!' % person

The bobo.query decorator integrates a plain function such as hello with the request handling machinery of the framework. We’ll cover decorators in Chapter 9—that’s not the point of this example here. The point is that Bobo introspects the hello function and finds out it needs one parameter named person to work, and it will retrieve a parameter with that name from the request and pass it to hello, so the programmer doesn’t need deal with request object directly. This also makes unit testing easier: there is no need to mock the request object to test the hello function.

If you install Bobo and point its development server to the script in Example 7-12 (e.g., bobo -f hello.py), a hit on the URL http://localhost:8080/ will produce the message “Missing form variable person” with a 403 HTTP code. This happens because Bobo understands that the person argument is required to call hello, but no such name was found in the request. Example 7-13 is a shell session using curl to show this behavior.

Example 7-13. Bobo issues a 403 forbidden response if there are missing function arguments in the request; curl -i is used to dump the headers to standard output
$ curl -i http://localhost:8080/
HTTP/1.0 403 Forbidden
Date: Thu, 21 Aug 2014 21:39:44 GMT
Server: WSGIServer/0.2 CPython/3.4.1
Content-Type: text/html; charset=UTF-8
Content-Length: 103

<html>
<head><title>Missing parameter</title></head>
<body>Missing form variable person</body>
</html>

However, if you get http://localhost:8080/?person=Jim, the response will be the string 'Hello Jim!'. See Example 7-14.

Example 7-14. Passing the person parameter is required for an OK response
$ curl -i http://localhost:8080/?person=Jim
HTTP/1.0 200 OK
Date: Thu, 21 Aug 2014 21:42:32 GMT
Server: WSGIServer/0.2 CPython/3.4.1
Content-Type: text/html; charset=UTF-8
Content-Length: 10

Hello Jim!

How does Bobo know which parameter names are required by the function, and whether they have default values or not?

Within a function object, the __defaults__ attribute holds a tuple with the default values of positional and keyword arguments. The defaults for keyword-only arguments appear in __kwdefaults__. The names of the arguments, however, are found within the __code__ attribute, which is a reference to a code object with many attributes of its own.

To demonstrate the use of these attributes, we will inspect the function clip in a module clip.py, listed in Example 7-15.

Example 7-15. Function to shorten a string by clipping at a space near the desired length
def clip(text, max_len=80):
    """Return text clipped at the last space before or after max_len
    """
    end = None
    if len(text) > max_len:
        space_before = text.rfind(' ', 0, max_len)
        if space_before >= 0:
            end = space_before
        else:
            space_after = text.rfind(' ', max_len)
            if space_after >= 0:
                end = space_after
    if end is None:  # no spaces were found
        return text.rstrip()
    return text[:end].rstrip()

Example 7-16 shows the values of __defaults__, __code__.co_varnames, and __code__.co_argcount for the clip function listed in Example 7-15.

Example 7-16. Extracting information about the function arguments
>>> from clip import clip
>>> clip.__defaults__
(80,)
>>> clip.__code__  # doctest: +ELLIPSIS
<code object clip at 0x...>
>>> clip.__code__.co_varnames
('text', 'max_len', 'end', 'space_before', 'space_after')
>>> clip.__code__.co_argcount
2

As you can see, this is not the most convenient arrangement of information. The argument names appear in __code__.co_varnames, but that also includes the names of the local variables created in the body of the function. Therefore, the argument names are the first N strings, where N is given by __code__.co_argcount which—by the way—does not include any variable arguments prefixed with * or **. The default values are identified only by their position in the __defaults__ tuple, so to link each with the respective argument, you have to scan from last to first. In the example, we have two arguments, text and max_len, and one default, 80, so it must belong to the last argument, max_len. This is awkward.

Fortunately, there is a better way: the inspect module.

Take a look at Example 7-17.

Example 7-17. Extracting the function signature
>>> from clip import clip
>>> from inspect import signature
>>> sig = signature(clip)
>>> sig  # doctest: +ELLIPSIS
<inspect.Signature object at 0x...>
>>> str(sig)
'(text, max_len=80)'
>>> for name, param in sig.parameters.items():
...     print(param.kind, ':', name, '=', param.default)
...
POSITIONAL_OR_KEYWORD : text = <class 'inspect._empty'>
POSITIONAL_OR_KEYWORD : max_len = 80

This is much better. inspect.signature returns an inspect.Signature object, which has a parameters attribute that lets you read an ordered mapping of names to inspect.Parameter objects. Each Parameter instance has attributes such as name, default, and kind. The special value inspect._empty denotes parameters with no default, which makes sense considering that None is a valid—and popular—default value.

The kind attribute holds one of five possible values from the _ParameterKind class:

POSITIONAL_OR_KEYWORD

A parameter that may be passed as a positional or as a keyword argument (most Python function parameters are of this kind).

VAR_POSITIONAL

A tuple of positional parameters.

VAR_KEYWORD

A dict of keyword parameters.

KEYWORD_ONLY

A keyword-only parameter (new in Python 3).

POSITIONAL_ONLY

A positional-only parameter; unsupported by function declaration syntax before Python 3.8, but exemplified by existing functions implemented in C—like divmod—that do not accept parameters passed by keyword.

Besides name, default, and kind, inspect.Parameter objects have an annotation attribute that is usually inspect._empty but may contain function signature metadata provided via the new annotations syntax in Python 3—covered in Chapter 8.

An inspect.Signature object has a bind method that takes any number of arguments and binds them to the parameters in the signature, applying the usual rules for matching actual arguments to formal parameters. This can be used by a framework to validate arguments prior to the actual function invocation. Example 7-18 shows how.

Example 7-18. Binding the function signature from the tag function in Example 7-10 to a dict of arguments
>>> import inspect
>>> sig = inspect.signature(tag)  1
>>> my_tag = {'name': 'img', 'title': 'Sunset Boulevard',
...           'src': 'sunset.jpg', 'cls': 'framed'}
>>> bound_args = sig.bind(**my_tag)  2
>>> bound_args
<inspect.BoundArguments object at 0x...>  3
>>> for name, value in bound_args.arguments.items():  4
...     print(name, '=', value)
...
name = img
cls = framed
attrs = {'title': 'Sunset Boulevard', 'src': 'sunset.jpg'}
>>> del my_tag['name']  5
>>> bound_args = sig.bind(**my_tag)  6
Traceback (most recent call last):
  ...
TypeError: 'name' parameter lacking default value
1

Get the signature from tag function in Example 7-10.

2

Pass a dict of arguments to .bind().

3

An inspect.BoundArguments object is produced.

4

Iterate over the items in bound_args.arguments, which is an OrderedDict, to display the names and values of the arguments.

5

Remove the mandatory argument name from my_tag.

6

Calling sig.bind(**my_tag) raises a TypeError complaining of the missing name parameter.

This example shows how the Python Data Model, with the help of inspect, exposes the same machinery the interpreter uses to bind arguments to formal parameters in function calls.

Frameworks and tools like IDEs can use this information to validate code. Another feature of Python 3, function annotations, enhances the possible uses of this, as we’ll see in Chapter 8.

After examining the anatomy of function objects, the remainder of this chapter covers the most useful packages in the standard library for programming in a functional style.

Packages for Functional Programming

Although Guido makes it clear that he did not design Python to be a functional programming language, a functional coding style can be used to good extent, thanks to first-class functions and the support of packages like operator and functools, which we cover in the next two sections.

The operator Module

Often in functional programming it is convenient to use an arithmetic operator as a function. For example, suppose you want to multiply a sequence of numbers to calculate factorials without using recursion. To perform summation, you can use sum, but there is no equivalent function for multiplication. You could use reduce—as we saw in “Modern Replacements for map, filter, and reduce”—but this requires a function to multiply two items of the sequence. Example 7-19 shows how to solve this using lambda.

Example 7-19. Factorial implemented with reduce and an anonymous function
from functools import reduce

def fact(n):
    return reduce(lambda a, b: a*b, range(1, n+1))

The operator module provides function equivalents for dozens of operators so you don’t have to code trivial functions like lambda a, b: a*b. With it, we can rewrite Example 7-19 as Example 7-20.

Example 7-20. Factorial implemented with reduce and operator.mul
from functools import reduce
from operator import mul

def fact(n):
    return reduce(mul, range(1, n+1))

Another group of one-trick lambdas that operator replaces are functions to pick items from sequences or read attributes from objects: itemgetter and attrgetter are factories that build custom functions to do that.

Example 7-21 shows a common use of itemgetter: sorting a list of tuples by the value of one field. In the example, the cities are printed sorted by country code (field 1). Essentially, itemgetter(1) create a function that, given a collection, returns the item at index 1. That’s easier to write and read than lambda fields: fields[1], which does the same:

Example 7-21. Demo of itemgetter to sort a list of tuples (data from Example 2-8)
>>> metro_data = [
...     ('Tokyo', 'JP', 36.933, (35.689722, 139.691667)),
...     ('Delhi NCR', 'IN', 21.935, (28.613889, 77.208889)),
...     ('Mexico City', 'MX', 20.142, (19.433333, -99.133333)),
...     ('New York-Newark', 'US', 20.104, (40.808611, -74.020386)),
...     ('Sao Paulo', 'BR', 19.649, (-23.547778, -46.635833)),
... ]
>>>
>>> from operator import itemgetter
>>> for city in sorted(metro_data, key=itemgetter(1)):
...     print(city)
...
('Sao Paulo', 'BR', 19.649, (-23.547778, -46.635833))
('Delhi NCR', 'IN', 21.935, (28.613889, 77.208889))
('Tokyo', 'JP', 36.933, (35.689722, 139.691667))
('Mexico City', 'MX', 20.142, (19.433333, -99.133333))
('New York-Newark', 'US', 20.104, (40.808611, -74.020386))

If you pass multiple index arguments to itemgetter, the function it builds will return tuples with the extracted values, which is useful for sorting on multiple keys:

>>> cc_name = itemgetter(1, 0)
>>> for city in metro_data:
...     print(cc_name(city))
...
('JP', 'Tokyo')
('IN', 'Delhi NCR')
('MX', 'Mexico City')
('US', 'New York-Newark')
('BR', 'Sao Paulo')
>>>

Because itemgetter uses the [] operator, it supports not only sequences but also mappings and any class that implements __getitem__.

A sibling of itemgetter is attrgetter, which creates functions to extract object attributes by name. If you pass attrgetter several attribute names as arguments, it also returns a tuple of values. In addition, if any argument name contains a . (dot), attrgetter navigates through nested objects to retrieve the attribute. These behaviors are shown in Example 7-22. This is not the shortest console session because we need to build a nested structure to showcase the handling of dotted attributes by attrgetter.

Example 7-22. Demo of attrgetter to process a previously defined list of namedtuple called metro_data (the same list that appears in Example 7-21)
>>> from collections import namedtuple
>>> LatLong = namedtuple('LatLong', 'lat long')  1
>>> Metropolis = namedtuple('Metropolis', 'name cc pop coord')  2
>>> metro_areas = [Metropolis(name, cc, pop, LatLong(lat, long))  3
...     for name, cc, pop, (lat, long) in metro_data]
>>> metro_areas[0]
Metropolis(name='Tokyo', cc='JP', pop=36.933, coord=LatLong(lat=35.689722,
long=139.691667))
>>> metro_areas[0].coord.lat  4
35.689722
>>> from operator import attrgetter
>>> name_lat = attrgetter('name', 'coord.lat')  5
>>>
>>> for city in sorted(metro_areas, key=attrgetter('coord.lat')):  6
...     print(name_lat(city))  7
...
('Sao Paulo', -23.547778)
('Mexico City', 19.433333)
('Delhi NCR', 28.613889)
('Tokyo', 35.689722)
('New York-Newark', 40.808611)
1

Use namedtuple to define LatLong.

2

Also define Metropolis.

3

Build metro_areas list with Metropolis instances; note the nested tuple unpacking to extract (lat, long) and use them to build the LatLong for the coord attribute of Metropolis.

4

Reach into element metro_areas[0] to get its latitude.

5

Define an attrgetter to retrieve the name and the coord.lat nested attribute.

6

Use attrgetter again to sort list of cities by latitude.

7

Use the attrgetter defined in 5 to show only city name and latitude.

Here is a partial list of functions defined in operator (names starting with _ are omitted, because they are mostly implementation details):

>>> [name for name in dir(operator) if not name.startswith('_')]
['abs', 'add', 'and_', 'attrgetter', 'concat', 'contains',
'countOf', 'delitem', 'eq', 'floordiv', 'ge', 'getitem', 'gt',
'iadd', 'iand', 'iconcat', 'ifloordiv', 'ilshift', 'imod', 'imul',
'index', 'indexOf', 'inv', 'invert', 'ior', 'ipow', 'irshift',
'is_', 'is_not', 'isub', 'itemgetter', 'itruediv', 'ixor', 'le',
'length_hint', 'lshift', 'lt', 'methodcaller', 'mod', 'mul', 'ne',
'neg', 'not_', 'or_', 'pos', 'pow', 'rshift', 'setitem', 'sub',
'truediv', 'truth', 'xor']

Most of the 52 names listed are self-evident. The group of names prefixed with i and the name of another operator—e.g., iadd, iand, etc.—correspond to the augmented assignment operators—e.g., +=, &=, etc. These change their first argument in place, if it is mutable; if not, the function works like the one without the i prefix: it simply returns the result of the operation.

Of the remaining operator functions, methodcaller is the last we will cover. It is somewhat similar to attrgetter and itemgetter in that it creates a function on the fly. The function it creates calls a method by name on the object given as argument, as shown in Example 7-23.

Example 7-23. Demo of methodcaller: second test shows the binding of extra arguments
>>> from operator import methodcaller
>>> s = 'The time has come'
>>> upcase = methodcaller('upper')
>>> upcase(s)
'THE TIME HAS COME'
>>> hiphenate = methodcaller('replace', ' ', '-')
>>> hiphenate(s)
'The-time-has-come'

The first test in Example 7-23 is there just to show methodcaller at work, but if you need to use the str.upper as a function, you can just call it on the str class and pass a string as argument, like this:

>>> str.upper(s)
'THE TIME HAS COME'

The second test in Example 7-23 shows that methodcaller can also do a partial application to freeze some arguments, like the functools.partial function does. That is our next subject.

Freezing Arguments with functools.partial

The functools module provides several higher-order functions. The best known of them is probably reduce, which was covered in “Modern Replacements for map, filter, and reduce”. Of the remaining functions in functools, the most useful is partial and its variation, partialmethod.

functools.partial is a higher-order function that allows partial application of a function. Given a function, a partial application produces a new callable with some of the arguments of the original function fixed. This is useful to adapt a function that takes one or more arguments to an API that requires a callback with fewer arguments. Example 7-24 is a trivial demonstration.

Example 7-24. Using partial to use a two-argument function where a one-argument callable is required
>>> from operator import mul
>>> from functools import partial
>>> triple = partial(mul, 3)  1
>>> triple(7)  2
21
>>> list(map(triple, range(1, 10)))  3
[3, 6, 9, 12, 15, 18, 21, 24, 27]
1

Create new triple function from mul, binding first positional argument to 3.

2

Test it.

3

Use triple with map; mul would not work with map in this example.

A more useful example involves the unicode.normalize function that we saw in “Normalizing Unicode for Reliable Comparisons”. If you work with text from many languages, you may want to apply unicode.normalize('NFC', s) to any string s before comparing or storing it. If you do that often, it’s handy to have an nfc function to do so, as in Example 7-25.

Example 7-25. Building a convenient Unicode normalizing function with partial
>>> import unicodedata, functools
>>> nfc = functools.partial(unicodedata.normalize, 'NFC')
>>> s1 = 'café'
>>> s2 = 'cafeu0301'
>>> s1, s2
('café', 'café')
>>> s1 == s2
False
>>> nfc(s1) == nfc(s2)
True

partial takes a callable as first argument, followed by an arbitrary number of positional and keyword arguments to bind.

Example 7-26 shows the use of partial with the tag function from Example 7-10, to freeze one positional argument and one keyword argument.

Example 7-26. Demo of partial applied to the function tag from Example 7-10
>>> from tagger import tag
>>> tag
<function tag at 0x10206d1e0>  1
>>> from functools import partial
>>> picture = partial(tag, 'img', cls='pic-frame')  2
>>> picture(src='wumpus.jpeg')
'<img class="pic-frame" src="wumpus.jpeg" />'  3
>>> picture
functools.partial(<function tag at 0x10206d1e0>, 'img', cls='pic-frame')  4
>>> picture.func  5
<function tag at 0x10206d1e0>
>>> picture.args
('img',)
>>> picture.keywords
{'cls': 'pic-frame'}
1

Import tag from Example 7-10 and show its ID.

2

Create picture function from tag by fixing the first positional argument with 'img' and the cls keyword argument with 'pic-frame'.

3

picture works as expected.

4

partial() returns a functools.partial object.3

5

A functools.partial object has attributes providing access to the original function and the fixed arguments.

The functools.partialmethod function (since Python 3.4) does the same job as partial, but is designed to work with methods.

An impressive functools function is lru_cache, which does memoization—an optimization that works by storing the results of function calls to avoid expensive recalculations. We will cover it in Chapter 9, where decorators are explained, along with other higher-order functions designed to be used as decorators: singledispatch and wraps.

Chapter Summary

The goal of this chapter was to explore the first-class nature of functions in Python. The main ideas are that you can assign functions to variables, pass them to other functions, store them in data structures, and access function attributes, allowing frameworks and tools to act on that information.

Higher-order functions, a staple of functional programming, are common in Python. Using map, filter, and reduce is not as common as it used to be—thanks to list comprehensions (and similar constructs like generator expressions) and the appearance of reducing built-ins like sum, all, and any. The sorted, min, max built-ins, and functools.partial are examples of commonly used higher-order functions in the language.

Callables come in nine different flavors since Python 3.6, from the simple functions created with lambda to instances of classes implementing __call__. Generators and coroutines are also callable, although their behavior is very different from other callables. All callables can be detected by the callable() built-in. Every callable supports the same rich syntax for declaring formal parameters, including keyword-only parameters and annotations—both new features introduced with Python 3.

Lastly, we covered some functions from the operator module and functools.partial, which facilitate functional programming by minimizing the need for the functionally challenged lambda syntax.

Further Reading

Don’t Make It Callable is a post from 2019 where Moshe Zadka—core developer of Python and Twisted—argues against implementing __call__ except in the very specific case of writing a DSL (Domain-Specific Language), which are specialized APIs where it is OK to break with convention to provide users with a “succint syntax dedicated to the task at hand.”

The next chapters continue our exploration of programming with function objects. Chapter 9 dives into function decorators—a special kind of higher-order function—and the closure mechanism that makes them work. Chapter 8 is devoted to type hints in function parameters and return values. Chapter 10 shows how first-class functions can simplify some classic object-oriented design patterns. The code in that chapter also show several uses of type hints.

In The Python Language Reference, “3.2. The standard type hierarchy” presents the nine callable types, along with all the other built-in types.

Chapter 7 of the Python Cookbook, Third Edition (O’Reilly), by David Beazley and Brian K. Jones, is an excellent complement to the current chapter as well as Chapter 9 of this book, covering mostly the same concepts with a different approach.

See PEP 3102 — Keyword-Only Arguments if you are interested in the rationale and use cases for that feature.

A great introduction to functional programming in Python is A. M. Kuchling’s Python Functional Programming HOWTO. The main focus of that text, however, is the use of iterators and generators, which are the subject of [Link to Come].

fn.py is a package to support functional programming in Python 2 and 3. According to its author, Alexey Kachayev, fn.py provides “implementation of missing features to enjoy FP” in Python. It includes a @recur.tco decorator that implements tail-call optimization for unlimited recursion in Python, among many other functions, data structures, and recipes. However, be aware that fn.py had its last updates in 2014, so it may be “done” or just abandoned.

The StackOverflow question “Python: Why is functools.partial necessary?” has a highly informative (and funny) reply by Alex Martelli, co-author of the classic Python in a Nutshell.

Jim Fulton’s Bobo was probably the first Web framework that deserved to be called object-oriented. If you were intrigued by it and want to learn more about its modern rewrite, start at its Introduction. A little of the early history of Bobo appears in a comment by Phillip J. Eby in a discussion at Joel Spolsky’s blog.

Reflecting on the question “Is Python a functional language?”, I created Beyond Paradigms, a talk presented at PyCaribbean, PyBay and PyConDE (slides, video).

1 “Origins of Python’s Functional Features”, from Guido’s The History of Python blog.

2 Calling a class usually creates an instance of the same class, but other behaviors are possible by overriding __new__. We’ll see an example of this in [Link to Come].

3 The source code for functools.py reveals that the functools.partial class is implemented in C and is used by default. If that is not available, a pure-Python implementation of partial is available since Python 3.4.in the functools module.

4 There is also the problem of lost indentation when pasting code to Web forums, but I digress.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
18.189.185.42