12.1 Decorators as higher-order functions

The core idea of a decorator is to transform some original function into a new function. Used like this, a decorator creates a composite function based on the decorator and the original function being decorated.

A decorator can be used in one of the two following ways:

• As a prefix that creates a new function with the same name as the base function, as follows:

  @decorator 
  def base_function() -> None: 
      pass

• As an explicit operation that returns a new function, possibly with a new name:

  def base_function() -> None: 
      pass 
   
  base_function = decorator(base_function)

These are two different syntaxes for the same operation. The prefix notation has the advantages of being tidy and succinct. The prefix location is also more visible to some readers. The suffix notation is explicit and slightly more flexible.

While the prefix notation is common, there is one reason for using the suffix notation: we may not want the resulting function to replace the original function. We may want to execute the following command, which allows us to use both the decorated and the undecorated functions:

new_function = decorator(base_function)

This will build a new function, named new_function(), from the original function. When using the @decorator syntax, the original function is no longer available for use. Indeed, once the name is reassigned to a new function object, the original object may have no remaining references, and the memory it once occupied may be eligible for reclamation.

A decorator is a function that accepts a function as an argument and returns a function as the result. This basic description is clearly a built-in feature of the language. Superficially, it may seem like we can update or adjust the internal code structure of a function.

Python doesn’t work by adjusting the internals of a function. Rather than messing about with the byte codes, Python uses a cleaner approach of defining a new function that wraps the original function. It’s easier to process the argument values or the result and leave the original function’s core processing alone.

We have two phases of higher-order functions involved in defining a decorator; they are as follows:

• At definition time, a decorator function applies a wrapper to a base function and returns the new, wrapped function. The decoration process can do some one-time-only evaluation as part of building the decorated function. Complex default values can be computed, for example.

• At evaluation time, the wrapping function can (and usually does) evaluate the base function. The wrapping function can pre-process the argument values or post-process the return value (or do both). It’s also possible that the wrapping function may avoid calling the base function. In the case of managing a cache, for example, the primary reason for wrapping is to avoid expensive calls to the base function.

Here’s an example of a decorator:

from collections.abc import Callable 
from functools import wraps 
 
def nullable(function: Callable[[float], float]) -> Callable[[float | None], float | None]: 
    @wraps(function) 
    def null_wrapper(value: float | None) -> float | None: 
        return None if value is None else function(value) 
    return null_wrapper

We almost always want to use the @wraps decorator when creating our own decorators to ensure that the decorated function retains the attributes of the original function. Copying the __name__ and __doc__ attributes, for example, ensures that the resulting decorated function has the name and docstring of the original function.

The resulting composite function, defined as the null_wrapper() function in the definition of the decorator, is also a type of higher-order function that combines the original function, the function callable object, in an expression that preserves the None values. Within the resulting null_wrapper() function, the original function callable object is not an explicit argument; it is a free variable that will get its value from the context in which the null_wrapper() function is defined.

The @nullable decorator’s return value is the newly minted function. It will be assigned to the original function’s name. It’s important that decorators only return functions and they don’t attempt to process data. Decorators use meta-programming: code that creates more code. The resulting null_wrapper() function is the function intended to process the application’s data.

As an example, we’ll assume we have a scaling function that converts input data from nautical miles to statute miles. This might be used with geolocation data that did calculations in nautical miles. The essential conversion from nautical miles, n, to statute miles, s, is a multiplication: s = 1.15078 × n.

We can apply our @nullable decorator to create a composite function as follows:

import math 
 
@nullable 
def st_miles(nm: float) -> float: 
    return 1.15078 * nm

This will create a function, st_miles(), which is a null-aware version of a small mathematical operation. The decoration process returns a version of the null_wrapper() function that invokes the original st_miles() function. This result will be named st_miles() and will have the composite behavior of both the wrapper and the original base function.

We can use this composite st_miles() function as follows:

>>> some_data = [8.7, 86.9, None, 43.4, 60] 
>>> scaled = map(st_miles, some_data) 
>>> list(scaled) 
[10.011785999999999, 100.002782, None, 49.94385199999999, 69.04679999999999]

We’ve applied the function to a collection of data values. The None value politely leads to a None result. There was no exception processing involved.

As a second example, here’s how we can create a null-aware rounding function using the same decorator:

@nullable 
def nround4(x: float) -> float: 
    return round(x, 4)

This function is a partial application of the round() function, wrapped to be null-aware. We can use this nround4() function to create a better test case for our st_miles() function as follows:

>>> some_data = [8.7, 86.9, None, 43.4, 60] 
>>> scaled = map(st_miles, some_data) 
>>> [nround4(v) for v in scaled] 
[10.0118, 100.0028, None, 49.9439, 69.0468]

This rounded result will be independent of any platform considerations. It’s very handy for doctest testing.

As an alternative implementation, we could also create these null-aware functions using the following code:

st_miles_2: Callable[[float | None], float | None] = ( 
    nullable(lambda nm: nm * 1.15078) 
) 
nround4_2: Callable[[float | None], float | None] = ( 
    nullable(lambda x: round(x, 4)) 
)

We didn’t use the @nullable decorator in front of the function definition def statement. Instead, we applied the nullable() function to another function defined as a lambda form. These expressions have the same effect as a decorator in front of a function definition.

The @nullable decorator makes an assumption that the decorated function is unary. We would need to revisit this design to create a more general-purpose null-aware decorator that works with arbitrary collections of arguments.

In Chapter 13, The PyMonad Library, we’ll look at an alternative approach to this problem of tolerating the None values. The PyMonad library defines a Maybe class of objects, which may have a proper value or may be the None value.

(’__module__’, ’__name__’, ’__qualname__’, ’__doc__’, 
’__annotations__’)

12.2 Cross-cutting concerns

One general principle behind decorators is to allow us to build a composite function from the decorator and the original function to which the decorator is applied. The idea is to have a library of common decorators that can provide implementations for common concerns.

We often call these cross-cutting concerns because they apply across several functions. These are the sorts of things that we would like to design once through a decorator and have them applied in relevant classes throughout an application or a framework.

Concerns that are often centralized as decorator definitions include the following:

• Logging

• Auditing

• Security

• Handling incomplete data

A logging decorator, for example, may write standardized messages to the application’s log file. An audit decorator may write details surrounding a database update. A security decorator may check some runtime context to be sure that the login user has the necessary permissions.

Our example of a null-aware wrapper for a function is a cross-cutting concern. In this case, we’d like to have a number of functions handle the None values by returning the None values instead of raising an exception. In applications where data is incomplete, we may need to process rows in a simple, uniform way without having to write lots of distracting if statements to handle missing values.

12.3 Composite design

The common mathematical notation for a composite function looks as follows:

The idea is that we can define a new function, f ∘g(x), that combines two other functions, f(y) and g(x).

Python’s multiple-line definition of a composition function can be done through the following code:

@f_deco 
def g(x): 
    something

The resulting function can be essentially equivalent to f ∘ g(x). The @f_deco decorator must define and return the composite function by merging an internal definition of f(y) with the provided base function, g(x).

The implementation details show that Python actually provides a slightly more complex kind of composition. The structure of a wrapper makes it helpful to think of Python decorator composition as follows:

A decorator applied to some application function, g(x), will include a wrapper function, w(y), that has two parts. One portion of the wrapper, w_α(y), applies to the arguments of the base function; the other portion, w_β(z), applies to the result of the base function.

Here’s a slightly more concrete idea, shown as a @stringify decorator definition:

def stringify(argument_function: Callable[[int, int], int]) -> Callable[[str], str]: 
    @wraps(argument_function) 
    def two_part_wrapper(text: str) -> str: 
        # The "before" part 
        arg1, arg2 = map(int, text.split(",")) 
        int_result = argument_function(arg1, arg2) 
        # The "after" part 
        return str(int_result) 
    return two_part_wrapper

This decorator inserts conversions from string to integer, and integer back to string. Concealing the details of string processing may be helpful when working with CSV files, where the content is always string data.

We can apply this decorator to a function:

>>> @stringify 
... def the_model(m: int, s: int) -> int: 
...     return m * 45 + s * 3 
... 
>>> the_model("5,6") 
’243’

This shows the two places to inject additional processing before as well as after the original function. This emphasizes an important distinction between the abstract concept of functional composition and the Python implementation: it’s possible that a decorator can create either f(g(x)), or g(f(x)), or a more complex f_βg(f_α(x)). The syntax of decoration doesn’t describe which kind of composition will be created.

The real value of decorators stems from the way any Python statement can be used in the wrapping function. A decorator can use if or for statements to transform a function into something used conditionally or iteratively. In the next section, the examples will leverage the try: statement to perform an operation with a standard recovery from bad data. There are many things that can be done within this general framework.

A great deal of functional programming follows the essential f ∘ g(x) design pattern. Defining a composite from two smaller functions can help to summarize complex processing. In other cases, it can be more informative to keep the two functions separate.

It’s easy to create composites of the common higher-order functions, such as map(), filter(), and functools.reduce(). Because these functions are relatively simple, a composite function is often easy to describe, and can help to make the code more expressive.

For example, an application may include map(f, map(g, x)). It may be more clear to create a composite function and use a map(f_g, x) expression to describe applying a composite to a collection. We can use f_g = lambda x: f(g(x)) to help explain a complex application as a composition of simpler functions. To make sure the type hints are correct, we’ll almost always want to define individual functions with the def statement.

It’s important to note that there’s no real performance advantage to either technique. The map() function is lazy: with two map() functions, one item will be taken from the source collection, x, processed by the g() function, and then processed by the f() function. With a single map() function, an item will be taken from the source collection, x, and then processed by the f_g() composite function; the memory use is the same.

In Chapter 13, The PyMonad Library, we’ll look at an alternative approach to this problem of creating composite functions from individual curried functions.

from collections.abc import Callable 
import decimal 
from typing import Any, Union, TypeVar, TypeAlias 
 
Number: TypeAlias = Union[decimal.Decimal, float] 
NumT = TypeVar("NumT", bound=Number) 
 
def bad_data( 
         function: Callable[[str], NumT] 
) -> Callable[[str], NumT]: 
    @wraps(function) 
    def wrap_bad_data(source: str, **kwargs: Any) -> NumT: 
        try: 
            return function(source, **kwargs) 
        except (ValueError, decimal.InvalidOperation): 
            cleaned = source.replace(",", "") 
            return function(cleaned, **kwargs) 
    return wrap_bad_data

from decimal import Decimal 
 
bd_int = bad_data(int) 
bd_float = bad_data(float) 
bd_decimal = bad_data(Decimal)

>>> bd_int("13") 
13 
>>> bd_int("1,371") 
1371 
>>> bd_int("1,371", base=16) 
4977

12.4 Adding a parameter to a decorator

A common requirement is to customize a decorator with additional parameters. Rather than simply creating a composite f ∘ g(x), we can do something a bit more complex. With parameterized decorators, we can create f(c) ∘ g(x). We’ve applied a parameter, c, as part of creating the wrapper, f(c). This parameterized composite function, f(c) ∘ g, can then be applied to the actual data, x.

In Python syntax, we can write it as follows:

@deco(arg) 
def func(x): 
    base function processing...

There are two steps to this. The first step applies the parameter to an abstract decorator to create a concrete decorator. Then the concrete decorator, the parameterized deco(arg) function, is applied to the base function definition to create the decorated function.

The effect is as follows:

concrete_deco = deco(arg) 
 
def func(x): 
    base function processing... 
 
func = concrete_deco(func)

The parameterized decorator worked by doing the following three things:

1. Applied the abstract decorator, deco(), to its argument, arg, to create a concrete decorator, concrete_deco().

2. Defined the base function, func().

3. Applied the concrete decorator, concrete_deco(), to the base function to create the decorated version of the function; in effect, it’s deco(arg)(func).

A decorator with arguments involves indirect construction of the final function. We seem to have moved beyond merely higher-order functions into something even more abstract: higher-order functions that create higher-order functions.

We can expand our bad-data-aware decorator to create a slightly more flexible conversion. We’ll define a @bad_char_remove decorator that can accept parameters of characters to remove. The following is a parameterized decorator:

from collections.abc import Callable 
import decimal 
from typing import Any, TypeVar 
 
T = TypeVar(’T’) 
 
def bad_char_remove( 
    *bad_chars: str 
) -> Callable[[Callable[[str], T]], Callable[[str], T]]: 
    def cr_decorator( 
            function: Callable[[str], T] 
    ) -> Callable[[str], T]: 
        def clean_list(text: str, *, to_replace: tuple[str, ...]) -> str: 
            if to_replace: 
                return clean_list( 
                    text.replace(to_replace[0], ""), 
                    to_replace=to_replace[1:] 
                ) 
            return text 
 
        @wraps(function) 
        def wrap_char_remove(text: str, **kwargs: Any) -> T: 
            try: 
                return function(text, **kwargs) 
            except (ValueError, decimal.InvalidOperation): 
                cleaned = clean_list(text, to_replace=bad_chars) 
                return function(cleaned, **kwargs) 
        return wrap_char_remove 
    return cr_decorator

A parameterized decorator has two internal function definitions:

• The concrete decorator; in this example, the cr_decorator() function. This will have the free variable, bad_chars, bound to the function being built. The concrete decorator is then returned; it will later be applied to a base function. When applied, the decorator will return a new function wrapped inside the wrap_char_remove() function. This new wrap_char_remove() function has type hints with a type variable, T, that claim the wrapped function’s type will be preserved by the new wrap_char_remove() function.

• The decorating wrapper, the wrap_char_remove() function in this example, will replace the original function with a wrapped version. Because of the @wraps decorator, the __name__ (and other attributes) of the new function will be replaced with the name of the base function being wrapped.

The overall decorator, the @bad_char_remove function in this example, has the job of binding the parameter, named bad_chars, to a function and returning the concrete decorator. The type hint clarifies the return value is a Callable object that transforms a Callable function into another Callable function. The language rules will then apply the concrete decorator to the following function definition.

The internal clean_list() function is used by the @bad_char_remove decorator to remove all characters in a given argument value. This is defined as a recursion to keep it very short. It can be optimized into an iteration if necessary. We’ve left that optimization as an exercise for the reader.

We can use the @bad_char_remove decorator to create conversion functions as follows:

from decimal import Decimal 
from typing import Any 
 
@bad_char_remove("$", ",") 
def currency(text: str, **kw: Any) -> Decimal: 
    return Decimal(text, **kw)

We’ve used our @bad_char_remove decorator to wrap a base currency() function. The essential feature of the currency() function is a reference to the decimal.Decimal constructor.

This currency() function will now handle some variant data formats:

>>> currency("13") 
Decimal(’13’) 
>>> currency("$3.14") 
Decimal(’3.14’) 
>>> currency("$1,701.00") 
Decimal(’1701.00’)

We can now process input data using a relatively simple map(currency, row) expression to convert source data from strings to usable Decimal values. The try:/except: error-handling has been isolated to a function that we’ve used to build a composite conversion function.

We can use a similar design to create null-tolerant functions. These functions would use a similar try:/except: wrapper, but would return the None values. This design variant is left as an exercise for the reader.

This decorator is limited to conversion functions that apply to a single string, and have a type hint like Callable[[str], T]. For generic decorators, it helps to follow the examples in PEP-612 and use the ParamSpec and Concatenate type hints to broaden the domain of the decorators. Because we’re interested in applying the internal clean_list() function to the first argument value, we can look at the conversion function as Callable[Concatenate[str, P], T]. We would define the first parameter as a string, and use a ParamSpec, P, to represent all other parameters of the conversion function.

12.5 Implementing more complex decorators

To create more complex compositions, Python allows the following kinds of function definitions:

@f_wrap 
@g_wrap 
def h(x): 
    return something...

Python permits stacking decorators that modify the results of other decorators. This has a meaning somewhat like f ∘g ∘h(x). However, the resulting name will be merely h(x), concealing the stack of decorations. Because of this potential confusion, we need to be cautious when creating functions that involve deeply nested decorators. If our intent is simply to handle some cross-cutting concerns, then each decorator should be designed to handle a separate concern while avoiding confusion.

While many things can be done with decoration, it’s essential to ask if using a decorator creates clear, succinct, expressive programming. When working with cross-cutting concerns, the features of the decorator are often essentially distinct from the function being decorated. This can be a wonderful simplification. Adding logging, debugging, or security checks through decoration is a widely followed practice.

12.6 Complicated design considerations

In the case of our data cleanup, the simplistic removal of stray characters may not be sufficient. When working with the geolocation data, we may have a wide variety of input formats that include simple degrees (37.549016197), degrees and minutes (37° 32.94097′), and degrees-minutes-seconds (37° 32′ 56.46′′). Of course, there can be even more subtle cleaning problems: some devices will create an output with the Unicode U+00BA character, º, the ”masculine ordinal indicator,” instead of the similar-looking degree character, °, which is U+00B0.

For this reason, it is often necessary to provide a separate cleansing function that’s bundled with the conversion function. This function will handle the more sophisticated conversions required by inputs that are as wildly inconsistent in format as latitudes and longitudes are.

How can we implement this? We have a number of choices. Simple higher-order functions are a good choice. A decorator, on the other hand, doesn’t work out terribly well. We’ll look at a decorator-based design to see some limitations to what makes sense in a decorator.

The requirements have the following two orthogonal design considerations:

• The output conversion from string to int, float or Decimal, summarized as Callable[str, T]

• The input cleaning; removing stray characters, reformatting coordinates; summarized as Callable[str, str]

Ideally, one of these aspects could be considered as the essential function that gets wrapped, and the other aspect is something that’s included via a decoration. The choice of essence versus wrapper isn’t always clear.

Considering the previous examples, it appears that this should be seen as a three-part composite:

• The output conversion from string to int, float, or decimal

• The input cleansing: either a simple replace or a more complex multiple-character replacement

• An overall processing function that first attempts the conversion, then does any cleansing as a response to an exception, and then attempts the conversion again

The third part—attempting the conversion and retrying—is the actual wrapper that also forms a part of the composite function. As we noted previously, a wrapper contains an argument phase and a return-value phase, which we can call w_α(x) and w_β(x), respectively.

We want to use this wrapper to create a composite of two additional functions. We have two choices for the design. We could include the cleansing function as an argument to the decorator on the conversion, as follows:

@cleanse_before(cleanser) 
def convert(text: str) -> int: 
    # code to convert the text, trusting it was clean 
    return # an int value

This first design claims that the conversion function is central, and the cleansing is an ancillary detail that will modify the behavior but preserve the original intent of the conversion.

Or, we could include the conversion function as an argument to the decorator for a cleansing function as follows:

@then_convert(converter) 
def cleanse(text: str) -> str: 
    # code to clean the text 
    return # the str value for later conversion

This second design claims that the cleansing is central and the conversion is an ancillary detail. This is a bit confusing because the cleansing type is generally Callable[[str], str], while the conversion’s type of Callable[[str], some other type] is what is required for the overall wrapped function.

While both of these approaches can create a usable composite function, the first version has an important advantage: the type signature of the conversion() function is also the type signature of the resulting composite function. This highlights a general design pattern for decorators: the type annotations—the signatures—of the function being decorated are the easiest to preserve.

Consequently, the @cleanse_before(cleaner) style decorator is preferred. The decorator definition looks like the following example:

from collections.abc import Callable 
from typing import Any, TypeVar 
 
# Defined Earlier: 
# T = TypeVar(’T’) 
 
def cleanse_before( 
    cleanse_function: Callable[[str], Any] 
) -> Callable[[Callable[[str], T]], Callable[[str], T]]: 
    def concrete_decorator(converter: Callable[[str], T]) -> Callable[[str], T]: 
        @wraps(converter) 
        def cc_wrapper(text: str, **kwargs: Any) -> T: 
            try: 
                return converter(text, **kwargs) 
            except (ValueError, decimal.InvalidOperation): 
                cleaned = cleanse_function(text) 
                return converter(cleaned, **kwargs) 
        return cc_wrapper 
    return concrete_decorator

We’ve defined the following multi-layer decorator:

• At the heart is the cc_wrapper() function that applies the converter() function. If this fails, then it uses the given cleanse_function() function and then tries the converter() function again.

• The cc_wrapper() function is built around the given cleanse_function() and a converter() function by the concrete_decorator() decorator. The converter() function is the function being decorated.

• The top-most layer is the concrete_decorator() function. This decorator has the cleanse_function() function as a free variable.

• The concrete decorator is created when the decorator interface, cleanse_before(), is evaluated. The interface is customized by providing the cleanse_function as an argument value.

The type hints emphasize the role of the @cleanse_before decorator. It expects some Callable function, named cleanse_function, and it creates a function, shown as Callable[[str], T], which will transform a function into a wrapped function. This is a helpful reminder of how parameterized decorators work.

We can now build a slightly more flexible cleanse and convert function, to_int(), as follows:

def drop_punct2(text: str) -> str: 
    return text.replace(",", "").replace("$", "") 
 
@cleanse_before(drop_punct2) 
def to_int(text: str, base: int = 10) -> int: 
    return int(text, base)

The integer conversion is decorated with a cleansing function. In this case, the cleansing function removes $ and , characters. The integer conversion is wrapped by this cleansing.

The to_int() function defined previously leverages the built-in int() function. An alternative definition that avoids the def statement would be the following:

to_int2 = cleanse_before(drop_punct2)(int)

This uses drop_punct2() to wrap the built-in int() conversion function. Using the mypy tool’s reveal_type() function shows that the type signature for to_int() matches the type signature for the built-in int(). It can be argued that this style is less readable than using a decorator.

We can use this enhanced integer conversion as follows:

>>> to_int("1,701") 
1701 
>>> to_int("42") 
42

The type hints for the underlying int() function have been rewritten (and simplified) for the decorated function, to_int(). This is a consequence of trying to use decorators to wrap built-in functions.

Because of the complexity of defining parameterized decorators, it appears that this is the edge of the envelope. The decorator model doesn’t seem to be ideal for this kind of design. It seems like a definition of a composite function would be more clear than the machinery required to build decorators.

The alternative is to duplicate a few lines of code that will be the same for all of the conversion functions. We could use:

def to_int_flat(text: str, base: int = 10) -> int: 
    try: 
        return int(text, base) 
    except (ValueError, decimal.InvalidOperation): 
        cleaned = drop_punct2(text) 
        return int(cleaned, base)

Each of the data type conversions will repeat the try-except block. The use of a decorator isolates this design feature in a way that can be applied to any number of conversion functions without explicitly restating the code. Later changes to the design when using this alternative may require editing a number of similar functions instead of changing one decorator.

Generally, decorators work well when we have a number of relatively simple and fixed aspects that we want to include with a given function (or a class). Decorators are also important when these additional aspects can be looked at as infrastructure or as support, and not something essential to the meaning of the application code.

For something that involves multiple orthogonal design aspects, we may want to result to a callable class definition with various kinds of plugin strategy objects. This might have a simpler class definition than the equivalent decorator. Another alternative to decorators is to look closely at creating higher-order functions. In some cases, partial functions with various combinations of parameters may be simpler than a decorator.

The typical examples for cross-cutting concerns include logging or security testing. These features can be considered as the kind of background processing that isn’t specific to the problem domain. When we have processing that is as ubiquitous as the air that surrounds us, then a decorator might be an appropriate design technique.

12.7 Summary

In this chapter, we’ve looked at two kinds of decorators: simple decorators with no arguments and parameterized decorators. We’ve seen how decorators involve an indirect composition between functions: the decorator wraps a function (defined inside the decorator) around another function.

Using the functools.wraps() decorator ensures that our decorators will properly copy attributes from the function being wrapped. This should be a piece of every decorator we write.

In the next chapter, we’ll look at the PyMonad library to express a functional programming concept directly in Python. We don’t require monads generally because Python is an imperative programming language under the hood.

12.8 Exercises

This chapter’s exercises are based on code available from Packt Publishing on GitHub. See https://github.com/PacktPublishing/Functional-Python-Programming-3rd-Edition.

In some cases, the reader will notice that the code provided on GitHub includes partial solutions to some of the exercises. These serve as hints, allowing the reader to explore alternative solutions.

In many cases, exercises will need unit test cases to confirm they actually solve the problem. These are often identical to the unit test cases already provided in the GitHub repository. The reader should replace the book’s example function name with their own solution to confirm that it works.

    def clean_list(text: str, *, to_replace: tuple[str, ...]) -> str: 
        ...

@none_tolerant 
def x2(x: float) -> float: 
    return 2 * x 
 
def test_x2() -> None: 
    assert x2(42.) == 84.0 
    assert x2(None) == None 
    assert list(map(x2, [1, 2, None, 3])) == [2, 3, None, 6]

from dataclasses import dataclass 
@dataclass(frozen=True) 
class Sample: 
    id: int 
    observation: float 
    date_time: datetime.datetime