Starting with Mypy

To begin type checking, I run the mypy command on the messages.py module:

…/no_hints/ $ pip install mypy
[lots of messages omitted...]
…/no_hints/ $ mypy messages.py
Success: no issues found in 1 source file

Mypy finds no problem with the code in Example 8-2 when run with its default settings.⁴

If a function signature has no annotations, Mypy ignores it by default. That’s the spirit of gradual typing.

For this example, I also have pytest unit tests. This is the code in messages_test.py.

from pytest import mark

from messages import show_count

@mark.parametrize('qty, expected', [
    (1, '1 part'),
    (2, '2 parts'),
])
def test_show_count(qty, expected):
    got = show_count(qty, 'part')
    assert got == expected

def test_show_count_zero():
    got = show_count(0, 'part')
    assert got == 'no part'

Let’s check messages_test.py:

$ mypy messages_test.py
messages_test.py:3: error: Skipping analyzing 'pytest':
 found module but no type hints or library stubs
messages_test.py:3: note: See
 https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports
Found 1 error in 1 file (checked 1 source file)

The problem is that messages_test imports pytest, which doesn’t have type hints as I write this.⁵ I add this comment to the import line to make Mypy ignore pytest:

from pytest import mark  # type: ignore

Now Mypy doesn’t report any issues with messages_test.py.

Making Mypy More Strict

The command-line option --disallow-untyped-defs makes Mypy flag any function definition that does not have type hints for all its parameters and for its return value.

Using --disallow-untyped-defs on the test file produces three errors and a note:

…/no_hints/ $ mypy --disallow-untyped-defs messages_test.py
messages.py:14: error: Function is missing a type annotation
messages_test.py:10: error: Function is missing a type annotation
messages_test.py:15: error: Function is missing a return type annotation
messages_test.py:15: note: Use "-> None" if function does not return a value
Found 3 errors in 2 files (checked 1 source file)

For the first steps with gradual typing, I prefer to use another option: --disallow-incomplete-defs. Initially, it tells me nothing:

…/no_hints/ $ mypy --disallow-incomplete-defs messages_test.py
Success: no issues found in 1 source file

But now, I may add just the return type to show_count in messages.py:

def show_count(count, word) -> str:

This is enough to make Mypy look at it. Using the same command line as before to check messages_test.py, will lead Mypy to look at messages.py again:

…/no_hints/ $ mypy --disallow-incomplete-defs messages_test.py
messages.py:14: error: Function is missing a type annotation for one or more arguments
Found 1 error in 1 file (checked 1 source file)

Now I can gradually add type hints function by function, without getting warnings about functions that I haven’t annotated. This is a fully annotated signature that satisfies Mypy:

def show_count(count: int, word: str) -> str:

[mypy]
python_version = 3.8
warn_unused_configs = True
disallow_incomplete_defs = True
[mypy-pytest]
ignore_missing_imports = True

A Default Parameter Value

The show_count function—first shown in Example 8-2—has an obvious limitation: it only works with regular nouns. If the plural can’t be spelled by appending an 's', we should let the user provide the plural form, like this:

>>> show_count(3, 'mouse', 'mice')
'3 mice'

Let’s do a little “type driven development”. First we add a test that uses that third argument. Don’t forget the return type hint -> None to the test function otherwise Mypy will not check it.

def test_irregular() -> None:
    got = show_count(2, 'child', 'children')
    assert got == '2 children'

Mypy detects the error:

…/hints_2/ $ mypy messages_test.py
messages_test.py:22: error: Too many arguments for "show_count"
Found 1 error in 1 file (checked 1 source file)

Now I edit show_count, adding the optional plural parameter:

def show_count(count: int, singular: str, plural: str = '') -> str:
    if count == 0:
        return f'no {singular}'
    elif count == 1:
        return f'1 {singular}'
    else:
        if plural:
            return f'{count} {plural}'
        else:
            return f'{count} {singular}s'

The following details are not mandatory, but are considered good style for type hints:

• There should be no space between the parameter name and the :, and one space after the :.

• There should be one space on each side of the = that precedes the default parameter value.

def munge(input: AnyStr, sep: AnyStr = None, limit=1000):

Using None as a default

In Example 8-3 the parameter plural is anotated as str, and the default value is '', so there is no type conflict.

I like that solution, but in other contexts None is a better default. If the optional parameter expects a mutable type, then None is the only sensible default—as we saw in “Mutable Types as Parameter Defaults: Bad Idea”.

To have None as the default for the plural parameter, here is how the signature would look like:

from typing import Optional

def show_count(count: int, singular: str, plural: Optional[str] = None) -> str:

Let’s unpack that:

• Optional[str] means plural may be a str or None.

• You must explicitly provide the default value = None.

If you don’t assign a default value to plural, the Python runtime will treat it as a required parameter. Remember: at runtime, type hints are ignored.

Note that we need to import Optional from the typing module. When importing types, it’s good practice to use the syntax from typing import X, to reduce the length of the function signatures.

Type hints for Python 2.7 and 3.x

Companies with large Python 2 codebases have learned that type hints are very helpful when migrating to Python 3. It’s possible to annotate code that will run on Python 2.7 and Python 3.x using special comments described in PEP 484.

This is how the final version of the show_count signature looks like using syntax that works in Python 2.7 and 3.x—also supported by Mypy and other type checkers:

from typing import Optional

def show_count(count, singular, plural=None):
    # type: (int, str, Optional[str]) -> str

Note that only the parameter types appear in the comment.

If the parameter list is too long, the signature may be annotated like this:

from typing import Optional

def show_count(count,       # type: int
               singular,    # type: str
               plural=None  # type: Optional[str]
               ):
    # type: (...) -> str

The last type comment would be exactly as shown: the ... replaces the parameter types already given, and the -> str defines the return type.

For more details, see Suggested syntax for Python 2.7 and straddling code in PEP 484.

Now that we’ve had a first practical view of gradual typing, let’s consider what the concept of type means in practice.

The Any type

The keystone of any gradual type system is the Any type, also known as the dynamic type. When a type checker sees an untyped function like this:

def double(n):
    return n * 2

It assumes this:

def double(n: Any) -> Any:
    return n * 2

That means the n argument and the return value can be of any type, including different types. Any is assumed to support every possible operation.

Contrast Any with object. Consider this signature:

def double(n: object) -> object:

This function also accepts arguments of every type, because every type is a subtype of object.

However, a type checker will reject this function:

def double(n: object) -> object:
    return n * 2

The problem is that object does not support the __mul__ operation. This is what Mypy reports:

…/birds/ $ mypy double_object.py
double_object.py:2: error: Unsupported operand types for * ("object" and "int")
Found 1 error in 1 file (checked 1 source file)

More general types have narrower interfaces, i.e. they support less operations. The object class implements fewer operations than abc.Sequence, which implements fewer operations than abc.MutableSequence, which implements fewer operations than list.

But Any is a magic type that sits at the top and the bottom of the type hierarchy. It’s simultaneously the most general type—so that an argument n: Any accepts values of every type—and the most specialized type, supporting every possible operation. At least, that’s how the type checker understands Any.

Of course, no type can support every possible operation, so using Any prevents the type checker from fulfilling its core mission: detecting potentially illegal operations before your program crashes with a runtime exception.

class C1:
    ...

class C2(C1):
    ...

def f1(p: C1) -> None:
    ...

o2 = C2()

f1(o2)  # OK

def f2(p: C2) -> None:
    ...

o1 = C1()

f2(o1)  # type error

def f3(p: Any) -> None
    ...

o0 = object()
o1 = C1()
o2 = C2()

f3(o0)  #
f3(o1)  #  all OK: rule #2
f3(o2)  #

def f4():  # implicit return type: `Any`
    ...

o4 = f4()  # inferred type: `Any`

f1(o4)  #
f2(o4)  #  all OK: rule #3
f3(o4)  #

Simple types and classes

Simple types like int, float, str, bytes may be used directly in type hints. Concrete classes from the standard library, external packages, or user defined— FrenchDeck, Vector2d, and Duck—may also be used in type hints.

Abstract Base Classes are also useful in type hints. We’ll get back to them as we study collection types, and in “Abstract Base Classes”.

Among classes, is-consistent-with is defined like is-subtype-of: a subclass is consistent with all its superclasses.

However, “practicality beats purity” so there is an important exception:

Optional and Union types

We saw the Optional special type in “Using None as a default”. It solves the problem of having None as a default, as in this example from that section:

from typing import Optional

def show_count(count: int, singular: str, plural: Optional[str] = None) -> str:

The construct Optional[str] is actually a shortcut for Union[str, None] which means the type of plural may be str or None.

The ord built-in function’s signature is a simple example of Union—it accepts str or bytes, and returns an int:¹⁰

def ord(c: Union[str, bytes]) -> int: ...

Here is an example of a function takes a str, but may return a str or a float:

from typing import Union

def parse_token(token: str) -> Union[str, float]:
    try:
        return float(token)
    except ValueError:
        return token

If possible, avoid creating functions that return Union types, as they put an extra burden on the user—forcing them to check the type of the returned value at runtime to know what to do with it. But the parse_token above is a reasonable use case in the context of a simple expression evaluator.

Union[] requires at least two types. Nested Union types have the same effect as a flattened Union. So this type hint:

Union[A, B, Union[C, D, E]]

is the same as:

Union[A, B, C, D, E]

Union is more useful with types that are not consistent with each other. For example: Union[int, float] is rarely useful as a type hint, because int is-consistent-with float. Usually it’s better to use float to annotate the parameter, then it will accept int values as well.

Generic collections

Most Python collections are heterogeneous: for example, you can put any mixture of different types in a list. However, in practice that is not very useful: if you put objects in a list, you are likely to want to operate on them later, and usually this means they share at least one common method.¹¹

With Python’s type hints, a collection can be annotated with a generic type to constrain the type of the elements in the collection. For example:

def tokenize(text: str) -> list[str]:
    return text.upper().split()

In Python 3.9, that means tokenize returns a list where all items are str.

However, list and the other built-in collections only support that notation in Python 3.7 and 3.8 with a __future__ import:

from __future__ import annotations

def tokenize(text: str) -> list[str]:
    return text.upper().split()

Sadly, that __future__ does not work with Python 3.5 or 3.6, nor is it supported by Mypy 0.770—which I am using as I write this.

So this is how to annotate tokenize in a way that works with Python ≥ 3.5 and Mypy in May 2020:

from typing import List

def tokenize(text: str) -> List[str]:
    return text.upper().split()

Note that you need to import the List type from the typing module.

To annotate a list that can hold any type of object, the type hint would be List[Any]. That’s the same as writing just List, or even list.

Besides List, the typing module defines dozens of types that are derived from existing standard library classes with the added feature of supporting generic type notation with []. Table 8-1 lists the collections that are not mappings.

Python 3.9 implements PEP 585—Type Hinting Generics In Standard Collections, which means the collection types in the left column of Table 8-1 support the generic [], so you can write set[str] and don’t need to import typing.Set.

As of May 2020, there is no good way to annotate array.array taking into account the typecode constructor argument which determines whether integers or floats are stored in the array.¹²

Tuple

There are three ways to annotate tuple types.

The first is using typing.Tuple and specifying the type of each field in the tuple. For example, to accept a tuple with city name, population and country—('Shanghai', 24.28, 'China')—the type hint would be Tuple[str, float, str].

Consider a function that takes a pair of geographic coordinates and returns a Geohash, used like this:

>>> shanghai = 31.2304, 121.4737
>>> geohash(shanghai)
'wtw3sjq6q'

This is how geohash is defined, using the geolib package from PyPI:

from typing import Tuple

from geolib import geohash as gh  # type: ignore

PRECISION = 9

def geohash(lat_lon = Tuple[float, float]) -> str:
    return gh.encode(*lat_lon, PRECISION)

The second way is using typing.NamedTuple—as seen in Chapter 5. Here is a variation of Example 8-7 with NamedTuple:

from typing import Tuple, NamedTuple

from geolib import geohash as gh  # type: ignore

PRECISION = 9

class Coordinate(NamedTuple):
    lat: float
    lon: float

def geohash(lat_lon: Coordinate) -> str:
    return gh.encode(*lat_lon, PRECISION)

def display(lat_lon: Tuple[float, float]) -> str:
    lat, lon = lat_lon
    ns = 'N' if lat >= 0 else 'S'
    ew = 'E' if lon >= 0 else 'W'
    return f'{abs(lat):0.1f}°{ns}, {abs(lon):0.1f}°{ew}'

As explained in “Overview of data class builders”, typing.NamedTuple is a factory for tuple subclasses, so Coordinate is-consistent-with Tuple[float, float] but the reverse is not true—after all, Coordinate has extra methods added by NamedTuple, like ._as_dict(), and could also have user-defined methods.

The third way is for annotating tuples of unspecified length that are used as immutable lists: you must specify a single type, followed by a comma and ... (that’s Python’s ellipsis token, made of three periods, not Unicode U+2026—HORIZONTAL ELLIPSIS).

For example, the type for a tuple with int elements is Tuple[int, ...].

The ellipsis indicates that any number of elements >= 1 is acceptable. There is no way to specify fields of different types for tuples of unspecified length.

Here is a columnize function that transforms a sequence into a table of rows and cells in the form of list of tuples with unspecified lengths. This is useful to display items in columns, like this:

>>> animals = 'drake fawn heron ibex koala lynx tahr xerus yak zapus'.split()
>>> table = columnize(animals)
>>> table
[('drake', 'koala', 'yak'), ('fawn', 'lynx', 'zapus'), ('heron', 'tahr'),
 ('ibex', 'xerus')]
>>> for row in table:
...     print(''.join(f'{word:10}' for word in row))
...
drake     koala     yak
fawn      lynx      zapus
heron     tahr
ibex      xerus

Example 8-9 shows the implementation of columnize. Note the return type, List[Tuple[str, ...]].

from typing import Sequence, List, Tuple

def columnize(sequence: Sequence[str], num_columns: int = 0) -> List[Tuple[str, ...]]:
    if num_columns == 0:
        num_columns = round(len(sequence) ** .5)
    num_rows, reminder = divmod(len(sequence), num_columns)
    num_rows += bool(reminder)
    return [tuple(sequence[i::num_rows]) for i in range(num_rows)]

The annotations Tuple[Any, ...], Tuple, and tuple mean the same thing.

In codebases supporting only Python 3.9 or later, the recommended signature for Example 8-9 is:

from collections.abc import Sequence

def columnize(sequence: abc.Sequence[str],
              num_columns: int = 0) -> list[tuple[str, ...]]:

Note there is no typing import. The list and tuple built-ins are used, as well as collections.abc.Sequence instead of typing.Sequence.

Generic mappings

Generic mapping types are annotated as MappingType[KeyType, ValueType].

For example, a JSON object must have string keys, but the values can be anything, so this would be written as Dict[str, Any].

Example 8-10 shows a practical use of a function returning an inverted index to search Unicode characters by name—a variation of Example 4-21 more suitable for server-side code that we’ll study in [Link to Come].

Given starting and ending Unicode character codes, name_index returns a Dict[str, Set[str]] which is an inverted index mapping each word to a set of characters that have that word in their names. For example, after indexing ASCII characters from 32 to 64, here are the sets of characters mapped to the words 'SIGN' and 'DIGIT', and how to find the character named 'DIGIT EIGHT':

>>> index = name_index(32, 65)
>>> index['SIGN']
{'$', '>', '=', '+', '<', '%', '#'}
>>> index['DIGIT']
{'8', '5', '6', '2', '3', '0', '1', '4', '7', '9'}
>>> index['DIGIT'] & index['EIGHT']
{'8'}

Below is the source code for charindex.py with the name_index function. Besides a Dict[] type hint, this example has three features appearing for the first time in the book.

import sys
import re
import unicodedata
from typing import Dict, Set, Iterator

RE_WORD = re.compile(r'w+')
STOP_CODE = sys.maxunicode + 1

def tokenize(text: str) -> Iterator[str]:  
    """return iterable of uppercased words"""
    for match in RE_WORD.finditer(text):
        yield match.group().upper()

def name_index(start: int = 32, end: int = STOP_CODE) -> Dict[str, Set[str]]:
    index: Dict[str, Set[str]] = {}  
    for char in (chr(i) for i in range(start, end)):
        if name := unicodedata.name(char, ''):  
            for word in tokenize(name):
                index.setdefault(word, set()).add(char)
    return index

tokenize is a generator function. [Link to Come] is about generators.

The local variable index is annotated. Without the hint, Mypy says: error: Need type annotation for 'index' (hint: "index: Dict[<type>, <type>] = ..."). We’ll come back to variable type hints in [Link to Come].

I used the walrus operator := in the if condition. It assigns the result of the unicodedata.name() call to name, and the whole expression evaluates to that result. When the result is '', that’s falsy and the index is not updated.¹³

The typing module also defines several mapping types and view types, listed in Table 8-2.

To instantiate an annotated defaultdict, a variable type hint is required for Python 3.8 and earlier—because typing.DefaultDict cannot be called to construct an object, and collections.defaultdict does not accept the generic syntax. Here is an example:

my_dict: DefaultDict[str, List[int]] = defaultdict(list)

TypedDict

The mapping types we just saw limit all values to have the same type.

However, Python dictionaries are often used as records, with field names as keys. For example, consider a record describing a book in JSON or Python:

{"isbn": "0134757599",
 "title": "Refactoring, 2e",
 "authors": ["Martin Fowler", "Kent Beck"],
 "pagecount": 478}

Before Python 3.8, there was no good way to annotate a record like that. Here are two possibilities, both unsatisfactory:

Dict[str, Any]

The values may be of any type.

Dict[str, Union[str, int, List[str]]]

Hard to read, and doesn’t preserve the relationship between field names and field types: title is supposed to be a str, it can’t be an int or a List[str].

Python 3.8 solved that problem by implementing PEP 589—TypedDict: Type Hints for Dictionaries with a Fixed Set of Keys.

Here is a simple TypedDict:

from typing import TypedDict, List
import json

class BookDict(TypedDict):
    isbn: str
    title: str
    authors: List[str]
    pagecount: int

At first glance, typing.TypedDict may seem like a data class builder, similar to the @dataclass decorator or typing.NamedTuple—both covered in Chapter 5.

The syntactic similarity is misleading. TypeDict is very different. It exists only for the benefit of type checkers, and has no runtime effect.

TypedDict provides two things:

1. Class-like syntax to annotate a dict with type hints for the value of each “field”.

2. A constructor that tells the type checker to expect a dict with the keys and values as specified.

At runtime, a TypeDict constructor such as BookDict is placebo: it has the same effect as calling the dict constructor with the same arguments.

The fact that BookDict creates a plain dict also means that:

• The “fields” in the pseudo-class definition don’t create instance attributes.

• You can’t write initializers with default values for the “fields”.

• Method definitions are not allowed.

Let’s explore the behavior of a BookDict at runtime.

>>> from books import BookDict
>>> pp = BookDict(title='Programming Pearls',  
...               authors='Jon Bentley',  
...               isbn='0201657880',
...               pagecount=256)
>>> pp  
{'title': 'Programming Pearls', 'authors': 'Jon Bentley', 'isbn': '0201657880',
 'pagecount': 256}
>>> type(pp)
<class 'dict'>
>>> pp.title  
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'dict' object has no attribute 'title'
>>> pp['title']
'Programming Pearls'
>>> BookDict.__annotations__  
{'isbn': <class 'str'>, 'title': <class 'str'>, 'authors': typing.List[str],
 'pagecount': <class 'int'>}

You can call BookDict like a dict constructor with keyword arguments, or passing a dict argument—including a dict literal.

Ooops… I forgot authors takes a list. But gradual typing means no type checking at runtime.

The result of calling BookDict is a plain dict…

… therefore you can’t read the data using object.field notation.

To get the type hints at runtime, read BookDict.__annotations__.

Without a type checker, TypedDict is as useful as comments: it may help people read the code, but that’s it. In contrast, the class builders from Chapter 5 are useful even if you don’t use a type checker because at runtime they generate or enhance a custom class that you can instantiate. They also provide several useful methods or functions listed in Table 5-1.

Example 8-13 builds a valid BookDict and tries some operations on it. This shows how TypedDict enables Mypy to catch errors, shown in Example 8-14.

from books import BookDict
from typing import TYPE_CHECKING

def demo() -> None:  
    book = BookDict(  
        isbn='0134757599',
        title='Refactoring, 2e',
        authors=['Martin Fowler', 'Kent Beck'],
        pagecount=478
    )
    authors = book['authors'] 
    if TYPE_CHECKING:  
        reveal_type(authors)  
    authors = 'Bob'  
    book['weight'] = 4.2
    del book['title']


if __name__ == '__main__':
    demo()

Remember to add a return type, so that Mypy doesn’t ignore the function.

This is a valid BookDict: all the keys are present, with values of the correct types.

Mypy will infer the type of authors from the annotation for the 'authors' key in BookDict.

typing.TYPE_CHECKING is only True when the program is being type checked. At runtime, it’s always false.

The previous if statement prevents reveal_type(authors) from being called at runtime. reveal_type is not a runtime Python function, but a debugging facility provided by Mypy. That’s why there is no import for it. See its output in Example 8-14.

The last three lines of the demo function are illegal. They will cause error messages in in Example 8-14.

Type checking demo_books.py from Example 8-13, this is what we get:

…/typedict/ $ mypy demo_books.py
demo_books.py:13: note: Revealed type is 'built-ins.list[built-ins.str]'  
demo_books.py:14: error: Incompatible types in assignment
                  (expression has type "str", variable has type "List[str]")  
demo_books.py:15: error: TypedDict "BookDict" has no key 'weight'  
demo_books.py:16: error: Key 'title' of TypedDict "BookDict" cannot be deleted  
Found 3 errors in 1 file (checked 1 source file)

This note is the result of reveal_type(authors).

The type of the authors variable was inferred from the type of the book['authors'] expression that initialized it. You can’t assign a str to a variable of type List[str]. Type checkers usually don’t allow the type of a variable to change.¹⁴

Cannot assign to a key that is not part of the BookDict definition.

Cannot delete a key that is part of the BookDict definition.

Now let’s see BookDict used in function signatures, to type check function calls.

Imagine you need to generate XML from book records, similar to this:

<BOOK>
  <ISBN>0134757599</ISBN>
  <TITLE>Refactoring, 2e</TITLE>
  <AUTHOR>Martin Fowler</AUTHOR>
  <AUTHOR>Kent Beck</AUTHOR>
  <PAGECOUNT>478</PAGECOUNT>
</BOOK>

If you were writing MicroPython code to embed in a tiny microcontroller, you might write a function like this:¹⁵

AUTHOR_EL = '<AUTHOR>{}</AUTHOR>'

def to_xml(book: BookDict) -> str:  
    elements: List[str] = []  
    for key, value in book.items():
        if isinstance(value, list):  
            elements.extend(
                AUTHOR_EL.format(n) for n in value)  
        else:
            tag = key.upper()
            elements.append(f'<{tag}>{value}</{tag}>')
    xml = '
	'.join(elements)
    return f'<BOOK>
	{xml}
</BOOK>'

The whole point of the example: using BookDict in the function signature.

It’s often necessary to annotate collections that start empty, otherwise Mypy can’t infer the type of the elements.¹⁶

Mypy understands isinstance checks, and treats value as a list in this block.

When I used key == 'authors' as the condition for the if guarding this block, Mypy found an error in this line: "object" has no attribute "__iter__", because it inferred the type of value returned from book.items() as object, which doesn’t support the __iter__ method required by the generator expression. With the isinstance check, this works because Mypy knows that value is a list in this block.

Here is a function that parses a JSON str and returns a BookDict:

def from_json(data: str) -> BookDict:
    whatever = json.loads(data)  
    return whatever  

The return type of json.loads() is Any.¹⁷

I can return whatever—of type Any—because Any is consistent with every type, including the declared return type, Bookdict.

The second point of Example 8-16 is very important to keep in mind: Mypy will not flag any problem in this code, but at runtime the value in whatever may not conform to the BookDict structure—in fact, it may not be a dict at all!

If you run Mypy with --disallow-any-expr it will complain about the two lines in the body of from_json:

…/typedict/ $ mypy books_any.py --disallow-any-expr
books.py:30: error: Expression has type "Any"
books.py:31: error: Expression has type "Any"
Found 2 errors in 1 file (checked 1 source file)

In this case, the type error can be silenced by adding a type hint to the initialization of the whatever variable, as in Example 8-17:

def from_json(data: str) -> BookDict:
    whatever: BookDict = json.loads(data)  
    return whatever  

--disallow-any-expr does not cause errors when an expression of type Any is immediately assigned to a variable with a type hint.

Now whatever is of type BookDict, the declared return type.

Static type checking is unable to prevent errors with code that is inherently dynamic, such as json.loads(), which builds a Python objects of different types at runtime. Example 8-18, Example 8-19, and Example 8-20 demonstrate.

from books import to_xml, from_json
from typing import TYPE_CHECKING

def demo() -> None:
    NOT_BOOK_JSON = """
        {"title": "Andromeda Strain",
         "flavor": "pistachio",
         "authors": true}
    """
    not_book = from_json(NOT_BOOK_JSON)  
    if TYPE_CHECKING:  
        reveal_type(not_book)
        reveal_type(not_book['authors'])

    print(not_book)  
    print(not_book['flavor'])  

    xml = to_xml(not_book)  
    print(xml)  


if __name__ == '__main__':
    demo()

This line does not produce a valid BookDict—see the content of NOT_BOOK_JSON.

Let’s have Mypy reveal a couple of types.

This should not be a problem, as print can handle object and every subtype.

BookDict has no 'flavor' key, but the JSON source does… what will happen?

Remember the signature: def to_xml(book: BookDict) -> str:

How will the XML output look like?

Checking demo_not_book.py with Mypy:

…/typedict/ $ mypy demo_not_book.py
demo_not_book.py:12: note: Revealed type is
   'TypedDict('books.BookDict', {'isbn': built-ins.str,
                                 'title': built-ins.str,
                                 'authors': built-ins.list[built-ins.str],
                                 'pagecount': built-ins.int})'  
demo_not_book.py:13: note: Revealed type is 'built-ins.list[built-ins.str]'  
demo_not_book.py:16: error: TypedDict "BookDict" has no key 'flavor'  
Found 1 error in 1 file (checked 1 source file)

The revealed type is the nominal type, not runtime content of not_book.

Again, this is the nominal type of not_book['authors'], as defined in BookDict. Not the runtime type.

This error is for line print(not_book['flavor']): that key does not exist in the nominal type.

Now let’s run demo_not_book.py.

…/typedict/ $ python3 demo_not_book.py
{'title': 'Andromeda Strain', 'flavor': 'pistachio', 'authors': True}  
pistachio  
<BOOK>  
        <TITLE>Andromeda Strain</TITLE>
        <FLAVOR>pistachio</FLAVOR>
        <AUTHORS>True</AUTHORS>
</BOOK>

This is not really a BookDict.

The value of not_book['flavor'].

to_xml takes a BookDict argument, but at runtime it’s more flexible: garbage in, garbage out.

Example 8-20 shows that demo_not_book.py outputs nonsense, but has no runtime errors. Using a TypeDict while handling JSON data did not provide much type safety.

If you look at the code for to_xml in Example 8-15 through the lens of duck typing, the argument book must provide an .items() method that returns an iterable of tuples like (key, value) where:

• key must have an .upper() method;

• value can be anything.

The point of this demonstration: when handling data with a dynamic structure, such as JSON or XML, TypeDict is absolutely not a replacement for data validation at runtime.

TypedDict has more features, including support for optional keys, a limited form of inheritance, and an alternative declaration syntax for Python versions before 3.6. These will be covered in [Link to Come], section [Link to Come].

Abstract Base Classes

Be conservative in what you send, be liberal in what you accept.

Postel’s law, a.k.a. the Robustness Principle

Table 8-1 and Table 8-2 list several abstract classes from collections.abc. Ideally, a function should accept arguments of those abstract types—or their type hint equivalents before Python 3.9—and not concrete types. This gives more flexibility to the caller.

Consider this function signature:

def name2hex(name: str, color_map: Mapping[str, int]) -> str:

Using typing.Mapping allows the caller to provide an instance of dict, defaultdict, ChainMap, a UserDict subclass or any other type that is a subtype of Mapping.

In contrast, consider this signature:

def name2hex(name: str, color_map: Dict[str, int]) -> str:

Now color_map must be a dict or one of its subtypes such as DefaultDict or OrderedDict. In particular, a subclass of collections.UserDict would not pass the type check for color_map, despite being the recommended way to create user-defined mappings, as we saw in “Subclassing UserDict”. Mypy would reject a UserDict or an instance of class derived from it, because UserDict is not a subclass of dict; they are siblings, both are subclasses of abc.MutableMapping.¹⁸

Therefore, in general it’s better to use typing.Mapping or typing.MutableMapping instead of dict or typing.Dict as a parameter type. If the name2hex function doesn’t need to mutate the given color_map, the most accurate type hint for color_map is typing.Mapping. That way, the caller doesn’t need to provide an object that implements methods like setdefault, pop and update which are part of the MutableMapping interface, but not of Mapping. This has to do with the second part of Postel’s law: “be liberal in what you accept”.

Postel’s law also tells us to be conservative in what we send. The return value of a function is always a concrete object, so the return type hint should be a concrete type, as in the example from “Generic collections”—which uses list[str] assuming the code will run on Python 3.9—otherwise the typing equivalent List[str] should be used.

def tokenize(text: str) -> list[str]:
    return text.upper().split()

Under the entry of typing.List, the Python documentation says:

Generic version of list. Useful for annotating return types. To annotate arguments it is preferred to use an abstract collection type such as Sequence or Iterable.

A similar comment appears in the entries for typing.Dict and typing.Set.

Remember that most ABCs from collections.abc and other concrete classes from collections, as well as built-in collections, support generic type hint notation like collections.deque[str] starting with Python 3.9. The corresponding typing collections will only be needed to support code written in Python 3.8 or earlier. The full list of classes that became generic appears in section Implementation of PEP 585—Type Hinting Generics In Standard Collections.

To wrap up our discussion of ABCs in type hints, we need to talk about Numbers.

Iterable

The typing.List documentation I just quoted recommends Sequence and Iterable for function parameter type hints.

One example of Iterable argument appears the math.fsum function from the standard library:

def fsum(__seq: Iterable[float]) -> float:

Example 8-21 is another example using an Iterable parameter that produces items that are Tuple[str, str]. Here is how the function is used:

>>> l33t = [('a', '4'), ('e', '3'), ('i', '1'), ('o', '0')]
>>> text = 'mad skilled noob powned leet'
>>> from replacer import zip_replace
>>> zip_replace(text, l33t)
'm4d sk1ll3d n00b p0wn3d l33t'

And here is how it’s implemented:

from typing import Iterable, Tuple

FromTo = Tuple[str, str]  

def zip_replace(text: str, changes: Iterable[FromTo]) -> str:  
    for from_, to in changes:
        text = text.replace(from_, to)
    return text

FromTo is a type alias: I assigned Tuple[str, str] to FromTo, to make the signature of zip_replace more readable.

changes needs to be an Iterable[FromTo]; that’s the same as Iterable[Tuple[str, str]], but shorter and easier to read.

from typing import TypeAlias, Tuple

FromTo: TypeAlias = Tuple[str, str]

Parameterized generics and TypeVar

A parameterized generic is a generic type, written as List[T] where T is a type variable that will be bound to a specific type with each usage. This allows a parameter type to be reflected on the result type.

Example 8-22 defines sample, a function that takes two arguments: a Sequence of elements of type T, and an int. It returns a List of elements of the same type T, picked at random from the first argument.

Here are two examples illustrate the behavior of sample:

1. If called with a tuple of type Tuple[int, ...]—which is-consistent-with Sequence[int]—then the type parameter is int, so the return type is List[int];

2. If called with a str—which is-consistent-with Sequence[str]—then the type parameter is str, so the return type is List[str].

This is the implementation:

from random import shuffle
from typing import Sequence, List, TypeVar

T = TypeVar('T')

def sample(population: Sequence[T], size: int) -> List[T]:
    if size < 1:
        raise ValueError('size must be >= 1')
    result = list(population)
    shuffle(result)
    return result[:size]

Another example is the statistics.mode function from the standard library, which returns the most common data point from a series.

Here is one usage example from the documentation:

>>> mode([1, 1, 2, 3, 3, 3, 3, 4])
3

Without using a TypeVar, mode could have this signature:

from collections import Counter
from typing import Iterable

def mode(data: Iterable[float]) -> float:
    pairs = Counter(data).most_common(1)
    if len(pairs) == 0:
        raise ValueError('no mode for empty data')
    return pairs[0][0]

Many uses of mode involve int or float values, but Python has other numerical types, and it is desirable that the return type follows the element type of the given Iterable. We can improve that TypeVar. Let’s start with a simple but wrong parameterized signature:

from typing import Iterable, TypeVar

T = TypeVar('T')

def mode(data: Iterable[T]) -> T:

When it first appears in the signature, the type parameter T can be any type. The second time it appears, it will mean the same type as the first.

Therefore, every iterable is-consistent-with Iterable[T], including iterables of unhashable types that collections.Counter cannot handle. We need to restrict the possible types assigned to T. We’ll see two ways of doing that in the next two sections.

from typing import Iterable, TypeVar
from decimal import Decimal
from fractions import Fraction

NumberT = TypeVar('NumberT', float, Decimal, Fraction)

def mode(data: Iterable[NumberT]) -> NumberT:

>>> mode(["red", "blue", "blue", "red", "green", "red", "red"])
'red'

NumberT = TypeVar('NumberT', float, Decimal, Fraction, str)

from typing import Iterable, Hashable

def mode(data: Iterable[Hashable]) -> Hashable:

from collections import Counter
from typing import Iterable, Hashable, TypeVar

HashableT = TypeVar('HashableT', bound=Hashable)

def mode(data: Iterable[HashableT]) -> HashableT:
    pairs = Counter(data).most_common(1)
    if len(pairs) == 0:
        raise ValueError('no mode for empty data')
    return pairs[0][0]

AnyStr = TypeVar('AnyStr', bytes, str)

Protocols

The Protocol type as presented in PEP 544—Protocols: Structural subtyping (static duck typing) is similar to interfaces in Go: a protocol type is defined by specifying one or more methods, and the type checker verifies that those methods are implemented where that protocol type is required.

In Python, a protocol definition is written as a typing.Protocol subclass. However, classes that implement a protocol don’t need to inherit, register or declare any relationship with the class that defines the protocol. It’s up to the type checker to find the available protocol types and enforce their usage.

Here is a problem that can be solved with the help of Protocol and TypeVar. Suppose you want to create a function top(it, n) that returns the largest n elements of the iterable it:

>>> top([4, 1, 5, 2, 6, 7, 3], 3)
[7, 6, 5]
>>> l = 'mango pear apple kiwi banana'.split()
>>> top(l, 3)
['pear', 'mango', 'kiwi']
>>>
>>> l2 = [(len(s), s) for s in l]
>>> l2
[(5, 'mango'), (4, 'pear'), (5, 'apple'), (4, 'kiwi'), (6, 'banana')]
>>> top(l2, 3)
[(6, 'banana'), (5, 'mango'), (5, 'apple')]

A parameterized generic top would look like this:

def top(series: Iterable[T], length: int) -> List[T]:
    ordered = sorted(series, reverse=True)
    return ordered[:length]

The problem is how to constrain T? It cannot be Any or object, because the series must work with sorted. The sorted built-in actually accepts Iterable[Any], but that’s because the optional argument key takes a function that computes an arbitrary sort key from each element. What happens if you don’t provide key and give a list of plain objects to sorted? Let’s try that:

>>> l = [object() for _ in range(4)]
>>> l
[<object object at 0x10fc2fca0>, <object object at 0x10fc2fbb0>,
<object object at 0x10fc2fbc0>, <object object at 0x10fc2fbd0>]
>>> sorted(l)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: '<' not supported between instances of 'object' and 'object'

That’s interesting: sorted needs the < operator on the elements of the iterable.

Is this all it takes? Let’s do another quick experiment:

>>> class Spam:
...     def __init__(self, n): self.n = n
...     def __lt__(self, other): return self.n < other.n
...     def __repr__(self): return f'Spam({self.n})'
...
>>> l = [Spam(n) for n in range(5, 0, -1)]
>>> l
[Spam(5), Spam(4), Spam(3), Spam(2), Spam(1)]
>>> sorted(l)
[Spam(1), Spam(2), Spam(3), Spam(4), Spam(5)]

That confirms it: I can sort a list of Spam because Spam implements __lt__—the special method that supports the < operator.²²

So the T type parameter in Example 8-25 should be limited to types that implement __lt__. In Example 8-24 we needed a type parameter that implemented __hash__, so we were able to use typing.Hashable as the upper bound for the type parameter. But now there is no suitable type in typing or abc to use, so we need to create it.

Here is the new Comparable type, a Protocol:

from typing import Protocol, Any

class Comparable(Protocol):  
    def __lt__(self, other: Any) -> bool: ...  

A protocol is a subclass of typing.Protocol.

The body of the protocol has one or more method definitions, with ... in their bodies.

A type T is-consistent-with a protocol P if T implements all the methods defined in P, with matching type signatures.

Given Comparable, we can now define this working version of top:

from typing import TypeVar, Iterable, List
from comparable import Comparable

CT = TypeVar('CT', bound=Comparable)

def top(series: Iterable[CT], length: int) -> List[CT]:
    ordered = sorted(series, reverse=True)
    return ordered[:length]

Let’s test-drive top. Example 8-28 shows part of a test suite for use with pytest. It tries calling top with a generator expression that yields of Tuple[int, str], and then with a list of object. With the list of object, we expect to get a TypeError exception.

def test_top_tuples() -> None:
    fruit = 'mango pear apple kiwi banana'.split()
    series: Iterator[Tuple[int, str]] = (
        (len(s), s) for s in fruit)
    length = 3
    expected = [(6, 'banana'), (5, 'mango'), (5, 'apple')]
    result = top(series, length)
    if TYPE_CHECKING:
        reveal_type(series)
        reveal_type(expected)
        reveal_type(result)
    assert result == expected

def test_top_objects_error() -> None:
    series = [object() for _ in range(4)]
    if TYPE_CHECKING:
        reveal_type(series)
    with pytest.raises(TypeError) as exc:
        top(series, 3)
    assert "'<' not supported" in str(exc)

The above tests pass—but they would pass anyway, even without type hints in top.py. More to the point, if I check that test file with Mypy, this is what I get:

…/comparable/ $ mypy top_test.py
top_test.py:27: note: Revealed type is 'typing.Iterator[Tuple[builtins.int, builtins.str]]'
top_test.py:28: note: Revealed type is 'builtins.list[Tuple[builtins.int, builtins.str]]'
top_test.py:29: note: Revealed type is 'builtins.list[Tuple[builtins.int, builtins.str]]'
top_test.py:35: note: Revealed type is 'builtins.list[builtins.object*]'
top_test.py:37: error: Value of type variable "CT" of "top" cannot be "object"
Found 1 error in 1 file (checked 1 source file)

The type check shows that the TypeVar is working as intended:

• in test_top_tuples, reveal_type confirms that the type returned by the top call is what we expected: given an Iterator[Tuple[int, str]], we got List[Tuple[str, int]];

• in test_top_objects_error, reveal_type shows the series argument type is List[object];

• Mypy flags the error: the element type of the series Iterable cannot be object.

A key advantage of a protocol type over ABCs is that a type needs no nominal connection with a specific protocol type to be consistent with it. I don’t need to derive or register str, tuple, float, set, etc. with Comparable to be able to use them where a Comparable parameter is expected. They only need to implement __lt__. And the type checker will still be able do its job, because Comparable is explicitly defined as a Protocol—in contrast with the implicit protocols that are common with duck typing, which are invisible to the type checker.

The special Procotol class was introduced in PEP 544—Protocols: Structural subtyping (static duck typing). Example 8-27 demonstrates why this feature is known as static duck typing: the solution to annotate the series parameter of top was to say “The nominal type of series doesn’t matter, as long as it implements __lt__“. Python’s duck typing always allowed us to say that implicitly, but the job of type checkers was much harder. A type checker can’t read CPython’s source code in C, or perform console experiments to find out that sorted only requires that the elements support <.

Now we are able to express this in code that the type checker can read. That’s why it makes sense to say that typing.Procotol gives us static duck typing.²³

There’s more to see about typing.Protocol, but we’ll leave that to Part IV, where [Link to Come] contrasts structural typing, duck typing, and ABCs—another approach to formalizing “classic” protocols.

Callable

To annotate callback parameters or function objects returned by higher-order functions, the typing module provides the Callable type, which is parameterized like this:

Callable[[ParamType1, ParamType2], ReturnType]

The parameter list—[ParamType1, ParamType2]—can have 0 or more types.

Here is an example in context:

def repl(input_fn: Callable[[Any], str] = input) -> None:

The repl function is part of a simple interactive interpreter.²⁴

During normal usage, the repl function uses Python’s input built-in to read expressions from the user. However, for automated testing or for integration with other input means, repl accepts an optional input_fn parameter: a Callable with the same parameter and return types as input.

The built-in input() has this signature on typeshed:

def input(__prompt: Any = ...) -> str: ...

That function is-consistent-with this Callable type hint:

Callable[[Any], str]

As another example, in Chapter 10, the Order.__init__ method in Example 10-3 uses this signature:

class Order:  # the Context

    def __init__(
        self,  
        customer: Customer,
        cart: Sequence[LineItem],
        promotion: Optional[Callable[['Order'], float]] = None,  
    ) -> None:  

self rarely needs a type hint.²⁵.

promotion may be None, or Callable[[Order], float]: a function that takes an Order and returns float.

__init__ always returns None, but I recommend recommend adding the return type hint for it anyway.²⁶

Note that the Order type appears as the string 'Order' in the Callable type hint, otherwise Python would raise NameError: name 'Order' is not defined—because the Order class is not defined until Python reads the whole body of the class—an issue we’ll discuss in [Link to Come]: Class Metaprogramming.

There is no syntax to annotate optional or keyword arguments in Callable[]. The documentation says “such function types are rarely used as callback types”. If you need a type hint to match a function with a dynamic signature, replace the whole parameter list with ..., like this: Callable[..., ReturnType].

NoReturn

This is a special type used only to annotate the return type of functions that never return. Usually, they exist to raise exceptions. There are dozens of such functions in the standard library.

For example: sys.exit() raises SystemExit, to terminate the Python process.

Its signature in typeshed is:

def exit(__status: object = ...) -> NoReturn: ...

The __status parameter is postional-only, and it has a default value. Stub files don’t spell out the default values: they use ... instead. The type of __status is object which means it may also be None, therefore it would be redundant to mark it Optional[object].

This ends our overview of the major groups of types used in type hints.