Main features

The different data class builders have a lot of common. Here we’ll discuss the main features they share. Table 5-1 summarizes.

No runtime effect

Type annotations don’t have any impact on the runtime behavior of Python programs. Check this out:

>>> import typing
>>> class Coordinate(typing.NamedTuple):
...     lat: float
...     long: float
...
>>> trash = Coordinate('foo', None)
>>> print(trash)
Coordinate(lat='Ni!', long=None)    

I told you: no type checking at runtime!

If you type the code of Example 5-9 in a Python module, replacing the last line with print(trash), it will happily run and display a meaningless Coordinate, with no error or warning:

$ python3 nocheck_demo.py
Coordinate(lat='Ni!', long=None)

The type hints are intended primarily to support third-party type checkers, like Mypy or the PyCharm IDE built-in type checker. These are static analysis tools: they check Python source code “at rest”, not running code.

To see the effect of type hints, you must run one of those tools on your code—like a linter. For instance, here is what Mypy has to say about the previous example:

$ mypy nocheck_demo.py
nocheck_demo.py:8: error: Argument 1 to "Coordinate" has
incompatible type "str"; expected "float"
nocheck_demo.py:8: error: Argument 2 to "Coordinate" has
incompatible type "None"; expected "float"

As you can see, given the definition of Coordinate, Mypy knows that both arguments to create an instance must be of type float, but the assignment to trash uses a str and None.⁵

Now let’s talk about the syntax and meaning of type hints.

Variable annotation Syntax

Both typing.NamedTuple and @dataclass use the syntax of variable annotations defined in PEP 526. This is quick introduction to that syntax in the context defining attributes in class statements.

The basic syntax of variable annotation is:

var_name: some_type

Acceptable type hints in PEP 484 explains what are acceptable types, but in the context of defining a concrete data class, these types are more useful:

• a concrete class, for example str or FrenchDeck;

• a parameterized collection type, like List[int], Tuple[str, float] etc.

• typing.Optional, for example Optional[str];

Although possible in theory, it’s not very useful to define fields with abstractions such as:

• special type constructs like Any, Union, Protocol etc. from the typing module;

• an ABC (Abstract Base Class);

You can also initialize the variable with a value. In a typing.NamedTuple or @dataclass declaration, that value will become the default for that attribute, if the corresponding argument is ommitted in the constructor call.

var_name: some_type = a_value

The meaning of variable annotations

We saw in “No runtime effect” that type hints have no effect at runtime. But at import time—when a module is loaded—Python does read them to build the __annotations__ dictionary that typing.NamedTuple and @dataclass then use to enhance the class.

We’ll start this exploration with a simple class, so that we can later see what extra features are added by typing.NamedTuple and @dataclass.

class DemoPlainClass:

    a: int           
    b: float = 1.1   
    c = 'spam'       

a becomes an annotation, but is otherwise discarded.

b is saved as an annotation, and also becomes a class attribute with value 1.1.

c is just a plain old class attribute, not an annotation.

We can verify that in the console, first reading the __annotations__ of the DemoPlainClass, then trying to get its attributes named a, b, and c:

>>> from demo_plain import DemoPlainClass
>>> DemoPlainClass.__annotations__
{'a': <class 'int'>, 'b': <class 'float'>}
>>> DemoPlainClass.a
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: type object 'DemoPlainClass' has no attribute 'a'
>>> DemoPlainClass.b
1.1
>>> DemoPlainClass.c
'spam'

Note that the __annotations__ special attribute is created by the interpreter to record the type hints that appear in the source code—even in a plain class.

The a survives only as an annotation. It doesn’t become a class attribute because no value is bound to it.⁶ The b and c are stored as class attributes because they are bound to values.

None of those three attributes will be in a new instance of DemoPlainClass. If you create an object o = DemoPlainClass(), o.a will raise AttributeError, while o.b and o.c will retrieve the class attributes with values 1.1 and 'spam'—that’s just normal Python object behavior.

import typing

class DemoNTClass(typing.NamedTuple):

    a: int           
    b: float = 1.1   
    c = 'spam'       

>>> from demo_nt import DemoNTClass
>>> DemoNTClass.__annotations__
{'a': <class 'int'>, 'b': <class 'float'>}
>>> DemoNTClass.a
<_collections._tuplegetter object at 0x101f0f940>
>>> DemoNTClass.b
<_collections._tuplegetter object at 0x101f0f8b0>
>>> DemoNTClass.c
'spam'

>>> DemoNTClass.__doc__
'DemoNTClass(a, b)'

>>> nt = DemoNTClass(8)
>>> nt.a
8
>>> nt.b
1.1
>>> nt.c
'spam'

from dataclasses import dataclass

@dataclass
class DemoDataClass:

    a: int           
    b: float = 1.1   
    c = 'spam'       

>>> from demo_dc import DemoDataClass
>>> DemoDataClass.__annotations__
{'a': <class 'int'>, 'b': <class 'float'>}
>>> DemoDataClass.__doc__
'DemoDataClass(a: int, b: float = 1.1)'
>>> DemoDataClass.a
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: type object 'DemoDataClass' has no attribute 'a'
>>> DemoDataClass.b
1.1
>>> DemoDataClass.c
'spam'

>>> dc = DemoDataClass(9)
>>> dc.a
9
>>> dc.b
1.1
>>> dc.c
'spam'

>>> dc.a = 10
>>> dc.b = 'oops'

>>> dc.c = 'whatever'
>>> dc.z = 'secret stash'

Field options

We’ve already seen most basic field option: providing or not a default value with the type hint. Note that fields are read in order, and after you declare a field with a default value, all remaining fields must also have default values. This limitation makes sense: the fields will become parameters in the generated __init__, and Python does not allow non-default parameters following parameters with defaults.

Mutable default values are a common source of bugs for beginning Python developers. In function definitions, a mutable default value is easily corrupted when one invocation of the function mutates the default, changing the behavior of further invocations—an issue we’ll explore in “Mutable Types as Parameter Defaults: Bad Idea” (Chapter 6). Class atributes are often used as default attribute values for instances, including in data classes. And @dataclass uses the default values in the type hints to generate parameters with defaults for __init__. To prevent bugs, @dataclass rejects the class definition in Example 5-13.

@dataclass
class ClubMember:

    name: str
    guests: list = []

If you load the module with that ClubMember class, this is what you get:

$ python3 club_wrong.py
Traceback (most recent call last):
  File "club_wrong.py", line 4, in <module>
    class ClubMember:
  ...several lines ommitted...
ValueError: mutable default <class 'list'> for field guests is not allowed:
use default_factory

The ValueError message explains the problem and suggests a solution: use default_factory. This is how to correct ClubMember:

from dataclasses import dataclass, field


@dataclass
class ClubMember:

    name: str
    guests: list = field(default_factory=list)

In the guests field of Example 5-14, instead of a literal list, the default value is set by calling the dataclasses.field function with default_factory=list. The default_factory parameter lets you provide a function, class, or any other callable, which will be invoked with zero arguments to build a default value each time an instance of the data class is created. This way, each instance of ClubMember will have its own list—instead of all instances sharing the same list from the class, which is rarely what we want and is often a bug.

If you browse the dataclasses module documentation, you’ll see a list field defined with a novel syntax, as in Example 5-15.

from dataclasses import dataclass, field
from typing import List                              

@dataclass
class ClubMember:

    name: str
    guests: List[str] = field(default_factory=list)  

Import the List type from typing.

List[str] means “a list of str”.

The new syntax List[str] is a generic type definition: the List class from typing accepts that bracket notation to specify the type of the list items. We’ll cover generics in [Link to Come]. For now, note that both Example 5-14 and Example 5-15 are correct, and the Mypy type checker does not complain about either of those class definitions. But the second one is more precise, and will allow the type checker to verify code that puts items in the list, or that read items from it.

The default_factory is by far the most frequently used option of the field function, but there are several others, listed in Table 5-3.

The default option exists because the field call takes the place of the default value in the field annotation. If you want to create an athlete field with default value of False, and also ommit that field from the __repr__ method, you’d write this:

@dataclass
class ClubMember:

    name: str
    guests: list = field(default_factory=list)
    athlete: bool = field(default=False, repr=False)

Post-init processing

The __init__ method generated by @dataclass only takes the arguments passed and assigns them—or their default values, if missing—to the instance attributes that are instance fields. But you may need to do more than that to initialize the instance. If that’s the case, you can provide a __post_init__ method. When that method exists, @dataclass will add code to the generated __init__ to call __post_init__ as the last step.

Common use cases for __post_init__ are validation and computing field values based on other fields. We’ll study a simple example that uses __post_init__ for both of these reasons.

First, let’s look at the expected behavior of a ClubMember subclass named HackerClubMember, as described by doctests in Example 5-16.

"""
``HackerClubMember`` objects accept an optional ``handle`` argument::

    >>> anna = HackerClubMember('Anna Ravenscroft', handle='AnnaRaven')
    >>> anna
    HackerClubMember(name='Anna Ravenscroft', guests=[], handle='AnnaRaven')

If ``handle`` is ommitted, it's set to the first part of the member's name::

    >>> leo = HackerClubMember('Leo Rochael')
    >>> leo
    HackerClubMember(name='Leo Rochael', guests=[], handle='Leo')

Members must have a unique handle. The following ``leo2`` will not be created,
because its ``handle`` would be 'Leo', which was taken by ``leo``::

    >>> leo2 = HackerClubMember('Leo DaVinci')
    Traceback (most recent call last):
      ...
    ValueError: handle 'Leo' already exists.

To fix, ``leo2`` must be created with an explicit ``handle``::

    >>> leo2 = HackerClubMember('Leo DaVinci', handle='Neo')
    >>> leo2
    HackerClubMember(name='Leo DaVinci', guests=[], handle='Neo')
"""

Note that we must provide handle as a keyword argument, because HackerClubMember inherits name and guests from ClubMember, and adds the handle field. The generated docstring for HackerClubMember shows the order of the fields in the constructor call:

>>> HackerClubMember.__doc__
"HackerClubMember(name: str, guests: list = <factory>, handle: str = '')"

Here, <factory> is a short way of saying that some callable will produce the default value for guests (in our case, the factory is the list class). The point is: to provide a handle but no guests, we must pass handle as a keyword argument.

The Inheritance section of the dataclasses module documentation explains how the order of the fields is computed when there are several levels of inheritance.

Example 5-17 is the implementation:

from dataclasses import dataclass
from club import ClubMember

@dataclass
class HackerClubMember(ClubMember):                         

    all_handles = set()                                     

    handle: str = ''                                        

    def __post_init__(self):
        cls = self.__class__                                
        if self.handle == '':                               
            self.handle = self.name.split()[0]
        if self.handle in cls.all_handles:                  
            msg = f'handle {self.handle!r} already exists.'
            raise ValueError(msg)
        cls.all_handles.add(self.handle)                    

HackerClubMember extends ClubMember.

all_handles is a class attribute.

handle is an instance field of type str with empty string as its default value; this makes it optional.

Get the class of the instance.

If self.handle is the empty string, set it to the first part of name.

If self.handle is in cls.all_handles, raise ValueError.

Add the new handle to cls.all_handles.

Example 5-17 works as intended, but is not satisfactory to a static type checker. Next, we’ll see why, and how to fix it.

Typed class attributes

If we typecheck Example 5-17 with Mypy, we are reprimanded:

$ mypy hackerclub.py
hackerclub.py:38: error: Need type annotation for 'all_handles'
(hint: "all_handles: Set[<type>] = ...")
Found 1 error in 1 file (checked 1 source file)

Unfortunately, the hint provided by Mypy (version 0.750 as I write this) is not helpful in the context of @dataclass usage. If we add a type hint like Set[…] to all_handles, @dataclass will find that annotation and make all_handles an instance field. We saw this happening in “Inspecting a class decorated with dataclass”.

The work-around defined in PEP 526—Syntax for Variable Annotations is a class variable annotation, written with a pseudo-type named typing.ClassVar, which leverages the generics [] notation to set the type of the variable and also declare it a class attribute.

To make the type checker happy, this is how we are supposed to declare all_handles in Example 5-17:

    all_handles: ClassVar[Set[str]] = set()

That type hint is saying:

all_handles is a class attribute of type set-of-str, with an empty set as its default value.

To code that annotation, we must import ClassVar and Set from the typing module.

The @dataclass decorator doesn’t care about the types in the annotations, except in two cases, and this is one of them: if the type is ClassVar, an instance field will not be generated for that attribute.

The other case where the type of the field is relevant to @dataclass is when declaring init-only variables, our next topic.

Initialization variables that are not fields

Sometimes you may neet to pass arguments to __init__ that are not instance fields. Such arguments are called init-only variables by the dataclasses documentation. To declare an argument like that, dataclasses module provides the pseudo-type InitVar, which uses the same syntax of typing.ClassVar. The example given in the documentation is a data class that has a field initialized from a database, and the database object must be passed to the constructor.

This is the code that illustrates the Init-only variables section:

@dataclass
class C:
    i: int
    j: int = None
    database: InitVar[DatabaseType] = None

    def __post_init__(self, database):
        if self.j is None and database is not None:
            self.j = database.lookup('j')

c = C(10, database=my_database)

Note how the database attribute is declared. InitVar will prevent @dataclass from treating database as a regular field. It will not be set as an instance attribute, and the dataclasses.fields function will not list it. However, database will be one of the arguments that the generated __init__ will accept, and it will be also passed to __post_init__—if you write that method, you must add a corresponding argument to the method signature, as shown in Example 5-18

This rather long overview of @dataclass covered the most useful features—some of them appeared in previous sections, like “Main features” where we covered all three data class builders in parallel. The dataclasses documentation and PEP 526 — Syntax for Variable Annotations have all details.

@dataclass Example: Dublin Core Resource Record

Often, classes built with @dataclass will have more fields than the very short examples presented so far. Dublin Core provides the foundation for a more typical @dataclass example.

The Dublin Core Schema is a small set of vocabulary terms that can be used to describe digital resources (video, images, web pages, etc.), as well as physical resources such as books or CDs, and objects like artworks.

Dublin Core on Wikipedia

The standard defines 15 optional fields, the Resource class in Example 5-19 uses 8 of them.

from dataclasses import dataclass, field
from typing import List, Optional
from enum import Enum, auto
from datetime import date


class ResourceType(Enum):                              
    BOOK = auto()
    EBOOK = auto()
    VIDEO = auto()


@dataclass
class Resource:
    """Media resource description."""
    identifier: str                                    
    title: str = '<untitled>'                          
    creators: List[str] = field(default_factory=list)
    date: Optional[date] = None                        
    type: ResourceType = ResourceType.BOOK             
    description: str = ''
    language: str = ''
    subjects: List[str] = field(default_factory=list)

This Enum will provide type-safe values for the Resource.type field.

identifier is the only required field.

title is the first field with a default. This forces all fields below to provide defaults.

The value of date can be a datetime.date instance, or None.

The type field default is ResourceType.BOOK.

Example 5-20 is a doctest to demonstrate how a Resource record appears in code:

    >>> description = 'Improving the design of existing code'
    >>> book = Resource('978-0-13-475759-9', 'Refactoring, 2nd Edition',
    ...     ['Martin Fowler', 'Kent Beck'], date(2018, 11, 19),
    ...     ResourceType.BOOK, description, 'EN',
    ...     ['computer programming', 'OOP'])
    >>> book  # doctest: +NORMALIZE_WHITESPACE
    Resource(identifier='978-0-13-475759-9', title='Refactoring, 2nd Edition',
    creators=['Martin Fowler', 'Kent Beck'], date=datetime.date(2018, 11, 19),
    type=<ResourceType.BOOK: 1>, description='Improving the design of existing code',
    language='EN', subjects=['computer programming', 'OOP'])

The __repr__ generated by @dataclass is OK, but we can do better. This is the format we want from repr(book):

    >>> book  # doctest: +NORMALIZE_WHITESPACE
    Resource(
        identifier = '978-0-13-475759-9',
        title = 'Refactoring, 2nd Edition',
        creators = ['Martin Fowler', 'Kent Beck'],
        date = datetime.date(2018, 11, 19),
        type = <ResourceType.BOOK: 1>,
        description = 'Improving the design of existing code',
        language = 'EN',
        subjects = ['computer programming', 'OOP'],
    )

Example 5-21 is the code of __repr__ to produce the format above. This example uses doctest.fields to get the names of the data class fields.

    def __repr__(self):
        cls = self.__class__
        cls_name = cls.__name__
        indent = ' ' * 4
        res = [f'{cls_name}(']                            
        for f in fields(cls):                             
            value = getattr(self, f.name)                 
            res.append(f'{indent}{f.name} = {value!r},')  

        res.append(')')                                   
        return '
'.join(res)                             

Start the res list to build the output string with the class name and open parenthesis.

For each field f in the class…

Get the named attribute from the instance.

Append an indented line with the name of the field and repr(value)—that’s what the !r does.

Append closing parenthesis.

Build multiline string from res and return it.

With this example inspired by the soul of Dublin, Ohio, we conclude our tour of Python’s data class builders.

Data classes are handy, but your project may suffer if you overuse them. The next section explains.

Data class as scaffolding

In this scenario, the data class is an initial, simplistic implementation of a class to jump start a new project or module. With time, the class should get its own methods, instead of relying on methods of other classes to operate on its instances. Scaffolding is temporary; eventually your custom class may become fully independent from the builder you used to start it.

Python is also used for quick problem solving and experimentation, and then it’s OK leave the scaffolding in place.

Data class as intermediate representation

A data class can be useful to build records about to be exported to JSON or some other interchange format, or to hold data that was just imported, crossing some system boundary. Python’s data class builders all provide a method or function to convert an instance to a plain dict, and you can always invoke the constructor with a dict used as keyword arguments expanded with **. Such a dict is very close to a JSON record.

In this scenario, the data class instances should be handled as immutable objects—even if the fields are mutable, you should not change them while they are in this intermediate form. If you do, you’re losing the key benefit of having data and behavior close together. When importing/exporting requires changing values, you should implement your own builder methods instead of using the given “as dict” methods or standard constructors.

After reviewing Python’s data class builders, we’ll end the chapter with the struct module, also used for importing/exporting records, but at a much lower level.

Structs and Memory Views

We saw in “Memory Views” that the memoryview type does not let you create or store byte sequences, but provides shared memory access to slices of data from other binary sequences, packed arrays, and buffers such as Python Imaging Library (PIL) images,¹¹ without copying the bytes.

Example 5-25 shows the use of memoryview and struct together to extract the width and height of a GIF image.

>>> import struct
>>> fmt = '<3s3sHH'  
>>> with open('filter.gif', 'rb') as fp:
...     img = memoryview(fp.read())  
...
>>> header = img[:10]  
>>> bytes(header)  
b'GIF89a+x02xe6x00'
>>> struct.unpack(fmt, header)  
(b'GIF', b'89a', 555, 230)
>>> del header  
>>> del img

struct format: < little-endian; 3s3s two sequences of 3 bytes; HH two 16-bit integers.

Create memoryview from file contents in memory…

…then another memoryview by slicing the first one; no bytes are copied here.

Convert to bytes for display only; 10 bytes are copied here.

Unpack memoryview into tuple of: type, version, width, and height.

Delete references to release the memory associated with the memoryview instances.

Note that slicing a memoryview returns a new memoryview, without copying bytes.¹²

I will not go deeper into memoryview or the struct module, but if you work with binary data, you’ll find it worthwhile to study their docs: Built-in Types » Memory Views and struct — Interpret bytes as packed binary data.

Should we use struct?

I only recommend struct to handle data from legacy systems or standardized binary formats.

Proprietary binary records in the real world are brittle and can be corrupted easily. Our super simple example exposed one of many caveats: a string field may be limited only by its size in bytes, it may be padded by spaces, or it may contain a null-terminated string followed by random garbage up to a certain size. There is also the problem of endianness: the order of the bytes used to represent integers and floats, which depends on the CPU architecture.

If you need to exchange binary data among Python systems, the pickle module is the easiest way, by far. If the exchange involves programs in other languages, use JSON or a multi-platform binary serialization format like MessagePack or Protocol Buffers.