Interfaces and Protocols in Python Culture

Python was already highly successful before ABCs were introduced, and most existing code does not use them at all. Since Chapter 1, we’ve been talking about duck typing and protocols. In “Protocols and Duck Typing”, protocols are defined as the informal interfaces that make polymorphism work in languages with dynamic typing like Python.

How do interfaces work in a dynamic-typed language? First, the basics: even without an interface keyword in the language, and regardless of ABCs, every class has an interface: the set public attributes (methods or data attributes) implemented or inherited by the class. This includes special methods, like __getitem__ or __add__.

By definition, protected and private attributes are not part of an interface, even if “protected” is merely a naming convention (the single leading underscore) and private attributes are easily accessed (recall “Private and “Protected” Attributes in Python”). It is bad form to violate these conventions.

On the other hand, it’s not a sin to have public data attributes as part of the interface of an object, because—if necessary—a data attribute can always be turned into a property implementing getter/setter logic without breaking client code that uses the plain obj.attr syntax. We did that in the Vector2d class: in Example 11-1, we see the first implementation with public x and y attributes.

class Vector2d:
    typecode = 'd'

    def __init__(self, x, y):
        self.x = float(x)
        self.y = float(y)

    def __iter__(self):
        return (i for i in (self.x, self.y))

    # more methods follow (omitted in this listing)

In Example 9-7, we turned x and y into read-only properties (Example 11-2). This is a significant refactoring, but an essential part of the interface of Vector2d is unchanged: users can still read my_vector.x and my_vector.y.

class Vector2d:
    typecode = 'd'

    def __init__(self, x, y):
        self.__x = float(x)
        self.__y = float(y)

    @property
    def x(self):
        return self.__x

    @property
    def y(self):
        return self.__y

    def __iter__(self):
        return (i for i in (self.x, self.y))

    # more methods follow (omitted in this listing)

A useful complementary definition of interface is: the subset of an object’s public methods that enable it to play a specific role in the system. That’s what is implied when the Python documentation mentions “a file-like object” or “an iterable,” without specifying a class. An interface seen as a set of methods to fulfill a role is what Smalltalkers called a procotol, and the term spread to other dynamic language communities. Protocols are independent of inheritance. A class may implement several protocols, enabling its instances to fulfill several roles.

Protocols are interfaces, but because they are informal—defined only by documentation and conventions—protocols cannot be enforced like formal interfaces can (we’ll see how ABCs enforce interface conformance later in this chapter). A protocol may be partially implemented in a particular class, and that’s OK. Sometimes all a specific API requires from “a file-like object” is that it has a .read() method that returns bytes. The remaining file methods may or may not be relevant in the context.

As I write this, the Python 3 documentation of memoryview says that it works with objects that “support the buffer protocol, which is only documented at the C API level. The bytearray constructor accepts an “an object conforming to the buffer interface.” Now there is a move to adopt “bytes-like object” as a friendlier term.² I point this out to emphasize that “X-like object,” “X protocol,” and “X interface” are synonyms in the minds of Pythonistas.

One of the most fundamental interfaces in Python is the sequence protocol. The interpreter goes out of its way to handle objects that provide even a minimal implementation of that protocol, as the next section demonstrates.

Python Digs Sequences

The philosophy of the Python data model is to cooperate with essential protocols as much as possible. When it comes to sequences, Python tries hard to work with even the simplest implementations.

Figure 11-1 shows how the formal Sequence interface is defined as an ABC.

Figure 11-1. UML class diagram for the Sequence ABC and related abstract classes from collections.abc. Inheritance arrows point from subclass to its superclasses. Names in italic are abstract methods.

Now, take a look at the Foo class in Example 11-3. It does not inherit from abc.Sequence, and it only implements one method of the sequence protocol: __getitem__ (__len__ is missing).

>>> class Foo:
...     def __getitem__(self, pos):
...         return range(0, 30, 10)[pos]
...
>>> f[1]
10
>>> f = Foo()
>>> for i in f: print(i)
...
0
10
20
>>> 20 in f
True
>>> 15 in f
False

There is no method __iter__ yet Foo instances are iterable because—as a fallback—when Python sees a __getitem__ method, it tries to iterate over the object by calling that method with integer indexes starting with 0. Because Python is smart enough to iterate over Foo instances, it can also make the in operator work even if Foo has no __contains__ method: it does a full scan to check if an item is present.

In summary, given the importance of the sequence protocol, in the absence __iter__ and __contains__ Python still manages to make iteration and the in operator work by invoking __getitem__.

Our original FrenchDeck from Chapter 1 does not subclass from abc.Sequence either, but it does implement both methods of the sequence protocol: __getitem__ and __len__. See Example 11-4.

import collections

Card = collections.namedtuple('Card', ['rank', 'suit'])

class FrenchDeck:
    ranks = [str(n) for n in range(2, 11)] + list('JQKA')
    suits = 'spades diamonds clubs hearts'.split()

    def __init__(self):
        self._cards = [Card(rank, suit) for suit in self.suits
                                        for rank in self.ranks]

    def __len__(self):
        return len(self._cards)

    def __getitem__(self, position):
        return self._cards[position]

A good part of the demos in Chapter 1 work because of the special treatment Python gives to anything vaguely resembling a sequence. Iteration in Python represents an extreme form of duck typing: the interpreter tries two different methods to iterate over objects.

Now let’s study another example emphasizing the dynamic nature of protocols.

Monkey-Patching to Implement a Protocol at Runtime

The FrenchDeck class from Example 11-4 has a major flaw: it cannot be shuffled. Years ago when I first wrote the FrenchDeck example I did implement a shuffle method. Later I had a Pythonic insight: if a FrenchDeck acts like a sequence, then it doesn’t need its own shuffle method because there is already random.shuffle, documented as “Shuffle the sequence x in place.”

The standard random.shuffle function is used like this:

>>> from random import shuffle
>>> l = list(range(10))
>>> shuffle(l)
>>> l
[5, 2, 9, 7, 8, 3, 1, 4, 0, 6]

However, if we try to shuffle a FrenchDeck instance, we get an exception, as in Example 11-5.

>>> from random import shuffle
>>> from frenchdeck import FrenchDeck
>>> deck = FrenchDeck()
>>> shuffle(deck)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../python3.3/random.py", line 265, in shuffle
    x[i], x[j] = x[j], x[i]
TypeError: 'FrenchDeck' object does not support item assignment

The error message is quite clear: “'FrenchDeck' object does not support item assignment.” The problem is that shuffle operates by swapping items inside the collection, and FrenchDeck only implements the immutable sequence protocol. Mutable sequences must also provide a __setitem__ method.

Because Python is dynamic, we can fix this at runtime, even at the interactive console. Example 11-6 shows how to do it.

>>> def set_card(deck, position, card):  
...     deck._cards[position] = card
...
>>> FrenchDeck.__setitem__ = set_card  
>>> shuffle(deck)  
>>> deck[:5]
[Card(rank='3', suit='hearts'), Card(rank='4', suit='diamonds'), Card(rank='4',
suit='clubs'), Card(rank='7', suit='hearts'), Card(rank='9', suit='spades')]

Create a function that takes deck, position, and card as arguments.

Assign that function to an attribute named __setitem__ in the FrenchDeck class.

deck can now be sorted because FrenchDeck now implements the necessary method of the mutable sequence protocol.

The signature of the __setitem__ special method is defined in The Python Language Reference in “3.3.6. Emulating container types”. Here we named the arguments deck, position, card—and not self, key, value as in the language reference—to show that every Python method starts life as a plain function, and naming the first argument self is merely a convention. This is OK in a console session, but in a Python source file it’s much better to use self, key, and value as documented.

The trick is that set_card knows that the deck object has an attribute named _cards, and _cards must be a mutable sequence. The set_card function is then attached to the FrenchDeck class as the __setitem__ special method. This is an example of monkey patching: changing a class or module at runtime, without touching the source code. Monkey patching is powerful, but the code that does the actual patching is very tightly coupled with the program to be patched, often handling private and undocumented parts.

Besides being an example of monkey patching, Example 11-6 highlights that protocols are dynamic: random.shuffle doesn’t care what type of argument it gets, it only needs the object to implement part of the mutable sequence protocol. It doesn’t even matter if the object was “born” with the necessary methods or if they were somehow acquired later.

The theme of this chapter so far has been “duck typing”: operating with objects regardless of their types, as long as they implement certain protocols.

When we did present diagrams with ABCs, the intent was to show how the protocols are related to the explicit interfaces documented in the abstract classes, but we did not actually inherit from any ABC so far.

In the following sections, we will leverage ABCs directly, and not just as documentation.

Alex Martelli’s Waterfowl

After reviewing the usual protocol-style interfaces of Python, we move to ABCs. But before diving into examples and details, Alex Martelli explains in a guest essay why ABCs were a great addition to Python.

Besides coining the “goose typing,” Alex makes the point that inheriting from an ABC is more than implementing the required methods: it’s also a clear declaration of intent by the developer. That intent can also be made explicit through registering a virtual subclass.

In addition, the use of isinstance and issubclass becomes more acceptable to test against ABCs. In the past, these functions worked against duck typing, but with ABCs they become more flexible. After all, if a component does not implement an ABC by subclassing, it can always be registered after the fact so it passes those explicit type checks.

However, even with ABCs, you should beware that excessive use of isinstance checks may be a code smell—a symptom of bad OO design. It’s usually not OK to have a chain of if/elif/elif with insinstance checks performing different actions depending on the type of an object: you should be using polymorphism for that—i.e., designing your classes so that the interpreter dispatches calls to the proper methods, instead of you hardcoding the dispatch logic in if/elif/elif blocks.

On the other hand, it’s usually OK to perform an insinstance check against an ABC if you must enforce an API contract: “Dude, you have to implement this if you want to call me,” as technical reviewer Lennart Regebro put it. That’s particularly useful in systems that have a plug-in architecture. Outside of frameworks, duck typing is often simpler and more flexible than type checks.

For example, in several classes in this book, when I needed to take a sequence of items and process them as a list, instead of requiring a list argument by type checking, I simply took the argument and immediately built a list from it: that way I can accept any iterable, and if the argument is not iterable, the call will fail soon enough with a very clear message. One example of this code pattern is in the __init__ method in Example 11-13, later in this chapter. Of course, this approach wouldn’t work if the sequence argument shouldn’t be copied, either because it’s too large or because my code needs to change it in place. Then an insinstance(x, abc.MutableSequence) would be better. If any iterable is acceptable, then calling iter(x) to obtain an iterator would be the way to go, as we’ll see in “Why Sequences Are Iterable: The iter Function”.

Another example is how you might imitate the handling of the field_names argument in collections.namedtuple: field_names accepts a single string with identifiers separated by spaces or commas, or a sequence of identifiers. It might be tempting to use isinstance, but Example 11-7 shows how I’d do it using duck typing.⁵

    try:  
        field_names = field_names.replace(',', ' ').split()  
    except AttributeError:  
        pass  
    field_names = tuple(field_names)  

Assume it’s a string (EAFP = it’s easier to ask forgiveness than permission).

Convert commas to spaces and split the result into a list of names.

Sorry, field_names doesn’t quack like a str… there’s either no .replace, or it returns something we can’t .split.

Now we assume it’s already an iterable of names.

To make sure it’s an iterable and to keep our own copy, create a tuple out of what we have.

Finally, in his essay, Alex reinforces more than once the need for restraint in the creation of ABCs. An ABC epidemic would be disastrous, imposing excessive ceremony in a language that became popular because it’s practical and pragmatic. During the Fluent Python review process, Alex wrote:

ABCs are meant to encapsulate very general concepts, abstractions, introduced by a framework—things like “a sequence” and “an exact number.” [Readers] most likely don’t need to write any new ABCs, just use existing ones correctly, to get 99.9% of the benefits without serious risk of misdesign.

Now let’s see goose typing in practice.

Subclassing an ABC

Following Martelli’s advice, we’ll leverage an existing ABC, collections.MutableSequence, before daring to invent our own. In Example 11-8, FrenchDeck2 is explicitly declared a subclass of collections.MutableSequence.

import collections

Card = collections.namedtuple('Card', ['rank', 'suit'])

class FrenchDeck2(collections.MutableSequence):
    ranks = [str(n) for n in range(2, 11)] + list('JQKA')
    suits = 'spades diamonds clubs hearts'.split()

    def __init__(self):
        self._cards = [Card(rank, suit) for suit in self.suits
                                        for rank in self.ranks]

    def __len__(self):
        return len(self._cards)

    def __getitem__(self, position):
        return self._cards[position]

    def __setitem__(self, position, value):  
        self._cards[position] = value

    def __delitem__(self, position):  
        del self._cards[position]

    def insert(self, position, value):  
        self._cards.insert(position, value)

__setitem__ is all we need to enable shuffling…

But subclassing MutableSequence forces us to implement __delitem__, an abstract method of that ABC.

We are also required to implement insert, the third abstract method of MutableSequence.

Python does not check for the implementation of the abstract methods at import time (when the frenchdeck2.py module is loaded and compiled), but only at runtime when we actually try to instantiate FrenchDeck2. Then, if we fail to implement any abstract method, we get a TypeError exception with a message such as "Can't instantiate abstract class FrenchDeck2 with abstract methods __delitem__, insert". That’s why we must implement __delitem__ and insert, even if our FrenchDeck2 examples do not need those behaviors: the MutableSequence ABC demands them.

As Figure 11-2 shows, not all methods of the Sequence and MutableSequence ABCs are abstract.

Figure 11-2. UML class diagram for the MutableSequence ABC and its superclasses from collections.abc (inheritance arrows point from subclasses to ancestors; names in italic are abstract classes and abstract methods)

From Sequence, FrenchDeck2 inherits the following ready-to-use concrete methods: __contains__, __iter__, __reversed__, index, and count. From MutableSequence, it gets append, reverse, extend, pop, remove, and __iadd__.

The concrete methods in each collections.abc ABC are implemented in terms of the public interface of the class, so they work without any knowledge of the internal structure of instances.

To use ABCs well, you need to know what’s available. We’ll review the collections ABCs next.

ABCs in the Standard Library

Since Python 2.6, ABCs are available in the standard library. Most are defined in the collections.abc module, but there are others. You can find ABCs in the numbers and io packages, for example. But the most widely used is collections.abc. Let’s see what is available there.

ABCs in collections.abc

Tip

There are two modules named abc in the standard library. Here we are talking about collections.abc. To reduce loading time, in Python 3.4, it’s implemented outside of the collections package, in Lib/_collections_abc.py), so it’s imported separately from collections. The other abc module is just abc (i.e., Lib/abc.py) where the abc.ABC class is defined. Every ABC depends on it, but we don’t need to import it ourselves except to create a new ABC.

Figure 11-3 is a summary UML class diagram (without attribute names) of all 16 ABCs defined in collections.abc as of Python 3.4. The official documentation of collections.abc has a nice table summarizing the ABCs, their relationships, and their abstract and concrete methods (called “mixin methods”). There is plenty of multiple inheritance going on in Figure 11-3. We’ll devote most of Chapter 12 to multiple inheritance, but for now it’s enough to say that it is usually not a problem when ABCs are concerned.⁶

Figure 11-3. UML class diagram for ABCs in collections.abc

Let’s review the clusters in Figure 11-3:

Iterable, Container, and Sized

Every collection should either inherit from these ABCs or at least implement compatible protocols. Iterable supports iteration with __iter__, Container supports the in operator with __contains__, and Sized supports len() with __len__.

Sequence, Mapping, and Set

These are the main immutable collection types, and each has a mutable subclass. A detailed diagram for MutableSequence is in Figure 11-2; for MutableMapping and MutableSet, there are diagrams in Chapter 3 (Figures 3-1 and 3-2).

MappingView

In Python 3, the objects returned from the mapping methods .items(), .keys(), and .values() inherit from ItemsView, ValuesView, and ValuesView, respectively. The first two also inherit the rich interface of Set, with all the operators we saw in “Set Operations”.

Callable and Hashable

These ABCs are not so closely related to collections, but collections.abc was the first package to define ABCs in the standard library, and these two were deemed important enough to be included. I’ve never seen subclasses of either Callable or Hashable. Their main use is to support the insinstance built-in as a safe way of determining whether an object is callable or hashable.⁷

Iterator

Note that iterator subclasses Iterable. We discuss this further in Chapter 14.

After the collections.abc package, the most useful package of ABCs in the standard library is numbers, covered next.

Defining and Using an ABC

To justify creating an ABC, we need to come up with a context for using it as an extension point in a framework. So here is our context: imagine you need to display advertisements on a website or a mobile app in random order, but without repeating an ad before the full inventory of ads is shown. Now let’s assume we are building an ad management framework called ADAM. One of its requirements is to support user-provided nonrepeating random-picking classes.⁸ To make it clear to ADAM users what is expected of a “nonrepeating random-picking” component, we’ll define an ABC.

Taking a clue from “stack” and “queue” (which describe abstract interfaces in terms of physical arrangements of objects), I will use a real-world metaphor to name our ABC: bingo cages and lottery blowers are machines designed to pick items at random from a finite set, without repeating, until the set is exhausted.

The ABC will be named Tombola, after the Italian name of bingo and the tumbling container that mixes the numbers.⁹

The Tombola ABC has four methods. The two abstract methods are:

• .load(…): put items into the container.

• .pick(): remove one item at random from the container, returning it.

The concrete methods are:

• .loaded(): return True if there is at least one item in the container.

• .inspect(): return a sorted tuple built from the items currently in the container, without changing its contents (its internal ordering is not preserved).

Figure 11-4 shows the Tombola ABC and three concrete implementations.

Figure 11-4. UML diagram for an ABC and three subclasses. The name of the Tombola ABC and its abstract methods are written in italics, per UML conventions. The dashed arrow is used for interface implementation, here we are using it to show that TomboList is a virtual subclass of Tombola because it is registered, as we will see later in this chapter.¹⁰

Example 11-9 shows the definition of the Tombola ABC.

import abc

class Tombola(abc.ABC):  

    @abc.abstractmethod
    def load(self, iterable):  
        """Add items from an iterable."""

    @abc.abstractmethod
    def pick(self):  
        """Remove item at random, returning it.

        This method should raise `LookupError` when the instance is empty.
        """

    def loaded(self):  
        """Return `True` if there's at least 1 item, `False` otherwise."""
        return bool(self.inspect())  


    def inspect(self):
        """Return a sorted tuple with the items currently inside."""
        items = []
        while True:  
            try:
                items.append(self.pick())
            except LookupError:
                break
        self.load(items)  
        return tuple(sorted(items))

To define an ABC, subclass abc.ABC.

An abstract method is marked with the @abstractmethod decorator, and often its body is empty except for a docstring.¹¹

The docstring instructs implementers to raise LookupError if there are no items to pick.

An ABC may include concrete methods.

Concrete methods in an ABC must rely only on the interface defined by the ABC (i.e., other concrete or abstract methods or properties of the ABC).

We can’t know how concrete subclasses will store the items, but we can build the inspect result by emptying the Tombola with successive calls to .pick()…

…then use .load(…) to put everything back.

The .inspect() method in Example 11-9 is perhaps a silly example, but it shows that, given .pick() and .load(…) we can inspect what’s inside the Tombola by picking all items and loading them back. The point of this example is to highlight that it’s OK to provide concrete methods in ABCs, as long as they only depend on other methods in the interface. Being aware of their internal data structures, concrete subclasses of Tombola may always override .inspect() with a smarter implementation, but they don’t have to.

The .loaded() method in Example 11-9 may not be as silly, but it’s expensive: it calls .inspect() to build the sorted tuple just to apply bool() on it. This works, but a concrete subclass can do much better, as we’ll see.

Note that our roundabout implementation of .inspect() requires that we catch a LookupError thrown by self.pick(). The fact that self.pick() may raise LookupError is also part of its interface, but there is no way to declare this in Python, except in the documentation (see the docstring for the abstract pick method in Example 11-9.)

I chose the LookupError exception because of its place in the Python hierarchy of exceptions in relation to IndexError and KeyError, the most likely exceptions to be raised by the data structures used to implement a concrete Tombola. Therefore, implementations can raise LookupError, IndexError, or KeyError to comply. See Example 11-10 (for a complete tree, see “5.4. Exception hierarchy” of The Python Standard Library).

BaseException
 ├── SystemExit
 ├── KeyboardInterrupt
 ├── GeneratorExit
 └── Exception
      ├── StopIteration
      ├── ArithmeticError
      │    ├── FloatingPointError
      │    ├── OverflowError
      │    └── ZeroDivisionError
      ├── AssertionError
      ├── AttributeError
      ├── BufferError
      ├── EOFError
      ├── ImportError
      ├── LookupError  
      │    ├── IndexError  
      │    └── KeyError  
      ├── MemoryError
      ... etc.

LookupError is the exception we handle in Tombola.inspect.

IndexError is the LookupError subclass raised when we try to get an item from a sequence with an index beyond the last position.

KeyError is raised when we use a nonexistent key to get an item from a mapping.

We now have our very own Tombola ABC. To witness the interface checking performed by an ABC, let’s try to fool Tombola with a defective implementation in Example 11-11.

>>> from tombola import Tombola
>>> class Fake(Tombola):  
...     def pick(self):
...         return 13
...
>>> Fake  
<class '__main__.Fake'>
<class 'abc.ABC'>, <class 'object'>)
>>> f = Fake()  
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: Can't instantiate abstract class Fake with abstract methods load

Declare Fake as a subclass of Tombola.

The class was created, no errors so far.

TypeError is raised when we try to instantiate Fake. The message is very clear: Fake is considered abstract because it failed to implement load, one of the abstract methods declared in the Tombola ABC.

So we have our first ABC defined, and we put it to work validating a class. We’ll soon subclass the Tombola ABC, but first we must cover some ABC coding rules.

class Tombola(metaclass=abc.ABCMeta):
    # ...

class Tombola(object):  # this is Python 2!!!
    __metaclass__ = abc.ABCMeta
    # ...

class MyABC(abc.ABC):
    @classmethod
    @abc.abstractmethod
    def an_abstract_classmethod(cls, ...):
        pass

import random

from tombola import Tombola


class BingoCage(Tombola):  

    def __init__(self, items):
        self._randomizer = random.SystemRandom()  
        self._items = []
        self.load(items)  

    def load(self, items):
        self._items.extend(items)
        self._randomizer.shuffle(self._items)  

    def pick(self):  
        try:
            return self._items.pop()
        except IndexError:
            raise LookupError('pick from empty BingoCage')

    def __call__(self):  
        self.pick()

import random

from tombola import Tombola


class LotteryBlower(Tombola):

    def __init__(self, iterable):
        self._balls = list(iterable)  

    def load(self, iterable):
        self._balls.extend(iterable)

    def pick(self):
        try:
            position = random.randrange(len(self._balls))  
        except ValueError:
            raise LookupError('pick from empty BingoCage')
        return self._balls.pop(position)  

    def loaded(self):  
        return bool(self._balls)

    def inspect(self):  
        return tuple(sorted(self._balls))

A Virtual Subclass of Tombola

An essential characteristic of goose typing—and the reason why it deserves a waterfowl name—is the ability to register a class as a virtual subclass of an ABC, even if it does not inherit from it. When doing so, we promise that the class faithfully implements the interface defined in the ABC—and Python will believe us without checking. If we lie, we’ll be caught by the usual runtime exceptions.

This is done by calling a register method on the ABC. The registered class then becomes a virtual subclass of the ABC, and will be recognized as such by functions like issubclass and isinstance, but it will not inherit any methods or attributes from the ABC.

Warning

Virtual subclasses do not inherit from their registered ABCs, and are not checked for conformance to the ABC interface at any time, not even when they are instantiated. It’s up to the subclass to actually implement all the methods needed to avoid runtime errors.

The register method is usually invoked as a plain function (see “Usage of register in Practice”), but it can also be used as a decorator. In Example 11-14, we use the decorator syntax and implement TomboList, a virtual subclass of Tombola depicted in Figure 11-5.

TomboList works as advertised, and the doctests that prove it are described in “How the Tombola Subclasses Were Tested”.

Figure 11-5. UML class diagram for the TomboList, a real subclass of list and a virtual subclass of Tombola

Example 11-14. tombolist.py: class TomboList is a virtual subclass of Tombola

from random import randrange

from tombola import Tombola

@Tombola.register  
class TomboList(list):  

    def pick(self):
        if self:  
            position = randrange(len(self))
            return self.pop(position)  
        else:
            raise LookupError('pop from empty TomboList')

    load = list.extend  

    def loaded(self):
        return bool(self)  

    def inspect(self):
        return tuple(sorted(self))

# Tombola.register(TomboList)  

Tombolist is registered as a virtual subclass of Tombola.

Tombolist extends list.

Tombolist inherits __bool__ from list, and that returns True if the list is not empty.

Our pick calls self.pop, inherited from list, passing a random item index.

Tombolist.load is the same as list.extend.

loaded delegates to bool.¹⁵

If you’re using Python 3.3 or earlier, you can’t use .register as a class decorator. You must use standard call syntax.

Note that because of the registration, the functions issubclass and isinstance act as if TomboList is a subclass of Tombola:

>>> from tombola import Tombola
>>> from tombolist import TomboList
>>> issubclass(TomboList, Tombola)
True
>>> t = TomboList(range(100))
>>> isinstance(t, Tombola)
True

However, inheritance is guided by a special class attribute named __mro__—the Method Resolution Order. It basically lists the class and its superclasses in the order Python uses to search for methods.¹⁶ If you inspect the __mro__ of TomboList, you’ll see that it lists only the “real” superclasses—list and object:

>>> TomboList.__mro__
(<class 'tombolist.TomboList'>, <class 'list'>, <class 'object'>)

Tombola is not in Tombolist.__mro__, so Tombolist does not inherit any methods from Tombola.

As I coded different classes to implement the same interface, I wanted a way to submit them all to the same suite of doctests. The next section shows how I leveraged the API of regular classes and ABCs to do it.

How the Tombola Subclasses Were Tested

The script I used to test the Tombola examples uses two class attributes that allow introspection of a class hierarchy:

__subclasses__()

Method that returns a list of the immediate subclasses of the class. The list does not include virtual subclasses.

_abc_registry

Data attribute—available only in ABCs—that is bound to a WeakSet with weak references to registered virtual subclasses of the abstract class.

To test all Tombola subclasses, I wrote a script to iterate over a list built from Tombola.__subclasses__() and Tombola._abc_registry, and bind each class to the name ConcreteTombola used in the doctests.

A successful run of the test script looks like this:

$ python3 tombola_runner.py
BingoCage        23 tests,  0 failed - OK
LotteryBlower    23 tests,  0 failed - OK
TumblingDrum     23 tests,  0 failed - OK
TomboList        23 tests,  0 failed - OK

The test script is Example 11-15 and the doctests are in Example 11-16.

import doctest

from tombola import Tombola

# modules to test
import bingo, lotto, tombolist, drum  

TEST_FILE = 'tombola_tests.rst'
TEST_MSG = '{0:16} {1.attempted:2} tests, {1.failed:2} failed - {2}'


def main(argv):
    verbose = '-v' in argv
    real_subclasses = Tombola.__subclasses__()  
    virtual_subclasses = list(Tombola._abc_registry)  

    for cls in real_subclasses + virtual_subclasses:  
        test(cls, verbose)


def test(cls, verbose=False):

    res = doctest.testfile(
            TEST_FILE,
            globs={'ConcreteTombola': cls},  
            verbose=verbose,
            optionflags=doctest.REPORT_ONLY_FIRST_FAILURE)
    tag = 'FAIL' if res.failed else 'OK'
    print(TEST_MSG.format(cls.__name__, res, tag))  


if __name__ == '__main__':
    import sys
    main(sys.argv)

Import modules containing real or virtual subclasses of Tombola for testing.

__subclasses__() lists the direct descendants that are alive in memory. That’s why we imported the modules to test, even if there is no further mention of them in the source code: to load the classes into memory.

Build a list from _abc_registry (which is a WeakSet) so we can concatenate it with the result of __subclasses__().

Iterate over the subclasses found, passing each to the test function.

The cls argument—the class to be tested—is bound to the name ConcreteTombola in the global namespace provided to run the doctest.

The test result is printed with the name of the class, the number of tests attempted, tests failed, and an 'OK' or 'FAIL' label.

The doctest file is Example 11-16.

==============
Tombola tests
==============

Every concrete subclass of Tombola should pass these tests.


Create and load instance from iterable::

    >>> balls = list(range(3))
    >>> globe = ConcreteTombola(balls)
    >>> globe.loaded()
    True
    >>> globe.inspect()
    (0, 1, 2)


Pick and collect balls::

    >>> picks = []
    >>> picks.append(globe.pick())
    >>> picks.append(globe.pick())
    >>> picks.append(globe.pick())


Check state and results::

    >>> globe.loaded()
    False
    >>> sorted(picks) == balls
    True


Reload::

    >>> globe.load(balls)
    >>> globe.loaded()
    True
    >>> picks = [globe.pick() for i in balls]
    >>> globe.loaded()
    False


Check that `LookupError` (or a subclass) is the exception
thrown when the device is empty::

    >>> globe = ConcreteTombola([])
    >>> try:
    ...     globe.pick()
    ... except LookupError as exc:
    ...     print('OK')
    OK


Load and pick 100 balls to verify that they all come out::

    >>> balls = list(range(100))
    >>> globe = ConcreteTombola(balls)
    >>> picks = []
    >>> while globe.inspect():
    ...     picks.append(globe.pick())
    >>> len(picks) == len(balls)
    True
    >>> set(picks) == set(balls)
    True


Check that the order has changed and is not simply reversed::

    >>> picks != balls
    True
    >>> picks[::-1] != balls
    True

Note: the previous 2 tests have a *very* small chance of failing
even if the implementation is OK. The probability of the 100
balls coming out, by chance, in the order they were inspect is
1/100!, or approximately 1.07e-158. It's much easier to win the
Lotto or to become a billionaire working as a programmer.

THE END

This concludes our Tombola ABC case study. In the next section, we’ll address how the register ABC function is used in the wild.

Usage of register in Practice

In Example 11-14, we used Tombola.register as a class decorator. Prior to Python 3.3, register could not be used like that—it had to be called as a plain function after the class definition, as suggested by the comment at the end of Example 11-14.

However, even if register can now be used as a decorator, it’s more widely deployed as a function to register classes defined elsewhere. For example, in the source code for the collections.abc module, the built-in types tuple, str, range, and memoryview are registered as virtual subclasses of Sequence like this:

Sequence.register(tuple)
Sequence.register(str)
Sequence.register(range)
Sequence.register(memoryview)

Several other built-in types are registered to ABCs in _collections_abc.py. Those registrations happen only when that module is imported, which is OK because you’ll have to import it anyway to get the ABCs: you need access to MutableMapping to be able to write isinstance(my_dict, MutableMapping).

We’ll wrap up this chapter by explaining a bit of ABC magic that Alex Martelli performed in “Waterfowl and ABCs”.

Geese Can Behave as Ducks

In his Waterfowl and ABCs essay, Alex shows that a class can be recognized as a virtual subclass of an ABC even without registration. Here is his example again, with an added test using issubclass:

>>> class Struggle:
...     def __len__(self): return 23
...
>>> from collections import abc
>>> isinstance(Struggle(), abc.Sized)
True
>>> issubclass(Struggle, abc.Sized)
True

Class Struggle is considered a subclass of abc.Sized by the issubclass function (and, consequently, by isinstance as well) because abc.Sized implements a special class method named __subclasshook__. See Example 11-17.

class Sized(metaclass=ABCMeta):

    __slots__ = ()

    @abstractmethod
    def __len__(self):
        return 0

    @classmethod
    def __subclasshook__(cls, C):
        if cls is Sized:
            if any("__len__" in B.__dict__ for B in C.__mro__):  
                return True  
        return NotImplemented  

If there is an attribute named __len__ in the __dict__ of any class listed in C.__mro__ (i.e., C and its superclasses)…

…return True, signaling that C is a virtual subclass of Sized.

Otherwise return NotImplemented to let the subclass check proceed.

If you are interested in the details of the subclass check, see the source code for the ABCMeta.__subclasscheck__ method in Lib/abc.py. Beware: it has lots of ifs and two recursive calls.

The __subclasshook__ adds some duck typing DNA to the whole goose typing proposition. You can have formal interface definitions with ABCs, you can make isinstance checks everywhere, and still have a completely unrelated class play along just because it implements a certain method (or because it does whatever it takes to convince a __subclasshook__ to vouch for it). Of course, this only works for ABCs that do provide a __subclasshook__.

Is it a good idea to implement __subclasshook__ in our own ABCs? Probably not. All the implementations of __subclasshook__ I’ve seen in the Python source code are in ABCs like Sized that declare just one special method, and they simply check for that special method name. Given their “special” status, you can be pretty sure that any method named __len__ does what you expect. But even in the realm of special methods and fundamental ABCs, it can be risky to make such assumptions. For example, mappings implement __len__, __getitem__, and __iter__ but they are rightly not considered a subtype of Sequence, because you can’t retrieve items using an integer offset and they make no guarantees about the ordering of items—except of course for OrderedDict, which preserves the insertion order, but does support item retrieval by offset either.

For ABCs that you and I may write, a __subclasshook__ would be even less dependable. I am not ready to believe that any class named Spam that implements or inherits load, pick, inspect, and loaded is guaranteed to behave as a Tombola. It’s better to let the programmer affirm it by subclassing Spam from Tombola, or at least registering: Tombola.register(Spam). Of course, your __subclasshook__ could also check method signatures and other features, but I just don’t think it’s worthwhile.

Chapter Summary

The goal of this chapter was to travel from the highly dynamic nature of informal interfaces—called protocols—visit the static interface declarations of ABCs, and conclude with the dynamic side of ABCs: virtual subclasses and dynamic subclass detection with __subclasshook__.

We started the journey by reviewing the traditional understanding of interfaces in the Python community. For most of the history of Python, we’ve been mindful of interfaces, but they were informal like the protocols from Smalltalk, and the official docs used language such as “foo protocol,” “foo interface,” and “foo-like object” interchangeably. Protocol-style interfaces have nothing to do with inheritance; each class stands alone when implementing a protocol. That’s what interfaces look like when you embrace duck typing.

With Example 11-3, we observed how deeply Python supports the sequence protocol. If a class implements __getitem__ and nothing else, Python manages to iterate over it, and the in operator just works. We then went back to the old FrenchDeck example of Chapter 1 to support shuffling by dynamically adding a method. This illustrated monkey patching and emphasized the dynamic nature of protocols. Again we saw how a partially implemented protocol can be useful: just adding __setitem__ from the mutable sequence protocol allowed us to leverage a ready-to-use function from the standard library: random.shuffle. Being aware of existing protocols lets us make the most of the rich Python standard library.

Alex Martelli then introduced the term “goose typing”¹⁷ to describe a new style of Python programming. With “goose typing,” ABCs are used to make interfaces explicit and classes may claim to implement an interface by subclassing an ABC or by registering with it—without requiring the strong and static link of an inheritance relationship.

The FrenchDeck2 example made clear the main drawbacks and advantages of explicit ABCs. Inheriting from abc.MutableSequence forced us to implement two methods we did not really need: insert and __delitem__. On the other hand, even a Python newbie can look at FrenchDeck2 and see that it’s a mutable sequence. And, as bonus, we inherited 11 ready-to-use methods from abc.MutableSequence (five indirectly from abc.Sequence).

After a panoramic view of existing ABCs from collections.abc in Figure 11-3, we wrote an ABC from scratch. Doug Hellmann, creator of the cool PyMOTW.com (Python Module of the Week) explains the motivation:

By defining an abstract base class, a common API can be established for a set of subclasses. This capability is especially useful in situations where someone less familiar with the source for an application is going to provide plug-in extensions…¹⁸

Putting the Tombola ABC to work, we created three concrete subclasses: two inheriting from Tombola, the other a virtual subclass registered with it, all passing the same suite of tests.

In concluding the chapter, we mentioned how several built-in types are registered to ABCs in the collections.abc module so you can ask isinstance(memoryview, abc.Sequence) and get True, even if memoryview does not inherit from abc.Sequence. And finally we went over the __subclasshook__ magic, which lets an ABC recognize any unregistered class as a subclass, as long as it passes a test that can be as simple or as complex as you like—the examples in the standard library merely check for method names.

To sum up, I’d like to restate Alex Martelli’s admonition that we should refrain from creating our own ABCs, except when we are building user-extensible frameworks—which most of the time we are not. On a daily basis, our contact with ABCs should be subclassing or registering classes with existing ABCs. Less often than subclassing or registering, we might use ABCs for isinstance checks. And even more rarely—if ever—we find occasion to write a new ABC from scratch.

After 15 years of Python, the first abstract class I ever wrote that is not a didactic example was the Board class of the Pingo project. The drivers that support different single board computers and controllers are subclasses of Board, thus sharing the same interface. In reality, although conceived and implemented as an abstract class, the pingo.Board class does not subclass abc.ABC as I write this.¹⁹ I intend to make Board an explicit ABC eventually—but there are more important things to do in the project.

Here is a fitting quote to end this chapter:

Although ABCs facilitate type checking, it’s not something that you should overuse in a program. At its heart, Python is a dynamic language that gives you great flexibility. Trying to enforce type constraints everywhere tends to result in code that is more complicated than it needs to be. You should embrace Python’s flexibility.²⁰

David Beazley and Brian Jones, Python Cookbook

Or, as technical reviewer Leonardo Rochael wrote: “If you feel tempted to create a custom ABC, please first try to solve your problem through regular duck-typing.”

Further Reading

Beazley and Jones’s Python Cookbook, 3rd Edition (O’Reilly) has a section about defining an ABC (Recipe 8.12). The book was written before Python 3.4, so they don’t use the now preferred syntax when declaring ABCs by subclassing from abc.ABC instead of using the metaclass keyword. Apart from this small detail, the recipe covers the major ABC features very well, and ends with the valuable advice quoted at the end of the previous section.

The Python Standard Library by Example by Doug Hellmann (Addison-Wesley), has a chapter about the abc module. It’s also available on the Web in Doug’s excellent PyMOTW — Python Module of the Week. Both the book and the site focus on Python 2; therefore, adjustments must be made if you are using Python 3. And for Python 3.4, remember that the only recommended ABC method decorator is @abstractmethod—the others were deprecated. The other quote about ABCs in the chapter summary is from Doug’s site and book.

When using ABCs, multiple inheritance is not only common but practically inevitable, because each of the fundamental collection ABCs—Sequence, Mapping, and Set—extends multiple ABCs (see Figure 11-3). Therefore, Chapter 12 is an important follow-up to this one.

PEP 3119 — Introducing Abstract Base Classes gives the rationale for ABCs, and PEP 3141 - A Type Hierarchy for Numbers presents the ABCs of the numbers module.

For a discussion of the pros and cons of dynamic typing, see Guido van Rossum’s interview to Bill Venners in “Contracts in Python: A Conversation with Guido van Rossum, Part IV”.

The zope.interface package provides a way of declaring interfaces, checking whether objects implement them, registering providers, and querying for providers of a given interface. The package started as a core piece of Zope 3, but it can and has been used outside of Zope. It is the basis of the flexible component architecture of large-scale Python projects like Twisted, Pyramid, and Plone. Lennart Regebro has a great introduction to zope.interface in “A Python Component Architecture”. Baiju M wrote an entire book about it: A Comprehensive Guide to Zope Component Architecture.