Creating your own objects in Python inevitably means implementing one or more of Python’s protocol methods—the magic methods whose names start and end with double underscores. These protocols are roughly the equivalent of interfaces in Python.
The lookup rules for the magic methods are slightly different from those of other attributes. Normal attribute lookup order[1] is instance -> class -> base classes. Magic method lookup goes directly to the class, skipping the instance. This means that you can’t override these methods by attaching a function directly to the instance (or through __getattr__); overriding has to happen at the class level. To provide these methods for classes themselves, they need to be implemented on the classes’ class, that is, its metaclass.[2]
This appendix is a reference to all the common magic methods.[3]
The object creation methods are called when a class is instantiated, as shown in table B.1.
Table B.1. Object creation
Method | Description |
|---|---|
| The object constructor. Responsible for creating new instances. It receives the class as the first argument, followed by all arguments passed in the instantiation call. To customize instance creation when inheriting from the built-in immutable types or .NET objects, you should override |
| The object initializer. Called after |
[a] Initializing immutable values is done in | |
The rich comparison methods are called during comparison operations involving the ==, !=, <, <=, >, and >= operators. They are also called by operations that implicitly involve comparisons, such as sorting a list. All these methods should return a Boolean. They can also return the NotImplemented singleton to indicate that the operation is not implemented for the two values being compared.
In Python there is no comparison fallback. To support all the comparison operators you need to implement all the comparison methods, shown in table B.2.
Classes of immutable objects that implement the comparison methods should also implement __hash__. Objects that compare equal should have the same hash.
Table B.2. Comparison methods
Method | Description |
|---|---|
| Called for equality operations ( |
| Called for inequality operations ( |
| Called for less-than operations ( |
| Called for less-than or equals operations ( |
| Called for greater-than operations ( |
| Called for greater-than or equals operations ( |
Table B.3 provides a selection of important methods that don’t fit into any other category.
Table B.3. Miscellaneous methods
Method | Description |
|---|---|
| Called to implement truth value testing and the built-in function |
| Available on classes only. Returns a list of all subclasses of the class. |
| Called when an object is called as a function. |
| Called when an object is used as a dictionary key, for set membership, and by the built-in function |
| Called when a Python object is about to be destroyed. |
Python has two protocols for objects that support subscription (indexing): the sequence protocol and the mapping protocol. The sequence protocol[4] is used for sequences (like lists, tuples, and strings) that are indexed with an integer. The mapping protocol[5] is used for types (like dictionaries) that map keys to values. Both sequences and mapping types are typically iterable, with sequences iterating over their values and mapping types iterating over their keys.
Both the mapping and sequence protocols use __getitem__, __setitem__, and __delitem__ to fetch, set, and delete items. The mapping and sequence protocol methods are shown in table B.4.
Table B.4. Mapping and sequence protocol methods
Method | Description |
|---|---|
| Called when an item or slice is fetched. If an item is indexed using slice syntax, then the index will be a slice[a] object. Classes may raise a |
| Called when an item or slice is set. |
| Called when an item or slice is deleted. |
| Called by the built-in function |
| Called by the |
| Called when iterating over an object or by the built-in function |
| Containers may implement this method to provide an optimized reverse iterator for instances. If this method is available, it is called by the built-in function |
| New in Python 2.5. This method is called on subclasses of the built-in dictionary ( |
| Iterators can implement this as an optimization so that Python can preallocate space for them. IronPython doesn’t use it (yet, anyway), but if you see this method, now you know what it is for. |
[a] The | |
Generator expressions are a form of iteration that we haven’t covered elsewhere. We’ve shown how list comprehensions allow for an extremely concise form of iteration and filter in a single expression:
result = [x for x in iterable if some_condition(x)]
This is comparable to LINQ over objects, which was introduced in C# 3.0 (.NET 3.5). There is an alternate form of list comprehensions called generator expressions, which arrived in Python 2.4. Instead of square brackets, they use parentheses to surround the expression. The major difference is that instead of being evaluated immediately, they are evaluated lazily. A generator expression returns a generator object (the same kind of object returned by a generator function). Like other iterators, this can be consumed by iterating over it, or you can consume individual items by calling the next method.
Because they are lazily evaluated, we can combine them:
from os import listdir
python_files = (f for f in listdir('.') if f.endswith('.py'))
first_lines = ((f, open(f).readline()) for f in python_files))
names_to_first_line = dict(first_lines)None of the generator expressions are actually executed until the final dict call. A nice side effect is that they look nicer when used in places that take any iterable:
total = sum(val for val in some_list if val > 0)
Both list comprehensions and generator expressions can take advantage of another feature new to Python 2.5: conditional expressions (also known as ternary expressions).
Conditional expressions have the basic syntax
X if Y else Z
This evaluates Y, and if it is True it returns X—otherwise, it returns Z. We can use this in list comprehensions and generator expressions:
generator = (f(a) if test(a) else g(a) for a in some_list)
This generator expression tests each member of some_list, yielding f(a) if test(a) returns True and g(a) if test(a) returns False.
Advanced tools for working with iterators and generators can be found in the standard library module itertools; see http://docs.python.org/library/itertools.html.
There are several ways that objects can be converted to strings, both implicitly and explicitly. Table B.5 lists the protocol methods that do this.
Table B.5. String conversion methods
Description | |
|---|---|
| Called by the built-in |
| Called by the built-in |
| Normally called by the built-in |
In Python you can completely customize attribute access on objects for fetching, setting, and deleting. Table B.6 shows the attribute access methods.
Table B.6. Attribute access methods
Method | Description |
|---|---|
| If this method is implemented, then it will be called with the attribute name when an attribute that doesn’t exist on the object is requested. If fetching the attribute fails, it should raise an |
| If this method is implemented, it will be called whenever an attribute is set on an instance. |
| If this method is implemented, it will be called whenever an attribute is deleted. |
| This method is always called for attribute access (new-style classes only). It is an important part of the descriptor protocol. It should return the attribute or raise an |
Numeric types are expected to implement a whole host of different methods in order to support the full range of numeric operations and conversions between the different built-in types. In addition, other types are free to implement individual methods to support a subset of the operations. For example, strings and lists both implement the addition methods so that you can add strings and lists together. Strings implement the modulo operator for string interpolation.
Table B.7 shows the methods that implement the binary arithmetic operations (+, -, *, /, //, true division, %, divmod(), pow(), **, <<, >>, &, ^, |). If one of those methods does not support the operation with the supplied arguments, it should return NotImplemented.
Table B.7. Binary arithmetic operations
Method | Description |
|---|---|
| Called for addition operations ( |
| Called for subtraction operations ( |
| Called for multiplication ( |
| Called for floor division ( |
| Called for division ( |
| Called for division when true division is on. |
| Called for the modulo operator ( |
| Called by the built-in |
| Called for power operations ( |
| Called for left-shift operations ( |
| Called for right-shift operations ( |
| Called for binary |
| Called for binary exclusive |
| Called for binary |
These methods all have an additional two forms.
If they are on the right-hand side of a binary operation, then the right-hand version of the operator method will be called. This has the same name as the normal operator but with an r prepended. For example, the right-hand addition operator method is __radd__.
All of the operators (except for the divmod function) also have an in-place equivalent. The in-place operators are +=, -=, /=, //=, %=, **=, <<=, >>=, &=, ^=, and |=. The in-place operator methods have the same name as the standard operator methods with an i prepended. For example, the in-place addition operator method is __iadd__. Because in-place operations rebind the name being operated on, mutable values that implement in-place operators should return self from the in-place operator methods (immutable values can return a new value as normal). If an in-place operator is used on an object that implements the standard operator method but not the in-place operator method, then the standard operator method will be used instead. Table B.8 shows the unary arithmetic operations.
The methods in table B.9 are used to convert objects between numeric types, usually by built-in functions.
Table B.9. Type conversion
Method | Description |
|---|---|
| Called by the built-in function |
| Called by the built-in function |
| Called by the built-in function |
| Called by the built-in function |
| Called by the built-in function |
| Called by the built-in function |
| Called whenever Python needs an integer object (such as in slicing). Must return an integer ( |
The Python with statement (new in Python 2.5)[6] supports the concept of a runtime context. This allows you to perform operations on an object, with a context manager to handle closing or disposing of the object once the operations are complete. This is useful for committing database transactions (or rolling them back in case of an error), synchronizing threads, or working with files.
The context management protocol involves implementing a pair of methods, __enter__ and __exit__. Table B.10 describes these methods.
Table B.10. The context management protocol
Method | Description |
|---|---|
| Enter the runtime context of the context manager. The object(s) returned by this method are bound to the names in the |
| Exit the runtime context associated with the context manager. If the Context managers should not re-raise exceptions; instead they should return a Boolean indicating whether or not they have handled the exception (a return value of |
Python defines these methods on files, so that you can use the following pattern:
with open(filename) as handle:
data = handle.read()
...When the with statement is executed,[7] __enter__ is called and the file handle returned is bound to the name handle. When the code inside the with block is exited (or an exception is raised), __exit__ is called, which closes the file.
The standard library module contextlib[8] is extremely useful for working with the context management protocol. contextlib.closing is a convenient decorator for creating context managers that automatically close a resource. When closing is insufficient, contextlib.contextmanager is the simplest way to create a context manager. When working with decorators in general, functools[9] is another extremely useful module. For example, functools.wrap preserves the name (__name__) and docstrings of decorated functions.
The descriptor protocol is an aspect of Python that is regarded as deep black magic, perhaps even more so than metaclasses. Like metaclasses, the rules are very simple once you understand them; it is just not very often that you need to explicitly implement them. The descriptor protocol is how properties, class methods, and static methods are implemented.
When an object is fetched from a class or instance dictionary, the __getattribute__ method checks for these methods and invokes them appropriately to fetch the object. This allows you to customize what happens when an object is accessed as a class or instance attribute. Table B.11 shows the descriptor protocol methods.
Table B.11. Descriptor protocol methods
Method | Description |
|---|---|
| Called to get attributes from an instance or a class ( |
| Called to set the attribute on an instance of the |
| Called to delete an attribute on an instance of the |
Python has a built-in property descriptor that allows you to invoke code when an attribute is requested on an instance. It doesn’t have an equivalent classproperty allowing you to create static properties as you can in C#. Using the descriptor protocol we can implement classproperty in a few lines of code:
class classproperty(object):
def __init__(self, function):
self.function = function
def __get__(self, instance, owner):
return self.function(owner)classproperty can be used as a decorator. When it decorates a method (which should receive the class as its only argument), it replaces the method in the class with a descriptor. The original method is stored as a function object, the function attribute on the descriptor.
When the descriptor is fetched from the class, its __get__ method is invoked, and the stored function is called with the class passed in. Despite its mystery-shrouded reputation, the descriptor protocol can be used very simply to add new language features.
As well as magic methods, many Python objects have magic attributes. These are attributes whose names start and end with double underscores and that have a special meaning to the Python interpreter. Table B.12 lists the most common of these attributes along with their description.
Table B.12. Python magic attributes
Name | Description |
|---|---|
| Most Python objects have a |
|
|
| When |
| A reference to the class for all class instances (everything in Python is an instance of a class). |
| A class attribute referencing a tuple of the base classes for the class. |
| An attribute (string) on modules, functions, and classes. It is assignable, allowing decorated functions to retain the same name as the function they wrap (very useful for tracebacks from decorated functions). |
| An optional attribute (list of strings) for modules. If available, this lists all the names that will be exported by the module when |
| A module attribute (string) containing the path the module was loaded from on the filesystem. This attribute does not exist for built-in modules. |
| A class, function, and method attribute (string) with the name of the module the object was defined in. |
| If this is defined as a class attribute, then the callable assigned will be called to create the class instead of type. It can also be used as a module-level attribute, and all old-style classes (without an explicit alternative metaclass or inheriting from a class with a metaclass) will use the callable assigned as their metaclass. A quick way to convert all old-style classes in a module into new-style classes is to add |
| A global variable (Boolean) indicating whether the interpreter is being run with optimizations on ( |
There are also a handful of modules and one built-in function whose names start and end with double underscores. Table B.13 lists the magic modules and functions.
Table B.13. Magic functions and modules
Name | Description |
|---|---|
| The Python module that contains all the built-in functions and exceptions. You can import this and patch it to affect the global scope of all modules. You can effectively create new built-ins, or point existing names to different objects, at runtime. |
| This module is the main script (as a module) that the interpreter is executing. |
| Python packages consist of directories containing an |
| A built-in function that provides programmatic access to the import machinery. The function signature is See the Python documentation on built-in functions[a] for more details. |
[1] This order assumes the usual caveat that the descriptor protocol makes the full lookup rules more complex. Section B.9 of this appendix describes the descriptor protocol.
[2] The IronPython generics support using the Array[int] syntax, which is implemented by adding a __getitem__ method to the type metaclass.
[3] This is only a summary; for full details refer to the Python documentation at http://docs.python.org/index.html.
[4] For additional methods commonly defined by mutable sequence types see http://docs.python.org/lib/typesseq-mutable.html.
[5] For additional methods commonly defined by mapping types see http://docs.python.org/lib/typesmapping.html.
[6] Using with in Python 2.5 (IronPython 2) requires the future import: from __future__ import with_statement.
[7] The with statement does not create a new scope.