13
WRITE LESS, CODE MORE
In this final chapter, I’ve compiled a few of Python’s more advanced features that I use to write better code. These are not limited to the Python Standard Library. We’ll cover how to make your code compatible with both Python 2 and 3, how to create a Lisp-like method dispatcher, how to use context managers, and how to create a boilerplate for classes with the attr module.
Using six for Python 2 and 3 Support
As you likely know, Python 3 breaks compatibility with Python 2 and shifts things around. However, the basics of the language haven’t changed between versions, which makes it possible to implement forward and backward compatibility, creating a bridge between Python 2 and Python 3.
Lucky for us, this module already exists! It’s called six—because 2 × 3 = 6.
The six module provides the useful six.PY3 variable, which is a Boolean that indicates whether you are running Python 3 or not. This is the pivot variable for any of your codebase that has two versions: one for Python 2 and one for Python 3. However, be careful not to abuse it; scattering your codebase with if six.PY3 is going to make it difficult for people to read and understand.
When we discussed generators in “Generators” on page 121, we saw that Python 3 has a great property whereby iterable objects are returned instead of lists in various built-in functions, such as map() or filter(). Python 3 therefore got rid of methods like dict.iteritems(), which was the iterable version of dict.items() in Python 2, in favor of making dict.items() return an iterator rather than a list. This change in methods and their return types can break your Python 2 code.
The six module provides six.iteritems() for such cases, which can be used to replace Python 2–specific code like this:
for k, v in mydict.iteritems():
print(k, v)
Using six, you would replace the mydict.iteritems() code with Python 2- and 3-compliant code like so:
import six
for k, v in six.iteritems(mydict):
print(k, v)
And voilà, both Python 2 and Python 3 compliance achieved in a snap! The six.iteritems() function will use either dict.iteritems() or dict.items() to return a generator, depending on the version of Python you’re using. The six module provides a lot of similar helper functions that can make it easy to support multiple Python versions.
Another example would be the six solution to the raise keyword, whose syntax is different between Python 2 and Python 3. In Python 2, raise will accept multiple arguments, but in Python 3, raise accepts an exception as its only argument and nothing else. Writing a raise statement with two or three arguments in Python 3 would result in a SyntaxError.
The six module provides a workaround here in the form of the function six.reraise(), which allows you to reraise an exception in whichever version of Python you use.
Strings and Unicode
Python 3’s enhanced ability to handle advanced encodings solved the string and unicode issues of Python 2. In Python 2, the basic string type is str, which can only handle basic ASCII strings. The type unicode, added later in Python 2.5, handles real strings of text.
In Python 3, the basic string type is still str, but it shares the properties of the Python 2 unicode class and can handle advanced encodings. The bytes type replaces the str type for handling basic character streams.
The six module again provides functions and constants, such as six.u and six.string_types, to handle the transition. The same compatibility is provided for integers, with six.integer_types that will handle the long type that has been removed from Python 3.
Handling Python Modules Moves
In the Python Standard Library, some modules have moved or have been renamed between Python 2 and 3. The six module provides a module called six.moves that handles a lot of these moves transparently.
For example, the ConfigParser module from Python 2 has been renamed to configparser in Python 3. Listing 13-1 shows how code can be ported and made compatible with both major Python versions using six.moves:
from six.moves.configparser import ConfigParser
conf = ConfigParser()
Listing 13-1: Using six.moves to use ConfigParser() with Python 2 and Python 3
You can also add your own moves via six.add_move to handle code transitions that six doesn’t handle natively.
In the event that the six library doesn’t cover all your use cases, it may be worth building a compatibility module encapsulating six itself, thereby ensuring that you will be able to enhance the module to fit future versions of Python or dispose of (part of) it when you want to stop supporting a particular version of the language. Also note that six is open source and that you can contribute to it rather than maintain your own hacks!
The modernize Module
Lastly, there is a tool named modernize that uses the six module to “modernize” your code by porting it to Python 3, rather than simply converting Python 2 syntax to Python 3 syntax. This provides support for both Python 2 and Python 3. The modernize tool helps to get your port off to a strong start by doing most of the grunt work for you, making this tool a better choice than the standard 2to3 tool.
Using Python Like Lisp to Make a Single Dispatcher
I like to say that Python is a good subset of the Lisp programming language, and as time passes, I find that this is more and more true. The PEP 443 proves that point: it describes a way to dispatch generic functions in a similar manner to what the Common Lisp Object System (CLOS) provides.
If you’re familiar with Lisp, this won’t be news to you. The Lisp object system, which is one of the basic components of Common Lisp, provides a simple, efficient way to define and handle method dispatching. I’ll show you how generic methods work in Lisp first.
Creating Generic Methods in Lisp
To begin with, let’s define a few very simple classes, without any parent classes or attributes, in Lisp:
(defclass snare-drum ()
())
(defclass cymbal ()
())
(defclass stick ()
())
(defclass brushes ()
())
This defines the classes snare-drum, cymbal, stick, and brushes without any parent class or attributes. These classes compose a drum kit, and we can combine them to play sound. For this, we define a play() method that takes two arguments and returns a sound as a string:
(defgeneric play (instrument accessory)
(:documentation "Play sound with instrument and accessory."))
This only defines a generic method that isn’t attached to any class and so cannot yet be called. At this stage, we’ve only informed the object system that the method is generic and might be called with two arguments named instrument and accessory. In Listing 13-2, we’ll implement versions of this method that simulate playing our snare drum.
(defmethod play ((instrument snare-drum) (accessory stick))
"POC!")
(defmethod play ((instrument snare-drum) (accessory brushes))
"SHHHH!")
(defmethod play ((instrument cymbal) (accessory brushes))
"FRCCCHHT!")
Listing 13-2: Defining generic methods in Lisp, independent of classes
Now we’ve defined concrete methods in code. Each method takes two arguments: instrument, which is an instance of snare-drum or cymbal, and accessory, which is an instance of stick or brushes.
At this stage, you should see the first major difference between this system and the Python (or similar) object systems: the method isn’t tied to any particular class. The methods are generic, and they can be implemented for any class.
Let’s try it. We can call our play() method with some objects:
* (play (make-instance 'snare-drum) (make-instance 'stick))
"POC!"
* (play (make-instance 'snare-drum) (make-instance 'brushes))
"SHHHH!"
As you can see, which function is called depends on the class of the arguments—the object system dispatches the function calls to the right function for us, based on the type of the arguments we pass. If we call play() with an object whose classes do not have a method defined, an error will be thrown.
In Listing 13-3, the play() method is called with a cymbal and a stick instance; however, the play() method has never been defined for those arguments, so it raises an error.
* (play (make-instance 'cymbal) (make-instance 'stick))
debugger invoked on a SIMPLE-ERROR in thread
#<THREAD "main thread" RUNNING {1002ADAF23}>:
There is no applicable method for the generic function
#<STANDARD-GENERIC-FUNCTION PLAY (2)>
when called with arguments
(#<CYMBAL {1002B801D3}> #<STICK {1002B82763}>).
Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.
restarts (invokable by number or by possibly abbreviated name):
0: [RETRY] Retry calling the generic function.
1: [ABORT] Exit debugger, returning to top level.
((:METHOD NO-APPLICABLE-METHOD (T)) #<STANDARD-GENERIC-FUNCTION PLAY (2)>
#<CYMBAL {1002B801D3}> #<STICK {1002B82763}>) [fast-method]
Listing 13-3: Calling a method with an unavailable signature
CLOS provides even more features, such as method inheritance or object-based dispatching, rather than using classes. If you’re really curious about the many features CLOS provides, I suggest reading “A Brief Guide to CLOS” by Jeff Dalton (http://www.aiai.ed.ac.uk/~jeff/clos-guide.html) as a starting point.
Generic Methods with Python
Python implements a simpler version of this workflow with the singledispatch() function, which has been distributed as part of the functools module since Python 3.4. In versions 2.6 to 3.3, the singledispatch() function is provided through the Python Package Index; for those eager to try it out, just run pip install singledispatch.
Listing 13-4 shows a rough equivalent of the Lisp program we built in Listing 13-2.
import functools
class SnareDrum(object): pass
class Cymbal(object): pass
class Stick(object): pass
class Brushes(object): pass
@functools.singledispatch
def play(instrument, accessory):
raise NotImplementedError("Cannot play these")
➊ @play.register(SnareDrum)
def _(instrument, accessory):
if isinstance(accessory, Stick):
return "POC!"
if isinstance(accessory, Brushes):
return "SHHHH!"
raise NotImplementedError("Cannot play these")
@play.register(Cymbal)
def _(instrument, accessory):
if isinstance(accessory, Brushes):
return "FRCCCHHT!"
raise NotImplementedError("Cannot play these")
Listing 13-4: Using singledispatch to dispatch method calls
This listing defines our four classes and a base play() function that raises NotImplementedError, indicating that by default we don’t know what to do.
We then write a specialized version of the play() function for a specific instrument, the SnareDrum ➊. This function checks which accessory type has been passed and returns the appropriate sound or raises NotImplementedError again if the accessory isn’t recognized.
If we run the program, it works as follows:
>>> play(SnareDrum(), Stick())
'POC!'
>>> play(SnareDrum(), Brushes())
'SHHHH!'
>>> play(Cymbal(), Stick())
Traceback (most recent call last):
NotImplementedError: Cannot play these
>>> play(SnareDrum(), Cymbal())
NotImplementedError: Cannot play these
The singledispatch module checks the class of the first argument passed and calls the appropriate version of the play() function. For the object class, the first defined version of the function is always the one that is run. Therefore, if our instrument is an instance of a class that we did not register, this base function will be called.
As we saw in the Lisp version of the code, CLOS provides a multiple dispatcher that can dispatch based on the type of any of the arguments defined in the method prototype, not just the first one. The Python dispatcher is named singledispatch for a good reason: it only knows how to dispatch based on the first argument.
In addition, singledispatch offers no way to call the parent function directly. There is no equivalent of the Python super() function; you’ll have to use various tricks to bypass this limitation.
While Python is improving its object system and dispatch mechanism, it still lacks a lot of the more advanced features that something like CLOS provides out of the box. That makes encountering singledispatch in the wild pretty rare. It’s still interesting to know it exists, as you may end up implementing such a mechanism yourself at some point.
Context Managers
The with statement introduced in Python 2.6 is likely to remind old-time Lispers of the various with-* macros that are often used in that language. Python provides a similar-looking mechanism with the use of objects that implement the context management protocol.
If you’ve never used the context management protocol, here’s how it works. The code block contained inside the with statement is surrounded by two function calls. The object being used in the with statement determines the two calls. Those objects are said to implement the context management protocol.
Objects like those returned by open() support this protocol; that’s why you can write code along these lines:
with open("myfile", "r") as f:
line = f.readline()
The object returned by open() has two methods: one called __enter__ and one called __exit__. These methods are called at the start of the with block and at the end of it, respectively.
A simple implementation of a context object is shown in Listing 13-5.
class MyContext(object):
def __enter__(self):
pass
def __exit__(self, exc_type, exc_value, traceback):
pass
Listing 13-5: A simple implementation of a context object
This implementation does not do anything, but it is valid and shows the signature of the methods that need to be defined to provide a class following the context protocol.
The context management protocol might be appropriate to use when you identify the following pattern in your code, where it is expected that a call to method B must always be done after a call to A:
Call method A.
Execute some code.
Call method B.
The open() function illustrates this pattern well: the constructor that opens the file and allocates a file descriptor internally is method A. The close() method that releases the file descriptor corresponds to method B. Obviously, the close() function is always meant to be called after you instantiate the file object.
It can be tedious to implement this protocol manually, so the contextlib standard library provides the contextmanager decorator to make implementation easier. The contextmanager decorator should be used on a generator function. The __enter__ and __exit__ methods will be dynamically implemented for you based on the code that wraps the yield statement of the generator.
In Listing 13-6, MyContext is defined as a context manager.
import contextlib
@contextlib.contextmanager
def MyContext():
print("do something first")
yield
print("do something else")
with MyContext():
print("hello world")
Listing 13-6: Using contextlib.contextmanager
The code before the yield statement will be executed before the with statement body is run; the code after the yield statement will be executed once the body of the with statement is over. When run, this program outputs the following:
do something first
hello world
do something else
There are a couple of things to handle here though. First, it’s possible to yield something inside our generator that can be used as part of the with block.
Listing 13-7 shows how to yield a value to the caller. The keyword as is used to store this value in a variable.
import contextlib
@contextlib.contextmanager
def MyContext():
print("do something first")
yield 42
print("do something else")
with MyContext() as value:
print(value)
Listing 13-7: Defining a context manager yielding a value
Listing 13-7 shows how to yield a value to the caller. The keyword as is used to store this value in a variable. When executed, the code outputs the following:
do something first
42
do something else
When using a context manager, you might need to handle exceptions that can be raised within the with code block. This can be done by surrounding the yield statement with a try...except block, as shown in Listing 13-8.
import contextlib
@contextlib.contextmanager
def MyContext():
print("do something first")
try:
yield 42
finally:
print("do something else")
with MyContext() as value:
print("about to raise")
➊ raise ValueError("let's try it")
print(value)
Listing 13-8: Handling exceptions in a context manager
Here, a ValueError is raised at the beginning of the with code block ➊; Python will propagate this error back to the context manager, and the yield statement will appear to raise the exception itself. We enclose the yield statement in try and finally to make sure the final print() is run.
When executed, Listing 13-8 outputs the following:
do something first
about to raise
do something else
Traceback (most recent call last):
File "<stdin>", line 3, in <module>
ValueError: let's try it
As you can see, the error is raised back to the context manager, and the program resumes and finishes execution because it ignored the exception using a try...finally block.
In some contexts, it can be useful to use several context managers at the same time, for example, when opening two files at the same time to copy their content, as shown in Listing 13-9.
with open("file1", "r") as source:
with open("file2", "w") as destination:
destination.write(source.read())
Listing 13-9: Opening two files at the same time to copy content
That being said, since the with statement supports multiple arguments, it’s actually more efficient to write a version using a single with, as shown in Listing 13-10.
with open("file1", "r") as source, open("file2", "w") as destination:
destination.write(source.read())
Listing 13-10: Opening two files at the same time using only one with statement
Context managers are extremely powerful design patterns that help to ensure your code flow is always correct, no matter what exception might occur. They can help to provide a consistent and clean programming interface in many situations in which code should be wrapped by other code and contextlib.contextmanager.
Less Boilerplate with attr
Writing Python classes can be cumbersome. You’ll often find yourself repeating just a few patterns because there are no other options. One of the most common examples, as illustrated in Listing 13-11, is when initializing an object with a few attributes passed to the constructor.
class Car(object):
def __init__(self, color, speed=0):
self.color = color
self.speed = speed
Listing 13-11: Common class initialization boilerplate
The process is always the same: you copy the value of the argument passed to the __init__ function to a few attributes stored in the object. Sometimes you’ll also have to check the value that is passed, compute a default, and so on.
Obviously, you also want your object to be represented correctly if printed, so you’ll have to implement a __repr__ method. There’s a chance some of your classes are simple enough to be converted to dictionaries for serialization. Things become even more complicated when talking about comparison and hashability (the ability to use hash on an object and store it in a set).
In reality, most Python programmers do none of this, because the burden of writing all those checks and methods is too heavy, especially when you’re not always sure you’ll need them. For example, you might find that __repr__ is useful in your program only that one time you’re trying to debug or trace it and decide to print objects in the standard output—and no other times.
The attr library aims for a straightforward solution by providing a generic boilerplate for all your classes and generating much of the code for you. You can install attr using pip with the command pip install attr. Get ready to enjoy!
Once installed, the attr.s decorator is your entry point into the wonderful world of attr. Use it above a class declaration and then use the function attr.ib() to declare attributes in your classes. Listing 13-12 shows a way to rewrite Listing 13-11 using attr.
import attr
@attr.s
class Car(object):
color = attr.ib()
speed = attr.ib(default=0)
Listing 13-12: Using attr.ib() to declare attributes
When declared this way, the class automatically gains a few useful methods for free, such as __repr__, which is called to represent objects when they are printed on stdout in the Python interpreter:
>>> Car("blue")
Car(color='blue', speed=0)
This output is cleaner than the default that __repr__ would have printed:
<__main__.Car object at 0x104ba4cf8>.
You can also add more validation on your attributes by using the validator and converter keyword arguments.
Listing 13-13 shows how the attr.ib() function can be used to declare an attribute with some constraints.
import attr
@attr.s
class Car(object):
color = attr.ib(converter=str)
speed = attr.ib(default=0)
@speed.validator
def speed_validator(self, attribute, value):
if value < 0:
raise ValueError("Value cannot be negative")
Listing 13-13: Using attr.ib() with its converter argument
The converter argument manages the conversion of whatever is passed to the constructor. The validator() function can be passed as an argument to attr.ib() or used as a decorator, as shown in Listing 13-13.
The attr module provides a few validators of its own (for example, attr.validators.instance_of() to check the type of the attribute), so be sure to check them out before wasting your time building your own.
The attr module also provides tweaks to make your object hashable so it can be used in a set or a dictionary key: just pass frozen=True to attr.s() to make the class instances immutable.
Listing 13-14 shows how using the frozen parameter changes the behavior of the class.
>>> import attr
>>> @attr.s(frozen=True)
... class Car(object):
... color = attr.ib()
...
>>> {Car("blue"), Car("blue"), Car("red")}
{Car(color='red'), Car(color='blue')}
>>> Car("blue").color = "red"
attr.exceptions.FrozenInstanceError
Listing 13-14: Using frozen=True
Listing 13-14 shows how using the frozen parameter changes the behavior of the Car class: it can be hashed and therefore stored in a set, but objects cannot be modified anymore.
In summary, attr provides the implementation for a ton of useful methods, thereby saving you from writing them yourself. I highly recommend leveraging attr for its efficiency when building your classes and modeling your software.
Summary
Congratulations! You made it to the end of the book. You’ve just upped your Python game and have a better idea of how to write efficient and productive Python code. I hope you enjoyed reading this book as much as I enjoyed writing it.
Python is a wonderful language and can be used in many different fields, and there are many more areas of Python that we did not touch on in this book. But every book needs an ending, right?
I highly recommend profiting from open source projects by reading the available source code out there and contributing to it. Having your code reviewed and discussed by other developers is often a great way to learn.
Happy hacking!