This book

My objective in writing this book is to share knowledge from different parts of the community and lessons learned over 15 years of writing Python code for a living. It will help you to be productive, both with the core language and add-on libraries. You will learn how to effectively use features of the language that are not strictly essential to be a productive programmer, such as asynchronous programming, packaging, and testing.

However, this book is aimed at people who want to write code, not people who are looking to understand deep magics. I will not delve too far into subjects that involve implementation details of Python. You will not be expected to grok¹ Python C extensions, metaclasses, or algorithms to benefit from this book.

Substantive code samples are shown as numbered listings, and the accompanying code for this book includes electronic versions of these listings. Some of these listings also have output shown directly beneath, rather than separately as a numbered figure.

The accompanying code for this book is also where you’ll find copies of the full codebase for the example on a chapter-by-chapter basis, as well as helper code for the exercises. In general, I would recommend that you follow along with the code by checking out the Git repository from the book’s website or the code distribution and changing to the relevant branch for the chapter you’re reading.

As well as listings, I show some console sessions. When lines which are formatted like code begin with >, that indicates that a shell session is being shown. These sections cover commands to be run from your operating system’s terminal. Any that involve >>> are demonstrating a Python console session and should be run from within a Python interpreter.

The example

This book’s example is that of a general-purpose data aggregator. If you work in DevOps, then it is very likely you use a program of this sort to track the resource utilization of servers. Alternatively, as a web developer, you may use something like this for statistics aggregation from different deployments of the same system. Some scientists use similar methods, for example, for aggregating the findings of air-quality sensors distributed across a city. It isn’t something that every developer will need to build, but it is a problem space that is familiar to many developers.

It has been picked not just because it’s a common task, but because it allows us to explore many of the subjects we want to cover in a natural, unified way. You will be able to follow the complete example perfectly well using any modern computer running any modern operating system,² without purchasing any additional hardware. You may find you get more out of some of the examples if you have additional computers to act as remote data sources.

I will be using a Raspberry Pi Zero equipped with some aftermarket sensors for my examples. This platform is widely available for approximately 5 US dollars and provides lots of interesting data. There are commercial sensor add-ons available from many Raspberry Pi stockists.

Although I’ll be recommending things specific to the Raspberry Pi to make following the examples easier, this book is not about the Internet of Things or the Raspberry Pi itself. It’s a means to an end; you should feel comfortable to adapt the examples to fit tasks that are more relevant to your interests if you like. Any of the similar problems mentioned earlier would follow the same design process.

Choice of topics

The topics covered by this book have been chosen to shine a light on a variety of different aspects of Python programming. All are underused or under-understood by the Python community as a whole, and none are things likely to be taught as a matter of course to beginners. That’s not to say that they are necessarily complex or hard to understand (although some certainly are), but they are techniques that I believe all Python programmers should be familiar with, even if they choose not to use them.

Chapter 1 will introduce you to different ways of approaching the writing of very simple programs in Python and, in particular, will cover Jupyter notebooks and an introduction to the use of the Python debugger. Although both are relatively well-known tools, many people are proficient in the use of one but not both. It will also cover ways of approaching the writing of command-line interfaces and some useful third-party libraries to support succinct command-line tool development.

Chapter 2 will cover tools that help you identify mistakes in your code, such as automated testing and linting tools. These tools all make it easier to write code that you can be confident in, whether it’s a large codebase, one that you rarely need to edit, or one that will garner contributions from third parties. The tools covered here are all ones I would recommend; however, the focus will be on understanding their advantages and disadvantages. You may have used one or more of these tools, and you may have opinions on whether some of them are appropriate to use. This chapter will help you understand the trade-offs to help you make informed decisions.

Chapter 3 covers code packaging and dependency distribution in Python. These are key features for writing applications that can be distributed to others and for designing deployment systems that work reliably. We will use this to convert our stand-alone script into an installable application.

Chapter 4 introduces plugin architectures. This is a powerful feature; it’s not uncommon for people who learn them to try and apply them everywhere, which means people can be wary of teaching them. For our example, a plugin architecture is a natural fit. It also covers some advanced techniques for command-line tools that can make debugging plugin-based systems easier.

Chapter 5 covers web interfaces and techniques such as decorators and closures to write complex functions. These techniques are idiomatic in Python but hard to express in many other programming languages. It also covers the appropriate use of abstract base classes. It’s common for people to advise against using ABCs because of the tendency of people who learn them to want to use them everywhere. There are definite advantages to a restrained use of ABCs in particular circumstances, especially when combined with some of the tools from Chapter 2.

Chapter 6 expands our example with another major component, the aggregation server that collects the data. This chapter also demonstrates some of the most useful third-party libraries you will use as a Python programmer, such as “requests.”

Chapter 7 covers threading and asynchronous programming in Python. Threading is often the source of subtle bugs. Asynchronous code can be used for similar tasks, but it is an idiom that many Python programmers haven’t used because the program behaves quite differently to synchronous programming. This chapter focuses on the real-world use of concurrency to achieve a result, rather than demonstrations of a simple example or the limits of what asynchronous programming can do. The objective is working code that is usable in the real world and a thorough understanding of the trade-offs, not a stand-alone technology demonstration.

Chapter 8 goes further with asynchronous programming, adding in the testing of asynchronous code and the various libraries that exist to write code that deals with external tools (such as databases) in an async context. We will also look briefly at some advanced techniques for writing good APIs that are helpful for asynchronous programming, like context managers and context variables.

Chapter 9 sees us return to Jupyter to use its features for data visualization and easy user interaction. We will look at how to use our asynchronous functions with widgets in Jupyter notebooks as well as advanced use of iterators and ways of implementing complex data types.

Chapter 10 details how to make Python code faster, using different types of caching and for which situations they are an appropriate choice. It covers benchmarking individual Python functions in your applications and how to interpret the results to find the reasons for slowdown.

Chapter 11 extends some of the concepts we’ve visited earlier in the book to handle faults more gracefully. We’ll look at ways that our plugin architecture can be modified to allow for handling errors seamlessly while retaining full backward compatibility, and we’ll take a closer look at designing processes that handle errors that they encounter.

In Chapter 12 , the final chapter, we use Python’s iterator and coroutine features to enhance the dashboards we’ve developed with features that aren’t passive data gatherers but actively introspect the data we’ve gathered, allowing us to build multistep analysis flows.

Python version

At the time of writing, the current release of Python is 3.8, and as such the examples in this book are being tested against 3.8 and first development versions of Python 3.9. I do not recommend using older versions. Very few code samples in this book do not work on Python 3.7 or Python 3.6.

You will need Python pip installed to follow along with this book. It should already be installed on your system if you have Python installed. Some operating systems intentionally remove pip from their default installations of Python, in which case you’ll need to install it using the operating system’s package manager explicitly. This is common on Debian-based systems, where it can be installed with sudo apt install python3-pip. On other operating systems, use python -m ensurepip --upgrade to have Python find the latest version of pip itself or find instructions specific to your operating system.

Electronic versions of code samples and errata are available from the publisher and the book’s website at https://advancedpython.dev . This should be your first port of call if you encounter any problems working through this book.