INTRODUCTION

If you’re reading this, the odds are good you’ve been working with Python for some time already. Maybe you learned it using some tutorials, delved into some existing programs, or started from scratch. Whatever the case, you’ve hacked your way into learning it. That’s exactly how I got familiar with Python up until I started working on big open source projects 10 years ago.

It is easy to think that you know and understand Python once you’ve written your first program. The language is that simple to grasp. However, it takes years to master it and to develop a deep comprehension of its advantages and shortcomings.

When I started Python, I built my own Python libraries and applications on a “garage project” scale. Things changed once I started working with hundreds of developers on software that thousands of users rely on. For example, the OpenStack platform—a project I contribute to—represents over 9 million lines of Python code, which collectively needs to be concise, efficient, and scalable to the needs of whatever cloud computing application its users require. When you have a project of this size, things like testing and documentation absolutely require automation, or else they won’t get done at all.

I thought I knew a lot about Python before working on projects of this scale—a scale I could hardly imagine when I started out—but I’ve learned a lot more. I’ve also had the opportunity to meet some of the best Python hackers in the industry and learn from them. They’ve taught me everything from general architecture and design principles to various helpful tips and tricks. Through this book, I hope to share the most important things I’ve learned so that you can build better Python programs—and build them more efficiently, too!

The first version of this book, The Hacker’s Guide to Python, came out in 2014. Now Serious Python is the fourth edition, with updated and entirely new contents. I hope you enjoy it!

Who Should Read This Book and Why

This book is intended for Python coders and developers who want to take their Python skills to the next level.

In it, you’ll find methods and advice that will help you get the most out of Python and build future-proof programs. If you’re already working on a project, you’ll be able to apply the techniques discussed right away to improve your current code. If you’re starting your first project, you’ll be able to create a blueprint with the best practice.

I’ll introduce you to some Python internals to give you a better understanding of how to write efficient code. You will gain a greater insight into the inner workings of the language that will help you understand problems or inefficiencies.

The book also provides applicable battle-tested solutions to problems such as testing, porting, and scaling Python code, applications, and libraries. This will help you avoid making the mistakes that others have made and discover strategies that will help you maintain your software in the long run.

About This Book

This book is not necessarily designed to be read from front to back. You should feel free to skip to sections that interest you or are relevant to your work. Throughout the book, you’ll find a wide range of advice and practical tips. Here’s a quick breakdown of what each chapter contains.

Chapter 1 provides guidance about what to consider before you undertake a project, with advice on structuring your project, numbering versions, setting up automated error checking, and more. At the end there’s an interview with Joshua Harlow.

Chapter 2 introduces Python modules, libraries, and frameworks and talks a little about how they work under the hood. You’ll find guidance on using the sys module, getting more from the pip package manager, choosing the best framework for you, and using standard and external libraries. There’s also an interview with Doug Hellmann.

Chapter 3 gives advice on documenting your projects and managing your APIs as your project evolves even after publication. You’ll get specific guidance on using Sphinx to automate certain documentation tasks. Here you’ll find an interview with Christophe de Vienne.

Chapter 4 covers the age-old issue of time zones and how best to handle them in your programs using datetime objects and tzinfo objects.

Chapter 5 helps you get your software to users with guidance on distribution. You’ll learn about packaging, distributions standards, the distutils and setuptools libraries, and how to easily discover dynamic features in a package using entry points. Nick Coghlan is interviewed.

Chapter 6 advises you on unit testing with best-practice tips and specific tutorials on automating unit tests with pytest. You’ll also look at using virtual environments to increase the isolation of your tests. The interview is with Robert Collins.

Chapter 7 digs into methods and decorators. This is a look at using Python for functional programming, with advice on how and when to use decorators and how to create decorators for decorators. We’ll also dig into static, class, and abstract methods and how to mix the three for a more robust program.

Chapter 8 shows you more functional programming tricks you can implement in Python. This chapter discusses generators, list comprehensions, functional functions and common tools for implementing them, and the useful functools library.

Chapter 9 peeks under the hood of the language itself and discusses the abstract syntax tree (AST) that is the inner structure of Python. We’ll also look at extending flake8 to work with the AST to introduce more sophisticated automatic checks into your programs. The chapter concludes with an interview with Paul Tagliamonte.

Chapter 10 is a guide to optimizing performance by using appropriate data structures, defining functions efficiently, and applying dynamic performance analysis to identify bottlenecks in your code. We’ll also touch on memoization and reducing waste in data copies. You’ll find an interview with Victor Stinner.

Chapter 11 tackles the difficult subject of multithreading, including how and when to use multithreading as opposed to multiprocessing and whether to use event-oriented or service-oriented architecture to create scalable programs.

Chapter 12 covers relational databases. We’ll take a look at how they work and how to use PostgreSQL to effectively manage and stream data. Dimitri Fontaine is interviewed.

Finally, Chapter 13 offers sound advice on a range of topics: making your code compatible with both Python 2 and 3, creating functional Lisp-like code, using context managers, and reducing repetition with the attr library.