Managing Complexity, Solving Business Problems

We both work for MADE.com, a European e-commerce company that sells furniture online; there, we apply the techniques in this book to build distributed systems that model real world business problems. Our example domain is the first system Bob built for MADE, and this book is an attempt to write down all the stuff we have to teach new programmers when they join one of our teams.

MADE.com operates a global supply chain of freight partners and manufacturers. To try and keep costs low, we try to optimize the delivery of stock to our warehouses so that we don’t have unsold goods lying around the place.

Ideally, the sofa that you want to buy will arrive in port on the very day that you decide to buy it, and we’ll ship it straight to your house without ever storing it. Getting the timing right is a tricky balancing act when goods take 3 months to arrive by container ship. Along the way things get broken, or water damaged; storms cause unexpected delays, logistics partners mishandle goods, paperwork goes missing, customers change their minds and amend their orders, and so on.

We solve those problems by building intelligent software that represents the kind of operations taking place in the real world so that we can automate as much of the business as possible.

Why Python?

If you’re reading this book, we probably don’t need to convince you that Python is great, so the real question is “Why does the Python community need a book like this?”

The answer is about Python’s popularity and maturity—although Python is probably the world’s fastest-growing programming language, and nearing the top of the absolute popularity tables, it’s only just starting to take on the kinds of problems that the C# and Java world have been working on for years. Startups become real businesses, web apps and scripted automations are becoming (whisper it) enterprise software.

In the Python world, we often quote the Zen of Python:¹ “there should be one—and preferably only one—obvious way to do it.” Unfortunately, as project size grows, the most obvious way of doing things isn’t always the way that helps you manage complexity and evolving requirements.

None of the techniques and patterns we’re going to discuss in this book are new, but they are mostly new to the Python world. And this book won’t be a replacement for the classics in the field like Eric Evans’ Domain-Driven Design or Martin Fowler’s Patterns of Enterprise Application Architecture (both of which we often refer to and encourage you to go and read).

But all the classic code examples in the literature do tend to be written in Java or C++/#, and if you’re a Python person and haven’t used either of those languages in a long time (or indeed ever), those code listings can be quite… trying. There’s a reason the latest edition of that other classic text, Refactoring, is in JavaScript.

TDD, DDD and Event-Driven Architecture

In order of notoriety, we know of three tools for managing complexity:

1. Test-Driven Development (TDD) helps us to build code that is correct, and enables us to refactor or add new features, without fear of regression. But it can be hard to get the best out of our tests: how do we make sure that they run as fast as possible? That we get as much coverage and feedback from fast, dependency-free unit tests, and have the minimum number of slower, flakey end-to-end tests?

2. Domain-Driven Design (DDD) asks us to focus our efforts on building a good model of the business domain, but how to we make sure that our models aren’t encumbered with infrastructure concerns and become hard to change?

3. Loosely coupled (micro)-services integrated via messages (sometimes called Reactive Microservices) are a well established answer to managing complexity across multiple applications or business domains. But it’s not always obvious how to make them fit with the established tools of the Python world—Flask, Django, Celery and so on.

Our aim with this book is to introduce several classic architectural patterns, and show how they support TDD, DDD and event-driven microservices. We hope it will serve as a reference for implementing them in a Pythonic way, and that people can use it as a first step towards further research in this field.

Who Should Read This Book

Here are a few things we assume about you, dear reader.

• We assume you’ve been close to some reasonably complex Python applications.

• We assume you’ve seen some of the pain that comes with trying to manage that complexity.

• We do not assume that you already know anything about DDD, or any of the classic application architecture patterns.

We structure our explorations of architectural patterns around an example app, building it up chapter by chapter. We use test-driven development (TDD) at work, so we tend to show listings of tests first, followed by implementation. If you’re not used to working test-first, it may feel a little strange at the beginning, but we hope you’ll soon get used to seeing code “being used,” i.e. from the outside, before you see how it’s built on the inside.

We use some specific Python (version 3) frameworks and technologies, like Flask, SQLAlchemy, and Pytest, as well as Docker and Redis. If you’re already familiar with them, that won’t hurt, but we don’t think it’s required. One of our main aims with this book is to build an architecture where specific technology choices become minor implementation details.

A note to our Early Release Readers

If you’re reading this book via the Early Release Programme, or if you’ve found your way to a free online version, we want to ask you one, hugely important favor. Please, please send feedback. It might be something as simple as a quick shout out on twitter (we are @hjwp and @bob_the_mighty), a quick email via [email protected], or opening up issues or PRs on GitHub. Even just hearing that people are reading it, with no actual feedback, cheers us on and validates that we have an audience. And if you have time to tell us your impressions, then we absolutely want to hear them, in any form. Tell us if we’re doing a good job explaining things, tell us where you found things confusing, tell us where we’re overexplaining. Tell us about your own problems, and whether or not this book is helping; if you’re struggling to see how this would apply “in the real world”, tell us about that especially! We’re hoping to spend a good bit of time in the final chapter talking about “how do I get there from here”, so the more that’s inspired by real-life stories, the better.

Most of all though, thanks for reading, and thanks for bearing with us while the book is still in its draft form!

A Brief Overview of What You’ll Learn

The book is divided into two parts; here’s a look at the topics we’ll cover, and which chapters they live in.

Example Code and Coding Along

You’re reading a book, but you’ll probably agree with us when we say that the best way to learn about code is to code. We learned most of what we know from pairing with people, writing code with them, and learning by doing, and we’d like to recreate that experience as much as possible for you in this book.

As a result, we’ve structured the book around a single example project (although we do sometimes throw in other examples), which we build up as we go, and the narrative of the book is as if you’re pairing with us as we go, and we’re explaining what we’re doing and why at each step.

But to really get to grips with these patterns, you need to mess about with the code and actually get a feel for how it works. You’ll find all the code on GitHub; each chapter has its own branch. You can find a list of them at github.com/cosmicpython/code/branches/all

Here are three different ways you might code along with the book:

• Start your own repo and try and build up the app as we do, following the examples from listings in the book, and occasionally looking to our repo for hints. A word of warning however, if you’ve read Harry’s previous book and coded along with that, you’ll find there is much more to figure out on your own; you may need to lean pretty heavily on the working versions on GitHub.

• Try to apply these each pattern, chapter-by-chapter, to your own (preferably small/toy) project, and see if you can make it work for your use case. This is high-risk / high-reward (and high effort besides!). It may take quite some work to get things working for the specifics of your project, but on the other hand you’re likely to learn the most.

• For less effort, in each chapter we’ll outline an “exercise for the reader,” and point you to a GitHub location where you can download some partially-finished code for the chapter with a few missing parts to write yourself.

Particularly if you’re intending to apply some of these patterns in your own projects, then working through a simple example is a great way to get some safe practice.

The code (and the online version of the book) is licensed under a Creative Commons CC-By-ND license, which means you are free to copy and share it with anyone you like, for non-commercial purposes, as long as you give attribution. If you want to re-use any of the content from this book and you have any worries about the license, contact O’Reilly at [email protected].

Conventions Used in This Book

The following typographical conventions are used in this book:

Italic

Indicates new terms, URLs, email addresses, filenames, and file extensions.

Constant width

Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords.

Constant width bold

Shows commands or other text that should be typed literally by the user.

Constant width italic

Shows text that should be replaced with user-supplied values or by values determined by context.

O’Reilly Online Learning

Our unique network of experts and innovators share their knowledge and expertise through books, articles, conferences, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, please visit http://oreilly.com.

For more information, please visit http://oreilly.com/safari.

How to Contact O’Reilly

Please address comments and questions concerning this book to the publisher:

• O’Reilly Media, Inc.
• 1005 Gravenstein Highway North
• Sebastopol, CA 95472
• 800-998-9938 (in the United States or Canada)
• 707-829-0515 (international or local)
• 707-829-0104 (fax)

We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at http://www.oreilly.com/catalog/catalogpage.

Email [email protected] to comment or ask technical questions about this book.

For more information about our books, courses, conferences, and news, see our website at http://www.oreilly.com.

Find us on Facebook: http://facebook.com/oreilly

Follow us on Twitter: http://twitter.com/oreillymedia

Watch us on YouTube: http://www.youtube.com/oreillymedia

Acknowledgments

To our Tech Reviewers, David Seddon, Ed Jung and Hynek Schlawack: we absolutely did not deserve you. You were all incredibly dedicated, conscientious and rigorous. Each one of you is immensely smart, and your different points of view were both useful and complementary with each other. Thank you from the bottom of our hearts.

Gigantic thanks also to our Early Release readers for their comments and suggestions: Ian Cooper, Abdullah Ariff, Jonathan Meier, Gil Gonçalves, Matthieu Choplin, Ben Judson, James Gregory, Łukasz Lechowicz, Clinton Roy, Vitorino Araújo, Susan Goodbody, Josh Harwood, Daniel Butler, Liu Haibin, Jimmy Davies, Ignacio Vergara Kausel, Gaia Canestrani, Renne Rocha, pedroabi, Ashia Zawaduk and many more, our apologies if we’ve missed your name on this list.

Super-mega-thanks to our editor Corbin Collins, for his gentle chivvying, and for being a tireless advocate of the reader; this book is immeasurably improved thanks to you.

Any errors remaining in the book are our own, naturally