7

Event-Driven Programming

In the previous chapter, we discussed various concurrency implementation models that are available in Python. To better explain the concept of concurrency, we used the following definition:

Two events are concurrent if neither can causally affect the other.

We often think about events as ordered points in time that happen one after another, often with some kind of cause-effect relationship. But, in programming, events are understood a bit differently. They are not necessarily "things that happen." Events in programming are more often understood as independent units of information that can be processed by the program. And that very notion of events is a real cornerstone of concurrency.

Concurrent programming is a programming paradigm for processing concurrent events. And there is a generalization of that paradigm that deals with the bare concept of events—no matter whether they are concurrent or not. This approach to programming, which treats programs as a flow of events, is called event-driven programming.

It is an important paradigm because it allows you to easily decouple even large and complex systems. It helps in defining clear boundaries between independent components and improves isolation between them.

In this chapter, we will cover the following topics:

• What exactly is event-driven programming?
• Various styles of event-driven programming
• Event-driven architectures

After reading this chapter, you will know the common techniques of event-driven programming and how to extrapolate these techniques to event-driven architectures. You'll also be able to easily identify problems that can be solved using event-driven programs.

Technical requirements

The following are Python packages that are mentioned in this chapter that you can download from PyPI:

• flask
• blinker

Information on how to install packages is included in Chapter 2, Modern Python Development Environments.

The code files for this chapter can be found at https://github.com/PacktPublishing/Expert-Python-Programming-Fourth-Edition/tree/main/Chapter%207.

In this chapter, we will build a small application using a Graphical User Interface (GUI) package named tkinter. To run the tkinter examples, you will need the Tk library for Python. It should be available by default with most Python distributions, but on some operating systems, it will require additional system packages to be installed. On Debian-based Linux distributions, this package is usually named python3-tk. Python installed though official macOS and Windows installers should already come with the Tk library.

What exactly is event-driven programming?

Event-driven programming focuses on the events (often called messages) and their flow between different software components. In fact, it can be found in many types of software. Historically, event-based programming is the most common paradigm for software that deals with direct human interaction. It means that it is a natural paradigm for GUIs. Anywhere the program needs to wait for some human input, that input can be modeled as events or messages. In such a framing, an event-driven program is often just a collection of event/message handlers that respond to human interaction.

Events of course don't have to be a direct result of user interaction. The architecture of any web application is also event-driven. Web browsers send requests to web servers on behalf of the user, and these requests are often processed as separate interaction events. Some of the requests will indeed be the result of direct user input (for example, submitting a form or clicking on a link), but don't always have to be. Many modern applications can asynchronously synchronize information with a web server without any interaction from the user, and that communication happens silently without the user noticing.

In summary, event-driven programming is a general way of coupling software components of various sizes and happens on various levels of software architecture. Depending on the scale and type of software architecture we're dealing with, it can take various forms:

• It can be a concurrency model directly supported by a semantic feature of a given programming language (for example, async/await in Python)
• It can be a way of structuring application code with event dispatchers/handlers, signals, and so on
• It can be a general inter-process or inter-service communication architecture that allows for the coupling of independent software components in a larger system

Let's discuss how event-driven programming is different from asynchronous programming in the next section.

Event-driven != asynchronous

Although event-driven programming is a paradigm that is extremely common for asynchronous systems, it doesn't mean that every event-driven application must be asynchronous. It also doesn't mean that event-driven programming is suited only for concurrent and asynchronous applications. Actually, the event-driven approach is extremely useful, even for decoupling problems that are strictly synchronous and definitely not concurrent.

Consider, for instance, database triggers, which are available in almost every relational database system. A database trigger is a stored procedure that is executed in response to a certain event that happens in the database. This is a common building block of database systems that, among other things, allows the database to maintain data consistency in scenarios that cannot be easily modeled with the mechanism of database constraints.

For instance, the PostgreSQL database distinguishes three types of row-level events that can occur in either a table or a view:

• INSERT: Emitted when a new row is inserted
• UPDATE: Emitted when an existing row is updated
• DELETE: Emitted when an existing row is deleted

In the case of table rows, triggers can be defined to be executed either BEFORE or AFTER a specific event. So, from the perspective of event-procedure coupling, we can treat each AFTER/BEFORE trigger as a separate event. To better understand this, let's consider the following example of database triggers in PostgreSQL:

CREATE TRIGGER before_user_update
    BEFORE UPDATE ON users
    FOR EACH ROW
    EXECUTE PROCEDURE check_user();
CREATE TRIGGER after_user_update
    AFTER UPDATE ON users
    FOR EACH ROW
    EXECUTE PROCEDURE log_user_update();

In the preceding example, we have two triggers that are executed when a row in the users table is updated. The first one is executed before a real update occurs and the second one is executed after the update is done. This means that BEFORE UPDATE and AFTER UPDATE events are casually dependent and cannot be handled concurrently. On the other hand, similar sets of events occurring on different rows from different sessions can still be concurrent, although that will depend on multiple factors (transaction or not, the isolation level, the scope of the trigger, and so on). This is a valid example of a situation where data modification in a database system can be modeled with event-based processing although the system as a whole isn't fully asynchronous.

In the next section, we'll take a look at event-driven programming in GUIs.

Event-driven programming in GUIs

GUIs are what many people think of when they hear the term "event-driven programming." Event-driven programming is an elegant way of coupling user input with code in GUIs because it naturally captures the way people interact with graphical interfaces. Such interfaces often present the user with a plethora of components to interact with, and that interaction is almost always nonlinear.

In complex interfaces, this interaction is often modeled through a collection of events that can be emitted by the user from different interface components.

The concept of events is common to most user interface libraries and frameworks, but different libraries use different design patterns to achieve event-driven communication. Some libraries even use other notions to describe their architecture (for example, signals in the Qt library). Still, the general pattern is almost always the same—every interface component (often called a widget) can emit events upon interaction. Other components receive those events either by subscription or by directly attaching themselves to emitters as their event handlers. Depending on the GUI library, events can just be plain named signals stating that something has happened (for example, "widget A was clicked"), or they can be more complex messages containing additional information about the nature of the interaction. Such messages can for instance contain the specific key that has been pressed or the position of the mouse when an event was emitted.

We will discuss the differences of actual design patterns later in the Various styles of event-driven programming section, but first let's take a look at the example Python GUI application that can be created with the use of the built-in tkinter module:

sudo apt-get install python3-tk

The following GUI application displays a single Python Zen button. When the button is clicked, the application will open a new window containing the Zen of Python text that was imported from the this module. The this module is a Python easter egg. After import, it prints on standard output the 19 aphorisms that are the guiding principles of Python's design.

import this
from tkinter import Tk, Frame, Button, LEFT, messagebox
rot13 = str.maketrans(
    "ABCDEFGHIJKLMabcdefghijklmNOPQRSTUVWXYZnopqrstuvwxyz",
    "NOPQRSTUVWXYZnopqrstuvwxyzABCDEFGHIJKLMabcdefghijklm"
)
def main_window(root: Tk):
    frame = Frame(root)
    frame.pack()
    zen_button = Button(frame, text="Python Zen", command=show_zen)
    zen_button.pack(side=LEFT)
def show_zen():
    messagebox.showinfo("Zen of Python", this.s.translate(rot13))
if __name__ == "__main__":
    root = Tk()
    main_window(root)
    root.mainloop()

Our script starts with imports and the definition of a simple string translation table. It is necessary because the text of the Zen of Python is encrypted inside the this module using an ROT13 letter substitution cipher (also known as a Caesar cipher). It is a simple encryption algorithm that shifts every letter in the alphabet by 13 positions.

The binding of events happens directly in the Button widget constructor:

Button(frame, text="Python Zen", command=show_zen)

The command keyword argument defines the event handler that will be executed when the user clicks the button. In our example, we have provided the show_zen() function, which will display the decoded text of the Zen of Python in a separate message box.

Most GUI frameworks work in a similar manner—you rarely work with raw keyboard and mouse inputs, but instead attach your commands/callbacks to higher-level events such as the following:

• Checkbox state change
• Button clicked
• Option selected
• Window closed

In the next section, we'll take a look at event-driven communication.

Event-driven communication

Event-driven programming is a very common practice for building distributed network applications. With event-driven programming, it is easier to split complex systems into isolated components that have a limited set of responsibilities, and because of that, it is especially popular in service-oriented and microservice architectures. In such architectures, the flow of events happens not between classes or functions living inside of a single computer process, but between many networked services. In large and distributed architectures, the flow of events between services is usually coordinated using special communication protocols (for example, AMQP and ZeroMQ), often with the help of dedicated services acting as message brokers. We will discuss some of these solutions later in the Event-driven architectures section.

However, you don't need to have a formalized way of coordinating events, nor a dedicated event-handling service, to consider your networked code an event-based application. Actually, if you take a more detailed look at a typical Python web application, you'll notice that most Python web frameworks have many things in common with GUI applications. Let's, for instance, consider a simple web application that was written using the Flask microframework:

import this
from flask import Flask
app = Flask(__name__)
rot13 = str.maketrans(
    "ABCDEFGHIJKLMabcdefghijklmNOPQRSTUVWXYZnopqrstuvwxyz",
    "NOPQRSTUVWXYZnopqrstuvwxyzABCDEFGHIJKLMabcdefghijklm"
)
def simple_html(body):
    return f"""
    <!DOCTYPE html>
    <html lang="en">
      <head>
        <meta charset="utf-8">
        <title>Book Example</title>
      </head>
      <body>
        {body}
      </body>
    </html>
    """
@app.route('/')
def hello():
    return simple_html("<a href=/zen>Python Zen</a>")
@app.route('/zen')
def zen():
    return simple_html(
        "<br>".join(this.s.translate(rot13).split("
"))
    )
if __name__ == '__main__':
    app.run()

If you compare the preceding listing with the example of the tkinter application from the previous section, you'll notice that, structurally, they are very similar. Specific routes (paths) of HTTP requests translate to dedicated handlers. If we consider our application to be event-driven, then the request path can be treated as a binding between a specific event type (for example, a link being clicked) and the action handler. Similar to events in GUI applications, HTTP requests can contain additional data about interaction context. This information is, of course, structured. The HTTP protocol defines multiple request methods (for example, POST, GET, PUT, and DELETE) and a few ways to transfer additional data (query string, request body, and headers).

The user does not communicate with our application directly as they would when using a GUI, but instead they use a web browser as their interface. This also makes it somewhat similar to traditional graphical applications, as many cross-platform user interface libraries (such as Tcl/Tk, Qt, and GTK+) are in fact just proxies between the application and the user's operating system's windowing APIs. So, in both cases, we deal with communication and events flowing through multiple system layers. It is just that, in web applications, layers are more evident and communication is always explicit.

Modern web applications often provide interactive interfaces based on JavaScript. They are very often built using event-driven frontend frameworks that communicate asynchronously with an application backend service through backend APIs. This only emphasizes the event-driven nature of web applications.

We've seen so far that depending on the use case, event-driven programming can be used in multiple types of applications. It can also take different forms. In the next section, we will go through the three major styles of event-driven programming.

Various styles of event-driven programming

As we already stated, event-driven programming can be implemented at various levels of software architecture. It is also often applied to very specific software engineering areas, such as networking, system programming, and GUI programming. So, event-driven programming isn't a single cohesive programming approach, but rather a collection of diverse patterns, tools, and algorithms that form a common paradigm that concentrates on programming around the flow of events.

Due to this, event-driven programming exists in different flavors and styles. The actual implementations of event-driven programming can be based on different design patterns and techniques. Some of these event-driven techniques and tools don't even use the term event. Despite this variety, we can easily identify three major event-driven programming styles that are the foundation for more concrete patterns:

• Callback-based style: This concentrates on the act of coupling event emitters with their handlers in a one-to-one fashion. In this style, event emitters are responsible for defining actions that will happen when a specific event occurs.
• Subject-based style: This concentrates on the one-to-many subscription of events originating at specific emitters. In this style, emitters are subjects of a subscription. Whoever wants to receive events needs to subscribe directly to the source of events.
• Topic-based style: This concentrates on the types of events rather than their origin and destination. In this style, event emitters are not necessarily aware of event subscribers and vice versa. Instead, communication happens through independent event channels—topics—that anyone can publish to or subscribe to.

In the next sections, we will do a brief review of the three major styles of event-driven programming that you may encounter when programming in Python.

Callback-based style

The callback-based style of event programming is one of the most common styles of event-driven programming. In this style, objects that emit events are the ones that are responsible for defining their event handlers. This means a one-to-one or (at most) many-to-one relationship between event emitters and event handlers.

This style of event-based programming is the dominant pattern among GUI frameworks and libraries. The reason for this is simple—it really captures how both users and programmers think about user interfaces. For every action we do, whether we toggle a switch, press a button, or tick a checkbox, we do it usually with a clear and single purpose.

We've already seen an example of callback-based event-driven programming and discussed an example of a graphical application written using the tkinter library (see the Event-driven programming in GUIs section). Let's recall one line from that application listing:

zen_button = Button(root, text="Python Zen", command=show_zen)

The previous instantiation of the Button class defines that the show_zen() function should be called whenever the button is pressed. Our event is implicit. The show_zen() callback (in tkinter, callbacks are called commands) does not receive any object that would describe the event behind the call. This makes sense because the responsibility of attaching event handlers lies closer to the event emitter. Here, it is the zen_button instance. The event handler is barely concerned about the actual origin of the event.

In some implementations of callback-based event-driven programming, the actual binding between event emitters and event handlers is a separate step that can be performed after the event emitter is initialized. This style of binding is possible in tkinter too, but only for raw user interaction events. The following is an updated excerpt of the previous tkinter application that uses this style of event binding:

def main_window(root):
    frame = Frame(root)
    zen_button = Button(frame, text="Python Zen")
    zen_button.bind("<ButtonRelease-1>", show_zen)
    zen_button.pack(side=LEFT)
def show_zen(event):
    messagebox.showinfo("Zen of Python", this.s.translate(rot13))

In the preceding example, the event is no longer implicit. Because of that, the show_zen() callback must be able to accept the event object. The event instance contains basic information about user interaction, such as the position of the mouse cursor, the time of the event, and the associated widget. What is important to remember is that this type of event binding is still unicast. This means that one event from one object can be bound to only one callback. It is possible to attach the same handler to multiple events and/or multiple objects, but a single event that comes from a single source can be dispatched to only one callback. Any attempt to attach a new callback using the bind() method will override the old one.

The unicast nature of callback-based event programming has obvious limitations as it requires the tight coupling of application components. The inability to attach multiple fine-grained handlers to single events often means that every handler is usually specialized to serve a single emitter and cannot be bound to objects of a different type.

The subject-based style is a style that reverses the relationship between event emitters and event handlers. Let's take a look at it in the next section.

Subject-based style

The subject-based style of event programming is a natural extension of unicast callback-based event handling. In this style of programming, event emitters (subjects) allow other objects to subscribe/register for notifications about their events. In practice, this is very similar to the callback-based style, as event emitters usually store a list of functions or methods to call when some new event happens.

In subject-based event programming, the focus moves from the event to the subject (the event emitter). The most common product of this style is the observer design pattern.

In short, the observer design pattern consists of two classes of objects—observers and subjects (sometimes called observables). The Subject instance is an object that maintains a list of Observer instances that are interested in what happens to the Subject instance. In other words, Subject is an event emitter and Observer instances are event handlers.

If we would like to define common interfaces for the observer design pattern, we could do that by creating the following abstract base classes:

from abc import ABC, abstractmethod
class ObserverABC(ABC):
    @abstractmethod
    def notify(self, event): ...
class SubjectABC(ABC):
    @abstractmethod
    def register(self, observer: ObserverABC): ...

The instances of the ObserverABC subclasses will be the event handlers. We will be able to register them as observers of subject events using the register() method of the SubjectABC subclass instances. What is interesting about this design is that it allows for multicast communication between components. A single observer can be registered in multiple subjects and a single subject can have multiple subscribers.

To better understand the potential of this mechanism, let's build a more practical example. We will try to build a naïve implementation of a grep-like utility. It will be able to recursively scan through the filesystem looking for files containing some specified text. We will use the built-in glob module for the recursive traversal of the filesystem and the re module for matching regular expressions.

The core of our program will be the Grepper class, which will be a subclass of SubjectABC. Let's start by defining the base scaffolding for the registration and notification of observers:

class Grepper(SubjectABC):
    _observers: list[ObserverABC]
    def __init__(self):
        self._observers = []
    def register(self, observer: ObserverABC):
        self._observers.append(observer)
    
    def notify_observers(self, event):
        for observer in self._observers:
            observer.notify(event)

The implementation is fairly simple. The __init__() function initializes an empty list of observers. Every new Grepper instance will start with no observers. The register() method was defined in the SubjectABC class as an abstract method, so we are obliged to provide the actual implementation of it. It is the only method that is able to add new observers to the subject state. Last is the notify_observers() method, which will pass the specified event to all registered observers.

Since our scaffolding is ready, we are now able to define the Grepper.grep() method, which will do the actual work:

from glob import glob
import os.path
import re
class Grepper(SubjectABC):
    ...
    def grep(self, path: str, pattern: str):
        r = re.compile(pattern)
        for item in glob(path, recursive=True):
            if not os.path.isfile(item):
                continue
            try:
                with open(item) as f:
                    self.notify_observers(("opened", item))
                    if r.findall(f.read()):
                        self.notify_observers(("matched", item))
            finally:
                self.notify_observers(("closed", item))

The glob(pattern, recursive=True) function allows us to do recursive filesystem path names search with "glob" patterns. We will use it to iterate over files in the location designated by the user. For searching through actual file contents, we use regular expressions provided in the re module.

As we don't know at this point what the possible observer use cases are, we decided to emit three types of events:

• "opened": Emitted when a new file has been opened
• "matched": Emitted when Grepper found a match in a file
• "closed": Emitted when a file has been closed

Let's save that class in a file named observers.py and finish it with the following fragment of code, which initializes the Grepper class instance with input arguments:

import sys
if __name__ == "__main__":
    if len(sys.argv) != 3:
        print("usage: program PATH PATTERN")
        sys.exit(1)
    grepper = Grepper()
    grepper.grep(sys.argv[1], sys.argv[2])

Our observers.py program is now able to search through files, but it won't output any visible output yet. If we would like to find out which file contents match our expression, we could change it by creating a subscriber that is able to respond to"matched" events. The following is an example of a Presenter subscriber that simply prints the name of the file associated with a "matched" event:

class Presenter(ObserverABC):
    def notify(self, event):
        event_type, file = event
        if event_type == "matched":
            print(f"Found in: {file}")

And here is how it could be attached to the Grepper class instance:

if __name__ == "__main__":
    if len(sys.argv) != 3:
        print("usage: program PATH PATTERN")
        sys.exit(1)
    grepper = Grepper()
    grepper.register(Presenter())
    grepper.grep(sys.argv[1], sys.argv[2])

If we would like to find out which of the examples from this chapter's code bundle contain the substring grep, we could use the following program invocation:

$ python observers.py 'Chapter 7/**' grep
Found in: Chapter 7/04 - Subject-based style/observers.py

The main benefit of this design pattern is extensibility. We can easily extend our application capabilities by introducing new observers. If, for instance, we would like to trace all opened files, we could create a special Auditor subscriber that logs all opened and closed files. It could be as simple as the following:

class Auditor(ObserverABC):
    def notify(self, event):
        event_type, file = event
        print(f"{event_type:8}: {file}")

Moreover, observers aren't tightly coupled to the subject and have only minimal assumptions on the nature of events delivered to them. If you decide to use a different matching mechanism (for instance, the fnmatch module for glob-like patterns instead of regular expressions from the re module), you can easily reuse existing observers by registering them to a completely new subject class.

Subject-based event programming allows for the looser coupling of components and thus increases application modularity. Unfortunately, the change of focus from events to subjects can become a burden. In our example, observers will be notified about every event emitted from the Subject class. They have no option to register for only specific types and we've seen already how the Presenter class had filtered out events other than "matched".

It is either the observer that must filter all incoming events or the subject that should allow observers to register for specific events at the source. The first approach will be inefficient if the number of events filtered out by every subscriber is large enough. The second approach may make the observer registration and event dispatch overly complex.

Despite the finer granularity of handlers and multicast capabilities, the subject-based approach to event programming rarely makes the application components more loosely coupled than the callback-based approach. This is why it isn't a good choice for the overall architecture of large applications, but rather a tool for specific problems.

This is mostly due to the focus on subjects, which requires all handlers to maintain a lot of assumptions about the observed subjects. Also, in the implementation of that style (that is, the observer design pattern), both observers and subjects must, at some point, meet in the same context. In other words, observers cannot register to events if there is no actual subject that would emit them.

Fortunately, there is a style of event-driven programming that allows fine-grained multicast event handling in a way that really fosters loose coupling of large applications. It is a topic-based style and is a natural evolution of subject-based event programming.

Topic-based style

Topic-based event programming concentrates on the types of events that are passed between software components without skewing toward either side of the emitter-handler relationship. Topic-based event programming is a generalization of previous styles. Event-driven applications written in the topic-based style allow components (for example, classes, objects, and functions) to both emit events and/or register to event types, completely ignoring the other side of the emitter-handler relation.

In other words, handlers can be registered to event types, even if there is no emitter that would emit them, and emitters can emit events even if there is no one subscribed to receive them. In this style of event-driven programming, events are first-class entities that are often defined separately from emitters and handlers. Such events are often given a dedicated class or are just global singleton instances of one generic Event class. This is why handlers can subscribe to events even if there is no object that would emit them.

Depending on the framework or library of choice, the abstraction that's used to encapsulate such observable event types/classes can be named differently. Popular terms are channels, topics, and signals. The term signal is particularly popular, and, because of that, this style of programming is sometimes called signal-driven programming. Signals can be found in such popular libraries and frameworks as Django (web framework), Flask (web microframework), SQLAlchemy (database ORM), and Scrapy (web crawling and scraping framework).

Amazingly, successful Python projects do not build their own signaling frameworks from scratch, but instead use an existing dedicated library. The most popular signaling library in Python seems to be blinker. It is characterized by extremely wide Python version compatibility (Python 2.4 or later, Python 3.0 or later, Jython 2.5 or later, or PyPy 1.6 or later) and has an extremely simple and concise API that allows it to be used in almost any project.

blinker is built on the concept of named signals. To create a new signal definition, you simply use the signal(name) constructor. Two separate calls to the signal() constructor with the same name value will return the same signal object. This allows you to easily refer to signals at any time. The following is an example of the SelfWatch class, which uses named signals to notify its instances every time a new sibling is created:

import itertools
from blinker import signal
class SelfWatch:
    _new_id = itertools.count(1)
    def __init__(self):
        self._id = next(self._new_id)
        init_signal = signal("SelfWatch.init")
        init_signal.send(self)
        init_signal.connect(self.receiver)
    def receiver(self, sender):
        print(f"{self}: received event from {sender}")
    def __str__(self):
        return f"<{self.__class__.__name__}: {self._id}>"

Let's save above code in topic_based_events.py file. The following transcript of the interactive session shows how new instances of the SelfWatch class notify the siblings about their initialization:

>>> from topic_based_events import SelfWatch
>>> selfwatch1 = SelfWatch()
>>> selfwatch2 = SelfWatch()
<SelfWatch: 1>: received event from <SelfWatch: 2>
>>> selfwatch3 = SelfWatch()
<SelfWatch: 2>: received event from <SelfWatch: 3>
<SelfWatch: 1>: received event from <SelfWatch: 3>
>>> selfwatch4 = SelfWatch()
<SelfWatch: 2>: received event from <SelfWatch: 4>
<SelfWatch: 3>: received event from <SelfWatch: 4>
<SelfWatch: 1>: received event from <SelfWatch: 4>

Other interesting features of the blinker library are as follows:

• Anonymous signals: Empty signal() calls always create a completely new anonymous signal. By storing the signal as a module variable or class attribute, you will avoid typos in string literals or accidental signal name collisions.
• Subject-aware subscription: The signal.connect() method allows us to select a specific sender; this allows you to use subject-based event dispatching on top of topic-based dispatching.
• Signal decorators: The signal.connect() method can be used as a decorator; this shortens code and makes event handling more evident in the code base.
• Data in signals: The signal.send() method accepts arbitrary keyword arguments that will be passed to the connected handler; this allows signals to be used as a message-passing mechanism.

One really interesting thing about the topic-based style of event-driven programming is that it does not enforce subject-dependent relations between components. Both sides of the relation can be event emitters and handlers to each other, depending on the situation. This way of event handling becomes just a communication mechanism. This makes topic-based event programming a good choice for the architectural pattern.

The loose coupling of software components allows for smaller incremental changes. Also, an application process that is loosely coupled internally through a system of events can be easily split into multiple services that communicate through message queues. This allows transforming event-driven applications into distributed event-driven architectures.

Let's take a look at event-driven architectures in the next section.

Event-driven architectures

From event-driven applications, there is only one minor step to event-driven architectures. Event-driven programming allows you to split your application into isolated components that communicate with each other only by exchanging events or signals. If you already did this, you should be also able to split your application into separate services that do the same, but transfer events to each other, either through some kind of inter-process communication (IPC) mechanism or over the network.

Event-driven architectures transfer the concept of event-driven programming to the level of inter-service communication. There are many good reasons for considering such architectures:

• Scalability and utilization of resources: If your workload can be split into many order-independent events, architectures that are event-driven allow the work to be easily distributed across many computing nodes (hosts). The amount of computing power can also be dynamically adjusted to the number of events being processed in the system at any given moment.
• Loose coupling: Systems that are composed of many (preferably small) services communicating over queues tend to be more loosely coupled than monolithic systems. Loose coupling allows for easier incremental changes and the steady evolution of system architecture.
• Failure resiliency: Event-driven systems with proper event transport technology (distributed message queues with built-in message persistency) tend to be more resilient to transient issues. Modern message queues, such as Kafka or RabbitMQ, offer multiple ways to ensure that the message will always be delivered to at least one recipient and are able to ensure that the message will be redelivered in the case of unexpected errors.

Event-driven architectures work best for problems that can be dealt with asynchronously, such as file processing or file/email delivery, or for systems that deal with regular and/or scheduled events (for example, cron jobs). In Python, it can also be used as a way of overcoming the CPython interpreter's performance limitations (such as Global Interpreter Lock (GIL), which was discussed in Chapter 6, Concurrency) by splitting the workload across multiple independent processes.

Last but not least, event-driven architectures seem to have a natural affinity for serverless computing. In this cloud-computing execution model, you're not concerned about infrastructure and you don't have to purchase computing capacity units. You leave all of the scaling and infrastructure management for your cloud service operator and provide them only with your code to run. Often, the pricing for such services is based only on the resources that are used by your code. The most prominent category of serverless computing services is Function as a Service (FaaS), which executes small units of code (functions) in response to events.

In the next section, we will discuss in more detail event and message queues, which form the foundation of most event-based architectures.

Event and message queues

In most single-process implementations of event-driven programming, events are handled as soon as they appear and are usually processed in a serial fashion. Whether it is a callback-based style of GUI application or full-fledged signaling in the style of the blinker library, an event-driven application usually maintains some kind of mapping between events and lists of handlers to execute.

This style of information passing in distributed applications is usually realized through a request-response communication model. Request-response is a bidirectional and obviously synchronous way of communication between services. It can definitely be a basis for simple event handling but has many downsides that make it really inefficient in large-scale or complex systems. The biggest problem with request-response communication is that it introduces relatively high coupling between components:

• Every communicating component needs to be able to locate dependent services. In other words, event emitters need to know the network addresses of network handlers.
• A subscription happens directly in the service that emits the event. This means that, in order to create a completely new event connection, usually more than one service has to be modified.
• Both sides of communication must agree on the communication protocol and message format. This makes potential changes more complex.
• A service that emits events must handle potential errors that are returned in responses from dependent services.
• Request-response communication often cannot be easily handled in an asynchronous way. This means that event-based architecture built on top of a request-response communication system rarely benefits from concurrent processing flows.

Due to the preceding reasons, event-driven architectures are usually implemented using the concept of message queues, rather than request-response cycles. A message queue is a communication mechanism in the form of a dedicated service or library that is only concerned with the messages and their intended delivery mechanism. It just acts as a communication hub between various parties. In a contract, the request-response flow requires both communicating parties to know each other and be "alive" during every information exchange.

Typically, writing a new message to the message queue is a fast operation as it does not require immediate action (a callback) to be executed on the subscriber's side. Moreover, event emitters don't need their subscribers to be running at the time that the new message is emitted, and asynchronous messaging can increase failure resilience. The request-response flow, in contrast, assumes that dependent services are always available, and the synchronous processing of events can introduce large processing delays.

Message queues allow for the loose coupling of services because they isolate event emitters and handlers from each other. Event emitters publish messages directly to the queue but don't need to care if any other service listens to its events. Similarly, event handlers consume events directly from the queue and don't need to worry about who produced the events (sometimes, information about the event emitter is important, but, in such situations, it is either in the contents of the delivered message or takes part in the message routing mechanism). In such a communication flow, there is never a direct synchronous connection between event emitters and event handlers, and all the exchange of information happens through the queue.

In some circumstances, this decoupling can be taken to such an extreme that a single service can communicate with itself by an external queuing mechanism. This isn't so surprising, because using message queues is already a great way of inter-thread communication that allows you to avoid locking (see Chapter 6, Concurrency).

Besides loose coupling, message queues (especially in the form of dedicated services) have many additional capabilities:

• Persistence: Most message queues are able to provide message persistence. This means that, even if a message queue's service dies, no messages will be lost.
• Retrying: Many message queues support message delivery/processing confirmations and allow you to define a retry mechanism for messages that fail to deliver. This, with the support of message persistency, guarantees that if a message was successfully submitted, it will eventually be processed, even in the case of transient network or service failures.
• Natural concurrency: Message queues are naturally concurrent. With various message distribution semantics (for example, fan-out and round-robin), it is a great basis for a highly scalable and distributed architecture.

When it comes to the actual implementation of the message queue, we can distinguish two major architectures:

• Brokered message queues: In this architecture, there is one service (or cluster of services) that is responsible for accepting and distributing events. The most common examples of open-source brokered message queue systems are RabbitMQ and Apache Kafka. A popular cloud-based service is Amazon SQS. These types of systems are most capable in terms of message persistence and built-in message delivery semantics.
• Brokerless message queues: These are implemented solely as programming libraries. A popular brokerless messaging library is ZeroMQ (often spelled as ØMQ or zmq). The biggest advantage of brokerless messaging is elasticity. Brokerless messaging libraries trade operational simplicity (no additional centralized service or cluster of services to maintain) for feature completeness and complexity (things like persistence and complex message delivery need to be implemented inside of services).

Both types of messaging approach have advantages and disadvantages. In brokered message queues, there is always an additional service to maintain in the case of open-source queues running on their own infrastructure, or an additional entry on your cloud provider invoice in the case of cloud-based services. Such messaging systems quickly become a critical part of your architecture. If not designed with high availability in mind, such a central message queue can become a single point of failure for your whole system architecture. Anyway, modern queue systems have a lot of features available out of the box, and integrating them into your code is usually a matter of proper configuration or a few API calls. With the AMQP standard, it's also quite easy to run local ad hoc queues for testing.

With brokerless messaging, your communication is often more distributed. This means that your system architecture does not depend on a single messaging service or cluster. Even if some services are dead, the rest of the system can still communicate. The downside of this approach is that you're usually on your own when it comes to things like message persistency, delivery/processing confirmations, delivery retries, and handling complex network failure scenarios like network splits. If you have such needs, you will either have to implement such capabilities directly in your services or build your own messaging broker from scratch using brokerless messaging libraries. For larger distributed applications, it is usually better to use proven and battle-tested message brokers.

Event-driven architectures encourage modularity and the decomposition of large applications into smaller services. This has both advantages and disadvantages. With many components communicating over the queues, it may be harder to debug applications and understand how they work.

On the other hand, good system architecture practices like separation of concerns, domain isolation, and the use of formal communication contracts improve overall architecture and make the development of separate components easier.

Summary

In this chapter, we discussed the elements of event-driven programming. We started with the most common examples and applications of event-driven programming to better introduce ourselves to this programming paradigm. Then, we precisely described the three main styles of event-driven programming, callback-based style, subject-based style, and topic-based style. There are many event-driven design patterns and programming techniques, but all of them fall into one of these three categories. The last part of this chapter focused on event-driven programming architectures.

With this chapter, we end something that we could call an "architecture and design arc." From now on, we will be talking less about architecture, design patterns, programming, and paradigms and more about Python internals and advanced syntax features.

The next chapter is all about metaprogramming in Python, that is, how to write programs that can treat themselves as data, analyze themselves, and modify themselves at runtime.