Creating a new template

We could use these templates as the basis for our aggregation process, but none of them exactly match the decisions we made in the earlier chapters. Instead, I’ll create a new template that captures this book’s recommendations for a minimal Python package. You can adapt this to match your preferences or create new templates to automate boilerplate code creation specific to your work.

We also need to create a directory that contains the templates we’re going to create. We can also use braces to include user-supplied values in filenames, so this should be called {{ cookiecutter.project_slug }} to create a directory whose name is the same as the project_slug value. We could use any value from cookiecutter.json; however, the project slug is the best choice. This directory will become the root of the new project’s git repository, so its name should match the expected repository name.

From here, we can create the various files that we want to include in every project of this type, such as the build files (setup.py, setup.cfg), the documentation (README.md, CHANGES.md, LICENCE), and the test/ and src/ directories.

There is a complication, however. The template includes a {{ cookiecutter.project_slug }}/ directory inside src/, which works fine for any packages that don’t contain a . in their slug, but if we were creating apd.sensors , we’d see a discrepancy between what the cookiecutter generates and what we want (Figure 6-1).

We need this additional level in the directory structure because apd is a namespace package. When we first created apd.sensors , we decided that apd would be a namespace, which allows us to create multiple packages within the namespace on the condition that no code is placed directly in the namespace packages, only the standard packages they contain.

We need some custom behavior here, above and beyond what is possible with a template alone.³ We need to recognize where there is a . in a slug and, in that case, split the slug and create nested directories for each of the parts. Cookiecutter supports this requirement through the use of a post-generation hook. In the root of the template, we can add a hooks directory with a post_gen_project.py file. Pre-generation hooks, stored as hooks/pre_gen_project.py, are used to manipulate and validate user input before generation starts; post-generation hooks , stored as hooks/post_gen_project.py, are used to manipulate the generated output.

Database types

The first thing we need to do is decide how the data should be stored in this application. There are lots of databases available, which cover a wide variety of feature sets. Developers very often choose a particular database according to the current fashion, rather than a dispassionate analysis of the pros and cons. Figure 6-2 is a decision tree that encapsulates the broad questions I ask myself when deciding what style of database to use. This only helps you find a broad category of database, not a particular piece of software, as the feature sets vary massively. Still, I believe it is helpful to ask these questions when deciding on a type of database.

The first question I ask myself is to rule out a few special cases of database technology. These are valuable technologies, and in their particular niche, they are excellent, but they are relatively infrequency required. These are append-only databases – where, once something is written, it can’t be (easily) removed or edited. This kind of database is a perfect match for logs, such as transaction logs or audit logs. The primary difference between a blockchain and an append-only database is trust; while both prevent editing or deleting data in the typical case, a standard append-only database can be edited by manipulating the underlying storage files. A blockchain is slightly different; it allows a group of people jointly to act as the maintainer. Data can only be edited or removed if at least 50% of the users agree. Any users that don’t agree can keep the old data and leave the group. At the time of writing, blockchains are the fashionable database du jour, but they are inappropriate for almost all applications.

Much more useful are the database types to the left of the diagram. They are the SQL and NoSQL databases. NoSQL databases were fashionable in the early 2010s. Relational databases have since adopted some of their features as extensions and additional data types. The use of SQL or not isn’t the critical way of distinguishing between these database types, but rather if they are schemaless or not. This difference is similar to Python with and without type hints; a schemaless database allows users to add data of any arbitrary shape,⁵ whereas a database that has a defined schema validates data to ensure it meets the expectations of the database author. A schemaless database might appear to be more appealing, but it can make querying or migrating data much more difficult. If there are no guarantees over what columns are present and their types, it’s possible to store data that appears to be correct but presents problems later in development.

For example, imagine we have a temperature log table which stores the time a temperature value is logged, the sensor that logged this temperature, and the value. The value would likely be declared to be a decimal number, but what would happen if a sensor provided a string like "21.2c" instead of 21.2? In a schema-enforcing database, this would raise an error, and the data would fail to insert. In a schemaless database, the insert would succeed but attempts to aggregate the data (such as calculating the mean) fail if one of these incorrectly formatted entries is present in the retrieved data set. As with Python’s type hinting, this doesn’t protect against all errors, just a type of error. A value of 70.2 would be accepted as it’s a valid number, even though a human can tell that it is a measurement in degrees Fahrenheit rather than Celsius.

The final thing we need to consider is how we’re going to be querying the data. Querying support is the hardest of these three questions to generalize, as there are significant differences within classes of database. People often describe relational databases as being better for querying and NoSQL databases as being more reliant on a natural key, like a path in an object store or a key in a key/value store. However, this is an oversimplification. For example, SQLite is a relational database, but it has a relatively minimal set of indexing options compared to alternatives such as PostgreSQL; and Elasticsearch is a NoSQL database designed for flexibility in indexing and search.

Our example

In our case, we find it very difficult to decide a single type for the value of a sensor, other than the fact that all values are JSON serializable. We want to be able to access the internals of this type, for example, the magnitude of a temperature value or the length of a list of IP addresses. If we were to build this with standard relational database constructs, we’d struggle to represent these options in a future-proof way. We’d have to write the database structure with foreknowledge of the different types of value that could be returned.

A better fit for us is to use a schemaless database, letting the JSON representation of the sensor returned from the API be the data that’s stored. We have a guarantee that we can restore this data accurately (assuming we have the same version of the sensor code), and there is no difficulty in finding a way of representing it.

This question has taken us to the lowest of the decision points on our decision tree; we now need to consider the relationships between items in the database. A single sensor value is related to other values by virtue of being generated by the same sensor type, by being retrieved from the same endpoint, as well as by being retrieved at the same time. That is, sensor values are related through sensor name, endpoint URL, and creation time. These multiple dimensions of relationship should steer us toward a database with rich indexing and query support, as it would help us to find related data. We would also want to look to a database with good querying support as we want to be able to find records from their values, not just the sensor and time.

This format balances some of the advantages of fixed-schema databases, in that it’s partially fixed. The name and collected_at fields are fixed columns, but the remaining data field is a schemaless field. In theory, we could store JSON or any other serialization format as a TEXT column in this table, but using the JSONB field allows us to write queries and indexes that introspect this value.

Object-relational mappers

It’s entirely possible to write SQL code directly as Python, but it’s relatively rare for people to do this. Databases are complex beasts, and SQL is infamous for being vulnerable to injection attacks. It’s not possible to completely abstract away the peculiarities of individual databases, but tools do exist that take care of table creation, column mapping, and SQL generation.

The most popular of these in the Python world is SQLAlchemy, written by Michael Bayer and others. SQLAlchemy is a very flexible object-relational mapper; it handles the translation between SQL statements and native Python objects, and it does so in an extensible way. Another commonly used ORM is the Django ORM, which is less flexible but offers an interface which requires less knowledge of how databases work. In general, you’ll only be using the Django ORM if you’re working on a Django project, and otherwise, SQLAlchemy is the most appropriate ORM.

To begin with, we need to install SQLAlchemy and a database driver. We need to add SQLAlchemy and psycopg2 to the install_requires section in setup.cfg and trigger these dependencies to be reevaluated using pipenv install -e . on the command line.

There are two ways of describing a database structure with SQLAlchemy, the classic and declarative styles. In the classic style, you instantiate Table objects and associate them with your existing classes. In the declarative style, you use a particular base class (which brings in a metaclass), then you define the columns directly on the user-facing class. In most cases, the Python style of the declarative method makes it the natural choice.

Versioning the database

There is a function in SQLAlchemy to create all the various tables, indexes, and constraints that have been defined in this database. This checks the tables and columns that have been defined and generates the matching database structure for them.

This function looks great at first, but it’s very limited. You will likely add more tables or columns in future or at least more indexes when you’ve done some performance testing. The create_all(...) function creates all things that do not yet exist, meaning any tables that are changed but did exist previously are not updated if you re-run create_all(...). As such, relying on create_all(...) can result in a database that has all the tables you expect but not all of the columns.

To combat this, people use a SQL migration framework. Alembic is the most popular one for SQLAlchemy. It works by connecting to an instance of the database and generating the actions that would be needed to bring the connected database in sync with the one defined in code. If you’re using the Django ORM, there is a built-in migration framework that instead works by analyzing all the past migrations and comparing that analyzed state with the current state of the code.

These frameworks allow us to make changes to the database and trust that they will be propagated to actual deployments, regardless of what versions of the software they’ve used in the past. If a user skips a version or three, any migrations between those versions will also be run.

The majority of the files are created in an alembic/ directory inside the package. We need to put the files here so that they’re accessible to people who install the package; files outside of this hierarchy aren’t distributed to end-users. The exception is alembic.ini, which provides the logging and database connection configuration. These are different for each end-user and so can’t be included as part of the package.

We need to modify the generated alembic.ini file , primarily to change the database URI to match the connection string we’re using. We can leave the value of script_location=src/apd/aggregation/alembic if we like, as in this development environment, we’re using an editable installation of apd.aggregation, but that path won’t be valid for end-users, so we should change it to reference an installed package, and we should include a minimal alembic.ini example in the readme file.

We also need to customize the generated code inside the package, starting with the env.py file. This file needs a reference to the metadata object that we looked at earlier when using the create_all(...) function , so it can determine what the state of the models is in code. It also contains functions for connecting to the database and for generating SQL files that represent the migration. These can be edited to allow customizing database connection options to match our project’s needs.

The revision command creates a file in the alembic/versions/ directory. The first part of the name is an opaque identifier which is randomly generated, but the second half is based on the message given above. The presence of the --autogenerate flag means that the generated file won’t be empty; it contains the migration operations required to match the current state of the code. The file is based on a template, script.py.mako in the alembic/ directory. This template is added automatically by Alembic. Although we can modify it if we want, the default is generally fine. The main reason to change this would be to modify the comments, perhaps with a checklist of things to check when generating a migration.

After running black on this file and removing comments containing instructions, it looks like this:

The four module-scope variables are used by Alembic to determine the order in which migrations should be run. These should not be altered. The bodies of the upgrade() and downgrade() functions are what we need to check, to make sure that they’re doing all the changes we expect and only the changes we expect. The most common change that’s needed is if there is an incorrect change detected, such as the migration altering a column but the target state being equal to the start state. This can happen if a database backup was incorrectly restored, for example.

A less common (but still regularly seen) problem is that sometimes an alembic migration includes import statements which introduce code from a dependency or elsewhere in user code, usually when developers are using a custom column type. In this case, the migration must be altered as it’s important that the migration code is entirely freestanding. Any constants should also be copied into the migration file, for this same reason.

If a migration imports external code, then its effects may change over time as that external code changes. Any migrations whose effects aren’t wholly deterministic could lead to real-world databases having inconsistent states, depending on which version of the dependency code was available at the time of the migration.

Finally, some changes are ambiguous. If we were to change the name of the datapoints table we’ve created here, it would not be clear to Alembic if this were a name change or the removal of one table and the creation of another that happens to have the same structure. Alembic always errs on the side of drop and recreate, so if a rename is intended, but the migration isn’t changed, data loss occurs.

Details on the available operations are available in the Alembic documentation, which provides all the everyday operations you might need. Operation plugins can offer new operation types, especially database-specific operations.

Loading data

Now we have the data model defined, and we can begin to load in data from the sensors. We’ll do this over HTTP with the excellent requests library. There is support for making HTTP requests built-in to Python, but the requests library has a better user interface. I recommend using requests over the standard library HTTP support in all situations. You should only use the standard library’s HTTP request support in cases where it’s not practical to use dependencies.

The lowest-level building block we need for pulling data from sensors is a function that, given the API details for an endpoint, makes a HTTP request to the API, parses the results, and creates DataPoint class instances for each sensor.

This function connects to the remote server and returns DataPoint objects for each sensor value present. It can also raise a ValueError representing an error encountered while attempting to read the data and performs some basic checking of the URL provided.

Then, we are using @click.argument for the first time. We use this to collect bare arguments to the function, not options with associated values. The nargs=-1 option to this argument states that we accept any number of arguments, rather than a specific number (usually 1). As such, the command could be invoked as collect_sensor_data http://localhost:8000/ (to collect data from localhost only), as collect_sensor_data http://one:8000/ http://two:8000/ (to collect data from two servers), or even as collect_sensor_data (no data would be collected, but the database connection would be tested implicitly).

The --api-key and --verbose options likely don’t need any explanation, but the --tolerate-failures option is one that we might not have considered. Without this option and its support code, we’d run the standalone(...) function with all the sensor locations, but if one failed, the entire script would fail. This option allows the user to specify that in cases where there are multiple servers specified, then any that succeed have their data saved and failing sensors are omitted. The code achieves this by using this option to decide if it should download data from [("http://one:8000/", "http://two:8000/")] or [("http://one:8000/", ), ("http://two:8000/", )]. The code for this command passes all the servers to standalone(...) in the normal case, but if --tolerate-failures is added, then there will be one call to standalone(...) for each of the server URLs. This is very much a convenience feature, but it’s one I would like if I were using this command myself.

Finally, the support functions are relatively simple. The add_data_from_sensors(...) function wraps the existing get_data_points(...) function and calls session.add(...) on each data point it returns. It then passes these through to the caller as a return value, but as a list rather than a generator. As we’re looping over the generators, it ensures that the iterator is fully consumed. Calls to add_data_from_sensors(...) have access to the DataPoint objects, but they are not obliged to iterate over them to consume a generator.

Databases

Pick a database that matches what you need to do with your data, not what is the current vogue. Some databases, like PostgreSQL, are a good default choice precisely because they offer so much flexibility, but flexibility comes at a complexity cost.

Use an ORM and a migration framework if you’re using a SQL-based database. In all but extreme edge cases, they serve you better than writing your own custom SQL. Don’t be fooled into thinking that the ORM would shield you from knowing about databases, however. It eases the interface, but you’ll have a tough time if you try to interact with a database without understanding its needs.

Custom attribute behavior

If you need something that acts like a calculated property, that is, something that behaves like an attribute on an object but actually builds its value from other sources, a @property is the best way to go. The same is true for one-off wrappers of values, where data is modified or reformatted. In this case, a property with a setter should be used.

If you are writing a behavior to be used multiple times in your codebase (and especially if you’re building a framework for others to use), a descriptor is usually a better choice. Anything that you can do with a property can be done with a custom descriptor, but you should prefer properties as they’re easier to understand at a glance. If you create a behavior, you should be careful to ensure that it does not stray too far from behavior other developers would expect from Python code.

Generators

Generators are appropriate for cases where you want to provide an infinite (or exceedingly long) stream of values to be looped over. They can be used to reduce memory consumption if the user of the generator does not need to keep a record of all previous values. This strength can also be their biggest drawback: code in a generator function is not guaranteed to execute unless you consume the whole generator.

Do not use generators except for in functions where you need to generate a list of items which would only be read once, where the generation is expected to be slow, and where you’re not certain that the caller needs to process all of the items.

Additional resources