2.1 Virtual Environments

Virtual environments are often misunderstood because the concept of environments is not clear. A Python environment refers to the root of the Python installation. The reason an environment is important is because of the lib/site-packages subdirectory of that root. The lib/site-packages subdirectory is where third-party packages are installed.

The most popular tool to add packages to an environment is pip, which is covered in the next section. Before using pip, it is important to understand how virtual environments work.

A real environment is based on Python installation, which means that to get a new real environment, a new Python must be installed and often rebuilt. This is sometimes an expensive proposition.

The advantage of a virtual environment is that it is cheap to set up and tear down. Some modern Python tooling takes advantage of that, setting up and tearing down virtual environments as a normal part of their operation. Setting up and tearing down virtual environments, being cheap and fast, is also a common part of Python developer workflow.

A virtual environment copies the minimum necessary out of the real environment to mislead Python into thinking it has a new root. The precise file structure is less important than remembering that the command to create a virtual environment is simple and fast.

Here, simple means that all the command does is copy some files and perhaps make a few symbolic links. Because of that, there are a few failure modes—mostly when file creation fails because of permission issues or a full disk.

There are two ways to use virtual environments: activated and inactivated. To use an inactivated virtual environment, which is most common in scripts and automated procedures, you explicitly call Python from the virtual environment.

This means that a virtual environment in /home/name/venvs/my-special-env calling /home/name/venvs/my-special-env/bin/python has a Python process that uses this environment. For example, /home/name/venvs/my-special-env/bin/python -m pip runs pip but installs in the virtual environment.

Note that entrypoint–based scripts are installed alongside Python, so running /home/name/venvs/my-special-env/bin/pip also installs packages in the virtual environment.

The sourcing sets a few environment variables, only one of which is important. The important variable is PATH, which gets prefixed by /home/name/venvs/my-special-env/bin. This means that commands like python or pip are found there first. Two cosmetic variables are set. $VIRTUAL_ENV points to the root of the environment. This is useful in management scripts that want to be aware of virtual environments. PS1 is prefixed with (my-special-env), which is useful for visualizing the virtual environment while working interactively in the console.

It is generally a good practice to only install third-party packages inside a virtual environment. Combined with the fact that virtual environments are cheap, if one gets into a bad state, it is best to remove the whole directory and start from scratch.

For example, imagine a bad package install that causes the Python start-up to fail. Even running pip uninstall is impossible since pip fails on start-up. However, the cheapness means you can remove the whole virtual environment and re-create it with a good set of packages.

A modern practice is to move increasingly toward treating virtual environments as semi-immutable. After creating them, there is a single stage for installing all required packages. Instead of modifying the virtual environment if an upgrade is required, destroy the environment, re-create, and reinstall.

The modern way to create virtual environments is to use the venv standard library module. This only works on Python 3. Since Python 2 has been strongly deprecated since the beginning of 2020, it is best avoided in any case.

venv is used as a command with python -m venv <directory>, as there is no dedicated entrypoint. It creates the directory for the environment.

It is best if this directory does not exist before that. A best practice is to remove it before creating the environment. There are also two options for creating the environment: which interpreter to use and what initial packages to install.

2.2 pip

The packaging tool for Python is pip. There have been other tools that have mostly been abandoned by the community and should not be used.

Installations of Python used to not come with pip out of the box. This has changed in recent versions, but many versions which are still supported do not have it. When running on such a version, python -m ensurepip installs it.

Some Python installations, especially system ones, disable ensurepip. When lacking ensurepip, there is a way of manually getting it: get-pip.py. This is a single downloadable file that, when executed, unpacks pip.

Luckily, pip is the only package that needs these weird gyrations to install. All other packages can, and should, be installed using pip.

The last command tests that glom has been properly installed. Glom is a package to handle deeply-nested data, and called with no arguments, outputs an empty Python dictionary. This makes it handy for quickly testing whether a new virtual environment can install new packages properly.

Internally, pip is also treated as a third-party package. Upgrading pip itself is done with pip install --upgrade pip.

Depending on how Python was installed, its real environment might or might not be modifiable by the user. Many instructions in various README files and blogs might encourage using sudo pip install. This is almost always the wrong thing to do; it installs the packages in the global environment.

The pip install command downloads and installs all dependencies. However, it can fail to downgrade incompatible packages. It is always possible to install explicit versions: pip install package-name==<version> installs this precise version. This is also a good way for local testing to get explicitly non-general-availability packages, such as release candidates, beta, or similar.

If wheel is installed, pip builds, and usually caches, wheels for packages. This is especially useful when dealing with a high virtual environment churn since installing a cached wheel is a fast operation. This is also highly useful when dealing with native or binary packages that need to be compiled with a C compiler. A wheel cache eliminates the need to build it again.

pip does allow uninstalling with pip uninstall <package>. This command, by default, requires manual confirmation. Except for exotic circumstances, this command is not used. If an unintended package has snuck in, the usual response is to destroy the environment and rebuild it. For similar reasons, pip install --upgrade <package> is not often needed; the common response is to destroy and re-create the environment. There is one situation where it is a good idea.

pip install supports a requirements file: pip install --requirements or pip install -r. The requirements file simply has one package per line. This is no different from specifying packages on the command line. However, requirement files often specify strict dependencies. A requirements file can be generated from an environment with pip freeze.

Like most individual packages or wheels, installing anything that is not strict and closed under requirements requires pip to decide which dependencies to install. The general problem of dependency resolution does not have an efficient and complete solution. Different strategies are possible to approach such a solution.

The way pip resolves dependencies is by using backtracking. This means that it optimistically tries to download the latest possible requirements recursively. If a dependency conflict is found, it backtracks; try a different option.

There are two base versions: 1.0 and 2.0. The package dependencies are setup.cfg files.

The base package has two versions: 1.0 and 2.0. It does not have any dependencies.

This solution has the advantage that it is complete, but it can take unfeasible amounts of time in certain cases. This is rare, but merely taking a long time is not.

One way to increase the speed is to include >= dependencies in the loose requirements. This is usually a good idea since packages are better at guaranteeing backward compatibility than forward compatibility. As a side benefit, this can dramatically reduce the solution space that pip needs to backtrack in.

In most scenarios, it is better to use strict requirements for day-to-day development and regenerate the strict requirements from the loose requirements (which can take a while) on a cadence that balances keeping up to date with churn.

2.3 Setup and Wheels

The term third party (as in third-party packages) refers to someone other than the Python core developers (first-party) or the local developers (second-party). I have covered how to install first-party packages in the installation section. You used pip and virtualenv to install third-party packages. It is time to finally turn your attention to the missing link: local development and installing local packages or second-party packages.

Note that the word package here means something different from post-installation. In Python, a package is an importable directory, a way to keep multiple modules together. The pedantic way to call installable things is distribution. A distribution can correspond to no packages (it can be a top-level single-module distribution) or multiple packages.

It is good to keep a 1-1-1 relationship when packaging things: a single distribution corresponding to one package and named the same. Even if there is only one file, put it as an __init__.py file under a directory.

Packaging is an area that has seen a lot of changes. Copying and pasting from existing packages is not a good idea; good packages are, for the most part, mature packages. Following the latest best practices means making changes to an existing working process.

The README is not strictly necessary. However, most source code management systems display it rendered, so it is best to break it out into its own file.

Even an empty pyproject.toml generates a package. However, almost all packages need at least a few more details.

Many systems can be used to build valid distributions. The setuptools system, which used to be the only possibility, is now one of several. However, it is still the most popular one.

For most popular code organizations, this is enough for the setuptools systems to find the code and create a correct package.

There are ways to have setuptools treat the version as dynamic and take it from a file or an attribute. An alternative is to take advantage of pyproject.toml in a structured format and manipulate it directly.

Some utilities keep the version synchronized between several files; for example, pyproject.toml and example_package/__init__.py. The best way to use these utilities is by not needing to do it.

This avoids needing to keep more than one place in sync.

The field dependencies in pyproject.toml is present on almost every package. This is how to mark other distributions that the code needs. It is a good practice to put loose dependencies in pyproject.toml. This is in contrast to exact dependencies, which specify a specific version. A loose dependency looks like Twisted>=17.5, specifying a minimum version but no maximum. Exact dependencies, like Twisted==18.1, are usually a bad idea in pyproject. toml. They should only be used in rare cases, for example, when using significant chunks of a private API package.

The pyproject.toml file also allows defining entrypoints. Some frameworks, like Pyramid, allow using entrypoints to add plugin-like features.

The syntax is package.....module:function. This function is called with no arguments when the script is being run.

You can build a distribution with pyproject.toml, a README.rst, and some Python code. There are several formats a distribution can take, but the one covered here is the wheel.

This creates a wheel under dist. If the wheel needs to be in a different directory, add --outdir <output directory> to the command.

You can do several things with the wheel, but it is important to note that one thing you can do is pip install <wheel file>. Doing this as part of continuous integration makes sure the wheel, as built by the current directory, is functional.

It is possible to combine the --sdist and --wheel arguments into one run of python -m build. This is also what python -m build does by default: create both a source distribution and a wheel.

This installs its dependencies in the current environment. While this is not a good fit for production use, this is a faster way to debug packaging issues.

2.4 Binary Wheels

Python is well known and often used as an integration language. One of the ways this integration happens is by linking to native code.

This is usually done using the C Application Binary Interface (C ABI). The C ABI is used not only for integrating with C libraries but with other languages, such as C++, Rust, or Swift, which can generate C ABI-compatible interfaces.

There needs to be some glue code bridging the C ABI to Python to integrate Python with such code. It is possible to write this code by hand.

This code is in the binary_module.pyx file. It is short and does just enough to be clear if it works correctly.

To build code with native integration, the files that describe the build are slightly more complicated.

This makes sure the cython package is installed before trying to build a wheel.

With these files, running python -m build --wheel builds a binary wheel in dist named something like binary_example-1.0-cp39-cp39-linux_x86_64.whl. Details of the name depend on the platform, the architecture, and the Python version.

This is a simple example that demonstrates the mechanics of binary packaging. It is designed to show how all the pieces fit together in a small example.

Realistic binary packages are usually more complicated, implementing subtle algorithms that can take advantage of the optimizations Cython gives or wrapping a native-code library.

2.5 manylinux Wheels

A binary wheel is not a pure Python wheel because at least one of its files contains native code. On a Linux system, this native code is a shared library, a file with the .so suffix for a shared object.

This shared library links against other libraries. For a library designed to wrap a specific native library, as with pygtk wrapping gtk, it links with the wrapped library.

In almost all cases, whether it is designed to wrap a specific library or not, it links against the standard C library. This is the library that has C functions like printf. Few things can be done in native code without linking against it.

On most modern Linux systems, this linking is usually dynamic. This means that the binary wheel does not contain the library it is linked with; it expects to load it at runtime.

If a wheel is built on a different system than the one it is installed on, a library that is binary compatible with the one it is linked with has to be installed on the system. If a binary compatible library is not installed, this leads to a failure at import time.

2.6 tox

tox is a tool to automatically manage virtual environments, usually for tests and builds. It is used to make sure that those run in well-defined environments and is smart about caching them to reduce churn. True to its roots as a test-running tool, tox is configured in test environments.

It uses a unique ini-based configuration format. This can make writing configurations difficult since remembering the subtleties of the file format can be hard. However, while hard to tap, there is a lot of power that can certainly configure tests and build clear and concise runs.

One thing that tox lacks is a notion of dependencies between build steps. This means that those are usually managed from the outside by running specific test runs after others and sharing artifacts somewhat ad hoc.

Note that if the name of the environment contains pyNM (for example, py36), then tox defaults to using CPython, the standard Python implementation, version N.M (3.6, in this case) as the Python interpreter for that test environment.

tox also supports name-based environment guessing for more esoteric implementations of Python. For example, PyPy, an implementation of Python in Python, is supported with the name pypyNM.

If the name does not include one of the supported short names, or if there is a need to override the default, a basepython field in the section can be used to indicate a specific Python version. By default, tox looks for Python available in the path. However, if the plug-in tox-pyenv is installed in the virtual environment that tox itself is installed in, tox will query pyenv if it cannot find the right Python on the path.

Let’s analyze a few tox configuration files in order of increasing complexity.

2.7 Pip Tools

The pip-tools PyPI package contains a dedicated command to freeze dependencies. The pip-compile command takes a loose requirements file as an input and produces one with strict requirements.

The usual names for the files are requirements.in for the input and requirements.txt for the output. Sometimes, when there are a few related variations of dependencies, the files are called requirements-<purpose>.in and requirements-<purpose>.txt, respectively.

More commonly, the loose requirements are already in a setup.cfg for the local code that uses the libraries. In those cases, pip-compile can take this as input directly.

It is usually best to also have a pyproject.toml, though it can be empty. Even though pip-compile does not depend on it, it does help with other parts of the workflow.

The output, requirements.txt, contains quite a few comment lines. The only non-comment line is the pinned gunicorn dependency. Note that the version is different when running pip-compile again in the future when a new package has been released.

This time, the pytest dependency generated more dependencies. All dependencies are pinned.

Note that while pip-tools does try to replicate the algorithm pip uses, there are some edge cases in which the resolution differs, or one might fail to find a resolution. This tends to happen in edge cases, and one way to improve things is to add specificity to some of the dependencies, often as >= details.

Note that even this relatively simple (hypothetical) Python program with two direct dependencies had nine total dependencies. This is typical; frozen, complete dependencies often number in the tens for simple programs and hundreds for many code bases.

Although requirements.txt and requirements-*.txt are generated files, checking them into the source code for applications is usually recommended. For libraries, it sometimes makes sense to check in requirements-test.txt to run at least one set of tests with known-good dependencies.

In either case, it is good to refresh the dependencies occasionally. Most libraries only fix bugs, and security issues, at their tip. Being too far behind the latest version means that if a bug fix or security patch becomes important, a library upgrade is potentially needed. These upgrades tend to be harder. Even with perfectly executed semantic versioning, the library might have bumped its major version number. More realistically, even without a major version number, some assumptions might be wrong. The bigger the jump, the more these assumptions interact and become complicated to resolve.

There are services that automatically open pull requests updating requirements.txt. If none of those are feasible to use, for any reason, writing a script that re-runs pip-compile and produces a pull request is possible. In that case, use the -U flag to pip-compile to let it know that all dependencies need to be upgraded to the latest mutually-consistent versions.

The most important thing is to do it regularly and merge the resulting pull request. These pull requests need to be treated as carefully, if not more so, than those that change the Python code itself. Whatever workflow is used for the latter is appropriate for these dependency-bump pull requests.

2.8 Poetry

Poetry is a package and dependency management system. It gives one tool which handles the entire Python development process: managing dependencies, creating virtual environments, building and publishing packages, and installing Python applications.

2.9 Pipenv

Pipenv is a tool to create virtual environments that match a specification and ways to evolve the specification. It relies on two files: Pipfile and Pipfile.lock.

Pipenv assumes that it controls a project’s directory. To start using it, create a new directory. Inside this directory, it is possible to install packages using pipenv add.

The Pipenv tool is not used for building packages. It is strictly a tool to improve local development or create virtual environments for deployment. For useful to be shippable distribution, in this case, it would need a setup.cfg and pyproject.toml.

2.10 DevPI

DevPI is a PyPI-compatible server that can be run locally. Though it does not scale to PyPI-like levels, it can be a powerful tool in several situations.

DevPI is made up of three parts. The most important one is devpi-server. For many use cases, this is the only part that needs to run. The server serves, first and foremost, as a caching proxy to PyPI. It takes advantage of the fact that packages on PyPI are immutable: once you have a package, it can never change.

There is also a web server that allows you to search in the local package directory. Since many use cases do not even involve searching on the PyPI website, this is optional. Finally, there is a client command-line tool that allows configuring various parameters on the running instance. The client is most useful in more esoteric use cases.

To apply it to all users, edit /etc/pip.conf on Unix. This can be useful, for example, when building container images against a DevPI.

DevPI is useful for disconnected operations. To install packages without a network, DevPI can be used to cache them. As mentioned earlier, virtual environments are disposable and often treated as mostly immutable. This means that a virtual environment with the right packages is not useful without a network. The chances are high that some situations will either require or suggest creating it from scratch.

However, a caching server is a different matter. If all package retrieval is done through a caching proxy, then destroying a virtual environment and rebuilding it is fine since the source of truth is the package cache. This is as useful for taking a laptop into the woods for disconnected development as maintaining proper firewall boundaries and having a consistent record of all installed software.

To warm up the DevPI cache (i.e., make sure it contains all needed packages), you need to use pip to install them. One way to do it is, after configuring DevPI and pip, to run tox against a source repository of software under development. Since tox goes through all test environments, it downloads all needed packages.

It is a good practice to also preinstall any requirements.txt files that are relevant in a disposable virtual environment.

However, the utility of DevPI is not limited to disconnected operations. Configuring one inside your build cluster, and pointing the build cluster at it, completely avoids the risk of a left-pad incident, where a package you rely on gets removed by the author from PyPI. It might also make builds faster and cut out a lot of outgoing traffic.

Note that this allows you to upload to an index that you only use explicitly, so you are not shadowing my-package for all environments that are not using this explicitly.

This makes your DevPI server a mirror of a local, upstream DevPI server. This allows you to upload private packages to the central DevPI server to share with your team. In those cases, the upstream DevPI server often needs to be run behind a proxy, and you need to have some tools to properly manage user access.

Running a centralized DevPI behind a simple proxy that asks for a username and password allows an effective private repository.

This means the root index no longer mirrors pypi. You can upload packages now directly to it. This type of server is often used with the --extra-index-url argument to pip, allowing pip to retrieve from the private and external sources. However, sometimes it is useful to have a DevPI instance that only serves specific packages. This allows enforcing rules about auditing before using any packages. It is downloaded, audited, and then added to the private repository whenever a new package is needed.

2.11 pex and shiv

While it is non-trivial to compile a Python program into one self-contained executable, you can do something almost as good. You can compile a Python program into a single file that only needs an installed interpreter to run. This takes advantage of the particular way Python handles start-ups.

When running python /path/to/directory/, Python acts as though you typed python /path/to/ directory/__main__.py.

A zip file is an end-oriented format. The metadata, and pointers to the data, are all at the end. Adding a prefix to a zip file does not change its contents.

So if you take a zip file and prefix it with #!/usr/bin/python<newline>, and mark it executable, then when running it, Python is running a zip file. If you put the right bootstrapping code in __main__.py and the right modules in the zip file, you can get all the third-party dependencies in one big file.

pex and shiv are tools for producing such files, but they both rely on the same underlying behavior of Python and zip files.

2.12 Summary

Much of the power of Python comes from its powerful third-party ecosystems. Whether for data science or networking code, there are many good options. Understanding how to install, use, and update third-party packages is crucial to using Python well.

With private package repositories, using Python packages for internal libraries, and distributing them in a way compatible with open source libraries, is often a good idea. It allows using the same machinery for internal distribution, versioning, and dependency management.