1.1 OS Packages

Volunteers have built ready-to-install packages for some of the more popular operating systems.

On macOS, the Homebrew third-party package manager has up-to-date Python packages. An introduction to Homebrew is beyond the scope of this book. Homebrew being a rolling release, the Python version is upgraded from time to time. While this means that it is a useful way to get an up-to-date Python, it makes it a poor target for reliably distributing tools.

It also has some downsides for doing day-to-day development. Since it upgrades quickly after a new Python release, development environments can break quickly and without warning. It also means that sometimes code can stop working; even if you are careful to watch upcoming Python versions for breaking changes, not all packages will. Homebrew is a good fit when needing a well-built up-to-date Python interpreter for a one-off task. Writing a quick script to analyze data, or automate some APIs, is a good use of Homebrew Python.

Finally, for Windows, it is possible to download an installer from Python.org for any version of Python.

1.2 Using pyenv

pyenv tends to offer the highest return on investment for installing Python for local development. The initial setup does have some subtleties. However, it allows installing many side-by-side Python versions as needed. It also allows you to manage how each is accessed—either using a per-user default or a per-directory default.

Installing pyenv itself depends on the operating system. On a macOS, the easiest way is to install it via Homebrew. Note that in this case, pyenv itself might need to be upgraded to install new versions of Python.

Unfortunately, pyenv does not work on Windows.

This allows pyenv to properly intercept all the necessary commands.

For CPython, <version> is the version number, such as 3.6.6 or 3.7.0rc1.

Local means they are available in a given directory. This is done by putting a python-version.txt file in this directory. This is important for version-controlled repositories, but a few different strategies are used to manage those. One strategy is to add this file to the ignored list. This is useful for heterogeneous teams or open source projects. Another strategy is to check in the file so that the same version of Python is used in the repository.

Note that since pyenv is designed to install side-by-side versions of Python, it has no concept of upgrading Python. A newer Python version needs to be installed with pyenv and then set as the default.

Let’s build a version that is pretty similar to binary versions from python.org.

1.3 Building Python from Source

The main challenge in building Python from source is that, in some sense, it is too forgiving. It is all too easy to build it with one of the built-in modules disabled because its dependency was not detected. This is why it is important to know which dependencies are fragile and how to make sure a local installation is good.

The first fragile dependency is SSL. It is disabled by default and must be enabled in Modules/Setup.dist. Carefully follow the instructions there about the location of the OpenSSL library. If you have installed OpenSSL via system packages, it is usually in /usr/. If you have installed it from source, it is usually in /usr/local.

The most important thing is to know how to test for it. When Python is done building, run ./python.exe -c 'import _ssl'. That .exe is not a mistake; this is how the build process calls the newly built executable, which is renamed to Python during installation. If this succeeds, the SSL module was built correctly.

Another extension that can fail to build is SQLite. Since it is a built-in, many third-party packages depend on it, even if you are not using it yourself. This means a Python installation without the SQLite module is pretty broken. Test it by running ./python.exe -c 'import sqlite3'.

In a Debian-based system (such as Ubuntu), libsqlite3-dev is required for this to succeed. In a Red Hat-based system (such as Fedora or CentOS), libsqlite3-dev is required for this to succeed.

Next, check for _ctypes with ./python.exe -c 'import _ctypes'. If this fails, likely, the libffi headers are not installed.

Finally, remember to run the built-in regression test suite after building from source. This ensures that there have been no silly mistakes while building the package.

1.4 PyPy

The usual implementation of Python is sometimes known as CPython to distinguish it from the language proper. The most popular alternative implementation is PyPy, a Python-based JIT implementation of Python in Python. Because it has a dynamic JIT (just-in-time) compilation to assembly, it can sometimes achieve phenomenal speed-ups (three times or even ten times) over regular Python.

There are sometimes challenges in using PyPy. Many tools and packages are tested only with CPython. However, sometimes spending the effort to check if PyPy is compatible with the environment is worth it if performance matters.

There are a few subtleties in installing Python from source. While it is theoretically possible to translate using CPython, in practice, the optimizations in PyPy mean that translating using PyPy works on more reasonable machines. Even when installing from source, it is better to first install a binary version to bootstrap.

The bootstrapping version should be PyPy, not PyPy3. PyPy is written in the Python 2 dialect, which is one of the only cases where worrying about the deprecation is irrelevant since PyPy is a Python 2 dialect interpreter. PyPy3 is the Python 3 dialect implementation, which is usually better in production as most packages are slowly dropping support for Python 2.

The latest PyPy3 supports 3.5 features of Python, as well as f-strings. However, the latest async features, added in Python 3.6, do not work.

1.5 Anaconda

The closest to a system Python that is still reasonable for use as a development platform is Anaconda, a metadistribution. It is, in essence, an operating system on top of the operating system. Anaconda has its grounding in the scientific computing community, and so its Python comes with easy-to-install modules for many scientific applications. Many of these modules are non-trivial to install from PyPI, requiring a complicated build environment.

It is possible to install multiple Anaconda environments on the same machine. This is handy when needing different Python versions or different versions of PyPI modules.

To bootstrap Anaconda, you can use the bash installer available at https://conda.io/miniconda.html. The installer also modifies ~/.bash_profile to add the path to conda, the installer.

conda environments are created using conda create --name <name> and activated using source conda activate <name>. There is no easy way to use inactivated environments. It is possible to create a conda environment while installing packages: conda create --name some-name python. You can specify the version using = – conda create --name some-name python=3.5. It is also possible to install more packages into a conda environment, using conda install package[=version], after the environment has been activated. Anaconda has a lot of prebuilt Python packages, especially ones that are non-trivial to build locally. This makes it a good choice if those packages are important to your use case.

1.6 Summary

Running a Python program requires an interpreter installed on the system. Depending on the operating system and the versions, there are several different ways to install Python. Using the system Python is a problematic option. On macOS and Unix systems, using pyenv is almost always the preferred option. On Windows, using the prepackaged installers from Python.org is often a good idea.