Pushing code to production is often the last step in taking an application from development to the customer. Though this is an important activity, it often gets overlooked in the scheme of importance in a software architect's checklist.
It is a pretty common and fatal mistake to assume that, if a system works in the development environment, it will also work dutifully in production. For one thing, the configuration of a production system is often very different from that of a development environment. Many optimizations and debugging that are available and taken for granted in a developer's box, are often not available in the production setup.
Deployment to production is an art rather than an exact science. The complexity of deployment of a system depends on a number of factors, such as the language the system is developed in, its runtime portability and performance, the number of configuration parameters, whether the system is deployed in a homogeneous or heterogeneous environment, binary dependencies, geographic distribution of the deployments, deployment automation tooling, and a host of other factors.
In recent years, Python, as an open source language, has matured in the level of automation and support it provides for deploying packages to production systems. With its rich availability of built-in and third-party support tools, the pain and hassle for production deployments and maintaining deployment systems up to date has decreased.
In this chapter, we will discuss, briefly, deployable systems and the concept of deployability. We'll take some time to understand the deployment of Python applications, and the tools and processes that the architect can add to his repertoire in order to ease the deploying and maintenance of his production systems' running applications, written using Python. We will also look at techniques and best practices that an architect can adopt to keep his production systems chugging along healthily and securely, without frequent downtimes.
Here are the list of topics we will be talking about in this chapter.
- Deployability
- Factors affecting deployability
- Tiers of software deployment architecture
- Software Deployment in Python
- Packaging Python code
- PIP
- Virtualenv
- Virtualenv and PIP
- PyPI—the Python Package Index
- Packaging and submission of an application
- PyPA
- Remote deployments using Fabric
- Remote deployments using Ansible
- Managing remote daemons using Supervisor
- Packaging Python code
- Deployment—patterns and best practices
The deployability of a software system is the ease with which it can be taken from development to production. It can be measured in terms of the effort—in terms of man-hours, or complexity—in terms of the number of disparate steps required for deploying code from a development to production environment.
It is a common mistake to assume that a code that runs well in a development or staging system would behave in a similar way in a production system. It is not often the case due to the vastly dissimilar requirements that a production system has when compared to a development one.
Here is a brief look at some of the factors that differentiate a production system from a development one, which can often give rise to unexpected issues in deployment leading to Production Gotchas:
- Optimizations and debugging: It is very common for development systems to turn off optimizations in code.
If your code is running in an interpreted runtime like Python, it is common to turn on debug configurations, which allows the programmer to generate generous tracebacks when an exception occurs. Also any Python interpreter optimizations are usually turned off.
On the other hand, in production systems, the reverse is true – as optimizations are turned on and debugging is turned off. This usually requires additional configuration to be enabled for the code to work in a similar way. It is also possible (though rare) that the program gives a different behavior upon optimization under certain circumstances than it does when running unoptimized.
- Dependencies and versions: A development environment, usually, has a rich installation of development and support libraries for running multiple applications that a developer may be working on. Quite often, these may be dependencies which are themselves not stale, since developers often work on bleeding edge code.
Production systems, on the other hand, need to be carefully prepared using a precompiled list of dependencies and their versions. It is quite common to specify only mature or stable versions for deployment on production systems. Hence, if a developer had relied on a feature or bug-fix which was available on an unstable (alpha, beta or release-candidate) version of a downstream dependency, one may find—too late—that the feature doesn't work in production as intended.
Another common problem is undocumented dependencies or dependencies that need to be compiled from source code—this is often a problem with first-time deployments.
- Resource configuration and access privileges: Development systems and production systems often differ in level, privilege, and details of access of resources locally and in the network. A development system may have a local database, whereas, production systems tend to use separate hosting for application and database systems. A development system may use a standard configuration file, while, in production, the configuration may have to be generated specifically for a host or an environment using specific scripts. Similarly, in production, the application may be required to run with lesser privileges as a specific user/group, whereas, in development, it may be common to run the program as the root or superuser. Such disparities in user privileges and configuration may affect resource access and might cause software to fail in production, when it runs fine on the development environment.
- Heterogeneous production environments: Code is usually developed in development environments, which are usually homogeneous. But it may often be required to be deployed on heterogeneous systems in production. For example, software may be developed on Linux, but there may be a requirement for a customer deployment on Windows.
The complexity of deployments increases proportionally to heterogeneity in environments. Well-managed staging and testing environments are required before such code is taken to production. Also, heterogeneous systems make dependency management more complex, as a separate list of dependencies needs to be maintained for each target system architecture.
- Security: In development and testing environments, it is somewhat common to give a wide berth to security aspects to save time and to reduce the configuration complexity for testing. For example, in a web application, routes which need logins may be disabled by using special development environment flags to facilitate quick programming and testing.
Similarly, systems used in development environments may often use easy-to-guess passwords, such as database systems, web application logins, and others, to make routine recall and usage easy. Also, role-based authorization may be ignored to facilitate testing.
However, security is critical in production, so these aspects require the opposite treatment. Routes which need logins should be enforced as such. Strong passwords should be used. Role-based authentication needs to be enforced. These can often cause subtle bugs in production where a feature which works in the development environment fails in production.
Since these and other similar problems are the bane of deploying code in production, standard practices have been defined to make the life of the DevOps practitioner a bit easier. Most companies follow the practice of using isolated environments to develop, test, and validate code and applications before pushing them to production. Let us take a look at this.