In Chapter 2, HTTP and Working with the Web, we explored the HTTP protocol—the primary protocol used by the World Wide Web—and we learned how to use Python as an HTTP client. In Chapter 3, APIs in Action, we expanded on this and looked at ways to consume web APIs. In this chapter, we'll be turning our focus around and looking at how we can use Python to build applications that serve responses to HTTP requests.
In this chapter, we'll cover the following:
- Python web frameworks
- A Python web application
- Hosting Python and WSGI
I should note up front that hosting modern web applications is a very large topic, and a complete treatment is well beyond the scope of this book, where we're focusing on applying Python code to network problems. Topics such as database access, selecting and configuring load balancers and reverse-proxies, containerization, and the system administration techniques needed to keep the whole show up and running won't be covered here. There are many great resources online though that can give you a start, and we'll try to mention as many as we can where relevant, as we go along.
Having said that, the technologies listed above aren't a requirement for creating and serving Python-based web applications, they're simply what a service comes to require as it reaches scale. As we'll see, there are options for easily manageable small-scale application hosting too.
To understand how we can employ Python in responding to HTTP requests, we need to know a bit about what typically needs to occur in order to respond to a request, and what tools and patterns already exist to do this.
A basic HTTP request and response might look like this:
Here our web client sends an HTTP request to a server, where a web server program interprets the request, creates a suitable HTTP response, and sends it back. In this case, the response body is simply the contents of an HTML file read from, with the response headers added by the web server program.
The web server is responsible for the entire process of responding to the client's request. The basic steps it needs to perform are:
First the web server program needs to accept the TCP connection attempt by the client. It then receives the HTTP request from the client over the TCP connection. The server needs to keep the TCP connection open while it generates the HTTP response, and it uses the connection to send the response back to the client. What the server does with the connection after that depends on the HTTP version in use and the value of a possible Connection header in the request (see the RFC for full details at http://tools.ietf.org/html/rfc7230#section-6.3).
Once the web server has received the request, it parses it, then generates the response. When the requested URL maps to a valid resource on the server, the server will respond with the resource at that URL. The resource could be a file on disk (so-called static content), as shown in the diagram of a basic HTTP request and response from before, it could be an HTTP redirect, or it could be a dynamically generated HTML page. If something goes wrong, or the URL is not valid, then instead the response will include a status code in the 4xx or 5xx range. Once the response is prepared, the server sends it back to the client over the TCP connection.
In the early days of the Web, when almost all requested resources consisted of static files read from disk, web servers could be written in a single language and could easily handle all four steps shown in the preceding image. However, as more and more dynamic content came into demand, such as shopping baskets and database-driven resources such as blogs, wikis, and social media, it was quickly found that hard-coding these functionalities into the web server itself was impractical. Instead, facilities were built into web servers to allow the invocation of external code as part of the page generation process.
Hence, web servers could be written in a fast language such as C and could deal with the low-level TCP connections, initial parsing and validating of requests, and handling static content, but then could invoke external code to handle page generation duties when a dynamic response was needed.
This external code is what we commonly refer to when we talk about web applications. So the response process duties can be split, as shown in the following figure:
Web applications can be written in any language that the web server is able to invoke, providing great flexibility and allowing higher level languages to be used. This can drastically reduce the time it takes to develop a new web service. These days there is a great range of languages that can be used to write web applications, and Python is no exception.