15.1.1 Injecting state through cookies

The addition of cookies changes the overall relationship between a client and server to become stateful. Interestingly, it involves no change to HTTP. The state information is communicated through headers on the request and the reply. The server will send cookies to the user agent in response headers. The user agent will save and reply with cookies in request headers.

The user agent or browser is required to retain a cache of cookie values, provided as part of a response, and include appropriate cookies in subsequent requests. The web server will look for cookies in the request header and provide updated cookies in the response header. The effect is to make the web server stateless; the state changes happen only in the client. Because a server sees cookies as additional arguments in a request and provides additional details in a response, this shapes our view of the function that responds to a request.

Cookies can contain anything that fits in 4,096 bytes. They are often encrypted to avoid exposing web server details to other applications running on the client computer. Transmitting large cookies can be slow, and should be avoided. The best practice is to keep session information in a database, and provide only a database key in a cookie. This makes the session persistent, and allows session processing to be handled by any available web server, allowing load-balancing among servers.

The concept of a session is a feature of the web application software, not HTTP. A session is commonly implemented via a cookie to retain session information. When an initial request is made, no cookie is available, and a new session cookie is created. Every subsequent request will include the cookie’s value. A logged-in user will have additional details in their session cookie. A session can last as long as the server is willing to accept the cookie; a cookie could be valid forever, or expire after a few minutes.

A RESTful approach to web services does not rely on sessions or cookies. Each REST request is distinct. In many cases, an Authorization header is provided with each request to provide credentials for authentication and authorization. This generally means that a separate client-facing application must create a pleasing user experience, often involving sessions. A common architecture is a front-end application, perhaps a mobile app or browser-based site to provide a view of the supporting RESTful web services.

We’ll focus on RESTful web services in this chapter. The RESTful approach fits well with stateless functional design patterns.

One consequence of sessionless REST processes is each individual REST request is separately authenticated. This generally means the REST service must also use Secure Socket Layer (SSL) protocols. The HTTPS scheme is required to transmit credentials securely from client to server.

15.1.2 Considering a server with a functional design

One core idea behind HTTP is that the server’s response is a function of the request. Conceptually, a web service should have a top-level implementation that can be summarized as follows:

response = httpd(request)

While this is the essence of HTTP, it lacks a number of important details. First, an HTTP request isn’t a simple, monolithic data structure. It has some required parts and some optional parts. A request may have headers, a method (e.g., GET, POST, PUT, PATCH, etc.), a URL, and there may be attachments. The URL has several optional parts including a path, a query string, and a fragment identifier. The attachments may include input from HTML forms or uploaded files, or both.

Second, the response, similarly, has three parts to it. It has a status code, headers, and a response body. Our simplistic model of a httpd() function doesn’t cover these additional details.

We’ll need to expand on this simplistic view to more accurately decompose web processing into useful functions.

15.1.3 Looking more deeply into the functional view

Both HTTP responses and requests have headers that are separate from the body. The request can also have some attached form data or other uploads. Therefore, we can more usefully think of a web server like this:

headers, content = httpd( 
    headers, request, [attachments, either forms or uploads] 
)

The request headers may include cookie values, which can be seen as adding more arguments. Additionally, a web server is often dependent on the OS environment in which it’s running. This OS environment data can be considered as yet more arguments being provided as part of the request.

The Multipurpose Internet Mail Extension (MIME) types define the kinds of content that a web service might return. MIME describes a large but reasonably well-defined spectrum of content. This can include plain text, HTML, JSON, XML, or any of the wide variety of non-text media that a website might serve.

There are some common features of HTTP request processing that we’d like to reuse. This idea of reusable elements is what leads to the creation of web service frameworks that fill a spectrum from simple to sophisticated. The ways that functional designs allow us to reuse functions indicate that the functional approach can help in building web services.

We’ll look at functional design of web services by examining how we can create a pipeline of the various elements of a service response. We’ll do this by nesting the functions for request processing so that inner elements are free from the generic overheads, which are provided by outer elements. This also allows the outer elements to act as filters: invalid requests can yield error responses, allowing the inner function to focus narrowly on the application processing.

15.1.4 Nesting the services

We can look at web request-handling as a number of layered contexts. The foundation might cover session management: examining the request to determine if this is another request in an existing session or a new session. Built on this foundation, another layer can provide tokens used for form processing that can detect Cross-Site Request Forgeries (CSRF). Another layer on top of these might handle user authentication within a session.

A conceptual view of the functions explained previously is something like this:

response = content( 
    authentication( 
        csrf( 
            session(headers, request, forms) 
        ) 
    ) 
)

The idea here is that each function can build on the results of the previous function. Each function either enriches the request or rejects it because it’s invalid. The session() function, for example, can use headers to determine if this is an existing session or a new session. The csrf() function will examine form input to ensure that proper tokens were used. The CSRF handling requires a valid session. The authentication() function can return an error response for a session that lacks valid credentials; it can enrich the request with user information when valid credentials are present.

The content() function is free from worrying about sessions, forgeries, and non- authenticated users. It can focus on parsing the path to determine what kind of content should be provided. In a more complex application, the content() function may include a rather complex mapping from path elements to the functions that determine the appropriate content.

This nested function view suffers from a profound problem. The stack of functions is defined to be used in a specific order. The csrf() function must be done first to provide useful information to the authentication() function. However, we can imagine a high-security scenario where authentication must be done before the CSRF tokens can be checked. We don’t want to have to define unique functions for each possible web architecture.

While each context must have a distinct focus, it would be more helpful to have a single, unified view of request and response processing. This allows pieces to be built independently. A useful website would be a composition of a number of disparate functions.

With a standardized interface, we can combine functions to implement the required features. This will fit the functional programming objectives of having succinct and expressive programs that provide web content. The WSGI standard provides a uniform way to build complex services as a composition of parts.

15.2.1 Raising exceptions during WSGI processing

One central feature of WSGI applications is that each stage along the chain is responsible for filtering the requests. The idea is to reject faulty requests as early in the processing as possible. When building a pipeline of independent WSGI applications, each stage has the following two essential choices:

• Evaluate the start_response() function to start a reply with an error status

• OR pass the request with an expanded environment to the next stage

Consider a WSGI application that provides small text files. A file may not exist, or a request may refer to a directory of files. We can define a WSGI application that provides static content as follows:

def headers(content: bytes) -> list[tuple[str, str]]: 
    return [ 
        ("Content-Type", ’text/plain;charset="utf-8"’), 
        ("Content-Length", str(len(content))), 
    ] 
 
def static_text_app( 
    environ: "WSGIEnvironment", 
    start_response: "StartResponse" 
) -> Iterable[bytes]: 
    log = environ[’wsgi.errors’] 
    try: 
        static_path = Path.cwd() / environ[’PATH_INFO’][1:] 
        with static_path.open() as static_file: 
            print(f"{static_path=}", file=log) 
            content = static_file.read().encode("utf-8") 
            start_response(’200 OK’, headers(content)) 
            return [content] 
    except IsADirectoryError as exc: 
        return index_app(environ, start_response) 
    except FileNotFoundError as exc: 
        print(f"{static_path=} {exc=}", file=log) 
        message = f"Not Found {environ[’PATH_INFO’]}".encode("utf-8") 
        start_response(’404 NOT FOUND’, headers(message)) 
        return [message]

This application creates a Path object from the current working directory and an element of the path provided as part of the requested URL. The path information is part of the WSGI environment, in an item with the ’PATH_INFO’ key. Because of the way the path is parsed, it will have a leading ”/”, which we discard by using environ[’PATH_INFO’][1:].

This application tries to open the requested path as a text file. There are two common problems, both of which are handled as exceptions:

• If the file is a directory, we’ll route the request to a different WSGI application, index_app, to present directory contents

• If the file is simply not found, we’ll return an HTTP 404 NOT FOUND response

Any other exceptions raised by this WSGI application will not be caught. The application that invoked this application should be designed with some generic error-response capability. If the application doesn’t handle the exceptions, a generic WSGI failure response will be used.

This small application shows the WSGI idea of either responding or passing the request onto another application that forms the response. This respond-now-or-forward design pattern enables the building of multi-stage pipelines. Each stage either rejects the request, handles it completely, or passes it on to some other application.

These pipelines are often called middleware because they are between a base server (like NGINX) and the final web application or RESTful API. The idea is to use middleware to perform a series of common filters or mappings for each request.

15.2.2 Pragmatic web applications

The intent of the WSGI standard is not to define a complete web framework; the intent is to define a minimum set of standards that allows flexible interoperability of web-related processing. This minimum fits well with functional programming concepts.

A web application framework is focused on the needs of developers. It should offer numerous simplifications to providing web services. The foundational interface must be compatible with WSGI, so that it can be used in a variety of contexts. The developer’s view, however, will diverge from the minimal WSGI definitions.

Web servers such as Apache httpd or NGINX have adapters to provide a WSGI-compatible interface from the web server to Python applications. For more information on WSGI implementations, visit https://wiki.python.org/moin/WSGIImplementations.

Embedding our applications in a larger server allows us to have a tidy separation of concerns. We can use Apache httpd or NGINX to serve the static content, such as .css, .js, and image files. For HTML pages, though, a server like NGINX can use the uwsgi module to hand off requests to a pool of Python processes. This focuses Python on handling the interestingly complex HTML portions of the web content.

Downloading static content requires little customization. There’s often no application-specific processing. This is best handled in a separate service that can be optimized to perform this fixed task.

The processing for dynamic content (often the HTML content of a web page) is where the interesting Python-based work happens. This work can be segregated to servers that are optimized to run this more complex application-specific computation.

Separating the static content from the dynamic content to provide optimized downloads means that we must either create a separate media server, or define our website to have two sets of paths. For smaller sites, a separate /media path works out nicely. For larger sites, distinct media servers are required.

An important consequence of the WSGI definition is the environ dictionary is often updated with additional configuration parameters. In this way, some WSGI applications can serve as gateways to enrich the environment with information extracted from cookies, headers, configuration files, or databases.

15.3.1 Flask application processing

We’ll use the Flask framework because it provides an easy-to-extend web services process. It supports a function-based design, with a mapping from a request path to a view function that builds the response. The framework also makes use of decorators, providing a good fit with functional programming concepts.

In order to bind all of the configuration and URL routing together, an overall Flask instance is used as a container. Our application will be an instance of the Flask class. As a simplification, each view function is defined separately and bound into the Flask instance via a routing table that maps URLs to functions. This routing table is built via decorators.

The core of the application is this collection of view functions. Generally, each view function needs to do three things:

1. Validate the request.

2. Perform the requested state change or data access.

3. Prepare a response.

Ideally, the view function does nothing more than this.

Here’s the initial Flask object that will contain the routes and their functions:

from flask import Flask 
 
app = Flask(__name__)

We’ve created the Flask instance and assigned it to the app variable. As a handy default, we’ve used the module’s name, __name__, as the name of the application. This is often sufficient. For complex applications, it may be better to provide a name that’s not specifically tied to a Python module or package name.

Most applications will need to have configuration parameters provided. In this case, the source data is a configurable value that might change.

For larger applications, it’s often necessary to locate an entire configuration file. For this small application, we’ll provide the configuration value as a literal:

from pathlib import Path 
 
app.config[’FILE_PATH’] = Path.cwd() / "Anscombe.txt"

Most of the view functions should be relatively small, focused functions that make use of other layers of the application. For this application, the web presentation depends on a data service tier to acquire and format the data. This leads to functions with the following three steps:

1. Validate the various inputs. This includes validating items like the path, any query parameters, form input data, uploaded files, header values, and even cookie values.

2. If the method involves a state change like POST, PUT, PATCH, or DELETE, perform the state-changing operation. These will often return a ”redirect” response to a path that will display the results of the change. If the method involves a GET request, gather the requested data.

3. Prepare the response.

What’s important about step 2 is all of the data manipulation is separate from the RESTful web application. The web presentation sits on a foundation of data access and manipulation. The web application is designed as a view or a presentation of the underlying structure.

We’ll look at two URL paths for the web application. The first path will provide an index of the available series in the Anscombe collection. The view function can be defined as follows:

from flask import request, abort, make_response, Response 
 
@app.route("/anscombe/") 
def index_view() -> Response: 
    # 1. Validate 
    response_format = format() 
    # 2. Get data 
    data = get_series_map(app.config[’FILE_PATH’]) 
    index_listofdicts = [{"Series": k} for k in data.keys()] 
    # 3. Prepare Response 
    try: 
        content_bytes = serialize(response_format, index_listofdicts, document_tag="Index", row_tag="Series") 
        response = make_response(content_bytes, 200, {"Content-Type": response_format}) 
        return response 
    except KeyError: 
        abort(404, f"Unknown {response_format=}")

This function has the Flask @app.route decorator. This shows what URLs should be processed by this view function. There are a fair number of options and alternatives available here. The view function will be evaluated when a request matches one of the available routes.

The format() function definition will be shown in a little while. It locates the user’s desired format by looking in two places: the query string, after the ? in the URL, and also in the Accept header. If the query string value is invalid, a 404 response will be created.

The get_series_map() function is an essential feature of the data service tier. This will locate the Anscombe series data and create a mapping from Series name to the data of the series.

The index information is in the form of a list-of-dict structure. This structure can be converted to JSON, CSV, and HTML without too much complication. Creating XML is a bit more difficult. The difficulty arises because the Python list and dictionary objects don’t have any specific class name, making it awkward to supply XML tags.

The data preparation is performed in two parts. First, the index information is serialized in the desired format. Second, a Flask Response object is built using the bytes, an HTTP status code of 200, and a specific value for the Content-Type header.

The abort() function stops process and returns an error response with the given code and reason information. For RESTful web services, it helps to add a small helper function to transform the result into JSON. The use of the abort() function during data validation and preparation makes it easy to end processing at the first problem with the request.

The format() function is defined as follows:

def format() -> str: 
    if arg := request.args.get(’form’): 
        try: 
            return { 
                ’xml’: ’application/xml’, 
                ’html’: ’text/html’, 
                ’json’: ’application/json’, 
                ’csv’: ’text/csv’, 
            }[arg] 
        except KeyError: 
            abort(404, "Unknown ?form=") 
    else: 
        return request.accept_mimetypes.best or "text/html"

This function looks for input from two attributes of the request object:

• The args will have the argument values that are present after the ”?” in the URL

• The accept_mimetypes will have the parsed values from the Accept header, allowing an application to locate a response that meets the client’s expectations

The request object is a bit of thread-local storage with the details of the web request being made. It is used like a global variable, making some functions look a little awkward. The use of a global like request tends to obscure the actual parameters to this function. Using explicit parameters requires also providing the underlying type information, which is little more than visual clutter.

The series_view() function to provide series data is defined as follows:

@app.route("/anscombe/<series_id>") 
def series_view(series_id: str, form: str | None = None) -> Response: 
    # 1. Validate 
    response_format = format() 
    # 2. Get data (and validate some more) 
    data = get_series_map(app.config[’FILE_PATH’]) 
    try: 
        dataset = anscombe_filter(series_id, data)._as_listofdicts() 
    except KeyError: 
        abort(404, "Unknown Series") 
    # 3. Prepare Response 
    try: 
        content_bytes = serialize(response_format, dataset, document_tag="Series", row_tag="Pair") 
        response = make_response( 
            content_bytes, 200, {"Content-Type": response_format} 
        ) 
        return response 
    except KeyError: 
        abort(404, f"Unknown {response_format=}")

This function has a similar structure to the previous index_view() function. The request is validated, the data acquired, and a response prepared. As with the previous function, the work is delegated to two other data access functions: get_series_map() and anscombe_filter(). These are separate from the web application, and could be part of a command-line application.

Both of these functions depend on an underlying data access layer. We’ll look at those functions in the next section.

15.3.2 The data access tier

The get_series_map() function is similar to the examples shown in the Cleaning raw data with generator functions section of Chapter 3, Functions, Iterators, and Generators. In this section, we’ll include some important changes. We’ll start with the following two NamedTuple definitions:

from Chapter03.ch03_ex4 import ( 
    series, head_split_fixed, row_iter) 
from collections.abc import Callable, Iterable 
from typing import NamedTuple, Any, cast 
 
class Pair(NamedTuple): 
    x: float 
    y: float 
 
    @classmethod 
    def create(cls: type["Pair"], source: Iterable[str]) -> "Pair": 
        return Pair(*map(float, source)) 
 
class Series(NamedTuple): 
    series: str 
    data: list[Pair] 
 
    @classmethod 
    def create(cls: type["Series"], name: str, source: Iterable[tuple[str, str]]) -> "Series": 
        return Series(name, list(map(Pair.create, source))) 
 
    def _as_listofdicts(self) -> list[dict[str, Any]]: 
        return [p._asdict() for p in self.data]

We’ve defined a Pair named tuple and provided a @classmethod to build instances of a Pair. This definition will automatically provide an _asdict() method that responds with a dictionary of the form dict[str, Any] containing the attribute names and values. This is helpful for serialization.

Similarly, we’ve defined a Series named tuple. The create() method can build a tuple from an iterable source of lists of values. The automatically provided _asdict() method can be helpful for serializing. For this application, however, we’ll make use of the _as_listofdicts method to create a list of dictionaries that can be serialized.

The function to produce the mapping from series name to Series object has the following definition:

from pathlib import Path 
 
def get_series_map(source_path: Path) -> dict[str, Series]: 
    with source_path.open() as source: 
        raw_data = list(head_split_fixed(row_iter(source))) 
        series_iter = ( 
            Series.create(id_str, series(id_num, raw_data)) 
            for id_num, id_str in enumerate( 
                [’I’, ’II’, ’III’, ’IV’]) 
        ) 
        mapping = { 
            series.series: series 
            for series in series_iter 
        } 
    return mapping

The get_series_map() function opens the local data file, and applies the row_iter() function to each line of the file. This parses the line into a row of separate items. The head_split_fixed() function is used to remove the heading from the file. The result is a tuple-of-list structure, which is assigned the variable raw_data.

From the raw_data structure, the Series.create() method is used to transform a sequence of values from the file into a Series object composed of individual Pair instances. The final step is to use a dictionary comprehension to collect the individual Series instances into a single mapping from series name to Series object.

Since the output from the get_series_map() function is a mapping, we can do something like the following example to pick a specific series by name:

>>> source = Path.cwd() / "Anscombe.txt" 
>>> get_series_map(source)[’I’] 
Series(series=’I’, data=[Pair(x=10.0, y=8.04), Pair(x=8.0, y=6.95), ...])

Given a key, for example, ’I’, the series is a list of Pair objects that have the x, y values for each item in the series.

def anscombe_filter( 
    set_id: str, raw_data_map: dict[str, Series] 
) -> Series: 
    return raw_data_map[set_id]

from collections.abc import Callable 
from typing import Any, TypeAlias 
 
Serializer: TypeAlias = Callable[[list[dict[str, Any]]], bytes]

SERIALIZERS: dict[str, Serializer] = {
'application/xml': serialize_xml,
'text/html': serialize_html,
'application/json': serialize_json,
'text/csv': serialize_csv,
}

def serialize( 
    format: str | None, 
    data: list[dict[str, Any]], 
    **kwargs: str 
) -> bytes: 
    """Relies on global SERIALIZERS, set separately""" 
    if format is None: 
        format = "text/html" 
    function = SERIALIZERS.get( 
        format.lower(), 
        serialize_html 
    ) 
    return function(data, **kwargs)

from collections.abc import Callable 
from typing import TypeVar, ParamSpec 
from functools import wraps 
 
T = TypeVar("T") 
P = ParamSpec("P") 
 
def to_bytes( 
    function: Callable[P, str] 
) -> Callable[P, bytes]: 
    @wraps(function) 
    def decorated(*args: P.args, **kwargs: P.kwargs) -> bytes: 
        text = function(*args, **kwargs) 
        return text.encode("utf-8") 
    return decorated

import json 
 
@to_bytes 
def serialize_json(data: list[dict[str, Any]], **kwargs: str) -> str: 
    text = json.dumps(data, sort_keys=True) 
    return text

import csv 
import io 
 
@to_bytes 
def serialize_csv(data: list[dict[str, Any]], **kwargs: str) -> str: 
    buffer = io.StringIO() 
    wtr = csv.DictWriter(buffer, sorted(data[0].keys())) 
    wtr.writeheader() 
    wtr.writerows(data) 
    return buffer.getvalue()

<?xml version="1.0" encoding="UTF-8"?> 
<Series> 
<Pair><x>2</x><y>3</y></Pair> 
<Pair><x>5</x><y>7</y></Pair> 
</Series>

15.6.1 WSGI application: welcome

In the The WSGI standard section of this chapter, a routing application was described. It showed three application routes, including paths starting with /demo and a special case for the path /index.html.

Creating applications via WSGI can be challenging. Build a function, welcome_app(), that displays an HTML page with some links for the demo app and the static download app.

A unit test for this application should use a mocked StartResponse function, and a mocked environment.

15.6.2 WSGI application: demo

In the The WSGI standard section of this chapter, a routing application was described. It showed three application routes, including paths starting with /demo and a special case for the /index.html path.

Build a function, demo_app(), to do some potentially useful activity. The intent here is to have a path that responds to an HTTP POST request to do some work, creating an entry in a log file. The result must be a redirect (status 303, usually) to a URL that uses the static_text_app() to download the log file. This behavior is described as Post/Redirect/Get, and allows for a good user experience when navigating back to a previous page. See https://www.geeksforgeeks.org/post-redirect-get-prg-design-pattern/ for more details on this design pattern.

Here are two examples of useful work that might be implemented by the demo application:

• A GET request can present an HTML page with a form. The submit button on the form can make a POST request to do a computation of some kind.

• A POST request can execute doctest.testfile() to run a unit test suite and collect the resulting log.

15.6.3 Serializing data with XML

In the Serializing data with XML and HTML section of this chapter, we described two additional features of the RESTful API built using Flask.

Extend the response in those examples to serialize the resulting data into XML in addition to CSV and JSON. One alternative to adding XML serialization is to download and install a library that will serialize Series and Pair objects. Another choice is to write a function that can work with a list[dict[str, Any]] object. Adding the XML serialization format also requires adding test cases to confirm the response has the expected format and content.

15.6.4 Serializing data with HTML

In the Serializing data with XML and HTML section of this chapter, we described two additional features of the RESTful API built using Flask.

Extend the response in those examples to serialize the resulting data into HTML in addition to CSV and JSON. HTML serialization can be more complex than XML serialization because there is quite a bit of overhead in an HTML presentation of data. Rather than a representation of the Pair objects, it is common practice to include an entire HTML table structure that mirrors the CSV rows and columns. Adding the HTML serialization format also requires adding test cases to confirm the response has the expected format and content.