As with any large software project, Django encompasses a large number of concepts, features, and tools, and the set of problems it was designed to solve—namely Web development—also has a large scope. To start learning the details about using Django, you need to understand those problems and the methods frameworks such as Django use to solve them.
This chapter introduces these basic ideas, starting with a tool-agnostic look at the Web, following with explanations of the Web framework model and its constituent parts, and wrapping up with the general development philosophy employed by the creators of Django. The high-level overview in the previous chapter also gives you a bit of context here.
An important note: If you already have a solid background in Web development, some of these concepts should be old hat—but even intermediate and experienced Web developers can benefit from taking a step back and reviewing the fundamentals of the practice. It is all too common to find ourselves mentally constrained by the specific language or toolset at hand, whereas a greater perspective often presents solutions previously hidden from view.
Having a solid handle on these core concepts makes you a better problem solver and allows you to make better choices, both during design and implementation. So please—don’t skip ahead!
At its heart, Web development is conceptually simple. Users request a document from the Web server; the Web server fetches or generates the requested document; the server returns the result to the user’s browser; and the browser renders the document. The details tend to vary, but that’s really all there is to it. Let’s break this down as it applies to Web frameworks such as Django.
HTTP (HyperText Transfer Protocol) encapsulates the entire process of serving Web pages and is the foundation for the Web. Because it’s a protocol for client-server communication, it largely consists of requests (client to server) and responses (server to client). What happens on the server between the two isn’t covered by HTTP and is up to the server software (see the following).
The concept of a request encapsulates the first part of the process—the client asking the server for a given document. The heart of a request is the URL—the “path” to the document being requested—but it can be further parameterized via a number of methods, enabling a single location or URL to exhibit multiple behaviors.
A response consists primarily of a body—usually the text of a Web page—and accompanying headers with extra information about the data in question, such as when it was last updated, how long it should be cached locally, its content type, and so forth. Other, non-HTML content included in a response could be plain text, images, documents (PDF, Word, Excel, and so on), audio clips, and so forth.
Django represents both requests and responses as relatively simple Python objects with attributes for the varying pieces of data and methods for more complex operations.
Looked at simply, the Web is about data transfer or the sharing of content (meaning, quite literally, anything—blog entries, financial data, ebooks, and so forth). In the Web’s early days, content consisted of HTML text files, written by hand, and stored on the server’s filesystem. This is known as static because requests to the same URL always return the same information. The “path” described previously was more primitive; there were no parameters, as it was merely a path on the server’s filesystem where the static content was located. Present day, most content is considered dynamic because the data returned by a given URL can vary tremendously depending on factors.
A large part of this dynamic nature is enabled by storing data in a database, where instead of a single string of text, one can create multipart pieces of data and link them to one another to represent relationships. To define and query the database, SQL (Structured Query Language) is used, often further abstracted by an ORM (Object-Relational Mapper), which enables object-oriented programming languages to represent the database as code objects.
SQL databases are organized into tables, each consisting of rows (for example, entries, items, objects) and columns (for example, attributes, fields), similar in overall organization to a spreadsheet. Django provides a powerful ORM, where Python classes represent tables, objects represent individual rows within those tables, and the table’s columns are attributes of those objects.
The final piece of the Web development puzzle is how to present or format the information requested and/or returned via HTTP and queried from the SQL database. Typically, this is done in HTML (HyperText Markup Language) or its newer, more XML-like cousin XHTML, along with the sister languages of JavaScript for dynamic browser-side functionality and CSS (Cascading Style Sheets) for visual styling. Newer applications also use JSON (a “light” data format) or XML to enable dynamic content.
To work with data being presented, most Web frameworks provide a template language, which blends raw HTML tags with a programming-like syntax for looping over collections of objects, performing logic operations, and other constructs that enable the dynamic behavior desired. A simple example could be an otherwise static HTML document with a piece of logic that says to display the username of the currently logged-in user or to display a “Login” link if the user is not yet logged in.
Some templating systems attempt to be fully XHTML compliant, implementing their programming-like commands as HTML attributes or tags, so the resultant document can be parsed as normal HTML. Others emulate regular programming languages more closely, sometimes with an “alternative tag” syntax where the programming constructs are surrounded with special characters for ease of reading and parsing. Django’s template language is one of the latter.
While organizing the Web into the three components outlined previously, one important aspect has been omitted: how they interact with one another. How does a Web application know to execute a SQL query based on a request, and how does it know what template to use when rendering the result?
The answer depends partly on the tools used: Each Web framework or language can approach things in a different way. However, there are generally more similarities than there are differences, and although the next two sections outline Django’s own approach, many of these concepts can be found in other frameworks as well.
As you’ve just seen, dynamic Web development is often broken down into a handful of core components. In this section, we expand further on those concepts, discussing the programming methodologies involved and an overview of how Django implements them (with details and examples in chapters to come).
The idea of breaking down a dynamic application (Web or otherwise) has been around for some time, usually applied to graphical client-side applications, and is generally known as the MVC (Model-View-Controller) paradigm. As you can expect, this means the application is segregated into the model, which controls the data, the view, which defines how to display data, and a controller, which mediates between the two and enables the user to request and manipulate the data.
Compartmentalizing an application in such a manner enables the programmer to be flexible and encourages code reuse among other things. For example, a given view—say, a module that knows how to display graphs of numeric data—could be used on top of various different sets of data, so long as the glue between the two is able to tie them together. Or a specific, single set of data could be displayed in multiple different output formats, such as the aforementioned graph view, a flat text file, or a sortable table. Multiple controllers could enable varying levels of access to the same data model for different users or enable data entry via a GUI application as well as via e-mail or the command line.
The key to successfully leveraging an MVC architecture lies in correctly segregating these different layers of the application. Having one’s data model store information about how it should be displayed, although potentially convenient for some setups, means it is much harder to completely swap out one view for another. And having database-specific access codes in one’s graphical layout code would cause no end of headaches if you decide to switch database platforms!
Django adheres to this separation of duties, although it does so in a slightly different manner than the norm. The model aspect stays the same: Django’s model layer deals solely with passing data into and out of a database. However, the “view” in Django isn’t the final step in displaying the data—Django’s views are closer to what are normally considered the “controller” aspects of MVC. They’re Python functions which tie together the model layer and the presentation layer (which consists of HTML and Django’s template language to be covered later in Chapter 6, “Templates and Form Processing”). To quote the Django development team:
In our interpretation of MVC, the “view” describes the data that gets presented to the user. It’s not necessarily how the data looks, but which data is presented. The view describes which data you see, not how you see it. It’s a subtle distinction.
In other words, Django splits the presentation layer in twain with a view method defining what data to display from the model and a template defining the final representation of that information. As for the controller, the framework itself serves as a controller of sorts—it provides mechanisms, which determine what view and template are used to respond to a given request.
The basis for any application, Web or otherwise, is the information it presents, gathers, and modifies. As such, if we examine an application as a set of layers, the model is the bottom layer, the foundation. Views and templates can come and go, changing how data enters and leaves the model and how it is presented, but the model is relatively set in stone.
From the perspective of designing a full-stack Web application, the model is possibly both the easiest to grasp and the hardest to master. Modeling a real-world problem in an object-oriented system is often a comparatively simple task, but for high-traffic Web sites the most realistic model isn’t always the most efficient.
The model encompasses a wide range of potential pitfalls, one of which is changing the model code after your application has been deployed. Although you are “just changing code,” you have actually altered your database schema under the covers, and this often causes side effects to the preexisting data stored in the database. We go over many of these real-life concerns in the chapters ahead when exploring the design of some example applications.
Views form much (sometimes most or all) of the logic in Django applications. Their definition is deceptively simple: Python functions are linked to one or more defined URLs, which return HTTP response objects. What happens in-between those two endpoints of Django’s HTTP mechanisms is entirely up to you. In practice, there are usually a few, similar tasks performed at this stage, such as displaying an object or list of objects from the model or adding new such objects, along with bookkeeping-like tasks such as checking the status of an authenticated application user and either enabling or rejecting access.
Django provides many shortcuts and helper functions for tasks such as these, but you can write everything yourself for full control over the process, for heavy use of the shortcuts for rapid prototyping and development, or for combining the two approaches. Flexibility and power are the name of the game here.
You should have noticed we just stated the view is responsible for displaying objects from the model. That’s not 100 percent true. To the extent that view methods just have to return an HTTP response, it’s true enough—you could write out a string in Python and return that and be none the worse for wear. However, in the vast majority of cases, it’s terribly inefficient to do so, and as mentioned previously, separation of layers is important to adhere to.
Instead, most Django developers use its template language to render the HTML that Web applications so often result in. Templates are essentially HTML text documents with special formatting denoting where to output dynamic values, enabling simple logic constructs such as loops, and so forth. When a view wants to return an HTML document, it usually specifies a template, gives it the information to display, and uses that rendered template in its response.
Although HTML is the most common format, templates don’t actually have to contain any—they can be used to create any text format, such as comma-separated values (CSV) or even e-mail message body text. The important thing is they enable a Django project to separate the presentation of its data from the view code which decides what data to present.
In this chapter thus far, we’ve covered some of the large architectural components which make up an actual Django system as well as supporting cast just outside its boundaries. Let’s put them all together to give you an overall perspective. In Figure 3.1, you can see the HTTP communication protocol is the closest to the user. Using URLs, they can send requests to Django Web applications and can receive responses back to their Web clients, which may also be running JavaScript with Ajax doing any out-of-band server access.
At the opposite end of the spectrum (at the bottom of the figure), you see the database is the persistent storage which is managed under the guidance of your models and the Django ORM, communicating to the database via Python’s DB-API as well as the database’s client library in the form of an adapter, usually written in C/C++ with a Python interface.
Last but not least, in between we have Django, the heart of the application. Django’s MVC paradigm is equivalently spelled out as “MTV” in Django-speak. The views, serving in controller capacity navigate between creating, updating, and deleting the data model through to the database via the ORM while managing the final view to users given its templates.
Connecting the pieces together, the HTTP requests that come in are forwarded by the Web server to Django, which accepts them starting at the request middleware layer. They are then dispatched based on URLconf patterns to the appropriate view, which performs the core part of the work required, using models and/or templates as necessary to create the response. The response then goes through one more layer of middleware that performs any final processing before returning the HTTP response back to the Web server to forward on to the user. Make sense?
As a full-stack Web framework initially developed by a small, tightly knit group of programmers, Django has been and continues to be designed with a fairly specific set of philosophies in mind. These ideals reflect the experiences (and to a degree, the personalities) of the core team, but at the same time they tend to line up very well with what any Web developer using any toolkit would agree are “best practices.” Understanding these philosophies helps you understand, and make better use of, the framework.
The programming language used, and that language’s community, is often one of the larger influences on any software project’s design, and this is no different with Django. Users of Python tend to describe things that mesh well with and generally adhere to the philosophies of the language as being Pythonic; although there is no explicit definition of the term, it generally means code exhibits various attributes common to other works in the language.
Among these attributes are the use of terse but powerful syntax (the default syntax of for loops or the even-more-concise tool of list comprehensions); the idea there is usually only one right way to do any given simple task (such “ways” are often incorporated into the language itself, such as the get dictionary method) and favoring the explicit more than the implicit (such as the requirement for a self argument in all object methods).
As we see in the example application chapters in Part III, “Django Applications by Example,” many Django conventions, methods, and design decisions are Pythonic or strive to be. This makes the framework easier to pick up for programmers with Python experience and also helps to ingrain good programming practices in less experienced developers.
One Pythonic attribute that deserves its own section is a principle common to almost all programming: DRY, or Don’t Repeat Yourself. DRY is perhaps the simplest programming idiom of all because it’s just plain old common sense: If you have information in more than one place and it needs to change, you’ve just made twice as much (or more) work for yourself.
As an example of DRY, consider the need to perform a simple calculation on a few pieces of data, such as the sum of a collection of bank accounts associated with a given individual. In a poorly designed system, this summation can be performed in multiple places throughout the code: pages listing individuals, a page for detailed per-individual information, or a page displaying grand totals for multiple individuals. In a system such as Django’s ORM, you can easily honor DRY by creating a Person class with a sum_accounts method defined only once and then used in all the previous locations.
Although DRY can be easy to apply to simple situations such as the previous example, it’s also one of the hardest commandments to adhere to strictly all the time; there are many places where it conflicts with other idioms, Pythonic and otherwise, where tradeoffs must be made. However, it is a worthy goal to strive toward and one which becomes easier with experience.
Django is a full-stack Web framework in the sense it provides all necessary components for a dynamic Web application: database access, request framework, application logic, templating, and so forth. However, an effort has been made to ensure that users’ options are left open: You can use as much or as little of Django as you need and can replace components with other tools as you see fit.
For example, some users dislike Django’s template system and prefer alternatives such as Kid or Cheetah. Django view methods don’t require that Django’s template system be used, so it’s entirely possible to have one’s views load up Kid or Cheetah, render a template written for those systems, and return the result as part of a Django response object.
The same goes for the database layer. Users who prefer SQLAlchemy or SQLObject, for example, can simply ignore Django’s ORM entirely and work with their data via other tools. Conversely, if less common, it’s possible to utilize only Django’s ORM for other projects (even non-Web-oriented ones); it functions with only a minimum of setup.
When all is said and done, however, such modularity comes at a price: Some of Django’s nicest shortcuts necessarily encompass the stack as a whole, such as the generic view methods enabling simple display and updating and creating database records. As such, this modular approach to Django is usually best avoided by those new to Python Web development.
Django was written with rapid, agile development in mind. Working in a fast-paced local newspaper shop, the core team needed a set of tools that would allow them to implement an idea in an extremely short amount of time. The open-sourcing of the framework has not changed the fact that it excels in this area.
Django provides shortcuts at a couple of different levels. The most obvious is the aforementioned generic view collection, which consists of perhaps a dozen or so common tasks. Combined with flexible and powerful parameterization, these generic views can, and often do, make up the entirety of a Web site, enabling creation and modification of database records, display of lists of objects (date-oriented and otherwise) and individual object pages, and more. With only three Python files—site-specific settings, a model declaration, and a map linking URLs to generic views—and some HTML templates, one can create an entire Web site in a matter of minutes or hours.
At a lower level, Django provides many shortcut methods for common tasks at the Python view level itself, so when generic views can’t provide what the programmer needs, they can still avoid a lot of boilerplate. Such shortcuts exist for rendering templates with data dictionaries, obtaining a database object, returning an HTTP error if one doesn’t exist, processing forms, and so forth.
Combined with the flexibility, terseness, and power of the Python language, these shortcuts enable programmers to focus on getting projects built and out the door quickly, and/or solving domain-specific problems without worrying about grunt work or so-called “glue code.”
We’ve covered a lot of ground in this chapter: the fundamentals of what Web development is all about, how Django and similar frameworks organize their approach to creating Web sites, and the underlying philosophies driving Django’s own development and design decisions. Regardless of what you brought to this chapter, we hope you’ve gained something from the overview.
At this point in the book, you should now have a decent background in both the basics of developing Web applications as well as the underlying theory and organization of a typical Web framework. In Part II, “Django in Depth,” we dive into the details of how to use Django, exploring the various classes, functions, and data structures it uses and showing you more code snippets to help it all make sense.