This chapter discusses how to use a major web application framework with the Python runtime environment. Java developers may be interested in the general discussion of frameworks that follows, but the rest of this chapter is specific to Python. Several frameworks for Java are known to work well with App Engine, and you can find information on these on the Web.
As with all major categories of software, web applications have a common set of problems that need to be solved in code. Most web apps need software to interface with the server’s networking layer, communicate using the HTTP protocol, define the resources and actions of the application, describe and implement the persistent data objects, enforce site-wide policies such as access control, and describe the browser interface in a way that makes it easily built and modified by designers. Many of these components involve complex and detailed best practices for interoperating with remote clients and protecting against a variety of security vulnerabilities.
A web application framework is a collection of solutions and best practices that you assemble and extend to make an app. A framework provides the structure for an app, and most frameworks can be run without changes to demonstrate that the initial skeleton is functional. You use the toolkit provided by the framework to build the data model, business logic, and user interface for your app, and the framework takes care of the details. Frameworks are so useful that selecting one is often the first step when starting a new web app project.
Notice that App Engine isn’t a web application framework, exactly. App Engine provides scaling infrastructure, services, and interfaces that solve many common problems, but these operate at a level of abstraction just below most web app frameworks. A better example of a framework is webapp2, a framework for Python included with the App Engine Python SDK that we’ve been using in examples throughout the book so far. webapp2 lets you implement request handlers as Python classes, and it takes care of the details of interfacing with the Python runtime environment and routing requests to handler classes.
Several major frameworks for Python work well with App Engine. Django, Pyramid, web2py, and Flask work well, and some frameworks have explicit support for App Engine. These frameworks are mature, robust, and widely used, and have large thriving support communities and substantial online documentation. You can buy books about some of these frameworks.
Not every feature of every framework works with App Engine. Most
notably, many frameworks include a mechanism for defining data models, but
these are usually implemented for relational databases, and don’t work with
the App Engine datastore. In some cases, you can just replace the
framework’s data modeling library with App Engine’s ext.db
library (or ext.ndb
). Some features of frameworks also have
issues running within App Engine’s sandbox restrictions, such as by
depending upon unsupported libraries. Developers have written adapter
components that work around many of these issues.
In general, to use a framework, you add the framework’s libraries to your application directory and then map all dynamic URLs (all URLs except those for static files) to a script that invokes the framework. Because the interface between the runtime environment and the app is WSGI, you can associate the framework’s WSGI adapter with the URL pattern in app.yaml, just as we did with webapp2. Most frameworks have their own mechanism for associating URL paths with request handlers, and it’s often easiest to send all dynamic requests to the framework and let it route them. You may still want to use app.yaml to institute Google Accounts–based access control for some URLs.
Django is a popular web application framework for Python, with a rich stack of features and pluggable components. It’s also large, consisting of thousands of files. To make it easier to use Django on App Engine, the Python runtime environment includes the Django libraries, so you do not have to upload all of Django with your application files. The Python SDK bundles several versions of Django as well.
App Engine 1.7.0 provides Django 1.3. This version of Django does not include explicit support for App Engine, but many features work as documented with minimal setup. In this chapter, we discuss how to use Django via the provided libraries, and discuss which of Django’s features work and which ones don’t when using Django this way.
We also look at a third-party open source project called django-nonrel, a fork of Django 1.3 that enables most of Django’s features for App Engine. In particular, it connects most of Django’s own data modeling features to the App Engine datastore, so components and features that need a database can work with App Engine, without knowing it’s App Engine behind the scenes. The data modeling layer also provides a layer of portability for your app, so your app code can run in environments that provide either a relational database or a nonrelational datastore (hence “nonrel”).
The official documentation for Django is famously good, although it relies heavily on the data modeling features for examples. For more information about Django, see the Django project website:
http://www.djangoproject.com/ |
As of this writing, Google is testing a new service for hosting SQL databases, called Google Cloud SQL. The service provides a fully functional MySQL instance running on Google servers, which you can use with software like the Django application framework. However, a hosted SQL database does not scale automatically like the App Engine datastore. Check the Google Cloud website for updates on this feature. In this chapter, we’ll assume your app will use the App Engine datastore for storage.
The App Engine 1.7.0 Python SDK provides Django 1.3 in its
lib/django_1_3/ subdirectory. With
Django, you use a command-line tool to set up a new web application
project. This tool expects lib/django_1_3/ to be in the Python library
load path, so it can load modules from the django
package.
One way to set this up is to add it to the PYTHONPATH
environment variable on your platform. For example, on the Mac OS X or
Linux command line, using a bash-compatible shell, run this command to
change the environment variable for the current session to load Django 1.3
from the Python SDK located at ~/google_appengine/:
export PYTHONPATH=$PYTHONPATH:~/google_appengine/lib/django_1_3
The commands that follow will assume the SDK is in ~/google_appengine/, and this
PYTHONPATH
is set.
The Django library is available in the runtime environment by using
a libraries:
directive in app.yaml, just like other libraries. We’ll see
an example of this in a moment.
Django 1.3 is the most recent version included with the Python runtime environment as of App Engine version 1.7.0. Later versions of Django are likely to be added to the runtime environment in future releases.
Instructions should be similar for later versions, although it
isn’t clear whether all future versions of Django will be added to the
Python SDK. All previously included versions must remain in the SDK for
compatibility, and the SDK might get a little large if it bundled every
version. You may need to install Django on your local computer
separately from the Python SDK in future versions. Django will likely be
included in the runtime environment itself, similar to other third-party
Python libraries. If you install Django yourself, you do not need to
adjust the PYTHONPATH
, and can run the Django commands
without the library path.
Check the App Engine website for updates on the inclusion of future versions of Django in the runtime environment.
For this tutorial, we will create an App Engine application that contains a Django project. In Django’s terminology, a project is a collection of code, configuration, and static files. A project consists of one or more subcomponents called apps. The Django model encourages designing apps to be reusable, with behavior controlled by the project’s configuration. The appearance of the overall website is also typically kept separate from apps by using a project-wide template directory.
To start, create the App Engine application root directory. From the Mac OS X or Linux command line, you would typically use these commands to make the directory and set it to be the current working directory:
mkdir myapp cd myapp
You create a new Django project by running a command called
django-admin.py startproject
. Run this command to create a
project named myproject
in a subdirectory called myproject/:
python ~/google_appengine/lib/django_1_3/django/bin/django-admin.py startproject myproject
This command creates the myproject/ subdirectory with several starter files:
A file that tells Python that code files in this directory can be imported as modules (this directory is a Python package).
A command-line utility you will use to build and manage this project, with many features.
Configuration for this project, in the form of a Python source file.
Mappings of URL paths to Python code, as a Python source file.
The django-admin.py
tool has many features, but
most of them are specific to managing SQL databases. This is the last time
we’ll use it here.
If you’re following along with a Django tutorial or book, the next
step is usually to start the Django development server by using the
manage.py
command. If you did so now, you would
be running the Django server, but it would know nothing of App Engine. We
want to run this application in the App Engine development server. To do
that, we need a couple of additional pieces.
To connect our Django project to App Engine, we need a short script that instantiates the Django WSGI adapter, and an app.yaml configuration file that maps all (nonstatic) URLs to the Django project.
Create a file named main.py in the application root directory with the following contents:
import os os.environ['DJANGO_SETTINGS_MODULE'] = 'myproject.settings' import django.core.handlers.wsgi application = django.core.handlers.wsgi.WSGIHandler()
The first two lines tell Django where to find the project’s
settings
module, which in this case is at the module path
myproject.settings
(the myproject/settings.py file). This must be set
before importing any Django modules. The remaining two lines import the
WSGI adapter, instantiate it, and store it in a global variable.
Next, create app.yaml in the application root directory, like so:
application: myapp version: 1 runtime: python27 api_version: 1 threadsafe: yes handlers: - url: .* script: main.application libraries: - name: django version: "1.3"
This should be familiar by now, but to review, this tells App Engine
this is an application with ID myapp
and version ID
1
running in the Python 2.7 runtime environment, with
multithreading enabled. All URLs are routed to the Django project we just
created, via the WSGI adapter instantiated in main.py. The libraries:
declaration selects Django 1.3 as the version to use when importing
django
modules.
Our directory structure so far looks like this:
myapp/ app.yaml main.py myproject/ __init__.py manage.py settings.py urls.py
We can now start this application in a development server. You can add this to the Python Launcher (
menu, ) and then start the server, or just start the development server from the command line, using the current working directory as the application root directory:dev_appserver.py .
Load the development server URL (such as http://localhost:8080/) in a browser, and enjoy the welcome screen (Figure 18-1).
The next step is to add a Django “app” to the Django
project. You create new Django apps by using the project’s
manage.py
script, which was generated when you created the
project. (You can also use django-admin.py
to create
apps.)
With the current working directory still set to the application
root, create a new app named bookstore
for this
project:
python myproject/manage.py startapp bookstore
This creates the subdirectory myproject/bookstore, with four new files:
This layout is Django’s way of encouraging a design philosophy that
separates data models and request handlers into separate, testable,
reusable units of code. This philosophy comes naturally when using the App
Engine datastore and ext.db
. Django does not depend on this
file layout directly, and you can change it as needed.
In addition to creating the app, it’s useful to tell Django
explicitly that this app will be used by this project. Edit settings.py in the project root directory, and
find the INSTALLED_APPS
value. Set it to a tuple containing
the Python path to the app’s package:
INSTALLED_APPS = ( 'myproject.bookstore', )
(Be sure to include the trailing comma, which tells Python this is a one-element tuple.)
The settings.py file is a
Python source file. It contains settings for the Django framework and
related components, in the form of variables. The
INSTALLED_APPS
setting lists all apps in active use by the
project, which enables some automatic features such as template loading.
(It’s primarily used by Django’s data modeling tools, which we won’t be
using, in favor of App Engine’s ext.db
.)
Let’s define our first custom response. Edit bookstore/views.py, and replace its contents with the following:
from django import http def home(request): return http.HttpResponse('Welcome to the Book Store!')
In Django, a view is a function that is called with a
django.http.HttpRequest
object and returns an
django.http.HttpResponse
object. This view
creates a simple response with some text. As you would expect, the
HttpRequest
provides access to request
parameters and headers. You can set headers and other aspects of the
HttpResponse
. Django provides several useful
ways to make HttpResponse
objects, as well as specialized
response classes for redirects and other return codes.
We still need to connect this view to a URL. URLs for views are set by the project, in the urls.py file in the project directory. This allows the project to control the URLs for all of the apps it uses. You can organize URLs such that each app specifies its own set of mappings of URL subpaths to views, and the project provides the path prefix for each app. For now, we’ll keep it simple, and just refer to the app’s view directly in the project’s URL configuration.
Edit urls.py to contain the following:
from django.conf.urls.defaults import patterns, url urlpatterns = patterns('myproject', url(r'^books/$', 'bookstore.views.home'), )
Like settings.py, urls.py is a Python module. It defines a global
variable named urlpatterns
whose value is returned by the
patterns()
function from the
django.conf.urls.defaults
module. Its first argument is a
Python path prefix that applies to all subsequent view paths; here, we set
it to 'myproject'
, which is the first part of the Python path
to every app in this project. Subsequent arguments are URL patterns, each
returned by the url()
function. In this case, we associate a
regular expression matching a URL to the Python path for the view function
we added.
With the development server still running, load the
/books/
URL in your browser. The app calls the view to
display some text.
Django includes a templating system for building web pages
and other displayable text. The Jinja2 templating library we’ve used
throughout the book so far is based on the Django template system. Their
syntax is mostly similar, but you’ll notice minor differences between the
two systems. As just one example, Jinja2 lets you call methods on template
values with arguments, and so requires that you use parentheses after the
method even when not using arguments: {{ someval.method() }}
.
In Django, templates can call methods of values but cannot pass arguments,
and so the parentheses are omitted: {{ someval.method
}}
.
Django templates are baked into the Django framework, so they’re easy to use. It’s possible to use Jinja2 templates with a Django application, but Jinja2 does not automatically support some of the organizational features of Django.
Let’s update the example to use a Django template. First, we need to
set up a template directory. In the project directory, create a directory
named templates
, and a subdirectory in there named
bookstore
:
mkdir -p templates/bookstore
(The -p
option to mkdir
tells it to create
the entire path of subdirectories, if any subdirectory does not
exist.)
Edit settings.py. We’re going
to make two changes: adding an import statement at the top, and replacing
the TEMPLATE_DIRS
value. It should look something like this,
with the rest of the file not shown:
import os # ... TEMPLATE_DIRS = ( os.path.join(os.path.dirname(__file__), 'templates'), ) # ...
The TEMPLATE_DIRS
setting is a list (actually a tuple)
of directory paths to check when a particular template file is requested.
These paths must be absolute paths in the operating system, so we use
os.path.dirname(__file__)
to get the full path to the project
directory (the directory containing the settings.py file), then
os.path.join(...)
to refer to the templates/ subdirectory.
Inside the templates/bookstore/ subdirectory, create the file index.html with the following template text:
<html> <body> <p>Welcome to The Book Store! {{ clock }}</p> </body> </html>
Finally, edit views.py to look like this:
from django.shortcuts import render_to_response import datetime def home(request): return render_to_response( 'bookstore/index.html', { 'clock': datetime.datetime.now() }, )
Reload the page to see the template displayed by the new view.
The render_to_response()
shortcut function takes as its
first argument the path to the template file. This path is relative to one
of the directories on the TEMPLATE_DIRS
template lookup path.
The second argument is a Python mapping that defines variables to be used
within the template. In this example, we set a template variable named
clock
to be the current datetime.datetime
.
Within the template, {{ code }}
interpolates this value as a
string.
The behavior of the template engine can be extended in many ways.
You can define custom tags and filters to use within templates. You can
also change how templates are loaded, with template loader classes and the
TEMPLATE_LOADERS
setting variable. The filesystem-based
behavior we’re using now is provided by the django.template.loaders.filesystem.Loader
class, which appears in the default settings file created by
Django.
Django includes a powerful feature for building web forms based on data model definitions. The Django forms library can generate HTML for forms, validate that submitted data meets the requirements of the model, and redisplay the form to the user with error messages. The default appearance is useful, and you can customize the appearance extensively.
Django’s data modeling library is similar to (and was the
inspiration for) ext.db
, but it was designed with SQL databases in mind. Django does not (yet) include its own
adapter layer for using its modeling library with the App Engine
datastore. (As we’ll see later in this chapter, this is a primary goal of
the django-nonrel project.) When using the Django library directly as
we’re doing now, we can’t use its data modeling library, nor can we use
any Django features that rely on it.
But there’s good news for web forms: App Engine includes an
adapter library so you can use the Django forms library with
ext.db
data models! This library is provided by the
google.appengine.ext.db.djangoforms
module.
We won’t go into the details of how Django forms work—see the Django
documentation for a complete explanation—but let’s walk through a quick
example to see how the pieces fit together. Our example will use the
following behavior for creating and editing Book
entities:
An HTTP GET request to /books/book/
displays an empty form for creating a new Book
.
An HTTP POST request to /books/book/
processes the book creation form, and either creates the book and
redirects to /books
(the book listing page) or redisplays
the form with errors, if any.
An HTTP GET request to /books/book/1234
displays
the form to edit the Book
entity, with the fields filled
out with the current values.
An HTTP POST request to /books/book/1234
updates
the book with that ID, with the same error-handling behavior as the
book creation form.
Edit urls.py to use a new view
function named book_form()
to handle these URLs:
from django.conf.urls.defaults import patterns, url urlpatterns = patterns('myproject', url(r'^books/book/(d*)', 'bookstore.views.book_form'), url(r'^books/', 'bookstore.views.home'), )
The regular expression '^books/book/(d*)'
captures the
book ID in the URL, if any, and passes it to the view function as an
argument.
Edit models.py in the bookstore/ app directory, and replace its
contents with the following ext.db
model definitions:
from google.appengine.ext import db class Book(db.Model): title = db.StringProperty() author = db.StringProperty() copyright_year = db.IntegerProperty() author_birthdate = db.DateProperty() class BookReview(db.Model): book = db.ReferenceProperty(Book, collection_name='reviews') review_author = db.UserProperty() review_text = db.TextProperty() rating = db.StringProperty(choices=['Poor', 'OK', 'Good', 'Very Good', 'Great'], default='Great') create_date = db.DateTimeProperty(auto_now_add=True)
No surprises here; these are just ext.db
models for
datastore entities of the kinds Book
and
BookReview
.
Now for the views. Edit views.py, and replace its contents with the following:
from django import template from django.http import HttpResponseRedirect from django.shortcuts import render_to_response from google.appengine.ext import db from google.appengine.ext.db import djangoforms from bookstore import models def home(request): q = models.Book.all().order('title') return render_to_response('bookstore/index.html', { 'books': q }) class BookForm(djangoforms.ModelForm): class Meta: model = models.Book def book_form(request, book_id=None): if request.method == 'POST': # The form was submitted. if book_id: # Fetch the existing Book and update it from the form. book = models.Book.get_by_id(int(book_id)) form = BookForm(request.POST, instance=book) else: # Create a new Book based on the form. form = BookForm(request.POST) if form.is_valid(): book = form.save(commit=False) book.put() return HttpResponseRedirect('/books/') # else fall through to redisplay the form with error messages else: # The user wants to see the form. if book_id: # Show the form to edit an existing Book. book = models.Book.get_by_id(int(book_id)) form = BookForm(instance=book) else: # Show the form to create a new Book. form = BookForm() return render_to_response('bookstore/bookform.html', { 'book_id': book_id, 'form': form, }, template.RequestContext(request))
We’ve updated the home()
view to set up a query for
Book
entities, and pass that query object to the template.
Edit templates/bookstore/index.html
to display this information:
<html> <body> <p>Welcome to The Book Store!</p> <p>Books in our catalog:</p> <ul> {% for book in books %} <li>{{ book.title }}, by {{ book.author }} ({{ book.copyright_year }}) [<a href="/books/book/{{ book.key.id }}">edit</a>]</li> {% endfor %} </ul> <p>[<a href="/books/book/">add a book</a>]</p> </body> </html>
Finally, create the template for the form used by the new
book_form()
view, named templates/bookstore/bookform.html:
<html> <body> {% if book_id %} <p>Edit book {{ book_id }}:</p> <form action="/books/book/{{ book_id }}" method="POST"> {% else %} <p>Create book:</p> <form action="/books/book/" method="POST"> {% endif %} {% csrf_token %} {{ form.as_p }} <input type="submit" /> </form> </body> </html>
The BookForm
class is a subclass of google.appengine.ext.db.djangoforms.ModelForm
.
It can examine an App Engine model class (a subclass of
db.Model
) and can render and process forms with fields based
on the model’s property declarations. It knows which model class to use
from the Meta
inner class, whose model
class
attribute is set to our Book
class. The
ModelForm
has useful default rendering and processing
behavior for each of the default property declaration types, and you can
customize this extensively. For now, we’ll use the defaults.
The book_form()
view function takes the HTTP request
object and the book_id
captured by the regular expression in
urls.py as arguments. If the request
method is 'POST'
, then it processes the submitted form;
otherwise it assumes the method is 'GET'
and just displays
the form. In either case, the form is represented by an instance of the
BookForm
class.
If constructed without arguments, the BookForm
represents an empty form for creating a new Book
entity. If
constructed with the instance
argument set to a
Book
object, the form’s fields are prepopulated with the
object’s property values.
To process a submitted form, you pass the dictionary of POST
parameters (request.POST
) to the BookForm
constructor as its first positional argument. If you also provide the
instance
argument, the instance sets the initial
values—including the entity key—and the form data overwrites everything
else, as provided.
The BookForm
object knows how to render the form based
on the model class and the provided model instance (if any). It also knows
how to validate data submitted by the user, and render the form with the
user’s input and any appropriate error messages included. The
is_valid()
method tells you if the submitted data is
acceptable for saving to the datastore. If it isn’t, you send the
BookForm
to the template just as you would when displaying
the form for the first time.
If the data submitted by the user is valid, the
BookForm
knows how to produce the final entity object. The
save()
method saves the entity and returns it; if you set the
commit=False
argument, it just returns
the entity and does not save it, so you can make further changes and save
it yourself. In this example, a successful create or update redirects the
user to /books/
(which we’ve hardcoded in the view for
simplicity) instead of rendering a template.
To display the form, we simply pass the BookForm
object
to a template. There are several methods on the object for rendering it in
different ways; we’ll use the as_p()
method to display the
form fields in <p>
elements. The template is
responsible for outputting the <form>
tag and the
Submit button. The BookForm
does the rest.
Restart the development server, then load the book list URL
(/books/
) in the browser. Click “add a book” to show the book
creation form. Enter some data for the book, then submit it to create
it.
The default form widget for a date field is just a text field, and
it’s finicky about the format. In this case, the “author birthdate”
field expects input in the form YYYY-MM-DD
, such as
1902-02-27
.
Continue the test by clicking the “edit” link next to one of the books listed. The form displays with that book’s data. Edit some of the data and then submit the form to update the entity.
Also try entering invalid data for a field, such as nonnumeric data for the “copyright year” field, or a date that doesn’t use the expected format. Notice that the form redisplays with your original input, and with error messages.
The main thing to notice about this example is that the data model
class itself (in this case Book
) completely describes the
default form, including the display of its form fields and the validation
logic. The default field names are based on the names of the properties.
You can change a field’s name by specifying a verbose_name
argument to the property declaration on the model class:
class Book(db.Model): title = db.StringProperty(verbose_name="Book title") # ...
See the Django documentation for more information about customizing the display and handling of forms, and other best practices regarding form handling.
What we’ve seen so far are just some of the features of the Django framework that are fully functional on App Engine. Django’s component composition, URL mapping, templating, and middleware let you build, organize, and collaborate on large, complex apps. With a little help from an App Engine library, you can use App Engine datastore data models with Django’s sophisticated form rendering features. And we’ve only scratched the surface.
The Achilles’ heel of this setup is the datastore. Django was built first and foremost on top of a data management library. Its earliest claim to fame was its web-based data administration panel, from which you could manage a website like a content management system, or debug database issues without a SQL command line. Any Django component that needs to persist global data uses the data library. Just one example is Django’s sessions feature, an important aspect of most modern websites.
This data library is built to work with any of several possible
database backends, but as of Django 1.3, the App Engine datastore is not
yet one of them. The django-nonrel project was founded to develop backends
for the Django data library that work with several popular nonrelational
datastores, including App Engine. The sister project djangoappengine
also implements backend
adapters for several other App Engine services, so Django library
components can do things like send email via the email service.
djangoappengine
also includes improved support for features
of the Django app management tools.
django-nonrel is a fork of Django 1.3. To use it, you download and unpack several components, which together make up the entirety of Django 1.3, with the necessary modifications. These files become your application directory. When you upload your application, all of these files go with it; it ignores the version of Django included in the runtime environment.
As of this writing, management of the django-nonrel project is in
transition. Even in it’s current form, it’s a useful addition to your
toolkit. To download django-nonrel for App Engine, and to read more about
its features and limitations, visit the djangoappengine
project website:
http://www.allbuttonspressed.com/projects/djangoappengine |
3.145.156.250