GridFS is actually a convention, implemented in the client, for storing large blobs of binary data in MongoDB. MongoDB documents are limited in size to (currently) 16 MB. This means that if your blob of data is larger than 16 MB, or might be larger than 16 MB, you need to split the data over multiple documents.

This is just what GridFS does. In GridFS, each blob is represented by:

While GridFS is not optimized in the same way as a traditional distributed filesystem, it is often more convenient to use. In particular, it’s convenient to use GridFS:

As always, it’s best to test with your own data to see if GridFS is a good fit for you.

GridFS stores data in two collections—in this case, cms.assets.files, which stores metadata, and cms.assets.chunks, which stores the data itself. Consider the following prototype document from the cms.assets.files collection:

{ _id: ObjectId(…),
  length: 123...,
  chunkSize: 262144,
  uploadDate: ISODate(…),
  contentType: 'image/jpeg',
  md5: 'ba49a...',
  metadata: {
    nonce: ObjectId(…),
    slug: '2012-03-invisible-bicycle',
    type: 'photo',
    locked: ISODate(...),
    parent_id: ObjectId(...),
    title: 'Kitteh',
    created: ISODate(…),
    author: { _id: ObjectId(…), name: 'Jared' },
    tags: [ … ],
    detail: {
      filename: 'kitteh_invisible_bike.jpg',
      resolution: [ 1600, 1600 ], … }
  }
}

Note that in this example, most of our document looks exactly the same as a basic page document. This helps to facilitate querying nodes, allowing us to use the same code for either a photo or a basic page. This is facilitated by the fact that GridFS reserves the metadata field for user-defined data.

The most common operations inside of a CMS center on creating and editing content. Consider the following insert operation:

db.cms.nodes.insert({
    'metadata': {
        'nonce': ObjectId(),
        'parent_id': ObjectId(...),
        'slug': '2012-03-noticed-the-news',
        'type': 'blog-entry',
        'title': 'Noticed in the News',
        'created': datetime.utcnow(),
        'author': { 'id': user_id, 'name': 'Rick' },
        'tags': [ 'news', 'musings' ],
        'detail': {
            'publish_on': datetime.utcnow(),
            'text': 'I noticed the news from Washington today…' }
        }
     })

One field that we’ve used but not yet explained is the nonce field. A nonce is a value that is designed to be only used once. In our CMS, we’ve used a nonce to mark each node when it is inserted or updated. This helps us to detect problems where two users might be modifying the same node simultaneously. Suppose, for example, that Alice and Bob both decide to update the About page for their CMS, and the sequence goes something like this (Figure 6-1):

Figure 6-1. Alice and Bob edit collision

A nonce can fix this problem by ensuring that updates to a node only succeed when you’re updating the same version of the document that you’re editing, as shown in Figure 6-2.

Figure 6-2. Alice and Bob edit collision detected

By using our nonce field, we can detect editing collisions and give the user the opportunity to resolve them. Our update operation for a content node, then, looks something like the following:

def update_text(parent_id, slug, nonce, text):
    result = db.cms.nodes.update(
        { 'metadata.parent_id': parent_id,
          'metadata.slug': slug,
          'metadata.nonce': nonce },
        { '$set':{'metadata.detail.text': text,
                    'metadata.nonce': ObjectId() } },
        safe=True) 
    if not result['updatedExisting']:
        raise ConflictError() 

We might also want to perform metadata edits to the item such as adding tags:

db.cms.nodes.update(
    { 'metadata.parent_id': parent_id, 'metadata.slug': slug },
    { '$addToSet': { 'tags': { '$each': [ 'interesting', 'funny' ] } } })

In this example, the $addToSet operator will only add values to the tags field if they do not already exist in the tags array; there’s no need to supply or update the nonce.

To support updates and queries on the metadata.parent_id and metadata.slug fields and to ensure that two editors don’t create two documents with the same parent or slug, we can create a unique index on these two fields:

>>> db.cms.nodes.ensure_index([
...    ('metadata.parent_id', 1), ('metadata.slug', 1)], unique=True)

Uploading photos to our CMS requires some extra attention, as the amount of data to be transferred can be substantially higher than for a “normal” content node. This led to our decision to use GridFS for the storage of photos. Furthermore, we would rather not load the entire photo’s data into memory at once, so the following code “streams” data through to MongoDB, one chunk at a time:

def upload_new_photo(
    input_file, parent_id, slug, title, author, tags, details):
    fs = GridFS(db, 'cms.assets')
    now = datetime.utcnow()
    with fs.new_file(
        content_type='image/jpeg', 
        metadata=dict(
            nonce=bson.ObjectId(),
            type='photo',
            locked=now, 
            parent_id=parent_id,
            slug=slug,
            title=title,
            created=now,
            author=author,
            tags=tags,
            detail=detail)) as upload_file:
        while True:
            chunk = input_file.read(upload_file.chunk_size) 
            if not chunk: break
            upload_file.write(chunk)
    # unlock the file
    db.assets.files.update( 
        {'_id': upload_file._id},
        {'$set': { 'locked': None } } )

Because uploading the photo spans multiple documents and is a nonatomic operation, we “lock” the file during upload by writing the current datetime in the record. The following code shows how the locked field is used to manage updates to the photo content:

def update_photo_content(input_file, parent_id, slug):
    fs = GridFS(db, 'cms.assets')

    # Delete the old version if it's unlocked or was locked more than 5
    #    minutes ago
    file_obj = db.cms.assets.find_one(
        { 'metadata.parent_id': parent_id,
          'metadata.slug': slug,
          'metadata.locked': None })
    if file_obj is None:
        threshold = datetime.utcnow() - timedelta(seconds=300)
        file_obj = db.cms.assets.find_one(
            { 'metadata.parent_id': parent_id,
              'metadata.slug': slug,
              'metadata.locked': { '$lt': threshold } })
    if file_obj is None: raise FileDoesNotExist()
    fs.delete(file_obj['_id']) 

    # update content, keep metadata unchanged
    file_obj['locked'] = datetime.utcnow()
    with fs.new_file(**file_obj):
        while True:
            chunk = input_file.read(upload_file.chunk_size)
            if not chunk: break
            upload_file.write(chunk)
    # unlock the file
    db.assets.files.update(
        {'_id': upload_file._id},
        {'$set': { 'locked': None } } )

As with the basic operations, editing tags is almost trivial:

db.cms.assets.files.update(
    { 'metadata.parent_id': parent_id, 'metadata.slug': slug },
    { '$addToSet': { 'metadata.tags': { '$each': [ 'interesting', 'funny']}}})

Since our queries tend to use both metadata.parent_id and metadata.slug, a unique index on this combination is sufficient to get good performance:

>>> db.cms.assets.files.ensure_index([
...    ('metadata.parent_id', 1), ('metadata.slug', 1)], unique=True)

To locate a “normal” node based on the value of metadata.parent_id and metadata.slug, we can use the find_one operation rather than find:

node = db.nodes.find_one({'metadata.parent_id': parent_id,
                          'metadata.slug': slug })

To locate an image based on the value of metadata.parent_id and metadata.slug, we use the GridFS method get_version:

code,sourceCode,python
fs = GridFS(db, 'cms.assets')
with fs.get_version({'metadata.parent_id': parent_id, 'metadata.slug': slug })
  as img_fpo:
     # do something with the image file

To retrieve a list of “normal” nodes based on their tags, the query is straightforward:

nodes = db.nodes.find({'metadata.tags': tag })

To retrieve a list of images based on their tags, we’ll perform a search on cms.assets.files directly:

image_file_objects = db.cms.assets.files.find({'metadata.tags': tag })
fs = GridFS(db, 'cms.assets')
for image_file_object in db.cms.assets.files.find(
    {'metadata.tags': tag }):
    image_file = fs.get(image_file_object['_id'])
    # do something with the image file

In order to make these queries perform well, of course, we need an index on metadata.tags:

>>> db.cms.nodes.ensure_index('metadata.tags')
>>> db.cms.assets.files.ensure_index('metadata.tags')

One common operation in a blog is to find the most recently published blog post, sorted in descending order by date, for use on the index page of the site, or in an RSS or ATOM feed:

articles = db.nodes.find({
    'metadata.parent_id': 'my-blog'
    'metadata.published': { '$lt': datetime.utcnow() } })
articles = articles.sort({'metadata.published': -1})

Since we’re now searching on parent_id and published, we need an index on these fields:

>>> db.cms.nodes.ensure_index(
...     [ ('metadata.parent_id', 1), ('metadata.published', -1) ])

To post a new comment in a chronologically ordered (i.e., without discussion threading) system, we just need to use a regular insert():

slug = generate_pseudorandom_slug()
db.comments.insert({
    'node_id': node_id,
    'slug': slug,
    'posted': datetime.utcnow(),
    'author': author_info,
    'text': comment_text })

To insert a comment for a system with threaded comments, we first need to generate the appropriate slug and full_slug values based on the parent comment:

posted = datetime.utcnow()

# generate the unique portions of the slug and full_slug
slug_part = generate_pseudorandom_slug()
full_slug_part = posted.strftime('%Y.%m.%d.%H.%M.%S') + ':' + slug_part
# load the parent comment (if any)
if parent_slug:
    parent = db.comments.find_one(
        {'node_id': node_id, 'slug': parent_slug })
    slug = parent['slug'] + '/' + slug_part
    full_slug = parent['full_slug'] + '/' + full_slug_part
else:
    slug = slug_part
    full_slug = full_slug_part

# actually insert the comment
db.comments.insert({
    'node_id': node_id,
    'slug': slug,
    'full_slug': full_slug,
    'posted': posted,
    'author': author_info,
    'text': comment_text })

To view comments that are not threaded, we just need to select all comments participating in a discussion and sort by the posted field. For example:

cursor = db.comments.find({'node_id': node_id})
cursor = cursor.sort('posted')
cursor = cursor.skip(page_num * page_size)
cursor = cursor.limit(page_size)

Since the full_slug field contains both hierarchical information (via the path) and chronological information, we can use a simple sort on the full_slug field to retrieve a threaded view:

cursor = db.comments.find({'node_id': node_id})
cursor = cursor.sort('full_slug')
cursor = cursor.skip(page_num * page_size)
cursor = cursor.limit(page_size)

To support these queries efficiently, maintain two compound indexes on node_id, posted and node_id, full_slug:

>>> db.comments.ensure_index([
...    ('node_id', 1), ('posted', 1)])
>>> db.comments.ensure_index([
...    ('node_id', 1), ('full_slug', 1)])

To directly retrieve a comment, without needing to page through all comments, we can select by the slug field:

comment = db.comments.find_one({
    'node_id': node_id,
    'slug': comment_slug})

We can also retrieve a “subdiscussion,” or a comment and all of its descendants recursively, by performing a regular expression prefix query on the full_slug field:

import re

subdiscussion = db.comments.find_one({
    'node_id': node_id,
    'full_slug': re.compile('^' + re.escape(parent_full_slug)) })
subdiscussion = subdiscussion.sort('full_slug')

Since we’ve already created indexes on { node_id: 1, full_slug: 1 } to support retrieving subdiscussions, we don’t need to add any other indexes here to achieve good performance.

To post a new comment in a chronologically ordered (i.e., unthreaded) system, we need the following update:

db.cms.nodes.update(
    { ... node specification ... },
    { '$push': { 'metadata.comments': {
        'posted': datetime.utcnow(),
        'author': author_info,
        'text': comment_text } } } )

The $push operator inserts comments into the comments array in correct chronological order. For threaded discussions, the update operation is more complex. To reply to a comment, the following code assumes that it can retrieve the path as a list of positions, for the parent comment:

if path != []:
    str_path = '.'.join('replies.%d' % part for part in path)
    str_path += '.replies'
else:
    str_path = 'replies'
db.cms.nodes.update(
    { ... node specification ... },
    { '$push': {
        'metadata.' + str_path: {
            'posted': datetime.utcnow(),
            'author': author_info,
            'text': comment_text } } } )

This constructs a field name of the form metadata.replies.0.replies.2... as str_path and then uses this value with the $push operator to insert the new comment into the replies array.

To view the comments in a nonthreaded design, we need to use the $slice operator:

node = db.cms.nodes.find_one(
    { ... node specification ... },
    { ... some fields relevant to your page from the root discussion ...,
      'metadata.comments': { '$slice': [ page_num * page_size, page_size ] }
    })

To return paginated comments for the threaded design, we must retrieve the whole document and paginate the comments within the application:

node = db.cms.nodes.find_one(... node specification ...)

def iter_comments(obj):
    for reply in obj['replies']:
        yield reply
        for subreply in iter_comments(reply):
            yield subreply

paginated_comments = itertools.slice(
    iter_comments(node),
    page_size * page_num,
    page_size * (page_num + 1))

Instead of retrieving comments via slugs as in Approach: One Document per Comment, the following example retrieves comments using their position in the comment list or tree. For chronological (i.e., nonthreaded) comments, we’ll just use the $slice operator to extract a single comment, as follows:

node = db.cms.nodes.find_one(
    {'node_id': node_id},
    {'comments': { '$slice': [ position, position ] } })
comment = node['comments'][0]

For threaded comments, we must know the correct path through the tree in our application, as follows:

node = db.cms.nodes.find_one(... node specification ...)
current = node.metadata
for part in path:
    current = current.replies[part]
comment = current

In order to post a new comment, we need to $push the comment onto the last page and $inc that page’s comment count. Consider the following example that adds a comment onto the last page of comments for some node:

def post_comment(node, comment):
    result = db.comment_pages.update(
        { 'node_id': node['_id'],
          'page': node['num_comment_pages'],
          'count': { '$lt': 100 } }, 
        { '$inc': { 'count': 1 },
          '$push': { 'comments': comment } },
        upsert=True)

    if not result['updatedExisting']:
        db.cms.nodes.update(
            { '_id': node['_id'],
              'num_comment_pages': node['num_comment_pages'] },
            { '$inc': { 'num_comment_pages': 1 } }) 
        db.comment_pages.update(
            { 'node_id': node['_id'],
              'page': node['num_comment_pages'] + 1},
            { '$inc': { 'count': 1 },
              '$push': { 'comments': comment } },
            upsert=True) 

There are a few things to note about this code:

To support the update operations, we need to maintain a compound index on node_id, page in the comment_pages collection:

>>> db.comment_pages.ensure_index([
...    ('node_id', 1), ('page', 1)])

The following function defines how to paginate comments where the number of comments on a page is not known precisely (i.e., with roughly 100 comments, as in this case):

def find_comments(discussion_id, skip, limit):
    query = db.comment_pages.find(
        { 'node_id': node_id } )
    query = query.sort('page')
    for page in query:
        new_skip = skip - page['count']
        if new_skip >= 0:
            skip = new_skip
            continue
        elif skip > 0:
            comments = page['comments'][skip:]
        else:
            comments = page['comments']
        skip = new_skip
        for comment in comments:
            if limit == 0:
                break
            limit -= 1
            yield comment
        if limit == 0: break

Here, we iterate through the pages until our skip requirement is satisfied, then yield comments until our limit requirement is satisfied. For example, if we have three pages of comments with 100, 102, 101, and 22 comments on each page, and we wish to retrieve comments where skip=300 and limit=50, we’d use the following algorithm:

To retrieve a comment directly without paging through all preceding pages of commentary, we’ll use the slug to find the correct page, and then use application logic to find the correct comment:

page = db.comment_pages.find_one(
    { 'node_id': node_id,
      'comments.slug': comment_slug},
    { 'comments': 1 })
for comment in page['comments']:
    if comment['slug'] = comment_slug:
        break

To perform this query efficiently, we’ll need a new index on node_id, comments.slug (this is assuming that slugs are only guaranteed unique within a node):

>>> db.comment_pages.ensure_index([
...    ('node_id', 1), ('comments.slug', 1)])