Schema Design Patterns

Schema design is important in MongoDB, as it impacts directly on application performance. There are many common issues in schema design that can be addressed through the use of known patterns, or “building blocks.” It is best practice in schema design to use one or more of these patterns together.

Scheme design patterns that might apply include:

Polymorphic pattern

This is suitable where all documents in a collection have a similar, but not identical, structure. It involves identifying the common fields across the documents that support the common queries that will be run by the application. Tracking specific fields in the documents or subdocuments will help identify the differences between the data and different code paths or classes/subclasses that can be coded in your application to manage these differences. This allows for the use of simple queries in a single collection of not-quite-identical documents to improve query performance.

Attribute pattern

This is suitable when there are a subset of fields in a document that share common features on which you want to sort or query, or when the fields you need to sort on only exist in a subset of the documents, or when both of these conditions are true. It involves reshaping the data into an array of key/value pairs and creating an index on the elements in this array. Qualifiers can be added as additional fields to these key/value pairs. This pattern assists in targeting many similar fields per document so that fewer indexes are required and queries become simpler to write.

Bucket pattern

This is suitable for time series data where the data is captured as a stream over a period of time. It is much more efficient in MongoDB to “bucket” this data into a set of documents each holding the data for a particular time range than it is to create a document per point in time/data point. For example, you might use a one-hour bucket and place all readings for that hour in an array in a single document. The document itself will have start and end times indicating the period this “bucket” covers.

Outlier pattern

This addresses the rare instances where a few queries of documents fall outside the normal pattern for the application. It is an advanced schema pattern designed for situations where popularity is a factor. This can be seen in social networks with major influencers, book sales, movie reviews, etc. It uses a flag to indicate the document is an outlier and stores the additional overflow into one or more documents that refer back to the first document via the "_id". The flag will be used by your application code to make the additional queries to retrieve the overflow document(s).

Computed pattern

This is used when data needs to be computed frequently, and it can also be used when the data access pattern is read-intensive. This pattern recommends that the calculations be done in the background, with the main document being updated periodically. This provides a valid approximation of the computed fields or documents without having to continuously generate these for individual queries. This can significantly reduce the strain on the CPU by avoiding repetition of the same calculations, particularly in use cases where reads trigger the calculation and you have a high read-to-write ratio.

Subset pattern

This is used when you have a working set that exceeds the available RAM of the machine. This can be caused by large documents that contain a lot of information that isn’t being used by your application. This pattern suggests that you split frequently used data and infrequently used data into two separate collections. A typical example might be an ecommerce application keeping the 10 most recent reviews of a product in the “main” (frequently accessed) collection and moving all the older reviews into a second collection queried only if the application needs more than the last 10 reviews.

Extended Reference pattern

This is used for scenarios where you have many different logical entities or “things,” each with their own collection, but you may want to gather these entities together for a specific function. A typical ecommerce schema might have separate collections for orders, customers, and inventory. This can have a negative performance impact when we want to collect together all the information for a single order from these separate collections. The solution is to identify the frequently accessed fields and duplicate these within the order document. In the case of an ecommerce order, this would be the name and address of the customer we are shipping the item to. This pattern trades off the duplication of data for a reduction in the number of queries necessary to collate the information together.

Approximation pattern

This is useful for situations where resource-expensive (time, memory, CPU cycles) calculations are needed but where exact precision is not absolutely required. An example of this is an image or post like/love counter or a page view counter, where knowing the exact count (e.g., whether it’s 999,535 or 1,000,0000) isn’t necessary. In these situations, applying this pattern can greatly reduce the number of writes—for example, by only updating the counter after every 100 or more views instead of after every view.

Tree pattern

This can be applied when you have a lot of queries and have data that is primarily hierarchical in structure. It follows the earlier concept of storing data together that is typically queried together. In MongoDB, you can easily store a hierarchy in an array within the same document. In the example of the ecommerce site, specifically its product catalog, there are often products that belong to multiple categories or to categories that are part of other categories. An example might be “Hard Drive,” which is itself a category but comes under the “Storage” category, which itself is under the “Computer Parts” category, which is part of the “Electronics” category. In this kind of scenario, we would have a field that would track the entire hierarchy and another field that would hold the immediate category (“Hard Drive”). The entire hierarchy field, kept in an array, provides the ability to use a multikey index on those values. This ensures all items related to categories in the hierarchy will be easily found. The immediate category field allows all items directly related to this category to be found.

Preallocation pattern

This was primarily used with the MMAP storage engine, but there are still uses for this pattern. The pattern recommends creating an initial empty structure that will be populated later. An example use could be for a reservation system that manages a resource on a day-by-day basis, keeping track of whether it is free or already booked/unavailable. A two-dimensional structure of resources (x) and days (y) makes it trivially easy to check availability and perform calculations.

Document Versioning pattern

This provides a mechanism to enable retention of older revisions of documents. It requires an extra field to be added to each document to track the document version in the “main” collection, and an additional collection that contains all the revisions of the documents. This pattern makes a few assumptions: specifically, that each document has a limited number of revisions, that there are not large numbers of documents that need to be versioned, and that the queries are primarily done on the current version of each document. In situations where these assumptions are not valid, you may need to modify the pattern or consider a different schema design pattern.

MongoDB provides several useful resources online on patterns and schema design. MongoDB University offers a free course, M320 Data Modeling, as well as a “Building with Patterns” blog series.

Examples of Data Representations

Suppose we are storing information about students and the classes that they are taking. One way to represent this would be to have a students collection (each student is one document) and a classes collection (each class is one document). Then we could have a third collection (studentClasses) that contains references to the students and the classes they are taking:

> db.studentClasses.findOne({"studentId" : id})
{
    "_id" : ObjectId("512512c1d86041c7dca81915"),
    "studentId" : ObjectId("512512a5d86041c7dca81914"),
    "classes" : [
        ObjectId("512512ced86041c7dca81916"),
        ObjectId("512512dcd86041c7dca81917"),
        ObjectId("512512e6d86041c7dca81918"),
        ObjectId("512512f0d86041c7dca81919")
    ]
}

If you are familiar with relational databases, you may have seen this type of join table before (although typically you’d have one student and one class per document, instead of a list of class "_id"s). It’s a bit more MongoDB-ish to put the classes in an array, but you usually wouldn’t want to store the data this way because it requires a lot of querying to get to the actual information.

Suppose we wanted to find the classes a student was taking. We’d query for the student in the students collection, query studentClasses for the course "_id"s, and then query the classes collection for the class information. Thus, finding this information would take three trips to the server. This is generally not the way you want to structure data in MongoDB, unless the classes and students are changing constantly and reading the data does not need to be done quickly.

We can remove one of the dereferencing queries by embedding class references in the student’s document:

{
    "_id" : ObjectId("512512a5d86041c7dca81914"),
    "name" : "John Doe",
    "classes" : [
        ObjectId("512512ced86041c7dca81916"),
        ObjectId("512512dcd86041c7dca81917"),
        ObjectId("512512e6d86041c7dca81918"),
        ObjectId("512512f0d86041c7dca81919")
    ]
}

The "classes" field keeps an array of "_id"s of classes that John Doe is taking. When we want to find out information about those classes, we can query the classes collection with those "_id"s. This only takes two queries. This is a fairly popular way to structure data that does not need to be instantly accessible and changes, but not constantly.

If we need to optimize reads further, we can get all of the information in a single query by fully denormalizing the data and storing each class as an embedded document in the "classes" field:

{
    "_id" : ObjectId("512512a5d86041c7dca81914"),
    "name" : "John Doe",
    "classes" : [
        {
            "class" : "Trigonometry",
            "credits" : 3,
            "room" : "204"
        },
        {
            "class" : "Physics",
            "credits" : 3,
            "room" : "159"
        },
        {
            "class" : "Women in Literature",
            "credits" : 3,
            "room" : "14b"
        },
        {
            "class" : "AP European History",
            "credits" : 4,
            "room" : "321"
        }
    ]
}

The upside of this is that it only takes one query to get the information. The downsides are that it takes up more space and is more difficult to keep in sync. For example, if it turns out that physics was supposed to be worth four credits (not three), every student in the physics class would need to have their document updated (instead of just updating a central “Physics” document).

Finally, you can use the Extended Reference pattern mentioned earlier, which is a hybrid of embedding and referencing—you create an array of subdocuments with the frequently used information, but with a reference to the actual document for more information:

{
    "_id" : ObjectId("512512a5d86041c7dca81914"),
    "name" : "John Doe",
    "classes" : [
        {
            "_id" : ObjectId("512512ced86041c7dca81916"),
            "class" : "Trigonometry"
        },
        {
            "_id" : ObjectId("512512dcd86041c7dca81917"),
            "class" : "Physics"
        },
        {
            "_id" : ObjectId("512512e6d86041c7dca81918"),
            "class" : "Women in Literature"
        },
        {
            "_id" : ObjectId("512512f0d86041c7dca81919"),
            "class" : "AP European History"
        }
    ]
}

This approach is also a nice option because the amount of information embedded can change over time as your requirements change: if you want to include more or less information on a page, you can embed more or less of it in the document.

Another important consideration is how often this information will change, versus how often it’s read. If it will be updated regularly, then normalizing it is a good idea. However, if it changes infrequently, then there is little benefit to optimizing the update process at the expense of every read your application performs.

For example, a textbook normalization use case is to store a user and their address in separate collections. However, people’s addresses rarely change, so you generally shouldn’t penalize every read on the off chance that someone’s moved. Your application should embed the address in the user document.

If you decide to use embedded documents and you need to update them, you should set up a cron job to ensure that any updates you do are successfully propagated to every document. For example, suppose you attempt to do a multi-update but the server crashes before all of the documents have been updated. You need a way to detect this and retry the update.

In terms of update operators, "$set" is idempotent but "$inc" is not. Idempotent operations will have the same outcome whether tried once or several times; in the case of a network error, retrying the operation will be sufficient for the update to occur. In the case of operators that are not idempotent, the operation should be broken into two operations that are individually idempotent and safe to retry. This can be achieved by including a unique pending token in the first operation and having the second operation use both a unique key and the unique pending token. This approach allows "$inc" to be idempotent because each individual updateOne operation is idempotent.

To some extent, the more information you are generating, the less of it you should embed. If the content of the embedded fields or number of embedded fields is supposed to grow without bound then they should generally be referenced, not embedded. Things like comment trees or activity lists should be stored as their own documents, not embedded. It is also worth considering using the Subset pattern (described in “Schema Design Patterns”) to store the most recent items (or some other subset) in the document.

Finally, the fields that are included should be integral to the data in the document. If a field is almost always excluded from your results when you query for a document, it’s a good sign that it may belong in another collection. These guidelines are summarized in Table 9-1.

Suppose we had a users collection. Here are some example fields we might have in the user documents and an indication of whether or not they should be embedded:

Account preferences

These are only relevant to this user document, and will probably be exposed with other user information in the document. Account preferences should generally be embedded.

Recent activity

This depends on how much recent activity grows and changes. If it is a fixed-size field (say, the last 10 things), it might be useful to embed this information or to implement the Subset pattern.

Friends

Generally this information should not be embedded, or at least not fully. See “Friends, Followers, and Other Inconveniences”.

All of the content this user has produced

This should not be embedded.

Cardinality

Cardinality is an indication of how many references a collection has to another collection. Common relationships are one-to-one, one-to-many, or many-to-many. For example, suppose we had a blog application. Each post has a title, so that’s a one-to-one relationship. Each author has many posts, so that’s a one-to-many relationship. And posts have many tags and tags refer to many posts, so that’s a many-to-many relationship.

When using MongoDB, it can be conceptually useful to split “many” into subcategories: “many” and “few.” For example, you might have a one-to-few relationship between authors and posts: each author only writes a few posts. You might have many-to-few relation between blog posts and tags: you probably have many more blog posts than you have tags. However, you’d have a one-to-many relationship between blog posts and comments: each post has many comments.

Determining few versus many relations can help you decide what to embed versus what to reference. Generally, “few” relationships will work better with embedding, and “many” relationships will work better as references.

Friends, Followers, and Other Inconveniences

Keep your friends close and your enemies embedded.

This section covers considerations for social graph data. Many social applications need to link people, content, followers, friends, and so on. Figuring out how to balance embedding and referencing this highly connected information can be tricky, but generally following, friending, or favoriting can be simplified to a publication/subscription system: one user is subscribing to notifications from another. Thus, there are two basic operations that need to be efficient: storing subscribers and notifying all interested parties of an event.

There are three ways people typically implement subscribing. The first option is to put the producer in the subscriber’s document, which looks something like this:

{
    "_id" : ObjectId("51250a5cd86041c7dca8190f"),
    "username" : "batman",
    "email" : "[email protected]"
    "following" : [
        ObjectId("51250a72d86041c7dca81910"), 
        ObjectId("51250a7ed86041c7dca81936")
    ]
}

Now, given a user’s document, you can issue a query like the following to find all of the activities that have been published that they might be interested in:

db.activities.find({"user" : {"$in" :
      user["following"]}})

However, if you need to find everyone who is interested in a newly published activity, you’d have to query the "following" field across all users.

Alternatively, you could append the followers to the producer’s document, like so:

{
    "_id" : ObjectId("51250a7ed86041c7dca81936"),
    "username" : "joker",
    "email" : "[email protected]"
    "followers" : [
        ObjectId("512510e8d86041c7dca81912"),
        ObjectId("51250a5cd86041c7dca8190f"),
        ObjectId("512510ffd86041c7dca81910")
    ]
}

Whenever this user does something, all the users you need to notify are right there. The downside is that now you need to query the whole users collection to find everyone a user follows (the opposite limitation as in the previous case).

Either of these options comes with an additional downside: they make your user documents larger and more volatile. The "following" (or "followers") field often won’t even need to be returned: how often do you want to list every follower? Thus, the final option neutralizes these downsides by normalizing even further and storing subscriptions in another collection. Normalizing this far is often overkill, but it can be useful for an extremely volatile field that often isn’t returned with the rest of the document. "followers" may be a sensible field to normalize this way.

In this case you keep a collection that matches publishers to subscribers, with documents that look something like this:

{
    "_id" : ObjectId("51250a7ed86041c7dca81936"), // followee's "_id"
    "followers" : [
        ObjectId("512510e8d86041c7dca81912"),
        ObjectId("51250a5cd86041c7dca8190f"),
        ObjectId("512510ffd86041c7dca81910")
    ]
}

This keeps your user documents svelte but means an extra query is needed to get the followers.

> db.users.find({"username" : "wil"})
{
    "_id" : ObjectId("51252871d86041c7dca8191a"),
    "username" : "wil",
    "email" : "[email protected]",
    "tbc" : [
        ObjectId("512528ced86041c7dca8191e"),
        ObjectId("5126510dd86041c7dca81924")
    ]
    "followers" : [
        ObjectId("512528a0d86041c7dca8191b"),
        ObjectId("512528a2d86041c7dca8191c"),
        ObjectId("512528a3d86041c7dca8191d"),
        ...
    ]
}
{
    "_id" : ObjectId("512528ced86041c7dca8191e"),
    "followers" : [
        ObjectId("512528f1d86041c7dca8191f"),
        ObjectId("512528f6d86041c7dca81920"),
        ObjectId("512528f8d86041c7dca81921"),
        ...
    ]
}
{
    "_id" : ObjectId("5126510dd86041c7dca81924"),
    "followers" : [
        ObjectId("512673e1d86041c7dca81925"),
        ObjectId("512650efd86041c7dca81922"),
        ObjectId("512650fdd86041c7dca81923"),
        ...
    ]
}