Chapter 3. Case Study: Social Feed Reader

Throughout this book, specific examples are used to explore the challenges and benefits of designing service-oriented applications. The case study presented in this chapter provides a basis for the discussion in the rest of this book. This chapter doesn’t implement and discuss every service that this application could contain, but it gives some good areas to focus on. Exploring how an existing application is set up highlights the shortcomings of a typical Rails stack and how to overcome those shortcomings through the intelligent use of services.

A Typical Rails Application

A typical Rails application starts out simple. You develop everything in a single code base, write tests (hopefully), and use a single database. In the beginning, it is likely to take up only two servers—the web and application server and the database—as shown in Figure 3.1.

Figure 3.1 A simple Rails deployment.

Simply setting up a few servers can get you pretty far with Rails. However, the complexity builds quickly. In a Rails application, you need to be able to run background tasks. For example, if a user takes an action that triggers sending an email message out to another user, you want the email to be sent in the background instead of during a request that is being made by a waiting user. For that, the application needs to be running additional background processes, as shown in Figure 3.2.

The background processing code is typically run on application servers. With a simple background processing system such as Delayed Job, the tasks to be performed are stored in the database in their own table. This means that the background job processing is fully integrated with the database and the Rails application code base (specifically, the models). If the logic for processing these jobs needs to be updated, the application code base must be changed and tested, and the full application must be redeployed.

Figure 3.2 A simple Rails deployment with background processing.

At this point, the application isn’t terribly complex. Only a few things run in the background, and the number of models is manageable. The next steps that are usually taken might include adding Memcached and adding more web servers to handle more requests, as shown in Figure 3.3.

Figure 3.3 A full Rails deployment with background processing.

Many Rails deployments look something like the one shown in Figure 3.3. The number of application servers may be greater or lower, but the general look is the same. There is a database server, the Rails application, and some background processing that is tightly integrated with the application. From there, the development proceeds and the application is built and iterated on.

While this setup is quite powerful, one of the problems that might be encountered is what to do when the database is unable to keep up with the number of requests. Typical approaches include performance logging to find spots to optimize queries, getting more powerful hardware for the database server, and splitting out reads and writes to separate replicated servers. What to do in this situation is the typical dilemma that people think of when they think of scaling. It’s also one of the issues that service-oriented design tackles.

The other scaling problem isn’t often discussed: scaling the team and the size of the project. As the application is improved and iterated, its code gets ever more complex. The project grows larger and has more lines of code added every day. While development continues, the application gains more models, controllers, views, and tests. This means longer times for the test suite to run and more code that each developer needs to be cognizant of while making updates. Taking a service-oriented approach also helps address this scaling problem.

Let’s now move from a generic Rails application to a more specific app.

The Rails Social Feed Reader Application

Many Rails developers have had to build some sort of social network at one point or another. These days, most web applications have at least a lightweight social network on top of them. This example considers a feed reader with a social network built on top in order to look at some of the advantages of services. Because most readers already have Rails applications in production, the concerns of migrating an existing application to services are very important. As this case study comes up throughout the book, it provides material for how you might perform the migration to services.

Features

For the sake of brevity, the feed reader needs to be fairly simple. Here is a high-level overview of the features of the current Rails social feed reader:

• User Profiles and Following—This includes basic profile information, user name, password, and which users are following each other.

• Activity Streams—This refers to the Facebook style of stream. Each user has a stream of activity from the people they follow. In addition, each user has a stream of activity for his or her profile that others can see. Activity can be either subscribing to a feed, commenting on an entry, or voting up or down on an entry.

• Feeds and Feed Entries—The system keeps track of which feeds need to be updated and the entries for each of these feeds.

• Feed Subscriptions—The system keeps track of which feeds a user is subscribed to.

• Votes—The system keeps track of all of the votes, up or down, a user casts on specific feed entries.

• Comments—Users can comment on any story they read from their feeds. These comments show up in the stream of any follower. The comments also show up on any entry when a user is reading through his or her list of feeds.

• Email Notifications—The system sends email notifications to users for account modifications or when other users start following them.

Current Setup

The current setup for the feed reader application looks like the typical Rails deployment mentioned earlier. There is an application server with background processing. The logic for sending out emails and performing updates of feeds occurs in these background jobs. Everything is tied together fairly tightly. There are separate models for the emails and the feeds, but the logic of actually running these tasks is held within a single process.

The code in the social reader application looks like a regular Rails application. There are models, controllers, and views. When looking at services, most of what gets pulled into a service is the code and logic that lies at the model level. It helps to have in mind what the model schema and relationships for the data look like. The following sections outline the models in the application. Each one is annotated with its database schema using the annotate models gem (see http://github.com/ctran/annotate_models/tree/master).

The User Model

The user model is the common starting place for most applications. Many of the other models are tied in some way to the user model, and the social feed reader application is no different. Every model can be reached through an association on the user model. When breaking up the data into services, some of these associations will have to cross service boundaries. The user model looks as follows:

This example shows only the associations, which happen to add the finders that are needed for the application. There are a few has_many : through associations. Users can access their followers, who they are following, the comments they have made, the feeds they are subscribed to, the activities they have performed in the system, and the activities of the users they are following.

The activities are contained in a single denormalized table. Another option would be to map activities through a polymorphic relationship, but such joins can be expensive. A denormalized structure makes retrieving the activities a quick operation. To get a sense of why the relationship for activities looks the way it does, let’s look at the activity model.

The Activity Model

The activity model is for keeping a record of user activity such as following another user, subscribing to a feed, or commenting or voting on an entry. It is a denormalized model, so there is some data duplication with the comment, follow, subscription, and vote models. The activity model looks as follows:

The activity model uses single-table inheritance (STI) to keep each type of activity in the same table. The parent class defines a write method that should be called when a comment, subscription, vote, or follow is created. First, it writes an activity without followed_user_id, which is used in the user model to find the activities that the specific user performed. Then write creates a new activity for each of the user’s followers. This is another instance of data duplication, but it cuts down on the number of joins that must be performed to pull the activity for all the users an individual is following.

The Follow Model

The follow model is the join model that specifies which users are following the others. It looks like this:

The follow model contains only the two user IDs of the follower and followee. The logic for creating activities after create is contained in the model.

The Feed Model

The feed model contains the basic data for RSS or Atom feeds in the system. Here’s how it looks:

The relationships for the feed model show that it has many entries (the specific blog posts) and many users through the subscriptions.

The Subscription Model

The subscription model maps users to the feeds they are subscribed to. It looks like this:

The subscription model is simple, with only a relationship with the user and the feed. The logic to create subscription activities is in the after create block.

The Entry Model

The entry model contains all the information for a specific article or blog post from a feed. Here’s how it looks:

The entry model has relationships to the feed that it belongs to and the comments associated with it. There are also counters for the number of up votes, down votes, and comments. It could also contain a has-many relationship to those votes, but from the entry’s point of view, the only important thing for the application to keep track of is the count of vote types.

The Vote Model

The vote model uses STI to define the two different types of votes, the up and down votes:

The parent class vote defines the relationship to user that both the up and down vote classes require. The up and down votes both define their relationships to the entry because of the automatic counter cache. This gets incremented in up_votes_count or down_votes_count on the entry object. Finally, only the up vote writes activity. This is because the users probably don’t want to see entries that the people they are following thought were bad.

The Comment Model

The comment model is very basic. It holds only the text of a comment from a user and the associated entry:

The relationships to the user and entry are here in the comment model. Also, a counter cache is kept up on the entry to store the number of comments. Finally, after creation of a new comment, the activity is written.

Converting to Services

The social feed reader application is fairly basic in terms of the number of models. However, the models are a decent cross-section of the different kinds of complexity found in a typical ActiveRecord-based Rails application. There are counter caches, STI, belongs-to, has-many, and has-many-through relationships. With all these things tied so closely together, it’s important to know how you might separate the application into discrete services.

Segmenting into Services

There are many different ways to break up the social feed reader application into services. Some choices are straightforward, while others require a little more thought. Answering the following questions helps determine how to redesign for services:

• Which data has high read and low write frequency?

• Which data has high write or update frequency?

• Which joins occur most frequently?

• Which parts of the application have clearly defined requirements and design?

Answering the first three questions will help determine where models might belong in different services. The last question helps determine whether a portion of the application should remain in a typical Rails environment instead of being split out into services. Typically, only portions of applications that are well defined should be considered for use as services. Chapter 4, “Service and API Design,” goes into the details of breaking these models into callable HTTP services.

Breaking Up the Application into Services

Finding the right places to split the Rails model-view-controller (MVC) design pattern can be tricky. For an existing application, the conversion must occur slowly over time, as parts are refactored into their own services. The progression of converting an application could take the form shown in Figure 3.4.

Figure 3.4 A Rails application.

Here, at the starting point, everything is contained in a single application. The first step is to implement services at the model layer, as shown in Figure 3.5.

Figure 3.5 A Rails application with services at the model layer.

Some or all of the models could be converted to services. For applications that rely heavily on background processing (for example, the social feed reader), this might be enough. Other applications that have many views and associated tests will benefit from having the controller and view layers broken into services. When the models are accessible as services from any application in the environment, this conversion can take place as shown in Figure 3.6.

Figure 3.6 A Rails application with services at all layers.

With the controllers and views broken into separate applications, the Rails application looks less like a regular application. However, the picture in Figure 3.5 isn’t yet complete. Once these view services have been made, you can convert the Rails application into nothing more than a proxy route request and stitch all the services together as shown in Figure 3.7.

Figure 3.7 A Rails application that connects multiple services.

The final step is to set up the Routing/ESI/Include Module in front of the view services. This has multiple names because it could take one of multiple forms. It could be a small Rails application that handles the routing of requests and builds final pages based on components handed back from services. It could also be an ESI layer that calls out to the services directly. Finally, it could be a Rack application that includes the services as modules. Some of these options are explored later in the book.

Conclusion

Ruby on Rails is perfect for quick prototyping, fast iterations, and smaller applications. As your code base, team, and traffic levels grow in volume and size, the complexity can become difficult to manage. This can result in large test suites that take a long time to run or unnecessary cognitive load for developers who have to keep too much of the application in their head. The case study in this chapter provides fodder for specific examples later in the book that illustrate how to convert an application to services and what the trade-offs and concerns are.