When you develop web applications of any kind, more than often, you will need a database to store data in. Without a proper persistence layer, only basic web apps can be built. In Chapter 5, Programming RethinkDB in Node.js, you learned how to connect to a RethinkDB cluster and run queries from a Node.js script. In this chapter, we're going to use the same syntax to use RethinkDB within an Express.js web app.
Fortunately, adding database connectivity capability to Express.js web apps is just a matter of loading an appropriate Node.js driver for the database in your app; however, you maybe wondering where exactly we put the code that interacts with the database, in the context of our Express.js web application. The answer lies within the MVC pattern.
If you've ever developed web applications before, you'll certainly be familiar with the model-view-controller pattern, which divides an application into three separate components: the model that stores data; the view that generates an output for the user; and the controller that updates the model and views. The database fits in the model component of the MVC pattern. The only question is how we can apply this to Express.js.
The model is the component that handles the data that's in our web application. It needs to have access to the RethinkDB Node.js module as the model will be the part of the app that interacts with the database and runs queries.
In Chapter 5, Programming RethinkDB in Node.js, you learned how to run queries from a simple Node.js script; however, you maybe wondering how the interaction with the database works in the context of a Node.js real-time web application. The traditional answer for most database systems is polling.
Traditional databases, such as MySQL, provide no way for the web application to detect changes within the database, so the only solution is for the client to continuously poll the database for changes. As you can imagine, this is not a great solution. With this type of solution, all clients will have to issue very frequent requests to the Node.js server that, in turn, will have to query the database continuously, for example, every second. This kind of approach is not at all practical for real-time web applications.
The diagrammatic representation of this approach is shown here:
There are several problems with this approach. First of all, sending frequent requests to the database and server will most certainly lead to a degraded performance due to the high volume of traffic. This will be even more evident when the web application has lots of connected clients. Another drawback with this approach is that data is not actually refreshed in real time, but rather, it is dependent on the time interval on which the database is polled. This approach is fine for some use cases, for example, a chat application, where new messages appear every second. However, if we want to develop an application that updates a graphical object in real time, this approach will fail.
There have been various solutions to this problem. In the next section, we'll outline a few of them.
One of the most common solutions to the previous problem is to go with external message queues or some kind of in-memory data store.
These software work by exchanging data in the form of messages between different parts of an application. Using a message queue, you can set up one client to continuously poll the database for changes and pass updates to other clients by sending a message through the message broker. This is shown in the following diagram:
These solutions can be very efficient in some cases as they definitely help reduce the server and database load; however, they involve adding a new software to your stack and adding more moving parts to your web application. Another common solution is to use an in-memory cache, such as Redis or Memcached. Using these solutions, you can cache the results of queries in the memory, thus clients will be able to query the cache instead of the database, achieving better performance and lowering system load.
However, these solutions present more than one problem. A new software must be added to your application and be configured, adding complexity, and more importantly, using an in-memory cache will often result in having stale data, especially in real-time web apps where data is updated very frequently. Fortunately, RethinkDB provides us with great features for the web application development.
RethinkDB Changefeeds provide an alternative to polling and using an external message queue as they allow you to push updated data from the database directly to the application.
Changefeeds are also extremely flexible as you can attach a Changefeed directly to a table to detect all changes, but you can also bind a Changefeed to more sophisticated queries to track updates on very specific data:
r.db('test').table('people').orderBy({index: 'age'}).limit(10).changes()In this example, we attach the changes command to a complex query that orders data using an index and only returns the top 10 results. This kind of query can be useful to write a real-time leader board or any kind of ranking where data needs to be updated in real-time.
Another important aspect to note is that Changefeeds are particularly useful in web applications that are designed to scale horizontally. The reason for this is that as we've seen in the previous section, when we use multiple machines to spread the load evenly across the cluster, we typically need an external software, such as a message queue or in-memory database, to propagate database updates between different servers. The power of Changefeeds resides in the fact that they move this message propagation feature from the application layer to the persistence layer: the database. This results in a simpler application architecture, eliminating the need for additional infrastructure, such as a message queue.
With Changefeeds, each instance of the application layer subscribes directly to the database to receive changes using the changes command. This way, when changes are available, RethinkDB sends them directly to each application server, which then broadcasts the updates to every client using WebSockets.
Now that you've learned all about the components that make up a real-time web application, including how the database fits in, it's time to put everything that you've learned together and develop a full-fledged, real-time web application!