What You’ll Learn in This Hour:
Paging documents from a large dataset in Node.js
Limiting which fields are returned from documents in Node.js
Using methods to generate lists of distinct field values in documents from a collection in Node.js
Implementing grouping from Node.js to group documents and build a return dataset
Applying an aggregation pipeline to build a dataset from documents in a collection from Node.js
This hour picks up from the last hour and continues to describe the Node.js MongoDB driver and how to implement it to retrieve data in your Node.js applications. This hour takes you through the process of using Node.js applications to limit the return results by limiting the number of documents returned, limiting the fields returned, and paging through large sets.
The hour also covers how to implement distinct grouping and aggregation operations from a Node.js application. These operations enable you to process data on the server before returning it to the Node.js application, reducing the amount of data sent and the work required in the application.
When finding documents on larger databases with more complex documents, you often want to limit what is being returned in requests to reduce the impact on the network, memory on both the server and client, and more. You can limit the result sets that match a specific query in three ways. You can simply accept a limited amount of documents, you can limit the fields that are returned, or you can page the results and get them in paged chunks.
The simplest method of limiting the amount of data returned in a find()
or other query request is to use the limit()
method on the Cursor
object that the find()
operation returns. The limit()
method limits the cursor so that it returns only a fixed number of items. This can save you from accidentally retrieving more objects than your application can handle.
For example, the following Node.js code displays only the first 10 documents in a collection, even though there could be thousands:
var cursor = wordsColl.find();
cursor.limit(10, function(err, items){
items.each(function(err, word){
if(word){
console.log(word);
}
}
});
Another extremely effective method of limiting the resulting data when retrieving documents is to limit which fields are returned. Documents might have many different fields that are useful in some circumstance but not in others. Consider which fields to include when retrieving documents from the MongoDB server, and request only the ones necessary.
To limit the fields returned from the server in a find()
operation on a Collection
object, you can use the fields
parameter. This parameter is a JavaScript object that contains the files with a value of true
to include or false
to exclude.
For example, to exclude the fields stats
, value
, and comments
when returning documents, you would use null
for the query
object because you are not finding all objects. You also would use the following fields
object:
var fields = {'stats' : false, 'value' : false, 'comments' : false);
var cursor = myColl.find({}, {'fields': fields});
Including just a few fields is often easier. For example, if you want to include only the word
and size
fields of documents in which the first
field equals t
, you would use
var query = {'first' : 't'};
var fields = {'word' : true, 'size' : true};
var cursor = myColl.find(query, fields);
A very common method of reducing the number of documents returned is to use paging. Paging involves specifying a number of documents to skip in the matching set, as well as limiting the documents returned. Then the skip value is incremented each time by the amount returned the previous time.
To implement paging on a set of documents, you need to implement the limit()
and skip()
methods on the Cursor
object. The skip()
method enables you to specify a number of documents to skip before returning documents.
By incrementing the value used in the skip()
method by the size used in limit()
each time you get another set of documents, you can effectively page through the dataset.
For example, the following statements find documents 11–20:
var cursor = collection.find();
cursor = cursor.limit(10);
cursor = cursor.skip(10);
Always include a sort
option when paging data, to ensure that the order of documents is the same.
A useful query against a MongoDB collection is to get a list of the distinct values for a single field in a set of documents. Distinct means that even though thousands of documents exist, you want to know only the unique values.
The distinct()
method on Collection
and Cursor
objects enables you to find a list of distinct values for a specific field. The syntax for the distinct()
method follows:
distinct(key, callback)
The key
parameter is the string value of the field name you want to get values for. You can specify subdocuments using dot syntax, such as stats.count
. If you want to get distinct values based on a subset of documents, you need to first generate a Cursor
object with a query
parameter and then call distinct()
on that Cursor
object.
For example, to find the distinct last names of users over 65 in a collection that has documents with first
, last
, and age
fields, you would use the following operation:
var query = {'age' : {'$gt' : 65}};
var cursor = myCollection.find(query);
cursor.distinct('last', function(err, lastNames){
console.log(lastNames);
}
The distinct()
method returns an array with the distinct values for the field specified. For example:
["Smith", "Jones", ...]
When performing operations on large datasets in Node.js, it is often useful to group the results based on the distinct values of one or more fields in a document. This could be done in code after retrieving the documents, but it is much more efficient to have the MongoDB server do it for you as part of a single request that is already iterating though the documents.
In Node.js, to group the results of a query, you can use the group()
method on the Collection
object. The group request collects all the documents that match a query
, adds a group object to an array based on distinct values of a set of keys
, performs operations on the group objects, and returns the array of group objects.
The syntax for the group()
methods follows:
group({keys, cond , initial, reduce, [finalize], callback})
The keys
, cond
, and initial
parameters are specified in JavaScript objects that define the fields to use, query to limit documents, and initial value settings. The reduce
and finalize
methods are String
objects that contain a string form of a JavaScript function that will be run on the server to reduce
and finalize
the request. See Hour 9, “Utilizing the Power of Grouping, Aggregation, and Map Reduce,” for more information on these parameters.
To illustrate, the following code implements a basic grouping by generating the key
, cond
, and initial
objects and passing in a reduce
function as a string:
var key = {'first' : true };
var cond = {'first' : 'a', 'size': 5};
var initial = {'count' : 0};
var reduce = "function (obj, prev) { prev.count++; }";
collection.group(key, cond, initial, reduce, function(err, results){
. . .
});
The result from the group method is an array that contains the groupings. To illustrate the results, the following code displays the items in the group one at a time:
for (var i in results){
console.log(results[i]);
}
Another valuable tool when working with MongoDB in Node.js applications is the aggregation framework. The Collection
object provides the aggregate()
method to perform aggregation operations on data. The syntax for the aggregate()
method follows:
aggregate(operator, [operator, ...], callback)
The operator
parameter is a list of one or more operator objects that provide the pipeline for aggregating the results. Each operator is a JavaScript object built with the operators. Hour 9 described the aggregation operators, so you should already be familiar with them.
As an example, the following code creates $group
and $limit
operators. The $group
operator groups by _id
of the word
field and adds an average
field using the $avg
that averages a field named size
. Notice that the field names must be prefixed with $
in aggregation operations:
var group = {'$group' :
{'_id' : '$word',
'average' : {'$avg' : '$size'}}};
var limit = {'$limit' : 10};
collection.aggregate([group, limit], function(err, results){
. . .
}):
The results from the aggregate()
method is an array that contains the aggregation results. Each element in the array is an aggregated result. To illustrate, the following code accesses and displays the items in the aggregated results one at a time:
for (var i in results){
console.log(results[i]);
}
In this hour, you learned how to use additional methods on the Collection
and Cursor
objects. You learned that the limit()
method can reduce the documents the cursor
returns and that limit()
and skip()
enable you to page through a large dataset. Using a fields
parameter on the find()
method enables you to reduce the number of fields returned from the database.
This hour also covered applying the distinct()
, group()
, and aggregate()
methods on the Collection
object to perform data gathering operations from a Node.js application. These operations enable you to process data on the server before returning it to the Node.js application, reducing the amount of data sent and the work required in the application.
Q. Is there a way to avoid all the callbacks when implementing MongoDB in Node.js?
A. Not really. That is the nature of Node.js, and each operation to MongoDB must be asynchronous to support the Node.js single event queue model. The exception is the Cursor
object, which can be called with methods such as sort()
, limit()
, and skip()
to generate new Cursor
objects without dealing with a callback. The Cursor
works the way it does for those methods because a new database request is not required to generate the new Cursor
object.
The workshop consists of a set of questions and answers designed to solidify your understanding of the material covered in this hour. Try answering the questions before looking at the answers.
1. How do you get the 21–30 documents represented by a Cursor
object in Node.js?
2. How do you find the different values for a specific field on documents in a collection in a Node.js application?
3. How do you return the first 10 documents from a collection in Node.js?
4. How do you prevent a specific field from being returned from a database query in Node.js?
1. Call limit(10)
and skip(20)
on the Cursor
object.
2. Use the distinct()
method on the Collection
object.
3. Use the limit(10)
method on the Cursor
object.
4. Set the field to false
in the fields
object of the options
passed to the find()
method.
1. Write a new Node.js application that finds words in the example dataset that start with n
, sort them by length descending, and then display the top five.
2. Extend the Node.jsAggregate.js
file to include a function that performs an aggregate that matches the words with a length of 4
, then limits it to only five documents, and finally projects the word as the _id
field and displays the stats. The matching MongoDB shell aggregation would look similar to the following:
{$match: {size:4}},
{$limit: 5},
{$project: {_id:"$word", stats:1}}
18.218.171.212