The aggregation pipeline can output results in three distinct ways:
- Inline as a document containing the result set
- In a collection
- Returning a cursor to the result set
Inline results are subject to the BSON maximum document size of 16 MB, meaning that we should use this only if our final result is of fixed size. An example of this would be outputting the ObjectIds of the top five most ordered items from an e-commerce site.
A contrary example to that would be outputting the top 1,000 ordered items, along with the product information, including the description and other fields of variable size.
Outputting results into a collection is the preferred solution if we want to perform further processing of data. We can either output into a new collection or replace the contents of an existing collection. The aggregation output results will only be visible once the aggregation command succeeds, or else not visible at all.
If the aggregation output violates indexes (including the built-in index on the unique ObjectId per document) or document validation rules, aggregation will fail.
With all three options, every document output must not exceed the BSON maximum document size of 16 MB.
Each pipeline stage can have documents exceeding the 16 MB limit as these are handled by MongoDB internally. Each pipeline stage, however, can only use up to 100 MB of memory. If we expect more data in our stages, we should set allowDiskUse: to true to allow exceeding data to overflow to disk at the expense of performance.
$graphLookup does not support datasets over 100 MB and will ignore any setting on allowDiskUse.