In particular circumstances, our application may be required to handle huge file uploads. This can be done by putting all of these in the memory, by creating a temporary file, or by acting directly on the stream. Out of these three, the last option works the best for us, as it removes I/O stream limitations (such as blocking, memory, and threads) and also eliminates the need to buffer (that is, acting on input at the rate needed).
Handling huge file uploads belongs to the set of unavoidable operations that can be heavy on resources. Some other tasks that belong to the same category are processing real-time data for monitoring, analysis, bulk data transfers, and processing large datasets. In this chapter, we will discuss the Iteratee approach used to handle such situations. This chapter covers the basics of handling data streams with a brief explanation of the following topics:
- Iteratees
- Enumerators
- Enumeratees
This chapter may seem intense at times but the topics discussed here will be helpful for some of the following chapters.
Consider that we connected a mobile device (such as a tablet, phone, MP3 player, and so on) to its charger and plugged it in. The consequences of this can be as follows:
- The device's battery starts charging and continues to do so until the occurrence of one of the other options
- The device's battery is completely charged and minimal power is drawn by the device to continue running
- The device's battery can not be charged due to malfunctioning of the device
Here, the power supply is the source, the device is the sink, while the charger is the channel that enables transfer of energy from the source to the sink. The processing or task performed by the device is that of charging its battery.
Well, this covers most of the Iteratee approach without any of the usual jargon. Simply put, the power supply represents a data source, the charger acts as the Enumerator, and the device as the Iteratee.
Oops, we missed the Enumeratee! Suppose that the energy from a regular power supply is not compatible with the device; then, in this case, the charger generally has an internal component that performs this transformation. For example, converting from A.C. (alternating current) to D.C. (direct current). In such cases, the charger can be considered a combination of the Enumerator and the Enumeratee. The component that collects energy from the power supply acts like the Enumerator, and the other component that transforms the energy is similar to an Enumeratee.
The concept of Iteratee, Enumerator, and Enumeratee originated from the Haskell library Iteratee I/O, which was developed by Oleg Kiselyov to overcome the problems faced with lazy I/O.
In Oleg's words, as seen on http://okmij.org/ftp/Streams.html:
Enumerator is an encapsulation of a data source, a stream producer—what folds an iteratee over the stream. An enumerator takes an iteratee and applies it to the stream data as they are being produced, until the source is depleted or the iteratee said it had enough. After disposing of buffers and other source-draining resources, enumerator returns the final value of the iteratee. Enumerator thus is an iteratee transformer.
Iteratees are stream consumers and an Iteratee can be in one of the following states:
- Completed or done: The Iteratee has completed processing
- Continuing: The current element has been processed but the Iteratee is not done yet and can accept the next element
- Error: The Iteratee has encountered an error
Enumeratee is both a consumer and a producer, incrementally decoding the outer stream and producing the nested stream of decoded data.
Although the enumerator knows how to get to the next element, it is completely unaware of the processing the Iteratee will perform on this element and vice versa.
Different libraries implement the Iteratee, Enumerator, and Enumeratee differently, based on these definitions. In the following sections, we will see how these are implemented in Play Framework and how we can use them in our application. Let's start with the Iteratee, as the Enumerator requires a one.