This chapter is the backbone for the chapters to come. We discussed a lot of challenges that every person/architect should be aware of when creating a data processing system within their organization. We discussed how to enable the large scale processing of data while keeping the overall system costs lower and how to keep the overall processing time within the defined SLA as the load on the processing system increases. Further, we learned that moving to the scale out architecture for data processing requires significant engineering effort as well as a shift in your perspective, and that the underlying components of the data processing system should not share resources but should be independent of each other. We covered the idea that failure is not an exception—the system should be built to handle failures. We also learned that the processing of data doesn't have to be always stream-like. The use case defines the processing pipeline design and not the other way around. We looked at when to process data, whether when it's moving or when it's at rest, and how to know what data to "not process". Finally, we explored when to decide not to traverse a particular network path, and how to effectively consume the processed data.

All of these concepts will feed into the next two chapters, where we will discuss each aspect in more detail. Stay tuned.