In this chapter, we have introduced a method for analyzing trends with TrendCalculus. We have outlined the fact that despite analysis of trends being a very common use case, there are few tools to aid the data scientist in this cause apart from very general-purpose visualization software. We have guided the reader through the TrendCalculus algorithm, demonstrating how we implement an efficient and scalable realization of the theory in Spark. We have described the process of identifying the key output of the algorithm: trend reversals on a named scale. Having calculated reversals, we used D3.js to visualize time series data that has been summarized for one-week windows, and plotted trend reversals. The chapter continued with an explanation of how to overcome the main edge case: the zero values found during simple trend calculation. We have concluded with a brief outline of the algorithm characteristics and potential use cases, demonstrating how the method is elegant and can be easily described and realized in Spark.

In the next chapter, we will be demystifying the topic of data security. We'll describe the most important areas of security from a data science perspective, concentrating on the theory and implementation of sanctioned access for the handling of highly confidential data.