Now that we have our algorithm coded, let's look at practical applications for this method on real data. We will start by understanding how the algorithm performs, so that we can determine where we might use it.
So, what are the characteristics of this algorithm? Below is a list of strengths and weaknesses.
The advantages are as follows:
- The algorithm is general, lending itself well to both stream based and Spark implementations
- The theory is simple, yet effective
- The implementation is fast and efficient
- The result is visual and interpretable
- The method is stackable and allows for multi scale studies; this is very simple when using Spark windows
The disadvantages are as follows:
- A lagging indicator the algorithm finds trend reversals that occurred in the past, and cannot be used directly to predict a trend change as it happens
- The lag accumulates for higher scales, meaning much more data (and thus time lag) is required to find long-range trend changes versus finding trend reversals on shorter timeframes
It's important to understand the limitations of this algorithm. We have created a very useful analysis tool that can be used for researching trends. However, it is not in itself a prediction tool, rather a tool to more easily identify trends for follow-on processing.
With our newly found ability to convert a time series into a list of change points, many use cases that were once difficult become easy. Let's take a look at some potential applications.
We can retrieve News Headlines from the GDELT feed at moments when trend changes occur, at major highs or lows, thus annotating our charts with context.
We can use the reduction in noise to compare trends across different time series, and devise calculations to measure which are co-trending.
We can use the algorithm to simplify time series and reduce data volumes, while retaining critical moments, stacking the algorithm allows for greater reductions.
We can view the change points as a novel form of index to the time series, allowing, for example, the retrieval of portions of the data where things were on a short time frame running counter to a trend on a longer time frame.
We can find change points on different time scales, and use the information to investigate the fractal dimensions of the time series.