We now have all the necessary tools to look at more complex problems that are more commonly called the big data problems. Armed with standard statistical algorithms—I understand that I have not covered many details and I am completely ready to accept the criticism—there is an entirely new ground to explore where we do not have clearly defined records, the variables in the datasets may be sparse and nested, and we have to cover a lot of ground and do a lot of preparatory work just to get to the stage where we can apply the standard statistical models. This is where Scala shines best.

In the next chapter, we will look more at working with unstructured data.