Conclusion to Part III

Part III, consisting of nine chapters, described stream data analytics techniques with emphasis on big data for insider threat detection. In particular, both supervised and unsupervised learning methods for insider threat detection were discussed.

Chapter 14 provided a discussion of our approach to insider threat detection using stream data analytics and discussed the big data issue with respect to the problem. That is, massive amounts of stream data are emanating from various devices and we need to analyze this data for insider threat detection. Chapter 15 described related work in both insider threat detection and stream data mining. In addition, aspects of the big data issue were also discussed. Chapter 16 described ensemble-based learning for insider threat detection. In particular, we have described techniques for both supervised and unsupervised learning and discussed the issues involved. We believe that ensemble-based approaches are suited for data streams as they are unbounded. Chapter 17 described the different classes of learning techniques for nonsequence data. It described exactly how each method arrives at detecting insider threats and how ensemble models are built, modified, and discarded. First, we discussed supervised learning in detail and then discussed unsupervised learning. In Chapter 18, we discussed our testing methodology and experimental results for mining data streams consisting of nonsequence data. We examined various aspects such as false positives, false negatives, and accuracy. Our results indicate that supervising learning yields better results for certain datasets. In Chapter 19, we described both supervised and unsupervised learning techniques for mining data streams for sequence data. Experimental results of the techniques discussed in Chapter 19 were presented in Chapter 20. In particular, we discussed our datasets and testing methodology as well as our experimental results. Chapter 21 discussed how big data technologies can be used for stream mining to handle insider threats. In particular, we examine one of the techniques we have designed and showed how it can be redesigned using big data technologies. We also discussed our experimental results. Finally, Chapter 22 concluded with an assessment of the viability of stream mining for real-world insider threat detection and the relevance to big data aspects.

Now that we have discussed the various aspects of stream data analytics, handling massive data streams, as well as applying the techniques for insider threat detection, in Part IV we will describe the various experimental systems we have designed and developed for BDMA and BDSP.