Anomalous pattern detection
by Krishna Choppella, Dr. Uday Kamath, Boštjan Kaluža, Jennifer L. Reese, Richard M
Machine Learning: End-to-End guide for Java developers
- Machine Learning: End-to-End guide for Java developers
- Table of Contents
- Machine Learning: End-to-End guide for Java developers
- Credits
- Preface
- 1. Module 1
- 1. Getting Started with Data Science
- Problems solved using data science
- Understanding the data science problem - solving approach
- Acquiring data for an application
- The importance and process of cleaning data
- Visualizing data to enhance understanding
- The use of statistical methods in data science
- Machine learning applied to data science
- Using neural networks in data science
- Deep learning approaches
- Performing text analysis
- Visual and audio analysis
- Improving application performance using parallel techniques
- Assembling the pieces
- Summary
- 2. Data Acquisition
- 3. Data Cleaning
- 4. Data Visualization
- 5. Statistical Data Analysis Techniques
- 6. Machine Learning
- 7. Neural Networks
- 8. Deep Learning
- 9. Text Analysis
- 10. Visual and Audio Analysis
- 11. Mathematical and Parallel Techniques for Data Analysis
- 12. Bringing It All Together
- 1. Getting Started with Data Science
- 2. Module 2
- 1. Applied Machine Learning Quick Start
- 2. Java Libraries and Platforms for Machine Learning
- 3. Basic Algorithms – Classification, Regression, and Clustering
- 4. Customer Relationship Prediction with Ensembles
- 5. Affinity Analysis
- 6. Recommendation Engine with Apache Mahout
- 7. Fraud and Anomaly Detection
- 8. Image Recognition with Deeplearning4j
- 9. Activity Recognition with Mobile Phone Sensors
- 10. Text Mining with Mallet – Topic Modeling and Spam Detection
- 11. What is Next?
- A. References
- 3. Module 3
- 1. Machine Learning Review
- Machine learning – history and definition
- What is not machine learning?
- Machine learning – concepts and terminology
- Machine learning – types and subtypes
- Datasets used in machine learning
- Machine learning applications
- Practical issues in machine learning
- Machine learning – roles and process
- Machine learning – tools and datasets
- Summary
- 2. Practical Approach to Real-World Supervised Learning
- Formal description and notation
- Data transformation and preprocessing
- Feature relevance analysis and dimensionality reduction
- Model building
- Model assessment, evaluation, and comparisons
- Case Study – Horse Colic Classification
- Summary
- References
- 3. Unsupervised Machine Learning Techniques
- Issues in common with supervised learning
- Issues specific to unsupervised learning
- Feature analysis and dimensionality reduction
- Clustering
- Outlier or anomaly detection
- Real-world case study
- Summary
- References
- 4. Semi-Supervised and Active Learning
- Semi-supervised learning
- Active learning
- Case study in active learning
- Summary
- References
- 5. Real-Time Stream Machine Learning
- Assumptions and mathematical notations
- Basic stream processing and computational techniques
- Concept drift and drift detection
- Incremental supervised learning
- Incremental unsupervised learning using clustering
- Modeling techniques
- Partition based
- Hierarchical based and micro clustering
- Density based
- Grid based
- Validation and evaluation techniques
- Key issues in stream cluster evaluation
- Evaluation measures
- Cluster Mapping Measures (CMM)
- V-Measure
- Other external measures
- Unsupervised learning using outlier detection
- Case study in stream learning
- Summary
- References
- 6. Probabilistic Graph Modeling
- Probability revisited
- Graph concepts
- Bayesian networks
- Representation
- Inference
- Learning
- Learning parameters
- Learning structures
- Measures to evaluate structures
- Methods for learning structures
- Constraint-based techniques
- Search and score-based techniques
- 7. Deep Learning
- Multi-layer feed-forward neural network
- Inputs, neurons, activation function, and mathematical notation
- Multi-layered neural network
- Structure and mathematical notations
- Activation functions in NN
- Training neural network
- Empirical risk minimization
- Parameter initialization
- Loss function
- Gradients
- Feed forward and backpropagation
- How does it work?
- Regularization
- L2 regularization
- L1 regularization
- Limitations of neural networks
- Deep learning
- Building blocks for deep learning
- Rectified linear activation function
- Restricted Boltzmann Machines
- Autoencoders
- Unsupervised pre-training and supervised fine-tuning
- Deep feed-forward NN
- Deep Autoencoders
- Deep Belief Networks
- Deep learning with dropouts
- Sparse coding
- Convolutional Neural Network
- CNN Layers
- Recurrent Neural Networks
- Building blocks for deep learning
- Case study
- Summary
- References
- 8. Text Mining and Natural Language Processing
- NLP, subfields, and tasks
- Text categorization
- Part-of-speech tagging (POS tagging)
- Text clustering
- Information extraction and named entity recognition
- Sentiment analysis and opinion mining
- Coreference resolution
- Word sense disambiguation
- Machine translation
- Semantic reasoning and inferencing
- Text summarization
- Automating question and answers
- Issues with mining unstructured data
- Text processing components and transformations
- Topics in text mining
- Tools and usage
- Summary
- References
- NLP, subfields, and tasks
- 9. Big Data Machine Learning – The Final Frontier
- What are the characteristics of Big Data?
- Big Data Machine Learning
- Batch Big Data Machine Learning
- Case study
- Business problem
- Machine Learning mapping
- Data collection
- Data sampling and transformation
- Spark MLlib as Big Data Machine Learning platform
- A. Linear Algebra
- B. Probability
- D. Bibliography
- Index
- Empirical risk minimization
- Multi-layer feed-forward neural network
- Modeling techniques
- 1. Machine Learning Review
The second approach uses the pattern library in an inverse fashion, meaning that the library encodes only positive patterns marked with green plus signs in the following image. When an observed behavior (blue circle) cannot be matched against the library, it is considered anomalous:
This approach requires us to model only what we have seen in the past, that is, normal patterns. If we return to the doctor example, the main reason we visited the doctor in the first place was because we did not feel fine. Our perceived state of feelings (for example, headache, sore skin) did not match our usual feelings, therefore, we decided to seek doctor. We don't know which disease caused this state nor do we know the treatment, but we were able to observe that it doesn't match the usual state.
A major advantage of this approach is that it does not require us to say anything about non-normal patterns; hence, it is appropriate for modeling known-unknowns and unknown-unknowns. On the other hand, it does not tell us what exactly is wrong.
Several approaches have been proposed to tackle the problem either way. We broadly classify anomalous and suspicious behavior detection in the following three categories: pattern analysis, transaction analysis, and plan recognition. In the following sections, we will quickly look into some real-life applications.
An active area of anomalous and suspicious behavior detection from patterns is based on visual modalities such as camera. Zhang et al (2007) proposed a system for a visual human motion analysis from a video sequence, which recognizes unusual behavior based on walking trajectories; Lin et al (2009) described a video surveillance system based on color features, distance features, and a count feature, where evolutionary techniques are used to measure observation similarity. The system tracks each person and classifies their behavior by analyzing their trajectory patterns. The system extracts a set of visual low-level features in different parts of the image, and performs a classification with SVMs to detect aggressive, cheerful, intoxicated, nervous, neutral, and tired behavior.
Transaction analysis assumes discrete states/transactions in contrast to continuous observations. A major research area is Intrusion Detection (ID) that aims at detecting attacks against information systems in general. There are two types of ID systems, signature-based and anomaly-based, that broadly follow the suspicious and anomalous pattern detection as described in the previous sections. A comprehensive review of ID approaches was published by Gyanchandani et al (2012).
Furthermore, applications in ambient-assisted living that are based on wearable sensors also fit to transaction analysis as sensing is typically event-based. Lymberopoulos et al (2008) proposed a system for automatic extraction of the users' spatio-temporal patterns encoded as sensor activations from the sensor network deployed inside their home. The proposed method, based on location, time, and duration, was able to extract frequent patterns using the Apriori algorithm and encode the most frequent patterns in the form of a Markov chain. Another area of related work includes Hidden Markov Models (HMMs) (Rabiner, 1989) that are widely used in traditional activity recognition for modeling a sequence of actions, but these topics are already out of scope of this book.
Plan recognition focuses on a mechanism for recognizing the unobservable state of an agent, given observations of its interaction with its environment (Avrahami-Zilberbrand, 2009). Most existing investigations assume discrete observations in the form of activities. To perform anomalous and suspicious behavior detection, plan recognition algorithms may use a hybrid approach, a symbolic plan recognizer is used to filter consistent hypotheses, passing them to an evaluation engine, which focuses on ranking.
These were advanced approaches applied to various real-life scenarios targeted at discovering anomalies. In the following sections, we'll dive into more basic approaches for suspicious and anomalous pattern detection.
-
No Comment