Restricted Boltzmann Machines
by Krishna Choppella, Dr. Uday Kamath, Boštjan Kaluža, Jennifer L. Reese, Richard M
Machine Learning: End-to-End guide for Java developers
- Machine Learning: End-to-End guide for Java developers
- Table of Contents
- Machine Learning: End-to-End guide for Java developers
- Credits
- Preface
- 1. Module 1
- 1. Getting Started with Data Science
- Problems solved using data science
- Understanding the data science problem - solving approach
- Acquiring data for an application
- The importance and process of cleaning data
- Visualizing data to enhance understanding
- The use of statistical methods in data science
- Machine learning applied to data science
- Using neural networks in data science
- Deep learning approaches
- Performing text analysis
- Visual and audio analysis
- Improving application performance using parallel techniques
- Assembling the pieces
- Summary
- 2. Data Acquisition
- 3. Data Cleaning
- 4. Data Visualization
- 5. Statistical Data Analysis Techniques
- 6. Machine Learning
- 7. Neural Networks
- 8. Deep Learning
- 9. Text Analysis
- 10. Visual and Audio Analysis
- 11. Mathematical and Parallel Techniques for Data Analysis
- 12. Bringing It All Together
- 1. Getting Started with Data Science
- 2. Module 2
- 1. Applied Machine Learning Quick Start
- 2. Java Libraries and Platforms for Machine Learning
- 3. Basic Algorithms – Classification, Regression, and Clustering
- 4. Customer Relationship Prediction with Ensembles
- 5. Affinity Analysis
- 6. Recommendation Engine with Apache Mahout
- 7. Fraud and Anomaly Detection
- 8. Image Recognition with Deeplearning4j
- 9. Activity Recognition with Mobile Phone Sensors
- 10. Text Mining with Mallet – Topic Modeling and Spam Detection
- 11. What is Next?
- A. References
- 3. Module 3
- 1. Machine Learning Review
- Machine learning – history and definition
- What is not machine learning?
- Machine learning – concepts and terminology
- Machine learning – types and subtypes
- Datasets used in machine learning
- Machine learning applications
- Practical issues in machine learning
- Machine learning – roles and process
- Machine learning – tools and datasets
- Summary
- 2. Practical Approach to Real-World Supervised Learning
- Formal description and notation
- Data transformation and preprocessing
- Feature relevance analysis and dimensionality reduction
- Model building
- Model assessment, evaluation, and comparisons
- Case Study – Horse Colic Classification
- Summary
- References
- 3. Unsupervised Machine Learning Techniques
- Issues in common with supervised learning
- Issues specific to unsupervised learning
- Feature analysis and dimensionality reduction
- Clustering
- Outlier or anomaly detection
- Real-world case study
- Summary
- References
- 4. Semi-Supervised and Active Learning
- Semi-supervised learning
- Active learning
- Case study in active learning
- Summary
- References
- 5. Real-Time Stream Machine Learning
- Assumptions and mathematical notations
- Basic stream processing and computational techniques
- Concept drift and drift detection
- Incremental supervised learning
- Incremental unsupervised learning using clustering
- Modeling techniques
- Partition based
- Hierarchical based and micro clustering
- Density based
- Grid based
- Validation and evaluation techniques
- Key issues in stream cluster evaluation
- Evaluation measures
- Cluster Mapping Measures (CMM)
- V-Measure
- Other external measures
- Unsupervised learning using outlier detection
- Case study in stream learning
- Summary
- References
- 6. Probabilistic Graph Modeling
- Probability revisited
- Graph concepts
- Bayesian networks
- Representation
- Inference
- Learning
- Learning parameters
- Learning structures
- Measures to evaluate structures
- Methods for learning structures
- Constraint-based techniques
- Search and score-based techniques
- 7. Deep Learning
- Multi-layer feed-forward neural network
- Inputs, neurons, activation function, and mathematical notation
- Multi-layered neural network
- Structure and mathematical notations
- Activation functions in NN
- Training neural network
- Empirical risk minimization
- Parameter initialization
- Loss function
- Gradients
- Feed forward and backpropagation
- How does it work?
- Regularization
- L2 regularization
- L1 regularization
- Limitations of neural networks
- Deep learning
- Building blocks for deep learning
- Rectified linear activation function
- Restricted Boltzmann Machines
- Autoencoders
- Unsupervised pre-training and supervised fine-tuning
- Deep feed-forward NN
- Deep Autoencoders
- Deep Belief Networks
- Deep learning with dropouts
- Sparse coding
- Convolutional Neural Network
- CNN Layers
- Recurrent Neural Networks
- Building blocks for deep learning
- Case study
- Summary
- References
- 8. Text Mining and Natural Language Processing
- NLP, subfields, and tasks
- Text categorization
- Part-of-speech tagging (POS tagging)
- Text clustering
- Information extraction and named entity recognition
- Sentiment analysis and opinion mining
- Coreference resolution
- Word sense disambiguation
- Machine translation
- Semantic reasoning and inferencing
- Text summarization
- Automating question and answers
- Issues with mining unstructured data
- Text processing components and transformations
- Topics in text mining
- Tools and usage
- Summary
- References
- NLP, subfields, and tasks
- 9. Big Data Machine Learning – The Final Frontier
- What are the characteristics of Big Data?
- Big Data Machine Learning
- Batch Big Data Machine Learning
- Case study
- Business problem
- Machine Learning mapping
- Data collection
- Data sampling and transformation
- Spark MLlib as Big Data Machine Learning platform
- A. Linear Algebra
- B. Probability
- D. Bibliography
- Index
- Empirical risk minimization
- Multi-layer feed-forward neural network
- Modeling techniques
- 1. Machine Learning Review
RBM is often used as part of a multi-layer deep belief network. The output of the RBM is used as an input to another layer. The use of the RBM is repeated until the final layer is reached.
Note
Deep Belief Networks (DBNs) consist of several RBMs stacked together. Each hidden layer provides the input for the subsequent layer. Within each layer, the nodes cannot communicate laterally and it becomes essentially a network of other single-layer networks. DBNs are especially helpful for classifying, clustering, and recognizing image data.
The term, continuous restricted Boltzmann machine, refers an RBM that uses values other than integers. Input data is normalized to values between zero and one.
Each node of the input layer is connected to each node of the second layer. No nodes of the same layer are connected to each other. That is, there is no intra-layer communication. This is what restricted means.
The number of input nodes for the visible layer is dependent on the problem being solved. For example, if we are looking at an image with 256 pixels, then we will need 256 input nodes. For an image, this is the number of rows times the number of columns for the image.
The Hidden Layer should contain fewer neurons than the Input Layer. Using close to the same number of neurons will sometimes result in the construction of an identity function. Too many neurons may result in overfitting. This means that datasets with a large number of inputs will require multiple layers. Smaller input sizes result in the need for fewer layers.
Stochastic, that is, random, values are assigned to each node's weights. The value for a node is multiplied by its weight and then added to a bias. This value, combined with the combined input from the other input nodes, is then fed into the activation function, where an output value is generated.
The RBM technique goes through a reconstruction phase. This is where the activations are fed back to the first layer and multiplied by the same weights used for the input. The sum of these values from each node of the second layer, plus another bias, represents an approximation of the original input. The idea is to train the model to minimize the difference between the original input values and the feedback values.
The difference in values is treated as an error. The process is repeated until an error minimum is reached. You can think of the reconstruction as guesses about the original input. These guesses are essentially a probability distribution of the original input. This is called generative learning, in contrast to discriminative learning, which occurs with classification techniques.
In a multi-layer model, each layer can be used to essentially identify a feature. In subsequent layers, a combination of features may be identified or generated. In this way, a seemingly random set of pixel values may be analyzed to identify the veins of a leaf, a leaf, a trunk, and then a tree.
The output of an RBM is a value that essentially represents a percentage. If it is not zero, then the machine has learned something about the input.
We will examine two different RBM configurations. The first one is minimal and we will see it again in Deep autoencoders. The second uses several additional methods and provides more insights into the various ways it can be configured.
The following statement creates a new layer using the RBM.Builder class. The input is computed based on the number of rows and columns of an image. The output is large, containing 1000 neurons. The loss function is RMSE_XENT. This loss function works better for some classification problems:
.layer(0, new RBM.Builder()
.nIn(numRows * numColumns).nOut(1000)
.lossFunction(LossFunctions.LossFunction.RMSE_XENT)
.build())
Next is a more complex RBM. We will not detail each of these methods here but will see them used in later examples:
.layer(new RBM.Builder()
.l2(1e-1).l1(1e-3)
.nIn(numRows * numColumns
.nOut(outputNum)
.activation("relu")
.weightInit(WeightInit.RELU)
.lossFunction(LossFunctions.LossFunction
.RECONSTRUCTION_CROSSENTROPY).k(3)
.hiddenUnit(HiddenUnit.RECTIFIED)
.visibleUnit(VisibleUnit.GAUSSIAN)
.updater(Updater.ADAGRAD)
.gradientNormalization(
GradientNormalization.ClipL2PerLayer)
.build())
A single-layer RBM is not always useful. A multi-layer autoencoder is often required. We will look at this option in the next section.
-
No Comment