This recipe covers the generalized regression model (GLM) implementation in Spark 2.0. There is a great parallel between this GeneralizedLinearRegression in Spark 2.0 and glmnet implementation in R. This API is a welcome addition that allows you to select and set both distribution family (for example, Gaussian) and link functions (for example, inverse log) with a coherent and well-designed API.
Generalized linear regression in Spark 2.0
by Shuen Mei, Broderick Hall, Meenakshi Rajendran, Siamak Amirghodsi
Apache Spark 2.x Machine Learning Cookbook
- Preface
- Practical Machine Learning with Spark Using Scala
- Introduction
- Downloading and installing the JDK
- Downloading and installing IntelliJ
- Downloading and installing Spark
- Configuring IntelliJ to work with Spark and run Spark ML sample codes
- Running a sample ML code from Spark
- Identifying data sources for practical machine learning
- Running your first program using Apache Spark 2.0 with the IntelliJ IDE
- How to add graphics to your Spark program
- Just Enough Linear Algebra for Machine Learning with Spark
- Introduction
- Package imports and initial setup for vectors and matrices
- Creating DenseVector and setup with Spark 2.0
- Creating SparseVector and setup with Spark
- Creating dense matrix and setup with Spark 2.0
- Using sparse local matrices with Spark 2.0
- Performing vector arithmetic using Spark 2.0
- Performing matrix arithmetic using Spark 2.0
- Exploring RowMatrix in Spark 2.0
- Exploring Distributed IndexedRowMatrix in Spark 2.0
- Exploring distributed CoordinateMatrix in Spark 2.0
- Exploring distributed BlockMatrix in Spark 2.0
- Spark's Three Data Musketeers for Machine Learning - Perfect Together
- Introduction
- Creating RDDs with Spark 2.0 using internal data sources
- Creating RDDs with Spark 2.0 using external data sources
- Transforming RDDs with Spark 2.0 using the filter() API
- Transforming RDDs with the super useful flatMap() API
- Transforming RDDs with set operation APIs
- RDD transformation/aggregation with groupBy() and reduceByKey()
- Transforming RDDs with the zip() API
- Join transformation with paired key-value RDDs
- Reduce and grouping transformation with paired key-value RDDs
- Creating DataFrames from Scala data structures
- Operating on DataFrames programmatically without SQL
- Loading DataFrames and setup from an external source
- Using DataFrames with standard SQL language - SparkSQL
- Working with the Dataset API using a Scala Sequence
- Creating and using Datasets from RDDs and back again
- Working with JSON using the Dataset API and SQL together
- Functional programming with the Dataset API using domain objects
- Common Recipes for Implementing a Robust Machine Learning System
- Introduction
- Spark's basic statistical API to help you build your own algorithms
- ML pipelines for real-life machine learning applications
- Normalizing data with Spark
- Splitting data for training and testing
- Common operations with the new Dataset API
- Creating and using RDD versus DataFrame versus Dataset from a text file in Spark 2.0
- LabeledPoint data structure for Spark ML
- Getting access to Spark cluster in Spark 2.0
- Getting access to Spark cluster pre-Spark 2.0
- Getting access to SparkContext vis-a-vis SparkSession object in Spark 2.0
- New model export and PMML markup in Spark 2.0
- Regression model evaluation using Spark 2.0
- Binary classification model evaluation using Spark 2.0
- Multiclass classification model evaluation using Spark 2.0
- Multilabel classification model evaluation using Spark 2.0
- Using the Scala Breeze library to do graphics in Spark 2.0
- Practical Machine Learning with Regression and Classification in Spark 2.0 - Part I
- Introduction
- Fitting a linear regression line to data the old fashioned way
- Generalized linear regression in Spark 2.0
- Linear regression API with Lasso and L-BFGS in Spark 2.0
- Linear regression API with Lasso and 'auto' optimization selection in Spark 2.0
- Linear regression API with ridge regression and 'auto' optimization selection in Spark 2.0
- Isotonic regression in Apache Spark 2.0
- Multilayer perceptron classifier in Apache Spark 2.0
- One-vs-Rest classifier (One-vs-All) in Apache Spark 2.0
- Survival regression – parametric AFT model in Apache Spark 2.0
- Practical Machine Learning with Regression and Classification in Spark 2.0 - Part II
- Introduction
- Linear regression with SGD optimization in Spark 2.0
- Logistic regression with SGD optimization in Spark 2.0
- Ridge regression with SGD optimization in Spark 2.0
- Lasso regression with SGD optimization in Spark 2.0
- Logistic regression with L-BFGS optimization in Spark 2.0
- Support Vector Machine (SVM) with Spark 2.0
- Naive Bayes machine learning with Spark 2.0 MLlib
- Exploring ML pipelines and DataFrames using logistic regression in Spark 2.0
- Recommendation Engine that Scales with Spark
- Introduction
- Setting up the required data for a scalable recommendation engine in Spark 2.0
- Exploring the movies data details for the recommendation system in Spark 2.0
- Exploring the ratings data details for the recommendation system in Spark 2.0
- Building a scalable recommendation engine using collaborative filtering in Spark 2.0
- Unsupervised Clustering with Apache Spark 2.0
- Introduction
- Building a KMeans classifying system in Spark 2.0
- Bisecting KMeans, the new kid on the block in Spark 2.0
- Using Gaussian Mixture and Expectation Maximization (EM) in Spark to classify data
- Classifying the vertices of a graph using Power Iteration Clustering (PIC) in Spark 2.0
- Latent Dirichlet Allocation (LDA) to classify documents and text into topics
- Streaming KMeans to classify data in near real-time
- Optimization - Going Down the Hill with Gradient Descent
- Introduction
- Optimizing a quadratic cost function and finding the minima using just math to gain insight
- Coding a quadratic cost function optimization using Gradient Descent (GD) from scratch
- Coding Gradient Descent optimization to solve Linear Regression from scratch
- Normal equations as an alternative for solving Linear Regression in Spark 2.0
- Building Machine Learning Systems with Decision Tree and Ensemble Models
- Introduction
- Getting and preparing real-world medical data for exploring Decision Trees and Ensemble models in Spark 2.0
- Building a classification system with Decision Trees in Spark 2.0
- Solving Regression problems with Decision Trees in Spark 2.0
- Building a classification system with Random Forest Trees in Spark 2.0
- Solving regression problems with Random Forest Trees in Spark 2.0
- Building a classification system with Gradient Boosted Trees (GBT) in Spark 2.0
- Solving regression problems with Gradient Boosted Trees (GBT) in Spark 2.0
- Curse of High-Dimensionality in Big Data
- Implementing Text Analytics with Spark 2.0 ML Library
- Introduction
- Doing term frequency with Spark - everything that counts
- Displaying similar words with Spark using Word2Vec
- Downloading a complete dump of Wikipedia for a real-life Spark ML project
- Using Latent Semantic Analysis for text analytics with Spark 2.0
- Topic modeling with Latent Dirichlet allocation in Spark 2.0
- Spark Streaming and Machine Learning Library
- Introduction
- Structured streaming for near real-time machine learning
- Streaming DataFrames for real-time machine learning
- Streaming Datasets for real-time machine learning
- Streaming data and debugging with queueStream
- Downloading and understanding the famous Iris data for unsupervised classification
- Streaming KMeans for a real-time on-line classifier
- Downloading wine quality data for streaming regression
- Streaming linear regression for a real-time regression
- Downloading Pima Diabetes data for supervised classification
- Streaming logistic regression for an on-line classifier
Generalized linear regression in Spark 2.0
-
No Comment
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.