In this section, we will see how to debug Spark applications that are running locally (on Eclipse or IntelliJ), standalone or cluster mode in YARN or Mesos. However, before diving deeper, it is necessary to know about logging in the Spark application.
- Preface
- Introduction to Scala
- Object-Oriented Scala
- Functional Programming Concepts
- Collection APIs
- Tackle Big Data – Spark Comes to the Party
- Start Working with Spark – REPL and RDDs
- Special RDD Operations
- Introduce a Little Structure - Spark SQL
- Stream Me Up, Scotty - Spark Streaming
- Everything is Connected - GraphX
- Learning Machine Learning - Spark MLlib and Spark ML
- Introduction to machine learning
- Spark machine learning APIs
- Feature extraction and transformation
- Creating a simple pipeline
- Unsupervised machine learning
- Binary and multiclass classification
- Summary
- My Name is Bayes, Naive Bayes
- Time to Put Some Order - Cluster Your Data with Spark MLlib
- Text Analytics Using Spark ML
- Spark Tuning
- Time to Go to ClusterLand - Deploying Spark on a Cluster
- Spark architecture in a cluster
- Deploying the Spark application on a cluster
- Submitting Spark jobs
- Hadoop YARN
- Configuring a single-node YARN cluster
- Step 1: Downloading Apache Hadoop
- Step 2: Setting the JAVA_HOME
- Step 3: Creating users and groups
- Step 4: Creating data and log directories
- Step 5: Configuring core-site.xml
- Step 6: Configuring hdfs-site.xml
- Step 7: Configuring mapred-site.xml
- Step 8: Configuring yarn-site.xml
- Step 9: Setting Java heap space
- Step 10: Formatting HDFS
- Step 11: Starting the HDFS
- Step 12: Starting YARN
- Step 13: Verifying on the web UI
- Submitting Spark jobs on YARN cluster
- Advance job submissions in a YARN cluster
- Configuring a single-node YARN cluster
- Apache Mesos
- Deploying on AWS
- Summary
- Testing and Debugging Spark
- PySpark and SparkR
Debugging Spark applications
-
No Comment
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.