"Harpists spend 90 percent of their lives tuning their harps and 10 percent playing out of tune."
- Igor Stravinsky
In this chapter, we will dig deeper into Apache Spark internals and see that while Spark is great in making us feel like we are using just another Scala collection, we don't have to forget that Spark actually runs in a distributed system. Therefore, some extra care should be taken. In a nutshell, the following topics will be covered in this chapter:
- Monitoring Spark jobs
- Spark configuration
- Common mistakes in Spark app development
- Optimization techniques