There are some common challenges associated with the tasks in an agile software development, and those challenges become more complex while testing the software in a distributed environment before deploying them eventually. Often team members need to merge the software components in parallel after the bugs proliferating. However, based on urgency, often the merging occurs before testing phase. Sometimes, many stakeholders are distributed across teams. Therefore, there's a huge potential for misunderstanding and teams often lose in between.

For example, Cloud Foundry (https://www.cloudfoundry.org/) is an open source heavily distributed PaaS software system for managing deployment and scalability of applications in the Cloud. It promises different features such as scalability, reliability, and elasticity that come inherently to deployments on Cloud Foundry require the underlying distributed system to implement measures to ensure robustness, resiliency, and failover.

The process of software testing is long known to comprise unit testing, integration testing, smoke testing, acceptance testing, scalability testing, performance testing, and quality of service testing. In Cloud Foundry, the process of testing a distributed system is shown in the following figure:

As shown in the preceding figure (first column), the process of testing in a distributed environment like Cloud starts with running unit tests against the smallest points of contract in the system. Following successful execution of all the unit tests, integration tests are run to validate the behavior of interacting components as part of a single coherent software system (second column) running on a single box (for example, a Virtual Machine (VM) or bare metal). However, while these tests validate the overall behavior of the system as a monolith, they do not guarantee system validity in a distributed deployment. Once integration tests pass, the next step (third column) is to validate distributed deployment of the system and run the smoke tests.

As you know, that the successful configuration of the software and execution of unit tests prepares us to validate acceptability of system behavior. This verification is done by running acceptance tests (fourth column). Now, to overcome the aforementioned issues and challenges in distributed environments, there are also other hidden challenges that need to be solved by researchers and big data engineers, but those are actually out of the scope of this book.

Now that we know what real challenges are for the software testing in a distributed environment, now let's start testing our Spark code a bit. The next section is dedicated to testing Spark applications.