Here we will discuss some major issues that need to be taken care of during the software and hardware testing so that Spark jobs run smoothly in cluster computing, which is essentially a distributed computing environment.
Note that all the issues are unavoidable, but we can at least tune them for betterment. You should follow the instructions and recommendations given in the previous chapter. According to Kamal Sheel Mishra and Anil Kumar Tripathi, Some Issues, Challenges and Problems of Distributed Software System, in International Journal of Computer Science and Information Technologies, Vol. 5 (4), 2014, 4922-4925. URL: https://pdfs.semanticscholar.org/4c6d/c4d739bad13bcd0398e5180c1513f18275d8.pdf, there are several issues that need to be addressed while working with software or hardware in a distributed environment:
- Scalability
- Heterogeneous languages, platform, and architecture
- Resource management
- Security and privacy
- Transparency
- Openness
- Interoperability
- Quality of service
- Failure management
- Synchronization
- Communications
- Software architectures
- Performance analysis
- Generating test data
- Component selection for testing
- Test sequence
- Testing for system scalability and performance
- Availability of source code
- Reproducibility of events
- Deadlocks and race conditions
- Testing for fault tolerance
- Scheduling issue for distributed system
- Distributed task allocation
- Testing distributed software
- Monitoring and control mechanism from the hardware abstraction level
It's true that we cannot fully solve all of these issues, but However, using Spark, we can at least control a few of them that are related to distributed system. For example, scalability, resource management, quality of service, failure management, synchronization, communications, scheduling issue for distributed system, distributed task allocation, and monitoring and control mechanism in testing distributed software. Most of them were discussed in the previous two chapters. On the other hand, we can address some issues in the testing and software side: such as software architectures, performance analysis, generating test data, component selection for testing, test sequence, testing for system scalability and performance, and availability of source code. These will be covered explicitly or implicitly in this chapter at least.