Hard Problems in Software Testing
Software testing is a critical phase of the software lifecycle, responsible for assuring the quality of a system under test; however, testing suffers from a variety of challenges—both new and timeless. In order to evaluate the ability of TaaS to overcome these challenges, the needs of the industry must first be identified.
In academia, a significant portion of testing-related research seems esoteric in nature. While the topics themselves are fascinating, the niche focus provides value to only a small portion of the field. The option arose to broaden the view and explore problems as they exist on a more universal scale. Many software testing challenges are well-known, especially those considered timeless; however, they need to be quantified to better determine which solutions could provide the most benefit to the industry.
The Hard Problems in Software Testing (HPST) survey was conducted to elicit the current issues plaguing the field in a quantifiable manner. Many software testing problems are challenging enough to demand sufficient research in the near future and important enough to the community that their resolution could advance the industry as a whole.
2.1 THE HPST SURVEY
The objective of the HPST survey was to obtain and quantify challenges that exist in the industry today, as encountered by testing professionals. The goal was to offer these findings as motivation for future research focused on the needs of the industry as a whole. The solutions themselves are far more complex than the issue at hand and were not considered as part of the initial survey effort.
The HPST survey was written in late 2011. The original survey is available in Appendix A. The following questions were asked:
• Tell us about yourself (name, affiliation, job title, email, and phone number).
• Describe five software problems you have struggled with in your job (these could be related to process, tools, education & training, and so on).
• Are you working in any of the following areas as a tester?
º Service-Oriented Architecture (SOA)
º Security/Information Assurance
º Cloud Computing
• What topics do you think the research community should focus on over the next five years to help you do your job better?
• Are you interested in collaborating with us? If so, on what problem area? In what capacity?
• Any additional comments.
The survey went live on SurveyMonkey [3] in November 2011 and was promoted on software testing forums. For the purpose of this research, answers were collected until November 2012. During that same time period, the survey was offered at seminars on Hard Problems in Software Testing at IBM CASCON 2011, STAREAST 2012, the 2012 Software Testing in the Cloud (STITC) Workshop, and at Florida Institute of Technology.
A total of sixty-four individuals participated in the HPST survey. Of these sixty-four, forty were industry professionals surveyed at conferences and on SurveyMonkey. These respondents represented over twenty companies including Lockheed Martin, Intel, Yahoo, and SAP. The experience of industry professionals is shown in Table 2.1. The four specific areas (SOA, security, cloud, mobile) were addressed in the survey because it was felt that they represented some of the key topics that testers are (or soon will be) concerned with.
Table 2.1: Areas of testing experience
Area |
Frequency |
Service-Oriented Architecture (SOA) |
16 |
Security / Information Assurance |
13 |
Cloud Computing |
8 |
Mobile / Apps |
10 |
Considered separately, the other twenty-four were master and doctoral candidates in computer science, software engineering, and other related fields at Florida Institute of Technology. These students were surveyed during a seminar on Hard Problems in Software Testing in 2012. A small percentage had sufficient industry experience to answer questions in the same vein as industry professionals. The remaining responses demanded that the student surveys be considered independently of the industry professionals’ surveys.
2.2 SURVEY RESULTS
For the purpose of this research, the second and fourth questions were examined in detail. Using a qualitative method called grounded theory [14], the results were explored systematically. The individual responses were repeatedly examined until they were all coded, allowing them to be placed within categories. The responses are shown in the following sections, organized by these categories and ordered by frequency. The results are discussed in Section 2.3.
2.2.1 SOFTWARE TESTING PROBLEMS
Question: “Describe five software problems you have struggled with in your job.”
The results of the HPST survey’s second question concerning testing problems with which individuals have struggled can be divided into several focus areas: testing, test data, personnel, metrics, estimation, process, platforms/environment, security, and tools. Testing contains the following subcategories: manual, automated, regression, unit, and other. Figure 2.1 shows the breakdown of responses per category.
Figure 2.1: Breakdown of responses per focus area.
The individual responses to this question are listed by category in the Tables 2.2–2.10. For each response, the frequency of that response—the number of individuals that responded with the same statement—is included on the right.
Table 2.2: Testing problems and frequency
Testing Problems |
Frequency |
Manual Testing |
|
Need to create a more efficient/effective testing process |
2 |
Manual tests are not repeatable |
2 |
Tests take too long to setup and run |
2 |
Testing too time consuming |
1 |
Automated Testing |
|
Difficulty setting/starting up automated testing |
4 |
Automated testing underdeveloped or underused |
2 |
Generated tests never fully utilized |
2 |
Concerns of affordability (time and cost) |
2 |
Need to reduce feedback time from automated tests |
2 |
Need to run a very large number of test cases |
1 |
Determining what should be automated |
1 |
Inability to automate for complex systems |
1 |
Regression Testing |
|
Cost to write and maintain tests |
5 |
Difficulty maintaining tests |
3 |
Misconceptions or misuse of regression testing |
3 |
Incomplete regression test suites |
2 |
Inability to store large amounts of test data |
1 |
No existing regression tests |
1 |
Unit Testing |
|
Unit testing not performed or insufficient |
4 |
Need to motivate developers to engage in unit testing |
3 |
Unit testing difficult for web applications |
1 |
Other |
|
Need for performance testing |
2 |
Need for security testing |
2 |
Need for uniform certification testing |
1 |
Difficulties starting up Testing as a Service |
1 |
Table 2.3: Test data problems and frequency
Test Data Problems |
Frequency |
Difficulty managing large sets of test data |
3 |
Need to maintain data privacy |
1 |
Difficulty generating realistic and unique sets of data |
1 |
Table 2.4: Personnel problems and frequency
| Personnel Problems | Frequency |
| Lack of education/experience in testing methods
∙ Lack of education in creating tests |
12 |
| Need to improve communication and coordination between test team and other groups | 4 |
| Developers acting as testers | 2 |
| Lack of knowledge on system under test | 2 |
| Complex systems require more training | 2 |
| Difficulty keeping up with current technologies | 2 |
| Turnover rates | 2 |
| Need for better methods of knowledge sharing between experienced and novice employees | 2 |
| Test team feels underappreciated | 2 |
| Getting people to understand the importance of testing | 2 |
| Roles of test team need to be better defined | 1 |
| Lack of testers | 1 |
| Difficulty recruiting talent | 1 |
| Lack of programming skills | 1 |
| Need for tester certification | 1 |
Table 2.5: Metrics problems and frequency
| Metrics Problems | Frequency |
| Determining test coverage | 3 |
| Finding powerful and valuable metrics | 3 |
| Need to write better bug reports | 3 |
| Need to quantify defects | 2 |
| Determining how much coverage is required | 1 |
| Metrics nonexistent or not maintained | 1 |
Table 2.6: Estimation problems and frequency
| Estimation Problems | Frequency |
| Creating realistic schedules | 5 |
| Need help with estimation | 3 |
| Generating realistic budgets | 2 |
| Budgeting for tools, training, and setup | 2 |
| Inability to determine cost-benefit for cloud migration | 2 |
Table 2.7: Process problems and frequency
| Process Problems | Frequency |
| Too much/all of testing at end of cycle | 5 |
| Need process for testing in agile environments | 4 |
| Nonexistent, poor, or lacking procedures or test plans | 2 |
| Better management of testing needed | 2 |
| Lots of exit criteria but limited or no entry criteria | 2 |
| Process not documented | 1 |
| Need for continuous integration | 1 |
Table 2.8: Platforms/environment problems and frequency
| Platforms/Environment Problems | Frequency |
| Environment scenarios unrealistic | 3 |
| Lack of adequate simulators | 3 |
| Keeping up with technology/evolution | 2 |
| Maintaining up-to-date platforms | 2 |
| Shortage of testing hardware | 1 |
| Resources limited | 1 |
| Variations in platforms produce inconsistent results | 1 |
| Environments need better management | 1 |
Table 2.9: Security problems and frequency
| Security Problems | Frequency |
| Maintaining data privacy and security, especially when outsourcing testing | 2 |
| Security concerns when testing | 1 |
Table 2.10: Tools problems and frequency
| Tools Problems | Frequency |
| Need for tools or need for better tools | 8 |
| Tools need to be: | |
| ∙ Adaptable to new technology | 3 |
| ∙ Better documented or packaged with training material | 2 |
| ∙ Easier to learn/simpler | 2 |
| ∙ Properly tested | 1 |
| ∙ Sturdier | 1 |
| ∙ Able to meet rigorous security demands | 1 |
| Inconsistency in tools (different tools needed for different languages) | 1 |
| Lack of tools for existing baseline | 1 |
| Need help evaluating tools | 1 |
| Vendors need to be knowledgeable and trustworthy to help guide the user community | 1 |
Students without industry experience offered a different set of problems, highly influenced by software testing classes and their software development coursework. Some of the stated challenges were universal, aligning closely with the industry responses. Examples include time constraints, considering all testing criteria, and compatibility between applications, tools, and versions. Others were far more focused on development, debugging, and self-testing of the students’ own code, such as null pointer exceptions, array out of bounds exceptions, memory errors, and validity of inputs. While the similarities and differences in software testing problems were interesting, further discussion in this research is limited to responses from industry professionals.
2.2.2 TOPICS FOR THE RESEARCH COMMUNITY
Question: “What topics do you think the research community should focus on over the next five years to help you do your job better?”
The HPST survey also questioned individuals about what topics they believed that the research community should focus on in the near future. The results varied, but generally followed a similar pattern as the previously discussed question on the software testing problems. Recommended research topics are listed in Tables 2.11–2.16 and are categorized as follows: testing, personnel, estimation, tools, standards, and miscellaneous areas of interest. Student responses were sparse and were not considered in the following tables.
Table 2.11: Testing research topics
| Testing Topics | Frequency |
| How to improve efficiency and effectiveness of the testing process | 4 |
| Agile testing: How to adequately test a system without testers | 3 |
| How to improve regression testing | 2 |
| Automated/Regression testing: How to reduce the cost of testing | 2 |
| How much should be automated and when to automate | 1 |
| What is the minimal set of devices that should be tested | 1 |
| Guidelines for uniform certification testing | 1 |
Table 2.12: Personnel research topics
| Personnel Topics | Frequency |
| How to train/certify testers | 4 |
| How to enforce knowledge sharing amongst employees | 1 |
| Guidelines for testing education | 1 |
| How to improve communication between the test and other teams | 1 |
Table 2.13: Estimation research topics
| Estimation Topics | Frequency |
| How to estimate effort | 1 |
| How to estimate risk | 1 |
| How to estimate quality | 1 |
Table 2.14: Tools research topics
| Tools Topics | Frequency |
| Better software testing tools | 2 |
| What tools are needed and what tools already exist for specific testing purposes | 1 |
Table 2.15: Standards research topics
| Standards Topics | Frequency |
| Creating standards for cloud computing | 1 |
| Creating standards for mobile testing | 1 |
Table 2.16: Miscellaneous research topics
| Miscellaneous Topics | Frequency |
| Creating a common infrastructure to prevent reinventing the wheel | 1 |
| Knowledge sharing of frequently identified bugs in embedded testing | 1 |
| Improving overall system security | 1 |
| How to bring qualitative methods to the masses | 1 |
| Improving data handling | 1 |
| Providing better methods of simulating time | 1 |
| How to improve Service Level Agreements | 1 |
2.3 DISCUSSION OF RESULTS
The HPST survey offered a qualitative approach to identifying some of the top challenges the industry faces today. From the gathered responses, the top items were examined more closely and relationships between them analyzed. Additionally, the most desirable research areas become obvious and oftentimes tie closely to the aforementioned challenges.
2.3.1 SOFTWARE TESTING PROBLEMS
While a large number of software testing problems were identified over the course of the survey, the top five were easily established. Worthy of note, the vast majority of these areas could be considered timeless challenges—those existing since software testing’s inception. The following represent the most commonly observed issues in software testing today:
1. A desire for education and training
3. Lacking, insufficient, or nonexistent testing
4. Generating realistic schedules
5. A deficiency in communication
The single most commonly discussed software testing problem was tester education and training. Testing is a broad and constantly evolving field. Truly comprehending every aspect of it is a lofty goal—one that may be unachievable for the majority of working professionals. Nevertheless, testers should endeavor to become a “jack of all trades” within their field. Training, certification, and education can all help, but common standards for these three areas do not exist. Furthermore, determining how much training is necessary and on which topics testers should focus is no easy task, especially on a universal scale.
Beyond training, respondents frequently cited a need for tools or a need for better tools. A wide variety of software testing tools exist but many of them lack flexibility, limiting the tool’s usage to a small portion of projects, or are too complicated to comprehend and apply within the already restricted project schedule. Even if tools are found that can cover certain aspects of testing, the team must spend time and effort learning and using the various tools in a manner that meets their needs. This process can be quite expensive, especially after factoring in the cost of tools.
Issues with testing itself made up the third most common testing problem. Specifically, respondents described lacking, insufficient, or nonexistent testing, particularly in the areas of automated, regression, and unit testing. A major factor contributing to this challenge is the aforementioned need for education and training—having the knowledge required to implement the various desired tests within schedule and budget. Another factor is having enough time allotted in the schedule during which tests can be written, updated, and performed.
Another timeless software testing problem is generating realistic schedules that prevent testing from falling at the very end of the project lifecycle. In traditional models of the software lifecycle, the testing phase falls last, dependent on the completion of the previous phases. With less flexible deployment deadlines, testing is further reduced by delays in the requirements, design, and development phases, leaving little time for assuring quality. Creating, and more importantly adhering to, a realistic schedule can provide the test team with much-needed time. Even so, testing still demands more time and attention than it typically receives. The best option is to perform testing in a more agile manner—as an integral part of development.
Finally, respondents felt that communication, both within the team and with other groups, was a major issue within software testing. The internal communication was primarily concerned with knowledge sharing between novice and experienced team members—another aspect of the lacking training and education. External communication included exchanges between both the test team and the development team and the test team and management. To some extent, this leads to two other cited issues: the test team feels underappreciated and they are unable to illustrate the importance of testing.
2.3.2 TOPICS FOR THE RESEARCH COMMUNITY
Several of the recommended research areas closely align with the most frequently identified software testing problems. The training and education of testers tops both lists, with individuals requesting that work be done on how to train and certify testers in addition to guidelines for tester education.
Equally important for research, respondents indicated a strong need for improving the efficiency and effectiveness of the testing process. This demand ties closely with the need for specific forms of testing, with related requests for research in improved regression testing and more affordable automated and regression testing. A few respondents also cited a need for research in testing without a tester, specifically in agile environments.
While tools received less attention under this question than the software testing problems question, the request for better software testing tools and for assistance in analyzing and selecting tools for specific purposes was highlighted. Similarly, a few individuals referenced the need for better methods of estimation and improved communication between the test team and others.