Chapter 14

Organization Structures for Testing Teams

14.1 DIMENSIONS OF ORGANIZATION STRUCTURES

In this chapter, we will look at the various organization structures in typical testing organizations. (We discuss only the organization structures directly pertaining to development and testing. Other parts of an organization such as finance, administration, and so on have not been covered.)

Organization structures directly relate to some of the people issues discussed in the previous chapter. In addition, the study of organization structures is important from the point of view of effectiveness because an appropriately designed organization structure can provide accountability to results. This accountability can promote better teamwork among the different constituents and create in better focus in the work. In addition, organization structures provide a road map for the team members to envision their career paths.

We have discussed organization structures based on two dimensions. The first dimension is organization type and the second dimension is on geographic distribution.

We have broadly classified types of organizations into two categories— product organizations and services organizations. Product organizations produce software products and have a “womb to tomb” (or conception through design, development, testing and maintenance to product obsolescence) responsibility for the entire product. Testing happens to be one of the phases or groups that are within the organization. The product organizations that we will discuss in this chapter are multi-product organizations. Service organizations do not have complete product responsibility. In the testing context, they are external organizations that provide testing services to other organizations that require them. In essence, testing services are outsourced to such organizations. Such testing services organizations provide specialized and committed personnel for testing. They also undertake specialized and niche areas of examining performance testing, internationalization testing, and so on.

A second factor that plays a significant role in deciding organization structures is the geographic distribution of teams. Product or service organizations that are involved in testing can be either single-site or multi-site. In a single-site team, all members are located at one place while in a multi-site team, the team is scattered across multiple locations. Multi-site teams introduce cultural and other factors that influence organization structures.

14.2 STRUCTURES IN SINGLE-PRODUCT COMPANIES

Product companies in general have a high-level organization structure similar to the one shown in Figure 14.1.

 

 

The CTO's office sets the high-level technology directions for the company. A business unit is in charge of each product that the company produces. (Sometimes the business unit may also handle related products to form a product line.) A product business unit is organized into a product management group and a product delivery group. The product management group has the responsibility of merging the CTO's directions with specific market needs to come out with a product road map. The product delivery group is responsible for delivering the product and handles both the development and testing functions. We use the term “project manager” to denote this head. Sometimes the term “development manager” or “delivery manager” is also used.

Figure 14.1 shows a typical multi-product organization. The internal organization of the delivery teams varies with different scenarios for single-and multi-product companies, as we will discuss below.

14.2.1 Testing Team Structures for Single-Product Companies

Most product companies start with a single product. During the initial stages of evolution, the organization does not work with many formalized processes. The product delivery team members distribute their time among multiple tasks and often wear multiple hats. All the engineers report into the project manager who is in charge of the entire project, with very little distinction between testing function and development functions. Thus, there is only a very thin line separating the “development team and “testing team.”

The model in Figure 14.2 is applicable in situations where the product is in the early stages of evolution. We have indicated in Figure 14.2 that a “project manager” handles part or all of a product. We have consciously avoided showing multiple layers of management for the development and testing functions. This is because, in a small organization in the early stages of development, there is very little management hierarchy and people playing the roles of “managers,” “leads” and so on actually are also “engineers” who are expected to act as individual contributors as well.

 

 

This model offers some advantages that are well suited to small organizations.

Exploits the rear-loading nature of testing activities    Even though testing activities are distributed throughout the project life cycle, the major concentration and pressure points for testing arise during the later part of the project life cycle. Thus, this organization structure provides an automatic load-balancing scheme wherein during the early part of the project, everyone chips in for development and during the later part, they wear a different hat and switch over to testing.

Enables engineers to gain experience in all aspects of life cycle    Since an engineer divides his or her time between development and testing, he or she can get to know what is involved in all the life cycle activities. This also makes them appreciate the importance of and difficulties in all the activities and prepares them for a better teamwork when the functions are broken down among different groups.

Is amenable to the fact that the organization mostly only has informal processes    An organization in its early days usually has only informal processes and there may not be well-defined entry and exit criteria for products to satisfy to move from one phase to another, say from development to testing. Thus, multiplexing the same resource between development and testing is in line with this informal nature of the organization.

Some defects may be detected early    Since the developers perform the testing functions, there is a possibility that they may catch the defects closer to the point of injection (much like what white box testing can do).

This model, however, has some serious disadvantages that cause an organization to move away quickly from it. These disadvantages are as follows.

Accountability for testing and quality reduces    Since the same persons perform both the development and testing functions, providing accountability for testing and quality becomes a challenge. Should a developer (or project manager) give importance to delivery dates or should he or she “steal cycles” from development to do testing? The temptation is to spend time on development and add some new functionality, even at the expense of testing thoroughly the existing features. Thus, accountability for testing suffers.

Developers do not in general like testing and hence the effectiveness of testing suffers    As we saw in the Chapter 13, People Issues in Testing, developers in general do not like to perform testing activities. Thus, in this model, they may take up the tasks of testing as a “price” they have to pay to do the “cool job” of development. This lack of intrinsic motivation for performing the testing function can have a telling effect on the effectiveness of testing.

Schedule pressures generally compromise testing    When push comes to shove, deadlines rule the roost! There are always lots of deadlines to meet and there may not be sufficient time for testing. This, coupled with a typical engineer's preference for development tasks than testing tasks, compromises the quality of testing.

Developers may not be able carry out the different types of tests    As we have seen in the earlier chapters there are a number of different types of tests. A developer may not be equipped to carry out all the tests, because such testing may require specialized infrastructure or skills. For example, performance testing may not be an area where a general product developer can perform effectively because he or she would have to be well versed in ascertaining typical workloads, using special tools, and so on. Thus, it would become necessary to separate at least these specialized testing functions into a different group.

As the product matures and the processes evolve, a homogeneous single-product organization doing both development and testing, splits into two distinct groups, one for development and one for testing. These two teams are considered as peer teams and both report to the project manager in charge of the entire product. In this model, some of the disadvantages of the previous model are done away with.

 

 

1. There is clear accountability for testing and development. The results and the expectations from the two teams can be more clearly set and demarcated.
2. Testing provides an external perspective. Since the testing and development teams are logically separated, there is not likely to be as much bias as in the previous case for the testers to prove that the product works. This external perspective can lead to uncovering more defects in the product.
3. Takes into account the different skill sets required for testing. As we have seen in the earlier chapters, the skill sets required for testing functions are quite different from that required for development functions. This model recognizes the difference in skill sets and proactively address the same.

There are certain precautions that must be taken to make this model effective. First, the project manager should not buckle under pressure and ignore the findings and recommendations of the testing team by releasing a product that fails the test criteria. Second, the project manager must ensure that the development and testing teams do not view each other as adversaries. This will erode the teamwork between the teams and ultimately affect the timeliness and quality of the product. Third, the testing team must participate in the project decision making and scheduling right from the start so that they do not come in at the “crunch time” of the project and face unrealistic schedules or expectations.

14.2.2 Component-Wise Testing Teams

Even if a company produces only one product, the product is made up of a number of components that fit together as a whole. In order to provide better accountability, each component may be developed and tested by separate teams and all the components integrated by a single integration test team reporting to the project manager. The structure of each of the component teams can be either a coalesced development-testing team (as in the first model above) or a team with distinct responsibilities for testing and development. This is because not all components are of the same complexity, not all components are at the same level of maturity. Hence, an informal mix-and-match of the different organization structures for the different components, with a central authority to ensure overall quality will be more effective. Figure 14.4 depicts this model.

 

14.3 STRUCTURES FOR MULTI-PRODUCT COMPANIES

When a company becomes successful as a single-product company, it may decide to diversify into other products. In such a case, each of the products is considered as a separate business unit, responsible for all activities of a product. In addition, as before, there will be common roles like the CTO.

The organization of test teams in multi-product companies is dictated largely by the following factors.

How tightly coupled the products are in terms of technology    If different products exploit similar technology, perhaps there may also be a synergy of testing of these similar products. For example, if an organization is making compilers for different languages, it may be possible to spread the testing resources across the multiple compilers. However, if an organization has a wide coverage of product types—ranging from system software products like operating systems or database to application software products like a payroll system—then it is unlikely that the testing resources can be shared across the products as the skill sets required for the two functions are different.

Dependence among various products    If the dependence among the products is high, then testing of the products would also be necessarily tightly coupled. A change in one product may have a significant impact on the functioning of another product. Thus, it may be necessary to have some overlap among the testing teams of the various products. At least, there may be a need for an organization-wide testing team that performs integration testing of the various dependent products.

How synchronous are the release cycles of products    If the release cycles of the various products are synchronous, it may mean two things. One, there is likely to be some dependency amongst the products. Two, the resources for testing are likely to be demanded at the same time. These appear like two conflicting demands—on the one hand, there needs to be commonality in the testing team because of dependencies (as discussed in the previous paragraph); on the other hand, because the resources are needed for different products at the same time, the resources may have be part of distinct teams with different reporting structures.

Customer base for each product and similarity among customer bases for various products    Since the type of testing and skill sets required for testing depend largely on the nature of the product which in turn is dictated by the customer base, the nature of the latter may have an influence on whether the teams for the different products should be same or different.

Based on the above factors, there are several options available for organizing testing teams for a multi-product company.

1. A central “test think-tank/brain trust” team, which formulates the test strategy for the organization
2. One test team for all the products
3. Different test teams for each product (or related products)
4. Different test teams for different types of tests
5. A hybrid of all the above models

14.3.1 Testing Teams as Part of “CTO's Office"

In a number of situations, the participation of the testing teams comes later in the product life cycle while the design and development teams get to participate early. However, testability of a product is as important (if not more important) as its development. Hence, it makes sense to assign the same level of importance to testing as to development. One way to accomplish this is to have a testing team report directly to the CTO as a peer to the design and development teams. The advantages that this model brings to the table are as follows.

1. Developing a product architecture that is testable or suitable for testing. For example, the non-functional test requirements are better addressed during architecture and design; by associating the testing team with the CTO, there is a better chance that the product design will keep the testing requirements in mind.
2. Testing team will have better product and technology skills. These skills can be built upfront during the product life cycle. In fact, the testing team can even make valuable contributions to product and technology choices.
3. The testing team can get a clear understating of what design and architecture are built for and plan their tests accordingly
4. The technical road map for product development and test suite development will be in better sync.
5. In the case of a multi-product company, the CTO's team can leverage and optimize the experiences of testing across the various product organizations/business units in the company.
6. The CTO's team can evolve a consistent, cost-effective strategy for test automation.
7. As the architecture and testing responsibilities are with the same person, that is the CTO, the end-to-end objectives of architecture such as performance, load conditions, availability requirements, and so on can be met without any ambiguity and planned upfront.

In this model, the CTO handles only the architecture and test teams. The actual development team working on the product code can report to a different person, who has operational responsibilities for the code. This ensures independence to the testing team.

This group reporting to the CTO addresses issues that have organization-wide ramifications and need proactive planning. A reason for making them report to the CTO is that this team is likely to be cross-divisional, and cross-functional. This reporting structure increases the credibility and authority of the team. Thus, their decisions are likely to be accepted with fewer questions by rest the of the organization, without much of a “this decision does not apply to my product as it was decided by someone else” kind of objections.

This structure also addresses career path issues of some of the top test engineers. Oftentimes, people perceive a plateau in the testing profession and harbor a misconception that in order to move ahead in their career, they have to go into development. This model, wherein a testing role reports to the CTO and has high visibility, will motivate them to have a good target to aim for.

In order that such a team reporting to the CTO be effective,

1. It should be small in number;
2. It should be a team of equals or at most very few hierarchies;
3. It should have organization-wide representation;
4. It should have decision-making and enforcing authority and not just be a recommending committee; and
5. It should be involved in periodic reviews to ensure that the operations are in line with the strategy.

14.3.2 Single Test Team for All Products

It may be possible to carry out the single-testing-team company model of a single-product company into a multi-product company. Earlier in this section, we discussed some criteria of how to organize testing teams. Based on those criteria, a single testing team for all the products would be possible when the line between the products is somewhat thin.

This model is similar to the case of a single-product team divided into multiple components and each of the components being developed by an independent team. The one major difference between the two is that in the earlier model, the project manager to whom the testing teams reports has direct delivery responsibilities whereas in the case of a multi-product company, since different groups/individuals have delivery responsibilities for different products, the single testing team must necessarily report to a different level. There are two possibilities.

1. The single testing team can form a “testing business unit” and report into this unit. This is similar to the “testing services” model to be discussed in the next section.
2. The testing team can be made to report to the “CTO think-tank” discussed earlier. This may make the implementation of standards and procedures somewhat easier but may dilute the function of the CTO think-tank to be less strategic and more operational.

14.3.3 Testing Teams Organized by Product

In a multi-product company, when the products are fairly independent of one another, having a single testing team may not be very natural. Accountability, decision making, and scheduling may all become issues with the single testing team. The most natural and effective way to organize the teams is to assign complete responsibility of all aspects of a product to the corresponding business unit and let the business unit head figure out how to organize the testing and development teams. This is very similar to the multi-component testing teams model.

Depending on the level of integration required among the products, there may be need for a central integration testing team. This team handles all the issues pertaining to the integration of the multiple products. Such an integration team should be cross-product and hence ideally report into the CTO think-tank.

14.3.4 Separate Testing Teams for Different Phases of Testing

So far, we have viewed “testing” as a single, homogeneous activity. In reality, however, it is not so.

• We have seen that there are different types of testing that need to be done—such as black box testing, system testing, performance testing, integration testing, internationalization testing, and so on.
• We have also seen that the skill sets required for performing each of these different test types are quite different from each other. For example, for white box testing, an intimate knowledge of the program code and programming language are needed. For black box testing, knowledge of external functionality is needed.
• Each of these different types of tests may be carried out at different points in time. For example, within internationalization testing, certain activities (such as enabling testing) are carried out early in the cycle and fake language testing is done before the product is localized.

As a result of these factors, it is common to split the testing function into different types and phases of testing. Since the nature of the different types of tests are different and because the people who can ascertain or be directly concerned with the specific types of tests are different, the people performing the different types of tests may end up reporting into different groups. A possible way of organizing the people performing the different types of testing is shown in Table 14.1.

 

Table 14.1 Organizing people to perform different types of testing.

Type of test	Reports into	Rationale
White box testing	Development team	White box testing is inherently close to code, the developers develop and (should) run the tests themselves.
Black box testing	Testing team	This is the first level of “external testing” that a product gets and hence ideally fits in the scope of the testing team, as described above.
Integration testing	Organization-wide testing team	Integration testing involves putting together multiple components or multiple products. Thus, this should ideally belong to an organization-wide testing team.
System testing	Product management orProduct marketing	System testing involves testing in real-life scenarios and hence—like acceptance testing —can be part of the product management or product marketing.
Performance testing	A central benchmarking group	Performance testing (as discussed in Chapter 7) is very specialized. This may also be related to benchmarking and industry standards thereof. In case of a multi-product company, there may also be inter-product dependencies that affect performance.
Acceptance testing	Product management orProduct marketing	Acceptance testing is actually a proxy for customer acceptance and hence is rightly carried out by product management or product marketing.
Internationalization testing	Internationalization team and some local teams	Internationalization involves testing at various levels. Some of the levels (as discussed in Chapter 9) involve even knowledge of local languages and conventions. Thus the responsibility of this may be distributed.
Regression testing	All test teams	Some of the regression tests are also run as a part of the smoke tests—these parts may continue to report into the product testing teams and the organization must establish processes to transfer these learnings across the groups.

Such an organization based on the testing types presents several advantages.

1. People with appropriate skill sets are used to perform a given type of test.
2. Defects can get detected better and closer to the point of injection.
3. This organization is in line with the V model and hence can lead to effective distribution of test resources.

The challenge to watch out for is that the test responsibilities are now distributed and hence it may seem that there is no single point of accountability for testing. The key to address this challenge is to define objectively the metrics for each of the phases or groups and track them to completion.

14.3.5 Hybrid Models

The above models are not mutually exclusive or disjoint models. In practice, a combination of all these models are used and the models chosen change from time to time, depending on the needs of the project. For example, during the crunch time of a project, when a product is near delivery, a multi-component team may act like a single-component team. During debugging situations, when a problem to do with the integration of multiple products comes up, the different product teams may work as a single team and report to the CTO/CEO for the duration of that debugging situation. We would like to view the various organization structures presented above as simply building blocks that can be put together in various permutations and combinations, depending on the need of the situation. The main aim of such hybrid organization structures should be effectiveness without losing sight of accountability.

14.4 EFFECTS OF GLOBALIZATION AND GEOGRAPHICALLY DISTRIBUTED TEAMS ON PRODUCT TESTING

14.4.1 Business Impact of Globalization

Globalization has revolutionized the way we produce and maintain software products. As we discussed earlier in the book.

1. Markets for software products are becoming global. Hence, a global distribution of production of software becomes necessary to exploit the knowledge of the local conditions.
2. Since the markets are global, the needs that a product must satisfy are increasing exponentially. Hence, it is impossible to meet all the demands from resources from just one location.
3. Several countries around the globe have a rich supply of talented people and this needs to be utilized effectively.
4. Some countries offer not only a rich supply of talent pool but also offer cost advantages that makes them a compelling business proposition.

It is increasingly common to find software product teams distributed in multiple countries, working together, developing testing and delivering a single product. Testing in particular tends to be a major beneficiary of globalization and helps maximize the potential of globalization. Several factors contribute to this.

1. In mature organizations, testing is fairly well defined as a process and hence can be handed over to a team anywhere provided they are well trained on the process and are equipped with the necessary technology infrastructure.
2. When test automation is not involved, testing is a manual process and hence it may not be possible to find people in all countries who can or will do the job.
3. When test automation is involved, automation development is usually an activity that has to go on in parallel with testing of the products to be released. This offers a scope for parallelism, which in turn enables use of resources in multiple locations.
4. Testing of older releases of a product is an activity that is perceived as low risk but essential for ensuring the sustenance of existing customers. This makes it possible to choose a low-cost location to perform such activities.

There are essentially two organization structures for distributing testing teams across multiple locations—Round the Clock Development/Testing Teams Model and Testing Competency Center Model.

14.4.2 Round the Clock Development/Testing Model

The basic premises that drive this model are as follows.

1. Development and testing happen alternately and iteratively, that is a piece of code is developed, tested, and defects removed. This cycle continues until the testing is deemed complete. Since development and testing alternate, they can be viewed as two separate work units.
2. Since development and testing alternate, it may be possible to perform development during one part of the day and testing during the other part.
3. The geographic time difference between two different locations can be exploited to “stretch” the day so that effectively work can be carried on during the night of one location, which is actually day for another location.

Consider an example of a product team split across California and India. Because of the time difference between India and California, effectively, when it is day time in California, it is night time in India and vice versa. If, say, we distribute the work between the two teams with development being done in California and testing being done in India, we will find that we are able to get a 24-hour coverage of work across the two teams, without either of the teams having to work continuously for 24-hours at a stretch. In this situation, a typical work flow could be as shown in Figure 14.5.

1. The developers in California work from 8 am to 6 pm their time. The corresponding times in India are 8:30 pm to 6:30 am (next day).
2. Before they sign off for the day, the development team in California check in the latest build into a configuration management repository.
3. The testers in India come to work at 8 am (which is 7:30 pm the previous day in California). They pick up the latest build from the configuration management repository, run whatever tests are required to be run (and possible to run), report the problems encountered into a defect repository. When the testers in India sign off for the day at 6 pm, it is 5:30 am in California. The developers are now ready for their days work to continue to fix the reported defects in the next build and go to the next cycle for the next day.

 

 

In the above model, we have actually stretched the day to be almost completely utilized—we have a gap of about three hours in a 24-hour window where no activity is taking place. This model can be extended to have multiple testing teams in multiple locations to utilize these hours also. However, one has to be careful that the communication overheads do not overwhelm the utilization of time.

This model is very natural and effective when

1. Testing as a process is well established and there is no need for developer intervention to make a test run successfully.
2. There is a communication channel available between the two teams during any part of the day or night. This is required so that when one team needs to get clarifications from the other team, they do not end up playing mail tags, wasting time.

Typically, this model is used when the product attains a reasonable level of maturity and stability. During the early stages of development of a product, the code is likely to be unstable and hence testing will very frequently encounter “show stopper” defects. That is, when the testing team starts testing, they may encounter a defect very early in the cycle that will prevent further testing. For example, if you are testing a network layer product and the product fails even while trying to establish a connection, then all further tests like error control, flow control, and so on cannot be executed until the connection defect is fixed. In such a case, the developers will have to get involved immediately—that is, in their night time—thus negating the whole idea of exploiting geographic time difference.

A typical workaround to get over this problem is to adopt a “shift” system wherein the teams in both locations have 24-hour representation. Even with this, the communication overheads arising out of an unstable product make this model unviable early in the product cycle.

14.4.3 Testing Competency Center Model

In this model, a logical organization called the Testing Competency Center is created. This organization could potentially span multiple locations to effectively exploit time difference and/or available skill sets. This Center is highly specialized in testing. It has the final say in the quality of the product and release readiness. The Testing Competency Center model has two variants.

In the first variant, the Center is a shared (and often scarce) resource that is shared across multiple product development groups. As mentioned before, this center has testing competency that is not available elsewhere. Hence, there is a demand for the services of this group. Product development groups “book” the time of this group in advance. With the high skill level of the group and the highly matured processes that this group has, the recommendations of this group are highly regarded by the management. This group could be internal to an organization or can be an external organization of high standing in the testing arena. One can regard this group or Center as a “certifying authority” whose “blessing” is needed before a product can be released. In other words, certification by this Center adds credibility to the product. This model is not to gain cost savings. In fact, it may even be expensive because of the specialized nature of the group. This model is not to exploit the time zone advantages. The placement of the group in any location or locations is determined purely by the availability of talent pool. This model can apply to product as well as service companies.

Another variant of the Testing Competency Center is to have a set of dedicated testing resources that gets involved in the full cycle of testing activities, right from the early stages of the project. This typically follows the process and structure given below.

1. The members of this Center get involved in planning for testing activities right from the early stages of product conception. Any activity that involves testing of a product always entails participation from members of this Center. Note that the physical location of the Center itself is immaterial in this variant.
2. When product functionality and release are being planned, the product team and the testing team from this Center identify and prioritize the product features to be tested and the location where each feature would be tested. This process ensures that the most important features get a higher priority for testing and that the right resources at the right location get involved in testing the right features at the right time.
3. Some of the factors for selecting features to be tested and allocating them to the various locations of the Center are
   1. Past experience in using similar features;
   2. Skill sets for performing the functions, including using the required test automation tools; and
   3. Availability of hardware and software resources (including local availability of support for these resources).
4. The testing team members from the Center spend time with the development team (and, if necessary, the product management team), early in the cycle, understanding the details of the product features allocated for testing. During this time, they get trained on the external and where necessary, internal aspects of the product. This forms the basis for the test planning and test case design.
5. The test case design thus generated gets translated to test cases by a larger team at the Center location(s). They also execute the tests to verify that the tests are tested (true to the dictum we saw in Chapter 1 to “test the tests first”). This creates the first baseline for the test cases.
6. If the product is fairly stable, then the test teams at the Center location(s) can also undertake the repeated testing of the code, as it evolves. If not, there may be a small team co-located with development team that will do the repeated execution in the early stages.
7. The Center in parallel undertakes the automation of the test cases and, over time, the running of the automated tests is done as a part of the build. Since this is automated, the exact location of the server is immaterial.

This second variant is most applicable when a product is undergoing version changes, which are usually upgrades with new features added/enhanced. In this variant, since the center is focused on developing testing as a competency, it is somewhat similar to “testing as a service” model, discussed later in this chapter.

14.4.4 Challenges in Global Teams

Cultural challenges    Whenever a global team works together, culture and language pose major barriers for communication. Without effective communication, the divide between development and testing will only widen. When a testing team is set up in a new location, it is important for the teams on both the locations to get a mutual appreciation of the culture, language accents, and so on. Some of the ways to achieve this are given below.

1. Planning periodic travel by the members from one location to another: Testing offers natural means to plan such travels — such as acceptance testing, integration testing of modules developed in multiple locations, training on new products to be tested. From a long-term team building perspective, it is important to plan such travel into the project and ensure that before and when an individual travels, he or she gets exposed to the cultural and language aspects of the other location.
2. Periodic video and audio conference calls.

Work allocation challenges    When teams are distributed, it is important to have clear work allocation policies that are articulated upfront and understood consistently by all the teams. There should not be a perception that all the “cool work” goes to one location while all the “not-so-cool work” goes to another location. The entire team must take pride in what they do. Only then can they perform effectively and develop themselves and make the product successful.

Parity across teams    When teams across different geographies work on similar tasks, they expect parity in the way they are treated and in their compensation and reward systems. “Parity” can be a debatable term. Firstly, the teams in different locations would like to have parity in job content. Next, the teams would like parity in ownership, authority and delegation, freedom, and so on. Last, they would also expect reasonable parity in terms of compensation and reward systems. A person working in USA may get a salary in terms of dollars whereas a person working in India may be getting a salary in Indian Rupees, which would not be the same in absolute value as what his or her US counterpart gets. Given the difference in purchasing levels and the economies of the two countries, this difference is natural and reasonable. However, the values that the employee carries may be different and this may even motivate him or her to work in a country with better monetary compensation (even though it may not mean anything). The solution for this is not to offer US salaries to people in India! Some of the ways to address this challenge are

1. Providing challenging job content;
2. Increasing career opportunities in testing, with models like competency centers;
3. Providing long-term retention incentives like stock options; and
4. Providing travel opportunities.

Ability to track effectively    When teams are distributed, tracking their progress becomes more difficult. Opportunities for informal tracking such as hallway chats are completely lost. Hence, the tracking mechanism has to be more robust, less person dependent. Some of the ways to increase the effectiveness of tracking include

1. Having planned communication exchanges such as weekly conference calls;
2. Having a clear allocation of tasks;
3. Having a commonly understood and well-enforced reporting mechanism; and
4. Automating as much of the tracking function as possible.

Dependence on communication infrastructure    Communication infrastructure through the Internet and Intranet form the lifeline of geographically distributed teams. This is especially true of the “round the clock” model of testing wherein the testers pick up the builds on a daily basis. Such a model mandates the availability of a high bandwidth and highly reliability communication infrastructure. While the communication technology has become fairly robust and this may not be a major issue, it is important to have alternative plans in case communication links snap for some reason. One way to mitigate this risk is to have a backup/mirror site at every location. This may be an expensive proposition. Another way could be to have a small “on site” team in different locations who can proxy for the “offsite” teams if the need arises.

Time difference    In the various models of distribution of work and organizing teams, time difference is normally an advantage. However, time difference can also present some challenges. For example, in the round the clock model of testing, when a tester in India, say, encounters a show stopper, he or she may have to call a developer during the developer's night time (assuming that the developer is in USA). If the developer is not available, that workday for the tester in India may be lost completely. In addition, when the developer fixes the problem, the tester should be available or another twelve hours will be lost! Thus, time difference can end up eating up 36 hours for solving a problem that would get fixed within minutes in a single-site team. The key to making effective use of time difference is to have sound processes and proactive quality assurance (so that such show stoppers can be minimized) and also make people work in shifts in various locations so that there is someone to contact at any point of time and in any location.

14.5 TESTING SERVICES ORGANIZATIONS

14.5.1 Business Need for Testing Services

Most of the discussions we have had so far on testing functions have implicitly assumed testing activities as part of the same organization that is responsible for producing, marketing and supporting the product. However, today it is common to find testing activities outsourced to external companies who specialize is testing and provide testing services. There are several business drivers for this model of testing as a service.

1. As we have seen throughout this book, testing is becoming increasingly diverse and a very specialized function.
2. The variety and complexity of the test automation tools further increase the challenge in testing. A specialized testing service organization may be able to effectively fulfill such a niche need.
3. Testing as a process is becoming better defined. This makes testing (at least some phases and types of testing) more amenable to outsourcing.
4. A product development organization which has expertise in understanding the domain of software may not necessarily have expertise in setting up and running an effective testing function (especially if it involves the use of complex automation tools).
5. An outsourced organization can offer location advantages. As we saw in the multi-site teams, testing teams that exist in different time zones can provide a better round-the-clock coverage.
6. An outsourced organization can provide cost advantages. Certain locations can be more economical than others, thus, the overall costs can be reduced. This rationale applies to outsourcing any function (including development and testing).

Thus, testing as a service has gained significant acceptance in the industry. Coupled with the time zone and cost advantages, testing services are normally outsourced to geographically distributed teams.

14.5.2 Differences between Testing as a Service and Product-Testing Organizations

Organization structures and the associated people issues in a testing services organization are driven by certain fundamental factors.

Testing service organizations are generally held at arm's length from product developers    In a typical product development organization (where testing is one of the functions), there is a much closer binding between the development and testing teams, as people generally move from one group to another. Furthermore, there are no Intellectual Property Rights (IPR) issues in the testing team getting exposed to the product code. When testing is outsourced as a service to an external organization, the members of the outsourced organization may not have the same level of rapport with product developers as testers internal to the product organization. This is especially the case if the outsourced team works in a different location, wherein opportunities for rapport building are reduced. The IPR issues of product code ownership also play a role in distancing the testing team from the rest of the product organization.

Testing service organizations may not always provide full breadth of coverage for testing types    Most testing service organizations provide services like black box testing, domain testing, and certain specialized testing such as performance testing or test automation. White box testing is not one of the commonly outsourced types of tests. Other than the IPR issues mentioned in the previous paragraph, the sheer logistics of co-ordination between the development and testing teams, need for a shared configuration management system that should be kept current all the time, and in-depth knowledge of the code required make outsourcing of white box testing to a testing services organization somewhat difficult.

Testing service organizations tend to be more homogeneous in terms of job titles and roles than testing teams in product organizations    In a typical product organization, there is a variety of job functions — development, maintenance, testing, support, and so on. Since a testing services company focuses only on testing, the role definitions and job titles tend to be more homogeneous. This could be a double-edged sword. People who look for diversity in job experience (covering all aspects of a software life cycle) may find working in a testing services organization uninteresting in the long run. On the other hand, given the misconceptions of testing as an “uninteresting” job function if a testing services organization can staff its team with people who are passionate about testing, it may easier to offer a progressive career path to the employees. Table 14.2 summarizes these and other key differences between product testing and testing as a services.

 

Table 14.2 Differences between product testing and testing as a service.

Attribute	Product testing	Testing as a service
Type of testing	Will have to address all types of testing as the entire product responsibility lies within the organization	An organization may be specializing in specific types of testing, for example, performance testing or internationalization testing
Exposure to code	There is likely to be a greater exposure to the product code for the product testing team. This is especially true for white box testing teams.	Generally, testing service organizations focus more on external functionality and hence work off some “product drops” on a SCM repository. In addition, the visibility to source code may be minimal or non-existent because of IPR or contractual reasons.
Accessibility to developers	Likely to be available as they are part of the same organization	Likely to be minimal
Career path issues	Since the organization has multiple roles and work functions, this may lead to people in the testing profession wanting to move on to other functions (for example, development) within the organization. Hence may be more difficult to provide testing-specific career paths.	Since the entire organization is fully focused on testing functions, it may be easier to provide career paths in testing for the employees. The expectations can be set and managed better.
Diversity of tools	It is likely that an entire organization (over time) will standardize on the same tools. Hence, there is usually no need for investing on acquisition and training on multiple tools.	A testing service organization may have to work with multiple clients who have different automation tools and hence may be forced to invest/train people on multiple tools.
Domain specialization	Since the product(s) is (are) usually specialization focused on a specific domain, it may be more feasible to hire and develop domain expertise within the team.	Since a testing service organization may have to work with different customers who are in different domains, the organization may either have to hire people with different domain expertise or retrain people across multiple domains.

14.5.3 Typical Roles and Responsibilities of Testing Services Organization

A testing services organization is made up of a number of accounts, with each account being responsible for a major customer. Each account is assigned an account manager, who is the single point of contact from the customer into the testing services organization. The account manager serves the following functions.

1. Is the single point of contact between the customer and the testing services organization for all matters and is also the point of escalation.
2. Develops rapport with the customer and is responsible for ensuring that current projects are delivered as promised and for getting new (repeat) business from the customer.
3. Participates in all strategic (and tactical, as needed) communication between the customer and the testing services organization.
4. Acts as a proxy for the customer within the testing services organization.

The account manager may be located close to the customer site or at the location of the testing services organization. To develop better rapport, this role would require frequent travel and face-to-face meetings with the customer. Usually this role has a single point of contact at the senior level from the customer organization to co-ordinate multiple projects from the account.

Since the testing services organization is a different entity from the customer, it is physically in a separate location. (Most often, these organizations are in countries with more competitive cost structures.)

The testing service team organizes its account team as a near-site team and a remote team. The near-site team is usually a small team. It is placed at or near the customer location. This team serves the following functions.

1. Be a point of direct first point of contact for the customer for tactical or urgent issues. For example, when a quick system study is required for understanding what needs to be tested for an incremental version, the near-site team, because of its proximity to the customer, can do this.
2. Act as a stop-gap to represent the remote team, in the event of emergencies like travel embargo infrastructure unavailability etc.
3. It also serves to increase the rapport between the customer's operational team and the testing services team.

This team is usually small and does not necessarily have a manager-engineer hierarchy.

The remote team is located on the site of the testing services organization. This team is usually large in number the team and does the bulk of the work. The organization of this team depends on the size and complexity of the testing project. The remote team manager manages the entire remote team and can have a peer-to-peer relationship with the near-site team or have the near-site team reporting to him or her. Depending on the need, there can be further hierarchies within the organization as test leads and test engineers much similar to the roles and responsibilities that we discussed in the last chapter.

 

 

The near-site and remote teams usually are rotated around. For example, some of the remote team members can be posted as a part of the near-site team. For testing services organizations that are in countries different from the customer location, this offers travel opportunities—a common employee-retention strategy. In addition, periodically, a remote team member may have to visit the customer site for any onsite testing and similar activities for which the near-site team may not be sufficient.

14.5.4 Challenges and Issues in Testing Services Organizations

All testing organizations face certain common challenges. In the case of a testing services organization to which testing is outsourced, some of these challenges are exacerbated, primarily because of the arm's length distance from the development team. We will now explore in detail these challenges and the ways to mitigate these challenges

14.5.4.1 The outsider effect and estimation of resources

The testing services organization is an “outsider” to the development organization. Some of the implications are as follows.

1. They do not necessarily have access to the product internals or code as much as a product testing team within the product organization would have.
2. They do not have access to product history in terms of which modules have historically been problem prone and thus may not have all the information needed to plan and prioritize test cases.
3. They may not necessarily have the same level of rapport with the development teams that an internal testing team may have.
4. Internal development and testing teams do not necessarily need to have information about the hardware and software resources required, whereas testing services organization would also have to estimate and plan for hardware and software resources.

These challenges make it more difficult for a testing services organization to estimate the resources needed for performing testing functions. There are no easy answers for these but all these point to the need for more upfront planning and more thorough tracking for a services organization than what is required for a product organization.

14.5.4.2 Domain expertise

A testing team in a product organization can develop domain expertise in a specific domain. For example, a testing team in an ERP software organization can develop specialized expertise in the ERP domain. The organization can hire a few domain specialists (who need not necessarily have computer expertise) who can augment the testing team. Such specialists will find it interesting to join a product organization because they can find a natural transition from their domain to the (apparently) more attractive IT arena. For example, for someone specialized in accounting, this can be a very natural transition to enter an IT product company.

However, a testing services organization may have to undertake projects from multiple customers. Since there is little product ownership in a testing service organization, it may be tougher to get domain experts to join a testing services company. (However, this disadvantage is fast disappearing as the size of the test services companies increases rapidly and the domain specialists see career growth opportunities.) Furthermore, the diversity of domains exacerbates the problem. Given that most testing service organizations perform functional testing (black box testing, integration testing, acceptance testing, and so on), having domain experts can go a long way in improving test quality. Product companies offer tough competition for these resources.

14.5.4.3 Privacy and customer isolation issues

For being viable, a testing services organization will have to work with multiple customers. As an organization works with customers in a given domain (for example, pharmaceutical or financial services), it develops not only general expertise but also domain expertise in a particular domain. As an organization develops domain expertise, it gains competitive advantage in offering testing services to other companies in the same domain. Similarly, when an organization develops expertise in a specific test automation tool, then it is placed in a position of strength to take on test automation projects for any organization that uses the same tool. However, with the strong linking of IT with organizational strategy in most businesses, a clear line of demarcation would have to be drawn between any two customers.

Two factors contribute to this being a major challenge. First, the testing service organization has a common infrastructure and hence physical isolation of the different teams may be difficult. Second, people move from one project (customer) to another. When this movement takes place, there should be full confidence and transparency in ensuring that the customer-specific knowledge acquired in one project is not taken to other projects. At the same time, for continuous improvement of the testing services organization, the learnings will have to be captured and abstracted at the right level, without compromising the privacy and confidentiality of customer-specific information.

The organization structure we discussed in the previous section addresses the above issues. First, each customer can be treated as a separate entity by associating an account manager working with a distinct and dedicated team for the customer. During the time the project is initiated, appropriate Non Disclosure Agreements (NDA) is draw up between the customer and the testing services organization. Such NDAs provide a framework ensuring that confidential information is not compromised. In certain cases, the teams are located in physically separate locations. Second, customer data for specific customers should also be isolated and secured. This leads us to the next issue of apportioning hardware and software resources and costs.

14.5.4.4 Apportioning hardware and software resources and costs

When a testing service organization bids for and works with multiple customers, it would have to use internal hardware and software resources. Some of these resources can be identified directly and specifically for a particular account. There are other resources that get multiplexed across multiple projects. For example, communication infrastructure such as satellite links; common infrastructure such email servers; and physical infrastructure costs.

These costs have to be apportioned across different projects while costing the project. The challenge is that it is difficult to know how much of these resources should be apportioned to each project.

14.5.4.5 Maintaining a “Bench”

A testing services organization should always have people ready to be deployed in any new project that may suddenly come from customers. In addition, some prospects may require some initial studies to be done or some demonstration of initial capability before they sign up fully. These would again need technical resources. There may also be cases where the customer would like to look at the resumes of specific individuals and have a say in handpicking people from the services organization to participate in the testing project. In all these cases, there may be a need to maintain “people on the bench,” that is, people not allocated to any project but ready in the wings to take on new projects or to convince customers.

The bench resources are not billable and hence have to be subsidized by the projects. Further, the people on the bench usually get frustrated, develop insecurity, and are at a high risk of attrition. A part of the planning, looking at the business pipeline, is to estimate the bench size—that is, the number of people that can be on the bench at any point of time.

14.6 SUCCESS FACTORS FOR TESTING ORGANIZATIONS

Whether it is product testing organizations or testing services organizations, there are some common success factors between them. These are as follows.

Communication and teamwork    As have discussed in Chapter 1, Principles of Testing, and Chapter 13, People Issues in Testing, the testing teams and development teams should work in a co-operative spirit, without viewing each other as adversaries. However, since the testing team's job is to find defects in the development team's work, this spirit of camaraderie is not easy to achieve. Some of the ways that product companies can achieve this spirit of co-operation and teamwork is to have periodic job rotation. This may not always be possible for testing service organizations that remain at arm's length from the development team. There should also be frequent opportunities for communication between the teams and some of this should be in non-work related environments and scenarios. The problem of communication and teamwork becomes magnified when the team is geographically distributed, when there is little opportunity for face-to-face interactions.

Bringing in customer perspective    The testing team should act as a proxy for the customer. This requires getting to know the customer perspective. How does a testing team get this perspective? One way could be to bring in domain experts or customer representatives into the testing team. Another way could be to rotate people from customer support into testing. In a product organization, customer support is the point of contact for the customers to report problems. They are usually familiar with the kind of problems customers face, what hurts the customers most, and what would be acceptable to customers. Having them participate in designing test cases or in performing ad hoc testing (see Chapter 10) can increase customer perspective. Another approach is to rotate the testing team members and even the development team members into product support, so that they get a first hand idea of the issues customers face.

Providing appropriate tools and environment    To be effective, a test organization must have appropriate supporting tools. Two of the most important ones are the defect repository and configuration management tools. A defect repository provides a way to track defects to completion. As discussed earlier, configuration management tools provide a mechanism for the teams to share the software environment and make sure the entire team gets one consistent view of the product source and object code and other associated files. (We will go into the details of some of these tools in Chapter 15, Test Management.

Providing periodic skill upgrades    Testing teams will need periodic skill upgrades to keep them current. These skill upgrades include.

1. Domain skills, to upgrade to new domains or new features in future versions;
2. Technology skills, for example, use of new languages;
3. Automation skills, to get acclimatized to new automation tools; and
4. Soft skills, including spoken communication, written communication, negotiation skills, and conflict resolution skills to be able to communicate better.

The challenge is to find the time for these skill upgrades amid the tight schedules and deadlines of the team. Career planning of individuals should budget time for these skill upgrades. Or else, the effectiveness of testing teams will reduce over time.

As mentioned in the previous chapter, the topics of people issues and organizational structures are not discussed at great length in literature. The organizational and project models discussed in [RAME-2002] for any geographically distributed project can be adapted and applied to testing teams as well.

1. When a company moves from a single product to multiple products, how can it leverage its past experience?
2. Which type of work distribution between development and testing is suited for each of the following cases?
   1. A product that is using emerging technologies that require niche skill sets
   2. A product that is going through very short release cycles with frequent builds
   3. A product wherein work on multiple releases are going on at the same time
   4. A product to which incremental features are being added on top of already existing features that are stable
   5. A product being developed in an organization where there are clear cut processes and methods of hand over of work from development.
3. A group was initially established to provide testing services to internal “customers” (application developers) in a bank. Soon the group developed competence and spun off as an independent company providing testing services to other banks and financial institutions. Outline some of the organizational and business challenges that this new company should watch out for and how it can leverage its past experience.
4. In which of the cases in Problem 2, would a geographically distributed team be appropriate?
5. A company makes multiple products in the systems and applications arena. Discuss the pros and cons of having separate teams for testing versus having a common team for testing.
6. A product has a nine-month release cycle, followed by a ten-month maintenance period. When would you decide to set up geographically distributed testing teams? What activities would you distribute at what timeframe? Justify your answer.
7. Which of the following testing activities can be outsourced to a testing services organization? Why?
   1. Code coverage testing
   2. Performance testing
   3. Ad hoc testing
   4. Integrating several products from different companies
   5. Hardware/software integration
8. We discussed specific people issues in the previous chapter. Take each of the issues and see what are the additional challenges you need to face when we add the dimension of multiple teams.