CHAPTER TWENTY-TWO

Testing and Quality Assurance for Workforce Management Systems1

IN THIS CHAPTER, TESTING AND quality assurance are discussed as an important component of the workforce management (WFM) implementation project. These activities are crucial checks on the system to avoid problems at a later date. This chapter describes the different types of testing that work best for the type of system that is being implemented. Considerations for determining testing strategies and preparation, such as the appropriate data sets and functions, are described. The systems development life cycle is revisited in an outlined quality assurance process that can be scaled to meet a specific organization's needs. Planning for effective testing begins in the analysis phase and continues through development and implementation. Testing and quality assurance for WFM systems are processes that can help the Workforce Asset Management Professional (WAM-Pro) organize and prepare for a project.

22.1 TESTING AND QUALITY ASSURANCE ON WFM SYSTEMS BEFORE RELEASE TO OPERATIONS

Testing a WFM system being implemented in an organization is often a misunderstood and questioned effort. The misunderstandings and questions are usually about what testing is, why it is needed in such detail, how much is needed, who should do it, and how it is done. WFM system testing is comprised of not only technology validation and certification, but also process and outcome assessment to verify whether the system is robust enough to deliver the desired tactical and strategic objectives. It is a matter not only of Is it working technically? but also Does it work operationally?

In this chapter, the example of overtime as both tactical (related to computation and compliance) and strategic (related to cost reduction) will be used to explore the various aspects of testing.

The WAM-Pro's job is to avoid misunderstandings that can lead to a lack of, inadequate, or inappropriate testing that does not fully verify that the WFM is ready for release. The risks of implementing a system without thorough testing include: significant problems and shortcomings that impact the organization financially and operationally, and a loss of credibility and confidence in the system by the user and organization. The WAM-Pro uses testing as a way to guard against this by undergoing an adequate testing process that verifies the WFM system is ready for use.

The formal term for testing is quality assurance. It is a process comprised of sequential events that examines the new system in detail. It requires preparation, consistency, and focus for it to be completed correctly. It is also an insurance policy that the WFM system will work appropriately and as expected based on the requirements defined by the business during the requirements phase of the overall project.

The following material outlines the quality assurance process recommended for the implementation of a WFM system. The chapter contains the core information and primary concepts about quality assurance as well as specific examples. As with many of the recommended processes and methods, quality assurance is a scalable effort based on the size and complexity of the system and organization. It is also one of the more important efforts of the WFM implementation project.

(a) Different Types of WFM Testing Systems

One of the areas of confusion when testing WFM systems is related to the type of system that is being implemented. In addition to understanding the size and complexity of the WFM being installed, the WAM-Pro must also understand what type of overall system is being implemented. For simplicity the types of systems have been classified as internally developed, commercial, service provided, and matrix.

Internally developed WFM systems are those that have their requirements defined, designed, and programmed internally by the organization that will use the system. These systems are the more challenging to test due to the newness of the software application code. With the availability of many commercial WFM systems today, few organizations develop their own WFM system from ground up. However, many companies develop supplemental functions and add-on capabilities to commercial systems that require testing.

Commercially developed WFM systems are those available in the marketplace from vendors specializing in those types of systems. There is a wide variety of this type of system with an equally wide range of functions and capabilities. Many of these systems have been in the marketplace for a length of time that has allowed testing and use by customers to find major problems that might occur with these kinds of software. In general, the commercial systems are configurable, which means they have a set of switches to allow changes and customization in some of the functions. In some cases the commercial software will allow for integrated or customized programming that could change how the functions work. And, in other cases the commercial software will allow add-on programs that do not change the original software but do allow the supplemental software to work and function as part of the WFM system. The situation with the commercial software will dictate the type of testing that will need to be performed.

Service-provided WFM systems are those where another company provides the system functions being requested by a customer. The services are usually standard. However, many of the servicing companies do have configurations that can be changed per the customer needs and they often allow add-on software to complete the requirements if needed. In these situations the testing is usually split between the service company and the customers. In general the service provider's WFM systems have well-tested software due to long-term usage. However, the configurations and any supplemental programming will still need to be tested.

A matrix WFM system is one that has a mix of internally developed software applications, commercial software, and possibly service-provided systems that have been integrated to create a complete WFM system for an organization. The strategy for this situation is more complex and requires more testing than the single WFM system type.

While the strategies for testing any of the just described systems may vary in complexity and length, testing is required for each of them to validate that they work as expected.

(b) Understanding Testing and Quality Assurance

Many of the aspects of testing a system and the processes associated with quality assurance are misunderstood. This is due to the variety of business and organization structures and sizes that occur. When involved in a systems project it is helpful to understand the following.

Companies that develop their own software usually have a quality assurance (QA) department that tests software. The complexity and type of software will dictate the size and approaches used by a QA department to complete the testing. Companies that do not develop their own software generally do not have systems QA departments to test their software and systems.

When purchasing and implementing a system from a vendor, the customer still needs to have a testing process to determine that the system is working and configured correctly to meet the unique requirements of that organization. WFM system testing is distinctive from other software applications and business processes for a number of reasons. There are typically more end users, a greater need for controls to reduce financial impact, administrative as well as operational requirements of system performance and outcome, potential hidden practices, numerous hardware components, and a host of regulations, rules, and expectations that are difficult to simulate and test. If the organization does not understand how to organize the testing process, then it will be necessary and worth the expense to engage with a consultant who knows how to complete the checklist of testing requirements.

The WAM-Pro should make certain that testing support is stipulated in the sales contract or a separate contract may need to be negotiated to provide testing assistance from a consulting firm.

(c) Determining the Strategy and Preparation for Testing

The following areas define a strategy that will utilize the selected testing process, lay out the organization of the testing process, and describe what will be considered acceptable results.

• Testing strategies and program. The overall strategy and program that will be used for testing the WFM should be documented with the processes well defined and metrics used to determine pass or fail for what is being tested. The strategy will also select the types of testing that will be used during the testing and QA process.
• The triple constraint—time, cost, and quality. In project management there is a concept called the triple constraint that is based on how long a project effort, such as testing, will take; how much it will cost; and at what level the quality will be. The concept is used by understanding that the longer an effort takes, the higher the cost, but the better the quality of the final result. If the schedule and timing are short or reduced it may cost less and be delivered quicker; however, the quality of the final result will be significantly lower. The WAM-Pro uses the triple constraint concept to determine how long will be considered adequate to complete testing and reduce the risk of failure, how much cost will be allocated to do the testing, and at what quality level testing will be set.
• Testing cycles. The process of testing, redo, testing redo, and release can be complex and confusing unless managed carefully. The testing cycles should be organized in a manner where the work that is first passed to the testing group should be documented for flaws and acceptance. The items returned for further work should be returned using a controlled process so it will be worked on and then reentered into the testing process.
• Timing and schedule estimates. One of the high pressure points with a project and WFM implementation is the schedule and length of time that testing will take. This will be a combination of the testing strategy and the organizational expectation for delivery. When developing the schedule, the following need to be considered. Testing is a sequential process. It cannot be started until after programming or configuring has been completed. In most cases testing is cyclical where there is testing, fixing, and then retesting. The WAM-Pro carefully estimates the amount of time it will take to complete testing, making sure that the implementation release schedule does not constrain the testing process. There is a tendency for companies to squeeze and reduce testing. The WAM-Pro helps to resist this with an explanation that the more testing is reduced, the higher the risk of failure and extensive follow-up work required to repair what should have been allowed to be caught during the testing and QA cycle.
  WFM system testing should cover a minimum of two payroll cycles. Consider whether the testing time frame should include holidays or other variables to the normal time and attendance cycle. Testing schedules should also include rhythms of related systems so that anomalies in those systems do not adversely impact the testing schedule or outcome. Avoid periods where other systems are being changed.
• Scoping and executing a pilot/parallel test. During a project there can be significant unknowns. This is one of the core concepts of testing and QA. Until testing the WAM-Pro does not know how a WFM system will perform. Due to this the pilot and parallel approaches are scaled down versions of the total planned system designed to test development and setup, as if preparing for a full release to the user community. The pilot test is a small group that begins to use the system before it is deployed to the entire organization. A pilot group may convert to the new system or concurrently use the old and new systems in a parallel test model. Either way, the pilot and parallel tests models inspect and compare the new system or features to the existing system. This approach can be fairly minimal or expansive and expensive in time. However, it does allow detailed testing and review of the associated business and operational processes that the WFM system supports. Not every scenario requires or allows for pilot and parallel testing; however, for large WFM system implementations where there are significant changes to the company, business, and processes, this type of testing is extremely beneficial. When developing the plan for pilot or parallel testing, keep in mind that it is difficult to test everything because the scope may be too large to do testing in a feasible time frame. The scope of the pilot/parallel test can be reduced with the assumption that some of the overall WFM system will work as designed (and tested during unit testing) and does not require full parallel testing.
• Distinguish the different types of parallel testing and their benefits. Parallel testing implies that two systems are running concurrently where one is used to validate the results of the other. There are various types of parallel testing that can be performed. The selection will depend on the situation. Some parallel testing will require results to be very nearly the same between the systems being tested. In other cases the results will be assumed to vary or be so different that a meaningful comparison is not possible (as with new functionality) so the processing flow is the concern of the testing. Parallel testing can be designed to turn on the new system while relying on the old system until the new system is validated. Reverse parallel testing relies on the new system while using the old system to validate and help resolve problems. Some situations call for a parallel and then a reverse parallel to wean the organization off the old system.
• Identify what to test and how. With certain testing and QA processes not everything will be tested. What will need to be tested, however, are functions and results that may have been changed, are new, or may be impacted by changes elsewhere in the program. When the testing plan is being created it should use the information developed during the project analysis (the business requirements and system mission) to determine which business, functional, or technical areas will be impacted and require testing.
  Some of the common and general things to test include pay rules, pay codes, the accounting structure, user rule sets (profiles), timecard functions, schedule templates, automated workflow, time clock functions, security settings, on-screen workspaces, canned and custom reports, query and navigational tools, interfaces such as accrual imports and payroll exports, new hire information, and changes to an employee's status. Testing plans go much deeper than this list, however, and should be specific such that the item and results to be tested are unambiguous and measurable.
• Identifying the appropriate standards, metrics, and expected results. When developing the strategy and testing plan, the appropriate standards, metrics, and expected results need to be set so it can be determined if a WFM system passes or fails in any of the testing areas. These would include aspects such as the acceptable variances from the legacy system during pay rule testing (e.g., will the new pay rules compute time differently such that a shift span might show 3.5 hours instead of 3.25 and still be accurate?), the amount of time for processes to complete, the number of records tested per item, and so on.
• Evaluate testing outcomes. Evaluating test results will ultimately lead to decisions as to whether the test passed or failed and what to do if the results are unexpected or borderline. The preferred way to prepare for this is to have the standards, metrics, and expected results defined before testing in a manner that will make the decision for the testers as to whether the testing was effective or not. The WAM-Pro provides clarity as to what the testing outcomes should be and the strategy for resolution. For instance, if certain test areas are borderline passing, what should the organization do? Or, if some areas pass but there is a borderline area, what should be done? Examples: 50 pay rules pass except for one rule, which needs rework. Or the payroll results were 90 percent of the legacy system during the pilot.

(d) Different Types of Testing

For WFM systems there are several testing phases. New or upgraded components go through unit testing, integration testing, system and stress testing, and user acceptance testing. Sometimes regression testing is done as well.

i. Unit Testing

Unit testing is performed on each specific component of system design. For timekeeping systems an isolated pay rule is tested to validate that the architecture of the many parts of the rule work under a set of conditions. Within a pay rule a meal rule would also be tested to verify that the parameters of proper meal rule computation are working. Unit testing is often done by the person who builds the component to make certain their design is workable.

An example of unit testing, daily overtime rules apply if an employer is subject to a state-based daily overtime regulation for hourly workers. The employer must pay time and half for all hours over 10 per day. These hours are paid in the Daily OT pay code. The hours should not count toward the weekly overtime calculation (over 40 per week).

ii. System and Stress Testing

System and stress testing are more comprehensive evaluations of how the system is working as a whole. WFM systems operate using combinations of features and functions and there is considerable interconnectedness between them. Schedules and timecards share and depend on the proper configuration. Users operate within the application on a number of levels and rely on the proper assignment of user rights and accessible data and functions. One kink in the underlying setup can make an important workspace or function unworkable. WFM systems also undergo cyclical periods of heavy use throughout the days and months. Payroll cycle deadlines increase the activity that WFM systems handle as well as shift schedules that result in peak times of employee inputs. Stress, or “load,” testing helps the WAM-Pro validate that the new or changed system can handle these requirements.

iii. Integration Testing

Integration testing essentially tests larger aspects of system workability. Integration system testing is used to test whether the system works well with other systems and whether the data delivered and received is usable and performs as expected in related systems. Interface testing for WFM systems, for example, is done to determine whether the data sent from the WFM system to the payroll system performs as expected and produces accurate payments.

iv. User Acceptance Testing

User acceptance testing (UAT) is the opportunity to allow testing participants from the end-user community to validate whether the system performs as expected. These testers test the expected and unexpected and help flush out potential gaps or problems before the system goes live. The WAM-Pro can also use the UAT phase to identify potential improvements where things work but not as well as might be possible. UAT testing can also help prove whether the training of the new system is acceptable or where training content should be enhanced or modified.

v. Regression Testing

Regression testing may be needed when WFM systems are reworked for bugs, patches, or moderate enhancements. In these situations, often none of the customer's unique setup changes, but the underlying software or code has been modified. The WAM-Pro leads the efforts to determine if the changes will change the expected behavior of the system and preserve any previous known problem fixes in the application. This is often done by rerunning prior test scenarios, testing for known problems and critical outputs, and perhaps executing stress testing exercises. When the specific fix is well documented, the scope of regression testing can be pared down to isolate the areas of potential impact rather than testing every aspect of the current system.

(e) Selecting Appropriate Testing Participants

Testing is an enterprise effort. Even though the WMO may be in charge, testing participants come from a number of areas because of the subject matter expertise needed for testing. This is especially true with user acceptance testing. One of the keys to testing is to have the right people selected and involved. It does no good to have someone test a system where they know nothing about the business areas or type of system they are testing.

The following is an example testing team. Depending on the size of the organization, one or more people may fill each role:

• Project leader
  • Manages the testing schedule, budget, and quality.
  • Regularly reviews the testing process.
  • Communicates with external stakeholders and related teams (e.g., CFO, CIO, payroll manager, and training developers).
  • Escalates testing issues, manages risks.
  • Works with external testing team members (e.g., consultants).
  • Owns the acceptance criteria.
  • Obtains leadership's system test sign-off
• Unit testers/programmers
  • Identifies test data and conditions.
  • Defines successful test criteria.
  • Executes test scenarios and documents results.
  • Resolves design problems.
  • Identifies system and stress test criteria and scenarios.
  • Reports system problems that cannot be resolved during design and build.
  • Participates in testing team meetings.
• Business analyst
  • Reviews the detailed test plans against operational objectives.
  • Understands the business rules in detail (e.g., familiar with specific overtime rules).
  • Defines testing procedures and user acceptance testing candidates.
  • Works to resolve design issues—liaison between system and business stakeholders.
  • Resolves and reports on business issues.
  • Participates in testing team meetings.
• Technical support
  • Builds the test environment (often making a copy of the development database and setting up test user accounts for the team).
  • Provides support during testing, making certain testing hardware and networks are operational.
  • May refresh applications with testing data as needed.
  • Monitors system performance during testing.
  • Suggests stress testing scenarios and helps to resolve system problems uncovered during testing.

(f) Tools for Testing—Automated and Tracking

There are a variety of tools available to assist with testing. These tools may range from automated testing systems that populate the system with data and complete rapid numbers of automatic testing to internal test environments prepopulated with customer-specific testing scenarios and data that are reused and enhanced over time to electronic or paper spreadsheets that track requirement and function testing that is completed manually. WFM system testing does not easily accommodate a preconfigured testing harness that comes with the application because the systems are designed to be highly configurable to the customer's business requirements. For example, not every employer operates in California so testing for those overtime computations is only appropriate for specific customers. Table 22.1 shows a sample spreadsheet layout identifying the test items and what type of difference, if any, the new configuration produced that was unexpected. The progression of testing moves from left to right and includes a place for the tester to document the modifications (remediation) used to correct the problem.

Table 22.1 Example Testing Spreadsheet: Unit Tester

Table 22.2 shows a sample testing worksheet used to identify and track testing items and test outcomes.

Table 22.2 System Testing

Each item is numbered and described, and a testing group is assigned (what set of employee data will be used). The number of test records is determined and the description of the requirement to be tested is listed. The tester should include the expected performance or outcome and any variance found during testing. For example: Clocks are updated with schedules every night, but the update routine takes four hours instead of 30 minutes. If a root cause is known, that is listed as well. For example: The updates include all information instead of just new or updated records increasing the size of the update file. The tester records their name. The test lead may prepopulate the testing documents with “weighting” that alerts everyone to the importance of certain test items as critical or not.

(g) Quality Assurance Process

QA is not simply the process of testing. It starts early with the analysis and design phases through implementation and concludes with the testing activities. The following is a generalized testing and QA process overview. Some organizations will have a more or less extensive process depending on the size and complexity of the project. Depending on the type of WFM technology being developed or purchased and configured, the actual tasks and sequence may vary. The WAM-Pro's job is to organize the process to be efficient while comprehensive enough to facilitate the successful release to the user community.

i. Analysis Phase of Project

The project finds its origins when the WAM-Pro receives the executive management goals and objectives for the WFM system. Prior to the actual analysis, the goals, objectives, and expectations should be well defined by management. This will provide the parameters for most of the requirements being developed and documented.

After the requirements have been defined they will need to be reviewed by the stakeholder community and approved. The WAM-Pro leads this effort, which includes an estimated costing effort to determine what implementing the requirements will cost the organization.

Next is the budgeting process. Based on the requirement and market research, the WAM-Pro begins work on estimating costs for the WFM system. In some cases, the cost will reduce the requirements and the activities that follow. Based on the approved requirements, a testing strategy will be developed. This will include the types of testing, the complexity and amount, standards, metrics, and expected results and processes.

The WAM-Pro then moves to the task of creating a compliance matrix. After the testing strategy and preparation have been completed, a matrix is created that includes the testing areas to be tested, indicating whether they have passed or failed and where the individual areas are in relation to the process.

The development of the test plan and testing preparation is the WAM-Pro's tactical approach and tools to execute the testing. It will include schedules, resources, what is being tested and how the overall process will be managed, the development of use cases, testing scripts, data development or preparation, and other materials that will verify the requirements are being delivered to address the organizational expectations.

The last activity in the analysis is to verify that all needed testing has been identified and that the materials to complete testing have been or will be created. The WAM-Pro creates the testing sign-off and makes certain that the testing process and outcome documentation is in place.

ii. Development Phase of Testing

The design and build phase of new systems and features is what most people focus on. However, the success of that phase is predicated on a solid, effective development of the testing. The following shows where testing fits in the system development process:

• Testing commercially developed software. Today large numbers of organizations will select commercial software or services to match their requirements rather than build them in-house. When software is purchased much of the work will be system configuration, which still requires testing even if less than for software developed in-house.
• Testing preparation. As the WFM system goes through the process of development or configuration, the WAM-Pro prepares the testing areas for the test process. This includes making sure that needed resources and testing systems are available and that the schedule is updated to begin tracking progress.
• Updates for requirements changes or additions. For WFM systems it is not atypical to experience a variety of changes to requirements and configurations during the development and configuration efforts. This may change the expected testing results. Due to this the WAM-Pro keeps track of such changes to determine whether any changes that have been made have been updated in the project documentation.
• Testing data preparation. The test data should have already been created or prepared. However, it may still need to be updated and placed in a defined location for the WFM system to use during testing. This includes the collection of data that is captured, generated, or calculated during the testing process.
• Testing process start. Prior to testing, development and changes to the software being tested should be complete and passed to the testing area. At that point in time the WAM-Pro communicates that testing can begin.
• Movement through testing process. During testing, the software will move through the previously selected testing types. The results of testing will send the software back to the developers or configuration specialist for repair, or it will be moved to the construction area of the implementation for final testing and release. The WAM-Pro makes certain that the acceptance criteria are being followed and rework is prioritized and updated in the testing schedule. This facet of testing is referred to as a redo cycle. If the software being tested fails, it must be repaired (redone). Keep in mind that this will require tracking to prevent confusion as to what has been tested and passed and what has failed. Some items may have to go through the redo cycle multiple times.
• Approval. There are a variety of approvals that may occur during testing. This includes the unit tests by the developer or configuration person as well as user acceptance. The WAM-Pro makes certain that each testing area has a zone of focus and manages where they may overlap with other areas of testing. Due to this, each person responsible for a test area needs to provide documented results and an approval that the WFM system has passed the testing effort.
• Release and promotion preparation. When software or a device has completed the testing process, it will be ready for assembly and placement into a system area that is used to prepare the item(s) for movement to the production system. At that time testing should be complete. The WAM-Pro will oversee the activities and scheduling for promoting changes from the test or QA environment into the production database.

iii. Implementation Phase of Project

The implementation phase is where the project takes on a heightened level of visibility. The WAM-Pro is crucial to orchestrating an uneventful go-live for the new system. Careful progress through this critical phase makes certain that the organization and system are both ready. The following is an outline of the primary steps in system deployment:

• Final user acceptance testing. Eventually in the overall testing process there will be a final user acceptance testing and acceptance. This is considered the place where testing that was planned to address possible failure areas has been completed and the results accepted. At this point the WAM-Pro is ready to release the new WFM system to the user community.
• Implementation release. The WAM-Pro works to schedule the implementation release so there is minimal disruption to the business, and when the release can be canceled if needed due to unforeseen problems. The implementation release means that the support for the system will be taken over by a designated group of staff and that any problems or fixes that are needed will fall into the proper process for system support and repairs.
• Release testing. As the WFM is released to the user community, there will still be another set of high-level testing to determine whether the system is performing as designed in the real-world user community environment.

Figure 22.1 shows the life cycle for the development or purchase of a WFM system and the testing and QA cycle.

Figure 22.1 Testing and Quality Assurance Cycle

iv. Testing and Quality Assurance Outcomes

The testing of any system or device before it is released to the user community is important. The ultimate outcomes of testing include:

• Confidence that the components meet the business and compliance requirements and are ready to support the organization.
• Validation that the components are designed to enhance the performance of the organization operationally, financially, and strategically as desired.
• Verifying that the released system will provide support and not disruption to the business.
• Opportunities to correct unexpected flaws in the developed software.
• Repair of identified flaws and omissions in system design.
• Adjustment of unexpected problems with configurations so settings work as required.
• Testing of the software environment so it can be determined that the software works as expected in the overall systems environment it will be placed in and adjusted if needed.
• Determining if unexpected areas of related systems have been broken due to the new system or changes and then repairing those areas/systems.

Many organizations that do not understand the purposes and benefits from testing and QA downplay the importance of the process only to find out that a failed system disrupts business and may not provide the management and business expectations for the organization. The WAM-Pro plays by the adage “expect what you inspect” and institutes a solid testing plan.

(h) Quality Assurance Terminology

The terms related to testing and QA are often confusing due to similarity and because many organizations customize their own testing processes and methods to fit their organization. The definition for a term in one organization may differ slightly from that of another. A word of advice: When working with any organization, request the definition of its testing terms, schedules, and processes since they will vary from organization to organization. For the list and definitions of many common testing and QA terms, see the glossary. By becoming familiar with the terminology, the WAM-Pro should be able to better communicate with those who are working on the system, as well as those who will work with it. Having a common glossary can help turn words into action during the phases of testing and QA.

Once organizations have moved onto the testing period, most of the planning has been accomplished and change is in motion. However, the testing and quality assurance activities are necessary checkpoints in the process. Actions that are repeated and sustained over time must be accurate and complete to serve the organization effectively. Organizations should position testing as a final opportunity to evaluate and assess the future usability, and ultimately the workability, of the system once it is in the workplace.

1. This chapter was contributed by James Thomas and Mike King.