10

Measuring ROI in an English-as- a-Second-Language Program: An Online Learning Solution

Performance Medica

BACKGROUND

The organization evaluated is a multinational research-based pharmaceutical company. In order to respect the privacy of the client company and for purposes of this project, the company will be called “Performance Medica.” The ROI evaluation described here represents the sixth year of evaluating the online English-as-a-second-language (ESL) course from a company called GlobalEnglish (GE). Due to its previous history as a holding company for many different types of businesses, Performance Medica has an HR function that does not include technical skills training, but focuses more on senior management and leadership development. While the company has shed many of the nonpharmaceutical businesses and now focuses exclusively on healthcare, technical training still resides in the various business units in the organization. As a result, the GE program was originally sponsored and rolled out by the company’s learning and performance (L&P) department. This group is responsible for training (directly and indirectly through global affiliates) more than 8,000 employees in the commercial organization (sales and marketing).

THE TRAINING NEED

Due to the size of the organization and the scope of the initiative, good metrics were very important to this project. As the process is described, this study references the 12 guiding principles of the ROI Methodology. For the company to meet its succession planning and career ladder goals, it needed to move a number of high-talent and high-potential people from country to country with an expected rotation in the United States. Succession planning was a stated business objective whereby the company was committed to develop and promote highly talented people. Those people were identified as tomorrow’s company leaders. As a U.S.-based multinational corporation, the lingua franca was English. Developing the English language skills for these non-native speakers was a major concern, but one that had been left up to the local affiliates to resolve on their own.

Historically, the affiliates tried to hire high-potential employees who already had suitable English language skills—although no standard for what was “suitable” was ever established. When ESL skills were less than “suitable,” it was up to the local affiliates to offer local ESL solutions to their staff. As a result, the outcomes of ESL training have been mixed. A number of affiliates suggested it would have been more effective if the company considered leveraging its global economy of scale to drive a standard ESL learning initiative.

KEY STAKEHOLDERS

A number of stakeholders were involved in this effort. However, it was a complex set of relationships. While the HR group (a key stakeholder) was responsible for the succession planning process, that group offered no ESL training to support the effort. A significant number of succession planning candidates in the affiliates came from their sales and marketing departments, therefore it was felt the global L&P department (another key stakeholder) should be tasked with supporting the ESL training. In addition, the affiliate HR managers were also key stakeholders responsible for implementing the local succession planning process. These managers were also critical decision makers in terms of both funding the ESL solution and enrolling affiliate learners.

ADDRESSING THE NEED: ONE FAILURE

Because succession planning was the driving factor in developing English language skills in high-potential affiliate personnel, the solution had to be cost-effective, minimize time away from work, and accommodate students at various levels of English capability. After repeated requests from overseas affiliates for support of this activity, several global ESL training companies were located and reviewed. Ultimately, a global contract meant to leverage global buying efficiencies was negotiated with Berlitz language services. However, after the first year the approach was abandoned for the following reasons:

• Despite global pricing, many affiliates were able to find local classroom training at lower prices—usually from a local university.

• Classroom training sponsored within the company’s local offices presented a challenge in scheduling.

• To keep costs reasonable, classes had a wide range of learners at various skill levels. Beginner, intermediate, and advanced learners sat in the same classroom, which created a lot of challenges.

The affiliates refused to participate in the Berlitz program after the first year, so the Berlitz contract was allowed to expire without renewal. An alternate solution needed to be located—one that addressed the problems experienced with the Berlitz program and one that the markets would embrace. An online ESL training system was found to be the potential solution. At the time, GlobalEnglish (GE) was one year old and had been started by a venture capital company. GE’s premise was simple:

• English is the global language of business.

• All resources would be put into developing interactive learning tools to support just one language—English.

• There were 11 levels of business English covered in the GE program.

• There were different course tracks for different English language skills, such as grammar, writing, speaking, and listening.

• The approach would allow a learner with level 10 skills in writing to take the level 10 writing track. However, if the learner had level five English grammar skills, the learner would be placed in the level five grammar track. This allowed for unprecedented customization not possible in a group classroom setting.

• Instructions on using the system in the first five levels are provided in local languages until the learner had sufficient English language skills to follow both lesson instructions and the lesson content completely in English.

A pilot program for GE was commenced for 50 people in several affiliates. The ROI Methodology was not applied to the pilot. What was used was a simple measure of Level 1–3 results. Specifically, learners provided feedback on their experience using the GE system (Level 1). Their improvement (or lack thereof) in English test scores from their original placement to when they completed the pilot were measured through assessment in the GE system (Level 2). Simulated application exercises, also within the GE system, were measured from the benchmark placement process at the beginning of the pilot to the learner’s final performance at the end of the pilot (Level 3). Based on the results of the pilot, which were favorable, the GE system was adopted.

MEASURING RESULTS

The L&P department, the program sponsor, was tasked with organizing the global rollout of the program. Since then, a limited ROI analysis has been added to the original pilot measures and is conducted in the fourth quarter every year. The original ROI method simply asked the learners to provide an estimate of how much time was saved due to their new English language skills. Once an “hour” value was established, conducting a ROI was relatively easy. On average, the company has seen an 800 percent ROI for each of the five years the program has been employed. However, many stakeholders were skeptical about the results of those earlier ROI analyses. Therefore, after five years, elements of the ROI Methodology were also added. For the first time, learners and their managers were asked: In addition to how much time was saved due to improved English language skills, how confident are learners and their managers in that estimate and how certain are they that the GE program was the reason for the results? This approach created a more defensible and credible ROI analysis. The balance of this study focuses on this improved process. The year the ROI Methodology was adopted there were 426 users on the system, an all-time high. Therefore, a more credible and rigorous ROI analysis was critical to measuring the value of the program.

Evaluation Objectives

Level 1, Reaction Objectives

1.  Determine learner satisfaction with the GE learning methodologies using a 5-point scale from “dissatisfied” to “extremely satisfied.”

2.  Determine learner self-satisfaction with progress improving English skills using a 5-point scale from “dissatisfied” to “extremely satisfied.” This last point can be correlated to actual Level 2 and 3 assessment results to see if learner perception matches reality.

Level 2, Learning Objectives

1.  Objective test scores for knowledge based on placement assessment and progress assessment in order to progress through the 11 levels. An assessment score of 80 percent or higher is required to move to the next level in a particular skill set.

2.  Learning objectives will focus on knowledge of vocabulary and rules of grammar.

Level 3, Application Objectives

1.  Objective test scores for skill application based on placement assessment and progress assessment in order to progress through the 11 levels. An assessment score of 70 percent or higher is required to move to the next level in a particular skill set.

2.  Application objectives, which focus on reading, writing, listening, and speaking skills as applied to specific, real job tasks, as differentiated from simulated tasks or assignments for reading, writing, listening, and speaking skills within GE. It is important to note that, with regard to reading, writing, listening, and speaking skills—whether applied to a simulated task (as within a course lesson) or when applied to a real job task or activity—the methods used for evaluation will be quite similar. The difference in this case is that the simulated Level 3 situations within GE can be automatically assessed by the administration module within the GE system. The real-world job application will need to be evaluated by qualified assessors on the job.

3.  Assessment by the learners and their managers (or qualified assessors) using an on-the-job checklist of 12 business situations will be conducted. Ratings will be ranked on a measure of five levels of improvement.

Levels 4 and 5, Business Impact and ROI Objectives

1.  Determine level of importance of English skills to the learner’s job and career aspirations.

2.  Determine learner’s estimate of time saved due to improved English language skills.

3.  Determine learner’s confidence estimate of how much time was saved due to improved English skills.

4.  Determine learner’s percent estimate of GE’s contribution to the improvement in the learner’s English language skills.

DATA COLLECTION PLAN

Table 10-1 provides a detailed look at the data collection plan. It is important to note that Level 2 and 3 data are actually collected through placement and progress assessments within the GE program administrative module and are not extraneous data collection tools. As an integral part of the program, these assessments meet the recognized generally accepted standards and criteria of ESL assessment methodology.

TABLE 10-1. Data Collection Plan

CONVERTING DATA TO MONETARY VALUES

Based on the data collected, the Level 1 through 3 objectives contribute to the development of a positive case for the use of GE in delivering ESL learning. Translating that learning into a monetary value comes from the Level 4 and 5 assessments, which ask learners how much time they and their managers believe was saved due to improved English language skills. To add to the credibility of this estimate, the respondents’ level of confidence in that estimate was captured and ultimately combined with the respondents’ level of confidence in GE’s contribution to that result to develop a very conservative and credible estimate of the time saved due to new ESL skills. Once a time saved value is determined (for instance, one hour per week), a financial value will be associated with that time (such as a fully loaded labor cost) and an ROI cost determined by taking into account the annual estimate of the value of the time saved, less the annual value of the time spent studying, divided by the annual cost per learner for the program. This approach will provide a very solid ROI case for the program.

ISOLATING THE IMPACT OF TRAINING

A chain of evidence is an essential component of demonstrating the impact and ROI of a training solution used as a performance improvement intervention. Specifically, conducting Level 1, 2, 3, 4, and 5 evaluations. Level 5 evaluation is critical to developing a supportable and credible foundation for the overall ROI analysis.

It is important to note that the learners and their managers are scattered around the globe. With 426 users (and associated supervisors) around the world, the most efficient method to gather learner reaction (Level 1), business impact (Level 4), and ROI (Level 5) was in a single survey. The critical element is that the survey had to be crafted in such a way that learner (and manager) reactions and expectations are aligned with a perception of the metrics that are relevant to the measure of success used by these learners, supervisors, and their organization. By linking Level 1, 4, and 5 data questions into a single survey, collating the data from potentially more than 850 people (426 learners and the same number of managers) from around the globe, the data collection became a much more manageable task. It is also important to keep in mind that in addition to this annual survey, which captures Levels 1, 4, and 5, the learners are constantly taking Level 2 and 3 assessments as they progress through the program.

Another factor in planning to conduct a global ROI study for an ESL program was the quality of English of both the beginning learners and the supervisors. Just because the learner is learning English language skills does not mean all the skills to answer a survey were mastered or that the learner’s supervisor(s) have mastered enough English as well (in order to respond to the survey). In addition, there are no resources to translate the survey into all the languages that might be required. As a result, it was important to keep the survey as short as possible, keep the form of English as simple as possible, and leave enough time for respondents to complete the survey. These factors were treated as prerequisite issues and potential constraints that had an impact on the design of the evaluation tools.

Consistent with the ROI Methodology Guiding Principle 5—“At least one method must be used to isolate the effects of the solution”—to isolate the contribution of training to improve job performance as a result of better English language skills, a number of the ROI Methodology techniques were employed. Two of the nine methods of data isolation were used in this ROI analysis.

Guiding Principle 7 states that estimates of improvements should be adjusted for the potential error of the estimate. To accomplish this, a questionnaire to gather data from learners was developed. The three key questions in order to capture data for the ROI analysis were:

1.  How much time has been saved due to improved English language skills? ______ hours have been saved each week.

2.  On a percentage basis, what is the confidence level concerning the number of hours saved each week? ___%

3.  On a percentage basis, what is the confidence level that GlobalEnglish is the reason this time has been saved? ___%

Again, consistent with Guiding Principle 7, a second questionnaire was developed to gather data from the learners’ supervisors. The key questions for the supervisors were:

1.  If one were to assume that improved English language skills allow an employee to work more effectively due to the ability to read and respond to English language emails, telephone calls, teleconferences, and meetings with greater skill and confidence, what would an estimate be of how many minutes or hours per week the learner(s) save(s) due to a perceived improvement in English language skills? ____

2.  On a percentage basis, what is the confidence level concerning the number of hours saved each week? ___%

3.  On a percentage basis, what is the confidence level that GlobalEnglish is the reason this time has been saved? ___%

With the data that resulted from these questions, a unit of measure, that is, time saved, was established, converted into a dollar value, and ultimately used to generate a benefit-cost ratio (BCR) and an ROI. This is an approach that, while not terribly sophisticated, is simple to measure, very defensible to management, and fits within the constraints of conducting a global ROI analysis.

EVALUATION RESULTS

Determining the Benefits

As mentioned earlier, for the company to meet its succession planning and career ladder goals it needs to move a number of high-talent and high-potential people from country to country with an expected rotation in the United States. No financial analysis of the value of such a program has been conducted by the company, and such an analysis is beyond the scope of this study. Succession planning is, however, a stated business objective whereby the company is committed to develop and promote highly talented people. The assumed and accepted benefit of such a program is key to the company’s growth, and effective English language skills are required for the succession planning process to be successful. That said, the financial benefit of the GE ESL program can be found in Table 10-2.

TABLE 10-2. Calculation of Dollar Benefits of the Online ESL Program

Calculating the Costs

For the five years since the GE program was introduced, an older annual ROI analysis has been conducted. Each previous year in which the older ROI analysis was conducted, the analysis was applied to just that year. This is consistent with Guiding Principle 9: “Only the first year of benefits [annual] should be used in the ROI analysis of short-term solutions.” Since most learners are only in the system for 12 to 15 months, each year can be considered the “first year” for that audience. That is why the survey is conducted annually. Its ongoing value to each year’s audience needs to be established. By adding the concepts offered by the Phillips Methodology, the quality and credibility of these annual ROI analyses will be enhanced. In fact, the costs that need to be captured for this project are quite easy to calculate. In the previous ROI calculations, a fully loaded labor cost per hour was provided by HR and it includes opportunity cost, that is, what work the employees could be doing if they were not taking the GE program. This aligns with Guiding Principle 10, which states, that the costs for the solution should be fully loaded for the ROI analysis. Additional costs include the per-user license fee for one year of access to GlobalEnglish, as well as the cost of broadband access. Because the cost of broadband access turned out to be miniscule, it was not used in the ROI analysis. This decision is in keeping with Guiding Principle 8, extreme data items and unsupported claims should not be used in the ROI analysis of short-term solutions. That measure, while having been calculated, turned out to be inconsequential in the final analysis. The details of that calculation can be found in Table 10-3.

TABLE 10-3. Annual Cost of T1 Broadband Access for 450 GE Learners

The entire calculation of all costs for the GE program implementation can be found in Table 10-4.

TABLE 10-4. Annual Cost of GlobalEnglish Usage for 426 Learners

Using the data from Tables 10-2, 10-3, and 10-4, the final ROI cost calculation is provided below:

The benefit-cost ratio (BCR) is calculated as follows:

COMMUNICATION PLAN

Guiding Principle 12 states, “The results from the ROI Methodology must be communicated to all the key stakeholders.” After determining the ROI and BCR for the GE learning initiative, it was critical to communicate those results to key stakeholders who influence the availability of resources to continue the program. Without communicating the results of the study, there is a very high risk that in a budgetary downturn the program could be cut because key stakeholders were unaware of the impact of the program. Putting together an effective communication plan required the following elements:

• Communication had to be timely.

• Communication was targeted to specific audiences.

• Media used were carefully selected.

• Communication was unbiased and modest.

• Communication was consistent.

• Testimonials were more effective coming from individuals the audience respects.

• The audience’s opinion of the learning and development staff and function had influence on the communication strategy.

CONCLUSIONS

This case study has demonstrated that by using a disciplined comprehensive approach to evaluating the GE online ESL program, it is quite possible to make a strong qualitative and quantitative case for investing in this learning intervention. The ROI Methodology approach has provided valid and persuasive tools and methods to tease out the financial impact, specifically the ROI, of implementing the program. The added rigor, discipline, and operating standards brought to the evaluation process helps to insulate the program and the study from critics who use subjective criteria to attack the value of the program. This process moves the entire evaluation methodology for learning programs from a soft, subjective assessment to a concrete, comprehensive, and objective analysis of the impact of the program on learner performance and company business results. These are very powerful tools for learning professionals to support their recommendations for investments in future technology-based learning programs.

DISCUSSION QUESTIONS

1.  Are the results of this study credible? Explain.

2.  Although participants are guided to think of program evaluation as including five distinct levels, for reasons of practicality, this study clearly combined the qualitative aspects of Levels 1, 3, 4, and 5 into one survey tool. Is this acceptable? If so, why? If not, why not?

3.  Why is the chain of impact so important in a study such as this one?

4.  An important part of this ROI case study was the ability to determine an increase in workplace application of improved English-as-a-second-language (ESL) skills by using percentages of time and converting those percentages to dollar values. Were there other methods one could use to determine the financial impact of new ESL skills? Please explain.

5.  How might the ROI process in this case study be improved?

REFERENCES

Phillips, J.J. (2003). Return on Investment in Training and Performance Improvement Programs. Oxford, UK: Butterworth-Heinemann.

Phillips, J.J. (2006). Return on Investment Measures Success [Electronic version]. Industrial Management, 48, 2.18–23.

ABOUT THE AUTHOR

Edward (Ed) P. Nathan PhD, CRP, CPT, has over 25 years’ experience in global learning and development. Having worked in more than 40 countries with global companies such as Pfizer, Wyeth Pharmaceuticals, Lederle Laboratories, and Walmart, he has conducted research in developing global e-learning programs and measuring the ROI of online ESL learning.

Nathan earned his PhD in education with a specialization in training and performance improvement from Capella University. He holds a CRP (Certified ROI Professional) from the ROI Institute and a CPT (Certified Performance Technologist) from the International Society for Performance Improvement. With numerous papers published, over a dozen conference presentations, and two textbook chapters, he is currently working on a book focusing on the use of organic learning methodologies in the classroom. Properly applied, these methods help facilitate learners’ self-mastery of complex business and academic processes. He makes his home in West Chester, Pennsylvania, and can be reached at [email protected].