4

Data Analysis

The data collected during the data collection phase must be analyzed. When using the ROI Methodology, data analysis should include the isolation of program effects on the data, calculation of fully loaded program costs, conversion of data to monetary values when appropriate, and the ROI calculation, if needed. This chapter provides a brief overview of each of these issues.

ISOLATING THE EFFECTS OF LEARNING

Isolating the effects of a program on business impact data is one of the most challenging, yet necessary, steps in the ROI Methodology. When addressed credibly, this step links learning through technology directly to business impact.

Other Factors Are Always Present

In almost every situation, multiple factors affect business results. During the time that programs are implemented, many other functions within the organization may be attempting to improve the same metrics addressed by the program. For example, marketing projects are designed to improve sales; and while a technology-based learning program should help a sales professional positively affect sales, the marketing project should also be positively affecting sales. In addition to internal factors, external factors may be affecting business results during the time that the program is occurring.

Without Isolation, There Is Only Evidence, Not Proof

Without taking steps to show the contribution of learning, there is no clear business linkage; instead, there is only evidence that learning may have made a difference. When business results improve during the learning program, it is possible that other factors may have contributed to that improvement. The proof that the program has made a difference on the business comes from isolating the effects of the program or initiative.

Other Factors Have Protective Owners

The owners of the other functions that influence business results are convinced that their processes or programs make the difference. Other processes or programs, such as advertisements, events, reward systems, and job design all have protective owners, and their arguments are plausible. Therefore, program owners are under pressure to build a credible argument for their case to claim value-add to the organization.

To Do It Right Is Not Always Easy

The challenge of isolating the effects of the learning program on the business is critical and can be done; but it is not easy for complex programs, especially when strong-willed owners of other processes are involved. It takes a determination to address this situation every time an ROI study is conducted. Fortunately, a variety of approaches are available.

TECHNIQUES TO ISOLATE THE EFFECTS OF LEARNING

Before reviewing the specific methods, it is helpful to highlight two important issues. First, although it is possible to isolate for the effect of the program at Level 3 application, isolation is usually applied to Level 4 impact. The business impact connected to the program is the key issue; when calculating ROI, the improvement in business measures is reported in monetary terms, and more than any other level, it must be credible. After the business impact data have been collected, the next step is to isolate the effects of the program. This step demonstrates the proof that the program made a difference, whereas reporting results along the chain of impact only presents evidence of the connection between the program and business results.

Another important issue is to attempt to identify the other factors that have contributed to the improvement in the business results measures. This step recognizes that other factors are almost always present and that the credit for improvement is shared with other functions in the organization. Just taking this step is likely to gain respect from the management team.

Several potential sources can help identify these influencing factors. The sponsors of the project may be able to identify the factors. Subject matter experts, process owners, and those who are most familiar with the situation may be able to indicate what has changed to influence the results. In many situations, participants know what other factors have actually influenced their performance. After all, it is their direct performance that is being measured and monitored.

By taking stock in this issue, all factors that contributed to improvement are revealed, indicating the seriousness of the issue and underscoring how difficult it is going to be to isolate the effects of the programs. A variety of techniques can help address the isolation issue, which are discussed next.

Comparison Group Analysis

The most accurate and credible approach to isolate the effects of a learning program is a comparison group analysis, known as the control group arrangement. This approach involves the use of an experimental group that participates in the learning program and a control group that does not. The composition of both groups should be as similar as possible, and, if feasible, the selection of participants for each group should be on a random basis. When this is possible and both groups are subjected to the same environmental influences (for example, market growth, and systems), the differences in the performance of the two groups can be attributed to the program. As illustrated in Figure 4-1, the control group and experimental group do not necessarily have preprogram measurements. Measurements are often taken after the program is implemented only, rather than prior to the program and then again after the program. The difference in the performance of the two groups shows the amount of improvement that is directly related to the program. Chapter 11 provides an example of a comparison group analysis.

FIGURE 4-1. Control Group Design (Post-Test Only)

Assumptions

For the comparison group analysis to be used, five conditions must exist.

1.  One or two outcome measures represent the consequence of the program. This is the business measure in question.

2.  In addition to the program, the factors that influence the outcome measures can be identified.

3.  There are enough participants available from which to select the two groups.

4.  The program can be withheld from the control group without any operational problems.

5.  The same environmental influences affect both groups during the experiment (except that one group participates in the program).

If these assumptions can be met, then there is a possibility for a control group arrangement.

Issues and Opportunities With Comparison Groups

Some issues surface with control group arrangement, which may make it difficult to apply in practice. When addressed properly the issues become opportunities. The first major issue is that the process is inappropriate for some situations. For some programs, it may not be proper to withhold the program from one particular group while it is conducted in another. This particular barrier may prevent many control groups from being used. However, in practice, there are many opportunities for a control group arrangement when a pilot program is implemented.

The second issue is that the control groups must be addressed early enough so that similar groups can be used in the comparison. Dozens of factors can affect employee performance, some of them individual and others contextual. To tackle the issue on a practical basis, it is best to select three to five variables that will have the greatest influence on performance.

A third issue with the control group arrangement is contamination, which can occur when program participants influence others in the control group. Sometimes the reverse situation occurs, when members of the control group model the behavior from the experimental group. In either case, the experiment becomes contaminated because the influence of the program filters to the control group.

Another issue is timing. The longer a control group and experimental group comparison operates, the greater the likelihood that other influences will affect the results. More variables will enter into the situation, contaminating the results. On the other end of the scale, there must be enough time so that a clear pattern can emerge between the two groups. Thus, the timing for control group comparisons must strike a delicate balance of waiting long enough for their performance differences to show, but not so long that the results become seriously contaminated.

A fifth issue occurs when the different groups function under different environmental influences. Because they may be in different locations, the groups may have different environmental influences. Sometimes the selection of the groups can help prevent this issue from occurring.

A sixth issue with using control groups is that it may appear to be too research-oriented for many business organizations. For example, management may not want to take the time to experiment before proceeding with a program, or they may not want to withhold a program from a group just to measure the impact of an experimental program. Because of this concern, some practitioners do not entertain the idea of using comparison groups.

When using a control group to study the effect of a technology-based program, it is important for the program impact to be isolated to a high level of accuracy; the primary advantage of the control group process is accuracy.

Trend-Line Analysis

Another technique used to isolate the impact of programs is trend-line analysis. This approach has credibility when it is feasible. It is also a simpler alternative to the control group arrangement.

A trend line is drawn using preprogram performance as a base and extending the trend into the future. After the program is implemented, actual performance is compared to the projected value, the trend line. Any improvement of performance over what the trend line predicted can then be reasonably attributed to the program. For this to work, the following assumptions must be verified:

• Preprogram data are available. Data represent the impact data—the proposed outcome of the program.

• Preprogram data should be stable, not erratic.

• The trend that has developed prior to the program is expected to continue if the technology-based program is not implemented to alter it.

• No other new variables entered the process after the program was conducted. The key word is “new,” realizing that the trend has been established because of the variables already in place, and no additional variables enter the process beyond the implementation of the program.

When the variance of the data is high, the stability of the trend line becomes an issue. If this is an extremely critical issue and the stability cannot be assessed from a direct plot of the data, more detailed statistical analyses can be used to determine if the data is stable enough to make the projection. The trend line can be projected with a simple formula available in software packages or office tools such as Microsoft Excel.

The key element in this approach is to track the trend using historical data; project where the trend would be without help from a program; then after the program occurs, track the actual data over the same period of time as the preprogram data. Then the comparison can be made between what the forecast data show and what the actual data show.

Example

Figure 4-2 shows an example of a trend-line analysis taken from a sales department of a book distribution company. The percentage reflects the level of actual sales compared to sales goals. Data are presented before and after program implementation in July. As shown in the figure, an upward trend on the data began prior to program implementation. Although the program apparently had an effect on sales, the trend line shows that some improvement would have occurred anyway, based on the trend that had previously been established. Program leaders may have been tempted to measure the improvement by comparing the average six months’ sales prior to the program (87.3 percent) to the average six months after the program (94.4 percent), yielding a 7.1 percent difference. However, a more accurate comparison is the six-month average after the program compared to the trend line (92.3 percent). In this analysis, the difference is 2.1 percent. Using this more conservative measure increases the accuracy and credibility of the process to isolate the impact of the program.

FIGURE 4-2. Sample Trend-Line Analysis

Advantages and Disadvantages of Trend-Line Analysis

The primary advantage of this approach is that it is simple and inexpensive. If historical data are available, a trend line can quickly be drawn and differences estimated. While the approach is not exact, it does provide a quick assessment of a program’s potential results.

A disadvantage of the trend-line approach is that it is not always accurate. The use of this approach assumes that the events that influenced the performance variable prior to the learning program are still in place after program implementation, except for implementation of the program. Also, it assumes that no new influences entered the situation at the time the learning program was implemented. Unfortunately, this is not the case most of the time.

Forecasting Methods

A more analytical approach to trend-line analysis is the use of forecasting methods that predict a change in performance variables based on the correlation of other variables. This approach represents a mathematical interpretation of the trend-line analysis when other variables enter the situation at the time the learning program is implemented.

The primary advantage of forecasting is that it can predict performance in business measures with some level of accuracy, if appropriate data and models are available. While there are no absolutes with any technique, using an appropriate level of statistical analysis can provide credible, reliable results.

A major disadvantage with forecasting occurs when several variables enter the process. The complexity multiplies, and the use of sophisticated statistical packages for multiple-variable analyses is necessary. Even then, a good fit of the data to the model may not be possible. Many organizations have not developed mathematical relationships for output variables as a function of one or more inputs. Without them, the forecasting method is difficult to use.

Expert Estimation

An easily implemented method to isolate the effect of a learning program is to obtain information directly from experts who understand the business performance measures. The experts could be any number of individuals. For most learning programs, the participants are the experts. After all, their performance is in question and the measure is reflecting their individual performance. They may know more about the relationships between the different factors, including the impact of the learning, than any other individual.

Because of the importance of estimations from participants, much of the discussion in this section relates to how to collect this information directly from them. The same methods would be used to collect data from others. The effectiveness of the approach rests on the assumption that participants are capable of determining how much of a performance improvement is related to the learning program. Because their actions have produced the improvement, participants may have an accurate perception of the issue. Although an estimate, this value will typically have credibility with management because participants are at the center of the change or improvement.

When using this technique, several assumptions are made:

• A learning program has been conducted with a variety of different activities, exercises, and learning opportunities all focused on improving performance.

• Business measures have been identified prior to the program and have been monitored following the program. Data monitoring has revealed an improvement in the business measure.

• There is a need to link the learning program to the specific amount of performance improvement and develop the monetary effect of the improvement. This information forms the basis for calculating the actual ROI.

• The participants are capable of providing knowledgeable input on the cause-and-effect relationship between the different factors, including learning and the output measure.

With these assumptions, the participants can pinpoint the actual results linked to the program and provide the data necessary to develop the ROI. This can be accomplished by using a focus group or a questionnaire.

Focus Group Approach

The focus group works extremely well for this challenge if the group size is relatively small—in the eight to 12 person range. If much larger, the group should be divided into multiple groups. Focus groups provide the opportunity for members to share information equally, avoiding domination by any one individual. The process taps the input, creativity, and reactions of the entire group.

When conducting a focus group, the following steps are recommended to arrive at the most credible value for learning program impact:

1.  Explain the task.

2.  Discuss the rules.

3.  Explain the importance of the process.

4.  Select the first measure and show the improvement.

5.  Identify the different factors that have contributed to the performance.

6.  Identify other factors that have contributed to the performance.

7.  Discuss the linkage.

8.  Repeat the process for each factor

9.  Allocate the improvement.

10. Provide a confidence estimate.

11. Ask the participants to multiply the two percentages.

Example

Participants who do not provide information are excluded from the analysis. Table 4-1 illustrates this approach with an example of one participant’s estimations. The participant allocates 50 percent of the improvement to the learning program. The confidence percentage is a reflection of the error in the estimate. A 70 percent confidence level reduces the estimate to an adjusted percentage of 35 percent (50% × 70% = 35%). In essence, this error adjustment assumes the lowest percentage in an error range. If a person is 70 percent confident in their estimate, that means they are 30 percent uncertain (a 30% error). Given this level of uncertainty, the margin of error is 50% × 30% = 15%. With a margin of error of +/– 15 percent, the range of improvement is 35 to 65 percent. To be conservative, the lowest end of the range, 35 percent, is reported as improvement. Participants who do not provide information are excluded from the analysis.

TABLE 4-1. Example of a Participant’s Estimation

The use of expert estimations provides a credible way to isolate the effects of technology-based learning when other methods will not work. It is often regarded as the low-cost solution to the problem because it takes only a few focus groups and a small amount of time to arrive at this conclusion.

Questionnaire Approach

Sometimes focus groups are not available or are considered unacceptable for the purposes of isolating the effects of a learning program. The participants may not be available for a group meeting, or the focus groups may become too expensive. In these situations, it may be helpful to collect similar information via a questionnaire. With this approach, participants address the same issues as those addressed in the focus group, but now on a series of impact questions imbedded into a follow-up questionnaire.

The questionnaire may focus solely on isolating the effects of the learning program, as detailed in the previous example, or it may focus on the monetary value derived from the program, with the isolation issue being only a part of the data collected. Using questionnaires is a more versatile approach when it is not certain exactly how participants will provide business impact data. In some programs, the precise measures that will be influenced by the learning program may not be accessible to the evaluation. This may be the case when participants are from many different organizations. In these situations, it is helpful to obtain information from participants on a series of impact questions, showing how they have used what they have learned and how the work unit has been affected. The recommended series of questions is shown in Table 4-2. It is important for participants to know about these questions before they receive the questionnaire. The surprise element can be disastrous in this type of data collection. The specific actions to improve response rates were presented in chapter 3.

TABLE 4-2. Recommended Series of Questions for Isolating Program Results

Although this is an estimate, the approach has considerable accuracy and credibility. Four adjustments are effectively used with this method to reflect a conservative approach:

• According to Guiding Principle 6, the individuals who do not respond to the questionnaire or provide usable data on the questionnaire are assumed to have no improvements.

• Extreme data and incomplete, unrealistic, and unsupported claims are omitted from the analysis, although they may be included in the intangible benefits.

• Since only annualized values are used, it is assumed that there are no benefits from the program after the first year of implementation.

• The confidence level, expressed as a percentage, is multiplied by the improvement value to reduce the amount of the improvement by the potential error. This is Guiding Principle 7.

Collecting an adequate amount of quality data from the series of impact questions is the critical challenge with this process. Participants must be primed to provide data, and this can be accomplished in several ways, which were explored in the previous chapter. Here are five important ones:

• Participants should know in advance that they are expected to provide this type of data along with an explanation of why the information is needed and how it will be used.

• Ideally, participants should see a copy of this questionnaire and discuss it while they are involved in the program.

• Participants should be reminded of the requirement prior to the time to collect data.

• Participants should be provided with examples of how the questionnaire can be completed, using likely scenarios and types of data.

• The immediate manager should guide participants through the process and review and approve the data, if necessary.

These steps help keep the data collection process with its chain of impact questions from being a surprise. It will also accomplish three critical tasks.

• The response rate will increase. Because participants commit to provide data during the session, a greater percentage will respond.

• The quantity of data will improve. Participants will understand the chain of impact and understand how data will be used. They will complete more questions.

• The quality of the data is enhanced. With up-front expectations, there is greater understanding of the type of data needed and improved confidence in the data provided.

The estimation process is an important technique to isolate the effect of technology- based programs. However, the process has some disadvantages. It is an estimate and, consequently, does not have the accuracy desired by some managers. Also, the input data may be unreliable since some participants are incapable of providing these types of estimates. They might not be aware of exactly which factors contributed to the results or they may be reluctant to provide data. If the questions come as a surprise, the data will be scarce.

Several advantages make this strategy attractive. It is a simple process, easily understood by most participants and by others who review evaluation data. It is inexpensive, takes very little time and analysis, and thus, results in an efficient addition to the evaluation process. Estimates originate from a credible source—the individuals who actually produced the improvement.

The advantages seem to offset the disadvantages. Isolating the effects of learning programs will never be exact, but this estimation process may result in data that are accurate enough for most stakeholders.

SELECTING ISOLATION TECHNIQUES

With several techniques available to isolate the impact of learning programs, selecting the most appropriate techniques for the specific program can be difficult. Estimates are simple and inexpensive, while others are more time consuming and costly. When attempting to make the selection decision, several factors should be considered:

• feasibility of the technique

• accuracy provided with the technique, when compared to the accuracy needed

• credibility of the technique with the target audience

• specific cost to implement the technique

• the amount of disruption in normal work activities as the technique is implemented

• participant, staff, and management time needed with the particular technique.

Multiple techniques should be considered if the reliability of one technique is in question. When multiple sources are used, the most conservative method is recommended. If two methods are credible, the lowest value is used (Guiding Principle 4). The target audience should always be provided with explanations of the process and the various subjective factors involved. Multiple sources allow an organization to experiment with different techniques and build confidence with a particular technique.

Because it is not unusual for the ROI in learning programs to be high, the audience should understand that, although every effort was made to isolate the impact, it is still a figure that is not exact and may contain error. It represents the best estimate of the impact given the constraints, conditions, and resources available.

By isolating the effects of learning, the outcomes are clearly connected to the program, accounting for other factors. By ignoring this step, the reported results lack credibility. To calculate the ROI for the learning program, the next step is to convert business impact data to money.

TYPES OF DATA

ROI is developed through the comparison of the monetary benefits of a program and the cost (or investment) in that program. It is an economic indicator, meaning that the metric indicates the financial return on the investment. To develop this measure, impact data (Level 4 results) are converted to monetary value then compared to the cost of the program. Before we describe the development of the ROI calculation, it is important to review the ways in which impacts are often described.

Hard Data vs. Soft Data

Data fall into one of two categories. They are either referred to as hard data or soft data. Hard data are easy to measure, quantifiable, objectively based, and immediately credible with management. They represent rational, undisputed facts and are usually easy to capture. Hard data can be broadly categorized as output, quality, cost, and time. Examples include sales, new accounts, time to revenue, sales cycle time, downtime, customer complaints, product returns, rework, waste, errors, operating costs, incidents, accidents, absenteeism, turnover, compliance discrepancies, and cycle time.

Soft data represent measures that are difficult to measure and quantify; they are subjectively based and behaviorally oriented. Compared to hard data, these measures, while important, are often perceived as less reliable or less credible when converted to monetary value, due to the inherent level of subjectivity. Examples of soft data are customer satisfaction, customer loyalty, brand awareness, and reputation.

Tangible vs. Intangible Data

Two other categories by which data are often referred are tangible and intangible data. On the surface, measures such as customer satisfaction, teamwork, job engagement, and creativity may seem like difficult soft data items to measure and value, but consider the following:

• Though customer satisfaction seems like a soft measure, quantitative values are assigned to customer satisfaction to create an index. These numbers quantify customer satisfaction.

• When participants apply their newly acquired leadership skills, it may result in improved teamwork. Take that a step further to realize that improved teamwork likely yields greater productivity, leading to increased output and reduced costs—both measures considered as hard data.

• When participants are more creative, this new creative thinking may lead to more efficient business meetings, which results in time savings that can be quantified.

• Employee engagement may seem like a soft measure, but when engagement results in more productivity (revenue per employee), it can be quantified.

Ultimately, soft data lead to hard measures. Some suggest that hard data represent tangible measures; others suggest that soft data represent intangibles. But this is not the case. Both tangible and intangible measures may evolve from either hard data or soft data. This is why to categorize a measure as hard or soft is ambiguous. An alternative categorization scheme is tangible versus intangible.

Tangible benefits of a program are those benefits that have been converted to money. Intangible benefits of a program are those benefits that have not been converted to money. Hard and soft data can be converted to monetary value. Hard data have a direct link to their monetary value, while soft data are converted by tying soft measures to hard measures. Then the measures are converted to money either by associating the measure with cost savings, cost avoidance, or revenue, which is then converted to profit, as shown in Figure 4-3.

FIGURE 4-3. Data Conversion

While all measures can be converted to money, several factors should be considered:

• Cost to convert the measure: The cost to convert data should not cost more than the evaluation itself.

• Importance of the measure: Some measures, such as customer satisfaction and employee satisfaction, stand alone quite well. When that is the case, you might think twice before attempting to convert the measure to money.

• Credibility: While most business decisions are made on somewhat subjective data, the source of the data, the perceived bias behind the data, and the motive in presenting the results are all concerns when data is somewhat questionable. Don’t risk credibility just to calculate an ROI. For those times when it is difficult to decide whether or not to convert a measure to monetary value, complete the four-part test shown in Figure 4-4.

DATA CONVERSION METHODS

There are a variety of techniques available to convert a measure to monetary value. These are listed in Table 4-3 in order of credibility. The success in converting data to monetary value is knowing which values are currently available. If values are not available, it is possible to develop them. The use of standard values is by far the most credible approach, because standard values have been accepted by the organization. Following those, however, are the operational techniques to convert a measure to money.

Standard Values

Many organizations have standard values for measures of turnover, productivity, and quality. Those organizations that are involved in Six Sigma or other quality initiatives have many measures and the monetary values of those measures. Look around the organization and talk with people to discover what is being measured in various functional areas of the organization. It may be possible to find a monetary value developed and accepted by the organization for a measure you are working with.

FIGURE 4-4. When to Convert a Measure to Monetary Value

TABLE 4-3. Techniques for Data Conversion

Standard values are defined as output to contribution, quality, and time. When considering output to contribution, the value is based on an additional output. For example, organizations that work on a for-profit basis consider the profit contribution, the profit from the sale, in monetizing an additional sale. Most organizations have a profit margin readily available.

The cost of quality is another standard value in organizations. Quality is a critical issue and its cost is an important measure in most manufacturing and service firms. Placing the monetary value on some measures of quality is quite easy. For example, waste, product returns, and complaints are often monitored in organizations and already have a monetary value placed on them. Other measures, such as errors, can be converted to monetary value by looking at the cost of the work. For example, when employees make mistakes and errors in the reporting, the cost of those mistakes—the value of those mistakes—is the cost incurred in reworking the report.

The third category of standard value is employees’ time, probably the simplest and most basic approach to data conversion. If time is saved due to a program, the first question to consider is, “Whose time is it?” Then, to convert time to monetary value, take time saved multiplied by labor cost and add the percentage of additional value for employee benefits. This benefits factor can easily be obtained from the human resources department. A word of caution: When considering employee time as a benefit, the time savings is only realized when the amount of time saved is actually used for productive work. So, if a manager saves time by reducing the number of ineffective meetings the manager attends, the time saved should be applied to more work that is productive.

Historical Costs Calculation

When no standard values exist, historical costs can be utilized by considering what the incident has cost in the past. Using this technique often requires more time and effort than desired. In the end, however, it is possible to develop a credible value for a given measure. This monetary value can eventually become a standard value.

Internal and External Experts

When standard values are unavailable and developing the monetary values through historical costs is not feasible, the next option is to use internal or external experts. When using this approach, ask the expert to provide the cost for the value of one unit of improvement for the measure under investigation. Internal experts have knowledge of the situation and the respect of management. External experts are well published and have the respect of the larger community. In either case, these experts have their own methodologies to develop the values. Therefore, it is important for the experts to understand the intent and the measure with which to develop the monetary value.

External Databases

Sometimes there are no standard values or resources available to develop a monetary value using historical costs. Additionally, there are times when there is no internal expert and it is not possible to locate an external expert who can provide the necessary information. When this is the case, go to external databases. The Internet can provide a wealth of information through online databases and research. External databases provide a variety of information, including the monetary value of many different business impact measures.

Linking With Other Measures

Another technique to convert a measure to monetary value is linking the value of that measure with other measures that have already been converted to monetary values. This approach involves identifying existing relationships showing a correlation between the measure under investigation and another measure to which a standard value has been applied. In some situations, the relationship between more than two measures is connected. Ultimately, this chain of measures is traced to a monetary value. For example, job engagement is linked to sales, productivity, safety, and employee retention. Credibility of data becomes an issue when the assumptions increase as the chain of measures develops further from the actual monetary value. Using this methodology based on the monetary value of other measures is often sufficient for converting measures when calculating the ROI of programs.

Estimations

When the previous methods are unavailable or inappropriate, an estimation process is used that has been proven conservative and credible with executives in a variety of organizations. The estimates of monetary value can come from participants, supervisors, managers, and even the program staff. The process of using estimation to convert a measure to monetary value is quite simple. The data can be collected through focus groups, interviews, or questionnaires. The key is clearly defining the measure so that those who are asked to provide the estimate have a clear understanding of that measure.

FIVE STEPS TO DATA CONVERSION

When it has been decided to convert a measure to monetary value and you’ve chosen the technique that you are going to use to calculate the monetary value, follow the five steps to complete the data conversion process.

1.  Focus on the unit of measure. The first step is to review one unit of the measure under investigation. For example, if evaluating a measure of productivity, and the output is one more credit card account, then one credit card account is the unit of measure.

2.  Determine the value of each unit. In determining the value of each unit, use standard values or one of the other operational techniques. For example, if one new account is worth $500 and that figure is based on standard values using profit contribution, the value is $500 in profit.

3.  Calculate the change in the performance of the measure. Step three is actually taken during the evaluation process. For example, change in performance or the improvement in the number of credit card accounts is determined during the Level 4 evaluation. How many new credit card accounts were due to the program? For this example, assume that an average two new credit card accounts were sold per month (after isolating all other factors).

4.  Determine the annual improvement in the measure. Annualize the improvement in the measure. Remember that Guiding Principle 9 says that for short-term programs, report only first-year benefits. You do not necessarily need to wait one year to see exactly how many new credit card accounts are achieved due to the program. Rather, pick a point in time to obtain the average improvement to that date and, then, annualize that figure. In the credit card account example, the unit of measure is one account and the value of the unit is $500. After establishing that the change in performance of the measure due to the program (after isolating the effects) is averaging two new accounts per month, determine the annual improvement in the measure by simply multiplying the change in performance by 12 months. So, two new accounts per month multiplied by 12 months equals 24 new accounts due to the program.

5.  Calculate the total monetary value of the improvement. Take the number from step four, annual improvement in the measure (24 in the example), and multiply it by the value of each unit using the standard profit margin ($500 per credit card account in the example). This provides a total monetary value of improvement of $12,000. This annual monetary benefit of the technology-based learning is the value that goes in the numerator of the equation.

FULLY LOADED COSTS

The next step in the move from Level 4 to Level 5 is tabulating the fully loaded cost of the program, which will go in the denominator of the ROI equation. When taking an evaluation to Level 4 only, this step is not necessary; although, regardless of how the learning programs are evaluated, it should be common practice to know the full costs of them. Fully loaded costs mean everything. Table 4-4 shows the recommended cost categories for a fully loaded conservative approach to tabulating and estimating costs.

TABLE 4-4. Project Cost Categories

Initial Analysis and Assessment

One of the most underestimated items is the cost of conducting the initial analysis and assessment. In a comprehensive program, this involves data collection, problem solving, assessment, and analysis. In some programs, this cost is near zero because the program is conducted without an appropriate assessment. However, as more program sponsors place increased attention on needs assessment and analysis, this item will become a significant cost in the future. All costs associated with the analysis and assessment should be captured to the fullest extent possible. These costs include time, direct expenses, and internal services and supplies used in the analysis. The total costs are usually allocated over the life of the program.

Development of Solutions

One of the more significant items is the cost of designing and developing the learning program. These costs include time in both the design and development and the purchase of supplies, technology, and other materials directly related to the solution. As with needs assessment costs, design and development costs are usually fully charged to the program. However, in some situations, the major expenditures may be prorated over several programs, if the solution can be used in other programs.

Acquisition Costs

In lieu of development costs, some executives purchase solutions from other sources to use directly or in a modified format. The acquisition costs for these solutions include the purchase price, support materials, and licensing agreements. Some programs have both acquisition costs and solution-development costs. Acquisition costs can be prorated if the acquired solutions can be used in other programs.

Application and Implementation Costs

Usually, the largest cost segment in a program is associated with implementation and delivery. Eight major categories are reviewed below:

• salaries and benefits for learning team time

• salaries and benefits for coordinators and organizers

• participants’ salaries and benefits

• program materials, if applicable

• hardware/software

• travel, lodging, and meals, if blended

• facilities (even in-house meetings), if blended

• capital expenditures, if appropriate.

Maintenance and Monitoring

Maintenance and monitoring involves routine expenses to maintain and operate the program. These represent ongoing expenses that allow the new program solution to continue. These may involve staff members and additional expenses, and could be significant for some programs.

Support and Overhead

Another charge is the cost of support and overhead, the additional costs of the program not directly related to a particular program. The overhead category represents any program cost not considered in the above calculations. Typical items include the cost of administrative/clerical support, telecommunication expenses, office expenses, salaries of client managers, and other fixed costs. This is usually an estimate allocated in some convenient way based on the number of learning hours, then estimating the overhead and support needed each hour. This becomes a standard value to use in calculations.

Evaluation and Reporting

The total evaluation cost should be included in the program costs to complete the fully loaded cost. Evaluation costs include the cost of developing the evaluation strategy, designing instruments, collecting data, analyzing data, preparing a report, and communicating the results. Cost categories include time, materials, purchased instruments, surveys, and any consulting fees.

ROI CALCULATION

As explained in chapter 2, ROI is reported in one of two ways: the benefit-cost ratio (BCR) and the ROI percentage. In simple terms, the BCR compares the economic benefits of the program with the cost of the program. A BCR of 2 to 1 says that for every $1 invested, $2 are provided in benefits.

The ROI formula, however, is reported as a percentage. The ROI is developed by calculating the net program benefits divided by program costs times 100. For example, a BCR of 2 to 1 translates into the ROI of 100 percent. This says that for every $1 spent on the learning program $1 is returned, after costs are captured. The formula used here is essentially the same as ROI in other types of investments, where the standard equation is annual earnings divided by investment.

For example, if after you convert Level 4 measures to money and you follow the five steps described previously, you find that the monetary benefits of a learning program result in a sales increase of $350,000, and the learning program cost $200,000, the BCR and ROI are:

The BCR explains that for every $1 invested in the learning program, the total financial benefit returned is $1.75. The ROI explains that for every $1 invested in learning, that $1 is recovered plus a net return of $0.75. ROI is the “return” on the investment, where the BCR is the total benefit including the investment itself.

So when do you use which? Many times both metrics are reported to give both perspectives. Because the BCR comes from the public sector, it is more often used in public sector reporting. However, the ROI is also gaining traction in those settings. For private sector organizations, the ROI is the primary metric.

Occasionally, a stakeholder will ask to see the time at which an investment will “pay off.” This payoff period is the estimated time at which a program will break even. It is then assumed that any time after that period will result in added benefit. The payback period equation is simply the BCR equation turned upside down. Take the total investment of the learning program, divide it by the benefits, and multiply by 12 to get the number of months. Using the previous numbers as the basis for the example, the payback period for the initiative would be:

This indicates that in approximately seven months, you can expect to break even on the investment.

INTANGIBLE BENEFITS

As described earlier, intangible benefits are those benefits that are not converted to monetary value; but they are important and sometimes just as important as the actual ROI calculation. When reporting as a result, there must be a connection to these intangibles. Typical intangible benefits not usually converted to monetary value are job satisfaction, organizational commitment, teamwork, and customer satisfaction. These could be converted to monetary value; however, when job satisfaction, organizational commitment, teamwork, and customer satisfaction are improved, the organization is usually satisfied with the improvement in these measures and the dollar value with that improvement is not necessary. The good news is that more of these measures are now being converted to money.

When you report ROI, always balance it with the intangible benefits. This balance places the ultimate benefits of the learning program into perspective.

FINAL THOUGHTS

This chapter discussed the various important aspects of ROI analysis. After following the steps in chapter 3 to collect data, it is important to analyze the data in a credible way. Perhaps the most important aspect of analysis is the isolation of the effects of the learning program on the data. It is crucial to know how this particular program affected the data, outside other influences. Another important part of data analysis is the conversion data to monetary value, various methods were discussed. Finally, the chapter discussed intangible benefits, or those not converted to monetary values. All of these items are critical in the ROI analysis. The next chapter focuses on communicating and using the results of the analysis.