2

A System for Measuring the Impact and ROI

While employee engagement is increasingly observed as a critical factor in organizations, the implications of measurement are also significant. To build a measurement process, certain steps need to be taken. It is in this context that the ROI Methodology is presented. In this chapter, we address using the ROI Methodology with engagement and how engagement teams are a conduit of change in building a measurement system.

HOW AND WHY ENGAGEMENT FITS WITH ROI

Engagement practitioners tend to be more socially driven (to help others) rather than financially driven. So what would drive an engagement practitioner to pursue ROI? Perhaps this is best answered by considering how and why ROI helps the process, and ultimately the client, by showing how the engagement has yielded the desired change.

The Engagement Practitioner as a Change Agent

Practitioners often view their role as one who influences an individual, group, or organization to a more desired change. The change agent plays a significant role in leading the change effort or collaborating with the team assigned to initiating change. Trying to create an environment that is measurement friendly also involves a change agent—someone to lead this effort and manage the change process within an organization.

It is important to remember that building a measurement system should be a strategic change. The change agent must set the stage with the “why” behind building a measurement culture, make sure that the change effort is in sync with what’s important for the organization, and include action planning and feedback to keep the momentum building. It is also helpful to involve people who are senior in the organization because they have the clout necessary to pave the way for building a measurement system.

Identify a System to Routinely Review Measures

Adopting a systematic way to plan, collect, analyze, and report on programs in the organization, such as the ROI Methodology, begins the process of communicating and reinforcing what is important to the organization, while sending a clear message to key stakeholders as to what needs to change to improve outcomes. This is particularly true when measurement has been planned in advance to collect data points that tell the story in a comprehensive way. Exhibit 2-1 shows a clear delineation between activity-based and results-based initiatives. The old “if you build it, they will come” mentality can be challenged through a series of filtering questions:

•  Is this initiative aligned with business impact or organization effectiveness outcomes?

•  Is there an assessment of performance that shows a gap in performance?

•  Is the work environment prepared to reinforce the implementation of engagement?

•  Have partnerships been established with key stakeholders to support this initiative?

•  Are there specific, measurable objectives for expected behavior change and business impact?

When employee engagement programs are aligned with results-based initiatives or what’s important to the organization, it becomes more likely that engagement programs are easily measured and supported. The old adage rings true in this context: What gets measured gets done.

EXHIBIT 2-1. Activity-Based Versus Results-Based Approach to Engagement

The ROI Methodology: A System for Accountability

The role of measurement and evaluation is crucial for establishing the impact and credibility of employee engagement. It is time for the field to fully accept its roots in a data-driven approach and understand the value inherent in measuring how and what we do. Several features about return on investment make it an effective measure for engagement:

•  To show bottom-line results for engagement. Return on investment represents the ultimate range of measurement—a comparison of the actual cost of a project with its monetary benefits. This is done by using the same standard ratio that accountants have used for years to show the return on investment for a variety of investments, such as technology, equipment, and buildings.

•  Return on investment has a rich history of application. The ROI Methodology is not a passing trend—it is a measure of accountability that has been in place for centuries. When resources are invested to address a business need, the ROI Methodology shows the financial impact of the investment.

•  To speak the same language as senior management. Most managers have knowledge and skills for managing a business; some have degrees in business administration. These managers understand the need for a process to establish solid business cases and calculate a return on investment. They use ROI for a variety of projects and are fluid in carrying on conversations that measure the monetary results from large investments.

•  Return on investment generates a high degree of attention among key stakeholders. Positive ROI outcomes create buzz and attention, particularly when the value exceeds expectations. Most stakeholders involved in engagement programs intuitively believe that the programs add value. But, they can use return on investment as a credible and valid measurement tool to confirm this hunch.

•  Using ROI Methodology forces the issue of strategic alignment. By following the steps in the ROI process and conducting diagnostics with the multilevel framework for understanding business and performance needs, engagement will be more closely aligned with the strategic and operational needs of the business.

Gone are the days of indiscriminately increasing investment in human capital without any evidence regarding its impact on the business. When budget cuts are being made, human capital often rises to the top, and engagement projects are no exception.

These five factors are foundational for engagement practitioners to rethink the use of ROI Methodology and to implement this type of evaluation in specific projects. By using ROI Methodology, stakeholder groups will receive a comprehensive set of significant and balanced information about the success of an employee engagement initiative.

TYPES OF DATA FOR THE ROI METHODOLOGY

At the heart of the ROI Methodology is the variety of data that are collected throughout the process and reported at different intervals. Sometimes data are assigned a level because they reflect a successive effect in which one type of data affects the next. A number of tasks and fields use very logical steps of succession, and their use in a sequence can be linked to a variety of guidelines and models. The medical field, for example, uses levels in running and analyzing blood work. This not only helps the clinician understand the categories represented by the levels, it also helps the patient understand the results. The ROI Methodology is based on levels of evaluation, as shown in Exhibit 2-2.

As the evaluation moves to the higher levels, the value ascribed to the data by the client increases. Accordingly, the degree of effort and cost of capturing the data for the higher levels of evaluation also generally increase. With proper project planning and preparation, costs can be minimized.

Project Input Data

Level 0, inputs and indicators, represents a category of data that reveals the volume, time, and cost of employee engagement programs. It includes the number of people involved and the time of their involvement, representing a fully loaded cost profile. It reflects all direct and indirect costs. Level 0 data do not represent outcome data, but they are important because they represent the investment in engagement.

EXHIBIT 2-2. Six Categories of Data

Reaction Data

The first category of outcome data collected from a program is basic reaction data (Level 1 evaluation). This type of data represents the immediate reaction to the program from a variety of key stakeholders, particularly participants who have the responsibility to make it work. At this level, a variety of basic reaction measures are taken, often representing five to 15 separate measures to gain insight into the value, importance, relevance, and usefulness of the employee engagement programs.

Learning Data

As the employee engagement programs continue, new information is acquired and new skills are learned. This level of measurement (Level 2) focuses on the changes in knowledge and skill acquisition, and details what still needs to be learned. Some programs have a high learning component, such as job design. Others may have a low learning component, such as brief engagement sessions. In some cases, the focus is on organizational learning or departmental skill development.

Application and Implementation Data

Application and implementation are key measures that show the extent to which employees are engaged, behavior has changed, and employee performance has improved. This type of data reflects how actions are taken, adjustments are made, new skills are applied, habits are changed, and steps in a new process are initiated as a result of the employee engagement programs.

This is one of the most powerful categories because it uncovers not only the extent to which the employee engagement programs are implemented, but also the reasons for lack of success. At this level, barriers and enablers to application and implementation are detailed, and a complete profile of performance change at the various steps of implementation is provided.

Business Impact Data (Tangible and Intangible)

As employees become more engaged, behavior change or actions taken in application and implementation have consequences. These can be described in one or more me sures representing an influence on the work environment, such as a direct impact to an individual, team, or department, or as an impact to other parts of the organization.

This level of data (Level 4) reflects the specific business impact and may include measures—such as output, quality, costs, time, job satisfaction, and customer satisfaction—that have been influenced by the application and implementation of the employee engagement program. A direct link between the business impact and the program must be established for the program to drive business value. At this level of analysis, a technique must be used to isolate the effects of the program from other influences that may be driving the same measure. Answering the following question is imperative: How do you know it was the employee engagement program that caused the improvement and not something else?

Chain of Value

Intangible data consist of measures that are not converted to monetary value. In some cases, converting certain measures to monetary values is not credible with a reasonable amount of resources. In these situations, data are listed as an intangible, but only if they are linked to the engagement program.

ROI Data

This level of measurement compares the monetary value of the business impact measures with the actual cost of the program. It is the ultimate level of accountability and represents the financial impact directly linked with the program, expressed as a benefit-cost ratio (BCR) or return-on-investment percentage. This measure is the fifth level of evaluation. It requires converting business impact data to monetary value and comparing that value with the fully loaded cost of the program.

Satisfaction leads to learning, which leads to application, which leads to business impact, and ultimately to return on investment. At the business impact level, the effects of the program must be isolated from other influences. In addition, business impact data are converted to monetary value and compared with the cost of the program to develop the return on investment. Exhibit 2-3 shows this connection as a chain of impact, which is necessary to drive business value. Stakeholders will more readily understand this chain of impact as they consider the long-term success of employee engagement. It is a novel yet pragmatic way to show results.

Fortunately, the ROI Methodology works extremely well in all types of environments and projects. The first level is critical, as a program would likely be unsuccessful if an adverse reaction occurred. An element of learning is also required to make a program successful: Those participating usually acquire knowledge and skills, and some projects even require significant skill development. However, learning does not guarantee success. Follow-up is needed to ensure that the knowledge and skills are being used appropriately. Therefore, application and implementation are critical for effectiveness; failure in these areas is typically what causes program failure overall.

The most important data set for those who sponsor projects is the impact, which is the consequence of application and is often expressed in business terms as output, quality, costs, and time. However, showing the impact of a program isn’t enough for some executives. They want the ultimate level of accountability: return on investment. ROI converts the amount of the improvement at the impact level (attributed to the program) to money, and compares that with the cost of the program.

EXHIBIT 2-3. Chain of Impact

SELECTING PROGRAMS FOR ROI ANALYSIS

Every employee engagement program should be evaluated in some way, even if it’s only collecting reaction data from those involved in the program. Reaction data alone may be sufficient for evaluating some programs, but the challenge is to collect additional data at higher levels, and to do so only when it is relevant and feasible.

Appropriate evaluation levels are usually determined when the program is initiated, recognizing that the evaluation level may change throughout the life of the program. If the cost of the program increases, sponsors may ask for an evaluation of the impact (Level 4) or even ROI (Level 5) of the program. Because of the resources required and the realistic barriers for ROI implementation, ROI analysis should be used only for those programs that are very expensive, linked to strategic objectives, important for solving organizational problems, and highly visible.

A comprehensive employee engagement program usually meets most, if not all, of these criteria. Deciding which type of evaluation to use is sometimes a trade-off depending on the resources available and the amount of disruption allowed for collecting the data. Because some data collection at this level may disrupt work at varying degrees or inconvenience those involved in some way, the evaluation needs to be balanced with the time, effort, and resources that can be committed to the process. Many organizations fall short of the ideal evaluation, instead settling for a feasible approach within existing constraints.

ROI PROCESS MODEL

Measurement and evaluation must be systematic, following a routine process that can be duplicated in a variety of projects. The ROI Process Model is a 10-step process, illustrated in Exhibit 2-4 (on the next page). The process begins with the end in mind by creating objectives, and proceeds until an impact report is generated. It is highly adaptable to the needs of the project in question and the evaluation can stop at any point along the process. The data collected during the program at Level 1 and Level 2, and data collected after the program at Levels 3 and 4, are steps along the way. This process will be explained in further detail in chapters 3 through 6.

FINAL THOUGHTS

This chapter presented a measurement culture and explained why that culture is so important in the context of employee engagement. Building on the premise that a measurement culture matters, this chapter explored ways for employee engagement practitioners to become change agents and build a measurement culture within the organizations in which they work. Finally, an introduction to the ROI Methodology was provided as a process to build a measurement culture.

EXHIBIT 2-4. ROI Methodology Process Model

From ROI Institute. All rights reserved.