10

Measuring the ROI of a New Engagement-Based Selection Process

International Premium Hotel Group

Jack J. Phillips and Patti P. Phillips

This case was prepared to serve as a basis for discussion rather than to illustrate either effective or ineffective administrative and management practices. All names, dates, places, and organizations have been disguised at the request of the author or organization.

BACKGROUND

International Premium Hotels (IPH) owns more than 70 upscale hotels operating in 12 countries. Through its different brands, IPH prides itself on providing excellent guest satisfaction delivered by employees who have superb hospitality skills. The hotel business is a very competitive market and guest satisfaction is a critical measure. IPH charges a premium price, and for that, it wants to offer a premium service. Providing a less-than-desired service level will affect the company’s image, ultimately leading to a reduction in revenue. At the same time, IPH must also be efficient and manage costs within appropriate ranges.

The Situation

The cost of high staff turnover in the first 60 days of employment has been increasing hotel costs. Management is concerned that the quality of the new staff is not where it should be in terms of being engaged with the work and customers. The IPH executives asked the HR team to experiment with new recruiting and selection processes to decrease the turnover of employees early in their tenure while also improving guest satisfaction and employee performance. The first pilot program would be for 24 hotels in the United States. If the concept proved effective, it would be adapted to different cultures in other countries and implemented globally.

The first step was for the IPH HR staff to examine why the company was facing unusually high turnover in the first 60 days. Focus groups were conducted using a nominal group technique to arrive at the cause of turnover (Phillips and Edwards 2009). The team found that the top reasons for early turnover were that employees were:

•  not adapting to the engagement culture

•  lacking hospitality experience

•  not providing the level of engagement needed

•  in a job that wasn’t what they had expected.

The definition of the turnover measure is avoidable turnover. In addition to voluntary turnovers, other turnovers could be avoided if actions were taken to prevent terminations. These could be prevented through counseling or improved selection processes.

SOLUTION

Armed with the information from the focus groups, the following solutions were developed and approved:

•  Change the recruitment efforts to a website and recruiting source that would attract experienced hospitality candidates.

•  Administer an engagement preference instrument that matches the candidate’s value systems with the hotel’s value systems.

•  Conduct customer engagement exercises where the candidates would view a variety of videos and make a decision about what to do in each case. The assessment is designed to pinpoint a candidate’s attitude toward customer engagement and customer satisfaction.

•  Provide a realistic job preview. The last stage in the selection process assigns a new candidate to a brief rotational process lasting about two hours to provide a sense of the customer engagement culture in the hotel. This was designed to have candidates self-select out of the process if they are not comfortable with the engagement culture.

Recruiting Method

The first solution is changing the recruiting method, which involved using a new website that serves as a job match for the hospitality industry. Employers can advertise jobs and include the requirements and necessary qualifications on the site. Candidates can apply and provide background information; they are forwarded to the company if their credentials meet the requirements the company requires. IPH had previously used local recruiting sources, job ads, and other online services to recruit the candidates. This new website was expected to provide more experienced candidates for the hospitality industry.

Engagement

The second solution is an engagement fit instrument that checked validity and reliability. The instrument assesses what candidates want and need at work and compares them with the engagement profile of successful employees to see if their profiles match. If a profile doesn’t match, the candidate should be rejected. The data show that this instrument can prevent early turnover because it ensures a proper match between the engagement culture of the organization and the work values of the candidate. A partial listing of the statements candidates agree or disagree with using a five-point scale includes:

1.   I prefer to work in a stress-free environment.

2.   My work is not successful until the customer is satisfied.

3.   I work best when I can control my schedule.

4.   I take full responsibility for my results.

5.   There is a limit to what we can do to satisfy a customer.

6.   I work best in a supportive team environment.

7.   A customer referral is more important than having a satisfied customer.

8.   My work must be important.

9.   My pay increases should be in proportion to my contribution.

10. I need to be recognized for my work.

11. I prefer individual rewards to team rewards.

12. I need clear expectations for my work.

13. Every customer service issue can be resolved.

14. I need opportunities to grow at work.

15. Quality of work is more important than quantity of work.

16. I need routine feedback on my progress.

Customer Service

The third solution is a customer service simulation, where the candidates view a variety of customer service videos and are asked to indicate a course of action. The situations require a person to use judgment, knowing when and how to respond to a customer as they are empowered to take that action. At the same time, they must not be too extreme with customer wishes. IPH struggles with this issue because it wants customer service staff to be empowered to take care of the customer at almost any cost, but know when the requests and issues are on the unreasonable side.

Realistic Job Preview

A fourth solution is the realistic job preview. This is based on the assumption that individuals need to see the actual job or some parts of it in operation. That way they can self-select out of the process if it isn’t a good fit. In this situation, the individual observes as many issues as possible in a two-hour period in a real-life setting. This is not a new concept and has a history of minimizing early turnovers.

Exhibit 10-1 shows the concept of this new selection process, where the pool of applicants is initially very broad but then the hospitality source reduces that number considerably. The engagement instrument narrows it even more, and the customer service exercise even reduces it further. Finally the realistic job preview allows some participants to self-select out of the process. The result is the eligible list of candidates.

EXHIBIT 10-1. New Components of Recruiting and Selection Process

Exhibit 10-2 shows the selection system and what has been added. The process is comprehensive, systematic, and can easily be administered by each hotel’s HR staff. The goal is to process as many qualified candidates through the system as quickly as possible, ensuring that those candidates fit in the organization properly and remain for a longer period of time.

EXHIBIT 10-2. The Revised Recruiting and Selection System

IMPLEMENTATION CHALLENGES

Cost

The new processes will increase the cost of recruiting and selection. The challenge is to keep these costs to a minimum while weighing the cost versus the monetary benefit of the change. This situation creates the desire to conduct the ROI analysis.

Time to Start

This new process must be implemented efficiently so that it does not add to the time to start, which is an important measure at the hotel. The time to start is a measure of time from the point that a particular job opening has been identified and a requisition has been signed to the day when the new candidate is on the job. Time to start is currently averaging 30 days and this measure needs to be at the same number (or less), even though the extra steps of the new system are extending the selection process.

Reaction to Hotel Staff

It is important for the HR staff and executives at each hotel to see the need for the change in recruitment and selection. They must see it as important to the success of the hotel and necessary to enhance guest satisfaction. They must also see the need to make the change based on the current level of performance and retention, which is linked to the selection of improper candidates.

Consistent Implementation

This process must be administered consistently and efficiently. One of the requirements of having a valid selection system is that the recruiting and selection process is always consistent. There is no option to administer some parts of the process but not others. For example, the same amount of time should be allocated for the realistic job previews each time a candidate participates in this step. But candidates must also proceed efficiently through the selection system.

OBJECTIVES

The program’s objectives are critical to maintain the focus on the solutions. The objectives are set at five levels.

Reaction

Based on the analysis and implementation issues described already, the following objectives were developed for reaction. After implementing this program, the HR recruiting and selecting team will see this program as important to the company’s success, important to guest satisfaction, necessary for recruiting qualified candidates, and one they intend to implement properly.

Learning

After initiating this project, the stakeholders will be able to properly use the new recruiting source, the engagement assessment instrument, a simulation on customer service, and the realistic job preview.

Application

When the project is implemented:

•  The new recruiting source will be used appropriately and routinely to attract candidates.

•  The engagement survey will be administered timely and consistently for each applicant.

•  The customer service simulation will be administered under consistent conditions.

•  The job preview will be implemented each time under the same conditions.

Impact

After the program is fully implemented:

•  The avoidable turnover rate (annualized) in the first three months of employment will be reduced from 28 percent to 18 percent, which is a reduction of 10 percentage points.

•  Guest satisfaction will be maintained or improved.

•  The performance of new employees will be improved one half of one point on a five-point scale.

•  The time to start measure will not exceed 30 days.

•  Employee job engagement will improve.

ROI

The ROI objective for this project is set at 20 percent, which is the minimum acceptable performance.

DATA COLLECTION

Exhibit 10-3 shows a data collection plan for this project. First a questionnaire will be administered to the HR team that is involved in the recruiting, testing, assessment, and selection processes. This questionnaire will capture reaction and learning data three weeks after the new system is implemented, using self-assessment input that explores any issues that are surfacing at this point. Two months after implementation, another questionnaire will be administered to ensure that all the four processes are operating properly.

For Level 4 assessments, the source of turnover and time to start data will be in the records. The rest of the impact measures will be in a questionnaire administered in four months. By then, the impact should materialize and this questionnaire will also be used to isolate the effects of this program on the impact data. Exhibit 10-4 shows the data integration by level.

EXHIBIT 10-3. Data Collection Plan for the New Recruiting and Selection System

EXHIBIT 10-4. Data Summary and Integration

ROI ANALYSIS PLAN

Exhibit 10-5 shows a completed ROI analysis plan. The plan starts with business impact data identified from the data collection (Level 4) and presents a method of isolation. For this project, an experimental versus control group would not be possible because immediate implementation was planned for all U.S. hotels. American executives didn’t want to withhold the processes for some hotels to see the effect on the control group because they wanted to resolve the problem as quickly as possible. Hotels outside the United States would not necessarily be a good match for use as a control group because of differences in global cultures. Instead, the HR team decided to use two other methods: trend line analysis and estimates from the stakeholders involved in the process. The trend line would work only if no other new process was implemented that would affect the impact measures during the four-month period. The estimations were a fallback approach.

The team relied on an acceptable standard value for the cost of turnover to convert the data to money. Regarding guest satisfaction, the hotel had some data that indicated the monetary payoff of improving guest satisfaction, but it was not considered to be credible. Employee performance, time to start, and employee engagement were not converted to money because of the difficulty of developing a credible value.

The costs of the four new processes are fully loaded. Some additional intangibles are anticipated, which means that some information must be collected to see the extent in which these intangibles are connected to the program. The groups for communicating results are typical, with particular emphasis placed on briefing with hotel executives and providing information to the key stakeholders.

EXHIBIT 10-5. Completed ROI Analysis Plan

RESULTS

The results are presented by the different levels, beginning with Level 1, moving through to Level 4 and 5 analyses, and ending with intangibles.

Reaction

The reaction results are presented in Exhibit 10-6. The reaction was satisfactory except for one issue: The individuals who are involved in recruiting and selection didn’t see how the project would affect guest satisfaction. Because of this, the project team put some extra effort into communications with the recruiting and selection staff to show them the connection between having a staff with a culture of customer service and guest satisfaction. Although the solution may prevent turnover in the early stages of employment, a mismatch early on in the hiring process can create problems with guest satisfaction. Having an employee with a longer tenure also provides continuity and continues to build client satisfaction.

EXHIBIT 10-6. Reaction Results

Participants Rate the Following:	Score
This program is important to hotel success	4.0
This program is important to guest satisfaction	3.2
This program is necessary for recruiting qualified candidates	4.5
This team intends to implement processes properly	4.1
	Average response = 4.0

Scale: 1 = Not at all, 2 = A little, 3 = Some, 4 = Much, 5 = Very much

Learning

Learning was obtained using a simple self-assessment on a questionnaire. The results are captured in Exhibit 10-7. The participants indicated that they could describe each of the new processes and use them properly. While there were no surprises at this level, the lowest ratings were for the customer service simulation and the realistic job previews. These were a little more complicated than the change in recruiting source and the engagement survey assessment. The excellent results at this level may be a reflection of the fact that the objectives were clearly defined.

Application

Exhibit 10-8 shows the application data and reveals that the new steps were being operated properly and consistently, except for the realistic job preview.

EXHIBIT 10-7. Learning Results

Participants are able to:	Score
Describe the new recruiting source and website	4.2
Use the new recruiting source and website	4.1
Use the engagement assessment	4.3
Describe the customer service simulated	3.9
Use the customer service simulated	4.0
Describe the realistic job preview process	3.8
Use the realistic job preview process	3.9
	Average response = 4.1

Scale: 1 = Not at all, 2 = Some, 3 = Moderate amount, 4 = Significant amount, 5 = Very significant amount

EXHIBIT 10-8. Application Results

Participants are able to:	Score
The new recruiting source is being used properly	4.2
The new recruiting source provides a steady stream of candidates	4.6
The engagement survey is administered for every candidate	4.7
The engagement survey is administered under the same conditions for each remaining candidate	4.6
The customer service simulation is administered	4.0
The customer service simulation is administered under the same conditions	4.1
The realistic job preview is administered for each remaining candidate	3.7
The realistic job preview is administered under the same conditions	3.8
The four new processes are successful	4.3

Scale: 1 = Not at all, 2 = Occasionally, 3 = Sometimes, 4 = Often, 5 = Always

Exhibit 10-9 shows the barriers to use and Exhibit 10-10 shows the enablers that support the use. The barriers reveal that the realistic job preview is a problem, with nearly 70 percent having difficulty with it. The issue is also connected to the second barrier. However, the enablers were very positive and encouraging.

EXHIBIT 10-9. Barriers to Effective Use

Barriers to Use	Percent Responding
The realistic job preview is difficult to execute	39%
The process takes too much time	28%
Candidates resist the two assessments	17%
It’s difficult to be consistent each time	8%
Other	14%

EXHIBIT 10-10. Enablers to Effective Use

Enablers to Use	Percent Responding
Each step is easy to use	64%
General manager supports this process	61%
Candidates appreciate the thoroughness of our process	52%
The process is very interesting and engaging	41%
It seems to fit our engagement culture	19%
Other	17%

Exhibit 10-11 shows the efficiencies that are coming through the system with these changes. Previously, only 54 percent of applicants made it through the resume review. With this new system, 81 percent are making it through the resume review, underscoring that the recruiting source is providing more qualified candidates. Thus, out of 100 initial applications, 81 move on to the engagement assessment round, which eliminates another 29 applications. The customer service simulation drops another 10, but the realistic job preview only removes one. So in essence, this process provides better candidates in the beginning, eliminating 40 percent of applicants that would potentially have left the organization in the early stages of employment. These are impressive results, with the engagement survey having the most impact thus far. In total, 79 of the 100 candidates who applied were eliminated from consideration. The engagement survey accounted for 29 (37 percent) of the total eliminations.

EXHIBIT 10-11. Recruiting and Selecting Efficiencies

Business Impact

The early turnover data are presented in Exhibit 10-12 for the four months prior to the program and four months after the program. As expected, the annualized turnover data are reduced.

The isolation by trend line analysis seems to work for this data set. It shows that 28 percent turnover would be expected, but after four months the actual turnover is 16 percent. The HR team indicated that the preprogram trend would probably have continued if nothing had changed in the system. More importantly, the HR team could not identify any other new influences that would have caused this improvement. Therefore it is safe to use the trend line analysis as a method of isolation.

Time to start is an important measure and Exhibit 10-13 shows the details. The program reduced the time to start to 27 days, compared with about 30 days prior to the program. However, there was already a downward trend in place the when the program started, and the projected trend line shows that time to start may have hit 27 days even without the program. This means that the program may have actually added more days to what the time to start could have been.

However, the team could not explain the downward trend on the program data, or the fact that it would have continued. Consequently, the HCA team suggested that the objective had been met.

EXHIBIT 10-12. Trend Line Analysis for Early Turnover

EXHIBIT 10-13. Trend Line Analysis for Time to Start

Converting Data to Money

In total, the company was expecting to hire 255 employees in the 24 U.S. hotels during the year of this study. To calculate the cost of early turnover, the hotel had been using 40 percent of annual pay, which was a number accepted by the management team. This is for early turnover, where the investment in new employees is not as much as for longer-term employees and the loss of knowledge and experience is not very high. However, there are still up-front variable costs related to replacements, so 40 percent is a conservative number that covers the cost of recruiting, selection, employment, onboarding, and initial training before new employees become productive. It also includes the cost of the disruption and the bottlenecks caused by turnover, as well as the time involved in addressing the turnover situation. Exhibit 10-14 shows the calculation for the cost savings of the reduction in turnover.

EXHIBIT 10-14. Turnover Analysis

255 Employees Turnover improvements: 28 - 16 = 12% annual 255 × 0.12 = 31 employees Average salary: $36,000 Turnover cost: 40% of salary 36,000 × 0.4 = 14,400 Annual = 14,400 × 31 = $446,400

The time to start decreased slightly but wasn’t converted to monetary value. It is left as an intangible.

Cost

The fully loaded cost of these four solutions, including the solution and purchase instruments, is shown in Exhibit 10-15. The team was very anxious to make sure all cost were included, both direct and indirect. Some of the costs included ongoing cost in the future because these services are being used for the recruiting database, instrument, and customer service simulation. The question that surfaced was how much cost to include. In the very beginning of the project, it was determined that this would be considered a short-term solution, which means the benefits would be accumulated for one year and the early turnover would be extrapolated for an entire year when the impact was actually determined. In other words, after four months, the change in turnover rate would be extrapolated for the year to arrive at the benefits in the previous section.

A case could easily have been made for this to be a longer-term solution, which would allow two or even three years of benefits to be used. However, the team decided to use one year to increase the credibility of the data.

EXHIBIT 10-15. Fully Loaded Costs of Four Solutions

Cost of Four Solutions at 24 Hotels (Annualized and Prorated)
Needs Assessment
Prorated Over Five Years	$1,200
Development and Purchase
Customer Service	15,000
Realistic Job Previews	8,000
Lease and Royalties
Recruiting Source (Annual)	20,000
Engagement (Annual)	17,200
Customer Service Simulation (Annual)	23,500
Learning Sessions for Recruiting and Selection Team	4,300
Time to Use the Process
Recruiting Source	Less than previous
Engagement	2,200
Administrative Issues
Coordination, Supplies, etc.	15,800
Customer Services	4,400
Realistic Job Preview	22,000
Evaluation	4,500
Total	$138,100

Finally the BCR and ROI were calculated, as shown below. Monetary benefits for one year were determined to be $446,400, whereas the cost for one year of the program was $138,100.

The BCR and the ROI are extremely high, making it difficult for people to accept. However, the impact data are credible because they come directly from the system. The high early turnover rate was substantially reduced, and the monetary benefits are realistic because management agreed on the cost of turnover. In addition, the program was not very expensive, particularly when considering that all of the costs for the four processes are included. The HCA team felt good that this could be defended in the communication of results.

Intangibles

This study influenced some important intangibles, with three measures that were initially identified as impact turning out to be intangibles. The time to start was not converted to money, as mentioned earlier, although there were certainly some costs avoided with improvement in that measure. Time to start was left intangible because it could not be developed credibly with a reasonable amount of resources.

The records of the individuals participating in the program were reviewed to determine employee performance. The performance was higher and slightly exceeded the requirement for one-half of one point in performance improvement. The average performance improvement rating increased from 3.8 to 4.4 on a five-point scale, where 3 is the average performance rating.

It was impossible to connect guest satisfaction directly to this group of employees. Although each hotel has guest satisfaction measures, those numbers aren’t broken down by individual person or even function. However, on the questionnaire the HR team asked employees to indicate the extent to which the program had improved guest satisfaction; they were able to connect the ranking of 4.1 on a five-point scale to guest satisfaction. Other intangible measures included reputation of the hotel, brand awareness, and recruiting image (Exhibit 10-16).

EXHIBIT 10-16. Intangible Benefits Connected to This Program

Intangible Benefits
Time to Start	Slight Reduction
Employee Performance	3.8–4.4
Guest Satisfaction	4.1
Recruiting Image	4.3
Brand Awareness	3.8
Hotel Reputation	3.2

Scale: 1 = No influence, 2 = Some influence, 3 = Moderate influence, 4 = Significant influence, 5 = Very significant influence

CONCLUSIONS AND RECOMMENDATIONS

This program was a success in terms of its overall connection with the impact measures. However, the realistic job preview doesn’t seem to be adding value, which is addressed in the recommendation. Data collection was smooth and efficient, participation was high, barriers were minimal, and enablers were very strong. On a conservative basis, the program adds tremendous value, with most of the improvement coming from the engagement survey. The program also underscores the fact that management support was extremely high, because the HCA team was handed a problem, but not the specific solution. Although management hinted at the solution—something had to be changed in the assessment, selection, and sourcing process to stem the turnover—the specific solution was left up to the team to uncover. This works much better, instead of having the management team implement or bring a predetermined solution, only to find out later that it was not the right one. By planning for the ROI in advance, this project brought a tremendous focus on the outcome, provided for early detection of problems, and helped achieve the results that were ultimately delivered.

In terms of recommendations, the HCA team recommended that this concept be applied to hotels in other countries, but in the context of the culture of that country. This may be more expensive because of language, labor laws, and the cultures of the different countries. For some, it might not even be worth it. However, each country should be evaluated individually and at least some parts of the process should be considered.

Additionally, the realistic job preview did not have a significant effect, with only one person self-selecting out in the four months of operation results. Given the cost of this process, the difficulties the hotel HR team is having with this task, and the amount of time it takes for other hotel staff to be involved, the HR team recommended that this part be dropped from the process going forward.

COMMUNICATION PLAN

With results in hand, the first stop was the executive group. The HR team gave the U.S. hotel executives a briefing during one of their regularly scheduled meetings. This 30-minute briefing gave the team the opportunity to explain the methodology that was used, describe their results, and gain approval for their recommendations. This briefing was carefully planned and orchestrated and was successful in meeting the agenda and achieving its purposes. Beyond that, a summary and full report was sent to the HR team involved in the project. This was followed with a webinar briefing, which used the same set of slides as the executive group briefing. In addition, all the hotel general managers were given a summary, a complete report, and a webinar on the results, with opportunities for Q&A. This hour and a half session accomplished the same goals as the executive briefing, but included a little more detail about operational issues at the hotel.

In addition, the team wrote a brief article for the employee newsletter explaining the study, what was done, and results that were achieved. A complete study was also made available to the HR managers at all the hotels in the United States and internationally. A webinar was offered to this complete group and was conducted in two sessions, one with the U.S. HR managers, and the other with those outside the United States, with some discussion around the applicability and feasibility of the same process in their country.

Finally, a brief paragraph of this study was included in the quarterly financial report that was released the quarter following the executive briefing. This was designed to alert investors of the progress being made to make the organization more successful in terms of guest satisfaction and controlling cost.

LESSONS LEARNED

The lessons learned from this program are very straightforward:

•  It is best to have a problem to work with so the team can find the right solution. This is in contrast to many requests when the solution is brought to the HR team.

•  The decision to measure the impact and ROI was made early, at the time of the analysis. This helped in planning the study, making it more focused, and keeping the attention on results throughout the implementation. This approach actually enhanced the results that were achieved.

•  Excellent support of the management group made it successful. The hotel general managers saw the problem, knew it was expensive, and were pleased to see the solution.

•  This type of analysis does take a little bit of time, but when it is all considered, it is worth it. The extra efforts are actually necessary steps in the process.

QUESTIONS FOR DISCUSSION

1.   Is this study credible? Please explain.

2.   Are there other potential solutions to this problem? Please explain.

3.   Critique the data collection. Are there other ways to collect data? If so, explain.

4.   What others ways could be used to isolate the effects of the program?

5.   Could the effects of the engagement be isolated from other parts of the solution? If so, how could this be accomplished?

6.   Could the ROI from the engagement survey be calculated? If so, how?

7.   Should this be a long-term or short-term solution? Explain.

8.   Could you defend this study with your executives? Please explain.

REFERENCE

Phillips, J.J., and L. Edwards. 2009. Managing Talent Retention: An ROI Approach. San Francisco: John Wiley.